]>
Commit | Line | Data |
---|---|---|
1a4d82fc JJ |
1 | ============================= |
2 | How To Validate a New Release | |
3 | ============================= | |
4 | ||
5 | .. contents:: | |
6 | :local: | |
7 | :depth: 1 | |
8 | ||
9 | Introduction | |
10 | ============ | |
11 | ||
12 | This document contains information about testing the release candidates that will | |
13 | ultimately be the next LLVM release. For more information on how to manage the | |
14 | actual release, please refer to :doc:`HowToReleaseLLVM`. | |
15 | ||
16 | Overview of the Release Process | |
17 | ------------------------------- | |
18 | ||
19 | Once the release process starts, the Release Manager will ask for volunteers, | |
20 | and it'll be the role of each volunteer to: | |
21 | ||
22 | * Test and benchmark the previous release | |
23 | ||
24 | * Test and benchmark each release candidate, comparing to the previous release and candidates | |
25 | ||
26 | * Identify, reduce and report every regression found during tests and benchmarks | |
27 | ||
28 | * Make sure the critical bugs get fixed and merged to the next release candidate | |
29 | ||
30 | Not all bugs or regressions are show-stoppers and it's a bit of a grey area what | |
31 | should be fixed before the next candidate and what can wait until the next release. | |
32 | ||
33 | It'll depend on: | |
34 | ||
35 | * The severity of the bug, how many people it affects and if it's a regression or a | |
36 | known bug. Known bugs are "unsupported features" and some bugs can be disabled if | |
37 | they have been implemented recently. | |
38 | ||
39 | * The stage in the release. Less critical bugs should be considered to be fixed between | |
40 | RC1 and RC2, but not so much at the end of it. | |
41 | ||
42 | * If it's a correctness or a performance regression. Performance regression tends to be | |
43 | taken more lightly than correctness. | |
44 | ||
45 | .. _scripts: | |
46 | ||
47 | Scripts | |
48 | ======= | |
49 | ||
50 | The scripts are in the ``utils/release`` directory. | |
51 | ||
52 | test-release.sh | |
53 | --------------- | |
54 | ||
55 | This script will check-out, configure and compile LLVM+Clang (+ most add-ons, like ``compiler-rt``, | |
56 | ``libcxx`` and ``clang-extra-tools``) in three stages, and will test the final stage. | |
57 | It'll have installed the final binaries on the Phase3/Releasei(+Asserts) directory, and | |
58 | that's the one you should use for the test-suite and other external tests. | |
59 | ||
60 | To run the script on a specific release candidate run:: | |
61 | ||
62 | ./test-release.sh \ | |
63 | -release 3.3 \ | |
64 | -rc 1 \ | |
65 | -no-64bit \ | |
66 | -test-asserts \ | |
67 | -no-compare-files | |
68 | ||
69 | Each system will require different options. For instance, x86_64 will obviously not need | |
70 | ``-no-64bit`` while 32-bit systems will, or the script will fail. | |
71 | ||
72 | The important flags to get right are: | |
73 | ||
74 | * On the pre-release, you should change ``-rc 1`` to ``-final``. On RC2, change it to ``-rc 2`` and so on. | |
75 | ||
76 | * On non-release testing, you can use ``-final`` in conjunction with ``-no-checkout``, but you'll have to | |
77 | create the ``final`` directory by hand and link the correct source dir to ``final/llvm.src``. | |
78 | ||
79 | * For release candidates, you need ``-test-asserts``, or it won't create a "Release+Asserts" directory, | |
80 | which is needed for release testing and benchmarking. This will take twice as long. | |
81 | ||
82 | * On the final candidate you just need Release builds, and that's the binary directory you'll have to pack. | |
83 | ||
84 | This script builds three phases of Clang+LLVM twice each (Release and Release+Asserts), so use | |
85 | screen or nohup to avoid headaches, since it'll take a long time. | |
86 | ||
87 | Use the ``--help`` option to see all the options and chose it according to your needs. | |
88 | ||
89 | ||
90 | findRegressions-nightly.py | |
91 | -------------------------- | |
92 | ||
93 | TODO | |
94 | ||
95 | .. _test-suite: | |
96 | ||
97 | Test Suite | |
98 | ========== | |
99 | ||
100 | .. contents:: | |
101 | :local: | |
102 | ||
103 | Follow the `LNT Quick Start Guide <http://llvm.org/docs/lnt/quickstart.html>`__ link on how to set-up the test-suite | |
104 | ||
105 | The binary location you'll have to use for testing is inside the ``rcN/Phase3/Release+Asserts/llvmCore-REL-RC.install``. | |
106 | Link that directory to an easier location and run the test-suite. | |
107 | ||
108 | An example on the run command line, assuming you created a link from the correct | |
109 | install directory to ``~/devel/llvm/install``:: | |
110 | ||
111 | ./sandbox/bin/python sandbox/bin/lnt runtest \ | |
112 | nt \ | |
113 | -j4 \ | |
114 | --sandbox sandbox \ | |
115 | --test-suite ~/devel/llvm/test/test-suite \ | |
116 | --cc ~/devel/llvm/install/bin/clang \ | |
117 | --cxx ~/devel/llvm/install/bin/clang++ | |
118 | ||
119 | It should have no new regressions, compared to the previous release or release candidate. You don't need to fix | |
120 | all the bugs in the test-suite, since they're not necessarily meant to pass on all architectures all the time. This is | |
121 | due to the nature of the result checking, which relies on direct comparison, and most of the time, the failures are | |
122 | related to bad output checking, rather than bad code generation. | |
123 | ||
124 | If the errors are in LLVM itself, please report every single regression found as blocker, and all the other bugs | |
125 | as important, but not necessarily blocking the release to proceed. They can be set as "known failures" and to be | |
126 | fix on a future date. | |
127 | ||
128 | .. _pre-release-process: | |
129 | ||
130 | Pre-Release Process | |
131 | =================== | |
132 | ||
133 | .. contents:: | |
134 | :local: | |
135 | ||
136 | When the release process is announced on the mailing list, you should prepare | |
137 | for the testing, by applying the same testing you'll do on the release candidates, | |
138 | on the previous release. | |
139 | ||
140 | You should: | |
141 | ||
142 | * Download the previous release sources from http://llvm.org/releases/download.html. | |
143 | ||
144 | * Run the test-release.sh script on ``final`` mode (change ``-rc 1`` to ``-final``). | |
145 | ||
146 | * Once all three stages are done, it'll test the final stage. | |
147 | ||
148 | * Using the ``Phase3/Release+Asserts/llvmCore-MAJ.MIN-final.install`` base, run the test-suite. | |
149 | ||
150 | If the final phase's ``make check-all`` failed, it's a good idea to also test the | |
151 | intermediate stages by going on the obj directory and running ``make check-all`` to find | |
152 | if there's at least one stage that passes (helps when reducing the error for bug report | |
153 | purposes). | |
154 | ||
155 | .. _release-process: | |
156 | ||
157 | Release Process | |
158 | =============== | |
159 | ||
160 | .. contents:: | |
161 | :local: | |
162 | ||
163 | When the Release Manager sends you the release candidate, download all sources, | |
164 | unzip on the same directory (there will be sym-links from the appropriate places | |
165 | to them), and run the release test as above. | |
166 | ||
167 | You should: | |
168 | ||
169 | * Download the current candidate sources from where the release manager points you | |
170 | (ex. http://llvm.org/pre-releases/3.3/rc1/). | |
171 | ||
172 | * Repeat the steps above with ``-rc 1``, ``-rc 2`` etc modes and run the test-suite | |
173 | the same way. | |
174 | ||
175 | * Compare the results, report all errors on Bugzilla and publish the binary blob | |
176 | where the release manager can grab it. | |
177 | ||
178 | Once the release manages announces that the latest candidate is the good one, you | |
179 | have to pack the ``Release`` (no Asserts) install directory on ``Phase3`` and that | |
180 | will be the official binary. | |
181 | ||
182 | * Rename (or link) ``clang+llvm-REL-ARCH-ENV`` to the .install directory | |
183 | ||
184 | * Tar that into the same name with ``.tar.gz`` extensioan from outside the directory | |
185 | ||
186 | * Make it available for the release manager to download | |
187 | ||
188 | .. _bug-reporting: | |
189 | ||
190 | Bug Reporting Process | |
191 | ===================== | |
192 | ||
193 | .. contents:: | |
194 | :local: | |
195 | ||
196 | If you found regressions or failures when comparing a release candidate with the | |
197 | previous release, follow the rules below: | |
198 | ||
199 | * Critical bugs on compilation should be fixed as soon as possible, possibly before | |
200 | releasing the binary blobs. | |
201 | ||
202 | * Check-all tests should be fixed before the next release candidate, but can wait | |
203 | until the test-suite run is finished. | |
204 | ||
205 | * Bugs in the test suite or unimportant check-all tests can be fixed in between | |
206 | release candidates. | |
207 | ||
208 | * New features or recent big changes, when close to the release, should have done | |
209 | in a way that it's easy to disable. If they misbehave, prefer disabling them than | |
210 | releasing an unstable (but untested) binary package. |