]>
Commit | Line | Data |
---|---|---|
970d7e83 LB |
1 | ================================= |
2 | LLVM Testing Infrastructure Guide | |
3 | ================================= | |
4 | ||
5 | .. contents:: | |
6 | :local: | |
7 | ||
8 | .. toctree:: | |
9 | :hidden: | |
10 | ||
11 | TestSuiteMakefileGuide | |
12 | ||
13 | Overview | |
14 | ======== | |
15 | ||
16 | This document is the reference manual for the LLVM testing | |
17 | infrastructure. It documents the structure of the LLVM testing | |
18 | infrastructure, the tools needed to use it, and how to add and run | |
19 | tests. | |
20 | ||
21 | Requirements | |
22 | ============ | |
23 | ||
1a4d82fc | 24 | In order to use the LLVM testing infrastructure, you will need all of the |
85aaf69f | 25 | software required to build LLVM, as well as `Python <http://python.org>`_ 2.7 or |
1a4d82fc | 26 | later. |
970d7e83 LB |
27 | |
28 | LLVM testing infrastructure organization | |
29 | ======================================== | |
30 | ||
31 | The LLVM testing infrastructure contains two major categories of tests: | |
32 | regression tests and whole programs. The regression tests are contained | |
33 | inside the LLVM repository itself under ``llvm/test`` and are expected | |
34 | to always pass -- they should be run before every commit. | |
35 | ||
36 | The whole programs tests are referred to as the "LLVM test suite" (or | |
37 | "test-suite") and are in the ``test-suite`` module in subversion. For | |
38 | historical reasons, these tests are also referred to as the "nightly | |
39 | tests" in places, which is less ambiguous than "test-suite" and remains | |
40 | in use although we run them much more often than nightly. | |
41 | ||
42 | Regression tests | |
43 | ---------------- | |
44 | ||
45 | The regression tests are small pieces of code that test a specific | |
46 | feature of LLVM or trigger a specific bug in LLVM. The language they are | |
47 | written in depends on the part of LLVM being tested. These tests are driven by | |
48 | the :doc:`Lit <CommandGuide/lit>` testing tool (which is part of LLVM), and | |
49 | are located in the ``llvm/test`` directory. | |
50 | ||
51 | Typically when a bug is found in LLVM, a regression test containing just | |
52 | enough code to reproduce the problem should be written and placed | |
53 | somewhere underneath this directory. For example, it can be a small | |
54 | piece of LLVM IR distilled from an actual application or benchmark. | |
55 | ||
56 | ``test-suite`` | |
57 | -------------- | |
58 | ||
59 | The test suite contains whole programs, which are pieces of code which | |
60 | can be compiled and linked into a stand-alone program that can be | |
61 | executed. These programs are generally written in high level languages | |
62 | such as C or C++. | |
63 | ||
64 | These programs are compiled using a user specified compiler and set of | |
65 | flags, and then executed to capture the program output and timing | |
66 | information. The output of these programs is compared to a reference | |
67 | output to ensure that the program is being compiled correctly. | |
68 | ||
69 | In addition to compiling and executing programs, whole program tests | |
70 | serve as a way of benchmarking LLVM performance, both in terms of the | |
71 | efficiency of the programs generated as well as the speed with which | |
72 | LLVM compiles, optimizes, and generates code. | |
73 | ||
74 | The test-suite is located in the ``test-suite`` Subversion module. | |
75 | ||
76 | Debugging Information tests | |
77 | --------------------------- | |
78 | ||
79 | The test suite contains tests to check quality of debugging information. | |
80 | The test are written in C based languages or in LLVM assembly language. | |
81 | ||
82 | These tests are compiled and run under a debugger. The debugger output | |
83 | is checked to validate of debugging information. See README.txt in the | |
84 | test suite for more information . This test suite is located in the | |
85 | ``debuginfo-tests`` Subversion module. | |
86 | ||
87 | Quick start | |
88 | =========== | |
89 | ||
90 | The tests are located in two separate Subversion modules. The | |
91 | regressions tests are in the main "llvm" module under the directory | |
92 | ``llvm/test`` (so you get these tests for free with the main LLVM tree). | |
93 | Use ``make check-all`` to run the regression tests after building LLVM. | |
94 | ||
95 | The more comprehensive test suite that includes whole programs in C and C++ | |
96 | is in the ``test-suite`` module. See :ref:`test-suite Quickstart | |
97 | <test-suite-quickstart>` for more information on running these tests. | |
98 | ||
99 | Regression tests | |
100 | ---------------- | |
101 | ||
102 | To run all of the LLVM regression tests, use the master Makefile in the | |
103 | ``llvm/test`` directory. LLVM Makefiles require GNU Make (read the :doc:`LLVM | |
104 | Makefile Guide <MakefileGuide>` for more details): | |
105 | ||
106 | .. code-block:: bash | |
107 | ||
108 | % make -C llvm/test | |
109 | ||
110 | or: | |
111 | ||
112 | .. code-block:: bash | |
113 | ||
114 | % make check | |
115 | ||
116 | If you have `Clang <http://clang.llvm.org/>`_ checked out and built, you | |
117 | can run the LLVM and Clang tests simultaneously using: | |
118 | ||
119 | .. code-block:: bash | |
120 | ||
121 | % make check-all | |
122 | ||
1a4d82fc JJ |
123 | To run the tests with Valgrind (Memcheck by default), use the ``LIT_ARGS`` make |
124 | variable to pass the required options to lit. For example, you can use: | |
970d7e83 LB |
125 | |
126 | .. code-block:: bash | |
127 | ||
1a4d82fc JJ |
128 | % make check LIT_ARGS="-v --vg --vg-leak" |
129 | ||
130 | to enable testing with valgrind and with leak checking enabled. | |
970d7e83 LB |
131 | |
132 | To run individual tests or subsets of tests, you can use the ``llvm-lit`` | |
133 | script which is built as part of LLVM. For example, to run the | |
134 | ``Integer/BitPacked.ll`` test by itself you can run: | |
135 | ||
136 | .. code-block:: bash | |
137 | ||
138 | % llvm-lit ~/llvm/test/Integer/BitPacked.ll | |
139 | ||
140 | or to run all of the ARM CodeGen tests: | |
141 | ||
142 | .. code-block:: bash | |
143 | ||
144 | % llvm-lit ~/llvm/test/CodeGen/ARM | |
145 | ||
146 | For more information on using the :program:`lit` tool, see ``llvm-lit --help`` | |
147 | or the :doc:`lit man page <CommandGuide/lit>`. | |
148 | ||
149 | Debugging Information tests | |
150 | --------------------------- | |
151 | ||
152 | To run debugging information tests simply checkout the tests inside | |
153 | clang/test directory. | |
154 | ||
155 | .. code-block:: bash | |
156 | ||
157 | % cd clang/test | |
158 | % svn co http://llvm.org/svn/llvm-project/debuginfo-tests/trunk debuginfo-tests | |
159 | ||
160 | These tests are already set up to run as part of clang regression tests. | |
161 | ||
162 | Regression test structure | |
163 | ========================= | |
164 | ||
165 | The LLVM regression tests are driven by :program:`lit` and are located in the | |
166 | ``llvm/test`` directory. | |
167 | ||
168 | This directory contains a large array of small tests that exercise | |
169 | various features of LLVM and to ensure that regressions do not occur. | |
170 | The directory is broken into several sub-directories, each focused on a | |
171 | particular area of LLVM. | |
172 | ||
173 | Writing new regression tests | |
174 | ---------------------------- | |
175 | ||
176 | The regression test structure is very simple, but does require some | |
177 | information to be set. This information is gathered via ``configure`` | |
178 | and is written to a file, ``test/lit.site.cfg`` in the build directory. | |
179 | The ``llvm/test`` Makefile does this work for you. | |
180 | ||
181 | In order for the regression tests to work, each directory of tests must | |
182 | have a ``lit.local.cfg`` file. :program:`lit` looks for this file to determine | |
183 | how to run the tests. This file is just Python code and thus is very | |
184 | flexible, but we've standardized it for the LLVM regression tests. If | |
185 | you're adding a directory of tests, just copy ``lit.local.cfg`` from | |
186 | another directory to get running. The standard ``lit.local.cfg`` simply | |
187 | specifies which files to look in for tests. Any directory that contains | |
188 | only directories does not need the ``lit.local.cfg`` file. Read the :doc:`Lit | |
189 | documentation <CommandGuide/lit>` for more information. | |
190 | ||
191 | Each test file must contain lines starting with "RUN:" that tell :program:`lit` | |
192 | how to run it. If there are no RUN lines, :program:`lit` will issue an error | |
193 | while running a test. | |
194 | ||
195 | RUN lines are specified in the comments of the test program using the | |
196 | keyword ``RUN`` followed by a colon, and lastly the command (pipeline) | |
197 | to execute. Together, these lines form the "script" that :program:`lit` | |
198 | executes to run the test case. The syntax of the RUN lines is similar to a | |
199 | shell's syntax for pipelines including I/O redirection and variable | |
200 | substitution. However, even though these lines may *look* like a shell | |
201 | script, they are not. RUN lines are interpreted by :program:`lit`. | |
202 | Consequently, the syntax differs from shell in a few ways. You can specify | |
203 | as many RUN lines as needed. | |
204 | ||
205 | :program:`lit` performs substitution on each RUN line to replace LLVM tool names | |
206 | with the full paths to the executable built for each tool (in | |
207 | ``$(LLVM_OBJ_ROOT)/$(BuildMode)/bin)``. This ensures that :program:`lit` does | |
208 | not invoke any stray LLVM tools in the user's path during testing. | |
209 | ||
210 | Each RUN line is executed on its own, distinct from other lines unless | |
211 | its last character is ``\``. This continuation character causes the RUN | |
212 | line to be concatenated with the next one. In this way you can build up | |
213 | long pipelines of commands without making huge line lengths. The lines | |
214 | ending in ``\`` are concatenated until a RUN line that doesn't end in | |
215 | ``\`` is found. This concatenated set of RUN lines then constitutes one | |
216 | execution. :program:`lit` will substitute variables and arrange for the pipeline | |
217 | to be executed. If any process in the pipeline fails, the entire line (and | |
218 | test case) fails too. | |
219 | ||
220 | Below is an example of legal RUN lines in a ``.ll`` file: | |
221 | ||
222 | .. code-block:: llvm | |
223 | ||
224 | ; RUN: llvm-as < %s | llvm-dis > %t1 | |
225 | ; RUN: llvm-dis < %s.bc-13 > %t2 | |
226 | ; RUN: diff %t1 %t2 | |
227 | ||
228 | As with a Unix shell, the RUN lines permit pipelines and I/O | |
1a4d82fc | 229 | redirection to be used. |
970d7e83 LB |
230 | |
231 | There are some quoting rules that you must pay attention to when writing | |
232 | your RUN lines. In general nothing needs to be quoted. :program:`lit` won't | |
233 | strip off any quote characters so they will get passed to the invoked program. | |
234 | To avoid this use curly braces to tell :program:`lit` that it should treat | |
235 | everything enclosed as one value. | |
236 | ||
237 | In general, you should strive to keep your RUN lines as simple as possible, | |
238 | using them only to run tools that generate textual output you can then examine. | |
1a4d82fc | 239 | The recommended way to examine output to figure out if the test passes is using |
970d7e83 LB |
240 | the :doc:`FileCheck tool <CommandGuide/FileCheck>`. *[The usage of grep in RUN |
241 | lines is deprecated - please do not send or commit patches that use it.]* | |
242 | ||
85aaf69f SL |
243 | Extra files |
244 | ----------- | |
245 | ||
246 | If your test requires extra files besides the file containing the ``RUN:`` | |
247 | lines, the idiomatic place to put them is in a subdirectory ``Inputs``. | |
248 | You can then refer to the extra files as ``%S/Inputs/foo.bar``. | |
249 | ||
250 | For example, consider ``test/Linker/ident.ll``. The directory structure is | |
251 | as follows:: | |
252 | ||
253 | test/ | |
254 | Linker/ | |
255 | ident.ll | |
256 | Inputs/ | |
257 | ident.a.ll | |
258 | ident.b.ll | |
259 | ||
260 | For convenience, these are the contents: | |
261 | ||
262 | .. code-block:: llvm | |
263 | ||
264 | ;;;;; ident.ll: | |
265 | ||
266 | ; RUN: llvm-link %S/Inputs/ident.a.ll %S/Inputs/ident.b.ll -S | FileCheck %s | |
267 | ||
268 | ; Verify that multiple input llvm.ident metadata are linked together. | |
269 | ||
270 | ; CHECK-DAG: !llvm.ident = !{!0, !1, !2} | |
271 | ; CHECK-DAG: "Compiler V1" | |
272 | ; CHECK-DAG: "Compiler V2" | |
273 | ; CHECK-DAG: "Compiler V3" | |
274 | ||
275 | ;;;;; Inputs/ident.a.ll: | |
276 | ||
277 | !llvm.ident = !{!0, !1} | |
278 | !0 = metadata !{metadata !"Compiler V1"} | |
279 | !1 = metadata !{metadata !"Compiler V2"} | |
280 | ||
281 | ;;;;; Inputs/ident.b.ll: | |
282 | ||
283 | !llvm.ident = !{!0} | |
284 | !0 = metadata !{metadata !"Compiler V3"} | |
285 | ||
286 | For symmetry reasons, ``ident.ll`` is just a dummy file that doesn't | |
287 | actually participate in the test besides holding the ``RUN:`` lines. | |
288 | ||
289 | .. note:: | |
290 | ||
291 | Some existing tests use ``RUN: true`` in extra files instead of just | |
292 | putting the extra files in an ``Inputs/`` directory. This pattern is | |
293 | deprecated. | |
294 | ||
970d7e83 LB |
295 | Fragile tests |
296 | ------------- | |
297 | ||
298 | It is easy to write a fragile test that would fail spuriously if the tool being | |
299 | tested outputs a full path to the input file. For example, :program:`opt` by | |
300 | default outputs a ``ModuleID``: | |
301 | ||
302 | .. code-block:: console | |
303 | ||
304 | $ cat example.ll | |
305 | define i32 @main() nounwind { | |
306 | ret i32 0 | |
307 | } | |
308 | ||
309 | $ opt -S /path/to/example.ll | |
310 | ; ModuleID = '/path/to/example.ll' | |
311 | ||
312 | define i32 @main() nounwind { | |
313 | ret i32 0 | |
314 | } | |
315 | ||
316 | ``ModuleID`` can unexpetedly match against ``CHECK`` lines. For example: | |
317 | ||
318 | .. code-block:: llvm | |
319 | ||
320 | ; RUN: opt -S %s | FileCheck | |
321 | ||
322 | define i32 @main() nounwind { | |
323 | ; CHECK-NOT: load | |
324 | ret i32 0 | |
325 | } | |
326 | ||
327 | This test will fail if placed into a ``download`` directory. | |
328 | ||
329 | To make your tests robust, always use ``opt ... < %s`` in the RUN line. | |
330 | :program:`opt` does not output a ``ModuleID`` when input comes from stdin. | |
331 | ||
1a4d82fc JJ |
332 | Platform-Specific Tests |
333 | ----------------------- | |
970d7e83 | 334 | |
1a4d82fc JJ |
335 | Whenever adding tests that require the knowledge of a specific platform, |
336 | either related to code generated, specific output or back-end features, | |
337 | you must make sure to isolate the features, so that buildbots that | |
338 | run on different architectures (and don't even compile all back-ends), | |
339 | don't fail. | |
970d7e83 | 340 | |
1a4d82fc JJ |
341 | The first problem is to check for target-specific output, for example sizes |
342 | of structures, paths and architecture names, for example: | |
970d7e83 | 343 | |
1a4d82fc JJ |
344 | * Tests containing Windows paths will fail on Linux and vice-versa. |
345 | * Tests that check for ``x86_64`` somewhere in the text will fail anywhere else. | |
346 | * Tests where the debug information calculates the size of types and structures. | |
970d7e83 | 347 | |
1a4d82fc JJ |
348 | Also, if the test rely on any behaviour that is coded in any back-end, it must |
349 | go in its own directory. So, for instance, code generator tests for ARM go | |
350 | into ``test/CodeGen/ARM`` and so on. Those directories contain a special | |
351 | ``lit`` configuration file that ensure all tests in that directory will | |
352 | only run if a specific back-end is compiled and available. | |
353 | ||
354 | For instance, on ``test/CodeGen/ARM``, the ``lit.local.cfg`` is: | |
355 | ||
356 | .. code-block:: python | |
357 | ||
358 | config.suffixes = ['.ll', '.c', '.cpp', '.test'] | |
359 | if not 'ARM' in config.root.targets: | |
360 | config.unsupported = True | |
361 | ||
362 | Other platform-specific tests are those that depend on a specific feature | |
363 | of a specific sub-architecture, for example only to Intel chips that support ``AVX2``. | |
364 | ||
365 | For instance, ``test/CodeGen/X86/psubus.ll`` tests three sub-architecture | |
366 | variants: | |
367 | ||
368 | .. code-block:: llvm | |
369 | ||
370 | ; RUN: llc -mcpu=core2 < %s | FileCheck %s -check-prefix=SSE2 | |
371 | ; RUN: llc -mcpu=corei7-avx < %s | FileCheck %s -check-prefix=AVX1 | |
372 | ; RUN: llc -mcpu=core-avx2 < %s | FileCheck %s -check-prefix=AVX2 | |
373 | ||
374 | And the checks are different: | |
375 | ||
376 | .. code-block:: llvm | |
377 | ||
378 | ; SSE2: @test1 | |
379 | ; SSE2: psubusw LCPI0_0(%rip), %xmm0 | |
380 | ; AVX1: @test1 | |
381 | ; AVX1: vpsubusw LCPI0_0(%rip), %xmm0, %xmm0 | |
382 | ; AVX2: @test1 | |
383 | ; AVX2: vpsubusw LCPI0_0(%rip), %xmm0, %xmm0 | |
384 | ||
385 | So, if you're testing for a behaviour that you know is platform-specific or | |
386 | depends on special features of sub-architectures, you must add the specific | |
387 | triple, test with the specific FileCheck and put it into the specific | |
388 | directory that will filter out all other architectures. | |
389 | ||
390 | ||
391 | Substitutions | |
392 | ------------- | |
970d7e83 | 393 | |
1a4d82fc JJ |
394 | Besides replacing LLVM tool names the following substitutions are performed in |
395 | RUN lines: | |
970d7e83 | 396 | |
1a4d82fc JJ |
397 | ``%%`` |
398 | Replaced by a single ``%``. This allows escaping other substitutions. | |
970d7e83 | 399 | |
1a4d82fc JJ |
400 | ``%s`` |
401 | File path to the test case's source. This is suitable for passing on the | |
402 | command line as the input to an LLVM tool. | |
970d7e83 | 403 | |
1a4d82fc | 404 | Example: ``/home/user/llvm/test/MC/ELF/foo_test.s`` |
970d7e83 | 405 | |
1a4d82fc JJ |
406 | ``%S`` |
407 | Directory path to the test case's source. | |
970d7e83 | 408 | |
1a4d82fc | 409 | Example: ``/home/user/llvm/test/MC/ELF`` |
970d7e83 | 410 | |
1a4d82fc JJ |
411 | ``%t`` |
412 | File path to a temporary file name that could be used for this test case. | |
970d7e83 LB |
413 | The file name won't conflict with other test cases. You can append to it |
414 | if you need multiple temporaries. This is useful as the destination of | |
415 | some redirected output. | |
416 | ||
1a4d82fc | 417 | Example: ``/home/user/llvm.build/test/MC/ELF/Output/foo_test.s.tmp`` |
970d7e83 | 418 | |
1a4d82fc JJ |
419 | ``%T`` |
420 | Directory of ``%t``. | |
970d7e83 | 421 | |
1a4d82fc JJ |
422 | Example: ``/home/user/llvm.build/test/MC/ELF/Output`` |
423 | ||
424 | ``%{pathsep}`` | |
425 | ||
426 | Expands to the path separator, i.e. ``:`` (or ``;`` on Windows). | |
427 | ||
428 | ||
429 | **LLVM-specific substitutions:** | |
430 | ||
431 | ``%shlibext`` | |
432 | The suffix for the host platforms shared library files. This includes the | |
433 | period as the first character. | |
434 | ||
435 | Example: ``.so`` (Linux), ``.dylib`` (OS X), ``.dll`` (Windows) | |
436 | ||
437 | ``%exeext`` | |
438 | The suffix for the host platforms executable files. This includes the | |
439 | period as the first character. | |
440 | ||
441 | Example: ``.exe`` (Windows), empty on Linux. | |
442 | ||
443 | ``%(line)``, ``%(line+<number>)``, ``%(line-<number>)`` | |
444 | The number of the line where this substitution is used, with an optional | |
445 | integer offset. This can be used in tests with multiple RUN lines, which | |
446 | reference test file's line numbers. | |
447 | ||
448 | ||
449 | **Clang-specific substitutions:** | |
450 | ||
451 | ``%clang`` | |
452 | Invokes the Clang driver. | |
453 | ||
454 | ``%clang_cpp`` | |
455 | Invokes the Clang driver for C++. | |
456 | ||
457 | ``%clang_cl`` | |
458 | Invokes the CL-compatible Clang driver. | |
459 | ||
460 | ``%clangxx`` | |
461 | Invokes the G++-compatible Clang driver. | |
462 | ||
463 | ``%clang_cc1`` | |
464 | Invokes the Clang frontend. | |
465 | ||
466 | ``%itanium_abi_triple``, ``%ms_abi_triple`` | |
467 | These substitutions can be used to get the current target triple adjusted to | |
468 | the desired ABI. For example, if the test suite is running with the | |
469 | ``i686-pc-win32`` target, ``%itanium_abi_triple`` will expand to | |
470 | ``i686-pc-mingw32``. This allows a test to run with a specific ABI without | |
471 | constraining it to a specific triple. | |
472 | ||
473 | To add more substituations, look at ``test/lit.cfg`` or ``lit.local.cfg``. | |
970d7e83 | 474 | |
970d7e83 LB |
475 | |
476 | Other Features | |
477 | -------------- | |
478 | ||
1a4d82fc JJ |
479 | To make RUN line writing easier, there are several helper programs. These |
480 | helpers are in the PATH when running tests, so you can just call them using | |
481 | their name. For example: | |
482 | ||
970d7e83 | 483 | ``not`` |
1a4d82fc | 484 | This program runs its arguments and then inverts the result code from it. |
970d7e83 LB |
485 | Zero result codes become 1. Non-zero result codes become 0. |
486 | ||
487 | Sometimes it is necessary to mark a test case as "expected fail" or | |
488 | XFAIL. You can easily mark a test as XFAIL just by including ``XFAIL:`` | |
489 | on a line near the top of the file. This signals that the test case | |
490 | should succeed if the test fails. Such test cases are counted separately | |
491 | by the testing tool. To specify an expected fail, use the XFAIL keyword | |
492 | in the comments of the test program followed by a colon and one or more | |
493 | failure patterns. Each failure pattern can be either ``*`` (to specify | |
494 | fail everywhere), or a part of a target triple (indicating the test | |
495 | should fail on that platform), or the name of a configurable feature | |
496 | (for example, ``loadable_module``). If there is a match, the test is | |
497 | expected to fail. If not, the test is expected to succeed. To XFAIL | |
498 | everywhere just specify ``XFAIL: *``. Here is an example of an ``XFAIL`` | |
499 | line: | |
500 | ||
501 | .. code-block:: llvm | |
502 | ||
503 | ; XFAIL: darwin,sun | |
504 | ||
505 | To make the output more useful, :program:`lit` will scan | |
506 | the lines of the test case for ones that contain a pattern that matches | |
507 | ``PR[0-9]+``. This is the syntax for specifying a PR (Problem Report) number | |
508 | that is related to the test case. The number after "PR" specifies the | |
509 | LLVM bugzilla number. When a PR number is specified, it will be used in | |
510 | the pass/fail reporting. This is useful to quickly get some context when | |
511 | a test fails. | |
512 | ||
513 | Finally, any line that contains "END." will cause the special | |
514 | interpretation of lines to terminate. This is generally done right after | |
515 | the last RUN: line. This has two side effects: | |
516 | ||
517 | (a) it prevents special interpretation of lines that are part of the test | |
518 | program, not the instructions to the test case, and | |
519 | ||
520 | (b) it speeds things up for really big test cases by avoiding | |
521 | interpretation of the remainder of the file. | |
522 | ||
523 | ``test-suite`` Overview | |
524 | ======================= | |
525 | ||
526 | The ``test-suite`` module contains a number of programs that can be | |
527 | compiled and executed. The ``test-suite`` includes reference outputs for | |
528 | all of the programs, so that the output of the executed program can be | |
529 | checked for correctness. | |
530 | ||
531 | ``test-suite`` tests are divided into three types of tests: MultiSource, | |
532 | SingleSource, and External. | |
533 | ||
534 | - ``test-suite/SingleSource`` | |
535 | ||
536 | The SingleSource directory contains test programs that are only a | |
537 | single source file in size. These are usually small benchmark | |
538 | programs or small programs that calculate a particular value. Several | |
539 | such programs are grouped together in each directory. | |
540 | ||
541 | - ``test-suite/MultiSource`` | |
542 | ||
543 | The MultiSource directory contains subdirectories which contain | |
544 | entire programs with multiple source files. Large benchmarks and | |
545 | whole applications go here. | |
546 | ||
547 | - ``test-suite/External`` | |
548 | ||
549 | The External directory contains Makefiles for building code that is | |
550 | external to (i.e., not distributed with) LLVM. The most prominent | |
551 | members of this directory are the SPEC 95 and SPEC 2000 benchmark | |
552 | suites. The ``External`` directory does not contain these actual | |
553 | tests, but only the Makefiles that know how to properly compile these | |
554 | programs from somewhere else. When using ``LNT``, use the | |
555 | ``--test-externals`` option to include these tests in the results. | |
556 | ||
557 | .. _test-suite-quickstart: | |
558 | ||
559 | ``test-suite`` Quickstart | |
560 | ------------------------- | |
561 | ||
562 | The modern way of running the ``test-suite`` is focused on testing and | |
563 | benchmarking complete compilers using the | |
564 | `LNT <http://llvm.org/docs/lnt>`_ testing infrastructure. | |
565 | ||
566 | For more information on using LNT to execute the ``test-suite``, please | |
567 | see the `LNT Quickstart <http://llvm.org/docs/lnt/quickstart.html>`_ | |
568 | documentation. | |
569 | ||
570 | ``test-suite`` Makefiles | |
571 | ------------------------ | |
572 | ||
573 | Historically, the ``test-suite`` was executed using a complicated setup | |
574 | of Makefiles. The LNT based approach above is recommended for most | |
575 | users, but there are some testing scenarios which are not supported by | |
576 | the LNT approach. In addition, LNT currently uses the Makefile setup | |
577 | under the covers and so developers who are interested in how LNT works | |
578 | under the hood may want to understand the Makefile based setup. | |
579 | ||
580 | For more information on the ``test-suite`` Makefile setup, please see | |
581 | the :doc:`Test Suite Makefile Guide <TestSuiteMakefileGuide>`. |