]> git.proxmox.com Git - ceph.git/blob - ceph/src/boost/tools/build/test/test_system.html
add subtree-ish sources for 12.0.3
[ceph.git] / ceph / src / boost / tools / build / test / test_system.html
1 <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN"
2 "http://www.w3.org/TR/html4/strict.dtd">
3
4 <html>
5 <head>
6 <meta name="generator" content=
7 "HTML Tidy for Linux/x86 (vers 1st March 2002), see www.w3.org">
8 <!--tidy options: -i -wrap 78 -->
9 <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
10
11 <title>A testing system for Boost.Build</title>
12 <style type="text/css">
13 hr { color: black }
14 p.revision { text-align: right; font-style: italic }
15 pre.code { margin-left: 2em }
16 pre.example { margin-left: 2em; border: solid black thin }
17 pre.output { margin-left: 2em }
18 img.banner { border: 0; float: left }
19 h1 { text-align: right }
20 br.clear { clear: left }
21 div.attention { color: red }
22
23 </style>
24 </head>
25
26 <body>
27 <p><a href="../../../../index.htm"><img class="banner" height="86" width=
28 "277" alt="C++ Boost" src="../../../../boost.png"></a></p>
29
30 <h1>A testing system for Boost.Build<br class="clear">
31 </h1>
32 <hr>
33
34 <dl class="page-index">
35 <dt><a href="#sec-intro">Introduction for users</a></dt>
36
37 <dd>
38 <dl class="page-index">
39 <dt><a href="#sec-command-line-options">Command line options</a></dt>
40 </dl>
41 </dd>
42
43 <dt><a href="#sec-developers">Introduction for developers</a></dt>
44
45 <dd>
46 <dl class="page-index">
47 <dt><a href="#sec-intro-changing">Changing the working
48 directory</a></dt>
49
50 <dt><a href="#sec-intro-examining">Examining the working directory and
51 changing it</a></dt>
52
53 <dt><a href="#sec-intro-results">Test result</a></dt>
54 </dl>
55 </dd>
56
57 <dt><a href="#sec-reference">Reference documentation</a></dt>
58
59 <dd>
60 <dl class="page-index">
61 <dt><a href="#method-__init__">Method __init__</a></dt>
62
63 <dt><a href="#method-set_tree">Method <tt>set_tree</tt></a></dt>
64
65 <dt><a href="#method-write">Method <tt>write</tt></a></dt>
66
67 <dt><a href="#method-copy">Method <tt>copy</tt></a></dt>
68
69 <dt><a href="#method-touch">Method <tt>touch</tt></a></dt>
70
71 <dt><a href="#method-run_build_system">Method
72 <tt>run_build_system</tt></a></dt>
73
74 <dt><a href="#method-read">Method <tt>read</tt></a></dt>
75
76 <dt><a href="#method-read_and_strip">Method
77 <tt>read_and_strip</tt></a></dt>
78
79 <dt><a href="#methods-expectations">Methods for declaring
80 expectations</a></dt>
81
82 <dt><a href="#methods-ignoring">Methods for ignoring
83 changes</a></dt>
84
85 <dt><a href="#methods-result">Methods for explicitly specifying
86 results</a></dt>
87
88 <dt><a href="#class-list">Helper class <tt>List</tt></a></dt>
89 </dl>
90 </dd>
91 </dl>
92 <hr>
93
94 <h2><a name="sec-intro">Introduction for users</a></h2>
95
96 <p>The testing system for Boost.Build is a small set of Python modules and
97 scripts for automatically testing user-obversable behaviour. It uses
98 components from testing systems of <a href="http://www.scons.org">Scons</a>
99 and <a href="http://subversion.tigris.org">Subversion</a>, together with
100 some additional functionality.</p>
101
102 <p>To run the tests you need to:</p>
103
104 <ol>
105 <li>Get the source tree of Boost.Build (located at <tt>tools/build</tt>
106 in Boost)</li>
107
108 <li>Have <a href="http://www.python.org">Python</a> installed. Version
109 2.1 is known to work.</li>
110
111 <li>Build Boost.Jam. See <a href=
112 "../engine/index.html">$boost_build_root/engine/index.html</a> for
113 instructions.</li>
114
115 <li>Configure at least one toolset. You can edit <tt>site-config.jam</tt>
116 or <tt>user-config.jam</tt> to add new toolsets. Or you can create file
117 <tt>test-config.jam</tt> in <tt>$boost_build_root/test</tt> directory. In
118 this case, <tt>site-config.jam</tt> and <tt>user-config.jam</tt> will be
119 ignored for testing.</li>
120 </ol>
121
122 <p>When all is set, you can run all the tests using the <tt>test_all.py</tt>
123 script or you can run a specific test by starting its Python script
124 directly.</p>
125
126 <p>Examples:</p>
127
128 <pre class="code">
129 python test_all.py
130 python generators_test.py
131 </pre>
132
133 <p>If everything is OK, you will see a list of passed tests. Otherwise, a
134 failure will be reported.</p>
135
136 <h3><a name="sec-command-line-options">Command line options</a></h3>
137
138 <p>Test scripts will use the toolset you configured to be the default or
139 you can specify a specific one on the command line:</p>
140
141 <pre class="code">
142 python test_all.py borland
143 python generators_test.py msvc-7.1
144 </pre>
145
146 <p>Other test script flags you can specify on the command line are:</p>
147
148 <ul>
149 <li><tt>--default-bjam</tt> -- By default the test system will use the
150 Boost Jam executable found built in its default development build
151 location. This option makes it use the default one available on your
152 system, i.e. the one found in the system path.</li>
153
154 <li><tt>--preserve</tt> -- In case of a failed test its working
155 directory will be copied to the "failed_test" directory under the
156 current directory.</li>
157
158 <li><tt>--verbose</tt> -- Makes the test system and the run build system
159 display additional output. Note though that this may cause tests that
160 check the build system output to fail.</li>
161 </ul>
162
163 <h2><a name="sec-developers">Introduction for developers</a></h2>
164
165 <p>It is suggested that every new functionality come together with tests,
166 and that bugfixes are accompanied by tests. There's no need to say that
167 tests are good, but two points are extremely important:</p>
168
169 <ul>
170 <li>For an interpreted language like Jam, without any static checks,
171 testing is simply the only sefeguard we can have.</li>
172
173 <li>Good tests allow us to change internal design much more safely, and we
174 have not gotten everything nailed down yet.</li>
175 </ul>
176
177 <p>Adding a new test is simple:</p>
178
179 <ol>
180 <li>Go to <tt>$boost_build_root/test/test_all.py</tt> and add new test
181 name to the list at the end of the file. Suppose the test name is "hello".
182 </li>
183
184 <li>Add a new python module, in this example "hello.py", to do the actual
185 testing.</li>
186 </ol>
187
188 <p>The module, in general will perform these basic actions:</p>
189
190 <ol>
191 <li>Set up the initial working directory state</li>
192
193 <li>
194 Run the build system and check the results:
195
196 <ol>
197 <li>generated output,</li>
198
199 <li>changes made to the working directory,</li>
200
201 <li>new content of the working directory.</li>
202 </ol>
203 </li>
204
205 <li>Add, remove or touch files or change their content and then repeat
206 the previous step until satisfied.</li>
207
208 <li>Clean up</li>
209 </ol>
210
211 <p>The "hello.py" module might contain:</p>
212 <pre class="example">
213 from BoostBuild import List
214
215 # Create a temporary working directory
216 t = BoostBuild.Tester()
217
218 # Create the needed files
219 t.write("jamroot.jam", "")
220 t.write("jamfile.jam", """
221 exe hello : hello.cpp ;
222 """)
223 t.write("hello.cpp", """
224 int main()
225 {
226 return 0;
227 }
228
229 """)
230
231 t.run_build_system()
232
233 # First, create a list of three pathnames.
234 file_list = List("bin/$toolset/debug/") * List("hello.exe hello.obj")
235 # Second, assert that those files were added as result of the last build system invocation.
236 t.expect_addition(file_list)
237
238 # Invoke the build system once again.
239 t.run_build_system("clean")
240 # Check if the files added previously were removed.
241 t.expect_removal(file_list)
242
243 # Remove temporary directories
244 t.cleanup()
245 </pre>
246
247 <p>The <tt>test</tt> directory contains a file "template.py" which can be
248 used as a start for your own tests.</p>
249
250 <p>Overview of the most important methods of class <tt>Tester</tt> follows.
251 </p>
252
253 <h3><a name="sec-intro-changing">Changing the working directory</a></h3>
254
255 <p>The class <tt>Tester</tt> creates a temporary directory in its
256 constructor and changes to that directory. It can be modified by calling
257 these methods:</p>
258
259 <ul>
260 <li><tt>set_tree</tt> -- sets the content of the working directory to be
261 equal to the content of the specified directory. This method is
262 preferrable when directory tree for testing is large.</li>
263
264 <li><tt>write</tt> -- sets the content of file in a working directory.
265 This is optimal if you want to create a directory tree with 3-4 small
266 files.</li>
267
268 <li><tt>touch</tt> -- changes the modification times of a file</li>
269 </ul>
270
271 <h3><a name="sec-intro-examining">Examining the working directory and
272 changing it</a></h3>
273
274 <p>The method <tt>read</tt>, inherited from the <tt>TestCmd</tt> class, can
275 be used to read any file in the working directory and check its content.
276 <tt>Tester</tt> adds another method for tracking changes. Whenever the build
277 system is run (using <a href="#method-run_build_system"><tt>run_build_system
278 </tt></a>), the working dir state before and after running is recorded. In
279 addition, difference between the two states -- i.e. lists of files that were
280 added, removed, modified or touched -- are stored in two member variables -
281 <tt>tree_difference</tt> and <tt>unexpected_difference</tt>.</p>
282
283 <p>After than, the test author may specify that some change is expected, for
284 example, by calling <tt>expect_addition("foo")</tt>. This call will check if
285 the file was indeed added, and if so, will remove its name from the list of
286 added files in <tt>unexpected_difference</tt>. Likewise, it is possible to
287 specify that some changes are not interesting, for example a call to
288 <tt>ignore("*.obj")</tt> will just remove every file with the ".obj"
289 extension from <tt>unexpected_difference</tt>.</p>
290
291 <p>When test has finished with expectations and ignoring, the member
292 <tt>unexpected_difference</tt> will contain the list of all changes not yet
293 accounted for. It is possible to assure that this list is empty by calling
294 the <tt>expect_nothing_more</tt> member function.</p>
295
296 <h3><a name="sec-intro-results">Test result</a></h3>
297
298 <p>Any of the <tt>expect*</tt> methods below will fail the test if the
299 expectation is not met. It is also possible to perform manually arbitrary
300 test and explicitly cause the test to either pass or fail. Ordinary
301 filesystem functions can be used to work with the directory tree. Methods
302 <tt>pass_test</tt> and <tt>fail_test</tt> are used to explicitly give the
303 test outcome.</p>
304
305 <p>Typically, after test termination, the working directory is erased. See
306 the <a href="#sec-command-line-options">"--preserve" command line option</a>
307 for information on how to preserve the working directory content for failed
308 tests for debugging purposes.</p>
309
310 <h2 id="sec-reference">Reference documentation</h2>
311
312 <p>The test system is composed of class <tt>Tester</tt>, derived form
313 <tt>TestCmd.TestCmd</tt>, and helper class <tt>List</tt>. <tt>Tester</tt>
314 and <tt>List</tt> methods are described below.</p>
315
316 <p>The documentation frequently refers to <tt>filename</tt>. In all cases,
317 files are specified in unix style: a sequence of components, separated by
318 "/". This is true on all platforms. In some contexts a list of files is
319 allowed. In those cases any object with a sequence interface is allowed.</p>
320
321 <h3><a name="method-__init__">Method <tt>__init__(self, arguments="",
322 executable="bjam", match=TestCmd.match_exact, boost_build_path=None,
323 translate_suffixes=True, pass_toolset=True, use_test_config=True,
324 ignore_toolset_requirements=True, workdir="", **keywords)</tt></a></h3>
325
326 <p><b>Optional arguments:</b></p>
327
328 <ul>
329 <li><tt>arguments</tt>
330 -- Arguments passed to the run executable.</li>
331 <li><tt>executable</tt>
332 -- Name of the executable to invoke.</li>
333 <li><tt>match</tt>
334 -- Function to use for compating actual and expected file contents.
335 </li>
336 <li><tt>boost_build_path</tt>
337 -- Boost build path to be passed to the run executable.</li>
338 <li><tt>translate_suffixes</tt>
339 -- Whether to update suffixes on the the file names passed from the
340 test script so they match those actually created by the current
341 toolset. For example, static library files are specified by using
342 the .lib suffix but when the 'gcc' toolset is used it actually
343 creates them using the .a suffix.</li>
344 <li><tt>pass_toolset</tt>
345 -- Whether the test system should pass the specified toolset to the
346 run executable.</li>
347 <li><tt>use_test_config</tt>
348 -- Whether the test system should tell the run executable to read in
349 the test_config.jam configuration file.</li>
350 <li><tt>ignore_toolset_requirements</tt>
351 -- Whether the test system should tell the run executable to ignore
352 toolset requirements.</li>
353 <li><tt>workdir</tt>
354 -- Indicates an absolute directory where the test will be run from.
355 </li>
356 </ul>
357
358 <p><b>Optional arguments inherited from the base class:</b></p>
359
360 <ul>
361 <li><tt>description</tt>
362 -- Test description string displayed in case of a failed test.</li>
363 <li><tt>subdir</tt>
364 -- List of subdirectories to automatically create under the working
365 directory. Each subdirectory needs to be specified separately
366 parent coming before its child.</li>
367 <li><tt>verbose</tt>
368 -- Flag that may be used to enable more verbose test system output.
369 Note that it does not also enable more verbose build system output
370 like the <a href="#sec-command-line-options">"--verbose" command
371 line option</a> does.</li>
372 </ul>
373
374 <p><b>Effects:</b></p>
375
376 <ol>
377 <li>Remembers the current working directory in member
378 <tt>original_workdir</tt>.</li>
379
380 <li>Determines the location of the executable (<code>bjam</code> by
381 default) and build system files, assuming that the current directory is
382 <tt>tools/build/test</tt>. Formulates jam invocation command, which
383 will include explicit setting for the <tt>BOOST_BUILD_PATH</tt> variable
384 and arguments passed to this methods, if any. This command will be used
385 by subsequent invocation of <a href="#method-run_build_system"><tt>
386 run_build_system</tt></a>. Finally, initializes the base class.</li>
387
388 <li>Changes the current working directory to the temporary working
389 directory created by the base constructor.</li>
390
391 <li>If you want to run a test in an existing directory, pass it as
392 <tt>workdir</tt>.</li>
393
394 <li> Most parameters passed to this constructor function may be overruled
395 for each specific test system run using <a href=
396 "#method-run_build_system"><tt>run_build_system</tt></a> parameters.
397 </ol>
398
399 <h3><a name="method-set_tree">Method <tt>set_tree(self,
400 tree_location)</tt></a></h3>
401
402 <p><b>Effects:</b></p>
403
404 <p>Replaces the content of the current working directory with the content
405 of directory at <tt>tree_location</tt>. If <tt>tree_location</tt> is not
406 absolute pathname, it will be treated as relative to
407 <tt>self.original_workdir</tt>. This methods also explicitly makes the
408 copied files writeable.</p>
409
410 <h3><a name="method-write">Method <tt>write(self, name,
411 content)</tt></a></h3>
412
413 <p><b>Effects:</b></p>
414
415 <p>Writes the specified content to the file given by <tt>name</tt> under
416 the temporary working directory. If the file already exists, it is
417 overwritten. Any required directories are automatically created.</p>
418
419 <h3><a name="method-copy">Method <tt>copy(self, src, dst)</tt></a></h3>
420
421 <p><b>Effects:</b></p>
422
423 <p>Equvivalent to <tt>self.write(self.read(src), dst)</tt>.</p>
424
425 <h3><a name="method-touch">Method <tt>touch(self, names)</tt></a></h3>
426
427 <p><b>Effects:</b></p>
428
429 <p>Sets the access and modification times for all files in <tt>names</tt> to
430 the current time. All the elements in <tt>names</tt> should be relative
431 paths.</p>
432
433 <h3><a name="method-run_build_system">Method <tt>run_build_system(self,
434 extra_args="", subdir="", stdout=None, stderr="", status=0, match=None,
435 pass_toolset=None, use_test_config=None, ignore_toolset_requirements=None,
436 expected_duration=None, **kw)</tt></a></h3>
437
438 <p><b>Effects:</b></p>
439
440 <ol>
441 <li>Stores the state of the working directory in
442 <tt>self.previous_tree</tt>.</li>
443
444 <li>Changes to <tt>subdir</tt>, if it is specified. It is relative to
445 the <tt>original_workdir</tt> or the workdir specified in
446 <tt>__init</tt>.</li>
447
448 <li>Invokes the <tt>bjam</tt> executable, passing <tt>extra_args</tt>
449 to it. The binary should be located under
450 <tt>&lt;test_invocation_dir&gt;/../jam/src/bin.&lt;platform&gt;</tt>.
451 This is to make sure tests use the version of jam build from CVS.</li>
452
453 <li>Compares the stdout, stderr and exit status of build system
454 invocation with values to appropriate parameters, if they are not
455 <tt>None</tt>. If any difference is found, the test fails.</li>
456
457 <li>If the <tt>expected_duration</tt> parameter is specified then it
458 represents the maximal allowed time in seconds for the test to run. The
459 test will be marked as failed if its duration is greater than the given
460 <tt>expected_duration</tt> parameter value.</li>
461
462 <li>Stores the new state of the working directory in <tt>self.tree</tt>.
463 Computes the difference between previous and current trees and stores them
464 in variables <tt>self.tree_difference</tt> and
465 <tt>self.unexpected_difference</tt>. Both variables are instances of class
466 <tt>tree.Trees_different</tt>, which have four attributes:
467 <tt>added_files</tt>, <tt>removed_files</tt>, <tt>modified_files</tt> and
468 <tt>touched_files</tt>. Each is a list of strings.</p></li>
469 </ol>
470
471 <h3><a name="method-read">Method <tt>read(self, name)</tt></a></h3>
472
473 <p><b>Effects:</b></p>
474
475 <p>Read the specified file and returns it content. Raises an exception is
476 the file is absent.</p>
477
478 <h3><a name="method-read_and_strip">Method <tt>read_and_strip(self, name)
479 </tt></a></h3>
480
481 <p><b>Effects:</b></p>
482
483 <p>Read the specified file and returns it content, after removing trailing
484 whitespace from every line. Raises an exception is the file is absent.</p>
485
486 <p><b>Rationale:</b></p>
487
488 <p>Althought this method is questionable, there are a lot of cases when jam
489 or shells it uses insert spaces. It seems that introducing this method is
490 much simpler than dealing with all those cases.</p>
491
492 <h3><a name="methods-expectations">Methods for declaring expectations</a>
493 </h3>
494
495 <p>Accordingly to the number of changes kinds that are detected, there are
496 four methods that specify that test author expects a specific change to
497 occur. They check <tt>self.unexpected_difference</tt>, and if the change is
498 present there, it is removed. Otherwise, test fails.</p>
499
500 <p>Each method accepts a list of names. Those names use <tt>/</tt> path
501 separator on all systems. Additionaly, the test system translates suffixes
502 appropriately. For the test to be portable, suffixes should use Windows
503 convention: <tt>exe</tt> for executables, <tt>dll</tt> for dynamic libraries
504 and <tt>lib</tt> for static libraries. Lastly, the string "$toolset" in file
505 names is replaced by the name of tested toolset.</p>
506
507 <p><b>Note:</b> The <tt>List</tt> helper class might be useful to create
508 lists of names.</p>
509
510 <p><b>Note:</b> The file content can be examined using the
511 <tt>TestCmd.read</tt> function.</p>
512
513 <p>The members are:</p>
514
515 <ul>
516 <li>expect_addition</li>
517 <li>expect_removal</li>
518 <li>expect_modification</li>
519 <li>expect_nothing</li>
520 </ul>
521
522 <p>Note that <tt>expect_modification</tt> is used to check that a either
523 file content or timestamp has changed. The rationale is that some compilers
524 change content even if sources does not change, and it's easier to have a
525 method which checks for both content and time changes.</p>
526
527 <p>There's also a member <tt>expect_nothing_more</tt>, which checks that all
528 the changes are either expected or ignored, in other words that
529 <tt>unexpected_difference</tt> is empty by now.</p>
530
531 <p>Lastly, there's a method to compare file content with expected content:
532 </p>
533 <tt>expect_content(self, name, content, exact=0)</tt>
534
535 <p>The method fails the test if the content of file identified by 'name' is
536 different from 'content'. If 'exact' is true, the file content is used
537 as-is, otherwise, two transformations are applied:</p>
538
539 <ul>
540 <li>The <tt>read_and_strip</tt> method is used to read the file, which
541 removes trailing whitespace</li>
542
543 <li>Each backslash in the file content is converted to forward slash.</li>
544 </ul>
545
546 <h3><a name="methods-ignoring">Methods for ignoring changes</a></h3>
547
548 <p>There are five methods which ignore changes made to the working tree.
549 They silently remove elements from <tt>self.unexpected_difference</tt>, and
550 don't generate error if element is not found. They accept shell style
551 wildcard.</p>
552
553 <p>The following methods correspond to four kinds of changes:</p>
554
555 <ul>
556 <li>ignore_addition(self, wildcard)</li>
557 <li>ignore_removal(self, wildcard)</li>
558 <li>ignore_modification(self, wildcard)</li>
559 <li>ignore_touch(self, wilcard)</li>
560 </ul>
561
562 <p>The method <tt>ignore(self, wildcard)</tt> ignores all the changes made
563 to files that match a wildcard.</p>
564
565 <h3><a name="methods-result">Methods for explicitly specifying results</a>
566 </h3>
567
568 <h4>Method <tt>pass_test(self, condition=1)</tt></h4>
569
570 <div class="attention">
571 At this moment, the method should not be used.
572 </div>
573
574 <h4>Method <tt>fail_test(self, condition=1)</tt></h4>
575
576 <p><b>Effects:</b> Cause the test to fail if <tt>condition</tt> is true.</p>
577
578 <h3><a name="class-list">Helper class <tt>List</tt></a></h3>
579 The class has sequence interface and two additional methods.
580
581 <h4>Method <tt>__init__(self, string)</tt></h4>
582
583 <p><b>Effects:</b> Splits the string on unescaped spaces and tabs. The split
584 components can further be retrieved using standard sequence access.</p>
585
586 <h4>Method <tt>__mul__(self, other)</tt></h4>
587
588 <p><b>Effects:</b> Returns an <tt>List</tt> instance, which elements are all
589 possible concatenations of two string, first of which is from <tt>self</tt>,
590 and second of which is from <tt>other</tt>.</p>
591
592 <p>The class also defines <tt>__str__</tt> and <tt>__repr__</tt> methods.
593 Finally, there's <tt>__coerce__</tt> method which allows to convert strings
594 to instances of <tt>List</tt>.</p>
595
596 <p><b>Example:</b></p>
597 <pre>
598 l = "a b" * List("c d")
599 for e in l:
600 print e
601 </pre>
602
603 <p>will output:</p>
604 <pre>
605 ac
606 ad
607 bc
608 bd
609
610 </pre>
611 <hr>
612 <p class="revision">Last modified: May 02, 2008</p>
613 <p>&copy; Copyright Vladimir Prus 2002, 2003, 2004, 2005.<br>
614 &copy; Copyright Jurko Gospodnetic 2008.<br>
615 Distributed under the Boost Software License, Version 1.0.
616 (See accompanying file LICENSE_1_0.txt or http://www.boost.org/LICENSE_1_0.txt)</p>
617 </body>
618 </html>