1 <!DOCTYPE HTML PUBLIC
"-//W3C//DTD HTML 4.01//EN"
2 "http://www.w3.org/TR/html4/strict.dtd">
6 <meta name=
"generator" content=
7 "HTML Tidy for Linux/x86 (vers 1st March 2002), see www.w3.org">
8 <!--tidy options: -i -wrap 78 -->
9 <meta http-equiv=
"Content-Type" content=
"text/html; charset=iso-8859-1">
11 <title>A testing system for B2
</title>
12 <style type=
"text/css">
14 p.revision { text-align: right; font-style: italic }
15 pre.code { margin-left:
2em }
16 pre.example { margin-left:
2em; border: solid black thin }
17 pre.output { margin-left:
2em }
18 img.banner { border:
0; float: left }
19 h1 { text-align: right }
20 br.clear { clear: left }
21 div.attention { color: red }
27 <p><a href=
"../../../../index.htm"><img class=
"banner" height=
"86" width=
28 "277" alt=
"C++ Boost" src=
"../../../../boost.png"></a></p>
30 <h1>A testing system for B2
<br class=
"clear">
34 <dl class=
"page-index">
35 <dt><a href=
"#sec-intro">Introduction for users
</a></dt>
38 <dl class=
"page-index">
39 <dt><a href=
"#sec-command-line-options">Command line options
</a></dt>
43 <dt><a href=
"#sec-developers">Introduction for developers
</a></dt>
46 <dl class=
"page-index">
47 <dt><a href=
"#sec-intro-changing">Changing the working
50 <dt><a href=
"#sec-intro-examining">Examining the working directory and
53 <dt><a href=
"#sec-intro-results">Test result
</a></dt>
57 <dt><a href=
"#sec-reference">Reference documentation
</a></dt>
60 <dl class=
"page-index">
61 <dt><a href=
"#method-__init__">Method __init__
</a></dt>
63 <dt><a href=
"#method-set_tree">Method
<tt>set_tree
</tt></a></dt>
65 <dt><a href=
"#method-write">Method
<tt>write
</tt></a></dt>
67 <dt><a href=
"#method-copy">Method
<tt>copy
</tt></a></dt>
69 <dt><a href=
"#method-touch">Method
<tt>touch
</tt></a></dt>
71 <dt><a href=
"#method-run_build_system">Method
72 <tt>run_build_system
</tt></a></dt>
74 <dt><a href=
"#method-read">Method
<tt>read
</tt></a></dt>
76 <dt><a href=
"#method-read_and_strip">Method
77 <tt>read_and_strip
</tt></a></dt>
79 <dt><a href=
"#methods-expectations">Methods for declaring
82 <dt><a href=
"#methods-ignoring">Methods for ignoring
85 <dt><a href=
"#methods-result">Methods for explicitly specifying
88 <dt><a href=
"#class-list">Helper class
<tt>List
</tt></a></dt>
94 <h2><a name=
"sec-intro">Introduction for users
</a></h2>
96 <p>The testing system for B2 is a small set of Python modules and
97 scripts for automatically testing user-obversable behaviour. It uses
98 components from testing systems of
<a href=
"http://www.scons.org">Scons
</a>
99 and
<a href=
"http://subversion.tigris.org">Subversion
</a>, together with
100 some additional functionality.
</p>
102 <p>To run the tests you need to:
</p>
105 <li>Get the source tree of B2 (located at
<tt>tools/build
</tt>
108 <li>Have
<a href=
"http://www.python.org">Python
</a> installed. Version
109 2.1 is known to work.
</li>
111 <li>Build Boost.Jam. See
<a href=
112 "../engine/index.html">$boost_build_root/engine/index.html
</a> for
115 <li>Configure at least one toolset. You can edit
<tt>site-config.jam
</tt>
116 or
<tt>user-config.jam
</tt> to add new toolsets. Or you can create file
117 <tt>test-config.jam
</tt> in
<tt>$boost_build_root/test
</tt> directory. In
118 this case,
<tt>site-config.jam
</tt> and
<tt>user-config.jam
</tt> will be
119 ignored for testing.
</li>
121 <li>Make sure that in the configuration jam file(s) that you use you generate
122 no console output, ie. with the Boost Build 'ECHO' rule. Such console output
123 in the configuration jam file(s) will cause a number of tests to automatically
124 fail which would otherwise succeed.
</li>
127 <p>When all is set, you can run all the tests using the
<tt>test_all.py
</tt>
128 script or you can run a specific test by starting its Python script
135 python generators_test.py
138 <p>If everything is OK, you will see a list of passed tests. Otherwise, a
139 failure will be reported.
</p>
141 <h3><a name=
"sec-command-line-options">Command line options
</a></h3>
143 <p>Test scripts will use the toolset you configured to be the default or
144 you can specify a specific one on the command line:
</p>
147 python test_all.py borland
148 python generators_test.py msvc-
7.1
151 <p>Other test script flags you can specify on the command line are:
</p>
154 <li><tt>--default-bjam
</tt> -- By default the test system will use the
155 Boost Jam executable found built in its default development build
156 location. This option makes it use the default one available on your
157 system, i.e. the one found in the system path.
</li>
159 <li><tt>--preserve
</tt> -- In case of a failed test its working
160 directory will be copied to the
"failed_test" directory under the
161 current directory.
</li>
163 <li><tt>--verbose
</tt> -- Makes the test system and the run build system
164 display additional output. Note though that this may cause tests that
165 check the build system output to fail.
</li>
168 <h2><a name=
"sec-developers">Introduction for developers
</a></h2>
170 <p>It is suggested that every new functionality come together with tests,
171 and that bugfixes are accompanied by tests. There's no need to say that
172 tests are good, but two points are extremely important:
</p>
175 <li>For an interpreted language like Jam, without any static checks,
176 testing is simply the only sefeguard we can have.
</li>
178 <li>Good tests allow us to change internal design much more safely, and we
179 have not gotten everything nailed down yet.
</li>
182 <p>Adding a new test is simple:
</p>
185 <li>Go to
<tt>$boost_build_root/test/test_all.py
</tt> and add new test
186 name to the list at the end of the file. Suppose the test name is
"hello".
189 <li>Add a new python module, in this example
"hello.py", to do the actual
193 <p>The module, in general will perform these basic actions:
</p>
196 <li>Set up the initial working directory state
</li>
199 Run the build system and check the results:
202 <li>generated output,
</li>
204 <li>changes made to the working directory,
</li>
206 <li>new content of the working directory.
</li>
210 <li>Add, remove or touch files or change their content and then repeat
211 the previous step until satisfied.
</li>
216 <p>The
"hello.py" module might contain:
</p>
217 <pre class=
"example">
218 from BoostBuild import List
220 # Create a temporary working directory
221 t = BoostBuild.Tester()
223 # Create the needed files
224 t.write(
"jamroot.jam",
"")
225 t.write(
"jamfile.jam",
"""
226 exe hello : hello.cpp ;
228 t.write(
"hello.cpp",
"""
238 # First, create a list of three pathnames.
239 file_list = List(
"bin/$toolset/debug/") * List(
"hello.exe hello.obj")
240 # Second, assert that those files were added as result of the last build system invocation.
241 t.expect_addition(file_list)
243 # Invoke the build system once again.
244 t.run_build_system(
"clean")
245 # Check if the files added previously were removed.
246 t.expect_removal(file_list)
248 # Remove temporary directories
252 <p>The
<tt>test
</tt> directory contains a file
"template.py" which can be
253 used as a start for your own tests.
</p>
255 <p>Overview of the most important methods of class
<tt>Tester
</tt> follows.
258 <h3><a name=
"sec-intro-changing">Changing the working directory
</a></h3>
260 <p>The class
<tt>Tester
</tt> creates a temporary directory in its
261 constructor and changes to that directory. It can be modified by calling
265 <li><tt>set_tree
</tt> -- sets the content of the working directory to be
266 equal to the content of the specified directory. This method is
267 preferable when directory tree for testing is large.
</li>
269 <li><tt>write
</tt> -- sets the content of file in a working directory.
270 This is optimal if you want to create a directory tree with
3-
4 small
273 <li><tt>touch
</tt> -- changes the modification times of a file
</li>
276 <h3><a name=
"sec-intro-examining">Examining the working directory and
279 <p>The method
<tt>read
</tt>, inherited from the
<tt>TestCmd
</tt> class, can
280 be used to read any file in the working directory and check its content.
281 <tt>Tester
</tt> adds another method for tracking changes. Whenever the build
282 system is run (using
<a href=
"#method-run_build_system"><tt>run_build_system
283 </tt></a>), the working dir state before and after running is recorded. In
284 addition, difference between the two states -- i.e. lists of files that were
285 added, removed, modified or touched -- are stored in two member variables -
286 <tt>tree_difference
</tt> and
<tt>unexpected_difference
</tt>.
</p>
288 <p>After than, the test author may specify that some change is expected, for
289 example, by calling
<tt>expect_addition(
"foo")
</tt>. This call will check if
290 the file was indeed added, and if so, will remove its name from the list of
291 added files in
<tt>unexpected_difference
</tt>. Likewise, it is possible to
292 specify that some changes are not interesting, for example a call to
293 <tt>ignore(
"*.obj")
</tt> will just remove every file with the
".obj"
294 extension from
<tt>unexpected_difference
</tt>.
</p>
296 <p>When test has finished with expectations and ignoring, the member
297 <tt>unexpected_difference
</tt> will contain the list of all changes not yet
298 accounted for. It is possible to assure that this list is empty by calling
299 the
<tt>expect_nothing_more
</tt> member function.
</p>
301 <h3><a name=
"sec-intro-results">Test result
</a></h3>
303 <p>Any of the
<tt>expect*
</tt> methods below will fail the test if the
304 expectation is not met. It is also possible to perform manually arbitrary
305 test and explicitly cause the test to either pass or fail. Ordinary
306 filesystem functions can be used to work with the directory tree. Methods
307 <tt>pass_test
</tt> and
<tt>fail_test
</tt> are used to explicitly give the
310 <p>Typically, after test termination, the working directory is erased. See
311 the
<a href=
"#sec-command-line-options">"--preserve" command line option
</a>
312 for information on how to preserve the working directory content for failed
313 tests for debugging purposes.
</p>
315 <h2 id=
"sec-reference">Reference documentation
</h2>
317 <p>The test system is composed of class
<tt>Tester
</tt>, derived form
318 <tt>TestCmd.TestCmd
</tt>, and helper class
<tt>List
</tt>.
<tt>Tester
</tt>
319 and
<tt>List
</tt> methods are described below.
</p>
321 <p>The documentation frequently refers to
<tt>filename
</tt>. In all cases,
322 files are specified in unix style: a sequence of components, separated by
323 "/". This is true on all platforms. In some contexts a list of files is
324 allowed. In those cases any object with a sequence interface is allowed.
</p>
326 <h3><a name=
"method-__init__">Method
<tt>__init__(self,
arguments=
"",
327 executable=
"bjam", match=TestCmd.match_exact, boost_build_path=None,
328 translate_suffixes=True, pass_toolset=True, use_test_config=True,
329 ignore_toolset_requirements=True,
workdir=
"", **keywords)
</tt></a></h3>
331 <p><b>Optional arguments:
</b></p>
334 <li><tt>arguments
</tt>
335 -- Arguments passed to the run executable.
</li>
336 <li><tt>executable
</tt>
337 -- Name of the executable to invoke.
</li>
339 -- Function to use for compating actual and expected file contents.
341 <li><tt>boost_build_path
</tt>
342 -- Boost build path to be passed to the run executable.
</li>
343 <li><tt>translate_suffixes
</tt>
344 -- Whether to update suffixes on the the file names passed from the
345 test script so they match those actually created by the current
346 toolset. For example, static library files are specified by using
347 the .lib suffix but when the 'gcc' toolset is used it actually
348 creates them using the .a suffix.
</li>
349 <li><tt>pass_toolset
</tt>
350 -- Whether the test system should pass the specified toolset to the
352 <li><tt>use_test_config
</tt>
353 -- Whether the test system should tell the run executable to read in
354 the test_config.jam configuration file.
</li>
355 <li><tt>ignore_toolset_requirements
</tt>
356 -- Whether the test system should tell the run executable to ignore
357 toolset requirements.
</li>
359 -- Indicates an absolute directory where the test will be run from.
363 <p><b>Optional arguments inherited from the base class:
</b></p>
366 <li><tt>description
</tt>
367 -- Test description string displayed in case of a failed test.
</li>
369 -- List of subdirectories to automatically create under the working
370 directory. Each subdirectory needs to be specified separately
371 parent coming before its child.
</li>
373 -- Flag that may be used to enable more verbose test system output.
374 Note that it does not also enable more verbose build system output
375 like the
<a href=
"#sec-command-line-options">"--verbose" command
376 line option
</a> does.
</li>
379 <p><b>Effects:
</b></p>
382 <li>Remembers the current working directory in member
383 <tt>original_workdir
</tt>.
</li>
385 <li>Determines the location of the executable (
<code>bjam
</code> by
386 default) and build system files, assuming that the current directory is
387 <tt>tools/build/test
</tt>. Formulates jam invocation command, which
388 will include explicit setting for the
<tt>BOOST_BUILD_PATH
</tt> variable
389 and arguments passed to this methods, if any. This command will be used
390 by subsequent invocation of
<a href=
"#method-run_build_system"><tt>
391 run_build_system
</tt></a>. Finally, initializes the base class.
</li>
393 <li>Changes the current working directory to the temporary working
394 directory created by the base constructor.
</li>
396 <li>If you want to run a test in an existing directory, pass it as
397 <tt>workdir
</tt>.
</li>
399 <li> Most parameters passed to this constructor function may be overruled
400 for each specific test system run using
<a href=
401 "#method-run_build_system"><tt>run_build_system
</tt></a> parameters.
404 <h3><a name=
"method-set_tree">Method
<tt>set_tree(self,
405 tree_location)
</tt></a></h3>
407 <p><b>Effects:
</b></p>
409 <p>Replaces the content of the current working directory with the content
410 of directory at
<tt>tree_location
</tt>. If
<tt>tree_location
</tt> is not
411 absolute pathname, it will be treated as relative to
412 <tt>self.original_workdir
</tt>. This methods also explicitly makes the
413 copied files writeable.
</p>
415 <h3><a name=
"method-write">Method
<tt>write(self, name,
416 content)
</tt></a></h3>
418 <p><b>Effects:
</b></p>
420 <p>Writes the specified content to the file given by
<tt>name
</tt> under
421 the temporary working directory. If the file already exists, it is
422 overwritten. Any required directories are automatically created.
</p>
424 <h3><a name=
"method-copy">Method
<tt>copy(self, src, dst)
</tt></a></h3>
426 <p><b>Effects:
</b></p>
428 <p>Equvivalent to
<tt>self.write(self.read(src), dst)
</tt>.
</p>
430 <h3><a name=
"method-touch">Method
<tt>touch(self, names)
</tt></a></h3>
432 <p><b>Effects:
</b></p>
434 <p>Sets the access and modification times for all files in
<tt>names
</tt> to
435 the current time. All the elements in
<tt>names
</tt> should be relative
438 <h3><a name=
"method-run_build_system">Method
<tt>run_build_system(self,
439 extra_args=
"",
subdir=
"", stdout=None,
stderr=
"", status=
0, match=None,
440 pass_toolset=None, use_test_config=None, ignore_toolset_requirements=None,
441 expected_duration=None, **kw)
</tt></a></h3>
443 <p><b>Effects:
</b></p>
446 <li>Stores the state of the working directory in
447 <tt>self.previous_tree
</tt>.
</li>
449 <li>Changes to
<tt>subdir
</tt>, if it is specified. It is relative to
450 the
<tt>original_workdir
</tt> or the workdir specified in
451 <tt>__init
</tt>.
</li>
453 <li>Invokes the
<tt>bjam
</tt> executable, passing
<tt>extra_args
</tt>
454 to it. The binary should be located under
455 <tt><test_invocation_dir
>/../jam/src/bin.
<platform
></tt>.
456 This is to make sure tests use the version of jam build from CVS.
</li>
458 <li>Compares the stdout, stderr and exit status of build system
459 invocation with values to appropriate parameters, if they are not
460 <tt>None
</tt>. If any difference is found, the test fails.
</li>
462 <li>If the
<tt>expected_duration
</tt> parameter is specified then it
463 represents the maximal allowed time in seconds for the test to run. The
464 test will be marked as failed if its duration is greater than the given
465 <tt>expected_duration
</tt> parameter value.
</li>
467 <li>Stores the new state of the working directory in
<tt>self.tree
</tt>.
468 Computes the difference between previous and current trees and stores them
469 in variables
<tt>self.tree_difference
</tt> and
470 <tt>self.unexpected_difference
</tt>. Both variables are instances of class
471 <tt>tree.Trees_different
</tt>, which have four attributes:
472 <tt>added_files
</tt>,
<tt>removed_files
</tt>,
<tt>modified_files
</tt> and
473 <tt>touched_files
</tt>. Each is a list of strings.
</p></li>
476 <h3><a name=
"method-read">Method
<tt>read(self, name)
</tt></a></h3>
478 <p><b>Effects:
</b></p>
480 <p>Read the specified file and returns it content. Raises an exception is
481 the file is absent.
</p>
483 <h3><a name=
"method-read_and_strip">Method
<tt>read_and_strip(self, name)
486 <p><b>Effects:
</b></p>
488 <p>Read the specified file and returns it content, after removing trailing
489 whitespace from every line. Raises an exception is the file is absent.
</p>
491 <p><b>Rationale:
</b></p>
493 <p>Although this method is questionable, there are a lot of cases when jam
494 or shells it uses insert spaces. It seems that introducing this method is
495 much simpler than dealing with all those cases.
</p>
497 <h3><a name=
"methods-expectations">Methods for declaring expectations
</a>
500 <p>Accordingly to the number of changes kinds that are detected, there are
501 four methods that specify that test author expects a specific change to
502 occur. They check
<tt>self.unexpected_difference
</tt>, and if the change is
503 present there, it is removed. Otherwise, test fails.
</p>
505 <p>Each method accepts a list of names. Those names use
<tt>/
</tt> path
506 separator on all systems. Additionally, the test system translates suffixes
507 appropriately. For the test to be portable, suffixes should use Windows
508 convention:
<tt>exe
</tt> for executables,
<tt>dll
</tt> for dynamic libraries
509 and
<tt>lib
</tt> for static libraries. Lastly, the string
"$toolset" in file
510 names is replaced by the name of tested toolset.
</p>
512 <p><b>Note:
</b> The
<tt>List
</tt> helper class might be useful to create
515 <p><b>Note:
</b> The file content can be examined using the
516 <tt>TestCmd.read
</tt> function.
</p>
518 <p>The members are:
</p>
521 <li>expect_addition
</li>
522 <li>expect_removal
</li>
523 <li>expect_modification
</li>
524 <li>expect_nothing
</li>
527 <p>Note that
<tt>expect_modification
</tt> is used to check that a either
528 file content or timestamp has changed. The rationale is that some compilers
529 change content even if sources does not change, and it's easier to have a
530 method which checks for both content and time changes.
</p>
532 <p>There's also a member
<tt>expect_nothing_more
</tt>, which checks that all
533 the changes are either expected or ignored, in other words that
534 <tt>unexpected_difference
</tt> is empty by now.
</p>
536 <p>Lastly, there's a method to compare file content with expected content:
538 <tt>expect_content(self, name, content, exact=
0)
</tt>
540 <p>The method fails the test if the content of file identified by 'name' is
541 different from 'content'. If 'exact' is true, the file content is used
542 as-is, otherwise, two transformations are applied:
</p>
545 <li>The
<tt>read_and_strip
</tt> method is used to read the file, which
546 removes trailing whitespace
</li>
548 <li>Each backslash in the file content is converted to forward slash.
</li>
551 <h3><a name=
"methods-ignoring">Methods for ignoring changes
</a></h3>
553 <p>There are five methods which ignore changes made to the working tree.
554 They silently remove elements from
<tt>self.unexpected_difference
</tt>, and
555 don't generate error if element is not found. They accept shell style
558 <p>The following methods correspond to four kinds of changes:
</p>
561 <li>ignore_addition(self, wildcard)
</li>
562 <li>ignore_removal(self, wildcard)
</li>
563 <li>ignore_modification(self, wildcard)
</li>
564 <li>ignore_touch(self, wildcard)
</li>
567 <p>The method
<tt>ignore(self, wildcard)
</tt> ignores all the changes made
568 to files that match a wildcard.
</p>
570 <h3><a name=
"methods-result">Methods for explicitly specifying results
</a>
573 <h4>Method
<tt>pass_test(self, condition=
1)
</tt></h4>
575 <div class=
"attention">
576 At this moment, the method should not be used.
579 <h4>Method
<tt>fail_test(self, condition=
1)
</tt></h4>
581 <p><b>Effects:
</b> Cause the test to fail if
<tt>condition
</tt> is true.
</p>
583 <h3><a name=
"class-list">Helper class
<tt>List
</tt></a></h3>
584 The class has sequence interface and two additional methods.
586 <h4>Method
<tt>__init__(self, string)
</tt></h4>
588 <p><b>Effects:
</b> Splits the string on unescaped spaces and tabs. The split
589 components can further be retrieved using standard sequence access.
</p>
591 <h4>Method
<tt>__mul__(self, other)
</tt></h4>
593 <p><b>Effects:
</b> Returns an
<tt>List
</tt> instance, which elements are all
594 possible concatenations of two string, first of which is from
<tt>self
</tt>,
595 and second of which is from
<tt>other
</tt>.
</p>
597 <p>The class also defines
<tt>__str__
</tt> and
<tt>__repr__
</tt> methods.
598 Finally, there's
<tt>__coerce__
</tt> method which allows to convert strings
599 to instances of
<tt>List
</tt>.
</p>
601 <p><b>Example:
</b></p>
603 l =
"a b" * List(
"c d")
617 <p class=
"revision">Last modified: May
02,
2008</p>
618 <p>© Copyright Vladimir Prus
2002,
2003,
2004,
2005.
<br>
619 © Copyright Jurko Gospodnetic
2008.
<br>
620 Distributed under the Boost Software License, Version
1.0.
621 (See accompanying file LICENSE.txt or https://www.bfgroup.xyz/b2/LICENSE.txt)
</p>