Planet
navi homePPSaboutscreenshotsdownloaddevelopmentforum

source: downloads/boost_1_34_1/tools/build/v2/test/test_system.html @ 29

Last change on this file since 29 was 29, checked in by landauf, 17 years ago

updated boost from 1_33_1 to 1_34_1

File size: 19.3 KB
Line 
1<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN"
2"http://www.w3.org/TR/html4/strict.dtd">
3
4<html>
5  <head>
6    <meta name="generator" content=
7    "HTML Tidy for Linux/x86 (vers 1st March 2002), see www.w3.org">
8    <!--tidy options: -i -wrap 78 -->
9    <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
10
11    <title>A testing system for Boost.Build</title>
12<style type="text/css">
13        hr { color: black }
14        p.revision { text-align: right; font-style: italic }
15        pre.code { margin-left: 2em } 
16        pre.example { margin-left: 2em; border: solid black thin } 
17        pre.output { margin-left: 2em }
18        img.banner { border: 0; float: left }
19        h1 { text-align: right }
20        br.clear { clear: left }
21        div.attention { color: red }
22   
23</style>
24  </head>
25
26  <body>
27    <p><a href="../../../../index.htm"><img class="banner" height="86" width=
28    "277" alt="C++ Boost" src="../../../../boost.png"></a></p>
29
30    <h1>A testing system for Boost.Build<br class="clear">
31    </h1>
32    <hr>
33
34    <dl class="page-index">
35      <dt><a href="#sec-intro">Introduction for users</a></dt>
36
37      <dt><a href="#sec-developers">Introduction for developers</a></dt>
38
39      <dd>
40        <dl class="page-index">
41          <dt><a href="#sec-intro-changing">Changing the working
42          directory</a></dt>
43
44          <dt><a href="#sec-intro-examining">Examining the working directory
45          and changing it</a></dt>
46
47          <dt><a href="#sec-intro-results">Test result</a></dt>
48        </dl>
49      </dd>
50
51      <dt><a href="#sec-reference">Reference documentation</a></dt>
52
53      <dd>
54        <dl class="page-index">
55          <dt><a href="#method-__init__">Method __init__</a></dt>
56
57          <dt><a href="#method-set_tree">Method <tt>set_tree</tt></a></dt>
58
59          <dt><a href="#method-write">Method <tt>write</tt></a></dt>
60
61          <dt><a href="#method-copy">Method <tt>copy</tt></a></dt>
62
63          <dt><a href="#method-touch">Method <tt>touch</tt></a></dt>
64
65          <dt><a href="#method-run_build_system">Method
66          <tt>run_build_system</tt></a></dt>
67
68          <dt><a href="#method-read">Method <tt>read</tt></a></dt>
69
70          <dt><a href="#method-read_and_strip">Method
71          <tt>read_and_strip</tt></a></dt>
72
73          <dt><a href="#methods-expectations">Methods for declaring
74          expectations</a></dt>
75
76          <dt><a href="#methods-ignoring">Methods for ignoring
77          changes</a></dt>
78
79          <dt><a href="#methods-result">Methods for explicitly specifying
80          results</a></dt>
81
82          <dt><a href="#class-list">Helper class <tt>List</tt></a></dt>
83        </dl>
84      </dd>
85    </dl>
86    <hr>
87
88    <h2><a name="sec-intro">Introduction for users</a></h2>
89
90    <p>The testing system for Boost.Build is a small set of Python modules
91    and scripts for automatically testing user-obversable behaviour. It uses
92    components from testing systems of <a href=
93    "http://www.scons.org">Scons</a> and <a href=
94    "http://subversion.tigris.org">Subverion</a>, together with some
95    additional functionality.</p>
96
97    <p>To run the tests you'd need:</p>
98
99    <ol>
100      <li>Get the source tree of Boost.Build (located at <tt>tools/build</tt>
101      in Boost)</li>
102
103      <li>Have <a href="http://www.python.org">Python</a> installed. Version
104      2.1 is known to work.</li>
105
106      <li>Build Boost.Jam. See <a href=
107      "../../../jam/index.html">$boost_build_root/jam/index.html</a> for
108      instructions.</li>
109
110      <li>Configure at least one toolset. You can edit
111      <tt>site-config.jam</tt> or <tt>user-config.jam</tt> to add new
112      toolsets. Or you can create file <tt>test-config.jam</tt> in
113      <tt>$boost_build_root/test</tt> directory. In this case,
114      <tt>site-config.jam</tt> and <tt>user-config.jam</tt> will be ignored
115      for testing.</li>
116    </ol>
117
118    <p>When all is done, you can run the tests with</p>
119<pre class="code">
120python test_all.py
121</pre>
122
123    <p>which will use the default toolset, or you can specify toolset on the
124    command line, for example:</p>
125<pre class="code">
126python test_all.py borland
127</pre>
128
129    <p>If everything's OK, you'll see a list of passed tests. Otherwise, a
130    failure will be reported.</p>
131
132    <p>It is possible to run a specific test, for example:</p>
133<pre class="code">
134python generators_test.py
135</pre>
136
137    <h2><a name="sec-developers">Introduction for developers</a></h2>
138
139    <p>It is suggested that every new functionality come together with tests,
140    and that bugfixes are accompanied by tests. There's no need to say that
141    tests are good, but two points are extremely important:</p>
142
143    <ul>
144      <li>For an interpreted language like Jam, without any static checks,
145      testing is simply the only sefeguard we can have.</li>
146
147      <li>Good tests allow to change internal design much more safely, and we
148      didn't nailed everything down yet.</li>
149    </ul>
150
151    <p>Adding a new test is simple:</p>
152
153    <ol>
154      <li>Go to <tt>$boost_build_root/test/test_all.py</tt> and add new test
155      name to the list at the end of file. Suppose the test name is
156      "hello".</li>
157
158      <li>Add a new python module, in this example "hello.py", to do actual
159      testing.</li>
160    </ol>
161
162    <p>The module, in general will perform these basic actions:</p>
163
164    <ol>
165      <li>Setting the initial working directory state</li>
166
167      <li>
168        Running the build system and checking:
169
170        <ol>
171          <li>generated output,</li>
172
173          <li>changes made to the working directory,</li>
174
175          <li>new content of the working directory.</li>
176        </ol>
177      </li>
178
179      <li>Adding, removing or touching files, or changing their content and
180      then repeating the previous step, until satisfied.</li>
181
182      <li>Cleaning up</li>
183    </ol>
184
185    <p>The "hello.py" module might contain:</p>
186<pre class="example">
187from BoostBuild import Tester, List
188
189# Create a temporary working directory
190t = Tester()
191
192# Create the needed files
193t.write("project-root.jam", "")
194t.write("Jamfile", """
195exe hello : hello.cpp ;
196""")
197t.write("hello.cpp", """
198int main()
199{
200    return 0;
201}
202
203""")
204
205t.run_build_system()
206
207# First, create a list of three pathnames
208file_list = List("bin/$toolset/debug/") * List("hello.exe hello.obj")
209# Second, assert that those files were added as result of the last build system invocation.
210t.expect_addition(file_list)
211
212# Invoke the build system once again
213t.run_build_system("clean")
214# Check if the files added previously were removed.
215t.expect_removal(file_list)
216
217# Remove temporary directories
218t.cleanup()
219</pre>
220
221    <p>The <tt>test</tt> directory contains a file "template.py" which can be
222    used as a start for your own tests.</p>
223
224    <p>Overview of the most important methods of class
225    <tt>TestBoostBuild</tt> follows.</p>
226
227    <h3><a name="sec-intro-changing">Changing the working directory</a></h3>
228
229    <p>The class <tt>TestBoostBuild</tt> creates a temporary directory in its
230    constructor and changes to that directory. It can be modified by calling
231    these methods:</p>
232
233    <ul>
234      <li><tt>set_tree</tt> -- sets the content of the working directory to
235      be equal to the content of the specified directory. This method is
236      preferrable when directory tree for testing is large.</li>
237
238      <li><tt>write</tt> -- sets the content of file in a working directory.
239      This is optimal if you want to create a directory tree with 3-4 small
240      files.</li>
241
242      <li><tt>touch</tt> -- changes the modification times of a file</li>
243    </ul>
244
245    <h3><a name="sec-intro-examining">Examining the working directory and
246    changing it</a></h3>
247
248    <p>The method <tt>read</tt>, inherited from the <tt>TestCmd</tt> class,
249    can be used to read any file in the working directory and check its
250    content. <tt>TestBoostBuild</tt> adds another method for tracking
251    changes. Whenever build system is run (via <tt>run_build_system</tt>),
252    the state of working dir before and after running is recorded. In
253    addition, difference between the two states -- i.e. lists of files that
254    were added, removed, modified or touched -- is stored in two member
255    variables, <tt>tree_difference</tt> and
256    <tt>unexpected_difference</tt>.</p>
257
258    <p>After than, the test author may specify that some change is expected,
259    for example, by calling <tt>expect_addition("foo")</tt>. This call will
260    check if the file was indeed added, and if so, will remove its name from
261    the list of added files in <tt>unexpected_difference</tt>. Likewise, it's
262    possible to specify that some changes are not interesting, for example a
263    call <tt>ignore("*.obj")</tt> will just remove every files with ".obj"
264    extension from <tt>unexpected_difference</tt>.</p>
265
266    <p>When test has finished with expectations and ignoring, the member
267    <tt>unexpected_difference</tt> will contain the list of all changes not
268    yet accounted for. It is possible to assure that this list is empty by
269    calling <tt>expect_nothing_more</tt> member function.</p>
270
271    <h3><a name="sec-intro-results">Test result</a></h3>
272
273    <p>Any of the <tt>expect*</tt> methods below will fail the test if the
274    expectation is not met. It is also possible to perform manually arbitrary
275    test and explicitly cause the test to either pass or fail. Ordinary
276    filesystem functions can be used to work with the directory tree. Methods
277    <tt>pass_test</tt> and <tt>fail_test</tt> are used to explicitly give the
278    test outcome.</p>
279
280    <p>Typically, after test termination, the working directory is erased. To
281    debug a failed test, you should add "--preserve" option when invoking
282    test. On failure, the working directory will be copied to "failed_test"
283    directory in the current dir.</p>
284
285    <h2 id="sec-reference">Reference documentation</h2>
286
287    <p>The test system is composed of class <tt>Tester</tt>, derived form
288    <tt>TestCmd.TestCmd</tt>, and helper class <tt>List</tt>. The methods of
289    <tt>Tester</tt>, and the class <tt>List</tt> are described below.</p>
290
291    <p>The documentation frequently refer to filename. In all cases, files
292    are specified in unix style: a sequence of components, separated by "/".
293    This is true on all platforms. In some contexts, a list of files is
294    allowed. In that case any object with sequence interface is allowed.</p>
295
296    <h3><a name="method-__init__">Method <tt>__init__(self, workdir='',
297    arguments='', executable='bjam')</tt></a></h3>
298
299    <p><b>Effects:</b></p>
300
301    <ol>
302      <li>Remembers the current working directory in member
303      <tt>original_workdir</tt>.</li>
304
305      <li>Determines the location of executable (<code>bjam</code> by
306      default) and build system files, assuming that the current directory is
307      <tt>tools/build/test</tt>. Formulates jam invocation command, which
308      will include explicit setting for <tt>BOOST_BUILD_PATH</tt> variable
309      and arguments passed to this methods, if any. This command will be used
310      by subsequent invocation of <a href=
311      "#method-run_build_system"><tt>run_build_system</tt></a>. Finally,
312      initializes the base class.</li>
313
314      <li>Changes current working dir to the temporary working directory
315      created by the base constructor.</li>
316
317      <li>If you want to run a test in a existing directory, pass it to
318      <tt>workdir</tt>.</li>
319    </ol>
320
321    <h3><a name="method-set_tree">Method <tt>set_tree(self,
322    tree_location)</tt></a></h3>
323
324    <p><b>Effects:</b></p>
325
326    <p>Replaces the content of the current working directory with the content
327    of directory at <tt>tree_location</tt>. If <tt>tree_location</tt> is not
328    absolute pathname, it will be treated as relative to
329    <tt>self.original_workdir</tt>. This methods also explicitly makes the
330    copied files writeable.</p>
331
332    <h3><a name="method-write">Method <tt>write(self, name,
333    content)</tt></a></h3>
334
335    <p><b>Effects:</b></p>
336
337    <p>Writes the specified content to the file given by <tt>name</tt> under
338    the temporary working directory. If the file already exists, it is
339    overwritten. Any required directories are automatically created.</p>
340
341    <h3><a name="method-copy">Method <tt>copy(self, src, dst)</tt></a></h3>
342
343    <p><b>Effects:</b></p>
344
345    <p>Equvivalent to <tt>self.write(self.read(src), dst)</tt>.</p>
346
347    <h3><a name="method-touch">Method <tt>touch(self, names)</tt></a></h3>
348
349    <p><b>Effects:</b></p>
350
351    <p>Sets the access and modification times for all files in <tt>names</tt>
352    to the current time. All the elements in <tt>names</tt> should be
353    relative paths.</p>
354
355    <h3><a name="method-run_build_system">Method <tt>run_build_system(self,
356    subdir='', extra_args='', stdout=None, stderr='', status=0,
357    **kw)</tt></a></h3>
358
359    <p><b>Effects:</b></p>
360
361    <ol>
362      <li>Stores the state of the working directory in
363      <tt>self.previous_tree</tt>.</li>
364
365      <li>Changes to <tt>subdir</tt>, if it is specified. It is relative to
366      the <tt>original_workdir</tt> or the workdir specified in
367      <tt>__init</tt>.</li>
368
369      <li>Invokes the <tt>bjam</tt> executable, passing <tt>extra_args</tt>
370      to it. The binary should be located under
371      <tt>&lt;test_invocation_dir&gt;/../jam/src/bin.&lt;platform&gt;</tt>.
372      This is to make sure tests use the version of jam build from CVS.</li>
373
374      <li>Compares the stdout, stderr and exit status of build system
375      invocation with values to appropriate parameters, if they are not
376      <tt>None</tt>. If any difference is found, the test fails.</li>
377
378      <li>
379        <p>Stores the new state of the working directory in
380        <tt>self.tree</tt>. Computes the difference between previous and
381        current trees and store them in variables
382        <tt>self.tree_difference</tt> and
383        <tt>self.unexpected_difference</tt>.</p>
384
385        <p>Both variables are instances of class
386        <tt>tree.Trees_different</tt>, which have four attributes:
387        <tt>added_files</tt>, <tt>removed_files</tt>, <tt>modified_files</tt>
388        and <tt>touched_files</tt>. Each is a list of strings.</p>
389      </li>
390    </ol>
391
392    <h3><a name="method-read">Method <tt>read(self, name)</tt></a></h3>
393
394    <p><b>Effects:</b></p>
395
396    <p>Read the specified file and returns it content. Raises an exception is
397    the file is absent.</p>
398
399    <h3><a name="method-read_and_strip">Method <tt>read_and_strip(self,
400    name)</tt></a></h3>
401
402    <p><b>Effects:</b></p>
403
404    <p>Read the specified file and returns it content, after removing
405    trailing whitespace from every line. Raises an exception is the file is
406    absent.</p>
407
408    <p><b>Rationale:</b></p>
409
410    <p>Althought this method is questionable, there are a lot of cases when
411    jam or shells it uses insert spaces. It seems that introducing this
412    method is much simpler than dealing with all those cases.</p>
413
414    <h3><a name="methods-expectations">Methods for declaring
415    expectations</a></h3>
416
417    <p>Accordingly to the number of changes kinds that are detected, there
418    are four methods that specify that test author expects a specific change
419    to occur. They check <tt>self.unexpected_difference</tt>, and if the
420    change is present there, it is removed. Otherwise, test fails.</p>
421
422    <p>Each method accepts a list of names. Those names use <tt>/</tt> path
423    separator on all systems. Additionaly, the test system translates
424    suffixes appropriately. For the test to be portable, suffixes should use
425    Windows convention: <tt>exe</tt> for executables, <tt>dll</tt> for
426    dynamic libraries and <tt>lib</tt> for static libraries. Lastly, the
427    string "$toolset" in file names is replaced by the name of tested
428    toolset.</p>
429
430    <p><b>Note:</b> The <tt>List</tt> helper class might be useful to create
431    lists of names.</p>
432
433    <p><b>Note:</b> The file content can be examined using
434    <tt>TestCmd.read</tt> function.</p>
435
436    <p>The members are:</p>
437
438    <ul>
439      <li>expect_addition</li>
440
441      <li>expect_removal</li>
442
443      <li>expect_modification</li>
444
445      <li>expect_nothing</li>
446    </ul>
447
448    <p>Note that <tt>expect_modification</tt> is used to check that a either
449    file content or timestamp has changed. The rationale is that some
450    compilers change content even if sources does not change, and it's easier
451    to have a method which checks for both content and time changes.</p>
452
453    <p>There's also a member <tt>expect_nothing_more</tt>, which checks that
454    all the changes are either expected or ignored, in other words that
455    <tt>unexpected_difference</tt> is empty by now.</p>
456
457    <p>Lastly, there's a method to compare file content with expected
458    content:</p>
459    <tt>expect_content(self, name, content, exact=0)</tt> 
460
461    <p>The method fails the test if the content of file identified by 'name'
462    is different from 'content'. If 'exact' is true, the file content is used
463    as-is, otherwise, two transformations are applied:</p>
464
465    <ul>
466      <li>The <tt>read_and_strip</tt> method is used to read the file, which
467      removes trailing whitespace</li>
468
469      <li>Each backslash in the file content is converted to forward
470      slash.</li>
471    </ul>
472
473    <h3><a name="methods-ignoring">Methods for ignoring changes</a></h3>
474
475    <p>There are five methods which ignore changes made to the working tree.
476    They silently remove elements from <tt>self.unexpected_difference</tt>,
477    and don't generate error if element is not found. They accept shell style
478    wildcard.</p>
479
480    <p>The following methods correspond to four kinds of changes:</p>
481
482    <ul>
483      <li>ignore_addition(self, wildcard)</li>
484
485      <li>ignore_removal(self, wildcard)</li>
486
487      <li>ignore_modification(self, wildcard)</li>
488
489      <li>ignore_touch(self, wilcard)</li>
490    </ul>
491
492    <p>The method <tt>ignore(self, wildcard)</tt> ignores all the changes
493    made to files that match a wildcard.</p>
494
495    <h3><a name="methods-result">Methods for explicitly specifying
496    results</a></h3>
497
498    <h4>Method <tt>pass_test(self, condition=1)</tt></h4>
499
500    <div class="attention">
501      At this moment, the method should not be used.
502    </div>
503
504    <h4>Method <tt>fail_test(self, condition=1)</tt></h4>
505
506    <p><b>Effects:</b> Cause the test to fail if <tt>condition</tt> is
507    true.</p>
508
509    <h3><a name="class-list">Helper class <tt>List</tt></a></h3>
510    The class has sequence interface and two additional methods.
511
512    <h4>Method <tt>__init__(self, string)</tt></h4>
513
514    <p><b>Effects:</b> Splits the string on unescaped spaces and tabs. The
515    split components can further be retrieved using standard sequence
516    access.</p>
517
518    <h4>Method <tt>__mul__(self, other)</tt></h4>
519
520    <p><b>Effects:</b> Returns an <tt>List</tt> instance, which elements are
521    all possible concatenations of two string, first of which is from
522    <tt>self</tt>, and second of which is from <tt>other</tt>.</p>
523
524    <p>The class also defines <tt>__str__</tt> and <tt>__repr__</tt> methods.
525    Finally, there's <tt>__coerce__</tt> method which allows to convert
526    strings to instances of <tt>List</tt>.</p>
527
528    <p><b>Example:</b></p>
529<pre>
530    l = "a b" * List("c d")
531    for e in l:
532        print e       
533   
534</pre>
535
536    <p>will output</p>
537<pre>
538    ac
539    ad
540    bc
541    bd
542   
543</pre>
544    <hr>
545
546    <p class="revision">Last modified: Mar 11, 2005</p>
547
548    <p>&copy; Copyright Vladimir Prus 2002, 2003, 2004, 2005. Permission to
549    copy, use, modify, sell and distribute this document is granted provided
550    this copyright notice appears in all copies. This document is provided
551    ``as is'' without express or implied warranty, and with no claim as to
552    its suitability for any purpose.</p>
553  </body>
554</html>
555
Note: See TracBrowser for help on using the repository browser.