tests/test-runner/man/test-runner.1

   1 .\"
   2 .\" This file and its contents are supplied under the terms of the
   3 .\" Common Development and Distribution License ("CDDL"), version 1.0.
   4 .\" You may only use this file in accordance with the terms of version
   5 .\" 1.0 of the CDDL.
   6 .\"
   7 .\" A full copy of the text of the CDDL should have accompanied this
   8 .\" source.  A copy of the CDDL is also available via the Internet at
   9 .\" http://www.illumos.org/license/CDDL.
  10 .\"
  11 .\"
  12 .\" Copyright (c) 2012 by Delphix. All rights reserved.
  13 .\"
  14 .TH run 1 "23 Sep 2012"
  15 .SH NAME
  16 run \- find, execute, and log the results of tests
  17 .SH SYNOPSIS
  18 .LP
  19 .nf
  20 \fBrun\fR [\fB-dgq] [\fB-o\fR \fIoutputdir\fR] [\fB-pP\fR \fIscript\fR] [\fB-t\fR \fIseconds\fR] [\fB-uxX\fR \fIusername\fR]
  21     \fIpathname\fR ...
  22 .fi
  23
  24 .LP
  25 .nf
  26 \fBrun\fR \fB-w\fR \fIrunfile\fR [\fB-gq\fR] [\fB-o\fR \fIoutputdir\fR] [\fB-pP\fR \fIscript\fR] [\fB-t\fR \fIseconds\fR]
  27     [\fB-uxX\fR \fIusername\fR] \fIpathname\fR ...
  28 .fi
  29
  30 .LP
  31 .nf
  32 \fBrun\fR \fB-c\fR \fIrunfile\fR [\fB-dq\fR]
  33 .fi
  34
  35 .LP
  36 .nf
  37 \fBrun\fR [\fB-h\fR]
  38 .fi
  39
  40 .SH DESCRIPTION
  41 .sp
  42 .LP
  43 The \fBrun\fR command has three basic modes of operation. With neither the
  44 \fB-c\fR nor the \fB-w\fR option, \fBrun\fR processes the arguments provided on
  45 the command line, adding them to the list for this run. If a specified
  46 \fIpathname\fR is an executable file, it is added as a test. If a specified
  47 \fIpathname\fR is a directory, the behavior depends upon the \fB-g\fR option.
  48 If \fB-g\fR is specified, the directory is treated as a test group. See the
  49 section on "Test Groups" below. Without the \fB-g\fR option, \fBrun\fR simply
  50 descends into the directory looking for executable files. The tests are then
  51 executed, and the results are logged.
  52
  53 With the \fB-w\fR option, \fBrun\fR finds tests in the manner described above.
  54 Rather than executing the tests and logging the results, the test configuration
  55 is stored in a \fIrunfile\fR which can be used in future invocations, or edited
  56 to modify which tests are executed and which options are applied. Options
  57 included on the command line with \fB-w\fR become defaults in the
  58 \fIrunfile\fR.
  59
  60 With the \fB-c\fR option, \fBrun\fR parses a \fIrunfile\fR, which can specify a
  61 series of tests and test groups to be executed. The tests are then executed,
  62 and the results are logged.
  63 .sp
  64 .SS "Test Groups"
  65 .sp
  66 .LP
  67 A test group is comprised of a set of executable files, all of which exist in
  68 one directory. The options specified on the command line or in a \fIrunfile\fR
  69 apply to individual tests in the group. The exception is options pertaining to
  70 pre and post scripts, which act on all tests as a group. Rather than running
  71 before and after each test, these scripts are run only once each at the start
  72 and end of the test group.
  73 .SS "Test Execution"
  74 .sp
  75 .LP
  76 The specified tests run serially, and are typically assigned results according
  77 to exit values. Tests that exit zero and non-zero are marked "PASS" and "FAIL"
  78 respectively. When a pre script fails for a test group, only the post script is
  79 executed, and the remaining tests are marked "SKIPPED." Any test that exceeds
  80 its \fItimeout\fR is terminated, and marked "KILLED."
  81
  82 By default, tests are executed with the credentials of the \fBrun\fR script.
  83 Executing tests with other credentials is done via \fBsudo\fR(1m), which must
  84 be configured to allow execution without prompting for a password. Environment
  85 variables from the calling shell are available to individual tests. During test
  86 execution, the working directory is changed to \fIoutputdir\fR.
  87 .SS "Output Logging"
  88 .sp
  89 .LP
  90 By default, \fBrun\fR will print one line on standard output at the conclusion
  91 of each test indicating the test name, result and elapsed time. Additionally,
  92 for each invocation of \fBrun\fR, a directory is created using the ISO 8601
  93 date format. Within this directory is a file named \fIlog\fR containing all the
  94 test output with timestamps, and a directory for each test. Within the test
  95 directories, there is one file each for standard output, standard error and
  96 merged output. The default location for the \fIoutputdir\fR is
  97 \fI/var/tmp/test_results\fR.
  98 .SS "Runfiles"
  99 .sp
 100 .LP
 101 The \fIrunfile\fR is an ini style configuration file that describes a test run.
 102 The file has one section named "DEFAULT," which contains configuration option
 103 names and their values in "name = value" format. The values in this section
 104 apply to all the subsequent sections, unless they are also specified there, in
 105 which case the default is overridden. The remaining section names are the
 106 absolute pathnames of files and direcotries, describing tests and test groups
 107 respectively. The legal option names are:
 108 .sp
 109 .ne 2
 110 .na
 111 \fBoutputdir\fR = \fIpathname\fR
 112 .ad
 113 .sp .6
 114 .RS 4n
 115 The name of the directory that holds test logs.
 116 .RE
 117 .sp
 118 .ne 2
 119 .na
 120 \fBpre\fR = \fIscript\fR
 121 .ad
 122 .sp .6
 123 .RS 4n
 124 Run \fIscript\fR prior to the test or test group.
 125 .RE
 126 .sp
 127 .ne 2
 128 .na
 129 \fBpre_user\fR = \fIusername\fR
 130 .ad
 131 .sp .6
 132 .RS 4n
 133 Execute the pre script as \fIusername\fR.
 134 .RE
 135 .sp
 136 .ne 2
 137 .na
 138 \fBpost\fR = \fIscript\fR
 139 .ad
 140 .sp .6
 141 .RS 4n
 142 Run \fIscript\fR after the test or test group.
 143 .RE
 144 .sp
 145 .ne 2
 146 .na
 147 \fBpost_user\fR = \fIusername\fR
 148 .ad
 149 .sp .6
 150 .RS 4n
 151 Execute the post script as \fIusername\fR.
 152 .RE
 153 .sp
 154 .ne 2
 155 .na
 156 \fBquiet\fR = [\fITrue\fR|\fIFalse\fR]
 157 .ad
 158 .sp .6
 159 .RS 4n
 160 If set to True, only the results summary is printed to standard out.
 161 .RE
 162 .sp
 163 .ne 2
 164 .na
 165 \fBtests\fR = [\fI'filename'\fR [,...]]
 166 .ad
 167 .sp .6
 168 .RS 4n
 169 Specify a list of \fIfilenames\fR for this test group. Only the basename of the
 170 absolute path is required. This option is only valid for test groups, and each
 171 \fIfilename\fR must be single quoted.
 172 .RE
 173 .sp
 174 .ne 2
 175 .na
 176 \fBtimeout\fR = \fIn\fR
 177 .ad
 178 .sp .6
 179 .RS 4n
 180 A timeout value of \fIn\fR seconds.
 181 .RE
 182 .sp
 183 .ne 2
 184 .na
 185 \fBuser\fR = \fIusername\fR
 186 .ad
 187 .sp .6
 188 .RS 4n
 189 Execute the test or test group as \fIusername\fR.
 190 .RE
 191
 192 .SH OPTIONS
 193 .sp
 194 .LP
 195 The following options are available for the \fBrun\fR command.
 196 .sp
 197 .ne 2
 198 .na
 199 \fB-c\fR \fIrunfile\fR
 200 .ad
 201 .RS 6n
 202 Specify a \fIrunfile\fR to be consumed by the run command.
 203 .RE
 204
 205 .ne 2
 206 .na
 207 \fB-d\fR
 208 .ad
 209 .RS 6n
 210 Dry run mode. Execute no tests, but print a description of each test that would
 211 have been run.
 212 .RE
 213
 214 .ne 2
 215 .na
 216 \fB-g\fR
 217 .ad
 218 .RS 6n
 219 Create test groups from any directories found while searching for tests.
 220 .RE
 221
 222 .ne 2
 223 .na
 224 \fB-o\fR \fIoutputdir\fR
 225 .ad
 226 .RS 6n
 227 Specify the directory in which to write test results.
 228 .RE
 229
 230 .ne 2
 231 .na
 232 \fB-p\fR \fIscript\fR
 233 .ad
 234 .RS 6n
 235 Run \fIscript\fR prior to any test or test group.
 236 .RE
 237
 238 .ne 2
 239 .na
 240 \fB-P\fR \fIscript\fR
 241 .ad
 242 .RS 6n
 243 Run \fIscript\fR after any test or test group.
 244 .RE
 245
 246 .ne 2
 247 .na
 248 \fB-q\fR
 249 .ad
 250 .RS 6n
 251 Print only the results sumary to the standard output.
 252 .RE
 253
 254 .ne 2
 255 .na
 256 \fB-t\fR \fIn\fR
 257 .ad
 258 .RS 6n
 259 Specify a timeout value of \fIn\fR seconds per test.
 260 .RE
 261
 262 .ne 2
 263 .na
 264 \fB-u\fR \fIusername\fR
 265 .ad
 266 .RS 6n
 267 Execute tests or test groups as \fIusername\fR.
 268 .RE
 269
 270 .ne 2
 271 .na
 272 \fB-w\fR \fIrunfile\fR
 273 .ad
 274 .RS 6n
 275 Specify the name of the \fIrunfile\fR to create.
 276 .RE
 277
 278 .ne 2
 279 .na
 280 \fB-x\fR \fIusername\fR
 281 .ad
 282 .RS 6n
 283 Execute the pre script as \fIusername\fR.
 284 .RE
 285
 286 .ne 2
 287 .na
 288 \fB-X\fR \fIusername\fR
 289 .ad
 290 .RS 6n
 291 Execute the post script as \fIusername\fR.
 292 .RE
 293
 294 .SH EXAMPLES
 295 .LP
 296 \fBExample 1\fR Running ad-hoc tests.
 297 .sp
 298 .LP
 299 This example demonstrates the simplest invocation of \fBrun\fR.
 300
 301 .sp
 302 .in +2
 303 .nf
 304 % \fBrun my-tests\fR
 305 Test: /home/jkennedy/my-tests/test-01                    [00:02] [PASS]
 306 Test: /home/jkennedy/my-tests/test-02                    [00:04] [PASS]
 307 Test: /home/jkennedy/my-tests/test-03                    [00:01] [PASS]
 308
 309 Results Summary
 310 PASS       3
 311
 312 Running Time:   00:00:07
 313 Percent passed: 100.0%
 314 Log directory:  /var/tmp/test_results/20120923T180654
 315 .fi
 316 .in -2
 317
 318 .LP
 319 \fBExample 2\fR Creating a \fIrunfile\fR for future use.
 320 .sp
 321 .LP
 322 This example demonstrates creating a \fIrunfile\fR with non default options.
 323
 324 .sp
 325 .in +2
 326 .nf
 327 % \fBrun -p setup -x root -g -w new-tests.run new-tests\fR
 328 % \fBcat new-tests.run\fR
 329 [DEFAULT]
 330 pre = setup
 331 post_user =
 332 quiet = False
 333 user =
 334 timeout = 60
 335 post =
 336 pre_user = root
 337 outputdir = /var/tmp/test_results
 338
 339 [/home/jkennedy/new-tests]
 340 tests = ['test-01', 'test-02', 'test-03']
 341 .fi
 342 .in -2
 343
 344 .SH EXIT STATUS
 345 .sp
 346 .LP
 347 The following exit values are returned:
 348 .sp
 349 .ne 2
 350 .na
 351 \fB\fB0\fR\fR
 352 .ad
 353 .sp .6
 354 .RS 4n
 355 Successful completion.
 356 .RE
 357 .sp
 358 .ne 2
 359 .na
 360 \fB\fB1\fR\fR
 361 .ad
 362 .sp .6
 363 .RS 4n
 364 An error occurred.
 365 .RE
 366
 367 .SH SEE ALSO
 368 .sp
 369 .LP
 370 \fBsudo\fR(1m)