]>
Commit | Line | Data |
---|---|---|
4eb99560 FZ |
1 | =============== |
2 | Testing in QEMU | |
3 | =============== | |
4 | ||
5 | This document describes the testing infrastructure in QEMU. | |
6 | ||
7 | Testing with "make check" | |
8 | ========================= | |
9 | ||
10 | The "make check" testing family includes most of the C based tests in QEMU. For | |
11 | a quick help, run ``make check-help`` from the source tree. | |
12 | ||
13 | The usual way to run these tests is: | |
14 | ||
15 | .. code:: | |
16 | ||
17 | make check | |
18 | ||
316082b1 TH |
19 | which includes QAPI schema tests, unit tests, QTests and some iotests. |
20 | Different sub-types of "make check" tests will be explained below. | |
4eb99560 FZ |
21 | |
22 | Before running tests, it is best to build QEMU programs first. Some tests | |
23 | expect the executables to exist and will fail with obscure messages if they | |
24 | cannot find them. | |
25 | ||
26 | Unit tests | |
27 | ---------- | |
28 | ||
29 | Unit tests, which can be invoked with ``make check-unit``, are simple C tests | |
30 | that typically link to individual QEMU object files and exercise them by | |
31 | calling exported functions. | |
32 | ||
33 | If you are writing new code in QEMU, consider adding a unit test, especially | |
34 | for utility modules that are relatively stateless or have few dependencies. To | |
35 | add a new unit test: | |
36 | ||
37 | 1. Create a new source file. For example, ``tests/foo-test.c``. | |
38 | ||
39 | 2. Write the test. Normally you would include the header file which exports | |
40 | the module API, then verify the interface behaves as expected from your | |
41 | test. The test code should be organized with the glib testing framework. | |
42 | Copying and modifying an existing test is usually a good idea. | |
43 | ||
44 | 3. Add the test to ``tests/Makefile.include``. First, name the unit test | |
45 | program and add it to ``$(check-unit-y)``; then add a rule to build the | |
26830e93 | 46 | executable. For example: |
4eb99560 FZ |
47 | |
48 | .. code:: | |
49 | ||
50 | check-unit-y += tests/foo-test$(EXESUF) | |
51 | tests/foo-test$(EXESUF): tests/foo-test.o $(test-util-obj-y) | |
52 | ... | |
4eb99560 FZ |
53 | |
54 | Since unit tests don't require environment variables, the simplest way to debug | |
55 | a unit test failure is often directly invoking it or even running it under | |
56 | ``gdb``. However there can still be differences in behavior between ``make`` | |
57 | invocations and your manual run, due to ``$MALLOC_PERTURB_`` environment | |
58 | variable (which affects memory reclamation and catches invalid pointers better) | |
59 | and gtester options. If necessary, you can run | |
60 | ||
61 | .. code:: | |
92970812 | 62 | |
4eb99560 FZ |
63 | make check-unit V=1 |
64 | ||
65 | and copy the actual command line which executes the unit test, then run | |
66 | it from the command line. | |
67 | ||
68 | QTest | |
69 | ----- | |
70 | ||
71 | QTest is a device emulation testing framework. It can be very useful to test | |
72 | device models; it could also control certain aspects of QEMU (such as virtual | |
73 | clock stepping), with a special purpose "qtest" protocol. Refer to the | |
74 | documentation in ``qtest.c`` for more details of the protocol. | |
75 | ||
76 | QTest cases can be executed with | |
77 | ||
78 | .. code:: | |
79 | ||
80 | make check-qtest | |
81 | ||
316082b1 TH |
82 | The QTest library is implemented by ``tests/qtest/libqtest.c`` and the API is |
83 | defined in ``tests/qtest/libqtest.h``. | |
4eb99560 FZ |
84 | |
85 | Consider adding a new QTest case when you are introducing a new virtual | |
86 | hardware, or extending one if you are adding functionalities to an existing | |
87 | virtual device. | |
88 | ||
89 | On top of libqtest, a higher level library, ``libqos``, was created to | |
90 | encapsulate common tasks of device drivers, such as memory management and | |
91 | communicating with system buses or devices. Many virtual device tests use | |
92 | libqos instead of directly calling into libqtest. | |
93 | ||
94 | Steps to add a new QTest case are: | |
95 | ||
96 | 1. Create a new source file for the test. (More than one file can be added as | |
316082b1 | 97 | necessary.) For example, ``tests/qtest/foo-test.c``. |
4eb99560 FZ |
98 | |
99 | 2. Write the test code with the glib and libqtest/libqos API. See also existing | |
100 | tests and the library headers for reference. | |
101 | ||
316082b1 TH |
102 | 3. Register the new test in ``tests/qtest/Makefile.include``. Add the test |
103 | executable name to an appropriate ``check-qtest-*-y`` variable. For example: | |
4eb99560 | 104 | |
316082b1 | 105 | ``check-qtest-generic-y = tests/qtest/foo-test$(EXESUF)`` |
4eb99560 FZ |
106 | |
107 | 4. Add object dependencies of the executable in the Makefile, including the | |
108 | test source file(s) and other interesting objects. For example: | |
109 | ||
316082b1 | 110 | ``tests/qtest/foo-test$(EXESUF): tests/qtest/foo-test.o $(libqos-obj-y)`` |
4eb99560 FZ |
111 | |
112 | Debugging a QTest failure is slightly harder than the unit test because the | |
113 | tests look up QEMU program names in the environment variables, such as | |
114 | ``QTEST_QEMU_BINARY`` and ``QTEST_QEMU_IMG``, and also because it is not easy | |
115 | to attach gdb to the QEMU process spawned from the test. But manual invoking | |
116 | and using gdb on the test is still simple to do: find out the actual command | |
117 | from the output of | |
118 | ||
119 | .. code:: | |
92970812 | 120 | |
4eb99560 FZ |
121 | make check-qtest V=1 |
122 | ||
123 | which you can run manually. | |
124 | ||
125 | QAPI schema tests | |
126 | ----------------- | |
127 | ||
128 | The QAPI schema tests validate the QAPI parser used by QMP, by feeding | |
129 | predefined input to the parser and comparing the result with the reference | |
130 | output. | |
131 | ||
132 | The input/output data is managed under the ``tests/qapi-schema`` directory. | |
133 | Each test case includes four files that have a common base name: | |
134 | ||
135 | * ``${casename}.json`` - the file contains the JSON input for feeding the | |
136 | parser | |
137 | * ``${casename}.out`` - the file contains the expected stdout from the parser | |
138 | * ``${casename}.err`` - the file contains the expected stderr from the parser | |
139 | * ``${casename}.exit`` - the expected error code | |
140 | ||
141 | Consider adding a new QAPI schema test when you are making a change on the QAPI | |
142 | parser (either fixing a bug or extending/modifying the syntax). To do this: | |
143 | ||
144 | 1. Add four files for the new case as explained above. For example: | |
145 | ||
146 | ``$EDITOR tests/qapi-schema/foo.{json,out,err,exit}``. | |
147 | ||
148 | 2. Add the new test in ``tests/Makefile.include``. For example: | |
149 | ||
150 | ``qapi-schema += foo.json`` | |
151 | ||
152 | check-block | |
153 | ----------- | |
154 | ||
316082b1 TH |
155 | ``make check-block`` runs a subset of the block layer iotests (the tests that |
156 | are in the "auto" group in ``tests/qemu-iotests/group``). | |
157 | See the "QEMU iotests" section below for more information. | |
4eb99560 FZ |
158 | |
159 | GCC gcov support | |
160 | ---------------- | |
161 | ||
31d2dda3 AB |
162 | ``gcov`` is a GCC tool to analyze the testing coverage by |
163 | instrumenting the tested code. To use it, configure QEMU with | |
164 | ``--enable-gcov`` option and build. Then run ``make check`` as usual. | |
990e6a27 AB |
165 | |
166 | If you want to gather coverage information on a single test the ``make | |
bf0e56a3 | 167 | clean-gcda`` target can be used to delete any existing coverage |
990e6a27 AB |
168 | information before running a single test. |
169 | ||
fe8bf5f6 | 170 | You can generate a HTML coverage report by executing ``make |
bf0e56a3 MAL |
171 | coverage-html`` which will create |
172 | ``meson-logs/coveragereport/index.html``. | |
fe8bf5f6 AB |
173 | |
174 | Further analysis can be conducted by running the ``gcov`` command | |
175 | directly on the various .gcda output files. Please read the ``gcov`` | |
176 | documentation for more information. | |
4eb99560 FZ |
177 | |
178 | QEMU iotests | |
179 | ============ | |
180 | ||
181 | QEMU iotests, under the directory ``tests/qemu-iotests``, is the testing | |
182 | framework widely used to test block layer related features. It is higher level | |
183 | than "make check" tests and 99% of the code is written in bash or Python | |
184 | scripts. The testing success criteria is golden output comparison, and the | |
185 | test files are named with numbers. | |
186 | ||
187 | To run iotests, make sure QEMU is built successfully, then switch to the | |
188 | ``tests/qemu-iotests`` directory under the build directory, and run ``./check`` | |
189 | with desired arguments from there. | |
190 | ||
191 | By default, "raw" format and "file" protocol is used; all tests will be | |
192 | executed, except the unsupported ones. You can override the format and protocol | |
193 | with arguments: | |
194 | ||
195 | .. code:: | |
196 | ||
197 | # test with qcow2 format | |
198 | ./check -qcow2 | |
199 | # or test a different protocol | |
200 | ./check -nbd | |
201 | ||
202 | It's also possible to list test numbers explicitly: | |
203 | ||
204 | .. code:: | |
205 | ||
206 | # run selected cases with qcow2 format | |
207 | ./check -qcow2 001 030 153 | |
208 | ||
209 | Cache mode can be selected with the "-c" option, which may help reveal bugs | |
210 | that are specific to certain cache mode. | |
211 | ||
212 | More options are supported by the ``./check`` script, run ``./check -h`` for | |
213 | help. | |
214 | ||
215 | Writing a new test case | |
216 | ----------------------- | |
217 | ||
218 | Consider writing a tests case when you are making any changes to the block | |
219 | layer. An iotest case is usually the choice for that. There are already many | |
220 | test cases, so it is possible that extending one of them may achieve the goal | |
221 | and save the boilerplate to create one. (Unfortunately, there isn't a 100% | |
222 | reliable way to find a related one out of hundreds of tests. One approach is | |
223 | using ``git grep``.) | |
224 | ||
225 | Usually an iotest case consists of two files. One is an executable that | |
226 | produces output to stdout and stderr, the other is the expected reference | |
227 | output. They are given the same number in file names. E.g. Test script ``055`` | |
228 | and reference output ``055.out``. | |
229 | ||
230 | In rare cases, when outputs differ between cache mode ``none`` and others, a | |
231 | ``.out.nocache`` file is added. In other cases, when outputs differ between | |
232 | image formats, more than one ``.out`` files are created ending with the | |
233 | respective format names, e.g. ``178.out.qcow2`` and ``178.out.raw``. | |
234 | ||
235 | There isn't a hard rule about how to write a test script, but a new test is | |
236 | usually a (copy and) modification of an existing case. There are a few | |
237 | commonly used ways to create a test: | |
238 | ||
239 | * A Bash script. It will make use of several environmental variables related | |
240 | to the testing procedure, and could source a group of ``common.*`` libraries | |
241 | for some common helper routines. | |
242 | ||
243 | * A Python unittest script. Import ``iotests`` and create a subclass of | |
244 | ``iotests.QMPTestCase``, then call ``iotests.main`` method. The downside of | |
245 | this approach is that the output is too scarce, and the script is considered | |
246 | harder to debug. | |
247 | ||
248 | * A simple Python script without using unittest module. This could also import | |
249 | ``iotests`` for launching QEMU and utilities etc, but it doesn't inherit | |
250 | from ``iotests.QMPTestCase`` therefore doesn't use the Python unittest | |
251 | execution. This is a combination of 1 and 2. | |
252 | ||
253 | Pick the language per your preference since both Bash and Python have | |
254 | comparable library support for invoking and interacting with QEMU programs. If | |
255 | you opt for Python, it is strongly recommended to write Python 3 compatible | |
256 | code. | |
257 | ||
f4a1b653 FZ |
258 | Both Python and Bash frameworks in iotests provide helpers to manage test |
259 | images. They can be used to create and clean up images under the test | |
260 | directory. If no I/O or any protocol specific feature is needed, it is often | |
261 | more convenient to use the pseudo block driver, ``null-co://``, as the test | |
262 | image, which doesn't require image creation or cleaning up. Avoid system-wide | |
263 | devices or files whenever possible, such as ``/dev/null`` or ``/dev/zero``. | |
264 | Otherwise, image locking implications have to be considered. For example, | |
265 | another application on the host may have locked the file, possibly leading to a | |
266 | test failure. If using such devices are explicitly desired, consider adding | |
267 | ``locking=off`` option to disable image locking. | |
268 | ||
f8ed349e AB |
269 | .. _docker-ref: |
270 | ||
4eb99560 FZ |
271 | Docker based tests |
272 | ================== | |
273 | ||
274 | Introduction | |
275 | ------------ | |
276 | ||
277 | The Docker testing framework in QEMU utilizes public Docker images to build and | |
278 | test QEMU in predefined and widely accessible Linux environments. This makes | |
279 | it possible to expand the test coverage across distros, toolchain flavors and | |
280 | library versions. | |
281 | ||
282 | Prerequisites | |
283 | ------------- | |
284 | ||
285 | Install "docker" with the system package manager and start the Docker service | |
286 | on your development machine, then make sure you have the privilege to run | |
287 | Docker commands. Typically it means setting up passwordless ``sudo docker`` | |
288 | command or login as root. For example: | |
289 | ||
290 | .. code:: | |
291 | ||
292 | $ sudo yum install docker | |
293 | $ # or `apt-get install docker` for Ubuntu, etc. | |
294 | $ sudo systemctl start docker | |
295 | $ sudo docker ps | |
296 | ||
297 | The last command should print an empty table, to verify the system is ready. | |
298 | ||
299 | An alternative method to set up permissions is by adding the current user to | |
300 | "docker" group and making the docker daemon socket file (by default | |
301 | ``/var/run/docker.sock``) accessible to the group: | |
302 | ||
303 | .. code:: | |
304 | ||
305 | $ sudo groupadd docker | |
29c33cc1 | 306 | $ sudo usermod $USER -a -G docker |
4eb99560 FZ |
307 | $ sudo chown :docker /var/run/docker.sock |
308 | ||
309 | Note that any one of above configurations makes it possible for the user to | |
310 | exploit the whole host with Docker bind mounting or other privileged | |
311 | operations. So only do it on development machines. | |
312 | ||
313 | Quickstart | |
314 | ---------- | |
315 | ||
316 | From source tree, type ``make docker`` to see the help. Testing can be started | |
317 | without configuring or building QEMU (``configure`` and ``make`` are done in | |
318 | the container, with parameters defined by the make target): | |
319 | ||
320 | .. code:: | |
321 | ||
322 | make docker-test-build@min-glib | |
323 | ||
324 | This will create a container instance using the ``min-glib`` image (the image | |
325 | is downloaded and initialized automatically), in which the ``test-build`` job | |
326 | is executed. | |
327 | ||
328 | Images | |
329 | ------ | |
330 | ||
331 | Along with many other images, the ``min-glib`` image is defined in a Dockerfile | |
2cd925da | 332 | in ``tests/docker/dockerfiles/``, called ``min-glib.docker``. ``make docker`` |
4eb99560 FZ |
333 | command will list all the available images. |
334 | ||
335 | To add a new image, simply create a new ``.docker`` file under the | |
336 | ``tests/docker/dockerfiles/`` directory. | |
337 | ||
338 | A ``.pre`` script can be added beside the ``.docker`` file, which will be | |
339 | executed before building the image under the build context directory. This is | |
340 | mainly used to do necessary host side setup. One such setup is ``binfmt_misc``, | |
341 | for example, to make qemu-user powered cross build containers work. | |
342 | ||
343 | Tests | |
344 | ----- | |
345 | ||
346 | Different tests are added to cover various configurations to build and test | |
347 | QEMU. Docker tests are the executables under ``tests/docker`` named | |
348 | ``test-*``. They are typically shell scripts and are built on top of a shell | |
349 | library, ``tests/docker/common.rc``, which provides helpers to find the QEMU | |
350 | source and build it. | |
351 | ||
352 | The full list of tests is printed in the ``make docker`` help. | |
353 | ||
354 | Tools | |
355 | ----- | |
356 | ||
357 | There are executables that are created to run in a specific Docker environment. | |
358 | This makes it easy to write scripts that have heavy or special dependencies, | |
359 | but are still very easy to use. | |
360 | ||
361 | Currently the only tool is ``travis``, which mimics the Travis-CI tests in a | |
362 | container. It runs in the ``travis`` image: | |
363 | ||
364 | .. code:: | |
365 | ||
366 | make docker-travis@travis | |
367 | ||
368 | Debugging a Docker test failure | |
369 | ------------------------------- | |
370 | ||
371 | When CI tasks, maintainers or yourself report a Docker test failure, follow the | |
372 | below steps to debug it: | |
373 | ||
374 | 1. Locally reproduce the failure with the reported command line. E.g. run | |
375 | ``make docker-test-mingw@fedora J=8``. | |
376 | 2. Add "V=1" to the command line, try again, to see the verbose output. | |
377 | 3. Further add "DEBUG=1" to the command line. This will pause in a shell prompt | |
378 | in the container right before testing starts. You could either manually | |
379 | build QEMU and run tests from there, or press Ctrl-D to let the Docker | |
380 | testing continue. | |
381 | 4. If you press Ctrl-D, the same building and testing procedure will begin, and | |
382 | will hopefully run into the error again. After that, you will be dropped to | |
383 | the prompt for debug. | |
384 | ||
385 | Options | |
386 | ------- | |
387 | ||
388 | Various options can be used to affect how Docker tests are done. The full | |
389 | list is in the ``make docker`` help text. The frequently used ones are: | |
390 | ||
391 | * ``V=1``: the same as in top level ``make``. It will be propagated to the | |
392 | container and enable verbose output. | |
393 | * ``J=$N``: the number of parallel tasks in make commands in the container, | |
394 | similar to the ``-j $N`` option in top level ``make``. (The ``-j`` option in | |
395 | top level ``make`` will not be propagated into the container.) | |
396 | * ``DEBUG=1``: enables debug. See the previous "Debugging a Docker test | |
397 | failure" section. | |
398 | ||
3b6882bd RF |
399 | Thread Sanitizer |
400 | ================ | |
401 | ||
402 | Thread Sanitizer (TSan) is a tool which can detect data races. QEMU supports | |
403 | building and testing with this tool. | |
404 | ||
405 | For more information on TSan: | |
406 | ||
407 | https://github.com/google/sanitizers/wiki/ThreadSanitizerCppManual | |
408 | ||
409 | Thread Sanitizer in Docker | |
410 | --------------------------- | |
411 | TSan is currently supported in the ubuntu2004 docker. | |
412 | ||
413 | The test-tsan test will build using TSan and then run make check. | |
414 | ||
415 | .. code:: | |
416 | ||
417 | make docker-test-tsan@ubuntu2004 | |
418 | ||
419 | TSan warnings under docker are placed in files located at build/tsan/. | |
420 | ||
421 | We recommend using DEBUG=1 to allow launching the test from inside the docker, | |
422 | and to allow review of the warnings generated by TSan. | |
423 | ||
424 | Building and Testing with TSan | |
425 | ------------------------------ | |
426 | ||
427 | It is possible to build and test with TSan, with a few additional steps. | |
428 | These steps are normally done automatically in the docker. | |
429 | ||
430 | There is a one time patch needed in clang-9 or clang-10 at this time: | |
431 | ||
432 | .. code:: | |
433 | ||
434 | sed -i 's/^const/static const/g' \ | |
435 | /usr/lib/llvm-10/lib/clang/10.0.0/include/sanitizer/tsan_interface.h | |
436 | ||
437 | To configure the build for TSan: | |
438 | ||
439 | .. code:: | |
440 | ||
441 | ../configure --enable-tsan --cc=clang-10 --cxx=clang++-10 \ | |
442 | --disable-werror --extra-cflags="-O0" | |
443 | ||
444 | The runtime behavior of TSAN is controlled by the TSAN_OPTIONS environment | |
445 | variable. | |
446 | ||
447 | More information on the TSAN_OPTIONS can be found here: | |
448 | ||
449 | https://github.com/google/sanitizers/wiki/ThreadSanitizerFlags | |
450 | ||
451 | For example: | |
452 | ||
453 | .. code:: | |
454 | ||
455 | export TSAN_OPTIONS=suppressions=<path to qemu>/tests/tsan/suppressions.tsan \ | |
456 | detect_deadlocks=false history_size=7 exitcode=0 \ | |
457 | log_path=<build path>/tsan/tsan_warning | |
458 | ||
459 | The above exitcode=0 has TSan continue without error if any warnings are found. | |
460 | This allows for running the test and then checking the warnings afterwards. | |
461 | If you want TSan to stop and exit with error on warnings, use exitcode=66. | |
462 | ||
463 | TSan Suppressions | |
464 | ----------------- | |
465 | Keep in mind that for any data race warning, although there might be a data race | |
466 | detected by TSan, there might be no actual bug here. TSan provides several | |
467 | different mechanisms for suppressing warnings. In general it is recommended | |
468 | to fix the code if possible to eliminate the data race rather than suppress | |
469 | the warning. | |
470 | ||
471 | A few important files for suppressing warnings are: | |
472 | ||
473 | tests/tsan/suppressions.tsan - Has TSan warnings we wish to suppress at runtime. | |
76ca4b58 | 474 | The comment on each suppression will typically indicate why we are |
3b6882bd RF |
475 | suppressing it. More information on the file format can be found here: |
476 | ||
477 | https://github.com/google/sanitizers/wiki/ThreadSanitizerSuppressions | |
478 | ||
479 | tests/tsan/blacklist.tsan - Has TSan warnings we wish to disable | |
480 | at compile time for test or debug. | |
481 | Add flags to configure to enable: | |
482 | ||
483 | "--extra-cflags=-fsanitize-blacklist=<src path>/tests/tsan/blacklist.tsan" | |
484 | ||
485 | More information on the file format can be found here under "Blacklist Format": | |
486 | ||
487 | https://github.com/google/sanitizers/wiki/ThreadSanitizerFlags | |
488 | ||
489 | TSan Annotations | |
490 | ---------------- | |
491 | include/qemu/tsan.h defines annotations. See this file for more descriptions | |
492 | of the annotations themselves. Annotations can be used to suppress | |
493 | TSan warnings or give TSan more information so that it can detect proper | |
494 | relationships between accesses of data. | |
495 | ||
496 | Annotation examples can be found here: | |
497 | ||
498 | https://github.com/llvm/llvm-project/tree/master/compiler-rt/test/tsan/ | |
499 | ||
500 | Good files to start with are: annotate_happens_before.cpp and ignore_race.cpp | |
501 | ||
502 | The full set of annotations can be found here: | |
503 | ||
504 | https://github.com/llvm/llvm-project/blob/master/compiler-rt/lib/tsan/rtl/tsan_interface_ann.cpp | |
505 | ||
4eb99560 FZ |
506 | VM testing |
507 | ========== | |
508 | ||
509 | This test suite contains scripts that bootstrap various guest images that have | |
510 | necessary packages to build QEMU. The basic usage is documented in ``Makefile`` | |
4f2f6276 | 511 | help which is displayed with ``make vm-help``. |
4eb99560 FZ |
512 | |
513 | Quickstart | |
514 | ---------- | |
515 | ||
4f2f6276 | 516 | Run ``make vm-help`` to list available make targets. Invoke a specific make |
4eb99560 FZ |
517 | command to run build test in an image. For example, ``make vm-build-freebsd`` |
518 | will build the source tree in the FreeBSD image. The command can be executed | |
519 | from either the source tree or the build dir; if the former, ``./configure`` is | |
520 | not needed. The command will then generate the test image in ``./tests/vm/`` | |
521 | under the working directory. | |
522 | ||
523 | Note: images created by the scripts accept a well-known RSA key pair for SSH | |
524 | access, so they SHOULD NOT be exposed to external interfaces if you are | |
525 | concerned about attackers taking control of the guest and potentially | |
526 | exploiting a QEMU security bug to compromise the host. | |
527 | ||
1e48931c WSM |
528 | QEMU binaries |
529 | ------------- | |
4eb99560 FZ |
530 | |
531 | By default, qemu-system-x86_64 is searched in $PATH to run the guest. If there | |
532 | isn't one, or if it is older than 2.10, the test won't work. In this case, | |
533 | provide the QEMU binary in env var: ``QEMU=/path/to/qemu-2.10+``. | |
534 | ||
1e48931c WSM |
535 | Likewise the path to qemu-img can be set in QEMU_IMG environment variable. |
536 | ||
4eb99560 FZ |
537 | Make jobs |
538 | --------- | |
539 | ||
540 | The ``-j$X`` option in the make command line is not propagated into the VM, | |
541 | specify ``J=$X`` to control the make jobs in the guest. | |
542 | ||
543 | Debugging | |
544 | --------- | |
545 | ||
546 | Add ``DEBUG=1`` and/or ``V=1`` to the make command to allow interactive | |
547 | debugging and verbose output. If this is not enough, see the next section. | |
41e3340a | 548 | ``V=1`` will be propagated down into the make jobs in the guest. |
4eb99560 FZ |
549 | |
550 | Manual invocation | |
551 | ----------------- | |
552 | ||
553 | Each guest script is an executable script with the same command line options. | |
554 | For example to work with the netbsd guest, use ``$QEMU_SRC/tests/vm/netbsd``: | |
555 | ||
556 | .. code:: | |
557 | ||
558 | $ cd $QEMU_SRC/tests/vm | |
559 | ||
560 | # To bootstrap the image | |
561 | $ ./netbsd --build-image --image /var/tmp/netbsd.img | |
562 | <...> | |
563 | ||
564 | # To run an arbitrary command in guest (the output will not be echoed unless | |
565 | # --debug is added) | |
566 | $ ./netbsd --debug --image /var/tmp/netbsd.img uname -a | |
567 | ||
568 | # To build QEMU in guest | |
569 | $ ./netbsd --debug --image /var/tmp/netbsd.img --build-qemu $QEMU_SRC | |
570 | ||
571 | # To get to an interactive shell | |
572 | $ ./netbsd --interactive --image /var/tmp/netbsd.img sh | |
573 | ||
574 | Adding new guests | |
575 | ----------------- | |
576 | ||
577 | Please look at existing guest scripts for how to add new guests. | |
578 | ||
579 | Most importantly, create a subclass of BaseVM and implement ``build_image()`` | |
580 | method and define ``BUILD_SCRIPT``, then finally call ``basevm.main()`` from | |
581 | the script's ``main()``. | |
582 | ||
583 | * Usually in ``build_image()``, a template image is downloaded from a | |
584 | predefined URL. ``BaseVM._download_with_cache()`` takes care of the cache and | |
585 | the checksum, so consider using it. | |
586 | ||
587 | * Once the image is downloaded, users, SSH server and QEMU build deps should | |
588 | be set up: | |
589 | ||
590 | - Root password set to ``BaseVM.ROOT_PASS`` | |
591 | - User ``BaseVM.GUEST_USER`` is created, and password set to | |
592 | ``BaseVM.GUEST_PASS`` | |
593 | - SSH service is enabled and started on boot, | |
594 | ``$QEMU_SRC/tests/keys/id_rsa.pub`` is added to ssh's ``authorized_keys`` | |
595 | file of both root and the normal user | |
596 | - DHCP client service is enabled and started on boot, so that it can | |
597 | automatically configure the virtio-net-pci NIC and communicate with QEMU | |
598 | user net (10.0.2.2) | |
599 | - Necessary packages are installed to untar the source tarball and build | |
600 | QEMU | |
601 | ||
602 | * Write a proper ``BUILD_SCRIPT`` template, which should be a shell script that | |
603 | untars a raw virtio-blk block device, which is the tarball data blob of the | |
604 | QEMU source tree, then configure/build it. Running "make check" is also | |
605 | recommended. | |
606 | ||
607 | Image fuzzer testing | |
608 | ==================== | |
609 | ||
610 | An image fuzzer was added to exercise format drivers. Currently only qcow2 is | |
611 | supported. To start the fuzzer, run | |
612 | ||
613 | .. code:: | |
614 | ||
615 | tests/image-fuzzer/runner.py -c '[["qemu-img", "info", "$test_img"]]' /tmp/test qcow2 | |
616 | ||
617 | Alternatively, some command different from "qemu-img info" can be tested, by | |
618 | changing the ``-c`` option. | |
c3d7e8c9 CR |
619 | |
620 | Acceptance tests using the Avocado Framework | |
621 | ============================================ | |
622 | ||
623 | The ``tests/acceptance`` directory hosts functional tests, also known | |
624 | as acceptance level tests. They're usually higher level tests, and | |
625 | may interact with external resources and with various guest operating | |
626 | systems. | |
627 | ||
628 | These tests are written using the Avocado Testing Framework (which must | |
629 | be installed separately) in conjunction with a the ``avocado_qemu.Test`` | |
630 | class, implemented at ``tests/acceptance/avocado_qemu``. | |
631 | ||
632 | Tests based on ``avocado_qemu.Test`` can easily: | |
633 | ||
634 | * Customize the command line arguments given to the convenience | |
635 | ``self.vm`` attribute (a QEMUMachine instance) | |
636 | ||
637 | * Interact with the QEMU monitor, send QMP commands and check | |
638 | their results | |
639 | ||
640 | * Interact with the guest OS, using the convenience console device | |
641 | (which may be useful to assert the effectiveness and correctness of | |
642 | command line arguments or QMP commands) | |
643 | ||
644 | * Interact with external data files that accompany the test itself | |
645 | (see ``self.get_data()``) | |
646 | ||
647 | * Download (and cache) remote data files, such as firmware and kernel | |
648 | images | |
649 | ||
650 | * Have access to a library of guest OS images (by means of the | |
651 | ``avocado.utils.vmimage`` library) | |
652 | ||
653 | * Make use of various other test related utilities available at the | |
654 | test class itself and at the utility library: | |
655 | ||
656 | - http://avocado-framework.readthedocs.io/en/latest/api/test/avocado.html#avocado.Test | |
657 | - http://avocado-framework.readthedocs.io/en/latest/api/utils/avocado.utils.html | |
658 | ||
a56931ee CR |
659 | Running tests |
660 | ------------- | |
661 | ||
662 | You can run the acceptance tests simply by executing: | |
663 | ||
664 | .. code:: | |
665 | ||
666 | make check-acceptance | |
667 | ||
668 | This involves the automatic creation of Python virtual environment | |
669 | within the build tree (at ``tests/venv``) which will have all the | |
670 | right dependencies, and will save tests results also within the | |
671 | build tree (at ``tests/results``). | |
c3d7e8c9 | 672 | |
a56931ee CR |
673 | Note: the build environment must be using a Python 3 stack, and have |
674 | the ``venv`` and ``pip`` packages installed. If necessary, make sure | |
675 | ``configure`` is called with ``--python=`` and that those modules are | |
676 | available. On Debian and Ubuntu based systems, depending on the | |
677 | specific version, they may be on packages named ``python3-venv`` and | |
678 | ``python3-pip``. | |
679 | ||
680 | The scripts installed inside the virtual environment may be used | |
681 | without an "activation". For instance, the Avocado test runner | |
682 | may be invoked by running: | |
683 | ||
684 | .. code:: | |
685 | ||
686 | tests/venv/bin/avocado run $OPTION1 $OPTION2 tests/acceptance/ | |
687 | ||
688 | Manual Installation | |
689 | ------------------- | |
690 | ||
691 | To manually install Avocado and its dependencies, run: | |
c3d7e8c9 CR |
692 | |
693 | .. code:: | |
694 | ||
695 | pip install --user avocado-framework | |
696 | ||
697 | Alternatively, follow the instructions on this link: | |
698 | ||
699 | http://avocado-framework.readthedocs.io/en/latest/GetStartedGuide.html#installing-avocado | |
700 | ||
701 | Overview | |
702 | -------- | |
703 | ||
805fac52 CR |
704 | The ``tests/acceptance/avocado_qemu`` directory provides the |
705 | ``avocado_qemu`` Python module, containing the ``avocado_qemu.Test`` | |
706 | class. Here's a simple usage example: | |
c3d7e8c9 CR |
707 | |
708 | .. code:: | |
709 | ||
710 | from avocado_qemu import Test | |
711 | ||
712 | ||
713 | class Version(Test): | |
714 | """ | |
c3d7e8c9 CR |
715 | :avocado: tags=quick |
716 | """ | |
717 | def test_qmp_human_info_version(self): | |
718 | self.vm.launch() | |
719 | res = self.vm.command('human-monitor-command', | |
720 | command_line='info version') | |
721 | self.assertRegexpMatches(res, r'^(\d+\.\d+\.\d)') | |
722 | ||
723 | To execute your test, run: | |
724 | ||
725 | .. code:: | |
726 | ||
727 | avocado run version.py | |
728 | ||
729 | Tests may be classified according to a convention by using docstring | |
730 | directives such as ``:avocado: tags=TAG1,TAG2``. To run all tests | |
731 | in the current directory, tagged as "quick", run: | |
732 | ||
733 | .. code:: | |
734 | ||
735 | avocado run -t quick . | |
736 | ||
737 | The ``avocado_qemu.Test`` base test class | |
738 | ----------------------------------------- | |
739 | ||
740 | The ``avocado_qemu.Test`` class has a number of characteristics that | |
741 | are worth being mentioned right away. | |
742 | ||
743 | First of all, it attempts to give each test a ready to use QEMUMachine | |
744 | instance, available at ``self.vm``. Because many tests will tweak the | |
745 | QEMU command line, launching the QEMUMachine (by using ``self.vm.launch()``) | |
746 | is left to the test writer. | |
747 | ||
b7287d42 CC |
748 | The base test class has also support for tests with more than one |
749 | QEMUMachine. The way to get machines is through the ``self.get_vm()`` | |
750 | method which will return a QEMUMachine instance. The ``self.get_vm()`` | |
751 | method accepts arguments that will be passed to the QEMUMachine creation | |
752 | and also an optional `name` attribute so you can identify a specific | |
753 | machine and get it more than once through the tests methods. A simple | |
754 | and hypothetical example follows: | |
755 | ||
756 | .. code:: | |
757 | ||
758 | from avocado_qemu import Test | |
759 | ||
760 | ||
761 | class MultipleMachines(Test): | |
762 | """ | |
763 | :avocado: enable | |
764 | """ | |
765 | def test_multiple_machines(self): | |
766 | first_machine = self.get_vm() | |
767 | second_machine = self.get_vm() | |
768 | self.get_vm(name='third_machine').launch() | |
769 | ||
770 | first_machine.launch() | |
771 | second_machine.launch() | |
772 | ||
773 | first_res = first_machine.command( | |
774 | 'human-monitor-command', | |
775 | command_line='info version') | |
776 | ||
777 | second_res = second_machine.command( | |
778 | 'human-monitor-command', | |
779 | command_line='info version') | |
780 | ||
781 | third_res = self.get_vm(name='third_machine').command( | |
782 | 'human-monitor-command', | |
783 | command_line='info version') | |
784 | ||
785 | self.assertEquals(first_res, second_res, third_res) | |
786 | ||
787 | At test "tear down", ``avocado_qemu.Test`` handles all the QEMUMachines | |
c3d7e8c9 CR |
788 | shutdown. |
789 | ||
790 | QEMUMachine | |
791 | ~~~~~~~~~~~ | |
792 | ||
793 | The QEMUMachine API is already widely used in the Python iotests, | |
794 | device-crash-test and other Python scripts. It's a wrapper around the | |
795 | execution of a QEMU binary, giving its users: | |
796 | ||
797 | * the ability to set command line arguments to be given to the QEMU | |
798 | binary | |
799 | ||
800 | * a ready to use QMP connection and interface, which can be used to | |
801 | send commands and inspect its results, as well as asynchronous | |
802 | events | |
803 | ||
804 | * convenience methods to set commonly used command line arguments in | |
805 | a more succinct and intuitive way | |
806 | ||
807 | QEMU binary selection | |
808 | ~~~~~~~~~~~~~~~~~~~~~ | |
809 | ||
810 | The QEMU binary used for the ``self.vm`` QEMUMachine instance will | |
811 | primarily depend on the value of the ``qemu_bin`` parameter. If it's | |
812 | not explicitly set, its default value will be the result of a dynamic | |
813 | probe in the same source tree. A suitable binary will be one that | |
814 | targets the architecture matching host machine. | |
815 | ||
816 | Based on this description, test writers will usually rely on one of | |
817 | the following approaches: | |
818 | ||
819 | 1) Set ``qemu_bin``, and use the given binary | |
820 | ||
821 | 2) Do not set ``qemu_bin``, and use a QEMU binary named like | |
64ed6f92 | 822 | "qemu-system-${arch}", either in the current |
c3d7e8c9 CR |
823 | working directory, or in the current source tree. |
824 | ||
825 | The resulting ``qemu_bin`` value will be preserved in the | |
826 | ``avocado_qemu.Test`` as an attribute with the same name. | |
827 | ||
828 | Attribute reference | |
829 | ------------------- | |
830 | ||
831 | Besides the attributes and methods that are part of the base | |
832 | ``avocado.Test`` class, the following attributes are available on any | |
833 | ``avocado_qemu.Test`` instance. | |
834 | ||
835 | vm | |
836 | ~~ | |
837 | ||
838 | A QEMUMachine instance, initially configured according to the given | |
839 | ``qemu_bin`` parameter. | |
840 | ||
2c44d68f CR |
841 | arch |
842 | ~~~~ | |
843 | ||
844 | The architecture can be used on different levels of the stack, e.g. by | |
845 | the framework or by the test itself. At the framework level, it will | |
846 | currently influence the selection of a QEMU binary (when one is not | |
847 | explicitly given). | |
848 | ||
849 | Tests are also free to use this attribute value, for their own needs. | |
850 | A test may, for instance, use the same value when selecting the | |
851 | architecture of a kernel or disk image to boot a VM with. | |
852 | ||
853 | The ``arch`` attribute will be set to the test parameter of the same | |
b910545f CR |
854 | name. If one is not given explicitly, it will either be set to |
855 | ``None``, or, if the test is tagged with one (and only one) | |
856 | ``:avocado: tags=arch:VALUE`` tag, it will be set to ``VALUE``. | |
2c44d68f | 857 | |
ba21bde9 CR |
858 | machine |
859 | ~~~~~~~ | |
860 | ||
861 | The machine type that will be set to all QEMUMachine instances created | |
862 | by the test. | |
863 | ||
864 | The ``machine`` attribute will be set to the test parameter of the same | |
865 | name. If one is not given explicitly, it will either be set to | |
866 | ``None``, or, if the test is tagged with one (and only one) | |
867 | ``:avocado: tags=machine:VALUE`` tag, it will be set to ``VALUE``. | |
868 | ||
c3d7e8c9 CR |
869 | qemu_bin |
870 | ~~~~~~~~ | |
871 | ||
872 | The preserved value of the ``qemu_bin`` parameter or the result of the | |
873 | dynamic probe for a QEMU binary in the current working directory or | |
874 | source tree. | |
875 | ||
876 | Parameter reference | |
877 | ------------------- | |
878 | ||
879 | To understand how Avocado parameters are accessed by tests, and how | |
880 | they can be passed to tests, please refer to:: | |
881 | ||
882 | http://avocado-framework.readthedocs.io/en/latest/WritingTests.html#accessing-test-parameters | |
883 | ||
884 | Parameter values can be easily seen in the log files, and will look | |
885 | like the following: | |
886 | ||
887 | .. code:: | |
888 | ||
64ed6f92 | 889 | PARAMS (key=qemu_bin, path=*, default=./qemu-system-x86_64) => './qemu-system-x86_64 |
c3d7e8c9 | 890 | |
2c44d68f CR |
891 | arch |
892 | ~~~~ | |
893 | ||
894 | The architecture that will influence the selection of a QEMU binary | |
895 | (when one is not explicitly given). | |
896 | ||
897 | Tests are also free to use this parameter value, for their own needs. | |
898 | A test may, for instance, use the same value when selecting the | |
899 | architecture of a kernel or disk image to boot a VM with. | |
900 | ||
901 | This parameter has a direct relation with the ``arch`` attribute. If | |
902 | not given, it will default to None. | |
903 | ||
ba21bde9 CR |
904 | machine |
905 | ~~~~~~~ | |
906 | ||
907 | The machine type that will be set to all QEMUMachine instances created | |
908 | by the test. | |
909 | ||
910 | ||
c3d7e8c9 CR |
911 | qemu_bin |
912 | ~~~~~~~~ | |
913 | ||
914 | The exact QEMU binary to be used on QEMUMachine. | |
915 | ||
916 | Uninstalling Avocado | |
917 | -------------------- | |
918 | ||
a56931ee CR |
919 | If you've followed the manual installation instructions above, you can |
920 | easily uninstall Avocado. Start by listing the packages you have | |
921 | installed:: | |
c3d7e8c9 CR |
922 | |
923 | pip list --user | |
924 | ||
925 | And remove any package you want with:: | |
926 | ||
927 | pip uninstall <package_name> | |
a56931ee CR |
928 | |
929 | If you've used ``make check-acceptance``, the Python virtual environment where | |
930 | Avocado is installed will be cleaned up as part of ``make check-clean``. | |
f8ed349e AB |
931 | |
932 | Testing with "make check-tcg" | |
933 | ============================= | |
934 | ||
935 | The check-tcg tests are intended for simple smoke tests of both | |
936 | linux-user and softmmu TCG functionality. However to build test | |
937 | programs for guest targets you need to have cross compilers available. | |
938 | If your distribution supports cross compilers you can do something as | |
939 | simple as:: | |
940 | ||
941 | apt install gcc-aarch64-linux-gnu | |
942 | ||
943 | The configure script will automatically pick up their presence. | |
944 | Sometimes compilers have slightly odd names so the availability of | |
945 | them can be prompted by passing in the appropriate configure option | |
946 | for the architecture in question, for example:: | |
947 | ||
948 | $(configure) --cross-cc-aarch64=aarch64-cc | |
949 | ||
950 | There is also a ``--cross-cc-flags-ARCH`` flag in case additional | |
951 | compiler flags are needed to build for a given target. | |
952 | ||
953 | If you have the ability to run containers as the user you can also | |
954 | take advantage of the build systems "Docker" support. It will then use | |
955 | containers to build any test case for an enabled guest where there is | |
956 | no system compiler available. See :ref: `_docker-ref` for details. | |
957 | ||
958 | Running subset of tests | |
959 | ----------------------- | |
960 | ||
961 | You can build the tests for one architecture:: | |
962 | ||
963 | make build-tcg-tests-$TARGET | |
964 | ||
965 | And run with:: | |
966 | ||
967 | make run-tcg-tests-$TARGET | |
968 | ||
969 | Adding ``V=1`` to the invocation will show the details of how to | |
970 | invoke QEMU for the test which is useful for debugging tests. | |
971 | ||
972 | TCG test dependencies | |
973 | --------------------- | |
974 | ||
975 | The TCG tests are deliberately very light on dependencies and are | |
976 | either totally bare with minimal gcc lib support (for softmmu tests) | |
977 | or just glibc (for linux-user tests). This is because getting a cross | |
978 | compiler to work with additional libraries can be challenging. | |
979 | ||
980 | Other TCG Tests | |
981 | --------------- | |
982 | ||
983 | There are a number of out-of-tree test suites that are used for more | |
984 | extensive testing of processor features. | |
985 | ||
986 | KVM Unit Tests | |
987 | ~~~~~~~~~~~~~~ | |
988 | ||
989 | The KVM unit tests are designed to run as a Guest OS under KVM but | |
990 | there is no reason why they can't exercise the TCG as well. It | |
991 | provides a minimal OS kernel with hooks for enabling the MMU as well | |
992 | as reporting test results via a special device:: | |
993 | ||
994 | https://git.kernel.org/pub/scm/virt/kvm/kvm-unit-tests.git | |
995 | ||
996 | Linux Test Project | |
997 | ~~~~~~~~~~~~~~~~~~ | |
998 | ||
999 | The LTP is focused on exercising the syscall interface of a Linux | |
1000 | kernel. It checks that syscalls behave as documented and strives to | |
1001 | exercise as many corner cases as possible. It is a useful test suite | |
1002 | to run to exercise QEMU's linux-user code:: | |
1003 | ||
1004 | https://linux-test-project.github.io/ |