5 This document describes the testing infrastructure in QEMU.
7 Testing with "make check"
8 =========================
10 The "make check" testing family includes most of the C based tests in QEMU. For
11 a quick help, run ``make check-help`` from the source tree.
13 The usual way to run these tests is:
19 which includes QAPI schema tests, unit tests, QTests and some iotests.
20 Different sub-types of "make check" tests will be explained below.
22 Before running tests, it is best to build QEMU programs first. Some tests
23 expect the executables to exist and will fail with obscure messages if they
29 Unit tests, which can be invoked with ``make check-unit``, are simple C tests
30 that typically link to individual QEMU object files and exercise them by
31 calling exported functions.
33 If you are writing new code in QEMU, consider adding a unit test, especially
34 for utility modules that are relatively stateless or have few dependencies. To
37 1. Create a new source file. For example, ``tests/unit/foo-test.c``.
39 2. Write the test. Normally you would include the header file which exports
40 the module API, then verify the interface behaves as expected from your
41 test. The test code should be organized with the glib testing framework.
42 Copying and modifying an existing test is usually a good idea.
44 3. Add the test to ``tests/unit/meson.build``. The unit tests are listed in a
45 dictionary called ``tests``. The values are any additional sources and
46 dependencies to be linked with the test. For a simple test whose source
47 is in ``tests/unit/foo-test.c``, it is enough to add an entry like::
55 Since unit tests don't require environment variables, the simplest way to debug
56 a unit test failure is often directly invoking it or even running it under
57 ``gdb``. However there can still be differences in behavior between ``make``
58 invocations and your manual run, due to ``$MALLOC_PERTURB_`` environment
59 variable (which affects memory reclamation and catches invalid pointers better)
60 and gtester options. If necessary, you can run
66 and copy the actual command line which executes the unit test, then run
67 it from the command line.
72 QTest is a device emulation testing framework. It can be very useful to test
73 device models; it could also control certain aspects of QEMU (such as virtual
74 clock stepping), with a special purpose "qtest" protocol. Refer to
75 :doc:`qtest` for more details.
77 QTest cases can be executed with
86 The QAPI schema tests validate the QAPI parser used by QMP, by feeding
87 predefined input to the parser and comparing the result with the reference
90 The input/output data is managed under the ``tests/qapi-schema`` directory.
91 Each test case includes four files that have a common base name:
93 * ``${casename}.json`` - the file contains the JSON input for feeding the
95 * ``${casename}.out`` - the file contains the expected stdout from the parser
96 * ``${casename}.err`` - the file contains the expected stderr from the parser
97 * ``${casename}.exit`` - the expected error code
99 Consider adding a new QAPI schema test when you are making a change on the QAPI
100 parser (either fixing a bug or extending/modifying the syntax). To do this:
102 1. Add four files for the new case as explained above. For example:
104 ``$EDITOR tests/qapi-schema/foo.{json,out,err,exit}``.
106 2. Add the new test in ``tests/Makefile.include``. For example:
108 ``qapi-schema += foo.json``
113 ``make check-block`` runs a subset of the block layer iotests (the tests that
114 are in the "auto" group).
115 See the "QEMU iotests" section below for more information.
120 ``gcov`` is a GCC tool to analyze the testing coverage by
121 instrumenting the tested code. To use it, configure QEMU with
122 ``--enable-gcov`` option and build. Then run ``make check`` as usual.
124 If you want to gather coverage information on a single test the ``make
125 clean-gcda`` target can be used to delete any existing coverage
126 information before running a single test.
128 You can generate a HTML coverage report by executing ``make
129 coverage-html`` which will create
130 ``meson-logs/coveragereport/index.html``.
132 Further analysis can be conducted by running the ``gcov`` command
133 directly on the various .gcda output files. Please read the ``gcov``
134 documentation for more information.
139 QEMU iotests, under the directory ``tests/qemu-iotests``, is the testing
140 framework widely used to test block layer related features. It is higher level
141 than "make check" tests and 99% of the code is written in bash or Python
142 scripts. The testing success criteria is golden output comparison, and the
143 test files are named with numbers.
145 To run iotests, make sure QEMU is built successfully, then switch to the
146 ``tests/qemu-iotests`` directory under the build directory, and run ``./check``
147 with desired arguments from there.
149 By default, "raw" format and "file" protocol is used; all tests will be
150 executed, except the unsupported ones. You can override the format and protocol
155 # test with qcow2 format
157 # or test a different protocol
160 It's also possible to list test numbers explicitly:
164 # run selected cases with qcow2 format
165 ./check -qcow2 001 030 153
167 Cache mode can be selected with the "-c" option, which may help reveal bugs
168 that are specific to certain cache mode.
170 More options are supported by the ``./check`` script, run ``./check -h`` for
173 Writing a new test case
174 -----------------------
176 Consider writing a tests case when you are making any changes to the block
177 layer. An iotest case is usually the choice for that. There are already many
178 test cases, so it is possible that extending one of them may achieve the goal
179 and save the boilerplate to create one. (Unfortunately, there isn't a 100%
180 reliable way to find a related one out of hundreds of tests. One approach is
183 Usually an iotest case consists of two files. One is an executable that
184 produces output to stdout and stderr, the other is the expected reference
185 output. They are given the same number in file names. E.g. Test script ``055``
186 and reference output ``055.out``.
188 In rare cases, when outputs differ between cache mode ``none`` and others, a
189 ``.out.nocache`` file is added. In other cases, when outputs differ between
190 image formats, more than one ``.out`` files are created ending with the
191 respective format names, e.g. ``178.out.qcow2`` and ``178.out.raw``.
193 There isn't a hard rule about how to write a test script, but a new test is
194 usually a (copy and) modification of an existing case. There are a few
195 commonly used ways to create a test:
197 * A Bash script. It will make use of several environmental variables related
198 to the testing procedure, and could source a group of ``common.*`` libraries
199 for some common helper routines.
201 * A Python unittest script. Import ``iotests`` and create a subclass of
202 ``iotests.QMPTestCase``, then call ``iotests.main`` method. The downside of
203 this approach is that the output is too scarce, and the script is considered
206 * A simple Python script without using unittest module. This could also import
207 ``iotests`` for launching QEMU and utilities etc, but it doesn't inherit
208 from ``iotests.QMPTestCase`` therefore doesn't use the Python unittest
209 execution. This is a combination of 1 and 2.
211 Pick the language per your preference since both Bash and Python have
212 comparable library support for invoking and interacting with QEMU programs. If
213 you opt for Python, it is strongly recommended to write Python 3 compatible
216 Both Python and Bash frameworks in iotests provide helpers to manage test
217 images. They can be used to create and clean up images under the test
218 directory. If no I/O or any protocol specific feature is needed, it is often
219 more convenient to use the pseudo block driver, ``null-co://``, as the test
220 image, which doesn't require image creation or cleaning up. Avoid system-wide
221 devices or files whenever possible, such as ``/dev/null`` or ``/dev/zero``.
222 Otherwise, image locking implications have to be considered. For example,
223 another application on the host may have locked the file, possibly leading to a
224 test failure. If using such devices are explicitly desired, consider adding
225 ``locking=off`` option to disable image locking.
227 Debugging a test case
228 -----------------------
229 The following options to the ``check`` script can be useful when debugging
232 * ``-gdb`` wraps every QEMU invocation in a ``gdbserver``, which waits for a
233 connection from a gdb client. The options given to ``gdbserver`` (e.g. the
234 address on which to listen for connections) are taken from the ``$GDB_OPTIONS``
235 environment variable. By default (if ``$GDB_OPTIONS`` is empty), it listens on
237 It is possible to connect to it for example with
238 ``gdb -iex "target remote $addr"``, where ``$addr`` is the address
239 ``gdbserver`` listens on.
240 If the ``-gdb`` option is not used, ``$GDB_OPTIONS`` is ignored,
241 regardless of whether it is set or not.
243 * ``-valgrind`` attaches a valgrind instance to QEMU. If it detects
244 warnings, it will print and save the log in
245 ``$TEST_DIR/<valgrind_pid>.valgrind``.
246 The final command line will be ``valgrind --log-file=$TEST_DIR/
247 <valgrind_pid>.valgrind --error-exitcode=99 $QEMU ...``
249 * ``-d`` (debug) just increases the logging verbosity, showing
250 for example the QMP commands and answers.
252 * ``-p`` (print) redirects QEMU’s stdout and stderr to the test output,
253 instead of saving it into a log file in
254 ``$TEST_DIR/qemu-machine-<random_string>``.
259 "Tests may belong to one or more test groups, which are defined in the form
260 of a comment in the test source file. By convention, test groups are listed
261 in the second line of the test file, after the "#!/..." line, like this:
265 #!/usr/bin/env python3
270 Another way of defining groups is creating the tests/qemu-iotests/group.local
271 file. This should be used only for downstream (this file should never appear
272 in upstream). This file may be used for defining some downstream test groups
273 or for temporarily disabling tests, like this:
277 # groups for some company downstream process
279 # ci - tests to run on build
280 # down - our downstream tests, not for upstream
282 # Format of each line is:
283 # TEST_NAME TEST_GROUP [TEST_GROUP ]...
288 our-ugly-workaround-test down ci
290 Note that the following group names have a special meaning:
292 - quick: Tests in this group should finish within a few seconds.
294 - auto: Tests in this group are used during "make check" and should be
295 runnable in any case. That means they should run with every QEMU binary
296 (also non-x86), with every QEMU configuration (i.e. must not fail if
297 an optional feature is not compiled in - but reporting a "skip" is ok),
298 work at least with the qcow2 file format, work with all kind of host
299 filesystems and users (e.g. "nobody" or "root") and must not take too
300 much memory and disk space (since CI pipelines tend to fail otherwise).
302 - disabled: Tests in this group are disabled and ignored by check.
306 Container based tests
307 =====================
312 The container testing framework in QEMU utilizes public images to
313 build and test QEMU in predefined and widely accessible Linux
314 environments. This makes it possible to expand the test coverage
315 across distros, toolchain flavors and library versions. The support
316 was originally written for Docker although we also support Podman as
317 an alternative container runtime. Although the many of the target
318 names and scripts are prefixed with "docker" the system will
319 automatically run on whichever is configured.
321 The container images are also used to augment the generation of tests
322 for testing TCG. See :ref:`checktcg-ref` for more details.
327 Install "docker" with the system package manager and start the Docker service
328 on your development machine, then make sure you have the privilege to run
329 Docker commands. Typically it means setting up passwordless ``sudo docker``
330 command or login as root. For example:
334 $ sudo yum install docker
335 $ # or `apt-get install docker` for Ubuntu, etc.
336 $ sudo systemctl start docker
339 The last command should print an empty table, to verify the system is ready.
341 An alternative method to set up permissions is by adding the current user to
342 "docker" group and making the docker daemon socket file (by default
343 ``/var/run/docker.sock``) accessible to the group:
347 $ sudo groupadd docker
348 $ sudo usermod $USER -a -G docker
349 $ sudo chown :docker /var/run/docker.sock
351 Note that any one of above configurations makes it possible for the user to
352 exploit the whole host with Docker bind mounting or other privileged
353 operations. So only do it on development machines.
358 Install "podman" with the system package manager.
362 $ sudo dnf install podman
365 The last command should print an empty table, to verify the system is ready.
370 From source tree, type ``make docker-help`` to see the help. Testing
371 can be started without configuring or building QEMU (``configure`` and
372 ``make`` are done in the container, with parameters defined by the
377 make docker-test-build@centos8
379 This will create a container instance using the ``centos8`` image (the image
380 is downloaded and initialized automatically), in which the ``test-build`` job
386 The QEMU project has a container registry hosted by GitLab at
387 ``registry.gitlab.com/qemu-project/qemu`` which will automatically be
388 used to pull in pre-built layers. This avoids unnecessary strain on
389 the distro archives created by multiple developers running the same
390 container build steps over and over again. This can be overridden
391 locally by using the ``NOCACHE`` build option:
395 make docker-image-debian10 NOCACHE=1
400 Along with many other images, the ``centos8`` image is defined in a Dockerfile
401 in ``tests/docker/dockerfiles/``, called ``centos8.docker``. ``make docker-help``
402 command will list all the available images.
404 To add a new image, simply create a new ``.docker`` file under the
405 ``tests/docker/dockerfiles/`` directory.
407 A ``.pre`` script can be added beside the ``.docker`` file, which will be
408 executed before building the image under the build context directory. This is
409 mainly used to do necessary host side setup. One such setup is ``binfmt_misc``,
410 for example, to make qemu-user powered cross build containers work.
415 Different tests are added to cover various configurations to build and test
416 QEMU. Docker tests are the executables under ``tests/docker`` named
417 ``test-*``. They are typically shell scripts and are built on top of a shell
418 library, ``tests/docker/common.rc``, which provides helpers to find the QEMU
421 The full list of tests is printed in the ``make docker-help`` help.
423 Debugging a Docker test failure
424 -------------------------------
426 When CI tasks, maintainers or yourself report a Docker test failure, follow the
427 below steps to debug it:
429 1. Locally reproduce the failure with the reported command line. E.g. run
430 ``make docker-test-mingw@fedora J=8``.
431 2. Add "V=1" to the command line, try again, to see the verbose output.
432 3. Further add "DEBUG=1" to the command line. This will pause in a shell prompt
433 in the container right before testing starts. You could either manually
434 build QEMU and run tests from there, or press Ctrl-D to let the Docker
436 4. If you press Ctrl-D, the same building and testing procedure will begin, and
437 will hopefully run into the error again. After that, you will be dropped to
438 the prompt for debug.
443 Various options can be used to affect how Docker tests are done. The full
444 list is in the ``make docker`` help text. The frequently used ones are:
446 * ``V=1``: the same as in top level ``make``. It will be propagated to the
447 container and enable verbose output.
448 * ``J=$N``: the number of parallel tasks in make commands in the container,
449 similar to the ``-j $N`` option in top level ``make``. (The ``-j`` option in
450 top level ``make`` will not be propagated into the container.)
451 * ``DEBUG=1``: enables debug. See the previous "Debugging a Docker test
457 Thread Sanitizer (TSan) is a tool which can detect data races. QEMU supports
458 building and testing with this tool.
460 For more information on TSan:
462 https://github.com/google/sanitizers/wiki/ThreadSanitizerCppManual
464 Thread Sanitizer in Docker
465 ---------------------------
466 TSan is currently supported in the ubuntu2004 docker.
468 The test-tsan test will build using TSan and then run make check.
472 make docker-test-tsan@ubuntu2004
474 TSan warnings under docker are placed in files located at build/tsan/.
476 We recommend using DEBUG=1 to allow launching the test from inside the docker,
477 and to allow review of the warnings generated by TSan.
479 Building and Testing with TSan
480 ------------------------------
482 It is possible to build and test with TSan, with a few additional steps.
483 These steps are normally done automatically in the docker.
485 There is a one time patch needed in clang-9 or clang-10 at this time:
489 sed -i 's/^const/static const/g' \
490 /usr/lib/llvm-10/lib/clang/10.0.0/include/sanitizer/tsan_interface.h
492 To configure the build for TSan:
496 ../configure --enable-tsan --cc=clang-10 --cxx=clang++-10 \
497 --disable-werror --extra-cflags="-O0"
499 The runtime behavior of TSAN is controlled by the TSAN_OPTIONS environment
502 More information on the TSAN_OPTIONS can be found here:
504 https://github.com/google/sanitizers/wiki/ThreadSanitizerFlags
510 export TSAN_OPTIONS=suppressions=<path to qemu>/tests/tsan/suppressions.tsan \
511 detect_deadlocks=false history_size=7 exitcode=0 \
512 log_path=<build path>/tsan/tsan_warning
514 The above exitcode=0 has TSan continue without error if any warnings are found.
515 This allows for running the test and then checking the warnings afterwards.
516 If you want TSan to stop and exit with error on warnings, use exitcode=66.
520 Keep in mind that for any data race warning, although there might be a data race
521 detected by TSan, there might be no actual bug here. TSan provides several
522 different mechanisms for suppressing warnings. In general it is recommended
523 to fix the code if possible to eliminate the data race rather than suppress
526 A few important files for suppressing warnings are:
528 tests/tsan/suppressions.tsan - Has TSan warnings we wish to suppress at runtime.
529 The comment on each suppression will typically indicate why we are
530 suppressing it. More information on the file format can be found here:
532 https://github.com/google/sanitizers/wiki/ThreadSanitizerSuppressions
534 tests/tsan/blacklist.tsan - Has TSan warnings we wish to disable
535 at compile time for test or debug.
536 Add flags to configure to enable:
538 "--extra-cflags=-fsanitize-blacklist=<src path>/tests/tsan/blacklist.tsan"
540 More information on the file format can be found here under "Blacklist Format":
542 https://github.com/google/sanitizers/wiki/ThreadSanitizerFlags
546 include/qemu/tsan.h defines annotations. See this file for more descriptions
547 of the annotations themselves. Annotations can be used to suppress
548 TSan warnings or give TSan more information so that it can detect proper
549 relationships between accesses of data.
551 Annotation examples can be found here:
553 https://github.com/llvm/llvm-project/tree/master/compiler-rt/test/tsan/
555 Good files to start with are: annotate_happens_before.cpp and ignore_race.cpp
557 The full set of annotations can be found here:
559 https://github.com/llvm/llvm-project/blob/master/compiler-rt/lib/tsan/rtl/tsan_interface_ann.cpp
564 This test suite contains scripts that bootstrap various guest images that have
565 necessary packages to build QEMU. The basic usage is documented in ``Makefile``
566 help which is displayed with ``make vm-help``.
571 Run ``make vm-help`` to list available make targets. Invoke a specific make
572 command to run build test in an image. For example, ``make vm-build-freebsd``
573 will build the source tree in the FreeBSD image. The command can be executed
574 from either the source tree or the build dir; if the former, ``./configure`` is
575 not needed. The command will then generate the test image in ``./tests/vm/``
576 under the working directory.
578 Note: images created by the scripts accept a well-known RSA key pair for SSH
579 access, so they SHOULD NOT be exposed to external interfaces if you are
580 concerned about attackers taking control of the guest and potentially
581 exploiting a QEMU security bug to compromise the host.
586 By default, qemu-system-x86_64 is searched in $PATH to run the guest. If there
587 isn't one, or if it is older than 2.10, the test won't work. In this case,
588 provide the QEMU binary in env var: ``QEMU=/path/to/qemu-2.10+``.
590 Likewise the path to qemu-img can be set in QEMU_IMG environment variable.
595 The ``-j$X`` option in the make command line is not propagated into the VM,
596 specify ``J=$X`` to control the make jobs in the guest.
601 Add ``DEBUG=1`` and/or ``V=1`` to the make command to allow interactive
602 debugging and verbose output. If this is not enough, see the next section.
603 ``V=1`` will be propagated down into the make jobs in the guest.
608 Each guest script is an executable script with the same command line options.
609 For example to work with the netbsd guest, use ``$QEMU_SRC/tests/vm/netbsd``:
613 $ cd $QEMU_SRC/tests/vm
615 # To bootstrap the image
616 $ ./netbsd --build-image --image /var/tmp/netbsd.img
619 # To run an arbitrary command in guest (the output will not be echoed unless
621 $ ./netbsd --debug --image /var/tmp/netbsd.img uname -a
623 # To build QEMU in guest
624 $ ./netbsd --debug --image /var/tmp/netbsd.img --build-qemu $QEMU_SRC
626 # To get to an interactive shell
627 $ ./netbsd --interactive --image /var/tmp/netbsd.img sh
632 Please look at existing guest scripts for how to add new guests.
634 Most importantly, create a subclass of BaseVM and implement ``build_image()``
635 method and define ``BUILD_SCRIPT``, then finally call ``basevm.main()`` from
636 the script's ``main()``.
638 * Usually in ``build_image()``, a template image is downloaded from a
639 predefined URL. ``BaseVM._download_with_cache()`` takes care of the cache and
640 the checksum, so consider using it.
642 * Once the image is downloaded, users, SSH server and QEMU build deps should
645 - Root password set to ``BaseVM.ROOT_PASS``
646 - User ``BaseVM.GUEST_USER`` is created, and password set to
647 ``BaseVM.GUEST_PASS``
648 - SSH service is enabled and started on boot,
649 ``$QEMU_SRC/tests/keys/id_rsa.pub`` is added to ssh's ``authorized_keys``
650 file of both root and the normal user
651 - DHCP client service is enabled and started on boot, so that it can
652 automatically configure the virtio-net-pci NIC and communicate with QEMU
654 - Necessary packages are installed to untar the source tarball and build
657 * Write a proper ``BUILD_SCRIPT`` template, which should be a shell script that
658 untars a raw virtio-blk block device, which is the tarball data blob of the
659 QEMU source tree, then configure/build it. Running "make check" is also
665 An image fuzzer was added to exercise format drivers. Currently only qcow2 is
666 supported. To start the fuzzer, run
670 tests/image-fuzzer/runner.py -c '[["qemu-img", "info", "$test_img"]]' /tmp/test qcow2
672 Alternatively, some command different from "qemu-img info" can be tested, by
673 changing the ``-c`` option.
675 Acceptance tests using the Avocado Framework
676 ============================================
678 The ``tests/acceptance`` directory hosts functional tests, also known
679 as acceptance level tests. They're usually higher level tests, and
680 may interact with external resources and with various guest operating
683 These tests are written using the Avocado Testing Framework (which must
684 be installed separately) in conjunction with a the ``avocado_qemu.Test``
685 class, implemented at ``tests/acceptance/avocado_qemu``.
687 Tests based on ``avocado_qemu.Test`` can easily:
689 * Customize the command line arguments given to the convenience
690 ``self.vm`` attribute (a QEMUMachine instance)
692 * Interact with the QEMU monitor, send QMP commands and check
695 * Interact with the guest OS, using the convenience console device
696 (which may be useful to assert the effectiveness and correctness of
697 command line arguments or QMP commands)
699 * Interact with external data files that accompany the test itself
700 (see ``self.get_data()``)
702 * Download (and cache) remote data files, such as firmware and kernel
705 * Have access to a library of guest OS images (by means of the
706 ``avocado.utils.vmimage`` library)
708 * Make use of various other test related utilities available at the
709 test class itself and at the utility library:
711 - http://avocado-framework.readthedocs.io/en/latest/api/test/avocado.html#avocado.Test
712 - http://avocado-framework.readthedocs.io/en/latest/api/utils/avocado.utils.html
717 You can run the acceptance tests simply by executing:
721 make check-acceptance
723 This involves the automatic creation of Python virtual environment
724 within the build tree (at ``tests/venv``) which will have all the
725 right dependencies, and will save tests results also within the
726 build tree (at ``tests/results``).
728 Note: the build environment must be using a Python 3 stack, and have
729 the ``venv`` and ``pip`` packages installed. If necessary, make sure
730 ``configure`` is called with ``--python=`` and that those modules are
731 available. On Debian and Ubuntu based systems, depending on the
732 specific version, they may be on packages named ``python3-venv`` and
735 It is also possible to run tests based on tags using the
736 ``make check-acceptance`` command and the ``AVOCADO_TAGS`` environment
741 make check-acceptance AVOCADO_TAGS=quick
743 Note that tags separated with commas have an AND behavior, while tags
744 separated by spaces have an OR behavior. For more information on Avocado
747 https://avocado-framework.readthedocs.io/en/latest/guides/user/chapters/tags.html
749 To run a single test file, a couple of them, or a test within a file
750 using the ``make check-acceptance`` command, set the ``AVOCADO_TESTS``
751 environment variable with the test files or test names. To run all
752 tests from a single file, use:
756 make check-acceptance AVOCADO_TESTS=$FILEPATH
758 The same is valid to run tests from multiple test files:
762 make check-acceptance AVOCADO_TESTS='$FILEPATH1 $FILEPATH2'
764 To run a single test within a file, use:
768 make check-acceptance AVOCADO_TESTS=$FILEPATH:$TESTCLASS.$TESTNAME
770 The same is valid to run single tests from multiple test files:
774 make check-acceptance AVOCADO_TESTS='$FILEPATH1:$TESTCLASS1.$TESTNAME1 $FILEPATH2:$TESTCLASS2.$TESTNAME2'
776 The scripts installed inside the virtual environment may be used
777 without an "activation". For instance, the Avocado test runner
778 may be invoked by running:
782 tests/venv/bin/avocado run $OPTION1 $OPTION2 tests/acceptance/
784 Note that if ``make check-acceptance`` was not executed before, it is
785 possible to create the Python virtual environment with the dependencies
792 It is also possible to run tests from a single file or a single test within
793 a test file. To run tests from a single file within the build tree, use:
797 tests/venv/bin/avocado run tests/acceptance/$TESTFILE
799 To run a single test within a test file, use:
803 tests/venv/bin/avocado run tests/acceptance/$TESTFILE:$TESTCLASS.$TESTNAME
805 Valid test names are visible in the output from any previous execution
806 of Avocado or ``make check-acceptance``, and can also be queried using:
810 tests/venv/bin/avocado list tests/acceptance
815 To manually install Avocado and its dependencies, run:
819 pip install --user avocado-framework
821 Alternatively, follow the instructions on this link:
823 https://avocado-framework.readthedocs.io/en/latest/guides/user/chapters/installing.html
828 The ``tests/acceptance/avocado_qemu`` directory provides the
829 ``avocado_qemu`` Python module, containing the ``avocado_qemu.Test``
830 class. Here's a simple usage example:
834 from avocado_qemu import Test
841 def test_qmp_human_info_version(self):
843 res = self.vm.command('human-monitor-command',
844 command_line='info version')
845 self.assertRegexpMatches(res, r'^(\d+\.\d+\.\d)')
847 To execute your test, run:
851 avocado run version.py
853 Tests may be classified according to a convention by using docstring
854 directives such as ``:avocado: tags=TAG1,TAG2``. To run all tests
855 in the current directory, tagged as "quick", run:
859 avocado run -t quick .
861 The ``avocado_qemu.Test`` base test class
862 -----------------------------------------
864 The ``avocado_qemu.Test`` class has a number of characteristics that
865 are worth being mentioned right away.
867 First of all, it attempts to give each test a ready to use QEMUMachine
868 instance, available at ``self.vm``. Because many tests will tweak the
869 QEMU command line, launching the QEMUMachine (by using ``self.vm.launch()``)
870 is left to the test writer.
872 The base test class has also support for tests with more than one
873 QEMUMachine. The way to get machines is through the ``self.get_vm()``
874 method which will return a QEMUMachine instance. The ``self.get_vm()``
875 method accepts arguments that will be passed to the QEMUMachine creation
876 and also an optional ``name`` attribute so you can identify a specific
877 machine and get it more than once through the tests methods. A simple
878 and hypothetical example follows:
882 from avocado_qemu import Test
885 class MultipleMachines(Test):
886 def test_multiple_machines(self):
887 first_machine = self.get_vm()
888 second_machine = self.get_vm()
889 self.get_vm(name='third_machine').launch()
891 first_machine.launch()
892 second_machine.launch()
894 first_res = first_machine.command(
895 'human-monitor-command',
896 command_line='info version')
898 second_res = second_machine.command(
899 'human-monitor-command',
900 command_line='info version')
902 third_res = self.get_vm(name='third_machine').command(
903 'human-monitor-command',
904 command_line='info version')
906 self.assertEquals(first_res, second_res, third_res)
908 At test "tear down", ``avocado_qemu.Test`` handles all the QEMUMachines
911 The ``avocado_qemu.LinuxTest`` base test class
912 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
914 The ``avocado_qemu.LinuxTest`` is further specialization of the
915 ``avocado_qemu.Test`` class, so it contains all the characteristics of
916 the later plus some extra features.
918 First of all, this base class is intended for tests that need to
919 interact with a fully booted and operational Linux guest. At this
920 time, it uses a Fedora 31 guest image. The most basic example looks
925 from avocado_qemu import LinuxTest
928 class SomeTest(LinuxTest):
931 self.launch_and_wait()
932 self.ssh_command('some_command_to_be_run_in_the_guest')
934 Please refer to tests that use ``avocado_qemu.LinuxTest`` under
935 ``tests/acceptance`` for more examples.
940 The QEMUMachine API is already widely used in the Python iotests,
941 device-crash-test and other Python scripts. It's a wrapper around the
942 execution of a QEMU binary, giving its users:
944 * the ability to set command line arguments to be given to the QEMU
947 * a ready to use QMP connection and interface, which can be used to
948 send commands and inspect its results, as well as asynchronous
951 * convenience methods to set commonly used command line arguments in
952 a more succinct and intuitive way
954 QEMU binary selection
955 ~~~~~~~~~~~~~~~~~~~~~
957 The QEMU binary used for the ``self.vm`` QEMUMachine instance will
958 primarily depend on the value of the ``qemu_bin`` parameter. If it's
959 not explicitly set, its default value will be the result of a dynamic
960 probe in the same source tree. A suitable binary will be one that
961 targets the architecture matching host machine.
963 Based on this description, test writers will usually rely on one of
964 the following approaches:
966 1) Set ``qemu_bin``, and use the given binary
968 2) Do not set ``qemu_bin``, and use a QEMU binary named like
969 "qemu-system-${arch}", either in the current
970 working directory, or in the current source tree.
972 The resulting ``qemu_bin`` value will be preserved in the
973 ``avocado_qemu.Test`` as an attribute with the same name.
978 Besides the attributes and methods that are part of the base
979 ``avocado.Test`` class, the following attributes are available on any
980 ``avocado_qemu.Test`` instance.
985 A QEMUMachine instance, initially configured according to the given
986 ``qemu_bin`` parameter.
991 The architecture can be used on different levels of the stack, e.g. by
992 the framework or by the test itself. At the framework level, it will
993 currently influence the selection of a QEMU binary (when one is not
996 Tests are also free to use this attribute value, for their own needs.
997 A test may, for instance, use the same value when selecting the
998 architecture of a kernel or disk image to boot a VM with.
1000 The ``arch`` attribute will be set to the test parameter of the same
1001 name. If one is not given explicitly, it will either be set to
1002 ``None``, or, if the test is tagged with one (and only one)
1003 ``:avocado: tags=arch:VALUE`` tag, it will be set to ``VALUE``.
1008 The cpu model that will be set to all QEMUMachine instances created
1011 The ``cpu`` attribute will be set to the test parameter of the same
1012 name. If one is not given explicitly, it will either be set to
1013 ``None ``, or, if the test is tagged with one (and only one)
1014 ``:avocado: tags=cpu:VALUE`` tag, it will be set to ``VALUE``.
1019 The machine type that will be set to all QEMUMachine instances created
1022 The ``machine`` attribute will be set to the test parameter of the same
1023 name. If one is not given explicitly, it will either be set to
1024 ``None``, or, if the test is tagged with one (and only one)
1025 ``:avocado: tags=machine:VALUE`` tag, it will be set to ``VALUE``.
1030 The preserved value of the ``qemu_bin`` parameter or the result of the
1031 dynamic probe for a QEMU binary in the current working directory or
1037 Besides the attributes present on the ``avocado_qemu.Test`` base
1038 class, the ``avocado_qemu.LinuxTest`` adds the following attributes:
1043 The name of the Linux distribution used as the guest image for the
1044 test. The name should match the **Provider** column on the list
1045 of images supported by the avocado.utils.vmimage library:
1047 https://avocado-framework.readthedocs.io/en/latest/guides/writer/libs/vmimage.html#supported-images
1052 The version of the Linux distribution as the guest image for the
1053 test. The name should match the **Version** column on the list
1054 of images supported by the avocado.utils.vmimage library:
1056 https://avocado-framework.readthedocs.io/en/latest/guides/writer/libs/vmimage.html#supported-images
1061 The sha256 hash of the guest image file used for the test.
1063 If this value is not set in the code or by a test parameter (with the
1064 same name), no validation on the integrity of the image will be
1070 To understand how Avocado parameters are accessed by tests, and how
1071 they can be passed to tests, please refer to::
1073 https://avocado-framework.readthedocs.io/en/latest/guides/writer/chapters/writing.html#accessing-test-parameters
1075 Parameter values can be easily seen in the log files, and will look
1080 PARAMS (key=qemu_bin, path=*, default=./qemu-system-x86_64) => './qemu-system-x86_64
1085 The architecture that will influence the selection of a QEMU binary
1086 (when one is not explicitly given).
1088 Tests are also free to use this parameter value, for their own needs.
1089 A test may, for instance, use the same value when selecting the
1090 architecture of a kernel or disk image to boot a VM with.
1092 This parameter has a direct relation with the ``arch`` attribute. If
1093 not given, it will default to None.
1098 The cpu model that will be set to all QEMUMachine instances created
1104 The machine type that will be set to all QEMUMachine instances created
1111 The exact QEMU binary to be used on QEMUMachine.
1116 Besides the parameters present on the ``avocado_qemu.Test`` base
1117 class, the ``avocado_qemu.LinuxTest`` adds the following parameters:
1122 The name of the Linux distribution used as the guest image for the
1123 test. The name should match the **Provider** column on the list
1124 of images supported by the avocado.utils.vmimage library:
1126 https://avocado-framework.readthedocs.io/en/latest/guides/writer/libs/vmimage.html#supported-images
1131 The version of the Linux distribution as the guest image for the
1132 test. The name should match the **Version** column on the list
1133 of images supported by the avocado.utils.vmimage library:
1135 https://avocado-framework.readthedocs.io/en/latest/guides/writer/libs/vmimage.html#supported-images
1140 The sha256 hash of the guest image file used for the test.
1142 If this value is not set in the code or by this parameter no
1143 validation on the integrity of the image will be performed.
1147 The Avocado framework provides Python decorators which allow for easily skip
1148 tests running under certain conditions. For example, on the lack of a binary
1149 on the test system or when the running environment is a CI system. For further
1150 information about those decorators, please refer to::
1152 https://avocado-framework.readthedocs.io/en/latest/guides/writer/chapters/writing.html#skipping-tests
1154 While the conditions for skipping tests are often specifics of each one, there
1155 are recurring scenarios identified by the QEMU developers and the use of
1156 environment variables became a kind of standard way to enable/disable tests.
1158 Here is a list of the most used variables:
1160 AVOCADO_ALLOW_LARGE_STORAGE
1161 ~~~~~~~~~~~~~~~~~~~~~~~~~~~
1162 Tests which are going to fetch or produce assets considered *large* are not
1163 going to run unless that ``AVOCADO_ALLOW_LARGE_STORAGE=1`` is exported on
1166 The definition of *large* is a bit arbitrary here, but it usually means an
1167 asset which occupies at least 1GB of size on disk when uncompressed.
1169 AVOCADO_ALLOW_UNTRUSTED_CODE
1170 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1171 There are tests which will boot a kernel image or firmware that can be
1172 considered not safe to run on the developer's workstation, thus they are
1173 skipped by default. The definition of *not safe* is also arbitrary but
1174 usually it means a blob which either its source or build process aren't
1177 You should export ``AVOCADO_ALLOW_UNTRUSTED_CODE=1`` on the environment in
1178 order to allow tests which make use of those kind of assets.
1180 AVOCADO_TIMEOUT_EXPECTED
1181 ~~~~~~~~~~~~~~~~~~~~~~~~
1182 The Avocado framework has a timeout mechanism which interrupts tests to avoid the
1183 test suite of getting stuck. The timeout value can be set via test parameter or
1184 property defined in the test class, for further details::
1186 https://avocado-framework.readthedocs.io/en/latest/guides/writer/chapters/writing.html#setting-a-test-timeout
1188 Even though the timeout can be set by the test developer, there are some tests
1189 that may not have a well-defined limit of time to finish under certain
1190 conditions. For example, tests that take longer to execute when QEMU is
1191 compiled with debug flags. Therefore, the ``AVOCADO_TIMEOUT_EXPECTED`` variable
1192 has been used to determine whether those tests should run or not.
1196 A number of tests are flagged to not run on the GitLab CI. Usually because
1197 they proved to the flaky or there are constraints on the CI environment which
1198 would make them fail. If you encounter a similar situation then use that
1199 variable as shown on the code snippet below to skip the test:
1203 @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab')
1207 Uninstalling Avocado
1208 --------------------
1210 If you've followed the manual installation instructions above, you can
1211 easily uninstall Avocado. Start by listing the packages you have
1216 And remove any package you want with::
1218 pip uninstall <package_name>
1220 If you've used ``make check-acceptance``, the Python virtual environment where
1221 Avocado is installed will be cleaned up as part of ``make check-clean``.
1225 Testing with "make check-tcg"
1226 =============================
1228 The check-tcg tests are intended for simple smoke tests of both
1229 linux-user and softmmu TCG functionality. However to build test
1230 programs for guest targets you need to have cross compilers available.
1231 If your distribution supports cross compilers you can do something as
1234 apt install gcc-aarch64-linux-gnu
1236 The configure script will automatically pick up their presence.
1237 Sometimes compilers have slightly odd names so the availability of
1238 them can be prompted by passing in the appropriate configure option
1239 for the architecture in question, for example::
1241 $(configure) --cross-cc-aarch64=aarch64-cc
1243 There is also a ``--cross-cc-flags-ARCH`` flag in case additional
1244 compiler flags are needed to build for a given target.
1246 If you have the ability to run containers as the user the build system
1247 will automatically use them where no system compiler is available. For
1248 architectures where we also support building QEMU we will generally
1249 use the same container to build tests. However there are a number of
1250 additional containers defined that have a minimal cross-build
1251 environment that is only suitable for building test cases. Sometimes
1252 we may use a bleeding edge distribution for compiler features needed
1253 for test cases that aren't yet in the LTS distros we support for QEMU
1256 See :ref:`container-ref` for more details.
1258 Running subset of tests
1259 -----------------------
1261 You can build the tests for one architecture::
1263 make build-tcg-tests-$TARGET
1267 make run-tcg-tests-$TARGET
1269 Adding ``V=1`` to the invocation will show the details of how to
1270 invoke QEMU for the test which is useful for debugging tests.
1272 TCG test dependencies
1273 ---------------------
1275 The TCG tests are deliberately very light on dependencies and are
1276 either totally bare with minimal gcc lib support (for softmmu tests)
1277 or just glibc (for linux-user tests). This is because getting a cross
1278 compiler to work with additional libraries can be challenging.
1283 There are a number of out-of-tree test suites that are used for more
1284 extensive testing of processor features.
1289 The KVM unit tests are designed to run as a Guest OS under KVM but
1290 there is no reason why they can't exercise the TCG as well. It
1291 provides a minimal OS kernel with hooks for enabling the MMU as well
1292 as reporting test results via a special device::
1294 https://git.kernel.org/pub/scm/virt/kvm/kvm-unit-tests.git
1299 The LTP is focused on exercising the syscall interface of a Linux
1300 kernel. It checks that syscalls behave as documented and strives to
1301 exercise as many corner cases as possible. It is a useful test suite
1302 to run to exercise QEMU's linux-user code::
1304 https://linux-test-project.github.io/