]>
Commit | Line | Data |
---|---|---|
7266ecce AB |
1 | .. _testing: |
2 | ||
4eb99560 FZ |
3 | Testing in QEMU |
4 | =============== | |
5 | ||
6 | This document describes the testing infrastructure in QEMU. | |
7 | ||
8 | Testing with "make check" | |
16e79e1b | 9 | ------------------------- |
4eb99560 FZ |
10 | |
11 | The "make check" testing family includes most of the C based tests in QEMU. For | |
12 | a quick help, run ``make check-help`` from the source tree. | |
13 | ||
14 | The usual way to run these tests is: | |
15 | ||
16 | .. code:: | |
17 | ||
18 | make check | |
19 | ||
316082b1 TH |
20 | which includes QAPI schema tests, unit tests, QTests and some iotests. |
21 | Different sub-types of "make check" tests will be explained below. | |
4eb99560 FZ |
22 | |
23 | Before running tests, it is best to build QEMU programs first. Some tests | |
24 | expect the executables to exist and will fail with obscure messages if they | |
25 | cannot find them. | |
26 | ||
27 | Unit tests | |
16e79e1b | 28 | ~~~~~~~~~~ |
4eb99560 FZ |
29 | |
30 | Unit tests, which can be invoked with ``make check-unit``, are simple C tests | |
31 | that typically link to individual QEMU object files and exercise them by | |
32 | calling exported functions. | |
33 | ||
34 | If you are writing new code in QEMU, consider adding a unit test, especially | |
35 | for utility modules that are relatively stateless or have few dependencies. To | |
36 | add a new unit test: | |
37 | ||
8db5c3e2 | 38 | 1. Create a new source file. For example, ``tests/unit/foo-test.c``. |
4eb99560 FZ |
39 | |
40 | 2. Write the test. Normally you would include the header file which exports | |
41 | the module API, then verify the interface behaves as expected from your | |
42 | test. The test code should be organized with the glib testing framework. | |
43 | Copying and modifying an existing test is usually a good idea. | |
44 | ||
8db5c3e2 | 45 | 3. Add the test to ``tests/unit/meson.build``. The unit tests are listed in a |
bab88ead PB |
46 | dictionary called ``tests``. The values are any additional sources and |
47 | dependencies to be linked with the test. For a simple test whose source | |
8db5c3e2 | 48 | is in ``tests/unit/foo-test.c``, it is enough to add an entry like:: |
bab88ead PB |
49 | |
50 | { | |
51 | ... | |
52 | 'foo-test': [], | |
53 | ... | |
54 | } | |
4eb99560 FZ |
55 | |
56 | Since unit tests don't require environment variables, the simplest way to debug | |
57 | a unit test failure is often directly invoking it or even running it under | |
58 | ``gdb``. However there can still be differences in behavior between ``make`` | |
59 | invocations and your manual run, due to ``$MALLOC_PERTURB_`` environment | |
60 | variable (which affects memory reclamation and catches invalid pointers better) | |
61 | and gtester options. If necessary, you can run | |
62 | ||
63 | .. code:: | |
92970812 | 64 | |
4eb99560 FZ |
65 | make check-unit V=1 |
66 | ||
67 | and copy the actual command line which executes the unit test, then run | |
68 | it from the command line. | |
69 | ||
70 | QTest | |
16e79e1b | 71 | ~~~~~ |
4eb99560 FZ |
72 | |
73 | QTest is a device emulation testing framework. It can be very useful to test | |
74 | device models; it could also control certain aspects of QEMU (such as virtual | |
a738a50e EH |
75 | clock stepping), with a special purpose "qtest" protocol. Refer to |
76 | :doc:`qtest` for more details. | |
4eb99560 FZ |
77 | |
78 | QTest cases can be executed with | |
79 | ||
80 | .. code:: | |
81 | ||
82 | make check-qtest | |
83 | ||
4eb99560 | 84 | QAPI schema tests |
16e79e1b | 85 | ~~~~~~~~~~~~~~~~~ |
4eb99560 FZ |
86 | |
87 | The QAPI schema tests validate the QAPI parser used by QMP, by feeding | |
88 | predefined input to the parser and comparing the result with the reference | |
89 | output. | |
90 | ||
91 | The input/output data is managed under the ``tests/qapi-schema`` directory. | |
92 | Each test case includes four files that have a common base name: | |
93 | ||
94 | * ``${casename}.json`` - the file contains the JSON input for feeding the | |
95 | parser | |
96 | * ``${casename}.out`` - the file contains the expected stdout from the parser | |
97 | * ``${casename}.err`` - the file contains the expected stderr from the parser | |
98 | * ``${casename}.exit`` - the expected error code | |
99 | ||
100 | Consider adding a new QAPI schema test when you are making a change on the QAPI | |
101 | parser (either fixing a bug or extending/modifying the syntax). To do this: | |
102 | ||
103 | 1. Add four files for the new case as explained above. For example: | |
104 | ||
105 | ``$EDITOR tests/qapi-schema/foo.{json,out,err,exit}``. | |
106 | ||
107 | 2. Add the new test in ``tests/Makefile.include``. For example: | |
108 | ||
109 | ``qapi-schema += foo.json`` | |
110 | ||
111 | check-block | |
16e79e1b | 112 | ~~~~~~~~~~~ |
4eb99560 | 113 | |
316082b1 | 114 | ``make check-block`` runs a subset of the block layer iotests (the tests that |
b25a9488 | 115 | are in the "auto" group). |
316082b1 | 116 | See the "QEMU iotests" section below for more information. |
4eb99560 | 117 | |
4eb99560 | 118 | QEMU iotests |
16e79e1b | 119 | ------------ |
4eb99560 FZ |
120 | |
121 | QEMU iotests, under the directory ``tests/qemu-iotests``, is the testing | |
122 | framework widely used to test block layer related features. It is higher level | |
123 | than "make check" tests and 99% of the code is written in bash or Python | |
124 | scripts. The testing success criteria is golden output comparison, and the | |
125 | test files are named with numbers. | |
126 | ||
127 | To run iotests, make sure QEMU is built successfully, then switch to the | |
128 | ``tests/qemu-iotests`` directory under the build directory, and run ``./check`` | |
129 | with desired arguments from there. | |
130 | ||
131 | By default, "raw" format and "file" protocol is used; all tests will be | |
132 | executed, except the unsupported ones. You can override the format and protocol | |
133 | with arguments: | |
134 | ||
135 | .. code:: | |
136 | ||
137 | # test with qcow2 format | |
138 | ./check -qcow2 | |
139 | # or test a different protocol | |
140 | ./check -nbd | |
141 | ||
142 | It's also possible to list test numbers explicitly: | |
143 | ||
144 | .. code:: | |
145 | ||
146 | # run selected cases with qcow2 format | |
147 | ./check -qcow2 001 030 153 | |
148 | ||
149 | Cache mode can be selected with the "-c" option, which may help reveal bugs | |
150 | that are specific to certain cache mode. | |
151 | ||
152 | More options are supported by the ``./check`` script, run ``./check -h`` for | |
153 | help. | |
154 | ||
155 | Writing a new test case | |
16e79e1b | 156 | ~~~~~~~~~~~~~~~~~~~~~~~ |
4eb99560 FZ |
157 | |
158 | Consider writing a tests case when you are making any changes to the block | |
159 | layer. An iotest case is usually the choice for that. There are already many | |
160 | test cases, so it is possible that extending one of them may achieve the goal | |
161 | and save the boilerplate to create one. (Unfortunately, there isn't a 100% | |
162 | reliable way to find a related one out of hundreds of tests. One approach is | |
163 | using ``git grep``.) | |
164 | ||
165 | Usually an iotest case consists of two files. One is an executable that | |
166 | produces output to stdout and stderr, the other is the expected reference | |
167 | output. They are given the same number in file names. E.g. Test script ``055`` | |
168 | and reference output ``055.out``. | |
169 | ||
170 | In rare cases, when outputs differ between cache mode ``none`` and others, a | |
171 | ``.out.nocache`` file is added. In other cases, when outputs differ between | |
172 | image formats, more than one ``.out`` files are created ending with the | |
173 | respective format names, e.g. ``178.out.qcow2`` and ``178.out.raw``. | |
174 | ||
175 | There isn't a hard rule about how to write a test script, but a new test is | |
176 | usually a (copy and) modification of an existing case. There are a few | |
177 | commonly used ways to create a test: | |
178 | ||
179 | * A Bash script. It will make use of several environmental variables related | |
180 | to the testing procedure, and could source a group of ``common.*`` libraries | |
181 | for some common helper routines. | |
182 | ||
183 | * A Python unittest script. Import ``iotests`` and create a subclass of | |
184 | ``iotests.QMPTestCase``, then call ``iotests.main`` method. The downside of | |
185 | this approach is that the output is too scarce, and the script is considered | |
186 | harder to debug. | |
187 | ||
188 | * A simple Python script without using unittest module. This could also import | |
189 | ``iotests`` for launching QEMU and utilities etc, but it doesn't inherit | |
190 | from ``iotests.QMPTestCase`` therefore doesn't use the Python unittest | |
191 | execution. This is a combination of 1 and 2. | |
192 | ||
193 | Pick the language per your preference since both Bash and Python have | |
194 | comparable library support for invoking and interacting with QEMU programs. If | |
195 | you opt for Python, it is strongly recommended to write Python 3 compatible | |
196 | code. | |
197 | ||
f4a1b653 FZ |
198 | Both Python and Bash frameworks in iotests provide helpers to manage test |
199 | images. They can be used to create and clean up images under the test | |
200 | directory. If no I/O or any protocol specific feature is needed, it is often | |
201 | more convenient to use the pseudo block driver, ``null-co://``, as the test | |
202 | image, which doesn't require image creation or cleaning up. Avoid system-wide | |
203 | devices or files whenever possible, such as ``/dev/null`` or ``/dev/zero``. | |
204 | Otherwise, image locking implications have to be considered. For example, | |
205 | another application on the host may have locked the file, possibly leading to a | |
206 | test failure. If using such devices are explicitly desired, consider adding | |
207 | ``locking=off`` option to disable image locking. | |
208 | ||
0193767b | 209 | Debugging a test case |
16e79e1b PB |
210 | ~~~~~~~~~~~~~~~~~~~~~ |
211 | ||
0193767b EGE |
212 | The following options to the ``check`` script can be useful when debugging |
213 | a failing test: | |
214 | ||
e92ecc32 EGE |
215 | * ``-gdb`` wraps every QEMU invocation in a ``gdbserver``, which waits for a |
216 | connection from a gdb client. The options given to ``gdbserver`` (e.g. the | |
217 | address on which to listen for connections) are taken from the ``$GDB_OPTIONS`` | |
218 | environment variable. By default (if ``$GDB_OPTIONS`` is empty), it listens on | |
219 | ``localhost:12345``. | |
220 | It is possible to connect to it for example with | |
221 | ``gdb -iex "target remote $addr"``, where ``$addr`` is the address | |
222 | ``gdbserver`` listens on. | |
223 | If the ``-gdb`` option is not used, ``$GDB_OPTIONS`` is ignored, | |
224 | regardless of whether it is set or not. | |
225 | ||
bd10a739 EGE |
226 | * ``-valgrind`` attaches a valgrind instance to QEMU. If it detects |
227 | warnings, it will print and save the log in | |
228 | ``$TEST_DIR/<valgrind_pid>.valgrind``. | |
229 | The final command line will be ``valgrind --log-file=$TEST_DIR/ | |
230 | <valgrind_pid>.valgrind --error-exitcode=99 $QEMU ...`` | |
231 | ||
0193767b EGE |
232 | * ``-d`` (debug) just increases the logging verbosity, showing |
233 | for example the QMP commands and answers. | |
234 | ||
8ffcda2a EGE |
235 | * ``-p`` (print) redirects QEMU’s stdout and stderr to the test output, |
236 | instead of saving it into a log file in | |
237 | ``$TEST_DIR/qemu-machine-<random_string>``. | |
238 | ||
b25a9488 | 239 | Test case groups |
16e79e1b | 240 | ~~~~~~~~~~~~~~~~ |
b25a9488 VSO |
241 | |
242 | "Tests may belong to one or more test groups, which are defined in the form | |
243 | of a comment in the test source file. By convention, test groups are listed | |
244 | in the second line of the test file, after the "#!/..." line, like this: | |
245 | ||
246 | .. code:: | |
247 | ||
248 | #!/usr/bin/env python3 | |
249 | # group: auto quick | |
250 | # | |
251 | ... | |
252 | ||
253 | Another way of defining groups is creating the tests/qemu-iotests/group.local | |
254 | file. This should be used only for downstream (this file should never appear | |
255 | in upstream). This file may be used for defining some downstream test groups | |
256 | or for temporarily disabling tests, like this: | |
257 | ||
258 | .. code:: | |
259 | ||
260 | # groups for some company downstream process | |
261 | # | |
262 | # ci - tests to run on build | |
263 | # down - our downstream tests, not for upstream | |
264 | # | |
265 | # Format of each line is: | |
266 | # TEST_NAME TEST_GROUP [TEST_GROUP ]... | |
267 | ||
268 | 013 ci | |
269 | 210 disabled | |
270 | 215 disabled | |
271 | our-ugly-workaround-test down ci | |
272 | ||
273 | Note that the following group names have a special meaning: | |
274 | ||
275 | - quick: Tests in this group should finish within a few seconds. | |
276 | ||
277 | - auto: Tests in this group are used during "make check" and should be | |
278 | runnable in any case. That means they should run with every QEMU binary | |
279 | (also non-x86), with every QEMU configuration (i.e. must not fail if | |
280 | an optional feature is not compiled in - but reporting a "skip" is ok), | |
281 | work at least with the qcow2 file format, work with all kind of host | |
282 | filesystems and users (e.g. "nobody" or "root") and must not take too | |
283 | much memory and disk space (since CI pipelines tend to fail otherwise). | |
284 | ||
285 | - disabled: Tests in this group are disabled and ignored by check. | |
286 | ||
663a041e | 287 | .. _container-ref: |
f8ed349e | 288 | |
663a041e | 289 | Container based tests |
16e79e1b | 290 | --------------------- |
4eb99560 FZ |
291 | |
292 | Introduction | |
16e79e1b | 293 | ~~~~~~~~~~~~ |
4eb99560 | 294 | |
9c1f491e AB |
295 | The container testing framework in QEMU utilizes public images to |
296 | build and test QEMU in predefined and widely accessible Linux | |
297 | environments. This makes it possible to expand the test coverage | |
298 | across distros, toolchain flavors and library versions. The support | |
299 | was originally written for Docker although we also support Podman as | |
0285a0ec | 300 | an alternative container runtime. Although many of the target |
9c1f491e AB |
301 | names and scripts are prefixed with "docker" the system will |
302 | automatically run on whichever is configured. | |
303 | ||
4583cdad AB |
304 | The container images are also used to augment the generation of tests |
305 | for testing TCG. See :ref:`checktcg-ref` for more details. | |
306 | ||
9c1f491e | 307 | Docker Prerequisites |
16e79e1b | 308 | ~~~~~~~~~~~~~~~~~~~~ |
4eb99560 FZ |
309 | |
310 | Install "docker" with the system package manager and start the Docker service | |
311 | on your development machine, then make sure you have the privilege to run | |
312 | Docker commands. Typically it means setting up passwordless ``sudo docker`` | |
313 | command or login as root. For example: | |
314 | ||
315 | .. code:: | |
316 | ||
317 | $ sudo yum install docker | |
318 | $ # or `apt-get install docker` for Ubuntu, etc. | |
319 | $ sudo systemctl start docker | |
320 | $ sudo docker ps | |
321 | ||
322 | The last command should print an empty table, to verify the system is ready. | |
323 | ||
324 | An alternative method to set up permissions is by adding the current user to | |
325 | "docker" group and making the docker daemon socket file (by default | |
326 | ``/var/run/docker.sock``) accessible to the group: | |
327 | ||
328 | .. code:: | |
329 | ||
330 | $ sudo groupadd docker | |
29c33cc1 | 331 | $ sudo usermod $USER -a -G docker |
4eb99560 FZ |
332 | $ sudo chown :docker /var/run/docker.sock |
333 | ||
334 | Note that any one of above configurations makes it possible for the user to | |
335 | exploit the whole host with Docker bind mounting or other privileged | |
336 | operations. So only do it on development machines. | |
337 | ||
9c1f491e | 338 | Podman Prerequisites |
16e79e1b | 339 | ~~~~~~~~~~~~~~~~~~~~ |
9c1f491e AB |
340 | |
341 | Install "podman" with the system package manager. | |
342 | ||
343 | .. code:: | |
344 | ||
345 | $ sudo dnf install podman | |
346 | $ podman ps | |
347 | ||
348 | The last command should print an empty table, to verify the system is ready. | |
349 | ||
4eb99560 | 350 | Quickstart |
16e79e1b | 351 | ~~~~~~~~~~ |
4eb99560 | 352 | |
9c1f491e AB |
353 | From source tree, type ``make docker-help`` to see the help. Testing |
354 | can be started without configuring or building QEMU (``configure`` and | |
355 | ``make`` are done in the container, with parameters defined by the | |
356 | make target): | |
4eb99560 FZ |
357 | |
358 | .. code:: | |
359 | ||
9c1f491e | 360 | make docker-test-build@centos8 |
4eb99560 | 361 | |
9c1f491e | 362 | This will create a container instance using the ``centos8`` image (the image |
4eb99560 FZ |
363 | is downloaded and initialized automatically), in which the ``test-build`` job |
364 | is executed. | |
365 | ||
9c1f491e | 366 | Registry |
16e79e1b | 367 | ~~~~~~~~ |
9c1f491e AB |
368 | |
369 | The QEMU project has a container registry hosted by GitLab at | |
370 | ``registry.gitlab.com/qemu-project/qemu`` which will automatically be | |
371 | used to pull in pre-built layers. This avoids unnecessary strain on | |
372 | the distro archives created by multiple developers running the same | |
373 | container build steps over and over again. This can be overridden | |
374 | locally by using the ``NOCACHE`` build option: | |
375 | ||
376 | .. code:: | |
377 | ||
d996f0ae | 378 | make docker-image-debian-arm64-cross NOCACHE=1 |
9c1f491e | 379 | |
4eb99560 | 380 | Images |
16e79e1b | 381 | ~~~~~~ |
4eb99560 | 382 | |
9c1f491e AB |
383 | Along with many other images, the ``centos8`` image is defined in a Dockerfile |
384 | in ``tests/docker/dockerfiles/``, called ``centos8.docker``. ``make docker-help`` | |
4eb99560 FZ |
385 | command will list all the available images. |
386 | ||
4eb99560 FZ |
387 | A ``.pre`` script can be added beside the ``.docker`` file, which will be |
388 | executed before building the image under the build context directory. This is | |
389 | mainly used to do necessary host side setup. One such setup is ``binfmt_misc``, | |
390 | for example, to make qemu-user powered cross build containers work. | |
391 | ||
4ebb040f DB |
392 | Most of the existing Dockerfiles were written by hand, simply by creating a |
393 | a new ``.docker`` file under the ``tests/docker/dockerfiles/`` directory. | |
394 | This has led to an inconsistent set of packages being present across the | |
395 | different containers. | |
396 | ||
397 | Thus going forward, QEMU is aiming to automatically generate the Dockerfiles | |
398 | using the ``lcitool`` program provided by the ``libvirt-ci`` project: | |
399 | ||
400 | https://gitlab.com/libvirt/libvirt-ci | |
401 | ||
402 | In that project, there is a ``mappings.yml`` file defining the distro native | |
403 | package names for a wide variety of third party projects. This is processed | |
404 | in combination with a project defined list of build pre-requisites to determine | |
405 | the list of native packages to install on each distribution. This can be used | |
406 | to generate dockerfiles, VM package lists and Cirrus CI variables needed to | |
407 | setup build environments across OS distributions with a consistent set of | |
408 | packages present. | |
409 | ||
410 | When preparing a patch series that adds a new build pre-requisite to QEMU, | |
411 | updates to various lcitool data files may be required. | |
412 | ||
413 | ||
414 | Adding new build pre-requisites | |
415 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
416 | ||
417 | In the simple case where the pre-requisite is already known to ``libvirt-ci`` | |
418 | the following steps are needed | |
419 | ||
420 | * Edit ``tests/lcitool/projects/qemu.yml`` and add the pre-requisite | |
421 | ||
422 | * Run ``make lcitool-refresh`` to re-generate all relevant build environment | |
423 | manifests | |
424 | ||
425 | In some cases ``libvirt-ci`` will not know about the build pre-requisite and | |
426 | thus some extra preparation steps will be required first | |
427 | ||
428 | * Fork the ``libvirt-ci`` project on gitlab | |
429 | ||
430 | * Edit the ``mappings.yml`` change to add an entry for the new build | |
431 | prerequisite, listing its native package name on as many OS distros | |
432 | as practical. | |
433 | ||
434 | * Commit the ``mappings.yml`` change and submit a merge request to | |
435 | the ``libvirt-ci`` project, noting in the description that this | |
436 | is a new build pre-requisite desired for use with QEMU | |
437 | ||
438 | * CI pipeline will run to validate that the changes to ``mappings.yml`` | |
439 | are correct, by attempting to install the newly listed package on | |
440 | all OS distributions supported by ``libvirt-ci``. | |
441 | ||
442 | * Once the merge request is accepted, go back to QEMU and update | |
443 | the ``libvirt-ci`` submodule to point to a commit that contains | |
444 | the ``mappings.yml`` update. | |
445 | ||
446 | ||
447 | Adding new OS distros | |
448 | ^^^^^^^^^^^^^^^^^^^^^ | |
449 | ||
450 | In some cases ``libvirt-ci`` will not know about the OS distro that is | |
451 | desired to be tested. Before adding a new OS distro, discuss the proposed | |
452 | addition: | |
453 | ||
454 | * Send a mail to qemu-devel, copying people listed in the | |
455 | MAINTAINERS file for ``Build and test automation``. | |
456 | ||
457 | There are limited CI compute resources available to QEMU, so the | |
458 | cost/benefit tradeoff of adding new OS distros needs to be considered. | |
459 | ||
460 | * File an issue at https://gitlab.com/libvirt/libvirt-ci/-/issues | |
461 | pointing to the qemu-devel mail thread in the archives. | |
462 | ||
463 | This alerts other people who might be interested in the work | |
464 | to avoid duplication, as well as to get feedback from libvirt-ci | |
465 | maintainers on any tips to ease the addition | |
466 | ||
467 | Assuming there is agreement to add a new OS distro then | |
468 | ||
469 | * Fork the ``libvirt-ci`` project on gitlab | |
470 | ||
471 | * Add metadata under ``guests/lcitool/lcitool/ansible/group_vars/`` | |
472 | for the new OS distro. There might be code changes required if | |
473 | the OS distro uses a package format not currently known. The | |
474 | ``libvirt-ci`` maintainers can advise on this when the issue | |
475 | is file. | |
476 | ||
477 | * Edit the ``mappings.yml`` change to update all the existing package | |
478 | entries, providing details of the new OS distro | |
479 | ||
480 | * Commit the ``mappings.yml`` change and submit a merge request to | |
481 | the ``libvirt-ci`` project, noting in the description that this | |
482 | is a new build pre-requisite desired for use with QEMU | |
483 | ||
484 | * CI pipeline will run to validate that the changes to ``mappings.yml`` | |
485 | are correct, by attempting to install the newly listed package on | |
486 | all OS distributions supported by ``libvirt-ci``. | |
487 | ||
488 | * Once the merge request is accepted, go back to QEMU and update | |
489 | the ``libvirt-ci`` submodule to point to a commit that contains | |
490 | the ``mappings.yml`` update. | |
491 | ||
492 | ||
4eb99560 | 493 | Tests |
16e79e1b | 494 | ~~~~~ |
4eb99560 FZ |
495 | |
496 | Different tests are added to cover various configurations to build and test | |
497 | QEMU. Docker tests are the executables under ``tests/docker`` named | |
498 | ``test-*``. They are typically shell scripts and are built on top of a shell | |
499 | library, ``tests/docker/common.rc``, which provides helpers to find the QEMU | |
500 | source and build it. | |
501 | ||
9c1f491e | 502 | The full list of tests is printed in the ``make docker-help`` help. |
4eb99560 | 503 | |
4eb99560 | 504 | Debugging a Docker test failure |
16e79e1b | 505 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
4eb99560 FZ |
506 | |
507 | When CI tasks, maintainers or yourself report a Docker test failure, follow the | |
508 | below steps to debug it: | |
509 | ||
510 | 1. Locally reproduce the failure with the reported command line. E.g. run | |
511 | ``make docker-test-mingw@fedora J=8``. | |
512 | 2. Add "V=1" to the command line, try again, to see the verbose output. | |
513 | 3. Further add "DEBUG=1" to the command line. This will pause in a shell prompt | |
514 | in the container right before testing starts. You could either manually | |
515 | build QEMU and run tests from there, or press Ctrl-D to let the Docker | |
516 | testing continue. | |
517 | 4. If you press Ctrl-D, the same building and testing procedure will begin, and | |
518 | will hopefully run into the error again. After that, you will be dropped to | |
519 | the prompt for debug. | |
520 | ||
521 | Options | |
16e79e1b | 522 | ~~~~~~~ |
4eb99560 FZ |
523 | |
524 | Various options can be used to affect how Docker tests are done. The full | |
525 | list is in the ``make docker`` help text. The frequently used ones are: | |
526 | ||
527 | * ``V=1``: the same as in top level ``make``. It will be propagated to the | |
528 | container and enable verbose output. | |
529 | * ``J=$N``: the number of parallel tasks in make commands in the container, | |
530 | similar to the ``-j $N`` option in top level ``make``. (The ``-j`` option in | |
531 | top level ``make`` will not be propagated into the container.) | |
532 | * ``DEBUG=1``: enables debug. See the previous "Debugging a Docker test | |
533 | failure" section. | |
534 | ||
3b6882bd | 535 | Thread Sanitizer |
16e79e1b | 536 | ---------------- |
3b6882bd RF |
537 | |
538 | Thread Sanitizer (TSan) is a tool which can detect data races. QEMU supports | |
539 | building and testing with this tool. | |
540 | ||
541 | For more information on TSan: | |
542 | ||
543 | https://github.com/google/sanitizers/wiki/ThreadSanitizerCppManual | |
544 | ||
545 | Thread Sanitizer in Docker | |
16e79e1b | 546 | ~~~~~~~~~~~~~~~~~~~~~~~~~~ |
3b6882bd RF |
547 | TSan is currently supported in the ubuntu2004 docker. |
548 | ||
549 | The test-tsan test will build using TSan and then run make check. | |
550 | ||
551 | .. code:: | |
552 | ||
553 | make docker-test-tsan@ubuntu2004 | |
554 | ||
555 | TSan warnings under docker are placed in files located at build/tsan/. | |
556 | ||
557 | We recommend using DEBUG=1 to allow launching the test from inside the docker, | |
558 | and to allow review of the warnings generated by TSan. | |
559 | ||
560 | Building and Testing with TSan | |
16e79e1b | 561 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
3b6882bd RF |
562 | |
563 | It is possible to build and test with TSan, with a few additional steps. | |
564 | These steps are normally done automatically in the docker. | |
565 | ||
566 | There is a one time patch needed in clang-9 or clang-10 at this time: | |
567 | ||
568 | .. code:: | |
569 | ||
570 | sed -i 's/^const/static const/g' \ | |
571 | /usr/lib/llvm-10/lib/clang/10.0.0/include/sanitizer/tsan_interface.h | |
572 | ||
573 | To configure the build for TSan: | |
574 | ||
575 | .. code:: | |
576 | ||
577 | ../configure --enable-tsan --cc=clang-10 --cxx=clang++-10 \ | |
578 | --disable-werror --extra-cflags="-O0" | |
579 | ||
580 | The runtime behavior of TSAN is controlled by the TSAN_OPTIONS environment | |
581 | variable. | |
582 | ||
583 | More information on the TSAN_OPTIONS can be found here: | |
584 | ||
585 | https://github.com/google/sanitizers/wiki/ThreadSanitizerFlags | |
586 | ||
587 | For example: | |
588 | ||
589 | .. code:: | |
590 | ||
591 | export TSAN_OPTIONS=suppressions=<path to qemu>/tests/tsan/suppressions.tsan \ | |
592 | detect_deadlocks=false history_size=7 exitcode=0 \ | |
593 | log_path=<build path>/tsan/tsan_warning | |
594 | ||
595 | The above exitcode=0 has TSan continue without error if any warnings are found. | |
596 | This allows for running the test and then checking the warnings afterwards. | |
597 | If you want TSan to stop and exit with error on warnings, use exitcode=66. | |
598 | ||
599 | TSan Suppressions | |
16e79e1b | 600 | ~~~~~~~~~~~~~~~~~ |
3b6882bd RF |
601 | Keep in mind that for any data race warning, although there might be a data race |
602 | detected by TSan, there might be no actual bug here. TSan provides several | |
603 | different mechanisms for suppressing warnings. In general it is recommended | |
604 | to fix the code if possible to eliminate the data race rather than suppress | |
605 | the warning. | |
606 | ||
607 | A few important files for suppressing warnings are: | |
608 | ||
609 | tests/tsan/suppressions.tsan - Has TSan warnings we wish to suppress at runtime. | |
76ca4b58 | 610 | The comment on each suppression will typically indicate why we are |
3b6882bd RF |
611 | suppressing it. More information on the file format can be found here: |
612 | ||
613 | https://github.com/google/sanitizers/wiki/ThreadSanitizerSuppressions | |
614 | ||
615 | tests/tsan/blacklist.tsan - Has TSan warnings we wish to disable | |
616 | at compile time for test or debug. | |
617 | Add flags to configure to enable: | |
618 | ||
619 | "--extra-cflags=-fsanitize-blacklist=<src path>/tests/tsan/blacklist.tsan" | |
620 | ||
621 | More information on the file format can be found here under "Blacklist Format": | |
622 | ||
623 | https://github.com/google/sanitizers/wiki/ThreadSanitizerFlags | |
624 | ||
625 | TSan Annotations | |
16e79e1b | 626 | ~~~~~~~~~~~~~~~~ |
3b6882bd RF |
627 | include/qemu/tsan.h defines annotations. See this file for more descriptions |
628 | of the annotations themselves. Annotations can be used to suppress | |
629 | TSan warnings or give TSan more information so that it can detect proper | |
630 | relationships between accesses of data. | |
631 | ||
632 | Annotation examples can be found here: | |
633 | ||
634 | https://github.com/llvm/llvm-project/tree/master/compiler-rt/test/tsan/ | |
635 | ||
636 | Good files to start with are: annotate_happens_before.cpp and ignore_race.cpp | |
637 | ||
638 | The full set of annotations can be found here: | |
639 | ||
640 | https://github.com/llvm/llvm-project/blob/master/compiler-rt/lib/tsan/rtl/tsan_interface_ann.cpp | |
641 | ||
396408ee AB |
642 | docker-binfmt-image-debian-% targets |
643 | ------------------------------------ | |
644 | ||
645 | It is possible to combine Debian's bootstrap scripts with a configured | |
646 | ``binfmt_misc`` to bootstrap a number of Debian's distros including | |
647 | experimental ports not yet supported by a released OS. This can | |
648 | simplify setting up a rootfs by using docker to contain the foreign | |
649 | rootfs rather than manually invoking chroot. | |
650 | ||
651 | Setting up ``binfmt_misc`` | |
652 | ~~~~~~~~~~~~~~~~~~~~~~~~~~ | |
653 | ||
654 | You can use the script ``qemu-binfmt-conf.sh`` to configure a QEMU | |
655 | user binary to automatically run binaries for the foreign | |
656 | architecture. While the scripts will try their best to work with | |
657 | dynamically linked QEMU's a statically linked one will present less | |
658 | potential complications when copying into the docker image. Modern | |
659 | kernels support the ``F`` (fix binary) flag which will open the QEMU | |
660 | executable on setup and avoids the need to find and re-open in the | |
661 | chroot environment. This is triggered with the ``--persistent`` flag. | |
662 | ||
663 | Example invocation | |
664 | ~~~~~~~~~~~~~~~~~~ | |
665 | ||
666 | For example to setup the HPPA ports builds of Debian:: | |
667 | ||
668 | make docker-binfmt-image-debian-sid-hppa \ | |
669 | DEB_TYPE=sid DEB_ARCH=hppa \ | |
670 | DEB_URL=http://ftp.ports.debian.org/debian-ports/ \ | |
671 | DEB_KEYRING=/usr/share/keyrings/debian-ports-archive-keyring.gpg \ | |
672 | EXECUTABLE=(pwd)/qemu-hppa V=1 | |
673 | ||
674 | The ``DEB_`` variables are substitutions used by | |
675 | ``debian-boostrap.pre`` which is called to do the initial debootstrap | |
676 | of the rootfs before it is copied into the container. The second stage | |
677 | is run as part of the build. The final image will be tagged as | |
678 | ``qemu/debian-sid-hppa``. | |
679 | ||
4eb99560 | 680 | VM testing |
16e79e1b | 681 | ---------- |
4eb99560 FZ |
682 | |
683 | This test suite contains scripts that bootstrap various guest images that have | |
684 | necessary packages to build QEMU. The basic usage is documented in ``Makefile`` | |
4f2f6276 | 685 | help which is displayed with ``make vm-help``. |
4eb99560 FZ |
686 | |
687 | Quickstart | |
16e79e1b | 688 | ~~~~~~~~~~ |
4eb99560 | 689 | |
4f2f6276 | 690 | Run ``make vm-help`` to list available make targets. Invoke a specific make |
4eb99560 FZ |
691 | command to run build test in an image. For example, ``make vm-build-freebsd`` |
692 | will build the source tree in the FreeBSD image. The command can be executed | |
693 | from either the source tree or the build dir; if the former, ``./configure`` is | |
694 | not needed. The command will then generate the test image in ``./tests/vm/`` | |
695 | under the working directory. | |
696 | ||
697 | Note: images created by the scripts accept a well-known RSA key pair for SSH | |
698 | access, so they SHOULD NOT be exposed to external interfaces if you are | |
699 | concerned about attackers taking control of the guest and potentially | |
700 | exploiting a QEMU security bug to compromise the host. | |
701 | ||
1e48931c | 702 | QEMU binaries |
16e79e1b | 703 | ~~~~~~~~~~~~~ |
4eb99560 | 704 | |
c5ba6219 PMD |
705 | By default, ``qemu-system-x86_64`` is searched in $PATH to run the guest. If |
706 | there isn't one, or if it is older than 2.10, the test won't work. In this case, | |
4eb99560 FZ |
707 | provide the QEMU binary in env var: ``QEMU=/path/to/qemu-2.10+``. |
708 | ||
c5ba6219 | 709 | Likewise the path to ``qemu-img`` can be set in QEMU_IMG environment variable. |
1e48931c | 710 | |
4eb99560 | 711 | Make jobs |
16e79e1b | 712 | ~~~~~~~~~ |
4eb99560 FZ |
713 | |
714 | The ``-j$X`` option in the make command line is not propagated into the VM, | |
715 | specify ``J=$X`` to control the make jobs in the guest. | |
716 | ||
717 | Debugging | |
16e79e1b | 718 | ~~~~~~~~~ |
4eb99560 FZ |
719 | |
720 | Add ``DEBUG=1`` and/or ``V=1`` to the make command to allow interactive | |
721 | debugging and verbose output. If this is not enough, see the next section. | |
41e3340a | 722 | ``V=1`` will be propagated down into the make jobs in the guest. |
4eb99560 FZ |
723 | |
724 | Manual invocation | |
16e79e1b | 725 | ~~~~~~~~~~~~~~~~~ |
4eb99560 FZ |
726 | |
727 | Each guest script is an executable script with the same command line options. | |
728 | For example to work with the netbsd guest, use ``$QEMU_SRC/tests/vm/netbsd``: | |
729 | ||
730 | .. code:: | |
731 | ||
732 | $ cd $QEMU_SRC/tests/vm | |
733 | ||
734 | # To bootstrap the image | |
735 | $ ./netbsd --build-image --image /var/tmp/netbsd.img | |
736 | <...> | |
737 | ||
738 | # To run an arbitrary command in guest (the output will not be echoed unless | |
739 | # --debug is added) | |
740 | $ ./netbsd --debug --image /var/tmp/netbsd.img uname -a | |
741 | ||
742 | # To build QEMU in guest | |
743 | $ ./netbsd --debug --image /var/tmp/netbsd.img --build-qemu $QEMU_SRC | |
744 | ||
745 | # To get to an interactive shell | |
746 | $ ./netbsd --interactive --image /var/tmp/netbsd.img sh | |
747 | ||
748 | Adding new guests | |
16e79e1b | 749 | ~~~~~~~~~~~~~~~~~ |
4eb99560 FZ |
750 | |
751 | Please look at existing guest scripts for how to add new guests. | |
752 | ||
753 | Most importantly, create a subclass of BaseVM and implement ``build_image()`` | |
754 | method and define ``BUILD_SCRIPT``, then finally call ``basevm.main()`` from | |
755 | the script's ``main()``. | |
756 | ||
757 | * Usually in ``build_image()``, a template image is downloaded from a | |
758 | predefined URL. ``BaseVM._download_with_cache()`` takes care of the cache and | |
759 | the checksum, so consider using it. | |
760 | ||
761 | * Once the image is downloaded, users, SSH server and QEMU build deps should | |
762 | be set up: | |
763 | ||
764 | - Root password set to ``BaseVM.ROOT_PASS`` | |
765 | - User ``BaseVM.GUEST_USER`` is created, and password set to | |
766 | ``BaseVM.GUEST_PASS`` | |
767 | - SSH service is enabled and started on boot, | |
768 | ``$QEMU_SRC/tests/keys/id_rsa.pub`` is added to ssh's ``authorized_keys`` | |
769 | file of both root and the normal user | |
770 | - DHCP client service is enabled and started on boot, so that it can | |
771 | automatically configure the virtio-net-pci NIC and communicate with QEMU | |
772 | user net (10.0.2.2) | |
773 | - Necessary packages are installed to untar the source tarball and build | |
774 | QEMU | |
775 | ||
776 | * Write a proper ``BUILD_SCRIPT`` template, which should be a shell script that | |
777 | untars a raw virtio-blk block device, which is the tarball data blob of the | |
778 | QEMU source tree, then configure/build it. Running "make check" is also | |
779 | recommended. | |
780 | ||
781 | Image fuzzer testing | |
16e79e1b | 782 | -------------------- |
4eb99560 FZ |
783 | |
784 | An image fuzzer was added to exercise format drivers. Currently only qcow2 is | |
785 | supported. To start the fuzzer, run | |
786 | ||
787 | .. code:: | |
788 | ||
789 | tests/image-fuzzer/runner.py -c '[["qemu-img", "info", "$test_img"]]' /tmp/test qcow2 | |
790 | ||
c5ba6219 | 791 | Alternatively, some command different from ``qemu-img info`` can be tested, by |
4eb99560 | 792 | changing the ``-c`` option. |
c3d7e8c9 | 793 | |
bbbd9b6e WR |
794 | Integration tests using the Avocado Framework |
795 | --------------------------------------------- | |
c3d7e8c9 | 796 | |
bbbd9b6e WR |
797 | The ``tests/avocado`` directory hosts integration tests. They're usually |
798 | higher level tests, and may interact with external resources and with | |
799 | various guest operating systems. | |
c3d7e8c9 CR |
800 | |
801 | These tests are written using the Avocado Testing Framework (which must | |
802 | be installed separately) in conjunction with a the ``avocado_qemu.Test`` | |
bbbd9b6e | 803 | class, implemented at ``tests/avocado/avocado_qemu``. |
c3d7e8c9 CR |
804 | |
805 | Tests based on ``avocado_qemu.Test`` can easily: | |
806 | ||
807 | * Customize the command line arguments given to the convenience | |
808 | ``self.vm`` attribute (a QEMUMachine instance) | |
809 | ||
810 | * Interact with the QEMU monitor, send QMP commands and check | |
811 | their results | |
812 | ||
813 | * Interact with the guest OS, using the convenience console device | |
814 | (which may be useful to assert the effectiveness and correctness of | |
815 | command line arguments or QMP commands) | |
816 | ||
817 | * Interact with external data files that accompany the test itself | |
818 | (see ``self.get_data()``) | |
819 | ||
820 | * Download (and cache) remote data files, such as firmware and kernel | |
821 | images | |
822 | ||
823 | * Have access to a library of guest OS images (by means of the | |
824 | ``avocado.utils.vmimage`` library) | |
825 | ||
826 | * Make use of various other test related utilities available at the | |
827 | test class itself and at the utility library: | |
828 | ||
829 | - http://avocado-framework.readthedocs.io/en/latest/api/test/avocado.html#avocado.Test | |
830 | - http://avocado-framework.readthedocs.io/en/latest/api/utils/avocado.utils.html | |
831 | ||
a56931ee | 832 | Running tests |
16e79e1b | 833 | ~~~~~~~~~~~~~ |
a56931ee | 834 | |
bbbd9b6e | 835 | You can run the avocado tests simply by executing: |
a56931ee CR |
836 | |
837 | .. code:: | |
838 | ||
bbbd9b6e | 839 | make check-avocado |
a56931ee CR |
840 | |
841 | This involves the automatic creation of Python virtual environment | |
842 | within the build tree (at ``tests/venv``) which will have all the | |
843 | right dependencies, and will save tests results also within the | |
844 | build tree (at ``tests/results``). | |
c3d7e8c9 | 845 | |
a56931ee CR |
846 | Note: the build environment must be using a Python 3 stack, and have |
847 | the ``venv`` and ``pip`` packages installed. If necessary, make sure | |
848 | ``configure`` is called with ``--python=`` and that those modules are | |
849 | available. On Debian and Ubuntu based systems, depending on the | |
850 | specific version, they may be on packages named ``python3-venv`` and | |
851 | ``python3-pip``. | |
852 | ||
23022794 | 853 | It is also possible to run tests based on tags using the |
bbbd9b6e | 854 | ``make check-avocado`` command and the ``AVOCADO_TAGS`` environment |
23022794 WR |
855 | variable: |
856 | ||
857 | .. code:: | |
858 | ||
bbbd9b6e | 859 | make check-avocado AVOCADO_TAGS=quick |
23022794 WR |
860 | |
861 | Note that tags separated with commas have an AND behavior, while tags | |
862 | separated by spaces have an OR behavior. For more information on Avocado | |
863 | tags, see: | |
864 | ||
865 | https://avocado-framework.readthedocs.io/en/latest/guides/user/chapters/tags.html | |
866 | ||
94c71462 | 867 | To run a single test file, a couple of them, or a test within a file |
bbbd9b6e | 868 | using the ``make check-avocado`` command, set the ``AVOCADO_TESTS`` |
94c71462 WR |
869 | environment variable with the test files or test names. To run all |
870 | tests from a single file, use: | |
871 | ||
872 | .. code:: | |
873 | ||
bbbd9b6e | 874 | make check-avocado AVOCADO_TESTS=$FILEPATH |
94c71462 WR |
875 | |
876 | The same is valid to run tests from multiple test files: | |
877 | ||
878 | .. code:: | |
879 | ||
bbbd9b6e | 880 | make check-avocado AVOCADO_TESTS='$FILEPATH1 $FILEPATH2' |
94c71462 WR |
881 | |
882 | To run a single test within a file, use: | |
883 | ||
884 | .. code:: | |
885 | ||
bbbd9b6e | 886 | make check-avocado AVOCADO_TESTS=$FILEPATH:$TESTCLASS.$TESTNAME |
94c71462 WR |
887 | |
888 | The same is valid to run single tests from multiple test files: | |
889 | ||
890 | .. code:: | |
891 | ||
bbbd9b6e | 892 | make check-avocado AVOCADO_TESTS='$FILEPATH1:$TESTCLASS1.$TESTNAME1 $FILEPATH2:$TESTCLASS2.$TESTNAME2' |
94c71462 | 893 | |
a56931ee CR |
894 | The scripts installed inside the virtual environment may be used |
895 | without an "activation". For instance, the Avocado test runner | |
896 | may be invoked by running: | |
897 | ||
898 | .. code:: | |
899 | ||
bbbd9b6e | 900 | tests/venv/bin/avocado run $OPTION1 $OPTION2 tests/avocado/ |
a56931ee | 901 | |
bbbd9b6e | 902 | Note that if ``make check-avocado`` was not executed before, it is |
6676f18f WR |
903 | possible to create the Python virtual environment with the dependencies |
904 | needed running: | |
905 | ||
906 | .. code:: | |
907 | ||
908 | make check-venv | |
909 | ||
910 | It is also possible to run tests from a single file or a single test within | |
911 | a test file. To run tests from a single file within the build tree, use: | |
912 | ||
913 | .. code:: | |
914 | ||
bbbd9b6e | 915 | tests/venv/bin/avocado run tests/avocado/$TESTFILE |
6676f18f WR |
916 | |
917 | To run a single test within a test file, use: | |
918 | ||
919 | .. code:: | |
920 | ||
bbbd9b6e | 921 | tests/venv/bin/avocado run tests/avocado/$TESTFILE:$TESTCLASS.$TESTNAME |
6676f18f WR |
922 | |
923 | Valid test names are visible in the output from any previous execution | |
bbbd9b6e | 924 | of Avocado or ``make check-avocado``, and can also be queried using: |
6676f18f WR |
925 | |
926 | .. code:: | |
927 | ||
bbbd9b6e | 928 | tests/venv/bin/avocado list tests/avocado |
6676f18f | 929 | |
a56931ee | 930 | Manual Installation |
16e79e1b | 931 | ~~~~~~~~~~~~~~~~~~~ |
a56931ee CR |
932 | |
933 | To manually install Avocado and its dependencies, run: | |
c3d7e8c9 CR |
934 | |
935 | .. code:: | |
936 | ||
937 | pip install --user avocado-framework | |
938 | ||
939 | Alternatively, follow the instructions on this link: | |
940 | ||
4c9ac672 | 941 | https://avocado-framework.readthedocs.io/en/latest/guides/user/chapters/installing.html |
c3d7e8c9 CR |
942 | |
943 | Overview | |
16e79e1b | 944 | ~~~~~~~~ |
c3d7e8c9 | 945 | |
bbbd9b6e | 946 | The ``tests/avocado/avocado_qemu`` directory provides the |
805fac52 CR |
947 | ``avocado_qemu`` Python module, containing the ``avocado_qemu.Test`` |
948 | class. Here's a simple usage example: | |
c3d7e8c9 CR |
949 | |
950 | .. code:: | |
951 | ||
2283b627 | 952 | from avocado_qemu import QemuSystemTest |
c3d7e8c9 CR |
953 | |
954 | ||
2283b627 | 955 | class Version(QemuSystemTest): |
c3d7e8c9 | 956 | """ |
c3d7e8c9 CR |
957 | :avocado: tags=quick |
958 | """ | |
959 | def test_qmp_human_info_version(self): | |
960 | self.vm.launch() | |
961 | res = self.vm.command('human-monitor-command', | |
962 | command_line='info version') | |
963 | self.assertRegexpMatches(res, r'^(\d+\.\d+\.\d)') | |
964 | ||
965 | To execute your test, run: | |
966 | ||
967 | .. code:: | |
968 | ||
969 | avocado run version.py | |
970 | ||
971 | Tests may be classified according to a convention by using docstring | |
972 | directives such as ``:avocado: tags=TAG1,TAG2``. To run all tests | |
973 | in the current directory, tagged as "quick", run: | |
974 | ||
975 | .. code:: | |
976 | ||
977 | avocado run -t quick . | |
978 | ||
979 | The ``avocado_qemu.Test`` base test class | |
16e79e1b | 980 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
c3d7e8c9 CR |
981 | |
982 | The ``avocado_qemu.Test`` class has a number of characteristics that | |
983 | are worth being mentioned right away. | |
984 | ||
985 | First of all, it attempts to give each test a ready to use QEMUMachine | |
986 | instance, available at ``self.vm``. Because many tests will tweak the | |
987 | QEMU command line, launching the QEMUMachine (by using ``self.vm.launch()``) | |
988 | is left to the test writer. | |
989 | ||
b7287d42 CC |
990 | The base test class has also support for tests with more than one |
991 | QEMUMachine. The way to get machines is through the ``self.get_vm()`` | |
992 | method which will return a QEMUMachine instance. The ``self.get_vm()`` | |
993 | method accepts arguments that will be passed to the QEMUMachine creation | |
1e235eda | 994 | and also an optional ``name`` attribute so you can identify a specific |
b7287d42 CC |
995 | machine and get it more than once through the tests methods. A simple |
996 | and hypothetical example follows: | |
997 | ||
998 | .. code:: | |
999 | ||
2283b627 | 1000 | from avocado_qemu import QemuSystemTest |
b7287d42 CC |
1001 | |
1002 | ||
2283b627 | 1003 | class MultipleMachines(QemuSystemTest): |
b7287d42 CC |
1004 | def test_multiple_machines(self): |
1005 | first_machine = self.get_vm() | |
1006 | second_machine = self.get_vm() | |
1007 | self.get_vm(name='third_machine').launch() | |
1008 | ||
1009 | first_machine.launch() | |
1010 | second_machine.launch() | |
1011 | ||
1012 | first_res = first_machine.command( | |
1013 | 'human-monitor-command', | |
1014 | command_line='info version') | |
1015 | ||
1016 | second_res = second_machine.command( | |
1017 | 'human-monitor-command', | |
1018 | command_line='info version') | |
1019 | ||
1020 | third_res = self.get_vm(name='third_machine').command( | |
1021 | 'human-monitor-command', | |
1022 | command_line='info version') | |
1023 | ||
1024 | self.assertEquals(first_res, second_res, third_res) | |
1025 | ||
1026 | At test "tear down", ``avocado_qemu.Test`` handles all the QEMUMachines | |
c3d7e8c9 CR |
1027 | shutdown. |
1028 | ||
1e4e7efa | 1029 | The ``avocado_qemu.LinuxTest`` base test class |
16e79e1b | 1030 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
1e4e7efa CR |
1031 | |
1032 | The ``avocado_qemu.LinuxTest`` is further specialization of the | |
1033 | ``avocado_qemu.Test`` class, so it contains all the characteristics of | |
1034 | the later plus some extra features. | |
1035 | ||
1036 | First of all, this base class is intended for tests that need to | |
1037 | interact with a fully booted and operational Linux guest. At this | |
1038 | time, it uses a Fedora 31 guest image. The most basic example looks | |
1039 | like this: | |
1040 | ||
1041 | .. code:: | |
1042 | ||
1043 | from avocado_qemu import LinuxTest | |
1044 | ||
1045 | ||
1046 | class SomeTest(LinuxTest): | |
1047 | ||
1048 | def test(self): | |
1049 | self.launch_and_wait() | |
1050 | self.ssh_command('some_command_to_be_run_in_the_guest') | |
1051 | ||
1052 | Please refer to tests that use ``avocado_qemu.LinuxTest`` under | |
bbbd9b6e | 1053 | ``tests/avocado`` for more examples. |
1e4e7efa | 1054 | |
c3d7e8c9 CR |
1055 | QEMUMachine |
1056 | ~~~~~~~~~~~ | |
1057 | ||
1058 | The QEMUMachine API is already widely used in the Python iotests, | |
1059 | device-crash-test and other Python scripts. It's a wrapper around the | |
1060 | execution of a QEMU binary, giving its users: | |
1061 | ||
1062 | * the ability to set command line arguments to be given to the QEMU | |
1063 | binary | |
1064 | ||
1065 | * a ready to use QMP connection and interface, which can be used to | |
1066 | send commands and inspect its results, as well as asynchronous | |
1067 | events | |
1068 | ||
1069 | * convenience methods to set commonly used command line arguments in | |
1070 | a more succinct and intuitive way | |
1071 | ||
1072 | QEMU binary selection | |
16e79e1b | 1073 | ^^^^^^^^^^^^^^^^^^^^^ |
c3d7e8c9 CR |
1074 | |
1075 | The QEMU binary used for the ``self.vm`` QEMUMachine instance will | |
1076 | primarily depend on the value of the ``qemu_bin`` parameter. If it's | |
1077 | not explicitly set, its default value will be the result of a dynamic | |
1078 | probe in the same source tree. A suitable binary will be one that | |
1079 | targets the architecture matching host machine. | |
1080 | ||
1081 | Based on this description, test writers will usually rely on one of | |
1082 | the following approaches: | |
1083 | ||
1084 | 1) Set ``qemu_bin``, and use the given binary | |
1085 | ||
1086 | 2) Do not set ``qemu_bin``, and use a QEMU binary named like | |
64ed6f92 | 1087 | "qemu-system-${arch}", either in the current |
c3d7e8c9 CR |
1088 | working directory, or in the current source tree. |
1089 | ||
1090 | The resulting ``qemu_bin`` value will be preserved in the | |
1091 | ``avocado_qemu.Test`` as an attribute with the same name. | |
1092 | ||
1093 | Attribute reference | |
16e79e1b PB |
1094 | ~~~~~~~~~~~~~~~~~~~ |
1095 | ||
1096 | Test | |
1097 | ^^^^ | |
c3d7e8c9 CR |
1098 | |
1099 | Besides the attributes and methods that are part of the base | |
1100 | ``avocado.Test`` class, the following attributes are available on any | |
1101 | ``avocado_qemu.Test`` instance. | |
1102 | ||
1103 | vm | |
16e79e1b | 1104 | '' |
c3d7e8c9 CR |
1105 | |
1106 | A QEMUMachine instance, initially configured according to the given | |
1107 | ``qemu_bin`` parameter. | |
1108 | ||
2c44d68f | 1109 | arch |
16e79e1b | 1110 | '''' |
2c44d68f CR |
1111 | |
1112 | The architecture can be used on different levels of the stack, e.g. by | |
1113 | the framework or by the test itself. At the framework level, it will | |
1114 | currently influence the selection of a QEMU binary (when one is not | |
1115 | explicitly given). | |
1116 | ||
1117 | Tests are also free to use this attribute value, for their own needs. | |
1118 | A test may, for instance, use the same value when selecting the | |
1119 | architecture of a kernel or disk image to boot a VM with. | |
1120 | ||
1121 | The ``arch`` attribute will be set to the test parameter of the same | |
b910545f CR |
1122 | name. If one is not given explicitly, it will either be set to |
1123 | ``None``, or, if the test is tagged with one (and only one) | |
1124 | ``:avocado: tags=arch:VALUE`` tag, it will be set to ``VALUE``. | |
2c44d68f | 1125 | |
20bbf846 | 1126 | cpu |
16e79e1b | 1127 | ''' |
20bbf846 WSM |
1128 | |
1129 | The cpu model that will be set to all QEMUMachine instances created | |
1130 | by the test. | |
1131 | ||
1132 | The ``cpu`` attribute will be set to the test parameter of the same | |
1133 | name. If one is not given explicitly, it will either be set to | |
1134 | ``None ``, or, if the test is tagged with one (and only one) | |
1135 | ``:avocado: tags=cpu:VALUE`` tag, it will be set to ``VALUE``. | |
1136 | ||
ba21bde9 | 1137 | machine |
16e79e1b | 1138 | ''''''' |
ba21bde9 CR |
1139 | |
1140 | The machine type that will be set to all QEMUMachine instances created | |
1141 | by the test. | |
1142 | ||
1143 | The ``machine`` attribute will be set to the test parameter of the same | |
1144 | name. If one is not given explicitly, it will either be set to | |
1145 | ``None``, or, if the test is tagged with one (and only one) | |
1146 | ``:avocado: tags=machine:VALUE`` tag, it will be set to ``VALUE``. | |
1147 | ||
c3d7e8c9 | 1148 | qemu_bin |
16e79e1b | 1149 | '''''''' |
c3d7e8c9 CR |
1150 | |
1151 | The preserved value of the ``qemu_bin`` parameter or the result of the | |
1152 | dynamic probe for a QEMU binary in the current working directory or | |
1153 | source tree. | |
1154 | ||
d5adf9d5 | 1155 | LinuxTest |
16e79e1b | 1156 | ^^^^^^^^^ |
d5adf9d5 CR |
1157 | |
1158 | Besides the attributes present on the ``avocado_qemu.Test`` base | |
1159 | class, the ``avocado_qemu.LinuxTest`` adds the following attributes: | |
1160 | ||
1161 | distro | |
16e79e1b | 1162 | '''''' |
d5adf9d5 CR |
1163 | |
1164 | The name of the Linux distribution used as the guest image for the | |
1165 | test. The name should match the **Provider** column on the list | |
1166 | of images supported by the avocado.utils.vmimage library: | |
1167 | ||
1168 | https://avocado-framework.readthedocs.io/en/latest/guides/writer/libs/vmimage.html#supported-images | |
1169 | ||
1170 | distro_version | |
16e79e1b | 1171 | '''''''''''''' |
d5adf9d5 CR |
1172 | |
1173 | The version of the Linux distribution as the guest image for the | |
1174 | test. The name should match the **Version** column on the list | |
1175 | of images supported by the avocado.utils.vmimage library: | |
1176 | ||
1177 | https://avocado-framework.readthedocs.io/en/latest/guides/writer/libs/vmimage.html#supported-images | |
1178 | ||
1179 | distro_checksum | |
16e79e1b | 1180 | ''''''''''''''' |
d5adf9d5 CR |
1181 | |
1182 | The sha256 hash of the guest image file used for the test. | |
1183 | ||
1184 | If this value is not set in the code or by a test parameter (with the | |
1185 | same name), no validation on the integrity of the image will be | |
1186 | performed. | |
1187 | ||
c3d7e8c9 | 1188 | Parameter reference |
16e79e1b | 1189 | ~~~~~~~~~~~~~~~~~~~ |
c3d7e8c9 CR |
1190 | |
1191 | To understand how Avocado parameters are accessed by tests, and how | |
1192 | they can be passed to tests, please refer to:: | |
1193 | ||
4c9ac672 | 1194 | https://avocado-framework.readthedocs.io/en/latest/guides/writer/chapters/writing.html#accessing-test-parameters |
c3d7e8c9 CR |
1195 | |
1196 | Parameter values can be easily seen in the log files, and will look | |
1197 | like the following: | |
1198 | ||
1199 | .. code:: | |
1200 | ||
64ed6f92 | 1201 | PARAMS (key=qemu_bin, path=*, default=./qemu-system-x86_64) => './qemu-system-x86_64 |
c3d7e8c9 | 1202 | |
16e79e1b PB |
1203 | Test |
1204 | ^^^^ | |
1205 | ||
2c44d68f | 1206 | arch |
16e79e1b | 1207 | '''' |
2c44d68f CR |
1208 | |
1209 | The architecture that will influence the selection of a QEMU binary | |
1210 | (when one is not explicitly given). | |
1211 | ||
1212 | Tests are also free to use this parameter value, for their own needs. | |
1213 | A test may, for instance, use the same value when selecting the | |
1214 | architecture of a kernel or disk image to boot a VM with. | |
1215 | ||
1216 | This parameter has a direct relation with the ``arch`` attribute. If | |
1217 | not given, it will default to None. | |
1218 | ||
20bbf846 | 1219 | cpu |
16e79e1b | 1220 | ''' |
20bbf846 WSM |
1221 | |
1222 | The cpu model that will be set to all QEMUMachine instances created | |
1223 | by the test. | |
1224 | ||
ba21bde9 | 1225 | machine |
16e79e1b | 1226 | ''''''' |
ba21bde9 CR |
1227 | |
1228 | The machine type that will be set to all QEMUMachine instances created | |
1229 | by the test. | |
1230 | ||
c3d7e8c9 | 1231 | qemu_bin |
16e79e1b | 1232 | '''''''' |
c3d7e8c9 CR |
1233 | |
1234 | The exact QEMU binary to be used on QEMUMachine. | |
1235 | ||
d5adf9d5 | 1236 | LinuxTest |
16e79e1b | 1237 | ^^^^^^^^^ |
d5adf9d5 CR |
1238 | |
1239 | Besides the parameters present on the ``avocado_qemu.Test`` base | |
1240 | class, the ``avocado_qemu.LinuxTest`` adds the following parameters: | |
1241 | ||
1242 | distro | |
16e79e1b | 1243 | '''''' |
d5adf9d5 CR |
1244 | |
1245 | The name of the Linux distribution used as the guest image for the | |
1246 | test. The name should match the **Provider** column on the list | |
1247 | of images supported by the avocado.utils.vmimage library: | |
1248 | ||
1249 | https://avocado-framework.readthedocs.io/en/latest/guides/writer/libs/vmimage.html#supported-images | |
1250 | ||
1251 | distro_version | |
16e79e1b | 1252 | '''''''''''''' |
d5adf9d5 CR |
1253 | |
1254 | The version of the Linux distribution as the guest image for the | |
1255 | test. The name should match the **Version** column on the list | |
1256 | of images supported by the avocado.utils.vmimage library: | |
1257 | ||
1258 | https://avocado-framework.readthedocs.io/en/latest/guides/writer/libs/vmimage.html#supported-images | |
1259 | ||
1260 | distro_checksum | |
16e79e1b | 1261 | ''''''''''''''' |
d5adf9d5 CR |
1262 | |
1263 | The sha256 hash of the guest image file used for the test. | |
1264 | ||
1265 | If this value is not set in the code or by this parameter no | |
1266 | validation on the integrity of the image will be performed. | |
1267 | ||
cf5891ec | 1268 | Skipping tests |
16e79e1b PB |
1269 | ~~~~~~~~~~~~~~ |
1270 | ||
cf5891ec WSM |
1271 | The Avocado framework provides Python decorators which allow for easily skip |
1272 | tests running under certain conditions. For example, on the lack of a binary | |
1273 | on the test system or when the running environment is a CI system. For further | |
1274 | information about those decorators, please refer to:: | |
1275 | ||
1276 | https://avocado-framework.readthedocs.io/en/latest/guides/writer/chapters/writing.html#skipping-tests | |
1277 | ||
1278 | While the conditions for skipping tests are often specifics of each one, there | |
1279 | are recurring scenarios identified by the QEMU developers and the use of | |
1280 | environment variables became a kind of standard way to enable/disable tests. | |
1281 | ||
1282 | Here is a list of the most used variables: | |
1283 | ||
1284 | AVOCADO_ALLOW_LARGE_STORAGE | |
16e79e1b | 1285 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
cf5891ec | 1286 | Tests which are going to fetch or produce assets considered *large* are not |
1e235eda | 1287 | going to run unless that ``AVOCADO_ALLOW_LARGE_STORAGE=1`` is exported on |
cf5891ec WSM |
1288 | the environment. |
1289 | ||
1290 | The definition of *large* is a bit arbitrary here, but it usually means an | |
1291 | asset which occupies at least 1GB of size on disk when uncompressed. | |
1292 | ||
1293 | AVOCADO_ALLOW_UNTRUSTED_CODE | |
16e79e1b | 1294 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
cf5891ec WSM |
1295 | There are tests which will boot a kernel image or firmware that can be |
1296 | considered not safe to run on the developer's workstation, thus they are | |
1297 | skipped by default. The definition of *not safe* is also arbitrary but | |
1298 | usually it means a blob which either its source or build process aren't | |
1299 | public available. | |
1300 | ||
1e235eda | 1301 | You should export ``AVOCADO_ALLOW_UNTRUSTED_CODE=1`` on the environment in |
cf5891ec WSM |
1302 | order to allow tests which make use of those kind of assets. |
1303 | ||
1304 | AVOCADO_TIMEOUT_EXPECTED | |
16e79e1b | 1305 | ^^^^^^^^^^^^^^^^^^^^^^^^ |
cf5891ec WSM |
1306 | The Avocado framework has a timeout mechanism which interrupts tests to avoid the |
1307 | test suite of getting stuck. The timeout value can be set via test parameter or | |
1308 | property defined in the test class, for further details:: | |
1309 | ||
1310 | https://avocado-framework.readthedocs.io/en/latest/guides/writer/chapters/writing.html#setting-a-test-timeout | |
1311 | ||
1312 | Even though the timeout can be set by the test developer, there are some tests | |
1313 | that may not have a well-defined limit of time to finish under certain | |
1314 | conditions. For example, tests that take longer to execute when QEMU is | |
1e235eda | 1315 | compiled with debug flags. Therefore, the ``AVOCADO_TIMEOUT_EXPECTED`` variable |
cf5891ec WSM |
1316 | has been used to determine whether those tests should run or not. |
1317 | ||
1318 | GITLAB_CI | |
16e79e1b | 1319 | ^^^^^^^^^ |
cf5891ec WSM |
1320 | A number of tests are flagged to not run on the GitLab CI. Usually because |
1321 | they proved to the flaky or there are constraints on the CI environment which | |
1322 | would make them fail. If you encounter a similar situation then use that | |
1323 | variable as shown on the code snippet below to skip the test: | |
1324 | ||
1325 | .. code:: | |
1326 | ||
1327 | @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab') | |
1328 | def test(self): | |
1329 | do_something() | |
1330 | ||
c3d7e8c9 | 1331 | Uninstalling Avocado |
16e79e1b | 1332 | ~~~~~~~~~~~~~~~~~~~~ |
c3d7e8c9 | 1333 | |
a56931ee CR |
1334 | If you've followed the manual installation instructions above, you can |
1335 | easily uninstall Avocado. Start by listing the packages you have | |
1336 | installed:: | |
c3d7e8c9 CR |
1337 | |
1338 | pip list --user | |
1339 | ||
1340 | And remove any package you want with:: | |
1341 | ||
1342 | pip uninstall <package_name> | |
a56931ee | 1343 | |
bbbd9b6e | 1344 | If you've used ``make check-avocado``, the Python virtual environment where |
a56931ee | 1345 | Avocado is installed will be cleaned up as part of ``make check-clean``. |
f8ed349e | 1346 | |
4583cdad AB |
1347 | .. _checktcg-ref: |
1348 | ||
f8ed349e | 1349 | Testing with "make check-tcg" |
16e79e1b | 1350 | ----------------------------- |
f8ed349e AB |
1351 | |
1352 | The check-tcg tests are intended for simple smoke tests of both | |
1353 | linux-user and softmmu TCG functionality. However to build test | |
1354 | programs for guest targets you need to have cross compilers available. | |
1355 | If your distribution supports cross compilers you can do something as | |
1356 | simple as:: | |
1357 | ||
1358 | apt install gcc-aarch64-linux-gnu | |
1359 | ||
1360 | The configure script will automatically pick up their presence. | |
1361 | Sometimes compilers have slightly odd names so the availability of | |
1362 | them can be prompted by passing in the appropriate configure option | |
1363 | for the architecture in question, for example:: | |
1364 | ||
1365 | $(configure) --cross-cc-aarch64=aarch64-cc | |
1366 | ||
479ca4cc | 1367 | There is also a ``--cross-cc-cflags-ARCH`` flag in case additional |
f8ed349e AB |
1368 | compiler flags are needed to build for a given target. |
1369 | ||
663a041e AB |
1370 | If you have the ability to run containers as the user the build system |
1371 | will automatically use them where no system compiler is available. For | |
1372 | architectures where we also support building QEMU we will generally | |
1373 | use the same container to build tests. However there are a number of | |
1374 | additional containers defined that have a minimal cross-build | |
1375 | environment that is only suitable for building test cases. Sometimes | |
1376 | we may use a bleeding edge distribution for compiler features needed | |
1377 | for test cases that aren't yet in the LTS distros we support for QEMU | |
1378 | itself. | |
1379 | ||
1380 | See :ref:`container-ref` for more details. | |
f8ed349e AB |
1381 | |
1382 | Running subset of tests | |
16e79e1b | 1383 | ~~~~~~~~~~~~~~~~~~~~~~~ |
f8ed349e AB |
1384 | |
1385 | You can build the tests for one architecture:: | |
1386 | ||
1387 | make build-tcg-tests-$TARGET | |
1388 | ||
1389 | And run with:: | |
1390 | ||
1391 | make run-tcg-tests-$TARGET | |
1392 | ||
1393 | Adding ``V=1`` to the invocation will show the details of how to | |
1394 | invoke QEMU for the test which is useful for debugging tests. | |
1395 | ||
1396 | TCG test dependencies | |
16e79e1b | 1397 | ~~~~~~~~~~~~~~~~~~~~~ |
f8ed349e AB |
1398 | |
1399 | The TCG tests are deliberately very light on dependencies and are | |
1400 | either totally bare with minimal gcc lib support (for softmmu tests) | |
1401 | or just glibc (for linux-user tests). This is because getting a cross | |
1402 | compiler to work with additional libraries can be challenging. | |
1403 | ||
1404 | Other TCG Tests | |
1405 | --------------- | |
1406 | ||
1407 | There are a number of out-of-tree test suites that are used for more | |
1408 | extensive testing of processor features. | |
1409 | ||
1410 | KVM Unit Tests | |
1411 | ~~~~~~~~~~~~~~ | |
1412 | ||
1413 | The KVM unit tests are designed to run as a Guest OS under KVM but | |
1414 | there is no reason why they can't exercise the TCG as well. It | |
1415 | provides a minimal OS kernel with hooks for enabling the MMU as well | |
1416 | as reporting test results via a special device:: | |
1417 | ||
1418 | https://git.kernel.org/pub/scm/virt/kvm/kvm-unit-tests.git | |
1419 | ||
1420 | Linux Test Project | |
1421 | ~~~~~~~~~~~~~~~~~~ | |
1422 | ||
1423 | The LTP is focused on exercising the syscall interface of a Linux | |
1424 | kernel. It checks that syscalls behave as documented and strives to | |
1425 | exercise as many corner cases as possible. It is a useful test suite | |
1426 | to run to exercise QEMU's linux-user code:: | |
1427 | ||
1428 | https://linux-test-project.github.io/ | |
9fce3601 PB |
1429 | |
1430 | GCC gcov support | |
16e79e1b | 1431 | ---------------- |
9fce3601 PB |
1432 | |
1433 | ``gcov`` is a GCC tool to analyze the testing coverage by | |
1434 | instrumenting the tested code. To use it, configure QEMU with | |
1435 | ``--enable-gcov`` option and build. Then run the tests as usual. | |
1436 | ||
1437 | If you want to gather coverage information on a single test the ``make | |
1438 | clean-gcda`` target can be used to delete any existing coverage | |
1439 | information before running a single test. | |
1440 | ||
1441 | You can generate a HTML coverage report by executing ``make | |
1442 | coverage-html`` which will create | |
1443 | ``meson-logs/coveragereport/index.html``. | |
1444 | ||
1445 | Further analysis can be conducted by running the ``gcov`` command | |
1446 | directly on the various .gcda output files. Please read the ``gcov`` | |
1447 | documentation for more information. |