]>
Commit | Line | Data |
---|---|---|
7266ecce AB |
1 | .. _testing: |
2 | ||
4eb99560 FZ |
3 | Testing in QEMU |
4 | =============== | |
5 | ||
6 | This document describes the testing infrastructure in QEMU. | |
7 | ||
8 | Testing with "make check" | |
16e79e1b | 9 | ------------------------- |
4eb99560 FZ |
10 | |
11 | The "make check" testing family includes most of the C based tests in QEMU. For | |
12 | a quick help, run ``make check-help`` from the source tree. | |
13 | ||
14 | The usual way to run these tests is: | |
15 | ||
16 | .. code:: | |
17 | ||
18 | make check | |
19 | ||
316082b1 TH |
20 | which includes QAPI schema tests, unit tests, QTests and some iotests. |
21 | Different sub-types of "make check" tests will be explained below. | |
4eb99560 FZ |
22 | |
23 | Before running tests, it is best to build QEMU programs first. Some tests | |
24 | expect the executables to exist and will fail with obscure messages if they | |
25 | cannot find them. | |
26 | ||
27 | Unit tests | |
16e79e1b | 28 | ~~~~~~~~~~ |
4eb99560 FZ |
29 | |
30 | Unit tests, which can be invoked with ``make check-unit``, are simple C tests | |
31 | that typically link to individual QEMU object files and exercise them by | |
32 | calling exported functions. | |
33 | ||
34 | If you are writing new code in QEMU, consider adding a unit test, especially | |
35 | for utility modules that are relatively stateless or have few dependencies. To | |
36 | add a new unit test: | |
37 | ||
8db5c3e2 | 38 | 1. Create a new source file. For example, ``tests/unit/foo-test.c``. |
4eb99560 FZ |
39 | |
40 | 2. Write the test. Normally you would include the header file which exports | |
41 | the module API, then verify the interface behaves as expected from your | |
42 | test. The test code should be organized with the glib testing framework. | |
43 | Copying and modifying an existing test is usually a good idea. | |
44 | ||
8db5c3e2 | 45 | 3. Add the test to ``tests/unit/meson.build``. The unit tests are listed in a |
bab88ead PB |
46 | dictionary called ``tests``. The values are any additional sources and |
47 | dependencies to be linked with the test. For a simple test whose source | |
8db5c3e2 | 48 | is in ``tests/unit/foo-test.c``, it is enough to add an entry like:: |
bab88ead PB |
49 | |
50 | { | |
51 | ... | |
52 | 'foo-test': [], | |
53 | ... | |
54 | } | |
4eb99560 FZ |
55 | |
56 | Since unit tests don't require environment variables, the simplest way to debug | |
57 | a unit test failure is often directly invoking it or even running it under | |
58 | ``gdb``. However there can still be differences in behavior between ``make`` | |
59 | invocations and your manual run, due to ``$MALLOC_PERTURB_`` environment | |
60 | variable (which affects memory reclamation and catches invalid pointers better) | |
61 | and gtester options. If necessary, you can run | |
62 | ||
63 | .. code:: | |
92970812 | 64 | |
4eb99560 FZ |
65 | make check-unit V=1 |
66 | ||
67 | and copy the actual command line which executes the unit test, then run | |
68 | it from the command line. | |
69 | ||
70 | QTest | |
16e79e1b | 71 | ~~~~~ |
4eb99560 FZ |
72 | |
73 | QTest is a device emulation testing framework. It can be very useful to test | |
74 | device models; it could also control certain aspects of QEMU (such as virtual | |
a738a50e EH |
75 | clock stepping), with a special purpose "qtest" protocol. Refer to |
76 | :doc:`qtest` for more details. | |
4eb99560 FZ |
77 | |
78 | QTest cases can be executed with | |
79 | ||
80 | .. code:: | |
81 | ||
82 | make check-qtest | |
83 | ||
0b49bc1b BM |
84 | Writing portable test cases |
85 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~ | |
86 | Both unit tests and qtests can run on POSIX hosts as well as Windows hosts. | |
87 | Care must be taken when writing portable test cases that can be built and run | |
88 | successfully on various hosts. The following list shows some best practices: | |
89 | ||
90 | * Use portable APIs from glib whenever necessary, e.g.: g_setenv(), | |
91 | g_mkdtemp(), g_mkdir(). | |
92 | * Avoid using hardcoded /tmp for temporary file directory. | |
93 | Use g_get_tmp_dir() instead. | |
94 | * Bear in mind that Windows has different special string representation for | |
95 | stdin/stdout/stderr and null devices. For example if your test case uses | |
96 | "/dev/fd/2" and "/dev/null" on Linux, remember to use "2" and "nul" on | |
97 | Windows instead. Also IO redirection does not work on Windows, so avoid | |
98 | using "2>nul" whenever necessary. | |
99 | * If your test cases uses the blkdebug feature, use relative path to pass | |
100 | the config and image file paths in the command line as Windows absolute | |
101 | path contains the delimiter ":" which will confuse the blkdebug parser. | |
1e458f11 | 102 | * Use double quotes in your extra QEMU command line in your test cases |
0b49bc1b BM |
103 | instead of single quotes, as Windows does not drop single quotes when |
104 | passing the command line to QEMU. | |
105 | * Windows opens a file in text mode by default, while a POSIX compliant | |
106 | implementation treats text files and binary files the same. So if your | |
107 | test cases opens a file to write some data and later wants to compare the | |
108 | written data with the original one, be sure to pass the letter 'b' as | |
109 | part of the mode string to fopen(), or O_BINARY flag for the open() call. | |
110 | * If a certain test case can only run on POSIX or Linux hosts, use a proper | |
111 | #ifdef in the codes. If the whole test suite cannot run on Windows, disable | |
112 | the build in the meson.build file. | |
113 | ||
4eb99560 | 114 | QAPI schema tests |
16e79e1b | 115 | ~~~~~~~~~~~~~~~~~ |
4eb99560 FZ |
116 | |
117 | The QAPI schema tests validate the QAPI parser used by QMP, by feeding | |
118 | predefined input to the parser and comparing the result with the reference | |
119 | output. | |
120 | ||
121 | The input/output data is managed under the ``tests/qapi-schema`` directory. | |
122 | Each test case includes four files that have a common base name: | |
123 | ||
124 | * ``${casename}.json`` - the file contains the JSON input for feeding the | |
125 | parser | |
126 | * ``${casename}.out`` - the file contains the expected stdout from the parser | |
127 | * ``${casename}.err`` - the file contains the expected stderr from the parser | |
128 | * ``${casename}.exit`` - the expected error code | |
129 | ||
130 | Consider adding a new QAPI schema test when you are making a change on the QAPI | |
131 | parser (either fixing a bug or extending/modifying the syntax). To do this: | |
132 | ||
133 | 1. Add four files for the new case as explained above. For example: | |
134 | ||
135 | ``$EDITOR tests/qapi-schema/foo.{json,out,err,exit}``. | |
136 | ||
137 | 2. Add the new test in ``tests/Makefile.include``. For example: | |
138 | ||
139 | ``qapi-schema += foo.json`` | |
140 | ||
141 | check-block | |
16e79e1b | 142 | ~~~~~~~~~~~ |
4eb99560 | 143 | |
316082b1 | 144 | ``make check-block`` runs a subset of the block layer iotests (the tests that |
b25a9488 | 145 | are in the "auto" group). |
316082b1 | 146 | See the "QEMU iotests" section below for more information. |
4eb99560 | 147 | |
4eb99560 | 148 | QEMU iotests |
16e79e1b | 149 | ------------ |
4eb99560 FZ |
150 | |
151 | QEMU iotests, under the directory ``tests/qemu-iotests``, is the testing | |
152 | framework widely used to test block layer related features. It is higher level | |
153 | than "make check" tests and 99% of the code is written in bash or Python | |
154 | scripts. The testing success criteria is golden output comparison, and the | |
155 | test files are named with numbers. | |
156 | ||
157 | To run iotests, make sure QEMU is built successfully, then switch to the | |
158 | ``tests/qemu-iotests`` directory under the build directory, and run ``./check`` | |
159 | with desired arguments from there. | |
160 | ||
161 | By default, "raw" format and "file" protocol is used; all tests will be | |
162 | executed, except the unsupported ones. You can override the format and protocol | |
163 | with arguments: | |
164 | ||
165 | .. code:: | |
166 | ||
167 | # test with qcow2 format | |
168 | ./check -qcow2 | |
169 | # or test a different protocol | |
170 | ./check -nbd | |
171 | ||
172 | It's also possible to list test numbers explicitly: | |
173 | ||
174 | .. code:: | |
175 | ||
176 | # run selected cases with qcow2 format | |
177 | ./check -qcow2 001 030 153 | |
178 | ||
179 | Cache mode can be selected with the "-c" option, which may help reveal bugs | |
180 | that are specific to certain cache mode. | |
181 | ||
182 | More options are supported by the ``./check`` script, run ``./check -h`` for | |
183 | help. | |
184 | ||
185 | Writing a new test case | |
16e79e1b | 186 | ~~~~~~~~~~~~~~~~~~~~~~~ |
4eb99560 FZ |
187 | |
188 | Consider writing a tests case when you are making any changes to the block | |
189 | layer. An iotest case is usually the choice for that. There are already many | |
190 | test cases, so it is possible that extending one of them may achieve the goal | |
191 | and save the boilerplate to create one. (Unfortunately, there isn't a 100% | |
192 | reliable way to find a related one out of hundreds of tests. One approach is | |
193 | using ``git grep``.) | |
194 | ||
195 | Usually an iotest case consists of two files. One is an executable that | |
196 | produces output to stdout and stderr, the other is the expected reference | |
197 | output. They are given the same number in file names. E.g. Test script ``055`` | |
198 | and reference output ``055.out``. | |
199 | ||
200 | In rare cases, when outputs differ between cache mode ``none`` and others, a | |
201 | ``.out.nocache`` file is added. In other cases, when outputs differ between | |
202 | image formats, more than one ``.out`` files are created ending with the | |
203 | respective format names, e.g. ``178.out.qcow2`` and ``178.out.raw``. | |
204 | ||
205 | There isn't a hard rule about how to write a test script, but a new test is | |
206 | usually a (copy and) modification of an existing case. There are a few | |
207 | commonly used ways to create a test: | |
208 | ||
209 | * A Bash script. It will make use of several environmental variables related | |
210 | to the testing procedure, and could source a group of ``common.*`` libraries | |
211 | for some common helper routines. | |
212 | ||
213 | * A Python unittest script. Import ``iotests`` and create a subclass of | |
214 | ``iotests.QMPTestCase``, then call ``iotests.main`` method. The downside of | |
215 | this approach is that the output is too scarce, and the script is considered | |
216 | harder to debug. | |
217 | ||
218 | * A simple Python script without using unittest module. This could also import | |
219 | ``iotests`` for launching QEMU and utilities etc, but it doesn't inherit | |
220 | from ``iotests.QMPTestCase`` therefore doesn't use the Python unittest | |
221 | execution. This is a combination of 1 and 2. | |
222 | ||
223 | Pick the language per your preference since both Bash and Python have | |
224 | comparable library support for invoking and interacting with QEMU programs. If | |
225 | you opt for Python, it is strongly recommended to write Python 3 compatible | |
226 | code. | |
227 | ||
f4a1b653 FZ |
228 | Both Python and Bash frameworks in iotests provide helpers to manage test |
229 | images. They can be used to create and clean up images under the test | |
230 | directory. If no I/O or any protocol specific feature is needed, it is often | |
231 | more convenient to use the pseudo block driver, ``null-co://``, as the test | |
232 | image, which doesn't require image creation or cleaning up. Avoid system-wide | |
233 | devices or files whenever possible, such as ``/dev/null`` or ``/dev/zero``. | |
234 | Otherwise, image locking implications have to be considered. For example, | |
235 | another application on the host may have locked the file, possibly leading to a | |
236 | test failure. If using such devices are explicitly desired, consider adding | |
237 | ``locking=off`` option to disable image locking. | |
238 | ||
0193767b | 239 | Debugging a test case |
16e79e1b PB |
240 | ~~~~~~~~~~~~~~~~~~~~~ |
241 | ||
0193767b EGE |
242 | The following options to the ``check`` script can be useful when debugging |
243 | a failing test: | |
244 | ||
e92ecc32 EGE |
245 | * ``-gdb`` wraps every QEMU invocation in a ``gdbserver``, which waits for a |
246 | connection from a gdb client. The options given to ``gdbserver`` (e.g. the | |
247 | address on which to listen for connections) are taken from the ``$GDB_OPTIONS`` | |
248 | environment variable. By default (if ``$GDB_OPTIONS`` is empty), it listens on | |
249 | ``localhost:12345``. | |
250 | It is possible to connect to it for example with | |
251 | ``gdb -iex "target remote $addr"``, where ``$addr`` is the address | |
252 | ``gdbserver`` listens on. | |
253 | If the ``-gdb`` option is not used, ``$GDB_OPTIONS`` is ignored, | |
254 | regardless of whether it is set or not. | |
255 | ||
bd10a739 EGE |
256 | * ``-valgrind`` attaches a valgrind instance to QEMU. If it detects |
257 | warnings, it will print and save the log in | |
258 | ``$TEST_DIR/<valgrind_pid>.valgrind``. | |
259 | The final command line will be ``valgrind --log-file=$TEST_DIR/ | |
260 | <valgrind_pid>.valgrind --error-exitcode=99 $QEMU ...`` | |
261 | ||
0193767b EGE |
262 | * ``-d`` (debug) just increases the logging verbosity, showing |
263 | for example the QMP commands and answers. | |
264 | ||
8ffcda2a EGE |
265 | * ``-p`` (print) redirects QEMU’s stdout and stderr to the test output, |
266 | instead of saving it into a log file in | |
267 | ``$TEST_DIR/qemu-machine-<random_string>``. | |
268 | ||
b25a9488 | 269 | Test case groups |
16e79e1b | 270 | ~~~~~~~~~~~~~~~~ |
b25a9488 VSO |
271 | |
272 | "Tests may belong to one or more test groups, which are defined in the form | |
273 | of a comment in the test source file. By convention, test groups are listed | |
274 | in the second line of the test file, after the "#!/..." line, like this: | |
275 | ||
276 | .. code:: | |
277 | ||
278 | #!/usr/bin/env python3 | |
279 | # group: auto quick | |
280 | # | |
281 | ... | |
282 | ||
283 | Another way of defining groups is creating the tests/qemu-iotests/group.local | |
284 | file. This should be used only for downstream (this file should never appear | |
285 | in upstream). This file may be used for defining some downstream test groups | |
286 | or for temporarily disabling tests, like this: | |
287 | ||
288 | .. code:: | |
289 | ||
290 | # groups for some company downstream process | |
291 | # | |
292 | # ci - tests to run on build | |
293 | # down - our downstream tests, not for upstream | |
294 | # | |
295 | # Format of each line is: | |
296 | # TEST_NAME TEST_GROUP [TEST_GROUP ]... | |
297 | ||
298 | 013 ci | |
299 | 210 disabled | |
300 | 215 disabled | |
301 | our-ugly-workaround-test down ci | |
302 | ||
303 | Note that the following group names have a special meaning: | |
304 | ||
305 | - quick: Tests in this group should finish within a few seconds. | |
306 | ||
307 | - auto: Tests in this group are used during "make check" and should be | |
308 | runnable in any case. That means they should run with every QEMU binary | |
309 | (also non-x86), with every QEMU configuration (i.e. must not fail if | |
310 | an optional feature is not compiled in - but reporting a "skip" is ok), | |
311 | work at least with the qcow2 file format, work with all kind of host | |
312 | filesystems and users (e.g. "nobody" or "root") and must not take too | |
313 | much memory and disk space (since CI pipelines tend to fail otherwise). | |
314 | ||
315 | - disabled: Tests in this group are disabled and ignored by check. | |
316 | ||
663a041e | 317 | .. _container-ref: |
f8ed349e | 318 | |
663a041e | 319 | Container based tests |
16e79e1b | 320 | --------------------- |
4eb99560 FZ |
321 | |
322 | Introduction | |
16e79e1b | 323 | ~~~~~~~~~~~~ |
4eb99560 | 324 | |
9c1f491e AB |
325 | The container testing framework in QEMU utilizes public images to |
326 | build and test QEMU in predefined and widely accessible Linux | |
327 | environments. This makes it possible to expand the test coverage | |
328 | across distros, toolchain flavors and library versions. The support | |
329 | was originally written for Docker although we also support Podman as | |
0285a0ec | 330 | an alternative container runtime. Although many of the target |
9c1f491e AB |
331 | names and scripts are prefixed with "docker" the system will |
332 | automatically run on whichever is configured. | |
333 | ||
4583cdad AB |
334 | The container images are also used to augment the generation of tests |
335 | for testing TCG. See :ref:`checktcg-ref` for more details. | |
336 | ||
9c1f491e | 337 | Docker Prerequisites |
16e79e1b | 338 | ~~~~~~~~~~~~~~~~~~~~ |
4eb99560 FZ |
339 | |
340 | Install "docker" with the system package manager and start the Docker service | |
341 | on your development machine, then make sure you have the privilege to run | |
342 | Docker commands. Typically it means setting up passwordless ``sudo docker`` | |
343 | command or login as root. For example: | |
344 | ||
345 | .. code:: | |
346 | ||
347 | $ sudo yum install docker | |
348 | $ # or `apt-get install docker` for Ubuntu, etc. | |
349 | $ sudo systemctl start docker | |
350 | $ sudo docker ps | |
351 | ||
352 | The last command should print an empty table, to verify the system is ready. | |
353 | ||
354 | An alternative method to set up permissions is by adding the current user to | |
355 | "docker" group and making the docker daemon socket file (by default | |
356 | ``/var/run/docker.sock``) accessible to the group: | |
357 | ||
358 | .. code:: | |
359 | ||
360 | $ sudo groupadd docker | |
29c33cc1 | 361 | $ sudo usermod $USER -a -G docker |
4eb99560 FZ |
362 | $ sudo chown :docker /var/run/docker.sock |
363 | ||
364 | Note that any one of above configurations makes it possible for the user to | |
365 | exploit the whole host with Docker bind mounting or other privileged | |
366 | operations. So only do it on development machines. | |
367 | ||
9c1f491e | 368 | Podman Prerequisites |
16e79e1b | 369 | ~~~~~~~~~~~~~~~~~~~~ |
9c1f491e AB |
370 | |
371 | Install "podman" with the system package manager. | |
372 | ||
373 | .. code:: | |
374 | ||
375 | $ sudo dnf install podman | |
376 | $ podman ps | |
377 | ||
378 | The last command should print an empty table, to verify the system is ready. | |
379 | ||
4eb99560 | 380 | Quickstart |
16e79e1b | 381 | ~~~~~~~~~~ |
4eb99560 | 382 | |
9c1f491e AB |
383 | From source tree, type ``make docker-help`` to see the help. Testing |
384 | can be started without configuring or building QEMU (``configure`` and | |
385 | ``make`` are done in the container, with parameters defined by the | |
386 | make target): | |
4eb99560 FZ |
387 | |
388 | .. code:: | |
389 | ||
9c1f491e | 390 | make docker-test-build@centos8 |
4eb99560 | 391 | |
9c1f491e | 392 | This will create a container instance using the ``centos8`` image (the image |
4eb99560 FZ |
393 | is downloaded and initialized automatically), in which the ``test-build`` job |
394 | is executed. | |
395 | ||
9c1f491e | 396 | Registry |
16e79e1b | 397 | ~~~~~~~~ |
9c1f491e AB |
398 | |
399 | The QEMU project has a container registry hosted by GitLab at | |
400 | ``registry.gitlab.com/qemu-project/qemu`` which will automatically be | |
401 | used to pull in pre-built layers. This avoids unnecessary strain on | |
402 | the distro archives created by multiple developers running the same | |
403 | container build steps over and over again. This can be overridden | |
404 | locally by using the ``NOCACHE`` build option: | |
405 | ||
406 | .. code:: | |
407 | ||
d996f0ae | 408 | make docker-image-debian-arm64-cross NOCACHE=1 |
9c1f491e | 409 | |
4eb99560 | 410 | Images |
16e79e1b | 411 | ~~~~~~ |
4eb99560 | 412 | |
9c1f491e AB |
413 | Along with many other images, the ``centos8`` image is defined in a Dockerfile |
414 | in ``tests/docker/dockerfiles/``, called ``centos8.docker``. ``make docker-help`` | |
4eb99560 FZ |
415 | command will list all the available images. |
416 | ||
4eb99560 FZ |
417 | A ``.pre`` script can be added beside the ``.docker`` file, which will be |
418 | executed before building the image under the build context directory. This is | |
419 | mainly used to do necessary host side setup. One such setup is ``binfmt_misc``, | |
420 | for example, to make qemu-user powered cross build containers work. | |
421 | ||
4ebb040f DB |
422 | Most of the existing Dockerfiles were written by hand, simply by creating a |
423 | a new ``.docker`` file under the ``tests/docker/dockerfiles/`` directory. | |
424 | This has led to an inconsistent set of packages being present across the | |
425 | different containers. | |
426 | ||
427 | Thus going forward, QEMU is aiming to automatically generate the Dockerfiles | |
428 | using the ``lcitool`` program provided by the ``libvirt-ci`` project: | |
429 | ||
430 | https://gitlab.com/libvirt/libvirt-ci | |
431 | ||
fa1ce1dd PB |
432 | ``libvirt-ci`` contains an ``lcitool`` program as well as a list of |
433 | mappings to distribution package names for a wide variety of third | |
434 | party projects. ``lcitool`` applies the mappings to a list of build | |
435 | pre-requisites in ``tests/lcitool/projects/qemu.yml``, determines the | |
436 | list of native packages to install on each distribution, and uses them | |
437 | to generate build environments (dockerfiles and Cirrus CI variable files) | |
438 | that are consistent across OS distribution. | |
4ebb040f DB |
439 | |
440 | ||
441 | Adding new build pre-requisites | |
442 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
443 | ||
fa1ce1dd PB |
444 | When preparing a patch series that adds a new build |
445 | pre-requisite to QEMU, the prerequisites should to be added to | |
446 | ``tests/lcitool/projects/qemu.yml`` in order to make the dependency | |
447 | available in the CI build environments. | |
448 | ||
4ebb040f | 449 | In the simple case where the pre-requisite is already known to ``libvirt-ci`` |
fa1ce1dd | 450 | the following steps are needed: |
4ebb040f DB |
451 | |
452 | * Edit ``tests/lcitool/projects/qemu.yml`` and add the pre-requisite | |
453 | ||
454 | * Run ``make lcitool-refresh`` to re-generate all relevant build environment | |
455 | manifests | |
456 | ||
fa1ce1dd PB |
457 | It may be that ``libvirt-ci`` does not know about the new pre-requisite. |
458 | If that is the case, some extra preparation steps will be required | |
459 | first to contribute the mapping to the ``libvirt-ci`` project: | |
4ebb040f DB |
460 | |
461 | * Fork the ``libvirt-ci`` project on gitlab | |
462 | ||
fa1ce1dd PB |
463 | * Add an entry for the new build prerequisite to |
464 | ``lcitool/facts/mappings.yml``, listing its native package name on as | |
465 | many OS distros as practical. Run ``python -m pytest --regenerate-output`` | |
466 | and check that the changes are correct. | |
4ebb040f | 467 | |
fa1ce1dd PB |
468 | * Commit the ``mappings.yml`` change together with the regenerated test |
469 | files, and submit a merge request to the ``libvirt-ci`` project. | |
470 | Please note in the description that this is a new build pre-requisite | |
471 | desired for use with QEMU. | |
4ebb040f DB |
472 | |
473 | * CI pipeline will run to validate that the changes to ``mappings.yml`` | |
474 | are correct, by attempting to install the newly listed package on | |
475 | all OS distributions supported by ``libvirt-ci``. | |
476 | ||
477 | * Once the merge request is accepted, go back to QEMU and update | |
fa1ce1dd PB |
478 | the ``tests/lcitool/libvirt-ci`` submodule to point to a commit that |
479 | contains the ``mappings.yml`` update. Then add the prerequisite and | |
480 | run ``make lcitool-refresh``. | |
4ebb040f | 481 | |
d2ab529e AS |
482 | * Please also trigger gitlab container generation pipelines on your change |
483 | for as many OS distros as practical to make sure that there are no | |
484 | obvious breakages when adding the new pre-requisite. Please see | |
485 | `CI <https://www.qemu.org/docs/master/devel/ci.html>`__ documentation | |
486 | page on how to trigger gitlab CI pipelines on your change. | |
487 | ||
2a851fca AS |
488 | * Please also trigger gitlab container generation pipelines on your change |
489 | for as many OS distros as practical to make sure that there are no | |
490 | obvious breakages when adding the new pre-requisite. Please see | |
491 | `CI <https://www.qemu.org/docs/master/devel/ci.html>`__ documentation | |
492 | page on how to trigger gitlab CI pipelines on your change. | |
493 | ||
32c06131 PB |
494 | For enterprise distros that default to old, end-of-life versions of the |
495 | Python runtime, QEMU uses a separate set of mappings that work with more | |
496 | recent versions. These can be found in ``tests/lcitool/mappings.yml``. | |
497 | Modifying this file should not be necessary unless the new pre-requisite | |
498 | is a Python library or tool. | |
499 | ||
4ebb040f DB |
500 | |
501 | Adding new OS distros | |
502 | ^^^^^^^^^^^^^^^^^^^^^ | |
503 | ||
504 | In some cases ``libvirt-ci`` will not know about the OS distro that is | |
505 | desired to be tested. Before adding a new OS distro, discuss the proposed | |
506 | addition: | |
507 | ||
508 | * Send a mail to qemu-devel, copying people listed in the | |
509 | MAINTAINERS file for ``Build and test automation``. | |
510 | ||
511 | There are limited CI compute resources available to QEMU, so the | |
512 | cost/benefit tradeoff of adding new OS distros needs to be considered. | |
513 | ||
514 | * File an issue at https://gitlab.com/libvirt/libvirt-ci/-/issues | |
515 | pointing to the qemu-devel mail thread in the archives. | |
516 | ||
517 | This alerts other people who might be interested in the work | |
518 | to avoid duplication, as well as to get feedback from libvirt-ci | |
519 | maintainers on any tips to ease the addition | |
520 | ||
521 | Assuming there is agreement to add a new OS distro then | |
522 | ||
523 | * Fork the ``libvirt-ci`` project on gitlab | |
524 | ||
fa1ce1dd PB |
525 | * Add metadata under ``lcitool/facts/targets/`` for the new OS |
526 | distro. There might be code changes required if the OS distro | |
527 | uses a package format not currently known. The ``libvirt-ci`` | |
528 | maintainers can advise on this when the issue is filed. | |
4ebb040f | 529 | |
fa1ce1dd PB |
530 | * Edit the ``lcitool/facts/mappings.yml`` change to add entries for |
531 | the new OS, listing the native package names for as many packages | |
532 | as practical. Run ``python -m pytest --regenerate-output`` and | |
533 | check that the changes are correct. | |
4ebb040f | 534 | |
fa1ce1dd PB |
535 | * Commit the changes to ``lcitool/facts`` and the regenerated test |
536 | files, and submit a merge request to the ``libvirt-ci`` project. | |
537 | Please note in the description that this is a new build pre-requisite | |
538 | desired for use with QEMU | |
4ebb040f DB |
539 | |
540 | * CI pipeline will run to validate that the changes to ``mappings.yml`` | |
541 | are correct, by attempting to install the newly listed package on | |
542 | all OS distributions supported by ``libvirt-ci``. | |
543 | ||
544 | * Once the merge request is accepted, go back to QEMU and update | |
545 | the ``libvirt-ci`` submodule to point to a commit that contains | |
546 | the ``mappings.yml`` update. | |
547 | ||
548 | ||
4eb99560 | 549 | Tests |
16e79e1b | 550 | ~~~~~ |
4eb99560 FZ |
551 | |
552 | Different tests are added to cover various configurations to build and test | |
553 | QEMU. Docker tests are the executables under ``tests/docker`` named | |
554 | ``test-*``. They are typically shell scripts and are built on top of a shell | |
555 | library, ``tests/docker/common.rc``, which provides helpers to find the QEMU | |
556 | source and build it. | |
557 | ||
9c1f491e | 558 | The full list of tests is printed in the ``make docker-help`` help. |
4eb99560 | 559 | |
4eb99560 | 560 | Debugging a Docker test failure |
16e79e1b | 561 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
4eb99560 FZ |
562 | |
563 | When CI tasks, maintainers or yourself report a Docker test failure, follow the | |
564 | below steps to debug it: | |
565 | ||
566 | 1. Locally reproduce the failure with the reported command line. E.g. run | |
098bfc01 | 567 | ``make docker-test-mingw@fedora-win64-cross J=8``. |
4eb99560 FZ |
568 | 2. Add "V=1" to the command line, try again, to see the verbose output. |
569 | 3. Further add "DEBUG=1" to the command line. This will pause in a shell prompt | |
570 | in the container right before testing starts. You could either manually | |
571 | build QEMU and run tests from there, or press Ctrl-D to let the Docker | |
572 | testing continue. | |
573 | 4. If you press Ctrl-D, the same building and testing procedure will begin, and | |
574 | will hopefully run into the error again. After that, you will be dropped to | |
575 | the prompt for debug. | |
576 | ||
577 | Options | |
16e79e1b | 578 | ~~~~~~~ |
4eb99560 FZ |
579 | |
580 | Various options can be used to affect how Docker tests are done. The full | |
581 | list is in the ``make docker`` help text. The frequently used ones are: | |
582 | ||
583 | * ``V=1``: the same as in top level ``make``. It will be propagated to the | |
584 | container and enable verbose output. | |
585 | * ``J=$N``: the number of parallel tasks in make commands in the container, | |
586 | similar to the ``-j $N`` option in top level ``make``. (The ``-j`` option in | |
587 | top level ``make`` will not be propagated into the container.) | |
588 | * ``DEBUG=1``: enables debug. See the previous "Debugging a Docker test | |
589 | failure" section. | |
590 | ||
3b6882bd | 591 | Thread Sanitizer |
16e79e1b | 592 | ---------------- |
3b6882bd RF |
593 | |
594 | Thread Sanitizer (TSan) is a tool which can detect data races. QEMU supports | |
595 | building and testing with this tool. | |
596 | ||
597 | For more information on TSan: | |
598 | ||
599 | https://github.com/google/sanitizers/wiki/ThreadSanitizerCppManual | |
600 | ||
601 | Thread Sanitizer in Docker | |
16e79e1b | 602 | ~~~~~~~~~~~~~~~~~~~~~~~~~~ |
171080d8 | 603 | TSan is currently supported in the ubuntu2204 docker. |
3b6882bd RF |
604 | |
605 | The test-tsan test will build using TSan and then run make check. | |
606 | ||
607 | .. code:: | |
608 | ||
171080d8 | 609 | make docker-test-tsan@ubuntu2204 |
3b6882bd RF |
610 | |
611 | TSan warnings under docker are placed in files located at build/tsan/. | |
612 | ||
613 | We recommend using DEBUG=1 to allow launching the test from inside the docker, | |
614 | and to allow review of the warnings generated by TSan. | |
615 | ||
616 | Building and Testing with TSan | |
16e79e1b | 617 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
3b6882bd RF |
618 | |
619 | It is possible to build and test with TSan, with a few additional steps. | |
620 | These steps are normally done automatically in the docker. | |
621 | ||
622 | There is a one time patch needed in clang-9 or clang-10 at this time: | |
623 | ||
624 | .. code:: | |
625 | ||
626 | sed -i 's/^const/static const/g' \ | |
627 | /usr/lib/llvm-10/lib/clang/10.0.0/include/sanitizer/tsan_interface.h | |
628 | ||
629 | To configure the build for TSan: | |
630 | ||
631 | .. code:: | |
632 | ||
633 | ../configure --enable-tsan --cc=clang-10 --cxx=clang++-10 \ | |
634 | --disable-werror --extra-cflags="-O0" | |
635 | ||
636 | The runtime behavior of TSAN is controlled by the TSAN_OPTIONS environment | |
637 | variable. | |
638 | ||
639 | More information on the TSAN_OPTIONS can be found here: | |
640 | ||
641 | https://github.com/google/sanitizers/wiki/ThreadSanitizerFlags | |
642 | ||
643 | For example: | |
644 | ||
645 | .. code:: | |
646 | ||
647 | export TSAN_OPTIONS=suppressions=<path to qemu>/tests/tsan/suppressions.tsan \ | |
648 | detect_deadlocks=false history_size=7 exitcode=0 \ | |
649 | log_path=<build path>/tsan/tsan_warning | |
650 | ||
651 | The above exitcode=0 has TSan continue without error if any warnings are found. | |
652 | This allows for running the test and then checking the warnings afterwards. | |
653 | If you want TSan to stop and exit with error on warnings, use exitcode=66. | |
654 | ||
655 | TSan Suppressions | |
16e79e1b | 656 | ~~~~~~~~~~~~~~~~~ |
3b6882bd RF |
657 | Keep in mind that for any data race warning, although there might be a data race |
658 | detected by TSan, there might be no actual bug here. TSan provides several | |
659 | different mechanisms for suppressing warnings. In general it is recommended | |
660 | to fix the code if possible to eliminate the data race rather than suppress | |
661 | the warning. | |
662 | ||
663 | A few important files for suppressing warnings are: | |
664 | ||
665 | tests/tsan/suppressions.tsan - Has TSan warnings we wish to suppress at runtime. | |
76ca4b58 | 666 | The comment on each suppression will typically indicate why we are |
3b6882bd RF |
667 | suppressing it. More information on the file format can be found here: |
668 | ||
669 | https://github.com/google/sanitizers/wiki/ThreadSanitizerSuppressions | |
670 | ||
671 | tests/tsan/blacklist.tsan - Has TSan warnings we wish to disable | |
672 | at compile time for test or debug. | |
673 | Add flags to configure to enable: | |
674 | ||
675 | "--extra-cflags=-fsanitize-blacklist=<src path>/tests/tsan/blacklist.tsan" | |
676 | ||
677 | More information on the file format can be found here under "Blacklist Format": | |
678 | ||
679 | https://github.com/google/sanitizers/wiki/ThreadSanitizerFlags | |
680 | ||
681 | TSan Annotations | |
16e79e1b | 682 | ~~~~~~~~~~~~~~~~ |
3b6882bd RF |
683 | include/qemu/tsan.h defines annotations. See this file for more descriptions |
684 | of the annotations themselves. Annotations can be used to suppress | |
685 | TSan warnings or give TSan more information so that it can detect proper | |
686 | relationships between accesses of data. | |
687 | ||
688 | Annotation examples can be found here: | |
689 | ||
690 | https://github.com/llvm/llvm-project/tree/master/compiler-rt/test/tsan/ | |
691 | ||
692 | Good files to start with are: annotate_happens_before.cpp and ignore_race.cpp | |
693 | ||
694 | The full set of annotations can be found here: | |
695 | ||
696 | https://github.com/llvm/llvm-project/blob/master/compiler-rt/lib/tsan/rtl/tsan_interface_ann.cpp | |
697 | ||
396408ee AB |
698 | docker-binfmt-image-debian-% targets |
699 | ------------------------------------ | |
700 | ||
701 | It is possible to combine Debian's bootstrap scripts with a configured | |
702 | ``binfmt_misc`` to bootstrap a number of Debian's distros including | |
703 | experimental ports not yet supported by a released OS. This can | |
704 | simplify setting up a rootfs by using docker to contain the foreign | |
705 | rootfs rather than manually invoking chroot. | |
706 | ||
707 | Setting up ``binfmt_misc`` | |
708 | ~~~~~~~~~~~~~~~~~~~~~~~~~~ | |
709 | ||
710 | You can use the script ``qemu-binfmt-conf.sh`` to configure a QEMU | |
711 | user binary to automatically run binaries for the foreign | |
712 | architecture. While the scripts will try their best to work with | |
713 | dynamically linked QEMU's a statically linked one will present less | |
714 | potential complications when copying into the docker image. Modern | |
715 | kernels support the ``F`` (fix binary) flag which will open the QEMU | |
716 | executable on setup and avoids the need to find and re-open in the | |
717 | chroot environment. This is triggered with the ``--persistent`` flag. | |
718 | ||
719 | Example invocation | |
720 | ~~~~~~~~~~~~~~~~~~ | |
721 | ||
722 | For example to setup the HPPA ports builds of Debian:: | |
723 | ||
724 | make docker-binfmt-image-debian-sid-hppa \ | |
725 | DEB_TYPE=sid DEB_ARCH=hppa \ | |
726 | DEB_URL=http://ftp.ports.debian.org/debian-ports/ \ | |
727 | DEB_KEYRING=/usr/share/keyrings/debian-ports-archive-keyring.gpg \ | |
728 | EXECUTABLE=(pwd)/qemu-hppa V=1 | |
729 | ||
730 | The ``DEB_`` variables are substitutions used by | |
731 | ``debian-boostrap.pre`` which is called to do the initial debootstrap | |
732 | of the rootfs before it is copied into the container. The second stage | |
733 | is run as part of the build. The final image will be tagged as | |
734 | ``qemu/debian-sid-hppa``. | |
735 | ||
4eb99560 | 736 | VM testing |
16e79e1b | 737 | ---------- |
4eb99560 FZ |
738 | |
739 | This test suite contains scripts that bootstrap various guest images that have | |
740 | necessary packages to build QEMU. The basic usage is documented in ``Makefile`` | |
4f2f6276 | 741 | help which is displayed with ``make vm-help``. |
4eb99560 FZ |
742 | |
743 | Quickstart | |
16e79e1b | 744 | ~~~~~~~~~~ |
4eb99560 | 745 | |
4f2f6276 | 746 | Run ``make vm-help`` to list available make targets. Invoke a specific make |
4eb99560 FZ |
747 | command to run build test in an image. For example, ``make vm-build-freebsd`` |
748 | will build the source tree in the FreeBSD image. The command can be executed | |
749 | from either the source tree or the build dir; if the former, ``./configure`` is | |
750 | not needed. The command will then generate the test image in ``./tests/vm/`` | |
751 | under the working directory. | |
752 | ||
753 | Note: images created by the scripts accept a well-known RSA key pair for SSH | |
754 | access, so they SHOULD NOT be exposed to external interfaces if you are | |
755 | concerned about attackers taking control of the guest and potentially | |
756 | exploiting a QEMU security bug to compromise the host. | |
757 | ||
1e48931c | 758 | QEMU binaries |
16e79e1b | 759 | ~~~~~~~~~~~~~ |
4eb99560 | 760 | |
c5ba6219 PMD |
761 | By default, ``qemu-system-x86_64`` is searched in $PATH to run the guest. If |
762 | there isn't one, or if it is older than 2.10, the test won't work. In this case, | |
4eb99560 FZ |
763 | provide the QEMU binary in env var: ``QEMU=/path/to/qemu-2.10+``. |
764 | ||
c5ba6219 | 765 | Likewise the path to ``qemu-img`` can be set in QEMU_IMG environment variable. |
1e48931c | 766 | |
4eb99560 | 767 | Make jobs |
16e79e1b | 768 | ~~~~~~~~~ |
4eb99560 FZ |
769 | |
770 | The ``-j$X`` option in the make command line is not propagated into the VM, | |
771 | specify ``J=$X`` to control the make jobs in the guest. | |
772 | ||
773 | Debugging | |
16e79e1b | 774 | ~~~~~~~~~ |
4eb99560 FZ |
775 | |
776 | Add ``DEBUG=1`` and/or ``V=1`` to the make command to allow interactive | |
777 | debugging and verbose output. If this is not enough, see the next section. | |
41e3340a | 778 | ``V=1`` will be propagated down into the make jobs in the guest. |
4eb99560 FZ |
779 | |
780 | Manual invocation | |
16e79e1b | 781 | ~~~~~~~~~~~~~~~~~ |
4eb99560 FZ |
782 | |
783 | Each guest script is an executable script with the same command line options. | |
784 | For example to work with the netbsd guest, use ``$QEMU_SRC/tests/vm/netbsd``: | |
785 | ||
786 | .. code:: | |
787 | ||
788 | $ cd $QEMU_SRC/tests/vm | |
789 | ||
790 | # To bootstrap the image | |
791 | $ ./netbsd --build-image --image /var/tmp/netbsd.img | |
792 | <...> | |
793 | ||
794 | # To run an arbitrary command in guest (the output will not be echoed unless | |
795 | # --debug is added) | |
796 | $ ./netbsd --debug --image /var/tmp/netbsd.img uname -a | |
797 | ||
798 | # To build QEMU in guest | |
799 | $ ./netbsd --debug --image /var/tmp/netbsd.img --build-qemu $QEMU_SRC | |
800 | ||
801 | # To get to an interactive shell | |
802 | $ ./netbsd --interactive --image /var/tmp/netbsd.img sh | |
803 | ||
804 | Adding new guests | |
16e79e1b | 805 | ~~~~~~~~~~~~~~~~~ |
4eb99560 FZ |
806 | |
807 | Please look at existing guest scripts for how to add new guests. | |
808 | ||
809 | Most importantly, create a subclass of BaseVM and implement ``build_image()`` | |
810 | method and define ``BUILD_SCRIPT``, then finally call ``basevm.main()`` from | |
811 | the script's ``main()``. | |
812 | ||
813 | * Usually in ``build_image()``, a template image is downloaded from a | |
814 | predefined URL. ``BaseVM._download_with_cache()`` takes care of the cache and | |
815 | the checksum, so consider using it. | |
816 | ||
817 | * Once the image is downloaded, users, SSH server and QEMU build deps should | |
818 | be set up: | |
819 | ||
820 | - Root password set to ``BaseVM.ROOT_PASS`` | |
821 | - User ``BaseVM.GUEST_USER`` is created, and password set to | |
822 | ``BaseVM.GUEST_PASS`` | |
823 | - SSH service is enabled and started on boot, | |
824 | ``$QEMU_SRC/tests/keys/id_rsa.pub`` is added to ssh's ``authorized_keys`` | |
825 | file of both root and the normal user | |
826 | - DHCP client service is enabled and started on boot, so that it can | |
827 | automatically configure the virtio-net-pci NIC and communicate with QEMU | |
828 | user net (10.0.2.2) | |
829 | - Necessary packages are installed to untar the source tarball and build | |
830 | QEMU | |
831 | ||
832 | * Write a proper ``BUILD_SCRIPT`` template, which should be a shell script that | |
833 | untars a raw virtio-blk block device, which is the tarball data blob of the | |
834 | QEMU source tree, then configure/build it. Running "make check" is also | |
835 | recommended. | |
836 | ||
837 | Image fuzzer testing | |
16e79e1b | 838 | -------------------- |
4eb99560 FZ |
839 | |
840 | An image fuzzer was added to exercise format drivers. Currently only qcow2 is | |
841 | supported. To start the fuzzer, run | |
842 | ||
843 | .. code:: | |
844 | ||
845 | tests/image-fuzzer/runner.py -c '[["qemu-img", "info", "$test_img"]]' /tmp/test qcow2 | |
846 | ||
c5ba6219 | 847 | Alternatively, some command different from ``qemu-img info`` can be tested, by |
4eb99560 | 848 | changing the ``-c`` option. |
c3d7e8c9 | 849 | |
bbbd9b6e WR |
850 | Integration tests using the Avocado Framework |
851 | --------------------------------------------- | |
c3d7e8c9 | 852 | |
bbbd9b6e WR |
853 | The ``tests/avocado`` directory hosts integration tests. They're usually |
854 | higher level tests, and may interact with external resources and with | |
855 | various guest operating systems. | |
c3d7e8c9 CR |
856 | |
857 | These tests are written using the Avocado Testing Framework (which must | |
858 | be installed separately) in conjunction with a the ``avocado_qemu.Test`` | |
bbbd9b6e | 859 | class, implemented at ``tests/avocado/avocado_qemu``. |
c3d7e8c9 CR |
860 | |
861 | Tests based on ``avocado_qemu.Test`` can easily: | |
862 | ||
863 | * Customize the command line arguments given to the convenience | |
864 | ``self.vm`` attribute (a QEMUMachine instance) | |
865 | ||
866 | * Interact with the QEMU monitor, send QMP commands and check | |
867 | their results | |
868 | ||
869 | * Interact with the guest OS, using the convenience console device | |
870 | (which may be useful to assert the effectiveness and correctness of | |
871 | command line arguments or QMP commands) | |
872 | ||
873 | * Interact with external data files that accompany the test itself | |
874 | (see ``self.get_data()``) | |
875 | ||
876 | * Download (and cache) remote data files, such as firmware and kernel | |
877 | images | |
878 | ||
879 | * Have access to a library of guest OS images (by means of the | |
880 | ``avocado.utils.vmimage`` library) | |
881 | ||
882 | * Make use of various other test related utilities available at the | |
883 | test class itself and at the utility library: | |
884 | ||
885 | - http://avocado-framework.readthedocs.io/en/latest/api/test/avocado.html#avocado.Test | |
886 | - http://avocado-framework.readthedocs.io/en/latest/api/utils/avocado.utils.html | |
887 | ||
a56931ee | 888 | Running tests |
16e79e1b | 889 | ~~~~~~~~~~~~~ |
a56931ee | 890 | |
bbbd9b6e | 891 | You can run the avocado tests simply by executing: |
a56931ee CR |
892 | |
893 | .. code:: | |
894 | ||
bbbd9b6e | 895 | make check-avocado |
a56931ee | 896 | |
c03f57fd PB |
897 | This involves the automatic installation, from PyPI, of all the |
898 | necessary avocado-framework dependencies into the QEMU venv within the | |
899 | build tree (at ``./pyvenv``). Test results are also saved within the | |
a56931ee | 900 | build tree (at ``tests/results``). |
c3d7e8c9 | 901 | |
a56931ee CR |
902 | Note: the build environment must be using a Python 3 stack, and have |
903 | the ``venv`` and ``pip`` packages installed. If necessary, make sure | |
904 | ``configure`` is called with ``--python=`` and that those modules are | |
905 | available. On Debian and Ubuntu based systems, depending on the | |
906 | specific version, they may be on packages named ``python3-venv`` and | |
907 | ``python3-pip``. | |
908 | ||
23022794 | 909 | It is also possible to run tests based on tags using the |
bbbd9b6e | 910 | ``make check-avocado`` command and the ``AVOCADO_TAGS`` environment |
23022794 WR |
911 | variable: |
912 | ||
913 | .. code:: | |
914 | ||
bbbd9b6e | 915 | make check-avocado AVOCADO_TAGS=quick |
23022794 WR |
916 | |
917 | Note that tags separated with commas have an AND behavior, while tags | |
918 | separated by spaces have an OR behavior. For more information on Avocado | |
919 | tags, see: | |
920 | ||
921 | https://avocado-framework.readthedocs.io/en/latest/guides/user/chapters/tags.html | |
922 | ||
94c71462 | 923 | To run a single test file, a couple of them, or a test within a file |
bbbd9b6e | 924 | using the ``make check-avocado`` command, set the ``AVOCADO_TESTS`` |
94c71462 WR |
925 | environment variable with the test files or test names. To run all |
926 | tests from a single file, use: | |
927 | ||
928 | .. code:: | |
929 | ||
bbbd9b6e | 930 | make check-avocado AVOCADO_TESTS=$FILEPATH |
94c71462 WR |
931 | |
932 | The same is valid to run tests from multiple test files: | |
933 | ||
934 | .. code:: | |
935 | ||
bbbd9b6e | 936 | make check-avocado AVOCADO_TESTS='$FILEPATH1 $FILEPATH2' |
94c71462 WR |
937 | |
938 | To run a single test within a file, use: | |
939 | ||
940 | .. code:: | |
941 | ||
bbbd9b6e | 942 | make check-avocado AVOCADO_TESTS=$FILEPATH:$TESTCLASS.$TESTNAME |
94c71462 WR |
943 | |
944 | The same is valid to run single tests from multiple test files: | |
945 | ||
946 | .. code:: | |
947 | ||
bbbd9b6e | 948 | make check-avocado AVOCADO_TESTS='$FILEPATH1:$TESTCLASS1.$TESTNAME1 $FILEPATH2:$TESTCLASS2.$TESTNAME2' |
94c71462 | 949 | |
a56931ee CR |
950 | The scripts installed inside the virtual environment may be used |
951 | without an "activation". For instance, the Avocado test runner | |
952 | may be invoked by running: | |
953 | ||
954 | .. code:: | |
955 | ||
c03f57fd | 956 | pyvenv/bin/avocado run $OPTION1 $OPTION2 tests/avocado/ |
a56931ee | 957 | |
bbbd9b6e | 958 | Note that if ``make check-avocado`` was not executed before, it is |
6676f18f WR |
959 | possible to create the Python virtual environment with the dependencies |
960 | needed running: | |
961 | ||
962 | .. code:: | |
963 | ||
964 | make check-venv | |
965 | ||
966 | It is also possible to run tests from a single file or a single test within | |
967 | a test file. To run tests from a single file within the build tree, use: | |
968 | ||
969 | .. code:: | |
970 | ||
c03f57fd | 971 | pyvenv/bin/avocado run tests/avocado/$TESTFILE |
6676f18f WR |
972 | |
973 | To run a single test within a test file, use: | |
974 | ||
975 | .. code:: | |
976 | ||
c03f57fd | 977 | pyvenv/bin/avocado run tests/avocado/$TESTFILE:$TESTCLASS.$TESTNAME |
6676f18f WR |
978 | |
979 | Valid test names are visible in the output from any previous execution | |
bbbd9b6e | 980 | of Avocado or ``make check-avocado``, and can also be queried using: |
6676f18f WR |
981 | |
982 | .. code:: | |
983 | ||
c03f57fd | 984 | pyvenv/bin/avocado list tests/avocado |
6676f18f | 985 | |
a56931ee | 986 | Manual Installation |
16e79e1b | 987 | ~~~~~~~~~~~~~~~~~~~ |
a56931ee CR |
988 | |
989 | To manually install Avocado and its dependencies, run: | |
c3d7e8c9 CR |
990 | |
991 | .. code:: | |
992 | ||
993 | pip install --user avocado-framework | |
994 | ||
995 | Alternatively, follow the instructions on this link: | |
996 | ||
4c9ac672 | 997 | https://avocado-framework.readthedocs.io/en/latest/guides/user/chapters/installing.html |
c3d7e8c9 CR |
998 | |
999 | Overview | |
16e79e1b | 1000 | ~~~~~~~~ |
c3d7e8c9 | 1001 | |
bbbd9b6e | 1002 | The ``tests/avocado/avocado_qemu`` directory provides the |
805fac52 CR |
1003 | ``avocado_qemu`` Python module, containing the ``avocado_qemu.Test`` |
1004 | class. Here's a simple usage example: | |
c3d7e8c9 CR |
1005 | |
1006 | .. code:: | |
1007 | ||
2283b627 | 1008 | from avocado_qemu import QemuSystemTest |
c3d7e8c9 CR |
1009 | |
1010 | ||
2283b627 | 1011 | class Version(QemuSystemTest): |
c3d7e8c9 | 1012 | """ |
c3d7e8c9 CR |
1013 | :avocado: tags=quick |
1014 | """ | |
1015 | def test_qmp_human_info_version(self): | |
1016 | self.vm.launch() | |
1017 | res = self.vm.command('human-monitor-command', | |
1018 | command_line='info version') | |
1019 | self.assertRegexpMatches(res, r'^(\d+\.\d+\.\d)') | |
1020 | ||
1021 | To execute your test, run: | |
1022 | ||
1023 | .. code:: | |
1024 | ||
1025 | avocado run version.py | |
1026 | ||
1027 | Tests may be classified according to a convention by using docstring | |
1028 | directives such as ``:avocado: tags=TAG1,TAG2``. To run all tests | |
1029 | in the current directory, tagged as "quick", run: | |
1030 | ||
1031 | .. code:: | |
1032 | ||
1033 | avocado run -t quick . | |
1034 | ||
1035 | The ``avocado_qemu.Test`` base test class | |
16e79e1b | 1036 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
c3d7e8c9 CR |
1037 | |
1038 | The ``avocado_qemu.Test`` class has a number of characteristics that | |
1039 | are worth being mentioned right away. | |
1040 | ||
1041 | First of all, it attempts to give each test a ready to use QEMUMachine | |
1042 | instance, available at ``self.vm``. Because many tests will tweak the | |
1043 | QEMU command line, launching the QEMUMachine (by using ``self.vm.launch()``) | |
1044 | is left to the test writer. | |
1045 | ||
b7287d42 CC |
1046 | The base test class has also support for tests with more than one |
1047 | QEMUMachine. The way to get machines is through the ``self.get_vm()`` | |
1048 | method which will return a QEMUMachine instance. The ``self.get_vm()`` | |
1049 | method accepts arguments that will be passed to the QEMUMachine creation | |
1e235eda | 1050 | and also an optional ``name`` attribute so you can identify a specific |
b7287d42 CC |
1051 | machine and get it more than once through the tests methods. A simple |
1052 | and hypothetical example follows: | |
1053 | ||
1054 | .. code:: | |
1055 | ||
2283b627 | 1056 | from avocado_qemu import QemuSystemTest |
b7287d42 CC |
1057 | |
1058 | ||
2283b627 | 1059 | class MultipleMachines(QemuSystemTest): |
b7287d42 CC |
1060 | def test_multiple_machines(self): |
1061 | first_machine = self.get_vm() | |
1062 | second_machine = self.get_vm() | |
1063 | self.get_vm(name='third_machine').launch() | |
1064 | ||
1065 | first_machine.launch() | |
1066 | second_machine.launch() | |
1067 | ||
1068 | first_res = first_machine.command( | |
1069 | 'human-monitor-command', | |
1070 | command_line='info version') | |
1071 | ||
1072 | second_res = second_machine.command( | |
1073 | 'human-monitor-command', | |
1074 | command_line='info version') | |
1075 | ||
1076 | third_res = self.get_vm(name='third_machine').command( | |
1077 | 'human-monitor-command', | |
1078 | command_line='info version') | |
1079 | ||
1080 | self.assertEquals(first_res, second_res, third_res) | |
1081 | ||
1082 | At test "tear down", ``avocado_qemu.Test`` handles all the QEMUMachines | |
c3d7e8c9 CR |
1083 | shutdown. |
1084 | ||
1e4e7efa | 1085 | The ``avocado_qemu.LinuxTest`` base test class |
16e79e1b | 1086 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
1e4e7efa CR |
1087 | |
1088 | The ``avocado_qemu.LinuxTest`` is further specialization of the | |
1089 | ``avocado_qemu.Test`` class, so it contains all the characteristics of | |
1090 | the later plus some extra features. | |
1091 | ||
1092 | First of all, this base class is intended for tests that need to | |
1093 | interact with a fully booted and operational Linux guest. At this | |
1094 | time, it uses a Fedora 31 guest image. The most basic example looks | |
1095 | like this: | |
1096 | ||
1097 | .. code:: | |
1098 | ||
1099 | from avocado_qemu import LinuxTest | |
1100 | ||
1101 | ||
1102 | class SomeTest(LinuxTest): | |
1103 | ||
1104 | def test(self): | |
1105 | self.launch_and_wait() | |
1106 | self.ssh_command('some_command_to_be_run_in_the_guest') | |
1107 | ||
1108 | Please refer to tests that use ``avocado_qemu.LinuxTest`` under | |
bbbd9b6e | 1109 | ``tests/avocado`` for more examples. |
1e4e7efa | 1110 | |
c3d7e8c9 CR |
1111 | QEMUMachine |
1112 | ~~~~~~~~~~~ | |
1113 | ||
1114 | The QEMUMachine API is already widely used in the Python iotests, | |
1115 | device-crash-test and other Python scripts. It's a wrapper around the | |
1116 | execution of a QEMU binary, giving its users: | |
1117 | ||
1118 | * the ability to set command line arguments to be given to the QEMU | |
1119 | binary | |
1120 | ||
1121 | * a ready to use QMP connection and interface, which can be used to | |
1122 | send commands and inspect its results, as well as asynchronous | |
1123 | events | |
1124 | ||
1125 | * convenience methods to set commonly used command line arguments in | |
1126 | a more succinct and intuitive way | |
1127 | ||
1128 | QEMU binary selection | |
16e79e1b | 1129 | ^^^^^^^^^^^^^^^^^^^^^ |
c3d7e8c9 CR |
1130 | |
1131 | The QEMU binary used for the ``self.vm`` QEMUMachine instance will | |
1132 | primarily depend on the value of the ``qemu_bin`` parameter. If it's | |
1133 | not explicitly set, its default value will be the result of a dynamic | |
1134 | probe in the same source tree. A suitable binary will be one that | |
1135 | targets the architecture matching host machine. | |
1136 | ||
1137 | Based on this description, test writers will usually rely on one of | |
1138 | the following approaches: | |
1139 | ||
1140 | 1) Set ``qemu_bin``, and use the given binary | |
1141 | ||
1142 | 2) Do not set ``qemu_bin``, and use a QEMU binary named like | |
64ed6f92 | 1143 | "qemu-system-${arch}", either in the current |
c3d7e8c9 CR |
1144 | working directory, or in the current source tree. |
1145 | ||
1146 | The resulting ``qemu_bin`` value will be preserved in the | |
1147 | ``avocado_qemu.Test`` as an attribute with the same name. | |
1148 | ||
1149 | Attribute reference | |
16e79e1b PB |
1150 | ~~~~~~~~~~~~~~~~~~~ |
1151 | ||
1152 | Test | |
1153 | ^^^^ | |
c3d7e8c9 CR |
1154 | |
1155 | Besides the attributes and methods that are part of the base | |
1156 | ``avocado.Test`` class, the following attributes are available on any | |
1157 | ``avocado_qemu.Test`` instance. | |
1158 | ||
1159 | vm | |
16e79e1b | 1160 | '' |
c3d7e8c9 CR |
1161 | |
1162 | A QEMUMachine instance, initially configured according to the given | |
1163 | ``qemu_bin`` parameter. | |
1164 | ||
2c44d68f | 1165 | arch |
16e79e1b | 1166 | '''' |
2c44d68f CR |
1167 | |
1168 | The architecture can be used on different levels of the stack, e.g. by | |
1169 | the framework or by the test itself. At the framework level, it will | |
1170 | currently influence the selection of a QEMU binary (when one is not | |
1171 | explicitly given). | |
1172 | ||
1173 | Tests are also free to use this attribute value, for their own needs. | |
1174 | A test may, for instance, use the same value when selecting the | |
1175 | architecture of a kernel or disk image to boot a VM with. | |
1176 | ||
1177 | The ``arch`` attribute will be set to the test parameter of the same | |
b910545f CR |
1178 | name. If one is not given explicitly, it will either be set to |
1179 | ``None``, or, if the test is tagged with one (and only one) | |
1180 | ``:avocado: tags=arch:VALUE`` tag, it will be set to ``VALUE``. | |
2c44d68f | 1181 | |
20bbf846 | 1182 | cpu |
16e79e1b | 1183 | ''' |
20bbf846 WSM |
1184 | |
1185 | The cpu model that will be set to all QEMUMachine instances created | |
1186 | by the test. | |
1187 | ||
1188 | The ``cpu`` attribute will be set to the test parameter of the same | |
1189 | name. If one is not given explicitly, it will either be set to | |
1190 | ``None ``, or, if the test is tagged with one (and only one) | |
1191 | ``:avocado: tags=cpu:VALUE`` tag, it will be set to ``VALUE``. | |
1192 | ||
ba21bde9 | 1193 | machine |
16e79e1b | 1194 | ''''''' |
ba21bde9 CR |
1195 | |
1196 | The machine type that will be set to all QEMUMachine instances created | |
1197 | by the test. | |
1198 | ||
1199 | The ``machine`` attribute will be set to the test parameter of the same | |
1200 | name. If one is not given explicitly, it will either be set to | |
1201 | ``None``, or, if the test is tagged with one (and only one) | |
1202 | ``:avocado: tags=machine:VALUE`` tag, it will be set to ``VALUE``. | |
1203 | ||
c3d7e8c9 | 1204 | qemu_bin |
16e79e1b | 1205 | '''''''' |
c3d7e8c9 CR |
1206 | |
1207 | The preserved value of the ``qemu_bin`` parameter or the result of the | |
1208 | dynamic probe for a QEMU binary in the current working directory or | |
1209 | source tree. | |
1210 | ||
d5adf9d5 | 1211 | LinuxTest |
16e79e1b | 1212 | ^^^^^^^^^ |
d5adf9d5 CR |
1213 | |
1214 | Besides the attributes present on the ``avocado_qemu.Test`` base | |
1215 | class, the ``avocado_qemu.LinuxTest`` adds the following attributes: | |
1216 | ||
1217 | distro | |
16e79e1b | 1218 | '''''' |
d5adf9d5 CR |
1219 | |
1220 | The name of the Linux distribution used as the guest image for the | |
1221 | test. The name should match the **Provider** column on the list | |
1222 | of images supported by the avocado.utils.vmimage library: | |
1223 | ||
1224 | https://avocado-framework.readthedocs.io/en/latest/guides/writer/libs/vmimage.html#supported-images | |
1225 | ||
1226 | distro_version | |
16e79e1b | 1227 | '''''''''''''' |
d5adf9d5 CR |
1228 | |
1229 | The version of the Linux distribution as the guest image for the | |
1230 | test. The name should match the **Version** column on the list | |
1231 | of images supported by the avocado.utils.vmimage library: | |
1232 | ||
1233 | https://avocado-framework.readthedocs.io/en/latest/guides/writer/libs/vmimage.html#supported-images | |
1234 | ||
1235 | distro_checksum | |
16e79e1b | 1236 | ''''''''''''''' |
d5adf9d5 CR |
1237 | |
1238 | The sha256 hash of the guest image file used for the test. | |
1239 | ||
1240 | If this value is not set in the code or by a test parameter (with the | |
1241 | same name), no validation on the integrity of the image will be | |
1242 | performed. | |
1243 | ||
c3d7e8c9 | 1244 | Parameter reference |
16e79e1b | 1245 | ~~~~~~~~~~~~~~~~~~~ |
c3d7e8c9 CR |
1246 | |
1247 | To understand how Avocado parameters are accessed by tests, and how | |
1248 | they can be passed to tests, please refer to:: | |
1249 | ||
4c9ac672 | 1250 | https://avocado-framework.readthedocs.io/en/latest/guides/writer/chapters/writing.html#accessing-test-parameters |
c3d7e8c9 CR |
1251 | |
1252 | Parameter values can be easily seen in the log files, and will look | |
1253 | like the following: | |
1254 | ||
1255 | .. code:: | |
1256 | ||
64ed6f92 | 1257 | PARAMS (key=qemu_bin, path=*, default=./qemu-system-x86_64) => './qemu-system-x86_64 |
c3d7e8c9 | 1258 | |
16e79e1b PB |
1259 | Test |
1260 | ^^^^ | |
1261 | ||
2c44d68f | 1262 | arch |
16e79e1b | 1263 | '''' |
2c44d68f CR |
1264 | |
1265 | The architecture that will influence the selection of a QEMU binary | |
1266 | (when one is not explicitly given). | |
1267 | ||
1268 | Tests are also free to use this parameter value, for their own needs. | |
1269 | A test may, for instance, use the same value when selecting the | |
1270 | architecture of a kernel or disk image to boot a VM with. | |
1271 | ||
1272 | This parameter has a direct relation with the ``arch`` attribute. If | |
1273 | not given, it will default to None. | |
1274 | ||
20bbf846 | 1275 | cpu |
16e79e1b | 1276 | ''' |
20bbf846 WSM |
1277 | |
1278 | The cpu model that will be set to all QEMUMachine instances created | |
1279 | by the test. | |
1280 | ||
ba21bde9 | 1281 | machine |
16e79e1b | 1282 | ''''''' |
ba21bde9 CR |
1283 | |
1284 | The machine type that will be set to all QEMUMachine instances created | |
1285 | by the test. | |
1286 | ||
c3d7e8c9 | 1287 | qemu_bin |
16e79e1b | 1288 | '''''''' |
c3d7e8c9 CR |
1289 | |
1290 | The exact QEMU binary to be used on QEMUMachine. | |
1291 | ||
d5adf9d5 | 1292 | LinuxTest |
16e79e1b | 1293 | ^^^^^^^^^ |
d5adf9d5 CR |
1294 | |
1295 | Besides the parameters present on the ``avocado_qemu.Test`` base | |
1296 | class, the ``avocado_qemu.LinuxTest`` adds the following parameters: | |
1297 | ||
1298 | distro | |
16e79e1b | 1299 | '''''' |
d5adf9d5 CR |
1300 | |
1301 | The name of the Linux distribution used as the guest image for the | |
1302 | test. The name should match the **Provider** column on the list | |
1303 | of images supported by the avocado.utils.vmimage library: | |
1304 | ||
1305 | https://avocado-framework.readthedocs.io/en/latest/guides/writer/libs/vmimage.html#supported-images | |
1306 | ||
1307 | distro_version | |
16e79e1b | 1308 | '''''''''''''' |
d5adf9d5 CR |
1309 | |
1310 | The version of the Linux distribution as the guest image for the | |
1311 | test. The name should match the **Version** column on the list | |
1312 | of images supported by the avocado.utils.vmimage library: | |
1313 | ||
1314 | https://avocado-framework.readthedocs.io/en/latest/guides/writer/libs/vmimage.html#supported-images | |
1315 | ||
1316 | distro_checksum | |
16e79e1b | 1317 | ''''''''''''''' |
d5adf9d5 CR |
1318 | |
1319 | The sha256 hash of the guest image file used for the test. | |
1320 | ||
1321 | If this value is not set in the code or by this parameter no | |
1322 | validation on the integrity of the image will be performed. | |
1323 | ||
cf5891ec | 1324 | Skipping tests |
16e79e1b PB |
1325 | ~~~~~~~~~~~~~~ |
1326 | ||
cf5891ec WSM |
1327 | The Avocado framework provides Python decorators which allow for easily skip |
1328 | tests running under certain conditions. For example, on the lack of a binary | |
1329 | on the test system or when the running environment is a CI system. For further | |
1330 | information about those decorators, please refer to:: | |
1331 | ||
1332 | https://avocado-framework.readthedocs.io/en/latest/guides/writer/chapters/writing.html#skipping-tests | |
1333 | ||
1334 | While the conditions for skipping tests are often specifics of each one, there | |
1335 | are recurring scenarios identified by the QEMU developers and the use of | |
1336 | environment variables became a kind of standard way to enable/disable tests. | |
1337 | ||
1338 | Here is a list of the most used variables: | |
1339 | ||
1340 | AVOCADO_ALLOW_LARGE_STORAGE | |
16e79e1b | 1341 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
cf5891ec | 1342 | Tests which are going to fetch or produce assets considered *large* are not |
1e235eda | 1343 | going to run unless that ``AVOCADO_ALLOW_LARGE_STORAGE=1`` is exported on |
cf5891ec WSM |
1344 | the environment. |
1345 | ||
1346 | The definition of *large* is a bit arbitrary here, but it usually means an | |
1347 | asset which occupies at least 1GB of size on disk when uncompressed. | |
1348 | ||
1349 | AVOCADO_ALLOW_UNTRUSTED_CODE | |
16e79e1b | 1350 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
cf5891ec WSM |
1351 | There are tests which will boot a kernel image or firmware that can be |
1352 | considered not safe to run on the developer's workstation, thus they are | |
1353 | skipped by default. The definition of *not safe* is also arbitrary but | |
1354 | usually it means a blob which either its source or build process aren't | |
1355 | public available. | |
1356 | ||
1e235eda | 1357 | You should export ``AVOCADO_ALLOW_UNTRUSTED_CODE=1`` on the environment in |
cf5891ec WSM |
1358 | order to allow tests which make use of those kind of assets. |
1359 | ||
1360 | AVOCADO_TIMEOUT_EXPECTED | |
16e79e1b | 1361 | ^^^^^^^^^^^^^^^^^^^^^^^^ |
cf5891ec WSM |
1362 | The Avocado framework has a timeout mechanism which interrupts tests to avoid the |
1363 | test suite of getting stuck. The timeout value can be set via test parameter or | |
1364 | property defined in the test class, for further details:: | |
1365 | ||
1366 | https://avocado-framework.readthedocs.io/en/latest/guides/writer/chapters/writing.html#setting-a-test-timeout | |
1367 | ||
1368 | Even though the timeout can be set by the test developer, there are some tests | |
1369 | that may not have a well-defined limit of time to finish under certain | |
1370 | conditions. For example, tests that take longer to execute when QEMU is | |
1e235eda | 1371 | compiled with debug flags. Therefore, the ``AVOCADO_TIMEOUT_EXPECTED`` variable |
cf5891ec WSM |
1372 | has been used to determine whether those tests should run or not. |
1373 | ||
1374 | GITLAB_CI | |
16e79e1b | 1375 | ^^^^^^^^^ |
cf5891ec WSM |
1376 | A number of tests are flagged to not run on the GitLab CI. Usually because |
1377 | they proved to the flaky or there are constraints on the CI environment which | |
1378 | would make them fail. If you encounter a similar situation then use that | |
1379 | variable as shown on the code snippet below to skip the test: | |
1380 | ||
1381 | .. code:: | |
1382 | ||
1383 | @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab') | |
1384 | def test(self): | |
1385 | do_something() | |
1386 | ||
a5754847 TH |
1387 | QEMU_TEST_FLAKY_TESTS |
1388 | ^^^^^^^^^^^^^^^^^^^^^ | |
1389 | Some tests are not working reliably and thus are disabled by default. | |
1390 | Set this environment variable to enable them. | |
1391 | ||
c3d7e8c9 | 1392 | Uninstalling Avocado |
16e79e1b | 1393 | ~~~~~~~~~~~~~~~~~~~~ |
c3d7e8c9 | 1394 | |
a56931ee CR |
1395 | If you've followed the manual installation instructions above, you can |
1396 | easily uninstall Avocado. Start by listing the packages you have | |
1397 | installed:: | |
c3d7e8c9 CR |
1398 | |
1399 | pip list --user | |
1400 | ||
1401 | And remove any package you want with:: | |
1402 | ||
1403 | pip uninstall <package_name> | |
a56931ee | 1404 | |
bbbd9b6e | 1405 | If you've used ``make check-avocado``, the Python virtual environment where |
a56931ee | 1406 | Avocado is installed will be cleaned up as part of ``make check-clean``. |
f8ed349e | 1407 | |
4583cdad AB |
1408 | .. _checktcg-ref: |
1409 | ||
f8ed349e | 1410 | Testing with "make check-tcg" |
16e79e1b | 1411 | ----------------------------- |
f8ed349e AB |
1412 | |
1413 | The check-tcg tests are intended for simple smoke tests of both | |
1414 | linux-user and softmmu TCG functionality. However to build test | |
1415 | programs for guest targets you need to have cross compilers available. | |
1416 | If your distribution supports cross compilers you can do something as | |
1417 | simple as:: | |
1418 | ||
1419 | apt install gcc-aarch64-linux-gnu | |
1420 | ||
1421 | The configure script will automatically pick up their presence. | |
1422 | Sometimes compilers have slightly odd names so the availability of | |
1423 | them can be prompted by passing in the appropriate configure option | |
1424 | for the architecture in question, for example:: | |
1425 | ||
1426 | $(configure) --cross-cc-aarch64=aarch64-cc | |
1427 | ||
479ca4cc | 1428 | There is also a ``--cross-cc-cflags-ARCH`` flag in case additional |
f8ed349e AB |
1429 | compiler flags are needed to build for a given target. |
1430 | ||
663a041e AB |
1431 | If you have the ability to run containers as the user the build system |
1432 | will automatically use them where no system compiler is available. For | |
1433 | architectures where we also support building QEMU we will generally | |
1434 | use the same container to build tests. However there are a number of | |
1435 | additional containers defined that have a minimal cross-build | |
1436 | environment that is only suitable for building test cases. Sometimes | |
1437 | we may use a bleeding edge distribution for compiler features needed | |
1438 | for test cases that aren't yet in the LTS distros we support for QEMU | |
1439 | itself. | |
1440 | ||
1441 | See :ref:`container-ref` for more details. | |
f8ed349e AB |
1442 | |
1443 | Running subset of tests | |
16e79e1b | 1444 | ~~~~~~~~~~~~~~~~~~~~~~~ |
f8ed349e AB |
1445 | |
1446 | You can build the tests for one architecture:: | |
1447 | ||
1448 | make build-tcg-tests-$TARGET | |
1449 | ||
1450 | And run with:: | |
1451 | ||
1452 | make run-tcg-tests-$TARGET | |
1453 | ||
1454 | Adding ``V=1`` to the invocation will show the details of how to | |
1455 | invoke QEMU for the test which is useful for debugging tests. | |
1456 | ||
1457 | TCG test dependencies | |
16e79e1b | 1458 | ~~~~~~~~~~~~~~~~~~~~~ |
f8ed349e AB |
1459 | |
1460 | The TCG tests are deliberately very light on dependencies and are | |
1461 | either totally bare with minimal gcc lib support (for softmmu tests) | |
1462 | or just glibc (for linux-user tests). This is because getting a cross | |
1463 | compiler to work with additional libraries can be challenging. | |
1464 | ||
1465 | Other TCG Tests | |
1466 | --------------- | |
1467 | ||
1468 | There are a number of out-of-tree test suites that are used for more | |
1469 | extensive testing of processor features. | |
1470 | ||
1471 | KVM Unit Tests | |
1472 | ~~~~~~~~~~~~~~ | |
1473 | ||
1474 | The KVM unit tests are designed to run as a Guest OS under KVM but | |
1475 | there is no reason why they can't exercise the TCG as well. It | |
1476 | provides a minimal OS kernel with hooks for enabling the MMU as well | |
1477 | as reporting test results via a special device:: | |
1478 | ||
1479 | https://git.kernel.org/pub/scm/virt/kvm/kvm-unit-tests.git | |
1480 | ||
1481 | Linux Test Project | |
1482 | ~~~~~~~~~~~~~~~~~~ | |
1483 | ||
1484 | The LTP is focused on exercising the syscall interface of a Linux | |
1485 | kernel. It checks that syscalls behave as documented and strives to | |
1486 | exercise as many corner cases as possible. It is a useful test suite | |
1487 | to run to exercise QEMU's linux-user code:: | |
1488 | ||
1489 | https://linux-test-project.github.io/ | |
9fce3601 PB |
1490 | |
1491 | GCC gcov support | |
16e79e1b | 1492 | ---------------- |
9fce3601 PB |
1493 | |
1494 | ``gcov`` is a GCC tool to analyze the testing coverage by | |
1495 | instrumenting the tested code. To use it, configure QEMU with | |
1496 | ``--enable-gcov`` option and build. Then run the tests as usual. | |
1497 | ||
1498 | If you want to gather coverage information on a single test the ``make | |
1499 | clean-gcda`` target can be used to delete any existing coverage | |
1500 | information before running a single test. | |
1501 | ||
1502 | You can generate a HTML coverage report by executing ``make | |
1503 | coverage-html`` which will create | |
1504 | ``meson-logs/coveragereport/index.html``. | |
1505 | ||
1506 | Further analysis can be conducted by running the ``gcov`` command | |
1507 | directly on the various .gcda output files. Please read the ``gcov`` | |
1508 | documentation for more information. |