]>
Commit | Line | Data |
---|---|---|
7266ecce AB |
1 | .. _testing: |
2 | ||
4eb99560 FZ |
3 | Testing in QEMU |
4 | =============== | |
5 | ||
6 | This document describes the testing infrastructure in QEMU. | |
7 | ||
8 | Testing with "make check" | |
16e79e1b | 9 | ------------------------- |
4eb99560 FZ |
10 | |
11 | The "make check" testing family includes most of the C based tests in QEMU. For | |
12 | a quick help, run ``make check-help`` from the source tree. | |
13 | ||
14 | The usual way to run these tests is: | |
15 | ||
16 | .. code:: | |
17 | ||
18 | make check | |
19 | ||
316082b1 TH |
20 | which includes QAPI schema tests, unit tests, QTests and some iotests. |
21 | Different sub-types of "make check" tests will be explained below. | |
4eb99560 FZ |
22 | |
23 | Before running tests, it is best to build QEMU programs first. Some tests | |
24 | expect the executables to exist and will fail with obscure messages if they | |
25 | cannot find them. | |
26 | ||
27 | Unit tests | |
16e79e1b | 28 | ~~~~~~~~~~ |
4eb99560 FZ |
29 | |
30 | Unit tests, which can be invoked with ``make check-unit``, are simple C tests | |
31 | that typically link to individual QEMU object files and exercise them by | |
32 | calling exported functions. | |
33 | ||
34 | If you are writing new code in QEMU, consider adding a unit test, especially | |
35 | for utility modules that are relatively stateless or have few dependencies. To | |
36 | add a new unit test: | |
37 | ||
8db5c3e2 | 38 | 1. Create a new source file. For example, ``tests/unit/foo-test.c``. |
4eb99560 FZ |
39 | |
40 | 2. Write the test. Normally you would include the header file which exports | |
41 | the module API, then verify the interface behaves as expected from your | |
42 | test. The test code should be organized with the glib testing framework. | |
43 | Copying and modifying an existing test is usually a good idea. | |
44 | ||
8db5c3e2 | 45 | 3. Add the test to ``tests/unit/meson.build``. The unit tests are listed in a |
bab88ead PB |
46 | dictionary called ``tests``. The values are any additional sources and |
47 | dependencies to be linked with the test. For a simple test whose source | |
8db5c3e2 | 48 | is in ``tests/unit/foo-test.c``, it is enough to add an entry like:: |
bab88ead PB |
49 | |
50 | { | |
51 | ... | |
52 | 'foo-test': [], | |
53 | ... | |
54 | } | |
4eb99560 FZ |
55 | |
56 | Since unit tests don't require environment variables, the simplest way to debug | |
57 | a unit test failure is often directly invoking it or even running it under | |
58 | ``gdb``. However there can still be differences in behavior between ``make`` | |
59 | invocations and your manual run, due to ``$MALLOC_PERTURB_`` environment | |
60 | variable (which affects memory reclamation and catches invalid pointers better) | |
61 | and gtester options. If necessary, you can run | |
62 | ||
63 | .. code:: | |
92970812 | 64 | |
4eb99560 FZ |
65 | make check-unit V=1 |
66 | ||
67 | and copy the actual command line which executes the unit test, then run | |
68 | it from the command line. | |
69 | ||
70 | QTest | |
16e79e1b | 71 | ~~~~~ |
4eb99560 FZ |
72 | |
73 | QTest is a device emulation testing framework. It can be very useful to test | |
74 | device models; it could also control certain aspects of QEMU (such as virtual | |
a738a50e EH |
75 | clock stepping), with a special purpose "qtest" protocol. Refer to |
76 | :doc:`qtest` for more details. | |
4eb99560 FZ |
77 | |
78 | QTest cases can be executed with | |
79 | ||
80 | .. code:: | |
81 | ||
82 | make check-qtest | |
83 | ||
0b49bc1b BM |
84 | Writing portable test cases |
85 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~ | |
86 | Both unit tests and qtests can run on POSIX hosts as well as Windows hosts. | |
87 | Care must be taken when writing portable test cases that can be built and run | |
88 | successfully on various hosts. The following list shows some best practices: | |
89 | ||
90 | * Use portable APIs from glib whenever necessary, e.g.: g_setenv(), | |
91 | g_mkdtemp(), g_mkdir(). | |
92 | * Avoid using hardcoded /tmp for temporary file directory. | |
93 | Use g_get_tmp_dir() instead. | |
94 | * Bear in mind that Windows has different special string representation for | |
95 | stdin/stdout/stderr and null devices. For example if your test case uses | |
96 | "/dev/fd/2" and "/dev/null" on Linux, remember to use "2" and "nul" on | |
97 | Windows instead. Also IO redirection does not work on Windows, so avoid | |
98 | using "2>nul" whenever necessary. | |
99 | * If your test cases uses the blkdebug feature, use relative path to pass | |
100 | the config and image file paths in the command line as Windows absolute | |
101 | path contains the delimiter ":" which will confuse the blkdebug parser. | |
1e458f11 | 102 | * Use double quotes in your extra QEMU command line in your test cases |
0b49bc1b BM |
103 | instead of single quotes, as Windows does not drop single quotes when |
104 | passing the command line to QEMU. | |
105 | * Windows opens a file in text mode by default, while a POSIX compliant | |
106 | implementation treats text files and binary files the same. So if your | |
107 | test cases opens a file to write some data and later wants to compare the | |
108 | written data with the original one, be sure to pass the letter 'b' as | |
109 | part of the mode string to fopen(), or O_BINARY flag for the open() call. | |
110 | * If a certain test case can only run on POSIX or Linux hosts, use a proper | |
111 | #ifdef in the codes. If the whole test suite cannot run on Windows, disable | |
112 | the build in the meson.build file. | |
113 | ||
4eb99560 | 114 | QAPI schema tests |
16e79e1b | 115 | ~~~~~~~~~~~~~~~~~ |
4eb99560 FZ |
116 | |
117 | The QAPI schema tests validate the QAPI parser used by QMP, by feeding | |
118 | predefined input to the parser and comparing the result with the reference | |
119 | output. | |
120 | ||
121 | The input/output data is managed under the ``tests/qapi-schema`` directory. | |
122 | Each test case includes four files that have a common base name: | |
123 | ||
124 | * ``${casename}.json`` - the file contains the JSON input for feeding the | |
125 | parser | |
126 | * ``${casename}.out`` - the file contains the expected stdout from the parser | |
127 | * ``${casename}.err`` - the file contains the expected stderr from the parser | |
128 | * ``${casename}.exit`` - the expected error code | |
129 | ||
130 | Consider adding a new QAPI schema test when you are making a change on the QAPI | |
131 | parser (either fixing a bug or extending/modifying the syntax). To do this: | |
132 | ||
133 | 1. Add four files for the new case as explained above. For example: | |
134 | ||
135 | ``$EDITOR tests/qapi-schema/foo.{json,out,err,exit}``. | |
136 | ||
137 | 2. Add the new test in ``tests/Makefile.include``. For example: | |
138 | ||
139 | ``qapi-schema += foo.json`` | |
140 | ||
141 | check-block | |
16e79e1b | 142 | ~~~~~~~~~~~ |
4eb99560 | 143 | |
316082b1 | 144 | ``make check-block`` runs a subset of the block layer iotests (the tests that |
b25a9488 | 145 | are in the "auto" group). |
316082b1 | 146 | See the "QEMU iotests" section below for more information. |
4eb99560 | 147 | |
4eb99560 | 148 | QEMU iotests |
16e79e1b | 149 | ------------ |
4eb99560 FZ |
150 | |
151 | QEMU iotests, under the directory ``tests/qemu-iotests``, is the testing | |
152 | framework widely used to test block layer related features. It is higher level | |
153 | than "make check" tests and 99% of the code is written in bash or Python | |
154 | scripts. The testing success criteria is golden output comparison, and the | |
155 | test files are named with numbers. | |
156 | ||
157 | To run iotests, make sure QEMU is built successfully, then switch to the | |
158 | ``tests/qemu-iotests`` directory under the build directory, and run ``./check`` | |
159 | with desired arguments from there. | |
160 | ||
161 | By default, "raw" format and "file" protocol is used; all tests will be | |
162 | executed, except the unsupported ones. You can override the format and protocol | |
163 | with arguments: | |
164 | ||
165 | .. code:: | |
166 | ||
167 | # test with qcow2 format | |
168 | ./check -qcow2 | |
169 | # or test a different protocol | |
170 | ./check -nbd | |
171 | ||
172 | It's also possible to list test numbers explicitly: | |
173 | ||
174 | .. code:: | |
175 | ||
176 | # run selected cases with qcow2 format | |
177 | ./check -qcow2 001 030 153 | |
178 | ||
179 | Cache mode can be selected with the "-c" option, which may help reveal bugs | |
180 | that are specific to certain cache mode. | |
181 | ||
182 | More options are supported by the ``./check`` script, run ``./check -h`` for | |
183 | help. | |
184 | ||
185 | Writing a new test case | |
16e79e1b | 186 | ~~~~~~~~~~~~~~~~~~~~~~~ |
4eb99560 FZ |
187 | |
188 | Consider writing a tests case when you are making any changes to the block | |
189 | layer. An iotest case is usually the choice for that. There are already many | |
190 | test cases, so it is possible that extending one of them may achieve the goal | |
191 | and save the boilerplate to create one. (Unfortunately, there isn't a 100% | |
192 | reliable way to find a related one out of hundreds of tests. One approach is | |
193 | using ``git grep``.) | |
194 | ||
195 | Usually an iotest case consists of two files. One is an executable that | |
196 | produces output to stdout and stderr, the other is the expected reference | |
197 | output. They are given the same number in file names. E.g. Test script ``055`` | |
198 | and reference output ``055.out``. | |
199 | ||
200 | In rare cases, when outputs differ between cache mode ``none`` and others, a | |
201 | ``.out.nocache`` file is added. In other cases, when outputs differ between | |
202 | image formats, more than one ``.out`` files are created ending with the | |
203 | respective format names, e.g. ``178.out.qcow2`` and ``178.out.raw``. | |
204 | ||
205 | There isn't a hard rule about how to write a test script, but a new test is | |
206 | usually a (copy and) modification of an existing case. There are a few | |
207 | commonly used ways to create a test: | |
208 | ||
209 | * A Bash script. It will make use of several environmental variables related | |
210 | to the testing procedure, and could source a group of ``common.*`` libraries | |
211 | for some common helper routines. | |
212 | ||
213 | * A Python unittest script. Import ``iotests`` and create a subclass of | |
214 | ``iotests.QMPTestCase``, then call ``iotests.main`` method. The downside of | |
215 | this approach is that the output is too scarce, and the script is considered | |
216 | harder to debug. | |
217 | ||
218 | * A simple Python script without using unittest module. This could also import | |
219 | ``iotests`` for launching QEMU and utilities etc, but it doesn't inherit | |
220 | from ``iotests.QMPTestCase`` therefore doesn't use the Python unittest | |
221 | execution. This is a combination of 1 and 2. | |
222 | ||
223 | Pick the language per your preference since both Bash and Python have | |
224 | comparable library support for invoking and interacting with QEMU programs. If | |
225 | you opt for Python, it is strongly recommended to write Python 3 compatible | |
226 | code. | |
227 | ||
f4a1b653 FZ |
228 | Both Python and Bash frameworks in iotests provide helpers to manage test |
229 | images. They can be used to create and clean up images under the test | |
230 | directory. If no I/O or any protocol specific feature is needed, it is often | |
231 | more convenient to use the pseudo block driver, ``null-co://``, as the test | |
232 | image, which doesn't require image creation or cleaning up. Avoid system-wide | |
233 | devices or files whenever possible, such as ``/dev/null`` or ``/dev/zero``. | |
234 | Otherwise, image locking implications have to be considered. For example, | |
235 | another application on the host may have locked the file, possibly leading to a | |
236 | test failure. If using such devices are explicitly desired, consider adding | |
237 | ``locking=off`` option to disable image locking. | |
238 | ||
0193767b | 239 | Debugging a test case |
16e79e1b PB |
240 | ~~~~~~~~~~~~~~~~~~~~~ |
241 | ||
0193767b EGE |
242 | The following options to the ``check`` script can be useful when debugging |
243 | a failing test: | |
244 | ||
e92ecc32 EGE |
245 | * ``-gdb`` wraps every QEMU invocation in a ``gdbserver``, which waits for a |
246 | connection from a gdb client. The options given to ``gdbserver`` (e.g. the | |
247 | address on which to listen for connections) are taken from the ``$GDB_OPTIONS`` | |
248 | environment variable. By default (if ``$GDB_OPTIONS`` is empty), it listens on | |
249 | ``localhost:12345``. | |
250 | It is possible to connect to it for example with | |
251 | ``gdb -iex "target remote $addr"``, where ``$addr`` is the address | |
252 | ``gdbserver`` listens on. | |
253 | If the ``-gdb`` option is not used, ``$GDB_OPTIONS`` is ignored, | |
254 | regardless of whether it is set or not. | |
255 | ||
bd10a739 EGE |
256 | * ``-valgrind`` attaches a valgrind instance to QEMU. If it detects |
257 | warnings, it will print and save the log in | |
258 | ``$TEST_DIR/<valgrind_pid>.valgrind``. | |
259 | The final command line will be ``valgrind --log-file=$TEST_DIR/ | |
260 | <valgrind_pid>.valgrind --error-exitcode=99 $QEMU ...`` | |
261 | ||
0193767b EGE |
262 | * ``-d`` (debug) just increases the logging verbosity, showing |
263 | for example the QMP commands and answers. | |
264 | ||
8ffcda2a EGE |
265 | * ``-p`` (print) redirects QEMU’s stdout and stderr to the test output, |
266 | instead of saving it into a log file in | |
267 | ``$TEST_DIR/qemu-machine-<random_string>``. | |
268 | ||
b25a9488 | 269 | Test case groups |
16e79e1b | 270 | ~~~~~~~~~~~~~~~~ |
b25a9488 VSO |
271 | |
272 | "Tests may belong to one or more test groups, which are defined in the form | |
273 | of a comment in the test source file. By convention, test groups are listed | |
274 | in the second line of the test file, after the "#!/..." line, like this: | |
275 | ||
276 | .. code:: | |
277 | ||
278 | #!/usr/bin/env python3 | |
279 | # group: auto quick | |
280 | # | |
281 | ... | |
282 | ||
283 | Another way of defining groups is creating the tests/qemu-iotests/group.local | |
284 | file. This should be used only for downstream (this file should never appear | |
285 | in upstream). This file may be used for defining some downstream test groups | |
286 | or for temporarily disabling tests, like this: | |
287 | ||
288 | .. code:: | |
289 | ||
290 | # groups for some company downstream process | |
291 | # | |
292 | # ci - tests to run on build | |
293 | # down - our downstream tests, not for upstream | |
294 | # | |
295 | # Format of each line is: | |
296 | # TEST_NAME TEST_GROUP [TEST_GROUP ]... | |
297 | ||
298 | 013 ci | |
299 | 210 disabled | |
300 | 215 disabled | |
301 | our-ugly-workaround-test down ci | |
302 | ||
303 | Note that the following group names have a special meaning: | |
304 | ||
305 | - quick: Tests in this group should finish within a few seconds. | |
306 | ||
307 | - auto: Tests in this group are used during "make check" and should be | |
308 | runnable in any case. That means they should run with every QEMU binary | |
309 | (also non-x86), with every QEMU configuration (i.e. must not fail if | |
310 | an optional feature is not compiled in - but reporting a "skip" is ok), | |
311 | work at least with the qcow2 file format, work with all kind of host | |
312 | filesystems and users (e.g. "nobody" or "root") and must not take too | |
313 | much memory and disk space (since CI pipelines tend to fail otherwise). | |
314 | ||
315 | - disabled: Tests in this group are disabled and ignored by check. | |
316 | ||
663a041e | 317 | .. _container-ref: |
f8ed349e | 318 | |
663a041e | 319 | Container based tests |
16e79e1b | 320 | --------------------- |
4eb99560 FZ |
321 | |
322 | Introduction | |
16e79e1b | 323 | ~~~~~~~~~~~~ |
4eb99560 | 324 | |
9c1f491e AB |
325 | The container testing framework in QEMU utilizes public images to |
326 | build and test QEMU in predefined and widely accessible Linux | |
327 | environments. This makes it possible to expand the test coverage | |
328 | across distros, toolchain flavors and library versions. The support | |
329 | was originally written for Docker although we also support Podman as | |
0285a0ec | 330 | an alternative container runtime. Although many of the target |
9c1f491e AB |
331 | names and scripts are prefixed with "docker" the system will |
332 | automatically run on whichever is configured. | |
333 | ||
4583cdad AB |
334 | The container images are also used to augment the generation of tests |
335 | for testing TCG. See :ref:`checktcg-ref` for more details. | |
336 | ||
9c1f491e | 337 | Docker Prerequisites |
16e79e1b | 338 | ~~~~~~~~~~~~~~~~~~~~ |
4eb99560 FZ |
339 | |
340 | Install "docker" with the system package manager and start the Docker service | |
341 | on your development machine, then make sure you have the privilege to run | |
342 | Docker commands. Typically it means setting up passwordless ``sudo docker`` | |
343 | command or login as root. For example: | |
344 | ||
345 | .. code:: | |
346 | ||
347 | $ sudo yum install docker | |
348 | $ # or `apt-get install docker` for Ubuntu, etc. | |
349 | $ sudo systemctl start docker | |
350 | $ sudo docker ps | |
351 | ||
352 | The last command should print an empty table, to verify the system is ready. | |
353 | ||
354 | An alternative method to set up permissions is by adding the current user to | |
355 | "docker" group and making the docker daemon socket file (by default | |
356 | ``/var/run/docker.sock``) accessible to the group: | |
357 | ||
358 | .. code:: | |
359 | ||
360 | $ sudo groupadd docker | |
29c33cc1 | 361 | $ sudo usermod $USER -a -G docker |
4eb99560 FZ |
362 | $ sudo chown :docker /var/run/docker.sock |
363 | ||
364 | Note that any one of above configurations makes it possible for the user to | |
365 | exploit the whole host with Docker bind mounting or other privileged | |
366 | operations. So only do it on development machines. | |
367 | ||
9c1f491e | 368 | Podman Prerequisites |
16e79e1b | 369 | ~~~~~~~~~~~~~~~~~~~~ |
9c1f491e AB |
370 | |
371 | Install "podman" with the system package manager. | |
372 | ||
373 | .. code:: | |
374 | ||
375 | $ sudo dnf install podman | |
376 | $ podman ps | |
377 | ||
378 | The last command should print an empty table, to verify the system is ready. | |
379 | ||
4eb99560 | 380 | Quickstart |
16e79e1b | 381 | ~~~~~~~~~~ |
4eb99560 | 382 | |
9c1f491e AB |
383 | From source tree, type ``make docker-help`` to see the help. Testing |
384 | can be started without configuring or building QEMU (``configure`` and | |
385 | ``make`` are done in the container, with parameters defined by the | |
386 | make target): | |
4eb99560 FZ |
387 | |
388 | .. code:: | |
389 | ||
9c1f491e | 390 | make docker-test-build@centos8 |
4eb99560 | 391 | |
9c1f491e | 392 | This will create a container instance using the ``centos8`` image (the image |
4eb99560 FZ |
393 | is downloaded and initialized automatically), in which the ``test-build`` job |
394 | is executed. | |
395 | ||
9c1f491e | 396 | Registry |
16e79e1b | 397 | ~~~~~~~~ |
9c1f491e AB |
398 | |
399 | The QEMU project has a container registry hosted by GitLab at | |
400 | ``registry.gitlab.com/qemu-project/qemu`` which will automatically be | |
401 | used to pull in pre-built layers. This avoids unnecessary strain on | |
402 | the distro archives created by multiple developers running the same | |
403 | container build steps over and over again. This can be overridden | |
404 | locally by using the ``NOCACHE`` build option: | |
405 | ||
406 | .. code:: | |
407 | ||
d996f0ae | 408 | make docker-image-debian-arm64-cross NOCACHE=1 |
9c1f491e | 409 | |
4eb99560 | 410 | Images |
16e79e1b | 411 | ~~~~~~ |
4eb99560 | 412 | |
9c1f491e AB |
413 | Along with many other images, the ``centos8`` image is defined in a Dockerfile |
414 | in ``tests/docker/dockerfiles/``, called ``centos8.docker``. ``make docker-help`` | |
4eb99560 FZ |
415 | command will list all the available images. |
416 | ||
4eb99560 FZ |
417 | A ``.pre`` script can be added beside the ``.docker`` file, which will be |
418 | executed before building the image under the build context directory. This is | |
419 | mainly used to do necessary host side setup. One such setup is ``binfmt_misc``, | |
420 | for example, to make qemu-user powered cross build containers work. | |
421 | ||
4ebb040f DB |
422 | Most of the existing Dockerfiles were written by hand, simply by creating a |
423 | a new ``.docker`` file under the ``tests/docker/dockerfiles/`` directory. | |
424 | This has led to an inconsistent set of packages being present across the | |
425 | different containers. | |
426 | ||
427 | Thus going forward, QEMU is aiming to automatically generate the Dockerfiles | |
428 | using the ``lcitool`` program provided by the ``libvirt-ci`` project: | |
429 | ||
430 | https://gitlab.com/libvirt/libvirt-ci | |
431 | ||
fa1ce1dd PB |
432 | ``libvirt-ci`` contains an ``lcitool`` program as well as a list of |
433 | mappings to distribution package names for a wide variety of third | |
434 | party projects. ``lcitool`` applies the mappings to a list of build | |
435 | pre-requisites in ``tests/lcitool/projects/qemu.yml``, determines the | |
436 | list of native packages to install on each distribution, and uses them | |
437 | to generate build environments (dockerfiles and Cirrus CI variable files) | |
438 | that are consistent across OS distribution. | |
4ebb040f DB |
439 | |
440 | ||
441 | Adding new build pre-requisites | |
442 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
443 | ||
fa1ce1dd PB |
444 | When preparing a patch series that adds a new build |
445 | pre-requisite to QEMU, the prerequisites should to be added to | |
446 | ``tests/lcitool/projects/qemu.yml`` in order to make the dependency | |
447 | available in the CI build environments. | |
448 | ||
4ebb040f | 449 | In the simple case where the pre-requisite is already known to ``libvirt-ci`` |
fa1ce1dd | 450 | the following steps are needed: |
4ebb040f DB |
451 | |
452 | * Edit ``tests/lcitool/projects/qemu.yml`` and add the pre-requisite | |
453 | ||
454 | * Run ``make lcitool-refresh`` to re-generate all relevant build environment | |
455 | manifests | |
456 | ||
fa1ce1dd PB |
457 | It may be that ``libvirt-ci`` does not know about the new pre-requisite. |
458 | If that is the case, some extra preparation steps will be required | |
459 | first to contribute the mapping to the ``libvirt-ci`` project: | |
4ebb040f DB |
460 | |
461 | * Fork the ``libvirt-ci`` project on gitlab | |
462 | ||
fa1ce1dd PB |
463 | * Add an entry for the new build prerequisite to |
464 | ``lcitool/facts/mappings.yml``, listing its native package name on as | |
465 | many OS distros as practical. Run ``python -m pytest --regenerate-output`` | |
466 | and check that the changes are correct. | |
4ebb040f | 467 | |
fa1ce1dd PB |
468 | * Commit the ``mappings.yml`` change together with the regenerated test |
469 | files, and submit a merge request to the ``libvirt-ci`` project. | |
470 | Please note in the description that this is a new build pre-requisite | |
471 | desired for use with QEMU. | |
4ebb040f DB |
472 | |
473 | * CI pipeline will run to validate that the changes to ``mappings.yml`` | |
474 | are correct, by attempting to install the newly listed package on | |
475 | all OS distributions supported by ``libvirt-ci``. | |
476 | ||
477 | * Once the merge request is accepted, go back to QEMU and update | |
fa1ce1dd PB |
478 | the ``tests/lcitool/libvirt-ci`` submodule to point to a commit that |
479 | contains the ``mappings.yml`` update. Then add the prerequisite and | |
480 | run ``make lcitool-refresh``. | |
4ebb040f | 481 | |
2a851fca AS |
482 | * Please also trigger gitlab container generation pipelines on your change |
483 | for as many OS distros as practical to make sure that there are no | |
484 | obvious breakages when adding the new pre-requisite. Please see | |
485 | `CI <https://www.qemu.org/docs/master/devel/ci.html>`__ documentation | |
486 | page on how to trigger gitlab CI pipelines on your change. | |
487 | ||
32c06131 PB |
488 | For enterprise distros that default to old, end-of-life versions of the |
489 | Python runtime, QEMU uses a separate set of mappings that work with more | |
490 | recent versions. These can be found in ``tests/lcitool/mappings.yml``. | |
491 | Modifying this file should not be necessary unless the new pre-requisite | |
492 | is a Python library or tool. | |
493 | ||
4ebb040f DB |
494 | |
495 | Adding new OS distros | |
496 | ^^^^^^^^^^^^^^^^^^^^^ | |
497 | ||
498 | In some cases ``libvirt-ci`` will not know about the OS distro that is | |
499 | desired to be tested. Before adding a new OS distro, discuss the proposed | |
500 | addition: | |
501 | ||
502 | * Send a mail to qemu-devel, copying people listed in the | |
503 | MAINTAINERS file for ``Build and test automation``. | |
504 | ||
505 | There are limited CI compute resources available to QEMU, so the | |
506 | cost/benefit tradeoff of adding new OS distros needs to be considered. | |
507 | ||
508 | * File an issue at https://gitlab.com/libvirt/libvirt-ci/-/issues | |
509 | pointing to the qemu-devel mail thread in the archives. | |
510 | ||
511 | This alerts other people who might be interested in the work | |
512 | to avoid duplication, as well as to get feedback from libvirt-ci | |
513 | maintainers on any tips to ease the addition | |
514 | ||
515 | Assuming there is agreement to add a new OS distro then | |
516 | ||
517 | * Fork the ``libvirt-ci`` project on gitlab | |
518 | ||
fa1ce1dd PB |
519 | * Add metadata under ``lcitool/facts/targets/`` for the new OS |
520 | distro. There might be code changes required if the OS distro | |
521 | uses a package format not currently known. The ``libvirt-ci`` | |
522 | maintainers can advise on this when the issue is filed. | |
4ebb040f | 523 | |
fa1ce1dd PB |
524 | * Edit the ``lcitool/facts/mappings.yml`` change to add entries for |
525 | the new OS, listing the native package names for as many packages | |
526 | as practical. Run ``python -m pytest --regenerate-output`` and | |
527 | check that the changes are correct. | |
4ebb040f | 528 | |
fa1ce1dd PB |
529 | * Commit the changes to ``lcitool/facts`` and the regenerated test |
530 | files, and submit a merge request to the ``libvirt-ci`` project. | |
531 | Please note in the description that this is a new build pre-requisite | |
532 | desired for use with QEMU | |
4ebb040f DB |
533 | |
534 | * CI pipeline will run to validate that the changes to ``mappings.yml`` | |
535 | are correct, by attempting to install the newly listed package on | |
536 | all OS distributions supported by ``libvirt-ci``. | |
537 | ||
538 | * Once the merge request is accepted, go back to QEMU and update | |
539 | the ``libvirt-ci`` submodule to point to a commit that contains | |
540 | the ``mappings.yml`` update. | |
541 | ||
542 | ||
4eb99560 | 543 | Tests |
16e79e1b | 544 | ~~~~~ |
4eb99560 FZ |
545 | |
546 | Different tests are added to cover various configurations to build and test | |
547 | QEMU. Docker tests are the executables under ``tests/docker`` named | |
548 | ``test-*``. They are typically shell scripts and are built on top of a shell | |
549 | library, ``tests/docker/common.rc``, which provides helpers to find the QEMU | |
550 | source and build it. | |
551 | ||
9c1f491e | 552 | The full list of tests is printed in the ``make docker-help`` help. |
4eb99560 | 553 | |
4eb99560 | 554 | Debugging a Docker test failure |
16e79e1b | 555 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
4eb99560 FZ |
556 | |
557 | When CI tasks, maintainers or yourself report a Docker test failure, follow the | |
558 | below steps to debug it: | |
559 | ||
560 | 1. Locally reproduce the failure with the reported command line. E.g. run | |
561 | ``make docker-test-mingw@fedora J=8``. | |
562 | 2. Add "V=1" to the command line, try again, to see the verbose output. | |
563 | 3. Further add "DEBUG=1" to the command line. This will pause in a shell prompt | |
564 | in the container right before testing starts. You could either manually | |
565 | build QEMU and run tests from there, or press Ctrl-D to let the Docker | |
566 | testing continue. | |
567 | 4. If you press Ctrl-D, the same building and testing procedure will begin, and | |
568 | will hopefully run into the error again. After that, you will be dropped to | |
569 | the prompt for debug. | |
570 | ||
571 | Options | |
16e79e1b | 572 | ~~~~~~~ |
4eb99560 FZ |
573 | |
574 | Various options can be used to affect how Docker tests are done. The full | |
575 | list is in the ``make docker`` help text. The frequently used ones are: | |
576 | ||
577 | * ``V=1``: the same as in top level ``make``. It will be propagated to the | |
578 | container and enable verbose output. | |
579 | * ``J=$N``: the number of parallel tasks in make commands in the container, | |
580 | similar to the ``-j $N`` option in top level ``make``. (The ``-j`` option in | |
581 | top level ``make`` will not be propagated into the container.) | |
582 | * ``DEBUG=1``: enables debug. See the previous "Debugging a Docker test | |
583 | failure" section. | |
584 | ||
3b6882bd | 585 | Thread Sanitizer |
16e79e1b | 586 | ---------------- |
3b6882bd RF |
587 | |
588 | Thread Sanitizer (TSan) is a tool which can detect data races. QEMU supports | |
589 | building and testing with this tool. | |
590 | ||
591 | For more information on TSan: | |
592 | ||
593 | https://github.com/google/sanitizers/wiki/ThreadSanitizerCppManual | |
594 | ||
595 | Thread Sanitizer in Docker | |
16e79e1b | 596 | ~~~~~~~~~~~~~~~~~~~~~~~~~~ |
171080d8 | 597 | TSan is currently supported in the ubuntu2204 docker. |
3b6882bd RF |
598 | |
599 | The test-tsan test will build using TSan and then run make check. | |
600 | ||
601 | .. code:: | |
602 | ||
171080d8 | 603 | make docker-test-tsan@ubuntu2204 |
3b6882bd RF |
604 | |
605 | TSan warnings under docker are placed in files located at build/tsan/. | |
606 | ||
607 | We recommend using DEBUG=1 to allow launching the test from inside the docker, | |
608 | and to allow review of the warnings generated by TSan. | |
609 | ||
610 | Building and Testing with TSan | |
16e79e1b | 611 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
3b6882bd RF |
612 | |
613 | It is possible to build and test with TSan, with a few additional steps. | |
614 | These steps are normally done automatically in the docker. | |
615 | ||
616 | There is a one time patch needed in clang-9 or clang-10 at this time: | |
617 | ||
618 | .. code:: | |
619 | ||
620 | sed -i 's/^const/static const/g' \ | |
621 | /usr/lib/llvm-10/lib/clang/10.0.0/include/sanitizer/tsan_interface.h | |
622 | ||
623 | To configure the build for TSan: | |
624 | ||
625 | .. code:: | |
626 | ||
627 | ../configure --enable-tsan --cc=clang-10 --cxx=clang++-10 \ | |
628 | --disable-werror --extra-cflags="-O0" | |
629 | ||
630 | The runtime behavior of TSAN is controlled by the TSAN_OPTIONS environment | |
631 | variable. | |
632 | ||
633 | More information on the TSAN_OPTIONS can be found here: | |
634 | ||
635 | https://github.com/google/sanitizers/wiki/ThreadSanitizerFlags | |
636 | ||
637 | For example: | |
638 | ||
639 | .. code:: | |
640 | ||
641 | export TSAN_OPTIONS=suppressions=<path to qemu>/tests/tsan/suppressions.tsan \ | |
642 | detect_deadlocks=false history_size=7 exitcode=0 \ | |
643 | log_path=<build path>/tsan/tsan_warning | |
644 | ||
645 | The above exitcode=0 has TSan continue without error if any warnings are found. | |
646 | This allows for running the test and then checking the warnings afterwards. | |
647 | If you want TSan to stop and exit with error on warnings, use exitcode=66. | |
648 | ||
649 | TSan Suppressions | |
16e79e1b | 650 | ~~~~~~~~~~~~~~~~~ |
3b6882bd RF |
651 | Keep in mind that for any data race warning, although there might be a data race |
652 | detected by TSan, there might be no actual bug here. TSan provides several | |
653 | different mechanisms for suppressing warnings. In general it is recommended | |
654 | to fix the code if possible to eliminate the data race rather than suppress | |
655 | the warning. | |
656 | ||
657 | A few important files for suppressing warnings are: | |
658 | ||
659 | tests/tsan/suppressions.tsan - Has TSan warnings we wish to suppress at runtime. | |
76ca4b58 | 660 | The comment on each suppression will typically indicate why we are |
3b6882bd RF |
661 | suppressing it. More information on the file format can be found here: |
662 | ||
663 | https://github.com/google/sanitizers/wiki/ThreadSanitizerSuppressions | |
664 | ||
665 | tests/tsan/blacklist.tsan - Has TSan warnings we wish to disable | |
666 | at compile time for test or debug. | |
667 | Add flags to configure to enable: | |
668 | ||
669 | "--extra-cflags=-fsanitize-blacklist=<src path>/tests/tsan/blacklist.tsan" | |
670 | ||
671 | More information on the file format can be found here under "Blacklist Format": | |
672 | ||
673 | https://github.com/google/sanitizers/wiki/ThreadSanitizerFlags | |
674 | ||
675 | TSan Annotations | |
16e79e1b | 676 | ~~~~~~~~~~~~~~~~ |
3b6882bd RF |
677 | include/qemu/tsan.h defines annotations. See this file for more descriptions |
678 | of the annotations themselves. Annotations can be used to suppress | |
679 | TSan warnings or give TSan more information so that it can detect proper | |
680 | relationships between accesses of data. | |
681 | ||
682 | Annotation examples can be found here: | |
683 | ||
684 | https://github.com/llvm/llvm-project/tree/master/compiler-rt/test/tsan/ | |
685 | ||
686 | Good files to start with are: annotate_happens_before.cpp and ignore_race.cpp | |
687 | ||
688 | The full set of annotations can be found here: | |
689 | ||
690 | https://github.com/llvm/llvm-project/blob/master/compiler-rt/lib/tsan/rtl/tsan_interface_ann.cpp | |
691 | ||
396408ee AB |
692 | docker-binfmt-image-debian-% targets |
693 | ------------------------------------ | |
694 | ||
695 | It is possible to combine Debian's bootstrap scripts with a configured | |
696 | ``binfmt_misc`` to bootstrap a number of Debian's distros including | |
697 | experimental ports not yet supported by a released OS. This can | |
698 | simplify setting up a rootfs by using docker to contain the foreign | |
699 | rootfs rather than manually invoking chroot. | |
700 | ||
701 | Setting up ``binfmt_misc`` | |
702 | ~~~~~~~~~~~~~~~~~~~~~~~~~~ | |
703 | ||
704 | You can use the script ``qemu-binfmt-conf.sh`` to configure a QEMU | |
705 | user binary to automatically run binaries for the foreign | |
706 | architecture. While the scripts will try their best to work with | |
707 | dynamically linked QEMU's a statically linked one will present less | |
708 | potential complications when copying into the docker image. Modern | |
709 | kernels support the ``F`` (fix binary) flag which will open the QEMU | |
710 | executable on setup and avoids the need to find and re-open in the | |
711 | chroot environment. This is triggered with the ``--persistent`` flag. | |
712 | ||
713 | Example invocation | |
714 | ~~~~~~~~~~~~~~~~~~ | |
715 | ||
716 | For example to setup the HPPA ports builds of Debian:: | |
717 | ||
718 | make docker-binfmt-image-debian-sid-hppa \ | |
719 | DEB_TYPE=sid DEB_ARCH=hppa \ | |
720 | DEB_URL=http://ftp.ports.debian.org/debian-ports/ \ | |
721 | DEB_KEYRING=/usr/share/keyrings/debian-ports-archive-keyring.gpg \ | |
722 | EXECUTABLE=(pwd)/qemu-hppa V=1 | |
723 | ||
724 | The ``DEB_`` variables are substitutions used by | |
725 | ``debian-boostrap.pre`` which is called to do the initial debootstrap | |
726 | of the rootfs before it is copied into the container. The second stage | |
727 | is run as part of the build. The final image will be tagged as | |
728 | ``qemu/debian-sid-hppa``. | |
729 | ||
4eb99560 | 730 | VM testing |
16e79e1b | 731 | ---------- |
4eb99560 FZ |
732 | |
733 | This test suite contains scripts that bootstrap various guest images that have | |
734 | necessary packages to build QEMU. The basic usage is documented in ``Makefile`` | |
4f2f6276 | 735 | help which is displayed with ``make vm-help``. |
4eb99560 FZ |
736 | |
737 | Quickstart | |
16e79e1b | 738 | ~~~~~~~~~~ |
4eb99560 | 739 | |
4f2f6276 | 740 | Run ``make vm-help`` to list available make targets. Invoke a specific make |
4eb99560 FZ |
741 | command to run build test in an image. For example, ``make vm-build-freebsd`` |
742 | will build the source tree in the FreeBSD image. The command can be executed | |
743 | from either the source tree or the build dir; if the former, ``./configure`` is | |
744 | not needed. The command will then generate the test image in ``./tests/vm/`` | |
745 | under the working directory. | |
746 | ||
747 | Note: images created by the scripts accept a well-known RSA key pair for SSH | |
748 | access, so they SHOULD NOT be exposed to external interfaces if you are | |
749 | concerned about attackers taking control of the guest and potentially | |
750 | exploiting a QEMU security bug to compromise the host. | |
751 | ||
1e48931c | 752 | QEMU binaries |
16e79e1b | 753 | ~~~~~~~~~~~~~ |
4eb99560 | 754 | |
c5ba6219 PMD |
755 | By default, ``qemu-system-x86_64`` is searched in $PATH to run the guest. If |
756 | there isn't one, or if it is older than 2.10, the test won't work. In this case, | |
4eb99560 FZ |
757 | provide the QEMU binary in env var: ``QEMU=/path/to/qemu-2.10+``. |
758 | ||
c5ba6219 | 759 | Likewise the path to ``qemu-img`` can be set in QEMU_IMG environment variable. |
1e48931c | 760 | |
4eb99560 | 761 | Make jobs |
16e79e1b | 762 | ~~~~~~~~~ |
4eb99560 FZ |
763 | |
764 | The ``-j$X`` option in the make command line is not propagated into the VM, | |
765 | specify ``J=$X`` to control the make jobs in the guest. | |
766 | ||
767 | Debugging | |
16e79e1b | 768 | ~~~~~~~~~ |
4eb99560 FZ |
769 | |
770 | Add ``DEBUG=1`` and/or ``V=1`` to the make command to allow interactive | |
771 | debugging and verbose output. If this is not enough, see the next section. | |
41e3340a | 772 | ``V=1`` will be propagated down into the make jobs in the guest. |
4eb99560 FZ |
773 | |
774 | Manual invocation | |
16e79e1b | 775 | ~~~~~~~~~~~~~~~~~ |
4eb99560 FZ |
776 | |
777 | Each guest script is an executable script with the same command line options. | |
778 | For example to work with the netbsd guest, use ``$QEMU_SRC/tests/vm/netbsd``: | |
779 | ||
780 | .. code:: | |
781 | ||
782 | $ cd $QEMU_SRC/tests/vm | |
783 | ||
784 | # To bootstrap the image | |
785 | $ ./netbsd --build-image --image /var/tmp/netbsd.img | |
786 | <...> | |
787 | ||
788 | # To run an arbitrary command in guest (the output will not be echoed unless | |
789 | # --debug is added) | |
790 | $ ./netbsd --debug --image /var/tmp/netbsd.img uname -a | |
791 | ||
792 | # To build QEMU in guest | |
793 | $ ./netbsd --debug --image /var/tmp/netbsd.img --build-qemu $QEMU_SRC | |
794 | ||
795 | # To get to an interactive shell | |
796 | $ ./netbsd --interactive --image /var/tmp/netbsd.img sh | |
797 | ||
798 | Adding new guests | |
16e79e1b | 799 | ~~~~~~~~~~~~~~~~~ |
4eb99560 FZ |
800 | |
801 | Please look at existing guest scripts for how to add new guests. | |
802 | ||
803 | Most importantly, create a subclass of BaseVM and implement ``build_image()`` | |
804 | method and define ``BUILD_SCRIPT``, then finally call ``basevm.main()`` from | |
805 | the script's ``main()``. | |
806 | ||
807 | * Usually in ``build_image()``, a template image is downloaded from a | |
808 | predefined URL. ``BaseVM._download_with_cache()`` takes care of the cache and | |
809 | the checksum, so consider using it. | |
810 | ||
811 | * Once the image is downloaded, users, SSH server and QEMU build deps should | |
812 | be set up: | |
813 | ||
814 | - Root password set to ``BaseVM.ROOT_PASS`` | |
815 | - User ``BaseVM.GUEST_USER`` is created, and password set to | |
816 | ``BaseVM.GUEST_PASS`` | |
817 | - SSH service is enabled and started on boot, | |
818 | ``$QEMU_SRC/tests/keys/id_rsa.pub`` is added to ssh's ``authorized_keys`` | |
819 | file of both root and the normal user | |
820 | - DHCP client service is enabled and started on boot, so that it can | |
821 | automatically configure the virtio-net-pci NIC and communicate with QEMU | |
822 | user net (10.0.2.2) | |
823 | - Necessary packages are installed to untar the source tarball and build | |
824 | QEMU | |
825 | ||
826 | * Write a proper ``BUILD_SCRIPT`` template, which should be a shell script that | |
827 | untars a raw virtio-blk block device, which is the tarball data blob of the | |
828 | QEMU source tree, then configure/build it. Running "make check" is also | |
829 | recommended. | |
830 | ||
831 | Image fuzzer testing | |
16e79e1b | 832 | -------------------- |
4eb99560 FZ |
833 | |
834 | An image fuzzer was added to exercise format drivers. Currently only qcow2 is | |
835 | supported. To start the fuzzer, run | |
836 | ||
837 | .. code:: | |
838 | ||
839 | tests/image-fuzzer/runner.py -c '[["qemu-img", "info", "$test_img"]]' /tmp/test qcow2 | |
840 | ||
c5ba6219 | 841 | Alternatively, some command different from ``qemu-img info`` can be tested, by |
4eb99560 | 842 | changing the ``-c`` option. |
c3d7e8c9 | 843 | |
bbbd9b6e WR |
844 | Integration tests using the Avocado Framework |
845 | --------------------------------------------- | |
c3d7e8c9 | 846 | |
bbbd9b6e WR |
847 | The ``tests/avocado`` directory hosts integration tests. They're usually |
848 | higher level tests, and may interact with external resources and with | |
849 | various guest operating systems. | |
c3d7e8c9 CR |
850 | |
851 | These tests are written using the Avocado Testing Framework (which must | |
852 | be installed separately) in conjunction with a the ``avocado_qemu.Test`` | |
bbbd9b6e | 853 | class, implemented at ``tests/avocado/avocado_qemu``. |
c3d7e8c9 CR |
854 | |
855 | Tests based on ``avocado_qemu.Test`` can easily: | |
856 | ||
857 | * Customize the command line arguments given to the convenience | |
858 | ``self.vm`` attribute (a QEMUMachine instance) | |
859 | ||
860 | * Interact with the QEMU monitor, send QMP commands and check | |
861 | their results | |
862 | ||
863 | * Interact with the guest OS, using the convenience console device | |
864 | (which may be useful to assert the effectiveness and correctness of | |
865 | command line arguments or QMP commands) | |
866 | ||
867 | * Interact with external data files that accompany the test itself | |
868 | (see ``self.get_data()``) | |
869 | ||
870 | * Download (and cache) remote data files, such as firmware and kernel | |
871 | images | |
872 | ||
873 | * Have access to a library of guest OS images (by means of the | |
874 | ``avocado.utils.vmimage`` library) | |
875 | ||
876 | * Make use of various other test related utilities available at the | |
877 | test class itself and at the utility library: | |
878 | ||
879 | - http://avocado-framework.readthedocs.io/en/latest/api/test/avocado.html#avocado.Test | |
880 | - http://avocado-framework.readthedocs.io/en/latest/api/utils/avocado.utils.html | |
881 | ||
a56931ee | 882 | Running tests |
16e79e1b | 883 | ~~~~~~~~~~~~~ |
a56931ee | 884 | |
bbbd9b6e | 885 | You can run the avocado tests simply by executing: |
a56931ee CR |
886 | |
887 | .. code:: | |
888 | ||
bbbd9b6e | 889 | make check-avocado |
a56931ee | 890 | |
9c6692db JS |
891 | This involves the automatic installation, from PyPI, of all the |
892 | necessary avocado-framework dependencies into the QEMU venv within the | |
893 | build tree (at ``./pyvenv``). Test results are also saved within the | |
a56931ee | 894 | build tree (at ``tests/results``). |
c3d7e8c9 | 895 | |
a56931ee CR |
896 | Note: the build environment must be using a Python 3 stack, and have |
897 | the ``venv`` and ``pip`` packages installed. If necessary, make sure | |
898 | ``configure`` is called with ``--python=`` and that those modules are | |
899 | available. On Debian and Ubuntu based systems, depending on the | |
900 | specific version, they may be on packages named ``python3-venv`` and | |
901 | ``python3-pip``. | |
902 | ||
23022794 | 903 | It is also possible to run tests based on tags using the |
bbbd9b6e | 904 | ``make check-avocado`` command and the ``AVOCADO_TAGS`` environment |
23022794 WR |
905 | variable: |
906 | ||
907 | .. code:: | |
908 | ||
bbbd9b6e | 909 | make check-avocado AVOCADO_TAGS=quick |
23022794 WR |
910 | |
911 | Note that tags separated with commas have an AND behavior, while tags | |
912 | separated by spaces have an OR behavior. For more information on Avocado | |
913 | tags, see: | |
914 | ||
915 | https://avocado-framework.readthedocs.io/en/latest/guides/user/chapters/tags.html | |
916 | ||
94c71462 | 917 | To run a single test file, a couple of them, or a test within a file |
bbbd9b6e | 918 | using the ``make check-avocado`` command, set the ``AVOCADO_TESTS`` |
94c71462 WR |
919 | environment variable with the test files or test names. To run all |
920 | tests from a single file, use: | |
921 | ||
922 | .. code:: | |
923 | ||
bbbd9b6e | 924 | make check-avocado AVOCADO_TESTS=$FILEPATH |
94c71462 WR |
925 | |
926 | The same is valid to run tests from multiple test files: | |
927 | ||
928 | .. code:: | |
929 | ||
bbbd9b6e | 930 | make check-avocado AVOCADO_TESTS='$FILEPATH1 $FILEPATH2' |
94c71462 WR |
931 | |
932 | To run a single test within a file, use: | |
933 | ||
934 | .. code:: | |
935 | ||
bbbd9b6e | 936 | make check-avocado AVOCADO_TESTS=$FILEPATH:$TESTCLASS.$TESTNAME |
94c71462 WR |
937 | |
938 | The same is valid to run single tests from multiple test files: | |
939 | ||
940 | .. code:: | |
941 | ||
bbbd9b6e | 942 | make check-avocado AVOCADO_TESTS='$FILEPATH1:$TESTCLASS1.$TESTNAME1 $FILEPATH2:$TESTCLASS2.$TESTNAME2' |
94c71462 | 943 | |
a56931ee CR |
944 | The scripts installed inside the virtual environment may be used |
945 | without an "activation". For instance, the Avocado test runner | |
946 | may be invoked by running: | |
947 | ||
948 | .. code:: | |
949 | ||
9c6692db | 950 | pyvenv/bin/avocado run $OPTION1 $OPTION2 tests/avocado/ |
a56931ee | 951 | |
bbbd9b6e | 952 | Note that if ``make check-avocado`` was not executed before, it is |
6676f18f WR |
953 | possible to create the Python virtual environment with the dependencies |
954 | needed running: | |
955 | ||
956 | .. code:: | |
957 | ||
958 | make check-venv | |
959 | ||
960 | It is also possible to run tests from a single file or a single test within | |
961 | a test file. To run tests from a single file within the build tree, use: | |
962 | ||
963 | .. code:: | |
964 | ||
9c6692db | 965 | pyvenv/bin/avocado run tests/avocado/$TESTFILE |
6676f18f WR |
966 | |
967 | To run a single test within a test file, use: | |
968 | ||
969 | .. code:: | |
970 | ||
9c6692db | 971 | pyvenv/bin/avocado run tests/avocado/$TESTFILE:$TESTCLASS.$TESTNAME |
6676f18f WR |
972 | |
973 | Valid test names are visible in the output from any previous execution | |
bbbd9b6e | 974 | of Avocado or ``make check-avocado``, and can also be queried using: |
6676f18f WR |
975 | |
976 | .. code:: | |
977 | ||
9c6692db | 978 | pyvenv/bin/avocado list tests/avocado |
6676f18f | 979 | |
a56931ee | 980 | Manual Installation |
16e79e1b | 981 | ~~~~~~~~~~~~~~~~~~~ |
a56931ee CR |
982 | |
983 | To manually install Avocado and its dependencies, run: | |
c3d7e8c9 CR |
984 | |
985 | .. code:: | |
986 | ||
987 | pip install --user avocado-framework | |
988 | ||
989 | Alternatively, follow the instructions on this link: | |
990 | ||
4c9ac672 | 991 | https://avocado-framework.readthedocs.io/en/latest/guides/user/chapters/installing.html |
c3d7e8c9 CR |
992 | |
993 | Overview | |
16e79e1b | 994 | ~~~~~~~~ |
c3d7e8c9 | 995 | |
bbbd9b6e | 996 | The ``tests/avocado/avocado_qemu`` directory provides the |
805fac52 CR |
997 | ``avocado_qemu`` Python module, containing the ``avocado_qemu.Test`` |
998 | class. Here's a simple usage example: | |
c3d7e8c9 CR |
999 | |
1000 | .. code:: | |
1001 | ||
2283b627 | 1002 | from avocado_qemu import QemuSystemTest |
c3d7e8c9 CR |
1003 | |
1004 | ||
2283b627 | 1005 | class Version(QemuSystemTest): |
c3d7e8c9 | 1006 | """ |
c3d7e8c9 CR |
1007 | :avocado: tags=quick |
1008 | """ | |
1009 | def test_qmp_human_info_version(self): | |
1010 | self.vm.launch() | |
1011 | res = self.vm.command('human-monitor-command', | |
1012 | command_line='info version') | |
1013 | self.assertRegexpMatches(res, r'^(\d+\.\d+\.\d)') | |
1014 | ||
1015 | To execute your test, run: | |
1016 | ||
1017 | .. code:: | |
1018 | ||
1019 | avocado run version.py | |
1020 | ||
1021 | Tests may be classified according to a convention by using docstring | |
1022 | directives such as ``:avocado: tags=TAG1,TAG2``. To run all tests | |
1023 | in the current directory, tagged as "quick", run: | |
1024 | ||
1025 | .. code:: | |
1026 | ||
1027 | avocado run -t quick . | |
1028 | ||
1029 | The ``avocado_qemu.Test`` base test class | |
16e79e1b | 1030 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
c3d7e8c9 CR |
1031 | |
1032 | The ``avocado_qemu.Test`` class has a number of characteristics that | |
1033 | are worth being mentioned right away. | |
1034 | ||
1035 | First of all, it attempts to give each test a ready to use QEMUMachine | |
1036 | instance, available at ``self.vm``. Because many tests will tweak the | |
1037 | QEMU command line, launching the QEMUMachine (by using ``self.vm.launch()``) | |
1038 | is left to the test writer. | |
1039 | ||
b7287d42 CC |
1040 | The base test class has also support for tests with more than one |
1041 | QEMUMachine. The way to get machines is through the ``self.get_vm()`` | |
1042 | method which will return a QEMUMachine instance. The ``self.get_vm()`` | |
1043 | method accepts arguments that will be passed to the QEMUMachine creation | |
1e235eda | 1044 | and also an optional ``name`` attribute so you can identify a specific |
b7287d42 CC |
1045 | machine and get it more than once through the tests methods. A simple |
1046 | and hypothetical example follows: | |
1047 | ||
1048 | .. code:: | |
1049 | ||
2283b627 | 1050 | from avocado_qemu import QemuSystemTest |
b7287d42 CC |
1051 | |
1052 | ||
2283b627 | 1053 | class MultipleMachines(QemuSystemTest): |
b7287d42 CC |
1054 | def test_multiple_machines(self): |
1055 | first_machine = self.get_vm() | |
1056 | second_machine = self.get_vm() | |
1057 | self.get_vm(name='third_machine').launch() | |
1058 | ||
1059 | first_machine.launch() | |
1060 | second_machine.launch() | |
1061 | ||
1062 | first_res = first_machine.command( | |
1063 | 'human-monitor-command', | |
1064 | command_line='info version') | |
1065 | ||
1066 | second_res = second_machine.command( | |
1067 | 'human-monitor-command', | |
1068 | command_line='info version') | |
1069 | ||
1070 | third_res = self.get_vm(name='third_machine').command( | |
1071 | 'human-monitor-command', | |
1072 | command_line='info version') | |
1073 | ||
1074 | self.assertEquals(first_res, second_res, third_res) | |
1075 | ||
1076 | At test "tear down", ``avocado_qemu.Test`` handles all the QEMUMachines | |
c3d7e8c9 CR |
1077 | shutdown. |
1078 | ||
1e4e7efa | 1079 | The ``avocado_qemu.LinuxTest`` base test class |
16e79e1b | 1080 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
1e4e7efa CR |
1081 | |
1082 | The ``avocado_qemu.LinuxTest`` is further specialization of the | |
1083 | ``avocado_qemu.Test`` class, so it contains all the characteristics of | |
1084 | the later plus some extra features. | |
1085 | ||
1086 | First of all, this base class is intended for tests that need to | |
1087 | interact with a fully booted and operational Linux guest. At this | |
1088 | time, it uses a Fedora 31 guest image. The most basic example looks | |
1089 | like this: | |
1090 | ||
1091 | .. code:: | |
1092 | ||
1093 | from avocado_qemu import LinuxTest | |
1094 | ||
1095 | ||
1096 | class SomeTest(LinuxTest): | |
1097 | ||
1098 | def test(self): | |
1099 | self.launch_and_wait() | |
1100 | self.ssh_command('some_command_to_be_run_in_the_guest') | |
1101 | ||
1102 | Please refer to tests that use ``avocado_qemu.LinuxTest`` under | |
bbbd9b6e | 1103 | ``tests/avocado`` for more examples. |
1e4e7efa | 1104 | |
c3d7e8c9 CR |
1105 | QEMUMachine |
1106 | ~~~~~~~~~~~ | |
1107 | ||
1108 | The QEMUMachine API is already widely used in the Python iotests, | |
1109 | device-crash-test and other Python scripts. It's a wrapper around the | |
1110 | execution of a QEMU binary, giving its users: | |
1111 | ||
1112 | * the ability to set command line arguments to be given to the QEMU | |
1113 | binary | |
1114 | ||
1115 | * a ready to use QMP connection and interface, which can be used to | |
1116 | send commands and inspect its results, as well as asynchronous | |
1117 | events | |
1118 | ||
1119 | * convenience methods to set commonly used command line arguments in | |
1120 | a more succinct and intuitive way | |
1121 | ||
1122 | QEMU binary selection | |
16e79e1b | 1123 | ^^^^^^^^^^^^^^^^^^^^^ |
c3d7e8c9 CR |
1124 | |
1125 | The QEMU binary used for the ``self.vm`` QEMUMachine instance will | |
1126 | primarily depend on the value of the ``qemu_bin`` parameter. If it's | |
1127 | not explicitly set, its default value will be the result of a dynamic | |
1128 | probe in the same source tree. A suitable binary will be one that | |
1129 | targets the architecture matching host machine. | |
1130 | ||
1131 | Based on this description, test writers will usually rely on one of | |
1132 | the following approaches: | |
1133 | ||
1134 | 1) Set ``qemu_bin``, and use the given binary | |
1135 | ||
1136 | 2) Do not set ``qemu_bin``, and use a QEMU binary named like | |
64ed6f92 | 1137 | "qemu-system-${arch}", either in the current |
c3d7e8c9 CR |
1138 | working directory, or in the current source tree. |
1139 | ||
1140 | The resulting ``qemu_bin`` value will be preserved in the | |
1141 | ``avocado_qemu.Test`` as an attribute with the same name. | |
1142 | ||
1143 | Attribute reference | |
16e79e1b PB |
1144 | ~~~~~~~~~~~~~~~~~~~ |
1145 | ||
1146 | Test | |
1147 | ^^^^ | |
c3d7e8c9 CR |
1148 | |
1149 | Besides the attributes and methods that are part of the base | |
1150 | ``avocado.Test`` class, the following attributes are available on any | |
1151 | ``avocado_qemu.Test`` instance. | |
1152 | ||
1153 | vm | |
16e79e1b | 1154 | '' |
c3d7e8c9 CR |
1155 | |
1156 | A QEMUMachine instance, initially configured according to the given | |
1157 | ``qemu_bin`` parameter. | |
1158 | ||
2c44d68f | 1159 | arch |
16e79e1b | 1160 | '''' |
2c44d68f CR |
1161 | |
1162 | The architecture can be used on different levels of the stack, e.g. by | |
1163 | the framework or by the test itself. At the framework level, it will | |
1164 | currently influence the selection of a QEMU binary (when one is not | |
1165 | explicitly given). | |
1166 | ||
1167 | Tests are also free to use this attribute value, for their own needs. | |
1168 | A test may, for instance, use the same value when selecting the | |
1169 | architecture of a kernel or disk image to boot a VM with. | |
1170 | ||
1171 | The ``arch`` attribute will be set to the test parameter of the same | |
b910545f CR |
1172 | name. If one is not given explicitly, it will either be set to |
1173 | ``None``, or, if the test is tagged with one (and only one) | |
1174 | ``:avocado: tags=arch:VALUE`` tag, it will be set to ``VALUE``. | |
2c44d68f | 1175 | |
20bbf846 | 1176 | cpu |
16e79e1b | 1177 | ''' |
20bbf846 WSM |
1178 | |
1179 | The cpu model that will be set to all QEMUMachine instances created | |
1180 | by the test. | |
1181 | ||
1182 | The ``cpu`` attribute will be set to the test parameter of the same | |
1183 | name. If one is not given explicitly, it will either be set to | |
1184 | ``None ``, or, if the test is tagged with one (and only one) | |
1185 | ``:avocado: tags=cpu:VALUE`` tag, it will be set to ``VALUE``. | |
1186 | ||
ba21bde9 | 1187 | machine |
16e79e1b | 1188 | ''''''' |
ba21bde9 CR |
1189 | |
1190 | The machine type that will be set to all QEMUMachine instances created | |
1191 | by the test. | |
1192 | ||
1193 | The ``machine`` attribute will be set to the test parameter of the same | |
1194 | name. If one is not given explicitly, it will either be set to | |
1195 | ``None``, or, if the test is tagged with one (and only one) | |
1196 | ``:avocado: tags=machine:VALUE`` tag, it will be set to ``VALUE``. | |
1197 | ||
c3d7e8c9 | 1198 | qemu_bin |
16e79e1b | 1199 | '''''''' |
c3d7e8c9 CR |
1200 | |
1201 | The preserved value of the ``qemu_bin`` parameter or the result of the | |
1202 | dynamic probe for a QEMU binary in the current working directory or | |
1203 | source tree. | |
1204 | ||
d5adf9d5 | 1205 | LinuxTest |
16e79e1b | 1206 | ^^^^^^^^^ |
d5adf9d5 CR |
1207 | |
1208 | Besides the attributes present on the ``avocado_qemu.Test`` base | |
1209 | class, the ``avocado_qemu.LinuxTest`` adds the following attributes: | |
1210 | ||
1211 | distro | |
16e79e1b | 1212 | '''''' |
d5adf9d5 CR |
1213 | |
1214 | The name of the Linux distribution used as the guest image for the | |
1215 | test. The name should match the **Provider** column on the list | |
1216 | of images supported by the avocado.utils.vmimage library: | |
1217 | ||
1218 | https://avocado-framework.readthedocs.io/en/latest/guides/writer/libs/vmimage.html#supported-images | |
1219 | ||
1220 | distro_version | |
16e79e1b | 1221 | '''''''''''''' |
d5adf9d5 CR |
1222 | |
1223 | The version of the Linux distribution as the guest image for the | |
1224 | test. The name should match the **Version** column on the list | |
1225 | of images supported by the avocado.utils.vmimage library: | |
1226 | ||
1227 | https://avocado-framework.readthedocs.io/en/latest/guides/writer/libs/vmimage.html#supported-images | |
1228 | ||
1229 | distro_checksum | |
16e79e1b | 1230 | ''''''''''''''' |
d5adf9d5 CR |
1231 | |
1232 | The sha256 hash of the guest image file used for the test. | |
1233 | ||
1234 | If this value is not set in the code or by a test parameter (with the | |
1235 | same name), no validation on the integrity of the image will be | |
1236 | performed. | |
1237 | ||
c3d7e8c9 | 1238 | Parameter reference |
16e79e1b | 1239 | ~~~~~~~~~~~~~~~~~~~ |
c3d7e8c9 CR |
1240 | |
1241 | To understand how Avocado parameters are accessed by tests, and how | |
1242 | they can be passed to tests, please refer to:: | |
1243 | ||
4c9ac672 | 1244 | https://avocado-framework.readthedocs.io/en/latest/guides/writer/chapters/writing.html#accessing-test-parameters |
c3d7e8c9 CR |
1245 | |
1246 | Parameter values can be easily seen in the log files, and will look | |
1247 | like the following: | |
1248 | ||
1249 | .. code:: | |
1250 | ||
64ed6f92 | 1251 | PARAMS (key=qemu_bin, path=*, default=./qemu-system-x86_64) => './qemu-system-x86_64 |
c3d7e8c9 | 1252 | |
16e79e1b PB |
1253 | Test |
1254 | ^^^^ | |
1255 | ||
2c44d68f | 1256 | arch |
16e79e1b | 1257 | '''' |
2c44d68f CR |
1258 | |
1259 | The architecture that will influence the selection of a QEMU binary | |
1260 | (when one is not explicitly given). | |
1261 | ||
1262 | Tests are also free to use this parameter value, for their own needs. | |
1263 | A test may, for instance, use the same value when selecting the | |
1264 | architecture of a kernel or disk image to boot a VM with. | |
1265 | ||
1266 | This parameter has a direct relation with the ``arch`` attribute. If | |
1267 | not given, it will default to None. | |
1268 | ||
20bbf846 | 1269 | cpu |
16e79e1b | 1270 | ''' |
20bbf846 WSM |
1271 | |
1272 | The cpu model that will be set to all QEMUMachine instances created | |
1273 | by the test. | |
1274 | ||
ba21bde9 | 1275 | machine |
16e79e1b | 1276 | ''''''' |
ba21bde9 CR |
1277 | |
1278 | The machine type that will be set to all QEMUMachine instances created | |
1279 | by the test. | |
1280 | ||
c3d7e8c9 | 1281 | qemu_bin |
16e79e1b | 1282 | '''''''' |
c3d7e8c9 CR |
1283 | |
1284 | The exact QEMU binary to be used on QEMUMachine. | |
1285 | ||
d5adf9d5 | 1286 | LinuxTest |
16e79e1b | 1287 | ^^^^^^^^^ |
d5adf9d5 CR |
1288 | |
1289 | Besides the parameters present on the ``avocado_qemu.Test`` base | |
1290 | class, the ``avocado_qemu.LinuxTest`` adds the following parameters: | |
1291 | ||
1292 | distro | |
16e79e1b | 1293 | '''''' |
d5adf9d5 CR |
1294 | |
1295 | The name of the Linux distribution used as the guest image for the | |
1296 | test. The name should match the **Provider** column on the list | |
1297 | of images supported by the avocado.utils.vmimage library: | |
1298 | ||
1299 | https://avocado-framework.readthedocs.io/en/latest/guides/writer/libs/vmimage.html#supported-images | |
1300 | ||
1301 | distro_version | |
16e79e1b | 1302 | '''''''''''''' |
d5adf9d5 CR |
1303 | |
1304 | The version of the Linux distribution as the guest image for the | |
1305 | test. The name should match the **Version** column on the list | |
1306 | of images supported by the avocado.utils.vmimage library: | |
1307 | ||
1308 | https://avocado-framework.readthedocs.io/en/latest/guides/writer/libs/vmimage.html#supported-images | |
1309 | ||
1310 | distro_checksum | |
16e79e1b | 1311 | ''''''''''''''' |
d5adf9d5 CR |
1312 | |
1313 | The sha256 hash of the guest image file used for the test. | |
1314 | ||
1315 | If this value is not set in the code or by this parameter no | |
1316 | validation on the integrity of the image will be performed. | |
1317 | ||
cf5891ec | 1318 | Skipping tests |
16e79e1b PB |
1319 | ~~~~~~~~~~~~~~ |
1320 | ||
cf5891ec WSM |
1321 | The Avocado framework provides Python decorators which allow for easily skip |
1322 | tests running under certain conditions. For example, on the lack of a binary | |
1323 | on the test system or when the running environment is a CI system. For further | |
1324 | information about those decorators, please refer to:: | |
1325 | ||
1326 | https://avocado-framework.readthedocs.io/en/latest/guides/writer/chapters/writing.html#skipping-tests | |
1327 | ||
1328 | While the conditions for skipping tests are often specifics of each one, there | |
1329 | are recurring scenarios identified by the QEMU developers and the use of | |
1330 | environment variables became a kind of standard way to enable/disable tests. | |
1331 | ||
1332 | Here is a list of the most used variables: | |
1333 | ||
1334 | AVOCADO_ALLOW_LARGE_STORAGE | |
16e79e1b | 1335 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
cf5891ec | 1336 | Tests which are going to fetch or produce assets considered *large* are not |
1e235eda | 1337 | going to run unless that ``AVOCADO_ALLOW_LARGE_STORAGE=1`` is exported on |
cf5891ec WSM |
1338 | the environment. |
1339 | ||
1340 | The definition of *large* is a bit arbitrary here, but it usually means an | |
1341 | asset which occupies at least 1GB of size on disk when uncompressed. | |
1342 | ||
1343 | AVOCADO_ALLOW_UNTRUSTED_CODE | |
16e79e1b | 1344 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
cf5891ec WSM |
1345 | There are tests which will boot a kernel image or firmware that can be |
1346 | considered not safe to run on the developer's workstation, thus they are | |
1347 | skipped by default. The definition of *not safe* is also arbitrary but | |
1348 | usually it means a blob which either its source or build process aren't | |
1349 | public available. | |
1350 | ||
1e235eda | 1351 | You should export ``AVOCADO_ALLOW_UNTRUSTED_CODE=1`` on the environment in |
cf5891ec WSM |
1352 | order to allow tests which make use of those kind of assets. |
1353 | ||
1354 | AVOCADO_TIMEOUT_EXPECTED | |
16e79e1b | 1355 | ^^^^^^^^^^^^^^^^^^^^^^^^ |
cf5891ec WSM |
1356 | The Avocado framework has a timeout mechanism which interrupts tests to avoid the |
1357 | test suite of getting stuck. The timeout value can be set via test parameter or | |
1358 | property defined in the test class, for further details:: | |
1359 | ||
1360 | https://avocado-framework.readthedocs.io/en/latest/guides/writer/chapters/writing.html#setting-a-test-timeout | |
1361 | ||
1362 | Even though the timeout can be set by the test developer, there are some tests | |
1363 | that may not have a well-defined limit of time to finish under certain | |
1364 | conditions. For example, tests that take longer to execute when QEMU is | |
1e235eda | 1365 | compiled with debug flags. Therefore, the ``AVOCADO_TIMEOUT_EXPECTED`` variable |
cf5891ec WSM |
1366 | has been used to determine whether those tests should run or not. |
1367 | ||
1368 | GITLAB_CI | |
16e79e1b | 1369 | ^^^^^^^^^ |
cf5891ec WSM |
1370 | A number of tests are flagged to not run on the GitLab CI. Usually because |
1371 | they proved to the flaky or there are constraints on the CI environment which | |
1372 | would make them fail. If you encounter a similar situation then use that | |
1373 | variable as shown on the code snippet below to skip the test: | |
1374 | ||
1375 | .. code:: | |
1376 | ||
1377 | @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab') | |
1378 | def test(self): | |
1379 | do_something() | |
1380 | ||
c3d7e8c9 | 1381 | Uninstalling Avocado |
16e79e1b | 1382 | ~~~~~~~~~~~~~~~~~~~~ |
c3d7e8c9 | 1383 | |
a56931ee CR |
1384 | If you've followed the manual installation instructions above, you can |
1385 | easily uninstall Avocado. Start by listing the packages you have | |
1386 | installed:: | |
c3d7e8c9 CR |
1387 | |
1388 | pip list --user | |
1389 | ||
1390 | And remove any package you want with:: | |
1391 | ||
1392 | pip uninstall <package_name> | |
a56931ee | 1393 | |
bbbd9b6e | 1394 | If you've used ``make check-avocado``, the Python virtual environment where |
a56931ee | 1395 | Avocado is installed will be cleaned up as part of ``make check-clean``. |
f8ed349e | 1396 | |
4583cdad AB |
1397 | .. _checktcg-ref: |
1398 | ||
f8ed349e | 1399 | Testing with "make check-tcg" |
16e79e1b | 1400 | ----------------------------- |
f8ed349e AB |
1401 | |
1402 | The check-tcg tests are intended for simple smoke tests of both | |
1403 | linux-user and softmmu TCG functionality. However to build test | |
1404 | programs for guest targets you need to have cross compilers available. | |
1405 | If your distribution supports cross compilers you can do something as | |
1406 | simple as:: | |
1407 | ||
1408 | apt install gcc-aarch64-linux-gnu | |
1409 | ||
1410 | The configure script will automatically pick up their presence. | |
1411 | Sometimes compilers have slightly odd names so the availability of | |
1412 | them can be prompted by passing in the appropriate configure option | |
1413 | for the architecture in question, for example:: | |
1414 | ||
1415 | $(configure) --cross-cc-aarch64=aarch64-cc | |
1416 | ||
479ca4cc | 1417 | There is also a ``--cross-cc-cflags-ARCH`` flag in case additional |
f8ed349e AB |
1418 | compiler flags are needed to build for a given target. |
1419 | ||
663a041e AB |
1420 | If you have the ability to run containers as the user the build system |
1421 | will automatically use them where no system compiler is available. For | |
1422 | architectures where we also support building QEMU we will generally | |
1423 | use the same container to build tests. However there are a number of | |
1424 | additional containers defined that have a minimal cross-build | |
1425 | environment that is only suitable for building test cases. Sometimes | |
1426 | we may use a bleeding edge distribution for compiler features needed | |
1427 | for test cases that aren't yet in the LTS distros we support for QEMU | |
1428 | itself. | |
1429 | ||
1430 | See :ref:`container-ref` for more details. | |
f8ed349e AB |
1431 | |
1432 | Running subset of tests | |
16e79e1b | 1433 | ~~~~~~~~~~~~~~~~~~~~~~~ |
f8ed349e AB |
1434 | |
1435 | You can build the tests for one architecture:: | |
1436 | ||
1437 | make build-tcg-tests-$TARGET | |
1438 | ||
1439 | And run with:: | |
1440 | ||
1441 | make run-tcg-tests-$TARGET | |
1442 | ||
1443 | Adding ``V=1`` to the invocation will show the details of how to | |
1444 | invoke QEMU for the test which is useful for debugging tests. | |
1445 | ||
1446 | TCG test dependencies | |
16e79e1b | 1447 | ~~~~~~~~~~~~~~~~~~~~~ |
f8ed349e AB |
1448 | |
1449 | The TCG tests are deliberately very light on dependencies and are | |
1450 | either totally bare with minimal gcc lib support (for softmmu tests) | |
1451 | or just glibc (for linux-user tests). This is because getting a cross | |
1452 | compiler to work with additional libraries can be challenging. | |
1453 | ||
1454 | Other TCG Tests | |
1455 | --------------- | |
1456 | ||
1457 | There are a number of out-of-tree test suites that are used for more | |
1458 | extensive testing of processor features. | |
1459 | ||
1460 | KVM Unit Tests | |
1461 | ~~~~~~~~~~~~~~ | |
1462 | ||
1463 | The KVM unit tests are designed to run as a Guest OS under KVM but | |
1464 | there is no reason why they can't exercise the TCG as well. It | |
1465 | provides a minimal OS kernel with hooks for enabling the MMU as well | |
1466 | as reporting test results via a special device:: | |
1467 | ||
1468 | https://git.kernel.org/pub/scm/virt/kvm/kvm-unit-tests.git | |
1469 | ||
1470 | Linux Test Project | |
1471 | ~~~~~~~~~~~~~~~~~~ | |
1472 | ||
1473 | The LTP is focused on exercising the syscall interface of a Linux | |
1474 | kernel. It checks that syscalls behave as documented and strives to | |
1475 | exercise as many corner cases as possible. It is a useful test suite | |
1476 | to run to exercise QEMU's linux-user code:: | |
1477 | ||
1478 | https://linux-test-project.github.io/ | |
9fce3601 PB |
1479 | |
1480 | GCC gcov support | |
16e79e1b | 1481 | ---------------- |
9fce3601 PB |
1482 | |
1483 | ``gcov`` is a GCC tool to analyze the testing coverage by | |
1484 | instrumenting the tested code. To use it, configure QEMU with | |
1485 | ``--enable-gcov`` option and build. Then run the tests as usual. | |
1486 | ||
1487 | If you want to gather coverage information on a single test the ``make | |
1488 | clean-gcda`` target can be used to delete any existing coverage | |
1489 | information before running a single test. | |
1490 | ||
1491 | You can generate a HTML coverage report by executing ``make | |
1492 | coverage-html`` which will create | |
1493 | ``meson-logs/coveragereport/index.html``. | |
1494 | ||
1495 | Further analysis can be conducted by running the ``gcov`` command | |
1496 | directly on the various .gcda output files. Please read the ``gcov`` | |
1497 | documentation for more information. |