]>
Commit | Line | Data |
---|---|---|
923072b8 | 1 | # Instrumentation-based Code Coverage |
17df50a5 | 2 | |
17df50a5 XL |
3 | ## Introduction |
4 | ||
5 | The Rust compiler includes two code coverage implementations: | |
6 | ||
7 | - A GCC-compatible, gcov-based coverage implementation, enabled with `-Z profile`, which derives coverage data based on DebugInfo. | |
5099ac24 | 8 | - A source-based code coverage implementation, enabled with `-C instrument-coverage`, which uses LLVM's native, efficient coverage instrumentation to generate very precise coverage data. |
17df50a5 | 9 | |
5099ac24 | 10 | This document describes how to enable and use the LLVM instrumentation-based coverage, via the `-C instrument-coverage` compiler flag. |
17df50a5 XL |
11 | |
12 | ## How it works | |
13 | ||
5099ac24 | 14 | When `-C instrument-coverage` is enabled, the Rust compiler enhances rust-based libraries and binaries by: |
17df50a5 XL |
15 | |
16 | - Automatically injecting calls to an LLVM intrinsic ([`llvm.instrprof.increment`]), at functions and branches in compiled code, to increment counters when conditional sections of code are executed. | |
a2a8927a | 17 | - Embedding additional information in the data section of each library and binary (using the [LLVM Code Coverage Mapping Format] _Version 5_, if compiling with LLVM 12, or _Version 6_, if compiling with LLVM 13 or higher), to define the code regions (start and end positions in the source code) being counted. |
17df50a5 XL |
18 | |
19 | When running a coverage-instrumented program, the counter values are written to a `profraw` file at program termination. LLVM bundles tools that read the counter results, combine those results with the coverage map (embedded in the program binary), and generate coverage reports in multiple formats. | |
20 | ||
21 | [`llvm.instrprof.increment`]: https://llvm.org/docs/LangRef.html#llvm-instrprof-increment-intrinsic | |
22 | [llvm code coverage mapping format]: https://llvm.org/docs/CoverageMappingFormat.html | |
23 | ||
5099ac24 | 24 | > **Note**: `-C instrument-coverage` also automatically enables `-C symbol-mangling-version=v0` (tracking issue [#60705]). The `v0` symbol mangler is strongly recommended. The `v0` demangler can be overridden by explicitly adding `-Z unstable-options -C symbol-mangling-version=legacy`. |
17df50a5 XL |
25 | |
26 | [#60705]: https://github.com/rust-lang/rust/issues/60705 | |
27 | ||
28 | ## Enable coverage profiling in the Rust compiler | |
29 | ||
5099ac24 | 30 | Rust's source-based code coverage requires the Rust "profiler runtime". Without it, compiling with `-C instrument-coverage` generates an error that the profiler runtime is missing. |
17df50a5 XL |
31 | |
32 | The Rust `nightly` distribution channel includes the profiler runtime, by default. | |
33 | ||
353b0b11 | 34 | > **Important**: If you are building the Rust compiler from the source distribution, the profiler runtime is _not_ enabled in the default `config.example.toml`. Edit your `config.toml` file and ensure the `profiler` feature is set it to `true` (either under the `[build]` section, or under the settings for an individual `[target.<triple>]`): |
17df50a5 XL |
35 | > |
36 | > ```toml | |
37 | > # Build the profiler runtime (required when compiling with options that depend | |
5099ac24 | 38 | > # on this runtime, such as `-C profile-generate` or `-C instrument-coverage`). |
17df50a5 XL |
39 | > profiler = true |
40 | > ``` | |
41 | ||
42 | ### Building the demangler | |
43 | ||
44 | LLVM coverage reporting tools generate results that can include function names and other symbol references, and the raw coverage results report symbols using the compiler's "mangled" version of the symbol names, which can be difficult to interpret. To work around this issue, LLVM coverage tools also support a user-specified symbol name demangler. | |
45 | ||
46 | One option for a Rust demangler is [`rustfilt`], which can be installed with: | |
47 | ||
48 | ```shell | |
49 | cargo install rustfilt | |
50 | ``` | |
51 | ||
52 | Another option, if you are building from the Rust compiler source distribution, is to use the `rust-demangler` tool included in the Rust source distribution, which can be built with: | |
53 | ||
54 | ```shell | |
55 | $ ./x.py build rust-demangler | |
56 | ``` | |
57 | ||
58 | [`rustfilt`]: https://crates.io/crates/rustfilt | |
59 | ||
60 | ## Compiling with coverage enabled | |
61 | ||
5099ac24 | 62 | Set the `-C instrument-coverage` compiler flag in order to enable LLVM source-based code coverage profiling. |
17df50a5 | 63 | |
5099ac24 | 64 | The default option generates coverage for all functions, including unused (never called) functions and generics. The compiler flag supports an optional value to tailor this behavior. (See [`-C instrument-coverage=<options>`](#-c-instrument-coverageoptions), below.) |
17df50a5 XL |
65 | |
66 | With `cargo`, you can instrument your program binary _and_ dependencies at the same time. | |
67 | ||
68 | For example (if your project's Cargo.toml builds a binary by default): | |
69 | ||
70 | ```shell | |
71 | $ cd your-project | |
72 | $ cargo clean | |
5099ac24 | 73 | $ RUSTFLAGS="-C instrument-coverage" cargo build |
17df50a5 XL |
74 | ``` |
75 | ||
76 | If `cargo` is not configured to use your `profiler`-enabled version of `rustc`, set the path explicitly via the `RUSTC` environment variable. Here is another example, using a `stage1` build of `rustc` to compile an `example` binary (from the [`json5format`] crate): | |
77 | ||
78 | ```shell | |
79 | $ RUSTC=$HOME/rust/build/x86_64-unknown-linux-gnu/stage1/bin/rustc \ | |
5099ac24 | 80 | RUSTFLAGS="-C instrument-coverage" \ |
17df50a5 XL |
81 | cargo build --example formatjson5 |
82 | ``` | |
83 | ||
5099ac24 | 84 | > **Note**: that some compiler options, combined with `-C instrument-coverage`, can produce LLVM IR and/or linked binaries that are incompatible with LLVM coverage maps. For example, coverage requires references to actual functions in LLVM IR. If any covered function is optimized out, the coverage tools may not be able to process the coverage results. If you need to pass additional options, with coverage enabled, test them early, to confirm you will get the coverage results you expect. |
17df50a5 XL |
85 | |
86 | ## Running the instrumented binary to generate raw coverage profiling data | |
87 | ||
88 | In the previous example, `cargo` generated the coverage-instrumented binary `formatjson5`: | |
89 | ||
90 | ```shell | |
91 | $ echo "{some: 'thing'}" | target/debug/examples/formatjson5 - | |
92 | ``` | |
93 | ||
94 | ```json5 | |
95 | { | |
96 | some: "thing", | |
97 | } | |
98 | ``` | |
99 | ||
f2b60f7d FG |
100 | After running this program, a new file named like `default_11699812450447639123_0_20944` should be in the current working directory. |
101 | A new, unique file name will be generated each time the program is run to avoid overwriting previous data. | |
102 | ||
103 | ```shell | |
104 | $ echo "{some: 'thing'}" | target/debug/examples/formatjson5 - | |
105 | ... | |
106 | $ ls default_*.profraw | |
107 | default_11699812450447639123_0_20944.profraw | |
108 | ``` | |
109 | ||
110 | You can also set a specific file name or path for the generated `.profraw` files by using the environment variable `LLVM_PROFILE_FILE`: | |
17df50a5 XL |
111 | |
112 | ```shell | |
113 | $ echo "{some: 'thing'}" \ | |
114 | | LLVM_PROFILE_FILE="formatjson5.profraw" target/debug/examples/formatjson5 - | |
115 | ... | |
116 | $ ls formatjson5.profraw | |
117 | formatjson5.profraw | |
118 | ``` | |
119 | ||
49aad941 | 120 | If `LLVM_PROFILE_FILE` contains a path to a nonexistent directory, the missing directory structure will be created. Additionally, the following special pattern strings are rewritten: |
17df50a5 XL |
121 | |
122 | - `%p` - The process ID. | |
123 | - `%h` - The hostname of the machine running the program. | |
124 | - `%t` - The value of the TMPDIR environment variable. | |
125 | - `%Nm` - the instrumented binary’s signature: The runtime creates a pool of N raw profiles, used for on-line profile merging. The runtime takes care of selecting a raw profile from the pool, locking it, and updating it before the program exits. `N` must be between `1` and `9`, and defaults to `1` if omitted (with simply `%m`). | |
126 | - `%c` - Does not add anything to the filename, but enables a mode (on some platforms, including Darwin) in which profile counter updates are continuously synced to a file. This means that if the instrumented program crashes, or is killed by a signal, perfect coverage information can still be recovered. | |
127 | ||
f2b60f7d FG |
128 | In the first example above, the value `11699812450447639123_0` in the generated filename is the instrumented binary's signature, |
129 | which replaced the `%m` pattern and the value `20944` is the process ID of the binary being executed. | |
130 | ||
17df50a5 XL |
131 | ## Installing LLVM coverage tools |
132 | ||
5099ac24 | 133 | LLVM's supplies two tools—`llvm-profdata` and `llvm-cov`—that process coverage data and generate reports. There are several ways to find and/or install these tools, but note that the coverage mapping data generated by the Rust compiler requires LLVM version 12 or higher, and processing the *raw* data may require exactly the LLVM version used by the compiler. (`llvm-cov --version` typically shows the tool's LLVM version number, and `rustc --verbose --version` shows the version of LLVM used by the Rust compiler.) |
17df50a5 | 134 | |
5099ac24 FG |
135 | - You can install compatible versions of these tools via the `rustup` component `llvm-tools-preview`. This component is the recommended path, though the specific tools available and their interface is not currently subject to Rust's usual stability guarantees. In this case, you may also find `cargo-binutils` useful as a wrapper around these tools. |
136 | - You can install a compatible version of LLVM tools from your operating system distribution, or from your distribution of LLVM. | |
17df50a5 | 137 | - If you are building the Rust compiler from source, you can optionally use the bundled LLVM tools, built from source. Those tool binaries can typically be found in your build platform directory at something like: `rust/build/x86_64-unknown-linux-gnu/llvm/bin/llvm-*`. |
17df50a5 | 138 | |
5099ac24 | 139 | The examples in this document show how to use the llvm tools directly. |
17df50a5 XL |
140 | |
141 | ## Creating coverage reports | |
142 | ||
5099ac24 | 143 | Raw profiles have to be indexed before they can be used to generate coverage reports. This is done using [`llvm-profdata merge`], which can combine multiple raw profiles and index them at the same time: |
17df50a5 XL |
144 | |
145 | ```shell | |
146 | $ llvm-profdata merge -sparse formatjson5.profraw -o formatjson5.profdata | |
147 | ``` | |
148 | ||
5099ac24 | 149 | Finally, the `.profdata` file is used, in combination with the coverage map (from the program binary) to generate coverage reports using [`llvm-cov report`], for a coverage summaries; and [`llvm-cov show`], to see detailed coverage of lines and regions (character ranges) overlaid on the original source code. |
17df50a5 XL |
150 | |
151 | These commands have several display and filtering options. For example: | |
152 | ||
153 | ```shell | |
154 | $ llvm-cov show -Xdemangler=rustfilt target/debug/examples/formatjson5 \ | |
155 | -instr-profile=formatjson5.profdata \ | |
156 | -show-line-counts-or-regions \ | |
157 | -show-instantiations \ | |
158 | -name=add_quoted_string | |
159 | ``` | |
160 | ||
04454e1e | 161 | <img alt="Screenshot of sample `llvm-cov show` result, for function add_quoted_string" src="images/llvm-cov-show-01.png" class="center"/> |
17df50a5 XL |
162 | <br/> |
163 | <br/> | |
164 | ||
165 | Some of the more notable options in this example include: | |
166 | ||
167 | - `--Xdemangler=rustfilt` - the command name or path used to demangle Rust symbols (`rustfilt` in the example, but this could also be a path to the `rust-demangler` tool) | |
168 | - `target/debug/examples/formatjson5` - the instrumented binary (from which to extract the coverage map) | |
169 | - `--instr-profile=<path-to-file>.profdata` - the location of the `.profdata` file created by `llvm-profdata merge` (from the `.profraw` file generated by the instrumented binary) | |
170 | - `--name=<exact-function-name>` - to show coverage for a specific function (or, consider using another filter option, such as `--name-regex=<pattern>`) | |
171 | ||
172 | [`llvm-profdata merge`]: https://llvm.org/docs/CommandGuide/llvm-profdata.html#profdata-merge | |
173 | [`llvm-cov report`]: https://llvm.org/docs/CommandGuide/llvm-cov.html#llvm-cov-report | |
174 | [`llvm-cov show`]: https://llvm.org/docs/CommandGuide/llvm-cov.html#llvm-cov-show | |
175 | ||
781aab86 | 176 | > **Note**: Coverage can also be disabled on an individual function by annotating the function with the [`coverage(off)` attribute] (which requires the feature flag `#![feature(coverage)]`). |
17df50a5 | 177 | |
781aab86 | 178 | [`coverage` attribute]: ../unstable-book/language-features/coverage.html |
17df50a5 XL |
179 | |
180 | ## Interpreting reports | |
181 | ||
182 | There are four statistics tracked in a coverage summary: | |
183 | ||
184 | - Function coverage is the percentage of functions that have been executed at least once. A function is considered to be executed if any of its instantiations are executed. | |
185 | - Instantiation coverage is the percentage of function instantiations that have been executed at least once. Generic functions and functions generated from macros are two kinds of functions that may have multiple instantiations. | |
186 | - Line coverage is the percentage of code lines that have been executed at least once. Only executable lines within function bodies are considered to be code lines. | |
187 | - Region coverage is the percentage of code regions that have been executed at least once. A code region may span multiple lines: for example, in a large function body with no control flow. In other cases, a single line can contain multiple code regions: `return x || (y && z)` has countable code regions for `x` (which may resolve the expression, if `x` is `true`), `|| (y && z)` (executed only if `x` was `false`), and `return` (executed in either situation). | |
188 | ||
189 | Of these four statistics, function coverage is usually the least granular while region coverage is the most granular. The project-wide totals for each statistic are listed in the summary. | |
190 | ||
191 | ## Test coverage | |
192 | ||
193 | A typical use case for coverage analysis is test coverage. Rust's source-based coverage tools can both measure your tests' code coverage as percentage, and pinpoint functions and branches not tested. | |
194 | ||
195 | The following example (using the [`json5format`] crate, for demonstration purposes) show how to generate and analyze coverage results for all tests in a crate. | |
196 | ||
f2b60f7d | 197 | Since `cargo test` both builds and runs the tests, we set the additional `RUSTFLAGS`, to add the `-C instrument-coverage` flag. |
17df50a5 XL |
198 | |
199 | ```shell | |
5099ac24 | 200 | $ RUSTFLAGS="-C instrument-coverage" \ |
17df50a5 XL |
201 | cargo test --tests |
202 | ``` | |
203 | ||
9c376795 FG |
204 | > **Note**: The default for `LLVM_PROFILE_FILE` is `default_%m_%p.profraw`. Versions prior to 1.65 had a default of `default.profraw`, so if using those earlier versions, it is recommended to explicitly set `LLVM_PROFILE_FILE="default_%m_%p.profraw"` to avoid having multiple tests overwrite the `.profraw` files. |
205 | ||
17df50a5 XL |
206 | Make note of the test binary file paths, displayed after the word "`Running`" in the test output: |
207 | ||
208 | ```text | |
209 | ... | |
210 | Compiling json5format v0.1.3 ($HOME/json5format) | |
211 | Finished test [unoptimized + debuginfo] target(s) in 14.60s | |
212 | ||
213 | Running target/debug/deps/json5format-fececd4653271682 | |
214 | running 25 tests | |
215 | ... | |
216 | test result: ok. 25 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out | |
217 | ||
218 | Running target/debug/deps/lib-30768f9c53506dc5 | |
219 | running 31 tests | |
220 | ... | |
221 | test result: ok. 31 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out | |
222 | ``` | |
223 | ||
224 | You should have one or more `.profraw` files now, one for each test binary. Run the `profdata` tool to merge them: | |
225 | ||
226 | ```shell | |
f2b60f7d | 227 | $ llvm-profdata merge -sparse default_*.profraw -o json5format.profdata |
17df50a5 XL |
228 | ``` |
229 | ||
230 | Then run the `cov` tool, with the `profdata` file and all test binaries: | |
231 | ||
232 | ```shell | |
5099ac24 | 233 | $ llvm-cov report \ |
17df50a5 XL |
234 | --use-color --ignore-filename-regex='/.cargo/registry' \ |
235 | --instr-profile=json5format.profdata \ | |
236 | --object target/debug/deps/lib-30768f9c53506dc5 \ | |
237 | --object target/debug/deps/json5format-fececd4653271682 | |
5099ac24 | 238 | $ llvm-cov show \ |
17df50a5 XL |
239 | --use-color --ignore-filename-regex='/.cargo/registry' \ |
240 | --instr-profile=json5format.profdata \ | |
241 | --object target/debug/deps/lib-30768f9c53506dc5 \ | |
242 | --object target/debug/deps/json5format-fececd4653271682 \ | |
243 | --show-instantiations --show-line-counts-or-regions \ | |
244 | --Xdemangler=rustfilt | less -R | |
245 | ``` | |
246 | ||
f2b60f7d FG |
247 | > **Note**: If overriding the default `profraw` file name via the `LLVM_PROFILE_FILE` environment variable, it's highly recommended to use the `%m` and `%p` special pattern strings to generate unique file names in the case of more than a single test binary being executed. |
248 | ||
17df50a5 XL |
249 | > **Note**: The command line option `--ignore-filename-regex=/.cargo/registry`, which excludes the sources for dependencies from the coverage results.\_ |
250 | ||
251 | ### Tips for listing the binaries automatically | |
252 | ||
253 | For `bash` users, one suggested way to automatically complete the `cov` command with the list of binaries is with a command like: | |
254 | ||
255 | ```bash | |
5099ac24 | 256 | $ llvm-cov report \ |
17df50a5 XL |
257 | $( \ |
258 | for file in \ | |
259 | $( \ | |
5099ac24 | 260 | RUSTFLAGS="-C instrument-coverage" \ |
17df50a5 XL |
261 | cargo test --tests --no-run --message-format=json \ |
262 | | jq -r "select(.profile.test == true) | .filenames[]" \ | |
263 | | grep -v dSYM - \ | |
264 | ); \ | |
265 | do \ | |
266 | printf "%s %s " -object $file; \ | |
267 | done \ | |
268 | ) \ | |
269 | --instr-profile=json5format.profdata --summary-only # and/or other options | |
270 | ``` | |
271 | ||
272 | Adding `--no-run --message-format=json` to the _same_ `cargo test` command used to run | |
273 | the tests (including the same environment variables and flags) generates output in a JSON | |
274 | format that `jq` can easily query. | |
275 | ||
276 | The `printf` command takes this list and generates the `--object <binary>` arguments | |
277 | for each listed test binary. | |
278 | ||
279 | ### Including doc tests | |
280 | ||
281 | The previous examples run `cargo test` with `--tests`, which excludes doc tests.[^79417] | |
282 | ||
283 | To include doc tests in the coverage results, drop the `--tests` flag, and apply the | |
5099ac24 FG |
284 | `-C instrument-coverage` flag, and some doc-test-specific options in the |
285 | `RUSTDOCFLAGS` environment variable. (The `llvm-profdata` command does not change.) | |
17df50a5 XL |
286 | |
287 | ```bash | |
5099ac24 FG |
288 | $ RUSTFLAGS="-C instrument-coverage" \ |
289 | RUSTDOCFLAGS="-C instrument-coverage -Z unstable-options --persist-doctests target/debug/doctestbins" \ | |
17df50a5 | 290 | cargo test |
f2b60f7d | 291 | $ llvm-profdata merge -sparse default_*.profraw -o json5format.profdata |
17df50a5 XL |
292 | ``` |
293 | ||
294 | The `-Z unstable-options --persist-doctests` flag is required, to save the test binaries | |
295 | (with their coverage maps) for `llvm-cov`. | |
296 | ||
297 | ```bash | |
5099ac24 | 298 | $ llvm-cov report \ |
17df50a5 XL |
299 | $( \ |
300 | for file in \ | |
301 | $( \ | |
5099ac24 FG |
302 | RUSTFLAGS="-C instrument-coverage" \ |
303 | RUSTDOCFLAGS="-C instrument-coverage -Z unstable-options --persist-doctests target/debug/doctestbins" \ | |
17df50a5 XL |
304 | cargo test --no-run --message-format=json \ |
305 | | jq -r "select(.profile.test == true) | .filenames[]" \ | |
306 | | grep -v dSYM - \ | |
307 | ) \ | |
308 | target/debug/doctestbins/*/rust_out; \ | |
309 | do \ | |
310 | [[ -x $file ]] && printf "%s %s " -object $file; \ | |
311 | done \ | |
312 | ) \ | |
313 | --instr-profile=json5format.profdata --summary-only # and/or other options | |
314 | ``` | |
315 | ||
5099ac24 FG |
316 | > **Note**: The differences in this `llvm-cov` invocation, compared with the |
317 | > version without doc tests, include: | |
17df50a5 XL |
318 | |
319 | - The `cargo test ... --no-run` command is updated with the same environment variables | |
f2b60f7d | 320 | and flags used to _build_ the tests, _including_ the doc tests. |
17df50a5 XL |
321 | - The file glob pattern `target/debug/doctestbins/*/rust_out` adds the `rust_out` |
322 | binaries generated for doc tests (note, however, that some `rust_out` files may not | |
323 | be executable binaries). | |
324 | - `[[ -x $file ]] &&` filters the files passed on to the `printf`, to include only | |
325 | executable binaries. | |
326 | ||
327 | [^79417]: | |
328 | There is ongoing work to resolve a known issue | |
329 | [(#79417)](https://github.com/rust-lang/rust/issues/79417) that doc test coverage | |
330 | generates incorrect source line numbers in `llvm-cov show` results. | |
331 | ||
5099ac24 | 332 | ## `-C instrument-coverage=<options>` |
17df50a5 | 333 | |
5099ac24 FG |
334 | - `-C instrument-coverage=all`: Instrument all functions, including unused functions and unused generics. (This is the same as `-C instrument-coverage`, with no value.) |
335 | - `-C instrument-coverage=off`: Do not instrument any functions. (This is the same as simply not including the `-C instrument-coverage` option.) | |
336 | - `-Zunstable-options -C instrument-coverage=except-unused-generics`: Instrument all functions except unused generics. | |
337 | - `-Zunstable-options -C instrument-coverage=except-unused-functions`: Instrument only used (called) functions and instantiated generic functions. | |
17df50a5 XL |
338 | |
339 | ## Other references | |
340 | ||
341 | Rust's implementation and workflow for source-based code coverage is based on the same library and tools used to implement [source-based code coverage in Clang]. (This document is partially based on the Clang guide.) | |
342 | ||
343 | [source-based code coverage in clang]: https://clang.llvm.org/docs/SourceBasedCodeCoverage.html | |
344 | [`json5format`]: https://crates.io/crates/json5format |