3 //! This module implements change-tracking so that Cargo can know whether or
4 //! not something needs to be recompiled. A Cargo `Unit` can be either "dirty"
5 //! (needs to be recompiled) or "fresh" (it does not need to be recompiled).
6 //! There are several mechanisms that influence a Unit's freshness:
8 //! - The `Fingerprint` is a hash, saved to the filesystem in the
9 //! `.fingerprint` directory, that tracks information about the Unit. If the
10 //! fingerprint is missing (such as the first time the unit is being
11 //! compiled), then the unit is dirty. If any of the fingerprint fields
12 //! change (like the name of the source file), then the Unit is considered
15 //! The `Fingerprint` also tracks the fingerprints of all its dependencies,
16 //! so a change in a dependency will propagate the "dirty" status up.
18 //! - Filesystem mtime tracking is also used to check if a unit is dirty.
19 //! See the section below on "Mtime comparison" for more details. There
20 //! are essentially two parts to mtime tracking:
22 //! 1. The mtime of a Unit's output files is compared to the mtime of all
23 //! its dependencies' output file mtimes (see `check_filesystem`). If any
24 //! output is missing, or is older than a dependency's output, then the
26 //! 2. The mtime of a Unit's source files is compared to the mtime of its
27 //! dep-info file in the fingerprint directory (see `find_stale_file`).
28 //! The dep-info file is used as an anchor to know when the last build of
29 //! the unit was done. See the "dep-info files" section below for more
30 //! details. If any input files are missing, or are newer than the
31 //! dep-info, then the unit is dirty.
33 //! Note: Fingerprinting is not a perfect solution. Filesystem mtime tracking
34 //! is notoriously imprecise and problematic. Only a small part of the
35 //! environment is captured. This is a balance of performance, simplicity, and
36 //! completeness. Sandboxing, hashing file contents, tracking every file
37 //! access, environment variable, and network operation would ensure more
38 //! reliable and reproducible builds at the cost of being complex, slow, and
39 //! platform-dependent.
41 //! ## Fingerprints and Metadata
43 //! The `Metadata` hash is a hash added to the output filenames to isolate
44 //! each unit. See the documentation in the `compilation_files` module for
45 //! more details. NOTE: Not all output files are isolated via filename hashes
46 //! (like dylibs). The fingerprint directory uses a hash, but sometimes units
47 //! share the same fingerprint directory (when they don't have Metadata) so
48 //! care should be taken to handle this!
50 //! Fingerprints and Metadata are similar, and track some of the same things.
51 //! The Metadata contains information that is required to keep Units separate.
52 //! The Fingerprint includes additional information that should cause a
53 //! recompile, but it is desired to reuse the same filenames. A comparison
54 //! of what is tracked:
56 //! Value | Fingerprint | Metadata
57 //! -------------------------------------------|-------------|----------
60 //! `cargo rustc` extra args | ✓ | ✓
61 //! CompileMode | ✓ | ✓
62 //! Target Name | ✓ | ✓
63 //! Target CompileKind (bin/lib/etc.) | ✓ | ✓
64 //! Enabled Features | ✓ | ✓
65 //! Immediate dependency’s hashes | ✓[^1] | ✓
66 //! Target or Host mode | | ✓
67 //! __CARGO_DEFAULT_LIB_METADATA[^4] | | ✓
69 //! authors, description, homepage, repo | ✓ |
70 //! Target src path relative to ws | ✓ |
71 //! Target flags (test/bench/for_host/edition) | ✓ |
72 //! -C incremental=… flag | ✓ |
73 //! mtime of sources | ✓[^3] |
74 //! RUSTFLAGS/RUSTDOCFLAGS | ✓ |
76 //! config settings[^5] | ✓ |
79 //! [^1]: Build script and bin dependencies are not included.
81 //! [^3]: See below for details on mtime tracking.
83 //! [^4]: `__CARGO_DEFAULT_LIB_METADATA` is set by rustbuild to embed the
84 //! release channel (bootstrap/stable/beta/nightly) in libstd.
86 //! [^5]: Config settings that are not otherwise captured anywhere else.
87 //! Currently, this is only `doc.extern-map`.
89 //! When deciding what should go in the Metadata vs the Fingerprint, consider
90 //! that some files (like dylibs) do not have a hash in their filename. Thus,
91 //! if a value changes, only the fingerprint will detect the change (consider,
92 //! for example, swapping between different features). Fields that are only in
93 //! Metadata generally aren't relevant to the fingerprint because they
94 //! fundamentally change the output (like target vs host changes the directory
95 //! where it is emitted).
97 //! ## Fingerprint files
99 //! Fingerprint information is stored in the
100 //! `target/{debug,release}/.fingerprint/` directory. Each Unit is stored in a
101 //! separate directory. Each Unit directory contains:
103 //! - A file with a 16 hex-digit hash. This is the Fingerprint hash, used for
104 //! quick loading and comparison.
105 //! - A `.json` file that contains details about the Fingerprint. This is only
106 //! used to log details about *why* a fingerprint is considered dirty.
107 //! `CARGO_LOG=cargo::core::compiler::fingerprint=trace cargo build` can be
108 //! used to display this log information.
109 //! - A "dep-info" file which contains a list of source filenames for the
110 //! target. See below for details.
111 //! - An `invoked.timestamp` file whose filesystem mtime is updated every time
112 //! the Unit is built. This is used for capturing the time when the build
113 //! starts, to detect if files are changed in the middle of the build. See
114 //! below for more details.
116 //! Note that some units are a little different. A Unit for *running* a build
117 //! script or for `rustdoc` does not have a dep-info file (it's not
118 //! applicable). Build script `invoked.timestamp` files are in the build
119 //! output directory.
121 //! ## Fingerprint calculation
123 //! After the list of Units has been calculated, the Units are added to the
124 //! `JobQueue`. As each one is added, the fingerprint is calculated, and the
125 //! dirty/fresh status is recorded. A closure is used to update the fingerprint
126 //! on-disk when the Unit successfully finishes. The closure will recompute the
127 //! Fingerprint based on the updated information. If the Unit fails to compile,
128 //! the fingerprint is not updated.
130 //! Fingerprints are cached in the `Context`. This makes computing
131 //! Fingerprints faster, but also is necessary for properly updating
132 //! dependency information. Since a Fingerprint includes the Fingerprints of
133 //! all dependencies, when it is updated, by using `Arc` clones, it
134 //! automatically picks up the updates to its dependencies.
136 //! ### dep-info files
138 //! Cargo passes the `--emit=dep-info` flag to `rustc` so that `rustc` will
139 //! generate a "dep info" file (with the `.d` extension). This is a
140 //! Makefile-like syntax that includes all of the source files used to build
141 //! the crate. This file is used by Cargo to know which files to check to see
142 //! if the crate will need to be rebuilt.
144 //! After `rustc` exits successfully, Cargo will read the dep info file and
145 //! translate it into a binary format that is stored in the fingerprint
146 //! directory (`translate_dep_info`). The mtime of the fingerprint dep-info
147 //! file itself is used as the reference for comparing the source files to
148 //! determine if any of the source files have been modified (see below for
151 //! There is also a third dep-info file. Cargo will extend the file created by
152 //! rustc with some additional information and saves this into the output
153 //! directory. This is intended for build system integration. See the
154 //! `output_depinfo` module for more detail.
156 //! #### -Zbinary-dep-depinfo
158 //! `rustc` has an experimental flag `-Zbinary-dep-depinfo`. This causes
159 //! `rustc` to include binary files (like rlibs) in the dep-info file. This is
160 //! primarily to support rustc development, so that Cargo can check the
161 //! implicit dependency to the standard library (which lives in the sysroot).
162 //! We want Cargo to recompile whenever the standard library rlib/dylibs
163 //! change, and this is a generic mechanism to make that work.
165 //! ### Mtime comparison
167 //! The use of modification timestamps is the most common way a unit will be
168 //! determined to be dirty or fresh between builds. There are many subtle
169 //! issues and edge cases with mtime comparisons. This gives a high-level
170 //! overview, but you'll need to read the code for the gritty details. Mtime
171 //! handling is different for different unit kinds. The different styles are
172 //! driven by the `Fingerprint.local` field, which is set based on the unit
175 //! The status of whether or not the mtime is "stale" or "up-to-date" is
176 //! stored in `Fingerprint.fs_status`.
178 //! All units will compare the mtime of its newest output file with the mtimes
179 //! of the outputs of all its dependencies. If any output file is missing,
180 //! then the unit is stale. If any dependency is newer, the unit is stale.
182 //! #### Normal package mtime handling
184 //! `LocalFingerprint::CheckDepinfo` is used for checking the mtime of
185 //! packages. It compares the mtime of the input files (the source files) to
186 //! the mtime of the dep-info file (which is written last after a build is
187 //! finished). If the dep-info is missing, the unit is stale (it has never
188 //! been built). The list of input files comes from the dep-info file. See the
189 //! section above for details on dep-info files.
191 //! Also note that although registry and git packages use `CheckDepInfo`, none
192 //! of their source files are included in the dep-info (see
193 //! `translate_dep_info`), so for those kinds no mtime checking is done
194 //! (unless `-Zbinary-dep-depinfo` is used). Repository and git packages are
195 //! static, so there is no need to check anything.
197 //! When a build is complete, the mtime of the dep-info file in the
198 //! fingerprint directory is modified to rewind it to the time when the build
199 //! started. This is done by creating an `invoked.timestamp` file when the
200 //! build starts to capture the start time. The mtime is rewound to the start
201 //! to handle the case where the user modifies a source file while a build is
202 //! running. Cargo can't know whether or not the file was included in the
203 //! build, so it takes a conservative approach of assuming the file was *not*
204 //! included, and it should be rebuilt during the next build.
206 //! #### Rustdoc mtime handling
208 //! Rustdoc does not emit a dep-info file, so Cargo currently has a relatively
209 //! simple system for detecting rebuilds. `LocalFingerprint::Precalculated` is
210 //! used for rustdoc units. For registry packages, this is the package
211 //! version. For git packages, it is the git hash. For path packages, it is
212 //! the a string of the mtime of the newest file in the package.
214 //! There are some known bugs with how this works, so it should be improved at
217 //! #### Build script mtime handling
219 //! Build script mtime handling runs in different modes. There is the "old
220 //! style" where the build script does not emit any `rerun-if` directives. In
221 //! this mode, Cargo will use `LocalFingerprint::Precalculated`. See the
222 //! "rustdoc" section above how it works.
224 //! In the new-style, each `rerun-if` directive is translated to the
225 //! corresponding `LocalFingerprint` variant. The `RerunIfChanged` variant
226 //! compares the mtime of the given filenames against the mtime of the
229 //! Similar to normal units, the build script "output" file mtime is rewound
230 //! to the time just before the build script is executed to handle mid-build
233 //! ## Considerations for inclusion in a fingerprint
235 //! Over time we've realized a few items which historically were included in
236 //! fingerprint hashings should not actually be included. Examples are:
238 //! * Modification time values. We strive to never include a modification time
239 //! inside a `Fingerprint` to get hashed into an actual value. While
240 //! theoretically fine to do, in practice this causes issues with common
241 //! applications like Docker. Docker, after a layer is built, will zero out
242 //! the nanosecond part of all filesystem modification times. This means that
243 //! the actual modification time is different for all build artifacts, which
244 //! if we tracked the actual values of modification times would cause
245 //! unnecessary recompiles. To fix this we instead only track paths which are
246 //! relevant. These paths are checked dynamically to see if they're up to
247 //! date, and the modification time doesn't make its way into the fingerprint
250 //! * Absolute path names. We strive to maintain a property where if you rename
251 //! a project directory Cargo will continue to preserve all build artifacts
252 //! and reuse the cache. This means that we can't ever hash an absolute path
253 //! name. Instead we always hash relative path names and the "root" is passed
254 //! in at runtime dynamically. Some of this is best effort, but the general
255 //! idea is that we assume all accesses within a crate stay within that
258 //! These are pretty tricky to test for unfortunately, but we should have a good
259 //! test suite nowadays and lord knows Cargo gets enough testing in the wild!
263 //! The *running* of a build script (`CompileMode::RunCustomBuild`) is treated
264 //! significantly different than all other Unit kinds. It has its own function
265 //! for calculating the Fingerprint (`calculate_run_custom_build`) and has some
266 //! unique considerations. It does not track the same information as a normal
267 //! Unit. The information tracked depends on the `rerun-if-changed` and
268 //! `rerun-if-env-changed` statements produced by the build script. If the
269 //! script does not emit either of these statements, the Fingerprint runs in
270 //! "old style" mode where an mtime change of *any* file in the package will
271 //! cause the build script to be re-run. Otherwise, the fingerprint *only*
272 //! tracks the individual "rerun-if" items listed by the build script.
274 //! The "rerun-if" statements from a *previous* build are stored in the build
275 //! output directory in a file called `output`. Cargo parses this file when
276 //! the Unit for that build script is prepared for the `JobQueue`. The
277 //! Fingerprint code can then use that information to compute the Fingerprint
278 //! and compare against the old fingerprint hash.
280 //! Care must be taken with build script Fingerprints because the
281 //! `Fingerprint::local` value may be changed after the build script runs
282 //! (such as if the build script adds or removes "rerun-if" items).
284 //! Another complication is if a build script is overridden. In that case, the
285 //! fingerprint is the hash of the output of the override.
287 //! ## Special considerations
289 //! Registry dependencies do not track the mtime of files. This is because
290 //! registry dependencies are not expected to change (if a new version is
291 //! used, the Package ID will change, causing a rebuild). Cargo currently
292 //! partially works with Docker caching. When a Docker image is built, it has
293 //! normal mtime information. However, when a step is cached, the nanosecond
294 //! portions of all files is zeroed out. Currently this works, but care must
295 //! be taken for situations like these.
297 //! HFS on macOS only supports 1 second timestamps. This causes a significant
298 //! number of problems, particularly with Cargo's testsuite which does rapid
299 //! builds in succession. Other filesystems have various degrees of
302 //! Various weird filesystems (such as network filesystems) also can cause
303 //! complications. Network filesystems may track the time on the server
304 //! (except when the time is set manually such as with
305 //! `filetime::set_file_times`). Not all filesystems support modifying the
308 //! See the `A-rebuild-detection` flag on the issue tracker for more:
309 //! <https://github.com/rust-lang/cargo/issues?q=is%3Aissue+is%3Aopen+label%3AA-rebuild-detection>
311 use std
::collections
::hash_map
::{Entry, HashMap}
;
313 use std
::hash
::{self, Hasher}
;
314 use std
::path
::{Path, PathBuf}
;
315 use std
::sync
::{Arc, Mutex}
;
316 use std
::time
::SystemTime
;
318 use anyhow
::{bail, format_err}
;
319 use filetime
::FileTime
;
320 use log
::{debug, info}
;
323 use serde
::{Deserialize, Serialize}
;
325 use crate::core
::compiler
::unit_graph
::UnitDep
;
326 use crate::core
::{InternedString, Package}
;
328 use crate::util
::errors
::{CargoResult, CargoResultExt}
;
329 use crate::util
::paths
;
330 use crate::util
::{internal, profile}
;
332 use super::custom_build
::BuildDeps
;
334 Freshness
::{Dirty, Fresh}
,
337 use super::{BuildContext, Context, FileFlavor, Unit}
;
339 /// Determines if a `unit` is up-to-date, and if not prepares necessary work to
340 /// update the persisted fingerprint.
342 /// This function will inspect `unit`, calculate a fingerprint for it, and then
343 /// return an appropriate `Job` to run. The returned `Job` will be a noop if
344 /// `unit` is considered "fresh", or if it was previously built and cached.
345 /// Otherwise the `Job` returned will write out the true fingerprint to the
346 /// filesystem, to be executed after the unit's work has completed.
348 /// The `force` flag is a way to force the `Job` to be "dirty", or always
349 /// update the fingerprint. **Beware using this flag** because it does not
350 /// transitively propagate throughout the dependency graph, it only forces this
351 /// one unit which is very unlikely to be what you want unless you're
352 /// exclusively talking about top-level units.
353 pub fn prepare_target(cx
: &mut Context
<'_
, '_
>, unit
: &Unit
, force
: bool
) -> CargoResult
<Job
> {
354 let _p
= profile
::start(format
!(
355 "fingerprint: {} / {}",
356 unit
.pkg
.package_id(),
360 let loc
= cx
.files().fingerprint_file_path(unit
, "");
362 debug
!("fingerprint at: {}", loc
.display());
364 // Figure out if this unit is up to date. After calculating the fingerprint
365 // compare it to an old version, if any, and attempt to print diagnostic
366 // information about failed comparisons to aid in debugging.
367 let fingerprint
= calculate(cx
, unit
)?
;
368 let mtime_on_use
= cx
.bcx
.config
.cli_unstable().mtime_on_use
;
369 let compare
= compare_old_fingerprint(&loc
, &*fingerprint
, mtime_on_use
);
370 log_compare(unit
, &compare
);
372 // If our comparison failed (e.g., we're going to trigger a rebuild of this
373 // crate), then we also ensure the source of the crate passes all
374 // verification checks before we build it.
376 // The `Source::verify` method is intended to allow sources to execute
377 // pre-build checks to ensure that the relevant source code is all
378 // up-to-date and as expected. This is currently used primarily for
379 // directory sources which will use this hook to perform an integrity check
380 // on all files in the source to ensure they haven't changed. If they have
381 // changed then an error is issued.
382 if compare
.is_err() {
383 let source_id
= unit
.pkg
.package_id().source_id();
384 let sources
= bcx
.packages
.sources();
387 .ok_or_else(|| internal("missing package source"))?
;
388 source
.verify(unit
.pkg
.package_id())?
;
391 if compare
.is_ok() && !force
{
392 return Ok(Job
::new(Work
::noop(), Fresh
));
395 // Clear out the old fingerprint file if it exists. This protects when
396 // compilation is interrupted leaving a corrupt file. For example, a
397 // project with a lib.rs and integration test (two units):
399 // 1. Build the library and integration test.
400 // 2. Make a change to lib.rs (NOT the integration test).
401 // 3. Build the integration test, hit Ctrl-C while linking. With gcc, this
402 // will leave behind an incomplete executable (zero size, or partially
403 // written). NOTE: The library builds successfully, it is the linking
404 // of the integration test that we are interrupting.
405 // 4. Build the integration test again.
407 // Without the following line, then step 3 will leave a valid fingerprint
408 // on the disk. Then step 4 will think the integration test is "fresh"
411 // - There is a valid fingerprint hash on disk (written in step 1).
412 // - The mtime of the output file (the corrupt integration executable
413 // written in step 3) is newer than all of its dependencies.
414 // - The mtime of the integration test fingerprint dep-info file (written
415 // in step 1) is newer than the integration test's source files, because
416 // we haven't modified any of its source files.
418 // But the executable is corrupt and needs to be rebuilt. Clearing the
419 // fingerprint at step 3 ensures that Cargo never mistakes a partially
420 // written output as up-to-date.
422 // Truncate instead of delete so that compare_old_fingerprint will
423 // still log the reason for the fingerprint failure instead of just
424 // reporting "failed to read fingerprint" during the next build if
426 paths
::write(&loc
, b
"")?
;
429 let write_fingerprint
= if unit
.mode
.is_run_custom_build() {
430 // For build scripts the `local` field of the fingerprint may change
431 // while we're executing it. For example it could be in the legacy
432 // "consider everything a dependency mode" and then we switch to "deps
433 // are explicitly specified" mode.
435 // To handle this movement we need to regenerate the `local` field of a
436 // build script's fingerprint after it's executed. We do this by
437 // using the `build_script_local_fingerprints` function which returns a
438 // thunk we can invoke on a foreign thread to calculate this.
439 let build_script_outputs
= Arc
::clone(&cx
.build_script_outputs
);
440 let pkg_id
= unit
.pkg
.package_id();
441 let metadata
= cx
.get_run_build_script_metadata(unit
);
442 let (gen_local
, _overridden
) = build_script_local_fingerprints(cx
, unit
);
443 let output_path
= cx
.build_explicit_deps
[unit
].build_script_output
.clone();
445 let outputs
= build_script_outputs
.lock().unwrap();
447 .get(pkg_id
, metadata
)
448 .expect("output must exist after running");
449 let deps
= BuildDeps
::new(&output_path
, Some(output
));
451 // FIXME: it's basically buggy that we pass `None` to `call_box`
452 // here. See documentation on `build_script_local_fingerprints`
453 // below for more information. Despite this just try to proceed and
454 // hobble along if it happens to return `Some`.
455 if let Some(new_local
) = (gen_local
)(&deps
, None
)?
{
456 *fingerprint
.local
.lock().unwrap() = new_local
;
459 write_fingerprint(&loc
, &fingerprint
)
462 Work
::new(move |_
| write_fingerprint(&loc
, &fingerprint
))
465 Ok(Job
::new(write_fingerprint
, Dirty
))
468 /// Dependency edge information for fingerprints. This is generated for each
469 /// dependency and is stored in a `Fingerprint` below.
471 struct DepFingerprint
{
472 /// The hash of the package id that this dependency points to
474 /// The crate name we're using for this dependency, which if we change we'll
475 /// need to recompile!
476 name
: InternedString
,
477 /// Whether or not this dependency is flagged as a public dependency or not.
479 /// Whether or not this dependency is an rmeta dependency or a "full"
480 /// dependency. In the case of an rmeta dependency our dependency edge only
481 /// actually requires the rmeta from what we depend on, so when checking
482 /// mtime information all files other than the rmeta can be ignored.
483 only_requires_rmeta
: bool
,
484 /// The dependency's fingerprint we recursively point to, containing all the
485 /// other hash information we'd otherwise need.
486 fingerprint
: Arc
<Fingerprint
>,
489 /// A fingerprint can be considered to be a "short string" representing the
490 /// state of a world for a package.
492 /// If a fingerprint ever changes, then the package itself needs to be
493 /// recompiled. Inputs to the fingerprint include source code modifications,
494 /// compiler flags, compiler version, etc. This structure is not simply a
495 /// `String` due to the fact that some fingerprints cannot be calculated lazily.
497 /// Path sources, for example, use the mtime of the corresponding dep-info file
498 /// as a fingerprint (all source files must be modified *before* this mtime).
499 /// This dep-info file is not generated, however, until after the crate is
500 /// compiled. As a result, this structure can be thought of as a fingerprint
501 /// to-be. The actual value can be calculated via `hash()`, but the operation
502 /// may fail as some files may not have been generated.
504 /// Note that dependencies are taken into account for fingerprints because rustc
505 /// requires that whenever an upstream crate is recompiled that all downstream
506 /// dependents are also recompiled. This is typically tracked through
507 /// `DependencyQueue`, but it also needs to be retained here because Cargo can
508 /// be interrupted while executing, losing the state of the `DependencyQueue`
510 #[derive(Serialize, Deserialize)]
511 pub struct Fingerprint
{
512 /// Hash of the version of `rustc` used.
514 /// Sorted list of cfg features enabled.
516 /// Hash of the `Target` struct, including the target name,
517 /// package-relative source path, edition, etc.
519 /// Hash of the `Profile`, `CompileMode`, and any extra flags passed via
520 /// `cargo rustc` or `cargo rustdoc`.
522 /// Hash of the path to the base source file. This is relative to the
523 /// workspace root for path members, or absolute for other sources.
525 /// Fingerprints of dependencies.
526 deps
: Vec
<DepFingerprint
>,
527 /// Information about the inputs that affect this Unit (such as source
528 /// file mtimes or build script environment variables).
529 local
: Mutex
<Vec
<LocalFingerprint
>>,
530 /// Cached hash of the `Fingerprint` struct. Used to improve performance
533 memoized_hash
: Mutex
<Option
<u64>>,
534 /// RUSTFLAGS/RUSTDOCFLAGS environment variable value (or config value).
535 rustflags
: Vec
<String
>,
536 /// Hash of some metadata from the manifest, such as "authors", or
537 /// "description", which are exposed as environment variables during
540 /// Hash of various config settings that change how things are compiled.
542 /// Description of whether the filesystem status for this unit is up to date
543 /// or should be considered stale.
546 /// Files, relative to `target_root`, that are produced by the step that
547 /// this `Fingerprint` represents. This is used to detect when the whole
548 /// fingerprint is out of date if this is missing, or if previous
549 /// fingerprints output files are regenerated and look newer than this one.
551 outputs
: Vec
<PathBuf
>,
554 /// Indication of the status on the filesystem for a particular unit.
556 /// This unit is to be considered stale, even if hash information all
557 /// matches. The filesystem inputs have changed (or are missing) and the
558 /// unit needs to subsequently be recompiled.
561 /// This unit is up-to-date. All outputs and their corresponding mtime are
562 /// listed in the payload here for other dependencies to compare against.
563 UpToDate { mtimes: HashMap<PathBuf, FileTime> }
,
567 fn up_to_date(&self) -> bool
{
569 FsStatus
::UpToDate { .. }
=> true,
570 FsStatus
::Stale
=> false,
575 impl Default
for FsStatus
{
576 fn default() -> FsStatus
{
581 impl Serialize
for DepFingerprint
{
582 fn serialize
<S
>(&self, ser
: S
) -> Result
<S
::Ok
, S
::Error
>
590 &self.fingerprint
.hash(),
596 impl<'de
> Deserialize
<'de
> for DepFingerprint
{
597 fn deserialize
<D
>(d
: D
) -> Result
<DepFingerprint
, D
::Error
>
599 D
: de
::Deserializer
<'de
>,
601 let (pkg_id
, name
, public
, hash
) = <(u64, String
, bool
, u64)>::deserialize(d
)?
;
604 name
: InternedString
::new(&name
),
606 fingerprint
: Arc
::new(Fingerprint
{
607 memoized_hash
: Mutex
::new(Some(hash
)),
610 // This field is never read since it's only used in
611 // `check_filesystem` which isn't used by fingerprints loaded from
613 only_requires_rmeta
: false,
618 /// A `LocalFingerprint` represents something that we use to detect direct
619 /// changes to a `Fingerprint`.
621 /// This is where we track file information, env vars, etc. This
622 /// `LocalFingerprint` struct is hashed and if the hash changes will force a
623 /// recompile of any fingerprint it's included into. Note that the "local"
624 /// terminology comes from the fact that it only has to do with one crate, and
625 /// `Fingerprint` tracks the transitive propagation of fingerprint changes.
627 /// Note that because this is hashed its contents are carefully managed. Like
628 /// mentioned in the above module docs, we don't want to hash absolute paths or
629 /// mtime information.
631 /// Also note that a `LocalFingerprint` is used in `check_filesystem` to detect
632 /// when the filesystem contains stale information (based on mtime currently).
633 /// The paths here don't change much between compilations but they're used as
634 /// inputs when we probe the filesystem looking at information.
635 #[derive(Debug, Serialize, Deserialize, Hash)]
636 enum LocalFingerprint
{
637 /// This is a precalculated fingerprint which has an opaque string we just
638 /// hash as usual. This variant is primarily used for rustdoc where we
639 /// don't have a dep-info file to compare against.
641 /// This is also used for build scripts with no `rerun-if-*` statements, but
642 /// that's overall a mistake and causes bugs in Cargo. We shouldn't use this
643 /// for build scripts.
644 Precalculated(String
),
646 /// This is used for crate compilations. The `dep_info` file is a relative
647 /// path anchored at `target_root(...)` to the dep-info file that Cargo
648 /// generates (which is a custom serialization after parsing rustc's own
649 /// `dep-info` output).
651 /// The `dep_info` file, when present, also lists a number of other files
652 /// for us to look at. If any of those files are newer than this file then
653 /// we need to recompile.
654 CheckDepInfo { dep_info: PathBuf }
,
656 /// This represents a nonempty set of `rerun-if-changed` annotations printed
657 /// out by a build script. The `output` file is a relative file anchored at
658 /// `target_root(...)` which is the actual output of the build script. That
659 /// output has already been parsed and the paths printed out via
660 /// `rerun-if-changed` are listed in `paths`. The `paths` field is relative
663 /// This is considered up-to-date if all of the `paths` are older than
664 /// `output`, otherwise we need to recompile.
670 /// This represents a single `rerun-if-env-changed` annotation printed by a
671 /// build script. The exact env var and value are hashed here. There's no
672 /// filesystem dependence here, and if the values are changed the hash will
673 /// change forcing a recompile.
674 RerunIfEnvChanged { var: String, val: Option<String> }
,
681 reference_mtime
: FileTime
,
683 stale_mtime
: FileTime
,
687 impl LocalFingerprint
{
688 /// Checks dynamically at runtime if this `LocalFingerprint` has a stale
691 /// This will use the absolute root paths passed in if necessary to guide
695 mtime_cache
: &mut HashMap
<PathBuf
, FileTime
>,
698 ) -> CargoResult
<Option
<StaleFile
>> {
700 // We need to parse `dep_info`, learn about all the files the crate
701 // depends on, and then see if any of them are newer than the
702 // dep_info file itself. If the `dep_info` file is missing then this
703 // unit has never been compiled!
704 LocalFingerprint
::CheckDepInfo { dep_info }
=> {
705 let dep_info
= target_root
.join(dep_info
);
706 if let Some(paths
) = parse_dep_info(pkg_root
, target_root
, &dep_info
)?
{
707 Ok(find_stale_file(mtime_cache
, &dep_info
, paths
.iter()))
709 Ok(Some(StaleFile
::Missing(dep_info
)))
713 // We need to verify that no paths listed in `paths` are newer than
714 // the `output` path itself, or the last time the build script ran.
715 LocalFingerprint
::RerunIfChanged { output, paths }
=> Ok(find_stale_file(
717 &target_root
.join(output
),
718 paths
.iter().map(|p
| pkg_root
.join(p
)),
721 // These have no dependencies on the filesystem, and their values
722 // are included natively in the `Fingerprint` hash so nothing
724 LocalFingerprint
::RerunIfEnvChanged { .. }
=> Ok(None
),
725 LocalFingerprint
::Precalculated(..) => Ok(None
),
729 fn kind(&self) -> &'
static str {
731 LocalFingerprint
::Precalculated(..) => "precalculated",
732 LocalFingerprint
::CheckDepInfo { .. }
=> "dep-info",
733 LocalFingerprint
::RerunIfChanged { .. }
=> "rerun-if-changed",
734 LocalFingerprint
::RerunIfEnvChanged { .. }
=> "rerun-if-env-changed",
740 struct MtimeSlot(Mutex
<Option
<FileTime
>>);
743 fn new() -> Fingerprint
{
749 features
: String
::new(),
751 local
: Mutex
::new(Vec
::new()),
752 memoized_hash
: Mutex
::new(None
),
753 rustflags
: Vec
::new(),
756 fs_status
: FsStatus
::Stale
,
761 /// For performance reasons fingerprints will memoize their own hash, but
762 /// there's also internal mutability with its `local` field which can
763 /// change, for example with build scripts, during a build.
765 /// This method can be used to bust all memoized hashes just before a build
766 /// to ensure that after a build completes everything is up-to-date.
767 pub fn clear_memoized(&self) {
768 *self.memoized_hash
.lock().unwrap() = None
;
771 fn hash(&self) -> u64 {
772 if let Some(s
) = *self.memoized_hash
.lock().unwrap() {
775 let ret
= util
::hash_u64(self);
776 *self.memoized_hash
.lock().unwrap() = Some(ret
);
780 /// Compares this fingerprint with an old version which was previously
781 /// serialized to filesystem.
783 /// The purpose of this is exclusively to produce a diagnostic message
784 /// indicating why we're recompiling something. This function always returns
785 /// an error, it will never return success.
786 fn compare(&self, old
: &Fingerprint
) -> CargoResult
<()> {
787 if self.rustc
!= old
.rustc
{
788 bail
!("rust compiler has changed")
790 if self.features
!= old
.features
{
792 "features have changed: {} != {}",
797 if self.target
!= old
.target
{
798 bail
!("target configuration has changed")
800 if self.path
!= old
.path
{
801 bail
!("path to the source has changed")
803 if self.profile
!= old
.profile
{
804 bail
!("profile configuration has changed")
806 if self.rustflags
!= old
.rustflags
{
808 "RUSTFLAGS has changed: {:?} != {:?}",
813 if self.metadata
!= old
.metadata
{
814 bail
!("metadata changed")
816 if self.config
!= old
.config
{
817 bail
!("configuration settings have changed")
819 let my_local
= self.local
.lock().unwrap();
820 let old_local
= old
.local
.lock().unwrap();
821 if my_local
.len() != old_local
.len() {
822 bail
!("local lens changed");
824 for (new
, old
) in my_local
.iter().zip(old_local
.iter()) {
826 (LocalFingerprint
::Precalculated(a
), LocalFingerprint
::Precalculated(b
)) => {
828 bail
!("precalculated components have changed: {} != {}", a
, b
)
832 LocalFingerprint
::CheckDepInfo { dep_info: adep }
,
833 LocalFingerprint
::CheckDepInfo { dep_info: bdep }
,
836 bail
!("dep info output changed: {:?} != {:?}", adep
, bdep
)
840 LocalFingerprint
::RerunIfChanged
{
844 LocalFingerprint
::RerunIfChanged
{
850 bail
!("rerun-if-changed output changed: {:?} != {:?}", aout
, bout
)
852 if apaths
!= bpaths
{
854 "rerun-if-changed output changed: {:?} != {:?}",
861 LocalFingerprint
::RerunIfEnvChanged
{
865 LocalFingerprint
::RerunIfEnvChanged
{
871 bail
!("env vars changed: {} != {}", akey
, bkey
);
873 if *avalue
!= *bvalue
{
875 "env var `{}` changed: previously {:?} now {:?}",
883 "local fingerprint type has changed ({} => {})",
890 if self.deps
.len() != old
.deps
.len() {
891 bail
!("number of dependencies has changed")
893 for (a
, b
) in self.deps
.iter().zip(old
.deps
.iter()) {
894 if a
.name
!= b
.name
{
895 let e
= format_err
!("`{}` != `{}`", a
.name
, b
.name
)
896 .context("unit dependency name changed");
900 if a
.fingerprint
.hash() != b
.fingerprint
.hash() {
902 "new ({}/{:x}) != old ({}/{:x})",
904 a
.fingerprint
.hash(),
908 .context("unit dependency information changed");
913 if !self.fs_status
.up_to_date() {
914 bail
!("current filesystem status shows we're outdated");
917 // This typically means some filesystem modifications happened or
918 // something transitive was odd. In general we should strive to provide
919 // a better error message than this, so if you see this message a lot it
920 // likely means this method needs to be updated!
921 bail
!("two fingerprint comparison turned up nothing obvious");
924 /// Dynamically inspect the local filesystem to update the `fs_status` field
925 /// of this `Fingerprint`.
927 /// This function is used just after a `Fingerprint` is constructed to check
928 /// the local state of the filesystem and propagate any dirtiness from
929 /// dependencies up to this unit as well. This function assumes that the
930 /// unit starts out as `FsStatus::Stale` and then it will optionally switch
931 /// it to `UpToDate` if it can.
934 mtime_cache
: &mut HashMap
<PathBuf
, FileTime
>,
937 ) -> CargoResult
<()> {
938 assert
!(!self.fs_status
.up_to_date());
940 let mut mtimes
= HashMap
::new();
942 // Get the `mtime` of all outputs. Optionally update their mtime
943 // afterwards based on the `mtime_on_use` flag. Afterwards we want the
944 // minimum mtime as it's the one we'll be comparing to inputs and
946 for output
in self.outputs
.iter() {
947 let mtime
= match paths
::mtime(output
) {
950 // This path failed to report its `mtime`. It probably doesn't
951 // exists, so leave ourselves as stale and bail out.
953 debug
!("failed to get mtime of {:?}: {}", output
, e
);
957 assert
!(mtimes
.insert(output
.clone(), mtime
).is_none());
960 let opt_max
= mtimes
.iter().max_by_key(|kv
| kv
.1);
961 let (max_path
, max_mtime
) = match opt_max
{
962 Some(mtime
) => mtime
,
964 // We had no output files. This means we're an overridden build
965 // script and we're just always up to date because we aren't
966 // watching the filesystem.
968 self.fs_status
= FsStatus
::UpToDate { mtimes }
;
973 "max output mtime for {:?} is {:?} {}",
974 pkg_root
, max_path
, max_mtime
977 for dep
in self.deps
.iter() {
978 let dep_mtimes
= match &dep
.fingerprint
.fs_status
{
979 FsStatus
::UpToDate { mtimes }
=> mtimes
,
980 // If our dependency is stale, so are we, so bail out.
981 FsStatus
::Stale
=> return Ok(()),
984 // If our dependency edge only requires the rmeta file to be present
985 // then we only need to look at that one output file, otherwise we
986 // need to consider all output files to see if we're out of date.
987 let (dep_path
, dep_mtime
) = if dep
.only_requires_rmeta
{
990 .find(|(path
, _mtime
)| {
991 path
.extension().and_then(|s
| s
.to_str()) == Some("rmeta")
993 .expect("failed to find rmeta")
995 match dep_mtimes
.iter().max_by_key(|kv
| kv
.1) {
996 Some(dep_mtime
) => dep_mtime
,
997 // If our dependencies is up to date and has no filesystem
998 // interactions, then we can move on to the next dependency.
1003 "max dep mtime for {:?} is {:?} {}",
1004 pkg_root
, dep_path
, dep_mtime
1007 // If the dependency is newer than our own output then it was
1008 // recompiled previously. We transitively become stale ourselves in
1009 // that case, so bail out.
1011 // Note that this comparison should probably be `>=`, not `>`, but
1012 // for a discussion of why it's `>` see the discussion about #5918
1013 // below in `find_stale`.
1014 if dep_mtime
> max_mtime
{
1016 "dependency on `{}` is newer than we are {} > {} {:?}",
1017 dep
.name
, dep_mtime
, max_mtime
, pkg_root
1023 // If we reached this far then all dependencies are up to date. Check
1024 // all our `LocalFingerprint` information to see if we have any stale
1025 // files for this package itself. If we do find something log a helpful
1026 // message and bail out so we stay stale.
1027 for local
in self.local
.get_mut().unwrap().iter() {
1028 if let Some(file
) = local
.find_stale_file(mtime_cache
, pkg_root
, target_root
)?
{
1034 // Everything was up to date! Record such.
1035 self.fs_status
= FsStatus
::UpToDate { mtimes }
;
1036 debug
!("filesystem up-to-date {:?}", pkg_root
);
1042 impl hash
::Hash
for Fingerprint
{
1043 fn hash
<H
: Hasher
>(&self, h
: &mut H
) {
1057 let local
= local
.lock().unwrap();
1059 rustc
, features
, target
, path
, profile
, &*local
, metadata
, config
, rustflags
,
1063 h
.write_usize(deps
.len());
1064 for DepFingerprint
{
1069 only_requires_rmeta
: _
, // static property, no need to hash
1075 // use memoized dep hashes to avoid exponential blowup
1076 h
.write_u64(Fingerprint
::hash(fingerprint
));
1081 impl hash
::Hash
for MtimeSlot
{
1082 fn hash
<H
: Hasher
>(&self, h
: &mut H
) {
1083 self.0.lock().unwrap().hash(h
)
1087 impl ser
::Serialize
for MtimeSlot
{
1088 fn serialize
<S
>(&self, s
: S
) -> Result
<S
::Ok
, S
::Error
>
1095 .map(|ft
| (ft
.unix_seconds(), ft
.nanoseconds()))
1100 impl<'de
> de
::Deserialize
<'de
> for MtimeSlot
{
1101 fn deserialize
<D
>(d
: D
) -> Result
<MtimeSlot
, D
::Error
>
1103 D
: de
::Deserializer
<'de
>,
1105 let kind
: Option
<(i64, u32)> = de
::Deserialize
::deserialize(d
)?
;
1106 Ok(MtimeSlot(Mutex
::new(
1107 kind
.map(|(s
, n
)| FileTime
::from_unix_time(s
, n
)),
1112 impl DepFingerprint
{
1113 fn new(cx
: &mut Context
<'_
, '_
>, parent
: &Unit
, dep
: &UnitDep
) -> CargoResult
<DepFingerprint
> {
1114 let fingerprint
= calculate(cx
, &dep
.unit
)?
;
1115 // We need to be careful about what we hash here. We have a goal of
1116 // supporting renaming a project directory and not rebuilding
1117 // everything. To do that, however, we need to make sure that the cwd
1118 // doesn't make its way into any hashes, and one source of that is the
1119 // `SourceId` for `path` packages.
1121 // We already have a requirement that `path` packages all have unique
1122 // names (sort of for this same reason), so if the package source is a
1123 // `path` then we just hash the name, but otherwise we hash the full
1124 // id as it won't change when the directory is renamed.
1125 let pkg_id
= if dep
.unit
.pkg
.package_id().source_id().is_path() {
1126 util
::hash_u64(dep
.unit
.pkg
.package_id().name())
1128 util
::hash_u64(dep
.unit
.pkg
.package_id())
1133 name
: dep
.extern_crate_name
,
1136 only_requires_rmeta
: cx
.only_requires_rmeta(parent
, &dep
.unit
),
1142 /// Use the `log` crate to log a hopefully helpful message in diagnosing
1143 /// what file is considered stale and why. This is intended to be used in
1144 /// conjunction with `CARGO_LOG` to determine why Cargo is recompiling
1145 /// something. Currently there's no user-facing usage of this other than
1149 StaleFile
::Missing(path
) => {
1150 info
!("stale: missing {:?}", path
);
1152 StaleFile
::Changed
{
1158 info
!("stale: changed {:?}", stale
);
1159 info
!(" (vs) {:?}", reference
);
1160 info
!(" {:?} != {:?}", reference_mtime
, stale_mtime
);
1166 /// Calculates the fingerprint for a `unit`.
1168 /// This fingerprint is used by Cargo to learn about when information such as:
1170 /// * A non-path package changes (changes version, changes revision, etc).
1171 /// * Any dependency changes
1172 /// * The compiler changes
1173 /// * The set of features a package is built with changes
1174 /// * The profile a target is compiled with changes (e.g., opt-level changes)
1175 /// * Any other compiler flags change that will affect the result
1177 /// Information like file modification time is only calculated for path
1179 fn calculate(cx
: &mut Context
<'_
, '_
>, unit
: &Unit
) -> CargoResult
<Arc
<Fingerprint
>> {
1180 // This function is slammed quite a lot, so the result is memoized.
1181 if let Some(s
) = cx
.fingerprints
.get(unit
) {
1182 return Ok(Arc
::clone(s
));
1184 let mut fingerprint
= if unit
.mode
.is_run_custom_build() {
1185 calculate_run_custom_build(cx
, unit
)?
1186 } else if unit
.mode
.is_doc_test() {
1187 panic
!("doc tests do not fingerprint");
1189 calculate_normal(cx
, unit
)?
1192 // After we built the initial `Fingerprint` be sure to update the
1193 // `fs_status` field of it.
1194 let target_root
= target_root(cx
);
1195 fingerprint
.check_filesystem(&mut cx
.mtime_cache
, unit
.pkg
.root(), &target_root
)?
;
1197 let fingerprint
= Arc
::new(fingerprint
);
1199 .insert(unit
.clone(), Arc
::clone(&fingerprint
));
1203 /// Calculate a fingerprint for a "normal" unit, or anything that's not a build
1204 /// script. This is an internal helper of `calculate`, don't call directly.
1205 fn calculate_normal(cx
: &mut Context
<'_
, '_
>, unit
: &Unit
) -> CargoResult
<Fingerprint
> {
1206 // Recursively calculate the fingerprint for all of our dependencies.
1208 // Skip fingerprints of binaries because they don't actually induce a
1209 // recompile, they're just dependencies in the sense that they need to be
1212 // Create Vec since mutable cx is needed in closure.
1213 let deps
= Vec
::from(cx
.unit_deps(unit
));
1216 .filter(|dep
| !dep
.unit
.target
.is_bin())
1217 .map(|dep
| DepFingerprint
::new(cx
, unit
, &dep
))
1218 .collect
::<CargoResult
<Vec
<_
>>>()?
;
1219 deps
.sort_by(|a
, b
| a
.pkg_id
.cmp(&b
.pkg_id
));
1221 // Afterwards calculate our own fingerprint information.
1222 let target_root
= target_root(cx
);
1223 let local
= if unit
.mode
.is_doc() {
1224 // rustdoc does not have dep-info files.
1225 let fingerprint
= pkg_fingerprint(cx
.bcx
, &unit
.pkg
).chain_err(|| {
1227 "failed to determine package fingerprint for documenting {}",
1231 vec
![LocalFingerprint
::Precalculated(fingerprint
)]
1233 let dep_info
= dep_info_loc(cx
, unit
);
1234 let dep_info
= dep_info
.strip_prefix(&target_root
).unwrap().to_path_buf();
1235 vec
![LocalFingerprint
::CheckDepInfo { dep_info }
]
1238 // Figure out what the outputs of our unit is, and we'll be storing them
1239 // into the fingerprint as well.
1243 .filter(|output
| !matches
!(output
.flavor
, FileFlavor
::DebugInfo
| FileFlavor
::Auxiliary
))
1244 .map(|output
| output
.path
.clone())
1247 // Fill out a bunch more information that we'll be tracking typically
1248 // hashed to take up less space on disk as we just need to know when things
1250 let extra_flags
= if unit
.mode
.is_doc() {
1251 cx
.bcx
.rustdocflags_args(unit
)
1253 cx
.bcx
.rustflags_args(unit
)
1257 let profile_hash
= util
::hash_u64((
1260 cx
.bcx
.extra_args_for(unit
),
1263 // Include metadata since it is exposed as environment variables.
1264 let m
= unit
.pkg
.manifest().metadata();
1265 let metadata
= util
::hash_u64((&m
.authors
, &m
.description
, &m
.homepage
, &m
.repository
));
1266 let config
= if unit
.mode
.is_doc() && cx
.bcx
.config
.cli_unstable().rustdoc_map
{
1270 .map_or(0, |map
| util
::hash_u64(map
))
1275 rustc
: util
::hash_u64(&cx
.bcx
.rustc().verbose_version
),
1276 target
: util
::hash_u64(&unit
.target
),
1277 profile
: profile_hash
,
1278 // Note that .0 is hashed here, not .1 which is the cwd. That doesn't
1279 // actually affect the output artifact so there's no need to hash it.
1280 path
: util
::hash_u64(super::path_args(cx
.bcx
, unit
).0),
1281 features
: format
!("{:?}", unit
.features
),
1283 local
: Mutex
::new(local
),
1284 memoized_hash
: Mutex
::new(None
),
1287 rustflags
: extra_flags
,
1288 fs_status
: FsStatus
::Stale
,
1293 /// Calculate a fingerprint for an "execute a build script" unit. This is an
1294 /// internal helper of `calculate`, don't call directly.
1295 fn calculate_run_custom_build(cx
: &mut Context
<'_
, '_
>, unit
: &Unit
) -> CargoResult
<Fingerprint
> {
1296 assert
!(unit
.mode
.is_run_custom_build());
1297 // Using the `BuildDeps` information we'll have previously parsed and
1298 // inserted into `build_explicit_deps` built an initial snapshot of the
1299 // `LocalFingerprint` list for this build script. If we previously executed
1300 // the build script this means we'll be watching files and env vars.
1301 // Otherwise if we haven't previously executed it we'll just start watching
1303 let (gen_local
, overridden
) = build_script_local_fingerprints(cx
, unit
);
1304 let deps
= &cx
.build_explicit_deps
[unit
];
1305 let local
= (gen_local
)(
1308 pkg_fingerprint(cx
.bcx
, &unit
.pkg
).chain_err(|| {
1310 "failed to determine package fingerprint for build script for {}",
1317 let output
= deps
.build_script_output
.clone();
1319 // Include any dependencies of our execution, which is typically just the
1320 // compilation of the build script itself. (if the build script changes we
1321 // should be rerun!). Note though that if we're an overridden build script
1322 // we have no dependencies so no need to recurse in that case.
1323 let deps
= if overridden
{
1324 // Overridden build scripts don't need to track deps.
1327 // Create Vec since mutable cx is needed in closure.
1328 let deps
= Vec
::from(cx
.unit_deps(unit
));
1330 .map(|dep
| DepFingerprint
::new(cx
, unit
, &dep
))
1331 .collect
::<CargoResult
<Vec
<_
>>>()?
1335 local
: Mutex
::new(local
),
1336 rustc
: util
::hash_u64(&cx
.bcx
.rustc().verbose_version
),
1338 outputs
: if overridden { Vec::new() }
else { vec![output] }
,
1340 // Most of the other info is blank here as we don't really include it
1341 // in the execution of the build script, but... this may be a latent
1343 ..Fingerprint
::new()
1347 /// Get ready to compute the `LocalFingerprint` values for a `RunCustomBuild`
1350 /// This function has, what's on the surface, a seriously wonky interface.
1351 /// You'll call this function and it'll return a closure and a boolean. The
1352 /// boolean is pretty simple in that it indicates whether the `unit` has been
1353 /// overridden via `.cargo/config`. The closure is much more complicated.
1355 /// This closure is intended to capture any local state necessary to compute
1356 /// the `LocalFingerprint` values for this unit. It is `Send` and `'static` to
1357 /// be sent to other threads as well (such as when we're executing build
1358 /// scripts). That deduplication is the rationale for the closure at least.
1360 /// The arguments to the closure are a bit weirder, though, and I'll apologize
1361 /// in advance for the weirdness too. The first argument to the closure is a
1362 /// `&BuildDeps`. This is the parsed version of a build script, and when Cargo
1363 /// starts up this is cached from previous runs of a build script. After a
1364 /// build script executes the output file is reparsed and passed in here.
1366 /// The second argument is the weirdest, it's *optionally* a closure to
1367 /// call `pkg_fingerprint` below. The `pkg_fingerprint` below requires access
1368 /// to "source map" located in `Context`. That's very non-`'static` and
1369 /// non-`Send`, so it can't be used on other threads, such as when we invoke
1370 /// this after a build script has finished. The `Option` allows us to for sure
1371 /// calculate it on the main thread at the beginning, and then swallow the bug
1372 /// for now where a worker thread after a build script has finished doesn't
1373 /// have access. Ideally there would be no second argument or it would be more
1374 /// "first class" and not an `Option` but something that can be sent between
1375 /// threads. In any case, it's a bug for now.
1377 /// This isn't the greatest of interfaces, and if there's suggestions to
1378 /// improve please do so!
1380 /// FIXME(#6779) - see all the words above
1381 fn build_script_local_fingerprints(
1382 cx
: &mut Context
<'_
, '_
>,
1388 Option
<&dyn Fn() -> CargoResult
<String
>>,
1389 ) -> CargoResult
<Option
<Vec
<LocalFingerprint
>>>
1394 assert
!(unit
.mode
.is_run_custom_build());
1395 // First up, if this build script is entirely overridden, then we just
1396 // return the hash of what we overrode it with. This is the easy case!
1397 if let Some(fingerprint
) = build_script_override_fingerprint(cx
, unit
) {
1398 debug
!("override local fingerprints deps {}", unit
.pkg
);
1401 move |_
: &BuildDeps
, _
: Option
<&dyn Fn() -> CargoResult
<String
>>| {
1402 Ok(Some(vec
![fingerprint
]))
1405 true, // this is an overridden build script
1409 // ... Otherwise this is a "real" build script and we need to return a real
1410 // closure. Our returned closure classifies the build script based on
1411 // whether it prints `rerun-if-*`. If it *doesn't* print this it's where the
1412 // magical second argument comes into play, which fingerprints a whole
1413 // package. Remember that the fact that this is an `Option` is a bug, but a
1414 // longstanding bug, in Cargo. Recent refactorings just made it painfully
1416 let pkg_root
= unit
.pkg
.root().to_path_buf();
1417 let target_dir
= target_root(cx
);
1419 move |deps
: &BuildDeps
, pkg_fingerprint
: Option
<&dyn Fn() -> CargoResult
<String
>>| {
1420 if deps
.rerun_if_changed
.is_empty() && deps
.rerun_if_env_changed
.is_empty() {
1421 match pkg_fingerprint
{
1422 // FIXME: this is somewhat buggy with respect to docker and
1423 // weird filesystems. The `Precalculated` variant
1424 // constructed below will, for `path` dependencies, contain
1425 // a stringified version of the mtime for the local crate.
1426 // This violates one of the things we describe in this
1427 // module's doc comment, never hashing mtimes. We should
1428 // figure out a better scheme where a package fingerprint
1429 // may be a string (like for a registry) or a list of files
1430 // (like for a path dependency). Those list of files would
1431 // be stored here rather than the the mtime of them.
1435 "old local fingerprints deps {:?} precalculated={:?}",
1438 return Ok(Some(vec
![LocalFingerprint
::Precalculated(s
)]));
1440 None
=> return Ok(None
),
1444 // Ok so now we're in "new mode" where we can have files listed as
1445 // dependencies as well as env vars listed as dependencies. Process
1447 Ok(Some(local_fingerprints_deps(deps
, &target_dir
, &pkg_root
)))
1450 // Note that `false` == "not overridden"
1451 (Box
::new(calculate
), false)
1454 /// Create a `LocalFingerprint` for an overridden build script.
1455 /// Returns None if it is not overridden.
1456 fn build_script_override_fingerprint(
1457 cx
: &mut Context
<'_
, '_
>,
1459 ) -> Option
<LocalFingerprint
> {
1460 // Build script output is only populated at this stage when it is
1462 let build_script_outputs
= cx
.build_script_outputs
.lock().unwrap();
1463 let metadata
= cx
.get_run_build_script_metadata(unit
);
1464 // Returns None if it is not overridden.
1465 let output
= build_script_outputs
.get(unit
.pkg
.package_id(), metadata
)?
;
1467 "overridden build state with hash: {}",
1468 util
::hash_u64(output
)
1470 Some(LocalFingerprint
::Precalculated(s
))
1473 /// Compute the `LocalFingerprint` values for a `RunCustomBuild` unit for
1474 /// non-overridden new-style build scripts only. This is only used when `deps`
1475 /// is already known to have a nonempty `rerun-if-*` somewhere.
1476 fn local_fingerprints_deps(
1480 ) -> Vec
<LocalFingerprint
> {
1481 debug
!("new local fingerprints deps {:?}", pkg_root
);
1482 let mut local
= Vec
::new();
1484 if !deps
.rerun_if_changed
.is_empty() {
1485 // Note that like the module comment above says we are careful to never
1486 // store an absolute path in `LocalFingerprint`, so ensure that we strip
1487 // absolute prefixes from them.
1489 .build_script_output
1490 .strip_prefix(target_root
)
1496 .map(|p
| p
.strip_prefix(pkg_root
).unwrap_or(p
).to_path_buf())
1498 local
.push(LocalFingerprint
::RerunIfChanged { output, paths }
);
1501 for var
in deps
.rerun_if_env_changed
.iter() {
1502 let val
= env
::var(var
).ok();
1503 local
.push(LocalFingerprint
::RerunIfEnvChanged
{
1512 fn write_fingerprint(loc
: &Path
, fingerprint
: &Fingerprint
) -> CargoResult
<()> {
1513 debug_assert_ne
!(fingerprint
.rustc
, 0);
1514 // fingerprint::new().rustc == 0, make sure it doesn't make it to the file system.
1515 // This is mostly so outside tools can reliably find out what rust version this file is for,
1516 // as we can use the full hash.
1517 let hash
= fingerprint
.hash();
1518 debug
!("write fingerprint ({:x}) : {}", hash
, loc
.display());
1519 paths
::write(loc
, util
::to_hex(hash
).as_bytes())?
;
1521 let json
= serde_json
::to_string(fingerprint
).unwrap();
1522 if cfg
!(debug_assertions
) {
1523 let f
: Fingerprint
= serde_json
::from_str(&json
).unwrap();
1524 assert_eq
!(f
.hash(), hash
);
1526 paths
::write(&loc
.with_extension("json"), json
.as_bytes())?
;
1530 /// Prepare for work when a package starts to build
1531 pub fn prepare_init(cx
: &mut Context
<'_
, '_
>, unit
: &Unit
) -> CargoResult
<()> {
1532 let new1
= cx
.files().fingerprint_dir(unit
);
1534 // Doc tests have no output, thus no fingerprint.
1535 if !new1
.exists() && !unit
.mode
.is_doc_test() {
1536 paths
::create_dir_all(&new1
)?
;
1542 /// Returns the location that the dep-info file will show up at for the `unit`
1544 pub fn dep_info_loc(cx
: &mut Context
<'_
, '_
>, unit
: &Unit
) -> PathBuf
{
1545 cx
.files().fingerprint_file_path(unit
, "dep-")
1548 /// Returns an absolute path that target directory.
1549 /// All paths are rewritten to be relative to this.
1550 fn target_root(cx
: &Context
<'_
, '_
>) -> PathBuf
{
1551 cx
.bcx
.ws
.target_dir().into_path_unlocked()
1554 fn compare_old_fingerprint(
1556 new_fingerprint
: &Fingerprint
,
1558 ) -> CargoResult
<()> {
1559 let old_fingerprint_short
= paths
::read(loc
)?
;
1562 // update the mtime so other cleaners know we used it
1563 let t
= FileTime
::from_system_time(SystemTime
::now());
1564 debug
!("mtime-on-use forcing {:?} to {}", loc
, t
);
1565 paths
::set_file_time_no_err(loc
, t
);
1568 let new_hash
= new_fingerprint
.hash();
1570 if util
::to_hex(new_hash
) == old_fingerprint_short
&& new_fingerprint
.fs_status
.up_to_date() {
1574 let old_fingerprint_json
= paths
::read(&loc
.with_extension("json"))?
;
1575 let old_fingerprint
: Fingerprint
= serde_json
::from_str(&old_fingerprint_json
)
1576 .chain_err(|| internal("failed to deserialize json"))?
;
1577 // Fingerprint can be empty after a failed rebuild (see comment in prepare_target).
1578 if !old_fingerprint_short
.is_empty() {
1579 debug_assert_eq
!(util
::to_hex(old_fingerprint
.hash()), old_fingerprint_short
);
1581 let result
= new_fingerprint
.compare(&old_fingerprint
);
1582 assert
!(result
.is_err());
1586 fn log_compare(unit
: &Unit
, compare
: &CargoResult
<()>) {
1587 let ce
= match compare
{
1592 "fingerprint error for {}/{:?}/{:?}",
1593 unit
.pkg
, unit
.mode
, unit
.target
,
1595 info
!(" err: {:?}", ce
);
1598 // Parse the dep-info into a list of paths
1599 pub fn parse_dep_info(
1603 ) -> CargoResult
<Option
<Vec
<PathBuf
>>> {
1604 let data
= match paths
::read_bytes(dep_info
) {
1606 Err(_
) => return Ok(None
),
1610 .filter(|x
| !x
.is_empty())
1612 let ty
= match DepInfoPathType
::from_byte(p
[0]) {
1614 None
=> return Err(internal("dep-info invalid")),
1616 let path
= util
::bytes2path(&p
[1..])?
;
1618 DepInfoPathType
::PackageRootRelative
=> Ok(pkg_root
.join(path
)),
1619 // N.B. path might be absolute here in which case the join will have no effect
1620 DepInfoPathType
::TargetRootRelative
=> Ok(target_root
.join(path
)),
1623 .collect
::<Result
<Vec
<_
>, _
>>()?
;
1627 fn pkg_fingerprint(bcx
: &BuildContext
<'_
, '_
>, pkg
: &Package
) -> CargoResult
<String
> {
1628 let source_id
= pkg
.package_id().source_id();
1629 let sources
= bcx
.packages
.sources();
1631 let source
= sources
1633 .ok_or_else(|| internal("missing package source"))?
;
1634 source
.fingerprint(pkg
)
1637 fn find_stale_file
<I
>(
1638 mtime_cache
: &mut HashMap
<PathBuf
, FileTime
>,
1641 ) -> Option
<StaleFile
>
1644 I
::Item
: AsRef
<Path
>,
1646 let reference_mtime
= match paths
::mtime(reference
) {
1648 Err(..) => return Some(StaleFile
::Missing(reference
.to_path_buf())),
1652 let path
= path
.as_ref();
1653 let path_mtime
= match mtime_cache
.entry(path
.to_path_buf()) {
1654 Entry
::Occupied(o
) => *o
.get(),
1655 Entry
::Vacant(v
) => {
1656 let mtime
= match paths
::mtime(path
) {
1658 Err(..) => return Some(StaleFile
::Missing(path
.to_path_buf())),
1665 // Note that equal mtimes should be considered "stale". For filesystems with
1666 // not much timestamp precision like 1s this is would be a conservative approximation
1667 // to handle the case where a file is modified within the same second after
1668 // a build starts. We want to make sure that incremental rebuilds pick that up!
1670 // For filesystems with nanosecond precision it's been seen in the wild that
1671 // its "nanosecond precision" isn't really nanosecond-accurate. It turns out that
1672 // kernels may cache the current time so files created at different times actually
1673 // list the same nanosecond precision. Some digging on #5919 picked up that the
1674 // kernel caches the current time between timer ticks, which could mean that if
1675 // a file is updated at most 10ms after a build starts then Cargo may not
1676 // pick up the build changes.
1678 // All in all, an equality check here would be a conservative assumption that,
1679 // if equal, files were changed just after a previous build finished.
1680 // Unfortunately this became problematic when (in #6484) cargo switch to more accurately
1681 // measuring the start time of builds.
1682 if path_mtime
<= reference_mtime
{
1686 return Some(StaleFile
::Changed
{
1687 reference
: reference
.to_path_buf(),
1689 stale
: path
.to_path_buf(),
1690 stale_mtime
: path_mtime
,
1695 "all paths up-to-date relative to {:?} mtime={}",
1696 reference
, reference_mtime
1702 enum DepInfoPathType
{
1703 // src/, e.g. src/lib.rs
1704 PackageRootRelative
= 1,
1705 // target/debug/deps/lib...
1706 // or an absolute path /.../sysroot/...
1707 TargetRootRelative
= 2,
1710 impl DepInfoPathType
{
1711 fn from_byte(b
: u8) -> Option
<DepInfoPathType
> {
1713 1 => Some(DepInfoPathType
::PackageRootRelative
),
1714 2 => Some(DepInfoPathType
::TargetRootRelative
),
1720 /// Parses the dep-info file coming out of rustc into a Cargo-specific format.
1722 /// This function will parse `rustc_dep_info` as a makefile-style dep info to
1723 /// learn about the all files which a crate depends on. This is then
1724 /// re-serialized into the `cargo_dep_info` path in a Cargo-specific format.
1726 /// The `pkg_root` argument here is the absolute path to the directory
1727 /// containing `Cargo.toml` for this crate that was compiled. The paths listed
1728 /// in the rustc dep-info file may or may not be absolute but we'll want to
1729 /// consider all of them relative to the `root` specified.
1731 /// The `rustc_cwd` argument is the absolute path to the cwd of the compiler
1732 /// when it was invoked.
1734 /// If the `allow_package` argument is true, then package-relative paths are
1735 /// included. If it is false, then package-relative paths are skipped and
1736 /// ignored (typically used for registry or git dependencies where we assume
1737 /// the source never changes, and we don't want the cost of running `stat` on
1738 /// all those files). See the module-level docs for the note about
1739 /// `-Zbinary-dep-depinfo` for more details on why this is done.
1741 /// The serialized Cargo format will contain a list of files, all of which are
1742 /// relative if they're under `root`. or absolute if they're elsewhere.
1743 pub fn translate_dep_info(
1744 rustc_dep_info
: &Path
,
1745 cargo_dep_info
: &Path
,
1749 allow_package
: bool
,
1750 ) -> CargoResult
<()> {
1751 let target
= parse_rustc_dep_info(rustc_dep_info
)?
;
1754 .ok_or_else(|| internal("malformed dep-info format, no targets".to_string()))?
1757 let target_root
= target_root
.canonicalize()?
;
1758 let pkg_root
= pkg_root
.canonicalize()?
;
1759 let mut new_contents
= Vec
::new();
1761 // The path may be absolute or relative, canonical or not. Make sure
1762 // it is canonicalized so we are comparing the same kinds of paths.
1763 let abs_file
= rustc_cwd
.join(file
);
1764 // If canonicalization fails, just use the abs path. There is currently
1765 // a bug where --remap-path-prefix is affecting .d files, causing them
1766 // to point to non-existent paths.
1767 let canon_file
= abs_file
.canonicalize().unwrap_or_else(|_
| abs_file
.clone());
1769 let (ty
, path
) = if let Ok(stripped
) = canon_file
.strip_prefix(&target_root
) {
1770 (DepInfoPathType
::TargetRootRelative
, stripped
)
1771 } else if let Ok(stripped
) = canon_file
.strip_prefix(&pkg_root
) {
1775 (DepInfoPathType
::PackageRootRelative
, stripped
)
1777 // It's definitely not target root relative, but this is an absolute path (since it was
1778 // joined to rustc_cwd) and as such re-joining it later to the target root will have no
1780 (DepInfoPathType
::TargetRootRelative
, &*abs_file
)
1782 new_contents
.push(ty
as u8);
1783 new_contents
.extend(util
::path2bytes(path
)?
);
1784 new_contents
.push(0);
1786 paths
::write(cargo_dep_info
, &new_contents
)?
;
1790 /// Parse the `.d` dep-info file generated by rustc.
1792 /// Result is a Vec of `(target, prerequisites)` tuples where `target` is the
1793 /// rule name, and `prerequisites` is a list of files that it depends on.
1794 pub fn parse_rustc_dep_info(rustc_dep_info
: &Path
) -> CargoResult
<Vec
<(String
, Vec
<String
>)>> {
1795 let contents
= paths
::read(rustc_dep_info
)?
;
1798 .filter_map(|l
| l
.find(": ").map(|i
| (l
, i
)))
1799 .map(|(line
, pos
)| {
1800 let target
= &line
[..pos
];
1801 let mut deps
= line
[pos
+ 2..].split_whitespace();
1803 let mut ret
= Vec
::new();
1804 while let Some(s
) = deps
.next() {
1805 let mut file
= s
.to_string();
1806 while file
.ends_with('
\\'
) {
1809 file
.push_str(deps
.next().ok_or_else(|| {
1810 internal("malformed dep-info format, trailing \\".to_string())
1815 Ok((target
.to_string(), ret
))