1 //! This module manages how the incremental compilation cache is represented in
4 //! Incremental compilation caches are managed according to a copy-on-write
5 //! strategy: Once a complete, consistent cache version is finalized, it is
6 //! never modified. Instead, when a subsequent compilation session is started,
7 //! the compiler will allocate a new version of the cache that starts out as
8 //! a copy of the previous version. Then only this new copy is modified and it
9 //! will not be visible to other processes until it is finalized. This ensures
10 //! that multiple compiler processes can be executed concurrently for the same
11 //! crate without interfering with each other or blocking each other.
13 //! More concretely this is implemented via the following protocol:
15 //! 1. For a newly started compilation session, the compiler allocates a
16 //! new `session` directory within the incremental compilation directory.
17 //! This session directory will have a unique name that ends with the suffix
18 //! "-working" and that contains a creation timestamp.
19 //! 2. Next, the compiler looks for the newest finalized session directory,
20 //! that is, a session directory from a previous compilation session that
21 //! has been marked as valid and consistent. A session directory is
22 //! considered finalized if the "-working" suffix in the directory name has
23 //! been replaced by the SVH of the crate.
24 //! 3. Once the compiler has found a valid, finalized session directory, it will
25 //! hard-link/copy its contents into the new "-working" directory. If all
26 //! goes well, it will have its own, private copy of the source directory and
27 //! subsequently not have to worry about synchronizing with other compiler
29 //! 4. Now the compiler can do its normal compilation process, which involves
30 //! reading and updating its private session directory.
31 //! 5. When compilation finishes without errors, the private session directory
32 //! will be in a state where it can be used as input for other compilation
33 //! sessions. That is, it will contain a dependency graph and cache artifacts
34 //! that are consistent with the state of the source code it was compiled
35 //! from, with no need to change them ever again. At this point, the compiler
36 //! finalizes and "publishes" its private session directory by renaming it
37 //! from "s-{timestamp}-{random}-working" to "s-{timestamp}-{SVH}".
38 //! 6. At this point the "old" session directory that we copied our data from
39 //! at the beginning of the session has become obsolete because we have just
40 //! published a more current version. Thus the compiler will delete it.
42 //! ## Garbage Collection
44 //! Naively following the above protocol might lead to old session directories
45 //! piling up if a compiler instance crashes for some reason before its able to
46 //! remove its private session directory. In order to avoid wasting disk space,
47 //! the compiler also does some garbage collection each time it is started in
48 //! incremental compilation mode. Specifically, it will scan the incremental
49 //! compilation directory for private session directories that are not in use
50 //! any more and will delete those. It will also delete any finalized session
51 //! directories for a given crate except for the most recent one.
53 //! ## Synchronization
55 //! There is some synchronization needed in order for the compiler to be able to
56 //! determine whether a given private session directory is not in use any more.
57 //! This is done by creating a lock file for each session directory and
58 //! locking it while the directory is still being used. Since file locks have
59 //! operating system support, we can rely on the lock being released if the
60 //! compiler process dies for some unexpected reason. Thus, when garbage
61 //! collecting private session directories, the collecting process can determine
62 //! whether the directory is still in use by trying to acquire a lock on the
63 //! file. If locking the file fails, the original process must still be alive.
64 //! If locking the file succeeds, we know that the owning process is not alive
65 //! any more and we can safely delete the directory.
66 //! There is still a small time window between the original process creating the
67 //! lock file and actually locking it. In order to minimize the chance that
68 //! another process tries to acquire the lock in just that instance, only
69 //! session directories that are older than a few seconds are considered for
70 //! garbage collection.
72 //! Another case that has to be considered is what happens if one process
73 //! deletes a finalized session directory that another process is currently
74 //! trying to copy from. This case is also handled via the lock file. Before
75 //! a process starts copying a finalized session directory, it will acquire a
76 //! shared lock on the directory's lock file. Any garbage collecting process,
77 //! on the other hand, will acquire an exclusive lock on the lock file.
78 //! Thus, if a directory is being collected, any reader process will fail
79 //! acquiring the shared lock and will leave the directory alone. Conversely,
80 //! if a collecting process can't acquire the exclusive lock because the
81 //! directory is currently being read from, it will leave collecting that
82 //! directory to another process at a later point in time.
83 //! The exact same scheme is also used when reading the metadata hashes file
84 //! from an extern crate. When a crate is compiled, the hash values of its
85 //! metadata are stored in a file in its session directory. When the
86 //! compilation session of another crate imports the first crate's metadata,
87 //! it also has to read in the accompanying metadata hashes. It thus will access
88 //! the finalized session directory of all crates it links to and while doing
89 //! so, it will also place a read lock on that the respective session directory
90 //! so that it won't be deleted while the metadata hashes are loaded.
94 //! This system relies on two features being available in the file system in
95 //! order to work really well: file locking and hard linking.
96 //! If hard linking is not available (like on FAT) the data in the cache
97 //! actually has to be copied at the beginning of each session.
98 //! If file locking does not work reliably (like on NFS), some of the
99 //! synchronization will go haywire.
100 //! In both cases we recommend to locate the incremental compilation directory
101 //! on a file system that supports these things.
102 //! It might be a good idea though to try and detect whether we are on an
103 //! unsupported file system and emit a warning in that case. This is not yet
107 use rustc_data_structures
::fx
::{FxHashSet, FxIndexSet}
;
108 use rustc_data_structures
::svh
::Svh
;
109 use rustc_data_structures
::unord
::{UnordMap, UnordSet}
;
110 use rustc_data_structures
::{base_n, flock}
;
111 use rustc_errors
::ErrorGuaranteed
;
112 use rustc_fs_util
::{link_or_copy, try_canonicalize, LinkOrCopy}
;
113 use rustc_session
::{Session, StableCrateId}
;
114 use rustc_span
::Symbol
;
116 use std
::fs
as std_fs
;
117 use std
::io
::{self, ErrorKind}
;
118 use std
::path
::{Path, PathBuf}
;
119 use std
::time
::{Duration, SystemTime, UNIX_EPOCH}
;
121 use rand
::{thread_rng, RngCore}
;
126 const LOCK_FILE_EXT
: &str = ".lock";
127 const DEP_GRAPH_FILENAME
: &str = "dep-graph.bin";
128 const STAGING_DEP_GRAPH_FILENAME
: &str = "dep-graph.part.bin";
129 const WORK_PRODUCTS_FILENAME
: &str = "work-products.bin";
130 const QUERY_CACHE_FILENAME
: &str = "query-cache.bin";
132 // We encode integers using the following base, so they are shorter than decimal
133 // or hexadecimal numbers (we want short file and directory names). Since these
134 // numbers will be used in file names, we choose an encoding that is not
135 // case-sensitive (as opposed to base64, for example).
136 const INT_ENCODE_BASE
: usize = base_n
::CASE_INSENSITIVE
;
138 /// Returns the path to a session's dependency graph.
139 pub(crate) fn dep_graph_path(sess
: &Session
) -> PathBuf
{
140 in_incr_comp_dir_sess(sess
, DEP_GRAPH_FILENAME
)
143 /// Returns the path to a session's staging dependency graph.
145 /// On the difference between dep-graph and staging dep-graph,
146 /// see `build_dep_graph`.
147 pub(crate) fn staging_dep_graph_path(sess
: &Session
) -> PathBuf
{
148 in_incr_comp_dir_sess(sess
, STAGING_DEP_GRAPH_FILENAME
)
151 pub(crate) fn work_products_path(sess
: &Session
) -> PathBuf
{
152 in_incr_comp_dir_sess(sess
, WORK_PRODUCTS_FILENAME
)
155 /// Returns the path to a session's query cache.
156 pub fn query_cache_path(sess
: &Session
) -> PathBuf
{
157 in_incr_comp_dir_sess(sess
, QUERY_CACHE_FILENAME
)
160 /// Locks a given session directory.
161 fn lock_file_path(session_dir
: &Path
) -> PathBuf
{
162 let crate_dir
= session_dir
.parent().unwrap();
164 let directory_name
= session_dir
.file_name().unwrap().to_string_lossy();
165 assert_no_characters_lost(&directory_name
);
167 let dash_indices
: Vec
<_
> = directory_name
.match_indices('
-'
).map(|(idx
, _
)| idx
).collect();
168 if dash_indices
.len() != 3 {
170 "Encountered incremental compilation session directory with \
172 session_dir
.display()
176 crate_dir
.join(&directory_name
[0..dash_indices
[2]]).with_extension(&LOCK_FILE_EXT
[1..])
179 /// Returns the path for a given filename within the incremental compilation directory
180 /// in the current session.
181 pub fn in_incr_comp_dir_sess(sess
: &Session
, file_name
: &str) -> PathBuf
{
182 in_incr_comp_dir(&sess
.incr_comp_session_dir(), file_name
)
185 /// Returns the path for a given filename within the incremental compilation directory,
186 /// not necessarily from the current session.
188 /// To ensure the file is part of the current session, use [`in_incr_comp_dir_sess`].
189 pub fn in_incr_comp_dir(incr_comp_session_dir
: &Path
, file_name
: &str) -> PathBuf
{
190 incr_comp_session_dir
.join(file_name
)
193 /// Allocates the private session directory.
195 /// If the result of this function is `Ok`, we have a valid incremental
196 /// compilation session directory. A valid session
197 /// directory is one that contains a locked lock file. It may or may not contain
198 /// a dep-graph and work products from a previous session.
200 /// This always attempts to load a dep-graph from the directory.
201 /// If loading fails for some reason, we fallback to a disabled `DepGraph`.
202 /// See [`rustc_interface::queries::dep_graph`].
204 /// If this function returns an error, it may leave behind an invalid session directory.
205 /// The garbage collection will take care of it.
207 /// [`rustc_interface::queries::dep_graph`]: ../../rustc_interface/struct.Queries.html#structfield.dep_graph
208 pub(crate) fn prepare_session_directory(
211 stable_crate_id
: StableCrateId
,
212 ) -> Result
<(), ErrorGuaranteed
> {
213 if sess
.opts
.incremental
.is_none() {
217 let _timer
= sess
.timer("incr_comp_prepare_session_directory");
219 debug
!("prepare_session_directory");
221 // {incr-comp-dir}/{crate-name-and-disambiguator}
222 let crate_dir
= crate_path(sess
, crate_name
, stable_crate_id
);
223 debug
!("crate-dir: {}", crate_dir
.display());
224 create_dir(sess
, &crate_dir
, "crate")?
;
226 // Hack: canonicalize the path *after creating the directory*
227 // because, on windows, long paths can cause problems;
228 // canonicalization inserts this weird prefix that makes windows
229 // tolerate long paths.
230 let crate_dir
= match try_canonicalize(&crate_dir
) {
233 return Err(sess
.emit_err(errors
::CanonicalizePath { path: crate_dir, err }
));
237 let mut source_directories_already_tried
= FxHashSet
::default();
240 // Generate a session directory of the form:
242 // {incr-comp-dir}/{crate-name-and-disambiguator}/s-{timestamp}-{random}-working
243 let session_dir
= generate_session_dir_path(&crate_dir
);
244 debug
!("session-dir: {}", session_dir
.display());
246 // Lock the new session directory. If this fails, return an
247 // error without retrying
248 let (directory_lock
, lock_file_path
) = lock_directory(sess
, &session_dir
)?
;
250 // Now that we have the lock, we can actually create the session
252 create_dir(sess
, &session_dir
, "session")?
;
254 // Find a suitable source directory to copy from. Ignore those that we
255 // have already tried before.
256 let source_directory
= find_source_directory(&crate_dir
, &source_directories_already_tried
);
258 let Some(source_directory
) = source_directory
else {
259 // There's nowhere to copy from, we're done
261 "no source directory found. Continuing with empty session \
265 sess
.init_incr_comp_session(session_dir
, directory_lock
);
269 debug
!("attempting to copy data from source: {}", source_directory
.display());
271 // Try copying over all files from the source directory
272 if let Ok(allows_links
) = copy_files(sess
, &session_dir
, &source_directory
) {
273 debug
!("successfully copied data from: {}", source_directory
.display());
276 sess
.emit_warning(errors
::HardLinkFailed { path: &session_dir }
);
279 sess
.init_incr_comp_session(session_dir
, directory_lock
);
282 debug
!("copying failed - trying next directory");
284 // Something went wrong while trying to copy/link files from the
285 // source directory. Try again with a different one.
286 source_directories_already_tried
.insert(source_directory
);
288 // Try to remove the session directory we just allocated. We don't
289 // know if there's any garbage in it from the failed copy action.
290 if let Err(err
) = safe_remove_dir_all(&session_dir
) {
291 sess
.emit_warning(errors
::DeletePartial { path: &session_dir, err }
);
294 delete_session_dir_lock_file(sess
, &lock_file_path
);
295 drop(directory_lock
);
300 /// This function finalizes and thus 'publishes' the session directory by
301 /// renaming it to `s-{timestamp}-{svh}` and releasing the file lock.
302 /// If there have been compilation errors, however, this function will just
303 /// delete the presumably invalid session directory.
304 pub fn finalize_session_directory(sess
: &Session
, svh
: Option
<Svh
>) {
305 if sess
.opts
.incremental
.is_none() {
308 // The svh is always produced when incr. comp. is enabled.
309 let svh
= svh
.unwrap();
311 let _timer
= sess
.timer("incr_comp_finalize_session_directory");
313 let incr_comp_session_dir
: PathBuf
= sess
.incr_comp_session_dir().clone();
315 if let Some(_
) = sess
.has_errors_or_span_delayed_bugs() {
316 // If there have been any errors during compilation, we don't want to
317 // publish this session directory. Rather, we'll just delete it.
320 "finalize_session_directory() - invalidating session directory: {}",
321 incr_comp_session_dir
.display()
324 if let Err(err
) = safe_remove_dir_all(&*incr_comp_session_dir
) {
325 sess
.emit_warning(errors
::DeleteFull { path: &incr_comp_session_dir, err }
);
328 let lock_file_path
= lock_file_path(&*incr_comp_session_dir
);
329 delete_session_dir_lock_file(sess
, &lock_file_path
);
330 sess
.mark_incr_comp_session_as_invalid();
333 debug
!("finalize_session_directory() - session directory: {}", incr_comp_session_dir
.display());
335 let old_sub_dir_name
= incr_comp_session_dir
.file_name().unwrap().to_string_lossy();
336 assert_no_characters_lost(&old_sub_dir_name
);
338 // Keep the 's-{timestamp}-{random-number}' prefix, but replace the
339 // '-working' part with the SVH of the crate
340 let dash_indices
: Vec
<_
> = old_sub_dir_name
.match_indices('
-'
).map(|(idx
, _
)| idx
).collect();
341 if dash_indices
.len() != 3 {
343 "Encountered incremental compilation session directory with \
345 incr_comp_session_dir
.display()
349 // State: "s-{timestamp}-{random-number}-"
350 let mut new_sub_dir_name
= String
::from(&old_sub_dir_name
[..=dash_indices
[2]]);
353 base_n
::push_str(svh
.as_u128(), INT_ENCODE_BASE
, &mut new_sub_dir_name
);
355 // Create the full path
356 let new_path
= incr_comp_session_dir
.parent().unwrap().join(new_sub_dir_name
);
357 debug
!("finalize_session_directory() - new path: {}", new_path
.display());
359 match rename_path_with_retry(&*incr_comp_session_dir
, &new_path
, 3) {
361 debug
!("finalize_session_directory() - directory renamed successfully");
363 // This unlocks the directory
364 sess
.finalize_incr_comp_session(new_path
);
367 // Warn about the error. However, no need to abort compilation now.
368 sess
.emit_warning(errors
::Finalize { path: &incr_comp_session_dir, err: e }
);
370 debug
!("finalize_session_directory() - error, marking as invalid");
371 // Drop the file lock, so we can garage collect
372 sess
.mark_incr_comp_session_as_invalid();
376 let _
= garbage_collect_session_directories(sess
);
379 pub(crate) fn delete_all_session_dir_contents(sess
: &Session
) -> io
::Result
<()> {
380 let sess_dir_iterator
= sess
.incr_comp_session_dir().read_dir()?
;
381 for entry
in sess_dir_iterator
{
383 safe_remove_file(&entry
.path())?
388 fn copy_files(sess
: &Session
, target_dir
: &Path
, source_dir
: &Path
) -> Result
<bool
, ()> {
389 // We acquire a shared lock on the lock file of the directory, so that
390 // nobody deletes it out from under us while we are reading from it.
391 let lock_file_path
= lock_file_path(source_dir
);
394 let Ok(_lock
) = flock
::Lock
::new(
396 false, // don't wait,
397 false, // don't create
400 // Could not acquire the lock, don't try to copy from here
404 let Ok(source_dir_iterator
) = source_dir
.read_dir() else {
408 let mut files_linked
= 0;
409 let mut files_copied
= 0;
411 for entry
in source_dir_iterator
{
414 let file_name
= entry
.file_name();
416 let target_file_path
= target_dir
.join(file_name
);
417 let source_path
= entry
.path();
419 debug
!("copying into session dir: {}", source_path
.display());
420 match link_or_copy(source_path
, target_file_path
) {
421 Ok(LinkOrCopy
::Link
) => files_linked
+= 1,
422 Ok(LinkOrCopy
::Copy
) => files_copied
+= 1,
423 Err(_
) => return Err(()),
426 Err(_
) => return Err(()),
430 if sess
.opts
.unstable_opts
.incremental_info
{
432 "[incremental] session directory: \
433 {files_linked} files hard-linked"
436 "[incremental] session directory: \
437 {files_copied} files copied"
441 Ok(files_linked
> 0 || files_copied
== 0)
444 /// Generates unique directory path of the form:
445 /// {crate_dir}/s-{timestamp}-{random-number}-working
446 fn generate_session_dir_path(crate_dir
: &Path
) -> PathBuf
{
447 let timestamp
= timestamp_to_string(SystemTime
::now());
448 debug
!("generate_session_dir_path: timestamp = {}", timestamp
);
449 let random_number
= thread_rng().next_u32();
450 debug
!("generate_session_dir_path: random_number = {}", random_number
);
452 let directory_name
= format
!(
455 base_n
::encode(random_number
as u128
, INT_ENCODE_BASE
)
457 debug
!("generate_session_dir_path: directory_name = {}", directory_name
);
458 let directory_path
= crate_dir
.join(directory_name
);
459 debug
!("generate_session_dir_path: directory_path = {}", directory_path
.display());
463 fn create_dir(sess
: &Session
, path
: &Path
, dir_tag
: &str) -> Result
<(), ErrorGuaranteed
> {
464 match std_fs
::create_dir_all(path
) {
466 debug
!("{} directory created successfully", dir_tag
);
469 Err(err
) => Err(sess
.emit_err(errors
::CreateIncrCompDir { tag: dir_tag, path, err }
)),
473 /// Allocate the lock-file and lock it.
477 ) -> Result
<(flock
::Lock
, PathBuf
), ErrorGuaranteed
> {
478 let lock_file_path
= lock_file_path(session_dir
);
479 debug
!("lock_directory() - lock_file: {}", lock_file_path
.display());
481 match flock
::Lock
::new(
484 true, // create the lock file
487 // the lock should be exclusive
488 Ok(lock
) => Ok((lock
, lock_file_path
)),
490 let is_unsupported_lock
= flock
::Lock
::error_unsupported(&lock_err
).then_some(());
491 Err(sess
.emit_err(errors
::CreateLock
{
495 is_cargo
: std
::env
::var_os("CARGO").map(|_
| ()),
501 fn delete_session_dir_lock_file(sess
: &Session
, lock_file_path
: &Path
) {
502 if let Err(err
) = safe_remove_file(lock_file_path
) {
503 sess
.emit_warning(errors
::DeleteLock { path: lock_file_path, err }
);
507 /// Finds the most recent published session directory that is not in the
509 fn find_source_directory(
511 source_directories_already_tried
: &FxHashSet
<PathBuf
>,
512 ) -> Option
<PathBuf
> {
516 .filter_map(|e
| e
.ok().map(|e
| e
.path()));
518 find_source_directory_in_iter(iter
, source_directories_already_tried
)
521 fn find_source_directory_in_iter
<I
>(
523 source_directories_already_tried
: &FxHashSet
<PathBuf
>,
526 I
: Iterator
<Item
= PathBuf
>,
528 let mut best_candidate
= (UNIX_EPOCH
, None
);
530 for session_dir
in iter
{
531 debug
!("find_source_directory_in_iter - inspecting `{}`", session_dir
.display());
533 let directory_name
= session_dir
.file_name().unwrap().to_string_lossy();
534 assert_no_characters_lost(&directory_name
);
536 if source_directories_already_tried
.contains(&session_dir
)
537 || !is_session_directory(&directory_name
)
538 || !is_finalized(&directory_name
)
540 debug
!("find_source_directory_in_iter - ignoring");
544 let timestamp
= match extract_timestamp_from_session_dir(&directory_name
) {
545 Ok(timestamp
) => timestamp
,
547 debug
!("unexpected incr-comp session dir: {}: {}", session_dir
.display(), e
);
552 if timestamp
> best_candidate
.0 {
553 best_candidate
= (timestamp
, Some(session_dir
.clone()));
560 fn is_finalized(directory_name
: &str) -> bool
{
561 !directory_name
.ends_with("-working")
564 fn is_session_directory(directory_name
: &str) -> bool
{
565 directory_name
.starts_with("s-") && !directory_name
.ends_with(LOCK_FILE_EXT
)
568 fn is_session_directory_lock_file(file_name
: &str) -> bool
{
569 file_name
.starts_with("s-") && file_name
.ends_with(LOCK_FILE_EXT
)
572 fn extract_timestamp_from_session_dir(directory_name
: &str) -> Result
<SystemTime
, &'
static str> {
573 if !is_session_directory(directory_name
) {
574 return Err("not a directory");
577 let dash_indices
: Vec
<_
> = directory_name
.match_indices('
-'
).map(|(idx
, _
)| idx
).collect();
578 if dash_indices
.len() != 3 {
579 return Err("not three dashes in name");
582 string_to_timestamp(&directory_name
[dash_indices
[0] + 1..dash_indices
[1]])
585 fn timestamp_to_string(timestamp
: SystemTime
) -> String
{
586 let duration
= timestamp
.duration_since(UNIX_EPOCH
).unwrap();
587 let micros
= duration
.as_secs() * 1_000_000 + (duration
.subsec_nanos() as u64) / 1000;
588 base_n
::encode(micros
as u128
, INT_ENCODE_BASE
)
591 fn string_to_timestamp(s
: &str) -> Result
<SystemTime
, &'
static str> {
592 let micros_since_unix_epoch
= u64::from_str_radix(s
, INT_ENCODE_BASE
as u32);
594 if micros_since_unix_epoch
.is_err() {
595 return Err("timestamp not an int");
598 let micros_since_unix_epoch
= micros_since_unix_epoch
.unwrap();
600 let duration
= Duration
::new(
601 micros_since_unix_epoch
/ 1_000_000,
602 1000 * (micros_since_unix_epoch
% 1_000_000) as u32,
604 Ok(UNIX_EPOCH
+ duration
)
607 fn crate_path(sess
: &Session
, crate_name
: Symbol
, stable_crate_id
: StableCrateId
) -> PathBuf
{
608 let incr_dir
= sess
.opts
.incremental
.as_ref().unwrap().clone();
610 let stable_crate_id
= base_n
::encode(stable_crate_id
.as_u64() as u128
, INT_ENCODE_BASE
);
612 let crate_name
= format
!("{crate_name}-{stable_crate_id}");
613 incr_dir
.join(crate_name
)
616 fn assert_no_characters_lost(s
: &str) {
617 if s
.contains('
\u{FFFD}'
) {
618 bug
!("Could not losslessly convert '{}'.", s
)
622 fn is_old_enough_to_be_collected(timestamp
: SystemTime
) -> bool
{
623 timestamp
< SystemTime
::now() - Duration
::from_secs(10)
626 /// Runs garbage collection for the current session.
627 pub(crate) fn garbage_collect_session_directories(sess
: &Session
) -> io
::Result
<()> {
628 debug
!("garbage_collect_session_directories() - begin");
630 let session_directory
= sess
.incr_comp_session_dir();
632 "garbage_collect_session_directories() - session directory: {}",
633 session_directory
.display()
636 let crate_directory
= session_directory
.parent().unwrap();
638 "garbage_collect_session_directories() - crate directory: {}",
639 crate_directory
.display()
642 // First do a pass over the crate directory, collecting lock files and
643 // session directories
644 let mut session_directories
= FxIndexSet
::default();
645 let mut lock_files
= UnordSet
::default();
647 for dir_entry
in crate_directory
.read_dir()?
{
648 let Ok(dir_entry
) = dir_entry
else {
653 let entry_name
= dir_entry
.file_name();
654 let entry_name
= entry_name
.to_string_lossy();
656 if is_session_directory_lock_file(&entry_name
) {
657 assert_no_characters_lost(&entry_name
);
658 lock_files
.insert(entry_name
.into_owned());
659 } else if is_session_directory(&entry_name
) {
660 assert_no_characters_lost(&entry_name
);
661 session_directories
.insert(entry_name
.into_owned());
663 // This is something we don't know, leave it alone
666 session_directories
.sort();
668 // Now map from lock files to session directories
669 let lock_file_to_session_dir
: UnordMap
<String
, Option
<String
>> = lock_files
671 .map(|lock_file_name
| {
672 assert
!(lock_file_name
.ends_with(LOCK_FILE_EXT
));
673 let dir_prefix_end
= lock_file_name
.len() - LOCK_FILE_EXT
.len();
675 let dir_prefix
= &lock_file_name
[0..dir_prefix_end
];
676 session_directories
.iter().find(|dir_name
| dir_name
.starts_with(dir_prefix
))
678 (lock_file_name
, session_dir
.map(String
::clone
))
682 // Delete all lock files, that don't have an associated directory. They must
683 // be some kind of leftover
684 for (lock_file_name
, directory_name
) in
685 lock_file_to_session_dir
.items().into_sorted_stable_ord()
687 if directory_name
.is_none() {
688 let Ok(timestamp
) = extract_timestamp_from_session_dir(lock_file_name
) else {
690 "found lock-file with malformed timestamp: {}",
691 crate_directory
.join(&lock_file_name
).display()
697 let lock_file_path
= crate_directory
.join(&*lock_file_name
);
699 if is_old_enough_to_be_collected(timestamp
) {
701 "garbage_collect_session_directories() - deleting \
702 garbage lock file: {}",
703 lock_file_path
.display()
705 delete_session_dir_lock_file(sess
, &lock_file_path
);
708 "garbage_collect_session_directories() - lock file with \
709 no session dir not old enough to be collected: {}",
710 lock_file_path
.display()
716 // Filter out `None` directories
717 let lock_file_to_session_dir
: UnordMap
<String
, String
> = lock_file_to_session_dir
719 .filter_map(|(lock_file_name
, directory_name
)| directory_name
.map(|n
| (lock_file_name
, n
)))
722 // Delete all session directories that don't have a lock file.
723 for directory_name
in session_directories
{
724 if !lock_file_to_session_dir
.items().any(|(_
, dir
)| *dir
== directory_name
) {
725 let path
= crate_directory
.join(directory_name
);
726 if let Err(err
) = safe_remove_dir_all(&path
) {
727 sess
.emit_warning(errors
::InvalidGcFailed { path: &path, err }
);
732 // Now garbage collect the valid session directories.
733 let deletion_candidates
=
734 lock_file_to_session_dir
.items().filter_map(|(lock_file_name
, directory_name
)| {
735 debug
!("garbage_collect_session_directories() - inspecting: {}", directory_name
);
737 let Ok(timestamp
) = extract_timestamp_from_session_dir(directory_name
) else {
739 "found session-dir with malformed timestamp: {}",
740 crate_directory
.join(directory_name
).display()
746 if is_finalized(directory_name
) {
747 let lock_file_path
= crate_directory
.join(lock_file_name
);
748 match flock
::Lock
::new(
751 false, // don't create the lock-file
754 // get an exclusive lock
757 "garbage_collect_session_directories() - \
758 successfully acquired lock"
761 "garbage_collect_session_directories() - adding \
762 deletion candidate: {}",
766 // Note that we are holding on to the lock
768 (timestamp
, crate_directory
.join(directory_name
)),
774 "garbage_collect_session_directories() - \
775 not collecting, still in use"
779 } else if is_old_enough_to_be_collected(timestamp
) {
780 // When cleaning out "-working" session directories, i.e.
781 // session directories that might still be in use by another
782 // compiler instance, we only look a directories that are
783 // at least ten seconds old. This is supposed to reduce the
784 // chance of deleting a directory in the time window where
785 // the process has allocated the directory but has not yet
786 // acquired the file-lock on it.
788 // Try to acquire the directory lock. If we can't, it
789 // means that the owning process is still alive and we
790 // leave this directory alone.
791 let lock_file_path
= crate_directory
.join(lock_file_name
);
792 match flock
::Lock
::new(
795 false, // don't create the lock-file
798 // get an exclusive lock
801 "garbage_collect_session_directories() - \
802 successfully acquired lock"
805 delete_old(sess
, &crate_directory
.join(directory_name
));
807 // Let's make it explicit that the file lock is released at this point,
808 // or rather, that we held on to it until here
813 "garbage_collect_session_directories() - \
814 not collecting, still in use"
820 "garbage_collect_session_directories() - not finalized, not \
826 let deletion_candidates
= deletion_candidates
.into();
828 // Delete all but the most recent of the candidates
829 all_except_most_recent(deletion_candidates
).into_items().all(|(path
, lock
)| {
830 debug
!("garbage_collect_session_directories() - deleting `{}`", path
.display());
832 if let Err(err
) = safe_remove_dir_all(&path
) {
833 sess
.emit_warning(errors
::FinalizedGcFailed { path: &path, err }
);
835 delete_session_dir_lock_file(sess
, &lock_file_path(&path
));
838 // Let's make it explicit that the file lock is released at this point,
839 // or rather, that we held on to it until here
847 fn delete_old(sess
: &Session
, path
: &Path
) {
848 debug
!("garbage_collect_session_directories() - deleting `{}`", path
.display());
850 if let Err(err
) = safe_remove_dir_all(path
) {
851 sess
.emit_warning(errors
::SessionGcFailed { path: path, err }
);
853 delete_session_dir_lock_file(sess
, &lock_file_path(path
));
857 fn all_except_most_recent(
858 deletion_candidates
: UnordMap
<(SystemTime
, PathBuf
), Option
<flock
::Lock
>>,
859 ) -> UnordMap
<PathBuf
, Option
<flock
::Lock
>> {
860 let most_recent
= deletion_candidates
.items().map(|(&(timestamp
, _
), _
)| timestamp
).max();
862 if let Some(most_recent
) = most_recent
{
865 .filter(|&((timestamp
, _
), _
)| timestamp
!= most_recent
)
866 .map(|((_
, path
), lock
)| (path
, lock
))
873 /// Since paths of artifacts within session directories can get quite long, we
874 /// need to support deleting files with very long paths. The regular
875 /// WinApi functions only support paths up to 260 characters, however. In order
876 /// to circumvent this limitation, we canonicalize the path of the directory
877 /// before passing it to std::fs::remove_dir_all(). This will convert the path
878 /// into the '\\?\' format, which supports much longer paths.
879 fn safe_remove_dir_all(p
: &Path
) -> io
::Result
<()> {
880 let canonicalized
= match try_canonicalize(p
) {
881 Ok(canonicalized
) => canonicalized
,
882 Err(err
) if err
.kind() == io
::ErrorKind
::NotFound
=> return Ok(()),
883 Err(err
) => return Err(err
),
886 std_fs
::remove_dir_all(canonicalized
)
889 fn safe_remove_file(p
: &Path
) -> io
::Result
<()> {
890 let canonicalized
= match try_canonicalize(p
) {
891 Ok(canonicalized
) => canonicalized
,
892 Err(err
) if err
.kind() == io
::ErrorKind
::NotFound
=> return Ok(()),
893 Err(err
) => return Err(err
),
896 match std_fs
::remove_file(canonicalized
) {
897 Err(err
) if err
.kind() == io
::ErrorKind
::NotFound
=> Ok(()),
902 // On Windows the compiler would sometimes fail to rename the session directory because
903 // the OS thought something was still being accessed in it. So we retry a few times to give
904 // the OS time to catch up.
905 // See https://github.com/rust-lang/rust/issues/86929.
906 fn rename_path_with_retry(from
: &Path
, to
: &Path
, mut retries_left
: usize) -> std
::io
::Result
<()> {
908 match std_fs
::rename(from
, to
) {
909 Ok(()) => return Ok(()),
911 if retries_left
> 0 && e
.kind() == ErrorKind
::PermissionDenied
{
912 // Try again after a short waiting period.
913 std
::thread
::sleep(Duration
::from_millis(50));