src/doc/rustc-dev-guide/src/parallel-rustc.md

   1 # Parallel Compilation
   2
   3 Most of the compiler is not parallel. This represents an opportunity for
   4 improving compiler performance.
   5
   6 As of <!-- date: 2021-07 --> July 2021, work on explicitly parallelizing the
   7 compiler has stalled. There is a lot of design and correctness work that needs
   8 to be done.
   9
  10 One can try out the current parallel compiler work by enabling it in the
  11 `config.toml`.
  12
  13 There are a few basic ideas in this effort:
  14
  15 - There are a lot of loops in the compiler that just iterate over all items in
  16   a crate. These can possibly be parallelized.
  17 - We can use (a custom fork of) [`rayon`] to run tasks in parallel. The custom
  18   fork allows the execution of DAGs of tasks, not just trees.
  19 - There are currently a lot of global data structures that need to be made
  20   thread-safe. A key strategy here has been converting interior-mutable
  21   data-structures (e.g. `Cell`) into their thread-safe siblings (e.g. `Mutex`).
  22
  23 [`rayon`]: https://crates.io/crates/rayon
  24
  25 As of <!-- date: 2021-02 --> February 2021, much of this effort is on hold due
  26 to lack of manpower. We have a working prototype with promising performance
  27 gains in many cases. However, there are two blockers:
  28
  29 - It's not clear what invariants need to be upheld that might not hold in the
  30   face of concurrency. An auditing effort was underway, but seems to have
  31   stalled at some point.
  32
  33 - There is a lot of lock contention, which actually degrades performance as the
  34   number of threads increases beyond 4.
  35
  36 Here are some resources that can be used to learn more (note that some of them
  37 are a bit out of date):
  38
  39 - [This IRLO thread by Zoxc, one of the pioneers of the effort][irlo0]
  40 - [This list of interior mutability in the compiler by nikomatsakis][imlist]
  41 - [This IRLO thread by alexchricton about performance][irlo1]
  42 - [This tracking issue][tracking]
  43
  44 [irlo0]: https://internals.rust-lang.org/t/parallelizing-rustc-using-rayon/6606
  45 [imlist]: https://github.com/nikomatsakis/rustc-parallelization/blob/master/interior-mutability-list.md
  46 [irlo1]: https://internals.rust-lang.org/t/help-test-parallel-rustc/11503
  47 [tracking]: https://github.com/rust-lang/rust/issues/48685