src/doc/rustc-dev-guide/src/compiler-src.md

   1 # High-level overview of the compiler source
   2
   3 <!-- toc -->
   4
   5 Now that we have [seen what the compiler does](./overview.md), let's take a
   6 look at the structure of the [`rust-lang/rust`] repository, where the rustc
   7 source code lives.
   8
   9 [`rust-lang/rust`]: https://github.com/rust-lang/rust
  10
  11 > You may find it helpful to read the ["Overview of the compiler"](./overview.md)
  12 > chapter, which introduces how the compiler works, before this one.
  13
  14 ## Workspace structure
  15
  16 The `rust-lang/rust` repository consists of a single large cargo workspace
  17 containing the compiler, the standard libraries (`core`, `alloc`, `std`,
  18 `proc_macro`, etc), and `rustdoc`, along with the build system and a bunch of
  19 tools and submodules for building a full Rust distribution.
  20
  21 The repository consists of three main directories:
  22
  23 - `compiler/` contains the source code for `rustc`. It consists of many crates
  24   that together make up the compiler.
  25
  26 - `library/` contains the standard libraries (`core`, `alloc`, `std`,
  27   `proc_macro`, `test`), as well as the Rust runtime (`backtrace`, `rtstartup`,
  28   `lang_start`).
  29
  30 - `tests/` contains the compiler tests.
  31
  32 - `src/` contains the source code for rustdoc, clippy, cargo, the build system,
  33   language docs, etc.
  34
  35 ## Compiler
  36
  37 The compiler is implemented in the various `compiler/` crates.
  38 The `compiler/` crates all have names starting with `rustc_*`. These are a
  39 collection of around 50 interdependent crates ranging in size from tiny to
  40 huge. There is also the `rustc` crate which is the actual binary (i.e. the
  41 `main` function); it doesn't actually do anything besides calling the
  42 `rustc_driver` crate, which drives the various parts of compilation in other
  43 crates.
  44
  45 The dependency structure of these crates is complex, but roughly it is
  46 something like this:
  47
  48 - `rustc` (the binary) calls [`rustc_driver::main`][main].
  49     - [`rustc_driver`] depends on a lot of other crates, but the main one is
  50       [`rustc_interface`].
  51         - [`rustc_interface`] depends on most of the other compiler crates. It
  52           is a fairly generic interface for driving the whole compilation.
  53             - Most of the other `rustc_*` crates depend on [`rustc_middle`],
  54               which defines a lot of central data structures in the compiler.
  55                 - [`rustc_middle`] and most of the other crates depend on a
  56                   handful of crates representing the early parts of the
  57                   compiler (e.g. the parser), fundamental data structures (e.g.
  58                   [`Span`]), or error reporting: [`rustc_data_structures`],
  59                   [`rustc_span`], [`rustc_errors`], etc.
  60
  61 [main]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_driver/fn.main.html
  62 [`rustc_driver`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_driver/index.html
  63 [`rustc_interface`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_interface/index.html
  64 [`rustc_middle`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_middle/index.html
  65 [`rustc_data_structures`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_data_structures/index.html
  66 [`rustc_span`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_span/index.html
  67 [`Span`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_span/struct.Span.html
  68 [`rustc_errors`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_errors/index.html
  69
  70 You can see the exact dependencies by reading the `Cargo.toml` for the various
  71 crates, just like a normal Rust crate.
  72
  73 One final thing: [`src/llvm-project`] is a submodule for our fork of LLVM.
  74 During bootstrapping, LLVM is built and the [`compiler/rustc_llvm`] crate
  75 contains Rust wrappers around LLVM (which is written in C++), so that the
  76 compiler can interface with it.
  77
  78 Most of this book is about the compiler, so we won't have any further
  79 explanation of these crates here.
  80
  81 [`src/llvm-project`]: https://github.com/rust-lang/rust/tree/master/src/
  82 [`compiler/rustc_llvm`]: https://github.com/rust-lang/rust/tree/master/compiler/rustc_llvm
  83
  84 ### Big picture
  85
  86 The dependency structure is influenced by two main factors:
  87
  88 1. Organization. The compiler is a _huge_ codebase; it would be an impossibly
  89    large crate. In part, the dependency structure reflects the code structure
  90    of the compiler.
  91 2. Compile time. By breaking the compiler into multiple crates, we can take
  92    better advantage of incremental/parallel compilation using cargo. In
  93    particular, we try to have as few dependencies between crates as possible so
  94    that we don't have to rebuild as many crates if you change one.
  95
  96 At the very bottom of the dependency tree are a handful of crates that are used
  97 by the whole compiler (e.g. [`rustc_span`]). The very early parts of the
  98 compilation process (e.g. parsing and the AST) depend on only these.
  99
 100 After the AST is constructed and other early analysis is done, the compiler's [query system][query]
 101 gets set up. The query system is set up in a clever way using function
 102 pointers. This allows us to break dependencies between crates, allowing more
 103 parallel compilation.
 104 The query system is defined in [`rustc_middle`], so nearly all
 105 subsequent parts of the compiler depend on this crate. It is a really large
 106 crate, leading to long compile times. Some efforts have been made to move stuff
 107 out of it with limited success. Another unfortunate side effect is that sometimes
 108 related functionality gets scattered across different crates. For example,
 109 linting functionality is scattered across earlier parts of the crate,
 110 [`rustc_lint`], [`rustc_middle`], and other places.
 111
 112 [`rustc_lint`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_lint/index.html
 113
 114 Ideally there would be fewer, more
 115 cohesive crates, with incremental and parallel compilation making sure compile
 116 times stay reasonable. However, our incremental and parallel compilation haven't
 117 gotten good enough for that yet, so breaking things into separate crates has
 118 been our solution so far.
 119
 120 At the top of the dependency tree are the [`rustc_interface`] and
 121 [`rustc_driver`] crates. [`rustc_interface`] is an unstable wrapper around the
 122 query system that helps to drive the various stages of compilation. Other
 123 consumers of the compiler may use this interface in different ways (e.g.
 124 rustdoc or maybe eventually rust-analyzer). The [`rustc_driver`] crate first
 125 parses command line arguments and then uses [`rustc_interface`] to drive the
 126 compilation to completion.
 127
 128 [query]: ./query.md
 129
 130 [orgch]: ./overview.md
 131
 132 ## rustdoc
 133
 134 The bulk of `rustdoc` is in [`librustdoc`]. However, the `rustdoc` binary
 135 itself is [`src/tools/rustdoc`], which does nothing except call [`rustdoc::main`].
 136
 137 There is also javascript and CSS for the rustdocs in [`src/tools/rustdoc-js`]
 138 and [`src/tools/rustdoc-themes`].
 139
 140 You can read more about rustdoc in [this chapter][rustdocch].
 141
 142 [`librustdoc`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustdoc/index.html
 143 [`rustdoc::main`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustdoc/fn.main.html
 144 [`src/tools/rustdoc`]:  https://github.com/rust-lang/rust/tree/master/src/tools/rustdoc
 145 [`src/tools/rustdoc-js`]: https://github.com/rust-lang/rust/tree/master/src/tools/rustdoc-js
 146 [`src/tools/rustdoc-themes`]: https://github.com/rust-lang/rust/tree/master/src/tools/rustdoc-themes
 147
 148 [rustdocch]: ./rustdoc.md
 149
 150 ## Tests
 151
 152 The test suite for all of the above is in [`tests/`]. You can read more
 153 about the test suite [in this chapter][testsch].
 154
 155 The test harness itself is in [`src/tools/compiletest`].
 156
 157 [testsch]: ./tests/intro.md
 158
 159 [`tests/`]: https://github.com/rust-lang/rust/tree/master/tests
 160 [`src/tools/compiletest`]: https://github.com/rust-lang/rust/tree/master/src/tools/compiletest
 161
 162 ## Build System
 163
 164 There are a number of tools in the repository just for building the compiler,
 165 standard library, rustdoc, etc, along with testing, building a full Rust
 166 distribution, etc.
 167
 168 One of the primary tools is [`src/bootstrap`]. You can read more about
 169 bootstrapping [in this chapter][bootstch]. The process may also use other tools
 170 from `src/tools/`, such as [`tidy`] or [`compiletest`].
 171
 172 [`src/bootstrap`]: https://github.com/rust-lang/rust/tree/master/src/bootstrap
 173 [`tidy`]: https://github.com/rust-lang/rust/tree/master/src/tools/tidy
 174 [`compiletest`]: https://github.com/rust-lang/rust/tree/master/src/tools/compiletest
 175
 176 [bootstch]: ./building/bootstrapping.md
 177
 178 ## Standard library
 179
 180 The standard library crates are all in `library/`. They have intuitive names
 181 like `std`, `core`, `alloc`, etc.  There is also `proc_macro`, `test`, and
 182 other runtime libraries.
 183
 184 This code is fairly similar to most other Rust crates except that it must be
 185 built in a special way because it can use unstable features.
 186
 187 ## Other
 188
 189 There are a lot of other things in the `rust-lang/rust` repo that are related
 190 to building a full Rust distribution. Most of the time you don't need to worry
 191 about them.
 192
 193 These include:
 194 - [`src/ci`]: The CI configuration. This actually quite extensive because we
 195   run a lot of tests on a lot of platforms.
 196 - [`src/doc`]: Various documentation, including submodules for a few books.
 197 - [`src/etc`]: Miscellaneous utilities.
 198 - And more...
 199
 200 [`src/ci`]: https://github.com/rust-lang/rust/tree/master/src/ci
 201 [`src/doc`]: https://github.com/rust-lang/rust/tree/master/src/doc
 202 [`src/etc`]: https://github.com/rust-lang/rust/tree/master/src/etc