src/doc/rustc-dev-guide/src/compiler-src.md

   1 # High-level overview of the compiler source
   2
   3 <!-- toc -->
   4
   5 Now that we have [seen what the compiler does](./overview.md), let's take a
   6 look at the structure of the contents of the rust-lang/rust repo.
   7
   8 ## Workspace structure
   9
  10 The `rust-lang/rust` repository consists of a single large cargo workspace
  11 containing the compiler, the standard libraries (`core`, `alloc`, `std`,
  12 `proc_macro`, etc), and `rustdoc`, along with the build system and a bunch of
  13 tools and submodules for building a full Rust distribution.
  14
  15 The repository consists of three main directories:
  16
  17 - `compiler/` contains the source code for `rustc`. It consists of many crates
  18   that together make up the compiler.
  19
  20 - `library/` contains the standard libraries (`core`, `alloc`, `std`,
  21   `proc_macro`, `test`), as well as the Rust runtime (`backtrace`, `rtstartup`,
  22   `lang_start`).
  23
  24 - `src/` contains the source code for rustdoc, clippy, cargo, the build system,
  25   language docs, etc.
  26
  27 ## Standard library
  28
  29 The standard library crates are all in `library/`. They have intuitive names
  30 like `std`, `core`, `alloc`, etc.  There is also `proc_macro`, `test`, and
  31 other runtime libraries.
  32
  33 This code is fairly similar to most other Rust crates except that it must be
  34 built in a special way because it can use unstable features.
  35
  36 ## Compiler
  37
  38 > You may find it helpful to read [The Overview Chapter](./overview.md) first,
  39 > which gives an overview of how the compiler works. The crates mentioned in
  40 > this section implement the compiler, and are underneath `compiler/`
  41
  42 The `compiler/` crates all have names starting with `rustc_*`. These are a
  43 collection of around 50 interdependent crates ranging in size from tiny to
  44 huge. There is also the `rustc` crate which is the actual binary (i.e. the
  45 `main` function); it doesn't actually do anything besides calling the
  46 `rustc_driver` crate, which drives the various parts of compilation in other
  47 crates.
  48
  49 The dependency structure of these crates is complex, but roughly it is
  50 something like this:
  51
  52 - `rustc` (the binary) calls [`rustc_driver::main`][main].
  53     - [`rustc_driver`] depends on a lot of other crates, but the main one is
  54       [`rustc_interface`].
  55         - [`rustc_interface`] depends on most of the other compiler crates. It
  56           is a fairly generic interface for driving the whole compilation.
  57             - Most of the other `rustc_*` crates depend on [`rustc_middle`],
  58               which defines a lot of central data structures in the compiler.
  59                 - [`rustc_middle`] and most of the other crates depend on a
  60                   handful of crates representing the early parts of the
  61                   compiler (e.g. the parser), fundamental data structures (e.g.
  62                   [`Span`]), or error reporting: [`rustc_data_structures`],
  63                   [`rustc_span`], [`rustc_errors`], etc.
  64
  65 [main]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_driver/fn.main.html
  66 [`rustc_driver`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_driver/index.html
  67 [`rustc_interface`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_interface/index.html
  68 [`rustc_middle`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_middle/index.html
  69 [`rustc_data_structures`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_data_structures/index.html
  70 [`rustc_span`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_span/index.html
  71 [`Span`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_span/struct.Span.html
  72 [`rustc_errors`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_errors/index.html
  73
  74 You can see the exact dependencies by reading the `Cargo.toml` for the various
  75 crates, just like a normal Rust crate.
  76
  77 One final thing: [`src/llvm-project`] is a submodule for our fork of LLVM.
  78 During bootstrapping, LLVM is built and the [`compiler/rustc_llvm`] crate
  79 contains rust wrappers around LLVM (which is written in C++), so that the
  80 compiler can interface with it.
  81
  82 Most of this book is about the compiler, so we won't have any further
  83 explanation of these crates here.
  84
  85 [`src/llvm-project`]: https://github.com/rust-lang/rust/tree/master/src/
  86 [`compiler/rustc_llvm`]: https://github.com/rust-lang/rust/tree/master/compiler/rustc_llvm
  87
  88 ### Big picture
  89
  90 The dependency structure is influenced strongly by two main factors:
  91
  92 1. Organization. The compiler is a _huge_ codebase; it would be an impossibly
  93    large crate. In part, the dependency structure reflects the code structure
  94    of the compiler.
  95 2. Compile time. By breaking the compiler into multiple crates, we can take
  96    better advantage of incremental/parallel compilation using cargo. In
  97    particular, we try to have as few dependencies between crates as possible so
  98    that we don't have to rebuild as many crates if you change one.
  99
 100 At the very bottom of the dependency tree are a handful of crates that are used
 101 by the whole compiler (e.g. [`rustc_span`]). The very early parts of the
 102 compilation process (e.g. parsing and the AST) depend on only these.
 103
 104 Pretty soon after the AST is constructed, the compiler's [query system][query]
 105 gets set up.  The query system is set up in a clever way using function
 106 pointers. This allows us to break dependencies between crates, allowing more
 107 parallel compilation.
 108
 109 However, since the query system is defined in [`rustc_middle`], nearly all
 110 subsequent parts of the compiler depend on this crate. It is a really large
 111 crate, leading to long compile times. Some efforts have been made to move stuff
 112 out of it with limited success. Another unfortunate side effect is that sometimes
 113 related functionality gets scattered across different crates. For example,
 114 linting functionality is scattered across earlier parts of the crate,
 115 [`rustc_lint`], [`rustc_middle`], and other places.
 116
 117 [`rustc_lint`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_lint/index.html
 118
 119 More generally, in an ideal world, it seems like there would be fewer, more
 120 cohesive crates, with incremental and parallel compilation making sure compile
 121 times stay reasonable. However, our incremental and parallel compilation haven't
 122 gotten good enough for that yet, so breaking things into separate crates has
 123 been our solution so far.
 124
 125 At the top of the dependency tree are the [`rustc_interface`] and
 126 [`rustc_driver`] crates. [`rustc_interface`] is an unstable wrapper around the
 127 query system that helps to drive the various stages of compilation. Other
 128 consumers of the compiler may use this interface in different ways (e.g.
 129 rustdoc or maybe eventually rust-analyzer). The [`rustc_driver`] crate first
 130 parses command line arguments and then uses [`rustc_interface`] to drive the
 131 compilation to completion.
 132
 133 [query]: ./query.md
 134
 135 [orgch]: ./overview.md
 136
 137 ## rustdoc
 138
 139 The bulk of `rustdoc` is in [`librustdoc`]. However, the `rustdoc` binary
 140 itself is [`src/tools/rustdoc`], which does nothing except call [`rustdoc::main`].
 141
 142 There is also javascript and CSS for the rustdocs in [`src/tools/rustdoc-js`]
 143 and [`src/tools/rustdoc-themes`].
 144
 145 You can read more about rustdoc in [this chapter][rustdocch].
 146
 147 [`librustdoc`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustdoc/index.html
 148 [`rustdoc::main`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustdoc/fn.main.html
 149 [`src/tools/rustdoc`]:  https://github.com/rust-lang/rust/tree/master/src/tools/rustdoc
 150 [`src/tools/rustdoc-js`]: https://github.com/rust-lang/rust/tree/master/src/tools/rustdoc-js
 151 [`src/tools/rustdoc-themes`]: https://github.com/rust-lang/rust/tree/master/src/tools/rustdoc-themes
 152
 153 [rustdocch]: ./rustdoc.md
 154
 155 ## Tests
 156
 157 The test suite for all of the above is in [`src/test/`]. You can read more
 158 about the test suite [in this chapter][testsch].
 159
 160 The test harness itself is in [`src/tools/compiletest`].
 161
 162 [testsch]: ./tests/intro.md
 163
 164 [`src/test/`]: https://github.com/rust-lang/rust/tree/master/src/test
 165 [`src/tools/compiletest`]: https://github.com/rust-lang/rust/tree/master/src/tools/compiletest
 166
 167 ## Build System
 168
 169 There are a number of tools in the repository just for building the compiler,
 170 standard library, rustdoc, etc, along with testing, building a full Rust
 171 distribution, etc.
 172
 173 One of the primary tools is [`src/bootstrap`]. You can read more about
 174 bootstrapping [in this chapter][bootstch]. The process may also use other tools
 175 from `src/tools/`, such as [`tidy`] or [`compiletest`].
 176
 177 [`src/bootstrap`]: https://github.com/rust-lang/rust/tree/master/src/bootstrap
 178 [`tidy`]: https://github.com/rust-lang/rust/tree/master/src/tools/tidy
 179 [`compiletest`]: https://github.com/rust-lang/rust/tree/master/src/tools/compiletest
 180
 181 [bootstch]: ./building/bootstrapping.md
 182
 183 ## Other
 184
 185 There are a lot of other things in the `rust-lang/rust` repo that are related
 186 to building a full rust distribution. Most of the time you don't need to worry
 187 about them.
 188
 189 These include:
 190 - [`src/ci`]: The CI configuration. This actually quite extensive because we
 191   run a lot of tests on a lot of platforms.
 192 - [`src/doc`]: Various documentation, including submodules for a few books.
 193 - [`src/etc`]: Miscellaneous utilities.
 194 - [`src/tools/rustc-workspace-hack`], and others: Various workarounds to make
 195   cargo work with bootstrapping.
 196 - And more...
 197
 198 [`src/ci`]: https://github.com/rust-lang/rust/tree/master/src/ci
 199 [`src/doc`]: https://github.com/rust-lang/rust/tree/master/src/doc
 200 [`src/etc`]: https://github.com/rust-lang/rust/tree/master/src/etc
 201 [`src/tools/rustc-workspace-hack`]: https://github.com/rust-lang/rust/tree/master/src/tools/rustc-workspace-hack