]>
Commit | Line | Data |
---|---|---|
a1dfa0c6 XL |
1 | # High-level overview of the compiler source |
2 | ||
6a06907d XL |
3 | <!-- toc --> |
4 | ||
5 | > **NOTE**: The structure of the repository is going through a lot of | |
6 | > transitions. In particular, we want to get to a point eventually where the | |
7 | > top-level directory has separate directories for the compiler, build-system, | |
8 | > std libs, etc, rather than one huge `src/` directory. | |
9 | > | |
10 | > As of <!-- date: 2021-01 --> January 2021, the standard libraries have been | |
11 | > moved to `library/` and the crates that make up the `rustc` compiler itself | |
12 | > have been moved to `compiler/`. | |
13 | ||
14 | Now that we have [seen what the compiler does](./overview.md), let's take a | |
15 | look at the structure of the contents of the rust-lang/rust repo. | |
16 | ||
17 | ## Workspace structure | |
18 | ||
19 | The `rust-lang/rust` repository consists of a single large cargo workspace | |
20 | containing the compiler, the standard libraries (`core`, `alloc`, `std`, | |
21 | `proc_macro`, etc), and `rustdoc`, along with the build system and a bunch of | |
22 | tools and submodules for building a full Rust distribution. | |
23 | ||
24 | The repository consists of three main directories: | |
25 | ||
26 | - `compiler/` contains the source code for `rustc`. It consists of many crates | |
27 | that together make up the compiler. | |
28 | ||
29 | - `library/` contains the standard libraries (`core`, `alloc`, `std`, | |
30 | `proc_macro`, `test`), as well as the Rust runtime (`backtrace`, `rtstartup`, | |
31 | `lang_start`). | |
32 | ||
33 | - `src/` contains the source code for rustdoc, clippy, cargo, the build system, | |
34 | language docs, etc. | |
35 | ||
36 | ## Standard library | |
37 | ||
38 | The standard library crates are all in `library/`. They have intuitive names | |
39 | like `std`, `core`, `alloc`, etc. There is also `proc_macro`, `test`, and | |
40 | other runtime libraries. | |
41 | ||
42 | This code is fairly similar to most other Rust crates except that it must be | |
43 | built in a special way because it can use unstable features. | |
44 | ||
45 | ## Compiler | |
46 | ||
47 | > You may find it helpful to read [The Overview Chapter](./overview.md) first, | |
48 | > which gives an overview of how the compiler works. The crates mentioned in | |
49 | > this section implement the compiler, and are underneath `compiler/` | |
50 | ||
51 | The `compiler/` crates all have names starting with `rustc_*`. These are a | |
52 | collection of around 50 interdependent crates ranging in size from tiny to | |
53 | huge. There is also the `rustc` crate which is the actual binary (i.e. the | |
54 | `main` function); it doesn't actually do anything besides calling the | |
55 | `rustc_driver` crate, which drives the various parts of compilation in other | |
56 | crates. | |
57 | ||
58 | The dependency structure of these crates is complex, but roughly it is | |
59 | something like this: | |
60 | ||
61 | - `rustc` (the binary) calls [`rustc_driver::main`][main]. | |
62 | - [`rustc_driver`] depends on a lot of other crates, but the main one is | |
63 | [`rustc_interface`]. | |
64 | - [`rustc_interface`] depends on most of the other compiler crates. It | |
65 | is a fairly generic interface for driving the whole compilation. | |
66 | - Most of the other `rustc_*` crates depend on [`rustc_middle`], | |
67 | which defines a lot of central data structures in the compiler. | |
68 | - [`rustc_middle`] and most of the other crates depend on a | |
69 | handful of crates representing the early parts of the | |
70 | compiler (e.g. the parser), fundamental data structures (e.g. | |
71 | [`Span`]), or error reporting: [`rustc_data_structures`], | |
72 | [`rustc_span`], [`rustc_errors`], etc. | |
73 | ||
74 | [main]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_driver/fn.main.html | |
75 | [`rustc_driver`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_driver/index.html | |
76 | [`rustc_interface`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_interface/index.html | |
77 | [`rustc_middle`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_middle/index.html | |
78 | [`rustc_data_structures`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_data_structures/index.html | |
79 | [`rustc_span`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_span/index.html | |
80 | [`Span`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_span/struct.Span.html | |
81 | [`rustc_errors`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_errors/index.html | |
82 | ||
83 | You can see the exact dependencies by reading the `Cargo.toml` for the various | |
84 | crates, just like a normal Rust crate. | |
85 | ||
86 | One final thing: [`src/llvm-project`] is a submodule for our fork of LLVM. | |
87 | During bootstrapping, LLVM is built and the [`compiler/rustc_llvm`] crate | |
88 | contains rust wrappers around LLVM (which is written in C++), so that the | |
89 | compiler can interface with it. | |
90 | ||
91 | Most of this book is about the compiler, so we won't have any further | |
92 | explanation of these crates here. | |
93 | ||
94 | [`src/llvm-project`]: https://github.com/rust-lang/rust/tree/master/src/ | |
95 | [`compiler/rustc_llvm`]: https://github.com/rust-lang/rust/tree/master/compiler/rustc_llvm | |
96 | ||
97 | ### Big picture | |
98 | ||
99 | The dependency structure is influenced strongly by two main factors: | |
100 | ||
101 | 1. Organization. The compiler is a _huge_ codebase; it would be an impossibly | |
102 | large crate. In part, the dependency structure reflects the code structure | |
103 | of the compiler. | |
104 | 2. Compile time. By breaking the compiler into multiple crates, we can take | |
105 | better advantage of incremental/parallel compilation using cargo. In | |
106 | particular, we try to have as few dependencies between crates as possible so | |
107 | that we don't have to rebuild as many crates if you change one. | |
108 | ||
109 | At the very bottom of the dependency tree are a handful of crates that are used | |
110 | by the whole compiler (e.g. [`rustc_span`]). The very early parts of the | |
111 | compilation process (e.g. parsing and the AST) depend on only these. | |
112 | ||
113 | Pretty soon after the AST is constructed, the compiler's [query system][query] | |
114 | gets set up. The query system is set up in a clever way using function | |
115 | pointers. This allows us to break dependencies between crates, allowing more | |
116 | parallel compilation. | |
117 | ||
118 | However, since the query system is defined in [`rustc_middle`], nearly all | |
119 | subsequent parts of the compiler depend on this crate. It is a really large | |
120 | crate, leading to long compile times. Some efforts have been made to move stuff | |
121 | out of it with limited success. Another unfortunate side effect is that sometimes | |
122 | related functionality gets scattered across different crates. For example, | |
123 | linting functionality is scattered across earlier parts of the crate, | |
124 | [`rustc_lint`], [`rustc_middle`], and other places. | |
125 | ||
126 | [`rustc_lint`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_lint/index.html | |
127 | ||
128 | More generally, in an ideal world, it seems like there would be fewer, more | |
129 | cohesive crates, with incremental and parallel compilation making sure compile | |
130 | times stay reasonable. However, our incremental and parallel compilation haven't | |
131 | gotten good enough for that yet, so breaking things into separate crates has | |
132 | been our solution so far. | |
133 | ||
134 | At the top of the dependency tree are the [`rustc_interface`] and | |
135 | [`rustc_driver`] crates. [`rustc_interface`] is an unstable wrapper around the | |
136 | query system that helps to drive the various stages of compilation. Other | |
137 | consumers of the compiler may use this interface in different ways (e.g. | |
138 | rustdoc or maybe eventually rust-analyzer). The [`rustc_driver`] crate first | |
139 | parses command line arguments and then uses [`rustc_interface`] to drive the | |
140 | compilation to completion. | |
141 | ||
142 | [query]: ./query.md | |
143 | ||
144 | [orgch]: ./overview.md | |
145 | ||
146 | ## rustdoc | |
147 | ||
148 | The bulk of `rustdoc` is in [`librustdoc`]. However, the `rustdoc` binary | |
149 | itself is [`src/tools/rustdoc`], which does nothing except call [`rustdoc::main`]. | |
150 | ||
151 | There is also javascript and CSS for the rustdocs in [`src/tools/rustdoc-js`] | |
152 | and [`src/tools/rustdoc-themes`]. | |
153 | ||
154 | You can read more about rustdoc in [this chapter][rustdocch]. | |
155 | ||
156 | [`librustdoc`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustdoc/index.html | |
157 | [`rustdoc::main`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustdoc/fn.main.html | |
158 | [`src/tools/rustdoc`]: https://github.com/rust-lang/rust/tree/master/src/tools/rustdoc | |
159 | [`src/tools/rustdoc-js`]: https://github.com/rust-lang/rust/tree/master/src/tools/rustdoc-js | |
160 | [`src/tools/rustdoc-themes`]: https://github.com/rust-lang/rust/tree/master/src/tools/rustdoc-themes | |
161 | ||
162 | [rustdocch]: ./rustdoc.md | |
163 | ||
164 | ## Tests | |
165 | ||
166 | The test suite for all of the above is in [`src/test/`]. You can read more | |
167 | about the test suite [in this chapter][testsch]. | |
168 | ||
169 | The test harness itself is in [`src/tools/compiletest`]. | |
170 | ||
171 | [testsch]: ./tests/intro.md | |
172 | ||
173 | [`src/test/`]: https://github.com/rust-lang/rust/tree/master/src/test | |
174 | [`src/tools/compiletest`]: https://github.com/rust-lang/rust/tree/master/src/tools/compiletest | |
175 | ||
176 | ## Build System | |
177 | ||
178 | There are a number of tools in the repository just for building the compiler, | |
179 | standard library, rustdoc, etc, along with testing, building a full Rust | |
180 | distribution, etc. | |
181 | ||
182 | One of the primary tools is [`src/bootstrap`]. You can read more about | |
183 | bootstrapping [in this chapter][bootstch]. The process may also use other tools | |
184 | from `src/tools/`, such as [`tidy`] or [`compiletest`]. | |
185 | ||
186 | [`src/bootstrap`]: https://github.com/rust-lang/rust/tree/master/src/bootstrap | |
187 | [`tidy`]: https://github.com/rust-lang/rust/tree/master/src/tools/tidy | |
188 | [`compiletest`]: https://github.com/rust-lang/rust/tree/master/src/tools/compiletest | |
189 | ||
190 | [bootstch]: ./building/bootstrapping.md | |
191 | ||
192 | ## Other | |
193 | ||
194 | There are a lot of other things in the `rust-lang/rust` repo that are related | |
195 | to building a full rust distribution. Most of the time you don't need to worry | |
196 | about them. | |
197 | ||
198 | These include: | |
199 | - [`src/ci`]: The CI configuration. This actually quite extensive because we | |
200 | run a lot of tests on a lot of platforms. | |
201 | - [`src/doc`]: Various documentation, including submodules for a few books. | |
202 | - [`src/etc`]: Miscellaneous utilities. | |
203 | - [`src/tools/rustc-workspace-hack`], and others: Various workarounds to make | |
204 | cargo work with bootstrapping. | |
205 | - And more... | |
206 | ||
207 | [`src/ci`]: https://github.com/rust-lang/rust/tree/master/src/ci | |
208 | [`src/doc`]: https://github.com/rust-lang/rust/tree/master/src/doc | |
209 | [`src/etc`]: https://github.com/rust-lang/rust/tree/master/src/etc | |
210 | [`src/tools/rustc-workspace-hack`]: https://github.com/rust-lang/rust/tree/master/src/tools/rustc-workspace-hack |