]>
Commit | Line | Data |
---|---|---|
83b02c13 | 1 | Document by Ximin Luo, Luca Bruno & Sylvestre Ledru |
ce1d3dbe | 2 | |
83b02c13 XL |
3 | This source package is unfortunately quite tricky and with several cutting |
4 | edges, due to the complexity of rust-lang bootstrapping system and the high | |
5 | rate of language changes still ongoing. | |
ad75087d | 6 | |
83b02c13 | 7 | We try to describe here inner packaging details and the reasons behind them. |
ad75087d | 8 | |
6c5c66c7 XL |
9 | If you are looking to help maintain this package, be sure to read the "Notes |
10 | for package maintainers" section further below. | |
2b7c35b5 | 11 | |
ad75087d | 12 | |
ad75087d LB |
13 | Embedded libraries |
14 | ================== | |
15 | ||
e2a86b73 XL |
16 | The upstream source package embeds many external libraries. We make a great |
17 | effort to remove them and use system versions where possible, but there are a | |
18 | few more remaining: | |
ad75087d | 19 | |
d41de177 | 20 | * vendor/dlmalloc, vendor/windows_*_gnu, vendor/windows_*_msvc |
ad75087d | 21 | |
e2a86b73 XL |
22 | These are small C libraries designed to be statically linked; their upstream |
23 | does not support building them as a shared library and they are too small to | |
24 | justify their own Debian package. | |
8f875a6a XL |
25 | |
26 | ||
27 | Building from source | |
28 | ==================== | |
29 | ||
30 | The Debian rustc package will use the system rustc to bootstrap itself from. | |
31 | The system rustc has to be either the previous or the same version as the rustc | |
32 | being built; the build will fail if this is not the case. | |
33 | ||
34 | sudo apt-get build-dep ./ | |
35 | dpkg-buildpackage | |
36 | # Or, to directly use what's in the Debian FTP archive | |
37 | sudo apt-get build-dep rustc | |
38 | apt-get source --compile rustc | |
39 | ||
40 | Alternatively, you may give the "pkg.rustc.dlstage0" DEB_BUILD_PROFILE to | |
41 | instead use the process defined by Rust upstream. This downloads the "official" | |
42 | stage0 compiler for the version being built from rust-lang.org. At the time of | |
43 | writing "official" means "the previous stable version". | |
44 | ||
45 | sudo apt-get build-dep -P pkg.rustc.dlstage0 ./ | |
3a2eac0a | 46 | dpkg-buildpackage -P pkg.rustc.dlstage0 |
8f875a6a XL |
47 | # Or, to directly use what's in the Debian FTP archive |
48 | sudo apt-get build-dep -P pkg.rustc.dlstage0 rustc | |
49 | apt-get source --compile -P pkg.rustc.dlstage0 rustc | |
50 | ||
51 | After [1] is fixed, both of these should in theory give identical results. | |
52 | ||
53 | If neither of these options are acceptable to you, e.g. because your distro | |
54 | does not have rustc already and your build process cannot access the network, | |
55 | see "Bootstrapping" below. | |
56 | ||
57 | [1] https://github.com/rust-lang/rust/issues/34902 | |
58 | ||
59 | ||
60 | Bootstrapping | |
61 | ============= | |
62 | ||
63 | To bootstrap rustc on a distro that does not have it or cargo available on any | |
64 | architecture (so cross-compiling is not an option) you can run `debian/rules | |
65 | source_orig-stage0`. This creates a .dsc that does not Build-Depend on rustc or | |
66 | cargo. Instead, it includes an extra orig-stage0 source tarball that contains | |
67 | the official stage0 compiler, pre-downloaded from rust-lang.org so that your | |
68 | build daemons don't need to access the network during the build. | |
69 | ||
70 | debian/rules source_orig-stage0 | |
71 | # Follow the final manual instructions that it outputs. Then: | |
72 | sbuild ../rustc_*.dsc && dput ../rustc_*.dsc | |
73 | ||
74 | To only bootstrap specific architectures, run this instead: | |
75 | ||
76 | upstream_bootstrap_arch="arm64 armhf" debian/rules source_orig-stage0 | |
77 | ||
78 | This way, other architectures will be omitted from the orig-stage0 tarball. You | |
79 | might want to do this e.g. if these other architectures are already present in | |
80 | your distro, but the $upstream_bootstrap_arch ones are not yet present. | |
81 | ||
82 | Notes | |
83 | ----- | |
84 | ||
85 | The approach bundles the upstream bootstrapping binaries inside the Debian | |
86 | source package. This is a nasty hack that stretches the definition of "source | |
87 | package", but has a few advantages explained below. | |
88 | ||
89 | The traditional Debian way of bootstrapping compilers - and other distros have | |
90 | similar approaches - is some variant of the following: | |
91 | ||
92 | 1. A developer locally installs some upstream bootstrapping binaries. | |
93 | 2. They locally build a Debian package, using these binaries as undeclared | |
94 | build dependencies. | |
95 | 3. They upload these binary packages to Debian, which can be used as declared | |
96 | Build-Depends in the future, including by the same package. | |
97 | ||
98 | The problem with this is, Debian does not have any policy nor infrastructure | |
99 | that can try to reproduce what this developer supposedly did. | |
100 | ||
101 | Using bootstrapping binary blobs *at some point of the process* is unavoidable. | |
102 | Rather than pretending we didn't do this, it is better to record *which blobs* | |
103 | we used, so it can be audited later. If we bundle non-Debian build-dependencies | |
104 | inside the source package, then we can do a *source-only upload*, and the | |
105 | building of the binary packages can be done by the normal build infrastructure. | |
106 | ||
107 | If the build process is reproducible [1] then we can be sure that *you* (as the | |
108 | developer that prepared the source-only upload) didn't backdoor the binaries, | |
109 | nor did the build daemons even if they were compromised during the build. | |
110 | ||
111 | The bootstrapping binaries may still have been backdoored, but this is true in | |
112 | both scenarios. So our arrangement is still a strict improvement in security, | |
113 | because it reduces the set of "things that may have been backdoored". Also, | |
114 | more people use the upstream binaries than the "magical original Debian | |
115 | package", so backdoors have a greater chance of being detected in the former. | |
116 | ||
117 | In the long run, this process is laying the foundations for doing Diverse | |
118 | Double-Compilation [2], where we use *many independent* bootstrapping binaries | |
119 | to reproduce bit-for-bit identical output compilers, giving confidence that | |
120 | nothing was backdoored along the way. | |
121 | ||
122 | [1] The build process for rustc is currently *not* reproducible but we're | |
123 | working towards it. https://github.com/rust-lang/rust/issues/34902 | |
124 | [2] http://www.dwheeler.com/trusting-trust/ | |
83b02c13 XL |
125 | |
126 | ||
6c5c66c7 XL |
127 | Maintaining this package |
128 | ======================== | |
129 | ||
130 | Import of a new upstream version | |
131 | -------------------------------- | |
132 | ||
d856205a | 133 | $ apt install equivs python3-magic |
0ae8a120 | 134 | $ sudo mk-build-deps -irt 'aptitude -R' |
0a1981fb XL |
135 | $ uscan --verbose # or debian/rules source_orig-beta, for beta |
136 | $ ver=UPDATE-ME # whatever it is, probably X.YY.Z or X.YY.Z~beta.N | |
c54b4d09 | 137 | |
51a673e4 XL |
138 | $ debian/refresh-early-patches.sh $ver |
139 | # This will require an understanding of how git-rebase and git-mergetool works | |
140 | # We recommend either kdiff3 or p4merge (proprietary) as the git-mergetool. | |
141 | ||
f9b5d476 | 142 | $ tar xf ../rustc-${ver/\~/-}-src.tar.xz && ( cd rustc-${ver/*~*/beta}-src/ && pwd && ../debian/prune-unused-deps ) && rm -rf rustc-${ver/*~*/beta}-src/ |
c54b4d09 XL |
143 | $ git diff |
144 | # Review the diff. If it removes too much stuff, it could mean that rustc | |
145 | # pulled in new unnecessary dependencies in this newer version. See if you can | |
146 | # drop them by amending the patch "d-0000-ignore-removed-submodules.patch". | |
147 | # Rerun the above "tar ..." commands again and check that your patch works. | |
148 | # For example, there is absolutely no reason why rustc should need openssl. | |
149 | ||
3d32fa04 | 150 | $ git commit -m "Update Files-Excluded for new upstream version ${ver/\~/-}" debian/copyright |
0a1981fb XL |
151 | $ uscan --verbose # yes, again, to pick up the new Files-Excluded stuff |
152 | # or debian/rules source_orig-beta, for beta | |
6c5c66c7 XL |
153 | |
154 | # Keep running this and follow its instructions, until it gives no output: | |
155 | $ debian/check-orig-suspicious.sh $ver | |
156 | # When you are satisfied with the above, proceed: | |
157 | ||
8598b294 | 158 | $ git checkout debian/experimental |
6c5c66c7 | 159 | $ gbp import-orig ../rustc_$ver+dfsg1.orig.tar.xz |
5c02168d | 160 | $ dch -v $ver+dfsg1-1~exp1 "New upstream release." |
6c5c66c7 XL |
161 | $ debian/rules update-version |
162 | # might also need to bump the version of the cargo Build-Depends | |
163 | # then refresh patches, etc etc | |
99f22c16 | 164 | # Use /usr/share/cargo/scripts/guess-crate-copyright to help update d/copyright quickly |
6c5c66c7 XL |
165 | |
166 | # If you need to repack again, bump the 'repacksuffix' in d/watch then run | |
167 | $ uscan --verbose --force-download | |
168 | # This will do a local repack using the new Files-Excluded rules, without | |
169 | # redownloading the orig tarball (despite the slightly misleading flag). | |
170 | ||
171 | ||
172 | Proceeding after build failure | |
173 | ------------------------------ | |
174 | ||
175 | If your build fails, don't run `./x.py` directly as that will detect it's being | |
176 | run with different settings, and run the build from scratch all over again. | |
177 | overwriting all intermediate files. Instead, do: | |
178 | ||
179 | $ debian/rules run_rustbuild X_CMD="build|test|install" X_FLAGS="whatever" | |
180 | ||
181 | Hopefully, this will directly proceed to the step that failed, without | |
182 | rebuilding everything in between. | |
183 | ||
184 | ||
185 | Comparing Debian rustc vs upstream rustc | |
186 | ---------------------------------------- | |
187 | ||
188 | This package does things the Debian way, which differs significantly from | |
189 | upstream practices. If you find a bug, you might want to check if it is present | |
5861ebac | 190 | in the upstream package. Run "debian/rules debian/config.toml" to generate our |
9244de54 XL |
191 | config.toml that you can then use in an upstream directory **unpacked from the |
192 | release tarball*. (It is more complex to get this working with their git repo.) | |
6c5c66c7 XL |
193 | |
194 | This will configure it in a "halfway" style between upstream and Debian. | |
195 | Specifically, it will not build LLVM nor download stuff from crates.io, yet | |
196 | Debian patches are *not* applied. These specific settings were chosen as a | |
197 | tradeoff between convenience vs being close to what upstream does - so that the | |
198 | chances of a bug here being a genuine upstream issue rather than a Debian bug, | |
199 | is much higher. Also, with the exception of LLVM, these are non-default modes | |
200 | *supported by* upstream so they would be happy to receive bug reports about it | |
201 | even if your issue only occurs here. | |
202 | ||
203 | OTOH if you need to test a completely clean upstream build, including all the | |
204 | annoying stuff like building LLVM and downloading dependencies from crates.io, | |
205 | simply unpack the tarball and run `./configure && ./x.py build` etc as normal. | |
206 | This can be useful for confirming that an issue is caused by Debian's LLVM. | |
207 | ||
208 | If you need to test a LLVM patch, do something like this: | |
209 | ||
210 | # build your patched LLVM debs, then: | |
211 | $ mkdir -p llvm-destdir && cd llvm-destdir | |
56548b60 XL |
212 | $ ver=4.0; VERSION=FIXME |
213 | $ for i in llvm-$ver llvm-$ver-dev llvm-$ver-runtime llvm-$ver-tools libllvm$ver; do \ | |
6c5c66c7 XL |
214 | dpkg -x ../"$i"_*${VERSION}_*.deb .; done |
215 | $ cd ../rustc | |
216 | $ debian/rules LLVM_DESTDIR=$PWD/../llvm-destdir build | |
217 | ||
56548b60 XL |
218 | If you need to test a patch to the stage0 rustc, do something like this: |
219 | ||
220 | # build your patched rustc debs or upstream rustc, then: | |
221 | $ mkdir -p rust-destdir && cd rust-destdir | |
222 | $ ver=1.20; VERSION=FIXME; | |
223 | $ for i in rustc libstd-rust-$ver libstd-rust-dev; do \ | |
224 | dpkg -x ../"$i"_*${VERSION}_*.deb .; done | |
225 | $ cd ../rustc | |
226 | $ debian/rules RUST_DESTDIR=$PWD/../rust-destdir build | |
227 | ||
6c5c66c7 | 228 | |
83b02c13 | 229 | Useful links |
6c5c66c7 | 230 | ------------ |
83b02c13 XL |
231 | |
232 | The Fedora rust team is more active than the Debian one. Here are their links: | |
233 | ||
234 | Source code | |
6a9fb0d7 | 235 | https://src.fedoraproject.org/rpms/rust/tree/ |
83b02c13 XL |
236 | |
237 | Binary packages and test logs | |
238 | https://kojipkgs.fedoraproject.org//packages/rust/ | |
239 | If the same test fails both on Fedora and Debian it's a good indication that | |
240 | we're not Doing It Wrong and can file a valid bug upstream. |