1 Document by Ximin Luo, Luca Bruno & Sylvestre Ledru
3 This source package is unfortunately quite tricky and with several cutting
4 edges, due to the complexity of rust-lang bootstrapping system and the high
5 rate of language changes still ongoing.
7 We try to describe here inner packaging details and the reasons behind them.
9 If you are looking to help maintain this package, be sure to read the "Notes
10 for package maintainers" section further below.
16 The upstream source package embeds many external libraries. We make a great
17 effort to remove them and use system versions where possible, but there are a
20 * vendor/backtrace-sys, vendor/dlmalloc, vendor/walkdir
22 These are small C libraries designed to be statically linked; their upstream
23 does not support building them as a shared library and they are too small to
24 justify their own Debian package.
30 The Debian rustc package will use the system rustc to bootstrap itself from.
31 The system rustc has to be either the previous or the same version as the rustc
32 being built; the build will fail if this is not the case.
34 sudo apt-get build-dep ./
36 # Or, to directly use what's in the Debian FTP archive
37 sudo apt-get build-dep rustc
38 apt-get source --compile rustc
40 Alternatively, you may give the "pkg.rustc.dlstage0" DEB_BUILD_PROFILE to
41 instead use the process defined by Rust upstream. This downloads the "official"
42 stage0 compiler for the version being built from rust-lang.org. At the time of
43 writing "official" means "the previous stable version".
45 sudo apt-get build-dep -P pkg.rustc.dlstage0 ./
47 # Or, to directly use what's in the Debian FTP archive
48 sudo apt-get build-dep -P pkg.rustc.dlstage0 rustc
49 apt-get source --compile -P pkg.rustc.dlstage0 rustc
51 After [1] is fixed, both of these should in theory give identical results.
53 If neither of these options are acceptable to you, e.g. because your distro
54 does not have rustc already and your build process cannot access the network,
55 see "Bootstrapping" below.
57 [1] https://github.com/rust-lang/rust/issues/34902
63 To bootstrap rustc on a distro that does not have it or cargo available on any
64 architecture (so cross-compiling is not an option) you can run `debian/rules
65 source_orig-stage0`. This creates a .dsc that does not Build-Depend on rustc or
66 cargo. Instead, it includes an extra orig-stage0 source tarball that contains
67 the official stage0 compiler, pre-downloaded from rust-lang.org so that your
68 build daemons don't need to access the network during the build.
70 debian/rules source_orig-stage0
71 # Follow the final manual instructions that it outputs. Then:
72 sbuild ../rustc_*.dsc && dput ../rustc_*.dsc
74 To only bootstrap specific architectures, run this instead:
76 upstream_bootstrap_arch="arm64 armhf" debian/rules source_orig-stage0
78 This way, other architectures will be omitted from the orig-stage0 tarball. You
79 might want to do this e.g. if these other architectures are already present in
80 your distro, but the $upstream_bootstrap_arch ones are not yet present.
85 The approach bundles the upstream bootstrapping binaries inside the Debian
86 source package. This is a nasty hack that stretches the definition of "source
87 package", but has a few advantages explained below.
89 The traditional Debian way of bootstrapping compilers - and other distros have
90 similar approaches - is some variant of the following:
92 1. A developer locally installs some upstream bootstrapping binaries.
93 2. They locally build a Debian package, using these binaries as undeclared
95 3. They upload these binary packages to Debian, which can be used as declared
96 Build-Depends in the future, including by the same package.
98 The problem with this is, Debian does not have any policy nor infrastructure
99 that can try to reproduce what this developer supposedly did.
101 Using bootstrapping binary blobs *at some point of the process* is unavoidable.
102 Rather than pretending we didn't do this, it is better to record *which blobs*
103 we used, so it can be audited later. If we bundle non-Debian build-dependencies
104 inside the source package, then we can do a *source-only upload*, and the
105 building of the binary packages can be done by the normal build infrastructure.
107 If the build process is reproducible [1] then we can be sure that *you* (as the
108 developer that prepared the source-only upload) didn't backdoor the binaries,
109 nor did the build daemons even if they were compromised during the build.
111 The bootstrapping binaries may still have been backdoored, but this is true in
112 both scenarios. So our arrangement is still a strict improvement in security,
113 because it reduces the set of "things that may have been backdoored". Also,
114 more people use the upstream binaries than the "magical original Debian
115 package", so backdoors have a greater chance of being detected in the former.
117 In the long run, this process is laying the foundations for doing Diverse
118 Double-Compilation [2], where we use *many independent* bootstrapping binaries
119 to reproduce bit-for-bit identical output compilers, giving confidence that
120 nothing was backdoored along the way.
122 [1] The build process for rustc is currently *not* reproducible but we're
123 working towards it. https://github.com/rust-lang/rust/issues/34902
124 [2] http://www.dwheeler.com/trusting-trust/
127 Maintaining this package
128 ========================
130 Import of a new upstream version
131 --------------------------------
133 $ apt install equivs python3-magic
134 $ sudo mk-build-deps -irt 'aptitude -R'
135 $ uscan --verbose # or debian/rules source_orig-beta, for beta
136 $ ver=UPDATE-ME # whatever it is, probably X.YY.Z or X.YY.Z~beta.N
138 $ tar xf ../rustc-${ver/\~/-}-src.tar.xz && ( cd rustc-${ver/*~*/beta}-src/ && pwd && ../debian/prune-unused-deps ) && rm -rf rustc-${ver/*~*/beta}-src/
139 # ^ If this fails, you probably need to refresh the patches used by debian/prune-unused-deps
141 # Review the diff. If it removes too much stuff, it could mean that rustc
142 # pulled in new unnecessary dependencies in this newer version. See if you can
143 # drop them by amending the patch "d-0000-ignore-removed-submodules.patch".
144 # Rerun the above "tar ..." commands again and check that your patch works.
145 # For example, there is absolutely no reason why rustc should need openssl.
147 $ git commit -m "Update Files-Excluded for new upstream version ${ver/\~/-}" debian/copyright
148 $ uscan --verbose # yes, again, to pick up the new Files-Excluded stuff
149 # or debian/rules source_orig-beta, for beta
151 # Keep running this and follow its instructions, until it gives no output:
152 $ debian/check-orig-suspicious.sh $ver
153 # When you are satisfied with the above, proceed:
155 $ git checkout debian/experimental
156 $ gbp import-orig ../rustc_$ver+dfsg1.orig.tar.xz
157 $ dch -v $ver+dfsg1-1~exp1 "New upstream release."
158 $ debian/rules update-version
159 # might also need to bump the version of the cargo Build-Depends
160 # then refresh patches, etc etc
161 # Use /usr/share/cargo/scripts/guess-crate-copyright to help update d/copyright quickly
163 # If you need to repack again, bump the 'repacksuffix' in d/watch then run
164 $ uscan --verbose --force-download
165 # This will do a local repack using the new Files-Excluded rules, without
166 # redownloading the orig tarball (despite the slightly misleading flag).
169 Proceeding after build failure
170 ------------------------------
172 If your build fails, don't run `./x.py` directly as that will detect it's being
173 run with different settings, and run the build from scratch all over again.
174 overwriting all intermediate files. Instead, do:
176 $ debian/rules run_rustbuild X_CMD="build|test|install" X_FLAGS="whatever"
178 Hopefully, this will directly proceed to the step that failed, without
179 rebuilding everything in between.
182 Comparing Debian rustc vs upstream rustc
183 ----------------------------------------
185 This package does things the Debian way, which differs significantly from
186 upstream practices. If you find a bug, you might want to check if it is present
187 in the upstream package. Run "debian/rules debian/config.toml" to generate our
188 config.toml that you can then use in an upstream directory **unpacked from the
189 release tarball*. (It is more complex to get this working with their git repo.)
191 This will configure it in a "halfway" style between upstream and Debian.
192 Specifically, it will not build LLVM nor download stuff from crates.io, yet
193 Debian patches are *not* applied. These specific settings were chosen as a
194 tradeoff between convenience vs being close to what upstream does - so that the
195 chances of a bug here being a genuine upstream issue rather than a Debian bug,
196 is much higher. Also, with the exception of LLVM, these are non-default modes
197 *supported by* upstream so they would be happy to receive bug reports about it
198 even if your issue only occurs here.
200 OTOH if you need to test a completely clean upstream build, including all the
201 annoying stuff like building LLVM and downloading dependencies from crates.io,
202 simply unpack the tarball and run `./configure && ./x.py build` etc as normal.
203 This can be useful for confirming that an issue is caused by Debian's LLVM.
205 If you need to test a LLVM patch, do something like this:
207 # build your patched LLVM debs, then:
208 $ mkdir -p llvm-destdir && cd llvm-destdir
209 $ ver=4.0; VERSION=FIXME
210 $ for i in llvm-$ver llvm-$ver-dev llvm-$ver-runtime llvm-$ver-tools libllvm$ver; do \
211 dpkg -x ../"$i"_*${VERSION}_*.deb .; done
213 $ debian/rules LLVM_DESTDIR=$PWD/../llvm-destdir build
215 If you need to test a patch to the stage0 rustc, do something like this:
217 # build your patched rustc debs or upstream rustc, then:
218 $ mkdir -p rust-destdir && cd rust-destdir
219 $ ver=1.20; VERSION=FIXME;
220 $ for i in rustc libstd-rust-$ver libstd-rust-dev; do \
221 dpkg -x ../"$i"_*${VERSION}_*.deb .; done
223 $ debian/rules RUST_DESTDIR=$PWD/../rust-destdir build
229 The Fedora rust team is more active than the Debian one. Here are their links:
232 https://src.fedoraproject.org/rpms/rust/tree/
234 Binary packages and test logs
235 https://kojipkgs.fedoraproject.org//packages/rust/
236 If the same test fails both on Fedora and Debian it's a good indication that
237 we're not Doing It Wrong and can file a valid bug upstream.