1 Document by Ximin Luo, Luca Bruno & Sylvestre Ledru
3 This source package is unfortunately quite tricky and with several cutting
4 edges, due to the complexity of rust-lang bootstrapping system and the high
5 rate of language changes still ongoing.
7 We try to describe here inner packaging details and the reasons behind them.
9 If you are looking to help maintain this package, be sure to read the "Notes
10 for package maintainers" section further below.
16 This source package embeds several external libraries (foeked and managed
17 by rust upstream as git submodules).
18 In early stages, many more libraries were forked/emebedded but we are steadily
19 progressing in splitting them out.
21 Here below the remaining ones, with the technical reasons.
23 * compiler-rt from https://github.com/rust-lang/compiler-rt
24 -> system-wide compiler-rt fails during linkage
26 Bug reported upstream, still to be fixed, see:
27 - https://github.com/rust-lang/rust/issues/15054
28 - https://github.com/rust-lang/rust/issues/15708
30 As a summary, we plan to:
31 * work with upstream to fix compiler-rt linkage soon.
33 -- Sylvestre Ledru <sylvestre@debian.org> Sat, 06 May 2017 13:26:08 +0200
39 The Debian rustc package will use the system rustc to bootstrap itself from.
40 The system rustc has to be either the previous or the same version as the rustc
41 being built; the build will fail if this is not the case.
43 sudo apt-get build-dep ./
45 # Or, to directly use what's in the Debian FTP archive
46 sudo apt-get build-dep rustc
47 apt-get source --compile rustc
49 Alternatively, you may give the "pkg.rustc.dlstage0" DEB_BUILD_PROFILE to
50 instead use the process defined by Rust upstream. This downloads the "official"
51 stage0 compiler for the version being built from rust-lang.org. At the time of
52 writing "official" means "the previous stable version".
54 sudo apt-get build-dep -P pkg.rustc.dlstage0 ./
56 # Or, to directly use what's in the Debian FTP archive
57 sudo apt-get build-dep -P pkg.rustc.dlstage0 rustc
58 apt-get source --compile -P pkg.rustc.dlstage0 rustc
60 After [1] is fixed, both of these should in theory give identical results.
62 If neither of these options are acceptable to you, e.g. because your distro
63 does not have rustc already and your build process cannot access the network,
64 see "Bootstrapping" below.
66 [1] https://github.com/rust-lang/rust/issues/34902
72 To bootstrap rustc on a distro that does not have it or cargo available on any
73 architecture (so cross-compiling is not an option) you can run `debian/rules
74 source_orig-stage0`. This creates a .dsc that does not Build-Depend on rustc or
75 cargo. Instead, it includes an extra orig-stage0 source tarball that contains
76 the official stage0 compiler, pre-downloaded from rust-lang.org so that your
77 build daemons don't need to access the network during the build.
79 debian/rules source_orig-stage0
80 # Follow the final manual instructions that it outputs. Then:
81 sbuild ../rustc_*.dsc && dput ../rustc_*.dsc
83 To only bootstrap specific architectures, run this instead:
85 upstream_bootstrap_arch="arm64 armhf" debian/rules source_orig-stage0
87 This way, other architectures will be omitted from the orig-stage0 tarball. You
88 might want to do this e.g. if these other architectures are already present in
89 your distro, but the $upstream_bootstrap_arch ones are not yet present.
94 The approach bundles the upstream bootstrapping binaries inside the Debian
95 source package. This is a nasty hack that stretches the definition of "source
96 package", but has a few advantages explained below.
98 The traditional Debian way of bootstrapping compilers - and other distros have
99 similar approaches - is some variant of the following:
101 1. A developer locally installs some upstream bootstrapping binaries.
102 2. They locally build a Debian package, using these binaries as undeclared
104 3. They upload these binary packages to Debian, which can be used as declared
105 Build-Depends in the future, including by the same package.
107 The problem with this is, Debian does not have any policy nor infrastructure
108 that can try to reproduce what this developer supposedly did.
110 Using bootstrapping binary blobs *at some point of the process* is unavoidable.
111 Rather than pretending we didn't do this, it is better to record *which blobs*
112 we used, so it can be audited later. If we bundle non-Debian build-dependencies
113 inside the source package, then we can do a *source-only upload*, and the
114 building of the binary packages can be done by the normal build infrastructure.
116 If the build process is reproducible [1] then we can be sure that *you* (as the
117 developer that prepared the source-only upload) didn't backdoor the binaries,
118 nor did the build daemons even if they were compromised during the build.
120 The bootstrapping binaries may still have been backdoored, but this is true in
121 both scenarios. So our arrangement is still a strict improvement in security,
122 because it reduces the set of "things that may have been backdoored". Also,
123 more people use the upstream binaries than the "magical original Debian
124 package", so backdoors have a greater chance of being detected in the former.
126 In the long run, this process is laying the foundations for doing Diverse
127 Double-Compilation [2], where we use *many independent* bootstrapping binaries
128 to reproduce bit-for-bit identical output compilers, giving confidence that
129 nothing was backdoored along the way.
131 [1] The build process for rustc is currently *not* reproducible but we're
132 working towards it. https://github.com/rust-lang/rust/issues/34902
133 [2] http://www.dwheeler.com/trusting-trust/
136 Maintaining this package
137 ========================
139 Import of a new upstream version
140 --------------------------------
142 # Make sure you apply the patch mentioned in #855464 to /usr/bin/mk-origtargz
144 $ sudo mk-build-deps -irt 'aptitude -R'
146 $ ver=UPDATE-ME # whatever it is, X.YY.0 or X.YY.0~beta probably
147 $ tar xf ../rustc-${ver/\~/-}-src.tar.xz && ( cd rustc-${ver/*~*/beta}-src/ && ../debian/prune-unused-deps ) && rm -rf rustc-${ver/*~*/beta}-src/
148 # ^ If this fails, you probably need to refresh patches or edit debian/prune-unused-deps
149 $ git commit -m "Update Files-Excluded for new upstream version ${ver/\~/-}" debian/copyright
150 $ uscan --verbose # yes, again, to pick up the new Files-Excluded stuff
152 # Keep running this and follow its instructions, until it gives no output:
153 $ debian/check-orig-suspicious.sh $ver
154 # When you are satisfied with the above, proceed:
156 $ git checkout debian/experimental
157 $ gbp import-orig ../rustc_$ver+dfsg1.orig.tar.xz
158 $ dch -v $ver+dfsg1-1~exp1 -m "New upstream release."
159 $ debian/rules update-version
160 # might also need to bump the version of the cargo Build-Depends
161 # then refresh patches, etc etc
162 # Use /usr/share/cargo/guess-crate-copyright to help update d/copyright quickly
164 # If you need to repack again, bump the 'repacksuffix' in d/watch then run
165 $ uscan --verbose --force-download
166 # This will do a local repack using the new Files-Excluded rules, without
167 # redownloading the orig tarball (despite the slightly misleading flag).
170 Proceeding after build failure
171 ------------------------------
173 If your build fails, don't run `./x.py` directly as that will detect it's being
174 run with different settings, and run the build from scratch all over again.
175 overwriting all intermediate files. Instead, do:
177 $ debian/rules run_rustbuild X_CMD="build|test|install" X_FLAGS="whatever"
179 Hopefully, this will directly proceed to the step that failed, without
180 rebuilding everything in between.
183 Comparing Debian rustc vs upstream rustc
184 ----------------------------------------
186 This package does things the Debian way, which differs significantly from
187 upstream practices. If you find a bug, you might want to check if it is present
188 in the upstream package. Run "debian/rules debian/config.toml" to generate our
189 config.toml that you can then use in an upstream directory **unpacked from the
190 release tarball*. (It is more complex to get this working with their git repo.)
192 This will configure it in a "halfway" style between upstream and Debian.
193 Specifically, it will not build LLVM nor download stuff from crates.io, yet
194 Debian patches are *not* applied. These specific settings were chosen as a
195 tradeoff between convenience vs being close to what upstream does - so that the
196 chances of a bug here being a genuine upstream issue rather than a Debian bug,
197 is much higher. Also, with the exception of LLVM, these are non-default modes
198 *supported by* upstream so they would be happy to receive bug reports about it
199 even if your issue only occurs here.
201 OTOH if you need to test a completely clean upstream build, including all the
202 annoying stuff like building LLVM and downloading dependencies from crates.io,
203 simply unpack the tarball and run `./configure && ./x.py build` etc as normal.
204 This can be useful for confirming that an issue is caused by Debian's LLVM.
206 If you need to test a LLVM patch, do something like this:
208 # build your patched LLVM debs, then:
209 $ mkdir -p llvm-destdir && cd llvm-destdir
210 $ ver=4.0; VERSION=FIXME
211 $ for i in llvm-$ver llvm-$ver-dev llvm-$ver-runtime llvm-$ver-tools libllvm$ver; do \
212 dpkg -x ../"$i"_*${VERSION}_*.deb .; done
214 $ debian/rules LLVM_DESTDIR=$PWD/../llvm-destdir build
216 If you need to test a patch to the stage0 rustc, do something like this:
218 # build your patched rustc debs or upstream rustc, then:
219 $ mkdir -p rust-destdir && cd rust-destdir
220 $ ver=1.20; VERSION=FIXME;
221 $ for i in rustc libstd-rust-$ver libstd-rust-dev; do \
222 dpkg -x ../"$i"_*${VERSION}_*.deb .; done
224 $ debian/rules RUST_DESTDIR=$PWD/../rust-destdir build
230 The Fedora rust team is more active than the Debian one. Here are their links:
233 https://src.fedoraproject.org/cgit/rpms/rust.git/tree/
235 Binary packages and test logs
236 https://kojipkgs.fedoraproject.org//packages/rust/
237 If the same test fails both on Fedora and Debian it's a good indication that
238 we're not Doing It Wrong and can file a valid bug upstream.
241 https://admin.fedoraproject.org/pkgdb/package/rpms/rust/