debian/README.source

   1 Document by Ximin Luo, Luca Bruno & Sylvestre Ledru
   2
   3 This source package is unfortunately quite tricky and with several cutting
   4 edges, due to the complexity of rust-lang bootstrapping system and the high
   5 rate of language changes still ongoing.
   6
   7 We try to describe here inner packaging details and the reasons behind them.
   8
   9 If you are looking to help maintain this package, be sure to read the "Notes
  10 for package maintainers" section further below.
  11
  12
  13 Embedded libraries
  14 ==================
  15
  16 The upstream source package embeds many external libraries. We make a great
  17 effort to remove them and use system versions where possible, but there are a
  18 few more remaining:
  19
  20  * compiler-rt from https://github.com/rust-lang/compiler-rt
  21    -> system-wide compiler-rt fails during linkage
  22
  23    Bug reported upstream, still to be fixed, see:
  24     - https://github.com/rust-lang/rust/issues/15054
  25     - https://github.com/rust-lang/rust/issues/15708
  26
  27  * vendor/backtrace-sys, vendor/dlmalloc, vendor/walkdir
  28
  29    These are small C libraries designed to be statically linked; their upstream
  30    does not support building them as a shared library and they are too small to
  31    justify their own Debian package.
  32
  33
  34 Building from source
  35 ====================
  36
  37 The Debian rustc package will use the system rustc to bootstrap itself from.
  38 The system rustc has to be either the previous or the same version as the rustc
  39 being built; the build will fail if this is not the case.
  40
  41     sudo apt-get build-dep ./
  42     dpkg-buildpackage
  43     # Or, to directly use what's in the Debian FTP archive
  44     sudo apt-get build-dep rustc
  45     apt-get source --compile rustc
  46
  47 Alternatively, you may give the "pkg.rustc.dlstage0" DEB_BUILD_PROFILE to
  48 instead use the process defined by Rust upstream. This downloads the "official"
  49 stage0 compiler for the version being built from rust-lang.org. At the time of
  50 writing "official" means "the previous stable version".
  51
  52     sudo apt-get build-dep -P pkg.rustc.dlstage0 ./
  53     dpkg-buildpackage
  54     # Or, to directly use what's in the Debian FTP archive
  55     sudo apt-get build-dep -P pkg.rustc.dlstage0 rustc
  56     apt-get source --compile -P pkg.rustc.dlstage0 rustc
  57
  58 After [1] is fixed, both of these should in theory give identical results.
  59
  60 If neither of these options are acceptable to you, e.g. because your distro
  61 does not have rustc already and your build process cannot access the network,
  62 see "Bootstrapping" below.
  63
  64 [1] https://github.com/rust-lang/rust/issues/34902
  65
  66
  67 Bootstrapping
  68 =============
  69
  70 To bootstrap rustc on a distro that does not have it or cargo available on any
  71 architecture (so cross-compiling is not an option) you can run `debian/rules
  72 source_orig-stage0`. This creates a .dsc that does not Build-Depend on rustc or
  73 cargo. Instead, it includes an extra orig-stage0 source tarball that contains
  74 the official stage0 compiler, pre-downloaded from rust-lang.org so that your
  75 build daemons don't need to access the network during the build.
  76
  77     debian/rules source_orig-stage0
  78     # Follow the final manual instructions that it outputs. Then:
  79     sbuild ../rustc_*.dsc && dput ../rustc_*.dsc
  80
  81 To only bootstrap specific architectures, run this instead:
  82
  83     upstream_bootstrap_arch="arm64 armhf" debian/rules source_orig-stage0
  84
  85 This way, other architectures will be omitted from the orig-stage0 tarball. You
  86 might want to do this e.g. if these other architectures are already present in
  87 your distro, but the $upstream_bootstrap_arch ones are not yet present.
  88
  89 Notes
  90 -----
  91
  92 The approach bundles the upstream bootstrapping binaries inside the Debian
  93 source package. This is a nasty hack that stretches the definition of "source
  94 package", but has a few advantages explained below.
  95
  96 The traditional Debian way of bootstrapping compilers - and other distros have
  97 similar approaches - is some variant of the following:
  98
  99 1. A developer locally installs some upstream bootstrapping binaries.
 100 2. They locally build a Debian package, using these binaries as undeclared
 101    build dependencies.
 102 3. They upload these binary packages to Debian, which can be used as declared
 103    Build-Depends in the future, including by the same package.
 104
 105 The problem with this is, Debian does not have any policy nor infrastructure
 106 that can try to reproduce what this developer supposedly did.
 107
 108 Using bootstrapping binary blobs *at some point of the process* is unavoidable.
 109 Rather than pretending we didn't do this, it is better to record *which blobs*
 110 we used, so it can be audited later. If we bundle non-Debian build-dependencies
 111 inside the source package, then we can do a *source-only upload*, and the
 112 building of the binary packages can be done by the normal build infrastructure.
 113
 114 If the build process is reproducible [1] then we can be sure that *you* (as the
 115 developer that prepared the source-only upload) didn't backdoor the binaries,
 116 nor did the build daemons even if they were compromised during the build.
 117
 118 The bootstrapping binaries may still have been backdoored, but this is true in
 119 both scenarios. So our arrangement is still a strict improvement in security,
 120 because it reduces the set of "things that may have been backdoored". Also,
 121 more people use the upstream binaries than the "magical original Debian
 122 package", so backdoors have a greater chance of being detected in the former.
 123
 124 In the long run, this process is laying the foundations for doing Diverse
 125 Double-Compilation [2], where we use *many independent* bootstrapping binaries
 126 to reproduce bit-for-bit identical output compilers, giving confidence that
 127 nothing was backdoored along the way.
 128
 129 [1] The build process for rustc is currently *not* reproducible but we're
 130     working towards it. https://github.com/rust-lang/rust/issues/34902
 131 [2] http://www.dwheeler.com/trusting-trust/
 132
 133
 134 Maintaining this package
 135 ========================
 136
 137 Import of a new upstream version
 138 --------------------------------
 139
 140 $ apt install equivs python3-magic
 141 $ sudo mk-build-deps -irt 'aptitude -R'
 142 $ uscan --verbose                       # or debian/rules source_orig-beta, for beta
 143 $ ver=UPDATE-ME                         # whatever it is, probably X.YY.Z or X.YY.Z~beta.N
 144 $ tar xf ../rustc-${ver/\~/-}-src.tar.xz && ( cd rustc-${ver/*~*/beta}-src/ && pwd && ../debian/prune-unused-deps ) && rm -rf rustc-${ver/*~*/beta}-src/
 145 # ^ If this fails, you probably need to refresh the patches used by debian/prune-unused-deps
 146 $ git commit -m "Update Files-Excluded for new upstream version ${ver/\~/-}" debian/copyright
 147 $ uscan --verbose                       # yes, again, to pick up the new Files-Excluded stuff
 148                                         # or debian/rules source_orig-beta, for beta
 149
 150 # Keep running this and follow its instructions, until it gives no output:
 151 $ debian/check-orig-suspicious.sh $ver
 152 # When you are satisfied with the above, proceed:
 153
 154 $ git checkout debian/experimental
 155 $ gbp import-orig ../rustc_$ver+dfsg1.orig.tar.xz
 156 $ dch -v $ver+dfsg1-1~exp1 "New upstream release."
 157 $ debian/rules update-version
 158 # might also need to bump the version of the cargo Build-Depends
 159 # then refresh patches, etc etc
 160 # Use /usr/share/cargo/scripts/guess-crate-copyright to help update d/copyright quickly
 161
 162 # If you need to repack again, bump the 'repacksuffix' in d/watch then run
 163 $ uscan --verbose --force-download
 164 # This will do a local repack using the new Files-Excluded rules, without
 165 # redownloading the orig tarball (despite the slightly misleading flag).
 166
 167
 168 Proceeding after build failure
 169 ------------------------------
 170
 171 If your build fails, don't run `./x.py` directly as that will detect it's being
 172 run with different settings, and run the build from scratch all over again.
 173 overwriting all intermediate files. Instead, do:
 174
 175 $ debian/rules run_rustbuild X_CMD="build|test|install" X_FLAGS="whatever"
 176
 177 Hopefully, this will directly proceed to the step that failed, without
 178 rebuilding everything in between.
 179
 180
 181 Comparing Debian rustc vs upstream rustc
 182 ----------------------------------------
 183
 184 This package does things the Debian way, which differs significantly from
 185 upstream practices. If you find a bug, you might want to check if it is present
 186 in the upstream package. Run "debian/rules debian/config.toml" to generate our
 187 config.toml that you can then use in an upstream directory **unpacked from the
 188 release tarball*. (It is more complex to get this working with their git repo.)
 189
 190 This will configure it in a "halfway" style between upstream and Debian.
 191 Specifically, it will not build LLVM nor download stuff from crates.io, yet
 192 Debian patches are *not* applied. These specific settings were chosen as a
 193 tradeoff between convenience vs being close to what upstream does - so that the
 194 chances of a bug here being a genuine upstream issue rather than a Debian bug,
 195 is much higher. Also, with the exception of LLVM, these are non-default modes
 196 *supported by* upstream so they would be happy to receive bug reports about it
 197 even if your issue only occurs here.
 198
 199 OTOH if you need to test a completely clean upstream build, including all the
 200 annoying stuff like building LLVM and downloading dependencies from crates.io,
 201 simply unpack the tarball and run `./configure && ./x.py build` etc as normal.
 202 This can be useful for confirming that an issue is caused by Debian's LLVM.
 203
 204 If you need to test a LLVM patch, do something like this:
 205
 206 # build your patched LLVM debs, then:
 207 $ mkdir -p llvm-destdir && cd llvm-destdir
 208 $ ver=4.0; VERSION=FIXME
 209 $ for i in llvm-$ver llvm-$ver-dev llvm-$ver-runtime llvm-$ver-tools libllvm$ver; do \
 210     dpkg -x ../"$i"_*${VERSION}_*.deb .; done
 211 $ cd ../rustc
 212 $ debian/rules LLVM_DESTDIR=$PWD/../llvm-destdir build
 213
 214 If you need to test a patch to the stage0 rustc, do something like this:
 215
 216 # build your patched rustc debs or upstream rustc, then:
 217 $ mkdir -p rust-destdir && cd rust-destdir
 218 $ ver=1.20; VERSION=FIXME;
 219 $ for i in rustc libstd-rust-$ver libstd-rust-dev; do \
 220     dpkg -x ../"$i"_*${VERSION}_*.deb .; done
 221 $ cd ../rustc
 222 $ debian/rules RUST_DESTDIR=$PWD/../rust-destdir build
 223
 224
 225 Useful links
 226 ------------
 227
 228 The Fedora rust team is more active than the Debian one. Here are their links:
 229
 230 Source code
 231 https://src.fedoraproject.org/rpms/rust/tree/
 232
 233 Binary packages and test logs
 234 https://kojipkgs.fedoraproject.org//packages/rust/
 235 If the same test fails both on Fedora and Debian it's a good indication that
 236 we're not Doing It Wrong and can file a valid bug upstream.