debian/README.source

   1 Document by Ximin Luo, Luca Bruno & Sylvestre Ledru
   2
   3 This source package is unfortunately quite tricky and with several cutting
   4 edges, due to the complexity of rust-lang bootstrapping system and the high
   5 rate of language changes still ongoing.
   6
   7 We try to describe here inner packaging details and the reasons behind them.
   8
   9 If you are looking to help maintain this package, be sure to read the "Notes
  10 for package maintainers" section further below.
  11
  12
  13 Embedded libraries
  14 ==================
  15
  16 This source package embeds several external libraries (foeked and managed
  17 by rust upstream as git submodules).
  18 In early stages, many more libraries were forked/emebedded but we are steadily
  19 progressing in splitting them out.
  20
  21 Here below the remaining ones, with the technical reasons.
  22
  23  * compiler-rt from https://github.com/rust-lang/compiler-rt
  24    -> system-wide compiler-rt fails during linkage
  25
  26    Bug reported upstream, still to be fixed, see:
  27     - https://github.com/rust-lang/rust/issues/15054
  28     - https://github.com/rust-lang/rust/issues/15708
  29
  30 As a summary, we plan to:
  31  * work with upstream to fix compiler-rt linkage soon.
  32
  33  -- Sylvestre Ledru <sylvestre@debian.org>  Sat, 06 May 2017 13:26:08 +0200
  34
  35
  36 Building from source
  37 ====================
  38
  39 The Debian rustc package will use the system rustc to bootstrap itself from.
  40 The system rustc has to be either the previous or the same version as the rustc
  41 being built; the build will fail if this is not the case.
  42
  43     sudo apt-get build-dep ./
  44     dpkg-buildpackage
  45     # Or, to directly use what's in the Debian FTP archive
  46     sudo apt-get build-dep rustc
  47     apt-get source --compile rustc
  48
  49 Alternatively, you may give the "pkg.rustc.dlstage0" DEB_BUILD_PROFILE to
  50 instead use the process defined by Rust upstream. This downloads the "official"
  51 stage0 compiler for the version being built from rust-lang.org. At the time of
  52 writing "official" means "the previous stable version".
  53
  54     sudo apt-get build-dep -P pkg.rustc.dlstage0 ./
  55     dpkg-buildpackage
  56     # Or, to directly use what's in the Debian FTP archive
  57     sudo apt-get build-dep -P pkg.rustc.dlstage0 rustc
  58     apt-get source --compile -P pkg.rustc.dlstage0 rustc
  59
  60 After [1] is fixed, both of these should in theory give identical results.
  61
  62 If neither of these options are acceptable to you, e.g. because your distro
  63 does not have rustc already and your build process cannot access the network,
  64 see "Bootstrapping" below.
  65
  66 [1] https://github.com/rust-lang/rust/issues/34902
  67
  68
  69 Bootstrapping
  70 =============
  71
  72 To bootstrap rustc on a distro that does not have it or cargo available on any
  73 architecture (so cross-compiling is not an option) you can run `debian/rules
  74 source_orig-stage0`. This creates a .dsc that does not Build-Depend on rustc or
  75 cargo. Instead, it includes an extra orig-stage0 source tarball that contains
  76 the official stage0 compiler, pre-downloaded from rust-lang.org so that your
  77 build daemons don't need to access the network during the build.
  78
  79     debian/rules source_orig-stage0
  80     # Follow the final manual instructions that it outputs. Then:
  81     sbuild ../rustc_*.dsc && dput ../rustc_*.dsc
  82
  83 To only bootstrap specific architectures, run this instead:
  84
  85     upstream_bootstrap_arch="arm64 armhf" debian/rules source_orig-stage0
  86
  87 This way, other architectures will be omitted from the orig-stage0 tarball. You
  88 might want to do this e.g. if these other architectures are already present in
  89 your distro, but the $upstream_bootstrap_arch ones are not yet present.
  90
  91 Notes
  92 -----
  93
  94 The approach bundles the upstream bootstrapping binaries inside the Debian
  95 source package. This is a nasty hack that stretches the definition of "source
  96 package", but has a few advantages explained below.
  97
  98 The traditional Debian way of bootstrapping compilers - and other distros have
  99 similar approaches - is some variant of the following:
 100
 101 1. A developer locally installs some upstream bootstrapping binaries.
 102 2. They locally build a Debian package, using these binaries as undeclared
 103    build dependencies.
 104 3. They upload these binary packages to Debian, which can be used as declared
 105    Build-Depends in the future, including by the same package.
 106
 107 The problem with this is, Debian does not have any policy nor infrastructure
 108 that can try to reproduce what this developer supposedly did.
 109
 110 Using bootstrapping binary blobs *at some point of the process* is unavoidable.
 111 Rather than pretending we didn't do this, it is better to record *which blobs*
 112 we used, so it can be audited later. If we bundle non-Debian build-dependencies
 113 inside the source package, then we can do a *source-only upload*, and the
 114 building of the binary packages can be done by the normal build infrastructure.
 115
 116 If the build process is reproducible [1] then we can be sure that *you* (as the
 117 developer that prepared the source-only upload) didn't backdoor the binaries,
 118 nor did the build daemons even if they were compromised during the build.
 119
 120 The bootstrapping binaries may still have been backdoored, but this is true in
 121 both scenarios. So our arrangement is still a strict improvement in security,
 122 because it reduces the set of "things that may have been backdoored". Also,
 123 more people use the upstream binaries than the "magical original Debian
 124 package", so backdoors have a greater chance of being detected in the former.
 125
 126 In the long run, this process is laying the foundations for doing Diverse
 127 Double-Compilation [2], where we use *many independent* bootstrapping binaries
 128 to reproduce bit-for-bit identical output compilers, giving confidence that
 129 nothing was backdoored along the way.
 130
 131 [1] The build process for rustc is currently *not* reproducible but we're
 132     working towards it. https://github.com/rust-lang/rust/issues/34902
 133 [2] http://www.dwheeler.com/trusting-trust/
 134
 135
 136 Maintaining this package
 137 ========================
 138
 139 Import of a new upstream version
 140 --------------------------------
 141
 142 # Make sure you apply the patch mentioned in #855464 to /usr/bin/mk-origtargz
 143
 144 $ sudo mk-build-deps -irt 'aptitude -R'
 145 $ uscan --verbose                       # or debian/rules source_orig-beta, for beta
 146 $ ver=UPDATE-ME                         # whatever it is, probably X.YY.Z or X.YY.Z~beta.N
 147 $ tar xf ../rustc-${ver/\~/-}-src.tar.xz && ( cd rustc-${ver/*~*/beta}-src/ && ../debian/prune-unused-deps ) && rm -rf rustc-${ver/*~*/beta}-src/
 148 # ^ If this fails, you probably need to refresh patches or edit debian/prune-unused-deps
 149 $ git commit -m "Update Files-Excluded for new upstream version ${ver/\~/-}" debian/copyright
 150 $ uscan --verbose                       # yes, again, to pick up the new Files-Excluded stuff
 151                                         # or debian/rules source_orig-beta, for beta
 152
 153 # Keep running this and follow its instructions, until it gives no output:
 154 $ debian/check-orig-suspicious.sh $ver
 155 # When you are satisfied with the above, proceed:
 156
 157 $ git checkout debian/experimental
 158 $ gbp import-orig ../rustc_$ver+dfsg1.orig.tar.xz
 159 $ dch -v $ver+dfsg1-1~exp1 "New upstream release."
 160 $ debian/rules update-version
 161 # might also need to bump the version of the cargo Build-Depends
 162 # then refresh patches, etc etc
 163 # Use /usr/share/cargo/guess-crate-copyright to help update d/copyright quickly
 164
 165 # If you need to repack again, bump the 'repacksuffix' in d/watch then run
 166 $ uscan --verbose --force-download
 167 # This will do a local repack using the new Files-Excluded rules, without
 168 # redownloading the orig tarball (despite the slightly misleading flag).
 169
 170
 171 Proceeding after build failure
 172 ------------------------------
 173
 174 If your build fails, don't run `./x.py` directly as that will detect it's being
 175 run with different settings, and run the build from scratch all over again.
 176 overwriting all intermediate files. Instead, do:
 177
 178 $ debian/rules run_rustbuild X_CMD="build|test|install" X_FLAGS="whatever"
 179
 180 Hopefully, this will directly proceed to the step that failed, without
 181 rebuilding everything in between.
 182
 183
 184 Comparing Debian rustc vs upstream rustc
 185 ----------------------------------------
 186
 187 This package does things the Debian way, which differs significantly from
 188 upstream practices. If you find a bug, you might want to check if it is present
 189 in the upstream package. Run "debian/rules debian/config.toml" to generate our
 190 config.toml that you can then use in an upstream directory **unpacked from the
 191 release tarball*. (It is more complex to get this working with their git repo.)
 192
 193 This will configure it in a "halfway" style between upstream and Debian.
 194 Specifically, it will not build LLVM nor download stuff from crates.io, yet
 195 Debian patches are *not* applied. These specific settings were chosen as a
 196 tradeoff between convenience vs being close to what upstream does - so that the
 197 chances of a bug here being a genuine upstream issue rather than a Debian bug,
 198 is much higher. Also, with the exception of LLVM, these are non-default modes
 199 *supported by* upstream so they would be happy to receive bug reports about it
 200 even if your issue only occurs here.
 201
 202 OTOH if you need to test a completely clean upstream build, including all the
 203 annoying stuff like building LLVM and downloading dependencies from crates.io,
 204 simply unpack the tarball and run `./configure && ./x.py build` etc as normal.
 205 This can be useful for confirming that an issue is caused by Debian's LLVM.
 206
 207 If you need to test a LLVM patch, do something like this:
 208
 209 # build your patched LLVM debs, then:
 210 $ mkdir -p llvm-destdir && cd llvm-destdir
 211 $ ver=4.0; VERSION=FIXME
 212 $ for i in llvm-$ver llvm-$ver-dev llvm-$ver-runtime llvm-$ver-tools libllvm$ver; do \
 213     dpkg -x ../"$i"_*${VERSION}_*.deb .; done
 214 $ cd ../rustc
 215 $ debian/rules LLVM_DESTDIR=$PWD/../llvm-destdir build
 216
 217 If you need to test a patch to the stage0 rustc, do something like this:
 218
 219 # build your patched rustc debs or upstream rustc, then:
 220 $ mkdir -p rust-destdir && cd rust-destdir
 221 $ ver=1.20; VERSION=FIXME;
 222 $ for i in rustc libstd-rust-$ver libstd-rust-dev; do \
 223     dpkg -x ../"$i"_*${VERSION}_*.deb .; done
 224 $ cd ../rustc
 225 $ debian/rules RUST_DESTDIR=$PWD/../rust-destdir build
 226
 227
 228 Useful links
 229 ------------
 230
 231 The Fedora rust team is more active than the Debian one. Here are their links:
 232
 233 Source code
 234 https://src.fedoraproject.org/cgit/rpms/rust.git/tree/
 235
 236 Binary packages and test logs
 237 https://kojipkgs.fedoraproject.org//packages/rust/
 238 If the same test fails both on Fedora and Debian it's a good indication that
 239 we're not Doing It Wrong and can file a valid bug upstream.
 240
 241 Package metadata
 242 https://admin.fedoraproject.org/pkgdb/package/rpms/rust/