debian/README.source

   1 Document by Ximin Luo, Luca Bruno & Sylvestre Ledru
   2
   3 This source package is unfortunately quite tricky and with several cutting
   4 edges, due to the complexity of rust-lang bootstrapping system and the high
   5 rate of language changes still ongoing.
   6
   7 We try to describe here inner packaging details and the reasons behind them.
   8
   9
  10 Import of a new upstream version
  11 ================================
  12
  13 $ uscan --verbose
  14 $ ver=UPDATE-ME # whatever it is, X.YY.0 probably
  15 $ tar xf ../rustc-$ver-src.tar.gz && ( cd rustc-$ver-src/ && ../debian/prune-unused-deps ) && rm -rf rustc-$ver-src/
  16 $ git commit -m "Update Files-Excluded for new upstream version $ver" debian/copyright
  17 $ uscan --verbose # yes, again, to pick up the new Files-Excluded stuff
  18
  19 # Now `tar tf ../rustc_$ver+dfsg1.orig.tar.xz | less` and look for any extra
  20 # stuff like new embedded C libraries. If so, add them to Files-Excluded: and
  21 # run uscan again. Repeat until there's no more issues. Some good stuff to run:
  22 $ tar xf ../rustc_$ver+dfsg1.orig.tar.xz && cd rustc-$ver-src/
  23 $ find . -type f -and -not -name '.cargo-checksum.json' -exec file '{}' \; | \
  24     sed -e 's/\btext\b\(.*\), with very long lines/verylongtext\1/g' | \
  25     grep -v '\b\(text\|empty\)\b' || true
  26 $ find src/vendor/ -name '*.c' -o -name '*.js'
  27 $ suspicious-source
  28
  29 $ gbp import-orig ../rustc_$ver+dfsg1.orig.tar.xz
  30 $ dch -v $ver+dfsg1-1 -m "New upstream release."
  31 $ debian/rules update-version
  32 # might also need to bump the version of the cargo Build-Depends
  33 # then refresh patches, etc etc
  34
  35 # If you need to repack again, bump the 'repacksuffix' in d/watch then run
  36 $ uscan --verbose --force-download
  37 # This will do a local repack using the new Files-Excluded rules, without
  38 # redownloading the orig tarball (despite the slightly misleading flag).
  39
  40
  41 Comparing Debian rustc vs upstream rustc
  42 ========================================
  43
  44 This package does things the Debian way, which differs significantly from
  45 upstream practices. If you find a bug, you might want to check if it is present
  46 in the upstream package. Run "debian/rules debian/configure-upstream" to
  47 generate a script, that you can then run in an unpacked upstream directory.
  48
  49 This will configure it in a "halfway" style between upstream and Debian.
  50 Specifically, it will not build LLVM nor download stuff from crates.io, yet
  51 Debian patches are *not* applied. These specific settings were chosen as a
  52 tradeoff between convenience vs being close to what upstream does - so that the
  53 chances of a bug here being a genuine upstream issue rather than a Debian bug,
  54 is much higher. Also, with the exception of LLVM, these are non-default modes
  55 *supported by* upstream so they would be happy to receive bug reports about it
  56 even if your issue only occurs here.
  57
  58 OTOH if you need to test a completely clean upstream build, including all the
  59 annoying stuff like building LLVM and downloading dependencies from crates.io,
  60 simply unpack the tarball and run `./configure && ./x.py build` etc as normal.
  61 This can be useful for confirming that an issue is caused by Debian's LLVM.
  62
  63
  64 Embedded libraries
  65 ==================
  66
  67 This source package embeds several external libraries (foeked and managed
  68 by rust upstream as git submodules).
  69 In early stages, many more libraries were forked/emebedded but we are steadily
  70 progressing in splitting them out.
  71
  72 Here below the remaining ones, with the technical reasons.
  73
  74  * jemalloc from https://github.com/rust-lang/jemalloc
  75    -> system-wide one can't be used due to rust using a "je_" prefix.
  76
  77    This is intentional upstream design and won't change soon, see:
  78     - https://github.com/rust-lang/rust/pull/18678
  79     - http://smallcultfollowing.com/babysteps/blog/2014/11/14/allocators-in-rust/
  80
  81  * compiler-rt from https://github.com/rust-lang/compiler-rt
  82    -> system-wide compiler-rt fails during linkage
  83
  84    Bug reported upstream, still to be fixed, see:
  85     - https://github.com/rust-lang/rust/issues/15054
  86     - https://github.com/rust-lang/rust/issues/15708
  87
  88 As a summary, we plan to:
  89  * keep embedding jemalloc (probably forever)
  90  * work with upstream to fix compiler-rt linkage soon.
  91
  92  -- Sylvestre Ledru <sylvestre@debian.org>  Sat, 06 May 2017 13:26:08 +0200
  93
  94
  95 Building from source
  96 ====================
  97
  98 The Debian rustc package will use the system rustc to bootstrap itself from.
  99 The system rustc has to be either the previous or the same version as the rustc
 100 being built; the build will fail if this is not the case.
 101
 102     sudo apt-get build-dep ./
 103     dpkg-buildpackage
 104     # Or, to directly use what's in the Debian FTP archive
 105     sudo apt-get build-dep rustc
 106     apt-get source --compile rustc
 107
 108 Alternatively, you may give the "pkg.rustc.dlstage0" DEB_BUILD_PROFILE to
 109 instead use the process defined by Rust upstream. This downloads the "official"
 110 stage0 compiler for the version being built from rust-lang.org. At the time of
 111 writing "official" means "the previous stable version".
 112
 113     sudo apt-get build-dep -P pkg.rustc.dlstage0 ./
 114     dpkg-buildpackage
 115     # Or, to directly use what's in the Debian FTP archive
 116     sudo apt-get build-dep -P pkg.rustc.dlstage0 rustc
 117     apt-get source --compile -P pkg.rustc.dlstage0 rustc
 118
 119 After [1] is fixed, both of these should in theory give identical results.
 120
 121 If neither of these options are acceptable to you, e.g. because your distro
 122 does not have rustc already and your build process cannot access the network,
 123 see "Bootstrapping" below.
 124
 125 [1] https://github.com/rust-lang/rust/issues/34902
 126
 127
 128 Bootstrapping
 129 =============
 130
 131 To bootstrap rustc on a distro that does not have it or cargo available on any
 132 architecture (so cross-compiling is not an option) you can run `debian/rules
 133 source_orig-stage0`. This creates a .dsc that does not Build-Depend on rustc or
 134 cargo. Instead, it includes an extra orig-stage0 source tarball that contains
 135 the official stage0 compiler, pre-downloaded from rust-lang.org so that your
 136 build daemons don't need to access the network during the build.
 137
 138     debian/rules source_orig-stage0
 139     # Follow the final manual instructions that it outputs. Then:
 140     sbuild ../rustc_*.dsc && dput ../rustc_*.dsc
 141
 142 To only bootstrap specific architectures, run this instead:
 143
 144     upstream_bootstrap_arch="arm64 armhf" debian/rules source_orig-stage0
 145
 146 This way, other architectures will be omitted from the orig-stage0 tarball. You
 147 might want to do this e.g. if these other architectures are already present in
 148 your distro, but the $upstream_bootstrap_arch ones are not yet present.
 149
 150 Notes
 151 -----
 152
 153 The approach bundles the upstream bootstrapping binaries inside the Debian
 154 source package. This is a nasty hack that stretches the definition of "source
 155 package", but has a few advantages explained below.
 156
 157 The traditional Debian way of bootstrapping compilers - and other distros have
 158 similar approaches - is some variant of the following:
 159
 160 1. A developer locally installs some upstream bootstrapping binaries.
 161 2. They locally build a Debian package, using these binaries as undeclared
 162    build dependencies.
 163 3. They upload these binary packages to Debian, which can be used as declared
 164    Build-Depends in the future, including by the same package.
 165
 166 The problem with this is, Debian does not have any policy nor infrastructure
 167 that can try to reproduce what this developer supposedly did.
 168
 169 Using bootstrapping binary blobs *at some point of the process* is unavoidable.
 170 Rather than pretending we didn't do this, it is better to record *which blobs*
 171 we used, so it can be audited later. If we bundle non-Debian build-dependencies
 172 inside the source package, then we can do a *source-only upload*, and the
 173 building of the binary packages can be done by the normal build infrastructure.
 174
 175 If the build process is reproducible [1] then we can be sure that *you* (as the
 176 developer that prepared the source-only upload) didn't backdoor the binaries,
 177 nor did the build daemons even if they were compromised during the build.
 178
 179 The bootstrapping binaries may still have been backdoored, but this is true in
 180 both scenarios. So our arrangement is still a strict improvement in security,
 181 because it reduces the set of "things that may have been backdoored". Also,
 182 more people use the upstream binaries than the "magical original Debian
 183 package", so backdoors have a greater chance of being detected in the former.
 184
 185 In the long run, this process is laying the foundations for doing Diverse
 186 Double-Compilation [2], where we use *many independent* bootstrapping binaries
 187 to reproduce bit-for-bit identical output compilers, giving confidence that
 188 nothing was backdoored along the way.
 189
 190 [1] The build process for rustc is currently *not* reproducible but we're
 191     working towards it. https://github.com/rust-lang/rust/issues/34902
 192 [2] http://www.dwheeler.com/trusting-trust/
 193
 194
 195 Useful links
 196 ============
 197
 198 The Fedora rust team is more active than the Debian one. Here are their links:
 199
 200 Source code
 201 https://src.fedoraproject.org/cgit/rpms/rust.git/tree/
 202
 203 Binary packages and test logs
 204 https://kojipkgs.fedoraproject.org//packages/rust/
 205 If the same test fails both on Fedora and Debian it's a good indication that
 206 we're not Doing It Wrong and can file a valid bug upstream.
 207
 208 Package metadata
 209 https://admin.fedoraproject.org/pkgdb/package/rpms/rust/