debian/README.source

   1 Document by Luca Bruno & Sylvestre Ledru
   2
   3 This source package is unfortunately quite tricky and with
   4 several cutting edges, due to the complexity of rust-lang
   5 bootstrapping system and the high rate of language changes
   6 still ongoing.
   7
   8 We try to describe here inner packaging details and the
   9 reasons behind them.
  10
  11
  12 Import of a new upstream version
  13 ================================
  14
  15 $ uscan
  16 $ gbp import-orig --upstream-branch=upstream/experimental --debian-branch=debian/experimental ../rustc_X.YY.0+dfsg1.orig.tar.xz
  17 $ dch -v X.YY.0+dfsg1-1
  18 $ debian/rules update-version
  19 # might also need to bump the version of the cargo Build-Depends
  20
  21
  22 Embedded libraries
  23 ==================
  24
  25 This source package embeds several external libraries (foeked and managed
  26 by rust upstream as git submodules).
  27 In early stages, many more libraries were forked/emebedded but we are steadily
  28 progressing in splitting them out.
  29
  30 Here below the remaining ones, with the technical reasons.
  31
  32  * jemalloc from https://github.com/rust-lang/jemalloc
  33    -> system-wide one can't be used due to rust using a "je_" prefix.
  34
  35    This is intentional upstream design and won't change soon, see:
  36     - https://github.com/rust-lang/rust/pull/18678
  37     - http://smallcultfollowing.com/babysteps/blog/2014/11/14/allocators-in-rust/
  38
  39  * compiler-rt from https://github.com/rust-lang/compiler-rt
  40    -> system-wide compiler-rt fails during linkage
  41
  42    Bug reported upstream, still to be fixed, see:
  43     - https://github.com/rust-lang/rust/issues/15054
  44     - https://github.com/rust-lang/rust/issues/15708
  45
  46 As a summary, we plan to:
  47  * keep embedding jemalloc (probably forever)
  48  * work with upstream to fix compiler-rt linkage soon.
  49
  50  -- Sylvestre Ledru <sylvestre@debian.org>  Sat, 06 May 2017 13:26:08 +0200
  51
  52
  53 Building from source
  54 ====================
  55
  56 The Debian rustc package will use the system rustc to bootstrap itself from.
  57 The system rustc has to be either the previous or the same version as the rustc
  58 being built; the build will fail if this is not the case.
  59
  60     sudo apt-get build-dep ./
  61     dpkg-buildpackage
  62     # Or, to directly use what's in the Debian FTP archive
  63     sudo apt-get build-dep rustc
  64     apt-get source --compile rustc
  65
  66 Alternatively, you may give the "pkg.rustc.dlstage0" DEB_BUILD_PROFILE to
  67 instead use the process defined by Rust upstream. This downloads the "official"
  68 stage0 compiler for the version being built from rust-lang.org. At the time of
  69 writing "official" means "the previous stable version".
  70
  71     sudo apt-get build-dep -P pkg.rustc.dlstage0 ./
  72     dpkg-buildpackage
  73     # Or, to directly use what's in the Debian FTP archive
  74     sudo apt-get build-dep -P pkg.rustc.dlstage0 rustc
  75     apt-get source --compile -P pkg.rustc.dlstage0 rustc
  76
  77 After [1] is fixed, both of these should in theory give identical results.
  78
  79 If neither of these options are acceptable to you, e.g. because your distro
  80 does not have rustc already and your build process cannot access the network,
  81 see "Bootstrapping" below.
  82
  83 [1] https://github.com/rust-lang/rust/issues/34902
  84
  85
  86 Bootstrapping
  87 =============
  88
  89 To bootstrap rustc on a distro that does not have it or cargo available on any
  90 architecture (so cross-compiling is not an option) you can run `debian/rules
  91 source_orig-stage0`. This creates a .dsc that does not Build-Depend on rustc or
  92 cargo. Instead, it includes an extra orig-stage0 source tarball that contains
  93 the official stage0 compiler, pre-downloaded from rust-lang.org so that your
  94 build daemons don't need to access the network during the build.
  95
  96     debian/rules source_orig-stage0
  97     # Follow the final manual instructions that it outputs. Then:
  98     sbuild ../rustc_*.dsc && dput ../rustc_*.dsc
  99
 100 To only bootstrap specific architectures, run this instead:
 101
 102     upstream_bootstrap_arch="arm64 armhf" debian/rules source_orig-stage0
 103
 104 This way, other architectures will be omitted from the orig-stage0 tarball. You
 105 might want to do this e.g. if these other architectures are already present in
 106 your distro, but the $upstream_bootstrap_arch ones are not yet present.
 107
 108 Notes
 109 -----
 110
 111 The approach bundles the upstream bootstrapping binaries inside the Debian
 112 source package. This is a nasty hack that stretches the definition of "source
 113 package", but has a few advantages explained below.
 114
 115 The traditional Debian way of bootstrapping compilers - and other distros have
 116 similar approaches - is some variant of the following:
 117
 118 1. A developer locally installs some upstream bootstrapping binaries.
 119 2. They locally build a Debian package, using these binaries as undeclared
 120    build dependencies.
 121 3. They upload these binary packages to Debian, which can be used as declared
 122    Build-Depends in the future, including by the same package.
 123
 124 The problem with this is, Debian does not have any policy nor infrastructure
 125 that can try to reproduce what this developer supposedly did.
 126
 127 Using bootstrapping binary blobs *at some point of the process* is unavoidable.
 128 Rather than pretending we didn't do this, it is better to record *which blobs*
 129 we used, so it can be audited later. If we bundle non-Debian build-dependencies
 130 inside the source package, then we can do a *source-only upload*, and the
 131 building of the binary packages can be done by the normal build infrastructure.
 132
 133 If the build process is reproducible [1] then we can be sure that *you* (as the
 134 developer that prepared the source-only upload) didn't backdoor the binaries,
 135 nor did the build daemons even if they were compromised during the build.
 136
 137 The bootstrapping binaries may still have been backdoored, but this is true in
 138 both scenarios. So our arrangement is still a strict improvement in security,
 139 because it reduces the set of "things that may have been backdoored". Also,
 140 more people use the upstream binaries than the "magical original Debian
 141 package", so backdoors have a greater chance of being detected in the former.
 142
 143 In the long run, this process is laying the foundations for doing Diverse
 144 Double-Compilation [2], where we use *many independent* bootstrapping binaries
 145 to reproduce bit-for-bit identical output compilers, giving confidence that
 146 nothing was backdoored along the way.
 147
 148 [1] The build process for rustc is currently *not* reproducible but we're
 149     working towards it. https://github.com/rust-lang/rust/issues/34902
 150 [2] http://www.dwheeler.com/trusting-trust/