rust_hacks.md

   1 # Scope of this document
   2
   3 This document aims to document some nice "hacks" and tricks to employ when packaging crates.
   4
   5 ## Getting an overview of missing crates
   6
   7 A nice tool to generate a graphical overview of a rust projects' dependency tree is `cargo debstatus`. Install it like that:
   8 `cargo install cargo-debstatus`. Then download either a release or clone the git project and `cd` into there. Run `cargo debstatus` to get a nice graph about dependencies and reverse dependencies.
   9
  10 ## Patching crates
  11 If a crate needs a) a newer dependency or b) an older dependency than the one in the archive you need to patch the crate. This is relatively common. You can also use this to patch out features in Cargo.toml or make changes to the source code.
  12
  13 Since the source is pulled from crates.io and not from github/lab, patching requires downloading the source from there. There are two ways to achieve this:
  14
  15 1) Install `cargo-download`: `cargo install cargo-download`
  16 1.1) Download the source: `cargo download foocrate version > foocrate.tar.gz`
  17 1.2) Extract the crate: `tar -xf foocrate.tar.gz`
  18 1.3) `cd foocrate-version && git init && git add . && git commit -m "d"`
  19 1.4) Edit e.g. Cargo.toml to relax the dependencies: Instead of version 0.4 of a crate bump it to 0.5 ( if that is the corresponding debian version) or down to 0.3 (if that is the debian version)
  20 1.5) Generate a patch with your changes: `git diff -p >> relax-deps.diff`
  21 1.6) `cp relax-deps.diff debcargo-conf/src/foocrate/debian`
  22 1.7) `cd debcargo-conf/src/foocrate/debian && mkdir patches`
  23 1.8) `mv relax-deps.diff patches/ && cd patches && echo relax-deps.diff >> series`
  24 1.9) `cd ../../../../`
  25 1.10) run `./update.sh foocrate` again. The patch should get applied and allow you to build against an older/newer dependency / with features disabled.
  26 1.11) Document your patches in d/changelog
  27 2) `wget http://crates.io/api/v1/crates/foocrate/version/download -O foocrate-version.tar.gz`
  28 Then follow 1.1)
  29
  30 Alternatively you can do it directly in the build directory with
  31 [quilt](https://wiki.debian.org/UsingQuilt):
  32
  33 1. In the root directory of `debcargo-conf`, `cd build/foo`
  34 2. `quilt series` to check existing patches, `quilt push -a` to test apply all
  35         of them, and `quilt pop -a` to unapply them
  36 3. `quilt new patch-name.patch` to create a new patch, `quilt edit
  37 path/to/file` to edit a file with changes saved in current patch, `quilt
  38 header -e --dep3` to add a DEP-3 patch header
  39 4. `quilt refresh` to update current patch and prevent fuzz (we don't allow
  40 fuzzes when building)
  41 5. `cp -r debian/patches ../../src/foo/debian/` to copy updated patches over
  42 6. In root directory, run `./update.sh foo` to update the build directory
  43
  44 Note that `update.sh` deletes and re-creates `build/foo`, so an open terminal
  45 in it needs to go up and down into the new directory.
  46
  47 capitol did a nice writeup which can be read here:
  48 https://blog.hackeriet.no/packaging-rust-part-II/
  49
  50
  51
  52
  53 ## Skipping tests / special d/rules overrides
  54
  55 Sometimes you get broken tests that can't be excluded or patched away. For instance if a test would require direct access to the RAM which isn't possible with `sbuild`. Then it is reasonable to skip those in `d/rules`.
  56
  57 1) `cd src/foocrate/debian && touch rules`
  58 2) `cd ../../../ && ./update.sh foocrate`. This will generate a `rules.debcargo.hint` file you can use as template, similar to d/copyright
  59 3) `cd src/foocrate/debian && cp rules.debcargo.hint rules`
  60 4) use your favorite editor to edit the rules file to your needs. For skipping tests you can do the following:
  61
  62 ```
  63 override_dh_auto_test:
  64         dh_auto_test -- test -- --skip tests::broken_test --skip tests::second_broken_test
  65 ```
  66
  67 if those are the broken tests. Then `cd` in the top-level directory and run `./update.sh` again. You might also need to skip the same tests in the autopkgtest runner. To do that, create a `tests/control` file in the `debian/` folder.
  68
  69
  70 ## Packaging binary crates
  71
  72 TODO
  73
  74 ## debcargo.toml tweaks
  75
  76 ### Excluding files
  77 In case you need to exclude certain files from `debcargo.toml`, there is an easy way to do that.
  78 Just add `excludes = ["foo/bar.rs", ""bar/non-dfsg-file"]` in debcargo.toml. This has the following usecases:
  79
  80 - Exclude non-dfsg/unnecessary files form the orig tarball (Example: svg-metadata)
  81   This is also **required** for some sys-crates to exclude vendored copies of the C library already in debian
  82 - Exclude broken tests that do not run (when in doubt, ask on #debian-rust) (Example: cxx )
  83
  84 ### Passing external packages to the buildsystem
  85
  86 If the crate needs external packages (such as -dev libraries) you can also pass those conveniently via `debcargo.toml` :
  87 ` depends = ["libfoo-dev"]  `. This is essentially needed for all -sys crates which provide rust bindings to C developement libraries such as GTK, for instance.
  88
  89 ### Passing runtime test dependencies to autopkgtest
  90
  91 In rare cases the autopkgtest can fail on the official runners compared to the local one because it has a slightly different setup.
  92 You can do this like that: `test_depends = ["foo"]`. That happens very rarely, see `cxx` for an example.
  93
  94 ### Whitelisting files
  95
  96 Sometimes debcargo marks files as suspicious, most of the time those are tests written in C for -sys crates. Whitelist them like that:
  97 `whitelist = [tests/foo.c"]` . (Example: lidadwaita-sys)
  98
  99 ### Allowing alpha/beta dependencies
 100
 101 Some crates depend on a crate with an alpha/beta version strings. debcargo will emit an error if that is the case. To allow those deps, pass the following:
 102 `allow_prerelease_deps = true`. Do this only if you are sure this will work !
 103
 104
 105 ### Collapsing features
 106 If a crate has features, `collapse_features = true` **should** be set in `debcargo.toml`. This is strongly recommended. See issue #17 in the debcargo repo for the reasoning.
 107
 108
 109 ### Marking feature tests as broken
 110
 111 Let's assume  you are building a package and get the following output at the end:
 112
 113 ```
 114
 115 autopkgtest [22:45:42]: test librust-foo-dev:: -----------------------]
 116 autopkgtest [22:45:42]: test librust-foo-dev::  - - - - - - - - - - results - - - - - - - - - -
 117 librust-foo-dev: PASS
 118 autopkgtest [22:45:42]: @@@@@@@@@@@@@@@@@@@@ summary
 119 rust-foo:@ FAIL non-zero exit status 101
 120 librust-foo-dev:default PASS
 121 librust-foo-dev: PASS
 122 librust-foo-dev:serde: FAIL non-zero exit status 101
 123
 124 ```
 125 As we can see, the `serde` and `@` feature fail. Since the other features (and the package itself) passes, we need to mark those test as flaky so the package can migrate to testing.
 126 Do it like that:
 127 ```
 128 #serde and @ feature fail the autopkgtest
 129 [packages.lib]
 130 test_is_broken = false
 131 [packages."lib+@"]
 132 test_is_broken = true
 133 [packages."lib+serde"]
 134 test_is_broken = true
 135 ```
 136 Examples: cxx, hashbrown, uom, ...
 137
 138 Do this only if *some* features fail. If all feature tests fail, read the test logs and look at the upstream test system. Make sure that some tests pass or that the tests aren't meant to be run. Some upstream projects run all test in a specific container or use specific setup. Also some crates are only tested with their default features enabled by upstream. If that's the case mark all test as broken:
 139
 140 ```
 141 # tests need a postgres container to run
 142 [packages.lib]
 143 test_is_broken = true
 144
 145 ```
 146 Examples: tokio-postgres, ...
 147
 148 Sometime you need to combine some tricks to run at least some tests: Exclude a faulty test, patch dependencies away and mark tests as broken.
 149
 150 ### More resources
 151 For a full documentation of all keywords available in `debcargo.toml` refer to debcargo.toml.example in the debcargo repo.
 152
 153
 154 ## Arch-specific failures
 155
 156 It can rarely happen that tests (read: autopkgtest) fail on specific arches because how bytes are addressed on that arch. The best course of action is to investigate first if it indeed is an arch-specific failure.
 157 If that's the case you need to write a patch that skips those faulty tests (on that arch) so the package can enter testing. Because arch names are different in rust, here is a handy table comparing them:
 158
 159
 160 | Debian arch name | rust arch name (target_arch) |
 161 |----------------------|---------------------------|
 162 | Arches autopkgtest runs on (needed for testing migration) | |
 163 | amd64            | x86-64         |
 164 | i386             | x86 |
 165 | arm64            | aarch64        |
 166 | armel            | arm        |
 167 | armhf            | arm |
 168 | ppc64el | powerpc64 |
 169 | s390x | powerpc64? |
 170 | **Other official arches¹** | |
 171 | mipsel | mips? |
 172 | mips64el | mips64 |
 173 | **Unoffical ports with rustc/cargo (not really relevant)** | |
 174 |  powerpc      | powerpc? |
 175 | ppc64         | powerpc? |
 176 | riscv64 | riscv64 |
 177 | sparc64 | sparc64? |
 178 | x32 | ? |
 179
 180 Arches without rustc/cargo:
 181 - sh4²
 182 - alpha
 183 - arc
 184 - hppa
 185 - hurd-i386
 186 - ia64
 187 - kfreebsd-amd64
 188 - kfreebsd-i386
 189 - m68k
 190
 191 Only the first seven are really relevant, I included the rest for completeness's sake.
 192 If you encounter a test failure e.g. on `armel`, add this macro before the `#[test]` macro:
 193  ` #[cfg(not(target_arch = "arm"))] `.
 194  Then generate a patch with the changes and include it in the usual way. Also notify upstream that this arch is broken and send them your patch.
 195 Footnotes:  [1] https://wiki.debian.org/SupportedArchitectures
 196             [2] sh4 has only one test failure for cargo
 197
 198