README

   1 KERNEL SOURCE:
   2 ==============
   3
   4 We currently use the Ubuntu kernel sources, available from our mirror:
   5
   6  https://git.proxmox.com/?p=mirror_ubuntu-kernels.git;a=summary
   7
   8 Ubuntu will maintain those kernels till:
   9
  10  https://wiki.ubuntu.com/Kernel/Dev/ExtendedStable
  11  or
  12  https://pve.proxmox.com/pve-docs/chapter-pve-faq.html#faq-support-table
  13
  14  whatever happens to be earlier.
  15
  16
  17 Additional/Updated Modules:
  18 ---------------------------
  19
  20 - include native OpenZFS filesystem kernel modules for Linux
  21
  22   * https://github.com/zfsonlinux/
  23
  24   For licensing questions, see: http://open-zfs.org/wiki/Talk:FAQ
  25
  26
  27 BUILD
  28 =====
  29
  30 As this is packaging for the Linux kernel with some extra integrations, like
  31 ZFS, this repo cannot be handled like a plain Linux kernel git repository.
  32
  33 The actual Linux kernel source lives in a git submodule.
  34
  35 For a build you should init the submodules and then handle it like most our
  36 Debian packaging builds. If unsure you can follow this:
  37
  38 Installing Build-Dependencies
  39 -----------------------------
  40
  41 You can either just check the package metadata template `debian/control.in`
  42 and install the packages listed in the `Build-Depends` section manually
  43 (replace `debhelper-compat` with just `debhelper`) or use a more automated way
  44 described below:
  45
  46  # install base build-dependencies and helpers
  47  apt update
  48  apt install devscripts
  49
  50  # create build-directory so that we got final packaging control files from the
  51  # .in templates generated
  52  make build-dir-fresh
  53
  54  # install build-dependencies (replace BUILD-DIR with actual one)
  55  mk-build-deps -ir BUILD-DIR/debian/control
  56
  57
  58 Package Build
  59 -------------
  60
  61  # start the actual build
  62  make deb
  63
  64 For simple KConfig modifications you can adapt the list in `debian/rules` file.
  65 For quick code changes to the actual kernel code you can do them directly in
  66 the submodule/ubuntu-kernels directory, then re-create the build-directory, e.g.:
  67
  68  make clean
  69  # now build again, explicitly creating the build-dir isn't required anymore
  70  # after one has the build-dependencies already installed.
  71  make deb
  72
  73
  74 Modify-Build-Test Cycles
  75 ------------------------
  76
  77 Ideally you avoid the need for doing a full package build and just directly
  78 build linux from the ubuntu-kernels or the mainline (stable) repo with copying
  79 over a build-config of a proxmox-kernel to that as .config and then using the
  80 `make olddefconfig` target.
  81
  82 If you need full package builds you can try to make changes inside the
  83 BUILD-DIR directly and then continue build from there, e.g., using
  84 `dpkg-buildpackage -b -uc -us --no-pre-clean`. Depending on what stage you want
  85 to continue build you might need to touch, or remove some *.prepared files.
  86 Just check `debian/rules` for how kernel build progress is tracked by make.
  87
  88 SUBMODULE
  89 =========
  90
  91 We track the current upstream repository as submodule. Besides obvious
  92 advantages over tracking binary tar archives this also has some implications.
  93
  94 For building the submodule directory gets copied into build/ and a few patches
  95 get applied with the `patch` tool. From a git point-of-view, the copied
  96 directory remains clean even with extra patches applied since it does not
  97 contain a .git directory, but a reference to the (still pristine) submodule:
  98
  99 $ cat build/ubuntu-kernel/.git
 100
 101 If you mistakenly cloned the upstream repo as "normal" clone (not via the
 102 submodule mechanics) this means that you have a real .git directory with its
 103 independent objects and tracking info when copying for building, thus git
 104 operates on the copied directory - and "sees" that it was dirtied by `patch`,
 105 and thus the kernel buildsystem sees this too and will add a '+' to the version
 106 as a result. This changes the output directories for modules and other build
 107 artefacts and let's then the build fail on packaging.
 108
 109 So always ensure that you really checked it out as submodule, not as full
 110 "normal" clone. You can also explicitly set the LOCALVERSION variable to
 111 undefined with: `export LOCALVERSION= but that should only be done for test
 112 builds.
 113
 114 RELATED PACKAGES:
 115 =================
 116
 117 proxmox-ve
 118 ----------
 119
 120 top level meta package, depends on current default kernel series meta package.
 121
 122 git clone git://git.proxmox.com/git/proxmox-ve.git
 123
 124 proxmox-default-kernel
 125 ----------------------
 126
 127 Depends on default kernel and header meta package, e.g., proxmox-kernel-6.2 /
 128 proxmox-headers-6.2.
 129
 130 git clone git://git.proxmox.com/git/pve-kernel-meta.git
 131
 132 proxmox-kernel-X.Y
 133 ------------------
 134
 135 Depends on the latest kernel (or header, in case of proxmox-headers-X.Y)
 136 package within a certain series.
 137
 138 e.g., proxmox-kernel-6.2 depends on proxmox-kernel-6.2.16-6-pve
 139
 140 NOTE: Since Proxmox VE 8, based on Debian 12 Bookworm, the kernel ABI is bumped
 141 with every version bump due to module signing. Since then the meta package was
 142 pulled into the kernel repo, before that it lived in pve-kernel-meta.git.
 143
 144 pve-firmware
 145 ------------
 146
 147 Contains the firmware for all released PVE kernels.
 148
 149 git clone git://git.proxmox.com/git/pve-firmware.git
 150
 151
 152 NOTES:
 153 ======
 154
 155 ABI versions, package versions and package name:
 156 ------------------------------------------------
 157
 158 We follow debian's versioning w.r.t ABI changes:
 159
 160 https://kernel-team.pages.debian.net/kernel-handbook/ch-versions.html
 161 https://wiki.debian.org/DebianKernelABIChanges
 162
 163 The debian/rules file has a target comparing the build kernel's ABI against the
 164 version stored in the repository and indicates when an ABI bump is necessary.
 165 An ABI bump within one upstream version consists of incrementing the KREL
 166 variable in the Makefile, rebuilding the packages and running 'make abiupdate'
 167 (the 'abiupdate' target in 'Makefile' contains the steps for consistently
 168 updating the repository).
 169
 170 Watchdog blacklist
 171 ------------------
 172
 173 By default, all watchdog modules are black-listed because it is totally undefined
 174 which device is actually used for /dev/watchdog.
 175 We ship this list in /lib/modprobe.d/blacklist_proxmox-kernel-<VERSION>.conf
 176 The user typically edit /etc/modules to enable a specific watchdog device.
 177
 178 Debug kernel and modules
 179 ------------------------
 180
 181 In order to build a -dbgsym package containing an unstripped copy of the kernel
 182 image and modules, enable the 'pkg.proxmox-kernel.debug' build profile (e.g. by
 183 exporting DEB_BUILD_PROFILES='pkg.proxmox-kernel.debug'). The resulting package can
 184 be used together with 'crash'/'kdump-tools' to debug kernel crashes.
 185
 186 Note: the -dbgsym package is only valid for the proxmox-kernel packages produced by
 187 the same build. A kernel/module from a different build will likely not match,
 188 even if both builds are of the same kernel and package version.
 189
 190 Additional information
 191 ----------------------
 192
 193 We use the default configuration provided by Ubuntu, and apply
 194 the following modifications:
 195
 196 NOTE: For the exact and current list see debian/rules (PVE_CONFIG_OPTS)
 197
 198 - enable INTEL_MEI_WDT=m (to allow disabling via patch)
 199
 200 - disable CONFIG_SND_PCM_OSS (enabled by default in Ubuntu, not needed)
 201
 202 - switch CONFIG_TRANSPARENT_HUGEPAGE to MADVISE from ALWAYS
 203
 204 - enable CONFIG_CEPH_FS=m (request from user)
 205
 206 - enable common CONFIG_BLK_DEV_XXX to avoid hardware detection
 207   problems (udev, update-initramfs have serious problems without that)
 208
 209          CONFIG_BLK_DEV_SD=y
 210          CONFIG_BLK_DEV_SR=y
 211          CONFIG_BLK_DEV_DM=y
 212
 213 - compile NBD and RBD modules
 214          CONFIG_BLK_DEV_NBD=m
 215          CONFIG_BLK_DEV_RBD=m
 216
 217 - enable IBM JFS file system as module
 218   requested by users (bug #64)
 219
 220 - enable apple HFS and HFSPLUS as module
 221   requested by users
 222
 223 - enable CONFIG_BCACHE=m (requested by user)
 224
 225 - enable CONFIG_BRIDGE=y
 226   to avoid warnings on boot, e.g. that net.bridge.bridge-nf-call-iptables is an unknown key
 227
 228 - enable CONFIG_DEFAULT_SECURITY_APPARMOR
 229   We need this for lxc
 230
 231 - set CONFIG_CPU_FREQ_DEFAULT_GOV_PERFORMANCE=y
 232   because if not set, it can give some dynamic memory or cpu frequencies
 233   change, and vms can crash (mainly windows guest).
 234   see http://forum.proxmox.com/threads/18238-Windows-7-x64-VMs-crashing-randomly-during-process-termination?p=93273#post93273
 235
 236 - use 'deadline' as default scheduler
 237   This is the suggested setting for KVM. We also measure bad fsync performance with ext4 and cfq.
 238
 239 - disable CONFIG_INPUT_EVBUG
 240   Module evbug is not blacklisted on debian, so we simply disable it to avoid
 241   key-event logs (which is a big security problem)
 242
 243 - enable CONFIG_MODVERSIONS (needed for ABI tracking)
 244
 245 - switch default UNWINDER to FRAME_POINTER
 246   the recently introduced ORC_UNWINDER is not 100% stable yet, especially in combination with ZFS
 247
 248 - enable CONFIG_PAGE_TABLE_ISOLATION (Meltdown mitigation)