system-booting.adoc

   1 [[sysboot]]
   2 Host Bootloader
   3 ---------------
   4 ifdef::wiki[]
   5 :pve-toplevel:
   6 endif::wiki[]
   7
   8 {pve} currently uses one of two bootloaders depending on the disk setup
   9 selected in the installer.
  10
  11 For EFI Systems installed with ZFS as the root filesystem `systemd-boot` is
  12 used. All other deployments use the standard `grub` bootloader (this usually
  13 also applies to systems which are installed on top of Debian).
  14
  15
  16 [[sysboot_installer_part_scheme]]
  17 Partitioning Scheme Used by the Installer
  18 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  19
  20 The {pve} installer creates 3 partitions on all disks selected for
  21 installation.
  22
  23 The created partitions are:
  24
  25 * a 1 MB BIOS Boot Partition (gdisk type EF02)
  26
  27 * a 512 MB EFI System Partition (ESP, gdisk type EF00)
  28
  29 * a third partition spanning the set `hdsize` parameter or the remaining space
  30     used for the chosen storage type
  31
  32 Systems using ZFS as root filesystem are booted with a kernel and initrd image
  33 stored on the 512 MB EFI System Partition. For legacy BIOS systems, `grub` is
  34 used, for EFI systems `systemd-boot` is used. Both are installed and configured
  35 to point to the ESPs.
  36
  37 `grub` in BIOS mode (`--target i386-pc`) is installed onto the BIOS Boot
  38 Partition of all selected disks on all systems booted with `grub`
  39 footnote:[These are all installs with root on `ext4` or `xfs` and installs
  40 with root on ZFS on non-EFI systems].
  41
  42
  43 [[sysboot_proxmox_boot_tool]]
  44 Synchronizing the content of the ESP with `proxmox-boot-tool`
  45 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  46
  47 `proxmox-boot-tool` is a utility used to keep the contents of the EFI System
  48 Partitions properly configured and synchronized. It copies certain kernel
  49 versions to all ESPs and configures the respective bootloader to boot from
  50 the `vfat` formatted ESPs. In the context of ZFS as root filesystem this means
  51 that you can use all optional features on your root pool instead of the subset
  52 which is also present in the ZFS implementation in `grub` or having to create a
  53 separate small boot-pool footnote:[Booting ZFS on root with grub
  54 https://github.com/zfsonlinux/zfs/wiki/Debian-Stretch-Root-on-ZFS].
  55
  56 In setups with redundancy all disks are partitioned with an ESP, by the
  57 installer. This ensures the system boots even if the first boot device fails
  58 or if the BIOS can only boot from a particular disk.
  59
  60 The ESPs are not kept mounted during regular operation. This helps to prevent
  61 filesystem corruption to the `vfat` formatted ESPs in case of a system crash,
  62 and removes the need to manually adapt `/etc/fstab` in case the primary boot
  63 device fails.
  64
  65 `proxmox-boot-tool` handles the following tasks:
  66
  67 * formatting and setting up a new partition
  68 * copying and configuring new kernel images and initrd images to all listed ESPs
  69 * synchronizing the configuration on kernel upgrades and other maintenance tasks
  70 * managing the list of kernel versions which are synchronized
  71 * configuring the boot-loader to boot a particular kernel version (pinning)
  72
  73
  74 You can view the currently configured ESPs and their state by running:
  75
  76 ----
  77 # proxmox-boot-tool status
  78 ----
  79
  80 [[sysboot_proxmox_boot_setup]]
  81 .Setting up a new partition for use as synced ESP
  82
  83 To format and initialize a partition as synced ESP, e.g., after replacing a
  84 failed vdev in an rpool, or when converting an existing system that pre-dates
  85 the sync mechanism, `proxmox-boot-tool` from `pve-kernel-helpers` can be used.
  86
  87 WARNING: the `format` command will format the `<partition>`, make sure to pass
  88 in the right device/partition!
  89
  90 For example, to format an empty partition `/dev/sda2` as ESP, run the following:
  91
  92 ----
  93 # proxmox-boot-tool format /dev/sda2
  94 ----
  95
  96 To setup an existing, unmounted ESP located on `/dev/sda2` for inclusion in
  97 {pve}'s kernel update synchronization mechanism, use the following:
  98
  99 ----
 100 # proxmox-boot-tool init /dev/sda2
 101 ----
 102
 103 Afterwards `/etc/kernel/proxmox-boot-uuids` should contain a new line with the
 104 UUID of the newly added partition. The `init` command will also automatically
 105 trigger a refresh of all configured ESPs.
 106
 107 [[sysboot_proxmox_boot_refresh]]
 108 .Updating the configuration on all ESPs
 109
 110 To copy and configure all bootable kernels and keep all ESPs listed in
 111 `/etc/kernel/proxmox-boot-uuids` in sync you just need to run:
 112
 113 ----
 114 # proxmox-boot-tool refresh
 115 ----
 116 (The equivalent to running `update-grub` systems with `ext4` or `xfs` on root).
 117
 118 This is necessary should you make changes to the kernel commandline, or want to
 119 sync all kernels and initrds.
 120
 121 NOTE: Both `update-initramfs` and `apt` (when necessary) will automatically
 122 trigger a refresh.
 123
 124 .Kernel Versions considered by `proxmox-boot-tool`
 125 The following kernel versions are configured by default:
 126
 127 * the currently running kernel
 128 * the version being newly installed on package updates
 129 * the two latest already installed kernels
 130 * the latest version of the second-to-last kernel series (e.g. 5.0, 5.3), if applicable
 131 * any manually selected kernels
 132
 133 .Manually keeping a kernel bootable
 134
 135 Should you wish to add a certain kernel and initrd image to the list of
 136 bootable kernels use `proxmox-boot-tool kernel add`.
 137
 138 For example run the following to add the kernel with ABI version `5.0.15-1-pve`
 139 to the list of kernels to keep installed and synced to all ESPs:
 140
 141 ----
 142 # proxmox-boot-tool kernel add 5.0.15-1-pve
 143 ----
 144
 145 `proxmox-boot-tool kernel list` will list all kernel versions currently selected
 146 for booting:
 147
 148 ----
 149 # proxmox-boot-tool kernel list
 150 Manually selected kernels:
 151 5.0.15-1-pve
 152
 153 Automatically selected kernels:
 154 5.0.12-1-pve
 155 4.15.18-18-pve
 156 ----
 157
 158 Run `proxmox-boot-tool kernel remove` to remove a kernel from the list of
 159 manually selected kernels, for example:
 160
 161 ----
 162 # proxmox-boot-tool kernel remove 5.0.15-1-pve
 163 ----
 164
 165 NOTE: It's required to run `proxmox-boot-tool refresh` to update all EFI System
 166 Partitions (ESPs) after a manual kernel addition or removal from above.
 167
 168
 169 [[sysboot_determine_bootloader_used]]
 170 Determine which Bootloader is Used
 171 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 172
 173 [thumbnail="screenshot/boot-grub.png", float="left"]
 174
 175 The simplest and most reliable way to determine which bootloader is used, is to
 176 watch the boot process of the {pve} node.
 177
 178 You will either see the blue box of `grub` or the simple black on white
 179 `systemd-boot`.
 180
 181 [thumbnail="screenshot/boot-systemdboot.png"]
 182
 183 Determining the bootloader from a running system might not be 100% accurate. The
 184 safest way is to run the following command:
 185
 186
 187 ----
 188 # efibootmgr -v
 189 ----
 190
 191 If it returns a message that EFI variables are not supported, `grub` is used in
 192 BIOS/Legacy mode.
 193
 194 If the output contains a line that looks similar to the following, `grub` is
 195 used in UEFI mode.
 196
 197 ----
 198 Boot0005* proxmox       [...] File(\EFI\proxmox\grubx64.efi)
 199 ----
 200
 201 If the output contains a line similar to the following, `systemd-boot` is used.
 202
 203 ----
 204 Boot0006* Linux Boot Manager    [...] File(\EFI\systemd\systemd-bootx64.efi)
 205 ----
 206
 207 By running:
 208
 209 ----
 210 # proxmox-boot-tool status
 211 ----
 212
 213 you can find out if `proxmox-boot-tool` is configured, which is a good
 214 indication of how the system is booted.
 215
 216
 217 [[sysboot_grub]]
 218 Grub
 219 ~~~~
 220
 221 `grub` has been the de-facto standard for booting Linux systems for many years
 222 and is quite well documented
 223 footnote:[Grub Manual https://www.gnu.org/software/grub/manual/grub/grub.html].
 224
 225 Configuration
 226 ^^^^^^^^^^^^^
 227 Changes to the `grub` configuration are done via the defaults file
 228 `/etc/default/grub` or config snippets in `/etc/default/grub.d`. To regenerate
 229 the configuration file after a change to the configuration run:
 230 footnote:[Systems using `proxmox-boot-tool` will call `proxmox-boot-tool
 231 refresh` upon `update-grub`.]
 232
 233 ----
 234 # update-grub
 235 ----
 236
 237
 238 [[sysboot_systemd_boot]]
 239 Systemd-boot
 240 ~~~~~~~~~~~~
 241
 242 `systemd-boot` is a lightweight EFI bootloader. It reads the kernel and initrd
 243 images directly from the EFI Service Partition (ESP) where it is installed.
 244 The main advantage of directly loading the kernel from the ESP is that it does
 245 not need to reimplement the drivers for accessing the storage. In {pve}
 246 xref:sysboot_proxmox_boot_tool[`proxmox-boot-tool`] is used to keep the
 247 configuration on the ESPs synchronized.
 248
 249 [[sysboot_systemd_boot_config]]
 250 Configuration
 251 ^^^^^^^^^^^^^
 252
 253 `systemd-boot` is configured via the file `loader/loader.conf` in the root
 254 directory of an EFI System Partition (ESP). See the `loader.conf(5)` manpage
 255 for details.
 256
 257 Each bootloader entry is placed in a file of its own in the directory
 258 `loader/entries/`
 259
 260 An example entry.conf looks like this (`/` refers to the root of the ESP):
 261
 262 ----
 263 title    Proxmox
 264 version  5.0.15-1-pve
 265 options   root=ZFS=rpool/ROOT/pve-1 boot=zfs
 266 linux    /EFI/proxmox/5.0.15-1-pve/vmlinuz-5.0.15-1-pve
 267 initrd   /EFI/proxmox/5.0.15-1-pve/initrd.img-5.0.15-1-pve
 268 ----
 269
 270
 271 [[sysboot_edit_kernel_cmdline]]
 272 Editing the Kernel Commandline
 273 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 274
 275 You can modify the kernel commandline in the following places, depending on the
 276 bootloader used:
 277
 278 .Grub
 279
 280 The kernel commandline needs to be placed in the variable
 281 `GRUB_CMDLINE_LINUX_DEFAULT` in the file `/etc/default/grub`. Running
 282 `update-grub` appends its content to all `linux` entries in
 283 `/boot/grub/grub.cfg`.
 284
 285 .Systemd-boot
 286
 287 The kernel commandline needs to be placed as one line in `/etc/kernel/cmdline`.
 288 To apply your changes, run `proxmox-boot-tool refresh`, which sets it as the
 289 `option` line for all config files in `loader/entries/proxmox-*.conf`.
 290
 291
 292 [[sysboot_kernel_pin]]
 293 Override the Kernel-Version for next Boot
 294 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 295
 296 To select a kernel that is not currently the default kernel, you can either:
 297
 298 * use the boot loader menu that is displayed at the beginning of the boot
 299   process
 300 * use the `proxmox-boot-tool` to `pin` the system to a kernel version either
 301   once or permanently (until pin is reset).
 302
 303 This should help you work around incompatibilities between a newer kernel
 304 version and the hardware.
 305
 306 NOTE: Such a pin should be removed as soon as possible so that all current
 307 security patches of the latest kernel are also applied to the system.
 308
 309 For example: To permanently select the version `5.15.30-1-pve` for booting you
 310 would run:
 311
 312 ----
 313 # proxmox-boot-tool kernel pin 5.15.30-1-pve
 314 ----
 315
 316 TIP: The pinning functionality works for all {pve} systems, not only those using
 317 `proxmox-boot-tool` to synchronize the contents of the ESPs, if your system
 318 does not use `proxmox-boot-tool` for synchronizing you can also skip the
 319 `proxmox-boot-tool refresh` call in the end.
 320
 321 You can also set a kernel version to be booted on the next system boot only.
 322 This is for example useful to test if an updated kernel has resolved an issue,
 323 which caused you to `pin` a version in the first place:
 324
 325 ----
 326 # proxmox-boot-tool kernel pin 5.15.30-1-pve --next-boot
 327 ----
 328
 329 To remove any pinned version configuration use the `unpin` subcommand:
 330
 331 ----
 332 # proxmox-boot-tool kernel unpin
 333 ----
 334
 335 While `unpin` has a `--next-boot` option as well, it is used to clear a pinned
 336 version set with `--next-boot`. As that happens already automatically on boot,
 337 invonking it manually is of little use.
 338
 339 After setting, or clearing pinned versions you also need to synchronize the
 340 content and configuration on the ESPs by running the `refresh` subcommand.
 341
 342 TIP: You will be prompted to automatically do for  `proxmox-boot-tool` managed
 343 systems if you call the tool interactively.
 344
 345 ----
 346 # proxmox-boot-tool refresh
 347 ----