system-booting.adoc

   1 [[sysboot]]
   2 Host Bootloader
   3 ---------------
   4 ifdef::wiki[]
   5 :pve-toplevel:
   6 endif::wiki[]
   7
   8 {pve} currently uses one of two bootloaders depending on the disk setup
   9 selected in the installer.
  10
  11 For EFI Systems installed with ZFS as the root filesystem `systemd-boot` is
  12 used. All other deployments use the standard `grub` bootloader (this usually
  13 also applies to systems which are installed on top of Debian).
  14
  15
  16 [[sysboot_installer_part_scheme]]
  17 Partitioning Scheme Used by the Installer
  18 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  19
  20 The {pve} installer creates 3 partitions on all disks selected for
  21 installation.
  22
  23 The created partitions are:
  24
  25 * a 1 MB BIOS Boot Partition (gdisk type EF02)
  26
  27 * a 512 MB EFI System Partition (ESP, gdisk type EF00)
  28
  29 * a third partition spanning the set `hdsize` parameter or the remaining space
  30     used for the chosen storage type
  31
  32 Systems using ZFS as root filesystem are booted with a kernel and initrd image
  33 stored on the 512 MB EFI System Partition. For legacy BIOS systems, `grub` is
  34 used, for EFI systems `systemd-boot` is used. Both are installed and configured
  35 to point to the ESPs.
  36
  37 `grub` in BIOS mode (`--target i386-pc`) is installed onto the BIOS Boot
  38 Partition of all selected disks on all systems booted with `grub`
  39 footnote:[These are all installs with root on `ext4` or `xfs` and installs
  40 with root on ZFS on non-EFI systems].
  41
  42
  43 [[sysboot_proxmox_boot_tool]]
  44 Synchronizing the content of the ESP with `proxmox-boot-tool`
  45 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  46
  47 `proxmox-boot-tool` is a utility used to keep the contents of the EFI System
  48 Partitions properly configured and synchronized. It copies certain kernel
  49 versions to all ESPs and configures the respective bootloader to boot from
  50 the `vfat` formatted ESPs. In the context of ZFS as root filesystem this means
  51 that you can use all optional features on your root pool instead of the subset
  52 which is also present in the ZFS implementation in `grub` or having to create a
  53 separate small boot-pool footnote:[Booting ZFS on root with grub
  54 https://github.com/zfsonlinux/zfs/wiki/Debian-Stretch-Root-on-ZFS].
  55
  56 In setups with redundancy all disks are partitioned with an ESP, by the
  57 installer. This ensures the system boots even if the first boot device fails
  58 or if the BIOS can only boot from a particular disk.
  59
  60 The ESPs are not kept mounted during regular operation. This helps to prevent
  61 filesystem corruption to the `vfat` formatted ESPs in case of a system crash,
  62 and removes the need to manually adapt `/etc/fstab` in case the primary boot
  63 device fails.
  64
  65 `proxmox-boot-tool` handles the following tasks:
  66
  67 * formatting and setting up a new partition
  68 * copying and configuring new kernel images and initrd images to all listed ESPs
  69 * synchronizing the configuration on kernel upgrades and other maintenance tasks
  70 * managing the list of kernel versions which are synchronized
  71
  72
  73 You can view the currently configured ESPs and their state by running:
  74
  75 ----
  76 # proxmox-boot-tool status
  77 ----
  78
  79 [[sysboot_proxmox_boot_setup]]
  80 .Setting up a new partition for use as synced ESP
  81
  82 To format and initialize a partition as synced ESP, e.g., after replacing a
  83 failed vdev in an rpool, or when converting an existing system that pre-dates
  84 the sync mechanism, `proxmox-boot-tool` from `pve-kernel-helpers` can be used.
  85
  86 WARNING: the `format` command will format the `<partition>`, make sure to pass
  87 in the right device/partition!
  88
  89 For example, to format an empty partition `/dev/sda2` as ESP, run the following:
  90
  91 ----
  92 # proxmox-boot-tool format /dev/sda2
  93 ----
  94
  95 To setup an existing, unmounted ESP located on `/dev/sda2` for inclusion in
  96 {pve}'s kernel update synchronization mechanism, use the following:
  97
  98 ----
  99 # proxmox-boot-tool init /dev/sda2
 100 ----
 101
 102 Afterwards `/etc/kernel/proxmox-boot-uuids` should contain a new line with the
 103 UUID of the newly added partition. The `init` command will also automatically
 104 trigger a refresh of all configured ESPs.
 105
 106 [[sysboot_proxmox_boot_refresh]]
 107 .Updating the configuration on all ESPs
 108
 109 To copy and configure all bootable kernels and keep all ESPs listed in
 110 `/etc/kernel/proxmox-boot-uuids` in sync you just need to run:
 111
 112 ----
 113 # proxmox-boot-tool refresh
 114 ----
 115 (The equivalent to running `update-grub` systems with `ext4` or `xfs` on root).
 116
 117 This is necessary should you make changes to the kernel commandline, or want to
 118 sync all kernels and initrds.
 119
 120 NOTE: Both `update-initramfs` and `apt` (when necessary) will automatically
 121 trigger a refresh.
 122
 123 .Kernel Versions considered by `proxmox-boot-tool`
 124 The following kernel versions are configured by default:
 125
 126 * the currently running kernel
 127 * the version being newly installed on package updates
 128 * the two latest already installed kernels
 129 * the latest version of the second-to-last kernel series (e.g. 5.0, 5.3), if applicable
 130 * any manually selected kernels
 131
 132 .Manually keeping a kernel bootable
 133
 134 Should you wish to add a certain kernel and initrd image to the list of
 135 bootable kernels use `proxmox-boot-tool kernel add`.
 136
 137 For example run the following to add the kernel with ABI version `5.0.15-1-pve`
 138 to the list of kernels to keep installed and synced to all ESPs:
 139
 140 ----
 141 # proxmox-boot-tool kernel add 5.0.15-1-pve
 142 ----
 143
 144 `proxmox-boot-tool kernel list` will list all kernel versions currently selected
 145 for booting:
 146
 147 ----
 148 # proxmox-boot-tool kernel list
 149 Manually selected kernels:
 150 5.0.15-1-pve
 151
 152 Automatically selected kernels:
 153 5.0.12-1-pve
 154 4.15.18-18-pve
 155 ----
 156
 157 Run `proxmox-boot-tool kernel remove` to remove a kernel from the list of
 158 manually selected kernels, for example:
 159
 160 ----
 161 # proxmox-boot-tool kernel remove 5.0.15-1-pve
 162 ----
 163
 164 NOTE: It's required to run `proxmox-boot-tool refresh` to update all EFI System
 165 Partitions (ESPs) after a manual kernel addition or removal from above.
 166
 167
 168 [[sysboot_determine_bootloader_used]]
 169 Determine which Bootloader is Used
 170 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 171
 172 [thumbnail="screenshot/boot-grub.png", float="left"]
 173
 174 The simplest and most reliable way to determine which bootloader is used, is to
 175 watch the boot process of the {pve} node.
 176
 177 You will either see the blue box of `grub` or the simple black on white
 178 `systemd-boot`.
 179
 180 [thumbnail="screenshot/boot-systemdboot.png"]
 181
 182 Determining the bootloader from a running system might not be 100% accurate. The
 183 safest way is to run the following command:
 184
 185
 186 ----
 187 # efibootmgr -v
 188 ----
 189
 190 If it returns a message that EFI variables are not supported, `grub` is used in
 191 BIOS/Legacy mode.
 192
 193 If the output contains a line that looks similar to the following, `grub` is
 194 used in UEFI mode.
 195
 196 ----
 197 Boot0005* proxmox       [...] File(\EFI\proxmox\grubx64.efi)
 198 ----
 199
 200 If the output contains a line similar to the following, `systemd-boot` is used.
 201
 202 ----
 203 Boot0006* Linux Boot Manager    [...] File(\EFI\systemd\systemd-bootx64.efi)
 204 ----
 205
 206 By running:
 207
 208 ----
 209 # proxmox-boot-tool status
 210 ----
 211
 212 you can find out if `proxmox-boot-tool` is configured, which is a good
 213 indication of how the system is booted.
 214
 215
 216 [[sysboot_grub]]
 217 Grub
 218 ~~~~
 219
 220 `grub` has been the de-facto standard for booting Linux systems for many years
 221 and is quite well documented
 222 footnote:[Grub Manual https://www.gnu.org/software/grub/manual/grub/grub.html].
 223
 224 Configuration
 225 ^^^^^^^^^^^^^
 226 Changes to the `grub` configuration are done via the defaults file
 227 `/etc/default/grub` or config snippets in `/etc/default/grub.d`. To regenerate
 228 the configuration file after a change to the configuration run:
 229 footnote:[Systems using `proxmox-boot-tool` will call `proxmox-boot-tool
 230 refresh` upon `update-grub`.]
 231
 232 ----
 233 # update-grub
 234 ----
 235
 236
 237 [[sysboot_systemd_boot]]
 238 Systemd-boot
 239 ~~~~~~~~~~~~
 240
 241 `systemd-boot` is a lightweight EFI bootloader. It reads the kernel and initrd
 242 images directly from the EFI Service Partition (ESP) where it is installed.
 243 The main advantage of directly loading the kernel from the ESP is that it does
 244 not need to reimplement the drivers for accessing the storage. In {pve}
 245 xref:sysboot_proxmox_boot_tool[`proxmox-boot-tool`] is used to keep the
 246 configuration on the ESPs synchronized.
 247
 248 [[sysboot_systemd_boot_config]]
 249 Configuration
 250 ^^^^^^^^^^^^^
 251
 252 `systemd-boot` is configured via the file `loader/loader.conf` in the root
 253 directory of an EFI System Partition (ESP). See the `loader.conf(5)` manpage
 254 for details.
 255
 256 Each bootloader entry is placed in a file of its own in the directory
 257 `loader/entries/`
 258
 259 An example entry.conf looks like this (`/` refers to the root of the ESP):
 260
 261 ----
 262 title    Proxmox
 263 version  5.0.15-1-pve
 264 options   root=ZFS=rpool/ROOT/pve-1 boot=zfs
 265 linux    /EFI/proxmox/5.0.15-1-pve/vmlinuz-5.0.15-1-pve
 266 initrd   /EFI/proxmox/5.0.15-1-pve/initrd.img-5.0.15-1-pve
 267 ----
 268
 269
 270 [[sysboot_edit_kernel_cmdline]]
 271 Editing the Kernel Commandline
 272 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 273
 274 You can modify the kernel commandline in the following places, depending on the
 275 bootloader used:
 276
 277 .Grub
 278
 279 The kernel commandline needs to be placed in the variable
 280 `GRUB_CMDLINE_LINUX_DEFAULT` in the file `/etc/default/grub`. Running
 281 `update-grub` appends its content to all `linux` entries in
 282 `/boot/grub/grub.cfg`.
 283
 284 .Systemd-boot
 285
 286 The kernel commandline needs to be placed as one line in `/etc/kernel/cmdline`.
 287 To apply your changes, run `proxmox-boot-tool refresh`, which sets it as the
 288 `option` line for all config files in `loader/entries/proxmox-*.conf`.