system-booting.adoc

   1 [[sysboot]]
   2 Host Bootloader
   3 ---------------
   4 ifdef::wiki[]
   5 :pve-toplevel:
   6 endif::wiki[]
   7
   8 {pve} currently uses one of two bootloaders depending on the disk setup
   9 selected in the installer.
  10
  11 For EFI Systems installed with ZFS as the root filesystem `systemd-boot` is
  12 used. All other deployments use the standard `grub` bootloader (this usually
  13 also applies to systems which are installed on top of Debian).
  14
  15
  16 [[sysboot_installer_part_scheme]]
  17 Partitioning Scheme Used by the Installer
  18 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  19
  20 The {pve} installer creates 3 partitions on the bootable disks selected for
  21 installation. The bootable disks are:
  22
  23 * For Installations with `ext4` or `xfs` the selected disk
  24
  25 * For ZFS installations all disks belonging to the first `vdev`:
  26 ** The first disk for RAID0
  27 ** All disks for RAID1, RAIDZ1, RAIDZ2, RAIDZ3
  28 ** The first two disks for RAID10
  29
  30 The created partitions are:
  31
  32 * a 1 MB BIOS Boot Partition (gdisk type EF02)
  33
  34 * a 512 MB EFI System Partition (ESP, gdisk type EF00)
  35
  36 * a third partition spanning the set `hdsize` parameter or the remaining space
  37     used for the chosen storage type
  38
  39 `grub` in BIOS mode (`--target i386-pc`) is installed onto the BIOS Boot
  40 Partition of all bootable disks for supporting older systems.
  41
  42
  43 [[sysboot_determine_bootloader_used]]
  44 Determine which Bootloader is Used
  45 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  46
  47 [thumbnail="screenshot/boot-grub.png", float="left"]
  48
  49 The simplest and most reliable way to determine which bootloader is used, is to
  50 watch the boot process of the {pve} node.
  51
  52 You will either see the blue box of `grub` or the simple black on white
  53 `systemd-boot`.
  54
  55 [thumbnail="screenshot/boot-systemdboot.png"]
  56
  57 Determining the bootloader from a running system might not be 100% accurate. The
  58 safest way is to run the following command:
  59
  60
  61 ----
  62 # efibootmgr -v
  63 ----
  64
  65 If it returns a message that EFI variables are not supported, `grub` is used in
  66 BIOS/Legacy mode.
  67
  68 If the output contains a line that looks similar to the following, `grub` is
  69 used in UEFI mode.
  70
  71 ----
  72 Boot0005* proxmox       [...] File(\EFI\proxmox\grubx64.efi)
  73 ----
  74
  75 If the output contains a line similar to the following, `systemd-bood` is used.
  76
  77 ----
  78 Boot0006* Linux Boot Manager    [...] File(\EFI\systemd\systemd-bootx64.efi)
  79 ----
  80
  81
  82 [[sysboot_grub]]
  83 Grub
  84 ~~~~
  85
  86 `grub` has been the de-facto standard for booting Linux systems for many years
  87 and is quite well documented
  88 footnote:[Grub Manual https://www.gnu.org/software/grub/manual/grub/grub.html].
  89
  90 The kernel and initrd images are taken from `/boot` and its configuration file
  91 `/boot/grub/grub.cfg` gets updated by the kernel installation process.
  92
  93 Configuration
  94 ^^^^^^^^^^^^^
  95 Changes to the `grub` configuration are done via the defaults file
  96 `/etc/default/grub` or config snippets in `/etc/default/grub.d`. To regenerate
  97 the `/boot/grub/grub.cfg` after a change to the configuration run:
  98
  99 ----
 100 `update-grub`.
 101 ----
 102
 103 [[sysboot_systemd_boot]]
 104 Systemd-boot
 105 ~~~~~~~~~~~~
 106
 107 `systemd-boot` is a lightweight EFI bootloader. It reads the kernel and initrd
 108 images directly from the EFI Service Partition (ESP) where it is installed.
 109 The main advantage of directly loading the kernel from the ESP is that it does
 110 not need to reimplement the drivers for accessing the storage.  In the context
 111 of ZFS as root filesystem this means that you can use all optional features on
 112 your root pool instead of the subset which is also present in the ZFS
 113 implementation in `grub` or having to create a separate small boot-pool
 114 footnote:[Booting ZFS on root with grub https://github.com/zfsonlinux/zfs/wiki/Debian-Stretch-Root-on-ZFS].
 115
 116 In setups with redundancy (RAID1, RAID10, RAIDZ*) all bootable disks (those
 117 being part of the first `vdev`) are partitioned with an ESP. This ensures the
 118 system boots even if the first boot device fails.  The ESPs are kept in sync by
 119 a kernel postinstall hook script `/etc/kernel/postinst.d/zz-pve-efiboot`. The
 120 script copies certain kernel versions and the initrd images to `EFI/proxmox/`
 121 on the root of each ESP and creates the appropriate config files in
 122 `loader/entries/proxmox-*.conf`. The `pve-efiboot-tool` script assists in
 123 managing both the synced ESPs themselves and their contents.
 124
 125 The following kernel versions are configured by default:
 126
 127 * the currently running kernel
 128 * the version being newly installed on package updates
 129 * the two latest already installed kernels
 130 * the latest version of the second-to-last kernel series (e.g. 4.15, 5.0), if applicable
 131 * any manually selected kernels (see below)
 132
 133 The ESPs are not kept mounted during regular operation, in contrast to `grub`,
 134 which keeps an ESP mounted on `/boot/efi`. This helps to prevent filesystem
 135 corruption to the `vfat` formatted ESPs in case of a system crash, and removes
 136 the need to manually adapt `/etc/fstab` in case the primary boot device fails.
 137
 138 [[sysboot_systemd_boot_config]]
 139 Configuration
 140 ^^^^^^^^^^^^^
 141
 142 `systemd-boot` is configured via the file `loader/loader.conf` in the root
 143 directory of an EFI System Partition (ESP). See the `loader.conf(5)` manpage
 144 for details.
 145
 146 Each bootloader entry is placed in a file of its own in the directory
 147 `loader/entries/`
 148
 149 An example entry.conf looks like this (`/` refers to the root of the ESP):
 150
 151 ----
 152 title    Proxmox
 153 version  5.0.15-1-pve
 154 options   root=ZFS=rpool/ROOT/pve-1 boot=zfs
 155 linux    /EFI/proxmox/5.0.15-1-pve/vmlinuz-5.0.15-1-pve
 156 initrd   /EFI/proxmox/5.0.15-1-pve/initrd.img-5.0.15-1-pve
 157 ----
 158
 159
 160 .Manually keeping a kernel bootable
 161
 162 Should you wish to add a certain kernel and initrd image to the list of
 163 bootable kernels use `pve-efiboot-tool kernel add`.
 164
 165 For example run the following to add the kernel with ABI version `5.0.15-1-pve`
 166 to the list of kernels to keep installed and synced to all ESPs:
 167
 168 ----
 169 pve-efiboot-tool kernel add 5.0.15-1-pve
 170 ----
 171
 172 `pve-efiboot-tool kernel list` will list all kernel versions currently selected
 173 for booting:
 174
 175 ----
 176 # pve-efiboot-tool kernel list
 177 Manually selected kernels:
 178 5.0.15-1-pve
 179
 180 Automatically selected kernels:
 181 5.0.12-1-pve
 182 4.15.18-18-pve
 183 ----
 184
 185 Run `pve-efiboot-tool remove` to remove a kernel from the list of manually
 186 selected kernels, for example:
 187
 188 ----
 189 pve-efiboot-tool kernel remove 5.0.15-1-pve
 190 ----
 191
 192 NOTE: It's required to run `pve-efiboot-tool refresh` to update all EFI System
 193 Partitions (ESPs) after a manual kernel addition or removal from above.
 194
 195 [[sysboot_systemd_boot_setup]]
 196 .Setting up a new partition for use as synced ESP
 197
 198 To format and initialize a partition as synced ESP, e.g., after replacing a
 199 failed vdev in an rpool, or when converting an existing system that pre-dates
 200 the sync mechanism, `pve-efiboot-tool` from `pve-kernel-helpers` can be used.
 201
 202 WARNING: the `format` command will format the `<partition>`, make sure to pass
 203 in the right device/partition!
 204
 205 For example, to format an empty partition `/dev/sda2` as ESP, run the following:
 206
 207 ----
 208 pve-efiboot-tool format /dev/sda2
 209 ----
 210
 211 To setup an existing, unmounted ESP located on `/dev/sda2` for inclusion in
 212 {pve}'s kernel update synchronization mechanism, use the following:
 213
 214 ----
 215 pve-efiboot-tool init /dev/sda2
 216 ----
 217
 218 Afterwards `/etc/kernel/pve-efiboot-uuids` should contain a new line with the
 219 UUID of the newly added partition. The `init` command will also automatically
 220 trigger a refresh of all configured ESPs.
 221
 222 [[sysboot_systemd_boot_refresh]]
 223 .Updating the configuration on all ESPs
 224
 225 To copy and configure all bootable kernels and keep all ESPs listed in
 226 `/etc/kernel/pve-efiboot-uuids` in sync you just need to run:
 227
 228 ----
 229  pve-efiboot-tool refresh
 230 ----
 231 (The equivalent to running `update-grub` on systems being booted with `grub`).
 232
 233 This is necessary should you make changes to the kernel commandline, or want to
 234 sync all kernels and initrds.
 235
 236 NOTE: Both `update-initramfs` and `apt` (when necessary) will automatically
 237 trigger a refresh.
 238
 239
 240 [[sysboot_edit_kernel_cmdline]]
 241 Editing the Kernel Commandline
 242 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 243
 244 You can modify the kernel commandline in the following places, depending on the
 245 bootloader used:
 246
 247 .Grub
 248
 249 The kernel commandline needs to be placed in the variable
 250 `GRUB_CMDLINE_LINUX_DEFAULT` in the file `/etc/default/grub`. Running
 251 `update-grub` appends its content to all `linux` entries in
 252 `/boot/grub/grub.cfg`.
 253
 254 .Systemd-boot
 255
 256 The kernel commandline needs to be placed as one line in `/etc/kernel/cmdline`.
 257 To apply your changes, run `pve-efiboot-tool refresh`, which sets it as the
 258 `option` line for all config files in `loader/entries/proxmox-*.conf`.