8 {pve} currently uses one of two bootloaders depending on the disk setup
9 selected in the installer.
11 For EFI Systems installed with ZFS as the root filesystem `systemd-boot` is
12 used. All other deployments use the standard `grub` bootloader (this usually
13 also applies to systems which are installed on top of Debian).
16 [[sysboot_installer_part_scheme]]
17 Partitioning Scheme Used by the Installer
18 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
20 The {pve} installer creates 3 partitions on all disks selected for
23 The created partitions are:
25 * a 1 MB BIOS Boot Partition (gdisk type EF02)
27 * a 512 MB EFI System Partition (ESP, gdisk type EF00)
29 * a third partition spanning the set `hdsize` parameter or the remaining space
30 used for the chosen storage type
32 Systems using ZFS as root filesystem are booted with a kernel and initrd image
33 stored on the 512 MB EFI System Partition. For legacy BIOS systems, `grub` is
34 used, for EFI systems `systemd-boot` is used. Both are installed and configured
37 `grub` in BIOS mode (`--target i386-pc`) is installed onto the BIOS Boot
38 Partition of all selected disks on all systems booted with `grub`
39 footnote:[These are all installs with root on `ext4` or `xfs` and installs
40 with root on ZFS on non-EFI systems].
43 [[sysboot_proxmox_boot_tool]]
44 Synchronizing the content of the ESP with `proxmox-boot-tool`
45 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
47 `proxmox-boot-tool` is a utility used to keep the contents of the EFI System
48 Partitions properly configured and synchronized. It copies certain kernel
49 versions to all ESPs and configures the respective bootloader to boot from
50 the `vfat` formatted ESPs. In the context of ZFS as root filesystem this means
51 that you can use all optional features on your root pool instead of the subset
52 which is also present in the ZFS implementation in `grub` or having to create a
53 separate small boot-pool footnote:[Booting ZFS on root with grub
54 https://github.com/zfsonlinux/zfs/wiki/Debian-Stretch-Root-on-ZFS].
56 In setups with redundancy all disks are partitioned with an ESP, by the
57 installer. This ensures the system boots even if the first boot device fails
58 or if the BIOS can only boot from a particular disk.
60 The ESPs are not kept mounted during regular operation. This helps to prevent
61 filesystem corruption to the `vfat` formatted ESPs in case of a system crash,
62 and removes the need to manually adapt `/etc/fstab` in case the primary boot
65 `proxmox-boot-tool` handles the following tasks:
67 * formatting and setting up a new partition
68 * copying and configuring new kernel images and initrd images to all listed ESPs
69 * synchronizing the configuration on kernel upgrades and other maintenance tasks
70 * managing the list of kernel versions which are synchronized
71 * configuring the boot-loader to boot a particular kernel version (pinning)
74 You can view the currently configured ESPs and their state by running:
77 # proxmox-boot-tool status
80 [[sysboot_proxmox_boot_setup]]
81 .Setting up a new partition for use as synced ESP
83 To format and initialize a partition as synced ESP, e.g., after replacing a
84 failed vdev in an rpool, or when converting an existing system that pre-dates
85 the sync mechanism, `proxmox-boot-tool` from `pve-kernel-helpers` can be used.
87 WARNING: the `format` command will format the `<partition>`, make sure to pass
88 in the right device/partition!
90 For example, to format an empty partition `/dev/sda2` as ESP, run the following:
93 # proxmox-boot-tool format /dev/sda2
96 To setup an existing, unmounted ESP located on `/dev/sda2` for inclusion in
97 {pve}'s kernel update synchronization mechanism, use the following:
100 # proxmox-boot-tool init /dev/sda2
103 Afterwards `/etc/kernel/proxmox-boot-uuids` should contain a new line with the
104 UUID of the newly added partition. The `init` command will also automatically
105 trigger a refresh of all configured ESPs.
107 [[sysboot_proxmox_boot_refresh]]
108 .Updating the configuration on all ESPs
110 To copy and configure all bootable kernels and keep all ESPs listed in
111 `/etc/kernel/proxmox-boot-uuids` in sync you just need to run:
114 # proxmox-boot-tool refresh
116 (The equivalent to running `update-grub` systems with `ext4` or `xfs` on root).
118 This is necessary should you make changes to the kernel commandline, or want to
119 sync all kernels and initrds.
121 NOTE: Both `update-initramfs` and `apt` (when necessary) will automatically
124 .Kernel Versions considered by `proxmox-boot-tool`
125 The following kernel versions are configured by default:
127 * the currently running kernel
128 * the version being newly installed on package updates
129 * the two latest already installed kernels
130 * the latest version of the second-to-last kernel series (e.g. 5.0, 5.3), if applicable
131 * any manually selected kernels
133 .Manually keeping a kernel bootable
135 Should you wish to add a certain kernel and initrd image to the list of
136 bootable kernels use `proxmox-boot-tool kernel add`.
138 For example run the following to add the kernel with ABI version `5.0.15-1-pve`
139 to the list of kernels to keep installed and synced to all ESPs:
142 # proxmox-boot-tool kernel add 5.0.15-1-pve
145 `proxmox-boot-tool kernel list` will list all kernel versions currently selected
149 # proxmox-boot-tool kernel list
150 Manually selected kernels:
153 Automatically selected kernels:
158 Run `proxmox-boot-tool kernel remove` to remove a kernel from the list of
159 manually selected kernels, for example:
162 # proxmox-boot-tool kernel remove 5.0.15-1-pve
165 NOTE: It's required to run `proxmox-boot-tool refresh` to update all EFI System
166 Partitions (ESPs) after a manual kernel addition or removal from above.
169 [[sysboot_determine_bootloader_used]]
170 Determine which Bootloader is Used
171 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
173 [thumbnail="screenshot/boot-grub.png", float="left"]
175 The simplest and most reliable way to determine which bootloader is used, is to
176 watch the boot process of the {pve} node.
178 You will either see the blue box of `grub` or the simple black on white
181 [thumbnail="screenshot/boot-systemdboot.png"]
183 Determining the bootloader from a running system might not be 100% accurate. The
184 safest way is to run the following command:
191 If it returns a message that EFI variables are not supported, `grub` is used in
194 If the output contains a line that looks similar to the following, `grub` is
198 Boot0005* proxmox [...] File(\EFI\proxmox\grubx64.efi)
201 If the output contains a line similar to the following, `systemd-boot` is used.
204 Boot0006* Linux Boot Manager [...] File(\EFI\systemd\systemd-bootx64.efi)
210 # proxmox-boot-tool status
213 you can find out if `proxmox-boot-tool` is configured, which is a good
214 indication of how the system is booted.
221 `grub` has been the de-facto standard for booting Linux systems for many years
222 and is quite well documented
223 footnote:[Grub Manual https://www.gnu.org/software/grub/manual/grub/grub.html].
227 Changes to the `grub` configuration are done via the defaults file
228 `/etc/default/grub` or config snippets in `/etc/default/grub.d`. To regenerate
229 the configuration file after a change to the configuration run:
230 footnote:[Systems using `proxmox-boot-tool` will call `proxmox-boot-tool
231 refresh` upon `update-grub`.]
238 [[sysboot_systemd_boot]]
242 `systemd-boot` is a lightweight EFI bootloader. It reads the kernel and initrd
243 images directly from the EFI Service Partition (ESP) where it is installed.
244 The main advantage of directly loading the kernel from the ESP is that it does
245 not need to reimplement the drivers for accessing the storage. In {pve}
246 xref:sysboot_proxmox_boot_tool[`proxmox-boot-tool`] is used to keep the
247 configuration on the ESPs synchronized.
249 [[sysboot_systemd_boot_config]]
253 `systemd-boot` is configured via the file `loader/loader.conf` in the root
254 directory of an EFI System Partition (ESP). See the `loader.conf(5)` manpage
257 Each bootloader entry is placed in a file of its own in the directory
260 An example entry.conf looks like this (`/` refers to the root of the ESP):
265 options root=ZFS=rpool/ROOT/pve-1 boot=zfs
266 linux /EFI/proxmox/5.0.15-1-pve/vmlinuz-5.0.15-1-pve
267 initrd /EFI/proxmox/5.0.15-1-pve/initrd.img-5.0.15-1-pve
271 [[sysboot_edit_kernel_cmdline]]
272 Editing the Kernel Commandline
273 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
275 You can modify the kernel commandline in the following places, depending on the
280 The kernel commandline needs to be placed in the variable
281 `GRUB_CMDLINE_LINUX_DEFAULT` in the file `/etc/default/grub`. Running
282 `update-grub` appends its content to all `linux` entries in
283 `/boot/grub/grub.cfg`.
287 The kernel commandline needs to be placed as one line in `/etc/kernel/cmdline`.
288 To apply your changes, run `proxmox-boot-tool refresh`, which sets it as the
289 `option` line for all config files in `loader/entries/proxmox-*.conf`.
292 [[sysboot_kernel_pin]]
293 Override the Kernel-Version for next Boot
294 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
296 To select a kernel that is not currently the default kernel, you can either:
298 * use the boot loader menu that is displayed at the beginning of the boot
300 * use the `proxmox-boot-tool` to `pin` the system to a kernel version either
301 once or permanently (until pin is reset).
303 This should help you work around incompatibilities between a newer kernel
304 version and the hardware.
306 NOTE: Such a pin should be removed as soon as possible so that all current
307 security patches of the latest kernel are also applied to the system.
309 For example: To permanently select the version `5.15.30-1-pve` for booting you
313 # proxmox-boot-tool kernel pin 5.15.30-1-pve
316 TIP: The pinning functionality works for all {pve} systems, not only those using
317 `proxmox-boot-tool` to synchronize the contents of the ESPs, if your system
318 does not use `proxmox-boot-tool` for synchronizing you can also skip the
319 `proxmox-boot-tool refresh` call in the end.
321 You can also set a kernel version to be booted on the next system boot only.
322 This is for example useful to test if an updated kernel has resolved an issue,
323 which caused you to `pin` a version in the first place:
326 # proxmox-boot-tool kernel pin 5.15.30-1-pve --next-boot
329 To remove any pinned version configuration use the `unpin` subcommand:
332 # proxmox-boot-tool kernel unpin
335 While `unpin` has a `--next-boot` option as well, it is used to clear a pinned
336 version set with `--next-boot`. As that happens already automatically on boot,
337 invonking it manually is of little use.
339 After setting, or clearing pinned versions you also need to synchronize the
340 content and configuration on the ESPs by running the `refresh` subcommand.
342 TIP: You will be prompted to automatically do for `proxmox-boot-tool` managed
343 systems if you call the tool interactively.
346 # proxmox-boot-tool refresh