8 {pve} currently uses one of two bootloaders depending on the disk setup
9 selected in the installer.
11 For EFI Systems installed with ZFS as the root filesystem `systemd-boot` is
12 used, unless Secure Boot is enabled. All other deployments use the standard
13 GRUB bootloader (this usually also applies to systems which are installed on
17 [[sysboot_installer_part_scheme]]
18 Partitioning Scheme Used by the Installer
19 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
21 The {pve} installer creates 3 partitions on all disks selected for
24 The created partitions are:
26 * a 1 MB BIOS Boot Partition (gdisk type EF02)
28 * a 512 MB EFI System Partition (ESP, gdisk type EF00)
30 * a third partition spanning the set `hdsize` parameter or the remaining space
31 used for the chosen storage type
33 Systems using ZFS as root filesystem are booted with a kernel and initrd image
34 stored on the 512 MB EFI System Partition. For legacy BIOS systems, and EFI
35 systems with Secure Boot enabled, GRUB is used, for EFI systems without
36 Secure Boot, `systemd-boot` is used. Both are installed and configured to point
39 GRUB in BIOS mode (`--target i386-pc`) is installed onto the BIOS Boot
40 Partition of all selected disks on all systems booted with GRUB
41 footnote:[These are all installs with root on `ext4` or `xfs` and installs
42 with root on ZFS on non-EFI systems].
45 [[sysboot_proxmox_boot_tool]]
46 Synchronizing the content of the ESP with `proxmox-boot-tool`
47 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
49 `proxmox-boot-tool` is a utility used to keep the contents of the EFI System
50 Partitions properly configured and synchronized. It copies certain kernel
51 versions to all ESPs and configures the respective bootloader to boot from
52 the `vfat` formatted ESPs. In the context of ZFS as root filesystem this means
53 that you can use all optional features on your root pool instead of the subset
54 which is also present in the ZFS implementation in GRUB or having to create a
55 separate small boot-pool footnote:[Booting ZFS on root with GRUB
56 https://github.com/zfsonlinux/zfs/wiki/Debian-Stretch-Root-on-ZFS].
58 In setups with redundancy all disks are partitioned with an ESP, by the
59 installer. This ensures the system boots even if the first boot device fails
60 or if the BIOS can only boot from a particular disk.
62 The ESPs are not kept mounted during regular operation. This helps to prevent
63 filesystem corruption to the `vfat` formatted ESPs in case of a system crash,
64 and removes the need to manually adapt `/etc/fstab` in case the primary boot
67 `proxmox-boot-tool` handles the following tasks:
69 * formatting and setting up a new partition
70 * copying and configuring new kernel images and initrd images to all listed ESPs
71 * synchronizing the configuration on kernel upgrades and other maintenance tasks
72 * managing the list of kernel versions which are synchronized
73 * configuring the boot-loader to boot a particular kernel version (pinning)
76 You can view the currently configured ESPs and their state by running:
79 # proxmox-boot-tool status
82 [[sysboot_proxmox_boot_setup]]
83 .Setting up a new partition for use as synced ESP
85 To format and initialize a partition as synced ESP, e.g., after replacing a
86 failed vdev in an rpool, or when converting an existing system that pre-dates
87 the sync mechanism, `proxmox-boot-tool` from `proxmox-kernel-helper` can be used.
89 WARNING: the `format` command will format the `<partition>`, make sure to pass
90 in the right device/partition!
92 For example, to format an empty partition `/dev/sda2` as ESP, run the following:
95 # proxmox-boot-tool format /dev/sda2
98 To setup an existing, unmounted ESP located on `/dev/sda2` for inclusion in
99 {pve}'s kernel update synchronization mechanism, use the following:
102 # proxmox-boot-tool init /dev/sda2
108 # proxmox-boot-tool init /dev/sda2 grub
111 to force initialization with GRUB instead of `systemd-boot`, for example for
114 Afterwards `/etc/kernel/proxmox-boot-uuids` should contain a new line with the
115 UUID of the newly added partition. The `init` command will also automatically
116 trigger a refresh of all configured ESPs.
118 [[sysboot_proxmox_boot_refresh]]
119 .Updating the configuration on all ESPs
121 To copy and configure all bootable kernels and keep all ESPs listed in
122 `/etc/kernel/proxmox-boot-uuids` in sync you just need to run:
125 # proxmox-boot-tool refresh
127 (The equivalent to running `update-grub` systems with `ext4` or `xfs` on root).
129 This is necessary should you make changes to the kernel commandline, or want to
130 sync all kernels and initrds.
132 NOTE: Both `update-initramfs` and `apt` (when necessary) will automatically
135 .Kernel Versions considered by `proxmox-boot-tool`
136 The following kernel versions are configured by default:
138 * the currently running kernel
139 * the version being newly installed on package updates
140 * the two latest already installed kernels
141 * the latest version of the second-to-last kernel series (e.g. 5.0, 5.3), if applicable
142 * any manually selected kernels
144 .Manually keeping a kernel bootable
146 Should you wish to add a certain kernel and initrd image to the list of
147 bootable kernels use `proxmox-boot-tool kernel add`.
149 For example run the following to add the kernel with ABI version `5.0.15-1-pve`
150 to the list of kernels to keep installed and synced to all ESPs:
153 # proxmox-boot-tool kernel add 5.0.15-1-pve
156 `proxmox-boot-tool kernel list` will list all kernel versions currently selected
160 # proxmox-boot-tool kernel list
161 Manually selected kernels:
164 Automatically selected kernels:
169 Run `proxmox-boot-tool kernel remove` to remove a kernel from the list of
170 manually selected kernels, for example:
173 # proxmox-boot-tool kernel remove 5.0.15-1-pve
176 NOTE: It's required to run `proxmox-boot-tool refresh` to update all EFI System
177 Partitions (ESPs) after a manual kernel addition or removal from above.
180 [[sysboot_determine_bootloader_used]]
181 Determine which Bootloader is Used
182 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
184 [thumbnail="screenshot/boot-grub.png", float="left"]
186 The simplest and most reliable way to determine which bootloader is used, is to
187 watch the boot process of the {pve} node.
189 You will either see the blue box of GRUB or the simple black on white
192 [thumbnail="screenshot/boot-systemdboot.png"]
194 Determining the bootloader from a running system might not be 100% accurate. The
195 safest way is to run the following command:
202 If it returns a message that EFI variables are not supported, GRUB is used in
205 If the output contains a line that looks similar to the following, GRUB is
209 Boot0005* proxmox [...] File(\EFI\proxmox\grubx64.efi)
212 If the output contains a line similar to the following, `systemd-boot` is used.
215 Boot0006* Linux Boot Manager [...] File(\EFI\systemd\systemd-bootx64.efi)
221 # proxmox-boot-tool status
224 you can find out if `proxmox-boot-tool` is configured, which is a good
225 indication of how the system is booted.
232 GRUB has been the de-facto standard for booting Linux systems for many years
233 and is quite well documented
234 footnote:[GRUB Manual https://www.gnu.org/software/grub/manual/grub/grub.html].
238 Changes to the GRUB configuration are done via the defaults file
239 `/etc/default/grub` or config snippets in `/etc/default/grub.d`. To regenerate
240 the configuration file after a change to the configuration run:
241 footnote:[Systems using `proxmox-boot-tool` will call `proxmox-boot-tool
242 refresh` upon `update-grub`.]
249 [[sysboot_systemd_boot]]
253 `systemd-boot` is a lightweight EFI bootloader. It reads the kernel and initrd
254 images directly from the EFI Service Partition (ESP) where it is installed.
255 The main advantage of directly loading the kernel from the ESP is that it does
256 not need to reimplement the drivers for accessing the storage. In {pve}
257 xref:sysboot_proxmox_boot_tool[`proxmox-boot-tool`] is used to keep the
258 configuration on the ESPs synchronized.
260 [[sysboot_systemd_boot_config]]
264 `systemd-boot` is configured via the file `loader/loader.conf` in the root
265 directory of an EFI System Partition (ESP). See the `loader.conf(5)` manpage
268 Each bootloader entry is placed in a file of its own in the directory
271 An example entry.conf looks like this (`/` refers to the root of the ESP):
276 options root=ZFS=rpool/ROOT/pve-1 boot=zfs
277 linux /EFI/proxmox/5.0.15-1-pve/vmlinuz-5.0.15-1-pve
278 initrd /EFI/proxmox/5.0.15-1-pve/initrd.img-5.0.15-1-pve
282 [[sysboot_edit_kernel_cmdline]]
283 Editing the Kernel Commandline
284 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
286 You can modify the kernel commandline in the following places, depending on the
291 The kernel commandline needs to be placed in the variable
292 `GRUB_CMDLINE_LINUX_DEFAULT` in the file `/etc/default/grub`. Running
293 `update-grub` appends its content to all `linux` entries in
294 `/boot/grub/grub.cfg`.
298 The kernel commandline needs to be placed as one line in `/etc/kernel/cmdline`.
299 To apply your changes, run `proxmox-boot-tool refresh`, which sets it as the
300 `option` line for all config files in `loader/entries/proxmox-*.conf`.
302 A complete list of kernel parameters can be found at
303 'https://www.kernel.org/doc/html/v<YOUR-KERNEL-VERSION>/admin-guide/kernel-parameters.html'.
304 replace <YOUR-KERNEL-VERSION> with the major.minor version, for example, for
305 kernels based on version 6.5 the URL would be:
306 https://www.kernel.org/doc/html/v6.5/admin-guide/kernel-parameters.html
308 You can find your kernel version by checking the web interface ('Node ->
309 Summary'), or by running
315 Use the first two numbers at the front of the output.
317 [[sysboot_kernel_pin]]
318 Override the Kernel-Version for next Boot
319 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
321 To select a kernel that is not currently the default kernel, you can either:
323 * use the boot loader menu that is displayed at the beginning of the boot
325 * use the `proxmox-boot-tool` to `pin` the system to a kernel version either
326 once or permanently (until pin is reset).
328 This should help you work around incompatibilities between a newer kernel
329 version and the hardware.
331 NOTE: Such a pin should be removed as soon as possible so that all current
332 security patches of the latest kernel are also applied to the system.
334 For example: To permanently select the version `5.15.30-1-pve` for booting you
338 # proxmox-boot-tool kernel pin 5.15.30-1-pve
341 TIP: The pinning functionality works for all {pve} systems, not only those using
342 `proxmox-boot-tool` to synchronize the contents of the ESPs, if your system
343 does not use `proxmox-boot-tool` for synchronizing you can also skip the
344 `proxmox-boot-tool refresh` call in the end.
346 You can also set a kernel version to be booted on the next system boot only.
347 This is for example useful to test if an updated kernel has resolved an issue,
348 which caused you to `pin` a version in the first place:
351 # proxmox-boot-tool kernel pin 5.15.30-1-pve --next-boot
354 To remove any pinned version configuration use the `unpin` subcommand:
357 # proxmox-boot-tool kernel unpin
360 While `unpin` has a `--next-boot` option as well, it is used to clear a pinned
361 version set with `--next-boot`. As that happens already automatically on boot,
362 invonking it manually is of little use.
364 After setting, or clearing pinned versions you also need to synchronize the
365 content and configuration on the ESPs by running the `refresh` subcommand.
367 TIP: You will be prompted to automatically do for `proxmox-boot-tool` managed
368 systems if you call the tool interactively.
371 # proxmox-boot-tool refresh
374 [[sysboot_secure_boot]]
378 Since {pve} 8.1, Secure Boot is supported out of the box via signed packages
379 and integration in `proxmox-boot-tool`.
381 The following packages are required for secure boot to work. You can
382 install them all at once by using the `proxmox-secure-boot-support'
385 - `shim-signed` (shim bootloader signed by Microsoft)
386 - `shim-helpers-amd64-signed` (fallback bootloader and MOKManager, signed by
388 - `grub-efi-amd64-signed` (GRUB EFI bootloader, signed by Proxmox)
389 - `proxmox-kernel-6.X.Y-Z-pve-signed` (Kernel image, signed by Proxmox)
391 Only GRUB is supported as bootloader out of the box, since other bootloader are
392 currently not eligible for secure boot code-signing.
394 Any new installation of {pve} will automatically have all of the above packages
397 More details about how Secure Boot works, and how to customize the setup, are
398 available in https://pve.proxmox.com/wiki/Secure_Boot_Setup[our wiki].
400 Switching an Existing Installation to Secure Boot
401 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
403 WARNING: This can lead to an unbootable installation in some cases if not done
404 correctly. Reinstalling the host will setup Secure Boot automatically if
405 available, without any extra interactions. **Make sure you have a working and
406 well-tested backup of your {pve} host!**
408 An existing UEFI installation can be switched over to Secure Boot if desired,
409 without having to reinstall {pve} from scratch.
411 First, ensure all your system is up-to-date. Next, install
412 `proxmox-secure-boot-support`. GRUB automatically creates the needed EFI boot
413 entry for booting via the default shim.
417 If `systemd-boot` is used as a bootloader (see
418 xref:sysboot_determine_bootloader_used[Determine which Bootloader is used]),
419 some additional setup is needed. This is only the case if {pve} was installed
422 To check the latter, run:
427 If the host is indeed using ZFS as root filesystem, the `FSTYPE` column
428 should contain `zfs`:
430 TARGET SOURCE FSTYPE OPTIONS
431 / rpool/ROOT/pve-1 zfs rw,relatime,xattr,noacl,casesensitive
434 Next, a suitable potential ESP (EFI system partition) must be found. This can be
435 done using the `lsblk` command as following:
440 The output should look something like this:
442 NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS FSTYPE
444 ├─sda1 8:1 0 1007K 0 part
445 ├─sda2 8:2 0 512M 0 part vfat
446 └─sda3 8:3 0 31.5G 0 part zfs_member
447 sdb 8:16 0 32G 0 disk
448 ├─sdb1 8:17 0 1007K 0 part
449 ├─sdb2 8:18 0 512M 0 part vfat
450 └─sdb3 8:19 0 31.5G 0 part zfs_member
453 In this case, the partitions `sda2` and `sdb2` are the targets. They can be
454 identified by the their size of 512M and their `FSTYPE` being `vfat`, in this
455 case on a ZFS RAID-1 installation.
457 These partitions must be properly set up for booting through GRUB using
458 `proxmox-boot-tool`. This command (using `sda2` as an example) must be run
459 separately for each individual ESP:
461 # proxmox-boot-tool init /dev/sda2 grub
464 Afterwards, you can sanity-check the setup by running the following command:
469 This list should contain an entry looking similar to this:
472 Boot0009* proxmox HD(2,GPT,..,0x800,0x100000)/File(\EFI\proxmox\shimx64.efi)
476 NOTE: The old `systemd-boot` bootloader will be kept, but GRUB will be
477 preferred. This way, if booting using GRUB in Secure Boot mode does not work for
478 any reason, the system can still be booted using `systemd-boot` with Secure Boot
481 Now the host can be rebooted and Secure Boot enabled in the UEFI firmware setup
484 On reboot, a new entry named `proxmox` should be selectable in the UEFI firmware
485 boot menu, which boots using the pre-signed EFI shim.
487 If, for any reason, no `proxmox` entry can be found in the UEFI boot menu, you
488 can try adding it manually (if supported by the firmware), by adding the file
489 `\EFI\proxmox\shimx64.efi` as a custom boot entry.
491 NOTE: Some UEFI firmwares are known to drop the `proxmox` boot option on reboot.
492 This can happen if the `proxmox` boot entry is pointing to a GRUB installation
493 on a disk, where the disk itself is not a boot option. If possible, try adding
494 the disk as a boot option in the UEFI firmware setup utility and run
495 `proxmox-boot-tool` again.
497 TIP: To enroll custom keys, see the accompanying
498 https://pve.proxmox.com/wiki/Secure_Boot_Setup#Setup_instructions_for_db_key_variant[Secure
501 Using DKMS/Third Party Modules With Secure Boot
502 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
504 On systems with Secure Boot enabled, the kernel will refuse to load modules
505 which are not signed by a trusted key. The default set of modules shipped with
506 the kernel packages is signed with an ephemeral key embedded in the kernel
507 image which is trusted by that specific version of the kernel image.
509 In order to load other modules, such as those built with DKMS or manually, they
510 need to be signed with a key trusted by the Secure Boot stack. The easiest way
511 to achieve this is to enroll them as Machine Owner Key (`MOK`) with `mokutil`.
513 The `dkms` tool will automatically generate a keypair and certificate in
514 `/var/lib/dkms/mok.key` and `/var/lib/dkms/mok.pub` and use it for signing
515 the kernel modules it builds and installs.
517 You can view the certificate contents with
520 # openssl x509 -in /var/lib/dkms/mok.pub -noout -text
523 and enroll it on your system using the following command:
526 # mokutil --import /var/lib/dkms/mok.pub
528 input password again:
531 The `mokutil` command will ask for a (temporary) password twice, this password
532 needs to be entered one more time in the next step of the process! Rebooting
533 the system should automatically boot into the `MOKManager` EFI binary, which
534 allows you to verify the key/certificate and confirm the enrollment using the
535 password selected when starting the enrollment using `mokutil`. Afterwards, the
536 kernel should allow loading modules built with DKMS (which are signed with the
537 enrolled `MOK`). The `MOK` can also be used to sign custom EFI binaries and
538 kernel images if desired.
540 The same procedure can also be used for custom/third-party modules not managed
541 with DKMS, but the key/certificate generation and signing steps need to be done
542 manually in that case.