selected in the installer.
For EFI Systems installed with ZFS as the root filesystem `systemd-boot` is
-used. All other deployments use the standard `grub` bootloader (this usually
-also applies to systems which are installed on top of Debian).
+used, unless Secure Boot is enabled. All other deployments use the standard
+`grub` bootloader (this usually also applies to systems which are installed on
+top of Debian).
+
[[sysboot_installer_part_scheme]]
-Partitioning scheme used by the installer
+Partitioning Scheme Used by the Installer
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-The {pve} installer creates 3 partitions on the bootable disks selected for
-installation. The bootable disks are:
-
-* For Installations with `ext4` or `xfs` the selected disk
-
-* For ZFS installations all disks belonging to the first `vdev`:
-** The first disk for RAID0
-** All disks for RAID1, RAIDZ1, RAIDZ2, RAIDZ3
-** The first two disks for RAID10
+The {pve} installer creates 3 partitions on all disks selected for
+installation.
The created partitions are:
* a third partition spanning the set `hdsize` parameter or the remaining space
used for the chosen storage type
+Systems using ZFS as root filesystem are booted with a kernel and initrd image
+stored on the 512 MB EFI System Partition. For legacy BIOS systems, and EFI
+systems with Secure Boot enabled, `grub` is used, for EFI systems without
+Secure Boot, `systemd-boot` is used. Both are installed and configured to point
+to the ESPs.
+
`grub` in BIOS mode (`--target i386-pc`) is installed onto the BIOS Boot
-Partition of all bootable disks for supporting older systems.
+Partition of all selected disks on all systems booted with `grub`
+footnote:[These are all installs with root on `ext4` or `xfs` and installs
+with root on ZFS on non-EFI systems].
+
+
+[[sysboot_proxmox_boot_tool]]
+Synchronizing the content of the ESP with `proxmox-boot-tool`
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+`proxmox-boot-tool` is a utility used to keep the contents of the EFI System
+Partitions properly configured and synchronized. It copies certain kernel
+versions to all ESPs and configures the respective bootloader to boot from
+the `vfat` formatted ESPs. In the context of ZFS as root filesystem this means
+that you can use all optional features on your root pool instead of the subset
+which is also present in the ZFS implementation in `grub` or having to create a
+separate small boot-pool footnote:[Booting ZFS on root with grub
+https://github.com/zfsonlinux/zfs/wiki/Debian-Stretch-Root-on-ZFS].
+
+In setups with redundancy all disks are partitioned with an ESP, by the
+installer. This ensures the system boots even if the first boot device fails
+or if the BIOS can only boot from a particular disk.
+
+The ESPs are not kept mounted during regular operation. This helps to prevent
+filesystem corruption to the `vfat` formatted ESPs in case of a system crash,
+and removes the need to manually adapt `/etc/fstab` in case the primary boot
+device fails.
+
+`proxmox-boot-tool` handles the following tasks:
+
+* formatting and setting up a new partition
+* copying and configuring new kernel images and initrd images to all listed ESPs
+* synchronizing the configuration on kernel upgrades and other maintenance tasks
+* managing the list of kernel versions which are synchronized
+* configuring the boot-loader to boot a particular kernel version (pinning)
+
+
+You can view the currently configured ESPs and their state by running:
+
+----
+# proxmox-boot-tool status
+----
+
+[[sysboot_proxmox_boot_setup]]
+.Setting up a new partition for use as synced ESP
+
+To format and initialize a partition as synced ESP, e.g., after replacing a
+failed vdev in an rpool, or when converting an existing system that pre-dates
+the sync mechanism, `proxmox-boot-tool` from `proxmox-kernel-helper` can be used.
+
+WARNING: the `format` command will format the `<partition>`, make sure to pass
+in the right device/partition!
+
+For example, to format an empty partition `/dev/sda2` as ESP, run the following:
+
+----
+# proxmox-boot-tool format /dev/sda2
+----
+
+To setup an existing, unmounted ESP located on `/dev/sda2` for inclusion in
+{pve}'s kernel update synchronization mechanism, use the following:
+
+----
+# proxmox-boot-tool init /dev/sda2
+----
+
+or
+
+----
+# proxmox-boot-tool init /dev/sda2 grub
+----
+
+to force initialization with Grub instead of systemd-boot, for example for
+Secure Boot support.
+
+Afterwards `/etc/kernel/proxmox-boot-uuids` should contain a new line with the
+UUID of the newly added partition. The `init` command will also automatically
+trigger a refresh of all configured ESPs.
+
+[[sysboot_proxmox_boot_refresh]]
+.Updating the configuration on all ESPs
+
+To copy and configure all bootable kernels and keep all ESPs listed in
+`/etc/kernel/proxmox-boot-uuids` in sync you just need to run:
+
+----
+# proxmox-boot-tool refresh
+----
+(The equivalent to running `update-grub` systems with `ext4` or `xfs` on root).
+
+This is necessary should you make changes to the kernel commandline, or want to
+sync all kernels and initrds.
+
+NOTE: Both `update-initramfs` and `apt` (when necessary) will automatically
+trigger a refresh.
+
+.Kernel Versions considered by `proxmox-boot-tool`
+The following kernel versions are configured by default:
+
+* the currently running kernel
+* the version being newly installed on package updates
+* the two latest already installed kernels
+* the latest version of the second-to-last kernel series (e.g. 5.0, 5.3), if applicable
+* any manually selected kernels
+
+.Manually keeping a kernel bootable
+
+Should you wish to add a certain kernel and initrd image to the list of
+bootable kernels use `proxmox-boot-tool kernel add`.
+
+For example run the following to add the kernel with ABI version `5.0.15-1-pve`
+to the list of kernels to keep installed and synced to all ESPs:
+
+----
+# proxmox-boot-tool kernel add 5.0.15-1-pve
+----
+
+`proxmox-boot-tool kernel list` will list all kernel versions currently selected
+for booting:
+
+----
+# proxmox-boot-tool kernel list
+Manually selected kernels:
+5.0.15-1-pve
+
+Automatically selected kernels:
+5.0.12-1-pve
+4.15.18-18-pve
+----
+
+Run `proxmox-boot-tool kernel remove` to remove a kernel from the list of
+manually selected kernels, for example:
+
+----
+# proxmox-boot-tool kernel remove 5.0.15-1-pve
+----
+
+NOTE: It's required to run `proxmox-boot-tool refresh` to update all EFI System
+Partitions (ESPs) after a manual kernel addition or removal from above.
+
+
+[[sysboot_determine_bootloader_used]]
+Determine which Bootloader is Used
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+[thumbnail="screenshot/boot-grub.png", float="left"]
+
+The simplest and most reliable way to determine which bootloader is used, is to
+watch the boot process of the {pve} node.
+
+You will either see the blue box of `grub` or the simple black on white
+`systemd-boot`.
+
+[thumbnail="screenshot/boot-systemdboot.png"]
+
+Determining the bootloader from a running system might not be 100% accurate. The
+safest way is to run the following command:
+
+
+----
+# efibootmgr -v
+----
+
+If it returns a message that EFI variables are not supported, `grub` is used in
+BIOS/Legacy mode.
+
+If the output contains a line that looks similar to the following, `grub` is
+used in UEFI mode.
+
+----
+Boot0005* proxmox [...] File(\EFI\proxmox\grubx64.efi)
+----
+
+If the output contains a line similar to the following, `systemd-boot` is used.
+
+----
+Boot0006* Linux Boot Manager [...] File(\EFI\systemd\systemd-bootx64.efi)
+----
+
+By running:
+
+----
+# proxmox-boot-tool status
+----
+
+you can find out if `proxmox-boot-tool` is configured, which is a good
+indication of how the system is booted.
[[sysboot_grub]]
and is quite well documented
footnote:[Grub Manual https://www.gnu.org/software/grub/manual/grub/grub.html].
-The kernel and initrd images are taken from `/boot` and its configuration file
-`/boot/grub/grub.cfg` gets updated by the kernel installation process.
-
Configuration
^^^^^^^^^^^^^
Changes to the `grub` configuration are done via the defaults file
`/etc/default/grub` or config snippets in `/etc/default/grub.d`. To regenerate
-the `/boot/grub/grub.cfg` after a change to the configuration run:
+the configuration file after a change to the configuration run:
+footnote:[Systems using `proxmox-boot-tool` will call `proxmox-boot-tool
+refresh` upon `update-grub`.]
----
-`update-grub`.
+# update-grub
----
+
[[sysboot_systemd_boot]]
Systemd-boot
~~~~~~~~~~~~
`systemd-boot` is a lightweight EFI bootloader. It reads the kernel and initrd
images directly from the EFI Service Partition (ESP) where it is installed.
The main advantage of directly loading the kernel from the ESP is that it does
-not need to reimplement the drivers for accessing the storage. In the context
-of ZFS as root filesystem this means that you can use all optional features on
-your root pool instead of the subset which is also present in the ZFS
-implementation in `grub` or having to create a separate small boot-pool
-footnote:[Booting ZFS on root with grub https://github.com/zfsonlinux/zfs/wiki/Debian-Stretch-Root-on-ZFS].
-
-In setups with redundancy (RAID1, RAID10, RAIDZ*) all bootable disks (those
-being part of the first `vdev`) are partitioned with an ESP. This ensures the
-system boots even if the first boot device fails. The ESPs are kept in sync by
-a kernel postinstall hook script `/etc/kernel/postinst.d/zz-pve-efiboot`. The
-script copies certain kernel versions and the initrd images to `EFI/proxmox/`
-on the root of each ESP and creates the appropriate config files in
-`loader/entries/proxmox-*.conf`. The `pve-efiboot-tool` script assists in
-managing both the synced ESPs themselves and their contents.
-
-The following kernel versions are configured by default:
-
-* the currently running kernel
-* the version being newly installed on package updates
-* the two latest kernels
-* the latest version of each kernel series (e.g. 4.15, 5.0).
-
-The ESPs are not kept mounted during regular operation, in contrast to `grub`,
-which keeps an ESP mounted on `/boot/efi`. This helps to prevent filesystem
-corruption to the `vfat` formatted ESPs in case of a system crash, and removes
-the need to manually adapt `/etc/fstab` in case the primary boot device fails.
+not need to reimplement the drivers for accessing the storage. In {pve}
+xref:sysboot_proxmox_boot_tool[`proxmox-boot-tool`] is used to keep the
+configuration on the ESPs synchronized.
[[sysboot_systemd_boot_config]]
Configuration
----
-.Manually keeping a kernel bootable
+[[sysboot_edit_kernel_cmdline]]
+Editing the Kernel Commandline
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-Should you wish to add a certain kernel and initrd image to the list of
-bootable kernel use `pve-efiboot-tool kernel add`.
+You can modify the kernel commandline in the following places, depending on the
+bootloader used:
-For example run the following to add the kernel with ABI version `5.0.15-1-pve`
-to the list of kernels to keep installed and synced to all ESPs:
+.Grub
+
+The kernel commandline needs to be placed in the variable
+`GRUB_CMDLINE_LINUX_DEFAULT` in the file `/etc/default/grub`. Running
+`update-grub` appends its content to all `linux` entries in
+`/boot/grub/grub.cfg`.
+
+.Systemd-boot
+
+The kernel commandline needs to be placed as one line in `/etc/kernel/cmdline`.
+To apply your changes, run `proxmox-boot-tool refresh`, which sets it as the
+`option` line for all config files in `loader/entries/proxmox-*.conf`.
+
+A complete list of kernel parameters can be found at
+'https://www.kernel.org/doc/html/v<YOUR-KERNEL-VERSION>/admin-guide/kernel-parameters.html'.
+replace <YOUR-KERNEL-VERSION> with the major.minor version, for example, for
+kernels based on version 6.5 the URL would be:
+https://www.kernel.org/doc/html/v6.5/admin-guide/kernel-parameters.html
+
+You can find your kernel version by checking the web interface ('Node ->
+Summary'), or by running
----
-pve-efiboot-tool kernel add 5.0.15-1-pve
+# uname -r
----
-`pve-efiboot-tool kernel list` will list all kernel versions currently selected
-for booting:
+Use the first two numbers at the front of the output.
+
+[[sysboot_kernel_pin]]
+Override the Kernel-Version for next Boot
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+To select a kernel that is not currently the default kernel, you can either:
+
+* use the boot loader menu that is displayed at the beginning of the boot
+ process
+* use the `proxmox-boot-tool` to `pin` the system to a kernel version either
+ once or permanently (until pin is reset).
+
+This should help you work around incompatibilities between a newer kernel
+version and the hardware.
+
+NOTE: Such a pin should be removed as soon as possible so that all current
+security patches of the latest kernel are also applied to the system.
+
+For example: To permanently select the version `5.15.30-1-pve` for booting you
+would run:
----
-# pve-efiboot-tool kernel list
-Manually selected kernels:
-5.0.15-1-pve
+# proxmox-boot-tool kernel pin 5.15.30-1-pve
+----
-Automatically selected kernels:
-5.0.12-1-pve
-4.15.18-18-pve
+TIP: The pinning functionality works for all {pve} systems, not only those using
+`proxmox-boot-tool` to synchronize the contents of the ESPs, if your system
+does not use `proxmox-boot-tool` for synchronizing you can also skip the
+`proxmox-boot-tool refresh` call in the end.
+
+You can also set a kernel version to be booted on the next system boot only.
+This is for example useful to test if an updated kernel has resolved an issue,
+which caused you to `pin` a version in the first place:
+
+----
+# proxmox-boot-tool kernel pin 5.15.30-1-pve --next-boot
----
-Run `pve-efiboot-tool remove` to remove a kernel from the list of manually
-selected kernels, for example:
+To remove any pinned version configuration use the `unpin` subcommand:
----
-pve-efiboot-tool kernel remove 5.0.15-1-pve
+# proxmox-boot-tool kernel unpin
----
+While `unpin` has a `--next-boot` option as well, it is used to clear a pinned
+version set with `--next-boot`. As that happens already automatically on boot,
+invonking it manually is of little use.
-[[sysboot_systemd_boot_setup]]
-.Setting up a new partition for use as synced ESP
+After setting, or clearing pinned versions you also need to synchronize the
+content and configuration on the ESPs by running the `refresh` subcommand.
-To format and initialize a partition as synced ESP, e.g., after replacing a
-failed vdev in an rpool, or when converting an existing system that pre-dates
-the sync mechanism, `pve-efiboot-tool` from `pve-kernel-helpers` can be used.
+TIP: You will be prompted to automatically do for `proxmox-boot-tool` managed
+systems if you call the tool interactively.
-WARNING: the `format` command will format the `<partition>`, make sure to pass
-in the right device/partition!
+----
+# proxmox-boot-tool refresh
+----
-For example, to format an empty partition `/dev/sda2` as ESP, run the following:
+[[sysboot_secure_boot]]
+Secure Boot
+~~~~~~~~~~~
+
+Since {pve} 8.1, Secure Boot is supported out of the box via signed packages
+and integration in `proxmox-boot-tool`.
+
+The following packages need to be installed for Secure Boot to be enabled:
+
+- shim-signed (shim bootloader signed by Microsoft)
+- shim-helpers-amd64-signed (fallback bootloader and MOKManager, signed by Proxmox)
+- grub-efi-amd64-signed (Grub EFI bootloader, signed by Proxmox)
+- proxmox-kernel-6.X.Y-Z-pve-signed (Kernel image, signed by Proxmox)
+
+Only Grub as bootloader is supported out of the box, since there are no other
+pre-signed bootloader packages available. Any new installation of {pve} will
+automatically have all of the above packages included.
+
+More details about how Secure Boot works, and how to customize the setup, are
+available in https://pve.proxmox.com/wiki/Secure_Boot_Setup[our wiki].
+
+Switching an Existing Installation to Secure Boot
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+WARNING: This can lead to an unbootable installation in some cases if not done
+correctly. Reinstalling the host will setup Secure Boot automatically if
+available, without any extra interactions. **Make sure you have a working and
+well-tested backup of your {pve} host!**
+
+An existing UEFI installation can be switched over to Secure Boot if desired,
+without having to reinstall {pve} from scratch.
+
+First, ensure all your system is up-to-date. Next, install all the required
+pre-signed packages as listed above. Grub automatically creates the needed EFI
+boot entry for booting via the default shim.
+
+.systemd-boot
+
+If `systemd-boot` is used as a bootloader (see
+xref:sysboot_determine_bootloader_used[Determine which Bootloader is used]),
+some additional setup is needed. This is only the case if {pve} was installed
+with ZFS-on-root.
+
+To check the latter, run:
----
-pve-efiboot-tool format /dev/sda2
+# findmnt /
----
-To setup an existing, unmounted ESP located on `/dev/sda2` for inclusion in
-{pve}'s kernel update synchronization mechanism, use the following:
+If the host is indeed running using ZFS as root filesystem, the `FSTYPE` column
+should contain `zfs`:
+----
+TARGET SOURCE FSTYPE OPTIONS
+/ rpool/ROOT/pve-1 zfs rw,relatime,xattr,noacl,casesensitive
+----
+Next, a suitable potential ESP (EFI system partition) must be found. This can be
+done using the `lsblk` command as following:
----
-pve-efiboot-tool init /dev/sda2
+# lsblk -o +FSTYPE
----
-Afterwards `/etc/kernel/pve-efiboot-uuids` should contain a new line with the
-UUID of the newly added partition. The `init` command will also automatically
-trigger a refresh of all configured ESPs.
+The output should look something like this:
+----
+NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS FSTYPE
+sda 8:0 0 32G 0 disk
+├─sda1 8:1 0 1007K 0 part
+├─sda2 8:2 0 512M 0 part vfat
+└─sda3 8:3 0 31.5G 0 part zfs_member
+sdb 8:16 0 32G 0 disk
+├─sdb1 8:17 0 1007K 0 part
+├─sdb2 8:18 0 512M 0 part vfat
+└─sdb3 8:19 0 31.5G 0 part zfs_member
+----
-[[sysboot_systemd_boot_refresh]]
-.Updating the configuration on all ESPs
+In this case, the partitions `sda2` and `sdb2` are the targets. They can be
+identified by the their size of 512M and their `FSTYPE` being `vfat`, in this
+case on a ZFS RAID-1 installation.
-To copy and configure all bootable kernels and keep all ESPs listed in
-`/etc/kernel/pve-efiboot-uuids` in sync you just need to run `pve-efiboot-tool
-refresh`.
-(The equivalent to running `update-grub` on Systems being booted with `grub`).
+These partitions must be properly set up for booting through Grub using
+`proxmox-boot-tool`. This command (using `sda2` as an example) must be run
+separately for each individual ESP:
+----
+# proxmox-boot-tool init /dev/sda2 grub
+----
-This is necessary should you make changes to the kernel commandline, or want to
-sync all kernels and initrds after regenerating the latter.
+Afterwards, you can sanity-check the setup by running the following command:
+----
+# efibootmgr -v
+----
-[[sysboot_edit_kernel_cmdline]]
-Editing the kernel commandline
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+This list should contain an entry looking similar to this:
+----
+[..]
+Boot0009* proxmox HD(2,GPT,..,0x800,0x100000)/File(\EFI\proxmox\shimx64.efi)
+[..]
+----
-You can modify the kernel commandline in the following places, depending on the
-bootloarder used:
+NOTE: The old `systemd-boot` bootloader will be kept, but Grub will be
+preferred. This way, if booting using Grub in Secure Boot mode does not work for
+any reason, the system can still be booted using `systemd-boot` with Secure Boot
+turned off.
-.Grub
+Now the host can be rebooted and Secure Boot enabled in the UEFI firmware setup
+utility.
-The kernel commandline needs to be placed in the variable
-`GRUB_CMDLINE_LINUX_DEFAULT` in the file `/etc/default/grub`. Running
-`update-grub` appends its content to all `linux` entries in
-`/boot/grub/grub.cfg`.
+On reboot, a new entry named `proxmox` should be selectable in the UEFI firmware
+boot menu, which boots using the pre-signed EFI shim.
-.Systemd-boot
+If, for any reason, no `proxmox` entry can be found in the UEFI boot menu, you
+can try adding it manually (if supported by the firmware), by adding the file
+`\EFI\proxmox\shimx64.efi` as a custom boot entry.
+
+NOTE: Some UEFI firmwares are known to drop the `proxmox` boot option on reboot.
+This can happen if the `proxmox` boot entry is pointing to a Grub installation
+on a disk, where the disk itself not a boot option. If possible, try adding the
+disk as a boot option in the UEFI firmware setup utility and run
+`proxmox-boot-tool` again.
+
+TIP: To enroll custom keys, see the accompanying
+https://pve.proxmox.com/wiki/Secure_Boot_Setup#Setup_instructions_for_db_key_variant[Secure
+Boot wiki page].
+
+Using DKMS/third party modules with Secure Boot
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+On systems with Secure Boot enabled, the kernel will refuse to load modules
+which are not signed by a trusted key. The default set of modules shipped with
+the kernel packages is signed with an ephemeral key embedded in the kernel
+image which is trusted by that specific version of the kernel image.
+
+In order to load other modules, such as those built with DKMS or manually, they
+need to be signed with a key trusted by the Secure Boot stack. The easiest way
+to achieve this is to enroll them as Machine Owner Key (`MOK`) with `mokutil`.
+
+The `dkms` tool will automatically generate a keypair and certificate in
+`/var/lib/dkms/mok.key` and `/var/lib/dkms/mok.pub` and use it for signing
+the kernel modules it builds and installs.
+
+You can view the certificate contents with
+
+----
+# openssl x509 -in /var/lib/dkms/mok.pub -noout -text
+----
+
+and enroll it on your system using the following command:
+
+----
+# mokutil --import /var/lib/dkms/mok.pub
+input password:
+input password again:
+----
-The kernel commandline needs to be placed as line in `/etc/kernel/cmdline`
-Running `/etc/kernel/postinst.d/zz-pve-efiboot` sets it as `option` line for
-all config files in `loader/entries/proxmox-*.conf`.
+The `mokutil` command will ask for a (temporary) password twice, this password
+needs to be entered one more time in the next step of the process! Rebooting
+the system should automatically boot into the `MOKManager` EFI binary, which
+allows you to verify the key/certificate and confirm the enrollment using the
+password selected when starting the enrollment using `mokutil`. Afterwards, the
+kernel should allow loading modules built with DKMS (which are signed with the
+enrolled `MOK`). The `MOK` can also be used to sign custom EFI binaries and
+kernel images if desired.
+
+The same procedure can also be used for custom/third-party modules not managed
+with DKMS, but the key/certificate generation and signing steps need to be done
+manually in that case.
projects / pve-docs.git / blobdiff