X-Git-Url: https://git.proxmox.com/?a=blobdiff_plain;f=system-booting.adoc;h=2b96409bffe57b8485522b9feae91c2c79f94735;hb=4622d5b4c5c725b623faf93922b80d90bc337e4f;hp=28507a2d4c1587e61e6b88e5b118c2e82425a3e0;hpb=0daaddbdc0bce7af1dddeea637d0ea69c73aa088;p=pve-docs.git diff --git a/system-booting.adoc b/system-booting.adoc index 28507a2..2b96409 100644 --- a/system-booting.adoc +++ b/system-booting.adoc @@ -12,19 +12,13 @@ For EFI Systems installed with ZFS as the root filesystem `systemd-boot` is used. All other deployments use the standard `grub` bootloader (this usually also applies to systems which are installed on top of Debian). + [[sysboot_installer_part_scheme]] -Partitioning scheme used by the installer +Partitioning Scheme Used by the Installer ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -The {pve} installer creates 3 partitions on the bootable disks selected for -installation. The bootable disks are: - -* For Installations with `ext4` or `xfs` the selected disk - -* For ZFS installations all disks belonging to the first `vdev`: -** The first disk for RAID0 -** All disks for RAID1, RAIDZ1, RAIDZ2, RAIDZ3 -** The first two disks for RAID10 +The {pve} installer creates 3 partitions on all disks selected for +installation. The created partitions are: @@ -35,156 +29,250 @@ The created partitions are: * a third partition spanning the set `hdsize` parameter or the remaining space used for the chosen storage type +Systems using ZFS as root filesystem are booted with a kernel and initrd image +stored on the 512 MB EFI System Partition. For legacy BIOS systems, `grub` is +used, for EFI systems `systemd-boot` is used. Both are installed and configured +to point to the ESPs. + `grub` in BIOS mode (`--target i386-pc`) is installed onto the BIOS Boot -Partition of all bootable disks for supporting older systems. +Partition of all selected disks on all systems booted with `grub` +footnote:[These are all installs with root on `ext4` or `xfs` and installs +with root on ZFS on non-EFI systems]. -[[sysboot_grub]] -Grub -~~~~ +[[sysboot_proxmox_boot_tool]] +Synchronizing the content of the ESP with `proxmox-boot-tool` +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -`grub` has been the de-facto standard for booting Linux systems for many years -and is quite well documented -footnote:[Grub Manual https://www.gnu.org/software/grub/manual/grub/grub.html]. +`proxmox-boot-tool` is a utility used to keep the contents of the EFI System +Partitions properly configured and synchronized. It copies certain kernel +versions to all ESPs and configures the respective bootloader to boot from +the `vfat` formatted ESPs. In the context of ZFS as root filesystem this means +that you can use all optional features on your root pool instead of the subset +which is also present in the ZFS implementation in `grub` or having to create a +separate small boot-pool footnote:[Booting ZFS on root with grub +https://github.com/zfsonlinux/zfs/wiki/Debian-Stretch-Root-on-ZFS]. -The kernel and initrd images are taken from `/boot` and its configuration file -`/boot/grub/grub.cfg` gets updated by the kernel installation process. +In setups with redundancy all disks are partitioned with an ESP, by the +installer. This ensures the system boots even if the first boot device fails +or if the BIOS can only boot from a particular disk. -Configuration -^^^^^^^^^^^^^ -Changes to the `grub` configuration are done via the defaults file -`/etc/default/grub` or config snippets in `/etc/default/grub.d`. To regenerate -the `/boot/grub/grub.cfg` after a change to the configuration run: +The ESPs are not kept mounted during regular operation. This helps to prevent +filesystem corruption to the `vfat` formatted ESPs in case of a system crash, +and removes the need to manually adapt `/etc/fstab` in case the primary boot +device fails. + +`proxmox-boot-tool` handles the following tasks: + +* formatting and setting up a new partition +* copying and configuring new kernel images and initrd images to all listed ESPs +* synchronizing the configuration on kernel upgrades and other maintenance tasks +* managing the list of kernel versions which are synchronized + + +You can view the currently configured ESPs and their state by running: ---- -`update-grub`. +# proxmox-boot-tool status ---- -[[sysboot_systemd_boot]] -Systemd-boot -~~~~~~~~~~~~ +[[sysboot_proxmox_boot_setup]] +.Setting up a new partition for use as synced ESP -`systemd-boot` is a lightweight EFI bootloader. It reads the kernel and initrd -images directly from the EFI Service Partition (ESP) where it is installed. -The main advantage of directly loading the kernel from the ESP is that it does -not need to reimplement the drivers for accessing the storage. In the context -of ZFS as root filesystem this means that you can use all optional features on -your root pool instead of the subset which is also present in the ZFS -implementation in `grub` or having to create a separate small boot-pool -footnote:[Booting ZFS on root with grub https://github.com/zfsonlinux/zfs/wiki/Debian-Stretch-Root-on-ZFS]. - -In setups with redundancy (RAID1, RAID10, RAIDZ*) all bootable disks (those -being part of the first `vdev`) are partitioned with an ESP. This ensures the -system boots even if the first boot device fails. The ESPs are kept in sync by -a kernel postinstall hook script `/etc/kernel/postinst.d/zz-pve-efiboot`. The -script copies certain kernel versions and the initrd images to `EFI/proxmox/` -on the root of each ESP and creates the appropriate config files in -`loader/entries/proxmox-*.conf`. +To format and initialize a partition as synced ESP, e.g., after replacing a +failed vdev in an rpool, or when converting an existing system that pre-dates +the sync mechanism, `proxmox-boot-tool` from `pve-kernel-helpers` can be used. + +WARNING: the `format` command will format the ``, make sure to pass +in the right device/partition! + +For example, to format an empty partition `/dev/sda2` as ESP, run the following: + +---- +# proxmox-boot-tool format /dev/sda2 +---- +To setup an existing, unmounted ESP located on `/dev/sda2` for inclusion in +{pve}'s kernel update synchronization mechanism, use the following: + +---- +# proxmox-boot-tool init /dev/sda2 +---- + +Afterwards `/etc/kernel/proxmox-boot-uuids` should contain a new line with the +UUID of the newly added partition. The `init` command will also automatically +trigger a refresh of all configured ESPs. + +[[sysboot_proxmox_boot_refresh]] +.Updating the configuration on all ESPs + +To copy and configure all bootable kernels and keep all ESPs listed in +`/etc/kernel/proxmox-boot-uuids` in sync you just need to run: + +---- +# proxmox-boot-tool refresh +---- +(The equivalent to running `update-grub` systems with `ext4` or `xfs` on root). + +This is necessary should you make changes to the kernel commandline, or want to +sync all kernels and initrds. + +NOTE: Both `update-initramfs` and `apt` (when necessary) will automatically +trigger a refresh. + +.Kernel Versions considered by `proxmox-boot-tool` The following kernel versions are configured by default: * the currently running kernel * the version being newly installed on package updates -* the two latest kernels -* the latest version of each kernel series (e.g. 4.15, 5.0). +* the two latest already installed kernels +* the latest version of the second-to-last kernel series (e.g. 5.0, 5.3), if applicable +* any manually selected kernels -The ESPs are not kept mounted during regular operation, in contrast to `grub`, -which keeps an ESP mounted on `/boot/efi`. This helps to prevent filesystem -corruption to the `vfat` formatted ESPs in case of a system crash, and removes -the need to manually adapt `/etc/fstab` in case the primary boot device fails. +.Manually keeping a kernel bootable -[[sysboot_systemd_boot_config]] -Configuration -^^^^^^^^^^^^^ +Should you wish to add a certain kernel and initrd image to the list of +bootable kernels use `proxmox-boot-tool kernel add`. -`systemd-boot` is configured via the file `loader/loader.conf` in the root -directory of an EFI System Partition (ESP). See the `loader.conf(5)` manpage -for details. +For example run the following to add the kernel with ABI version `5.0.15-1-pve` +to the list of kernels to keep installed and synced to all ESPs: -Each bootloader entry is placed in a file of its own in the directory -`loader/entries/` +---- +# proxmox-boot-tool kernel add 5.0.15-1-pve +---- -An example entry.conf looks like this (`/` refers to the root of the ESP): +`proxmox-boot-tool kernel list` will list all kernel versions currently selected +for booting: ---- -title Proxmox -version 5.0.15-1-pve -options root=ZFS=rpool/ROOT/pve-1 boot=zfs -linux /EFI/proxmox/5.0.15-1-pve/vmlinuz-5.0.15-1-pve -initrd /EFI/proxmox/5.0.15-1-pve/initrd.img-5.0.15-1-pve +# proxmox-boot-tool kernel list +Manually selected kernels: +5.0.15-1-pve + +Automatically selected kernels: +5.0.12-1-pve +4.15.18-18-pve ---- +Run `proxmox-boot-tool kernel remove` to remove a kernel from the list of +manually selected kernels, for example: -.Manually keeping a kernel bootable +---- +# proxmox-boot-tool kernel remove 5.0.15-1-pve +---- -Should you wish to add a certain kernel and initrd image to the list of -bootable kernels you need to: +NOTE: It's required to run `proxmox-boot-tool refresh` to update all EFI System +Partitions (ESPs) after a manual kernel addition or removal from above. + + +[[sysboot_determine_bootloader_used]] +Determine which Bootloader is Used +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +[thumbnail="screenshot/boot-grub.png", float="left"] -* create a directory on the ESP (e.g. `/EFI/personalkernel`) -* copy the kernel and initrd image to that directory -* create a entry for this kernel in `/loader/entries/*.conf` +The simplest and most reliable way to determine which bootloader is used, is to +watch the boot process of the {pve} node. -NOTE: do not use `/EFI/proxmox` as directory since all entries there can be -removed by `/etc/kernel/postinst.d/zz-pve-efiboot` +You will either see the blue box of `grub` or the simple black on white +`systemd-boot`. + +[thumbnail="screenshot/boot-systemdboot.png"] + +Determining the bootloader from a running system might not be 100% accurate. The +safest way is to run the following command: -Example (keeping kernel 5.0.15-1-pve and copying to an ESP mounted on -`/mnt/esp`): ---- -mkdir /mnt/esp/EFI/preferred-kernel -cp /boot/initrd.img-5.0.15-1-pve /boot/vmlinuz-5.0.15-1-pve /mnt/esp/EFI/preferred-kernel -echo -n "title Preferred Kernel -version 5.0.15-1-pve -linux /mnt/esp/EFI/preferred-kernel/vmlinuz-5.0.15-1-pve -initrd /mnt/esp/EFI/preferred-kernel/initrd.img-5.0.15-1-pve -options " > /mnt/esp/loader/entries/preferred.conf -cat /etc/kernel/cmdline >> /mnt/esp/loader/entries/preferred.conf +# efibootmgr -v ---- -[[sysboot_systemd_boot_setup]] -.Setting up a new partition for use as synced ESP +If it returns a message that EFI variables are not supported, `grub` is used in +BIOS/Legacy mode. -To format and initialize a partition as synced ESP, e.g., after replacing a -failed vdev in an rpool, or when converting an existing system that pre-dates -the sync mechanism, `pve-efiboot-tool` from `pve-kernel-helpers` can be used. +If the output contains a line that looks similar to the following, `grub` is +used in UEFI mode. -WARNING: the `format` command will format the ``, make sure to pass -in the right device/partition! +---- +Boot0005* proxmox [...] File(\EFI\proxmox\grubx64.efi) +---- -For example, to format an empty partition `/dev/sda2` as ESP, run the following: +If the output contains a line similar to the following, `systemd-boot` is used. ---- -pve-efiboot-tool format /dev/sda2 +Boot0006* Linux Boot Manager [...] File(\EFI\systemd\systemd-bootx64.efi) ---- -To setup an existing, unmounted ESP located on `/dev/sda2` for inclusion in -{pve}'s synchronization mechanism, use the following: +By running: ---- -pve-efiboot-tool init /dev/sda2 +# proxmox-boot-tool status ---- -Afterwards `/etc/kernel/pve-efiboot-uuids` should contain a new line with the -UUID of the newly added partition. The `init` command will also automatically -trigger a refresh of all configured ESPs. +you can find out if `proxmox-boot-tool` is configured, which is a good +indication of how the system is booted. -[[sysboot_systemd_boot_refresh]] -.Updating the configuration on all ESPs -To copy and configure all bootable kernels and keep all ESPs listed in -`/etc/kernel/pve-efiboot-uuids` in sync you just need to run `pve-efiboot-tool -refresh`. -(The equivalent to running `update-grub` on Systems being booted with `grub`). +[[sysboot_grub]] +Grub +~~~~ + +`grub` has been the de-facto standard for booting Linux systems for many years +and is quite well documented +footnote:[Grub Manual https://www.gnu.org/software/grub/manual/grub/grub.html]. + +Configuration +^^^^^^^^^^^^^ +Changes to the `grub` configuration are done via the defaults file +`/etc/default/grub` or config snippets in `/etc/default/grub.d`. To regenerate +the configuration file after a change to the configuration run: +footnote:[Systems using `proxmox-boot-tool` will call `proxmox-boot-tool +refresh` upon `update-grub`.] + +---- +# update-grub +---- + + +[[sysboot_systemd_boot]] +Systemd-boot +~~~~~~~~~~~~ + +`systemd-boot` is a lightweight EFI bootloader. It reads the kernel and initrd +images directly from the EFI Service Partition (ESP) where it is installed. +The main advantage of directly loading the kernel from the ESP is that it does +not need to reimplement the drivers for accessing the storage. In {pve} +xref:sysboot_proxmox_boot_tool[`proxmox-boot-tool`] is used to keep the +configuration on the ESPs synchronized. + +[[sysboot_systemd_boot_config]] +Configuration +^^^^^^^^^^^^^ + +`systemd-boot` is configured via the file `loader/loader.conf` in the root +directory of an EFI System Partition (ESP). See the `loader.conf(5)` manpage +for details. + +Each bootloader entry is placed in a file of its own in the directory +`loader/entries/` + +An example entry.conf looks like this (`/` refers to the root of the ESP): + +---- +title Proxmox +version 5.0.15-1-pve +options root=ZFS=rpool/ROOT/pve-1 boot=zfs +linux /EFI/proxmox/5.0.15-1-pve/vmlinuz-5.0.15-1-pve +initrd /EFI/proxmox/5.0.15-1-pve/initrd.img-5.0.15-1-pve +---- -This is necessary should you make changes to the kernel commandline, or want to -sync all kernels and initrds after regenerating the latter. [[sysboot_edit_kernel_cmdline]] -Editing the kernel commandline +Editing the Kernel Commandline ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ You can modify the kernel commandline in the following places, depending on the -bootloarder used: +bootloader used: .Grub @@ -195,6 +283,6 @@ The kernel commandline needs to be placed in the variable .Systemd-boot -The kernel commandline needs to be placed as line in `/etc/kernel/cmdline` -Running `/etc/kernel/postinst.d/zz-pve-efiboot` sets it as `option` line for -all config files in `loader/entries/proxmox-*.conf`. +The kernel commandline needs to be placed as one line in `/etc/kernel/cmdline`. +To apply your changes, run `proxmox-boot-tool refresh`, which sets it as the +`option` line for all config files in `loader/entries/proxmox-*.conf`.