X-Git-Url: https://git.proxmox.com/?a=blobdiff_plain;f=system-booting.adoc;h=30621a6a7bfccaf8bb2c04bc43a01c64739994d0;hb=5a8fec5903ce678f27608f5a28e79f2be30f5b96;hp=f40a469a64e53cbac53d77a75b68c6d2cc890ebd;hpb=1722c45bca48cb64d22a13c03e0a5398f11d4fc9;p=pve-docs.git diff --git a/system-booting.adoc b/system-booting.adoc index f40a469..30621a6 100644 --- a/system-booting.adoc +++ b/system-booting.adoc @@ -12,19 +12,13 @@ For EFI Systems installed with ZFS as the root filesystem `systemd-boot` is used. All other deployments use the standard `grub` bootloader (this usually also applies to systems which are installed on top of Debian). + [[sysboot_installer_part_scheme]] -Partitioning scheme used by the installer +Partitioning Scheme Used by the Installer ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -The {pve} installer creates 3 partitions on the bootable disks selected for -installation. The bootable disks are: - -* For Installations with `ext4` or `xfs` the selected disk - -* For ZFS installations all disks belonging to the first `vdev`: -** The first disk for RAID0 -** All disks for RAID1, RAIDZ1, RAIDZ2, RAIDZ3 -** The first two disks for RAID10 +The {pve} installer creates 3 partitions on all disks selected for +installation. The created partitions are: @@ -35,103 +29,124 @@ The created partitions are: * a third partition spanning the set `hdsize` parameter or the remaining space used for the chosen storage type +Systems using ZFS as root filesystem are booted with a kernel and initrd image +stored on the 512 MB EFI System Partition. For legacy BIOS systems, `grub` is +used, for EFI systems `systemd-boot` is used. Both are installed and configured +to point to the ESPs. + `grub` in BIOS mode (`--target i386-pc`) is installed onto the BIOS Boot -Partition of all bootable disks for supporting older systems. +Partition of all selected disks on all systems booted with `grub` +footnote:[These are all installs with root on `ext4` or `xfs` and installs +with root on ZFS on non-EFI systems]. -[[sysboot_grub]] -Grub -~~~~ +[[sysboot_proxmox_boot_tool]] +Synchronizing the content of the ESP with `proxmox-boot-tool` +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -`grub` has been the de-facto standard for booting Linux systems for many years -and is quite well documented -footnote:[Grub Manual https://www.gnu.org/software/grub/manual/grub/grub.html]. +`proxmox-boot-tool` is a utility used to keep the contents of the EFI System +Partitions properly configured and synchronized. It copies certain kernel +versions to all ESPs and configures the respective bootloader to boot from +the `vfat` formatted ESPs. In the context of ZFS as root filesystem this means +that you can use all optional features on your root pool instead of the subset +which is also present in the ZFS implementation in `grub` or having to create a +separate small boot-pool footnote:[Booting ZFS on root with grub +https://github.com/zfsonlinux/zfs/wiki/Debian-Stretch-Root-on-ZFS]. -The kernel and initrd images are taken from `/boot` and its configuration file -`/boot/grub/grub.cfg` gets updated by the kernel installation process. +In setups with redundancy all disks are partitioned with an ESP, by the +installer. This ensures the system boots even if the first boot device fails +or if the BIOS can only boot from a particular disk. -Configuration -^^^^^^^^^^^^^ -Changes to the `grub` configuration are done via the defaults file -`/etc/default/grub` or config snippets in `/etc/default/grub.d`. To regenerate -the `/boot/grub/grub.cfg` after a change to the configuration run: +The ESPs are not kept mounted during regular operation. This helps to prevent +filesystem corruption to the `vfat` formatted ESPs in case of a system crash, +and removes the need to manually adapt `/etc/fstab` in case the primary boot +device fails. + +`proxmox-boot-tool` handles the following tasks: + +* formatting and setting up a new partition +* copying and configuring new kernel images and initrd images to all listed ESPs +* synchronizing the configuration on kernel upgrades and other maintenance tasks +* managing the list of kernel versions which are synchronized +* configuring the boot-loader to boot a particular kernel version (pinning) + + +You can view the currently configured ESPs and their state by running: ---- -`update-grub`. +# proxmox-boot-tool status ---- -[[sysboot_systemd_boot]] -Systemd-boot -~~~~~~~~~~~~ +[[sysboot_proxmox_boot_setup]] +.Setting up a new partition for use as synced ESP -`systemd-boot` is a lightweight EFI bootloader. It reads the kernel and initrd -images directly from the EFI Service Partition (ESP) where it is installed. -The main advantage of directly loading the kernel from the ESP is that it does -not need to reimplement the drivers for accessing the storage. In the context -of ZFS as root filesystem this means that you can use all optional features on -your root pool instead of the subset which is also present in the ZFS -implementation in `grub` or having to create a separate small boot-pool -footnote:[Booting ZFS on root with grub https://github.com/zfsonlinux/zfs/wiki/Debian-Stretch-Root-on-ZFS]. - -In setups with redundancy (RAID1, RAID10, RAIDZ*) all bootable disks (those -being part of the first `vdev`) are partitioned with an ESP. This ensures the -system boots even if the first boot device fails. The ESPs are kept in sync by -a kernel postinstall hook script `/etc/kernel/postinst.d/zz-pve-efiboot`. The -script copies certain kernel versions and the initrd images to `EFI/proxmox/` -on the root of each ESP and creates the appropriate config files in -`loader/entries/proxmox-*.conf`. +To format and initialize a partition as synced ESP, e.g., after replacing a +failed vdev in an rpool, or when converting an existing system that pre-dates +the sync mechanism, `proxmox-boot-tool` from `pve-kernel-helpers` can be used. -The following kernel versions are configured by default: +WARNING: the `format` command will format the ``, make sure to pass +in the right device/partition! -* the currently running kernel -* the version being newly installed on package updates -* the two latest kernels -* the latest version of each kernel series (e.g. 4.15, 5.0). +For example, to format an empty partition `/dev/sda2` as ESP, run the following: -The ESPs are not kept mounted during regular operation, in contrast to `grub`, -which keeps an ESP mounted on `/boot/efi`. This helps to prevent filesystem -corruption to the `vfat` formatted ESPs in case of a system crash, and removes -the need to manually adapt `/etc/fstab` in case the primary boot device fails. +---- +# proxmox-boot-tool format /dev/sda2 +---- -[[sysboot_systemd_boot_config]] -Configuration -^^^^^^^^^^^^^ +To setup an existing, unmounted ESP located on `/dev/sda2` for inclusion in +{pve}'s kernel update synchronization mechanism, use the following: -`systemd-boot` is configured via the file `loader/loader.conf` in the root -directory of an EFI System Partition (ESP). See the `loader.conf(5)` manpage -for details. +---- +# proxmox-boot-tool init /dev/sda2 +---- -Each bootloader entry is placed in a file of its own in the directory -`loader/entries/` +Afterwards `/etc/kernel/proxmox-boot-uuids` should contain a new line with the +UUID of the newly added partition. The `init` command will also automatically +trigger a refresh of all configured ESPs. -An example entry.conf looks like this (`/` refers to the root of the ESP): +[[sysboot_proxmox_boot_refresh]] +.Updating the configuration on all ESPs + +To copy and configure all bootable kernels and keep all ESPs listed in +`/etc/kernel/proxmox-boot-uuids` in sync you just need to run: ---- -title Proxmox -version 5.0.15-1-pve -options root=ZFS=rpool/ROOT/pve-1 boot=zfs -linux /EFI/proxmox/5.0.15-1-pve/vmlinuz-5.0.15-1-pve -initrd /EFI/proxmox/5.0.15-1-pve/initrd.img-5.0.15-1-pve +# proxmox-boot-tool refresh ---- +(The equivalent to running `update-grub` systems with `ext4` or `xfs` on root). + +This is necessary should you make changes to the kernel commandline, or want to +sync all kernels and initrds. + +NOTE: Both `update-initramfs` and `apt` (when necessary) will automatically +trigger a refresh. +.Kernel Versions considered by `proxmox-boot-tool` +The following kernel versions are configured by default: + +* the currently running kernel +* the version being newly installed on package updates +* the two latest already installed kernels +* the latest version of the second-to-last kernel series (e.g. 5.0, 5.3), if applicable +* any manually selected kernels .Manually keeping a kernel bootable Should you wish to add a certain kernel and initrd image to the list of -bootable kernel use `pve-efiboot-tool kernel add`. +bootable kernels use `proxmox-boot-tool kernel add`. For example run the following to add the kernel with ABI version `5.0.15-1-pve` to the list of kernels to keep installed and synced to all ESPs: ---- -pve-efiboot-tool kernel add 5.0.15-1-pve +# proxmox-boot-tool kernel add 5.0.15-1-pve ---- -`pve-efiboot-tool kernel list` will list all kernel versions currently selected +`proxmox-boot-tool kernel list` will list all kernel versions currently selected for booting: ---- -# pve-efiboot-tool kernel list +# proxmox-boot-tool kernel list Manually selected kernels: 5.0.15-1-pve @@ -140,58 +155,125 @@ Automatically selected kernels: 4.15.18-18-pve ---- -Run `pve-efiboot-tool remove` to remove a kernel from the list of manually -selected kernels, for example: +Run `proxmox-boot-tool kernel remove` to remove a kernel from the list of +manually selected kernels, for example: ---- -pve-efiboot-tool kernel remove 5.0.15-1-pve +# proxmox-boot-tool kernel remove 5.0.15-1-pve ---- +NOTE: It's required to run `proxmox-boot-tool refresh` to update all EFI System +Partitions (ESPs) after a manual kernel addition or removal from above. -[[sysboot_systemd_boot_setup]] -.Setting up a new partition for use as synced ESP -To format and initialize a partition as synced ESP, e.g., after replacing a -failed vdev in an rpool, or when converting an existing system that pre-dates -the sync mechanism, `pve-efiboot-tool` from `pve-kernel-helpers` can be used. +[[sysboot_determine_bootloader_used]] +Determine which Bootloader is Used +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -WARNING: the `format` command will format the ``, make sure to pass -in the right device/partition! +[thumbnail="screenshot/boot-grub.png", float="left"] + +The simplest and most reliable way to determine which bootloader is used, is to +watch the boot process of the {pve} node. + +You will either see the blue box of `grub` or the simple black on white +`systemd-boot`. + +[thumbnail="screenshot/boot-systemdboot.png"] + +Determining the bootloader from a running system might not be 100% accurate. The +safest way is to run the following command: -For example, to format an empty partition `/dev/sda2` as ESP, run the following: ---- -pve-efiboot-tool format /dev/sda2 +# efibootmgr -v ---- -To setup an existing, unmounted ESP located on `/dev/sda2` for inclusion in -{pve}'s kernel update synchronization mechanism, use the following: +If it returns a message that EFI variables are not supported, `grub` is used in +BIOS/Legacy mode. + +If the output contains a line that looks similar to the following, `grub` is +used in UEFI mode. ---- -pve-efiboot-tool init /dev/sda2 +Boot0005* proxmox [...] File(\EFI\proxmox\grubx64.efi) ---- -Afterwards `/etc/kernel/pve-efiboot-uuids` should contain a new line with the -UUID of the newly added partition. The `init` command will also automatically -trigger a refresh of all configured ESPs. +If the output contains a line similar to the following, `systemd-boot` is used. -[[sysboot_systemd_boot_refresh]] -.Updating the configuration on all ESPs +---- +Boot0006* Linux Boot Manager [...] File(\EFI\systemd\systemd-bootx64.efi) +---- -To copy and configure all bootable kernels and keep all ESPs listed in -`/etc/kernel/pve-efiboot-uuids` in sync you just need to run `pve-efiboot-tool -refresh`. -(The equivalent to running `update-grub` on Systems being booted with `grub`). +By running: + +---- +# proxmox-boot-tool status +---- + +you can find out if `proxmox-boot-tool` is configured, which is a good +indication of how the system is booted. + + +[[sysboot_grub]] +Grub +~~~~ + +`grub` has been the de-facto standard for booting Linux systems for many years +and is quite well documented +footnote:[Grub Manual https://www.gnu.org/software/grub/manual/grub/grub.html]. + +Configuration +^^^^^^^^^^^^^ +Changes to the `grub` configuration are done via the defaults file +`/etc/default/grub` or config snippets in `/etc/default/grub.d`. To regenerate +the configuration file after a change to the configuration run: +footnote:[Systems using `proxmox-boot-tool` will call `proxmox-boot-tool +refresh` upon `update-grub`.] + +---- +# update-grub +---- + + +[[sysboot_systemd_boot]] +Systemd-boot +~~~~~~~~~~~~ + +`systemd-boot` is a lightweight EFI bootloader. It reads the kernel and initrd +images directly from the EFI Service Partition (ESP) where it is installed. +The main advantage of directly loading the kernel from the ESP is that it does +not need to reimplement the drivers for accessing the storage. In {pve} +xref:sysboot_proxmox_boot_tool[`proxmox-boot-tool`] is used to keep the +configuration on the ESPs synchronized. + +[[sysboot_systemd_boot_config]] +Configuration +^^^^^^^^^^^^^ + +`systemd-boot` is configured via the file `loader/loader.conf` in the root +directory of an EFI System Partition (ESP). See the `loader.conf(5)` manpage +for details. + +Each bootloader entry is placed in a file of its own in the directory +`loader/entries/` + +An example entry.conf looks like this (`/` refers to the root of the ESP): + +---- +title Proxmox +version 5.0.15-1-pve +options root=ZFS=rpool/ROOT/pve-1 boot=zfs +linux /EFI/proxmox/5.0.15-1-pve/vmlinuz-5.0.15-1-pve +initrd /EFI/proxmox/5.0.15-1-pve/initrd.img-5.0.15-1-pve +---- -This is necessary should you make changes to the kernel commandline, or want to -sync all kernels and initrds after regenerating the latter. [[sysboot_edit_kernel_cmdline]] -Editing the kernel commandline +Editing the Kernel Commandline ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ You can modify the kernel commandline in the following places, depending on the -bootloarder used: +bootloader used: .Grub @@ -202,6 +284,64 @@ The kernel commandline needs to be placed in the variable .Systemd-boot -The kernel commandline needs to be placed as line in `/etc/kernel/cmdline` -Running `/etc/kernel/postinst.d/zz-pve-efiboot` sets it as `option` line for -all config files in `loader/entries/proxmox-*.conf`. +The kernel commandline needs to be placed as one line in `/etc/kernel/cmdline`. +To apply your changes, run `proxmox-boot-tool refresh`, which sets it as the +`option` line for all config files in `loader/entries/proxmox-*.conf`. + + +[[sysboot_kernel_pin]] +Override the Kernel-Version for next Boot +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +To select a kernel that is not currently the default kernel, you can either: + +* use the boot loader menu that is displayed at the beginning of the boot + process +* use the `proxmox-boot-tool` to `pin` the system to a kernel version either + once or permanently (until pin is reset). + +This should help you work around incompatibilities between a newer kernel +version and the hardware. + +NOTE: Such a pin should be removed as soon as possible so that all current +security patches of the latest kernel are also applied to the system. + +For example: To permanently select the version `5.15.30-1-pve` for booting you +would run: + +---- +# proxmox-boot-tool kernel pin 5.15.30-1-pve +---- + +TIP: The pinning functionality works for all {pve} systems, not only those using +`proxmox-boot-tool` to synchronize the contents of the ESPs, if your system +does not use `proxmox-boot-tool` for synchronizing you can also skip the +`proxmox-boot-tool refresh` call in the end. + +You can also set a kernel version to be booted on the next system boot only. +This is for example useful to test if an updated kernel has resolved an issue, +which caused you to `pin` a version in the first place: + +---- +# proxmox-boot-tool kernel pin 5.15.30-1-pve --next-boot +---- + +To remove any pinned version configuration use the `unpin` subcommand: + +---- +# proxmox-boot-tool kernel unpin +---- + +While `unpin` has a `--next-boot` option as well, it is used to clear a pinned +version set with `--next-boot`. As that happens already automatically on boot, +invonking it manually is of little use. + +After setting, or clearing pinned versions you also need to synchronize the +content and configuration on the ESPs by running the `refresh` subcommand. + +TIP: You will be prompted to automatically do for `proxmox-boot-tool` managed +systems if you call the tool interactively. + +---- +# proxmox-boot-tool refresh +----