2 .. _chapter-systembooting:
7 `Proxmox Backup`_ currently uses one of two bootloaders, depending on the disk setup
8 selected in the installer.
10 For EFI Systems installed with ZFS as the root filesystem ``systemd-boot`` is
11 used. All other deployments use the standard ``grub`` bootloader (this usually
12 also applies to systems which are installed on top of Debian).
15 .. _systembooting-installer-part-scheme:
17 Partitioning Scheme Used by the Installer
18 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
20 The `Proxmox Backup`_ installer creates 3 partitions on all disks selected for
23 The created partitions are:
25 * A 1 MB BIOS Boot Partition (gdisk type EF02)
27 * A 512 MB EFI System Partition (ESP, gdisk type EF00)
29 * A third partition spanning the configured ``hdsize`` parameter or the
30 remaining space available for the chosen storage type
32 Systems using ZFS as a root filesystem are booted with a kernel and initrd image
33 stored on the 512 MB EFI System Partition. For legacy BIOS systems, ``grub`` is
34 used, for EFI systems ``systemd-boot`` is used. Both are installed and configured
37 ``grub`` in BIOS mode (``--target i386-pc``) is installed onto the BIOS Boot
38 Partition of all selected disks on all systems booted with ``grub`` (that is,
39 all installs with root on ``ext4`` or ``xfs``, and installs with root on ZFS on
43 .. _systembooting-proxmox-boot-tool:
45 Synchronizing the Content of the ESP with ``proxmox-boot-tool``
46 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
48 ``proxmox-boot-tool`` is a utility used to keep the contents of the EFI System
49 Partitions properly configured and synchronized. It copies certain kernel
50 versions to all ESPs and configures the respective bootloader to boot from
51 the ``vfat`` formatted ESPs. In the context of ZFS as root filesystem, this means
52 that you can use all the optional features on your root pool, instead of the subset
53 which is also present in the ZFS implementation in ``grub`` or having to create a
54 small, separate boot-pool (see: `Booting ZFS on root with grub
55 <https://github.com/zfsonlinux/zfs/wiki/Debian-Stretch-Root-on-ZFS>`_).
57 In setups with redundancy, all disks are partitioned with an ESP by the
58 installer. This ensures the system boots, even if the first boot device fails
59 or if the BIOS can only boot from a particular disk.
61 The ESPs are not kept mounted during regular operation. This helps to prevent
62 filesystem corruption in the ``vfat`` formatted ESPs in case of a system crash,
63 and removes the need to manually adapt ``/etc/fstab`` in case the primary boot
66 ``proxmox-boot-tool`` handles the following tasks:
68 * Formatting and setting up a new partition
69 * Copying and configuring new kernel images and initrd images to all listed ESPs
70 * Synchronizing the configuration on kernel upgrades and other maintenance tasks
71 * Managing the list of kernel versions which are synchronized
72 * Configuring the boot-loader to boot a particular kernel version (pinning)
75 You can view the currently configured ESPs and their state by running:
77 .. code-block:: console
79 # proxmox-boot-tool status
81 .. _systembooting-proxmox-boot-setup:
83 Setting up a New Partition for use as Synced ESP
84 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
86 To format and initialize a partition as synced ESP, for example, after replacing a
87 failed vdev in an rpool, ``proxmox-boot-tool`` from ``pve-kernel-helper`` can be used.
89 WARNING: the ``format`` command will format the ``<partition>``. Make sure to pass
90 in the right device/partition!
92 For example, to format an empty partition ``/dev/sda2`` as ESP, run the following:
94 .. code-block:: console
96 # proxmox-boot-tool format /dev/sda2
98 To setup an existing, unmounted ESP located on ``/dev/sda2`` for inclusion in
99 `Proxmox Backup`_'s kernel update synchronization mechanism, use the following:
101 .. code-block:: console
103 # proxmox-boot-tool init /dev/sda2
105 Following this, `/etc/kernel/proxmox-boot-uuids`` should contain a new line with the
106 UUID of the newly added partition. The ``init`` command will also automatically
107 trigger a refresh of all configured ESPs.
109 .. _systembooting-proxmox-boot-refresh:
111 Updating the Configuration on all ESPs
112 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
114 To copy and configure all bootable kernels and keep all ESPs listed in
115 ``/etc/kernel/proxmox-boot-uuids`` in sync, you just need to run:
117 .. code-block:: console
119 # proxmox-boot-tool refresh
121 (Equivalent to running ``update-grub`` on systems with ``ext4`` or ``xfs`` on root).
123 This is necessary after making changes to the kernel commandline, or if you want
124 to sync all kernels and initrds.
126 .. NOTE:: Both ``update-initramfs`` and ``apt`` (when necessary) will automatically
129 Kernel Versions Considered by ``proxmox-boot-tool``
130 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
132 The following kernel versions are configured by default:
134 * The currently running kernel
135 * The version being newly installed on package updates
136 * The two latest, already installed kernels
137 * The latest version of the second-to-last kernel series (e.g. 5.0, 5.3), if applicable
138 * Any manually selected kernels
140 Manually Keeping a Kernel Bootable
141 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
143 Should you wish to add a certain kernel and initrd image to the list of
144 bootable kernels, use ``proxmox-boot-tool kernel add``.
146 For example, run the following to add the kernel with ABI version ``5.0.15-1-pve``
147 to the list of kernels to keep installed and synced to all ESPs:
149 .. code-block:: console
151 # proxmox-boot-tool kernel add 5.0.15-1-pve
154 ``proxmox-boot-tool kernel list`` will list all kernel versions currently selected
157 .. code-block:: console
159 # proxmox-boot-tool kernel list
160 Manually selected kernels:
163 Automatically selected kernels:
167 Run ``proxmox-boot-tool kernel remove`` to remove a kernel from the list of
168 manually selected kernels, for example:
170 .. code-block:: console
172 # proxmox-boot-tool kernel remove 5.0.15-1-pve
175 .. NOTE:: It's required to run ``proxmox-boot-tool refresh`` to update all EFI System
176 Partitions (ESPs) after a manual kernel addition or removal from above.
179 .. _systembooting-determine-bootloader:
181 Determine which Bootloader is Used
182 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
184 .. image:: images/screenshots/boot-grub.png
185 :target: _images/boot-grub.png
187 :alt: Grub boot screen
189 The simplest and most reliable way to determine which bootloader is used, is to
190 watch the boot process of the `Proxmox Backup`_ node.
193 You will either see the blue box of ``grub`` or the simple black on white
196 .. image:: images/screenshots/boot-systemdboot.png
197 :target: _images/boot-systemdboot.png
199 :alt: systemd-boot screen
201 Determining the bootloader from a running system might not be 100% accurate. The
202 most reliable way is to run the following command:
205 .. code-block:: console
210 If it returns a message that EFI variables are not supported, ``grub`` is used in
213 If the output contains a line that looks similar to the following, ``grub`` is
216 .. code-block:: console
218 Boot0005* proxmox [...] File(\EFI\proxmox\grubx64.efi)
221 If the output contains a line similar to the following, ``systemd-boot`` is used.
223 .. code-block:: console
225 Boot0006* Linux Boot Manager [...] File(\EFI\systemd\systemd-bootx64.efi)
228 By running the following command, you can find out if ``proxmox-boot-tool`` is
229 configured, which is a good indication of how the system is booted:
231 .. code-block:: console
233 # proxmox-boot-tool status
236 .. _systembooting-grub:
241 ``grub`` has been the de facto standard for booting Linux systems for many years
242 and is quite well documented
243 (see the `Grub Manual
244 <https://www.gnu.org/software/grub/manual/grub/grub.html>`_).
249 Changes to the ``grub`` configuration are done via the defaults file
250 ``/etc/default/grub`` or via config snippets in ``/etc/default/grub.d``. To
251 regenerate the configuration file after a change to the configuration, run:
253 .. code-block:: console
257 .. NOTE:: Systems using ``proxmox-boot-tool`` will call
258 ``proxmox-boot-tool refresh`` upon ``update-grub``
260 .. _systembooting-systemdboot:
265 ``systemd-boot`` is a lightweight EFI bootloader. It reads the kernel and initrd
266 images directly from the EFI Service Partition (ESP) where it is installed.
267 The main advantage of directly loading the kernel from the ESP is that it does
268 not need to reimplement the drivers for accessing the storage. In `Proxmox
269 Backup`_, :ref:`proxmox-boot-tool <systembooting-proxmox-boot-tool>` is used to
270 keep the configuration on the ESPs synchronized.
272 .. _systembooting-systemd-boot-config:
277 ``systemd-boot`` is configured via the file ``loader/loader.conf`` in the root
278 directory of an EFI System Partition (ESP). See the ``loader.conf(5)`` manpage
281 Each bootloader entry is placed in a file of its own, in the directory
284 An example entry.conf looks like this (``/`` refers to the root of the ESP):
286 .. code-block:: console
290 options root=ZFS=rpool/ROOT/pve-1 boot=zfs
291 linux /EFI/proxmox/5.0.15-1-pve/vmlinuz-5.0.15-1-pve
292 initrd /EFI/proxmox/5.0.15-1-pve/initrd.img-5.0.15-1-pve
295 .. _systembooting-edit-kernel-cmdline:
297 Editing the Kernel Commandline
298 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
300 You can modify the kernel commandline in the following places, depending on the
306 The kernel commandline needs to be placed in the variable
307 ``GRUB_CMDLINE_LINUX_DEFAULT`` in the file ``/etc/default/grub``. Running
308 ``update-grub`` appends its content to all ``linux`` entries in
309 ``/boot/grub/grub.cfg``.
314 The kernel commandline needs to be placed as one line in ``/etc/kernel/cmdline``.
315 To apply your changes, run ``proxmox-boot-tool refresh``, which sets it as the
316 ``option`` line for all config files in ``loader/entries/proxmox-*.conf``.
319 .. _systembooting-kernel-pin:
321 Override the Kernel-Version for next Boot
322 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
324 To select a kernel that is not currently the default kernel, you can either:
326 * Use the boot loader menu that is displayed at the beginning of the boot
328 * Use the ``proxmox-boot-tool`` to ``pin`` the system to a kernel version either
329 once or permanently (until pin is reset).
331 This should help you work around incompatibilities between a newer kernel
332 version and the hardware.
334 .. NOTE:: Such a pin should be removed as soon as possible, so that all recent
335 security patches from the latest kernel are also applied to the system.
337 For example, to permanently select the version ``5.15.30-1-pve`` for booting, you
340 .. code-block:: console
342 # proxmox-boot-tool kernel pin 5.15.30-1-pve
345 .. TIP:: The pinning functionality works for all `Proxmox Backup`_ systems, not only those using
346 ``proxmox-boot-tool`` to synchronize the contents of the ESPs, if your system
347 does not use ``proxmox-boot-tool`` for synchronizing, you can also skip the
348 ``proxmox-boot-tool refresh`` call in the end.
350 You can also set a kernel version to be booted on the next system boot only.
351 This is useful, for example, to test if an updated kernel has resolved an issue,
352 which caused you to ``pin`` a version in the first place:
354 .. code-block:: console
356 # proxmox-boot-tool kernel pin 5.15.30-1-pve --next-boot
359 To remove any pinned version configuration, use the ``unpin`` subcommand:
361 .. code-block:: console
363 # proxmox-boot-tool kernel unpin
365 While ``unpin`` has a ``--next-boot`` option as well, it is used to clear a pinned
366 version set with ``--next-boot``. As that happens already automatically on boot,
367 invoking it manually is of little use.
369 After setting or clearing pinned versions, you also need to synchronize the
370 content and configuration on the ESPs by running the ``refresh`` subcommand.
372 .. TIP:: You will be prompted to automatically do for ``proxmox-boot-tool`` managed
373 systems if you call the tool interactively.
375 .. code-block:: console
377 # proxmox-boot-tool refresh