]>
Commit | Line | Data |
---|---|---|
69055103 TL |
1 | [[sysboot]] |
2 | Host Bootloader | |
3 | --------------- | |
62de1c14 SI |
4 | ifdef::wiki[] |
5 | :pve-toplevel: | |
6 | endif::wiki[] | |
7 | ||
69055103 TL |
8 | {pve} currently uses one of two bootloaders depending on the disk setup |
9 | selected in the installer. | |
62de1c14 SI |
10 | |
11 | For EFI Systems installed with ZFS as the root filesystem `systemd-boot` is | |
952ee606 FG |
12 | used, unless Secure Boot is enabled. All other deployments use the standard |
13 | `grub` bootloader (this usually also applies to systems which are installed on | |
14 | top of Debian). | |
62de1c14 | 15 | |
dfa919a3 TL |
16 | |
17 | [[sysboot_installer_part_scheme]] | |
18 | Partitioning Scheme Used by the Installer | |
19 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | |
20 | ||
cb04e768 SI |
21 | The {pve} installer creates 3 partitions on all disks selected for |
22 | installation. | |
dfa919a3 TL |
23 | |
24 | The created partitions are: | |
25 | ||
26 | * a 1 MB BIOS Boot Partition (gdisk type EF02) | |
27 | ||
28 | * a 512 MB EFI System Partition (ESP, gdisk type EF00) | |
29 | ||
30 | * a third partition spanning the set `hdsize` parameter or the remaining space | |
31 | used for the chosen storage type | |
32 | ||
cb04e768 | 33 | Systems using ZFS as root filesystem are booted with a kernel and initrd image |
952ee606 FG |
34 | stored on the 512 MB EFI System Partition. For legacy BIOS systems, and EFI |
35 | systems with Secure Boot enabled, `grub` is used, for EFI systems without | |
36 | Secure Boot, `systemd-boot` is used. Both are installed and configured to point | |
37 | to the ESPs. | |
cb04e768 | 38 | |
dfa919a3 | 39 | `grub` in BIOS mode (`--target i386-pc`) is installed onto the BIOS Boot |
cb04e768 SI |
40 | Partition of all selected disks on all systems booted with `grub` |
41 | footnote:[These are all installs with root on `ext4` or `xfs` and installs | |
42 | with root on ZFS on non-EFI systems]. | |
43 | ||
44 | ||
45 | [[sysboot_proxmox_boot_tool]] | |
46 | Synchronizing the content of the ESP with `proxmox-boot-tool` | |
47 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | |
48 | ||
49 | `proxmox-boot-tool` is a utility used to keep the contents of the EFI System | |
50 | Partitions properly configured and synchronized. It copies certain kernel | |
51 | versions to all ESPs and configures the respective bootloader to boot from | |
52 | the `vfat` formatted ESPs. In the context of ZFS as root filesystem this means | |
53 | that you can use all optional features on your root pool instead of the subset | |
54 | which is also present in the ZFS implementation in `grub` or having to create a | |
55 | separate small boot-pool footnote:[Booting ZFS on root with grub | |
56 | https://github.com/zfsonlinux/zfs/wiki/Debian-Stretch-Root-on-ZFS]. | |
57 | ||
58 | In setups with redundancy all disks are partitioned with an ESP, by the | |
59 | installer. This ensures the system boots even if the first boot device fails | |
60 | or if the BIOS can only boot from a particular disk. | |
61 | ||
62 | The ESPs are not kept mounted during regular operation. This helps to prevent | |
63 | filesystem corruption to the `vfat` formatted ESPs in case of a system crash, | |
64 | and removes the need to manually adapt `/etc/fstab` in case the primary boot | |
65 | device fails. | |
66 | ||
67 | `proxmox-boot-tool` handles the following tasks: | |
68 | ||
69 | * formatting and setting up a new partition | |
70 | * copying and configuring new kernel images and initrd images to all listed ESPs | |
71 | * synchronizing the configuration on kernel upgrades and other maintenance tasks | |
72 | * managing the list of kernel versions which are synchronized | |
a9a2cdbe | 73 | * configuring the boot-loader to boot a particular kernel version (pinning) |
cb04e768 SI |
74 | |
75 | ||
76 | You can view the currently configured ESPs and their state by running: | |
77 | ||
78 | ---- | |
79 | # proxmox-boot-tool status | |
80 | ---- | |
81 | ||
82 | [[sysboot_proxmox_boot_setup]] | |
83 | .Setting up a new partition for use as synced ESP | |
84 | ||
85 | To format and initialize a partition as synced ESP, e.g., after replacing a | |
86 | failed vdev in an rpool, or when converting an existing system that pre-dates | |
3ad9d61c | 87 | the sync mechanism, `proxmox-boot-tool` from `proxmox-kernel-helper` can be used. |
cb04e768 SI |
88 | |
89 | WARNING: the `format` command will format the `<partition>`, make sure to pass | |
90 | in the right device/partition! | |
91 | ||
92 | For example, to format an empty partition `/dev/sda2` as ESP, run the following: | |
93 | ||
94 | ---- | |
95 | # proxmox-boot-tool format /dev/sda2 | |
96 | ---- | |
97 | ||
98 | To setup an existing, unmounted ESP located on `/dev/sda2` for inclusion in | |
99 | {pve}'s kernel update synchronization mechanism, use the following: | |
100 | ||
101 | ---- | |
102 | # proxmox-boot-tool init /dev/sda2 | |
103 | ---- | |
104 | ||
952ee606 FG |
105 | or |
106 | ||
107 | ---- | |
108 | # proxmox-boot-tool init /dev/sda2 grub | |
109 | ---- | |
110 | ||
111 | to force initialization with Grub instead of systemd-boot, for example for | |
112 | Secure Boot support. | |
113 | ||
cb04e768 SI |
114 | Afterwards `/etc/kernel/proxmox-boot-uuids` should contain a new line with the |
115 | UUID of the newly added partition. The `init` command will also automatically | |
116 | trigger a refresh of all configured ESPs. | |
117 | ||
118 | [[sysboot_proxmox_boot_refresh]] | |
119 | .Updating the configuration on all ESPs | |
120 | ||
121 | To copy and configure all bootable kernels and keep all ESPs listed in | |
122 | `/etc/kernel/proxmox-boot-uuids` in sync you just need to run: | |
123 | ||
124 | ---- | |
125 | # proxmox-boot-tool refresh | |
126 | ---- | |
127 | (The equivalent to running `update-grub` systems with `ext4` or `xfs` on root). | |
128 | ||
129 | This is necessary should you make changes to the kernel commandline, or want to | |
130 | sync all kernels and initrds. | |
131 | ||
132 | NOTE: Both `update-initramfs` and `apt` (when necessary) will automatically | |
133 | trigger a refresh. | |
134 | ||
135 | .Kernel Versions considered by `proxmox-boot-tool` | |
136 | The following kernel versions are configured by default: | |
137 | ||
138 | * the currently running kernel | |
139 | * the version being newly installed on package updates | |
140 | * the two latest already installed kernels | |
141 | * the latest version of the second-to-last kernel series (e.g. 5.0, 5.3), if applicable | |
142 | * any manually selected kernels | |
143 | ||
144 | .Manually keeping a kernel bootable | |
145 | ||
146 | Should you wish to add a certain kernel and initrd image to the list of | |
147 | bootable kernels use `proxmox-boot-tool kernel add`. | |
148 | ||
149 | For example run the following to add the kernel with ABI version `5.0.15-1-pve` | |
150 | to the list of kernels to keep installed and synced to all ESPs: | |
151 | ||
152 | ---- | |
153 | # proxmox-boot-tool kernel add 5.0.15-1-pve | |
154 | ---- | |
155 | ||
156 | `proxmox-boot-tool kernel list` will list all kernel versions currently selected | |
157 | for booting: | |
158 | ||
159 | ---- | |
160 | # proxmox-boot-tool kernel list | |
161 | Manually selected kernels: | |
162 | 5.0.15-1-pve | |
163 | ||
164 | Automatically selected kernels: | |
165 | 5.0.12-1-pve | |
166 | 4.15.18-18-pve | |
167 | ---- | |
168 | ||
169 | Run `proxmox-boot-tool kernel remove` to remove a kernel from the list of | |
170 | manually selected kernels, for example: | |
171 | ||
172 | ---- | |
173 | # proxmox-boot-tool kernel remove 5.0.15-1-pve | |
174 | ---- | |
175 | ||
176 | NOTE: It's required to run `proxmox-boot-tool refresh` to update all EFI System | |
177 | Partitions (ESPs) after a manual kernel addition or removal from above. | |
dfa919a3 TL |
178 | |
179 | ||
48a1482b | 180 | [[sysboot_determine_bootloader_used]] |
dfa919a3 | 181 | Determine which Bootloader is Used |
48a1482b AL |
182 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
183 | ||
184 | [thumbnail="screenshot/boot-grub.png", float="left"] | |
185 | ||
186 | The simplest and most reliable way to determine which bootloader is used, is to | |
187 | watch the boot process of the {pve} node. | |
188 | ||
189 | You will either see the blue box of `grub` or the simple black on white | |
190 | `systemd-boot`. | |
191 | ||
192 | [thumbnail="screenshot/boot-systemdboot.png"] | |
193 | ||
194 | Determining the bootloader from a running system might not be 100% accurate. The | |
195 | safest way is to run the following command: | |
196 | ||
197 | ||
198 | ---- | |
199 | # efibootmgr -v | |
200 | ---- | |
201 | ||
202 | If it returns a message that EFI variables are not supported, `grub` is used in | |
203 | BIOS/Legacy mode. | |
204 | ||
205 | If the output contains a line that looks similar to the following, `grub` is | |
206 | used in UEFI mode. | |
207 | ||
208 | ---- | |
209 | Boot0005* proxmox [...] File(\EFI\proxmox\grubx64.efi) | |
210 | ---- | |
211 | ||
cb04e768 | 212 | If the output contains a line similar to the following, `systemd-boot` is used. |
48a1482b AL |
213 | |
214 | ---- | |
215 | Boot0006* Linux Boot Manager [...] File(\EFI\systemd\systemd-bootx64.efi) | |
216 | ---- | |
217 | ||
cb04e768 SI |
218 | By running: |
219 | ||
220 | ---- | |
221 | # proxmox-boot-tool status | |
222 | ---- | |
223 | ||
224 | you can find out if `proxmox-boot-tool` is configured, which is a good | |
225 | indication of how the system is booted. | |
226 | ||
48a1482b | 227 | |
69055103 | 228 | [[sysboot_grub]] |
62de1c14 SI |
229 | Grub |
230 | ~~~~ | |
231 | ||
232 | `grub` has been the de-facto standard for booting Linux systems for many years | |
233 | and is quite well documented | |
234 | footnote:[Grub Manual https://www.gnu.org/software/grub/manual/grub/grub.html]. | |
235 | ||
62de1c14 SI |
236 | Configuration |
237 | ^^^^^^^^^^^^^ | |
238 | Changes to the `grub` configuration are done via the defaults file | |
239 | `/etc/default/grub` or config snippets in `/etc/default/grub.d`. To regenerate | |
cb04e768 SI |
240 | the configuration file after a change to the configuration run: |
241 | footnote:[Systems using `proxmox-boot-tool` will call `proxmox-boot-tool | |
242 | refresh` upon `update-grub`.] | |
62de1c14 SI |
243 | |
244 | ---- | |
cb04e768 | 245 | # update-grub |
62de1c14 SI |
246 | ---- |
247 | ||
cb04e768 | 248 | |
69055103 | 249 | [[sysboot_systemd_boot]] |
62de1c14 SI |
250 | Systemd-boot |
251 | ~~~~~~~~~~~~ | |
252 | ||
253 | `systemd-boot` is a lightweight EFI bootloader. It reads the kernel and initrd | |
254 | images directly from the EFI Service Partition (ESP) where it is installed. | |
255 | The main advantage of directly loading the kernel from the ESP is that it does | |
cb04e768 SI |
256 | not need to reimplement the drivers for accessing the storage. In {pve} |
257 | xref:sysboot_proxmox_boot_tool[`proxmox-boot-tool`] is used to keep the | |
258 | configuration on the ESPs synchronized. | |
62de1c14 | 259 | |
69055103 | 260 | [[sysboot_systemd_boot_config]] |
62de1c14 SI |
261 | Configuration |
262 | ^^^^^^^^^^^^^ | |
263 | ||
264 | `systemd-boot` is configured via the file `loader/loader.conf` in the root | |
265 | directory of an EFI System Partition (ESP). See the `loader.conf(5)` manpage | |
266 | for details. | |
267 | ||
268 | Each bootloader entry is placed in a file of its own in the directory | |
269 | `loader/entries/` | |
270 | ||
271 | An example entry.conf looks like this (`/` refers to the root of the ESP): | |
272 | ||
273 | ---- | |
274 | title Proxmox | |
275 | version 5.0.15-1-pve | |
276 | options root=ZFS=rpool/ROOT/pve-1 boot=zfs | |
277 | linux /EFI/proxmox/5.0.15-1-pve/vmlinuz-5.0.15-1-pve | |
278 | initrd /EFI/proxmox/5.0.15-1-pve/initrd.img-5.0.15-1-pve | |
279 | ---- | |
280 | ||
281 | ||
69055103 | 282 | [[sysboot_edit_kernel_cmdline]] |
dfa919a3 | 283 | Editing the Kernel Commandline |
62de1c14 SI |
284 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
285 | ||
286 | You can modify the kernel commandline in the following places, depending on the | |
f11fb629 | 287 | bootloader used: |
62de1c14 SI |
288 | |
289 | .Grub | |
290 | ||
291 | The kernel commandline needs to be placed in the variable | |
292 | `GRUB_CMDLINE_LINUX_DEFAULT` in the file `/etc/default/grub`. Running | |
293 | `update-grub` appends its content to all `linux` entries in | |
294 | `/boot/grub/grub.cfg`. | |
295 | ||
296 | .Systemd-boot | |
297 | ||
0aebb0d9 | 298 | The kernel commandline needs to be placed as one line in `/etc/kernel/cmdline`. |
cb04e768 | 299 | To apply your changes, run `proxmox-boot-tool refresh`, which sets it as the |
0aebb0d9 | 300 | `option` line for all config files in `loader/entries/proxmox-*.conf`. |
a9a2cdbe | 301 | |
9dbab4f8 NU |
302 | A complete list of kernel parameters can be found at |
303 | 'https://www.kernel.org/doc/html/v<YOUR-KERNEL-VERSION>/admin-guide/kernel-parameters.html'. | |
c9f91b12 TL |
304 | replace <YOUR-KERNEL-VERSION> with the major.minor version, for example, for |
305 | kernels based on version 6.5 the URL would be: | |
306 | https://www.kernel.org/doc/html/v6.5/admin-guide/kernel-parameters.html | |
307 | ||
308 | You can find your kernel version by checking the web interface ('Node -> | |
309 | Summary'), or by running | |
9dbab4f8 NU |
310 | |
311 | ---- | |
312 | # uname -r | |
313 | ---- | |
a9a2cdbe | 314 | |
c9f91b12 TL |
315 | Use the first two numbers at the front of the output. |
316 | ||
a9a2cdbe | 317 | [[sysboot_kernel_pin]] |
027bbe9a TL |
318 | Override the Kernel-Version for next Boot |
319 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | |
320 | ||
321 | To select a kernel that is not currently the default kernel, you can either: | |
322 | ||
323 | * use the boot loader menu that is displayed at the beginning of the boot | |
324 | process | |
325 | * use the `proxmox-boot-tool` to `pin` the system to a kernel version either | |
326 | once or permanently (until pin is reset). | |
327 | ||
328 | This should help you work around incompatibilities between a newer kernel | |
329 | version and the hardware. | |
a9a2cdbe | 330 | |
027bbe9a TL |
331 | NOTE: Such a pin should be removed as soon as possible so that all current |
332 | security patches of the latest kernel are also applied to the system. | |
333 | ||
334 | For example: To permanently select the version `5.15.30-1-pve` for booting you | |
335 | would run: | |
a9a2cdbe SI |
336 | |
337 | ---- | |
338 | # proxmox-boot-tool kernel pin 5.15.30-1-pve | |
339 | ---- | |
340 | ||
027bbe9a TL |
341 | TIP: The pinning functionality works for all {pve} systems, not only those using |
342 | `proxmox-boot-tool` to synchronize the contents of the ESPs, if your system | |
343 | does not use `proxmox-boot-tool` for synchronizing you can also skip the | |
344 | `proxmox-boot-tool refresh` call in the end. | |
345 | ||
346 | You can also set a kernel version to be booted on the next system boot only. | |
347 | This is for example useful to test if an updated kernel has resolved an issue, | |
348 | which caused you to `pin` a version in the first place: | |
a9a2cdbe SI |
349 | |
350 | ---- | |
027bbe9a | 351 | # proxmox-boot-tool kernel pin 5.15.30-1-pve --next-boot |
a9a2cdbe SI |
352 | ---- |
353 | ||
354 | To remove any pinned version configuration use the `unpin` subcommand: | |
355 | ||
356 | ---- | |
357 | # proxmox-boot-tool kernel unpin | |
358 | ---- | |
359 | ||
360 | While `unpin` has a `--next-boot` option as well, it is used to clear a pinned | |
027bbe9a TL |
361 | version set with `--next-boot`. As that happens already automatically on boot, |
362 | invonking it manually is of little use. | |
363 | ||
364 | After setting, or clearing pinned versions you also need to synchronize the | |
365 | content and configuration on the ESPs by running the `refresh` subcommand. | |
a9a2cdbe | 366 | |
027bbe9a TL |
367 | TIP: You will be prompted to automatically do for `proxmox-boot-tool` managed |
368 | systems if you call the tool interactively. | |
a9a2cdbe SI |
369 | |
370 | ---- | |
371 | # proxmox-boot-tool refresh | |
372 | ---- | |
952ee606 FG |
373 | |
374 | [[sysboot_secure_boot]] | |
375 | Secure Boot | |
376 | ~~~~~~~~~~~ | |
377 | ||
378 | Since {pve} 8.1, Secure Boot is supported out of the box via signed packages | |
379 | and integration in `proxmox-boot-tool`. | |
380 | ||
381 | The following packages need to be installed for Secure Boot to be enabled: | |
382 | ||
383 | - shim-signed (shim bootloader signed by Microsoft) | |
384 | - shim-helpers-amd64-signed (fallback bootloader and MOKManager, signed by Proxmox) | |
385 | - grub-efi-amd64-signed (Grub EFI bootloader, signed by Proxmox) | |
386 | - proxmox-kernel-6.X.Y-Z-pve-signed (Kernel image, signed by Proxmox) | |
387 | ||
388 | Only Grub as bootloader is supported out of the box, since there are no other | |
389 | pre-signed bootloader packages available. Any new installation of {pve} will | |
390 | automatically have all of the above packages included. | |
391 | ||
392 | More details about how Secure Boot works, and how to customize the setup, are | |
393 | available in https://pve.proxmox.com/wiki/Secure_Boot_Setup[our wiki]. |