]>
Commit | Line | Data |
---|---|---|
16b31cc9 AZ |
1 | [[chapter_firmware_updates]] |
2 | Firmware Updates | |
3 | ---------------- | |
4 | ifdef::wiki[] | |
5 | :pve-toplevel: | |
6 | endif::wiki[] | |
16b31cc9 AZ |
7 | Firmware updates from this chapter should be applied when running {pve} on a |
8 | bare-metal server. Whether configuring firmware updates is appropriate within | |
9 | guests, e.g. when using device pass-through, depends strongly on your setup and | |
10 | is therefore out of scope. | |
11 | ||
48ae5721 AZ |
12 | In addition to regular software updates, firmware updates are also important |
13 | for reliable and secure operation. | |
14 | ||
15 | When obtaining and applying firmware updates, a combination of available options | |
16 | is recommended to get them as early as possible or at all. | |
16b31cc9 | 17 | |
48ae5721 AZ |
18 | The term firmware is usually divided linguistically into microcode (for CPUs) |
19 | and firmware (for other devices). | |
16b31cc9 AZ |
20 | |
21 | ||
22 | [[sysadmin_firmware_persistent]] | |
23 | Persistent Firmware | |
24 | ~~~~~~~~~~~~~~~~~~~ | |
48ae5721 AZ |
25 | This section is suitable for all devices. Updated microcode, which is usually |
26 | included in a BIOS/UEFI update, is stored on the motherboard, whereas other | |
27 | firmware is stored on the respective device. This persistent method is | |
28 | especially important for the CPU, as it enables the earliest possible regular | |
29 | loading of the updated microcode at boot time. | |
16b31cc9 | 30 | |
48ae5721 AZ |
31 | CAUTION: With some updates, such as for BIOS/UEFI or storage controller, the |
32 | device configuration could be reset. Please follow the vendor's instructions | |
33 | carefully and back up the current configuration. | |
16b31cc9 | 34 | |
48ae5721 | 35 | Please check with your vendor which update methods are available. |
16b31cc9 | 36 | |
48ae5721 AZ |
37 | * Convenient update methods for servers can include Dell's Lifecycle Manager or |
38 | Service Packs from HPE. | |
16b31cc9 | 39 | |
48ae5721 | 40 | * Sometimes there are Linux utilities available as well. Examples are |
16b31cc9 AZ |
41 | https://network.nvidia.com/support/firmware/mlxup-mft/['mlxup'] for NVIDIA |
42 | ConnectX or | |
43 | https://techdocs.broadcom.com/us/en/storage-and-ethernet-connectivity/ethernet-nic-controllers/bcm957xxx/adapters/software-installation/updating-the-firmware/manually-updating-the-adapter-firmware-on-linuxesx.html['bnxtnvm'/'niccli'] | |
44 | for Broadcom network cards. | |
45 | ||
905fff49 SI |
46 | * https://fwupd.org[LVFS] is also an option if there is a cooperation with |
47 | the https://fwupd.org/lvfs/vendors/[hardware vendor] and | |
48ae5721 | 48 | https://fwupd.org/lvfs/devices/[supported hardware] in use. The technical |
905fff49 SI |
49 | requirement for this is that the system was manufactured after 2014 and is |
50 | booted via UEFI. | |
51 | ||
a3806d64 | 52 | {pve} ships its own version of the `fwupd` package to enable Secure Boot |
62ef2acb | 53 | Support with the Proxmox signing key. This package consciously dropped the |
a3806d64 TL |
54 | dependency recommendation for the `udisks2` package, due to observed issues with |
55 | its use on hypervisors. That means you must explicitly configure the correct | |
56 | mount point of the EFI partition in `/etc/fwupd/daemon.conf`, for example: | |
905fff49 SI |
57 | |
58 | .File `/etc/fwupd/daemon.conf` | |
59 | ---- | |
60 | # Override the location used for the EFI system partition (ESP) path. | |
61 | EspLocation=/boot/efi | |
62 | ---- | |
16b31cc9 | 63 | |
48ae5721 AZ |
64 | TIP: If the update instructions require a host reboot, make sure that it can be |
65 | done safely. See also xref:ha_manager_node_maintenance[Node Maintenance]. | |
16b31cc9 AZ |
66 | |
67 | ||
68 | [[sysadmin_firmware_runtime_files]] | |
69 | Runtime Firmware Files | |
70 | ~~~~~~~~~~~~~~~~~~~~~~ | |
48ae5721 AZ |
71 | This method stores firmware on the {pve} operating system and will pass it to a |
72 | device if its xref:sysadmin_firmware_persistent[persisted firmware] is less | |
73 | recent. It is supported by devices such as network and graphics cards, but not | |
74 | by those that rely on persisted firmware such as the motherboard and hard disks. | |
16b31cc9 AZ |
75 | |
76 | In {pve} the package `pve-firmware` is already installed by default. Therefore, | |
48ae5721 AZ |
77 | with the normal xref:system_software_updates[system updates (APT)], included |
78 | firmware of common hardware is automatically kept up to date. | |
79 | ||
80 | An additional xref:sysadmin_debian_firmware_repo[Debian Firmware Repository] | |
81 | exists, but is not configured by default. | |
82 | ||
83 | If you try to install an additional firmware package but it conflicts, APT will | |
84 | abort the installation. Perhaps the particular firmware can be obtained in | |
85 | another way. | |
86 | ||
87 | ||
88 | [[sysadmin_firmware_cpu]] | |
89 | CPU Microcode Updates | |
90 | ~~~~~~~~~~~~~~~~~~~~~ | |
91 | Microcode updates are intended to fix found security vulnerabilities and other | |
92 | serious CPU bugs. While the CPU performance can be affected, a patched microcode | |
93 | is usually still more performant than an unpatched microcode where the kernel | |
94 | itself has to do mitigations. Depending on the CPU type, it is possible that | |
95 | performance results of the flawed factory state can no longer be achieved | |
96 | without knowingly running the CPU in an unsafe state. | |
97 | ||
98 | To get an overview of present CPU vulnerabilities and their mitigations, run | |
99 | `lscpu`. Current real-world known vulnerabilities can only show up if the | |
100 | {pve} host is xref:system_software_updates[up to date], its version not | |
101 | xref:faq-support-table[end of life], and has at least been rebooted since the | |
102 | last kernel update. | |
103 | ||
104 | Besides the recommended microcode update via | |
105 | xref:sysadmin_firmware_persistent[persistent] BIOS/UEFI updates, there is also | |
106 | an independent method via *Early OS Microcode Updates*. It is convenient to use | |
107 | and also quite helpful when the motherboard vendor no longer provides BIOS/UEFI | |
108 | updates. Regardless of the method in use, a reboot is always needed to apply a | |
109 | microcode update. | |
110 | ||
111 | ||
112 | Set up Early OS Microcode Updates | |
113 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
113290b8 TL |
114 | To set up microcode updates that are applied early on boot by the Linux kernel, |
115 | you need to: | |
116 | ||
117 | . Enable the xref:sysadmin_debian_firmware_repo[Debian Firmware Repository] | |
118 | . Get the latest available packages `apt update` (or use the web interface, | |
119 | under Node -> Updates) | |
120 | . Install the CPU-vendor specific microcode package: | |
121 | - For Intel CPUs: `apt install intel-microcode` | |
122 | - For AMD CPUs: `apt install amd64-microcode` | |
123 | . Reboot the {pve} host | |
124 | ||
125 | Any future microcode update will also require a reboot to be loaded. | |
48ae5721 AZ |
126 | |
127 | Microcode Version | |
128 | ^^^^^^^^^^^^^^^^^ | |
129 | To get the current running microcode revision for comparison or debugging | |
130 | purposes: | |
131 | ||
132 | ---- | |
133 | # grep microcode /proc/cpuinfo | uniq | |
134 | microcode : 0xf0 | |
135 | ---- | |
136 | ||
8bad0dea TL |
137 | A microcode package has updates for many different CPUs. But updates |
138 | specifically for your CPU might not come often. So, just looking at the date on | |
139 | the package won't tell you when the company actually released an update for your | |
140 | specific CPU. | |
141 | ||
142 | If you've installed a new microcode package and rebooted your {pve} host, and | |
143 | this new microcode is newer than both, the version baked into the CPU and the | |
144 | one from the motherboard's firmware, you'll see a message in the system log | |
145 | saying "microcode updated early". | |
48ae5721 AZ |
146 | |
147 | ---- | |
148 | # dmesg | grep microcode | |
149 | [ 0.000000] microcode: microcode updated early to revision 0xf0, date = 2021-11-12 | |
150 | [ 0.896580] microcode: Microcode Update Driver: v2.2. | |
151 | ---- | |
152 | ||
153 | ||
154 | [[sysadmin_firmware_troubleshooting]] | |
155 | Troubleshooting | |
156 | ^^^^^^^^^^^^^^^ | |
157 | For debugging purposes, the set up Early OS Microcode Update applied regularly | |
158 | at system boot can be temporarily disabled as follows: | |
159 | ||
160 | 1. make sure that the host can be rebooted xref:ha_manager_node_maintenance[safely] | |
161 | 2. reboot the host to get to the GRUB menu (hold `SHIFT` if it is hidden) | |
162 | 3. at the desired {pve} boot entry press `E` | |
163 | 4. go to the line which starts with `linux` and append separated by a space | |
164 | *`dis_ucode_ldr`* | |
165 | 5. press `CTRL-X` to boot this time without an Early OS Microcode Update | |
166 | ||
167 | If a problem related to a recent microcode update is suspected, a package | |
168 | downgrade should be considered instead of package removal | |
169 | (`apt purge <intel-microcode|amd64-microcode>`). Otherwise, a too old | |
170 | xref:sysadmin_firmware_persistent[persisted] microcode might be loaded, even | |
171 | though a more recent one would run without problems. | |
172 | ||
173 | A downgrade is possible if an earlier microcode package version is | |
174 | available in the Debian repository, as shown in this example: | |
175 | ||
176 | ---- | |
177 | # apt list -a intel-microcode | |
178 | Listing... Done | |
179 | intel-microcode/stable-security,now 3.20230808.1~deb12u1 amd64 [installed] | |
180 | intel-microcode/stable 3.20230512.1 amd64 | |
181 | ---- | |
182 | ---- | |
183 | # apt install intel-microcode=3.202305* | |
184 | ... | |
185 | Selected version '3.20230512.1' (Debian:12.1/stable [amd64]) for 'intel-microcode' | |
186 | ... | |
187 | dpkg: warning: downgrading intel-microcode from 3.20230808.1~deb12u1 to 3.20230512.1 | |
188 | ... | |
189 | intel-microcode: microcode will be updated at next boot | |
190 | ... | |
191 | ---- | |
192 | ||
193 | Make sure (again) that the host can be rebooted | |
194 | xref:ha_manager_node_maintenance[safely]. To apply an older microcode | |
195 | potentially included in the microcode package for your CPU type, reboot now. | |
196 | ||
197 | [TIP] | |
198 | ==== | |
199 | It makes sense to hold the downgraded package for a while and try more recent | |
200 | versions again at a later time. Even if the package version is the same in the | |
201 | future, system updates may have fixed the experienced problem in the meantime. | |
202 | ---- | |
203 | # apt-mark hold intel-microcode | |
204 | intel-microcode set on hold. | |
205 | ---- | |
206 | ---- | |
207 | # apt-mark unhold intel-microcode | |
208 | # apt update | |
209 | # apt upgrade | |
210 | ---- | |
211 | ==== |