]>
Commit | Line | Data |
---|---|---|
16b31cc9 AZ |
1 | [[chapter_firmware_updates]] |
2 | Firmware Updates | |
3 | ---------------- | |
4 | ifdef::wiki[] | |
5 | :pve-toplevel: | |
6 | endif::wiki[] | |
16b31cc9 AZ |
7 | Firmware updates from this chapter should be applied when running {pve} on a |
8 | bare-metal server. Whether configuring firmware updates is appropriate within | |
9 | guests, e.g. when using device pass-through, depends strongly on your setup and | |
10 | is therefore out of scope. | |
11 | ||
48ae5721 AZ |
12 | In addition to regular software updates, firmware updates are also important |
13 | for reliable and secure operation. | |
14 | ||
15 | When obtaining and applying firmware updates, a combination of available options | |
16 | is recommended to get them as early as possible or at all. | |
16b31cc9 | 17 | |
48ae5721 AZ |
18 | The term firmware is usually divided linguistically into microcode (for CPUs) |
19 | and firmware (for other devices). | |
16b31cc9 AZ |
20 | |
21 | ||
22 | [[sysadmin_firmware_persistent]] | |
23 | Persistent Firmware | |
24 | ~~~~~~~~~~~~~~~~~~~ | |
48ae5721 AZ |
25 | This section is suitable for all devices. Updated microcode, which is usually |
26 | included in a BIOS/UEFI update, is stored on the motherboard, whereas other | |
27 | firmware is stored on the respective device. This persistent method is | |
28 | especially important for the CPU, as it enables the earliest possible regular | |
29 | loading of the updated microcode at boot time. | |
16b31cc9 | 30 | |
48ae5721 AZ |
31 | CAUTION: With some updates, such as for BIOS/UEFI or storage controller, the |
32 | device configuration could be reset. Please follow the vendor's instructions | |
33 | carefully and back up the current configuration. | |
16b31cc9 | 34 | |
48ae5721 | 35 | Please check with your vendor which update methods are available. |
16b31cc9 | 36 | |
48ae5721 AZ |
37 | * Convenient update methods for servers can include Dell's Lifecycle Manager or |
38 | Service Packs from HPE. | |
16b31cc9 | 39 | |
48ae5721 | 40 | * Sometimes there are Linux utilities available as well. Examples are |
16b31cc9 AZ |
41 | https://network.nvidia.com/support/firmware/mlxup-mft/['mlxup'] for NVIDIA |
42 | ConnectX or | |
43 | https://techdocs.broadcom.com/us/en/storage-and-ethernet-connectivity/ethernet-nic-controllers/bcm957xxx/adapters/software-installation/updating-the-firmware/manually-updating-the-adapter-firmware-on-linuxesx.html['bnxtnvm'/'niccli'] | |
44 | for Broadcom network cards. | |
45 | ||
48ae5721 AZ |
46 | * https://fwupd.org[LVFS] could also be an option if there is a cooperation with |
47 | a https://fwupd.org/lvfs/vendors/[vendor] and | |
48 | https://fwupd.org/lvfs/devices/[supported hardware] in use. The technical | |
49 | requirement for this is that the system was manufactured after 2014, is booted | |
50 | via UEFI and the easiest way is to mount the EFI partition from which you boot | |
51 | (`mount /dev/disk/by-partuuid/<from efibootmgr -v> /boot/efi`) before installing | |
52 | 'fwupd'. | |
16b31cc9 | 53 | |
48ae5721 AZ |
54 | TIP: If the update instructions require a host reboot, make sure that it can be |
55 | done safely. See also xref:ha_manager_node_maintenance[Node Maintenance]. | |
16b31cc9 AZ |
56 | |
57 | ||
58 | [[sysadmin_firmware_runtime_files]] | |
59 | Runtime Firmware Files | |
60 | ~~~~~~~~~~~~~~~~~~~~~~ | |
48ae5721 AZ |
61 | This method stores firmware on the {pve} operating system and will pass it to a |
62 | device if its xref:sysadmin_firmware_persistent[persisted firmware] is less | |
63 | recent. It is supported by devices such as network and graphics cards, but not | |
64 | by those that rely on persisted firmware such as the motherboard and hard disks. | |
16b31cc9 AZ |
65 | |
66 | In {pve} the package `pve-firmware` is already installed by default. Therefore, | |
48ae5721 AZ |
67 | with the normal xref:system_software_updates[system updates (APT)], included |
68 | firmware of common hardware is automatically kept up to date. | |
69 | ||
70 | An additional xref:sysadmin_debian_firmware_repo[Debian Firmware Repository] | |
71 | exists, but is not configured by default. | |
72 | ||
73 | If you try to install an additional firmware package but it conflicts, APT will | |
74 | abort the installation. Perhaps the particular firmware can be obtained in | |
75 | another way. | |
76 | ||
77 | ||
78 | [[sysadmin_firmware_cpu]] | |
79 | CPU Microcode Updates | |
80 | ~~~~~~~~~~~~~~~~~~~~~ | |
81 | Microcode updates are intended to fix found security vulnerabilities and other | |
82 | serious CPU bugs. While the CPU performance can be affected, a patched microcode | |
83 | is usually still more performant than an unpatched microcode where the kernel | |
84 | itself has to do mitigations. Depending on the CPU type, it is possible that | |
85 | performance results of the flawed factory state can no longer be achieved | |
86 | without knowingly running the CPU in an unsafe state. | |
87 | ||
88 | To get an overview of present CPU vulnerabilities and their mitigations, run | |
89 | `lscpu`. Current real-world known vulnerabilities can only show up if the | |
90 | {pve} host is xref:system_software_updates[up to date], its version not | |
91 | xref:faq-support-table[end of life], and has at least been rebooted since the | |
92 | last kernel update. | |
93 | ||
94 | Besides the recommended microcode update via | |
95 | xref:sysadmin_firmware_persistent[persistent] BIOS/UEFI updates, there is also | |
96 | an independent method via *Early OS Microcode Updates*. It is convenient to use | |
97 | and also quite helpful when the motherboard vendor no longer provides BIOS/UEFI | |
98 | updates. Regardless of the method in use, a reboot is always needed to apply a | |
99 | microcode update. | |
100 | ||
101 | ||
102 | Set up Early OS Microcode Updates | |
103 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
113290b8 TL |
104 | To set up microcode updates that are applied early on boot by the Linux kernel, |
105 | you need to: | |
106 | ||
107 | . Enable the xref:sysadmin_debian_firmware_repo[Debian Firmware Repository] | |
108 | . Get the latest available packages `apt update` (or use the web interface, | |
109 | under Node -> Updates) | |
110 | . Install the CPU-vendor specific microcode package: | |
111 | - For Intel CPUs: `apt install intel-microcode` | |
112 | - For AMD CPUs: `apt install amd64-microcode` | |
113 | . Reboot the {pve} host | |
114 | ||
115 | Any future microcode update will also require a reboot to be loaded. | |
48ae5721 AZ |
116 | |
117 | Microcode Version | |
118 | ^^^^^^^^^^^^^^^^^ | |
119 | To get the current running microcode revision for comparison or debugging | |
120 | purposes: | |
121 | ||
122 | ---- | |
123 | # grep microcode /proc/cpuinfo | uniq | |
124 | microcode : 0xf0 | |
125 | ---- | |
126 | ||
8bad0dea TL |
127 | A microcode package has updates for many different CPUs. But updates |
128 | specifically for your CPU might not come often. So, just looking at the date on | |
129 | the package won't tell you when the company actually released an update for your | |
130 | specific CPU. | |
131 | ||
132 | If you've installed a new microcode package and rebooted your {pve} host, and | |
133 | this new microcode is newer than both, the version baked into the CPU and the | |
134 | one from the motherboard's firmware, you'll see a message in the system log | |
135 | saying "microcode updated early". | |
48ae5721 AZ |
136 | |
137 | ---- | |
138 | # dmesg | grep microcode | |
139 | [ 0.000000] microcode: microcode updated early to revision 0xf0, date = 2021-11-12 | |
140 | [ 0.896580] microcode: Microcode Update Driver: v2.2. | |
141 | ---- | |
142 | ||
143 | ||
144 | [[sysadmin_firmware_troubleshooting]] | |
145 | Troubleshooting | |
146 | ^^^^^^^^^^^^^^^ | |
147 | For debugging purposes, the set up Early OS Microcode Update applied regularly | |
148 | at system boot can be temporarily disabled as follows: | |
149 | ||
150 | 1. make sure that the host can be rebooted xref:ha_manager_node_maintenance[safely] | |
151 | 2. reboot the host to get to the GRUB menu (hold `SHIFT` if it is hidden) | |
152 | 3. at the desired {pve} boot entry press `E` | |
153 | 4. go to the line which starts with `linux` and append separated by a space | |
154 | *`dis_ucode_ldr`* | |
155 | 5. press `CTRL-X` to boot this time without an Early OS Microcode Update | |
156 | ||
157 | If a problem related to a recent microcode update is suspected, a package | |
158 | downgrade should be considered instead of package removal | |
159 | (`apt purge <intel-microcode|amd64-microcode>`). Otherwise, a too old | |
160 | xref:sysadmin_firmware_persistent[persisted] microcode might be loaded, even | |
161 | though a more recent one would run without problems. | |
162 | ||
163 | A downgrade is possible if an earlier microcode package version is | |
164 | available in the Debian repository, as shown in this example: | |
165 | ||
166 | ---- | |
167 | # apt list -a intel-microcode | |
168 | Listing... Done | |
169 | intel-microcode/stable-security,now 3.20230808.1~deb12u1 amd64 [installed] | |
170 | intel-microcode/stable 3.20230512.1 amd64 | |
171 | ---- | |
172 | ---- | |
173 | # apt install intel-microcode=3.202305* | |
174 | ... | |
175 | Selected version '3.20230512.1' (Debian:12.1/stable [amd64]) for 'intel-microcode' | |
176 | ... | |
177 | dpkg: warning: downgrading intel-microcode from 3.20230808.1~deb12u1 to 3.20230512.1 | |
178 | ... | |
179 | intel-microcode: microcode will be updated at next boot | |
180 | ... | |
181 | ---- | |
182 | ||
183 | Make sure (again) that the host can be rebooted | |
184 | xref:ha_manager_node_maintenance[safely]. To apply an older microcode | |
185 | potentially included in the microcode package for your CPU type, reboot now. | |
186 | ||
187 | [TIP] | |
188 | ==== | |
189 | It makes sense to hold the downgraded package for a while and try more recent | |
190 | versions again at a later time. Even if the package version is the same in the | |
191 | future, system updates may have fixed the experienced problem in the meantime. | |
192 | ---- | |
193 | # apt-mark hold intel-microcode | |
194 | intel-microcode set on hold. | |
195 | ---- | |
196 | ---- | |
197 | # apt-mark unhold intel-microcode | |
198 | # apt update | |
199 | # apt upgrade | |
200 | ---- | |
201 | ==== |