]>
Commit | Line | Data |
---|---|---|
16b31cc9 AZ |
1 | [[chapter_firmware_updates]] |
2 | Firmware Updates | |
3 | ---------------- | |
4 | ifdef::wiki[] | |
5 | :pve-toplevel: | |
6 | endif::wiki[] | |
16b31cc9 AZ |
7 | Firmware updates from this chapter should be applied when running {pve} on a |
8 | bare-metal server. Whether configuring firmware updates is appropriate within | |
9 | guests, e.g. when using device pass-through, depends strongly on your setup and | |
10 | is therefore out of scope. | |
11 | ||
48ae5721 AZ |
12 | In addition to regular software updates, firmware updates are also important |
13 | for reliable and secure operation. | |
14 | ||
15 | When obtaining and applying firmware updates, a combination of available options | |
16 | is recommended to get them as early as possible or at all. | |
16b31cc9 | 17 | |
48ae5721 AZ |
18 | The term firmware is usually divided linguistically into microcode (for CPUs) |
19 | and firmware (for other devices). | |
16b31cc9 AZ |
20 | |
21 | ||
22 | [[sysadmin_firmware_persistent]] | |
23 | Persistent Firmware | |
24 | ~~~~~~~~~~~~~~~~~~~ | |
48ae5721 AZ |
25 | This section is suitable for all devices. Updated microcode, which is usually |
26 | included in a BIOS/UEFI update, is stored on the motherboard, whereas other | |
27 | firmware is stored on the respective device. This persistent method is | |
28 | especially important for the CPU, as it enables the earliest possible regular | |
29 | loading of the updated microcode at boot time. | |
16b31cc9 | 30 | |
48ae5721 AZ |
31 | CAUTION: With some updates, such as for BIOS/UEFI or storage controller, the |
32 | device configuration could be reset. Please follow the vendor's instructions | |
33 | carefully and back up the current configuration. | |
16b31cc9 | 34 | |
48ae5721 | 35 | Please check with your vendor which update methods are available. |
16b31cc9 | 36 | |
48ae5721 AZ |
37 | * Convenient update methods for servers can include Dell's Lifecycle Manager or |
38 | Service Packs from HPE. | |
16b31cc9 | 39 | |
48ae5721 | 40 | * Sometimes there are Linux utilities available as well. Examples are |
16b31cc9 AZ |
41 | https://network.nvidia.com/support/firmware/mlxup-mft/['mlxup'] for NVIDIA |
42 | ConnectX or | |
43 | https://techdocs.broadcom.com/us/en/storage-and-ethernet-connectivity/ethernet-nic-controllers/bcm957xxx/adapters/software-installation/updating-the-firmware/manually-updating-the-adapter-firmware-on-linuxesx.html['bnxtnvm'/'niccli'] | |
44 | for Broadcom network cards. | |
45 | ||
48ae5721 AZ |
46 | * https://fwupd.org[LVFS] could also be an option if there is a cooperation with |
47 | a https://fwupd.org/lvfs/vendors/[vendor] and | |
48 | https://fwupd.org/lvfs/devices/[supported hardware] in use. The technical | |
49 | requirement for this is that the system was manufactured after 2014, is booted | |
50 | via UEFI and the easiest way is to mount the EFI partition from which you boot | |
51 | (`mount /dev/disk/by-partuuid/<from efibootmgr -v> /boot/efi`) before installing | |
52 | 'fwupd'. | |
16b31cc9 | 53 | |
48ae5721 AZ |
54 | TIP: If the update instructions require a host reboot, make sure that it can be |
55 | done safely. See also xref:ha_manager_node_maintenance[Node Maintenance]. | |
16b31cc9 AZ |
56 | |
57 | ||
58 | [[sysadmin_firmware_runtime_files]] | |
59 | Runtime Firmware Files | |
60 | ~~~~~~~~~~~~~~~~~~~~~~ | |
48ae5721 AZ |
61 | This method stores firmware on the {pve} operating system and will pass it to a |
62 | device if its xref:sysadmin_firmware_persistent[persisted firmware] is less | |
63 | recent. It is supported by devices such as network and graphics cards, but not | |
64 | by those that rely on persisted firmware such as the motherboard and hard disks. | |
16b31cc9 AZ |
65 | |
66 | In {pve} the package `pve-firmware` is already installed by default. Therefore, | |
48ae5721 AZ |
67 | with the normal xref:system_software_updates[system updates (APT)], included |
68 | firmware of common hardware is automatically kept up to date. | |
69 | ||
70 | An additional xref:sysadmin_debian_firmware_repo[Debian Firmware Repository] | |
71 | exists, but is not configured by default. | |
72 | ||
73 | If you try to install an additional firmware package but it conflicts, APT will | |
74 | abort the installation. Perhaps the particular firmware can be obtained in | |
75 | another way. | |
76 | ||
77 | ||
78 | [[sysadmin_firmware_cpu]] | |
79 | CPU Microcode Updates | |
80 | ~~~~~~~~~~~~~~~~~~~~~ | |
81 | Microcode updates are intended to fix found security vulnerabilities and other | |
82 | serious CPU bugs. While the CPU performance can be affected, a patched microcode | |
83 | is usually still more performant than an unpatched microcode where the kernel | |
84 | itself has to do mitigations. Depending on the CPU type, it is possible that | |
85 | performance results of the flawed factory state can no longer be achieved | |
86 | without knowingly running the CPU in an unsafe state. | |
87 | ||
88 | To get an overview of present CPU vulnerabilities and their mitigations, run | |
89 | `lscpu`. Current real-world known vulnerabilities can only show up if the | |
90 | {pve} host is xref:system_software_updates[up to date], its version not | |
91 | xref:faq-support-table[end of life], and has at least been rebooted since the | |
92 | last kernel update. | |
93 | ||
94 | Besides the recommended microcode update via | |
95 | xref:sysadmin_firmware_persistent[persistent] BIOS/UEFI updates, there is also | |
96 | an independent method via *Early OS Microcode Updates*. It is convenient to use | |
97 | and also quite helpful when the motherboard vendor no longer provides BIOS/UEFI | |
98 | updates. Regardless of the method in use, a reboot is always needed to apply a | |
99 | microcode update. | |
100 | ||
101 | ||
102 | Set up Early OS Microcode Updates | |
103 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
104 | After enabling the | |
105 | xref:sysadmin_debian_firmware_repo[Debian Firmware Repository], run | |
106 | `apt install <intel-microcode|amd64-microcode>` and reboot the {pve} host | |
107 | xref:ha_manager_node_maintenance[safely] afterwards. | |
108 | ||
109 | Infrequent future microcode updates also require a reboot to be loaded. | |
110 | ||
111 | ||
112 | Microcode Version | |
113 | ^^^^^^^^^^^^^^^^^ | |
114 | To get the current running microcode revision for comparison or debugging | |
115 | purposes: | |
116 | ||
117 | ---- | |
118 | # grep microcode /proc/cpuinfo | uniq | |
119 | microcode : 0xf0 | |
120 | ---- | |
121 | ||
122 | Since a microcode package contains microcode for different CPU types, an updated | |
123 | microcode for your CPU will only be included occasionally. Therefore, date | |
124 | information from the package can also not be matched to a specific release date | |
125 | from the vendor. | |
126 | ||
127 | If the microcode package is installed, the system has been rebooted, and the | |
128 | microcode included in the package is more recent than that on the motherboard | |
129 | and CPU, the message "microcode updated early" appears in the log: | |
130 | ||
131 | ---- | |
132 | # dmesg | grep microcode | |
133 | [ 0.000000] microcode: microcode updated early to revision 0xf0, date = 2021-11-12 | |
134 | [ 0.896580] microcode: Microcode Update Driver: v2.2. | |
135 | ---- | |
136 | ||
137 | ||
138 | [[sysadmin_firmware_troubleshooting]] | |
139 | Troubleshooting | |
140 | ^^^^^^^^^^^^^^^ | |
141 | For debugging purposes, the set up Early OS Microcode Update applied regularly | |
142 | at system boot can be temporarily disabled as follows: | |
143 | ||
144 | 1. make sure that the host can be rebooted xref:ha_manager_node_maintenance[safely] | |
145 | 2. reboot the host to get to the GRUB menu (hold `SHIFT` if it is hidden) | |
146 | 3. at the desired {pve} boot entry press `E` | |
147 | 4. go to the line which starts with `linux` and append separated by a space | |
148 | *`dis_ucode_ldr`* | |
149 | 5. press `CTRL-X` to boot this time without an Early OS Microcode Update | |
150 | ||
151 | If a problem related to a recent microcode update is suspected, a package | |
152 | downgrade should be considered instead of package removal | |
153 | (`apt purge <intel-microcode|amd64-microcode>`). Otherwise, a too old | |
154 | xref:sysadmin_firmware_persistent[persisted] microcode might be loaded, even | |
155 | though a more recent one would run without problems. | |
156 | ||
157 | A downgrade is possible if an earlier microcode package version is | |
158 | available in the Debian repository, as shown in this example: | |
159 | ||
160 | ---- | |
161 | # apt list -a intel-microcode | |
162 | Listing... Done | |
163 | intel-microcode/stable-security,now 3.20230808.1~deb12u1 amd64 [installed] | |
164 | intel-microcode/stable 3.20230512.1 amd64 | |
165 | ---- | |
166 | ---- | |
167 | # apt install intel-microcode=3.202305* | |
168 | ... | |
169 | Selected version '3.20230512.1' (Debian:12.1/stable [amd64]) for 'intel-microcode' | |
170 | ... | |
171 | dpkg: warning: downgrading intel-microcode from 3.20230808.1~deb12u1 to 3.20230512.1 | |
172 | ... | |
173 | intel-microcode: microcode will be updated at next boot | |
174 | ... | |
175 | ---- | |
176 | ||
177 | Make sure (again) that the host can be rebooted | |
178 | xref:ha_manager_node_maintenance[safely]. To apply an older microcode | |
179 | potentially included in the microcode package for your CPU type, reboot now. | |
180 | ||
181 | [TIP] | |
182 | ==== | |
183 | It makes sense to hold the downgraded package for a while and try more recent | |
184 | versions again at a later time. Even if the package version is the same in the | |
185 | future, system updates may have fixed the experienced problem in the meantime. | |
186 | ---- | |
187 | # apt-mark hold intel-microcode | |
188 | intel-microcode set on hold. | |
189 | ---- | |
190 | ---- | |
191 | # apt-mark unhold intel-microcode | |
192 | # apt update | |
193 | # apt upgrade | |
194 | ---- | |
195 | ==== |