firmware-updates.adoc

   1 [[chapter_firmware_updates]]
   2 Firmware Updates
   3 ----------------
   4 ifdef::wiki[]
   5 :pve-toplevel:
   6 endif::wiki[]
   7 Firmware updates from this chapter should be applied when running {pve} on a
   8 bare-metal server. Whether configuring firmware updates is appropriate within
   9 guests, e.g. when using device pass-through, depends strongly on your setup and
  10 is therefore out of scope.
  11
  12 In addition to regular software updates, firmware updates are also important
  13 for reliable and secure operation.
  14
  15 When obtaining and applying firmware updates, a combination of available options
  16 is recommended to get them as early as possible or at all.
  17
  18 The term firmware is usually divided linguistically into microcode (for CPUs)
  19 and firmware (for other devices).
  20
  21
  22 [[sysadmin_firmware_persistent]]
  23 Persistent Firmware
  24 ~~~~~~~~~~~~~~~~~~~
  25 This section is suitable for all devices. Updated microcode, which is usually
  26 included in a BIOS/UEFI update, is stored on the motherboard, whereas other
  27 firmware is stored on the respective device. This persistent method is
  28 especially important for the CPU, as it enables the earliest possible regular
  29 loading of the updated microcode at boot time.
  30
  31 CAUTION: With some updates, such as for BIOS/UEFI or storage controller, the
  32 device configuration could be reset. Please follow the vendor's instructions
  33 carefully and back up the current configuration.
  34
  35 Please check with your vendor which update methods are available.
  36
  37 * Convenient update methods for servers can include Dell's Lifecycle Manager or
  38 Service Packs from HPE.
  39
  40 * Sometimes there are Linux utilities available as well. Examples are
  41 https://network.nvidia.com/support/firmware/mlxup-mft/['mlxup'] for NVIDIA
  42 ConnectX or
  43 https://techdocs.broadcom.com/us/en/storage-and-ethernet-connectivity/ethernet-nic-controllers/bcm957xxx/adapters/software-installation/updating-the-firmware/manually-updating-the-adapter-firmware-on-linuxesx.html['bnxtnvm'/'niccli']
  44 for Broadcom network cards.
  45
  46 * https://fwupd.org[LVFS] could also be an option if there is a cooperation with
  47 a https://fwupd.org/lvfs/vendors/[vendor] and
  48 https://fwupd.org/lvfs/devices/[supported hardware] in use. The technical
  49 requirement for this is that the system was manufactured after 2014, is booted
  50 via UEFI and the easiest way is to mount the EFI partition from which you boot
  51 (`mount /dev/disk/by-partuuid/<from efibootmgr -v> /boot/efi`) before installing
  52 'fwupd'.
  53
  54 TIP: If the update instructions require a host reboot, make sure that it can be
  55 done safely. See also xref:ha_manager_node_maintenance[Node Maintenance].
  56
  57
  58 [[sysadmin_firmware_runtime_files]]
  59 Runtime Firmware Files
  60 ~~~~~~~~~~~~~~~~~~~~~~
  61 This method stores firmware on the {pve} operating system and will pass it to a
  62 device if its xref:sysadmin_firmware_persistent[persisted firmware] is less
  63 recent. It is supported by devices such as network and graphics cards, but not
  64 by those that rely on persisted firmware such as the motherboard and hard disks.
  65
  66 In {pve} the package `pve-firmware` is already installed by default. Therefore,
  67 with the normal xref:system_software_updates[system updates (APT)], included
  68 firmware of common hardware is automatically kept up to date.
  69
  70 An additional xref:sysadmin_debian_firmware_repo[Debian Firmware Repository]
  71 exists, but is not configured by default.
  72
  73 If you try to install an additional firmware package but it conflicts, APT will
  74 abort the installation. Perhaps the particular firmware can be obtained in
  75 another way.
  76
  77
  78 [[sysadmin_firmware_cpu]]
  79 CPU Microcode Updates
  80 ~~~~~~~~~~~~~~~~~~~~~
  81 Microcode updates are intended to fix found security vulnerabilities and other
  82 serious CPU bugs. While the CPU performance can be affected, a patched microcode
  83 is usually still more performant than an unpatched microcode where the kernel
  84 itself has to do mitigations. Depending on the CPU type, it is possible that
  85 performance results of the flawed factory state can no longer be achieved
  86 without knowingly running the CPU in an unsafe state.
  87
  88 To get an overview of present CPU vulnerabilities and their mitigations, run
  89 `lscpu`. Current real-world known vulnerabilities can only show up if the
  90 {pve} host is xref:system_software_updates[up to date], its version not
  91 xref:faq-support-table[end of life], and has at least been rebooted since the
  92 last kernel update.
  93
  94 Besides the recommended microcode update via
  95 xref:sysadmin_firmware_persistent[persistent] BIOS/UEFI updates, there is also
  96 an independent method via *Early OS Microcode Updates*. It is convenient to use
  97 and also quite helpful when the motherboard vendor no longer provides BIOS/UEFI
  98 updates. Regardless of the method in use, a reboot is always needed to apply a
  99 microcode update.
 100
 101
 102 Set up Early OS Microcode Updates
 103 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 104 To set up microcode updates that are applied early on boot by the Linux kernel,
 105 you need to:
 106
 107 . Enable the xref:sysadmin_debian_firmware_repo[Debian Firmware Repository]
 108 . Get the latest available packages `apt update` (or use the web interface,
 109   under Node -> Updates)
 110 . Install the CPU-vendor specific microcode package:
 111   - For Intel CPUs:  `apt install intel-microcode`
 112   - For AMD CPUs:  `apt install amd64-microcode`
 113 . Reboot the {pve} host
 114
 115 Any future microcode update will also require a reboot to be loaded.
 116
 117 Microcode Version
 118 ^^^^^^^^^^^^^^^^^
 119 To get the current running microcode revision for comparison or debugging
 120 purposes:
 121
 122 ----
 123 # grep microcode /proc/cpuinfo | uniq
 124 microcode       : 0xf0
 125 ----
 126
 127 A microcode package has updates for many different CPUs. But updates
 128 specifically for your CPU might not come often. So, just looking at the date on
 129 the package won't tell you when the company actually released an update for your
 130 specific CPU.
 131
 132 If you've installed a new microcode package and rebooted your {pve} host, and
 133 this new microcode is newer than both, the version baked into the CPU and the
 134 one from the motherboard's firmware, you'll see a message in the system log
 135 saying "microcode updated early".
 136
 137 ----
 138 # dmesg | grep microcode
 139 [    0.000000] microcode: microcode updated early to revision 0xf0, date = 2021-11-12
 140 [    0.896580] microcode: Microcode Update Driver: v2.2.
 141 ----
 142
 143
 144 [[sysadmin_firmware_troubleshooting]]
 145 Troubleshooting
 146 ^^^^^^^^^^^^^^^
 147 For debugging purposes, the set up Early OS Microcode Update applied regularly
 148 at system boot can be temporarily disabled as follows:
 149
 150 1. make sure that the host can be rebooted xref:ha_manager_node_maintenance[safely]
 151 2. reboot the host to get to the GRUB menu (hold `SHIFT` if it is hidden)
 152 3. at the desired {pve} boot entry press `E`
 153 4. go to the line which starts with `linux` and append separated by a space
 154 *`dis_ucode_ldr`*
 155 5. press `CTRL-X` to boot this time without an Early OS Microcode Update
 156
 157 If a problem related to a recent microcode update is suspected, a package
 158 downgrade should be considered instead of package removal
 159 (`apt purge <intel-microcode|amd64-microcode>`). Otherwise, a too old
 160 xref:sysadmin_firmware_persistent[persisted] microcode might be loaded, even
 161 though a more recent one would run without problems.
 162
 163 A downgrade is possible if an earlier microcode package version is
 164 available in the Debian repository, as shown in this example:
 165
 166 ----
 167 # apt list -a intel-microcode
 168 Listing... Done
 169 intel-microcode/stable-security,now 3.20230808.1~deb12u1 amd64 [installed]
 170 intel-microcode/stable 3.20230512.1 amd64
 171 ----
 172 ----
 173 # apt install intel-microcode=3.202305*
 174 ...
 175 Selected version '3.20230512.1' (Debian:12.1/stable [amd64]) for 'intel-microcode'
 176 ...
 177 dpkg: warning: downgrading intel-microcode from 3.20230808.1~deb12u1 to 3.20230512.1
 178 ...
 179 intel-microcode: microcode will be updated at next boot
 180 ...
 181 ----
 182
 183 Make sure (again) that the host can be rebooted
 184 xref:ha_manager_node_maintenance[safely]. To apply an older microcode
 185 potentially included in the microcode package for your CPU type, reboot now.
 186
 187 [TIP]
 188 ====
 189 It makes sense to hold the downgraded package for a while and try more recent
 190 versions again at a later time. Even if the package version is the same in the
 191 future, system updates may have fixed the experienced problem in the meantime.
 192 ----
 193 # apt-mark hold intel-microcode
 194 intel-microcode set on hold.
 195 ----
 196 ----
 197 # apt-mark unhold intel-microcode
 198 # apt update
 199 # apt upgrade
 200 ----
 201 ====