firmware: rework setting up OS ucode updates to actionable list

[pve-docs.git] / firmware-updates.adoc

[[chapter_firmware_updates]]
Firmware Updates
----------------
ifdef::wiki[]
:pve-toplevel:
endif::wiki[]
Firmware updates from this chapter should be applied when running {pve} on a
bare-metal server. Whether configuring firmware updates is appropriate within
guests, e.g. when using device pass-through, depends strongly on your setup and
is therefore out of scope.

In addition to regular software updates, firmware updates are also important
for reliable and secure operation.

When obtaining and applying firmware updates, a combination of available options
is recommended to get them as early as possible or at all.

The term firmware is usually divided linguistically into microcode (for CPUs)
and firmware (for other devices).


[[sysadmin_firmware_persistent]]
Persistent Firmware
~~~~~~~~~~~~~~~~~~~
This section is suitable for all devices. Updated microcode, which is usually
included in a BIOS/UEFI update, is stored on the motherboard, whereas other
firmware is stored on the respective device. This persistent method is
especially important for the CPU, as it enables the earliest possible regular
loading of the updated microcode at boot time.

CAUTION: With some updates, such as for BIOS/UEFI or storage controller, the
device configuration could be reset. Please follow the vendor's instructions
carefully and back up the current configuration.

Please check with your vendor which update methods are available.

* Convenient update methods for servers can include Dell's Lifecycle Manager or
Service Packs from HPE.

* Sometimes there are Linux utilities available as well. Examples are
https://network.nvidia.com/support/firmware/mlxup-mft/['mlxup'] for NVIDIA
ConnectX or
https://techdocs.broadcom.com/us/en/storage-and-ethernet-connectivity/ethernet-nic-controllers/bcm957xxx/adapters/software-installation/updating-the-firmware/manually-updating-the-adapter-firmware-on-linuxesx.html['bnxtnvm'/'niccli']
for Broadcom network cards.

* https://fwupd.org[LVFS] could also be an option if there is a cooperation with
a https://fwupd.org/lvfs/vendors/[vendor] and
https://fwupd.org/lvfs/devices/[supported hardware] in use. The technical
requirement for this is that the system was manufactured after 2014, is booted
via UEFI and the easiest way is to mount the EFI partition from which you boot
(`mount /dev/disk/by-partuuid/<from efibootmgr -v> /boot/efi`) before installing
'fwupd'.

TIP: If the update instructions require a host reboot, make sure that it can be
done safely. See also xref:ha_manager_node_maintenance[Node Maintenance].


[[sysadmin_firmware_runtime_files]]
Runtime Firmware Files
~~~~~~~~~~~~~~~~~~~~~~
This method stores firmware on the {pve} operating system and will pass it to a
device if its xref:sysadmin_firmware_persistent[persisted firmware] is less
recent. It is supported by devices such as network and graphics cards, but not
by those that rely on persisted firmware such as the motherboard and hard disks.

In {pve} the package `pve-firmware` is already installed by default. Therefore,
with the normal xref:system_software_updates[system updates (APT)], included
firmware of common hardware is automatically kept up to date.

An additional xref:sysadmin_debian_firmware_repo[Debian Firmware Repository]
exists, but is not configured by default.

If you try to install an additional firmware package but it conflicts, APT will
abort the installation. Perhaps the particular firmware can be obtained in
another way.


[[sysadmin_firmware_cpu]]
CPU Microcode Updates
~~~~~~~~~~~~~~~~~~~~~
Microcode updates are intended to fix found security vulnerabilities and other
serious CPU bugs. While the CPU performance can be affected, a patched microcode
is usually still more performant than an unpatched microcode where the kernel
itself has to do mitigations. Depending on the CPU type, it is possible that
performance results of the flawed factory state can no longer be achieved
without knowingly running the CPU in an unsafe state.

To get an overview of present CPU vulnerabilities and their mitigations, run
`lscpu`. Current real-world known vulnerabilities can only show up if the
{pve} host is xref:system_software_updates[up to date], its version not
xref:faq-support-table[end of life], and has at least been rebooted since the
last kernel update.

Besides the recommended microcode update via
xref:sysadmin_firmware_persistent[persistent] BIOS/UEFI updates, there is also
an independent method via *Early OS Microcode Updates*. It is convenient to use
and also quite helpful when the motherboard vendor no longer provides BIOS/UEFI
updates. Regardless of the method in use, a reboot is always needed to apply a
microcode update.


Set up Early OS Microcode Updates
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
To set up microcode updates that are applied early on boot by the Linux kernel,
you need to:

. Enable the xref:sysadmin_debian_firmware_repo[Debian Firmware Repository]
. Get the latest available packages `apt update` (or use the web interface,
  under Node -> Updates)
. Install the CPU-vendor specific microcode package:
  - For Intel CPUs:  `apt install intel-microcode`
  - For AMD CPUs:  `apt install amd64-microcode`
. Reboot the {pve} host

Any future microcode update will also require a reboot to be loaded.

Microcode Version
^^^^^^^^^^^^^^^^^
To get the current running microcode revision for comparison or debugging
purposes:

----
# grep microcode /proc/cpuinfo | uniq
microcode	: 0xf0
----

Since a microcode package contains microcode for different CPU types, an updated
microcode for your CPU will only be included occasionally. Therefore, date
information from the package can also not be matched to a specific release date
from the vendor.

If the microcode package is installed, the system has been rebooted, and the
microcode included in the package is more recent than that on the motherboard
and CPU, the message "microcode updated early" appears in the log:

----
# dmesg | grep microcode
[    0.000000] microcode: microcode updated early to revision 0xf0, date = 2021-11-12
[    0.896580] microcode: Microcode Update Driver: v2.2.
----


[[sysadmin_firmware_troubleshooting]]
Troubleshooting
^^^^^^^^^^^^^^^
For debugging purposes, the set up Early OS Microcode Update applied regularly
at system boot can be temporarily disabled as follows:

1. make sure that the host can be rebooted xref:ha_manager_node_maintenance[safely]
2. reboot the host to get to the GRUB menu (hold `SHIFT` if it is hidden)
3. at the desired {pve} boot entry press `E`
4. go to the line which starts with `linux` and append separated by a space
*`dis_ucode_ldr`*
5. press `CTRL-X` to boot this time without an Early OS Microcode Update

If a problem related to a recent microcode update is suspected, a package
downgrade should be considered instead of package removal
(`apt purge <intel-microcode|amd64-microcode>`). Otherwise, a too old
xref:sysadmin_firmware_persistent[persisted] microcode might be loaded, even
though a more recent one would run without problems.

A downgrade is possible if an earlier microcode package version is
available in the Debian repository, as shown in this example:

----
# apt list -a intel-microcode
Listing... Done
intel-microcode/stable-security,now 3.20230808.1~deb12u1 amd64 [installed]
intel-microcode/stable 3.20230512.1 amd64
----
----
# apt install intel-microcode=3.202305*
...
Selected version '3.20230512.1' (Debian:12.1/stable [amd64]) for 'intel-microcode'
...
dpkg: warning: downgrading intel-microcode from 3.20230808.1~deb12u1 to 3.20230512.1
...
intel-microcode: microcode will be updated at next boot
...
----

Make sure (again) that the host can be rebooted
xref:ha_manager_node_maintenance[safely]. To apply an older microcode
potentially included in the microcode package for your CPU type, reboot now.

[TIP]
====
It makes sense to hold the downgraded package for a while and try more recent
versions again at a later time. Even if the package version is the same in the
future, system updates may have fixed the experienced problem in the meantime.
----
# apt-mark hold intel-microcode
intel-microcode set on hold.
----
----
# apt-mark unhold intel-microcode
# apt update
# apt upgrade
----
====