Administration
==============
-The Administration GUI allows you to do common tasks
-such as updating software packages, managing quarantine, viewing service
-status, and managing mail queues. It also provides server statistics in
-order to verify server health.
+The Administration GUI allows you to carry out common tasks
+such as updating software packages, managing quarantines, viewing the
+status of services, and managing mail queues. It also provides server
+statistics, in order to verify server health.
Server Administration
---------------------
-Server status
-~~~~~~~~~~~~~
+Status
+~~~~~~
-[thumbnail="pmg-gui-server-status.png", big=1]
+[thumbnail="screenshot/pmg-gui-server-status.png", big=1]
-This page shows server statistics about CPU, memory, disk and network
-usage. You can select the displayed time span on the upper right.
+This page shows statistics about server CPU, memory, disk and network
+usage. You can select the displayed time span from the upper right.
Administrators can open a terminal window using the 'Console'
button. It is also possible to trigger a server 'Restart' or
Services
~~~~~~~~
-[thumbnail="pmg-gui-service-status.png", big=1]
+[thumbnail="screenshot/pmg-gui-service-status.png", big=1]
-This panel lists all major services used for mail processing and
+This panel lists all the major services used for mail processing and
cluster synchronization. If necessary, you can start, stop or restart
-them. The 'Syslog' button shows the system log filtered for the
+them. The 'Syslog' button shows the system log, filtered for the
selected service.
Please note that {pmg} uses {systemd} to manage services, so you can
-also use the standard `systemctl` command line tool to manage or view
+also use the standard `systemctl` command-line tool to manage or view
service status, for example:
-----
-----
+[[pmg_updates]]
Updates
~~~~~~~
-[thumbnail="pmg-gui-updates.png", big=1]
+[thumbnail="screenshot/pmg-gui-updates.png", big=1]
We release software updates on a regular basis, and it is recommended
to always run the latest available version. This page shows the
the 'Upgrade' button.
See section xref:pmg_package_repositories[Package Repositories] for
-details abaout available package repositories.
+details about the available package repositories.
Syslog and Tasks
~~~~~~~~~~~~~~~~
-[thumbnail="pmg-gui-syslog.png", big=1]
+[thumbnail="screenshot/pmg-gui-syslog.png", big=1]
-The syslog page gives you a quick real-time log view. You can use the
+The Syslog page gives you a quick real-time log view. You can use the
xref:pmg_tracking_center[Tracking Center] to search the logs.
+The Tasks page provides a history of the administration tasks that you
+carried out on the server, such as upgrading the system. Each task
+entry provides status information about the task, as well as the
+output.
Quarantine
----------
+[[pmgadministration_spam_quarantine]]
Spam
~~~~
-[thumbnail="pmg-gui-spam-quarantine.png", big=1]
+[thumbnail="screenshot/pmg-gui-spam-quarantine.png", big=1]
This panel lets you inspect the mail quarantine. Emails can be safely
previewed and if desired, delivered to the original user.
-The email preview on the web interface is very secure as malicious
+The email preview on the web interface is very secure, as malicious
code (attacking your operating system or email client) is removed by
{pmg}.
+Users can access their personalized quarantine via the daily spam report or by
+navigating to the URL configured for the quarantine (defaults to
+`https://<pmg-host>:8006/quarantine`) and logging in with their LDAP credentials
+(email address and password).
+
+You can additionally enable user self-service for sending an access link from
+the Quarantine Login page.
+To enable this on the Quarantine Login page, edit `/etc/pmg/pmg.conf`. See section
+xref:pmgconfig_spamdetector_quarantine[Spam Detector Configuration - Quarantine]
+for more details about the available settings.
+
Virus
~~~~~
Tracking Center
---------------
-[thumbnail="pmg-gui-tracking-center.png", big=1]
+[thumbnail="screenshot/pmg-gui-tracking-center.png", big=1]
Email processing is a complex task and involves several service
daemons. Each daemon logs information to the syslog service. The
[cols="s,5d"]
|====
-|Time | Timestamp of first found syslog entry.
-|From | Envelope 'From' address (the sender).
-|To | The email receiver address.
-|Status | Delivery status.
-|Syslog | The corresponding syslog entries are shown if you double click such
-entry, or if you press the '+' button on the left.
+|Time | Timestamp of first syslog entry found
+|From | Envelope 'From' address (the sender)
+|To | The email receiver address
+|Status | Delivery status
+|Syslog | The corresponding syslog entries are shown if you double
+click such an entry or if you press the '+' button on the left
|====
-You can specify filters, and most importantly you can set
-a 'Start' and 'End' time. By default the start time is set to one hour
-ago. If you still get too many entries, you can try to restrict
+To narrow the search down further, you can specify filters and set
+a 'Start' and 'End' time. By default, the start time is set to the
+last hour. If you still get too many entries, you can try to restrict
the search to a specific sender or receiver address, or search for a
-specific text in the logs ('Filter' entry).
+specific text string in the logs ('Filter' entry).
NOTE: Search is faster if you use a shorter time interval.
processes them and finally sends the result to the receiver.
The first phase is receiving the mail. The proxy may reject the mail
-early, or instead accepts the mail and feeds it into the filter. The filter
-rules can block or accept the mail.
+early or accept the mail and feed it into the filter. The filter
+rules can then block or accept the mail.
In the second phase, accepted mails need to be delivered to the
receiver. This action may also fail or succeed. 'Status'
-combines the result from the first and second phase.
-
-When before-queue filtering is enabled, we don't have the same delivery information as
-with after-queue filtering. In this case, if the mail is accepted by the filter,
-the status is shown as 'queued/accepted' instead of 'accepted/<status>'.
+combines the results from the first and second phase.
[options="header",cols="2s,1d,5d"]
|====
|Status |Phase |Description
-|rejected |1 | Email rejected (e.g. sender IP is listed on a IP blacklist)
+|rejected |1 | Email rejected (for example, the sender IP is listed on an IP blacklist)
|greylisted |1 | Email temporarily rejected by greylisting
-|queued/deferred |1 | Internal Email was queued, still trying to deliver
-|queued/bounced |1 | Internal Email was queued but not accepted by the target email server (for example user unknown)
-|queued/accepted |1 | Email was accepted, no delivery information available (before-queue only, replaces accepted/<status>)
+|queued/deferred |1 | Internal email was queued, still trying to deliver
+|queued/bounced |1 | Internal email was queued but not accepted by the target email server (for example, user unknown)
+|queued/delivered |1 | Internal email was queued and delivered
|quarantine |1 | Email was moved to quarantine
|blocked |1 | Email was blocked by filter rules
|accepted/deferred |2 | Email accepted, still trying to deliver
-|accepted/bounced |2 | Email accepted but not accepted by the target email server (for example user unknown)
+|accepted/bounced |2 | Email accepted, but not accepted by the target email server (for example, user unknown)
|accepted/delivered |2 | Email accepted and delivered
|====
-
+[[postfix_queue_administration]]
Postfix Queue Administration
----------------------------
+[thumbnail="screenshot/pmg-gui-queue-admin-summary.png", big=1]
Mail-queues are one of the central concepts of the SMTP protocol. Once a
-mailserver accepts a mail for further processing it saves it to a queue.
+mail server accepts a mail for further processing it saves it to a queue.
After the mail is either relayed to another system, stored locally
-or discarded it is deleted from the local mail-queue.
+or discarded, it is deleted from the local mail-queue.
If immediate processing is not possible, for example because a downstream
-mailserver is not reachable, the mail remains on the queue for later
+mail server is not reachable, the mail remains on the queue for later
processing.
-The Queue Administration view provides a succinct overview, as generated by
-the 'qshape (1)' utility, about the current state of the postfix mail-queue.
-
-It shows mails for which domains were not delivered, and how long they have
+The 'Queue Administration' panel provides a summary about the current state
+of the postfix mail-queue, similar to the 'qshape (1)' command-line utility.
+It shows domains for which mails were not delivered, and how long they have
been queued.
The three Action Buttons on top provide the most common queue operations:
'Delete All Messages'::
Delete all currently queued mail, for example if the queue contains only spam.
-Attempt to deliver all currently queued mail.
'Discard address verification database'::
Clear the recipient verification cache.
+A sudden increase in queued mails should be closely inspected. This
+increase can indicate issues connecting to downstream servers or that
+one of the servers for which you relay emails sends spam itself.
+
+Deferred Mail
+~~~~~~~~~~~~~
+[thumbnail="screenshot/pmg-gui-queue-admin-deferred.png"]
+
+In the 'Deferred Mail' tab, you can examine each deferred email
+separately. In addition to providing contact information about the
+sender and receiver, you can also check the reason for which an email
+remains queued.
+
+You can view the complete headers and filter by sender or receiver of
+queued emails.
+
+Here, you can also flush or delete each deferred email independently.
+
+
+[[pmg_firmware_updates]]
+Firmware Updates
+----------------
+Firmware updates from this chapter should be applied when running {pmg} or
+Debian on a bare-metal server. Whether configuring firmware updates is
+appropriate within a virtualized environment, e.g. when using device
+pass-through, depends strongly on your setup and is therefore out of scope.
+
+In addition to regular software updates, firmware updates are also important for
+reliable and secure operation.
+
+When obtaining and applying firmware updates, a combination of available options
+is recommended to get them as early as possible or at all.
-In the 'Deferred Mail' tab you can filter and examine deferred mail in more
-detail by looking at the headers.
+The term firmware is usually divided linguistically into microcode (for CPUs)
+and firmware (for other devices).
+
+
+[[pmg_firmware_persistent]]
+Persistent Firmware
+~~~~~~~~~~~~~~~~~~~
+This section is suitable for all devices. Updated microcode, which is usually
+included in a BIOS/UEFI update, is stored on the motherboard, whereas other
+firmware is stored on the respective device. This persistent method is
+especially important for the CPU, as it enables the earliest possible regular
+loading of the updated microcode at boot time.
+
+CAUTION: With some updates, such as for BIOS/UEFI or storage controller, the
+device configuration could be reset. Please follow the vendor's instructions
+carefully and back up the current configuration.
+
+Please check with your vendor which update methods are available.
+
+* Convenient update methods for servers can include Dell's Lifecycle Manager or
+Service Packs from HPE.
+
+* Sometimes there are Linux utilities available as well. Examples are
+https://network.nvidia.com/support/firmware/mlxup-mft/['mlxup'] for NVIDIA
+ConnectX or
+https://techdocs.broadcom.com/us/en/storage-and-ethernet-connectivity/ethernet-nic-controllers/bcm957xxx/adapters/software-installation/updating-the-firmware/manually-updating-the-adapter-firmware-on-linuxesx.html['bnxtnvm'/'niccli']
+for Broadcom network cards.
+
+* https://fwupd.org[LVFS] could also be an option if there is a cooperation with
+a https://fwupd.org/lvfs/vendors/[vendor] and
+https://fwupd.org/lvfs/devices/[supported hardware] in use. The technical
+requirement for this is that the system was manufactured after 2014, is booted
+via UEFI and the easiest way is to mount the EFI partition from which you boot
+(`mount /dev/disk/by-partuuid/<from efibootmgr -v> /boot/efi`) before installing
+'fwupd'.
+
+TIP: If the update instructions require a host reboot, please do not forget
+about it.
+
+
+[[pmg_firmware_runtime_files]]
+Runtime Firmware Files
+~~~~~~~~~~~~~~~~~~~~~~
+This method stores firmware on the {pmg} operating system and will pass it to a
+device if its xref:pmg_firmware_persistent[persisted firmware] is less recent.
+It is supported by devices such as network and graphics cards, but not by those
+that rely on persisted firmware such as the motherboard and hard disks.
+
+In {pmg} the package `pve-firmware` is already installed by default. Therefore,
+with the normal xref:pmg_updates[system updates (APT)], included firmware of
+common hardware is automatically kept up to date.
+
+An additional xref:pmg_debian_firmware_repo[Debian Firmware Repository] exists,
+but is not configured by default.
+
+If you try to install an additional firmware package but it conflicts, APT will
+abort the installation. Perhaps the particular firmware can be obtained in
+another way.
+
+
+[[pmg_firmware_cpu]]
+CPU Microcode Updates
+~~~~~~~~~~~~~~~~~~~~~
+Microcode updates are intended to fix found security vulnerabilities and other
+serious CPU bugs. While the CPU performance can be affected, a patched microcode
+is usually still more performant than an unpatched microcode where the kernel
+itself has to do mitigations. Depending on the CPU type, it is possible that
+performance results of the flawed factory state can no longer be achieved
+without knowingly running the CPU in an unsafe state.
+
+To get an overview of present CPU vulnerabilities and their mitigations, run
+`lscpu`. Current real-world known vulnerabilities can only show up if the {pmg}
+host is xref:pmg_updates[up to date], its version not
+xref:faq-support-table[end of life], and has at least been rebooted since the
+last kernel update.
+
+Besides the recommended microcode update via
+xref:pmg_firmware_persistent[persistent] BIOS/UEFI updates, there is also an
+independent method via *Early OS Microcode Updates*. It is convenient to use and
+also quite helpful when the motherboard vendor no longer provides BIOS/UEFI
+updates. Regardless of the method in use, a reboot is always needed to apply a
+microcode update.
+
+
+Set up Early OS Microcode Updates
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+To set up microcode updates that are applied early on boot by the Linux kernel,
+you need to:
+
+. Enable the xref:pmg_debian_firmware_repo[Debian Firmware Repository]
+. Get the latest available packages: `apt update` (or use the web interface,
+ under Administration -> Updates)
+. Install the CPU-vendor specific microcode package:
+ - For Intel CPUs: `apt install intel-microcode`
+ - For AMD CPUs: `apt install amd64-microcode`
+. Reboot the {pmg} host
+
+Any future microcode update will also require a reboot to be loaded.
+
+
+Microcode Version
+^^^^^^^^^^^^^^^^^
+To get the current running microcode revision for comparison or debugging
+purposes:
+
+----
+# grep microcode /proc/cpuinfo | uniq
+microcode : 0xf0
+----
+
+A microcode package has updates for many different CPUs. But updates
+specifically for your CPU might not come often. So, just looking at the date on
+the package won't tell you when the company actually released an update for your
+specific CPU.
+
+If you've installed a new microcode package and rebooted your {pmg} host, and
+this new microcode is newer than both, the version baked into the CPU and the
+one from the motherboard's firmware, you'll see a message in the system log
+saying "microcode updated early".
+
+----
+# dmesg | grep microcode
+[ 0.000000] microcode: microcode updated early to revision 0xf0, date = 2021-11-12
+[ 0.896580] microcode: Microcode Update Driver: v2.2.
+----
+
+
+[[pmg_firmware_troubleshooting]]
+Troubleshooting
+^^^^^^^^^^^^^^^
+For debugging purposes, the set up Early OS Microcode Update applied regularly
+at system boot can be temporarily disabled as follows:
+
+. Reboot the host to get to the GRUB menu (hold `SHIFT` if it is hidden)
+. At the desired {pmg} boot entry press `E`
+. Go to the line which starts with `linux` and append separated by a space
+*`dis_ucode_ldr`*
+. Press `CTRL-X` to boot this time without an Early OS Microcode Update
+
+If a problem related to a recent microcode update is suspected, a package
+downgrade should be considered instead of package removal
+(`apt purge <intel-microcode|amd64-microcode>`). Otherwise, a too old
+xref:pmg_firmware_persistent[persisted] microcode might be loaded, even
+though a more recent one would run without problems.
+
+A downgrade is possible if an earlier microcode package version is
+available in the Debian repository, as shown in this example:
+
+----
+# apt list -a intel-microcode
+Listing... Done
+intel-microcode/stable-security,now 3.20230808.1~deb12u1 amd64 [installed]
+intel-microcode/stable 3.20230512.1 amd64
+----
+----
+# apt install intel-microcode=3.202305*
+...
+Selected version '3.20230512.1' (Debian:12.1/stable [amd64]) for 'intel-microcode'
+...
+dpkg: warning: downgrading intel-microcode from 3.20230808.1~deb12u1 to 3.20230512.1
+...
+intel-microcode: microcode will be updated at next boot
+...
+----
+
+To apply an older microcode potentially included in the microcode package for
+your CPU type, reboot now.
+
+[TIP]
+====
+It makes sense to hold the downgraded package for a while and try more recent
+versions again at a later time. Even if the package version is the same in the
+future, system updates may have fixed the experienced problem in the meantime.
+----
+# apt-mark hold intel-microcode
+intel-microcode set on hold.
+----
+----
+# apt-mark unhold intel-microcode
+# apt update
+# apt upgrade
+----
+====
+
+
+include::system-booting.adoc[]
projects / pmg-docs.git / blobdiff