admin: include system-booting docs

[pmg-docs.git] / pmg-administration.adoc

Administration
==============

The Administration GUI allows you to carry out common tasks
such as updating software packages, managing quarantines, viewing the
status of services, and managing mail queues. It also provides server
statistics, in order to verify server health.


Server Administration
---------------------

Status
~~~~~~

[thumbnail="pmg-gui-server-status.png", big=1]

This page shows statistics about server CPU, memory, disk and network
usage. You can select the displayed time span from the upper right.

Administrators can open a terminal window using the 'Console'
button. It is also possible to trigger a server 'Restart' or
'Shutdown'.


Services
~~~~~~~~

[thumbnail="pmg-gui-service-status.png", big=1]

This panel lists all the major services used for mail processing and
cluster synchronization. If necessary, you can start, stop or restart
them. The 'Syslog' button shows the system log, filtered for the
selected service.

Please note that {pmg} uses {systemd} to manage services, so you can
also use the standard `systemctl` command-line tool to manage or view
service status, for example:

-----
systemctl status postfix
-----


[[pmg_updates]]
Updates
~~~~~~~

[thumbnail="pmg-gui-updates.png", big=1]

We release software updates on a regular basis, and it is recommended
to always run the latest available version. This page shows the
available updates, and administrators can run an upgrade by pressing
the 'Upgrade' button.

See section xref:pmg_package_repositories[Package Repositories] for
details about the available package repositories.


Syslog and Tasks
~~~~~~~~~~~~~~~~

[thumbnail="pmg-gui-syslog.png", big=1]

The Syslog page gives you a quick real-time log view. You can use the
xref:pmg_tracking_center[Tracking Center] to search the logs.

The Tasks page provides a history of the administration tasks that you
carried out on the server, such as upgrading the system. Each task
entry provides status information about the task, as well as the
output.

Quarantine
----------

[[pmgadministration_spam_quarantine]]
Spam
~~~~

[thumbnail="pmg-gui-spam-quarantine.png", big=1]

This panel lets you inspect the mail quarantine. Emails can be safely
previewed and if desired, delivered to the original user.

The email preview on the web interface is very secure, as malicious
code (attacking your operating system or email client) is removed by
{pmg}.

Users can access their personalized quarantine via the daily spam report or by
navigating to the URL configured for the quarantine (defaults to
`https://<pmg-host>:8006/quarantine`) and logging in with their LDAP credentials
(email address and password).

You can additionally enable user self-service for sending an access link from
the Quarantine Login page.
To enable this on the Quarantine Login page, edit `/etc/pmg/pmg.conf`. See section
xref:pmgconfig_spamdetector_quarantine[Spam Detector Configuration - Quarantine]
for more details about the available settings.


Virus
~~~~~

Allows administrators to inspect quarantined virus mails.


Attachment
~~~~~~~~~~

Allows administrators to inspect quarantined mails and download their
attachments or deliver/delete them.

NOTE: Use the options of the 'Remove attachment' action to control the Attachment Quarantine.


[[pmg_userblackwhitelist]]
User White- and Blacklist
~~~~~~~~~~~~~~~~~~~~~~~~~

This is mostly useful to debug or verify white- and blacklist user
settings. The administrator should not change these values because
users can manage this themselves.


[[pmg_tracking_center]]
Tracking Center
---------------

[thumbnail="pmg-gui-tracking-center.png", big=1]

Email processing is a complex task and involves several service
daemons. Each daemon logs information to the syslog service. The
problem is that a server analyzes many emails in parallel, so it is
usually very hard to find all logs corresponding to a specific mail.

The Tracking Center simplifies the search for
emails dramatically. We use highly optimized and safe Rust footnote:[A language
empowering everyone to build reliable and efficient software.
https://www.rust-lang.org/] code to search the available syslog data. This is
very fast and powerful, and works for sites processing several million emails
per day.

The result is a list of received mails, including the following data:

[cols="s,5d"]
|====
|Time | Timestamp of first syslog entry found
|From | Envelope 'From' address (the sender)
|To   | The email receiver address
|Status | Delivery status
|Syslog | The corresponding syslog entries are shown if you double
click such an entry or if you press the '+' button on the left
|====

To narrow the search down further, you can specify filters and set
a 'Start' and 'End' time. By default, the start time is set to the
last hour. If you still get too many entries, you can try to restrict
the search to a specific sender or receiver address, or search for a
specific text string in the logs ('Filter' entry).

NOTE: Search is faster if you use a shorter time interval.

The 'Status' field summarizes what happened with an email. {pmg} is a
mail proxy, meaning that the proxy receives mails from outside,
processes them and finally sends the result to the receiver.

The first phase is receiving the mail. The proxy may reject the mail
early or accept the mail and feed it into the filter. The filter
rules can then block or accept the mail.

In the second phase, accepted mails need to be delivered to the
receiver. This action may also fail or succeed. 'Status'
combines the results from the first and second phase.

[options="header",cols="2s,1d,5d"]
|====
|Status |Phase |Description
|rejected             |1 | Email rejected (for example, the sender IP is listed on an IP blacklist)
|greylisted           |1 | Email temporarily rejected by greylisting
|queued/deferred      |1 | Internal email was queued, still trying to deliver
|queued/bounced       |1 | Internal email was queued but not accepted by the target email server (for example, user unknown)
|queued/delivered     |1 | Internal email was queued and delivered
|quarantine           |1 | Email was moved to quarantine
|blocked              |1 | Email was blocked by filter rules
|accepted/deferred    |2 | Email accepted, still trying to deliver
|accepted/bounced     |2 | Email accepted, but not accepted by the target email server (for example, user unknown)
|accepted/delivered   |2 | Email accepted and delivered
|====

[[postfix_queue_administration]]
Postfix Queue Administration
----------------------------

[thumbnail="pmg-gui-queue-admin-summary.png", big=1]

Mail-queues are one of the central concepts of the SMTP protocol. Once a
mail server accepts a mail for further processing it saves it to a queue.
After the mail is either relayed to another system, stored locally
or discarded, it is deleted from the local mail-queue.

If immediate processing is not possible, for example because a downstream
mail server is not reachable, the mail remains on the queue for later
processing.

The 'Queue Administration' panel provides a summary about the current state
of the postfix mail-queue, similar to the 'qshape (1)' command-line utility.
It shows domains for which mails were not delivered, and how long they have
been queued.

The three Action Buttons on top provide the most common queue operations:

'Flush Queue'::

Attempt to deliver all currently queued mail, for example if a downstream
server has become available again.

'Delete All Messages'::

Delete all currently queued mail, for example if the queue contains only spam.

'Discard address verification database'::

Clear the recipient verification cache.

A sudden increase in queued mails should be closely inspected. This
increase can indicate issues connecting to downstream servers or that
one of the servers for which you relay emails sends spam itself.

Deferred Mail
~~~~~~~~~~~~~
[thumbnail="pmg-gui-queue-admin-deferred.png"]

In the 'Deferred Mail' tab, you can examine each deferred email
separately. In addition to providing contact information about the
sender and receiver, you can also check the reason for which an email
remains queued.

You can view the complete headers and filter by sender or receiver of
queued emails.

Here, you can also flush or delete each deferred email independently.


[[pmg_firmware_updates]]
Firmware Updates
----------------
Firmware updates from this chapter should be applied when running {pmg} or
Debian on a bare-metal server. Whether configuring firmware updates is
appropriate within a virtualized environment, e.g. when using device
pass-through, depends strongly on your setup and is therefore out of scope.

In addition to regular software updates, firmware updates are also important for
reliable and secure operation.

When obtaining and applying firmware updates, a combination of available options
is recommended to get them as early as possible or at all.

The term firmware is usually divided linguistically into microcode (for CPUs)
and firmware (for other devices).


[[pmg_firmware_persistent]]
Persistent Firmware
~~~~~~~~~~~~~~~~~~~
This section is suitable for all devices. Updated microcode, which is usually
included in a BIOS/UEFI update, is stored on the motherboard, whereas other
firmware is stored on the respective device. This persistent method is
especially important for the CPU, as it enables the earliest possible regular
loading of the updated microcode at boot time.

CAUTION: With some updates, such as for BIOS/UEFI or storage controller, the
device configuration could be reset. Please follow the vendor's instructions
carefully and back up the current configuration.

Please check with your vendor which update methods are available.

* Convenient update methods for servers can include Dell's Lifecycle Manager or
Service Packs from HPE.

* Sometimes there are Linux utilities available as well. Examples are
https://network.nvidia.com/support/firmware/mlxup-mft/['mlxup'] for NVIDIA
ConnectX or
https://techdocs.broadcom.com/us/en/storage-and-ethernet-connectivity/ethernet-nic-controllers/bcm957xxx/adapters/software-installation/updating-the-firmware/manually-updating-the-adapter-firmware-on-linuxesx.html['bnxtnvm'/'niccli']
for Broadcom network cards.

* https://fwupd.org[LVFS] could also be an option if there is a cooperation with
a https://fwupd.org/lvfs/vendors/[vendor] and
https://fwupd.org/lvfs/devices/[supported hardware] in use. The technical
requirement for this is that the system was manufactured after 2014, is booted
via UEFI and the easiest way is to mount the EFI partition from which you boot
(`mount /dev/disk/by-partuuid/<from efibootmgr -v> /boot/efi`) before installing
'fwupd'.

TIP: If the update instructions require a host reboot, please do not forget
about it.


[[pmg_firmware_runtime_files]]
Runtime Firmware Files
~~~~~~~~~~~~~~~~~~~~~~
This method stores firmware on the {pmg} operating system and will pass it to a
device if its xref:pmg_firmware_persistent[persisted firmware] is less recent.
It is supported by devices such as network and graphics cards, but not by those
that rely on persisted firmware such as the motherboard and hard disks.

In {pmg} the package `pve-firmware` is already installed by default. Therefore,
with the normal xref:pmg_updates[system updates (APT)], included firmware of
common hardware is automatically kept up to date.

An additional xref:pmg_debian_firmware_repo[Debian Firmware Repository] exists,
but is not configured by default.

If you try to install an additional firmware package but it conflicts, APT will
abort the installation. Perhaps the particular firmware can be obtained in
another way.


[[pmg_firmware_cpu]]
CPU Microcode Updates
~~~~~~~~~~~~~~~~~~~~~
Microcode updates are intended to fix found security vulnerabilities and other
serious CPU bugs. While the CPU performance can be affected, a patched microcode
is usually still more performant than an unpatched microcode where the kernel
itself has to do mitigations. Depending on the CPU type, it is possible that
performance results of the flawed factory state can no longer be achieved
without knowingly running the CPU in an unsafe state.

To get an overview of present CPU vulnerabilities and their mitigations, run
`lscpu`. Current real-world known vulnerabilities can only show up if the {pmg}
host is xref:pmg_updates[up to date], its version not
xref:faq-support-table[end of life], and has at least been rebooted since the
last kernel update.

Besides the recommended microcode update via
xref:pmg_firmware_persistent[persistent] BIOS/UEFI updates, there is also an
independent method via *Early OS Microcode Updates*. It is convenient to use and
also quite helpful when the motherboard vendor no longer provides BIOS/UEFI
updates. Regardless of the method in use, a reboot is always needed to apply a
microcode update.


Set up Early OS Microcode Updates
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
To set up microcode updates that are applied early on boot by the Linux kernel,
you need to:

. Enable the xref:pmg_debian_firmware_repo[Debian Firmware Repository]
. Get the latest available packages: `apt update` (or use the web interface,
  under Administration -> Updates)
. Install the CPU-vendor specific microcode package:
  - For Intel CPUs:  `apt install intel-microcode`
  - For AMD CPUs:  `apt install amd64-microcode`
. Reboot the {pmg} host

Any future microcode update will also require a reboot to be loaded.


Microcode Version
^^^^^^^^^^^^^^^^^
To get the current running microcode revision for comparison or debugging
purposes:

----
# grep microcode /proc/cpuinfo | uniq
microcode       : 0xf0
----

A microcode package has updates for many different CPUs. But updates
specifically for your CPU might not come often. So, just looking at the date on
the package won't tell you when the company actually released an update for your
specific CPU.

If you've installed a new microcode package and rebooted your {pmg} host, and
this new microcode is newer than both, the version baked into the CPU and the
one from the motherboard's firmware, you'll see a message in the system log
saying "microcode updated early".

----
# dmesg | grep microcode
[    0.000000] microcode: microcode updated early to revision 0xf0, date = 2021-11-12
[    0.896580] microcode: Microcode Update Driver: v2.2.
----


[[pmg_firmware_troubleshooting]]
Troubleshooting
^^^^^^^^^^^^^^^
For debugging purposes, the set up Early OS Microcode Update applied regularly
at system boot can be temporarily disabled as follows:

. Reboot the host to get to the GRUB menu (hold `SHIFT` if it is hidden)
. At the desired {pmg} boot entry press `E`
. Go to the line which starts with `linux` and append separated by a space
*`dis_ucode_ldr`*
. Press `CTRL-X` to boot this time without an Early OS Microcode Update

If a problem related to a recent microcode update is suspected, a package
downgrade should be considered instead of package removal
(`apt purge <intel-microcode|amd64-microcode>`). Otherwise, a too old
xref:pmg_firmware_persistent[persisted] microcode might be loaded, even
though a more recent one would run without problems.

A downgrade is possible if an earlier microcode package version is
available in the Debian repository, as shown in this example:

----
# apt list -a intel-microcode
Listing... Done
intel-microcode/stable-security,now 3.20230808.1~deb12u1 amd64 [installed]
intel-microcode/stable 3.20230512.1 amd64
----
----
# apt install intel-microcode=3.202305*
...
Selected version '3.20230512.1' (Debian:12.1/stable [amd64]) for 'intel-microcode'
...
dpkg: warning: downgrading intel-microcode from 3.20230808.1~deb12u1 to 3.20230512.1
...
intel-microcode: microcode will be updated at next boot
...
----

To apply an older microcode potentially included in the microcode package for
your CPU type, reboot now.

[TIP]
====
It makes sense to hold the downgraded package for a while and try more recent
versions again at a later time. Even if the package version is the same in the
future, system updates may have fixed the experienced problem in the meantime.
----
# apt-mark hold intel-microcode
intel-microcode set on hold.
----
----
# apt-mark unhold intel-microcode
# apt update
# apt upgrade
----
====


include::system-booting.adoc[]