X-Git-Url: https://git.proxmox.com/?p=pmg-docs.git;a=blobdiff_plain;f=pmg-administration.adoc;h=1018db4941a8fc39905950906668ce741a9188f2;hp=020558670c26d34d3878db93d6a06dff647f154c;hb=HEAD;hpb=268f309d0cc7d4f03ede0dca2f02ee1a05b5f694 diff --git a/pmg-administration.adoc b/pmg-administration.adoc index 0205586..8f632dc 100644 --- a/pmg-administration.adoc +++ b/pmg-administration.adoc @@ -1,22 +1,22 @@ Administration ============== -The Administration GUI allows you to do common administration tasks -like updating software packages, manage quarantine, view service -status and manage mail queues. It also provides server statistics in -order to verify server health. +The Administration GUI allows you to carry out common tasks +such as updating software packages, managing quarantines, viewing the +status of services, and managing mail queues. It also provides server +statistics, in order to verify server health. Server Administration --------------------- -Server status -~~~~~~~~~~~~~ +Status +~~~~~~ -image::images/screenshot/pmg-gui-server-status.png[] +[thumbnail="screenshot/pmg-gui-server-status.png", big=1] -This page shows server statistics about CPU, memory, disk and network -usage. You can select the displayed time span on the upper right. +This page shows statistics about server CPU, memory, disk and network +usage. You can select the displayed time span from the upper right. Administrators can open a terminal window using the 'Console' button. It is also possible to trigger a server 'Restart' or @@ -26,15 +26,15 @@ button. It is also possible to trigger a server 'Restart' or Services ~~~~~~~~ -image::images/screenshot/pmg-gui-service-status.png[] +[thumbnail="screenshot/pmg-gui-service-status.png", big=1] -This panel lists all major services used for mail processing and +This panel lists all the major services used for mail processing and cluster synchronization. If necessary, you can start, stop or restart -them. The 'Syslog' button shows the system log filtered for the +them. The 'Syslog' button shows the system log, filtered for the selected service. Please note that {pmg} uses {systemd} to manage services, so you can -also use the standard `systemctl` command line tool to manage or view +also use the standard `systemctl` command-line tool to manage or view service status, for example: ----- @@ -42,85 +42,402 @@ systemctl status postfix ----- +[[pmg_updates]] Updates ~~~~~~~ -image::images/screenshot/pmg-gui-updates.png[] +[thumbnail="screenshot/pmg-gui-updates.png", big=1] We release software updates on a regular basis, and it is recommended to always run the latest available version. This page shows the -available updates, and administrator can run an upgrade by pressing +available updates, and administrators can run an upgrade by pressing the 'Upgrade' button. See section xref:pmg_package_repositories[Package Repositories] for -details abaout available package repositories. +details about the available package repositories. Syslog and Tasks ~~~~~~~~~~~~~~~~ -image::images/screenshot/pmg-gui-syslog.png[] +[thumbnail="screenshot/pmg-gui-syslog.png", big=1] -The syslog page gives you a quick real-time log view. Please use the +The Syslog page gives you a quick real-time log view. You can use the xref:pmg_tracking_center[Tracking Center] to search the logs. +The Tasks page provides a history of the administration tasks that you +carried out on the server, such as upgrading the system. Each task +entry provides status information about the task, as well as the +output. Quarantine ---------- +[[pmgadministration_spam_quarantine]] Spam ~~~~ -DODO +[thumbnail="screenshot/pmg-gui-spam-quarantine.png", big=1] + +This panel lets you inspect the mail quarantine. Emails can be safely +previewed and if desired, delivered to the original user. + +The email preview on the web interface is very secure, as malicious +code (attacking your operating system or email client) is removed by +{pmg}. + +Users can access their personalized quarantine via the daily spam report or by +navigating to the URL configured for the quarantine (defaults to +`https://:8006/quarantine`) and logging in with their LDAP credentials +(email address and password). + +You can additionally enable user self-service for sending an access link from +the Quarantine Login page. +To enable this on the Quarantine Login page, edit `/etc/pmg/pmg.conf`. See section +xref:pmgconfig_spamdetector_quarantine[Spam Detector Configuration - Quarantine] +for more details about the available settings. + Virus ~~~~~ -TODO +Allows administrators to inspect quarantined virus mails. + + +Attachment +~~~~~~~~~~ +Allows administrators to inspect quarantined mails and download their +attachments or deliver/delete them. + +NOTE: Use the options of the 'Remove attachment' action to control the Attachment Quarantine. + + +[[pmg_userblackwhitelist]] User White- and Blacklist ~~~~~~~~~~~~~~~~~~~~~~~~~ -TODO +This is mostly useful to debug or verify white- and blacklist user +settings. The administrator should not change these values because +users can manage this themselves. + [[pmg_tracking_center]] Tracking Center --------------- -image::images/screenshot/pmg-gui-tracking-center.png[] +[thumbnail="screenshot/pmg-gui-tracking-center.png", big=1] Email processing is a complex task and involves several service daemons. Each daemon logs information to the syslog service. The -problem is that a servers analyzes many emails in parallel, so it is +problem is that a server analyzes many emails in parallel, so it is usually very hard to find all logs corresponding to a specific mail. -Introduced in {pmg} 2.1, the tracking center simplifies the search for -emails dramatically. We use highly optimized C-code to search the -available syslog data. This is very fast and powerful, and works for -sites processing several million emails per day. +The Tracking Center simplifies the search for +emails dramatically. We use highly optimized and safe Rust footnote:[A language +empowering everyone to build reliable and efficient software. +https://www.rust-lang.org/] code to search the available syslog data. This is +very fast and powerful, and works for sites processing several million emails +per day. The result is a list of received mails, including the following data: [cols="s,5d"] |==== -|Time | Timestamp of first found syslog entry. -|From | Envelope 'From' address (the sender). -|To | The email receiver address. -|Status | Delivery status. -|Syslog | The corresponding syslog entries are shown if you double click such -entry, or if you press the '+' button on the left. +|Time | Timestamp of first syslog entry found +|From | Envelope 'From' address (the sender) +|To | The email receiver address +|Status | Delivery status +|Syslog | The corresponding syslog entries are shown if you double +click such an entry or if you press the '+' button on the left |==== -Please notice that you can specify filters, most important you can set -a 'Start' and 'End' time. By default the start time is set to one hour -ago. If you still get to much result entries, you can try to restrict -the search to specific sender or receiver addresses, or search for a -specific text in the logs ('Filter' entry). +To narrow the search down further, you can specify filters and set +a 'Start' and 'End' time. By default, the start time is set to the +last hour. If you still get too many entries, you can try to restrict +the search to a specific sender or receiver address, or search for a +specific text string in the logs ('Filter' entry). -NOTE: Search is faster if you use a short time interval. +NOTE: Search is faster if you use a shorter time interval. +The 'Status' field summarizes what happened with an email. {pmg} is a +mail proxy, meaning that the proxy receives mails from outside, +processes them and finally sends the result to the receiver. +The first phase is receiving the mail. The proxy may reject the mail +early or accept the mail and feed it into the filter. The filter +rules can then block or accept the mail. + +In the second phase, accepted mails need to be delivered to the +receiver. This action may also fail or succeed. 'Status' +combines the results from the first and second phase. + +[options="header",cols="2s,1d,5d"] +|==== +|Status |Phase |Description +|rejected |1 | Email rejected (for example, the sender IP is listed on an IP blacklist) +|greylisted |1 | Email temporarily rejected by greylisting +|queued/deferred |1 | Internal email was queued, still trying to deliver +|queued/bounced |1 | Internal email was queued but not accepted by the target email server (for example, user unknown) +|queued/delivered |1 | Internal email was queued and delivered +|quarantine |1 | Email was moved to quarantine +|blocked |1 | Email was blocked by filter rules +|accepted/deferred |2 | Email accepted, still trying to deliver +|accepted/bounced |2 | Email accepted, but not accepted by the target email server (for example, user unknown) +|accepted/delivered |2 | Email accepted and delivered +|==== + +[[postfix_queue_administration]] Postfix Queue Administration ---------------------------- -TODO +[thumbnail="screenshot/pmg-gui-queue-admin-summary.png", big=1] + +Mail-queues are one of the central concepts of the SMTP protocol. Once a +mail server accepts a mail for further processing it saves it to a queue. +After the mail is either relayed to another system, stored locally +or discarded, it is deleted from the local mail-queue. + +If immediate processing is not possible, for example because a downstream +mail server is not reachable, the mail remains on the queue for later +processing. + +The 'Queue Administration' panel provides a summary about the current state +of the postfix mail-queue, similar to the 'qshape (1)' command-line utility. +It shows domains for which mails were not delivered, and how long they have +been queued. + +The three Action Buttons on top provide the most common queue operations: + +'Flush Queue':: + +Attempt to deliver all currently queued mail, for example if a downstream +server has become available again. + +'Delete All Messages':: + +Delete all currently queued mail, for example if the queue contains only spam. + +'Discard address verification database':: + +Clear the recipient verification cache. + +A sudden increase in queued mails should be closely inspected. This +increase can indicate issues connecting to downstream servers or that +one of the servers for which you relay emails sends spam itself. + +Deferred Mail +~~~~~~~~~~~~~ +[thumbnail="screenshot/pmg-gui-queue-admin-deferred.png"] + +In the 'Deferred Mail' tab, you can examine each deferred email +separately. In addition to providing contact information about the +sender and receiver, you can also check the reason for which an email +remains queued. + +You can view the complete headers and filter by sender or receiver of +queued emails. + +Here, you can also flush or delete each deferred email independently. + + +[[pmg_firmware_updates]] +Firmware Updates +---------------- +Firmware updates from this chapter should be applied when running {pmg} or +Debian on a bare-metal server. Whether configuring firmware updates is +appropriate within a virtualized environment, e.g. when using device +pass-through, depends strongly on your setup and is therefore out of scope. + +In addition to regular software updates, firmware updates are also important for +reliable and secure operation. + +When obtaining and applying firmware updates, a combination of available options +is recommended to get them as early as possible or at all. + +The term firmware is usually divided linguistically into microcode (for CPUs) +and firmware (for other devices). + + +[[pmg_firmware_persistent]] +Persistent Firmware +~~~~~~~~~~~~~~~~~~~ +This section is suitable for all devices. Updated microcode, which is usually +included in a BIOS/UEFI update, is stored on the motherboard, whereas other +firmware is stored on the respective device. This persistent method is +especially important for the CPU, as it enables the earliest possible regular +loading of the updated microcode at boot time. + +CAUTION: With some updates, such as for BIOS/UEFI or storage controller, the +device configuration could be reset. Please follow the vendor's instructions +carefully and back up the current configuration. + +Please check with your vendor which update methods are available. + +* Convenient update methods for servers can include Dell's Lifecycle Manager or +Service Packs from HPE. + +* Sometimes there are Linux utilities available as well. Examples are +https://network.nvidia.com/support/firmware/mlxup-mft/['mlxup'] for NVIDIA +ConnectX or +https://techdocs.broadcom.com/us/en/storage-and-ethernet-connectivity/ethernet-nic-controllers/bcm957xxx/adapters/software-installation/updating-the-firmware/manually-updating-the-adapter-firmware-on-linuxesx.html['bnxtnvm'/'niccli'] +for Broadcom network cards. + +* https://fwupd.org[LVFS] could also be an option if there is a cooperation with +a https://fwupd.org/lvfs/vendors/[vendor] and +https://fwupd.org/lvfs/devices/[supported hardware] in use. The technical +requirement for this is that the system was manufactured after 2014, is booted +via UEFI and the easiest way is to mount the EFI partition from which you boot +(`mount /dev/disk/by-partuuid/ /boot/efi`) before installing +'fwupd'. + +TIP: If the update instructions require a host reboot, please do not forget +about it. + + +[[pmg_firmware_runtime_files]] +Runtime Firmware Files +~~~~~~~~~~~~~~~~~~~~~~ +This method stores firmware on the {pmg} operating system and will pass it to a +device if its xref:pmg_firmware_persistent[persisted firmware] is less recent. +It is supported by devices such as network and graphics cards, but not by those +that rely on persisted firmware such as the motherboard and hard disks. + +In {pmg} the package `pve-firmware` is already installed by default. Therefore, +with the normal xref:pmg_updates[system updates (APT)], included firmware of +common hardware is automatically kept up to date. + +An additional xref:pmg_debian_firmware_repo[Debian Firmware Repository] exists, +but is not configured by default. + +If you try to install an additional firmware package but it conflicts, APT will +abort the installation. Perhaps the particular firmware can be obtained in +another way. + + +[[pmg_firmware_cpu]] +CPU Microcode Updates +~~~~~~~~~~~~~~~~~~~~~ +Microcode updates are intended to fix found security vulnerabilities and other +serious CPU bugs. While the CPU performance can be affected, a patched microcode +is usually still more performant than an unpatched microcode where the kernel +itself has to do mitigations. Depending on the CPU type, it is possible that +performance results of the flawed factory state can no longer be achieved +without knowingly running the CPU in an unsafe state. + +To get an overview of present CPU vulnerabilities and their mitigations, run +`lscpu`. Current real-world known vulnerabilities can only show up if the {pmg} +host is xref:pmg_updates[up to date], its version not +xref:faq-support-table[end of life], and has at least been rebooted since the +last kernel update. + +Besides the recommended microcode update via +xref:pmg_firmware_persistent[persistent] BIOS/UEFI updates, there is also an +independent method via *Early OS Microcode Updates*. It is convenient to use and +also quite helpful when the motherboard vendor no longer provides BIOS/UEFI +updates. Regardless of the method in use, a reboot is always needed to apply a +microcode update. + + +Set up Early OS Microcode Updates +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +To set up microcode updates that are applied early on boot by the Linux kernel, +you need to: + +. Enable the xref:pmg_debian_firmware_repo[Debian Firmware Repository] +. Get the latest available packages: `apt update` (or use the web interface, + under Administration -> Updates) +. Install the CPU-vendor specific microcode package: + - For Intel CPUs: `apt install intel-microcode` + - For AMD CPUs: `apt install amd64-microcode` +. Reboot the {pmg} host + +Any future microcode update will also require a reboot to be loaded. + + +Microcode Version +^^^^^^^^^^^^^^^^^ +To get the current running microcode revision for comparison or debugging +purposes: + +---- +# grep microcode /proc/cpuinfo | uniq +microcode : 0xf0 +---- + +A microcode package has updates for many different CPUs. But updates +specifically for your CPU might not come often. So, just looking at the date on +the package won't tell you when the company actually released an update for your +specific CPU. + +If you've installed a new microcode package and rebooted your {pmg} host, and +this new microcode is newer than both, the version baked into the CPU and the +one from the motherboard's firmware, you'll see a message in the system log +saying "microcode updated early". + +---- +# dmesg | grep microcode +[ 0.000000] microcode: microcode updated early to revision 0xf0, date = 2021-11-12 +[ 0.896580] microcode: Microcode Update Driver: v2.2. +---- + + +[[pmg_firmware_troubleshooting]] +Troubleshooting +^^^^^^^^^^^^^^^ +For debugging purposes, the set up Early OS Microcode Update applied regularly +at system boot can be temporarily disabled as follows: + +. Reboot the host to get to the GRUB menu (hold `SHIFT` if it is hidden) +. At the desired {pmg} boot entry press `E` +. Go to the line which starts with `linux` and append separated by a space +*`dis_ucode_ldr`* +. Press `CTRL-X` to boot this time without an Early OS Microcode Update + +If a problem related to a recent microcode update is suspected, a package +downgrade should be considered instead of package removal +(`apt purge `). Otherwise, a too old +xref:pmg_firmware_persistent[persisted] microcode might be loaded, even +though a more recent one would run without problems. + +A downgrade is possible if an earlier microcode package version is +available in the Debian repository, as shown in this example: + +---- +# apt list -a intel-microcode +Listing... Done +intel-microcode/stable-security,now 3.20230808.1~deb12u1 amd64 [installed] +intel-microcode/stable 3.20230512.1 amd64 +---- +---- +# apt install intel-microcode=3.202305* +... +Selected version '3.20230512.1' (Debian:12.1/stable [amd64]) for 'intel-microcode' +... +dpkg: warning: downgrading intel-microcode from 3.20230808.1~deb12u1 to 3.20230512.1 +... +intel-microcode: microcode will be updated at next boot +... +---- + +To apply an older microcode potentially included in the microcode package for +your CPU type, reboot now. + +[TIP] +==== +It makes sense to hold the downgraded package for a while and try more recent +versions again at a later time. Even if the package version is the same in the +future, system updates may have fixed the experienced problem in the meantime. +---- +# apt-mark hold intel-microcode +intel-microcode set on hold. +---- +---- +# apt-mark unhold intel-microcode +# apt update +# apt upgrade +---- +==== + + +include::system-booting.adoc[]