Stoiko Ivanov [Tue, 2 Feb 2021 13:03:16 +0000 (14:03 +0100)]
utils: allow '/' inside email address localpart
The change is motivated by a report in our community forum [0], where
a mail addressed to an address containing '/' in its local-part ended
up in the quarantine.
This is permitted by RFC5322 ([1]), and, probably more relevant,
happily accepted and processed by postfix.
Once inside the quarantine (or the statistic database) the records cannot
be displayed (due to the parameter verification failure).
This leaves the user unable to delete the quarantined mail.
Apart from the quarantine and statistics the 'pmg-email-address'
format is only used in the PBSConfig and the fetchmail configuration
(both of which are available only to the admin and can be still be
edited irrespective of the set localpart).
Stoiko Ivanov [Thu, 21 Jan 2021 15:51:04 +0000 (16:51 +0100)]
api: statistics: refactor detail calls
the API calls returning the detailed statistics for a particular
email use much common code.
This patch introduces a sub to be used in all detail api calls.
Stoiko Ivanov [Tue, 19 Jan 2021 10:38:14 +0000 (11:38 +0100)]
api: spamassassin: update local channels
This patch adds a helper to loop over all present Spamassassin
channels files in /etc/mail/spamassassin/channel.d and:
* import the included gpg key into sa-update's keyring
* run sa-update for each channel separately
the verbose argument of the helper is for reusing the code in
pmg-daily (where we only want to log errors and be less informative)
the $SA_UPDATE variable hardcoding the path of /usr/bin/sa-update was
dropped in favor of using 'sa-update' without path since we do have a
sensible setting of PATH everywhere, and hardcoding paths is
problematic (especially in usr-merged systems).
The choice of invoking sa-update for each channel separately, instead
of providing multiple '--channel' and '--gpgkey' options to a single
command was made to prevent downloading signatures, which were signed
by a key not configured for the channel.
Importing gpg-keys is also done with individual sa-update invocations,
because sa-update only imports the last present --import argument
(wrong use of Getopt::Long)
Stoiko Ivanov [Tue, 19 Jan 2021 10:38:12 +0000 (11:38 +0100)]
add helper for parsing SA channel.d files
RHEL/CentOS based SpamAssassin implementations ship an update script,
which reads shell snippets from
/etc/mail/spamassassin/channel.d/*.conf and uses the information there
to update SA rules from the configured channels [0].
Noticed the existence of this directory/mechanism while reading the
announcement of the updatechannel for the KAM ruleset [1].
Parsing the file as text, instead of sourcing it in a shell, since I
hope that the channel files distributed don't rely on running commands
to get the ruleset url and gpg key.
The parser has some minimal tests added (inspired by the
convert_size_test.pl from pve-common)
Dominik Csapak [Wed, 18 Nov 2020 10:59:36 +0000 (11:59 +0100)]
api2/quarantine: add global sendlink api call
this api call takes an email, checks it against the relay domains,
and prepares a custom quarantinelink for that email and sends it there
this has to happen unauthenticated, since the idea is that the user
want to access the quarantine but has no current ticket (and no
old spam report with a ticket)
we rate limit the requests by allowing only a request per 5 seconds
(to prevent dos'ing the internal mail server) and only
one request per user/hour
this api call is disabled by default
if admins want even more ratelimiting, they can setup something
like fail2ban to block hosts hitting this api call often
Stoiko Ivanov [Wed, 18 Nov 2020 14:52:54 +0000 (15:52 +0100)]
do not create /cluster/<cid> unconditionally
while looking through the spooldir creation we noticed the mkdir call
on a relative path. This creates a '/cluster/<cid>/' directory on each system
which has a cluster.conf (<cid> being the node's clusterid). This is not used
since the spooldirs are in '/var/spool/pmg/cluster/'
Simply drop the mkdir call, since the spooldirs get created upon cluster
creation (PMG::API2::Cluster::create) and joining to an existing cluster.
Stoiko Ivanov [Wed, 18 Nov 2020 14:52:53 +0000 (15:52 +0100)]
fix clustersync after node-deletion
This patch creates the spoolsdirs for a newly joining clusternode on the
master (/var/spool/pmg/cluster/<newnode-cid>/(spam|attachment|virus).
This is necessary in order to prevent a failing cluster-sync for nodes, joining
the cluster after that node has been deleted. (This happens if you remove
a node from the cluster and directly rejoin it to the same masternode):
On the first sync after a node was deleted (there is no section config for a
number < maxcid) each node tries to sync the quarantine for the deleted node
from the cluster (in order to be promotable to new master). This rsync
fails because the spooldir for that node never got created on the master.
The spooldir for a node gets created on the master on the first sync of a node
which can be 2 minutes after joining the cluster (and leaving it again).
Thomas Lamprecht [Wed, 18 Nov 2020 09:28:15 +0000 (10:28 +0100)]
pmgbackup: fix missing semicolon leading to weird effects
without as semicolon the code run at "compile" (initial parse) time,
not after that. Thus, every code error was attributed to an
compilation error, i.e., bad syntax or the like, not a runtime die.
This was visible as the program executed when using perl check
> # perl -wc src/bin/pmgbackup
which should normally not be the case
Originally-by: Stoiko Ivanov <s.ivanov@proxmox.com> Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Stoiko Ivanov [Thu, 29 Oct 2020 17:49:16 +0000 (18:49 +0100)]
reinject_email: fix connecting for ipv6-only hosts
When configuring PMG only with ipv6 addresses, reinject_email after processing
fails to connect to the postfix/smtpd instance (with EINVAL).
Setting the host to '::FFFF:127.0.0.1' fixes the issue.
Tested with:
* an ipv6only host (no ipv4 configured)
* a host with ipv6 disabled via sysctl:
```
net.ipv6.conf.all.disable_ipv6=1
net.ipv6.conf.default.disable_ipv6=1
net.ipv6.conf.lo.disable_ipv6=1
```
* a host with dual-stack setup
Stoiko Ivanov [Mon, 16 Nov 2020 11:01:13 +0000 (12:01 +0100)]
pbs-integration: add CLI calls to pmgbackup
This patch adds to new categories for commands to pmgbackup:
* pmgbackup remote - for managing PBS instances' configuration, cluster-wide
* pmgbackup pbsjob - for managing backups, restores, pruning
Stoiko Ivanov [Wed, 28 Oct 2020 18:54:21 +0000 (19:54 +0100)]
Restore: optionally restore from directory
In preparation for integrating PMG with PBS decide based on the type of the
provided filename, whether or not to untar:
* if it's a directory skip untarring (PBS)
* if it's a filename untar (local backup)
Stoiko Ivanov [Wed, 28 Oct 2020 18:54:20 +0000 (19:54 +0100)]
Backup: split backup creation and creating tar
In preparation for integrating PMG with PBS split the current creation of
a PMG backup into 2 methods:
* create all files in a backup in a target directory
* create a tarball from a backup in a temporary directory
Stoiko Ivanov [Thu, 18 Jun 2020 11:33:17 +0000 (13:33 +0200)]
add logging to disclaimer action
the disclaimer action currently does not log, if a disclaimer got added or not.
given that there are a few not directly obvious cases where a disclaimer does
not get added (e.g. it depends on the mail's encoding) - logging success or
failure should help in debugging
Tested by sending mails, where adding the disclaimer works, and where it fails.
verified that the log-tracker also adds those lines to its output.
Stoiko Ivanov [Wed, 17 Jun 2020 15:04:05 +0000 (17:04 +0200)]
prefix message-id in attachment-quarantine
This patch fixes #2785.
When using the attachment quarantine - the message is:
a) stored in the quarantine unaltered
b) sent on with the attachment removed
Currently we do not change the message in any other way - in particular
we do not change the message-id header of any of the 2 mails.
When a mail is released from the attachment quarantine it is sent by PMG
with the same message-id as the mail with the attachments removed.
This is a violation of RFC 5322 (see [0]), and additionally newer versions
of Exchange do accept 2 mails with the same message-id but silently discard
the second version, thus making the attachment quarantine unusable for
Exchange users.
This patch simply prefixes the message-id with 'pmg-aquar-$$' (where $$ is
the pid of the pmg-smtp-filter process) for the mail without attachment.
By keeping the original message-id in the headers tracing the mailflow should
be facilitated.
The Message-ID is left intact on the original message in order to keep DKIM
signatures valid (they are invalidated on the modified mail by the removal
of the attachment anyways).
Stoiko Ivanov [Thu, 28 May 2020 08:04:58 +0000 (10:04 +0200)]
fix #1976: optionally sort postfix queue result
The PostfixMailQueue widget uses an Ext.data.BufferedStore, due to
the potential size of the resultset, which does only support remoteSorting [0]
By adding two optional parameters ('sortfield' and 'sortdir') we can use
them for sorting the mailq output accordingly.
The sorting is kept in PMG::API2::Postfix instead of PMG::Postfix, because
sorting (as opposed to filtering) needs to happen after the complete result
is known, and there is no gain in pushing it further down.
[0] only mentioned in the source-code - not in the referencedoc
to keep the log from growing without end. Rotate monthly and keep 12 logs,
since the logs should not be too large (e.g. a productive instance, with
~20 users using the quarantine and some configuration changes amounts to
108M over 2.5 years)
the logrotate snippet is placed in /etc/logrotate.d/pmg-api by
dh_installlogrotate(1).
Thomas Lamprecht [Fri, 24 Apr 2020 06:59:19 +0000 (08:59 +0200)]
pmgsync.service: really order before postfix@-.service
followup for commit 0c4cf3f2cfa2b40d4fb1ded7501989b884c73eae
which assumed that we can order on templated base units, which we
cannot (at least under the systemd version of buster). So depend on
the actual instance of the main postfix template.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
pmgsync.service updates the database-schema (pmgdb init) and generates and
updates configuration files from the templates and reloads the respective
services.
When first booting up after installation, it adapts the config from what's
shipped in the default debian packages for the first time.
The postfix configuration is also rendered, including settings where a
restart is necessary (listening on the internal port (26)).
While the unit already starts before postfix.service, the postfix service
files are designed to start multiple instances of postfix via instantiation
(by default postfix@-.service is the single instance (and the service file
which actually starts postfix)).
Since both pmgsync and postfix@- have no ordering relation between them, they
are started in parallel, which leads to postfix starting with the stock config
upon first boot.
Tested by running the installer in debug mode and applying this patch in the
last debug shell.
the usepolicy variable is used by the templateing system to decide whether
pmgpolicy should be asked by postfix and should also be enabled if greylisting
is only active for ipv6.