Stoiko Ivanov [Mon, 1 Mar 2021 14:12:20 +0000 (15:12 +0100)]
backup: add notify parameter to 'classic' backup
for feature-parity, and since we recently had a user in our community
forum, who does regular backups via cron+rsync (small site w/o
dedicated backup host). Those setups could also benefit from this.
Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com> Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Stoiko Ivanov [Mon, 1 Mar 2021 14:12:19 +0000 (15:12 +0100)]
backup: fix #3146 add email notifications
this patch addresses the missing email notification for scheduled
backup related tasks, which we have in all our other products, for our
mail product.
the parameter names are inspired by PBS' datastore config.
the default is 'never' in order to stay consistent with the current
code.
it uses the templateing system for the notification, because this
results in less code and a bit of added flexibility for the users.
the recipient address is currently hardcoded to the admin address in
pmg.conf as we also send the (admin) pmgreport there, and I did not
want to overengineer this (even more).
I shortly considered adding a $SIG{'__DIE__'} handler to the
run_backup API call but dropped the idea due to the warning in
perlvar(1).
Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com> Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Stoiko Ivanov [Mon, 1 Mar 2021 14:12:18 +0000 (15:12 +0100)]
backup: pbs: prevent race in concurrent backups
If two pbs backup-creation calls happen simultaneously, it is possible
that the first removes the backup dir before the other is done
creating or sending it to the pbs remote.
This patch takes the same route as non-PBS backups - creating a unique
tempdir indexed by remote, PID and current time.
the tmp-dir now also needs to be removed in case of error while
backing up. (before the next invocation would have wiped it).
Noticed while having 2 schedules to different PBS instances with the
same interval and w/o random delay.
Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com> Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Stoiko Ivanov [Tue, 2 Feb 2021 13:03:16 +0000 (14:03 +0100)]
utils: allow '/' inside email address localpart
The change is motivated by a report in our community forum [0], where
a mail addressed to an address containing '/' in its local-part ended
up in the quarantine.
This is permitted by RFC5322 ([1]), and, probably more relevant,
happily accepted and processed by postfix.
Once inside the quarantine (or the statistic database) the records cannot
be displayed (due to the parameter verification failure).
This leaves the user unable to delete the quarantined mail.
Apart from the quarantine and statistics the 'pmg-email-address'
format is only used in the PBSConfig and the fetchmail configuration
(both of which are available only to the admin and can be still be
edited irrespective of the set localpart).
Stoiko Ivanov [Thu, 21 Jan 2021 15:51:04 +0000 (16:51 +0100)]
api: statistics: refactor detail calls
the API calls returning the detailed statistics for a particular
email use much common code.
This patch introduces a sub to be used in all detail api calls.
Stoiko Ivanov [Tue, 19 Jan 2021 10:38:14 +0000 (11:38 +0100)]
api: spamassassin: update local channels
This patch adds a helper to loop over all present Spamassassin
channels files in /etc/mail/spamassassin/channel.d and:
* import the included gpg key into sa-update's keyring
* run sa-update for each channel separately
the verbose argument of the helper is for reusing the code in
pmg-daily (where we only want to log errors and be less informative)
the $SA_UPDATE variable hardcoding the path of /usr/bin/sa-update was
dropped in favor of using 'sa-update' without path since we do have a
sensible setting of PATH everywhere, and hardcoding paths is
problematic (especially in usr-merged systems).
The choice of invoking sa-update for each channel separately, instead
of providing multiple '--channel' and '--gpgkey' options to a single
command was made to prevent downloading signatures, which were signed
by a key not configured for the channel.
Importing gpg-keys is also done with individual sa-update invocations,
because sa-update only imports the last present --import argument
(wrong use of Getopt::Long)
Stoiko Ivanov [Tue, 19 Jan 2021 10:38:12 +0000 (11:38 +0100)]
add helper for parsing SA channel.d files
RHEL/CentOS based SpamAssassin implementations ship an update script,
which reads shell snippets from
/etc/mail/spamassassin/channel.d/*.conf and uses the information there
to update SA rules from the configured channels [0].
Noticed the existence of this directory/mechanism while reading the
announcement of the updatechannel for the KAM ruleset [1].
Parsing the file as text, instead of sourcing it in a shell, since I
hope that the channel files distributed don't rely on running commands
to get the ruleset url and gpg key.
The parser has some minimal tests added (inspired by the
convert_size_test.pl from pve-common)
Dominik Csapak [Wed, 18 Nov 2020 10:59:36 +0000 (11:59 +0100)]
api2/quarantine: add global sendlink api call
this api call takes an email, checks it against the relay domains,
and prepares a custom quarantinelink for that email and sends it there
this has to happen unauthenticated, since the idea is that the user
want to access the quarantine but has no current ticket (and no
old spam report with a ticket)
we rate limit the requests by allowing only a request per 5 seconds
(to prevent dos'ing the internal mail server) and only
one request per user/hour
this api call is disabled by default
if admins want even more ratelimiting, they can setup something
like fail2ban to block hosts hitting this api call often
Stoiko Ivanov [Wed, 18 Nov 2020 14:52:54 +0000 (15:52 +0100)]
do not create /cluster/<cid> unconditionally
while looking through the spooldir creation we noticed the mkdir call
on a relative path. This creates a '/cluster/<cid>/' directory on each system
which has a cluster.conf (<cid> being the node's clusterid). This is not used
since the spooldirs are in '/var/spool/pmg/cluster/'
Simply drop the mkdir call, since the spooldirs get created upon cluster
creation (PMG::API2::Cluster::create) and joining to an existing cluster.
Stoiko Ivanov [Wed, 18 Nov 2020 14:52:53 +0000 (15:52 +0100)]
fix clustersync after node-deletion
This patch creates the spoolsdirs for a newly joining clusternode on the
master (/var/spool/pmg/cluster/<newnode-cid>/(spam|attachment|virus).
This is necessary in order to prevent a failing cluster-sync for nodes, joining
the cluster after that node has been deleted. (This happens if you remove
a node from the cluster and directly rejoin it to the same masternode):
On the first sync after a node was deleted (there is no section config for a
number < maxcid) each node tries to sync the quarantine for the deleted node
from the cluster (in order to be promotable to new master). This rsync
fails because the spooldir for that node never got created on the master.
The spooldir for a node gets created on the master on the first sync of a node
which can be 2 minutes after joining the cluster (and leaving it again).
Thomas Lamprecht [Wed, 18 Nov 2020 09:28:15 +0000 (10:28 +0100)]
pmgbackup: fix missing semicolon leading to weird effects
without as semicolon the code run at "compile" (initial parse) time,
not after that. Thus, every code error was attributed to an
compilation error, i.e., bad syntax or the like, not a runtime die.
This was visible as the program executed when using perl check
> # perl -wc src/bin/pmgbackup
which should normally not be the case
Originally-by: Stoiko Ivanov <s.ivanov@proxmox.com> Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Stoiko Ivanov [Thu, 29 Oct 2020 17:49:16 +0000 (18:49 +0100)]
reinject_email: fix connecting for ipv6-only hosts
When configuring PMG only with ipv6 addresses, reinject_email after processing
fails to connect to the postfix/smtpd instance (with EINVAL).
Setting the host to '::FFFF:127.0.0.1' fixes the issue.
Tested with:
* an ipv6only host (no ipv4 configured)
* a host with ipv6 disabled via sysctl:
```
net.ipv6.conf.all.disable_ipv6=1
net.ipv6.conf.default.disable_ipv6=1
net.ipv6.conf.lo.disable_ipv6=1
```
* a host with dual-stack setup
Stoiko Ivanov [Mon, 16 Nov 2020 11:01:13 +0000 (12:01 +0100)]
pbs-integration: add CLI calls to pmgbackup
This patch adds to new categories for commands to pmgbackup:
* pmgbackup remote - for managing PBS instances' configuration, cluster-wide
* pmgbackup pbsjob - for managing backups, restores, pruning
Stoiko Ivanov [Wed, 28 Oct 2020 18:54:21 +0000 (19:54 +0100)]
Restore: optionally restore from directory
In preparation for integrating PMG with PBS decide based on the type of the
provided filename, whether or not to untar:
* if it's a directory skip untarring (PBS)
* if it's a filename untar (local backup)
Stoiko Ivanov [Wed, 28 Oct 2020 18:54:20 +0000 (19:54 +0100)]
Backup: split backup creation and creating tar
In preparation for integrating PMG with PBS split the current creation of
a PMG backup into 2 methods:
* create all files in a backup in a target directory
* create a tarball from a backup in a temporary directory
Stoiko Ivanov [Thu, 18 Jun 2020 11:33:17 +0000 (13:33 +0200)]
add logging to disclaimer action
the disclaimer action currently does not log, if a disclaimer got added or not.
given that there are a few not directly obvious cases where a disclaimer does
not get added (e.g. it depends on the mail's encoding) - logging success or
failure should help in debugging
Tested by sending mails, where adding the disclaimer works, and where it fails.
verified that the log-tracker also adds those lines to its output.
Stoiko Ivanov [Wed, 17 Jun 2020 15:04:05 +0000 (17:04 +0200)]
prefix message-id in attachment-quarantine
This patch fixes #2785.
When using the attachment quarantine - the message is:
a) stored in the quarantine unaltered
b) sent on with the attachment removed
Currently we do not change the message in any other way - in particular
we do not change the message-id header of any of the 2 mails.
When a mail is released from the attachment quarantine it is sent by PMG
with the same message-id as the mail with the attachments removed.
This is a violation of RFC 5322 (see [0]), and additionally newer versions
of Exchange do accept 2 mails with the same message-id but silently discard
the second version, thus making the attachment quarantine unusable for
Exchange users.
This patch simply prefixes the message-id with 'pmg-aquar-$$' (where $$ is
the pid of the pmg-smtp-filter process) for the mail without attachment.
By keeping the original message-id in the headers tracing the mailflow should
be facilitated.
The Message-ID is left intact on the original message in order to keep DKIM
signatures valid (they are invalidated on the modified mail by the removal
of the attachment anyways).
Stoiko Ivanov [Thu, 28 May 2020 08:04:58 +0000 (10:04 +0200)]
fix #1976: optionally sort postfix queue result
The PostfixMailQueue widget uses an Ext.data.BufferedStore, due to
the potential size of the resultset, which does only support remoteSorting [0]
By adding two optional parameters ('sortfield' and 'sortdir') we can use
them for sorting the mailq output accordingly.
The sorting is kept in PMG::API2::Postfix instead of PMG::Postfix, because
sorting (as opposed to filtering) needs to happen after the complete result
is known, and there is no gain in pushing it further down.
[0] only mentioned in the source-code - not in the referencedoc
projects / pmg-api.git / log