Dominik Csapak [Wed, 18 Nov 2020 10:59:36 +0000 (11:59 +0100)]
api2/quarantine: add global sendlink api call
this api call takes an email, checks it against the relay domains,
and prepares a custom quarantinelink for that email and sends it there
this has to happen unauthenticated, since the idea is that the user
want to access the quarantine but has no current ticket (and no
old spam report with a ticket)
we rate limit the requests by allowing only a request per 5 seconds
(to prevent dos'ing the internal mail server) and only
one request per user/hour
this api call is disabled by default
if admins want even more ratelimiting, they can setup something
like fail2ban to block hosts hitting this api call often
Stoiko Ivanov [Wed, 18 Nov 2020 14:52:54 +0000 (15:52 +0100)]
do not create /cluster/<cid> unconditionally
while looking through the spooldir creation we noticed the mkdir call
on a relative path. This creates a '/cluster/<cid>/' directory on each system
which has a cluster.conf (<cid> being the node's clusterid). This is not used
since the spooldirs are in '/var/spool/pmg/cluster/'
Simply drop the mkdir call, since the spooldirs get created upon cluster
creation (PMG::API2::Cluster::create) and joining to an existing cluster.
Stoiko Ivanov [Wed, 18 Nov 2020 14:52:53 +0000 (15:52 +0100)]
fix clustersync after node-deletion
This patch creates the spoolsdirs for a newly joining clusternode on the
master (/var/spool/pmg/cluster/<newnode-cid>/(spam|attachment|virus).
This is necessary in order to prevent a failing cluster-sync for nodes, joining
the cluster after that node has been deleted. (This happens if you remove
a node from the cluster and directly rejoin it to the same masternode):
On the first sync after a node was deleted (there is no section config for a
number < maxcid) each node tries to sync the quarantine for the deleted node
from the cluster (in order to be promotable to new master). This rsync
fails because the spooldir for that node never got created on the master.
The spooldir for a node gets created on the master on the first sync of a node
which can be 2 minutes after joining the cluster (and leaving it again).
Thomas Lamprecht [Wed, 18 Nov 2020 09:28:15 +0000 (10:28 +0100)]
pmgbackup: fix missing semicolon leading to weird effects
without as semicolon the code run at "compile" (initial parse) time,
not after that. Thus, every code error was attributed to an
compilation error, i.e., bad syntax or the like, not a runtime die.
This was visible as the program executed when using perl check
> # perl -wc src/bin/pmgbackup
which should normally not be the case
Originally-by: Stoiko Ivanov <s.ivanov@proxmox.com> Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Stoiko Ivanov [Thu, 29 Oct 2020 17:49:16 +0000 (18:49 +0100)]
reinject_email: fix connecting for ipv6-only hosts
When configuring PMG only with ipv6 addresses, reinject_email after processing
fails to connect to the postfix/smtpd instance (with EINVAL).
Setting the host to '::FFFF:127.0.0.1' fixes the issue.
Tested with:
* an ipv6only host (no ipv4 configured)
* a host with ipv6 disabled via sysctl:
```
net.ipv6.conf.all.disable_ipv6=1
net.ipv6.conf.default.disable_ipv6=1
net.ipv6.conf.lo.disable_ipv6=1
```
* a host with dual-stack setup
Stoiko Ivanov [Mon, 16 Nov 2020 11:01:13 +0000 (12:01 +0100)]
pbs-integration: add CLI calls to pmgbackup
This patch adds to new categories for commands to pmgbackup:
* pmgbackup remote - for managing PBS instances' configuration, cluster-wide
* pmgbackup pbsjob - for managing backups, restores, pruning
Stoiko Ivanov [Wed, 28 Oct 2020 18:54:21 +0000 (19:54 +0100)]
Restore: optionally restore from directory
In preparation for integrating PMG with PBS decide based on the type of the
provided filename, whether or not to untar:
* if it's a directory skip untarring (PBS)
* if it's a filename untar (local backup)
Stoiko Ivanov [Wed, 28 Oct 2020 18:54:20 +0000 (19:54 +0100)]
Backup: split backup creation and creating tar
In preparation for integrating PMG with PBS split the current creation of
a PMG backup into 2 methods:
* create all files in a backup in a target directory
* create a tarball from a backup in a temporary directory
Stoiko Ivanov [Thu, 18 Jun 2020 11:33:17 +0000 (13:33 +0200)]
add logging to disclaimer action
the disclaimer action currently does not log, if a disclaimer got added or not.
given that there are a few not directly obvious cases where a disclaimer does
not get added (e.g. it depends on the mail's encoding) - logging success or
failure should help in debugging
Tested by sending mails, where adding the disclaimer works, and where it fails.
verified that the log-tracker also adds those lines to its output.
Stoiko Ivanov [Wed, 17 Jun 2020 15:04:05 +0000 (17:04 +0200)]
prefix message-id in attachment-quarantine
This patch fixes #2785.
When using the attachment quarantine - the message is:
a) stored in the quarantine unaltered
b) sent on with the attachment removed
Currently we do not change the message in any other way - in particular
we do not change the message-id header of any of the 2 mails.
When a mail is released from the attachment quarantine it is sent by PMG
with the same message-id as the mail with the attachments removed.
This is a violation of RFC 5322 (see [0]), and additionally newer versions
of Exchange do accept 2 mails with the same message-id but silently discard
the second version, thus making the attachment quarantine unusable for
Exchange users.
This patch simply prefixes the message-id with 'pmg-aquar-$$' (where $$ is
the pid of the pmg-smtp-filter process) for the mail without attachment.
By keeping the original message-id in the headers tracing the mailflow should
be facilitated.
The Message-ID is left intact on the original message in order to keep DKIM
signatures valid (they are invalidated on the modified mail by the removal
of the attachment anyways).
Stoiko Ivanov [Thu, 28 May 2020 08:04:58 +0000 (10:04 +0200)]
fix #1976: optionally sort postfix queue result
The PostfixMailQueue widget uses an Ext.data.BufferedStore, due to
the potential size of the resultset, which does only support remoteSorting [0]
By adding two optional parameters ('sortfield' and 'sortdir') we can use
them for sorting the mailq output accordingly.
The sorting is kept in PMG::API2::Postfix instead of PMG::Postfix, because
sorting (as opposed to filtering) needs to happen after the complete result
is known, and there is no gain in pushing it further down.
[0] only mentioned in the source-code - not in the referencedoc
to keep the log from growing without end. Rotate monthly and keep 12 logs,
since the logs should not be too large (e.g. a productive instance, with
~20 users using the quarantine and some configuration changes amounts to
108M over 2.5 years)
the logrotate snippet is placed in /etc/logrotate.d/pmg-api by
dh_installlogrotate(1).
Thomas Lamprecht [Fri, 24 Apr 2020 06:59:19 +0000 (08:59 +0200)]
pmgsync.service: really order before postfix@-.service
followup for commit 0c4cf3f2cfa2b40d4fb1ded7501989b884c73eae
which assumed that we can order on templated base units, which we
cannot (at least under the systemd version of buster). So depend on
the actual instance of the main postfix template.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
pmgsync.service updates the database-schema (pmgdb init) and generates and
updates configuration files from the templates and reloads the respective
services.
When first booting up after installation, it adapts the config from what's
shipped in the default debian packages for the first time.
The postfix configuration is also rendered, including settings where a
restart is necessary (listening on the internal port (26)).
While the unit already starts before postfix.service, the postfix service
files are designed to start multiple instances of postfix via instantiation
(by default postfix@-.service is the single instance (and the service file
which actually starts postfix)).
Since both pmgsync and postfix@- have no ordering relation between them, they
are started in parallel, which leads to postfix starting with the stock config
upon first boot.
Tested by running the installer in debug mode and applying this patch in the
last debug shell.
the usepolicy variable is used by the templateing system to decide whether
pmgpolicy should be asked by postfix and should also be enabled if greylisting
is only active for ipv6.
Instead of hardcoding the netmask used for comparing greylistentries to
the current ip (24 for ipv4 and 64 for ipv6) - make them configurable in
pmg.conf
This should help with some cloud providers who send the same mail with
different ips from a large network - which all get greylisted separately.
In the worst case the sending cloud drops the mail, after it got defered
too often.
adds a new configuration flag in the 'mail' configuration section to
selectively enable greylisting for IPv6 and leaves its default as false to
maintain backward compatibility.
this change also enables SPF verification of IPv6 addresses if 'spf' is set
in the 'mail' section as a side-effect
In preparation for adding support for a configurable greylist netmask [0]
and greylisting for ipv6 hosts the width of the IPNet column of the cgreylist
table needs to be extended to 49 [1].
Instead of comparing the first 3 octets of a ipv4 address we store the
complete network definition (i.e. for 192.0.2.127/24 - 192.0.2.0/24)
The last octet is not saved, but written as 0 (the information is not
needed, and not used currently). The generation of the network is done
with postgresql's functions for the inet and cidr datatypes [2,3].
The change of the column width instead of using the inet datatype prevents
errors while syncing or downgrading, although older nodes in a cluster (or
downgraded nodes) will not match new records.
When syncing from a node with old-style data the rows are inserted in the
new format.
Upon upgrade (`pmgdb init` in the postinst script) the data is changed to
the new format and matched for duplicates (in case one node in the cluster
got upgraded and it's contents were synced we should not edit the data
again). This process does cause the Cgreylist table to be scanned, which
takes time linear in the number of rows (e.g. with a test-dataset of
~ 1 million rows the upgrade is blocked for ~50 seconds on an
average testinstallation).
Changing only the column datatype does not lock the table and is almost
instantenous [4].
[0] defining from which neighbors a mail is accepted on the second attempt
[1] INET6_ADDRSTRLEN is 46 + 4 for the netmask ('/128) - \0
[2] https://www.postgresql.org/docs/11/datatype-net-types.html
[3] https://www.postgresql.org/docs/11/functions-net.html
[4] Notes section of https://www.postgresql.org/docs/11/sql-altertable.html
This behaves like the 'ArchiveFilter' to 'ContentTypeFilter', in that
it matches the filenames in archives, as well as the filenames of
attachments (via filename property in the mime header).
mirroring the changed default both in upstream [0] and debian [1].
Additionally since the upgrade to clamav 0.102.2 we had a number of reports
from users (both in forum and in our enterprise support), which noticed
that the upgrades don't work due to running into the timeout
(might be related to the use of libcurl in freshclam in 0.102.0)
absence of 'ReceiveTimeout' in freshclam.conf (5) defaults to 0 (no timeout).
[0] https://blog.clamav.net/2020/02/clamav-01022-security-patch-released.html
[1] apt changelog clamav-freshclam (for version 0.102.2+dfsg-1)
freshclam.conf.in: make ScriptedUpdates a variable
The 'ScriptedUpdate' setting in freshclam.conf (5) defines whether signature
updates should be fetched incrementally or whether the whole database should
be downloaded.
Since the upgrade of clamav to 0.102.1 led to some problems when downloading
the complete file [0], and some users who switched to incremental updates
later had problems with that, it seems the most comfortable and futureproof
solution is to have a simple way for users to selectively switch to
whichever mechanism works. AFAIR signature downloads of clamav also had
similar problems in previous versions.
Additionally the rendered boolean value was changed from 'yes' to 'true',
to be a bit more consistent with the other booleans in our template.
ucf(1) is a utility to track changes in config files which are not shipped in
the debian package (but e.g. get generated through the postinst script)
While the template overriding mechanism of PMG does not directly write those
config files - users who override a config-file currently need to manually
compare the templates shipped in '/var/lib/pmg/templates' on every upgrade.
By selectively registering the existing template overrides with ucf, users get
asked once upon the next upgrade regarding their changes, and then will
always get prompted when the shipped default template changes.
The alternative of unconditionally registering all templates with ucf, as done
by dh_ucf (1), would copy all templates to /etc/pmg/templates, which I deemed
less elegant.
The postrm script's check for the existance of 'ucf' only should be sufficient,
since the other ucf executables used ('ucfq' and 'ucfr' are all shipped with
the 'ucf' package)
Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
[ dropped some extra lines left over from v1 ] Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Stoiko Ivanov [Wed, 18 Mar 2020 10:23:46 +0000 (11:23 +0100)]
Add tls options for lmtp to main.cf template
With the addition of supporting lmtp as downstream server, we should also
set the relevant configuration options if TLS support is enabled.
(postfix does not use the smtp settings for lmtp)
Tested by sending a few mails to a downstream lmtp-server (dovecot)
and comparing the traffic with tcpdump/wireshark
Stoiko Ivanov [Wed, 18 Mar 2020 10:23:45 +0000 (11:23 +0100)]
TLSPolicy: rename domain to destination
TLS policies are applied to destinations - for the outbound case these are
domains. However when setting a policy for a next-hop destination the
property name 'domain' is misleading, and should be renamed to 'destination'
In order to maintain backward compatibility in the API, we need to keep the
'domain' property in all return defintions, and also accept it (or the new
'destination') for the create call (all other calls get the destination from
the path)
the create call warns of the deprecation if the domain property is provided
(which ends up in the journal), but still works.
We can then get rid of the 'domain' property with the release of PMG 7.0
Stoiko Ivanov [Wed, 18 Mar 2020 10:23:44 +0000 (11:23 +0100)]
fix #1948: allow setting TLS policy for transports
As described in postfix TLS Readme [0] the key of the tls_policy table need
not be a destination domain - it can also contain an entry from the transport
table.
By adding a new format, which matches the current format of 'transport-domain'
and additionally the possible values for a smtp/lmtp next-hop (see `man smtp`)
users can now also set a stricter tls policy for their configured downstream
servers (e.g. to enforce TLS, or to disable it, if the downstream server's
TLS implementation is broken).
Tested locally by sending mails to a downstream server with policy 'may' set
(STARTTLS is used), and 'none' (mail goes unecrypted) - verified with tcpdump.
If a next-hop is provided it needs to be literally the same entry as present
in the transport table (w/o the 'smtp:' or 'lmtp:inet:' prefix) - i.e.
it is significant if the entry is enclosed in brackets, or if the (defacto
optional) 'ipv6:' prefix is present in the transport entry.
Stoiko Ivanov [Wed, 18 Mar 2020 10:23:43 +0000 (11:23 +0100)]
allow for optional 'ipv6:' prefix in transports
according to the smtp(8) and transport(5) manuals literal ipv6 addresses must
be written as '[ipv6:2001:db8::ff]', in accordance with rfc2821 [0].
Postfix does work irrespective of this prefix (as it has been working until
now), but we should allow for administrators to enter the addresses with the
prefix present.
Stoiko Ivanov [Wed, 18 Mar 2020 10:23:42 +0000 (11:23 +0100)]
fix rendering of ipv(4|6) literal lmtp transports
While reviewing support for lmtp as transport one thing I forgot to
test was adding a lmtp-transport pointing to an IPv6 address.
Using the use_mx flag (which only makes sense for domain-names) to
provide the information of whether the next-hop/hostname should be
written out in square brackets or not is a bit confusing, and leads
to ambiguous results when providing ipv6 literal addresses:
> host: 2001:db8:25::25
> port: 24
> gets rendered as
> lmtp:inet:2001:db8:25::25:24
Which postfix oddly enough parses 'correctly'; postfix splits on ':'
and uses the last part as port. For ip4 literals and dns-names this
works for ip6 literals it only works if you provide a port so it
makes more sense to always write ip(4|6) literals in brackets
By introducing a explicit flag "$bracket_host" and reordering the
conditions lmtp and smtp entries get rendered correctly (see `man
smtp`).
Additionally fixes an indentation glitch in read_transport_map.
Stoiko Ivanov [Thu, 19 Mar 2020 13:02:27 +0000 (14:02 +0100)]
pmgspamreport: purge before sending reports
By cleaning the quarantine (database and files in spool directory), before
trying to send out mail we can prevent the file-system and database from
filling up because of an unrelated problem in sending out mails.
Originally discovered while analyzing why the quarantine spooldir on a
non-master node fills up, while being limted in size on the master-node:
the call to send out the reports errored out, because the node was not master.
Stoiko Ivanov [Thu, 19 Mar 2020 13:02:26 +0000 (14:02 +0100)]
pmgqm: warn if not running on master
pmgqm should exit when running on a non-master node (instead of e.g. sending
out spam report mails multiple times).
However this condition should not be an error - a warning printed should
suffice.
changing the die to warn + return in the calls for 'status' and 'send'
achieves this effect.
projects / pmg-api.git / log