Stoiko Ivanov [Thu, 29 Oct 2020 17:49:16 +0000 (18:49 +0100)]
reinject_email: fix connecting for ipv6-only hosts
When configuring PMG only with ipv6 addresses, reinject_email after processing
fails to connect to the postfix/smtpd instance (with EINVAL).
Setting the host to '::FFFF:127.0.0.1' fixes the issue.
Tested with:
* an ipv6only host (no ipv4 configured)
* a host with ipv6 disabled via sysctl:
```
net.ipv6.conf.all.disable_ipv6=1
net.ipv6.conf.default.disable_ipv6=1
net.ipv6.conf.lo.disable_ipv6=1
```
* a host with dual-stack setup
Stoiko Ivanov [Mon, 16 Nov 2020 11:01:13 +0000 (12:01 +0100)]
pbs-integration: add CLI calls to pmgbackup
This patch adds to new categories for commands to pmgbackup:
* pmgbackup remote - for managing PBS instances' configuration, cluster-wide
* pmgbackup pbsjob - for managing backups, restores, pruning
Stoiko Ivanov [Wed, 28 Oct 2020 18:54:21 +0000 (19:54 +0100)]
Restore: optionally restore from directory
In preparation for integrating PMG with PBS decide based on the type of the
provided filename, whether or not to untar:
* if it's a directory skip untarring (PBS)
* if it's a filename untar (local backup)
Stoiko Ivanov [Wed, 28 Oct 2020 18:54:20 +0000 (19:54 +0100)]
Backup: split backup creation and creating tar
In preparation for integrating PMG with PBS split the current creation of
a PMG backup into 2 methods:
* create all files in a backup in a target directory
* create a tarball from a backup in a temporary directory
Stoiko Ivanov [Thu, 18 Jun 2020 11:33:17 +0000 (13:33 +0200)]
add logging to disclaimer action
the disclaimer action currently does not log, if a disclaimer got added or not.
given that there are a few not directly obvious cases where a disclaimer does
not get added (e.g. it depends on the mail's encoding) - logging success or
failure should help in debugging
Tested by sending mails, where adding the disclaimer works, and where it fails.
verified that the log-tracker also adds those lines to its output.
Stoiko Ivanov [Wed, 17 Jun 2020 15:04:05 +0000 (17:04 +0200)]
prefix message-id in attachment-quarantine
This patch fixes #2785.
When using the attachment quarantine - the message is:
a) stored in the quarantine unaltered
b) sent on with the attachment removed
Currently we do not change the message in any other way - in particular
we do not change the message-id header of any of the 2 mails.
When a mail is released from the attachment quarantine it is sent by PMG
with the same message-id as the mail with the attachments removed.
This is a violation of RFC 5322 (see [0]), and additionally newer versions
of Exchange do accept 2 mails with the same message-id but silently discard
the second version, thus making the attachment quarantine unusable for
Exchange users.
This patch simply prefixes the message-id with 'pmg-aquar-$$' (where $$ is
the pid of the pmg-smtp-filter process) for the mail without attachment.
By keeping the original message-id in the headers tracing the mailflow should
be facilitated.
The Message-ID is left intact on the original message in order to keep DKIM
signatures valid (they are invalidated on the modified mail by the removal
of the attachment anyways).
Stoiko Ivanov [Thu, 28 May 2020 08:04:58 +0000 (10:04 +0200)]
fix #1976: optionally sort postfix queue result
The PostfixMailQueue widget uses an Ext.data.BufferedStore, due to
the potential size of the resultset, which does only support remoteSorting [0]
By adding two optional parameters ('sortfield' and 'sortdir') we can use
them for sorting the mailq output accordingly.
The sorting is kept in PMG::API2::Postfix instead of PMG::Postfix, because
sorting (as opposed to filtering) needs to happen after the complete result
is known, and there is no gain in pushing it further down.
[0] only mentioned in the source-code - not in the referencedoc
to keep the log from growing without end. Rotate monthly and keep 12 logs,
since the logs should not be too large (e.g. a productive instance, with
~20 users using the quarantine and some configuration changes amounts to
108M over 2.5 years)
the logrotate snippet is placed in /etc/logrotate.d/pmg-api by
dh_installlogrotate(1).
Thomas Lamprecht [Fri, 24 Apr 2020 06:59:19 +0000 (08:59 +0200)]
pmgsync.service: really order before postfix@-.service
followup for commit 0c4cf3f2cfa2b40d4fb1ded7501989b884c73eae
which assumed that we can order on templated base units, which we
cannot (at least under the systemd version of buster). So depend on
the actual instance of the main postfix template.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
pmgsync.service updates the database-schema (pmgdb init) and generates and
updates configuration files from the templates and reloads the respective
services.
When first booting up after installation, it adapts the config from what's
shipped in the default debian packages for the first time.
The postfix configuration is also rendered, including settings where a
restart is necessary (listening on the internal port (26)).
While the unit already starts before postfix.service, the postfix service
files are designed to start multiple instances of postfix via instantiation
(by default postfix@-.service is the single instance (and the service file
which actually starts postfix)).
Since both pmgsync and postfix@- have no ordering relation between them, they
are started in parallel, which leads to postfix starting with the stock config
upon first boot.
Tested by running the installer in debug mode and applying this patch in the
last debug shell.
the usepolicy variable is used by the templateing system to decide whether
pmgpolicy should be asked by postfix and should also be enabled if greylisting
is only active for ipv6.
Instead of hardcoding the netmask used for comparing greylistentries to
the current ip (24 for ipv4 and 64 for ipv6) - make them configurable in
pmg.conf
This should help with some cloud providers who send the same mail with
different ips from a large network - which all get greylisted separately.
In the worst case the sending cloud drops the mail, after it got defered
too often.
adds a new configuration flag in the 'mail' configuration section to
selectively enable greylisting for IPv6 and leaves its default as false to
maintain backward compatibility.
this change also enables SPF verification of IPv6 addresses if 'spf' is set
in the 'mail' section as a side-effect
In preparation for adding support for a configurable greylist netmask [0]
and greylisting for ipv6 hosts the width of the IPNet column of the cgreylist
table needs to be extended to 49 [1].
Instead of comparing the first 3 octets of a ipv4 address we store the
complete network definition (i.e. for 192.0.2.127/24 - 192.0.2.0/24)
The last octet is not saved, but written as 0 (the information is not
needed, and not used currently). The generation of the network is done
with postgresql's functions for the inet and cidr datatypes [2,3].
The change of the column width instead of using the inet datatype prevents
errors while syncing or downgrading, although older nodes in a cluster (or
downgraded nodes) will not match new records.
When syncing from a node with old-style data the rows are inserted in the
new format.
Upon upgrade (`pmgdb init` in the postinst script) the data is changed to
the new format and matched for duplicates (in case one node in the cluster
got upgraded and it's contents were synced we should not edit the data
again). This process does cause the Cgreylist table to be scanned, which
takes time linear in the number of rows (e.g. with a test-dataset of
~ 1 million rows the upgrade is blocked for ~50 seconds on an
average testinstallation).
Changing only the column datatype does not lock the table and is almost
instantenous [4].
[0] defining from which neighbors a mail is accepted on the second attempt
[1] INET6_ADDRSTRLEN is 46 + 4 for the netmask ('/128) - \0
[2] https://www.postgresql.org/docs/11/datatype-net-types.html
[3] https://www.postgresql.org/docs/11/functions-net.html
[4] Notes section of https://www.postgresql.org/docs/11/sql-altertable.html
This behaves like the 'ArchiveFilter' to 'ContentTypeFilter', in that
it matches the filenames in archives, as well as the filenames of
attachments (via filename property in the mime header).
mirroring the changed default both in upstream [0] and debian [1].
Additionally since the upgrade to clamav 0.102.2 we had a number of reports
from users (both in forum and in our enterprise support), which noticed
that the upgrades don't work due to running into the timeout
(might be related to the use of libcurl in freshclam in 0.102.0)
absence of 'ReceiveTimeout' in freshclam.conf (5) defaults to 0 (no timeout).
[0] https://blog.clamav.net/2020/02/clamav-01022-security-patch-released.html
[1] apt changelog clamav-freshclam (for version 0.102.2+dfsg-1)
freshclam.conf.in: make ScriptedUpdates a variable
The 'ScriptedUpdate' setting in freshclam.conf (5) defines whether signature
updates should be fetched incrementally or whether the whole database should
be downloaded.
Since the upgrade of clamav to 0.102.1 led to some problems when downloading
the complete file [0], and some users who switched to incremental updates
later had problems with that, it seems the most comfortable and futureproof
solution is to have a simple way for users to selectively switch to
whichever mechanism works. AFAIR signature downloads of clamav also had
similar problems in previous versions.
Additionally the rendered boolean value was changed from 'yes' to 'true',
to be a bit more consistent with the other booleans in our template.
ucf(1) is a utility to track changes in config files which are not shipped in
the debian package (but e.g. get generated through the postinst script)
While the template overriding mechanism of PMG does not directly write those
config files - users who override a config-file currently need to manually
compare the templates shipped in '/var/lib/pmg/templates' on every upgrade.
By selectively registering the existing template overrides with ucf, users get
asked once upon the next upgrade regarding their changes, and then will
always get prompted when the shipped default template changes.
The alternative of unconditionally registering all templates with ucf, as done
by dh_ucf (1), would copy all templates to /etc/pmg/templates, which I deemed
less elegant.
The postrm script's check for the existance of 'ucf' only should be sufficient,
since the other ucf executables used ('ucfq' and 'ucfr' are all shipped with
the 'ucf' package)
Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
[ dropped some extra lines left over from v1 ] Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Stoiko Ivanov [Wed, 18 Mar 2020 10:23:46 +0000 (11:23 +0100)]
Add tls options for lmtp to main.cf template
With the addition of supporting lmtp as downstream server, we should also
set the relevant configuration options if TLS support is enabled.
(postfix does not use the smtp settings for lmtp)
Tested by sending a few mails to a downstream lmtp-server (dovecot)
and comparing the traffic with tcpdump/wireshark
Stoiko Ivanov [Wed, 18 Mar 2020 10:23:45 +0000 (11:23 +0100)]
TLSPolicy: rename domain to destination
TLS policies are applied to destinations - for the outbound case these are
domains. However when setting a policy for a next-hop destination the
property name 'domain' is misleading, and should be renamed to 'destination'
In order to maintain backward compatibility in the API, we need to keep the
'domain' property in all return defintions, and also accept it (or the new
'destination') for the create call (all other calls get the destination from
the path)
the create call warns of the deprecation if the domain property is provided
(which ends up in the journal), but still works.
We can then get rid of the 'domain' property with the release of PMG 7.0
Stoiko Ivanov [Wed, 18 Mar 2020 10:23:44 +0000 (11:23 +0100)]
fix #1948: allow setting TLS policy for transports
As described in postfix TLS Readme [0] the key of the tls_policy table need
not be a destination domain - it can also contain an entry from the transport
table.
By adding a new format, which matches the current format of 'transport-domain'
and additionally the possible values for a smtp/lmtp next-hop (see `man smtp`)
users can now also set a stricter tls policy for their configured downstream
servers (e.g. to enforce TLS, or to disable it, if the downstream server's
TLS implementation is broken).
Tested locally by sending mails to a downstream server with policy 'may' set
(STARTTLS is used), and 'none' (mail goes unecrypted) - verified with tcpdump.
If a next-hop is provided it needs to be literally the same entry as present
in the transport table (w/o the 'smtp:' or 'lmtp:inet:' prefix) - i.e.
it is significant if the entry is enclosed in brackets, or if the (defacto
optional) 'ipv6:' prefix is present in the transport entry.
Stoiko Ivanov [Wed, 18 Mar 2020 10:23:43 +0000 (11:23 +0100)]
allow for optional 'ipv6:' prefix in transports
according to the smtp(8) and transport(5) manuals literal ipv6 addresses must
be written as '[ipv6:2001:db8::ff]', in accordance with rfc2821 [0].
Postfix does work irrespective of this prefix (as it has been working until
now), but we should allow for administrators to enter the addresses with the
prefix present.
Stoiko Ivanov [Wed, 18 Mar 2020 10:23:42 +0000 (11:23 +0100)]
fix rendering of ipv(4|6) literal lmtp transports
While reviewing support for lmtp as transport one thing I forgot to
test was adding a lmtp-transport pointing to an IPv6 address.
Using the use_mx flag (which only makes sense for domain-names) to
provide the information of whether the next-hop/hostname should be
written out in square brackets or not is a bit confusing, and leads
to ambiguous results when providing ipv6 literal addresses:
> host: 2001:db8:25::25
> port: 24
> gets rendered as
> lmtp:inet:2001:db8:25::25:24
Which postfix oddly enough parses 'correctly'; postfix splits on ':'
and uses the last part as port. For ip4 literals and dns-names this
works for ip6 literals it only works if you provide a port so it
makes more sense to always write ip(4|6) literals in brackets
By introducing a explicit flag "$bracket_host" and reordering the
conditions lmtp and smtp entries get rendered correctly (see `man
smtp`).
Additionally fixes an indentation glitch in read_transport_map.
Stoiko Ivanov [Thu, 19 Mar 2020 13:02:27 +0000 (14:02 +0100)]
pmgspamreport: purge before sending reports
By cleaning the quarantine (database and files in spool directory), before
trying to send out mail we can prevent the file-system and database from
filling up because of an unrelated problem in sending out mails.
Originally discovered while analyzing why the quarantine spooldir on a
non-master node fills up, while being limted in size on the master-node:
the call to send out the reports errored out, because the node was not master.
Stoiko Ivanov [Thu, 19 Mar 2020 13:02:26 +0000 (14:02 +0100)]
pmgqm: warn if not running on master
pmgqm should exit when running on a non-master node (instead of e.g. sending
out spam report mails multiple times).
However this condition should not be an error - a warning printed should
suffice.
changing the die to warn + return in the calls for 'status' and 'send'
achieves this effect.
Dominik Csapak [Mon, 9 Mar 2020 11:18:16 +0000 (12:18 +0100)]
api/quarantine: add safer endpoint for user white/blacklist address deletion
having the entry as part of the url causes many problems since it can
contain special characters like '/.,' etc., and that can break API
call path-to-method resolution.
Passing it as parameter makes it easier for callers (frontends) and
safer for backend to use
Note that the new api calls overwrites the parameter pattern with '',
so no formatting limits for the entries
Stoiko Ivanov [Mon, 24 Feb 2020 18:16:48 +0000 (19:16 +0100)]
fix #2525: encode notifications in UTF-8
the Notify action is one of the places where we already encode the data as
UTF-8, before writing it to the DB (and decoding it when reading).
as laid out in rt.cpan.org [0] Mime::Body does expect encoded bytes, and not
perl characters.
Tested by creating a notification with the body supplied in #2591 (which is a
duplicate of #2525) and additionally with cyrillic characters in the subject.
A minimal test case is a body consisting of a Euro sign (since its Unicode
codepoint is larger than one byte).
Should the table contain invalid UTF-8 sequences (AFAIU only possible by
direct DB-manipulation) the byte gets replaced with \x{fffd} (Unicode
replacement character).
Dominik Csapak [Tue, 3 Mar 2020 08:33:35 +0000 (09:33 +0100)]
fix #2622: include all spam levels in total spam statistic
by using 'LIMIT 10' for the spamlevels, we only got the first
10 spamlevels back from the database. This is only ok if there are
only <= 10 different spamlevels in the database, but not if there are
more, as then the bucket for spamlevel >= 10 missed entries.
The call site of this uses the combined spam count of this query
result for calculating the 'rest' (meaning the mails with spam level
0), but this is obviously wrong if not all spamlevels are counted so
simply return all available levels.
Mira Limbeck [Tue, 18 Feb 2020 15:36:40 +0000 (16:36 +0100)]
add pmg-smtp-filter ID to reply
For the pmg-log-tracker to match the pmg-smtp-filter on a reject, we
need some kind of information. With the addition of the pmg-smtp-filter
ID we can match it the same way we do for an accept.
Signed-off-by: Mira Limbeck <m.limbeck@proxmox.com> Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Thomas Lamprecht [Fri, 21 Feb 2020 13:08:41 +0000 (14:08 +0100)]
replace lookup_node_ip with get_ip_from_hostname from pve-common
besides some irrelevant implementation details 'lookup_node_ip' is
identical to pve-common's get_ip_from_hostname, as they both rely on
'PVE::Tools::getaddrinfo_all' to get the addresses. So just reuse
the one from common instead of shipping a copy here.
The pve-common's one was recently improved by checking not only the
first IP it gets from getaddrinfo_all, but all and only complain if
none of those is a "real WAN (non-local)" IP. This will help
container installations of PMG with DHCP as their network
configuration option, as those often also have a hosts entry from
hostname to loopback addresses. Also, static setups often have both,
WAN and loopback addresses in /etc/hosts - as getaddrinfo_all gives
use them all do not just check the first.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
projects / pmg-api.git / log