to keep the log from growing without end. Rotate monthly and keep 12 logs,
since the logs should not be too large (e.g. a productive instance, with
~20 users using the quarantine and some configuration changes amounts to
108M over 2.5 years)
the logrotate snippet is placed in /etc/logrotate.d/pmg-api by
dh_installlogrotate(1).
Thomas Lamprecht [Fri, 24 Apr 2020 06:59:19 +0000 (08:59 +0200)]
pmgsync.service: really order before postfix@-.service
followup for commit 0c4cf3f2cfa2b40d4fb1ded7501989b884c73eae
which assumed that we can order on templated base units, which we
cannot (at least under the systemd version of buster). So depend on
the actual instance of the main postfix template.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
pmgsync.service updates the database-schema (pmgdb init) and generates and
updates configuration files from the templates and reloads the respective
services.
When first booting up after installation, it adapts the config from what's
shipped in the default debian packages for the first time.
The postfix configuration is also rendered, including settings where a
restart is necessary (listening on the internal port (26)).
While the unit already starts before postfix.service, the postfix service
files are designed to start multiple instances of postfix via instantiation
(by default postfix@-.service is the single instance (and the service file
which actually starts postfix)).
Since both pmgsync and postfix@- have no ordering relation between them, they
are started in parallel, which leads to postfix starting with the stock config
upon first boot.
Tested by running the installer in debug mode and applying this patch in the
last debug shell.
the usepolicy variable is used by the templateing system to decide whether
pmgpolicy should be asked by postfix and should also be enabled if greylisting
is only active for ipv6.
Instead of hardcoding the netmask used for comparing greylistentries to
the current ip (24 for ipv4 and 64 for ipv6) - make them configurable in
pmg.conf
This should help with some cloud providers who send the same mail with
different ips from a large network - which all get greylisted separately.
In the worst case the sending cloud drops the mail, after it got defered
too often.
adds a new configuration flag in the 'mail' configuration section to
selectively enable greylisting for IPv6 and leaves its default as false to
maintain backward compatibility.
this change also enables SPF verification of IPv6 addresses if 'spf' is set
in the 'mail' section as a side-effect
In preparation for adding support for a configurable greylist netmask [0]
and greylisting for ipv6 hosts the width of the IPNet column of the cgreylist
table needs to be extended to 49 [1].
Instead of comparing the first 3 octets of a ipv4 address we store the
complete network definition (i.e. for 192.0.2.127/24 - 192.0.2.0/24)
The last octet is not saved, but written as 0 (the information is not
needed, and not used currently). The generation of the network is done
with postgresql's functions for the inet and cidr datatypes [2,3].
The change of the column width instead of using the inet datatype prevents
errors while syncing or downgrading, although older nodes in a cluster (or
downgraded nodes) will not match new records.
When syncing from a node with old-style data the rows are inserted in the
new format.
Upon upgrade (`pmgdb init` in the postinst script) the data is changed to
the new format and matched for duplicates (in case one node in the cluster
got upgraded and it's contents were synced we should not edit the data
again). This process does cause the Cgreylist table to be scanned, which
takes time linear in the number of rows (e.g. with a test-dataset of
~ 1 million rows the upgrade is blocked for ~50 seconds on an
average testinstallation).
Changing only the column datatype does not lock the table and is almost
instantenous [4].
[0] defining from which neighbors a mail is accepted on the second attempt
[1] INET6_ADDRSTRLEN is 46 + 4 for the netmask ('/128) - \0
[2] https://www.postgresql.org/docs/11/datatype-net-types.html
[3] https://www.postgresql.org/docs/11/functions-net.html
[4] Notes section of https://www.postgresql.org/docs/11/sql-altertable.html
This behaves like the 'ArchiveFilter' to 'ContentTypeFilter', in that
it matches the filenames in archives, as well as the filenames of
attachments (via filename property in the mime header).
mirroring the changed default both in upstream [0] and debian [1].
Additionally since the upgrade to clamav 0.102.2 we had a number of reports
from users (both in forum and in our enterprise support), which noticed
that the upgrades don't work due to running into the timeout
(might be related to the use of libcurl in freshclam in 0.102.0)
absence of 'ReceiveTimeout' in freshclam.conf (5) defaults to 0 (no timeout).
[0] https://blog.clamav.net/2020/02/clamav-01022-security-patch-released.html
[1] apt changelog clamav-freshclam (for version 0.102.2+dfsg-1)
freshclam.conf.in: make ScriptedUpdates a variable
The 'ScriptedUpdate' setting in freshclam.conf (5) defines whether signature
updates should be fetched incrementally or whether the whole database should
be downloaded.
Since the upgrade of clamav to 0.102.1 led to some problems when downloading
the complete file [0], and some users who switched to incremental updates
later had problems with that, it seems the most comfortable and futureproof
solution is to have a simple way for users to selectively switch to
whichever mechanism works. AFAIR signature downloads of clamav also had
similar problems in previous versions.
Additionally the rendered boolean value was changed from 'yes' to 'true',
to be a bit more consistent with the other booleans in our template.
ucf(1) is a utility to track changes in config files which are not shipped in
the debian package (but e.g. get generated through the postinst script)
While the template overriding mechanism of PMG does not directly write those
config files - users who override a config-file currently need to manually
compare the templates shipped in '/var/lib/pmg/templates' on every upgrade.
By selectively registering the existing template overrides with ucf, users get
asked once upon the next upgrade regarding their changes, and then will
always get prompted when the shipped default template changes.
The alternative of unconditionally registering all templates with ucf, as done
by dh_ucf (1), would copy all templates to /etc/pmg/templates, which I deemed
less elegant.
The postrm script's check for the existance of 'ucf' only should be sufficient,
since the other ucf executables used ('ucfq' and 'ucfr' are all shipped with
the 'ucf' package)
Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
[ dropped some extra lines left over from v1 ] Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Stoiko Ivanov [Wed, 18 Mar 2020 10:23:46 +0000 (11:23 +0100)]
Add tls options for lmtp to main.cf template
With the addition of supporting lmtp as downstream server, we should also
set the relevant configuration options if TLS support is enabled.
(postfix does not use the smtp settings for lmtp)
Tested by sending a few mails to a downstream lmtp-server (dovecot)
and comparing the traffic with tcpdump/wireshark
Stoiko Ivanov [Wed, 18 Mar 2020 10:23:45 +0000 (11:23 +0100)]
TLSPolicy: rename domain to destination
TLS policies are applied to destinations - for the outbound case these are
domains. However when setting a policy for a next-hop destination the
property name 'domain' is misleading, and should be renamed to 'destination'
In order to maintain backward compatibility in the API, we need to keep the
'domain' property in all return defintions, and also accept it (or the new
'destination') for the create call (all other calls get the destination from
the path)
the create call warns of the deprecation if the domain property is provided
(which ends up in the journal), but still works.
We can then get rid of the 'domain' property with the release of PMG 7.0
Stoiko Ivanov [Wed, 18 Mar 2020 10:23:44 +0000 (11:23 +0100)]
fix #1948: allow setting TLS policy for transports
As described in postfix TLS Readme [0] the key of the tls_policy table need
not be a destination domain - it can also contain an entry from the transport
table.
By adding a new format, which matches the current format of 'transport-domain'
and additionally the possible values for a smtp/lmtp next-hop (see `man smtp`)
users can now also set a stricter tls policy for their configured downstream
servers (e.g. to enforce TLS, or to disable it, if the downstream server's
TLS implementation is broken).
Tested locally by sending mails to a downstream server with policy 'may' set
(STARTTLS is used), and 'none' (mail goes unecrypted) - verified with tcpdump.
If a next-hop is provided it needs to be literally the same entry as present
in the transport table (w/o the 'smtp:' or 'lmtp:inet:' prefix) - i.e.
it is significant if the entry is enclosed in brackets, or if the (defacto
optional) 'ipv6:' prefix is present in the transport entry.
Stoiko Ivanov [Wed, 18 Mar 2020 10:23:43 +0000 (11:23 +0100)]
allow for optional 'ipv6:' prefix in transports
according to the smtp(8) and transport(5) manuals literal ipv6 addresses must
be written as '[ipv6:2001:db8::ff]', in accordance with rfc2821 [0].
Postfix does work irrespective of this prefix (as it has been working until
now), but we should allow for administrators to enter the addresses with the
prefix present.
Stoiko Ivanov [Wed, 18 Mar 2020 10:23:42 +0000 (11:23 +0100)]
fix rendering of ipv(4|6) literal lmtp transports
While reviewing support for lmtp as transport one thing I forgot to
test was adding a lmtp-transport pointing to an IPv6 address.
Using the use_mx flag (which only makes sense for domain-names) to
provide the information of whether the next-hop/hostname should be
written out in square brackets or not is a bit confusing, and leads
to ambiguous results when providing ipv6 literal addresses:
> host: 2001:db8:25::25
> port: 24
> gets rendered as
> lmtp:inet:2001:db8:25::25:24
Which postfix oddly enough parses 'correctly'; postfix splits on ':'
and uses the last part as port. For ip4 literals and dns-names this
works for ip6 literals it only works if you provide a port so it
makes more sense to always write ip(4|6) literals in brackets
By introducing a explicit flag "$bracket_host" and reordering the
conditions lmtp and smtp entries get rendered correctly (see `man
smtp`).
Additionally fixes an indentation glitch in read_transport_map.
Stoiko Ivanov [Thu, 19 Mar 2020 13:02:27 +0000 (14:02 +0100)]
pmgspamreport: purge before sending reports
By cleaning the quarantine (database and files in spool directory), before
trying to send out mail we can prevent the file-system and database from
filling up because of an unrelated problem in sending out mails.
Originally discovered while analyzing why the quarantine spooldir on a
non-master node fills up, while being limted in size on the master-node:
the call to send out the reports errored out, because the node was not master.
Stoiko Ivanov [Thu, 19 Mar 2020 13:02:26 +0000 (14:02 +0100)]
pmgqm: warn if not running on master
pmgqm should exit when running on a non-master node (instead of e.g. sending
out spam report mails multiple times).
However this condition should not be an error - a warning printed should
suffice.
changing the die to warn + return in the calls for 'status' and 'send'
achieves this effect.
Dominik Csapak [Mon, 9 Mar 2020 11:18:16 +0000 (12:18 +0100)]
api/quarantine: add safer endpoint for user white/blacklist address deletion
having the entry as part of the url causes many problems since it can
contain special characters like '/.,' etc., and that can break API
call path-to-method resolution.
Passing it as parameter makes it easier for callers (frontends) and
safer for backend to use
Note that the new api calls overwrites the parameter pattern with '',
so no formatting limits for the entries
Stoiko Ivanov [Mon, 24 Feb 2020 18:16:48 +0000 (19:16 +0100)]
fix #2525: encode notifications in UTF-8
the Notify action is one of the places where we already encode the data as
UTF-8, before writing it to the DB (and decoding it when reading).
as laid out in rt.cpan.org [0] Mime::Body does expect encoded bytes, and not
perl characters.
Tested by creating a notification with the body supplied in #2591 (which is a
duplicate of #2525) and additionally with cyrillic characters in the subject.
A minimal test case is a body consisting of a Euro sign (since its Unicode
codepoint is larger than one byte).
Should the table contain invalid UTF-8 sequences (AFAIU only possible by
direct DB-manipulation) the byte gets replaced with \x{fffd} (Unicode
replacement character).
Dominik Csapak [Tue, 3 Mar 2020 08:33:35 +0000 (09:33 +0100)]
fix #2622: include all spam levels in total spam statistic
by using 'LIMIT 10' for the spamlevels, we only got the first
10 spamlevels back from the database. This is only ok if there are
only <= 10 different spamlevels in the database, but not if there are
more, as then the bucket for spamlevel >= 10 missed entries.
The call site of this uses the combined spam count of this query
result for calculating the 'rest' (meaning the mails with spam level
0), but this is obviously wrong if not all spamlevels are counted so
simply return all available levels.
Mira Limbeck [Tue, 18 Feb 2020 15:36:40 +0000 (16:36 +0100)]
add pmg-smtp-filter ID to reply
For the pmg-log-tracker to match the pmg-smtp-filter on a reject, we
need some kind of information. With the addition of the pmg-smtp-filter
ID we can match it the same way we do for an accept.
Signed-off-by: Mira Limbeck <m.limbeck@proxmox.com> Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Thomas Lamprecht [Fri, 21 Feb 2020 13:08:41 +0000 (14:08 +0100)]
replace lookup_node_ip with get_ip_from_hostname from pve-common
besides some irrelevant implementation details 'lookup_node_ip' is
identical to pve-common's get_ip_from_hostname, as they both rely on
'PVE::Tools::getaddrinfo_all' to get the addresses. So just reuse
the one from common instead of shipping a copy here.
The pve-common's one was recently improved by checking not only the
first IP it gets from getaddrinfo_all, but all and only complain if
none of those is a "real WAN (non-local)" IP. This will help
container installations of PMG with DHCP as their network
configuration option, as those often also have a hosts entry from
hostname to loopback addresses. Also, static setups often have both,
WAN and loopback addresses in /etc/hosts - as getaddrinfo_all gives
use them all do not just check the first.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Since the settings were overridden in the smtpd instances the error only
shows for locally generated mail (e.g. from cronjobs):
```
warning: connect to transport private/scan: Connection refused
```
Instead of globally setting 'smtpd_proxy_filter' (and associated options) in
'main.cf', as is done for 'content_filter' (for afterqueue filtering), this
patch removes the global 'content_filter' setting. This is done since
'smtp_proxy_filter' only applies to smtp sessions [0] and all incoming smtpd
processes get the setting in 'master.cf.in'.
Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
[0] see http://www.postfix.org/SMTPD_PROXY_README.html - the proxy speaks
smtp, and it's answers are sent to the sending server.
Dominik Csapak [Fri, 17 Jan 2020 09:52:47 +0000 (10:52 +0100)]
fix disclaimer encoding for html entities
we also want to encode the disclaimer for text/html parts and not
only for text/plain. while doing this, combine those two cases,
as they differ only by the variable to be encoded
this also fixes a missing charset, which we would ignore, but
should actually be treated as US-ASCII[0] so that an ascii disclaimer
still gets appended
the only (non-rfc-compliant) use case this breaks is if:
* the part has no charet defined (unusual)
* the clients of both the sender and receiver treat a missing charset
as 'iso-8859-1' (non-rfc-compliant)
* the disclaimer contains characters from 'iso-8859-1' and was added
to the pmg before encoding this to utf-8 (unlikely)
Stoiko Ivanov [Tue, 14 Jan 2020 18:31:35 +0000 (19:31 +0100)]
dkim: add selector list api call
The fix for #2504 left the GUI with an unsatisfactory UX:
Users can change the selector to any newly created or existing one, but
don't know which ones exist (without looking on the commandline)
By adding a method under '/config/dkim/selectors' which lists all existing
files matching the pattern '/etc/pmg/dkim/.*\.private' the GUI can display
all currently existing selectors.
Stoiko Ivanov [Tue, 14 Jan 2020 18:31:34 +0000 (19:31 +0100)]
fix #2504: do not overwrite existing selector key
This patch changes the behavior of DKIM selector creation. Instead of blindly
overwriting an already present file, add a force parameter to overwrite it (and
behave like the current code).
Overwriting an existing selector can potentially be quite destructive (e.g.
a setup where the admin has already posted the DNS-record for one selector to
many domains, then wants to quickly experiment with a larger keysize, and tries
to go back to the existing behavior).
The new behavior without force set to true, when a private key for the selector
already exists is to die if the file is either not a private RSA key, or has
the wrong size, or else just set the selector in pmg.conf
Julian Zehnter [Sun, 5 Jan 2020 14:53:07 +0000 (15:53 +0100)]
Feature #2438 add support for lmtp delivery to downstream servers
new feature lmtp support for simplifying setups
with lmtp capable downstream servers (e.g. dovecot)
Postfix support lmtp out of the box and can now deliver
mails directly to internal mailbox servers without
one more smtp connection
extending the api code for new lmtp option:
Config.pm:
Adding new variable "relayprotocol"
Extending the read_transport_map & write_transport_map
for parsing the /etc/pmg/transport
Transport.pm:
Add new protcol varialbe for smtp/lmtp setting
Generalizing some "SMTP" keywords
Templates:
Adapting the main.cf templates for adding the lmtp keyword
Dominik Csapak [Thu, 28 Nov 2019 10:22:43 +0000 (11:22 +0100)]
pmgproxy: implement cache workaround
we include the version of the packages in the temlate, so that
we can tell the browser that the file has changed and thus preventing
to load an old gui js file
the index template has to add the version to the get parameter
Stoiko Ivanov [Fri, 22 Nov 2019 09:40:57 +0000 (10:40 +0100)]
Don't add DKIM signature without domain
When the DKIMSign module fails to determine the domain for signing
(the one added to the header and used for retrieving the publickey record)
the code logs that no signing will take place, but only does not set the
domain - resulting in a generated and added signature with domain 'example.com'
Fixed by returning the success-status from signing_domain and only signing if
it was successful.
Stoiko Ivanov [Mon, 18 Nov 2019 09:03:17 +0000 (10:03 +0100)]
create pmg-scores.cf unconditionally
with the recent addtion of adjustable SA-rule scores, we introduced
'/etc/mail/spamassassin/pmg-scores.cf' as a new file and included it in the
cluster synchronization.
If the file does not exist the `rsync` command complains leading to misleading
errors in the journal.
Unconditionally creating the file (like we do for the already existing
'custom.cf') removes the warnings.
Reported-by: Martin Maurer <martin@proxmox.com> Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
Dominik Csapak [Thu, 14 Nov 2019 11:18:53 +0000 (12:18 +0100)]
add SACustom Package and API Calls for custom SpamAssassin scores
this uses our INotify interface to parse and write a custom sa config
in /etc/mail/spamassassin/pmg-scores.cf with a shadow file in
/var/cache/pmg-scores.cf (to track the diff)
add also api calls to create a new/delete/edit/revert/apply those custom
rules
Dominik Csapak [Thu, 14 Nov 2019 11:18:52 +0000 (12:18 +0100)]
implement force_restart for cluster sync
the 'force_restart' hash was unused, but it is actually necessary to
restart the pmg-smtp-filter if the custom.cf has changed
use it for saving the daemon name that needs to be restarted, this
way we can reuse it in the future to force restart other daemons
or for different reasons if we need to
Stoiko Ivanov [Thu, 14 Nov 2019 16:35:07 +0000 (17:35 +0100)]
add support for before queue filtering
Support for rejecting mails during the SMTP-dialog with 550 (permanent failure)
instead of seemingly accepting the mail (250 OK) and dropping it if it is
rejected by the rule-system is also known as 'before queue filtering' [0].
This patch adds minimal support for before queue filtering to pmg-smtp-filter.
Since pmg-smtp-filter is currently called via LMTP (and the 'scan' service in
'master.cf') we can adapt the already existing branch dealing with SMTP to
send a 550 selectively.
We can reply with 554 (permfail) if all recepients are blocked.
In the case that some accept the mail, we reply with 250 OK.
Depending on the setting of 'ndr_on_block' we generate ndrs
for all blocking recepients. (This is also the behavior that postfix
has when not enabling receiver verification and the downstream server rejects
recepients).
Configuration of before-queue filtering is done via the
'before_queue_filtering' boolean in the 'mail' section of 'pmg.conf':
the before_queue_filtering flag is used when rendering '/etc/postfix/master.cf'
to adapt the needed config-options for both, inbound and outbound, smtpd
servers. The settings were adapted from [0].
Stoiko Ivanov [Thu, 14 Nov 2019 16:35:06 +0000 (17:35 +0100)]
add support for sending NDRs on Block
This patch adds a flag to the mail-section of pmg.conf - 'ndr_on_block',
defaulting to false.
If enabled pmg-smtp-filter replies with 554 (permanent fail) for the recipient
who blocked the mail.
Since pmg-smtp-filter is currently called with LMTP it can respond with a
separate code for each recipient. Postfix records the answers and generates
and sends the NDR to the sender.
Stoiko Ivanov [Tue, 12 Nov 2019 14:16:13 +0000 (15:16 +0100)]
add generate_ndr to PMG::SMTP
In order to selectively accept and reject mails when multiple recipients
are present for a mail, without silently dropping mails to certain recipients
the sender needs to be informed about those recipients which did not receive
the mail (via a non-delivery report - NDR).
The format of delivery status notifications (of which an ndr is a subset) is
specified in RFC 6522 [0]. The format for the 'message/delivery-status'
MIME-Type is specified in RFC 6533 [1]
The message text was adapted from Postfix' default bounce-messages, however
we do not attach the original mail to the report. This is acceptable by [0]
and makes sense (not sending mail, which is likely spam or a virus, back to the
(potentially faked) sender).
Dominik Csapak [Wed, 23 Oct 2019 07:36:34 +0000 (09:36 +0200)]
Quarantine API: extend download call to download whole mails
this makes the attachmentid parameter optional and if it is not
given, open the whole mail for download
The permission check, that a quser only is able to get their own
mails/attachments, happens in get_and_check_mail, thus we can add the
quser to the "sufficient permissions" list without opening up the
real permissions the users has.
projects / pmg-api.git / log