Dominik Csapak [Mon, 9 Mar 2020 11:18:16 +0000 (12:18 +0100)]
api/quarantine: add safer endpoint for user white/blacklist address deletion
having the entry as part of the url causes many problems since it can
contain special characters like '/.,' etc., and that can break API
call path-to-method resolution.
Passing it as parameter makes it easier for callers (frontends) and
safer for backend to use
Note that the new api calls overwrites the parameter pattern with '',
so no formatting limits for the entries
Stoiko Ivanov [Mon, 24 Feb 2020 18:16:48 +0000 (19:16 +0100)]
fix #2525: encode notifications in UTF-8
the Notify action is one of the places where we already encode the data as
UTF-8, before writing it to the DB (and decoding it when reading).
as laid out in rt.cpan.org [0] Mime::Body does expect encoded bytes, and not
perl characters.
Tested by creating a notification with the body supplied in #2591 (which is a
duplicate of #2525) and additionally with cyrillic characters in the subject.
A minimal test case is a body consisting of a Euro sign (since its Unicode
codepoint is larger than one byte).
Should the table contain invalid UTF-8 sequences (AFAIU only possible by
direct DB-manipulation) the byte gets replaced with \x{fffd} (Unicode
replacement character).
Dominik Csapak [Tue, 3 Mar 2020 08:33:35 +0000 (09:33 +0100)]
fix #2622: include all spam levels in total spam statistic
by using 'LIMIT 10' for the spamlevels, we only got the first
10 spamlevels back from the database. This is only ok if there are
only <= 10 different spamlevels in the database, but not if there are
more, as then the bucket for spamlevel >= 10 missed entries.
The call site of this uses the combined spam count of this query
result for calculating the 'rest' (meaning the mails with spam level
0), but this is obviously wrong if not all spamlevels are counted so
simply return all available levels.
Mira Limbeck [Tue, 18 Feb 2020 15:36:40 +0000 (16:36 +0100)]
add pmg-smtp-filter ID to reply
For the pmg-log-tracker to match the pmg-smtp-filter on a reject, we
need some kind of information. With the addition of the pmg-smtp-filter
ID we can match it the same way we do for an accept.
Signed-off-by: Mira Limbeck <m.limbeck@proxmox.com> Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Thomas Lamprecht [Fri, 21 Feb 2020 13:08:41 +0000 (14:08 +0100)]
replace lookup_node_ip with get_ip_from_hostname from pve-common
besides some irrelevant implementation details 'lookup_node_ip' is
identical to pve-common's get_ip_from_hostname, as they both rely on
'PVE::Tools::getaddrinfo_all' to get the addresses. So just reuse
the one from common instead of shipping a copy here.
The pve-common's one was recently improved by checking not only the
first IP it gets from getaddrinfo_all, but all and only complain if
none of those is a "real WAN (non-local)" IP. This will help
container installations of PMG with DHCP as their network
configuration option, as those often also have a hosts entry from
hostname to loopback addresses. Also, static setups often have both,
WAN and loopback addresses in /etc/hosts - as getaddrinfo_all gives
use them all do not just check the first.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Since the settings were overridden in the smtpd instances the error only
shows for locally generated mail (e.g. from cronjobs):
```
warning: connect to transport private/scan: Connection refused
```
Instead of globally setting 'smtpd_proxy_filter' (and associated options) in
'main.cf', as is done for 'content_filter' (for afterqueue filtering), this
patch removes the global 'content_filter' setting. This is done since
'smtp_proxy_filter' only applies to smtp sessions [0] and all incoming smtpd
processes get the setting in 'master.cf.in'.
Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
[0] see http://www.postfix.org/SMTPD_PROXY_README.html - the proxy speaks
smtp, and it's answers are sent to the sending server.
Dominik Csapak [Fri, 17 Jan 2020 09:52:47 +0000 (10:52 +0100)]
fix disclaimer encoding for html entities
we also want to encode the disclaimer for text/html parts and not
only for text/plain. while doing this, combine those two cases,
as they differ only by the variable to be encoded
this also fixes a missing charset, which we would ignore, but
should actually be treated as US-ASCII[0] so that an ascii disclaimer
still gets appended
the only (non-rfc-compliant) use case this breaks is if:
* the part has no charet defined (unusual)
* the clients of both the sender and receiver treat a missing charset
as 'iso-8859-1' (non-rfc-compliant)
* the disclaimer contains characters from 'iso-8859-1' and was added
to the pmg before encoding this to utf-8 (unlikely)
Stoiko Ivanov [Tue, 14 Jan 2020 18:31:35 +0000 (19:31 +0100)]
dkim: add selector list api call
The fix for #2504 left the GUI with an unsatisfactory UX:
Users can change the selector to any newly created or existing one, but
don't know which ones exist (without looking on the commandline)
By adding a method under '/config/dkim/selectors' which lists all existing
files matching the pattern '/etc/pmg/dkim/.*\.private' the GUI can display
all currently existing selectors.
Stoiko Ivanov [Tue, 14 Jan 2020 18:31:34 +0000 (19:31 +0100)]
fix #2504: do not overwrite existing selector key
This patch changes the behavior of DKIM selector creation. Instead of blindly
overwriting an already present file, add a force parameter to overwrite it (and
behave like the current code).
Overwriting an existing selector can potentially be quite destructive (e.g.
a setup where the admin has already posted the DNS-record for one selector to
many domains, then wants to quickly experiment with a larger keysize, and tries
to go back to the existing behavior).
The new behavior without force set to true, when a private key for the selector
already exists is to die if the file is either not a private RSA key, or has
the wrong size, or else just set the selector in pmg.conf
Julian Zehnter [Sun, 5 Jan 2020 14:53:07 +0000 (15:53 +0100)]
Feature #2438 add support for lmtp delivery to downstream servers
new feature lmtp support for simplifying setups
with lmtp capable downstream servers (e.g. dovecot)
Postfix support lmtp out of the box and can now deliver
mails directly to internal mailbox servers without
one more smtp connection
extending the api code for new lmtp option:
Config.pm:
Adding new variable "relayprotocol"
Extending the read_transport_map & write_transport_map
for parsing the /etc/pmg/transport
Transport.pm:
Add new protcol varialbe for smtp/lmtp setting
Generalizing some "SMTP" keywords
Templates:
Adapting the main.cf templates for adding the lmtp keyword
Dominik Csapak [Thu, 28 Nov 2019 10:22:43 +0000 (11:22 +0100)]
pmgproxy: implement cache workaround
we include the version of the packages in the temlate, so that
we can tell the browser that the file has changed and thus preventing
to load an old gui js file
the index template has to add the version to the get parameter
Stoiko Ivanov [Fri, 22 Nov 2019 09:40:57 +0000 (10:40 +0100)]
Don't add DKIM signature without domain
When the DKIMSign module fails to determine the domain for signing
(the one added to the header and used for retrieving the publickey record)
the code logs that no signing will take place, but only does not set the
domain - resulting in a generated and added signature with domain 'example.com'
Fixed by returning the success-status from signing_domain and only signing if
it was successful.
Stoiko Ivanov [Mon, 18 Nov 2019 09:03:17 +0000 (10:03 +0100)]
create pmg-scores.cf unconditionally
with the recent addtion of adjustable SA-rule scores, we introduced
'/etc/mail/spamassassin/pmg-scores.cf' as a new file and included it in the
cluster synchronization.
If the file does not exist the `rsync` command complains leading to misleading
errors in the journal.
Unconditionally creating the file (like we do for the already existing
'custom.cf') removes the warnings.
Reported-by: Martin Maurer <martin@proxmox.com> Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
Dominik Csapak [Thu, 14 Nov 2019 11:18:53 +0000 (12:18 +0100)]
add SACustom Package and API Calls for custom SpamAssassin scores
this uses our INotify interface to parse and write a custom sa config
in /etc/mail/spamassassin/pmg-scores.cf with a shadow file in
/var/cache/pmg-scores.cf (to track the diff)
add also api calls to create a new/delete/edit/revert/apply those custom
rules
Dominik Csapak [Thu, 14 Nov 2019 11:18:52 +0000 (12:18 +0100)]
implement force_restart for cluster sync
the 'force_restart' hash was unused, but it is actually necessary to
restart the pmg-smtp-filter if the custom.cf has changed
use it for saving the daemon name that needs to be restarted, this
way we can reuse it in the future to force restart other daemons
or for different reasons if we need to
Stoiko Ivanov [Thu, 14 Nov 2019 16:35:07 +0000 (17:35 +0100)]
add support for before queue filtering
Support for rejecting mails during the SMTP-dialog with 550 (permanent failure)
instead of seemingly accepting the mail (250 OK) and dropping it if it is
rejected by the rule-system is also known as 'before queue filtering' [0].
This patch adds minimal support for before queue filtering to pmg-smtp-filter.
Since pmg-smtp-filter is currently called via LMTP (and the 'scan' service in
'master.cf') we can adapt the already existing branch dealing with SMTP to
send a 550 selectively.
We can reply with 554 (permfail) if all recepients are blocked.
In the case that some accept the mail, we reply with 250 OK.
Depending on the setting of 'ndr_on_block' we generate ndrs
for all blocking recepients. (This is also the behavior that postfix
has when not enabling receiver verification and the downstream server rejects
recepients).
Configuration of before-queue filtering is done via the
'before_queue_filtering' boolean in the 'mail' section of 'pmg.conf':
the before_queue_filtering flag is used when rendering '/etc/postfix/master.cf'
to adapt the needed config-options for both, inbound and outbound, smtpd
servers. The settings were adapted from [0].
Stoiko Ivanov [Thu, 14 Nov 2019 16:35:06 +0000 (17:35 +0100)]
add support for sending NDRs on Block
This patch adds a flag to the mail-section of pmg.conf - 'ndr_on_block',
defaulting to false.
If enabled pmg-smtp-filter replies with 554 (permanent fail) for the recipient
who blocked the mail.
Since pmg-smtp-filter is currently called with LMTP it can respond with a
separate code for each recipient. Postfix records the answers and generates
and sends the NDR to the sender.
Stoiko Ivanov [Tue, 12 Nov 2019 14:16:13 +0000 (15:16 +0100)]
add generate_ndr to PMG::SMTP
In order to selectively accept and reject mails when multiple recipients
are present for a mail, without silently dropping mails to certain recipients
the sender needs to be informed about those recipients which did not receive
the mail (via a non-delivery report - NDR).
The format of delivery status notifications (of which an ndr is a subset) is
specified in RFC 6522 [0]. The format for the 'message/delivery-status'
MIME-Type is specified in RFC 6533 [1]
The message text was adapted from Postfix' default bounce-messages, however
we do not attach the original mail to the report. This is acceptable by [0]
and makes sense (not sending mail, which is likely spam or a virus, back to the
(potentially faked) sender).
Dominik Csapak [Wed, 23 Oct 2019 07:36:34 +0000 (09:36 +0200)]
Quarantine API: extend download call to download whole mails
this makes the attachmentid parameter optional and if it is not
given, open the whole mail for download
The permission check, that a quser only is able to get their own
mails/attachments, happens in get_and_check_mail, thus we can add the
quser to the "sufficient permissions" list without opening up the
real permissions the users has.
Stoiko Ivanov [Mon, 21 Oct 2019 17:23:35 +0000 (19:23 +0200)]
pmgproxy: add proxmox-widget-toolkit css path
With adding DKIM to the GUI the pmx-hint css class (defined in
proxmox-widget-toolkit) is needed in the GUI, therefore the css-file
needs to be available via pmgproxy
Stoiko Ivanov [Mon, 21 Oct 2019 17:23:34 +0000 (19:23 +0200)]
add /etc/pmg/dkim to cluster-sync
The clustersync already uses rsync to sync most files (there are excludes
for '*.db', '*~', and the certificates of the node) from the master's
'/etc/pmg' directory to all nodes. Moving files from the syncdir to the node's
actual '/etc/pmg' happen only for explicitly listed files.
This patch adds a second list of explicit directories and adds
'/etc/pmg/templates' and '/etc/pmg/dkim' to that list.
The syncing of all files inside this directory list is done via
`rsync -aq --delete-after`. This is a semantic change to the loop used
for the templates directory, in which only regular files and symlinks were
copied (rsync copies everything recursively).
Stoiko Ivanov [Mon, 21 Oct 2019 17:23:32 +0000 (19:23 +0200)]
add DKIM API paths
A new path is added at /config/dkim with 2 subpaths:
* /config/dkim/domains gives access to the dkimdomains
* /config/dkim/selector gives access to the private key for the configured
selector
Stoiko Ivanov [Mon, 21 Oct 2019 17:23:30 +0000 (19:23 +0200)]
refactor API2::Domains for reuse in DKIMSign
both DKIM Signed Domains and Relay Domains are lists of domains (DKIMSign falls
back to Relay Domains). By refactoring the method creation we can reuse most
of the code for the handling of DKIMSign domains
Stoiko Ivanov [Mon, 21 Oct 2019 17:23:29 +0000 (19:23 +0200)]
DKIM sign outbound mail if configured
The signing is done in the Accept-Action just before the mail gets handed to
the outbound postifx process, thus ensuring that all modifications done by
the rule-system don't invalidate the signature
The PMG::DKIMSign/DKIM::Signer object is not cached, since subsequent calls to
the same object lead to invalid signatures.
Stoiko Ivanov [Mon, 21 Oct 2019 17:23:28 +0000 (19:23 +0200)]
add PMG::DKIMSign module
the module serves 3 purposes:
* it extends Mail::DKIM::Signer:
* it provides a glue layer between MIME::Entity's output method (which
expects print and uses \n as line terminator) and Mail::DKIM::Signer's
PRINT method (which expects \r\n)
* it integrates with PMG's config
* the domain which should be used for signing is selected based on the
sender's e-mail address and the DKIM-settings in PMG-configuration
* it provides a method which takes a MIME::Entity and returns it with
signature
* certain headers get oversigned (in order to prevent adding a previously
non-existing header (e.g. Reply-To) and retaining a valid signature).
the list of headers which are oversigned is inspired by rspamd's choice [0].
for rationale see [1,2]
* it provides methods for handling selectors and keys.
Stoiko Ivanov [Mon, 21 Oct 2019 17:23:27 +0000 (19:23 +0200)]
add DKIM options to PMG::Config
This patch adds a new registered file to PMG::Config: /etc/pmg/dkim/domains.
* It holds a list of domains for which mail will be signed if received on the
internal port (26).
* The domain of a mail is determined by it's envelope-sender
(not the From: header)
* If the file does not exist mail is signed for the relay domains as a sensible
fallback
Additionally it adds 3 new options to the admin section of pmg.conf:
* dkim_sign - whether DKIM signing should be attempted
* dkim_sign_all_mail - wheter the outbound mail should be signed irrespective
of sender-domain
* dkim_selector - the selector used for signing (a label of the DKIM TXT record
and present in the signature - thus linking the signature to the used key
- see RFC6376)
Stoiko Ivanov [Mon, 21 Oct 2019 17:23:26 +0000 (19:23 +0200)]
fix #2371: reload pmg-smtp-filter on config change
the external services (postfix, clamav,...) are restarted if their configfile
changes (which Template::Toolkit tells us).
By writing a current-config to '/run/pmg-smtp-filter.cfg' we can use the same
logic to reload it on a config-change affecting it - currently hide_received
Dominik Csapak [Thu, 10 Oct 2019 09:21:57 +0000 (11:21 +0200)]
RuleDB/Remove: improve attachment detection for 'remove all'
this patch improves the attachment detection in such a way that
instead of leaving the first part if it has a content-type of
text/* and removing all other parts we now leave the 'real' message, meaning:
* the first text/plain or text/html part if no multipart
* the first text/plain or text/html part if multipart/mixed
* all text/plain and text/html parts of the first multipart/alternative
(if we did not already see the message)
this way the 'real' message including the text/html alternative
(or the only text/html part if no text/plain was sent) is
left in the message that is being sent to the receiver
also add missing '$rulename' parameter to the recursive call to
'delete_marked_parts'
Dominik Csapak [Thu, 10 Oct 2019 09:21:52 +0000 (11:21 +0200)]
HTTPServer: extend download functionality
allow object as return value for 'download' api calls
this way, we can give additional information for download apis
(like the content-type and if it should be deleted after)
While perl5 returns the result of the last expression implicitly, if
no explicit return statement is in a submethod's code path, it's much
nicer to do the things explicit most of the times, and a slight
modification could even break this.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
fix #2360: cluster: reload pmg-smtp-filter on rulechange
When a node detects a change of the rule-database during clustersync,
the database update got done, however the running `pmg-smtp-filter`
instance was not notified of the updated ruleset. Adding a call to
PMG::DBTools::reload_ruledb (like in the API2 paths) fixes the issue.
As this is guarded by a digest change check it'll only reload if the
ruleDB really changed.
We do not pass the currewnt ruleDB instance to the reload method, as
it is only used for "rewrite_postfix_whitelist", which is already
executed through the pmgmirror::cluster_sync call to
PMG::Config->rewrite_config, if needed, so don't do that twice.
Tested by
* creating a 2 node PMG cluster
* adding and removing e-mail-addresses to the Blacklist (which was used in
a high priority, active rule)
* sending e-mails from those addresses through PMG
* observing the action taken before/after the Rule DB got synced
Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com> Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
This is a fix for a small cosmetic issue, which keeps coming up on our various
support-channels:
pmgpolicy as user of 'Mail::SPF::Server' logs lines referencing the (now
defunct) url: http://www.openspf.org/Why
By overriding 'default_authority_explanation' [0] the logs now contain a simple
logmessage without any reference to an external url.
All macros used before are used in the new message as well, so the information
content stays the same.
Given that we are (by far) not the only users of this module I additionally
opened an issue upstream [1] (and linked it to the debian bug-report [2]).
Tested by providing a SPF policy not allowing the ip of the sender in my
testsetup and observing the logs.
Stoiko Ivanov [Mon, 19 Aug 2019 16:32:14 +0000 (18:32 +0200)]
close #2324 - improve docs on encrypted archives
The option for ClamAV 'Block Encrypted Archives and Documents', actually
triggers a Heuristic match from ClamAV, which is used by PMG to rais the
spam score of a message by the value of 'Heuristic Score' configured
at the 'Spam Detector' settings.
Since this has caused confusion (also for myself) a few times already, this
patch tries to address the issue by referring to the other option in the API
description of both properties
and should enable us to update pve-common in PMG 5.x along with PVE again.
Tested by:
* logging into PMG (get csrf token)
a) installing only the previous patch:
* POSTing (with the previously obtained token) -> 401
b) installing previous and this patch
* POSTing (with the previously obtained token) -> 200
Stoiko Ivanov [Fri, 9 Aug 2019 07:07:30 +0000 (09:07 +0200)]
adapt journalctl invocation to buster
With Debian Buster the behavior of `journalctl` has changed when it finds
no entries for a given selection:
* The exit code was 0 in stretch, but is 1 in buster
* The output changed slightly - a header got added
Since PMG::Utils::scan_journal_for_rbl_rejects uses journalctl for reading the
IPs blocked by postscreen it needs to adapt for the new behavior (otherwise
run_command dies because of the exit code 1)
The patch addresses the problem by using the json-output of `journalctl`, which
still exits with 0 if no entries are present . Additionally the json-output
adds the current cursor to the output by default, removing the need to
explicitly scanning for it.(the exit code of 1 was due to '--show-cursor'
without a single result line).
Stoiko Ivanov [Thu, 8 Aug 2019 16:09:22 +0000 (18:09 +0200)]
adapt clamav.conf.in to new upstream version
The new version of clamav has changed a few options, which we ship in our
config template - see [0] for the details.
This patch changes all affected options to the new names and additionally
expands the Description of 'archiveblockencrypted' to reflect that the option
is used for both archives _and_ documents.
projects / pmg-api.git / log