ruledb: match field: validate regular expressions on addition
Do not save rules if they die during an execution test, which is done
by using them once on an empty string.
Since users may have saved already invalid ones, only warn if we
encounter such a regex in 'parse_entity' during execution instead of
dying. Otherwise pmg-smtp-filter will exit and restart, possibly
leading to wrongly denying mails (and possibly sending out NDRs)
before spam checking was done.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com> Tested-by: Mira Limbeck <m.limbeck@proxmox.com> Reviewed-by: Mira Limbeck <m.limbeck@proxmox.com>
[ T: touch up commit subject/message ] Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Stoiko Ivanov [Mon, 27 Mar 2023 19:18:13 +0000 (21:18 +0200)]
quarantine: delete Delivered-To and Return-Path when reinjecting
The removal of those 2 headers was dropped in the recent rework for
quarantine delivery.
Leading to mails from quarantine being bounced by postfix 'local'
delivery agent (as the comment in the original code stated)
Reproduced by delivering a mail from quarantine to a postfix instance,
which routes it to a local account
Fixes: e51fe74 ("quarantine: use reinject_local_mail to deliver quarantined mail") Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
Stoiko Ivanov [Fri, 17 Mar 2023 18:44:54 +0000 (19:44 +0100)]
api: quarantine: decode addresses before delivery/userlisting
With the change of using reinject_local_mail for the quarantine
delivery the issue of not properly decoding the entries we get from
the database before delivering became apparent
The database returns utf-8 encoded strings, reinject_local_mail and
add_to_blackwhite expects perl-strings (with wide characters) and
encodes them (a second time) - this patch decodes the database strings
before passing it on.
add_to_black_white is used in a few API calls (via
read_or_modify_user_bw_list), therefore the approach of decode (from
database), and encode (for database) was chosen.
Stoiko Ivanov [Fri, 17 Mar 2023 18:44:53 +0000 (19:44 +0100)]
quarantine: use reinject_local_mail to deliver quarantined mail
the current delivery looks quite similar to reinject_local_mail,
apart from the database handling and sending the mail-contents from a
file instead of a MIME::Entity.
While reparsing the mail might seem expensive, the quarantine code
does so multiple times when users click in the quarantine GUI (see
PMG::HTMLMail, and the attachment quarantine)
The issue of MIME::Parser being lossy [0] (parsing and then printing
the entity, might not return the original mail byte-by-byte), is
already present in our code-base anyways (when the mail gets
quarantined (or sent on) it is from a parsed MIME::Entity).
Stoiko Ivanov [Fri, 17 Mar 2023 18:44:52 +0000 (19:44 +0100)]
reinject mail: improve error logging
this patch unifies the error handling for mail and rcpt with
the data command: all now die with sensible error (which gets logged
in the error-handling of the eval), and it sets the respose message
and code for those commands as well.
additionally it adds a '\n' to all die statements.
this makes it possible to provide information what went wrong at
call-sites (instead of only having it in syslog)
Stoiko Ivanov [Fri, 17 Mar 2023 18:44:51 +0000 (19:44 +0100)]
config: make smtputf8 configurable through the API
the flag is simply a boolean which is used to:
* add smtputf8_enable = no to postfix' main.cf if it is disabled
(the default is to enable it, and not adding it unconditionally,
should cause the fewest surprises for users with modified templates)
* decide if locally generated mail should be scanned for utf8 headers
and addresses (to set the parameter to the MAIL command)
This should match postfix own implementation w.r.t. smtputf8 behavior.
Additionally, since quite a few users need to disable it because
their downstream servers do not support it (Zimbra, OpenXchange,
MS Exchange), this should make for a better user experience.
Stoiko Ivanov [Fri, 17 Mar 2023 18:44:50 +0000 (19:44 +0100)]
smtputf8: keep smtputf8 from incoming postfix, detect for local mail
This patch changes the detection if smtputf8 is needed as option to
the 'MAIL' command:
* for mail arriving through postfix it is only added if the mail
originally was received with it (Accept and BCC actions)
* for locally generated mail (Notify, reports, quarantine-link and
ndrs) it is decided based on utf8 characters in the mail-addresses
or headers - this is done by `reinject_local_mail`, as a new helper
This should match postfix own behavior in those cases quite
closely:
https://www.postfix.org/SMTPUTF8_README.html#using
Notable difference is that we check the complete e-mail address and
not only the domain part, but I assume non-ascii local-parts to be a
very fringe edge-case in environments where smtputf8 is not supported.
Stefan Sterz [Thu, 9 Feb 2023 11:41:23 +0000 (12:41 +0100)]
fix #4521: api/tasks: replace upid as filename for task log downloads
previously the upid would just be used without a file extension when
downloading a task log. this lead to rather strange filenames that
appeared unfamiliar to users as the upid is not very prevalent in the
gui. set a proper file name based on the node name, worker type and a
time stamp instead. also add the ".log" file extension to indicate
that these files contain logs.
Stoiko Ivanov [Mon, 20 Mar 2023 20:26:38 +0000 (21:26 +0100)]
config: warn on parse errors for tls related config files
this unifies the error-handling with transports and mynetworks -
resulting in wrong entries being logged, the remaining entries
displayed in the GUI, and the wrong entries being dropped upon
editing.
Christoph Heiss [Mon, 20 Mar 2023 10:35:46 +0000 (11:35 +0100)]
fix #2437: api: Add endpoint for managing tls_inbound_domains entries
Add a new API endpoint `/config/tlsinbounddomains` for managing entries
of the `tls_inbound_domains` postfix map. Modelled after the
`DestinationTLSPolicy` implementation.
Signed-off-by: Christoph Heiss <c.heiss@proxmox.com>
Christoph Heiss [Mon, 20 Mar 2023 10:35:45 +0000 (11:35 +0100)]
fix #2437: config: Add new tls_inbound_domains postfix map
Add a new configuration file /etc/pmg/tls_inbound_domains, which is a
postfix map containing all domains having `reject_plaintext_session`
action set. This is the only allowed action value and enforced while
parsing.
This map is then used for `smtpd_sender_restriction` in the main.cf
template.
Signed-off-by: Christoph Heiss <c.heiss@proxmox.com>
Stoiko Ivanov [Mon, 13 Mar 2023 21:23:50 +0000 (22:23 +0100)]
templates: enable DMARC plugin in v400.pre.in
This module needs Mail::DMARC (libmail-dmarc-perl) as prerequisite.
It is currently only available in sid and bookworm, but can be
trivially rebuild for bullseye.
the dmarc tests are skipped if only internal relays are used/present
in the headers, so I could not explicitly test this
Stoiko Ivanov [Mon, 13 Mar 2023 21:23:49 +0000 (22:23 +0100)]
templates: enable DecodeShortUrls for SpamAssassin 4.0.0
enabled if they system has rbl_checks enabled.
The module resolves url-shortener (e.g. bit.ly) chains.
the KAM rulset has a number of url-shorteners configured
(KAM_urlshorteners.cf).
While the functionality also works without the configured caching
module, it worked well in my tests.
Stoiko Ivanov [Mon, 13 Mar 2023 21:23:48 +0000 (22:23 +0100)]
config: add spam option for extract_text
toggling the configuration options for the ExtractText SA plugin (see
[0]).
The config is copied from the module itself, the informational headers
were not added, as I don't see too much gain, apart from verifying
that the plugin is working.
the external dependencies for the plugin to work are added as
Recommends, as it is a possible config to not have them installed and
simply disable the option
Stoiko Ivanov [Mon, 13 Mar 2023 21:23:44 +0000 (22:23 +0100)]
ruledb: spam: adapt to spamassassin 4.0.0
find_all_addrs_in_line was changed to require an instantiated
Mail::SpamAssassin instance in:
https://github.com/apache/spamassassin/commit/139adfb5901b27fa13dccbf3a66c53ca7613f733
(read-only git mirror of the authoritative SVN)
Noticed while using `mutt` and bouncing mails, which adds Resent
headers.
Stefan Sterz [Thu, 9 Mar 2023 08:00:15 +0000 (09:00 +0100)]
proxy: add support for switching themes
parse the theme cookie so users can switch between themes in the ui
this requires a bump of the pmg-gui, which in turn needs a bump for
the widget toolkit, so the parameters passed to the template are
handled appropriately.
Leo Nunner [Fri, 3 Mar 2023 15:56:11 +0000 (16:56 +0100)]
fix #4536: parse original filenames from gzip files
GZIP provides the possiblity to store the original filename in the
optional FNAME header field, which we can use for 'Match Archive
Filename' rules.
IO::Uncompress::Gunzip is explicitly recommended for this purpose by the
documentation on Compress::Zlib, so an additional imnport was
introduced here. Consequently, libio-compress-perl was added as an extra
dependency to the debian control file.
Stoiko Ivanov [Mon, 23 Jan 2023 15:55:20 +0000 (16:55 +0100)]
utils: skip checking headers for non-ascii characters
The fix for smtputf8 enablement in 191e470 ("utils: fix mailflow if
smtputf8 is disabled") was a bit too eager and broke the mail flow of
a few users, who have smtputf8 disabled in their postfix config
(because their downstream servers do not support this):
The issue here is that the mails they process have had
(non-rfc-compliant) non-ascii header contents, which used to work
before the patch. The postfix smtputf8 how-to explains quite well
that postfix never cared too much about headers (or local-parts of
addresses) before smtputf8 [1]
```
Postfix already permitted UTF-8 in message header values and in
address localparts. This does not change.
```
While the patch only ignores the headers and still could cause issues
with non-ascii local-parts, those should occur far less frequently in
the wild (none of the reporters in our forum had this).
Tested with a mail with an Euro sign in a 'X-From:' header.
Note that this is a stop-gap for re-working how to decide if smtputf8
is needed in a manner that better matches real world systems.
Christoph Heiss [Thu, 29 Dec 2022 09:45:17 +0000 (10:45 +0100)]
fix #4410: Remove non-null host bits from CIDR when writing postfix config
This will drop non-null host bits from `mynetworks` CIDRs when writing
the `main.cf` postfix template.
Backwards-compatibility with old entries in `/etc/pmg/mynetworks` is
thus also preserved.
Add an additional comment to the mynetworks API, indicating that unused
fields can/should be dropped with the next PMG version.
No GUI changes. The entries are written to `/etc/pmg/mynetworks` as the
user enters them. Suggested by Stoiko, see discussion in v2 thread [0].
Stoiko Ivanov [Wed, 21 Dec 2022 14:53:43 +0000 (15:53 +0100)]
utils: fix mailflow if smtputf8 is disabled
with the recent addition of smtputf8 support for the rulesystem setups
explicitly disabling smtputf8 in postfix got broken.
This is mostly noticeable for the spamreports (the receivers are taken
from the database and potentially decoded from utf-8, which sets the
'is_utf8' flag, and then tries to use the smtputf8 extension when
reinjecting the mail, which fails (since smtputf8 is disabled)
Instead of checking for the internal flag, we check for occurence of
characters which are not ascii printable (everything excluding
controlcharacters - '[\x20-\x7E]') in the envelope-addresses and
headers (there also for [\r\n\t], due to searching all headers and
folding). - see
https://perldoc.perl.org/perlunifaq#What-is-%22the-UTF8-flag%22? and
https://perldoc.perl.org/perlrecharclass#POSIX-Character-Classes
The only diversion from the requirements in the smptutf8 rfc
https://www.rfc-editor.org/rfc/rfc6531
is that we do not check the headers of all parts of a multipart
message (think suggested filename for an attachment), but I assume
that this should not be an issue in mail-transit
the addresses now always get encoded as UTF-8, as this is robust for
aascii-only addresses.
reported in our community forum:
https://forum.proxmox.com/threads/.119387/
issue is reproducible by setting
`smtputf8_enable = no` in postfix main.cf
and sending a spamreport using `pmgqm`
regular mailflow should not be affected in those setups (as no utf-8
addresses would come into the system)
Stoiko Ivanov [Tue, 20 Dec 2022 10:57:35 +0000 (11:57 +0100)]
rulecache: sort rules additionally by id
When more rules have the same priority currently their order is not
stable - postgres returns them in a stable way, based on their last
changetime - e.g. disabling and reenabling a rule puts it in the front
of evaluation. Sortin by id (the primary key) in addition should make
rule evaluation robust to such updates
While there is no guarantee of ordering (within the same priority)
unexpected changes in which rule fires can cause confusion (at least
it confused me quite a bit).
Thomas Lamprecht [Mon, 12 Dec 2022 12:05:00 +0000 (13:05 +0100)]
backup restore: code style cleanup
find also takes a code ref directly, and the "wanted" name is a bit
non-ideal anyway, as it has no control over what find "wants" (i.e.,
descends into), but its return value is completely ignored.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Stoiko Ivanov [Wed, 30 Nov 2022 17:07:27 +0000 (18:07 +0100)]
backup: restore: keep directories in /etc/pmg for inotify
By wiping the subdirectories in /etc/pmg/, we lose the inotify
watchers upon restore (/etc/pmg itself and thus most configs are
currently handled by the keep_root flag to rmtree)
This can lead to inconsistencies after restoring for parts relying on
config in a subdirectory (e.g. /etc/pmg/pbs/pbs.conf).
This patch uses File::Find (included in perl-modules-$perlver) to keep
all directories an unlink everything else.
This was chosen for future robustness over keeping an explicit list of
directories to keep, in case a new directory gets added.
quickly tested with a fifo, chardev, and socket in the directory.
an alternative approach would be to simply reload pmgdaemon/pmgproxy
upon config-restore, but that feels more likely to miss some
(potentially future) service, expecting inotify to work.
Stoiko Ivanov [Tue, 29 Nov 2022 15:52:58 +0000 (16:52 +0100)]
api: tasks: keep early exit if rpcenv is not initialized
while $rpcenv is only used in the end to set_result_attrib, the get()
call can cause an early exit if the RESTEnvironment is not
initialized. keep it that way (as with most/all API calls)
Stoiko Ivanov [Mon, 28 Nov 2022 18:17:29 +0000 (19:17 +0100)]
user accesslists: reword logging and hits for newer SA rule sets
This commit adapts the sa-hits and the logging for the user
block/welcomelist to be consistent with the terms used in the
SpamAssassin 4.0 release, which tries to avoid some terms that might
be interpreted as racially charged.
This patch is a (small part) of the fix for #3755, which will be
addressed along with the upgrade to SpamAssassin 4.0 (to be
consistent with the (quite well thought-through) namings used by SA)
Keeping the USER_IN_BLACKLIST hit when loading the descriptions
catches mails put in quarantine before the patched version was
installed.
Stoiko Ivanov [Mon, 28 Nov 2022 18:17:28 +0000 (19:17 +0100)]
user-bl: use custom description of USER_IN_BLACKLIST consistently
The USER_IN_BLACKLIST spamassassin hit is created by the Spam What
object, if the sending e-mail is in the receivers blacklist.
This 'hit' is kept on the PMG only - it is not written to the SPAMINFO
macro - and only visible in the quarantine interface afaict.
The description shown in the quarantine interface, however is read
from SpamAssassin sources.
They have recently changed to include a 'DEPRECATED' prefix, since the
rules containing 'blacklist' and 'whitelist' have been renamed to
'blocklist' and 'welcomelist' for the upcoming 4.0 series of
spamassassin.
In any case we should keep our description consistent, thus the move
to a sub of its own for reusing in both locations.
The mechanism for welcomlisted/whitelisted mails does not create an
'internal' sa-rule (but simply drops the SA hits for analysis) - so no
symmetric change is needed.
Reported-by: Thomas Lamprecht <t.lamprecht@proxmox.com> Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
The read_tasklog API call now stream the whole log file if the query
parameter 'download' is set to true.
This is done in preparation for the task log download button to be
added in the TaskViewer.
I saw an opportunity here to clear some redundant code for displaying
the tasklog and replaced it with a call to dump_logfile(), akin to how
this is handled in pve-manager.
Signed-off-by: Daniel Tschlatscher <d.tschlatscher@proxmox.com>
Dominik Csapak [Thu, 24 Nov 2022 12:21:11 +0000 (13:21 +0100)]
ldap: improve unicode support
when we receive mails with SMTPUTF8 encoded sender/recipient,
we have to encode these values for our ldapcache to work,
otherwise pmg-smtp-filter fails with when trying to insert
perl strings.
on read from the cache we have to decode these values again so
that the webui can show them correctly
also encode/decode dn and group names, since according to rfc4514[0]
utf-8 should be ok here
Stoiko Ivanov [Thu, 24 Nov 2022 12:21:07 +0000 (13:21 +0100)]
pmgqm: handle smtputf8 data
$data->{pmail} is both used in the template rendering ('Spam Report for
$pmail'), and as content for the To header, which need different
treatment. Thus introduce 'pmail_raw' additionally.
Stoiko Ivanov [Thu, 24 Nov 2022 12:21:06 +0000 (13:21 +0100)]
quarantine: handle utf8 data
use try_decode_utf8 for sender/receiver of the smtp dialog and mail
headers since they're either ASCII (not SMTPUTF8) or UTF-8 (with SMTPUTF8)
encoded
change the mail regex for wl/bl to basic email/domain syntax without
the restriction of ascii only. (whitespace and backslashes are
forbidden, but they shouldn't normally occur in email addresses and
domains)
Stoiko Ivanov [Thu, 24 Nov 2022 12:21:05 +0000 (13:21 +0100)]
partially fix #2465: handle smtputf8 addresses in the rule-system
the envelope addresses are used in the rule-system for lookups and
statistics. When the mail is received with smtputf8 the addresses are
decoded (multi-byte perl-strings) and thus need encoding before using
them as parameter in a database query.
This patch encodes the addresses as utf-8 for the relevant queries
unconditionally, because envelope-senders should either be:
* (a subset of) ascii (no smtputf8) - which is invariant for utf-8
encoding
* valid utf-8 (smtputf8)
The patch does not address the issues with multi-byte addresses in our
LDAP-implementation (hence the partial fix), but should still be an
improvment for many deployments
Stoiko Ivanov [Thu, 24 Nov 2022 12:21:03 +0000 (13:21 +0100)]
fix #2541 ruledb: encode relevant values as utf-8 in database
This patch adds support for storing rule names, comments(info), and
most relevant values (e.g. the header content to match) in utf-8 in
the database.
backwards-compatibility should not be an issue:
* currently the database should not contain any utf-8 multibyte
characters, as our tooling prevented this due to sending
wide-characters, which causes an exception in DBI.
* any character > 127 and < 256 will be correctly interpreted when
stored in a perl-string (this happens if the decode fails in
try_decode_utf8), and will be correctly encoded when storing into
the database.
the database is created with SQL_ASCII encoding - which behaves by
interpreting bytes <= 127 as ascii and those > 127 are not interpreted
(see [0], which just means that we have to explicitly en-/decode upon
storing/reading from there)
This patch currently omits most Who objects:
* for email/domain we'd still need to consider how to store them
(puny-code for the domain part, or everything as UTF-8) and it would
need changes to the API-types.
* the LDAP objects currently would not work too well, since our LDAPCache
is not UTF-8 safe - and fixing warants its own patch-series
* WhoRegex should work and be able to handle many use-cases
The ContentType values should also contain only ascii characters per
RFC6838 [1] and RFC2045 [2].
Stoiko Ivanov [Thu, 24 Nov 2022 12:21:02 +0000 (13:21 +0100)]
ruledb: properly substitute prox_vars in headers
by storing the variables as perl-string (not mime-encoded, and not
utf-8 encoded), and appropriately dealing with multi-line values to
input (folding the headers and encoding as mime).
Stoiko Ivanov [Thu, 24 Nov 2022 12:21:01 +0000 (13:21 +0100)]
utils: return perl string from decode_rfc1522
decode_rfc1522 is a more robust version of decode_mimewords (in
scalar context) - adapt it to return a perlstring, under the
assumption that data is utf-8 encoded (holds true for ascii and
smtputf8 mails)
the try_decode_utf8 helper sub backwards will be used extensively in
later patches and is inspired by commit 43f8112f0bb424f99057106d57d32276d7d422a6 in pve-storage:
We consider that the valid multibyte utf-8 characters do not really
yield sensible combinations of single-byte perl characters (starting
with a byte > 127 - e.g. "£") so if something decodes without error
from utf-8 it will in all likelyhood have been utf-8 to begin with
Dominik Csapak [Wed, 23 Nov 2022 14:52:21 +0000 (15:52 +0100)]
fix #3287: add pmail parameter to virus/attch. quarantine list
so that we can filter by the recipient email
for that we also have to add the quarantine type to the 'spamusers' api
call, or else we cannot list which recipients have mails in the
respective quarantine
Stoiko Ivanov [Wed, 9 Nov 2022 18:27:25 +0000 (19:27 +0100)]
ruledb: add deprecation warnings for unused actions
* ReportSpam
* Attach
* Counter
are all still present since (at least) the release of PMG 5.0, but
were never exposed in the API/GUI.
All of them in their current form don't seem to fit well nowadays, or
their functionality was taken over by some other Action:
* Attach - the functionality is currently present in the Notify action
(attach original mail)
* Counter - without a matching What object simply increasing a counter
by one in the database serves no purpose
* ReportSpam - sending potentially sensitive mail automatically to the
public SpamAssassin project does not seem to fit well nowadays
Instead of dropping them right away - this patch adds logging when
they are encountered while loading or when they are run, to keep
backwards-compatibility for users who have very long-running PMG
instances (not sure if the actions were ever used in the pre git-days
of PMG)
MIME::Words::encode_mimewords does not deal with multiline headers
(the warning about this being a 'quick and dirty' solution [0]
partially tells as much).
Instead - split the replacement value after variable substition on:
'\r?\n\s*' (to capture multi-line values like __SPAM_INFO__, but also
already folded headers, which are separated by '\r?\n\s+') and do the
substitution for each line seperately.
reported in our community forum:
https://forum.proxmox.com/threads/.118001/
Dominik Csapak [Fri, 4 Nov 2022 15:04:20 +0000 (16:04 +0100)]
api/quarantine: allow 'listattachments' for quarantine users
we use 'get_attachments' which uses 'get_and_check_mail'. that already
checks the correct permsissions (quser are only able to retriever their
own mails/attachments) so it's ok here to allow it
Dominik Csapak [Wed, 5 Oct 2022 07:49:41 +0000 (09:49 +0200)]
RuleDB/Notify: properly en-/decode the mail subject
we need to mime decode the subject after reading it, so that we get
the 'real' subject instead of the (possibly) mime encoded one (which
might be base64 or quoted-printable encoded). To get a proper subject in
the notification mail again, we have to encode it again before passing
it MIME::Entity->build
fix #4269: rule cache: from match: cope with undefined IP
No semantic change, just avoids an ugly warning. Can normally only
happen if a mail is send/inject directly from the PMG host.
fwiw, all the rule implementation that actually use $ip got an early
return 0 if $ip evaluates to false(y), one might actually consider
checking the counterpart too for false-y in this case, and return a
match if both are false (or maybe better, make the check a
definedness one); but as this is for an edge case we might just keep
it as is for now, worked ok for more than a decade..
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
projects / pmg-api.git / log