users: add endpoint for unlocking the TFA of a user
add /access/users/<userid>/unlock-tfa api call which can be used for
unlocking a user after their TFA got locked due to many failed
consecutive retries.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com> Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Stoiko Ivanov [Tue, 27 Jun 2023 09:00:27 +0000 (11:00 +0200)]
report: adapt to changes in SpamAssassin DNS api
SpamAssassin 4.0 changed the way it does DNS-lookups a bit (switched
to asynchronous lookups) - this broke pmg-system-report, since we use
the SpamAssassin API to check that DNS-resolution works. The reason
for this is that SA used to take only the first entry from
/etc/resolv.conf - and SA being able to do correct resolution is
critical for it to work.
This patch fixes the incompatible use of the DNS-API, but does not
change to the asynchronous model.
Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com> Tested-by: Friedrich Weber <f.weber@proxmox.com>
Stoiko Ivanov [Mon, 26 Jun 2023 20:45:10 +0000 (22:45 +0200)]
postgresql compat: cast result from EXTRACT to INTEGER
Postgresql has changed the return type of the EXTRACT function to
numeric from float8 [0] in version 14, and I strongly assume that this
change is the reason why:
`SELECT EXTRACT (EPOCH FROM now());`
now returns a floating point instead of an integer value, which in
turn is not accepted in the prepared statements throughout our
codebase.
Dominik Csapak [Mon, 26 Jun 2023 14:10:26 +0000 (16:10 +0200)]
dbtools: grant permissions public schema for created databases
since postgres 15, the public schema is not world writeable anymore for
security reasons. In our environment, where the db is not externaly
reachable and no database users should exists except the ones we create,
we can safely give the permissions again to be able to use
the root/www-data user without modification of the remaining
code/privileges for postgres.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com> Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Dominik Csapak [Mon, 26 Jun 2023 12:30:43 +0000 (14:30 +0200)]
introduce pmg7to8 cli helper
mostly copied from pve7to8 (without the pve specific tests) with some
notable additions to check some basic things for the pmg upgrade:
* check if the cluster is healthy
* check if the services are stopped(pre-upgrade)/started(post-upgrade)
* check if the db was upgraded (post upgrade)
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com> Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Thomas Lamprecht [Mon, 26 Jun 2023 08:43:00 +0000 (10:43 +0200)]
d/control: depend on rsyslog
required for our current tracking center implementation, a central
feature for PMG, which uses rsyslog log files and format.
Note that we evaluated switching to the journal there, but that was
deemed to be too slow (albeit could have only been start-up time
penalty) – anyhow, as of now this is a requirement to get the full
functionality, once the log-tracker can understand other formats in
an efficient way too we can add those as alternatives.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Stoiko Ivanov [Fri, 31 Mar 2023 11:27:47 +0000 (13:27 +0200)]
d/maintscripts: prevent aborting on errors in some commands
in case something goes wrong it is often better to not leave the
packaging state broken.
failure in the commands masked by this patch are either transient
(pmgconfig sync -restart 1 failing when services are masked), or will
be noticed quite instantly (failed database or config initialization
upon first install)
the deb-systemd-invoke change was based on a quick grep in
/var/lib/dpkg/info on my system
I quickly considered masking even more errors (e.g. related to the ucf
handling) - but they don't seem to cause issues (in the past 3 years)
- and if something breaks there it is probably worth to get a report
reported in our community forum:
https://forum.proxmox.com/threads/.125088/
ruledb: match field: validate regular expressions on addition
Do not save rules if they die during an execution test, which is done
by using them once on an empty string.
Since users may have saved already invalid ones, only warn if we
encounter such a regex in 'parse_entity' during execution instead of
dying. Otherwise pmg-smtp-filter will exit and restart, possibly
leading to wrongly denying mails (and possibly sending out NDRs)
before spam checking was done.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com> Tested-by: Mira Limbeck <m.limbeck@proxmox.com> Reviewed-by: Mira Limbeck <m.limbeck@proxmox.com>
[ T: touch up commit subject/message ] Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Stoiko Ivanov [Mon, 27 Mar 2023 19:18:13 +0000 (21:18 +0200)]
quarantine: delete Delivered-To and Return-Path when reinjecting
The removal of those 2 headers was dropped in the recent rework for
quarantine delivery.
Leading to mails from quarantine being bounced by postfix 'local'
delivery agent (as the comment in the original code stated)
Reproduced by delivering a mail from quarantine to a postfix instance,
which routes it to a local account
Fixes: e51fe74 ("quarantine: use reinject_local_mail to deliver quarantined mail") Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
Stoiko Ivanov [Fri, 17 Mar 2023 18:44:54 +0000 (19:44 +0100)]
api: quarantine: decode addresses before delivery/userlisting
With the change of using reinject_local_mail for the quarantine
delivery the issue of not properly decoding the entries we get from
the database before delivering became apparent
The database returns utf-8 encoded strings, reinject_local_mail and
add_to_blackwhite expects perl-strings (with wide characters) and
encodes them (a second time) - this patch decodes the database strings
before passing it on.
add_to_black_white is used in a few API calls (via
read_or_modify_user_bw_list), therefore the approach of decode (from
database), and encode (for database) was chosen.
Stoiko Ivanov [Fri, 17 Mar 2023 18:44:53 +0000 (19:44 +0100)]
quarantine: use reinject_local_mail to deliver quarantined mail
the current delivery looks quite similar to reinject_local_mail,
apart from the database handling and sending the mail-contents from a
file instead of a MIME::Entity.
While reparsing the mail might seem expensive, the quarantine code
does so multiple times when users click in the quarantine GUI (see
PMG::HTMLMail, and the attachment quarantine)
The issue of MIME::Parser being lossy [0] (parsing and then printing
the entity, might not return the original mail byte-by-byte), is
already present in our code-base anyways (when the mail gets
quarantined (or sent on) it is from a parsed MIME::Entity).
Stoiko Ivanov [Fri, 17 Mar 2023 18:44:52 +0000 (19:44 +0100)]
reinject mail: improve error logging
this patch unifies the error handling for mail and rcpt with
the data command: all now die with sensible error (which gets logged
in the error-handling of the eval), and it sets the respose message
and code for those commands as well.
additionally it adds a '\n' to all die statements.
this makes it possible to provide information what went wrong at
call-sites (instead of only having it in syslog)
Stoiko Ivanov [Fri, 17 Mar 2023 18:44:51 +0000 (19:44 +0100)]
config: make smtputf8 configurable through the API
the flag is simply a boolean which is used to:
* add smtputf8_enable = no to postfix' main.cf if it is disabled
(the default is to enable it, and not adding it unconditionally,
should cause the fewest surprises for users with modified templates)
* decide if locally generated mail should be scanned for utf8 headers
and addresses (to set the parameter to the MAIL command)
This should match postfix own implementation w.r.t. smtputf8 behavior.
Additionally, since quite a few users need to disable it because
their downstream servers do not support it (Zimbra, OpenXchange,
MS Exchange), this should make for a better user experience.
Stoiko Ivanov [Fri, 17 Mar 2023 18:44:50 +0000 (19:44 +0100)]
smtputf8: keep smtputf8 from incoming postfix, detect for local mail
This patch changes the detection if smtputf8 is needed as option to
the 'MAIL' command:
* for mail arriving through postfix it is only added if the mail
originally was received with it (Accept and BCC actions)
* for locally generated mail (Notify, reports, quarantine-link and
ndrs) it is decided based on utf8 characters in the mail-addresses
or headers - this is done by `reinject_local_mail`, as a new helper
This should match postfix own behavior in those cases quite
closely:
https://www.postfix.org/SMTPUTF8_README.html#using
Notable difference is that we check the complete e-mail address and
not only the domain part, but I assume non-ascii local-parts to be a
very fringe edge-case in environments where smtputf8 is not supported.
Stefan Sterz [Thu, 9 Feb 2023 11:41:23 +0000 (12:41 +0100)]
fix #4521: api/tasks: replace upid as filename for task log downloads
previously the upid would just be used without a file extension when
downloading a task log. this lead to rather strange filenames that
appeared unfamiliar to users as the upid is not very prevalent in the
gui. set a proper file name based on the node name, worker type and a
time stamp instead. also add the ".log" file extension to indicate
that these files contain logs.
Stoiko Ivanov [Mon, 20 Mar 2023 20:26:38 +0000 (21:26 +0100)]
config: warn on parse errors for tls related config files
this unifies the error-handling with transports and mynetworks -
resulting in wrong entries being logged, the remaining entries
displayed in the GUI, and the wrong entries being dropped upon
editing.
Christoph Heiss [Mon, 20 Mar 2023 10:35:46 +0000 (11:35 +0100)]
fix #2437: api: Add endpoint for managing tls_inbound_domains entries
Add a new API endpoint `/config/tlsinbounddomains` for managing entries
of the `tls_inbound_domains` postfix map. Modelled after the
`DestinationTLSPolicy` implementation.
Signed-off-by: Christoph Heiss <c.heiss@proxmox.com>
Christoph Heiss [Mon, 20 Mar 2023 10:35:45 +0000 (11:35 +0100)]
fix #2437: config: Add new tls_inbound_domains postfix map
Add a new configuration file /etc/pmg/tls_inbound_domains, which is a
postfix map containing all domains having `reject_plaintext_session`
action set. This is the only allowed action value and enforced while
parsing.
This map is then used for `smtpd_sender_restriction` in the main.cf
template.
Signed-off-by: Christoph Heiss <c.heiss@proxmox.com>
Stoiko Ivanov [Mon, 13 Mar 2023 21:23:50 +0000 (22:23 +0100)]
templates: enable DMARC plugin in v400.pre.in
This module needs Mail::DMARC (libmail-dmarc-perl) as prerequisite.
It is currently only available in sid and bookworm, but can be
trivially rebuild for bullseye.
the dmarc tests are skipped if only internal relays are used/present
in the headers, so I could not explicitly test this
Stoiko Ivanov [Mon, 13 Mar 2023 21:23:49 +0000 (22:23 +0100)]
templates: enable DecodeShortUrls for SpamAssassin 4.0.0
enabled if they system has rbl_checks enabled.
The module resolves url-shortener (e.g. bit.ly) chains.
the KAM rulset has a number of url-shorteners configured
(KAM_urlshorteners.cf).
While the functionality also works without the configured caching
module, it worked well in my tests.
Stoiko Ivanov [Mon, 13 Mar 2023 21:23:48 +0000 (22:23 +0100)]
config: add spam option for extract_text
toggling the configuration options for the ExtractText SA plugin (see
[0]).
The config is copied from the module itself, the informational headers
were not added, as I don't see too much gain, apart from verifying
that the plugin is working.
the external dependencies for the plugin to work are added as
Recommends, as it is a possible config to not have them installed and
simply disable the option
Stoiko Ivanov [Mon, 13 Mar 2023 21:23:44 +0000 (22:23 +0100)]
ruledb: spam: adapt to spamassassin 4.0.0
find_all_addrs_in_line was changed to require an instantiated
Mail::SpamAssassin instance in:
https://github.com/apache/spamassassin/commit/139adfb5901b27fa13dccbf3a66c53ca7613f733
(read-only git mirror of the authoritative SVN)
Noticed while using `mutt` and bouncing mails, which adds Resent
headers.
Stefan Sterz [Thu, 9 Mar 2023 08:00:15 +0000 (09:00 +0100)]
proxy: add support for switching themes
parse the theme cookie so users can switch between themes in the ui
this requires a bump of the pmg-gui, which in turn needs a bump for
the widget toolkit, so the parameters passed to the template are
handled appropriately.
Leo Nunner [Fri, 3 Mar 2023 15:56:11 +0000 (16:56 +0100)]
fix #4536: parse original filenames from gzip files
GZIP provides the possiblity to store the original filename in the
optional FNAME header field, which we can use for 'Match Archive
Filename' rules.
IO::Uncompress::Gunzip is explicitly recommended for this purpose by the
documentation on Compress::Zlib, so an additional imnport was
introduced here. Consequently, libio-compress-perl was added as an extra
dependency to the debian control file.
Stoiko Ivanov [Mon, 23 Jan 2023 15:55:20 +0000 (16:55 +0100)]
utils: skip checking headers for non-ascii characters
The fix for smtputf8 enablement in 191e470 ("utils: fix mailflow if
smtputf8 is disabled") was a bit too eager and broke the mail flow of
a few users, who have smtputf8 disabled in their postfix config
(because their downstream servers do not support this):
The issue here is that the mails they process have had
(non-rfc-compliant) non-ascii header contents, which used to work
before the patch. The postfix smtputf8 how-to explains quite well
that postfix never cared too much about headers (or local-parts of
addresses) before smtputf8 [1]
```
Postfix already permitted UTF-8 in message header values and in
address localparts. This does not change.
```
While the patch only ignores the headers and still could cause issues
with non-ascii local-parts, those should occur far less frequently in
the wild (none of the reporters in our forum had this).
Tested with a mail with an Euro sign in a 'X-From:' header.
Note that this is a stop-gap for re-working how to decide if smtputf8
is needed in a manner that better matches real world systems.
Christoph Heiss [Thu, 29 Dec 2022 09:45:17 +0000 (10:45 +0100)]
fix #4410: Remove non-null host bits from CIDR when writing postfix config
This will drop non-null host bits from `mynetworks` CIDRs when writing
the `main.cf` postfix template.
Backwards-compatibility with old entries in `/etc/pmg/mynetworks` is
thus also preserved.
Add an additional comment to the mynetworks API, indicating that unused
fields can/should be dropped with the next PMG version.
No GUI changes. The entries are written to `/etc/pmg/mynetworks` as the
user enters them. Suggested by Stoiko, see discussion in v2 thread [0].
Stoiko Ivanov [Wed, 21 Dec 2022 14:53:43 +0000 (15:53 +0100)]
utils: fix mailflow if smtputf8 is disabled
with the recent addition of smtputf8 support for the rulesystem setups
explicitly disabling smtputf8 in postfix got broken.
This is mostly noticeable for the spamreports (the receivers are taken
from the database and potentially decoded from utf-8, which sets the
'is_utf8' flag, and then tries to use the smtputf8 extension when
reinjecting the mail, which fails (since smtputf8 is disabled)
Instead of checking for the internal flag, we check for occurence of
characters which are not ascii printable (everything excluding
controlcharacters - '[\x20-\x7E]') in the envelope-addresses and
headers (there also for [\r\n\t], due to searching all headers and
folding). - see
https://perldoc.perl.org/perlunifaq#What-is-%22the-UTF8-flag%22? and
https://perldoc.perl.org/perlrecharclass#POSIX-Character-Classes
The only diversion from the requirements in the smptutf8 rfc
https://www.rfc-editor.org/rfc/rfc6531
is that we do not check the headers of all parts of a multipart
message (think suggested filename for an attachment), but I assume
that this should not be an issue in mail-transit
the addresses now always get encoded as UTF-8, as this is robust for
aascii-only addresses.
reported in our community forum:
https://forum.proxmox.com/threads/.119387/
issue is reproducible by setting
`smtputf8_enable = no` in postfix main.cf
and sending a spamreport using `pmgqm`
regular mailflow should not be affected in those setups (as no utf-8
addresses would come into the system)
Stoiko Ivanov [Tue, 20 Dec 2022 10:57:35 +0000 (11:57 +0100)]
rulecache: sort rules additionally by id
When more rules have the same priority currently their order is not
stable - postgres returns them in a stable way, based on their last
changetime - e.g. disabling and reenabling a rule puts it in the front
of evaluation. Sortin by id (the primary key) in addition should make
rule evaluation robust to such updates
While there is no guarantee of ordering (within the same priority)
unexpected changes in which rule fires can cause confusion (at least
it confused me quite a bit).
Thomas Lamprecht [Mon, 12 Dec 2022 12:05:00 +0000 (13:05 +0100)]
backup restore: code style cleanup
find also takes a code ref directly, and the "wanted" name is a bit
non-ideal anyway, as it has no control over what find "wants" (i.e.,
descends into), but its return value is completely ignored.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Stoiko Ivanov [Wed, 30 Nov 2022 17:07:27 +0000 (18:07 +0100)]
backup: restore: keep directories in /etc/pmg for inotify
By wiping the subdirectories in /etc/pmg/, we lose the inotify
watchers upon restore (/etc/pmg itself and thus most configs are
currently handled by the keep_root flag to rmtree)
This can lead to inconsistencies after restoring for parts relying on
config in a subdirectory (e.g. /etc/pmg/pbs/pbs.conf).
This patch uses File::Find (included in perl-modules-$perlver) to keep
all directories an unlink everything else.
This was chosen for future robustness over keeping an explicit list of
directories to keep, in case a new directory gets added.
quickly tested with a fifo, chardev, and socket in the directory.
an alternative approach would be to simply reload pmgdaemon/pmgproxy
upon config-restore, but that feels more likely to miss some
(potentially future) service, expecting inotify to work.
projects / pmg-api.git / log