Stoiko Ivanov [Thu, 25 Nov 2021 17:48:13 +0000 (18:48 +0100)]
fix #2795: add support for DSN
store the esmtp parameters for the MAIL and RCPT command needed to
support Delivery status notifications (DSN - RFC 3464 [0]) and pass
them to the outbound postfix instance (port 10025) used for sending
the mail further (see also [1]).
Postfix does syntax-checking before passing the mail to the proxy
also in before-queue filtering mode.
Since the handling is done by postfix we don't need to generate any
DSN in the regular case.
For mail put into quarantine I decided to skip sending a delivery
notification (on the expectation, that few people are using quarantine
outbound, and that I would not consider a mail put in quarantine as
delivered successfully)
We only store a whitelist of parameters, instead of passing all,
because some parameters might not be valid anymore after processing
(e.g. SIZE)
The DSN EHLO keyword was added for the after-queue filtering case -
else the inbound postfix is the system that sends out the
notification.
tested with various combinations of the -V, -N and -R parameters to
sendmail (e.g.):
```
/usr/sbin/sendmail -N success,delay,failure \
-V '<xxxxxxxx@test.proxmox.com>'\
-R hdrs test@test.domain.example
```
tested the following scenarios in before and after-queue filter mode:
* successful delivery
* successful delivery with set DSN
* failed delivery (recipient rejects with 544)
* failed delivery with DSN
* delivering a mail with empty envelope sender (bounce)
some tests with invalid combinations were also done with netcat.
Stoiko Ivanov [Thu, 25 Nov 2021 17:44:11 +0000 (18:44 +0100)]
partially fix #2795: allow for '>' in smtp parameters
The regular expressions parsing the MAIL and RCPT commands do not
cover the case where a esmtp parameter may contain angle brackets
(e.g. the ENVID parameter for the delivery status notification
extension - RFC3464 [0]).
following section 4.1.2 of RFC5321 [1] the regex is changed to:
* consider everything up to the first '>' the mailbox
* consider everything afterwards (if it starts with a ' ') as
parameters
* since the parameter group might not match (in case no parameters are
set - e.g. after-queue filtering) - default to '' if it's not
defined
This is fairly robusts, only not parsing correctly if the local part
contains '>' (as quoted text) - but this did not work before anyways
(and causes problems in other places as well).
Dominik Csapak [Thu, 25 Nov 2021 14:14:41 +0000 (15:14 +0100)]
fix #3734: scrub 'url' from style tags/attributes
if 'view images' for the quarantine is disabled, it is expected that
*no* images will be loaded. but in addition to img (src/href/etc.)
also css can load external images via the 'url' directive
since html scrubber does not parse/iterate over css, we simply remove
the url+protocol part of those tags/attributes. this technically leaves behind
invalid css, but the browsers should cope with that.
(we cannot 'cleanly' remove without much more effort because of quoting)
also we have to scrub the style tags in 'dump_html' since HTML::Scrubber
does not have a way to modify the *content* of a tag, only the
attributes...
Stoiko Ivanov [Wed, 24 Nov 2021 21:00:48 +0000 (22:00 +0100)]
rulesystem: limit linelength of disclaimer to 998 bytes
As described in
http://www.postfix.org/postconf.5.html#smtp_line_length_limit
postfix splits lines which are longer by inserting <cr><lf><space> to
adhere with RFC 5322 (section 2.1.1):
https://datatracker.ietf.org/doc/html/rfc5322#section-2.1.1
(or actually section 4.5.3.1.6. where characters are translated to
octets)
If a longer line is part of the disclaimer pmg-smtp-filter adds it
without this modification, which breaks DKIM signatures (since the
body is modified by postfix after the body hash is computed)
regular-expression matching is used instead of length(), because the
limit is on line-length (and a disclaimer can contain multiple lines)
reported in our community forum:
https://forum.proxmox.com/threads/.97919/
Stoiko Ivanov [Wed, 24 Nov 2021 16:04:09 +0000 (17:04 +0100)]
api-daemons: set oom-policy to continue
OOMPolicy [0] defaults to stop - resulting in the complete daemon to
be killed.
Our Daemon class does start new workers automatically if it detects
that fewer than configured are running.
Dominik Csapak [Wed, 24 Nov 2021 14:48:52 +0000 (15:48 +0100)]
api: journal: stream the journal data to the client
instead of accumulating the whole output of 'mini-journalreader' in
the api call (this can be quite big), use the download mechanic of the
http-server to stream the output to the client.
we lose some error handling possibilities, but we do not have
to allocate anything here, and since perl does not free memory after
allocating[0] this is our desired behaviour.
to keep api compatiblitiy, we need to give the journalreader the '-j'
flag to let it output json.
also tell the http server that the encoding is gzip and pipe
the output through it.
Stoiko Ivanov [Mon, 22 Nov 2021 19:49:39 +0000 (20:49 +0100)]
fix #3712: strip trailing dot from searchdomain
having a trailing '.' in the search domain is perfectly legal syntax
(for domain names in general). postfix refuses to use a fqdn with
trailing dot as hostname[0].
The restriction might be due to section 2.3.5 (Domain Names) of
RFC5321 (a top-level domain is a single string without any dots) [1]
[0] src/util/valid_hostname.c in the postfix source
[1] https://datatracker.ietf.org/doc/html/rfc5321#section-2.3.5
Thomas Lamprecht [Mon, 20 Sep 2021 06:52:28 +0000 (08:52 +0200)]
services: add drop weird binary-exists condition
The package that ships the service is the same as the one that ships
the binaries, so quite the useless check and a remainder from initial
switch from sysv to systemd in ~2015 (when it was not 100% clear
what/how systemd features should be integrated or units encoded).
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Thomas Lamprecht [Mon, 20 Sep 2021 06:50:46 +0000 (08:50 +0200)]
services: add After=network-online.target and update
while we indirectly got that by the remote-fs ordering constraint its
better to encode it explicitly, especially as the remote-fs does not
make much sense and may get removed soon
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Thomas Lamprecht [Mon, 20 Sep 2021 06:38:08 +0000 (08:38 +0200)]
config: fix "var declared in conditional statement"
This is actually buggy and can lead to unexpected issues as in the
case the check on the declared variable did not evaluates to true it
gets (or better keeps) the value from the previous time when it was
actually assigned. Found with perlcritic, which reports the highest
severity for this mistake.
Refactor out the "is current file equal to generated config" check
which fixes three instances of that on its own and reduces code bloat
a bit.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Thomas Lamprecht [Sat, 18 Sep 2021 13:17:21 +0000 (15:17 +0200)]
pmgbanner: retry getting local ip for a bit
basically only useful for setups using (hopefully static) DHCP for
the PMG host, but we can have that in evaluation, especially when
using CTs or installing on top of a plain Debian.
This was favored over adding an After=network-online.target order
constraint for the pmgbanner service, as it'd delay the console-getty
service needlessly in most setups
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Thomas Lamprecht [Sat, 18 Sep 2021 13:06:57 +0000 (15:06 +0200)]
prefer more flexible get_local_ip where possible
get_ip_from_hostname does only check getaddrinfo, which can fail for
the local node in some environments, especially container ones.
Rather, use the new get_local_ip helper, that still tries to do a gai
call first, but falls back to configured (/etc/network/interfaces)
IPs and also on the currently, from kernel POV active ones.
A big bonus is that the new helper is much less likely to die, so it
won't break service startup in restricted (CTs) envs after initial
setup as often anymore.
While yes, if no addr is resolved, configured or active the PMG won't
work, but killing pmg proxy/daemon won't better that situation either
;)
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Oguz Bektas [Thu, 2 Sep 2021 08:47:12 +0000 (10:47 +0200)]
api: apt: use actual pmg-style permission for endpoint schema
it wrongly uses the permission model from PVE, which caused the
endpoints to be root-only as a side effect, since PMG API doesn't
recognize the PVE-specific permissions.
fix those to allow PMG users with administrator role to add/delete
repositories, and auditor role to view the repositories.
Thomas Lamprecht [Fri, 16 Jul 2021 10:27:01 +0000 (12:27 +0200)]
api: implement live network reload with ifupdown2
Like most of the other call here, copied over from PVE, with the SDN
stuff dropped and some task-log feedback if we actually moved a
pending change in. Also adding error handling for the rename, both
should be added to PVE too.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
ssh public keys are base64 encoded, thus can potentially contain =.
until now the RSA keys generated by Debian were 2048 bits long and did
not need padding
with bullseye (openssh (1:8.0p1-1)) the RSA keysize got increased to
3072 bits, and now does contain a =
noticed while trying to join a PMG container from a bullseye template
to my existing cluster (the error happens on the new node).
Stoiko Ivanov [Wed, 30 Jun 2021 16:39:55 +0000 (18:39 +0200)]
config: freshclam: default to incremental downloads
clamav recently started yielding 429 (too many requests) response
codes on even comparatively low attempts to download the complete
signature files (cvd)(see [0]), instead of the incremental changes
(cdiff) (see [1] for some background)
changing the default to scriptedupdates (a.k.a. cdiff download) seems
sensible for most situations.
Stoiko Ivanov [Wed, 30 Jun 2021 15:42:57 +0000 (17:42 +0200)]
cluster: fix missing import
The missing use PMG::Ticket import is problematic during ACME cert
renewal from pmg-daily->PMG::API2::Certificates->renew_acme_cert,
since pmg-daily does not import it.
Reported-by: Martin Maurer <martin@proxmox.com> Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
Stoiko Ivanov [Mon, 17 May 2021 14:02:57 +0000 (16:02 +0200)]
fix #2013 spamreport: remove ticket if authmode is ldap
Currently the 'authmode' setting for the spamquarantine is not used
anywhere. According to documentation setting it to 'ldap' should allow
access to the quarantine only with ldap credentials.
This patch addresses the issue by not generating a quarantineticket,
and adapting all links accordingly if the authmode is 'ldap'.
tested by changing the authmode and running
`pmgqm send -receiver <email-address> -debug 1`
Stoiko Ivanov [Wed, 16 Jun 2021 18:36:40 +0000 (20:36 +0200)]
api: nodeconfig: validate acme config before writing
Currently it is possible to add the same domains as different
acmedomainX keys to the node config, which prevents the user from
ordering certificates later.
This patch adds a call to get_acme_conf, which does the semantic
validation (and is also used in all other sites, which read the
config).
Reported in our community forum:
https://forum.proxmox.com/threads/lets-encrypt-cert-on-gui-not-working.91014/
quickly tested in my setup, by successfully adding the same domain
twice without the patch, and failing to do so with it applied.
Thomas Lamprecht [Mon, 28 Jun 2021 12:15:28 +0000 (14:15 +0200)]
d/control: drop transitional apt-transport-https, provided by apt
It was actually integrated into apt quite a bit before version 2.0
but it does not really hurts and version 2 is available since Q1 2020
on sid, bullseye will have 2.2.x so using (>= 2~) is just fine.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Stoiko Ivanov [Fri, 11 Jun 2021 15:54:46 +0000 (17:54 +0200)]
greylisting: drop unneeded Host column form cgreylist table
With the changes added in f61d54891d4820b21ef9e53f7ce0ebb1d5be1f73
greylisting does the matches based on a configurable netmask, and
does not use the 'Host' column in the cgreylist table anymore.
Drop it now with PMG 7.0
Quickly tested the following scenarios (all successfully):
* Upgrading from a previous version
* Restoring a pmg-backup taken with PMG 5.2 (the greylist table is
excluded from the backup)
* Adding a node with the changes to an existing cluster without the
change
* Adding a node without the changes to a master-node having them
Stoiko Ivanov [Mon, 31 May 2021 13:53:05 +0000 (13:53 +0000)]
utils: do not hardcode postgres version
PMG::Utils::lookup_real_service_name is only called
for translating the service names provided as arguments
to PMG::API2::Nodes::syslog (for fetching the journal
for specific units). Instead of hardcoding the
version getting it with a call to `psql` seems justified.
Stoiko Ivanov [Tue, 8 Jun 2021 17:25:29 +0000 (17:25 +0000)]
api: statistics: drop deprecated detail statistic methods
in e89b61c5190e3e374c2c3bcb3dce444c64c718cf we introduced a method
taking the address as explicit parameter instead of path component
(local-parts can contain '/'). now we can drop the old paths.
Stoiko Ivanov [Tue, 8 Jun 2021 16:06:50 +0000 (16:06 +0000)]
api: nodes: drop deprecated 'upgrade' option of termproxy
The termproxy api was adapted to the changes from PVE and PBS
in d9e79ff4b7f0f9b2c49f06484091546353980c5e
We can now drop the 'upgrade' option kept for backwards compatibility