Dominik Csapak [Thu, 10 Oct 2019 09:21:57 +0000 (11:21 +0200)]
RuleDB/Remove: improve attachment detection for 'remove all'
this patch improves the attachment detection in such a way that
instead of leaving the first part if it has a content-type of
text/* and removing all other parts we now leave the 'real' message, meaning:
* the first text/plain or text/html part if no multipart
* the first text/plain or text/html part if multipart/mixed
* all text/plain and text/html parts of the first multipart/alternative
(if we did not already see the message)
this way the 'real' message including the text/html alternative
(or the only text/html part if no text/plain was sent) is
left in the message that is being sent to the receiver
also add missing '$rulename' parameter to the recursive call to
'delete_marked_parts'
Dominik Csapak [Thu, 10 Oct 2019 09:21:52 +0000 (11:21 +0200)]
HTTPServer: extend download functionality
allow object as return value for 'download' api calls
this way, we can give additional information for download apis
(like the content-type and if it should be deleted after)
While perl5 returns the result of the last expression implicitly, if
no explicit return statement is in a submethod's code path, it's much
nicer to do the things explicit most of the times, and a slight
modification could even break this.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
fix #2360: cluster: reload pmg-smtp-filter on rulechange
When a node detects a change of the rule-database during clustersync,
the database update got done, however the running `pmg-smtp-filter`
instance was not notified of the updated ruleset. Adding a call to
PMG::DBTools::reload_ruledb (like in the API2 paths) fixes the issue.
As this is guarded by a digest change check it'll only reload if the
ruleDB really changed.
We do not pass the currewnt ruleDB instance to the reload method, as
it is only used for "rewrite_postfix_whitelist", which is already
executed through the pmgmirror::cluster_sync call to
PMG::Config->rewrite_config, if needed, so don't do that twice.
Tested by
* creating a 2 node PMG cluster
* adding and removing e-mail-addresses to the Blacklist (which was used in
a high priority, active rule)
* sending e-mails from those addresses through PMG
* observing the action taken before/after the Rule DB got synced
Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com> Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
This is a fix for a small cosmetic issue, which keeps coming up on our various
support-channels:
pmgpolicy as user of 'Mail::SPF::Server' logs lines referencing the (now
defunct) url: http://www.openspf.org/Why
By overriding 'default_authority_explanation' [0] the logs now contain a simple
logmessage without any reference to an external url.
All macros used before are used in the new message as well, so the information
content stays the same.
Given that we are (by far) not the only users of this module I additionally
opened an issue upstream [1] (and linked it to the debian bug-report [2]).
Tested by providing a SPF policy not allowing the ip of the sender in my
testsetup and observing the logs.
Stoiko Ivanov [Mon, 19 Aug 2019 16:32:14 +0000 (18:32 +0200)]
close #2324 - improve docs on encrypted archives
The option for ClamAV 'Block Encrypted Archives and Documents', actually
triggers a Heuristic match from ClamAV, which is used by PMG to rais the
spam score of a message by the value of 'Heuristic Score' configured
at the 'Spam Detector' settings.
Since this has caused confusion (also for myself) a few times already, this
patch tries to address the issue by referring to the other option in the API
description of both properties
and should enable us to update pve-common in PMG 5.x along with PVE again.
Tested by:
* logging into PMG (get csrf token)
a) installing only the previous patch:
* POSTing (with the previously obtained token) -> 401
b) installing previous and this patch
* POSTing (with the previously obtained token) -> 200
Stoiko Ivanov [Fri, 9 Aug 2019 07:07:30 +0000 (09:07 +0200)]
adapt journalctl invocation to buster
With Debian Buster the behavior of `journalctl` has changed when it finds
no entries for a given selection:
* The exit code was 0 in stretch, but is 1 in buster
* The output changed slightly - a header got added
Since PMG::Utils::scan_journal_for_rbl_rejects uses journalctl for reading the
IPs blocked by postscreen it needs to adapt for the new behavior (otherwise
run_command dies because of the exit code 1)
The patch addresses the problem by using the json-output of `journalctl`, which
still exits with 0 if no entries are present . Additionally the json-output
adds the current cursor to the output by default, removing the need to
explicitly scanning for it.(the exit code of 1 was due to '--show-cursor'
without a single result line).
Stoiko Ivanov [Thu, 8 Aug 2019 16:09:22 +0000 (18:09 +0200)]
adapt clamav.conf.in to new upstream version
The new version of clamav has changed a few options, which we ship in our
config template - see [0] for the details.
This patch changes all affected options to the new names and additionally
expands the Description of 'archiveblockencrypted' to reflect that the option
is used for both archives _and_ documents.
Stoiko Ivanov [Thu, 8 Aug 2019 13:36:11 +0000 (15:36 +0200)]
adapt postgresql.conf template to variable version
pass the postgres major version in template_vars->{postgres}->{version} and
use that in the template. Additionally use the version for the config file
location.
The start and end tags are changed to '<*' and '*>' since the original
postgresql.conf contains an occurence of the default start tag '[%' which
would need to get escaped
Stoiko Ivanov [Thu, 8 Aug 2019 13:36:10 +0000 (15:36 +0200)]
add get_pg_server_version in PMG::Utils
PMG renders the postgresql.conf through its templating system (currently the
shipped template does not use any variables). postgresql.conf (in most
installations and in both debian and upstream packages) contains a few
occurrences (datadir, config files, pid-file, cluster name) of the postgres
major version number (see [0], for a description and why 9.6 and 11 are major
version numbers). The rendered config should use the correct version number
for the config of the currently used postgres installation (the one listening
on the default port (5432) and socket).
This fixes a bug observed while testing the upgrade to buster and postgres 11:
* a long running service (pmgmirror, pmgdaemon) still has the old config
path in memory (/etc/postgresql/9.6/)
* while upgrading the pmg-api package the shipped template changes to one
with the new major number (11)
* the next restart of the postgresql cluster fails, with an error not directly
related to the broken config file
By reading [1] the version number through a connection to the current
postgresql server we rewrite the fitting configfile with the correct paths.
Stoiko Ivanov [Thu, 8 Aug 2019 13:36:09 +0000 (15:36 +0200)]
move postgres_admin_command into PMG::Utils
Take postgres_admin_command from PMG::DBTools and put it in PMG::Utils.
This avoids a cyclic dependency (PMG::DBTools calls
PMG::Config::rewrite_postfix_whitelist when reloading the RuleDB, and
PMG::Config needs access to the Postgres major version for rendering
the postgresql.conf)
fix #2276: restore line format for pmg-log-tracker
pmg-log-tracker requires a specific format of the output of the
smtp-filter to correctly detect and recognize emails and message ids
commit 365d5b9549d25a910c82cd37034f05e1c906565a
changed the format of some lines (by including the rule name)
so that pmg-log-tacker did not correctly parse it anymore
this patch changes the format in a way that the log-tracker
can parse and still display the new information
Stoiko Ivanov [Fri, 17 May 2019 12:45:45 +0000 (14:45 +0200)]
avast: change 'scan' invocation
Change the invocation of avast's 'scan' executable from a hardcoded '/bin/scan'
to 'scan', so that it checks for the executable in the PATH.
The hardcoded path became apparent, while testing the new upstream release
of avast (3.0.1), where 'scan' got moved from '/bin/scan' to '/usr/bin/scan'
Mira Limbeck [Tue, 11 Jun 2019 13:27:28 +0000 (15:27 +0200)]
fall back to hostname only if no domain defined
fall back to hostname only if no domain is defined in /etc/resolv.conf.
this removes the 'Use of uninitialized value' warning for
'$msginfo->{domain}'.
Signed-off-by: Mira Limbeck <m.limbeck@proxmox.com>
Alexander Plank [Thu, 6 Jun 2019 12:54:56 +0000 (14:54 +0200)]
extended fix #1974: traffic_stat_graph: go through all entries
Extends a fix for #1974.
The commit cb609ca098823734dde590fcf42164f72bbfbf37 fixed the graph
for postscreen_stat_graph. The code from the above commit was
adopted to fix the same error for traffic_stat_graph.
Signed-off-by: Alexander Plank <alexander.plank@siconnex.com> Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Stoiko Ivanov [Thu, 6 Jun 2019 09:26:37 +0000 (11:26 +0200)]
fix #2232: set rulename for default accept
Logging the rulename along with the action taken introduced in 365d5b9549d25a910c82cd37034f05e1c906565a, introduced a regression, for the
default action (accept), since the accept-rule is instantiated directly it
did not set a name, resulting in an 'Use of uninitialized value' being written
to the mail.log for every mail not triggering any explicit rule.
Stoiko Ivanov [Tue, 28 May 2019 10:32:57 +0000 (12:32 +0200)]
Drop sa-awl output from pmg-system-report
After gathering some initial experience with `pmg-system-report` it seems
that the output of `sa-awl` is not too helpful in narrowing down problems.
Additionally the output tends to be quite large (multiple MiB) on larger/
longer-running installations, leading to timeouts or problems when sending
the report to the support.
Should the AWL checks skew the SA-results in a negative way this already shows
in the logs (which log the score for each rule).
Stoiko Ivanov [Wed, 22 May 2019 15:31:52 +0000 (17:31 +0200)]
limit precision of bayes-score in log
Spamassassin's bayes_score is a float, and is written to the log during
filtering. Limiting the precision for the log to 2 decimal places keeps
logs a bit shorter and also prevents misreadings of values like:
5.55111571207834e-17
With complicated rulesets knowing which rule is responsible for the action
applied to a mail can become complex. Since relevant actions [0] do log a line
when being executed adding the rule's name to this logline should simplify
understanding and debugging complex rules.
Additionally the mix of string interpolation and formatstrings in Quarantine.pm
got unified to formatstrings.
[0] Attach is covered via Notify, Disclaimer and ModField could result in too
verbose logging (i.e. users adding many headers to a mail) without any gain
PMG::DBTools::postgres_admin_cmd switches the euid to postgres. The error
handling expected that the setresuid (2) call failed if $! was != 0, without
explicitly setting it to 0 beforehand. This lead to a false positive if errno
was set from a previous library call.
This patch changes the code to explicitly call the setresuid syscall (exposed
via a separate patch to pve-common) and check for an error.
Steps to reproduce:
* install nscd on a system
* try installing pmg-api (the postinst script invokes `pmgdb init`)
PMG::Config::rewrite_config is called from various places (e.g.
pmgmirror for clustered setups, pmgconfig sync --restart for CLI
operations) for rendering the config-templates and conditionally
restarting services. This patch adds a syslog call for each service
that gets restarted.
fix #2172: sort mynetworks template var to make postfix config rewrite stable
The mynetworks template_var is written to postfix/main.cf, causing a
postfix restart on every change. Since mynetworks is a hash the order
of the networks potentially changes with every invocation. This shows
quite readily in clustered setups where pmgmirror writes the configs
and checks for changes once every 2 minutes.
reload postfix instead of restart on config change
From `man 1 postfix`:
```
Note: in order to refresh the Postfix mail system after a
configuration change, do not use the start and stop commands in
succession. Use the reload command instead.
```
Additionally restarting postfix, while a mail has been passed to
pmg-smtp-filter but has not beed fed back to postfix again, causes it
to get passed a second time to pmg-smtp-filter (by qmgr) and this
results in duplicate maildelivery
Thomas Lamprecht [Tue, 19 Mar 2019 07:04:08 +0000 (08:04 +0100)]
pmgversion: be compatible with minimized container installation
One can now install PMG in a CT envrionment through the new
proxmox-mailgateway-container meta package, but this wasn't correctly
checked by the API PMG versions code.
Add the new meta package as optional one, and if it is installed
replace the bare-metal meta packge info with it, both can never be
installed at the same time. Also move pve-firmware to the optional
package list.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Thomas Lamprecht [Tue, 19 Mar 2019 07:04:07 +0000 (08:04 +0100)]
pmgversion: sort packages
Similar adaptions as PVE got, initally list the most important
packages, pmg-api and pmg-gui, then the kernels sorted by real
version order, then the rest of intresting packages in alphabetical
order.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Dominik Csapak [Mon, 18 Mar 2019 13:58:32 +0000 (14:58 +0100)]
better error handling for ldap connect
even though we create the ldap connection with 'onerror' => 'die',
it returns undef and sets $@ on error during connect, so we want to use
that instead of $!
Stoiko Ivanov [Thu, 14 Mar 2019 17:20:45 +0000 (18:20 +0100)]
fix #2129: allow (some) filters for dnsbl-entry
currently we only handle dnsbl-sites with optional <WEIGHT>, but postfix also
allows for an optional <FILTER> (which dns-answers to interpret as hit) [0].
The regex is extended to also allow for a filter with singular answers, as
well as ranges ([0..255])for each octet. Filters relying on 'lists' of numbers
split by ';' break the use of JSONSchema's '-list' format matching (it
uses split_list, which splits on ';') and were thus excluded.
Stoiko Ivanov [Wed, 13 Mar 2019 20:39:41 +0000 (21:39 +0100)]
add custom_check handling
This patch enables users to create their own script for analyzing mails.
The 'custom_check' needs to be enabled via pmg.conf (optionally the check's
executable path ('custom_check_path') can be set, defaulting to
'/usr/local/bin/pmg-custom-check').
'pmg-smtp-filter' calls the check before analyze_virus (which in turn calls
clamav or avast). The custom_check 'api' is kept simple:
* Input: the check gets 2 arguments:
* the 'api-version' (currently 'v1') - for potential future change of the
invocation
* the 'queue-file-name' - a filename, which contains the complete e-mail as
rfc822/eml file
* Output: the check needs to return 2 lines on STDOUT:
* the 'api-version' (currently 'v1') - see above
* one of the following 3 results:
* 'OK' - mail is ok
* 'VIRUS: <virusdescription>' - mail is treated as if it contained a virus
(the virusdescription is logged and added to the mail's headers)
* 'SCORE: <number>' - <number> is added (negative numbers are also possible)
to the mail's spamscore
* The check will be killed after a 5 minute timeout - and the mail is
treated as OK
* All output written to STDERR by the check is written to the journal/mail.log
(with priority 'err')
Stoiko Ivanov [Tue, 26 Feb 2019 14:02:30 +0000 (15:02 +0100)]
pmg-system-report: check for existing sa-awl db
When run via pmgdaemon the pmg-system-report threw an error, while running
`sa-awl` (it relies on the environment variable HOME being set, for the default
location of the awl database).
This patch checks for the existance of /root/.spamassassin/auto-whitelist and
conditionally runs sa-awl with the file as argument.
projects / pmg-api.git / log