fix #2276: restore line format for pmg-log-tracker
pmg-log-tracker requires a specific format of the output of the
smtp-filter to correctly detect and recognize emails and message ids
commit 365d5b9549d25a910c82cd37034f05e1c906565a
changed the format of some lines (by including the rule name)
so that pmg-log-tacker did not correctly parse it anymore
this patch changes the format in a way that the log-tracker
can parse and still display the new information
Stoiko Ivanov [Fri, 17 May 2019 12:45:45 +0000 (14:45 +0200)]
avast: change 'scan' invocation
Change the invocation of avast's 'scan' executable from a hardcoded '/bin/scan'
to 'scan', so that it checks for the executable in the PATH.
The hardcoded path became apparent, while testing the new upstream release
of avast (3.0.1), where 'scan' got moved from '/bin/scan' to '/usr/bin/scan'
Mira Limbeck [Tue, 11 Jun 2019 13:27:28 +0000 (15:27 +0200)]
fall back to hostname only if no domain defined
fall back to hostname only if no domain is defined in /etc/resolv.conf.
this removes the 'Use of uninitialized value' warning for
'$msginfo->{domain}'.
Signed-off-by: Mira Limbeck <m.limbeck@proxmox.com>
Alexander Plank [Thu, 6 Jun 2019 12:54:56 +0000 (14:54 +0200)]
extended fix #1974: traffic_stat_graph: go through all entries
Extends a fix for #1974.
The commit cb609ca098823734dde590fcf42164f72bbfbf37 fixed the graph
for postscreen_stat_graph. The code from the above commit was
adopted to fix the same error for traffic_stat_graph.
Signed-off-by: Alexander Plank <alexander.plank@siconnex.com> Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Stoiko Ivanov [Thu, 6 Jun 2019 09:26:37 +0000 (11:26 +0200)]
fix #2232: set rulename for default accept
Logging the rulename along with the action taken introduced in 365d5b9549d25a910c82cd37034f05e1c906565a, introduced a regression, for the
default action (accept), since the accept-rule is instantiated directly it
did not set a name, resulting in an 'Use of uninitialized value' being written
to the mail.log for every mail not triggering any explicit rule.
Stoiko Ivanov [Tue, 28 May 2019 10:32:57 +0000 (12:32 +0200)]
Drop sa-awl output from pmg-system-report
After gathering some initial experience with `pmg-system-report` it seems
that the output of `sa-awl` is not too helpful in narrowing down problems.
Additionally the output tends to be quite large (multiple MiB) on larger/
longer-running installations, leading to timeouts or problems when sending
the report to the support.
Should the AWL checks skew the SA-results in a negative way this already shows
in the logs (which log the score for each rule).
Stoiko Ivanov [Wed, 22 May 2019 15:31:52 +0000 (17:31 +0200)]
limit precision of bayes-score in log
Spamassassin's bayes_score is a float, and is written to the log during
filtering. Limiting the precision for the log to 2 decimal places keeps
logs a bit shorter and also prevents misreadings of values like:
5.55111571207834e-17
With complicated rulesets knowing which rule is responsible for the action
applied to a mail can become complex. Since relevant actions [0] do log a line
when being executed adding the rule's name to this logline should simplify
understanding and debugging complex rules.
Additionally the mix of string interpolation and formatstrings in Quarantine.pm
got unified to formatstrings.
[0] Attach is covered via Notify, Disclaimer and ModField could result in too
verbose logging (i.e. users adding many headers to a mail) without any gain
PMG::DBTools::postgres_admin_cmd switches the euid to postgres. The error
handling expected that the setresuid (2) call failed if $! was != 0, without
explicitly setting it to 0 beforehand. This lead to a false positive if errno
was set from a previous library call.
This patch changes the code to explicitly call the setresuid syscall (exposed
via a separate patch to pve-common) and check for an error.
Steps to reproduce:
* install nscd on a system
* try installing pmg-api (the postinst script invokes `pmgdb init`)
PMG::Config::rewrite_config is called from various places (e.g.
pmgmirror for clustered setups, pmgconfig sync --restart for CLI
operations) for rendering the config-templates and conditionally
restarting services. This patch adds a syslog call for each service
that gets restarted.
fix #2172: sort mynetworks template var to make postfix config rewrite stable
The mynetworks template_var is written to postfix/main.cf, causing a
postfix restart on every change. Since mynetworks is a hash the order
of the networks potentially changes with every invocation. This shows
quite readily in clustered setups where pmgmirror writes the configs
and checks for changes once every 2 minutes.
reload postfix instead of restart on config change
From `man 1 postfix`:
```
Note: in order to refresh the Postfix mail system after a
configuration change, do not use the start and stop commands in
succession. Use the reload command instead.
```
Additionally restarting postfix, while a mail has been passed to
pmg-smtp-filter but has not beed fed back to postfix again, causes it
to get passed a second time to pmg-smtp-filter (by qmgr) and this
results in duplicate maildelivery
Thomas Lamprecht [Tue, 19 Mar 2019 07:04:08 +0000 (08:04 +0100)]
pmgversion: be compatible with minimized container installation
One can now install PMG in a CT envrionment through the new
proxmox-mailgateway-container meta package, but this wasn't correctly
checked by the API PMG versions code.
Add the new meta package as optional one, and if it is installed
replace the bare-metal meta packge info with it, both can never be
installed at the same time. Also move pve-firmware to the optional
package list.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Thomas Lamprecht [Tue, 19 Mar 2019 07:04:07 +0000 (08:04 +0100)]
pmgversion: sort packages
Similar adaptions as PVE got, initally list the most important
packages, pmg-api and pmg-gui, then the kernels sorted by real
version order, then the rest of intresting packages in alphabetical
order.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Dominik Csapak [Mon, 18 Mar 2019 13:58:32 +0000 (14:58 +0100)]
better error handling for ldap connect
even though we create the ldap connection with 'onerror' => 'die',
it returns undef and sets $@ on error during connect, so we want to use
that instead of $!
Stoiko Ivanov [Thu, 14 Mar 2019 17:20:45 +0000 (18:20 +0100)]
fix #2129: allow (some) filters for dnsbl-entry
currently we only handle dnsbl-sites with optional <WEIGHT>, but postfix also
allows for an optional <FILTER> (which dns-answers to interpret as hit) [0].
The regex is extended to also allow for a filter with singular answers, as
well as ranges ([0..255])for each octet. Filters relying on 'lists' of numbers
split by ';' break the use of JSONSchema's '-list' format matching (it
uses split_list, which splits on ';') and were thus excluded.
Stoiko Ivanov [Wed, 13 Mar 2019 20:39:41 +0000 (21:39 +0100)]
add custom_check handling
This patch enables users to create their own script for analyzing mails.
The 'custom_check' needs to be enabled via pmg.conf (optionally the check's
executable path ('custom_check_path') can be set, defaulting to
'/usr/local/bin/pmg-custom-check').
'pmg-smtp-filter' calls the check before analyze_virus (which in turn calls
clamav or avast). The custom_check 'api' is kept simple:
* Input: the check gets 2 arguments:
* the 'api-version' (currently 'v1') - for potential future change of the
invocation
* the 'queue-file-name' - a filename, which contains the complete e-mail as
rfc822/eml file
* Output: the check needs to return 2 lines on STDOUT:
* the 'api-version' (currently 'v1') - see above
* one of the following 3 results:
* 'OK' - mail is ok
* 'VIRUS: <virusdescription>' - mail is treated as if it contained a virus
(the virusdescription is logged and added to the mail's headers)
* 'SCORE: <number>' - <number> is added (negative numbers are also possible)
to the mail's spamscore
* The check will be killed after a 5 minute timeout - and the mail is
treated as OK
* All output written to STDERR by the check is written to the journal/mail.log
(with priority 'err')
Stoiko Ivanov [Tue, 26 Feb 2019 14:02:30 +0000 (15:02 +0100)]
pmg-system-report: check for existing sa-awl db
When run via pmgdaemon the pmg-system-report threw an error, while running
`sa-awl` (it relies on the environment variable HOME being set, for the default
location of the awl database).
This patch checks for the existance of /root/.spamassassin/auto-whitelist and
conditionally runs sa-awl with the file as argument.
Dominik Csapak [Mon, 25 Feb 2019 09:52:22 +0000 (10:52 +0100)]
Quarantine: reuse raw parameter for non htmlmail formatter
when we download a mail, we want the raw, unmodified header
and content in full size, so we reuse the raw parameter for
json/extjs formatter to get the full email, not only the first 4k
Dominik Csapak [Mon, 18 Feb 2019 16:12:10 +0000 (17:12 +0100)]
improve pmg-email-address regex
the '|' is not necessary since the first option is empty (mitigated by
the minLength of 3) and add the '\' to forbidden characters since
they make problems with browser requests (browsers convert '\' to '/')
Stoiko Ivanov [Mon, 11 Feb 2019 14:43:07 +0000 (15:43 +0100)]
extend `pmgdb dump`
add priority, direction and an active flag to `pmgdb dump` output.
pmgdb dump provides the complete ruleset including all rules and related
objects. The information whether a rule is active and in which direction it
works is necessary to get an overview about the setup for pmg-smtp-filter.
Additionally the priority was explicitly added to the output for easier matching
with the GUI.
Stoiko Ivanov [Mon, 11 Feb 2019 14:43:06 +0000 (15:43 +0100)]
close #1917: add pmg-system-report command
pmg-system-report gathers information about a PMG installation, like
pvereport does for PVE.
The name was chosen because pmgreport is already taken (for the daily
reportmails).
The DNS resolution check uses SpamAssassin's internal DnsResolver, since
SpamAssassin has a few pecularities, e.g. only using the first entry in
/etc/resolv.conf - see [0] and Mail::SpamAssassin::DnsResolver and
spam-detection is abysmal if SpamAssassin cannot resolve RBL-entries.
The SpamAssassin initialization is taken from pmg-smtp-filter (except that
local_tests_only is unconditionally disabled (otherwise it would not do DNS
Resolution).
Stoiko Ivanov [Fri, 8 Feb 2019 10:11:49 +0000 (11:11 +0100)]
add rule's score to pmg-smtp-filter logline
We already log which Spamassassin rules apply to a mail. Given that the scores
depend on configuration and setup (e.g. AWL) writing them in the log provides
a quick overview of Spamassassin performance, and spares admins and support from
having to gather the complete mail just for assessing Spamassessin.
fix #1974: postscreen_stat_graph: go through all entries
When the GUI requests the values for a whole month
containing a DST switch it will request a range a little
longer or shorter than a month, eg. 31.04166 days for
October 2018 in CET.
Since we use integer math to calculate the number of entries
we expect, the database then returns one more value than
expected, and we forget to fill in the last time value.
For example, requesting Oct. 2018 from CET causes the
equivalent of this query:
# pmgsh get /statistics/rejectcount --starttime=1538344800 --endtime=$[1541026800] --timespan=86400
400 Result verification failed
[31].time: property is missing and it is not optional
This also happens when for example taking the working range
for the month and simply subtracting 1 second from the
end-time. Our division will then round down by a day while
the database timestamps still cause that day to be included
in the result.
Dominik Csapak [Wed, 7 Nov 2018 14:47:56 +0000 (15:47 +0100)]
fix #1978: always give encoding/collate explicitly when creating db
already existing clusters still have the wrong encoding,
so if a user has a problem with it, they have to either recreate
the slave db with pmgdb delete && pmgdb init,
or remove the slave and add it again after this patch
add pmg_verify_tls_policy_strict and use it in API
This patch splits the parsing of tls_policies in 2 parts:
While reading we just require a line to start with one of the valid tls_policies,
while writing we only accept one of the policies w/o any attributes.
This should help users, who already have a manually crafted file in place, to
use API-calls for adding/modifying entries.
to handle /etc/pmg/tls_policy via API, using PMG::API2::Transport
as base/inspiration.
This enables PMG to enforce TLS on a per-domain basis.
See http://www.postfix.org/postconf.5.html#smtp_tls_policy_maps and
http://www.postfix.org/TLS_README.html#client_tls_policy for reference.
fix #1876: allow node status for admin/manager/auditors
users can already see the status for the local host
via /config/cluster/status or rrddata on all nodes
so allow them to directly get the status via /nodes/nodename/status
this fixes the permission error on the dashboard in clustered setups
projects / pmg-api.git / log