Stoiko Ivanov [Wed, 5 May 2021 14:36:24 +0000 (16:36 +0200)]
daemon: add compat code for pmgproxy 6.x
The changes to the listening behavior introduced with PVE 6.4 break
backwardscompatibility w.r.t. listening address and logging, which
should not be changed without explictly notifying the user.
This patch re-adds the family parameter, which is still used by
pmgproxy and based on its existence creates the socket as before.
compared to the IO::Socket::IP->new call used before 390fc10dc4a696dd30646cbdd018ad08d855175f, the only change is the
renaming of 'LocalAddr' to 'LocalHost' (which are synonymous in
IO::Socket::IP [0])
It can simply be reverted with the release of pmg-api 7.0 (where
we'll record the change in the release-notes and upgrade-page)
Stoiko Ivanov [Wed, 5 May 2021 14:36:23 +0000 (16:36 +0200)]
daemon: explicitly bind to wildcard address.
with the recent change in pve-manager pveproxy (and spiceproxy)
try binding to '::' per default. This fails for hosts having disabled
ipv6 via kernel commandline.
Our desired behavior of binding on '::' and only falling back to
'0.0.0.0' in case this is not supported is not directly possible with
IO::Socket::IP->new (or rather by Socket::GetAddrInfo, which at least
on my system always returns the v4 wildcard-address first).
the code now binds to:
* the provided $host if not undef
* '::' if $host is not set
* '0.0.0.0' if $host is not set and binding on '::' yields undef,
which means that it failed to create a socket which normally means
that IPv6 is disabled
Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com> Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Stoiko Ivanov [Wed, 5 May 2021 14:36:22 +0000 (16:36 +0200)]
daemon: drop Domain parameter from create_reusable_socket
The Domain parameter for IO::Socket::IP is not used/needed.
It is needed to create a IP Socket when calling IO::Socket->new,
but here we call IO::Socket::IP-new directly (see [0]).
Fabian Ebner [Thu, 18 Mar 2021 08:44:18 +0000 (09:44 +0100)]
schema: check format: parse list formats as arrays
Previously, the returned value would be only the last element or undef in case
of an empty list. There's only a handful of callers of check_format() that look
at the return value and AFAICT none of the exisitng ones is for a -list format.
But best to avoid any future surprises.
It has not shown any real value in the last decade+ it was enabled,
and it can actually add quite some performance overhead. E.g., if an
API endpoint returns a few 100k of relatively simple entries we can
easily require several seconds, even tens of seconds, to run the
return validation - making it easier to run into timeouts along the
transmit path to the client.
The CLI handler has it still enabled, normally there's no timeout
there as no HTTP transmit path is involved, and d.csapak had a slight
preference for that in an off-list discussion.
The actual implementations in PMG or PVE could enable it too if
running under debug mode.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
allow workers to count warnings and finish tasks in a WARNINGS state
as is already supported by the UI (and PBS).
A nice bonus is that warn() can be used by both workers and non-workers. For
workers, the output is redirected/duplicated as set up by {fork,tee}_worker(),
and non-erroring workers that issued a warning will end in a WARNINGS state.
Stefan Reiter [Thu, 22 Apr 2021 15:34:50 +0000 (17:34 +0200)]
PBSClient: add file_restore_extract function
*_prepare creates a fifo for streaming data back to clients directly,
filefile_restore_extract blocks and should be called from a background
worker - while it is running outcoming data can be read from the FIFO.
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
Oguz Bektas [Wed, 24 Mar 2021 13:20:36 +0000 (14:20 +0100)]
daemon: create_reusable_socket: listen on IPv6 and IPv4
When the 'Domain' configuration of IO::Socket::IP is set to PF_INET6,
it will prefer IPv6, but since we don't set 'V6Only' it will also
listen on IPv4 interfaces[0].
get_options: don't set optional positional params to `undef`
Currently this happened if (and only if) at least one
positional parameter was passed.
We run into this with
`pmgconfig cert delete <type> [<restart>]`
vs
`pvenode cert delete [<restart>]`
where in the PVE case the `restart` option was simply
omitted, whereas for PMG due to the existence of `<type>`
the `restart` option was explicitly passedset in the $opts
hash but ended up being `undef`.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Stefan Reiter [Wed, 17 Feb 2021 13:32:16 +0000 (14:32 +0100)]
format: fix render_bytes with CLIFormatter
CLIFormatter passes in an options hash as a second paramter, which so
far was ignored. Now that we treat it as a precision parameter, check if
it is a hash and extract the option from it before using it.
Otherwise perl puts HASH(0x...) into the format and sprintf chokes.
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
to avoid erroring out when "multiple" addresses are configured on live
migration, when in fact it's the same IP multiple times.
Seems like the same problem for a caller in pve-cluster was fixed by checking
the uniqueness afterwards, see commit 266041169beb36c8892ca54265e2d91335307ffb
in pve-cluster. But there doesn't seem to be any caller relying on the current
behavior, and no additional information other than the addresses are returned,
so fix it here.
Fabian Ebner [Mon, 15 Feb 2021 12:24:55 +0000 (13:24 +0100)]
sendmail: use more complete email regex and shellquote
Shellquote is needed for '~', and while it doesn't help with '-',
there should be no problem, because options are separated from mailto
since commit 216a3f4f131693dc4bbad5e06e96a61baef5f5e9.
Fabian Ebner [Mon, 21 Dec 2020 13:48:19 +0000 (14:48 +0100)]
SectionConfig: parse_config: add errors to result
so that callers can know about them. This is useful in places where we'd rather
abort then continue with a faulty configuration. For example, when reading the
storage configuration before executing a backup job.
Originally-by: Thomas Lamprecht <t.lamprecht@proxmox.com> Signed-off-by: Fabian Ebner <f.ebner@proxmox.com> Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
sync_mountpoint takes a path, gets an open filedescriptor and calls
syncfs(2) on it.
by opening with O_PATH the syncfs call fails with EBADF (see open(2)).
found by running:
```
pkill -f 'pvedaemon worker';
strace -yyttT -s 512 -o /tmp/trace -fp $(pgrep -f pvedaemon$)
```
This patch enables the sendmail helper sub to send emails with a non-existant
display name in the from address. This is used to replace the direct call to
the sendmail binary in pve-manager/PVE/API2/APT.pm.
PVE::Tools::sendmail currently always sends a multipart/alternative message
irrespective of the actual content of the mail (e.g. a plain-text only mail
need not be sent as multipart message).
Additionally a few small refactorings based on the discussion
in https://lists.proxmox.com/pipermail/pbs-devel/2020-August/000423.html
and commited in 66004f22c6475ceb0146cf2df1f380f9f0274be4 in the
rust proxmox repository git://git.proxmox.com/git/proxmox.git
were carried over.
tested by creating a backup of a VM and setting an e-mail address, having
ha-manager send a mail after fencing as well as sending a few small mails via
'perl -e'
Thomas Lamprecht [Wed, 26 Aug 2020 19:21:03 +0000 (21:21 +0200)]
get_options: allow optional arguments "arg_params" if no ambiguity
If we run out of passed arguments from the user but still had defined
"arg_params" (those params which went after the command in fixed
order without option -- dashes) we always errored out with "not
enough arguments". But, there are situations where the remaining
arg_params are all marked as optional in the schema, so we do not
need to error out in that case.
A prime (future) use case is "pvesm prune-backups". Currently the
usage is:
> pvesm prune-backups storeid --prune-backups keep-last=1,keep-...
Because the "prune-backups" keep retention property is optional as it
can fallback to the one defined in the storage configuration.
With this patch we can make it an argument and allow the following
two usages:
1. As above, but avoiding the extra ugly --prune-backups
> pvesm prune-backups storeid keep-last=1,keep-...
2. Fallback to storage config:
> pvesm prune-backups storeid
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
run_command: improve performance for logging and long lines
to call out/err/logfunc with each line, we search for a newline and call
outfunc/logfunc with everything before that
since we do a select/read (with 4096 size) in a loop, this means
that if we have very long lines, we search for a newline in an
ever growing buffer (for which we know does not contain a newline)
this is (via file_get_contents and file_copy) used for reading from
pmxcfs, which has a file size limit of 512k. since quite a number of
call sites would need to explicitly override this (and then get updated
if we bump the limit on the pmxcfs side again in the future), making our
default file reader compatible by default seems the better solution.
Stefan Reiter [Thu, 25 Jun 2020 11:35:39 +0000 (13:35 +0200)]
JSONSchema: add format validator support and cleanup check_format
Adds a third, optional parameter to register_format that allows specifying
a function that will be called after parsing and can validate the parsed
data. A validator should die on failed validation, and can also change the
parsed object by returning a modified version of it.
This is useful so one can register a format with its hash, thus allowing
documentation to be generated automatically, while still enforcing certain
validation rules.
The validator only needs to be called in parse_property_string, since
check_format always calls parse_property_string if there is a
possibility of a validator existing at all. parse_property_string should
then be called with named formats for best effect, as only then can
validators be used.
Clean up 'check_format' as well (which pretty much amounts to a rewrite).
No existing functionality is intentionally changed.
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
projects / pve-common.git / log