pve-common.git
14 months agogenerate_usage_str: show short usage help for unknown commands
Thomas Lamprecht [Wed, 13 Jun 2018 06:30:35 +0000 (08:30 +0200)]
generate_usage_str: show short usage help for unknown commands

followup for previous commit to show the full short usage string for
the CLI tool in the case of an unknown command

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
14 months agogenerate_usage_str: do no generate help for unknown commands
Dietmar Maurer [Wed, 13 Jun 2018 06:00:22 +0000 (08:00 +0200)]
generate_usage_str: do no generate help for unknown commands

Before:

...
USAGE: pvesm aaa zfsscan

With this patch applied:

no such command 'aaa'

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
14 months agoadd a generalized 'read and confirm password' sub
Dominik Csapak [Tue, 12 Jun 2018 10:33:47 +0000 (12:33 +0200)]
add a generalized 'read and confirm password' sub

to use everywhere we read two passwords and compare them

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
14 months agoCLIHandler.pm: fix command line completion for simple commands
Dietmar Maurer [Tue, 12 Jun 2018 08:57:45 +0000 (10:57 +0200)]
CLIHandler.pm: fix command line completion for simple commands

You can simply test behavior using 'qmrestore' ...

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
14 months agoapi_dump: add $raw_dump options
Dietmar Maurer [Mon, 11 Jun 2018 09:23:19 +0000 (11:23 +0200)]
api_dump: add $raw_dump options

Allow to return the original tree with all refs. We use this
with our new pveclient which needs the full api definition.
Keeping refs makes it possible to store the tree more efficiently.

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
14 months agobash completion: complete fully specified command
Wolfgang Bumiller [Thu, 7 Jun 2018 09:59:02 +0000 (11:59 +0200)]
bash completion: complete fully specified command

This contains 2 functional changes:

First: resolve_cmd no longer keeps a hash of which arguments
were expanded. This information is not required and not used
properly: For one it would conflict if the same word
appeared twice in a longer subcommand, and secondly we lose
the information when recursing into an alias anyway. And
lastly, we do not support tab completing multiple parameters
simultaneously anyway (as in, `pveum u a<tab>` does not
become `pveum user add`).
So now we simply return the expanded version of the last
command or undef if it was unknown in place of the hash we
returned previously.

The second change is how we use the new returned value:
Previously if resolve_cmd() returned a new subcommand in
$def we skipped over finishing the last word. Of course, if
the command was already fully specified (but no space put
after it), we already considered it complete and returned
the new $def.
This condition can be detected as in this case the $prev
command equals the $cur command. (Additionally, the $cur
command is either '' (after the space) or also the $prev
command (before the space), but checking this would only be
required when the same word can actually appear multiple
times in a row in a sub command chain...)
This case now takes precedence over looking through the
nested $def, so that bash will put the space after a full
command which requires another subcommand to be added.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
14 months agocleanup: fix variable typo
Wolfgang Bumiller [Thu, 7 Jun 2018 09:59:01 +0000 (11:59 +0200)]
cleanup: fix variable typo

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
14 months agofix getopt when invoking without arguments
Dietmar Maurer [Wed, 6 Jun 2018 04:24:32 +0000 (06:24 +0200)]
fix getopt when invoking without arguments

14 months agotools: unbless errors in run_fork_with_timeout
Wolfgang Bumiller [Mon, 4 Jun 2018 08:39:34 +0000 (10:39 +0200)]
tools: unbless errors in run_fork_with_timeout

We cannot properly encode blessed objects as json, so
instead, we should stringify them. This happened for
instance if a VM's systemd scope wasn't cleaned up as we
got an error as a Net::DBus::Error object causing a
"malformed json string" error to appear instead of the
actual message.

Additionally, add a 'must_stringify' helper: The above error
object implements a '""' operator for stringification (as
all error should), but in theory that could die as well, in
which case we just return a generic error string we'll
hopefully never see...

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
14 months agobump version to 5.0-32
Thomas Lamprecht [Tue, 29 May 2018 06:28:29 +0000 (08:28 +0200)]
bump version to 5.0-32

15 months agofix #1766: compare task starttime numerically
Wolfgang Bumiller [Wed, 16 May 2018 06:54:55 +0000 (08:54 +0200)]
fix #1766: compare task starttime numerically

Not only because <=> is correct, but using 'cmp' also has
the side effect that it adds a string version to the
variable and the API's json output turns into a string as
well, and this only happens once a task has completed
(while it's an integer while it's still running...)

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
15 months agoapi_dump_remove_refs: prepare API tree for use with to_json($tree)
Dietmar Maurer [Fri, 18 May 2018 07:01:46 +0000 (09:01 +0200)]
api_dump_remove_refs: prepare API tree for use with to_json($tree)

We want to use this with the extractapi.pl helper (pve-docs, pve-api-client).

15 months agobump version to 5.0-31
Thomas Lamprecht [Fri, 4 May 2018 10:57:25 +0000 (12:57 +0200)]
bump version to 5.0-31

15 months agobuild: install ACME files
Fabian Grünbichler [Thu, 19 Apr 2018 12:01:35 +0000 (14:01 +0200)]
build: install ACME files

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
15 months agoacme: add challenge plugins
Fabian Grünbichler [Mon, 30 Apr 2018 10:14:54 +0000 (12:14 +0200)]
acme: add challenge plugins

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
15 months agoacme: add ACME library
Fabian Grünbichler [Thu, 19 Apr 2018 12:01:33 +0000 (14:01 +0200)]
acme: add ACME library

this implements those parts of draft-ietf-acme-acme-09 which are needed
to use Let's Encrypt's v2 API.

(based on an internal implement for the Let's Encrypt v1 API)
Co-Authored-By: Wolfgang Bumiller <w.bumiller@proxmox.com>
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
15 months agoadd Certificate helper
Fabian Grünbichler [Thu, 19 Apr 2018 12:01:32 +0000 (14:01 +0200)]
add Certificate helper

general purpose certificate related helper functions

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
15 months agonew helper PVE::Tools::du() - get disk usage
Dietmar Maurer [Wed, 25 Apr 2018 08:29:58 +0000 (10:29 +0200)]
new helper PVE::Tools::du() - get disk usage

We simply call the external binary 'du', so that we can abort the command
when we run into a timeout.

15 months agountaint df return values
Dominik Csapak [Tue, 24 Apr 2018 08:15:14 +0000 (10:15 +0200)]
untaint df return values

since we sometimes use their length in a format string for printf

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Co-authored-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
15 months agoREADME.dev: inkscape no longer needed, s/git-core/git/
Thomas Lamprecht [Mon, 23 Apr 2018 06:21:48 +0000 (08:21 +0200)]
README.dev: inkscape no longer needed, s/git-core/git/

15 months agoREADME.dev: s/jessie/stretch
Thomas Lamprecht [Mon, 23 Apr 2018 06:20:56 +0000 (08:20 +0200)]
README.dev: s/jessie/stretch

16 months agouse ssh -X for upload target
Dietmar Maurer [Thu, 5 Apr 2018 10:23:17 +0000 (12:23 +0200)]
use ssh -X for upload target

16 months agobump version to 5.0-30
Thomas Lamprecht [Thu, 22 Mar 2018 08:09:50 +0000 (09:09 +0100)]
bump version to 5.0-30

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
17 months agopty: fix read_password prototype
Wolfgang Bumiller [Mon, 12 Mar 2018 11:44:52 +0000 (12:44 +0100)]
pty: fix read_password prototype

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
17 months agocli: more generic interactive parameter definition
Wolfgang Bumiller [Mon, 12 Mar 2018 12:04:15 +0000 (13:04 +0100)]
cli: more generic interactive parameter definition

Instead of hardcoding 'password' as a special case in the
JSONSchema's getopt handling, extend the new parameter
mapping to allow defining a parameters as 'interactive'.
They also take an optional argument on the command line
directly.

This effectively deprecates the password special case which
should be replaced in pct/pveum/... and then dropped in
pve-common.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
17 months agobump version to 5.0-29
Dietmar Maurer [Fri, 9 Mar 2018 07:56:27 +0000 (08:56 +0100)]
bump version to 5.0-29

17 months agoCLIHandler: fix command usage string generation
Thomas Lamprecht [Tue, 6 Mar 2018 08:08:11 +0000 (09:08 +0100)]
CLIHandler: fix command usage string generation

track our command string, i.e. everything which cannot be an argument
for a specific command, in resolve_cmd, as we go through the commando
definition there anyway and know if a ARGV element is part of the
command itself or its arguments.

Fixes a problem where a invalid command had all the passed parameter
attached in the resulting USAGE output.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
17 months agofix #1682: handle relative years absolutely
Fabian Grünbichler [Wed, 28 Feb 2018 09:42:56 +0000 (10:42 +0100)]
fix #1682: handle relative years absolutely

the timegm(gmtime()) and timelocal(localtime(()) constructs are
problematic in the following case: - $last is such that $year gets set
to a two-digit value (e.g., the referred to timestamp is somewhere in
the range of 1900-1999) - the current date is such that the value of
$year gets interpreted wrongly (e.g., anything other than 1950).

the exact breakage depends on the actual current date AND value of
$last, since localtime/gmtime will interpret two-digit years as (perldoc
Time::Local):
    [...] shorthand for years in the rolling "current century," defined
    as 50 years on either side of the current year. Thus, today, in
    1999, 0 would refer to 2000, and 45 to 2045, but 55 would refer to
    1955.  Twenty years from now, 55 would instead refer to 2055.

fix it by adding 1900 to force 4-digit $year values, as the localtime
documentation suggests.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
17 months agouse better name for string_param_file_mapping (param_mapping).
Dietmar Maurer [Fri, 2 Mar 2018 11:37:23 +0000 (12:37 +0100)]
use better name for string_param_file_mapping (param_mapping).

But keep old one for compatibility.

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
17 months agorename $can_read_pass to $read_password_func
Dietmar Maurer [Fri, 2 Mar 2018 11:37:22 +0000 (12:37 +0100)]
rename $can_read_pass to $read_password_func

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
17 months agorename $pwcallback to $read_password_func
Dietmar Maurer [Fri, 2 Mar 2018 11:37:21 +0000 (12:37 +0100)]
rename $pwcallback to $read_password_func

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
17 months agorename $can_str_param_fmap to $param_mapping_func
Dietmar Maurer [Fri, 2 Mar 2018 11:37:20 +0000 (12:37 +0100)]
rename $can_str_param_fmap to $param_mapping_func

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
17 months agorename $stringfilemap to $param_mapping_func
Dietmar Maurer [Fri, 2 Mar 2018 11:37:19 +0000 (12:37 +0100)]
rename $stringfilemap to $param_mapping_func

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
17 months agointroduce compute_param_mapping_hash helper
Dietmar Maurer [Fri, 2 Mar 2018 11:37:18 +0000 (12:37 +0100)]
introduce compute_param_mapping_hash helper

This allows us to specify a arbitrary mapping func for any param.

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
17 months agobump version to 5.0-28
Fabian Grünbichler [Mon, 19 Feb 2018 10:41:34 +0000 (11:41 +0100)]
bump version to 5.0-28

18 months agoreplace brctl with iproute2 calls
Wolfgang Bumiller [Wed, 7 Feb 2018 13:15:19 +0000 (14:15 +0100)]
replace brctl with iproute2 calls

And add a few helpers for the common cases.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
18 months agoschema: add bwlimit standard option and format
Wolfgang Bumiller [Tue, 30 Jan 2018 12:20:06 +0000 (13:20 +0100)]
schema: add bwlimit standard option and format

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
18 months agoschema: allow ipv6 prefix lengths up to 128
Wolfgang Bumiller [Mon, 29 Jan 2018 14:06:30 +0000 (15:06 +0100)]
schema: allow ipv6 prefix lengths up to 128

There's no technical reason for *us* to limit this to 120.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
18 months agonet: remove flowid in traffic limit commands
Wolfgang Bumiller [Mon, 29 Jan 2018 09:49:14 +0000 (10:49 +0100)]
net: remove flowid in traffic limit commands

We don't use them and iproute2 4.13.0 has an issue parsing
parameters after a policing description.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
18 months agobump version to 5.0-27
Wolfgang Bumiller [Mon, 22 Jan 2018 14:09:59 +0000 (15:09 +0100)]
bump version to 5.0-27

Now breaks qemu-server<<5.0-21

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
18 months agoINotify.pm - new helper poll_changes
Dietmar Maurer [Mon, 22 Jan 2018 11:12:41 +0000 (12:12 +0100)]
INotify.pm - new helper poll_changes

Useful to detect file changes.

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
18 months agoread_file: replace $versions->{$filename} with $cver
Dietmar Maurer [Mon, 22 Jan 2018 11:12:40 +0000 (12:12 +0100)]
read_file: replace $versions->{$filename} with $cver

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
18 months agoCLIHandler: use resolved command definition
Thomas Lamprecht [Mon, 22 Jan 2018 10:00:07 +0000 (11:00 +0100)]
CLIHandler: use resolved command definition

For sub commands we resolve the real $cmd, $def and its arguments,
thus we should also get the handler from the resolved $def, not the
global one.

No change for normal (consisting of only the first argument)
commands, for them $cmddef == $def.
This sneaked in in a respin/rebase of the series.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
19 months agobump version to 5.0-26
Wolfgang Bumiller [Thu, 18 Jan 2018 08:31:02 +0000 (09:31 +0100)]
bump version to 5.0-26

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
19 months agocli: document $cmddef structure
Thomas Lamprecht [Tue, 9 Jan 2018 12:25:33 +0000 (13:25 +0100)]
cli: document $cmddef structure

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Acked-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
19 months agocli: allow to pass sub commands to help as array
Thomas Lamprecht [Tue, 9 Jan 2018 12:25:32 +0000 (13:25 +0100)]
cli: allow to pass sub commands to help as array

Improves usabillity by allowing to pass a sub copmmand unquoted to
the help command, e.g.:

 # pveum help user delete

without this only a quoted version worked, e.g.:
 # pveum help "user delete"

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Acked-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
19 months agocli: allow specifying sub commands through $cmddef
Thomas Lamprecht [Tue, 9 Jan 2018 12:25:31 +0000 (13:25 +0100)]
cli: allow specifying sub commands through $cmddef

allow to use sub commands alá
 # pveum user add

The new resolve_cmd traverses $cmddef, resolves one level of aliases
and returns the respective sub command, its cmddef, arguments and if
it was expanded (e.g., pveum u d ... => pveum user delete ...) which
allows quite easy integration in the usage/synopsis generator, bash
completion helper and command handler.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
19 months agocli: factor out generate usage string
Thomas Lamprecht [Tue, 9 Jan 2018 12:25:30 +0000 (13:25 +0100)]
cli: factor out generate usage string

reduce code reuse and prepare for sub commands

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
19 months agocli: refactor print_bash_completion
Thomas Lamprecht [Tue, 9 Jan 2018 12:25:29 +0000 (13:25 +0100)]
cli: refactor print_bash_completion

move variables nearer to where they actually used.
drop program name early from argv array
drop unnecessary variables

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
19 months agoJSONSchema: add fingerprint-sha256 standard option
Thomas Lamprecht [Mon, 8 Jan 2018 08:38:52 +0000 (09:38 +0100)]
JSONSchema: add fingerprint-sha256 standard option

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
19 months agofork_worker: guard more setup code with eval
Wolfgang Bumiller [Wed, 27 Dec 2017 10:06:07 +0000 (11:06 +0100)]
fork_worker: guard more setup code with eval

As it might die with an error which should end up in the
_exit() code path rather than bailing out into the upper
scope.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Reviewed-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
19 months agofork_worker: use correct handle type for POSIX::write
Wolfgang Bumiller [Wed, 27 Dec 2017 10:11:05 +0000 (11:11 +0100)]
fork_worker: use correct handle type for POSIX::write

$resfh can be a pipe from POSIX::pipe() or the upid output
handle, which is an IO::File, so we need to take its
fileno().

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Fixes: ed52a8435a6d ("fork_worker: use separate pipe for status messages")
Reviewed-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
19 months agocli: factor out abort
Thomas Lamprecht [Mon, 18 Dec 2017 09:21:39 +0000 (10:21 +0100)]
cli: factor out abort

will be reused in later patches too

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Reviewed-by: Dominik Csapak <d.csapak@proxmox.com>
Acked-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
19 months agocli: refactor comand name helper
Thomas Lamprecht [Mon, 18 Dec 2017 09:21:37 +0000 (10:21 +0100)]
cli: refactor comand name helper

use shorter and also a bit faster methods to expand and get comand
names

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Acked-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
19 months agocli: factor out initialisation check
Thomas Lamprecht [Mon, 18 Dec 2017 09:21:36 +0000 (10:21 +0100)]
cli: factor out initialisation check

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Reviewed-by: Dominik Csapak <d.csapak@proxmox.com>
Acked-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
19 months agocli: refactor and use $cmddef directly
Thomas Lamprecht [Mon, 18 Dec 2017 09:21:35 +0000 (10:21 +0100)]
cli: refactor and use $cmddef directly

passing one param less to the helper methods

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Reviewed-by: Dominik Csapak <d.csapak@proxmox.com>
Acked-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
19 months agoTools/df: reuse run_fork_with_timeout
Thomas Lamprecht [Mon, 18 Dec 2017 13:58:26 +0000 (14:58 +0100)]
Tools/df: reuse run_fork_with_timeout

Use the later added run_fork_with_timeout for encapsulating df in a
safely manner.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
20 months agorun_command: add 'quiet' parameter for omitting STD* prints
Thomas Lamprecht [Tue, 19 Dec 2017 10:50:16 +0000 (11:50 +0100)]
run_command: add 'quiet' parameter for omitting STD* prints

Without this patch we printed to STDOUT and STDERR, respectively, if
no $outfunc or $errfunc was passed.

Sometimes it's useful if one, or even both, of those prints can
be suppressed, currently this can only be done by either using an array
of arrays or a whole string for the command and redirecting STDOUT
and STDERR.

Add a 'quiet' option which allows to do this in an easier way.
It allows to silent STDERR or STDOUT or both.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
20 months agofork_worker: factor out synced worker output mirroring
Thomas Lamprecht [Fri, 15 Dec 2017 16:00:31 +0000 (17:00 +0100)]
fork_worker: factor out synced worker output mirroring

When running in sync (CLI environment) we mirror the workers output
to both, STDOUT and th task log file, a similar function as the unix
comand line tool tee provides, thus we borrow its name for the
factored out sub method.

This moves ~60 lines of code out of the big fork_worker sub and makes
it easier to read track what happens there.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
20 months agofork_worker: use separate pipe for status messages
Thomas Lamprecht [Fri, 15 Dec 2017 16:00:30 +0000 (17:00 +0100)]
fork_worker: use separate pipe for status messages

We forced line wise flushing of the workers STDOUT and STDERR to
capture the final status (TASK OK/TASK ERROR).
Thus, if the code executed in the worker wanted to flush explicitly,
e.g., when the last output wasn't new line terminated but needed to
reach the users eyes, the parent just ignored that.
This leads to confusing results in CLI handlers using fork_workers.

So remove the buffering logic completely and introduce a separate
pipe for sending the final status.
Said pipe gets once read after the child closes (EOF) its STDOUT.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
20 months agofork_worker: refactor passing $upid to parent for sync
Thomas Lamprecht [Fri, 15 Dec 2017 16:00:29 +0000 (17:00 +0100)]
fork_worker: refactor passing $upid to parent for sync

STDOUT and $psync[1] are the same here, so no need to differ.
Also we do this only for letting the parent know tha we're ready, the
parent knows the UPID already as it was generated before forking.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
20 months agobump version to 5.0-25
Wolfgang Bumiller [Fri, 15 Dec 2017 12:25:49 +0000 (13:25 +0100)]
bump version to 5.0-25

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
20 months agoticket: raise UNAUTHORIZED not FORBIDDEN in verify subs
Thomas Lamprecht [Fri, 15 Dec 2017 05:41:49 +0000 (06:41 +0100)]
ticket: raise UNAUTHORIZED not FORBIDDEN in verify subs

In the ticket and CSRF prevention token verification methods we used
a raise_perm exception to tell our caller about a failure of such a
verification. raise_perm uses HTTP_FORBIDDEN (403) as code.

Earlier, all such exceptions or die's where caught when the anyevent
http server called the auth_handler method and transformed to
HTTP_UNAUTHORIZED (401).

With commit d8327719e353198a1dffad88c246fee065054a6b from
pve-http-server we gained the ability to tell a client about a server
internal 5XX error, so that clients do not get wrongly logged out if
we have a internal error.
This resulted also in the effect that the exceptions of the
verify_rsa_ticket and verify_csrf_prevention_token sub methods where
passed to the client.

If an old, now invalid, ticket was sent to the server a client got
403 (FORBIDDEN) instead of the 401 (UNAUTHORIZED) - which he was used
to, and thus meant that he did some wrong doing, instead of knowing
that he just needs to login.

As we are not yet logged in here, and thus cannot possibly know if
the call is forbidden or not, HTTP_FORBIDDEN seems the wrong code.
Change it to HTTP_UNAUTHORIZED, which restores it to the code we told
API clients since ever and is the correct one here.

Also RFC 2068 section 10.4.4 [1] defines that for the afformentioned
verify methods FORBIDDEN was not really correct:

 > 403 Forbidden
 >
 >    The server understood the request, but is refusing to fulfill it.
 >    Authorization will not help and the request SHOULD NOT be
 >    repeated. [...]

With a invalid ticket or CSRF prevention token we have a
authorization problem for the current call, not a permission problem
(we may have, but we can't tell yet).

[1] https://tools.ietf.org/html/rfc2068#section-10.4.4

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
20 months agofix convert_size with decimal numbers and add tests
Dominik Csapak [Fri, 15 Dec 2017 09:58:10 +0000 (10:58 +0100)]
fix convert_size with decimal numbers and add tests

converting from 0.5 gb to mb resulted in 0 mb
with this patch it correctly returns 512

also add tests and catch more errors

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
20 months agobump version to 5.0-24
Wolfgang Bumiller [Wed, 13 Dec 2017 14:05:07 +0000 (15:05 +0100)]
bump version to 5.0-24

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
20 months agoread_password improvements
Wolfgang Bumiller [Wed, 13 Dec 2017 10:47:23 +0000 (11:47 +0100)]
read_password improvements

* Cancel on Ctrl+C (die())
* Finish on Ctrl+D (eof/eot) without appending a newline
* Also finish on \n to be sure.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Reviewed-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Tested-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
20 months agowait_for_vnc_port: allow to enforce IP family
Thomas Lamprecht [Mon, 4 Dec 2017 10:30:11 +0000 (11:30 +0100)]
wait_for_vnc_port: allow to enforce IP family

Most times a port was requested for a specified IP family (v4, v6)
only. Thus also ensure that the port from the respective family got
ready, else we may return on a false positive.

As we had no user setting the $timeout param we can add the $family
param as second one, it'll get used more often, so no need to put it
at the back.

As we do nothing if not defined this does not changes the behavior of
our users yet.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
20 months agowait_for_vnc_port: die if port did not get ready
Thomas Lamprecht [Mon, 4 Dec 2017 10:30:10 +0000 (11:30 +0100)]
wait_for_vnc_port: die if port did not get ready

All of our users expected this behavior and did not check for undef

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
20 months agobump version to 5.0-23
Wolfgang Bumiller [Mon, 4 Dec 2017 09:17:13 +0000 (10:17 +0100)]
bump version to 5.0-23

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
20 months agobuildsys: cleanup and add PTY.pm to install files
Wolfgang Bumiller [Fri, 24 Nov 2017 09:57:10 +0000 (10:57 +0100)]
buildsys: cleanup and add PTY.pm to install files

There was no obvious order to this list, now there is.
Plus, tabs after non-tabs are evil.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
20 months agopty: add read_password helper
Wolfgang Bumiller [Fri, 24 Nov 2017 09:56:15 +0000 (10:56 +0100)]
pty: add read_password helper

Short helper which doesn't need readline and/or deal with
history cleanup.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
20 months agoAdd PVE::PTY helper class
Wolfgang Bumiller [Fri, 24 Nov 2017 09:54:38 +0000 (10:54 +0100)]
Add PVE::PTY helper class

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
20 months agoRemove deprecated run_cli subroutine
Philip Abernethy [Mon, 16 Oct 2017 09:03:33 +0000 (11:03 +0200)]
Remove deprecated run_cli subroutine

20 months agobump version to 5.0-22
Wolfgang Bumiller [Mon, 20 Nov 2017 13:22:24 +0000 (14:22 +0100)]
bump version to 5.0-22

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
20 months agodaemon: fix send termination to all workers on exit
Thomas Lamprecht [Fri, 17 Nov 2017 13:05:48 +0000 (14:05 +0100)]
daemon: fix send termination to all workers on exit

The hash slice did not work as intented here, it only return the keys
from the last elemend defined in the slice, thus not all workers got
a TERM.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
21 months agobump version to 5.0-21
Dietmar Maurer [Tue, 14 Nov 2017 07:01:56 +0000 (08:01 +0100)]
bump version to 5.0-21

21 months agopartially revert: daemon: refactor and cleanup
Wolfgang Bumiller [Mon, 13 Nov 2017 09:31:58 +0000 (10:31 +0100)]
partially revert: daemon: refactor and cleanup

Reverts a hunk of 0da5a3e43b16 which removed checking &
untainting of pids from the PVE_DAEMON_WORKER_PIDS env var.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
21 months agodaemon: add missing parenthesis around list
Wolfgang Bumiller [Fri, 10 Nov 2017 11:24:05 +0000 (12:24 +0100)]
daemon: add missing parenthesis around list

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Fixes: 0da5a3e43b16 ("daemon: refactor and cleanup")

21 months agodaemon: terminate_server reduce code reuse
Thomas Lamprecht [Fri, 10 Nov 2017 11:09:29 +0000 (12:09 +0100)]
daemon: terminate_server reduce code reuse

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
21 months agodaemon: refactor and cleanup
Thomas Lamprecht [Fri, 10 Nov 2017 11:09:28 +0000 (12:09 +0100)]
daemon: refactor and cleanup

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
21 months agodaemon: don't send SIGTERM before restart on leave_children_open_on_reload
Thomas Lamprecht [Fri, 10 Nov 2017 11:09:27 +0000 (12:09 +0100)]
daemon: don't send SIGTERM before restart on leave_children_open_on_reload

Else this options is not really useful. First, sending a SIGTERM lets
the children exit, not quite what "leave_children_open_on_reload"
promises.

The problem this causes is that we may get a time window where no
worker is active and thus, for example, our API daemon would not
accept connections during a restart (or better said, reload).

So, don't request termination of any child worker, if this option is
set, but rather just restart (re-exec) ourself, startup a new set of
workers and only then request the termination of the old ones,
allowing a fully seamless reload.

This is only done on `$daemon-exe restart` and thus on
`systemctl reload $daemon`, systemctl restart or any other stop start
cycles always exit all other workers first.

This expects that the worker can do a graceful termination on
SIGTERM, which is already the case for anything using our AnyEvent
based class (which is base of our HTTPServer module).
With graceful termination is meant the following: the worker accepts
no new work and exits immediately after the current queued work is
done.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
21 months agotests: remove acquire-lock newline termination
Wolfgang Bumiller [Fri, 10 Nov 2017 10:29:52 +0000 (11:29 +0100)]
tests: remove acquire-lock newline termination

Not needed anymore.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
21 months agolock_file_full: add missing trailing newline
Thomas Lamprecht [Fri, 10 Nov 2017 09:24:25 +0000 (10:24 +0100)]
lock_file_full: add missing trailing newline

When we do not instantly get the lock we print a respective message
to stderr. This shows also up in the task logs, and if it's the last
message before a 'Task OK' the UI gets confused an shows the task as
erroneous.

Keep the message as its a good feedback for the user to see why an op
seems to do nothing, so simply add a trailing newline.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
21 months agoJSONSchema.pm - add a 'download' attribute
Dietmar Maurer [Wed, 8 Nov 2017 08:42:58 +0000 (09:42 +0100)]
JSONSchema.pm - add a 'download' attribute

To mark methods which downloads file content to the client.

22 months agobump version to 5.0-20
Fabian Grünbichler [Tue, 17 Oct 2017 13:08:57 +0000 (15:08 +0200)]
bump version to 5.0-20

22 months agocli: whitespace cleanup
Philip Abernethy [Tue, 17 Oct 2017 11:16:26 +0000 (13:16 +0200)]
cli: whitespace cleanup

22 months agocli: code cleanup
Philip Abernethy [Tue, 17 Oct 2017 11:16:25 +0000 (13:16 +0200)]
cli: code cleanup

Removes obsolete subroutine and unused Dumper

22 months agotools: more general run_fork_with_timeout + run_fork
Wolfgang Bumiller [Tue, 10 Oct 2017 08:08:12 +0000 (10:08 +0200)]
tools: more general run_fork_with_timeout + run_fork

22 months agobuild: reformat debian/control
Fabian Grünbichler [Wed, 4 Oct 2017 09:05:33 +0000 (11:05 +0200)]
build: reformat debian/control

using wrap-and-sort -abt

22 months agobump version to 5.0-19
Dietmar Maurer [Tue, 3 Oct 2017 09:35:09 +0000 (11:35 +0200)]
bump version to 5.0-19

22 months agoPVE/Subscription.pm - check if we have a key
Dietmar Maurer [Tue, 3 Oct 2017 10:04:47 +0000 (12:04 +0200)]
PVE/Subscription.pm -  check if we have a key

22 months agoPVE/Subscription.pm - avoid warn, and return error message instead
Dietmar Maurer [Tue, 3 Oct 2017 10:02:45 +0000 (12:02 +0200)]
PVE/Subscription.pm - avoid warn, and return error message instead

22 months agoPVE::Subscription - new class to simplify subscription management
Dietmar Maurer [Fri, 29 Sep 2017 07:17:01 +0000 (09:17 +0200)]
PVE::Subscription - new class to simplify subscription management

22 months agoProcFSTools:read_proc_mounts: increase read size
Alexandre Derumier [Fri, 16 Dec 2016 16:26:10 +0000 (17:26 +0100)]
ProcFSTools:read_proc_mounts: increase read size

Signed-off-by: Alexandre Derumier <aderumier@odiso.com>
23 months agotools: df: handle a failing df
Wolfgang Bumiller [Mon, 11 Sep 2017 07:20:08 +0000 (09:20 +0200)]
tools: df: handle a failing df

This function assumed df() will work or hang, but it can
also actually fail and return undef which results in
warnings - let's silence those.

23 months agobump version to 5.0-18
Wolfgang Bumiller [Tue, 12 Sep 2017 11:43:55 +0000 (13:43 +0200)]
bump version to 5.0-18

23 months agorun_fork_with_timeout: allow returning complex structures
Thomas Lamprecht [Tue, 12 Sep 2017 11:25:30 +0000 (13:25 +0200)]
run_fork_with_timeout: allow returning complex structures

Encode the result or the error in JSON. This way complex objects or
exceptions may be passed to the parent in a generic way.

This allows to remove the second pipe 'pipe_err'.

Allow also to return undef without any warnings to our caller.
This avoids a "use of uninitialized variable ..." warning

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
23 months agoUse double hyphens when prefixing command options in the documentation
Emmanuel Kasper [Wed, 6 Sep 2017 12:37:33 +0000 (14:37 +0200)]
Use double hyphens when prefixing command options in the documentation

This makes our man pages follow the GNU long option recommandations
where non-single character options are prefixed with a double hyphen
(https://www.gnu.org/software/libc/manual/html_node/Argument-Syntax.html)

The benefit for PVE is that our documentation looks more similar to what
a user with previous Linux knowledge is used to.

Our bash autocompletion helper only completes options using double hyphens too.

23 months agoTools: add `convert_size` for generic byte conversion
Thomas Lamprecht [Mon, 11 Sep 2017 08:41:34 +0000 (10:41 +0200)]
Tools: add `convert_size` for generic byte conversion

We often need to convert between file sizes, for formatting output,
but also code-internal. Some methods expect kilobytes, some gigabytes
and sometimes we need bytes.

While conversion from smaller to bigger units can be simply done with
a left-shift, the opposite conversion may need more attention -
depending on the used context.

If we allocate disks this is quite critical. For example, if we need
to allocate a disk with size 1023 bytes using the
PVE::Storage::vdisk_alloc method (which expects kilobytes) a
right shift by 10 (<=> division by 1024) would result in "0", which
obviously fails.

Thus we round up the converted value if a remainder was lost on the
transformation in this new method. This behaviour is opt-out, to be
on the safe side.

The method can be used in a clear way, as it gives information about
the source and target unit size, unlike "$var *= 1024", which doesn't
gives direct information at all, if not commented or derived
somewhere from its context.

For example:
 > my $size = convert_unit($value, 'gb' => 'kb');
is more clear than:
 > my $size = $value*1024*1024;

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
23 months agorun_fork_with_timeout: do not overwrite global signal handlers
Thomas Lamprecht [Wed, 6 Sep 2017 11:29:03 +0000 (13:29 +0200)]
run_fork_with_timeout: do not overwrite global signal handlers

perls 'local' must be either used in front of each $SIG{...}
assignments or they must be put in a list, else it affects only the
first variable and the rest are *not* in local context.

This may cause weird behaviour where daemons seemingly do not get
terminating signals delivered correctly and thus may not shutdown
gracefully anymore.

As we only send SIGINT to processes if a manual stop action gets
triggered just catch this one here.

As this is a general method which allows to pass an arbitrary code
payload we cannot sanely handle all signals here, so remove trapping
all other besides SIGINT, if those need to be trapped that should be
done by the caller on a case by case basis.

Fixes: #1495

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>