]> git.proxmox.com Git - qemu-server.git/log
qemu-server.git
6 months agoapi: reduce overly lengthy comment
Thomas Lamprecht [Tue, 10 Oct 2023 10:57:00 +0000 (12:57 +0200)]
api: reduce overly lengthy comment

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
6 months agoFix ACPI-suspended VMs resuming after migration
Filip Schauer [Mon, 9 Oct 2023 13:25:19 +0000 (15:25 +0200)]
Fix ACPI-suspended VMs resuming after migration

Add checks for "suspended" and "prelaunch" runstates when checking
whether a VM is paused.

This fixes the following issues:
* ACPI-suspended VMs automatically resuming after migration
* Shutdown and reboot commands timing out instead of failing
  immediately on suspended VMs

Signed-off-by: Filip Schauer <f.schauer@proxmox.com>
6 months agovm start: set higher timeout if using PCI passthrough
Friedrich Weber [Fri, 6 Oct 2023 12:15:33 +0000 (14:15 +0200)]
vm start: set higher timeout if using PCI passthrough

The default VM startup timeout is `max(30, VM memory in GiB)` seconds.
Multiple reports in the forum [0] [1] and the bug tracker [2] suggest
this is too short when using PCI passthrough with a large amount of VM
memory, since QEMU needs to map the whole memory during startup (see
comment #2 in [2]). As a result, VM startup fails with "got timeout".

To work around this, set a larger default timeout if at least one PCI
device is passed through. The question remains how to choose an
appropriate timeout. Users reported the following startup times:

ref | RAM | time  | ratio (s/GiB)
---------------------------------
[1] | 60G |  135s |  2.25
[1] | 70G |  157s |  2.24
[1] | 80G |  277s |  3.46
[2] | 65G |  213s |  3.28
[2] | 96G | >290s | >3.02

The data does not really indicate any simple (e.g. linear)
relationship between RAM and startup time (even data from the same
source). However, to keep the heuristic simple, assume linear growth
and multiply the default timeout by 4 if at least one `hostpci[n]`
option is present, obtaining `4 * max(30, VM memory in GiB)`. This
covers all cases above, and should still leave some headroom.

[0]: https://forum.proxmox.com/threads/83765/post-552071
[1]: https://forum.proxmox.com/threads/126398/post-592826
[2]: https://bugzilla.proxmox.com/show_bug.cgi?id=3502

Suggested-by: Fiona Ebner <f.ebner@proxmox.com>
Signed-off-by: Friedrich Weber <f.weber@proxmox.com>
7 months agoadd memory parser
Alexandre Derumier [Mon, 4 Sep 2023 11:39:49 +0000 (13:39 +0200)]
add memory parser

In preparation to add more properties to the memory configuration like
maximum hotpluggable memory and whether virtio-mem devices should be
used.

This also allows to get rid of the cyclic include of PVE::QemuServer
in the memory module.

Signed-off-by: Alexandre Derumier <aderumier@odiso.com>
[FE: also convert new usage in get_derived_property
     remove cyclic include of PVE::QemuServer
     add commit message]
Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
7 months agointroduce QMPHelpers module
Fiona Ebner [Mon, 4 Sep 2023 11:39:48 +0000 (13:39 +0200)]
introduce QMPHelpers module

moving qemu_{device,object}{add,del} helpers there for now.

In preparation to remove the cyclic include of PVE::QemuServer in the
memory module and generally for better modularity in the future.

No functional change intended.

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
7 months agomemory: replace deprecated check_running() call
Fiona Ebner [Mon, 4 Sep 2023 11:39:47 +0000 (13:39 +0200)]
memory: replace deprecated check_running() call

PVE::QemuServer::check_running() does both
PVE::QemuConfig::assert_config_exists_on_node()
PVE::QemuServer::Helpers::vm_running_locally()

The former one isn't needed here when doing hotplug, because the API
already assert that the VM config exists. It also would introduce a
new cyclic dependency between PVE::QemuServer::Memory <->
PVE::QemuConfig with the proposed virtio-mem patch set.

In preparation to remove the cyclic include of PVE::QemuServer in the
memory module.

No functional change intended.

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
7 months agomove NUMA-related code into memory module
Fiona Ebner [Mon, 4 Sep 2023 11:39:46 +0000 (13:39 +0200)]
move NUMA-related code into memory module

which is the only user of the parse_numa() helper. While at it, avoid
the duplication of MAX_NUMA.

In preparation to remove the cyclic include of PVE::QemuServer in the
memory module.

No functional change intended.

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
7 months agomove parse_number_sets() helper to helpers module
Fiona Ebner [Mon, 4 Sep 2023 11:39:45 +0000 (13:39 +0200)]
move parse_number_sets() helper to helpers module

In preparation to move parse_numa() to the memory module.

No functional change intended.

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
7 months agorestore vma: inline one timeout variable and move other closer to usage
Fiona Ebner [Tue, 12 Sep 2023 09:16:17 +0000 (11:16 +0200)]
restore vma: inline one timeout variable and move other closer to usage

No functional change intended.

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
7 months agorestore vma: add comment describing timeout
Fiona Ebner [Tue, 12 Sep 2023 09:16:16 +0000 (11:16 +0200)]
restore vma: add comment describing timeout

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
7 months agofix #2816: restore: remove timeout when allocating disks
Fiona Ebner [Tue, 12 Sep 2023 09:16:15 +0000 (11:16 +0200)]
fix #2816: restore: remove timeout when allocating disks

10 minutes is not long enough when disks are large and/or network
storages are used when preallocation is not disabled. The default is
metadata preallocation for qcow2, so there are still reports of the
issue [0][1]. If allocation really does not finish like the comment
describing the timeout feared, just let the user cancel it.

Also note that when restoring a PBS backup, there is no timeout for
disk allocation, and there don't seem to be any user complaints yet.

The 5 second timeout for receiving the config from vma is kept,
because certain corruptions in the VMA header can lead to the
operation hanging there.

There is no need for the $tmp variable before setting back the old
timeout, because that is at least one second, so we'll always be able
to set the $oldtimeout variable to undef in time in practice.
Currently, there shouldn't even be an outer timeout in the first
place, because the only call path leading to here is via the create
API (also used by qmrestore), both of which don't set a timeout.

[0]: https://forum.proxmox.com/threads/126825/
[1]: https://forum.proxmox.com/threads/128093/

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
7 months agodrive: Fix typo in description of efitype
Filip Schauer [Tue, 5 Sep 2023 08:42:31 +0000 (10:42 +0200)]
drive: Fix typo in description of efitype

Signed-off-by: Filip Schauer <f.schauer@proxmox.com>
7 months agomemory: use static_memory in foreach_dimm
Alexandre Derumier [Mon, 19 Jun 2023 07:28:34 +0000 (09:28 +0200)]
memory: use static_memory in foreach_dimm

Signed-off-by: Alexandre Derumier <aderumier@odiso.com>
8 months agobump version to 8.0.7
Fiona Ebner [Mon, 21 Aug 2023 09:53:38 +0000 (11:53 +0200)]
bump version to 8.0.7

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
8 months agotests: use valid machine types for snapshot tests
Markus Frank [Wed, 18 Jan 2023 13:57:56 +0000 (14:57 +0100)]
tests: use valid machine types for snapshot tests

In preparation to turn the 'machine' parameter into a property string.

parse_property_string checks for the regex, therefore the test-cases
with 'somemachine' and 'someothermachine' would fail.

To avoid that, replace 'somemachine' and 'someothermachine' with 'q35'
and 'pc' with sed:

sed -i 's/somemachine/q35/g'
sed -i 's/someothermachine/pc/g'

Signed-off-by: Markus Frank <m.frank@proxmox.com>
[FE: improve wording in commit message]
Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
8 months agofix #3963: Skip TPM startup for template VMs
Filip Schauer [Thu, 10 Aug 2023 09:20:36 +0000 (11:20 +0200)]
fix #3963: Skip TPM startup for template VMs

Skip the software TPM startup when starting a template VM for performing
a backup. This fixes an error that occurs when the TPM state disk is
write-protected.

Signed-off-by: Filip Schauer <f.schauer@proxmox.com>
8 months agocfg2cmd: netdev: fix value for tx_queue_size
Fiona Ebner [Thu, 27 Jul 2023 09:10:01 +0000 (11:10 +0200)]
cfg2cmd: netdev: fix value for tx_queue_size

Quoting from QEMU commit 4271f40383 ("virtio-net: correctly report
maximum tx_queue_size value"):

> Maximum value for tx_queue_size depends on the backend type.
> 1024 for vDPA/vhost-user, 256 for all the others.

> So the parameter is silently ignored and ethtool reports a different
> value than the one provided by the user.

Indeed, for a non-vDPA/vhost-user netdev, the guest will see TX: 256
instead of the specified 1024 here. With the mentioned QEMU commit (in
master and will be part of 8.1), using 1024 will be a hard error:

> Invalid tx_queue_size (= 1024), must be a power of 2 between 256 and 256

Since neither vhost-user, nor vhost-vdpa netdev types are exposed by
Proxmox VE, just changing the limit to the correct 256 should be fine.
No obvious issue during live-migration found.

Fixes: 620d6b32 ("virtio-net: increase defaults rx|tx-queue-size to 1024")
Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
8 months agomigration: alloc nbd disks: fix fall-back for remote live migration
Fiona Ebner [Mon, 17 Jul 2023 14:00:20 +0000 (16:00 +0200)]
migration: alloc nbd disks: fix fall-back for remote live migration

While the comment sated
>    # order of precedence, filtered by whether storage supports it:
>    # 1. explicit requested format
>    # 2. format of current volume
>    # 3. default format of storage

the code did not fall back to the default format in the case of remote
migration, because the format was already set and the code used
> $format //= $defFormat;

This made remote migration from dir with qcow2 to e.g. LVM-thin fail.

Move extracting the format from the volume name to the call side for
local migration. This allows the logic here to be much simpler.

Reported-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Suggested-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
8 months agomigration: alloc nbd disks: base format hint off source storage
Fiona Ebner [Mon, 17 Jul 2023 14:00:19 +0000 (16:00 +0200)]
migration: alloc nbd disks: base format hint off source storage

Previously, qemu_img_format() was called with the target storage's
$scfg and the source storage's volume name.

This mismatch should only be relevant for certain special kinds of
storage plugins:
- no path, but does support an additional QEMU image format besides
  'raw', in short NPAF.
- no path, volume name can match QEMU_FORMAT_RE, in short NPVM.

Note that all integrated plugins are neither NPAF nor NPVM.

Note that for NPAF plugins, qemu_img_format() already always returns
'raw' because there is no path. It's a bit unlikely such a plugin
exists, because there were no bug reports about qemu_img_format()
misbehaving there yet.

Let's go through the cases:
- If source and target storage both have or don't have a path,
  qemu_img_format($scfg, $volname) returns the same for both $scfg's.
- If source storage has a path, but target storage does not, the
  format hint was previously 'raw', but can only be more correct now
  (being what the source image actually is):
  - For non-NPAF targets, since we know there is no path, it follows
    that 'raw' is the only supported QEMU image format.
  - For NPAF targets, the format will be preserved now (if actually
    supported).
- If source storage does not have a path, but target storage does, the
  format hint will be 'raw' now.
  - For non-NPVM sources, QEMU_FORMAT_RE didn't match when
    qemu_img_format() was called with the target storage's $scfg, so
    the hint also was 'raw' before this commit.
  - For NPVM sources, qemu_img_format() might've guessed a format from
    the source volume name when called with the target's $scfg before
    this commit. If the target storage supports the previously guessed
    format, it was preserved before this commit, but will not be
    anymore. In theory, the guess might've also been wrong, and in
    this case, this commit avoids the wrong guess.

To summarize, there is only one edge case with an exotic kind of third
party storage plugin where format preservation would be lost and in
another edge case, format preservation is gained.

In preparation to simplify the format fallback logic implementation.

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
8 months agofix: api: fix permission check for cloudinit drive update
Friedrich Weber [Mon, 13 Mar 2023 12:56:25 +0000 (13:56 +0100)]
fix: api: fix permission check for cloudinit drive update

Trying to regenerate a cloudinit drive as a non-root user via the API
currently throws a Perl error, as reported in the forum [1]. This is
due to a type mismatch in the permission check, where a string is
passed but an array is expected.

[1] https://forum.proxmox.com/threads/regenerate-cloudinit-by-put-api-return-500.124099/

Signed-off-by: Friedrich Weber <f.weber@proxmox.com>
9 months agocloudinit: allow non-root users to set ciupgrade option
Friedrich Weber [Mon, 24 Jul 2023 11:33:48 +0000 (13:33 +0200)]
cloudinit: allow non-root users to set ciupgrade option

The new ciupgrade option was missing in $cloudinitoptions in
PVE::API2::Qemu, so $check_vm_modify_config_perm defaulted to
requiring root@pam for modifying the option. To fix this, add
ciupgrade to $cloudinitoptions. This also fixes an issue where
ciupgrade was missing in the output of `qm cloudinit pending`,
as it also relies on $cloudinitoptions.

This issue was originally reported in the forum [0].

Also add a comment to avoid similar issues when adding new options in
the future.

[0]: https://forum.proxmox.com/threads/131043/

Signed-off-by: Friedrich Weber <f.weber@proxmox.com>
9 months agofix #4620: cfg2cmd: drive device: correctly handle IDE for q35
Fiona Ebner [Thu, 20 Jul 2023 09:45:02 +0000 (11:45 +0200)]
fix #4620: cfg2cmd: drive device: correctly handle IDE for q35

Only unit 0 for IDE is supported with machine type q35. Currently,
QEMU will fail startup with machine type q35 with an error like
> Can't create IDE unit 1, bus supports only 1 units
when ide1 or ide3 is configured.

Make sure to keep backwards compat form migration by leaving ide0 and
ide2 fixed. Since starting with ide1 or ide3 never worked, they can be
moved to a controller with a higher ID without issue.

Reported in the community forum:
https://forum.proxmox.com/threads/124615/post-543127
https://forum.proxmox.com/threads/130815/

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
9 months agotest: add tests with multiple IDE CD-ROM drives
Fiona Ebner [Thu, 20 Jul 2023 09:45:01 +0000 (11:45 +0200)]
test: add tests with multiple IDE CD-ROM drives

None of the configured test storages support the content type iso
right now, just add it to cifs-store.

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
9 months agoupdate_vm_api: properly wrap arguments
Fabian Grünbichler [Wed, 19 Jul 2023 10:22:04 +0000 (12:22 +0200)]
update_vm_api: properly wrap arguments

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
9 months agoapi: update: also check access for currently configured bridge
Fiona Ebner [Mon, 17 Jul 2023 07:15:18 +0000 (09:15 +0200)]
api: update: also check access for currently configured bridge

Relevant when modifying or removing an existing network device.

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
9 months agoqmeventd: VMID from PID: avoid goto
Fiona Ebner [Mon, 10 Jul 2023 08:53:02 +0000 (10:53 +0200)]
qmeventd: VMID from PID: avoid goto

If strtol() returns an errno, make sure not to print the error message
twice after the refactoring.

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
9 months agoqmeventd: VMID from PID: don't fail immediately when encountering unexpected entry
Fiona Ebner [Mon, 10 Jul 2023 08:53:01 +0000 (10:53 +0200)]
qmeventd: VMID from PID: don't fail immediately when encountering unexpected entry

While such entries shuoldn't exist in practice without manually
messing around, it's not hard to imagine, and there's still a good
chance the actual VMID can still be found.

It's still unexpected to encounter such entries so keep the log line.

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
9 months agoqmeventd: further improve getting VMID from PID
Fiona Ebner [Mon, 10 Jul 2023 08:53:00 +0000 (10:53 +0200)]
qmeventd: further improve getting VMID from PID

by also expecting the ".scope" part and trying the next entry if it is
not present instead of immediately failing.

It's still unexpected to encounter such entries, so keep the log line.

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
9 months agoqmeventd: improve getting VMID from PID in presence of legacy cgroup entries
Fiona Ebner [Mon, 10 Jul 2023 08:52:59 +0000 (10:52 +0200)]
qmeventd: improve getting VMID from PID in presence of legacy cgroup entries

On a hybrid cgroup system, the /proc/<PID>/cgroup file looks like

> 13:pids:/qemu.slice/110.scope
> 12:perf_event:/
> 11:devices:/qemu.slice
> 10:misc:/
> 9:hugetlb:/
> 8:freezer:/
> 7:cpu,cpuacct:/qemu.slice/110.scope
> 6:memory:/qemu.slice/110.scope
> 5:rdma:/
> 4:cpuset:/
> 3:blkio:/qemu.slice
> 2:net_cls,net_prio:/
> 1:name=systemd:/qemu.slice/110.scope
> 0::/qemu.slice/110.scope

but the order doesn't seem to be deterministic, so it can happen that
an entry like '11:devices:/qemu.slice' is the first to match the
'/qemu.slice' part, which previously made the code expect to find the
VMID.

To improve detection, as a first step, match the trailing slash too.

Reported in the community forum:
https://forum.proxmox.com/threads/129320/post-571654

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
9 months agovzdump: pbs: factor out getting and checking encryption keys
Thomas Lamprecht [Thu, 6 Jul 2023 15:42:11 +0000 (17:42 +0200)]
vzdump: pbs: factor out getting and checking encryption keys

factor the common checks for disk-less and "normal" backups out into
its own helper, avoiding code duplication and ensuring that the
messages and checks stay in sync.

The use sites for key and master key are a bit clearer, as it all
just depends on them being defined or not.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
9 months agovzdump: reword "master-key but no encryption key" message
Thomas Lamprecht [Wed, 5 Jul 2023 16:25:21 +0000 (18:25 +0200)]
vzdump: reword "master-key but no encryption key" message

.. and make it use a warn level, which can then also mark the whole
task as potentially problematic as with a new enough pve-guest-common
the REST environment worker warn counters are then increased.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
9 months agovzdump: log only once that encryption is enabled
Thomas Lamprecht [Wed, 5 Jul 2023 05:40:26 +0000 (07:40 +0200)]
vzdump: log only once that encryption is enabled

our backup logs are still quite noise at the task start part so avoid
logging that the task is running with encryption enabled twice for
the master-key feature.

The definedness check on master_keyfile isn't required anymore, it
was never for the no-disk case, and for the standard case it isn't
since  781fb80 ("vzdump: error out for master-key backup but no QEMU
support")

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
9 months agovzdump: error out for master-key backup but no QEMU support
Thomas Lamprecht [Thu, 6 Jul 2023 11:01:14 +0000 (13:01 +0200)]
vzdump: error out for master-key backup but no QEMU support

Our QEMU gained master-key support for Proxmox VE 6.4 with initial
QEMU 5.2.0 packaging in 0b8da68 ("add PBS master key support")
version.
As we're now two major releases in the future any VM needs to run
with a newer QEMU version we can just make this a hard-error, as
there really should be no use-case left. After all we only support
upgrading directly to the next major release, so one needs to do at
least a migration (or shutdown) of the VM to reboot the node for
upgrading to Proxmox VE 8, so the lowest QEMU version baseline is 6.0
for Proxmox VE 8 (i.e., the version from PVE 7.0).

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
9 months agovzdump: drop unused pathlist variable for PBS no-disk code path
Thomas Lamprecht [Wed, 5 Jul 2023 16:31:31 +0000 (18:31 +0200)]
vzdump: drop unused pathlist variable for PBS no-disk code path

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
9 months agofix #4822: vzdump: fix pbs encryption for no-disk guests
Fabian Grünbichler [Mon, 3 Jul 2023 07:03:31 +0000 (09:03 +0200)]
fix #4822: vzdump: fix pbs encryption for no-disk guests

these are backed up directly with proxmox-backup-client, and the invocation was
lacking the key parameters.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
10 months agobump version to 8.0.6
Thomas Lamprecht [Wed, 21 Jun 2023 11:03:04 +0000 (13:03 +0200)]
bump version to 8.0.6

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
10 months agomigration: fix issue with qcow2 cloudinit disk live migration
Fiona Ebner [Wed, 21 Jun 2023 10:43:39 +0000 (12:43 +0200)]
migration: fix issue with qcow2 cloudinit disk live migration

The check for with_snapshots for qcow2 needs to happen for these too.

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
10 months agomigration: add trailing newline to aliased volumes error message
Fiona Ebner [Wed, 21 Jun 2023 10:22:34 +0000 (12:22 +0200)]
migration: add trailing newline to aliased volumes error message

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
10 months agoforeach volid helper: make $pending parameter behave like a boolean
Fiona Ebner [Wed, 21 Jun 2023 10:20:57 +0000 (12:20 +0200)]
foreach volid helper: make $pending parameter behave like a boolean

Avoids potential future mistake with passing in an explicit 0.

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
10 months agotests: add migration alias check
Aaron Lauterer [Mon, 19 Jun 2023 09:29:34 +0000 (11:29 +0200)]
tests: add migration alias check

Reviewed-by: Fiona Ebner <f.ebner@proxmox.com>
Signed-off-by: Aaron Lauterer <a.lauterer@proxmox.com>
10 months agomigration: fail when aliased volume is detected
Aaron Lauterer [Mon, 19 Jun 2023 09:29:33 +0000 (11:29 +0200)]
migration: fail when aliased volume is detected

Aliased volids can lead to unexpected behavior in a migration.

An aliased volid can happen if we have two storage configurations,
pointing to the same place. The resulting 'path' for a disk image
will be the same.
Therefore, stop the migration in such a case.

The check works by comparing the path returned by the storage plugin.

We decided against checking the storages themselves being aliased. It is
not possible to infer that reliably from just the storage configuration
options alone.

Reviewed-by: Fiona Ebner <f.ebner@proxmox.com>
Signed-off-by: Aaron Lauterer <a.lauterer@proxmox.com>
10 months agotests: add migration test for pending disk
Aaron Lauterer [Mon, 19 Jun 2023 09:29:32 +0000 (11:29 +0200)]
tests: add migration test for pending disk

Reviewed-by: Fiona Ebner <f.ebner@proxmox.com>
Signed-off-by: Aaron Lauterer <a.lauterer@proxmox.com>
10 months agoqemuserver: migration: test_volid: change attr name and ref handling
Aaron Lauterer [Mon, 19 Jun 2023 09:29:31 +0000 (11:29 +0200)]
qemuserver: migration: test_volid: change attr name and ref handling

Since we don't scan all storages for matching disk images anymore for a
migration we don't have any images found via storage alone. They will be
referenced in the config somewhere.

Therefore, there is no need for the 'storage' ref.
The 'referenced_in_config' is not really needed and can apply to both,
attached and unused disk images.

Therefore the QemuServer::foreach_volid() will change the
'referenced_in_config' attribute to an 'is_attached' one that only
applies to disk images that are in the _main_ config part and are not
unused.

In QemuMigrate::scan_local_volumes() we can then quite easily map the
refs to each state, attached, unused, referenced_in_{pending,snapshot}.

The refs are mostly used for informational use to print out in the logs
why a disk image is part of the migration. Except for the 'attached' case.

In the future the extra step of the refs in QemuMigrate could probably
be streamlined even more.

Signed-off-by: Aaron Lauterer <a.lauterer@proxmox.com>
10 months agomigration: only migrate disks used by the guest
Aaron Lauterer [Mon, 19 Jun 2023 09:29:30 +0000 (11:29 +0200)]
migration: only migrate disks used by the guest

When scanning all configured storages for disk images belonging to the
VM, the migration could easily fail if a storage is not available, but
enabled. That storage might not even be used by the VM at all.

By not scanning all storages and only looking at the disk images
referenced in the VM config, we can avoid unnecessary failures.
Some information that used to be provided by the storage scanning needs
to be fetched explicilty (size, format).

Behaviorally the biggest change is that unreferenced disk images will
not be migrated anymore. Only images referenced in the config will be
migrated.

The tests have been adapted accordingly.

Signed-off-by: Aaron Lauterer <a.lauterer@proxmox.com>
10 months agoqemuserver: foreach_volid: always include pending disks
Aaron Lauterer [Mon, 19 Jun 2023 09:29:29 +0000 (11:29 +0200)]
qemuserver: foreach_volid: always include pending disks

All calling sites except for QemuConfig.pm::get_replicatable_volumes()
already enabled it. Making it the non-configurable default results in a
change in the VM replication.  Now a disk image only referenced in the
pending section will also be replicated.

Signed-off-by: Aaron Lauterer <a.lauterer@proxmox.com>
10 months agoqemuserver: foreach_volid: include pending volumes
Aaron Lauterer [Mon, 19 Jun 2023 09:29:28 +0000 (11:29 +0200)]
qemuserver: foreach_volid: include pending volumes

Make it possible to optionally iterate over disks in the pending section
of VMs, similar as to how snapshots are handled already.

This is for example useful in the migration if we don't want to rely on
the scanning of all storages.

All calling sites are adapted and enable it, except for
QemuConfig::get_replicatable_volumes as that would cause a change for
the replication if pending disks would be included.

The following lists the calling sites and if they should be fine with
the change (source [0]):

1. QemuMigrate: scan_local_volumes(): needed to include pending disk
   images
2. API2/Qemu.pm: check_vm_disks_local() for migration precondition:
   related to migration, so more consistent with pending
3. QemuConfig.pm: get_replicatable_volumes(): would change the behavior
   of the replication, will not use it for now.
4. QemuServer.pm: get_vm_volumes(): is used multiple times by:
4a. vm_stop_cleanup() to deactivate/unmap: should also be fine with
    including pending
4b. QemuMigrate.pm: in prepare(): part of migration, so more consistent
    with pending
4c. QemuMigrate.pm: in phase3_cleanup() for deactivation: part of
    migration, so more consistent with pending

[0] https://lists.proxmox.com/pipermail/pve-devel/2023-May/056868.html

Signed-off-by: Aaron Lauterer <a.lauterer@proxmox.com>
10 months agocloudinit: restore previous default for package upgrades
Fiona Ebner [Wed, 21 Jun 2023 10:04:32 +0000 (12:04 +0200)]
cloudinit: restore previous default for package upgrades

Commit efa3355d ("fix #3428: cloudinit: add parameter for upgrade on
boot") changed the default, but this is a breaking change. The bug
report was only about making the option configurable.

The commit doesn't give an explicit reason for why, and arguably,
doing the upgrade is not an issue for most users. It also leads to a
different cloud-init instance ID, because of the different setting,
which in turn leads to ssh host key regeneration within the VM.

Reported-by: Friedrich Weber <f.weber@proxmox.com>
Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
10 months agoschema: cloudinit: document default for ciupgrade
Fiona Ebner [Wed, 21 Jun 2023 10:04:31 +0000 (12:04 +0200)]
schema: cloudinit: document default for ciupgrade

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
10 months agobump version to 8.0.5
Thomas Lamprecht [Wed, 21 Jun 2023 07:17:44 +0000 (09:17 +0200)]
bump version to 8.0.5

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
10 months agovm start: always reset any failed-state of the VM systemd scope
Thomas Lamprecht [Wed, 21 Jun 2023 07:13:50 +0000 (09:13 +0200)]
vm start: always reset any failed-state of the VM systemd scope

The scope can get into failed state for some issues like OOM kills of
the whole scope, in that case a user cannot re-start the VM until
they manually reset it.

Do this for now inline to avoid a pve-common bump as done in [0]
(location was suggested by me thinking we could maybe do it over
dbus, but as we have a stop command here already it probably doesn't
matters)

[0]: https://lists.proxmox.com/pipermail/pve-devel/2023-June/057770.html

Originally-by: Fiona Ebner <f.ebner@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
10 months agovm start: factor out silencing systemd stop-scope command
Thomas Lamprecht [Wed, 21 Jun 2023 07:12:54 +0000 (09:12 +0200)]
vm start: factor out silencing systemd stop-scope command

will be reused in the next commit

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
10 months agorestore: extend permissions checks
Fabian Grünbichler [Tue, 20 Jun 2023 07:41:49 +0000 (09:41 +0200)]
restore: extend permissions checks

to allow early checking of the merged config, if the backup archive
passed in is a proper volume where extraction is possible.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
10 months agorestore: refactor archive parsing
Fabian Grünbichler [Tue, 20 Jun 2023 07:41:48 +0000 (09:41 +0200)]
restore: refactor archive parsing

to avoid duplicate work, always set 'volid' to the backup volume's volid, if it
was successfully parsed as such.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
10 months agocpuconfig: add missing qemu 8.0 cpu models
Alexandre Derumier [Mon, 19 Jun 2023 12:25:02 +0000 (14:25 +0200)]
cpuconfig: add missing qemu 8.0 cpu models

Signed-off-by: Alexandre Derumier <aderumier@odiso.com>
10 months agobump version to 8.0.4
Thomas Lamprecht [Mon, 19 Jun 2023 05:24:52 +0000 (07:24 +0200)]
bump version to 8.0.4

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
10 months agoadd test for mapped pci devices
Dominik Csapak [Fri, 16 Jun 2023 13:05:27 +0000 (15:05 +0200)]
add test for mapped pci devices

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Tested-By:  Markus Frank <m.frank@proxmox.com>
10 months agomigration: check for mapped resources
Dominik Csapak [Fri, 16 Jun 2023 13:05:26 +0000 (15:05 +0200)]
migration: check for mapped resources

they can only be migrated to nodes where there exists a mapping and if
the migration is done offline

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Tested-By:  Markus Frank <m.frank@proxmox.com>
10 months agoapi: migrate preconditions: use new check_local_resources info
Dominik Csapak [Fri, 16 Jun 2023 13:05:25 +0000 (15:05 +0200)]
api: migrate preconditions: use new check_local_resources info

for offline migration, limit the allowed nodes to the ones where the
mapped resources are available

this adds new info to the api call namely the 'mapped-resources' list,
as well as the 'unavailable-resources' info in the 'not_allowed_nodes'
object

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Tested-By:  Markus Frank <m.frank@proxmox.com>
10 months agocheck local resources: extend for mapped resources
Dominik Csapak [Fri, 16 Jun 2023 13:05:24 +0000 (15:05 +0200)]
check local resources: extend for mapped resources

by adding them to their own list, saving the nodes where they are not
allowed, and return those on 'wantarray' so we don't break existing
callers that don't expect it.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Tested-By:  Markus Frank <m.frank@proxmox.com>
10 months agoenable cluster mapped PCI devices for guests
Dominik Csapak [Fri, 16 Jun 2023 13:05:23 +0000 (15:05 +0200)]
enable cluster mapped PCI devices for guests

this patch allows configuring pci devices that are mapped via cluster
resource mapping when the user has 'Resource.Use' on the ACL path
'/mapping/pci/{ID}' (in  addition to the usual required vm config
privileges)

When given multiple mappings in the config, we use them as alternatives
for the passthrough, and will select the first free one on startup.
It is using our regular pci reservation mechanism for regular devices and
we introduce a selection mechanism for mediated devices.

A few changes to the inner workings were required to make this work well:
* parse_hostpci now returns a different structure where we have a list
  of lists (first level is for the different alternatives and second
  level is for the different devices that should be passed through
  together)
* factor out the 'parse_hostpci_devices' which parses each device from
  the config and does some precondition checks
* reserve_pci_usage now behaves slightly different when trying to
  reserve an device with the same VMID that's already reserved for,
  since for checking which alternative we can use, we already must
  reserve one (this means that qm showcmd can actually reserve devices,
  albeit only for up to 10 seconds)
* configuring a mediated device on a multifunction device is not
  supported anymore, and results in failure to start (previously, it
  just chose the first device to do it). This is a breaking change
* configuring a single pci device twice on different hostpci slots now
  fails during commandline generation instead on qemu start, so we had
  to adapt one test where this occurred (it could never have worked
  anyway)
* we now check permissions during clone/restore, meaning raw/real
  devices can only be cloned/restored by root@pam from now on.
  this is a breaking change.

Fixes #3574: Improve SR-IOV usability
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Tested-By:  Markus Frank <m.frank@proxmox.com>
10 months agoenable cluster mapped USB devices for guests
Dominik Csapak [Fri, 16 Jun 2023 13:05:22 +0000 (15:05 +0200)]
enable cluster mapped USB devices for guests

this patch allows configuring usb devices that are mapped via
cluster resource mapping when the user has 'Mapping.Use' on the ACL
path '/mapping/usb/{ID}' (in addition to the usual required vm config
privileges)

for now, this is only valid if there is exactly one mapping for the
host, since we don't track passed through usb devices yet

This now also checks permissions on clone/restore, meaning a
'non-mapped' device can only be cloned/restored as root@pam user.
That is a breaking change.

Refactor the checks for restoring into a sub, so we have central place
where we can add such checks

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Tested-By:  Markus Frank <m.frank@proxmox.com>
10 months agousb: refactor usb code and move some into USB module
Dominik Csapak [Fri, 16 Jun 2023 13:05:21 +0000 (15:05 +0200)]
usb: refactor usb code and move some into USB module

similar to how we handle the PCI module and format. This makes the
'verify_usb_device' method and format unnecessary since
we simply check the format with a regex.

while doing tihs, i noticed that we don't correctly check for the
case-insensitive variant for 'spice' during hotplug, so fix that too

With this we can also remove some parameters from the get_usb_devices
and get_usb_controllers functions

while were at it, refactor the permission checks for the usb config too
and use the new 'my sub' style for the functions

also make print_usbdevice_full parse the device itself, so we don't have
to do it in multiple places (especially in places where we don't see
that this is needed)

No functional change intended

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Tested-By:  Markus Frank <m.frank@proxmox.com>
10 months agomigration: add target_storage_check_available
Aaron Lauterer [Fri, 16 Jun 2023 09:57:01 +0000 (11:57 +0200)]
migration: add target_storage_check_available

We use this in a few places. By factoring it into its own function, we
can avoid running slightly different checks in various places.

Signed-off-by: Aaron Lauterer <a.lauterer@proxmox.com>
10 months agohelpers: actualy future proof and allow also checking releases
Thomas Lamprecht [Fri, 16 Jun 2023 11:52:30 +0000 (13:52 +0200)]
helpers: actualy future proof and allow also checking releases

ensuring the editor state is saved helps -.-

Reported-by: Fiona Ebner <f.ebner@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
10 months agohelpers: future proof and allow also checking releases
Thomas Lamprecht [Fri, 16 Jun 2023 11:26:43 +0000 (13:26 +0200)]
helpers: future proof and allow also checking releases

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
10 months agofix #4784: helpers: cope with native versions in manager version check
Thomas Lamprecht [Fri, 16 Jun 2023 11:22:36 +0000 (13:22 +0200)]
fix #4784: helpers: cope with native versions in manager version check

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
10 months agostatus: fix description of qmpstatus property
Fiona Ebner [Fri, 3 Mar 2023 07:35:03 +0000 (08:35 +0100)]
status: fix description of qmpstatus property

Using the word 'agent' is highly confusing here as there is no QMP
agent and thus wrongly suggests that the value is related to the
guest agent[0].

[0]: https://forum.proxmox.com/threads/123590/post-537716

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
10 months agocpuconfig: add new x86-64-vX models
Alexandre Derumier [Thu, 8 Jun 2023 09:51:05 +0000 (11:51 +0200)]
cpuconfig: add new x86-64-vX models

https://gitlab.com/x86-psABIs/x86-64-ABI/
https://lists.gnu.org/archive/html/qemu-devel/2021-06/msg01592.html
"
In 2020, AMD, Intel, Red Hat, and SUSE worked together to define
three microarchitecture levels on top of the historical x86-64
baseline:

  * x86-64:    original x86_64 baseline instruction set
  * x86-64-v2: vector instructions up to Streaming SIMD
               Extensions 4.2 (SSE4.2)  and Supplemental
               Streaming SIMD Extensions 3 (SSSE3), the
               POPCNT instruction, and CMPXCHG16B
  * x86-64-v3: vector instructions up to AVX2, MOVBE,
               and additional bit-manipulation instructions.
  * x86-64-v4: vector instructions from some of the
               AVX-512 variants.
"

This patch add new builtin model derivated from qemu64 model,
to be compatible between intel/amd.

mandatory flags from qemu-doc generator:
https://gitlab.com/qemu/qemu/-/blob/master/scripts/cpu-x86-uarch-abi.py

levels = [
    [ # x86-64 baseline
        "cmov",
        "cx8",
        "fpu",
        "fxsr",
        "mmx",
        "syscall",
        "sse",
        "sse2",
    ],
    [ # x86-64-v2
        "cx16",
        "lahf-lm",
        "popcnt",
        "pni",
        "sse4.1",
        "sse4.2",
        "ssse3",
    ],
    [ # x86-64-v3
        "avx",
        "avx2",
        "bmi1",
        "bmi2",
        "f16c",
        "fma",
        "abm",
        "movbe",
"xsave"  #missing from qemu doc currently
    ],
    [ # x86-64-v4
        "avx512f",
        "avx512bw",
        "avx512cd",
        "avx512dq",
        "avx512vl",
    ],
]

x86-64-v1 : I'm skipping it, as it's basicaly qemu64|kvm64 -vme,-cx16 for compat Opteron_G1 from 2004
            so will use it as qemu64|kvm64 is higher are not working on opteron_g1 anyway

x86-64-v2 : Derived from qemu, +popcnt;+pni;+sse4.1;+sse4.2;+ssse3

min intel: Nehalem
min amd : Opteron_G3

x86-64-v2-AES : Derived from qemu, +aes;+popcnt;+pni;+sse4.1;+sse4.2;+ssse3

min intel: Westmere
min amd : Opteron_G3

x86-64-v3 : Derived from qemu64 +aes;+popcnt;+pni;+sse4.1;+sse4.2;+ssse3;+avx;+avx2;+bmi1;+bmi2;+f16c;+fma;+abm;+movbe+xsave

min intel: Haswell
min amd : EPYC_v1

x86-64-v4 : Derived from qemu64 +aes;+popcnt;+pni;+sse4.1;+sse4.2;+ssse3;+avx;+avx2;+bmi1;+bmi2;+f16c;+fma;+abm;+movbe;+xsave;+avx512f;+avx512bw;+avx512cd;+avx512dq;+avx512vl

min intel: Skylake
min amd : EPYC_v4

Signed-off-by: Alexandre Derumier <aderumier@odiso.com>
10 months agobump version to 8.0.3
Thomas Lamprecht [Fri, 9 Jun 2023 10:20:45 +0000 (12:20 +0200)]
bump version to 8.0.3

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
10 months agoqemu: fix permission check call
Fabian Grünbichler [Fri, 9 Jun 2023 10:10:46 +0000 (12:10 +0200)]
qemu: fix permission check call

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
10 months agobump version to 8.0.2
Thomas Lamprecht [Fri, 9 Jun 2023 08:26:24 +0000 (10:26 +0200)]
bump version to 8.0.2

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
10 months agod/control: bump versioned dependency for libpve-access-control-perl
Thomas Lamprecht [Fri, 9 Jun 2023 08:24:05 +0000 (10:24 +0200)]
d/control: bump versioned dependency for libpve-access-control-perl

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
10 months agoqm: remote migration: improve error when storage cannot be found
Fiona Ebner [Fri, 9 Jun 2023 07:37:34 +0000 (09:37 +0200)]
qm: remote migration: improve error when storage cannot be found

It can also be a permission issue, so the current error can be
a bit confusing.

Reported in the community forum:
https://forum.proxmox.com/threads/120619/post-562660

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
10 months agod/control: bump versioned dependency for libpve-guest-common-perl
Thomas Lamprecht [Thu, 8 Jun 2023 15:25:05 +0000 (17:25 +0200)]
d/control: bump versioned dependency for libpve-guest-common-perl

to ensure we got the vnet access check helper available for us, and
also that the get_derived_property interface is available for our
users

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
10 months agorestore: check bridge access when actual config is available
Thomas Lamprecht [Thu, 8 Jun 2023 15:19:43 +0000 (17:19 +0200)]
restore: check bridge access when actual config is available

This was not only rather inefficient (getting the config from the
archive twice) but also wrong, as we can override options on restore,
so we can do the check only when the backed-up config and override
config got merged.

If this is to late from POV of volume deletion or the like, then the
issue is that those things happen to early, as we can only know what
to do with the actual target config, so destructive actions that
happen before that are wrong by design.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
10 months agomove helper to check bridge access out of api
Thomas Lamprecht [Thu, 8 Jun 2023 15:16:24 +0000 (17:16 +0200)]
move helper to check bridge access out of api

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
10 months agoapi: check bridge access for create, update, clone & restore
Alexandre Derumier [Wed, 7 Jun 2023 12:03:49 +0000 (14:03 +0200)]
api: check bridge access for create, update, clone & restore

Signed-off-by: Alexandre Derumier <aderumier@odiso.com>
Reviewed-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
10 months agoconfig: implement method to calculate derived properties from a config
Fiona Ebner [Tue, 28 Feb 2023 10:54:09 +0000 (11:54 +0100)]
config: implement method to calculate derived properties from a config

See the corresponding commit in guest-common for more information.

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
10 months agoqmeventd: extract vmid from cgroup file instead of cmdline
Fiona Ebner [Wed, 24 May 2023 13:56:50 +0000 (15:56 +0200)]
qmeventd: extract vmid from cgroup file instead of cmdline

This is the single remaining user of the id argument. The id argument
is a Proxmox-specific extension to QEMU, which we'd like to drop to
reduce our differences with upstream QEMU.

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
10 months agoremove left-over mentions of to-be-dropped, outdated QMP commands
Fiona Ebner [Wed, 24 May 2023 13:56:49 +0000 (15:56 +0200)]
remove left-over mentions of to-be-dropped, outdated QMP commands

The commands snapshot-drive and delete-drive-snapshot have been unused
by qemu-server since commit eba2b721 ("use qemu's blockdev-snapshot
functions") and are now going to be dropped in our QEMU builds too, so
get rid of these left-overs.

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
10 months agocloudinit: pass through hostname via fqdn field
Leo Nunner [Tue, 23 May 2023 07:14:19 +0000 (09:14 +0200)]
cloudinit: pass through hostname via fqdn field

If no FQDN is provided, we simply set it to the current hostname. This
ensures that the hostname *really* gets set, since we encountered an
issue on Fedora and CentOS based systems where no hostname got set at
all.

When there's no FQDN set in the cloudinit config, this leads to the
following entry:

    127.0.1.1 <hostname> <hostname>

Which doesn't seem to cause any issues.

Tested on:
 - Ubuntu 23.04
 - CentOS 8
 - Fedora 38
 - Debian 11
 - SUSE 15.4

Signed-off-by: Leo Nunner <l.nunner@proxmox.com>
10 months agofix #2315: api: have resize endpoint spawn a worker task
Fiona Ebner [Tue, 30 May 2023 13:52:04 +0000 (15:52 +0200)]
fix #2315: api: have resize endpoint spawn a worker task

Similar to the corresponding endpoint for containers. Because disks
are involved, this can be a longer running operation, as is also
indicated by the 60 seconds timeout used in qemu_block_resize() which
is called by this endpoint.

This is a breaking API change.

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
10 months agod/control: bump minimal QEMU version to 7.1
Thomas Lamprecht [Wed, 7 Jun 2023 17:23:12 +0000 (19:23 +0200)]
d/control: bump minimal QEMU version to 7.1

just to be safe and have this on a more recent one, as we def. do not
support 3.0 anymore

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
10 months agofix #517: api: allow resizing qcow2 disk with snapshots
Fiona Ebner [Tue, 30 May 2023 13:52:03 +0000 (15:52 +0200)]
fix #517: api: allow resizing qcow2 disk with snapshots

Support for this was added in QEMU 5.1 by commit 7fa140abf6 ("qcow2:
Allow resize of images with internal snapshots").

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
10 months agofast plug options: add migrate_downtime and migrate_speed
Fiona Ebner [Thu, 11 May 2023 12:30:12 +0000 (14:30 +0200)]
fast plug options: add migrate_downtime and migrate_speed

for convenience. These options do not influence the QEMU instance
directly, but are only used for migration, so no need to keep them in
pending.

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
10 months agofast plug options: order alphabetically
Fiona Ebner [Thu, 11 May 2023 12:30:11 +0000 (14:30 +0200)]
fast plug options: order alphabetically

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
10 months agocloudinit: fix 'pending' api endpoint
Leo Nunner [Thu, 11 May 2023 08:03:17 +0000 (10:03 +0200)]
cloudinit: fix 'pending' api endpoint

This patch partially reverts commit 1b5706cd168fedc5e494e24300069ee4ff25761f,
by reintroducing the old format for return values (key, value, pending,
delete), but drops the "force-delete" return value. Right now, this
endpoint does not conform to its own format, because the return values
are as follows:

{
    key => {
old => 'foo',
new => 'bar',
    },
    […]
}

While the format specified is

[
    {
key => 'baz',
old => 'foo',
new => 'bar',
    },
    […]
]

This leads to the endpoint being broken when used through 'qm' and
'pvesh'. Using the API works fine, because the format doesn't get
verified there. Reverting this change brings the advantage that we can
also use PVE::GuestHelpers::format_pending when calling the endpoint
through qm again.

Signed-off-by: Leo Nunner <l.nunner@proxmox.com>
10 months agofix #3428: cloudinit: add parameter for upgrade on boot
Leo Nunner [Thu, 4 May 2023 10:55:01 +0000 (12:55 +0200)]
fix #3428: cloudinit: add parameter for upgrade on boot

up until now, we did an automatic upgrade after the first boot in our
standard cloud-init config. This has been requested to be toggleable
several times [1][2]. With this patch, "package_upgrade" is disabled by
default, and needs to be enabled manually, diverging from the previous
behaviour.

[1] https://forum.proxmox.com/threads/how-to-prevent-automatic-apt-upgrade-during-the-first-boot-with-cloud-init.68472/
[2] https://forum.proxmox.com/threads/cloud-init-ohne-package-upgrade.123841/

Signed-off-by: Leo Nunner <l.nunner@proxmox.com>
10 months agoallow setting ipconfigX with VM.Config.Cloudinit
Fabian Grünbichler [Wed, 3 May 2023 07:52:03 +0000 (09:52 +0200)]
allow setting ipconfigX with VM.Config.Cloudinit

these config keys only affect the cloudinit drive contents (and state of the
guest inside the VM), they are not used anywhere on the hypervisor side, so
they should not require VM.Config.Network (which allows a lot more, such as
changing vNIC VLAN tags or the bridges they are connected to).

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
10 months agoschema: avoid using deprecated -no-hpet in example for 'args' property
Fiona Ebner [Mon, 3 Apr 2023 12:09:45 +0000 (14:09 +0200)]
schema: avoid using deprecated -no-hpet in example for 'args' property

instead use a recent example that served as a workaround in #4625.

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
10 months agocfg2cmd: replace deprecated no-hpet option with hpet=off machine flag
Fiona Ebner [Mon, 3 Apr 2023 12:09:44 +0000 (14:09 +0200)]
cfg2cmd: replace deprecated no-hpet option with hpet=off machine flag

like the deprecation message printed by QEMU suggests.

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
10 months agocfg2cmd: replace deprecated no-acpi option with acpi=off machine flag
Fiona Ebner [Mon, 3 Apr 2023 12:09:43 +0000 (14:09 +0200)]
cfg2cmd: replace deprecated no-acpi option with acpi=off machine flag

like the deprecation message printed by QEMU suggests.

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
10 months agocfg2cmd: use actual backend names instead of removed tty and paraport aliases
Fiona Ebner [Mon, 3 Apr 2023 12:09:42 +0000 (14:09 +0200)]
cfg2cmd: use actual backend names instead of removed tty and paraport aliases

As described in:
https://qemu-project.gitlab.io/qemu/about/removed-features.html#chardev-backend-aliases-tty-and-parport-removed-in-8-0

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
10 months agofixup d/changelog
Wolfgang Bumiller [Wed, 7 Jun 2023 11:50:14 +0000 (13:50 +0200)]
fixup d/changelog

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
10 months agobump version to 8.0.1
Wolfgang Bumiller [Wed, 7 Jun 2023 11:47:58 +0000 (13:47 +0200)]
bump version to 8.0.1

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
10 months agobump common dep to 8.0.2, guest-common to 5.0.1
Wolfgang Bumiller [Wed, 7 Jun 2023 11:47:52 +0000 (13:47 +0200)]
bump common dep to 8.0.2, guest-common to 5.0.1

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
10 months agoapi: switch agent api call to 'array' type
Dominik Csapak [Tue, 6 Jun 2023 13:08:51 +0000 (15:08 +0200)]
api: switch agent api call to 'array' type

we don't want to use the '-alist' formats anymore in favor of real arrays

Acked-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
10 months agoblock resize: avoid passing zero size to QMP command
Fiona Ebner [Fri, 28 Apr 2023 12:32:07 +0000 (14:32 +0200)]
block resize: avoid passing zero size to QMP command

Commit 7246e8f9 ("Set zero $size and continue if volume_resize()
returns false") mentions that this is needed for "some storages with
backing block devices to do online resize" and since this patch came
together [0] with pve-storage commit a4aee43 ("Fix RBD resize with
krbd option enabled."), it's safe to assume that RBD with krbd is
meant. But it should be the same situation for any external plugin
relying on the same behavior.

Other storages backed by block devices like LVM(-thin) and ZFS return
1 and the new size respectively, and the code is older than the above
mentioned commits. So really, the RBD plugin just should have returned
a positive value to be in-line with those and there should be no need
to pass 0 to the block_resize QMP command either.

Actually, it's a hack, because the block_resize QMP command does not
actually do special handling for the value 0. It's just that in the
case of a block device, QEMU won't try to resize it (and not fail for
shrinkage). But the size in the raw driver's BlockDriverState is
temporarily set to 0 (which is not nice), until the sector count is
refreshed, where raw_co_getlength is called, which queries the new
size and sets the size in the raw driver's BlockDriverState again as a
side effect. It's not known to cause any issues, but bdrv_getlength is
a coroutine wrapper starting from QEMU 8.0.0, and it's just better to
avoid setting a completely wrong value even temporarily. Just pass the
actually requested size like is done for LVM(thin) and ZFS.

[0]: https://lists.proxmox.com/pipermail/pve-devel/2017-January/025060.html

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
10 months agonet: Skip and warn of interfaces without bridge
Christian Ebner [Wed, 12 Apr 2023 08:45:13 +0000 (10:45 +0200)]
net: Skip and warn of interfaces without bridge

Handle and warn about network interfaces which are not attached to
any bridge because the user actively removed it from the VM config.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
10 months agoqmrestore: improve description of bwlimit parameter
Stefan Hanreich [Wed, 29 Mar 2023 12:34:34 +0000 (14:34 +0200)]
qmrestore: improve description of bwlimit parameter

This makes the description consistent with the other places that
have bwlimit as a parameter as well.

Signed-off-by: Stefan Hanreich <s.hanreich@proxmox.com>