pve-storage.git
10 days agobump version to 7.0-13 master
Thomas Lamprecht [Thu, 14 Oct 2021 09:22:55 +0000 (11:22 +0200)]
bump version to 7.0-13

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
10 days agotest: also pass format for backing base image
Thomas Lamprecht [Thu, 14 Oct 2021 09:17:40 +0000 (11:17 +0200)]
test: also pass format for backing base image

mirroring what commit 9177cc2eda87c9a8f85a5ba73fa5f8e45cdd44de did
for the plugin system already, with QEMU 6.1 this is now a hard
requirement

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
10 days agofix #3580: plugins: make preallocation mode selectable for qcow2 and raw images
Lorenz Stechauner [Tue, 12 Oct 2021 12:32:32 +0000 (14:32 +0200)]
fix #3580: plugins: make preallocation mode selectable for qcow2 and raw images

the plugins for file based storages
 * BTRFS
 * CIFS
 * Dir
 * Glusterfs
 * NFS
now allow the option 'preallocation'.

'preallocation' can have four values:
 * default
 * off
 * metadata
 * falloc
 * full
see man pages for `qemu-img` for what these mean exactly. [0]

the defualt value was chosen to be
 * qcow2: metadata (as previously)
 * raw: off

when using 'metadata' as preallocation mode, for raw images 'off'
is used.

[0] https://qemu.readthedocs.io/en/latest/system/images.html#disk-image-file-formats

Signed-off-by: Lorenz Stechauner <l.stechauner@proxmox.com>
Reviewed-by: Fabian Ebner <f.ebner@proxmox.com>
Tested-by: Fabian Ebner <f.ebner@proxmox.com>
11 days agoct templates: support zstd compressed archives
Thomas Lamprecht [Wed, 13 Oct 2021 07:06:51 +0000 (09:06 +0200)]
ct templates: support zstd compressed archives

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2 weeks agoapi: disks: create: set correct partition type
Fabian Ebner [Wed, 6 Oct 2021 09:18:46 +0000 (11:18 +0200)]
api: disks: create: set correct partition type

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2 weeks agopartially fix #2285: api: disks: allow partitions for creation paths
Fabian Ebner [Wed, 6 Oct 2021 09:18:45 +0000 (11:18 +0200)]
partially fix #2285: api: disks: allow partitions for creation paths

The calls for directory and ZFS need slight adaptations. Except for
those, the only thing that needs to be done is support partitions in
the disk_is_used helper.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2 weeks agoapi: disks: initgpt: explicitly abort for partitions
Fabian Ebner [Wed, 6 Oct 2021 09:18:44 +0000 (11:18 +0200)]
api: disks: initgpt: explicitly abort for partitions

In preparation to extend disk_is_used to support partitions. Without
this new check, initgpt would also allow partitions once disk_is_used
supports partitions, which is not desirable.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2 weeks agodiskmanage: don't set usage for unused partitions
Fabian Ebner [Wed, 6 Oct 2021 09:18:43 +0000 (11:18 +0200)]
diskmanage: don't set usage for unused partitions

The disk type is already 'partition' so there's no additional
information here. And it would need to serve as a code-word for
unused partitions. The cleaner approach is to not set the usage.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2 weeks agodiskmanage: wipe blockdev: also change partition type
Fabian Ebner [Wed, 6 Oct 2021 09:18:42 +0000 (11:18 +0200)]
diskmanage: wipe blockdev: also change partition type

when called with a partition. Since get_disks uses the partition type
(among other things) to detect LVM and ZFS volumes, such volumes would
still be seen as in-use after wiping. Thus, also change the partition
type and simply use 0x83 "Linux filesystem".

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2 weeks agodiskmanage: add change_parttype and is_partition helpers
Fabian Ebner [Wed, 6 Oct 2021 09:18:41 +0000 (11:18 +0200)]
diskmanage: add change_parttype and is_partition helpers

For change_parttype, only GPT-partitioned disks are supported, as I
didn't see an option for sgdisk to make it also work with
MBR-partitioned disks. And while sfdisk could be used instead (or
additionally) it would be a new dependency, and AFAICS require some
conversion of partition type GUIDs to MBR types on our part.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2 weeks agobtrfs: free image: only remove snapshots for current subvol
Fabian Ebner [Mon, 13 Sep 2021 09:01:42 +0000 (11:01 +0200)]
btrfs: free image: only remove snapshots for current subvol

instead of all in the same directory.

Reported in the community forum:
https://forum.proxmox.com/threads/error-could-not-statfs-no-such-file-or-directory.96057/

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2 weeks agobump version to 7.0-12
Thomas Lamprecht [Tue, 5 Oct 2021 04:25:08 +0000 (06:25 +0200)]
bump version to 7.0-12

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2 weeks agoimport: don't check for 1K aligned size
Stefan Reiter [Mon, 4 Oct 2021 15:29:19 +0000 (17:29 +0200)]
import: don't check for 1K aligned size

TPM state disks on directory storages may have completely unaligned
sizes, this check doesn't make sense for them.

This appears to just be a (weak) safeguard and not serve an actual
functional purpose, so simply get rid of it to allow migration of TPM
state.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2 weeks agod/control: break libpve-http-server-perl (<< 4.0-3)
Thomas Lamprecht [Mon, 4 Oct 2021 08:23:31 +0000 (10:23 +0200)]
d/control: break libpve-http-server-perl (<< 4.0-3)

commit e4d56f096ed28761d6b9a9e348be0fc682928040 removes a `sleep 1`
hack for removal of a tempfile which earlier happened in the
pve-http-server but we do ourself now.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2 weeks agostatus: fix tmpfile cleanup
Lorenz Stechauner [Tue, 31 Aug 2021 10:16:32 +0000 (12:16 +0200)]
status: fix tmpfile cleanup

$tmpfilename already gets unlinked after executing the cmd.

furthermore, because this is a local file, it is wrong to delete
it via the ssh command on a remote node.

small change: added \n to the error message.

Signed-off-by: Lorenz Stechauner <l.stechauner@proxmox.com>
2 weeks agofix #3505: status: add checksum and algorithm to file upload
Lorenz Stechauner [Tue, 31 Aug 2021 10:16:31 +0000 (12:16 +0200)]
fix #3505: status: add checksum and algorithm to file upload

Signed-off-by: Lorenz Stechauner <l.stechauner@proxmox.com>
2 weeks agostatus: remove sleep(1) in file upload
Lorenz Stechauner [Tue, 31 Aug 2021 10:16:30 +0000 (12:16 +0200)]
status: remove sleep(1) in file upload

this racey sleep(1) is only there for legacy reasons: because
we don't use apache anymore and only emulate its behabiour
regarding removing temp files, this is under our own control
now and so we can improve this whole situation.

this change requires a pve-http-server version, in which the
tmpfile gets not automatically removed anymore.

Signed-off-by: Lorenz Stechauner <l.stechauner@proxmox.com>
3 weeks agodiskmanage: allow passing partitions to get_disks
Fabian Ebner [Tue, 28 Sep 2021 11:39:48 +0000 (13:39 +0200)]
diskmanage: allow passing partitions to get_disks

Requires that the $include_partitions parameter is set too, which:
1. Makes sense, because the partition won't be included in the result
   otherwise.
2. Ensures backwards compatibility for existing callers that don't
   use $include_partitions. No existing callers use both $disks and
   $include_partitions at the same time, so nothing learns to
   "support" partitions by accident.

Moving the strip_dev helper to the top, so it can be used everywhere.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
3 weeks agodiskmanage: allow partitions for get_udev_info
Fabian Ebner [Tue, 28 Sep 2021 11:39:47 +0000 (13:39 +0200)]
diskmanage: allow partitions for get_udev_info

both existing callers only call this with non-partitions currently, so
the change should be backwards compatible.

In preparation to enable ZFS creation on top of partitions (where the
udev info is used to get the stable by-id path of a device).

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
3 weeks agoapi: disk: work around udev bug to ensure its database is updated
Fabian Ebner [Tue, 28 Sep 2021 11:39:42 +0000 (13:39 +0200)]
api: disk: work around udev bug to ensure its database is updated

There is a udev bug [0] which can ultimately lead to the udev database
for certain devices not being actively updated. Determining whether a
disk is used or not in get_disks() (in part) relies upon lsblk, which
queries the udev database. Ensure the information is updated by
manually calling 'udevadm trigger' for the changed devices.

It's most important for the 'directory' API path, as mounting depends
on the '/dev/disk/by-uuid'-symlink to be generated.

[0]: https://github.com/systemd/systemd/issues/18525

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
3 weeks agoapi: disks: create: re-check disk after fork/lock
Fabian Ebner [Tue, 28 Sep 2021 11:39:41 +0000 (13:39 +0200)]
api: disks: create: re-check disk after fork/lock

Because then it might not be unused anymore. If there really is a
race, this prevents e.g. sgdisk creating a partition on a device
already in use by LVM or LVM destroying a partitioned device.

For ZFS, also get the latest udev info once inside the worker.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
4 weeks agobtrfs: call free_image correctly
Fabian Ebner [Mon, 20 Sep 2021 10:23:02 +0000 (12:23 +0200)]
btrfs: call free_image correctly

Currently, 'PVE::Storage::DirPlugin' is implicitly passed along as
$class, which means that if the base class's free_image calls another
method (e.g.  filesystem_path) then the DirPlugin's method will be
used, rather than the one from BTRFSPlugin. Change it so that $class
itself is passed along.

See also commit 279d9de5108f6fc6f2d31f77f1b41d6dc7a67cb9 for context,
where the approach in this patch was suggested.

Suggested-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
5 weeks agocifs: negotiates the highest SMB2+ version supported by default
Thomas Lamprecht [Tue, 14 Sep 2021 14:23:23 +0000 (16:23 +0200)]
cifs: negotiates the highest SMB2+ version supported by default

instead of hardcoding it to a potential outdated value.

For `smbclient` we only set max-protocol version and that could only
be smb2 or smb3 (no finer granularity) any how, so this was not
really correct.

Nowadays the kernel dropped SMB1 and tries to go for SMB2.1 or higher
by default, depending on what client and server supports. SMB2.1 is
Windows 7/2008R2 - both EOL since quite a bit, so ok as default lower
boundary.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
5 weeks agocifs: allow "3" and "default" for version
Thomas Lamprecht [Tue, 14 Sep 2021 12:28:04 +0000 (14:28 +0200)]
cifs: allow "3" and "default" for version

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
5 weeks agofix #3609: cifs: add support to SMB 3.11
Moayad Almalat [Mon, 13 Sep 2021 12:15:55 +0000 (14:15 +0200)]
fix #3609: cifs: add support to SMB 3.11

Added support for the SMB version SMB3_11 When the `min protocol =
SMB3_11` in the smb.conf, the CIFS mount will return with the
following error:
```
CIFS VFS: cifs_mount failed w/return code = -95
```
added an optional option to use the `vers=3.11`

Signed-off-by: Moayad Almalat <m.almalat@proxmox.com>
Tested-by: Fabian Ebner <f.ebner@proxmox.com>
[ Thomas: move text from cover letter to commit message &
  add S-o-b ]
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
6 weeks agofix #3610: properly build ZFS detail tree
Fabian Ebner [Fri, 10 Sep 2021 11:45:35 +0000 (13:45 +0200)]
fix #3610: properly build ZFS detail tree

Previously, top-level vdevs like log or special were wrongly added as
children of the previous outer vdev instead of the root.

Fix it by also showing the vdev with the same name as the pool and
start counting from level 1 (the pool itself serves as the root and
should be the only one with level 0). This results in the same kind
of structure as in PBS and (except for the root) zpool status itself.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
6 weeks agofix prune-backups validation (again)
Fabian Ebner [Fri, 10 Sep 2021 09:37:18 +0000 (11:37 +0200)]
fix prune-backups validation (again)

Commit a000e26ce7d30ba2b938402164a9a15e66dd123e caused a test failure
in pve-manager, because now 'keep-all=0' is not thrown out upon
validation anymore. Fix the issue the commit addressed differently,
by simply creating a copy of the (shallow) hash first, and using
the logic from before the commit.

Reported-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
6 weeks agoprune {validate, mark}: preserve input parameter
Fabian Ebner [Thu, 9 Sep 2021 09:58:01 +0000 (11:58 +0200)]
prune {validate, mark}: preserve input parameter

While the current way to detect settings like { 'keep-last' => 0 } is
concise, it's also wrong, because but the delete operation is visible
to the caller. This resulted in e.g.
    # $hash is { 'keep-all' => 1 }
    my $s = print_property_string($hash, 'prune-backups');
    # $hash is now {}, $s is 'keep-all=1'
because validation is called in print_property_string. The same issue
is present when calling prune_mark_backup_group.

Because validation complains when keep-all and something else is set,
this shouldn't have caused any real issues, besides vzdump with
keep-all=1 wrongly taking the removal path, but without any settings,
so not removing anything:
    INFO: prune older backups with retention:
    INFO: pruned 0 backup(s)

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
6 weeks agobtrfs: style: add missing semicolon
Fabian Ebner [Wed, 8 Sep 2021 11:26:51 +0000 (13:26 +0200)]
btrfs: style: add missing semicolon

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
6 weeks agobtrfs: avoid undef warnings with format
Fabian Ebner [Wed, 8 Sep 2021 11:26:50 +0000 (13:26 +0200)]
btrfs: avoid undef warnings with format

which is only set by parse_volname when the volume is a VM or
container image, but not for other content types.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
6 weeks agobump version to 7.0-11
Thomas Lamprecht [Mon, 6 Sep 2021 06:40:39 +0000 (08:40 +0200)]
bump version to 7.0-11

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
6 weeks agoapi: followup style/comment improvements
Thomas Lamprecht [Mon, 6 Sep 2021 06:32:15 +0000 (08:32 +0200)]
api: followup style/comment improvements

try to comment why not what, what is already described good enough by
the code here.

Also, we want to go up to 100cc text-width if it improves
readability, which for post-if's it most often does.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
6 weeks agostatus: move unlink from http-server to enpoint
Lorenz Stechauner [Tue, 31 Aug 2021 10:16:29 +0000 (12:16 +0200)]
status: move unlink from http-server to enpoint

this is the first step in which not the http server removes the
temporary file, but the worker itself.

Signed-off-by: Lorenz Stechauner <l.stechauner@proxmox.com>
6 weeks agobtrfs: fix calling alloc_image from DirPlugin
Thomas Lamprecht [Mon, 6 Sep 2021 06:25:45 +0000 (08:25 +0200)]
btrfs: fix calling alloc_image from DirPlugin

similar to commit 279d9de5108f6fc6f2d31f77f1b41d6dc7a67cb9

This calling style is pretty dangerous in general for such plugin
systems...

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
8 weeks agoCeph: add keyring parameter for external clusters
Aaron Lauterer [Thu, 26 Aug 2021 10:03:32 +0000 (12:03 +0200)]
Ceph: add keyring parameter for external clusters

By adding the keyring for RBD storage or the secret for CephFS ones, it
is possible to add an external Ceph cluster with only one API call.

Previously the keyring / secret file needed to be placed in
/etc/pve/priv/ceph/$storeID.{keyring,secret} manually.

Signed-off-by: Aaron Lauterer <a.lauterer@proxmox.com>
8 weeks agoCephConfig: add optional $secret parameter
Aaron Lauterer [Thu, 26 Aug 2021 10:03:31 +0000 (12:03 +0200)]
CephConfig: add optional $secret parameter

This allows us to manually pass the used RBD keyring or CephFS secret.
Useful mostly when adding external Ceph clusters where we have no other
means to fetch them.

I renamed the previous $secret to $cephfs_secret to be able to use
$secret as parameter.

Signed-off-by: Aaron Lauterer <a.lauterer@proxmox.com>
2 months agozfs: fix unmount request
Fabian Ebner [Thu, 5 Aug 2021 08:33:43 +0000 (10:33 +0200)]
zfs: fix unmount request

by not dying when the dataset is already unmounted. Can be triggered
for a container by doing two rollbacks in a row.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2 months agobuildsys: change upload dist to bullseye
Thomas Lamprecht [Fri, 30 Jul 2021 13:37:13 +0000 (15:37 +0200)]
buildsys: change upload dist to bullseye

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2 months agobump version to 7.0-10
Thomas Lamprecht [Fri, 30 Jul 2021 13:23:19 +0000 (15:23 +0200)]
bump version to 7.0-10

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2 months agoapi: disks: allow zstd compression for zfs pools
Dominik Csapak [Fri, 30 Jul 2021 11:34:15 +0000 (13:34 +0200)]
api: disks: allow zstd compression for zfs pools

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2 months agofix #3555: BTRFS: call DirPlugin's free_image correctly
Hannes Laimer [Fri, 30 Jul 2021 11:04:55 +0000 (13:04 +0200)]
fix #3555: BTRFS: call DirPlugin's free_image correctly

The method is only derived in the DirPlugin module from the base
Plugin, so we do not have it available there through a static module
method call using ::, but only when using a class dereference.

Other fix options would have been:

  PVE::Storage::Plugin::free_image(@_);

or:
  $class->SUPER::free_image($storeid, ...);

Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
[ Thomas: add some background to the commit message ]
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2 months agoapi: status: rework err cleanup
Thomas Lamprecht [Thu, 29 Jul 2021 15:12:02 +0000 (17:12 +0200)]
api: status: rework err cleanup

avoid open3 on local node.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2 months agoapi: status: import run_command
Thomas Lamprecht [Thu, 29 Jul 2021 15:11:41 +0000 (17:11 +0200)]
api: status: import run_command

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2 months agoapi: status: fix unlink on file upload
Lorenz Stechauner [Fri, 25 Jun 2021 07:22:40 +0000 (09:22 +0200)]
api: status: fix unlink on file upload

after an error while copying the file to its destination the local
path of the destination was unlinked in every case, even when on the
destination was copied to via scp.

Signed-off-by: Lorenz Stechauner <l.stechauner@proxmox.com>
2 months agoapi: status: sort/cleanup module use-statements
Thomas Lamprecht [Thu, 29 Jul 2021 09:55:26 +0000 (11:55 +0200)]
api: status: sort/cleanup module use-statements

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
3 months agostatus: add max length and disclaimer to file upload's filename
Lorenz Stechauner [Wed, 21 Jul 2021 12:52:07 +0000 (14:52 +0200)]
status: add max length and disclaimer to file upload's filename

Signed-off-by: Lorenz Stechauner <l.stechauner@proxmox.com>
3 months agostatus: add enum for file upload content type
Lorenz Stechauner [Wed, 21 Jul 2021 12:51:55 +0000 (14:51 +0200)]
status: add enum for file upload content type

the addition of this enum does not change API behaviour, because
it is checked for 'iso' or 'vztmpl' aftwerwards anyway.

Signed-off-by: Lorenz Stechauner <l.stechauner@proxmox.com>
3 months agobump version to 7.0-9
Thomas Lamprecht [Tue, 6 Jul 2021 14:28:24 +0000 (16:28 +0200)]
bump version to 7.0-9

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
3 months agoextract backup config: less precise matching for broken pipe detection
Fabian Ebner [Tue, 6 Jul 2021 13:47:29 +0000 (15:47 +0200)]
extract backup config: less precise matching for broken pipe detection

Extracting the config for zstd compressed vma files was broken:
    Failed to extract config from VMA archive: zstd: error 70 : Write
    error : cannot write decoded block : Broken pipe (500)
since the error message changed and wouldn't match anymore.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
3 months agobump version to 7.0-8
Thomas Lamprecht [Tue, 6 Jul 2021 10:41:42 +0000 (12:41 +0200)]
bump version to 7.0-8

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
3 months agolvm: wipe signatures on lvcreate
Stoiko Ivanov [Tue, 6 Jul 2021 09:50:26 +0000 (11:50 +0200)]
lvm: wipe signatures on lvcreate

With PVE 7.0 we use upstream's lvm2 packages, which seem to detect
'more' signatures (and refuse creating lvs when they are present)

This prevents creating new disks on LVM (thick) storages as reported
on pve-user [0].

Adding -Wy to wipe signatures, and --yes (to actually wipe them
instead of prompting) fixes the aborted lvcreate.

Adding only to LVMPlugin and not to the lvcreate calls in
LvmThinPlugin, since I assume (and my quick tests confirm) that thin
pools are not affected by this issue..

Tested on a virtual test-setup with a LVM storage on a (virtual) iscsi
target and a local lvmthin storage.

[0] https://lists.proxmox.com/pipermail/pve-user/2021-July/172660.html

Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
3 months agocifs: improve warning for password but no username set
Thomas Lamprecht [Tue, 6 Jul 2021 05:50:29 +0000 (07:50 +0200)]
cifs: improve warning for password but no username set

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
3 months agocifs: fix sensitive parameter name for on-update/add
Thomas Lamprecht [Tue, 6 Jul 2021 05:50:06 +0000 (07:50 +0200)]
cifs: fix sensitive parameter name for on-update/add

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
3 months agobump version to 7.0-7
Thomas Lamprecht [Mon, 5 Jul 2021 17:01:15 +0000 (19:01 +0200)]
bump version to 7.0-7

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
3 months agobtrfs: fix path_is_mounted invocation
Wolfgang Bumiller [Mon, 28 Jun 2021 06:40:03 +0000 (08:40 +0200)]
btrfs: fix path_is_mounted invocation

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
4 months agobump version to 7.0-6
Thomas Lamprecht [Thu, 24 Jun 2021 14:13:12 +0000 (16:13 +0200)]
bump version to 7.0-6

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
4 months agobtrfs: support newer prune-backups for backup retention
Thomas Lamprecht [Thu, 24 Jun 2021 14:11:48 +0000 (16:11 +0200)]
btrfs: support newer prune-backups for backup retention

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
4 months agobump version to 7.0-5
Thomas Lamprecht [Thu, 24 Jun 2021 09:43:59 +0000 (11:43 +0200)]
bump version to 7.0-5

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
4 months agobtrfs: add mkdir as option for now
Thomas Lamprecht [Thu, 24 Jun 2021 09:45:38 +0000 (11:45 +0200)]
btrfs: add mkdir as option for now

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
4 months agobtrfs: do not reuse DirPlugins activate_storage directrly
Thomas Lamprecht [Thu, 24 Jun 2021 09:15:28 +0000 (11:15 +0200)]
btrfs: do not reuse DirPlugins activate_storage directrly

as then the btrfs assertion would happen after we already created
subdirectories on some path, leaving those left-over..

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
4 months agobtrfs: check for btrfs in on_add_hook and activate
Wolfgang Bumiller [Thu, 24 Jun 2021 08:45:55 +0000 (10:45 +0200)]
btrfs: check for btrfs in on_add_hook and activate

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
4 months agobump version to 7.0-4
Thomas Lamprecht [Wed, 23 Jun 2021 20:51:02 +0000 (22:51 +0200)]
bump version to 7.0-4

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
4 months agoapi: file download: limit filename length and encode it for the worker id
Thomas Lamprecht [Wed, 23 Jun 2021 20:40:40 +0000 (22:40 +0200)]
api: file download: limit filename length and encode it for the worker id

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
4 months agoapi: file download: code cleanups
Thomas Lamprecht [Wed, 23 Jun 2021 20:40:05 +0000 (22:40 +0200)]
api: file download: code cleanups

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
4 months agostatus: add download_url method
Lorenz Stechauner [Tue, 22 Jun 2021 09:19:22 +0000 (11:19 +0200)]
status: add download_url method

uses common function PVE::Tools::download_file_from_url to download
iso files.

Only users with permissions `Sys.Audit` and `Sys.Modify` on `/` are
permitted to perform this action. This restriction is due to the
fact, that the download function is able to download files from
internal networks (which are not visible/accessible from outside).
Users with these permissions anyway have the means to alter node
(network) config, so this does not create any further security risk.

Signed-off-by: Lorenz Stechauner <l.stechauner@proxmox.com>
4 months agostatus: factoring out normalize_content_filename
Lorenz Stechauner [Tue, 22 Jun 2021 09:19:21 +0000 (11:19 +0200)]
status: factoring out normalize_content_filename

Signed-off-by: Lorenz Stechauner <l.stechauner@proxmox.com>
4 months agoapi: content: correctly handle warnings status for delayed task
Fabian Ebner [Wed, 12 May 2021 12:32:55 +0000 (14:32 +0200)]
api: content: correctly handle warnings status for delayed task

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
4 months agobtrfs: drop qcow2 and vmdk for now
Thomas Lamprecht [Wed, 23 Jun 2021 18:22:50 +0000 (20:22 +0200)]
btrfs: drop qcow2 and vmdk for now

the web-interface always prefers qcow2 once that is in the list,
itself a bug on it's own as the preferred one from the backend should
be preferred too, but still, vmdk support should not be extended we
can only cope with that in a limited way anyway, and both can always
get enabled later easily, if there's actual user-request for it.
Disabling is never that easy, at least if one cares about backward
compat.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
4 months agobtrfs: make NOCOW optional
Wolfgang Bumiller [Tue, 22 Jun 2021 12:18:23 +0000 (14:18 +0200)]
btrfs: make NOCOW optional

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
4 months agobtrfs: add 'btrfs' import/export format
Wolfgang Bumiller [Tue, 22 Jun 2021 12:18:22 +0000 (14:18 +0200)]
btrfs: add 'btrfs' import/export format

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
4 months agobump storage API: update import/export methods
Wolfgang Bumiller [Tue, 22 Jun 2021 12:18:21 +0000 (14:18 +0200)]
bump storage API: update import/export methods

Bumps APIVER to 9 and resets APIAGE to zero.

The import methods (volume_import, volume_import_formats):

These additionally get the '$snapshot' parameter which is
already present on the export side as an informational piece
to know which of the snapshots is the *current* one.
This parameter is inserted *in the middle* of the current
parameters, so the import & export format methods now have
the same signatures.
The current "disk" state will be set to this snapshot.
This, too, is required for our btrfs implementation.
  `volume_import_formats` can obviously not make much
*use* of this parameter, but it'll still be useful to know
that the information is actually available in the import
call, so its presence will be checked in the btrfs
implementation.

Currently this is intended to be used for btrfs send/recv
support, which in theory could also get additional metadata
similar to how we do the "tar+size" format, however, we
currently only really use this within this repository in
storage_migrate() which has this information readily
available anyway.

On the export side (volume_export, volume_export_formats):

The `$with_snapshots` option is now "defined" to be an
ordered array of snapshots to include, as a hint for
storages which need this. (As of the next commit this is
only btrfs, and only when also specifying a base snapshot,
which is a case we can currently not run into except on the
command line interface.)
  The current providers of the `with_snapshot` option will
still treat it as a boolean (since eg. for ZFS you cannot
really "skip" snapshots AFAIK).
  This is mainly intended for storages which do not have a
strong association between snapshots and the originals, or
an ordering (eg. btrfs and lvm-thin allow creating
arbitrary snapshot trees, and with btrfs you can even
create a "circular" connection between subvolumes, also we
could consider reflink based copies snapshots on xfs in
the future maybe?)

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
4 months agoadd BTRFS storage plugin
Wolfgang Bumiller [Tue, 22 Jun 2021 12:18:20 +0000 (14:18 +0200)]
add BTRFS storage plugin

This is mostly the same as a directory storage, with 2 major
differences:

* 'subvol' volumes are actual btrfs subvolumes and therefore
  allow snapshots
* 'raw' files are placed *into* a subvolume and therefore
  also allow snapshots, the raw file for volume
  `btrstore:100/vm-100-disk-1.raw` can be found under
  `$path/images/100/vm-100-disk-1/disk.raw`
* in both cases, snapshots add an '@name' suffix to the
  subvolume's directory name, so snapshot 'foo' of the above
  would be found under
  `$path/images/100/vm-100-disk-1@foo/disk.raw`
  or for format "subvol":
  `$path/images/100/subvol-100-disk-1.subvol@foo`

Note that qgroups aren't included in btrfs-send streams,
therefore for now we will only be using *unsized* subvolumes
for containers and place a regular raw+ext4 file for sized
containers.
We could extend the import/export stream format to include
the information at the front (similar to how we do the
"tar+size" format, but we need to include the size of all
the contained snapshots as well, since they can technically
change). (But before enabling quotas we should do some
performance testing on bigger file systems with multiple
snapshots as there are quite a few reports of the fs slowing
down considerably in such scenarios).

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
4 months agofactoring out regex for vztmpl
Lorenz Stechauner [Wed, 23 Jun 2021 13:10:15 +0000 (15:10 +0200)]
factoring out regex for vztmpl

stores the regex definition in PVE::Storage.

One test had to be adapted because it tested obsolete code. Namely:
it expects vztmpl to only end with .tar.gz, but the new regex also
includes .tar.xz, there is nothing against allowing .tar.xz files as
vztmpl files.

Signed-off-by: Lorenz Stechauner <l.stechauner@proxmox.com>
4 months agofile size info: return early if we cannot parse json
Thomas Lamprecht [Wed, 23 Jun 2021 06:25:20 +0000 (08:25 +0200)]
file size info: return early if we cannot parse json

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
4 months agofile/volume size info: add actual errors to untaint messages
Thomas Lamprecht [Wed, 23 Jun 2021 06:24:58 +0000 (08:24 +0200)]
file/volume size info: add actual errors to untaint messages

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
4 months agoplugins: untaint volume_size_info retuns
Stoiko Ivanov [Tue, 22 Jun 2021 16:39:54 +0000 (18:39 +0200)]
plugins: untaint volume_size_info retuns

the size returned by volume_size_info is used for creating the new
destination image in PVE::QemuServer::clone_disk (and probably
elsewhere). In certain cases the return values are tainted - they are
obtained by a run_command call and depending on the format and length
of the parsed output can still have their tainted attribute.

One example of a tainted return has been reported in our
community-forum:
https://forum.proxmox.com/threads/cannot-clone-vm-or-move-disk-with-more-than-13-snapshots.89628/

A qcow2 image with 13 snapshots generates a output > 4k in length from
`qemu-img info --output=json`, which in turn causes the output to be
considered tainted.

This patch untaints the returns where applicable. The other
storage-plugins are not affected:
* LVMPlugin returns a single number and a newline (thus gets untainted
  by run_command)
* RBDPlugin untaints the complete json before decoding
* ZFSPoolplugin and ISCSIDirectPlugin explicitly untaint their
  returns.

Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
4 months agotree-wide: fix typos with codespell
Thomas Lamprecht [Wed, 23 Jun 2021 06:07:20 +0000 (08:07 +0200)]
tree-wide: fix typos with codespell

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
4 months agopbs: fix typo
Fabian Grünbichler [Tue, 22 Jun 2021 11:44:06 +0000 (13:44 +0200)]
pbs: fix typo

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
4 months agobump version to 7.0-3
Thomas Lamprecht [Mon, 21 Jun 2021 09:30:10 +0000 (11:30 +0200)]
bump version to 7.0-3

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
4 months agoconfig: prevent empty content list when content type 'none' is not supported
Fabian Ebner [Fri, 18 Jun 2021 10:59:36 +0000 (12:59 +0200)]
config: prevent empty content list when content type 'none' is not supported

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
4 months agovdisk_list: only scan storages with the correct content type(s)
Fabian Ebner [Fri, 18 Jun 2021 10:59:35 +0000 (12:59 +0200)]
vdisk_list: only scan storages with the correct content type(s)

The enabled check in the lower loop is now redundant and can be removed.

If storeid is provided, initialize the result hash accordingly, mainly for
backwards compatibility (needed by a caller in pve-manager's Ceph/Pools.pm and
the migration code in pve-container and qemu-server), but it also is less
surprising in general.

Remaining vdisk_list users that do not specify a content type are:
    1. pve-manager's Pool/Ceph.pm, but the content type for RBD can only be
       rootdir and images, so the storage is scanned (if enabled, same as
       before).
    2. pve-container migration
    3. qemu-server migration
For the latter two, it's planned to enforce content type, so the change is fine
too.

This also means that for iscsi(direct) plugins with content type 'none', i.e.
"use LUNs directly" does not return the list of images anymore, but that was
rather a bug anyways as they're not virtual disks then:
    0.0.0.scsi-36001405b8f2772e13a04b8e9390db13d
All of the remaining callers not using content types (see above) are fine with
that change too.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
4 months agolvm: volume import: handle worker returned by free_image
Fabian Ebner [Tue, 4 May 2021 09:52:54 +0000 (11:52 +0200)]
lvm: volume import: handle worker returned by free_image

only affects LVM storages with 'saferemove 1' where the import fails at a rather
advanced stage. Previously in such cases, the renamed (by free_image) volume
del-vm-XYZ-disk-N would be left over.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
4 months agopbs: free image: explicitly return undef
Fabian Ebner [Tue, 4 May 2021 09:52:53 +0000 (11:52 +0200)]
pbs: free image: explicitly return undef

Storage.pm's vdisk_free interprets truthy return values as worker subs, so be
explicit about returning undef here. Not an issue at the moment, because
run_client_command already returns undef, but better be safe than sorry.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
4 months agoapi: status: sort index and add missing "file-restore"
Thomas Lamprecht [Mon, 21 Jun 2021 07:31:17 +0000 (09:31 +0200)]
api: status: sort index and add missing "file-restore"

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
4 months agoplugin loader: text-width cleanup
Thomas Lamprecht [Fri, 18 Jun 2021 16:33:20 +0000 (18:33 +0200)]
plugin loader: text-width cleanup

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
4 months agopostinst: remove old file if new one is identical
Fabian Ebner [Thu, 17 Jun 2021 08:58:26 +0000 (10:58 +0200)]
postinst: remove old file if new one is identical

Suggested-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
4 months agopostinst: avoid spawning subshell
Fabian Ebner [Thu, 17 Jun 2021 08:58:25 +0000 (10:58 +0200)]
postinst: avoid spawning subshell

which makes the continue not behave as intended.

Reported by shellcheck: SC2106: This [i.e. continue] only exits the subshell
caused by the (..) group

Also factor out long message for readability.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
4 months agoapi: content/delete: die with newline to avoid addign file-context
Thomas Lamprecht [Wed, 16 Jun 2021 17:24:24 +0000 (19:24 +0200)]
api: content/delete: die with newline to avoid addign file-context

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
4 months agocephfs: update reminder for systemd_netmount removal
Fabian Ebner [Wed, 16 Jun 2021 07:27:00 +0000 (09:27 +0200)]
cephfs: update reminder for systemd_netmount removal

Commit d9ece228fbbbf0eb575c8a067a70ed7023078f84 introduced the workaround with
using systemd units and 25e222ca0d96a0da0d491e749b437d2822215e9e re-used the
functionality for fuse-mounts too.

The latter commit suggests to switch to using mount.fuse.ceph for the '_netdev'
option, but it doesn't seem to work:

 root@pve701 / # mount -t fuse.ceph 10.10.10.11,10.10.10.12,10.10.10.13:/ /mnttest/fuse -o 'ceph.id=admin,ceph.keyfile=/etc/pve/priv/ceph/cephfs.secret,ceph.conf=/etc/pve/ceph.conf,_netdev'
 ceph-fuse[20729]: starting ceph client
 2021-06-15T14:22:00.631+0200 7f995f878080 -1 init, newargv = 0x55e09fc11a40 newargc=11
 ceph-fuse[20729]: starting fuse
 root@pve701 / # mount -t ceph 10.10.10.11,10.10.10.12,10.10.10.13:/ /mnttest/normal -o 'name=admin,secretfile=/etc/pve/priv/ceph/cephfs.secret,conf=/etc/pve/ceph.conf,_netdev'
 root@pve701 / # mount | grep mnttest
 ceph-fuse on /mnttest/fuse type fuse.ceph-fuse (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
 10.10.10.11,10.10.10.12,10.10.10.13:/ on /mnttest/normal type ceph (rw,relatime,name=admin,secret=<hidden>,acl,_netdev)

Also, the return value is not propagated by mount.fuse.ceph, meaning the output
would need to be parsed...

 root@pve701 ~ # mount -t fuse.ceph 10.10.10.11,10.10.10.12,10.10.10.13:/ /mnttest/fuse -o 'ceph.id=admin,ceph.keyfile=/etc/pve/priv/ceph/cephfs.secret,ceph.conf=/etc/pve/ceph.conf,_netdev'
 2021-06-15T14:42:56.326+0200 7f634edae080 -1 init, newargv = 0x560cdb5e0a40 newargc=11
 ceph-fuse[34480]: starting ceph client
 fuse: mountpoint is not empty
 fuse: if you are sure this is safe, use the 'nonempty' mount option
 ceph-fuse[34480]: fuse failed to start
 2021-06-15T14:42:56.338+0200 7f634edae080 -1
 fuse_mount(mountpoint=/mnttest/fuse) failed.
 Mount failed with status code: 5
 root@pve701 ~ # echo $?
 0

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
4 months agocephfs: revert safe-guard check for Luminous
Fabian Ebner [Wed, 16 Jun 2021 07:26:59 +0000 (09:26 +0200)]
cephfs: revert safe-guard check for Luminous

It's necessary to be on Nautilus before upgrading to 7.x, so the check is no
longer needed. See commit e54c3e334760491954bc42f3585a8b5b136d4b1d. It didn't
cleanly revert, because there were cleanups made afterwards.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
4 months agoconfig: add backup content type to default local storage
Fabian Ebner [Wed, 16 Jun 2021 07:26:58 +0000 (09:26 +0200)]
config: add backup content type to default local storage

which is used if there is no ('dir'-type) 'local' entry. Storage configurations
made by the installer also support backups for the 'local' storage, and the
'prune-backups' parameter is not really useful otherwise.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
4 months agoconfig: mention that maxfiles is deprecated
Fabian Ebner [Wed, 16 Jun 2021 07:26:57 +0000 (09:26 +0200)]
config: mention that maxfiles is deprecated

Don't add an explicit deprecation warning on parsing (yet), this already done in
the pve6to7 script. Also, automatic conversion to 'prune-backups' happens when
the section config is read, so over time fewer users should be affected.
Postpone explicit warning/dropping the parameter to a future major release.

Also switch the setting for the default 'local' storage to 'prune-backups'.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
4 months agopostinst: move cifs credential files into subdirectory upon update
Fabian Ebner [Wed, 16 Jun 2021 07:26:56 +0000 (09:26 +0200)]
postinst: move cifs credential files into subdirectory upon update

and drop the compat code.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
4 months agofix find_free_disk_name invocations
Wolfgang Bumiller [Wed, 9 Jun 2021 13:18:46 +0000 (15:18 +0200)]
fix find_free_disk_name invocations

The interface takes the storeid now, not the image dir.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
4 months agoprune backups: activate storage
Fabian Ebner [Fri, 16 Apr 2021 08:51:27 +0000 (10:51 +0200)]
prune backups: activate storage

which also checks whether the storage is even enabled. VZDump jobs already
activate the storage, but more direct calls via API/CLI didn't do so yet.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
4 months agodiskmanage: fix determining array length
Fabian Ebner [Mon, 14 Jun 2021 11:40:40 +0000 (13:40 +0200)]
diskmanage: fix determining array length

$#* is the last index, not the length.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
4 months agoapi: get rid of moved 'usb' call
Fabian Ebner [Fri, 4 Jun 2021 13:49:33 +0000 (15:49 +0200)]
api: get rid of moved 'usb' call

pve-manger commit bd328734deb1dcea296858bb38d085e392adb99e changed the frontend
to use the new call.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
4 months agobump version to 7.0-2
Thomas Lamprecht [Wed, 2 Jun 2021 14:32:23 +0000 (16:32 +0200)]
bump version to 7.0-2

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
4 months agodisks: fixup join usage
Thomas Lamprecht [Wed, 2 Jun 2021 12:19:53 +0000 (14:19 +0200)]
disks: fixup join usage

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
4 months agodisks: wipe blockdev: pass all child partitions to wipefs
Thomas Lamprecht [Wed, 2 Jun 2021 11:13:26 +0000 (13:13 +0200)]
disks: wipe blockdev: pass all child partitions to wipefs

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>