]> git.proxmox.com Git - pve-storage.git/log
pve-storage.git
3 years agoFix: path_to_volume_id returned wrong content
Alwin Antreich [Tue, 28 Apr 2020 13:58:20 +0000 (15:58 +0200)]
Fix: path_to_volume_id returned wrong content

type for backup files. Patch includes changes of the test as well.

Signed-off-by: Alwin Antreich <a.antreich@proxmox.com>
3 years agotest: path_to_volume_id
Alwin Antreich [Tue, 28 Apr 2020 13:58:19 +0000 (15:58 +0200)]
test: path_to_volume_id

Test to reduce the potential for accidental breakage on regex changes.

Signed-off-by: Alwin Antreich <a.antreich@proxmox.com>
3 years agoFix: backup: ctime was from stat not file name
Alwin Antreich [Tue, 28 Apr 2020 13:58:18 +0000 (15:58 +0200)]
Fix: backup: ctime was from stat not file name

The vzdump file was passed with the full path to the regex. That regex
captures the time from the file name, to calculate the epoch.

As the regex didn't match, the ctime from stat was taken instead. This
resulted in the ctime shown when the file was changed, not when the
backup was made.

Signed-off-by: Alwin Antreich <a.antreich@proxmox.com>
3 years agotest: list_volumes
Alwin Antreich [Tue, 28 Apr 2020 13:58:17 +0000 (15:58 +0200)]
test: list_volumes

Test to reduce the potential for accidental breakage on regex changes.

Co-Authored-by: Dominic Jaeger <d.jaeger@proxmox.com>
Signed-off-by: Alwin Antreich <a.antreich@proxmox.com>
3 years agotest: parse_volname
Alwin Antreich [Tue, 28 Apr 2020 13:58:16 +0000 (15:58 +0200)]
test: parse_volname

Test to reduce the potential for accidental breakage on regex changes.
And to make sure that all vtype_subdirs are parsed.

Signed-off-by: Alwin Antreich <a.antreich@proxmox.com>
3 years agostorage: replace build-in stat occurrences
Alwin Antreich [Tue, 28 Apr 2020 13:58:15 +0000 (15:58 +0200)]
storage: replace build-in stat occurrences

with File::stat::stat to minimize variable declarations. And allow to
mock this method in tests instead of the perl build-in stat.

Signed-off-by: Alwin Antreich <a.antreich@proxmox.com>
3 years agostorage: test: split archive format/compressor
Alwin Antreich [Tue, 28 Apr 2020 13:58:13 +0000 (15:58 +0200)]
storage: test: split archive format/compressor

detection into separate functions so they are reusable and easier
modifiable. This patch also adds the test for archive_info.

Signed-off-by: Alwin Antreich <a.antreich@proxmox.com>
4 years agos/ceph_version/local_ceph_version/ for clarity
Thomas Lamprecht [Sat, 25 Apr 2020 09:37:26 +0000 (11:37 +0200)]
s/ceph_version/local_ceph_version/ for clarity

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
4 years agod/control: bump ceph dependency to 12.2
Thomas Lamprecht [Sat, 25 Apr 2020 09:27:30 +0000 (11:27 +0200)]
d/control: bump ceph dependency to 12.2

A newer than the Luminous version is shipped with buster, and our
ceph repos are on Nautilus (14.2) in PVE 6.

Allows to drop a check for really old ceph versions (< 10, so
Infernalis and older).

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
4 years agofollowup: only parse version if required, fix whitespace error
Thomas Lamprecht [Sat, 25 Apr 2020 09:18:03 +0000 (11:18 +0200)]
followup: only parse version if required, fix whitespace error

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
4 years agoFix #2705: cephfs: mount fails with bad option
Alwin Antreich [Fri, 24 Apr 2020 15:29:47 +0000 (17:29 +0200)]
Fix #2705: cephfs: mount fails with bad option

dmesg: libceph: bad option at 'conf=/etc/pve/ceph.conf'

After the upgrade to PVE 6 with Ceph Luminous, the mount.ceph helper
doesn't understand the conf= option yet. And the CephFS mount with the
kernel client fails. After upgrading to Ceph Nautilus the option exists
in the mount.ceph helper.

Signed-off-by: Alwin Antreich <a.antreich@proxmox.com>
4 years agoconfig: add missing whitespace
Thomas Lamprecht [Sat, 25 Apr 2020 09:12:48 +0000 (11:12 +0200)]
config: add missing whitespace

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
4 years agofix nvme wearout parsing
Dominik Csapak [Tue, 14 Apr 2020 14:17:33 +0000 (16:17 +0200)]
fix nvme wearout parsing

the '.*' was greedy, also consuming all but one digits of the real percentage

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
switch to \s* instead of .*?, to prevent mis-interpreting potential
strings like '< 50%' or '0-50%'

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
4 years agodisk_tests: improve nvme smart test
Dominik Csapak [Tue, 14 Apr 2020 14:17:32 +0000 (16:17 +0200)]
disk_tests: improve nvme smart test

by using an actual percentage for 'Percentage Used' instead of 0%

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
4 years agoAdd comment for volume_has_feature
Fabian Ebner [Wed, 8 Apr 2020 11:40:07 +0000 (13:40 +0200)]
Add comment for volume_has_feature

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
4 years agoZFS: use -p flag where possible
Aaron Lauterer [Tue, 7 Apr 2020 12:25:37 +0000 (14:25 +0200)]
ZFS: use -p flag where possible

Signed-off-by: Aaron Lauterer <a.lauterer@proxmox.com>
4 years agoZFS: use -p flag and remove zfs_parse_size
Aaron Lauterer [Tue, 7 Apr 2020 12:25:36 +0000 (14:25 +0200)]
ZFS: use -p flag and remove zfs_parse_size

ZFS supports the -p flag in the list command since a few years now.
Let us use the real byte values and avoid the error prone calculation
from human readable numbers that can lead to incorrect numbers if the
reported human readable value is a rounded number.

Signed-off-by: Aaron Lauterer <a.lauterer@proxmox.com>
4 years agoZFSPoolPlugin: fix #2662 get volume size correctly
Aaron Lauterer [Tue, 7 Apr 2020 12:25:35 +0000 (14:25 +0200)]
ZFSPoolPlugin: fix #2662 get volume size correctly

Getting the volume sizes as byte values instead of converted to human
readable units helps to avoid rounding errors in the further processing
if the volume size is more on the odd side.

The `zfs list` command supports the -p(arseable) flag since a few years
now.
When returning the size in bytes there is no  calculation performed and
thus we need to explicitly cast the size to an integer before returning
it.

Signed-off-by: Aaron Lauterer <a.lauterer@proxmox.com>
4 years agobump version to 6.1-6
Fabian Grünbichler [Thu, 9 Apr 2020 06:43:17 +0000 (08:43 +0200)]
bump version to 6.1-6

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
4 years agod/control: wrap-and-sort
Fabian Grünbichler [Thu, 9 Apr 2020 06:38:31 +0000 (08:38 +0200)]
d/control: wrap-and-sort

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
4 years agod/control: break consumers of storage_migrate
Fabian Grünbichler [Thu, 9 Apr 2020 06:38:04 +0000 (08:38 +0200)]
d/control: break consumers of storage_migrate

since we changed the signature, and want to ensure smooth upgrades.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
4 years agostorage_migrate: add volname_for_storage helper
Fabian Ebner [Wed, 8 Apr 2020 09:25:14 +0000 (11:25 +0200)]
storage_migrate: add volname_for_storage helper

to guess a valid volname for a targetstorage of a different type.
This makes it possible to migrate raw volumes between 'dir' and 'lvm'
storages.

It is only used when the storage type for the source storage X
and target storage Y differ and should work as long as Y uses
the standard naming scheme (VMID/vm-VMID-name.fmt respectively vm-VMID-name).
If it doesn't, we get an invalid name and fail, which is the old
behavior (except if X and Y have different types but the same
non-standard naming-scheme, where the old behavior did work).

The original name is preserved, except for a possible extension
and it's also checked if the format is valid for the target storage.
Example: mylvm:vm-123-disk-4 <-> mydir:123/vm-123-disk-4.raw

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
4 years agoIntroduce allow_rename parameter for pvesm import and storage_migrate
Fabian Ebner [Wed, 8 Apr 2020 09:25:05 +0000 (11:25 +0200)]
Introduce allow_rename parameter for pvesm import and storage_migrate

and also return the ID of the allocated volume. This option
allows plugins to choose a new name if there is a collision.

In storage_migrate, the API version of the receiving side is checked.

In Storage.pm's volume_import, when a plugin returns 'undef',
it can be assumed that the import with the requested volid was
successful (it should've died otherwise) and so volid is returned.
This is done for backwards compatibility with foreign plugins.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
4 years agoAdd apiinfo helper to pvesm
Fabian Ebner [Wed, 8 Apr 2020 09:25:04 +0000 (11:25 +0200)]
Add apiinfo helper to pvesm

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
4 years agoCollect optional parameters for storage_migrate into $opts
Fabian Ebner [Wed, 8 Apr 2020 09:24:59 +0000 (11:24 +0200)]
Collect optional parameters for storage_migrate into $opts

Sanitizing $with_snapshots is done on extraction to save a line.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
4 years agoFix 2647: Add snippet content type for Gluster
Dominic Jäger [Thu, 26 Mar 2020 09:33:28 +0000 (10:33 +0100)]
Fix 2647: Add snippet content type for Gluster

Our wiki mentions snippets as supported content type for GlusterFS storages [0]
and all other directory based storages have it enabled already [1]

[0] https://pve.proxmox.com/wiki/Storage:_GlusterFS
[1] https://git.proxmox.com/?p=pve-storage.git;a=commit;h=d1eb35ea74cf27713625ab7e7c3767a8254a4aee

Signed-off-by: Dominic Jäger <d.jaeger@proxmox.com>
4 years agofix #2474: always show iscsi content
Dominik Csapak [Tue, 7 Apr 2020 07:25:24 +0000 (09:25 +0200)]
fix #2474: always show iscsi content

Instead of relying on list_volumes of Plugin.pm (which filters by
the content types set in the config), use our own to always
show the luns of an iscsi.

This makes sense here, since we need it to show the luns when using
it as base storage for LVM (where we have content type 'none' set).

It does not interfere with the rest of the GUI, since on e.g. disk
creation, we already filter the storages in the dropdown by content
type, iow. an iscsi storage used this way still does not show up
when trying to create a disk.

This also shows the luns now in the 'Content' tab, but this is also
OK, since the user cannot actually do anything there with the luns.
(Besides looking at them)

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
4 years agostorage_migrate: check if target storage supports content type
Fabian Ebner [Thu, 26 Mar 2020 08:09:31 +0000 (09:09 +0100)]
storage_migrate: check if target storage supports content type

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
4 years agoAllow passing options to volume_has_feature
Fabian Ebner [Mon, 23 Mar 2020 11:18:50 +0000 (12:18 +0100)]
Allow passing options to volume_has_feature

With the option valid_target_formats it's possible
to let the caller specify possible formats for the target
of an operation.
[0]: If the option is not set, assume that every format is valid.

In most cases the format of the the target and the format
of the source will agree (and therefore assumption [0] is
not actually assuming very much and ensures backwards
compatability). But when cloning a volume on a storage
using Plugin.pm's implementation (e.g. directory based
storages), the result is always a qcow2 image.

When cloning containers, the new option can be used to detect
that qcow2 is not valid and hence the clone feature is not
available.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
4 years agobase plugin: get_subdir_files: split stat variables into single lines
Thomas Lamprecht [Fri, 6 Mar 2020 18:27:24 +0000 (19:27 +0100)]
base plugin: get_subdir_files: split stat variables into single lines

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
4 years agoapi: storage/content: fix return schema property descriptions
Thomas Lamprecht [Fri, 6 Mar 2020 18:24:32 +0000 (19:24 +0100)]
api: storage/content: fix return schema property descriptions

a small grammar fix, and we now return ctime of all files, as
remaining storages are planned for the future omit this hint
completely.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
4 years agoPBS plugin: code cleanup
Thomas Lamprecht [Fri, 6 Mar 2020 18:15:53 +0000 (19:15 +0100)]
PBS plugin: code cleanup

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
4 years agoLVM list_images: return creation time
Dietmar Maurer [Wed, 4 Mar 2020 10:46:22 +0000 (11:46 +0100)]
LVM list_images: return creation time

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
4 years agobase plugin: return ctime for vm images
Dietmar Maurer [Wed, 4 Mar 2020 10:46:21 +0000 (11:46 +0100)]
base plugin: return ctime for vm images

Changed file_size_info() to additionally return ctime to avoid
another stat() call.

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
4 years agobase plugin: add ctime for all files
Dietmar Maurer [Wed, 4 Mar 2020 10:46:20 +0000 (11:46 +0100)]
base plugin: add ctime for all files

Creation time makes sense for other file types also.

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
4 years agoPBS Plugin: list_volumes: add ctime
Dietmar Maurer [Wed, 4 Mar 2020 10:46:18 +0000 (11:46 +0100)]
PBS Plugin: list_volumes: add ctime

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
4 years agobase plugin: add ctime for backup files
Dietmar Maurer [Wed, 4 Mar 2020 10:46:17 +0000 (11:46 +0100)]
base plugin: add ctime for backup files

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
4 years agoapi: storage/content: add ctime to return schema
Dietmar Maurer [Wed, 4 Mar 2020 10:46:19 +0000 (11:46 +0100)]
api: storage/content: add ctime to return schema

to allow implementation from plugins in future patches

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
4 years agoiscis: add iscsi_session helper
Thomas Lamprecht [Tue, 3 Mar 2020 09:49:11 +0000 (10:49 +0100)]
iscis: add iscsi_session helper

allows to write some code sligthly nicer

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
4 years agoiscsi: sort and split module usage
Thomas Lamprecht [Tue, 3 Mar 2020 09:43:02 +0000 (10:43 +0100)]
iscsi: sort and split module usage

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
4 years agoISCSI: whitespace cleanup
Aaron Lauterer [Mon, 2 Mar 2020 07:52:06 +0000 (08:52 +0100)]
ISCSI: whitespace cleanup

Signed-off-by: Aaron Lauterer <a.lauterer@proxmox.com>
4 years agofix #2620: storage API: iSCSI: return active field as integer
Aaron Lauterer [Mon, 2 Mar 2020 07:52:05 +0000 (08:52 +0100)]
fix #2620: storage API: iSCSI: return active field as integer

If active, the return value was a string: "1" and not an integer.

Signed-off-by: Aaron Lauterer <a.lauterer@proxmox.com>
4 years agofollowup: fix VMID regex, use same as JSONSchema does
Thomas Lamprecht [Fri, 21 Feb 2020 14:58:06 +0000 (15:58 +0100)]
followup: fix VMID regex, use same as JSONSchema does

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
4 years agolist_volumes: try to return vmid also for backups
Dominik Csapak [Fri, 21 Feb 2020 10:02:09 +0000 (11:02 +0100)]
list_volumes: try to return vmid also for backups

this way the content listing api also returns the vmid on content
listings which, among other things, is useful for the gui for
filtering

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
4 years agobump version to 6.1-5
Thomas Lamprecht [Thu, 20 Feb 2020 12:46:24 +0000 (13:46 +0100)]
bump version to 6.1-5

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
4 years agocifs: followup fix for credential fallback
Thomas Lamprecht [Thu, 20 Feb 2020 12:55:43 +0000 (13:55 +0100)]
cifs: followup fix for credential fallback

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
4 years agocifs: move password credential file to storage subdirectory
Thomas Lamprecht [Thu, 20 Feb 2020 12:07:06 +0000 (13:07 +0100)]
cifs: move password credential file to storage subdirectory

Do not pollute top-level private directory, use "storage" folder but
with backward compatibility.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
4 years agoPBSPlugin.pm: fix password handling using new on_update_hook
Dietmar Maurer [Thu, 20 Feb 2020 11:38:21 +0000 (12:38 +0100)]
PBSPlugin.pm: fix password handling using new on_update_hook

4 years agoCIFSPlugin.pm: fix crediential handling using new on_update_hook
Dietmar Maurer [Thu, 20 Feb 2020 11:33:59 +0000 (12:33 +0100)]
CIFSPlugin.pm: fix crediential handling using new on_update_hook

4 years agoPVE/Storage/Plugin.pm: introduce on_update_hook
Dietmar Maurer [Thu, 20 Feb 2020 11:33:58 +0000 (12:33 +0100)]
PVE/Storage/Plugin.pm: introduce on_update_hook

We need this to correctly update the password file.

4 years agopbs: ensure storage secret file directory exists
Thomas Lamprecht [Thu, 20 Feb 2020 10:12:29 +0000 (11:12 +0100)]
pbs: ensure storage secret file directory exists

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
4 years agoPBSPlugin.pm - extract_vzdump_config: fix call to run_raw_client_cmd
Dietmar Maurer [Thu, 20 Feb 2020 09:45:34 +0000 (10:45 +0100)]
PBSPlugin.pm - extract_vzdump_config: fix call to run_raw_client_cmd

4 years agovolume_resize: align size to 1 KiB
Fabian Ebner [Wed, 19 Feb 2020 10:31:31 +0000 (11:31 +0100)]
volume_resize: align size to 1 KiB

1. Avoids the error
qemu-img: The new size must be a multiple of 512
for qcow2 disks.
2. Because volume_import expects disk sizes to be a multiple of 1 KiB.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
4 years agonamespace storage specific secret files to 'priv/storage' folder
Thomas Lamprecht [Wed, 19 Feb 2020 13:51:00 +0000 (14:51 +0100)]
namespace storage specific secret files to 'priv/storage' folder

As /etc/pve/priv is already pretty polluted, having a
"<storage-id>.pw" file there smells like it could make problems in
the future.

So let the pbs pw file generator use /etc/pve/priv/storages as base
path.
Other storage should move also to that path in the future, if they
save such secrets anywhere in /etc/pve.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
4 years agocheck if client executable ist installed before running command
Thomas Lamprecht [Wed, 19 Feb 2020 13:50:42 +0000 (14:50 +0100)]
check if client executable ist installed before running command

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
4 years agouse one liner closure for outfunc
Thomas Lamprecht [Wed, 19 Feb 2020 13:50:04 +0000 (14:50 +0100)]
use one liner closure for outfunc

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
4 years agoindentation fixes
Thomas Lamprecht [Wed, 19 Feb 2020 13:49:38 +0000 (14:49 +0100)]
indentation fixes

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
4 years agoimplement extract_vzdump_config for PBSPlugin
Dietmar Maurer [Wed, 19 Feb 2020 12:55:28 +0000 (13:55 +0100)]
implement extract_vzdump_config for PBSPlugin

4 years agoPVE/Storage/PBSPlugin.pm: start new proxmox backup server plugin
Dietmar Maurer [Wed, 19 Feb 2020 12:55:27 +0000 (13:55 +0100)]
PVE/Storage/PBSPlugin.pm: start new proxmox backup server plugin

4 years agoCheck whether 'zfs get mountpoint' returns a valid absolute path
Fabian Ebner [Tue, 18 Feb 2020 10:14:59 +0000 (11:14 +0100)]
Check whether 'zfs get mountpoint' returns a valid absolute path

The command 'zfs get mountpoint' can return 'none' and so 'mountpoint
none' was written to storage.cfg, which would block the fall-back to
using the default mount point when requesting a path, see [0].

[0]: https://forum.proxmox.com/threads/zfs-backup-with-snapshot-mode-fails.61927/#post-284123

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
4 years agosystemd unit name escape helpers moved to common, use them
Thomas Lamprecht [Wed, 5 Feb 2020 16:14:39 +0000 (17:14 +0100)]
systemd unit name escape helpers moved to common, use them

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
4 years agodirectory/cephfs: sort module usage
Thomas Lamprecht [Wed, 5 Feb 2020 16:13:30 +0000 (17:13 +0100)]
directory/cephfs: sort module usage

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
4 years agostorage_migrate: only set errfunc for send stream
Fabian Grünbichler [Wed, 5 Feb 2020 08:37:09 +0000 (09:37 +0100)]
storage_migrate: only set errfunc for send stream

since we redirect the output to our (insecure) socket, logfunc is only
used for STDERR anyway, so we might as well make it explicit on the
caller side.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
4 years agostorage_migrate: also log with an insecure connection if there is a log function
Fabian Ebner [Wed, 29 Jan 2020 13:30:02 +0000 (14:30 +0100)]
storage_migrate: also log with an insecure connection if there is a log function

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
4 years agoRemove unused string
Fabian Ebner [Wed, 29 Jan 2020 13:30:00 +0000 (14:30 +0100)]
Remove unused string

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
4 years agobump version to 6.1-4
Thomas Lamprecht [Wed, 29 Jan 2020 19:04:03 +0000 (20:04 +0100)]
bump version to 6.1-4

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
4 years agocephcfg sort keys in write_ceph_config
Thomas Lamprecht [Wed, 29 Jan 2020 18:59:08 +0000 (19:59 +0100)]
cephcfg sort keys in write_ceph_config

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
4 years agocephfs mount: reload systemd if existing unit gets regenerated
Thomas Lamprecht [Wed, 29 Jan 2020 18:48:49 +0000 (19:48 +0100)]
cephfs mount: reload systemd if existing unit gets regenerated

One the first write bringing the unit file in existence we can just
start it, after that we need to tell systemd that we want to actively
reload it.

While this is slightly shaky due to the fact that we do not check all
paths where such a unit could reside, it is something we can do
because earlier one couldn't have a unit/overwrite anyway (from
procfs mountinfo generated one do not support that) and does adding
such override ones from now on should work.

Also note that we can only get here in the "user does no weird stuff"
case when "cephfs_is_mounted" actively tells that there is no cephfs
mounted at the $mountpoint - at which time we can safely re-write the
potential updated unit file, reload and mount again.

So let's make our life a bit easier here until a user actually
complains about a rational issue for this, maybe we have PVE 7.0 then
and can get rid of that anyway :)

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
4 years agocephfs: mount fuse through systemd with correct order dependencies
Thomas Lamprecht [Wed, 29 Jan 2020 18:41:09 +0000 (19:41 +0100)]
cephfs: mount fuse through systemd with correct order dependencies

This fixes a potential races where fuse get's unmouted to late in the
shutdown process, i.e., at a time where network was down and it could
not talk to any MDS or monitor anymore.

We could fix it the same way we did once with the kernel based mount,
i.e., adding _netdev, but doing so would require to switch over from
"ceph-fuse" to "mount.fuse.ceph" which has better compatibility with
the common mount tool API.

As that helper exists we can reuse the newer systemd_netmount
ephemeral unit generator, only some options differ in name between
fuse and kernel variant.

So besides solving a potential issue we get a more unified handling
of those two cases.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
4 years agofix random hangs on reboot with active CephFS mount ordering cycle
Thomas Lamprecht [Wed, 29 Jan 2020 17:02:13 +0000 (18:02 +0100)]
fix random hangs on reboot with active CephFS mount ordering cycle

commit 54e0b0034bd6654c566cb4ae7d4a5953c48cd1ca introduced the
"_netdev" option, for PVE 5.3. The systemd generator then correctly
resolved that in the following resulting order-dependencies:
> Wants=network-online.target
> Before=umount.target remote-fs.target
> After=remote-fs-pre.target system.slice network.target network-online.target -.mount

This worked well and all were happy. With the current systemd in 6.0
we sometimes get the local-fs ones there generated too. This is a
fallout from a try to better handling nested mount hierachies, where
a .mount unit needs to be mounter or unmounted, before or after,
respectively, the parent mount was processed. It seems that sometime
that glitches and thus a "RequireMountFor=/mnt/pve" gets thrown in
and result sometimes in the local-fs order constraints being added.

The issue now is, that one must not have ordering depends to all,
local-fs, local-fs-pre, remote-fs, remote-fs-pre, as that gets you a
ordering cycle. Systemd tries to solve that cycle by randomly
dropping one constraint and retrying. By luck this is a not so
important unit, and all goes on well. Most of the time one isn't that
lucky and something important gets dropped, for example:

> Jan 24 18:43:05 prod1 systemd[1]: sysinit.target: Found ordering cycle on systemd-timesyncd.service/stop
> Jan 24 18:43:05 prod1 systemd[1]: sysinit.target: Found dependency on systemd-tmpfiles-setup.service/stop
> Jan 24 18:43:05 prod1 systemd[1]: sysinit.target: Found dependency on local-fs.target/stop
> Jan 24 18:43:05 prod1 systemd[1]: sysinit.target: Found dependency on mnt-pve-cephfs.mount/stop
> Jan 24 18:43:05 prod1 systemd[1]: sysinit.target: Found dependency on remote-fs-pre.target/stop
> Jan 24 18:43:05 prod1 systemd[1]: sysinit.target: Found dependency on rbdmap.service/stop
> Jan 24 18:43:05 prod1 systemd[1]: sysinit.target: Found dependency on sysinit.target/stop
> Jan 24 18:43:05 prod1 systemd[1]: sysinit.target: Job remote-fs-pre.target/stop deleted to break ordering cycle starting with sysinit.target/stop

Then, most of the time the host reboot hangs for ~10 minutes, often
showing scapegoat units like the pve-ha-lrm being the cause of the
hang (even if no HA is configure >.<).

This behavior is fixed with newer systemd versions, e.g., the v244
from buster-backports, but that is not a real option for us for now.

So until 7.0 we generate the unit with the correct dependencies
directly in the ephemeral /run/ tmpfs backed systemd/system path and
start it.

While FUSE gets only the local-fs ordering constraint, it seems to cope
very well regarding such symptoms. But it _is_ racy and probably only
works due to systemd stopping it early as it has not much ordering
constraints at all.. It should be moved in the future nonetheless, as
there's a mount.fuse.ceph helper that should be not an issue.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
4 years agoCephFSPlugin: copy over systemd_escape
Thomas Lamprecht [Wed, 29 Jan 2020 17:00:34 +0000 (18:00 +0100)]
CephFSPlugin: copy over systemd_escape

This is but a hack, but we have no general helper/tools module here
and I do not want to do versioned dependencies for this fast-tracked
bugfix to pve-common, so I'll have to live with the shame for now.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
4 years agodon't (accidentally) modify PVE::Cluster::vmlist
Fabian Grünbichler [Tue, 7 Jan 2020 11:24:13 +0000 (12:24 +0100)]
don't (accidentally) modify PVE::Cluster::vmlist

when listing volumes, otherwise an empty hash can be persisted into the
current worker's $vmlist, which could cause issues at various other API
endpoints.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
4 years agocifs-plugin: Add bwlimit storage option
Dominic Jäger [Fri, 20 Dec 2019 09:12:31 +0000 (10:12 +0100)]
cifs-plugin: Add bwlimit storage option

This is already implemented in all other storage plugins.

Signed-off-by: Dominic Jäger <d.jaeger@proxmox.com>
4 years agobump version to 6.1-3
Thomas Lamprecht [Fri, 13 Dec 2019 11:18:40 +0000 (12:18 +0100)]
bump version to 6.1-3

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
4 years agogrammar fix: s/does not exists/does not exist/g
Thomas Lamprecht [Fri, 13 Dec 2019 11:11:26 +0000 (12:11 +0100)]
grammar fix: s/does not exists/does not exist/g

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
4 years agobase find_free_diskname: refactor mapping from disks to volid array
Thomas Lamprecht [Thu, 12 Dec 2019 11:59:57 +0000 (12:59 +0100)]
base find_free_diskname: refactor mapping from disks to volid array

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
4 years agoUse a common interface for find_free_diskname
Fabian Ebner [Wed, 11 Dec 2019 09:25:49 +0000 (10:25 +0100)]
Use a common interface for find_free_diskname

We can use 'list_images' to get the desired volume IDs in
'find_free_diskname' for most plugins. For the two LVM plugins, 'list_images'
potentially skips untagged volumes, so we keep the custom version. For the
RBD plugin, 'list_images' is much more costly than the custom version, so we
keep the custom version.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
4 years agorbd: unprotect all snapshots on image removal
Fabian Grünbichler [Fri, 29 Nov 2019 11:00:37 +0000 (12:00 +0100)]
rbd: unprotect all snapshots on image removal

we need to unprotect more snapshots than just the base one, since we
allow linked clones of regular VM snapshots. unprotection will only work
if no linked clones exist anymore.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
4 years agoWhen resizing a ZFS volume, align size to 1M
Fabian Ebner [Mon, 9 Dec 2019 07:25:53 +0000 (08:25 +0100)]
When resizing a ZFS volume, align size to 1M

The size is required to be a multiple of volblocksize. Make sure
that the requirement is always met, so ZFS won't complain when we do
things like 'qm resize 102 scsi1 +0.01G'.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
4 years agocalculate reasonable metadatasize for lvm thin pools on creation
Tim Marx [Fri, 6 Dec 2019 13:04:45 +0000 (14:04 +0100)]
calculate reasonable metadatasize for lvm thin pools on creation

Letting LVM set the meta-data size internally was not a good idea, as
it produces really small metadata LVs. Adapts the same logic as the
installer.

Signed-off-by: Tim Marx <t.marx@proxmox.com>
Reviewed-By: Dominik Csapak <d.csapak@proxmox.com>
Tested-By: Dominik Csapak <d.csapak@proxmox.com>
4 years agolvm: vg create: followup for bogus warnings
Thomas Lamprecht [Mon, 2 Dec 2019 14:15:52 +0000 (15:15 +0100)]
lvm: vg create: followup for bogus warnings

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
4 years agobump version to 6.1-2
Thomas Lamprecht [Sat, 30 Nov 2019 18:38:17 +0000 (19:38 +0100)]
bump version to 6.1-2

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
4 years agoLVM commands: ignore "No medium found" bogus warnings
Thomas Lamprecht [Fri, 29 Nov 2019 13:44:13 +0000 (14:44 +0100)]
LVM commands: ignore "No medium found" bogus warnings

Those come normally from virtual devices, like a IPMI disk, if no
media is attached. They spam the log really often on operations like
migrate, and are quite scare-mongering. So filter them out.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
4 years agorbd: update features of image when mapping snapshot
Fabian Grünbichler [Fri, 29 Nov 2019 12:30:30 +0000 (13:30 +0100)]
rbd: update features of image when mapping snapshot

in order to fix the features even if the image itself never gets mapped.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
4 years agorbd: don't attempt to update features of snapshots
Fabian Grünbichler [Fri, 29 Nov 2019 09:57:17 +0000 (10:57 +0100)]
rbd: don't attempt to update features of snapshots

it does not work:

disable RBD image features this kernel RBD drivers is not compatible with: fast-diff,object-map,deep-flatten
clone failed: could not disable krbd-incompatible image features 'fast-diff,object-map,deep-flatten' for rbd image: vm-123123123-disk-0@test: rbd: snapshot name specified for a command that doesn't use it

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
4 years agobump version to 6.1-1
Thomas Lamprecht [Wed, 27 Nov 2019 13:46:53 +0000 (14:46 +0100)]
bump version to 6.1-1

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
4 years agopvesm import: make error messages consistent
Fabian Grünbichler [Wed, 27 Nov 2019 12:48:59 +0000 (13:48 +0100)]
pvesm import: make error messages consistent

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
4 years agopvesm import: improve handling of interrupted export
Fabian Grünbichler [Wed, 27 Nov 2019 12:48:58 +0000 (13:48 +0100)]
pvesm import: improve handling of interrupted export

since 'pvesm export' and 'pvesm import' are connected via a pipe and
SSH, a fatal error in the former can lead to no valid header being
written to the pipe. handle this more gracefully by printing an easier
to understand error message, instead of uninitialized warnings with no
context.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
4 years agofix #2309: allow LVM Volume Group names starting with a digit
Thomas Lamprecht [Tue, 26 Nov 2019 13:17:19 +0000 (14:17 +0100)]
fix #2309: allow LVM Volume Group names starting with a digit

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
4 years agobump version to 6.0-12
Thomas Lamprecht [Mon, 25 Nov 2019 17:28:57 +0000 (18:28 +0100)]
bump version to 6.0-12

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
4 years agod/controll: add zfs-zed to Recommends
Thomas Lamprecht [Mon, 25 Nov 2019 13:57:23 +0000 (14:57 +0100)]
d/controll: add zfs-zed to Recommends

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
4 years agod/control: bump version dependency to pve-common
Thomas Lamprecht [Sat, 23 Nov 2019 15:15:56 +0000 (16:15 +0100)]
d/control: bump version dependency to pve-common

ensures we have the new kernel_version helper available.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
4 years agoRBD: disable and enable features depending on kernel version
Thomas Lamprecht [Sat, 23 Nov 2019 14:37:46 +0000 (15:37 +0100)]
RBD: disable and enable features depending on kernel version

Modern kernel, like 5.3, support all those features ('fast-diff',
'object-map', 'deep-flatten'), so we do not want to disable them
there. 5.0 already supports exclusive-locks, so no need to disable
exclusive locking there.

Further, we also want to profit from new features available, so let's
enable those which can be enabled "live" (i.e., after image creation)
if their available.

While we could also parse the kernel information directly from:
/sys/module/libceph/parameters/supported_features
there's not much advantage to that, features cannot be disabled with
KConfig, their also very dependent of the kernel version booted.
So for us it's enough to check that one.

This only affects container and VMs backed by a storage with KRBD
explicitly enabled. But as the enabling and disabling happens
transparently, it has no effect on the running guest.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
4 years agorbd: group and sort module usage
Thomas Lamprecht [Fri, 22 Nov 2019 17:15:25 +0000 (18:15 +0100)]
rbd: group and sort module usage

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
4 years agobump version to 6.0-11
Thomas Lamprecht [Fri, 22 Nov 2019 15:06:20 +0000 (16:06 +0100)]
bump version to 6.0-11

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
4 years agoRBD: fix ceph version detection
Thomas Lamprecht [Fri, 22 Nov 2019 15:00:59 +0000 (16:00 +0100)]
RBD: fix ceph version detection

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
4 years agocleanuo: no need to negate we have not-equal
Thomas Lamprecht [Thu, 21 Nov 2019 11:28:49 +0000 (12:28 +0100)]
cleanuo: no need to negate we have not-equal

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
4 years agochange var name ct to type to prevent confusion with container
Tim Marx [Thu, 21 Nov 2019 10:43:22 +0000 (11:43 +0100)]
change var name ct to type to prevent confusion with container

Signed-off-by: Tim Marx <t.marx@proxmox.com>
4 years agofix #2467: avoid duplicate volumes & tag with correct content type
Tim Marx [Thu, 21 Nov 2019 10:43:20 +0000 (11:43 +0100)]
fix #2467: avoid duplicate volumes & tag with correct content type

The bugfix for #2317 introduced a kind of odd API behavior, where
each volume was returned twice from our API if a storage has both
'rootdir' & 'images' content types enabled. To give the content type
of the volume an actual meaning, it is now inferred from the
associated guest, if there's no guest or we don't have an owner for
that volume we default to 'images'.

At the volume level, there is no option to list volumes based on
content types, since the volumes do not know what type they are
actually used for.

Signed-off-by: Tim Marx <t.marx@proxmox.com>
4 years agofollowup: remove no-op JSONSchema::check_format on mountpoint
Thomas Lamprecht [Thu, 21 Nov 2019 11:41:56 +0000 (12:41 +0100)]
followup: remove no-op JSONSchema::check_format on mountpoint

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>