pve-storage.git
4 days agobuildsys: no need to include arch detection for arch-independent package master
Thomas Lamprecht [Tue, 21 May 2019 20:24:42 +0000 (22:24 +0200)]
buildsys: no need to include arch detection for arch-independent package

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
4 days agobuildsys: switch upload dist over to buster
Thomas Lamprecht [Tue, 21 May 2019 18:57:05 +0000 (20:57 +0200)]
buildsys: switch upload dist over to buster

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
4 days agobump version to 6.0-0+1
Thomas Lamprecht [Tue, 21 May 2019 18:54:49 +0000 (20:54 +0200)]
bump version to 6.0-0+1

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
4 days agobuildsys: use dpkg-dev makefile helpers for pkg info
Thomas Lamprecht [Tue, 21 May 2019 18:39:36 +0000 (20:39 +0200)]
buildsys: use dpkg-dev makefile helpers for pkg info

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
4 days agod/control: fix priority-extra-is-replaced-by-priority-optional
Thomas Lamprecht [Tue, 21 May 2019 18:31:42 +0000 (20:31 +0200)]
d/control: fix priority-extra-is-replaced-by-priority-optional

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
11 days agobump version to 5.0-43
Thomas Lamprecht [Wed, 15 May 2019 14:16:58 +0000 (16:16 +0200)]
bump version to 5.0-43

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
11 days agoapi/config update: indentation and whitespace fixes
Thomas Lamprecht [Wed, 15 May 2019 08:36:01 +0000 (10:36 +0200)]
api/config update: indentation and whitespace fixes

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
11 days agoapi/config update: only iterate over hash keys, not values
Thomas Lamprecht [Wed, 15 May 2019 08:34:12 +0000 (10:34 +0200)]
api/config update: only iterate over hash keys, not values

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
12 days agopvesm set: handle deletion of properties
Thomas Lamprecht [Tue, 14 May 2019 14:18:50 +0000 (16:18 +0200)]
pvesm set: handle deletion of properties

the delete parameter get's injected by the SectionConfigs
updateSchem, but we need to handle it ourself in the code
This makes the following possible:

pvesm set STORAGEID --delete property

Also the API equivalent is now possible. Adapted from the HA
managers Resource update API call.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
3 weeks agobump version to 5.0-42
Thomas Lamprecht [Tue, 30 Apr 2019 13:56:23 +0000 (13:56 +0000)]
bump version to 5.0-42

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
3 weeks agomap_volume: fall back to 'path'
Mira Limbeck [Mon, 29 Apr 2019 13:00:38 +0000 (15:00 +0200)]
map_volume: fall back to 'path'

Adds a fallback to 'Plugin::path' in the default implementation of
'map_volume' to simplify a common case of calling 'map_volume' followed
by a defined-check and a call to path if it is not. The path is now
always returned if the plugin in question does not override
'map_volume'.

Signed-off-by: Mira Limbeck <m.limbeck@proxmox.com>
5 weeks agozpool: handle race with other zpool imports
Thomas Lamprecht [Wed, 17 Apr 2019 15:00:20 +0000 (15:00 +0000)]
zpool: handle race with other zpool imports

The underlying issue is that a zpool can get imported only once, so
we first check if it's in `zpool list`, and thus imported, and only
if it does not shows up there we try to import it.

But, this can race with either:
* parallel running activate_storage call, through CLI/API/daemon
* a zpool import from an admin (a bit unlikely, but hey that's the
  thing with race conditions ;))

So refactor the "is pool imported" check into a closure, and call it
addditionally if the import failed, and silent the error if the pool
is now listed, and thus imported. This makes it a little bit nicer to
read too, IMO.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
5 weeks agoadd zfsutils-linux to build-dependencies
Fabian Grünbichler [Thu, 18 Apr 2019 08:00:45 +0000 (10:00 +0200)]
add zfsutils-linux to build-dependencies

to enable our ZFS tests and fix broken disk list tests.

5 weeks agozPool: fixup timeout setting for import
Thomas Lamprecht [Wed, 17 Apr 2019 14:39:37 +0000 (14:39 +0000)]
zPool: fixup timeout setting for import

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
5 weeks agozpool: cleanup zfs_request command a bit
Thomas Lamprecht [Wed, 17 Apr 2019 14:31:05 +0000 (14:31 +0000)]
zpool: cleanup zfs_request command a bit

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
5 weeks agoZFS Pool: improve error output from activate_storage
Wolfgang Link [Tue, 16 Apr 2019 13:18:22 +0000 (15:18 +0200)]
ZFS Pool: improve error output from activate_storage

related to #2154 which has some issues on zpool list, but we do not
see the error messages from that step : Buggy "pvesm status" output

6 weeks agoStatus: Include command error in error message when storage activation fails
Christian Ebner [Wed, 10 Apr 2019 15:17:14 +0000 (17:17 +0200)]
Status: Include command error in error message when storage activation fails

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
6 weeks agofollowup: code cleanup
Thomas Lamprecht [Wed, 10 Apr 2019 08:14:46 +0000 (10:14 +0200)]
followup: code cleanup

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
6 weeks agoFix #318: Delete vzdump log when deleting a backup
Dominic Jäger [Tue, 9 Apr 2019 07:41:25 +0000 (09:41 +0200)]
Fix #318: Delete vzdump log when deleting a backup

Vzdump log files were not deleted when a backup was deleted.
Consequently, the folder continuously filled with .log files.
Now they get deleted after the backup is removed.

Signed-off-by: Dominic Jäger <d.jaeger@proxmox.com>
6 weeks agobump version to 5.0-41
Thomas Lamprecht [Mon, 8 Apr 2019 15:54:08 +0000 (17:54 +0200)]
bump version to 5.0-41

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
6 weeks agocode cleanup
Thomas Lamprecht [Mon, 8 Apr 2019 15:48:39 +0000 (17:48 +0200)]
code cleanup

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
6 weeks agoDiskmanage: don't run zpool if not present
Stoiko Ivanov [Mon, 8 Apr 2019 12:38:36 +0000 (14:38 +0200)]
Diskmanage: don't run zpool if not present

Since zfsutils are not a hard dependency of our stack it is possible to not have
`zpool` available.

Checking for existance of `zpool` before calling it suppresses spurious warnings
in the logs (e.g. when creating Ceph OSDs or accessing the 'Disk' Tab in the
GUI).

Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
7 weeks agoadd 3 bwlimit tests
Stoiko Ivanov [Fri, 5 Apr 2019 14:33:35 +0000 (16:33 +0200)]
add 3 bwlimit tests

1) checking that the empty storage list is treated correctly (only override
and datacenter config limit considered)
2) checking that the empty storage list is treated correctly (as with 1).
3) checking that undef can be passed as one element of the storage list (it is
ignored)

Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
7 weeks agoget_bandwidt_limits: ignore 'undef' as storage
Stoiko Ivanov [Fri, 5 Apr 2019 14:33:34 +0000 (16:33 +0200)]
get_bandwidt_limits: ignore 'undef' as storage

If one of the storages passed in $storage_list was not defined
get_bandwidth_limit died (see [0], of an occurence of this).
This patch changes the behavior to ignore undef as storage instead.

[0] https://pve.proxmox.com/pipermail/pve-devel/2019-April/036515.html

Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
7 weeks agobump version to 5.0-40
Thomas Lamprecht [Thu, 4 Apr 2019 14:19:51 +0000 (16:19 +0200)]
bump version to 5.0-40

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
7 weeks agozfs: don't generate/update cachefile on pool import
Fabian Grünbichler [Wed, 3 Apr 2019 09:06:07 +0000 (11:06 +0200)]
zfs: don't generate/update cachefile on pool import

during storage activation.

for pools that don't get imported at boot (e.g. because their vdevs are
not available when zfs-import-*.service runs) it is fatal to include
them in the cachefile, for those that do get imported at boot this code
should never run anyway as they are already imported.

in any case, a fallback to import without cachefile is the safe variant.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
8 weeks agoworkaround zfs create -V error for unaligned sizes
Mira Limbeck [Fri, 29 Mar 2019 09:13:10 +0000 (10:13 +0100)]
workaround zfs create -V error for unaligned sizes

fixes the 'cannot create 'nvme/foo': volume size must be a multiple of
volume block size' error by always rounding the size up to the next 1M
boundary. this is a workaround until
https://github.com/zfsonlinux/zfs/issues/8541 is solved.
the current manpage says 128k is the maximum blocksize, but a local test
showed that values up to 1M are allowed. it might be possible to
increase it even further (see f1512ee61).

Signed-off-by: Mira Limbeck <m.limbeck@proxmox.com>
8 weeks agoStorage::get_bandwidth_limit: fix if condition
Stoiko Ivanov [Fri, 29 Mar 2019 07:27:59 +0000 (08:27 +0100)]
Storage::get_bandwidth_limit: fix if condition

Passing 'undef' as '$storage_list' led to a warning about using an
uninitialized value as array_ref.

Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
2 months agoadd libfile-chdir-perl to build dependencies
Thomas Lamprecht [Thu, 21 Mar 2019 09:51:19 +0000 (10:51 +0100)]
add libfile-chdir-perl to build dependencies

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2 months agore-bump version to 5.0-39
Thomas Lamprecht [Thu, 7 Mar 2019 11:22:57 +0000 (12:22 +0100)]
re-bump version to 5.0-39

this wasn't released yet to any repo and I'd like to have the iSCSI
fix included, so just re-run 'dch -r'

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2 months agofix tests when one has iscsi devices
Dominik Csapak [Thu, 7 Mar 2019 09:49:04 +0000 (10:49 +0100)]
fix tests when one has iscsi devices

the test would read the real device and if one is an iscsi device
it would fail, move the test code to a sub and mock it in the tests

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2 months agofix content listing for user mode iscsi plugin
Dominik Csapak [Thu, 7 Mar 2019 09:49:03 +0000 (10:49 +0100)]
fix content listing for user mode iscsi plugin

the format is a required in the result schema

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2 months agobump version to 5.0-39
Thomas Lamprecht [Wed, 6 Mar 2019 09:39:18 +0000 (10:39 +0100)]
bump version to 5.0-39

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2 months agofix #585: remove leftover disks/directory after VM creation failed
Christian Ebner [Mon, 4 Mar 2019 15:47:06 +0000 (16:47 +0100)]
fix #585: remove leftover disks/directory after VM creation failed

When trying to create a qcow2 disk image with a size larger than available on the
storage, this will fail.
As qemu-img does not clean up the disk afterwards, it needs to be deleted
explicitly. Further, the vmid folder is cleaned up once it is empty.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
3 months agod/control: bump version dependency to pve-doc-generator
Thomas Lamprecht [Fri, 22 Feb 2019 12:31:32 +0000 (13:31 +0100)]
d/control: bump version dependency to pve-doc-generator

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
3 months ago1891 Add zsh command completion for pvesm
Christian Ebner [Thu, 21 Feb 2019 13:25:02 +0000 (14:25 +0100)]
1891 Add zsh command completion for pvesm

This adds the zsh command completion generation for pvesm.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
3 months agofollowup: try to be a bit more like systemd-escape
Thomas Lamprecht [Wed, 20 Feb 2019 15:31:25 +0000 (16:31 +0100)]
followup: try to be a bit more like systemd-escape

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
3 months agofollowup comment that we do not escape completely like systemd
Thomas Lamprecht [Wed, 20 Feb 2019 15:30:46 +0000 (16:30 +0100)]
followup comment that we do not escape completely like systemd

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
3 months agofix #2099: escape systemd path names in mount unit
Dominik Csapak [Wed, 20 Feb 2019 15:04:34 +0000 (16:04 +0100)]
fix #2099: escape systemd path names in mount unit

we only allow '-' '_' and '.' int storage-ids and names,
and we do not need to escape '_' and '.' (see man 5 systemd.unit)

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
3 months agobump version to 5.0-38
Thomas Lamprecht [Thu, 7 Feb 2019 14:15:39 +0000 (15:15 +0100)]
bump version to 5.0-38

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
3 months agofollowup: improve comment
Thomas Lamprecht [Tue, 5 Feb 2019 16:19:35 +0000 (17:19 +0100)]
followup: improve comment

while the commit message tells it nicely a comment should add
additional info for people just giving this a quick look

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
3 months agocheck_volume_access: tighten checks for iso/tmpl
Fabian Grünbichler [Tue, 5 Feb 2019 15:01:40 +0000 (16:01 +0100)]
check_volume_access: tighten checks for iso/tmpl

(custom) templates might contain sensitive data, so require at least
read access on the underlying storage to access ISO and template files.

the same permissions are already needed for listing them, so this is
unlikely to cause fallout.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
3 months agod/control: bump version dependency of libpve-common-perl
Thomas Lamprecht [Mon, 4 Feb 2019 11:02:21 +0000 (12:02 +0100)]
d/control: bump version dependency of libpve-common-perl

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
3 months agodiskmanage: fix device encoding handling
Wolfgang Bumiller [Mon, 4 Feb 2019 09:42:01 +0000 (10:42 +0100)]
diskmanage: fix device encoding handling

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
3 months agobump version 5.0-37
Thomas Lamprecht [Fri, 1 Feb 2019 11:39:32 +0000 (12:39 +0100)]
bump version 5.0-37

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
3 months agoallow snippets by default for new dir storages
Dominik Csapak [Tue, 29 Jan 2019 15:28:40 +0000 (16:28 +0100)]
allow snippets by default for new dir storages

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
3 months agoenable snippets content type for all directory based storages
Dominik Csapak [Tue, 29 Jan 2019 15:28:39 +0000 (16:28 +0100)]
enable snippets content type for all directory based storages

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
3 months agoadd new content type 'snippets'
Dominik Csapak [Tue, 29 Jan 2019 15:28:38 +0000 (16:28 +0100)]
add new content type 'snippets'

will be used to contain files which can be executed as hookscripts or
contain custom cloud-init configs

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
3 months agofix #1598: use glusterfs daemon default port for online check
Thomas Lamprecht [Mon, 28 Jan 2019 13:37:05 +0000 (14:37 +0100)]
fix #1598: use glusterfs daemon default port for online check

use the port where the main glusterfs daemon listens on as ping port,
this one is also used by QEMU as default.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
4 months agoFix #1941: remove empty directories when freeing image on FS based storages
Christian Ebner [Thu, 24 Jan 2019 13:58:22 +0000 (14:58 +0100)]
Fix #1941: remove empty directories when freeing image on FS based storages

Remove directories if they are empty, which can happen if all images
from a VM got deleted, e.g., after destroying said VM.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
4 months agoFix #2050: only provide 'conv=sparse' for LvmThin
Stoiko Ivanov [Thu, 17 Jan 2019 15:31:51 +0000 (16:31 +0100)]
Fix #2050: only provide 'conv=sparse' for LvmThin

LVMPlugin->volume_import (used by storage_migrate on either offline
migration with local disks, or online migration with storage-only
referenced disks) passed 'conv=sparse' to `dd`. This can lead to
data-corruption, if the target volume is not zero-initialized.

dropping the sparse argument completely would fix the problem, but
breaks keeping data sparse for LvmThinPlugin.

This patch moves the dd out into (LVM*) plugin specific sub so that
each can control the parameters.

Steps for reproducing the issue:
* create a cluster with (at least) 2 nodes A and B, with a free
  disk-device (/dev/sdx)
* write a recognizable pattern to /dev/sdx on B:
  `dd if=/dev/zero bs=10M | tr '\000' '\255' | dd of=/dev/sdb bs=10M`
  (would be grateful for alternatives to the dd| tr| dd)
* on both A and B create a lvm-vg (pvcreate, vgcreate)
* add it as _not_ shared storage, which is available on nodes A and B
* create a small guest on A
* fill a file in the guest with zeros
  `dd if=/dev/zero of=/zerofil bs=10M`
* stop the guest, migrate it to B
* start the guest - check that the file `/zerofil` contains `ad`
  instead of `00`

Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
4 months agoupdate version to 5.0-36
Thomas Lamprecht [Tue, 15 Jan 2019 12:18:28 +0000 (13:18 +0100)]
update version to 5.0-36

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
4 months agoLVM: Add '--refresh' when activating volumes
Stoiko Ivanov [Fri, 4 Jan 2019 13:06:25 +0000 (14:06 +0100)]
LVM: Add '--refresh' when activating volumes

From `man 8 lvchange`:
  --refresh
      If the logical volume is active, reload its metadata. This is not
      necessary in normal operation, but may be useful ... if you're doing
      clustering manually without a clustered lock manager.

Fixes migration in a shared LVM (iscsi) setup, where a disk gets resized on one
node A and the guest is afterwards migrated to another node B: B still presents
the old size to the guest, leading to data corruption.

It is necessary to run `lvchange` twice because the options `-ay` and
`--refresh` are mutually exclusive.

Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
4 months agofix #2046 add volume_size_info to LVMPlugin
Stoiko Ivanov [Fri, 4 Jan 2019 13:06:24 +0000 (14:06 +0100)]
fix #2046 add volume_size_info to LVMPlugin

Without volume_size_info a Storage plugin falls back to the Implementation
in PVE/Storage/Plugin.pm, which relies on `qemu-img info`.

`qemu-img info` returns wrong results on a node in the case of shared volume
groups (e.g. when sharing disks via iSCSI), if a disk was resized on another
node (it lseeks to the end of the block-device, and this yields the old size).

Using lvs directly fixes the issue, since the LVM metadata gets updated when
invoking lvs.

Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
4 months agoget_monaddr_list: also ensure that returned 'mon addr' are defined
Thomas Lamprecht [Wed, 9 Jan 2019 14:23:39 +0000 (15:23 +0100)]
get_monaddr_list: also ensure that returned 'mon addr' are defined

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
4 months agoceph: get_monaddr_list: exclude general monitor section
Alwin Antreich [Wed, 9 Jan 2019 10:59:12 +0000 (11:59 +0100)]
ceph: get_monaddr_list: exclude general monitor section

Else, if a general MON section existed in the ceph.conf, we added a
undefined entry and a cephfs storage can't be mounted anymore.

Signed-off-by: Alwin Antreich <a.antreich@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
4 months agofix use of uninitialized value in parse_ceph_config
David Limbeck [Thu, 3 Jan 2019 08:44:59 +0000 (09:44 +0100)]
fix use of uninitialized value in parse_ceph_config

Signed-off-by: David Limbeck <d.limbeck@proxmox.com>
5 months agobump version to 5.0-35
Thomas Lamprecht [Thu, 20 Dec 2018 09:50:45 +0000 (10:50 +0100)]
bump version to 5.0-35

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
5 months agod/control: bump pve-cluster version dependency to 5.0-32
Thomas Lamprecht [Thu, 20 Dec 2018 09:49:06 +0000 (10:49 +0100)]
d/control: bump pve-cluster version dependency to 5.0-32

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
5 months agoregister ceph.conf parser/writer
Dominik Csapak [Wed, 19 Dec 2018 10:24:40 +0000 (11:24 +0100)]
register ceph.conf parser/writer

With this we can use cfs_read_file/cfs_write_file and cfs_lock_file.

Code for writing is mostly copied from pve-managers CephTools.pm,
with the addition of mgr sections.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
5 months agomove Storage/CephTools to CephConfig
Dominik Csapak [Wed, 19 Dec 2018 10:24:39 +0000 (11:24 +0100)]
move Storage/CephTools to CephConfig

it is not a storage plugin, and it makes more sense to have it
top-level, but there we cannot name it CephTools because of the
existing ones in pve-manager

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
5 months agobump version to 5.0-34
Thomas Lamprecht [Fri, 14 Dec 2018 11:22:22 +0000 (12:22 +0100)]
bump version to 5.0-34

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
5 months agoFix #2020: use /sys to map nvmeXnY to nvmeX
Wolfgang Bumiller [Mon, 10 Dec 2018 13:17:06 +0000 (14:17 +0100)]
Fix #2020: use /sys to map nvmeXnY to nvmeX

`nvmeX` devices nodes are apparently allocated independently
from their namespace block devices `nvmeXnY` and therefore
they are not strictly related by name. For instance:
  $ readlink /sys/block/nvme0n1/device
  ../../nvme1
  $ readlink /sys/block/nvme1n1/device
  ../../nvme0

Here /dev/nvme0n1 is the first namespace of /dev/nvme1 while
/dev/nvme1n1 is the first namespace of /dev/nvme0.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
5 months agoFix #2019: CephFS storage misses maxfiles
Alwin Antreich [Fri, 7 Dec 2018 08:02:58 +0000 (09:02 +0100)]
Fix #2019: CephFS storage misses maxfiles

Signed-off-by: Alwin Antreich <a.antreich@proxmox.com>
5 months agobump version to 5.0-33
Thomas Lamprecht [Tue, 27 Nov 2018 12:23:48 +0000 (13:23 +0100)]
bump version to 5.0-33

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
5 months agocephfs: tell systemd that mount mount requires network
Thomas Lamprecht [Tue, 27 Nov 2018 11:02:01 +0000 (12:02 +0100)]
cephfs: tell systemd that mount mount requires network

As we mount this manually and thus systemd doesn't know about any
dependency for cephFS mounts, this got umounted only at the last
stage of shutdown, where network wasn't active anymore.

But, CephFS needs to be connected to an active MDS for a clean
unmount so without network this mount would delay shutdown for quite
a bit, until after some minutes systemd gave up and forced unmount.

So tell systemd that this mount requires network, which can be done
with the '_netdev'[0] mount option, that lucky for us can be also
passed to a mount call and isn't only available for fstab.

with this a mount gets, among others:
> Wants=network-online.target
> Before=umount.target remote-fs.target
> After=remote-fs-pre.target system.slice network.target network-online.target -.mount

Which does the trick for us.

[0]: https://www.freedesktop.org/software/systemd/man/systemd.mount.html#_netdev

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
6 months agopvesm: add scan subcommands
Thomas Lamprecht [Mon, 19 Nov 2018 15:23:25 +0000 (16:23 +0100)]
pvesm: add scan subcommands

Change to a cleaner sub command interface grouping all scan commands.

Alias to old command names for backward compatibility
Best viewed with the ignore whitespace/indent change '-w' flag from
git

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
6 months agobump version to 5.0-32
Thomas Lamprecht [Mon, 19 Nov 2018 14:19:56 +0000 (15:19 +0100)]
bump version to 5.0-32

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
6 months agod/control: bump pve-common dependency and break older pve-manager
Thomas Lamprecht [Mon, 19 Nov 2018 13:44:46 +0000 (14:44 +0100)]
d/control: bump pve-common dependency and break older pve-manager

To ensure that we have the new SysFSTools available and as previously
we hosted the API scan endoints which pve-manager registered so
record the break of older versions.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
6 months agoremove usb scan code
Dominik Csapak [Fri, 16 Nov 2018 15:17:59 +0000 (16:17 +0100)]
remove usb scan code

this is now in PVE::SysFSTools

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
6 months agoremove PVE/API2/Storage/Scan.pm
Dominik Csapak [Fri, 16 Nov 2018 15:17:58 +0000 (16:17 +0100)]
remove PVE/API2/Storage/Scan.pm

since those are now defined in pve-manager

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
6 months agomove Scan API calls from PVE/API2/Storage/Scan.pm to pvesm.pm
Dominik Csapak [Fri, 16 Nov 2018 15:17:57 +0000 (16:17 +0100)]
move Scan API calls from PVE/API2/Storage/Scan.pm to pvesm.pm

since the calls for the real API are defined now in pve-managers Scan.pm

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
6 months agoAPIAGE followup: fix typo and print versions in error message
Thomas Lamprecht [Mon, 19 Nov 2018 11:18:03 +0000 (12:18 +0100)]
APIAGE followup: fix typo and print versions in error message

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
6 months agoplugin loader: add an APIAGE
Wolfgang Bumiller [Fri, 16 Nov 2018 12:54:44 +0000 (13:54 +0100)]
plugin loader: add an APIAGE

With the addition of the map/unmap_volume() methods we made
an (actually unnecessary) API version bump.
All current users of these methods fall back to path() when
they return undef, so plugins implementing version 1 are
in fact compatible currently. (In fact, the default
Plugin::map_volume() could fall back to it on its own, but
doesn't currently).

For now let's just allow plugins older plugins to also be
loaded by introducing an API age variable. With it, if we
have a reason to break older plugins, we can have a
deprecation period during which older plugins cause a
warning instead of refusing to load altogether.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
6 months agoLVM: lock on volume_resize
Thomas Lamprecht [Wed, 14 Nov 2018 13:43:29 +0000 (14:43 +0100)]
LVM: lock on volume_resize

This is important for shared LVM storages. As with deletes and
creates of images, as else we may have not the up-to-date metadata
and extents may get reused if another node created an image during
the same time, for example.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
6 months agofix #862: do not resolve portal adress on storage add
Thomas Lamprecht [Wed, 14 Nov 2018 17:11:02 +0000 (18:11 +0100)]
fix #862: do not resolve portal adress on storage add

as described in #862:

> I experienced a problem with ISCSI portal when using a hostname and
> not IP.
> The GUI resolves the hostname to an IP and writes it to storage.cfg.
> As my setup requires hostnames, i needed to change the config
> manually back to the hostname which is working fine.
>
> Why is this conversion done? If I enter a hostname, i want to have a
> hostname. If i enter an IP address i want to have an IP address.

This makes sense to me, a feature of using domains is that they
are/should be resolved when actually using (i.e., connecting to them)
so resolving it once on add does not seems like a good idea (if I do
not miss something - as this is a classic "imported from SVN" I do
not have any rationale to look at).

So save the work and pass it as is.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
6 months agobump version to 5.0-31
Thomas Lamprecht [Fri, 9 Nov 2018 14:30:24 +0000 (15:30 +0100)]
bump version to 5.0-31

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
6 months agoimplement map_volume and unmap_volume
Dietmar Maurer [Thu, 8 Nov 2018 13:05:17 +0000 (14:05 +0100)]
implement map_volume and unmap_volume

This allows to request a mapped device/path explicitly, regardles of
the storage option, eg. krbd option in the RBDplugin.

Bump of the storage ABI => 2

Co-authored-by: Alwin Antreich <a.antreich@proxmox.com>
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
6 months agofolowup: fix whitespace errors and s/and/&&/ for consistency
Thomas Lamprecht [Fri, 9 Nov 2018 14:27:06 +0000 (15:27 +0100)]
folowup: fix whitespace errors and s/and/&&/ for consistency

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
6 months agostorage zfs: removed unused variable
Tim Marx [Tue, 6 Nov 2018 13:04:34 +0000 (14:04 +0100)]
storage zfs: removed unused variable

Signed-off-by: Tim Marx <t.marx@proxmox.com>
6 months agostorage zfs: changed return value description & optionals
Tim Marx [Tue, 6 Nov 2018 13:04:33 +0000 (14:04 +0100)]
storage zfs: changed return value description & optionals

Signed-off-by: Tim Marx <t.marx@proxmox.com>
6 months agoclose #1949: storage zfs: changed zpool command parser
Tim Marx [Tue, 6 Nov 2018 13:04:32 +0000 (14:04 +0100)]
close #1949: storage zfs: changed zpool command parser

Signed-off-by: Tim Marx <t.marx@proxmox.com>
6 months agoFix: api zfs: changed return value name to errors
Tim Marx [Thu, 25 Oct 2018 10:51:08 +0000 (12:51 +0200)]
Fix: api zfs: changed return value name to errors

Signed-off-by: Tim Marx <t.marx@proxmox.com>
7 months agod/control: record pve-cluster build dependency
Thomas Lamprecht [Thu, 18 Oct 2018 11:25:34 +0000 (13:25 +0200)]
d/control: record pve-cluster build dependency

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
7 months agobump version to 5.0-30
Thomas Lamprecht [Wed, 3 Oct 2018 15:02:43 +0000 (17:02 +0200)]
bump version to 5.0-30

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
7 months agoremove outdated unused API2::Storage::Replication module
Thomas Lamprecht [Wed, 3 Oct 2018 14:58:55 +0000 (16:58 +0200)]
remove outdated unused API2::Storage::Replication module

this was not installed, had references to non-existing modules
(e.g., PVE::ReplicationTools) and the things it probably was intended
for are done in pve-manager, which has full access to all pve perl
libs and API methods (from a dependency POV)

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
7 months agodiskmanage: rename check_unused to assert_disk_unused
Thomas Lamprecht [Wed, 3 Oct 2018 12:51:35 +0000 (14:51 +0200)]
diskmanage: rename check_unused to assert_disk_unused

to have a clear method name for this. check_XYZ suggests also that we
return true if the check was OK, but we don't.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
7 months agorename check_available to assert_sid_unused
Thomas Lamprecht [Wed, 3 Oct 2018 12:49:14 +0000 (14:49 +0200)]
rename check_available to assert_sid_unused

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
7 months agofix #1929: only check storage if user want to create one
Dominik Csapak [Tue, 25 Sep 2018 08:38:02 +0000 (10:38 +0200)]
fix #1929: only check storage if user want to create one

this is useful if a user wants to create similar storage on each host

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
7 months agoadd missing storage check in LVM Disk API
Dominik Csapak [Tue, 25 Sep 2018 08:38:01 +0000 (10:38 +0200)]
add missing storage check in LVM Disk API

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
7 months agorefactor disk/storage checks for Disk API
Dominik Csapak [Tue, 25 Sep 2018 08:38:00 +0000 (10:38 +0200)]
refactor disk/storage checks for Disk API

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
7 months agopvesm iscsiscan: fix cli parameters
Dietmar Maurer [Mon, 1 Oct 2018 08:53:31 +0000 (10:53 +0200)]
pvesm iscsiscan: fix cli parameters

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
8 months agobump version to 5.0-29
Thomas Lamprecht [Wed, 19 Sep 2018 09:20:52 +0000 (11:20 +0200)]
bump version to 5.0-29

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
8 months agofind_free_diskname: fixup regex match operator
Thomas Lamprecht [Wed, 19 Sep 2018 09:04:33 +0000 (11:04 +0200)]
find_free_diskname: fixup regex match operator

Co-developed-by: Stoiko Ivanov <s.ivanov@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
8 months agoFix #1925: untaint rbd JSON output
Dietmar Maurer [Wed, 19 Sep 2018 04:43:12 +0000 (06:43 +0200)]
Fix #1925: untaint rbd JSON output

Reviewed-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Tested-by: Stoiko Ivanov <s.ivanov@proxmox.com>
Reviewed-by: Stoiko Ivanov <s.ivanov@proxmox.com>
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
8 months agobump version to 5.0-28
Thomas Lamprecht [Fri, 14 Sep 2018 09:42:23 +0000 (11:42 +0200)]
bump version to 5.0-28

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
8 months agofix #1912: find the partition instead of guessing it
Dominik Csapak [Fri, 14 Sep 2018 09:13:35 +0000 (11:13 +0200)]
fix #1912: find the partition instead of guessing it

instead of using '$dev1' as partition,
get the information from /sys/block/$dev

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
8 months agoget_vm_disk_number: follow up cleanup
Thomas Lamprecht [Fri, 14 Sep 2018 07:17:06 +0000 (09:17 +0200)]
get_vm_disk_number: follow up cleanup

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
8 months agoFix #1913: get_vm_disk_number: clone plugindata to avoid side effects
Stoiko Ivanov [Thu, 13 Sep 2018 18:36:40 +0000 (20:36 +0200)]
Fix #1913: get_vm_disk_number: clone plugindata to avoid side effects

Accessing a non-existing 'format' key in plugindata (e.g., in LvmThinPlugin),
created it by autovivication, thus breaking the fallback to the default value
'raw' upon any following access.

Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
8 months agobump version to 5.0-27
Thomas Lamprecht [Wed, 12 Sep 2018 13:00:31 +0000 (15:00 +0200)]
bump version to 5.0-27

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>