pve-storage.git
13 days agobump version to 6.0-7 master
Fabian Grünbichler [Tue, 6 Aug 2019 12:08:02 +0000 (14:08 +0200)]
bump version to 6.0-7

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
13 days agoZFS: refactor waiting for zvol symlinks
Fabian Grünbichler [Wed, 31 Jul 2019 08:58:04 +0000 (10:58 +0200)]
ZFS: refactor waiting for zvol symlinks

and actually do that not just for creating zvols, but also when
activating them. this should fix a range of issues/races that sometimes
occured on bootup, snapshot rollback or similar operations.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
13 days agofix 2317: list images on 'rootdir' only storages
Fabian Grünbichler [Tue, 6 Aug 2019 07:23:19 +0000 (09:23 +0200)]
fix 2317: list images on 'rootdir' only storages

plugins can still override list_volumes if they want separate methods to
list rootdir and images content.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
3 weeks agobump version to 6.0-6
Thomas Lamprecht [Thu, 25 Jul 2019 11:35:07 +0000 (13:35 +0200)]
bump version to 6.0-6

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
3 weeks agoFix #582: Add DELETE delay logic
Dominic Jäger [Mon, 15 Jul 2019 10:28:20 +0000 (12:28 +0200)]
Fix #582: Add DELETE delay logic

Previously, the web GUI timed out when removing content (e.g. backup) took
too long. Doing the main part of the API DELETE call in a fork_worker solves
this.

Signed-off-by: Dominic Jäger <d.jaeger@proxmox.com>
4 weeks agostorage plugin: move get_subdir call
Fabian Grünbichler [Fri, 19 Jul 2019 11:53:45 +0000 (13:53 +0200)]
storage plugin: move get_subdir call

since list_volumes is only supposed to be called with filtered content
types, this should ensure that get_subdir is only called for plugins
that have a defined 'path' property, like the old code in
PVE::Storage::template_list did.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
4 weeks agostorage plugin: new list_volumes plugin method
Dietmar Maurer [Fri, 19 Jul 2019 09:56:16 +0000 (11:56 +0200)]
storage plugin: new list_volumes plugin method

This cleanup improve code reuse, and allows plugins to override list_volumes.

5 weeks agoDiskmanage: add flag for encrypted osds
Dominik Csapak [Fri, 12 Jul 2019 10:15:15 +0000 (12:15 +0200)]
Diskmanage: add flag for encrypted osds

we can only do this here, since the ceph cluster is not aware of
osd encryption, only the local node is (via ceph-volume and lv tags)

this way, we are able to show an 'encrypted' flag in the disk gui at least

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
5 weeks agoDiskmanage: fix usage test for sdd
Dominik Csapak [Fri, 12 Jul 2019 10:13:51 +0000 (12:13 +0200)]
Diskmanage: fix usage test for sdd

sdd1 and sdd2 were listed in the zfs output but were not included
in the partlist, so the zfs detection test was wrong

now it is correct and more in line with the real world

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
5 weeks agoDiskmanage: add test for lsblk by simulating zfs disk
Dominik Csapak [Fri, 12 Jul 2019 10:13:50 +0000 (12:13 +0200)]
Diskmanage: add test for lsblk by simulating zfs disk

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
5 weeks agoDiskmanage: change parttype uuid detection
Dominik Csapak [Fri, 12 Jul 2019 10:13:49 +0000 (12:13 +0200)]
Diskmanage: change parttype uuid detection

previously ceph included a udev rule to populate
/dev/disk/by-parttypeuuid/

but not anymore, so we now use 'lsblk --json -o path,parttype' to
get a mapping between parttype uuid and partition

fix the test by simulating empty lsblk output

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
5 weeks agobump version to 6.0-5
Thomas Lamprecht [Thu, 11 Jul 2019 17:37:31 +0000 (19:37 +0200)]
bump version to 6.0-5

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
5 weeks agopvesm import: allow to pass a tcp://<IP> as file
Thomas Lamprecht [Thu, 27 Jun 2019 14:16:09 +0000 (16:16 +0200)]
pvesm import: allow to pass a tcp://<IP> as file

To allow getting closer to finally drop "pvecm mtunnel".
Code parts taken from pipe_socket_to_command

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
[regex fixup]
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
5 weeks agofix #2266: Diskmanage: get correct osd id
Dominik Csapak [Mon, 8 Jul 2019 13:48:14 +0000 (15:48 +0200)]
fix #2266: Diskmanage: get correct osd id

The osdid can consist of multiple digits, cope with that and add more
regression tests for this

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
6 weeks agobump version to 6.0-4
Thomas Lamprecht [Thu, 4 Jul 2019 14:00:48 +0000 (16:00 +0200)]
bump version to 6.0-4

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
6 weeks agocephfs: make is_mounted check less strict
Dominik Csapak [Wed, 3 Jul 2019 07:42:13 +0000 (09:42 +0200)]
cephfs: make is_mounted check less strict

checking '$server:$subdir' is too strict to work in all cirumcstances,
e.g. adding/removing a monitor would mean that it is not the same
anymore, same if one is adding/removing the ports from the config

check only if the subdir is the same and if it is a cephfs
this way, it still returns true if someone changes the config

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
6 weeks agofix missing osd info for osd 0
Dominik Csapak [Wed, 3 Jul 2019 06:43:12 +0000 (08:43 +0200)]
fix missing osd info for osd 0

0 is falsy, we have to check for definedness
also adapt the tests so we test for this

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
7 weeks agobump version to 6.0-3
Thomas Lamprecht [Thu, 27 Jun 2019 14:18:03 +0000 (16:18 +0200)]
bump version to 6.0-3

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
7 weeks agoCephConfig: read monitor addresses also from mon_host for cephfs
Dominik Csapak [Thu, 27 Jun 2019 08:43:12 +0000 (10:43 +0200)]
CephConfig: read monitor addresses also from mon_host for cephfs

since we write only the mon_host config beginning with nautilus,
we have to get the monitor ips from there as well

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
7 weeks agoCephConfig: refactor host port parsing
Dominik Csapak [Thu, 27 Jun 2019 08:43:11 +0000 (10:43 +0200)]
CephConfig: refactor host port parsing

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
7 weeks agoCephConfig: do not always interpret '; ' as a comment
Dominik Csapak [Thu, 27 Jun 2019 08:43:10 +0000 (10:43 +0200)]
CephConfig: do not always interpret '; ' as a comment

; is the beginning of a comment, but in some configuration settings
it is also valid syntax, e.g. for mon_host it is a valid
seperator for hosts (sigh ...)

only remove lines when it starts with a ';'

since we remove all comments anyway any time we write the ceph conf
it should not really matter for our users

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
7 weeks agobump version to 6.0-2
Thomas Lamprecht [Mon, 24 Jun 2019 16:48:27 +0000 (18:48 +0200)]
bump version to 6.0-2

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2 months agofollowup: comment reword
Thomas Lamprecht [Wed, 19 Jun 2019 08:23:01 +0000 (10:23 +0200)]
followup: comment reword

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2 months agoCephConfig: map special config key characters to _
Dominik Csapak [Wed, 19 Jun 2019 07:17:52 +0000 (09:17 +0200)]
CephConfig: map special config key characters to _

we want a consistent config has, regardless of how the user or a tool
adds it to the config, so we map ' ' and '-' to '_' in the keys

this way we can always access the correct key without trying multiple
times

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2 months agopartially revert follow up
Thomas Lamprecht [Mon, 17 Jun 2019 10:35:51 +0000 (12:35 +0200)]
partially revert follow up

see
https://pve.proxmox.com/pipermail/pve-devel/2019-June/037575.html

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2 months agofollowup: free_image: just check if a file in general exist
Thomas Lamprecht [Mon, 17 Jun 2019 09:51:27 +0000 (11:51 +0200)]
followup: free_image: just check if a file in general exist

don't require any specific file types, if something is here which can
be requested to delete over API/CLI it either comes from an API/CLI
operation and should be thus OK to delete, else a user caused the
creation of the special file and it either works and all is good or
the user gets notified as we check if unlink succeeded anyway

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2 months agofix #2226: allow broken symlinks to be deleted via API
Stefan Reiter [Mon, 17 Jun 2019 09:05:32 +0000 (11:05 +0200)]
fix #2226: allow broken symlinks to be deleted via API

Symlinks with a non-existing target fail Perls '-f' test and were thus
not deleteable via the API (failing with '$path does not exist').

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2 months agofix #2226: ignore broken symlinks in *_list
Stefan Reiter [Mon, 17 Jun 2019 09:05:31 +0000 (11:05 +0200)]
fix #2226: ignore broken symlinks in *_list

Broken symlinks (and other files without a size) will now show up as 0
byte instead of causing a format validation error in the API.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2 months agobump version to 6.0-1
Thomas Lamprecht [Fri, 14 Jun 2019 18:44:46 +0000 (20:44 +0200)]
bump version to 6.0-1

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2 months agocephfs: Exclude _netdev when mounting with fuse
Alwin Antreich [Wed, 12 Jun 2019 10:32:05 +0000 (12:32 +0200)]
cephfs: Exclude _netdev when mounting with fuse

Since ceph-fuse is called directly in the CephFS storage plugin, which
can not process the _netdev option, mounting the CephFS storage fails
when fuse is set in the storage.cfg.

This patch moves the _netdev option into the else part of the if fuse is
set statement. _netdev is only added if the CephFS kernel client mounts
the storage.

It seems _netdev is not needed anyway for the fuse mount, as the
connection is closed, once the fuse process gets killed on shutdown.

Signed-off-by: Alwin Antreich <a.antreich@proxmox.com>
2 months agofollowup: reword comment a bit
Thomas Lamprecht [Thu, 13 Jun 2019 09:59:58 +0000 (11:59 +0200)]
followup: reword comment a bit

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2 months agofix #1427: Set file mode on uploaded templates/ISOs
Stefan Reiter [Thu, 13 Jun 2019 09:33:28 +0000 (11:33 +0200)]
fix #1427: Set file mode on uploaded templates/ISOs

simply chmod the temp file before copying to the "correct" permission
mode, where all users with access to the directory can read the file,
to mirror the behavior one gets for a apl_download call.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2 months agodrop un-maintained sheepdog plugin
Thomas Lamprecht [Tue, 4 Jun 2019 14:57:40 +0000 (16:57 +0200)]
drop un-maintained sheepdog plugin

as already announced over two months ago[0], remove the unofficial
SheepDog plugin now completely. Besides that it was never fully
supported in Proxmox VE one of its main developer and ex-maintainer
declared it as abandoned[1], and thus just let's remove it, git
allows to resurrect it any time if a wonder happens anyway.

[0]: https://pve.proxmox.com/pipermail/pve-user/2019-March/170497.html
[1]: http://lists.wpkg.org/pipermail/sheepdog/2019-March/068449.html

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2 months agobuildsys: split storage plugins to single lines
Thomas Lamprecht [Tue, 4 Jun 2019 14:56:46 +0000 (16:56 +0200)]
buildsys: split storage plugins to single lines

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2 months agoDiskmanage: correctly add wearout value of 0
Dominik Csapak [Tue, 4 Jun 2019 11:05:29 +0000 (13:05 +0200)]
Diskmanage: correctly add wearout value of 0

if wearout is 0 we showed 'N/A' instead of 100%
(wearout is really the 'life left' value)

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2 months agoDiskmanage: fix incorrect variable usage
Dominik Csapak [Tue, 4 Jun 2019 11:05:28 +0000 (13:05 +0200)]
Diskmanage: fix incorrect variable usage

sysdir is a string, we wanted sysdata

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2 months agoDiskmanage: extract nvme wearout from smartctl text
Dominik Csapak [Tue, 4 Jun 2019 11:05:27 +0000 (13:05 +0200)]
Diskmanage: extract nvme wearout from smartctl text

extract the info from the line:
Percentage Used: XX%

also adapt the tests

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2 months agotests: test get_disks single/multi disk filter
Dominik Csapak [Tue, 4 Jun 2019 10:35:24 +0000 (12:35 +0200)]
tests: test get_disks single/multi disk filter

this executes all tests again with each disk as a single parameter
and all disks again as an array ref

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2 months agoDiskmanage: add append_partition sub
Dominik Csapak [Tue, 4 Jun 2019 10:35:23 +0000 (12:35 +0200)]
Diskmanage: add append_partition sub

we will use this for adding a partition to a disk when using a device
for ceph osd db/wal which already has partitions on it

first we search for the highest partition number, then add the partition
and search for the resulting device (we cannot assume to simply
append the number, e.g. from /dev/nvme0n1 we get /dev/nvme0n1pX)

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2 months agoDiskmanage: allow get_disks to take multiple disks
Dominik Csapak [Tue, 4 Jun 2019 10:35:22 +0000 (12:35 +0200)]
Diskmanage: allow get_disks to take multiple disks

we now expect the first parameter to be either a string with a single
disk, or an array ref with a list of disks

this way we can get the info of multiple disks simultaneously while
not iterating over all disks

this will be used to get the info for osd/db/wal disk

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2 months agoDisks API: allow also unused disks for db/wal
Dominik Csapak [Tue, 4 Jun 2019 10:35:21 +0000 (12:35 +0200)]
Disks API: allow also unused disks for db/wal

since we will create a pv/vg/lv on it with nautilus

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2 months agotests: improve error handling of run_disk_tests
Dominik Csapak [Tue, 4 Jun 2019 07:21:17 +0000 (09:21 +0200)]
tests: improve error handling of run_disk_tests

if Diskmanage.pm has a syntax error, this will now catch it
during build

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2 months agoLVMPlugin: factor out the lv creation
Dominik Csapak [Tue, 4 Jun 2019 07:21:16 +0000 (09:21 +0200)]
LVMPlugin: factor out the lv creation

we will want to create lvs manually for ceph nautilus db/wal devices

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2 months agofollowup: get_disks: use own variable for frequent access
Thomas Lamprecht [Fri, 31 May 2019 10:04:37 +0000 (12:04 +0200)]
followup: get_disks: use own variable for frequent access

Less reading and the own name for the variable should helps to grasp
more quickly what it should contain

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2 months agofollouwp: get_ceph_volume_infos: code and comment cleanup
Thomas Lamprecht [Fri, 31 May 2019 10:04:13 +0000 (12:04 +0200)]
follouwp: get_ceph_volume_infos: code and comment cleanup

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2 months agoDiskmanage: detect osds/journals/etc. created with ceph-volume
Dominik Csapak [Wed, 29 May 2019 13:48:06 +0000 (15:48 +0200)]
Diskmanage: detect osds/journals/etc. created with ceph-volume

ceph-volume creates osds/journal/etc. on LVM instead of partitions,
so to detect them, we have to parse the lv_tags of the LVs and
match them with the underlying device

also add tests for this detection

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2 months agoallow disk with LVM for journal devices
Dominik Csapak [Wed, 29 May 2019 13:48:05 +0000 (15:48 +0200)]
allow disk with LVM for journal devices

With this, users can select disks with LVM on it for journal/db devices,
which will be necessary for ceph nautilus, since there
journals/db/wal will be put on an LV

of course when creating an osd, we have to detect if that
is ok (probably based on the vg name on it)

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2 months agobuildsys: no need to include arch detection for arch-independent package
Thomas Lamprecht [Tue, 21 May 2019 20:24:42 +0000 (22:24 +0200)]
buildsys: no need to include arch detection for arch-independent package

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2 months agobuildsys: switch upload dist over to buster
Thomas Lamprecht [Tue, 21 May 2019 18:57:05 +0000 (20:57 +0200)]
buildsys: switch upload dist over to buster

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2 months agobump version to 6.0-0+1
Thomas Lamprecht [Tue, 21 May 2019 18:54:49 +0000 (20:54 +0200)]
bump version to 6.0-0+1

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2 months agobuildsys: use dpkg-dev makefile helpers for pkg info
Thomas Lamprecht [Tue, 21 May 2019 18:39:36 +0000 (20:39 +0200)]
buildsys: use dpkg-dev makefile helpers for pkg info

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2 months agod/control: fix priority-extra-is-replaced-by-priority-optional
Thomas Lamprecht [Tue, 21 May 2019 18:31:42 +0000 (20:31 +0200)]
d/control: fix priority-extra-is-replaced-by-priority-optional

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
3 months agobump version to 5.0-43
Thomas Lamprecht [Wed, 15 May 2019 14:16:58 +0000 (16:16 +0200)]
bump version to 5.0-43

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
3 months agoapi/config update: indentation and whitespace fixes
Thomas Lamprecht [Wed, 15 May 2019 08:36:01 +0000 (10:36 +0200)]
api/config update: indentation and whitespace fixes

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
3 months agoapi/config update: only iterate over hash keys, not values
Thomas Lamprecht [Wed, 15 May 2019 08:34:12 +0000 (10:34 +0200)]
api/config update: only iterate over hash keys, not values

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
3 months agopvesm set: handle deletion of properties
Thomas Lamprecht [Tue, 14 May 2019 14:18:50 +0000 (16:18 +0200)]
pvesm set: handle deletion of properties

the delete parameter get's injected by the SectionConfigs
updateSchem, but we need to handle it ourself in the code
This makes the following possible:

pvesm set STORAGEID --delete property

Also the API equivalent is now possible. Adapted from the HA
managers Resource update API call.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
3 months agobump version to 5.0-42
Thomas Lamprecht [Tue, 30 Apr 2019 13:56:23 +0000 (13:56 +0000)]
bump version to 5.0-42

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
3 months agomap_volume: fall back to 'path'
Mira Limbeck [Mon, 29 Apr 2019 13:00:38 +0000 (15:00 +0200)]
map_volume: fall back to 'path'

Adds a fallback to 'Plugin::path' in the default implementation of
'map_volume' to simplify a common case of calling 'map_volume' followed
by a defined-check and a call to path if it is not. The path is now
always returned if the plugin in question does not override
'map_volume'.

Signed-off-by: Mira Limbeck <m.limbeck@proxmox.com>
4 months agozpool: handle race with other zpool imports
Thomas Lamprecht [Wed, 17 Apr 2019 15:00:20 +0000 (15:00 +0000)]
zpool: handle race with other zpool imports

The underlying issue is that a zpool can get imported only once, so
we first check if it's in `zpool list`, and thus imported, and only
if it does not shows up there we try to import it.

But, this can race with either:
* parallel running activate_storage call, through CLI/API/daemon
* a zpool import from an admin (a bit unlikely, but hey that's the
  thing with race conditions ;))

So refactor the "is pool imported" check into a closure, and call it
addditionally if the import failed, and silent the error if the pool
is now listed, and thus imported. This makes it a little bit nicer to
read too, IMO.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
4 months agoadd zfsutils-linux to build-dependencies
Fabian Grünbichler [Thu, 18 Apr 2019 08:00:45 +0000 (10:00 +0200)]
add zfsutils-linux to build-dependencies

to enable our ZFS tests and fix broken disk list tests.

4 months agozPool: fixup timeout setting for import
Thomas Lamprecht [Wed, 17 Apr 2019 14:39:37 +0000 (14:39 +0000)]
zPool: fixup timeout setting for import

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
4 months agozpool: cleanup zfs_request command a bit
Thomas Lamprecht [Wed, 17 Apr 2019 14:31:05 +0000 (14:31 +0000)]
zpool: cleanup zfs_request command a bit

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
4 months agoZFS Pool: improve error output from activate_storage
Wolfgang Link [Tue, 16 Apr 2019 13:18:22 +0000 (15:18 +0200)]
ZFS Pool: improve error output from activate_storage

related to #2154 which has some issues on zpool list, but we do not
see the error messages from that step : Buggy "pvesm status" output

4 months agoStatus: Include command error in error message when storage activation fails
Christian Ebner [Wed, 10 Apr 2019 15:17:14 +0000 (17:17 +0200)]
Status: Include command error in error message when storage activation fails

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
4 months agofollowup: code cleanup
Thomas Lamprecht [Wed, 10 Apr 2019 08:14:46 +0000 (10:14 +0200)]
followup: code cleanup

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
4 months agoFix #318: Delete vzdump log when deleting a backup
Dominic Jäger [Tue, 9 Apr 2019 07:41:25 +0000 (09:41 +0200)]
Fix #318: Delete vzdump log when deleting a backup

Vzdump log files were not deleted when a backup was deleted.
Consequently, the folder continuously filled with .log files.
Now they get deleted after the backup is removed.

Signed-off-by: Dominic Jäger <d.jaeger@proxmox.com>
4 months agobump version to 5.0-41
Thomas Lamprecht [Mon, 8 Apr 2019 15:54:08 +0000 (17:54 +0200)]
bump version to 5.0-41

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
4 months agocode cleanup
Thomas Lamprecht [Mon, 8 Apr 2019 15:48:39 +0000 (17:48 +0200)]
code cleanup

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
4 months agoDiskmanage: don't run zpool if not present
Stoiko Ivanov [Mon, 8 Apr 2019 12:38:36 +0000 (14:38 +0200)]
Diskmanage: don't run zpool if not present

Since zfsutils are not a hard dependency of our stack it is possible to not have
`zpool` available.

Checking for existance of `zpool` before calling it suppresses spurious warnings
in the logs (e.g. when creating Ceph OSDs or accessing the 'Disk' Tab in the
GUI).

Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
4 months agoadd 3 bwlimit tests
Stoiko Ivanov [Fri, 5 Apr 2019 14:33:35 +0000 (16:33 +0200)]
add 3 bwlimit tests

1) checking that the empty storage list is treated correctly (only override
and datacenter config limit considered)
2) checking that the empty storage list is treated correctly (as with 1).
3) checking that undef can be passed as one element of the storage list (it is
ignored)

Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
4 months agoget_bandwidt_limits: ignore 'undef' as storage
Stoiko Ivanov [Fri, 5 Apr 2019 14:33:34 +0000 (16:33 +0200)]
get_bandwidt_limits: ignore 'undef' as storage

If one of the storages passed in $storage_list was not defined
get_bandwidth_limit died (see [0], of an occurence of this).
This patch changes the behavior to ignore undef as storage instead.

[0] https://pve.proxmox.com/pipermail/pve-devel/2019-April/036515.html

Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
4 months agobump version to 5.0-40
Thomas Lamprecht [Thu, 4 Apr 2019 14:19:51 +0000 (16:19 +0200)]
bump version to 5.0-40

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
4 months agozfs: don't generate/update cachefile on pool import
Fabian Grünbichler [Wed, 3 Apr 2019 09:06:07 +0000 (11:06 +0200)]
zfs: don't generate/update cachefile on pool import

during storage activation.

for pools that don't get imported at boot (e.g. because their vdevs are
not available when zfs-import-*.service runs) it is fatal to include
them in the cachefile, for those that do get imported at boot this code
should never run anyway as they are already imported.

in any case, a fallback to import without cachefile is the safe variant.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
4 months agoworkaround zfs create -V error for unaligned sizes
Mira Limbeck [Fri, 29 Mar 2019 09:13:10 +0000 (10:13 +0100)]
workaround zfs create -V error for unaligned sizes

fixes the 'cannot create 'nvme/foo': volume size must be a multiple of
volume block size' error by always rounding the size up to the next 1M
boundary. this is a workaround until
https://github.com/zfsonlinux/zfs/issues/8541 is solved.
the current manpage says 128k is the maximum blocksize, but a local test
showed that values up to 1M are allowed. it might be possible to
increase it even further (see f1512ee61).

Signed-off-by: Mira Limbeck <m.limbeck@proxmox.com>
4 months agoStorage::get_bandwidth_limit: fix if condition
Stoiko Ivanov [Fri, 29 Mar 2019 07:27:59 +0000 (08:27 +0100)]
Storage::get_bandwidth_limit: fix if condition

Passing 'undef' as '$storage_list' led to a warning about using an
uninitialized value as array_ref.

Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
4 months agoadd libfile-chdir-perl to build dependencies
Thomas Lamprecht [Thu, 21 Mar 2019 09:51:19 +0000 (10:51 +0100)]
add libfile-chdir-perl to build dependencies

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
5 months agore-bump version to 5.0-39
Thomas Lamprecht [Thu, 7 Mar 2019 11:22:57 +0000 (12:22 +0100)]
re-bump version to 5.0-39

this wasn't released yet to any repo and I'd like to have the iSCSI
fix included, so just re-run 'dch -r'

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
5 months agofix tests when one has iscsi devices
Dominik Csapak [Thu, 7 Mar 2019 09:49:04 +0000 (10:49 +0100)]
fix tests when one has iscsi devices

the test would read the real device and if one is an iscsi device
it would fail, move the test code to a sub and mock it in the tests

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
5 months agofix content listing for user mode iscsi plugin
Dominik Csapak [Thu, 7 Mar 2019 09:49:03 +0000 (10:49 +0100)]
fix content listing for user mode iscsi plugin

the format is a required in the result schema

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
5 months agobump version to 5.0-39
Thomas Lamprecht [Wed, 6 Mar 2019 09:39:18 +0000 (10:39 +0100)]
bump version to 5.0-39

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
5 months agofix #585: remove leftover disks/directory after VM creation failed
Christian Ebner [Mon, 4 Mar 2019 15:47:06 +0000 (16:47 +0100)]
fix #585: remove leftover disks/directory after VM creation failed

When trying to create a qcow2 disk image with a size larger than available on the
storage, this will fail.
As qemu-img does not clean up the disk afterwards, it needs to be deleted
explicitly. Further, the vmid folder is cleaned up once it is empty.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
5 months agod/control: bump version dependency to pve-doc-generator
Thomas Lamprecht [Fri, 22 Feb 2019 12:31:32 +0000 (13:31 +0100)]
d/control: bump version dependency to pve-doc-generator

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
5 months ago1891 Add zsh command completion for pvesm
Christian Ebner [Thu, 21 Feb 2019 13:25:02 +0000 (14:25 +0100)]
1891 Add zsh command completion for pvesm

This adds the zsh command completion generation for pvesm.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
5 months agofollowup: try to be a bit more like systemd-escape
Thomas Lamprecht [Wed, 20 Feb 2019 15:31:25 +0000 (16:31 +0100)]
followup: try to be a bit more like systemd-escape

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
5 months agofollowup comment that we do not escape completely like systemd
Thomas Lamprecht [Wed, 20 Feb 2019 15:30:46 +0000 (16:30 +0100)]
followup comment that we do not escape completely like systemd

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
5 months agofix #2099: escape systemd path names in mount unit
Dominik Csapak [Wed, 20 Feb 2019 15:04:34 +0000 (16:04 +0100)]
fix #2099: escape systemd path names in mount unit

we only allow '-' '_' and '.' int storage-ids and names,
and we do not need to escape '_' and '.' (see man 5 systemd.unit)

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
6 months agobump version to 5.0-38
Thomas Lamprecht [Thu, 7 Feb 2019 14:15:39 +0000 (15:15 +0100)]
bump version to 5.0-38

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
6 months agofollowup: improve comment
Thomas Lamprecht [Tue, 5 Feb 2019 16:19:35 +0000 (17:19 +0100)]
followup: improve comment

while the commit message tells it nicely a comment should add
additional info for people just giving this a quick look

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
6 months agocheck_volume_access: tighten checks for iso/tmpl
Fabian Grünbichler [Tue, 5 Feb 2019 15:01:40 +0000 (16:01 +0100)]
check_volume_access: tighten checks for iso/tmpl

(custom) templates might contain sensitive data, so require at least
read access on the underlying storage to access ISO and template files.

the same permissions are already needed for listing them, so this is
unlikely to cause fallout.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
6 months agod/control: bump version dependency of libpve-common-perl
Thomas Lamprecht [Mon, 4 Feb 2019 11:02:21 +0000 (12:02 +0100)]
d/control: bump version dependency of libpve-common-perl

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
6 months agodiskmanage: fix device encoding handling
Wolfgang Bumiller [Mon, 4 Feb 2019 09:42:01 +0000 (10:42 +0100)]
diskmanage: fix device encoding handling

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
6 months agobump version 5.0-37
Thomas Lamprecht [Fri, 1 Feb 2019 11:39:32 +0000 (12:39 +0100)]
bump version 5.0-37

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
6 months agoallow snippets by default for new dir storages
Dominik Csapak [Tue, 29 Jan 2019 15:28:40 +0000 (16:28 +0100)]
allow snippets by default for new dir storages

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
6 months agoenable snippets content type for all directory based storages
Dominik Csapak [Tue, 29 Jan 2019 15:28:39 +0000 (16:28 +0100)]
enable snippets content type for all directory based storages

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
6 months agoadd new content type 'snippets'
Dominik Csapak [Tue, 29 Jan 2019 15:28:38 +0000 (16:28 +0100)]
add new content type 'snippets'

will be used to contain files which can be executed as hookscripts or
contain custom cloud-init configs

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
6 months agofix #1598: use glusterfs daemon default port for online check
Thomas Lamprecht [Mon, 28 Jan 2019 13:37:05 +0000 (14:37 +0100)]
fix #1598: use glusterfs daemon default port for online check

use the port where the main glusterfs daemon listens on as ping port,
this one is also used by QEMU as default.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
6 months agoFix #1941: remove empty directories when freeing image on FS based storages
Christian Ebner [Thu, 24 Jan 2019 13:58:22 +0000 (14:58 +0100)]
Fix #1941: remove empty directories when freeing image on FS based storages

Remove directories if they are empty, which can happen if all images
from a VM got deleted, e.g., after destroying said VM.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
7 months agoFix #2050: only provide 'conv=sparse' for LvmThin
Stoiko Ivanov [Thu, 17 Jan 2019 15:31:51 +0000 (16:31 +0100)]
Fix #2050: only provide 'conv=sparse' for LvmThin

LVMPlugin->volume_import (used by storage_migrate on either offline
migration with local disks, or online migration with storage-only
referenced disks) passed 'conv=sparse' to `dd`. This can lead to
data-corruption, if the target volume is not zero-initialized.

dropping the sparse argument completely would fix the problem, but
breaks keeping data sparse for LvmThinPlugin.

This patch moves the dd out into (LVM*) plugin specific sub so that
each can control the parameters.

Steps for reproducing the issue:
* create a cluster with (at least) 2 nodes A and B, with a free
  disk-device (/dev/sdx)
* write a recognizable pattern to /dev/sdx on B:
  `dd if=/dev/zero bs=10M | tr '\000' '\255' | dd of=/dev/sdb bs=10M`
  (would be grateful for alternatives to the dd| tr| dd)
* on both A and B create a lvm-vg (pvcreate, vgcreate)
* add it as _not_ shared storage, which is available on nodes A and B
* create a small guest on A
* fill a file in the guest with zeros
  `dd if=/dev/zero of=/zerofil bs=10M`
* stop the guest, migrate it to B
* start the guest - check that the file `/zerofil` contains `ad`
  instead of `00`

Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
7 months agoupdate version to 5.0-36
Thomas Lamprecht [Tue, 15 Jan 2019 12:18:28 +0000 (13:18 +0100)]
update version to 5.0-36

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
7 months agoLVM: Add '--refresh' when activating volumes
Stoiko Ivanov [Fri, 4 Jan 2019 13:06:25 +0000 (14:06 +0100)]
LVM: Add '--refresh' when activating volumes

From `man 8 lvchange`:
  --refresh
      If the logical volume is active, reload its metadata. This is not
      necessary in normal operation, but may be useful ... if you're doing
      clustering manually without a clustered lock manager.

Fixes migration in a shared LVM (iscsi) setup, where a disk gets resized on one
node A and the guest is afterwards migrated to another node B: B still presents
the old size to the guest, leading to data corruption.

It is necessary to run `lvchange` twice because the options `-ay` and
`--refresh` are mutually exclusive.

Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>