Max Carrara [Tue, 2 Apr 2024 14:55:20 +0000 (16:55 +0200)]
cephconfig: align our parser with Ceph's parser
This commit rewrites the entire parser for ceph.conf, aligning its
behaviour as closely as possible with Ceph's parser grammar [0].
The most notable improvements are as follows:
1. The characters '#' and ';' now both mark comments, instead of
just the '#' character.
2. Any character, including comment literals ('#' and ';'), may now
be escaped.
3. Quoted values (single and double) are now supported.
4. Line continuations are now supported (lines ending with '\').
5. Repeated whitespace characters in keys are now treated as a
single space character.
6. Dashes '-' are not treated the same as spaces and underscores
anymore, as Ceph's grammar doesn't treat them that way.
* Paired with 5., this means that repeated whitespace is now
equivalent to a single underscore.
7. Escaped comment literals are now un-escaped.
8. Although not too crucial, the parser now also supports empty
sections and will just initialize them with an empty hash.
Furthermore, the original grammar's more quirky behaviours are also
respected where sanely possible.
This commit changes the code style of subroutine `write_ceph_config`
to match our style guide [0] more.
Furthermore, the repeated calls to the inner subroutine are replaced
with a loop, while the regular expressions used by the inner `sub` are
now quoted with `qr` to prevent any accidental mis-quotings in the
future.
Instead of just using it as a warning and then trying to parse an
empty string as json.
For example, trying to parse unsupported vmdks, previously we'd see
something like this:
qemu-img: Could not open
'/run/pve/import/esxi/foo/mnt/ha-datacenter/vsanDatastore/asdf/asdf-000001.vmdk':
Unsupported image type 'vsanSparse'
could not parse qemu-img info command output for
'/run/pve/import/esxi/foo/mnt/ha-datacenter/vsanDatastore/asdf/asdf-000001.vmdk'
- malformed JSON string, neither tag, array, object, number, string
or atom, at character offset 0 (before "(end of string)") at
src/PVE/Storage/Plugin.pm line 962, <DATA> line 960.
Now it simply shows:
qemu-img: Could not open
'/run/pve/import/esxi/foo/mnt/ha-datacenter/vsanDatastore/asdf/asdf-000001.vmdk':
Unsupported image type 'vsanSparse'
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Thomas Lamprecht [Wed, 27 Mar 2024 12:11:22 +0000 (13:11 +0100)]
esxi: reduce cache invalidation time to 30s
Reduce the time the cache stays valid from 60s to 30s, while this
could double the amount of requests in the worst case, it's still not
that frequent and also halves the maximal time a user has to wait to
see changes on the ESXi side to appear here.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Gabriel Goller [Thu, 21 Mar 2024 09:07:52 +0000 (10:07 +0100)]
esxi: detect correct os type in 'other' family
This patch introduces the conversion table for all possible OS Types
that are in the VMWare 'other' family and sets the pve counterpart.
Our default OS Type is 'linux', so including mappings to 'other' makes
sense.
Signed-off-by: Gabriel Goller <g.goller@proxmox.com>
following pve-esxi-import-tools's commits: 3ee5c3b ("esxi-folder-fuse: add --insecure option") c292c67 ("listvms.py: add --insecure parameter, verify cert by
default") 34c87be ("rename --insecure option to --skip-cert-verification")
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
[ TL: rename 'insecure' to 'skip-cert-verification' to better convey
what it means ] Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
[ TL: fix wrong comparison with >= and avoid undef warning if file
does not yet exist at all ] Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
[ TL: squash removal of both subs into one commit ] Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Thomas Lamprecht [Sun, 10 Mar 2024 18:25:34 +0000 (19:25 +0100)]
api: import-metadata: make warnings structured & merge ignored-volumes
This allows the frontends to translate them and avoids somewhat
duplicated info by having some warnings explicitly (ignored-volumes)
while others are in the warnings array.
By passing along the key and the value the frontend can also show the
warnings in-line, e.g. by marking a disk-entry in a grid as having
potential problems.
Ideally we'd have a central list of known types used for the API
return schema enum and to check when calling the $warn closure, but as
we only got three warnings keep this as is and only add a comment.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
This will be used for returning the base meta information of a
external VM that is about to be imported into Proxmox VE.
A front-end can use this endpoint to show the proposed configs with
potential override switches to the user, so that they can adapt the
most important options to ensure that import can work.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
[ TL: add more commit message with some background ] Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Add a new 'import' content type which will be the corner stone for a
better API and UI integrated way to import virtual guests into Proxmox
VE.
For starters this will be used to implement a ESXi adapter, so that
those VMs can get imported nicely.
Later we want to integrate the OVF/OVA import skeletons we got in
qemu-server to something more usable here.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
[ TL: add more commit message with some background ] Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Thomas Lamprecht [Mon, 19 Feb 2024 16:13:41 +0000 (17:13 +0100)]
fix #5254: api: allow usage of download-url with Sys.AccessNetwork
The download-url API endpoint has some implications that admins are
unaware of, namely that it basically allow to scan the whole network
via HTTP URLs, and potentially even download some image that the user
should not have access to and adding to a VM that the user controls.
That's why in addition to the Datastore.AllocateTemplate privilege on
the storage, the Sys.Modify on the whole Cluster was required to use
the API call. That design was chosen as we were not fully sure if a
separate privilege is warranted, but user feedback has shown that the
(not so big) cost of adding such a new privilege is justified.
Change the permission check to allow the combination of
Datastore.AllocateTemplate on the storage and either 'Sys.Modify' on
/, for backwards compatibility, or the newer 'Sys.AccessNetwork' on
the node that handles the download.
Using a node-specific ACL path allows admins to e.g. prepare one
specific node's firewall so that pveproxy can access only a safe set
of hosts via outgoing HTTP (not stemming from valid connection
tracking to the PVE API), and thus even further limit the privileges
of users or tools that are trusted to download images to a storage.
Buglink: https://bugzilla.proxmox.com/show_bug.cgi?id=5254 Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com> Tested-by: Hannes Duerr <h.duerr@proxmox.com> Reviewed-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Thomas Lamprecht [Sun, 19 Nov 2023 19:05:50 +0000 (20:05 +0100)]
btrfs: fix calling parent create_base method in fall-back
If we want to forward to the create_base of the directory plugin while
making that use our $class for the operations that call might do, we
cannot use the -> notation (which would resolve the next actual
implementation) but rather pass the class directly.
But, DirPlugin reuses the create_base method from the base Plugin
method, so we also need to call that, because on direct call notation
the inheritance fallback to super methods isn't available.
Reported in the forum:
https://forum.proxmox.com/threads/95684/post-606535
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
fix #254: iscsi: add support for multipath targets
With this patch Proxmox now tries to login to all discovered portals
in case some of them are not logged yet.
In case of multipath configuration when initially configured portal is
missing for some reason Proxmox don't lose iSCSI storage now and can
successfully restore iSCSI connection between reboots.
Fiona Ebner [Tue, 27 Jun 2023 07:48:49 +0000 (09:48 +0200)]
cifs: bubble up NT_STATUS_INVALID_PARAMETER during connection check
instead of claiming that the storage is not online.
Would've made the issue fixed by b27da68 ("cifs: fix check connection
call") more obvious, because (the UI passes along an empty string for
domain if not set and) the smbclient call returns that status with
> -W ''
in Bookworm.
Philipp Hufnagl [Mon, 14 Aug 2023 14:42:17 +0000 (16:42 +0200)]
fix #4849: download-url: allow download and decompression of compressed ISOs
adds information for how to decompress isos.
generates the compressor regex from a list of comression formats (to
avoid redundancy)
extends the download_url wtih the functionality to handley compression
for images
Signed-off-by: Philipp Hufnagl <p.hufnagl@proxmox.com>
Thomas Lamprecht [Sat, 17 Jun 2023 12:53:05 +0000 (14:53 +0200)]
disk api: only ask for Datastore.Allocate if adding to storage config
The Proxmox VE storage systems doesn't cares at all if the
Datastore.Allocate privilege is present if no Proxmox VE storage will
be allocated.
Note, if we want to restrict this further as Sys.Modify on /, which
is already quite a powerful permission, we should probably add a new
one under the Sys. space, e.g., Sys.Disk.Use or the like.
This is a step in splitting the disk manage code out of the
pve-storage package, and maybe even repository
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Thomas Lamprecht [Sat, 17 Jun 2023 12:22:28 +0000 (14:22 +0200)]
disk api: avoid using unrelated Datastore.Audit priv for disk management
Local disk and storage creation and listing is something rather
different than the Proxmox VE storage client ABI that provides an
abstract access to a variety of storage types, specifically targeted
to virtual guests images, templates and backups.
The Datastore.* privilege group is specifically made for auditing the
abstract configuration, here the name must be interpreted in context
and not just assumed that due to "datastore" sounding like it could
have to do something with disks or creation of local storage it just
must be a good fit.
Luckily, Sys.Audit was already used too, which is the correct one
here, this is for node specific (HW) details, not some config for
accessing datastore in a restricted way.
This is a step in splitting the disk manage code out of the
pve-storage package, and maybe even repository.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Friedrich Weber [Thu, 15 Jun 2023 11:36:58 +0000 (13:36 +0200)]
content-dirs check: silently skip paths that cannot be resolved
Since commit 8e623a2930f7aee4b3309b1f297613a250ee4698, the inequality
check for content-dirs prints a warning if a content directory path
could not be resolved, i.e., if `abs_path` returns undef. Among other
things, `abs_path` returns undef if the path has an inner (= any but
last) component that does not exist. This can happen for a storage
with content type `iso,vztmpl` and `create-subdirs` set to 0, in case
`template/` does not exist. In this case, the warnings printed by
pvestatd are quite noisy.
As missing content directories are not a problem per se, remove the
warning and just ignore the directory during the inequality check.
Signed-off-by: Friedrich Weber <f.weber@proxmox.com>
Fiona Ebner [Mon, 12 Jun 2023 14:27:33 +0000 (16:27 +0200)]
api: config: add/update storage: check for type mismatch first
This avoids confusing errors about other properties when the storage
type doesn't match. By highlighting that the type doesn't match, users
should know right away what the issue is.
content dirs: skip creation if either mkdir or create-subdirs is false
This is slightly confusing due to both options, the legacy convoluted
one and the new targeted one, exist, but before the rework we skip if
either of those sub-expressions was true, so doing it needs both to
be true.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
activate storage: ensure content directories are created before checking them
checking the content dirs for clashes via abs_path must be done after
the logic for creating them ran, as abs_path is working on actual
filesystem level, so it will return undf if the directory does not
exist, in which case we then set a hash entry for "undef", and the
next for loop round then resolved again to "undef", resulting in a
false-positive of the check.
Avoid the dangerous "return if" stanzas and reverse them to an actual
if block, which is much safer to adapt. Then move the check for
duplicate content-dir usage after that.
best viewed with white space change ignored: git show -w
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Stefan Hrdlicka [Wed, 1 Mar 2023 12:13:25 +0000 (13:13 +0100)]
fix #2920: cifs: add options parameter
This makes it possible to add all mount options offered by mount.cifs.
NFS & CIFS now share the options parameter since they use it for the
same purpose.
Signed-off-by: Stefan Hrdlicka <s.hrdlicka@proxmox.com>
[FE: rebase + style fixes] Signed-off-by: Fiona Ebner <f.ebner@proxmox.com> Tested-by: Friedrich Weber <f.weber@proxmox.com>
[T: fix merge conflict ] Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
rbd: volume resize: avoid passing floating point value to rbd
which causes an error "the argument for option '--size' is invalid".
Just round up to the nearest integer to have at least the requested
size. This is similar to what is done for ZFS with d3e3e5d ("When
resizing a ZFS volume, align size to 1M") and makes commands like 'qm
resize 102 scsi1 +0.01G' work.
It was introduced by commit 4b7dd9d ("allow --allow-shrink on RBD
resize"), but doesn't give a rationale. A mail gives more[0],
indicating that the user also uses the function to shrink images.
However, the volume_resize function is only reachable via the resize
API endpoints for VMs and containers, which have an explicit check to
disallow shrinkage. If somebody really wants to shrink the image, just
let them use the storage's tools directly. Calling into Proxmox VE's
perl functions directly is not supported.
plugin: simplify and fix create-base-path vs mkdir logic
In the previous code, if `create-base-path` was explicitly
set to false, it would be treated the same as if it was
undef, falling through to whatever 'mkdir' was.
Instead, the new options should always be preferred, and the
logic can be simplified to a single line.
Here's the table showing the difference, 'u' being 'undef':
config: mkdir: u 0 1 u 0 1 u 0 1
create: u u u 0 0 0 1 1 1
=========================
mkpath: old: 1 0 1 0 0 1 1 1 1
new: 1 0 1 0 0 0 1 1 1
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
projects / pve-storage.git / log