Dominik Csapak [Mon, 18 Nov 2024 15:29:14 +0000 (16:29 +0100)]
plugin: file_size_info: warn on parent images with unusual path
If the base image (parent) of an image contains e.g. whitespace in it's
path, the current untainting would not match and it would seem there was
no parent.
Since untrusted files are not allowed to have backing parts, just warn,
when encountering this case to keep backwards compatibility.
Dominik Csapak [Mon, 18 Nov 2024 15:29:10 +0000 (16:29 +0100)]
ovf: implement parsing nics
by iterating over the relevant parts and trying to parse out the
'ResourceSubType'. The content of that is not standardized, but I only
ever found examples that are compatible with vmware, meaning it's
either 'e1000', 'e1000e' or 'vmxnet3' (in various capitalizations; thus
the `lc()`)
As a fallback i used e1000, since that is our default too, and should
work for most guest operating systems.
Dominik Csapak [Mon, 18 Nov 2024 15:29:08 +0000 (16:29 +0100)]
ovf: implement parsing out firmware type
it seems there is no part of the ovf standard that handles which type of
bios there is (at least i could not find it). Every ovf/ova i tested
either has no info about it, or has it in a vmware specific property
which we parse here.
Dominik Csapak [Mon, 18 Nov 2024 15:29:06 +0000 (16:29 +0100)]
ovf: improve and simplify path checking code
moves the filepath code a bit more closer to where it's actually used
checks the contained path before trying to find it's absolute path
properly add error handling to realpath
instead of checking the combined ovf_path + filepath, just make sure
filepath can't point to anythign besides a file in this directory
by checking for '.' and '..' (slashes are not allowed in SAFE_CHAR_CLASS_RE)
Dominik Csapak [Mon, 18 Nov 2024 15:29:05 +0000 (16:29 +0100)]
plugin: dir: handle ova files for import
since we want to handle ova files (which are only ovf+images bundled in
a tar file) for import, add code that handles that.
we introduce a valid volname for files contained in ovas like this:
storage:import/archive.ova/disk-1.vmdk
by basically treating the last part of the path as the name for the
contained disk we want.
in that case we return 'import' as type with 'vmdk/qcow2/raw' as format
(we cannot use something like 'ova+vmdk' without extending the 'format'
parsing to that for all storages/formats. This is because it runs
though a verify format check at least once)
we then provide a function to use for that:
* extract_disk_from_import_file: this actually extracts the file from
the archive. Currently only ova is supported, so the extraction with
'tar' is hardcoded, but again we can easily extend/modify that should
we need to.
we currently extract into the either the import storage or a given
target storage in the images directory so if the cleanup does not
happen, the user can still see and interact with the image via
api/cli/gui
we have to modify the `parse_ovf` a bit to handle the missing disk
images, and we parse the size out of the ovf part (since this is
informal only, it should be no problem if we cannot parse it sometimes)
Dominik Csapak [Mon, 18 Nov 2024 15:29:04 +0000 (16:29 +0100)]
plugin: dir: implement import content type
in DirPlugin and not Plugin (because of cyclic dependency of
Plugin -> OVF -> Storage -> Plugin otherwise)
only ovf is currently supported (though ova will be shown in import
listing), expects the files to not be in a subdir, and adjacent to the
ovf file.
listed will be all ovf/qcow2/raw/vmdk files.
ovf because it can be imported, and the rest because they can be used
in the 'import-from' part of qemu-server.
Thomas Lamprecht [Mon, 18 Nov 2024 15:01:35 +0000 (16:01 +0100)]
d/control: bump versioned dependency for libpve-common-perl
to ensure the new verification callback for downloaded files is
available, while it's not a hard failure if it's missing it's good to
ensure that it's active and can work as intended.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Dominik Csapak [Mon, 18 Nov 2024 14:31:12 +0000 (15:31 +0100)]
api: iso up/download: check file content
by letting it run through 'file_size_info' as 'untrusted', since that
does the necessary checks. We do this so we don't accidentally
up/download a file that is not a valid iso
Dominik Csapak [Fri, 15 Nov 2024 15:17:23 +0000 (16:17 +0100)]
move OVF module over from qemu-server
Copy over the PVE::QemuServer::OVF module and relevant OVF tests from
the qemu-server package/repo.
We need it here for implementing the import content type support to
generic directory based plugins.
So it will also use PVE::Storage modules and thus anything higher, or
a different package, makes things only harder for now.
Put the OVF module under the new PVE::GuestImport module namespace.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
[ TL: rework commit message to avoid file endings and clarify
intentions ] Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
this allows checking some extra attributes for images which come from
a potentially malicious source.
since file_size_info is not part of the plugin API, no API bump is
needed. if desired, a similar check could also be implemented in
volume_size_info, which would entail bumping both APIVER and APIAGE
(since the additional parameter would make checking untrusted volumes
opt-in for external plugins).
Mira Limbeck [Mon, 11 Nov 2024 15:01:03 +0000 (16:01 +0100)]
iscsi: verify volume disks are part of target
We build the disk path by appending the last part of the volname to
/dev/disk/by-id. These could in theory be any other disk found under
there instead of a LUN provided by the target configured.
This patch adds a way to verify the disk used is actually provided by
the target. To do so `udevadm` is used to get the devpath
(/devices/...). This can then be checked under `/sys` for a session.
With the session the targetname can be looked up under /sys/class and
compared with the configured target of the storage.
In case of multipath, all disks backing the multipath device are checked
recursively (in case of nested device mapper devices), and verification
succeeds if at least one backing disk is part of the iSCSI target.
Mixing disks from different iSCSI targets is allowed as long as one
corresponds to the right target.
udevadm input is limited to `/dev/` paths since we only pass those either
explicitly, or via Cwd::realpath on a /dev/disk/by-id path returned by
filesystem_path.
According to [0] /sys/subsystems should be preferred over /sys/class if
available, but neither kernel 6.8 nor kernel 6.11 provided it. It is
mentioned that in the future this will be moved to /sys/subsystems. So
this has to be kept in mind for future kernels.
Friedrich Weber [Tue, 5 Nov 2024 16:37:44 +0000 (17:37 +0100)]
iscsi: fix activation of second iSCSI storage on other cluster nodes
Assume a cluster that already has an iSCSI storage A configured. After
adding a new iSCSI storage B with a different target on node 1, B will
only become active on node 1, not on the other nodes. On other nodes,
pvestatd logs 'storage B is not online'. The storage does not become
available even after a reboot. A workaround is to manually perform
iSCSI discovery against B's targets on the other nodes once.
This happens because the connectivity check of the iSCSI plugin on
node B does not correctly handle the case that iscsiadm already knows
portals (i.e., A's portals) but not B's portals.
The connectivity check calls `iscsi_portals` to determine the portals
to ping, which calls `iscsiadm -m node` to query all known portals,
and extracts all portals to the storage's target. If the iscsiadm
command fails, `iscsi_portals` returns the portal given in the storage
config. This works as expected if the storage is the first iSCSI
storage, because then iscsiadm does not know any portals and thus
exits with code 21.
However, since there already is an iSCSI storage A, iscsiadm exits
cleanly but its output does not contain any portals for B's target.
Hence, `iscsi_portals` returns an empty array of portals, so the
connectivity check fails and node 2 never performs discovery for B.
To fix this, let `iscsi_portals` also return the portal from B's
storage config if iscsiadm exited cleanly but its output contained no
matching portal.
Signed-off-by: Friedrich Weber <f.weber@proxmox.com>
Friedrich Weber [Fri, 11 Oct 2024 12:58:21 +0000 (14:58 +0200)]
iscsi: disable Open-iSCSI login retries
Since 90c1b10 ("fix #254: iscsi: add support for multipath targets"),
iSCSI storage activation checks whether a session exists for each
discovered portal. If there is a discovered portal without a session,
it performs a discovery and login in the hope of establishing a
session to the portal. If the portal is unreachable when trying to log
in, Open-iSCSI's default behavior is to retry for up to 2 minutes, as
explained in /etc/iscsi/iscid.conf:
> # The default node.session.initial_login_retry_max is 8 and
> # node.conn[0].timeo.login_timeout is 15 so we have:
> #
> # node.conn[0].timeo.login_timeout * \
> node.session.initial_login_retry_max = 120s
If pvestatd is activating the storage, it will be blocked during that
time, which is undesirable. This is particularly unfortunate if the
target announces portals that the host permanently cannot reach. In
that case, every pvestatd iteration will take 2 minutes. While it can
be argued that such setups are misconfigured, it is still desirable to
keep the fallout of that misconfiguration as low as possible.
In order to reduce the time Open-iSCSI tries to log in, instruct
Open-ISCSI to not perform login retries for that target. For this, set
node.session.initial_login_retry_max for the target to 0. This setting
is stored in Open-iSCSI's records under /etc/iscsi/nodes. As these
records are overwritten with the defaults from /etc/iscsi/iscsid.conf
on discovery, the setting needs to be applied after discovery.
With this setting, one login attempt should take at most 15 seconds.
This is still higher than pvestatd's iteration time of 10 seconds, but
more tolerable. Logins will still be continuously retried by pvestatd
in every iteration until there is a session to each discovered portal.
Signed-off-by: Friedrich Weber <f.weber@proxmox.com> Tested-by: Mira Limbeck <m.limbeck@proxmox.com> Reviewed-by: Mira Limbeck <m.limbeck@proxmox.com>
It's not enough to check whether $! is set. From "perldoc perlvar":
> Many system or library calls set "errno" if they fail, to
> indicate the cause of failure. They usually do not set "errno"
> to zero if they succeed and may set "errno" to a non-zero value
> on success. This means "errno", hence $!, is meaningful only
> *immediately* after a failure:
To protect against potential issues, check the return value of unlink
and only check $! if it failed.
Adds the ability to change the owner of a guest image.
Btrfs does not need special commands to rename a subvolume and this can
be achieved the same as in Storage/plugin.pm's rename_volume taking
special care of how the directory structure used by Btrfs.
Daniel Kral [Wed, 21 Aug 2024 13:57:47 +0000 (15:57 +0200)]
esxi: fix #5587: add support for older version of vmx storage filepaths
Allow the ESXi storage disk entry property "fileName" to be flatcased
("filename") in addition to being camelcased ("fileName"). This adds
compatibility with older ESXi .vmx configuration files.
Fiona Ebner [Mon, 10 Jun 2024 09:04:15 +0000 (11:04 +0200)]
volume import: assume target API version is at least 9
The storage API version has been bumped to at least 9 since
libpve-storage = 7.0-4. If the source node is on Proxmox VE 8, where
this change will come in, then the target node can be assumed to be
running either Proxmox VE 8 or, during upgrade, the latest version of
Proxmox VE 7.4, so it's safe to assume a storage API version of at
least 9 in all cases.
As reported by Maximiliano, the fact that the 'apiinfo' call was
guarded with a quiet eval could lead to strange errors for replication
on a customer system where an SSH connection could not always be
established, because the target's API version would fall back to 1.
Because of that, the '-base' argument would be missing for the import
call on the target which would in turn lead to an error about the
target ZFS volume already existing (rather than doing an incremental
sync).
Reported-by: Maximiliano Sandoval <m.sandoval@proxmox.com> Signed-off-by: Fiona Ebner <f.ebner@proxmox.com> Reviewed-by: Max Carrara <m.carrara@proxmox.com>
plugin: move definition for 'port' option to base plugin
Commit 7020491 ("esxi: add 'port' config parameter") started using
the 'port' option in a second plugin, but the definition stayed in the
PBS plugin. Avoid the hidden dependency and move the definition to the
base plugin instead.
It is necessary to mark it as optional or it would be required always.
Clarify that the option is not used by NFS and CIFS.
This diverts stderr of the fuse process to a pipe, which makes no
sense as it runs daemonized in a scope, also, the pipe fd was used as
a ready-signal, which now does not trigger anymore.
Dominik Csapak [Fri, 10 May 2024 13:56:58 +0000 (15:56 +0200)]
esxi: improve error handling for fuse mount tool
if the fuse tool encounters an error early, it prints it like:
Error: some error message
on stderr.
Redirect STDERR of the child process (which mounts the ESXi instance) to
the pipe of the parent (API) process, so that it can pass a hopefully
more meaningful message to the user than just an erroneous return code.
This prevents importing from vmdks with whitespaces in file names.
Further, some operations that include file sizes (like listing disks)
would potentially fail entirely if a custom disk with a badly name
backing device exists in a VM images directory since they don't expect
this. Specifically, since we don't necessarily know the actual naming
scheme of the current storage in the plain Plugin.pm version, we don't
check the full name anyway, so why bother with whitespaces...
See-also: https://forum.proxmox.com/threads/new-import-wizard-available-for-migrating-vmware-esxi-based-virtual-machines.144023/page-16#post-658697 Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Mira Limbeck [Wed, 17 Apr 2024 09:48:57 +0000 (11:48 +0200)]
fix insecure migration failing if waiting on lock
both STDOUT and STDERR are written into `$info` which is then parsed for
IP and port of the target socket listening.
when the ports file can't be locked immediately `trying to acquire
lock...` is printed on STDERR and in turn written into `$info`.
trying to parse the IP then fails, resulting in a migration or
replication failing.
the bare open3 call is replaced by the run_command wrapper from
pve-common to use a safe wrapper around open3 with the same
functionality.
STDERR is read separatey from STDOUT and the last line of STDERR is
kept in case of errors.
Fixes: 57acd6a ("fix #1452: also log stderr of remote command with
insecure storage migration")
Signed-off-by: Mira Limbeck <m.limbeck@proxmox.com>
Max Carrara [Tue, 2 Apr 2024 14:55:20 +0000 (16:55 +0200)]
cephconfig: align our parser with Ceph's parser
This commit rewrites the entire parser for ceph.conf, aligning its
behaviour as closely as possible with Ceph's parser grammar [0].
The most notable improvements are as follows:
1. The characters '#' and ';' now both mark comments, instead of
just the '#' character.
2. Any character, including comment literals ('#' and ';'), may now
be escaped.
3. Quoted values (single and double) are now supported.
4. Line continuations are now supported (lines ending with '\').
5. Repeated whitespace characters in keys are now treated as a
single space character.
6. Dashes '-' are not treated the same as spaces and underscores
anymore, as Ceph's grammar doesn't treat them that way.
* Paired with 5., this means that repeated whitespace is now
equivalent to a single underscore.
7. Escaped comment literals are now un-escaped.
8. Although not too crucial, the parser now also supports empty
sections and will just initialize them with an empty hash.
Furthermore, the original grammar's more quirky behaviours are also
respected where sanely possible.
This commit changes the code style of subroutine `write_ceph_config`
to match our style guide [0] more.
Furthermore, the repeated calls to the inner subroutine are replaced
with a loop, while the regular expressions used by the inner `sub` are
now quoted with `qr` to prevent any accidental mis-quotings in the
future.
Instead of just using it as a warning and then trying to parse an
empty string as json.
For example, trying to parse unsupported vmdks, previously we'd see
something like this:
qemu-img: Could not open
'/run/pve/import/esxi/foo/mnt/ha-datacenter/vsanDatastore/asdf/asdf-000001.vmdk':
Unsupported image type 'vsanSparse'
could not parse qemu-img info command output for
'/run/pve/import/esxi/foo/mnt/ha-datacenter/vsanDatastore/asdf/asdf-000001.vmdk'
- malformed JSON string, neither tag, array, object, number, string
or atom, at character offset 0 (before "(end of string)") at
src/PVE/Storage/Plugin.pm line 962, <DATA> line 960.
Now it simply shows:
qemu-img: Could not open
'/run/pve/import/esxi/foo/mnt/ha-datacenter/vsanDatastore/asdf/asdf-000001.vmdk':
Unsupported image type 'vsanSparse'
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Thomas Lamprecht [Wed, 27 Mar 2024 12:11:22 +0000 (13:11 +0100)]
esxi: reduce cache invalidation time to 30s
Reduce the time the cache stays valid from 60s to 30s, while this
could double the amount of requests in the worst case, it's still not
that frequent and also halves the maximal time a user has to wait to
see changes on the ESXi side to appear here.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Gabriel Goller [Thu, 21 Mar 2024 09:07:52 +0000 (10:07 +0100)]
esxi: detect correct os type in 'other' family
This patch introduces the conversion table for all possible OS Types
that are in the VMWare 'other' family and sets the pve counterpart.
Our default OS Type is 'linux', so including mappings to 'other' makes
sense.
Signed-off-by: Gabriel Goller <g.goller@proxmox.com>
following pve-esxi-import-tools's commits: 3ee5c3b ("esxi-folder-fuse: add --insecure option") c292c67 ("listvms.py: add --insecure parameter, verify cert by
default") 34c87be ("rename --insecure option to --skip-cert-verification")
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
[ TL: rename 'insecure' to 'skip-cert-verification' to better convey
what it means ] Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
[ TL: fix wrong comparison with >= and avoid undef warning if file
does not yet exist at all ] Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
[ TL: squash removal of both subs into one commit ] Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Thomas Lamprecht [Sun, 10 Mar 2024 18:25:34 +0000 (19:25 +0100)]
api: import-metadata: make warnings structured & merge ignored-volumes
This allows the frontends to translate them and avoids somewhat
duplicated info by having some warnings explicitly (ignored-volumes)
while others are in the warnings array.
By passing along the key and the value the frontend can also show the
warnings in-line, e.g. by marking a disk-entry in a grid as having
potential problems.
Ideally we'd have a central list of known types used for the API
return schema enum and to check when calling the $warn closure, but as
we only got three warnings keep this as is and only add a comment.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
projects / pve-storage.git / log