datastore: factor type out of ListGroups into ListGroupsType
In the API we want to iterate over all backup groups
belonging to a particular type at least once, and iterating
through *everything* and simply "skipping" over every single
entry from another type makes no sense given that the groups
are organized into subdirectories based on their type.
Let's have an `.iter_backup_type()` method which returns an
iterator over all the groups of a specific type named
ListGroupsType and factorize the type level iterator out of
ListGroups for reuse.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Dominik Csapak [Mon, 20 Jun 2022 07:51:13 +0000 (09:51 +0200)]
restore-daemon: make file listing 'streaming'
this prevents an oom kill when listing large directories.
Without this, i'd get an oom kill in the restore vm when
i tried to list a directory with ~60000 entries, but with this,
i'd get the response for even 250000 entries
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com> Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Stefan Sterz [Fri, 10 Jun 2022 08:13:25 +0000 (10:13 +0200)]
config: remove duplicate privilege lookup in cached_user_info
`lookup_privs` just uses `lookup_privs_details` but ignores the
propagated privileges it returns. thus, the lookup here is redundant
as it is immediately followed by a call to `lookup_privs_details` with
the same parameters.
The previous implementation had one issue with not handling API
tokens correctly.
In general, AclTree(Node) operates on the role level, not the priv
level - the latter is handled by cached_user_info.rs
Accordingly, the ACL tree helpers now return a list of paths where *any*
role is defined for the given AuthId, and any_priv_below then maps those
paths to privs via the regular helpers for priv lookup/checking. this
approach should also be robust if groups and group ACLs are ever
introduced.
api: tape restore: code cleanup to reduce indentation level
No semantic change intended. IMO the interface of "both a datastore
and NS mapping must be present" is still a bit weird, at least in how
its used here to decide what to skip and what not, maybe we can
implement this in a more clear way (or maybe I'm just overlooking
something that makes it clearer as is).
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Dominik Csapak [Thu, 2 Jun 2022 12:18:04 +0000 (14:18 +0200)]
replace 'disk_usage' with 'fs_info' from proxmox-sys
Use the moved 'fs_info' helpers from the proxmox-sys crate (available
from there since proxmox-sys 0.3.0) as replacement for 'disk_usage'
in the workspace local tools crate and remove the latter as we do not
need it anymore.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
[ T: squashed in removal of now unused import and reworded commit
message to include version availability info, among other things ] Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
api: list datastore: avoid iterating over NS for priv check, use AclTree
Make the assumption that if a user has any privilege that would make
an NS and (parts) of its content visible they also should be able to
know about the datastore and very basic errors on lookup (path
existence and maintenance mode) even if that NS doesn't even exists
(yet), as they could, e.g., make or view a backup and find out
anyway.
This avoids iterating over parts of the whole datastore folder tree
on disk, doing a priv check on each, swapping IO to virtual in memory
checks on info we got available already anyway, is always a good idea
after all
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
config: any_priv_below: plural name & switch to slice of &str for path
s/any_priv_below/any_privs_below/ for consistency and switch from a
single &str for the path param to the slice-ref string variant, as
that allows to use it more often without allocation.
Also allow passing the whole path as single &str element in the slice
by splitting each component on '/' like we do in other parts
nowadays. Note though that we need to omit the leading slash then.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Stefan Sterz [Fri, 3 Jun 2022 15:32:24 +0000 (17:32 +0200)]
pbs-config: acl-tree: add any_priv_below
`any_priv_below()` checks if a given AuthId has any given privileges
on a sub-tree of the AclTree. to do so, it first takes into account
propagating privileges on the path itself and then uses a depth-first
search to check if any of the provided privileges are set on any
node of the sub-tree pointed to by the path.
Signed-off-by: Stefan Sterz <s.sterz@proxmox.com> Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
datastore: swap ConfigVersionCache with digest for change detection
We got the digest available anyway, and it's only 16 bytes more to
save (compared to last_generation and the recently removed last_time,
both being 64 bit = 8 bytes each)
Side benefit, we detect config changes made manually (e.g., `vim
datacenter.cfg`) immediately.
Note that we could restructure the maintenance mode checking to only
be done after checking if there's a cached datastore, in which case
using the generation could make sense to decide if we need to re-load
it again before blindly loading the config anyway. As that's not only
some (not exactly hard but not really trivial like a typo fix either)
restructuring work but also means we'd lose the "detect manual
changes" again I'd rather keep using the digest.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
to avoid the problematic open fresh datastore with fresh chunkstore
with, and that's the actual problematic part, fresh process locker.
As the latter uses posix record locks which are pretty dangreous as
they operate on a path level (not FD level) and thus closing any file
opened (even if it wasn't opened for locking at all) drops all active
locks on the same file on completely unrelated file descriptors -.-
Also, no operation wasn't exactly correct for this thing in the first
place, but we cannot use Operation::Lookup either, as we're currently
indeed using a rather stupid-simple way and *are* reading.
So until we optimize this to allow querying the AclTree if there's
any priv XYZ below a path, use the Operation::Read.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Dominik Csapak [Thu, 2 Jun 2022 14:27:44 +0000 (16:27 +0200)]
datastore: lookup: reuse ChunkStore on stale datastore re-open
When re-opening a datastore due to the cached entry being stale
(config change) but also if the last re-open was >60s ago). On
datastore open the chunk store was also re-opened, which in turn
creates a new ProcessLocker, loosing any existing shared lock which
can cause conflicts between long running (24h+) backups and GC.
To fix this, reuse the existing ChunkStore, and thus its
ProcessLocker, when creating a up-to-date datastore instance on
lookup, since only the datastore config should be reloaded. This is
fine as the ChunkStore path is not updatable over our API.
This was always a potential issue but got exposed in practice by
commit 118deb4db8e709b02704bc66c0551bfa7e4369ed which introduced the
unconditional "re-open after 60s" mechanism.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
[ T: reword commit message a bit and reference commit that made the
issue much more likely ] Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
along with the rest of tokio/futures/hyper/openssl being updated - this
is the only one we explicitly depend on that had a non-compatible
version number.
projects / proxmox-backup.git / log