]> git.proxmox.com Git - proxmox-backup.git/log
proxmox-backup.git
2 years agoapi: backup create: enforce that namespace exists
Thomas Lamprecht [Thu, 5 May 2022 17:26:20 +0000 (19:26 +0200)]
api: backup create: enforce that namespace exists

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2 years agoapi: backup create: make permission check namespace aware
Thomas Lamprecht [Thu, 5 May 2022 17:26:04 +0000 (19:26 +0200)]
api: backup create: make permission check namespace aware

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2 years agoapi: datastore: make permission checks namespace aware
Thomas Lamprecht [Thu, 5 May 2022 17:07:28 +0000 (19:07 +0200)]
api: datastore: make permission checks namespace aware

We probably can combine the base permission + owner check, but for
now add explicit ones to upfront so that the change is simpler as
only one thing is done.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2 years agoapi: namespace management endpoints
Thomas Lamprecht [Sun, 24 Apr 2022 18:24:42 +0000 (20:24 +0200)]
api: namespace management endpoints

allow to list any namespace with privileges on it and allow to create
and delete namespaces if the user has modify permissions on the parent
namespace.

Creation is only allowed if the parent NS already exists.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2 years agodatastore: add max-depth to recursive namespace iter
Thomas Lamprecht [Thu, 5 May 2022 15:17:02 +0000 (17:17 +0200)]
datastore: add max-depth to recursive namespace iter

on depth == 0 we only yield the anchor ns, this simplifies usage in
the API as for, e.g. list-snapthos the depth == 0 means only the
snapshots from the passed namespace, nothing below.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2 years agoapi: add NS_MAX_DEPTH_SCHEMA
Fabian Grünbichler [Fri, 29 Apr 2022 11:48:54 +0000 (13:48 +0200)]
api: add NS_MAX_DEPTH_SCHEMA

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
 [ T: renamed from NAMESPACE_RECURSION_DEPTH_SCHEMA & moved to from
   jobs to datastore ]
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2 years agoapi: add prefix-mapping helper to BackupNamespace
Fabian Grünbichler [Fri, 29 Apr 2022 11:43:58 +0000 (13:43 +0200)]
api: add prefix-mapping helper to BackupNamespace

given a namespace, a source prefix and a target prefix this helper
strips the source prefix and replaces it with the target one (erroring
out if the prefix doesn't match).

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2 years agoapi: derive UpdaterType for BackupNamespace
Fabian Grünbichler [Fri, 29 Apr 2022 11:42:58 +0000 (13:42 +0200)]
api: derive UpdaterType for BackupNamespace

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2 years agoconfig: acl tree: allow path components to be paths too
Thomas Lamprecht [Wed, 4 May 2022 16:11:43 +0000 (18:11 +0200)]
config: acl tree: allow path components to be paths too

will be used for namespaces

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2 years agodatastore: add create_namespace
Thomas Lamprecht [Tue, 26 Apr 2022 17:00:19 +0000 (19:00 +0200)]
datastore: add create_namespace

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2 years agoBackupNamespace: fix deserialize of root NS
Fabian Grünbichler [Fri, 29 Apr 2022 11:25:13 +0000 (13:25 +0200)]
BackupNamespace: fix deserialize of root NS

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2 years agoapi-types: fixup backup-ns being optional
Wolfgang Bumiller [Fri, 29 Apr 2022 10:11:32 +0000 (12:11 +0200)]
api-types: fixup backup-ns being optional

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2 years agopbs-client: don't include empty backup-ns in requests
Wolfgang Bumiller [Wed, 27 Apr 2022 12:55:58 +0000 (14:55 +0200)]
pbs-client: don't include empty backup-ns in requests

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2 years agoclient: add --ns parameter to snapshot list
Wolfgang Bumiller [Wed, 27 Apr 2022 12:35:36 +0000 (14:35 +0200)]
client: add --ns parameter to snapshot list

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2 years agoapi types: BackupNamespace add pop & parent helpers
Thomas Lamprecht [Thu, 28 Apr 2022 07:37:10 +0000 (09:37 +0200)]
api types: BackupNamespace add pop & parent helpers

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2 years agoapi-types: more regex fixups
Wolfgang Bumiller [Mon, 25 Apr 2022 09:52:12 +0000 (11:52 +0200)]
api-types: more regex fixups

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2 years agoapi-types: add missing slash in optional ns path regex
Wolfgang Bumiller [Mon, 25 Apr 2022 09:33:12 +0000 (11:33 +0200)]
api-types: add missing slash in optional ns path regex

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2 years agoclient: more backup namespace support
Wolfgang Bumiller [Mon, 25 Apr 2022 09:23:15 +0000 (11:23 +0200)]
client: more backup namespace support

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2 years agodatastore: include namespace in full_path
Wolfgang Bumiller [Mon, 25 Apr 2022 08:58:46 +0000 (10:58 +0200)]
datastore: include namespace in full_path

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2 years agoapi: add remaining missing backup-ns parameters
Wolfgang Bumiller [Mon, 25 Apr 2022 08:34:53 +0000 (10:34 +0200)]
api: add remaining missing backup-ns parameters

these are the ones for non-#[api] methods, also fill in the
namespace in prune operations

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2 years agodatastore: add helpers to destroy whole namespaces
Thomas Lamprecht [Mon, 25 Apr 2022 09:37:57 +0000 (11:37 +0200)]
datastore: add helpers to destroy whole namespaces

The behavior on "any snapshot was protected" isn't yet ideal, as we
then do not cleanup any (sub) namespace, even if some of them where
cleaned from groups & snapshots completely. But that isn't easy to do
with our current depth-first pre-order iterator behavior, and it's
also not completely wrong either, the user can re-do the removal on
the sub-namespaces, so leave that for later.

Should get moved to a datastore::BackupNamespace type once/if we get
one

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2 years agodatastore: add single-level and recursive namespace iterators
Thomas Lamprecht [Sun, 24 Apr 2022 18:23:30 +0000 (20:23 +0200)]
datastore: add single-level and recursive namespace iterators

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2 years agoapi types: namespace: add from_parent_ns helper
Thomas Lamprecht [Sun, 24 Apr 2022 18:21:57 +0000 (20:21 +0200)]
api types: namespace: add from_parent_ns helper

will be used in the (recursive) namespace iterator

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2 years agoapi types: namespace: include problematic component in error
Thomas Lamprecht [Sun, 24 Apr 2022 18:21:45 +0000 (20:21 +0200)]
api types: namespace: include problematic component in error

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2 years agodatastore: add backup_ns accessor
Wolfgang Bumiller [Mon, 25 Apr 2022 08:42:23 +0000 (10:42 +0200)]
datastore: add backup_ns accessor

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2 years agoapi-types: add namespace to BackupGroup
Wolfgang Bumiller [Thu, 21 Apr 2022 13:04:59 +0000 (15:04 +0200)]
api-types: add namespace to BackupGroup

Make it easier by adding an helper accepting either group or
directory

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2 years agons: max depth: set constant to upper inclusive boundary
Thomas Lamprecht [Thu, 5 May 2022 15:15:31 +0000 (17:15 +0200)]
ns: max depth: set constant to upper inclusive boundary

makes usage a bit simpler, e.g., the api maximum can use that 1:1
then.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2 years agoapi-types: add BackupNamespace type
Thomas Lamprecht [Wed, 7 Apr 2021 10:28:50 +0000 (12:28 +0200)]
api-types: add BackupNamespace type

The idea is to have namespaces in a datastore to allow grouping and
namespacing backups from different (but similar trusted) sources,
e.g., different PVE clusters, geo sites, use-cases or company
service-branches, without separating the underlying
deduplication domain and thus blowing up data and (GC/verify)
resource usage.

To avoid namespace ID clashes with anything existing or future
usecases use a intermediate `ns` level on *each* depth.

The current implementation treats that as internal and thus hides
that fact from the API, iow., the namespace path the users passes
along or gets returned won't include the `ns` level, they do not
matter there at all.

The max-depth of 8 is chosen with the following in mind:
- assume that end-users already are in a deeper level of a hierarchy,
  most often they'll start at level one or two, as the higher ones
  are used by the seller/admin to namespace different users/groups,
  so lower than four would be very limiting for a lot of target use
  cases

- all the more, a PBS could be used as huge second level archive in a
  big company, so one could imagine a namespace structure like:
  /<state>/<intra-state-location>/<datacenter>/<company-branch>/<workload-type>/<service-type>/
  e.g.: /us/east-coast/dc12345/financial/report-storage/cassandra/
  that's six levels that one can imagine for a reasonable use-case,
  leave some room for the ones harder to imagine ;-)

- on the other hand, we do not want to allow unlimited levels as we
  have request parameter limits and deep nesting can create other
  issues as well (e.g., stack exhaustion), so doubling the minimum
  level of 4 (1st point) we got room to breath even for the
  more odd (or huge) use cases (2nd point)

- a per-level length of 32 (-1 due to separator) is enough to use
  telling names, making lives of users and admin simpler, but not
  blowing up parameter total length with the max depth of 8

- 8 * 32 = 256 which is nice buffer size

Much thanks for Wolfgang for all the great work on the type
implementation and assisting greatly with the design.

Co-authored-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Co-authored-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2 years agoapi types: BackupType: add iter for enum
Thomas Lamprecht [Tue, 3 May 2022 10:39:47 +0000 (12:39 +0200)]
api types: BackupType: add iter for enum

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2 years agocompletion: fix 'group-filter' parameter name
Fabian Grünbichler [Fri, 6 May 2022 08:27:00 +0000 (10:27 +0200)]
completion: fix 'group-filter' parameter name

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2 years agoproxmox-backup-manager: add limit to pull
Fabian Grünbichler [Fri, 6 May 2022 08:25:31 +0000 (10:25 +0200)]
proxmox-backup-manager: add limit to pull

seems to have been forgotten initially.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2 years agodatastore: chunk store: leverage new format str variable reference
Thomas Lamprecht [Tue, 10 May 2022 07:39:17 +0000 (09:39 +0200)]
datastore: chunk store: leverage new format str variable reference

makes it often compact enough for rustfmt to move it into a single
line

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2 years agochunk_store: insert_chunk: write chunk again if it is empty on disk
Dominik Csapak [Mon, 9 May 2022 14:09:06 +0000 (16:09 +0200)]
chunk_store: insert_chunk: write chunk again if it is empty on disk

and issue a warning. We can do this, because we know an empty chunk
cannot be valid, and we (assumedly) have a valid chunk in memory.

Having empty chunks on disk is currently possible when PBS crashes,
but the rename of the chunk was flushed to disk, when the actual data
was not.

If it's not empty but there is a size mismatch, return an error.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2 years agoapi: tape/restore: skip snapshot if owner check failed
Dominik Csapak [Mon, 9 May 2022 10:41:19 +0000 (12:41 +0200)]
api: tape/restore: skip snapshot if owner check failed

instead of aborting the whole restore

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2 years agoapi: tape/restore: fix wrong datastore locking
Dominik Csapak [Mon, 9 May 2022 10:41:18 +0000 (12:41 +0200)]
api: tape/restore: fix wrong datastore locking

used_datastores returned the 'target', but in the full_restore_worker,
we interpreted it as the source and searched for a mapping
(which we then locked)

since we cannot return a HashSet of Arc<T> (missing Hash trait on DataStore),
we have now a map of source -> target

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2 years agodrop mut on some http client usages
Thomas Lamprecht [Thu, 5 May 2022 08:49:03 +0000 (10:49 +0200)]
drop mut on some http client usages

thanks to commit 70142e607dda43fc778f39d52dc7bb3bba088cd3 from
proxmox repos's proxmox-http crate

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2 years agoupdate proxmox-http b-d to 0.6.1
Thomas Lamprecht [Thu, 5 May 2022 08:50:23 +0000 (10:50 +0200)]
update proxmox-http b-d to 0.6.1

so that we can drop some mut on http client usages

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2 years agorouter change made one level of rpcenv mut superfluous
Thomas Lamprecht [Tue, 3 May 2022 07:43:04 +0000 (09:43 +0200)]
router change made one level of rpcenv mut superfluous

Created via `cargo fix`.  see commit
47acc8dc8f68ed2c5db69b1678b479e05b0a3194 from proxmox-rs

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2 years agoupdate proxmox-router b-d to 1.2.2
Thomas Lamprecht [Thu, 5 May 2022 07:54:18 +0000 (09:54 +0200)]
update proxmox-router b-d to 1.2.2

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2 years agobump version to 2.1.8-1
Thomas Lamprecht [Mon, 2 May 2022 15:36:17 +0000 (17:36 +0200)]
bump version to 2.1.8-1

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2 years agopull: add some comments
Fabian Grünbichler [Fri, 29 Apr 2022 09:17:27 +0000 (11:17 +0200)]
pull: add some comments

and remove already fixed fixmes.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
 [ T: squash in cargo fmt fixup for some trailing ws ]
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2 years agopull: remove unnecessary `pub` visibility
Fabian Grünbichler [Fri, 29 Apr 2022 09:17:26 +0000 (11:17 +0200)]
pull: remove unnecessary `pub` visibility

pull_store is the entrypoint used by other code, the rest does not need
to be visible at all.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2 years agopull: filter local removal candidates by owner
Fabian Grünbichler [Fri, 29 Apr 2022 09:17:25 +0000 (11:17 +0200)]
pull: filter local removal candidates by owner

else this might remove groups which are not part of the pull scope. note
that setting/using remove_vanished already checks the required privs
earlier.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2 years agoproxmox-backup-proxy: stop accept() loop on daemon shutdown
Dominik Csapak [Fri, 29 Apr 2022 10:04:52 +0000 (12:04 +0200)]
proxmox-backup-proxy: stop accept() loop on daemon shutdown

On reload the old process hands over to the new process but needs to
keep running until all its worker tasks are finished to avoid
breaking a in-progress action like a xterm.js web shell or a backup
creation/restore.

During that wait time the receiving channel was already closed, but
the TCP sockt accept listener was still left active by mistake.

That paired with the `SO_REUSEPORT` being set on the underlying
socket, made the kernel choose either the old or new process for new
incoming connections, both still listened for them after all and
reuse-port + multiple processes is often used as load-balancer
mechanism.

As the old proxy accepted connections but didn't process them anymore
one could observer sporadic connection failures on any API call, well
any new connection to the proxy, depending on which process got the
it assigned.

The fix is to stop accepting new connections one we shutdown, so poll
the shutdown_future too during accept and just exit the accept-loop
on shutdown.

Note: This part of the code, nor other parts that could influence it,
wasn't changed at all in recent times, so it's still unresolved for
why it pops up only now.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Co-authored-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 [ T: add more (root cause) info and reword a bit ]
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2 years agoapi: status: return gc-status again
Dominik Csapak [Fri, 29 Apr 2022 10:07:12 +0000 (12:07 +0200)]
api: status: return gc-status again

Returning the GC status was dropped by mistake in commit 762f7d15
("datastore status: factor out api type DataStoreStatusListItem")

As this is considered a breaking change which we also felt, due to
the gc-status being used in the web interface for the datastore
overview list (not the dashboard), re add it.

Fixes: 762f7d15
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
 [ T: add reference to breaking commit, reword message ]
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2 years agocargo fmt
Thomas Lamprecht [Thu, 28 Apr 2022 08:26:00 +0000 (10:26 +0200)]
cargo fmt

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2 years agobump version to 2.1.7-1
Thomas Lamprecht [Wed, 27 Apr 2022 17:54:27 +0000 (19:54 +0200)]
bump version to 2.1.7-1

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2 years agoui: add tooltip to datastore in maintenance mode
Hannes Laimer [Tue, 26 Apr 2022 06:23:35 +0000 (06:23 +0000)]
ui: add tooltip to datastore in maintenance mode

Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
2 years agoui: utils: add function for parsing maintenance mode
Hannes Laimer [Tue, 26 Apr 2022 06:23:34 +0000 (06:23 +0000)]
ui: utils: add function for parsing maintenance mode

...since the same code is used is more than one place

Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
2 years agoui: update datastore list more often
Hannes Laimer [Tue, 26 Apr 2022 06:23:33 +0000 (06:23 +0000)]
ui: update datastore list more often

Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
2 years agoui: update icon in datastore list when in maintenance mode
Hannes Laimer [Tue, 26 Apr 2022 06:23:32 +0000 (06:23 +0000)]
ui: update icon in datastore list when in maintenance mode

Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
2 years agoapi2: DataStoreListItem add maintenance info
Hannes Laimer [Tue, 26 Apr 2022 06:23:31 +0000 (06:23 +0000)]
api2: DataStoreListItem add maintenance info

Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
2 years agoui: add summary mask when in maintenance mode
Hannes Laimer [Tue, 26 Apr 2022 06:23:30 +0000 (06:23 +0000)]
ui: add summary mask when in maintenance mode

Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
2 years agofile-restore: add 'timeout' and 'json-error' parameter
Dominik Csapak [Tue, 26 Apr 2022 10:13:58 +0000 (12:13 +0200)]
file-restore: add 'timeout' and 'json-error' parameter

timeout limits the code with the given timeout in seconds, and
'json-error' return json to stdout when the call returns an error like
this:

{
    "msg": "error message",
    "error": true,
    "code": <HTTP_STATUS_CODE>, // if it was an http error
}

with both options set, a client can more easily determine if the call
ran into a timeout (since it will return a 503 error), and can poll
it again

both is done behind new parameters, so that we can stay backwards-compatible

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2 years agofile-restore: factor out 'list_files'
Dominik Csapak [Tue, 26 Apr 2022 10:13:57 +0000 (12:13 +0200)]
file-restore: factor out 'list_files'

we'll want to reuse that in a later patch

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2 years agorestore-daemon: avoid auto-mounting zpools
Dominik Csapak [Tue, 26 Apr 2022 10:13:56 +0000 (12:13 +0200)]
restore-daemon: avoid auto-mounting zpools

the duration of mounting zpools not only correspond to the number of disks,
but also to the content (many subvols for example) which we cannot know
beforehand. so avoid mounting them at the start, and mount it only when
the user requests a listing/extraction with the zpool in path

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2 years agorestore-daemon: put blocking code into 'block_in_place'
Dominik Csapak [Tue, 26 Apr 2022 10:13:55 +0000 (12:13 +0200)]
restore-daemon: put blocking code into 'block_in_place'

DISK_STATE.lock() and '.resolve()' can both block since they access
the disks. Putting them into a 'block_in_place' makes tokio move it
out in its own thread to avoid that the executor isn't able to
progress any other futures in the mean time.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2 years agorestore-daemon: start disk initialization in parallel to the api
Dominik Csapak [Tue, 26 Apr 2022 10:13:54 +0000 (12:13 +0200)]
restore-daemon: start disk initialization in parallel to the api

this way, the vm can start up faster, and the actual disk init happens
in parallel. this avoids unnecessary timeouts when starting the vm

if the call panics, we still abort the vm with an error

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2 years agoapi types: datastore status: reword doc comment of estimated_full_date
Thomas Lamprecht [Mon, 25 Apr 2022 09:48:14 +0000 (11:48 +0200)]
api types: datastore status: reword doc comment of estimated_full_date

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2 years agoui: update generated OnlineHelpInfo map
Thomas Lamprecht [Mon, 25 Apr 2022 08:17:21 +0000 (10:17 +0200)]
ui: update generated OnlineHelpInfo map

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2 years agofix #3067: ui: add a separate notes view for longer markdown notes
Stefan Sterz [Tue, 12 Apr 2022 10:34:23 +0000 (12:34 +0200)]
fix #3067: ui: add a separate notes view for longer markdown notes

since markdown notes might be rather long, this commit adds a tab
similar to pve's datacenter or node notes. requires a bump of the
widget toolkit in order to use the `pmxNotesView`.

Signed-off-by: Stefan Sterz <s.sterz@proxmox.com>
2 years agofix #3067: docs: add markdown primer from pve to pbs
Stefan Sterz [Tue, 12 Apr 2022 10:34:19 +0000 (12:34 +0200)]
fix #3067: docs: add markdown primer from pve to pbs

this copies the markdown primer from the pve docs to allow access to
it via the help buttons in the gui

Signed-off-by: Stefan Sterz <s.sterz@proxmox.com>
2 years agodatastore: move blob loading into BackupDir impl and adapt call sites
Thomas Lamprecht [Sun, 24 Apr 2022 17:09:38 +0000 (19:09 +0200)]
datastore: move blob loading into BackupDir impl and adapt call sites

data blobs can only appear in a BackupDir (snapshot) in the backup
hierachy, so makes more sense that it lives in there.

As it wasn't widely used anyway it's easy to move the single
non-package call site over to the new one directly and drop the
implementation from Datastore completely.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2 years agodatastore: move destroying group or dir into respective impl
Thomas Lamprecht [Sun, 24 Apr 2022 16:49:09 +0000 (18:49 +0200)]
datastore: move destroying group or dir into respective impl

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2 years agodatastore: move manifest locking into BackupDir impl
Thomas Lamprecht [Sun, 24 Apr 2022 16:37:15 +0000 (18:37 +0200)]
datastore: move manifest locking into BackupDir impl

the manifest is owned by the backup dir (snapshot) so it should also
handle locking, makes no sense to have the implementation somewhere
higher up.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2 years agodatastore: s/fail_if_not_exist/assert_exists/
Thomas Lamprecht [Sun, 24 Apr 2022 17:50:51 +0000 (19:50 +0200)]
datastore: s/fail_if_not_exist/assert_exists/

avoid putting whole sentences in parameter names

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2 years agodatastore: improve backup group/snapshot iters
Thomas Lamprecht [Sun, 24 Apr 2022 16:06:17 +0000 (18:06 +0200)]
datastore: improve backup group/snapshot iters

move the check for directory before doing the OSString -> String
conversion, which should be a bit more efficient.

Also let the match return the entry in the non-skip/return case to
reduce indentation level for the inner "yield element" part, making
it slightly easier to follow.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2 years agodatastore: replace manual path assembly by group/dir full_path
Thomas Lamprecht [Sun, 24 Apr 2022 16:03:27 +0000 (18:03 +0200)]
datastore: replace manual path assembly by group/dir full_path

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2 years agodatastore: list snapshots iter: report group dir in error
Thomas Lamprecht [Sun, 24 Apr 2022 15:58:12 +0000 (17:58 +0200)]
datastore: list snapshots iter: report group dir in error

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2 years agopbs-client: extract: add top-level dir in tar.zst
Dominik Csapak [Tue, 19 Apr 2022 10:28:28 +0000 (12:28 +0200)]
pbs-client: extract: add top-level dir in tar.zst

when download a folder, include that folder as first entry (except '/')

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2 years agopbs-client: extract: rewrite create_zip with sequential decoder
Dominik Csapak [Tue, 19 Apr 2022 10:28:27 +0000 (12:28 +0200)]
pbs-client: extract: rewrite create_zip with sequential decoder

instead of an async recursive function. Not only is it less code,
recursive futures are not really nice and it should be faster too.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2 years agoserver pull: fix comment w.r.t. initial downloaded chunk capacity
Thomas Lamprecht [Thu, 21 Apr 2022 13:54:59 +0000 (15:54 +0200)]
server pull: fix comment w.r.t. initial downloaded chunk capacity

> The hash set will be able to hold at least capacity elements
> without reallocating. If capacity is 0, the hash set will not
> allocate.
-- rustdoc, HashSet::with_capacity

So, the number we pass is the amount of chunk "IDs" we safe, which is
then 64Ki, not 16Ki and thus the size we can reference too is also
256 GiB, not 64 GiB.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2 years agopbs-tape: sgutils2: check sense data when status is 'CHECK_CONDITION'
Dietmar Maurer [Thu, 21 Apr 2022 07:34:36 +0000 (09:34 +0200)]
pbs-tape: sgutils2: check sense data when status is 'CHECK_CONDITION'

Some raid controllers return a 'transport error' when we expected a
'sense error'. it seems the correct way to check the sense data is when
either the result category is 'SENSE' or when the status is 'CHECK_CONDITION',
so do that. (similar to how 'sg_raw' returns the errors)

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
2 years agoreference the datastore in BackupGroup/Dir
Wolfgang Bumiller [Wed, 20 Apr 2022 13:30:04 +0000 (15:30 +0200)]
reference the datastore in BackupGroup/Dir

And drop the base_path parameter on a first bunch of
functions (more reordering will follow).

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2 years agoapi-types: DataStoreConfig::new for testing
Wolfgang Bumiller [Wed, 20 Apr 2022 13:06:28 +0000 (15:06 +0200)]
api-types: DataStoreConfig::new for testing

so our examples can more easily access a datastore without
going over a configuration & cache

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2 years agoapi2: read_remote: also return RemoteWithoutPassword
Fabian Grünbichler [Wed, 20 Apr 2022 11:58:41 +0000 (13:58 +0200)]
api2: read_remote: also return RemoteWithoutPassword

like for the index, instead of manually stripping it.

this (and the previous change) is backwards-compatible since `Remote`
already skipped serializing empty strings, so the returned JSON is
identical.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2 years agodatastore: cleanup and document backup group/dir openers
Wolfgang Bumiller [Wed, 20 Apr 2022 11:24:57 +0000 (13:24 +0200)]
datastore: cleanup and document backup group/dir openers

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2 years agodatastore: drop Hash from BackupGroup
Wolfgang Bumiller [Wed, 20 Apr 2022 11:08:43 +0000 (13:08 +0200)]
datastore: drop Hash from BackupGroup

same as for Eq/Ord/...

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2 years agodatastore: drop PartialEq and PartialOrd from BackupGroup
Wolfgang Bumiller [Wed, 20 Apr 2022 10:23:13 +0000 (12:23 +0200)]
datastore: drop PartialEq and PartialOrd from BackupGroup

Same as previous commits: this will be linked to a
particular DataStore and Eq/Ord is now only part of the
api types, for now.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2 years agodatastore: drop Eq and PartialEq from BackupDir
Wolfgang Bumiller [Wed, 20 Apr 2022 10:21:50 +0000 (12:21 +0200)]
datastore: drop Eq and PartialEq from BackupDir

Same as previous commit: this is supposed to be connected to
a datastore and Eq/PartialEq only make sense for the
api-type part.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2 years agodatastore: drop Ord from BackupGroup
Wolfgang Bumiller [Wed, 20 Apr 2022 10:20:28 +0000 (12:20 +0200)]
datastore: drop Ord from BackupGroup

This one is supposed to be linked to a datastore instance,
so it won't be Ord for now.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2 years agomake datastore BackupGroup/Dir ctors private
Wolfgang Bumiller [Tue, 19 Apr 2022 08:38:46 +0000 (10:38 +0200)]
make datastore BackupGroup/Dir ctors private

And use the api-types for their contents.

These are supposed to be instances for a datastore, the pure
specifications are the ones in pbs_api_types which should be
preferred in crates like clients which do not need to deal
with the datastore directly.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2 years agoapi-types: use BackupType for GroupFilter::BackupType
Wolfgang Bumiller [Wed, 20 Apr 2022 09:45:53 +0000 (11:45 +0200)]
api-types: use BackupType for GroupFilter::BackupType

instead of a string

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2 years agorename BackupDir's group_path to relative_group_path
Wolfgang Bumiller [Tue, 19 Apr 2022 07:45:22 +0000 (09:45 +0200)]
rename BackupDir's group_path to relative_group_path

datastore's group_path will be moved to BackupDir soon and
this is required to be able to properly distinguish them

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2 years agoapi-types: datastore type improvements
Wolfgang Bumiller [Tue, 19 Apr 2022 10:11:16 +0000 (12:11 +0200)]
api-types: datastore type improvements

let BackupGroup implement Hash

let BackupGroup and BackupDir be AsRef<BackupGroup>
let BackupDir be AsRef<BackupDir>

the pbs-datastore types will implement these AsRefs as well

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2 years agodatastore: remove unused list_files function
Wolfgang Bumiller [Wed, 20 Apr 2022 07:58:12 +0000 (09:58 +0200)]
datastore: remove unused list_files function

it also doesn't belong into this type

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2 years agodatastore: move last_backup from BackupInfo to BackupGroup
Wolfgang Bumiller [Wed, 20 Apr 2022 07:55:48 +0000 (09:55 +0200)]
datastore: move last_backup from BackupInfo to BackupGroup

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2 years agoAuthId: derive Ord and PartialOrd
Dietmar Maurer [Wed, 20 Apr 2022 07:58:15 +0000 (09:58 +0200)]
AuthId: derive Ord and PartialOrd

So the we can sort...

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
2 years agoRemoteWithoutPassword: new API type
Dietmar Maurer [Wed, 20 Apr 2022 07:40:05 +0000 (09:40 +0200)]
RemoteWithoutPassword: new API type

To make it explicit that we do not return the password.

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
2 years agoapi-types: introduce BackupType enum and Group/Dir api types
Wolfgang Bumiller [Thu, 14 Apr 2022 13:05:58 +0000 (15:05 +0200)]
api-types: introduce BackupType enum and Group/Dir api types

The type is a real enum.

All are API types and implement Display and FromStr. The
ordering is the same as it is in pbs-datastore.

Also, they are now flattened into a few structs instead of
being copied manually.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2 years agodatastore: add snapshot iterator and provide example
Thomas Lamprecht [Fri, 15 Apr 2022 10:24:56 +0000 (12:24 +0200)]
datastore: add snapshot iterator and provide example

will be more used in the future, when the upend-datastore master plan
comes in effect.

also a preparatory work for namespaces

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2 years agobackup: switch over to streaming Iterator improving memory usage
Thomas Lamprecht [Fri, 15 Apr 2022 10:20:51 +0000 (12:20 +0200)]
backup: switch over to streaming Iterator improving memory usage

Avoid collecting the whole group list in memory only to iterate and
filter over it again.

Note that the change could result in a indentation change, so best
viewed with `-w` flag.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2 years agodatastore: add helper to get a iterator for backup groups
Thomas Lamprecht [Fri, 15 Apr 2022 10:15:54 +0000 (12:15 +0200)]
datastore: add helper to get a iterator for backup groups

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2 years agodatastore: implement Iterator for backup group listing
Thomas Lamprecht [Fri, 15 Apr 2022 09:02:36 +0000 (11:02 +0200)]
datastore: implement Iterator for backup group listing

While currently it's still only used in a collected() way, most call
sites can be switched over to use the iterator directly, as often
they already convert the not-so-cheap, in-memory vector back in
.into_iter() anyway.

somewhat also preparatory (yak shaving) work for namespaces

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2 years agoapi datastore: some code cleanups
Thomas Lamprecht [Fri, 15 Apr 2022 08:59:57 +0000 (10:59 +0200)]
api datastore: some code cleanups

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2 years agodatastore: backup info: drop deprecated list_backup_groups
Thomas Lamprecht [Fri, 15 Apr 2022 08:53:57 +0000 (10:53 +0200)]
datastore: backup info: drop deprecated list_backup_groups

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2 years agoreplace deprecated list_backup_group from BackupInfo with Datastore one
Thomas Lamprecht [Fri, 15 Apr 2022 08:50:28 +0000 (10:50 +0200)]
replace deprecated list_backup_group from BackupInfo with Datastore one

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2 years agodatastore: move list_backup_groups into Datastore impl
Thomas Lamprecht [Fri, 15 Apr 2022 07:03:13 +0000 (09:03 +0200)]
datastore: move list_backup_groups into Datastore impl

Having that as static method in BackupInfo makes zero sense and just
complicates call sites, which need to extract the base_path from the
store manually upfront.

Mark old fn as deprecated so that we can do the move in a separate
step.

It's also planned to add an Iterator impl for this to allow more
efficient usage in the future.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2 years agoclient: rustfmt
Thomas Lamprecht [Thu, 14 Apr 2022 12:08:48 +0000 (14:08 +0200)]
client: rustfmt

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2 years agobackup client: rustfmt
Thomas Lamprecht [Thu, 14 Apr 2022 12:06:15 +0000 (14:06 +0200)]
backup client: rustfmt

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>