the snapshot string format is not backwards compatible since it now has
an in-line namespace prefix. it's possible to select which magic to use
at the start of the backup, since a tape backup job knows whether it
operates on non-root namespaces up-front.
the MediaCatalog itself also has a similar incompatible change, but
there
- updating existing catalogs in-place
- not knowing what the catalog will contain in the future when initially
creating/opening it
makes bumping the magic there harder. since the tape contents are
sufficiently guarded by the other two bumps, ignoring the
backwards-incomaptible change of the on-disk catalogs seems like an okay
tradeoff.
Dominik Csapak [Thu, 5 May 2022 13:59:36 +0000 (15:59 +0200)]
api: tape/restore: add namespace mapping
by adding a new parameter 'namespaces', which contains a mapping
for a namespace like this:
store=datastore,source=foo,target=bar,max-depth=2
if source or target are omitted the root namespace is used for its value
this mapping can be given several times (on the cli) or as an array (via
api) to have mappings for multiple datastores
if a specific snapshot list is given simultaneously, the given snapshots
will be restored according to this mapping, or to the source namespace
if no mapping was found.
to do this, we reutilize the restore_list_worker, but change it so that
it does not hold a lock for the duration of the restore, but fails
if the snapshot does exist at the end. also the snapshot will now
be temporarily restored into the target datastore into the
'.tmp/<media-set-uuid>' folder.
Dominik Csapak [Tue, 10 May 2022 14:09:42 +0000 (16:09 +0200)]
tape: fix snapshot path in catalog and snapshot_archive
both used the 'Display' trait of pbs_datastore::BackupDir, which is not
intended to be serialized anywhere. Instead, manually format the path
using the print_ns_and_snapshot helper, and conversely, parse with
'parse_ns_and_snapshot'. to be a bit safer, change the register_snapshot
signature to take a BackupNamespace and BackupDir instead of a string.
Dominik Csapak [Tue, 10 May 2022 14:06:41 +0000 (16:06 +0200)]
pbs-api-types: add parse and print ns_and_snapshot
these are helpers for the few cases where we want to print and parse
from a format that has the namespace and snapshot combined, like for
the on-tape catalog and snapshot archive.
Dominik Csapak [Tue, 3 May 2022 06:07:08 +0000 (08:07 +0200)]
tape: notify when arriving at end of media
when continuing a media set, we first move to the end of the tape and
start with the next (chunk) archive. If that takes long, the task logs
last line is 'moving to end of media' even if we already startet
writing. To make this less confusing, log that we arrived at the
end of the media.
to handle the unlikely case of `ns` being deeper than `remote-ns`,
`max-depth` being set to `None` and a too-deep sub-ns of `ns` existing.
such a sub-ns cannot have been created by a previous run of this sync
job, so avoid unexpectedly removing it.
into the regular one (with default == MAX) and the one used for
pull/sync, where the default is 'None' which actually means the remote
end reduces the scope of sync automatically (or, if needed,
backwards-compat mode without any remote namespaces at all).
pull/sync: detect remote lack of namespace support
and fall back to only syncing the root namespace, if possible. the sync
job will still be marked as failed to prompt the admin to resolve the
situation:
- explicitly mark the job as syncing *only* the root namespace
- or upgrade remote end to support namespaces
Thomas Lamprecht [Thu, 12 May 2022 13:40:55 +0000 (15:40 +0200)]
datastore: drop bogus chunk size check, can cause trouble
other sizes can happen in legitimate and illegitimate ways:
- illegitimate: encryped chunks and bad actor client
- legitimate: same chunk but newer zstd version (or compression
level) can compress it better (or worse) so the
Ideally we could take the actual smaller chunk so that improved zstd
tech gets leveraged, but we could only allow to do that for
un-encrypted ones.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Dominik Csapak [Thu, 12 May 2022 12:23:52 +0000 (14:23 +0200)]
ui: navigation tree: fix losing datastore selection on store load
instead of using 'replaceChild', simply set the appropriate
properties. When using the 'nodeUpdate' (protected function of extjs,
intended to be overwritten) instead of the private 'updateNode', it
will be called when the properties change
This way, the treenode stays the same and it can keep the selection
Thomas Lamprecht [Thu, 12 May 2022 10:55:37 +0000 (12:55 +0200)]
ui: content: fix various tree-checks from action handlers
they all still used some odd side effects of the tree structure to
decided what record type they operated on, just move them over to the
new `ty` record.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Fabian Ebner [Wed, 4 May 2022 11:33:24 +0000 (13:33 +0200)]
rest server: daemon: update PID file before sending MAINPID notification
There is a race upon reload, where it can happen that:
1. systemd forks off /bin/kill -HUP $MAINPID
2. Current instance forks off new one and notifies systemd with the
new MAINPID.
3. systemd sets new MAINPID.
4. systemd receives SIGCHLD for the kill process (which is the current
control process for the service) and reads the PID of the old
instance from the PID file, resetting MAINPID to the PID of the old
instance.
5. Old instance exits.
6. systemd receives SIGCHLD for the old instance, reads the PID of the
old instance from the PID file once more. systemd sees that the
MAINPID matches the child PID and considers the service exited.
7. systemd receivese notification from the new PID and is confused.
The service won't get active, because the notification wasn't
handled.
To fix it, update the PID file before sending the MAINPID
notification, similar to what a comment in systemd's
src/core/service.c suggests:
> /* Forking services may occasionally move to a new PID.
> * As long as they update the PID file before exiting the old
> * PID, they're fine. */
but for our Type=notify "before sending the notification" rather than
"before exiting", because otherwise, the mix-up in 4. could still
happen (although it might not actually be problematic without the
mix-up in 6., it still seems better to avoid).
Thomas Lamprecht [Thu, 12 May 2022 09:34:48 +0000 (11:34 +0200)]
GC scheduling: avoid triggering operation tracking error for upfront checks
without that one gets a "failed to lookup datastore X" in the log for
every datastore that is in read-only or offline maintenance mode,
even if they aren't scheduled for GC anyway.
Avoid that by first opening the datastore through a Lookup operation,
and only re-open it as Write op once we know that GC needs to get
scheduled for it.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Thomas Lamprecht [Thu, 12 May 2022 09:31:07 +0000 (11:31 +0200)]
datastore: add new Lookup for operations tracking
We sometimes need to do some in-memory only stuff, e.g., to check if
GC is already running for a datastore, which is a try_lock on a mutex
that is in-memory.
Actually the whole thing would be nicer if we could guarantee to hold
the correct contract statically, e.g., like
https://docs.rust-embedded.org/book/static-guarantees/design-contracts.html
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Stoiko Ivanov [Wed, 11 May 2022 14:06:32 +0000 (14:06 +0000)]
docs: add system-booting from pve-docs
and transform to reST.
semantic changes to the content are:
* s/{pve}/`Proxmox Backup`_/g
* changing footnotes to parenthesized notes (did not see footnote use in
the current docs)
* removed the comment about systems setup before the introduction of
p-b-t (which was introduced before pbs)
Stoiko Ivanov [Wed, 11 May 2022 14:06:31 +0000 (14:06 +0000)]
docs: sysadmin: adapt kernel-specifics for PBS
while all statements here are technically true - adding all
virtualization improvements is not relevant for proxmox backup in most
cases.
The intel nic driver seems like a left-over from a time (pre PVE 5.1)
where the pve-kernel included the out-of-tree drivers.
Thomas Lamprecht [Thu, 12 May 2022 07:17:54 +0000 (09:17 +0200)]
ui: content: show namespaces also inline and rework node type detection
this not only makes the action disable/hide checks simpler, but also
prepares the view a bit for the idea of adding a new API endpoint
that returns the whole datastore content tree as structured JSON so
that it can be directly loaded into a tree store.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Thomas Lamprecht [Tue, 10 May 2022 10:24:25 +0000 (12:24 +0200)]
ui: content view: improve empty text
reference NS so that users get a hint where they are currently
hierarchy-wise, and clarify that we found no *accessible* snapshots,
on this level, i.e., there can be some that we just cannot see due to
only having access on lover level NS or being different owners.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
the helper now takes both high-privilege and lesser-privilege privs, so
the resulting bool can be used to quickly check whether additional
checks like group ownership are needed or not.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com> Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
split the namespace out of BackupGroup/Dir api types
We decided to go this route because it'll most likely be
safer in the API as we need to explicitly add namespaces
support to the various API endpoints this way.
For example, 'pull' should have 2 namespaces: local and
remote, and the GroupFilter (which would otherwise contain
exactly *one* namespace parameter) needs to be applied for
both sides (to decide what to pull from the remote, and what
to *remove* locally as cleanup).
The *datastore* types still contain the namespace and have a
`.backup_ns()` getter.
Note that the datastore's `Display` implementations are no
longer safe to use as a deserializable string.
Additionally, some datastore based methods now have been
exposed via the BackupGroup/BackupDir types to avoid a
"round trip" in code.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com> Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
api: datastore: make permission checks namespace aware
We probably can combine the base permission + owner check, but for
now add explicit ones to upfront so that the change is simpler as
only one thing is done.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Thomas Lamprecht [Sun, 24 Apr 2022 18:24:42 +0000 (20:24 +0200)]
api: namespace management endpoints
allow to list any namespace with privileges on it and allow to create
and delete namespaces if the user has modify permissions on the parent
namespace.
Creation is only allowed if the parent NS already exists.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
projects / proxmox-backup.git / log