commenting that version_cache.increase_datastore_generation increases
the, well, version is rather superfluous. Also avoid the use of "we",
which is always ambiguous in code comments.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Dominik Csapak [Fri, 25 Feb 2022 14:26:11 +0000 (15:26 +0100)]
pbs-datastore: use ConfigVersionCache for datastore
instead of relying on the content of some configs
previously, we always read and parsed the config file, and only
generated a new config object when the path or the 'verify-new' option
changed.
now, we increase the datastore generation on config save, and if that
changed (or the last load is 1 minute in the past), we always
generate a new object
Dominik Csapak [Mon, 28 Feb 2022 11:20:07 +0000 (12:20 +0100)]
docs: explain retention time for event allocation policy in more detail
'when the calendar event' triggers was too vague, it could mean for
the current media-set or the next time. Apart from that, it was not
technically correct all the time, since we take the start time of
the next media set if that exists first.
The idea here is that we begin the retention when the media set is
finished.
Dominik Csapak [Tue, 22 Feb 2022 14:57:49 +0000 (15:57 +0100)]
datastore: add tuning option for chunk order
currently, we sort chunks by inode when verifying or backing up to tape.
we get the inode# by stat'ing each chunk, which may be more expensive
than the gains of reading the chunks in order
Since that is highly dependent on the underlying storage of the datastore,
introduce a tuning option so that the admin can tune that behaviour
for each datastore.
Markus Frank [Fri, 18 Feb 2022 09:55:38 +0000 (10:55 +0100)]
fix #3323: cli client: add dry-run option for backup command
adds a dry-run parameter for "proxmox-backup-client backup".
With this parameter on it simply prints out what would be uploaded,
instead of uploading it.
Signed-off-by: Markus Frank <m.frank@proxmox.com> Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Dominik Csapak [Thu, 17 Feb 2022 13:50:40 +0000 (14:50 +0100)]
tape/pool_writer: skip already backed up chunks in iterator
currently, the iterator goes over *all* chunks of the index, even
those already backed up by a previous snapshots in the same tape
backup. this is bad since for each iterator, we stat each chunk to
sort by inode number. so to avoid stat'ing the same chunks over
and over for consecutive snapshots, add a 'skip_fn' to the iterator
and in the pool writer and check the catalog_set if we can skip it
this means we can drop the later check for the catalog_set
(since we don't modify that here)
Stefan Sterz [Thu, 10 Feb 2022 14:23:24 +0000 (15:23 +0100)]
fix #3853: api: add force option to tape key change-passphrase
When force is used, the current passphrase is not required. Instead
it will be read from the file pointed to by TAPE_KEYS_FILENAME and
the old key configuration will be overwritten using the new
passphrase. Requires super user privileges.
Aaron Lauterer [Mon, 7 Feb 2022 13:37:29 +0000 (14:37 +0100)]
report: move subscription info further up
This is something that is checked all the time. Having it further up
saves on scrolling and brings it into better alignment with PVE & PMG
regarding where in the report the info is located.
ui: webauthn: fix stopping store upgrades on destroy
`deactivate` is only triggered if we switch to a different tab on
the same navigation level, but if we switch to a completely different
component (e.g., fom `Options -> Others` to `Datastore foo`) we can
only work with the destroy event, use the before one as else we
cannot access the view controllers method anymore.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Markus Frank [Thu, 3 Feb 2022 11:57:26 +0000 (12:57 +0100)]
fix #3856 hint parameter is not optional
For the API the parameter --hint is not optional. This patch fixes
the man page and cli command doesn't send an API call, if the
parameter does not exist.
Dominik Csapak [Fri, 28 Jan 2022 14:10:30 +0000 (15:10 +0100)]
traffic-control: use SocketAddr from 'accept()'
instead of getting the 'peer_addr()' from the socket.
The advantage is that we must get this and thus can drop the mapping
from result -> option, and can drop the testing for None and a test case
verifying a single snapshot is now never skipped because of recent verify
verifying a group will now reverify after 29 days to be consistent
with the 'All OK (old)' display
Thomas Lamprecht [Wed, 26 Jan 2022 15:08:19 +0000 (16:08 +0100)]
file restore: scale per-round delay up dynamically
Avoids latency for restore-VMs that are finished fast but not ready
yet the first round while not checking to often for slower ones, iow,
we assume that the start up distribution is looking like a chi-square
Χ² with k=3.
With 25*round we get at max 45 rounds totalling to 25.875 s delay and
1.125 max between-round delay, which still provides an ok reaction
time.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Thomas Lamprecht [Wed, 26 Jan 2022 15:04:41 +0000 (16:04 +0100)]
file restore: always wait up to 25s
the timeout for connecting may be much shorter if we get a response
(which doesn't needs to be Ok, e.g., "Connection refused"), so
instead of trying a fixed amount of 60 times lets try for 25s
independent of how often that will be then.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Dietmar Maurer [Wed, 26 Jan 2022 06:10:59 +0000 (07:10 +0100)]
Set MMAP_THRESHOLD to a fixed value (128K)
glibc's malloc has a misguided heuristic to detect transient allocations that
will just result in allocation sizes below 32 MiB never using mmap.
That it turn means that those relatively big allocations are on the heap where
cleanup and returning memory to the OS is harder to do and easier to be blocked
by long living, small allocations at the top (end) of the heap.
Observing the malloc size distribution in a file-level backup run:
We see that all allocations will be on the heap, and that while most
allocations are small, the relatively few big ones will still make up most of
the RSS and if blocked from being released back to the OS result in much higher
peak and average usage for the program than actually required.
Avoiding the "dynamic" mmap-threshold increasement algorithm and fixing it at
the original default of 128 KiB reduces RSS size by factor 10-20 when running
backups. As with memory mappings other mappings or the heap can never block
freeing the memory fully back to the OS.
But, the drawback of using mmap is more wasted space for unaligned or small
allocation sizes, and the fact that the kernel allegedly zeros out the data
before giving it to user space. The former doesn't really matter for us when
using it only for allocations bigger than 128 KiB, and the latter is a
trade-off, using 10 to 20 times less memory brings its own performance
improvement possibilities for the whole system after all ;-)
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
[ Thomas: added to comment & commit message + extra-empty-line fixes ] Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
these need to be checked (and are) via libssl anyway before persisting,
and newer versions might contain new ciphers/variants/... (and things
like @STRENGTH or @SECLEVEL=n were missing).
there isn't really a concept of 'nodes' in PBS (yet) anyway - and if
there ever is, it needs to be handled by the rest-server / specific API
endpoints (like in PVE), and not by the schema.
this allows dropping proxmox-sys from pbs-api-types (and thus nix and
some other transitive deps as well).
Thomas Lamprecht [Wed, 12 Jan 2022 11:36:36 +0000 (12:36 +0100)]
ui: sys config: merge webauthn and general options into one tab
To much wasted space else.
Also rename "Options" to "Others", while it's not _that_ much better
it's slightly more intuitive than config -> options (which has some
redundancy)...
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Dominik Csapak [Mon, 10 Jan 2022 11:08:14 +0000 (12:08 +0100)]
fix #3743: extract zfs dataset io stats from /proc/spl/kstat/zfs/POOL/objset-*
Recently, ZFS removed the pool global io stats from
/proc/spl/kstat/zfs/POOL/io with no replacement.
To gather stats about the datastores, access now the objset specific
entries there. To be able to make that efficient, cache a map of
dataset <-> obset ids, so that we do not have to parse all files each time.
We update the cache each time we try to get the info for a dataset
where we do not have a mapping.
We cannot update it on datastore add/remove since that happens in the
proxmox-backup daemon, while we need the info here in proxmox-backup-proxy.
Sadly with this we lose the io wait metric, but it seems that this is no
longer tracked in zfs at all, so nothing we can do for that.
the part number cannot go above 255 at the moment, but if it ever gets
bumped to a bigger integer type this boundary wouldn't cause a
compile-error. explicitly checking for overflowing u8 makes this a bit
more future-proof, and shuts up clippy as well ;)
projects / proxmox-backup.git / log