rest server: log rotation: fix off-by-one for max_days
The entries in a file go from oldest end-time in the first time to
newest end-time in the last line. So, just because the first line is
older than the cut-off time, the remaining one doesn't necessarily
have to be old enough too. What we can know for sure that older than
the current checked rotations of the task archive are definitively up
for deletion.
Another possibility would be to check the last line, but as scanning
backwards is more expensive/complex to do while only being an actual
improvement in a very specific edge case (it's more likely to have a
mixed time-cutoff vs. task-log-file boundary than that those are
aligned)
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Similar to 874bd545 ("pxar: fix anchored exclusion at archive root"),
but this time for inclusion. Because of the inconsistency, it could
happen that a file included in generate_directory_file_list() got
excluded in add_entry(), e.g. with a .pxarexclude file like
> *
> !/supposed-to-be-included
use BufReader/Writer for Files passed to serde_json::from_reader/writer
As serde_json will otherwise read files 1 byte at a time.
Writing is a bit better, but syntacitcal elements (quotes, braces,
commas) still often show up as single write syscalls, so use BufWriter
there as well.
Note that while we do store the file in the resulting objects, we do not
need to keep the buffered read/writers as we always `seek` to the
beginning on further file operations.
Markus Frank [Thu, 24 Mar 2022 11:49:30 +0000 (12:49 +0100)]
fix #3854 paperkey import to proxmox-tape
added a parameter to the cli for importing tape key via a json-parameter or
via reading a exported paperkey-file or json-file.
For this i also added a backupkey parameter to the api, but here it only
accepts json.
The cli interprets the parameter first as json-string, then json-file
and last as paperkey-file.
functionality:
proxmox-tape key paperkey [fingerprint of existing key] > paperkey.backup
proxmox-tape key restore --backupkey paperkey.backup # key from line above
proxmox-tape key restore --backupkey paperkey.json # only the json
proxmox-tape key restore --backupkey '{"kdf": {"Scrypt": ...' # json as string
for importing the key as paperkey-file it is irrelevant, if the paperkey got exported as html
or txt.
pbs-client: print error when we couldn't download previous fidx/didx
When we have a previous manifest, we try to download the fidx/didx files
to get the known chunks list. We continue if that fails (which is ok),
but we did not print any error, leading to a confusing backup output,
since the users would expect that chunks will be reused.
Printing the error should at least make it apparent that something did
not work correctly.
the latest changes to this api call changed/removed some things that
were actually necessary for the gui. Readd those and document them this
time.
The change from u64 to i64 limits us to 8EiB of Datastore sizes (instead if
16EiB) but if we reach that, we must adapt most other parts to use 128bit
sizes anyway
Dominik Csapak [Tue, 8 Mar 2022 12:20:56 +0000 (13:20 +0100)]
proxmox-backup-debug api: parse parameters before sending to api
when we use http to make the api call, we have to parse the parameters
before, else we might send the string "true" instead of the boolean true
and the api rejects it with a 'Parameter verification error'.
We already have all api call schemas here, so parsing is possible.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com> Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Dominik Csapak [Mon, 7 Mar 2022 08:13:30 +0000 (09:13 +0100)]
tools: zfs_dataset_stats: remove dataset <-> obset file mapping on error
this can only real fail for two reasons:
* the format is wrong:
this should not happen unless the format changed, then it will
happen every time
* the file can't be read:
this can happen if a user deletes and recreates a dataset manually,
since the mapped file does not exist anymore but the dataset does
for the second case, delete the mapping from the hashmap, so that the
next call will refresh the mapping with the correct file
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com> Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
commenting that version_cache.increase_datastore_generation increases
the, well, version is rather superfluous. Also avoid the use of "we",
which is always ambiguous in code comments.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Dominik Csapak [Fri, 25 Feb 2022 14:26:11 +0000 (15:26 +0100)]
pbs-datastore: use ConfigVersionCache for datastore
instead of relying on the content of some configs
previously, we always read and parsed the config file, and only
generated a new config object when the path or the 'verify-new' option
changed.
now, we increase the datastore generation on config save, and if that
changed (or the last load is 1 minute in the past), we always
generate a new object
Dominik Csapak [Mon, 28 Feb 2022 11:20:07 +0000 (12:20 +0100)]
docs: explain retention time for event allocation policy in more detail
'when the calendar event' triggers was too vague, it could mean for
the current media-set or the next time. Apart from that, it was not
technically correct all the time, since we take the start time of
the next media set if that exists first.
The idea here is that we begin the retention when the media set is
finished.
Dominik Csapak [Tue, 22 Feb 2022 14:57:49 +0000 (15:57 +0100)]
datastore: add tuning option for chunk order
currently, we sort chunks by inode when verifying or backing up to tape.
we get the inode# by stat'ing each chunk, which may be more expensive
than the gains of reading the chunks in order
Since that is highly dependent on the underlying storage of the datastore,
introduce a tuning option so that the admin can tune that behaviour
for each datastore.
Markus Frank [Fri, 18 Feb 2022 09:55:38 +0000 (10:55 +0100)]
fix #3323: cli client: add dry-run option for backup command
adds a dry-run parameter for "proxmox-backup-client backup".
With this parameter on it simply prints out what would be uploaded,
instead of uploading it.
Signed-off-by: Markus Frank <m.frank@proxmox.com> Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Dominik Csapak [Thu, 17 Feb 2022 13:50:40 +0000 (14:50 +0100)]
tape/pool_writer: skip already backed up chunks in iterator
currently, the iterator goes over *all* chunks of the index, even
those already backed up by a previous snapshots in the same tape
backup. this is bad since for each iterator, we stat each chunk to
sort by inode number. so to avoid stat'ing the same chunks over
and over for consecutive snapshots, add a 'skip_fn' to the iterator
and in the pool writer and check the catalog_set if we can skip it
this means we can drop the later check for the catalog_set
(since we don't modify that here)
Stefan Sterz [Thu, 10 Feb 2022 14:23:24 +0000 (15:23 +0100)]
fix #3853: api: add force option to tape key change-passphrase
When force is used, the current passphrase is not required. Instead
it will be read from the file pointed to by TAPE_KEYS_FILENAME and
the old key configuration will be overwritten using the new
passphrase. Requires super user privileges.
Aaron Lauterer [Mon, 7 Feb 2022 13:37:29 +0000 (14:37 +0100)]
report: move subscription info further up
This is something that is checked all the time. Having it further up
saves on scrolling and brings it into better alignment with PVE & PMG
regarding where in the report the info is located.
ui: webauthn: fix stopping store upgrades on destroy
`deactivate` is only triggered if we switch to a different tab on
the same navigation level, but if we switch to a completely different
component (e.g., fom `Options -> Others` to `Datastore foo`) we can
only work with the destroy event, use the before one as else we
cannot access the view controllers method anymore.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Markus Frank [Thu, 3 Feb 2022 11:57:26 +0000 (12:57 +0100)]
fix #3856 hint parameter is not optional
For the API the parameter --hint is not optional. This patch fixes
the man page and cli command doesn't send an API call, if the
parameter does not exist.
projects / proxmox-backup.git / log