Dominik Csapak [Thu, 29 Oct 2020 09:50:13 +0000 (10:50 +0100)]
server/worker_task: simplify task log writing
instead of prerotating 1000 tasks
(which resulted in 2 writes each time an active worker was finished)
simply append finished tasks to the archive (which will be rotated)
page cache should be good enough so that we can get the task logs fast
since existing installations might have an 'index' file, we
still have to read tasks from there, but only if it exists
this simplifies the TaskListInfoIterator a good amount
Dominik Csapak [Thu, 29 Oct 2020 10:51:26 +0000 (11:51 +0100)]
api: datstore status: introduce proper structs and restore compatibility
by moving the properties of the storage status out again to the top
level object
also introduce proper structs for the types used, to get type-safety
and better documentation for the api calls
this changes the backup counts from an array of [groups,snapshots] to
an object/struct with { groups, snapshots } and include 'other' types
(though we do not have any at this moment)
this way it is better documented
this also adapts the ui code to cope with the api changes
Dominik Csapak [Thu, 22 Oct 2020 07:05:37 +0000 (09:05 +0200)]
fix #2998: encode mtime as i64 instead of u64
saves files mtime as i64 instead of u64 which enables backup of
files with negative mtime
the catalog_decode_i64 is compatible to encoded u64 values (if < 2^63)
but not reverse, so all "old" catalogs can be read with the new
decoder, but catalogs that contain negative mtimes will decode wrongly
on older clients
also remove the arbitrary maximum value of 2^63 - 1 for
encode_u64 (we just use up to 10 bytes now) and correctly
decode them and update the comments accordingly
adds also test for i64 encode/decode and for compatibility between
u64 encode and i64 decode
Dominik Csapak [Wed, 28 Oct 2020 09:58:00 +0000 (10:58 +0100)]
tools/logrotate: fix compression logic
we never actually compressed any files, since we only looked at
the extension:
* if it was 'zst' (which was always true for newly rotated files), we
would not compress it
* even if it was not 'zst', we compressed it inplace, never adding '.zst'
(possibly compressing them multiple times as zstd)
now we add new rotated files simply as '.X' and add a 'target' to the
compress fn, where we rename it to (but now we have to unlink the source
path)
Thomas Lamprecht [Wed, 28 Oct 2020 17:25:47 +0000 (18:25 +0100)]
ui: datastore: show comment, allow to edit notes
the "comment" is the first line of the "notes" field from a manifest,
show it in the grid and allow editing the full notes.
Hack the click event listener a bit together for the right aligned
edit action button, but it works out well and is efficient (only one
event listener is much cheaper than per-buttons ones).
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Thomas Lamprecht [Wed, 28 Oct 2020 17:22:20 +0000 (18:22 +0100)]
ui: datastore: use simple V. for verify action button
Choosing a good icon is hard here, while the magnifying glass is
somewhat relatable, it reminds to much of a "Search" function, which
can be quite confusing here.
So use a simple "V.", even if it's probably also not to ideal..
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Thomas Lamprecht [Wed, 28 Oct 2020 14:33:04 +0000 (15:33 +0100)]
do_verification_job: fix "never-reverify" and refactor/comment
commit a4915dfc2bc7bef03354f97f5bbce9fe2df4e0d6 made a wrong fix, as
it did not observed that the last expressions was done under the
invariant that we had a last verification result, because if none
could be loaded we already returned true (include).
It thus broke the case for "never re-verify", which is important when
using multiple schedules, a more high frequent one for new,
unverified snapshots, and a low frequency to re-verify older snapshots,
e.g., monthly.
Fix this case again, rework the code to avoid this easy to oversee
invariant. Use a nested match to better express the implication of
each setting, and add some comments.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Dominik Csapak [Tue, 27 Oct 2020 15:20:10 +0000 (16:20 +0100)]
ui: MainView/NavigationTree: improve tree selection handling
this fixes some bugs related to selection handling in the treelist:
* datastores were not selected after a reload
* reloading when in a tabpanel on any tab but the first, would
not select a treenode
* changing between datastores on any tab but the first would
not select the same tab on the new datastore
fixed those by mostly rewriting the changePath handling for
datastores and tabpanels in general
Dominik Csapak [Tue, 27 Oct 2020 15:20:09 +0000 (16:20 +0100)]
ui: NavigationTree: add 'Add Datastore' button below datastore list
and make 'Datastore' unclickable
since we have all options and information on the relevant datastore panels,
we do not need a datastore config anymore (besides the creation,
which we add here)
this also fixes the sorted insertion and removal of new/old datastores
Dominik Csapak [Tue, 27 Oct 2020 15:20:07 +0000 (16:20 +0100)]
ui: add DataStoreSummary and move Statistics into it
this adds a 'Summary' panel to the datastores, similar to what we have
for PVE's nodes/guests/storages
contains an info panel with useful information, a comment field, and
the charts from the statistics panel (which can be deleted since it is
not necessary any more)
Stefan Reiter [Wed, 21 Oct 2020 09:41:16 +0000 (11:41 +0200)]
apt: add /changelog API call similar to PVE
For proxmox packages it works the same way as PVE, by retrieving the
changelog URL and issuing a HTTP GET to it, forwarding the output to the
client. As this is only supposed to be a workaround removed in the
future, a simple block_on is used to avoid async.
For debian packages we can simply call 'apt-get changelog' and forward
it's output.
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
Stefan Reiter [Wed, 21 Oct 2020 09:41:15 +0000 (11:41 +0200)]
fix #2934: list to-be-installed packages in updates
As always, libapt is mocking us with complexity, but we can get the
approximate result we want by retrieving dependencies of all
to-be-updated packages and then seeing if they are missing.
If they are, we assume they will be installed.
For this, query_detailed_info is extended to allow reading details for
non-installed packages, and this is also exposed in
list_installed_apt_packages via 'all_versions_for'. This is necessary so
we can retrieve changelogs for such packages.
Note that we cannot retrieve all that information all the time, as
querying details for packages that aren't installed takes a rather long
time.
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
Stefan Reiter [Wed, 21 Oct 2020 09:41:12 +0000 (11:41 +0200)]
apt: use 'apt-get changelog --print-uris' in get_changelog_url
Avoids custom hardcoded logic, but can only be used for debian packages
as of now. Adds a FIXME to switch over to use --print-uris only once our
package repos support that changelog format.
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
Dominik Csapak [Wed, 21 Oct 2020 07:29:08 +0000 (09:29 +0200)]
api2/admin/datastore/pxar_file_download: download directory as zip
by using the new ZipEncoder and recursively add files to it
the zip only contains directories, normal files and hardlinks (by simply
copying the content), no symlinks, etc.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com> Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Dylan Whyte [Tue, 20 Oct 2020 09:29:16 +0000 (11:29 +0200)]
fix #3038: check user before renewing ticket
Fixes a bug in which the userid of the ticket cache is updated,
when a user connects, but the ticket itself is not.
This means a newly connected user has a previously connected
user's ticket and thus, cannot do anything, as the client will
attempt to use the invalid ticket.
e.g. if john@pbs connected to the server first, followed by
mike@pbs, the following would be stored in the ticket cache.
Thomas Lamprecht [Tue, 20 Oct 2020 08:26:28 +0000 (10:26 +0200)]
log rotate: do NOT compress first rotation
The first rotation is normally the one still opened by one or more
processes for writing, so it must NOT be replaced, removed, ..., as
this then makes the remaining logging, until those processes are
noticed that they should reopen the logfile due to rotation, goes
into nirvana, which is far from ideal for a log.
Only rotating (renaming) is OK for this active file, as this does not
invalidates the file and keeps open FDs intact.
So start compressing with the second rotation, which should be clear
to use, as all writers must have been told to reopen the log during
the last rotation, reopen is a fast operation and normally triggered
at least day ago (at least if one did not dropped the state file
manually), so we are fine to archive that one for real.
If we plan to allow faster rotation the whole rotation+reopen should
be locked, so that we can guarantee that all writers switched over,
but this is unlikely to be needed.
Again, this is was logrotate sanely does by default since forever.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Thomas Lamprecht [Tue, 20 Oct 2020 08:05:51 +0000 (10:05 +0200)]
log rotate: do NOT overwrite file with possible writers
this is not the job of logrotate, and the real 20+ years battle
tested log rotate binary does not do so either as it's actually
pretty dangerous.
If we "replace" the file we break any logger which already opened a
new one here, e.g., a dameon starting up, and thus that writer would
log to nirvana.
It's the job of a logger to create a file if not existing, it makes
no sense to do it here.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Stefan Reiter [Tue, 20 Oct 2020 08:08:25 +0000 (10:08 +0200)]
fix #2988: allow verification after finishing a snapshot
To cater to the paranoid, a new datastore-wide setting "verify-new" is
introduced. When set, a verify job will be spawned right after a new
backup is added to the store (only verifying the added snapshot).
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
Stefan Reiter [Mon, 19 Oct 2020 14:45:22 +0000 (16:45 +0200)]
datastore: cleanup open and load config only once
Force consumers to use the lookup_datastore method instead of
potentially opening a datastore twice, and pass the config we have
already loaded into open_with_path, removing the need for open(1).
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>