Dietmar Maurer [Wed, 26 Jan 2022 06:10:59 +0000 (07:10 +0100)]
Set MMAP_THRESHOLD to a fixed value (128K)
glibc's malloc has a misguided heuristic to detect transient allocations that
will just result in allocation sizes below 32 MiB never using mmap.
That it turn means that those relatively big allocations are on the heap where
cleanup and returning memory to the OS is harder to do and easier to be blocked
by long living, small allocations at the top (end) of the heap.
Observing the malloc size distribution in a file-level backup run:
We see that all allocations will be on the heap, and that while most
allocations are small, the relatively few big ones will still make up most of
the RSS and if blocked from being released back to the OS result in much higher
peak and average usage for the program than actually required.
Avoiding the "dynamic" mmap-threshold increasement algorithm and fixing it at
the original default of 128 KiB reduces RSS size by factor 10-20 when running
backups. As with memory mappings other mappings or the heap can never block
freeing the memory fully back to the OS.
But, the drawback of using mmap is more wasted space for unaligned or small
allocation sizes, and the fact that the kernel allegedly zeros out the data
before giving it to user space. The former doesn't really matter for us when
using it only for allocations bigger than 128 KiB, and the latter is a
trade-off, using 10 to 20 times less memory brings its own performance
improvement possibilities for the whole system after all ;-)
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
[ Thomas: added to comment & commit message + extra-empty-line fixes ] Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
these need to be checked (and are) via libssl anyway before persisting,
and newer versions might contain new ciphers/variants/... (and things
like @STRENGTH or @SECLEVEL=n were missing).
there isn't really a concept of 'nodes' in PBS (yet) anyway - and if
there ever is, it needs to be handled by the rest-server / specific API
endpoints (like in PVE), and not by the schema.
this allows dropping proxmox-sys from pbs-api-types (and thus nix and
some other transitive deps as well).
Thomas Lamprecht [Wed, 12 Jan 2022 11:36:36 +0000 (12:36 +0100)]
ui: sys config: merge webauthn and general options into one tab
To much wasted space else.
Also rename "Options" to "Others", while it's not _that_ much better
it's slightly more intuitive than config -> options (which has some
redundancy)...
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Dominik Csapak [Mon, 10 Jan 2022 11:08:14 +0000 (12:08 +0100)]
fix #3743: extract zfs dataset io stats from /proc/spl/kstat/zfs/POOL/objset-*
Recently, ZFS removed the pool global io stats from
/proc/spl/kstat/zfs/POOL/io with no replacement.
To gather stats about the datastores, access now the objset specific
entries there. To be able to make that efficient, cache a map of
dataset <-> obset ids, so that we do not have to parse all files each time.
We update the cache each time we try to get the info for a dataset
where we do not have a mapping.
We cannot update it on datastore add/remove since that happens in the
proxmox-backup daemon, while we need the info here in proxmox-backup-proxy.
Sadly with this we lose the io wait metric, but it seems that this is no
longer tracked in zfs at all, so nothing we can do for that.
the part number cannot go above 255 at the moment, but if it ever gets
bumped to a bigger integer type this boundary wouldn't cause a
compile-error. explicitly checking for overflowing u8 makes this a bit
more future-proof, and shuts up clippy as well ;)
Dominik Csapak [Thu, 16 Dec 2021 07:51:33 +0000 (08:51 +0100)]
api: zfs: create zpool with relatime=on
some operations (e.g. garbage collection/restore/etc.) are very read
intensive on the chunks, and having atime=on and relatime=off (zfs default)
makes those write intensive operations too. Additionally, 'ext4' defaults to
relatime, so also change the default for api-created zpools.
Dominik Csapak [Thu, 2 Dec 2021 14:26:31 +0000 (15:26 +0100)]
ui: fix opening settings window in datastore panel
When a user directly opened the webui with a fragment that is not
the summary, opening of the 'my settings' window fails because the
initial set of the columns field triggers a state change, which in turn
tries to trigger 'updateColumns'. That fails though, since the columns
were not even rendered yet (because we are on a different tab).
To fix this, simply return when the panel is not rendered yet.
Thomas Lamprecht [Wed, 15 Dec 2021 13:25:30 +0000 (14:25 +0100)]
buildsys: drop hack that moved testing after dh_install
the motivation for this was that we required to build some stuff with
different feature flags before the big-split when openid (that still
links to the dependency-greedy) got added, to avoid that binaries
that do not use openid at all also got linked to its dependencies.
This is now fixed since a bit and thus we should be able to drop the
test-reorder hack.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Thomas Lamprecht [Wed, 15 Dec 2021 13:13:46 +0000 (14:13 +0100)]
fix #3794: api types: set backup time lower limit to 1
Some users want to import historical backups but they run into the
original lower backuo-time limit one can pass. That original limit
was derived from the initial PBS development start in 2019, it was
assumed that no older backup can exist with PBS before it existing,
but imports of older backups is a legitimate thing.
I pondered using 683071200 (1991-08-25), aka the first time Linux was
publicly announced by Linus Torvalds as new limit but at the end I
did not wanted to risk that and backup software is IMO to serious for
such easter eggs, so I went for 1, to differ between the bogus 0 some
tools fallback too if there's something off with time.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
ui: calendar event: add once daily example and clarify workday one
similar to PVE, copying over the remaining commit message:
Using 00:00 with relying on the implied default is sub optimal as its
a bit of a magic example that new users may not understand as easily.
So spell it out explicitly, even if there'd be a shorter version
possible.
We also had some request for the once-daily every day, and its a
sensible example to have in general, could help getting the
difference between an hour list and a single one.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Dominik Csapak [Wed, 1 Dec 2021 10:57:08 +0000 (11:57 +0100)]
ui: form/GroupFilter: copy records for the pbsGroupSelectors
store.getData() returns an 'Ext.util.Collection' which is a special
class that does more than being an array of records. Namely, it can
have 'observers' which can react on the change of the collection
Here, the 'onWidgetAttach' callback will be called twice on the first
row add and the widgets (and thus stores) are cached by extjs. When
doing a 'setData' of a Collection, it tries to add the store as an
observer, but due to the above caching and multiple calling this fails
since the store is already an observer.
For this reason, we want to actually copy the records (which neither
the store, nor the Collection has a method for...)
This gives us an additional benefit: The different pbsGroupSelectors can
sort independently now, before it was all linked to the original store's
collection.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com> Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Dominik Csapak [Mon, 29 Nov 2021 14:39:41 +0000 (15:39 +0100)]
ui: add GroupFilter form field(container)
this contains a grid + button + hidden field which lets the user
add group filters one by one. the first column is the type selector
(type, group, regex) and the second column shows the relevant
input field (groupselector, kvcombobox for type, and textfield for regex)
i had to hack a little to get access to the widgets of the
fieldcontainer, since we cannot simply access the widget of a column
from another column (which we need to show the correct one when changing
the type), also we cannot traverse the widget hirachy in the usual way,
since extjs seems to build it differently for widgetcolumns.
to solve this, i added references of the widgets to the record, and a
reference of the record to the widgets. since this is now a cyclic
reference, i solve that in 'removeFilter' and in 'beforedestroy' of the grid
by removing the references again
also contains a small css style to remove the padding in the rows
Dietmar Maurer [Tue, 23 Nov 2021 16:57:00 +0000 (17:57 +0100)]
update to proxmox-sys 0.2 crate
- imported pbs-api-types/src/common_regex.rs from old proxmox crate
- use hex crate to generate/parse hex digest
- remove all reference to proxmox crate (use proxmox-sys and
proxmox-serde instead)
Thomas Lamprecht [Mon, 22 Nov 2021 10:21:23 +0000 (11:21 +0100)]
ui: datastore content: rework rendering protection state
avoid that there's the same icon rendered twice, once clickable and
once as status. Also indicate the protection with a literal text and
by highlighting the single shield with green, if protected.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
projects / proxmox-backup.git / log