Christian Ebner [Tue, 23 Apr 2024 17:25:55 +0000 (19:25 +0200)]
file restore: cover split metadata and payload archives
Attach the payload data archive as input stream to the decoder
and accessor instances for split archives.
Allows to restore contents from split archives via the
`proxmox-file-restore extract` command, by passing the metadata
archive name.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Christian Ebner [Wed, 21 Feb 2024 19:39:48 +0000 (20:39 +0100)]
catalog: shell: make split pxar archives accessible
Cover the cases where the pxar archive was uploaded as split payload
data and metadata streams. Instantiate the required reader and
decoder instances to access the metadata and payload data archives,
using the corresponding helper methods.
Allows to restore split metadata and payload stream pxar archives via
the catalog shell.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Christian Ebner [Thu, 15 Feb 2024 11:47:32 +0000 (12:47 +0100)]
client: mount: make split pxar archives mountable
Cover the cases where the pxar archive was uploaded as split payload
data and metadata streams. Instantiate the required reader and
decoder instances to access the metadata and payload data archives.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Christian Ebner [Wed, 21 Feb 2024 10:51:52 +0000 (11:51 +0100)]
client: restore: read payload from dedicated index
Whenever a split pxar archive is encountered, instantiate and attach
the required dedicated reader instance to the decoder instance on
restore.
Piping the output to stdout is not possible for these, as this would
require a decoder instance which can decode the input stream, while
maintaining the pxar stream format as output.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Christian Ebner [Tue, 21 May 2024 15:18:38 +0000 (17:18 +0200)]
client: tools: helper to check pxar filename extensions
With the introduction of split pxar archives, the allowed extensions
are now `.pxar`, `.mpxar` and `.ppxar`. Add a helper function to
allow to check for all valid variants, including the optional
additional `.didx` in case of a server archive name.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Christian Ebner [Tue, 19 Mar 2024 08:43:22 +0000 (09:43 +0100)]
client: helper: add method for split archive name mapping
Helper method that takes an archive name as input and checks if the
given archive is present in the manifest, by also taking possible
split archive extensions into account.
Returns the pxar archive name if found or the split archive names if
the split archive variant is present in the manifest.
If neither is matched, an error is returned signaling that nothing
matched entries in the manifest.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Christian Ebner [Mon, 22 Apr 2024 15:39:00 +0000 (17:39 +0200)]
client: pxar: optionally split metadata and payload streams
... and attach the split payload writer variant to the pxar archive
creation. By this, metadata and payload data will create different
dynamic indexes, allowing to lookup and reuse payload chunks without
the additional overhead of the pxar archive's metadata.
For now this functionality remains disabled and will be enabled in a
later patch once the logic for reusing the payload chunks is in
place.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Christian Ebner [Mon, 22 Apr 2024 14:23:40 +0000 (16:23 +0200)]
client: pxar: combine writers into struct
Introduce a `PxarWriters` struct to bundle all writer instances
required for the pxar archive creation into a single object to limit
the number of function call parameters.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Christian Ebner [Tue, 20 Feb 2024 16:07:08 +0000 (17:07 +0100)]
client: pxar: switch to stack based encoder state
... and adapt to the new reader/writer variant for encoder or
decoder/accessor to attach a dedicated payload input/output for split
pxar archives.
In preparation for look-ahead caching, where a passing around of
per-directory level encoder instances with internal references is
not feasible.
Previously, for each directory level a new encoder instance has been
generated, restricting possible implementation errors. These encoder
instances have been internally linked by references to keep track of
the state changes in a parent child relationship.
This is however not feasible when the encoder has to be passed by
mutable reference, as required by the look-ahead cache
implementation. The encoder has therefore been adapted to use a
single instance implementation with an internal stack keeping track
of the state.
Depends on the bumped pxar library version, including the patches to
attach the corresponding variant for the pxar reader/writer
instantiation.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Christian Ebner [Tue, 28 May 2024 09:42:33 +0000 (11:42 +0200)]
datastore: dynamic index: add method to get digest
In preparation for injecting reused payload chunks in payload streams
for regular files with unchanged metaddata. Allows to get the digest
of a dynamic index entry to construct a reusable dynamic entry from
it.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Christian Ebner [Tue, 28 May 2024 09:42:11 +0000 (11:42 +0200)]
api: datastore: refactor getting local chunk reader
Move the code to get the local chunk reader to a dedicated function
to make it reusable. The same code is required to get the local chunk
reader for the payload stream for split stream archives.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Christian Ebner [Tue, 28 May 2024 09:42:10 +0000 (11:42 +0200)]
client: backup: factor out extension from backup target
Instead of composing the backup target name and pushing it to the
backup list, push the archive name and extension separately, only
constructing it while iterating the list later.
By this it remains possible to additionally prefix the extension, as
required with the separate pxar metadata and payload indexes.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Shannon Sterz [Thu, 23 May 2024 11:25:59 +0000 (13:25 +0200)]
auth: add locking to `PbsAuthenticator` to avoid race conditions
currently we don't lock the shadow file when removing or storing a
password. by adding locking here we avoid a situation where storing
and/or removing a password concurrently could lead to a race
condition. in this scenario it is possible that a password isn't
persisted or a password isn't removed. we already do this for
the "token.shadow" file, so just use the same mechanism here.
Thomas Lamprecht [Wed, 22 May 2024 17:03:05 +0000 (19:03 +0200)]
tape: rework setting MAM Host type attributes
The product name is Proxmox Backup Server, not just Backup Server,
that makes no sense on its own and it really cannot be expected by
tools extracting any Medium Auxiliary Memory (MAM) info to render it
as `${app_vendor} ${app_name}`.
Drop the comment about ignoring errors, that's pretty clear with
the only-log-error construct.
Instead, add some comments about what the hex numbers refers too and
what their respective length (limit) is. The names where taken from
Table 315 "MAM Host type attributes" in the "IBM LTO SCSI Reference"
for LTO 9.
Slightly off-topic: The tape code really is a mess with sprinkling
those hex numbers hard coded all over the place, often with some
unchecked coupling in other places (like here, the list of set MAM
attrs and the one that get cleared can easily get out of sync..), but
that's for another time to clean-up (I need to cut a release).
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
docs: document notification-mode and merge old notification section
This new section describes how the notification-mode parameter works.
The section also contains also parts of the old notification section
from the maintenance chapter, reusing the description of the
`notify` and `notify-user` parameters.
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com> Reviewed-by: Gabriel Goller <g.goller@proxmox.com>
Gabriel Goller [Wed, 15 May 2024 09:58:45 +0000 (11:58 +0200)]
notifications: fix legacy sync notifications
When using the legacy notifications the sync mode would pick up the
settings from the prune-job, which default to Error. This completely
disables notifications for successful sync-jobs when using the legacy
system.
Reported in the forum: https://forum.proxmox.com/threads/147018/
Signed-off-by: Gabriel Goller <g.goller@proxmox.com> Tested-by: Max Carrara <m.carrara@proxmox.com> Reviewed-by: Lukas Wagner <l.wagner@proxmox.com>
Stefan Sterz [Wed, 6 Mar 2024 12:36:09 +0000 (13:36 +0100)]
auth: use auth-api when generating keys and generate ec keys
this commit switches pbs over to generating ed25519 keys when
generating new auth api keys. this also removes the last direct
usages of openssl here and further unifies key handling in the auth
api.
Stefan Sterz [Wed, 6 Mar 2024 12:36:08 +0000 (13:36 +0100)]
auth: move to auth-api's private and public keys when loading keys
this commit moves away from using openssl's `PKey` and uses the
wrappers from proxmox-auth-api. this allows us to handle keys in a
more flexible way and enables as to move to ec based crypto for the
authkey in the future.
Stefan Sterz [Wed, 6 Mar 2024 12:36:07 +0000 (13:36 +0100)]
auth: upgrade hashes on user log in
if a users password is not hashed with the latest password hashing
function, re-hash the password with the newest hashing function. we
can only do this on login and after the password has been validated,
as this is the only point at which we have access to the plain text
password and also know that it matched the original password.
Stefan Sterz [Wed, 6 Mar 2024 12:36:06 +0000 (13:36 +0100)]
auth: move to hmac keys for csrf tokens
previously we used a self-rolled implementation for csrf tokens. while
it's unlikely to cause issues in reality, as csrf tokens are only
valid for a given tickets lifetime, there are still theoretical
attacks on our implementation. so move all of this code into the
proxmox-auth-api crate and use hmac instead.
this change should not impact existing installations for now, as this
falls back to the old implementation if a key is already present. hmac
keys will only be used for new installations and if users manually
remove the old key and
Thomas Lamprecht [Tue, 21 May 2024 09:30:07 +0000 (11:30 +0200)]
ui: garbage-collection: use different state-id for global and per-datastore view
For one these different views have different columns shown, and more
importantly: with the state being shared one could change sorting in
the global view and then have that applied in the per-datastore view
too, even if one cannot sort that view explicitly otherwise as there's
just one row anyway. This small glitch might lead to a bit of
confusion in the worst case and looks unpolished in any way.
Note that I explicitly decided against encoding the datastore in the
state-id for the per-datastore views for now, as most users will want
to adapt layout (like column width) for all per-datastores views.
Having to re-do that for every datastore separately can be quite a
nuisance while the same user wanting different layout for each
datastore in their per-datastore view seems rather to be an edge case.
And we can always change this, so starting out with the slightly more
restricted design that has less browser local data to be saved seems
better w.r.t. maintainability.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Gabriel Goller [Thu, 16 May 2024 09:18:41 +0000 (11:18 +0200)]
fix #5422: ui: garbage-collection: make columns in global view sortable
Make columns sortable in the global 'Prune & GC Jobs' view. In the
per-datastore view the columns will not be sortable as there can only be
one job.
Fixes: db3fd213 ("fix #3217: ui: global prune and gc job view") Co-authored-by: Thomas Lamprecht <t.lamprecht@proxmox.com> Signed-off-by: Gabriel Goller <g.goller@proxmox.com> Tested-by: Max Carrara <m.carrara@proxmox.com>
Dominik Csapak [Wed, 15 May 2024 09:55:13 +0000 (11:55 +0200)]
restore daemon: search disk also with truncated serial
the disk serial given to virtio disks only can be 20 characters, so
looking for a disk with a longer serial will always fail (like
'drive-tpmstate0-backup'). If the serial is longer, also try with the
truncated one. Leave the first try in place in case the limit changes.
Dominik Csapak [Wed, 15 May 2024 09:55:12 +0000 (11:55 +0200)]
restore daemon: log some errors for dir traversal
in case we cannot stat a file in the restore vm, log the path and reason
why. This should normally not happen, but when it does, the path and
error might help us find the issue.
Dominik Csapak [Mon, 13 May 2024 10:49:26 +0000 (12:49 +0200)]
tape: include drive activity in status
Since we don't query each drives status seperately, but rely on a single
call to the drives listing parameter for that, we now add the option
to query the activity there too. This makes that data avaiable for us
to show in a seperate (by default hidden) column.
Also we show the activity in the 'State' column when the drive is idle
from our perspective. This is useful when e.g. an LTO-9 tape is loaded
the first time and is calibrating, since that happens automatically.
Dominik Csapak [Mon, 13 May 2024 10:49:25 +0000 (12:49 +0200)]
tape: drive status: make some depend on the activity
when the tape drive has an activity (and the tape is in motion), certain
calls block until the operation is finished. Since we cannot predict how
long it's going to be and it can be quite long in certain cases,
skip those calls when the drive is doing anything.
If we cannot determine the activity, try to do the queries.
We have to extend the check for a loaded drive in the UI, since the
position is not available during any activity.
Dominik Csapak [Tue, 7 May 2024 13:45:52 +0000 (15:45 +0200)]
tape: improve throughput by not unnecessarily syncing/committing
When writing data on tape, the idea was to sync/committing to tape and
the catalog to disk every 128GiB of data. For that the counter
'bytes_written' was introduced and checked after every chunk/snapshot
archive.
Sadly we forgot to reset the counter after doing so, which meant that
after 128GiB was written onto the tape, we synced/committed after every
archive on the tape for the remaining length of the tape.
Since syncing to tape and writing to disk takes a bit of time, the drive
had to slow down every time and reduced the available throughput. (In
our tests here from ~300MB/s to ~255MB/s).
By resetting the value to zero after syncing, we avoid that and increase
throughput performance when backups are bigger than 128GiB on tape.
Gabriel Goller [Fri, 26 Apr 2024 14:02:43 +0000 (16:02 +0200)]
api-types: remove influxdb bucket name restrictions
Remove the regex for influxdb organizations and buckets. Influxdb does
not place any constraints on these names and allows all characters. This
allows influxdb organization names with slashes.
Also remove a duplicate comment and add some missing ones.
This also aligns the behavior to PVE as there are no restrictions there
either.
The motivation for this patch is this forum post:
https://forum.proxmox.com/threads/influx-db-organization-doesnt-allow-slash.145402/
Signed-off-by: Gabriel Goller <g.goller@proxmox.com>
ui: sync job: fix error if local namespace is selected first
When creating a new sync job and a local namespace is configured
without setting a remote first, the createMaxPrefixLength
was passed an array instead of a string/undefined/null, which
triggered a 'ns2.match is not a funtion exception', making the UI
glitchy afterwards.
Fixed by explicitly checking for a string. Verified that the other
user of NamespaceMaxDepthReduced, the prune job edit window, does not
break after the change.
Stefan Sterz [Fri, 5 Apr 2024 14:12:57 +0000 (16:12 +0200)]
fix: tape ui: unset `deleteEmpty` in `TapeBackupWindow`
since the api rejects unknown parameters, deleteEmpty needs to be
unset here, because the endpoint for creating backups does not support
deleting parameters. otherwise a user will get a fairly cryptic error
message in the gui.
Stefan Lendl [Thu, 4 Apr 2024 10:00:35 +0000 (12:00 +0200)]
api: create and update vlan interfaces
* Implement setting vlan-id and vlan-raw-device in the create and update api.
* Checking if the provided vlan-raw-device exists
* Moved VLAN_INTERFACE_REGEX to top level network module to use it in
the checking functions there. Changed to match with named capture groups.
* Unit tests to verify parsing vlan_id and vlan_raw_device from name.
Signed-off-by: Stefan Lendl <s.lendl@proxmox.com> Tested-by: Lukas Wagner <l.wagner@proxmox.com> Reviewed-by: Lukas Wagner <l.wagner@proxmox.com> Tested-by: Folke Gleumes <f.gleumes@proxmox.com> Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
* Add lexer Token enum variants for vlan-id and vlan-raw-device and parse
them in parse_iface_attributes.
* Add tests to verify this works in the above scenarios
Signed-off-by: Stefan Lendl <s.lendl@proxmox.com> Tested-by: Lukas Wagner <l.wagner@proxmox.com> Reviewed-by: Lukas Wagner <l.wagner@proxmox.com> Tested-by: Folke Gleumes <f.gleumes@proxmox.com> Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Stefan Lendl [Thu, 4 Apr 2024 10:00:31 +0000 (12:00 +0200)]
config: write vlan network interface
* Add vlan_id and vlan_raw_device fields to the Interface api type
* Write to the network config the vlan specific properties for vlan
interface type
* Add several tests to verify the functionally
Signed-off-by: Stefan Lendl <s.lendl@proxmox.com> Tested-by: Lukas Wagner <l.wagner@proxmox.com> Reviewed-by: Lukas Wagner <l.wagner@proxmox.com> Tested-by: Folke Gleumes <f.gleumes@proxmox.com> Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Dominik Csapak [Wed, 6 Mar 2024 09:39:45 +0000 (10:39 +0100)]
api: tape: don't allow overwriting of ids in changer/drive config
by checking the whole section config for an existing id, not only the
ones of the given type.
This prevents creation of a drive config with the same name as an
existing changer and vice versa, as it is confusing that existing things
get deleted, and we can get in the situation that we reference a changer
that does not exist anymore, i.e. consider this:
* create a changer with name `foo`
* create a drive with name `foo` and select changer `foo` for it
this would delete the changer config, but still reference it, leading
to errors when trying to use it.
We could implement support for separate id namespaces in section configs
for different types, but this is much more easier to do and be enough
for now.
installation: add section about unattended/automatic installation
Mention and briefly explain it. The main part of the documentation will
live in the Wiki for now as it applies to not just Proxmox Mail Gateway.
Signed-off-by: Aaron Lauterer <a.lauterer@proxmox.com>
[ TL: adapt to changes made in the wiki article ] Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Stefan Sterz [Fri, 12 Apr 2024 12:31:56 +0000 (14:31 +0200)]
auth: request a write lock when exposing the `LockedTfaConfig`
this function is called every time a user tries to log in to check
whether a tfa challenge is required. since the tfa config may need to
be written by the auth api (e.g. when a recovery key is used) this
needs to use a write lock instead of a read lock in order to avoid
potential races.
Christoph Heiss [Fri, 12 Jan 2024 16:16:04 +0000 (17:16 +0100)]
realm sync: add sync job for AD realms
Basically just a thin wrapper over the existing LDAP-based realm sync
job, which retrieves the appropriate config and sets the correct user
attributes.
api: notification: also list datastores if user has only Backup privs
Use the /admin/datatore API instead of /config/datastore to get a list
of all available datastores - this ensures that users can see
datastores even if they only have Datastore.Backup privs.
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com> Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
ui: notifications: pull in UX improvements for match rules creation
These changes have not been applied yet in widget toolkit, but
are very valuable for the initial integration in PBS.
We override modified components and replace them with the patched
variants.
The changes change the edit window such that known field names and
values are suggested in a combobox. Also, the 'exact' match mode
can now match multiple values.
This can and *should* be removed once the changes from [1] are
merged into the widget toolkit.
proxmox-backup-manager: add CLI for SMTP endpoints
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com> Tested-by: Gabriel Goller <g.goller@proxmox.com> Reviewed-by: Gabriel Goller <g.goller@proxmox.com> Tested-by: Maximiliano Sandoval <m.sandoval@proxmox.com> Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
proxmox-backup-manager: add CLI for sendmail endpoints
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com> Tested-by: Gabriel Goller <g.goller@proxmox.com> Reviewed-by: Gabriel Goller <g.goller@proxmox.com> Tested-by: Maximiliano Sandoval <m.sandoval@proxmox.com> Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
proxmox-backup-manager: add CLI for gotify endpoints
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com> Tested-by: Gabriel Goller <g.goller@proxmox.com> Reviewed-by: Gabriel Goller <g.goller@proxmox.com> Tested-by: Maximiliano Sandoval <m.sandoval@proxmox.com> Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
proxmox-backup-manager: add CLI for notification matchers
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com> Tested-by: Gabriel Goller <g.goller@proxmox.com> Reviewed-by: Gabriel Goller <g.goller@proxmox.com> Tested-by: Maximiliano Sandoval <m.sandoval@proxmox.com> Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
proxmox-backup-manager: add CLI for notification targets
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com> Tested-by: Gabriel Goller <g.goller@proxmox.com> Reviewed-by: Gabriel Goller <g.goller@proxmox.com> Tested-by: Maximiliano Sandoval <m.sandoval@proxmox.com> Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
ui: permissions paths: add /system/notifications to combobox
The /system/notifications ACL path is used for configuring the
notification system.
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com> Tested-by: Gabriel Goller <g.goller@proxmox.com> Reviewed-by: Gabriel Goller <g.goller@proxmox.com> Tested-by: Maximiliano Sandoval <m.sandoval@proxmox.com> Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
ui: datastore edit: make new stores use notification system by default
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com> Tested-by: Gabriel Goller <g.goller@proxmox.com> Reviewed-by: Gabriel Goller <g.goller@proxmox.com> Tested-by: Maximiliano Sandoval <m.sandoval@proxmox.com> Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
ui: utils: add overrides for known notification metadata fields/values
This mechanism allows having nice, translatable notification event
types and fields.
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com> Tested-by: Gabriel Goller <g.goller@proxmox.com> Reviewed-by: Gabriel Goller <g.goller@proxmox.com> Tested-by: Maximiliano Sandoval <m.sandoval@proxmox.com> Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com> Tested-by: Gabriel Goller <g.goller@proxmox.com> Reviewed-by: Gabriel Goller <g.goller@proxmox.com> Tested-by: Maximiliano Sandoval <m.sandoval@proxmox.com> Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
projects / proxmox-backup.git / log