Stefan Lendl [Thu, 7 Mar 2024 14:23:43 +0000 (15:23 +0100)]
gitignore: ignore generated synopsis and config docs
Signed-off-by: Stefan Lendl <s.lendl@proxmox.com>
[ TL: condense this to something more general ] Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Dominik Csapak [Wed, 6 Mar 2024 14:04:25 +0000 (15:04 +0100)]
ui: tape: transfer: increase timeout to 3 minutes
the default timeout of 30 seconds is too short to properly wait for a
slot transfer. Increase the timeout to a value of 3 minutes. In my
tests, it took about 60 seconds in a very basic changer to move a tape
between two slots, so triple that to account for bigger and more
complicated libraries.
Christian Ebner [Wed, 6 Mar 2024 14:11:53 +0000 (15:11 +0100)]
server: sync job: format downloaded amount in human readable units
Use the methods provided by HumanByte for the output for consistency
with the rest of the task log and better readability.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com> Reviewed-by: Max Carrara <m.carrara@proxmox.com> Tested-by: Max Carrara <m.carrara@proxmox.com> Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Adds a summary to the end of the task log showing the size and number
of chunks pulled as well as the average transfer rate.
Such an entry looks something like:
> Summary: sync job pulled 214.445 MiB in 166 chunks (average rate: 111.012 MiB/s)
Link: https://bugzilla.proxmox.com/show_bug.cgi?id=5285 Signed-off-by: Christian Ebner <c.ebner@proxmox.com> Reviewed-by: Max Carrara <m.carrara@proxmox.com> Tested-by: Max Carrara <m.carrara@proxmox.com> Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Christian Ebner [Wed, 6 Mar 2024 14:11:51 +0000 (15:11 +0100)]
server: sync job: return `PullStats` for pull related methods
Return basic statistics on pull related methods via `PullStats`
objects, in order to construct a global summary for sync jobs.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com> Reviewed-by: Max Carrara <m.carrara@proxmox.com> Tested-by: Max Carrara <m.carrara@proxmox.com> Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
values.username just does not exist, and we do not need to delete the
username part anyway, as that field is used to assemble the full
userid by concatenating the name@realm parts.
While at it move this over to let-assignments and do not call setting
expiry explicitly a hack, it's fine and warranted code, because if one
wants to use a datefield's empty value as 0 one needs to do so
explicitly, nothing hacky there..
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Stefan Sterz [Wed, 6 Mar 2024 11:45:56 +0000 (12:45 +0100)]
ui: user edit: don't send realm property
the api does not accept a realm property here, it is only needed to
construct a proper user id of the form `{username}@{realm}`. so
remove it before sending it to the api and getting an error in return.
Dominik Csapak [Wed, 6 Mar 2024 11:21:03 +0000 (12:21 +0100)]
ui: sync job: don't send delete value on creation
since that's not a valid api parameter there
we have to pass the `isCreate` value through to the inputpanel, we even
used it there already but it was never set.
Dominik Csapak [Wed, 6 Mar 2024 11:21:02 +0000 (12:21 +0100)]
ui: prune: fix sending invalid parameters
the prune input panel is used in various contexts (add/editing a
prunejob, adding a datastore, executing a prune). These different api
calls don't all take the same parameters, so we have to correctly set
the `isCreate` to not send a `delete` paramter for those request if
there was an empty field.
Also set 'max-depth:0' only when recursive was not set *and* we can
set 'recursive', because for creating a datastore that is not supported
by the api, and for the prune job editing we override the whole
onGetValues anyway so that's not an issue there.
Dominik Csapak [Wed, 6 Mar 2024 11:21:01 +0000 (12:21 +0100)]
ui: tape backup job: don't send delete value on creation
this is not a valid parameter for the create call. To do that in the
onGetValues method, we have to pass the 'isCreate' value through to the
input panels via cbind.
Dominik Csapak [Wed, 6 Mar 2024 11:21:00 +0000 (12:21 +0100)]
ui: metrics: don't send digest when creating a new influxdbupd host
we accidentally always tried to load an existing config, even when
creating a new entry. This returned the list of all configured ones plus
the digest (which gets set by the edit window). When the digest is set,
the edit window will send it along, but that does not exist for the
create api call, so it failed.
To fix it, guard the load behind the `serverid` property, which is only
set when we edit an existing entry.
Max Carrara [Fri, 1 Mar 2024 13:49:06 +0000 (14:49 +0100)]
fix #5217: api: send missing header when upgrading to HTTP/2
The "Connection: upgrade" header is strictly expected to be included
in the response sent by the server when an upgrade to a different
protocol is requested by the client.
A detailed explanation as well as additional context follows below.
Background
----------
Neither RFC 9110 (HTTP Semantics) [0] or RFC 7540 (HTTP/2) [1]
*explicitly state* that the "Connection: upgrade" header must be
included *in the server's response* when a client requests an upgrade
to a different protocol. For clients, however, it is specified [2]:
> A sender of Upgrade MUST also send an "Upgrade" connection option in
> the Connection header field (Section 7.6.1) to inform intermediaries
> not to forward this field.
Yet, the example for a response provided in RFC 9110 [3] does include
the header:
> HTTP/1.1 101 Switching Protocols
> Connection: upgrade
> Upgrade: websocket
>
> [... data stream switches to websocket with an appropriate response
> (as defined by new protocol) to the "GET /hello" request ...]
The example in RFC 7540 [4] also includes the header:
Additionally, RFC 9113 [5], which obsoletes RFC 7540 [1], mentions:
> The HTTP/1.1 Upgrade mechanism is deprecated and no longer specified
> in this document. It was never widely deployed, with plaintext
> HTTP/2 users choosing to use the prior-knowledge implementation
> instead.
I therefore initially concluded that whether the "Connection: upgrade"
header should / should not / must / must not be included in the
server's response was unspecified.
Further Revelations
-------------------
As per Thomas's suggestion [6], I opened a discussion over at Caddy's
GitHub issue tracker [7]. This discussion revealed that RFC 7230 [8],
which is obsoleted by RFC 9110 [1], does in fact specify that the
header must be included [9], thus proving my initial conclusion to be
incorrect:
> When a header field aside from Connection is used to supply control
> information for or about the current connection, the sender MUST
> list the corresponding field-name within the Connection header
> field. [...]
The discussion [7] also revealed that the WebSocket RFC 6455 [10]
specifies the usage of the "Connection" header in more detail [11]:
> 3. If the response lacks a |Connection| header field or the
> |Connection| header field doesn't contain a token that is an ASCII
> case-insensitive match for the value "Upgrade", the client MUST
> _Fail the WebSocket Connection_.
Furthermore [12]:
> 5. If the server chooses to accept the incoming connection, it
> MUST reply with a valid HTTP response indicating the following.
>
> [...]
>
> 3. A |Connection| header field with value "Upgrade".
Although we're using the upgrade mechanism for HTTP/2, the WebSocket
RFC [10] specifies its usage more clearly and most importantly, in an
explicit manner.
Final Conclusion
----------------
The "Connection: upgrade" header must therefore definitely be included
as per RFC 7230 section 6.1 [8], even if the newer RFC 9110 [1] does
not specify this explicitly anymore.
Finally, this fixes bug #5217 [13] and allows PBS to be deployed
behind Caddy. Also tested with nginx, which still works as expected.
Dominik Csapak [Fri, 1 Mar 2024 09:03:41 +0000 (10:03 +0100)]
ui: system config: fix deleting DNS entries
While PVE and PMG use a rather brittle "replace whole config" style on
their DNS entry CRUD API, the PBS one was made with a per-entry level
granularity, so that single entries can modified, or deleted, without
touching the others.
But the UI from the widget-toolkit was made for the older PVE/PMG
behavior and did not sent along the delete-array of to-be-deleted
keys.
Since widget-toolkit commit 8d161ac ("dns: update comment to avoid
coupling to downstream dependency") the DNS edit window supports
opting into that by setting the new `deleteEmpty` config parameter.
Stefan Sterz [Fri, 1 Mar 2024 09:40:19 +0000 (10:40 +0100)]
ui: trim whitespaces when adding a subscription key
users that add the correct subscription key just get unnecessarily
confused with a "value does not match the regex pattern" error if
they accidentally have a stray whitespace at the end or beginning
otherwise.
Switch to using our `proxmoxtextfield` component that provides a
`trimValue` config option since widget-toolkit commit 5d7d30d ("text
field: add trimValue config") that was made just for this case.
Signed-off-by: Stefan Sterz <s.sterz@proxmox.com>
[ TL: reference widget toolkit commit ] Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Gabriel Goller [Wed, 14 Feb 2024 09:52:01 +0000 (10:52 +0100)]
fix #4975: client: ignore E2BIG error flag
Some filesystems (f.e. zfs) support xattrs bigger than 64kB, sadly we
can't get them because the kernel vfs limits us. The syscalls listxattr
and getxattr will return a E2BIG error in this case.
Added a flag --ignore-e2big-xattr to the client, this will ignore the
metadata (but still backup the file) if this error occurs.
Signed-off-by: Gabriel Goller <g.goller@proxmox.com> Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
```
warning: in a `match` scrutinee, avoid complex blocks or closures with blocks; instead, move the block or closure higher and bind it with a `let`
--> src/bin/proxmox-backup-proxy.rs:874:58
|
874 | let stats = match tokio::task::spawn_blocking(|| {
| __________________________________________________________^
875 | | let hoststats = collect_host_stats_sync();
876 | | let (hostdisk, datastores) = collect_disk_stats_sync();
877 | | Arc::new((hoststats, hostdisk, datastores))
878 | | })
| |_________^
|
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#blocks_in_conditions
= note: `#[warn(clippy::blocks_in_conditions)]` on by default
```
Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
```
warning: `to_string` applied to a type that implements `Display` in `writeln!` args
--> src/server/report.rs:141:72
|
141 | let _ = writeln!(out, "error during read-dir - {}", err.to_string());
| ^^^^^^^^^^^^ help: remove this
|
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#to_string_in_format_args
```
Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
access first element with first() rather than get(0)
Fixes the clippy lint
```
warning: accessing first element with `self.transports.get(0)`
--> pbs-tape/src/lib.rs:283:9
|
283 | / self.transports
284 | | .get(0)
| |___________________^ help: try: `self.transports.first()`
|
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#get_first
= note: `#[warn(clippy::get_first)]` on by default
```
Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
use or_default instead of or_insert_with(Default::default)
We need to annotate some cases to allow the compile to infer the types.
Fixes the clippy lint:
```
warning: use of `or_insert_with` to construct default value
--> src/api2/tape/restore.rs:750:18
|
750 | .or_insert_with(Vec::new);
| ^^^^^^^^^^^^^^^^^^^^^^^^ help: try: `or_default()`
|
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#unwrap_or_default
= note: `#[warn(clippy::unwrap_or_default)]` on by default
```
Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
```
warning: using `SeekFrom::Current` to start from current position
--> src/tape/media_catalog.rs:798:23
|
798 | let pos = file.seek(SeekFrom::Current(0))?; // get current pos
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ help: replace with: `file.stream_position()`
|
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#seek_from_current
= note: `#[warn(clippy::seek_from_current)]` on by default
```
Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
```
warning: the borrowed expression implements the required traits
--> src/server/report.rs:193:47
|
193 | get_directory_content(&path)
| ^^^^^ help: change this to: `path`
|
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#needless_borrows_for_generic_args
= note: `#[warn(clippy::needless_borrows_for_generic_args)]` on by default
```
Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
```
warning: redundant explicit link target
--> src/tools/mod.rs:47:42
|
47 | /// Returns a new instance of [`Client`](proxmox_http::client::Client) configured for PBS usage.
| -------- ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ explicit target is redundant
| |
| because label contains path that resolves to same destination
|
= note: when a link's destination is not specified,
the label is used to resolve intra-doc links
= note: `#[warn(rustdoc::redundant_explicit_links)]` on by default
help: remove explicit link target
|
47 | /// Returns a new instance of [`Client`] configured for PBS usage.
| ~~~~~~~~~~
```
Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
Dominik Csapak [Mon, 12 Feb 2024 12:27:34 +0000 (13:27 +0100)]
fix #5229: tape: remove max sequence number limit
The idea was to limit the number of tapes in a media set, but this was
not enforced when adding a medium to a media set, only on read/parsing
the inventory. With that, it is possible to create media sets greater
than the limit which in turn blocks access to most functions via
api/cli/gui due to the check.
Instead of enforcing an arbitrary limit, simply warn on creation when
the media-set is very large (20).
To restore the whole media set, the time taken would still be at least 38
hours for LTO-4 and 250 hours for LTO-9.
We already have a section in the docs where we tell about the
disadvantages of large media sets.
Gabriel Goller [Tue, 6 Feb 2024 10:09:07 +0000 (11:09 +0100)]
fix #5190: api: OIDC: accept generic URIs for the ACR value
Allow more complex strings for the acr-value when using openid. The
openid documentation only specifies the acr-value *should* be an URI
[0]. Implemented a regex that loosely disallows some of the reserved
URI characters specified in the RFC [1].
Currently values like:
- "urn:mace:incommon:iap:silver"
- "urn:comsolve.nl:idp:contract:rba:location"
do NOT work, although they are correct URI's and common acr tokens.
For Proxmox VE we had to actually make this more strict to align with
each other, as there we accepted any string.
Since we additonally also support delcaring a "type" property for
`oneOf` schemas (to use with serde's *internally* tagged enum
representation, this contains an additional `typeProperty` and
`typeSchema` value.
It dumps as follows:
{
"type": "object",
"description": ...,
"typeProperty": "name-of-type-property",
"typeSchema": {
"type": "string",
"enum": [ ... ], // technically not enforced by the code
},
"oneOf": [
{
"title": "<value from the above 'enum' array>",
<schema>,
},
{
"title": "<value from the above 'enum' array>",
<schema>,
},
... <one for each 'enum' above>
// ^ exact match is not technically enforced by code
}
}
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
tape: factor getting encryption fingerprint tuple out
makes it a bit more readable as there's less "noise" in the read_label
function and as the separate new fn allows us to nicely use ? to early
return as it has an option in the return signature avoiding 5 lines of
code while not really getting more terse.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Dominik Csapak [Wed, 31 Jan 2024 13:42:33 +0000 (14:42 +0100)]
tape: fix regression in restoring key from medium
Since commit 1343dcaf we automatically try to load the key into the
drive after reading the media-set label, this cannot work for the case
where we actually restore the key from the tape itself.
To address this special case while preserving the automatic key
loading, everything except the setup of the key has been separated
from the 'read_label' method into a new function named
'read_label_without_loading_key'. Consequently, the 'restore-key' API
endpoint can be switched to utilize this new method, thereby avoiding
the issue.
Fixes: 1343dcaf ("tape: move 'set_encryption' calls to the TapeDriver") Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
[ TL: reword and shorten commit message ] Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Nightly rustc now warns about unused private fields in the case of a
non-pub newtype struct, so use an underscore-prefixed dummy field name
to get rid of the warning.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Dominik Csapak [Mon, 22 Jan 2024 11:50:34 +0000 (12:50 +0100)]
tape: assert encryption mode when using the PoolWriter
by introducing an 'assert_encryption_mode' that checks the desired
state, and bails out if it's different, called directly where we
previously set the encryption mode (which is now done automatically)
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
[ TL: add drive_ prefix and fleece in comment ] Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Dominik Csapak [Mon, 22 Jan 2024 11:50:33 +0000 (12:50 +0100)]
tape: move 'set_encryption' calls to the TapeDriver (and implementation)
namely everytime we know what the key for the tape has to be:
* after we write the MediaSetLabel
* after reading the MediaSetLabel
When handling data on tape, we always have to have the MediaSetLabel, so
we should always trigger one of these. Because of that, we should not be
able to forget to set the encryption mode.
Dominik Csapak [Mon, 22 Jan 2024 11:50:32 +0000 (12:50 +0100)]
tape: fix wrongly unloading encryption key
For security, we want to automatically unload the encryption key from
the drive when we're done, so there was a Drop handler for SgTape that
handles that. Sadly, our tool we use to set it in the first place, also
invoked the Drop handler, thus unloading the keys again immediately
To fix that, move the Drop handler one logical level higher to the
LtoTapeHandle, which is not used by the 'sg-tape-cmd'.
Dominik Csapak [Mon, 22 Jan 2024 11:50:30 +0000 (12:50 +0100)]
tape: use SgTape in sg-tape-cmd
instead of LtoTapeHandle. This way, we can simply always call the binary
from LtoTapeHandle, and don't have to concern ourselves with the sg_tape
calling.
Thomas Lamprecht [Thu, 18 Jan 2024 14:13:32 +0000 (15:13 +0100)]
report: change output contract of functions
let them manage it completely themselves, as we cannot really say if a
code-block fits for the whole output, like it was the case for the
function that returned a limited output of a 'top' process status
command.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Dominik Csapak [Thu, 11 Jan 2024 10:40:34 +0000 (11:40 +0100)]
api: tape: don't allow duplicate media label-texts
quite a few parts of our code assumes that the label-text is unique in
the inventory, which leads to rather unexpected behaviour when having
more than one tape with the same label-text, e.g. a
`proxmox-tape media destroy <LABEL>`
destroys the first one in the config
(same with moving to vault, etc.)
since having multiple tapes with the same human readable name is always
confusing, simply disallow that here
Dominik Csapak [Thu, 11 Jan 2024 10:40:32 +0000 (11:40 +0100)]
tape: handle duplicate label-texts in inventory
find_media_by_label_text assumes that the label-texts are unique, but
currently this is not necessarily the case. To properly handle that,
change the signature to return a result, and in case there are duplicate
ones, return an error.
Philipp Hufnagl [Tue, 2 Jan 2024 11:06:53 +0000 (12:06 +0100)]
ui: Show if Filter includes or excludes
To make the UI compatible, the Group Filter dialogue has been extended
by a second list, so it now features a list for all include filter and
one for all exclude filters.
Internally, all include as well as exclude filter are managed into one
list. The 2 list view is just for a cleaner representation in the UI.
Signed-off-by: Philipp Hufnagl <p.hufnagl@proxmox.com>
Philipp Hufnagl [Tue, 2 Jan 2024 11:06:52 +0000 (12:06 +0100)]
fix #4315: jobs: modify GroupFilter so include/exclude is tracked
After some discussion I canged the include/exclude behavior to first run
all include filter and after that all exclude filter (rather then
allowing to alternate inbetween). This is done by splitting them into 2
lists, running include first.
A lot of discussion happened how edge cases should be handled and we
came to following conclusion:
no include filter + no exclude filter => include all
some include filter + no exclude filter => filter as always
no include filter + some exclude filter => include all then exclude
Since a GroupFilter now also features an behavior, the Struct has been
renamed To GroupType (since simply type is a keyword). The new
GroupFilter now has a behaviour as a flag 'is_exclude'.
I considered calling it 'is_include' but a reader later then might not
know what the opposite of 'include' is (do not include? deactivate?). I
also considered making a new enum 'behaviour' but since there are only 2
values I considered it over engeneered.
Signed-off-by: Philipp Hufnagl <p.hufnagl@proxmox.com>
projects / proxmox-backup.git / log