]>
git.proxmox.com Git - proxmox-backup.git/log
Dietmar Maurer [Wed, 29 May 2019 05:08:34 +0000 (07:08 +0200)]
src/backup/fixed_index.rs: remove ChunkStat from struct
Dietmar Maurer [Wed, 29 May 2019 05:02:45 +0000 (07:02 +0200)]
src/backup/fixed_index.rs: improve consistency checks
Christian Ebner [Tue, 28 May 2019 09:16:21 +0000 (11:16 +0200)]
pxar: change logic and impl all command line flags for xattrs/fcaps/acls
Allows to individually set the flags for storing/dumping/restoring of
xattrs/fcaps/acls in the cli of pxar.
Changes logic so that each of them can be threated individually.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Dietmar Maurer [Tue, 28 May 2019 10:55:22 +0000 (12:55 +0200)]
src/api2/admin/datastore/backup.rs: log sucessful finish call
Dietmar Maurer [Tue, 28 May 2019 08:12:44 +0000 (10:12 +0200)]
src/bin/proxmox-backup-client.rs: use new BackupClient to upload data
Dietmar Maurer [Tue, 28 May 2019 07:35:08 +0000 (09:35 +0200)]
src/tools.rs: implement image_size helper
Dietmar Maurer [Tue, 28 May 2019 07:12:38 +0000 (09:12 +0200)]
src/api2/admin/datastore/backup/environment.rs: add more consistency checks
Dietmar Maurer [Tue, 28 May 2019 07:01:01 +0000 (09:01 +0200)]
src/client/http_client.rs: use start of chunk as offset
Dietmar Maurer [Tue, 28 May 2019 06:53:27 +0000 (08:53 +0200)]
src/client/http_client.rs: code cleanup
Dietmar Maurer [Tue, 28 May 2019 06:47:03 +0000 (08:47 +0200)]
src/backup/fixed_index.rs: improve bound checks inside add_digest
Dietmar Maurer [Tue, 28 May 2019 06:27:57 +0000 (08:27 +0200)]
src/backup/fixed_index.rs: store index_length to avoid code duplication
Dietmar Maurer [Tue, 28 May 2019 05:13:19 +0000 (07:13 +0200)]
src/api2/admin/datastore/backup.rs: fix typo
Christian Ebner [Mon, 27 May 2019 12:16:13 +0000 (14:16 +0200)]
src/bin/pxar.rs: set archive permissions to 0o640
As the archive can contain potentially sensitive data such as key files, it makes
sense to restrict the permissions.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Christian Ebner [Mon, 27 May 2019 11:59:17 +0000 (13:59 +0200)]
src/pxar/encoder.rs: Refactor file stat
Introduce helper functions to check file stats
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Christian Ebner [Mon, 27 May 2019 11:01:38 +0000 (13:01 +0200)]
src/pxar/encoder.rs: Don't bail if endpoint does not support xattrs.
The encoder bailed if a endpoint which did not support xattrs was encountered.
Instead of bailing, we ignore these errors and simply do not store xattrs for
such endpoints.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Dietmar Maurer [Tue, 28 May 2019 04:18:55 +0000 (06:18 +0200)]
src/api2/admin/datastore/backup.rs: implement fixed sized chunk upload api
Dietmar Maurer [Tue, 28 May 2019 04:18:24 +0000 (06:18 +0200)]
src/backup/fixed_index.rs: make chunk_size public
Dietmar Maurer [Mon, 27 May 2019 07:35:30 +0000 (09:35 +0200)]
src/client/http_client.rs: correctly parse error response in download_chunk_list
Dietmar Maurer [Mon, 27 May 2019 05:57:43 +0000 (07:57 +0200)]
src/api2/admin/datastore/backup/upload_chunk.rs: rename upload_dynamic_chunk into upload_chunk
Dietmar Maurer [Mon, 27 May 2019 05:40:50 +0000 (07:40 +0200)]
src/api2/admin/datastore/backup.rs: use smaller window sizes
Dietmar Maurer [Mon, 27 May 2019 05:24:32 +0000 (07:24 +0200)]
src/client/http_client.rs: fix request pipelining
Dietmar Maurer [Sun, 26 May 2019 08:52:56 +0000 (10:52 +0200)]
src/client/merge_known_chunks.rs: use MergedChunkInfo as input
Dietmar Maurer [Sun, 26 May 2019 07:57:34 +0000 (09:57 +0200)]
src/client/http_client.rs: append chunks in correct order
We need to wait for response from upload_chunk. After that we can
add the chunk to the index file.
Dietmar Maurer [Fri, 24 May 2019 08:05:22 +0000 (10:05 +0200)]
src/api2/admin/datastore/backup.rs: verify chunk offset
This clearly shows that the current approach does not work - seems we get
chunks out of order.
Dietmar Maurer [Fri, 24 May 2019 07:10:59 +0000 (09:10 +0200)]
src/server/rest.rs: correctly verify json parameters
Dietmar Maurer [Fri, 24 May 2019 06:32:55 +0000 (08:32 +0200)]
src/client/http_client.rs: encode parameters inside body
To avoid problems with long uris.
Dietmar Maurer [Fri, 24 May 2019 05:42:58 +0000 (07:42 +0200)]
src/client/merge_known_chunks.rs: reduce RAM usage
There is no need to store chunk data for known chunks.
Dietmar Maurer [Fri, 24 May 2019 05:36:09 +0000 (07:36 +0200)]
src/client/merge_known_chunks.rs: avoid VecDequeu - a single buffer is enough
Christian Ebner [Thu, 23 May 2019 16:09:15 +0000 (18:09 +0200)]
src/pxar/sequential_decoder.rs: impl support to dump/restore ACLs from pxar archives
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Christian Ebner [Thu, 23 May 2019 16:08:56 +0000 (18:08 +0200)]
src/pxar/encoder.rs: impl support to store ACLs in pxar achives
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Christian Ebner [Thu, 23 May 2019 16:08:47 +0000 (18:08 +0200)]
src/pxar/format_definition.rs: define types and structs needed to impl ACL support in pxar
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Christian Ebner [Thu, 23 May 2019 11:10:20 +0000 (13:10 +0200)]
pxar: pass feature_flags to encoder/decoder instead of individual flags
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Christian Ebner [Thu, 23 May 2019 10:14:22 +0000 (12:14 +0200)]
src/tools/xattr.rs: add test for is_valid_xattr_name()
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Christian Ebner [Thu, 23 May 2019 09:58:40 +0000 (11:58 +0200)]
Minor refactoring of pxars xattr encoder/decoder, mostly reformatting and renaming.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Wolfgang Bumiller [Thu, 23 May 2019 13:19:20 +0000 (15:19 +0200)]
doc-test fixup
cargo test by default compiles and runs all code snippets
found in the documentation...
oops...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Wolfgang Bumiller [Thu, 23 May 2019 09:27:45 +0000 (11:27 +0200)]
RFC: schema: make enums static
I don't see a reason to allow these to be dynamically
modifiable.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Dietmar Maurer [Thu, 23 May 2019 10:29:33 +0000 (12:29 +0200)]
src/client/merge_known_chunks.rs: merge known chunks
To decrease the number of api calls required...
Dietmar Maurer [Thu, 23 May 2019 07:42:37 +0000 (09:42 +0200)]
src/client/http_client.rs: use ChunkInfo streams
This will make out of order uploads possible...
Wolfgang Bumiller [Wed, 22 May 2019 13:02:16 +0000 (15:02 +0200)]
tree-wide: use the new vec/io tools modules
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Wolfgang Bumiller [Wed, 22 May 2019 13:41:20 +0000 (15:41 +0200)]
buildsys: make valgrind=1 to enable valgrind support
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Wolfgang Bumiller [Wed, 22 May 2019 12:43:24 +0000 (14:43 +0200)]
tools: add helpful vector and read operations
After importing the I/O ops trait via:
use crate::tools::io::ops::*;
Instead of:
let mut buffer = vec![0u8; 65536];
file.read_exact(&mut buffer)?;
use:
let buffer = file.read_exact_allocated(65536)?;
After importing the vector helpers via:
use crate::tools::vec::{self, ops::*};
For a buffer which *could* be uninitialized but you prefer
zero-initialization anyway for security reasons, instead of:
let mut buffer = vec![0u8; len];
use:
let mut buffer = vec::undefined(len);
which zero-initializes, but, if the `valgrind` feature flag
is enabled, marks the vector as having undefined contents,
so reading from it will cause valgrind errors.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Dietmar Maurer [Thu, 23 May 2019 06:50:36 +0000 (08:50 +0200)]
src/api2/admin/datastore/backup.rs: verify file size and chunk count on close
Dietmar Maurer [Thu, 23 May 2019 06:15:32 +0000 (08:15 +0200)]
src/server/rest.rs: improve error handling
Dietmar Maurer [Thu, 23 May 2019 06:05:39 +0000 (08:05 +0200)]
src/api2/admin/datastore/backup/service.rs: improve error logging
Dietmar Maurer [Thu, 23 May 2019 05:20:05 +0000 (07:20 +0200)]
src/bin/upload-speed.rs: code cleanup
Christian Ebner [Wed, 22 May 2019 15:50:44 +0000 (17:50 +0200)]
tests/catar.rs: Pass dummy xattr/fcaps feature_flags to encoder.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Christian Ebner [Wed, 22 May 2019 15:50:43 +0000 (17:50 +0200)]
src/bin/pxar.rs: impl CLI flags to disable archiving/restoring of xattrs and fcaps.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Christian Ebner [Wed, 22 May 2019 15:50:42 +0000 (17:50 +0200)]
src/client/: pass dummy flags for xattrs and fcaps to pxar encoder/decoder.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Christian Ebner [Wed, 22 May 2019 15:50:41 +0000 (17:50 +0200)]
src/pxar/decoder.rs: pass flags for xattrs and fcaps to decoder.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Christian Ebner [Wed, 22 May 2019 15:50:40 +0000 (17:50 +0200)]
src/pxar/sequential_decoder.rs: Conditional restore xattrs and fcaps if feature_flags is set
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Christian Ebner [Wed, 22 May 2019 15:50:39 +0000 (17:50 +0200)]
src/pxar/format_definitions.rs: do not store the feature_flags in pxar archive.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Christian Ebner [Wed, 22 May 2019 15:50:38 +0000 (17:50 +0200)]
src/pxar/encoder: Use encoder feature_flags, do not store them within the archive.
Instead of testing and storing the feature_flags in the pxar archive, set and use
the feature flags within the encoder.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Christian Ebner [Wed, 22 May 2019 13:02:41 +0000 (15:02 +0200)]
tests/pxar.rs: Make sure the correct pxar binary is run in the tests.
If compiled in debug mode, the debug binary should be tested, in release mode
the release binary has to be invoked.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Christian Ebner [Wed, 22 May 2019 10:00:23 +0000 (12:00 +0200)]
Makefile: make the tests folder available in the build folder on 'make deb'
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Dietmar Maurer [Thu, 23 May 2019 04:28:10 +0000 (06:28 +0200)]
src/client/http_client.rs: re-order code - public functions first
Dietmar Maurer [Wed, 22 May 2019 15:28:25 +0000 (17:28 +0200)]
src/client/http_client.rs: move low level H2 code into separate class
Dietmar Maurer [Wed, 22 May 2019 11:24:33 +0000 (13:24 +0200)]
src/client/http_client.rs: code cleanup
Dietmar Maurer [Wed, 22 May 2019 11:05:51 +0000 (13:05 +0200)]
src/client/http_client.rs: cleanup h2 backup client
Dietmar Maurer [Wed, 22 May 2019 11:04:58 +0000 (13:04 +0200)]
src/api2/admin/datastore/backup.rs: suse POST for finish() api
Dietmar Maurer [Wed, 22 May 2019 09:14:56 +0000 (11:14 +0200)]
src/bin/h2test.rs: removed
This test does no work anymore.
Dietmar Maurer [Wed, 22 May 2019 08:55:27 +0000 (10:55 +0200)]
src/api2/admin/datastore/backup.rs: remove test callbacks
Christian Ebner [Tue, 21 May 2019 15:54:49 +0000 (17:54 +0200)]
Makefile: Allow to pass parameter tests=pattern to make test
`make test tests=pattern` will only run the subset of tests containing pattern,
therefore allowing to specify which subset of tests to run.
If the parameter is not specified, all tests are run.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Christian Ebner [Tue, 21 May 2019 12:58:05 +0000 (14:58 +0200)]
src/tools/xattr.rs: impl unit tests for fsetattr and fgetattr
As git does not preserve xattrs they have to be set before they can be read.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Dietmar Maurer [Wed, 22 May 2019 07:46:02 +0000 (09:46 +0200)]
src/client/http_client.rs: implement download_chunk_list
Dietmar Maurer [Wed, 22 May 2019 07:39:02 +0000 (09:39 +0200)]
src/backup/chunk_stream.rs: use BytesMut in ChunkStream
Dietmar Maurer [Wed, 22 May 2019 07:18:05 +0000 (09:18 +0200)]
src/backup/chunk_stream.rs: use more generics
Dietmar Maurer [Wed, 22 May 2019 07:05:35 +0000 (09:05 +0200)]
src/backup/chunk_stream.rs: optimize FixedChunkStream (use BytesMut)
Dietmar Maurer [Wed, 22 May 2019 06:05:27 +0000 (08:05 +0200)]
src/backup/chunk_stream.rs: fix behaviour for large input buffers
Dietmar Maurer [Wed, 22 May 2019 05:44:51 +0000 (07:44 +0200)]
src/backup/chunk_stream.rs: assemble data with correct order
Dietmar Maurer [Wed, 22 May 2019 05:43:54 +0000 (07:43 +0200)]
src/api2/admin/datastore/backup.rs: improve error message
Dietmar Maurer [Tue, 21 May 2019 11:42:13 +0000 (13:42 +0200)]
src/tools.rs: fix hex_to_digest
Dietmar Maurer [Tue, 21 May 2019 10:28:44 +0000 (12:28 +0200)]
src/backup/index.rs: rename ChunkListReader into DigestListEncoder
Dietmar Maurer [Tue, 21 May 2019 10:21:22 +0000 (12:21 +0200)]
src/backup/index.rs: implement DigestListDecoder
Dietmar Maurer [Tue, 21 May 2019 07:54:11 +0000 (09:54 +0200)]
src/api2/admin/datastore/backup.rs: ignore error after finish, register known_chunks
Dietmar Maurer [Tue, 21 May 2019 07:52:19 +0000 (09:52 +0200)]
src/backup/dynamic_index.rs: add chunk_info method
Christian Ebner [Mon, 20 May 2019 16:08:56 +0000 (18:08 +0200)]
tests/pxar.rs: Integrity test for pxar
Test the integrity of the restored folder content of data previously archived with pxar
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Christian Ebner [Fri, 17 May 2019 12:23:35 +0000 (14:23 +0200)]
src/pxar/sequential_decoder.rs: impl functionality to restore xattrs/fcaps from pxar achives
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Christian Ebner [Fri, 17 May 2019 12:23:34 +0000 (14:23 +0200)]
src/pxar/encoder.rs: impl functionality to encode xattrs/fcaps in pxar
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Christian Ebner [Fri, 17 May 2019 12:23:33 +0000 (14:23 +0200)]
src/tools/xattr.rs: add functions name_store and security_capability
These functions allow to check if the name for xattrs/fcaps are valid.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Dietmar Maurer [Mon, 20 May 2019 16:05:10 +0000 (18:05 +0200)]
src/api2/admin/datastore/backup/environment.rs: register/lookup known chunks
Dietmar Maurer [Mon, 20 May 2019 15:29:00 +0000 (17:29 +0200)]
src/client/pxar_backup_stream.rs: fix docu
Dietmar Maurer [Mon, 20 May 2019 12:19:24 +0000 (14:19 +0200)]
src/client/http_client.rs: implement upload_stream
Dietmar Maurer [Mon, 20 May 2019 09:20:33 +0000 (11:20 +0200)]
src/client/pxar_backup_stream.rs: close pipe inside drop
Dietmar Maurer [Sun, 19 May 2019 10:15:46 +0000 (12:15 +0200)]
src/tools.rs: implement hex_to_digest
Dietmar Maurer [Sun, 19 May 2019 09:09:02 +0000 (11:09 +0200)]
src/bin/test_chunk_speed2.rs: test max chunk size
Dietmar Maurer [Sun, 19 May 2019 09:05:56 +0000 (11:05 +0200)]
Revert "src/backup/chunk_stream.rs: simplify code"
This reverts commit
e058744d8f4dc1d87fe2a28cd79bc385e934f563 .
The optimization was wrong, and produces wrong chunk boundaries.
Dietmar Maurer [Sun, 19 May 2019 08:18:51 +0000 (10:18 +0200)]
src/api_schema/schema.rs: improve error messages
Dietmar Maurer [Sat, 18 May 2019 13:13:31 +0000 (15:13 +0200)]
src/client/pxar_backup_stream.rs: use WrappedStreamReader
to make it fully async ...
Dietmar Maurer [Sat, 18 May 2019 10:57:43 +0000 (12:57 +0200)]
src/client/pxar_backup_stream.rs: use 1M pipe buffer size
Dietmar Maurer [Sat, 18 May 2019 09:59:17 +0000 (11:59 +0200)]
src/bin/test_chunk_speed2.rs: test chunker speed with real data
Dietmar Maurer [Sat, 18 May 2019 08:54:05 +0000 (10:54 +0200)]
src/backup/chunk_stream.rs: simplify code
Dietmar Maurer [Sat, 18 May 2019 08:46:29 +0000 (10:46 +0200)]
src/backup/chunk_stream.rs: implement fixed size chunk stream
Dietmar Maurer [Fri, 17 May 2019 09:51:14 +0000 (11:51 +0200)]
src/client/http_client.rs: improve speed test
pipeline responses to a separate async channel ...
Wolfgang Bumiller [Thu, 16 May 2019 09:58:38 +0000 (11:58 +0200)]
add qemu-io crate, AioContext reactor helper
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Christian Ebner [Wed, 15 May 2019 13:27:34 +0000 (15:27 +0200)]
src/tools/xattrs.rs: impl libc wrapper functions to get/set xattrs and fcaps
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Christian Ebner [Wed, 15 May 2019 13:27:33 +0000 (15:27 +0200)]
src/pxar/format_definitions.rs: add format definitions for xattrs and fcaps
In addition to the format definition, the traits needed for sorting of xattr
entries by name are derived.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Christian Ebner [Wed, 15 May 2019 11:43:42 +0000 (13:43 +0200)]
src/tools/acl.rs: impl add_entry_full for ACL and set_permissions for ACLEntry
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Dietmar Maurer [Thu, 16 May 2019 08:24:23 +0000 (10:24 +0200)]
src/api2/admin/datastore/backup.rs: add speedtest api, improve upload speed
We need to disable tcp Nagle algorythm (set_nodelay), and use larger window size for http2
Dietmar Maurer [Wed, 15 May 2019 10:58:55 +0000 (12:58 +0200)]
src/api2/admin/datastore/backup.rs: implement finish_backup
Dietmar Maurer [Wed, 15 May 2019 08:09:46 +0000 (10:09 +0200)]
src/api2/admin/datastore/backup.rs: improve logging