src/bin/proxmox-backup-client.rs: verify blob/catlog checksums

src/pxar/sequential_decoder.rs: make functions needed in non-sequential decoder accessible from within crate

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>

src/pxar/format_definitions.rs: move header type in PxarHeader and hash in PxarGoodbyeItem to top of struct

By this it is possible to read and check just the first u64 of the corresponding
structs in order to identify the items.
This is needed for the fuse implementation in order to get entries based on the
archive offset, used as inode.
Directories are referenced by the offset to the goodbye tail while other items
are referenced by the offset of the filename followed by the entry.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>

src/pxar/fuse.rs: add function to map offset to inode and vice versa

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>

src/pxar/decoder.rs: add getter functions to obtain start/end offsets for directories

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>

src/pxar/decoder.rs: rustfmt fixups

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>

src/bin/proxmox-backup-client.rs: refactoring of match statement

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>

src/bin/proxmox-backup-client.rs: code cleanup - factor out some functions

src/backup.rs: define INDEX_BLOB_NAME here

src/bin/proxmox-backup-client.rs - restore: verify checksums

src/backup/fixed_index.rs: new helper to compute checksum and file size

src/backup/dynamic_index.rs: new helper to compute checksum and file size

src/client/http_client.rs: fix dynamic index checksum

comment fixup

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>

tools: add tokio::main() replacement

to deal with block_on() not allowing blocking()

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>

add WrappedReaderStream test

Mostly to just document the rt.block_on() issue with
blocking().

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>

Makefile: fixup: remove references to proxmox-protocol

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>

merge fixup

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>

add Cargo.lock

because we're currently using alpha releases...

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>

update to tokio 0.2.0-alpha.4

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>

drop hyper_openssl

We can use .await now, which means the whole connection
state machine doesn't need to be typed out as "types"
anymore, so, at least until hyper_openssl gets updated to
proper dependencies, let's drop it.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>

add generic either-or AsyncRead/Write type

The HttpsConnector will use this. Instead of implementing a
specialized MaybeTlsStream, this is simply a generic "either
this or that kind of Async Read/Write type".

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>

src/server/state.rs: update to tokio alpha.2

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>

src/server/rest.rs: use tokio::timer::delay

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>

tests/worker-task-abort.rs: switch to async

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>

src/tools/wrapped_reader_stream.rs: switch to async

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>

src/tools/futures.rs: switch to async

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>

src/tools/daemon.rs: switch to async

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>

src/tools/broadcast_future.rs: switch to async

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>

src/tools/async_mutex.rs: switch to async

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>

src/server/worker_task.rs: switch to async

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>

src/server/state.rs: switch to async

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>

src/server/rest.rs: switch to async

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>

src/server/h2service.rs: switch to async

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>

src/server/command_socket.rs: switch to async

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>

src/client/remote_chunk_reader.rs: switch to async

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>

src/client/pxar_backup_stream.rs: switch to async

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>

src/client/pipe_to_stream.rs: switch to async

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>

src/client/merge_known_chunks.rs: switch to async

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>

src/client/http_client.rs: switch to async

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>

src/bin/upload-speed.rs: switch to async

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>

src/bin/test_chunk_speed2.rs: switch to async

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>

src/bin/proxmox-backup-proxy.rs: switch to async

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>

src/bin/proxmox-backup-client.rs: switch to async

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>

src/bin/proxmox-backup-api.rs: switch to async

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>

src/bin/h2s-server.rs: switch to async

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>

src/bin/h2server.rs: switch to async

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>

src/bin/h2s-client.rs: switch to async

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>

src/bin/h2client.rs: switch to async

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>

src/bin/download-speed.rs: switch to async

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>

src/backup/index.rs: switch to async

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>

src/backup/chunk_stream.rs: switch to async

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>

src/api_schema/router.rs: switch to async

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>

src/api2/reader.rs: switch to async

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>

src/api2/backup/upload_chunk.rs: switch to async

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>

src/api2/backup.rs: switch to async

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>

src/api2/admin/datastore.rs: switch to async

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>

implement Sync for DynamicIndexReader and FixedIndexReader

hyper's wrap_stream now needs this

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>

drop src/storage/futures.rs

it's unused and uses a tokio::Task I don't want to lookup
the std-future conversion for...

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>

switch to futures 0.3

* cleanup Cargo.toml
* pull in tower_service (required for newer hyper)
* use local hyper-openssl fixup (its Cargo.toml has broken
  outdated dependencies)
* add pin-utils dependency

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>

rc/bin/proxmox-backup-client.rs - restore: always download index.json.blob

src/bin/proxmox-backup-client.rs - reatore: avoid loading (large) blobs into memory

bump zstd-sys version

src/client/http_client.rs: Refactor handling Option and Result types

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>

tools: remove SignalFd

We don't use it anymore and it'll just be more work when
switching to async-await.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>

backup/dynamic_index: explicitly zero-initialize the header

The writer.write_all() call accessed data marked as
undefined to valgrind. Note that we shouldn't write out
uninitialized memory for security reasons anyway.
(note that vec::undefined already did zero-initialize the
data, but also marked it as undefined for valgrind when
compiling with the valgrind feature)

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>

backup/dynamic_index: use static assertion, fix size

The type was sized properly but the number was still wrong,
fixed this.
TODO! Once unions with non-Copy values are stable make this
a `union { full: [u8; 4096], data: TheActualHeader }`;

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>

pxar/fuse: add pxar cli flag to set single- or multi-threaded session loop

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>

src/pxar/sequentail_decoder.rs: followup: cleanup S_IFMT masking

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>

src/pxar/sequential_decoder.rs: style fixups

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>

src/pxar/dir_stack.rs: style fixups

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>

src/pxar/fuse.rs: style fixups

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>

src/pxar/match_pattern.rs: style fixups

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>

src/pxar/format_definition.rs: style fixup

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>

remove proxmox-protocol subcrate

AFAICT we have no use for it anymore, its api entry points
are gone. If we do end up needing something from it, it's
still in the git history anyway. (And about two thirds of it
can be made much less awkward by utilizing async-await
anyway, so no love lost there...)

Moved the chunker back into src/backup/chunker.rs

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>

more formatting & use statement fixups

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>

import rustfmt.toml

This just sets edition to 2018.
Unfortunately pretty much all other configuration options
are considered unstable for now ...

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>

src/pxar/encoder.rs: style fixups by rustfmt

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>

(pxar: replace deprecated std::mem::uninitialized()

... and use std::mem::MaybeUninit or proxmox::tools::vec::uninitialized() instead.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>

{dynamic,fixed}_index: replace mem::uninitialized

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>

src/backup/index: style fixup & unsafe copy removal

We can use the safe .copy_from_slice alternative in this
case.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>

src/backup/index: replace mem::uninitialized

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>

api2/node/time: replace mem::uninitialized

and fixup use statements

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>

tools: tty: replace mem::uninitialized and style fixup

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>

tools: timer: style fixup

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>

timer: replace mem::uninitialized

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>

tools: style & use statement fixups

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>

src/config: more style fixups

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>

src/config: style fixup

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>

src/config: use statement cleanup

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>

/var/run -> /run

/var/run is considered deprecated and for instance in
systemd unit files lintian complains...

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>

disable offensive test

Tests we run with `make test` should not require root
privileges or write to system paths! This one does both!

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>

buildsys: defer some variable expansions

So that local.mak needs to only contain a PREFIX.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>

switch from create_dir_chown to create_path

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>

update to nix 0.15

The proxmox crate now needs it.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>

.gitignore: add local.mak

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>

pxar/decoder: avoid failing on hardlinks and return corresponding target file entry

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
[Fixup: remove unnecessary 'mut']
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>

src/pxar/decoder.rs: get correct stats for root directory

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>

src/pxar/decoder.rs: move GOODBYE_ITEM_SIZE from function to module scope.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>

use statement cleanup

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>