``rustup`` Toolchain
====================
-If you use ``rustup``, you need to configure what toolchain you want
-to use. We set the default in the file ``./rust-toolchain`` to ``system``:
-
- # cat rust-toolchain
- system
-
-This works if you have installed the ``rustc`` Debian packages, and told
-``rustup`` the location of that link:
+We normally want to build with the ``rustc`` Debian package. To do that
+you can set the following ``rustup`` configuration:
# rustup toolchain link system /usr
+ # rustup default system
Versioning of proxmox helper crates
To use current git master code of the proxmox* helper crates, add::
- git = "ssh://gitolite3@proxdev.maurer-it.com/rust/proxmox"
+ git = "git://git.proxmox.com/git/proxmox"
or::
to the proxmox dependency, and update the version to reflect the current,
pre-release version number (e.g., "0.1.1-dev.1" instead of "0.1.0").
+
Local cargo config
==================
To reference new dependencies (or updated versions) that are not yet packaged,
the dependency needs to point directly to a path or git source (e.g., see
example for proxmox crate above).
+
+
+Build
+=====
+on Debian Buster
+
+Setup:
+ 1. # echo 'deb http://download.proxmox.com/debian/devel/ buster main' >> /etc/apt/sources.list.d/proxmox-devel.list
+ 2. # sudo wget http://download.proxmox.com/debian/proxmox-ve-release-6.x.gpg -O /etc/apt/trusted.gpg.d/proxmox-ve-release-6.x.gpg
+ 3. # sudo apt update
+ 4. # sudo apt install devscripts debcargo clang
+ 5. # git clone git://git.proxmox.com/git/proxmox-backup.git
+ 6. # sudo mk-build-deps -ir
+
+Note: 2. may be skipped if you already added the PVE or PBS package repository
+
+You are now able to build using the Makefile or cargo itself.
+
+
+Design Notes
+============
+
+Here are some random thought about the software design (unless I find a better place).
+
+
+Large chunk sizes
+-----------------
+
+It is important to notice that large chunk sizes are crucial for
+performance. We have a multi-user system, where different people can do
+different operations on a datastore at the same time, and most operation
+involves reading a series of chunks.
+
+So what is the maximal theoretical speed we can get when reading a
+series of chunks? Reading a chunk sequence need the following steps:
+
+- seek to the first chunk start location
+- read the chunk data
+- seek to the first chunk start location
+- read the chunk data
+- ...
+
+Lets use the following disk performance metrics:
+
+:AST: Average Seek Time (second)
+:MRS: Maximum sequential Read Speed (bytes/second)
+:ACS: Average Chunk Size (bytes)
+
+The maximum performance you can get is::
+
+ MAX(ACS) = ACS /(AST + ACS/MRS)
+
+Please note that chunk data is likely to be sequential arranged on disk, but
+this it is sort of a best case assumption.
+
+For a typical rotational disk, we assume the following values::
+
+ AST: 10ms
+ MRS: 170MB/s
+
+ MAX(4MB) = 115.37 MB/s
+ MAX(1MB) = 61.85 MB/s;
+ MAX(64KB) = 6.02 MB/s;
+ MAX(4KB) = 0.39 MB/s;
+ MAX(1KB) = 0.10 MB/s;
+
+Modern SSD are much faster, lets assume the following::
+
+ max IOPS: 20000 => AST = 0.00005
+ MRS: 500Mb/s
+
+ MAX(4MB) = 474 MB/s
+ MAX(1MB) = 465 MB/s;
+ MAX(64KB) = 354 MB/s;
+ MAX(4KB) = 67 MB/s;
+ MAX(1KB) = 18 MB/s;
+
+
+Also, the average chunk directly relates to the number of chunks produced by
+a backup::
+
+ CHUNK_COUNT = BACKUP_SIZE / ACS
+
+Here are some staticics from my developer worstation::
+
+ Disk Usage: 65 GB
+ Directories: 58971
+ Files: 726314
+ Files < 64KB: 617541
+
+As you see, there are really many small files. If we would do file
+level deduplication, i.e. generate one chunk per file, we end up with
+more than 700000 chunks.
+
+Instead, our current algorithm only produce large chunks with an
+average chunks size of 4MB. With above data, this produce about 15000
+chunks (factor 50 less chunks).
projects / proxmox-backup.git / blobdiff