[proxmox-backup.git] / README.rst

``rustup`` Toolchain
====================

We normally want to build with the ``rustc`` Debian package. To do that
you can set the following ``rustup`` configuration:

    # rustup toolchain link system /usr
    # rustup default system


Versioning of proxmox helper crates
===================================

To use current git master code of the proxmox* helper crates, add::

   git = "git://git.proxmox.com/git/proxmox"

or::

   path = "../proxmox/proxmox"

to the proxmox dependency, and update the version to reflect the current,
pre-release version number (e.g., "0.1.1-dev.1" instead of "0.1.0").


Local cargo config
==================

This repository ships with a ``.cargo/config`` that replaces the crates.io
registry with packaged crates located in ``/usr/share/cargo/registry``.

A similar config is also applied building with dh_cargo. Cargo.lock needs to be
deleted when switching between packaged crates and crates.io, since the
checksums are not compatible.

To reference new dependencies (or updated versions) that are not yet packaged,
the dependency needs to point directly to a path or git source (e.g., see
example for proxmox crate above).


Build
=====
on Debian Buster

Setup:
  1. # echo 'deb http://download.proxmox.com/debian/devel/ buster main' >> /etc/apt/sources.list.d/proxmox-devel.list
  2. # sudo wget http://download.proxmox.com/debian/proxmox-ve-release-6.x.gpg -O /etc/apt/trusted.gpg.d/proxmox-ve-release-6.x.gpg
  3. # sudo apt update
  4. # sudo apt install devscripts debcargo clang
  5. # git clone git://git.proxmox.com/git/proxmox-backup.git
  6. # sudo mk-build-deps -ir

Note: 2. may be skipped if you already added the PVE or PBS package repository

You are now able to build using the Makefile or cargo itself.


Design Notes
============

Here are some random thought about the software design (unless I find a better place).


Large chunk sizes
-----------------

It is important to notice that large chunk sizes are crucial for
performance. We have a multi-user system, where different people can do
different operations on a datastore at the same time, and most operation
involves reading a series of chunks.

So what is the maximal theoretical speed we can get when reading a
series of chunks? Reading a chunk sequence need the following steps:

- seek to the first chunk start location
- read the chunk data
- seek to the first chunk start location
- read the chunk data
- ...

Lets use the following disk performance metrics:

:AST: Average Seek Time (second)
:MRS: Maximum sequential Read Speed (bytes/second)
:ACS: Average Chunk Size (bytes)

The maximum performance you can get is::

  MAX(ACS) = ACS /(AST + ACS/MRS)

Please note that chunk data is likely to be sequential arranged on disk, but
this it is sort of a best case assumption.

For a typical rotational disk, we assume the following values::

  AST: 10ms
  MRS: 170MB/s

  MAX(4MB)  = 115.37 MB/s
  MAX(1MB)  =  61.85 MB/s;
  MAX(64KB) =   6.02 MB/s;
  MAX(4KB)  =   0.39 MB/s;
  MAX(1KB)  =   0.10 MB/s;

Modern SSD are much faster, lets assume the following::

  max IOPS: 20000 => AST = 0.00005
  MRS: 500Mb/s

  MAX(4MB)  = 474 MB/s
  MAX(1MB)  = 465 MB/s;
  MAX(64KB) = 354 MB/s;
  MAX(4KB)  =  67 MB/s;
  MAX(1KB)  =  18 MB/s;
Commit	Line	Data
	1	``rustup`` Toolchain
	2	====================
	3
	4	We normally want to build with the ``rustc`` Debian package. To do that
	5	you can set the following ``rustup`` configuration:
	6
	7	# rustup toolchain link system /usr
	8	# rustup default system
	9
	10
	11	Versioning of proxmox helper crates
	12	===================================
	13
	14	To use current git master code of the proxmox* helper crates, add::
	15
	16	git = "git://git.proxmox.com/git/proxmox"
	17
	18	or::
	19
	20	path = "../proxmox/proxmox"
	21
	22	to the proxmox dependency, and update the version to reflect the current,
	23	pre-release version number (e.g., "0.1.1-dev.1" instead of "0.1.0").
	24
	25
	26	Local cargo config
	27	==================
	28
	29	This repository ships with a ``.cargo/config`` that replaces the crates.io
	30	registry with packaged crates located in ``/usr/share/cargo/registry``.
	31
	32	A similar config is also applied building with dh_cargo. Cargo.lock needs to be
	33	deleted when switching between packaged crates and crates.io, since the
	34	checksums are not compatible.
	35
	36	To reference new dependencies (or updated versions) that are not yet packaged,
	37	the dependency needs to point directly to a path or git source (e.g., see
	38	example for proxmox crate above).
	39
	40
	41	Build
	42	=====
	43	on Debian Buster
	44
	45	Setup:
	46	1. # echo 'deb http://download.proxmox.com/debian/devel/ buster main' >> /etc/apt/sources.list.d/proxmox-devel.list
	47	2. # sudo wget http://download.proxmox.com/debian/proxmox-ve-release-6.x.gpg -O /etc/apt/trusted.gpg.d/proxmox-ve-release-6.x.gpg
	48	3. # sudo apt update
	49	4. # sudo apt install devscripts debcargo clang
	50	5. # git clone git://git.proxmox.com/git/proxmox-backup.git
	51	6. # sudo mk-build-deps -ir
	52
	53	Note: 2. may be skipped if you already added the PVE or PBS package repository
	54
	55	You are now able to build using the Makefile or cargo itself.
	56
	57
	58	Design Notes
	59	============
	60
	61	Here are some random thought about the software design (unless I find a better place).
	62
	63
	64	Large chunk sizes
	65	-----------------
	66
	67	It is important to notice that large chunk sizes are crucial for
	68	performance. We have a multi-user system, where different people can do
	69	different operations on a datastore at the same time, and most operation
	70	involves reading a series of chunks.
	71
	72	So what is the maximal theoretical speed we can get when reading a
	73	series of chunks? Reading a chunk sequence need the following steps:
	74
	75	- seek to the first chunk start location
	76	- read the chunk data
	77	- seek to the first chunk start location
	78	- read the chunk data
	79	- ...
	80
	81	Lets use the following disk performance metrics:
	82
	83	:AST: Average Seek Time (second)
	84	:MRS: Maximum sequential Read Speed (bytes/second)
	85	:ACS: Average Chunk Size (bytes)
	86
	87	The maximum performance you can get is::
	88
	89	MAX(ACS) = ACS /(AST + ACS/MRS)
	90
	91	Please note that chunk data is likely to be sequential arranged on disk, but
	92	this it is sort of a best case assumption.
	93
	94	For a typical rotational disk, we assume the following values::
	95
	96	AST: 10ms
	97	MRS: 170MB/s
	98
	99	MAX(4MB) = 115.37 MB/s
	100	MAX(1MB) = 61.85 MB/s;
	101	MAX(64KB) = 6.02 MB/s;
	102	MAX(4KB) = 0.39 MB/s;
	103	MAX(1KB) = 0.10 MB/s;
	104
	105	Modern SSD are much faster, lets assume the following::
	106
	107	max IOPS: 20000 => AST = 0.00005
	108	MRS: 500Mb/s
	109
	110	MAX(4MB) = 474 MB/s
	111	MAX(1MB) = 465 MB/s;
	112	MAX(64KB) = 354 MB/s;
	113	MAX(4KB) = 67 MB/s;
	114	MAX(1KB) = 18 MB/s;