4 .. The administration guide.
5 .. todo:: either add a bit more explanation or remove the previous sentence
13 When doing deduplication, there are different strategies to get
14 optimal results in terms of performance and/or deduplication rates.
15 Depending on the type of data, it can be split into *fixed* or *variable*
18 Fixed sized chunking requires minimal CPU power, and is used to
19 backup virtual machine images.
21 Variable sized chunking needs more CPU power, but is essential to get
22 good deduplication rates for file archives.
24 The Proxmox Backup Server supports both strategies.
27 File Archives: ``<name>.pxar``
28 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
30 .. see https://moinakg.wordpress.com/2013/06/22/high-performance-content-defined-chunking/
32 A file archive stores a full directory tree. Content is stored using
33 the :ref:`pxar-format`, split into variable-sized chunks. The format
34 is optimized to achieve good deduplication rates.
37 Image Archives: ``<name>.img``
38 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
40 This is used for virtual machine images and other large binary
41 data. Content is split into fixed-sized chunks.
47 This type is used to store smaller (< 16MB) binary data such as
48 configuration files. Larger files should be stored as image archive.
50 .. caution:: Please do not store all files as BLOBs. Instead, use the
51 file archive to store whole directory trees.
54 Catalog File: ``catalog.pcat1``
55 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
57 The catalog file is an index for file archives. It contains
58 the list of files and is used to speed up search operations.
61 The Manifest: ``index.json``
62 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
64 The manifest contains the list of all backup files, their
65 sizes and checksums. It is used to verify the consistency of a
72 The backup server groups backups by *type*, where *type* is one of:
75 This type is used for :term:`virtual machine`\ s. Typically
76 consists of the virtual machine's configuration file and an image archive
80 This type is used for :term:`container`\ s. Consists of the container's
81 configuration and a single file archive for the filesystem content.
84 This type is used for backups created from within the backed up machine.
85 Typically this would be a physical host but could also be a virtual machine
86 or container. Such backups may contain file and image archives, there are no restrictions in this regard.
92 A unique ID. Usually the virtual machine or container ID. ``host``
93 type backups normally use the hostname.
99 The time when the backup was made.
105 The tuple ``<type>/<ID>`` is called a backup group. Such a group
106 may contain one or more backup snapshots.
112 The triplet ``<type>/<ID>/<time>`` is called a backup snapshot. It
113 uniquely identifies a specific backup within a datastore.
115 .. code-block:: console
116 :caption: Backup Snapshot Examples
118 vm/104/2019-10-09T08:01:06Z
119 host/elsa/2019-11-08T09:48:14Z
121 As you can see, the time format is RFC3399_ with Coordinated
122 Universal Time (UTC_, identified by the trailing *Z*).
124 Backup Server Management
125 ------------------------
127 The command line tool to configure and manage the backup server is called
128 :command:`proxmox-backup-manager`.
135 A datastore is a place where backups are stored. The current implementation
136 uses a directory inside a standard unix file system (``ext4``, ``xfs``
137 or ``zfs``) to store the backup data.
139 Datastores are identified by a simple *ID*. You can configure it
140 when setting up the backup server.
142 .. note:: The `File Layout`_ requires the file system to support at least *65538*
143 subdirectories per directory. That number comes from the 2\ :sup:`16`
144 pre-created chunk namespace directories, and the ``.`` and ``..`` default
145 directory entries. This requirement excludes certain filesystems and
146 filesystem configuration from being supported for a datastore. For example,
147 ``ext3`` as a whole or ``ext4`` with the ``dir_nlink`` feature manually disabled.
151 Proxmox Backup Server comes with a set of disk utilities, which are
152 accessed using the ``disk`` subcommand. This subcommand allows you to initialize
153 disks, create various filesystems, and get information about the disks.
155 To view the disks connected to the system, use the ``list`` subcommand of
158 .. code-block:: console
160 # proxmox-backup-manager disk list
161 ┌──────┬────────┬─────┬───────────┬─────────────┬───────────────┬─────────┬────────┐
162 │ name │ used │ gpt │ disk-type │ size │ model │ wearout │ status │
163 ╞══════╪════════╪═════╪═══════════╪═════════════╪═══════════════╪═════════╪════════╡
164 │ sda │ lvm │ 1 │ hdd │ 34359738368 │ QEMU_HARDDISK │ - │ passed │
165 ├──────┼────────┼─────┼───────────┼─────────────┼───────────────┼─────────┼────────┤
166 │ sdb │ unused │ 1 │ hdd │ 68719476736 │ QEMU_HARDDISK │ - │ passed │
167 ├──────┼────────┼─────┼───────────┼─────────────┼───────────────┼─────────┼────────┤
168 │ sdc │ unused │ 1 │ hdd │ 68719476736 │ QEMU_HARDDISK │ - │ passed │
169 └──────┴────────┴─────┴───────────┴─────────────┴───────────────┴─────────┴────────┘
171 To initialize a disk with a new GPT, use the ``initialize`` subcommand:
173 .. code-block:: console
175 # proxmox-backup-manager disk initialize sdX
177 You can create an ``ext4`` or ``xfs`` filesystem on a disk, using ``fs
178 create``. The following command creates an ``ext4`` filesystem and passes the
179 ``--add-datastore`` parameter, in order to automatically create a datastore on
180 the disk (in this case ``sdd``). This will create a datastore at the location
181 ``/mnt/datastore/store1``:
183 .. code-block:: console
185 # proxmox-backup-manager disk fs create store1 --disk sdd --filesystem ext4 --add-datastore true
186 register worker thread
188 FILE: "/var/log/proxmox-backup/tasks/EA/UPID:pbs:000016F6:001851EA:00000000:5F3D0A26:dircreate:store1:root@pam:"
189 create datastore 'store1' on disk sdd
194 Detected stopped UPID UPID:pbs:000016F6:001851EA:00000000:5F3D0A26:dircreate:store1:root@pam:
197 You can also create a ``zpool`` with various raid levels. The command below
198 creates a mirrored ``zpool`` using two disks (``sdb`` & ``sdc``) and mounts it
199 on the root directory (default):
201 .. code-block:: console
203 # proxmox-backup-manager disk zpool create zpool1 --devices sdb,sdc --raidlevel mirror
204 register worker thread
206 FILE: "/var/log/proxmox-backup/tasks/F5/UPID:pbs:00001544:001814F5:00000000:5F3D098A:zfscreate:zpool1:root@pam:"
207 create Mirror zpool 'zpool1' on devices 'sdb,sdc'
208 # "zpool" "create" "-o" "ashift=12" "zpool1" "mirror" "sdb" "sdc"
211 Detected stopped UPID UPID:pbs:00001544:001814F5:00000000:5F3D098A:zfscreate:zpool1:root@pam:
215 You can also pass the ``--add-datastore`` parameter here, to automatically
216 create a datastore from the disk.
218 You can use ``disk fs list`` and ``disk zpool list`` to keep track of your
219 filesystems and zpools respectively.
221 If a disk supports S.M.A.R.T. capability, and you have this enabled, you can
222 display S.M.A.R.T. attributes using the command:
224 .. code-block:: console
226 # proxmox-backup-manager disk smart-attributes sdX
228 Datastore Configuration
229 ~~~~~~~~~~~~~~~~~~~~~~~
231 You can configure multiple datastores. Minimum one datastore needs to be
232 configured. The datastore is identified by a simple `name` and points to a
233 directory on the filesystem. Each datastore also has associated retention
234 settings of how many backup snapshots for each interval of ``hourly``,
235 ``daily``, ``weekly``, ``monthly``, ``yearly`` as well as a time-independent
236 number of backups to keep in that store. :ref:`Pruning <pruning>` and
237 :ref:`garbage collection <garbage-collection>` can also be configured to run
238 periodically based on a configured :term:`schedule` per datastore.
240 The following command creates a new datastore called ``store1`` on :file:`/backup/disk1/store1`
242 .. code-block:: console
244 # proxmox-backup-manager datastore create store1 /backup/disk1/store1
246 To list existing datastores run:
248 .. code-block:: console
250 # proxmox-backup-manager datastore list
251 ┌────────┬──────────────────────┬─────────────────────────────┐
252 │ name │ path │ comment │
253 ╞════════╪══════════════════════╪═════════════════════════════╡
254 │ store1 │ /backup/disk1/store1 │ This is my default storage. │
255 └────────┴──────────────────────┴─────────────────────────────┘
257 You can change settings of a datastore, for example to set a prune and garbage
258 collection schedule or retention settings using ``update`` subcommand and view
259 a datastore with the ``show`` subcommand:
261 .. code-block:: console
263 # proxmox-backup-manager datastore update store1 --keep-last 7 --prune-schedule daily --gc-schedule 'Tue 04:27'
264 # proxmox-backup-manager datastore show store1
265 ┌────────────────┬─────────────────────────────┐
267 ╞════════════════╪═════════════════════════════╡
269 ├────────────────┼─────────────────────────────┤
270 │ path │ /backup/disk1/store1 │
271 ├────────────────┼─────────────────────────────┤
272 │ comment │ This is my default storage. │
273 ├────────────────┼─────────────────────────────┤
274 │ gc-schedule │ Tue 04:27 │
275 ├────────────────┼─────────────────────────────┤
277 ├────────────────┼─────────────────────────────┤
278 │ prune-schedule │ daily │
279 └────────────────┴─────────────────────────────┘
281 Finally, it is possible to remove the datastore configuration:
283 .. code-block:: console
285 # proxmox-backup-manager datastore remove store1
287 .. note:: The above command removes only the datastore configuration. It does
288 not delete any data from the underlying directory.
294 After creating a datastore, the following default layout will appear:
296 .. code-block:: console
298 # ls -arilh /backup/disk1/store1
299 276493 -rw-r--r-- 1 backup backup 0 Jul 8 12:35 .lock
300 276490 drwxr-x--- 1 backup backup 1064960 Jul 8 12:35 .chunks
302 `.lock` is an empty file used for process locking.
304 The `.chunks` directory contains folders, starting from `0000` and taking hexadecimal values until `ffff`. These
305 directories will store the chunked data after a backup operation has been executed.
307 .. code-block:: console
309 # ls -arilh /backup/disk1/store1/.chunks
310 545824 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 ffff
311 545823 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 fffe
312 415621 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 fffd
313 415620 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 fffc
314 353187 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 fffb
315 344995 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 fffa
316 144079 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 fff9
317 144078 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 fff8
318 144077 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 fff7
320 403180 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 000c
321 403179 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 000b
322 403177 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 000a
323 402530 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 0009
324 402513 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 0008
325 402509 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 0007
326 276509 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 0006
327 276508 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 0005
328 276507 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 0004
329 276501 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 0003
330 276499 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 0002
331 276498 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 0001
332 276494 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 0000
333 276489 drwxr-xr-x 3 backup backup 4.0K Jul 8 12:35 ..
334 276490 drwxr-x--- 1 backup backup 1.1M Jul 8 12:35 .
341 Proxmox Backup Server supports several authentication realms, and you need to
342 choose the realm when you add a new user. Possible realms are:
344 :pam: Linux PAM standard authentication. Use this if you want to
345 authenticate as Linux system user (Users need to exist on the
348 :pbs: Proxmox Backup Server realm. This type stores hashed passwords in
349 ``/etc/proxmox-backup/shadow.json``.
351 After installation, there is a single user ``root@pam``, which
352 corresponds to the Unix superuser. You can use the
353 ``proxmox-backup-manager`` command line tool to list or manipulate
356 .. code-block:: console
358 # proxmox-backup-manager user list
359 ┌─────────────┬────────┬────────┬───────────┬──────────┬────────────────┬────────────────────┐
360 │ userid │ enable │ expire │ firstname │ lastname │ email │ comment │
361 ╞═════════════╪════════╪════════╪═══════════╪══════════╪════════════════╪════════════════════╡
362 │ root@pam │ 1 │ │ │ │ │ Superuser │
363 └─────────────┴────────┴────────┴───────────┴──────────┴────────────────┴────────────────────┘
365 The superuser has full administration rights on everything, so you
366 normally want to add other users with less privileges:
368 .. code-block:: console
370 # proxmox-backup-manager user create john@pbs --email john@example.com
372 The create command lets you specify many options like ``--email`` or
373 ``--password``. You can update or change any of them using the
374 update command later:
376 .. code-block:: console
378 # proxmox-backup-manager user update john@pbs --firstname John --lastname Smith
379 # proxmox-backup-manager user update john@pbs --comment "An example user."
381 .. todo:: Mention how to set password without passing plaintext password as cli argument.
384 The resulting user list looks like this:
386 .. code-block:: console
388 # proxmox-backup-manager user list
389 ┌──────────┬────────┬────────┬───────────┬──────────┬──────────────────┬──────────────────┐
390 │ userid │ enable │ expire │ firstname │ lastname │ email │ comment │
391 ╞══════════╪════════╪════════╪═══════════╪══════════╪══════════════════╪══════════════════╡
392 │ john@pbs │ 1 │ │ John │ Smith │ john@example.com │ An example user. │
393 ├──────────┼────────┼────────┼───────────┼──────────┼──────────────────┼──────────────────┤
394 │ root@pam │ 1 │ │ │ │ │ Superuser │
395 └──────────┴────────┴────────┴───────────┴──────────┴──────────────────┴──────────────────┘
397 Newly created users do not have any permissions. Please read the next
398 section to learn how to set access permissions.
400 If you want to disable a user account, you can do that by setting ``--enable`` to ``0``
402 .. code-block:: console
404 # proxmox-backup-manager user update john@pbs --enable 0
406 Or completely remove the user with:
408 .. code-block:: console
410 # proxmox-backup-manager user remove john@pbs
416 By default new users do not have any permission. Instead you need to
417 specify what is allowed and what is not. You can do this by assigning
418 roles to users on specific objects like datastores or remotes. The
419 following roles exist:
422 Disable Access - nothing is allowed.
428 Can view things, but is not allowed to change settings.
431 Can do anything on datastores.
434 Can view datastore settings and list content. But
435 is not allowed to read the actual data.
438 Can Inspect datastore content and can do restores.
441 Can backup and restore owned backups.
443 **DatastorePowerUser**
444 Can backup, restore, and prune owned backups.
447 Can do anything on remotes.
450 Can view remote settings.
452 **RemoteSyncOperator**
453 Is allowed to read data from a remote.
455 You can use the ``acl`` subcommand to manage and monitor user permissions. For
456 example, the command below will add the user ``john@pbs`` as a
457 **DatastoreAdmin** for the data store ``store1``, located at ``/backup/disk1/store1``:
459 .. code-block:: console
461 # proxmox-backup-manager acl update /datastore/store1 DatastoreAdmin --userid john@pbs
463 You can monitor the roles of each user using the following command:
465 .. code-block:: console
467 # proxmox-backup-manager acl list
468 ┌──────────┬──────────────────┬───────────┬────────────────┐
469 │ ugid │ path │ propagate │ roleid │
470 ╞══════════╪══════════════════╪═══════════╪════════════════╡
471 │ john@pbs │ /datastore/disk1 │ 1 │ DatastoreAdmin │
472 └──────────┴──────────────────┴───────────┴────────────────┘
474 A single user can be assigned multiple permission sets for different data stores.
477 Naming convention is important here. For data stores on the host,
478 you must use the convention ``/datastore/{storename}``. For example, to set
479 permissions for a data store mounted at ``/mnt/backup/disk4/store2``, you would use
480 ``/datastore/store2`` for the path. For remote stores, use the convention
481 ``/remote/{remote}/{storename}``, where ``{remote}`` signifies the name of the
482 remote (see `Remote` below) and ``{storename}`` is the name of the data store on
488 A remote refers to a separate Proxmox Backup Server installation and a user on that
489 installation, from which you can `sync` datastores to a local datastore with a
492 To add a remote, you need its hostname or ip, a userid and password on the
493 remote, and its certificate fingerprint. To get the fingerprint, use the
494 ``proxmox-backup-manager cert info`` command on the remote.
496 .. code-block:: console
498 # proxmox-backup-manager cert info |grep Fingerprint
499 Fingerprint (sha256): 64:d3:ff:3a:50:38:53:5a:9b:f7:50:...:ab:fe
501 Using the information specified above, add the remote with:
503 .. code-block:: console
505 # proxmox-backup-manager remote create pbs2 --host pbs2.mydomain.example --userid sync@pam --password 'SECRET' --fingerprint 64:d3:ff:3a:50:38:53:5a:9b:f7:50:...:ab:fe
507 Use the ``list``, ``show``, ``update``, ``remove`` subcommands of
508 ``proxmox-backup-manager remote`` to manage your remotes:
510 .. code-block:: console
512 # proxmox-backup-manager remote update pbs2 --host pbs2.example
513 # proxmox-backup-manager remote list
514 ┌──────┬──────────────┬──────────┬───────────────────────────────────────────┬─────────┐
515 │ name │ host │ userid │ fingerprint │ comment │
516 ╞══════╪══════════════╪══════════╪═══════════════════════════════════════════╪═════════╡
517 │ pbs2 │ pbs2.example │ sync@pam │64:d3:ff:3a:50:38:53:5a:9b:f7:50:...:ab:fe │ │
518 └──────┴──────────────┴──────────┴───────────────────────────────────────────┴─────────┘
519 # proxmox-backup-manager remote remove pbs2
525 Sync jobs are configured to pull the contents of a datastore on a `Remote` to a
526 local datastore. You can either start the sync job manually on the GUI or
527 provide it with a :term:`schedule` to run regularly. The
528 ``proxmox-backup-manager sync-job`` command is used to manage sync jobs:
530 .. code-block:: console
532 # proxmox-backup-manager sync-job create pbs2-local --remote pbs2 --remote-store local --store local --schedule 'Wed 02:30'
533 # proxmox-backup-manager sync-job update pbs2-local --comment 'offsite'
534 # proxmox-backup-manager sync-job list
535 ┌────────────┬───────┬────────┬──────────────┬───────────┬─────────┐
536 │ id │ store │ remote │ remote-store │ schedule │ comment │
537 ╞════════════╪═══════╪════════╪══════════════╪═══════════╪═════════╡
538 │ pbs2-local │ local │ pbs2 │ local │ Wed 02:30 │ offsite │
539 └────────────┴───────┴────────┴──────────────┴───────────┴─────────┘
540 # proxmox-backup-manager sync-job remove pbs2-local
546 The command line client is called :command:`proxmox-backup-client`.
552 The client uses the following notation to specify a datastore repository
553 on the backup server.
555 [[username@]server:]datastore
557 The default value for ``username`` ist ``root``. If no server is specified,
558 the default is the local host (``localhost``).
560 You can pass the repository with the ``--repository`` command
561 line option, or by setting the ``PBS_REPOSITORY`` environment
565 Environment Variables
566 ~~~~~~~~~~~~~~~~~~~~~
569 The default backup repository.
572 When set, this value is used for the password required for the
575 ``PBS_ENCRYPTION_PASSWORD``
576 When set, this value is used to access the secret encryption key (if
577 protected by password).
579 ``PBS_FINGERPRINT`` When set, this value is used to verify the server
580 certificate (only used if the system CA certificates cannot
581 validate the certificate).
587 Most commands support the ``--output-format`` parameter. It accepts
588 the following values:
590 :``text``: Text format (default). Structured data is rendered as a table.
592 :``json``: JSON (single line).
594 :``json-pretty``: JSON (multiple lines, nicely formatted).
597 Please use the following environment variables to modify output behavior:
599 ``PROXMOX_OUTPUT_FORMAT``
600 Defines the default output format.
602 ``PROXMOX_OUTPUT_NO_BORDER``
603 If set (to any value), do not render table borders.
605 ``PROXMOX_OUTPUT_NO_HEADER``
606 If set (to any value), do not render table headers.
608 .. note:: The ``text`` format is designed to be human readable, and
609 not meant to be parsed by automation tools. Please use the ``json``
610 format if you need to process the output.
613 .. _creating-backups:
618 This section explains how to create a backup from within the machine. This can
619 be a physical host, a virtual machine, or a container. Such backups may contain file
620 and image archives. There are no restrictions in this case.
622 .. note:: If you want to backup virtual machines or containers on Proxmox VE, see :ref:`pve-integration`.
624 For the following example you need to have a backup server set up, working
625 credentials and need to know the repository name.
626 In the following examples we use ``backup-server:store1``.
628 .. code-block:: console
630 # proxmox-backup-client backup root.pxar:/ --repository backup-server:store1
631 Starting backup: host/elsa/2019-12-03T09:35:01Z
633 skip mount point: "/boot/efi"
634 skip mount point: "/dev"
635 skip mount point: "/run"
636 skip mount point: "/sys"
637 Uploaded 12129 chunks in 87 seconds (564 MB/s).
638 End Time: 2019-12-03T10:36:29+01:00
640 This will prompt you for a password and then uploads a file archive named
641 ``root.pxar`` containing all the files in the ``/`` directory.
643 .. Caution:: Please note that the proxmox-backup-client does not
644 automatically include mount points. Instead, you will see a short
645 ``skip mount point`` notice for each of them. The idea is to
646 create a separate file archive for each mounted disk. You can
647 explicitly include them using the ``--include-dev`` option
648 (i.e. ``--include-dev /boot/efi``). You can use this option
649 multiple times for each mount point that should be included.
651 The ``--repository`` option can get quite long and is used by all
652 commands. You can avoid having to enter this value by setting the
653 environment variable ``PBS_REPOSITORY``. Note that if you would like this to remain set
654 over multiple sessions, you should instead add the below line to your
657 .. code-block:: console
659 # export PBS_REPOSITORY=backup-server:store1
661 After this you can execute all commands without specifying the ``--repository``
664 One single backup is allowed to contain more than one archive. For example, if
665 you want to backup two disks mounted at ``/mmt/disk1`` and ``/mnt/disk2``:
667 .. code-block:: console
669 # proxmox-backup-client backup disk1.pxar:/mnt/disk1 disk2.pxar:/mnt/disk2
671 This creates a backup of both disks.
673 The backup command takes a list of backup specifications, which
674 include the archive name on the server, the type of the archive, and the
675 archive source at the client. The format is:
677 <archive-name>.<type>:<source-path>
679 Common types are ``.pxar`` for file archives, and ``.img`` for block
680 device images. To create a backup of a block device run the following command:
682 .. code-block:: console
684 # proxmox-backup-client backup mydata.img:/dev/mylvm/mydata
686 Excluding files/folders from a backup
687 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
689 Sometimes it is desired to exclude certain files or folders from a backup archive.
690 To tell the Proxmox Backup client when and how to ignore files and directories,
691 place a text file called ``.pxarexclude`` in the filesystem hierarchy.
692 Whenever the backup client encounters such a file in a directory, it interprets
693 each line as glob match patterns for files and directories that are to be excluded
696 The file must contain a single glob pattern per line. Empty lines are ignored.
697 The same is true for lines starting with ``#``, which indicates a comment.
698 A ``!`` at the beginning of a line reverses the glob match pattern from an exclusion
699 to an explicit inclusion. This makes it possible to exclude all entries in a
700 directory except for a few single files/subdirectories.
701 Lines ending in ``/`` match only on directories.
702 The directory containing the ``.pxarexclude`` file is considered to be the root of
703 the given patterns. It is only possible to match files in this directory and its subdirectories.
705 ``\`` is used to escape special glob characters.
706 ``?`` matches any single character.
707 ``*`` matches any character, including an empty string.
708 ``**`` is used to match subdirectories. It can be used to, for example, exclude
709 all files ending in ``.tmp`` within the directory or subdirectories with the
710 following pattern ``**/*.tmp``.
711 ``[...]`` matches a single character from any of the provided characters within
712 the brackets. ``[!...]`` does the complementary and matches any single character
713 not contained within the brackets. It is also possible to specify ranges with two
714 characters separated by ``-``. For example, ``[a-z]`` matches any lowercase
715 alphabetic character and ``[0-9]`` matches any one single digit.
717 The order of the glob match patterns defines whether a file is included or
718 excluded, that is to say later entries override previous ones.
719 This is also true for match patterns encountered deeper down the directory tree,
720 which can override a previous exclusion.
721 Be aware that excluded directories will **not** be read by the backup client.
722 Thus, a ``.pxarexclude`` file in an excluded subdirectory will have no effect.
723 ``.pxarexclude`` files are treated as regular files and will be included in the
726 For example, consider the following directory structure:
728 .. code-block:: console
732 . .. .pxarexclude subfolder0 subfolder1
735 . .. file0 file1 file2 file3 .pxarexclude
738 . .. file0 file1 file2 file3
740 The different ``.pxarexclude`` files contain the following:
742 .. code-block:: console
744 # cat folder/.pxarexclude
749 .. code-block:: console
751 # cat folder/subfolder0/.pxarexclude
754 This would exclude ``file1`` and ``file3`` in ``subfolder0`` and all of
755 ``subfolder1`` except ``file2``.
757 Restoring this backup will result in:
759 .. code-block:: console
763 . .. .pxarexclude subfolder0 subfolder1
766 . .. file0 file2 .pxarexclude
774 Proxmox Backup supports client-side encryption with AES-256 in GCM_
775 mode. To set this up, you first need to create an encryption key:
777 .. code-block:: console
779 # proxmox-backup-client key create my-backup.key
780 Encryption Key Password: **************
782 The key is password protected by default. If you do not need this
783 extra protection, you can also create it without a password:
785 .. code-block:: console
787 # proxmox-backup-client key create /path/to/my-backup.key --kdf none
789 Having created this key, it is now possible to create an encrypted backup, by
790 passing the ``--keyfile`` parameter, with the path to the key file.
792 .. code-block:: console
794 # proxmox-backup-client backup etc.pxar:/etc --keyfile /path/to/my-backup.key
796 Encryption Key Password: **************
799 .. Note:: If you do not specify the name of the backup key, the key will be
800 created in the default location
801 ``~/.config/proxmox-backup/encryption-key.json``. ``proxmox-backup-client``
802 will also search this location by default, in case the ``--keyfile``
803 parameter is not specified.
805 You can avoid entering the passwords by setting the environment
806 variables ``PBS_PASSWORD`` and ``PBS_ENCRYPTION_PASSWORD``.
808 Using a master key to store and recover encryption keys
809 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
811 You can also use ``proxmox-backup-client key`` to create an RSA public/private
812 key pair, which can be used to store an encrypted version of the symmetric
813 backup encryption key alongside each backup and recover it later.
815 To set up a master key:
817 1. Create an encryption key for the backup:
819 .. code-block:: console
821 # proxmox-backup-client key create
822 creating default key at: "~/.config/proxmox-backup/encryption-key.json"
823 Encryption Key Password: **********
826 The resulting file will be saved to ``~/.config/proxmox-backup/encryption-key.json``.
828 2. Create an RSA public/private key pair:
830 .. code-block:: console
832 # proxmox-backup-client key create-master-key
833 Master Key Password: *********
836 This will create two files in your current directory, ``master-public.pem``
837 and ``master-private.pem``.
839 3. Import the newly created ``master-public.pem`` public certificate, so that
840 ``proxmox-backup-client`` can find and use it upon backup.
842 .. code-block:: console
844 # proxmox-backup-client key import-master-pubkey /path/to/master-public.pem
845 Imported public master key to "~/.config/proxmox-backup/master-public.pem"
847 4. With all these files in place, run a backup job:
849 .. code-block:: console
851 # proxmox-backup-client backup etc.pxar:/etc
853 The key will be stored in your backup, under the name ``rsa-encrypted.key``.
855 .. Note:: The ``--keyfile`` parameter can be excluded, if the encryption key
856 is in the default path. If you specified another path upon creation, you
857 must pass the ``--keyfile`` parameter.
859 5. To test that everything worked, you can restore the key from the backup:
861 .. code-block:: console
863 # proxmox-backup-client restore /path/to/backup/ rsa-encrypted.key /path/to/target
865 .. Note:: You should not need an encryption key to extract this file. However, if
866 a key exists at the default location
867 (``~/.config/proxmox-backup/encryption-key.json``) the program will prompt
868 you for an encryption key password. Simply moving ``encryption-key.json``
869 out of this directory will fix this issue.
871 6. Then, use the previously generated master key to decrypt the file:
873 .. code-block:: console
875 # openssl rsautl -decrypt -inkey master-private.pem -in rsa-encrypted.key -out /path/to/target
876 Enter pass phrase for ./master-private.pem: *********
878 7. The target file will now contain the encryption key information in plain
879 text. The success of this can be confirmed by passing the resulting ``json``
880 file, with the ``--keyfile`` parameter, when decrypting files from the backup.
882 .. warning:: Without their key, backed up files will be inaccessible. Thus, you should
883 keep keys ordered and in a place that is separate from the contents being
884 backed up. It can happen, for example, that you back up an entire system, using
885 a key on that system. If the system then becomes inaccessable for any reason
886 and needs to be restored, this will not be possible as the encryption key will be
887 lost along with the broken system. In preparation for the worst case scenario,
888 you should consider keeping a paper copy of this key locked away in
894 The regular creation of backups is a necessary step to avoiding data
895 loss. More importantly, however, is the restoration. It is good practice to perform
896 periodic recovery tests to ensure that you can access the data in
899 First, you need to find the snapshot which you want to restore. The snapshot
900 command provides a list of all the snapshots on the server:
902 .. code-block:: console
904 # proxmox-backup-client snapshots
905 ┌────────────────────────────────┬─────────────┬────────────────────────────────────┐
906 │ snapshot │ size │ files │
907 ╞════════════════════════════════╪═════════════╪════════════════════════════════════╡
908 │ host/elsa/2019-12-03T09:30:15Z │ 51788646825 │ root.pxar catalog.pcat1 index.json │
909 ├────────────────────────────────┼─────────────┼────────────────────────────────────┤
910 │ host/elsa/2019-12-03T09:35:01Z │ 51790622048 │ root.pxar catalog.pcat1 index.json │
911 ├────────────────────────────────┼─────────────┼────────────────────────────────────┤
914 You can inspect the catalog to find specific files.
916 .. code-block:: console
918 # proxmox-backup-client catalog dump host/elsa/2019-12-03T09:35:01Z
920 d "./root.pxar.didx/etc/cifs-utils"
921 l "./root.pxar.didx/etc/cifs-utils/idmap-plugin"
922 d "./root.pxar.didx/etc/console-setup"
925 The restore command lets you restore a single archive from the
928 .. code-block:: console
930 # proxmox-backup-client restore host/elsa/2019-12-03T09:35:01Z root.pxar /target/path/
932 To get the contents of any archive, you can restore the ``index.json`` file in the
933 repository to the target path '-'. This will dump the contents to the standard output.
935 .. code-block:: console
937 # proxmox-backup-client restore host/elsa/2019-12-03T09:35:01Z index.json -
943 If you only want to restore a few individual files, it is often easier
944 to use the interactive recovery shell.
946 .. code-block:: console
948 # proxmox-backup-client catalog shell host/elsa/2019-12-03T09:35:01Z root.pxar
949 Starting interactive shell
951 bin boot dev etc home lib lib32
954 The interactive recovery shell is a minimalistic command line interface that
955 utilizes the metadata stored in the catalog to quickly list, navigate and
956 search files in a file archive.
957 To restore files, you can select them individually or match them with a glob
960 Using the catalog for navigation reduces the overhead considerably because only
961 the catalog needs to be downloaded and, optionally, decrypted.
962 The actual chunks are only accessed if the metadata in the catalog is not enough
963 or for the actual restore.
965 Similar to common UNIX shells ``cd`` and ``ls`` are the commands used to change
966 working directory and list directory contents in the archive.
967 ``pwd`` shows the full path of the current working directory with respect to the
970 Being able to quickly search the contents of the archive is a commmonly needed feature.
971 That's where the catalog is most valuable.
974 .. code-block:: console
976 pxar:/ > find etc/**/*.txt --select
978 pxar:/ > list-selected
980 pxar:/ > restore-selected /target/path
983 This will find and print all files ending in ``.txt`` located in ``etc/`` or a
984 subdirectory and add the corresponding pattern to the list for subsequent restores.
985 ``list-selected`` shows these patterns and ``restore-selected`` finally restores
986 all files in the archive matching the patterns to ``/target/path`` on the local
987 host. This will scan the whole archive.
989 With ``restore /target/path`` you can restore the sub-archive given by the current
990 working directory to the local target path ``/target/path`` on your host.
991 By additionally passing a glob pattern with ``--pattern <glob>``, the restore is
992 further limited to files matching the pattern.
995 .. code-block:: console
998 pxar:/etc/ > restore /target/ --pattern **/*.conf
1001 The above will scan trough all the directories below ``/etc`` and restore all
1002 files ending in ``.conf``.
1004 .. todo:: Explain interactive restore in more detail
1006 Mounting of Archives via FUSE
1007 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
1009 The :term:`FUSE` implementation for the pxar archive allows you to mount a
1010 file archive as a read-only filesystem to a mountpoint on your host.
1012 .. code-block:: console
1014 # proxmox-backup-client mount host/backup-client/2020-01-29T11:29:22Z root.pxar /mnt/mountpoint
1015 # ls /mnt/mountpoint
1016 bin dev home lib32 libx32 media opt root sbin sys usr
1017 boot etc lib lib64 lost+found mnt proc run srv tmp var
1019 This allows you to access the full contents of the archive in a seamless manner.
1021 .. note:: As the FUSE connection needs to fetch and decrypt chunks from the
1022 backup server's datastore, this can cause some additional network and CPU
1023 load on your host, depending on the operations you perform on the mounted
1026 To unmount the filesystem use the ``umount`` command on the mountpoint:
1028 .. code-block:: console
1030 # umount /mnt/mountpoint
1035 The client tool prompts you to enter the logon password as soon as you
1036 want to access the backup server. The server checks your credentials
1037 and responds with a ticket that is valid for two hours. The client
1038 tool automatically stores that ticket and uses it for further requests
1041 You can also manually trigger this login/logout using the login and
1044 .. code-block:: console
1046 # proxmox-backup-client login
1047 Password: **********
1049 To remove the ticket, issue a logout:
1051 .. code-block:: console
1053 # proxmox-backup-client logout
1058 Pruning and Removing Backups
1059 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1061 You can manually delete a backup snapshot using the ``forget``
1064 .. code-block:: console
1066 # proxmox-backup-client forget <snapshot>
1069 .. caution:: This command removes all archives in this backup
1070 snapshot. They will be inaccessible and unrecoverable.
1073 Although manual removal is sometimes required, the ``prune``
1074 command is normally used to systematically delete older backups. Prune lets
1075 you specify which backup snapshots you want to keep. The
1076 following retention options are available:
1079 Keep the last ``<N>`` backup snapshots.
1081 ``--keep-hourly <N>``
1082 Keep backups for the last ``<N>`` hours. If there is more than one
1083 backup for a single hour, only the latest is kept.
1085 ``--keep-daily <N>``
1086 Keep backups for the last ``<N>`` days. If there is more than one
1087 backup for a single day, only the latest is kept.
1089 ``--keep-weekly <N>``
1090 Keep backups for the last ``<N>`` weeks. If there is more than one
1091 backup for a single week, only the latest is kept.
1093 .. note:: Weeks start on Monday and end on Sunday. The software
1094 uses the `ISO week date`_ system and handles weeks at
1095 the end of the year correctly.
1097 ``--keep-monthly <N>``
1098 Keep backups for the last ``<N>`` months. If there is more than one
1099 backup for a single month, only the latest is kept.
1101 ``--keep-yearly <N>``
1102 Keep backups for the last ``<N>`` years. If there is more than one
1103 backup for a single year, only the latest is kept.
1105 The retention options are processed in the order given above. Each option
1106 only covers backups within its time period. The next option does not take care
1107 of already covered backups. It will only consider older backups.
1109 Unfinished and incomplete backups will be removed by the prune command unless
1110 they are newer than the last successful backup. In this case, the last failed
1113 .. code-block:: console
1115 # proxmox-backup-client prune <group> --keep-daily 7 --keep-weekly 4 --keep-monthly 3
1118 You can use the ``--dry-run`` option to test your settings. This only
1119 shows the list of existing snapshots and what actions prune would take.
1121 .. code-block:: console
1123 # proxmox-backup-client prune host/elsa --dry-run --keep-daily 1 --keep-weekly 3
1124 ┌────────────────────────────────┬──────┐
1126 ╞════════════════════════════════╪══════╡
1127 │ host/elsa/2019-12-04T13:20:37Z │ 1 │
1128 ├────────────────────────────────┼──────┤
1129 │ host/elsa/2019-12-03T09:35:01Z │ 0 │
1130 ├────────────────────────────────┼──────┤
1131 │ host/elsa/2019-11-22T11:54:47Z │ 1 │
1132 ├────────────────────────────────┼──────┤
1133 │ host/elsa/2019-11-21T12:36:25Z │ 0 │
1134 ├────────────────────────────────┼──────┤
1135 │ host/elsa/2019-11-10T10:42:20Z │ 1 │
1136 └────────────────────────────────┴──────┘
1138 .. note:: Neither the ``prune`` command nor the ``forget`` command free space
1139 in the chunk-store. The chunk-store still contains the data blocks. To free
1140 space you need to perform :ref:`garbage-collection`.
1143 .. _garbage-collection:
1148 The ``prune`` command removes only the backup index files, not the data
1149 from the data store. This task is left to the garbage collection
1150 command. It is recommended to carry out garbage collection on a regular basis.
1152 The garbage collection works in two phases. In the first phase, all
1153 data blocks that are still in use are marked. In the second phase,
1154 unused data blocks are removed.
1156 .. note:: This command needs to read all existing backup index files
1157 and touches the complete chunk-store. This can take a long time
1158 depending on the number of chunks and the speed of the underlying
1161 .. note:: The garbage collection will only remove chunks that haven't been used
1162 for at least one day (exactly 24h 5m). This grace period is necessary because
1163 chunks in use are marked by touching the chunk which updates the ``atime``
1164 (access time) property. Filesystems are mounted with the ``relatime`` option
1165 by default. This results in a better performance by only updating the
1166 ``atime`` property if the last access has been at least 24 hours ago. The
1167 downside is, that touching a chunk within these 24 hours will not always
1168 update its ``atime`` property.
1170 Chunks in the grace period will be logged at the end of the garbage
1171 collection task as *Pending removals*.
1173 .. code-block:: console
1175 # proxmox-backup-client garbage-collect
1176 starting garbage collection on store store2
1177 Start GC phase1 (mark used chunks)
1178 Start GC phase2 (sweep unused chunks)
1179 percentage done: 1, chunk count: 219
1180 percentage done: 2, chunk count: 453
1182 percentage done: 99, chunk count: 21188
1183 Removed bytes: 411368505
1185 Original data bytes: 327160886391
1186 Disk bytes: 52767414743 (16 %)
1188 Average chunk size: 2486565
1192 .. todo:: howto run garbage-collection at regular intervalls (cron)
1196 The backup client also comes with a benchmarking tool. This tool measures
1197 various metrics relating to compression and encryption speeds. You can run a
1198 benchmark using the ``benchmark`` subcommand of ``proxmox-backup-client``:
1200 .. code-block:: console
1202 # proxmox-backup-client benchmark
1203 Uploaded 656 chunks in 5 seconds.
1204 Time per request: 7659 microseconds.
1205 TLS speed: 547.60 MB/s
1206 SHA256 speed: 585.76 MB/s
1207 Compression speed: 1923.96 MB/s
1208 Decompress speed: 7885.24 MB/s
1209 AES256/GCM speed: 3974.03 MB/s
1210 ┌───────────────────────────────────┬─────────────────────┐
1212 ╞═══════════════════════════════════╪═════════════════════╡
1213 │ TLS (maximal backup upload speed) │ 547.60 MB/s (93%) │
1214 ├───────────────────────────────────┼─────────────────────┤
1215 │ SHA256 checksum computation speed │ 585.76 MB/s (28%) │
1216 ├───────────────────────────────────┼─────────────────────┤
1217 │ ZStd level 1 compression speed │ 1923.96 MB/s (89%) │
1218 ├───────────────────────────────────┼─────────────────────┤
1219 │ ZStd level 1 decompression speed │ 7885.24 MB/s (98%) │
1220 ├───────────────────────────────────┼─────────────────────┤
1221 │ AES256 GCM encryption speed │ 3974.03 MB/s (104%) │
1222 └───────────────────────────────────┴─────────────────────┘
1224 You can also pass the ``--output-format`` parameter to output stats in ``json``,
1225 rather than the default table format.
1227 .. _pve-integration:
1229 `Proxmox VE`_ integration
1230 -------------------------
1232 You need to define a new storage with type 'pbs' on your `Proxmox VE`_
1233 node. The following example uses ``store2`` as storage name, and
1234 assumes the server address is ``localhost``, and you want to connect
1237 .. code-block:: console
1239 # pvesm add pbs store2 --server localhost --datastore store2
1240 # pvesm set store2 --username user1@pbs --password <secret>
1242 If your backup server uses a self signed certificate, you need to add
1243 the certificate fingerprint to the configuration. You can get the
1244 fingerprint by running the following command on the backup server:
1246 .. code-block:: console
1248 # proxmox-backup-manager cert info |grep Fingerprint
1249 Fingerprint (sha256): 64:d3:ff:3a:50:38:53:5a:9b:f7:50:...:ab:fe
1251 Please add that fingerprint to your configuration to establish a trust
1254 .. code-block:: console
1256 # pvesm set store2 --fingerprint 64:d3:ff:3a:50:38:53:5a:9b:f7:50:...:ab:fe
1258 After that you should be able to see storage status with:
1260 .. code-block:: console
1262 # pvesm status --storage store2
1263 Name Type Status Total Used Available %
1264 store2 pbs active 3905109820 1336687816 2568422004 34.23%
1268 .. include:: command-line-tools.rst
1270 .. include:: services.rst