4 .. The administration guide.
5 .. todo:: either add a bit more explanation or remove the previous sentence
13 When doing deduplication, there are different strategies to get
14 optimal results in terms of performance and/or deduplication rates.
15 Depending on the type of data, it can be split into *fixed* or *variable*
18 Fixed sized chunking requires minimal CPU power, and is used to
19 backup virtual machine images.
21 Variable sized chunking needs more CPU power, but is essential to get
22 good deduplication rates for file archives.
24 The Proxmox Backup Server supports both strategies.
27 File Archives: ``<name>.pxar``
28 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
30 .. see https://moinakg.wordpress.com/2013/06/22/high-performance-content-defined-chunking/
32 A file archive stores a full directory tree. Content is stored using
33 the :ref:`pxar-format`, split into variable-sized chunks. The format
34 is optimized to achieve good deduplication rates.
37 Image Archives: ``<name>.img``
38 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
40 This is used for virtual machine images and other large binary
41 data. Content is split into fixed-sized chunks.
47 This type is used to store smaller (< 16MB) binary data such as
48 configuration files. Larger files should be stored as image archive.
50 .. caution:: Please do not store all files as BLOBs. Instead, use the
51 file archive to store whole directory trees.
54 Catalog File: ``catalog.pcat1``
55 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
57 The catalog file is an index for file archives. It contains
58 the list of files and is used to speed up search operations.
61 The Manifest: ``index.json``
62 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
64 The manifest contains the list of all backup files, their
65 sizes and checksums. It is used to verify the consistency of a
72 The backup server groups backups by *type*, where *type* is one of:
75 This type is used for :term:`virtual machine`\ s. Typically
76 consists of the virtual machine's configuration file and an image archive
80 This type is used for :term:`container`\ s. Consists of the container's
81 configuration and a single file archive for the filesystem content.
84 This type is used for backups created from within the backed up machine.
85 Typically this would be a physical host but could also be a virtual machine
86 or container. Such backups may contain file and image archives, there are no restrictions in this regard.
92 A unique ID. Usually the virtual machine or container ID. ``host``
93 type backups normally use the hostname.
99 The time when the backup was made.
105 The tuple ``<type>/<ID>`` is called a backup group. Such a group
106 may contain one or more backup snapshots.
112 The triplet ``<type>/<ID>/<time>`` is called a backup snapshot. It
113 uniquely identifies a specific backup within a datastore.
115 .. code-block:: console
116 :caption: Backup Snapshot Examples
118 vm/104/2019-10-09T08:01:06Z
119 host/elsa/2019-11-08T09:48:14Z
121 As you can see, the time format is RFC3399_ with Coordinated
122 Universal Time (UTC_, identified by the trailing *Z*).
124 Backup Server Management
125 ------------------------
127 The command line tool to configure and manage the backup server is called
128 :command:`proxmox-backup-manager`.
135 A datastore is a place where backups are stored. The current implementation
136 uses a directory inside a standard unix file system (``ext4``, ``xfs``
137 or ``zfs``) to store the backup data.
139 Datastores are identified by a simple *ID*. You can configure it
140 when setting up the backup server.
142 .. note:: The `File Layout`_ requires the file system to support at least *65538*
143 subdirectories per directory. That number comes from the 2\ :sup:`16`
144 pre-created chunk namespace directories, and the ``.`` and ``..`` default
145 directory entries. This requirement excludes certain filesystems and
146 filesystem configuration from being supported for a datastore. For example,
147 ``ext3`` as a whole or ``ext4`` with the ``dir_nlink`` feature manually disabled.
150 Datastore Configuration
151 ~~~~~~~~~~~~~~~~~~~~~~~
153 You can configure multiple datastores. Minimum one datastore needs to be
154 configured. The datastore is identified by a simple `name` and points to a
155 directory on the filesystem. Each datastore also has associated retention
156 settings of how many backup snapshots for each interval of ``hourly``,
157 ``daily``, ``weekly``, ``monthly``, ``yearly`` as well as a time-independent
158 number of backups to keep in that store. :ref:`Pruning <pruning>` and
159 :ref:`garbage collection <garbage-collection>` can also be configured to run
160 periodically based on a configured :term:`schedule` per datastore.
162 The following command creates a new datastore called ``store1`` on :file:`/backup/disk1/store1`
164 .. code-block:: console
166 # proxmox-backup-manager datastore create store1 /backup/disk1/store1
168 To list existing datastores run:
170 .. code-block:: console
172 # proxmox-backup-manager datastore list
173 ┌────────┬──────────────────────┬─────────────────────────────┐
174 │ name │ path │ comment │
175 ╞════════╪══════════════════════╪═════════════════════════════╡
176 │ store1 │ /backup/disk1/store1 │ This is my default storage. │
177 └────────┴──────────────────────┴─────────────────────────────┘
179 You can change settings of a datastore, for example to set a prune and garbage
180 collection schedule or retention settings using ``update`` subcommand and view
181 a datastore with the ``show`` subcommand:
183 .. code-block:: console
185 # proxmox-backup-manager datastore update store1 --keep-last 7 --prune-schedule daily --gc-schedule 'Tue 04:27'
186 # proxmox-backup-manager datastore show store1
187 ┌────────────────┬─────────────────────────────┐
189 ╞════════════════╪═════════════════════════════╡
191 ├────────────────┼─────────────────────────────┤
192 │ path │ /backup/disk1/store1 │
193 ├────────────────┼─────────────────────────────┤
194 │ comment │ This is my default storage. │
195 ├────────────────┼─────────────────────────────┤
196 │ gc-schedule │ Tue 04:27 │
197 ├────────────────┼─────────────────────────────┤
199 ├────────────────┼─────────────────────────────┤
200 │ prune-schedule │ daily │
201 └────────────────┴─────────────────────────────┘
203 Finally, it is possible to remove the datastore configuration:
205 .. code-block:: console
207 # proxmox-backup-manager datastore remove store1
209 .. note:: The above command removes only the datastore configuration. It does
210 not delete any data from the underlying directory.
216 After creating a datastore, the following default layout will appear:
218 .. code-block:: console
220 # ls -arilh /backup/disk1/store1
221 276493 -rw-r--r-- 1 backup backup 0 Jul 8 12:35 .lock
222 276490 drwxr-x--- 1 backup backup 1064960 Jul 8 12:35 .chunks
224 `.lock` is an empty file used for process locking.
226 The `.chunks` directory contains folders, starting from `0000` and taking hexadecimal values until `ffff`. These
227 directories will store the chunked data after a backup operation has been executed.
229 .. code-block:: console
231 # ls -arilh /backup/disk1/store1/.chunks
232 545824 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 ffff
233 545823 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 fffe
234 415621 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 fffd
235 415620 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 fffc
236 353187 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 fffb
237 344995 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 fffa
238 144079 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 fff9
239 144078 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 fff8
240 144077 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 fff7
242 403180 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 000c
243 403179 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 000b
244 403177 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 000a
245 402530 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 0009
246 402513 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 0008
247 402509 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 0007
248 276509 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 0006
249 276508 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 0005
250 276507 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 0004
251 276501 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 0003
252 276499 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 0002
253 276498 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 0001
254 276494 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 0000
255 276489 drwxr-xr-x 3 backup backup 4.0K Jul 8 12:35 ..
256 276490 drwxr-x--- 1 backup backup 1.1M Jul 8 12:35 .
263 Proxmox Backup Server supports several authentication realms, and you need to
264 choose the realm when you add a new user. Possible realms are:
266 :pam: Linux PAM standard authentication. Use this if you want to
267 authenticate as Linux system user (Users need to exist on the
270 :pbs: Proxmox Backup Server realm. This type stores hashed passwords in
271 ``/etc/proxmox-backup/shadow.json``.
273 After installation, there is a single user ``root@pam``, which
274 corresponds to the Unix superuser. You can use the
275 ``proxmox-backup-manager`` command line tool to list or manipulate
278 .. code-block:: console
280 # proxmox-backup-manager user list
281 ┌─────────────┬────────┬────────┬───────────┬──────────┬────────────────┬────────────────────┐
282 │ userid │ enable │ expire │ firstname │ lastname │ email │ comment │
283 ╞═════════════╪════════╪════════╪═══════════╪══════════╪════════════════╪════════════════════╡
284 │ root@pam │ 1 │ │ │ │ │ Superuser │
285 └─────────────┴────────┴────────┴───────────┴──────────┴────────────────┴────────────────────┘
287 The superuser has full administration rights on everything, so you
288 normally want to add other users with less privileges:
290 .. code-block:: console
292 # proxmox-backup-manager user create john@pbs --email john@example.com
294 The create command lets you specify many options like ``--email`` or
295 ``--password``. You can update or change any of them using the
296 update command later:
298 .. code-block:: console
300 # proxmox-backup-manager user update john@pbs --firstname John --lastname Smith
301 # proxmox-backup-manager user update john@pbs --comment "An example user."
303 .. todo:: Mention how to set password without passing plaintext password as cli argument.
306 The resulting user list looks like this:
308 .. code-block:: console
310 # proxmox-backup-manager user list
311 ┌──────────┬────────┬────────┬───────────┬──────────┬──────────────────┬──────────────────┐
312 │ userid │ enable │ expire │ firstname │ lastname │ email │ comment │
313 ╞══════════╪════════╪════════╪═══════════╪══════════╪══════════════════╪══════════════════╡
314 │ john@pbs │ 1 │ │ John │ Smith │ john@example.com │ An example user. │
315 ├──────────┼────────┼────────┼───────────┼──────────┼──────────────────┼──────────────────┤
316 │ root@pam │ 1 │ │ │ │ │ Superuser │
317 └──────────┴────────┴────────┴───────────┴──────────┴──────────────────┴──────────────────┘
319 Newly created users do not have any permissions. Please read the next
320 section to learn how to set access permissions.
322 If you want to disable a user account, you can do that by setting ``--enable`` to ``0``
324 .. code-block:: console
326 # proxmox-backup-manager user update john@pbs --enable 0
328 Or completely remove the user with:
330 .. code-block:: console
332 # proxmox-backup-manager user remove john@pbs
338 By default new users do not have any permission. Instead you need to
339 specify what is allowed and what is not. You can do this by assigning
340 roles to users on specific objects like datastores or remotes. The
341 following roles exist:
344 Disable Access - nothing is allowed.
347 The Administrator can do anything.
350 An Auditor can view things, but is not allowed to change settings.
353 Can do anything on datastores.
356 Can view datastore settings and list content. But
357 is not allowed to read the actual data.
360 Can Inspect datastore content and can do restores.
363 Can backup and restore owned backups.
365 **DatastorePowerUser**
366 Can backup, restore, and prune owned backups.
369 Can do anything on remotes.
372 Can view remote settings.
374 **RemoteSyncOperator**
375 Is allowed to read data from a remote.
381 A remote refers to a separate Proxmox Backup Server installation and a user on that
382 installation, from which you can `sync` datastores to a local datastore with a
385 To add a remote, you need its hostname or ip, a userid and password on the
386 remote, and its certificate fingerprint. To get the fingerprint, use the
387 ``proxmox-backup-manager cert info`` command on the remote.
389 .. code-block:: console
391 # proxmox-backup-manager cert info |grep Fingerprint
392 Fingerprint (sha256): 64:d3:ff:3a:50:38:53:5a:9b:f7:50:...:ab:fe
394 Using the information specified above, add the remote with:
396 .. code-block:: console
398 # proxmox-backup-manager remote create pbs2 --host pbs2.mydomain.example --userid sync@pam --password 'SECRET' --fingerprint 64:d3:ff:3a:50:38:53:5a:9b:f7:50:...:ab:fe
400 Use the ``list``, ``show``, ``update``, ``remove`` subcommands of
401 ``proxmox-backup-manager remote`` to manage your remotes:
403 .. code-block:: console
405 # proxmox-backup-manager remote update pbs2 --host pbs2.example
406 # proxmox-backup-manager remote list
407 ┌──────┬──────────────┬──────────┬───────────────────────────────────────────┬─────────┐
408 │ name │ host │ userid │ fingerprint │ comment │
409 ╞══════╪══════════════╪══════════╪═══════════════════════════════════════════╪═════════╡
410 │ pbs2 │ pbs2.example │ sync@pam │64:d3:ff:3a:50:38:53:5a:9b:f7:50:...:ab:fe │ │
411 └──────┴──────────────┴──────────┴───────────────────────────────────────────┴─────────┘
412 # proxmox-backup-manager remote remove pbs2
418 Sync jobs are configured to pull the contents of a datastore on a `Remote` to a
419 local datastore. You can either start the sync job manually on the GUI or
420 provide it with a :term:`schedule` to run regularly. The
421 ``proxmox-backup-manager sync-job`` command is used to manage sync jobs:
423 .. code-block:: console
425 # proxmox-backup-manager sync-job create pbs2-local --remote pbs2 --remote-store local --store local --schedule 'Wed 02:30'
426 # proxmox-backup-manager sync-job update pbs2-local --comment 'offsite'
427 # proxmox-backup-manager sync-job list
428 ┌────────────┬───────┬────────┬──────────────┬───────────┬─────────┐
429 │ id │ store │ remote │ remote-store │ schedule │ comment │
430 ╞════════════╪═══════╪════════╪══════════════╪═══════════╪═════════╡
431 │ pbs2-local │ local │ pbs2 │ local │ Wed 02:30 │ offsite │
432 └────────────┴───────┴────────┴──────────────┴───────────┴─────────┘
433 # proxmox-backup-manager sync-job remove pbs2-local
439 The command line client is called :command:`proxmox-backup-client`.
445 The client uses the following notation to specify a datastore repository
446 on the backup server.
448 [[username@]server:]datastore
450 The default value for ``username`` ist ``root``. If no server is specified,
451 the default is the local host (``localhost``).
453 You can pass the repository with the ``--repository`` command
454 line option, or by setting the ``PBS_REPOSITORY`` environment
458 Environment Variables
459 ~~~~~~~~~~~~~~~~~~~~~
462 The default backup repository.
465 When set, this value is used for the password required for the
468 ``PBS_ENCRYPTION_PASSWORD``
469 When set, this value is used to access the secret encryption key (if
470 protected by password).
472 ``PBS_FINGERPRINT`` When set, this value is used to verify the server
473 certificate (only used if the system CA certificates cannot
474 validate the certificate).
480 Most commands support the ``--output-format`` parameter. It accepts
481 the following values:
483 :``text``: Text format (default). Structured data is rendered as a table.
485 :``json``: JSON (single line).
487 :``json-pretty``: JSON (multiple lines, nicely formatted).
490 Please use the following environment variables to modify output behavior:
492 ``PROXMOX_OUTPUT_FORMAT``
493 Defines the default output format.
495 ``PROXMOX_OUTPUT_NO_BORDER``
496 If set (to any value), do not render table borders.
498 ``PROXMOX_OUTPUT_NO_HEADER``
499 If set (to any value), do not render table headers.
501 .. note:: The ``text`` format is designed to be human readable, and
502 not meant to be parsed by automation tools. Please use the ``json``
503 format if you need to process the output.
506 .. _creating-backups:
511 This section explains how to create a backup from within the machine. This can
512 be a physical host, a virtual machine, or a container. Such backups may contain file
513 and image archives. There are no restrictions in this case.
515 .. note:: If you want to backup virtual machines or containers on Proxmox VE, see :ref:`pve-integration`.
517 For the following example you need to have a backup server set up, working
518 credentials and need to know the repository name.
519 In the following examples we use ``backup-server:store1``.
521 .. code-block:: console
523 # proxmox-backup-client backup root.pxar:/ --repository backup-server:store1
524 Starting backup: host/elsa/2019-12-03T09:35:01Z
526 skip mount point: "/boot/efi"
527 skip mount point: "/dev"
528 skip mount point: "/run"
529 skip mount point: "/sys"
530 Uploaded 12129 chunks in 87 seconds (564 MB/s).
531 End Time: 2019-12-03T10:36:29+01:00
533 This will prompt you for a password and then uploads a file archive named
534 ``root.pxar`` containing all the files in the ``/`` directory.
536 .. Caution:: Please note that the proxmox-backup-client does not
537 automatically include mount points. Instead, you will see a short
538 ``skip mount point`` notice for each of them. The idea is to
539 create a separate file archive for each mounted disk. You can
540 explicitly include them using the ``--include-dev`` option
541 (i.e. ``--include-dev /boot/efi``). You can use this option
542 multiple times for each mount point that should be included.
544 The ``--repository`` option can get quite long and is used by all
545 commands. You can avoid having to enter this value by setting the
546 environment variable ``PBS_REPOSITORY``.
548 .. code-block:: console
550 # export PBS_REPOSITORY=backup-server:store1
552 After this you can execute all commands without specifying the ``--repository``
555 One single backup is allowed to contain more than one archive. For example, if
556 you want to backup two disks mounted at ``/mmt/disk1`` and ``/mnt/disk2``:
558 .. code-block:: console
560 # proxmox-backup-client backup disk1.pxar:/mnt/disk1 disk2.pxar:/mnt/disk2
562 This creates a backup of both disks.
564 The backup command takes a list of backup specifications, which
565 include the archive name on the server, the type of the archive, and the
566 archive source at the client. The format is:
568 <archive-name>.<type>:<source-path>
570 Common types are ``.pxar`` for file archives, and ``.img`` for block
571 device images. To create a backup of a block device run the following command:
573 .. code-block:: console
575 # proxmox-backup-client backup mydata.img:/dev/mylvm/mydata
577 Excluding files/folders from a backup
578 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
580 Sometimes it is desired to exclude certain files or folders from a backup archive.
581 To tell the Proxmox backup client when and how to ignore files and directories,
582 place a text file called ``.pxarexclude`` in the filesystem hierarchy.
583 Whenever the backup client encounters such a file in a directory, it interprets
584 each line as glob match patterns for files and directories that are to be excluded
587 The file must contain a single glob pattern per line. Empty lines are ignored.
588 The same is true for lines starting with ``#``, which indicates a comment.
589 A ``!`` at the beginning of a line reverses the glob match pattern from an exclusion
590 to an explicit inclusion. This makes it possible to exclude all entries in a
591 directory except for a few single files/subdirectories.
592 Lines ending in ``/`` match only on directories.
593 The directory containing the ``.pxarexclude`` file is considered to be the root of
594 the given patterns. It is only possible to match files in this directory and its subdirectories.
596 ``\`` is used to escape special glob characters.
597 ``?`` matches any single character.
598 ``*`` matches any character, including an empty string.
599 ``**`` is used to match subdirectories. It can be used to, for example, exclude
600 all files ending in ``.tmp`` within the directory or subdirectories with the
601 following pattern ``**/*.tmp``.
602 ``[...]`` matches a single character from any of the provided characters within
603 the brackets. ``[!...]`` does the complementary and matches any single character
604 not contained within the brackets. It is also possible to specify ranges with two
605 characters separated by ``-``. For example, ``[a-z]`` matches any lowercase
606 alphabetic character and ``[0-9]`` matches any one single digit.
608 The order of the glob match patterns defines whether a file is included or
609 excluded, that is to say later entries override previous ones.
610 This is also true for match patterns encountered deeper down the directory tree,
611 which can override a previous exclusion.
612 Be aware that excluded directories will **not** be read by the backup client.
613 Thus, a ``.pxarexclude`` file in an excluded subdirectory will have no effect.
614 ``.pxarexclude`` files are treated as regular files and will be included in the
617 For example, consider the following directory structure:
619 .. code-block:: console
623 . .. .pxarexclude subfolder0 subfolder1
626 . .. file0 file1 file2 file3 .pxarexclude
629 . .. file0 file1 file2 file3
631 The different ``.pxarexclude`` files contain the following:
633 .. code-block:: console
635 # cat folder/.pxarexclude
640 .. code-block:: console
642 # cat folder/subfolder0/.pxarexclude
645 This would exclude ``file1`` and ``file3`` in ``subfolder0`` and all of
646 ``subfolder1`` except ``file2``.
648 Restoring this backup will result in:
650 .. code-block:: console
654 . .. .pxarexclude subfolder0 subfolder1
657 . .. file0 file2 .pxarexclude
665 Proxmox Backup supports client-side encryption with AES-256 in GCM_
666 mode. To set this up, you first need to create an encryption key:
668 .. code-block:: console
670 # proxmox-backup-client key create my-backup.key
671 Encryption Key Password: **************
673 The key is password protected by default. If you do not need this
674 extra protection, you can also create it without a password:
676 .. code-block:: console
678 # proxmox-backup-client key create /path/to/my-backup.key --kdf none
681 .. code-block:: console
683 # proxmox-backup-client backup etc.pxar:/etc --keyfile /path/to/my-backup.key
685 Encryption Key Password: **************
689 You can avoid entering the passwords by setting the environment
690 variables ``PBS_PASSWORD`` and ``PBS_ENCRYPTION_PASSWORD``.
692 .. todo:: Explain master-key
698 The regular creation of backups is a necessary step to avoiding data
699 loss. More importantly, however, is the restoration. It is good practice to perform
700 periodic recovery tests to ensure that you can access the data in
703 First, you need to find the snapshot which you want to restore. The snapshot
704 command provides a list of all the snapshots on the server:
706 .. code-block:: console
708 # proxmox-backup-client snapshots
709 ┌────────────────────────────────┬─────────────┬────────────────────────────────────┐
710 │ snapshot │ size │ files │
711 ╞════════════════════════════════╪═════════════╪════════════════════════════════════╡
712 │ host/elsa/2019-12-03T09:30:15Z │ 51788646825 │ root.pxar catalog.pcat1 index.json │
713 ├────────────────────────────────┼─────────────┼────────────────────────────────────┤
714 │ host/elsa/2019-12-03T09:35:01Z │ 51790622048 │ root.pxar catalog.pcat1 index.json │
715 ├────────────────────────────────┼─────────────┼────────────────────────────────────┤
718 You can inspect the catalog to find specific files.
720 .. code-block:: console
722 # proxmox-backup-client catalog dump host/elsa/2019-12-03T09:35:01Z
724 d "./root.pxar.didx/etc/cifs-utils"
725 l "./root.pxar.didx/etc/cifs-utils/idmap-plugin"
726 d "./root.pxar.didx/etc/console-setup"
729 The restore command lets you restore a single archive from the
732 .. code-block:: console
734 # proxmox-backup-client restore host/elsa/2019-12-03T09:35:01Z root.pxar /target/path/
736 To get the contents of any archive, you can restore the ``ìndex.json`` file in the
737 repository to the target path '-'. This will dump the contents to the standard output.
739 .. code-block:: console
741 # proxmox-backup-client restore host/elsa/2019-12-03T09:35:01Z index.json -
747 If you only want to restore a few individual files, it is often easier
748 to use the interactive recovery shell.
750 .. code-block:: console
752 # proxmox-backup-client catalog shell host/elsa/2019-12-03T09:35:01Z root.pxar
753 Starting interactive shell
755 bin boot dev etc home lib lib32
758 The interactive recovery shell is a minimalistic command line interface that
759 utilizes the metadata stored in the catalog to quickly list, navigate and
760 search files in a file archive.
761 To restore files, you can select them individually or match them with a glob
764 Using the catalog for navigation reduces the overhead considerably because only
765 the catalog needs to be downloaded and, optionally, decrypted.
766 The actual chunks are only accessed if the metadata in the catalog is not enough
767 or for the actual restore.
769 Similar to common UNIX shells ``cd`` and ``ls`` are the commands used to change
770 working directory and list directory contents in the archive.
771 ``pwd`` shows the full path of the current working directory with respect to the
774 Being able to quickly search the contents of the archive is a commmonly needed feature.
775 That's where the catalog is most valuable.
778 .. code-block:: console
780 pxar:/ > find etc/ **/*.txt --select
782 pxar:/ > list-selected
784 pxar:/ > restore-selected /target/path
787 This will find and print all files ending in ``.txt`` located in ``etc/`` or a
788 subdirectory and add the corresponding pattern to the list for subsequent restores.
789 ``list-selected`` shows these patterns and ``restore-selected`` finally restores
790 all files in the archive matching the patterns to ``/target/path`` on the local
791 host. This will scan the whole archive.
793 With ``restore /target/path`` you can restore the sub-archive given by the current
794 working directory to the local target path ``/target/path`` on your host.
795 By additionally passing a glob pattern with ``--pattern <glob>``, the restore is
796 further limited to files matching the pattern.
799 .. code-block:: console
802 pxar:/etc/ > restore /target/ --pattern **/*.conf
805 The above will scan trough all the directories below ``/etc`` and restore all
806 files ending in ``.conf``.
808 .. todo:: Explain interactive restore in more detail
810 Mounting of Archives via FUSE
811 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
813 The :term:`FUSE` implementation for the pxar archive allows you to mount a
814 file archive as a read-only filesystem to a mountpoint on your host.
816 .. code-block:: console
818 # proxmox-backup-client mount host/backup-client/2020-01-29T11:29:22Z root.pxar /mnt
820 bin dev home lib32 libx32 media opt root sbin sys usr
821 boot etc lib lib64 lost+found mnt proc run srv tmp var
823 This allows you to access the full contents of the archive in a seamless manner.
825 .. note:: As the FUSE connection needs to fetch and decrypt chunks from the
826 backup server's datastore, this can cause some additional network and CPU
827 load on your host, depending on the operations you perform on the mounted
830 To unmount the filesystem use the ``umount`` command on the mountpoint:
832 .. code-block:: console
839 The client tool prompts you to enter the logon password as soon as you
840 want to access the backup server. The server checks your credentials
841 and responds with a ticket that is valid for two hours. The client
842 tool automatically stores that ticket and uses it for further requests
845 You can also manually trigger this login/logout using the login and
848 .. code-block:: console
850 # proxmox-backup-client login
853 To remove the ticket, issue a logout:
855 .. code-block:: console
857 # proxmox-backup-client logout
862 Pruning and Removing Backups
863 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
865 You can manually delete a backup snapshot using the ``forget``
868 .. code-block:: console
870 # proxmox-backup-client forget <snapshot>
873 .. caution:: This command removes all archives in this backup
874 snapshot. They will be inaccessible and unrecoverable.
877 The manual removal is sometimes required, but normally the prune
878 command is used to systematically delete older backups. Prune lets
879 you specify which backup snapshots you want to keep. The
880 following retention options are available:
883 Keep the last ``<N>`` backup snapshots.
885 ``--keep-hourly <N>``
886 Keep backups for the last ``<N>`` hours. If there is more than one
887 backup for a single hour, only the latest is kept.
890 Keep backups for the last ``<N>`` days. If there is more than one
891 backup for a single day, only the latest is kept.
893 ``--keep-weekly <N>``
894 Keep backups for the last ``<N>`` weeks. If there is more than one
895 backup for a single week, only the latest is kept.
897 .. note:: Weeks start on Monday and end on Sunday. The software
898 uses the `ISO week date`_ system and handles weeks at
899 the end of the year correctly.
901 ``--keep-monthly <N>``
902 Keep backups for the last ``<N>`` months. If there is more than one
903 backup for a single month, only the latest is kept.
905 ``--keep-yearly <N>``
906 Keep backups for the last ``<N>`` years. If there is more than one
907 backup for a single year, only the latest is kept.
909 The retention options are processed in the order given above. Each option
910 only covers backups within its time period. The next option does not take care
911 of already covered backups. It will only consider older backups.
913 Unfinished and incomplete backups will be removed by the prune command unless
914 they are newer than the last successful backup. In this case, the last failed
917 .. code-block:: console
919 # proxmox-backup-client prune <group> --keep-daily 7 --keep-weekly 4 --keep-monthly 3
922 You can use the ``--dry-run`` option to test your settings. This only
923 shows the list of existing snapshots and what actions prune would take.
925 .. code-block:: console
927 # proxmox-backup-client prune host/elsa --dry-run --keep-daily 1 --keep-weekly 3
928 ┌────────────────────────────────┬──────┐
930 ╞════════════════════════════════╪══════╡
931 │ host/elsa/2019-12-04T13:20:37Z │ 1 │
932 ├────────────────────────────────┼──────┤
933 │ host/elsa/2019-12-03T09:35:01Z │ 0 │
934 ├────────────────────────────────┼──────┤
935 │ host/elsa/2019-11-22T11:54:47Z │ 1 │
936 ├────────────────────────────────┼──────┤
937 │ host/elsa/2019-11-21T12:36:25Z │ 0 │
938 ├────────────────────────────────┼──────┤
939 │ host/elsa/2019-11-10T10:42:20Z │ 1 │
940 └────────────────────────────────┴──────┘
942 .. note:: Neither the ``prune`` command nor the ``forget`` command free space
943 in the chunk-store. The chunk-store still contains the data blocks. To free
944 space you need to perform :ref:`garbage-collection`.
947 .. _garbage-collection:
952 The ``prune`` command removes only the backup index files, not the data
953 from the data store. This task is left to the garbage collection
954 command. It is recommended to carry out garbage collection on a regular basis.
956 The garbage collection works in two phases. In the first phase, all
957 data blocks that are still in use are marked. In the second phase,
958 unused data blocks are removed.
960 .. note:: This command needs to read all existing backup index files
961 and touches the complete chunk-store. This can take a long time
962 depending on the number of chunks and the speed of the underlying
965 .. note:: The garbage collection will only remove chunks that haven't been used
966 for at least one day (exactly 24h 5m). This grace period is necessary because
967 chunks in use are marked by touching the chunk which updates the ``atime``
968 (access time) property. Filesystems are mounted with the ``relatime`` option
969 by default. This results in a better performance by only updating the
970 ``atime`` property if the last access has been at least 24 hours ago. The
971 downside is, that touching a chunk within these 24 hours will not always
972 update its ``atime`` property.
974 Chunks in the grace period will be logged at the end of the garbage
975 collection task as *Pending removals*.
977 .. code-block:: console
979 # proxmox-backup-client garbage-collect
980 starting garbage collection on store store2
981 Start GC phase1 (mark used chunks)
982 Start GC phase2 (sweep unused chunks)
983 percentage done: 1, chunk count: 219
984 percentage done: 2, chunk count: 453
986 percentage done: 99, chunk count: 21188
987 Removed bytes: 411368505
989 Original data bytes: 327160886391
990 Disk bytes: 52767414743 (16 %)
992 Average chunk size: 2486565
996 .. todo:: howto run garbage-collection at regular intervalls (cron)
1001 `Proxmox VE`_ integration
1002 -------------------------
1004 You need to define a new storage with type 'pbs' on your `Proxmox VE`_
1005 node. The following example uses ``store2`` as storage name, and
1006 assumes the server address is ``localhost``, and you want to connect
1009 .. code-block:: console
1011 # pvesm add pbs store2 --server localhost --datastore store2
1012 # pvesm set store2 --username user1@pbs --password <secret>
1014 If your backup server uses a self signed certificate, you need to add
1015 the certificate fingerprint to the configuration. You can get the
1016 fingerprint by running the following command on the backup server:
1018 .. code-block:: console
1020 # proxmox-backup-manager cert info |grep Fingerprint
1021 Fingerprint (sha256): 64:d3:ff:3a:50:38:53:5a:9b:f7:50:...:ab:fe
1023 Please add that fingerprint to your configuration to establish a trust
1026 .. code-block:: console
1028 # pvesm set store2 --fingerprint 64:d3:ff:3a:50:38:53:5a:9b:f7:50:...:ab:fe
1030 After that you should be able to see storage status with:
1032 .. code-block:: console
1034 # pvesm status --storage store2
1035 Name Type Status Total Used Available %
1036 store2 pbs active 3905109820 1336687816 2568422004 34.23%
1040 .. include:: command-line-tools.rst
1042 .. include:: services.rst