4 .. The administration guide.
5 .. todo:: either add a bit more explanation or remove the previous sentence
13 When doing deduplication, there are different strategies to get
14 optimal results in terms of performance and/or deduplication rates.
15 Depending on the type of data, it can be split into *fixed* or *variable*
18 Fixed sized chunking requires minimal CPU power, and is used to
19 backup virtual machine images.
21 Variable sized chunking needs more CPU power, but is essential to get
22 good deduplication rates for file archives.
24 The Proxmox Backup Server supports both strategies.
27 Image Archives: ``<name>.img``
28 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
30 This is used for virtual machine images and other large binary
31 data. Content is split into fixed-sized chunks.
34 File Archives: ``<name>.pxar``
35 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
37 .. see https://moinakg.wordpress.com/2013/06/22/high-performance-content-defined-chunking/
39 A file archive stores a full directory tree. Content is stored using
40 the :ref:`pxar-format`, split into variable-sized chunks. The format
41 is optimized to achieve good deduplication rates.
47 This type is used to store smaller (< 16MB) binary data such as
48 configuration files. Larger files should be stored as image archive.
50 .. caution:: Please do not store all files as BLOBs. Instead, use the
51 file archive to store whole directory trees.
54 Catalog File: ``catalog.pcat1``
55 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
57 The catalog file is an index for file archives. It contains
58 the list of files and is used to speed up search operations.
61 The Manifest: ``index.json``
62 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
64 The manifest contains the list of all backup files, their
65 sizes and checksums. It is used to verify the consistency of a
72 The backup server groups backups by *type*, where *type* is one of:
75 This type is used for :term:`virtual machine`\ s. Typically
76 consists of the virtual machine's configuration file and an image archive
80 This type is used for :term:`container`\ s. Consists of the container's
81 configuration and a single file archive for the filesystem content.
84 This type is used for backups created from within the backed up machine.
85 Typically this would be a physical host but could also be a virtual machine
86 or container. Such backups may contain file and image archives, there are no restrictions in this regard.
92 A unique ID. Usually the virtual machine or container ID. ``host``
93 type backups normally use the hostname.
99 The time when the backup was made.
105 The tuple ``<type>/<ID>`` is called a backup group. Such a group
106 may contain one or more backup snapshots.
112 The triplet ``<type>/<ID>/<time>`` is called a backup snapshot. It
113 uniquely identifies a specific backup within a datastore.
115 .. code-block:: console
116 :caption: Backup Snapshot Examples
118 vm/104/2019-10-09T08:01:06Z
119 host/elsa/2019-11-08T09:48:14Z
121 As you can see, the time format is RFC3399_ with Coordinated
122 Universal Time (UTC_, identified by the trailing *Z*).
124 Backup Server Management
125 ------------------------
127 The command line tool to configure and manage the backup server is called
128 :command:`proxmox-backup-manager`.
135 A datastore is a place where backups are stored. The current implementation
136 uses a directory inside a standard unix file system (``ext4``, ``xfs``
137 or ``zfs``) to store the backup data.
139 Datastores are identified by a simple *ID*. You can configure it
140 when setting up the backup server.
142 .. note:: The `File Layout`_ requires the file system to support at least *65538*
143 subdirectories per directory. That number comes from the 2\ :sup:`16`
144 pre-created chunk namespace directories, and the ``.`` and ``..`` default
145 directory entries. This requirement excludes certain filesystems and
146 filesystem configuration from being supported for a datastore. For example,
147 ``ext3`` as a whole or ``ext4`` with the ``dir_nlink`` feature manually disabled.
152 .. image:: images/screenshots/pbs-gui-disks.png
156 Proxmox Backup Server comes with a set of disk utilities, which are
157 accessed using the ``disk`` subcommand. This subcommand allows you to initialize
158 disks, create various filesystems, and get information about the disks.
160 To view the disks connected to the system, navigate to **Administration ->
161 Disks** in the web interface or use the ``list`` subcommand of
164 .. code-block:: console
166 # proxmox-backup-manager disk list
167 ┌──────┬────────┬─────┬───────────┬─────────────┬───────────────┬─────────┬────────┐
168 │ name │ used │ gpt │ disk-type │ size │ model │ wearout │ status │
169 ╞══════╪════════╪═════╪═══════════╪═════════════╪═══════════════╪═════════╪════════╡
170 │ sda │ lvm │ 1 │ hdd │ 34359738368 │ QEMU_HARDDISK │ - │ passed │
171 ├──────┼────────┼─────┼───────────┼─────────────┼───────────────┼─────────┼────────┤
172 │ sdb │ unused │ 1 │ hdd │ 68719476736 │ QEMU_HARDDISK │ - │ passed │
173 ├──────┼────────┼─────┼───────────┼─────────────┼───────────────┼─────────┼────────┤
174 │ sdc │ unused │ 1 │ hdd │ 68719476736 │ QEMU_HARDDISK │ - │ passed │
175 └──────┴────────┴─────┴───────────┴─────────────┴───────────────┴─────────┴────────┘
177 To initialize a disk with a new GPT, use the ``initialize`` subcommand:
179 .. code-block:: console
181 # proxmox-backup-manager disk initialize sdX
183 .. image:: images/screenshots/pbs-gui-disks-dir-create.png
185 :alt: Create a directory
187 You can create an ``ext4`` or ``xfs`` filesystem on a disk using ``fs
188 create``, or by navigating to **Administration -> Disks -> Directory** in the
189 web interface and creating one from there. The following command creates an
190 ``ext4`` filesystem and passes the ``--add-datastore`` parameter, in order to
191 automatically create a datastore on the disk (in this case ``sdd``). This will
192 create a datastore at the location ``/mnt/datastore/store1``:
194 .. code-block:: console
196 # proxmox-backup-manager disk fs create store1 --disk sdd --filesystem ext4 --add-datastore true
198 .. image:: images/screenshots/pbs-gui-disks-zfs-create.png
200 :alt: Create a directory
202 You can also create a ``zpool`` with various raid levels from **Administration
203 -> Disks -> Zpool** in the web interface, or by using ``zpool create``. The command
204 below creates a mirrored ``zpool`` using two disks (``sdb`` & ``sdc``) and
205 mounts it on the root directory (default):
207 .. code-block:: console
209 # proxmox-backup-manager disk zpool create zpool1 --devices sdb,sdc --raidlevel mirror
212 You can also pass the ``--add-datastore`` parameter here, to automatically
213 create a datastore from the disk.
215 You can use ``disk fs list`` and ``disk zpool list`` to keep track of your
216 filesystems and zpools respectively.
218 If a disk supports S.M.A.R.T. capability, and you have this enabled, you can
219 display S.M.A.R.T. attributes from the web interface or by using the command:
221 .. code-block:: console
223 # proxmox-backup-manager disk smart-attributes sdX
226 Datastore Configuration
227 ~~~~~~~~~~~~~~~~~~~~~~~
229 .. image:: images/screenshots/pbs-gui-datastore.png
231 :alt: Datastore Overview
233 You can configure multiple datastores. Minimum one datastore needs to be
234 configured. The datastore is identified by a simple *name* and points to a
235 directory on the filesystem. Each datastore also has associated retention
236 settings of how many backup snapshots for each interval of ``hourly``,
237 ``daily``, ``weekly``, ``monthly``, ``yearly`` as well as a time-independent
238 number of backups to keep in that store. :ref:`Pruning <pruning>` and
239 :ref:`garbage collection <garbage-collection>` can also be configured to run
240 periodically based on a configured :term:`schedule` per datastore.
244 .. image:: images/screenshots/pbs-gui-datastore-create-general.png
246 :alt: Create a datastore
248 You can create a new datastore from the web GUI, by navigating to **Datastore** in
249 the menu tree and clicking **Create**. Here:
251 * *Name* refers to the name of the datastore
252 * *Backing Path* is the path to the directory upon which you want to create the
254 * *GC Schedule* refers to the time and intervals at which garbage collection
256 * *Prune Schedule* refers to the frequency at which pruning takes place
257 * *Prune Options* set the amount of backups which you would like to keep (see :ref:`Pruning <pruning>`).
259 Alternatively you can create a new datastore from the command line. The
260 following command creates a new datastore called ``store1`` on :file:`/backup/disk1/store1`
262 .. code-block:: console
264 # proxmox-backup-manager datastore create store1 /backup/disk1/store1
269 To list existing datastores from the command line run:
271 .. code-block:: console
273 # proxmox-backup-manager datastore list
274 ┌────────┬──────────────────────┬─────────────────────────────┐
275 │ name │ path │ comment │
276 ╞════════╪══════════════════════╪═════════════════════════════╡
277 │ store1 │ /backup/disk1/store1 │ This is my default storage. │
278 └────────┴──────────────────────┴─────────────────────────────┘
280 You can change the garbage collection and prune settings of a datastore, by
281 editing the datastore from the GUI or by using the ``update`` subcommand. For
282 example, the below command changes the garbage collection schedule using the
283 ``update`` subcommand and prints the properties of the datastore with the
286 .. code-block:: console
288 # proxmox-backup-manager datastore update store1 --gc-schedule 'Tue 04:27'
289 # proxmox-backup-manager datastore show store1
290 ┌────────────────┬─────────────────────────────┐
292 ╞════════════════╪═════════════════════════════╡
294 ├────────────────┼─────────────────────────────┤
295 │ path │ /backup/disk1/store1 │
296 ├────────────────┼─────────────────────────────┤
297 │ comment │ This is my default storage. │
298 ├────────────────┼─────────────────────────────┤
299 │ gc-schedule │ Tue 04:27 │
300 ├────────────────┼─────────────────────────────┤
302 ├────────────────┼─────────────────────────────┤
303 │ prune-schedule │ daily │
304 └────────────────┴─────────────────────────────┘
306 Finally, it is possible to remove the datastore configuration:
308 .. code-block:: console
310 # proxmox-backup-manager datastore remove store1
312 .. note:: The above command removes only the datastore configuration. It does
313 not delete any data from the underlying directory.
319 After creating a datastore, the following default layout will appear:
321 .. code-block:: console
323 # ls -arilh /backup/disk1/store1
324 276493 -rw-r--r-- 1 backup backup 0 Jul 8 12:35 .lock
325 276490 drwxr-x--- 1 backup backup 1064960 Jul 8 12:35 .chunks
327 `.lock` is an empty file used for process locking.
329 The `.chunks` directory contains folders, starting from `0000` and taking hexadecimal values until `ffff`. These
330 directories will store the chunked data after a backup operation has been executed.
332 .. code-block:: console
334 # ls -arilh /backup/disk1/store1/.chunks
335 545824 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 ffff
336 545823 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 fffe
337 415621 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 fffd
338 415620 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 fffc
339 353187 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 fffb
340 344995 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 fffa
341 144079 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 fff9
342 144078 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 fff8
343 144077 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 fff7
345 403180 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 000c
346 403179 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 000b
347 403177 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 000a
348 402530 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 0009
349 402513 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 0008
350 402509 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 0007
351 276509 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 0006
352 276508 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 0005
353 276507 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 0004
354 276501 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 0003
355 276499 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 0002
356 276498 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 0001
357 276494 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 0000
358 276489 drwxr-xr-x 3 backup backup 4.0K Jul 8 12:35 ..
359 276490 drwxr-x--- 1 backup backup 1.1M Jul 8 12:35 .
366 .. image:: images/screenshots/pbs-gui-user-management.png
368 :alt: User management
370 Proxmox Backup Server supports several authentication realms, and you need to
371 choose the realm when you add a new user. Possible realms are:
373 :pam: Linux PAM standard authentication. Use this if you want to
374 authenticate as Linux system user (Users need to exist on the
377 :pbs: Proxmox Backup Server realm. This type stores hashed passwords in
378 ``/etc/proxmox-backup/shadow.json``.
380 After installation, there is a single user ``root@pam``, which
381 corresponds to the Unix superuser. You can use the
382 ``proxmox-backup-manager`` command line tool to list or manipulate
385 .. code-block:: console
387 # proxmox-backup-manager user list
388 ┌─────────────┬────────┬────────┬───────────┬──────────┬────────────────┬────────────────────┐
389 │ userid │ enable │ expire │ firstname │ lastname │ email │ comment │
390 ╞═════════════╪════════╪════════╪═══════════╪══════════╪════════════════╪════════════════════╡
391 │ root@pam │ 1 │ │ │ │ │ Superuser │
392 └─────────────┴────────┴────────┴───────────┴──────────┴────────────────┴────────────────────┘
394 .. image:: images/screenshots/pbs-gui-user-management-add-user.png
398 The superuser has full administration rights on everything, so you
399 normally want to add other users with less privileges. You can create a new
400 user with the ``user create`` subcommand or through the web interface, under
401 **Configuration -> User Management**. The ``create`` subcommand lets you specify
402 many options like ``--email`` or ``--password``. You can update or change any
403 user properties using the ``update`` subcommand later (**Edit** in the GUI):
406 .. code-block:: console
408 # proxmox-backup-manager user create john@pbs --email john@example.com
409 # proxmox-backup-manager user update john@pbs --firstname John --lastname Smith
410 # proxmox-backup-manager user update john@pbs --comment "An example user."
412 .. todo:: Mention how to set password without passing plaintext password as cli argument.
415 The resulting user list looks like this:
417 .. code-block:: console
419 # proxmox-backup-manager user list
420 ┌──────────┬────────┬────────┬───────────┬──────────┬──────────────────┬──────────────────┐
421 │ userid │ enable │ expire │ firstname │ lastname │ email │ comment │
422 ╞══════════╪════════╪════════╪═══════════╪══════════╪══════════════════╪══════════════════╡
423 │ john@pbs │ 1 │ │ John │ Smith │ john@example.com │ An example user. │
424 ├──────────┼────────┼────────┼───────────┼──────────┼──────────────────┼──────────────────┤
425 │ root@pam │ 1 │ │ │ │ │ Superuser │
426 └──────────┴────────┴────────┴───────────┴──────────┴──────────────────┴──────────────────┘
428 Newly created users do not have any permissions. Please read the next
429 section to learn how to set access permissions.
431 If you want to disable a user account, you can do that by setting ``--enable`` to ``0``
433 .. code-block:: console
435 # proxmox-backup-manager user update john@pbs --enable 0
437 Or completely remove the user with:
439 .. code-block:: console
441 # proxmox-backup-manager user remove john@pbs
447 By default new users do not have any permission. Instead you need to
448 specify what is allowed and what is not. You can do this by assigning
449 roles to users on specific objects like datastores or remotes. The
450 following roles exist:
453 Disable Access - nothing is allowed.
459 Can view things, but is not allowed to change settings.
462 Can do anything on datastores.
465 Can view datastore settings and list content. But
466 is not allowed to read the actual data.
469 Can Inspect datastore content and can do restores.
472 Can backup and restore owned backups.
474 **DatastorePowerUser**
475 Can backup, restore, and prune owned backups.
478 Can do anything on remotes.
481 Can view remote settings.
483 **RemoteSyncOperator**
484 Is allowed to read data from a remote.
487 :alt: Add permissions for user
489 You can manage datastore permissions from **Configuration -> Permissions** in
490 the web interface. Likewise, you can use the ``acl`` subcommand to manage and
491 monitor user permissions from the command line. For example, the command below
492 will add the user ``john@pbs`` as a **DatastoreAdmin** for the datastore
493 ``store1``, located at ``/backup/disk1/store1``:
495 .. code-block:: console
497 # proxmox-backup-manager acl update /datastore/store1 DatastoreAdmin --userid john@pbs
499 You can monitor the roles of each user using the following command:
501 .. code-block:: console
503 # proxmox-backup-manager acl list
504 ┌──────────┬──────────────────┬───────────┬────────────────┐
505 │ ugid │ path │ propagate │ roleid │
506 ╞══════════╪══════════════════╪═══════════╪════════════════╡
507 │ john@pbs │ /datastore/disk1 │ 1 │ DatastoreAdmin │
508 └──────────┴──────────────────┴───────────┴────────────────┘
510 A single user can be assigned multiple permission sets for different datastores.
513 Naming convention is important here. For datastores on the host,
514 you must use the convention ``/datastore/{storename}``. For example, to set
515 permissions for a datastore mounted at ``/mnt/backup/disk4/store2``, you would use
516 ``/datastore/store2`` for the path. For remote stores, use the convention
517 ``/remote/{remote}/{storename}``, where ``{remote}`` signifies the name of the
518 remote (see `Remote` below) and ``{storename}`` is the name of the datastore on
524 Proxmox Backup Server provides both a web interface and a command line tool for
525 network configuration. You can find the configuration options in the web
526 interface under the **Network Interfaces** section of the **Configuration** menu
527 tree item. The command line tool is accessed via the ``network`` subcommand.
528 These interfaces allow you to carry out some basic network management tasks,
529 such as adding, configuring, and removing network interfaces.
531 .. note:: Any changes made to the network configuration are not
532 applied, until you click on **Apply Configuration** or enter the ``network
533 reload`` command. This allows you to make many changes at once. It also allows
534 you to ensure that your changes are correct before applying them, as making a
535 mistake here can render the server inaccessible over the network.
537 To get a list of available interfaces, use the following command:
539 .. code-block:: console
541 # proxmox-backup-manager network list
542 ┌───────┬────────┬───────────┬────────┬─────────────┬──────────────┬──────────────┐
543 │ name │ type │ autostart │ method │ address │ gateway │ ports/slaves │
544 ╞═══════╪════════╪═══════════╪════════╪═════════════╪══════════════╪══════════════╡
545 │ bond0 │ bond │ 1 │ static │ x.x.x.x/x │ x.x.x.x │ ens18 ens19 │
546 ├───────┼────────┼───────────┼────────┼─────────────┼──────────────┼──────────────┤
547 │ ens18 │ eth │ 1 │ manual │ │ │ │
548 ├───────┼────────┼───────────┼────────┼─────────────┼──────────────┼──────────────┤
549 │ ens19 │ eth │ 1 │ manual │ │ │ │
550 └───────┴────────┴───────────┴────────┴─────────────┴──────────────┴──────────────┘
552 .. image:: images/screenshots/pbs-gui-network-create-bond.png
554 :alt: Add a network interface
556 To add a new network interface, use the ``create`` subcommand with the relevant
557 parameters. The following command shows a template for creating the bond shown
560 .. code-block:: console
562 # proxmox-backup-manager network create bond0 --type bond --bond_mode active-backup --slaves ens18,ens19 --autostart true --cidr x.x.x.x/x --gateway x.x.x.x
564 You can make changes to the configuration of a network interface with the
565 ``update`` subcommand:
567 .. code-block:: console
569 # proxmox-backup-manager network update bond0 --cidr y.y.y.y/y
571 You can also remove a network interface:
573 .. code-block:: console
575 # proxmox-backup-manager network remove bond0
577 The pending changes for the network configuration file will appear at the bottom of the
578 web interface. You can also view these changes, by using the command:
580 .. code-block:: console
582 # proxmox-backup-manager network changes
584 If you would like to cancel all changes at this point, you can either click on
585 the **Revert** button or use the following command:
587 .. code-block:: console
589 # proxmox-backup-manager network revert
591 If you are happy with the changes and would like to write them into the
592 configuration file, select **Apply Configuration**. The corresponding command
595 .. code-block:: console
597 # proxmox-backup-manager network reload
599 You can also configure DNS settings, from the **DNS** section
600 of **Configuration** or by using the ``dns`` subcommand of
601 ``proxmox-backup-manager``.
606 A remote refers to a separate Proxmox Backup Server installation and a user on that
607 installation, from which you can `sync` datastores to a local datastore with a
608 `Sync Job`. You can configure remotes in the web interface, under **Configuration
609 -> Remotes**. Alternatively, you can use the ``remote`` subcommand.
611 .. image:: images/screenshots/pbs-gui-remote-add.png
612 .. image:: images/screenshots/pbs-gui-permissions-add.png
616 To add a remote, you need its hostname or ip, a userid and password on the
617 remote, and its certificate fingerprint. To get the fingerprint, use the
618 ``proxmox-backup-manager cert info`` command on the remote, or navigate to
619 **Dashboard** in the remote's web interface and select **Show Fingerprint**.
621 .. code-block:: console
623 # proxmox-backup-manager cert info |grep Fingerprint
624 Fingerprint (sha256): 64:d3:ff:3a:50:38:53:5a:9b:f7:50:...:ab:fe
626 Using the information specified above, you can add a remote from the **Remotes**
627 configuration panel, or by using the command:
629 .. code-block:: console
631 # proxmox-backup-manager remote create pbs2 --host pbs2.mydomain.example --userid sync@pam --password 'SECRET' --fingerprint 64:d3:ff:3a:50:38:53:5a:9b:f7:50:...:ab:fe
633 Use the ``list``, ``show``, ``update``, ``remove`` subcommands of
634 ``proxmox-backup-manager remote`` to manage your remotes:
636 .. code-block:: console
638 # proxmox-backup-manager remote update pbs2 --host pbs2.example
639 # proxmox-backup-manager remote list
640 ┌──────┬──────────────┬──────────┬───────────────────────────────────────────┬─────────┐
641 │ name │ host │ userid │ fingerprint │ comment │
642 ╞══════╪══════════════╪══════════╪═══════════════════════════════════════════╪═════════╡
643 │ pbs2 │ pbs2.example │ sync@pam │64:d3:ff:3a:50:38:53:5a:9b:f7:50:...:ab:fe │ │
644 └──────┴──────────────┴──────────┴───────────────────────────────────────────┴─────────┘
645 # proxmox-backup-manager remote remove pbs2
651 .. image:: images/screenshots/pbs-gui-syncjob-add.png
655 Sync jobs are configured to pull the contents of a datastore on a **Remote** to a
656 local datastore. You can either start a sync job manually on the GUI or
657 provide it with a :term:`schedule` to run regularly. You can manage sync jobs
658 under **Configuration -> Sync Jobs** in the web interface, or using the
659 ``proxmox-backup-manager sync-job`` command:
661 .. code-block:: console
663 # proxmox-backup-manager sync-job create pbs2-local --remote pbs2 --remote-store local --store local --schedule 'Wed 02:30'
664 # proxmox-backup-manager sync-job update pbs2-local --comment 'offsite'
665 # proxmox-backup-manager sync-job list
666 ┌────────────┬───────┬────────┬──────────────┬───────────┬─────────┐
667 │ id │ store │ remote │ remote-store │ schedule │ comment │
668 ╞════════════╪═══════╪════════╪══════════════╪═══════════╪═════════╡
669 │ pbs2-local │ local │ pbs2 │ local │ Wed 02:30 │ offsite │
670 └────────────┴───────┴────────┴──────────────┴───────────┴─────────┘
671 # proxmox-backup-manager sync-job remove pbs2-local
676 You can monitor and run :ref:`garbage collection <garbage-collection>` on the
677 Proxmox Backup Server using the ``garbage-collection`` subcommand of
678 ``proxmox-backup-manager``. You can use the ``start`` subcommand to manually start garbage
679 collection on an entire datastore and the ``status`` subcommand to see
680 attributes relating to the :ref:`garbage collection <garbage-collection>`.
686 The command line client is called :command:`proxmox-backup-client`.
692 The client uses the following notation to specify a datastore repository
693 on the backup server.
695 [[username@]server:]datastore
697 The default value for ``username`` ist ``root``. If no server is specified,
698 the default is the local host (``localhost``).
700 You can pass the repository with the ``--repository`` command
701 line option, or by setting the ``PBS_REPOSITORY`` environment
705 Environment Variables
706 ~~~~~~~~~~~~~~~~~~~~~
709 The default backup repository.
712 When set, this value is used for the password required for the
715 ``PBS_ENCRYPTION_PASSWORD``
716 When set, this value is used to access the secret encryption key (if
717 protected by password).
719 ``PBS_FINGERPRINT`` When set, this value is used to verify the server
720 certificate (only used if the system CA certificates cannot
721 validate the certificate).
727 Most commands support the ``--output-format`` parameter. It accepts
728 the following values:
730 :``text``: Text format (default). Structured data is rendered as a table.
732 :``json``: JSON (single line).
734 :``json-pretty``: JSON (multiple lines, nicely formatted).
737 Please use the following environment variables to modify output behavior:
739 ``PROXMOX_OUTPUT_FORMAT``
740 Defines the default output format.
742 ``PROXMOX_OUTPUT_NO_BORDER``
743 If set (to any value), do not render table borders.
745 ``PROXMOX_OUTPUT_NO_HEADER``
746 If set (to any value), do not render table headers.
748 .. note:: The ``text`` format is designed to be human readable, and
749 not meant to be parsed by automation tools. Please use the ``json``
750 format if you need to process the output.
753 .. _creating-backups:
758 This section explains how to create a backup from within the machine. This can
759 be a physical host, a virtual machine, or a container. Such backups may contain file
760 and image archives. There are no restrictions in this case.
762 .. note:: If you want to backup virtual machines or containers on Proxmox VE, see :ref:`pve-integration`.
764 For the following example you need to have a backup server set up, working
765 credentials and need to know the repository name.
766 In the following examples we use ``backup-server:store1``.
768 .. code-block:: console
770 # proxmox-backup-client backup root.pxar:/ --repository backup-server:store1
771 Starting backup: host/elsa/2019-12-03T09:35:01Z
773 skip mount point: "/boot/efi"
774 skip mount point: "/dev"
775 skip mount point: "/run"
776 skip mount point: "/sys"
777 Uploaded 12129 chunks in 87 seconds (564 MB/s).
778 End Time: 2019-12-03T10:36:29+01:00
780 This will prompt you for a password and then uploads a file archive named
781 ``root.pxar`` containing all the files in the ``/`` directory.
783 .. Caution:: Please note that the proxmox-backup-client does not
784 automatically include mount points. Instead, you will see a short
785 ``skip mount point`` notice for each of them. The idea is to
786 create a separate file archive for each mounted disk. You can
787 explicitly include them using the ``--include-dev`` option
788 (i.e. ``--include-dev /boot/efi``). You can use this option
789 multiple times for each mount point that should be included.
791 The ``--repository`` option can get quite long and is used by all
792 commands. You can avoid having to enter this value by setting the
793 environment variable ``PBS_REPOSITORY``. Note that if you would like this to remain set
794 over multiple sessions, you should instead add the below line to your
797 .. code-block:: console
799 # export PBS_REPOSITORY=backup-server:store1
801 After this you can execute all commands without specifying the ``--repository``
804 One single backup is allowed to contain more than one archive. For example, if
805 you want to backup two disks mounted at ``/mmt/disk1`` and ``/mnt/disk2``:
807 .. code-block:: console
809 # proxmox-backup-client backup disk1.pxar:/mnt/disk1 disk2.pxar:/mnt/disk2
811 This creates a backup of both disks.
813 The backup command takes a list of backup specifications, which
814 include the archive name on the server, the type of the archive, and the
815 archive source at the client. The format is:
817 <archive-name>.<type>:<source-path>
819 Common types are ``.pxar`` for file archives, and ``.img`` for block
820 device images. To create a backup of a block device run the following command:
822 .. code-block:: console
824 # proxmox-backup-client backup mydata.img:/dev/mylvm/mydata
826 Excluding files/folders from a backup
827 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
829 Sometimes it is desired to exclude certain files or folders from a backup archive.
830 To tell the Proxmox Backup client when and how to ignore files and directories,
831 place a text file called ``.pxarexclude`` in the filesystem hierarchy.
832 Whenever the backup client encounters such a file in a directory, it interprets
833 each line as glob match patterns for files and directories that are to be excluded
836 The file must contain a single glob pattern per line. Empty lines are ignored.
837 The same is true for lines starting with ``#``, which indicates a comment.
838 A ``!`` at the beginning of a line reverses the glob match pattern from an exclusion
839 to an explicit inclusion. This makes it possible to exclude all entries in a
840 directory except for a few single files/subdirectories.
841 Lines ending in ``/`` match only on directories.
842 The directory containing the ``.pxarexclude`` file is considered to be the root of
843 the given patterns. It is only possible to match files in this directory and its subdirectories.
845 ``\`` is used to escape special glob characters.
846 ``?`` matches any single character.
847 ``*`` matches any character, including an empty string.
848 ``**`` is used to match subdirectories. It can be used to, for example, exclude
849 all files ending in ``.tmp`` within the directory or subdirectories with the
850 following pattern ``**/*.tmp``.
851 ``[...]`` matches a single character from any of the provided characters within
852 the brackets. ``[!...]`` does the complementary and matches any single character
853 not contained within the brackets. It is also possible to specify ranges with two
854 characters separated by ``-``. For example, ``[a-z]`` matches any lowercase
855 alphabetic character and ``[0-9]`` matches any one single digit.
857 The order of the glob match patterns defines whether a file is included or
858 excluded, that is to say later entries override previous ones.
859 This is also true for match patterns encountered deeper down the directory tree,
860 which can override a previous exclusion.
861 Be aware that excluded directories will **not** be read by the backup client.
862 Thus, a ``.pxarexclude`` file in an excluded subdirectory will have no effect.
863 ``.pxarexclude`` files are treated as regular files and will be included in the
866 For example, consider the following directory structure:
868 .. code-block:: console
872 . .. .pxarexclude subfolder0 subfolder1
875 . .. file0 file1 file2 file3 .pxarexclude
878 . .. file0 file1 file2 file3
880 The different ``.pxarexclude`` files contain the following:
882 .. code-block:: console
884 # cat folder/.pxarexclude
889 .. code-block:: console
891 # cat folder/subfolder0/.pxarexclude
894 This would exclude ``file1`` and ``file3`` in ``subfolder0`` and all of
895 ``subfolder1`` except ``file2``.
897 Restoring this backup will result in:
899 .. code-block:: console
903 . .. .pxarexclude subfolder0 subfolder1
906 . .. file0 file2 .pxarexclude
914 Proxmox Backup supports client-side encryption with AES-256 in GCM_
915 mode. To set this up, you first need to create an encryption key:
917 .. code-block:: console
919 # proxmox-backup-client key create my-backup.key
920 Encryption Key Password: **************
922 The key is password protected by default. If you do not need this
923 extra protection, you can also create it without a password:
925 .. code-block:: console
927 # proxmox-backup-client key create /path/to/my-backup.key --kdf none
929 Having created this key, it is now possible to create an encrypted backup, by
930 passing the ``--keyfile`` parameter, with the path to the key file.
932 .. code-block:: console
934 # proxmox-backup-client backup etc.pxar:/etc --keyfile /path/to/my-backup.key
936 Encryption Key Password: **************
939 .. Note:: If you do not specify the name of the backup key, the key will be
940 created in the default location
941 ``~/.config/proxmox-backup/encryption-key.json``. ``proxmox-backup-client``
942 will also search this location by default, in case the ``--keyfile``
943 parameter is not specified.
945 You can avoid entering the passwords by setting the environment
946 variables ``PBS_PASSWORD`` and ``PBS_ENCRYPTION_PASSWORD``.
948 Using a master key to store and recover encryption keys
949 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
951 You can also use ``proxmox-backup-client key`` to create an RSA public/private
952 key pair, which can be used to store an encrypted version of the symmetric
953 backup encryption key alongside each backup and recover it later.
955 To set up a master key:
957 1. Create an encryption key for the backup:
959 .. code-block:: console
961 # proxmox-backup-client key create
962 creating default key at: "~/.config/proxmox-backup/encryption-key.json"
963 Encryption Key Password: **********
966 The resulting file will be saved to ``~/.config/proxmox-backup/encryption-key.json``.
968 2. Create an RSA public/private key pair:
970 .. code-block:: console
972 # proxmox-backup-client key create-master-key
973 Master Key Password: *********
976 This will create two files in your current directory, ``master-public.pem``
977 and ``master-private.pem``.
979 3. Import the newly created ``master-public.pem`` public certificate, so that
980 ``proxmox-backup-client`` can find and use it upon backup.
982 .. code-block:: console
984 # proxmox-backup-client key import-master-pubkey /path/to/master-public.pem
985 Imported public master key to "~/.config/proxmox-backup/master-public.pem"
987 4. With all these files in place, run a backup job:
989 .. code-block:: console
991 # proxmox-backup-client backup etc.pxar:/etc
993 The key will be stored in your backup, under the name ``rsa-encrypted.key``.
995 .. Note:: The ``--keyfile`` parameter can be excluded, if the encryption key
996 is in the default path. If you specified another path upon creation, you
997 must pass the ``--keyfile`` parameter.
999 5. To test that everything worked, you can restore the key from the backup:
1001 .. code-block:: console
1003 # proxmox-backup-client restore /path/to/backup/ rsa-encrypted.key /path/to/target
1005 .. Note:: You should not need an encryption key to extract this file. However, if
1006 a key exists at the default location
1007 (``~/.config/proxmox-backup/encryption-key.json``) the program will prompt
1008 you for an encryption key password. Simply moving ``encryption-key.json``
1009 out of this directory will fix this issue.
1011 6. Then, use the previously generated master key to decrypt the file:
1013 .. code-block:: console
1015 # openssl rsautl -decrypt -inkey master-private.pem -in rsa-encrypted.key -out /path/to/target
1016 Enter pass phrase for ./master-private.pem: *********
1018 7. The target file will now contain the encryption key information in plain
1019 text. The success of this can be confirmed by passing the resulting ``json``
1020 file, with the ``--keyfile`` parameter, when decrypting files from the backup.
1022 .. warning:: Without their key, backed up files will be inaccessible. Thus, you should
1023 keep keys ordered and in a place that is separate from the contents being
1024 backed up. It can happen, for example, that you back up an entire system, using
1025 a key on that system. If the system then becomes inaccessable for any reason
1026 and needs to be restored, this will not be possible as the encryption key will be
1027 lost along with the broken system. In preparation for the worst case scenario,
1028 you should consider keeping a paper copy of this key locked away in
1034 The regular creation of backups is a necessary step to avoiding data
1035 loss. More importantly, however, is the restoration. It is good practice to perform
1036 periodic recovery tests to ensure that you can access the data in
1039 First, you need to find the snapshot which you want to restore. The snapshot
1040 command provides a list of all the snapshots on the server:
1042 .. code-block:: console
1044 # proxmox-backup-client snapshots
1045 ┌────────────────────────────────┬─────────────┬────────────────────────────────────┐
1046 │ snapshot │ size │ files │
1047 ╞════════════════════════════════╪═════════════╪════════════════════════════════════╡
1048 │ host/elsa/2019-12-03T09:30:15Z │ 51788646825 │ root.pxar catalog.pcat1 index.json │
1049 ├────────────────────────────────┼─────────────┼────────────────────────────────────┤
1050 │ host/elsa/2019-12-03T09:35:01Z │ 51790622048 │ root.pxar catalog.pcat1 index.json │
1051 ├────────────────────────────────┼─────────────┼────────────────────────────────────┤
1054 You can inspect the catalog to find specific files.
1056 .. code-block:: console
1058 # proxmox-backup-client catalog dump host/elsa/2019-12-03T09:35:01Z
1060 d "./root.pxar.didx/etc/cifs-utils"
1061 l "./root.pxar.didx/etc/cifs-utils/idmap-plugin"
1062 d "./root.pxar.didx/etc/console-setup"
1065 The restore command lets you restore a single archive from the
1068 .. code-block:: console
1070 # proxmox-backup-client restore host/elsa/2019-12-03T09:35:01Z root.pxar /target/path/
1072 To get the contents of any archive, you can restore the ``index.json`` file in the
1073 repository to the target path '-'. This will dump the contents to the standard output.
1075 .. code-block:: console
1077 # proxmox-backup-client restore host/elsa/2019-12-03T09:35:01Z index.json -
1080 Interactive Restores
1081 ^^^^^^^^^^^^^^^^^^^^
1083 If you only want to restore a few individual files, it is often easier
1084 to use the interactive recovery shell.
1086 .. code-block:: console
1088 # proxmox-backup-client catalog shell host/elsa/2019-12-03T09:35:01Z root.pxar
1089 Starting interactive shell
1091 bin boot dev etc home lib lib32
1094 The interactive recovery shell is a minimalistic command line interface that
1095 utilizes the metadata stored in the catalog to quickly list, navigate and
1096 search files in a file archive.
1097 To restore files, you can select them individually or match them with a glob
1100 Using the catalog for navigation reduces the overhead considerably because only
1101 the catalog needs to be downloaded and, optionally, decrypted.
1102 The actual chunks are only accessed if the metadata in the catalog is not enough
1103 or for the actual restore.
1105 Similar to common UNIX shells ``cd`` and ``ls`` are the commands used to change
1106 working directory and list directory contents in the archive.
1107 ``pwd`` shows the full path of the current working directory with respect to the
1110 Being able to quickly search the contents of the archive is a commmonly needed feature.
1111 That's where the catalog is most valuable.
1114 .. code-block:: console
1116 pxar:/ > find etc/**/*.txt --select
1118 pxar:/ > list-selected
1120 pxar:/ > restore-selected /target/path
1123 This will find and print all files ending in ``.txt`` located in ``etc/`` or a
1124 subdirectory and add the corresponding pattern to the list for subsequent restores.
1125 ``list-selected`` shows these patterns and ``restore-selected`` finally restores
1126 all files in the archive matching the patterns to ``/target/path`` on the local
1127 host. This will scan the whole archive.
1129 With ``restore /target/path`` you can restore the sub-archive given by the current
1130 working directory to the local target path ``/target/path`` on your host.
1131 By additionally passing a glob pattern with ``--pattern <glob>``, the restore is
1132 further limited to files matching the pattern.
1135 .. code-block:: console
1138 pxar:/etc/ > restore /target/ --pattern **/*.conf
1141 The above will scan trough all the directories below ``/etc`` and restore all
1142 files ending in ``.conf``.
1144 .. todo:: Explain interactive restore in more detail
1146 Mounting of Archives via FUSE
1147 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
1149 The :term:`FUSE` implementation for the pxar archive allows you to mount a
1150 file archive as a read-only filesystem to a mountpoint on your host.
1152 .. code-block:: console
1154 # proxmox-backup-client mount host/backup-client/2020-01-29T11:29:22Z root.pxar /mnt/mountpoint
1155 # ls /mnt/mountpoint
1156 bin dev home lib32 libx32 media opt root sbin sys usr
1157 boot etc lib lib64 lost+found mnt proc run srv tmp var
1159 This allows you to access the full contents of the archive in a seamless manner.
1161 .. note:: As the FUSE connection needs to fetch and decrypt chunks from the
1162 backup server's datastore, this can cause some additional network and CPU
1163 load on your host, depending on the operations you perform on the mounted
1166 To unmount the filesystem use the ``umount`` command on the mountpoint:
1168 .. code-block:: console
1170 # umount /mnt/mountpoint
1175 The client tool prompts you to enter the logon password as soon as you
1176 want to access the backup server. The server checks your credentials
1177 and responds with a ticket that is valid for two hours. The client
1178 tool automatically stores that ticket and uses it for further requests
1181 You can also manually trigger this login/logout using the login and
1184 .. code-block:: console
1186 # proxmox-backup-client login
1187 Password: **********
1189 To remove the ticket, issue a logout:
1191 .. code-block:: console
1193 # proxmox-backup-client logout
1198 Pruning and Removing Backups
1199 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1201 You can manually delete a backup snapshot using the ``forget``
1204 .. code-block:: console
1206 # proxmox-backup-client forget <snapshot>
1209 .. caution:: This command removes all archives in this backup
1210 snapshot. They will be inaccessible and unrecoverable.
1213 Although manual removal is sometimes required, the ``prune``
1214 command is normally used to systematically delete older backups. Prune lets
1215 you specify which backup snapshots you want to keep. The
1216 following retention options are available:
1219 Keep the last ``<N>`` backup snapshots.
1221 ``--keep-hourly <N>``
1222 Keep backups for the last ``<N>`` hours. If there is more than one
1223 backup for a single hour, only the latest is kept.
1225 ``--keep-daily <N>``
1226 Keep backups for the last ``<N>`` days. If there is more than one
1227 backup for a single day, only the latest is kept.
1229 ``--keep-weekly <N>``
1230 Keep backups for the last ``<N>`` weeks. If there is more than one
1231 backup for a single week, only the latest is kept.
1233 .. note:: Weeks start on Monday and end on Sunday. The software
1234 uses the `ISO week date`_ system and handles weeks at
1235 the end of the year correctly.
1237 ``--keep-monthly <N>``
1238 Keep backups for the last ``<N>`` months. If there is more than one
1239 backup for a single month, only the latest is kept.
1241 ``--keep-yearly <N>``
1242 Keep backups for the last ``<N>`` years. If there is more than one
1243 backup for a single year, only the latest is kept.
1245 The retention options are processed in the order given above. Each option
1246 only covers backups within its time period. The next option does not take care
1247 of already covered backups. It will only consider older backups.
1249 Unfinished and incomplete backups will be removed by the prune command unless
1250 they are newer than the last successful backup. In this case, the last failed
1253 .. code-block:: console
1255 # proxmox-backup-client prune <group> --keep-daily 7 --keep-weekly 4 --keep-monthly 3
1258 You can use the ``--dry-run`` option to test your settings. This only
1259 shows the list of existing snapshots and what actions prune would take.
1261 .. code-block:: console
1263 # proxmox-backup-client prune host/elsa --dry-run --keep-daily 1 --keep-weekly 3
1264 ┌────────────────────────────────┬──────┐
1266 ╞════════════════════════════════╪══════╡
1267 │ host/elsa/2019-12-04T13:20:37Z │ 1 │
1268 ├────────────────────────────────┼──────┤
1269 │ host/elsa/2019-12-03T09:35:01Z │ 0 │
1270 ├────────────────────────────────┼──────┤
1271 │ host/elsa/2019-11-22T11:54:47Z │ 1 │
1272 ├────────────────────────────────┼──────┤
1273 │ host/elsa/2019-11-21T12:36:25Z │ 0 │
1274 ├────────────────────────────────┼──────┤
1275 │ host/elsa/2019-11-10T10:42:20Z │ 1 │
1276 └────────────────────────────────┴──────┘
1278 .. note:: Neither the ``prune`` command nor the ``forget`` command free space
1279 in the chunk-store. The chunk-store still contains the data blocks. To free
1280 space you need to perform :ref:`garbage-collection`.
1283 .. _garbage-collection:
1288 The ``prune`` command removes only the backup index files, not the data
1289 from the datastore. This task is left to the garbage collection
1290 command. It is recommended to carry out garbage collection on a regular basis.
1292 The garbage collection works in two phases. In the first phase, all
1293 data blocks that are still in use are marked. In the second phase,
1294 unused data blocks are removed.
1296 .. note:: This command needs to read all existing backup index files
1297 and touches the complete chunk-store. This can take a long time
1298 depending on the number of chunks and the speed of the underlying
1301 .. note:: The garbage collection will only remove chunks that haven't been used
1302 for at least one day (exactly 24h 5m). This grace period is necessary because
1303 chunks in use are marked by touching the chunk which updates the ``atime``
1304 (access time) property. Filesystems are mounted with the ``relatime`` option
1305 by default. This results in a better performance by only updating the
1306 ``atime`` property if the last access has been at least 24 hours ago. The
1307 downside is, that touching a chunk within these 24 hours will not always
1308 update its ``atime`` property.
1310 Chunks in the grace period will be logged at the end of the garbage
1311 collection task as *Pending removals*.
1313 .. code-block:: console
1315 # proxmox-backup-client garbage-collect
1316 starting garbage collection on store store2
1317 Start GC phase1 (mark used chunks)
1318 Start GC phase2 (sweep unused chunks)
1319 percentage done: 1, chunk count: 219
1320 percentage done: 2, chunk count: 453
1322 percentage done: 99, chunk count: 21188
1323 Removed bytes: 411368505
1325 Original data bytes: 327160886391
1326 Disk bytes: 52767414743 (16 %)
1328 Average chunk size: 2486565
1332 .. todo:: howto run garbage-collection at regular intervalls (cron)
1336 The backup client also comes with a benchmarking tool. This tool measures
1337 various metrics relating to compression and encryption speeds. You can run a
1338 benchmark using the ``benchmark`` subcommand of ``proxmox-backup-client``:
1340 .. code-block:: console
1342 # proxmox-backup-client benchmark
1343 Uploaded 656 chunks in 5 seconds.
1344 Time per request: 7659 microseconds.
1345 TLS speed: 547.60 MB/s
1346 SHA256 speed: 585.76 MB/s
1347 Compression speed: 1923.96 MB/s
1348 Decompress speed: 7885.24 MB/s
1349 AES256/GCM speed: 3974.03 MB/s
1350 ┌───────────────────────────────────┬─────────────────────┐
1352 ╞═══════════════════════════════════╪═════════════════════╡
1353 │ TLS (maximal backup upload speed) │ 547.60 MB/s (93%) │
1354 ├───────────────────────────────────┼─────────────────────┤
1355 │ SHA256 checksum computation speed │ 585.76 MB/s (28%) │
1356 ├───────────────────────────────────┼─────────────────────┤
1357 │ ZStd level 1 compression speed │ 1923.96 MB/s (89%) │
1358 ├───────────────────────────────────┼─────────────────────┤
1359 │ ZStd level 1 decompression speed │ 7885.24 MB/s (98%) │
1360 ├───────────────────────────────────┼─────────────────────┤
1361 │ AES256 GCM encryption speed │ 3974.03 MB/s (104%) │
1362 └───────────────────────────────────┴─────────────────────┘
1364 .. note:: The percentages given in the output table correspond to a
1365 comparison against a Ryzen 7 2700X. The TLS test connects to the
1366 local host, so there is no network involved.
1368 You can also pass the ``--output-format`` parameter to output stats in ``json``,
1369 rather than the default table format.
1371 .. _pve-integration:
1373 `Proxmox VE`_ integration
1374 -------------------------
1376 You need to define a new storage with type 'pbs' on your `Proxmox VE`_
1377 node. The following example uses ``store2`` as storage name, and
1378 assumes the server address is ``localhost``, and you want to connect
1381 .. code-block:: console
1383 # pvesm add pbs store2 --server localhost --datastore store2
1384 # pvesm set store2 --username user1@pbs --password <secret>
1386 .. note:: If you would rather not pass your password as plain text, you can pass
1387 the ``--password`` parameter, without any arguments. This will cause the
1388 program to prompt you for a password upon entering the command.
1390 If your backup server uses a self signed certificate, you need to add
1391 the certificate fingerprint to the configuration. You can get the
1392 fingerprint by running the following command on the backup server:
1394 .. code-block:: console
1396 # proxmox-backup-manager cert info |grep Fingerprint
1397 Fingerprint (sha256): 64:d3:ff:3a:50:38:53:5a:9b:f7:50:...:ab:fe
1399 Please add that fingerprint to your configuration to establish a trust
1402 .. code-block:: console
1404 # pvesm set store2 --fingerprint 64:d3:ff:3a:50:38:53:5a:9b:f7:50:...:ab:fe
1406 After that you should be able to see storage status with:
1408 .. code-block:: console
1410 # pvesm status --storage store2
1411 Name Type Status Total Used Available %
1412 store2 pbs active 3905109820 1336687816 2568422004 34.23%
1416 .. include:: command-line-tools.rst
1418 .. include:: services.rst