]> git.proxmox.com Git - proxmox-backup.git/blame - docs/backup-client.rst
docs: added shell prompt
[proxmox-backup.git] / docs / backup-client.rst
CommitLineData
04e24b14
DW
1Backup Client Usage
2===================
3
34407477 4The command-line client for `Proxmox Backup`_ Server is called
5fb852af 5:command:`proxmox-backup-client`.
04e24b14 6
654c56e0 7.. _client_repository:
04e24b14 8
fd8b00ae
TL
9Backup Repository Locations
10---------------------------
04e24b14 11
5fb852af
DW
12The client uses the following format to specify a datastore repository
13on the backup server (where username is specified in the form of user@realm):
04e24b14
DW
14
15 [[username@]server[:port]:]datastore
16
4aef06f1 17The default value for ``username`` is ``root@pam``. If no server is specified,
04e24b14
DW
18the default is the local host (``localhost``).
19
5fb852af
DW
20You can specify a port if your backup server is only reachable on a non-default
21port (for example, with NAT and port forwarding configurations).
04e24b14 22
5fb852af 23Note that if the server uses an IPv6 address, you have to write it with square
4aef06f1 24brackets (for example, `[fe80::01]`).
04e24b14 25
8c83b75a 26You can pass the repository with the ``--repository`` command-line option, or
4aef06f1 27by setting the ``PBS_REPOSITORY`` environment variable.
04e24b14 28
ca8cf671
DC
29The web interface provides copyable repository text in the datastore summary
30with the `Show Connection Information` button.
31
5fb852af
DW
32Below are some examples of valid repositories and their corresponding real
33values:
04e24b14 34
c0af05e1 35================================ ================== ================== ===========
4aef06f1 36Example User Host:Port Datastore
c0af05e1 37================================ ================== ================== ===========
4aef06f1
TL
38mydatastore ``root@pam`` localhost:8007 mydatastore
39myhostname:mydatastore ``root@pam`` myhostname:8007 mydatastore
40user@pbs@myhostname:mydatastore ``user@pbs`` myhostname:8007 mydatastore
4f04ecb2 41user\@pbs!token@host:store ``user@pbs!token`` host:8007 store
4aef06f1
TL
42192.168.55.55:1234:mydatastore ``root@pam`` 192.168.55.55:1234 mydatastore
43[ff80::51]:mydatastore ``root@pam`` [ff80::51]:8007 mydatastore
44[ff80::51]:1234:mydatastore ``root@pam`` [ff80::51]:1234 mydatastore
c0af05e1 45================================ ================== ================== ===========
04e24b14
DW
46
47Environment Variables
48---------------------
49
50``PBS_REPOSITORY``
51 The default backup repository.
52
53``PBS_PASSWORD``
5fb852af
DW
54 When set, this value is used as the password for the backup server.
55 You can also set this to an API token secret.
04e24b14 56
65ab4ca9
TL
57``PBS_PASSWORD_FD``, ``PBS_PASSWORD_FILE``, ``PBS_PASSWORD_CMD``
58 Like ``PBS_PASSWORD``, but read data from an open file descriptor, a file
59 name or from the `stdout` of a command, respectively. The first defined
60 environment variable from the order above is preferred.
99a73fad 61
04e24b14
DW
62``PBS_ENCRYPTION_PASSWORD``
63 When set, this value is used to access the secret encryption key (if
64 protected by password).
65
65ab4ca9
TL
66``PBS_ENCRYPTION_PASSWORD_FD``, ``PBS_ENCRYPTION_PASSWORD_FILE``, ``PBS_ENCRYPTION_PASSWORD_CMD``
67 Like ``PBS_ENCRYPTION_PASSWORD``, but read data from an open file descriptor,
68 a file name or from the `stdout` of a command, respectively. The first
69 defined environment variable from the order above is preferred.
99a73fad 70
5fb852af
DW
71``PBS_FINGERPRINT``
72 When set, this value is used to verify the server certificate (only used if
73 the system CA certificates cannot validate the certificate).
04e24b14 74
fc65ec43
SH
75``ALL_PROXY``
76 When set, the client uses the specified HTTP proxy for all connections to the
77 backup server. Currently only HTTP proxies are supported. Valid proxy
78 configurations have the following format:
79 `[http://][user:password@]<host>[:port]`. Default `port` is 1080, if not
80 otherwise specified.
81
82
83.. Note:: The recommended solution for shielding hosts is using tunnels such as
84 wireguard, instead of using an HTTP proxy.
85
04e24b14 86
5fb852af 87.. Note:: Passwords must be valid UTF-8 and may not contain newlines. For your
c10a6755 88 convenience, Proxmox Backup Server only uses the first line as password, so
5fb852af 89 you can add arbitrary comments after the first newline.
99a73fad
DM
90
91
04e24b14
DW
92Output Format
93-------------
94
f458e97f 95.. include:: output-format.rst
04e24b14
DW
96
97
ec1ae7e6 98.. _client_creating_backups:
04e24b14
DW
99
100Creating Backups
101----------------
102
103This section explains how to create a backup from within the machine. This can
5fb852af
DW
104be a physical host, a virtual machine, or a container. Such backups may contain
105file and image archives. There are no restrictions in this case.
04e24b14 106
5fb852af
DW
107.. Note:: If you want to backup virtual machines or containers on Proxmox VE,
108 see :ref:`pve-integration`.
04e24b14 109
5fb852af
DW
110For the following example, you need to have a backup server set up, have working
111credentials, and know the repository name.
112In the following examples, we use ``backup-server:store1``.
04e24b14
DW
113
114.. code-block:: console
115
116 # proxmox-backup-client backup root.pxar:/ --repository backup-server:store1
117 Starting backup: host/elsa/2019-12-03T09:35:01Z
118 Client name: elsa
119 skip mount point: "/boot/efi"
120 skip mount point: "/dev"
121 skip mount point: "/run"
122 skip mount point: "/sys"
123 Uploaded 12129 chunks in 87 seconds (564 MB/s).
124 End Time: 2019-12-03T10:36:29+01:00
125
5fb852af 126This will prompt you for a password, then upload a file archive named
04e24b14
DW
127``root.pxar`` containing all the files in the ``/`` directory.
128
5fb852af 129.. Caution:: Please note that proxmox-backup-client does not
04e24b14 130 automatically include mount points. Instead, you will see a short
5fb852af 131 ``skip mount point`` message for each of them. The idea is to
04e24b14
DW
132 create a separate file archive for each mounted disk. You can
133 explicitly include them using the ``--include-dev`` option
134 (i.e. ``--include-dev /boot/efi``). You can use this option
135 multiple times for each mount point that should be included.
136
a3b10267
TL
137The ``--repository`` option can get quite long and is used by all commands. You
138can avoid having to enter this value by setting the environment variable
139``PBS_REPOSITORY``. Note that if you would like this to remain set over
140multiple sessions, you should instead add the below line to your ``.bashrc``
141file.
04e24b14
DW
142
143.. code-block:: console
144
145 # export PBS_REPOSITORY=backup-server:store1
146
5fb852af
DW
147After this, you can execute all commands without having to specify the
148``--repository`` option.
04e24b14 149
5fb852af
DW
150A single backup is allowed to contain more than one archive. For example, if
151you want to back up two disks mounted at ``/mnt/disk1`` and ``/mnt/disk2``:
04e24b14
DW
152
153.. code-block:: console
154
155 # proxmox-backup-client backup disk1.pxar:/mnt/disk1 disk2.pxar:/mnt/disk2
156
157This creates a backup of both disks.
158
6481fd24 159If you want to use a namespace for the backup target, you can add the `--ns`
f71a4ce6
TL
160parameter:
161
162.. code-block:: console
163
164 # proxmox-backup-client backup disk1.pxar:/mnt/disk1 disk2.pxar:/mnt/disk2 --ns a/b/c
165
166The backup command takes a list of backup specifications, which include the
167archive name on the server, the type of the archive, and the archive source at
168the client. The format is:
04e24b14
DW
169
170 <archive-name>.<type>:<source-path>
171
5fb852af
DW
172Common types are ``.pxar`` for file archives and ``.img`` for block
173device images. To create a backup of a block device, run the following command:
04e24b14
DW
174
175.. code-block:: console
176
177 # proxmox-backup-client backup mydata.img:/dev/mylvm/mydata
178
179
5fb852af
DW
180Excluding Files/Directories from a Backup
181~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
04e24b14 182
a3b10267
TL
183Sometimes it is desired to exclude certain files or directories from a backup
184archive. To tell the Proxmox Backup client when and how to ignore files and
185directories, place a text file named ``.pxarexclude`` in the filesystem
186hierarchy. Whenever the backup client encounters such a file in a directory,
187it interprets each line as a glob match pattern for files and directories that
188are to be excluded from the backup.
04e24b14 189
5fb852af
DW
190The file must contain a single glob pattern per line. Empty lines and lines
191starting with ``#`` (indicating a comment) are ignored.
a3b10267
TL
192A ``!`` at the beginning of a line reverses the glob match pattern from an
193exclusion to an explicit inclusion. This makes it possible to exclude all
194entries in a directory except for a few single files/subdirectories.
04e24b14 195Lines ending in ``/`` match only on directories.
a3b10267
TL
196The directory containing the ``.pxarexclude`` file is considered to be the root
197of the given patterns. It is only possible to match files in this directory and
198its subdirectories.
04e24b14 199
b2df21bb
FE
200.. Note:: Patterns without a leading ``/`` will also match in subdirectories,
201 while patterns with a leading ``/`` will only match in the current directory.
202
04e24b14
DW
203``\`` is used to escape special glob characters.
204``?`` matches any single character.
205``*`` matches any character, including an empty string.
5fb852af
DW
206``**`` is used to match current directory and subdirectories. For example, with
207the pattern ``**/*.tmp``, it would exclude all files ending in ``.tmp`` within
208a directory and its subdirectories.
04e24b14 209``[...]`` matches a single character from any of the provided characters within
a3b10267
TL
210the brackets. ``[!...]`` does the complementary and matches any single
211character not contained within the brackets. It is also possible to specify
212ranges with two characters separated by ``-``. For example, ``[a-z]`` matches
213any lowercase alphabetic character, and ``[0-9]`` matches any single digit.
04e24b14
DW
214
215The order of the glob match patterns defines whether a file is included or
5fb852af 216excluded, that is to say, later entries override earlier ones.
a3b10267
TL
217This is also true for match patterns encountered deeper down the directory
218tree, which can override a previous exclusion.
5fb852af
DW
219
220.. Note:: Excluded directories will **not** be read by the backup client. Thus,
221 a ``.pxarexclude`` file in an excluded subdirectory will have no effect.
222 ``.pxarexclude`` files are treated as regular files and will be included in
223 the backup archive.
04e24b14
DW
224
225For example, consider the following directory structure:
226
227.. code-block:: console
228
229 # ls -aR folder
230 folder/:
231 . .. .pxarexclude subfolder0 subfolder1
232
233 folder/subfolder0:
234 . .. file0 file1 file2 file3 .pxarexclude
235
236 folder/subfolder1:
237 . .. file0 file1 file2 file3
238
239The different ``.pxarexclude`` files contain the following:
240
241.. code-block:: console
242
243 # cat folder/.pxarexclude
244 /subfolder0/file1
245 /subfolder1/*
246 !/subfolder1/file2
247
248.. code-block:: console
249
250 # cat folder/subfolder0/.pxarexclude
251 file3
252
253This would exclude ``file1`` and ``file3`` in ``subfolder0`` and all of
254``subfolder1`` except ``file2``.
255
256Restoring this backup will result in:
257
258.. code-block:: console
259
bc9b8990 260 # ls -aR restored
04e24b14
DW
261 restored/:
262 . .. .pxarexclude subfolder0 subfolder1
263
264 restored/subfolder0:
265 . .. file0 file2 .pxarexclude
266
267 restored/subfolder1:
268 . .. file2
269
93f2568f
GG
270The same syntax can also be used directly in the cli with the ``--exclude``
271parameter. For example:
272
273.. code-block:: console
274
275 # proxmox-backup-client backup.pxar:./linux --exclude /usr
276
277Multiple paths can be excluded like this:
278
279.. code-block:: console
280
281 # proxmox-backup-client backup.pxar:./linux --exclude=/usr --exclude=/rust
04e24b14 282
ec1ae7e6 283.. _client_encryption:
d9f365d7 284
04e24b14
DW
285Encryption
286----------
287
288Proxmox Backup supports client-side encryption with AES-256 in GCM_
289mode. To set this up, you first need to create an encryption key:
290
291.. code-block:: console
292
293 # proxmox-backup-client key create my-backup.key
294 Encryption Key Password: **************
295
296The key is password protected by default. If you do not need this
297extra protection, you can also create it without a password:
298
299.. code-block:: console
300
301 # proxmox-backup-client key create /path/to/my-backup.key --kdf none
302
303Having created this key, it is now possible to create an encrypted backup, by
304passing the ``--keyfile`` parameter, with the path to the key file.
305
306.. code-block:: console
307
308 # proxmox-backup-client backup etc.pxar:/etc --keyfile /path/to/my-backup.key
309 Password: *********
310 Encryption Key Password: **************
311 ...
312
313.. Note:: If you do not specify the name of the backup key, the key will be
314 created in the default location
315 ``~/.config/proxmox-backup/encryption-key.json``. ``proxmox-backup-client``
316 will also search this location by default, in case the ``--keyfile``
317 parameter is not specified.
318
319You can avoid entering the passwords by setting the environment
320variables ``PBS_PASSWORD`` and ``PBS_ENCRYPTION_PASSWORD``.
321
322
5fb852af 323Using a Master Key to Store and Recover Encryption Keys
04e24b14
DW
324~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
325
326You can also use ``proxmox-backup-client key`` to create an RSA public/private
327key pair, which can be used to store an encrypted version of the symmetric
328backup encryption key alongside each backup and recover it later.
329
330To set up a master key:
331
3321. Create an encryption key for the backup:
333
334 .. code-block:: console
335
336 # proxmox-backup-client key create
337 creating default key at: "~/.config/proxmox-backup/encryption-key.json"
338 Encryption Key Password: **********
339 ...
340
341 The resulting file will be saved to ``~/.config/proxmox-backup/encryption-key.json``.
342
3432. Create an RSA public/private key pair:
344
345 .. code-block:: console
346
347 # proxmox-backup-client key create-master-key
348 Master Key Password: *********
349 ...
350
351 This will create two files in your current directory, ``master-public.pem``
352 and ``master-private.pem``.
353
3543. Import the newly created ``master-public.pem`` public certificate, so that
355 ``proxmox-backup-client`` can find and use it upon backup.
356
357 .. code-block:: console
358
359 # proxmox-backup-client key import-master-pubkey /path/to/master-public.pem
360 Imported public master key to "~/.config/proxmox-backup/master-public.pem"
361
3624. With all these files in place, run a backup job:
363
364 .. code-block:: console
365
366 # proxmox-backup-client backup etc.pxar:/etc
367
368 The key will be stored in your backup, under the name ``rsa-encrypted.key``.
369
370 .. Note:: The ``--keyfile`` parameter can be excluded, if the encryption key
371 is in the default path. If you specified another path upon creation, you
372 must pass the ``--keyfile`` parameter.
373
3745. To test that everything worked, you can restore the key from the backup:
375
376 .. code-block:: console
377
378 # proxmox-backup-client restore /path/to/backup/ rsa-encrypted.key /path/to/target
379
380 .. Note:: You should not need an encryption key to extract this file. However, if
381 a key exists at the default location
382 (``~/.config/proxmox-backup/encryption-key.json``) the program will prompt
383 you for an encryption key password. Simply moving ``encryption-key.json``
384 out of this directory will fix this issue.
385
3866. Then, use the previously generated master key to decrypt the file:
387
388 .. code-block:: console
389
2ea5abcd
FG
390 # proxmox-backup-client key import-with-master-key /path/to/target --master-keyfile /path/to/master-private.pem --encrypted-keyfile /path/to/rsa-encrypted.key
391 Master Key Password: ******
392 New Password: ******
393 Verify Password: ******
04e24b14
DW
394
3957. The target file will now contain the encryption key information in plain
396 text. The success of this can be confirmed by passing the resulting ``json``
397 file, with the ``--keyfile`` parameter, when decrypting files from the backup.
398
399.. warning:: Without their key, backed up files will be inaccessible. Thus, you should
400 keep keys ordered and in a place that is separate from the contents being
401 backed up. It can happen, for example, that you back up an entire system, using
cdbc18fc 402 a key on that system. If the system then becomes inaccessible for any reason
5fb852af 403 and needs to be restored, this will not be possible, as the encryption key will be
4accbc58
DW
404 lost along with the broken system.
405
c5608cf8
DW
406It is recommended that you keep your master key safe, but easily accessible, in
407order for quick disaster recovery. For this reason, the best place to store it
408is in your password manager, where it is immediately recoverable. As a backup to
6aa3eeb5 409this, you should also save the key to a USB flash drive and store that in a secure
c5608cf8
DW
410place. This way, it is detached from any system, but is still easy to recover
411from, in case of emergency. Finally, in preparation for the worst case scenario,
412you should also consider keeping a paper copy of your master key locked away in
413a safe place. The ``paperkey`` subcommand can be used to create a QR encoded
414version of your master key. The following command sends the output of the
415``paperkey`` command to a text file, for easy printing.
4accbc58
DW
416
417.. code-block:: console
418
419 proxmox-backup-client key paperkey --output-format text > qrkey.txt
04e24b14
DW
420
421
422Restoring Data
423--------------
424
5fb852af
DW
425The regular creation of backups is a necessary step in avoiding data loss. More
426importantly, however, is the restoration. It is good practice to perform
427periodic recovery tests to ensure that you can access the data in case of
428disaster.
04e24b14
DW
429
430First, you need to find the snapshot which you want to restore. The snapshot
abf9b6da 431list command provides a list of all the snapshots on the server:
04e24b14
DW
432
433.. code-block:: console
434
abf9b6da 435 # proxmox-backup-client snapshot list
04e24b14
DW
436 ┌────────────────────────────────┬─────────────┬────────────────────────────────────┐
437 │ snapshot │ size │ files │
438 ╞════════════════════════════════╪═════════════╪════════════════════════════════════╡
439 │ host/elsa/2019-12-03T09:30:15Z │ 51788646825 │ root.pxar catalog.pcat1 index.json │
440 ├────────────────────────────────┼─────────────┼────────────────────────────────────┤
441 │ host/elsa/2019-12-03T09:35:01Z │ 51790622048 │ root.pxar catalog.pcat1 index.json │
442 ├────────────────────────────────┼─────────────┼────────────────────────────────────┤
443 ...
444
f71a4ce6
TL
445
446.. tip:: List will by default only output the backup snapshots of the root
447 namespace itself. To list backups from another namespace use the ``--ns
448 <ns>`` option
449
04e24b14
DW
450You can inspect the catalog to find specific files.
451
452.. code-block:: console
453
454 # proxmox-backup-client catalog dump host/elsa/2019-12-03T09:35:01Z
455 ...
456 d "./root.pxar.didx/etc/cifs-utils"
457 l "./root.pxar.didx/etc/cifs-utils/idmap-plugin"
458 d "./root.pxar.didx/etc/console-setup"
459 ...
460
461The restore command lets you restore a single archive from the
462backup.
463
464.. code-block:: console
465
466 # proxmox-backup-client restore host/elsa/2019-12-03T09:35:01Z root.pxar /target/path/
467
468To get the contents of any archive, you can restore the ``index.json`` file in the
469repository to the target path '-'. This will dump the contents to the standard output.
470
471.. code-block:: console
472
473 # proxmox-backup-client restore host/elsa/2019-12-03T09:35:01Z index.json -
474
475
476Interactive Restores
477~~~~~~~~~~~~~~~~~~~~
478
479If you only want to restore a few individual files, it is often easier
480to use the interactive recovery shell.
481
482.. code-block:: console
483
484 # proxmox-backup-client catalog shell host/elsa/2019-12-03T09:35:01Z root.pxar
485 Starting interactive shell
486 pxar:/ > ls
487 bin boot dev etc home lib lib32
488 ...
489
8c83b75a 490The interactive recovery shell is a minimal command-line interface that
04e24b14 491utilizes the metadata stored in the catalog to quickly list, navigate and
5fb852af 492search for files in a file archive.
04e24b14
DW
493To restore files, you can select them individually or match them with a glob
494pattern.
495
496Using the catalog for navigation reduces the overhead considerably because only
497the catalog needs to be downloaded and, optionally, decrypted.
5fb852af
DW
498The actual chunks are only accessed if the metadata in the catalog is
499insufficient or for the actual restore.
04e24b14 500
5fb852af 501Similar to common UNIX shells, ``cd`` and ``ls`` are the commands used to change
04e24b14
DW
502working directory and list directory contents in the archive.
503``pwd`` shows the full path of the current working directory with respect to the
504archive root.
505
5fb852af
DW
506The ability to quickly search the contents of the archive is a commonly required
507feature. That's where the catalog is most valuable. For example:
04e24b14
DW
508
509.. code-block:: console
510
511 pxar:/ > find etc/**/*.txt --select
512 "/etc/X11/rgb.txt"
513 pxar:/ > list-selected
514 etc/**/*.txt
515 pxar:/ > restore-selected /target/path
516 ...
517
5fb852af
DW
518This will find and print all files ending in ``.txt`` located in ``etc/`` or its
519subdirectories, and add the corresponding pattern to the list for subsequent restores.
04e24b14
DW
520``list-selected`` shows these patterns and ``restore-selected`` finally restores
521all files in the archive matching the patterns to ``/target/path`` on the local
522host. This will scan the whole archive.
523
985e84e3
DW
524The ``restore`` command can be used to restore all the files contained within
525the backup archive. This is most helpful when paired with the ``--pattern
526<glob>`` option, as it allows you to restore all files matching a specific
527pattern. For example, if you wanted to restore configuration files
528located in ``/etc``, you could do the following:
04e24b14
DW
529
530.. code-block:: console
531
985e84e3 532 pxar:/ > restore target/ --pattern etc/**/*.conf
04e24b14
DW
533 ...
534
5f3b2330 535The above will scan through all the directories below ``/etc`` and restore all
04e24b14
DW
536files ending in ``.conf``.
537
538.. todo:: Explain interactive restore in more detail
539
540Mounting of Archives via FUSE
541~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
542
543The :term:`FUSE` implementation for the pxar archive allows you to mount a
5fb852af 544file archive as a read-only filesystem to a mount point on your host.
04e24b14
DW
545
546.. code-block:: console
547
548 # proxmox-backup-client mount host/backup-client/2020-01-29T11:29:22Z root.pxar /mnt/mountpoint
549 # ls /mnt/mountpoint
550 bin dev home lib32 libx32 media opt root sbin sys usr
551 boot etc lib lib64 lost+found mnt proc run srv tmp var
552
553This allows you to access the full contents of the archive in a seamless manner.
554
555.. note:: As the FUSE connection needs to fetch and decrypt chunks from the
556 backup server's datastore, this can cause some additional network and CPU
557 load on your host, depending on the operations you perform on the mounted
558 filesystem.
559
5fb852af 560To unmount the filesystem, use the ``umount`` command on the mount point:
04e24b14
DW
561
562.. code-block:: console
563
564 # umount /mnt/mountpoint
565
566Login and Logout
567----------------
568
5fb852af 569The client tool prompts you to enter the login password as soon as you
04e24b14
DW
570want to access the backup server. The server checks your credentials
571and responds with a ticket that is valid for two hours. The client
572tool automatically stores that ticket and uses it for further requests
573to this server.
574
575You can also manually trigger this login/logout using the login and
576logout commands:
577
578.. code-block:: console
579
580 # proxmox-backup-client login
581 Password: **********
582
583To remove the ticket, issue a logout:
584
585.. code-block:: console
586
587 # proxmox-backup-client logout
588
589
08379a21
DW
590.. _changing-backup-owner:
591
592Changing the Owner of a Backup Group
593------------------------------------
594
595By default, the owner of a backup group is the user which was used to originally
596create that backup group (or in the case of sync jobs, ``root@pam``). This
597means that if a user ``mike@pbs`` created a backup, another user ``john@pbs``
5fb852af 598can not be used to create backups in that same backup group. In case you want
08379a21
DW
599to change the owner of a backup, you can do so with the below command, using a
600user that has ``Datastore.Modify`` privileges on the datastore.
601
602.. code-block:: console
603
604 # proxmox-backup-client change-owner vm/103 john@pbs
605
606This can also be done from within the web interface, by navigating to the
a3b10267
TL
607`Content` section of the datastore that contains the backup group and selecting
608the user icon under the `Actions` column. Common cases for this could be to
609change the owner of a sync job from ``root@pam``, or to repurpose a backup
610group.
08379a21
DW
611
612
4c3efb53 613.. _backup-pruning:
04e24b14
DW
614
615Pruning and Removing Backups
616----------------------------
617
f71a4ce6 618You can manually delete a backup snapshot using the ``forget`` command:
04e24b14
DW
619
620.. code-block:: console
621
abf9b6da 622 # proxmox-backup-client snapshot forget <snapshot>
04e24b14
DW
623
624
f71a4ce6
TL
625.. caution:: This command removes all archives in this backup snapshot. They
626 will be inaccessible and *unrecoverable*.
627
628Don't forget to add the namespace ``--ns`` parameter if you want to forget a
629snapshot that is contained in the root namespace:
630
631.. code-block:: console
632
633 # proxmox-backup-client snapshot forget <snapshot> --ns <ns>
634
635
04e24b14
DW
636
637
638Although manual removal is sometimes required, the ``prune``
639command is normally used to systematically delete older backups. Prune lets
640you specify which backup snapshots you want to keep. The
641following retention options are available:
642
643``--keep-last <N>``
644 Keep the last ``<N>`` backup snapshots.
645
646``--keep-hourly <N>``
8df3a17c
FE
647 Keep backups for the last ``<N>`` hours. If there is more than one backup for
648 a single hour, only the latest is kept. Hours without backups do not count.
04e24b14
DW
649
650``--keep-daily <N>``
8df3a17c
FE
651 Keep backups for the last ``<N>`` days. If there is more than one backup for a
652 single day, only the latest is kept. Days without backups do not count.
04e24b14
DW
653
654``--keep-weekly <N>``
8df3a17c
FE
655 Keep backups for the last ``<N>`` weeks. If there is more than one backup for
656 a single week, only the latest is kept. Weeks without backup do not count.
04e24b14
DW
657
658 .. note:: Weeks start on Monday and end on Sunday. The software
659 uses the `ISO week date`_ system and handles weeks at
660 the end of the year correctly.
661
662``--keep-monthly <N>``
8df3a17c
FE
663 Keep backups for the last ``<N>`` months. If there is more than one backup for
664 a single month, only the latest is kept. Months without backups do not count.
04e24b14
DW
665
666``--keep-yearly <N>``
8df3a17c
FE
667 Keep backups for the last ``<N>`` years. If there is more than one backup for
668 a single year, only the latest is kept. Years without backups do not count.
04e24b14
DW
669
670The retention options are processed in the order given above. Each option
671only covers backups within its time period. The next option does not take care
672of already covered backups. It will only consider older backups.
673
674Unfinished and incomplete backups will be removed by the prune command unless
675they are newer than the last successful backup. In this case, the last failed
676backup is retained.
677
678.. code-block:: console
679
680 # proxmox-backup-client prune <group> --keep-daily 7 --keep-weekly 4 --keep-monthly 3
681
682
683You can use the ``--dry-run`` option to test your settings. This only
684shows the list of existing snapshots and what actions prune would take.
685
686.. code-block:: console
687
688 # proxmox-backup-client prune host/elsa --dry-run --keep-daily 1 --keep-weekly 3
689 ┌────────────────────────────────┬──────┐
690 │ snapshot │ keep │
691 ╞════════════════════════════════╪══════╡
692 │ host/elsa/2019-12-04T13:20:37Z │ 1 │
693 ├────────────────────────────────┼──────┤
694 │ host/elsa/2019-12-03T09:35:01Z │ 0 │
695 ├────────────────────────────────┼──────┤
696 │ host/elsa/2019-11-22T11:54:47Z │ 1 │
697 ├────────────────────────────────┼──────┤
698 │ host/elsa/2019-11-21T12:36:25Z │ 0 │
699 ├────────────────────────────────┼──────┤
700 │ host/elsa/2019-11-10T10:42:20Z │ 1 │
701 └────────────────────────────────┴──────┘
702
703.. note:: Neither the ``prune`` command nor the ``forget`` command free space
704 in the chunk-store. The chunk-store still contains the data blocks. To free
a98e2287 705 space you need to perform :ref:`client_garbage-collection`.
04e24b14 706
38517ca0
DC
707It is also possible to protect single snapshots from being pruned or deleted:
708
709.. code-block:: console
710
711 # proxmox-backup-client snapshot protected update <snapshot> true
712
713This will set the protected flag on the snapshot and prevent pruning or manual
6481fd24 714deletion of this snapshot until the flag is removed again with:
38517ca0
DC
715
716.. code-block:: console
717
718 # proxmox-backup-client snapshot protected update <snapshot> false
719
6481fd24
DW
720When a group with a protected snapshot is deleted, only the non-protected
721ones are removed, and the rest will remain.
38517ca0
DC
722
723.. note:: This flag will not be synced when using pull or sync jobs. If you
6481fd24 724 want to protect a synced snapshot, you have to do this again manually on
56d3b59c 725 the target backup server.
04e24b14 726
ec1ae7e6 727.. _client_garbage-collection:
04e24b14
DW
728
729Garbage Collection
730------------------
731
732The ``prune`` command removes only the backup index files, not the data
733from the datastore. This task is left to the garbage collection
734command. It is recommended to carry out garbage collection on a regular basis.
735
736The garbage collection works in two phases. In the first phase, all
737data blocks that are still in use are marked. In the second phase,
738unused data blocks are removed.
739
740.. note:: This command needs to read all existing backup index files
741 and touches the complete chunk-store. This can take a long time
742 depending on the number of chunks and the speed of the underlying
743 disks.
744
745.. note:: The garbage collection will only remove chunks that haven't been used
746 for at least one day (exactly 24h 5m). This grace period is necessary because
747 chunks in use are marked by touching the chunk which updates the ``atime``
748 (access time) property. Filesystems are mounted with the ``relatime`` option
749 by default. This results in a better performance by only updating the
750 ``atime`` property if the last access has been at least 24 hours ago. The
5fb852af 751 downside is that touching a chunk within these 24 hours will not always
04e24b14
DW
752 update its ``atime`` property.
753
754 Chunks in the grace period will be logged at the end of the garbage
755 collection task as *Pending removals*.
756
757.. code-block:: console
758
759 # proxmox-backup-client garbage-collect
760 starting garbage collection on store store2
761 Start GC phase1 (mark used chunks)
762 Start GC phase2 (sweep unused chunks)
763 percentage done: 1, chunk count: 219
764 percentage done: 2, chunk count: 453
765 ...
766 percentage done: 99, chunk count: 21188
767 Removed bytes: 411368505
768 Removed chunks: 203
769 Original data bytes: 327160886391
770 Disk bytes: 52767414743 (16 %)
771 Disk chunks: 21221
772 Average chunk size: 2486565
773 TASK OK
774
35e80f3e 775Garbage collection can also be scheduled using ``proxmox-backup-manager`` or
5fb852af 776from the Proxmox Backup Server's web interface.
04e24b14
DW
777
778Benchmarking
779------------
780
781The backup client also comes with a benchmarking tool. This tool measures
654c56e0
TL
782various metrics relating to compression and encryption speeds. If a Proxmox
783Backup repository (remote or local) is specified, the TLS upload speed will get
784measured too.
785
786You can run a benchmark using the ``benchmark`` subcommand of
787``proxmox-backup-client``:
788
789.. note:: The TLS speed test is only included if a :ref:`backup server
790 repository is specified <client_repository>`.
04e24b14
DW
791
792.. code-block:: console
793
794 # proxmox-backup-client benchmark
81f5d03e
DM
795 Uploaded 1517 chunks in 5 seconds.
796 Time per request: 3309 microseconds.
797 TLS speed: 1267.41 MB/s
798 SHA256 speed: 2066.73 MB/s
799 Compression speed: 775.11 MB/s
800 Decompress speed: 1233.35 MB/s
801 AES256/GCM speed: 3688.27 MB/s
802 Verify speed: 783.43 MB/s
04e24b14
DW
803 ┌───────────────────────────────────┬─────────────────────┐
804 │ Name │ Value │
805 ╞═══════════════════════════════════╪═════════════════════╡
81f5d03e 806 │ TLS (maximal backup upload speed) │ 1267.41 MB/s (103%) │
04e24b14 807 ├───────────────────────────────────┼─────────────────────┤
81f5d03e 808 │ SHA256 checksum computation speed │ 2066.73 MB/s (102%) │
04e24b14 809 ├───────────────────────────────────┼─────────────────────┤
81f5d03e 810 │ ZStd level 1 compression speed │ 775.11 MB/s (103%) │
04e24b14 811 ├───────────────────────────────────┼─────────────────────┤
81f5d03e 812 │ ZStd level 1 decompression speed │ 1233.35 MB/s (103%) │
04e24b14 813 ├───────────────────────────────────┼─────────────────────┤
81f5d03e
DM
814 │ Chunk verification speed │ 783.43 MB/s (103%) │
815 ├───────────────────────────────────┼─────────────────────┤
816 │ AES256 GCM encryption speed │ 3688.27 MB/s (101%) │
04e24b14
DW
817 └───────────────────────────────────┴─────────────────────┘
818
81f5d03e 819
04e24b14 820.. note:: The percentages given in the output table correspond to a
654c56e0 821 comparison against a Ryzen 7 2700X.
04e24b14
DW
822
823You can also pass the ``--output-format`` parameter to output stats in ``json``,
824rather than the default table format.