]> git.proxmox.com Git - proxmox-backup.git/blame - docs/backup-client.rst
docs: simplify list of ENV var alternative
[proxmox-backup.git] / docs / backup-client.rst
CommitLineData
04e24b14
DW
1Backup Client Usage
2===================
3
4The command line client is called :command:`proxmox-backup-client`.
5
654c56e0 6.. _client_repository:
04e24b14 7
fd8b00ae
TL
8Backup Repository Locations
9---------------------------
04e24b14
DW
10
11The client uses the following notation to specify a datastore repository
12on the backup server.
13
14 [[username@]server[:port]:]datastore
15
4aef06f1 16The default value for ``username`` is ``root@pam``. If no server is specified,
04e24b14
DW
17the default is the local host (``localhost``).
18
19You can specify a port if your backup server is only reachable on a different
20port (e.g. with NAT and port forwarding).
21
4aef06f1
TL
22Note that if the server is an IPv6 address, you have to write it with square
23brackets (for example, `[fe80::01]`).
04e24b14 24
4aef06f1
TL
25You can pass the repository with the ``--repository`` command line option, or
26by setting the ``PBS_REPOSITORY`` environment variable.
04e24b14
DW
27
28Here some examples of valid repositories and the real values
29
c0af05e1 30================================ ================== ================== ===========
4aef06f1 31Example User Host:Port Datastore
c0af05e1 32================================ ================== ================== ===========
4aef06f1
TL
33mydatastore ``root@pam`` localhost:8007 mydatastore
34myhostname:mydatastore ``root@pam`` myhostname:8007 mydatastore
35user@pbs@myhostname:mydatastore ``user@pbs`` myhostname:8007 mydatastore
90e94aa2 36user\@pbs!token@host:store ``user@pbs!token`` myhostname:8007 mydatastore
4aef06f1
TL
37192.168.55.55:1234:mydatastore ``root@pam`` 192.168.55.55:1234 mydatastore
38[ff80::51]:mydatastore ``root@pam`` [ff80::51]:8007 mydatastore
39[ff80::51]:1234:mydatastore ``root@pam`` [ff80::51]:1234 mydatastore
c0af05e1 40================================ ================== ================== ===========
04e24b14
DW
41
42Environment Variables
43---------------------
44
45``PBS_REPOSITORY``
46 The default backup repository.
47
48``PBS_PASSWORD``
4aef06f1
TL
49 When set, this value is used for the password required for the backup server.
50 You can also set this to a API token secret.
04e24b14 51
65ab4ca9
TL
52``PBS_PASSWORD_FD``, ``PBS_PASSWORD_FILE``, ``PBS_PASSWORD_CMD``
53 Like ``PBS_PASSWORD``, but read data from an open file descriptor, a file
54 name or from the `stdout` of a command, respectively. The first defined
55 environment variable from the order above is preferred.
99a73fad 56
04e24b14
DW
57``PBS_ENCRYPTION_PASSWORD``
58 When set, this value is used to access the secret encryption key (if
59 protected by password).
60
65ab4ca9
TL
61``PBS_ENCRYPTION_PASSWORD_FD``, ``PBS_ENCRYPTION_PASSWORD_FILE``, ``PBS_ENCRYPTION_PASSWORD_CMD``
62 Like ``PBS_ENCRYPTION_PASSWORD``, but read data from an open file descriptor,
63 a file name or from the `stdout` of a command, respectively. The first
64 defined environment variable from the order above is preferred.
99a73fad 65
04e24b14 66``PBS_FINGERPRINT`` When set, this value is used to verify the server
4aef06f1
TL
67 certificate (only used if the system CA certificates cannot validate the
68 certificate).
04e24b14
DW
69
70
99a73fad
DM
71.. Note:: Passwords must be valid UTF8 an may not contain
72 newlines. For your convienience, we just use the first line as
73 password, so you can add arbitrary comments after the
74 first newline.
75
76
04e24b14
DW
77Output Format
78-------------
79
f458e97f 80.. include:: output-format.rst
04e24b14
DW
81
82
ec1ae7e6 83.. _client_creating_backups:
04e24b14
DW
84
85Creating Backups
86----------------
87
88This section explains how to create a backup from within the machine. This can
89be a physical host, a virtual machine, or a container. Such backups may contain file
90and image archives. There are no restrictions in this case.
91
92.. note:: If you want to backup virtual machines or containers on Proxmox VE, see :ref:`pve-integration`.
93
94For the following example you need to have a backup server set up, working
95credentials and need to know the repository name.
96In the following examples we use ``backup-server:store1``.
97
98.. code-block:: console
99
100 # proxmox-backup-client backup root.pxar:/ --repository backup-server:store1
101 Starting backup: host/elsa/2019-12-03T09:35:01Z
102 Client name: elsa
103 skip mount point: "/boot/efi"
104 skip mount point: "/dev"
105 skip mount point: "/run"
106 skip mount point: "/sys"
107 Uploaded 12129 chunks in 87 seconds (564 MB/s).
108 End Time: 2019-12-03T10:36:29+01:00
109
110This will prompt you for a password and then uploads a file archive named
111``root.pxar`` containing all the files in the ``/`` directory.
112
113.. Caution:: Please note that the proxmox-backup-client does not
114 automatically include mount points. Instead, you will see a short
115 ``skip mount point`` notice for each of them. The idea is to
116 create a separate file archive for each mounted disk. You can
117 explicitly include them using the ``--include-dev`` option
118 (i.e. ``--include-dev /boot/efi``). You can use this option
119 multiple times for each mount point that should be included.
120
121The ``--repository`` option can get quite long and is used by all
122commands. You can avoid having to enter this value by setting the
123environment variable ``PBS_REPOSITORY``. Note that if you would like this to remain set
124over multiple sessions, you should instead add the below line to your
125``.bashrc`` file.
126
127.. code-block:: console
128
129 # export PBS_REPOSITORY=backup-server:store1
130
131After this you can execute all commands without specifying the ``--repository``
132option.
133
134One single backup is allowed to contain more than one archive. For example, if
3bbb70b3 135you want to backup two disks mounted at ``/mnt/disk1`` and ``/mnt/disk2``:
04e24b14
DW
136
137.. code-block:: console
138
139 # proxmox-backup-client backup disk1.pxar:/mnt/disk1 disk2.pxar:/mnt/disk2
140
141This creates a backup of both disks.
142
143The backup command takes a list of backup specifications, which
144include the archive name on the server, the type of the archive, and the
145archive source at the client. The format is:
146
147 <archive-name>.<type>:<source-path>
148
149Common types are ``.pxar`` for file archives, and ``.img`` for block
150device images. To create a backup of a block device run the following command:
151
152.. code-block:: console
153
154 # proxmox-backup-client backup mydata.img:/dev/mylvm/mydata
155
156
157Excluding files/folders from a backup
158~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
159
160Sometimes it is desired to exclude certain files or folders from a backup archive.
161To tell the Proxmox Backup client when and how to ignore files and directories,
162place a text file called ``.pxarexclude`` in the filesystem hierarchy.
163Whenever the backup client encounters such a file in a directory, it interprets
164each line as glob match patterns for files and directories that are to be excluded
165from the backup.
166
167The file must contain a single glob pattern per line. Empty lines are ignored.
168The same is true for lines starting with ``#``, which indicates a comment.
169A ``!`` at the beginning of a line reverses the glob match pattern from an exclusion
170to an explicit inclusion. This makes it possible to exclude all entries in a
171directory except for a few single files/subdirectories.
172Lines ending in ``/`` match only on directories.
173The directory containing the ``.pxarexclude`` file is considered to be the root of
174the given patterns. It is only possible to match files in this directory and its subdirectories.
175
176``\`` is used to escape special glob characters.
177``?`` matches any single character.
178``*`` matches any character, including an empty string.
179``**`` is used to match subdirectories. It can be used to, for example, exclude
180all files ending in ``.tmp`` within the directory or subdirectories with the
181following pattern ``**/*.tmp``.
182``[...]`` matches a single character from any of the provided characters within
183the brackets. ``[!...]`` does the complementary and matches any single character
184not contained within the brackets. It is also possible to specify ranges with two
185characters separated by ``-``. For example, ``[a-z]`` matches any lowercase
186alphabetic character and ``[0-9]`` matches any one single digit.
187
188The order of the glob match patterns defines whether a file is included or
189excluded, that is to say later entries override previous ones.
190This is also true for match patterns encountered deeper down the directory tree,
191which can override a previous exclusion.
192Be aware that excluded directories will **not** be read by the backup client.
193Thus, a ``.pxarexclude`` file in an excluded subdirectory will have no effect.
194``.pxarexclude`` files are treated as regular files and will be included in the
195backup archive.
196
197For example, consider the following directory structure:
198
199.. code-block:: console
200
201 # ls -aR folder
202 folder/:
203 . .. .pxarexclude subfolder0 subfolder1
204
205 folder/subfolder0:
206 . .. file0 file1 file2 file3 .pxarexclude
207
208 folder/subfolder1:
209 . .. file0 file1 file2 file3
210
211The different ``.pxarexclude`` files contain the following:
212
213.. code-block:: console
214
215 # cat folder/.pxarexclude
216 /subfolder0/file1
217 /subfolder1/*
218 !/subfolder1/file2
219
220.. code-block:: console
221
222 # cat folder/subfolder0/.pxarexclude
223 file3
224
225This would exclude ``file1`` and ``file3`` in ``subfolder0`` and all of
226``subfolder1`` except ``file2``.
227
228Restoring this backup will result in:
229
230.. code-block:: console
231
232 ls -aR restored
233 restored/:
234 . .. .pxarexclude subfolder0 subfolder1
235
236 restored/subfolder0:
237 . .. file0 file2 .pxarexclude
238
239 restored/subfolder1:
240 . .. file2
241
242
ec1ae7e6 243.. _client_encryption:
d9f365d7 244
04e24b14
DW
245Encryption
246----------
247
248Proxmox Backup supports client-side encryption with AES-256 in GCM_
249mode. To set this up, you first need to create an encryption key:
250
251.. code-block:: console
252
253 # proxmox-backup-client key create my-backup.key
254 Encryption Key Password: **************
255
256The key is password protected by default. If you do not need this
257extra protection, you can also create it without a password:
258
259.. code-block:: console
260
261 # proxmox-backup-client key create /path/to/my-backup.key --kdf none
262
263Having created this key, it is now possible to create an encrypted backup, by
264passing the ``--keyfile`` parameter, with the path to the key file.
265
266.. code-block:: console
267
268 # proxmox-backup-client backup etc.pxar:/etc --keyfile /path/to/my-backup.key
269 Password: *********
270 Encryption Key Password: **************
271 ...
272
273.. Note:: If you do not specify the name of the backup key, the key will be
274 created in the default location
275 ``~/.config/proxmox-backup/encryption-key.json``. ``proxmox-backup-client``
276 will also search this location by default, in case the ``--keyfile``
277 parameter is not specified.
278
279You can avoid entering the passwords by setting the environment
280variables ``PBS_PASSWORD`` and ``PBS_ENCRYPTION_PASSWORD``.
281
282
283Using a master key to store and recover encryption keys
284~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
285
286You can also use ``proxmox-backup-client key`` to create an RSA public/private
287key pair, which can be used to store an encrypted version of the symmetric
288backup encryption key alongside each backup and recover it later.
289
290To set up a master key:
291
2921. Create an encryption key for the backup:
293
294 .. code-block:: console
295
296 # proxmox-backup-client key create
297 creating default key at: "~/.config/proxmox-backup/encryption-key.json"
298 Encryption Key Password: **********
299 ...
300
301 The resulting file will be saved to ``~/.config/proxmox-backup/encryption-key.json``.
302
3032. Create an RSA public/private key pair:
304
305 .. code-block:: console
306
307 # proxmox-backup-client key create-master-key
308 Master Key Password: *********
309 ...
310
311 This will create two files in your current directory, ``master-public.pem``
312 and ``master-private.pem``.
313
3143. Import the newly created ``master-public.pem`` public certificate, so that
315 ``proxmox-backup-client`` can find and use it upon backup.
316
317 .. code-block:: console
318
319 # proxmox-backup-client key import-master-pubkey /path/to/master-public.pem
320 Imported public master key to "~/.config/proxmox-backup/master-public.pem"
321
3224. With all these files in place, run a backup job:
323
324 .. code-block:: console
325
326 # proxmox-backup-client backup etc.pxar:/etc
327
328 The key will be stored in your backup, under the name ``rsa-encrypted.key``.
329
330 .. Note:: The ``--keyfile`` parameter can be excluded, if the encryption key
331 is in the default path. If you specified another path upon creation, you
332 must pass the ``--keyfile`` parameter.
333
3345. To test that everything worked, you can restore the key from the backup:
335
336 .. code-block:: console
337
338 # proxmox-backup-client restore /path/to/backup/ rsa-encrypted.key /path/to/target
339
340 .. Note:: You should not need an encryption key to extract this file. However, if
341 a key exists at the default location
342 (``~/.config/proxmox-backup/encryption-key.json``) the program will prompt
343 you for an encryption key password. Simply moving ``encryption-key.json``
344 out of this directory will fix this issue.
345
3466. Then, use the previously generated master key to decrypt the file:
347
348 .. code-block:: console
349
2ea5abcd
FG
350 # proxmox-backup-client key import-with-master-key /path/to/target --master-keyfile /path/to/master-private.pem --encrypted-keyfile /path/to/rsa-encrypted.key
351 Master Key Password: ******
352 New Password: ******
353 Verify Password: ******
04e24b14
DW
354
3557. The target file will now contain the encryption key information in plain
356 text. The success of this can be confirmed by passing the resulting ``json``
357 file, with the ``--keyfile`` parameter, when decrypting files from the backup.
358
359.. warning:: Without their key, backed up files will be inaccessible. Thus, you should
360 keep keys ordered and in a place that is separate from the contents being
361 backed up. It can happen, for example, that you back up an entire system, using
cdbc18fc 362 a key on that system. If the system then becomes inaccessible for any reason
04e24b14 363 and needs to be restored, this will not be possible as the encryption key will be
4accbc58
DW
364 lost along with the broken system.
365
c5608cf8
DW
366It is recommended that you keep your master key safe, but easily accessible, in
367order for quick disaster recovery. For this reason, the best place to store it
368is in your password manager, where it is immediately recoverable. As a backup to
369this, you should also save the key to a USB drive and store that in a secure
370place. This way, it is detached from any system, but is still easy to recover
371from, in case of emergency. Finally, in preparation for the worst case scenario,
372you should also consider keeping a paper copy of your master key locked away in
373a safe place. The ``paperkey`` subcommand can be used to create a QR encoded
374version of your master key. The following command sends the output of the
375``paperkey`` command to a text file, for easy printing.
4accbc58
DW
376
377.. code-block:: console
378
379 proxmox-backup-client key paperkey --output-format text > qrkey.txt
04e24b14
DW
380
381
382Restoring Data
383--------------
384
385The regular creation of backups is a necessary step to avoiding data
386loss. More importantly, however, is the restoration. It is good practice to perform
387periodic recovery tests to ensure that you can access the data in
388case of problems.
389
390First, you need to find the snapshot which you want to restore. The snapshot
abf9b6da 391list command provides a list of all the snapshots on the server:
04e24b14
DW
392
393.. code-block:: console
394
abf9b6da 395 # proxmox-backup-client snapshot list
04e24b14
DW
396 ┌────────────────────────────────┬─────────────┬────────────────────────────────────┐
397 │ snapshot │ size │ files │
398 ╞════════════════════════════════╪═════════════╪════════════════════════════════════╡
399 │ host/elsa/2019-12-03T09:30:15Z │ 51788646825 │ root.pxar catalog.pcat1 index.json │
400 ├────────────────────────────────┼─────────────┼────────────────────────────────────┤
401 │ host/elsa/2019-12-03T09:35:01Z │ 51790622048 │ root.pxar catalog.pcat1 index.json │
402 ├────────────────────────────────┼─────────────┼────────────────────────────────────┤
403 ...
404
405You can inspect the catalog to find specific files.
406
407.. code-block:: console
408
409 # proxmox-backup-client catalog dump host/elsa/2019-12-03T09:35:01Z
410 ...
411 d "./root.pxar.didx/etc/cifs-utils"
412 l "./root.pxar.didx/etc/cifs-utils/idmap-plugin"
413 d "./root.pxar.didx/etc/console-setup"
414 ...
415
416The restore command lets you restore a single archive from the
417backup.
418
419.. code-block:: console
420
421 # proxmox-backup-client restore host/elsa/2019-12-03T09:35:01Z root.pxar /target/path/
422
423To get the contents of any archive, you can restore the ``index.json`` file in the
424repository to the target path '-'. This will dump the contents to the standard output.
425
426.. code-block:: console
427
428 # proxmox-backup-client restore host/elsa/2019-12-03T09:35:01Z index.json -
429
430
431Interactive Restores
432~~~~~~~~~~~~~~~~~~~~
433
434If you only want to restore a few individual files, it is often easier
435to use the interactive recovery shell.
436
437.. code-block:: console
438
439 # proxmox-backup-client catalog shell host/elsa/2019-12-03T09:35:01Z root.pxar
440 Starting interactive shell
441 pxar:/ > ls
442 bin boot dev etc home lib lib32
443 ...
444
3bbb70b3 445The interactive recovery shell is a minimal command line interface that
04e24b14
DW
446utilizes the metadata stored in the catalog to quickly list, navigate and
447search files in a file archive.
448To restore files, you can select them individually or match them with a glob
449pattern.
450
451Using the catalog for navigation reduces the overhead considerably because only
452the catalog needs to be downloaded and, optionally, decrypted.
453The actual chunks are only accessed if the metadata in the catalog is not enough
454or for the actual restore.
455
456Similar to common UNIX shells ``cd`` and ``ls`` are the commands used to change
457working directory and list directory contents in the archive.
458``pwd`` shows the full path of the current working directory with respect to the
459archive root.
460
3bbb70b3 461Being able to quickly search the contents of the archive is a commonly needed feature.
04e24b14
DW
462That's where the catalog is most valuable.
463For example:
464
465.. code-block:: console
466
467 pxar:/ > find etc/**/*.txt --select
468 "/etc/X11/rgb.txt"
469 pxar:/ > list-selected
470 etc/**/*.txt
471 pxar:/ > restore-selected /target/path
472 ...
473
474This will find and print all files ending in ``.txt`` located in ``etc/`` or a
475subdirectory and add the corresponding pattern to the list for subsequent restores.
476``list-selected`` shows these patterns and ``restore-selected`` finally restores
477all files in the archive matching the patterns to ``/target/path`` on the local
478host. This will scan the whole archive.
479
985e84e3
DW
480The ``restore`` command can be used to restore all the files contained within
481the backup archive. This is most helpful when paired with the ``--pattern
482<glob>`` option, as it allows you to restore all files matching a specific
483pattern. For example, if you wanted to restore configuration files
484located in ``/etc``, you could do the following:
04e24b14
DW
485
486.. code-block:: console
487
985e84e3 488 pxar:/ > restore target/ --pattern etc/**/*.conf
04e24b14
DW
489 ...
490
5f3b2330 491The above will scan through all the directories below ``/etc`` and restore all
04e24b14
DW
492files ending in ``.conf``.
493
494.. todo:: Explain interactive restore in more detail
495
496Mounting of Archives via FUSE
497~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
498
499The :term:`FUSE` implementation for the pxar archive allows you to mount a
500file archive as a read-only filesystem to a mountpoint on your host.
501
502.. code-block:: console
503
504 # proxmox-backup-client mount host/backup-client/2020-01-29T11:29:22Z root.pxar /mnt/mountpoint
505 # ls /mnt/mountpoint
506 bin dev home lib32 libx32 media opt root sbin sys usr
507 boot etc lib lib64 lost+found mnt proc run srv tmp var
508
509This allows you to access the full contents of the archive in a seamless manner.
510
511.. note:: As the FUSE connection needs to fetch and decrypt chunks from the
512 backup server's datastore, this can cause some additional network and CPU
513 load on your host, depending on the operations you perform on the mounted
514 filesystem.
515
516To unmount the filesystem use the ``umount`` command on the mountpoint:
517
518.. code-block:: console
519
520 # umount /mnt/mountpoint
521
522Login and Logout
523----------------
524
525The client tool prompts you to enter the logon password as soon as you
526want to access the backup server. The server checks your credentials
527and responds with a ticket that is valid for two hours. The client
528tool automatically stores that ticket and uses it for further requests
529to this server.
530
531You can also manually trigger this login/logout using the login and
532logout commands:
533
534.. code-block:: console
535
536 # proxmox-backup-client login
537 Password: **********
538
539To remove the ticket, issue a logout:
540
541.. code-block:: console
542
543 # proxmox-backup-client logout
544
545
08379a21
DW
546.. _changing-backup-owner:
547
548Changing the Owner of a Backup Group
549------------------------------------
550
551By default, the owner of a backup group is the user which was used to originally
552create that backup group (or in the case of sync jobs, ``root@pam``). This
553means that if a user ``mike@pbs`` created a backup, another user ``john@pbs``
554can not be used to create backups in that same backup group. In case you want
555to change the owner of a backup, you can do so with the below command, using a
556user that has ``Datastore.Modify`` privileges on the datastore.
557
558.. code-block:: console
559
560 # proxmox-backup-client change-owner vm/103 john@pbs
561
562This can also be done from within the web interface, by navigating to the
563`Content` section of the datastore that contains the backup group and
564selecting the user icon under the `Actions` column. Common cases for this could
565be to change the owner of a sync job from ``root@pam``, or to repurpose a
566backup group.
567
568
4c3efb53 569.. _backup-pruning:
04e24b14
DW
570
571Pruning and Removing Backups
572----------------------------
573
574You can manually delete a backup snapshot using the ``forget``
575command:
576
577.. code-block:: console
578
abf9b6da 579 # proxmox-backup-client snapshot forget <snapshot>
04e24b14
DW
580
581
582.. caution:: This command removes all archives in this backup
583 snapshot. They will be inaccessible and unrecoverable.
584
585
586Although manual removal is sometimes required, the ``prune``
587command is normally used to systematically delete older backups. Prune lets
588you specify which backup snapshots you want to keep. The
589following retention options are available:
590
591``--keep-last <N>``
592 Keep the last ``<N>`` backup snapshots.
593
594``--keep-hourly <N>``
595 Keep backups for the last ``<N>`` hours. If there is more than one
596 backup for a single hour, only the latest is kept.
597
598``--keep-daily <N>``
599 Keep backups for the last ``<N>`` days. If there is more than one
600 backup for a single day, only the latest is kept.
601
602``--keep-weekly <N>``
603 Keep backups for the last ``<N>`` weeks. If there is more than one
604 backup for a single week, only the latest is kept.
605
606 .. note:: Weeks start on Monday and end on Sunday. The software
607 uses the `ISO week date`_ system and handles weeks at
608 the end of the year correctly.
609
610``--keep-monthly <N>``
611 Keep backups for the last ``<N>`` months. If there is more than one
612 backup for a single month, only the latest is kept.
613
614``--keep-yearly <N>``
615 Keep backups for the last ``<N>`` years. If there is more than one
616 backup for a single year, only the latest is kept.
617
618The retention options are processed in the order given above. Each option
619only covers backups within its time period. The next option does not take care
620of already covered backups. It will only consider older backups.
621
622Unfinished and incomplete backups will be removed by the prune command unless
623they are newer than the last successful backup. In this case, the last failed
624backup is retained.
625
626.. code-block:: console
627
628 # proxmox-backup-client prune <group> --keep-daily 7 --keep-weekly 4 --keep-monthly 3
629
630
631You can use the ``--dry-run`` option to test your settings. This only
632shows the list of existing snapshots and what actions prune would take.
633
634.. code-block:: console
635
636 # proxmox-backup-client prune host/elsa --dry-run --keep-daily 1 --keep-weekly 3
637 ┌────────────────────────────────┬──────┐
638 │ snapshot │ keep │
639 ╞════════════════════════════════╪══════╡
640 │ host/elsa/2019-12-04T13:20:37Z │ 1 │
641 ├────────────────────────────────┼──────┤
642 │ host/elsa/2019-12-03T09:35:01Z │ 0 │
643 ├────────────────────────────────┼──────┤
644 │ host/elsa/2019-11-22T11:54:47Z │ 1 │
645 ├────────────────────────────────┼──────┤
646 │ host/elsa/2019-11-21T12:36:25Z │ 0 │
647 ├────────────────────────────────┼──────┤
648 │ host/elsa/2019-11-10T10:42:20Z │ 1 │
649 └────────────────────────────────┴──────┘
650
651.. note:: Neither the ``prune`` command nor the ``forget`` command free space
652 in the chunk-store. The chunk-store still contains the data blocks. To free
a98e2287 653 space you need to perform :ref:`client_garbage-collection`.
04e24b14
DW
654
655
ec1ae7e6 656.. _client_garbage-collection:
04e24b14
DW
657
658Garbage Collection
659------------------
660
661The ``prune`` command removes only the backup index files, not the data
662from the datastore. This task is left to the garbage collection
663command. It is recommended to carry out garbage collection on a regular basis.
664
665The garbage collection works in two phases. In the first phase, all
666data blocks that are still in use are marked. In the second phase,
667unused data blocks are removed.
668
669.. note:: This command needs to read all existing backup index files
670 and touches the complete chunk-store. This can take a long time
671 depending on the number of chunks and the speed of the underlying
672 disks.
673
674.. note:: The garbage collection will only remove chunks that haven't been used
675 for at least one day (exactly 24h 5m). This grace period is necessary because
676 chunks in use are marked by touching the chunk which updates the ``atime``
677 (access time) property. Filesystems are mounted with the ``relatime`` option
678 by default. This results in a better performance by only updating the
679 ``atime`` property if the last access has been at least 24 hours ago. The
680 downside is, that touching a chunk within these 24 hours will not always
681 update its ``atime`` property.
682
683 Chunks in the grace period will be logged at the end of the garbage
684 collection task as *Pending removals*.
685
686.. code-block:: console
687
688 # proxmox-backup-client garbage-collect
689 starting garbage collection on store store2
690 Start GC phase1 (mark used chunks)
691 Start GC phase2 (sweep unused chunks)
692 percentage done: 1, chunk count: 219
693 percentage done: 2, chunk count: 453
694 ...
695 percentage done: 99, chunk count: 21188
696 Removed bytes: 411368505
697 Removed chunks: 203
698 Original data bytes: 327160886391
699 Disk bytes: 52767414743 (16 %)
700 Disk chunks: 21221
701 Average chunk size: 2486565
702 TASK OK
703
704
3bbb70b3 705.. todo:: howto run garbage-collection at regular intervals (cron)
04e24b14
DW
706
707Benchmarking
708------------
709
710The backup client also comes with a benchmarking tool. This tool measures
654c56e0
TL
711various metrics relating to compression and encryption speeds. If a Proxmox
712Backup repository (remote or local) is specified, the TLS upload speed will get
713measured too.
714
715You can run a benchmark using the ``benchmark`` subcommand of
716``proxmox-backup-client``:
717
718.. note:: The TLS speed test is only included if a :ref:`backup server
719 repository is specified <client_repository>`.
04e24b14
DW
720
721.. code-block:: console
722
723 # proxmox-backup-client benchmark
81f5d03e
DM
724 Uploaded 1517 chunks in 5 seconds.
725 Time per request: 3309 microseconds.
726 TLS speed: 1267.41 MB/s
727 SHA256 speed: 2066.73 MB/s
728 Compression speed: 775.11 MB/s
729 Decompress speed: 1233.35 MB/s
730 AES256/GCM speed: 3688.27 MB/s
731 Verify speed: 783.43 MB/s
04e24b14
DW
732 ┌───────────────────────────────────┬─────────────────────┐
733 │ Name │ Value │
734 ╞═══════════════════════════════════╪═════════════════════╡
81f5d03e 735 │ TLS (maximal backup upload speed) │ 1267.41 MB/s (103%) │
04e24b14 736 ├───────────────────────────────────┼─────────────────────┤
81f5d03e 737 │ SHA256 checksum computation speed │ 2066.73 MB/s (102%) │
04e24b14 738 ├───────────────────────────────────┼─────────────────────┤
81f5d03e 739 │ ZStd level 1 compression speed │ 775.11 MB/s (103%) │
04e24b14 740 ├───────────────────────────────────┼─────────────────────┤
81f5d03e 741 │ ZStd level 1 decompression speed │ 1233.35 MB/s (103%) │
04e24b14 742 ├───────────────────────────────────┼─────────────────────┤
81f5d03e
DM
743 │ Chunk verification speed │ 783.43 MB/s (103%) │
744 ├───────────────────────────────────┼─────────────────────┤
745 │ AES256 GCM encryption speed │ 3688.27 MB/s (101%) │
04e24b14
DW
746 └───────────────────────────────────┴─────────────────────┘
747
81f5d03e 748
04e24b14 749.. note:: The percentages given in the output table correspond to a
654c56e0 750 comparison against a Ryzen 7 2700X.
04e24b14
DW
751
752You can also pass the ``--output-format`` parameter to output stats in ``json``,
753rather than the default table format.