]>
Commit | Line | Data |
---|---|---|
cb01363c DM |
1 | Administration Guide |
2 | ==================== | |
7e688b71 | 3 | |
85e139b7 | 4 | The administration guide. |
7e688b71 | 5 | |
4f3db187 | 6 | .. todo:: either add a bit more explanation or remove the previous sentence |
c4f1b69f | 7 | |
fea8789c DM |
8 | Terminology |
9 | ----------- | |
10 | ||
85e139b7 DM |
11 | Backup Content |
12 | ~~~~~~~~~~~~~~ | |
13 | ||
14 | When doing deduplication, there are different strategies to get | |
15 | optimal results in terms of performance and/or deduplication rates. | |
4f3db187 | 16 | Depending on the type of data, one can split data into *fixed* or *variable* |
85e139b7 DM |
17 | sized chunks. |
18 | ||
19 | Fixed sized chunking needs almost no CPU performance, and is used to | |
20 | backup virtual machine images. | |
21 | ||
22 | Variable sized chunking needs more CPU power, but is essential to get | |
23 | good deduplication rates for file archives. | |
24 | ||
4f3db187 | 25 | The backup server supports both strategies. |
85e139b7 DM |
26 | |
27 | ||
57905a61 DM |
28 | File Archives: ``<name>.pxar`` |
29 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
85e139b7 DM |
30 | |
31 | .. see https://moinakg.wordpress.com/2013/06/22/high-performance-content-defined-chunking/ | |
32 | ||
4f3db187 | 33 | A file archive stores a full directory tree. Content is stored using |
85e139b7 | 34 | the :ref:`pxar-format`, split into variable sized chunks. The format |
4f3db187 | 35 | is optimized to achieve good deduplication rates. |
85e139b7 DM |
36 | |
37 | ||
57905a61 DM |
38 | Image Archives: ``<name>.img`` |
39 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
85e139b7 DM |
40 | |
41 | This is used for virtual machine images and other large binary | |
42 | data. Content is split into fixed sized chunks. | |
43 | ||
44 | ||
45 | Binary Data (BLOBs) | |
46 | ^^^^^^^^^^^^^^^^^^^ | |
47 | ||
4f3db187 AL |
48 | This type is used to store smaller (< 16MB) binary data such as |
49 | configuration files. Larger files should be stored as image archive. | |
85e139b7 DM |
50 | |
51 | .. caution:: Please do not store all files as BLOBs. Instead, use the | |
52 | file archive to store whole directory trees. | |
53 | ||
54 | ||
57905a61 DM |
55 | Catalog File: ``catalog.pcat1`` |
56 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
57 | ||
4f3db187 AL |
58 | The catalog file is an index for file archives. It contains |
59 | the list of files and is used to speed-up search operations. | |
57905a61 DM |
60 | |
61 | ||
62 | The Manifest: ``index.json`` | |
63 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
64 | ||
4f3db187 AL |
65 | The manifest contains the list of all backup files, their |
66 | sizes and checksums. It is used to verify the consistency of a | |
57905a61 DM |
67 | backup. |
68 | ||
69 | ||
fea8789c DM |
70 | Backup Type |
71 | ~~~~~~~~~~~ | |
72 | ||
73 | The backup server groups backups by *type*, where *type* is one of: | |
74 | ||
75 | ``vm`` | |
a129fdd9 | 76 | This type is used for :term:`virtual machine`\ s. Typically |
4f3db187 | 77 | contains the virtual machine's configuration and an image archive |
fea8789c DM |
78 | for each disk. |
79 | ||
80 | ``ct`` | |
4f3db187 | 81 | This type is used for :term:`container`\ s. Contains the container's |
fea8789c DM |
82 | configuration and a single file archive for the container content. |
83 | ||
84 | ``host`` | |
4f3db187 AL |
85 | This type is used for backups created from within the backed up machine. |
86 | Typically this would be a physical host but could also be a virtual machine | |
87 | or container. Such backups may contain file and image archives, there are no restrictions in this regard. | |
fea8789c DM |
88 | |
89 | ||
90 | Backup ID | |
91 | ~~~~~~~~~ | |
92 | ||
93 | An unique ID. Usually the virtual machine or container ID. ``host`` | |
94 | type backups normally use the hostname. | |
95 | ||
96 | ||
97 | Backup Time | |
98 | ~~~~~~~~~~~ | |
99 | ||
100 | The time when the backup was made. | |
101 | ||
102 | ||
6e5a0c03 DM |
103 | Backup Group |
104 | ~~~~~~~~~~~~ | |
105 | ||
4f3db187 AL |
106 | The tuple ``<type>/<ID>`` is called a backup group. Such a group |
107 | may contain one or more backup snapshots. | |
6e5a0c03 DM |
108 | |
109 | ||
fea8789c DM |
110 | Backup Snapshot |
111 | ~~~~~~~~~~~~~~~ | |
112 | ||
4f3db187 | 113 | The triplet ``<type>/<ID>/<time>`` is called a backup snapshot. It |
fea8789c DM |
114 | uniquely identifies a specific backup within a datastore. |
115 | ||
116 | .. code-block:: console | |
117 | :caption: Backup Snapshot Examples | |
118 | ||
119 | vm/104/2019-10-09T08:01:06Z | |
120 | host/elsa/2019-11-08T09:48:14Z | |
121 | ||
4f3db187 | 122 | As you can see, the time format is RFC3399_ with Coordinated |
fea8789c DM |
123 | Universal Time (UTC_, identified by the trailing *Z*). |
124 | ||
125 | ||
126 | :term:`DataStore` | |
127 | ~~~~~~~~~~~~~~~~~ | |
128 | ||
4f3db187 | 129 | A datastore is a place where backups are stored. The current implementation |
fea8789c | 130 | uses a directory inside a standard unix file system (``ext4``, ``xfs`` |
4f3db187 | 131 | or ``zfs``) to store the backup data. |
fea8789c | 132 | |
4f3db187 | 133 | Datastores are identified by a simple *ID*. You can configure it |
fea8789c DM |
134 | when setting up the backup server. |
135 | ||
136 | ||
cb01363c DM |
137 | Backup Server Management |
138 | ------------------------ | |
58ea88c8 | 139 | |
4f3db187 | 140 | The command line tool to configure and manage the backup server is called |
58ea88c8 DM |
141 | :command:`proxmox-backup-manager`. |
142 | ||
143 | ||
144 | Datastore Configuration | |
145 | ~~~~~~~~~~~~~~~~~~~~~~~ | |
146 | ||
147 | A :term:`datastore` is a place to store backups. You can configure | |
17ec699d DM |
148 | multiple datastores. At least one datastore needs to be |
149 | configured. The datastore is identified by a simple `name` and points | |
150 | to a directory. | |
58ea88c8 DM |
151 | |
152 | The following command creates a new datastore called ``store1`` on :file:`/backup/disk1/store1` | |
153 | ||
154 | .. code-block:: console | |
155 | ||
156 | # proxmox-backup-manager datastore create store1 /backup/disk1/store1 | |
157 | ||
4f3db187 | 158 | To list existing datastores run: |
58ea88c8 DM |
159 | |
160 | .. code-block:: console | |
161 | ||
162 | # proxmox-backup-manager datastore list | |
17ec699d DM |
163 | ┌────────┬──────────────────────┬─────────────────────────────┐ |
164 | │ name │ path │ comment │ | |
165 | ╞════════╪══════════════════════╪═════════════════════════════╡ | |
166 | │ store1 │ /backup/disk1/store1 │ This is my default storage. │ | |
167 | └────────┴──────────────────────┴─────────────────────────────┘ | |
58ea88c8 | 168 | |
4f3db187 | 169 | Finally, it is possible to remove the datastore configuration: |
58ea88c8 DM |
170 | |
171 | .. code-block:: console | |
172 | ||
173 | # proxmox-backup-manager datastore remove store1 | |
174 | ||
4f3db187 | 175 | .. note:: The above command removes only the datastore configuration. It does |
58ea88c8 DM |
176 | not delete any data from the underlying directory. |
177 | ||
178 | ||
fea8789c DM |
179 | File Layout |
180 | ^^^^^^^^^^^ | |
181 | ||
182 | .. todo:: Add datastore file layout example | |
183 | ||
184 | ||
17ec699d DM |
185 | User Management |
186 | ~~~~~~~~~~~~~~~ | |
187 | ||
188 | Proxmox Backup support several authentication realms, and you need to | |
189 | choose the realm when you add a new user. Possible realms are: | |
190 | ||
191 | :pam: Linux PAM standard authentication. Use this if you want to | |
192 | authenticate as Linux system user (Users needs to exist on the | |
193 | system). | |
194 | ||
195 | :pbs: Proxmox Backup Server realm. This type stores hashed passwords in | |
196 | ``/etc/proxmox-backup/shadow.json``. | |
197 | ||
198 | After installation, there is a single user ``root@pam``, which | |
199 | corresponds to the Unix superuser. You can use the | |
200 | ``proxmox-backup-manager`` command line tool to list or manipulate | |
201 | users: | |
202 | ||
203 | .. code-block:: console | |
204 | ||
205 | # proxmox-backup-manager user list | |
26d29e0e DM |
206 | ┌─────────────┬────────┬────────┬───────────┬──────────┬────────────────┬────────────────────┐ |
207 | │ userid │ enable │ expire │ firstname │ lastname │ email │ comment │ | |
208 | ╞═════════════╪════════╪════════╪═══════════╪══════════╪════════════════╪════════════════════╡ | |
209 | │ root@pam │ 1 │ │ │ │ │ Superuser │ | |
210 | └─────────────┴────────┴────────┴───────────┴──────────┴────────────────┴────────────────────┘ | |
17ec699d DM |
211 | |
212 | The superuser has full administration rights on everything, so you | |
213 | normally want to add other users with less privileges: | |
214 | ||
215 | .. code-block:: console | |
216 | ||
217 | # proxmox-backup-manager user create john@pbs --email john@example.com | |
218 | ||
219 | The create command lets you specify many option like ``--email`` or | |
220 | ``--password``, but you can update or change any of them using the | |
221 | update command later: | |
222 | ||
223 | .. code-block:: console | |
224 | ||
225 | # proxmox-backup-manager user update john@pbs --firstname John --lastname Smith | |
226 | # proxmox-backup-manager user update john@pbs --comment "An example user." | |
227 | ||
228 | ||
229 | .. todo:: Mention how to set password without passing plaintext password as cli argument. | |
230 | ||
231 | ||
232 | The resulting use list looks like this: | |
233 | ||
234 | .. code-block:: console | |
235 | ||
236 | # proxmox-backup-manager user list | |
237 | ┌──────────┬────────┬────────┬───────────┬──────────┬──────────────────┬──────────────────┐ | |
238 | │ userid │ enable │ expire │ firstname │ lastname │ email │ comment │ | |
239 | ╞══════════╪════════╪════════╪═══════════╪══════════╪══════════════════╪══════════════════╡ | |
240 | │ john@pbs │ 1 │ │ John │ Smith │ john@example.com │ An example user. │ | |
241 | ├──────────┼────────┼────────┼───────────┼──────────┼──────────────────┼──────────────────┤ | |
242 | │ root@pam │ 1 │ │ │ │ │ Superuser │ | |
243 | └──────────┴────────┴────────┴───────────┴──────────┴──────────────────┴──────────────────┘ | |
244 | ||
245 | Newly created users do not have an permissions. Please read the next | |
246 | section to learn how to set access permissions. | |
247 | ||
8f3b3cc1 DM |
248 | If you want to disable an user account, you can do that by setting ``--enable`` to ``0`` |
249 | ||
250 | .. code-block:: console | |
251 | ||
252 | # proxmox-backup-manager user update john@pbs --enable 0 | |
253 | ||
254 | Or completely remove the users with: | |
255 | ||
256 | .. code-block:: console | |
257 | ||
258 | # proxmox-backup-manager user remove john@pbs | |
259 | ||
260 | ||
17ec699d DM |
261 | Access Control |
262 | ~~~~~~~~~~~~~~ | |
263 | ||
8df51d48 DM |
264 | Users do not have any permission by default. Instead you need to |
265 | specify what is allowed and what not. You can do this by assigning | |
266 | roles to users on specific objects like datastores or remotes. The | |
267 | following roles exist: | |
268 | ||
269 | **Admin** | |
270 | The Administrator can do anything. | |
271 | ||
272 | **Audit** | |
273 | An Auditor can view things, but is not allowed to change settings. | |
274 | ||
275 | **NoAccess** | |
276 | Disable Access - nothing is allowed. | |
277 | ||
278 | **DatastoreAdmin** | |
279 | Can do anything on datastores. | |
280 | ||
281 | **DatastoreAudit** | |
282 | Can view datastore settings and list content. But | |
283 | is not allowed to read the actual data. | |
284 | ||
285 | **DataStoreReader** | |
286 | Can Inspect datastore content and can do restores. | |
287 | ||
288 | **DataStoreBackup** | |
289 | Can backup and restore owned backups. | |
290 | ||
291 | **DatastorePowerUser** | |
292 | Can backup, restore, and prune owned backups. | |
293 | ||
294 | **RemoteAdmin** | |
295 | Can do anything on remotes. | |
296 | ||
297 | **RemoteAudit** | |
298 | Can view remote settings. | |
299 | ||
300 | **RemoteSyncOperator** | |
301 | Is allowed to read data from a remote. | |
302 | ||
17ec699d DM |
303 | |
304 | ||
cb01363c DM |
305 | Backup Client usage |
306 | ------------------- | |
58ea88c8 DM |
307 | |
308 | The command line client is called :command:`proxmox-backup-client`. | |
309 | ||
a129fdd9 | 310 | |
58ea88c8 DM |
311 | Respository Locations |
312 | ~~~~~~~~~~~~~~~~~~~~~ | |
313 | ||
4f3db187 | 314 | The client uses the following notation to specify a datastore repository |
58ea88c8 DM |
315 | on the backup server. |
316 | ||
317 | [[username@]server:]datastore | |
318 | ||
4f3db187 AL |
319 | The default value for ``username`` ist ``root``. If no server is specified, the |
320 | default is the local host (``localhost``). | |
58ea88c8 | 321 | |
4f3db187 AL |
322 | You can pass the repository with the ``--repository`` command |
323 | line option, or by setting the ``PBS_REPOSITORY`` environment | |
58ea88c8 DM |
324 | variable. |
325 | ||
326 | ||
327 | Environment Variables | |
53ea6556 | 328 | ~~~~~~~~~~~~~~~~~~~~~ |
58ea88c8 DM |
329 | |
330 | ``PBS_REPOSITORY`` | |
331 | The default backup repository. | |
332 | ||
333 | ``PBS_PASSWORD`` | |
334 | When set, this value is used for the password required for the | |
335 | backup server. | |
336 | ||
337 | ``PBS_ENCRYPTION_PASSWORD`` | |
58ea88c8 DM |
338 | When set, this value is used to access the secret encryption key (if |
339 | protected by password). | |
340 | ||
3243f93c DM |
341 | ``PBS_FINGERPRINT`` When set, this value is used to verify the server |
342 | certificate (only used if the system CA certificates cannot | |
343 | validate the certificate). | |
344 | ||
53ea6556 DM |
345 | |
346 | Output Format | |
347 | ~~~~~~~~~~~~~ | |
348 | ||
4f3db187 AL |
349 | Most commands support the ``--output-format`` parameter. It accepts |
350 | the following values: | |
53ea6556 DM |
351 | |
352 | :``text``: Text format (default). Structured data is rendered as a table. | |
353 | ||
354 | :``json``: JSON (single line). | |
355 | ||
356 | :``json-pretty``: JSON (multiple lines, nicely formatted). | |
357 | ||
358 | ||
359 | Please use the following environment variables to modify output behavior: | |
360 | ||
361 | ``PROXMOX_OUTPUT_FORMAT`` | |
362 | Defines the default output format. | |
363 | ||
364 | ``PROXMOX_OUTPUT_NO_BORDER`` | |
365 | If set (to any value), do not render table borders. | |
366 | ||
367 | ``PROXMOX_OUTPUT_NO_HEADER`` | |
368 | If set (to any value), do not render table headers. | |
369 | ||
4f3db187 | 370 | .. note:: The ``text`` format is designed to be human readable, and |
53ea6556 | 371 | not meant to be parsed by automation tools. Please use the ``json`` |
4f3db187 | 372 | format if you need to process the output. |
53ea6556 DM |
373 | |
374 | ||
cee53b34 | 375 | .. _creating-backups: |
58ea88c8 DM |
376 | |
377 | Creating Backups | |
378 | ~~~~~~~~~~~~~~~~ | |
379 | ||
4f3db187 AL |
380 | This section explains how to create a backup from within the machine. This can |
381 | be a physical host, a virtual machine, or a container. Such backups may contain file | |
382 | and image archives. There are no restrictions in this case. | |
a129fdd9 | 383 | |
4f3db187 | 384 | .. note:: If you want to backup virtual machines or containers on Proxmov VE, see :ref:`pve-integration`. |
a129fdd9 | 385 | |
4f3db187 AL |
386 | For the following example you need to have a backup server set up, working |
387 | credentials and need to know the repository name. | |
388 | In the following examples we use ``backup-server:store1``. | |
a129fdd9 DM |
389 | |
390 | .. code-block:: console | |
391 | ||
392 | # proxmox-backup-client backup root.pxar:/ --repository backup-server:store1 | |
393 | Starting backup: host/elsa/2019-12-03T09:35:01Z | |
394 | Client name: elsa | |
395 | skip mount point: "/boot/efi" | |
396 | skip mount point: "/dev" | |
397 | skip mount point: "/run" | |
398 | skip mount point: "/sys" | |
399 | Uploaded 12129 chunks in 87 seconds (564 MB/s). | |
400 | End Time: 2019-12-03T10:36:29+01:00 | |
401 | ||
402 | This will prompt you for a password and then uploads a file archive named | |
403 | ``root.pxar`` containing all the files in the ``/`` directory. | |
404 | ||
4f3db187 | 405 | .. Caution:: Please note that the proxmox-backup-client does not |
ed858b0a | 406 | automatically include mount points. Instead, you will see a short |
4f3db187 AL |
407 | ``skip mount point`` notice for each of them. The idea is to |
408 | create a separate file archive for each mounted disk. You can | |
a129fdd9 DM |
409 | explicitly include them using the ``--include-dev`` option |
410 | (i.e. ``--include-dev /boot/efi``). You can use this option | |
4f3db187 | 411 | multiple times for each mount point that should be included. |
a129fdd9 | 412 | |
4f3db187 | 413 | The ``--repository`` option can get quite long and is used by all |
a129fdd9 DM |
414 | commands. You can avoid having to enter this value by setting the |
415 | environment variable ``PBS_REPOSITORY``. | |
416 | ||
417 | .. code-block:: console | |
418 | ||
419 | # export PBS_REPOSTORY=backup-server:store1 | |
420 | ||
4f3db187 | 421 | After this you can execute all commands without specifying the ``--repository`` |
a129fdd9 DM |
422 | option. |
423 | ||
4f3db187 AL |
424 | One single backup is allowed to contain more than one archive. For example, if |
425 | you want to backup two disks mounted at ``/mmt/disk1`` and ``/mnt/disk2``: | |
a129fdd9 DM |
426 | |
427 | .. code-block:: console | |
428 | ||
429 | # proxmox-backup-client backup disk1.pxar:/mnt/disk1 disk2.pxar:/mnt/disk2 | |
430 | ||
4f3db187 | 431 | This creates a backup of both disks. |
a129fdd9 DM |
432 | |
433 | The backup command takes a list of backup specifications, which | |
4f3db187 AL |
434 | include the archive name on the server, the type of the archive, and the |
435 | archive source at the client. The format is: | |
a129fdd9 DM |
436 | |
437 | <archive-name>.<type>:<source-path> | |
438 | ||
439 | Common types are ``.pxar`` for file archives, and ``.img`` for block | |
4f3db187 | 440 | device images. To create a backup of a block device run the following command: |
a129fdd9 DM |
441 | |
442 | .. code-block:: console | |
443 | ||
444 | # proxmox-backup-client backup mydata.img:/dev/mylvm/mydata | |
445 | ||
50b8f9dd CE |
446 | Excluding files/folders from a backup |
447 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
448 | ||
449 | Sometimes it is desired to exclude certain files or folders from a backup archive. | |
4f3db187 AL |
450 | To tell the Proxmox backup client when and how to ignore files and directories, |
451 | place a text file called ``.pxarexclude`` in the filesystem hierarchy. | |
452 | Whenever the backup client encounters such a file in a directory, it interprets | |
453 | each line as glob match patterns for files and directories that are to be excluded | |
454 | from the backup. | |
455 | ||
456 | The file must contain a single glob pattern per line. Empty lines are ignored. | |
457 | The same is true for lines starting with ``#``, which indicates a comment. | |
458 | A ``!`` at the beginning of a line reverses the glob match pattern from an exclusion | |
459 | to an explicit inclusion. This makes it possible to exclude all entries in a | |
460 | directory except for a few single files/subdirectories. | |
461 | Lines ending in ``/`` match only on directories. | |
462 | The directory containing the ``.pxarexclude`` file is considered to be the root of | |
463 | the given patterns. It is only possible to match files in this directory and its subdirectories. | |
464 | ||
465 | ``\`` is used to escape special glob characters. | |
466 | ``?`` matches any single character. | |
467 | ``*`` matches any character, including an empty string. | |
468 | ``**`` is used to match subdirectories. It can be used to, for example, exclude | |
469 | all files ending in ``.tmp`` within the directory or subdirectories with the | |
50b8f9dd CE |
470 | following pattern ``**/*.tmp``. |
471 | ``[...]`` matches a single character from any of the provided characters within | |
472 | the brackets. ``[!...]`` does the complementary and matches any singe character | |
4f3db187 AL |
473 | not contained within the brackets. It is also possible to specify ranges with two |
474 | characters separated by ``-``. For example, ``[a-z]`` matches any lowercase | |
475 | alphabetic character and ``[0-9]`` matches any one single digit. | |
50b8f9dd | 476 | |
4f3db187 | 477 | The order of the glob match patterns defines if a file is included or |
50b8f9dd CE |
478 | excluded, later entries win over previous ones. |
479 | This is also true for match patterns encountered deeper down the directory tree, | |
4f3db187 AL |
480 | which can override a previous exclusion. |
481 | Be aware that excluded directories will **not** be read by the backup client. | |
482 | A ``.pxarexclude`` file in a subdirectory will have no effect. | |
483 | ``.pxarexclude`` files are treated as regular files and will be included in the | |
50b8f9dd CE |
484 | backup archive. |
485 | ||
4f3db187 | 486 | For example, consider the following directory structure: |
50b8f9dd CE |
487 | |
488 | .. code-block:: console | |
489 | ||
490 | # ls -aR folder | |
491 | folder/: | |
492 | . .. .pxarexclude subfolder0 subfolder1 | |
493 | ||
494 | folder/subfolder0: | |
495 | . .. file0 file1 file2 file3 .pxarexclude | |
496 | ||
497 | folder/subfolder1: | |
498 | . .. file0 file1 file2 file3 | |
499 | ||
4f3db187 | 500 | The different ``.pxarexclude`` files contain the following: |
50b8f9dd CE |
501 | |
502 | .. code-block:: console | |
503 | ||
504 | # cat folder/.pxarexclude | |
505 | /subfolder0/file1 | |
506 | /subfolder1/* | |
507 | !/subfolder1/file2 | |
508 | ||
509 | .. code-block:: console | |
510 | ||
511 | # cat folder/subfolder0/.pxarexclude | |
512 | file3 | |
513 | ||
514 | This would exclude ``file1`` and ``file3`` in ``subfolder0`` and all of | |
515 | ``subfolder1`` except ``file2``. | |
516 | ||
4f3db187 | 517 | Restoring this backup will result in: |
50b8f9dd CE |
518 | |
519 | .. code-block:: console | |
520 | ||
521 | ls -aR restored | |
522 | restored/: | |
523 | . .. .pxarexclude subfolder0 subfolder1 | |
524 | ||
525 | restored/subfolder0: | |
526 | . .. file0 file2 .pxarexclude | |
527 | ||
528 | restored/subfolder1: | |
529 | . .. file2 | |
a129fdd9 | 530 | |
58ea88c8 DM |
531 | Encryption |
532 | ^^^^^^^^^^ | |
533 | ||
4f3db187 AL |
534 | Proxmox backup supports client side encryption with AES-256 in GCM_ |
535 | mode. First you need to create an encryption key: | |
5a499f32 DM |
536 | |
537 | .. code-block:: console | |
538 | ||
539 | # proxmox-backup-client key create my-backup.key | |
540 | Encryption Key Password: ************** | |
541 | ||
542 | The key is password protected by default. If you do not need this | |
543 | extra protection, you can also create it without a password: | |
544 | ||
545 | .. code-block:: console | |
546 | ||
4f3db187 | 547 | # proxmox-backup-client key create /path/to/my-backup.key --kdf none |
5a499f32 DM |
548 | |
549 | ||
550 | .. code-block:: console | |
551 | ||
552 | # proxmox-backup-client backup etc.pxar:/etc --keyfile /path/to/my-backup.key | |
553 | Password: ********* | |
554 | Encryption Key Password: ************** | |
555 | ... | |
556 | ||
557 | ||
4f3db187 | 558 | You can avoid entering the passwords by setting the environment |
5a499f32 DM |
559 | variables ``PBS_PASSWORD`` and ``PBS_ENCRYPTION_PASSWORD``. |
560 | ||
561 | .. todo:: Explain master-key | |
562 | ||
58ea88c8 DM |
563 | |
564 | Restoring Data | |
565 | ~~~~~~~~~~~~~~ | |
566 | ||
64b85116 | 567 | The regular creation of backups is a necessary step to avoid data |
4f3db187 AL |
568 | loss. More important, however, is the restoration. It is good practice to perform |
569 | periodic recovery tests to ensure that you can access the data in | |
64b85116 DM |
570 | case of problems. |
571 | ||
4f3db187 AL |
572 | First, you need to find the snapshot which you want to restore. The snapshot |
573 | command gives a list of all snapshots on the server: | |
64b85116 DM |
574 | |
575 | .. code-block:: console | |
576 | ||
577 | # proxmox-backup-client snapshots | |
96feecd6 DM |
578 | ┌────────────────────────────────┬─────────────┬────────────────────────────────────┐ |
579 | │ snapshot │ size │ files │ | |
580 | ╞════════════════════════════════╪═════════════╪════════════════════════════════════╡ | |
581 | │ host/elsa/2019-12-03T09:30:15Z │ 51788646825 │ root.pxar catalog.pcat1 index.json │ | |
582 | ├────────────────────────────────┼─────────────┼────────────────────────────────────┤ | |
583 | │ host/elsa/2019-12-03T09:35:01Z │ 51790622048 │ root.pxar catalog.pcat1 index.json │ | |
584 | ├────────────────────────────────┼─────────────┼────────────────────────────────────┤ | |
64b85116 DM |
585 | ... |
586 | ||
4f3db187 | 587 | You can inspect the catalog to find specific files. |
64b85116 DM |
588 | |
589 | .. code-block:: console | |
590 | ||
3c50a9d8 | 591 | # proxmox-backup-client catalog dump host/elsa/2019-12-03T09:35:01Z |
64b85116 DM |
592 | ... |
593 | d "./root.pxar.didx/etc/cifs-utils" | |
594 | l "./root.pxar.didx/etc/cifs-utils/idmap-plugin" | |
595 | d "./root.pxar.didx/etc/console-setup" | |
596 | ... | |
597 | ||
598 | The restore command lets you restore a single archive from the | |
599 | backup. | |
600 | ||
601 | .. code-block:: console | |
602 | ||
603 | # proxmox-backup-client restore host/elsa/2019-12-03T09:35:01Z root.pxar /target/path/ | |
604 | ||
4f3db187 AL |
605 | To get the contents of any archive you can restore the ``ìndex.json`` file in the |
606 | repository and restore it to '-'. This will dump the content to the standard output. | |
64b85116 DM |
607 | |
608 | .. code-block:: console | |
609 | ||
610 | # proxmox-backup-client restore host/elsa/2019-12-03T09:35:01Z index.json - | |
611 | ||
612 | ||
613 | Interactive Restores | |
614 | ^^^^^^^^^^^^^^^^^^^^ | |
615 | ||
616 | If you only want to restore a few individual files, it is often easier | |
617 | to use the interactive recovery shell. | |
618 | ||
619 | .. code-block:: console | |
620 | ||
3c50a9d8 | 621 | # proxmox-backup-client catalog shell host/elsa/2019-12-03T09:35:01Z root.pxar |
64b85116 DM |
622 | Starting interactive shell |
623 | pxar:/ > ls | |
624 | bin boot dev etc home lib lib32 | |
625 | ... | |
626 | ||
3f0983b7 | 627 | The interactive recovery shell is a minimalistic command line interface that |
4f3db187 AL |
628 | utilizes the metadata stored in the catalog to quickly list, navigate and |
629 | search files in a file archive. | |
630 | To restore files, you can select them individually or match them with a glob | |
631 | pattern. | |
632 | ||
633 | Using the catalog for navigation reduces the overhead considerably because only | |
634 | the catalog needs to be downloaded and, optionally, decrypted. | |
3f0983b7 CE |
635 | The actual chunks are only accessed if the metadata in the catalog is not enough |
636 | or for the actual restore. | |
637 | ||
638 | Similar to common UNIX shells ``cd`` and ``ls`` are the commands used to change | |
4f3db187 | 639 | working directory and list directory contents in the archive. |
3f0983b7 CE |
640 | ``pwd`` shows the full path of the current working directory with respect to the |
641 | archive root. | |
642 | ||
643 | Being able to quickly search the contents of the archive is a often needed feature. | |
644 | That's where the catalog is most valuable. | |
645 | For example: | |
646 | ||
647 | .. code-block:: console | |
648 | ||
649 | pxar:/ > find etc/ **/*.txt --select | |
650 | "/etc/X11/rgb.txt" | |
651 | pxar:/ > list-selected | |
652 | etc/**/*.txt | |
653 | pxar:/ > restore-selected /target/path | |
654 | ... | |
655 | ||
656 | This will find and print all files ending in ``.txt`` located in ``etc/`` or a | |
657 | subdirectory and add the corresponding pattern to the list for subsequent restores. | |
658 | ``list-selected`` shows these patterns and ``restore-selected`` finally restores | |
659 | all files in the archive matching the patterns to ``/target/path`` on the local | |
660 | host. This will scan the whole archive. | |
661 | ||
662 | With ``restore /target/path`` you can restore the sub-archive given by the current | |
663 | working directory to the local target path ``/target/path`` on your host. | |
664 | By additionally passing a glob pattern with ``--pattern <glob>``, the restore is | |
665 | further limited to files matching the pattern. | |
666 | For example: | |
667 | ||
668 | .. code-block:: console | |
669 | ||
670 | pxar:/ > cd /etc/ | |
671 | pxar:/etc/ > restore /target/ --pattern **/*.conf | |
672 | ... | |
673 | ||
674 | The above will scan trough all the directories below ``/etc`` and restore all | |
675 | files ending in ``.conf``. | |
676 | ||
677 | .. todo:: Explain interactive restore in more detail | |
64b85116 | 678 | |
c7971d7f CE |
679 | Mounting of Archives via FUSE |
680 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
681 | ||
682 | The :term:`FUSE` implementation for the pxar archive allows you to mount a | |
683 | file archive as a read-only filesystem to a mountpoint on your host. | |
684 | ||
685 | .. code-block:: console | |
686 | ||
687 | # proxmox-backup-client mount host/backup-client/2020-01-29T11:29:22Z root.pxar /mnt | |
688 | # ls /mnt | |
689 | bin dev home lib32 libx32 media opt root sbin sys usr | |
690 | boot etc lib lib64 lost+found mnt proc run srv tmp var | |
691 | ||
692 | This allows you to access the full content of the archive in a seamless manner. | |
693 | ||
694 | .. note:: As the FUSE connection needs to fetch and decrypt chunks from the | |
695 | backup servers datastore, this can cause some additional network and CPU | |
696 | load on your host, depending on the operations you perform on the mounted | |
697 | filesystem. | |
698 | ||
4f3db187 | 699 | To unmount the filesystem use the ``umount`` command on the mountpoint: |
c7971d7f CE |
700 | |
701 | .. code-block:: console | |
702 | ||
703 | # umount /mnt | |
58ea88c8 | 704 | |
ac456d85 DM |
705 | Login and Logout |
706 | ~~~~~~~~~~~~~~~~ | |
707 | ||
708 | The client tool prompts you to enter the logon password as soon as you | |
709 | want to access the backup server. The server checks your credentials | |
710 | and responds with a ticket that is valid for two hours. The client | |
4f3db187 | 711 | tool automatically stores that ticket and uses it for further requests |
ac456d85 DM |
712 | to this server. |
713 | ||
714 | You can also manually trigger this login/logout using the login and | |
715 | logout commands: | |
716 | ||
717 | .. code-block:: console | |
718 | ||
719 | # proxmox-backup-client login | |
720 | Password: ********** | |
721 | ||
4f3db187 | 722 | To remove the ticket, issue a logout: |
ac456d85 DM |
723 | |
724 | .. code-block:: console | |
725 | ||
726 | # proxmox-backup-client logout | |
727 | ||
728 | ||
6e5a0c03 DM |
729 | Pruning and Removing Backups |
730 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | |
731 | ||
732 | You can manually delete a backup snapshot using the ``forget`` | |
733 | command: | |
734 | ||
735 | .. code-block:: console | |
736 | ||
737 | # proxmox-backup-client forget <snapshot> | |
738 | ||
739 | ||
4f3db187 AL |
740 | .. caution:: This command removes all archives in this backup |
741 | snapshot. They will be inaccessible and unrecoverable. | |
6e5a0c03 DM |
742 | |
743 | ||
4f3db187 | 744 | The manual removal is sometimes required, but normally the prune |
52b2be97 | 745 | command is used to systematically delete older backups. Prune lets |
4f3db187 AL |
746 | you specify which backup snapshots you want to keep. The |
747 | following retention options are available: | |
52b2be97 DM |
748 | |
749 | ``--keep-last <N>`` | |
750 | Keep the last ``<N>`` backup snapshots. | |
751 | ||
102d8d41 | 752 | ``--keep-hourly <N>`` |
4f3db187 AL |
753 | Keep backups for the last ``<N>`` hours. If there is more than one |
754 | backup for a single hour, only the latest is kept. | |
102d8d41 | 755 | |
52b2be97 | 756 | ``--keep-daily <N>`` |
4f3db187 AL |
757 | Keep backups for the last ``<N>`` days. If there is more than one |
758 | backup for a single day, only the latest is kept. | |
52b2be97 DM |
759 | |
760 | ``--keep-weekly <N>`` | |
4f3db187 AL |
761 | Keep backups for the last ``<N>`` weeks. If there is more than one |
762 | backup for a single week, only the latest is kept. | |
52b2be97 | 763 | |
4f3db187 AL |
764 | .. note:: Weeks start on Monday and end on Sunday. The software |
765 | uses the `ISO week date`_ system and handles weeks at | |
766 | the end of the year correctly. | |
1af66370 | 767 | |
52b2be97 | 768 | ``--keep-monthly <N>`` |
4f3db187 AL |
769 | Keep backups for the last ``<N>`` months. If there is more than one |
770 | backup for a single month, only the latest is kept. | |
52b2be97 DM |
771 | |
772 | ``--keep-yearly <N>`` | |
4f3db187 AL |
773 | Keep backups for the last ``<N>`` years. If there is more than one |
774 | backup for a single year, only the latest is kept. | |
775 | ||
776 | The retention options are processed in the order given above. Each option | |
777 | only covers backups within its time period. The next option does not take care | |
778 | of already covered backups. It will only consider older backups. | |
52b2be97 | 779 | |
4f3db187 AL |
780 | Unfinished and incomplete backups will be removed by the prune command unless |
781 | they are newer than the last successful backup. In this case, the last failed | |
782 | backup is retained. | |
02d22dec | 783 | |
6e5a0c03 DM |
784 | .. code-block:: console |
785 | ||
786 | # proxmox-backup-client prune <group> --keep-daily 7 --keep-weekly 4 --keep-monthly 3 | |
787 | ||
788 | ||
4f3db187 AL |
789 | You can use the ``--dry-run`` option to test your settings. This only |
790 | shows the list of existing snapshots and which action prune would take. | |
84322d8c DM |
791 | |
792 | .. code-block:: console | |
793 | ||
794 | # proxmox-backup-client prune host/elsa --dry-run --keep-daily 1 --keep-weekly 3 | |
a66d5898 DM |
795 | ┌────────────────────────────────┬──────┐ |
796 | │ snapshot │ keep │ | |
797 | ╞════════════════════════════════╪══════╡ | |
798 | │ host/elsa/2019-12-04T13:20:37Z │ 1 │ | |
799 | ├────────────────────────────────┼──────┤ | |
800 | │ host/elsa/2019-12-03T09:35:01Z │ 0 │ | |
801 | ├────────────────────────────────┼──────┤ | |
802 | │ host/elsa/2019-11-22T11:54:47Z │ 1 │ | |
803 | ├────────────────────────────────┼──────┤ | |
804 | │ host/elsa/2019-11-21T12:36:25Z │ 0 │ | |
805 | ├────────────────────────────────┼──────┤ | |
806 | │ host/elsa/2019-11-10T10:42:20Z │ 1 │ | |
807 | └────────────────────────────────┴──────┘ | |
84322d8c | 808 | |
52b2be97 | 809 | .. note:: Neither the ``prune`` command nor the ``forget`` command free space |
4f3db187 AL |
810 | in the chunk-store. The chunk-store still contains the data blocks. To free |
811 | space you need to perform :ref:`garbage-collection`. | |
6e5a0c03 DM |
812 | |
813 | ||
814 | .. _garbage-collection: | |
815 | ||
816 | Garbage Collection | |
817 | ~~~~~~~~~~~~~~~~~~ | |
818 | ||
e1c356ec DM |
819 | The ``prune`` command removes only the backup index files, not the data |
820 | from the data store. This task is left to the garbage collection | |
4f3db187 | 821 | command. It is recommended to carry out garbage collection on a regular basis. |
e1c356ec DM |
822 | |
823 | The garbage collection works in two phases. In the first phase, all | |
824 | data blocks that are still in use are marked. In the second phase, | |
825 | unused data blocks are removed. | |
826 | ||
827 | .. note:: This command needs to read all existing backup index files | |
f0188322 CE |
828 | and touches the complete chunk-store. This can take a long time |
829 | depending on the number of chunks and the speed of the underlying | |
e1c356ec DM |
830 | disks. |
831 | ||
832 | ||
833 | .. code-block:: console | |
834 | ||
835 | # proxmox-backup-client garbage-collect | |
836 | starting garbage collection on store store2 | |
837 | Start GC phase1 (mark used chunks) | |
838 | Start GC phase2 (sweep unused chunks) | |
839 | percentage done: 1, chunk count: 219 | |
840 | percentage done: 2, chunk count: 453 | |
841 | ... | |
842 | percentage done: 99, chunk count: 21188 | |
843 | Removed bytes: 411368505 | |
844 | Removed chunks: 203 | |
845 | Original data bytes: 327160886391 | |
846 | Disk bytes: 52767414743 (16 %) | |
847 | Disk chunks: 21221 | |
848 | Average chunk size: 2486565 | |
849 | TASK OK | |
850 | ||
851 | ||
852 | .. todo:: howto run garbage-collection at regular intervalls (cron) | |
6e5a0c03 DM |
853 | |
854 | ||
a129fdd9 DM |
855 | .. _pve-integration: |
856 | ||
58ea88c8 DM |
857 | `Proxmox VE`_ integration |
858 | ------------------------- | |
cb01363c | 859 | |
f9dcfa41 DM |
860 | You need to define a new storage with type 'pbs' on your `Proxmox VE`_ |
861 | node. The following example uses ``store2`` as storage name, and | |
862 | assumes the server address is ``localhost``, and you want to connect | |
863 | as ``user1@pbs``. | |
864 | ||
865 | .. code-block:: console | |
866 | ||
867 | # pvesm add pbs store2 --server localhost --datastore store2 | |
868 | # pvesm set store2 --username user1@pbs --password <secret> | |
869 | ||
870 | If your backup server uses a self signed certificate, you need to add | |
871 | the certificate fingerprint to the configuration. You can get the | |
872 | fingerprint by running the following command on the backup server: | |
873 | ||
874 | .. code-block:: console | |
875 | ||
876 | # proxmox-backup-manager cert info |grep Fingerprint | |
877 | Fingerprint (sha256): 64:d3:ff:3a:50:38:53:5a:9b:f7:50:...:ab:fe | |
878 | ||
879 | Please add that fingerprint to your configuration to establish a trust | |
880 | relationship: | |
881 | ||
882 | .. code-block:: console | |
883 | ||
884 | # pvesm set store2 --fingerprint 64:d3:ff:3a:50:38:53:5a:9b:f7:50:...:ab:fe | |
885 | ||
886 | After that you should be able to see storage status with: | |
887 | ||
888 | .. code-block:: console | |
889 | ||
890 | # pvesm status --storage store2 | |
891 | Name Type Status Total Used Available % | |
892 | store2 pbs active 3905109820 1336687816 2568422004 34.23% | |
893 | ||
894 | ||
cb01363c DM |
895 | |
896 | .. include:: command-line-tools.rst | |
897 | ||
898 | .. include:: services.rst | |
85325c25 DM |
899 | |
900 | .. include host system admin at the end | |
901 | ||
902 | .. include:: sysadmin.rst |