]>
Commit | Line | Data |
---|---|---|
cb01363c DM |
1 | Administration Guide |
2 | ==================== | |
7e688b71 | 3 | |
85e139b7 | 4 | The administration guide. |
7e688b71 | 5 | |
c4f1b69f | 6 | |
fea8789c DM |
7 | Terminology |
8 | ----------- | |
9 | ||
85e139b7 DM |
10 | Backup Content |
11 | ~~~~~~~~~~~~~~ | |
12 | ||
13 | When doing deduplication, there are different strategies to get | |
14 | optimal results in terms of performance and/or deduplication rates. | |
57905a61 | 15 | Depending on the type of data, one can split data into fixed or variable |
85e139b7 DM |
16 | sized chunks. |
17 | ||
18 | Fixed sized chunking needs almost no CPU performance, and is used to | |
19 | backup virtual machine images. | |
20 | ||
21 | Variable sized chunking needs more CPU power, but is essential to get | |
22 | good deduplication rates for file archives. | |
23 | ||
24 | Therefore, the backup server supports both strategies. | |
25 | ||
26 | ||
57905a61 DM |
27 | File Archives: ``<name>.pxar`` |
28 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
85e139b7 DM |
29 | |
30 | .. see https://moinakg.wordpress.com/2013/06/22/high-performance-content-defined-chunking/ | |
31 | ||
32 | A file archive stores a whole directory tree. Content is stored using | |
33 | the :ref:`pxar-format`, split into variable sized chunks. The format | |
34 | is specially optimized to achieve good deduplication rates. | |
35 | ||
36 | ||
57905a61 DM |
37 | Image Archives: ``<name>.img`` |
38 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
85e139b7 DM |
39 | |
40 | This is used for virtual machine images and other large binary | |
41 | data. Content is split into fixed sized chunks. | |
42 | ||
43 | ||
44 | Binary Data (BLOBs) | |
45 | ^^^^^^^^^^^^^^^^^^^ | |
46 | ||
47 | This type is used to store smaller (< 16MB) binaries like | |
48 | configuration data. Larger files should be stored as image archive. | |
49 | ||
50 | .. caution:: Please do not store all files as BLOBs. Instead, use the | |
51 | file archive to store whole directory trees. | |
52 | ||
53 | ||
57905a61 DM |
54 | Catalog File: ``catalog.pcat1`` |
55 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
56 | ||
57 | The catalog file is basically an index for file archive. It contains | |
58 | the list of files, and is used to speedup search operations. | |
59 | ||
60 | ||
61 | The Manifest: ``index.json`` | |
62 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
63 | ||
64 | The manifest contains the list of all backup files, including | |
65 | file sizes and checksums. It is used to verify the consistency of a | |
66 | backup. | |
67 | ||
68 | ||
fea8789c DM |
69 | Backup Type |
70 | ~~~~~~~~~~~ | |
71 | ||
72 | The backup server groups backups by *type*, where *type* is one of: | |
73 | ||
74 | ``vm`` | |
a129fdd9 | 75 | This type is used for :term:`virtual machine`\ s. Typically |
fea8789c DM |
76 | contains the virtual machine configuration and an image archive |
77 | for each disk. | |
78 | ||
79 | ``ct`` | |
a129fdd9 | 80 | This type is used for :term:`container`\ s. Contains the container |
fea8789c DM |
81 | configuration and a single file archive for the container content. |
82 | ||
83 | ``host`` | |
a129fdd9 DM |
84 | This type is used for physical host, or if you want to run backups |
85 | manually from inside virtual machines or containers. Such backups | |
86 | may contain file and image archives (no restrictions here). | |
fea8789c DM |
87 | |
88 | ||
89 | Backup ID | |
90 | ~~~~~~~~~ | |
91 | ||
92 | An unique ID. Usually the virtual machine or container ID. ``host`` | |
93 | type backups normally use the hostname. | |
94 | ||
95 | ||
96 | Backup Time | |
97 | ~~~~~~~~~~~ | |
98 | ||
99 | The time when the backup was made. | |
100 | ||
101 | ||
6e5a0c03 DM |
102 | Backup Group |
103 | ~~~~~~~~~~~~ | |
104 | ||
105 | We call the tuple ``<type>/<ID>`` a backup group. Such group | |
106 | may contains one or more backup snapshots. | |
107 | ||
108 | ||
fea8789c DM |
109 | Backup Snapshot |
110 | ~~~~~~~~~~~~~~~ | |
111 | ||
112 | We call the triplet ``<type>/<ID>/<time>`` a backup snapshot. It | |
113 | uniquely identifies a specific backup within a datastore. | |
114 | ||
115 | .. code-block:: console | |
116 | :caption: Backup Snapshot Examples | |
117 | ||
118 | vm/104/2019-10-09T08:01:06Z | |
119 | host/elsa/2019-11-08T09:48:14Z | |
120 | ||
121 | As you can see, the time is formatted as RFC3399_ using Coordinated | |
122 | Universal Time (UTC_, identified by the trailing *Z*). | |
123 | ||
124 | ||
125 | :term:`DataStore` | |
126 | ~~~~~~~~~~~~~~~~~ | |
127 | ||
128 | A datastore is a place to store backups. The current implementation | |
129 | uses a directory inside a standard unix file system (``ext4``, ``xfs`` | |
130 | or ``zfs``) to store backup data. | |
131 | ||
132 | Datastores are identified by a simple *ID*. You can configure that | |
133 | when setting up the backup server. | |
134 | ||
135 | ||
cb01363c DM |
136 | Backup Server Management |
137 | ------------------------ | |
58ea88c8 DM |
138 | |
139 | The command line tool to configure and manage the server is called | |
140 | :command:`proxmox-backup-manager`. | |
141 | ||
142 | ||
143 | Datastore Configuration | |
144 | ~~~~~~~~~~~~~~~~~~~~~~~ | |
145 | ||
146 | A :term:`datastore` is a place to store backups. You can configure | |
147 | several datastores, but you need at least one of them. The datastore is identified by a simple `name` and point to a directory. | |
148 | ||
149 | The following command creates a new datastore called ``store1`` on :file:`/backup/disk1/store1` | |
150 | ||
151 | .. code-block:: console | |
152 | ||
153 | # proxmox-backup-manager datastore create store1 /backup/disk1/store1 | |
154 | ||
155 | To list existing datastores use: | |
156 | ||
157 | .. code-block:: console | |
158 | ||
159 | # proxmox-backup-manager datastore list | |
160 | store1 /backup/disk1/store1 | |
161 | ||
162 | Finally, it is also possible to remove the datastore configuration: | |
163 | ||
164 | .. code-block:: console | |
165 | ||
166 | # proxmox-backup-manager datastore remove store1 | |
167 | ||
168 | .. note:: Above command removes the datastore configuration. It does | |
169 | not delete any data from the underlying directory. | |
170 | ||
171 | ||
fea8789c DM |
172 | File Layout |
173 | ^^^^^^^^^^^ | |
174 | ||
175 | .. todo:: Add datastore file layout example | |
176 | ||
177 | ||
cb01363c DM |
178 | Backup Client usage |
179 | ------------------- | |
58ea88c8 DM |
180 | |
181 | The command line client is called :command:`proxmox-backup-client`. | |
182 | ||
a129fdd9 | 183 | |
58ea88c8 DM |
184 | Respository Locations |
185 | ~~~~~~~~~~~~~~~~~~~~~ | |
186 | ||
187 | The client uses a special repository notation to specify a datastore | |
188 | on the backup server. | |
189 | ||
190 | [[username@]server:]datastore | |
191 | ||
192 | If you do not specify a ``username`` the default is ``root@pam``. The | |
193 | default for server is to use the local host (``localhost``). | |
194 | ||
195 | You can pass the repository by setting the ``--repository`` command | |
196 | line options, or by setting the ``PBS_REPOSITORY`` environment | |
197 | variable. | |
198 | ||
199 | ||
200 | Environment Variables | |
201 | ~~~~~~~~~~~~~~~~~~~~~~ | |
202 | ||
203 | ``PBS_REPOSITORY`` | |
204 | The default backup repository. | |
205 | ||
206 | ``PBS_PASSWORD`` | |
207 | When set, this value is used for the password required for the | |
208 | backup server. | |
209 | ||
210 | ``PBS_ENCRYPTION_PASSWORD`` | |
58ea88c8 DM |
211 | When set, this value is used to access the secret encryption key (if |
212 | protected by password). | |
213 | ||
214 | ||
215 | Creating Backups | |
216 | ~~~~~~~~~~~~~~~~ | |
217 | ||
a129fdd9 DM |
218 | This section explains how to create backup on physical host, or from |
219 | inside virtual machines or containers. Such backups may contain file | |
220 | and image archives (no restrictions here). | |
221 | ||
222 | .. note:: If you want to backup virtual machines or containers see :ref:`pve-integration`. | |
223 | ||
224 | The prerequisite is that you have already set up (or can access) a | |
225 | backup server. It is assumed that you know the repository name and | |
226 | credentials. In the following examples we simply use ``backup-server:store1``. | |
227 | ||
228 | .. code-block:: console | |
229 | ||
230 | # proxmox-backup-client backup root.pxar:/ --repository backup-server:store1 | |
231 | Starting backup: host/elsa/2019-12-03T09:35:01Z | |
232 | Client name: elsa | |
233 | skip mount point: "/boot/efi" | |
234 | skip mount point: "/dev" | |
235 | skip mount point: "/run" | |
236 | skip mount point: "/sys" | |
237 | Uploaded 12129 chunks in 87 seconds (564 MB/s). | |
238 | End Time: 2019-12-03T10:36:29+01:00 | |
239 | ||
240 | This will prompt you for a password and then uploads a file archive named | |
241 | ``root.pxar`` containing all the files in the ``/`` directory. | |
242 | ||
243 | .. Caution:: Please note that proxmox-backup-client does not | |
ed858b0a | 244 | automatically include mount points. Instead, you will see a short |
a129fdd9 DM |
245 | ``skip mount point`` notice for each of them. The idea is that you |
246 | create a separate file archive for each mounted disk. You can also | |
247 | explicitly include them using the ``--include-dev`` option | |
248 | (i.e. ``--include-dev /boot/efi``). You can use this option | |
249 | multiple times, once for each mount point you want to include. | |
250 | ||
251 | The ``--repository`` option is sometimes quite long and is used by all | |
252 | commands. You can avoid having to enter this value by setting the | |
253 | environment variable ``PBS_REPOSITORY``. | |
254 | ||
255 | .. code-block:: console | |
256 | ||
257 | # export PBS_REPOSTORY=backup-server:store1 | |
258 | ||
259 | You can then execute all commands without specifying the ``--repository`` | |
260 | option. | |
261 | ||
262 | One signle backup is allowed to contain more than one archive. For example, assume you want to backup two disks mounted at ``/mmt/disk1`` and ``/mnt/disk2``: | |
263 | ||
264 | .. code-block:: console | |
265 | ||
266 | # proxmox-backup-client backup disk1.pxar:/mnt/disk1 disk2.pxar:/mnt/disk2 | |
267 | ||
268 | This create a backup of both disks. | |
269 | ||
270 | The backup command takes a list of backup specifications, which | |
271 | include archive name on the server, the type of the archive, and the | |
272 | archive source at the client. The format is quite simple to understand: | |
273 | ||
274 | <archive-name>.<type>:<source-path> | |
275 | ||
276 | Common types are ``.pxar`` for file archives, and ``.img`` for block | |
277 | device images. Thus it is quite easy to create a backup for a block | |
278 | device: | |
279 | ||
280 | .. code-block:: console | |
281 | ||
282 | # proxmox-backup-client backup mydata.img:/dev/mylvm/mydata | |
283 | ||
284 | ||
58ea88c8 DM |
285 | Encryption |
286 | ^^^^^^^^^^ | |
287 | ||
5a499f32 DM |
288 | Proxmox backup support client side encryption using AES-256 in GCM_ |
289 | mode. You first need to create an encryption key in order to use that: | |
290 | ||
291 | .. code-block:: console | |
292 | ||
293 | # proxmox-backup-client key create my-backup.key | |
294 | Encryption Key Password: ************** | |
295 | ||
296 | The key is password protected by default. If you do not need this | |
297 | extra protection, you can also create it without a password: | |
298 | ||
299 | .. code-block:: console | |
300 | ||
301 | # proxmox-backup-client key create /path/to/my-backup.key --kdf none | |
302 | ||
303 | ||
304 | .. code-block:: console | |
305 | ||
306 | # proxmox-backup-client backup etc.pxar:/etc --keyfile /path/to/my-backup.key | |
307 | Password: ********* | |
308 | Encryption Key Password: ************** | |
309 | ... | |
310 | ||
311 | ||
312 | You can avoid having to enter the passwords by setting the environment | |
313 | variables ``PBS_PASSWORD`` and ``PBS_ENCRYPTION_PASSWORD``. | |
314 | ||
315 | .. todo:: Explain master-key | |
316 | ||
58ea88c8 DM |
317 | |
318 | Restoring Data | |
319 | ~~~~~~~~~~~~~~ | |
320 | ||
64b85116 DM |
321 | The regular creation of backups is a necessary step to avoid data |
322 | loss. More important, however, is the restoration. Be sure to perform | |
323 | periodic recovery tests to ensure that you can access your data in | |
324 | case of problems. | |
325 | ||
326 | First, you need to find the snapshot you want to restore. The snapshot | |
327 | command gives you a list of all snapshots on the server: | |
328 | ||
329 | .. code-block:: console | |
330 | ||
331 | # proxmox-backup-client snapshots | |
332 | ... | |
333 | host/elsa/2019-12-03T09:30:15Z | 51788646825 | root.pxar catalog.pcat1 index.json | |
334 | host/elsa/2019-12-03T09:35:01Z | 51790622048 | root.pxar catalog.pcat1 index.json | |
335 | ... | |
336 | ||
337 | You can also inspect the catalog to find specific files. | |
338 | ||
339 | .. code-block:: console | |
340 | ||
341 | # proxmox-backup-client catalog host/elsa/2019-12-03T09:35:01Z | |
342 | ... | |
343 | d "./root.pxar.didx/etc/cifs-utils" | |
344 | l "./root.pxar.didx/etc/cifs-utils/idmap-plugin" | |
345 | d "./root.pxar.didx/etc/console-setup" | |
346 | ... | |
347 | ||
348 | The restore command lets you restore a single archive from the | |
349 | backup. | |
350 | ||
351 | .. code-block:: console | |
352 | ||
353 | # proxmox-backup-client restore host/elsa/2019-12-03T09:35:01Z root.pxar /target/path/ | |
354 | ||
355 | You can instead simply download the contents of any archive using '-' | |
356 | instead of ``/target/path``. This dumps the content to standard | |
357 | output: | |
358 | ||
359 | .. code-block:: console | |
360 | ||
361 | # proxmox-backup-client restore host/elsa/2019-12-03T09:35:01Z index.json - | |
362 | ||
363 | ||
364 | Interactive Restores | |
365 | ^^^^^^^^^^^^^^^^^^^^ | |
366 | ||
367 | If you only want to restore a few individual files, it is often easier | |
368 | to use the interactive recovery shell. | |
369 | ||
370 | .. code-block:: console | |
371 | ||
372 | # proxmox-backup-client shell host/elsa/2019-12-03T09:35:01Z root.pxar | |
373 | Starting interactive shell | |
374 | pxar:/ > ls | |
375 | bin boot dev etc home lib lib32 | |
376 | ... | |
377 | ||
378 | .. todo:: Explain interactive restore | |
379 | ||
58ea88c8 | 380 | |
ac456d85 DM |
381 | Login and Logout |
382 | ~~~~~~~~~~~~~~~~ | |
383 | ||
384 | The client tool prompts you to enter the logon password as soon as you | |
385 | want to access the backup server. The server checks your credentials | |
386 | and responds with a ticket that is valid for two hours. The client | |
387 | tool automatically stores that ticket and use it for further requests | |
388 | to this server. | |
389 | ||
390 | You can also manually trigger this login/logout using the login and | |
391 | logout commands: | |
392 | ||
393 | .. code-block:: console | |
394 | ||
395 | # proxmox-backup-client login | |
396 | Password: ********** | |
397 | ||
398 | To remove the ticket, simply issue a logout: | |
399 | ||
400 | .. code-block:: console | |
401 | ||
402 | # proxmox-backup-client logout | |
403 | ||
404 | ||
6e5a0c03 DM |
405 | Pruning and Removing Backups |
406 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | |
407 | ||
408 | You can manually delete a backup snapshot using the ``forget`` | |
409 | command: | |
410 | ||
411 | .. code-block:: console | |
412 | ||
413 | # proxmox-backup-client forget <snapshot> | |
414 | ||
415 | ||
416 | .. caution:: This command removes all the archives in this backup | |
417 | snapshot so that they are inaccessible and unrecoverable. | |
418 | ||
419 | ||
52b2be97 DM |
420 | Such manual removal is sometimes required, but normally the prune |
421 | command is used to systematically delete older backups. Prune lets | |
422 | you specify which backup snapshots you want to keep. There are the | |
423 | following retention options: | |
424 | ||
425 | ``--keep-last <N>`` | |
426 | Keep the last ``<N>`` backup snapshots. | |
427 | ||
428 | ``--keep-daily <N>`` | |
429 | Keep backups for ``<N>`` different days. If there is more than one | |
430 | backup for a single day, only the latest one is kept. | |
431 | ||
432 | ``--keep-weekly <N>`` | |
433 | Keep backups for ``<N>`` different weeks. If there is more than one | |
434 | backup for a single week, only the latest one is kept. | |
435 | ||
436 | ``--keep-monthly <N>`` | |
437 | Keep backups for ``<N>`` different months. If there is more than one | |
438 | backup for a single month, only the latest one is kept. | |
439 | ||
440 | ``--keep-yearly <N>`` | |
441 | Keep backups for ``<N>`` different year. If there is more than one | |
442 | backup for a single year, only the latest one is kept. | |
443 | ||
444 | ||
445 | Those retention options are processed in the order given above. Each | |
446 | option covers a specific period of time. We say that backups within | |
447 | this period are covered by this option. The next option does not take | |
448 | care of already covered backups and only considers older backups. | |
449 | ||
02d22dec DM |
450 | The prune command also looks for unfinished and incomplete backups and |
451 | removes them unless they are newer than the last successful backup. In | |
452 | this case, the last failed backup is retained. | |
453 | ||
6e5a0c03 DM |
454 | .. code-block:: console |
455 | ||
456 | # proxmox-backup-client prune <group> --keep-daily 7 --keep-weekly 4 --keep-monthly 3 | |
457 | ||
458 | ||
84322d8c DM |
459 | You can use the ``--dry-run`` option to test your settings. This just |
460 | shows the list of existing snapshots and what action prune would take | |
461 | on that. | |
462 | ||
463 | .. code-block:: console | |
464 | ||
465 | # proxmox-backup-client prune host/elsa --dry-run --keep-daily 1 --keep-weekly 3 | |
466 | host/elsa/2019-12-04T13:20:37Z keep | |
467 | host/elsa/2019-12-03T09:35:01Z remove | |
468 | host/elsa/2019-11-22T11:54:47Z keep | |
469 | host/elsa/2019-11-21T12:36:25Z remove | |
470 | host/elsa/2019-11-10T10:42:20Z keep | |
471 | ||
472 | ||
52b2be97 | 473 | .. note:: Neither the ``prune`` command nor the ``forget`` command free space |
6e5a0c03 DM |
474 | in the chunk-store. The chunk-store still contains the data blocks |
475 | unless you are performing :ref:`garbage-collection`. | |
476 | ||
477 | ||
478 | .. _garbage-collection: | |
479 | ||
480 | Garbage Collection | |
481 | ~~~~~~~~~~~~~~~~~~ | |
482 | ||
483 | .. todo:: write garbage-collection section | |
484 | ||
485 | ||
a129fdd9 DM |
486 | .. _pve-integration: |
487 | ||
58ea88c8 DM |
488 | `Proxmox VE`_ integration |
489 | ------------------------- | |
cb01363c DM |
490 | |
491 | ||
492 | .. include:: command-line-tools.rst | |
493 | ||
494 | .. include:: services.rst | |
85325c25 DM |
495 | |
496 | .. include host system admin at the end | |
497 | ||
498 | .. include:: sysadmin.rst |