]>
Commit | Line | Data |
---|---|---|
5f79dc28 DM |
1 | Tape Backup |
2 | =========== | |
3 | ||
4 | Our tape backup solution provides a easy way to store datastore | |
54d96866 | 5 | contents to a tape. This increases data safety because you get: |
5f79dc28 DM |
6 | |
7 | - an additional copy of the data | |
8 | - to a different media type (tape) | |
9 | - to an additional location (you can move tape offsite) | |
10 | ||
54d96866 DM |
11 | Statistics show that 95% of all restore jobs restores the last |
12 | backup. Restore requests further declines the older the data gets. | |
13 | Considering that, tape backup may also help to reduce disk usage, | |
14 | because you can safely remove data from disk once archived on tape. | |
15 | This is especially true if you need to keep data for several years. | |
16 | ||
17 | Tape backups do not provide random access to the stored data. Instead, | |
18 | you need to restore the data to disk before you can access it | |
19 | again. Also, if you store your tapes offsite (using some kind of tape | |
20 | vaulting service), you need to bring them onsite before you can do any | |
21 | restore. So please consider that restores from tapes can take much | |
22 | longer than restores from disk. | |
5f79dc28 DM |
23 | |
24 | ||
25 | Tape Technology Primer | |
26 | ---------------------- | |
27 | ||
28 | .. _Linear Tape Open: https://en.wikipedia.org/wiki/Linear_Tape-Open | |
29 | ||
30 | As of 2021, the only broadly available tape technology standard is | |
31 | `Linear Tape Open`_, and different vendors offers LTO Ultrium tape | |
54d96866 DM |
32 | drives, autoloaders and LTO tape cartridges. |
33 | ||
34 | Of cause, there are a few vendor offering proprietary drives with | |
35 | slight advantages in performance and capacity, but they have | |
36 | significat disadvantages: | |
37 | ||
38 | - proprietary (single vendor) | |
39 | - a much higher purchase cost | |
40 | ||
41 | So we currently do no test such drives. | |
5f79dc28 DM |
42 | |
43 | In general, LTO tapes offer the following advantages: | |
44 | ||
45 | - Durable (30 years) | |
46 | - High Capacity (12 TB) | |
47 | - Relatively low cost per TB | |
48 | - Cold Media | |
49 | - Movable (storable inside vault) | |
50 | - Multiple vendors (for both media and drives) | |
d5efa18a | 51 | |
a174854a | 52 | Please note that `Proxmox Backup Server` already stores compressed |
54d96866 DM |
53 | data, so we do not need/use the tape compression feature. Same applies |
54 | to encryption. | |
55 | ||
d5efa18a DM |
56 | |
57 | Supported Hardware | |
58 | ------------------ | |
59 | ||
60 | Proxmox Backup Server supports `Linear Tape Open`_ genertion 3 | |
61 | (LTO3) or later. In general, all SCSI2 tape drives supported by | |
62 | the Linux kernel should work. | |
63 | ||
64 | Tape changer support is done using the Linux 'mtx' command line | |
65 | tool. So any changer devive supported by that tool work work. | |
66 | ||
67 | ||
8a192bed DM |
68 | Drive Performance |
69 | ~~~~~~~~~~~~~~~~~ | |
70 | ||
71 | Current LTO8 tapes provide read/write speeds up to 360MB/s. Please | |
72 | note that it still takes a minimum of 9 hours to completely write or | |
73 | read a single tape (even at maximum speed). | |
74 | ||
75 | The only way to speed up that data rate is to use more than one | |
76 | drive. That way you can run several backup jobs in parallel, or run | |
77 | restore jobs while the other dives are used for backups. | |
78 | ||
baefbc44 DM |
79 | Also consider that you need to read data first from your datastore |
80 | (disk). But a single spinning disk is unable to deliver data at this | |
81 | rate. We meassured a maximum rate about 100MB/s in practive, so it | |
82 | takes 33 hours to read 12TB to fill up a LTO8 tape. So if you want to | |
83 | run your tape at full speed, please make sure that the source | |
84 | datastore is able to delive that performance (use SSDs). | |
85 | ||
d5efa18a DM |
86 | |
87 | Terminology | |
88 | ----------- | |
89 | ||
90 | :Tape Labels: are used to uniquely indentify a tape. You normally use | |
91 | some sticky paper labels and apply them on the front of the | |
92 | cartridge. We additionally store the label text magnetically on the | |
93 | tape (first file on tape). | |
94 | ||
95 | .. _Code 39: https://en.wikipedia.org/wiki/Code_39 | |
96 | ||
97 | .. _LTO Ultrium Cartridge Label Specification: https://www.ibm.com/support/pages/ibm-lto-ultrium-cartridge-label-specification | |
98 | ||
8001c82e DM |
99 | .. _LTO Barcode Generator: lto-barcode/index.html |
100 | ||
d5efa18a DM |
101 | :Barcodes: are a special form of tape labels, which are electronically |
102 | readable. Most LTO tape robots use an 8 character string encoded as | |
103 | `Code 39`_, as definded in the `LTO Ultrium Cartridge Label | |
104 | Specification`_. | |
105 | ||
106 | You can either bye such barcode labels from your cartidge vendor, | |
8001c82e DM |
107 | or print them yourself. You can use our `LTO Barcode Generator`_ App |
108 | for that. | |
d5efa18a DM |
109 | |
110 | .. Note:: Physical labels and the associated adhesive shall have an | |
111 | environmental performance to match or exceed the environmental | |
112 | specifications of the cartridge to which it is applied. | |
113 | ||
114 | :Media Pools: A media pool is a logical container for tapes. A backup | |
115 | job targets one media pool, so a job only uses tapes from that | |
08d8b2a4 DM |
116 | pool. The pool additionally defines how long a backup job can |
117 | append data to tapes (allocation policy) and how long you want to | |
118 | keep the data (retention policy). | |
119 | ||
120 | :Media Set: A group of continuously written tapes (all from the same | |
121 | media pool). | |
d5efa18a DM |
122 | |
123 | :Tape drive: The decive used to read and write data to the tape. There | |
124 | are standalone drives, but drives often ship within tape libraries. | |
125 | ||
126 | :Tape changer: A device which can change the tapes inside a tape drive | |
127 | (tape robot). They are usually part of a tape library. | |
128 | ||
129 | .. _Tape Library: https://en.wikipedia.org/wiki/Tape_library | |
130 | ||
131 | :`Tape library`_: A storage device that contains one or more tape drives, | |
132 | a number of slots to hold tape cartridges, a barcode reader to | |
133 | identify tape cartridges and an automated method for loading tapes | |
134 | (a robot). | |
135 | ||
136 | People als call this 'autoloader', 'tape robot' or 'tape jukebox'. | |
8a192bed DM |
137 | |
138 | ||
139 | Tape Quickstart | |
140 | --------------- | |
141 | ||
142 | 1. Configure your tape hardware (drives and changers) | |
143 | ||
144 | 2. Configure one or more media pools | |
145 | ||
146 | 3. Label your tape cartridges. | |
147 | ||
148 | 4. Start your first tape backup job ... | |
baefbc44 DM |
149 | |
150 | ||
151 | Configuration | |
152 | ------------- | |
153 | ||
154 | Please note that you can configure anything using the graphical user | |
155 | interface or the command line interface. Both methods results in the | |
156 | same configuration. | |
157 | ||
158 | ||
159 | Tape changers | |
160 | ~~~~~~~~~~~~~ | |
161 | ||
162 | Tape changers (robots) are part of a `Tape Library`_. You can skip | |
163 | this step if you are using a standalone drive. | |
164 | ||
165 | Linux is able to auto detect those devices, and you can get a list | |
a174854a | 166 | of available devices using:: |
baefbc44 DM |
167 | |
168 | # proxmox-tape changer scan | |
a174854a DM |
169 | ┌─────────────────────────────┬─────────┬──────────────┬────────┐ |
170 | │ path │ vendor │ model │ serial │ | |
171 | ╞═════════════════════════════╪═════════╪══════════════╪════════╡ | |
172 | │ /dev/tape/by-id/scsi-CC2C52 │ Quantum │ Superloader3 │ CC2C52 │ | |
173 | └─────────────────────────────┴─────────┴──────────────┴────────┘ | |
baefbc44 DM |
174 | |
175 | In order to use that device with Proxmox, you need to create a | |
04eba29c | 176 | configuration entry:: |
baefbc44 | 177 | |
a174854a | 178 | # proxmox-tape changer create sl3 --path /dev/tape/by-id/scsi-CC2C52 |
baefbc44 DM |
179 | |
180 | Where ``sl3`` is an arbitrary name you can choose. | |
181 | ||
04eba29c | 182 | .. Note:: Please use stable device path names from inside |
baefbc44 DM |
183 | ``/dev/tape/by-id/``. Names like ``/dev/sg0`` may point to a |
184 | different device after reboot, and that is not what you want. | |
185 | ||
a174854a | 186 | You can show the final configuration with:: |
baefbc44 | 187 | |
04eba29c DM |
188 | # proxmox-tape changer config sl3 |
189 | ┌──────┬─────────────────────────────┐ | |
190 | │ Name │ Value │ | |
191 | ╞══════╪═════════════════════════════╡ | |
192 | │ name │ sl3 │ | |
193 | ├──────┼─────────────────────────────┤ | |
194 | │ path │ /dev/tape/by-id/scsi-CC2C52 │ | |
195 | └──────┴─────────────────────────────┘ | |
196 | ||
197 | Or simply list all configured changer devices:: | |
198 | ||
baefbc44 | 199 | # proxmox-tape changer list |
a174854a DM |
200 | ┌──────┬─────────────────────────────┬─────────┬──────────────┬────────────┐ |
201 | │ name │ path │ vendor │ model │ serial │ | |
202 | ╞══════╪═════════════════════════════╪═════════╪══════════════╪════════════╡ | |
203 | │ sl3 │ /dev/tape/by-id/scsi-CC2C52 │ Quantum │ Superloader3 │ CC2C52 │ | |
204 | └──────┴─────────────────────────────┴─────────┴──────────────┴────────────┘ | |
baefbc44 DM |
205 | |
206 | The Vendor, Model and Serial number are auto detected, but only shown | |
207 | if the device is online. | |
208 | ||
a174854a DM |
209 | To test your setup, please query the status of the changer device with:: |
210 | ||
211 | # proxmox-tape changer status sl3 | |
212 | ┌───────────────┬──────────┬────────────┬─────────────┐ | |
213 | │ entry-kind │ entry-id │ changer-id │ loaded-slot │ | |
214 | ╞═══════════════╪══════════╪════════════╪═════════════╡ | |
215 | │ drive │ 0 │ vtape1 │ 1 │ | |
216 | ├───────────────┼──────────┼────────────┼─────────────┤ | |
217 | │ slot │ 1 │ │ │ | |
218 | ├───────────────┼──────────┼────────────┼─────────────┤ | |
219 | │ slot │ 2 │ vtape2 │ │ | |
220 | ├───────────────┼──────────┼────────────┼─────────────┤ | |
221 | │ ... │ ... │ │ │ | |
222 | ├───────────────┼──────────┼────────────┼─────────────┤ | |
223 | │ slot │ 16 │ │ │ | |
224 | └───────────────┴──────────┴────────────┴─────────────┘ | |
225 | ||
0912878e DM |
226 | Tape libraries usually provide some special import/export slots (also |
227 | called "mail slots"). Tapes inside those slots are acessible from | |
228 | outside, making it easy to add/remove tapes to/from the library. Those | |
229 | tapes are considered to be "offline", so backup jobs will not use | |
230 | them. Those special slots are auto-detected and marked as | |
231 | ``import-export`` slot in the status command. | |
232 | ||
233 | It's worth noting that some of the smaller tape libraries don't have | |
234 | such slots. While they have something called "Mail Slot", that slot | |
235 | is just a way to grab the tape from the gripper. But they are unable | |
236 | to hold media while the robot does other things. They also do not | |
237 | expose that "Mail Slot" over the SCSI interface, so you wont see them in | |
238 | the status output. | |
239 | ||
240 | As a workaround, you can mark some of the normal slots as export | |
241 | slot. The software treats those slots like real ``import-export`` | |
242 | slots, and the media inside those slots is considered to be 'offline' | |
243 | (not available for backup):: | |
244 | ||
245 | # proxmox-tape changer update sl3 --export-slots 15,16 | |
246 | ||
247 | After that, you can see those artificial ``import-export`` slots in | |
248 | the status output:: | |
249 | ||
04eba29c | 250 | # proxmox-tape changer status sl3 |
0912878e DM |
251 | ┌───────────────┬──────────┬────────────┬─────────────┐ |
252 | │ entry-kind │ entry-id │ changer-id │ loaded-slot │ | |
253 | ╞═══════════════╪══════════╪════════════╪═════════════╡ | |
254 | │ drive │ 0 │ vtape1 │ 1 │ | |
255 | ├───────────────┼──────────┼────────────┼─────────────┤ | |
256 | │ import-export │ 15 │ │ │ | |
257 | ├───────────────┼──────────┼────────────┼─────────────┤ | |
258 | │ import-export │ 16 │ │ │ | |
259 | ├───────────────┼──────────┼────────────┼─────────────┤ | |
260 | │ slot │ 1 │ │ │ | |
261 | ├───────────────┼──────────┼────────────┼─────────────┤ | |
262 | │ slot │ 2 │ vtape2 │ │ | |
263 | ├───────────────┼──────────┼────────────┼─────────────┤ | |
264 | │ ... │ ... │ │ │ | |
265 | ├───────────────┼──────────┼────────────┼─────────────┤ | |
266 | │ slot │ 14 │ │ │ | |
267 | └───────────────┴──────────┴────────────┴─────────────┘ | |
a174854a | 268 | |
baefbc44 DM |
269 | |
270 | Tape drives | |
271 | ~~~~~~~~~~~ | |
272 | ||
04eba29c DM |
273 | Linux is able to auto detect tape drives, and you can get a list |
274 | of available tape drives using:: | |
275 | ||
276 | # proxmox-tape drive scan | |
277 | ┌────────────────────────────────┬────────┬─────────────┬────────┐ | |
278 | │ path │ vendor │ model │ serial │ | |
279 | ╞════════════════════════════════╪════════╪═════════════╪════════╡ | |
280 | │ /dev/tape/by-id/scsi-12345-nst │ IBM │ ULT3580-TD4 │ 12345 │ | |
281 | └────────────────────────────────┴────────┴─────────────┴────────┘ | |
282 | ||
283 | In order to use that drive with Proxmox, you need to create a | |
284 | configuration entry:: | |
285 | ||
286 | # proxmox-tape drive create mydrive --path /dev/tape/by-id/scsi-12345-nst | |
287 | ||
288 | .. Note:: Please use stable device path names from inside | |
289 | ``/dev/tape/by-id/``. Names like ``/dev/nst0`` may point to a | |
290 | different device after reboot, and that is not what you want. | |
291 | ||
292 | If you have a tape library, you also need to set the associated | |
293 | changer device:: | |
294 | ||
295 | # proxmox-tape drive update mydrive --changer sl3 --changer-drive-id 0 | |
296 | ||
297 | The ``--changer-drive-id`` is only necessary if the tape library | |
298 | includes more than one drive (The changer status command lists all | |
299 | drive IDs). | |
300 | ||
301 | You can show the final configuration with:: | |
302 | ||
303 | # proxmox-tape drive config mydrive | |
304 | ┌─────────┬────────────────────────────────┐ | |
305 | │ Name │ Value │ | |
306 | ╞═════════╪════════════════════════════════╡ | |
307 | │ name │ mydrive │ | |
308 | ├─────────┼────────────────────────────────┤ | |
309 | │ path │ /dev/tape/by-id/scsi-12345-nst │ | |
310 | ├─────────┼────────────────────────────────┤ | |
311 | │ changer │ sl3 │ | |
312 | └─────────┴────────────────────────────────┘ | |
313 | ||
314 | .. NOTE:: The ``changer-drive-id`` value 0 is not stored in the | |
315 | configuration, because that is the default. | |
316 | ||
317 | To list all configured drives use:: | |
318 | ||
319 | # proxmox-tape drive list | |
320 | ┌──────────┬────────────────────────────────┬─────────┬────────┬─────────────┬────────┐ | |
321 | │ name │ path │ changer │ vendor │ model │ serial │ | |
322 | ╞══════════╪════════════════════════════════╪═════════╪════════╪═════════════╪════════╡ | |
323 | │ mydrive │ /dev/tape/by-id/scsi-12345-nst │ sl3 │ IBM │ ULT3580-TD4 │ 12345 │ | |
324 | └──────────┴────────────────────────────────┴─────────┴────────┴─────────────┴────────┘ | |
325 | ||
326 | The Vendor, Model and Serial number are auto detected, but only shown | |
327 | if the device is online. | |
328 | ||
632756b6 DM |
329 | For testing, you can simply query the drive status with:: |
330 | ||
331 | # proxmox-tape status --drive mydrive | |
332 | ┌───────────┬────────────────────────┐ | |
333 | │ Name │ Value │ | |
334 | ╞═══════════╪════════════════════════╡ | |
335 | │ blocksize │ 0 │ | |
336 | ├───────────┼────────────────────────┤ | |
337 | │ status │ DRIVE_OPEN | IM_REP_EN │ | |
338 | └───────────┴────────────────────────┘ | |
339 | ||
340 | .. NOTE:: Blocksize should always be 0 (variable block size | |
341 | mode). This is the default anyways. | |
342 | ||
baefbc44 DM |
343 | |
344 | Media Pools | |
345 | ~~~~~~~~~~~ | |
346 | ||
08d8b2a4 DM |
347 | A media pool is a logical container for tapes. A backup job targets |
348 | one media pool, so a job only uses tapes from that pool. | |
349 | ||
350 | .. topic:: Media Set | |
351 | ||
352 | A media set is a group of continuously written tapes, used to split | |
353 | the larger pool into smaller, restorable units. One or more backup | |
59217472 DM |
354 | jobs write to a media set, producing an ordered group of |
355 | tapes. Media sets are identified by an unique ID. That ID and the | |
356 | sequence number is stored on each tape of that set (tape label). | |
08d8b2a4 DM |
357 | |
358 | Media sets are the basic unit for restore tasks, i.e. you need all | |
59217472 | 359 | tapes in the set to restore the media set content. Data is fully |
08d8b2a4 DM |
360 | deduplicated inside a media set. |
361 | ||
362 | ||
363 | .. topic:: Media Set Allocation Policy | |
364 | ||
365 | The pool additionally defines how long backup jobs can append data | |
59217472 | 366 | to a media set. The following settings are possible: |
08d8b2a4 | 367 | |
59217472 | 368 | - Try to use the current media set. |
08d8b2a4 | 369 | |
59217472 DM |
370 | This setting produce one large media set. While this is very |
371 | space efficient (deduplication, no unused space), it can lead to | |
372 | long restore times, because restore jobs needs to read all tapes in the | |
373 | set. | |
08d8b2a4 | 374 | |
59217472 DM |
375 | .. NOTE:: Data is fully deduplicated inside a media set. That |
376 | also means that data is randomly distributed over the tapes in | |
377 | the set. So even if you restore a single VM, this may have to | |
378 | read data from all tapes inside the media set. | |
379 | ||
380 | Larger media sets are also more error prone, because a single | |
381 | damaged media makes the restore fail. | |
382 | ||
383 | Usage scenario: Mostly used with tape libraries, and you manually | |
384 | trigger new set creation by running a backup job with the | |
385 | ``--export`` option. | |
386 | ||
387 | - Always create a new media set. | |
388 | ||
389 | With this setting each backup job creates a new media set. This | |
390 | is less space efficient, because the last media from the last set | |
391 | may not be fully written, leaving the remaining space unused. | |
392 | ||
393 | The advantage is that this procudes media sets of minimal | |
394 | size. Small set are easier to handle, you can move sets to an | |
395 | off-site vault, and restore is much faster. | |
396 | ||
397 | - Create a new set when the specified Calendar Event triggers. | |
398 | ||
399 | .. _systemd.time manpage: https://manpages.debian.org/buster/systemd/systemd.time.7.en.html | |
400 | ||
401 | This allows you to specify points in time by using systemd like | |
402 | Calendar Event specifications (see `systemd.time manpage`_). | |
403 | ||
404 | For example, the value ``weekly`` (or ``Mon *-*-* 00:00:00``) | |
405 | will create a new set each week. | |
406 | ||
407 | This balances between space efficency and media count. | |
08d8b2a4 DM |
408 | |
409 | Additionally, the following events may allocate a new media set: | |
410 | ||
59217472 | 411 | - Required tape is offline (and you use a tape library). |
08d8b2a4 DM |
412 | |
413 | - Current set contains damaged of retired tapes. | |
414 | ||
59217472 DM |
415 | - Database consistency errors, e.g. if the inventory does not |
416 | contain required media info, or contain conflicting infos | |
417 | (outdated data). | |
08d8b2a4 DM |
418 | |
419 | .. topic:: Retention Policy | |
420 | ||
3a2cc5c6 DM |
421 | Defines how long we want to keep the data. Retention period starts |
422 | with the existence of a newer media set, e.g. when the allocation | |
423 | policy calendar event triggers. | |
08d8b2a4 | 424 | |
3838ce33 DM |
425 | - Always overwrite media. |
426 | ||
427 | - Protect data for the duration specified. | |
428 | ||
429 | We use systemd like time spans to specify durations, e.g. ``2 | |
430 | weeks`` (see `systemd.time manpage`_). | |
431 | ||
432 | - Never overwrite data. | |
08d8b2a4 DM |
433 | |
434 | .. NOTE:: FIXME: Add note about global content namespace. (We do not store | |
435 | the source datastore, so it is impossible to distinguish | |
436 | store1:/vm/100 from store2:/vm/100. Please use different media | |
437 | pools if the source is from a different name space) | |
438 | ||
439 | ||
440 | Empty Media Pool | |
441 | ^^^^^^^^^^^^^^^^ | |
442 | ||
443 | It is possible to label tapes with no pool assignment. Such tapes may | |
444 | be used by any pool. Once used by a pool, media stays in that pool. | |
445 | ||
446 | ||
baefbc44 DM |
447 | |
448 | Tape Jobs | |
449 | ~~~~~~~~~ | |
450 | ||
451 | ||
452 | Administration | |
453 | -------------- | |
454 | ||
44ce25e7 DM |
455 | Many sub-command of the ``proxmox-tape`` command line tools take a |
456 | parameter called ``--drive``, which specifies the tape drive you want | |
457 | to work on. For convenience, you can set that in an environment | |
458 | variable:: | |
459 | ||
460 | # export PROXMOX_TAPE_DRIVE=mydrive | |
461 | ||
462 | You can then omit the ``--drive`` parameter from the command. If the | |
463 | drive has an associated changer device, you may also omit the changer | |
464 | parameter from commands that needs a changer device, for example:: | |
465 | ||
466 | # proxmox-tape changer status | |
467 | ||
468 | Should displays the changer status of the changer device associated with | |
469 | drive ``mydrive``. | |
470 | ||
471 | ||
baefbc44 DM |
472 | Label Tapes |
473 | ~~~~~~~~~~~ | |
474 | ||
8001c82e DM |
475 | By default, tape cartidges all looks the same, so you need to put a |
476 | label on them for unique identification. So first, put a sticky paper | |
477 | label with some human readable text on the cartridge. | |
478 | ||
479 | If you use a `Tape Library`_, you should use an 8 character string | |
480 | encoded as `Code 39`_, as definded in the `LTO Ultrium Cartridge Label | |
481 | Specification`_. You can either bye such barcode labels from your | |
482 | cartidge vendor, or print them yourself. You can use our `LTO Barcode | |
483 | Generator`_ App for that. | |
484 | ||
485 | Next, you need to write that same label text to the tape, so that the | |
486 | software can uniquely identify the tape too. | |
487 | ||
488 | For a standalone drive, manually insert the new tape cartidge into the | |
44ce25e7 | 489 | drive and run:: |
8001c82e DM |
490 | |
491 | # proxmox-tape label --changer-id <label-text> --drive <drive-name> | |
492 | ||
493 | .. Note:: For safety reasons, this command fails if the tape contain | |
494 | any data. If you want to overwrite it anways, erase the tape first. | |
495 | ||
44ce25e7 | 496 | You can verify success by reading back the label:: |
8001c82e DM |
497 | |
498 | # proxmox-tape read-label --drive <drive-name> | |
499 | ||
500 | If you have a tape library, apply the sticky barcode label to the tape | |
501 | cartridges first. Then load those empty tapes into the library. You | |
44ce25e7 | 502 | can then label all unlabeled tapes with a single command:: |
8001c82e DM |
503 | |
504 | # proxmox-tape barcode-label --drive <drive-name> | |
505 | ||
506 | ||
baefbc44 DM |
507 | Run Tape Backups |
508 | ~~~~~~~~~~~~~~~~ | |
509 | ||
510 | Restore from Tape | |
511 | ~~~~~~~~~~~~~~~~~ | |
512 | ||
513 | Update Inventory | |
514 | ~~~~~~~~~~~~~~~~ | |
515 | ||
516 | Restore Catalog | |
517 | ~~~~~~~~~~~~~~~ |