[proxmox-backup.git] / docs / storage.rst

Backup Storage
==============

.. _storage_disk_management:

Disk Management
---------------

.. image:: images/screenshots/pbs-gui-disks.png
  :align: right
  :alt: List of disks

Proxmox Backup Server comes with a set of disk utilities, which are
accessed using the ``disk`` subcommand. This subcommand allows you to initialize
disks, create various filesystems, and get information about the disks.

To view the disks connected to the system, navigate to **Administration ->
Storage/Disks** in the web interface or use the ``list`` subcommand of
``disk``:

.. code-block:: console

  # proxmox-backup-manager disk list
  ┌──────┬────────┬─────┬───────────┬─────────────┬───────────────┬─────────┬────────┐
  │ name │ used   │ gpt │ disk-type │        size │ model         │ wearout │ status │
  ╞══════╪════════╪═════╪═══════════╪═════════════╪═══════════════╪═════════╪════════╡
  │ sda  │ lvm    │   1 │ hdd       │ 34359738368 │ QEMU_HARDDISK │       - │ passed │
  ├──────┼────────┼─────┼───────────┼─────────────┼───────────────┼─────────┼────────┤
  │ sdb  │ unused │   1 │ hdd       │ 68719476736 │ QEMU_HARDDISK │       - │ passed │
  ├──────┼────────┼─────┼───────────┼─────────────┼───────────────┼─────────┼────────┤
  │ sdc  │ unused │   1 │ hdd       │ 68719476736 │ QEMU_HARDDISK │       - │ passed │
  └──────┴────────┴─────┴───────────┴─────────────┴───────────────┴─────────┴────────┘

To initialize a disk with a new GPT, use the ``initialize`` subcommand:

.. code-block:: console

  # proxmox-backup-manager disk initialize sdX

.. image:: images/screenshots/pbs-gui-disks-dir-create.png
  :align: right
  :alt: Create a directory

You can create an ``ext4`` or ``xfs`` filesystem on a disk using ``fs
create``, or by navigating to **Administration -> Storage/Disks -> Directory**
in the web interface and creating one from there. The following command creates
an ``ext4`` filesystem and passes the ``--add-datastore`` parameter, in order to
automatically create a datastore on the disk (in this case ``sdd``). This will
create a datastore at the location ``/mnt/datastore/store1``:

.. code-block:: console

  # proxmox-backup-manager disk fs create store1 --disk sdd --filesystem ext4 --add-datastore true

.. image:: images/screenshots/pbs-gui-disks-zfs-create.png
  :align: right
  :alt: Create ZFS

You can also create a ``zpool`` with various raid levels from **Administration
-> Storage/Disks -> ZFS** in the web interface, or by using ``zpool create``. The command
below creates a mirrored ``zpool`` using two disks (``sdb`` & ``sdc``) and
mounts it under ``/mnt/datastore/zpool1``:

.. code-block:: console

  # proxmox-backup-manager disk zpool create zpool1 --devices sdb,sdc --raidlevel mirror

.. note:: You can also pass the ``--add-datastore`` parameter here, to automatically
  create a datastore from the disk.

You can use ``disk fs list`` and ``disk zpool list`` to keep track of your
filesystems and zpools respectively.

Proxmox Backup Server uses the package smartmontools. This is a set of tools
used to monitor and control the S.M.A.R.T. system for local hard disks. If a
disk supports S.M.A.R.T. capability, and you have this enabled, you can
display S.M.A.R.T. attributes from the web interface or by using the command:

.. code-block:: console

  # proxmox-backup-manager disk smart-attributes sdX

.. note:: This functionality may also be accessed directly through the use of
  the ``smartctl`` command, which comes as part of the smartmontools package
  (see ``man smartctl`` for more details).


.. _datastore_intro:

:term:`Datastore`
-----------------

A datastore refers to a location at which backups are stored. The current
implementation uses a directory inside a standard Unix file system (``ext4``,
``xfs`` or ``zfs``) to store the backup data.

Datastores are identified by a simple *ID*. You can configure this
when setting up the datastore. The configuration information for datastores
is stored in the file ``/etc/proxmox-backup/datastore.cfg``.

.. note:: The `File Layout`_ requires the file system to support at least *65538*
   subdirectories per directory. That number comes from the 2\ :sup:`16`
   pre-created chunk namespace directories, and the ``.`` and ``..`` default
   directory entries. This requirement excludes certain filesystems and
   filesystem configurations from being supported for a datastore. For example,
   ``ext3`` as a whole or ``ext4`` with the ``dir_nlink`` feature manually disabled.


Datastore Configuration
~~~~~~~~~~~~~~~~~~~~~~~

.. image:: images/screenshots/pbs-gui-datastore-content.png
  :align: right
  :alt: Datastore Overview

You can configure multiple datastores. A minimum of one datastore needs to be
configured. The datastore is identified by a simple *name* and points to a
directory on the filesystem. Each datastore also has associated retention
settings of how many backup snapshots for each interval of ``hourly``,
``daily``, ``weekly``, ``monthly``, ``yearly`` as well as a time-independent
number of backups to keep in that store. :ref:`backup-pruning` and
:ref:`garbage collection <client_garbage-collection>` can also be configured to
run periodically, based on a configured schedule (see
:ref:`calendar-event-scheduling`) per datastore.


.. _storage_datastore_create:

Creating a Datastore
^^^^^^^^^^^^^^^^^^^^
.. image:: images/screenshots/pbs-gui-datastore-create-general.png
  :align: right
  :alt: Create a datastore

You can create a new datastore from the web interface, by clicking **Add
Datastore** in the side menu, under the **Datastore** section. In the setup
window:

* *Name* refers to the name of the datastore
* *Backing Path* is the path to the directory upon which you want to create the
  datastore
* *GC Schedule* refers to the time and intervals at which garbage collection
  runs
* *Prune Schedule* refers to the frequency at which pruning takes place
* *Prune Options* set the amount of backups which you would like to keep (see
  :ref:`backup-pruning`).
* *Comment* can be used to add some contextual information to the datastore.

Alternatively you can create a new datastore from the command line. The
following command creates a new datastore called ``store1`` on
:file:`/backup/disk1/store1`

.. code-block:: console

  # proxmox-backup-manager datastore create store1 /backup/disk1/store1


Managing Datastores
^^^^^^^^^^^^^^^^^^^

To list existing datastores from the command line, run:

.. code-block:: console

  # proxmox-backup-manager datastore list
  ┌────────┬──────────────────────┬─────────────────────────────┐
  │ name   │ path                 │ comment                     │
  ╞════════╪══════════════════════╪═════════════════════════════╡
  │ store1 │ /backup/disk1/store1 │ This is my default storage. │
  └────────┴──────────────────────┴─────────────────────────────┘

You can change the garbage collection and prune settings of a datastore, by
editing the datastore from the GUI or by using the ``update`` subcommand. For
example, the below command changes the garbage collection schedule using the
``update`` subcommand and prints the properties of the datastore with the
``show`` subcommand:

.. code-block:: console

  # proxmox-backup-manager datastore update store1 --gc-schedule 'Tue 04:27'
  # proxmox-backup-manager datastore show store1
  ┌────────────────┬─────────────────────────────┐
  │ Name           │ Value                       │
  ╞════════════════╪═════════════════════════════╡
  │ name           │ store1                      │
  ├────────────────┼─────────────────────────────┤
  │ path           │ /backup/disk1/store1        │
  ├────────────────┼─────────────────────────────┤
  │ comment        │ This is my default storage. │
  ├────────────────┼─────────────────────────────┤
  │ gc-schedule    │ Tue 04:27                   │
  ├────────────────┼─────────────────────────────┤
  │ keep-last      │ 7                           │
  ├────────────────┼─────────────────────────────┤
  │ prune-schedule │ daily                       │
  └────────────────┴─────────────────────────────┘

Finally, it is possible to remove the datastore configuration:

.. code-block:: console

  # proxmox-backup-manager datastore remove store1

.. note:: The above command removes only the datastore configuration. It does
   not delete any data from the underlying directory.


File Layout
^^^^^^^^^^^

After creating a datastore, the following default layout will appear:

.. code-block:: console

  # ls -arilh /backup/disk1/store1
  276493 -rw-r--r-- 1 backup backup       0 Jul  8 12:35 .lock
  276490 drwxr-x--- 1 backup backup 1064960 Jul  8 12:35 .chunks

`.lock` is an empty file used for process locking.

The `.chunks` directory contains folders, starting from `0000` and increasing in
hexadecimal values until `ffff`. These directories will store the chunked data,
categorized by checksum, after a backup operation has been executed.

.. code-block:: console

 # ls -arilh /backup/disk1/store1/.chunks
 545824 drwxr-x--- 2 backup backup 4.0K Jul  8 12:35 ffff
 545823 drwxr-x--- 2 backup backup 4.0K Jul  8 12:35 fffe
 415621 drwxr-x--- 2 backup backup 4.0K Jul  8 12:35 fffd
 415620 drwxr-x--- 2 backup backup 4.0K Jul  8 12:35 fffc
 353187 drwxr-x--- 2 backup backup 4.0K Jul  8 12:35 fffb
 344995 drwxr-x--- 2 backup backup 4.0K Jul  8 12:35 fffa
 144079 drwxr-x--- 2 backup backup 4.0K Jul  8 12:35 fff9
 144078 drwxr-x--- 2 backup backup 4.0K Jul  8 12:35 fff8
 144077 drwxr-x--- 2 backup backup 4.0K Jul  8 12:35 fff7
 ...
 403180 drwxr-x--- 2 backup backup 4.0K Jul  8 12:35 000c
 403179 drwxr-x--- 2 backup backup 4.0K Jul  8 12:35 000b
 403177 drwxr-x--- 2 backup backup 4.0K Jul  8 12:35 000a
 402530 drwxr-x--- 2 backup backup 4.0K Jul  8 12:35 0009
 402513 drwxr-x--- 2 backup backup 4.0K Jul  8 12:35 0008
 402509 drwxr-x--- 2 backup backup 4.0K Jul  8 12:35 0007
 276509 drwxr-x--- 2 backup backup 4.0K Jul  8 12:35 0006
 276508 drwxr-x--- 2 backup backup 4.0K Jul  8 12:35 0005
 276507 drwxr-x--- 2 backup backup 4.0K Jul  8 12:35 0004
 276501 drwxr-x--- 2 backup backup 4.0K Jul  8 12:35 0003
 276499 drwxr-x--- 2 backup backup 4.0K Jul  8 12:35 0002
 276498 drwxr-x--- 2 backup backup 4.0K Jul  8 12:35 0001
 276494 drwxr-x--- 2 backup backup 4.0K Jul  8 12:35 0000
 276489 drwxr-xr-x 3 backup backup 4.0K Jul  8 12:35 ..
 276490 drwxr-x--- 1 backup backup 1.1M Jul  8 12:35 .
Commit	Line	Data
55334cf4 TL	1	Backup Storage
55334cf4 TL	2	==============
04e24b14	3
ee0ab12d TL	4	.. _storage_disk_management:
ee0ab12d TL	5
04e24b14 DW	6	Disk Management
	7	---------------
	8
	9	.. image:: images/screenshots/pbs-gui-disks.png
	10	:align: right
	11	:alt: List of disks
	12
	13	Proxmox Backup Server comes with a set of disk utilities, which are
	14	accessed using the ``disk`` subcommand. This subcommand allows you to initialize
	15	disks, create various filesystems, and get information about the disks.
	16
	17	To view the disks connected to the system, navigate to **Administration ->
60589e60	18	Storage/Disks** in the web interface or use the ``list`` subcommand of
04e24b14 DW	19	``disk``:
	20
	21	.. code-block:: console
	22
	23	# proxmox-backup-manager disk list
	24	┌──────┬────────┬─────┬───────────┬─────────────┬───────────────┬─────────┬────────┐
	25	│ name │ used │ gpt │ disk-type │ size │ model │ wearout │ status │
	26	╞══════╪════════╪═════╪═══════════╪═════════════╪═══════════════╪═════════╪════════╡
	27	│ sda │ lvm │ 1 │ hdd │ 34359738368 │ QEMU_HARDDISK │ - │ passed │
	28	├──────┼────────┼─────┼───────────┼─────────────┼───────────────┼─────────┼────────┤
	29	│ sdb │ unused │ 1 │ hdd │ 68719476736 │ QEMU_HARDDISK │ - │ passed │
	30	├──────┼────────┼─────┼───────────┼─────────────┼───────────────┼─────────┼────────┤
	31	│ sdc │ unused │ 1 │ hdd │ 68719476736 │ QEMU_HARDDISK │ - │ passed │
	32	└──────┴────────┴─────┴───────────┴─────────────┴───────────────┴─────────┴────────┘
	33
	34	To initialize a disk with a new GPT, use the ``initialize`` subcommand:
	35
	36	.. code-block:: console
	37
	38	# proxmox-backup-manager disk initialize sdX
	39
	40	.. image:: images/screenshots/pbs-gui-disks-dir-create.png
	41	:align: right
	42	:alt: Create a directory
	43
	44	You can create an ``ext4`` or ``xfs`` filesystem on a disk using ``fs
60589e60 DW	45	create``, or by navigating to Administration -> Storage/Disks -> Directory
	46	in the web interface and creating one from there. The following command creates
	47	an ``ext4`` filesystem and passes the ``--add-datastore`` parameter, in order to
04e24b14 DW	48	automatically create a datastore on the disk (in this case ``sdd``). This will
	49	create a datastore at the location ``/mnt/datastore/store1``:
	50
	51	.. code-block:: console
	52
	53	# proxmox-backup-manager disk fs create store1 --disk sdd --filesystem ext4 --add-datastore true
	54
	55	.. image:: images/screenshots/pbs-gui-disks-zfs-create.png
	56	:align: right
	57	:alt: Create ZFS
	58
	59	You can also create a ``zpool`` with various raid levels from **Administration
60589e60	60	-> Storage/Disks -> ZFS** in the web interface, or by using ``zpool create``. The command
04e24b14	61	below creates a mirrored ``zpool`` using two disks (``sdb`` & ``sdc``) and
f5d9f253	62	mounts it under ``/mnt/datastore/zpool1``:
04e24b14 DW	63
	64	.. code-block:: console
	65
	66	# proxmox-backup-manager disk zpool create zpool1 --devices sdb,sdc --raidlevel mirror
	67
	68	.. note:: You can also pass the ``--add-datastore`` parameter here, to automatically
	69	create a datastore from the disk.
	70
	71	You can use ``disk fs list`` and ``disk zpool list`` to keep track of your
	72	filesystems and zpools respectively.
	73
	74	Proxmox Backup Server uses the package smartmontools. This is a set of tools
	75	used to monitor and control the S.M.A.R.T. system for local hard disks. If a
	76	disk supports S.M.A.R.T. capability, and you have this enabled, you can
	77	display S.M.A.R.T. attributes from the web interface or by using the command:
	78
	79	.. code-block:: console
	80
	81	# proxmox-backup-manager disk smart-attributes sdX
	82
	83	.. note:: This functionality may also be accessed directly through the use of
	84	the ``smartctl`` command, which comes as part of the smartmontools package
	85	(see ``man smartctl`` for more details).
	86
	87
	88	.. _datastore_intro:
	89
030464d3	90	:term:`Datastore`
04e24b14 DW	91	-----------------
	92
	93	A datastore refers to a location at which backups are stored. The current
3bbb70b3	94	implementation uses a directory inside a standard Unix file system (``ext4``,
04e24b14 DW	95	``xfs`` or ``zfs``) to store the backup data.
	96
	97	Datastores are identified by a simple ID. You can configure this
	98	when setting up the datastore. The configuration information for datastores
	99	is stored in the file ``/etc/proxmox-backup/datastore.cfg``.
	100
	101	.. note:: The `File Layout`_ requires the file system to support at least 65538
	102	subdirectories per directory. That number comes from the 2\ :sup:`16`
	103	pre-created chunk namespace directories, and the ``.`` and ``..`` default
	104	directory entries. This requirement excludes certain filesystems and
717ce406	105	filesystem configurations from being supported for a datastore. For example,
04e24b14 DW	106	``ext3`` as a whole or ``ext4`` with the ``dir_nlink`` feature manually disabled.
	107
	108
	109	Datastore Configuration
	110	~~~~~~~~~~~~~~~~~~~~~~~
	111
f608e74c	112	.. image:: images/screenshots/pbs-gui-datastore-content.png
04e24b14 DW	113	:align: right
	114	:alt: Datastore Overview
	115
717ce406	116	You can configure multiple datastores. A minimum of one datastore needs to be
04e24b14 DW	117	configured. The datastore is identified by a simple name and points to a
	118	directory on the filesystem. Each datastore also has associated retention
	119	settings of how many backup snapshots for each interval of ``hourly``,
	120	``daily``, ``weekly``, ``monthly``, ``yearly`` as well as a time-independent
4c3efb53	121	number of backups to keep in that store. :ref:`backup-pruning` and
717ce406 DW	122	:ref:`garbage collection <client_garbage-collection>` can also be configured to
	123	run periodically, based on a configured schedule (see
	124	:ref:`calendar-event-scheduling`) per datastore.
04e24b14 DW	125
04e24b14 DW	126
ee0ab12d TL	127	.. _storage_datastore_create:
ee0ab12d TL	128
04e24b14 DW	129	Creating a Datastore
	130	^^^^^^^^^^^^^^^^^^^^
	131	.. image:: images/screenshots/pbs-gui-datastore-create-general.png
	132	:align: right
	133	:alt: Create a datastore
	134
f608e74c DW	135	You can create a new datastore from the web interface, by clicking **Add
	136	Datastore in the side menu, under the Datastore** section. In the setup
	137	window:
04e24b14 DW	138
	139	* Name refers to the name of the datastore
	140	* Backing Path is the path to the directory upon which you want to create the
	141	datastore
	142	* GC Schedule refers to the time and intervals at which garbage collection
	143	runs
	144	* Prune Schedule refers to the frequency at which pruning takes place
f608e74c DW	145	* Prune Options set the amount of backups which you would like to keep (see
	146	:ref:`backup-pruning`).
	147	* Comment can be used to add some contextual information to the datastore.
04e24b14 DW	148
04e24b14 DW	149	Alternatively you can create a new datastore from the command line. The
717ce406 DW	150	following command creates a new datastore called ``store1`` on
717ce406 DW	151	:file:`/backup/disk1/store1`
04e24b14 DW	152
	153	.. code-block:: console
	154
	155	# proxmox-backup-manager datastore create store1 /backup/disk1/store1
	156
	157
	158	Managing Datastores
	159	^^^^^^^^^^^^^^^^^^^
	160
717ce406	161	To list existing datastores from the command line, run:
04e24b14 DW	162
	163	.. code-block:: console
	164
	165	# proxmox-backup-manager datastore list
	166	┌────────┬──────────────────────┬─────────────────────────────┐
	167	│ name │ path │ comment │
	168	╞════════╪══════════════════════╪═════════════════════════════╡
	169	│ store1 │ /backup/disk1/store1 │ This is my default storage. │
	170	└────────┴──────────────────────┴─────────────────────────────┘
	171
	172	You can change the garbage collection and prune settings of a datastore, by
	173	editing the datastore from the GUI or by using the ``update`` subcommand. For
	174	example, the below command changes the garbage collection schedule using the
	175	``update`` subcommand and prints the properties of the datastore with the
	176	``show`` subcommand:
	177
	178	.. code-block:: console
	179
	180	# proxmox-backup-manager datastore update store1 --gc-schedule 'Tue 04:27'
	181	# proxmox-backup-manager datastore show store1
	182	┌────────────────┬─────────────────────────────┐
	183	│ Name │ Value │
	184	╞════════════════╪═════════════════════════════╡
	185	│ name │ store1 │
	186	├────────────────┼─────────────────────────────┤
	187	│ path │ /backup/disk1/store1 │
	188	├────────────────┼─────────────────────────────┤
	189	│ comment │ This is my default storage. │
	190	├────────────────┼─────────────────────────────┤
	191	│ gc-schedule │ Tue 04:27 │
	192	├────────────────┼─────────────────────────────┤
	193	│ keep-last │ 7 │
	194	├────────────────┼─────────────────────────────┤
	195	│ prune-schedule │ daily │
	196	└────────────────┴─────────────────────────────┘
	197
	198	Finally, it is possible to remove the datastore configuration:
	199
	200	.. code-block:: console
	201
	202	# proxmox-backup-manager datastore remove store1
	203
	204	.. note:: The above command removes only the datastore configuration. It does
	205	not delete any data from the underlying directory.
	206
	207
	208	File Layout
	209	^^^^^^^^^^^
	210
	211	After creating a datastore, the following default layout will appear:
	212
	213	.. code-block:: console
	214
	215	# ls -arilh /backup/disk1/store1
	216	276493 -rw-r--r-- 1 backup backup 0 Jul 8 12:35 .lock
	217	276490 drwxr-x--- 1 backup backup 1064960 Jul 8 12:35 .chunks
	218
	219	`.lock` is an empty file used for process locking.
	220
717ce406 DW	221	The `.chunks` directory contains folders, starting from `0000` and increasing in
	222	hexadecimal values until `ffff`. These directories will store the chunked data,
	223	categorized by checksum, after a backup operation has been executed.
04e24b14 DW	224
	225	.. code-block:: console
	226
	227	# ls -arilh /backup/disk1/store1/.chunks
	228	545824 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 ffff
	229	545823 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 fffe
	230	415621 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 fffd
	231	415620 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 fffc
	232	353187 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 fffb
	233	344995 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 fffa
	234	144079 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 fff9
	235	144078 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 fff8
	236	144077 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 fff7
	237	...
	238	403180 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 000c
	239	403179 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 000b
	240	403177 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 000a
	241	402530 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 0009
	242	402513 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 0008
	243	402509 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 0007
	244	276509 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 0006
	245	276508 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 0005
	246	276507 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 0004
	247	276501 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 0003
	248	276499 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 0002
	249	276498 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 0001
	250	276494 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 0000
	251	276489 drwxr-xr-x 3 backup backup 4.0K Jul 8 12:35 ..
	252	276490 drwxr-x--- 1 backup backup 1.1M Jul 8 12:35 .
	253
	254