docs/storage.rst

   1 Backup Storage
   2 ==============
   3
   4 .. _storage_disk_management:
   5
   6 Disk Management
   7 ---------------
   8
   9 .. image:: images/screenshots/pbs-gui-disks.png
  10   :align: right
  11   :alt: List of disks
  12
  13 Proxmox Backup Server comes with a set of disk utilities, which are
  14 accessed using the ``disk`` subcommand. This subcommand allows you to initialize
  15 disks, create various filesystems, and get information about the disks.
  16
  17 To view the disks connected to the system, navigate to **Administration ->
  18 Storage/Disks** in the web interface or use the ``list`` subcommand of
  19 ``disk``:
  20
  21 .. code-block:: console
  22
  23   # proxmox-backup-manager disk list
  24   ┌──────┬────────┬─────┬───────────┬─────────────┬───────────────┬─────────┬────────┐
  25   │ name │ used   │ gpt │ disk-type │        size │ model         │ wearout │ status │
  26   ╞══════╪════════╪═════╪═══════════╪═════════════╪═══════════════╪═════════╪════════╡
  27   │ sda  │ lvm    │   1 │ hdd       │ 34359738368 │ QEMU_HARDDISK │       - │ passed │
  28   ├──────┼────────┼─────┼───────────┼─────────────┼───────────────┼─────────┼────────┤
  29   │ sdb  │ unused │   1 │ hdd       │ 68719476736 │ QEMU_HARDDISK │       - │ passed │
  30   ├──────┼────────┼─────┼───────────┼─────────────┼───────────────┼─────────┼────────┤
  31   │ sdc  │ unused │   1 │ hdd       │ 68719476736 │ QEMU_HARDDISK │       - │ passed │
  32   └──────┴────────┴─────┴───────────┴─────────────┴───────────────┴─────────┴────────┘
  33
  34 To initialize a disk with a new GPT, use the ``initialize`` subcommand:
  35
  36 .. code-block:: console
  37
  38   # proxmox-backup-manager disk initialize sdX
  39
  40 .. image:: images/screenshots/pbs-gui-disks-dir-create.png
  41   :align: right
  42   :alt: Create a directory
  43
  44 You can create an ``ext4`` or ``xfs`` filesystem on a disk using ``fs
  45 create``, or by navigating to **Administration -> Storage/Disks -> Directory**
  46 in the web interface and creating one from there. The following command creates
  47 an ``ext4`` filesystem and passes the ``--add-datastore`` parameter, in order to
  48 automatically create a datastore on the disk (in this case ``sdd``). This will
  49 create a datastore at the location ``/mnt/datastore/store1``:
  50
  51 .. code-block:: console
  52
  53   # proxmox-backup-manager disk fs create store1 --disk sdd --filesystem ext4 --add-datastore true
  54
  55 .. image:: images/screenshots/pbs-gui-disks-zfs-create.png
  56   :align: right
  57   :alt: Create ZFS
  58
  59 You can also create a ``zpool`` with various raid levels from **Administration
  60 -> Storage/Disks -> ZFS** in the web interface, or by using ``zpool create``. The command
  61 below creates a mirrored ``zpool`` using two disks (``sdb`` & ``sdc``) and
  62 mounts it under ``/mnt/datastore/zpool1``:
  63
  64 .. code-block:: console
  65
  66   # proxmox-backup-manager disk zpool create zpool1 --devices sdb,sdc --raidlevel mirror
  67
  68 .. note:: You can also pass the ``--add-datastore`` parameter here, to automatically
  69   create a datastore from the disk.
  70
  71 You can use ``disk fs list`` and ``disk zpool list`` to keep track of your
  72 filesystems and zpools respectively.
  73
  74 Proxmox Backup Server uses the package smartmontools. This is a set of tools
  75 used to monitor and control the S.M.A.R.T. system for local hard disks. If a
  76 disk supports S.M.A.R.T. capability, and you have this enabled, you can
  77 display S.M.A.R.T. attributes from the web interface or by using the command:
  78
  79 .. code-block:: console
  80
  81   # proxmox-backup-manager disk smart-attributes sdX
  82
  83 .. note:: This functionality may also be accessed directly through the use of
  84   the ``smartctl`` command, which comes as part of the smartmontools package
  85   (see ``man smartctl`` for more details).
  86
  87
  88 .. _datastore_intro:
  89
  90 :term:`Datastore`
  91 -----------------
  92
  93 A datastore refers to a location at which backups are stored. The current
  94 implementation uses a directory inside a standard Unix file system (``ext4``,
  95 ``xfs`` or ``zfs``) to store the backup data.
  96
  97 Datastores are identified by a simple *ID*. You can configure this
  98 when setting up the datastore. The configuration information for datastores
  99 is stored in the file ``/etc/proxmox-backup/datastore.cfg``.
 100
 101 .. note:: The `File Layout`_ requires the file system to support at least *65538*
 102    subdirectories per directory. That number comes from the 2\ :sup:`16`
 103    pre-created chunk namespace directories, and the ``.`` and ``..`` default
 104    directory entries. This requirement excludes certain filesystems and
 105    filesystem configurations from being supported for a datastore. For example,
 106    ``ext3`` as a whole or ``ext4`` with the ``dir_nlink`` feature manually disabled.
 107
 108
 109 Datastore Configuration
 110 ~~~~~~~~~~~~~~~~~~~~~~~
 111
 112 .. image:: images/screenshots/pbs-gui-datastore-content.png
 113   :align: right
 114   :alt: Datastore Overview
 115
 116 You can configure multiple datastores. A minimum of one datastore needs to be
 117 configured. The datastore is identified by a simple *name* and points to a
 118 directory on the filesystem. Each datastore also has associated retention
 119 settings of how many backup snapshots for each interval of ``hourly``,
 120 ``daily``, ``weekly``, ``monthly``, ``yearly`` as well as a time-independent
 121 number of backups to keep in that store. :ref:`backup-pruning` and
 122 :ref:`garbage collection <client_garbage-collection>` can also be configured to
 123 run periodically, based on a configured schedule (see
 124 :ref:`calendar-event-scheduling`) per datastore.
 125
 126
 127 .. _storage_datastore_create:
 128
 129 Creating a Datastore
 130 ^^^^^^^^^^^^^^^^^^^^
 131 .. image:: images/screenshots/pbs-gui-datastore-create-general.png
 132   :align: right
 133   :alt: Create a datastore
 134
 135 You can create a new datastore from the web interface, by clicking **Add
 136 Datastore** in the side menu, under the **Datastore** section. In the setup
 137 window:
 138
 139 * *Name* refers to the name of the datastore
 140 * *Backing Path* is the path to the directory upon which you want to create the
 141   datastore
 142 * *GC Schedule* refers to the time and intervals at which garbage collection
 143   runs
 144 * *Prune Schedule* refers to the frequency at which pruning takes place
 145 * *Prune Options* set the amount of backups which you would like to keep (see
 146   :ref:`backup-pruning`).
 147 * *Comment* can be used to add some contextual information to the datastore.
 148
 149 Alternatively you can create a new datastore from the command line. The
 150 following command creates a new datastore called ``store1`` on
 151 :file:`/backup/disk1/store1`
 152
 153 .. code-block:: console
 154
 155   # proxmox-backup-manager datastore create store1 /backup/disk1/store1
 156
 157
 158 Managing Datastores
 159 ^^^^^^^^^^^^^^^^^^^
 160
 161 To list existing datastores from the command line, run:
 162
 163 .. code-block:: console
 164
 165   # proxmox-backup-manager datastore list
 166   ┌────────┬──────────────────────┬─────────────────────────────┐
 167   │ name   │ path                 │ comment                     │
 168   ╞════════╪══════════════════════╪═════════════════════════════╡
 169   │ store1 │ /backup/disk1/store1 │ This is my default storage. │
 170   └────────┴──────────────────────┴─────────────────────────────┘
 171
 172 You can change the garbage collection and prune settings of a datastore, by
 173 editing the datastore from the GUI or by using the ``update`` subcommand. For
 174 example, the below command changes the garbage collection schedule using the
 175 ``update`` subcommand and prints the properties of the datastore with the
 176 ``show`` subcommand:
 177
 178 .. code-block:: console
 179
 180   # proxmox-backup-manager datastore update store1 --gc-schedule 'Tue 04:27'
 181   # proxmox-backup-manager datastore show store1
 182   ┌────────────────┬─────────────────────────────┐
 183   │ Name           │ Value                       │
 184   ╞════════════════╪═════════════════════════════╡
 185   │ name           │ store1                      │
 186   ├────────────────┼─────────────────────────────┤
 187   │ path           │ /backup/disk1/store1        │
 188   ├────────────────┼─────────────────────────────┤
 189   │ comment        │ This is my default storage. │
 190   ├────────────────┼─────────────────────────────┤
 191   │ gc-schedule    │ Tue 04:27                   │
 192   ├────────────────┼─────────────────────────────┤
 193   │ keep-last      │ 7                           │
 194   ├────────────────┼─────────────────────────────┤
 195   │ prune-schedule │ daily                       │
 196   └────────────────┴─────────────────────────────┘
 197
 198 Finally, it is possible to remove the datastore configuration:
 199
 200 .. code-block:: console
 201
 202   # proxmox-backup-manager datastore remove store1
 203
 204 .. note:: The above command removes only the datastore configuration. It does
 205    not delete any data from the underlying directory.
 206
 207
 208 File Layout
 209 ^^^^^^^^^^^
 210
 211 After creating a datastore, the following default layout will appear:
 212
 213 .. code-block:: console
 214
 215   # ls -arilh /backup/disk1/store1
 216   276493 -rw-r--r-- 1 backup backup       0 Jul  8 12:35 .lock
 217   276490 drwxr-x--- 1 backup backup 1064960 Jul  8 12:35 .chunks
 218
 219 `.lock` is an empty file used for process locking.
 220
 221 The `.chunks` directory contains folders, starting from `0000` and increasing in
 222 hexadecimal values until `ffff`. These directories will store the chunked data,
 223 categorized by checksum, after a backup operation has been executed.
 224
 225 .. code-block:: console
 226
 227  # ls -arilh /backup/disk1/store1/.chunks
 228  545824 drwxr-x--- 2 backup backup 4.0K Jul  8 12:35 ffff
 229  545823 drwxr-x--- 2 backup backup 4.0K Jul  8 12:35 fffe
 230  415621 drwxr-x--- 2 backup backup 4.0K Jul  8 12:35 fffd
 231  415620 drwxr-x--- 2 backup backup 4.0K Jul  8 12:35 fffc
 232  353187 drwxr-x--- 2 backup backup 4.0K Jul  8 12:35 fffb
 233  344995 drwxr-x--- 2 backup backup 4.0K Jul  8 12:35 fffa
 234  144079 drwxr-x--- 2 backup backup 4.0K Jul  8 12:35 fff9
 235  144078 drwxr-x--- 2 backup backup 4.0K Jul  8 12:35 fff8
 236  144077 drwxr-x--- 2 backup backup 4.0K Jul  8 12:35 fff7
 237  ...
 238  403180 drwxr-x--- 2 backup backup 4.0K Jul  8 12:35 000c
 239  403179 drwxr-x--- 2 backup backup 4.0K Jul  8 12:35 000b
 240  403177 drwxr-x--- 2 backup backup 4.0K Jul  8 12:35 000a
 241  402530 drwxr-x--- 2 backup backup 4.0K Jul  8 12:35 0009
 242  402513 drwxr-x--- 2 backup backup 4.0K Jul  8 12:35 0008
 243  402509 drwxr-x--- 2 backup backup 4.0K Jul  8 12:35 0007
 244  276509 drwxr-x--- 2 backup backup 4.0K Jul  8 12:35 0006
 245  276508 drwxr-x--- 2 backup backup 4.0K Jul  8 12:35 0005
 246  276507 drwxr-x--- 2 backup backup 4.0K Jul  8 12:35 0004
 247  276501 drwxr-x--- 2 backup backup 4.0K Jul  8 12:35 0003
 248  276499 drwxr-x--- 2 backup backup 4.0K Jul  8 12:35 0002
 249  276498 drwxr-x--- 2 backup backup 4.0K Jul  8 12:35 0001
 250  276494 drwxr-x--- 2 backup backup 4.0K Jul  8 12:35 0000
 251  276489 drwxr-xr-x 3 backup backup 4.0K Jul  8 12:35 ..
 252  276490 drwxr-x--- 1 backup backup 1.1M Jul  8 12:35 .
 253
 254