docs/storage.rst

   1 Storage
   2 =======
   3
   4 Disk Management
   5 ---------------
   6
   7 .. image:: images/screenshots/pbs-gui-disks.png
   8   :align: right
   9   :alt: List of disks
  10
  11 Proxmox Backup Server comes with a set of disk utilities, which are
  12 accessed using the ``disk`` subcommand. This subcommand allows you to initialize
  13 disks, create various filesystems, and get information about the disks.
  14
  15 To view the disks connected to the system, navigate to **Administration ->
  16 Disks** in the web interface or use the ``list`` subcommand of
  17 ``disk``:
  18
  19 .. code-block:: console
  20
  21   # proxmox-backup-manager disk list
  22   ┌──────┬────────┬─────┬───────────┬─────────────┬───────────────┬─────────┬────────┐
  23   │ name │ used   │ gpt │ disk-type │        size │ model         │ wearout │ status │
  24   ╞══════╪════════╪═════╪═══════════╪═════════════╪═══════════════╪═════════╪════════╡
  25   │ sda  │ lvm    │   1 │ hdd       │ 34359738368 │ QEMU_HARDDISK │       - │ passed │
  26   ├──────┼────────┼─────┼───────────┼─────────────┼───────────────┼─────────┼────────┤
  27   │ sdb  │ unused │   1 │ hdd       │ 68719476736 │ QEMU_HARDDISK │       - │ passed │
  28   ├──────┼────────┼─────┼───────────┼─────────────┼───────────────┼─────────┼────────┤
  29   │ sdc  │ unused │   1 │ hdd       │ 68719476736 │ QEMU_HARDDISK │       - │ passed │
  30   └──────┴────────┴─────┴───────────┴─────────────┴───────────────┴─────────┴────────┘
  31
  32 To initialize a disk with a new GPT, use the ``initialize`` subcommand:
  33
  34 .. code-block:: console
  35
  36   # proxmox-backup-manager disk initialize sdX
  37
  38 .. image:: images/screenshots/pbs-gui-disks-dir-create.png
  39   :align: right
  40   :alt: Create a directory
  41
  42 You can create an ``ext4`` or ``xfs`` filesystem on a disk using ``fs
  43 create``, or by navigating to **Administration -> Disks -> Directory** in the
  44 web interface and creating one from there. The following command creates an
  45 ``ext4`` filesystem and passes the ``--add-datastore`` parameter, in order to
  46 automatically create a datastore on the disk (in this case ``sdd``). This will
  47 create a datastore at the location ``/mnt/datastore/store1``:
  48
  49 .. code-block:: console
  50
  51   # proxmox-backup-manager disk fs create store1 --disk sdd --filesystem ext4 --add-datastore true
  52
  53 .. image:: images/screenshots/pbs-gui-disks-zfs-create.png
  54   :align: right
  55   :alt: Create ZFS
  56
  57 You can also create a ``zpool`` with various raid levels from **Administration
  58 -> Disks -> Zpool** in the web interface, or by using ``zpool create``. The command
  59 below creates a mirrored ``zpool`` using two disks (``sdb`` & ``sdc``) and
  60 mounts it on the root directory (default):
  61
  62 .. code-block:: console
  63
  64   # proxmox-backup-manager disk zpool create zpool1 --devices sdb,sdc --raidlevel mirror
  65
  66 .. note:: You can also pass the ``--add-datastore`` parameter here, to automatically
  67   create a datastore from the disk.
  68
  69 You can use ``disk fs list`` and ``disk zpool list`` to keep track of your
  70 filesystems and zpools respectively.
  71
  72 Proxmox Backup Server uses the package smartmontools. This is a set of tools
  73 used to monitor and control the S.M.A.R.T. system for local hard disks. If a
  74 disk supports S.M.A.R.T. capability, and you have this enabled, you can
  75 display S.M.A.R.T. attributes from the web interface or by using the command:
  76
  77 .. code-block:: console
  78
  79   # proxmox-backup-manager disk smart-attributes sdX
  80
  81 .. note:: This functionality may also be accessed directly through the use of
  82   the ``smartctl`` command, which comes as part of the smartmontools package
  83   (see ``man smartctl`` for more details).
  84
  85
  86 .. _datastore_intro:
  87
  88 :term:`DataStore`
  89 -----------------
  90
  91 A datastore refers to a location at which backups are stored. The current
  92 implementation uses a directory inside a standard unix file system (``ext4``,
  93 ``xfs`` or ``zfs``) to store the backup data.
  94
  95 Datastores are identified by a simple *ID*. You can configure this
  96 when setting up the datastore. The configuration information for datastores
  97 is stored in the file ``/etc/proxmox-backup/datastore.cfg``.
  98
  99 .. note:: The `File Layout`_ requires the file system to support at least *65538*
 100    subdirectories per directory. That number comes from the 2\ :sup:`16`
 101    pre-created chunk namespace directories, and the ``.`` and ``..`` default
 102    directory entries. This requirement excludes certain filesystems and
 103    filesystem configuration from being supported for a datastore. For example,
 104    ``ext3`` as a whole or ``ext4`` with the ``dir_nlink`` feature manually disabled.
 105
 106
 107 Datastore Configuration
 108 ~~~~~~~~~~~~~~~~~~~~~~~
 109
 110 .. image:: images/screenshots/pbs-gui-datastore.png
 111   :align: right
 112   :alt: Datastore Overview
 113
 114 You can configure multiple datastores. Minimum one datastore needs to be
 115 configured. The datastore is identified by a simple *name* and points to a
 116 directory on the filesystem. Each datastore also has associated retention
 117 settings of how many backup snapshots for each interval of ``hourly``,
 118 ``daily``, ``weekly``, ``monthly``, ``yearly`` as well as a time-independent
 119 number of backups to keep in that store. :ref:`backup-pruning` and
 120 :ref:`garbage collection <garbage-collection>` can also be configured to run
 121 periodically based on a configured schedule (see :ref:`calendar-events`) per datastore.
 122
 123
 124 Creating a Datastore
 125 ^^^^^^^^^^^^^^^^^^^^
 126 .. image:: images/screenshots/pbs-gui-datastore-create-general.png
 127   :align: right
 128   :alt: Create a datastore
 129
 130 You can create a new datastore from the web GUI, by navigating to **Datastore** in
 131 the menu tree and clicking **Create**. Here:
 132
 133 * *Name* refers to the name of the datastore
 134 * *Backing Path* is the path to the directory upon which you want to create the
 135   datastore
 136 * *GC Schedule* refers to the time and intervals at which garbage collection
 137   runs
 138 * *Prune Schedule* refers to the frequency at which pruning takes place
 139 * *Prune Options* set the amount of backups which you would like to keep (see :ref:`backup-pruning`).
 140
 141 Alternatively you can create a new datastore from the command line. The
 142 following command creates a new datastore called ``store1`` on :file:`/backup/disk1/store1`
 143
 144 .. code-block:: console
 145
 146   # proxmox-backup-manager datastore create store1 /backup/disk1/store1
 147
 148
 149 Managing Datastores
 150 ^^^^^^^^^^^^^^^^^^^
 151
 152 To list existing datastores from the command line run:
 153
 154 .. code-block:: console
 155
 156   # proxmox-backup-manager datastore list
 157   ┌────────┬──────────────────────┬─────────────────────────────┐
 158   │ name   │ path                 │ comment                     │
 159   ╞════════╪══════════════════════╪═════════════════════════════╡
 160   │ store1 │ /backup/disk1/store1 │ This is my default storage. │
 161   └────────┴──────────────────────┴─────────────────────────────┘
 162
 163 You can change the garbage collection and prune settings of a datastore, by
 164 editing the datastore from the GUI or by using the ``update`` subcommand. For
 165 example, the below command changes the garbage collection schedule using the
 166 ``update`` subcommand and prints the properties of the datastore with the
 167 ``show`` subcommand:
 168
 169 .. code-block:: console
 170
 171   # proxmox-backup-manager datastore update store1 --gc-schedule 'Tue 04:27'
 172   # proxmox-backup-manager datastore show store1
 173   ┌────────────────┬─────────────────────────────┐
 174   │ Name           │ Value                       │
 175   ╞════════════════╪═════════════════════════════╡
 176   │ name           │ store1                      │
 177   ├────────────────┼─────────────────────────────┤
 178   │ path           │ /backup/disk1/store1        │
 179   ├────────────────┼─────────────────────────────┤
 180   │ comment        │ This is my default storage. │
 181   ├────────────────┼─────────────────────────────┤
 182   │ gc-schedule    │ Tue 04:27                   │
 183   ├────────────────┼─────────────────────────────┤
 184   │ keep-last      │ 7                           │
 185   ├────────────────┼─────────────────────────────┤
 186   │ prune-schedule │ daily                       │
 187   └────────────────┴─────────────────────────────┘
 188
 189 Finally, it is possible to remove the datastore configuration:
 190
 191 .. code-block:: console
 192
 193   # proxmox-backup-manager datastore remove store1
 194
 195 .. note:: The above command removes only the datastore configuration. It does
 196    not delete any data from the underlying directory.
 197
 198
 199 File Layout
 200 ^^^^^^^^^^^
 201
 202 After creating a datastore, the following default layout will appear:
 203
 204 .. code-block:: console
 205
 206   # ls -arilh /backup/disk1/store1
 207   276493 -rw-r--r-- 1 backup backup       0 Jul  8 12:35 .lock
 208   276490 drwxr-x--- 1 backup backup 1064960 Jul  8 12:35 .chunks
 209
 210 `.lock` is an empty file used for process locking.
 211
 212 The `.chunks` directory contains folders, starting from `0000` and taking hexadecimal values until `ffff`. These
 213 directories will store the chunked data after a backup operation has been executed.
 214
 215 .. code-block:: console
 216
 217  # ls -arilh /backup/disk1/store1/.chunks
 218  545824 drwxr-x--- 2 backup backup 4.0K Jul  8 12:35 ffff
 219  545823 drwxr-x--- 2 backup backup 4.0K Jul  8 12:35 fffe
 220  415621 drwxr-x--- 2 backup backup 4.0K Jul  8 12:35 fffd
 221  415620 drwxr-x--- 2 backup backup 4.0K Jul  8 12:35 fffc
 222  353187 drwxr-x--- 2 backup backup 4.0K Jul  8 12:35 fffb
 223  344995 drwxr-x--- 2 backup backup 4.0K Jul  8 12:35 fffa
 224  144079 drwxr-x--- 2 backup backup 4.0K Jul  8 12:35 fff9
 225  144078 drwxr-x--- 2 backup backup 4.0K Jul  8 12:35 fff8
 226  144077 drwxr-x--- 2 backup backup 4.0K Jul  8 12:35 fff7
 227  ...
 228  403180 drwxr-x--- 2 backup backup 4.0K Jul  8 12:35 000c
 229  403179 drwxr-x--- 2 backup backup 4.0K Jul  8 12:35 000b
 230  403177 drwxr-x--- 2 backup backup 4.0K Jul  8 12:35 000a
 231  402530 drwxr-x--- 2 backup backup 4.0K Jul  8 12:35 0009
 232  402513 drwxr-x--- 2 backup backup 4.0K Jul  8 12:35 0008
 233  402509 drwxr-x--- 2 backup backup 4.0K Jul  8 12:35 0007
 234  276509 drwxr-x--- 2 backup backup 4.0K Jul  8 12:35 0006
 235  276508 drwxr-x--- 2 backup backup 4.0K Jul  8 12:35 0005
 236  276507 drwxr-x--- 2 backup backup 4.0K Jul  8 12:35 0004
 237  276501 drwxr-x--- 2 backup backup 4.0K Jul  8 12:35 0003
 238  276499 drwxr-x--- 2 backup backup 4.0K Jul  8 12:35 0002
 239  276498 drwxr-x--- 2 backup backup 4.0K Jul  8 12:35 0001
 240  276494 drwxr-x--- 2 backup backup 4.0K Jul  8 12:35 0000
 241  276489 drwxr-xr-x 3 backup backup 4.0K Jul  8 12:35 ..
 242  276490 drwxr-x--- 1 backup backup 1.1M Jul  8 12:35 .
 243
 244