]> git.proxmox.com Git - pve-docs.git/blame - pvesm.adoc
fix #1818: use NTP instead of Servers in timesyncd.conf
[pve-docs.git] / pvesm.adoc
CommitLineData
80c0adcb 1[[chapter_storage]]
aa039b0f 2ifdef::manvolnum[]
b2f242ab
DM
3pvesm(1)
4========
5f09af76
DM
5:pve-toplevel:
6
aa039b0f
DM
7NAME
8----
9
10pvesm - Proxmox VE Storage Manager
11
12
49a5e11c 13SYNOPSIS
aa039b0f
DM
14--------
15
16include::pvesm.1-synopsis.adoc[]
17
18DESCRIPTION
19-----------
20endif::manvolnum[]
aa039b0f
DM
21ifndef::manvolnum[]
22{pve} Storage
23=============
194d2f29 24:pve-toplevel:
aa039b0f 25endif::manvolnum[]
5f09af76 26ifdef::wiki[]
cb84ed18 27:title: Storage
5f09af76
DM
28endif::wiki[]
29
aa039b0f
DM
30The {pve} storage model is very flexible. Virtual machine images
31can either be stored on one or several local storages, or on shared
32storage like NFS or iSCSI (NAS, SAN). There are no limits, and you may
33configure as many storage pools as you like. You can use all
34storage technologies available for Debian Linux.
35
36One major benefit of storing VMs on shared storage is the ability to
37live-migrate running machines without any downtime, as all nodes in
38the cluster have direct access to VM disk images. There is no need to
39copy VM image data, so live migration is very fast in that case.
40
8c1189b6 41The storage library (package `libpve-storage-perl`) uses a flexible
aa039b0f
DM
42plugin system to provide a common interface to all storage types. This
43can be easily adopted to include further storage types in future.
44
45
46Storage Types
47-------------
48
49There are basically two different classes of storage types:
50
51Block level storage::
52
53Allows to store large 'raw' images. It is usually not possible to store
54other files (ISO, backups, ..) on such storage types. Most modern
55block level storage implementations support snapshots and clones.
0fdc757c 56RADOS, Sheepdog and GlusterFS are distributed systems, replicating storage
aa039b0f
DM
57data to different nodes.
58
59File level storage::
60
61They allow access to a full featured (POSIX) file system. They are
62more flexible, and allows you to store any content type. ZFS is
63probably the most advanced system, and it has full support for
64snapshots and clones.
65
66
67.Available storage types
68[width="100%",cols="<d,1*m,4*d",options="header"]
69|===========================================================
70|Description |PVE type |Level |Shared|Snapshots|Stable
71|ZFS (local) |zfspool |file |no |yes |yes
db7f8770
DC
72|Directory |dir |file |no |no^1^ |yes
73|NFS |nfs |file |yes |no^1^ |yes
de14ebff 74|CIFS |cifs |file |yes |no^1^ |yes
db7f8770
DC
75|GlusterFS |glusterfs |file |yes |no^1^ |yes
76|LVM |lvm |block |no^2^ |no |yes
9801e1c3 77|LVM-thin |lvmthin |block |no |yes |yes
aa039b0f
DM
78|iSCSI/kernel |iscsi |block |yes |no |yes
79|iSCSI/libiscsi |iscsidirect |block |yes |no |yes
80|Ceph/RBD |rbd |block |yes |yes |yes
81|Sheepdog |sheepdog |block |yes |yes |beta
aa039b0f
DM
82|ZFS over iSCSI |zfs |block |yes |yes |yes
83|=========================================================
84
db7f8770
DC
85^1^: On file based storages, snapshots are possible with the 'qcow2' format.
86
87^2^: It is possible to use LVM on top of an iSCSI storage. That way
8c1189b6 88you get a `shared` LVM storage.
aa039b0f 89
5eba0743
FG
90
91Thin Provisioning
2afe468c 92~~~~~~~~~~~~~~~~~
ebc15cbc 93
8c1189b6
FG
94A number of storages, and the Qemu image format `qcow2`, support 'thin
95provisioning'. With thin provisioning activated, only the blocks that
2afe468c 96the guest system actually use will be written to the storage.
ebc15cbc 97
2afe468c 98Say for instance you create a VM with a 32GB hard disk, and after
5eba0743 99installing the guest system OS, the root file system of the VM contains
2afe468c
DM
1003 GB of data. In that case only 3GB are written to the storage, even
101if the guest VM sees a 32GB hard drive. In this way thin provisioning
102allows you to create disk images which are larger than the currently
103available storage blocks. You can create large disk images for your
104VMs, and when the need arises, add more disks to your storage without
5eba0743 105resizing the VMs' file systems.
2afe468c 106
8c1189b6 107All storage types which have the ``Snapshots'' feature also support thin
2afe468c 108provisioning.
ebc15cbc 109
ba1d96fd 110CAUTION: If a storage runs full, all guests using volumes on that
38d1cf56 111storage receive IO errors. This can cause file system inconsistencies
ba1d96fd
DM
112and may corrupt your data. So it is advisable to avoid
113over-provisioning of your storage resources, or carefully observe
114free space to avoid such conditions.
ebc15cbc 115
5eba0743 116
aa039b0f
DM
117Storage Configuration
118---------------------
119
120All {pve} related storage configuration is stored within a single text
8c1189b6 121file at `/etc/pve/storage.cfg`. As this file is within `/etc/pve/`, it
aa039b0f
DM
122gets automatically distributed to all cluster nodes. So all nodes
123share the same storage configuration.
124
125Sharing storage configuration make perfect sense for shared storage,
8c1189b6 126because the same ``shared'' storage is accessible from all nodes. But is
aa039b0f
DM
127also useful for local storage types. In this case such local storage
128is available on all nodes, but it is physically different and can have
129totally different content.
130
5eba0743 131
aa039b0f
DM
132Storage Pools
133~~~~~~~~~~~~~
134
5eba0743
FG
135Each storage pool has a `<type>`, and is uniquely identified by its
136`<STORAGE_ID>`. A pool configuration looks like this:
aa039b0f
DM
137
138----
139<type>: <STORAGE_ID>
140 <property> <value>
141 <property> <value>
142 ...
143----
144
aa039b0f 145The `<type>: <STORAGE_ID>` line starts the pool definition, which is then
871e1fd6
FG
146followed by a list of properties. Most properties have values, but some of
147them come with reasonable default. In that case you can omit the value.
aa039b0f 148
9c41b54d
DM
149To be more specific, take a look at the default storage configuration
150after installation. It contains one special local storage pool named
8c1189b6 151`local`, which refers to the directory `/var/lib/vz` and is always
9c41b54d
DM
152available. The {pve} installer creates additional storage entries
153depending on the storage type chosen at installation time.
154
8c1189b6 155.Default storage configuration (`/etc/pve/storage.cfg`)
9801e1c3
DM
156----
157dir: local
aa039b0f 158 path /var/lib/vz
9801e1c3
DM
159 content iso,vztmpl,backup
160
9c41b54d 161# default image store on LVM based installation
9801e1c3
DM
162lvmthin: local-lvm
163 thinpool data
164 vgname pve
165 content rootdir,images
9c41b54d
DM
166
167# default image store on ZFS based installation
168zfspool: local-zfs
169 pool rpool/data
170 sparse
171 content images,rootdir
9801e1c3 172----
aa039b0f 173
5eba0743 174
aa039b0f
DM
175Common Storage Properties
176~~~~~~~~~~~~~~~~~~~~~~~~~
177
871e1fd6 178A few storage properties are common among different storage types.
aa039b0f
DM
179
180nodes::
181
182List of cluster node names where this storage is
183usable/accessible. One can use this property to restrict storage
184access to a limited set of nodes.
185
186content::
187
188A storage can support several content types, for example virtual disk
189images, cdrom iso images, container templates or container root
871e1fd6 190directories. Not all storage types support all content types. One can set
aa039b0f
DM
191this property to select for what this storage is used for.
192
193images:::
194
195KVM-Qemu VM images.
196
197rootdir:::
198
871e1fd6 199Allow to store container data.
aa039b0f
DM
200
201vztmpl:::
202
203Container templates.
204
205backup:::
206
8c1189b6 207Backup files (`vzdump`).
aa039b0f
DM
208
209iso:::
210
211ISO images
212
213shared::
214
215Mark storage as shared.
216
217disable::
218
219You can use this flag to disable the storage completely.
220
221maxfiles::
222
5eba0743 223Maximum number of backup files per VM. Use `0` for unlimited.
aa039b0f
DM
224
225format::
226
227Default image format (`raw|qcow2|vmdk`)
228
229
230WARNING: It is not advisable to use the same storage pool on different
871e1fd6 231{pve} clusters. Some storage operation need exclusive access to the
aa039b0f 232storage, so proper locking is required. While this is implemented
871e1fd6 233within a cluster, it does not work between different clusters.
aa039b0f
DM
234
235
236Volumes
237-------
238
239We use a special notation to address storage data. When you allocate
871e1fd6 240data from a storage pool, it returns such a volume identifier. A volume
aa039b0f
DM
241is identified by the `<STORAGE_ID>`, followed by a storage type
242dependent volume name, separated by colon. A valid `<VOLUME_ID>` looks
243like:
244
245 local:230/example-image.raw
246
247 local:iso/debian-501-amd64-netinst.iso
248
249 local:vztmpl/debian-5.0-joomla_1.5.9-1_i386.tar.gz
250
251 iscsi-storage:0.0.2.scsi-14f504e46494c4500494b5042546d2d646744372d31616d61
252
5eba0743 253To get the file system path for a `<VOLUME_ID>` use:
aa039b0f
DM
254
255 pvesm path <VOLUME_ID>
256
5eba0743 257
aa039b0f
DM
258Volume Ownership
259~~~~~~~~~~~~~~~~
260
8c1189b6 261There exists an ownership relation for `image` type volumes. Each such
aa039b0f
DM
262volume is owned by a VM or Container. For example volume
263`local:230/example-image.raw` is owned by VM 230. Most storage
264backends encodes this ownership information into the volume name.
265
871e1fd6 266When you remove a VM or Container, the system also removes all
aa039b0f
DM
267associated volumes which are owned by that VM or Container.
268
269
270Using the Command Line Interface
271--------------------------------
272
871e1fd6
FG
273It is recommended to familiarize yourself with the concept behind storage
274pools and volume identifiers, but in real life, you are not forced to do any
aa039b0f
DM
275of those low level operations on the command line. Normally,
276allocation and removal of volumes is done by the VM and Container
277management tools.
278
8c1189b6
FG
279Nevertheless, there is a command line tool called `pvesm` (``{pve}
280Storage Manager''), which is able to perform common storage management
aa039b0f
DM
281tasks.
282
283
284Examples
285~~~~~~~~
286
287Add storage pools
288
289 pvesm add <TYPE> <STORAGE_ID> <OPTIONS>
290 pvesm add dir <STORAGE_ID> --path <PATH>
291 pvesm add nfs <STORAGE_ID> --path <PATH> --server <SERVER> --export <EXPORT>
292 pvesm add lvm <STORAGE_ID> --vgname <VGNAME>
293 pvesm add iscsi <STORAGE_ID> --portal <HOST[:PORT]> --target <TARGET>
294
295Disable storage pools
296
297 pvesm set <STORAGE_ID> --disable 1
298
299Enable storage pools
300
301 pvesm set <STORAGE_ID> --disable 0
302
303Change/set storage options
304
305 pvesm set <STORAGE_ID> <OPTIONS>
306 pvesm set <STORAGE_ID> --shared 1
307 pvesm set local --format qcow2
308 pvesm set <STORAGE_ID> --content iso
309
310Remove storage pools. This does not delete any data, and does not
311disconnect or unmount anything. It just removes the storage
312configuration.
313
314 pvesm remove <STORAGE_ID>
315
316Allocate volumes
317
318 pvesm alloc <STORAGE_ID> <VMID> <name> <size> [--format <raw|qcow2>]
319
320Allocate a 4G volume in local storage. The name is auto-generated if
321you pass an empty string as `<name>`
322
323 pvesm alloc local <VMID> '' 4G
324
5eba0743 325Free volumes
aa039b0f
DM
326
327 pvesm free <VOLUME_ID>
328
329WARNING: This really destroys all volume data.
330
331List storage status
332
333 pvesm status
334
335List storage contents
336
337 pvesm list <STORAGE_ID> [--vmid <VMID>]
338
339List volumes allocated by VMID
340
341 pvesm list <STORAGE_ID> --vmid <VMID>
342
343List iso images
344
345 pvesm list <STORAGE_ID> --iso
346
347List container templates
348
349 pvesm list <STORAGE_ID> --vztmpl
350
5eba0743 351Show file system path for a volume
aa039b0f
DM
352
353 pvesm path <VOLUME_ID>
354
deb4673f
DM
355ifdef::wiki[]
356
357See Also
358--------
359
f532afb7 360* link:/wiki/Storage:_Directory[Storage: Directory]
deb4673f 361
f532afb7 362* link:/wiki/Storage:_GlusterFS[Storage: GlusterFS]
deb4673f 363
f532afb7 364* link:/wiki/Storage:_User_Mode_iSCSI[Storage: User Mode iSCSI]
deb4673f 365
f532afb7 366* link:/wiki/Storage:_iSCSI[Storage: iSCSI]
deb4673f 367
f532afb7 368* link:/wiki/Storage:_LVM[Storage: LVM]
deb4673f 369
f532afb7 370* link:/wiki/Storage:_LVM_Thin[Storage: LVM Thin]
deb4673f 371
f532afb7 372* link:/wiki/Storage:_NFS[Storage: NFS]
deb4673f 373
de14ebff
WL
374* link:/wiki/Storage:_CIFS[Storage: CIFS]
375
f532afb7 376* link:/wiki/Storage:_RBD[Storage: RBD]
deb4673f 377
f532afb7 378* link:/wiki/Storage:_ZFS[Storage: ZFS]
deb4673f 379
032e755c 380* link:/wiki/Storage:_ZFS_over_iSCSI[Storage: ZFS over iSCSI]
deb4673f
DM
381
382endif::wiki[]
383
251666be
DM
384ifndef::wiki[]
385
aa039b0f
DM
386// backend documentation
387
388include::pve-storage-dir.adoc[]
389
390include::pve-storage-nfs.adoc[]
391
de14ebff
WL
392include::pve-storage-cifs.adoc[]
393
aa039b0f
DM
394include::pve-storage-glusterfs.adoc[]
395
396include::pve-storage-zfspool.adoc[]
397
398include::pve-storage-lvm.adoc[]
399
9801e1c3
DM
400include::pve-storage-lvmthin.adoc[]
401
aa039b0f
DM
402include::pve-storage-iscsi.adoc[]
403
404include::pve-storage-iscsidirect.adoc[]
405
406include::pve-storage-rbd.adoc[]
407
408
251666be 409
aa039b0f
DM
410ifdef::manvolnum[]
411include::pve-copyright.adoc[]
412endif::manvolnum[]
413
251666be
DM
414endif::wiki[]
415