]> git.proxmox.com Git - pve-docs.git/blame - local-zfs.adoc
qm-cloud-init: make install inside VM more obvious
[pve-docs.git] / local-zfs.adoc
CommitLineData
0235c741 1[[chapter_zfs]]
9ee94323
DM
2ZFS on Linux
3------------
5f09af76
DM
4ifdef::wiki[]
5:pve-toplevel:
6endif::wiki[]
7
9ee94323
DM
8ZFS is a combined file system and logical volume manager designed by
9Sun Microsystems. Starting with {pve} 3.4, the native Linux
10kernel port of the ZFS file system is introduced as optional
5eba0743
FG
11file system and also as an additional selection for the root
12file system. There is no need for manually compile ZFS modules - all
9ee94323
DM
13packages are included.
14
5eba0743 15By using ZFS, its possible to achieve maximum enterprise features with
9ee94323
DM
16low budget hardware, but also high performance systems by leveraging
17SSD caching or even SSD only setups. ZFS can replace cost intense
18hardware raid cards by moderate CPU and memory load combined with easy
19management.
20
21.General ZFS advantages
22
23* Easy configuration and management with {pve} GUI and CLI.
24
25* Reliable
26
27* Protection against data corruption
28
5eba0743 29* Data compression on file system level
9ee94323
DM
30
31* Snapshots
32
33* Copy-on-write clone
34
447596fd
SH
35* Various raid levels: RAID0, RAID1, RAID10, RAIDZ-1, RAIDZ-2, RAIDZ-3,
36dRAID, dRAID2, dRAID3
9ee94323
DM
37
38* Can use SSD for cache
39
40* Self healing
41
42* Continuous integrity checking
43
44* Designed for high storage capacities
45
9ee94323
DM
46* Asynchronous replication over network
47
48* Open Source
49
50* Encryption
51
52* ...
53
54
55Hardware
56~~~~~~~~
57
58ZFS depends heavily on memory, so you need at least 8GB to start. In
60ed554f 59practice, use as much as you can get for your hardware/budget. To prevent
9ee94323
DM
60data corruption, we recommend the use of high quality ECC RAM.
61
d48bdcf2 62If you use a dedicated cache and/or log disk, you should use an
0d4a93dc 63enterprise class SSD. This can
9ee94323
DM
64increase the overall performance significantly.
65
60ed554f
DW
66IMPORTANT: Do not use ZFS on top of a hardware RAID controller which has its
67own cache management. ZFS needs to communicate directly with the disks. An
68HBA adapter or something like an LSI controller flashed in ``IT'' mode is more
69appropriate.
9ee94323
DM
70
71If you are experimenting with an installation of {pve} inside a VM
8c1189b6 72(Nested Virtualization), don't use `virtio` for disks of that VM,
60ed554f
DW
73as they are not supported by ZFS. Use IDE or SCSI instead (also works
74with the `virtio` SCSI controller type).
9ee94323
DM
75
76
5eba0743 77Installation as Root File System
9ee94323
DM
78~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
79
80When you install using the {pve} installer, you can choose ZFS for the
81root file system. You need to select the RAID type at installation
82time:
83
84[horizontal]
8c1189b6
FG
85RAID0:: Also called ``striping''. The capacity of such volume is the sum
86of the capacities of all disks. But RAID0 does not add any redundancy,
9ee94323
DM
87so the failure of a single drive makes the volume unusable.
88
8c1189b6 89RAID1:: Also called ``mirroring''. Data is written identically to all
9ee94323
DM
90disks. This mode requires at least 2 disks with the same size. The
91resulting capacity is that of a single disk.
92
93RAID10:: A combination of RAID0 and RAID1. Requires at least 4 disks.
94
95RAIDZ-1:: A variation on RAID-5, single parity. Requires at least 3 disks.
96
97RAIDZ-2:: A variation on RAID-5, double parity. Requires at least 4 disks.
98
99RAIDZ-3:: A variation on RAID-5, triple parity. Requires at least 5 disks.
100
101The installer automatically partitions the disks, creates a ZFS pool
8c1189b6
FG
102called `rpool`, and installs the root file system on the ZFS subvolume
103`rpool/ROOT/pve-1`.
9ee94323 104
8c1189b6 105Another subvolume called `rpool/data` is created to store VM
9ee94323 106images. In order to use that with the {pve} tools, the installer
8c1189b6 107creates the following configuration entry in `/etc/pve/storage.cfg`:
9ee94323
DM
108
109----
110zfspool: local-zfs
111 pool rpool/data
112 sparse
113 content images,rootdir
114----
115
116After installation, you can view your ZFS pool status using the
8c1189b6 117`zpool` command:
9ee94323
DM
118
119----
120# zpool status
121 pool: rpool
122 state: ONLINE
123 scan: none requested
124config:
125
126 NAME STATE READ WRITE CKSUM
127 rpool ONLINE 0 0 0
128 mirror-0 ONLINE 0 0 0
129 sda2 ONLINE 0 0 0
130 sdb2 ONLINE 0 0 0
131 mirror-1 ONLINE 0 0 0
132 sdc ONLINE 0 0 0
133 sdd ONLINE 0 0 0
134
135errors: No known data errors
136----
137
8c1189b6 138The `zfs` command is used configure and manage your ZFS file
9ee94323
DM
139systems. The following command lists all file systems after
140installation:
141
142----
143# zfs list
144NAME USED AVAIL REFER MOUNTPOINT
145rpool 4.94G 7.68T 96K /rpool
146rpool/ROOT 702M 7.68T 96K /rpool/ROOT
147rpool/ROOT/pve-1 702M 7.68T 702M /
148rpool/data 96K 7.68T 96K /rpool/data
149rpool/swap 4.25G 7.69T 64K -
150----
151
152
e4262cac
AL
153[[sysadmin_zfs_raid_considerations]]
154ZFS RAID Level Considerations
155~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
156
157There are a few factors to take into consideration when choosing the layout of
158a ZFS pool. The basic building block of a ZFS pool is the virtual device, or
159`vdev`. All vdevs in a pool are used equally and the data is striped among them
160(RAID0). Check the `zpool(8)` manpage for more details on vdevs.
161
162[[sysadmin_zfs_raid_performance]]
163Performance
164^^^^^^^^^^^
165
166Each `vdev` type has different performance behaviors. The two
167parameters of interest are the IOPS (Input/Output Operations per Second) and
168the bandwidth with which data can be written or read.
169
f1b7d1a3
MH
170A 'mirror' vdev (RAID1) will approximately behave like a single disk in regard
171to both parameters when writing data. When reading data the performance will
172scale linearly with the number of disks in the mirror.
e4262cac
AL
173
174A common situation is to have 4 disks. When setting it up as 2 mirror vdevs
175(RAID10) the pool will have the write characteristics as two single disks in
f1b7d1a3 176regard to IOPS and bandwidth. For read operations it will resemble 4 single
e4262cac
AL
177disks.
178
179A 'RAIDZ' of any redundancy level will approximately behave like a single disk
f1b7d1a3 180in regard to IOPS with a lot of bandwidth. How much bandwidth depends on the
e4262cac
AL
181size of the RAIDZ vdev and the redundancy level.
182
183For running VMs, IOPS is the more important metric in most situations.
184
185
186[[sysadmin_zfs_raid_size_space_usage_redundancy]]
187Size, Space usage and Redundancy
188^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
189
190While a pool made of 'mirror' vdevs will have the best performance
191characteristics, the usable space will be 50% of the disks available. Less if a
192mirror vdev consists of more than 2 disks, for example in a 3-way mirror. At
193least one healthy disk per mirror is needed for the pool to stay functional.
194
195The usable space of a 'RAIDZ' type vdev of N disks is roughly N-P, with P being
196the RAIDZ-level. The RAIDZ-level indicates how many arbitrary disks can fail
197without losing data. A special case is a 4 disk pool with RAIDZ2. In this
198situation it is usually better to use 2 mirror vdevs for the better performance
199as the usable space will be the same.
200
201Another important factor when using any RAIDZ level is how ZVOL datasets, which
202are used for VM disks, behave. For each data block the pool needs parity data
203which is at least the size of the minimum block size defined by the `ashift`
204value of the pool. With an ashift of 12 the block size of the pool is 4k. The
205default block size for a ZVOL is 8k. Therefore, in a RAIDZ2 each 8k block
206written will cause two additional 4k parity blocks to be written,
2078k + 4k + 4k = 16k. This is of course a simplified approach and the real
208situation will be slightly different with metadata, compression and such not
209being accounted for in this example.
210
211This behavior can be observed when checking the following properties of the
212ZVOL:
213
214 * `volsize`
215 * `refreservation` (if the pool is not thin provisioned)
216 * `used` (if the pool is thin provisioned and without snapshots present)
217
218----
219# zfs get volsize,refreservation,used <pool>/vm-<vmid>-disk-X
220----
221
222`volsize` is the size of the disk as it is presented to the VM, while
223`refreservation` shows the reserved space on the pool which includes the
224expected space needed for the parity data. If the pool is thin provisioned, the
225`refreservation` will be set to 0. Another way to observe the behavior is to
226compare the used disk space within the VM and the `used` property. Be aware
227that snapshots will skew the value.
228
229There are a few options to counter the increased use of space:
230
231* Increase the `volblocksize` to improve the data to parity ratio
232* Use 'mirror' vdevs instead of 'RAIDZ'
233* Use `ashift=9` (block size of 512 bytes)
234
235The `volblocksize` property can only be set when creating a ZVOL. The default
236value can be changed in the storage configuration. When doing this, the guest
237needs to be tuned accordingly and depending on the use case, the problem of
b2444770 238write amplification is just moved from the ZFS layer up to the guest.
e4262cac
AL
239
240Using `ashift=9` when creating the pool can lead to bad
241performance, depending on the disks underneath, and cannot be changed later on.
242
243Mirror vdevs (RAID1, RAID10) have favorable behavior for VM workloads. Use
f4abc68a 244them, unless your environment has specific needs and characteristics where
e4262cac
AL
245RAIDZ performance characteristics are acceptable.
246
247
447596fd
SH
248ZFS dRAID
249~~~~~~~~~
250
251In a ZFS dRAID (declustered RAID) the hot spare drive(s) participate in the RAID.
252Their spare capacity is reserved and used for rebuilding when one drive fails.
253This provides, depending on the configuration, faster rebuilding compared to a
254RAIDZ in case of drive failure. More information can be found in the official
255OpenZFS documentation. footnote:[OpenZFS dRAID
256https://openzfs.github.io/openzfs-docs/Basic%20Concepts/dRAID%20Howto.html]
257
258NOTE: dRAID is intended for more than 10-15 disks in a dRAID. A RAIDZ
259setup should be better for a lower amount of disks in most use cases.
260
261NOTE: The GUI requires one more disk than the minimum (i.e. dRAID1 needs 3). It
262expects that a spare disk is added as well.
263
264 * `dRAID1` or `dRAID`: requires at least 2 disks, one can fail before data is
265lost
266 * `dRAID2`: requires at least 3 disks, two can fail before data is lost
267 * `dRAID3`: requires at least 4 disks, three can fail before data is lost
268
269
270Additional information can be found on the manual page:
271
272----
273# man zpoolconcepts
274----
275
276Spares and Data
277^^^^^^^^^^^^^^^
278The number of `spares` tells the system how many disks it should keep ready in
279case of a disk failure. The default value is 0 `spares`. Without spares,
280rebuilding won't get any speed benefits.
281
282`data` defines the number of devices in a redundancy group. The default value is
2838. Except when `disks - parity - spares` equal something less than 8, the lower
284number is used. In general, a smaller number of `data` devices leads to higher
285IOPS, better compression ratios and faster resilvering, but defining fewer data
286devices reduces the available storage capacity of the pool.
287
288
9ee94323
DM
289Bootloader
290~~~~~~~~~~
291
cb04e768
SI
292{pve} uses xref:sysboot_proxmox_boot_tool[`proxmox-boot-tool`] to manage the
293bootloader configuration.
3a433e9b 294See the chapter on xref:sysboot[{pve} host bootloaders] for details.
9ee94323
DM
295
296
297ZFS Administration
298~~~~~~~~~~~~~~~~~~
299
300This section gives you some usage examples for common tasks. ZFS
301itself is really powerful and provides many options. The main commands
8c1189b6
FG
302to manage ZFS are `zfs` and `zpool`. Both commands come with great
303manual pages, which can be read with:
9ee94323
DM
304
305----
306# man zpool
307# man zfs
308-----
309
42449bdf
TL
310[[sysadmin_zfs_create_new_zpool]]
311Create a new zpool
312^^^^^^^^^^^^^^^^^^
9ee94323 313
25b89d16
TL
314To create a new pool, at least one disk is needed. The `ashift` should have the
315same sector-size (2 power of `ashift`) or larger as the underlying disk.
9ee94323 316
eaefe614
FE
317----
318# zpool create -f -o ashift=12 <pool> <device>
319----
9ee94323 320
25b89d16
TL
321[TIP]
322====
323Pool names must adhere to the following rules:
324
325* begin with a letter (a-z or A-Z)
326* contain only alphanumeric, `-`, `_`, `.`, `:` or ` ` (space) characters
327* must *not begin* with one of `mirror`, `raidz`, `draid` or `spare`
328* must not be `log`
329====
330
e06707f2 331To activate compression (see section <<zfs_compression,Compression in ZFS>>):
9ee94323 332
eaefe614
FE
333----
334# zfs set compression=lz4 <pool>
335----
9ee94323 336
42449bdf
TL
337[[sysadmin_zfs_create_new_zpool_raid0]]
338Create a new pool with RAID-0
339^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
9ee94323 340
dc2d00a0 341Minimum 1 disk
9ee94323 342
eaefe614
FE
343----
344# zpool create -f -o ashift=12 <pool> <device1> <device2>
345----
9ee94323 346
42449bdf
TL
347[[sysadmin_zfs_create_new_zpool_raid1]]
348Create a new pool with RAID-1
349^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
9ee94323 350
dc2d00a0 351Minimum 2 disks
9ee94323 352
eaefe614
FE
353----
354# zpool create -f -o ashift=12 <pool> mirror <device1> <device2>
355----
9ee94323 356
42449bdf
TL
357[[sysadmin_zfs_create_new_zpool_raid10]]
358Create a new pool with RAID-10
359^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
9ee94323 360
dc2d00a0 361Minimum 4 disks
9ee94323 362
eaefe614
FE
363----
364# zpool create -f -o ashift=12 <pool> mirror <device1> <device2> mirror <device3> <device4>
365----
9ee94323 366
42449bdf
TL
367[[sysadmin_zfs_create_new_zpool_raidz1]]
368Create a new pool with RAIDZ-1
369^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
9ee94323 370
dc2d00a0 371Minimum 3 disks
9ee94323 372
eaefe614
FE
373----
374# zpool create -f -o ashift=12 <pool> raidz1 <device1> <device2> <device3>
375----
9ee94323 376
42449bdf
TL
377Create a new pool with RAIDZ-2
378^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
9ee94323 379
dc2d00a0 380Minimum 4 disks
9ee94323 381
eaefe614
FE
382----
383# zpool create -f -o ashift=12 <pool> raidz2 <device1> <device2> <device3> <device4>
384----
9ee94323 385
8a1de6bf
TL
386Please read the section for
387xref:sysadmin_zfs_raid_considerations[ZFS RAID Level Considerations]
388to get a rough estimate on how IOPS and bandwidth expectations before setting up
389a pool, especially when wanting to use a RAID-Z mode.
390
42449bdf
TL
391[[sysadmin_zfs_create_new_zpool_with_cache]]
392Create a new pool with cache (L2ARC)
393^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
9ee94323 394
5f440d2c
TL
395It is possible to use a dedicated device, or partition, as second-level cache to
396increase the performance. Such a cache device will especially help with
397random-read workloads of data that is mostly static. As it acts as additional
398caching layer between the actual storage, and the in-memory ARC, it can also
399help if the ARC must be reduced due to memory constraints.
9ee94323 400
5f440d2c 401.Create ZFS pool with a on-disk cache
eaefe614 402----
5f440d2c 403# zpool create -f -o ashift=12 <pool> <device> cache <cache-device>
eaefe614 404----
9ee94323 405
5f440d2c
TL
406Here only a single `<device>` and a single `<cache-device>` was used, but it is
407possible to use more devices, like it's shown in
408xref:sysadmin_zfs_create_new_zpool_raid0[Create a new pool with RAID].
409
410Note that for cache devices no mirror or raid modi exist, they are all simply
411accumulated.
412
413If any cache device produces errors on read, ZFS will transparently divert that
414request to the underlying storage layer.
415
416
42449bdf
TL
417[[sysadmin_zfs_create_new_zpool_with_log]]
418Create a new pool with log (ZIL)
419^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
9ee94323 420
5f440d2c
TL
421It is possible to use a dedicated drive, or partition, for the ZFS Intent Log
422(ZIL), it is mainly used to provide safe synchronous transactions, so often in
423performance critical paths like databases, or other programs that issue `fsync`
424operations more frequently.
9ee94323 425
5f440d2c
TL
426The pool is used as default ZIL location, diverting the ZIL IO load to a
427separate device can, help to reduce transaction latencies while relieving the
428main pool at the same time, increasing overall performance.
9ee94323 429
5f440d2c
TL
430For disks to be used as log devices, directly or through a partition, it's
431recommend to:
432
433- use fast SSDs with power-loss protection, as those have much smaller commit
434 latencies.
435
436- Use at least a few GB for the partition (or whole device), but using more than
437 half of your installed memory won't provide you with any real advantage.
438
439.Create ZFS pool with separate log device
eaefe614 440----
5f440d2c 441# zpool create -f -o ashift=12 <pool> <device> log <log-device>
eaefe614 442----
9ee94323 443
5f440d2c
TL
444In above example a single `<device>` and a single `<log-device>` is used, but you
445can also combine this with other RAID variants, as described in the
446xref:sysadmin_zfs_create_new_zpool_raid0[Create a new pool with RAID] section.
447
448You can also mirror the log device to multiple devices, this is mainly useful to
449ensure that performance doesn't immediately degrades if a single log device
450fails.
451
452If all log devices fail the ZFS main pool itself will be used again, until the
453log device(s) get replaced.
454
42449bdf
TL
455[[sysadmin_zfs_add_cache_and_log_dev]]
456Add cache and log to an existing pool
457^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
9ee94323 458
5f440d2c
TL
459If you have a pool without cache and log you can still add both, or just one of
460them, at any time.
461
462For example, let's assume you got a good enterprise SSD with power-loss
463protection that you want to use for improving the overall performance of your
464pool.
465
466As the maximum size of a log device should be about half the size of the
467installed physical memory, it means that the ZIL will mostly likely only take up
468a relatively small part of the SSD, the remaining space can be used as cache.
9ee94323 469
5f440d2c 470First you have to create two GPT partitions on the SSD with `parted` or `gdisk`.
9ee94323 471
5f440d2c 472Then you're ready to add them to an pool:
9ee94323 473
5f440d2c 474.Add both, a separate log device and a second-level cache, to an existing pool
eaefe614 475----
237007eb 476# zpool add -f <pool> log <device-part1> cache <device-part2>
eaefe614 477----
9ee94323 478
5f440d2c
TL
479Just replay `<pool>`, `<device-part1>` and `<device-part2>` with the pool name
480and the two `/dev/disk/by-id/` paths to the partitions.
481
482You can also add ZIL and cache separately.
483
484.Add a log device to an existing ZFS pool
485----
486# zpool add <pool> log <log-device>
487----
488
489
42449bdf
TL
490[[sysadmin_zfs_change_failed_dev]]
491Changing a failed device
492^^^^^^^^^^^^^^^^^^^^^^^^
9ee94323 493
eaefe614 494----
5f440d2c 495# zpool replace -f <pool> <old-device> <new-device>
eaefe614 496----
1748211a 497
11a6e022
AL
498.Changing a failed bootable device
499
69c2b2e5
SI
500Depending on how {pve} was installed it is either using `systemd-boot` or `grub`
501through `proxmox-boot-tool`
cb04e768
SI
502footnote:[Systems installed with {pve} 6.4 or later, EFI systems installed with
503{pve} 5.4 or later] or plain `grub` as bootloader (see
504xref:sysboot[Host Bootloader]). You can check by running:
505
506----
507# proxmox-boot-tool status
508----
11a6e022
AL
509
510The first steps of copying the partition table, reissuing GUIDs and replacing
511the ZFS partition are the same. To make the system bootable from the new disk,
512different steps are needed which depend on the bootloader in use.
1748211a 513
eaefe614
FE
514----
515# sgdisk <healthy bootable device> -R <new device>
516# sgdisk -G <new device>
517# zpool replace -f <pool> <old zfs partition> <new zfs partition>
11a6e022
AL
518----
519
44aee838 520NOTE: Use the `zpool status -v` command to monitor how far the resilvering
11a6e022
AL
521process of the new disk has progressed.
522
cb04e768 523.With `proxmox-boot-tool`:
11a6e022
AL
524
525----
cb04e768 526# proxmox-boot-tool format <new disk's ESP>
952ee606 527# proxmox-boot-tool init <new disk's ESP> [grub]
eaefe614 528----
0daaddbd
FG
529
530NOTE: `ESP` stands for EFI System Partition, which is setup as partition #2 on
531bootable disks setup by the {pve} installer since version 5.4. For details, see
cb04e768 532xref:sysboot_proxmox_boot_setup[Setting up a new partition for use as synced ESP].
9ee94323 533
952ee606
FG
534NOTE: make sure to pass 'grub' as mode to `proxmox-boot-tool init` if
535`proxmox-boot-tool status` indicates your current disks are using Grub,
536especially if Secure Boot is enabled!
537
69c2b2e5 538.With plain `grub`:
11a6e022
AL
539
540----
541# grub-install <new disk>
542----
69c2b2e5
SI
543NOTE: plain `grub` is only used on systems installed with {pve} 6.3 or earlier,
544which have not been manually migrated to using `proxmox-boot-tool` yet.
545
9ee94323 546
aa425868
FE
547Configure E-Mail Notification
548~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
9ee94323 549
aa425868
FE
550ZFS comes with an event daemon `ZED`, which monitors events generated by the ZFS
551kernel module. The daemon can also send emails on ZFS events like pool errors.
552Newer ZFS packages ship the daemon in a separate `zfs-zed` package, which should
553already be installed by default in {pve}.
e280a948 554
aa425868
FE
555You can configure the daemon via the file `/etc/zfs/zed.d/zed.rc` with your
556favorite editor. The required setting for email notification is
557`ZED_EMAIL_ADDR`, which is set to `root` by default.
9ee94323 558
083adc34 559--------
9ee94323 560ZED_EMAIL_ADDR="root"
083adc34 561--------
9ee94323 562
8c1189b6 563Please note {pve} forwards mails to `root` to the email address
9ee94323
DM
564configured for the root user.
565
9ee94323 566
42449bdf 567[[sysadmin_zfs_limit_memory_usage]]
5eba0743 568Limit ZFS Memory Usage
9ee94323
DM
569~~~~~~~~~~~~~~~~~~~~~~
570
9060abb9
TL
571ZFS uses '50 %' of the host memory for the **A**daptive **R**eplacement
572**C**ache (ARC) by default. Allocating enough memory for the ARC is crucial for
573IO performance, so reduce it with caution. As a general rule of thumb, allocate
574at least +2 GiB Base + 1 GiB/TiB-Storage+. For example, if you have a pool with
575+8 TiB+ of available storage space then you should use +10 GiB+ of memory for
576the ARC.
577
578You can change the ARC usage limit for the current boot (a reboot resets this
579change again) by writing to the +zfs_arc_max+ module parameter directly:
580
581----
582 echo "$[10 * 1024*1024*1024]" >/sys/module/zfs/parameters/zfs_arc_max
583----
584
585To *permanently change* the ARC limits, add the following line to
586`/etc/modprobe.d/zfs.conf`:
9ee94323 587
5eba0743
FG
588--------
589options zfs zfs_arc_max=8589934592
590--------
9ee94323 591
9060abb9 592This example setting limits the usage to 8 GiB ('8 * 2^30^').
9ee94323 593
beed14f8
FG
594IMPORTANT: In case your desired +zfs_arc_max+ value is lower than or equal to
595+zfs_arc_min+ (which defaults to 1/32 of the system memory), +zfs_arc_max+ will
596be ignored unless you also set +zfs_arc_min+ to at most +zfs_arc_max - 1+.
597
598----
599echo "$[8 * 1024*1024*1024 - 1]" >/sys/module/zfs/parameters/zfs_arc_min
600echo "$[8 * 1024*1024*1024]" >/sys/module/zfs/parameters/zfs_arc_max
601----
602
603This example setting (temporarily) limits the usage to 8 GiB ('8 * 2^30^') on
604systems with more than 256 GiB of total memory, where simply setting
605+zfs_arc_max+ alone would not work.
606
9ee94323
DM
607[IMPORTANT]
608====
9060abb9 609If your root file system is ZFS, you must update your initramfs every
5eba0743 610time this value changes:
9ee94323 611
eaefe614 612----
abdfbbbb 613# update-initramfs -u -k all
eaefe614 614----
9060abb9
TL
615
616You *must reboot* to activate these changes.
9ee94323
DM
617====
618
619
dc74fc63 620[[zfs_swap]]
4128e7ff
TL
621SWAP on ZFS
622~~~~~~~~~~~
9ee94323 623
dc74fc63 624Swap-space created on a zvol may generate some troubles, like blocking the
9ee94323
DM
625server or generating a high IO load, often seen when starting a Backup
626to an external Storage.
627
628We strongly recommend to use enough memory, so that you normally do not
dc74fc63 629run into low memory situations. Should you need or want to add swap, it is
3a433e9b 630preferred to create a partition on a physical disk and use it as a swap device.
dc74fc63
SI
631You can leave some space free for this purpose in the advanced options of the
632installer. Additionally, you can lower the
8c1189b6 633``swappiness'' value. A good value for servers is 10:
9ee94323 634
eaefe614
FE
635----
636# sysctl -w vm.swappiness=10
637----
9ee94323 638
8c1189b6 639To make the swappiness persistent, open `/etc/sysctl.conf` with
9ee94323
DM
640an editor of your choice and add the following line:
641
083adc34
FG
642--------
643vm.swappiness = 10
644--------
9ee94323 645
8c1189b6 646.Linux kernel `swappiness` parameter values
9ee94323
DM
647[width="100%",cols="<m,2d",options="header"]
648|===========================================================
649| Value | Strategy
650| vm.swappiness = 0 | The kernel will swap only to avoid
651an 'out of memory' condition
652| vm.swappiness = 1 | Minimum amount of swapping without
653disabling it entirely.
654| vm.swappiness = 10 | This value is sometimes recommended to
655improve performance when sufficient memory exists in a system.
656| vm.swappiness = 60 | The default value.
657| vm.swappiness = 100 | The kernel will swap aggressively.
658|===========================================================
cca0540e
FG
659
660[[zfs_encryption]]
4128e7ff
TL
661Encrypted ZFS Datasets
662~~~~~~~~~~~~~~~~~~~~~~
cca0540e 663
500e5ab3
ML
664WARNING: Native ZFS encryption in {pve} is experimental. Known limitations and
665issues include Replication with encrypted datasets
666footnote:[https://bugzilla.proxmox.com/show_bug.cgi?id=2350],
667as well as checksum errors when using Snapshots or ZVOLs.
668footnote:[https://github.com/openzfs/zfs/issues/11688]
669
cca0540e
FG
670ZFS on Linux version 0.8.0 introduced support for native encryption of
671datasets. After an upgrade from previous ZFS on Linux versions, the encryption
229426eb 672feature can be enabled per pool:
cca0540e
FG
673
674----
675# zpool get feature@encryption tank
676NAME PROPERTY VALUE SOURCE
677tank feature@encryption disabled local
678
679# zpool set feature@encryption=enabled
680
681# zpool get feature@encryption tank
682NAME PROPERTY VALUE SOURCE
683tank feature@encryption enabled local
684----
685
686WARNING: There is currently no support for booting from pools with encrypted
687datasets using Grub, and only limited support for automatically unlocking
688encrypted datasets on boot. Older versions of ZFS without encryption support
689will not be able to decrypt stored data.
690
691NOTE: It is recommended to either unlock storage datasets manually after
692booting, or to write a custom unit to pass the key material needed for
693unlocking on boot to `zfs load-key`.
694
695WARNING: Establish and test a backup procedure before enabling encryption of
5dfeeece 696production data. If the associated key material/passphrase/keyfile has been
cca0540e
FG
697lost, accessing the encrypted data is no longer possible.
698
699Encryption needs to be setup when creating datasets/zvols, and is inherited by
700default to child datasets. For example, to create an encrypted dataset
701`tank/encrypted_data` and configure it as storage in {pve}, run the following
702commands:
703
704----
705# zfs create -o encryption=on -o keyformat=passphrase tank/encrypted_data
706Enter passphrase:
707Re-enter passphrase:
708
709# pvesm add zfspool encrypted_zfs -pool tank/encrypted_data
710----
711
712All guest volumes/disks create on this storage will be encrypted with the
713shared key material of the parent dataset.
714
715To actually use the storage, the associated key material needs to be loaded
7353437b 716and the dataset needs to be mounted. This can be done in one step with:
cca0540e
FG
717
718----
7353437b 719# zfs mount -l tank/encrypted_data
cca0540e
FG
720Enter passphrase for 'tank/encrypted_data':
721----
722
723It is also possible to use a (random) keyfile instead of prompting for a
724passphrase by setting the `keylocation` and `keyformat` properties, either at
229426eb 725creation time or with `zfs change-key` on existing datasets:
cca0540e
FG
726
727----
728# dd if=/dev/urandom of=/path/to/keyfile bs=32 count=1
729
730# zfs change-key -o keyformat=raw -o keylocation=file:///path/to/keyfile tank/encrypted_data
731----
732
733WARNING: When using a keyfile, special care needs to be taken to secure the
734keyfile against unauthorized access or accidental loss. Without the keyfile, it
735is not possible to access the plaintext data!
736
737A guest volume created underneath an encrypted dataset will have its
738`encryptionroot` property set accordingly. The key material only needs to be
739loaded once per encryptionroot to be available to all encrypted datasets
740underneath it.
741
742See the `encryptionroot`, `encryption`, `keylocation`, `keyformat` and
743`keystatus` properties, the `zfs load-key`, `zfs unload-key` and `zfs
744change-key` commands and the `Encryption` section from `man zfs` for more
745details and advanced usage.
68029ec8
FE
746
747
e06707f2
FE
748[[zfs_compression]]
749Compression in ZFS
750~~~~~~~~~~~~~~~~~~
751
752When compression is enabled on a dataset, ZFS tries to compress all *new*
753blocks before writing them and decompresses them on reading. Already
754existing data will not be compressed retroactively.
755
756You can enable compression with:
757
758----
759# zfs set compression=<algorithm> <dataset>
760----
761
762We recommend using the `lz4` algorithm, because it adds very little CPU
763overhead. Other algorithms like `lzjb` and `gzip-N`, where `N` is an
764integer from `1` (fastest) to `9` (best compression ratio), are also
765available. Depending on the algorithm and how compressible the data is,
766having compression enabled can even increase I/O performance.
767
768You can disable compression at any time with:
769
770----
771# zfs set compression=off <dataset>
772----
773
774Again, only new blocks will be affected by this change.
775
776
42449bdf 777[[sysadmin_zfs_special_device]]
68029ec8
FE
778ZFS Special Device
779~~~~~~~~~~~~~~~~~~
780
781Since version 0.8.0 ZFS supports `special` devices. A `special` device in a
782pool is used to store metadata, deduplication tables, and optionally small
783file blocks.
784
785A `special` device can improve the speed of a pool consisting of slow spinning
51e544b6
TL
786hard disks with a lot of metadata changes. For example workloads that involve
787creating, updating or deleting a large number of files will benefit from the
788presence of a `special` device. ZFS datasets can also be configured to store
789whole small files on the `special` device which can further improve the
790performance. Use fast SSDs for the `special` device.
68029ec8
FE
791
792IMPORTANT: The redundancy of the `special` device should match the one of the
793pool, since the `special` device is a point of failure for the whole pool.
794
795WARNING: Adding a `special` device to a pool cannot be undone!
796
797.Create a pool with `special` device and RAID-1:
798
eaefe614
FE
799----
800# zpool create -f -o ashift=12 <pool> mirror <device1> <device2> special mirror <device3> <device4>
801----
68029ec8
FE
802
803.Add a `special` device to an existing pool with RAID-1:
804
eaefe614
FE
805----
806# zpool add <pool> special mirror <device1> <device2>
807----
68029ec8
FE
808
809ZFS datasets expose the `special_small_blocks=<size>` property. `size` can be
810`0` to disable storing small file blocks on the `special` device or a power of
9deec2e2 811two in the range between `512B` to `1M`. After setting the property new file
68029ec8
FE
812blocks smaller than `size` will be allocated on the `special` device.
813
814IMPORTANT: If the value for `special_small_blocks` is greater than or equal to
51e544b6
TL
815the `recordsize` (default `128K`) of the dataset, *all* data will be written to
816the `special` device, so be careful!
68029ec8
FE
817
818Setting the `special_small_blocks` property on a pool will change the default
819value of that property for all child ZFS datasets (for example all containers
820in the pool will opt in for small file blocks).
821
51e544b6 822.Opt in for all file smaller than 4K-blocks pool-wide:
68029ec8 823
eaefe614
FE
824----
825# zfs set special_small_blocks=4K <pool>
826----
68029ec8
FE
827
828.Opt in for small file blocks for a single dataset:
829
eaefe614
FE
830----
831# zfs set special_small_blocks=4K <pool>/<filesystem>
832----
68029ec8
FE
833
834.Opt out from small file blocks for a single dataset:
835
eaefe614
FE
836----
837# zfs set special_small_blocks=0 <pool>/<filesystem>
838----
18d0d68e
SI
839
840[[sysadmin_zfs_features]]
841ZFS Pool Features
842~~~~~~~~~~~~~~~~~
843
844Changes to the on-disk format in ZFS are only made between major version changes
845and are specified through *features*. All features, as well as the general
846mechanism are well documented in the `zpool-features(5)` manpage.
847
848Since enabling new features can render a pool not importable by an older version
849of ZFS, this needs to be done actively by the administrator, by running
850`zpool upgrade` on the pool (see the `zpool-upgrade(8)` manpage).
851
852Unless you need to use one of the new features, there is no upside to enabling
853them.
854
855In fact, there are some downsides to enabling new features:
856
857* A system with root on ZFS, that still boots using `grub` will become
858 unbootable if a new feature is active on the rpool, due to the incompatible
859 implementation of ZFS in grub.
860* The system will not be able to import any upgraded pool when booted with an
861 older kernel, which still ships with the old ZFS modules.
862* Booting an older {pve} ISO to repair a non-booting system will likewise not
863 work.
864
27adc096
TL
865IMPORTANT: Do *not* upgrade your rpool if your system is still booted with
866`grub`, as this will render your system unbootable. This includes systems
867installed before {pve} 5.4, and systems booting with legacy BIOS boot (see
18d0d68e
SI
868xref:sysboot_determine_bootloader_used[how to determine the bootloader]).
869
27adc096 870.Enable new features for a ZFS pool:
18d0d68e
SI
871----
872# zpool upgrade <pool>
873----