2 .\" Copyright (c) 2012, 2018 by Delphix. All rights reserved.
3 .\" Copyright (c) 2013 by Saso Kiselkov. All rights reserved.
4 .\" Copyright (c) 2014, Joyent, Inc. All rights reserved.
5 .\" The contents of this file are subject to the terms of the Common Development
6 .\" and Distribution License (the "License"). You may not use this file except
7 .\" in compliance with the License. You can obtain a copy of the license at
8 .\" usr/src/OPENSOLARIS.LICENSE or http://www.opensolaris.org/os/licensing.
10 .\" See the License for the specific language governing permissions and
11 .\" limitations under the License. When distributing Covered Code, include this
12 .\" CDDL HEADER in each file and include the License file at
13 .\" usr/src/OPENSOLARIS.LICENSE. If applicable, add the following below this
14 .\" CDDL HEADER, with the fields enclosed by brackets "[]" replaced with your
15 .\" own identifying information:
16 .\" Portions Copyright [yyyy] [name of copyright owner]
17 .\" Copyright (c) 2019, Klara Inc.
18 .\" Copyright (c) 2019, Allan Jude
19 .\" Copyright (c) 2021, Colm Buckley <colm@tuatha.org>
27 .Nd description of ZFS pool features
30 ZFS pool on-disk format versions are specified via "features" which replace
31 the old on-disk format numbers (the last supported on-disk format number is 28).
32 To enable a feature on a pool use the
33 .Nm zpool Cm upgrade ,
35 .Sy feature Ns @ Ns Ar feature-name
39 .Sx Compatibility feature sets
40 section for information on how sets of features may be enabled together.
42 The pool format does not affect file system version compatibility or the ability
43 to send file systems between pools.
45 Since most features can be enabled independently of each other, the on-disk
46 format of the pool is specified by the set of all features marked as
49 If the pool was created by another software version
50 this set may include unsupported features.
52 .Ss Identifying features
53 Every feature has a GUID of the form
54 .Ar com.example : Ns Ar feature-name .
55 The reversed DNS name ensures that the feature's GUID is unique across all ZFS
57 When unsupported features are encountered on a pool they will
58 be identified by their GUIDs.
59 Refer to the documentation for the ZFS
60 implementation that created the pool for information about those features.
62 Each supported feature also has a short name.
63 By convention a feature's short name is the portion of its GUID which follows the
66 .Ar com.example : Ns Ar feature-name
67 would have the short name
69 however a feature's short name may differ across ZFS implementations if
70 following the convention would result in name conflicts.
73 Features can be in one of three states:
74 .Bl -tag -width "disabled"
76 This feature's on-disk format changes are in effect on the pool.
77 Support for this feature is required to import the pool in read-write mode.
78 If this feature is not read-only compatible,
79 support is also required to import the pool in read-only mode
80 .Pq see Sx Read-only compatibility .
82 An administrator has marked this feature as enabled on the pool, but the
83 feature's on-disk format changes have not been made yet.
84 The pool can still be imported by software that does not support this feature,
85 but changes may be made to the on-disk format at any time
86 which will move the feature to the
89 Some features may support returning to the
93 See feature-specific documentation for details.
95 This feature's on-disk format changes have not been made and will not be made
96 unless an administrator moves the feature to the
99 Features cannot be disabled once they have been enabled.
102 The state of supported features is exposed through pool properties of the form
103 .Sy feature Ns @ Ns Ar short-name .
105 .Ss Read-only compatibility
106 Some features may make on-disk format changes that do not interfere with other
107 software's ability to read from the pool.
108 These features are referred to as
109 .Dq read-only compatible .
110 If all unsupported features on a pool are read-only compatible,
111 the pool can be imported in read-only mode by setting the
113 property during import (see
115 for details on importing pools).
117 .Ss Unsupported features
118 For each unsupported feature enabled on an imported pool, a pool property
120 .Sy unsupported Ns @ Ns Ar feature-name
121 will indicate why the import was allowed despite the unsupported feature.
122 Possible values for this property are:
123 .Bl -tag -width "readonly"
125 The feature is in the
127 state and therefore the pool's on-disk
128 format is still compatible with software that does not support this feature.
130 The feature is read-only compatible and the pool has been imported in
134 .Ss Feature dependencies
135 Some features depend on other features being enabled in order to function.
136 Enabling a feature will automatically enable any features it depends on.
138 .Ss Compatibility feature sets
139 It is sometimes necessary for a pool to maintain compatibility with a
140 specific on-disk format, by enabling and disabling particular features.
143 feature facilitates this by allowing feature sets to be read from text files.
146 (the default), compatibility feature sets are disabled
147 (i.e. all features are enabled); when set to
149 no features are enabled.
150 When set to a comma-separated list of filenames
151 (each filename may either be an absolute path, or relative to
152 .Pa /etc/zfs/compatibility.d
154 .Pa /usr/share/zfs/compatibility.d ) ,
155 the lists of requested features are read from those files,
156 separated by whitespace and/or commas.
157 Only features present in all files are enabled.
159 Simple sanity checks are applied to the files:
160 they must be between 1B and 16kB in size, and must end with a newline character.
162 The requested features are applied when a pool is created using
163 .Nm zpool Cm create Fl o Sy compatibility Ns = Ns Ar …
164 and controls which features are enabled when using
165 .Nm zpool Cm upgrade .
167 will not show a warning about disabled features which are not part
168 of the requested feature set.
172 prevents any features from being enabled, either via
175 .Nm zpool Cm set Sy feature Ns @ Ns Ar feature-name Ns = Ns Sy enabled .
176 This setting also prevents pools from being upgraded to newer on-disk versions.
177 This is a safety measure to prevent new features from being
178 accidentally enabled, breaking compatibility.
180 By convention, compatibility files in
181 .Pa /usr/share/zfs/compatibility.d
182 are provided by the distribution, and include feature sets
183 supported by important versions of popular distributions, and feature
184 sets commonly supported at the start of each year.
185 Compatibility files in
186 .Pa /etc/zfs/compatibility.d ,
187 if present, will take precedence over files with the same name in
188 .Pa /usr/share/zfs/compatibility.d .
190 If an unrecognized feature is found in these files, an error message will
192 If the unrecognized feature is in a file in
193 .Pa /etc/zfs/compatibility.d ,
194 this is treated as an error and processing will stop.
195 If the unrecognized feature is under
196 .Pa /usr/share/zfs/compatibility.d ,
197 this is treated as a warning and processing will continue.
198 This difference is to allow distributions to include features
199 which might not be recognized by the currently-installed binaries.
201 Compatibility files may include comments:
204 to the end of the line is ignored.
207 .Bd -literal -compact -offset 4n
208 .No example# Nm cat Pa /usr/share/zfs/compatibility.d/grub2
209 # Features which are supported by GRUB2
222 .No example# Nm zpool Cm create Fl o Sy compatibility Ns = Ns Ar grub2 Ar bootpool Ar vdev
229 for more information on how these commands are affected by feature sets.
233 .Bl -tag -compact -width "READ-ONLY COMPATIBLE"
239 .if !"\\$5"" , \fB\\$5\fP\c
240 .if !"\\$6"" , \fB\\$6\fP\c
241 .if !"\\$7"" , \fB\\$7\fP\c
242 .if !"\\$8"" , \fB\\$8\fP\c
243 .if !"\\$9"" , \fB\\$9\fP\c
245 .It READ-ONLY COMPATIBLE
252 .No This feature becomes Sy active No as soon as it is enabled \
253 and will never return to being Sy enabled .
255 .ds remount-upgrade \
256 .No Each filesystem will be upgraded automatically when remounted, \
257 or when a new file is created under that filesystem. \
258 The upgrade can also be triggered on filesystems via \
259 Nm zfs Cm set Sy version Ns = Ns Sy current Ar fs . \
260 No The upgrade process runs in the background and may take a while to complete \
261 for filesystems containing large amounts of files.
268 the administrator can turn on the
270 checksum on any dataset using
271 .Nm zfs Cm set Sy checksum Ns = Ns Sy \\$1 Ar dset
272 .Po see Xr zfs-set 8 Pc .
277 property has been set to
279 and will return to being
281 once all filesystems that have ever had their checksum set to
287 The following features are supported on this system:
289 .feature org.zfsonlinux allocation_classes yes
290 This feature enables support for separate allocation classes.
294 when a dedicated allocation class vdev (dedup or special) is created with the
295 .Nm zpool Cm create No or Nm zpool Cm add No commands .
296 With device removal, it can be returned to the
298 state if all the dedicated allocation class vdevs are removed.
300 .feature com.delphix async_destroy yes
301 Destroying a file system requires traversing all of its data in order to
302 return its used space to the pool.
305 the file system is not fully removed until all space has been reclaimed.
306 If the destroy operation is interrupted by a reboot or power outage,
307 the next attempt to open the pool will need to complete the destroy
308 operation synchronously.
312 is enabled, the file system's data will be reclaimed by a background process,
313 allowing the destroy operation to complete
314 without traversing the entire file system.
315 The background process is able to resume
316 interrupted destroys after the pool has been opened, eliminating the need
317 to finish interrupted destroys as part of the open operation.
318 The amount of space remaining to be reclaimed by the background process
319 is available through the
329 .feature com.delphix bookmarks yes extensible_dataset
330 This feature enables use of the
336 while any bookmarks exist in the pool.
337 All bookmarks in the pool can be listed by running
338 .Nm zfs Cm list Fl t Sy bookmark Fl r Ar poolname .
340 .feature com.datto bookmark_v2 no bookmark extensible_dataset
341 This feature enables the creation and management of larger bookmarks which are
342 needed for other features in ZFS.
346 when a v2 bookmark is created and will be returned to the
348 state when all v2 bookmarks are destroyed.
350 .feature com.delphix bookmark_written no bookmark extensible_dataset bookmark_v2
351 This feature enables additional bookmark accounting fields, enabling the
352 .Sy written Ns # Ns Ar bookmark
353 property (space written since a bookmark) and estimates of
354 send stream sizes for incrementals from bookmarks.
358 when a bookmark is created and will be
361 state when all bookmarks with these fields are destroyed.
363 .feature org.openzfs device_rebuild yes
364 This feature enables the ability for the
368 commands to perform sequential reconstruction
369 (instead of healing reconstruction) when resilvering.
371 Sequential reconstruction resilvers a device in LBA order without immediately
372 verifying the checksums.
373 Once complete, a scrub is started, which then verifies the checksums.
374 This approach allows full redundancy to be restored to the pool
375 in the minimum amount of time.
376 This two-phase approach will take longer than a healing resilver
377 when the time to verify the checksums is included.
378 However, unless there is additional pool damage,
379 no checksum errors should be reported by the scrub.
380 This feature is incompatible with raidz configurations.
384 while a sequential resilver is in progress, and returns to
386 when the resilver completes.
388 .feature com.delphix device_removal no
389 This feature enables the
391 command to remove top-level vdevs,
392 evacuating them to reduce the total size of the pool.
399 on a top-level vdev, and will never return to being
402 .feature org.openzfs draid no
403 This feature enables use of the
406 dRAID is a variant of raidz which provides integrated distributed
407 hot spares that allow faster resilvering while retaining the benefits of raidz.
408 Data, parity, and spare space are organized in redundancy groups
409 and distributed evenly over all of the devices.
413 when creating a pool which uses the
415 vdev type, or when adding a new
417 vdev to an existing pool.
419 .feature org.illumos edonr no extensible_dataset
420 This feature enables the use of the Edon-R hash algorithm for checksum,
421 including for nopwrite (if compression is also enabled, an overwrite of
422 a block whose checksum matches the data being written will be ignored).
423 In an abundance of caution, Edon-R requires verification when used with
425 .Nm zfs Cm set Sy dedup Ns = Ns Sy edonr , Ns Sy verify
426 .Po see Xr zfs-set 8 Pc .
428 Edon-R is a very high-performance hash algorithm that was part
429 of the NIST SHA-3 competition.
430 It provides extremely high hash performance (over 350% faster than SHA-256),
431 but was not selected because of its unsuitability
432 as a general purpose secure hash algorithm.
433 This implementation utilizes the new salted checksumming functionality
434 in ZFS, which means that the checksum is pre-seeded with a secret
435 256-bit random key (stored on the pool) before being fed the data block
437 Thus the produced checksums are unique to a given pool,
438 preventing hash collision attacks on systems with dedup.
440 .checksum-spiel edonr
442 .Fx does not support the Sy edonr No feature.
444 .feature com.delphix embedded_data no
445 This feature improves the performance and compression ratio of
446 highly-compressible blocks.
447 Blocks whose contents can compress to 112 bytes
448 or smaller can take advantage of this feature.
450 When this feature is enabled, the contents of highly-compressible blocks are
451 stored in the block "pointer" itself (a misnomer in this case, as it contains
452 the compressed data, rather than a pointer to its location on disk).
453 Thus the space of the block (one sector, typically 512B or 4kB) is saved,
454 and no additional I/O is needed to read and write the data block.
458 .feature com.delphix empty_bpobj yes
459 This feature increases the performance of creating and using a large
460 number of snapshots of a single filesystem or volume, and also reduces
461 the disk space required.
463 When there are many snapshots, each snapshot uses many Block Pointer
464 Objects (bpobjs) to track blocks associated with that snapshot.
465 However, in common use cases, most of these bpobjs are empty.
466 This feature allows us to create each bpobj on-demand,
467 thus eliminating the empty bpobjs.
471 while there are any filesystems, volumes,
472 or snapshots which were created after enabling this feature.
474 .feature com.delphix enabled_txg yes
475 Once this feature is enabled, ZFS records the transaction group number
476 in which new features are enabled.
477 This has no user-visible impact, but other features may depend on this feature.
481 as soon as it is enabled and will
482 never return to being
485 .feature com.datto encryption no bookmark_v2 extensible_dataset
486 This feature enables the creation and management of natively encrypted datasets.
490 when an encrypted dataset is created and will be returned to the
492 state when all datasets that use this feature are destroyed.
494 .feature com.delphix extensible_dataset no
495 This feature allows more flexible use of internal ZFS data structures,
496 and exists for other features to depend on.
500 when the first dependent feature uses it, and will be returned to the
502 state when all datasets that use this feature are destroyed.
504 .feature com.joyent filesystem_limits yes extensible_dataset
505 This feature enables filesystem and snapshot limits.
506 These limits can be used to control how many filesystems and/or snapshots
507 can be created at the point in the tree on which the limits are set.
511 once either of the limit properties has been set on a dataset.
512 Once activated the feature is never deactivated.
514 .feature com.delphix hole_birth no enabled_txg
515 This feature has/had bugs, the result of which is that, if you do a
517 .Pq or Fl R , No since it uses Fl i
518 from an affected dataset, the receiving party will not see any checksum
519 or other errors, but the resulting destination snapshot
520 will not match the source.
523 has been disabled by default
524 .Pq see Sy send_holes_without_birth_time No in Xr zfs-module-parameters 5 .
526 This feature improves performance of incremental sends
527 .Pq Nm zfs Cm send Fl i
528 and receives for objects with many holes.
529 The most common case of hole-filled objects is zvols.
531 An incremental send stream from snapshot
532 .Sy A No to snapshot Sy B
533 contains information about every block that changed between
535 Blocks which did not change between those snapshots can be
536 identified and omitted from the stream using a piece of metadata called
537 the "block birth time", but birth times are not recorded for holes
538 (blocks filled only with zeroes).
539 Since holes created after
540 .Sy A No cannot be distinguished from holes created before Sy A ,
541 information about every hole in the entire filesystem or zvol
542 is included in the send stream.
544 For workloads where holes are rare this is not a problem.
545 However, when incrementally replicating filesystems or zvols with many holes
546 (for example a zvol formatted with another filesystem) a lot of time will
547 be spent sending and receiving unnecessary information about holes that
548 already exist on the receiving side.
552 feature has been enabled the block birth times
553 of all new holes will be recorded.
554 Incremental sends between snapshots created after this feature is enabled
555 will use this new metadata to avoid sending information about holes that
556 already exist on the receiving side.
560 .feature org.open-zfs large_blocks no extensible_dataset
561 This feature allows the record size on a dataset to be set larger than 128kB.
565 once a dataset contains a file with a block size larger than 128kB,
566 and will return to being
568 once all filesystems that have ever had their recordsize larger than 128kB
571 .feature org.zfsonlinux large_dnode no extensible_dataset
572 This feature allows the size of dnodes in a dataset to be set larger than 512B.
576 once a dataset contains an object with a dnode larger than 512B,
577 which occurs as a result of setting the
579 dataset property to a value other than
581 The feature will return to being
583 once all filesystems that have ever contained a dnode larger than 512B
585 Large dnodes allow more data to be stored in the bonus buffer,
586 thus potentially improving performance by avoiding the use of spill blocks.
588 .feature com.delphix livelist yes
589 This feature allows clones to be deleted faster than the traditional method
590 when a large number of random/sparse writes have been made to the clone.
591 All blocks allocated and freed after a clone is created are tracked by the
592 the clone's livelist which is referenced during the deletion of the clone.
593 The feature is activated when a clone is created and remains
595 until all clones have been destroyed.
597 .feature com.delphix log_spacemap yes com.delphix:spacemap_v2
598 This feature improves performance for heavily-fragmented pools,
599 especially when workloads are heavy in random-writes.
600 It does so by logging all the metaslab changes on a single spacemap every TXG
601 instead of scattering multiple writes to all the metaslab spacemaps.
605 .feature org.illumos lz4_compress no
607 is a high-performance real-time compression algorithm that
608 features significantly faster compression and decompression as well as a
609 higher compression ratio than the older
614 compression is approximately 50% faster on compressible data and 200% faster
615 on incompressible data than
617 It is also approximately 80% faster on decompression,
618 while giving approximately a 10% better compression ratio.
624 the administrator can turn on
626 compression on any dataset on the pool using the
629 All newly written metadata will be compressed with the
635 .feature com.joyent multi_vdev_crash_dump no
636 This feature allows a dump device to be configured with a pool comprised
638 Those vdevs may be arranged in any mirrored or raidz configuration.
641 .Sy multi_vdev_crash_dump
644 the administrator can use
646 to configure a dump device on a pool comprised of multiple vdevs.
650 and Linux this feature is unused, but registered for compatibility.
651 New pools created on these systems will have the feature
653 but will never transition to
655 as this functionality is not required for crash dump support.
656 Existing pools where this feature is
660 .feature com.delphix obsolete_counts yes device_removal
661 This feature is an enhancement of
663 which will over time reduce the memory used to track removed devices.
664 When indirect blocks are freed or remapped,
665 we note that their part of the indirect mapping is "obsolete" – no longer needed.
671 command is used on a top-level vdev, and will never return to being
674 .feature org.zfsonlinux project_quota yes extensible_dataset
675 This feature allows administrators to account the spaces and objects usage
676 information against the project identifier (ID).
678 The project ID is an object-based attribute.
679 When upgrading an existing filesystem,
680 objects without a project ID will be assigned a zero project ID.
681 When this feature is enabled, newly created objects inherit
682 their parent directories' project ID if the parent's inherit flag is set
683 .Pq via Nm chattr Sy [+-]P No or Nm zfs Cm project Fl s Ns | Ns Fl C .
684 Otherwise, the new object's project ID will be zero.
685 An object's project ID can be changed at any time by the owner
686 (or privileged user) via
687 .Nm chattr Fl p Ar prjid
689 .Nm zfs Cm project Fl p Ar prjid .
691 This feature will become
693 as soon as it is enabled and will never return to being
697 .feature com.delphix redaction_bookmarks no bookmarks extensible_dataset
698 This feature enables the use of redacted
699 .Nm zfs Cm send Ns s ,
700 which create redaction bookmarks storing the list of blocks
701 redacted by the send that created them.
702 For more information about redacted sends, see
705 .feature com.delphix redacted_datasets no extensible_dataset
706 This feature enables the receiving of redacted
708 streams. which create redacted datasets when received.
709 These datasets are missing some of their blocks,
710 and so cannot be safely mounted, and their contents cannot be safely read.
711 For more information about redacted receives, see
714 .feature com.datto resilver_defer yes
715 This feature allows ZFS to postpone new resilvers if an existing one is already
717 Without this feature, any new resilvers will cause the currently
718 running one to be immediately restarted from the beginning.
722 once a resilver has been deferred, and returns to being
724 when the deferred resilver begins.
726 .feature org.illumos sha512 no extensible_dataset
727 This feature enables the use of the SHA-512/256 truncated hash algorithm
728 (FIPS 180-4) for checksum and dedup.
729 The native 64-bit arithmetic of SHA-512 provides an approximate 50%
730 performance boost over SHA-256 on 64-bit hardware
731 and is thus a good minimum-change replacement candidate
732 for systems where hash performance is important,
733 but these systems cannot for whatever reason utilize the faster
734 .Sy skein No and Sy edonr
737 .checksum-spiel sha512
739 .feature org.illumos skein no extensible_dataset
740 This feature enables the use of the Skein hash algorithm for checksum and dedup.
741 Skein is a high-performance secure hash algorithm that was a
742 finalist in the NIST SHA-3 competition.
743 It provides a very high security margin and high performance on 64-bit hardware
744 (80% faster than SHA-256).
745 This implementation also utilizes the new salted checksumming
746 functionality in ZFS, which means that the checksum is pre-seeded with a
747 secret 256-bit random key (stored on the pool) before being fed the data
748 block to be checksummed.
749 Thus the produced checksums are unique to a given pool,
750 preventing hash collision attacks on systems with dedup.
752 .checksum-spiel skein
754 .feature com.delphix spacemap_histogram yes
755 This features allows ZFS to maintain more information about how free space
756 is organized within the pool.
759 it will be activated when a new space map object is created, or
760 an existing space map is upgraded to the new format,
761 and never returns back to being
764 .feature com.delphix spacemap_v2 yes
765 This feature enables the use of the new space map encoding which
766 consists of two words (instead of one) whenever it is advantageous.
767 The new encoding allows space maps to represent large regions of
768 space more efficiently on-disk while also increasing their maximum
775 and never returns back to being
778 .feature org.zfsonlinux userobj_accounting yes extensible_dataset
779 This feature allows administrators to account the object usage information
785 .feature com.delphix zpool_checkpoint yes
786 This feature enables the
787 .Nm zpool Cm checkpoint
788 command that can checkpoint the state of the pool
789 at the time it was issued and later rewind back to it or discard it.
794 .Nm zpool Cm checkpoint
795 command is used to checkpoint the pool.
796 The feature will only return back to being
798 when the pool is rewound or the checkpoint has been discarded.
800 .feature org.freebsd zstd_compress no extensible_dataset
802 is a high-performance compression algorithm that features a
803 combination of high compression ratios and high speed.
807 offers slightly better compression at much higher speeds.
811 offers much better compression while being only modestly slower.
814 compression speed ranges from 250 to 500 MB/s per thread
815 and decompression speed is over 1 GB/s per thread.
821 the administrator can turn on
823 compression of any dataset using
824 .Nm zfs Cm set Sy compress Ns = Ns Sy zstd Ar dset
825 .Po see Xr zfs-set 8 Pc .
830 property has been set to
832 and will return to being
834 once all filesystems that have ever had their
projects / mirror_zfs.git / blob