.\" Copyright 2011 Nexenta Systems, Inc. All rights reserved.
.\" Copyright (c) 2013 by Delphix. All rights reserved.
.\" Copyright (c) 2012 Cyril Plisko. All Rights Reserved.
+.\" Copyright (c) 2017 Datto Inc.
.\" The contents of this file are subject to the terms of the Common Development
.\" and Distribution License (the "License"). You may not use this file except
.\" in compliance with the License. You can obtain a copy of the license at
.\" CDDL HEADER, with the fields enclosed by brackets "[]" replaced with your
.\" own identifying information:
.\" Portions Copyright [yyyy] [name of copyright owner]
-.TH zpool 8 "May 11, 2016" "ZFS pool 28, filesystem 5" "System Administration Commands"
+.TH zpool 8 "April 12, 2017" "ZFS pool 28, filesystem 5" "System Administration Commands"
.SH NAME
zpool \- configures ZFS storage pools
.SH SYNOPSIS
.LP
.nf
-\fB\fBzpool iostat\fR [\fB-c\fR \fBCMD\fR] [\fB-T\fR \fBd\fR | \fBu\fR] [\fB-ghHLpPvy\fR] [\fB-lq\fR]|[\fB-r\fR|-\fBw\fR]]
+\fB\fBzpool iostat\fR [[[\fB-c\fR \fBSCRIPT\fR] [\fB-lq\fR]] | \fB-rw\fR] [\fB-T\fR \fBd\fR | \fBu\fR] [\fB-ghHLpPvy\fR]
[[\fIpool\fR ...]|[\fIpool vdev\fR ...]|[\fIvdev\fR ...]] [\fIinterval\fR[\fIcount\fR]]\fR
.fi
.LP
.nf
-\fBzpool offline\fR [\fB-t\fR] \fIpool\fR \fIdevice\fR ...
+\fBzpool offline\fR [\fB-f\fR] [\fB-t\fR] \fIpool\fR \fIdevice\fR ...
.fi
.LP
.LP
.nf
-\fBzpool status\fR [\fB-c\fR \fBCMD\fR] [\fB-gLPvxD\fR] [\fB-T\fR d | u] [\fIpool\fR] ... [\fIinterval\fR [\fIcount\fR]]
+\fBzpool status\fR [\fB-c\fR \fBSCRIPT\fR] [\fB-gLPvxD\fR] [\fB-T\fR d | u] [\fIpool\fR] ... [\fIinterval\fR [\fIcount\fR]]
+.fi
+
+.LP
+.nf
+\fBzpool sync\fR [\fBpool\fR] ...
.fi
.LP
.sp
.LP
All datasets within a storage pool share the same space. See \fBzfs\fR(8) for information on managing datasets.
-.SS "Virtual Devices (\fBvdev\fRs)"
+.SS "Virtual Devices (vdevs)"
.sp
.LP
A "virtual device" describes a single device or a collection of devices organized according to certain performance and fault characteristics. The following virtual devices are supported:
.LP
The space usage properties report actual physical space available to the storage pool. The physical space can be different from the total amount of space that any contained datasets can actually use. The amount of space used in a \fBraidz\fR configuration depends on the characteristics of the data being written. In addition, \fBZFS\fR reserves some space for internal accounting that the \fBzfs\fR(8) command takes into account, but the \fBzpool\fR command does not. For non-full pools of a reasonable size, these effects should be invisible. For small pools, or pools that are close to being completely full, these discrepancies may become more noticeable.
-.sp
-.LP
-The following property can be set at creation time:
-.sp
-.ne 2
-.na
-\fB\fBashift\fR=\fIashift\fR\fR
-.ad
-.sp .6
-.RS 4n
-Pool sector size exponent, to the power of 2 (internally referred to as "ashift"). Values from 9 to 13, inclusive, are valid; also, the special value 0 (the default) means to auto-detect using the kernel's block layer and a ZFS internal exception list. I/O operations will be aligned to the specified size boundaries. Additionally, the minimum (disk) write size will be set to the specified size, so this represents a space vs. performance trade-off. The typical case for setting this property is when performance is important and the underlying disks use 4KiB sectors but report 512B sectors to the OS (for compatibility reasons); in that case, set \fBashift=12\fR (which is 1<<12 = 4096).
-.LP
-For optimal performance, the pool sector size should be greater than or equal to the sector size of the underlying disks. Since the property cannot be changed after pool creation, if in a given pool, you \fIever\fR want to use drives that \fIreport\fR 4KiB sectors, you must set \fBashift=12\fR at pool creation time.
-.LP
-Keep in mind is that the \fBashift\fR is \fIvdev\fR specific and is not a \fIpool\fR global. This means that when adding new vdevs to an existing pool you may need to specify the \fBashift\fR.
-.RE
-
.sp
.LP
The following property can be set at creation time and import time:
.sp
.LP
The following properties can be set at creation time and import time, and later changed with the \fBzpool set\fR command:
+.sp
+.ne 2
+.na
+\fB\fBashift\fR=\fIashift\fR\fR
+.ad
+.sp .6
+.RS 4n
+Pool sector size exponent, to the power of 2 (internally referred to as "ashift"). Values from 9 to 16, inclusive, are valid; also, the special value 0 (the default) means to auto-detect using the kernel's block layer and a ZFS internal exception list. I/O operations will be aligned to the specified size boundaries. Additionally, the minimum (disk) write size will be set to the specified size, so this represents a space vs. performance trade-off. For optimal performance, the pool sector size should be greater than or equal to the sector size of the underlying disks. The typical case for setting this property is when performance is important and the underlying disks use 4KiB sectors but report 512B sectors to the OS (for compatibility reasons); in that case, set \fBashift=12\fR (which is 1<<12 = 4096).
+.LP
+When set, this property is used as the default hint value in \fIsubsequent\fR vdev operations (add, attach and replace). Changing this value will \fInot\fR modify any existing vdev, not even on disk replacement; however it can be used, for instance, to replace a dying 512B sectors disk with a newer 4KiB sectors device: this will probably result in bad performance but at the same time could prevent loss of data.
+.RE
+
.sp
.ne 2
.na
.sp
.ne 2
.na
-\fB\fBcachefile\fR=fBnone\fR | \fIpath\fR\fR
+\fB\fBcachefile\fR=\fBnone\fR | \fIpath\fR\fR
.ad
.sp .6
.RS 4n
.ad
.sp .6
.RS 4n
-Sets the given pool properties. See the "Properties" section for a list of valid properties that can be set. The only property supported at the moment is \fBashift\fR. \fBDo note\fR that some properties (among them \fBashift\fR) are \fInot\fR inherited from a previous vdev. They are vdev specific, not pool specific.
+Sets the given pool properties. See the "Properties" section for a list of valid properties that can be set. The only property supported at the moment is \fBashift\fR.
.RE
Do not add a disk that is currently configured as a quorum device to a zpool. After a disk is in the pool, that disk can then be configured as a quorum device.
.ad
.sp .6
.RS 4n
-Sets the given pool properties. See the "Properties" section for a list of valid properties that can be set. The only property supported at the moment is "ashift".
+Sets the given pool properties. See the "Properties" section for a list of valid properties that can be set. The only property supported at the moment is \fBashift\fR.
.RE
.RE
.sp
.ne 2
.na
-\fB\fBzpool iostat\fR [\fB-c\fR \fBCMD\fR] [\fB-T\fR \fBd\fR | \fBu\fR] [\fB-ghHLpPvy\fR] [[\fB-lq\fR]|[\fB-r\fR|\fB-w\fR]] [[\fIpool\fR ...]|[\fIpool vdev\fR ...]|[\fIvdev\fR ...]] [\fIinterval\fR[\fIcount\fR]]\fR
+\fB\fBzpool iostat\fR [[[\fB-c\fR \fBSCRIPT\fR] [\fB-lq\fR]] | \fB-rw\fR] [\fB-T\fR \fBd\fR | \fBu\fR] [\fB-ghHLpPvy\fR] [[\fIpool\fR ...]|[\fIpool vdev\fR ...]|[\fIvdev\fR ...]] [\fIinterval\fR[\fIcount\fR]]\fR
.ad
.sp .6
.sp
.ne 2
.na
-\fB\fB-c\fR \fBCMD\fR
+\fB\fB-c\fR \fB[SCRIPT1,SCRIPT2,...]\fR
.ad
.RS 12n
-Run a command on each vdev and include first line of output
+Run a script (or scripts) on each vdev and include the output in zpool iostat
+.sp
+The \fB-c\fR option allows you to run script(s) for each vdev and display the
+output in zpool iostat. For security reasons, a user can only execute scripts
+found in the /<etc>/zfs/zpool.d directory as an unprivileged user. However, a
+privileged user can run \fB-c\fR if they have the ZPOOL_SCRIPTS_AS_ROOT
+environment variable set. If a script requires the use of a privileged
+command (like smartctl) then it's recommended you allow the user access to it in
+/etc/sudoers. For example, to allow user "zfsuser" access to "smartctl -a", add
+the following to /etc/sudoers:
+
+zfsuser ALL=NOPASSWD: /usr/sbin/smartctl -a /dev/sd[a-z]*, NOEXEC: /usr/sbin/smartctl -a /dev/sd[a-z]*`
+
+If \fB-c\fR is passed without a script name, it prints a list of all scripts.
+\fB-c\fR also sets verbose mode (\fB-v\fR).
+
+Script output should be in the form of "name=value". The column name is
+set to "name" and the value is set to "value". Multiple lines can be used to
+output multiple columns. The first line of output not in the "name=value"
+format is displayed without a column title, and no more output after that is
+displayed. This can be useful for printing error messages. Blank or NULL
+values are printed as a '-' to make output awk-able.
+
+The following environment variables are set before running each script:
.sp
-The \fB-c\fR option allows you to run an arbitrary command on each vdev and
-display the first line of output in zpool iostat. The environment vars
-\fBVDEV_PATH\fR and \fBVDEV_UPATH\fR are set to the vdev's path and "underlying
-path" before running the command. For device mapper, multipath, or partitioned
-vdevs, \fBVDEV_UPATH\fR is the actual underlying /dev/sd* disk. This can be
-useful if the command you're running requires a /dev/sd* device. Commands run
-in parallel for each vdev for performance.
+\fB$VDEV_PATH\fR: Full path to the vdev.
+.LP
+\fB$VDEV_UPATH\fR: "Underlying path" to the vdev. For device mapper, multipath, or
+partitioned vdevs, \fBVDEV_UPATH\fR is the actual underlying /dev/sd* disk.
+This can be useful if the command you're running requires a /dev/sd* device.
+.LP
+\fB$VDEV_ENC_SYSFS_PATH\fR: The sysfs path to the vdev's enclosure LEDs (if any).
.RE
.sp
\fB\fB-o\fR \fIprops\fR\fR
.ad
.RS 12n
-Comma-separated list of properties to display. See the "Properties" section for a list of valid properties. The default list is "name, size, used, available, fragmentation, expandsize, capacity, dedupratio, health, altroot"
+Comma-separated list of properties to display. See the "Properties" section for a list of valid properties. The default list is "name, size, alloc, free, fragmentation, expandsize, capacity, dedupratio, health, altroot"
.RE
.sp
.sp
.ne 2
.na
-\fB\fBzpool offline\fR [\fB-t\fR] \fIpool\fR \fIdevice\fR ...\fR
+\fB\fBzpool offline\fR [\fB-f\fR] [\fB-t\fR] \fIpool\fR \fIdevice\fR ...\fR
.ad
.sp .6
.RS 4n
Takes the specified physical device offline. While the \fIdevice\fR is offline, no attempt is made to read or write to the device.
.sp
-This command is not applicable to spares or cache devices.
+.ne 2
+.na
+\fB\fB-f\fR\fR
+.ad
+.RS 6n
+Force fault. Instead of offlining the disk, put it into a faulted state. The
+fault will persist across imports unless the \fB-t\fR flag was specified.
+.RE
+
.sp
.ne 2
.na
.ad
.sp .6n
.RS 6n
-Sets the given pool properties. See the "Properties" section for a list of valid properties that can be set. The only property supported at the moment is \fBashift\fR. \fBDo note\fR that some properties (among them \fBashift\fR) are \fInot\fR inherited from a previous vdev. They are vdev specific, not pool specific.
+Sets the given pool properties. See the "Properties" section for a list of valid properties that can be set. The only property supported at the moment is \fBashift\fR.
.RE
.RE
.sp
.ne 2
.na
-\fBzpool status\fR [\fB-c\fR \fBCMD\fR] [\fB-gLPvxD\fR] [\fB-T\fR d | u] [\fIpool\fR] ... [\fIinterval\fR [\fIcount\fR]]
+\fBzpool status\fR [\fB-c\fR \fB[SCRIPT1,SCRIPT2,...] \fR] [\fB-gLPvxD\fR] [\fB-T\fR d | u] [\fIpool\fR] ... [\fIinterval\fR [\fIcount\fR]]
.ad
.sp .6
.RS 4n
.sp
.ne 2
.na
-\fB\fB-c\fR \fBCMD\fR
+\fB\fB-c\fR \fB[SCRIPT1,SCRIPT2,...]\fR
.ad
.RS 12n
-Run a command on each vdev and include first line of output
+Run a script (or scripts) on each vdev and include the output in zpool status
.sp
-The \fB-c\fR option allows you to run an arbitrary command on each vdev and
-display the first line of output in zpool status. The environment vars
-\fBVDEV_PATH\fR and \fBVDEV_UPATH\fR are set to the vdev's path and "underlying
-path" before running the command. For device mapper, multipath, or partitioned
-vdevs, \fBVDEV_UPATH\fR is the actual underlying /dev/sd* disk. This can be
-useful if the command you're running requires a /dev/sd* device. Commands run
-in parallel for each vdev for performance.
+The \fB-c\fR option allows you to run script(s) for each vdev and display the
+output in zpool iostat. For security reasons, a user can only execute scripts
+found in the /<etc>/zfs/zpool.d directory as an unprivileged user. However, a
+privileged user can run \fB-c\fR if they have the ZPOOL_SCRIPTS_AS_ROOT
+environment variable set. If a script requires the use of a privileged
+command (like smartctl) then it's recommended you allow the user access to it in
+/etc/sudoers. For example, to allow user "zfsuser" access to "smartctl -a", add
+the following to /etc/sudoers:
+
+zfsuser ALL=NOPASSWD: /usr/sbin/smartctl -a /dev/sd[a-z]*, NOEXEC: /usr/sbin/smartctl -a /dev/sd[a-z]*`
+
+If \fB-c\fR is passed without a script name, it prints a list of all scripts.
+
+Script output should be in the form of "name=value". The column name is
+set to "name" and the value is set to "value". Multiple lines can be used to
+output multiple columns. The first line of output not in the "name=value"
+format is displayed without a column title, and no more output after that is
+displayed. This can be useful for printing error messages. Blank or NULL
+values are printed as a '-' to make output awk-able.
+
+The following environment variables are set before running each command:
+.sp
+\fB$VDEV_PATH\fR: Full path to the vdev.
+.LP
+\fB$VDEV_UPATH\fR: "Underlying path" to the vdev. For device mapper, multipath, or
+partitioned vdevs, \fBVDEV_UPATH\fR is the actual underlying /dev/sd* disk.
+This can be useful if the command you're running requires a /dev/sd* device.
+.LP
+\fB$VDEV_ENC_SYSFS_PATH\fR: The sysfs path to the vdev's enclosure LEDs (if any).
.RE
.sp
.RE
+.sp
+.ne 2
+.na
+\fB\fBzpool sync\fR\fR [\fBpool\fR] ...
+.ad
+.sp .6
+.RS 4n
+This command forces all in-core dirty data to be written to the primary pool
+storage and not the ZIL. It will also update administrative information
+including quota reporting.
+Without arguments, \fBzpool sync\fR will sync all pools on the system.
+Otherwise, it will sync only the specified pool(s).
+.RE
+
.sp
.ne 2
.na
\fBExample 16 \fRRunning commands in zpool status and zpool iostat with -c
.sp
.LP
-Some examples of using the command (-c) option with zpool status and zpool
-iostat:
.sp
.in +2
.nf
-# \fBzpool status -c \[aq]echo I am $VDEV_PATH, $VDEV_UPATH\[aq]\fR
-NAME STATE READ WRITE CKSUM
-mypool ONLINE 0 0 0
+# zpool status -c vendor,model,size,enc
+...
+NAME STATE READ WRITE CKSUM vendor model size enc
+tank ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
- mpatha ONLINE 0 0 0 I am /dev/mapper/mpatha, /dev/sdc
- sdb ONLINE 0 0 0 I am /dev/sdb1, /dev/sdb
+ U1 ONLINE 0 0 0 SEAGATE ST8000NM0075 7.3T 0:0:0:0
+ U10 ONLINE 0 0 0 SEAGATE ST8000NM0075 7.3T 0:0:0:0
+ U11 ONLINE 0 0 0 SEAGATE ST8000NM0075 7.3T 0:0:0:0
+ U12 ONLINE 0 0 0 SEAGATE ST8000NM0075 7.3T 0:0:0:0
+ U13 ONLINE 0 0 0 SEAGATE ST8000NM0075 7.3T 0:0:0:0
+ U14 ONLINE 0 0 0 SEAGATE ST8000NM0075 7.3T 0:0:0:0
.fi
.in -2
.sp
.in +2
.nf
-# \fBzpool iostat -v -c \[aq]smartctl -a $VDEV_UPATH | grep "Current Drive Temperature"\[aq]\fR
-mypool 997M 7.25T 0 0 105K 106K
- mirror 997M 7.25T 0 0 105K 106K
- B0 - - 0 0 17.4K 15.2K Current Drive Temperature: 25 C
- B1 - - 0 0 17.4K 15.2K Current Drive Temperature: 24 C
- B2 - - 0 0 17.5K 15.2K Current Drive Temperature: 24 C
- B3 - - 0 0 0 15.1K Current Drive Temperature: 24 C
-logs - - - - - -
- B8 0 7.25T 0 0 1.14K 20.2K Current Drive Temperature: 23 C
+# zpool iostat -vc slaves,locate_led
+ capacity operations bandwidth
+pool alloc free read write read write slaves locate_led
+---------- ----- ----- ----- ----- ----- ----- --------- ----------
+tank 20.4G 7.23T 26 152 20.7M 21.6M
+ mirror 20.4G 7.23T 26 152 20.7M 21.6M
+ U1 - - 0 31 1.46K 20.6M sdb sdff 0
+ U10 - - 0 1 3.77K 13.3K sdas sdgw 0
+ U11 - - 0 1 288K 13.3K sdat sdgx 1
+ U12 - - 0 1 78.4K 13.3K sdau sdgy 0
+ U13 - - 0 1 128K 13.3K sdav sdgz 0
+ U14 - - 0 1 63.2K 13.3K sdfk sdg 0
.fi
.in -2
A pool can be stripped of any "devid" values on import or prevented from adding
them on \fBzpool create\fR or \fBzpool add\fR by setting ZFS_VDEV_DEVID_OPT_OUT.
+.TP
+.B "ZPOOL_SCRIPTS_AS_ROOT"
+Allow a privilaged user to run the \fBzpool status/iostat\fR with the \fB-c\fR
+option. Normally, only unprivilaged users are allowed to run \fB-c\fR.
.SH SEE ALSO
.sp