mirror_zfs.git
2 months agoLinux 5.1 compat: get_ds() removed master
Brian Behlendorf [Thu, 7 Mar 2019 22:44:23 +0000 (14:44 -0800)]
Linux 5.1 compat: get_ds() removed

Commit torvalds/linux@736706bee has removed the get_fs() function
as a bit of cleanup.  It has been defined as KERNEL_DS on all
architectures for all supported kernels.  Replace get_fs() with
KERNEL_DS as was done in the kernel.

Reviewed-by: Tom Caputi <tcaputi@datto.com>
Reviewed-by: Tony Hutter <hutter2@llnl.gov>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #8479

2 months agokernel_fpu fixes
Tony Hutter [Thu, 7 Mar 2019 00:03:03 +0000 (16:03 -0800)]
kernel_fpu fixes

This patch fixes a few issues when detecting which kernel_fpu functions
are available.

- Use kernel_fpu_begin() if it's exported on newer kernels.

- Use ZFS_LINUX_TRY_COMPILE_SYMBOL() to choose the right kernel_fpu
  function when using --enable-linux-builtin.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Tony Hutter <hutter2@llnl.gov>
Closes #8259
Closes #8363

2 months agoStack overflow in recursive bpobj_iterate_impl
Paul Zuchowski [Wed, 6 Mar 2019 17:50:55 +0000 (12:50 -0500)]
Stack overflow in recursive bpobj_iterate_impl

The function bpobj_iterate_impl overflows the stack when bpobjs
are deeply nested. Rewrite the function to eliminate the recursion.

Reviewed-by: Serapheim Dimitropoulos <serapheim@delphix.com>
Reviewed-by: Matt Ahrens <mahrens@delphix.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Paul Zuchowski <pzuchowski@datto.com>
Closes #7674
Closes #7675
Closes #7908

2 months agoFix race in vdev_initialize_thread
Brian Behlendorf [Wed, 6 Mar 2019 17:17:53 +0000 (09:17 -0800)]
Fix race in vdev_initialize_thread

Before allowing new allocations to the metaslab we need to ensure
that any issued initializing writes have been synced.  Otherwise,
it's possible for metaslab_block_alloc() to allocate a range which
is about to be overwritten by an initializing IO.

Serapheim Dimitropoulos <serapheim@delphix.com>
Reviewed-by: Richard Elling <Richard.Elling@RichardElling.com>
Reviewed-by: Tim Chase <tim@chase2k.com>
Reviewed-by: George Wilson <george.wilson@delphix.com>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #8461

2 months agoconfig: better libtirpc detection
Rafael Kitover [Sun, 3 Mar 2019 00:19:05 +0000 (16:19 -0800)]
config: better libtirpc detection

Improve the autoconf code for finding libtirpc and do not assume the
headers are in /usr/include/tirpc.

Also remove this assumption from the `rpc/xdr.h` header in libspl and
use the same `#include_next` mechanism that is used for other libspl
headers.

Include pkg.m4 from pkg-config in config/ for PKG_CHECK_MODULES(), the
file license allows this.

Include ax_save_flags.m4 and ax_restore_flags.m4 from autoconf-archive,
the file licenses are compatible. Use the 2012 versions so as not rely
on a more recent autoconf feature AS_VAR_COPY(), which breaks some build
slaves.

Add new macro library `config/find_system_library.m4` which defines the
FIND_SYSTEM_LIBRARY() macro which is a convenience wrapper over using
PKG_CHECK_MODULES() with a fallback to standard library locations and
some sanity checks.

The parameters are:

```
FIND_SYSTEM_LIBRARY(VARIABLE-PREFIX, MODULE, HEADER, HEADER-PREFIXES,
    LIBRARY, FUNCTIONS, [ACTION-IF-FOUND], [ACTION-IF-NOT-FOUND])
```

`HEADER-PREFIXES` and `FUNCTIONS` are comma-separated m4 lists.

For libtirpc we are using:

```
FIND_SYSTEM_LIBRARY(LIBTIRPC, [libtirpc], [rpc/xdr.h], [tirpc], [tirpc],
    [xdrmem_create], [], [...])
```

The headers are first checked for without the prefixes and then with.

This system works with pkg-config and falls back on checking standard
header/library locations, it can be easily overridden by the user by
setting the `PREFIX_CFLAGS` and `PREFIX_LIBS` variables which are
automatically added to the `./configure --help` output.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Rafael Kitover <rkitover@gmail.com>
Closes #7422
Closes #8313

2 months agoFix style of spl_kmem_cache_create()
Matthew Ahrens [Fri, 1 Mar 2019 01:57:47 +0000 (17:57 -0800)]
Fix style of spl_kmem_cache_create()

Fix indentation of code in ifdef's.
Remove obsolete comment.
Make if/else statements more readable by adding braces.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Tony Hutter <hutter2@llnl.gov>
Signed-off-by: Matthew Ahrens <mahrens@delphix.com>
Closes #8459

2 months agoDo not resume a pool if multihost is enabled
Olaf Faaland [Fri, 1 Mar 2019 01:56:19 +0000 (17:56 -0800)]
Do not resume a pool if multihost is enabled

When multihost is enabled, and a pool is suspended, return
EINVAL in response to "zpool clear <pool>".  The pool
may have been imported on another host while I/O was suspended.

Reviewed-by: loli10K <ezomori.nozomu@gmail.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Olaf Faaland <faaland1@llnl.gov>
Closes #6933
Closes #8460

2 months agoWarn user about accidentally sharing devices
Olaf Faaland [Fri, 1 Mar 2019 01:54:36 +0000 (17:54 -0800)]
Warn user about accidentally sharing devices

Improve the man page text to warn the user about the risk of adding
the same device to multiple pools via simultaneous "zpool create",
"zpool add", "zpool replace", etc.

State that MMP/multihost does not protect against these scenarios.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: Tony Hutter <hutter2@llnl.gov>
Signed-off-by: Olaf Faaland <faaland1@llnl.gov>
Closes #6473
Closes #8457

2 months agoabd_alloc should use scatter for >1K allocations
Matthew Ahrens [Fri, 1 Mar 2019 01:52:55 +0000 (17:52 -0800)]
abd_alloc should use scatter for >1K allocations

abd_alloc() normally does scatter allocations, thus solving the problem
that ABD originally set out to: the bulk of ZFS's allocations are single
pages, which are faster to allocate and free, and don't suffer from
internal fragmentation (and the inability to reclaim memory because some
buffers in the slab are still allocated).

However, the current code does linear allocations for 4KB and smaller
allocations, defeating the purpose of ABD.

Scatter ABD's use at least one page each, so sub-page allocations waste
some space when allocated as scatter (e.g. 2KB scatter allocation wastes
half of each page).  Using linear ABD's for small allocations means that
they will be put on slabs which contain many allocations.  This can
improve memory efficiency, but it also makes it much harder for ARC
evictions to actually free pages, because all the buffers on one slab
need to be freed in order for the slab (and underlying pages) to be
freed.  Typically, 512B and 1KB kmem caches have 16 buffers per slab, so
it's possible for them to actually waste more memory than scatter (one
page per buf = wasting 3/4 or 7/8th; one buf per slab = wasting
15/16th).

Spill blocks are typically 512B and are heavily used on systems running
selinux with the default dnode size and the `xattr=sa` property set.

By default we will use linear allocations for 512B and 1KB, and scatter
allocations for larger (1.5KB and up).

Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: DHE <git@dehacked.net>
Reviewed-by: Chunwei Chen <tuxoko@gmail.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Don Brady <don.brady@delphix.com>
Signed-off-by: Matthew Ahrens <mahrens@delphix.com>
Closes #8455

2 months agoRemove zfs-zed hard dep from zfs-share init script
beren12 [Thu, 28 Feb 2019 20:07:03 +0000 (15:07 -0500)]
Remove zfs-zed hard dep from zfs-share init script

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: bunder2015 <omfgbunder@gmail.com>
Signed-off-by: Chris Zubrzycki <github@mid-earth.net>
Closes #8447

2 months agoinitramfs/debian: use panic() instead of directly calling /bin/sh
Michael Niewöhner [Thu, 28 Feb 2019 20:05:55 +0000 (21:05 +0100)]
initramfs/debian: use panic() instead of directly calling /bin/sh

Debian has a panic() function which makes it possible to disable shell
access in initramfs by setting the panic kernel parameter. Use it.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Kash Pande <kash@tripleback.net>
Signed-off-by: Michael Niewöhner <foss@mniewoehner.de>
Closes #8448

2 months agozstreamdump: include embedded writes when dumping raw data (-d)
Allan Jude [Thu, 28 Feb 2019 01:55:25 +0000 (20:55 -0500)]
zstreamdump: include embedded writes when dumping raw data (-d)

When feeding a replication stream to `zstreamdump -d` (raw dump mode),
it does not print the raw data for DRR_WRITE_EMBEDDED records.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed by: Matt Ahrens <mahrens@delphix.com>
Signed-off-by: Allan Jude <allanjude@freebsd.org>
Closes #8430

2 months agoFix overly broad spa config lock
Brian Behlendorf [Wed, 27 Feb 2019 18:49:22 +0000 (10:49 -0800)]
Fix overly broad spa config lock

The spa_txg_history_init_io() and spa_txg_history_fini_io() were
mistakenly taking SCL_ALL when only SCL_CONFIG is required to
access the vdev stats.  This could result in a deadlock which
was observed when running ztest.

Reviewed-by: Olaf Faaland <faaland1@llnl.gov>
Reviewed-by: Tim Chase <tim@chase2k.com>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #8445

2 months agozfs.8 has wrong description of "zfs program -t"
Matthew Ahrens [Tue, 26 Feb 2019 19:15:28 +0000 (11:15 -0800)]
zfs.8 has wrong description of "zfs program -t"

The "-t" argument to "zfs program" specifies a limit on the number of
LUA instructions that can be executed.  The zfs.8 manpage has the wrong
description.  It should be updated to match what's in zfs-program.8

Also fix the formatting of the zfs help message.

Reviewed by: Allan Jude <allanjude@freebsd.org>
Reviewed-by: loli10K <ezomori.nozomu@gmail.com>
Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Matthew Ahrens <mahrens@delphix.com>
Closes #8410

2 months agoSort by full path name instead of by GUID when importing
kpande [Tue, 26 Feb 2019 19:13:15 +0000 (11:13 -0800)]
Sort by full path name instead of by GUID when importing

Preferentially sort by the full path name instead of GUID when determining
which device links to use.  This helps ensure that the pool vdevs are named
consistently when multiple links for a device appear in the same directory.
For example, the /dev/disk/by-id/scsi* and /dev/disk/by-id/wwn* links.

Reviewed-by: Alek Pinchuk <apinchuk@datto.com>
Reviewed-by: Richard Elling <Richard.Elling@RichardElling.com>
Authored-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Kash Pande <kash@tripleback.net>
Closes #8108
Closes #8440

2 months agoImprove error message for zfs create with @ or # in name
Damian Wojsław [Mon, 25 Feb 2019 19:20:07 +0000 (20:20 +0100)]
Improve error message for zfs create with @ or # in name

Reorder the `zfs create` error messages in order to return the most
specific one first.  If none of them apply then an expanded version of
the invalid name message is used.

Reviewed by: Tom Caputi <tcaputi@datto.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed by: Matt Ahrens <mahrens@delphix.com>
Signed-off-by: Damian Wojsław <damian@wojslaw.pl>
Closes #8155
Closes #8352

2 months agozfs(8): improve document of compression behaviours
DeHackEd [Mon, 25 Feb 2019 19:10:16 +0000 (14:10 -0500)]
zfs(8): improve document of compression behaviours

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed by: Allan Jude <allanjude@freebsd.org>
Reviewed-by: bunder2015 <omfgbunder@gmail.com>
Signed-off-by: DHE <git@dehacked.net>
Closes #4660
Closes #8423

2 months agoError path in metaslab_load_impl() forgets to drop ms_sync_lock
Serapheim Dimitropoulos [Mon, 25 Feb 2019 19:08:52 +0000 (11:08 -0800)]
Error path in metaslab_load_impl() forgets to drop ms_sync_lock

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Giuseppe Di Natale <guss80@gmail.com>
Reviewed-by: George Melikov <mail@gmelikov.ru>
Signed-off-by: Serapheim Dimitropoulos <serapheim@delphix.com>
Closes #8444

2 months agozvol: allow rename of in use ZVOL dataset
loli10K [Fri, 22 Feb 2019 23:38:42 +0000 (00:38 +0100)]
zvol: allow rename of in use ZVOL dataset

While ZFS allow renaming of in use ZVOLs at the DSL level without issues
the ZVOL layer does not correctly update the renamed dataset if the
device node is open (zv->zv_open_count > 0): trying to access the stale
dataset name, for instance during a zfs receive, will cause the
following failure:

VERIFY3(zv->zv_objset->os_dsl_dataset->ds_owner == zv) failed ((null) == ffff8800dbb6fc00)
PANIC at zvol.c:1255:zvol_resume()
Showing stack for process 1390
CPU: 0 PID: 1390 Comm: zfs Tainted: P           O  3.16.0-4-amd64 #1 Debian 3.16.51-3
Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
 0000000000000000 ffffffff8151ea00 ffffffffa0758a80 ffff88028aefba30
 ffffffffa0417219 ffff880037179220 ffffffff00000030 ffff88028aefba40
 ffff88028aefb9e0 2833594649524556 6f5f767a3e2d767a 6f3e2d7465736a62
Call Trace:
 [<0>] ? dump_stack+0x5d/0x78
 [<0>] ? spl_panic+0xc9/0x110 [spl]
 [<0>] ? mutex_lock+0xe/0x2a
 [<0>] ? zfs_refcount_remove_many+0x1ad/0x250 [zfs]
 [<0>] ? rrw_exit+0xc8/0x2e0 [zfs]
 [<0>] ? mutex_lock+0xe/0x2a
 [<0>] ? dmu_objset_from_ds+0x9a/0x250 [zfs]
 [<0>] ? dmu_objset_hold_flags+0x71/0xc0 [zfs]
 [<0>] ? zvol_resume+0x178/0x280 [zfs]
 [<0>] ? zfs_ioc_recv_impl+0x88b/0xf80 [zfs]
 [<0>] ? zfs_refcount_remove_many+0x1ad/0x250 [zfs]
 [<0>] ? zfs_ioc_recv+0x1c2/0x2a0 [zfs]
 [<0>] ? dmu_buf_get_user+0x13/0x20 [zfs]
 [<0>] ? __alloc_pages_nodemask+0x166/0xb50
 [<0>] ? zfsdev_ioctl+0x896/0x9c0 [zfs]
 [<0>] ? handle_mm_fault+0x464/0x1140
 [<0>] ? do_vfs_ioctl+0x2cf/0x4b0
 [<0>] ? __do_page_fault+0x177/0x410
 [<0>] ? SyS_ioctl+0x81/0xa0
 [<0>] ? async_page_fault+0x28/0x30
 [<0>] ? system_call_fast_compare_end+0x10/0x15

Reviewed by: Tom Caputi <tcaputi@datto.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: loli10K <ezomori.nozomu@gmail.com>
Closes #6263
Closes #8371

2 months agozpool reports 16E expandsize on disks with oddball number of sectors
loli10K [Fri, 22 Feb 2019 23:36:34 +0000 (00:36 +0100)]
zpool reports 16E expandsize on disks with oddball number of sectors

The issue is caused by a small discrepancy in how userland creates the
partition layout and the kernel estimates available space:

 * zpool command: subtract 9M from the usable device size, then align
   to 1M boundary. 9M is the sum of 1M "start" partition alignment + 8M
   EFI "reserved" partition.

 * kernel module: subtract 10M from the device size. 10M is the sum of
   1M "start" partition alignment + 1m "end" partition alignment + 8M
   EFI "reserved" partition.

For devices where the number of sectors is not a multiple of the
alignment size the zpool command will create a partition layout which
reserves less than 1M after the 8M EFI "reserved" partition:

  Disk /dev/sda: 1024 MiB, 1073739776 bytes, 2097148 sectors
  Units: sectors of 1 * 512 = 512 bytes
  Sector size (logical/physical): 512 bytes / 512 bytes
  I/O size (minimum/optimal): 512 bytes / 512 bytes
  Disklabel type: gpt
  Disk identifier: 49811D40-16F4-4E41-84A9-387703950D7F

  Device       Start     End Sectors  Size Type
  /dev/sda1     2048 2078719 2076672 1014M Solaris /usr & Apple ZFS
  /dev/sda9  2078720 2095103   16384    8M Solaris reserved 1

When the kernel module vdev_open() the device its max_asize ends up
being slightly smaller than asize: this results in a huge number (16E)
reported by metaslab_class_expandable_space().

This change prevents bdev_max_capacity() from returing a size smaller
than bdev_capacity().

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: George Wilson <george.wilson@delphix.com>
Reviewed by: Sara Hartse <sara.hartse@delphix.com>
Signed-off-by: loli10K <ezomori.nozomu@gmail.com>
Closes #1468
Closes #8391

2 months agoFix dnode_hold_impl() soft lockup
lidongyang [Fri, 22 Feb 2019 17:48:37 +0000 (04:48 +1100)]
Fix dnode_hold_impl() soft lockup

Soft lockups could happen when multiple threads trying
to get zrl on the same dnode handle in order to allocate
and initialize the dnode marked as DN_SLOT_ALLOCATED.

Don't loop from beginning when we can't get zrl, otherwise
we would increase the zrl refcount and nobody can actually
lock it.

Reviewed by: Tom Caputi <tcaputi@datto.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Li Dongyang <dongyangli@ddn.com>
Closes #8433

2 months agoClarify zpool iostat statistics reporting
kpande [Thu, 21 Feb 2019 22:00:48 +0000 (14:00 -0800)]
Clarify zpool iostat statistics reporting

Document expected behavior for  zpool iostat statistics reporting.

Reviewed by: Matt Ahrens <mahrens@delphix.com>
Reviewed-by: Alek Pinchuk <apinchuk@datto.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed by: Allan Jude <allanjude@freebsd.org>
Signed-off-by: Kash Pande <kash@tripleback.net>
Closes #2888
Closes #8417

2 months agoFix '-T u|d' descriptions in zpool(8)
Anatoly Borodin [Thu, 21 Feb 2019 19:22:06 +0000 (20:22 +0100)]
Fix '-T u|d' descriptions in zpool(8)

In

-T u|d  Display a time stamp.  Specify -u for a printed
representation of the internal representation of time.
See time(2).  Specify -d for standard date format.
See date(1).

'Specify u' and 'Specify d' should be used instead. `zpool list -T -u`
does not work.

Bring the descriptions in `zpool list` and `zpool status` in sync with
`zpool iostat`.

Reviewed by: Allan Jude <allanjude@freebsd.org>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Anatoly Borodin <anatoly.borodin@gmail.com>
Closes #8438

2 months agoDon't enter zvol's rangelock for read bio with size 0
Tomohiro Kusumi [Wed, 20 Feb 2019 18:14:36 +0000 (03:14 +0900)]
Don't enter zvol's rangelock for read bio with size 0

The SCST driver (SCSI target driver implementation) and possibly
others may issue read bio's with a length of zero bytes. Although
this is unusual, such bio's issued under certain condition can cause
kernel oops, due to how rangelock is implemented.

rangelock_add_reader() is not made to handle overlap of two (or more)
ranges from read bio's with the same offset when one of them has size
of 0, even though they conceptually overlap. Allowing them to enter
rangelock results in kernel oops by dereferencing invalid pointer,
or assertion failure on AVL tree manipulation with debug enabled
kernel module.

For example, this happens when read bio whose (offset, size) is
(0, 0) enters rangelock followed by another read bio with (0, 4096)
when (0, 0) rangelock is still locked, when there are no pending
write bio's. It can also happen with reverse order, which is (0, N)
followed by (0, 0) when (0, N) is still locked. More details
mentioned in #8379.

Kernel Oops on ->make_request_fn() of ZFS volume
https://github.com/zfsonlinux/zfs/issues/8379

Prevent this by returning bio with size 0 as success without entering
rangelock. This has been done for write bio after checking flusher
bio case (though not for the same reason), but not for read bio.

Reviewed-by: Alek Pinchuk <apinchuk@datto.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@osnexus.com>
Closes #8379
Closes #8401

2 months agoAdd diffutils dependency for dkms build
kpande [Wed, 20 Feb 2019 18:04:05 +0000 (10:04 -0800)]
Add diffutils dependency for dkms build

The cmp and diff utilities are required at configure time.  Add
a dependency on diffutils to ensure they are installed.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Kash Pande <kash@tripleback.net>
Closes #5205
Closes #8428

2 months agoIntroduce auxiliary metaslab histograms
Serapheim Dimitropoulos [Wed, 20 Feb 2019 17:59:57 +0000 (09:59 -0800)]
Introduce auxiliary metaslab histograms

This patch introduces 3 new histograms per metaslab. These
histograms track segments that have made it to the metaslab's
space map histogram (and are part of the spacemap) but have
not yet reached the ms_allocatable tree on loaded metaslab's
because these metaslab's are currently syncing and haven't
gone through metaslab_sync_done() yet.

The histograms help when we decide whether to load an unloaded
metaslab in-order to allocate from it. When calculating the
weight of an unloaded metaslab traditionally, we look at the
highest bucket of its spacemap's histogram.  The problem is
that we are not guaranteed to be able to allocated that
segment when we load the metaslab because it may still be at
the freeing, freed, or defer trees. The new histograms are
used when we try to calculate an unloaded metaslab's weight
to deal with this issue by removing segments that have would
not be in the allocatable tree at runtime. Note, that this
method of dealing with this is not completely accurate as
adjacent segments are not always consolidated in the space
map histogram of a metaslab.

In addition and to make things deterministic, we always reset
the weight of unloaded metaslabs based on their space map
weight (instead of doing that on a need basis). Thus, every
time a metaslab is loaded and its weight is reset again (from
the weight based on its space map to the one based on its
allocatable range tree) we expect (and assert) that this
change in weight can only get better if it doesn't stay the
same.

Reviewed by: Paul Dagnelie <pcd@delphix.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed by: Matt Ahrens <mahrens@delphix.com>
Signed-off-by: Serapheim Dimitropoulos <serapheim@delphix.com>
Closes #8358

3 months agoPrevent user accounting on readonly pool
loli10K [Wed, 20 Feb 2019 02:41:18 +0000 (03:41 +0100)]
Prevent user accounting on readonly pool

Trying to mount a dataset from a readonly pool could inadvertently start
the user accounting upgrade task, leading to the following failure:

VERIFY3(tx->tx_threads == 2) failed (0 == 2)
PANIC at txg.c:680:txg_wait_synced()
Showing stack for process 2541
CPU: 2 PID: 2541 Comm: z_upgrade Tainted: P           O  3.16.0-4-amd64 #1 Debian 3.16.51-3
Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
Call Trace:
 [<0>] ? dump_stack+0x5d/0x78
 [<0>] ? spl_panic+0xc9/0x110 [spl]
 [<0>] ? dnode_next_offset+0x1d4/0x2c0 [zfs]
 [<0>] ? dmu_object_next+0x77/0x130 [zfs]
 [<0>] ? dnode_rele_and_unlock+0x4d/0x120 [zfs]
 [<0>] ? txg_wait_synced+0x91/0x220 [zfs]
 [<0>] ? dmu_objset_id_quota_upgrade_cb+0x10f/0x140 [zfs]
 [<0>] ? dmu_objset_upgrade_task_cb+0xe3/0x170 [zfs]
 [<0>] ? taskq_thread+0x2cc/0x5d0 [spl]
 [<0>] ? wake_up_state+0x10/0x10
 [<0>] ? taskq_thread_should_stop.part.3+0x70/0x70 [spl]
 [<0>] ? kthread+0xbd/0xe0
 [<0>] ? kthread_create_on_node+0x180/0x180
 [<0>] ? ret_from_fork+0x58/0x90
 [<0>] ? kthread_create_on_node+0x180/0x180

This patch updates both functions responsible for checking if we can
perform user accounting to verify the pool is not readonly.

Reviewed-by: Alek Pinchuk <apinchuk@datto.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: loli10K <ezomori.nozomu@gmail.com>
Closes #8424

3 months agoAdd missing copyright notice to large_dnode tests
Ned Bass [Wed, 20 Feb 2019 02:39:10 +0000 (18:39 -0800)]
Add missing copyright notice to large_dnode tests

Missing copyright notices were noticed during the Illumos
RTI process. Add LLNS 2016 copyright based on original merge
date.

Reviewed-by: Giuseppe Di Natale <guss80@gmail.com>
Reviewed-by: Alek Pinchuk <apinchuk@datto.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Ned Bass <bass6@llnl.gov>
Closes #8435

3 months agoFix zdb crash
Igor K [Tue, 19 Feb 2019 19:15:22 +0000 (22:15 +0300)]
Fix zdb crash

We have to use umem_free() instead of free() if we are using
umem_zalloc()

Reviewed-by: Olaf Faaland <faaland1@llnl.gov>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Igor Kozhukhov <igor@dilos.org>
Closes #8402

3 months agoZTS: user_property_002_pos fails to destroy volume
John Wren Kennedy [Tue, 19 Feb 2019 19:12:47 +0000 (12:12 -0700)]
ZTS: user_property_002_pos fails to destroy volume

During the cleanup function of this test, an attempt to destroy a volume
can fail because the volume is busy. This leaves the system with
unexpected datasets which in turn causes subsequent failures.

Reviewed-by: bunder2015 <omfgbunder@gmail.com>
Reviewed-by: Igor Kozhukhov <igor@dilos.org>
Reviewed-by: Giuseppe Di Natale <guss80@gmail.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: John Kennedy <john.kennedy@delphix.com>
Closes #8422

3 months agozfs mount man page should document legacy behaviour
kpande [Tue, 19 Feb 2019 19:10:57 +0000 (11:10 -0800)]
zfs mount man page should document legacy behaviour

Document legacy mount behavior.

Reviewed by: Allan Jude <allanjude@freebsd.org>
Reviewed by: Matt Ahrens <mahrens@delphix.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: bunder2015 <omfgbunder@gmail.com>
Reviewed-by: loli10K <ezomori.nozomu@gmail.com>
Signed-off-by: Kash Pande <kash@tripleback.net>
Closes #2900
Closes #8414

3 months agoDelay injection can cause indefinitely hung zios
Sara Hartse [Fri, 15 Feb 2019 22:44:56 +0000 (14:44 -0800)]
Delay injection can cause indefinitely hung zios

If we hit the (NSEC_TO_TICK(diff) == 0) condition in
zio_delay_interrupt, zio_interrupt is never called and the
zio does not progress.

Reviewed by: Matt Ahrens <mahrens@delphix.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: George Melikov <mail@gmelikov.ru>
Signed-off-by: sara hartse <sara.hartse@delphix.com>
Closes #8404

3 months agoZFS mounted NFSv3 shares fail lock reclaims
Don Brady [Fri, 15 Feb 2019 22:40:16 +0000 (15:40 -0700)]
ZFS mounted NFSv3 shares fail lock reclaims

ZFS NFS shares mounted on a client with NFSv3 and with open
locks will fail to reclaim those locks after a server reboot.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: George Wilson <george.wilson@delphix.com>
Reviewed-by: George Melikov <mail@gmelikov.ru>
Signed-off-by: Don Brady <don.brady@delphix.com>
Closes #8398

3 months agoZTS: clone_001_pos fails in cleanup on busy dataset
John Wren Kennedy [Fri, 15 Feb 2019 20:45:46 +0000 (13:45 -0700)]
ZTS: clone_001_pos fails in cleanup on busy dataset

The "cleanup_all" function in this test calls "zfs destroy" which
fails approximately 30% of the time in our environment due to the
dataset being busy. Since the failure happens during cleanup, the
error is propagated to subsequent tests.

Tested by running the snapshot test group in a loop without seeing
any failures.

Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: Igor Kozhukhov <igor@dilos.org>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: John Kennedy <john.kennedy@delphix.com>
Closes #8409

3 months agozio_deadman_impl() fix and enhancement
Tim Chase [Fri, 15 Feb 2019 20:44:24 +0000 (14:44 -0600)]
zio_deadman_impl() fix and enhancement

Add the zio_deadman_log_all tunable to print all zios in
zio_deadman_impl().  Also, in all cases, display the depth of the
zio relative to the original parent zio.  This is meant to be used by
developers to gain diagnostic information for hangs which don't involve
fully set-up zio trees or are otherwise stuck or hung in an early stage.

Reviewed-by: Olaf Faaland <faaland1@llnl.gov>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: loli10K <ezomori.nozomu@gmail.com>
Signed-off-by: Tim Chase <tim@chase2k.com>
Closes #8362

3 months agozfs should optionally send holds
Paul Zuchowski [Fri, 15 Feb 2019 20:41:38 +0000 (15:41 -0500)]
zfs should optionally send holds

Add -h switch to zfs send command to send dataset holds. If
holds are present in the stream, zfs receive will create them
on the target dataset, unless the zfs receive -h option is used
to skip receive of holds.

Reviewed-by: Alek Pinchuk <apinchuk@datto.com>
Reviewed-by: loli10K <ezomori.nozomu@gmail.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed by: Paul Dagnelie <pcd@delphix.com>
Signed-off-by: Paul Zuchowski <pzuchowski@datto.com>
Closes #7513

3 months agoLinux 4.20 compat: Fix VERIFY(RW_READ_HELD(&hash->mh_contents))
Tony Hutter [Fri, 15 Feb 2019 20:37:20 +0000 (12:37 -0800)]
Linux 4.20 compat: Fix VERIFY(RW_READ_HELD(&hash->mh_contents))

The 4.20 kernel changed the meaning of the rw_semaphore.owner bits,
causing an assertion when loading the module under the 4.20 kernel.
This patch fixes the issue.

Reviewed-by: Chunwei Chen <tuxoko@gmail.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Tony Hutter <hutter2@llnl.gov>
Closes #8360
Closes #8389

3 months agoFix obsolete comment on rangelock
Tomohiro Kusumi [Thu, 14 Feb 2019 23:13:58 +0000 (08:13 +0900)]
Fix obsolete comment on rangelock

5d43cc9a59 renamed it to rangelock_enter().

Reviewed-by: Matt Ahrens <mahrens@delphix.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@osnexus.com>
Closes #8408

3 months agozdb: replace label_t to zdb_label_t for reduce collisions
Igor K [Wed, 13 Feb 2019 19:28:36 +0000 (22:28 +0300)]
zdb: replace label_t to zdb_label_t for reduce collisions

with builds on illumos based platform we can see build issue
because label_t has been redefined.

for reduce build issues on others platforms we should rename
label_t to zdb_label_t.

Reviewed-by: loli10K <ezomori.nozomu@gmail.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: George Melikov <mail@gmelikov.ru>
Signed-off-by: Igor Kozhukhov <igor@dilos.org>
Closes #8397

3 months agoFreeing throttle should account for holes
Alek P [Tue, 12 Feb 2019 20:01:08 +0000 (12:01 -0800)]
Freeing throttle should account for holes

Deletion throttle currently does not account for holes in a file.
This means that it can activate when it shouldn't.
To fix it we switch the throttle to be based on the number of
L1 blocks we will have to dirty when freeing

Reviewed by: Tom Caputi <tcaputi@datto.com>
Reviewed by: Matt Ahrens <mahrens@delphix.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Alek Pinchuk <apinchuk@datto.com>
Closes #7725
Closes #7888

3 months agoport async unlinked drain from illumos-nexenta
Alek P [Tue, 12 Feb 2019 18:41:15 +0000 (10:41 -0800)]
port async unlinked drain from illumos-nexenta

This patch is an async implementation of the existing sync
zfs_unlinked_drain() function. This function is called at mount time and
is responsible for freeing znodes that we didn't get to freeing before.
We don't have to hold mounting of the dataset until the unlinked list is
fully drained as is done now. Since we can process the unlinked set
asynchronously this results in a better user experience when mounting a
dataset with entries in the unlinked set.

Reviewed by: Jorgen Lundman <lundman@lundman.net>
Reviewed by: Tom Caputi <tcaputi@datto.com>
Reviewed by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Matt Ahrens <mahrens@delphix.com>
Reviewed by: Paul Dagnelie <pcd@delphix.com>
Signed-off-by: Alek Pinchuk <apinchuk@datto.com>
Closes #8142

3 months agoGet rid of space_map_update() for ms_synced_length
Serapheim Dimitropoulos [Tue, 12 Feb 2019 18:38:11 +0000 (10:38 -0800)]
Get rid of space_map_update() for ms_synced_length

Initially, metaslabs and space maps used to be the same thing
in ZFS. Later, we started differentiating them by referring
to the space map as the on-disk state of the metaslab, making
the metaslab a higher-level concept that is metadata that deals
with space accounting. Today we've managed to split that code
furthermore, with the space map being its own on-disk data
structure used in areas of ZFS besides metaslabs (e.g. the
vdev-wide space maps used for zpool checkpoint or vdev removal
features).

This patch refactors the space map code to further split the
space map code from the metaslab code. It does so by getting
rid of the idea that the space map can have a different in-core
and on-disk length (sm_length vs smp_length) which is something
that is only used for the metaslab code, and other consumers
of space maps just have to deal with. Instead, this patch
introduces changes that move the old in-core length of the
metaslab's space map to the metaslab structure itself (see
ms_synced_length field) while making the space map code only
care about the actual space map's length on-disk.

The result of this is that space map consumers no longer have
to deal with syncing two different lengths for the same
structure (e.g. space_map_update() goes away) while metaslab
specific behavior stays within the metaslab code. Specifically,
the ms_synced_length field keeps track of the amount of data
metaslab_load() can read from the metaslab's space map while
working concurrently with metaslab_sync() that may be
appending to that same space map.

As a side note, the patch also adds a few comments around
the metaslab code documenting some assumptions and expected
behavior.

Reviewed-by: Matt Ahrens <mahrens@delphix.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed by: Pavel Zakharov <pavel.zakharov@delphix.com>
Signed-off-by: Serapheim Dimitropoulos <serapheim@delphix.com>
Closes #8328

3 months agoZVOLs should not be allowed to have children
loli10K [Fri, 8 Feb 2019 23:44:15 +0000 (00:44 +0100)]
ZVOLs should not be allowed to have children

zfs create, receive and rename can bypass this hierarchy rule. Update
both userland and kernel module to prevent this issue and use pyzfs
unit tests to exercise the ioctls directly.

Note: this commit slightly changes zfs_ioc_create() ABI. This allow to
differentiate a generic error (EINVAL) from the specific case where we
tried to create a dataset below a ZVOL (ZFS_ERR_WRONG_PARENT).

Reviewed-by: Paul Dagnelie <pcd@delphix.com>
Reviewed-by: Matt Ahrens <mahrens@delphix.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Tom Caputi <tcaputi@datto.com>
Signed-off-by: loli10K <ezomori.nozomu@gmail.com>
3 months agoPool allocation classes misplacing small file blocks
loli10K [Fri, 8 Feb 2019 20:32:12 +0000 (21:32 +0100)]
Pool allocation classes misplacing small file blocks

Due to an off-by-one condition in spa_preferred_class() we are picking
the "normal" allocation class instead of the "special" one for file
blocks with size equal to the special_small_blocks property value.

This change fix the small code issue, update the ZFS Test Suite and the
zfs(8) man page.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Don Brady <don.brady@delphix.com>
Signed-off-by: loli10K <ezomori.nozomu@gmail.com>
Closes #8351
Closes #8361

3 months agoFix ARC stats for embedded blkptrs
Tim Chase [Mon, 4 Feb 2019 17:33:30 +0000 (11:33 -0600)]
Fix ARC stats for embedded blkptrs

Re-factor arc_read() to better account for embedded data blkptrs.
Previously, reading the payload from an embedded blkptr would cause
arcstats such as demand_metadata_misses to be bumped when there was
actually no cache "miss" because the data are already available in
the blkptr.

The following test procedure was used to demonstrate the problem:

   zpool create tank ...
   zfs create -o compression=lz4 tank/fs
   echo blah > /tank/fs/blah
   stat /tank/fs/blah
   grep 'meta.*mis' /proc/spl/kstat/zfs/arcstats

and repeating the last two steps to watch the metadata miss counter
increment.  This can also be demonstrated via the  zfs_arc_miss DTRACE4
probe in arc_read().

Reviewed-by: loli10K <ezomori.nozomu@gmail.com>
Reviewed-by: George Wilson <george.wilson@delphix.com>
Reviewed-by: Matt Ahrens <mahrens@delphix.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: George Melikov <mail@gmelikov.ru>
Signed-off-by: Tim Chase <tim@chase2k.com>
Closes #8319

3 months agoOpenZFS 9185 - Enable testing over NFS in ZFS performance tests
Ahmed Ghanem [Sun, 11 Feb 2018 23:11:59 +0000 (16:11 -0700)]
OpenZFS 9185 - Enable testing over NFS in ZFS performance tests

This change makes additions to the ZFS test suite that allows the
performance tests to run over NFS. The test is run and performance data
collected from the server side, while IO is generated on the NFS client.

This has been tested with Linux and illumos NFS clients.

Authored by: Ahmed Ghanem <ahmedg@delphix.com>
Reviewed by: Dan Kimmel <dan.kimmel@delphix.com>
Reviewed by: John Kennedy <john.kennedy@delphix.com>
Reviewed by: Kevin Greene <kevin.greene@delphix.com>
Reviewed-by: Richard Elling <Richard.Elling@RichardElling.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Ported-by: John Kennedy <john.kennedy@delphix.com>
Signed-off-by: John Kennedy <john.kennedy@delphix.com>
OpenZFS-issue: https://www.illumos.org/issues/9185
Closes #8367

3 months agozstreamdump: -d option is not documented in manpage
loli10K [Mon, 4 Feb 2019 17:13:00 +0000 (18:13 +0100)]
zstreamdump: -d option is not documented in manpage

This change simply documents the missing -d (dump contents) option in
zstreamdump(8).

Reviewed-by: bunder2015 <omfgbunder@gmail.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Igor Kozhukhov <igor@dilos.org>
Reviewed-by: George Melikov <mail@gmelikov.ru>
Signed-off-by: loli10K <ezomori.nozomu@gmail.com>
Closes #8369

3 months agoshellcheck pass
bunder2015 [Mon, 4 Feb 2019 17:07:19 +0000 (12:07 -0500)]
shellcheck pass

note: which is non-standard. Use builtin 'command -v' instead. [SC2230]
note: Use -n instead of ! -z. [SC2236]

Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: Giuseppe Di Natale <guss80@gmail.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: bunder2015 <omfgbunder@gmail.com>
Closes #8367

3 months agoflake8 pass
bunder2015 [Mon, 4 Feb 2019 17:02:46 +0000 (12:02 -0500)]
flake8 pass

F632 use ==/!= to compare str, bytes, and int literals

Reviewed-by: Håkan Johansson <f96hajo@chalmers.se>
Reviewed-by: Giuseppe Di Natale <guss80@gmail.com>
Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: bunder2015 <omfgbunder@gmail.com>
Closes #8368

3 months agoFix zpool iostat -w header names
Tony Hutter [Thu, 31 Jan 2019 18:51:18 +0000 (10:51 -0800)]
Fix zpool iostat -w header names

The zpool iostat latency histograms (-w) has column names
'sync_queue' and 'async_queue', which do not match the man page, nor
the equivalent columns in average latency.  Change the column
names to be 'syncq_wait' and 'asyncq_wait' to be consistent.

Reviewed-by: Olaf Faaland <faaland1@llnl.gov>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Tony Hutter <hutter2@llnl.gov>
Closes #8338

3 months agoSimplify log vdev removal code
Serapheim Dimitropoulos [Thu, 31 Jan 2019 17:17:52 +0000 (09:17 -0800)]
Simplify log vdev removal code

Get rid of the majority metaslab metadata when removing log vdevs
in spa_vdev_remove_log() with a call to metaslab_fini() instead
of duplicating a lot of that in vdev_remove_empty_log().

Reviewed-by: Matt Ahrens <mahrens@delphix.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Serapheim Dimitropoulos <serapheim@delphix.com>
Closes #8347

3 months agovs_alloc can underflow in L2ARC vdevs
Serapheim Dimitropoulos [Thu, 31 Jan 2019 17:16:39 +0000 (09:16 -0800)]
vs_alloc can underflow in L2ARC vdevs

The current L2 ARC device code consistently uses psize to
increment vs_alloc but varies between psize and lsize when
decrementing it. The result of this behavior is that
vs_alloc can be decremented more that it is incremented
and underflow. This patch changes the code so asize is
used anywhere.

In addition, it ensures that vs_alloc gets incremented by
the L2 ARC device code as buffers are written and not at
the end of the l2arc_write_buffers() routine. The latter
(and old) way would temporarily underflow vs_alloc as
buffers that were just written, would be destroyed while
l2arc_write_buffers() was still looping.

Reviewed-by: Matt Ahrens <mahrens@delphix.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Serapheim Dimitropoulos <serapheim@delphix.com>
Closes #8298

3 months agoDon't acquire zthr_request_lock in zthr_wakeup
Sara Hartse [Wed, 30 Jan 2019 20:31:16 +0000 (12:31 -0800)]
Don't acquire zthr_request_lock in zthr_wakeup

Address a deadlock caused by simultaneous wakeup and cancel on a zthr
by remove the hold of zthr_request_lock from zthr_wakeup. This
allows thr_wakeup to not block a thread that is in the process of
being cancelled.

Reviewed-by: Matt Ahrens <mahrens@delphix.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Serapheim Dimitropoulos <serapheim@delphix.com>
Signed-off-by: Sara Hartse <sara.hartse@delphix.com>
Closes #8333

3 months agozdb -L should skip leak detection altogether
Serapheim Dimitropoulos [Wed, 30 Jan 2019 17:54:27 +0000 (09:54 -0800)]
zdb -L should skip leak detection altogether

Currently the point of -L option in zdb is to  disable leak
tracing and the loading of space maps because they are expensive,
yet still do leak detection in terms of space. Unfortunately,
there is a scenario where this is a lie. If we are using zdb -L
on a pool where a vdev is being removed, zdb_claim_removing()
will open the metaslab space maps of that device.

This patch makes it so zdb -L skips leak detection altogether
and ensures that no space maps are loaded.

Reviewed-by: Matt Ahrens <mahrens@delphix.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Serapheim Dimitropoulos <serapheim@delphix.com>
Closes #8335

3 months agoExclude test-runner.py from the rpmbuild shebang check
Tony Hutter [Wed, 23 Jan 2019 19:31:24 +0000 (11:31 -0800)]
Exclude test-runner.py from the rpmbuild shebang check

Exclude test-runner.py from the rpmbuild shebang check to allow it to
run under Python 2 and 3.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Tony Hutter <hutter2@llnl.gov>
Closes #8331

3 months agoGCC 9.0: Fix ztest "directive argument is not a nul-terminated string"
Tony Hutter [Wed, 23 Jan 2019 19:17:52 +0000 (11:17 -0800)]
GCC 9.0: Fix ztest "directive argument is not a nul-terminated string"

GCC 9.0 is complaining because we're trying to print strings that
are defined like this:

.zo_pool = { 'z', 't', 'e', 's', 't', '\0' },

Fix them by making them actual strings.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Tony Hutter <hutter2@llnl.gov>
Closes #8330

3 months agoLinux 5.0 compat: Fix bio_set_dev()
Brian Behlendorf [Wed, 16 Jan 2019 18:39:19 +0000 (10:39 -0800)]
Linux 5.0 compat: Fix bio_set_dev()

The Linux 5.0 kernel updated the bio_set_dev() macro so it calls the
GPL-only bio_associate_blkg() symbol thus inadvertently converting
the entire macro.  Provide a minimal version which always assigns the
request queue's root_blkg to the bio.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #8287

3 months agoLinux 5.0 compat: Disable vector instructions on 5.0+ kernels
Tony Hutter [Sat, 12 Jan 2019 02:01:28 +0000 (18:01 -0800)]
Linux 5.0 compat: Disable vector instructions on 5.0+ kernels

The 5.0 kernel no longer exports the functions we need to do vector
(SSE/SSE2/SSE3/AVX...) instructions.  Disable vector-based checksum
algorithms when building against those kernels.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Tony Hutter <hutter2@llnl.gov>
Closes #8259

3 months agoLinux 5.0 compat: Fix SUBDIRs
Tony Hutter [Fri, 11 Jan 2019 01:07:05 +0000 (17:07 -0800)]
Linux 5.0 compat: Fix SUBDIRs

SUBDIRs has been deprecated for a long time, and was finally removed in
the 5.0 kernel.  Use "M=" instead.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Tony Hutter <hutter2@llnl.gov>
Closes #8257

3 months agoLinux 5.0 compat: Convert MS_* macros to SB_*
Tony Hutter [Thu, 10 Jan 2019 23:28:44 +0000 (15:28 -0800)]
Linux 5.0 compat: Convert MS_* macros to SB_*

In the 5.0 kernel, only the mount namespace code should use the MS_*
macos. Filesystems should use the SB_* ones.

https://patchwork.kernel.org/patch/10552493/

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Tony Hutter <hutter2@llnl.gov>
Closes #8264

3 months agoLinux 5.0 compat: Use totalram_pages()
Tony Hutter [Thu, 10 Jan 2019 22:28:10 +0000 (14:28 -0800)]
Linux 5.0 compat: Use totalram_pages()

totalram_pages() was converted to an atomic variable in 5.0:

https://patchwork.kernel.org/patch/10652795/

Its value should now be read though the totalram_pages() helper
function.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Tony Hutter <hutter2@llnl.gov>
Closes #8263

3 months agoLinux 5.0 compat: access_ok() drops 'type' parameter
Tony Hutter [Thu, 10 Jan 2019 19:03:40 +0000 (11:03 -0800)]
Linux 5.0 compat: access_ok() drops 'type' parameter

access_ok no longer needs a 'type' parameter in the 5.0 kernel.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Tony Hutter <hutter2@llnl.gov>
Closes #8261

3 months agoLinux 4.18 compat: Use ktime_get_coarse_real_ts64()
Tony Hutter [Wed, 9 Jan 2019 21:16:39 +0000 (13:16 -0800)]
Linux 4.18 compat: Use ktime_get_coarse_real_ts64()

Newer kernels remove current_kernel_time64().  Use
ktime_get_coarse_real_ts64() in its place.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Tony Hutter <hutter2@llnl.gov>
Closes #8258

3 months agoChange target size of metaslabs from 256GB to 16GB
Serapheim Dimitropoulos [Sat, 26 Jan 2019 00:38:27 +0000 (16:38 -0800)]
Change target size of metaslabs from 256GB to 16GB

= Old behavior

For vdev sizes 100GB to 50TB we keep ~200 metaslabs per
vdev and the metaslab size grows from 512MB to 256GB.
For vdev's bigger than that we start increasing the
number of metaslabs until we hit the 128K limit.

= New Behavior

For vdev sizes 100GB to 3TB we keep ~200 metaslabs per
vdev and the metaslab size grows from 512MB to 16GB.
For vdev's bigger than that we start increasing the
number of metaslabs until we hit the 128K limit.

= Reasoning

The old behavior makes metaslabs grow in size when
the vdev range is between 3TB (ms_size 16GB) and
32PB (ms_size 256GB). Even though keeping the number
of metaslabs is good in terms of potential number of
I/Os per TXG, these bigger metaslabs take longer
to be loaded and after they are loaded they can
take up a lot of memory because of their range trees.

This change tries to put a boundary in memory and
loading time for the specific range of vdev sizes.

Reviewed-by: Matt Ahrens <mahrens@delphix.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Don Brady <don.brady@delphix.com>
Signed-off-by: Serapheim Dimitropoulos <serapheim@delphix.com>
Closes #8324

3 months agoRename range_tree_verify to range_tree_verify_not_present
Serapheim Dimitropoulos [Fri, 25 Jan 2019 17:51:24 +0000 (09:51 -0800)]
Rename range_tree_verify to range_tree_verify_not_present

The range_tree_verify function looks for a segment in a
range tree and panics if the segment is present on the
tree. This patch gives the function a more descriptive
name.

Reviewed-by: Matt Ahrens <mahrens@delphix.com>
Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Serapheim Dimitropoulos <serapheim@delphix.com>
Closes #8327

3 months agoUse proper tag for spa config refcounts in mmp_write_uberblock()
Tim Chase [Fri, 25 Jan 2019 17:50:06 +0000 (11:50 -0600)]
Use proper tag for spa config refcounts in mmp_write_uberblock()

This allows the spa config refcounts to use tracking in debug builds
without triggering the "No such hold %p on refcount" panic.

Reviewed-by: Olaf Faaland <faaland1@llnl.gov>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Tim Chase <tim@chase2k.com>
Closes #8326

3 months agozfs userspace dumps core when used on ZVOLs
loli10K [Fri, 25 Jan 2019 17:47:52 +0000 (18:47 +0100)]
zfs userspace dumps core when used on ZVOLs

If you try to get the userspace, groupspace or projectspace on a ZVOL,
the generated error results in passing EINVAL to
zfs_standard_error_fmt() when we should return a specific error to
inform the user that those properties aren't available on volumes.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed by: Tom Caputi <tcaputi@datto.com>
Signed-off-by: loli10K <ezomori.nozomu@gmail.com>
Closes #8279

3 months agozpool iostat should print headers when terminal fills
Damian Wojsław [Wed, 23 Jan 2019 21:29:49 +0000 (22:29 +0100)]
zpool iostat should print headers when terminal fills

When `zpool iostat` fills the terminal the headers should be
printed again.  `zpool iostat -n` can be used to suppress this.

If the command is not attached to a tty, headers will not be
printed so as to not break existing scripts.

Reviewed-by: Joshua M. Clulow <josh@sysmgr.org>
Reviewed-by: Giuseppe Di Natale <guss80@gmail.com>
Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Tony Hutter <hutter2@llnl.gov>
Signed-off-by: Damian Wojsław <damian@wojslaw.pl>
Closes #8235
Closes #8262

3 months agoFix bad kmem_free() in zvol_rename_minors_impl()
Tom Caputi [Wed, 23 Jan 2019 19:38:05 +0000 (14:38 -0500)]
Fix bad kmem_free() in zvol_rename_minors_impl()

Currently, zvol_rename_minors_impl() calls kmem_asprintf()
to allocate and initialize a string. This function is a thin
wrapper around the kernel's kvasprintf() and does not call
into the SPL's kmem tracking code when it is enabled. However,
this function frees the string with the tracked kmem_free()
instead of the untracked strfree(), which causes the SPL
kmem tracking code to believe that the function is attempting
to free memory it never allocated, triggering an ASSERT. This
patch simply corrects this issue.

Reviewed by: Matt Ahrens <mahrens@delphix.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Tom Caputi <tcaputi@datto.com>
Closes #8307

4 months agoztest: creates partially initialized root dataset
loli10K [Fri, 18 Jan 2019 19:14:01 +0000 (20:14 +0100)]
ztest: creates partially initialized root dataset

Since d8fdfc2 was integrated dsl_pool_create() does not call
dmu_objset_create_impl() for the root dataset when running in
userland (ztest): this creates a pool with a partially initialized
root dataset. Trying to import and use this pool results in both
zpool and zfs executables dumping core.

Fix this by adopting an alternative change suggested in OpenZFS 8607
code review.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed by: Tom Caputi <tcaputi@datto.com>
Original-patch-by: Robert Mustacchi <rm@joyent.com>
Signed-off-by: loli10K <ezomori.nozomu@gmail.com>
Closes #8277

4 months agoRemove zfs_sync() panicking kernel check
Brian Behlendorf [Fri, 18 Jan 2019 19:11:47 +0000 (11:11 -0800)]
Remove zfs_sync() panicking kernel check

This check provides no real additional protection and unnecessarily
introduces a dependency on the "oops_in_progress" kernel symbol.
Remove the check, it there are special circumstances on other
platforms which make this a requirement it can be reintroduced
for all relevant call paths in a more portable comprehensive manor.

Reviewed-by: Matt Ahrens <mahrens@delphix.com>
Reviewed-by: George Melikov <mail@gmelikov.ru>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #8297

4 months agoFactor metaslab_load_wait() in metaslab_load()
Serapheim Dimitropoulos [Fri, 18 Jan 2019 19:10:32 +0000 (11:10 -0800)]
Factor metaslab_load_wait() in metaslab_load()

Most callers that need to operate on a loaded metaslab, always
call metaslab_load_wait() before loading the metaslab just in
case someone else is already doing the work.

Factoring metaslab_load_wait() within metaslab_load() makes the
later more robust, as callers won't have to do the load-wait
check explicitly every time they need to load a metaslab.

Reviewed-by: Matt Ahrens <mahrens@delphix.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Serapheim Dimitropoulos <serapheim@delphix.com>
Closes #8290

4 months agoFix 0 byte memory leak in zfs receive
Tom Caputi [Fri, 18 Jan 2019 19:06:48 +0000 (14:06 -0500)]
Fix 0 byte memory leak in zfs receive

Currently, when a DRR_OBJECT record is read into memory in
receive_read_record(), memory is allocated for the bonus buffer.
However, if the object doesn't have a bonus buffer the code will
still "allocate" the zero bytes, but the memory will not be passed
to the processing thread for cleanup later. This causes the spl
kmem tracking code to report a leak. This patch simply changes the
code so that it only allocates this memory if it has a non-zero
length.

Reviewed by: Matt Ahrens <mahrens@delphix.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Tom Caputi <tcaputi@datto.com>
Closes #8266

4 months agoDocument guidelines for usage of zfs_dbgmsg
Serapheim Dimitropoulos [Fri, 18 Jan 2019 18:16:56 +0000 (10:16 -0800)]
Document guidelines for usage of zfs_dbgmsg

Reviewed-by: Richard Elling <Richard.Elling@RichardElling.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Matt Ahrens <matt@delphix.com>
Reviewed-by: Igor Kozhukhov <igor@dilos.org>
Reviewed-by: George Melikov <mail@gmelikov.ru>
Signed-off-by: Serapheim Dimitropoulos <serapheim@delphix.com>
Closes #8299

4 months agodkms: Enable debuginfo option to be set with zfs sysconfig file
Neal Gompa (ニール・ゴンパ) [Fri, 18 Jan 2019 18:10:24 +0000 (13:10 -0500)]
dkms: Enable debuginfo option to be set with zfs sysconfig file

On some Linux distributions, the kernel module build will not
default to building with debuginfo symbols, which can make it
difficult for debugging and testing.

For this case, we provide a flag to override the build to force
debuginfo to be produced for the kernel module build.

Reviewed-by: Tony Hutter <hutter2@llnl.gov>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Co-authored-by: Neal Gompa <ngompa@datto.com>
Co-authored-by: Simon Watson <swatson@datto.com>
Signed-off-by: Neal Gompa <ngompa@datto.com>
Signed-off-by: Simon Watson <swatson@datto.com>
Closes #8304

4 months agoOff-by-one in zap_leaf_array_create()
loli10K [Fri, 18 Jan 2019 17:58:46 +0000 (18:58 +0100)]
Off-by-one in zap_leaf_array_create()

Trying to set user properties with their length 1 byte shorter than the
maximum size triggers an assertion failure in zap_leaf_array_create():

  panic[cpu0]/thread=ffffff000a092c40:
  assertion failed: num_integers * integer_size < (8<<10) (0x2000 < 0x2000), file: ../../common/fs/zfs/zap_leaf.c, line: 233

  ffffff000a092500 genunix:process_type+167c35 ()
  ffffff000a0925a0 zfs:zap_leaf_array_create+1d2 ()
  ffffff000a092650 zfs:zap_entry_create+1be ()
  ffffff000a092720 zfs:fzap_update+ed ()
  ffffff000a0927d0 zfs:zap_update+1a5 ()
  ffffff000a0928d0 zfs:dsl_prop_set_sync_impl+5c6 ()
  ffffff000a092970 zfs:dsl_props_set_sync_impl+fc ()
  ffffff000a0929b0 zfs:dsl_props_set_sync+79 ()
  ffffff000a0929f0 zfs:dsl_sync_task_sync+10a ()
  ffffff000a092a80 zfs:dsl_pool_sync+3a3 ()
  ffffff000a092b50 zfs:spa_sync+4e6 ()
  ffffff000a092c20 zfs:txg_sync_thread+297 ()
  ffffff000a092c30 unix:thread_start+8 ()

This patch simply corrects the assertion.

Reviewed-by: Matt Ahrens <mahrens@delphix.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: loli10K <ezomori.nozomu@gmail.com>
Closes #8278

4 months agoSimplify spa_sync by breaking it up to smaller functions
Serapheim Dimitropoulos [Fri, 18 Jan 2019 17:50:16 +0000 (09:50 -0800)]
Simplify spa_sync by breaking it up to smaller functions

The point of this refactoring is to break the high-level conceptual
steps of spa_sync() to their own helper functions. In general large
functions can enhance readability if structured well, but in this
case the amount of conceptual steps taken could use the help of
helper functions.

Reviewed-by: Matt Ahrens <mahrens@delphix.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Serapheim Dimitropoulos <serapheim@delphix.com>
Closes #8293

4 months agoztest: scrub verification
Brian Behlendorf [Fri, 18 Jan 2019 17:47:55 +0000 (09:47 -0800)]
ztest: scrub verification

By design ztest will never inject non-repairable damage in to the
pool.  Update the ztest_scrub() test case such that it waits for
the scrub to complete and verifies the pool is always repairable.

After enabling scrub verification two scenarios were encountered
which are the result of how ztest manages failure injection.

The first case is straight forward and pertains to detaching a
mirror vdev.  In this case, the pool must always be scrubbed prior
the detach.  Failure to do so can potentially lock in previously
repairable data corruption by removing all good copies of a block
leaving only damaged ones.

The second is a little more subtle.  The child/offset selection
logic in ztest_fault_inject() depends on the calculated number of
leaves always remaining constant between injection passes.  This
is true within a single execution of ztest, but when using zloop.sh
random values are selected for each restart.  Therefore, when ztest
imports an existing pool it must be scrubbed before failure injection
can be safely enabled.  Otherwise it is possible that it will inject
non-repairable damage.

Reviewed by: Matt Ahrens <mahrens@delphix.com>
Reviewed by: Tom Caputi <tcaputi@datto.com>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #8269

4 months agoFix error handling incallers of dbuf_hold_level()
Tom Caputi [Thu, 17 Jan 2019 23:47:08 +0000 (18:47 -0500)]
Fix error handling incallers of dbuf_hold_level()

Currently, the functions dbuf_prefetch_indirect_done() and
dmu_assign_arcbuf_by_dnode() assume that dbuf_hold_level() cannot
fail. In the event of an error the former will cause a NULL pointer
dereference and the later will trigger a VERIFY. This patch adds
error handling to these functions and their callers where necessary.

Reviewed by: Matt Ahrens <mahrens@delphix.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Tom Caputi <tcaputi@datto.com>
Closes #8291

4 months agoRemove unused vdev_t fields
Serapheim Dimitropoulos [Thu, 17 Jan 2019 23:41:12 +0000 (15:41 -0800)]
Remove unused vdev_t fields

The following fields from the vdev_t struct are not used anywhere.

Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Tony Hutter <hutter2@llnl.gov>
Signed-off-by: Serapheim Dimitropoulos <serapheim@delphix.com>
Closes #8285

4 months agoztest: scrub ddt repair
Brian Behlendorf [Thu, 17 Jan 2019 23:25:00 +0000 (15:25 -0800)]
ztest: scrub ddt repair

The ztest_ddt_repair() test is designed inflict damage to the
ddt which can be repairable by a scrub.  Unfortunately, this
repair logic was broken at some point and it went undetected.
This issue is not specific to ztest, but thankfully this extra
redundancy is rarely enabled and even more rarely needed.

The root cause was identified to be the ddt_bp_create()
function called by dsl_scan_ddt_entry() which did not set the
dedup bit of the generated block pointer.

The consequence of this was that the ZIO_DDT_READ_PIPELINE was
never enabled for the block pointer during the scrub, and the
dedup ditto repair logic was never run.  Note that for demand
reads which don't rely on ddt_bp_create() the required pipeline
stages would be enabled and the repair performed.

This was resolved by unconditionally setting the dedup bit in
ddt_bp_create().  This way all codes paths which may need to
perform a repair from a block pointer generated from the dtt
entry will be able too.  The only exception is that the dedup
bit is cleared in ddt_phys_free() which is required to avoid
leaking space.

Reviewed by: Matt Ahrens <mahrens@delphix.com>
Reviewed by: Tom Caputi <tcaputi@datto.com>
Reviewed by: Serapheim Dimitropoulos <serapheim@delphix.com>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #8270

4 months agoUpdate vdev_is_spacemap_addressable() for new spacemap encoding
Serapheim Dimitropoulos [Wed, 16 Jan 2019 23:06:20 +0000 (15:06 -0800)]
Update vdev_is_spacemap_addressable() for new spacemap encoding

Since the new spacemap encoding was ported to ZoL that's no longer
a limitation. This patch updates vdev_is_spacemap_addressable()
that was performing that check.

It also updates the appropriate test to ensure that the same
functionality is tested.  The test does so by creating pools that
don't have the new spacemap encoding enabled - just the checkpoint
feature. This patch also reorganizes that same tests in order to
cut in half its memory consumption.

Reviewed by: Matt Ahrens <mahrens@delphix.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Serapheim Dimitropoulos <serapheim@delphix.com>
Closes #8286

4 months agoztest: split block reconstruction
Brian Behlendorf [Wed, 16 Jan 2019 22:10:02 +0000 (14:10 -0800)]
ztest: split block reconstruction

Increase the default allowed number of reconstruction attempts.
There's not an exact right number for this setting.  It needs
to be set large enough to cover any realistic failure scenarios
and small enough to avoid stalling the IO pipeline and invoking
the dead man detection.

The current value of 256 was empirically determined to be too
low based on multi-day runs of ztest.  The fault injection code
would inject more damage than could be reconstructed given the
relatively small number of attempts.  However, in all observed
cases the block could be reconstructed using a slightly higher
limit.

Based on local testing increasing the default value to 4096 was
determined to strike the best balance.  Checking all combinations
takes less than 10s in the worst case, and has so far eliminated
the vast majority of false positives detected by ztest.  This
delay is roughly on par with how long retries may be performed
to a misbehaving HDD and was deemed to be reasonable.  Better to
err on the side of a brief delay rather than fail to reconstruct
the data.

Lastly, the -Y flag has been added to zdb to make it easy to try all
possible combinations when performing split block reconstruction.
For badly damaged blocks with 18 splits, they can be fully enumerated
within a few minutes.  This has been done to ensure permanent errors
are never incorrectly reported when ztest verifies the pool with zdb.

Reviewed by: Tom Caputi <tcaputi@datto.com>
Reviewed by: Matt Ahrens <mahrens@delphix.com>
Reviewed by: Serapheim Dimitropoulos <serapheim@delphix.com>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #8271

4 months agoMake zdb results for checkpoint tests consistent
Serapheim Dimitropoulos [Wed, 16 Jan 2019 18:41:47 +0000 (10:41 -0800)]
Make zdb results for checkpoint tests consistent

This patch exports and re-imports the pool when these tests are
analyzed with zdb to get consistent results.

Reviewed by: Igor Kozhukhov <igor@dilos.org>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: George Melikov <mail@gmelikov.ru>
Signed-off-by: Serapheim Dimitropoulos <serapheim@delphix.com>
Closes #8292

4 months agoDisable 'zfs remap' command
Brian Behlendorf [Tue, 15 Jan 2019 23:46:58 +0000 (15:46 -0800)]
Disable 'zfs remap' command

The implementation of 'zfs remap' has proven to be problematic since
it modifies the objset (but not its logical contents) by dirtying
metadata without owning it.  The consequence of which is that
dmu_objset_remap_indirects() is vulnerable to certain races.

For example, if we are in the middle of receiving into the filesystem
while it is being remapped.  Then it is possible we could evict the
objset when the receive completes (see dsl_dataset_clone_swap_sync_impl,
or dmu_recv_end_sync), but dmu_objset_remap_indirects() may be still
using the objset.  The result of which would be a panic.

Extended runs of ztest(8) have exposed other possible races which
can occur when using 'zfs remap'.  Several of these have been fixed
but there may be others which have not yet been encountered and
diagnosed.

Furthermore, the ability to manually remap a filesystem is no longer
particularly useful now that the removal code can map large chunks.
Coupled with the fact that explaining what this command does and why
it may be useful requires a detailed understanding of the internals
of device removal.  These are details users should not be bothered
with.

Therefore, the 'zfs remap' command is being disabled but not entirely
removed.  It may be removed in the future or potentially reworked
to address the issues described above.  Since 'zfs remap' has never
been part of a tagged release its removal is expected to have
minimal impact.

The ZTS tests have been updated to continue to exercise the command
to prevent atrophy, but it has been removed entirely from ztest(8).

Reviewed by: Matt Ahrens <mahrens@delphix.com>
Reviewed by: Tom Caputi <tcaputi@datto.com>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #8238

4 months agoFix zio leak in dbuf_read()
Tom Caputi [Tue, 15 Jan 2019 20:23:40 +0000 (15:23 -0500)]
Fix zio leak in dbuf_read()

Currently, dbuf_read() may decide to create a zio_root which is
used as a parent for any child zios created in dbuf_read_impl().
However, if there is an error in dbuf_read_impl(), this zio is
never executed and ends up leaked. This patch simply ensures
that we always execute the root zio, even i it has no real work
to do.

Reviewed-by: Matt Ahrens <mahrens@delphix.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Tom Caputi <tcaputi@datto.com>
Closes #8267

4 months agoVerify .gitignore entries
loli10K [Tue, 15 Jan 2019 19:56:29 +0000 (20:56 +0100)]
Verify .gitignore entries

This change adds a make target 'vcscheck' which scans the git workspace
for new, untracked files missing from the .gitignore configuration; this
is done to help prevent adding unwanted build artifacts to the source
tree during development.

Reviewed-by: Neal Gompa <ngompa@datto.com>
Reviewed-by: Giuseppe Di Natale <dinatale2@llnl.gov>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: loli10K <ezomori.nozomu@gmail.com>
Closes #8281

4 months agoTag 0.8.0-rc3
Brian Behlendorf [Thu, 10 Jan 2019 23:58:15 +0000 (15:58 -0800)]
Tag 0.8.0-rc3

Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
4 months agoMinor spelling corrections
Brian Behlendorf [Sun, 13 Jan 2019 18:11:52 +0000 (10:11 -0800)]
Minor spelling corrections

Some minor spelling mistakes and typos.  No functional changes.

Reviewed-by: Neal Gompa <ngompa@datto.com>
Reviewed by: Matt Ahrens <mahrens@delphix.com>
Reviewed-by: Giuseppe Di Natale <guss80@gmail.com>
Reviewed-by: bunder2015 <omfgbunder@gmail.com>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #8272

4 months agoSerialize ZTHR operations to eliminate races
Serapheim Dimitropoulos [Sun, 13 Jan 2019 18:09:46 +0000 (10:09 -0800)]
Serialize ZTHR operations to eliminate races

Adds a new lock for serializing operations on zthrs.
The commit also includes some code cleanup and
refactoring.

Reviewed by: Matt Ahrens <mahrens@delphix.com>
Reviewed by: Tom Caputi <tcaputi@datto.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Serapheim Dimitropoulos <serapheim@delphix.com>
Closes #8229

4 months agozfs filesystem skipped by df -h
Paul Zuchowski [Sun, 13 Jan 2019 18:06:13 +0000 (13:06 -0500)]
zfs filesystem skipped by df -h

On full pool when pool root filesystem references very few bytes,
the f_blocks returned to statvfs is 0 but should be at least 1.

Reviewed by: Tom Caputi <tcaputi@datto.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Paul Zuchowski <pzuchowski@datto.com>
Closes #8253
Closes #8254

4 months agoAdd contrib/pyzfs/setup.py to .gitignore
Tom Caputi [Sun, 13 Jan 2019 18:04:38 +0000 (13:04 -0500)]
Add contrib/pyzfs/setup.py to .gitignore

As of 9ef798b77, setup.py is now generated from setup.py.in, but
this file was never moved to the .gitignore. This patch simply
corrects this issue.

Reviewed-by: Neal Gompa <ngompa@datto.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Tom Caputi <tcaputi@datto.com>
Closes #8268

4 months agoAdd pyzfs BuildRequires for mock(1)
Brian Behlendorf [Sun, 13 Jan 2019 18:02:33 +0000 (10:02 -0800)]
Add pyzfs BuildRequires for mock(1)

When building pyzfs under mock the python-cffi and python-setuptools
packages need to be installed and have been added to the BuildRequires.

Reviewed-by: Neal Gompa <ngompa@datto.com>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #8265

4 months agoZTS: zpool_resilver_restart
Brian Behlendorf [Sun, 13 Jan 2019 18:01:31 +0000 (10:01 -0800)]
ZTS: zpool_resilver_restart

Since the vdev initialize feature was integrated the ZTS
zpool_resilver_restart test has been hitting its internal
timeout more frequently.  This happens most often on
the coverage builder but not exclusively.  Increasing the
timeout for this test case prevents any false positives.

Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #8273

4 months agoProvide more flexible object allocation interface
Brian Behlendorf [Thu, 10 Jan 2019 22:37:43 +0000 (14:37 -0800)]
Provide more flexible object allocation interface

Object allocation performance can be improved for complex operations
by providing an interface which returns the newly allocated dnode.
This allows the caller to immediately use the dnode without incurring
the expense of looking up the dnode by object number.

The functions dmu_object_alloc_hold(), zap_create_hold(), and
dmu_bonus_hold_by_dnode() were added for this purpose.

The zap_create_* functions have been updated to take advantage of
this new functionality.  The dmu_bonus_hold_impl() function should
really have never been included in sys/dmu.h and was removed.
It's sole caller was converted to use dmu_bonus_hold_by_dnode().

The new symbols have been exported for use by Lustre.

Reviewed-by: Tony Hutter <hutter2@llnl.gov>
Reviewed by: Matt Ahrens <mahrens@delphix.com>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #8015

4 months agoDon't allow dnode allocation if dn_holds != 0
Tom Caputi [Thu, 10 Jan 2019 22:36:23 +0000 (17:36 -0500)]
Don't allow dnode allocation if dn_holds != 0

This patch simply fixes a small bug where dnode_hold_impl() could
attempt to allocate a dnode that was in the process of being freed,
but which still had active references. This patch simply adds the
required check.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Tom Caputi <tcaputi@datto.com>
Closes #8249

4 months agoRemoved suggestion to use root dataset as bootfs
Gregor Kopka [Wed, 9 Jan 2019 00:15:30 +0000 (01:15 +0100)]
Removed suggestion to use root dataset as bootfs

The dracut howto proposed to boot from the root dataset of a pool.
Apart from this giving problems when booting (as the code seems to
expect a child dataset and creates an illegal dataset name when using
the root dataset) the technical limitations of the root dataset
(among others the inability to rename or destroy through the `zfs`
command) resulted in the general consensus to only use it as a
container for the datasets in the pool - not as a filesystem itself.

Removed the idea to boot from the root dataset.

Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: bunder2015 <omfgbunder@gmail.com>
Signed-off-by: Gregor Kopka <gregor@kopka.net>
Closes #8247

4 months agoUse ZFS version for pyzfs & drop unused reqs file
Neal Gompa (ニール・ゴンパ) [Tue, 8 Jan 2019 23:56:42 +0000 (18:56 -0500)]
Use ZFS version for pyzfs & drop unused reqs file

Now that 'pyzfs' is part of the ZFS codebase, it should be
versioned the same as the rest of the source tree. This eliminates
confusion on what version of the bindings are being used, especially
for dependent Python projects that may use the Python dist metadata
to identify compatible versions of pyzfs to work from.

In addition, a trivial change to drop the unused requirements.txt
file is included, simply because it's unused and a leftover from
before it was imported into the ZFS codebase and wired into the
autotools build scripts.

Reviewed-by: loli10K <ezomori.nozomu@gmail.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Neal Gompa <ngompa@datto.com>
Closes #8243

4 months agozfs receive and rollback can skew filesystem_count
loli10K [Tue, 8 Jan 2019 18:17:46 +0000 (19:17 +0100)]
zfs receive and rollback can skew filesystem_count

This commit fixes a small issue which causes both zfs receive and
rollback operations to incorrectly increase the "filesystem_count"
property value.

This change also adds a new test group "limits" to the ZFS Test Suite
to exercise both filesystem_count/limit and snapshot_count/limit
functionality.

Reviewed by: Jerry Jelinek <jerry.jelinek@joyent.com>
Reviewed by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: loli10K <ezomori.nozomu@gmail.com>
Closes #8232

4 months agoOpenZFS 8473 - scrub does not detect errors on active spares
asomers [Wed, 20 Sep 2017 16:31:00 +0000 (16:31 +0000)]
OpenZFS 8473 - scrub does not detect errors on active spares

Scrubbing is supposed to detect and repair all errors in the pool.
However, it wrongly ignores active spare devices. The problem can
easily be reproduced in OpenZFS at git rev 0ef125d with these
commands:

    truncate -s 64m /tmp/a /tmp/b /tmp/c
    sudo zpool create testpool mirror /tmp/a /tmp/b spare /tmp/c
    sudo zpool replace testpool /tmp/a /tmp/c
    /bin/dd if=/dev/zero bs=1024k count=63 oseek=1 conv=notrunc of=/tmp/c
    sync
    sudo zpool scrub testpool
    zpool status testpool # Will show 0 errors, which is wrong
    sudo zpool offline testpool /tmp/a
    sudo zpool scrub testpool
    zpool status testpool # Will show errors on /tmp/c,
                          # which should've already been fixed

FreeBSD head is partially affected: the first scrub will detect
some errors, but the second scrub will detect more.  This same
test was run on Linux before applying the fix and the FreeBSD
head behavior was observed.

Authored by: asomers <asomers@FreeBSD.org>
Reviewed by: Andy Stormont <astormont@racktopsystems.com>
Reviewed by: Matt Ahrens <mahrens@delphix.com>
Reviewed by: George Wilson <george.wilson@delphix.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Approved by: Richard Lowe <richlowe@richlowe.net>
Ported-by: Brian Behlendorf <behlendorf1@llnl.gov>
Sponsored by: Spectra Logic Corp

OpenZFS-issue: https://www.illumos.org/issues/8473
FreeBSD-commit: https://github.com/freebsd/freebsd/commit/e20ec8879
OpenZFS-commit: https://github.com/illumos/illumos-gate/commit/554675ee
Closes #8251