George Melikov [Mon, 23 Jan 2017 17:36:24 +0000 (20:36 +0300)]
OpenZFS 7054 - dmu_tx_hold_t should use refcount_t to track space
Authored by: Igor Kozhukhov ikozhukhov@gmail.com
Reviewed by: George Wilson <george.wilson@delphix.com>
Reviewed by: Paul Dagnelie <pcd@delphix.com>
Reviewed by: Igor Kozhukhov <ikozhukhov@gmail.com>
Approved by: Dan McDonald <danmcd@omniti.com> Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Ported-by: George Melikov mail@gmelikov.ru
OpenZFS-issue: https://www.illumos.org/issues/7054
OpenZFS-commit: https://github.com/openzfs/openzfs/commit/0c779ad
Closes #5600
Paul Dagnelie [Tue, 17 Jan 2017 22:47:50 +0000 (01:47 +0300)]
OpenZFS 6459 - cstyle doesn't detect opening braces on the same line as function header
Authored by: Paul Dagnelie <pcd@delphix.com>
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: Alex Reece <alex@delphix.com>
Reviewed by: Albert Lee <trisk@omniti.com>
Reviewed by: Dan McDonald <danmcd@omniti.com>
Approved by: Robert Mustacchi <rm@joyent.com> Ported-by: George Melikov mail@gmelikov.ru
OpenZFS-issue: https://www.illumos.org/issues/6459
OpenZFS-commit: https://github.com/openzfs/openzfs/commit/c4567a6
Porting notes:
These changes are adopted for ZoL codebase because of
many false positive warnings.
Brian Behlendorf [Fri, 20 Jan 2017 01:09:04 +0000 (17:09 -0800)]
Fix unused variable warning
The local mg variable is unused in non-debug builds.
Wrap the variable in ASSERTV() so that it's only present
in the debug build. Introduced by OpenZFS 7303.
Reviewed-by: Don Brady <don.brady@intel.com> Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #5616
Chunwei Chen [Thu, 19 Jan 2017 21:56:36 +0000 (13:56 -0800)]
Suspend/resume zvol for recv and rollback
When doing recv and rollback, dsl_dataset_clone_swap_sync_impl will be
called to swap out the ds_objset and do dmu_objset_evict on the old one.
However, currently zv->zv_objset will not be swapped out accordingly, so
if anyone currently holds a fd on the zvol, we risk hitting a use-after-free.
We fix this by introducing the suspend and resume mechanism of zsb to
zv. Before recv or rollback, we use zvol_suspend to block all access to
zv_objset and shut it down. After the recv or rollback, we use zvol_resume
to swap in zv_objset with the new ds_objset and unblock the access.
George Melikov [Thu, 19 Jan 2017 21:50:22 +0000 (00:50 +0300)]
OpenZFS 6529 - Properly handle updates of variably-sized SA entries
Porting notes:
- This issue was first fixed in ZoL by commit d862cb0d. That fix was
then modified and an equivalent version of the patch landed in the
upstream code base. For additional details see the discussion in
https://github.com/openzfs/openzfs/pull/24 .
This commit aligns ZoL with OpenZFS codebase.
Authored by: Andriy Gapon <avg@icyb.net.ua>
Reviewed by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: Ned Bass <bass6@llnl.gov>
Reviewed by: Tim Chase <tim@chase2k.com>
Approved by: Gordon Ross <gwr@nexenta.com> Ported-by: George Melikov mail@gmelikov.ru
OpenZFS-issue: https://www.illumos.org/issues/6529
OpenZFS-commit: https://github.com/openzfs/openzfs/commit/e7e978b
Closes #5606
George Melikov [Tue, 17 Jan 2017 22:52:17 +0000 (01:52 +0300)]
OpenZFS 7071 - lzc_snapshot does not fill in errlist on ENOENT
Authored by: Igor Kozhukhov ikozhukhov@gmail.com
Reviewed by: Igor Kozhukhov <ikozhukhov@gmail.com>
Reviewed by: George Wilson <george.wilson@delphix.com>
Reviewed by: Dan Kimmel <dan.kimmel@delphix.com>
Approved by: Robert Mustacchi <rm@joyent.com> Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Ported-by: George Melikov <mail@gmelikov.ru>
OpenZFS-issue: https://www.illumos.org/issues/7071
OpenZFS-commit: https://github.com/openzfs/openzfs/commit/25f7d99
Closes #5597
George Melikov [Tue, 17 Jan 2017 22:49:24 +0000 (01:49 +0300)]
OpenZFS 7082 - bptree_iterate() passes wrong args to zfs_dbgmsg()
Authored by: Igor Kozhukhov <ikozhukhov@gmail.com>
Reviewed by: Dan Kimmel <dan.kimmel@delphix.com>
Reviewed by: George Wilson <george.wilson@delphix.com>
Reviewed by: Igor Kozhukhov <ikozhukhov@gmail.com>
Approved by: Dan McDonald <danmcd@omniti.com> Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Reviewed-by: Giuseppe Di Natale <dinatale2@llnl.gov> Ported-by: George Melikov <mail@gmelikov.ru>
OpenZFS-issue: https://www.illumos.org/issues/7082
OpenZFS-commit: https://github.com/openzfs/openzfs/commit/10e67aa
Closes #5596
Brian Behlendorf [Tue, 17 Jan 2017 22:45:02 +0000 (14:45 -0800)]
OpenZFS 6550 - cmd/zfs: cleanup gcc warnings
Porting Notes:
- Many of the fixes proposed by this patch were already applied.
In the cases where a different but equivalent fix was made the
code was updated with the OpenZFS version to minimize differences.
Authored by: Igor Kozhukhov <ikozhukhov@gmail.com>
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: Andy Stormont <astormont@racktopsystems.com>
Approved by: Dan McDonald <danmcd@omniti.com> Reviewed-by: George Melikov <mail@gmelikov.ru> Ported-by: Brian Behlendorf <behlendorf1@llnl.gov>
OpenZFS-issue: https://www.illumos.org/issues/6550
OpenZFS-commit: https://github.com/openzfs/openzfs/commit/c16bcc4
Closes #5591
Brian Behlendorf [Tue, 17 Jan 2017 22:42:56 +0000 (14:42 -0800)]
OpenZFS 6551 - cmd/zpool: cleanup gcc warnings
Porting Notes:
- Many of the fixes proposed by this patch were already applied.
In the cases where a different but equivalent fix was made the
code was updated with the OpenZFS version to minimize differences.
- The zpool_get_vdev_by_name() function was previously removed
by commit 235db0a.
Authored by: Igor Kozhukhov <ikozhukhov@gmail.com>
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: Andy Stormont <astormont@racktopsystems.com>
Approved by: Robert Mustacchi <rm@joyent.com> Reviewed-by: George Melikov <mail@gmelikov.ru> Reviewed-by: Haakan T Johansson <f96hajo@chalmers.se> Ported-by: Brian Behlendorf <behlendorf1@llnl.gov>
OpenZFS-issue: https://www.illumos.org/issues/6551
OpenZFS-commit: https://github.com/openzfs/openzfs/commit/b327cd3
Closes #5590
LOLi [Fri, 13 Jan 2017 23:47:34 +0000 (00:47 +0100)]
Fix unallocated object detection for large_dnode datasets
Fix dmu_object_next() to correctly handle unallocated objects on
large_dnode datasets.
We implement this by scanning the dnode block until we find the correct
offset to be used in dnode_next_offset(). This is necessary because we
can't assume *objectp is a hole even if dmu_object_info() returns
ENOENT.
This fixes a couple of issues with zfs receive on large_dnode datasets.
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Ned Bass <bass6@llnl.gov> Signed-off-by: loli10K <ezomori.nozomu@gmail.com>
Closes #5027
Closes #5532
Brian Behlendorf [Fri, 13 Jan 2017 23:29:32 +0000 (15:29 -0800)]
OpenZFS 7181 - race between zfs_mount and zfs_ioc_rollback
Authored by: Andriy Gapon <andriy.gapon@clusterhq.com>
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Approved by: Gordon Ross <gordon.w.ross@gmail.com> Reviewed-by: Richard Yao <ryao@gentoo.org> Reviewed-by: Giuseppe Di Natale <dinatale2@llnl.gov> Ported-by: Brian Behlendorf <behlendorf1@llnl.gov>
OpenZFS-issue: https://www.illumos.org/issues/7181
OpenZFS-commit: https://github.com/openzfs/openzfs/commit/90f2c09
Closes #5585
bzzz77 [Fri, 13 Jan 2017 22:58:41 +0000 (01:58 +0300)]
Add *_by-dnode routines
Add *_by_dnode() routines for accessing objects given their
dnode_t *, this is more efficient than accessing the object by
(objset_t *, uint64_t object). This change converts some but
not all of the existing consumers. As performance-sensitive
code paths are discovered they should be converted to use
these routines.
Reviewed-by: Matthew Ahrens <mahrens@delphix.com> Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Closes #5534
Issue #4802
Don Brady [Fri, 13 Jan 2017 21:50:22 +0000 (14:50 -0700)]
OpenZFS 7743 - per-vdev-zaps init path for upgrade
Authored by: Paul Dagnelie <pcd@delphix.com>
Reviewed by: Matt Ahrens <mahrens@delphix.com>
Reviewed by: Pavel Zakharov <pavel.zakharov@delphix.com>
Reviewed by: George Wilson <george.wilson@delphix.com>
Reviewed by: Don Brady <don.brady@intel.com> Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Reviewed-by: Joe Stein <jas14@cs.brown.edu> Ported-by: Don Brady <don.brady@intel.com>
When loading a pool that had been created before the existance of
per-vdev zaps, on a system that knows about per-vdev zaps, the
per-vdev zaps will not be allocated and initialized.
This appears to be because the logic that would have done so, in
spa_sync_config_object(), is not reached under normal operation. It is
only reached if spa_config_dirty_list is non-empty.
The fix is to add another `AVZ_ACTION_` enum that will allow this code
to be reached when we detect that we're loading an old pool, even when
there are no dirty configs.
Don Brady [Thu, 12 Jan 2017 19:52:56 +0000 (12:52 -0700)]
OpenZFS 7303 - dynamic metaslab selection
This change introduces a new weighting algorithm to improve
metaslab selection. The new weighting algorithm relies on the
SPACEMAP_HISTOGRAM feature. As a result, the metaslab weight
now encodes the type of weighting algorithm used (size-based
vs segment-based).
Porting Notes: The metaslab allocation tracing code is conditionally
removed on linux (dependent on mdb debugger).
Authored by: George Wilson <george.wilson@delphix.com>
Reviewed by: Alex Reece <alex@delphix.com>
Reviewed by: Chris Siden <christopher.siden@delphix.com>
Reviewed by: Dan Kimmel <dan.kimmel@delphix.com>
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: Paul Dagnelie <paul.dagnelie@delphix.com>
Reviewed by: Pavel Zakharov pavel.zakharov@delphix.com
Reviewed by: Prakash Surya <prakash.surya@delphix.com>
Reviewed by: Don Brady <don.brady@intel.com> Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Ported-by: Don Brady <don.brady@intel.com>
OpenZFS-issue: https://www.illumos.org/issues/7303
OpenZFS-commit: https://github.com/openzfs/openzfs/commit/d5190931bd
Closes #5404
George Melikov [Thu, 12 Jan 2017 19:25:27 +0000 (22:25 +0300)]
OpenZFS 6637 - replacing "dontclose" with "should_close"
Authored by: David Schwartz <dschwartz783@gmail.com>
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: Prakash Surya <prakash.surya@delphix.com>
Approved by: Robert Mustacchi <rm@joyent.com> Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Reviewed-by: Giuseppe Di Natale <dinatale2@llnl.gov> Ported-by: George Melikov <mail@gmelikov.ru>
I find that this is a lot easier to read. "not don't close" is somewhat tough on the eyes.
Johnny Stenback [Tue, 3 Jan 2017 18:29:23 +0000 (10:29 -0800)]
Fix TypeError: unorderable types: str() > int() in arc_summary.py
Running arc_summary.py with a l2arc cache device around produces
the following error:
Traceback (most recent call last):
File "/usr/bin/arc_summary.py", line 1148, in <module>
main()
File "/usr/bin/arc_summary.py", line 1144, in main
page(Kstat)
File "/usr/bin/arc_summary.py", line 724, in _l2arc_summary
arc["l2_arc_evicts"]["reading"] > 0:
TypeError: unorderable types: str() > int()
This is due to arc["l2_arc_evicts"]['lock_retries'] and
arc["l2_arc_evicts"]["reading"] both being strings, returned
from fHits() earlier. Rather than adding them up and checking
if the result is > 0, this checks if either string is != '0'.
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Reviewed-by: Giuseppe Di Natale <dinatale2@llnl.gov>
Closes #5538
George Melikov [Tue, 3 Jan 2017 18:03:05 +0000 (21:03 +0300)]
OpenZFS 7259 - DS_FIELD_LARGE_BLOCKS is unused
Authored by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: Dan Kimmel <dan.kimmel@delphix.com>
Reviewed by: Prakash Surya <prakash.surya@delphix.com>
Reviewed by: Dan McDonald <danmcd@omniti.com>
Reviewed by: Igor Kozhukhov <ikozhukhov@gmail.com> Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Ported-by: George Melikov <mail@gmelikov.ru>
The DS_FIELD_LARGE_BLOCKS macro has been unused since the integration of
this patch: 241b541 Illumos 5959 - clean up per-dataset feature count code.
This patch simply removes this macro from dsl_dataset.h.
ka7 [Tue, 3 Jan 2017 17:31:18 +0000 (18:31 +0100)]
Fix spelling
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov Reviewed-by: Giuseppe Di Natale <dinatale2@llnl.gov>> Reviewed-by: George Melikov <mail@gmelikov.ru> Reviewed-by: Haakan T Johansson <f96hajo@chalmers.se>
Closes #5547
Closes #5543
Tim Chase [Fri, 30 Dec 2016 22:03:59 +0000 (16:03 -0600)]
4.10 compat - BIO flag changes and others
[bio] The req_op enum was changed to req_opf. Update the "Linux 4.8 API"
autotools checks to use an int to determine whether the various REQ_OP
values are defined. This should work properly on kernels >= 4.8.
[bio] bio_set_op_attrs() is now an inline function and can't be detected
with #ifdef. Add a configure check to determine whether bio_set_op_attrs()
is defined. Move the local definition of it from vdev_disk.c to
blkdev_compat.h for consistency with other related compability shims.
[bio] The read/write flags and their modifiers, including WRITE_FLUSH,
WRITE_FUA and WRITE_FLUSH_FUA have been removed from fs.h. Add the new
bio_set_flush() compatibility wrapper to replace VDEV_WRITE_FLUSH_FUA
and set the flags appropriately for each supported kernel version.
[vfs] The generic_readlink() function has been made static. If .readlink
in inode_operations is NULL, generic_readlink() is used.
[zol typo] Completely unrelated to 4.10 compat, fix a typo in the check
for REQ_OP_SECURE_ERASE so that the proper macro is defined:
GeLiXin [Wed, 21 Dec 2016 19:27:24 +0000 (03:27 +0800)]
Fix coverity defects: CID 147587
CID 147587: Out-of-bounds read
Future changes may cause an array overrun of 4096 bytes at byte
offset 4096 by dereferencing pointer dstp. Adding this additional
check ensures correctness.
Chunwei Chen [Wed, 21 Dec 2016 18:47:15 +0000 (10:47 -0800)]
Use a dedicated taskq for vdev_file
The introduction of parallel zvol prefetch causes deadlock when using
vdev_file.
spa_async->(spa_namespace_lock)->txg_wait_synced->(wait for txg_sync)
txg_sync->zio_wait->(wait for vdev_file_io_fsync on system_taskq)
zvol_prefetch_minors_impl (on system_taskq)->spa_open_common->(wait for spa_namespace_lock)
We fix this by using dedicated taskq for vdev_file. This same change
was originally made in commit bc25c93 but reverted in commit aa9af22
when dynamic taskqs were added.
LOLi [Wed, 21 Dec 2016 02:46:59 +0000 (03:46 +0100)]
Fix dsl_props_set_sync_impl to work with nested nvlist
When iterating over the input nvlist in dsl_props_set_sync_impl() when we don't
preserve the nvpair name before looking up ZPROP_VALUE, so when we later go to
process it nvpair_name() is always "value" and not the actual property name.
This fixes a couple of bugs in zfs_ioc_recv():
* Received properties were not restored correctly when failing to receive an
incremental send stream
* Received properties were not completely replaced by the new ones when
successfully receiving an incremental send stream
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: loli10K <ezomori.nozomu@gmail.com>
Closes #5497
Brian Behlendorf [Mon, 19 Dec 2016 21:01:10 +0000 (13:01 -0800)]
Fix file attributes
This branch contains the following fixes/improvements.
* Fix setting i_flags
* Fix wrong operator in xvattr.h
* Fix fchange macro in zpl_ioctl_setflags()
* Added configure check to use inode_set_flags()
* Added a test case for chattr for better test coverage
Reviewed-by: Tim Chase <tim@chase2k.com> Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Chunwei Chen <david.chen@osnexus.com>
Closes #5486
Closes #5470
Closes #5469
Chunwei Chen [Mon, 19 Dec 2016 17:46:29 +0000 (09:46 -0800)]
Fix zmo leak when zfs_sb_create fails
zfs_sb_create would normally takes ownership of zmo, and it will be freed in
zfs_sb_free. However, when zfs_sb_create fails we need to explicit free it.
Tony Hutter [Sat, 17 Dec 2016 00:10:45 +0000 (16:10 -0800)]
Don't run 'zpool iostat -c CMD' command on all vdevs, if vdevs specified
zpool iostat allows you to specify only certain vdevs to display.
Currently, if you run 'zpool iostat -c CMD vdev1 vdev2 ...'
on specific vdevs, it will actually run the command on *all* vdevs,
and just display the results for the vdevs you specify. This patch
corrects the behavior to only run the command on the specified vdevs,
and also enables the zpool_iostat_005_pos.ksh tests.
Reviewed-by: Giuseppe Di Natale <dinatale2@llnl.gov> Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Tony Hutter <hutter2@llnl.gov>
Closes #5443
Chunwei Chen [Fri, 16 Dec 2016 20:41:56 +0000 (12:41 -0800)]
Fix fchange in zpl_ioctl_setflags
The fchange in zpl_ioctl_setflags was for detecting flag change. However it
was incorrect and would always fail to detect a flag change from set to unset,
causing users without CAP_LINUX_IMMUTABLE to be able to unset flags.
cao [Fri, 16 Dec 2016 17:11:17 +0000 (01:11 +0800)]
Fix coverity defects: CID 147534
CID 147534: Negative array index read
Reviewed-by: Giuseppe Di Natale <dinatale2@llnl.gov> Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: cao.xuewen <cao.xuewen@zte.com.cn>
Closes #5467
Brian Behlendorf [Wed, 14 Dec 2016 17:36:14 +0000 (09:36 -0800)]
Skip xfstests on Ubuntu 16.04 and CentOS 7
The ZFS enabled versions of xfstests fails to build cleanly on
Ubuntu 16.04 and CentOS 7. This issue should be resolved by
rebasing the ZFS patches against the latest xfstests and pushing
those patches upstream. This would allow us to use an unmodified
xfstests.
Reviewed-by: Giuseppe Di Natale <dinatale2@llnl.gov> Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Issue #5481
Closes #5482
Brian Behlendorf [Wed, 14 Dec 2016 17:33:07 +0000 (09:33 -0800)]
Skip slow tests when kmemleak is enabled
When running the ZFS Test Suite with a kmemleak enabled kernel
the following test cases run far slower than usual and may hit
their timeout threshold. Skip the following test cases.
Test: cli_root/zfs_get/zfs_get_009_pos (run as root) [55:43]
Test: cli_root/zpool_clear/zpool_clear_001_pos (run as root) [11:32]
Test: cli_root/zpool_create/zpool_create_024_pos (run as root) [11:01]
Test: features/async_destroy/async_destroy_001_pos (run as root) [41:15]
Test: inheritance/inherit_001_pos (run as root) [09:08]
Brian Behlendorf [Mon, 12 Dec 2016 18:46:26 +0000 (10:46 -0800)]
Use cstyle -cpP in `make cstyle` check
Enable picky cstyle checks and resolve the new warnings. The vast
majority of the changes needed were to handle minor issues with
whitespace formatting. This patch contains no functional changes.
Non-whitespace changes are as follows:
* 8 times ; to { } in for/while loop
* fix missing ; in cmd/zed/agents/zfs_diagnosis.c
* comment (confim -> confirm)
* change endline , to ; in cmd/zpool/zpool_main.c
* a number of /* BEGIN CSTYLED */ /* END CSTYLED */ blocks
* /* CSTYLED */ markers
* change == 0 to !
* ulong to unsigned long in module/zfs/dsl_scan.c
* rearrangement of module_param lines in module/zfs/metaslab.c
* add { } block around statement after for_each_online_node
Reviewed-by: Giuseppe Di Natale <dinatale2@llnl.gov> Reviewed-by: Håkan Johansson <f96hajo@chalmers.se> Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #5465
George Melikov [Fri, 9 Dec 2016 19:48:12 +0000 (20:48 +0100)]
Add CONTRIBUTING information and templates
Guidelines for developers and users describing how they can
participle in the project.
Reviewed-by: Manuel Mendez <mmendez534@gmail.com> Reviewed-by: Giuseppe Di Natale <dinatale2@llnl.gov> Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: George Melikov <mail@gmelikov.ru>
Closes #672
Closes #4776
Closes #5361
luozhengzheng [Thu, 8 Dec 2016 21:48:09 +0000 (05:48 +0800)]
Fix coverity defects: CID 154617
CID 154617: Memory - illegal accesses (UNINIT)
The value here just needs to be initialized to make Coverity happy.
When dsize == 0, then value of daiter.iter_mapaddr is irrelevant. That
address won't be accessed, it's only used for some arithmetic. dsize
can be zero either if dabd is null, or if code column is longer than the
current data column.
* Add system delay taskq
* Parallel prefetch zvol dnodes during zvol_create_minors
* Parallel zvol_free during zvol_remove_minors
* Reduce list linear search using ida and hash
Reviewed-by: Boris Protopopov <boris.protopopov@actifio.com> Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Chunwei Chen <david.chen@osnexus.com>
Closes #5433
It was observed that even when the txg history is disabled by
setting `zfs_txg_history=0` the txg_sync thread still fetches
the vdev stats unnecessarily.
This patch refactors the code such that vdev_get_stats() is no
longer called when `zfs_txg_history=0`. And it further reduces
the differences between upstream and the ZoL txg_sync_thread()
function.
Reviewed-by: Tony Hutter <hutter2@llnl.gov> Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #5412
Chunwei Chen [Wed, 30 Nov 2016 21:56:50 +0000 (13:56 -0800)]
zvol_remove_minors do parallel zvol_free
On some kernel version, blk_cleanup_queue and put_disk will wait for more then
10ms. So a pool with a lot of zvols will easily wait for more then 1 min if we
do zvol_free sequentially.
Chunwei Chen [Wed, 30 Nov 2016 21:56:50 +0000 (13:56 -0800)]
zpool_create_minors parallel prefetch
Do parallel prefetch all zvol dnodes before actually creating each individual.
This will greatly reduce the import time when having a lot of zvols and disk
is slow.
The zpool_scrub_004_pos test case currently fails when testing on
a 32-bit system. Conditionally skip this test case on 32-bit
systems until the root cause is identified and resolved.
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Issue #5444
Closes #5445
OpenZFS 7143 - dbuf_read() creates unnecessary zio_root() for bonus buf
dbuf_read() creates a zio_root() to track and wait for all the
zio's that may happen as part of this call. However, if the blkptr_t
for this buffer is NULL or a hole, we will not create any more zio's,
so this zio_root() is unnecessary. This is always the case when calling
dbuf_read() on a bonus buffer, because it has no blkptr (it's part of
the containing dnode). For workloads that read a lot of bonus buffers
(e.g. file creation and removal), creating and destroying these
unnecessary zio's can decrease performance by around 3%.
The fix is to only create/destroy the zio_root() in dbuf_read() if
the blkptr is not NULL and not a hole.
Changes sponsored by Intel Corp.
Authored by: Matthew Ahrens <mahrens@delphix.com> Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com> Ported-by: Brian Behlendorf <behlendorf1@llnl.gov>
Issue openzfs/openzfs#137
Closes #4803
Closes #5382
cao [Thu, 1 Dec 2016 23:39:06 +0000 (07:39 +0800)]
Remove unused sa_update_from_cb()
It looks like this was functionality which was added in the
original SA implementation and then never needed. It can
be safely removed now and easily added back if we find a
use for it.
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: cao.xuewen <cao.xuewen@zte.com.cn>
Closes #5440
cao [Thu, 1 Dec 2016 23:36:25 +0000 (07:36 +0800)]
Compile zio.h and zio_impl.h mutual include
zio.h includes zio_impl.h but zio_impl.h also includes zio.h, so the
header files to contain each other. Get rid of the zio_impl.h include
in zio.h and update zio_inject.c to include zio.h instead of zio_impl.h.
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: cao.xuewen <cao.xuewen@zte.com.cn>
Closes #5439
Chunwei Chen [Wed, 30 Nov 2016 21:56:50 +0000 (13:56 -0800)]
Use system_delay_taskq for long delay tasks
Use it for spa_deadman, zpl_posix_acl_free, snapentry_expire.
This free system_taskq from the above long delay tasks, and allow us to do
taskq_wait_outstanding on system_taskq without being blocked forever, making
system_taskq more generic and useful.
Håkan Johansson [Thu, 1 Dec 2016 00:46:16 +0000 (01:46 +0100)]
Do not force VDEV_NAME_TYPE_ID in max_width()
Do not force VDEV_NAME_TYPE_ID in max_width(), instead add it
in the relevant calls to max_width().
The first location of max_width() where VDEV_NAME_TYPE_ID is
now added in show_import() is followed by print_import_config() and
print_logs(). Both these print children vdev names that have been
retrieved using an explicit VDEV_NAME_TYPE_ID added.
The second location is in status_callback(). This is followed by
print_status_config(), print_logs(), print_l2cache(), and
print_spares(). For l2cache and spares it should not matter as there
are no mirror-X or raidz-X involved. print_status_config() as above
retrieves the name using explicit VDEV_NAME_TYPE_ID before
calling itself to print children.
The call of max_width() in get_namewidth() is not changed, as this is
used by zpool_do_iostat(), followed by print_iostat(), which does not
add VDEV_NAME_TYPE_ID.
Overall, we should consider adding VDEV_NAME_TYPE_ID to the
relevant name_flags / cb_name_flags fields, and remove the explicit
adding in called routines.
Reviewed-by: Tony Hutter <hutter2@llnl.gov> Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Haakan T Johansson <f96hajo@chalmers.se>
Closes #5401
Brian Behlendorf [Wed, 30 Nov 2016 23:18:20 +0000 (16:18 -0700)]
Convert zio_buf_alloc() consumers
In multiple cases zio_buf_alloc() was used instead of kmem_alloc()
or vmem_alloc(). This was often done because the allocations
could be large and it was easy to use zfs_buf_alloc() for them.
But this isn't ideal for allocations which are small or short
lived. In these cases it is better to use kmem_alloc() or
vmem_alloc(). If possible we want to avoid the case where
we have slabs allocated for kmem caches which are rarely used.
Note for small allocations vmem_alloc() will be internally
converted to kmem_alloc(). Therefore as long as large
allocations are infrequent and short lived the penalty for
using vmem_alloc() is small.
Reviewed-by: Chunwei Chen <david.chen@osnexus.com> Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #5409
Brian Behlendorf [Wed, 30 Nov 2016 21:48:16 +0000 (14:48 -0700)]
Introduce ARC Buffer Data (ABD)
ZFS currently uses ARC buffers which are backed by virtual memory.
While functional, there are some major problems with this approach
which can be observed on all OpenZFS platforms. ABD was designed
to address these issues and includes contributions from OpenZFS
developers from multiple platforms.
While all OpenZFS platforms will benefit from ABD this functionality
is critical for Linux. Unlike the other OpenZFS platforms the Linux
kernel discourages extensive use of virtual memory. The provided
interfaces are not optimized for frequent allocations from the virtual
address space. To maintain good performance a kmem cache is
used which contains relatively long lived slabs backed by virtual
memory. The downside to the approach is that those slabs can
become highly fragmented resulting in an inefficient use of memory.
Another issue is that on 32-bit systems the available virtual
address space in the kernel is only a small fraction of total
system memory. This means the ARC size is highly constrained
which hurts performance and make allocating memory difficult
and OOMs more likely.
ABD is designed to address these issues by using scatter lists
of pages for data buffers. This removes the need for slabs
which resolves the fragmentation issue. It also allows high
memory pages to be allocated which alleviates the virtual
address space pressure on 32-bit systems.
For metadata buffers, which are small, linear ABDs are allocated
from the slab. This is preferable because there are many places
in the code which expect to be able to read from a given offset
in the buffer. Using linear ABDs means none of that code needs
to be modified. The majority of these buffers are allocated with
kmalloc so there's minimal impact of the virtual address space.
Tested-by: Kash Pande <kash@tripleback.net> Tested-by: kernelOfTruth <kerneloftruth@gmail.com> Tested-by: RageLtMan <rageltman@sempervictus> Tested-by: DHE <git@dehacked.net> Reviewed-by: Chunwei Chen <david.chen@osnexus.com> Reviewed-by: Dan Kimmel <dan.kimmel@delphix.com> Reviewed-by: David Quigley <david.quigley@intel.com> Reviewed-by: Gvozden Neskovic <neskovic@gmail.com> Reviewed-by: Tom Caputi <tcaputi@datto.com> Reviewed-by: Isaac Huang <he.huang@intel.com> Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com> Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #3441
Closes #5135
ChaoyuZhang [Wed, 30 Nov 2016 18:27:04 +0000 (02:27 +0800)]
Enable ro_props_001_pos
This script was disabled as the avail/used space changed slightly.
Add sync_pool() and a short delay after snapshots are created to
ensure everything in flight has been written.
Chunwei Chen [Wed, 26 Oct 2016 04:32:23 +0000 (00:32 -0400)]
ABD optimized page allocation code
* Convert ABD to use the Linux Kernel scatterlist implementation
instead of the hand rolled one from illumos.
* Scatter ABDs are preferentially populated with higher order
compound pages from a single zone. Allocation size is
progressively decreased until it can be satisfied without
performing reclaim or compaction.
* An alternate page allocator is provided for kernels older
than 3.6 and for CONFIG_HIGHMEM systems. This allocator
is designed as a fallback for maximum compatibility.
* Extended abdstats to provide visibility in the the allocator.
* Add cached value for PAGESIZE in userspace.
Contributions-by:
Chunwei Chen <david.chen@osnexus.com>
Gvozden Neskovic <neskovic@gmail.com>
Jinshan Xiong <jinshan.xiong@intel.com>
Isaac Huang <he.huang@intel.com>
David Quigley <david.quigley@intel.com>
Brian Behlendorf <behlendorf1@llnl.gov>
Gvozden Neskovic [Wed, 24 Aug 2016 13:51:33 +0000 (15:51 +0200)]
ABD Vectorized raidz
Enable vectorized raidz code on ABD buffers. The avx512f,
avx512bw, neon and aarch64_neonx2 are disabled in this commit.
With the exception of avx512bw these implementations are
updated for ABD in the subsequent commits.
Gvozden Neskovic [Wed, 24 Aug 2016 13:42:51 +0000 (15:42 +0200)]
ABD changes for vectorized RAIDZ
* userspace: aligned buffers. Minimum of 32B alignment is
needed for AVX2. Kernel buffers are aligned 512B or more.
* add abd_get_offset_size() interface
* abd_iter_map(): fix calculation of iter_mapsize
* add abd_raidz_gen_iterate() and abd_raidz_rec_iterate()
Tony Hutter [Tue, 29 Nov 2016 21:45:38 +0000 (13:45 -0800)]
Add -c to zpool iostat & status to run command
This patch adds a command (-c) option to zpool status and zpool iostat. The
-c option allows you to run an arbitrary command on each vdev and display
the first line of output in zpool status/iostat. The environment vars
VDEV_PATH and VDEV_UPATH are set to the vdev's path and "underlying path"
before running the command. For device mapper, multipath, or partitioned
vdevs, VDEV_UPATH is the actual underlying /dev/sd* disk. This can be useful
if the command you're running requires a /dev/sd* device.
The patch also uses /sys/block/<dev>/slaves/ to lookup the underlying device
instead of using libdevmapper. This not only removes the libdevmapper
requirement at build time, but also allows you to resolve device mapper
devices without being root. This means that UDEV_UPATH get set correctly
when running zpool status/iostat as an unprivileged user.
Example:
$ zpool status -c 'echo I am $VDEV_PATH, $VDEV_UPATH'
NAME STATE READ WRITE CKSUM
mypool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
mpatha ONLINE 0 0 0 I am /dev/mapper/mpatha, /dev/sdc
sdb ONLINE 0 0 0 I am /dev/sdb1, /dev/sdb
Reviewed-by: Giuseppe Di Natale <dinatale2@llnl.gov> Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Tony Hutter <hutter2@llnl.gov>
Closes #5368
LOLi [Tue, 29 Nov 2016 19:22:38 +0000 (20:22 +0100)]
Allow zfs unshare <protocol> -a
Allow `zfs unshare <protocol> -a` command to share or unshare all datasets
of a given protocol, nfs or smb.
Additionally, enable most of ZFS Test Suite zfs_share/zfs_unshare test cases.
To work around some Illumos-specific functionalities ($SHARE/$UNSHARE) some
function wrappers were added around them.
Finally, fix and issue in smb_is_share_active() that would leave SMB shares
exported when invoking 'zfs unshare -a'
Reviewed-by: Giuseppe Di Natale <dinatale2@llnl.gov> Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Turbo Fredriksson <turbo@bayour.com> Signed-off-by: loli10K <ezomori.nozomu@gmail.com>
Closes #3238
Closes #5367
Ensure that perf regression tests cleanup properly
Each test in the performance regression test suite
creates a pool and a dataset for use. Unfortunately,
these tests do not cleanup the pool and dataset
correctly once they complete. Each test now kills
fio and iostat, destroys the dataset, and finally
destroys the pool. Each test also now traps the
SIGTERM signal to handle cases where test-runner
kills a test.
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Giuseppe Di Natale <dinatale2@llnl.gov>
Requires-builders: all
Closes #5407
projects / mirror_zfs.git / log