- Unconditionally inject `CONFIG_MODULES` make variable
and `#define CONFIG_MODULES` to Kbuild in `ZFS_LINUX_COMPILE`
autoconf function to emulate loadable kernel modules support.
This allows OpenZFS to perform Linux checks despite
`CONFIG_MODULES=n` in the actual Linux config.
- Add `ZFS_AC_KERNEL_CONFIG_MODULES` check which encompasses
the logic from `ZFS_AC_KERNEL_TEST_MODULE` with additional
diagnostic messages to the user
- Removed `ZFS_AC_KERNEL_TEST_MODULE` as it merely duplicates
every check in `ZFS_AC_KERNEL_CONFIG_DEFINED`
- Moved `ZFS_AC_MODULE_SYMVERS` after `ZFS_AC_KERNEL_CONFIG_DEFINED`
so the user has a chance to see the proper diagnostic from the
steps before.
kbuild: unify modules(_install) for in-tree and external modules
If you attempt to build or install modules ('make modules(_install)'
with CONFIG_MODULES disabled, you will get a clear error message, but
nothing for external module builds.
Factor out the modules and modules_install rules into the common part,
so you will get the same error message when you try to build external
modules with CONFIG_MODULES=n.
Alexander Motin [Tue, 26 Apr 2022 17:44:21 +0000 (13:44 -0400)]
Improve log spacemap load time
Previous flushing algorithm limited only total number of log blocks to
the minimum of 256K and 4x number of metaslabs in the pool. As result,
system with 1500 disks with 1000 metaslabs each, touching several new
metaslabs each TXG could grow spacemap log to huge size without much
benefits. We've observed one of such systems importing pool for about
45 minutes.
This patch improves the situation from five sides:
- By limiting maximum period for each metaslab to be flushed to 1000
TXGs, that effectively limits maximum number of per-TXG spacemap logs
to load to the same number.
- By making flushing more smooth via accounting number of metaslabs
that were touched after the last flush and actually need another flush,
not just ms_unflushed_txg bump.
- By applying zfs_unflushed_log_block_pct to the number of metaslabs
that were touched after the last flush, not all metaslabs in the pool.
- By aggressively prefetching per-TXG spacemap logs up to 16 TXGs in
advance, making log spacemap load process for wide HDD pool CPU-bound,
accelerating it by many times.
- By reducing zfs_unflushed_log_block_max from 256K to 128K, reducing
single-threaded by nature log processing time from ~10 to ~5 minutes.
As further optimization we could skip bumping ms_unflushed_txg for
metaslabs not touched since the last flush, but that would be an
incompatible change, requiring new pool feature.
Reviewed-by: Matthew Ahrens <mahrens@delphix.com> Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Alexander Motin <mav@FreeBSD.org> Sponsored-By: iXsystems, Inc.
Closes #12789
George Amanakis [Tue, 26 Apr 2022 00:25:42 +0000 (02:25 +0200)]
Improve zpool status output, list all affected datasets
Currently, determining which datasets are affected by corruption is
a manual process.
The primary difficulty in reporting the list of affected snapshots is
that since the error was initially found, the snapshot where the error
originally occurred in, may have been deleted. To solve this issue, we
add the ID of the head dataset of the original snapshot which the error
was detected in, to the stored error report. Then any time a filesystem
is deleted, the errors associated with it are deleted as well. Any time
a clone promote occurs, we modify reports associated with the original
head to refer to the new head. The stored error reports are identified
by this head ID, the birth time of the block which the error occurred
in, as well as some information about the error itself are also stored.
Once this information is stored, we can find the set of datasets
affected by an error by walking back the list of snapshots in the given
head until we find one with the appropriate birth txg, and then traverse
through the snapshots of the clone family, terminating a branch if the
block was replaced in a given snapshot. Then we report this information
back to libzfs, and to the zpool status command, where it is displayed
as follows:
pool: test
state: ONLINE
status: One or more devices has experienced an error resulting in data
corruption. Applications may be affected.
action: Restore the file in question if possible. Otherwise restore the
entire pool from backup.
see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-8A
scan: scrub repaired 0B in 00:00:00 with 800 errors on Fri Dec 3
08:27:57 2021
config:
NAME STATE READ WRITE CKSUM
test ONLINE 0 0 0
sdb ONLINE 0 0 1.58K
errors: Permanent errors have been detected in the following files:
A new feature flag is introduced to mark the presence of this change, as
well as promotion and backwards compatibility logic. This is an updated
version of #9175. Rebase required fixing the tests, updating the ABI of
libzfs, updating the man pages, fixing bugs, fixing the error returns,
and updating the old on-disk error logs to the new format when
activating the feature.
Reviewed-by: Matthew Ahrens <mahrens@delphix.com> Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Reviewed-by: Mark Maybee <mark.maybee@delphix.com> Reviewed-by: Tony Hutter <hutter2@llnl.gov> Co-authored-by: TulsiJain <tulsi.jain@delphix.com> Signed-off-by: George Amanakis <gamanakis@gmail.com>
Closes #9175
Closes #12812
Richard Laager [Fri, 22 Apr 2022 22:37:03 +0000 (17:37 -0500)]
zvol_wait: Ignore locked zvols
"When an encrypted zvol is locked the zfs-volume-wait service does not
start. The /sbin/zvol_wait should not wait for links when the volume
has property keystatus=unavailable."
-- https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1888405
Reviewed-by: Tony Hutter <hutter2@llnl.gov> Reviewed-by: Damian Szuberski <szuberskidamian@gmail.com>
Thanks: James Dingwall <james-launchpad@dingwall.me.uk> Signed-off-by: Richard Laager <rlaager@wiktel.com>
Closes #10662
- Add `CONFIG_BLOCK` Linux config requirement to
`ZFS_AC_KERNEL_CONFIG_DEFINED`. OpenZFS won't compile without
that block device support due to large amount of functional
dependencies on it.
- Remove dependency on `groups_alloc()` in
`ZFS_AC_KERNEL_SRC_GROUP_INFO_GID` to circumvent the missing stub
in Linux 4.X kernel headers.
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: szubersk <szuberskidamian@gmail.com>
Closes #13351
contrib; dracut: centralise root= parsing, actually support root=s
So far, everything parsed root= manually, which meant that while
zfs-parse.sh was updated, and supposedly supported + -> ' ' conversion,
it meant nothing
Instead, centralise parsing, and allow:
root=
root=zfs
root=zfs:
root=zfs:AUTO
root=ZFS=data/set
root=zfs:data/set
root=zfs:ZFS=data/set (as a side-effect; allowed but undocumented)
rootfstype=zfs AND root=data/set <=> root=data/set
rootfstype=zfs AND root= <=> root=zfs:AUTO
So rootfstype=zfs /also/ behaves as expected, and + decoding works
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Ahelenia Ziemiańska <nabijaczleweli@nabijaczleweli.xyz>
Closes #13291
contrib: dracut: parse-zfs: stop pretending we support FILESYSTEM=
It was added in the original ae26d0465a ("Add dracut support") commit
in 2011, and was then broken a bit later with the advent of
dracut-zfs-generator, or maybe earlier as part of other churn
Either way, it's broken, and has been in 2.0+ as well, and no-one
complained. Stop pretending we support it at all
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Ahelenia Ziemiańska <nabijaczleweli@nabijaczleweli.xyz>
Closes #13291
It turns out, no, in fact, ZERO_RANGE and PUNCH_HOLE do
have differing semantics in some ways - in particular,
one requires KEEP_SIZE, and the other does not.
Also added a zero-range test to catch this, corrected a flaw
that made the punch-hole test succeed vacuously, and a typo
in file_write.
Alexander Motin [Wed, 20 Apr 2022 23:05:38 +0000 (19:05 -0400)]
FreeBSD: Fix translation from ABD to physical pages
In hypothetical case of non-linear ABD with single segment, multiple
to page size but not aligned to it, vdev_geom_fill_unmap_cb() could
fill one page less into bio_ma array.
I am not sure it is exploitable, but better to be safe than sorry.
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Reviewed-by: Ryan Moeller <ryan@iXsystems.com> Reported-by: Mark Johnston <markj@FreeBSD.org> Signed-off-by: Alexander Motin <mav@FreeBSD.org>
Closes #13345
наб [Thu, 31 Mar 2022 15:20:50 +0000 (17:20 +0200)]
linux: module: weld all but spl.ko into zfs.ko
Originally it was thought it would be useful to split up the kmods
by functionality. This would allow external consumers to only load
what was needed. However, in practice we've never had a case where
this functionality would be needed, and conversely managing multiple
kmods can be awkward. Therefore, this change merges all but the
spl.ko kmod in to a single zfs.ko kmod.
Reviewed-by: Tony Hutter <hutter2@llnl.gov> Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Ahelenia Ziemiańska <nabijaczleweli@nabijaczleweli.xyz>
Closes #13274
Allan Jude [Wed, 20 Apr 2022 20:16:25 +0000 (16:16 -0400)]
Improve the inline descriptions of the ARC module parameters
These are displayed as the descriptions of the sysctl's on FreeBSD
Reviewed-by: George Melikov <mail@gmelikov.ru> Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Allan Jude <allan@klarasystems.com>
Closes #13334
Paul Dagnelie [Tue, 19 Apr 2022 20:55:04 +0000 (13:55 -0700)]
Fix style checking error introduced by zfs-mount changes
Reviewed-by: Matthew Ahrens <mahrens@delphix.com> Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Reviewed-by: Tony Nguyen <tony.nguyen@delphix.com> Signed-off-by: Paul Dagnelie <pcd@delphix.com>
Closes #13347
Brian Behlendorf [Tue, 19 Apr 2022 17:38:04 +0000 (10:38 -0700)]
Linux 5.17 compat: GENHD_FL_EXT_DEVT / GENHD_FL_NO_PART_SCAN
As of the 5.17 kernel the GENHD_FL_EXT_DEVT flag has been removed
and the GENHD_FL_NO_PART_SCAN flag renamed GENHD_FL_NO_PART. Update
zvol_alloc() to set GENHD_FL_NO_PART for the newer kernels which
is sufficient. The behavior for prior kernels remains unchanged.
1ebe2e5f ("block: remove GENHD_FL_EXT_DEVT") 46e7eac6 ("block: rename GENHD_FL_NO_PART_SCAN to GENHD_FL_NO_PART")
Reviewed-by: Tony Hutter <hutter2@llnl.gov> Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #13294
Closes #13297
Brian Behlendorf [Tue, 12 Apr 2022 00:26:01 +0000 (17:26 -0700)]
ZTS: Retry auto_spare_multiple.ksh
The auto_spare_multiple.ksh test may incorrectly fail for a similar
reason as the auto_spare_shared.ksh test. Add it to known list of
exceptions which should be retried to prevent failures in the CI.
Reviewed-by: George Melikov <mail@gmelikov.ru> Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #13318
Brian Behlendorf [Mon, 11 Apr 2022 23:18:51 +0000 (16:18 -0700)]
ZTS: Retry redundancy_draid_spare[1,3].ksh
The redundancy_draid_spare1.ksh and redundancy_draid_spare3.ksh test
cases are a little to strict for the sequential resilver case. While
unlikely it is possible that a handful of correctable checksum errors
will be reported resulting in a test failure. Update the zts-report.py
script to allow this the test case to be retried if requested.
Reviewed-by: George Melikov <mail@gmelikov.ru> Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #13318
Mark Johnston [Thu, 7 Apr 2022 21:13:18 +0000 (17:13 -0400)]
FreeBSD: Return Mach error codes from VOP_(GET|PUT)PAGES
FreeBSD's memory management system uses its own error numbers and gets
confused when these VOPs return EIO.
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Reviewed-by: Ryan Moeller <ryan@iXsystems.com> Reported-by: Peter Holm <pho@FreeBSD.org> Signed-off-by: Mark Johnston <markj@FreeBSD.org>
Closes #13311
Mark Johnston [Thu, 7 Apr 2022 21:11:00 +0000 (17:11 -0400)]
FreeBSD: Parameterize ZFS_ENTER/ZFS_VERIFY_VP with an error code
For legacy reasons, a couple of VOPs have to return error numbers that
don't come from the usual errno namespace. To handle the cases where
ZFS_ENTER or ZFS_VERIFY_ZP fail, we need to be able to override the
default error return value of EIO. Extend the macros to permit this.
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Reviewed-by: Ryan Moeller <ryan@iXsystems.com> Signed-off-by: Mark Johnston <markj@FreeBSD.org>
Closes #13311
The logic above skips completely `mount.zfs` and prevents us
from reading filesystem properties and applying mount options.
For comparison, the coreutils `mount(8)` implementation does:
```
openat(AT_FDCWD, "/proc/filesystems", O_RDONLY|O_CLOEXEC) = 3
// figure out that zfs is a `nodev` filesystem and look for a helper
newfstatat(AT_FDCWD, "/sbin/mount.zfs" ...) = 0
execve("/sbin/mount.zfs" ...) = 0
```
Using `mount.zfs` in initramfs would help circumvent deficiencies
of some of `mount(8)` implementations. `mount -t zfs` translates
to `mount.zfs` invocation, except for cases when explicitly disabled
by `-i`.
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: szubersk <szuberskidamian@gmail.com>
Closes #13305
cmd: zed: rc: drop "should be owned by root and 0600"
It doesn't matter, 0600 are Weird Permissions, and it's even weirder to
spec them for no reason ‒ it's perfectly fine if it's the usual 0:0 644,
or literally anything else, so long as unprivileged users can't edit it
(which (a) 644 accomplishes and (b) is at the administrator's
discretion, it's not unheard of to have adm users and having it
be 664 in that case is just as good; it's not our place to say)
наб [Sat, 26 Mar 2022 10:19:17 +0000 (11:19 +0100)]
tests: clean out more temporary files
What remains is a bunch of anonymous untraceable /tmp/tmp.XXXXXXXXXX
files and bak.root.receive.staff1.3835 from an error branch, testdir.1,
testdir.3, and testroot454470 (with children) in testroot
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Reviewed-by: John Kennedy <john.kennedy@delphix.com> Reviewed-by: Ryan Moeller <ryan@iXsystems.com> Signed-off-by: Ahelenia Ziemiańska <nabijaczleweli@nabijaczleweli.xyz>
Closes #13259
наб [Wed, 23 Mar 2022 01:21:56 +0000 (02:21 +0100)]
tests: pam: use absolute path to module .so
This is a valid configuration and both (a) skips the tests if it's
unbuilt/not installed and (b) makes it work even if installed outside
the system directory (like in /u/l/l/s instead of /l/s)
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Reviewed-by: John Kennedy <john.kennedy@delphix.com> Reviewed-by: Ryan Moeller <ryan@iXsystems.com> Signed-off-by: Ahelenia Ziemiańska <nabijaczleweli@nabijaczleweli.xyz>
Closes #13259
наб [Wed, 23 Mar 2022 20:07:33 +0000 (21:07 +0100)]
tests: rsend/send-c_props: make it chooch
Original error:
23:47:40.59 SUCCESS: eval zfs receive -dFv testpool2 < /mnt/testroot/backdir-rsend/pool-final-p
23:47:40.61 1,23d0
23:47:40.61 < type filesystem -
23:47:40.61 < origin POOL@psnap -
23:47:40.61 < volblocksize - -
23:47:40.61 < acltype nfsv4 inherited from POOL
23:47:40.61 < dnodesize legacy inherited from POOL
23:47:40.61 < atime off local
23:47:40.61 < canmount off local
23:47:40.61 < checksum off local
23:47:40.61 < compression off local
23:47:40.61 < copies 3 local
23:47:40.61 < devices off local
23:47:40.61 < exec off local
23:47:40.61 < quota none default
23:47:40.61 < readonly on local
23:47:40.61 < recordsize 128K local
23:47:40.61 < reservation none default
23:47:40.61 < setuid off local
23:47:40.61 < snapdir hidden local
23:47:40.61 < version 5 -
23:47:40.61 < volsize - -
23:47:40.61 < xattr off local
23:47:40.61 < mountpoint /PREFIX inherited from POOL
23:47:40.61 < jailed on local
23:47:40.62 cannot open 'testpool2/pclone': dataset does not exist
23:47:40.62 ERROR: cmp_ds_prop testpool/pclone testpool2/pclone exited 1
So: (a) actually send all the datasets in -p mode and
(b) drop origin for clones sent with -p:
This fixes rsend_012_pos:
20:28:50.50 SUCCESS: eval zfs receive -d -F testpool2 < /mnt/testroot/backdir-rsend/pool-final-R
20:28:50.53 4,6c4,6
20:28:50.53 < acltype off local
20:28:50.53 < dnodesize 4k local
20:28:50.53 < atime off local
20:28:50.53 ---
20:28:50.53 > acltype off received
20:28:50.53 > dnodesize 4k received
20:28:50.53 > atime off received
20:28:50.53 8,13c8,13
20:28:50.53 < checksum sha256 local
20:28:50.53 < compression off local
20:28:50.53 < copies 2 local
20:28:50.53 < devices on local
20:28:50.53 < exec on local
20:28:50.53 < quota 1G local
20:28:50.53 ---
20:28:50.53 > checksum sha256 received
20:28:50.53 > compression off received
20:28:50.53 > copies 2 received
20:28:50.53 > devices on received
20:28:50.53 > exec on received
20:28:50.53 > quota 1G received
20:28:50.53 15c15
20:28:50.53 < recordsize 128K local
20:28:50.53 ---
20:28:50.53 > recordsize 128K received
20:28:50.53 17,18c17,18
20:28:50.53 < setuid off local
20:28:50.53 < snapdir visible local
20:28:50.53 ---
20:28:50.53 > setuid off received
20:28:50.53 > snapdir visible received
20:28:50.53 ERROR: cmp_ds_prop testpool testpool2 exited 1
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Reviewed-by: John Kennedy <john.kennedy@delphix.com> Reviewed-by: Ryan Moeller <ryan@iXsystems.com> Signed-off-by: Ahelenia Ziemiańska <nabijaczleweli@nabijaczleweli.xyz>
Closes #13250
Closes #13259