ztest: split block reconstruction

Increase the default allowed number of reconstruction attempts.
There's not an exact right number for this setting.  It needs
to be set large enough to cover any realistic failure scenarios
and small enough to avoid stalling the IO pipeline and invoking
the dead man detection.

The current value of 256 was empirically determined to be too
low based on multi-day runs of ztest.  The fault injection code
would inject more damage than could be reconstructed given the
relatively small number of attempts.  However, in all observed
cases the block could be reconstructed using a slightly higher
limit.

Based on local testing increasing the default value to 4096 was
determined to strike the best balance.  Checking all combinations
takes less than 10s in the worst case, and has so far eliminated
the vast majority of false positives detected by ztest.  This
delay is roughly on par with how long retries may be performed
to a misbehaving HDD and was deemed to be reasonable.  Better to
err on the side of a brief delay rather than fail to reconstruct
the data.

Lastly, the -Y flag has been added to zdb to make it easy to try all
possible combinations when performing split block reconstruction.
For badly damaged blocks with 18 splits, they can be fully enumerated
within a few minutes.  This has been done to ensure permanent errors
are never incorrectly reported when ztest verifies the pool with zdb.

Reviewed by: Tom Caputi <tcaputi@datto.com>
Reviewed by: Matt Ahrens <mahrens@delphix.com>
Reviewed by: Serapheim Dimitropoulos <serapheim@delphix.com>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #8271

diff --git a/cmd/zdb/zdb.c b/cmd/zdb/zdb.c
index 1a9303d12ceea10ab025e070fd4e8869f6a01af8..c436139a22fbc78762e693e8a0d8abc97e59b007 100644 (file)
--- a/cmd/zdb/zdb.c
+++ b/cmd/zdb/zdb.c
@@ -100,6 +100,7 @@ extern int zfs_recover;
 extern uint64_t zfs_arc_max, zfs_arc_meta_limit;
 extern int zfs_vdev_async_read_max_active;
 extern boolean_t spa_load_verify_dryrun;
+extern int zfs_reconstruct_indirect_combinations_max;
 
 static const char cmdname[] = "zdb";
 uint8_t dump_opt[256];
@@ -215,6 +216,8 @@ usage(void)
            "dump all read blocks into specified directory\n");
        (void) fprintf(stderr, "        -X attempt extreme rewind (does not "
            "work with dataset)\n");
+       (void) fprintf(stderr, "        -Y attempt all reconstruction "
+           "combinations for split blocks\n");
        (void) fprintf(stderr, "Specify an option more than once (e.g. -bb) "
            "to make only that option verbose\n");
        (void) fprintf(stderr, "Default is to dump everything non-verbosely\n");
@@ -5871,7 +5874,7 @@ main(int argc, char **argv)
                spa_config_path = spa_config_path_env;
 
        while ((c = getopt(argc, argv,
-           "AbcCdDeEFGhiI:klLmMo:Op:PqRsSt:uU:vVx:X")) != -1) {
+           "AbcCdDeEFGhiI:klLmMo:Op:PqRsSt:uU:vVx:XY")) != -1) {
                switch (c) {
                case 'b':
                case 'c':
@@ -5903,6 +5906,10 @@ main(int argc, char **argv)
                case 'X':
                        dump_opt[c]++;
                        break;
+               case 'Y':
+                       zfs_reconstruct_indirect_combinations_max = INT_MAX;
+                       zfs_deadman_enabled = 0;
+                       break;
                /* NB: Sort single match options below. */
                case 'I':
                        max_inflight = strtoull(optarg, NULL, 0);

diff --git a/cmd/ztest/ztest.c b/cmd/ztest/ztest.c
index 678e29ff6681e45f9583bf502d07f515f2510066..3686fab7679d0c142f30525ec144404b06db46c3 100644 (file)
--- a/cmd/ztest/ztest.c
+++ b/cmd/ztest/ztest.c
@@ -6447,8 +6447,7 @@ ztest_run_zdb(char *pool)
        ztest_get_zdb_bin(bin, len);
 
        (void) sprintf(zdb,
-           "%s -bcc%s%s -G -d -U %s "
-           "-o zfs_reconstruct_indirect_combinations_max=65536 %s",
+           "%s -bcc%s%s -G -d -Y -U %s %s",
            bin,
            ztest_opts.zo_verbose >= 3 ? "s" : "",
            ztest_opts.zo_verbose >= 4 ? "v" : "",

diff --git a/man/man5/zfs-module-parameters.5 b/man/man5/zfs-module-parameters.5
index 1dbf865f435f0c1e991321aa328afe44c90193e6..c9dfceb7eca692810f11223ba8718d756472e044 100644 (file)
--- a/man/man5/zfs-module-parameters.5
+++ b/man/man5/zfs-module-parameters.5
@@ -2025,7 +2025,7 @@ combinations each time the block is accessed.  This allows all segment
 copies to participate fairly in the reconstruction when all combinations
 cannot be checked and prevents repeated use of one bad copy.
 .sp
-Default value: \fB256\fR.
+Default value: \fB4096\fR.
 .RE
 
 .sp

diff --git a/man/man8/zdb.8 b/man/man8/zdb.8
index f00d9c0cfef730f371f66b8f96df0185867b51f1..79d6f8af7082826068cc990549368eba916e8e93 100644 (file)
--- a/man/man8/zdb.8
+++ b/man/man8/zdb.8
@@ -23,7 +23,7 @@
 .Nd display zpool debugging and consistency information
 .Sh SYNOPSIS
 .Nm
-.Op Fl AbcdDFGhikLMPsvX
+.Op Fl AbcdDFGhikLMPsvXY
 .Op Fl e Oo Fl V Oc Op Fl p Ar path ...
 .Op Fl I Ar inflight I/Os
 .Oo Fl o Ar var Ns = Ns Ar value Oc Ns ...
@@ -50,7 +50,7 @@
 .Ar device
 .Nm
 .Fl m
-.Op Fl AFLPX
+.Op Fl AFLPXY
 .Op Fl e Oo Fl V Oc Op Fl p Ar path ...
 .Op Fl t Ar txg
 .Op Fl U Ar cache
@@ -349,6 +349,10 @@ Attempt
 transaction rewind, that is attempt the same recovery as
 .Fl F
 but read transactions otherwise deemed too old.
+.It Fl Y
+Attempt all possible combinations when reconstructing indirect split blocks.
+This flag disables the individual I/O deadman timer in order to allow as
+much time as required for the attempted reconstruction.
 .El
 .Pp
 Specifying a display option more than once enables verbosity for only that

diff --git a/module/zfs/vdev_indirect.c b/module/zfs/vdev_indirect.c
index d0725add8922ed0d7ac1ce15df1d0b9a89bfdd92..2f8268f0fab663b76944593e7353d2c047daf0a1 100644 (file)
--- a/module/zfs/vdev_indirect.c
+++ b/module/zfs/vdev_indirect.c
@@ -213,8 +213,7 @@ int zfs_condense_indirect_commit_entry_delay_ms = 0;
  * copies to participate fairly in the reconstruction when all combinations
  * cannot be checked and prevents repeated use of one bad copy.
  */
-int zfs_reconstruct_indirect_combinations_max = 256;
-
+int zfs_reconstruct_indirect_combinations_max = 4096;
 
 /*
  * Enable to simulate damaged segments and validate reconstruction.  This