ZTS: Provide for nested cleanup routines

Shared test library functions lack a simple way to ensure proper
cleanup in the event of a failure.  The `log_onexit` cleanup pattern
cannot be used in library functions because it uses one global
variable to store the cleanup command.

An example of where this is a serious issue is when a tunable that
artifically stalls kernel progress gets activated and then some check
fails.  Unless the caller knows about the tunable and sets it back,
the system will be left in a bad state.

To solve this problem, turn the global cleanup variable into a stack.
Provide push and pop functions to add additional cleanup steps and
remove them after it is safe again.

The first use of this new functionality is in attempt_during_removal,
which sets REMOVAL_SUSPEND_PROGRESS.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: John Kennedy <john.kennedy@delphix.com>
Signed-off-by: Ryan Moeller <ryan@iXsystems.com>
Closes #10080

diff --git a/tests/test-runner/include/logapi.shlib b/tests/test-runner/include/logapi.shlib
index cd7982a94a0b0696bb8ac45f95d822687376a46d..3c8324d57c5f246531daaf04dcd137202f16a537 100644 (file)
--- a/tests/test-runner/include/logapi.shlib
+++ b/tests/test-runner/include/logapi.shlib
@@ -281,7 +281,23 @@ function log_pos
 
 function log_onexit
 {
-       _CLEANUP="$@"
+       _CLEANUP=("$*")
+}
+
+# Push an exit handler on the cleanup stack
+#
+# $@ - function(s) to perform on exit
+
+function log_onexit_push
+{
+       _CLEANUP+=("$*")
+}
+
+# Pop an exit handler off the cleanup stack
+
+function log_onexit_pop
+{
+       _CLEANUP=("${_CLEANUP[@]:0:${#_CLEANUP[@]}-1}")
 }
 
 #
@@ -425,12 +441,14 @@ function _endlog
                _execute_testfail_callbacks
        fi
 
-       if [[ -n $_CLEANUP ]] ; then
-               typeset cleanup=$_CLEANUP
-               log_onexit ""
+       typeset stack=("${_CLEANUP[@]}")
+       log_onexit ""
+       typeset i=${#stack[@]}
+       while (( i-- )); do
+               typeset cleanup="${stack[i]}"
                log_note "Performing local cleanup via log_onexit ($cleanup)"
                $cleanup
-       fi
+       done
 
        exit $exitcode
 }

diff --git a/tests/zfs-tests/tests/functional/removal/removal.kshlib b/tests/zfs-tests/tests/functional/removal/removal.kshlib
index e1f43fbe76c49ac7fef5c7af035ac600b14fa2fd..231991e82c574a7cfb8b0bc76cd57aa9bc8abb6b 100644 (file)
--- a/tests/zfs-tests/tests/functional/removal/removal.kshlib
+++ b/tests/zfs-tests/tests/functional/removal/removal.kshlib
@@ -60,6 +60,7 @@ function attempt_during_removal # pool disk callback [args]
        typeset callback=$3
 
        shift 3
+       log_onexit_push set_tunable32 REMOVAL_SUSPEND_PROGRESS 0
        set_tunable32 REMOVAL_SUSPEND_PROGRESS 1
 
        log_must zpool remove $pool $disk
@@ -80,6 +81,7 @@ function attempt_during_removal # pool disk callback [args]
        log_must is_pool_removing $pool
 
        set_tunable32 REMOVAL_SUSPEND_PROGRESS 0
+       log_onexit_pop
 
        log_must wait_for_removal $pool
        log_mustnot vdevs_in_pool $pool $disk