Documentation/fault-injection/fault-injection.txt

   1 Fault injection capabilities infrastructure
   2 ===========================================
   3
   4 See also drivers/md/faulty.c and "every_nth" module option for scsi_debug.
   5
   6
   7 Available fault injection capabilities
   8 --------------------------------------
   9
  10 o failslab
  11
  12   injects slab allocation failures. (kmalloc(), kmem_cache_alloc(), ...)
  13
  14 o fail_page_alloc
  15
  16   injects page allocation failures. (alloc_pages(), get_free_pages(), ...)
  17
  18 o fail_futex
  19
  20   injects futex deadlock and uaddr fault errors.
  21
  22 o fail_make_request
  23
  24   injects disk IO errors on devices permitted by setting
  25   /sys/block/<device>/make-it-fail or
  26   /sys/block/<device>/<partition>/make-it-fail. (generic_make_request())
  27
  28 o fail_mmc_request
  29
  30   injects MMC data errors on devices permitted by setting
  31   debugfs entries under /sys/kernel/debug/mmc0/fail_mmc_request
  32
  33 Configure fault-injection capabilities behavior
  34 -----------------------------------------------
  35
  36 o debugfs entries
  37
  38 fault-inject-debugfs kernel module provides some debugfs entries for runtime
  39 configuration of fault-injection capabilities.
  40
  41 - /sys/kernel/debug/fail*/probability:
  42
  43         likelihood of failure injection, in percent.
  44         Format: <percent>
  45
  46         Note that one-failure-per-hundred is a very high error rate
  47         for some testcases.  Consider setting probability=100 and configure
  48         /sys/kernel/debug/fail*/interval for such testcases.
  49
  50 - /sys/kernel/debug/fail*/interval:
  51
  52         specifies the interval between failures, for calls to
  53         should_fail() that pass all the other tests.
  54
  55         Note that if you enable this, by setting interval>1, you will
  56         probably want to set probability=100.
  57
  58 - /sys/kernel/debug/fail*/times:
  59
  60         specifies how many times failures may happen at most.
  61         A value of -1 means "no limit".
  62
  63 - /sys/kernel/debug/fail*/space:
  64
  65         specifies an initial resource "budget", decremented by "size"
  66         on each call to should_fail(,size).  Failure injection is
  67         suppressed until "space" reaches zero.
  68
  69 - /sys/kernel/debug/fail*/verbose
  70
  71         Format: { 0 | 1 | 2 }
  72         specifies the verbosity of the messages when failure is
  73         injected.  '0' means no messages; '1' will print only a single
  74         log line per failure; '2' will print a call trace too -- useful
  75         to debug the problems revealed by fault injection.
  76
  77 - /sys/kernel/debug/fail*/task-filter:
  78
  79         Format: { 'Y' | 'N' }
  80         A value of 'N' disables filtering by process (default).
  81         Any positive value limits failures to only processes indicated by
  82         /proc/<pid>/make-it-fail==1.
  83
  84 - /sys/kernel/debug/fail*/require-start:
  85 - /sys/kernel/debug/fail*/require-end:
  86 - /sys/kernel/debug/fail*/reject-start:
  87 - /sys/kernel/debug/fail*/reject-end:
  88
  89         specifies the range of virtual addresses tested during
  90         stacktrace walking.  Failure is injected only if some caller
  91         in the walked stacktrace lies within the required range, and
  92         none lies within the rejected range.
  93         Default required range is [0,ULONG_MAX) (whole of virtual address space).
  94         Default rejected range is [0,0).
  95
  96 - /sys/kernel/debug/fail*/stacktrace-depth:
  97
  98         specifies the maximum stacktrace depth walked during search
  99         for a caller within [require-start,require-end) OR
 100         [reject-start,reject-end).
 101
 102 - /sys/kernel/debug/fail_page_alloc/ignore-gfp-highmem:
 103
 104         Format: { 'Y' | 'N' }
 105         default is 'N', setting it to 'Y' won't inject failures into
 106         highmem/user allocations.
 107
 108 - /sys/kernel/debug/failslab/ignore-gfp-wait:
 109 - /sys/kernel/debug/fail_page_alloc/ignore-gfp-wait:
 110
 111         Format: { 'Y' | 'N' }
 112         default is 'N', setting it to 'Y' will inject failures
 113         only into non-sleep allocations (GFP_ATOMIC allocations).
 114
 115 - /sys/kernel/debug/fail_page_alloc/min-order:
 116
 117         specifies the minimum page allocation order to be injected
 118         failures.
 119
 120 - /sys/kernel/debug/fail_futex/ignore-private:
 121
 122         Format: { 'Y' | 'N' }
 123         default is 'N', setting it to 'Y' will disable failure injections
 124         when dealing with private (address space) futexes.
 125
 126 o Boot option
 127
 128 In order to inject faults while debugfs is not available (early boot time),
 129 use the boot option:
 130
 131         failslab=
 132         fail_page_alloc=
 133         fail_make_request=
 134         fail_futex=
 135         mmc_core.fail_request=<interval>,<probability>,<space>,<times>
 136
 137 o proc entries
 138
 139 - /proc/self/task/<current-tid>/fail-nth:
 140
 141         Write to this file of integer N makes N-th call in the task fail.
 142         Read from this file returns a integer value. A value of '0' indicates
 143         that the fault setup with a previous write to this file was injected.
 144         A positive integer N indicates that the fault wasn't yet injected.
 145         Note that this file enables all types of faults (slab, futex, etc).
 146         This setting takes precedence over all other generic debugfs settings
 147         like probability, interval, times, etc. But per-capability settings
 148         (e.g. fail_futex/ignore-private) take precedence over it.
 149
 150         This feature is intended for systematic testing of faults in a single
 151         system call. See an example below.
 152
 153 How to add new fault injection capability
 154 -----------------------------------------
 155
 156 o #include <linux/fault-inject.h>
 157
 158 o define the fault attributes
 159
 160   DECLARE_FAULT_INJECTION(name);
 161
 162   Please see the definition of struct fault_attr in fault-inject.h
 163   for details.
 164
 165 o provide a way to configure fault attributes
 166
 167 - boot option
 168
 169   If you need to enable the fault injection capability from boot time, you can
 170   provide boot option to configure it. There is a helper function for it:
 171
 172         setup_fault_attr(attr, str);
 173
 174 - debugfs entries
 175
 176   failslab, fail_page_alloc, and fail_make_request use this way.
 177   Helper functions:
 178
 179         fault_create_debugfs_attr(name, parent, attr);
 180
 181 - module parameters
 182
 183   If the scope of the fault injection capability is limited to a
 184   single kernel module, it is better to provide module parameters to
 185   configure the fault attributes.
 186
 187 o add a hook to insert failures
 188
 189   Upon should_fail() returning true, client code should inject a failure.
 190
 191         should_fail(attr, size);
 192
 193 Application Examples
 194 --------------------
 195
 196 o Inject slab allocation failures into module init/exit code
 197
 198 #!/bin/bash
 199
 200 FAILTYPE=failslab
 201 echo Y > /sys/kernel/debug/$FAILTYPE/task-filter
 202 echo 10 > /sys/kernel/debug/$FAILTYPE/probability
 203 echo 100 > /sys/kernel/debug/$FAILTYPE/interval
 204 echo -1 > /sys/kernel/debug/$FAILTYPE/times
 205 echo 0 > /sys/kernel/debug/$FAILTYPE/space
 206 echo 2 > /sys/kernel/debug/$FAILTYPE/verbose
 207 echo 1 > /sys/kernel/debug/$FAILTYPE/ignore-gfp-wait
 208
 209 faulty_system()
 210 {
 211         bash -c "echo 1 > /proc/self/make-it-fail && exec $*"
 212 }
 213
 214 if [ $# -eq 0 ]
 215 then
 216         echo "Usage: $0 modulename [ modulename ... ]"
 217         exit 1
 218 fi
 219
 220 for m in $*
 221 do
 222         echo inserting $m...
 223         faulty_system modprobe $m
 224
 225         echo removing $m...
 226         faulty_system modprobe -r $m
 227 done
 228
 229 ------------------------------------------------------------------------------
 230
 231 o Inject page allocation failures only for a specific module
 232
 233 #!/bin/bash
 234
 235 FAILTYPE=fail_page_alloc
 236 module=$1
 237
 238 if [ -z $module ]
 239 then
 240         echo "Usage: $0 <modulename>"
 241         exit 1
 242 fi
 243
 244 modprobe $module
 245
 246 if [ ! -d /sys/module/$module/sections ]
 247 then
 248         echo Module $module is not loaded
 249         exit 1
 250 fi
 251
 252 cat /sys/module/$module/sections/.text > /sys/kernel/debug/$FAILTYPE/require-start
 253 cat /sys/module/$module/sections/.data > /sys/kernel/debug/$FAILTYPE/require-end
 254
 255 echo N > /sys/kernel/debug/$FAILTYPE/task-filter
 256 echo 10 > /sys/kernel/debug/$FAILTYPE/probability
 257 echo 100 > /sys/kernel/debug/$FAILTYPE/interval
 258 echo -1 > /sys/kernel/debug/$FAILTYPE/times
 259 echo 0 > /sys/kernel/debug/$FAILTYPE/space
 260 echo 2 > /sys/kernel/debug/$FAILTYPE/verbose
 261 echo 1 > /sys/kernel/debug/$FAILTYPE/ignore-gfp-wait
 262 echo 1 > /sys/kernel/debug/$FAILTYPE/ignore-gfp-highmem
 263 echo 10 > /sys/kernel/debug/$FAILTYPE/stacktrace-depth
 264
 265 trap "echo 0 > /sys/kernel/debug/$FAILTYPE/probability" SIGINT SIGTERM EXIT
 266
 267 echo "Injecting errors into the module $module... (interrupt to stop)"
 268 sleep 1000000
 269
 270 Tool to run command with failslab or fail_page_alloc
 271 ----------------------------------------------------
 272 In order to make it easier to accomplish the tasks mentioned above, we can use
 273 tools/testing/fault-injection/failcmd.sh.  Please run a command
 274 "./tools/testing/fault-injection/failcmd.sh --help" for more information and
 275 see the following examples.
 276
 277 Examples:
 278
 279 Run a command "make -C tools/testing/selftests/ run_tests" with injecting slab
 280 allocation failure.
 281
 282         # ./tools/testing/fault-injection/failcmd.sh \
 283                 -- make -C tools/testing/selftests/ run_tests
 284
 285 Same as above except to specify 100 times failures at most instead of one time
 286 at most by default.
 287
 288         # ./tools/testing/fault-injection/failcmd.sh --times=100 \
 289                 -- make -C tools/testing/selftests/ run_tests
 290
 291 Same as above except to inject page allocation failure instead of slab
 292 allocation failure.
 293
 294         # env FAILCMD_TYPE=fail_page_alloc \
 295                 ./tools/testing/fault-injection/failcmd.sh --times=100 \
 296                 -- make -C tools/testing/selftests/ run_tests
 297
 298 Systematic faults using fail-nth
 299 ---------------------------------
 300
 301 The following code systematically faults 0-th, 1-st, 2-nd and so on
 302 capabilities in the socketpair() system call.
 303
 304 #include <sys/types.h>
 305 #include <sys/stat.h>
 306 #include <sys/socket.h>
 307 #include <sys/syscall.h>
 308 #include <fcntl.h>
 309 #include <unistd.h>
 310 #include <string.h>
 311 #include <stdlib.h>
 312 #include <stdio.h>
 313 #include <errno.h>
 314
 315 int main()
 316 {
 317         int i, err, res, fail_nth, fds[2];
 318         char buf[128];
 319
 320         system("echo N > /sys/kernel/debug/failslab/ignore-gfp-wait");
 321         sprintf(buf, "/proc/self/task/%ld/fail-nth", syscall(SYS_gettid));
 322         fail_nth = open(buf, O_RDWR);
 323         for (i = 1;; i++) {
 324                 sprintf(buf, "%d", i);
 325                 write(fail_nth, buf, strlen(buf));
 326                 res = socketpair(AF_LOCAL, SOCK_STREAM, 0, fds);
 327                 err = errno;
 328                 pread(fail_nth, buf, sizeof(buf), 0);
 329                 if (res == 0) {
 330                         close(fds[0]);
 331                         close(fds[1]);
 332                 }
 333                 printf("%d-th fault %c: res=%d/%d\n", i, atoi(buf) ? 'N' : 'Y',
 334                         res, err);
 335                 if (atoi(buf))
 336                         break;
 337         }
 338         return 0;
 339 }
 340
 341 An example output:
 342
 343 1-th fault Y: res=-1/23
 344 2-th fault Y: res=-1/23
 345 3-th fault Y: res=-1/12
 346 4-th fault Y: res=-1/12
 347 5-th fault Y: res=-1/23
 348 6-th fault Y: res=-1/23
 349 7-th fault Y: res=-1/23
 350 8-th fault Y: res=-1/12
 351 9-th fault Y: res=-1/12
 352 10-th fault Y: res=-1/12
 353 11-th fault Y: res=-1/12
 354 12-th fault Y: res=-1/12
 355 13-th fault Y: res=-1/12
 356 14-th fault Y: res=-1/12
 357 15-th fault Y: res=-1/12
 358 16-th fault N: res=0/12