]>
Commit | Line | Data |
---|---|---|
f9d7b4b3 SH |
1 | Block I/O error injection using blkdebug |
2 | ---------------------------------------- | |
3 | Copyright (C) 2014 Red Hat Inc | |
4 | ||
5 | This work is licensed under the terms of the GNU GPL, version 2 or later. See | |
6 | the COPYING file in the top-level directory. | |
7 | ||
8 | The blkdebug block driver is a rule-based error injection engine. It can be | |
9 | used to exercise error code paths in block drivers including ENOSPC (out of | |
10 | space) and EIO. | |
11 | ||
12 | This document gives an overview of the features available in blkdebug. | |
13 | ||
14 | Background | |
15 | ---------- | |
16 | Block drivers have many error code paths that handle I/O errors. Image formats | |
17 | are especially complex since metadata I/O errors during cluster allocation or | |
18 | while updating tables happen halfway through request processing and require | |
19 | discipline to keep image files consistent. | |
20 | ||
21 | Error injection allows test cases to trigger I/O errors at specific points. | |
22 | This way, all error paths can be tested to make sure they are correct. | |
23 | ||
24 | Rules | |
25 | ----- | |
26 | The blkdebug block driver takes a list of "rules" that tell the error injection | |
27 | engine when to fail an I/O request. | |
28 | ||
29 | Each I/O request is evaluated against the rules. If a rule matches the request | |
30 | then its "action" is executed. | |
31 | ||
32 | Rules can be placed in a configuration file; the configuration file | |
33 | follows the same .ini-like format used by QEMU's -readconfig option, and | |
34 | each section of the file represents a rule. | |
35 | ||
36 | The following configuration file defines a single rule: | |
37 | ||
38 | $ cat blkdebug.conf | |
39 | [inject-error] | |
40 | event = "read_aio" | |
41 | errno = "28" | |
42 | ||
43 | This rule fails all aio read requests with ENOSPC (28). Note that the errno | |
44 | value depends on the host. On Linux, see | |
45 | /usr/include/asm-generic/errno-base.h for errno values. | |
46 | ||
47 | Invoke QEMU as follows: | |
48 | ||
49 | $ qemu-system-x86_64 | |
50 | -drive if=none,cache=none,file=blkdebug:blkdebug.conf:test.img,id=drive0 \ | |
51 | -device virtio-blk-pci,drive=drive0,id=virtio-blk-pci0 | |
52 | ||
53 | Rules support the following attributes: | |
54 | ||
55 | event - which type of operation to match (e.g. read_aio, write_aio, | |
56 | flush_to_os, flush_to_disk). See the "Events" section for | |
57 | information on events. | |
58 | ||
59 | state - (optional) the engine must be in this state number in order for this | |
60 | rule to match. See the "State transitions" section for information | |
61 | on states. | |
62 | ||
63 | errno - the numeric errno value to return when a request matches this rule. | |
64 | The errno values depend on the host since the numeric values are not | |
65 | standarized in the POSIX specification. | |
66 | ||
67 | sector - (optional) a sector number that the request must overlap in order to | |
68 | match this rule | |
69 | ||
70 | once - (optional, default "off") only execute this action on the first | |
71 | matching request | |
72 | ||
7c84b1b8 MA |
73 | immediately - (optional, default "off") return a NULL BlockAIOCB |
74 | pointer and fail without an errno instead. This | |
75 | exercises the code path where BlockAIOCB fails and the | |
097310b5 | 76 | caller's BlockCompletionFunc is not invoked. |
f9d7b4b3 SH |
77 | |
78 | Events | |
79 | ------ | |
80 | Block drivers provide information about the type of I/O request they are about | |
81 | to make so rules can match specific types of requests. For example, the qcow2 | |
82 | block driver tells blkdebug when it accesses the L1 table so rules can match | |
83 | only L1 table accesses and not other metadata or guest data requests. | |
84 | ||
85 | The core events are: | |
86 | ||
87 | read_aio - guest data read | |
88 | ||
89 | write_aio - guest data write | |
90 | ||
91 | flush_to_os - write out unwritten block driver state (e.g. cached metadata) | |
92 | ||
93 | flush_to_disk - flush the host block device's disk cache | |
94 | ||
95 | See block/blkdebug.c:event_names[] for the full list of events. You may need | |
96 | to grep block driver source code to understand the meaning of specific events. | |
97 | ||
98 | State transitions | |
99 | ----------------- | |
100 | There are cases where more power is needed to match a particular I/O request in | |
101 | a longer sequence of requests. For example: | |
102 | ||
103 | write_aio | |
104 | flush_to_disk | |
105 | write_aio | |
106 | ||
107 | How do we match the 2nd write_aio but not the first? This is where state | |
108 | transitions come in. | |
109 | ||
110 | The error injection engine has an integer called the "state" that always starts | |
111 | initialized to 1. The state integer is internal to blkdebug and cannot be | |
112 | observed from outside but rules can interact with it for powerful matching | |
113 | behavior. | |
114 | ||
115 | Rules can be conditional on the current state and they can transition to a new | |
116 | state. | |
117 | ||
118 | When a rule's "state" attribute is non-zero then the current state must equal | |
119 | the attribute in order for the rule to match. | |
120 | ||
121 | For example, to match the 2nd write_aio: | |
122 | ||
123 | [set-state] | |
124 | event = "write_aio" | |
125 | state = "1" | |
126 | new_state = "2" | |
127 | ||
128 | [inject-error] | |
129 | event = "write_aio" | |
130 | state = "2" | |
131 | errno = "5" | |
132 | ||
133 | The first write_aio request matches the set-state rule and transitions from | |
134 | state 1 to state 2. Once state 2 has been entered, the set-state rule no | |
135 | longer matches since it requires state 1. But the inject-error rule now | |
136 | matches the next write_aio request and injects EIO (5). | |
137 | ||
138 | State transition rules support the following attributes: | |
139 | ||
140 | event - which type of operation to match (e.g. read_aio, write_aio, | |
141 | flush_to_os, flush_to_disk). See the "Events" section for | |
142 | information on events. | |
143 | ||
144 | state - (optional) the engine must be in this state number in order for this | |
145 | rule to match | |
146 | ||
147 | new_state - transition to this state number | |
148 | ||
149 | Suspend and resume | |
150 | ------------------ | |
151 | Exercising code paths in block drivers may require specific ordering amongst | |
152 | concurrent requests. The "breakpoint" feature allows requests to be halted on | |
153 | a blkdebug event and resumed later. This makes it possible to achieve | |
154 | deterministic ordering when multiple requests are in flight. | |
155 | ||
156 | Breakpoints on blkdebug events are associated with a user-defined "tag" string. | |
157 | This tag serves as an identifier by which the request can be resumed at a later | |
158 | point. | |
159 | ||
160 | See the qemu-io(1) break, resume, remove_break, and wait_break commands for | |
161 | details. |