]>
Commit | Line | Data |
---|---|---|
a241ec65 PM |
1 | RCU Torture Test Operation |
2 | ||
3 | ||
4 | CONFIG_RCU_TORTURE_TEST | |
5 | ||
6 | The CONFIG_RCU_TORTURE_TEST config option is available for all RCU | |
7 | implementations. It creates an rcutorture kernel module that can | |
8 | be loaded to run a torture test. The test periodically outputs | |
9 | status messages via printk(), which can be examined via the dmesg | |
72e9bb54 | 10 | command (perhaps grepping for "torture"). The test is started |
a241ec65 PM |
11 | when the module is loaded, and stops when the module is unloaded. |
12 | ||
31a72bce PM |
13 | CONFIG_RCU_TORTURE_TEST_RUNNABLE |
14 | ||
15 | It is also possible to specify CONFIG_RCU_TORTURE_TEST=y, which will | |
16 | result in the tests being loaded into the base kernel. In this case, | |
17 | the CONFIG_RCU_TORTURE_TEST_RUNNABLE config option is used to specify | |
18 | whether the RCU torture tests are to be started immediately during | |
19 | boot or whether the /proc/sys/kernel/rcutorture_runnable file is used | |
20 | to enable them. This /proc file can be used to repeatedly pause and | |
21 | restart the tests, regardless of the initial state specified by the | |
22 | CONFIG_RCU_TORTURE_TEST_RUNNABLE config option. | |
23 | ||
24 | You will normally -not- want to start the RCU torture tests during boot | |
25 | (and thus the default is CONFIG_RCU_TORTURE_TEST_RUNNABLE=n), but doing | |
26 | this can sometimes be useful in finding boot-time bugs. | |
a241ec65 PM |
27 | |
28 | ||
29 | MODULE PARAMETERS | |
30 | ||
31 | This module has the following parameters: | |
32 | ||
4c54005c PM |
33 | fqs_duration Duration (in microseconds) of artificially induced bursts |
34 | of force_quiescent_state() invocations. In RCU | |
35 | implementations having force_quiescent_state(), these | |
36 | bursts help force races between forcing a given grace | |
37 | period and that grace period ending on its own. | |
38 | ||
39 | fqs_holdoff Holdoff time (in microseconds) between consecutive calls | |
40 | to force_quiescent_state() within a burst. | |
41 | ||
42 | fqs_stutter Wait time (in seconds) between consecutive bursts | |
43 | of calls to force_quiescent_state(). | |
44 | ||
0729fbf3 PM |
45 | irqreaders Says to invoke RCU readers from irq level. This is currently |
46 | done via timers. Defaults to "1" for variants of RCU that | |
47 | permit this. (Or, more accurately, variants of RCU that do | |
48 | -not- permit this know to ignore this variable.) | |
a241ec65 | 49 | |
b772e1dd JT |
50 | nfakewriters This is the number of RCU fake writer threads to run. Fake |
51 | writer threads repeatedly use the synchronous "wait for | |
52 | current readers" function of the interface selected by | |
53 | torture_type, with a delay between calls to allow for various | |
54 | different numbers of writers running in parallel. | |
55 | nfakewriters defaults to 4, which provides enough parallelism | |
56 | to trigger special cases caused by multiple writers, such as | |
57 | the synchronize_srcu() early return optimization. | |
58 | ||
0729fbf3 PM |
59 | nreaders This is the number of RCU reading threads supported. |
60 | The default is twice the number of CPUs. Why twice? | |
61 | To properly exercise RCU implementations with preemptible | |
62 | read-side critical sections. | |
63 | ||
64 | shuffle_interval | |
65 | The number of seconds to keep the test threads affinitied | |
66 | to a particular subset of the CPUs, defaults to 3 seconds. | |
67 | Used in conjunction with test_no_idle_hz. | |
68 | ||
a241ec65 PM |
69 | stat_interval The number of seconds between output of torture |
70 | statistics (via printk()). Regardless of the interval, | |
71 | statistics are printed when the module is unloaded. | |
72 | Setting the interval to zero causes the statistics to | |
73 | be printed -only- when the module is unloaded, and this | |
74 | is the default. | |
75 | ||
d120f65f PM |
76 | stutter The length of time to run the test before pausing for this |
77 | same period of time. Defaults to "stutter=5", so as | |
78 | to run and pause for (roughly) five-second intervals. | |
79 | Specifying "stutter=0" causes the test to run continuously | |
80 | without pausing, which is the old default behavior. | |
81 | ||
29766f1e PM |
82 | test_no_idle_hz Whether or not to test the ability of RCU to operate in |
83 | a kernel that disables the scheduling-clock interrupt to | |
84 | idle CPUs. Boolean parameter, "1" to test, "0" otherwise. | |
f85d6c71 | 85 | Defaults to omitting this test. |
29766f1e | 86 | |
20d2e428 JT |
87 | torture_type The type of RCU to test: "rcu" for the rcu_read_lock() API, |
88 | "rcu_sync" for rcu_read_lock() with synchronous reclamation, | |
11a14701 | 89 | "rcu_bh" for the rcu_read_lock_bh() API, "rcu_bh_sync" for |
4b6c2cca | 90 | rcu_read_lock_bh() with synchronous reclamation, "srcu" for |
240ebbf8 PM |
91 | the "srcu_read_lock()" API, "sched" for the use of |
92 | preempt_disable() together with synchronize_sched(), | |
93 | and "sched_expedited" for the use of preempt_disable() | |
94 | with synchronize_sched_expedited(). | |
72e9bb54 | 95 | |
a241ec65 PM |
96 | verbose Enable debug printk()s. Default is disabled. |
97 | ||
98 | ||
99 | OUTPUT | |
100 | ||
101 | The statistics output is as follows: | |
102 | ||
72e9bb54 PM |
103 | rcu-torture: --- Start of test: nreaders=16 stat_interval=0 verbose=0 |
104 | rcu-torture: rtc: 0000000000000000 ver: 1916 tfle: 0 rta: 1916 rtaf: 0 rtf: 1915 | |
105 | rcu-torture: Reader Pipe: 1466408 9747 0 0 0 0 0 0 0 0 0 | |
106 | rcu-torture: Reader Batch: 1464477 11678 0 0 0 0 0 0 0 0 | |
107 | rcu-torture: Free-Block Circulation: 1915 1915 1915 1915 1915 1915 1915 1915 1915 1915 0 | |
108 | rcu-torture: --- End of test | |
a241ec65 | 109 | |
72e9bb54 | 110 | The command "dmesg | grep torture:" will extract this information on |
a241ec65 PM |
111 | most systems. On more esoteric configurations, it may be necessary to |
112 | use other commands to access the output of the printk()s used by | |
113 | the RCU torture test. The printk()s use KERN_ALERT, so they should | |
114 | be evident. ;-) | |
115 | ||
116 | The entries are as follows: | |
117 | ||
a241ec65 PM |
118 | o "rtc": The hexadecimal address of the structure currently visible |
119 | to readers. | |
120 | ||
121 | o "ver": The number of times since boot that the rcutw writer task | |
122 | has changed the structure visible to readers. | |
123 | ||
124 | o "tfle": If non-zero, indicates that the "torture freelist" | |
125 | containing structure to be placed into the "rtc" area is empty. | |
126 | This condition is important, since it can fool you into thinking | |
127 | that RCU is working when it is not. :-/ | |
128 | ||
129 | o "rta": Number of structures allocated from the torture freelist. | |
130 | ||
131 | o "rtaf": Number of allocations from the torture freelist that have | |
132 | failed due to the list being empty. | |
133 | ||
134 | o "rtf": Number of frees into the torture freelist. | |
135 | ||
136 | o "Reader Pipe": Histogram of "ages" of structures seen by readers. | |
137 | If any entries past the first two are non-zero, RCU is broken. | |
138 | And rcutorture prints the error flag string "!!!" to make sure | |
139 | you notice. The age of a newly allocated structure is zero, | |
140 | it becomes one when removed from reader visibility, and is | |
141 | incremented once per grace period subsequently -- and is freed | |
142 | after passing through (RCU_TORTURE_PIPE_LEN-2) grace periods. | |
143 | ||
144 | The output displayed above was taken from a correctly working | |
145 | RCU. If you want to see what it looks like when broken, break | |
146 | it yourself. ;-) | |
147 | ||
148 | o "Reader Batch": Another histogram of "ages" of structures seen | |
149 | by readers, but in terms of counter flips (or batches) rather | |
150 | than in terms of grace periods. The legal number of non-zero | |
f85d6c71 PM |
151 | entries is again two. The reason for this separate view is that |
152 | it is sometimes easier to get the third entry to show up in the | |
a241ec65 PM |
153 | "Reader Batch" list than in the "Reader Pipe" list. |
154 | ||
155 | o "Free-Block Circulation": Shows the number of torture structures | |
156 | that have reached a given point in the pipeline. The first element | |
157 | should closely correspond to the number of structures allocated, | |
158 | the second to the number that have been removed from reader view, | |
159 | and all but the last remaining to the corresponding number of | |
160 | passes through a grace period. The last entry should be zero, | |
161 | as it is only incremented if a torture structure's counter | |
162 | somehow gets incremented farther than it should. | |
163 | ||
b2896d2e PM |
164 | Different implementations of RCU can provide implementation-specific |
165 | additional information. For example, SRCU provides the following: | |
166 | ||
167 | srcu-torture: rtc: f8cf46a8 ver: 355 tfle: 0 rta: 356 rtaf: 0 rtf: 346 rtmbe: 0 | |
168 | srcu-torture: Reader Pipe: 559738 939 0 0 0 0 0 0 0 0 0 | |
169 | srcu-torture: Reader Batch: 560434 243 0 0 0 0 0 0 0 0 | |
170 | srcu-torture: Free-Block Circulation: 355 354 353 352 351 350 349 348 347 346 0 | |
171 | srcu-torture: per-CPU(idx=1): 0(0,1) 1(0,1) 2(0,0) 3(0,1) | |
172 | ||
173 | The first four lines are similar to those for RCU. The last line shows | |
174 | the per-CPU counter state. The numbers in parentheses are the values | |
175 | of the "old" and "current" counters for the corresponding CPU. The | |
176 | "idx" value maps the "old" and "current" values to the underlying array, | |
177 | and is useful for debugging. | |
178 | ||
240ebbf8 PM |
179 | Similarly, sched_expedited RCU provides the following: |
180 | ||
181 | sched_expedited-torture: rtc: d0000000016c1880 ver: 1090796 tfle: 0 rta: 1090796 rtaf: 0 rtf: 1090787 rtmbe: 0 nt: 27713319 | |
182 | sched_expedited-torture: Reader Pipe: 12660320201 95875 0 0 0 0 0 0 0 0 0 | |
183 | sched_expedited-torture: Reader Batch: 12660424885 0 0 0 0 0 0 0 0 0 0 | |
184 | sched_expedited-torture: Free-Block Circulation: 1090795 1090795 1090794 1090793 1090792 1090791 1090790 1090789 1090788 1090787 0 | |
240ebbf8 | 185 | |
a241ec65 PM |
186 | |
187 | USAGE | |
188 | ||
189 | The following script may be used to torture RCU: | |
190 | ||
191 | #!/bin/sh | |
192 | ||
193 | modprobe rcutorture | |
194 | sleep 100 | |
195 | rmmod rcutorture | |
72e9bb54 | 196 | dmesg | grep torture: |
a241ec65 PM |
197 | |
198 | The output can be manually inspected for the error flag of "!!!". | |
199 | One could of course create a more elaborate script that automatically | |
29766f1e PM |
200 | checked for such errors. The "rmmod" command forces a "SUCCESS" or |
201 | "FAILURE" indication to be printk()ed. |