]>
Commit | Line | Data |
---|---|---|
4c54005c PM |
1 | Using RCU's CPU Stall Detector |
2 | ||
a00e0d71 PM |
3 | The rcu_cpu_stall_suppress module parameter enables RCU's CPU stall |
4 | detector, which detects conditions that unduly delay RCU grace periods. | |
5 | This module parameter enables CPU stall detection by default, but | |
6 | may be overridden via boot-time parameter or at runtime via sysfs. | |
7 | The stall detector's idea of what constitutes "unduly delayed" is | |
8 | controlled by a set of kernel configuration variables and cpp macros: | |
4c54005c | 9 | |
a00e0d71 | 10 | CONFIG_RCU_CPU_STALL_TIMEOUT |
4c54005c | 11 | |
a00e0d71 PM |
12 | This kernel configuration parameter defines the period of time |
13 | that RCU will wait from the beginning of a grace period until it | |
14 | issues an RCU CPU stall warning. This time period is normally | |
64d3b7a1 | 15 | 21 seconds. |
4c54005c | 16 | |
24cd7fd0 PM |
17 | This configuration parameter may be changed at runtime via the |
18 | /sys/module/rcutree/parameters/rcu_cpu_stall_timeout, however | |
19 | this parameter is checked only at the beginning of a cycle. | |
64d3b7a1 | 20 | So if you are 10 seconds into a 40-second stall, setting this |
24cd7fd0 PM |
21 | sysfs parameter to (say) five will shorten the timeout for the |
22 | -next- stall, or the following warning for the current stall | |
23 | (assuming the stall lasts long enough). It will not affect the | |
24 | timing of the next warning for the current stall. | |
4c54005c | 25 | |
24cd7fd0 PM |
26 | Stall-warning messages may be enabled and disabled completely via |
27 | /sys/module/rcutree/parameters/rcu_cpu_stall_suppress. | |
28 | ||
29 | CONFIG_RCU_CPU_STALL_VERBOSE | |
30 | ||
31 | This kernel configuration parameter causes the stall warning to | |
32 | also dump the stacks of any tasks that are blocking the current | |
33 | RCU-preempt grace period. | |
34 | ||
64d3b7a1 | 35 | CONFIG_RCU_CPU_STALL_INFO |
24cd7fd0 PM |
36 | |
37 | This kernel configuration parameter causes the stall warning to | |
38 | print out additional per-CPU diagnostic information, including | |
39 | information on scheduling-clock ticks and RCU's idle-CPU tracking. | |
40 | ||
41 | RCU_STALL_DELAY_DELTA | |
42 | ||
43 | Although the lockdep facility is extremely useful, it does add | |
44 | some overhead. Therefore, under CONFIG_PROVE_RCU, the | |
45 | RCU_STALL_DELAY_DELTA macro allows five extra seconds before | |
64d3b7a1 PM |
46 | giving an RCU CPU stall warning message. (This is a cpp |
47 | macro, not a kernel configuration parameter.) | |
4c54005c PM |
48 | |
49 | RCU_STALL_RAT_DELAY | |
50 | ||
f1d507be PM |
51 | The CPU stall detector tries to make the offending CPU print its |
52 | own warnings, as this often gives better-quality stack traces. | |
53 | However, if the offending CPU does not detect its own stall in | |
54 | the number of jiffies specified by RCU_STALL_RAT_DELAY, then | |
55 | some other CPU will complain. This delay is normally set to | |
64d3b7a1 PM |
56 | two jiffies. (This is a cpp macro, not a kernel configuration |
57 | parameter.) | |
4c54005c | 58 | |
f1d507be PM |
59 | When a CPU detects that it is stalling, it will print a message similar |
60 | to the following: | |
61 | ||
62 | INFO: rcu_sched_state detected stall on CPU 5 (t=2500 jiffies) | |
63 | ||
64 | This message indicates that CPU 5 detected that it was causing a stall, | |
65 | and that the stall was affecting RCU-sched. This message will normally be | |
66 | followed by a stack dump of the offending CPU. On TREE_RCU kernel builds, | |
67 | RCU and RCU-sched are implemented by the same underlying mechanism, | |
68 | while on TREE_PREEMPT_RCU kernel builds, RCU is instead implemented | |
69 | by rcu_preempt_state. | |
70 | ||
71 | On the other hand, if the offending CPU fails to print out a stall-warning | |
72 | message quickly enough, some other CPU will print a message similar to | |
73 | the following: | |
74 | ||
75 | INFO: rcu_bh_state detected stalls on CPUs/tasks: { 3 5 } (detected by 2, 2502 jiffies) | |
76 | ||
77 | This message indicates that CPU 2 detected that CPUs 3 and 5 were both | |
78 | causing stalls, and that the stall was affecting RCU-bh. This message | |
79 | will normally be followed by stack dumps for each CPU. Please note that | |
80 | TREE_PREEMPT_RCU builds can be stalled by tasks as well as by CPUs, | |
81 | and that the tasks will be indicated by PID, for example, "P3421". | |
82 | It is even possible for a rcu_preempt_state stall to be caused by both | |
83 | CPUs -and- tasks, in which case the offending CPUs and tasks will all | |
84 | be called out in the list. | |
85 | ||
86 | Finally, if the grace period ends just as the stall warning starts | |
87 | printing, there will be a spurious stall-warning message: | |
88 | ||
89 | INFO: rcu_bh_state detected stalls on CPUs/tasks: { } (detected by 4, 2502 jiffies) | |
90 | ||
64d3b7a1 PM |
91 | This is rare, but does happen from time to time in real life. It is also |
92 | possible for a zero-jiffy stall to be flagged in this case, depending | |
93 | on how the stall warning and the grace-period initialization happen to | |
94 | interact. Please note that it is not possible to entirely eliminate this | |
95 | sort of false positive without resorting to things like stop_machine(), | |
96 | which is overkill for this sort of problem. | |
f1d507be | 97 | |
24cd7fd0 PM |
98 | If the CONFIG_RCU_CPU_STALL_INFO kernel configuration parameter is set, |
99 | more information is printed with the stall-warning message, for example: | |
100 | ||
101 | INFO: rcu_preempt detected stall on CPU | |
6231069b | 102 | 0: (63959 ticks this GP) idle=241/3fffffffffffffff/0 softirq=82/543 |
24cd7fd0 PM |
103 | (t=65000 jiffies) |
104 | ||
105 | In kernels with CONFIG_RCU_FAST_NO_HZ, even more information is | |
106 | printed: | |
107 | ||
108 | INFO: rcu_preempt detected stall on CPU | |
6231069b | 109 | 0: (64628 ticks this GP) idle=dd5/3fffffffffffffff/0 softirq=82/543 last_accelerate: a345/d342 nonlazy_posted: 25 .D |
24cd7fd0 PM |
110 | (t=65000 jiffies) |
111 | ||
112 | The "(64628 ticks this GP)" indicates that this CPU has taken more | |
113 | than 64,000 scheduling-clock interrupts during the current stalled | |
114 | grace period. If the CPU was not yet aware of the current grace | |
115 | period (for example, if it was offline), then this part of the message | |
116 | indicates how many grace periods behind the CPU is. | |
117 | ||
118 | The "idle=" portion of the message prints the dyntick-idle state. | |
119 | The hex number before the first "/" is the low-order 12 bits of the | |
120 | dynticks counter, which will have an even-numbered value if the CPU is | |
121 | in dyntick-idle mode and an odd-numbered value otherwise. The hex | |
122 | number between the two "/"s is the value of the nesting, which will | |
123 | be a small positive number if in the idle loop and a very large positive | |
124 | number (as shown above) otherwise. | |
125 | ||
6231069b PM |
126 | The "softirq=" portion of the message tracks the number of RCU softirq |
127 | handlers that the stalled CPU has executed. The number before the "/" | |
128 | is the number that had executed since boot at the time that this CPU | |
129 | last noted the beginning of a grace period, which might be the current | |
130 | (stalled) grace period, or it might be some earlier grace period (for | |
131 | example, if the CPU might have been in dyntick-idle mode for an extended | |
132 | time period. The number after the "/" is the number that have executed | |
133 | since boot until the current time. If this latter number stays constant | |
134 | across repeated stall-warning messages, it is possible that RCU's softirq | |
135 | handlers are no longer able to execute on this CPU. This can happen if | |
136 | the stalled CPU is spinning with interrupts are disabled, or, in -rt | |
137 | kernels, if a high-priority process is starving RCU's softirq handler. | |
138 | ||
139 | For CONFIG_RCU_FAST_NO_HZ kernels, the "last_accelerate:" prints the | |
140 | low-order 16 bits (in hex) of the jiffies counter when this CPU last | |
141 | invoked rcu_try_advance_all_cbs() from rcu_needs_cpu() or last invoked | |
142 | rcu_accelerate_cbs() from rcu_prepare_for_idle(). The "nonlazy_posted:" | |
143 | prints the number of non-lazy callbacks posted since the last call to | |
144 | rcu_needs_cpu(). Finally, an "L" indicates that there are currently | |
145 | no non-lazy callbacks ("." is printed otherwise, as shown above) and | |
146 | "D" indicates that dyntick-idle processing is enabled ("." is printed | |
147 | otherwise, for example, if disabled via the "nohz=" kernel boot parameter). | |
24cd7fd0 PM |
148 | |
149 | ||
150 | Multiple Warnings From One Stall | |
151 | ||
152 | If a stall lasts long enough, multiple stall-warning messages will be | |
153 | printed for it. The second and subsequent messages are printed at | |
154 | longer intervals, so that the time between (say) the first and second | |
155 | message will be about three times the interval between the beginning | |
156 | of the stall and the first message. | |
157 | ||
158 | ||
159 | What Causes RCU CPU Stall Warnings? | |
160 | ||
f1d507be PM |
161 | So your kernel printed an RCU CPU stall warning. The next question is |
162 | "What caused it?" The following problems can result in RCU CPU stall | |
163 | warnings: | |
4c54005c PM |
164 | |
165 | o A CPU looping in an RCU read-side critical section. | |
166 | ||
f1d507be PM |
167 | o A CPU looping with interrupts disabled. This condition can |
168 | result in RCU-sched and RCU-bh stalls. | |
4c54005c | 169 | |
f1d507be PM |
170 | o A CPU looping with preemption disabled. This condition can |
171 | result in RCU-sched stalls and, if ksoftirqd is in use, RCU-bh | |
172 | stalls. | |
173 | ||
174 | o A CPU looping with bottom halves disabled. This condition can | |
175 | result in RCU-sched and RCU-bh stalls. | |
4c54005c PM |
176 | |
177 | o For !CONFIG_PREEMPT kernels, a CPU looping anywhere in the kernel | |
178 | without invoking schedule(). | |
179 | ||
2c96c775 PM |
180 | o A CPU-bound real-time task in a CONFIG_PREEMPT kernel, which might |
181 | happen to preempt a low-priority task in the middle of an RCU | |
182 | read-side critical section. This is especially damaging if | |
183 | that low-priority task is not permitted to run on any other CPU, | |
184 | in which case the next RCU grace period can never complete, which | |
185 | will eventually cause the system to run out of memory and hang. | |
186 | While the system is in the process of running itself out of | |
187 | memory, you might see stall-warning messages. | |
188 | ||
189 | o A CPU-bound real-time task in a CONFIG_PREEMPT_RT kernel that | |
190 | is running at a higher priority than the RCU softirq threads. | |
191 | This will prevent RCU callbacks from ever being invoked, | |
192 | and in a CONFIG_TREE_PREEMPT_RCU kernel will further prevent | |
193 | RCU grace periods from ever completing. Either way, the | |
194 | system will eventually run out of memory and hang. In the | |
195 | CONFIG_TREE_PREEMPT_RCU case, you might see stall-warning | |
196 | messages. | |
197 | ||
2c01531f PM |
198 | o A hardware or software issue shuts off the scheduler-clock |
199 | interrupt on a CPU that is not in dyntick-idle mode. This | |
200 | problem really has happened, and seems to be most likely to | |
3451d024 | 201 | result in RCU CPU stall warnings for CONFIG_NO_HZ_COMMON=n kernels. |
2c01531f | 202 | |
4c54005c PM |
203 | o A bug in the RCU implementation. |
204 | ||
205 | o A hardware failure. This is quite unlikely, but has occurred | |
f1d507be | 206 | at least once in real life. A CPU failed in a running system, |
4c54005c PM |
207 | becoming unresponsive, but not causing an immediate crash. |
208 | This resulted in a series of RCU CPU stall warnings, eventually | |
209 | leading the realization that the CPU had failed. | |
210 | ||
9ceae0e2 PM |
211 | The RCU, RCU-sched, and RCU-bh implementations have CPU stall warning. |
212 | SRCU does not have its own CPU stall warnings, but its calls to | |
213 | synchronize_sched() will result in RCU-sched detecting RCU-sched-related | |
214 | CPU stalls. Please note that RCU only detects CPU stalls when there is | |
215 | a grace period in progress. No grace period, no CPU stall warnings. | |
f1d507be PM |
216 | |
217 | To diagnose the cause of the stall, inspect the stack traces. | |
218 | The offending function will usually be near the top of the stack. | |
219 | If you have a series of stall warnings from a single extended stall, | |
220 | comparing the stack traces can often help determine where the stall | |
221 | is occurring, which will usually be in the function nearest the top of | |
222 | that portion of the stack which remains the same from trace to trace. | |
223 | If you can reliably trigger the stall, ftrace can be quite helpful. | |
4c54005c | 224 | |
24cd7fd0 | 225 | RCU bugs can often be debugged with the help of CONFIG_RCU_TRACE |
64d3b7a1 PM |
226 | and with RCU's event tracing. For information on RCU's event tracing, |
227 | see include/trace/events/rcu.h. |