]>
Commit | Line | Data |
---|---|---|
4c54005c PM |
1 | Using RCU's CPU Stall Detector |
2 | ||
a00e0d71 PM |
3 | The rcu_cpu_stall_suppress module parameter enables RCU's CPU stall |
4 | detector, which detects conditions that unduly delay RCU grace periods. | |
5 | This module parameter enables CPU stall detection by default, but | |
6 | may be overridden via boot-time parameter or at runtime via sysfs. | |
7 | The stall detector's idea of what constitutes "unduly delayed" is | |
8 | controlled by a set of kernel configuration variables and cpp macros: | |
4c54005c | 9 | |
a00e0d71 | 10 | CONFIG_RCU_CPU_STALL_TIMEOUT |
4c54005c | 11 | |
a00e0d71 PM |
12 | This kernel configuration parameter defines the period of time |
13 | that RCU will wait from the beginning of a grace period until it | |
14 | issues an RCU CPU stall warning. This time period is normally | |
64d3b7a1 | 15 | 21 seconds. |
4c54005c | 16 | |
24cd7fd0 | 17 | This configuration parameter may be changed at runtime via the |
84596ccb | 18 | /sys/module/rcupdate/parameters/rcu_cpu_stall_timeout, however |
24cd7fd0 | 19 | this parameter is checked only at the beginning of a cycle. |
64d3b7a1 | 20 | So if you are 10 seconds into a 40-second stall, setting this |
24cd7fd0 PM |
21 | sysfs parameter to (say) five will shorten the timeout for the |
22 | -next- stall, or the following warning for the current stall | |
23 | (assuming the stall lasts long enough). It will not affect the | |
24 | timing of the next warning for the current stall. | |
4c54005c | 25 | |
24cd7fd0 | 26 | Stall-warning messages may be enabled and disabled completely via |
96224daa | 27 | /sys/module/rcupdate/parameters/rcu_cpu_stall_suppress. |
24cd7fd0 | 28 | |
64d3b7a1 | 29 | CONFIG_RCU_CPU_STALL_INFO |
24cd7fd0 PM |
30 | |
31 | This kernel configuration parameter causes the stall warning to | |
32 | print out additional per-CPU diagnostic information, including | |
33 | information on scheduling-clock ticks and RCU's idle-CPU tracking. | |
34 | ||
35 | RCU_STALL_DELAY_DELTA | |
36 | ||
37 | Although the lockdep facility is extremely useful, it does add | |
38 | some overhead. Therefore, under CONFIG_PROVE_RCU, the | |
39 | RCU_STALL_DELAY_DELTA macro allows five extra seconds before | |
64d3b7a1 PM |
40 | giving an RCU CPU stall warning message. (This is a cpp |
41 | macro, not a kernel configuration parameter.) | |
4c54005c PM |
42 | |
43 | RCU_STALL_RAT_DELAY | |
44 | ||
f1d507be PM |
45 | The CPU stall detector tries to make the offending CPU print its |
46 | own warnings, as this often gives better-quality stack traces. | |
47 | However, if the offending CPU does not detect its own stall in | |
48 | the number of jiffies specified by RCU_STALL_RAT_DELAY, then | |
49 | some other CPU will complain. This delay is normally set to | |
64d3b7a1 PM |
50 | two jiffies. (This is a cpp macro, not a kernel configuration |
51 | parameter.) | |
4c54005c | 52 | |
37fe5f0e PM |
53 | rcupdate.rcu_task_stall_timeout |
54 | ||
55 | This boot/sysfs parameter controls the RCU-tasks stall warning | |
56 | interval. A value of zero or less suppresses RCU-tasks stall | |
57 | warnings. A positive value sets the stall-warning interval | |
58 | in jiffies. An RCU-tasks stall warning starts wtih the line: | |
59 | ||
60 | INFO: rcu_tasks detected stalls on tasks: | |
61 | ||
62 | And continues with the output of sched_show_task() for each | |
63 | task stalling the current RCU-tasks grace period. | |
64 | ||
65 | For non-RCU-tasks flavors of RCU, when a CPU detects that it is stalling, | |
66 | it will print a message similar to the following: | |
f1d507be PM |
67 | |
68 | INFO: rcu_sched_state detected stall on CPU 5 (t=2500 jiffies) | |
69 | ||
70 | This message indicates that CPU 5 detected that it was causing a stall, | |
71 | and that the stall was affecting RCU-sched. This message will normally be | |
72 | followed by a stack dump of the offending CPU. On TREE_RCU kernel builds, | |
73 | RCU and RCU-sched are implemented by the same underlying mechanism, | |
28f6569a | 74 | while on PREEMPT_RCU kernel builds, RCU is instead implemented |
f1d507be PM |
75 | by rcu_preempt_state. |
76 | ||
77 | On the other hand, if the offending CPU fails to print out a stall-warning | |
78 | message quickly enough, some other CPU will print a message similar to | |
79 | the following: | |
80 | ||
81 | INFO: rcu_bh_state detected stalls on CPUs/tasks: { 3 5 } (detected by 2, 2502 jiffies) | |
82 | ||
83 | This message indicates that CPU 2 detected that CPUs 3 and 5 were both | |
84 | causing stalls, and that the stall was affecting RCU-bh. This message | |
85 | will normally be followed by stack dumps for each CPU. Please note that | |
28f6569a | 86 | PREEMPT_RCU builds can be stalled by tasks as well as by CPUs, |
f1d507be PM |
87 | and that the tasks will be indicated by PID, for example, "P3421". |
88 | It is even possible for a rcu_preempt_state stall to be caused by both | |
89 | CPUs -and- tasks, in which case the offending CPUs and tasks will all | |
90 | be called out in the list. | |
91 | ||
92 | Finally, if the grace period ends just as the stall warning starts | |
93 | printing, there will be a spurious stall-warning message: | |
94 | ||
95 | INFO: rcu_bh_state detected stalls on CPUs/tasks: { } (detected by 4, 2502 jiffies) | |
96 | ||
64d3b7a1 PM |
97 | This is rare, but does happen from time to time in real life. It is also |
98 | possible for a zero-jiffy stall to be flagged in this case, depending | |
99 | on how the stall warning and the grace-period initialization happen to | |
100 | interact. Please note that it is not possible to entirely eliminate this | |
101 | sort of false positive without resorting to things like stop_machine(), | |
102 | which is overkill for this sort of problem. | |
f1d507be | 103 | |
24cd7fd0 PM |
104 | If the CONFIG_RCU_CPU_STALL_INFO kernel configuration parameter is set, |
105 | more information is printed with the stall-warning message, for example: | |
106 | ||
107 | INFO: rcu_preempt detected stall on CPU | |
6231069b | 108 | 0: (63959 ticks this GP) idle=241/3fffffffffffffff/0 softirq=82/543 |
24cd7fd0 PM |
109 | (t=65000 jiffies) |
110 | ||
111 | In kernels with CONFIG_RCU_FAST_NO_HZ, even more information is | |
112 | printed: | |
113 | ||
114 | INFO: rcu_preempt detected stall on CPU | |
6231069b | 115 | 0: (64628 ticks this GP) idle=dd5/3fffffffffffffff/0 softirq=82/543 last_accelerate: a345/d342 nonlazy_posted: 25 .D |
24cd7fd0 PM |
116 | (t=65000 jiffies) |
117 | ||
118 | The "(64628 ticks this GP)" indicates that this CPU has taken more | |
119 | than 64,000 scheduling-clock interrupts during the current stalled | |
120 | grace period. If the CPU was not yet aware of the current grace | |
121 | period (for example, if it was offline), then this part of the message | |
122 | indicates how many grace periods behind the CPU is. | |
123 | ||
124 | The "idle=" portion of the message prints the dyntick-idle state. | |
125 | The hex number before the first "/" is the low-order 12 bits of the | |
126 | dynticks counter, which will have an even-numbered value if the CPU is | |
127 | in dyntick-idle mode and an odd-numbered value otherwise. The hex | |
128 | number between the two "/"s is the value of the nesting, which will | |
129 | be a small positive number if in the idle loop and a very large positive | |
130 | number (as shown above) otherwise. | |
131 | ||
6231069b PM |
132 | The "softirq=" portion of the message tracks the number of RCU softirq |
133 | handlers that the stalled CPU has executed. The number before the "/" | |
134 | is the number that had executed since boot at the time that this CPU | |
135 | last noted the beginning of a grace period, which might be the current | |
136 | (stalled) grace period, or it might be some earlier grace period (for | |
137 | example, if the CPU might have been in dyntick-idle mode for an extended | |
138 | time period. The number after the "/" is the number that have executed | |
139 | since boot until the current time. If this latter number stays constant | |
140 | across repeated stall-warning messages, it is possible that RCU's softirq | |
141 | handlers are no longer able to execute on this CPU. This can happen if | |
142 | the stalled CPU is spinning with interrupts are disabled, or, in -rt | |
143 | kernels, if a high-priority process is starving RCU's softirq handler. | |
144 | ||
145 | For CONFIG_RCU_FAST_NO_HZ kernels, the "last_accelerate:" prints the | |
146 | low-order 16 bits (in hex) of the jiffies counter when this CPU last | |
147 | invoked rcu_try_advance_all_cbs() from rcu_needs_cpu() or last invoked | |
148 | rcu_accelerate_cbs() from rcu_prepare_for_idle(). The "nonlazy_posted:" | |
149 | prints the number of non-lazy callbacks posted since the last call to | |
150 | rcu_needs_cpu(). Finally, an "L" indicates that there are currently | |
151 | no non-lazy callbacks ("." is printed otherwise, as shown above) and | |
152 | "D" indicates that dyntick-idle processing is enabled ("." is printed | |
153 | otherwise, for example, if disabled via the "nohz=" kernel boot parameter). | |
24cd7fd0 | 154 | |
fb81a44b PM |
155 | If the relevant grace-period kthread has been unable to run prior to |
156 | the stall warning, the following additional line is printed: | |
157 | ||
158 | rcu_preempt kthread starved for 2023 jiffies! | |
159 | ||
160 | Starving the grace-period kthreads of CPU time can of course result in | |
161 | RCU CPU stall warnings even when all CPUs and tasks have passed through | |
162 | the required quiescent states. | |
163 | ||
24cd7fd0 PM |
164 | |
165 | Multiple Warnings From One Stall | |
166 | ||
167 | If a stall lasts long enough, multiple stall-warning messages will be | |
168 | printed for it. The second and subsequent messages are printed at | |
169 | longer intervals, so that the time between (say) the first and second | |
170 | message will be about three times the interval between the beginning | |
171 | of the stall and the first message. | |
172 | ||
173 | ||
174 | What Causes RCU CPU Stall Warnings? | |
175 | ||
f1d507be PM |
176 | So your kernel printed an RCU CPU stall warning. The next question is |
177 | "What caused it?" The following problems can result in RCU CPU stall | |
178 | warnings: | |
4c54005c PM |
179 | |
180 | o A CPU looping in an RCU read-side critical section. | |
181 | ||
f1d507be PM |
182 | o A CPU looping with interrupts disabled. This condition can |
183 | result in RCU-sched and RCU-bh stalls. | |
4c54005c | 184 | |
f1d507be PM |
185 | o A CPU looping with preemption disabled. This condition can |
186 | result in RCU-sched stalls and, if ksoftirqd is in use, RCU-bh | |
187 | stalls. | |
188 | ||
189 | o A CPU looping with bottom halves disabled. This condition can | |
190 | result in RCU-sched and RCU-bh stalls. | |
4c54005c | 191 | |
37fe5f0e PM |
192 | o For !CONFIG_PREEMPT kernels, a CPU looping anywhere in the |
193 | kernel without invoking schedule(). Note that cond_resched() | |
194 | does not necessarily prevent RCU CPU stall warnings. Therefore, | |
195 | if the looping in the kernel is really expected and desirable | |
196 | behavior, you might need to replace some of the cond_resched() | |
197 | calls with calls to cond_resched_rcu_qs(). | |
4c54005c | 198 | |
6ccd2ecd PM |
199 | o Anything that prevents RCU's grace-period kthreads from running. |
200 | This can result in the "All QSes seen" console-log message. | |
201 | This message will include information on when the kthread last | |
202 | ran and how often it should be expected to run. | |
203 | ||
2c96c775 PM |
204 | o A CPU-bound real-time task in a CONFIG_PREEMPT kernel, which might |
205 | happen to preempt a low-priority task in the middle of an RCU | |
206 | read-side critical section. This is especially damaging if | |
207 | that low-priority task is not permitted to run on any other CPU, | |
208 | in which case the next RCU grace period can never complete, which | |
209 | will eventually cause the system to run out of memory and hang. | |
210 | While the system is in the process of running itself out of | |
211 | memory, you might see stall-warning messages. | |
212 | ||
213 | o A CPU-bound real-time task in a CONFIG_PREEMPT_RT kernel that | |
214 | is running at a higher priority than the RCU softirq threads. | |
215 | This will prevent RCU callbacks from ever being invoked, | |
28f6569a | 216 | and in a CONFIG_PREEMPT_RCU kernel will further prevent |
2c96c775 PM |
217 | RCU grace periods from ever completing. Either way, the |
218 | system will eventually run out of memory and hang. In the | |
28f6569a | 219 | CONFIG_PREEMPT_RCU case, you might see stall-warning |
2c96c775 PM |
220 | messages. |
221 | ||
2c01531f PM |
222 | o A hardware or software issue shuts off the scheduler-clock |
223 | interrupt on a CPU that is not in dyntick-idle mode. This | |
224 | problem really has happened, and seems to be most likely to | |
3451d024 | 225 | result in RCU CPU stall warnings for CONFIG_NO_HZ_COMMON=n kernels. |
2c01531f | 226 | |
4c54005c PM |
227 | o A bug in the RCU implementation. |
228 | ||
229 | o A hardware failure. This is quite unlikely, but has occurred | |
f1d507be | 230 | at least once in real life. A CPU failed in a running system, |
4c54005c PM |
231 | becoming unresponsive, but not causing an immediate crash. |
232 | This resulted in a series of RCU CPU stall warnings, eventually | |
233 | leading the realization that the CPU had failed. | |
234 | ||
37fe5f0e PM |
235 | The RCU, RCU-sched, RCU-bh, and RCU-tasks implementations have CPU stall |
236 | warning. Note that SRCU does -not- have CPU stall warnings. Please note | |
237 | that RCU only detects CPU stalls when there is a grace period in progress. | |
238 | No grace period, no CPU stall warnings. | |
f1d507be PM |
239 | |
240 | To diagnose the cause of the stall, inspect the stack traces. | |
241 | The offending function will usually be near the top of the stack. | |
242 | If you have a series of stall warnings from a single extended stall, | |
243 | comparing the stack traces can often help determine where the stall | |
244 | is occurring, which will usually be in the function nearest the top of | |
245 | that portion of the stack which remains the same from trace to trace. | |
246 | If you can reliably trigger the stall, ftrace can be quite helpful. | |
4c54005c | 247 | |
24cd7fd0 | 248 | RCU bugs can often be debugged with the help of CONFIG_RCU_TRACE |
64d3b7a1 PM |
249 | and with RCU's event tracing. For information on RCU's event tracing, |
250 | see include/trace/events/rcu.h. |