]>
Commit | Line | Data |
---|---|---|
49717cb4 PM |
1 | REDUCING OS JITTER DUE TO PER-CPU KTHREADS |
2 | ||
3 | This document lists per-CPU kthreads in the Linux kernel and presents | |
4 | options to control their OS jitter. Note that non-per-CPU kthreads are | |
5 | not listed here. To reduce OS jitter from non-per-CPU kthreads, bind | |
6 | them to a "housekeeping" CPU dedicated to such work. | |
7 | ||
8 | ||
9 | REFERENCES | |
10 | ||
11 | o Documentation/IRQ-affinity.txt: Binding interrupts to sets of CPUs. | |
12 | ||
09c3bcce | 13 | o Documentation/cgroup-v1: Using cgroups to bind tasks to sets of CPUs. |
49717cb4 PM |
14 | |
15 | o man taskset: Using the taskset command to bind tasks to sets | |
16 | of CPUs. | |
17 | ||
18 | o man sched_setaffinity: Using the sched_setaffinity() system | |
19 | call to bind tasks to sets of CPUs. | |
20 | ||
21 | o /sys/devices/system/cpu/cpuN/online: Control CPU N's hotplug state, | |
22 | writing "0" to offline and "1" to online. | |
23 | ||
24 | o In order to locate kernel-generated OS jitter on CPU N: | |
25 | ||
26 | cd /sys/kernel/debug/tracing | |
27 | echo 1 > max_graph_depth # Increase the "1" for more detail | |
28 | echo function_graph > current_tracer | |
29 | # run workload | |
30 | cat per_cpu/cpuN/trace | |
31 | ||
32 | ||
33 | KTHREADS | |
34 | ||
35 | Name: ehca_comp/%u | |
36 | Purpose: Periodically process Infiniband-related work. | |
37 | To reduce its OS jitter, do any of the following: | |
38 | 1. Don't use eHCA Infiniband hardware, instead choosing hardware | |
39 | that does not require per-CPU kthreads. This will prevent these | |
40 | kthreads from being created in the first place. (This will | |
41 | work for most people, as this hardware, though important, is | |
42 | relatively old and is produced in relatively low unit volumes.) | |
43 | 2. Do all eHCA-Infiniband-related work on other CPUs, including | |
44 | interrupts. | |
45 | 3. Rework the eHCA driver so that its per-CPU kthreads are | |
46 | provisioned only on selected CPUs. | |
47 | ||
48 | ||
49 | Name: irq/%d-%s | |
50 | Purpose: Handle threaded interrupts. | |
51 | To reduce its OS jitter, do the following: | |
52 | 1. Use irq affinity to force the irq threads to execute on | |
53 | some other CPU. | |
54 | ||
55 | Name: kcmtpd_ctr_%d | |
56 | Purpose: Handle Bluetooth work. | |
57 | To reduce its OS jitter, do one of the following: | |
58 | 1. Don't use Bluetooth, in which case these kthreads won't be | |
59 | created in the first place. | |
60 | 2. Use irq affinity to force Bluetooth-related interrupts to | |
61 | occur on some other CPU and furthermore initiate all | |
62 | Bluetooth activity on some other CPU. | |
63 | ||
64 | Name: ksoftirqd/%u | |
65 | Purpose: Execute softirq handlers when threaded or when under heavy load. | |
66 | To reduce its OS jitter, each softirq vector must be handled | |
67 | separately as follows: | |
68 | TIMER_SOFTIRQ: Do all of the following: | |
69 | 1. To the extent possible, keep the CPU out of the kernel when it | |
70 | is non-idle, for example, by avoiding system calls and by forcing | |
71 | both kernel threads and interrupts to execute elsewhere. | |
72 | 2. Build with CONFIG_HOTPLUG_CPU=y. After boot completes, force | |
73 | the CPU offline, then bring it back online. This forces | |
74 | recurring timers to migrate elsewhere. If you are concerned | |
75 | with multiple CPUs, force them all offline before bringing the | |
76 | first one back online. Once you have onlined the CPUs in question, | |
77 | do not offline any other CPUs, because doing so could force the | |
78 | timer back onto one of the CPUs in question. | |
79 | NET_TX_SOFTIRQ and NET_RX_SOFTIRQ: Do all of the following: | |
80 | 1. Force networking interrupts onto other CPUs. | |
81 | 2. Initiate any network I/O on other CPUs. | |
82 | 3. Once your application has started, prevent CPU-hotplug operations | |
83 | from being initiated from tasks that might run on the CPU to | |
84 | be de-jittered. (It is OK to force this CPU offline and then | |
85 | bring it back online before you start your application.) | |
86 | BLOCK_SOFTIRQ: Do all of the following: | |
87 | 1. Force block-device interrupts onto some other CPU. | |
88 | 2. Initiate any block I/O on other CPUs. | |
89 | 3. Once your application has started, prevent CPU-hotplug operations | |
90 | from being initiated from tasks that might run on the CPU to | |
91 | be de-jittered. (It is OK to force this CPU offline and then | |
92 | bring it back online before you start your application.) | |
511cbce2 | 93 | IRQ_POLL_SOFTIRQ: Do all of the following: |
49717cb4 PM |
94 | 1. Force block-device interrupts onto some other CPU. |
95 | 2. Initiate any block I/O and block-I/O polling on other CPUs. | |
96 | 3. Once your application has started, prevent CPU-hotplug operations | |
97 | from being initiated from tasks that might run on the CPU to | |
98 | be de-jittered. (It is OK to force this CPU offline and then | |
99 | bring it back online before you start your application.) | |
100 | TASKLET_SOFTIRQ: Do one or more of the following: | |
101 | 1. Avoid use of drivers that use tasklets. (Such drivers will contain | |
102 | calls to things like tasklet_schedule().) | |
103 | 2. Convert all drivers that you must use from tasklets to workqueues. | |
104 | 3. Force interrupts for drivers using tasklets onto other CPUs, | |
105 | and also do I/O involving these drivers on other CPUs. | |
106 | SCHED_SOFTIRQ: Do all of the following: | |
107 | 1. Avoid sending scheduler IPIs to the CPU to be de-jittered, | |
108 | for example, ensure that at most one runnable kthread is present | |
109 | on that CPU. If a thread that expects to run on the de-jittered | |
110 | CPU awakens, the scheduler will send an IPI that can result in | |
111 | a subsequent SCHED_SOFTIRQ. | |
44c65ff2 PM |
112 | 2. CONFIG_NO_HZ_FULL=y and ensure that the CPU to be de-jittered |
113 | is marked as an adaptive-ticks CPU using the "nohz_full=" | |
114 | boot parameter. This reduces the number of scheduler-clock | |
115 | interrupts that the de-jittered CPU receives, minimizing its | |
116 | chances of being selected to do the load balancing work that | |
117 | runs in SCHED_SOFTIRQ context. | |
49717cb4 PM |
118 | 3. To the extent possible, keep the CPU out of the kernel when it |
119 | is non-idle, for example, by avoiding system calls and by | |
120 | forcing both kernel threads and interrupts to execute elsewhere. | |
121 | This further reduces the number of scheduler-clock interrupts | |
122 | received by the de-jittered CPU. | |
123 | HRTIMER_SOFTIRQ: Do all of the following: | |
124 | 1. To the extent possible, keep the CPU out of the kernel when it | |
125 | is non-idle. For example, avoid system calls and force both | |
126 | kernel threads and interrupts to execute elsewhere. | |
127 | 2. Build with CONFIG_HOTPLUG_CPU=y. Once boot completes, force the | |
128 | CPU offline, then bring it back online. This forces recurring | |
129 | timers to migrate elsewhere. If you are concerned with multiple | |
130 | CPUs, force them all offline before bringing the first one | |
131 | back online. Once you have onlined the CPUs in question, do not | |
132 | offline any other CPUs, because doing so could force the timer | |
133 | back onto one of the CPUs in question. | |
134 | RCU_SOFTIRQ: Do at least one of the following: | |
135 | 1. Offload callbacks and keep the CPU in either dyntick-idle or | |
136 | adaptive-ticks state by doing all of the following: | |
44c65ff2 PM |
137 | a. CONFIG_NO_HZ_FULL=y and ensure that the CPU to be |
138 | de-jittered is marked as an adaptive-ticks CPU using the | |
139 | "nohz_full=" boot parameter. Bind the rcuo kthreads to | |
140 | housekeeping CPUs, which can tolerate OS jitter. | |
49717cb4 PM |
141 | b. To the extent possible, keep the CPU out of the kernel |
142 | when it is non-idle, for example, by avoiding system | |
143 | calls and by forcing both kernel threads and interrupts | |
144 | to execute elsewhere. | |
145 | 2. Enable RCU to do its processing remotely via dyntick-idle by | |
146 | doing all of the following: | |
147 | a. Build with CONFIG_NO_HZ=y and CONFIG_RCU_FAST_NO_HZ=y. | |
148 | b. Ensure that the CPU goes idle frequently, allowing other | |
149 | CPUs to detect that it has passed through an RCU quiescent | |
150 | state. If the kernel is built with CONFIG_NO_HZ_FULL=y, | |
151 | userspace execution also allows other CPUs to detect that | |
152 | the CPU in question has passed through a quiescent state. | |
153 | c. To the extent possible, keep the CPU out of the kernel | |
154 | when it is non-idle, for example, by avoiding system | |
155 | calls and by forcing both kernel threads and interrupts | |
156 | to execute elsewhere. | |
157 | ||
f7bac9b8 PM |
158 | Name: kworker/%u:%d%s (cpu, id, priority) |
159 | Purpose: Execute workqueue requests | |
160 | To reduce its OS jitter, do any of the following: | |
161 | 1. Run your workload at a real-time priority, which will allow | |
162 | preempting the kworker daemons. | |
bbf393b0 PM |
163 | 2. A given workqueue can be made visible in the sysfs filesystem |
164 | by passing the WQ_SYSFS to that workqueue's alloc_workqueue(). | |
165 | Such a workqueue can be confined to a given subset of the | |
166 | CPUs using the /sys/devices/virtual/workqueue/*/cpumask sysfs | |
167 | files. The set of WQ_SYSFS workqueues can be displayed using | |
168 | "ls sys/devices/virtual/workqueue". That said, the workqueues | |
169 | maintainer would like to caution people against indiscriminately | |
170 | sprinkling WQ_SYSFS across all the workqueues. The reason for | |
171 | caution is that it is easy to add WQ_SYSFS, but because sysfs is | |
172 | part of the formal user/kernel API, it can be nearly impossible | |
173 | to remove it, even if its addition was a mistake. | |
174 | 3. Do any of the following needed to avoid jitter that your | |
f7bac9b8 PM |
175 | application cannot tolerate: |
176 | a. Build your kernel with CONFIG_SLUB=y rather than | |
177 | CONFIG_SLAB=y, thus avoiding the slab allocator's periodic | |
178 | use of each CPU's workqueues to run its cache_reap() | |
179 | function. | |
180 | b. Avoid using oprofile, thus avoiding OS jitter from | |
181 | wq_sync_buffer(). | |
182 | c. Limit your CPU frequency so that a CPU-frequency | |
183 | governor is not required, possibly enlisting the aid of | |
184 | special heatsinks or other cooling technologies. If done | |
185 | correctly, and if you CPU architecture permits, you should | |
186 | be able to build your kernel with CONFIG_CPU_FREQ=n to | |
187 | avoid the CPU-frequency governor periodically running | |
188 | on each CPU, including cs_dbs_timer() and od_dbs_timer(). | |
189 | WARNING: Please check your CPU specifications to | |
190 | make sure that this is safe on your particular system. | |
89bf5d82 PM |
191 | d. As of v3.18, Christoph Lameter's on-demand vmstat workers |
192 | commit prevents OS jitter due to vmstat_update() on | |
193 | CONFIG_SMP=y systems. Before v3.18, is not possible | |
194 | to entirely get rid of the OS jitter, but you can | |
195 | decrease its frequency by writing a large value to | |
196 | /proc/sys/vm/stat_interval. The default value is HZ, | |
197 | for an interval of one second. Of course, larger values | |
198 | will make your virtual-memory statistics update more | |
199 | slowly. Of course, you can also run your workload at | |
200 | a real-time priority, thus preempting vmstat_update(), | |
64f26e5c PM |
201 | but if your workload is CPU-bound, this is a bad idea. |
202 | However, there is an RFC patch from Christoph Lameter | |
203 | (based on an earlier one from Gilad Ben-Yossef) that | |
204 | reduces or even eliminates vmstat overhead for some | |
205 | workloads at https://lkml.org/lkml/2013/9/4/379. | |
f1360570 PM |
206 | e. Boot with "elevator=noop" to avoid workqueue use by |
207 | the block layer. | |
208 | f. If running on high-end powerpc servers, build with | |
f7bac9b8 PM |
209 | CONFIG_PPC_RTAS_DAEMON=n. This prevents the RTAS |
210 | daemon from running on each CPU every second or so. | |
211 | (This will require editing Kconfig files and will defeat | |
212 | this platform's RAS functionality.) This avoids jitter | |
213 | due to the rtas_event_scan() function. | |
214 | WARNING: Please check your CPU specifications to | |
215 | make sure that this is safe on your particular system. | |
f1360570 | 216 | g. If running on Cell Processor, build your kernel with |
f7bac9b8 PM |
217 | CBE_CPUFREQ_SPU_GOVERNOR=n to avoid OS jitter from |
218 | spu_gov_work(). | |
219 | WARNING: Please check your CPU specifications to | |
220 | make sure that this is safe on your particular system. | |
f1360570 | 221 | h. If running on PowerMAC, build your kernel with |
f7bac9b8 PM |
222 | CONFIG_PMAC_RACKMETER=n to disable the CPU-meter, |
223 | avoiding OS jitter from rackmeter_do_timer(). | |
224 | ||
49717cb4 PM |
225 | Name: rcuc/%u |
226 | Purpose: Execute RCU callbacks in CONFIG_RCU_BOOST=y kernels. | |
227 | To reduce its OS jitter, do at least one of the following: | |
228 | 1. Build the kernel with CONFIG_PREEMPT=n. This prevents these | |
229 | kthreads from being created in the first place, and also obviates | |
230 | the need for RCU priority boosting. This approach is feasible | |
231 | for workloads that do not require high degrees of responsiveness. | |
232 | 2. Build the kernel with CONFIG_RCU_BOOST=n. This prevents these | |
233 | kthreads from being created in the first place. This approach | |
234 | is feasible only if your workload never requires RCU priority | |
235 | boosting, for example, if you ensure frequent idle time on all | |
236 | CPUs that might execute within the kernel. | |
44c65ff2 PM |
237 | 3. Build with CONFIG_RCU_NOCB_CPU=y and boot with the rcu_nocbs= |
238 | boot parameter offloading RCU callbacks from all CPUs susceptible | |
239 | to OS jitter. This approach prevents the rcuc/%u kthreads from | |
240 | having any work to do, so that they are never awakened. | |
49717cb4 PM |
241 | 4. Ensure that the CPU never enters the kernel, and, in particular, |
242 | avoid initiating any CPU hotplug operations on this CPU. This is | |
243 | another way of preventing any callbacks from being queued on the | |
244 | CPU, again preventing the rcuc/%u kthreads from having any work | |
245 | to do. | |
246 | ||
247 | Name: rcuob/%d, rcuop/%d, and rcuos/%d | |
248 | Purpose: Offload RCU callbacks from the corresponding CPU. | |
249 | To reduce its OS jitter, do at least one of the following: | |
250 | 1. Use affinity, cgroups, or other mechanism to force these kthreads | |
251 | to execute on some other CPU. | |
b9651622 | 252 | 2. Build with CONFIG_RCU_NOCB_CPU=n, which will prevent these |
49717cb4 PM |
253 | kthreads from being created in the first place. However, please |
254 | note that this will not eliminate OS jitter, but will instead | |
255 | shift it to RCU_SOFTIRQ. | |
256 | ||
257 | Name: watchdog/%u | |
258 | Purpose: Detect software lockups on each CPU. | |
259 | To reduce its OS jitter, do at least one of the following: | |
260 | 1. Build with CONFIG_LOCKUP_DETECTOR=n, which will prevent these | |
261 | kthreads from being created in the first place. | |
f1360570 PM |
262 | 2. Boot with "nosoftlockup=0", which will also prevent these kthreads |
263 | from being created. Other related watchdog and softlockup boot | |
8c27ceff | 264 | parameters may be found in Documentation/admin-guide/kernel-parameters.rst |
f1360570 PM |
265 | and Documentation/watchdog/watchdog-parameters.txt. |
266 | 3. Echo a zero to /proc/sys/kernel/watchdog to disable the | |
49717cb4 | 267 | watchdog timer. |
f1360570 | 268 | 4. Echo a large number of /proc/sys/kernel/watchdog_thresh in |
49717cb4 PM |
269 | order to reduce the frequency of OS jitter due to the watchdog |
270 | timer down to a level that is acceptable for your workload. |