]>
Commit | Line | Data |
---|---|---|
72f924f6 VG |
1 | Block IO Controller |
2 | =================== | |
3 | Overview | |
4 | ======== | |
5 | cgroup subsys "blkio" implements the block io controller. There seems to be | |
6 | a need of various kinds of IO control policies (like proportional BW, max BW) | |
7 | both at leaf nodes as well as at intermediate nodes in a storage hierarchy. | |
8 | Plan is to use the same cgroup based management interface for blkio controller | |
9 | and based on user options switch IO policies in the background. | |
10 | ||
2786c4e5 VG |
11 | Currently two IO control policies are implemented. First one is proportional |
12 | weight time based division of disk policy. It is implemented in CFQ. Hence | |
13 | this policy takes effect only on leaf nodes when CFQ is being used. The second | |
14 | one is throttling policy which can be used to specify upper IO rate limits | |
15 | on devices. This policy is implemented in generic block layer and can be | |
16 | used on leaf nodes as well as higher level logical devices like device mapper. | |
72f924f6 VG |
17 | |
18 | HOWTO | |
19 | ===== | |
2786c4e5 VG |
20 | Proportional Weight division of bandwidth |
21 | ----------------------------------------- | |
72f924f6 VG |
22 | You can do a very simple testing of running two dd threads in two different |
23 | cgroups. Here is what you can do. | |
24 | ||
afc24d49 VG |
25 | - Enable Block IO controller |
26 | CONFIG_BLK_CGROUP=y | |
27 | ||
72f924f6 VG |
28 | - Enable group scheduling in CFQ |
29 | CONFIG_CFQ_GROUP_IOSCHED=y | |
30 | ||
31 | - Compile and boot into kernel and mount IO controller (blkio). | |
32 | ||
33 | mount -t cgroup -o blkio none /cgroup | |
34 | ||
35 | - Create two cgroups | |
36 | mkdir -p /cgroup/test1/ /cgroup/test2 | |
37 | ||
38 | - Set weights of group test1 and test2 | |
39 | echo 1000 > /cgroup/test1/blkio.weight | |
40 | echo 500 > /cgroup/test2/blkio.weight | |
41 | ||
42 | - Create two same size files (say 512MB each) on same disk (file1, file2) and | |
43 | launch two dd threads in different cgroup to read those files. | |
44 | ||
45 | sync | |
46 | echo 3 > /proc/sys/vm/drop_caches | |
47 | ||
48 | dd if=/mnt/sdb/zerofile1 of=/dev/null & | |
49 | echo $! > /cgroup/test1/tasks | |
50 | cat /cgroup/test1/tasks | |
51 | ||
52 | dd if=/mnt/sdb/zerofile2 of=/dev/null & | |
53 | echo $! > /cgroup/test2/tasks | |
54 | cat /cgroup/test2/tasks | |
55 | ||
56 | - At macro level, first dd should finish first. To get more precise data, keep | |
57 | on looking at (with the help of script), at blkio.disk_time and | |
58 | blkio.disk_sectors files of both test1 and test2 groups. This will tell how | |
59 | much disk time (in milli seconds), each group got and how many secotors each | |
60 | group dispatched to the disk. We provide fairness in terms of disk time, so | |
61 | ideally io.disk_time of cgroups should be in proportion to the weight. | |
62 | ||
2786c4e5 VG |
63 | Throttling/Upper Limit policy |
64 | ----------------------------- | |
65 | - Enable Block IO controller | |
66 | CONFIG_BLK_CGROUP=y | |
67 | ||
68 | - Enable throttling in block layer | |
69 | CONFIG_BLK_DEV_THROTTLING=y | |
70 | ||
71 | - Mount blkio controller | |
72 | mount -t cgroup -o blkio none /cgroup/blkio | |
73 | ||
74 | - Specify a bandwidth rate on particular device for root group. The format | |
75 | for policy is "<major>:<minor> <byes_per_second>". | |
76 | ||
77 | echo "8:16 1048576" > /cgroup/blkio/blkio.read_bps_device | |
78 | ||
79 | Above will put a limit of 1MB/second on reads happening for root group | |
80 | on device having major/minor number 8:16. | |
81 | ||
82 | - Run dd to read a file and see if rate is throttled to 1MB/s or not. | |
83 | ||
84 | # dd if=/mnt/common/zerofile of=/dev/null bs=4K count=1024 | |
85 | # iflag=direct | |
86 | 1024+0 records in | |
87 | 1024+0 records out | |
88 | 4194304 bytes (4.2 MB) copied, 4.0001 s, 1.0 MB/s | |
89 | ||
90 | Limits for writes can be put using blkio.write_bps_device file. | |
91 | ||
bdc85df7 VG |
92 | Hierarchical Cgroups |
93 | ==================== | |
94 | - Currently none of the IO control policy supports hierarhical groups. But | |
95 | cgroup interface does allow creation of hierarhical cgroups and internally | |
96 | IO policies treat them as flat hierarchy. | |
97 | ||
98 | So this patch will allow creation of cgroup hierarhcy but at the backend | |
99 | everything will be treated as flat. So if somebody created a hierarchy like | |
100 | as follows. | |
101 | ||
102 | root | |
103 | / \ | |
104 | test1 test2 | |
105 | | | |
106 | test3 | |
107 | ||
108 | CFQ and throttling will practically treat all groups at same level. | |
109 | ||
110 | pivot | |
111 | / | \ \ | |
112 | root test1 test2 test3 | |
113 | ||
114 | Down the line we can implement hierarchical accounting/control support | |
115 | and also introduce a new cgroup file "use_hierarchy" which will control | |
116 | whether cgroup hierarchy is viewed as flat or hierarchical by the policy.. | |
117 | This is how memory controller also has implemented the things. | |
118 | ||
72f924f6 VG |
119 | Various user visible config options |
120 | =================================== | |
72f924f6 | 121 | CONFIG_BLK_CGROUP |
afc24d49 | 122 | - Block IO controller. |
72f924f6 VG |
123 | |
124 | CONFIG_DEBUG_BLK_CGROUP | |
afc24d49 VG |
125 | - Debug help. Right now some additional stats file show up in cgroup |
126 | if this option is enabled. | |
127 | ||
128 | CONFIG_CFQ_GROUP_IOSCHED | |
129 | - Enables group scheduling in CFQ. Currently only 1 level of group | |
130 | creation is allowed. | |
72f924f6 | 131 | |
2786c4e5 VG |
132 | CONFIG_BLK_DEV_THROTTLING |
133 | - Enable block device throttling support in block layer. | |
134 | ||
72f924f6 VG |
135 | Details of cgroup files |
136 | ======================= | |
2786c4e5 VG |
137 | Proportional weight policy files |
138 | -------------------------------- | |
72f924f6 | 139 | - blkio.weight |
da69da18 GJ |
140 | - Specifies per cgroup weight. This is default weight of the group |
141 | on all the devices until and unless overridden by per device rule. | |
142 | (See blkio.weight_device). | |
df457f84 | 143 | Currently allowed range of weights is from 10 to 1000. |
72f924f6 | 144 | |
da69da18 GJ |
145 | - blkio.weight_device |
146 | - One can specify per cgroup per device rules using this interface. | |
147 | These rules override the default value of group weight as specified | |
148 | by blkio.weight. | |
149 | ||
150 | Following is the format. | |
151 | ||
152 | #echo dev_maj:dev_minor weight > /path/to/cgroup/blkio.weight_device | |
153 | Configure weight=300 on /dev/sdb (8:16) in this cgroup | |
154 | # echo 8:16 300 > blkio.weight_device | |
155 | # cat blkio.weight_device | |
156 | dev weight | |
157 | 8:16 300 | |
158 | ||
159 | Configure weight=500 on /dev/sda (8:0) in this cgroup | |
160 | # echo 8:0 500 > blkio.weight_device | |
161 | # cat blkio.weight_device | |
162 | dev weight | |
163 | 8:0 500 | |
164 | 8:16 300 | |
165 | ||
166 | Remove specific weight for /dev/sda in this cgroup | |
167 | # echo 8:0 0 > blkio.weight_device | |
168 | # cat blkio.weight_device | |
169 | dev weight | |
170 | 8:16 300 | |
171 | ||
72f924f6 VG |
172 | - blkio.time |
173 | - disk time allocated to cgroup per device in milliseconds. First | |
174 | two fields specify the major and minor number of the device and | |
175 | third field specifies the disk time allocated to group in | |
176 | milliseconds. | |
177 | ||
178 | - blkio.sectors | |
179 | - number of sectors transferred to/from disk by the group. First | |
180 | two fields specify the major and minor number of the device and | |
181 | third field specifies the number of sectors transferred by the | |
182 | group to/from the device. | |
183 | ||
84c124da DS |
184 | - blkio.io_service_bytes |
185 | - Number of bytes transferred to/from the disk by the group. These | |
186 | are further divided by the type of operation - read or write, sync | |
187 | or async. First two fields specify the major and minor number of the | |
188 | device, third field specifies the operation type and the fourth field | |
189 | specifies the number of bytes. | |
190 | ||
191 | - blkio.io_serviced | |
192 | - Number of IOs completed to/from the disk by the group. These | |
193 | are further divided by the type of operation - read or write, sync | |
194 | or async. First two fields specify the major and minor number of the | |
195 | device, third field specifies the operation type and the fourth field | |
196 | specifies the number of IOs. | |
197 | ||
198 | - blkio.io_service_time | |
199 | - Total amount of time between request dispatch and request completion | |
200 | for the IOs done by this cgroup. This is in nanoseconds to make it | |
201 | meaningful for flash devices too. For devices with queue depth of 1, | |
202 | this time represents the actual service time. When queue_depth > 1, | |
203 | that is no longer true as requests may be served out of order. This | |
204 | may cause the service time for a given IO to include the service time | |
205 | of multiple IOs when served out of order which may result in total | |
206 | io_service_time > actual time elapsed. This time is further divided by | |
207 | the type of operation - read or write, sync or async. First two fields | |
208 | specify the major and minor number of the device, third field | |
209 | specifies the operation type and the fourth field specifies the | |
210 | io_service_time in ns. | |
211 | ||
212 | - blkio.io_wait_time | |
213 | - Total amount of time the IOs for this cgroup spent waiting in the | |
214 | scheduler queues for service. This can be greater than the total time | |
215 | elapsed since it is cumulative io_wait_time for all IOs. It is not a | |
216 | measure of total time the cgroup spent waiting but rather a measure of | |
217 | the wait_time for its individual IOs. For devices with queue_depth > 1 | |
218 | this metric does not include the time spent waiting for service once | |
219 | the IO is dispatched to the device but till it actually gets serviced | |
220 | (there might be a time lag here due to re-ordering of requests by the | |
221 | device). This is in nanoseconds to make it meaningful for flash | |
222 | devices too. This time is further divided by the type of operation - | |
223 | read or write, sync or async. First two fields specify the major and | |
224 | minor number of the device, third field specifies the operation type | |
225 | and the fourth field specifies the io_wait_time in ns. | |
226 | ||
812d4026 DS |
227 | - blkio.io_merged |
228 | - Total number of bios/requests merged into requests belonging to this | |
229 | cgroup. This is further divided by the type of operation - read or | |
230 | write, sync or async. | |
231 | ||
cdc1184c DS |
232 | - blkio.io_queued |
233 | - Total number of requests queued up at any given instant for this | |
234 | cgroup. This is further divided by the type of operation - read or | |
235 | write, sync or async. | |
236 | ||
237 | - blkio.avg_queue_size | |
afc24d49 | 238 | - Debugging aid only enabled if CONFIG_DEBUG_BLK_CGROUP=y. |
cdc1184c DS |
239 | The average queue size for this cgroup over the entire time of this |
240 | cgroup's existence. Queue size samples are taken each time one of the | |
241 | queues of this cgroup gets a timeslice. | |
242 | ||
812df48d | 243 | - blkio.group_wait_time |
afc24d49 | 244 | - Debugging aid only enabled if CONFIG_DEBUG_BLK_CGROUP=y. |
812df48d DS |
245 | This is the amount of time the cgroup had to wait since it became busy |
246 | (i.e., went from 0 to 1 request queued) to get a timeslice for one of | |
247 | its queues. This is different from the io_wait_time which is the | |
248 | cumulative total of the amount of time spent by each IO in that cgroup | |
249 | waiting in the scheduler queue. This is in nanoseconds. If this is | |
250 | read when the cgroup is in a waiting (for timeslice) state, the stat | |
251 | will only report the group_wait_time accumulated till the last time it | |
252 | got a timeslice and will not include the current delta. | |
253 | ||
254 | - blkio.empty_time | |
afc24d49 | 255 | - Debugging aid only enabled if CONFIG_DEBUG_BLK_CGROUP=y. |
812df48d DS |
256 | This is the amount of time a cgroup spends without any pending |
257 | requests when not being served, i.e., it does not include any time | |
258 | spent idling for one of the queues of the cgroup. This is in | |
259 | nanoseconds. If this is read when the cgroup is in an empty state, | |
260 | the stat will only report the empty_time accumulated till the last | |
261 | time it had a pending request and will not include the current delta. | |
262 | ||
263 | - blkio.idle_time | |
afc24d49 | 264 | - Debugging aid only enabled if CONFIG_DEBUG_BLK_CGROUP=y. |
812df48d DS |
265 | This is the amount of time spent by the IO scheduler idling for a |
266 | given cgroup in anticipation of a better request than the exising ones | |
267 | from other queues/cgroups. This is in nanoseconds. If this is read | |
268 | when the cgroup is in an idling state, the stat will only report the | |
269 | idle_time accumulated till the last idle period and will not include | |
270 | the current delta. | |
271 | ||
72f924f6 | 272 | - blkio.dequeue |
afc24d49 | 273 | - Debugging aid only enabled if CONFIG_DEBUG_BLK_CGROUP=y. This |
72f924f6 VG |
274 | gives the statistics about how many a times a group was dequeued |
275 | from service tree of the device. First two fields specify the major | |
276 | and minor number of the device and third field specifies the number | |
277 | of times a group was dequeued from a particular device. | |
278 | ||
2786c4e5 VG |
279 | Throttling/Upper limit policy files |
280 | ----------------------------------- | |
281 | - blkio.throttle.read_bps_device | |
282 | - Specifies upper limit on READ rate from the device. IO rate is | |
283 | specified in bytes per second. Rules are per deivce. Following is | |
284 | the format. | |
285 | ||
286 | echo "<major>:<minor> <rate_bytes_per_second>" > /cgrp/blkio.read_bps_device | |
287 | ||
288 | - blkio.throttle.write_bps_device | |
289 | - Specifies upper limit on WRITE rate to the device. IO rate is | |
290 | specified in bytes per second. Rules are per deivce. Following is | |
291 | the format. | |
292 | ||
293 | echo "<major>:<minor> <rate_bytes_per_second>" > /cgrp/blkio.write_bps_device | |
294 | ||
295 | - blkio.throttle.read_iops_device | |
296 | - Specifies upper limit on READ rate from the device. IO rate is | |
297 | specified in IO per second. Rules are per deivce. Following is | |
298 | the format. | |
299 | ||
300 | echo "<major>:<minor> <rate_io_per_second>" > /cgrp/blkio.read_iops_device | |
301 | ||
302 | - blkio.throttle.write_iops_device | |
303 | - Specifies upper limit on WRITE rate to the device. IO rate is | |
304 | specified in io per second. Rules are per deivce. Following is | |
305 | the format. | |
306 | ||
307 | echo "<major>:<minor> <rate_io_per_second>" > /cgrp/blkio.write_iops_device | |
308 | ||
309 | Note: If both BW and IOPS rules are specified for a device, then IO is | |
310 | subjectd to both the constraints. | |
311 | ||
312 | - blkio.throttle.io_serviced | |
313 | - Number of IOs (bio) completed to/from the disk by the group (as | |
314 | seen by throttling policy). These are further divided by the type | |
315 | of operation - read or write, sync or async. First two fields specify | |
316 | the major and minor number of the device, third field specifies the | |
317 | operation type and the fourth field specifies the number of IOs. | |
318 | ||
319 | blkio.io_serviced does accounting as seen by CFQ and counts are in | |
320 | number of requests (struct request). On the other hand, | |
321 | blkio.throttle.io_serviced counts number of IO in terms of number | |
322 | of bios as seen by throttling policy. These bios can later be | |
323 | merged by elevator and total number of requests completed can be | |
324 | lesser. | |
325 | ||
326 | - blkio.throttle.io_service_bytes | |
327 | - Number of bytes transferred to/from the disk by the group. These | |
328 | are further divided by the type of operation - read or write, sync | |
329 | or async. First two fields specify the major and minor number of the | |
330 | device, third field specifies the operation type and the fourth field | |
331 | specifies the number of bytes. | |
332 | ||
333 | These numbers should roughly be same as blkio.io_service_bytes as | |
334 | updated by CFQ. The difference between two is that | |
335 | blkio.io_service_bytes will not be updated if CFQ is not operating | |
336 | on request queue. | |
337 | ||
338 | Common files among various policies | |
339 | ----------------------------------- | |
84c124da DS |
340 | - blkio.reset_stats |
341 | - Writing an int to this file will result in resetting all the stats | |
342 | for that cgroup. | |
343 | ||
72f924f6 VG |
344 | CFQ sysfs tunable |
345 | ================= | |
6d6ac1c1 VG |
346 | /sys/block/<disk>/queue/iosched/slice_idle |
347 | ------------------------------------------ | |
348 | On a faster hardware CFQ can be slow, especially with sequential workload. | |
349 | This happens because CFQ idles on a single queue and single queue might not | |
350 | drive deeper request queue depths to keep the storage busy. In such scenarios | |
351 | one can try setting slice_idle=0 and that would switch CFQ to IOPS | |
352 | (IO operations per second) mode on NCQ supporting hardware. | |
353 | ||
354 | That means CFQ will not idle between cfq queues of a cfq group and hence be | |
355 | able to driver higher queue depth and achieve better throughput. That also | |
356 | means that cfq provides fairness among groups in terms of IOPS and not in | |
357 | terms of disk time. | |
358 | ||
359 | /sys/block/<disk>/queue/iosched/group_idle | |
360 | ------------------------------------------ | |
361 | If one disables idling on individual cfq queues and cfq service trees by | |
362 | setting slice_idle=0, group_idle kicks in. That means CFQ will still idle | |
363 | on the group in an attempt to provide fairness among groups. | |
364 | ||
365 | By default group_idle is same as slice_idle and does not do anything if | |
366 | slice_idle is enabled. | |
367 | ||
368 | One can experience an overall throughput drop if you have created multiple | |
369 | groups and put applications in that group which are not driving enough | |
370 | IO to keep disk busy. In that case set group_idle=0, and CFQ will not idle | |
371 | on individual groups and throughput should improve. | |
372 | ||
72f924f6 VG |
373 | What works |
374 | ========== | |
375 | - Currently only sync IO queues are support. All the buffered writes are | |
376 | still system wide and not per group. Hence we will not see service | |
377 | differentiation between buffered writes between groups. |