]>
Commit | Line | Data |
---|---|---|
72f924f6 VG |
1 | Block IO Controller |
2 | =================== | |
3 | Overview | |
4 | ======== | |
5 | cgroup subsys "blkio" implements the block io controller. There seems to be | |
6 | a need of various kinds of IO control policies (like proportional BW, max BW) | |
7 | both at leaf nodes as well as at intermediate nodes in a storage hierarchy. | |
8 | Plan is to use the same cgroup based management interface for blkio controller | |
9 | and based on user options switch IO policies in the background. | |
10 | ||
11 | In the first phase, this patchset implements proportional weight time based | |
12 | division of disk policy. It is implemented in CFQ. Hence this policy takes | |
13 | effect only on leaf nodes when CFQ is being used. | |
14 | ||
15 | HOWTO | |
16 | ===== | |
17 | You can do a very simple testing of running two dd threads in two different | |
18 | cgroups. Here is what you can do. | |
19 | ||
20 | - Enable group scheduling in CFQ | |
21 | CONFIG_CFQ_GROUP_IOSCHED=y | |
22 | ||
23 | - Compile and boot into kernel and mount IO controller (blkio). | |
24 | ||
25 | mount -t cgroup -o blkio none /cgroup | |
26 | ||
27 | - Create two cgroups | |
28 | mkdir -p /cgroup/test1/ /cgroup/test2 | |
29 | ||
30 | - Set weights of group test1 and test2 | |
31 | echo 1000 > /cgroup/test1/blkio.weight | |
32 | echo 500 > /cgroup/test2/blkio.weight | |
33 | ||
34 | - Create two same size files (say 512MB each) on same disk (file1, file2) and | |
35 | launch two dd threads in different cgroup to read those files. | |
36 | ||
37 | sync | |
38 | echo 3 > /proc/sys/vm/drop_caches | |
39 | ||
40 | dd if=/mnt/sdb/zerofile1 of=/dev/null & | |
41 | echo $! > /cgroup/test1/tasks | |
42 | cat /cgroup/test1/tasks | |
43 | ||
44 | dd if=/mnt/sdb/zerofile2 of=/dev/null & | |
45 | echo $! > /cgroup/test2/tasks | |
46 | cat /cgroup/test2/tasks | |
47 | ||
48 | - At macro level, first dd should finish first. To get more precise data, keep | |
49 | on looking at (with the help of script), at blkio.disk_time and | |
50 | blkio.disk_sectors files of both test1 and test2 groups. This will tell how | |
51 | much disk time (in milli seconds), each group got and how many secotors each | |
52 | group dispatched to the disk. We provide fairness in terms of disk time, so | |
53 | ideally io.disk_time of cgroups should be in proportion to the weight. | |
54 | ||
55 | Various user visible config options | |
56 | =================================== | |
57 | CONFIG_CFQ_GROUP_IOSCHED | |
58 | - Enables group scheduling in CFQ. Currently only 1 level of group | |
59 | creation is allowed. | |
60 | ||
61 | CONFIG_DEBUG_CFQ_IOSCHED | |
62 | - Enables some debugging messages in blktrace. Also creates extra | |
63 | cgroup file blkio.dequeue. | |
64 | ||
65 | Config options selected automatically | |
66 | ===================================== | |
67 | These config options are not user visible and are selected/deselected | |
68 | automatically based on IO scheduler configuration. | |
69 | ||
70 | CONFIG_BLK_CGROUP | |
71 | - Block IO controller. Selected by CONFIG_CFQ_GROUP_IOSCHED. | |
72 | ||
73 | CONFIG_DEBUG_BLK_CGROUP | |
74 | - Debug help. Selected by CONFIG_DEBUG_CFQ_IOSCHED. | |
75 | ||
76 | Details of cgroup files | |
77 | ======================= | |
78 | - blkio.weight | |
79 | - Specifies per cgroup weight. | |
72f924f6 VG |
80 | Currently allowed range of weights is from 100 to 1000. |
81 | ||
82 | - blkio.time | |
83 | - disk time allocated to cgroup per device in milliseconds. First | |
84 | two fields specify the major and minor number of the device and | |
85 | third field specifies the disk time allocated to group in | |
86 | milliseconds. | |
87 | ||
88 | - blkio.sectors | |
89 | - number of sectors transferred to/from disk by the group. First | |
90 | two fields specify the major and minor number of the device and | |
91 | third field specifies the number of sectors transferred by the | |
92 | group to/from the device. | |
93 | ||
84c124da DS |
94 | - blkio.io_service_bytes |
95 | - Number of bytes transferred to/from the disk by the group. These | |
96 | are further divided by the type of operation - read or write, sync | |
97 | or async. First two fields specify the major and minor number of the | |
98 | device, third field specifies the operation type and the fourth field | |
99 | specifies the number of bytes. | |
100 | ||
101 | - blkio.io_serviced | |
102 | - Number of IOs completed to/from the disk by the group. These | |
103 | are further divided by the type of operation - read or write, sync | |
104 | or async. First two fields specify the major and minor number of the | |
105 | device, third field specifies the operation type and the fourth field | |
106 | specifies the number of IOs. | |
107 | ||
108 | - blkio.io_service_time | |
109 | - Total amount of time between request dispatch and request completion | |
110 | for the IOs done by this cgroup. This is in nanoseconds to make it | |
111 | meaningful for flash devices too. For devices with queue depth of 1, | |
112 | this time represents the actual service time. When queue_depth > 1, | |
113 | that is no longer true as requests may be served out of order. This | |
114 | may cause the service time for a given IO to include the service time | |
115 | of multiple IOs when served out of order which may result in total | |
116 | io_service_time > actual time elapsed. This time is further divided by | |
117 | the type of operation - read or write, sync or async. First two fields | |
118 | specify the major and minor number of the device, third field | |
119 | specifies the operation type and the fourth field specifies the | |
120 | io_service_time in ns. | |
121 | ||
122 | - blkio.io_wait_time | |
123 | - Total amount of time the IOs for this cgroup spent waiting in the | |
124 | scheduler queues for service. This can be greater than the total time | |
125 | elapsed since it is cumulative io_wait_time for all IOs. It is not a | |
126 | measure of total time the cgroup spent waiting but rather a measure of | |
127 | the wait_time for its individual IOs. For devices with queue_depth > 1 | |
128 | this metric does not include the time spent waiting for service once | |
129 | the IO is dispatched to the device but till it actually gets serviced | |
130 | (there might be a time lag here due to re-ordering of requests by the | |
131 | device). This is in nanoseconds to make it meaningful for flash | |
132 | devices too. This time is further divided by the type of operation - | |
133 | read or write, sync or async. First two fields specify the major and | |
134 | minor number of the device, third field specifies the operation type | |
135 | and the fourth field specifies the io_wait_time in ns. | |
136 | ||
812d4026 DS |
137 | - blkio.io_merged |
138 | - Total number of bios/requests merged into requests belonging to this | |
139 | cgroup. This is further divided by the type of operation - read or | |
140 | write, sync or async. | |
141 | ||
cdc1184c DS |
142 | - blkio.io_queued |
143 | - Total number of requests queued up at any given instant for this | |
144 | cgroup. This is further divided by the type of operation - read or | |
145 | write, sync or async. | |
146 | ||
147 | - blkio.avg_queue_size | |
148 | - Debugging aid only enabled if CONFIG_DEBUG_CFQ_IOSCHED=y. | |
149 | The average queue size for this cgroup over the entire time of this | |
150 | cgroup's existence. Queue size samples are taken each time one of the | |
151 | queues of this cgroup gets a timeslice. | |
152 | ||
812df48d DS |
153 | - blkio.group_wait_time |
154 | - Debugging aid only enabled if CONFIG_DEBUG_CFQ_IOSCHED=y. | |
155 | This is the amount of time the cgroup had to wait since it became busy | |
156 | (i.e., went from 0 to 1 request queued) to get a timeslice for one of | |
157 | its queues. This is different from the io_wait_time which is the | |
158 | cumulative total of the amount of time spent by each IO in that cgroup | |
159 | waiting in the scheduler queue. This is in nanoseconds. If this is | |
160 | read when the cgroup is in a waiting (for timeslice) state, the stat | |
161 | will only report the group_wait_time accumulated till the last time it | |
162 | got a timeslice and will not include the current delta. | |
163 | ||
164 | - blkio.empty_time | |
165 | - Debugging aid only enabled if CONFIG_DEBUG_CFQ_IOSCHED=y. | |
166 | This is the amount of time a cgroup spends without any pending | |
167 | requests when not being served, i.e., it does not include any time | |
168 | spent idling for one of the queues of the cgroup. This is in | |
169 | nanoseconds. If this is read when the cgroup is in an empty state, | |
170 | the stat will only report the empty_time accumulated till the last | |
171 | time it had a pending request and will not include the current delta. | |
172 | ||
173 | - blkio.idle_time | |
174 | - Debugging aid only enabled if CONFIG_DEBUG_CFQ_IOSCHED=y. | |
175 | This is the amount of time spent by the IO scheduler idling for a | |
176 | given cgroup in anticipation of a better request than the exising ones | |
177 | from other queues/cgroups. This is in nanoseconds. If this is read | |
178 | when the cgroup is in an idling state, the stat will only report the | |
179 | idle_time accumulated till the last idle period and will not include | |
180 | the current delta. | |
181 | ||
72f924f6 VG |
182 | - blkio.dequeue |
183 | - Debugging aid only enabled if CONFIG_DEBUG_CFQ_IOSCHED=y. This | |
184 | gives the statistics about how many a times a group was dequeued | |
185 | from service tree of the device. First two fields specify the major | |
186 | and minor number of the device and third field specifies the number | |
187 | of times a group was dequeued from a particular device. | |
188 | ||
84c124da DS |
189 | - blkio.reset_stats |
190 | - Writing an int to this file will result in resetting all the stats | |
191 | for that cgroup. | |
192 | ||
72f924f6 VG |
193 | CFQ sysfs tunable |
194 | ================= | |
195 | /sys/block/<disk>/queue/iosched/group_isolation | |
196 | ||
197 | If group_isolation=1, it provides stronger isolation between groups at the | |
198 | expense of throughput. By default group_isolation is 0. In general that | |
199 | means that if group_isolation=0, expect fairness for sequential workload | |
200 | only. Set group_isolation=1 to see fairness for random IO workload also. | |
201 | ||
202 | Generally CFQ will put random seeky workload in sync-noidle category. CFQ | |
203 | will disable idling on these queues and it does a collective idling on group | |
204 | of such queues. Generally these are slow moving queues and if there is a | |
205 | sync-noidle service tree in each group, that group gets exclusive access to | |
206 | disk for certain period. That means it will bring the throughput down if | |
207 | group does not have enough IO to drive deeper queue depths and utilize disk | |
208 | capacity to the fullest in the slice allocated to it. But the flip side is | |
209 | that even a random reader should get better latencies and overall throughput | |
210 | if there are lots of sequential readers/sync-idle workload running in the | |
211 | system. | |
212 | ||
213 | If group_isolation=0, then CFQ automatically moves all the random seeky queues | |
214 | in the root group. That means there will be no service differentiation for | |
215 | that kind of workload. This leads to better throughput as we do collective | |
216 | idling on root sync-noidle tree. | |
217 | ||
218 | By default one should run with group_isolation=0. If that is not sufficient | |
219 | and one wants stronger isolation between groups, then set group_isolation=1 | |
220 | but this will come at cost of reduced throughput. | |
221 | ||
222 | What works | |
223 | ========== | |
224 | - Currently only sync IO queues are support. All the buffered writes are | |
225 | still system wide and not per group. Hence we will not see service | |
226 | differentiation between buffered writes between groups. |