]>
Commit | Line | Data |
---|---|---|
1da177e4 LT |
1 | Tools that manage md devices can be found at |
2 | http://www.<country>.kernel.org/pub/linux/utils/raid/.... | |
3 | ||
4 | ||
5 | Boot time assembly of RAID arrays | |
6 | --------------------------------- | |
7 | ||
8 | You can boot with your md device with the following kernel command | |
9 | lines: | |
10 | ||
11 | for old raid arrays without persistent superblocks: | |
12 | md=<md device no.>,<raid level>,<chunk size factor>,<fault level>,dev0,dev1,...,devn | |
13 | ||
14 | for raid arrays with persistent superblocks | |
15 | md=<md device no.>,dev0,dev1,...,devn | |
16 | or, to assemble a partitionable array: | |
17 | md=d<md device no.>,dev0,dev1,...,devn | |
18 | ||
19 | md device no. = the number of the md device ... | |
20 | 0 means md0, | |
21 | 1 md1, | |
22 | 2 md2, | |
23 | 3 md3, | |
24 | 4 md4 | |
25 | ||
26 | raid level = -1 linear mode | |
27 | 0 striped mode | |
28 | other modes are only supported with persistent super blocks | |
29 | ||
30 | chunk size factor = (raid-0 and raid-1 only) | |
31 | Set the chunk size as 4k << n. | |
32 | ||
33 | fault level = totally ignored | |
34 | ||
35 | dev0-devn: e.g. /dev/hda1,/dev/hdc1,/dev/sda1,/dev/sdb1 | |
36 | ||
37 | A possible loadlin line (Harald Hoyer <HarryH@Royal.Net>) looks like this: | |
38 | ||
39 | e:\loadlin\loadlin e:\zimage root=/dev/md0 md=0,0,4,0,/dev/hdb2,/dev/hdc3 ro | |
40 | ||
41 | ||
42 | Boot time autodetection of RAID arrays | |
43 | -------------------------------------- | |
44 | ||
45 | When md is compiled into the kernel (not as module), partitions of | |
46 | type 0xfd are scanned and automatically assembled into RAID arrays. | |
47 | This autodetection may be suppressed with the kernel parameter | |
48 | "raid=noautodetect". As of kernel 2.6.9, only drives with a type 0 | |
49 | superblock can be autodetected and run at boot time. | |
50 | ||
51 | The kernel parameter "raid=partitionable" (or "raid=part") means | |
52 | that all auto-detected arrays are assembled as partitionable. | |
53 | ||
6ff8d8ec N |
54 | Boot time assembly of degraded/dirty arrays |
55 | ------------------------------------------- | |
56 | ||
57 | If a raid5 or raid6 array is both dirty and degraded, it could have | |
58 | undetectable data corruption. This is because the fact that it is | |
59 | 'dirty' means that the parity cannot be trusted, and the fact that it | |
60 | is degraded means that some datablocks are missing and cannot reliably | |
61 | be reconstructed (due to no parity). | |
62 | ||
63 | For this reason, md will normally refuse to start such an array. This | |
64 | requires the sysadmin to take action to explicitly start the array | |
fff9289b | 65 | despite possible corruption. This is normally done with |
6ff8d8ec N |
66 | mdadm --assemble --force .... |
67 | ||
68 | This option is not really available if the array has the root | |
69 | filesystem on it. In order to support this booting from such an | |
70 | array, md supports a module parameter "start_dirty_degraded" which, | |
71 | when set to 1, bypassed the checks and will allows dirty degraded | |
72 | arrays to be started. | |
73 | ||
74 | So, to boot with a root filesystem of a dirty degraded raid[56], use | |
75 | ||
76 | md-mod.start_dirty_degraded=1 | |
77 | ||
1da177e4 LT |
78 | |
79 | Superblock formats | |
80 | ------------------ | |
81 | ||
82 | The md driver can support a variety of different superblock formats. | |
83 | Currently, it supports superblock formats "0.90.0" and the "md-1" format | |
84 | introduced in the 2.5 development series. | |
85 | ||
86 | The kernel will autodetect which format superblock is being used. | |
87 | ||
88 | Superblock format '0' is treated differently to others for legacy | |
89 | reasons - it is the original superblock format. | |
90 | ||
91 | ||
92 | General Rules - apply for all superblock formats | |
93 | ------------------------------------------------ | |
94 | ||
95 | An array is 'created' by writing appropriate superblocks to all | |
96 | devices. | |
97 | ||
98 | It is 'assembled' by associating each of these devices with an | |
99 | particular md virtual device. Once it is completely assembled, it can | |
100 | be accessed. | |
101 | ||
102 | An array should be created by a user-space tool. This will write | |
103 | superblocks to all devices. It will usually mark the array as | |
104 | 'unclean', or with some devices missing so that the kernel md driver | |
105 | can create appropriate redundancy (copying in raid1, parity | |
106 | calculation in raid4/5). | |
107 | ||
108 | When an array is assembled, it is first initialized with the | |
109 | SET_ARRAY_INFO ioctl. This contains, in particular, a major and minor | |
110 | version number. The major version number selects which superblock | |
111 | format is to be used. The minor number might be used to tune handling | |
112 | of the format, such as suggesting where on each device to look for the | |
113 | superblock. | |
114 | ||
115 | Then each device is added using the ADD_NEW_DISK ioctl. This | |
116 | provides, in particular, a major and minor number identifying the | |
117 | device to add. | |
118 | ||
119 | The array is started with the RUN_ARRAY ioctl. | |
120 | ||
121 | Once started, new devices can be added. They should have an | |
122 | appropriate superblock written to them, and then passed be in with | |
123 | ADD_NEW_DISK. | |
124 | ||
125 | Devices that have failed or are not yet active can be detached from an | |
126 | array using HOT_REMOVE_DISK. | |
127 | ||
128 | ||
129 | Specific Rules that apply to format-0 super block arrays, and | |
130 | arrays with no superblock (non-persistent). | |
131 | ------------------------------------------------------------- | |
132 | ||
133 | An array can be 'created' by describing the array (level, chunksize | |
134 | etc) in a SET_ARRAY_INFO ioctl. This must has major_version==0 and | |
135 | raid_disks != 0. | |
136 | ||
137 | Then uninitialized devices can be added with ADD_NEW_DISK. The | |
138 | structure passed to ADD_NEW_DISK must specify the state of the device | |
139 | and it's role in the array. | |
140 | ||
141 | Once started with RUN_ARRAY, uninitialized spares can be added with | |
142 | HOT_ADD_DISK. | |
bb636547 N |
143 | |
144 | ||
145 | ||
146 | MD devices in sysfs | |
147 | ------------------- | |
148 | md devices appear in sysfs (/sys) as regular block devices, | |
149 | e.g. | |
150 | /sys/block/md0 | |
151 | ||
152 | Each 'md' device will contain a subdirectory called 'md' which | |
153 | contains further md-specific information about the device. | |
154 | ||
155 | All md devices contain: | |
156 | level | |
d33a56d3 N |
157 | a text file indicating the 'raid level'. e.g. raid0, raid1, |
158 | raid5, linear, multipath, faulty. | |
bb636547 | 159 | If no raid level has been set yet (array is still being |
d33a56d3 N |
160 | assembled), the value will reflect whatever has been written |
161 | to it, which may be a name like the above, or may be a number | |
162 | such as '0', '5', etc. | |
bb636547 N |
163 | |
164 | raid_disks | |
165 | a text file with a simple number indicating the number of devices | |
166 | in a fully functional array. If this is not yet known, the file | |
167 | will be empty. If an array is being resized (not currently | |
168 | possible) this will contain the larger of the old and new sizes. | |
da943b99 N |
169 | Some raid level (RAID1) allow this value to be set while the |
170 | array is active. This will reconfigure the array. Otherwise | |
171 | it can only be set while assembling an array. | |
bb636547 | 172 | |
3b34380a N |
173 | chunk_size |
174 | This is the size if bytes for 'chunks' and is only relevant to | |
175 | raid levels that involve striping (1,4,5,6,10). The address space | |
176 | of the array is conceptually divided into chunks and consecutive | |
177 | chunks are striped onto neighbouring devices. | |
3f6dee9b | 178 | The size should be at least PAGE_SIZE (4k) and should be a power |
3b34380a N |
179 | of 2. This can only be set while assembling an array |
180 | ||
a35b0d69 N |
181 | component_size |
182 | For arrays with data redundancy (i.e. not raid0, linear, faulty, | |
183 | multipath), all components must be the same size - or at least | |
184 | there must a size that they all provide space for. This is a key | |
185 | part or the geometry of the array. It is measured in sectors | |
186 | and can be read from here. Writing to this value may resize | |
187 | the array if the personality supports it (raid1, raid5, raid6), | |
188 | and if the component drives are large enough. | |
189 | ||
8bb93aac N |
190 | metadata_version |
191 | This indicates the format that is being used to record metadata | |
192 | about the array. It can be 0.90 (traditional format), 1.0, 1.1, | |
193 | 1.2 (newer format in varying locations) or "none" indicating that | |
194 | the kernel isn't managing metadata at all. | |
195 | ||
d4dbd025 N |
196 | layout |
197 | The "layout" for the array for the particular level. This is | |
198 | simply a number that is interpretted differently by different | |
199 | levels. It can be written while assembling an array. | |
200 | ||
a94213b1 N |
201 | resync_start |
202 | The point at which resync should start. If no resync is needed, | |
203 | this will be a very large number. At array creation it will | |
204 | default to 0, though starting the array as 'clean' will | |
205 | set it much larger. | |
206 | ||
6d7ff738 N |
207 | new_dev |
208 | This file can be written but not read. The value written should | |
209 | be a block device number as major:minor. e.g. 8:0 | |
210 | This will cause that device to be attached to the array, if it is | |
211 | available. It will then appear at md/dev-XXX (depending on the | |
212 | name of the device) and further configuration is then possible. | |
213 | ||
16f17b39 N |
214 | safe_mode_delay |
215 | When an md array has seen no write requests for a certain period | |
216 | of time, it will be marked as 'clean'. When another write | |
fff9289b ML |
217 | request arrives, the array is marked as 'dirty' before the write |
218 | commences. This is known as 'safe_mode'. | |
16f17b39 N |
219 | The 'certain period' is controlled by this file which stores the |
220 | period as a number of seconds. The default is 200msec (0.200). | |
221 | Writing a value of 0 disables safemode. | |
222 | ||
9e653b63 N |
223 | array_state |
224 | This file contains a single word which describes the current | |
225 | state of the array. In many cases, the state can be set by | |
226 | writing the word for the desired state, however some states | |
227 | cannot be explicitly set, and some transitions are not allowed. | |
228 | ||
229 | clear | |
230 | No devices, no size, no level | |
231 | Writing is equivalent to STOP_ARRAY ioctl | |
232 | inactive | |
233 | May have some settings, but array is not active | |
234 | all IO results in error | |
235 | When written, doesn't tear down array, but just stops it | |
236 | suspended (not supported yet) | |
237 | All IO requests will block. The array can be reconfigured. | |
238 | Writing this, if accepted, will block until array is quiessent | |
239 | readonly | |
240 | no resync can happen. no superblocks get written. | |
241 | write requests fail | |
242 | read-auto | |
243 | like readonly, but behaves like 'clean' on a write request. | |
244 | ||
245 | clean - no pending writes, but otherwise active. | |
246 | When written to inactive array, starts without resync | |
247 | If a write request arrives then | |
248 | if metadata is known, mark 'dirty' and switch to 'active'. | |
249 | if not known, block and switch to write-pending | |
250 | If written to an active array that has pending writes, then fails. | |
251 | active | |
252 | fully active: IO and resync can be happening. | |
253 | When written to inactive array, starts with resync | |
254 | ||
255 | write-pending | |
256 | clean, but writes are blocked waiting for 'active' to be written. | |
257 | ||
258 | active-idle | |
259 | like active, but no writes have been seen for a while (safe_mode_delay). | |
260 | ||
261 | ||
88202a0c N |
262 | sync_speed_min |
263 | sync_speed_max | |
264 | This are similar to /proc/sys/dev/raid/speed_limit_{min,max} | |
265 | however they only apply to the particular array. | |
266 | If no value has been written to these, of if the word 'system' | |
267 | is written, then the system-wide value is used. If a value, | |
268 | in kibibytes-per-second is written, then it is used. | |
269 | When the files are read, they show the currently active value | |
270 | followed by "(local)" or "(system)" depending on whether it is | |
271 | a locally set or system-wide value. | |
272 | ||
273 | sync_completed | |
274 | This shows the number of sectors that have been completed of | |
275 | whatever the current sync_action is, followed by the number of | |
276 | sectors in total that could need to be processed. The two | |
277 | numbers are separated by a '/' thus effectively showing one | |
278 | value, a fraction of the process that is complete. | |
279 | ||
280 | sync_speed | |
281 | This shows the current actual speed, in K/sec, of the current | |
282 | sync_action. It is averaged over the last 30 seconds. | |
283 | ||
6d7ff738 | 284 | |
bb636547 N |
285 | As component devices are added to an md array, they appear in the 'md' |
286 | directory as new directories named | |
287 | dev-XXX | |
288 | where XXX is a name that the kernel knows for the device, e.g. hdb1. | |
289 | Each directory contains: | |
290 | ||
291 | block | |
292 | a symlink to the block device in /sys/block, e.g. | |
293 | /sys/block/md0/md/dev-hdb1/block -> ../../../../block/hdb/hdb1 | |
294 | ||
295 | super | |
296 | A file containing an image of the superblock read from, or | |
297 | written to, that device. | |
298 | ||
299 | state | |
300 | A file recording the current state of the device in the array | |
301 | which can be a comma separated list of | |
302 | faulty - device has been kicked from active use due to | |
303 | a detected fault | |
304 | in_sync - device is a fully in-sync member of the array | |
f655675b N |
305 | writemostly - device will only be subject to read |
306 | requests if there are no other options. | |
307 | This applies only to raid1 arrays. | |
bb636547 N |
308 | spare - device is working, but not a full member. |
309 | This includes spares that are in the process | |
310 | of being recoverred to | |
311 | This list make grow in future. | |
45dc2de1 N |
312 | This can be written to. |
313 | Writing "faulty" simulates a failure on the device. | |
314 | Writing "remove" removes the device from the array. | |
f655675b N |
315 | Writing "writemostly" sets the writemostly flag. |
316 | Writing "-writemostly" clears the writemostly flag. | |
bb636547 | 317 | |
4dbcdc75 N |
318 | errors |
319 | An approximate count of read errors that have been detected on | |
320 | this device but have not caused the device to be evicted from | |
321 | the array (either because they were corrected or because they | |
322 | happened while the array was read-only). When using version-1 | |
323 | metadata, this value persists across restarts of the array. | |
324 | ||
325 | This value can be written while assembling an array thus | |
326 | providing an ongoing count for arrays with metadata managed by | |
327 | userspace. | |
328 | ||
014236d2 N |
329 | slot |
330 | This gives the role that the device has in the array. It will | |
331 | either be 'none' if the device is not active in the array | |
332 | (i.e. is a spare or has failed) or an integer less than the | |
992caacf | 333 | 'raid_disks' number for the array indicating which position |
014236d2 N |
334 | it currently fills. This can only be set while assembling an |
335 | array. A device for which this is set is assumed to be working. | |
336 | ||
93c8cad0 N |
337 | offset |
338 | This gives the location in the device (in sectors from the | |
339 | start) where data from the array will be stored. Any part of | |
340 | the device before this offset us not touched, unless it is | |
341 | used for storing metadata (Formats 1.1 and 1.2). | |
342 | ||
83303b61 N |
343 | size |
344 | The amount of the device, after the offset, that can be used | |
345 | for storage of data. This will normally be the same as the | |
346 | component_size. This can be written while assembling an | |
347 | array. If a value less than the current component_size is | |
348 | written, component_size will be reduced to this value. | |
349 | ||
bb636547 N |
350 | |
351 | An active md device will also contain and entry for each active device | |
352 | in the array. These are named | |
353 | ||
354 | rdNN | |
355 | ||
992caacf | 356 | where 'NN' is the position in the array, starting from 0. |
bb636547 N |
357 | So for a 3 drive array there will be rd0, rd1, rd2. |
358 | These are symbolic links to the appropriate 'dev-XXX' entry. | |
359 | Thus, for example, | |
360 | cat /sys/block/md*/md/rd*/state | |
361 | will show 'in_sync' on every line. | |
362 | ||
363 | ||
364 | ||
365 | Active md devices for levels that support data redundancy (1,4,5,6) | |
366 | also have | |
367 | ||
368 | sync_action | |
369 | a text file that can be used to monitor and control the rebuild | |
370 | process. It contains one word which can be one of: | |
371 | resync - redundancy is being recalculated after unclean | |
372 | shutdown or creation | |
373 | recover - a hot spare is being built to replace a | |
374 | failed/missing device | |
375 | idle - nothing is happening | |
376 | check - A full check of redundancy was requested and is | |
377 | happening. This reads all block and checks | |
378 | them. A repair may also happen for some raid | |
379 | levels. | |
380 | repair - A full check and repair is happening. This is | |
381 | similar to 'resync', but was requested by the | |
382 | user, and the write-intent bitmap is NOT used to | |
383 | optimise the process. | |
384 | ||
385 | This file is writable, and each of the strings that could be | |
386 | read are meaningful for writing. | |
387 | ||
388 | 'idle' will stop an active resync/recovery etc. There is no | |
389 | guarantee that another resync/recovery may not be automatically | |
390 | started again, though some event will be needed to trigger | |
391 | this. | |
392 | 'resync' or 'recovery' can be used to restart the | |
393 | corresponding operation if it was stopped with 'idle'. | |
394 | 'check' and 'repair' will start the appropriate process | |
395 | providing the current state is 'idle'. | |
396 | ||
397 | mismatch_count | |
398 | When performing 'check' and 'repair', and possibly when | |
399 | performing 'resync', md will count the number of errors that are | |
400 | found. The count in 'mismatch_cnt' is the number of sectors | |
401 | that were re-written, or (for 'check') would have been | |
402 | re-written. As most raid levels work in units of pages rather | |
403 | than sectors, this my be larger than the number of actual errors | |
404 | by a factor of the number of sectors in a page. | |
405 | ||
9b1d1dac PC |
406 | bitmap_set_bits |
407 | If the array has a write-intent bitmap, then writing to this | |
408 | attribute can set bits in the bitmap, indicating that a resync | |
409 | would need to check the corresponding blocks. Either individual | |
410 | numbers or start-end pairs can be written. Multiple numbers | |
411 | can be separated by a space. | |
412 | Note that the numbers are 'bit' numbers, not 'block' numbers. | |
413 | They should be scaled by the bitmap_chunksize. | |
414 | ||
bb636547 N |
415 | Each active md device may also have attributes specific to the |
416 | personality module that manages it. | |
417 | These are specific to the implementation of the module and could | |
418 | change substantially if the implementation changes. | |
419 | ||
420 | These currently include | |
421 | ||
422 | stripe_cache_size (currently raid5 only) | |
423 | number of entries in the stripe cache. This is writable, but | |
424 | there are upper and lower limits (32768, 16). Default is 128. | |
425 | strip_cache_active (currently raid5 only) | |
426 | number of active entries in the stripe cache |