]>
Commit | Line | Data |
---|---|---|
da9bb1d2 AC |
1 | |
2 | ||
3 | EDAC - Error Detection And Correction | |
4 | ||
87f24c3a | 5 | Written by Doug Thompson <dougthompson@xmission.com> |
da9bb1d2 | 6 | 7 Dec 2005 |
87f24c3a | 7 | 17 Jul 2007 Updated |
da9bb1d2 | 8 | |
31983a04 MCC |
9 | (c) Mauro Carvalho Chehab <mchehab@redhat.com> |
10 | 05 Aug 2009 Nehalem interface | |
da9bb1d2 | 11 | |
87f24c3a | 12 | EDAC is maintained and written by: |
da9bb1d2 | 13 | |
87f24c3a DT |
14 | Doug Thompson, Dave Jiang, Dave Peterson et al, |
15 | original author: Thayne Harbaugh, | |
16 | ||
17 | Contact: | |
18 | website: bluesmoke.sourceforge.net | |
19 | mailing list: bluesmoke-devel@lists.sourceforge.net | |
20 | ||
21 | "bluesmoke" was the name for this device driver when it was "out-of-tree" | |
22 | and maintained at sourceforge.net. When it was pushed into 2.6.16 for the | |
23 | first time, it was renamed to 'EDAC'. | |
24 | ||
25 | The bluesmoke project at sourceforge.net is now utilized as a 'staging area' | |
26 | for EDAC development, before it is sent upstream to kernel.org | |
27 | ||
19f59460 ML |
28 | At the bluesmoke/EDAC project site is a series of quilt patches against |
29 | recent kernels, stored in a SVN repository. For easier downloading, there | |
87f24c3a | 30 | is also a tarball snapshot available. |
da9bb1d2 AC |
31 | |
32 | ============================================================================ | |
33 | EDAC PURPOSE | |
34 | ||
35 | The 'edac' kernel module goal is to detect and report errors that occur | |
87f24c3a DT |
36 | within the computer system running under linux. |
37 | ||
38 | MEMORY | |
39 | ||
40 | In the initial release, memory Correctable Errors (CE) and Uncorrectable | |
41 | Errors (UE) are the primary errors being harvested. These types of errors | |
42 | are harvested by the 'edac_mc' class of device. | |
da9bb1d2 AC |
43 | |
44 | Detecting CE events, then harvesting those events and reporting them, | |
45 | CAN be a predictor of future UE events. With CE events, the system can | |
f3479816 | 46 | continue to operate, but with less safety. Preventive maintenance and |
da9bb1d2 AC |
47 | proactive part replacement of memory DIMMs exhibiting CEs can reduce |
48 | the likelihood of the dreaded UE events and system 'panics'. | |
49 | ||
87f24c3a DT |
50 | NON-MEMORY |
51 | ||
52 | A new feature for EDAC, the edac_device class of device, was added in | |
53 | the 2.6.23 version of the kernel. | |
54 | ||
55 | This new device type allows for non-memory type of ECC hardware detectors | |
56 | to have their states harvested and presented to userspace via the sysfs | |
57 | interface. | |
58 | ||
59 | Some architectures have ECC detectors for L1, L2 and L3 caches, along with DMA | |
60 | engines, fabric switches, main data path switches, interconnections, | |
61 | and various other hardware data paths. If the hardware reports it, then | |
62 | a edac_device device probably can be constructed to harvest and present | |
63 | that to userspace. | |
64 | ||
65 | ||
66 | PCI BUS SCANNING | |
da9bb1d2 AC |
67 | |
68 | In addition, PCI Bus Parity and SERR Errors are scanned for on PCI devices | |
69 | in order to determine if errors are occurring on data transfers. | |
87f24c3a | 70 | |
da9bb1d2 | 71 | The presence of PCI Parity errors must be examined with a grain of salt. |
f3479816 | 72 | There are several add-in adapters that do NOT follow the PCI specification |
da9bb1d2 AC |
73 | with regards to Parity generation and reporting. The specification says |
74 | the vendor should tie the parity status bits to 0 if they do not intend | |
75 | to generate parity. Some vendors do not do this, and thus the parity bit | |
76 | can "float" giving false positives. | |
77 | ||
19f59460 | 78 | In the kernel there is a PCI device attribute located in sysfs that is |
87f24c3a | 79 | checked by the EDAC PCI scanning code. If that attribute is set, |
19f59460 | 80 | PCI parity/error scanning is skipped for that device. The attribute |
87f24c3a DT |
81 | is: |
82 | ||
83 | broken_parity_status | |
84 | ||
e281d75e | 85 | as is located in /sys/devices/pci<XXX>/0000:XX:YY.Z directories for |
87f24c3a DT |
86 | PCI devices. |
87 | ||
88 | FUTURE HARDWARE SCANNING | |
da9bb1d2 | 89 | |
49c0dab7 DT |
90 | EDAC will have future error detectors that will be integrated with |
91 | EDAC or added to it, in the following list: | |
da9bb1d2 AC |
92 | |
93 | MCE Machine Check Exception | |
94 | MCA Machine Check Architecture | |
95 | NMI NMI notification of ECC errors | |
96 | MSRs Machine Specific Register error cases | |
97 | and other mechanisms. | |
98 | ||
99 | These errors are usually bus errors, ECC errors, thermal throttling | |
100 | and the like. | |
101 | ||
102 | ||
103 | ============================================================================ | |
104 | EDAC VERSIONING | |
105 | ||
87f24c3a | 106 | EDAC is composed of a "core" module (edac_core.ko) and several Memory |
da9bb1d2 AC |
107 | Controller (MC) driver modules. On a given system, the CORE |
108 | is loaded and one MC driver will be loaded. Both the CORE and | |
87f24c3a DT |
109 | the MC driver (or edac_device driver) have individual versions that reflect |
110 | current release level of their respective modules. | |
111 | ||
112 | Thus, to "report" on what version a system is running, one must report both | |
113 | the CORE's and the MC driver's versions. | |
da9bb1d2 AC |
114 | |
115 | ||
116 | LOADING | |
117 | ||
118 | If 'edac' was statically linked with the kernel then no loading is | |
119 | necessary. If 'edac' was built as modules then simply modprobe the | |
120 | 'edac' pieces that you need. You should be able to modprobe | |
121 | hardware-specific modules and have the dependencies load the necessary core | |
122 | modules. | |
123 | ||
124 | Example: | |
125 | ||
126 | $> modprobe amd76x_edac | |
127 | ||
128 | loads both the amd76x_edac.ko memory controller module and the edac_mc.ko | |
129 | core module. | |
130 | ||
131 | ||
132 | ============================================================================ | |
133 | EDAC sysfs INTERFACE | |
134 | ||
135 | EDAC presents a 'sysfs' interface for control, reporting and attribute | |
136 | reporting purposes. | |
137 | ||
87f24c3a DT |
138 | EDAC lives in the /sys/devices/system/edac directory. |
139 | ||
140 | Within this directory there currently reside 2 'edac' components: | |
da9bb1d2 AC |
141 | |
142 | mc memory controller(s) system | |
49c0dab7 | 143 | pci PCI control and status system |
da9bb1d2 AC |
144 | |
145 | ||
146 | ============================================================================ | |
147 | Memory Controller (mc) Model | |
148 | ||
149 | First a background on the memory controller's model abstracted in EDAC. | |
49c0dab7 | 150 | Each 'mc' device controls a set of DIMM memory modules. These modules are |
f3479816 | 151 | laid out in a Chip-Select Row (csrowX) and Channel table (chX). There can |
49c0dab7 | 152 | be multiple csrows and multiple channels. |
da9bb1d2 AC |
153 | |
154 | Memory controllers allow for several csrows, with 8 csrows being a typical value. | |
155 | Yet, the actual number of csrows depends on the electrical "loading" | |
156 | of a given motherboard, memory controller and DIMM characteristics. | |
157 | ||
158 | Dual channels allows for 128 bit data transfers to the CPU from memory. | |
49c0dab7 DT |
159 | Some newer chipsets allow for more than 2 channels, like Fully Buffered DIMMs |
160 | (FB-DIMMs). The following example will assume 2 channels: | |
da9bb1d2 AC |
161 | |
162 | ||
163 | Channel 0 Channel 1 | |
164 | =================================== | |
165 | csrow0 | DIMM_A0 | DIMM_B0 | | |
166 | csrow1 | DIMM_A0 | DIMM_B0 | | |
167 | =================================== | |
168 | ||
169 | =================================== | |
170 | csrow2 | DIMM_A1 | DIMM_B1 | | |
171 | csrow3 | DIMM_A1 | DIMM_B1 | | |
172 | =================================== | |
173 | ||
174 | In the above example table there are 4 physical slots on the motherboard | |
175 | for memory DIMMs: | |
176 | ||
177 | DIMM_A0 | |
178 | DIMM_B0 | |
179 | DIMM_A1 | |
180 | DIMM_B1 | |
181 | ||
182 | Labels for these slots are usually silk screened on the motherboard. Slots | |
f3479816 | 183 | labeled 'A' are channel 0 in this example. Slots labeled 'B' |
da9bb1d2 AC |
184 | are channel 1. Notice that there are two csrows possible on a |
185 | physical DIMM. These csrows are allocated their csrow assignment | |
186 | based on the slot into which the memory DIMM is placed. Thus, when 1 DIMM | |
187 | is placed in each Channel, the csrows cross both DIMMs. | |
188 | ||
189 | Memory DIMMs come single or dual "ranked". A rank is a populated csrow. | |
190 | Thus, 2 single ranked DIMMs, placed in slots DIMM_A0 and DIMM_B0 above | |
191 | will have 1 csrow, csrow0. csrow1 will be empty. On the other hand, | |
f3479816 | 192 | when 2 dual ranked DIMMs are similarly placed, then both csrow0 and |
da9bb1d2 AC |
193 | csrow1 will be populated. The pattern repeats itself for csrow2 and |
194 | csrow3. | |
195 | ||
196 | The representation of the above is reflected in the directory tree | |
197 | in EDAC's sysfs interface. Starting in directory | |
198 | /sys/devices/system/edac/mc each memory controller will be represented | |
5989f11b | 199 | by its own 'mcX' directory, where 'X' is the index of the MC. |
da9bb1d2 AC |
200 | |
201 | ||
202 | ..../edac/mc/ | |
203 | | | |
204 | |->mc0 | |
205 | |->mc1 | |
206 | |->mc2 | |
207 | .... | |
208 | ||
209 | Under each 'mcX' directory each 'csrowX' is again represented by a | |
5989f11b | 210 | 'csrowX', where 'X' is the csrow index: |
da9bb1d2 AC |
211 | |
212 | ||
213 | .../mc/mc0/ | |
214 | | | |
215 | |->csrow0 | |
216 | |->csrow2 | |
217 | |->csrow3 | |
218 | .... | |
219 | ||
220 | Notice that there is no csrow1, which indicates that csrow0 is | |
221 | composed of a single ranked DIMMs. This should also apply in both | |
222 | Channels, in order to have dual-channel mode be operational. Since | |
223 | both csrow2 and csrow3 are populated, this indicates a dual ranked | |
224 | set of DIMMs for channels 0 and 1. | |
225 | ||
226 | ||
327dafb1 | 227 | Within each of the 'mcX' and 'csrowX' directories are several |
da9bb1d2 AC |
228 | EDAC control and attribute files. |
229 | ||
da9bb1d2 AC |
230 | ============================================================================ |
231 | 'mcX' DIRECTORIES | |
232 | ||
233 | ||
234 | In 'mcX' directories are EDAC control and attribute files for | |
5989f11b | 235 | this 'X' instance of the memory controllers: |
da9bb1d2 AC |
236 | |
237 | ||
238 | Counter reset control file: | |
239 | ||
240 | 'reset_counters' | |
241 | ||
242 | This write-only control file will zero all the statistical counters | |
243 | for UE and CE errors. Zeroing the counters will also reset the timer | |
244 | indicating how long since the last counter zero. This is useful | |
245 | for computing errors/time. Since the counters are always reset at | |
246 | driver initialization time, no module/kernel parameter is available. | |
247 | ||
248 | RUN TIME: echo "anything" >/sys/devices/system/edac/mc/mc0/counter_reset | |
249 | ||
250 | This resets the counters on memory controller 0 | |
251 | ||
252 | ||
253 | Seconds since last counter reset control file: | |
254 | ||
255 | 'seconds_since_reset' | |
256 | ||
257 | This attribute file displays how many seconds have elapsed since the | |
258 | last counter reset. This can be used with the error counters to | |
259 | measure error rates. | |
260 | ||
261 | ||
262 | ||
da9bb1d2 AC |
263 | Memory Controller name attribute file: |
264 | ||
265 | 'mc_name' | |
266 | ||
267 | This attribute file displays the type of memory controller | |
268 | that is being utilized. | |
269 | ||
270 | ||
da9bb1d2 AC |
271 | Total memory managed by this memory controller attribute file: |
272 | ||
273 | 'size_mb' | |
274 | ||
275 | This attribute file displays, in count of megabytes, of memory | |
276 | that this instance of memory controller manages. | |
277 | ||
278 | ||
279 | Total Uncorrectable Errors count attribute file: | |
280 | ||
281 | 'ue_count' | |
282 | ||
283 | This attribute file displays the total count of uncorrectable | |
284 | errors that have occurred on this memory controller. If panic_on_ue | |
285 | is set this counter will not have a chance to increment, | |
286 | since EDAC will panic the system. | |
287 | ||
288 | ||
289 | Total UE count that had no information attribute fileY: | |
290 | ||
291 | 'ue_noinfo_count' | |
292 | ||
e281d75e DB |
293 | This attribute file displays the number of UEs that have occurred |
294 | with no information as to which DIMM slot is having errors. | |
da9bb1d2 AC |
295 | |
296 | ||
297 | Total Correctable Errors count attribute file: | |
298 | ||
299 | 'ce_count' | |
300 | ||
301 | This attribute file displays the total count of correctable | |
302 | errors that have occurred on this memory controller. This | |
303 | count is very important to examine. CEs provide early | |
304 | indications that a DIMM is beginning to fail. This count | |
305 | field should be monitored for non-zero values and report | |
306 | such information to the system administrator. | |
307 | ||
308 | ||
309 | Total Correctable Errors count attribute file: | |
310 | ||
311 | 'ce_noinfo_count' | |
312 | ||
313 | This attribute file displays the number of CEs that | |
f65e51d7 | 314 | have occurred wherewith no information as to which DIMM slot |
da9bb1d2 AC |
315 | is having errors. Memory is handicapped, but operational, |
316 | yet no information is available to indicate which slot | |
317 | the failing memory is in. This count field should be also | |
318 | be monitored for non-zero values. | |
319 | ||
320 | Device Symlink: | |
321 | ||
322 | 'device' | |
323 | ||
4f423ddf FJ |
324 | Symlink to the memory controller device. |
325 | ||
326 | Sdram memory scrubbing rate: | |
327 | ||
328 | 'sdram_scrub_rate' | |
329 | ||
330 | Read/Write attribute file that controls memory scrubbing. The scrubbing | |
d9195881 | 331 | rate is set by writing a minimum bandwidth in bytes/sec to the attribute |
4f423ddf FJ |
332 | file. The rate will be translated to an internal value that gives at |
333 | least the specified rate. | |
334 | ||
335 | Reading the file will return the actual scrubbing rate employed. | |
336 | ||
5e8e19bf BP |
337 | If configuration fails or memory scrubbing is not implemented, accessing |
338 | that attribute will fail. | |
da9bb1d2 AC |
339 | |
340 | ||
341 | ||
342 | ============================================================================ | |
343 | 'csrowX' DIRECTORIES | |
344 | ||
345 | In the 'csrowX' directories are EDAC control and attribute files for | |
5989f11b | 346 | this 'X' instance of csrow: |
da9bb1d2 AC |
347 | |
348 | ||
349 | Total Uncorrectable Errors count attribute file: | |
350 | ||
351 | 'ue_count' | |
352 | ||
353 | This attribute file displays the total count of uncorrectable | |
354 | errors that have occurred on this csrow. If panic_on_ue is set | |
355 | this counter will not have a chance to increment, since EDAC | |
356 | will panic the system. | |
357 | ||
358 | ||
359 | Total Correctable Errors count attribute file: | |
360 | ||
361 | 'ce_count' | |
362 | ||
363 | This attribute file displays the total count of correctable | |
364 | errors that have occurred on this csrow. This | |
365 | count is very important to examine. CEs provide early | |
366 | indications that a DIMM is beginning to fail. This count | |
367 | field should be monitored for non-zero values and report | |
368 | such information to the system administrator. | |
369 | ||
370 | ||
371 | Total memory managed by this csrow attribute file: | |
372 | ||
373 | 'size_mb' | |
374 | ||
375 | This attribute file displays, in count of megabytes, of memory | |
f3479816 | 376 | that this csrow contains. |
da9bb1d2 AC |
377 | |
378 | ||
379 | Memory Type attribute file: | |
380 | ||
381 | 'mem_type' | |
382 | ||
383 | This attribute file will display what type of memory is currently | |
384 | on this csrow. Normally, either buffered or unbuffered memory. | |
49c0dab7 DT |
385 | Examples: |
386 | Registered-DDR | |
387 | Unbuffered-DDR | |
da9bb1d2 AC |
388 | |
389 | ||
390 | EDAC Mode of operation attribute file: | |
391 | ||
392 | 'edac_mode' | |
393 | ||
394 | This attribute file will display what type of Error detection | |
395 | and correction is being utilized. | |
396 | ||
397 | ||
398 | Device type attribute file: | |
399 | ||
400 | 'dev_type' | |
401 | ||
49c0dab7 DT |
402 | This attribute file will display what type of DRAM device is |
403 | being utilized on this DIMM. | |
404 | Examples: | |
405 | x1 | |
406 | x2 | |
407 | x4 | |
408 | x8 | |
da9bb1d2 AC |
409 | |
410 | ||
411 | Channel 0 CE Count attribute file: | |
412 | ||
413 | 'ch0_ce_count' | |
414 | ||
415 | This attribute file will display the count of CEs on this | |
416 | DIMM located in channel 0. | |
417 | ||
418 | ||
419 | Channel 0 UE Count attribute file: | |
420 | ||
421 | 'ch0_ue_count' | |
422 | ||
423 | This attribute file will display the count of UEs on this | |
424 | DIMM located in channel 0. | |
425 | ||
426 | ||
427 | Channel 0 DIMM Label control file: | |
428 | ||
429 | 'ch0_dimm_label' | |
430 | ||
431 | This control file allows this DIMM to have a label assigned | |
432 | to it. With this label in the module, when errors occur | |
433 | the output can provide the DIMM label in the system log. | |
434 | This becomes vital for panic events to isolate the | |
435 | cause of the UE event. | |
436 | ||
437 | DIMM Labels must be assigned after booting, with information | |
438 | that correctly identifies the physical slot with its | |
439 | silk screen label. This information is currently very | |
440 | motherboard specific and determination of this information | |
441 | must occur in userland at this time. | |
442 | ||
443 | ||
444 | Channel 1 CE Count attribute file: | |
445 | ||
446 | 'ch1_ce_count' | |
447 | ||
448 | This attribute file will display the count of CEs on this | |
449 | DIMM located in channel 1. | |
450 | ||
451 | ||
452 | Channel 1 UE Count attribute file: | |
453 | ||
454 | 'ch1_ue_count' | |
455 | ||
456 | This attribute file will display the count of UEs on this | |
457 | DIMM located in channel 0. | |
458 | ||
459 | ||
460 | Channel 1 DIMM Label control file: | |
461 | ||
462 | 'ch1_dimm_label' | |
463 | ||
464 | This control file allows this DIMM to have a label assigned | |
465 | to it. With this label in the module, when errors occur | |
466 | the output can provide the DIMM label in the system log. | |
467 | This becomes vital for panic events to isolate the | |
468 | cause of the UE event. | |
469 | ||
470 | DIMM Labels must be assigned after booting, with information | |
471 | that correctly identifies the physical slot with its | |
472 | silk screen label. This information is currently very | |
473 | motherboard specific and determination of this information | |
474 | must occur in userland at this time. | |
475 | ||
da9bb1d2 AC |
476 | ============================================================================ |
477 | SYSTEM LOGGING | |
478 | ||
479 | If logging for UEs and CEs are enabled then system logs will have | |
480 | error notices indicating errors that have been detected: | |
481 | ||
49c0dab7 | 482 | EDAC MC0: CE page 0x283, offset 0xce0, grain 8, syndrome 0x6ec3, row 0, |
da9bb1d2 AC |
483 | channel 1 "DIMM_B1": amd76x_edac |
484 | ||
49c0dab7 | 485 | EDAC MC0: CE page 0x1e5, offset 0xfb0, grain 8, syndrome 0xb741, row 0, |
da9bb1d2 AC |
486 | channel 1 "DIMM_B1": amd76x_edac |
487 | ||
488 | ||
489 | The structure of the message is: | |
490 | the memory controller (MC0) | |
491 | Error type (CE) | |
492 | memory page (0x283) | |
493 | offset in the page (0xce0) | |
494 | the byte granularity (grain 8) | |
495 | or resolution of the error | |
496 | the error syndrome (0xb741) | |
497 | memory row (row 0) | |
498 | memory channel (channel 1) | |
499 | DIMM label, if set prior (DIMM B1 | |
500 | and then an optional, driver-specific message that may | |
501 | have additional information. | |
502 | ||
503 | Both UEs and CEs with no info will lack all but memory controller, | |
504 | error type, a notice of "no info" and then an optional, | |
505 | driver-specific error message. | |
506 | ||
507 | ||
da9bb1d2 AC |
508 | ============================================================================ |
509 | PCI Bus Parity Detection | |
510 | ||
511 | ||
512 | On Header Type 00 devices the primary status is looked at | |
513 | for any parity error regardless of whether Parity is enabled on the | |
514 | device. (The spec indicates parity is generated in some cases). | |
515 | On Header Type 01 bridges, the secondary status register is also | |
f3479816 | 516 | looked at to see if parity occurred on the bus on the other side of |
da9bb1d2 AC |
517 | the bridge. |
518 | ||
519 | ||
520 | SYSFS CONFIGURATION | |
521 | ||
522 | Under /sys/devices/system/edac/pci are control and attribute files as follows: | |
523 | ||
524 | ||
525 | Enable/Disable PCI Parity checking control file: | |
526 | ||
527 | 'check_pci_parity' | |
528 | ||
529 | ||
530 | This control file enables or disables the PCI Bus Parity scanning | |
531 | operation. Writing a 1 to this file enables the scanning. Writing | |
532 | a 0 to this file disables the scanning. | |
533 | ||
534 | Enable: | |
535 | echo "1" >/sys/devices/system/edac/pci/check_pci_parity | |
536 | ||
537 | Disable: | |
538 | echo "0" >/sys/devices/system/edac/pci/check_pci_parity | |
539 | ||
540 | ||
327dafb1 AJ |
541 | Parity Count: |
542 | ||
543 | 'pci_parity_count' | |
544 | ||
545 | This attribute file will display the number of parity errors that | |
546 | have been detected. | |
547 | ||
548 | ||
549 | ============================================================================ | |
550 | MODULE PARAMETERS | |
551 | ||
552 | Panic on UE control file: | |
553 | ||
554 | 'edac_mc_panic_on_ue' | |
555 | ||
556 | An uncorrectable error will cause a machine panic. This is usually | |
557 | desirable. It is a bad idea to continue when an uncorrectable error | |
558 | occurs - it is indeterminate what was uncorrected and the operating | |
559 | system context might be so mangled that continuing will lead to further | |
560 | corruption. If the kernel has MCE configured, then EDAC will never | |
561 | notice the UE. | |
562 | ||
563 | LOAD TIME: module/kernel parameter: edac_mc_panic_on_ue=[0|1] | |
564 | ||
565 | RUN TIME: echo "1" > /sys/module/edac_core/parameters/edac_mc_panic_on_ue | |
566 | ||
567 | ||
568 | Log UE control file: | |
569 | ||
570 | 'edac_mc_log_ue' | |
571 | ||
572 | Generate kernel messages describing uncorrectable errors. These errors | |
573 | are reported through the system message log system. UE statistics | |
574 | will be accumulated even when UE logging is disabled. | |
575 | ||
576 | LOAD TIME: module/kernel parameter: edac_mc_log_ue=[0|1] | |
577 | ||
578 | RUN TIME: echo "1" > /sys/module/edac_core/parameters/edac_mc_log_ue | |
579 | ||
580 | ||
581 | Log CE control file: | |
582 | ||
583 | 'edac_mc_log_ce' | |
584 | ||
585 | Generate kernel messages describing correctable errors. These | |
586 | errors are reported through the system message log system. | |
587 | CE statistics will be accumulated even when CE logging is disabled. | |
588 | ||
589 | LOAD TIME: module/kernel parameter: edac_mc_log_ce=[0|1] | |
590 | ||
591 | RUN TIME: echo "1" > /sys/module/edac_core/parameters/edac_mc_log_ce | |
592 | ||
593 | ||
594 | Polling period control file: | |
595 | ||
596 | 'edac_mc_poll_msec' | |
597 | ||
598 | The time period, in milliseconds, for polling for error information. | |
599 | Too small a value wastes resources. Too large a value might delay | |
600 | necessary handling of errors and might loose valuable information for | |
601 | locating the error. 1000 milliseconds (once each second) is the current | |
602 | default. Systems which require all the bandwidth they can get, may | |
603 | increase this. | |
604 | ||
605 | LOAD TIME: module/kernel parameter: edac_mc_poll_msec=[0|1] | |
606 | ||
607 | RUN TIME: echo "1000" > /sys/module/edac_core/parameters/edac_mc_poll_msec | |
608 | ||
da9bb1d2 AC |
609 | |
610 | Panic on PCI PARITY Error: | |
611 | ||
612 | 'panic_on_pci_parity' | |
613 | ||
614 | ||
f3479816 | 615 | This control files enables or disables panicking when a parity |
da9bb1d2 AC |
616 | error has been detected. |
617 | ||
618 | ||
327dafb1 | 619 | module/kernel parameter: edac_panic_on_pci_pe=[0|1] |
da9bb1d2 AC |
620 | |
621 | Enable: | |
327dafb1 | 622 | echo "1" > /sys/module/edac_core/parameters/edac_panic_on_pci_pe |
da9bb1d2 AC |
623 | |
624 | Disable: | |
327dafb1 | 625 | echo "0" > /sys/module/edac_core/parameters/edac_panic_on_pci_pe |
da9bb1d2 AC |
626 | |
627 | ||
628 | ||
da9bb1d2 | 629 | ======================================================================= |
87f24c3a DT |
630 | |
631 | ||
632 | EDAC_DEVICE type of device | |
633 | ||
634 | In the header file, edac_core.h, there is a series of edac_device structures | |
635 | and APIs for the EDAC_DEVICE. | |
636 | ||
637 | User space access to an edac_device is through the sysfs interface. | |
638 | ||
639 | At the location /sys/devices/system/edac (sysfs) new edac_device devices will | |
640 | appear. | |
641 | ||
642 | There is a three level tree beneath the above 'edac' directory. For example, | |
643 | the 'test_device_edac' device (found at the bluesmoke.sourceforget.net website) | |
644 | installs itself as: | |
645 | ||
646 | /sys/devices/systm/edac/test-instance | |
647 | ||
648 | in this directory are various controls, a symlink and one or more 'instance' | |
649 | directorys. | |
650 | ||
651 | The standard default controls are: | |
652 | ||
653 | log_ce boolean to log CE events | |
654 | log_ue boolean to log UE events | |
655 | panic_on_ue boolean to 'panic' the system if an UE is encountered | |
656 | (default off, can be set true via startup script) | |
657 | poll_msec time period between POLL cycles for events | |
658 | ||
659 | The test_device_edac device adds at least one of its own custom control: | |
660 | ||
661 | test_bits which in the current test driver does nothing but | |
662 | show how it is installed. A ported driver can | |
663 | add one or more such controls and/or attributes | |
664 | for specific uses. | |
665 | One out-of-tree driver uses controls here to allow | |
666 | for ERROR INJECTION operations to hardware | |
667 | injection registers | |
668 | ||
669 | The symlink points to the 'struct dev' that is registered for this edac_device. | |
670 | ||
671 | INSTANCES | |
672 | ||
673 | One or more instance directories are present. For the 'test_device_edac' case: | |
674 | ||
675 | test-instance0 | |
676 | ||
677 | ||
678 | In this directory there are two default counter attributes, which are totals of | |
679 | counter in deeper subdirectories. | |
680 | ||
681 | ce_count total of CE events of subdirectories | |
682 | ue_count total of UE events of subdirectories | |
683 | ||
684 | BLOCKS | |
685 | ||
686 | At the lowest directory level is the 'block' directory. There can be 0, 1 | |
687 | or more blocks specified in each instance. | |
688 | ||
689 | test-block0 | |
690 | ||
691 | ||
692 | In this directory the default attributes are: | |
693 | ||
694 | ce_count which is counter of CE events for this 'block' | |
695 | of hardware being monitored | |
696 | ue_count which is counter of UE events for this 'block' | |
697 | of hardware being monitored | |
698 | ||
699 | ||
700 | The 'test_device_edac' device adds 4 attributes and 1 control: | |
701 | ||
702 | test-block-bits-0 for every POLL cycle this counter | |
703 | is incremented | |
704 | test-block-bits-1 every 10 cycles, this counter is bumped once, | |
705 | and test-block-bits-0 is set to 0 | |
706 | test-block-bits-2 every 100 cycles, this counter is bumped once, | |
707 | and test-block-bits-1 is set to 0 | |
708 | test-block-bits-3 every 1000 cycles, this counter is bumped once, | |
709 | and test-block-bits-2 is set to 0 | |
710 | ||
711 | ||
712 | reset-counters writing ANY thing to this control will | |
713 | reset all the above counters. | |
714 | ||
715 | ||
716 | Use of the 'test_device_edac' driver should any others to create their own | |
717 | unique drivers for their hardware systems. | |
718 | ||
719 | The 'test_device_edac' sample driver is located at the | |
720 | bluesmoke.sourceforge.net project site for EDAC. | |
721 | ||
31983a04 MCC |
722 | ======================================================================= |
723 | NEHALEM USAGE OF EDAC APIs | |
724 | ||
725 | This chapter documents some EXPERIMENTAL mappings for EDAC API to handle | |
726 | Nehalem EDAC driver. They will likely be changed on future versions | |
727 | of the driver. | |
728 | ||
729 | Due to the way Nehalem exports Memory Controller data, some adjustments | |
730 | were done at i7core_edac driver. This chapter will cover those differences | |
731 | ||
732 | 1) On Nehalem, there are one Memory Controller per Quick Patch Interconnect | |
c3444363 MCC |
733 | (QPI). At the driver, the term "socket" means one QPI. This is |
734 | associated with a physical CPU socket. | |
31983a04 MCC |
735 | |
736 | Each MC have 3 physical read channels, 3 physical write channels and | |
737 | 3 logic channels. The driver currenty sees it as just 3 channels. | |
738 | Each channel can have up to 3 DIMMs. | |
739 | ||
740 | The minimum known unity is DIMMs. There are no information about csrows. | |
c3444363 MCC |
741 | As EDAC API maps the minimum unity is csrows, the driver sequencially |
742 | maps channel/dimm into different csrows. | |
743 | ||
25985edc | 744 | For example, supposing the following layout: |
c3444363 MCC |
745 | Ch0 phy rd0, wr0 (0x063f4031): 2 ranks, UDIMMs |
746 | dimm 0 1024 Mb offset: 0, bank: 8, rank: 1, row: 0x4000, col: 0x400 | |
747 | dimm 1 1024 Mb offset: 4, bank: 8, rank: 1, row: 0x4000, col: 0x400 | |
748 | Ch1 phy rd1, wr1 (0x063f4031): 2 ranks, UDIMMs | |
749 | dimm 0 1024 Mb offset: 0, bank: 8, rank: 1, row: 0x4000, col: 0x400 | |
750 | Ch2 phy rd3, wr3 (0x063f4031): 2 ranks, UDIMMs | |
751 | dimm 0 1024 Mb offset: 0, bank: 8, rank: 1, row: 0x4000, col: 0x400 | |
752 | The driver will map it as: | |
753 | csrow0: channel 0, dimm0 | |
754 | csrow1: channel 0, dimm1 | |
755 | csrow2: channel 1, dimm0 | |
756 | csrow3: channel 2, dimm0 | |
757 | ||
758 | exports one | |
31983a04 MCC |
759 | DIMM per csrow. |
760 | ||
c3444363 | 761 | Each QPI is exported as a different memory controller. |
31983a04 MCC |
762 | |
763 | 2) Nehalem MC has the hability to generate errors. The driver implements this | |
764 | functionality via some error injection nodes: | |
765 | ||
766 | For injecting a memory error, there are some sysfs nodes, under | |
c3444363 | 767 | /sys/devices/system/edac/mc/mc?/: |
31983a04 | 768 | |
35be9544 | 769 | inject_addrmatch/*: |
31983a04 MCC |
770 | Controls the error injection mask register. It is possible to specify |
771 | several characteristics of the address to match an error code: | |
772 | dimm = the affected dimm. Numbers are relative to a channel; | |
773 | rank = the memory rank; | |
774 | channel = the channel that will generate an error; | |
775 | bank = the affected bank; | |
776 | page = the page address; | |
777 | column (or col) = the address column. | |
778 | each of the above values can be set to "any" to match any valid value. | |
779 | ||
780 | At driver init, all values are set to any. | |
781 | ||
782 | For example, to generate an error at rank 1 of dimm 2, for any channel, | |
783 | any bank, any page, any column: | |
35be9544 MCC |
784 | echo 2 >/sys/devices/system/edac/mc/mc0/inject_addrmatch/dimm |
785 | echo 1 >/sys/devices/system/edac/mc/mc0/inject_addrmatch/rank | |
31983a04 MCC |
786 | |
787 | To return to the default behaviour of matching any, you can do: | |
35be9544 MCC |
788 | echo any >/sys/devices/system/edac/mc/mc0/inject_addrmatch/dimm |
789 | echo any >/sys/devices/system/edac/mc/mc0/inject_addrmatch/rank | |
31983a04 MCC |
790 | |
791 | inject_eccmask: | |
792 | specifies what bits will have troubles, | |
793 | ||
794 | inject_section: | |
795 | specifies what ECC cache section will get the error: | |
796 | 3 for both | |
797 | 2 for the highest | |
798 | 1 for the lowest | |
799 | ||
31983a04 MCC |
800 | inject_type: |
801 | specifies the type of error, being a combination of the following bits: | |
802 | bit 0 - repeat | |
803 | bit 1 - ecc | |
804 | bit 2 - parity | |
805 | ||
806 | inject_enable starts the error generation when something different | |
807 | than 0 is written. | |
808 | ||
809 | All inject vars can be read. root permission is needed for write. | |
810 | ||
811 | Datasheet states that the error will only be generated after a write on an | |
812 | address that matches inject_addrmatch. It seems, however, that reading will | |
813 | also produce an error. | |
814 | ||
815 | For example, the following code will generate an error for any write access | |
816 | at socket 0, on any DIMM/address on channel 2: | |
817 | ||
35be9544 | 818 | echo 2 >/sys/devices/system/edac/mc/mc0/inject_addrmatch/channel |
31983a04 MCC |
819 | echo 2 >/sys/devices/system/edac/mc/mc0/inject_type |
820 | echo 64 >/sys/devices/system/edac/mc/mc0/inject_eccmask | |
821 | echo 3 >/sys/devices/system/edac/mc/mc0/inject_section | |
31983a04 MCC |
822 | echo 1 >/sys/devices/system/edac/mc/mc0/inject_enable |
823 | dd if=/dev/mem of=/dev/null seek=16k bs=4k count=1 >& /dev/null | |
824 | ||
c3444363 MCC |
825 | For socket 1, it is needed to replace "mc0" by "mc1" at the above |
826 | commands. | |
827 | ||
31983a04 MCC |
828 | The generated error message will look like: |
829 | ||
830 | EDAC MC0: UE row 0, channel-a= 0 channel-b= 0 labels "-": NON_FATAL (addr = 0x0075b980, socket=0, Dimm=0, Channel=2, syndrome=0x00000040, count=1, Err=8c0000400001009f:4000080482 (read error: read ECC error)) | |
831 | ||
832 | 3) Nehalem specific Corrected Error memory counters | |
833 | ||
35be9544 MCC |
834 | Nehalem have some registers to count memory errors. The driver uses those |
835 | registers to report Corrected Errors on devices with Registered Dimms. | |
31983a04 | 836 | |
35be9544 MCC |
837 | However, those counters don't work with Unregistered Dimms. As the chipset |
838 | offers some counters that also work with UDIMMS (but with a worse level of | |
839 | granularity than the default ones), the driver exposes those registers for | |
840 | UDIMM memories. | |
c3444363 | 841 | |
35be9544 | 842 | They can be read by looking at the contents of all_channel_counts/ |
31983a04 | 843 | |
35be9544 MCC |
844 | $ for i in /sys/devices/system/edac/mc/mc0/all_channel_counts/*; do echo $i; cat $i; done |
845 | /sys/devices/system/edac/mc/mc0/all_channel_counts/udimm0 | |
846 | 0 | |
847 | /sys/devices/system/edac/mc/mc0/all_channel_counts/udimm1 | |
848 | 0 | |
849 | /sys/devices/system/edac/mc/mc0/all_channel_counts/udimm2 | |
850 | 0 | |
c3444363 MCC |
851 | |
852 | What happens here is that errors on different csrows, but at the same | |
853 | dimm number will increment the same counter. | |
854 | So, in this memory mapping: | |
855 | csrow0: channel 0, dimm0 | |
856 | csrow1: channel 0, dimm1 | |
857 | csrow2: channel 1, dimm0 | |
858 | csrow3: channel 2, dimm0 | |
35be9544 MCC |
859 | The hardware will increment udimm0 for an error at the first dimm at either |
860 | csrow0, csrow2 or csrow3; | |
861 | The hardware will increment udimm1 for an error at the second dimm at either | |
862 | csrow0, csrow2 or csrow3; | |
863 | The hardware will increment udimm2 for an error at the third dimm at either | |
864 | csrow0, csrow2 or csrow3; | |
c3444363 MCC |
865 | |
866 | 4) Standard error counters | |
867 | ||
868 | The standard error counters are generated when an mcelog error is received | |
35be9544 MCC |
869 | by the driver. Since, with udimm, this is counted by software, it is |
870 | possible that some errors could be lost. With rdimm's, they displays the | |
871 | contents of the registers |