]>
Commit | Line | Data |
---|---|---|
da9bb1d2 | 1 | EDAC - Error Detection And Correction |
043b4318 | 2 | ===================================== |
87f24c3a | 3 | |
e34217c0 BP |
4 | "bluesmoke" was the name for this device driver when it |
5 | was "out-of-tree" and maintained at sourceforge.net - | |
6 | bluesmoke.sourceforge.net. That site is mostly archaic now and can be | |
7 | used only for historical purposes. | |
8 | ||
9 | When the subsystem was pushed into 2.6.16 for the first time, it was | |
10 | renamed to 'EDAC'. | |
87f24c3a | 11 | |
043b4318 BP |
12 | PURPOSE |
13 | ------- | |
da9bb1d2 | 14 | |
043b4318 BP |
15 | The 'edac' kernel module's goal is to detect and report hardware errors |
16 | that occur within the computer system running under linux. | |
87f24c3a DT |
17 | |
18 | MEMORY | |
043b4318 | 19 | ------ |
87f24c3a | 20 | |
043b4318 BP |
21 | Memory Correctable Errors (CE) and Uncorrectable Errors (UE) are the |
22 | primary errors being harvested. These types of errors are harvested by | |
23 | the 'edac_mc' device. | |
da9bb1d2 AC |
24 | |
25 | Detecting CE events, then harvesting those events and reporting them, | |
043b4318 BP |
26 | *can* but must not necessarily be a predictor of future UE events. With |
27 | CE events only, the system can and will continue to operate as no data | |
28 | has been damaged yet. | |
29 | ||
30 | However, preventive maintenance and proactive part replacement of memory | |
31 | DIMMs exhibiting CEs can reduce the likelihood of the dreaded UE events | |
32 | and system panics. | |
da9bb1d2 | 33 | |
043b4318 BP |
34 | OTHER HARDWARE ELEMENTS |
35 | ----------------------- | |
87f24c3a DT |
36 | |
37 | A new feature for EDAC, the edac_device class of device, was added in | |
38 | the 2.6.23 version of the kernel. | |
39 | ||
40 | This new device type allows for non-memory type of ECC hardware detectors | |
41 | to have their states harvested and presented to userspace via the sysfs | |
42 | interface. | |
43 | ||
043b4318 BP |
44 | Some architectures have ECC detectors for L1, L2 and L3 caches, |
45 | along with DMA engines, fabric switches, main data path switches, | |
46 | interconnections, and various other hardware data paths. If the hardware | |
47 | reports it, then a edac_device device probably can be constructed to | |
48 | harvest and present that to userspace. | |
87f24c3a DT |
49 | |
50 | ||
51 | PCI BUS SCANNING | |
043b4318 | 52 | ---------------- |
da9bb1d2 | 53 | |
043b4318 BP |
54 | In addition, PCI devices are scanned for PCI Bus Parity and SERR Errors |
55 | in order to determine if errors are occurring during data transfers. | |
87f24c3a | 56 | |
da9bb1d2 | 57 | The presence of PCI Parity errors must be examined with a grain of salt. |
043b4318 | 58 | There are several add-in adapters that do *not* follow the PCI specification |
da9bb1d2 AC |
59 | with regards to Parity generation and reporting. The specification says |
60 | the vendor should tie the parity status bits to 0 if they do not intend | |
61 | to generate parity. Some vendors do not do this, and thus the parity bit | |
62 | can "float" giving false positives. | |
63 | ||
043b4318 BP |
64 | There is a PCI device attribute located in sysfs that is checked by |
65 | the EDAC PCI scanning code. If that attribute is set, PCI parity/error | |
66 | scanning is skipped for that device. The attribute is: | |
87f24c3a DT |
67 | |
68 | broken_parity_status | |
69 | ||
043b4318 | 70 | and is located in /sys/devices/pci<XXX>/0000:XX:YY.Z directories for |
87f24c3a DT |
71 | PCI devices. |
72 | ||
da9bb1d2 | 73 | |
043b4318 BP |
74 | VERSIONING |
75 | ---------- | |
da9bb1d2 | 76 | |
87f24c3a | 77 | EDAC is composed of a "core" module (edac_core.ko) and several Memory |
043b4318 BP |
78 | Controller (MC) driver modules. On a given system, the CORE is loaded |
79 | and one MC driver will be loaded. Both the CORE and the MC driver (or | |
80 | edac_device driver) have individual versions that reflect current | |
81 | release level of their respective modules. | |
87f24c3a | 82 | |
043b4318 BP |
83 | Thus, to "report" on what version a system is running, one must report |
84 | both the CORE's and the MC driver's versions. | |
da9bb1d2 AC |
85 | |
86 | ||
87 | LOADING | |
043b4318 | 88 | ------- |
da9bb1d2 | 89 | |
043b4318 BP |
90 | If 'edac' was statically linked with the kernel then no loading |
91 | is necessary. If 'edac' was built as modules then simply modprobe | |
92 | the 'edac' pieces that you need. You should be able to modprobe | |
93 | hardware-specific modules and have the dependencies load the necessary | |
94 | core modules. | |
da9bb1d2 AC |
95 | |
96 | Example: | |
97 | ||
98 | $> modprobe amd76x_edac | |
99 | ||
100 | loads both the amd76x_edac.ko memory controller module and the edac_mc.ko | |
101 | core module. | |
102 | ||
103 | ||
043b4318 BP |
104 | SYSFS INTERFACE |
105 | --------------- | |
da9bb1d2 | 106 | |
043b4318 BP |
107 | EDAC presents a 'sysfs' interface for control and reporting purposes. It |
108 | lives in the /sys/devices/system/edac directory. | |
87f24c3a | 109 | |
043b4318 | 110 | Within this directory there currently reside 2 components: |
da9bb1d2 AC |
111 | |
112 | mc memory controller(s) system | |
49c0dab7 | 113 | pci PCI control and status system |
da9bb1d2 AC |
114 | |
115 | ||
043b4318 | 116 | |
da9bb1d2 | 117 | Memory Controller (mc) Model |
043b4318 | 118 | ---------------------------- |
da9bb1d2 | 119 | |
043b4318 BP |
120 | Each 'mc' device controls a set of DIMM memory modules. These modules |
121 | are laid out in a Chip-Select Row (csrowX) and Channel table (chX). | |
122 | There can be multiple csrows and multiple channels. | |
da9bb1d2 | 123 | |
043b4318 BP |
124 | Memory controllers allow for several csrows, with 8 csrows being a |
125 | typical value. Yet, the actual number of csrows depends on the layout of | |
126 | a given motherboard, memory controller and DIMM characteristics. | |
da9bb1d2 | 127 | |
043b4318 BP |
128 | Dual channels allows for 128 bit data transfers to/from the CPU from/to |
129 | memory. Some newer chipsets allow for more than 2 channels, like Fully | |
130 | Buffered DIMMs (FB-DIMMs). The following example will assume 2 channels: | |
da9bb1d2 AC |
131 | |
132 | ||
133 | Channel 0 Channel 1 | |
134 | =================================== | |
135 | csrow0 | DIMM_A0 | DIMM_B0 | | |
136 | csrow1 | DIMM_A0 | DIMM_B0 | | |
137 | =================================== | |
138 | ||
139 | =================================== | |
140 | csrow2 | DIMM_A1 | DIMM_B1 | | |
141 | csrow3 | DIMM_A1 | DIMM_B1 | | |
142 | =================================== | |
143 | ||
144 | In the above example table there are 4 physical slots on the motherboard | |
145 | for memory DIMMs: | |
146 | ||
147 | DIMM_A0 | |
148 | DIMM_B0 | |
149 | DIMM_A1 | |
150 | DIMM_B1 | |
151 | ||
043b4318 BP |
152 | Labels for these slots are usually silk-screened on the motherboard. |
153 | Slots labeled 'A' are channel 0 in this example. Slots labeled 'B' are | |
154 | channel 1. Notice that there are two csrows possible on a physical DIMM. | |
155 | These csrows are allocated their csrow assignment based on the slot into | |
156 | which the memory DIMM is placed. Thus, when 1 DIMM is placed in each | |
157 | Channel, the csrows cross both DIMMs. | |
da9bb1d2 AC |
158 | |
159 | Memory DIMMs come single or dual "ranked". A rank is a populated csrow. | |
160 | Thus, 2 single ranked DIMMs, placed in slots DIMM_A0 and DIMM_B0 above | |
161 | will have 1 csrow, csrow0. csrow1 will be empty. On the other hand, | |
f3479816 | 162 | when 2 dual ranked DIMMs are similarly placed, then both csrow0 and |
da9bb1d2 AC |
163 | csrow1 will be populated. The pattern repeats itself for csrow2 and |
164 | csrow3. | |
165 | ||
043b4318 BP |
166 | The representation of the above is reflected in the directory |
167 | tree in EDAC's sysfs interface. Starting in directory | |
da9bb1d2 | 168 | /sys/devices/system/edac/mc each memory controller will be represented |
5989f11b | 169 | by its own 'mcX' directory, where 'X' is the index of the MC. |
da9bb1d2 AC |
170 | |
171 | ||
172 | ..../edac/mc/ | |
173 | | | |
174 | |->mc0 | |
175 | |->mc1 | |
176 | |->mc2 | |
177 | .... | |
178 | ||
179 | Under each 'mcX' directory each 'csrowX' is again represented by a | |
5989f11b | 180 | 'csrowX', where 'X' is the csrow index: |
da9bb1d2 AC |
181 | |
182 | ||
183 | .../mc/mc0/ | |
184 | | | |
185 | |->csrow0 | |
186 | |->csrow2 | |
187 | |->csrow3 | |
188 | .... | |
189 | ||
043b4318 BP |
190 | Notice that there is no csrow1, which indicates that csrow0 is composed |
191 | of a single ranked DIMMs. This should also apply in both Channels, in | |
192 | order to have dual-channel mode be operational. Since both csrow2 and | |
193 | csrow3 are populated, this indicates a dual ranked set of DIMMs for | |
194 | channels 0 and 1. | |
da9bb1d2 AC |
195 | |
196 | ||
043b4318 BP |
197 | Within each of the 'mcX' and 'csrowX' directories are several EDAC |
198 | control and attribute files. | |
da9bb1d2 | 199 | |
da9bb1d2 | 200 | |
043b4318 BP |
201 | 'mcX' directories |
202 | ----------------- | |
da9bb1d2 AC |
203 | |
204 | In 'mcX' directories are EDAC control and attribute files for | |
8b6f04ce | 205 | this 'X' instance of the memory controllers. |
da9bb1d2 | 206 | |
8b6f04ce | 207 | For a description of the sysfs API, please see: |
3aae9edd | 208 | Documentation/ABI/testing/sysfs-devices-edac |
da9bb1d2 AC |
209 | |
210 | ||
032d0ab7 MCC |
211 | ``dimmX`` or ``rankX`` directories |
212 | ---------------------------------- | |
213 | ||
214 | The recommended way to use the EDAC subsystem is to look at the information | |
215 | provided by the ``dimmX`` or ``rankX`` directories [#f5]_. | |
216 | ||
217 | A typical EDAC system has the following structure under | |
218 | ``/sys/devices/system/edac/``\ [#f6]_:: | |
219 | ||
220 | /sys/devices/system/edac/ | |
221 | ├── mc | |
222 | │  ├── mc0 | |
223 | │  │  ├── ce_count | |
224 | │  │  ├── ce_noinfo_count | |
225 | │  │  ├── dimm0 | |
226 | │  │  │  ├── dimm_dev_type | |
227 | │  │  │  ├── dimm_edac_mode | |
228 | │  │  │  ├── dimm_label | |
229 | │  │  │  ├── dimm_location | |
230 | │  │  │  ├── dimm_mem_type | |
231 | │  │  │  ├── size | |
232 | │  │  │  └── uevent | |
233 | │  │  ├── max_location | |
234 | │  │  ├── mc_name | |
235 | │  │  ├── reset_counters | |
236 | │  │  ├── seconds_since_reset | |
237 | │  │  ├── size_mb | |
238 | │  │  ├── ue_count | |
239 | │  │  ├── ue_noinfo_count | |
240 | │  │  └── uevent | |
241 | │  ├── mc1 | |
242 | │  │  ├── ce_count | |
243 | │  │  ├── ce_noinfo_count | |
244 | │  │  ├── dimm0 | |
245 | │  │  │  ├── dimm_dev_type | |
246 | │  │  │  ├── dimm_edac_mode | |
247 | │  │  │  ├── dimm_label | |
248 | │  │  │  ├── dimm_location | |
249 | │  │  │  ├── dimm_mem_type | |
250 | │  │  │  ├── size | |
251 | │  │  │  └── uevent | |
252 | │  │  ├── max_location | |
253 | │  │  ├── mc_name | |
254 | │  │  ├── reset_counters | |
255 | │  │  ├── seconds_since_reset | |
256 | │  │  ├── size_mb | |
257 | │  │  ├── ue_count | |
258 | │  │  ├── ue_noinfo_count | |
259 | │  │  └── uevent | |
260 | │  └── uevent | |
261 | └── uevent | |
262 | ||
263 | In the ``dimmX`` directories are EDAC control and attribute files for | |
264 | this ``X`` memory module: | |
265 | ||
266 | - ``size`` - Total memory managed by this csrow attribute file | |
267 | ||
268 | This attribute file displays, in count of megabytes, the memory | |
269 | that this csrow contains. | |
270 | ||
271 | - ``dimm_dev_type`` - Device type attribute file | |
272 | ||
273 | This attribute file will display what type of DRAM device is | |
274 | being utilized on this DIMM. | |
275 | Examples: | |
276 | ||
277 | - x1 | |
278 | - x2 | |
279 | - x4 | |
280 | - x8 | |
281 | ||
282 | - ``dimm_edac_mode`` - EDAC Mode of operation attribute file | |
283 | ||
284 | This attribute file will display what type of Error detection | |
285 | and correction is being utilized. | |
286 | ||
287 | - ``dimm_label`` - memory module label control file | |
288 | ||
289 | This control file allows this DIMM to have a label assigned | |
290 | to it. With this label in the module, when errors occur | |
291 | the output can provide the DIMM label in the system log. | |
292 | This becomes vital for panic events to isolate the | |
293 | cause of the UE event. | |
294 | ||
295 | DIMM Labels must be assigned after booting, with information | |
296 | that correctly identifies the physical slot with its | |
297 | silk screen label. This information is currently very | |
298 | motherboard specific and determination of this information | |
299 | must occur in userland at this time. | |
300 | ||
301 | - ``dimm_location`` - location of the memory module | |
302 | ||
303 | The location can have up to 3 levels, and describe how the | |
304 | memory controller identifies the location of a memory module. | |
305 | Depending on the type of memory and memory controller, it | |
306 | can be: | |
307 | ||
308 | - *csrow* and *channel* - used when the memory controller | |
309 | doesn't identify a single DIMM - e. g. in ``rankX`` dir; | |
310 | - *branch*, *channel*, *slot* - typically used on FB-DIMM memory | |
311 | controllers; | |
312 | - *channel*, *slot* - used on Nehalem and newer Intel drivers. | |
313 | ||
314 | - ``dimm_mem_type`` - Memory Type attribute file | |
315 | ||
316 | This attribute file will display what type of memory is currently | |
317 | on this csrow. Normally, either buffered or unbuffered memory. | |
318 | Examples: | |
319 | ||
320 | - Registered-DDR | |
321 | - Unbuffered-DDR | |
322 | ||
323 | .. [#f5] On some systems, the memory controller doesn't have any logic | |
324 | to identify the memory module. On such systems, the directory is called ``rankX`` and works on a similar way as the ``csrowX`` directories. | |
325 | On modern Intel memory controllers, the memory controller identifies the | |
326 | memory modules directly. On such systems, the directory is called ``dimmX``. | |
327 | ||
328 | .. [#f6] There are also some ``power`` directories and ``subsystem`` | |
329 | symlinks inside the sysfs mapping that are automatically created by | |
330 | the sysfs subsystem. Currently, they serve no purpose. | |
da9bb1d2 | 331 | |
043b4318 BP |
332 | 'csrowX' directories |
333 | -------------------- | |
334 | ||
335 | When CONFIG_EDAC_LEGACY_SYSFS is enabled, sysfs will contain the csrowX | |
336 | directories. As this API doesn't work properly for Rambus, FB-DIMMs and | |
337 | modern Intel Memory Controllers, this is being deprecated in favor of | |
338 | dimmX directories. | |
8b6f04ce | 339 | |
da9bb1d2 | 340 | In the 'csrowX' directories are EDAC control and attribute files for |
5989f11b | 341 | this 'X' instance of csrow: |
da9bb1d2 AC |
342 | |
343 | ||
344 | Total Uncorrectable Errors count attribute file: | |
345 | ||
346 | 'ue_count' | |
347 | ||
348 | This attribute file displays the total count of uncorrectable | |
349 | errors that have occurred on this csrow. If panic_on_ue is set | |
350 | this counter will not have a chance to increment, since EDAC | |
351 | will panic the system. | |
352 | ||
353 | ||
354 | Total Correctable Errors count attribute file: | |
355 | ||
356 | 'ce_count' | |
357 | ||
358 | This attribute file displays the total count of correctable | |
043b4318 BP |
359 | errors that have occurred on this csrow. This count is very |
360 | important to examine. CEs provide early indications that a | |
361 | DIMM is beginning to fail. This count field should be | |
362 | monitored for non-zero values and report such information | |
363 | to the system administrator. | |
da9bb1d2 AC |
364 | |
365 | ||
366 | Total memory managed by this csrow attribute file: | |
367 | ||
368 | 'size_mb' | |
369 | ||
3aae9edd | 370 | This attribute file displays, in count of megabytes, the memory |
f3479816 | 371 | that this csrow contains. |
da9bb1d2 AC |
372 | |
373 | ||
374 | Memory Type attribute file: | |
375 | ||
376 | 'mem_type' | |
377 | ||
378 | This attribute file will display what type of memory is currently | |
379 | on this csrow. Normally, either buffered or unbuffered memory. | |
49c0dab7 DT |
380 | Examples: |
381 | Registered-DDR | |
382 | Unbuffered-DDR | |
da9bb1d2 AC |
383 | |
384 | ||
385 | EDAC Mode of operation attribute file: | |
386 | ||
387 | 'edac_mode' | |
388 | ||
389 | This attribute file will display what type of Error detection | |
390 | and correction is being utilized. | |
391 | ||
392 | ||
393 | Device type attribute file: | |
394 | ||
395 | 'dev_type' | |
396 | ||
49c0dab7 DT |
397 | This attribute file will display what type of DRAM device is |
398 | being utilized on this DIMM. | |
399 | Examples: | |
400 | x1 | |
401 | x2 | |
402 | x4 | |
403 | x8 | |
da9bb1d2 AC |
404 | |
405 | ||
406 | Channel 0 CE Count attribute file: | |
407 | ||
408 | 'ch0_ce_count' | |
409 | ||
410 | This attribute file will display the count of CEs on this | |
411 | DIMM located in channel 0. | |
412 | ||
413 | ||
414 | Channel 0 UE Count attribute file: | |
415 | ||
416 | 'ch0_ue_count' | |
417 | ||
418 | This attribute file will display the count of UEs on this | |
419 | DIMM located in channel 0. | |
420 | ||
421 | ||
422 | Channel 0 DIMM Label control file: | |
423 | ||
424 | 'ch0_dimm_label' | |
425 | ||
426 | This control file allows this DIMM to have a label assigned | |
427 | to it. With this label in the module, when errors occur | |
428 | the output can provide the DIMM label in the system log. | |
429 | This becomes vital for panic events to isolate the | |
430 | cause of the UE event. | |
431 | ||
432 | DIMM Labels must be assigned after booting, with information | |
433 | that correctly identifies the physical slot with its | |
434 | silk screen label. This information is currently very | |
435 | motherboard specific and determination of this information | |
436 | must occur in userland at this time. | |
437 | ||
438 | ||
439 | Channel 1 CE Count attribute file: | |
440 | ||
441 | 'ch1_ce_count' | |
442 | ||
443 | This attribute file will display the count of CEs on this | |
444 | DIMM located in channel 1. | |
445 | ||
446 | ||
447 | Channel 1 UE Count attribute file: | |
448 | ||
449 | 'ch1_ue_count' | |
450 | ||
451 | This attribute file will display the count of UEs on this | |
452 | DIMM located in channel 0. | |
453 | ||
454 | ||
455 | Channel 1 DIMM Label control file: | |
456 | ||
457 | 'ch1_dimm_label' | |
458 | ||
459 | This control file allows this DIMM to have a label assigned | |
460 | to it. With this label in the module, when errors occur | |
461 | the output can provide the DIMM label in the system log. | |
462 | This becomes vital for panic events to isolate the | |
463 | cause of the UE event. | |
464 | ||
465 | DIMM Labels must be assigned after booting, with information | |
466 | that correctly identifies the physical slot with its | |
467 | silk screen label. This information is currently very | |
468 | motherboard specific and determination of this information | |
469 | must occur in userland at this time. | |
470 | ||
043b4318 BP |
471 | |
472 | ||
da9bb1d2 | 473 | SYSTEM LOGGING |
043b4318 | 474 | -------------- |
da9bb1d2 | 475 | |
043b4318 BP |
476 | If logging for UEs and CEs is enabled, then system logs will contain |
477 | information indicating that errors have been detected: | |
da9bb1d2 | 478 | |
49c0dab7 | 479 | EDAC MC0: CE page 0x283, offset 0xce0, grain 8, syndrome 0x6ec3, row 0, |
da9bb1d2 AC |
480 | channel 1 "DIMM_B1": amd76x_edac |
481 | ||
49c0dab7 | 482 | EDAC MC0: CE page 0x1e5, offset 0xfb0, grain 8, syndrome 0xb741, row 0, |
da9bb1d2 AC |
483 | channel 1 "DIMM_B1": amd76x_edac |
484 | ||
485 | ||
486 | The structure of the message is: | |
487 | the memory controller (MC0) | |
488 | Error type (CE) | |
489 | memory page (0x283) | |
490 | offset in the page (0xce0) | |
491 | the byte granularity (grain 8) | |
492 | or resolution of the error | |
493 | the error syndrome (0xb741) | |
494 | memory row (row 0) | |
495 | memory channel (channel 1) | |
496 | DIMM label, if set prior (DIMM B1 | |
497 | and then an optional, driver-specific message that may | |
498 | have additional information. | |
499 | ||
043b4318 BP |
500 | Both UEs and CEs with no info will lack all but memory controller, error |
501 | type, a notice of "no info" and then an optional, driver-specific error | |
502 | message. | |
da9bb1d2 AC |
503 | |
504 | ||
da9bb1d2 | 505 | PCI Bus Parity Detection |
043b4318 | 506 | ------------------------ |
da9bb1d2 | 507 | |
043b4318 BP |
508 | On Header Type 00 devices, the primary status is looked at for any |
509 | parity error regardless of whether parity is enabled on the device or | |
510 | not. (The spec indicates parity is generated in some cases). On Header | |
511 | Type 01 bridges, the secondary status register is also looked at to see | |
512 | if parity occurred on the bus on the other side of the bridge. | |
da9bb1d2 AC |
513 | |
514 | ||
515 | SYSFS CONFIGURATION | |
043b4318 | 516 | ------------------- |
da9bb1d2 AC |
517 | |
518 | Under /sys/devices/system/edac/pci are control and attribute files as follows: | |
519 | ||
520 | ||
521 | Enable/Disable PCI Parity checking control file: | |
522 | ||
523 | 'check_pci_parity' | |
524 | ||
525 | ||
526 | This control file enables or disables the PCI Bus Parity scanning | |
527 | operation. Writing a 1 to this file enables the scanning. Writing | |
528 | a 0 to this file disables the scanning. | |
529 | ||
530 | Enable: | |
531 | echo "1" >/sys/devices/system/edac/pci/check_pci_parity | |
532 | ||
533 | Disable: | |
534 | echo "0" >/sys/devices/system/edac/pci/check_pci_parity | |
535 | ||
536 | ||
327dafb1 AJ |
537 | Parity Count: |
538 | ||
539 | 'pci_parity_count' | |
540 | ||
541 | This attribute file will display the number of parity errors that | |
542 | have been detected. | |
543 | ||
544 | ||
043b4318 | 545 | |
327dafb1 | 546 | MODULE PARAMETERS |
043b4318 | 547 | ----------------- |
327dafb1 AJ |
548 | |
549 | Panic on UE control file: | |
550 | ||
551 | 'edac_mc_panic_on_ue' | |
552 | ||
553 | An uncorrectable error will cause a machine panic. This is usually | |
554 | desirable. It is a bad idea to continue when an uncorrectable error | |
555 | occurs - it is indeterminate what was uncorrected and the operating | |
556 | system context might be so mangled that continuing will lead to further | |
557 | corruption. If the kernel has MCE configured, then EDAC will never | |
558 | notice the UE. | |
559 | ||
560 | LOAD TIME: module/kernel parameter: edac_mc_panic_on_ue=[0|1] | |
561 | ||
562 | RUN TIME: echo "1" > /sys/module/edac_core/parameters/edac_mc_panic_on_ue | |
563 | ||
564 | ||
565 | Log UE control file: | |
566 | ||
567 | 'edac_mc_log_ue' | |
568 | ||
569 | Generate kernel messages describing uncorrectable errors. These errors | |
570 | are reported through the system message log system. UE statistics | |
571 | will be accumulated even when UE logging is disabled. | |
572 | ||
573 | LOAD TIME: module/kernel parameter: edac_mc_log_ue=[0|1] | |
574 | ||
575 | RUN TIME: echo "1" > /sys/module/edac_core/parameters/edac_mc_log_ue | |
576 | ||
577 | ||
578 | Log CE control file: | |
579 | ||
580 | 'edac_mc_log_ce' | |
581 | ||
582 | Generate kernel messages describing correctable errors. These | |
583 | errors are reported through the system message log system. | |
584 | CE statistics will be accumulated even when CE logging is disabled. | |
585 | ||
586 | LOAD TIME: module/kernel parameter: edac_mc_log_ce=[0|1] | |
587 | ||
588 | RUN TIME: echo "1" > /sys/module/edac_core/parameters/edac_mc_log_ce | |
589 | ||
590 | ||
591 | Polling period control file: | |
592 | ||
593 | 'edac_mc_poll_msec' | |
594 | ||
595 | The time period, in milliseconds, for polling for error information. | |
596 | Too small a value wastes resources. Too large a value might delay | |
597 | necessary handling of errors and might loose valuable information for | |
598 | locating the error. 1000 milliseconds (once each second) is the current | |
599 | default. Systems which require all the bandwidth they can get, may | |
600 | increase this. | |
601 | ||
602 | LOAD TIME: module/kernel parameter: edac_mc_poll_msec=[0|1] | |
603 | ||
604 | RUN TIME: echo "1000" > /sys/module/edac_core/parameters/edac_mc_poll_msec | |
605 | ||
da9bb1d2 AC |
606 | |
607 | Panic on PCI PARITY Error: | |
608 | ||
609 | 'panic_on_pci_parity' | |
610 | ||
611 | ||
3aae9edd | 612 | This control file enables or disables panicking when a parity |
da9bb1d2 AC |
613 | error has been detected. |
614 | ||
615 | ||
327dafb1 | 616 | module/kernel parameter: edac_panic_on_pci_pe=[0|1] |
da9bb1d2 AC |
617 | |
618 | Enable: | |
327dafb1 | 619 | echo "1" > /sys/module/edac_core/parameters/edac_panic_on_pci_pe |
da9bb1d2 AC |
620 | |
621 | Disable: | |
327dafb1 | 622 | echo "0" > /sys/module/edac_core/parameters/edac_panic_on_pci_pe |
da9bb1d2 AC |
623 | |
624 | ||
625 | ||
043b4318 BP |
626 | EDAC device type |
627 | ---------------- | |
87f24c3a DT |
628 | |
629 | In the header file, edac_core.h, there is a series of edac_device structures | |
630 | and APIs for the EDAC_DEVICE. | |
631 | ||
632 | User space access to an edac_device is through the sysfs interface. | |
633 | ||
634 | At the location /sys/devices/system/edac (sysfs) new edac_device devices will | |
635 | appear. | |
636 | ||
637 | There is a three level tree beneath the above 'edac' directory. For example, | |
638 | the 'test_device_edac' device (found at the bluesmoke.sourceforget.net website) | |
639 | installs itself as: | |
640 | ||
641 | /sys/devices/systm/edac/test-instance | |
642 | ||
643 | in this directory are various controls, a symlink and one or more 'instance' | |
c98be0c9 | 644 | directories. |
87f24c3a DT |
645 | |
646 | The standard default controls are: | |
647 | ||
648 | log_ce boolean to log CE events | |
649 | log_ue boolean to log UE events | |
650 | panic_on_ue boolean to 'panic' the system if an UE is encountered | |
651 | (default off, can be set true via startup script) | |
652 | poll_msec time period between POLL cycles for events | |
653 | ||
654 | The test_device_edac device adds at least one of its own custom control: | |
655 | ||
656 | test_bits which in the current test driver does nothing but | |
657 | show how it is installed. A ported driver can | |
658 | add one or more such controls and/or attributes | |
659 | for specific uses. | |
660 | One out-of-tree driver uses controls here to allow | |
661 | for ERROR INJECTION operations to hardware | |
662 | injection registers | |
663 | ||
664 | The symlink points to the 'struct dev' that is registered for this edac_device. | |
665 | ||
666 | INSTANCES | |
043b4318 | 667 | --------- |
87f24c3a DT |
668 | |
669 | One or more instance directories are present. For the 'test_device_edac' case: | |
670 | ||
671 | test-instance0 | |
672 | ||
673 | ||
674 | In this directory there are two default counter attributes, which are totals of | |
675 | counter in deeper subdirectories. | |
676 | ||
677 | ce_count total of CE events of subdirectories | |
678 | ue_count total of UE events of subdirectories | |
679 | ||
680 | BLOCKS | |
043b4318 | 681 | ------ |
87f24c3a DT |
682 | |
683 | At the lowest directory level is the 'block' directory. There can be 0, 1 | |
684 | or more blocks specified in each instance. | |
685 | ||
686 | test-block0 | |
687 | ||
688 | ||
689 | In this directory the default attributes are: | |
690 | ||
691 | ce_count which is counter of CE events for this 'block' | |
692 | of hardware being monitored | |
693 | ue_count which is counter of UE events for this 'block' | |
694 | of hardware being monitored | |
695 | ||
696 | ||
697 | The 'test_device_edac' device adds 4 attributes and 1 control: | |
698 | ||
699 | test-block-bits-0 for every POLL cycle this counter | |
700 | is incremented | |
701 | test-block-bits-1 every 10 cycles, this counter is bumped once, | |
702 | and test-block-bits-0 is set to 0 | |
703 | test-block-bits-2 every 100 cycles, this counter is bumped once, | |
704 | and test-block-bits-1 is set to 0 | |
705 | test-block-bits-3 every 1000 cycles, this counter is bumped once, | |
706 | and test-block-bits-2 is set to 0 | |
707 | ||
708 | ||
709 | reset-counters writing ANY thing to this control will | |
710 | reset all the above counters. | |
711 | ||
712 | ||
3aae9edd | 713 | Use of the 'test_device_edac' driver should enable any others to create their own |
87f24c3a DT |
714 | unique drivers for their hardware systems. |
715 | ||
716 | The 'test_device_edac' sample driver is located at the | |
717 | bluesmoke.sourceforge.net project site for EDAC. | |
718 | ||
043b4318 | 719 | |
31983a04 | 720 | NEHALEM USAGE OF EDAC APIs |
043b4318 | 721 | -------------------------- |
31983a04 MCC |
722 | |
723 | This chapter documents some EXPERIMENTAL mappings for EDAC API to handle | |
724 | Nehalem EDAC driver. They will likely be changed on future versions | |
725 | of the driver. | |
726 | ||
727 | Due to the way Nehalem exports Memory Controller data, some adjustments | |
728 | were done at i7core_edac driver. This chapter will cover those differences | |
729 | ||
3aae9edd | 730 | 1) On Nehalem, there is one Memory Controller per Quick Patch Interconnect |
c3444363 MCC |
731 | (QPI). At the driver, the term "socket" means one QPI. This is |
732 | associated with a physical CPU socket. | |
31983a04 MCC |
733 | |
734 | Each MC have 3 physical read channels, 3 physical write channels and | |
c94bed8e | 735 | 3 logic channels. The driver currently sees it as just 3 channels. |
31983a04 MCC |
736 | Each channel can have up to 3 DIMMs. |
737 | ||
738 | The minimum known unity is DIMMs. There are no information about csrows. | |
3aae9edd | 739 | As EDAC API maps the minimum unity is csrows, the driver sequentially |
c3444363 MCC |
740 | maps channel/dimm into different csrows. |
741 | ||
25985edc | 742 | For example, supposing the following layout: |
c3444363 MCC |
743 | Ch0 phy rd0, wr0 (0x063f4031): 2 ranks, UDIMMs |
744 | dimm 0 1024 Mb offset: 0, bank: 8, rank: 1, row: 0x4000, col: 0x400 | |
745 | dimm 1 1024 Mb offset: 4, bank: 8, rank: 1, row: 0x4000, col: 0x400 | |
746 | Ch1 phy rd1, wr1 (0x063f4031): 2 ranks, UDIMMs | |
747 | dimm 0 1024 Mb offset: 0, bank: 8, rank: 1, row: 0x4000, col: 0x400 | |
748 | Ch2 phy rd3, wr3 (0x063f4031): 2 ranks, UDIMMs | |
749 | dimm 0 1024 Mb offset: 0, bank: 8, rank: 1, row: 0x4000, col: 0x400 | |
750 | The driver will map it as: | |
751 | csrow0: channel 0, dimm0 | |
752 | csrow1: channel 0, dimm1 | |
753 | csrow2: channel 1, dimm0 | |
754 | csrow3: channel 2, dimm0 | |
755 | ||
756 | exports one | |
31983a04 MCC |
757 | DIMM per csrow. |
758 | ||
c3444363 | 759 | Each QPI is exported as a different memory controller. |
31983a04 | 760 | |
3aae9edd | 761 | 2) Nehalem MC has the ability to generate errors. The driver implements this |
31983a04 MCC |
762 | functionality via some error injection nodes: |
763 | ||
764 | For injecting a memory error, there are some sysfs nodes, under | |
c3444363 | 765 | /sys/devices/system/edac/mc/mc?/: |
31983a04 | 766 | |
35be9544 | 767 | inject_addrmatch/*: |
31983a04 MCC |
768 | Controls the error injection mask register. It is possible to specify |
769 | several characteristics of the address to match an error code: | |
770 | dimm = the affected dimm. Numbers are relative to a channel; | |
771 | rank = the memory rank; | |
772 | channel = the channel that will generate an error; | |
773 | bank = the affected bank; | |
774 | page = the page address; | |
775 | column (or col) = the address column. | |
776 | each of the above values can be set to "any" to match any valid value. | |
777 | ||
778 | At driver init, all values are set to any. | |
779 | ||
780 | For example, to generate an error at rank 1 of dimm 2, for any channel, | |
781 | any bank, any page, any column: | |
35be9544 MCC |
782 | echo 2 >/sys/devices/system/edac/mc/mc0/inject_addrmatch/dimm |
783 | echo 1 >/sys/devices/system/edac/mc/mc0/inject_addrmatch/rank | |
31983a04 MCC |
784 | |
785 | To return to the default behaviour of matching any, you can do: | |
35be9544 MCC |
786 | echo any >/sys/devices/system/edac/mc/mc0/inject_addrmatch/dimm |
787 | echo any >/sys/devices/system/edac/mc/mc0/inject_addrmatch/rank | |
31983a04 MCC |
788 | |
789 | inject_eccmask: | |
790 | specifies what bits will have troubles, | |
791 | ||
792 | inject_section: | |
793 | specifies what ECC cache section will get the error: | |
794 | 3 for both | |
795 | 2 for the highest | |
796 | 1 for the lowest | |
797 | ||
31983a04 MCC |
798 | inject_type: |
799 | specifies the type of error, being a combination of the following bits: | |
800 | bit 0 - repeat | |
801 | bit 1 - ecc | |
802 | bit 2 - parity | |
803 | ||
804 | inject_enable starts the error generation when something different | |
805 | than 0 is written. | |
806 | ||
807 | All inject vars can be read. root permission is needed for write. | |
808 | ||
809 | Datasheet states that the error will only be generated after a write on an | |
810 | address that matches inject_addrmatch. It seems, however, that reading will | |
811 | also produce an error. | |
812 | ||
813 | For example, the following code will generate an error for any write access | |
814 | at socket 0, on any DIMM/address on channel 2: | |
815 | ||
35be9544 | 816 | echo 2 >/sys/devices/system/edac/mc/mc0/inject_addrmatch/channel |
31983a04 MCC |
817 | echo 2 >/sys/devices/system/edac/mc/mc0/inject_type |
818 | echo 64 >/sys/devices/system/edac/mc/mc0/inject_eccmask | |
819 | echo 3 >/sys/devices/system/edac/mc/mc0/inject_section | |
31983a04 MCC |
820 | echo 1 >/sys/devices/system/edac/mc/mc0/inject_enable |
821 | dd if=/dev/mem of=/dev/null seek=16k bs=4k count=1 >& /dev/null | |
822 | ||
c3444363 MCC |
823 | For socket 1, it is needed to replace "mc0" by "mc1" at the above |
824 | commands. | |
825 | ||
31983a04 MCC |
826 | The generated error message will look like: |
827 | ||
828 | EDAC MC0: UE row 0, channel-a= 0 channel-b= 0 labels "-": NON_FATAL (addr = 0x0075b980, socket=0, Dimm=0, Channel=2, syndrome=0x00000040, count=1, Err=8c0000400001009f:4000080482 (read error: read ECC error)) | |
829 | ||
830 | 3) Nehalem specific Corrected Error memory counters | |
831 | ||
35be9544 MCC |
832 | Nehalem have some registers to count memory errors. The driver uses those |
833 | registers to report Corrected Errors on devices with Registered Dimms. | |
31983a04 | 834 | |
35be9544 MCC |
835 | However, those counters don't work with Unregistered Dimms. As the chipset |
836 | offers some counters that also work with UDIMMS (but with a worse level of | |
837 | granularity than the default ones), the driver exposes those registers for | |
838 | UDIMM memories. | |
c3444363 | 839 | |
35be9544 | 840 | They can be read by looking at the contents of all_channel_counts/ |
31983a04 | 841 | |
35be9544 MCC |
842 | $ for i in /sys/devices/system/edac/mc/mc0/all_channel_counts/*; do echo $i; cat $i; done |
843 | /sys/devices/system/edac/mc/mc0/all_channel_counts/udimm0 | |
844 | 0 | |
845 | /sys/devices/system/edac/mc/mc0/all_channel_counts/udimm1 | |
846 | 0 | |
847 | /sys/devices/system/edac/mc/mc0/all_channel_counts/udimm2 | |
848 | 0 | |
c3444363 MCC |
849 | |
850 | What happens here is that errors on different csrows, but at the same | |
851 | dimm number will increment the same counter. | |
852 | So, in this memory mapping: | |
853 | csrow0: channel 0, dimm0 | |
854 | csrow1: channel 0, dimm1 | |
855 | csrow2: channel 1, dimm0 | |
856 | csrow3: channel 2, dimm0 | |
35be9544 MCC |
857 | The hardware will increment udimm0 for an error at the first dimm at either |
858 | csrow0, csrow2 or csrow3; | |
859 | The hardware will increment udimm1 for an error at the second dimm at either | |
860 | csrow0, csrow2 or csrow3; | |
861 | The hardware will increment udimm2 for an error at the third dimm at either | |
862 | csrow0, csrow2 or csrow3; | |
c3444363 MCC |
863 | |
864 | 4) Standard error counters | |
865 | ||
866 | The standard error counters are generated when an mcelog error is received | |
35be9544 | 867 | by the driver. Since, with udimm, this is counted by software, it is |
3aae9edd | 868 | possible that some errors could be lost. With rdimm's, they display the |
35be9544 | 869 | contents of the registers |
043b4318 | 870 | |
6b7464b7 AG |
871 | AMD64_EDAC REFERENCE DOCUMENTS USED |
872 | ----------------------------------- | |
873 | amd64_edac module is based on the following documents | |
874 | (available from http://support.amd.com/en-us/search/tech-docs): | |
875 | ||
876 | 1. Title: BIOS and Kernel Developer's Guide for AMD Athlon 64 and AMD | |
877 | Opteron Processors | |
878 | AMD publication #: 26094 | |
879 | Revision: 3.26 | |
880 | Link: http://support.amd.com/TechDocs/26094.PDF | |
881 | ||
882 | 2. Title: BIOS and Kernel Developer's Guide for AMD NPT Family 0Fh | |
883 | Processors | |
884 | AMD publication #: 32559 | |
885 | Revision: 3.00 | |
886 | Issue Date: May 2006 | |
887 | Link: http://support.amd.com/TechDocs/32559.pdf | |
888 | ||
889 | 3. Title: BIOS and Kernel Developer's Guide (BKDG) For AMD Family 10h | |
890 | Processors | |
891 | AMD publication #: 31116 | |
892 | Revision: 3.00 | |
893 | Issue Date: September 07, 2007 | |
894 | Link: http://support.amd.com/TechDocs/31116.pdf | |
895 | ||
896 | 4. Title: BIOS and Kernel Developer's Guide (BKDG) for AMD Family 15h | |
897 | Models 30h-3Fh Processors | |
898 | AMD publication #: 49125 | |
899 | Revision: 3.06 | |
900 | Issue Date: 2/12/2015 (latest release) | |
901 | Link: http://support.amd.com/TechDocs/49125_15h_Models_30h-3Fh_BKDG.pdf | |
902 | ||
903 | 5. Title: BIOS and Kernel Developer's Guide (BKDG) for AMD Family 15h | |
904 | Models 60h-6Fh Processors | |
905 | AMD publication #: 50742 | |
906 | Revision: 3.01 | |
907 | Issue Date: 7/23/2015 (latest release) | |
908 | Link: http://support.amd.com/TechDocs/50742_15h_Models_60h-6Fh_BKDG.pdf | |
909 | ||
910 | 6. Title: BIOS and Kernel Developer's Guide (BKDG) for AMD Family 16h | |
911 | Models 00h-0Fh Processors | |
912 | AMD publication #: 48751 | |
913 | Revision: 3.03 | |
914 | Issue Date: 2/23/2015 (latest release) | |
915 | Link: http://support.amd.com/TechDocs/48751_16h_bkdg.pdf | |
916 | ||
043b4318 BP |
917 | CREDITS: |
918 | ======== | |
919 | ||
920 | Written by Doug Thompson <dougthompson@xmission.com> | |
921 | 7 Dec 2005 | |
922 | 17 Jul 2007 Updated | |
923 | ||
924 | (c) Mauro Carvalho Chehab | |
925 | 05 Aug 2009 Nehalem interface | |
926 | ||
927 | EDAC authors/maintainers: | |
928 | ||
929 | Doug Thompson, Dave Jiang, Dave Peterson et al, | |
930 | Mauro Carvalho Chehab | |
931 | Borislav Petkov | |
932 | original author: Thayne Harbaugh |