1.. SPDX-License-Identifier: GPL-2.0 2.. include:: <isonum.txt> 3 4================================================== 5Reliability, Availability and Serviceability (RAS) 6================================================== 7 8This documents different aspects of the RAS functionality present in the 9kernel. 10 11RAS concepts 12************ 13 14Reliability, Availability and Serviceability (RAS) is a concept used on 15servers meant to measure their robustness. 16 17Reliability 18 is the probability that a system will produce correct outputs. 19 20 * Generally measured as Mean Time Between Failures (MTBF) 21 * Enhanced by features that help to avoid, detect and repair hardware faults 22 23Availability 24 is the probability that a system is operational at a given time 25 26 * Generally measured as a percentage of downtime per a period of time 27 * Often uses mechanisms to detect and correct hardware faults in 28 runtime; 29 30Serviceability (or maintainability) 31 is the simplicity and speed with which a system can be repaired or 32 maintained 33 34 * Generally measured on Mean Time Between Repair (MTBR) 35 36Improving RAS 37------------- 38 39In order to reduce systems downtime, a system should be capable of detecting 40hardware errors, and, when possible correcting them in runtime. It should 41also provide mechanisms to detect hardware degradation, in order to warn 42the system administrator to take the action of replacing a component before 43it causes data loss or system downtime. 44 45Among the monitoring measures, the most usual ones include: 46 47* CPU ��� detect errors at instruction execution and at L1/L2/L3 caches; 48* Memory ��� add error correction logic (ECC) to detect and correct errors; 49* I/O ��� add CRC checksums for transferred data; 50* Storage ��� RAID, journal file systems, checksums, 51 Self-Monitoring, Analysis and Reporting Technology (SMART). 52 53By monitoring the number of occurrences of error detections, it is possible 54to identify if the probability of hardware errors is increasing, and, on such 55case, do a preventive maintenance to replace a degraded component while 56those errors are correctable. 57 58Types of errors 59--------------- 60 61Most mechanisms used on modern systems use technologies like Hamming 62Codes that allow error correction when the number of errors on a bit packet 63is below a threshold. If the number of errors is above, those mechanisms 64can indicate with a high degree of confidence that an error happened, but 65they can't correct. 66 67Also, sometimes an error occur on a component that it is not used. For 68example, a part of the memory that it is not currently allocated. 69 70That defines some categories of errors: 71 72* **Correctable Error (CE)** - the error detection mechanism detected and 73 corrected the error. Such errors are usually not fatal, although some 74 Kernel mechanisms allow the system administrator to consider them as fatal. 75 76* **Uncorrected Error (UE)** - the amount of errors happened above the error 77 correction threshold, and the system was unable to auto-correct. 78 79* **Fatal Error** - when an UE error happens on a critical component of the 80 system (for example, a piece of the Kernel got corrupted by an UE), the 81 only reliable way to avoid data corruption is to hang or reboot the machine. 82 83* **Non-fatal Error** - when an UE error happens on an unused component, 84 like a CPU in power down state or an unused memory bank, the system may 85 still run, eventually replacing the affected hardware by a hot spare, 86 if available. 87 88 Also, when an error happens on a userspace process, it is also possible to 89 kill such process and let userspace restart it. 90 91The mechanism for handling non-fatal errors is usually complex and may 92require the help of some userspace application, in order to apply the 93policy desired by the system administrator. 94 95Identifying a bad hardware component 96------------------------------------ 97 98Just detecting a hardware flaw is usually not enough, as the system needs 99to pinpoint to the minimal replaceable unit (MRU) that should be exchanged 100to make the hardware reliable again. 101 102So, it requires not only error logging facilities, but also mechanisms that 103will translate the error message to the silkscreen or component label for 104the MRU. 105 106Typically, it is very complex for memory, as modern CPUs interlace memory 107from different memory modules, in order to provide a better performance. The 108DMI BIOS usually have a list of memory module labels, with can be obtained 109using the ``dmidecode`` tool. For example, on a desktop machine, it shows:: 110 111 Memory Device 112 Total Width: 64 bits 113 Data Width: 64 bits 114 Size: 16384 MB 115 Form Factor: SODIMM 116 Set: None 117 Locator: ChannelA-DIMM0 118 Bank Locator: BANK 0 119 Type: DDR4 120 Type Detail: Synchronous 121 Speed: 2133 MHz 122 Rank: 2 123 Configured Clock Speed: 2133 MHz 124 125On the above example, a DDR4 SO-DIMM memory module is located at the 126system's memory labeled as "BANK 0", as given by the *bank locator* field. 127Please notice that, on such system, the *total width* is equal to the 128*data width*. It means that such memory module doesn't have error 129detection/correction mechanisms. 130 131Unfortunately, not all systems use the same field to specify the memory 132bank. On this example, from an older server, ``dmidecode`` shows:: 133 134 Memory Device 135 Array Handle: 0x1000 136 Error Information Handle: Not Provided 137 Total Width: 72 bits 138 Data Width: 64 bits 139 Size: 8192 MB 140 Form Factor: DIMM 141 Set: 1 142 Locator: DIMM_A1 143 Bank Locator: Not Specified 144 Type: DDR3 145 Type Detail: Synchronous Registered (Buffered) 146 Speed: 1600 MHz 147 Rank: 2 148 Configured Clock Speed: 1600 MHz 149 150There, the DDR3 RDIMM memory module is located at the system's memory labeled 151as "DIMM_A1", as given by the *locator* field. Please notice that this 152memory module has 64 bits of *data width* and 72 bits of *total width*. So, 153it has 8 extra bits to be used by error detection and correction mechanisms. 154Such kind of memory is called Error-correcting code memory (ECC memory). 155 156To make things even worse, it is not uncommon that systems with different 157labels on their system's board to use exactly the same BIOS, meaning that 158the labels provided by the BIOS won't match the real ones. 159 160ECC memory 161---------- 162 163As mentioned in the previous section, ECC memory has extra bits to be 164used for error correction. In the above example, a memory module has 16564 bits of *data width*, and 72 bits of *total width*. The extra 8 166bits which are used for the error detection and correction mechanisms 167are referred to as the *syndrome*\ [#f1]_\ [#f2]_. 168 169So, when the cpu requests the memory controller to write a word with 170*data width*, the memory controller calculates the *syndrome* in real time, 171using Hamming code, or some other error correction code, like SECDED+, 172producing a code with *total width* size. Such code is then written 173on the memory modules. 174 175At read, the *total width* bits code is converted back, using the same 176ECC code used on write, producing a word with *data width* and a *syndrome*. 177The word with *data width* is sent to the CPU, even when errors happen. 178 179The memory controller also looks at the *syndrome* in order to check if 180there was an error, and if the ECC code was able to fix such error. 181If the error was corrected, a Corrected Error (CE) happened. If not, an 182Uncorrected Error (UE) happened. 183 184The information about the CE/UE errors is stored on some special registers 185at the memory controller and can be accessed by reading such registers, 186either by BIOS, by some special CPUs or by Linux EDAC driver. On x86 64 187bit CPUs, such errors can also be retrieved via the Machine Check 188Architecture (MCA)\ [#f3]_. 189 190.. [#f1] Please notice that several memory controllers allow operation on a 191 mode called "Lock-Step", where it groups two memory modules together, 192 doing 128-bit reads/writes. That gives 16 bits for error correction, with 193 significantly improves the error correction mechanism, at the expense 194 that, when an error happens, there's no way to know what memory module is 195 to blame. So, it has to blame both memory modules. 196 197.. [#f2] Some memory controllers also allow using memory in mirror mode. 198 On such mode, the same data is written to two memory modules. At read, 199 the system checks both memory modules, in order to check if both provide 200 identical data. On such configuration, when an error happens, there's no 201 way to know what memory module is to blame. So, it has to blame both 202 memory modules (or 4 memory modules, if the system is also on Lock-step 203 mode). 204 205.. [#f3] For more details about the Machine Check Architecture (MCA), 206 please read Documentation/arch/x86/x86_64/machinecheck.rst at the Kernel tree. 207 208EDAC - Error Detection And Correction 209************************************* 210 211.. note:: 212 213 "bluesmoke" was the name for this device driver subsystem when it 214 was "out-of-tree" and maintained at http://bluesmoke.sourceforge.net. 215 That site is mostly archaic now and can be used only for historical 216 purposes. 217 218 When the subsystem was pushed upstream for the first time, on 219 Kernel 2.6.16, it was renamed to ``EDAC``. 220 221Purpose 222------- 223 224The ``edac`` kernel module's goal is to detect and report hardware errors 225that occur within the computer system running under linux. 226 227Memory 228------ 229 230Memory Correctable Errors (CE) and Uncorrectable Errors (UE) are the 231primary errors being harvested. These types of errors are harvested by 232the ``edac_mc`` device. 233 234Detecting CE events, then harvesting those events and reporting them, 235**can** but must not necessarily be a predictor of future UE events. With 236CE events only, the system can and will continue to operate as no data 237has been damaged yet. 238 239However, preventive maintenance and proactive part replacement of memory 240modules exhibiting CEs can reduce the likelihood of the dreaded UE events 241and system panics. 242 243Other hardware elements 244----------------------- 245 246A new feature for EDAC, the ``edac_device`` class of device, was added in 247the 2.6.23 version of the kernel. 248 249This new device type allows for non-memory type of ECC hardware detectors 250to have their states harvested and presented to userspace via the sysfs 251interface. 252 253Some architectures have ECC detectors for L1, L2 and L3 caches, 254along with DMA engines, fabric switches, main data path switches, 255interconnections, and various other hardware data paths. If the hardware 256reports it, then a edac_device device probably can be constructed to 257harvest and present that to userspace. 258 259 260PCI bus scanning 261---------------- 262 263In addition, PCI devices are scanned for PCI Bus Parity and SERR Errors 264in order to determine if errors are occurring during data transfers. 265 266The presence of PCI Parity errors must be examined with a grain of salt. 267There are several add-in adapters that do **not** follow the PCI specification 268with regards to Parity generation and reporting. The specification says 269the vendor should tie the parity status bits to 0 if they do not intend 270to generate parity. Some vendors do not do this, and thus the parity bit 271can "float" giving false positives. 272 273There is a PCI device attribute located in sysfs that is checked by 274the EDAC PCI scanning code. If that attribute is set, PCI parity/error 275scanning is skipped for that device. The attribute is:: 276 277 broken_parity_status 278 279and is located in ``/sys/devices/pci<XXX>/0000:XX:YY.Z`` directories for 280PCI devices. 281 282 283Versioning 284---------- 285 286EDAC is composed of a "core" module (``edac_core.ko``) and several Memory 287Controller (MC) driver modules. On a given system, the CORE is loaded 288and one MC driver will be loaded. Both the CORE and the MC driver (or 289``edac_device`` driver) have individual versions that reflect current 290release level of their respective modules. 291 292Thus, to "report" on what version a system is running, one must report 293both the CORE's and the MC driver's versions. 294 295 296Loading 297------- 298 299If ``edac`` was statically linked with the kernel then no loading 300is necessary. If ``edac`` was built as modules then simply modprobe 301the ``edac`` pieces that you need. You should be able to modprobe 302hardware-specific modules and have the dependencies load the necessary 303core modules. 304 305Example:: 306 307 $ modprobe amd76x_edac 308 309loads both the ``amd76x_edac.ko`` memory controller module and the 310``edac_mc.ko`` core module. 311 312 313Sysfs interface 314--------------- 315 316EDAC presents a ``sysfs`` interface for control and reporting purposes. It 317lives in the /sys/devices/system/edac directory. 318 319Within this directory there currently reside 2 components: 320 321 ======= ============================== 322 mc memory controller(s) system 323 pci PCI control and status system 324 ======= ============================== 325 326 327 328Memory Controller (mc) Model 329---------------------------- 330 331Each ``mc`` device controls a set of memory modules [#f4]_. These modules 332are laid out in a Chip-Select Row (``csrowX``) and Channel table (``chX``). 333There can be multiple csrows and multiple channels. 334 335.. [#f4] Nowadays, the term DIMM (Dual In-line Memory Module) is widely 336 used to refer to a memory module, although there are other memory 337 packaging alternatives, like SO-DIMM, SIMM, etc. The UEFI 338 specification (Version 2.7) defines a memory module in the Common 339 Platform Error Record (CPER) section to be an SMBIOS Memory Device 340 (Type 17). Along this document, and inside the EDAC subsystem, the term 341 "dimm" is used for all memory modules, even when they use a 342 different kind of packaging. 343 344Memory controllers allow for several csrows, with 8 csrows being a 345typical value. Yet, the actual number of csrows depends on the layout of 346a given motherboard, memory controller and memory module characteristics. 347 348Dual channels allow for dual data length (e. g. 128 bits, on 64 bit systems) 349data transfers to/from the CPU from/to memory. Some newer chipsets allow 350for more than 2 channels, like Fully Buffered DIMMs (FB-DIMMs) memory 351controllers. The following example will assume 2 channels: 352 353 +------------+-----------------------+ 354 | CS Rows | Channels | 355 +------------+-----------+-----------+ 356 | | ``ch0`` | ``ch1`` | 357 +============+===========+===========+ 358 | |**DIMM_A0**|**DIMM_B0**| 359 +------------+-----------+-----------+ 360 | ``csrow0`` | rank0 | rank0 | 361 +------------+-----------+-----------+ 362 | ``csrow1`` | rank1 | rank1 | 363 +------------+-----------+-----------+ 364 | |**DIMM_A1**|**DIMM_B1**| 365 +------------+-----------+-----------+ 366 | ``csrow2`` | rank0 | rank0 | 367 +------------+-----------+-----------+ 368 | ``csrow3`` | rank1 | rank1 | 369 +------------+-----------+-----------+ 370 371In the above example, there are 4 physical slots on the motherboard 372for memory DIMMs: 373 374 +---------+---------+ 375 | DIMM_A0 | DIMM_B0 | 376 +---------+---------+ 377 | DIMM_A1 | DIMM_B1 | 378 +---------+---------+ 379 380Labels for these slots are usually silk-screened on the motherboard. 381Slots labeled ``A`` are channel 0 in this example. Slots labeled ``B`` are 382channel 1. Notice that there are two csrows possible on a physical DIMM. 383These csrows are allocated their csrow assignment based on the slot into 384which the memory DIMM is placed. Thus, when 1 DIMM is placed in each 385Channel, the csrows cross both DIMMs. 386 387Memory DIMMs come single or dual "ranked". A rank is a populated csrow. 388In the example above 2 dual ranked DIMMs are similarly placed. Thus, 389both csrow0 and csrow1 are populated. On the other hand, when 2 single 390ranked DIMMs are placed in slots DIMM_A0 and DIMM_B0, then they will 391have just one csrow (csrow0) and csrow1 will be empty. The pattern 392repeats itself for csrow2 and csrow3. Also note that some memory 393controllers don't have any logic to identify the memory module, see 394``rankX`` directories below. 395 396The representation of the above is reflected in the directory 397tree in EDAC's sysfs interface. Starting in directory 398``/sys/devices/system/edac/mc``, each memory controller will be 399represented by its own ``mcX`` directory, where ``X`` is the 400index of the MC:: 401 402 ..../edac/mc/ 403 | 404 |->mc0 405 |->mc1 406 |->mc2 407 .... 408 409Under each ``mcX`` directory each ``csrowX`` is again represented by a 410``csrowX``, where ``X`` is the csrow index:: 411 412 .../mc/mc0/ 413 | 414 |->csrow0 415 |->csrow2 416 |->csrow3 417 .... 418 419Notice that there is no csrow1, which indicates that csrow0 is composed 420of a single ranked DIMMs. This should also apply in both Channels, in 421order to have dual-channel mode be operational. Since both csrow2 and 422csrow3 are populated, this indicates a dual ranked set of DIMMs for 423channels 0 and 1. 424 425Within each of the ``mcX`` and ``csrowX`` directories are several EDAC 426control and attribute files. 427 428``mcX`` directories 429------------------- 430 431In ``mcX`` directories are EDAC control and attribute files for 432this ``X`` instance of the memory controllers. 433 434For a description of the sysfs API, please see: 435 436 Documentation/ABI/testing/sysfs-devices-edac 437 438 439``dimmX`` or ``rankX`` directories 440---------------------------------- 441 442The recommended way to use the EDAC subsystem is to look at the information 443provided by the ``dimmX`` or ``rankX`` directories [#f5]_. 444 445A typical EDAC system has the following structure under 446``/sys/devices/system/edac/``\ [#f6]_:: 447 448 /sys/devices/system/edac/ 449 ��������� mc 450 ������� ��������� mc0 451 ������� ������� ��������� ce_count 452 ������� ������� ��������� ce_noinfo_count 453 ������� ������� ��������� dimm0 454 ������� ������� ������� ��������� dimm_ce_count 455 ������� ������� ������� ��������� dimm_dev_type 456 ������� ������� ������� ��������� dimm_edac_mode 457 ������� ������� ������� ��������� dimm_label 458 ������� ������� ������� ��������� dimm_location 459 ������� ������� ������� ��������� dimm_mem_type 460 ������� ������� ������� ��������� dimm_ue_count 461 ������� ������� ������� ��������� size 462 ������� ������� ������� ��������� uevent 463 ������� ������� ��������� max_location 464 ������� ������� ��������� mc_name 465 ������� ������� ��������� reset_counters 466 ������� ������� ��������� seconds_since_reset 467 ������� ������� ��������� size_mb 468 ������� ������� ��������� ue_count 469 ������� ������� ��������� ue_noinfo_count 470 ������� ������� ��������� uevent 471 ������� ��������� mc1 472 ������� ������� ��������� ce_count 473 ������� ������� ��������� ce_noinfo_count 474 ������� ������� ��������� dimm0 475 ������� ������� ������� ��������� dimm_ce_count 476 ������� ������� ������� ��������� dimm_dev_type 477 ������� ������� ������� ��������� dimm_edac_mode 478 ������� ������� ������� ��������� dimm_label 479 ������� ������� ������� ��������� dimm_location 480 ������� ������� ������� ��������� dimm_mem_type 481 ������� ������� ������� ��������� dimm_ue_count 482 ������� ������� ������� ��������� size 483 ������� ������� ������� ��������� uevent 484 ������� ������� ��������� max_location 485 ������� ������� ��������� mc_name 486 ������� ������� ��������� reset_counters 487 ������� ������� ��������� seconds_since_reset 488 ������� ������� ��������� size_mb 489 ������� ������� ��������� ue_count 490 ������� ������� ��������� ue_noinfo_count 491 ������� ������� ��������� uevent 492 ������� ��������� uevent 493 ��������� uevent 494 495In the ``dimmX`` directories are EDAC control and attribute files for 496this ``X`` memory module: 497 498- ``size`` - Total memory managed by this csrow attribute file 499 500 This attribute file displays, in count of megabytes, the memory 501 that this csrow contains. 502 503- ``dimm_ue_count`` - Uncorrectable Errors count attribute file 504 505 This attribute file displays the total count of uncorrectable 506 errors that have occurred on this DIMM. If panic_on_ue is set 507 this counter will not have a chance to increment, since EDAC 508 will panic the system. 509 510- ``dimm_ce_count`` - Correctable Errors count attribute file 511 512 This attribute file displays the total count of correctable 513 errors that have occurred on this DIMM. This count is very 514 important to examine. CEs provide early indications that a 515 DIMM is beginning to fail. This count field should be 516 monitored for non-zero values and report such information 517 to the system administrator. 518 519- ``dimm_dev_type`` - Device type attribute file 520 521 This attribute file will display what type of DRAM device is 522 being utilized on this DIMM. 523 Examples: 524 525 - x1 526 - x2 527 - x4 528 - x8 529 530- ``dimm_edac_mode`` - EDAC Mode of operation attribute file 531 532 This attribute file will display what type of Error detection 533 and correction is being utilized. 534 535- ``dimm_label`` - memory module label control file 536 537 This control file allows this DIMM to have a label assigned 538 to it. With this label in the module, when errors occur 539 the output can provide the DIMM label in the system log. 540 This becomes vital for panic events to isolate the 541 cause of the UE event. 542 543 DIMM Labels must be assigned after booting, with information 544 that correctly identifies the physical slot with its 545 silk screen label. This information is currently very 546 motherboard specific and determination of this information 547 must occur in userland at this time. 548 549- ``dimm_location`` - location of the memory module 550 551 The location can have up to 3 levels, and describe how the 552 memory controller identifies the location of a memory module. 553 Depending on the type of memory and memory controller, it 554 can be: 555 556 - *csrow* and *channel* - used when the memory controller 557 doesn't identify a single DIMM - e. g. in ``rankX`` dir; 558 - *branch*, *channel*, *slot* - typically used on FB-DIMM memory 559 controllers; 560 - *channel*, *slot* - used on Nehalem and newer Intel drivers. 561 562- ``dimm_mem_type`` - Memory Type attribute file 563 564 This attribute file will display what type of memory is currently 565 on this csrow. Normally, either buffered or unbuffered memory. 566 Examples: 567 568 - Registered-DDR 569 - Unbuffered-DDR 570 571.. [#f5] On some systems, the memory controller doesn't have any logic 572 to identify the memory module. On such systems, the directory is called ``rankX`` and works on a similar way as the ``csrowX`` directories. 573 On modern Intel memory controllers, the memory controller identifies the 574 memory modules directly. On such systems, the directory is called ``dimmX``. 575 576.. [#f6] There are also some ``power`` directories and ``subsystem`` 577 symlinks inside the sysfs mapping that are automatically created by 578 the sysfs subsystem. Currently, they serve no purpose. 579 580``csrowX`` directories 581---------------------- 582 583When CONFIG_EDAC_LEGACY_SYSFS is enabled, sysfs will contain the ``csrowX`` 584directories. As this API doesn't work properly for Rambus, FB-DIMMs and 585modern Intel Memory Controllers, this is being deprecated in favor of 586``dimmX`` directories. 587 588In the ``csrowX`` directories are EDAC control and attribute files for 589this ``X`` instance of csrow: 590 591 592- ``ue_count`` - Total Uncorrectable Errors count attribute file 593 594 This attribute file displays the total count of uncorrectable 595 errors that have occurred on this csrow. If panic_on_ue is set 596 this counter will not have a chance to increment, since EDAC 597 will panic the system. 598 599 600- ``ce_count`` - Total Correctable Errors count attribute file 601 602 This attribute file displays the total count of correctable 603 errors that have occurred on this csrow. This count is very 604 important to examine. CEs provide early indications that a 605 DIMM is beginning to fail. This count field should be 606 monitored for non-zero values and report such information 607 to the system administrator. 608 609 610- ``size_mb`` - Total memory managed by this csrow attribute file 611 612 This attribute file displays, in count of megabytes, the memory 613 that this csrow contains. 614 615 616- ``mem_type`` - Memory Type attribute file 617 618 This attribute file will display what type of memory is currently 619 on this csrow. Normally, either buffered or unbuffered memory. 620 Examples: 621 622 - Registered-DDR 623 - Unbuffered-DDR 624 625 626- ``edac_mode`` - EDAC Mode of operation attribute file 627 628 This attribute file will display what type of Error detection 629 and correction is being utilized. 630 631 632- ``dev_type`` - Device type attribute file 633 634 This attribute file will display what type of DRAM device is 635 being utilized on this DIMM. 636 Examples: 637 638 - x1 639 - x2 640 - x4 641 - x8 642 643 644- ``ch0_ce_count`` - Channel 0 CE Count attribute file 645 646 This attribute file will display the count of CEs on this 647 DIMM located in channel 0. 648 649 650- ``ch0_ue_count`` - Channel 0 UE Count attribute file 651 652 This attribute file will display the count of UEs on this 653 DIMM located in channel 0. 654 655 656- ``ch0_dimm_label`` - Channel 0 DIMM Label control file 657 658 659 This control file allows this DIMM to have a label assigned 660 to it. With this label in the module, when errors occur 661 the output can provide the DIMM label in the system log. 662 This becomes vital for panic events to isolate the 663 cause of the UE event. 664 665 DIMM Labels must be assigned after booting, with information 666 that correctly identifies the physical slot with its 667 silk screen label. This information is currently very 668 motherboard specific and determination of this information 669 must occur in userland at this time. 670 671 672- ``ch1_ce_count`` - Channel 1 CE Count attribute file 673 674 675 This attribute file will display the count of CEs on this 676 DIMM located in channel 1. 677 678 679- ``ch1_ue_count`` - Channel 1 UE Count attribute file 680 681 682 This attribute file will display the count of UEs on this 683 DIMM located in channel 0. 684 685 686- ``ch1_dimm_label`` - Channel 1 DIMM Label control file 687 688 This control file allows this DIMM to have a label assigned 689 to it. With this label in the module, when errors occur 690 the output can provide the DIMM label in the system log. 691 This becomes vital for panic events to isolate the 692 cause of the UE event. 693 694 DIMM Labels must be assigned after booting, with information 695 that correctly identifies the physical slot with its 696 silk screen label. This information is currently very 697 motherboard specific and determination of this information 698 must occur in userland at this time. 699 700 701System Logging 702-------------- 703 704If logging for UEs and CEs is enabled, then system logs will contain 705information indicating that errors have been detected:: 706 707 EDAC MC0: CE page 0x283, offset 0xce0, grain 8, syndrome 0x6ec3, row 0, channel 1 "DIMM_B1": amd76x_edac 708 EDAC MC0: CE page 0x1e5, offset 0xfb0, grain 8, syndrome 0xb741, row 0, channel 1 "DIMM_B1": amd76x_edac 709 710 711The structure of the message is: 712 713 +---------------------------------------+-------------+ 714 | Content | Example | 715 +=======================================+=============+ 716 | The memory controller | MC0 | 717 +---------------------------------------+-------------+ 718 | Error type | CE | 719 +---------------------------------------+-------------+ 720 | Memory page | 0x283 | 721 +---------------------------------------+-------------+ 722 | Offset in the page | 0xce0 | 723 +---------------------------------------+-------------+ 724 | The byte granularity | grain 8 | 725 | or resolution of the error | | 726 +---------------------------------------+-------------+ 727 | The error syndrome | 0xb741 | 728 +---------------------------------------+-------------+ 729 | Memory row | row 0 | 730 +---------------------------------------+-------------+ 731 | Memory channel | channel 1 | 732 +---------------------------------------+-------------+ 733 | DIMM label, if set prior | DIMM B1 | 734 +---------------------------------------+-------------+ 735 | And then an optional, driver-specific | | 736 | message that may have additional | | 737 | information. | | 738 +---------------------------------------+-------------+ 739 740Both UEs and CEs with no info will lack all but memory controller, error 741type, a notice of "no info" and then an optional, driver-specific error 742message. 743 744 745PCI Bus Parity Detection 746------------------------ 747 748On Header Type 00 devices, the primary status is looked at for any 749parity error regardless of whether parity is enabled on the device or 750not. (The spec indicates parity is generated in some cases). On Header 751Type 01 bridges, the secondary status register is also looked at to see 752if parity occurred on the bus on the other side of the bridge. 753 754 755Sysfs configuration 756------------------- 757 758Under ``/sys/devices/system/edac/pci`` are control and attribute files as 759follows: 760 761 762- ``check_pci_parity`` - Enable/Disable PCI Parity checking control file 763 764 This control file enables or disables the PCI Bus Parity scanning 765 operation. Writing a 1 to this file enables the scanning. Writing 766 a 0 to this file disables the scanning. 767 768 Enable:: 769 770 echo "1" >/sys/devices/system/edac/pci/check_pci_parity 771 772 Disable:: 773 774 echo "0" >/sys/devices/system/edac/pci/check_pci_parity 775 776 777- ``pci_parity_count`` - Parity Count 778 779 This attribute file will display the number of parity errors that 780 have been detected. 781 782 783Module parameters 784----------------- 785 786- ``edac_mc_panic_on_ue`` - Panic on UE control file 787 788 An uncorrectable error will cause a machine panic. This is usually 789 desirable. It is a bad idea to continue when an uncorrectable error 790 occurs - it is indeterminate what was uncorrected and the operating 791 system context might be so mangled that continuing will lead to further 792 corruption. If the kernel has MCE configured, then EDAC will never 793 notice the UE. 794 795 LOAD TIME:: 796 797 module/kernel parameter: edac_mc_panic_on_ue=[0|1] 798 799 RUN TIME:: 800 801 echo "1" > /sys/module/edac_core/parameters/edac_mc_panic_on_ue 802 803 804- ``edac_mc_log_ue`` - Log UE control file 805 806 807 Generate kernel messages describing uncorrectable errors. These errors 808 are reported through the system message log system. UE statistics 809 will be accumulated even when UE logging is disabled. 810 811 LOAD TIME:: 812 813 module/kernel parameter: edac_mc_log_ue=[0|1] 814 815 RUN TIME:: 816 817 echo "1" > /sys/module/edac_core/parameters/edac_mc_log_ue 818 819 820- ``edac_mc_log_ce`` - Log CE control file 821 822 823 Generate kernel messages describing correctable errors. These 824 errors are reported through the system message log system. 825 CE statistics will be accumulated even when CE logging is disabled. 826 827 LOAD TIME:: 828 829 module/kernel parameter: edac_mc_log_ce=[0|1] 830 831 RUN TIME:: 832 833 echo "1" > /sys/module/edac_core/parameters/edac_mc_log_ce 834 835 836- ``edac_mc_poll_msec`` - Polling period control file 837 838 839 The time period, in milliseconds, for polling for error information. 840 Too small a value wastes resources. Too large a value might delay 841 necessary handling of errors and might loose valuable information for 842 locating the error. 1000 milliseconds (once each second) is the current 843 default. Systems which require all the bandwidth they can get, may 844 increase this. 845 846 LOAD TIME:: 847 848 module/kernel parameter: edac_mc_poll_msec=[0|1] 849 850 RUN TIME:: 851 852 echo "1000" > /sys/module/edac_core/parameters/edac_mc_poll_msec 853 854 855- ``panic_on_pci_parity`` - Panic on PCI PARITY Error 856 857 858 This control file enables or disables panicking when a parity 859 error has been detected. 860 861 862 module/kernel parameter:: 863 864 edac_panic_on_pci_pe=[0|1] 865 866 Enable:: 867 868 echo "1" > /sys/module/edac_core/parameters/edac_panic_on_pci_pe 869 870 Disable:: 871 872 echo "0" > /sys/module/edac_core/parameters/edac_panic_on_pci_pe 873 874 875 876EDAC device type 877---------------- 878 879In the header file, edac_pci.h, there is a series of edac_device structures 880and APIs for the EDAC_DEVICE. 881 882User space access to an edac_device is through the sysfs interface. 883 884At the location ``/sys/devices/system/edac`` (sysfs) new edac_device devices 885will appear. 886 887There is a three level tree beneath the above ``edac`` directory. For example, 888the ``test_device_edac`` device (found at the http://bluesmoke.sourceforget.net 889website) installs itself as:: 890 891 /sys/devices/system/edac/test-instance 892 893in this directory are various controls, a symlink and one or more ``instance`` 894directories. 895 896The standard default controls are: 897 898 ============== ======================================================= 899 log_ce boolean to log CE events 900 log_ue boolean to log UE events 901 panic_on_ue boolean to ``panic`` the system if an UE is encountered 902 (default off, can be set true via startup script) 903 poll_msec time period between POLL cycles for events 904 ============== ======================================================= 905 906The test_device_edac device adds at least one of its own custom control: 907 908 ============== ================================================== 909 test_bits which in the current test driver does nothing but 910 show how it is installed. A ported driver can 911 add one or more such controls and/or attributes 912 for specific uses. 913 One out-of-tree driver uses controls here to allow 914 for ERROR INJECTION operations to hardware 915 injection registers 916 ============== ================================================== 917 918The symlink points to the 'struct dev' that is registered for this edac_device. 919 920Instances 921--------- 922 923One or more instance directories are present. For the ``test_device_edac`` 924case: 925 926 +----------------+ 927 | test-instance0 | 928 +----------------+ 929 930 931In this directory there are two default counter attributes, which are totals of 932counter in deeper subdirectories. 933 934 ============== ==================================== 935 ce_count total of CE events of subdirectories 936 ue_count total of UE events of subdirectories 937 ============== ==================================== 938 939Blocks 940------ 941 942At the lowest directory level is the ``block`` directory. There can be 0, 1 943or more blocks specified in each instance: 944 945 +-------------+ 946 | test-block0 | 947 +-------------+ 948 949In this directory the default attributes are: 950 951 ============== ================================================ 952 ce_count which is counter of CE events for this ``block`` 953 of hardware being monitored 954 ue_count which is counter of UE events for this ``block`` 955 of hardware being monitored 956 ============== ================================================ 957 958 959The ``test_device_edac`` device adds 4 attributes and 1 control: 960 961 ================== ==================================================== 962 test-block-bits-0 for every POLL cycle this counter 963 is incremented 964 test-block-bits-1 every 10 cycles, this counter is bumped once, 965 and test-block-bits-0 is set to 0 966 test-block-bits-2 every 100 cycles, this counter is bumped once, 967 and test-block-bits-1 is set to 0 968 test-block-bits-3 every 1000 cycles, this counter is bumped once, 969 and test-block-bits-2 is set to 0 970 ================== ==================================================== 971 972 973 ================== ==================================================== 974 reset-counters writing ANY thing to this control will 975 reset all the above counters. 976 ================== ==================================================== 977 978 979Use of the ``test_device_edac`` driver should enable any others to create their own 980unique drivers for their hardware systems. 981 982The ``test_device_edac`` sample driver is located at the 983http://bluesmoke.sourceforge.net project site for EDAC. 984 985 986Usage of EDAC APIs on Nehalem and newer Intel CPUs 987-------------------------------------------------- 988 989On older Intel architectures, the memory controller was part of the North 990Bridge chipset. Nehalem, Sandy Bridge, Ivy Bridge, Haswell, Sky Lake and 991newer Intel architectures integrated an enhanced version of the memory 992controller (MC) inside the CPUs. 993 994This chapter will cover the differences of the enhanced memory controllers 995found on newer Intel CPUs, such as ``i7core_edac``, ``sb_edac`` and 996``sbx_edac`` drivers. 997 998.. note:: 999 1000 The Xeon E7 processor families use a separate chip for the memory 1001 controller, called Intel Scalable Memory Buffer. This section doesn't 1002 apply for such families. 1003 10041) There is one Memory Controller per Quick Patch Interconnect 1005 (QPI). At the driver, the term "socket" means one QPI. This is 1006 associated with a physical CPU socket. 1007 1008 Each MC have 3 physical read channels, 3 physical write channels and 1009 3 logic channels. The driver currently sees it as just 3 channels. 1010 Each channel can have up to 3 DIMMs. 1011 1012 The minimum known unity is DIMMs. There are no information about csrows. 1013 As EDAC API maps the minimum unity is csrows, the driver sequentially 1014 maps channel/DIMM into different csrows. 1015 1016 For example, supposing the following layout:: 1017 1018 Ch0 phy rd0, wr0 (0x063f4031): 2 ranks, UDIMMs 1019 dimm 0 1024 Mb offset: 0, bank: 8, rank: 1, row: 0x4000, col: 0x400 1020 dimm 1 1024 Mb offset: 4, bank: 8, rank: 1, row: 0x4000, col: 0x400 1021 Ch1 phy rd1, wr1 (0x063f4031): 2 ranks, UDIMMs 1022 dimm 0 1024 Mb offset: 0, bank: 8, rank: 1, row: 0x4000, col: 0x400 1023 Ch2 phy rd3, wr3 (0x063f4031): 2 ranks, UDIMMs 1024 dimm 0 1024 Mb offset: 0, bank: 8, rank: 1, row: 0x4000, col: 0x400 1025 1026 The driver will map it as:: 1027 1028 csrow0: channel 0, dimm0 1029 csrow1: channel 0, dimm1 1030 csrow2: channel 1, dimm0 1031 csrow3: channel 2, dimm0 1032 1033 exports one DIMM per csrow. 1034 1035 Each QPI is exported as a different memory controller. 1036 10372) The MC has the ability to inject errors to test drivers. The drivers 1038 implement this functionality via some error injection nodes: 1039 1040 For injecting a memory error, there are some sysfs nodes, under 1041 ``/sys/devices/system/edac/mc/mc?/``: 1042 1043 - ``inject_addrmatch/*``: 1044 Controls the error injection mask register. It is possible to specify 1045 several characteristics of the address to match an error code:: 1046 1047 dimm = the affected dimm. Numbers are relative to a channel; 1048 rank = the memory rank; 1049 channel = the channel that will generate an error; 1050 bank = the affected bank; 1051 page = the page address; 1052 column (or col) = the address column. 1053 1054 each of the above values can be set to "any" to match any valid value. 1055 1056 At driver init, all values are set to any. 1057 1058 For example, to generate an error at rank 1 of dimm 2, for any channel, 1059 any bank, any page, any column:: 1060 1061 echo 2 >/sys/devices/system/edac/mc/mc0/inject_addrmatch/dimm 1062 echo 1 >/sys/devices/system/edac/mc/mc0/inject_addrmatch/rank 1063 1064 To return to the default behaviour of matching any, you can do:: 1065 1066 echo any >/sys/devices/system/edac/mc/mc0/inject_addrmatch/dimm 1067 echo any >/sys/devices/system/edac/mc/mc0/inject_addrmatch/rank 1068 1069 - ``inject_eccmask``: 1070 specifies what bits will have troubles, 1071 1072 - ``inject_section``: 1073 specifies what ECC cache section will get the error:: 1074 1075 3 for both 1076 2 for the highest 1077 1 for the lowest 1078 1079 - ``inject_type``: 1080 specifies the type of error, being a combination of the following bits:: 1081 1082 bit 0 - repeat 1083 bit 1 - ecc 1084 bit 2 - parity 1085 1086 - ``inject_enable``: 1087 starts the error generation when something different than 0 is written. 1088 1089 All inject vars can be read. root permission is needed for write. 1090 1091 Datasheet states that the error will only be generated after a write on an 1092 address that matches inject_addrmatch. It seems, however, that reading will 1093 also produce an error. 1094 1095 For example, the following code will generate an error for any write access 1096 at socket 0, on any DIMM/address on channel 2:: 1097 1098 echo 2 >/sys/devices/system/edac/mc/mc0/inject_addrmatch/channel 1099 echo 2 >/sys/devices/system/edac/mc/mc0/inject_type 1100 echo 64 >/sys/devices/system/edac/mc/mc0/inject_eccmask 1101 echo 3 >/sys/devices/system/edac/mc/mc0/inject_section 1102 echo 1 >/sys/devices/system/edac/mc/mc0/inject_enable 1103 dd if=/dev/mem of=/dev/null seek=16k bs=4k count=1 >& /dev/null 1104 1105 For socket 1, it is needed to replace "mc0" by "mc1" at the above 1106 commands. 1107 1108 The generated error message will look like:: 1109 1110 EDAC MC0: UE row 0, channel-a= 0 channel-b= 0 labels "-": NON_FATAL (addr = 0x0075b980, socket=0, Dimm=0, Channel=2, syndrome=0x00000040, count=1, Err=8c0000400001009f:4000080482 (read error: read ECC error)) 1111 11123) Corrected Error memory register counters 1113 1114 Those newer MCs have some registers to count memory errors. The driver 1115 uses those registers to report Corrected Errors on devices with Registered 1116 DIMMs. 1117 1118 However, those counters don't work with Unregistered DIMM. As the chipset 1119 offers some counters that also work with UDIMMs (but with a worse level of 1120 granularity than the default ones), the driver exposes those registers for 1121 UDIMM memories. 1122 1123 They can be read by looking at the contents of ``all_channel_counts/``:: 1124 1125 $ for i in /sys/devices/system/edac/mc/mc0/all_channel_counts/*; do echo $i; cat $i; done 1126 /sys/devices/system/edac/mc/mc0/all_channel_counts/udimm0 1127 0 1128 /sys/devices/system/edac/mc/mc0/all_channel_counts/udimm1 1129 0 1130 /sys/devices/system/edac/mc/mc0/all_channel_counts/udimm2 1131 0 1132 1133 What happens here is that errors on different csrows, but at the same 1134 dimm number will increment the same counter. 1135 So, in this memory mapping:: 1136 1137 csrow0: channel 0, dimm0 1138 csrow1: channel 0, dimm1 1139 csrow2: channel 1, dimm0 1140 csrow3: channel 2, dimm0 1141 1142 The hardware will increment udimm0 for an error at the first dimm at either 1143 csrow0, csrow2 or csrow3; 1144 1145 The hardware will increment udimm1 for an error at the second dimm at either 1146 csrow0, csrow2 or csrow3; 1147 1148 The hardware will increment udimm2 for an error at the third dimm at either 1149 csrow0, csrow2 or csrow3; 1150 11514) Standard error counters 1152 1153 The standard error counters are generated when an mcelog error is received 1154 by the driver. Since, with UDIMM, this is counted by software, it is 1155 possible that some errors could be lost. With RDIMM's, they display the 1156 contents of the registers 1157 1158Reference documents used on ``amd64_edac`` 1159------------------------------------------ 1160 1161``amd64_edac`` module is based on the following documents 1162(available from http://support.amd.com/en-us/search/tech-docs): 1163 11641. :Title: BIOS and Kernel Developer's Guide for AMD Athlon 64 and AMD 1165 Opteron Processors 1166 :AMD publication #: 26094 1167 :Revision: 3.26 1168 :Link: http://support.amd.com/TechDocs/26094.PDF 1169 11702. :Title: BIOS and Kernel Developer's Guide for AMD NPT Family 0Fh 1171 Processors 1172 :AMD publication #: 32559 1173 :Revision: 3.00 1174 :Issue Date: May 2006 1175 :Link: http://support.amd.com/TechDocs/32559.pdf 1176 11773. :Title: BIOS and Kernel Developer's Guide (BKDG) For AMD Family 10h 1178 Processors 1179 :AMD publication #: 31116 1180 :Revision: 3.00 1181 :Issue Date: September 07, 2007 1182 :Link: http://support.amd.com/TechDocs/31116.pdf 1183 11844. :Title: BIOS and Kernel Developer's Guide (BKDG) for AMD Family 15h 1185 Models 30h-3Fh Processors 1186 :AMD publication #: 49125 1187 :Revision: 3.06 1188 :Issue Date: 2/12/2015 (latest release) 1189 :Link: http://support.amd.com/TechDocs/49125_15h_Models_30h-3Fh_BKDG.pdf 1190 11915. :Title: BIOS and Kernel Developer's Guide (BKDG) for AMD Family 15h 1192 Models 60h-6Fh Processors 1193 :AMD publication #: 50742 1194 :Revision: 3.01 1195 :Issue Date: 7/23/2015 (latest release) 1196 :Link: http://support.amd.com/TechDocs/50742_15h_Models_60h-6Fh_BKDG.pdf 1197 11986. :Title: BIOS and Kernel Developer's Guide (BKDG) for AMD Family 16h 1199 Models 00h-0Fh Processors 1200 :AMD publication #: 48751 1201 :Revision: 3.03 1202 :Issue Date: 2/23/2015 (latest release) 1203 :Link: http://support.amd.com/TechDocs/48751_16h_bkdg.pdf 1204 1205Credits 1206======= 1207 1208* Written by Doug Thompson <dougthompson@xmission.com> 1209 1210 - 7 Dec 2005 1211 - 17 Jul 2007 Updated 1212 1213* |copy| Mauro Carvalho Chehab 1214 1215 - 05 Aug 2009 Nehalem interface 1216 - 26 Oct 2016 Converted to ReST and cleanups at the Nehalem section 1217 1218* EDAC authors/maintainers: 1219 1220 - Doug Thompson, Dave Jiang, Dave Peterson et al, 1221 - Mauro Carvalho Chehab 1222 - Borislav Petkov 1223 - original author: Thayne Harbaugh 1224