1====================== 2Firmware-Assisted Dump 3====================== 4 5July 2011 6 7The goal of firmware-assisted dump is to enable the dump of 8a crashed system, and to do so from a fully-reset system, and 9to minimize the total elapsed time until the system is back 10in production use. 11 12- Firmware-Assisted Dump (FADump) infrastructure is intended to replace 13 the existing phyp assisted dump. 14- Fadump uses the same firmware interfaces and memory reservation model 15 as phyp assisted dump. 16- Unlike phyp dump, FADump exports the memory dump through /proc/vmcore 17 in the ELF format in the same way as kdump. This helps us reuse the 18 kdump infrastructure for dump capture and filtering. 19- Unlike phyp dump, userspace tool does not need to refer any sysfs 20 interface while reading /proc/vmcore. 21- Unlike phyp dump, FADump allows user to release all the memory reserved 22 for dump, with a single operation of echo 1 > /sys/kernel/fadump_release_mem. 23- Once enabled through kernel boot parameter, FADump can be 24 started/stopped through /sys/kernel/fadump_registered interface (see 25 sysfs files section below) and can be easily integrated with kdump 26 service start/stop init scripts. 27 28Comparing with kdump or other strategies, firmware-assisted 29dump offers several strong, practical advantages: 30 31- Unlike kdump, the system has been reset, and loaded 32 with a fresh copy of the kernel. In particular, 33 PCI and I/O devices have been reinitialized and are 34 in a clean, consistent state. 35- Once the dump is copied out, the memory that held the dump 36 is immediately available to the running kernel. And therefore, 37 unlike kdump, FADump doesn't need a 2nd reboot to get back 38 the system to the production configuration. 39 40The above can only be accomplished by coordination with, 41and assistance from the Power firmware. The procedure is 42as follows: 43 44- The first kernel registers the sections of memory with the 45 Power firmware for dump preservation during OS initialization. 46 These registered sections of memory are reserved by the first 47 kernel during early boot. 48 49- When system crashes, the Power firmware will copy the registered 50 low memory regions (boot memory) from source to destination area. 51 It will also save hardware PTE's. 52 53 NOTE: 54 The term 'boot memory' means size of the low memory chunk 55 that is required for a kernel to boot successfully when 56 booted with restricted memory. By default, the boot memory 57 size will be the larger of 5% of system RAM or 256MB. 58 Alternatively, user can also specify boot memory size 59 through boot parameter 'crashkernel=' which will override 60 the default calculated size. Use this option if default 61 boot memory size is not sufficient for second kernel to 62 boot successfully. For syntax of crashkernel= parameter, 63 refer to Documentation/admin-guide/kdump/kdump.rst. If any 64 offset is provided in crashkernel= parameter, it will be 65 ignored as FADump uses a predefined offset to reserve memory 66 for boot memory dump preservation in case of a crash. 67 68- After the low memory (boot memory) area has been saved, the 69 firmware will reset PCI and other hardware state. It will 70 *not* clear the RAM. It will then launch the bootloader, as 71 normal. 72 73- The freshly booted kernel will notice that there is a new node 74 (rtas/ibm,kernel-dump on pSeries or ibm,opal/dump/mpipl-boot 75 on OPAL platform) in the device tree, indicating that 76 there is crash data available from a previous boot. During 77 the early boot OS will reserve rest of the memory above 78 boot memory size effectively booting with restricted memory 79 size. This will make sure that this kernel (also, referred 80 to as second kernel or capture kernel) will not touch any 81 of the dump memory area. 82 83- User-space tools will read /proc/vmcore to obtain the contents 84 of memory, which holds the previous crashed kernel dump in ELF 85 format. The userspace tools may copy this info to disk, or 86 network, nas, san, iscsi, etc. as desired. 87 88- Once the userspace tool is done saving dump, it will echo 89 '1' to /sys/kernel/fadump_release_mem to release the reserved 90 memory back to general use, except the memory required for 91 next firmware-assisted dump registration. 92 93 e.g.:: 94 95 # echo 1 > /sys/kernel/fadump_release_mem 96 97Please note that the firmware-assisted dump feature 98is only available on POWER6 and above systems on pSeries 99(PowerVM) platform and POWER9 and above systems with OP940 100or later firmware versions on PowerNV (OPAL) platform. 101Note that, OPAL firmware exports ibm,opal/dump node when 102FADump is supported on PowerNV platform. 103 104On OPAL based machines, system first boots into an intermittent 105kernel (referred to as petitboot kernel) before booting into the 106capture kernel. This kernel would have minimal kernel and/or 107userspace support to process crash data. Such kernel needs to 108preserve previously crash'ed kernel's memory for the subsequent 109capture kernel boot to process this crash data. Kernel config 110option CONFIG_PRESERVE_FA_DUMP has to be enabled on such kernel 111to ensure that crash data is preserved to process later. 112 113-- On OPAL based machines (PowerNV), if the kernel is build with 114 CONFIG_OPAL_CORE=y, OPAL memory at the time of crash is also 115 exported as /sys/firmware/opal/mpipl/core file. This procfs file is 116 helpful in debugging OPAL crashes with GDB. The kernel memory 117 used for exporting this procfs file can be released by echo'ing 118 '1' to /sys/firmware/opal/mpipl/release_core node. 119 120 e.g. 121 # echo 1 > /sys/firmware/opal/mpipl/release_core 122 123Implementation details: 124----------------------- 125 126During boot, a check is made to see if firmware supports 127this feature on that particular machine. If it does, then 128we check to see if an active dump is waiting for us. If yes 129then everything but boot memory size of RAM is reserved during 130early boot (See Fig. 2). This area is released once we finish 131collecting the dump from user land scripts (e.g. kdump scripts) 132that are run. If there is dump data, then the 133/sys/kernel/fadump_release_mem file is created, and the reserved 134memory is held. 135 136If there is no waiting dump data, then only the memory required to 137hold CPU state, HPTE region, boot memory dump, and FADump header is 138usually reserved at an offset greater than boot memory size (see Fig. 1). 139This area is *not* released: this region will be kept permanently 140reserved, so that it can act as a receptacle for a copy of the boot 141memory content in addition to CPU state and HPTE region, in the case 142a crash does occur. 143 144Since this reserved memory area is used only after the system crash, 145there is no point in blocking this significant chunk of memory from 146production kernel. Hence, the implementation uses the Linux kernel's 147Contiguous Memory Allocator (CMA) for memory reservation if CMA is 148configured for kernel. With CMA reservation this memory will be 149available for applications to use it, while kernel is prevented from 150using it. With this FADump will still be able to capture all of the 151kernel memory and most of the user space memory except the user pages 152that were present in CMA region:: 153 154 o Memory Reservation during first kernel 155 156 Low memory Top of memory 157 0 boot memory size |<------ Reserved dump area ----->| | 158 | | | Permanent Reservation | | 159 V V | | V 160 +-----------+-----/ /---+---+----+-----------+-------+----+-----+ 161 | | |///|////| DUMP | HDR |////| | 162 +-----------+-----/ /---+---+----+-----------+-------+----+-----+ 163 | ^ ^ ^ ^ ^ 164 | | | | | | 165 \ CPU HPTE / | | 166 -------------------------------- | | 167 Boot memory content gets transferred | | 168 to reserved area by firmware at the | | 169 time of crash. | | 170 FADump Header | 171 (meta area) | 172 | 173 | 174 Metadata: This area holds a metadata structure whose 175 address is registered with f/w and retrieved in the 176 second kernel after crash, on platforms that support 177 tags (OPAL). Having such structure with info needed 178 to process the crashdump eases dump capture process. 179 180 Fig. 1 181 182 183 o Memory Reservation during second kernel after crash 184 185 Low memory Top of memory 186 0 boot memory size | 187 | |<------------ Crash preserved area ------------>| 188 V V |<--- Reserved dump area --->| | 189 +----+---+--+-----/ /---+---+----+-------+-----+-----+-------+ 190 | |ELF| | |///|////| DUMP | HDR |/////| | 191 +----+---+--+-----/ /---+---+----+-------+-----+-----+-------+ 192 | | | | | | 193 ----- ------------------------------ --------------- 194 \ | | 195 \ | | 196 \ | | 197 \ | ---------------------------- 198 \ | / 199 \ | / 200 \ | / 201 /proc/vmcore 202 203 204 +---+ 205 |///| -> Regions (CPU, HPTE & Metadata) marked like this in the above 206 +---+ figures are not always present. For example, OPAL platform 207 does not have CPU & HPTE regions while Metadata region is 208 not supported on pSeries currently. 209 210 +---+ 211 |ELF| -> elfcorehdr, it is created in second kernel after crash. 212 +---+ 213 214 Note: Memory from 0 to the boot memory size is used by second kernel 215 216 Fig. 2 217 218 219Currently the dump will be copied from /proc/vmcore to a new file upon 220user intervention. The dump data available through /proc/vmcore will be 221in ELF format. Hence the existing kdump infrastructure (kdump scripts) 222to save the dump works fine with minor modifications. KDump scripts on 223major Distro releases have already been modified to work seamlessly (no 224user intervention in saving the dump) when FADump is used, instead of 225KDump, as dump mechanism. 226 227The tools to examine the dump will be same as the ones 228used for kdump. 229 230How to enable firmware-assisted dump (FADump): 231---------------------------------------------- 232 2331. Set config option CONFIG_FA_DUMP=y and build kernel. 2342. Boot into linux kernel with 'fadump=on' kernel cmdline option. 235 By default, FADump reserved memory will be initialized as CMA area. 236 Alternatively, user can boot linux kernel with 'fadump=nocma' to 237 prevent FADump to use CMA. 2383. Optionally, user can also set 'crashkernel=' kernel cmdline 239 to specify size of the memory to reserve for boot memory dump 240 preservation. 241 242NOTE: 243 1. 'fadump_reserve_mem=' parameter has been deprecated. Instead 244 use 'crashkernel=' to specify size of the memory to reserve 245 for boot memory dump preservation. 246 2. If firmware-assisted dump fails to reserve memory then it 247 will fallback to existing kdump mechanism if 'crashkernel=' 248 option is set at kernel cmdline. 249 3. if user wants to capture all of user space memory and ok with 250 reserved memory not available to production system, then 251 'fadump=nocma' kernel parameter can be used to fallback to 252 old behaviour. 253 254Sysfs/debugfs files: 255-------------------- 256 257Firmware-assisted dump feature uses sysfs file system to hold 258the control files and debugfs file to display memory reserved region. 259 260Here is the list of files under kernel sysfs: 261 262 /sys/kernel/fadump_enabled 263 This is used to display the FADump status. 264 265 - 0 = FADump is disabled 266 - 1 = FADump is enabled 267 268 This interface can be used by kdump init scripts to identify if 269 FADump is enabled in the kernel and act accordingly. 270 271 /sys/kernel/fadump_registered 272 This is used to display the FADump registration status as well 273 as to control (start/stop) the FADump registration. 274 275 - 0 = FADump is not registered. 276 - 1 = FADump is registered and ready to handle system crash. 277 278 To register FADump echo 1 > /sys/kernel/fadump_registered and 279 echo 0 > /sys/kernel/fadump_registered for un-register and stop the 280 FADump. Once the FADump is un-registered, the system crash will not 281 be handled and vmcore will not be captured. This interface can be 282 easily integrated with kdump service start/stop. 283 284 /sys/kernel/fadump/mem_reserved 285 286 This is used to display the memory reserved by FADump for saving the 287 crash dump. 288 289 /sys/kernel/fadump_release_mem 290 This file is available only when FADump is active during 291 second kernel. This is used to release the reserved memory 292 region that are held for saving crash dump. To release the 293 reserved memory echo 1 to it:: 294 295 echo 1 > /sys/kernel/fadump_release_mem 296 297 After echo 1, the content of the /sys/kernel/debug/powerpc/fadump_region 298 file will change to reflect the new memory reservations. 299 300 The existing userspace tools (kdump infrastructure) can be easily 301 enhanced to use this interface to release the memory reserved for 302 dump and continue without 2nd reboot. 303 304Note: /sys/kernel/fadump_release_opalcore sysfs has moved to 305 /sys/firmware/opal/mpipl/release_core 306 307 /sys/firmware/opal/mpipl/release_core 308 309 This file is available only on OPAL based machines when FADump is 310 active during capture kernel. This is used to release the memory 311 used by the kernel to export /sys/firmware/opal/mpipl/core file. To 312 release this memory, echo '1' to it: 313 314 echo 1 > /sys/firmware/opal/mpipl/release_core 315 316Note: The following FADump sysfs files are deprecated. 317 318+----------------------------------+--------------------------------+ 319| Deprecated | Alternative | 320+----------------------------------+--------------------------------+ 321| /sys/kernel/fadump_enabled | /sys/kernel/fadump/enabled | 322+----------------------------------+--------------------------------+ 323| /sys/kernel/fadump_registered | /sys/kernel/fadump/registered | 324+----------------------------------+--------------------------------+ 325| /sys/kernel/fadump_release_mem | /sys/kernel/fadump/release_mem | 326+----------------------------------+--------------------------------+ 327 328Here is the list of files under powerpc debugfs: 329(Assuming debugfs is mounted on /sys/kernel/debug directory.) 330 331 /sys/kernel/debug/powerpc/fadump_region 332 This file shows the reserved memory regions if FADump is 333 enabled otherwise this file is empty. The output format 334 is:: 335 336 <region>: [<start>-<end>] <reserved-size> bytes, Dumped: <dump-size> 337 338 and for kernel DUMP region is: 339 340 DUMP: Src: <src-addr>, Dest: <dest-addr>, Size: <size>, Dumped: # bytes 341 342 e.g. 343 Contents when FADump is registered during first kernel:: 344 345 # cat /sys/kernel/debug/powerpc/fadump_region 346 CPU : [0x0000006ffb0000-0x0000006fff001f] 0x40020 bytes, Dumped: 0x0 347 HPTE: [0x0000006fff0020-0x0000006fff101f] 0x1000 bytes, Dumped: 0x0 348 DUMP: [0x0000006fff1020-0x0000007fff101f] 0x10000000 bytes, Dumped: 0x0 349 350 Contents when FADump is active during second kernel:: 351 352 # cat /sys/kernel/debug/powerpc/fadump_region 353 CPU : [0x0000006ffb0000-0x0000006fff001f] 0x40020 bytes, Dumped: 0x40020 354 HPTE: [0x0000006fff0020-0x0000006fff101f] 0x1000 bytes, Dumped: 0x1000 355 DUMP: [0x0000006fff1020-0x0000007fff101f] 0x10000000 bytes, Dumped: 0x10000000 356 : [0x00000010000000-0x0000006ffaffff] 0x5ffb0000 bytes, Dumped: 0x5ffb0000 357 358 359NOTE: 360 Please refer to Documentation/filesystems/debugfs.rst on 361 how to mount the debugfs filesystem. 362 363 364TODO: 365----- 366 - Need to come up with the better approach to find out more 367 accurate boot memory size that is required for a kernel to 368 boot successfully when booted with restricted memory. 369 370Author: Mahesh Salgaonkar <mahesh@linux.vnet.ibm.com> 371 372This document is based on the original documentation written for phyp 373 374assisted dump by Linas Vepstas and Manish Ahuja. 375