]>
Commit | Line | Data |
---|---|---|
dc851a0f DW |
1 | ================================================================ |
2 | Documentation for Kdump - The kexec-based Crash Dumping Solution | |
b089f4a6 VG |
3 | ================================================================ |
4 | ||
dc851a0f DW |
5 | This document includes overview, setup and installation, and analysis |
6 | information. | |
b089f4a6 | 7 | |
dc851a0f DW |
8 | Overview |
9 | ======== | |
b089f4a6 | 10 | |
dc851a0f DW |
11 | Kdump uses kexec to quickly boot to a dump-capture kernel whenever a |
12 | dump of the system kernel's memory needs to be taken (for example, when | |
13 | the system panics). The system kernel's memory image is preserved across | |
14 | the reboot and is accessible to the dump-capture kernel. | |
b089f4a6 | 15 | |
dc851a0f DW |
16 | You can use common Linux commands, such as cp and scp, to copy the |
17 | memory image to a dump file on the local disk, or across the network to | |
18 | a remote system. | |
b089f4a6 | 19 | |
9c61a446 | 20 | Kdump and kexec are currently supported on the x86, x86_64, ppc64 and IA64 |
dc851a0f | 21 | architectures. |
b089f4a6 | 22 | |
dc851a0f DW |
23 | When the system kernel boots, it reserves a small section of memory for |
24 | the dump-capture kernel. This ensures that ongoing Direct Memory Access | |
25 | (DMA) from the system kernel does not corrupt the dump-capture kernel. | |
26 | The kexec -p command loads the dump-capture kernel into this reserved | |
27 | memory. | |
b089f4a6 | 28 | |
dc851a0f DW |
29 | On x86 machines, the first 640 KB of physical memory is needed to boot, |
30 | regardless of where the kernel loads. Therefore, kexec backs up this | |
31 | region just before rebooting into the dump-capture kernel. | |
b089f4a6 | 32 | |
dc851a0f DW |
33 | All of the necessary information about the system kernel's core image is |
34 | encoded in the ELF format, and stored in a reserved area of memory | |
35 | before a crash. The physical address of the start of the ELF header is | |
36 | passed to the dump-capture kernel through the elfcorehdr= boot | |
37 | parameter. | |
38 | ||
39 | With the dump-capture kernel, you can access the memory image, or "old | |
40 | memory," in two ways: | |
41 | ||
42 | - Through a /dev/oldmem device interface. A capture utility can read the | |
43 | device file and write out the memory in raw format. This is a raw dump | |
44 | of memory. Analysis and capture tools must be intelligent enough to | |
45 | determine where to look for the right information. | |
46 | ||
47 | - Through /proc/vmcore. This exports the dump as an ELF-format file that | |
48 | you can write out using file copy commands such as cp or scp. Further, | |
49 | you can use analysis tools such as the GNU Debugger (GDB) and the Crash | |
50 | tool to debug the dump file. This method ensures that the dump pages are | |
51 | correctly ordered. | |
52 | ||
53 | ||
54 | Setup and Installation | |
55 | ====================== | |
56 | ||
9c61a446 VG |
57 | Install kexec-tools |
58 | ------------------- | |
dc851a0f DW |
59 | |
60 | 1) Login as the root user. | |
61 | ||
62 | 2) Download the kexec-tools user-space package from the following URL: | |
63 | ||
9c61a446 | 64 | http://www.kernel.org/pub/linux/kernel/people/horms/kexec-tools/kexec-tools-testing-20061214.tar.gz |
dc851a0f | 65 | |
9c61a446 | 66 | Note: Latest kexec-tools-testing git tree is available at |
dc851a0f | 67 | |
9c61a446 VG |
68 | git://git.kernel.org/pub/scm/linux/kernel/git/horms/kexec-tools-testing.git |
69 | or | |
70 | http://www.kernel.org/git/?p=linux/kernel/git/horms/kexec-tools-testing.git;a=summary | |
dc851a0f | 71 | |
9c61a446 | 72 | 3) Unpack the tarball with the tar command, as follows: |
dc851a0f | 73 | |
9c61a446 | 74 | tar xvpzf kexec-tools-testing-20061214.tar.gz |
dc851a0f | 75 | |
9c61a446 | 76 | 4) Change to the kexec-tools-1.101 directory, as follows: |
dc851a0f | 77 | |
9c61a446 | 78 | cd kexec-tools-testing-20061214 |
dc851a0f | 79 | |
9c61a446 | 80 | 5) Configure the package, as follows: |
dc851a0f DW |
81 | |
82 | ./configure | |
83 | ||
9c61a446 | 84 | 6) Compile the package, as follows: |
dc851a0f DW |
85 | |
86 | make | |
87 | ||
9c61a446 | 88 | 7) Install the package, as follows: |
dc851a0f DW |
89 | |
90 | make install | |
91 | ||
92 | ||
9c61a446 VG |
93 | Build the system and dump-capture kernels |
94 | ----------------------------------------- | |
95 | There are two possible methods of using Kdump. | |
96 | ||
97 | 1) Build a separate custom dump-capture kernel for capturing the | |
98 | kernel core dump. | |
99 | ||
100 | 2) Or use the system kernel binary itself as dump-capture kernel and there is | |
101 | no need to build a separate dump-capture kernel. This is possible | |
102 | only with the architecutres which support a relocatable kernel. As | |
103 | of today i386 and ia64 architectures support relocatable kernel. | |
104 | ||
105 | Building a relocatable kernel is advantageous from the point of view that | |
106 | one does not have to build a second kernel for capturing the dump. But | |
107 | at the same time one might want to build a custom dump capture kernel | |
108 | suitable to his needs. | |
dc851a0f | 109 | |
9c61a446 VG |
110 | Following are the configuration setting required for system and |
111 | dump-capture kernels for enabling kdump support. | |
dc851a0f | 112 | |
9c61a446 VG |
113 | System kernel config options |
114 | ---------------------------- | |
dc851a0f DW |
115 | |
116 | 1) Enable "kexec system call" in "Processor type and features." | |
117 | ||
118 | CONFIG_KEXEC=y | |
119 | ||
120 | 2) Enable "sysfs file system support" in "Filesystem" -> "Pseudo | |
121 | filesystems." This is usually enabled by default. | |
122 | ||
123 | CONFIG_SYSFS=y | |
124 | ||
125 | Note that "sysfs file system support" might not appear in the "Pseudo | |
126 | filesystems" menu if "Configure standard kernel features (for small | |
127 | systems)" is not enabled in "General Setup." In this case, check the | |
128 | .config file itself to ensure that sysfs is turned on, as follows: | |
129 | ||
130 | grep 'CONFIG_SYSFS' .config | |
131 | ||
132 | 3) Enable "Compile the kernel with debug info" in "Kernel hacking." | |
133 | ||
134 | CONFIG_DEBUG_INFO=Y | |
135 | ||
136 | This causes the kernel to be built with debug symbols. The dump | |
137 | analysis tools require a vmlinux with debug symbols in order to read | |
138 | and analyze a dump file. | |
139 | ||
9c61a446 VG |
140 | Dump-capture kernel config options (Arch Independent) |
141 | ----------------------------------------------------- | |
dc851a0f | 142 | |
9c61a446 VG |
143 | 1) Enable "kernel crash dumps" support under "Processor type and |
144 | features": | |
dc851a0f | 145 | |
9c61a446 | 146 | CONFIG_CRASH_DUMP=y |
dc851a0f | 147 | |
9c61a446 | 148 | 2) Enable "/proc/vmcore support" under "Filesystems" -> "Pseudo filesystems". |
b089f4a6 | 149 | |
9c61a446 VG |
150 | CONFIG_PROC_VMCORE=y |
151 | (CONFIG_PROC_VMCORE is set by default when CONFIG_CRASH_DUMP is selected.) | |
dc851a0f | 152 | |
9c61a446 VG |
153 | Dump-capture kernel config options (Arch Dependent, i386) |
154 | -------------------------------------------------------- | |
155 | 1) On x86, enable high memory support under "Processor type and | |
dc851a0f DW |
156 | features": |
157 | ||
158 | CONFIG_HIGHMEM64G=y | |
159 | or | |
160 | CONFIG_HIGHMEM4G | |
161 | ||
9c61a446 | 162 | 2) On x86 and x86_64, disable symmetric multi-processing support |
dc851a0f DW |
163 | under "Processor type and features": |
164 | ||
165 | CONFIG_SMP=n | |
9c61a446 | 166 | |
dc851a0f DW |
167 | (If CONFIG_SMP=y, then specify maxcpus=1 on the kernel command line |
168 | when loading the dump-capture kernel, see section "Load the Dump-capture | |
169 | Kernel".) | |
170 | ||
9c61a446 VG |
171 | 3) If one wants to build and use a relocatable kernel, |
172 | Enable "Build a relocatable kernel" support under "Processor type and | |
173 | features" | |
dc851a0f | 174 | |
9c61a446 | 175 | CONFIG_RELOCATABLE=y |
dc851a0f | 176 | |
9c61a446 VG |
177 | 4) Use a suitable value for "Physical address where the kernel is |
178 | loaded" (under "Processor type and features"). This only appears when | |
179 | "kernel crash dumps" is enabled. A suitable value depends upon | |
180 | whether kernel is relocatable or not. | |
181 | ||
182 | If you are using a relocatable kernel use CONFIG_PHYSICAL_START=0x100000 | |
183 | This will compile the kernel for physical address 1MB, but given the fact | |
184 | kernel is relocatable, it can be run from any physical address hence | |
185 | kexec boot loader will load it in memory region reserved for dump-capture | |
186 | kernel. | |
187 | ||
188 | Otherwise it should be the start of memory region reserved for | |
189 | second kernel using boot parameter "crashkernel=Y@X". Here X is | |
190 | start of memory region reserved for dump-capture kernel. | |
191 | Generally X is 16MB (0x1000000). So you can set | |
192 | CONFIG_PHYSICAL_START=0x1000000 | |
193 | ||
194 | 5) Make and install the kernel and its modules. DO NOT add this kernel | |
195 | to the boot loader configuration files. | |
dc851a0f | 196 | |
9c61a446 VG |
197 | Dump-capture kernel config options (Arch Dependent, x86_64) |
198 | ---------------------------------------------------------- | |
199 | 1) On x86 and x86_64, disable symmetric multi-processing support | |
200 | under "Processor type and features": | |
201 | ||
202 | CONFIG_SMP=n | |
203 | ||
204 | (If CONFIG_SMP=y, then specify maxcpus=1 on the kernel command line | |
205 | when loading the dump-capture kernel, see section "Load the Dump-capture | |
206 | Kernel".) | |
dc851a0f | 207 | |
9c61a446 | 208 | 2) Use a suitable value for "Physical address where the kernel is |
dc851a0f DW |
209 | loaded" (under "Processor type and features"). This only appears when |
210 | "kernel crash dumps" is enabled. By default this value is 0x1000000 | |
211 | (16MB). It should be the same as X in the "crashkernel=Y@X" boot | |
9c61a446 | 212 | parameter. |
dc851a0f | 213 | |
9c61a446 | 214 | For x86_64, normally "CONFIG_PHYSICAL_START=0x1000000". |
dc851a0f | 215 | |
9c61a446 VG |
216 | 3) Make and install the kernel and its modules. DO NOT add this kernel |
217 | to the boot loader configuration files. | |
dc851a0f | 218 | |
9c61a446 VG |
219 | Dump-capture kernel config options (Arch Dependent, ppc64) |
220 | ---------------------------------------------------------- | |
dc851a0f | 221 | |
9c61a446 | 222 | - Make and install the kernel and its modules. DO NOT add this kernel |
dc851a0f DW |
223 | to the boot loader configuration files. |
224 | ||
9c61a446 VG |
225 | Dump-capture kernel config options (Arch Dependent, ia64) |
226 | ---------------------------------------------------------- | |
227 | (To be filled) | |
228 | ||
229 | ||
230 | Boot into System Kernel | |
231 | ======================= | |
232 | ||
233 | 1) Make and install the kernel and its modules. Update the boot loader | |
234 | (such as grub, yaboot, or lilo) configuration files as necessary. | |
235 | ||
236 | 2) Boot the system kernel with the boot parameter "crashkernel=Y@X", | |
237 | where Y specifies how much memory to reserve for the dump-capture kernel | |
238 | and X specifies the beginning of this reserved memory. For example, | |
239 | "crashkernel=64M@16M" tells the system kernel to reserve 64 MB of memory | |
240 | starting at physical address 0x01000000 (16MB) for the dump-capture kernel. | |
241 | ||
242 | On x86 and x86_64, use "crashkernel=64M@16M". | |
243 | ||
244 | On ppc64, use "crashkernel=128M@32M". | |
dc851a0f DW |
245 | |
246 | Load the Dump-capture Kernel | |
247 | ============================ | |
248 | ||
9c61a446 VG |
249 | After booting to the system kernel, dump-capture kernel needs to be |
250 | loaded. | |
251 | ||
252 | Based on the architecture and type of image (relocatable or not), one | |
253 | can choose to load the uncompressed vmlinux or compressed bzImage/vmlinuz | |
254 | of dump-capture kernel. Following is the summary. | |
255 | ||
256 | For i386: | |
257 | - Use vmlinux if kernel is not relocatable. | |
258 | - Use bzImage/vmlinuz if kernel is relocatable. | |
259 | For x86_64: | |
260 | - Use vmlinux | |
261 | For ppc64: | |
262 | - Use vmlinux | |
263 | For ia64: | |
264 | (To be filled) | |
265 | ||
266 | If you are using a uncompressed vmlinux image then use following command | |
267 | to load dump-capture kernel. | |
dc851a0f | 268 | |
9c61a446 | 269 | kexec -p <dump-capture-kernel-vmlinux-image> \ |
dc851a0f | 270 | --initrd=<initrd-for-dump-capture-kernel> --args-linux \ |
9c61a446 | 271 | --append="root=<root-dev> <arch-specific-options>" |
dc851a0f | 272 | |
9c61a446 VG |
273 | If you are using a compressed bzImage/vmlinuz, then use following command |
274 | to load dump-capture kernel. | |
dc851a0f | 275 | |
9c61a446 VG |
276 | kexec -p <dump-capture-kernel-bzImage> \ |
277 | --initrd=<initrd-for-dump-capture-kernel> \ | |
278 | --append="root=<root-dev> <arch-specific-options>" | |
279 | ||
280 | Following are the arch specific command line options to be used while | |
281 | loading dump-capture kernel. | |
282 | ||
283 | For i386 and x86_64: | |
284 | "init 1 irqpoll maxcpus=1" | |
285 | ||
286 | For ppc64: | |
287 | "init 1 maxcpus=1 noirqdistrib" | |
dc851a0f | 288 | |
9c61a446 VG |
289 | For IA64 |
290 | (To be filled) | |
291 | ||
292 | ||
293 | Notes on loading the dump-capture kernel: | |
dc851a0f DW |
294 | |
295 | * By default, the ELF headers are stored in ELF64 format to support | |
296 | systems with more than 4GB memory. The --elf32-core-headers option can | |
297 | be used to force the generation of ELF32 headers. This is necessary | |
298 | because GDB currently cannot open vmcore files with ELF64 headers on | |
299 | 32-bit systems. ELF32 headers can be used on non-PAE systems (that is, | |
300 | less than 4GB of memory). | |
301 | ||
302 | * The "irqpoll" boot parameter reduces driver initialization failures | |
303 | due to shared interrupts in the dump-capture kernel. | |
304 | ||
305 | * You must specify <root-dev> in the format corresponding to the root | |
306 | device name in the output of mount command. | |
307 | ||
308 | * "init 1" boots the dump-capture kernel into single-user mode without | |
309 | networking. If you want networking, use "init 3." | |
310 | ||
9c61a446 VG |
311 | * We generally don' have to bring up a SMP kernel just to capture the |
312 | dump. Hence generally it is useful either to build a UP dump-capture | |
313 | kernel or specify maxcpus=1 option while loading dump-capture kernel. | |
dc851a0f DW |
314 | |
315 | Kernel Panic | |
316 | ============ | |
317 | ||
318 | After successfully loading the dump-capture kernel as previously | |
319 | described, the system will reboot into the dump-capture kernel if a | |
320 | system crash is triggered. Trigger points are located in panic(), | |
321 | die(), die_nmi() and in the sysrq handler (ALT-SysRq-c). | |
322 | ||
323 | The following conditions will execute a crash trigger point: | |
324 | ||
325 | If a hard lockup is detected and "NMI watchdog" is configured, the system | |
326 | will boot into the dump-capture kernel ( die_nmi() ). | |
327 | ||
328 | If die() is called, and it happens to be a thread with pid 0 or 1, or die() | |
329 | is called inside interrupt context or die() is called and panic_on_oops is set, | |
330 | the system will boot into the dump-capture kernel. | |
331 | ||
670e9f34 | 332 | On powererpc systems when a soft-reset is generated, die() is called by all cpus and the system will boot into the dump-capture kernel. |
dc851a0f DW |
333 | |
334 | For testing purposes, you can trigger a crash by using "ALT-SysRq-c", | |
335 | "echo c > /proc/sysrq-trigger or write a module to force the panic. | |
336 | ||
337 | Write Out the Dump File | |
338 | ======================= | |
339 | ||
340 | After the dump-capture kernel is booted, write out the dump file with | |
341 | the following command: | |
b089f4a6 VG |
342 | |
343 | cp /proc/vmcore <dump-file> | |
344 | ||
dc851a0f DW |
345 | You can also access dumped memory as a /dev/oldmem device for a linear |
346 | and raw view. To create the device, use the following command: | |
b089f4a6 | 347 | |
dc851a0f | 348 | mknod /dev/oldmem c 1 12 |
b089f4a6 | 349 | |
dc851a0f DW |
350 | Use the dd command with suitable options for count, bs, and skip to |
351 | access specific portions of the dump. | |
b089f4a6 | 352 | |
dc851a0f | 353 | To see the entire memory, use the following command: |
b089f4a6 | 354 | |
dc851a0f | 355 | dd if=/dev/oldmem of=oldmem.001 |
a7e670d8 | 356 | |
dc851a0f DW |
357 | |
358 | Analysis | |
b089f4a6 | 359 | ======== |
b089f4a6 | 360 | |
dc851a0f DW |
361 | Before analyzing the dump image, you should reboot into a stable kernel. |
362 | ||
363 | You can do limited analysis using GDB on the dump file copied out of | |
364 | /proc/vmcore. Use the debug vmlinux built with -g and run the following | |
365 | command: | |
366 | ||
367 | gdb vmlinux <dump-file> | |
b089f4a6 | 368 | |
dc851a0f DW |
369 | Stack trace for the task on processor 0, register display, and memory |
370 | display work fine. | |
b089f4a6 | 371 | |
dc851a0f DW |
372 | Note: GDB cannot analyze core files generated in ELF64 format for x86. |
373 | On systems with a maximum of 4GB of memory, you can generate | |
374 | ELF32-format headers using the --elf32-core-headers kernel option on the | |
375 | dump kernel. | |
b089f4a6 | 376 | |
dc851a0f DW |
377 | You can also use the Crash utility to analyze dump files in Kdump |
378 | format. Crash is available on Dave Anderson's site at the following URL: | |
a7e670d8 | 379 | |
dc851a0f DW |
380 | http://people.redhat.com/~anderson/ |
381 | ||
382 | ||
383 | To Do | |
384 | ===== | |
a7e670d8 | 385 | |
dc851a0f DW |
386 | 1) Provide a kernel pages filtering mechanism, so core file size is not |
387 | extreme on systems with huge memory banks. | |
a7e670d8 | 388 | |
dc851a0f DW |
389 | 2) Relocatable kernel can help in maintaining multiple kernels for |
390 | crash_dump, and the same kernel as the system kernel can be used to | |
391 | capture the dump. | |
b089f4a6 | 392 | |
dc851a0f DW |
393 | |
394 | Contact | |
b089f4a6 | 395 | ======= |
dc851a0f | 396 | |
b089f4a6 | 397 | Vivek Goyal (vgoyal@in.ibm.com) |
d58831e4 | 398 | Maneesh Soni (maneesh@in.ibm.com) |
dc851a0f DW |
399 | |
400 | ||
401 | Trademark | |
402 | ========= | |
403 | ||
404 | Linux is a trademark of Linus Torvalds in the United States, other | |
405 | countries, or both. |