]>
Commit | Line | Data |
---|---|---|
dc851a0f DW |
1 | ================================================================ |
2 | Documentation for Kdump - The kexec-based Crash Dumping Solution | |
b089f4a6 VG |
3 | ================================================================ |
4 | ||
dc851a0f DW |
5 | This document includes overview, setup and installation, and analysis |
6 | information. | |
b089f4a6 | 7 | |
dc851a0f DW |
8 | Overview |
9 | ======== | |
b089f4a6 | 10 | |
dc851a0f DW |
11 | Kdump uses kexec to quickly boot to a dump-capture kernel whenever a |
12 | dump of the system kernel's memory needs to be taken (for example, when | |
13 | the system panics). The system kernel's memory image is preserved across | |
14 | the reboot and is accessible to the dump-capture kernel. | |
b089f4a6 | 15 | |
dc851a0f DW |
16 | You can use common Linux commands, such as cp and scp, to copy the |
17 | memory image to a dump file on the local disk, or across the network to | |
18 | a remote system. | |
b089f4a6 | 19 | |
dc851a0f DW |
20 | Kdump and kexec are currently supported on the x86, x86_64, and ppc64 |
21 | architectures. | |
b089f4a6 | 22 | |
dc851a0f DW |
23 | When the system kernel boots, it reserves a small section of memory for |
24 | the dump-capture kernel. This ensures that ongoing Direct Memory Access | |
25 | (DMA) from the system kernel does not corrupt the dump-capture kernel. | |
26 | The kexec -p command loads the dump-capture kernel into this reserved | |
27 | memory. | |
b089f4a6 | 28 | |
dc851a0f DW |
29 | On x86 machines, the first 640 KB of physical memory is needed to boot, |
30 | regardless of where the kernel loads. Therefore, kexec backs up this | |
31 | region just before rebooting into the dump-capture kernel. | |
b089f4a6 | 32 | |
dc851a0f DW |
33 | All of the necessary information about the system kernel's core image is |
34 | encoded in the ELF format, and stored in a reserved area of memory | |
35 | before a crash. The physical address of the start of the ELF header is | |
36 | passed to the dump-capture kernel through the elfcorehdr= boot | |
37 | parameter. | |
38 | ||
39 | With the dump-capture kernel, you can access the memory image, or "old | |
40 | memory," in two ways: | |
41 | ||
42 | - Through a /dev/oldmem device interface. A capture utility can read the | |
43 | device file and write out the memory in raw format. This is a raw dump | |
44 | of memory. Analysis and capture tools must be intelligent enough to | |
45 | determine where to look for the right information. | |
46 | ||
47 | - Through /proc/vmcore. This exports the dump as an ELF-format file that | |
48 | you can write out using file copy commands such as cp or scp. Further, | |
49 | you can use analysis tools such as the GNU Debugger (GDB) and the Crash | |
50 | tool to debug the dump file. This method ensures that the dump pages are | |
51 | correctly ordered. | |
52 | ||
53 | ||
54 | Setup and Installation | |
55 | ====================== | |
56 | ||
57 | Install kexec-tools and the Kdump patch | |
58 | --------------------------------------- | |
59 | ||
60 | 1) Login as the root user. | |
61 | ||
62 | 2) Download the kexec-tools user-space package from the following URL: | |
63 | ||
64 | http://www.xmission.com/~ebiederm/files/kexec/kexec-tools-1.101.tar.gz | |
65 | ||
66 | 3) Unpack the tarball with the tar command, as follows: | |
67 | ||
68 | tar xvpzf kexec-tools-1.101.tar.gz | |
69 | ||
70 | 4) Download the latest consolidated Kdump patch from the following URL: | |
71 | ||
72 | http://lse.sourceforge.net/kdump/ | |
73 | ||
74 | (This location is being used until all the user-space Kdump patches | |
75 | are integrated with the kexec-tools package.) | |
76 | ||
77 | 5) Change to the kexec-tools-1.101 directory, as follows: | |
78 | ||
79 | cd kexec-tools-1.101 | |
80 | ||
81 | 6) Apply the consolidated patch to the kexec-tools-1.101 source tree | |
82 | with the patch command, as follows. (Modify the path to the downloaded | |
83 | patch as necessary.) | |
84 | ||
85 | patch -p1 < /path-to-kdump-patch/kexec-tools-1.101-kdump.patch | |
86 | ||
87 | 7) Configure the package, as follows: | |
88 | ||
89 | ./configure | |
90 | ||
91 | 8) Compile the package, as follows: | |
92 | ||
93 | make | |
94 | ||
95 | 9) Install the package, as follows: | |
96 | ||
97 | make install | |
98 | ||
99 | ||
100 | Download and build the system and dump-capture kernels | |
101 | ------------------------------------------------------ | |
102 | ||
103 | Download the mainline (vanilla) kernel source code (2.6.13-rc1 or newer) | |
104 | from http://www.kernel.org. Two kernels must be built: a system kernel | |
105 | and a dump-capture kernel. Use the following steps to configure these | |
106 | kernels with the necessary kexec and Kdump features: | |
107 | ||
108 | System kernel | |
109 | ------------- | |
110 | ||
111 | 1) Enable "kexec system call" in "Processor type and features." | |
112 | ||
113 | CONFIG_KEXEC=y | |
114 | ||
115 | 2) Enable "sysfs file system support" in "Filesystem" -> "Pseudo | |
116 | filesystems." This is usually enabled by default. | |
117 | ||
118 | CONFIG_SYSFS=y | |
119 | ||
120 | Note that "sysfs file system support" might not appear in the "Pseudo | |
121 | filesystems" menu if "Configure standard kernel features (for small | |
122 | systems)" is not enabled in "General Setup." In this case, check the | |
123 | .config file itself to ensure that sysfs is turned on, as follows: | |
124 | ||
125 | grep 'CONFIG_SYSFS' .config | |
126 | ||
127 | 3) Enable "Compile the kernel with debug info" in "Kernel hacking." | |
128 | ||
129 | CONFIG_DEBUG_INFO=Y | |
130 | ||
131 | This causes the kernel to be built with debug symbols. The dump | |
132 | analysis tools require a vmlinux with debug symbols in order to read | |
133 | and analyze a dump file. | |
134 | ||
135 | 4) Make and install the kernel and its modules. Update the boot loader | |
136 | (such as grub, yaboot, or lilo) configuration files as necessary. | |
137 | ||
138 | 5) Boot the system kernel with the boot parameter "crashkernel=Y@X", | |
139 | where Y specifies how much memory to reserve for the dump-capture kernel | |
140 | and X specifies the beginning of this reserved memory. For example, | |
141 | "crashkernel=64M@16M" tells the system kernel to reserve 64 MB of memory | |
142 | starting at physical address 0x01000000 for the dump-capture kernel. | |
143 | ||
144 | On x86 and x86_64, use "crashkernel=64M@16M". | |
145 | ||
146 | On ppc64, use "crashkernel=128M@32M". | |
147 | ||
148 | ||
149 | The dump-capture kernel | |
150 | ----------------------- | |
b089f4a6 | 151 | |
dc851a0f DW |
152 | 1) Under "General setup," append "-kdump" to the current string in |
153 | "Local version." | |
154 | ||
155 | 2) On x86, enable high memory support under "Processor type and | |
156 | features": | |
157 | ||
158 | CONFIG_HIGHMEM64G=y | |
159 | or | |
160 | CONFIG_HIGHMEM4G | |
161 | ||
162 | 3) On x86 and x86_64, disable symmetric multi-processing support | |
163 | under "Processor type and features": | |
164 | ||
165 | CONFIG_SMP=n | |
166 | (If CONFIG_SMP=y, then specify maxcpus=1 on the kernel command line | |
167 | when loading the dump-capture kernel, see section "Load the Dump-capture | |
168 | Kernel".) | |
169 | ||
170 | 4) On ppc64, disable NUMA support and enable EMBEDDED support: | |
171 | ||
172 | CONFIG_NUMA=n | |
173 | CONFIG_EMBEDDED=y | |
174 | CONFIG_EEH=N for the dump-capture kernel | |
175 | ||
176 | 5) Enable "kernel crash dumps" support under "Processor type and | |
177 | features": | |
178 | ||
179 | CONFIG_CRASH_DUMP=y | |
180 | ||
181 | 6) Use a suitable value for "Physical address where the kernel is | |
182 | loaded" (under "Processor type and features"). This only appears when | |
183 | "kernel crash dumps" is enabled. By default this value is 0x1000000 | |
184 | (16MB). It should be the same as X in the "crashkernel=Y@X" boot | |
185 | parameter discussed above. | |
186 | ||
187 | On x86 and x86_64, use "CONFIG_PHYSICAL_START=0x1000000". | |
188 | ||
189 | On ppc64 the value is automatically set at 32MB when | |
190 | CONFIG_CRASH_DUMP is set. | |
191 | ||
192 | 6) Optionally enable "/proc/vmcore support" under "Filesystems" -> | |
193 | "Pseudo filesystems". | |
194 | ||
195 | CONFIG_PROC_VMCORE=y | |
196 | (CONFIG_PROC_VMCORE is set by default when CONFIG_CRASH_DUMP is selected.) | |
197 | ||
198 | 7) Make and install the kernel and its modules. DO NOT add this kernel | |
199 | to the boot loader configuration files. | |
200 | ||
201 | ||
202 | Load the Dump-capture Kernel | |
203 | ============================ | |
204 | ||
205 | After booting to the system kernel, load the dump-capture kernel using | |
206 | the following command: | |
207 | ||
208 | kexec -p <dump-capture-kernel> \ | |
209 | --initrd=<initrd-for-dump-capture-kernel> --args-linux \ | |
210 | --append="root=<root-dev> init 1 irqpoll" | |
211 | ||
212 | ||
213 | Notes on loading the dump-capture kernel: | |
214 | ||
215 | * <dump-capture-kernel> must be a vmlinux image (that is, an | |
216 | uncompressed ELF image). bzImage does not work at this time. | |
217 | ||
218 | * By default, the ELF headers are stored in ELF64 format to support | |
219 | systems with more than 4GB memory. The --elf32-core-headers option can | |
220 | be used to force the generation of ELF32 headers. This is necessary | |
221 | because GDB currently cannot open vmcore files with ELF64 headers on | |
222 | 32-bit systems. ELF32 headers can be used on non-PAE systems (that is, | |
223 | less than 4GB of memory). | |
224 | ||
225 | * The "irqpoll" boot parameter reduces driver initialization failures | |
226 | due to shared interrupts in the dump-capture kernel. | |
227 | ||
228 | * You must specify <root-dev> in the format corresponding to the root | |
229 | device name in the output of mount command. | |
230 | ||
231 | * "init 1" boots the dump-capture kernel into single-user mode without | |
232 | networking. If you want networking, use "init 3." | |
233 | ||
234 | ||
235 | Kernel Panic | |
236 | ============ | |
237 | ||
238 | After successfully loading the dump-capture kernel as previously | |
239 | described, the system will reboot into the dump-capture kernel if a | |
240 | system crash is triggered. Trigger points are located in panic(), | |
241 | die(), die_nmi() and in the sysrq handler (ALT-SysRq-c). | |
242 | ||
243 | The following conditions will execute a crash trigger point: | |
244 | ||
245 | If a hard lockup is detected and "NMI watchdog" is configured, the system | |
246 | will boot into the dump-capture kernel ( die_nmi() ). | |
247 | ||
248 | If die() is called, and it happens to be a thread with pid 0 or 1, or die() | |
249 | is called inside interrupt context or die() is called and panic_on_oops is set, | |
250 | the system will boot into the dump-capture kernel. | |
251 | ||
252 | On powererpc systems when a soft-reset is generated, die() is called by all cpus and the system system will boot into the dump-capture kernel. | |
253 | ||
254 | For testing purposes, you can trigger a crash by using "ALT-SysRq-c", | |
255 | "echo c > /proc/sysrq-trigger or write a module to force the panic. | |
256 | ||
257 | Write Out the Dump File | |
258 | ======================= | |
259 | ||
260 | After the dump-capture kernel is booted, write out the dump file with | |
261 | the following command: | |
b089f4a6 VG |
262 | |
263 | cp /proc/vmcore <dump-file> | |
264 | ||
dc851a0f DW |
265 | You can also access dumped memory as a /dev/oldmem device for a linear |
266 | and raw view. To create the device, use the following command: | |
b089f4a6 | 267 | |
dc851a0f | 268 | mknod /dev/oldmem c 1 12 |
b089f4a6 | 269 | |
dc851a0f DW |
270 | Use the dd command with suitable options for count, bs, and skip to |
271 | access specific portions of the dump. | |
b089f4a6 | 272 | |
dc851a0f | 273 | To see the entire memory, use the following command: |
b089f4a6 | 274 | |
dc851a0f | 275 | dd if=/dev/oldmem of=oldmem.001 |
a7e670d8 | 276 | |
dc851a0f DW |
277 | |
278 | Analysis | |
b089f4a6 | 279 | ======== |
b089f4a6 | 280 | |
dc851a0f DW |
281 | Before analyzing the dump image, you should reboot into a stable kernel. |
282 | ||
283 | You can do limited analysis using GDB on the dump file copied out of | |
284 | /proc/vmcore. Use the debug vmlinux built with -g and run the following | |
285 | command: | |
286 | ||
287 | gdb vmlinux <dump-file> | |
b089f4a6 | 288 | |
dc851a0f DW |
289 | Stack trace for the task on processor 0, register display, and memory |
290 | display work fine. | |
b089f4a6 | 291 | |
dc851a0f DW |
292 | Note: GDB cannot analyze core files generated in ELF64 format for x86. |
293 | On systems with a maximum of 4GB of memory, you can generate | |
294 | ELF32-format headers using the --elf32-core-headers kernel option on the | |
295 | dump kernel. | |
b089f4a6 | 296 | |
dc851a0f DW |
297 | You can also use the Crash utility to analyze dump files in Kdump |
298 | format. Crash is available on Dave Anderson's site at the following URL: | |
a7e670d8 | 299 | |
dc851a0f DW |
300 | http://people.redhat.com/~anderson/ |
301 | ||
302 | ||
303 | To Do | |
304 | ===== | |
a7e670d8 | 305 | |
dc851a0f DW |
306 | 1) Provide a kernel pages filtering mechanism, so core file size is not |
307 | extreme on systems with huge memory banks. | |
a7e670d8 | 308 | |
dc851a0f DW |
309 | 2) Relocatable kernel can help in maintaining multiple kernels for |
310 | crash_dump, and the same kernel as the system kernel can be used to | |
311 | capture the dump. | |
b089f4a6 | 312 | |
dc851a0f DW |
313 | |
314 | Contact | |
b089f4a6 | 315 | ======= |
dc851a0f | 316 | |
b089f4a6 | 317 | Vivek Goyal (vgoyal@in.ibm.com) |
d58831e4 | 318 | Maneesh Soni (maneesh@in.ibm.com) |
dc851a0f DW |
319 | |
320 | ||
321 | Trademark | |
322 | ========= | |
323 | ||
324 | Linux is a trademark of Linus Torvalds in the United States, other | |
325 | countries, or both. |