]>
Commit | Line | Data |
---|---|---|
5f096274 JC |
1 | .. highlight:: none |
2 | ||
bda1a921 JK |
3 | Debugging kernel and modules via gdb |
4 | ==================================== | |
5 | ||
6 | The kernel debugger kgdb, hypervisors like QEMU or JTAG-based hardware | |
7 | interfaces allow to debug the Linux kernel and its modules during runtime | |
8 | using gdb. Gdb comes with a powerful scripting interface for python. The | |
9 | kernel provides a collection of helper scripts that can simplify typical | |
10 | kernel debugging steps. This is a short tutorial about how to enable and use | |
11 | them. It focuses on QEMU/KVM virtual machines as target, but the examples can | |
12 | be transferred to the other gdb stubs as well. | |
13 | ||
14 | ||
15 | Requirements | |
16 | ------------ | |
17 | ||
5f096274 JC |
18 | - gdb 7.2+ (recommended: 7.4+) with python support enabled (typically true |
19 | for distributions) | |
bda1a921 JK |
20 | |
21 | ||
22 | Setup | |
23 | ----- | |
24 | ||
5f096274 JC |
25 | - Create a virtual Linux machine for QEMU/KVM (see www.linux-kvm.org and |
26 | www.qemu.org for more details). For cross-development, | |
27 | http://landley.net/aboriginal/bin keeps a pool of machine images and | |
28 | toolchains that can be helpful to start from. | |
bda1a921 | 29 | |
5f096274 JC |
30 | - Build the kernel with CONFIG_GDB_SCRIPTS enabled, but leave |
31 | CONFIG_DEBUG_INFO_REDUCED off. If your architecture supports | |
32 | CONFIG_FRAME_POINTER, keep it enabled. | |
bda1a921 | 33 | |
24159749 ZZ |
34 | - Install that kernel on the guest, turn off KASLR if necessary by adding |
35 | "nokaslr" to the kernel command line. | |
5f096274 JC |
36 | Alternatively, QEMU allows to boot the kernel directly using -kernel, |
37 | -append, -initrd command line switches. This is generally only useful if | |
38 | you do not depend on modules. See QEMU documentation for more details on | |
24159749 ZZ |
39 | this mode. In this case, you should build the kernel with |
40 | CONFIG_RANDOMIZE_BASE disabled if the architecture supports KASLR. | |
bda1a921 | 41 | |
5f096274 | 42 | - Enable the gdb stub of QEMU/KVM, either |
bda1a921 | 43 | |
bda1a921 | 44 | - at VM startup time by appending "-s" to the QEMU command line |
5f096274 JC |
45 | |
46 | or | |
47 | ||
bda1a921 JK |
48 | - during runtime by issuing "gdbserver" from the QEMU monitor |
49 | console | |
50 | ||
5f096274 | 51 | - cd /path/to/linux-build |
bda1a921 | 52 | |
5f096274 | 53 | - Start gdb: gdb vmlinux |
bda1a921 | 54 | |
5f096274 JC |
55 | Note: Some distros may restrict auto-loading of gdb scripts to known safe |
56 | directories. In case gdb reports to refuse loading vmlinux-gdb.py, add:: | |
bda1a921 JK |
57 | |
58 | add-auto-load-safe-path /path/to/linux-build | |
59 | ||
5f096274 JC |
60 | to ~/.gdbinit. See gdb help for more details. |
61 | ||
62 | - Attach to the booted guest:: | |
bda1a921 | 63 | |
bda1a921 JK |
64 | (gdb) target remote :1234 |
65 | ||
66 | ||
67 | Examples of using the Linux-provided gdb helpers | |
68 | ------------------------------------------------ | |
69 | ||
5f096274 JC |
70 | - Load module (and main kernel) symbols:: |
71 | ||
bda1a921 JK |
72 | (gdb) lx-symbols |
73 | loading vmlinux | |
74 | scanning for modules in /home/user/linux/build | |
75 | loading @0xffffffffa0020000: /home/user/linux/build/net/netfilter/xt_tcpudp.ko | |
76 | loading @0xffffffffa0016000: /home/user/linux/build/net/netfilter/xt_pkttype.ko | |
77 | loading @0xffffffffa0002000: /home/user/linux/build/net/netfilter/xt_limit.ko | |
78 | loading @0xffffffffa00ca000: /home/user/linux/build/net/packet/af_packet.ko | |
79 | loading @0xffffffffa003c000: /home/user/linux/build/fs/fuse/fuse.ko | |
80 | ... | |
81 | loading @0xffffffffa0000000: /home/user/linux/build/drivers/ata/ata_generic.ko | |
82 | ||
5f096274 JC |
83 | - Set a breakpoint on some not yet loaded module function, e.g.:: |
84 | ||
bda1a921 JK |
85 | (gdb) b btrfs_init_sysfs |
86 | Function "btrfs_init_sysfs" not defined. | |
87 | Make breakpoint pending on future shared library load? (y or [n]) y | |
88 | Breakpoint 1 (btrfs_init_sysfs) pending. | |
89 | ||
5f096274 JC |
90 | - Continue the target:: |
91 | ||
bda1a921 JK |
92 | (gdb) c |
93 | ||
5f096274 JC |
94 | - Load the module on the target and watch the symbols being loaded as well as |
95 | the breakpoint hit:: | |
96 | ||
bda1a921 JK |
97 | loading @0xffffffffa0034000: /home/user/linux/build/lib/libcrc32c.ko |
98 | loading @0xffffffffa0050000: /home/user/linux/build/lib/lzo/lzo_compress.ko | |
99 | loading @0xffffffffa006e000: /home/user/linux/build/lib/zlib_deflate/zlib_deflate.ko | |
100 | loading @0xffffffffa01b1000: /home/user/linux/build/fs/btrfs/btrfs.ko | |
101 | ||
102 | Breakpoint 1, btrfs_init_sysfs () at /home/user/linux/fs/btrfs/sysfs.c:36 | |
103 | 36 btrfs_kset = kset_create_and_add("btrfs", NULL, fs_kobj); | |
104 | ||
5f096274 JC |
105 | - Dump the log buffer of the target kernel:: |
106 | ||
bda1a921 JK |
107 | (gdb) lx-dmesg |
108 | [ 0.000000] Initializing cgroup subsys cpuset | |
109 | [ 0.000000] Initializing cgroup subsys cpu | |
110 | [ 0.000000] Linux version 3.8.0-rc4-dbg+ (... | |
111 | [ 0.000000] Command line: root=/dev/sda2 resume=/dev/sda1 vga=0x314 | |
112 | [ 0.000000] e820: BIOS-provided physical RAM map: | |
113 | [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable | |
114 | [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved | |
115 | .... | |
116 | ||
5f096274 JC |
117 | - Examine fields of the current task struct:: |
118 | ||
bda1a921 JK |
119 | (gdb) p $lx_current().pid |
120 | $1 = 4998 | |
121 | (gdb) p $lx_current().comm | |
122 | $2 = "modprobe\000\000\000\000\000\000\000" | |
123 | ||
5f096274 JC |
124 | - Make use of the per-cpu function for the current or a specified CPU:: |
125 | ||
bda1a921 JK |
126 | (gdb) p $lx_per_cpu("runqueues").nr_running |
127 | $3 = 1 | |
128 | (gdb) p $lx_per_cpu("runqueues", 2).nr_running | |
129 | $4 = 0 | |
130 | ||
5f096274 JC |
131 | - Dig into hrtimers using the container_of helper:: |
132 | ||
bda1a921 JK |
133 | (gdb) set $next = $lx_per_cpu("hrtimer_bases").clock_base[0].active.next |
134 | (gdb) p *$container_of($next, "struct hrtimer", "node") | |
135 | $5 = { | |
136 | node = { | |
137 | node = { | |
138 | __rb_parent_color = 18446612133355256072, | |
139 | rb_right = 0x0 <irq_stack_union>, | |
140 | rb_left = 0x0 <irq_stack_union> | |
141 | }, | |
142 | expires = { | |
143 | tv64 = 1835268000000 | |
144 | } | |
145 | }, | |
146 | _softexpires = { | |
147 | tv64 = 1835268000000 | |
148 | }, | |
149 | function = 0xffffffff81078232 <tick_sched_timer>, | |
150 | base = 0xffff88003fd0d6f0, | |
151 | state = 1, | |
152 | start_pid = 0, | |
153 | start_site = 0xffffffff81055c1f <hrtimer_start_range_ns+20>, | |
154 | start_comm = "swapper/2\000\000\000\000\000\000" | |
155 | } | |
156 | ||
157 | ||
158 | List of commands and functions | |
159 | ------------------------------ | |
160 | ||
161 | The number of commands and convenience functions may evolve over the time, | |
5f096274 | 162 | this is just a snapshot of the initial version:: |
bda1a921 JK |
163 | |
164 | (gdb) apropos lx | |
165 | function lx_current -- Return current task | |
166 | function lx_module -- Find module by name and return the module variable | |
167 | function lx_per_cpu -- Return per-cpu variable | |
168 | function lx_task_by_pid -- Find Linux task by PID and return the task_struct variable | |
169 | function lx_thread_info -- Calculate Linux thread_info from task variable | |
170 | lx-dmesg -- Print Linux kernel log buffer | |
171 | lx-lsmod -- List currently loaded modules | |
172 | lx-symbols -- (Re-)load symbols of Linux kernel and currently loaded modules | |
173 | ||
174 | Detailed help can be obtained via "help <command-name>" for commands and "help | |
175 | function <function-name>" for convenience functions. |