Display the list of valid subcommands. The help subcommand can be reduced at only one topic. The actual list of topics is:
KDB(0)> ? ? help topics: basic subcommands trace subcommands break points/steps dumps/display/decode modify memory namelists/symbols kdbx subcommands (dont use directly) watch subcommands conditionnal subcommand calculator converter machine status loader subcommands address translation system table net subcommands vmm subcommands trampolin subcommands SMP subcommands bat/Block Address Translation btac/BRAT subcommand machdep subcommands KDB(7)> ? step CMD ALIAS ALIAS FUNCTION ARG *** break points/steps *** b brk set/list break point(s) [-p/-v] [addr] lb lbrk set/list local bp(s) [-p/-v] [addr] c cl clear break point [slot|[-p/-v] addr] lc lcl clear local bp [slot|[-p/-v] addr [ctx]] ca clear all break points r return go to end of function gt go until address [-p/-v] addr n nexti next instruction [count] s stepi single step [count] S step on bl/blr B step on branch
The print history subcommand prints history subcommands. Argument can specify a number of subcommands. Each subcommand can be edited and executed with usual control characters (ala emacs).
KDB(3)> his ? Usage: hist [line count] ...... CTRL_A go to beginning of the line ...... CTRL_B one char backward ...... CTRL_D delete one char ...... CTRL_E go to end of line ...... CTRL_F one char forward ...... CTRL_N next command ...... CTRL_P previous command ...... CTRL_U kill line KDB(3)> his tpid f s 11 r n 11 p proc+001680 c dc .kforkx+30 11 mw .kforkx+000040 48005402 . his ? KDB(3)>
The exit subcommand leaves KDB session and returns to the system, all break-points are installed in memory. To leave KDB without break-points, the clear all subcommand must be invoked.
When KDB is called after a kernel panic, the dump argument can be specified to force an AIX dump to be processed.
How to take a dump depending on the type of system crash.
WARNING: You *must* know the key position AT BOOT TIME, whatever the current position is !!! Then type quit, several times (one time per cpu !)
You have "_0c9" on leds while the dump is copied on disk (either on hd7 or hd6) If you entered the debugger through panic, then control is returned to the debugger when the dump is over, and LEDS show xxxx If you entered the debugger through halt_display (c20 on the LEDS), the display shows: 888 102 700 0c0 on the leds when the dump is over.
The setup subcommand may be used to list and set kdb toggles. Current list of toggles is:
Toggles display_stack_frames and display_stacked_regs can be used to find arguments of routines. Arguments are saved in non-volatile registers or in the current stack. It is an easy way to look for them.
KDB(1)> set No toggle name current value 1 no_symbol false 2 mst_wanted true 3 screen_size 24 4 power_pc_syntax true 5 hardware_target 604 6 Unix symbols start from 3500 7 hexadecimal_wanted true 8 screen_previous false 9 display_stack_frames false 10 display_stacked_regs false 11 64_bit false 12 emacs_window false 13 Thread attached local breakpoint 14 KDB stops all processors 15 tweq_r1_r1 true 16 kext_IF_active true 17 kext_IF_active false 18 origin 00000000 KDB(1)> dw 000034CC display memory 000034CC: 00000002 00000008 00010006 00000020 KDB(1)> set 6 1000 toggle change Unix symbols start from 1000 KDB(1)> dw 000034CC display memory _system_configuration+000000: 00000002 00000008 00010006 00000020 KDB(4)> sw 464 Switch to thread: <thread+015C00> KDB(4)> sw u to see user code KDB(4)> dc 1000A14C 1000A14C bl <1000A1A4> KDB(4)> set 17 trace_back_lookup is true KDB(4)> dc 1000A14C .get_superblk+00007C bl <.validate_super> KDB(0)> set origin 002C5338 origin = 002C5338 KDB(0)> b init_heap1 .init_heap1+000000 (real address:002C55F4) permanent & global KDB(0)> e Breakpoint .init_heap1+000000 (ORG+000002BC) stmw r24,FFFFFFE0(stkp) <.mainstk+001EB8> r24=00003A60,FFFFFFE0(stkp)=00384B74 KDB(0)> In the listing you can see ... | 000000 PDEF init_heap1 0| PROC heap_addr,numpages,flags,heapx,pages,gr3-gr8 0| 0002BC stm BF01FFE0 8 STM #stack(gr1,-32)=gr24-gr31 ...
The stack subcommand displays all the stacks from the current instruction as deep as possible. Interrupts and system calls are crossed and the user stack is also seen. In the user space, trace back allows to have symbolic names. But KDB can not access directly these symbols. Use the +x toogle to have binary addresses (by example, to put a break point on one of these addresses). This toggle is valid until -x is entered with another stack subcommand To have shorter display, you can suppressed the mst display with the corresponding toggle mst_wanted. To have longer display, you can set the corresponding toggle display_stack_wanted. It is possible to display the stack of a specified thread (by its slot entry or address).
If running code is -O compiled, routine parameters are not saved in the stack. KDB warns about that with [??] at the end of the line. In this case, showed parameters may be wrong.
In the following example, we set a break point on v_gettlock, and when the break point is encountered, the stack is displayed. Then we try to display the first argument of the open() syscall. Looking at the code, we can see that argument is saved by copen() in register R31, and this register is saved in the stack by openpath(). Looking at memory pointed by register R31, argument is found: /dev/ptc
KDB(2)> f show the stack thread+012540 STACK: [0004AC84]v_gettlock+000000 (00012049, C0011E80, 00000080, 00000000 [??]) [00085C18]v_pregettlock+0000B4 (??, ??, ??, ??) [000132E8]isync_vcs1+0000D8 (??, ??) ____ Exception (2FF3B400) ____ [000131FC].backt+000000 (00012049, C0011E80 [??]) [0004B220]vm_gettlock+000020 (??, ??) [0019A64C]iwrite+00013C (??) [0019D194]finicom+0000A0 (??, ??) [0019D4F0]comlist+0001CC (??, ??) [0019D5BC]_commit+000030 (00000000, 00000001, 09C6E9E8, 399028AA, 0000A46F, 0000E2AA, 2D3A4EAA, 2FF3A730) [001E1B18]jfs_setattr+000258 (??, ??, ??, ??, ??, ??) [001A5ED4]vnop_setattr+000018 (??, ??, ??, ??, ??, ??) [001E9008]spec_setattr+00017C (??, ??, ??, ??, ??, ??) [001A5ED4]vnop_setattr+000018 (??, ??, ??, ??, ??, ??) [01B655C8]pty_vsetattr+00002C (??, ??, ??, ??, ??, ??) [01B6584C]pty_setname+000084 (??, ??, ??, ??, ??, ??) [01B60810]pty_create_ptp+0002C4 (??, ??, ??, ??, ??) [01B60210]pty_open_comm+00015C (??, ??, ??, ??) [01B5FFC0]call_pty_open_comm+0000B8 (??, ??, ??, ??) [01B6526C]ptm_open+000140 (??, ??, ??, ??, ??) (2)> more (^C to quit) ? [01A9A124]open_wrapper+0000D0 (??) [01A8DF74]csq_protect+000258 (??, ??, ??, ??, ??, ??) [01A96348]osr_open+0000BC (??) [01A9C1C8]pse_clone_open+000164 (??, ??, ??, ??) [001ADCC8]spec_clone+000178 (??, ??, ??, ??, ??) [001B3FC4]openpnp+0003AC (??, ??, ??, ??, ??) [001B4178]openpath+000064 (??, ??, ??, ??, ??, ??) [001B43E8]copen+000130 (??, ??, ??, ??, ??) [001B44BC]open+000014 (??, ??, ??) [000037D8].sys_call+000000 () [10002E74]doit+00003C (??, ??, ??) [10003924]main+0004CC (??, ??) [1000014C].__start+00004C () KDB(2)> set 10 show saved registers display_stacked_regs is true KDB(2)> f show the stack thread+012540 STACK: [0004AC84]v_gettlock+000000 (00012049, C0011E80, 00000080, 00000000 [??]) ... [001B3FC4]openpnp+0003AC (??, ??, ??, ??, ??) r24 : 2FF3B6E0 r25 : 2FF3B400 r26 : 10002E78 r27 : 00000000 r28 : 00000002 r29 : 2FF3B3C0 r30 : 00000000 r31 : 20000510 [001B4178]openpath+000064 (??, ??, ??, ??, ??, ??) [001B43E8]copen+000130 (??, ??, ??, ??, ??) r27 : 2A22A424 r28 : E3014000 r29 : E6012540 r30 : 0C87B000 r31 : 00000000 [001B44BC]open+000014 (??, ??, ??) ... KDB(2)> dc open 6 look for argument R3 .open+000000 stwu stkp,FFFFFFC0(stkp) .open+000004 mflr r0 .open+000008 addic r7,stkp,38 .open+00000C stw r0,48(stkp) .open+000010 li r6,0 .open+000014 bl <.copen> KDB(2)> dc copen 9 look for argument R3 .copen+000000 stmw r27,FFFFFFEC(stkp) .copen+000004 addi r28,r4,0 .copen+000008 mflr r0 .copen+00000C lwz r4,D5C(toc) D5C(toc)=audit_flag .copen+000010 stw r0,8(stkp) .copen+000014 stwu stkp,FFFFFFA0(stkp) .copen+000018 cmpi cr0,r4,0 .copen+00001C mtcrf cr5,r28 .copen+000020 addi r31,r3,0 KDB(2)> d 20000510 display memory location @R31 20000510: 2F64 6576 2F70 7463 0000 0000 416C 6C20 /dev/ptc....All
In the following example, the problem is to find what is lsfs subcommand waiting for. The answer is given with getfssize arguments, and there are saved in the stack.
# ps -ef|grep lsfs root 63046 39258 0 Apr 01 pts/1 0:00 lsfs # kdb Preserving 587377 bytes of symbol table First symbol sys_resource PFT: id....................0007 raddr.............01000000 eaddr.............B0000000 size..............01000000 align.............01000000 valid..1 ros....0 holes..0 io.....0 seg....0 wimg...2 PVT: id....................0008 raddr.............003BC000 eaddr.............B2000000 size..............001FFDA0 align.............00001000 valid..1 ros....0 holes..0 io.....0 seg....0 wimg...2 (0)> dcal 63046 print hexa value of PID Value decimal: 63046 Value hexa: 0000F646 (0)> tpid 0000F646 show threads of this PID SLOT NAME STATE TID PRI CPUID CPU FLAGS WCHAN thread+025440 795 lsfs SLEEP 31B31 03C 000 00000004 057DB5BC (0)> sw 795 set current context on this thread Switch to thread: <thread+025440> (0)> f show the stack thread+025440 STACK: [000205C0]e_block_thread+000250 () [00020B1C]e_sleep_thread+000040 (??, ??, ??) [0002AAA0]iowait+00004C (??) [0002B40C]bread+0000DC (??, ??) [0020AF4C]readblk+0000AC (??, ??, ??, ??) [001E90D8]spec_rdwr+00007C (??, ??, ??, ??, ??, ??, ??, ??) [001A6328]vnop_rdwr+000070 (??, ??, ??, ??, ??, ??, ??, ??) [00198278]rwuio+0000CC (??, ??, ??, ??, ??, ??, ??, ??) [001986AC]rdwr+000184 (??, ??, ??, ??, ??, ??) [001984D4]kreadv+000064 (??, ??, ??, ??) [000037D8].sys_call+000000 () [D0046A18]read+000028 (??, ??, ??) [1000A0E4]get_superblk+000054 (??, ??, ??) [100035F8]read_super+000024 (??, ??, ??, ??) [10005C00]getfssize+0000A0 (??, ??, ??) [10002D18]prnt_stanza+0001E8 (??, ??, ??) [1000349C]do_ls+000294 (??, ??) [10000524]main+0001E8 (??, ??) [1000014C].__start+00004C () (0)> sw u enable user context of the thread (0)> dc 10005C00-a0 8 look for arguments R3, R4, R5 10005B60 mflr r0 10005B64 stw r31,FFFFFFFC(stkp) 10005B68 stw r0,8(stkp) 10005B6C stwu stkp,FFFFFEE0(stkp) 10005B70 stw r3,108(stkp) 10005B74 stw r4,104(stkp) 10005B78 stw r5,10C(stkp) 10005B7C addi r3,r4,0 (0)> set 9 print stack frame display_stack_frames is true (0)> f show the stack thread+025440 STACK: [000205C0]e_block_thread+000250 () ... [100035F8]read_super+000024 (??, ??, ??, ??) ======================================================================= 2FF225D0: 2FF2 26F0 2A20 2429 1000 5C04 F071 71C0 /.&.* $)..\..qq. 2FF225E0: 2FF2 2620 2000 4D74 D000 4E18 F071 F83C /.& .Mt..N..q.< 2FF225F0: F075 2FF8 F074 36A4 F075 0FE0 F075 1FF8 .u/..t6..u...u.. 2FF22600: F071 AE80 8080 8080 0000 0004 0000 0006 .q.............. ======================================================================= [10005C00]getfssize+0000A0 (??, ??, ??) ... (0)> dw 2FF225D0+104 print arguments (offset 0x104 0x108 0x10c) 2FF226D4: 2000DCC8 2000DC78 00000000 00000004 (0)> d 2000DC78 20 print first argument 2000DC78: 2F74 6D70 2F73 7472 6970 655F 6673 2E32 /tmp/stripe_fs.2 2000DC88: 3433 3632 0000 0000 0000 0000 0000 0004 4362............ (0)> d 2000DCC8 20 print second argument 2000DCC8: 2F64 6576 2F73 6C76 3234 3336 3200 0000 /dev/slv24362... 2000DCD8: 0000 0000 0000 0000 0000 0000 0000 0004 ................ (0)> q leave debugger #
Note: This subcommand is only available within the kdb command; it is not included in the KDB Kernel Debugger.
The context subcommand is used in user mode to analyse memory dump. By default, KDB shows the current AIX context. But it is possible to elect the current kernel KDB context, and to see more information in stack trace subcommand. For instance, the complete stack of a kernel panic is seen. Cpu number is given in argument. No argument allows to restore initial context.
WARNING: KDB context is available only if the running kernel is booted with KDB.
$ kdb dump unix dump analysis Preserving 628325 bytes of symbol table First symbol sys_resource Component Names: 1) proc 2) thrd 3) errlg 4) bos 5) vmm 6) bscsi 7) scdisk 8) lvm 9) tty 10) netstat 11) lent_dd PFT: id....................0007 raddr.....0000000001000000 eaddr.....0000000001000000 size..............00800000 align.............00800000 valid..1 ros....0 holes..0 io.....0 seg....1 wimg...2 PVT: id....................0008 raddr.....00000000004B8000 eaddr.....00000000004B8000 size..............000FFD60 align.............00001000 valid..1 ros....0 holes..0 io.....0 seg....1 wimg...2 Dump analysis on POWER_PC POWER_604 machine with 8 cpu(s) Processing symbol table... .......................done (0)> stat machine status RS6K_SMP_MCA POWER_PC POWER_604 machine with 8 cpu(s) .......... SYSTEM STATUS sysname... AIX nodename.. jumbo32 release... 3 version... 4 machine... 00920312A0 nid....... 920312A0 time of crash: Tue Jul 22 09:46:22 1997 age of system: 1 day, 0 min., 35 sec. .......... PANIC STRING assert(v_lookup(sid,pno) == -1) .......... SYSTEM MESSAGES AIX Version 4.3 Starting physical processor #1 as logical #1... done. Starting physical processor #2 as logical #2... done. Starting physical processor #3 as logical #3... done. Starting physical processor #4 as logical #4... done. Starting physical processor #5 as logical #5... done. Starting physical processor #6 as logical #6... done. Starting physical processor #7 as logical #7... done. [v_lists.c #727] <- end_of_buffer (0)> ctx 0 KDB context of CPU 0 Switch to KDB context of cpu 0 (0)> dr iar current instruction iar : 00009414 .unlock_enable+000110 lwz r0,8(stkp) r0=0,8(stkp)=mststack+00AD18 (0)> ctx 1 KDB context of CPU 1 Switch to KDB context of cpu 1 (1)> dr iar current instruction iar : 000BDB68 .kunlockl+000118 blr <.ld_usecount+0005BC> r3=0000000B (1)> ctx 2 KDB context of CPU 2 Switch to KDB context of cpu 2 (2)> dr iar current instruction iar : 00027634 .tstart+000284 blr <.sys_timer+000964> r3=00000005 (2)> ctx 3 KDB context of CPU 3 Switch to KDB context of cpu 3 (3)> dr iar current instruction iar : 01B6A580 01B6A580 ori r3,r31,0 <00000089> r3=50001000,r31=00000089 (3)> ctx 4 KDB context of CPU 4 Switch to KDB context of cpu 4 (4)> dr iar current instruction iar : 00014BFC .panic_trap+000004 bl <.panic_dump> r3=_$STATIC+000294 (4)> f current stack __kdb_thread+0002F0 STACK: [00014BFC].panic_trap+000004 () [0003ACAC]v_inspft+000104 (??, ??, ??) [00048DA8]v_inherit+0004A0 (??, ??, ??) [000A7ECC]v_preinherit+000058 (??, ??, ??) [00027BFC]begbt_603_patch_2+000008 (??, ??) Machine State Save Area [2FF3B400] iar : 00027AEC msr : 000010B0 cr : 22222222 lr : 00243E58 ctr : 00000000 xer : 00000000 mq : 00000000 r0 : 000A7E74 r1 : 2FF3B220 r2 : 002EBC70 r3 : 00013350 r4 : 00000000 r5 : 00000100 r6 : 00009030 r7 : 2FF3B400 r8 : 00000106 r9 : 00000000 r10 : 00243E58 r11 : 2FF3B400 r12 : 000010B0 r13 : 000C1C80 r14 : 2FF22A88 r15 : 20022DB8 r16 : 20006A98 r17 : 20033128 r18 : 00000000 r19 : 0008AD56 r20 : B02A6038 r21 : 0000006A r22 : 00000000 r23 : 0000FFFF r24 : 00000100 r25 : 00003262 r26 : 00000000 r27 : B02B8AEC r28 : B02A9F70 r29 : 00000001 r30 : 00003350 r31 : 00013350 s0 : 00000000 s1 : 007FFFFF s2 : 0000864B s3 : 007FFFFF s4 : 007FFFFF s5 : 007FFFFF s6 : 007FFFFF s7 : 007FFFFF s8 : 007FFFFF s9 : 007FFFFF s10 : 007FFFFF s11 : 00001001 s12 : 00002002 s13 : 6001F01F s14 : 00004004 s15 : 007FFFFF prev 00000000 kjmpbuf 00000000 stackfix 00000000 intpri 0B curid 0008AD56 sralloc E01E0000 ioalloc 00000000 backt 00 flags 00 tid 00000000 excp_type 00000000 fpscr 00000000 fpeu 01 fpinfo 00 fpscrx 00000000 o_iar 00000000 o_toc 00000000 o_arg1 00000000 excbranch 00000000 o_vaddr 00000000 mstext 00000000 Except : csr 00000000 dsisr 40000000 bit set: DSISR_PFT srval 6000864B dar 2FF22FF8 dsirr 00000106 [00027AEC].backt+000000 (00013350, 00000000 [??]) [00243E54]vms_delete+0004DC (??) [00256838]shmfreews+0000B0 () [000732B4]freeuspace+000010 () [00072EAC]kexitx+000688 (??) (4)> ctx AIX context of CPU 4 Restore initial context (4)> f current stack thread+031920 STACK: [00027AEC].backt+000000 (00013350, 00000000 [??]) [00243E54]vms_delete+0004DC (??) [00256838]shmfreews+0000B0 () [000732B4]freeuspace+000010 () [00072EAC]kexitx+000688 (??) (4)>
Note: This subcommand is only available within the kdb command; it is not included in the KDB Kernel Debugger.
The cdt subcommand is used in user mode to analyse memory dump. Any part of component dump can be printed. Usage is cdt [-d] [<index>] [<entry>].
(0)> cdt 1) CDT head name proc, len 001D80E8, entries 96676 2) CDT head name thrd, len 003ABE4C, entries 192489 3) CDT head name errlg, len 00000054, entries 3 4) CDT head name bos, len 00000040, entries 2 5) CDT head name vmm, len 000003D8, entries 30 6) CDT head name sscsidd, len 0000007C, entries 5 7) CDT head name dptSR, len 00000054, entries 3 8) CDT head name scdisk, len 00000130, entries 14 9) CDT head name lvm, len 00000040, entries 2 10) CDT head name SSAGS, len 000000A4, entries 7 11) CDT head name SSAES, len 00000054, entries 3 12) CDT head name ssagateway, len 0000007C, entries 5 13) CDT head name tty, len 00000068, entries 4 14) CDT head name sio_dd, len 00000054, entries 3 15) CDT head name netstat, len 000000E0, entries 10 16) CDT head name ent2104x, len 00000054, entries 3 17) CDT head name cstokdd, len 0000007C, entries 5 18) CDT head name atm_dd_charm, len 00000040, entries 2 19) CDT head name ssadisk, len 000002AC, entries 33 20) CDT head name SSADS, len 00000040, entries 2 21) CDT head name osi_frame, len 0000002C, entries 1 (0)> cdt 12 12) CDT head name ssagateway, len 0000007C, entries 5 CDT 1 name HashTbl addr 0000000001A25CF0, len 00000040 CDT 2 name CfgdAdap addr 0000000001A0E044, len 00000004 CDT 3 name OpenAdap addr 0000000001A0E048, len 00000004 CDT 4 name LockWord addr 0000000001A0E04C, len 00000004 CDT 5 name ssa0 addr 0000000001A2D000, len 00000B88 (0)> cdt -d 12 4 12) CDT head name ssagateway, len 0000007C, entries 5 CDT 4 name LockWord addr 0000000001A0E04C, len 00000004 01A0E04C: FFFFFFFF ....