KDB(2)> hp print kernel heap information Pinned heap 0FFC4000 sanity..... 48454150 base....... F11B7000 lock@...... 0FFC4008 lock....... 00000000 alt........ 00000001 numpages... 0000EE49 amount..... 002D2750 pinflag.... 00000001 newheap.... 00000000 protect.... 00000000 limit...... 00000000 heap64..... 00000000 vmrelflag.. 00000000 rhash...... 00000000 pagtot..... 00000000 pagused.... 00000000 frtot[00].. 00000000 [01].. 00000000 [02].. 00000000 [03].. 00000000 frtot[04].. 00000000 [05].. 00000000 [06].. 00000000 [07].. 00000000 frtot[08].. 00000000 [09].. 00000000 [10].. 00000000 [11].. 00000000 frused[00]. 00000000 [01].. 00000000 [02].. 00000000 [03].. 00000000 frused[04]. 00000000 [05].. 00000000 [06].. 00000000 [07].. 00000000 frused[08]. 00000000 [09].. 00000000 [10].. 00000000 [11].. 00000000 fr[00]..... 00FFFFFF [01].. 00FFFFFF [02].. 00FFFFFF [03].. 00FFFFFF fr[04]..... 00003C22 [05].. 00004167 [06].. 00004A05 [07].. 00004845 fr[08]..... 000043B5 [09].. 00000002 [10].. 0000443A [11].. 00004842 Kernel heap 0FFC40B8 sanity..... 48454150 base....... F11B6F48 lock@...... 0FFC40C0 lock....... 00000000 alt........ 00000000 numpages... 0000EE49 amount..... 04732CF0 pinflag.... 00000000 newheap.... 00000000 protect.... 00000000 limit...... 00000000 heap64..... 00000000 vmrelflag.. 00000000 rhash...... 00000000 pagtot..... 00000000 pagused.... 00000000 frtot[00].. 00000000 [01].. 00000000 [02].. 00000000 [03].. 00000000 frtot[04].. 00000000 [05].. 00000000 [06].. 00000000 [07].. 00000000 frtot[08].. 00000000 [09].. 00000000 [10].. 00000000 [11].. 00000000 frused[00]. 00000000 [01].. 00000000 [02].. 00000000 [03].. 00000000 frused[04]. 00000000 [05].. 00000000 [06].. 00000000 [07].. 00000000 frused[08]. 00000000 [09].. 00000000 [10].. 00000000 [11].. 00000000 fr[00]..... 00FFFFFF [01].. 00FFFFFF [02].. 00FFFFFF [03].. 00FFFFFF fr[04]..... 000049E9 [05].. 00003C26 [06].. 0000484E [07].. 00004737 fr[08]..... 00003C0A [09].. 00004A07 [10].. 00004855 [11].. 00004A11 addr...... 0000000000000000 maxpages.......... 00000000 peakpage.......... 00000000 limit_callout..... 00000000 newseg_callout.... 00000000 pagesoffset....... 0FFC4194 pages_sid......... 00000000 Heap anchor ... 0FFC4190 pageno FFFFFFFF pages.type.. 00 allocpage offset... 00004A08 Heap Free list ... 0FFD69B4 pageno 00004A08 pages.type.. 02 freepage offset... 00004A0C ... 0FFD69C4 pageno 00004A0C pages.type.. 03 freerange offset... 00004A17 ... 0FFD69C8 pageno 00004A0D pages.type.. 04 freesize size..... 00000005 ... 0FFD69D4 pageno 00004A10 pages.type.. 05 freerangeend offset... 00004A0C ... 0FFD69F0 pageno 00004A17 pages.type.. 03 freerange offset... NO_PAGE ... 0FFD69F4 pageno 00004A18 pages.type.. 04 freesize size..... 0000A432 ... 0FFFFAB4 pageno 0000EE48 pages.type.. 05 freerangeend offset... 00004A17 Heap Alloc list ... 0FFC41B0 pageno 00000007 pages.type.. 01 allocrange offset... NO_PAGE ... 0FFC41B4 pageno 00000008 pages.type.. 06 allocsize size..... 00001E00 ... 0FFCB9AC pageno 00001E06 pages.type.. 07 allocrangeend offset... 00000007 ... 0FFCB9B0 pageno 00001E07 pages.type.. 01 allocrange offset... NO_PAGE ... 0FFCB9B4 pageno 00001E08 pages.type.. 06 allocsize size..... 00001E00 ... 0FFD31AC pageno 00003C06 pages.type.. 07 allocrangeend offset... 00001E07 ... 0FFD31B4 pageno 00003C08 pages.type.. 01 allocrange offset... 00003C42 ... 0FFD31B8 pageno 00003C09 pages.type.. 06 allocsize size..... 00000002 ... 0FFD31C4 pageno 00003C0C pages.type.. 01 allocrange offset... NO_PAGE ... 0FFD31C8 pageno 00003C0D pages.type.. 06 allocsize size..... 00000009 ... 0FFD31E4 pageno 00003C14 pages.type.. 07 allocrangeend offset... 00003C0C ... 0FFD31E8 pageno 00003C15 pages.type.. 01 allocrange offset... NO_PAGE ... 0FFD31EC pageno 00003C16 pages.type.. 06 allocsize size..... 00000009 ... 0FFD3208 pageno 00003C1D pages.type.. 07 allocrangeend offset... 00003C15 ... 0FFD320C pageno 00003C1E pages.type.. 01 allocrange offset... NO_PAGE ... KDB(3)> dw msg_heap 8 look at message heap msg_heap+000000: 0000A02A CFFBF0B8 0000B02B CFFBF0B8 ...*.......+.... msg_heap+000010: 0000C02C CFFBF0B8 0000D02D CFFBF0B8 ...,.......-.... KDB(3)> mr s12 set SR12 with message heap SID s12 : 007FFFFF = 0000A02A KDB(3)> heap CFFBF0B8 print message heap Heap CFFBF000 sanity..... 48454150 base....... F0041000 lock@...... CFFBF008 lock....... 00000000 alt........ 00000001 numpages... 0000FFBF amount..... 00000000 pinflag.... 00000000 newheap.... 00000000 protect.... 00000000 limit...... 00000000 heap64..... 00000000 vmrelflag.. 00000000 rhash...... 00000000 pagtot..... 00000000 pagused.... 00000000 frtot[00].. 00000000 [01].. 00000000 [02].. 00000000 [03].. 00000000 frtot[04].. 00000000 [05].. 00000000 [06].. 00000000 [07].. 00000000 frtot[08].. 00000000 [09].. 00000000 [10].. 00000000 [11].. 00000000 frused[00]. 00000000 [01].. 00000000 [02].. 00000000 [03].. 00000000 frused[04]. 00000000 [05].. 00000000 [06].. 00000000 [07].. 00000000 frused[08]. 00000000 [09].. 00000000 [10].. 00000000 [11].. 00000000 fr[00]..... 00FFFFFF [01].. 00FFFFFF [02].. 00FFFFFF [03].. 00FFFFFF fr[04]..... 00FFFFFF [05].. 00FFFFFF [06].. 00FFFFFF [07].. 00FFFFFF fr[08]..... 00FFFFFF [09].. 00FFFFFF [10].. 00FFFFFF [11].. 00FFFFFF Heap CFFBF0B8 sanity..... 48454150 base....... F0040F48 lock@...... CFFBF0C0 lock....... 00000000 alt........ 00000000 numpages... 0000FFBF amount..... 00000100 pinflag.... 00000000 newheap.... 00000000 protect.... 00000000 limit...... 00000000 heap64..... 00000000 vmrelflag.. 00000000 rhash...... 00000000 pagtot..... 00000000 pagused.... 00000000 frtot[00].. 00000000 [01].. 00000000 [02].. 00000000 [03].. 00000000 frtot[04].. 00000000 [05].. 00000000 [06].. 00000000 [07].. 00000000 frtot[08].. 00000000 [09].. 00000000 [10].. 00000000 [11].. 00000000 frused[00]. 00000000 [01].. 00000000 [02].. 00000000 [03].. 00000000 frused[04]. 00000000 [05].. 00000000 [06].. 00000000 [07].. 00000000 frused[08]. 00000000 [09].. 00000000 [10].. 00000000 [11].. 00000000 fr[00]..... 00FFFFFF [01].. 00FFFFFF [02].. 00FFFFFF [03].. 00FFFFFF fr[04]..... 00FFFFFF [05].. 00FFFFFF [06].. 00FFFFFF [07].. 00FFFFFF fr[08]..... 00000000 [09].. 00FFFFFF [10].. 00FFFFFF [11].. 00FFFFFF addr...... 0000000000000000 maxpages.......... 00000000 peakpage.......... 00000000 limit_callout..... 00000000 newseg_callout.... 00000000 pagesoffset....... 00000194 pages_sid......... 00000000 Heap anchor ... CFFBF190 pageno FFFFFFFF pages.type.. 00 allocpage offset... 00000001 Heap Free list ... CFFBF198 pageno 00000001 pages.type.. 03 freerange offset... NO_PAGE ... CFFBF19C pageno 00000002 pages.type.. 04 freesize size..... 0000FFBE ... CFFFF08C pageno 0000FFBE pages.type.. 05 freerangeend offset... 00000001 Heap Alloc list KDB(3)> mr s12 reset SR12 s12 : 0000A02A = 007FFFFF
(0)> stat
RS6K_SMP_MCA POWER_PC POWER_604 machine with 8 cpu(s)
.......... SYSTEM STATUS
sysname... AIX nodename.. jumbo32
release... 3 version... 4
machine... 00920312A0 nid....... 920312A0
time of crash: Fri Jul 11 08:07:01 1997
age of system: 1 day, 20 hr., 31 min., 17 sec.
.......... PANIC STRING
Memdbg: *w == pat
(0)> xm -? Display usage
xmalloc <addr>
Print all available xmalloc information about <addr>.
If debug xmalloc kernel is available, also print out
information from xmalloc -s and xmalloc -h.
xmalloc -s <addr>
Print debug xmalloc allocation records matching associated with <addr>
(Debug xmalloc kernel only.)
xmalloc -h <addr>
Print records in debug xmalloc kernel free list associated with <addr>
(Debug xmalloc kernel only.)
xmalloc [-l] -f
Print allocation records on free list, from first-freed to last-freed.
If "-l" is specified, verbose records are printed
(Debug xmalloc kernel only.)
xmalloc [-l] -a
Print allocation record table.
If "-l" is specified, verbose records are printed.
(Debug xmalloc kernel only.)
xmalloc [-l] -p <pageno>
Print page descriptor information for page <pageno>.
If "-l" is specified, print extra info, even on failure.
xmalloc -d <addr>
Print debug xmalloc kernel allocation record hash chain that
is associated with the record hash value for <addr>.
xmalloc -v
Verifies allocation trailers of allocated records,
and free fill patterns of freed records.
(Debug xmalloc kernel only.)
xmalloc -u
Print xmalloc usage histogram. It tells the size the kernel heap
and how much of it has been used. Next, it prints a list of
allocated memory blocks sorted by allocation size, one per line.
Memory allocations of the same size from the same routine are coalesced,
keeping track of how many there were.
(0)> xm -s Display debug xmalloc status
Debug kernel error message: The xmfree service has found data written beyond the
end of the memory buffer that is being freed.
Address at fault was 0x09410200
(0)> xm -h 0x09410200 Display debug xmalloc records associated with addr
0B78DAB0: addr......... 09410200 req_size..... 128 freed unpinned
0B78DAB0: pid.......... 00043158 comm......... bcross
Trace during xmalloc() Trace during xmfree()
002329E4(.xmalloc+0000A8) 002328F0(.xmfree+0000FC)
00235CD4(.dlistadd+000040) 00234F04(.setbitmaps+0001BC)
00235520(.newblk+00006C) 00236894(.finicom+0001A4)
0B645120: addr......... 09410200 req_size..... 128 freed unpinned
0B645120: pid.......... 0007DCAC comm......... bcross
Trace during xmalloc() Trace during xmfree()
002329E4(.xmalloc+0000A8) 002328F0(.xmfree+0000FC)
00235CD4(.dlistadd+000040) 00236614(.logdfree+0001E8)
00236574(.logdfree+000148) 00236720(.finicom+000030)
0B7A3750: addr......... 09410200 req_size..... 128 freed unpinned
0B7A3750: pid.......... 000010BA comm......... syncd
Trace during xmalloc() Trace during xmfree()
002329E4(.xmalloc+0000A8) 002328F0(.xmfree+0000FC)
00235CD4(.dlistadd+000040) 00234F04(.setbitmaps+0001BC)
00235520(.newblk+00006C) 00236894(.finicom+0001A4)
0B52B330: addr......... 09410200 req_size..... 128 freed unpinned
0B52B330: pid.......... 00058702 comm......... bcross
Trace during xmalloc() Trace during xmfree()
002329E4(.xmalloc+0000A8) 002328F0(.xmfree+0000FC)
00235CD4(.dlistadd+000040) 00236698(.logdfree+00026C)
00236510(.logdfree+0000E4) 00236720(.finicom+000030)
07A33840: addr......... 09410200 req_size..... 133 freed unpinned
07A33840: pid.......... 00042C24 comm......... ksh
Trace during xmalloc() Trace during xmfree()
002329E4(.xmalloc+0000A8) 002328F0(.xmfree+0000FC)
00271F28(.ld_pathopen+000160) 00271D24(.ld_pathclear+00008C)
0027FB6C(.ld_getlib+000074) 002ABF04(.ld_execload+00075C)
0B796480: addr......... 09410200 req_size..... 133 freed unpinned
0B796480: pid.......... 0005C2E0 comm......... ksh
Trace during xmalloc() Trace during xmfree()
002329E4(.xmalloc+0000A8) 002328F0(.xmfree+0000FC)
00271F28(.ld_pathopen+000160) 00271D24(.ld_pathclear+00008C)
0027FB6C(.ld_getlib+000074) 002ABF04(.ld_execload+00075C)
07A31420: addr......... 09410200 req_size..... 135 freed unpinned
07A31420: pid.......... 0007161A comm......... ksh
Trace during xmalloc() Trace during xmfree()
002329E4(.xmalloc+0000A8) 002328F0(.xmfree+0000FC)
00271F28(.ld_pathopen+000160) 00271D24(.ld_pathclear+00008C)
0027FB6C(.ld_getlib+000074) 002ABF04(.ld_execload+00075C)
07A38630: addr......... 09410200 req_size..... 125 freed unpinned
07A38630: pid.......... 0001121E comm......... ksh
Trace during xmalloc() Trace during xmfree()
002329E4(.xmalloc+0000A8) 002328F0(.xmfree+0000FC)
00271F28(.ld_pathopen+000160) 00271D24(.ld_pathclear+00008C)
0027FB6C(.ld_getlib+000074) 002ABF04(.ld_execload+00075C)
07A3D240: addr......... 09410200 req_size..... 133 freed unpinned
07A3D240: pid.......... 0000654C comm......... ksh
Trace during xmalloc() Trace during xmfree()
002329E4(.xmalloc+0000A8) 002328F0(.xmfree+0000FC)
00271F28(.ld_pathopen+000160) 00271D24(.ld_pathclear+00008C)
0027FB6C(.ld_getlib+000074) 002ABF04(.ld_execload+00075C)
The xm subcommand can be used to find memory location of any heap record, knowing the page index (pageno) , or to find the heap record knowing the allocated memory location.
(0)> heap ... Heap Alloc list ... 0FFC41B0 pageno 00000007 pages.type.. 01 allocrange offset... NO_PAGE ... 0FFC41B4 pageno 00000008 pages.type.. 06 allocsize size..... 00001E00 ... 0FFCB9AC pageno 00001E06 pages.type.. 07 allocrangeend offset... 00000007 ... 0FFCB9B0 pageno 00001E07 pages.type.. 01 allocrange offset... NO_PAGE ... 0FFCB9B4 pageno 00001E08 pages.type.. 06 allocsize size..... 00001E00 ... 0FFD31AC pageno 00003C06 pages.type.. 07 allocrangeend offset... 00001E07 ... 0FFD31B4 pageno 00003C08 pages.type.. 01 allocrange offset... 00003C42 ... 0FFD31B8 pageno 00003C09 pages.type.. 06 allocsize size..... 00000002 ... 0FFD31C4 pageno 00003C0C pages.type.. 01 allocrange offset... NO_PAGE ... 0FFD31C8 pageno 00003C0D pages.type.. 06 allocsize size..... 00000009 ... 0FFD31E4 pageno 00003C14 pages.type.. 07 allocrangeend offset... 00003C0C ... (0)> xm -l -p 00001E07 how to find memory address of heap index 00001E07 type..................... 1 (P_allocrange) page_addr................ 02F82000 pinned................... 0 size..................... 00000000 offset................... 00FFFFFF page_descriptor_address.. 0FFCB9B0 (0)> xm -l 02F82000 how to find page index in kernel heap of 02F82000 P_allocrange (range of 2 or more allocated full pages) page........... 00001E07 start.......... 02F82000 page_cnt....... 00001E00 allocated_size. 01E00000 pinned......... unknown (0)> xm -l -p 00003C08 how to find memory address of heap index 00003C08 type..................... 1 (P_allocrange) page_addr................ 04D83000 pinned................... 0 size..................... 00000000 offset................... 00003C42 page_descriptor_address.. 0FFD31B4 (0)> xm -l 04D83000 ow to find page index in kernel heap of 04D83000 P_allocrange (range of 2 or more allocated full pages) page........... 00003C08 start.......... 04D83000 page_cnt....... 00000002 allocated_size. 00002000 pinned......... unknown
The bucket subcommand prints kernel memory allocator buckets.
KDB(0)> bucket ? print usage Usage: bucket [-l] [-c cpu] [-i index] [symb/eaddr] -l to display bucket free list -c to display only a cpu buckets -i to display only a bucket index symb/eaddr to display only one bucket KDB(0)> bucket -l -c 4 -i 13 print processor 4 8K bytes buckets displaying kmembucket for cpu 4 offset 13 size 0x00002000 address..................00376404 b_next..(x)..............0659F000 b_calls..(x).............0000AEBB b_total..(x).............00000003 b_totalfree..(x).........00000003 b_elmpercl..(x)..........00000001 b_highwat..(x)...........0000000A b_couldfree (sic)..(x)...00000000 b_failed..(x)............00000000 lock..(x)................00000000 Bucket free list..... 1 next...0659F000, kmemusage...09B57268 [000D 0001 00000004] 2 next...0619E000, kmemusage...09B55260 [000D 0001 00000004] 3 next...06687000, kmemusage...09B579A8 [000D 0001 00000004] KDB(0)> bucket -c 3 print all processor 3 buckets displaying kmembucket for cpu 3 offset 0 size 0x00000002 address..................00375F3C b_next..(x)..............00000000 b_calls..(x).............00000000 b_total..(x).............00000000 b_totalfree..(x).........00000000 b_elmpercl..(x)..........00001000 b_highwat..(x)...........00005000 b_couldfree (sic)..(x)...00000000 b_failed..(x)............00000000 lock..(x)................00000000 displaying kmembucket for cpu 3 offset 1 size 0x00000004 address..................00375F60 b_next..(x)..............00000000 b_calls..(x).............00000000 b_total..(x).............00000000 b_totalfree..(x).........00000000 b_elmpercl..(x)..........00000800 b_highwat..(x)...........00002800 b_couldfree (sic)..(x)...00000000 (0)> more (^C to quit) ? continue b_failed..(x)............00000000 lock..(x)................00000000 ... displaying kmembucket for cpu 3 offset 8 size 0x00000100 address..................0037605C b_next..(x)..............062A2700 b_calls..(x).............00B3F6EA b_total..(x).............00000330 b_totalfree..(x).........00000031 b_elmpercl..(x)..........00000010 b_highwat..(x)...........00000180 b_couldfree (sic)..(x)...00000000 b_failed..(x)............00000000 lock..(x)................00000000 displaying kmembucket for cpu 3 offset 9 size 0x00000200 address..................00376080 b_next..(x)..............05D30000 b_calls..(x).............0000A310 b_total..(x).............00000010 b_totalfree..(x).........0000000C b_elmpercl..(x)..........00000008 b_highwat..(x)...........00000028 b_couldfree (sic)..(x)...00000000 b_failed..(x)............00000000 lock..(x)................00000000 ... displaying kmembucket for cpu 3 offset 20 size 0x00200000 (0)> more (^C to quit) ? continue address..................0037620C b_next..(x)..............00000000 b_calls..(x).............00000000 b_total..(x).............00000000 b_totalfree..(x).........00000000 b_elmpercl..(x)..........00000001 b_highwat..(x)...........0000000A b_couldfree (sic)..(x)...00000000 b_failed..(x)............00000000 lock..(x)................00000000 KDB(0)>
The kmstats subcommand prints kernel allocator memory statistics.
KDB(0)> kmstats print allocator statistics displaying kmemstats for offset 0 free address..................0025C120 inuse..(x)...............00000000 calls..(x)...............00000000 memuse..(x)..............00000000 limit blocks..(x)........00000000 map blocks..(x)..........00000000 maxused..(x).............00000000 limit..(x)...............02666680 failed..(x)..............00000000 lock..(x)................00000000 displaying kmemstats for offset 1 mbuf address..................0025C144 inuse..(x)...............0000000D calls..(x)...............002C4E54 memuse..(x)..............00000D00 limit blocks..(x)........00000000 map blocks..(x)..........00000000 maxused..(x).............0001D700 limit..(x)...............02666680 (0)> more (^C to quit) ? continue failed..(x)..............00000000 lock..(x)................00000000 displaying kmemstats for offset 2 mcluster address..................0025C168 inuse..(x)...............00000002 calls..(x)...............00023D4E memuse..(x)..............00000900 limit blocks..(x)........00000000 map blocks..(x)..........00000000 maxused..(x).............00079C00 limit..(x)...............02666680 failed..(x)..............00000000 lock..(x)................00000000 ... displaying kmemstats for offset 48 kalloc address..................0025C7E0 inuse..(x)...............00000000 calls..(x)...............00000000 memuse..(x)..............00000000 limit blocks..(x)........00000000 map blocks..(x)..........00000000 maxused..(x).............00000000 limit..(x)...............02666680 failed..(x)..............00000000 lock..(x)................00000000 displaying kmemstats for offset 49 temp address..................0025C804 inuse..(x)...............00000007 calls..(x)...............00000007 memuse..(x)..............00003500 (0)> more (^C to quit) ? continue limit blocks..(x)........00000000 map blocks..(x)..........00000000 maxused..(x).............00003500 limit..(x)...............02666680 failed..(x)..............00000000 lock..(x)................00000000 KDB(0)>