[Devel] Kernel Panic - Why and Troubleshooting?

[Hans Zaunere]

Hello,

Recently one of our host systems had the following panic and had to be power
cycled.


Nov  3 15:05:40 vzhn1 kernel: general protection fault: 0000 [1] SMP
Nov  3 15:05:40 vzhn1 kernel: last sysfs file:
Nov  3 15:05:40 vzhn1 kernel: CPU: 0
Nov  3 15:05:40 vzhn1 kernel: Modules linked in: vzethdev(U) simfs(U)
vzrst(U) ip_nat(U) vzcpt(U) ip_conntrack(U) nfnetlink(U) vzdquota(U)
xt_tcpudp(U) xt_length(U) ipt_ttl(U) x
t_tcpmss(U) ipt_TCPMSS(U) iptable_mangle(U) iptable_filter(U)
xt_multiport(U) xt_limit(U) ipt_tos(U) ipt_REJECT(U) ip_tables(U)
x_tables(U) vznetdev(U) vzmon(U) ipv6(U) xfrm_nal
go(U) crypto_api(U) vzdev(U) dm_mirror(U) dm_multipath(U) scsi_dh(U)
video(U) hwmon(U) backlight(U) sbs(U) i2c_ec(U) button(U) battery(U)
asus_acpi(U) acpi_memhotplug(U) ac(U) l
p(U) snd_hda_intel(U) snd_seq_dummy(U) snd_seq_oss(U) snd_seq_midi_event(U)
snd_seq(U) snd_seq_device(U) snd_pcm_oss(U) snd_mixer_oss(U) snd_pcm(U)
snd_timer(U) snd_page_alloc(U
) snd_hwdep(U) parport_pc(U) snd(U) r8169(U) i2c_i801(U) sg(U) e1000e(U)
i2c_core(U) soundcore(U) mii(U) parport(U) pcspkr(U) serio_raw(U)
dm_raid45(U) dm_message(U) dm_region_h
ash(U) dm_log(U) dm_mod(U) dm_mem_cache(U) ahci(U) libata(U) shpchp(U)
aacraid(U) sd_mod(U) scsi_mod(U) ext3(U) jbd(U) uhci_hcd(U) ohci_hcd(
Nov  3 15:05:40 vzhn1 kernel: ) ehci_hcd(U)
Nov  3 15:05:40 vzhn1 kernel: Pid: 309, comm: kswapd0 Tainted:  P
2.6.18-128.1.1.el5.028stab062.3 #1 028stab062
Nov  3 15:05:40 vzhn1 kernel: RIP: 0060:[<ffffffff800f0fb1>]
[<ffffffff800f0fb1>] d_free+0x10/0x48
Nov  3 15:05:40 vzhn1 kernel: RSP: 0000:ffff81022a45fd00  EFLAGS: 00010206
Nov  3 15:05:40 vzhn1 kernel: RAX: 4000000000000000 RBX: ffff810169bdbb18
RCX: ffff810169bdbb30
Nov  3 15:05:40 vzhn1 kernel: RDX: ffff8100a935f1c8 RSI: 0000000000000282
RDI: ffff810169bdbb18
Nov  3 15:05:40 vzhn1 kernel: RBP: ffff810169bdbb18 R08: 0000000000000282
R09: ffff810107c24d00
Nov  3 15:05:40 vzhn1 kernel: R10: 000000000000000a R11: 0000000000000282
R12: ffff810169bdbb18
Nov  3 15:05:40 vzhn1 kernel: R13: 0000000000000075 R14: 00000000000000c8
R15: 0000000000000100
Nov  3 15:05:40 vzhn1 kernel: FS:  0000000000000000(0000)
GS:ffffffff803b7000(0000) knlGS:0000000000000000
Nov  3 15:05:40 vzhn1 kernel: CS:  0060 DS: 0018 ES: 0018 CR0:
000000008005003b
Nov  3 15:05:40 vzhn1 kernel: CR2: 00000000b74ac000 CR3: 00000001df214000
CR4: 00000000000006e0
Nov  3 15:05:40 vzhn1 kernel: Process kswapd0 (pid: 309, veid=0, threadinfo
ffff81022a45e000, task ffff81022a45abb0)
Nov  3 15:05:40 vzhn1 kernel: Stack:  ffff8100a935f168 ffffffff800f1017
0000000000000282 ffff810229bb5470
Nov  3 15:05:40 vzhn1 kernel:  0000000000000000 ffffffff800f10d4
ffff810229bb5470 ffffffff8002f9c6
Nov  3 15:05:40 vzhn1 kernel:  ffff81022a45fd40 00000000ffffffff
00000000002a570b 00000000ac3fb70c
Nov  3 15:05:40 vzhn1 kernel: Call Trace:
Nov  3 15:05:40 vzhn1 kernel:  [<ffffffff800f1017>] d_kill+0x2e/0x43
Nov  3 15:05:40 vzhn1 kernel:  [<ffffffff800f10d4>]
prune_one_dentry+0xa8/0xc2
Nov  3 15:05:40 vzhn1 kernel:  [<ffffffff8002f9c6>] prune_dcache+0x10f/0x149
Nov  3 15:05:40 vzhn1 kernel:  [<ffffffff800f1de3>]
shrink_dcache_memory+0x34/0xef
Nov  3 15:05:40 vzhn1 kernel:  [<ffffffff80040b70>] shrink_slab+0x108/0x187
Nov  3 15:05:40 vzhn1 kernel:  [<ffffffff8005a60e>] kswapd+0x37a/0x48f
Nov  3 15:05:40 vzhn1 kernel:  [<ffffffff8009ee6b>]
autoremove_wake_function+0x0/0x2e
Nov  3 15:05:40 vzhn1 kernel:  [<ffffffff8009ec45>]
keventd_create_kthread+0x0/0xc4
Nov  3 15:05:40 vzhn1 kernel:  [<ffffffff8005a294>] kswapd+0x0/0x48f
Nov  3 15:05:40 vzhn1 kernel:  [<ffffffff8009ec45>]
keventd_create_kthread+0x0/0xc4
Nov  3 15:05:40 vzhn1 kernel:  [<ffffffff80033577>] kthread+0xfe/0x132
Nov  3 15:05:40 vzhn1 kernel:  [<ffffffff80061001>] child_rip+0xa/0x11
Nov  3 15:05:40 vzhn1 kernel:  [<ffffffff8009ec45>]
keventd_create_kthread+0x0/0xc4
Nov  3 15:05:40 vzhn1 kernel:  [<ffffffff80033479>] kthread+0x0/0x132
Nov  3 15:05:40 vzhn1 kernel:  [<ffffffff80060ff7>] child_rip+0x0/0x11
Nov  3 15:05:40 vzhn1 kernel:
Nov  3 15:05:40 vzhn1 kernel:
Nov  3 15:05:40 vzhn1 kernel: Code: 48 8b 40 20 48 85 c0 74 02 ff d0 48 8b
bb a0 00 00 00 48 85
Nov  3 15:05:40 vzhn1 kernel: RIP  [<ffffffff800f0fb1>] d_free+0x10/0x48
Nov  3 15:05:40 vzhn1 kernel:  RSP <ffff81022a45fd00>
Nov  3 15:05:40 vzhn1 kernel: Kernel panic - not syncing: Fatal exception

uname:
Linux xxx.xxx.xxx 2.6.18-128.1.1.el5.028stab062.3 #1 SMP Sun May 10 18:54:51
MSD 2009 x86_64 x86_64 x86_64 GNU/Linux

Any thoughts on the cause and preventing this from happening again?  Or
ideas on how to trouble shoot further?

Thanks,

Hans