[Devel] 2.6.26 panic (skb->dev==NULL), NFS support

[Karel Tuma]

Hello list,

I hope this is the right place to post.

I've recently moved to 2.6.26 git (i hope that's the bleeding edge) OpenVZ
to find out how usable it is. I'm running it on a box under fair IO load
(100-300 BIO/s). The thing panics in net/ipv4/tcp_ipv4.c:tcp_v4_send_ack()
once every couple of hours, during io spikes. Apparently skb->dev is NULL,
inside an irq context on top of that. It's very hard to pinpoint the actual
trigger. Vanilla 2.6.26 runs just fine.

Treating the symptoms, rather than the reason behind it, by testing
the pointer seems to work ok, without disrupting any service. However,
it would be nice to figure out the real cause to avoid such a hideous hack.

NFS seems to work (mounting from CT0 so far) after minor changes for
/proc interfacing (it was throwing ENOMEM). It's mounted in udp so it's
unlikely to be the cause for tcp panic.

oops, ugly fix and nfs patches are attached.

With regards,

// Karel, http://leet.cz

-------------- next part --------------
diff --git a/fs/nfs/client.c b/fs/nfs/client.c
index f2a092c..77b47de 100644
--- a/fs/nfs/client.c
+++ b/fs/nfs/client.c
@@ -1502,7 +1502,7 @@ int __init nfs_fs_proc_init(void)
 {
 	struct proc_dir_entry *p;
 
-	proc_fs_nfs = proc_mkdir("fs/nfsfs", NULL);
+	proc_fs_nfs = proc_mkdir("fs/nfsfs", &glob_proc_root);
 	if (!proc_fs_nfs)
 		goto error_0;
 
@@ -1536,7 +1536,7 @@ void nfs_fs_proc_exit(void)
 {
 	remove_proc_entry("volumes", proc_fs_nfs);
 	remove_proc_entry("servers", proc_fs_nfs);
-	remove_proc_entry("fs/nfsfs", NULL);
+	remove_proc_entry("fs/nfsfs", &glob_proc_root);
 }
 
 #endif /* CONFIG_PROC_FS */

-------------- next part --------------
BUG: unable to handle kernel NULL pointer dereference at 00000000000003a8
IP: [<ffffffff804337a1>] tcp_v4_send_ack+0xf1/0x170
PGD 712ca067 PUD 7eec8067 PMD 0
Oops: 0000 [1]
CPU: 0
Modules linked in: simfs vzethdev vzmon vzdquota vzdev nfs lockd nfs_acl sunrpc dm_drivenc tun 8021q bridge llc ipv6 fuse dm_snapshot dm_mirror dm_log dm_mod netconsole loop evdev pcspkr serio_raw i2c_i801 sky2 e1000 [last unloaded: firmware_class]
Pid: 0, comm: swapper Not tainted 2.6.26 #6 036test001
RIP: 0010:[<ffffffff804337a1>]  [<ffffffff804337a1>] tcp_v4_send_ack+0xf1/0x170
RSP: 0018:ffffffff805c1cd0  EFLAGS: 00010246
RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000014
RDX: ffffffff805c1cd0 RSI: ffff81007efaa580 RDI: 0000000000000000
RBP: ffff810030a12800 R08: 00000000000016d0 R09: 0000000000000014
R10: ffff8100713ee434 R11: 000000005b139152 R12: 0000000000000000
R13: ffff81007efaa580 R14: ffff810073705340 R15: ffff8100711fa468
FS:  0000000000000000(0000) GS:ffffffff8057d000(0000) knlGS:0000000000000000
CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 00000000000003a8 CR3: 0000000030b40000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process swapper (pid: 0, veid=0, threadinfo ffffffff80584000, task ffffffff80543340)
Stack:  ffffffff805c1cf0 0000000000000014 000000082ddef414 0000000000000000
5b1391520604bd01 d016105095bdbbf6 ffff810000000000 ffff810073705340
0000000000000000 ffffffff80437068 ffffffffa00bd050 0000000080000000
Call Trace:
<IRQ>  [<ffffffff80437068>] ? tcp_check_req+0x368/0x3f0
[<ffffffffa00bd050>] ? :bridge:br_nf_pre_routing_finish+0x0/0x300
[<ffffffff80434988>] ? tcp_v4_do_rcv+0x178/0x240
[<ffffffff80436795>] ? tcp_v4_rcv+0x5c5/0x6c0
[<ffffffff8041839d>] ? ip_local_deliver_finish+0x10d/0x1e0
[<ffffffff80418074>] ? ip_rcv_finish+0x144/0x360
[<ffffffff804187cf>] ? ip_rcv+0x23f/0x2e0
[<ffffffff803f5f17>] ? netif_receive_skb+0x337/0x480
[<ffffffff803f8c8f>] ? process_backlog+0x6f/0xd0
[<ffffffff803f882e>] ? net_rx_action+0xce/0x180
[<ffffffffa0000f86>] ? :e1000:e1000_set_itr+0x86/0x150
[<ffffffff8022ceba>] ? __do_softirq+0x8a/0x120
[<ffffffff8020c14c>] ? call_softirq+0x1c/0x30
[<ffffffff8020e005>] ? do_softirq+0x35/0x70
[<ffffffff8020e381>] ? do_IRQ+0x61/0xc0
[<ffffffff80211920>] ? mwait_idle+0x0/0x50
[<ffffffff8020b9a1>] ? ret_from_intr+0x0/0xa
<EOI>  [<ffffffff804234c0>] ? tcp_poll+0x0/0x200
[<ffffffff8021195e>] ? mwait_idle+0x3e/0x50
[<ffffffff8020a103>] ? cpu_idle+0x33/0x60
Code: 0c 13 41 10 11 d0 83 d0 00 48 85 ff 89 44 24 10 c7 44 24 14 08 00 00 00 74 07 8b 47 04 89 44 24 18 48 8b 46 20 48 89 e2 44 89 c9 <48> 8b 80 a8 03 00 00 48 8b b8 30 01 00 00 e8 0c 9f fe  RSP <ffffffff805c1cd0>
CR2: 00000000000003a8
Kernel panic - not syncing: Fatal exception
Rebooting in 1 seconds..md: bind<sdi2>

-------------- next part --------------
diff --git a/net/ipv4/tcp_ipv4.c b/net/ipv4/tcp_ipv4.c
index ca6b5d3..f7571c1 100644
--- a/net/ipv4/tcp_ipv4.c
+++ b/net/ipv4/tcp_ipv4.c
@@ -689,6 +689,9 @@ static void tcp_v4_send_ack(struct tcp_timewait_sock *twsk,
 	if (twsk)
 		arg.bound_dev_if = twsk->tw_sk.tw_bound_dev_if;
 
+	if (!skb->dev) { printk("Hey, skb->dev is NULL, this is bad. I'll save you from panic this time ... How did you trigger this anyway?"); return; };
 	ip_send_reply(dev_net(skb->dev)->ipv4.tcp_sock, skb,
 		      &arg, arg.iov[0].iov_len);