<html><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">Hello<div><br></div><div>We've been running the ovz kernel for a while, but we're experiencing random kernel panics on a moderately loaded server.</div><div><br></div><div>The server is Centos 5.2, 2.6.18-92.1.13.el5.028stab059.6PAE kernel running the ata-over-ethernet driver version aoe6-69 from <a href="http://support.coraid.com/support/linux/">http://support.coraid.com/support/linux/</a></div><div><br></div><div>The server in question ran a memtest for over 48 hours and showed nothing, so I don't suspect memory errors...</div><div><br></div><div>I have several servers with 'normal' centos kernel (no openvz) with aoe driver accessing storage heavily, these have been running stable for months. As soon as ovz kernel is booted the machine starts crashing.</div><div><br></div><div>I suspect there is a problem between openvz kernel patches and the aoe driver, but attempts to track down the problem with the aoe driver maintainer have been unsuccessful so far...</div><div><br></div><div>Here are his last comments in case it helps any:</div><div><br></div><div><blockquote type="cite" class=""><span class="Apple-style-span" style="color: rgb(0, 0, 0); ">When I looked at your last trace, it appeared that the list of free<br>pages was the one containing the list corruption that triggered the<br>BUG inside of list_del. Backtraces are often incomplete, and here is<br>an expanded version of what I think happened between<br>get_page_from_freelist and the BUG in list_del.<br><br> mm/page_alloc.c:<span class="Apple-tab-span" style="white-space: pre; "> </span>get_page_from_freelist calls buffered_rmqueue<br><br> mm/page_alloc.c:<span class="Apple-tab-span" style="white-space: pre; "> </span>buffered_rmqueue calls list_del(&page->lru);<br><br> lib/list_debug.c:<span class="Apple-tab-span" style="white-space: pre; "> </span>list_del finds that ...<br><span class="Apple-tab-span" style="white-space: pre; "> </span><span class="Apple-tab-span" style="white-space: pre; "> </span><span class="Apple-tab-span" style="white-space: pre; "> </span>page->lru->prev->next == page->lru, but<br><span class="Apple-tab-span" style="white-space: pre; "> </span><span class="Apple-tab-span" style="white-space: pre; "> </span><span class="Apple-tab-span" style="white-space: pre; "> </span>page->lru->next->prev != page->lru<br><br>It could mean that something is keeping a pointer to a page on the<br>free list after it has been freed. That's why I thought the patches<br>related to the page->_count would be relevant: the aoe driver has to<br>manipulate the count so that the put_page done by the network layer<br>doesn't free the page (associated with an AoE read or write) before<br>the block layer is done with it.</span></blockquote></div><div><br></div><div>I found the following link <a href="http://bugs.centos.org/view.php?id=3192">http://bugs.centos.org/view.php?id=3192</a> pointing at a similar bug/error in the centos kernel but this problem appeared to go away for that user in -92.1.17. So I tried upgrading to 2.6.18-92.1.18.el5.028stab060.2PAE yesterday but it has since crashed two more times.</div><div><br></div><div>I have lots and lots of crash traces, but they all look pretty much the same so I will just post one below. If anyone has any ideas what might be going on here I'd be very interested. If any more info is needed, just let me know.</div><div><br></div><div>There are some notes here about the setup process I used for this server: <a href="http://code.fubra.com/wiki/AoeOpenvzTesting">http://code.fubra.com/wiki/AoeOpenvzTesting</a></div><div><br></div><div>Thanks!</div><div><br></div><div>Mark</div><div><br></div><div>>>TRACE FOLLOWS>></div><div>list_add corruption. next->prev should be c06cdb0c, but was c14c6170<br>------------[ cut here ]------------<br>kernel BUG at lib/list_debug.c:26!<br>invalid opcode: 0000 [#1]<br>SMP<br>last sysfs file:<br>Modules linked in: nfs(U) lockd(U) nfs_acl(U) vzethdev(U) vznetdev(U) simfs(U) vzrst(U) ip_nat(U) vzcpt(U) ip_conntrack(U) nfnetlink(U) ipip(U) tunnel4(U) tun(U) vzmon(U) xt_tcpudp(U) xt_length(U) ipt_ttl(U) xt_tcpmss(U) ipt_TCPMSS(U) iptable_mangle(U) iptable_filter(U) xt_multiport(U) xt_limit(U) ipt_tos(U) ipt_REJECT(U) ip_tables(U) x_tables(U) aoe(U) vzdquota(U) autofs4(U) hidp(U) rfcomm(U) l2cap(U) bluetooth(U) sunrpc(U) 8021q(U) ipv6(U) xfrm_nalgo(U) crypto_api(U) vzdev(U) dm_multipath(U) video(U) sbs(U) backlight(U) i2c_ec(U) container(U) button(U) battery(U) asus_acpi(U) ac(U) parport_pc(U) lp(U) parport(U) e7xxx_edac(U) edac_mc(U) e100(U) i2c_i801(U) serio_raw(U) eepro100(U) i2c_core(U) mii(U) e1000(U) pcspkr(U) e1000e(U) dm_snapshot(U) dm_zero(U) dm_mirror(U) dm_mod(U) ata_piix(U) libata(U) sd_mod(U) scsi_mod(U) ext3(U) jbd(U) ehci_hcd(U) ohci_hcd(U) uhci_hcd(U)<br>CPU: 0, VCPU: 0.0<br>EIP: 0060:[<c04db0be>] Tainted: P VLI<br>EFLAGS: 00010046 (2.6.18-92.1.18.el5.028stab060.2PAE #1 028stab060)<br>EIP is at __list_add+0x1c/0x58<br>eax: 00000048 ebx: c14a3c28 ecx: f517dea0 edx: c063572e<br>esi: c06cdb0c edi: c14ead1c ebp: c06cdb0c esp: f517de9c<br>ds: 007b es: 007b ss: 0068<br>Process aoe_ktio (pid: 8451, veid: 0, ti=f517c000 task=f7c3a670 task.ti=f517c000)<br>Stack: c063572e c06cdb0c c14c6170 c14ead04 c06cdb00 00000046 c0457b85 c06cda80<br> f7f4afc0 e1c863a0 c14ead04 00000000 c0456560 e0c64380 e1c863a0 e0c64300<br> c045dd33 00000000 f7f4afc0 00000000 e0c64300 00001000 c045de6e 00000000<br>Call Trace:<br>[<c0457b85>] free_hot_cold_page+0x110/0x13a<br>[<c0456560>] mempool_free+0x5f/0x63<br>[<c045dd33>] bounce_end_io+0x51/0x80<br>[<c045de6e>] bounce_end_io_write+0x0/0x1b<br>[<c045de84>] bounce_end_io_write+0x16/0x1b<br>[<c0476c05>] bio_endio+0x50/0x55<br>[<c04c9e20>] __end_that_request_first+0x185/0x47c<br>[<f904f542>] aoe_end_request+0x46/0x78 [aoe]<br>[<f9050def>] ktio+0x3dc/0x475 [aoe]<br>[<f9050730>] kthread+0x87/0xdf [aoe]<br>[<c041acd2>] default_wake_function+0x0/0xc<br>[<f90506a9>] kthread+0x0/0xdf [aoe]<br>[<c04349db>] kthread+0xc0/0xed<br>[<c043491b>] kthread+0x0/0xed<br>[<c0607fcb>] kernel_thread_helper+0x7/0x10<br>=======================<br>Code: c7 43 04 00 02 20 00 c7 03 00 01 10 00 5b c3 57 89 c7 56 89 d6 53 8b 41 04 89 cb 39 d0 74 1a 50 52 68 2e 57 63 c0 e8 a3 7f f4 ff <0f> 0b 66 b8 1a 00 b8 e0 56 63 c0 83 c4 0c 8b 06 39 d8 74 1a 50<br>EIP: [<c04db0be>] __list_add+0x1c/0x58 SS:ESP 0068:f517de9c<br>Kernel panic - not syncing: Fatal exception<br>BUG: warning at arch/i386/kernel/smp.c:543/smp_call_function() (Tainted: P )<br>[<c04115bc>] stop_this_cpu+0x0/0x33<br>[<c04112ef>] smp_call_function+0x57/0xc3<br>[<c0423079>] printk+0x18/0x8e<br>[<c041136e>] smp_send_stop+0x13/0x1c<br>[<c04222cf>] panic+0x4c/0x165<br>[<c04056dc>] die+0x249/0x260<br>[<c0405e16>] do_invalid_op+0x0/0x9d<br>[<c0405ea7>] do_invalid_op+0x91/0x9d<br>[<c04db0be>] __list_add+0x1c/0x58<br>[<c0422833>] wake_up_klogd+0x2b/0x2d<br>[<c042287c>] release_console_sem+0x47/0x4a<br>[<c046e07b>] free_block+0xd0/0xef<br>[<c046e1a5>] cache_flusharray+0x76/0xa2<br>[<c0607dfb>] error_code+0x4f/0x54<br>[<c04db0be>] __list_add+0x1c/0x58<br>[<c0457b85>] free_hot_cold_page+0x110/0x13a<br>[<c0456560>] mempool_free+0x5f/0x63<br>[<c045dd33>] bounce_end_io+0x51/0x80<br>[<c045de6e>] bounce_end_io_write+0x0/0x1b<br>[<c045de84>] bounce_end_io_write+0x16/0x1b<br>[<c0476c05>] bio_endio+0x50/0x55<br>[<c04c9e20>] __end_that_request_first+0x185/0x47c<br>[<f904f542>] aoe_end_request+0x46/0x78 [aoe]<br>[<f9050def>] ktio+0x3dc/0x475 [aoe]<br>[<f9050730>] kthread+0x87/0xdf [aoe]<br>[<c041acd2>] default_wake_function+0x0/0xc<br>[<f90506a9>] kthread+0x0/0xdf [aoe]<br>[<c04349db>] kthread+0xc0/0xed<br>[<c043491b>] kthread+0x0/0xed<br>[<c0607fcb>] kernel_thread_helper+0x7/0x10<br>=======================</div><div><br></div></body></html>