[Users] stab93.5 crashing processes

Mon Dec 8 05:00:01 PST 2014

Dear Benjamin,

similar issues are processed in
https://bugzilla.openvz.org/show_bug.cgi?id=3140

Thank you,
	Vasily Averin

On 03.12.2014 18:27, Benjamin Henrion wrote:
> On Wed, Dec 3, 2014 at 4:13 PM, Benjamin Henrion <zoobab at gmail.com> wrote:
>> Hi,
>>
>> Just to let you know that I have multiple HNs with the kernel 93.5
>> crashing processes with the following kernel messages:
>>
>> ===========================================================
>> [3676904.929199] ------------[ cut here ]------------
>> [3676904.929243] WARNING: at fs/ext4/super.c:250
>> ext4_journal_start_sb+0xce/0xe0 [ext4]() (Tainted: G        W
>> ---------------   )
>> [3676904.929247] Hardware name: System x3650 M3 -[7945K3G]-
>> [3676904.929249] Modules linked in: dm_snapshot vzethdev vznetdev
>> pio_nfs pio_direct pfmt_raw pfmt_ploop1 ploop simfs vzrst vzcpt nfs
>> lockd fscache auth_rpcgss nfs_acl sunrpc vziolimit vzmon ipt_REDIRECT
>> nf_nat_irc nf_nat_ftp iptable_nat nf_nat xt_helper xt_conntrack
>> nf_conntrack_irc nf_conntrack_ftp xt_length ipt_LOG xt_hl xt_tcpmss
>> xt_TCPMSS xt_DSCP xt_dscp xt_limit iptable_mangle fuse vzdquota
>> vzevent xt_multiport autofs4 bridge vzdev bonding 8021q garp stp llc
>> ipt_REJECT iptable_filter nf_conntrack_ipv4 nf_defrag_ipv4 ip_tables
>> ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack
>> ip6table_filter ip6_tables ipv6 vfat fat tpm_tis tpm tpm_bios iTCO_wdt
>> iTCO_vendor_support ipmi_devintf bnx2 cdc_ether usbnet mii serio_raw
>> i2c_i801 i2c_core lpc_ich mfd_core sg ioatdma dca i7core_edac
>> edac_core shpchp ext4 jbd2 mbcache aesni_intel ablk_helper cryptd lrw
>> glue_helper aes_x86_64 aes_generic xts gf128mul dm_crypt sr_mod cdrom
>> sd_mod crc_t10dif pata_acpi ata_generic ata_piix usb_storage
>> megaraid_sas dm_mirror dm_region_hash dm_log dm_mod [last unloaded:
>> sunrpc]
>> [3676904.929313] Pid: 572345, comm: gunicorn: worke veid: 29301
>> Tainted: G        W  ---------------    2.6.32-042stab093.5 #1
>> [3676904.929316] Call Trace:
>> [3676904.929326]  [<ffffffff810755d7>] ? warn_slowpath_common+0x87/0xc0
>> [3676904.929330]  [<ffffffff8107562a>] ? warn_slowpath_null+0x1a/0x20
>> [3676904.929342]  [<ffffffffa0130d8e>] ? ext4_journal_start_sb+0xce/0xe0 [ext4]
>> [3676904.929353]  [<ffffffffa0117d8a>] ? ext4_dirty_inode+0x2a/0x60 [ext4]
>> [3676904.929360]  [<ffffffff811dbd3b>] ? __mark_inode_dirty+0x3b/0x190
>> [3676904.929365]  [<ffffffff811cc95c>] ? inode_setattr+0x4c/0x60
>> [3676904.929376]  [<ffffffffa011ab91>] ? ext4_setattr+0x101/0x3a0 [ext4]
>> [3676904.929382]  [<ffffffff811a8e8b>] ? put_unused_fd+0x3b/0x90
>> [3676904.929385]  [<ffffffff811cccd1>] ? notify_change+0x111/0x340
>> [3676904.929389]  [<ffffffff811aa08a>] ? sys_fchmod+0xfa/0x130
>> [3676904.929395]  [<ffffffff810f4e9e>] ? __audit_syscall_exit+0x25e/0x290
>> [3676904.929403]  [<ffffffff8100b102>] ? system_call_fastpath+0x16/0x1b
>> [3676904.929406] ---[ end trace 1a72102dcc568583 ]---
>> [3676904.929408] Tainting kernel with flag 0x9
>> [3676904.929410] Pid: 572345, comm: gunicorn: worke veid: 29301
>> Tainted: G        W  ---------------    2.6.32-042stab093.5 #1
>> [3676904.929413] Call Trace:
>> [3676904.929416]  [<ffffffff81075461>] ? add_taint+0x71/0x80
>> [3676904.929419]  [<ffffffff810755e4>] ? warn_slowpath_common+0x94/0xc0
>> [3676904.929423]  [<ffffffff8107562a>] ? warn_slowpath_null+0x1a/0x20
>> [3676904.929434]  [<ffffffffa0130d8e>] ? ext4_journal_start_sb+0xce/0xe0 [ext4]
>> [3676904.929445]  [<ffffffffa0117d8a>] ? ext4_dirty_inode+0x2a/0x60 [ext4]
>> [3676904.929448]  [<ffffffff811dbd3b>] ? __mark_inode_dirty+0x3b/0x190
>> [3676904.929451]  [<ffffffff811cc95c>] ? inode_setattr+0x4c/0x60
>> [3676904.929462]  [<ffffffffa011ab91>] ? ext4_setattr+0x101/0x3a0 [ext4]
>> [3676904.929465]  [<ffffffff811a8e8b>] ? put_unused_fd+0x3b/0x90
>> [3676904.929468]  [<ffffffff811cccd1>] ? notify_change+0x111/0x340
>> [3676904.929472]  [<ffffffff811aa08a>] ? sys_fchmod+0xfa/0x130
>> [3676904.929475]  [<ffffffff810f4e9e>] ? __audit_syscall_exit+0x25e/0x290
>> [3676904.929479]  [<ffffffff8100b102>] ? system_call_fastpath+0x16/0x1b
>> ===========================================================
>>
>> I have upgraded to 2.6.32-042stab094.7 and the bug seems to disappear.
> 
> I have found a machine with "2.6.32-042stab094.7", and this critical
> bug is still present.
> 
> I smell that it might be linked to ext4:
> 
> ==================================================================
> [ 5017.634363] device veth10801.2 entered promiscuous mode
> [ 5017.634490] vzbr2: topology change detected, propagating
> [ 5017.634500] vzbr2: port 2(veth10801.2) entering forwarding state
> [13529.132994] Core dump to |/usr/libexec/abrt-hook-ccpp 11 0 20953 0
> 0 1417105614 e pipe failed
> [13529.133878] Core dump to |/usr/libexec/abrt-hook-ccpp 11 0 20976 0
> 0 1417105614 e pipe failed
> [155454.976611] Core dump to |/usr/libexec/abrt-hook-ccpp 11 0 19603 0
> 0 1417247584 e pipe failed
> [178839.543158] Core dump to |/usr/libexec/abrt-hook-ccpp 11 0 29355 0
> 0 1417270976 e pipe failed
> [214904.940995] EXT4-fs (dm-6): ext4_orphan_cleanup: deleting
> unreferenced inode 9971401
> [214904.941021] EXT4-fs (dm-6): ext4_orphan_cleanup: deleting
> unreferenced inode 9971399
> [214904.941029] EXT4-fs (dm-6): ext4_orphan_cleanup: deleting
> unreferenced inode 9971394
> [214904.941037] EXT4-fs (dm-6): ext4_orphan_cleanup: deleting
> unreferenced inode 9971393
> [214904.941046] EXT4-fs (dm-6): ext4_orphan_cleanup: deleting
> unreferenced inode 9971392
> [214904.950216] EXT4-fs (dm-6): ext4_orphan_cleanup: deleting
> unreferenced inode 10528588
> [214904.958245] EXT4-fs (dm-6): ext4_orphan_cleanup: deleting
> unreferenced inode 10285608
> [214904.960827] EXT4-fs (dm-6): 7 orphan inodes deleted
> [214904.962010] EXT4-fs (dm-6): recovery complete
> [214904.962165] EXT4-fs (dm-6): mounted filesystem with ordered data mode. Opts:
> [220030.090424] EXT4-fs (dm-6): ext4_orphan_cleanup: deleting
> unreferenced inode 9971401
> [220030.090902] EXT4-fs (dm-6): ext4_orphan_cleanup: deleting
> unreferenced inode 9971399
> [220030.091063] EXT4-fs (dm-6): ext4_orphan_cleanup: deleting
> unreferenced inode 9971394
> [220030.091222] EXT4-fs (dm-6): ext4_orphan_cleanup: deleting
> unreferenced inode 9971393
> [220030.092382] EXT4-fs (dm-6): ext4_orphan_cleanup: deleting
> unreferenced inode 9971392
> [220030.112088] EXT4-fs (dm-6): ext4_orphan_cleanup: deleting
> unreferenced inode 10528588
> [220030.120215] EXT4-fs (dm-6): ext4_orphan_cleanup: deleting
> unreferenced inode 10285608
> [220030.131763] EXT4-fs (dm-6): 7 orphan inodes deleted
> [220030.139871] EXT4-fs (dm-6): recovery complete
> [220030.140168] EXT4-fs (dm-6): mounted filesystem with ordered data mode. Opts:
> [223371.666000] ------------[ cut here ]------------
> [223371.666031] WARNING: at fs/ext4/super.c:250
> ext4_journal_start_sb+0xce/0xe0 [ext4]() (Not tainted)
> [223371.666034] Hardware name: PowerEdge R620
> [223371.666036] Modules linked in: dm_snapshot vzethdev pio_nfs
> pio_direct pfmt_raw pfmt_ploop1 ploop simfs vzrst vzcpt nfs lockd
> fscache auth_rpcgss nfs_acl sunrpc vziolimit vzdquota ipt_REDIRECT
> nf_nat_irc nf_nat_ftp xt_helper xt_conntrack nf_conntrack_irc
> nf_conntrack_ftp xt_length ipt_LOG xt_hl xt_tcpmss xt_TCPMSS xt_DSCP
> xt_dscp xt_limit vzevent xt_multiport autofs4 bridge vznetdev vzmon
> vzdev bonding 8021q garp stp llc ipt_REJECT iptable_filter
> iptable_mangle iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4
> ip_tables ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state
> nf_conntrack ip6table_filter ip6_tables ipv6 ipmi_devintf iTCO_wdt
> iTCO_vendor_support acpi_pad power_meter dcdbas sb_edac edac_core
> lpc_ich mfd_core shpchp sg tg3 ptp pps_core ext4 jbd2 mbcache
> aesni_intel ablk_helper cryptd lrw glue_helper aes_x86_64 aes_generic
> xts gf128mul dm_crypt sr_mod cdrom sd_mod crc_t10dif ahci wmi
> megaraid_sas dm_mirror dm_region_hash dm_log dm_mod [last unloaded:
> scsi_wait_scan]
> [223371.666106] Pid: 389820, comm: perl veid: 1041 Not tainted
> 2.6.32-042stab094.7 #1
> [223371.666109] Call Trace:
> [223371.666118]  [<ffffffff81075647>] ? warn_slowpath_common+0x87/0xc0
> [223371.666123]  [<ffffffff8107569a>] ? warn_slowpath_null+0x1a/0x20
> [223371.666136]  [<ffffffffa0116d8e>] ? ext4_journal_start_sb+0xce/0xe0 [ext4]
> [223371.666149]  [<ffffffffa0108f37>] ? ext4_create+0x77/0x1a0 [ext4]
> [223371.666154]  [<ffffffff811ba6b3>] ? generic_permission+0x23/0xb0
> [223371.666158]  [<ffffffff811bc5f0>] ? vfs_create+0xd0/0xf0
> ==================================================================
> 
> Best,
> 