[Users] stab93.5 crashing processes

Benjamin Henrion zoobab at gmail.com
Wed Dec 3 07:27:01 PST 2014


On Wed, Dec 3, 2014 at 4:13 PM, Benjamin Henrion <zoobab at gmail.com> wrote:
> Hi,
>
> Just to let you know that I have multiple HNs with the kernel 93.5
> crashing processes with the following kernel messages:
>
> ===========================================================
> [3676904.929199] ------------[ cut here ]------------
> [3676904.929243] WARNING: at fs/ext4/super.c:250
> ext4_journal_start_sb+0xce/0xe0 [ext4]() (Tainted: G        W
> ---------------   )
> [3676904.929247] Hardware name: System x3650 M3 -[7945K3G]-
> [3676904.929249] Modules linked in: dm_snapshot vzethdev vznetdev
> pio_nfs pio_direct pfmt_raw pfmt_ploop1 ploop simfs vzrst vzcpt nfs
> lockd fscache auth_rpcgss nfs_acl sunrpc vziolimit vzmon ipt_REDIRECT
> nf_nat_irc nf_nat_ftp iptable_nat nf_nat xt_helper xt_conntrack
> nf_conntrack_irc nf_conntrack_ftp xt_length ipt_LOG xt_hl xt_tcpmss
> xt_TCPMSS xt_DSCP xt_dscp xt_limit iptable_mangle fuse vzdquota
> vzevent xt_multiport autofs4 bridge vzdev bonding 8021q garp stp llc
> ipt_REJECT iptable_filter nf_conntrack_ipv4 nf_defrag_ipv4 ip_tables
> ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack
> ip6table_filter ip6_tables ipv6 vfat fat tpm_tis tpm tpm_bios iTCO_wdt
> iTCO_vendor_support ipmi_devintf bnx2 cdc_ether usbnet mii serio_raw
> i2c_i801 i2c_core lpc_ich mfd_core sg ioatdma dca i7core_edac
> edac_core shpchp ext4 jbd2 mbcache aesni_intel ablk_helper cryptd lrw
> glue_helper aes_x86_64 aes_generic xts gf128mul dm_crypt sr_mod cdrom
> sd_mod crc_t10dif pata_acpi ata_generic ata_piix usb_storage
> megaraid_sas dm_mirror dm_region_hash dm_log dm_mod [last unloaded:
> sunrpc]
> [3676904.929313] Pid: 572345, comm: gunicorn: worke veid: 29301
> Tainted: G        W  ---------------    2.6.32-042stab093.5 #1
> [3676904.929316] Call Trace:
> [3676904.929326]  [<ffffffff810755d7>] ? warn_slowpath_common+0x87/0xc0
> [3676904.929330]  [<ffffffff8107562a>] ? warn_slowpath_null+0x1a/0x20
> [3676904.929342]  [<ffffffffa0130d8e>] ? ext4_journal_start_sb+0xce/0xe0 [ext4]
> [3676904.929353]  [<ffffffffa0117d8a>] ? ext4_dirty_inode+0x2a/0x60 [ext4]
> [3676904.929360]  [<ffffffff811dbd3b>] ? __mark_inode_dirty+0x3b/0x190
> [3676904.929365]  [<ffffffff811cc95c>] ? inode_setattr+0x4c/0x60
> [3676904.929376]  [<ffffffffa011ab91>] ? ext4_setattr+0x101/0x3a0 [ext4]
> [3676904.929382]  [<ffffffff811a8e8b>] ? put_unused_fd+0x3b/0x90
> [3676904.929385]  [<ffffffff811cccd1>] ? notify_change+0x111/0x340
> [3676904.929389]  [<ffffffff811aa08a>] ? sys_fchmod+0xfa/0x130
> [3676904.929395]  [<ffffffff810f4e9e>] ? __audit_syscall_exit+0x25e/0x290
> [3676904.929403]  [<ffffffff8100b102>] ? system_call_fastpath+0x16/0x1b
> [3676904.929406] ---[ end trace 1a72102dcc568583 ]---
> [3676904.929408] Tainting kernel with flag 0x9
> [3676904.929410] Pid: 572345, comm: gunicorn: worke veid: 29301
> Tainted: G        W  ---------------    2.6.32-042stab093.5 #1
> [3676904.929413] Call Trace:
> [3676904.929416]  [<ffffffff81075461>] ? add_taint+0x71/0x80
> [3676904.929419]  [<ffffffff810755e4>] ? warn_slowpath_common+0x94/0xc0
> [3676904.929423]  [<ffffffff8107562a>] ? warn_slowpath_null+0x1a/0x20
> [3676904.929434]  [<ffffffffa0130d8e>] ? ext4_journal_start_sb+0xce/0xe0 [ext4]
> [3676904.929445]  [<ffffffffa0117d8a>] ? ext4_dirty_inode+0x2a/0x60 [ext4]
> [3676904.929448]  [<ffffffff811dbd3b>] ? __mark_inode_dirty+0x3b/0x190
> [3676904.929451]  [<ffffffff811cc95c>] ? inode_setattr+0x4c/0x60
> [3676904.929462]  [<ffffffffa011ab91>] ? ext4_setattr+0x101/0x3a0 [ext4]
> [3676904.929465]  [<ffffffff811a8e8b>] ? put_unused_fd+0x3b/0x90
> [3676904.929468]  [<ffffffff811cccd1>] ? notify_change+0x111/0x340
> [3676904.929472]  [<ffffffff811aa08a>] ? sys_fchmod+0xfa/0x130
> [3676904.929475]  [<ffffffff810f4e9e>] ? __audit_syscall_exit+0x25e/0x290
> [3676904.929479]  [<ffffffff8100b102>] ? system_call_fastpath+0x16/0x1b
> ===========================================================
>
> I have upgraded to 2.6.32-042stab094.7 and the bug seems to disappear.

I have found a machine with "2.6.32-042stab094.7", and this critical
bug is still present.

I smell that it might be linked to ext4:

==================================================================
[ 5017.634363] device veth10801.2 entered promiscuous mode
[ 5017.634490] vzbr2: topology change detected, propagating
[ 5017.634500] vzbr2: port 2(veth10801.2) entering forwarding state
[13529.132994] Core dump to |/usr/libexec/abrt-hook-ccpp 11 0 20953 0
0 1417105614 e pipe failed
[13529.133878] Core dump to |/usr/libexec/abrt-hook-ccpp 11 0 20976 0
0 1417105614 e pipe failed
[155454.976611] Core dump to |/usr/libexec/abrt-hook-ccpp 11 0 19603 0
0 1417247584 e pipe failed
[178839.543158] Core dump to |/usr/libexec/abrt-hook-ccpp 11 0 29355 0
0 1417270976 e pipe failed
[214904.940995] EXT4-fs (dm-6): ext4_orphan_cleanup: deleting
unreferenced inode 9971401
[214904.941021] EXT4-fs (dm-6): ext4_orphan_cleanup: deleting
unreferenced inode 9971399
[214904.941029] EXT4-fs (dm-6): ext4_orphan_cleanup: deleting
unreferenced inode 9971394
[214904.941037] EXT4-fs (dm-6): ext4_orphan_cleanup: deleting
unreferenced inode 9971393
[214904.941046] EXT4-fs (dm-6): ext4_orphan_cleanup: deleting
unreferenced inode 9971392
[214904.950216] EXT4-fs (dm-6): ext4_orphan_cleanup: deleting
unreferenced inode 10528588
[214904.958245] EXT4-fs (dm-6): ext4_orphan_cleanup: deleting
unreferenced inode 10285608
[214904.960827] EXT4-fs (dm-6): 7 orphan inodes deleted
[214904.962010] EXT4-fs (dm-6): recovery complete
[214904.962165] EXT4-fs (dm-6): mounted filesystem with ordered data mode. Opts:
[220030.090424] EXT4-fs (dm-6): ext4_orphan_cleanup: deleting
unreferenced inode 9971401
[220030.090902] EXT4-fs (dm-6): ext4_orphan_cleanup: deleting
unreferenced inode 9971399
[220030.091063] EXT4-fs (dm-6): ext4_orphan_cleanup: deleting
unreferenced inode 9971394
[220030.091222] EXT4-fs (dm-6): ext4_orphan_cleanup: deleting
unreferenced inode 9971393
[220030.092382] EXT4-fs (dm-6): ext4_orphan_cleanup: deleting
unreferenced inode 9971392
[220030.112088] EXT4-fs (dm-6): ext4_orphan_cleanup: deleting
unreferenced inode 10528588
[220030.120215] EXT4-fs (dm-6): ext4_orphan_cleanup: deleting
unreferenced inode 10285608
[220030.131763] EXT4-fs (dm-6): 7 orphan inodes deleted
[220030.139871] EXT4-fs (dm-6): recovery complete
[220030.140168] EXT4-fs (dm-6): mounted filesystem with ordered data mode. Opts:
[223371.666000] ------------[ cut here ]------------
[223371.666031] WARNING: at fs/ext4/super.c:250
ext4_journal_start_sb+0xce/0xe0 [ext4]() (Not tainted)
[223371.666034] Hardware name: PowerEdge R620
[223371.666036] Modules linked in: dm_snapshot vzethdev pio_nfs
pio_direct pfmt_raw pfmt_ploop1 ploop simfs vzrst vzcpt nfs lockd
fscache auth_rpcgss nfs_acl sunrpc vziolimit vzdquota ipt_REDIRECT
nf_nat_irc nf_nat_ftp xt_helper xt_conntrack nf_conntrack_irc
nf_conntrack_ftp xt_length ipt_LOG xt_hl xt_tcpmss xt_TCPMSS xt_DSCP
xt_dscp xt_limit vzevent xt_multiport autofs4 bridge vznetdev vzmon
vzdev bonding 8021q garp stp llc ipt_REJECT iptable_filter
iptable_mangle iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4
ip_tables ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state
nf_conntrack ip6table_filter ip6_tables ipv6 ipmi_devintf iTCO_wdt
iTCO_vendor_support acpi_pad power_meter dcdbas sb_edac edac_core
lpc_ich mfd_core shpchp sg tg3 ptp pps_core ext4 jbd2 mbcache
aesni_intel ablk_helper cryptd lrw glue_helper aes_x86_64 aes_generic
xts gf128mul dm_crypt sr_mod cdrom sd_mod crc_t10dif ahci wmi
megaraid_sas dm_mirror dm_region_hash dm_log dm_mod [last unloaded:
scsi_wait_scan]
[223371.666106] Pid: 389820, comm: perl veid: 1041 Not tainted
2.6.32-042stab094.7 #1
[223371.666109] Call Trace:
[223371.666118]  [<ffffffff81075647>] ? warn_slowpath_common+0x87/0xc0
[223371.666123]  [<ffffffff8107569a>] ? warn_slowpath_null+0x1a/0x20
[223371.666136]  [<ffffffffa0116d8e>] ? ext4_journal_start_sb+0xce/0xe0 [ext4]
[223371.666149]  [<ffffffffa0108f37>] ? ext4_create+0x77/0x1a0 [ext4]
[223371.666154]  [<ffffffff811ba6b3>] ? generic_permission+0x23/0xb0
[223371.666158]  [<ffffffff811bc5f0>] ? vfs_create+0xd0/0xf0
==================================================================

Best,

-- 
Benjamin Henrion <bhenrion at ffii.org>
FFII Brussels - +32-484-566109 - +32-2-4148403
"In July 2005, after several failed attempts to legalise software
patents in Europe, the patent establishment changed its strategy.
Instead of explicitly seeking to sanction the patentability of
software, they are now seeking to create a central European patent
court, which would establish and enforce patentability rules in their
favor, without any possibility of correction by competing courts or
democratically elected legislators."


More information about the Users mailing list