[Users] Full CT looks to have caused problem with a ext4 fs

[Mark Johanson]

If I am reading this right, it looks like it might be a reemergence of a
previously fixed bug:

https://bugzilla.redhat.com/show_bug.cgi?id=522615

vzctl = 4.0
kernel = 2.6.32-042stab057.1 (yes I know we are behind a couple kernels)
CentOS release 6.3 (Final)

Nov 20 19:50:52 node9 kernel: [422315.522828] VZ QUOTA: disk hardlimit
reached for id=xxxx
....
Nov 20 22:52:57 node9 kernel: [433223.958088] EXT4-fs (sdc): delayed
block allocation failed for inode 12732632 at logical offset 240 with
max blocks 253 with error -122
Nov 20 22:52:57 node9 kernel: [433223.958235] 
Nov 20 22:52:57 node9 kernel: [433223.958236] This should not happen!!
Data will be lost
Nov 20 22:52:57 node9 kernel: [433223.964372] EXT4-fs (sdc): delayed
block allocation failed for inode 12732644 at logical offset 4775 with
max blocks 36 with error -122
Nov 20 22:52:57 node9 kernel: [433223.964564] 
Nov 20 22:52:57 node9 kernel: [433223.964566] This should not happen!!
Data will be lost
Nov 20 22:52:57 node9 kernel: [433223.964694] EXT4-fs (sdc): delayed
block allocation failed for inode 12732644 at logical offset 4814 with
max blocks 69 with error -122
Nov 20 22:52:57 node9 kernel: [433223.964874] 
Nov 20 22:52:57 node9 kernel: [433223.964875] This should not happen!!
Data will be lost
Nov 20 22:52:57 node9 kernel: [433223.965089] EXT4-fs (sdc): delayed
block allocation failed for inode 12731800 at logical offset 187 with
max blocks 8 with error -122
Nov 20 22:52:57 node9 kernel: [433223.965281] 
Nov 20 22:52:57 node9 kernel: [433223.965283] This should not happen!!
Data will be lost
Nov 20 22:55:37 node9 kernel: [433383.616500] INFO: task flush-8:32:1642
blocked for more than 120 seconds.
Nov 20 22:55:37 node9 kernel: [433383.616582] "echo 0
> /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 20 22:55:37 node9 kernel: [433383.616719] flush-8:32    D
ffff88033a82c3c0     0  1642      2    0 0x00000080
Nov 20 22:55:37 node9 kernel: [433383.616725]  ffff880332193950
0000000000000046 0000000000000000 ffff8803321939b0
Nov 20 22:55:37 node9 kernel: [433383.616732]  ffff880332193a28
0000000000000000 000000000000000e ffff8803321939a0
Nov 20 22:55:37 node9 kernel: [433383.616737]  ffff880332193930
ffff88033a82c978 ffff880332193fd8 ffff880332193fd8
Nov 20 22:55:37 node9 kernel: [433383.616742] Call Trace:
Nov 20 22:55:37 node9 kernel: [433383.616772]  [<ffffffffa0035f7a>]
start_this_handle+0x26a/0x4e0 [jbd2]
Nov 20 22:55:37 node9 kernel: [433383.616788]  [<ffffffff810959c0>] ?
autoremove_wake_function+0x0/0x40
Nov 20 22:55:37 node9 kernel: [433383.616803]  [<ffffffffa00363d5>]
jbd2_journal_start+0xb5/0x100 [jbd2]
Nov 20 22:55:37 node9 kernel: [433383.616825]  [<ffffffffa0066705>] ?
ext4_meta_trans_blocks+0x75/0xf0 [ext4]
Nov 20 22:55:37 node9 kernel: [433383.616848]  [<ffffffffa0080968>]
ext4_journal_start_sb+0x58/0x90 [ext4]
Nov 20 22:55:37 node9 kernel: [433383.616867]  [<ffffffffa006a7bc>]
ext4_da_writepages+0x28c/0x690 [ext4]
Nov 20 22:55:37 node9 kernel: [433383.616888]  [<ffffffff81138f10>] ?
__writepage+0x0/0x40
Nov 20 22:55:37 node9 kernel: [433383.616900]  [<ffffffff8113a041>]
do_writepages+0x21/0x40
Nov 20 22:55:37 node9 kernel: [433383.616914]  [<ffffffff811bdb2d>]
__writeback_single_inode+0xdd/0x2c0
Nov 20 22:55:37 node9 kernel: [433383.616926]  [<ffffffff811bdd93>]
writeback_single_inode+0x83/0xc0
Nov 20 22:55:37 node9 kernel: [433383.616940]  [<ffffffff811ad300>] ?
iput+0x30/0x70
Nov 20 22:55:37 node9 kernel: [433383.616951]  [<ffffffff811be051>]
writeback_sb_inodes+0xf1/0x210
Nov 20 22:55:37 node9 kernel: [433383.616963]  [<ffffffff811be2c0>]
writeback_inodes_wb+0x150/0x1a0
Nov 20 22:55:37 node9 kernel: [433383.616975]  [<ffffffff811be5eb>]
wb_writeback+0x2db/0x430
Nov 20 22:55:37 node9 kernel: [433383.616990]  [<ffffffff814e9f9e>] ?
thread_return+0x4e/0x7d0
Nov 20 22:55:37 node9 kernel: [433383.617002]  [<ffffffff811be8e9>]
wb_do_writeback+0x1a9/0x250
Nov 20 22:55:37 node9 kernel: [433383.617015]  [<ffffffff8107e7a0>] ?
process_timeout+0x0/0x10
Nov 20 22:55:37 node9 kernel: [433383.617028]  [<ffffffff811be9f3>]
bdi_writeback_task+0x63/0x1b0
Nov 20 22:55:37 node9 kernel: [433383.617039]  [<ffffffff81095897>] ?
bit_waitqueue+0x17/0xc0
Nov 20 22:55:37 node9 kernel: [433383.617053]  [<ffffffff8114dd70>] ?
bdi_start_fn+0x0/0x110
Nov 20 22:55:37 node9 kernel: [433383.617064]  [<ffffffff8114de05>]
bdi_start_fn+0x95/0x110
Nov 20 22:55:37 node9 kernel: [433383.617075]  [<ffffffff8114dd70>] ?
bdi_start_fn+0x0/0x110
Nov 20 22:55:37 node9 kernel: [433383.617089]  [<ffffffff810953e6>]
kthread+0x96/0xa0
Nov 20 22:55:37 node9 kernel: [433383.617103]  [<ffffffff8100c20a>]
child_rip+0xa/0x20
Nov 20 22:55:37 node9 kernel: [433383.617114]  [<ffffffff81095350>] ?
kthread+0x0/0xa0
Nov 20 22:55:37 node9 kernel: [433383.617125]  [<ffffffff8100c200>] ?
child_rip+0x0/0x20


I didn't see any quota info that might have lead to this being fixed
already, and it is possible this was a fluke that the user hit their
quota and not too long after the etx4 filesystem has issues.

Anyone else seen this issue recently?