[Devel] [PATCH RHEL7 COMMIT] block: suppress hard lockup warning in elv_drain_elevator

Konstantin Khorenko khorenko at virtuozzo.com
Thu Mar 24 07:42:21 PDT 2016


The commit is pushed to "branch-rh7-3.10.0-327.10.1.vz7.12.x-ovz" and will appear at https://src.openvz.org/scm/ovz/vzkernel.git
after rh7-3.10.0-327.10.1.vz7.12.3
------>
commit ef777429c0e02a80b1f4487112b14ac866af8645
Author: Vladimir Davydov <vdavydov at virtuozzo.com>
Date:   Thu Mar 24 18:42:20 2016 +0400

    block: suppress hard lockup warning in elv_drain_elevator
    
    Under heavy io sg_io() might keep busy-looping in elv_drain_elevator()
    under queue_lock and irqs disabled for quite a bit while trying to
    insert a request to the tail of the queue, resulting in hard lockup:
    
      Kernel panic - not syncing: Watchdog detected hard LOCKUP on cpu 1
      CPU: 1 PID: 642 Comm: smartd ve: 0 Not tainted 3.10.0-327.3.1.vz7.10.13 #1 10.13
      Hardware name:   Intel Corporation Blackford & ESB2 Chipset/Blackford CRB, BIOS F8  01/24/2007
       ffff88042fc45c40 00000000b0b67a0e ffff88042fc45af0 ffffffff81630786
       ffff88042fc45b70 ffffffff8162ac09 0000000000000010 ffff88042fc45b80
       ffff88042fc45b20 00000000b0b67a0e ffffffff8101cc99 0000000000000001
      Call Trace:
       <NMI>  [<ffffffff81630786>] dump_stack+0x19/0x1b
       [<ffffffff8162ac09>] panic+0xd8/0x1e7
       [<ffffffff8101cc99>] ? sched_clock+0x9/0x10
       [<ffffffff8112c610>] ? restart_watchdog_hrtimer+0x50/0x50
       [<ffffffff8112c6d2>] watchdog_overflow_callback+0xc2/0xd0
       [<ffffffff81171411>] __perf_event_overflow+0xa1/0x250
       [<ffffffff81171ee4>] perf_event_overflow+0x14/0x20
       [<ffffffff81032e98>] intel_pmu_handle_irq+0x1e8/0x470
       [<ffffffff8101cc45>] ? native_sched_clock+0x35/0x80
       [<ffffffff810bf30d>] ? sched_clock_local+0x1d/0x80
       [<ffffffff8163a18b>] perf_event_nmi_handler+0x2b/0x50
       [<ffffffff816398d9>] nmi_handle.isra.0+0x69/0xb0
       [<ffffffff816399f0>] do_nmi+0xd0/0x340
       [<ffffffff81638cb1>] end_repeat_nmi+0x1e/0x2e
       [<ffffffff812bec77>] ? elv_dispatch_sort+0x87/0xe0
       [<ffffffff812bec77>] ? elv_dispatch_sort+0x87/0xe0
       [<ffffffff812bec77>] ? elv_dispatch_sort+0x87/0xe0
       <<EOE>>  [<ffffffff812e74d8>] cfq_dispatch_insert+0x158/0x280
       [<ffffffff8101002f>] ? perf_trace_xen_mmu_set_pud+0x19f/0x1b0
       [<ffffffff810b7960>] ? finish_task_switch+0xe0/0x180
       [<ffffffff812ea427>] cfq_dispatch_requests+0xae7/0xc20
       [<ffffffff812e9252>] ? cfq_set_request+0xa2/0x440
       [<ffffffff811db350>] ? kmem_cache_alloc+0xf0/0x220
       [<ffffffff8117e545>] ? mempool_alloc_slab+0x15/0x20
       [<ffffffff812bf1f2>] elv_drain_elevator+0x22/0x70
       [<ffffffff812bf2fb>] __elv_add_request+0xbb/0x2d0
       [<ffffffff812cb82d>] blk_execute_rq_nowait+0xad/0x180
       [<ffffffff812c544a>] ? get_request+0x39a/0x780
       [<ffffffff812cb98b>] blk_execute_rq+0x8b/0x150
       [<ffffffff810a8720>] ? wake_up_atomic_t+0x30/0x30
       [<ffffffff8108951e>] ? ns_capable+0x2e/0x60
       [<ffffffff810895a7>] ? capable+0x17/0x20
       [<ffffffff812d774b>] sg_io+0x27b/0x450
       [<ffffffff812d8017>] scsi_cmd_ioctl+0x337/0x4d0
       [<ffffffff811e8bbc>] ? __memcg_kmem_get_cache+0x4c/0x150
       [<ffffffff812d81f2>] scsi_cmd_blk_ioctl+0x42/0x50
       [<ffffffffa00c4a5e>] sd_ioctl+0xbe/0x140 [sd_mod]
       [<ffffffff812d480f>] blkdev_ioctl+0x2df/0x770
       [<ffffffff81236ff1>] block_ioctl+0x41/0x50
       [<ffffffff8120de15>] do_vfs_ioctl+0x255/0x4f0
       [<ffffffff81218e17>] ? __fd_install+0x47/0x60
       [<ffffffff8120e104>] SyS_ioctl+0x54/0xa0
       [<ffffffff81640ec9>] system_call_fastpath+0x16/0x1b
    
    There's nothing critical about it - the system should recover sooner or
    later - so let's suppress watchdog there.
    
    https://jira.sw.ru/browse/PSBM-44480
    
    Signed-off-by: Vladimir Davydov <vdavydov at virtuozzo.com>
    Acked-by: Dmitry Monakhov <dmonakhov at virtuozzo.com>
---
 block/cfq-iosched.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/block/cfq-iosched.c b/block/cfq-iosched.c
index af66fad..85e091c 100644
--- a/block/cfq-iosched.c
+++ b/block/cfq-iosched.c
@@ -14,6 +14,7 @@
 #include <linux/rbtree.h>
 #include <linux/ioprio.h>
 #include <linux/blktrace_api.h>
+#include <linux/nmi.h>
 #include <bc/io_acct.h>
 
 #include "blk.h"
@@ -3237,6 +3238,7 @@ static int cfq_forced_dispatch(struct cfq_data *cfqd)
 	while ((cfqq = cfq_get_next_queue_forced(cfqd)) != NULL) {
 		__cfq_set_active_queue(cfqd, cfqq);
 		dispatched += __cfq_forced_dispatch_cfqq(cfqq);
+		touch_nmi_watchdog();
 	}
 
 	BUG_ON(cfqd->busy_queues);


More information about the Devel mailing list