[Devel] [PATCH 1/1] ms/virtio_ring: fix infinite loop in virtnet_poll_cleantx when device is broken

[Denis V. Lunev]

From: Jinqian Yang <yangjinqian1 at huawei.com>

virtnet_poll_cleantx() contains a do-while loop that cleans up
transmitted TX buffers and calls virtqueue_enable_cb_delayed() to check
whether more buffers need processing. When the virtio backend stops
responding during guest reboot, used->idx is never updated, so
virtqueue_enable_cb_delayed() always returns false and the loop never
terminates. Then it will block reboot process, and the guest will hang.

The problem occurs during guest reboot under network traffic:

  1. kernel_restart() -> device_shutdown() traverses the device list
  2. virtio_dev_shutdown() calls virtio_break_device() which sets
     vq->broken = true
  3. virtio_dev_shutdown() then calls virtio_synchronize_cbs() to wait
     for in-flight callbacks to complete
  4. A virtio interrupt fires, softirq is deferred to ksoftirqd which
     calls net_rx_action() -> virtnet_poll() -> virtnet_poll_cleantx()
  5. virtnet_poll_cleantx() enters the do-while loop and never exits
     because the QEMU backend has stopped updating used->idx, despite
     vq->broken having been set to true in step 2.

Since the loop runs inside ksoftirqd (a SCHED_OTHER kthread), it is
visible to the scheduler and does not trigger a hard lockup. However,
the kthread never leaves the loop, so RCU detects it as a CPU stall
and reports it periodically. Meanwhile, the reboot process remains
blocked in device_shutdown() because virtio_dev_shutdown() cannot
complete its synchronization step, and the guest hangs permanently.

This can be reproduced on a guest with a virtio-net device: run iperf3
traffic in the guest, then trigger reboot. The reboot occasionally hangs
permanently with RCU stall on ksoftirqd.

Observed on ARM64 KVM guest:

  CPU#1 RCU stall (ksoftirqd/1), repeated periodically:
    virtqueue_enable_cb_delayed_split <- virtnet_poll <- __napi_poll <-
    net_rx_action <- handle_softirqs <- run_ksoftirqd <-
    smpboot_thread_fn <- kthread

Fix by adding a vq->broken check in virtqueue_enable_cb_delayed(), so
that the loop exits immediately when the device is broken, allowing
the device shutdown to proceed.

Signed-off-by: Jinqian Yang <yangjinqian1 at huawei.com>
Reviewed-by: Xuan Zhuo <xuanzhuo at linux.alibaba.com>
Signed-off-by: Michael S. Tsirkin <mst at redhat.com>
Message-ID: <20260716115940.394832-1-yangjinqian1 at huawei.com>

(cherry picked from commit 0d0eff39ceb3dcbf7847a6f4517086c207c60081)

Applied by hand: this tree has no data_race() annotation on
vq->event_triggered and dispatches the split and packed helpers
directly, so only the surrounding context differs. The added check is
identical.

This tree is hit harder than the kernel the fix was written against.
Upstream commit e13b6da7045f ("virtio-net: tweak for better TX
performance in NAPI mode") replaced the same do-while loop in
start_xmit() with a single check and is not here yet, so the livelock
is also reachable from a timer softirq: a delack timer transmitting on
a broken queue never returns from start_xmit(), and softlockup_panic
turns that into a panic rather than an RCU stall on ksoftirqd.

https://virtuozzo.atlassian.net/browse/VSTOR-143525
Feature: fix ms/virtio_ring
Signed-off-by: Denis V. Lunev <den at openvz.org>
---
 drivers/virtio/virtio_ring.c | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c
index 65674af37918..8097e49456d3 100644
--- a/drivers/virtio/virtio_ring.c
+++ b/drivers/virtio/virtio_ring.c
@@ -2529,6 +2529,14 @@ bool virtqueue_enable_cb_delayed(struct virtqueue *_vq)
 {
 	struct vring_virtqueue *vq = to_vvq(_vq);
 
+	/*
+	 * When the device is broken there is no point in polling used->idx,
+	 * the backend will never update it. Return true to let callers
+	 * exit their cleanup loops instead of spinning forever.
+	 */
+	if (unlikely(vq->broken))
+		return true;
+
 	if (vq->event_triggered)
 		vq->event_triggered = false;
 
-- 
2.53.0