[Devel] [PATCH hci-8.1] fs/fuse kio: start connect once per rpc_queue_work batch
Alexey Kuznetsov
kuznet at virtuozzo.com
Tue Sep 22 00:17:19 MSK 2026
IMHO if we are inside unpatched rpc_queue_work() it can continue to loop,
it is not a big deal, it will finish the work eventually, hours or even days.
We just prevent future works queued after it from slow execution.
Normally, cluster does not even sense this background job. It is
single rescuer thread
which is not used for normal work. It starts to be
sensed when it is required to flush all pending requests like it
happens on unmount.
Then it can be fixed manually, it is simple with some iptables ... -j REJECT.
Main thing is that RK did not fail, then the problem will disappear
after the first run
is finally finished. But if customer has already got huge request
queues (like with the last case it was 15 rpcs 200000-300000 reqs each),
and needs urgent unmount, he has to wait till current rpc_queue_work() finsihes,
or support can add a temporary iptables rule.
On Tue, Sep 22, 2026 at 4:56 AM Vladimir Riabchun
<vladimir.riabchun at virtuozzo.com> wrote:
>
>
> On 9/21/26 13:28, Liu Kui wrote:
> > pcs_rpc_connect() is synchronous: it performs a full connect+auth
> > cycle while holding ep->mutex. rpc_queue_work() splices the entire
> > ep->input_queue onto an on-stack list and used to trigger the connect
> > from pcs_rpc_send() on the first staged message. When the connect
> > fails, pcs_rpc_reset() purges ep->input_queue and ep->state_queue,
> > but the messages still sitting on the on-stack list are invisible to
> > the purge - each of them re-triggers another blocking connect attempt
> > under the same mutex hold. With a large backlog and an unreachable
> > peer this keeps ep->mutex held for O(messages * connect timeout):
> > in one crashdump the rescuer of pcs_wq ground through such a batch
> > for days, starving calendar_work/rpc_close_work on the same ep and
> > wedging umount in flush_workqueue(pcs_wq).
> >
> > Stage all messages from the batch on ep->state_queue first and start
> > the connect once afterwards. A failed connect now aborts the whole
> > batch in a single rpc_abort(), bounding the mutex hold to one connect
> > attempt. This also restores the meaning of state_queue, which
> > otherwise never held more than one message.
> >
> > https://virtuozzo.atlassian.net/browse/VSTOR-135626
> >
> > Signed-off-by: Liu Kui <kui.liu at virtuozzo.com>> ---
> > Fix a pathological case seen in ACSSP-872 (and ASUP-3562) crashdumps:
> > a single rpc_queue_work() invocation could hold ep->mutex for days,
> > re-running a blocking connect+auth once per queued message to an
> > unreachable peer, wedging the pcs_wq rescuer, calendar/close works
> > and umount in flush_workqueue().
> >
> > https://virtuozzo.atlassian.net/browse/ACSSP-872
> > ---
> > fs/fuse/kio/pcs/pcs_rpc.c | 18 +++++++++++++++---
> > 1 file changed, 15 insertions(+), 3 deletions(-)
> >
> > diff --git a/fs/fuse/kio/pcs/pcs_rpc.c b/fs/fuse/kio/pcs/pcs_rpc.c
> > index 540225006726..c50e31c08a31 100644
> > --- a/fs/fuse/kio/pcs/pcs_rpc.c
> > +++ b/fs/fuse/kio/pcs/pcs_rpc.c
> > @@ -702,9 +702,6 @@ static void pcs_rpc_send(struct pcs_rpc * ep, struct pcs_msg * msg, bool requeue
> >
> > list_add_tail(&msg->list, &ep->state_queue);
> > msg->stage = PCS_MSG_STAGE_UNSENT;
> > -
> > - if (ep->state == PCS_RPC_UNCONN)
> > - pcs_rpc_connect(ep);
> > }
> >
> > void pcs_rpc_kick_queue(struct pcs_rpc * ep)
> > @@ -930,6 +927,21 @@ static void rpc_queue_work(struct work_struct *w)
> > list_del_init(&msg->list);
> > pcs_rpc_send(ep, msg, 0);
> > }
> > +
> > + /* pcs_rpc_connect() is synchronous: it can block for a full
> > + * connect+auth cycle while holding ep->mutex. Start it once, after
> > + * every message from input_q has been staged on ep->state_queue,
> > + * instead of from pcs_rpc_send() on the first message. This way a
> > + * failed connect aborts the whole batch at once: rpc_abort() purges
> > + * ep->state_queue, whereas messages still sitting on the on-stack
> > + * input_q are invisible to the purge and would each trigger another
> > + * blocking connect attempt under the same mutex hold. It also
> > + * restores the meaning of state_queue, which otherwise never holds
> > + * more than one message.
> > + */
> > + if (ep->state == PCS_RPC_UNCONN && !list_empty(&ep->state_queue))
> > + pcs_rpc_connect(ep);
> > +
>
> Now I seem to get the problem we are fixing. This will work fine when
> no problems pre-existed, but if the thread already collected a bunch of
> messages in its private list, it will prevent live patching.
>
> Ref: https://docs.kernel.org/livepatch/livepatch.html#consistency-model
>
> rpc_queue_work task has patched function on its stack (itself), so
> approach 1 is not for us. We can't poke kernel thread task with any signals
> and waking it up in klp_send_signals will not help as well.
>
> The doc I referenced above describes /sys/kernel/livepatch/<patch>/force
> mechanism, that can force patch transition for all tasks.
> We may make use of it to fixup the thread.
>
> I checked disassembly and pcs_rpc_send is _not_ inlined in rpc_queue_work,
> so after forcing livepatch, new versions of pcs_rpc_send will be executed.
> However, rpc_queue_work will be old one - jump to new function version
> happens in ftrace header, we missed it and never come there for a running
> kthread.
>
> So with this forced scenario we can clear local input_q list, but nothing
> here (I don't know about other places) will then call pcs_rpc_connect
> and flush ep->state_queue.
> After moving all input_q to ep->state_queue, without pcs_rpc_connect
> this state_queue will move to local state_q, which will then do
> patched pcs_rpc_send with requeue = 1, again without pcs_rpc_connect,
> and put these messages back to ep->state_queue.
>
> My question is:
> Will anything blow up in this case if we just move these messages
> from one queue to another without calling connect?
> For me it seems that it will be a fast journey, all messages will
> go to ep->state_queue and we will leave the kthread (repeat = 0).
> Am I right and just forcing livepatch will work, or we need something
> smarter to fix stuck threads?
>
> > list_splice_tail_init(&ep->state_queue, &state_q);
> > while (!list_empty(&state_q)) {
> > struct pcs_msg * msg = list_first_entry(&state_q, struct pcs_msg, list);
> >
> > ---
> > base-commit: 3851feefb321d7dee9e57cbe7926410aa6dd043f
> > change-id: 20260921-acssp872-batch-connect-bec05e9db04d
> >
> > Best regards,
> > --
> > Liu Kui <kui.liu at virtuozzo.com>
> >
> > _______________________________________________
> > Devel mailing list
> > Devel at openvz.org
> > https://lists.openvz.org/mailman/listinfo/devel
>
> --
> Best regards, Riabchun Vladimir
> Linux Kernel Developer, Virtuozzo
More information about the Devel
mailing list