[CRIU] Does CRIU handle Futex system calls ?

[Alexander Mikhalitsyn]

You are welcome! :)

On Fri, Jul 8, 2022 at 3:12 PM Dorian Goepp <goepp at i3s.unice.fr> wrote:
>
> Hello Alex,
>
> Thanks again for your time and response. It is now all clear to me.
>
> I am amazed at the work put in to ensure CRIU does everything right.
>
>
> Best regards,
>
> Dorian Goepp
>
> Le 2022-07-07 16:54, Alexander Mikhalitsyn a écrit :
>
> Hello, Dorian!
>
> There is one thing I would just like to be sure to have understood properly from your message. Do you mean that the process will redo the futex_wait system calls on restoration ?
>
> yes.
>
> More detailed:
> when CRIU comes to the dump process it uses ptrace() to "seize" them.
> This procedure acts as a signal on the processes, so all (almost)
> syscalls which was executed by the collectable
> processes at the moment of dump get interrupted.
> You can observe that behaviour if you write a "buggy" program that
> incorrectly uses the sleep() call and does not handle EINTR. In this
> case the program will sleep less than it has to.
> But for futex() syscall handling is different, CRIU will restart
> syscall for you because from the kernel side futex will return
> -ERESTARTSYS which is handled by CRIU in compel_get_task_regs and
> leads to "automatic" syscall restart.
> In case of nanosleep syscall the kernel will return
> -ERESTART_RESTARTBLOCK which means that kernel should not perform the
> autorestart for this syscall, so in this case CRIU will "fixup" the
> "ax" register to -EINTR value to be fully transparent to mimic the
> generic kernel behavior for this syscalls group.
>
> General idea here is to be fully invisible to the userspace. If
> syscall has to return EINTR on signal, then CRIU will do the same, if
> syscall has to be restarted after execution of a signal handler then
> CRIU
> will restore the process with the syscall restarted.
>
> I've omitted details about the SA_RESTART flag, it's not so important
> to understand the basic idea here :)
> You can refer to the handle_signal() kernel function to get a better
> understanding of the details.
>
> References:
> https://github.com/torvalds/linux/blob/8cb1ae19bfae92def42c985417cd6e894ddaa047/kernel/futex/waitwake.c#L670
> https://github.com/torvalds/linux/blob/8cb1ae19bfae92def42c985417cd6e894ddaa047/kernel/futex/waitwake.c#L552
> https://github.com/torvalds/linux/blob/d6ecaa0024485effd065124fe774de2e22095f2d/arch/x86/kernel/signal.c#L796
> signal(7) man
>
> Best regards,
> Alex
>
> On Thu, Jul 7, 2022 at 5:11 PM Dorian Goepp <goepp at i3s.unice.fr> wrote:
>
>
> Hello,
>
>
> Thanks a lot Alex for your response.
>
> There is one thing I would just like to be sure to have understood properly from your message. Do you mean that the process will redo the futex_wait system calls on restoration ?
>
> Best regards,
>
> Dorian Goepp
>
> Le 2022-07-07 14:34, Alexander Mikhalitsyn a écrit :
>
> Hello, dear friends,
>
> Yep, CRIU definitely handles futexes carefully. It's one of the most
> important things.
> We support both robust and regular futexes.
>
> As far as you know futexes work only on a shared memory basis, so for
> non-robust futex
> there is no need to have any special handling. We are just dumping the
> whole process memory contents.
> So after the restore we just need to ensure that threads instruction
> pointers (IP) are properly set
> (for instance we need to perform manual "syscall restart" for futexes
> (see compel_get_task_regs()).
>
> For robust futexes we have a special handling here:
> https://github.com/checkpoint-restore/criu/blob/c8f9880adab038481f7806173b698fc6e17ba76a/criu/cr-dump.c#L565
> https://github.com/checkpoint-restore/criu/blob/c8f9880adab038481f7806173b698fc6e17ba76a/criu/pie/restorer.c#L532
>
> Regards,
> Alex
>
> On Thu, Jul 7, 2022 at 3:13 PM Adrian Reber <adrian at lisas.de> wrote:
>
>
> Please try to submit your question as a github issue. Much higher
> chances of getting an answer there.
>
>                 Adrian
>
> On Thu, Jul 07, 2022 at 01:39:17PM +0200, Dorian Goepp wrote:
>
> Hi,
>
> Is this the right place to ask questions about CRIU's features and
> internals?
>
> If so, I have been considering CRIU for dumping and restoring a process
> with a set of threads synchronised through futexes [1]. It seems to
> work, but, I cannot tell from the documentation (wiki) whether it is
> officially supported, or just accidentally works for the cases I tested
> it with.
>
> I could not find which parts would take care of the state of the
> futex_wait system call in CRIU's source code, except maybe for
> `get_task_regs()` in crui/compel/arch/x86/src/lib/infect.c (I run it on
> an amd64 processor). This function issues the warning "Will restore %d
> with interrupted system call" when I dump the futex-heavy process. Is it
> enough to save the process's register state to resume the futex system
> call correctly ?
>
> Best regards,
>
> Dorian Goepp
>
> Links:
> ------
> [1] https://man7.org/linux/man-pages/man2/futex.2.html
>
>
> _______________________________________________
> CRIU mailing list
> CRIU at openvz.org
> https://lists.openvz.org/mailman/listinfo/criu
>
> _______________________________________________
> CRIU mailing list
> CRIU at openvz.org
> https://lists.openvz.org/mailman/listinfo/criu