[CRIU] LXC integration

[Deyan Doychev]

Hello,

I am copying the lxc-devel mailing list because there will be patches to
LXC as well.

Thanks for the fast response.

On 03/18/2014 09:43 AM, Pavel Emelyanov wrote:
> On 03/18/2014 01:35 AM, Deyan Doychev wrote:
>> Hello everybody!
>>
>> For the last few weeks I've been trying to dump LXC containers with
>> CRIU. Everything seems to work just fine when there are no consoles
>> inside the containers. I can dump the init process and the whole process
>> tree of the container and restore it later. I'm able to add the
>> processes to the right cgroup and fix host side network configuration
>> with an action script.
> That's great! Presumably, these scripts should be submitted to the
> LXC guys, so that they are always consistent with the rest of the
> code.
>
> What guest template do you use, by the way?

Currently I am testing with the base Debian template and in the near
future we plan using CentOS and several other templates.

>> However the following two problems are present:
>>
>> 1) After restoring the previously dumped container, LXC doesn't know
>> about it and all the LXC commands claim it is stopped. This is normal
>> since there is no lxc-start-like process on top with its socket and main
>> loop.                                                                                               
> Yes, we've seen this too. Andrey (in Cc) is also working on that. We
> currently researching two directions how we can re-attach restored CT
> back to LXC daemon.
>
> 1. We tell to CRIU, that after restoring container is should execve()
>    lxc properly explaining to it that there's a new container hanging
>    around.
>
> 2. We make LXC set himself as child subreaper, then fork() criu and ask
>    it to detach (-d) from restore container afterwards. Being a subreaper,
>    it should get the container's init into his child list after it.
>
> We don't yet know which one would work better, since we have to resolve
> issues with the console (below).
>
> Anyway, Andrey will keep us updated with his progress :)
>
>> 2) I have containers that need their consoles and a working lxc-console
>> command.
> This should be resolved with plugins. Plugins itself should be submitted
> to the LXC guys, together with networking action scripts.
>
>> Currently I have the following ideas for fixing these problems:
>>
>> 1) Adding an argument to the criu restore command that will make it
>> exec() a custom command when restore is complete. Thus, the newly
>> created init process will become a child of this command. Then we could
>> create a new lxc-start-like command that will take pid as an argument
>> and work with it instead of the pid returned by clone(). This way LXC
>> will be able to intercept the SIGCHLD signal from the restored init
>> process on container shutdown/restart and also send appropriate signals.
> Yup, this is exactly the way #1 I've described above :) And it looks like
> this one will be the final one, as the way #2 cannot be implemented when
> criu runs ar RPC service.
>
> Deyan, do you have some code doing this ready? We'd be happy if you could
> participate in LXC support development.

I still don't have any code on this as I wanted a second opinion before
working on it but I'll start implementing it right away.

>> 2) Currently /dev/console and /dev/ttyX in the container are actually
>> bind-mounted /dev/pts/Y files. I've been thinking about adding code that
>> creates master-slave pty pairs when restoring mount-points and
>> bind-mounts them to the /dev/ttyX files. After that, with a small patch
>> to the tty.c file we'll be able to use the already allocated slaves,
>> while the lxc-start-like command from 1) will get the masters' open file
>> descriptors upon exec().
> Absolutely. But the code should be implemented as plugin.

I wrote a simple plugin that handles the external tty/console mounts by
simply doing nothing. It was just a test out of curiosity. Unfortunately
to get it to successfully dump the container and later restore it I had
to make a few changes to tty.c - we still end up there because we have
pts files opened.

Do you have any idea of how should I get around this using only a plugin?

Maybe implement another plugin hook for restoring external ttys and call
this from tty.c?

>
>> Before actually beginning with the implementation I'll be glad to hear
>> what you think about these solutions. Also, I'm open to any other
>> suggestions or improvement ideas.
> Everything seems to be in sync with what we plan to do. I think, that even if
> you start doing the criu->exec->lxc reattach thing, you wouldn't conflict with
> Andrey, as he's looking more into the live-migrating LXC container with P.Haul.

Great, I will start from the criu->exec->lxc implementation.

>> Regards,
>> Deyan Doychev
> Thanks,
> Pavel

Regards,
Deyan

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: OpenPGP digital signature
URL: <http://lists.openvz.org/pipermail/criu/attachments/20140318/716a110b/attachment-0001.sig>