[CRIU] LXC integration

[Pavel Emelyanov]

On 03/18/2014 01:35 AM, Deyan Doychev wrote:
> Hello everybody!
> 
> For the last few weeks I've been trying to dump LXC containers with
> CRIU. Everything seems to work just fine when there are no consoles
> inside the containers. I can dump the init process and the whole process
> tree of the container and restore it later. I'm able to add the
> processes to the right cgroup and fix host side network configuration
> with an action script.

That's great! Presumably, these scripts should be submitted to the
LXC guys, so that they are always consistent with the rest of the
code.

What guest template do you use, by the way?

> However the following two problems are present:
> 
> 1) After restoring the previously dumped container, LXC doesn't know
> about it and all the LXC commands claim it is stopped. This is normal
> since there is no lxc-start-like process on top with its socket and main
> loop.                                                                                               

Yes, we've seen this too. Andrey (in Cc) is also working on that. We
currently researching two directions how we can re-attach restored CT
back to LXC daemon.

1. We tell to CRIU, that after restoring container is should execve()
   lxc properly explaining to it that there's a new container hanging
   around.

2. We make LXC set himself as child subreaper, then fork() criu and ask
   it to detach (-d) from restore container afterwards. Being a subreaper,
   it should get the container's init into his child list after it.

We don't yet know which one would work better, since we have to resolve
issues with the console (below).

Anyway, Andrey will keep us updated with his progress :)

> 2) I have containers that need their consoles and a working lxc-console
> command.

This should be resolved with plugins. Plugins itself should be submitted
to the LXC guys, together with networking action scripts.

> Currently I have the following ideas for fixing these problems:
> 
> 1) Adding an argument to the criu restore command that will make it
> exec() a custom command when restore is complete. Thus, the newly
> created init process will become a child of this command. Then we could
> create a new lxc-start-like command that will take pid as an argument
> and work with it instead of the pid returned by clone(). This way LXC
> will be able to intercept the SIGCHLD signal from the restored init
> process on container shutdown/restart and also send appropriate signals.

Yup, this is exactly the way #1 I've described above :) And it looks like
this one will be the final one, as the way #2 cannot be implemented when
criu runs ar RPC service.

Deyan, do you have some code doing this ready? We'd be happy if you could
participate in LXC support development.

> 2) Currently /dev/console and /dev/ttyX in the container are actually
> bind-mounted /dev/pts/Y files. I've been thinking about adding code that
> creates master-slave pty pairs when restoring mount-points and
> bind-mounts them to the /dev/ttyX files. After that, with a small patch
> to the tty.c file we'll be able to use the already allocated slaves,
> while the lxc-start-like command from 1) will get the masters' open file
> descriptors upon exec().

Absolutely. But the code should be implemented as plugin.

> Before actually beginning with the implementation I'll be glad to hear
> what you think about these solutions. Also, I'm open to any other
> suggestions or improvement ideas.

Everything seems to be in sync with what we plan to do. I think, that even if
you start doing the criu->exec->lxc reattach thing, you wouldn't conflict with
Andrey, as he's looking more into the live-migrating LXC container with P.Haul.

> Regards,
> Deyan Doychev

Thanks,
Pavel