[CRIU] Live migration for containers - Was "RE: [libvirt] libvirt support for LXC"

[Daniel P. Berrange]

On Sat, Nov 02, 2013 at 12:56:37AM +0000, Christian Benvenuti (benve) wrote:
> Hello,
>   based on the 3D below, it seems that the most logical way to
> add support for
> 
> 	 container live migration
> 
> to libvirt is to integrate the latter with CRIU.
> If I understand it correctly, Daniel's suggestion below would be
> that of
> - 1st converting CRIU to a library
> - 2nd making libvirt use that library to C/R the container/s
> 
> CRIU has recently announced support for
> 
> 	CRIU as a service
> 
> and the reason why they opted for a service instead of a library [1] seems
> to be associated with a use case they had:
> 
> 	ability for an application to invoke a self-dump C/R
> 
> In Libvirt's case it would not be the container to ask for a self dump, but it would
> be libvirt itself to orchestrate it.
> 
> In light of the new CRIU as a service feature, is libvirt's preference still
> that of using a library? Would a service be equally good?

I can't easily answer this question, without delving into the technical
details of it. The big question is how this would fit into libvirt's
architecture. I'm not fixed on any single approach - just thought that
having it as a library might be the easiest way to integrate - we tend
to prefer APIs over forking external programs whereever available, since
they're often easier to do good error reporting with, and lower overhead
to use.

Currently libvirtd spawns libvirt_lxc which then spawns the actual
container. The libvirt_lxc process is daemonized and is set as the
parent of the container init process, and has a UNIX socket back
to libvirtd.  The libvirt_lxc process also owns the master side of two
psuedo TTYs, one in the host's /dev/pts instance, and one in the guest's
/dev/pts instance.

What I'm fuzzy on is where the C/R would best take place. eg would it
cover the container processes only, or the container process + libvirt_lxc.
In the former case, I'm not sure how the libvirt_lxc container wouuld get
back a handle to the master PTY. In the latter case, I'm not sure how
libvirtd would re-establish the UNIX socket connection to libvirt_lxc.
I've heard that there's some problems with CRIU and UNIX sockets when
only one end of the socket is being C/R'd.

Also curious how CRIU deals with network interfaces. When libvirt
starts a container, if using an SRIOV NIC with multiple virtual
functions, then libvirt will select a random function to assign to
the container at startup. We can assume this same function is still
available when restoring the containe - we'd likely need to select
a different virtual function to give to the container as its "eth0"

Similarly with SELinux, we dynamically assign a unique MCS level to
containers when starting them. This will again need to be newly
allocated at restore time and will almost certain differ from the
previous MCS level.

I've not looked at CRIU in enough detail know if it copes with stuff
like this or not.

> Is there anyone actively working or looking at this (libvirt+CRIU)?

Not that I am aware of.

Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|