[CRIU] [lxc-devel] LXC live migrate

[Marian Marinov]

On 11/27/2013 10:54 AM, Pavel Emelyanov wrote:
> On 11/27/2013 06:19 AM, Qiang Huang wrote:
>> On 2013/11/27 0:19, Marian Marinov wrote:
>>>
>>> On my test setup it works for processes like apache, dovecot and mysql.
>>>
>>> However it does not work with containers:
>>>
>>> root at s321:~# criu dump -D deb1 -t 19332 --file-locks
>>> (00.004962) Error (namespaces.c:155): Can't dump nested pid namespace for 28352
>>> (00.004985) Error (namespaces.c:321): Can't make pidns id
>>> (00.005327) Error (cr-dump.c:1811): Dumping FAILED.
>>> root at s321:~#
>>> When I try to dump the init process(which I believe I should not do), here is what I see:
>>>     http://pastebin.com/DFC0ADpp
>>>
>>> (00.291294) Error (tty.c:222): tty: Unexpected format on path /dev/tty1
>>> (00.291315) Error (cr-dump.c:1491): Dump files (pid: 29702) failed with -1
>>> (00.291892) Error (cr-dump.c:1811): Dumping FAILED.
>>>
>>> This is my setup:
>>> 19332 ?        Ss     0:00 lxc-start -n deb1 -d
>>> 28352 ?        Ss     0:00  \_ init [3]
>>> 28393 ?        Ss     0:00      \_ /usr/sbin/apache2 -k start
>>> 28419 ?        S      0:00      |   \_ /usr/sbin/apache2 -k start
>>> 28422 ?        Sl     0:00      |   \_ /usr/sbin/apache2 -k start
>>> 28423 ?        Sl     0:00      |   \_ /usr/sbin/apache2 -k start
>>> 28489 ?        S      0:00      \_ /bin/sh /usr/bin/mysqld_safe
>>> 28620 ?        Sl     0:00      |   \_ /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql
>>> --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/run/mysqld/mysqld.sock --port
>>> 28621 ?        S      0:00      |   \_ logger -t mysqld -p daemon.error
>>> 28598 ?        Ss     0:00      \_ /usr/sbin/sshd
>>> 29702 pts/0    Ss+    0:00      \_ /sbin/getty 38400 tty1 linux
>>>
>>> I rebooted the container without getty on tty1 and then I got this:
>>>
>>> (00.260757) Error (mount.c:255): 86:/dev/tty4 doesn't have a proper root mount
>
> This is the reason. That's container's console which is a bind mounted tty from
> the host. And since this is an external connection, CRIU doesn't dump one.
>
> There are two ways to resolve this. The first is disable container's console. It's
> fast, but ugly. The second way is supporting one, but it would require criu hacking.
> We should detect, that this is an external tty, decide, that we're OK to disconnect
> it after dump and on restore -- connect it back.

The ugly fix does not work either. Because even if you comment the lxc.tty option criu is complaining:

(00.243390) Error (mount.c:255): 82:/dev/console doesn't have a proper root mount
(00.243626) Error (namespaces.c:445): Namespaces dumping finished with error 65280
(00.244029) Error (cr-dump.c:1811): Dumping FAILED.

>
>>> (00.261007) Error (namespaces.c:445): Namespaces dumping finished with error 65280
>>> (00.261454) Error (cr-dump.c:1811): Dumping FAILED.
>>>
>>> This ithe relevant container config
>>> ## Device config
>>> lxc.cgroup.devices.deny = a
>>> # /dev/null and zero
>>> lxc.cgroup.devices.allow = c 1:3 rwm
>>> lxc.cgroup.devices.allow = c 1:5 rwm
>>> # consoles
>>> lxc.cgroup.devices.allow = c 5:1 rwm
>>> lxc.cgroup.devices.allow = c 5:0 rwm
>>> lxc.cgroup.devices.allow = c 4:0 rwm
>>> lxc.cgroup.devices.allow = c 4:1 rwm
>>> # /dev/{,u}random
>>> lxc.cgroup.devices.allow = c 1:9 rwm
>>> lxc.cgroup.devices.allow = c 1:8 rwm
>>> lxc.cgroup.devices.allow = c 136:* rwm
>>> lxc.cgroup.devices.allow = c 5:2 rwm
>>> # rtc
>>> lxc.cgroup.devices.allow = c 254:0 rm
>>>
>>> # mounts point
>>> lxc.mount.entry = devpts dev/pts devpts gid=5,mode=620 0 0
>>> lxc.mount.auto = proc:mixed sys:ro
>>>
>>>
>>> Am I doing something wrong?
>>
>> According to the criu TODO list: http://criu.org/Todo
>> cgroups in container is not supported yet, so I doubt it would work for
>> normal containers.

Pavel, can you give me some pointers for this? I would be interested in helping out with this part.

>
> AFAIK cgroups are used _inside_ containers only with recent guest templates.
> In OpenVZ we use more old ones (and more stable) so haven't meet this yet.
> And yes, cgroups are in plans for the nearest future :)
>
>> I'm interested in this too, so let's cc CRIU list and find out what is wrong :)
>>
>>>
>>> Marian
>>>
>
> Thanks,
> Pavel
>
>
>