[CRIU] Restore CRIU image across a host

[Pavel Emelyanov]

On 08/05/2015 11:14 PM, Hui Kang wrote:
> Hi, Tycho
> 
> On Wed, Aug 5, 2015 at 9:39 AM, Tycho Andersen <tycho.andersen at canonical.com <mailto:tycho.andersen at canonical.com>> wrote:
> 
>     Hi Hui,
> 
>     On Tue, Aug 04, 2015 at 11:15:25PM -0400, Hui Kang wrote:
>     > Hi, Tycho
>     > I created a patch
>     > (http://lists.openvz.org/pipermail/criu/2015-August/021576.html) and
>     > pass cgrou_manage as CG_MODE_FULL to the criu service. From the
>     > restore.log, I can see that criu will smartly create all the cgroup
>     > directories. However, the restore still fails at this cpu of wrting to
>     >
>     > (cgroup.c:901): cg: Can't move into cpuset//[PID]/tasks (-1/0): No
>     > space left on device
>     > (00.082249) Error (cr-restore.c:1919): Restoring FAILED.
>     >
>     > As Tycho pointed that:
>     >
>     > >> >        ENOSPC Attempted to write(2) the process ID (PID) of  a  process
>     > >> > to  a  cpuset
>     > >> >               tasks file when the cpuset had an empty cpuset.cpus or empty
>     > >> > cpuset.mems
>     > >> >               setting.
>     > >> >
>     > >> >        ENOSPC Attempted to write(2) an empty cpuset.cpus or cpuset.mems
>     > >> > setting  to  a
>     > >> >               cpuset that has tasks attached.
> 
>     There are a few different things going on here,
> 
>     > I verify that cpuset.cpus (created by CRIU full cg manage) in the
>     > destination host is empty, while cpuset.cpus in the source host is
>     > "0-3".
>     >
>     > Further, when I attemped to write "0-3" to the cpuset.cpus in the
>     > destination host before restoring, it gave permission error.
>     >
>     > echo "0-3" > /sys/fs/cgroup/cpuset/[PID]/cpuset.cpus
>     > -bash: echo: write error: Permission denied
> 
>     You're probably not root here, so you need to do something like:
> 
>     $ echo "0-3" | sudo tee /sys/fs/cgroup/cpuset/foo/cpuset.cpus
>     0-3
> 
>     otherwise you'll get EPERM.
> 
> 
> I tried both sudo and root to restore and write to the cpuset.cpus file; both failed and gave me the error
> 
> tee: /sys/fs/cgroup/cpuset/foo/cpuset.cpus: Permission denied

It depends on the way to try :) If you just do sudo echo ... then it might not work (I
didn't research why). You should switch to root first with e.g. "sudo su -", then
do the echo command.

>     > Should CRIU write some default value to the cpuset.cpus file when
>     > creating the director? This could help restoring processes with cgroup
>     > across host. If so, I would be more than happy to work on a patch.
>     > Thanks. 
> 
> 
>     It should already try to do this in restore_special_cpuset_props(),
>     although it seems that bit isn't working for you. If you want to
>     debug, that's probably a good place to start.
> 
> 
> In fact the restoration of cgroup on a different host fails in the function restore_cgroup_prop. The error message I got is
> 
> prepare_cgroup_dirs
> (00.001362) cg: Created cgroup dir cpuset/foo
> (00.001429) Error (cgroup.c:978): cg: Failed closing cpuset/foo/cpuset.cpus: Permission denied
> (00.001441) Error (cgroup.c:1085): cg: Restoring special cpuset props failed!
> 
>  Then I changed chmod 777 to cpuset/foot and got those "permission denied" or "no space left" error.

-ENOSPC means that the allowed cpu/mem mask is empty and no tasks allowed there. Maybe
there're problems propagating the mask up the tree, need to strace ciru restore.

-EPERM is all about security.

> I am guessing that current criu implementation failed to propogate the cgroup dir across host. Have you ever test this? Thanks.

We did. P.haul project does live migration and (when done under root) works well.

-- Pavel