[CRIU] Restore CRIU image across a host

[Hui Kang]

Hi, Tycho
I created a patch
(http://lists.openvz.org/pipermail/criu/2015-August/021576.html) and
pass cgrou_manage as CG_MODE_FULL to the criu service. From the
restore.log, I can see that criu will smartly create all the cgroup
directories. However, the restore still fails at this cpu of wrting to

(cgroup.c:901): cg: Can't move into cpuset//[PID]/tasks (-1/0): No
space left on device
(00.082249) Error (cr-restore.c:1919): Restoring FAILED.

As Tycho pointed that:

>> >        ENOSPC Attempted to write(2) the process ID (PID) of  a  process
>> > to  a  cpuset
>> >               tasks file when the cpuset had an empty cpuset.cpus or empty
>> > cpuset.mems
>> >               setting.
>> >
>> >        ENOSPC Attempted to write(2) an empty cpuset.cpus or cpuset.mems
>> > setting  to  a
>> >               cpuset that has tasks attached.

I verify that cpuset.cpus (created by CRIU full cg manage) in the
destination host is empty, while cpuset.cpus in the source host is
"0-3".

Further, when I attemped to write "0-3" to the cpuset.cpus in the
destination host before restoring, it gave permission error.

echo "0-3" > /sys/fs/cgroup/cpuset/[PID]/cpuset.cpus
-bash: echo: write error: Permission denied

Should CRIU write some default value to the cpuset.cpus file when
creating the director? This could help restoring processes with cgroup
across host. If so, I would be more than happy to work on a patch.
Thanks.

- Hui


On Tue, Aug 4, 2015 at 5:43 PM, Tycho Andersen
<tycho.andersen at canonical.com> wrote:
> On Tue, Aug 04, 2015 at 05:40:48PM -0400, Hui Kang wrote:
>> Hi, Tycho
>> Thanks for your suggestion. I am using criu RPC calls, where I can not find
>> the four mode for cgroup-manager. Instead, it is a bool type value.
>>
>> optional bool                 manage_cgroups  = 24;
>>
>> Should I update my criu version (I am using v1.6)?
>
> Oh, that's probably because the RPC struct wasn't updated to match the
> new CLI options. It looks like in the current tree it is also a bool,
> so updating to master won't help you. I'm sure a patch would be
> welcome, though :)
>
> Tycho
>
>> - Hui
>>
>> On Tue, Aug 4, 2015 at 5:15 PM, Tycho Andersen <tycho.andersen at canonical.com
>> > wrote:
>>
>> > On Tue, Aug 04, 2015 at 05:01:14PM -0400, Hui Kang wrote:
>> > > Hi,
>> > > I am trying to restore the process on a different criu host by
>> > transferring
>> > > the CRIU images. The checkpointed criu images are scp-ed to a different
>> > > host; however the restore fails. The restore.log shows that
>> > >
>> > > (00.075765)      1: nr_restored_pages: 66
>> > > (00.075772)      1: nr_shared_pages:   0
>> > > (00.075779)      1: nr_droped_pages:   0
>> > > (00.075829)      1: Found fd 1 (id pipe:[13922]) in inherit fd list
>> > (caller
>> > > close_old_fds)
>> > > (00.075841)      1: Found fd 2 (id pipe:[13923]) in inherit fd list
>> > (caller
>> > > close_old_fds)
>> > > (00.075859)      1: cg: Move into 2
>> > >  1: cg:   `-> blkio//[PID]/tasks
>> > > (00.075906)      1: cg:   `-> cpu//[PID]/tasks
>> > > (00.075946)      1: cg:   `-> cpuacct//[PID]/tasks
>> > > (00.075967)      1: cg:   `-> cpuset//[PID]/tasks
>> > > (00.076008)      1: Error (cgroup.c:901): cg: Can't move into
>> > > cpuset//[PID]/tasks (-1/0): No space left on device
>> > > (00.082249) Error (cr-restore.c:1919): Restoring FAILED.
>> > >
>> > > Note that the process being migrated has its cgroup directory in the
>> > source
>> > > node, which apparently does not exist in the destination host. So I first
>> > > mkdir /sys/fs/cgroup/cpu/PID .... and copy the content from the source to
>> > > destination.
>> >
>> > You might want to use one of the new `--manage-cgroups full` mode to
>> > do this for you. That said, moving something into cpuset's tasks fails
>> > with ENOSPC for the following reasons (from cpuset(7)):
>> >
>> >        ENOSPC Attempted to write(2) the process ID (PID) of  a  process
>> > to  a  cpuset
>> >               tasks file when the cpuset had an empty cpuset.cpus or empty
>> > cpuset.mems
>> >               setting.
>> >
>> >        ENOSPC Attempted to write(2) an empty cpuset.cpus or cpuset.mems
>> > setting  to  a
>> >               cpuset that has tasks attached.
>> >
>> > So depending on what you copied over, it can cause some problems. In
>> > `--manage-cgroups full` mode, criu's code should be smart enough to handle
>> > things, though.
>> >
>> > Tycho
>> >
>> > > Is there anything wrong in these steps? Thanks.
>> > >
>> > > - Hui
>> >
>> > > _______________________________________________
>> > > CRIU mailing list
>> > > CRIU at openvz.org
>> > > https://lists.openvz.org/mailman/listinfo/criu
>> >
>> >