[CRIU] criu_restore() in Open MPI problems

[Pavel Emelyanov]

On 03/19/2014 06:05 PM, Adrian Reber wrote:
> On Wed, Mar 19, 2014 at 05:41:40PM +0400, Andrew Vagin wrote:
>> On Wed, Mar 19, 2014 at 11:29:51AM +0100, Adrian Reber wrote:
>>> On Wed, Mar 19, 2014 at 12:33:30PM +0400, Andrew Vagin wrote:
>>>> Could you try out the attached patch?
>>>
>>> With this patch it actually tries to restore process but fails with:
>>>
>>> (00.026193)  15852: tty: open type pts id 0x2 index 11 (master 0 sid 0 pgrp 0 inherit 1)
>>> (00.026198)  15852: Error (tty.c:541): tty: Can't dup SELF_STDIN_OFF: Bad file descriptor
>>> (00.026782) Error (cr-restore.c:1035): 15852 exited, status=255
>>> (00.026810) Error (cr-restore.c:1577): Restoring FAILED.
>>>
>>> Full log at http://lisas.de/~adrian/criu.log
>>>
>>> Which is probably related to the way Open MPI handles stdout/stderr of
>>> its child processes. I need to find out how this exactly works.
>>
>> As far as I understand you are executing criu as a service, aren't you?
> 
> Yes. Criu as a service and libcriu linked into Open MPI. The code is
> something like:
> 
>     criu_set_images_dir_fd(fd);
> 
>     criu_set_log_file(mca_crs_criu_component.log_file);
>     criu_set_log_level(mca_crs_criu_component.log_level);
>     criu_set_tcp_established(mca_crs_criu_component.tcp_established);
>     criu_set_shell_job(mca_crs_criu_component.shell_job);
>     criu_set_ext_unix_sk(mca_crs_criu_component.ext_unix_sk);
>     criu_set_leave_running(mca_crs_criu_component.leave_running);
> 
>     ret = criu_restore();
> 
> 
>> We have understood that the shell_job option on restore can't work
>> correctly in this case, because a link on parent and a session can't be
>> restored correctly. Both this parameters can be inhereted and can not be
>> set.
>>
>> Looks like we have only one way to execute "criu restore" directly.
>> Maybe we will need to set the suid bit on criu, because it requires
>> CAP_SYS_ADMIN and CAP_SYS_RESOURCE.
>>
>> Adrian, I want to know a bit more about structure of a process tree,
>> could you provide a bit more info:
>>
>> * ps axf -o sid,gid,pid,cmd,uid,gid
> 
>  9042   500 19148   500   500  |   \_ /home/adrian/devel/openmpi-trunk/bin/orterun --am ft-enable-cr -np 1 orte-test2
>  9042   500 19151   500   500  |       \_ orte-test2

It looks like service is not the right choice for such kind of restoration -- it
cannot re-parent restored subtree into arbitrary task.

Maybe in that case we should put suid bit in crtoos binary and just fork-and-exec one?