[CRIU] CRIU segfaulting when restoring a process

Dmitry Safonov dsafonov at virtuozzo.com
Fri Aug 19 04:40:49 PDT 2016


On 08/19/2016 01:25 PM, Nikolay Borisov wrote:
>
>
> On 08/19/2016 01:03 PM, Dmitry Safonov wrote:
>> On 08/19/2016 11:16 AM, Nikolay Borisov wrote:
>>>
>>>
>>> On 08/18/2016 06:13 PM, Dmitry Safonov wrote:
>>>> On 08/18/2016 04:44 PM, Nikolay Borisov wrote:
>>>>> Hello,
>>>>
>>>> Hi Nikolay,
>>>>
>>>>> I've built CRIU 2.5 from source + some patches which move around stuff
>>>>> in the headers to facilitate compilation on centos 6.7 with external
>>>>> glibc 2.19. My CRIU is built the following way:
>>>>>
>>>>> make -j8 USERCFLAGS="-I/opt/glibc-2.19/include/ -L/opt/glibc-2.19//lib/
>>>>> -Wl,-dynamic-linker=/opt/glibc-2.19//lib/ld-linux-x86-64.so.2 -Wl,-
>>>>> rpath=/opt/glibc-2.19/lib/:/usr/lib64/:/lib64/"
>>>>>
>>>>> This way I can happily dump a simple process a la
>>>>> https://criu.org/Simple_loop style. However my problems begin when I
>>>>> try
>>>>> to restore the process, since CRIU segfaults. Here is a restore log as
>>>>> well as strace from the restore process:
>>>>>
>>>>> http://sprunge.us/DcIh - restore.log
>>>>> http://sprunge.us/CVBG - strace.log
>>>>>
>>>>> I'd happy if you could shed some light what I might be causing the
>>>>> problem. One thing I thought might be the difference between the way
>>>>> the
>>>>> process being restore - bash is compiled and criu. Here is a comparison
>>>>> how they libraries look like: http://paste.ubuntu.com/23067392/ should
>>>>> it matter of course
>>>>
>>>> So, the problem seems to be in the restorer blob:
>>>> Switching to the restorer was sucessful:
>>>>> 23537 write(199999, "(00.188706)  23537: task_args:
>>>> 0x20000\ntask_args->pid: 23537\ntask_args->nr_threads:
>>>> 1\ntask_args->clo"..., 155) = 155
>>>>> 23537 getpid()                          = 23537
>>>>
>>>> which is sys_getpid() in __export_restore_task().
>>>> The fault address is very strange one:
>>>> 23537 --- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR,
>>>> si_addr=0x12024b48d48} ---
>>>>
>>>> So, the fail is somwhere between getpid() and sigaction() calls in
>>>> __export_restore_task() (criu/pie/restorer.c), as we would see
>>>> sys_sigaction() if it has been called.
>>>>
>>>> Could you give a shot with the next diff and paste strace output before
>>>> failure?
>>>>
>>>> --->8--->8--->8---8<---8<---8<---
>>>> diff --git a/criu/pie/restorer.c b/criu/pie/restorer.c
>>>> index 7cc735c96870..8503078a82d9 100644
>>>> --- a/criu/pie/restorer.c
>>>> +++ b/criu/pie/restorer.c
>>>> @@ -1123,6 +1123,7 @@ long __export_restore_task(struct
>>>> task_restore_args *args)
>>>>      n_helpers = args->helpers_n;
>>>>      zombies = args->zombies;
>>>>      n_zombies = args->zombies_n;
>>>> +    sys_getpid();
>>>>      *args->breakpoint = rst_sigreturn;
>>>>
>>>>      ksigfillset(&act.rt_sa_mask);
>>>>
>>>> --->8--->8--->8---8<---8<---8<---
>>>>
>>>> I suspect that the problem is in *args pointer to be garbage by some
>>>> reason -- if we find another getpid() call in strace log, that's not
>>>> the reason and it's somewhere in ksigfillset() (which is unlikely).
>>>>
>>>> And let me think a while, why *args may have such strange junk inside
>>>> (0x12024b48d48).
>>>>
>>>
>>>
>>> So here is an strace with your patch applied, it looks a bit different
>>> indeed - http://sprunge.us/HBEV
>>
>> Hmm, I don't see the second call to getpid(), but *args, which are in
>> %rdi looks quite normal (0x20000).
>>
>>> I checked the disassembly my compiler produces for the
>>> __export_restore_task and it indeed has a prologue, setting up the
>>> stack. So that looks good indeed.
>>>
>>> Also here are the register state at the time the crash occurs with your
>>> patch applied:
>>>
>>> (gdb) info register
>>> rax            0x23000    143360
>>> rbx            0x20000    131072
>>> rcx            0x12dad    77229
>>> rdx            0x48000042b0058948    5188147057151805768
>>> rsi            0x6fc8a0    7325856
>>> rdi            0x20000    131072
>>> rbp            0x5b16    0x5b16
>>> rsp            0x1eec0    0x1eec0
>>> r8             0x1    1
>>> r9             0x1    1
>>> r10            0x7fffbf9c4d70    140736408079728
>>> r11            0x206    518
>>> r12            0x1f070    127088
>>> r13            0x703e20    7355936
>>> r14            0x203c0    132032
>>> r15            0x7fffffffde60    140737488346720
>>> rip            0x10b27    0x10b27
>>> eflags         0x10206    [ PF IF RF ]
>>> cs             0x33    51
>>> ss             0x2b    43
>>> ds             0x0    0
>>> es             0x0    0
>>> fs             0x0    0
>>> gs             0x0    0
>>
>> So %rdi is fine, %rsp also, AFAICS, everything looks just fine.
>> Could you provide disassembly for __export_restore_task -- till 0xb27
>> address for this binary?
>> Like: $ objdump -dS criu/pie/restorer.built-in.o
>> I belive it's a load from *args, but...
>
> Does this help:  http://paste.ubuntu.com/23069854/ ?

Yes, thanks.
So, it does crash here:
b27:       48 89 02                mov    %rax,(%rdx)
And %rdx was set earlier here:
afa:       48 8b 15 00 00 00 00    mov    0x0(%rip),%rdx

This is simple R_X86_64_PC32 relocation at this place.
So, as compel successfuly patched this relative relocations here
(in the same function earlier):
      b01:       48 89 05 00 00 00 00    mov    %rax,0x0(%rip)
      b0f:       89 05 00 00 00 00       mov    %eax,0x0(%rip)
      b1c:       48 89 05 00 00 00 00    mov    %rax,0x0(%rip)

It should have resolved this place too (the reason of fail)
afa:       48 8b 15 00 00 00 00    mov    0x0(%rip),%rdx

The question, has compel patched it?
To check it, it would be worth if you do the following:
1. compile CRIU, like `make -j5`
2. then do `touch criu/pie/restorer.c`
3. copy, please, the output of `make V=1` to pastebin.

>>
>>> My theory here is that since CRIU is compiled with a non-standard glibc,
>>> it has started being interpreted by glibc 2.19's interpreter
>>> (ld-linux-x86-64.so.2) and when it's time to restore the process which
>>> is bash it starts executing it with the new interpreted and my bash
>>> indeed doesn't work with it and segfaults. Does this sound possible?
>>
>> Well, it's a good theory, but it can't be applied as the application
>> wasn't restored fully. So your restoree process yet has not gained the
>> control and wasn't resumed to execute. The fail is in restore process,
>> not in the result of it.
>> So, I think there shouldn't be anything special about using external
>> glibc -- all should C/R as normal.
>
> One way to test this would be to create a simple C-based application and
> link it against glibc 2.19 and then try to C/R that one. This will
> completely debunk the theory that the glibc is the culprit.

Well, will see, but at this moment I don't see any problem with this.

>>
>> The only concern is restorer binary size -- for me address is:
>> 0000000000001520 <__export_restore_task>
>>
>> And the size at run-time is:
>> (00.035761)  29972: Found bootstrap VMA hint at: 0x10000 (needs ~104K)
>>
>> While yours is on a page lesser -- but that's likely just ok.
>
> Regarding your VM question - this is a simple centos 6.7 installation,
> running on kernel 4.4 (CRIU check says that everything looks good) and
> I've built criu with the following build script:
> http://paste.ubuntu.com/23069868/

Thanks for the script -- if we don't debug it through emails, I guess,
I'll try to do the same on the weekend maybe :D


-- 
              Dmitry


More information about the CRIU mailing list