[CRIU] [PATCH] test, pipes: Exhaustive test of shared pipes

Mon Dec 12 05:46:40 PST 2016

On 12.12.2016 16:37, Pavel Emelyanov wrote:
> On 12/12/2016 04:29 PM, Kirill Tkhai wrote:
>> On 12.12.2016 16:27, Pavel Emelyanov wrote:
>>> On 12/12/2016 04:08 PM, Kirill Tkhai wrote:
>>>> On 12.12.2016 15:52, Pavel Emelyanov wrote:
>>>>> On 12/12/2016 03:46 PM, Kirill Tkhai wrote:
>>>>>> On 12.12.2016 15:46, Pavel Emelyanov wrote:
>>>>>>> On 12/12/2016 01:04 PM, Kirill Tkhai wrote:
>>>>>>>> On 09.12.2016 16:59, Pavel Emelyanov wrote:
>>>>>>>>> So, here's the next test that just enumerates all possible states and checks
>>>>>>>>> that CRIU C/R-s it well. This time -- pipes. The goal of the test is to load
>>>>>>>>> the fd-sharing engine, so pipes are chosen, as they not only generate shared
>>>>>>>>> struct files, but also produce 2 descriptors in CRIU's fdesc->open callback
>>>>>>>>> which is handled separately.
>>>>>>>>>
>>>>>>>>> It's implemented slightly differently from the unix test, since we don't want
>>>>>>>>> to check sequences of syscalls on objects, we need to check the task to pipe
>>>>>>>>> relations in all possible ways.
>>>>>>>>>
>>>>>>>>> The 'state' is several tasks, several pipes and each generated test includes
>>>>>>>>> pipe ends sitting in all possible combinations in the tasks' FDTs.
>>>>>>>>>
>>>>>>>>> Also note, that states, that seem to be equal to each other, e.g. pipe between
>>>>>>>>> tasks A->B and pipe B->A, are really different as CRIU picks the pipe-restorer
>>>>>>>>> based in task PIDs. So whether the picked task has read end or write end at 
>>>>>>>>> his FDT makes a difference on restore.
>>>>>>>>>
>>>>>>>>> Number of tasks is limited with --tasks option, number of pipes with the
>>>>>>>>> --pipes one. Test just runs all -- generates states, makes them and C/R-s
>>>>>>>>> them. To check the restored result the /proc/pid/fd/ and /proc/pid/fdinfo/
>>>>>>>>> for all restored tasks is analyzed.
>>>>>>>>>
>>>>>>>>> Right now CRIU works OK for --tasks 2 --pipes 2 (for more -- didn't check).
>>>>>>>>> Kirill, please, check that your patches pass this test.
>>>>>>>>
>>>>>>>> I applied the test to criu-dev (without my patches) and run it:
>>>>>>>>
>>>>>>>> ~/criu# python ./test/exhaustive/pipe.py --tasks 2 --pipes 2
>>>>>>>> Checking [(0, 0), (0, 0)]
>>>>>>>> 	Make pipes
>>>>>>>> 		Make pipes for 0
>>>>>>>> 		Make pipes for 1
>>>>>>>> 	Wait for C/R
>>>>>>>> C/R test
>>>>>>>> `- dump fail
>>>>>>>> Wake up test
>>>>>>>> Done (1, pid == 23208)
>>>>>>>> FAIL
>>>>>>>>
>>>>>>>> Hm... Do I run it in a wrong way?
>>>>>>>
>>>>>>> Did you build criu before running the test?
>>>>>>
>>>>>> Yes, it's definitely built:
>>>>>>
>>>>>> root at pro:/home/kirill/criu# file criu/criu
>>>>>> criu/criu: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.32, BuildID[sha1]=9933205650ef97e01e3258b42dc41eedb9e3efdd, not stripped
>>>>>> root at pro:/home/kirill/criu# python ./test/exhaustive/pipe.py --tasks 2 --pipes 2
>>>>>
>>>>> OK. Try to do "cd test/exhaustive" before running it. Path to criu binary is
>>>>> hard-coded to be "../../criu/criu"
>>>>
>>>> Now it works. Thanks.
>>>>
>>>> But vanila criu-dev branch has just failed once:
>>>>
>>>> ...
>>>>
>>>> C/R test
>>>> Wake up test
>>>> 	Check pipes
>>>> Done (0, pid == 24684)
>>>> Checking [(2, 1), (3, 2)]
>>>> 	Make pipes
>>>> 		Make pipes for 0
>>>> 		Make pipes for 1
>>>> 	Wait for C/R
>>>> C/R test
>>>> Wake up test
>>>> 	Check pipes
>>>> Done (0, pid == 24687)
>>>> Checking [(2, 1), (3, 3)]
>>>> 	Make pipes
>>>> 		Make pipes for 0
>>>> 		Make pipes for 1
>>>> 	Wait for C/R
>>>> C/R test
>>>> `- restore fail
>>>
>>> What's in restore log?
>>
>> (00.000050) Version: 2.8 (gitid v2.8-532-g13618be)
>> (00.009443) Pagemap is fully functional
>> (00.009609) Found task size of 7ffffffff000
>> (00.015720) cpu: fpu:1 fxsr:1 xsave:1
>> (00.016083) vdso: Parsing at 7fff3fb58000 7fff3fb5a000
>> (00.016101) vdso: PT_LOAD p_vaddr: 0
>> (00.016105) vdso: DT_HASH: 120
>> (00.016108) vdso: DT_STRTAB: 298
>> (00.016111) vdso: DT_SYMTAB: 1a8
>> (00.016357) vdso: DT_STRSZ: 5e
>> (00.016367) vdso: DT_SYMENT: 18
>> (00.016371) vdso: nbucket 3 nchain a bucket 7fff3fb58128 chain 7fff3fb58134
>> (00.016395) vdso: rt [vdso] 7fff3fb58000-7fff3fb5a000 [vvar] 7fff3fb56000-7fff3fb58000
>> (00.016534) kernel pid_max=32768
>> (00.016540) Reading image tree
>> (00.016618) Add mnt ns 5 pid 7814
>> (00.016650) pstree pid_max=7816
>> (00.016674) Migrating process tree (GID 7776->7776 SID 4638->4638)
>> (00.016680) Will restore in 0 namespaces
>> (00.016685) NS mask to use 0
>> (00.016698) Collecting 39/18 (flags 1)
>> (00.016750) Collected [usr/bin/python2.7] ID 0x1
>> (00.016757) Collected [usr/lib/x86_64-linux-gnu/libcrypto.so.1.1] ID 0x2
>> (00.016762) Collected [usr/lib/x86_64-linux-gnu/libssl.so.1.1] ID 0x3
>> (00.016766) Collected [usr/lib/python2.7/lib-dynload/_ssl.x86_64-linux-gnu.so] ID 0x4
>> (00.016771) Collected [usr/lib/locale/locale-archive] ID 0x5
>> (00.016775) Collected [lib/x86_64-linux-gnu/libc-2.24.so] ID 0x6
>> (00.016780) Collected [lib/x86_64-linux-gnu/libm-2.24.so] ID 0x7
>> (00.016786) Collected [lib/x86_64-linux-gnu/libz.so.1.2.8] ID 0x8
>> (00.016791) Collected [lib/x86_64-linux-gnu/libutil-2.24.so] ID 0x9
>> (00.016796) Collected [lib/x86_64-linux-gnu/libdl-2.24.so] ID 0xa
>> (00.016801) Collected [lib/x86_64-linux-gnu/libpthread-2.24.so] ID 0xb
>> (00.016805) Collected [lib/x86_64-linux-gnu/ld-2.24.so] ID 0xc
>> (00.016809) Collected [dev/pts/1] ID 0xd
>> (00.016817) Collected [home/kirill/criu/test/exhaustive] ID 0xf
>> (00.016822) Collected [.] ID 0x10
>> (00.016828)  `- ... done
>> (00.016832) Collecting 53/57 (flags 0)
>> (00.016839) No remap-fpath.img image
>> (00.016846)  `- ... done
>> (00.016850) Collecting 42/21 (flags 0)
>> (00.016856) No inetsk.img image
>> (00.016860)  `- ... done
>> (00.016891) No cgroup.img image
>> (00.016923) Running pre-restore scripts
>> (00.017013) No mountpoints-5.img image
>> (00.017018) mnt: Reading mountpoint images (id 5 pid 7814)
>> (00.017167) Forking task with 7814 pid (flags 0x8000)
>> (00.017517) PID: real 7815 virt 7814
>> (00.017640) Wait until namespaces are created
>> (00.017798) Error (criu/cr-restore.c:1363): Pid 7815 do not match expected 7814
> 
> Ah :( Pid conflict ... Looks like the test itself is still racy :\
> 
> As a workaround -- try to pin the whole test to a single CPU to reduce
> this possibility.

Still fails :( I tried run with taskset 0x1 ...

If you need a reproduction, kernel compilation in background seems to be good for that...