[CRIU] [PATCH] test, pipes: Exhaustive test of shared pipes

Mon Dec 12 05:37:51 PST 2016

On 12/12/2016 04:29 PM, Kirill Tkhai wrote:
> On 12.12.2016 16:27, Pavel Emelyanov wrote:
>> On 12/12/2016 04:08 PM, Kirill Tkhai wrote:
>>> On 12.12.2016 15:52, Pavel Emelyanov wrote:
>>>> On 12/12/2016 03:46 PM, Kirill Tkhai wrote:
>>>>> On 12.12.2016 15:46, Pavel Emelyanov wrote:
>>>>>> On 12/12/2016 01:04 PM, Kirill Tkhai wrote:
>>>>>>> On 09.12.2016 16:59, Pavel Emelyanov wrote:
>>>>>>>> So, here's the next test that just enumerates all possible states and checks
>>>>>>>> that CRIU C/R-s it well. This time -- pipes. The goal of the test is to load
>>>>>>>> the fd-sharing engine, so pipes are chosen, as they not only generate shared
>>>>>>>> struct files, but also produce 2 descriptors in CRIU's fdesc->open callback
>>>>>>>> which is handled separately.
>>>>>>>>
>>>>>>>> It's implemented slightly differently from the unix test, since we don't want
>>>>>>>> to check sequences of syscalls on objects, we need to check the task to pipe
>>>>>>>> relations in all possible ways.
>>>>>>>>
>>>>>>>> The 'state' is several tasks, several pipes and each generated test includes
>>>>>>>> pipe ends sitting in all possible combinations in the tasks' FDTs.
>>>>>>>>
>>>>>>>> Also note, that states, that seem to be equal to each other, e.g. pipe between
>>>>>>>> tasks A->B and pipe B->A, are really different as CRIU picks the pipe-restorer
>>>>>>>> based in task PIDs. So whether the picked task has read end or write end at 
>>>>>>>> his FDT makes a difference on restore.
>>>>>>>>
>>>>>>>> Number of tasks is limited with --tasks option, number of pipes with the
>>>>>>>> --pipes one. Test just runs all -- generates states, makes them and C/R-s
>>>>>>>> them. To check the restored result the /proc/pid/fd/ and /proc/pid/fdinfo/
>>>>>>>> for all restored tasks is analyzed.
>>>>>>>>
>>>>>>>> Right now CRIU works OK for --tasks 2 --pipes 2 (for more -- didn't check).
>>>>>>>> Kirill, please, check that your patches pass this test.
>>>>>>>
>>>>>>> I applied the test to criu-dev (without my patches) and run it:
>>>>>>>
>>>>>>> ~/criu# python ./test/exhaustive/pipe.py --tasks 2 --pipes 2
>>>>>>> Checking [(0, 0), (0, 0)]
>>>>>>> 	Make pipes
>>>>>>> 		Make pipes for 0
>>>>>>> 		Make pipes for 1
>>>>>>> 	Wait for C/R
>>>>>>> C/R test
>>>>>>> `- dump fail
>>>>>>> Wake up test
>>>>>>> Done (1, pid == 23208)
>>>>>>> FAIL
>>>>>>>
>>>>>>> Hm... Do I run it in a wrong way?
>>>>>>
>>>>>> Did you build criu before running the test?
>>>>>
>>>>> Yes, it's definitely built:
>>>>>
>>>>> root at pro:/home/kirill/criu# file criu/criu
>>>>> criu/criu: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.32, BuildID[sha1]=9933205650ef97e01e3258b42dc41eedb9e3efdd, not stripped
>>>>> root at pro:/home/kirill/criu# python ./test/exhaustive/pipe.py --tasks 2 --pipes 2
>>>>
>>>> OK. Try to do "cd test/exhaustive" before running it. Path to criu binary is
>>>> hard-coded to be "../../criu/criu"
>>>
>>> Now it works. Thanks.
>>>
>>> But vanila criu-dev branch has just failed once:
>>>
>>> ...
>>>
>>> C/R test
>>> Wake up test
>>> 	Check pipes
>>> Done (0, pid == 24684)
>>> Checking [(2, 1), (3, 2)]
>>> 	Make pipes
>>> 		Make pipes for 0
>>> 		Make pipes for 1
>>> 	Wait for C/R
>>> C/R test
>>> Wake up test
>>> 	Check pipes
>>> Done (0, pid == 24687)
>>> Checking [(2, 1), (3, 3)]
>>> 	Make pipes
>>> 		Make pipes for 0
>>> 		Make pipes for 1
>>> 	Wait for C/R
>>> C/R test
>>> `- restore fail
>>
>> What's in restore log?
> 
> (00.000050) Version: 2.8 (gitid v2.8-532-g13618be)
> (00.009443) Pagemap is fully functional
> (00.009609) Found task size of 7ffffffff000
> (00.015720) cpu: fpu:1 fxsr:1 xsave:1
> (00.016083) vdso: Parsing at 7fff3fb58000 7fff3fb5a000
> (00.016101) vdso: PT_LOAD p_vaddr: 0
> (00.016105) vdso: DT_HASH: 120
> (00.016108) vdso: DT_STRTAB: 298
> (00.016111) vdso: DT_SYMTAB: 1a8
> (00.016357) vdso: DT_STRSZ: 5e
> (00.016367) vdso: DT_SYMENT: 18
> (00.016371) vdso: nbucket 3 nchain a bucket 7fff3fb58128 chain 7fff3fb58134
> (00.016395) vdso: rt [vdso] 7fff3fb58000-7fff3fb5a000 [vvar] 7fff3fb56000-7fff3fb58000
> (00.016534) kernel pid_max=32768
> (00.016540) Reading image tree
> (00.016618) Add mnt ns 5 pid 7814
> (00.016650) pstree pid_max=7816
> (00.016674) Migrating process tree (GID 7776->7776 SID 4638->4638)
> (00.016680) Will restore in 0 namespaces
> (00.016685) NS mask to use 0
> (00.016698) Collecting 39/18 (flags 1)
> (00.016750) Collected [usr/bin/python2.7] ID 0x1
> (00.016757) Collected [usr/lib/x86_64-linux-gnu/libcrypto.so.1.1] ID 0x2
> (00.016762) Collected [usr/lib/x86_64-linux-gnu/libssl.so.1.1] ID 0x3
> (00.016766) Collected [usr/lib/python2.7/lib-dynload/_ssl.x86_64-linux-gnu.so] ID 0x4
> (00.016771) Collected [usr/lib/locale/locale-archive] ID 0x5
> (00.016775) Collected [lib/x86_64-linux-gnu/libc-2.24.so] ID 0x6
> (00.016780) Collected [lib/x86_64-linux-gnu/libm-2.24.so] ID 0x7
> (00.016786) Collected [lib/x86_64-linux-gnu/libz.so.1.2.8] ID 0x8
> (00.016791) Collected [lib/x86_64-linux-gnu/libutil-2.24.so] ID 0x9
> (00.016796) Collected [lib/x86_64-linux-gnu/libdl-2.24.so] ID 0xa
> (00.016801) Collected [lib/x86_64-linux-gnu/libpthread-2.24.so] ID 0xb
> (00.016805) Collected [lib/x86_64-linux-gnu/ld-2.24.so] ID 0xc
> (00.016809) Collected [dev/pts/1] ID 0xd
> (00.016817) Collected [home/kirill/criu/test/exhaustive] ID 0xf
> (00.016822) Collected [.] ID 0x10
> (00.016828)  `- ... done
> (00.016832) Collecting 53/57 (flags 0)
> (00.016839) No remap-fpath.img image
> (00.016846)  `- ... done
> (00.016850) Collecting 42/21 (flags 0)
> (00.016856) No inetsk.img image
> (00.016860)  `- ... done
> (00.016891) No cgroup.img image
> (00.016923) Running pre-restore scripts
> (00.017013) No mountpoints-5.img image
> (00.017018) mnt: Reading mountpoint images (id 5 pid 7814)
> (00.017167) Forking task with 7814 pid (flags 0x8000)
> (00.017517) PID: real 7815 virt 7814
> (00.017640) Wait until namespaces are created
> (00.017798) Error (criu/cr-restore.c:1363): Pid 7815 do not match expected 7814

Ah :( Pid conflict ... Looks like the test itself is still racy :\

As a workaround -- try to pin the whole test to a single CPU to reduce
this possibility.

> (00.018610) Error (criu/cr-restore.c:2013): Restoring FAILED.
> .
> 

-- Pavel