[CRIU] [PATCH] test, pipes: Exhaustive test of shared pipes

Mon Dec 12 11:10:25 PST 2016

On 12/12/2016 04:46 PM, Kirill Tkhai wrote:
> On 12.12.2016 16:37, Pavel Emelyanov wrote:
>> On 12/12/2016 04:29 PM, Kirill Tkhai wrote:
>>> On 12.12.2016 16:27, Pavel Emelyanov wrote:
>>>> On 12/12/2016 04:08 PM, Kirill Tkhai wrote:
>>>>> On 12.12.2016 15:52, Pavel Emelyanov wrote:
>>>>>> On 12/12/2016 03:46 PM, Kirill Tkhai wrote:
>>>>>>> On 12.12.2016 15:46, Pavel Emelyanov wrote:
>>>>>>>> On 12/12/2016 01:04 PM, Kirill Tkhai wrote:
>>>>>>>>> On 09.12.2016 16:59, Pavel Emelyanov wrote:
>>>>>>>>>> So, here's the next test that just enumerates all possible states and checks
>>>>>>>>>> that CRIU C/R-s it well. This time -- pipes. The goal of the test is to load
>>>>>>>>>> the fd-sharing engine, so pipes are chosen, as they not only generate shared
>>>>>>>>>> struct files, but also produce 2 descriptors in CRIU's fdesc->open callback
>>>>>>>>>> which is handled separately.
>>>>>>>>>>
>>>>>>>>>> It's implemented slightly differently from the unix test, since we don't want
>>>>>>>>>> to check sequences of syscalls on objects, we need to check the task to pipe
>>>>>>>>>> relations in all possible ways.
>>>>>>>>>>
>>>>>>>>>> The 'state' is several tasks, several pipes and each generated test includes
>>>>>>>>>> pipe ends sitting in all possible combinations in the tasks' FDTs.
>>>>>>>>>>
>>>>>>>>>> Also note, that states, that seem to be equal to each other, e.g. pipe between
>>>>>>>>>> tasks A->B and pipe B->A, are really different as CRIU picks the pipe-restorer
>>>>>>>>>> based in task PIDs. So whether the picked task has read end or write end at 
>>>>>>>>>> his FDT makes a difference on restore.
>>>>>>>>>>
>>>>>>>>>> Number of tasks is limited with --tasks option, number of pipes with the
>>>>>>>>>> --pipes one. Test just runs all -- generates states, makes them and C/R-s
>>>>>>>>>> them. To check the restored result the /proc/pid/fd/ and /proc/pid/fdinfo/
>>>>>>>>>> for all restored tasks is analyzed.
>>>>>>>>>>
>>>>>>>>>> Right now CRIU works OK for --tasks 2 --pipes 2 (for more -- didn't check).
>>>>>>>>>> Kirill, please, check that your patches pass this test.
>>>>>>>>>
>>>>>>>>> I applied the test to criu-dev (without my patches) and run it:
>>>>>>>>>
>>>>>>>>> ~/criu# python ./test/exhaustive/pipe.py --tasks 2 --pipes 2
>>>>>>>>> Checking [(0, 0), (0, 0)]
>>>>>>>>> 	Make pipes
>>>>>>>>> 		Make pipes for 0
>>>>>>>>> 		Make pipes for 1
>>>>>>>>> 	Wait for C/R
>>>>>>>>> C/R test
>>>>>>>>> `- dump fail
>>>>>>>>> Wake up test
>>>>>>>>> Done (1, pid == 23208)
>>>>>>>>> FAIL
>>>>>>>>>
>>>>>>>>> Hm... Do I run it in a wrong way?
>>>>>>>>
>>>>>>>> Did you build criu before running the test?
>>>>>>>
>>>>>>> Yes, it's definitely built:
>>>>>>>
>>>>>>> root at pro:/home/kirill/criu# file criu/criu
>>>>>>> criu/criu: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.32, BuildID[sha1]=9933205650ef97e01e3258b42dc41eedb9e3efdd, not stripped
>>>>>>> root at pro:/home/kirill/criu# python ./test/exhaustive/pipe.py --tasks 2 --pipes 2
>>>>>>
>>>>>> OK. Try to do "cd test/exhaustive" before running it. Path to criu binary is
>>>>>> hard-coded to be "../../criu/criu"
>>>>>
>>>>> Now it works. Thanks.
>>>>>
>>>>> But vanila criu-dev branch has just failed once:
>>>>>
>>>>> ...
>>>>>
>>>>> C/R test
>>>>> Wake up test
>>>>> 	Check pipes
>>>>> Done (0, pid == 24684)
>>>>> Checking [(2, 1), (3, 2)]
>>>>> 	Make pipes
>>>>> 		Make pipes for 0
>>>>> 		Make pipes for 1
>>>>> 	Wait for C/R
>>>>> C/R test
>>>>> Wake up test
>>>>> 	Check pipes
>>>>> Done (0, pid == 24687)
>>>>> Checking [(2, 1), (3, 3)]
>>>>> 	Make pipes
>>>>> 		Make pipes for 0
>>>>> 		Make pipes for 1
>>>>> 	Wait for C/R
>>>>> C/R test
>>>>> `- restore fail
>>>>
>>>> What's in restore log?
>>>
>>> (00.000050) Version: 2.8 (gitid v2.8-532-g13618be)
>>> (00.009443) Pagemap is fully functional
>>> (00.009609) Found task size of 7ffffffff000
>>> (00.015720) cpu: fpu:1 fxsr:1 xsave:1
>>> (00.016083) vdso: Parsing at 7fff3fb58000 7fff3fb5a000
>>> (00.016101) vdso: PT_LOAD p_vaddr: 0
>>> (00.016105) vdso: DT_HASH: 120
>>> (00.016108) vdso: DT_STRTAB: 298
>>> (00.016111) vdso: DT_SYMTAB: 1a8
>>> (00.016357) vdso: DT_STRSZ: 5e
>>> (00.016367) vdso: DT_SYMENT: 18
>>> (00.016371) vdso: nbucket 3 nchain a bucket 7fff3fb58128 chain 7fff3fb58134
>>> (00.016395) vdso: rt [vdso] 7fff3fb58000-7fff3fb5a000 [vvar] 7fff3fb56000-7fff3fb58000
>>> (00.016534) kernel pid_max=32768
>>> (00.016540) Reading image tree
>>> (00.016618) Add mnt ns 5 pid 7814
>>> (00.016650) pstree pid_max=7816
>>> (00.016674) Migrating process tree (GID 7776->7776 SID 4638->4638)
>>> (00.016680) Will restore in 0 namespaces
>>> (00.016685) NS mask to use 0
>>> (00.016698) Collecting 39/18 (flags 1)
>>> (00.016750) Collected [usr/bin/python2.7] ID 0x1
>>> (00.016757) Collected [usr/lib/x86_64-linux-gnu/libcrypto.so.1.1] ID 0x2
>>> (00.016762) Collected [usr/lib/x86_64-linux-gnu/libssl.so.1.1] ID 0x3
>>> (00.016766) Collected [usr/lib/python2.7/lib-dynload/_ssl.x86_64-linux-gnu.so] ID 0x4
>>> (00.016771) Collected [usr/lib/locale/locale-archive] ID 0x5
>>> (00.016775) Collected [lib/x86_64-linux-gnu/libc-2.24.so] ID 0x6
>>> (00.016780) Collected [lib/x86_64-linux-gnu/libm-2.24.so] ID 0x7
>>> (00.016786) Collected [lib/x86_64-linux-gnu/libz.so.1.2.8] ID 0x8
>>> (00.016791) Collected [lib/x86_64-linux-gnu/libutil-2.24.so] ID 0x9
>>> (00.016796) Collected [lib/x86_64-linux-gnu/libdl-2.24.so] ID 0xa
>>> (00.016801) Collected [lib/x86_64-linux-gnu/libpthread-2.24.so] ID 0xb
>>> (00.016805) Collected [lib/x86_64-linux-gnu/ld-2.24.so] ID 0xc
>>> (00.016809) Collected [dev/pts/1] ID 0xd
>>> (00.016817) Collected [home/kirill/criu/test/exhaustive] ID 0xf
>>> (00.016822) Collected [.] ID 0x10
>>> (00.016828)  `- ... done
>>> (00.016832) Collecting 53/57 (flags 0)
>>> (00.016839) No remap-fpath.img image
>>> (00.016846)  `- ... done
>>> (00.016850) Collecting 42/21 (flags 0)
>>> (00.016856) No inetsk.img image
>>> (00.016860)  `- ... done
>>> (00.016891) No cgroup.img image
>>> (00.016923) Running pre-restore scripts
>>> (00.017013) No mountpoints-5.img image
>>> (00.017018) mnt: Reading mountpoint images (id 5 pid 7814)
>>> (00.017167) Forking task with 7814 pid (flags 0x8000)
>>> (00.017517) PID: real 7815 virt 7814
>>> (00.017640) Wait until namespaces are created
>>> (00.017798) Error (criu/cr-restore.c:1363): Pid 7815 do not match expected 7814
>>
>> Ah :( Pid conflict ... Looks like the test itself is still racy :\
>>
>> As a workaround -- try to pin the whole test to a single CPU to reduce
>> this possibility.
> 
> Still fails :( I tried run with taskset 0x1 ...

Erm :\  It's hackish, but try to add 'time.sleep(0.2) before the line 252
"print 'Done (%d, pid == %d)' % (st, pid)". Just to proceed with checking
patches.

> If you need a reproduction, kernel compilation in background seems to be good for that...
> .
> 