[CRIU] [PATCH] test, pipes: Exhaustive test of shared pipes

Kirill Tkhai ktkhai at virtuozzo.com
Tue Dec 13 03:46:12 PST 2016



On 12.12.2016 22:10, Pavel Emelyanov wrote:
> On 12/12/2016 04:46 PM, Kirill Tkhai wrote:
>> On 12.12.2016 16:37, Pavel Emelyanov wrote:
>>> On 12/12/2016 04:29 PM, Kirill Tkhai wrote:
>>>> On 12.12.2016 16:27, Pavel Emelyanov wrote:
>>>>> On 12/12/2016 04:08 PM, Kirill Tkhai wrote:
>>>>>> On 12.12.2016 15:52, Pavel Emelyanov wrote:
>>>>>>> On 12/12/2016 03:46 PM, Kirill Tkhai wrote:
>>>>>>>> On 12.12.2016 15:46, Pavel Emelyanov wrote:
>>>>>>>>> On 12/12/2016 01:04 PM, Kirill Tkhai wrote:
>>>>>>>>>> On 09.12.2016 16:59, Pavel Emelyanov wrote:
>>>>>>>>>>> So, here's the next test that just enumerates all possible states and checks
>>>>>>>>>>> that CRIU C/R-s it well. This time -- pipes. The goal of the test is to load
>>>>>>>>>>> the fd-sharing engine, so pipes are chosen, as they not only generate shared
>>>>>>>>>>> struct files, but also produce 2 descriptors in CRIU's fdesc->open callback
>>>>>>>>>>> which is handled separately.
>>>>>>>>>>>
>>>>>>>>>>> It's implemented slightly differently from the unix test, since we don't want
>>>>>>>>>>> to check sequences of syscalls on objects, we need to check the task to pipe
>>>>>>>>>>> relations in all possible ways.
>>>>>>>>>>>
>>>>>>>>>>> The 'state' is several tasks, several pipes and each generated test includes
>>>>>>>>>>> pipe ends sitting in all possible combinations in the tasks' FDTs.
>>>>>>>>>>>
>>>>>>>>>>> Also note, that states, that seem to be equal to each other, e.g. pipe between
>>>>>>>>>>> tasks A->B and pipe B->A, are really different as CRIU picks the pipe-restorer
>>>>>>>>>>> based in task PIDs. So whether the picked task has read end or write end at 
>>>>>>>>>>> his FDT makes a difference on restore.
>>>>>>>>>>>
>>>>>>>>>>> Number of tasks is limited with --tasks option, number of pipes with the
>>>>>>>>>>> --pipes one. Test just runs all -- generates states, makes them and C/R-s
>>>>>>>>>>> them. To check the restored result the /proc/pid/fd/ and /proc/pid/fdinfo/
>>>>>>>>>>> for all restored tasks is analyzed.
>>>>>>>>>>>
>>>>>>>>>>> Right now CRIU works OK for --tasks 2 --pipes 2 (for more -- didn't check).
>>>>>>>>>>> Kirill, please, check that your patches pass this test.
>>>>>>>>>>
>>>>>>>>>> I applied the test to criu-dev (without my patches) and run it:
>>>>>>>>>>
>>>>>>>>>> ~/criu# python ./test/exhaustive/pipe.py --tasks 2 --pipes 2
>>>>>>>>>> Checking [(0, 0), (0, 0)]
>>>>>>>>>> 	Make pipes
>>>>>>>>>> 		Make pipes for 0
>>>>>>>>>> 		Make pipes for 1
>>>>>>>>>> 	Wait for C/R
>>>>>>>>>> C/R test
>>>>>>>>>> `- dump fail
>>>>>>>>>> Wake up test
>>>>>>>>>> Done (1, pid == 23208)
>>>>>>>>>> FAIL
>>>>>>>>>>
>>>>>>>>>> Hm... Do I run it in a wrong way?
>>>>>>>>>
>>>>>>>>> Did you build criu before running the test?
>>>>>>>>
>>>>>>>> Yes, it's definitely built:
>>>>>>>>
>>>>>>>> root at pro:/home/kirill/criu# file criu/criu
>>>>>>>> criu/criu: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.32, BuildID[sha1]=9933205650ef97e01e3258b42dc41eedb9e3efdd, not stripped
>>>>>>>> root at pro:/home/kirill/criu# python ./test/exhaustive/pipe.py --tasks 2 --pipes 2
>>>>>>>
>>>>>>> OK. Try to do "cd test/exhaustive" before running it. Path to criu binary is
>>>>>>> hard-coded to be "../../criu/criu"
>>>>>>
>>>>>> Now it works. Thanks.
>>>>>>
>>>>>> But vanila criu-dev branch has just failed once:
>>>>>>
>>>>>> ...
>>>>>>
>>>>>> C/R test
>>>>>> Wake up test
>>>>>> 	Check pipes
>>>>>> Done (0, pid == 24684)
>>>>>> Checking [(2, 1), (3, 2)]
>>>>>> 	Make pipes
>>>>>> 		Make pipes for 0
>>>>>> 		Make pipes for 1
>>>>>> 	Wait for C/R
>>>>>> C/R test
>>>>>> Wake up test
>>>>>> 	Check pipes
>>>>>> Done (0, pid == 24687)
>>>>>> Checking [(2, 1), (3, 3)]
>>>>>> 	Make pipes
>>>>>> 		Make pipes for 0
>>>>>> 		Make pipes for 1
>>>>>> 	Wait for C/R
>>>>>> C/R test
>>>>>> `- restore fail
>>>>>
>>>>> What's in restore log?
>>>>
>>>> (00.000050) Version: 2.8 (gitid v2.8-532-g13618be)
>>>> (00.009443) Pagemap is fully functional
>>>> (00.009609) Found task size of 7ffffffff000
>>>> (00.015720) cpu: fpu:1 fxsr:1 xsave:1
>>>> (00.016083) vdso: Parsing at 7fff3fb58000 7fff3fb5a000
>>>> (00.016101) vdso: PT_LOAD p_vaddr: 0
>>>> (00.016105) vdso: DT_HASH: 120
>>>> (00.016108) vdso: DT_STRTAB: 298
>>>> (00.016111) vdso: DT_SYMTAB: 1a8
>>>> (00.016357) vdso: DT_STRSZ: 5e
>>>> (00.016367) vdso: DT_SYMENT: 18
>>>> (00.016371) vdso: nbucket 3 nchain a bucket 7fff3fb58128 chain 7fff3fb58134
>>>> (00.016395) vdso: rt [vdso] 7fff3fb58000-7fff3fb5a000 [vvar] 7fff3fb56000-7fff3fb58000
>>>> (00.016534) kernel pid_max=32768
>>>> (00.016540) Reading image tree
>>>> (00.016618) Add mnt ns 5 pid 7814
>>>> (00.016650) pstree pid_max=7816
>>>> (00.016674) Migrating process tree (GID 7776->7776 SID 4638->4638)
>>>> (00.016680) Will restore in 0 namespaces
>>>> (00.016685) NS mask to use 0
>>>> (00.016698) Collecting 39/18 (flags 1)
>>>> (00.016750) Collected [usr/bin/python2.7] ID 0x1
>>>> (00.016757) Collected [usr/lib/x86_64-linux-gnu/libcrypto.so.1.1] ID 0x2
>>>> (00.016762) Collected [usr/lib/x86_64-linux-gnu/libssl.so.1.1] ID 0x3
>>>> (00.016766) Collected [usr/lib/python2.7/lib-dynload/_ssl.x86_64-linux-gnu.so] ID 0x4
>>>> (00.016771) Collected [usr/lib/locale/locale-archive] ID 0x5
>>>> (00.016775) Collected [lib/x86_64-linux-gnu/libc-2.24.so] ID 0x6
>>>> (00.016780) Collected [lib/x86_64-linux-gnu/libm-2.24.so] ID 0x7
>>>> (00.016786) Collected [lib/x86_64-linux-gnu/libz.so.1.2.8] ID 0x8
>>>> (00.016791) Collected [lib/x86_64-linux-gnu/libutil-2.24.so] ID 0x9
>>>> (00.016796) Collected [lib/x86_64-linux-gnu/libdl-2.24.so] ID 0xa
>>>> (00.016801) Collected [lib/x86_64-linux-gnu/libpthread-2.24.so] ID 0xb
>>>> (00.016805) Collected [lib/x86_64-linux-gnu/ld-2.24.so] ID 0xc
>>>> (00.016809) Collected [dev/pts/1] ID 0xd
>>>> (00.016817) Collected [home/kirill/criu/test/exhaustive] ID 0xf
>>>> (00.016822) Collected [.] ID 0x10
>>>> (00.016828)  `- ... done
>>>> (00.016832) Collecting 53/57 (flags 0)
>>>> (00.016839) No remap-fpath.img image
>>>> (00.016846)  `- ... done
>>>> (00.016850) Collecting 42/21 (flags 0)
>>>> (00.016856) No inetsk.img image
>>>> (00.016860)  `- ... done
>>>> (00.016891) No cgroup.img image
>>>> (00.016923) Running pre-restore scripts
>>>> (00.017013) No mountpoints-5.img image
>>>> (00.017018) mnt: Reading mountpoint images (id 5 pid 7814)
>>>> (00.017167) Forking task with 7814 pid (flags 0x8000)
>>>> (00.017517) PID: real 7815 virt 7814
>>>> (00.017640) Wait until namespaces are created
>>>> (00.017798) Error (criu/cr-restore.c:1363): Pid 7815 do not match expected 7814
>>>
>>> Ah :( Pid conflict ... Looks like the test itself is still racy :\
>>>
>>> As a workaround -- try to pin the whole test to a single CPU to reduce
>>> this possibility.
>>
>> Still fails :( I tried run with taskset 0x1 ...
> 
> Erm :\  It's hackish, but try to add 'time.sleep(0.2) before the line 252
> "print 'Done (%d, pid == %d)' % (st, pid)". Just to proceed with checking
> patches.

I add time.sleep(1) and it still fails... :(
 
>> If you need a reproduction, kernel compilation in background seems to be good for that...
>> .
>>
> 


More information about the CRIU mailing list