[CRIU] criu check --extra output and dump failure

Dmitry Safonov 0x7f454c46 at gmail.com
Fri Apr 14 10:58:41 PDT 2017


2017-04-13 17:54 GMT+03:00 Dmitry Safonov <0x7f454c46 at gmail.com>:
> 2017-04-13 17:33 GMT+03:00 Dmitry Safonov <0x7f454c46 at gmail.com>:
>> Let's keep in on the criu-ml, if anyone also has the same issue.
>> If you don't mind, CC: criu-ml
>>
>> 2017-04-13 16:59 GMT+03:00 Brinkmann, Harald
>> <Harald.Brinkmann at bst-international.com>:
>>>
>>> Hi Dmitry,
>>>
>>> On Wed, 2017-04-12 at 22:26 +0300, Dmitry Safonov wrote:
>>>> 2017-04-12 18:22 GMT+03:00 Dmitry Safonov <0x7f454c46 at gmail.com>:
>>>> > 2017-04-12 14:46 GMT+03:00 Brinkmann, Harald
>>>> > <Harald.Brinkmann at bst-international.com>:
>>>> >> On Tue, 2017-04-11 at 18:47 +0300, Dmitry Safonov wrote:
>>>> >>> 2017-04-11 16:16 GMT+03:00 Brinkmann, Harald
>>>> >>> <Harald.Brinkmann at bst-international.com>:
>>>> >>> > On Tue, 2017-04-11 at 14:51 +0300, Dmitry Safonov wrote:
>> [...]
>>>> >>> > This is from my objdump:
>>>> >>> >
>>>> >>> > 00000000 <__export_parasite_head_start>:
>>>> >>> >    0:   e24f2008        sub     r2, pc, #8
>>>> >>> >    4:   e28f0018        add     r0, pc, #24
>>>> >>> >    8:   e5900000        ldr     r0, [r0]
>>>> >>> >    c:   e28f100c        add     r1, pc, #12
>>>> >>> >   10:   e5911000        ldr     r1, [r1]
>>>> >>> >   14:   e0811002        add     r1, r1, r2
>>>> >>> >   18:   ebfffffe        bl      15a4 <parasite_service>
>>>> >>> >   1c:   e7f001f0        .word   0xe7f001f0
>>>> >>> >
>>>> >>> > And it looks like the jump to parasite_service() is broken as you first
>>>> >>> > suspected. Maybe this does give a signal 11 after all - as I understand
>>>> >>> > it is sort of an illegal address after all.
>>>> >>>
>>>> >>> Looks like 11 was generated by
>>>> >>> >   1c:   e7f001f0        .word   0xe7f001f0
>>>> >>> according to
>>>> >>> > [  154.088060] pc : [<76e78680>]    lr : [<76e7701c>]    psr: 20080010
>>>> >>
>>>> >> Forgive me for being a bit slow today, but how did you work that out?
>>>> >
>>>> > That's from dmesg you sent me.
>>>
>>> Yes, I know, but how do you tell from the dmesg line that it is object
>>> dump line '1c' which caused it?
>>
>> Well, surely I could tell more clearly:
>> in your dump.log there is a line, which says, where the parasite
>> was put:
>> (00.169005) Putting parasite blob into 0x76dd4000->0x76e77000
>>
>> And having in mind LR address from dmesg 76e7701c it's clearly
>> that parasite stepped at 1c and then jumped to unmapped address
>> which caused the segmentation violation.
>> So, this 1c is a guard for such situations, if you glance the source,
>> here it is (0x18-0x1c):
>>     bl    parasite_service
>>     .byte   0xf0, 0x01, 0xf0, 0xe7        @ the instruction UDF #32
>> generates the signal SIGTRAP in Linux
>>
>>>
>>>> So, I did cross-compilation with the following toolchain
>>>> gcc-linaro-arm-linux-gnueabihf-4.8-2014.04_linux, and hmm, all looks
>>>> good at the objdump:
>>>>
>>>> 00000000 <__export_parasite_head_start>:
>>>>        0:       e24f2008        sub     r2, pc, #8
>>>>        4:       e28f0018        add     r0, pc, #24
>>>>        8:       e5900000        ldr     r0, [r0]
>>>>        c:       e28f100c        add     r1, pc, #12
>>>>       10:       e5911000        ldr     r1, [r1]
>>>>       14:       e0811002        add     r1, r1, r2
>>>>       18:       fa000814        blx     2070 <parasite_service>
>>>>       1c:       e7f001f0        .word   0xe7f001f0
>>>>
>>>> Could you try Linaro cross-toolchain to check if there is something
>>>> with your build procedure or tools?
>>>>
>>>> Linaro releases are here:
>>>> https://releases.linaro.org/components/toolchain/binaries/
>>>
>>> I had no luck getting Linaro going today.
>>>
>>> Just wondering, is this possibly caused by protobuf or protobuf-c not
>>> being incorporated properly?
>>
>> No, that shouldn't be the cause of it.
>>
>>> Just wondering, could you send me the actual compiler commands generated
>>> by 'make' around the generation of the criu/pie/parasite.built-in.o
>>> file. Maybe there is a hint there somewhere that I could follow up?
>>
>> Here it is, note that I didn't do full criu building, rather only
>> parasite/restorer
>> PIEs - because I was lazy to copy shared libraries to host for linking
>> criu binary.
>> By that reason, there is /tmp/include path in log, as I've copied headers there.
>>
>> arm-linux-gnueabihf-gcc -M -MT criu/pie/util.d -MT criu/pie/util.o -I
>> /usr/include/google/ -I images -DCONFIG_ARMV7 -I /tmp/include -I
>> /usr/include/google/ -I images -DCONFIG_ARMV7 -I /tmp/include -O2 -g
>> -Wall -Wformat-security -Werror -DCONFIG_ARMV7 -D_FILE_OFFSET_BITS=64
>> -D_GNU_SOURCE -iquote include/ -fno-stack-protector -DCR_NOGLIBC -fpie
>> -fno-strict-aliasing -iquote criu/include -iquote include -iquote
>> images -iquote criu/arch/arm/include -iquote . -I/usr/include/libnl3
>> -I ./compel/include/uapi -I ./compel/include/uapi -marm
>> criu/pie/util.c -o criu/pie/util.d
>> arm-linux-gnueabihf-gcc -c -I /usr/include/google/ -I images
>> -DCONFIG_ARMV7 -I /tmp/include -I /usr/include/google/ -I images
>> -DCONFIG_ARMV7 -I /tmp/include -O2 -g -Wall -Wformat-security -Werror
>> -DCONFIG_ARMV7 -D_FILE_OFFSET_BITS=64 -D_GNU_SOURCE -iquote include/
>> -fno-stack-protector -DCR_NOGLIBC -fpie -fno-strict-aliasing -iquote
>> criu/include -iquote include -iquote images -iquote
>> criu/arch/arm/include -iquote . -I/usr/include/libnl3 -I
>> ./compel/include/uapi -I ./compel/include/uapi -marm criu/pie/util.c
>> -o criu/pie/util.o
>> arm-linux-gnueabihf-ar -rcs   criu/pie/pie.lib.a  criu/pie/util.o
>> make -r -R -f /home/japdoll/tools/criu/scripts/nmk/scripts/main.mk
>> makefile=Makefile obj=criu/pie all
>> arm-linux-gnueabihf-gcc -M -MT criu/pie/restorer.d -MT
>> criu/pie/restorer.o -I /usr/include/google/ -I images -DCONFIG_ARMV7
>> -I /tmp/include -I /usr/include/google/ -I images -DCONFIG_ARMV7 -I
>> /tmp/include -O2 -g -Wall -Wformat-security -Werror -DCONFIG_ARMV7
>> -D_FILE_OFFSET_BITS=64 -D_GNU_SOURCE -iquote include/
>> -Wstrict-prototypes -fno-stack-protector -nostdlib
>> -fomit-frame-pointer -fpie -I ./compel/include/uapi
>> -fno-strict-aliasing -iquote criu/include -iquote include -iquote
>> images -iquote criu/arch/arm/include -iquote . -I/usr/include/libnl3
>> -I ./compel/include/uapi -I ./compel/include/uapi -DCR_NOGLIBC
>> -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=0 -Wp,-U_FORTIFY_SOURCE
>> -Wp,-D_FORTIFY_SOURCE=0 -marm criu/pie/restorer.c -o
>> criu/pie/restorer.d
>> arm-linux-gnueabihf-gcc -M -MT criu/pie/parasite.d -MT
>> criu/pie/parasite.o -I /usr/include/google/ -I images -DCONFIG_ARMV7
>> -I /tmp/include -I /usr/include/google/ -I images -DCONFIG_ARMV7 -I
>> /tmp/include -O2 -g -Wall -Wformat-security -Werror -DCONFIG_ARMV7
>> -D_FILE_OFFSET_BITS=64 -D_GNU_SOURCE -iquote include/
>> -Wstrict-prototypes -fno-stack-protector -nostdlib
>> -fomit-frame-pointer -fpie -I ./compel/include/uapi
>> -fno-strict-aliasing -iquote criu/include -iquote include -iquote
>> images -iquote criu/arch/arm/include -iquote . -I/usr/include/libnl3
>> -I ./compel/include/uapi -I ./compel/include/uapi -DCR_NOGLIBC
>> -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=0 -Wp,-U_FORTIFY_SOURCE
>> -Wp,-D_FORTIFY_SOURCE=0 -marm criu/pie/parasite.c -o
>> criu/pie/parasite.d
>> arm-linux-gnueabihf-gcc -c -I /usr/include/google/ -I images
>> -DCONFIG_ARMV7 -I /tmp/include -I /usr/include/google/ -I images
>> -DCONFIG_ARMV7 -I /tmp/include -O2 -g -Wall -Wformat-security -Werror
>> -DCONFIG_ARMV7 -D_FILE_OFFSET_BITS=64 -D_GNU_SOURCE -iquote include/
>> -Wstrict-prototypes -fno-stack-protector -nostdlib
>> -fomit-frame-pointer -fpie -I ./compel/include/uapi
>> -fno-strict-aliasing -iquote criu/include -iquote include -iquote
>> images -iquote criu/arch/arm/include -iquote . -I/usr/include/libnl3
>> -I ./compel/include/uapi -I ./compel/include/uapi -DCR_NOGLIBC
>> -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=0 -Wp,-U_FORTIFY_SOURCE
>> -Wp,-D_FORTIFY_SOURCE=0 -marm criu/pie/parasite.c -o
>> criu/pie/parasite.o
>> arm-linux-gnueabihf-ld  -z noexecstack -T
>> ./compel/arch/arm/scripts/compel-pack.lds.S  -o
>> criu/pie/parasite.built-in.o  criu/pie/parasite.o criu/pie/pie.lib.a
>> ./compel/plugins/std.lib.a
>> ./compel/compel-host hgen -f criu/pie/parasite.built-in.o -o
>> criu/pie/parasite-blob.h
>> criu/pie/parasite-blob.h generated successfully.
>> arm-linux-gnueabihf-gcc -c -I /usr/include/google/ -I images
>> -DCONFIG_ARMV7 -I /tmp/include -I /usr/include/google/ -I images
>> -DCONFIG_ARMV7 -I /tmp/include -O2 -g -Wall -Wformat-security -Werror
>> -DCONFIG_ARMV7 -D_FILE_OFFSET_BITS=64 -D_GNU_SOURCE -iquote include/
>> -Wstrict-prototypes -fno-stack-protector -nostdlib
>> -fomit-frame-pointer -fpie -I ./compel/include/uapi
>> -fno-strict-aliasing -iquote criu/include -iquote include -iquote
>> images -iquote criu/arch/arm/include -iquote . -I/usr/include/libnl3
>> -I ./compel/include/uapi -I ./compel/include/uapi -DCR_NOGLIBC
>> -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=0 -Wp,-U_FORTIFY_SOURCE
>> -Wp,-D_FORTIFY_SOURCE=0 -marm criu/pie/restorer.c -o
>> criu/pie/restorer.o
>> arm-linux-gnueabihf-ld  -z noexecstack -T
>> ./compel/arch/arm/scripts/compel-pack.lds.S  -o
>> criu/pie/restorer.built-in.o  ./criu/arch/arm/restorer.o
>> criu/pie/restorer.o criu/pie/pie.lib.a ./compel/plugins/std.lib.a
>> ./compel/compel-host hgen -f criu/pie/restorer.built-in.o -o
>> criu/pie/restorer-blob.h
>> criu/pie/restorer-blob.h generated successfully.
>>
>> The only hint, that was the solution of the very similar issue as yours
>> previously, is that ld line shouldn't have `-r' flag. Otherwise, toolchains
>> that I've tried didn't do relocations in resulting PIE. There was also a
>> couple of arm reports about the same relocation issue, so removing this
>> flag solved issues for me and community people.
>>
>>> arm-linux-gnueabihf-ld  -z noexecstack -T ./compel/arch/arm/scripts/compel-pack.lds.S  -o criu/pie/parasite.built-in.o  criu/pie/parasite.o criu/pie/pie.lib.a ./compel/plugins/std.lib.a
>
>
> Another hint: as you do full cross-compilation of rootfs for
> your target, check ld flags for building some shared library,
> and check the difference between them and linking
> parasite.built-in.o
> For a test you can take parasite-head.S and add small
> stub.c file which provides in C a function named parasite_service()
> I've attached files for linking test.
>
> Try to build it with your toolchain like:
> arm-linux-gnueabihf-gcc -c stub.c parasite-head.S
> arm-linux-gnueabihf-ld -z noexecstack -o built-in.o stub.o
> parasite-head.o -T compel-pack.lds.S
>
> In result you should have in built-in.o relocated relative call
> to parasite_service():
> arm-linux-gnueabihf-objdump -dS built-in.o | head -n20
> [...]
>   18:    fa000004     blx    30 <parasite_service>
>   1c:    e7f001f0     .word    0xe7f001f0
> [...]
> 00000030 <parasite_service>:
>   30:    b480          push    {r7}
>   32:    b083          sub    sp, #12
>   34:    af00          add    r7, sp, #0
>   36:    6078          str    r0, [r7, #4]
>   38:    6039          str    r1, [r7, #0]
> [...]
>
> If it's not what you have - you'll need to find how it works for
> shared libraries in your environment.

Any news on that?

-- 
             Dmitry


More information about the CRIU mailing list