[CRIU] criu check --extra output and dump failure
Dmitry Safonov
0x7f454c46 at gmail.com
Fri Apr 14 10:58:41 PDT 2017
2017-04-13 17:54 GMT+03:00 Dmitry Safonov <0x7f454c46 at gmail.com>:
> 2017-04-13 17:33 GMT+03:00 Dmitry Safonov <0x7f454c46 at gmail.com>:
>> Let's keep in on the criu-ml, if anyone also has the same issue.
>> If you don't mind, CC: criu-ml
>>
>> 2017-04-13 16:59 GMT+03:00 Brinkmann, Harald
>> <Harald.Brinkmann at bst-international.com>:
>>>
>>> Hi Dmitry,
>>>
>>> On Wed, 2017-04-12 at 22:26 +0300, Dmitry Safonov wrote:
>>>> 2017-04-12 18:22 GMT+03:00 Dmitry Safonov <0x7f454c46 at gmail.com>:
>>>> > 2017-04-12 14:46 GMT+03:00 Brinkmann, Harald
>>>> > <Harald.Brinkmann at bst-international.com>:
>>>> >> On Tue, 2017-04-11 at 18:47 +0300, Dmitry Safonov wrote:
>>>> >>> 2017-04-11 16:16 GMT+03:00 Brinkmann, Harald
>>>> >>> <Harald.Brinkmann at bst-international.com>:
>>>> >>> > On Tue, 2017-04-11 at 14:51 +0300, Dmitry Safonov wrote:
>> [...]
>>>> >>> > This is from my objdump:
>>>> >>> >
>>>> >>> > 00000000 <__export_parasite_head_start>:
>>>> >>> > 0: e24f2008 sub r2, pc, #8
>>>> >>> > 4: e28f0018 add r0, pc, #24
>>>> >>> > 8: e5900000 ldr r0, [r0]
>>>> >>> > c: e28f100c add r1, pc, #12
>>>> >>> > 10: e5911000 ldr r1, [r1]
>>>> >>> > 14: e0811002 add r1, r1, r2
>>>> >>> > 18: ebfffffe bl 15a4 <parasite_service>
>>>> >>> > 1c: e7f001f0 .word 0xe7f001f0
>>>> >>> >
>>>> >>> > And it looks like the jump to parasite_service() is broken as you first
>>>> >>> > suspected. Maybe this does give a signal 11 after all - as I understand
>>>> >>> > it is sort of an illegal address after all.
>>>> >>>
>>>> >>> Looks like 11 was generated by
>>>> >>> > 1c: e7f001f0 .word 0xe7f001f0
>>>> >>> according to
>>>> >>> > [ 154.088060] pc : [<76e78680>] lr : [<76e7701c>] psr: 20080010
>>>> >>
>>>> >> Forgive me for being a bit slow today, but how did you work that out?
>>>> >
>>>> > That's from dmesg you sent me.
>>>
>>> Yes, I know, but how do you tell from the dmesg line that it is object
>>> dump line '1c' which caused it?
>>
>> Well, surely I could tell more clearly:
>> in your dump.log there is a line, which says, where the parasite
>> was put:
>> (00.169005) Putting parasite blob into 0x76dd4000->0x76e77000
>>
>> And having in mind LR address from dmesg 76e7701c it's clearly
>> that parasite stepped at 1c and then jumped to unmapped address
>> which caused the segmentation violation.
>> So, this 1c is a guard for such situations, if you glance the source,
>> here it is (0x18-0x1c):
>> bl parasite_service
>> .byte 0xf0, 0x01, 0xf0, 0xe7 @ the instruction UDF #32
>> generates the signal SIGTRAP in Linux
>>
>>>
>>>> So, I did cross-compilation with the following toolchain
>>>> gcc-linaro-arm-linux-gnueabihf-4.8-2014.04_linux, and hmm, all looks
>>>> good at the objdump:
>>>>
>>>> 00000000 <__export_parasite_head_start>:
>>>> 0: e24f2008 sub r2, pc, #8
>>>> 4: e28f0018 add r0, pc, #24
>>>> 8: e5900000 ldr r0, [r0]
>>>> c: e28f100c add r1, pc, #12
>>>> 10: e5911000 ldr r1, [r1]
>>>> 14: e0811002 add r1, r1, r2
>>>> 18: fa000814 blx 2070 <parasite_service>
>>>> 1c: e7f001f0 .word 0xe7f001f0
>>>>
>>>> Could you try Linaro cross-toolchain to check if there is something
>>>> with your build procedure or tools?
>>>>
>>>> Linaro releases are here:
>>>> https://releases.linaro.org/components/toolchain/binaries/
>>>
>>> I had no luck getting Linaro going today.
>>>
>>> Just wondering, is this possibly caused by protobuf or protobuf-c not
>>> being incorporated properly?
>>
>> No, that shouldn't be the cause of it.
>>
>>> Just wondering, could you send me the actual compiler commands generated
>>> by 'make' around the generation of the criu/pie/parasite.built-in.o
>>> file. Maybe there is a hint there somewhere that I could follow up?
>>
>> Here it is, note that I didn't do full criu building, rather only
>> parasite/restorer
>> PIEs - because I was lazy to copy shared libraries to host for linking
>> criu binary.
>> By that reason, there is /tmp/include path in log, as I've copied headers there.
>>
>> arm-linux-gnueabihf-gcc -M -MT criu/pie/util.d -MT criu/pie/util.o -I
>> /usr/include/google/ -I images -DCONFIG_ARMV7 -I /tmp/include -I
>> /usr/include/google/ -I images -DCONFIG_ARMV7 -I /tmp/include -O2 -g
>> -Wall -Wformat-security -Werror -DCONFIG_ARMV7 -D_FILE_OFFSET_BITS=64
>> -D_GNU_SOURCE -iquote include/ -fno-stack-protector -DCR_NOGLIBC -fpie
>> -fno-strict-aliasing -iquote criu/include -iquote include -iquote
>> images -iquote criu/arch/arm/include -iquote . -I/usr/include/libnl3
>> -I ./compel/include/uapi -I ./compel/include/uapi -marm
>> criu/pie/util.c -o criu/pie/util.d
>> arm-linux-gnueabihf-gcc -c -I /usr/include/google/ -I images
>> -DCONFIG_ARMV7 -I /tmp/include -I /usr/include/google/ -I images
>> -DCONFIG_ARMV7 -I /tmp/include -O2 -g -Wall -Wformat-security -Werror
>> -DCONFIG_ARMV7 -D_FILE_OFFSET_BITS=64 -D_GNU_SOURCE -iquote include/
>> -fno-stack-protector -DCR_NOGLIBC -fpie -fno-strict-aliasing -iquote
>> criu/include -iquote include -iquote images -iquote
>> criu/arch/arm/include -iquote . -I/usr/include/libnl3 -I
>> ./compel/include/uapi -I ./compel/include/uapi -marm criu/pie/util.c
>> -o criu/pie/util.o
>> arm-linux-gnueabihf-ar -rcs criu/pie/pie.lib.a criu/pie/util.o
>> make -r -R -f /home/japdoll/tools/criu/scripts/nmk/scripts/main.mk
>> makefile=Makefile obj=criu/pie all
>> arm-linux-gnueabihf-gcc -M -MT criu/pie/restorer.d -MT
>> criu/pie/restorer.o -I /usr/include/google/ -I images -DCONFIG_ARMV7
>> -I /tmp/include -I /usr/include/google/ -I images -DCONFIG_ARMV7 -I
>> /tmp/include -O2 -g -Wall -Wformat-security -Werror -DCONFIG_ARMV7
>> -D_FILE_OFFSET_BITS=64 -D_GNU_SOURCE -iquote include/
>> -Wstrict-prototypes -fno-stack-protector -nostdlib
>> -fomit-frame-pointer -fpie -I ./compel/include/uapi
>> -fno-strict-aliasing -iquote criu/include -iquote include -iquote
>> images -iquote criu/arch/arm/include -iquote . -I/usr/include/libnl3
>> -I ./compel/include/uapi -I ./compel/include/uapi -DCR_NOGLIBC
>> -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=0 -Wp,-U_FORTIFY_SOURCE
>> -Wp,-D_FORTIFY_SOURCE=0 -marm criu/pie/restorer.c -o
>> criu/pie/restorer.d
>> arm-linux-gnueabihf-gcc -M -MT criu/pie/parasite.d -MT
>> criu/pie/parasite.o -I /usr/include/google/ -I images -DCONFIG_ARMV7
>> -I /tmp/include -I /usr/include/google/ -I images -DCONFIG_ARMV7 -I
>> /tmp/include -O2 -g -Wall -Wformat-security -Werror -DCONFIG_ARMV7
>> -D_FILE_OFFSET_BITS=64 -D_GNU_SOURCE -iquote include/
>> -Wstrict-prototypes -fno-stack-protector -nostdlib
>> -fomit-frame-pointer -fpie -I ./compel/include/uapi
>> -fno-strict-aliasing -iquote criu/include -iquote include -iquote
>> images -iquote criu/arch/arm/include -iquote . -I/usr/include/libnl3
>> -I ./compel/include/uapi -I ./compel/include/uapi -DCR_NOGLIBC
>> -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=0 -Wp,-U_FORTIFY_SOURCE
>> -Wp,-D_FORTIFY_SOURCE=0 -marm criu/pie/parasite.c -o
>> criu/pie/parasite.d
>> arm-linux-gnueabihf-gcc -c -I /usr/include/google/ -I images
>> -DCONFIG_ARMV7 -I /tmp/include -I /usr/include/google/ -I images
>> -DCONFIG_ARMV7 -I /tmp/include -O2 -g -Wall -Wformat-security -Werror
>> -DCONFIG_ARMV7 -D_FILE_OFFSET_BITS=64 -D_GNU_SOURCE -iquote include/
>> -Wstrict-prototypes -fno-stack-protector -nostdlib
>> -fomit-frame-pointer -fpie -I ./compel/include/uapi
>> -fno-strict-aliasing -iquote criu/include -iquote include -iquote
>> images -iquote criu/arch/arm/include -iquote . -I/usr/include/libnl3
>> -I ./compel/include/uapi -I ./compel/include/uapi -DCR_NOGLIBC
>> -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=0 -Wp,-U_FORTIFY_SOURCE
>> -Wp,-D_FORTIFY_SOURCE=0 -marm criu/pie/parasite.c -o
>> criu/pie/parasite.o
>> arm-linux-gnueabihf-ld -z noexecstack -T
>> ./compel/arch/arm/scripts/compel-pack.lds.S -o
>> criu/pie/parasite.built-in.o criu/pie/parasite.o criu/pie/pie.lib.a
>> ./compel/plugins/std.lib.a
>> ./compel/compel-host hgen -f criu/pie/parasite.built-in.o -o
>> criu/pie/parasite-blob.h
>> criu/pie/parasite-blob.h generated successfully.
>> arm-linux-gnueabihf-gcc -c -I /usr/include/google/ -I images
>> -DCONFIG_ARMV7 -I /tmp/include -I /usr/include/google/ -I images
>> -DCONFIG_ARMV7 -I /tmp/include -O2 -g -Wall -Wformat-security -Werror
>> -DCONFIG_ARMV7 -D_FILE_OFFSET_BITS=64 -D_GNU_SOURCE -iquote include/
>> -Wstrict-prototypes -fno-stack-protector -nostdlib
>> -fomit-frame-pointer -fpie -I ./compel/include/uapi
>> -fno-strict-aliasing -iquote criu/include -iquote include -iquote
>> images -iquote criu/arch/arm/include -iquote . -I/usr/include/libnl3
>> -I ./compel/include/uapi -I ./compel/include/uapi -DCR_NOGLIBC
>> -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=0 -Wp,-U_FORTIFY_SOURCE
>> -Wp,-D_FORTIFY_SOURCE=0 -marm criu/pie/restorer.c -o
>> criu/pie/restorer.o
>> arm-linux-gnueabihf-ld -z noexecstack -T
>> ./compel/arch/arm/scripts/compel-pack.lds.S -o
>> criu/pie/restorer.built-in.o ./criu/arch/arm/restorer.o
>> criu/pie/restorer.o criu/pie/pie.lib.a ./compel/plugins/std.lib.a
>> ./compel/compel-host hgen -f criu/pie/restorer.built-in.o -o
>> criu/pie/restorer-blob.h
>> criu/pie/restorer-blob.h generated successfully.
>>
>> The only hint, that was the solution of the very similar issue as yours
>> previously, is that ld line shouldn't have `-r' flag. Otherwise, toolchains
>> that I've tried didn't do relocations in resulting PIE. There was also a
>> couple of arm reports about the same relocation issue, so removing this
>> flag solved issues for me and community people.
>>
>>> arm-linux-gnueabihf-ld -z noexecstack -T ./compel/arch/arm/scripts/compel-pack.lds.S -o criu/pie/parasite.built-in.o criu/pie/parasite.o criu/pie/pie.lib.a ./compel/plugins/std.lib.a
>
>
> Another hint: as you do full cross-compilation of rootfs for
> your target, check ld flags for building some shared library,
> and check the difference between them and linking
> parasite.built-in.o
> For a test you can take parasite-head.S and add small
> stub.c file which provides in C a function named parasite_service()
> I've attached files for linking test.
>
> Try to build it with your toolchain like:
> arm-linux-gnueabihf-gcc -c stub.c parasite-head.S
> arm-linux-gnueabihf-ld -z noexecstack -o built-in.o stub.o
> parasite-head.o -T compel-pack.lds.S
>
> In result you should have in built-in.o relocated relative call
> to parasite_service():
> arm-linux-gnueabihf-objdump -dS built-in.o | head -n20
> [...]
> 18: fa000004 blx 30 <parasite_service>
> 1c: e7f001f0 .word 0xe7f001f0
> [...]
> 00000030 <parasite_service>:
> 30: b480 push {r7}
> 32: b083 sub sp, #12
> 34: af00 add r7, sp, #0
> 36: 6078 str r0, [r7, #4]
> 38: 6039 str r1, [r7, #0]
> [...]
>
> If it's not what you have - you'll need to find how it works for
> shared libraries in your environment.
Any news on that?
--
Dmitry
More information about the CRIU
mailing list