[CRIU] criu check --extra output and dump failure
Dmitry Safonov
0x7f454c46 at gmail.com
Thu Apr 13 07:54:06 PDT 2017
2017-04-13 17:33 GMT+03:00 Dmitry Safonov <0x7f454c46 at gmail.com>:
> Let's keep in on the criu-ml, if anyone also has the same issue.
> If you don't mind, CC: criu-ml
>
> 2017-04-13 16:59 GMT+03:00 Brinkmann, Harald
> <Harald.Brinkmann at bst-international.com>:
>>
>> Hi Dmitry,
>>
>> On Wed, 2017-04-12 at 22:26 +0300, Dmitry Safonov wrote:
>>> 2017-04-12 18:22 GMT+03:00 Dmitry Safonov <0x7f454c46 at gmail.com>:
>>> > 2017-04-12 14:46 GMT+03:00 Brinkmann, Harald
>>> > <Harald.Brinkmann at bst-international.com>:
>>> >> On Tue, 2017-04-11 at 18:47 +0300, Dmitry Safonov wrote:
>>> >>> 2017-04-11 16:16 GMT+03:00 Brinkmann, Harald
>>> >>> <Harald.Brinkmann at bst-international.com>:
>>> >>> > On Tue, 2017-04-11 at 14:51 +0300, Dmitry Safonov wrote:
> [...]
>>> >>> > This is from my objdump:
>>> >>> >
>>> >>> > 00000000 <__export_parasite_head_start>:
>>> >>> > 0: e24f2008 sub r2, pc, #8
>>> >>> > 4: e28f0018 add r0, pc, #24
>>> >>> > 8: e5900000 ldr r0, [r0]
>>> >>> > c: e28f100c add r1, pc, #12
>>> >>> > 10: e5911000 ldr r1, [r1]
>>> >>> > 14: e0811002 add r1, r1, r2
>>> >>> > 18: ebfffffe bl 15a4 <parasite_service>
>>> >>> > 1c: e7f001f0 .word 0xe7f001f0
>>> >>> >
>>> >>> > And it looks like the jump to parasite_service() is broken as you first
>>> >>> > suspected. Maybe this does give a signal 11 after all - as I understand
>>> >>> > it is sort of an illegal address after all.
>>> >>>
>>> >>> Looks like 11 was generated by
>>> >>> > 1c: e7f001f0 .word 0xe7f001f0
>>> >>> according to
>>> >>> > [ 154.088060] pc : [<76e78680>] lr : [<76e7701c>] psr: 20080010
>>> >>
>>> >> Forgive me for being a bit slow today, but how did you work that out?
>>> >
>>> > That's from dmesg you sent me.
>>
>> Yes, I know, but how do you tell from the dmesg line that it is object
>> dump line '1c' which caused it?
>
> Well, surely I could tell more clearly:
> in your dump.log there is a line, which says, where the parasite
> was put:
> (00.169005) Putting parasite blob into 0x76dd4000->0x76e77000
>
> And having in mind LR address from dmesg 76e7701c it's clearly
> that parasite stepped at 1c and then jumped to unmapped address
> which caused the segmentation violation.
> So, this 1c is a guard for such situations, if you glance the source,
> here it is (0x18-0x1c):
> bl parasite_service
> .byte 0xf0, 0x01, 0xf0, 0xe7 @ the instruction UDF #32
> generates the signal SIGTRAP in Linux
>
>>
>>> So, I did cross-compilation with the following toolchain
>>> gcc-linaro-arm-linux-gnueabihf-4.8-2014.04_linux, and hmm, all looks
>>> good at the objdump:
>>>
>>> 00000000 <__export_parasite_head_start>:
>>> 0: e24f2008 sub r2, pc, #8
>>> 4: e28f0018 add r0, pc, #24
>>> 8: e5900000 ldr r0, [r0]
>>> c: e28f100c add r1, pc, #12
>>> 10: e5911000 ldr r1, [r1]
>>> 14: e0811002 add r1, r1, r2
>>> 18: fa000814 blx 2070 <parasite_service>
>>> 1c: e7f001f0 .word 0xe7f001f0
>>>
>>> Could you try Linaro cross-toolchain to check if there is something
>>> with your build procedure or tools?
>>>
>>> Linaro releases are here:
>>> https://releases.linaro.org/components/toolchain/binaries/
>>
>> I had no luck getting Linaro going today.
>>
>> Just wondering, is this possibly caused by protobuf or protobuf-c not
>> being incorporated properly?
>
> No, that shouldn't be the cause of it.
>
>> Just wondering, could you send me the actual compiler commands generated
>> by 'make' around the generation of the criu/pie/parasite.built-in.o
>> file. Maybe there is a hint there somewhere that I could follow up?
>
> Here it is, note that I didn't do full criu building, rather only
> parasite/restorer
> PIEs - because I was lazy to copy shared libraries to host for linking
> criu binary.
> By that reason, there is /tmp/include path in log, as I've copied headers there.
>
> arm-linux-gnueabihf-gcc -M -MT criu/pie/util.d -MT criu/pie/util.o -I
> /usr/include/google/ -I images -DCONFIG_ARMV7 -I /tmp/include -I
> /usr/include/google/ -I images -DCONFIG_ARMV7 -I /tmp/include -O2 -g
> -Wall -Wformat-security -Werror -DCONFIG_ARMV7 -D_FILE_OFFSET_BITS=64
> -D_GNU_SOURCE -iquote include/ -fno-stack-protector -DCR_NOGLIBC -fpie
> -fno-strict-aliasing -iquote criu/include -iquote include -iquote
> images -iquote criu/arch/arm/include -iquote . -I/usr/include/libnl3
> -I ./compel/include/uapi -I ./compel/include/uapi -marm
> criu/pie/util.c -o criu/pie/util.d
> arm-linux-gnueabihf-gcc -c -I /usr/include/google/ -I images
> -DCONFIG_ARMV7 -I /tmp/include -I /usr/include/google/ -I images
> -DCONFIG_ARMV7 -I /tmp/include -O2 -g -Wall -Wformat-security -Werror
> -DCONFIG_ARMV7 -D_FILE_OFFSET_BITS=64 -D_GNU_SOURCE -iquote include/
> -fno-stack-protector -DCR_NOGLIBC -fpie -fno-strict-aliasing -iquote
> criu/include -iquote include -iquote images -iquote
> criu/arch/arm/include -iquote . -I/usr/include/libnl3 -I
> ./compel/include/uapi -I ./compel/include/uapi -marm criu/pie/util.c
> -o criu/pie/util.o
> arm-linux-gnueabihf-ar -rcs criu/pie/pie.lib.a criu/pie/util.o
> make -r -R -f /home/japdoll/tools/criu/scripts/nmk/scripts/main.mk
> makefile=Makefile obj=criu/pie all
> arm-linux-gnueabihf-gcc -M -MT criu/pie/restorer.d -MT
> criu/pie/restorer.o -I /usr/include/google/ -I images -DCONFIG_ARMV7
> -I /tmp/include -I /usr/include/google/ -I images -DCONFIG_ARMV7 -I
> /tmp/include -O2 -g -Wall -Wformat-security -Werror -DCONFIG_ARMV7
> -D_FILE_OFFSET_BITS=64 -D_GNU_SOURCE -iquote include/
> -Wstrict-prototypes -fno-stack-protector -nostdlib
> -fomit-frame-pointer -fpie -I ./compel/include/uapi
> -fno-strict-aliasing -iquote criu/include -iquote include -iquote
> images -iquote criu/arch/arm/include -iquote . -I/usr/include/libnl3
> -I ./compel/include/uapi -I ./compel/include/uapi -DCR_NOGLIBC
> -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=0 -Wp,-U_FORTIFY_SOURCE
> -Wp,-D_FORTIFY_SOURCE=0 -marm criu/pie/restorer.c -o
> criu/pie/restorer.d
> arm-linux-gnueabihf-gcc -M -MT criu/pie/parasite.d -MT
> criu/pie/parasite.o -I /usr/include/google/ -I images -DCONFIG_ARMV7
> -I /tmp/include -I /usr/include/google/ -I images -DCONFIG_ARMV7 -I
> /tmp/include -O2 -g -Wall -Wformat-security -Werror -DCONFIG_ARMV7
> -D_FILE_OFFSET_BITS=64 -D_GNU_SOURCE -iquote include/
> -Wstrict-prototypes -fno-stack-protector -nostdlib
> -fomit-frame-pointer -fpie -I ./compel/include/uapi
> -fno-strict-aliasing -iquote criu/include -iquote include -iquote
> images -iquote criu/arch/arm/include -iquote . -I/usr/include/libnl3
> -I ./compel/include/uapi -I ./compel/include/uapi -DCR_NOGLIBC
> -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=0 -Wp,-U_FORTIFY_SOURCE
> -Wp,-D_FORTIFY_SOURCE=0 -marm criu/pie/parasite.c -o
> criu/pie/parasite.d
> arm-linux-gnueabihf-gcc -c -I /usr/include/google/ -I images
> -DCONFIG_ARMV7 -I /tmp/include -I /usr/include/google/ -I images
> -DCONFIG_ARMV7 -I /tmp/include -O2 -g -Wall -Wformat-security -Werror
> -DCONFIG_ARMV7 -D_FILE_OFFSET_BITS=64 -D_GNU_SOURCE -iquote include/
> -Wstrict-prototypes -fno-stack-protector -nostdlib
> -fomit-frame-pointer -fpie -I ./compel/include/uapi
> -fno-strict-aliasing -iquote criu/include -iquote include -iquote
> images -iquote criu/arch/arm/include -iquote . -I/usr/include/libnl3
> -I ./compel/include/uapi -I ./compel/include/uapi -DCR_NOGLIBC
> -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=0 -Wp,-U_FORTIFY_SOURCE
> -Wp,-D_FORTIFY_SOURCE=0 -marm criu/pie/parasite.c -o
> criu/pie/parasite.o
> arm-linux-gnueabihf-ld -z noexecstack -T
> ./compel/arch/arm/scripts/compel-pack.lds.S -o
> criu/pie/parasite.built-in.o criu/pie/parasite.o criu/pie/pie.lib.a
> ./compel/plugins/std.lib.a
> ./compel/compel-host hgen -f criu/pie/parasite.built-in.o -o
> criu/pie/parasite-blob.h
> criu/pie/parasite-blob.h generated successfully.
> arm-linux-gnueabihf-gcc -c -I /usr/include/google/ -I images
> -DCONFIG_ARMV7 -I /tmp/include -I /usr/include/google/ -I images
> -DCONFIG_ARMV7 -I /tmp/include -O2 -g -Wall -Wformat-security -Werror
> -DCONFIG_ARMV7 -D_FILE_OFFSET_BITS=64 -D_GNU_SOURCE -iquote include/
> -Wstrict-prototypes -fno-stack-protector -nostdlib
> -fomit-frame-pointer -fpie -I ./compel/include/uapi
> -fno-strict-aliasing -iquote criu/include -iquote include -iquote
> images -iquote criu/arch/arm/include -iquote . -I/usr/include/libnl3
> -I ./compel/include/uapi -I ./compel/include/uapi -DCR_NOGLIBC
> -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=0 -Wp,-U_FORTIFY_SOURCE
> -Wp,-D_FORTIFY_SOURCE=0 -marm criu/pie/restorer.c -o
> criu/pie/restorer.o
> arm-linux-gnueabihf-ld -z noexecstack -T
> ./compel/arch/arm/scripts/compel-pack.lds.S -o
> criu/pie/restorer.built-in.o ./criu/arch/arm/restorer.o
> criu/pie/restorer.o criu/pie/pie.lib.a ./compel/plugins/std.lib.a
> ./compel/compel-host hgen -f criu/pie/restorer.built-in.o -o
> criu/pie/restorer-blob.h
> criu/pie/restorer-blob.h generated successfully.
>
> The only hint, that was the solution of the very similar issue as yours
> previously, is that ld line shouldn't have `-r' flag. Otherwise, toolchains
> that I've tried didn't do relocations in resulting PIE. There was also a
> couple of arm reports about the same relocation issue, so removing this
> flag solved issues for me and community people.
>
>> arm-linux-gnueabihf-ld -z noexecstack -T ./compel/arch/arm/scripts/compel-pack.lds.S -o criu/pie/parasite.built-in.o criu/pie/parasite.o criu/pie/pie.lib.a ./compel/plugins/std.lib.a
Another hint: as you do full cross-compilation of rootfs for
your target, check ld flags for building some shared library,
and check the difference between them and linking
parasite.built-in.o
For a test you can take parasite-head.S and add small
stub.c file which provides in C a function named parasite_service()
I've attached files for linking test.
Try to build it with your toolchain like:
arm-linux-gnueabihf-gcc -c stub.c parasite-head.S
arm-linux-gnueabihf-ld -z noexecstack -o built-in.o stub.o
parasite-head.o -T compel-pack.lds.S
In result you should have in built-in.o relocated relative call
to parasite_service():
arm-linux-gnueabihf-objdump -dS built-in.o | head -n20
[...]
18: fa000004 blx 30 <parasite_service>
1c: e7f001f0 .word 0xe7f001f0
[...]
00000030 <parasite_service>:
30: b480 push {r7}
32: b083 sub sp, #12
34: af00 add r7, sp, #0
36: 6078 str r0, [r7, #4]
38: 6039 str r1, [r7, #0]
[...]
If it's not what you have - you'll need to find how it works for
shared libraries in your environment.
--
Dmitry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: compel-pack.lds.S
Type: text/x-asm
Size: 601 bytes
Desc: not available
URL: <http://lists.openvz.org/pipermail/criu/attachments/20170413/ef6ccf37/attachment.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: parasite-head.S
Type: text/x-asm
Size: 747 bytes
Desc: not available
URL: <http://lists.openvz.org/pipermail/criu/attachments/20170413/ef6ccf37/attachment-0001.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: stub.c
Type: text/x-csrc
Size: 54 bytes
Desc: not available
URL: <http://lists.openvz.org/pipermail/criu/attachments/20170413/ef6ccf37/attachment-0002.bin>
More information about the CRIU
mailing list