[CRIU] Fwd: Checkpoint failure on arm64 platform

[Christopher Covington]

Hi Vijay,

I'm glad to see someone else using the AArch64 port. Sorry it has so
many cobwebs.

On 12/22/2015 06:48 AM, Vijay Kilari wrote:
> On Tue, Dec 22, 2015 at 4:13 PM, Pavel Emelyanov <xemul at parallels.com> wrote:
>> On 12/22/2015 09:03 AM, Vijay Kilari wrote:

>>> After this changes + changing PAGE_SIZE to 64KB, there is no error
>>> reporting during checkpoint.
>>
>> Good :) Christopher (in Cc) once started to fix the page-size problem for
>> arm and aarch64.

Our current hack, that we've grown complacent with, is remove the check
in crtools.c that PAGE_SIZE is PAGE_IMAGE_SIZE (4096), and build twice,
once with PAGE_SIZE at the default of 4096, and again with
-DPAGE_SIZE=65536. We rename each criu executable so that we essentially
have "criu_4096" and "criu_65536". We then use a wrapper script named
"criu" to run criu_$(getconf PAGE_SIZE).

Hopefully the foundation has been laid to finally take the 4K build, run
it on 64K (and/or vice versa), see what crashes, and change each
breaking, hard-coded PAGE_SIZE usage to
run-time-detected-on-arches-that-need-it page_size(). When I tried this
previously, support from within the parasite code was most difficult,
but over the summer I finally figured out how to pass through such
information with the very similar task_args->task_size = kdat.task_size
change (7451fc). Lastly, the image files should include the page size as
well. Pavel suggested doing this with cpuinf dump/check images
(cpuinfo.proto). Proper runtime probing might even preemptively add 16K
support, in case anyone is interested in that.

32-bit ARM (AArch32) only has 4K pages, although there was recent work
on runtime probing for TASK_SIZE, if I recall correctly.

> The clone() system is failing because, arm64 requires stack pointer 
> to be aligned with 16 bytes. With below changes in cr-restore.c, 
> clone() is ok I will send patch for this as well. Many places inside
> test/zdtm also requires similar changes.
> 
> struct cr_clone_arg {
>         /*
>          * Reserve some space for clone() to locate arguments
>          * and retcode in this place
>          */
> -        char stack[128] __attribute__((aligned (8)));
> +       char stack[128] __attribute__((aligned (16)));
>         char stack_ptr[0];
>         struct pstree_item *item;
>         unsigned long clone_flags;
>         int fd;
> 
>         CoreEntry *core;
> };

This is correct. Apologies for not having already sent this patch.

> However now restore fails with signal 11 when executing
>  vdso_fill_symtable(). Any idea?
> 
> restore.log:
> ---------------
> pie: vdso: Parsing at 0x3ff81110000 0x3ff81120000
> pie: vdso: PT_LOAD p_vaddr: 0x0
> pie: vdso: DT_HASH: 0x120
> pie: vdso: DT_STRTAB: 0x1f8
> pie: vdso: DT_SYMTAB: 0x150
> pie: vdso: DT_STRSZ: 0x77
> pie: vdso: DT_SYMENT: 0x18
> pie: vdso: nbucket 0x3 nchain 0x7 bucket 0x3ff81110128 chain 0x3ff8111>
> pie: 0134
> (01.062166) Error (cr-restore.c:1266): 24818 killed by signal 11
> (01.103990) Error (cr-restore.c:1266): 24818 killed by signal 9
> (01.294182) Error (cr-restore.c:1999): Restoring FAILED.

Try running the test code attached to the following email:

https://lists.openvz.org/pipermail/criu/2015-March/019161.html

If that crashes, try the following patch:

http://www.spinics.net/lists/linux-arm-msm/msg18291.html

While commit 871da9 seemed to fix the issue for cross-compilation using
Linaro compilers, native building under Arch results in the char
*vdso_symbols requiring relocations. We're currently getting by with a
hacky workaround. Unless some compiler option that we've not yet tried
gets GCC to actually generated position independent code, piegen
probably will need to be ported to AArch64 (and perhaps AArch32 as well)
to resolve the load-time relocations.

Christopher Covington

-- 
Qualcomm Innovation Center, Inc.
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
a Linux Foundation Collaborative Project