[CRIU] [PATCH 1/4] pie/x86_64: syscall clobbers rcx and r11

[Tycho Andersen]

Hi Cyrill,

On Tue, Jun 16, 2015 at 10:10:53PM +0300, Cyrill Gorcunov wrote:
> On Tue, Jun 16, 2015 at 07:34:55AM -0600, Tycho Andersen wrote:
> > On Tue, Jun 16, 2015 at 04:31:50PM +0300, Cyrill Gorcunov wrote:
> > > On Tue, Jun 16, 2015 at 07:20:56AM -0600, Tycho Andersen wrote:
> > > > 
> > > > Ok, I just added rcx because the above link said it could be clobbered
> > > > too, but I can send a patch to remove it if you want.
> > > 
> > > no need to, i'm fine with as is.
> > 
> > Ok. I think the reason we've never seen a seg fault here is that the
> > code path hasn't changed in a while (and neither has gcc's register
> > allocator apparently, yay reaload.c). When I changed it for this
> > patchset, gcc decided to use r11 for something (viz. the dissasembly
> > in the commit message), which is what caused the fault.
>
> I still wonder why it's happened ;) Look, the kernel saves all the data
> it going to modify inside syscall
> 
> ENTRY(system_call)
> 	...
> 	SAVE_ARGS 8,0
> 
> the SAVE_ARGS macro expands to (see thunk_64.S)
> 
> 	...
>      3e6:       48 83 ec 50             sub    $0x50,%rsp
>      3ea:       48 89 7c 24 40          mov    %rdi,0x40(%rsp)
>      3ef:       48 89 74 24 38          mov    %rsi,0x38(%rsp)
>      3f4:       48 89 54 24 30          mov    %rdx,0x30(%rsp)
>      3f9:       48 89 44 24 20          mov    %rax,0x20(%rsp)
>      3fe:       4c 89 44 24 18          mov    %r8,0x18(%rsp)
>      403:       4c 89 4c 24 10          mov    %r9,0x10(%rsp)
>      408:       4c 89 54 24 08          mov    %r10,0x8(%rsp)
>      40d:       4c 89 1c 24             mov    %r11,(%rsp)
>      411:       48 89 44 24 48          mov    %rax,0x48(%rsp)
>      416:       48 89 4c 24 50          mov    %rcx,0x50(%rsp)
>      41b:       f7 84 24 88 e0 ff ff    testl  $0x100801d1,-0x1f78(%rsp)
>      422:       d1 01 08 10 
>      426:       0f 85 d6 00 00 00       jne    502 <tracesys>
> ...
> 
> then when it's time to return from syscall we move them back
> 
> 	RESTORE_ARGS 1,-ARG_SKIP,0
> 
> which expands to
> 
>      467:       48 8b 4c 24 50          mov    0x50(%rsp),%rcx
>      46c:       4c 8b 1c 24             mov    (%rsp),%r11
>      470:       4c 8b 54 24 08          mov    0x8(%rsp),%r10
>      475:       4c 8b 4c 24 10          mov    0x10(%rsp),%r9
>      47a:       4c 8b 44 24 18          mov    0x18(%rsp),%r8
>      47f:       48 8b 44 24 20          mov    0x20(%rsp),%rax
>      484:       48 8b 54 24 30          mov    0x30(%rsp),%rdx
>      489:       48 8b 74 24 38          mov    0x38(%rsp),%rsi
>      48e:       48 8b 7c 24 40          mov    0x40(%rsp),%rdi
>      493:       65 48 8b 24 25 00 00    mov    %gs:0x0,%rsp
>      49a:       00 00 
>      49c:       0f 01 f8                swapgs 
>      49f:       48 0f 07                sysretq 
> 
> Until I miss something obvious (which is very possible), we should not have r11 corruption here.
> Could you please send me pie/restorer.built-in.o which was failing?
> Actually better both one which failed and one with your patch applied.

Sure, I think you can revert the patch in master and build it yourself
and you should see it with the futex test (among others):

criu:~/criu/test master sudo ./zdtm.sh -r static/futex
Execute static/futex
./futex --pidfile=futex.pid --outfile=futex.out
Dump 29325
Restore
Test: zdtm/live/static/futex, Result: FAIL
==================================== ERROR ====================================
Test: zdtm/live/static/futex, Namespace: 
Dump log   : /home/ubuntu/criu/test/dump/static/futex/29325/1/dump.log
--------------------------------- grep Error ---------------------------------
------------------------------------- END -------------------------------------
Restore log: /home/ubuntu/criu/test/dump/static/futex/29325/1/restore.log
--------------------------------- grep Error ---------------------------------
(00.110702) Error (cr-restore.c:1225): 29325 killed by signal 11
(00.110774) Error (cr-restore.c:1905): Restoring FAILED.
------------------------------------- END -------------------------------------
================================= ERROR OVER =================================

but I've uploaded it to http://files.tycho.ws/tmp/restorer.built-in.o and
http://files.tycho.ws/tmp/restorer.built-in.o.good just in case.

Tycho