[CRIU] [PATCHv7 00/33] kernel: Introduce Time Namespace

[Andrei Vagin]

On Thu, Oct 17, 2019 at 11:24:45AM +0200, Thomas Gleixner wrote:
> On Fri, 11 Oct 2019, Dmitry Safonov wrote:
> > We wrote two small benchmarks. The first one gettime_perf.c calls
> > clock_gettime() in a loop for 3 seconds. It shows us performance with
> > a hot CPU cache (more clock_gettime() cycles - the better):
> > 
> >         | before    | CONFIG_TIME_NS=n | host      | inside timens
> > --------------------------------------------------------------
> >         | 153242367 | 153567617        | 150933203 | 139310914
> >         | 153324800 | 153115132        | 150919828 | 139299761
> >         | 153125401 | 153686868        | 150930471 | 139273917
> >         | 153399355 | 153694866        | 151083410 | 139286081
> >         | 153489417 | 153739716        | 150997262 | 139146403
> >         | 153494270 | 153724332        | 150035651 | 138835612
> > -----------------------------------------------------------
> > avg     | 153345935 | 153588088        | 150816637 | 139192114
> > diff %  | 100       | 100.1            | 98.3      | 90.7
> 
> 
> That host 98.3% number is weird and does not match the tests I did with the
> fallback code I provided you. On my limited testing that fallback hidden in
> the slowpath did not show any difference to the TIME_NS=n case when not
> inside a time namespace.

You did your experiments without a small optimization that we introduced
in the 18-th patch:

[PATCHv7 18/33] lib/vdso: Add unlikely() hint into vdso_read_begin()

When I did my measurements in the first time, I found that with this
timens change clock_gettime() shows a better performance when
CONFIG_TIME_NS isn't set. This looked weird for me, because I don't
expect to see this improvement. After analyzing a disassembled code of
vdso.so, I found that we can add the unlikely() hint into
vdso_read_begin() and this gives us 2% improvement of clock_gettime
performance on the upsteam kernel.

In my table, the "before" column is actually for the upstream kernel
with the 18-th patch. Here is the table with the real "before" column:

        | before    | with 18/33 | CONFIG_TIME_NS=n | host      | inside timens
------------------------------------------------------------------------------
avg     | 150331408 | 153345935  | 153588088        | 150816637 | 139192114
------------------------------------------------------------------------------
diff %  |       98  |      100   | 100.1            | 98.3      | 90.7
------------------------------------------------------------------------------
stdev % |       0.3 |     0.09   | 0.15             | 0.25      | 0.13

If we compare numbers in "before", "host" and "inside timens" columns, we
see the same results that you had. clock_gettime() works with the
same performance in the host namespace and 7% slower in a time
namespace.

Now let's look why we have these 2% degradation in the host time
namespace. For that, we cat look at disassembled code of do_hres:

Before:
   0:   55                      push   %rbp
   1:   48 63 f6                movslq %esi,%rsi
   4:   49 89 d1                mov    %rdx,%r9
   7:   49 89 c8                mov    %rcx,%r8
   a:   48 c1 e6 04             shl    $0x4,%rsi
   e:   48 01 fe                add    %rdi,%rsi
  11:   48 89 e5                mov    %rsp,%rbp
  14:   41 54                   push   %r12
  16:   53                      push   %rbx
  17:   44 8b 17                mov    (%rdi),%r10d
  1a:   41 f6 c2 01             test   $0x1,%r10b
  1e:   0f 85 fb 00 00 00       jne    11f <do_hres.isra.0+0x11f>
  24:   8b 47 04                mov    0x4(%rdi),%eax
  27:   83 f8 01                cmp    $0x1,%eax
  2a:   74 0f                   je     3b <do_hres.isra.0+0x3b>
  2c:   83 f8 02                cmp    $0x2,%eax
  2f:   74 72                   je     a3 <do_hres.isra.0+0xa3>
  31:   5b                      pop    %rbx
  32:   b8 ff ff ff ff          mov    $0xffffffff,%eax
  37:   41 5c                   pop    %r12
  39:   5d                      pop    %rbp
  3a:   c3                      retq
...

After:
   0:   55                      push   %rbp
   1:   4c 63 ce                movslq %esi,%r9
   4:   49 89 d0                mov    %rdx,%r8
   7:   49 c1 e1 04             shl    $0x4,%r9
   b:   49 01 f9                add    %rdi,%r9
   e:   48 89 e5                mov    %rsp,%rbp
  11:   41 56                   push   %r14
  13:   41 55                   push   %r13
  15:   41 54                   push   %r12
  17:   53                      push   %rbx
  18:   44 8b 17                mov    (%rdi),%r10d
  1b:   44 89 d0                mov    %r10d,%eax
  1e:   f7 d0                   not    %eax
  20:   83 e0 01                and    $0x1,%eax
  23:   89 c3                   mov    %eax,%ebx
  25:   0f 84 03 01 00 00       je     12e <do_hres+0x12e>
  2b:   8b 47 04                mov    0x4(%rdi),%eax
  2e:   83 f8 01                cmp    $0x1,%eax
  31:   74 13                   je     46 <do_hres+0x46>
  33:   83 f8 02                cmp    $0x2,%eax
  36:   74 7b                   je     b3 <do_hres+0xb3>
  38:   b8 ff ff ff ff          mov    $0xffffffff,%eax
  3d:   5b                      pop    %rbx
  3e:   41 5c                   pop    %r12
  40:   41 5d                   pop    %r13
  42:   41 5e                   pop    %r14
  44:   5d                      pop    %rbp
  45:   c3                      retq
...

So I think we see these 2% degradation in the host time namespace,
because we need to save to extra registers on stack. If we want to avoid
this degradation, we can mark do_hres_timens as noinline. In this case,
the disassembled code will be the same as before these changes:

  0000000000000160 <do_hres>:
  do_hres():
   160:   55                      push   %rbp
   161:   4c 63 ce                movslq %esi,%r9
   164:   49 89 d0                mov    %rdx,%r8
   167:   49 c1 e1 04             shl    $0x4,%r9
   16b:   49 01 f9                add    %rdi,%r9
   16e:   48 89 e5                mov    %rsp,%rbp
   171:   41 54                   push   %r12
   173:   53                      push   %rbx
   174:   44 8b 17                mov    (%rdi),%r10d
   177:   41 f6 c2 01             test   $0x1,%r10b
   17b:   0f 85 fc 00 00 00       jne    27d <do_hres+0x11d>
   181:   8b 47 04                mov    0x4(%rdi),%eax
   184:   83 f8 01                cmp    $0x1,%eax
   187:   74 0f                   je     198 <do_hres+0x38>
   189:   83 f8 02                cmp    $0x2,%eax
   18c:   74 73                   je     201 <do_hres+0xa1>
   18e:   5b                      pop    %rbx
   18f:   b8 ff ff ff ff          mov    $0xffffffff,%eax
   194:   41 5c                   pop    %r12
   196:   5d                      pop    %rbp
   197:   c3                      retq
...

But this change will affect the performance of clock_gettime in a time
namespace.

My experiments shows that with the noinline annotation for
do_hres_timens, clock_gettime will work with the same performance in the
host time namespace, but it will be slower on 11% in a time namespace.

Thomas, what do you think about this? Do we need to mark do_hres_timens
as noinline?

Thanks,
Andrei