[CRIU] [PATCH 0/9] vDSO rework, part 2/3

Dmitry Safonov dsafonov at virtuozzo.com
Thu Jun 22 14:04:23 MSK 2017


Hi guys,

This is the second part of the set, it consists
mostly in refactoring - preparation for the last third part.
Also I've included the 9-th patch-fix to reduce nr. of patches
in the last part of the set, as the fix is one-liner and
doesn't collide with fixes in the 3 part.

No functional change in this part is expected,
with the exception for ia32 fix.

Attaching original set's cover:

=== Full description of the set ===

After facing a bug with mismatched pfn for vdso in vz7 CTs
and proposing a way to solve it and make vdso C/R faster:
https://lists.openvz.org/pipermail/criu/2017-June/037982.html

I've started working on it. Then I've moved vdso symtable to
criu.kdat file (as I previously have proposed to Pasha)...
And found that vDSO code is not very readable and another
fast-fix on the top may roll a yacht over 

During the refactoring I've also meet some bugs, which were fixed:
1. Leaving vdso/vvar after restore in task which didn't have them.
2. Bug with unmapping original vvar vma on the second C/R if
   jump trampolines & rt-vdso is used.
3. Keeping rt-vvar after each C/R with inserted jump trampolines.
4. Bug with ia32 unmapping vdso after trampolines insertion.

For (2), (3), I did introduce rt-vdso mark v3.

There are two new tests:
o Task with unmapped vdso (vdso02)
o Iterative C/R with inserting jump-trampolines (vdso-proxy)

Then, the whole process of vdso's C/R has being reworked with
criu.kdat file keeping in mind:

=== Process of vDSO C/R ===

On *Dump*:
        Before                                  After
        ------                                  -----
Checking vdso's pfn or filling          No need to do this on post v3.16
symtable to find if VMa is vdso         kernels, as "[vdso]" mark stays
or is mishinted by /proc/../maps        in maps file after mremap().
        ------                                  -----
Parsing of self-maps to find
vdso/vvar position for remapping        No need to do this on dump.
into restorer's parking zone.
        ------                                  -----
Parsing vdso's symtable to find         No need to do this on dump.
if image's vdso matches host's vdso.


On *Restore*:
        Before                                  After
        ------                                  -----
Parsing of self-maps to find            Mapping vdso/vvar with
vdso/vvar position for remapping        arch_prctl(MAP_VDSO_*)
into restorer's parking zone.           to save sys_mremap() syscalls.
        ------                                  -----
Parsing vdso's symtable to find         Keeping vdso symtable in criu.kdat.
if image's vdso matches host's vdso.
        ------                                  -----
Checking vdso's pfn or filling
symtable to find if VMa is vdso         No need to do this on restore.
or is mishinted by /proc/../maps

=== Result: some numbers ===

I've tested the performance impact of the patches set on:
4.12.0-rc5+ kernel (needed arch_prctl() is available from v4.9 kernel)
With the current criu-dev af6399cc5 ("vma: Fix badly inherited FD
in filemap_open")
And my `wip/new-vdso' branch on github.
Done on Qemu with 2Gb of memory and 4 CPUs on fedora-26.
Only values those differ are presented.

Mean test results on 30 iterations on busyloop00 (single-process impact)
and session00 (~10 processes impact).
Deviation is from session00 (before) test, as it's the largest there.

*Dump*                         | === busyloop00 === | === session00 === max dev
                               | before:     after: | before:    after:
syscalls:sys_enter_splice      |      13         12 |      96        96
syscalls:sys_enter_fcntl       |      30         29 |     200       199
syscalls:sys_enter_unlinkat    |      25         26 |      31        31
syscalls:sys_enter_pipe        |       6          5 |      45        44
syscalls:sys_enter_newfstatat  |     157        159 |     334       334
syscalls:sys_enter_read        |     100         96 |     378       374
syscalls:sys_enter_write       |      17         11 |      57        56
syscalls:sys_enter_pread64     |      16         17 |     137        73
syscalls:sys_enter_writev      |       2          0 |       4         4
syscalls:sys_enter_sendfile64  |       3          0 |       0         0
syscalls:sys_enter_open        |      85         83 |      83        83
syscalls:sys_enter_openat      |      68         62 |     436       426
syscalls:sys_enter_close       |     172        160 |     655       641
syscalls:sys_enter_brk         |      10          4 |      14        14
syscalls:sys_enter_munmap      |       5          2 |      12         9
syscalls:sys_enter_kcmp        |       5          5 |     111       105 +-0.45%
syscalls:sys_enter_getpid      |      25         24 |      88        87
syscalls:sys_enter_kill        |       2          0 |       2         0
syscalls:sys_enter_exit        |       1          0 |       1         0
syscalls:sys_enter_wait4       |      19         17 |     138       136
syscalls:sys_enter_mmap        |      26         25 |      33        32
syscalls:sys_enter_arch_prctl  |       2          1 |       2         1
seconds time elapsed           |0.085441   0.082569 |0.107197  0.103572 +-2.98%

*Restore*                      | === busyloop00 === | === session00 === max dev
                               | before:     after: | before:    after:
syscalls:sys_enter_dup2        |     245        241 |    1748      1615 +-6.45%
syscalls:sys_enter_fcntl       |     627        617 |    4450      4119 +-6.34%
syscalls:sys_enter_pipe        |       1          0 |       5         4
syscalls:sys_enter_read        |      50         43 |     140       133
syscalls:sys_enter_write       |      16         15 |      33        32
syscalls:sys_enter_pread64     |       1          0 |     176       168
syscalls:sys_enter_open        |     165        163 |     915       849 +-6.17%
syscalls:sys_enter_openat      |      82         78 |     208       204
syscalls:sys_enter_close       |     355        343 |    2074      1933 +-5.44%
syscalls:sys_enter_mremap      |       8          6 |     319       303
syscalls:sys_enter_munmap      |      14         11 |      70        67
syscalls:sys_enter_futex       |      32         32 |     472       426 +-7.29%
syscalls:sys_enter_getpid      |      36         33 |     133       130
syscalls:sys_enter_rt_sigproc..|     266        262 |    1786      1654 +-6.32%
syscalls:sys_enter_kill        |     122        118 |     878       810 +-6.43%
syscalls:sys_enter_exit        |       1          0 |       1         0
syscalls:sys_enter_wait4       |      22         20 |     392       350 +-7.47%
syscalls:sys_enter_mmap        |      68         67 |      96        95
syscalls:sys_enter_arch_prctl  |       6          6 |      20        27
seconds time elapsed           |0.041698  0.0328033 |0.106096  0.097462 +-1.83%

Looking at the deviation, dumping takes about the same time, heh,
but less number of syscalls anyway.

In time-values:
Around 21% faster restoring on single busyloop!
And 8% faster restore for ~10 processes.

Cc: Cyrill Gorcunov <gorcunov at openvz.org>

Dmitry Safonov (9):
  vdso: Introduce type for checking vdso hint in maps file
  util: Add fd parameter to vaddr_to_pfn()
  vdso: Use vaddr_to_pfn() to get vdso's pfn
  vdso: Separate function for checking vdso's pfn
  vdso: Separate checks for {vdso,vvar}-alikeness
  vdso: Separate remap_rt_vdso() from vdso_proxify()
  vdso: Cleanup vdso_do_park()
  vdso: Separate add_vdso_proxy()
  vdso/ia32: Don't unmap rt-vdso

 criu/arch/x86/restorer_unmap.S |   1 +
 criu/include/util.h            |   2 +-
 criu/kerndat.c                 |   2 +-
 criu/pie/parasite-vdso.c       | 196 ++++++++++++++++++++++-------------------
 criu/pie/restorer.c            |   2 +-
 criu/util.c                    |  23 +++--
 criu/vdso.c                    | 149 +++++++++++++++++++------------
 7 files changed, 222 insertions(+), 153 deletions(-)

-- 
2.13.1



More information about the CRIU mailing list