[CRIU] criu restore performance

[Pavel Emelyanov]

On 06/10/2014 09:55 PM, J F wrote:
> Hello,
> 
> I'm having fun with crtools, and I'm interested in exploring techniques to speed up the time it takes to restore a process (other than getting a better machine).
> 
> Some questions:
> 
> 1) Is 'criu restore' operation time complexity mostly CPU bound, 
>    IO bound, or memory bound?

It's mostly CPU-bound, but for images with large amount of process memory
it can become mem-bound.

> 2) Are any of the operations in 'criu restore" parallelizable or are 
>    they intrinsically sequential and require synchronization due to 
>    kernel interactions?

Currently restore procedure looks like this (top level):

1. fork all tasks
2. make each task restore itself
3. resume tasks execution

Between each step there are global synchronization points.
Other than this in stage 2 tasks may sometimes wait for each other
to restore shared resources, e.g. opened files.

> 3) Is performance of restore a function of the size of the images folder? 

Well, yes. The more data we have to restore the more time it takes. But
the dependency is not researched, but it's non-linear for sure.

> 4) Any tricks/advice/hacks to speed up restore?

It's a WIP at the moment. We do know some things that slow restore (and
dump), but the list is not complete and is not fully fixed yet. E.g.

1. More image files we have the slower it works. Currently criu generates
   8 files per-task, we try to make less of them.

2. Criu writes data into images with small portions. This behaves badly 
   due to many actions taken by kernel on every write() call especially
   for disk FS-s (even for page-cache writes).

3. /proc interface we use heavily on dump is too damn slow.

4. Shared file descriptors can be inherited by tasks on restore. Instead
   we share them via unix sockets which is slower.

5. Potentially COW-ed pages in memory mapping are memcmp-ed on restore to
   decide whether or not to COW the page. No good ideas how to deal with it.

Thanks,
Pavel