[CRIU] Soft dirty pages in expanded VMA

[Diyu Zhou]

Hey Mike,

Sorry for the late reply. It took me a while to understand what's really
going
on.  I have written a simple program to illustrate my point.

The program first creates a large heap and then block.

//40MB heap
#define TOT_HEAP_SIZE (40 * 1024 * 1024)

char * start = sbrk(0);
sbrk(TOT_HEAP_SIZE);
memset(start, 0, TOT_HEAP_SIZE);
getchar(); //CRIU dump here!

I invoke CRIU to checkpoint the process when it is blocked at getchar().

Then the program increase the heap size and then block  again:

sbrk(4096);
getchar(); //CRIU dump again here.


I invoke CRIU to checkpoint the process again when the process is blocked at
getchar().  What I observe is that the second dump from CRIU will copy all
the
pages in the heap (40MB).  I have attached the testing program and the
scripts
to invoke CRIU in case someone wants to play with it.

This seems to me can be further optimized.  I found that the cause of this
behaviour is that the kernel code will mark a VMA as soft-dirty when the
VMA is
expanded. As a result, all the pages within the VMA will be marked as
soft-dirty when reading from /proc/pid/pagemap. Is there a reason for that?

Thanks in advance for your help!

Best,
Diyu

On Wed, May 15, 2019 at 4:27 AM Mike Rapoport <rppt at linux.ibm.com> wrote:

> Hi,
>
> On Tue, May 14, 2019 at 12:10:26PM -0700, Diyu Zhou wrote:
> > Hey Mike,
> >
> > Thanks for your reply. I understand what you wrote. However, what I
> observe is
> > that some of the "clean" pages are copied by the CRIU.
> >
> > For example, let's say CRIU performs iterative checkpoints (let's say
> using
> > predump) on a dumpee process. After one checkpoint is completed, the
> dumpee
> > process calls sbrk to increase the size of the heap. What I observe is
> that
> > even the dumpee program didn't touch any pages in the heap, all the
> present
> > pages in the heap will be marked as dirty and copied in the next
> checkpoint
> > operation. Is there an explanation for such a behavior?
>
> Are you running with '--track-mem'?
>
> If I understand correctly, the size of pages*img increases for each
> iteration?
>
> Can you check what's the contents of pagemap*img with
> 'crit show /path/to/pagemap-<pid>.img'
> and see if the second and the next iterations contain areas marked as
> 'PE_PRESENT'?
>
> > Thanks in advance for your help.
> >
> > Best,
> > Diyu
>
> --
> Sincerely yours,
> Mike.
>
> > On Tue, May 14, 2019 at 9:04 AM Mike Rapoport <rppt at linux.ibm.com>
> wrote:
> >
> >     On Mon, May 13, 2019 at 04:10:54PM -0700, Diyu Zhou wrote:
> >     > Hey,
> >     >
> >     > Firstly, thanks for the great tool! It is very very helpful. I can
> see
> >     the
> >     > development of such a tool
> >     > is very challenging and you guys do several clever tricks to make
> it
> >     possible.
> >     > I enjoy a lot using
> >     > the tool as well as reading the source code and have learned a lot
> from
> >     it.
> >     >
> >     > I have a question regarding the implementation of the soft dirty
> pages.
> >     From
> >     > what I can tell, if
> >     > a VMA is expanded, all the pages within it will be marked as soft
> dirty.
> >     This
> >     > seems to me will
> >     > result in unnecessarily transferring clean pages when iterative
> dumps are
> >     > performed.
> >     > Is there any reason not just mark the pages in the expanded part
> of the
> >     VMA
> >     > soft dirty?
> >
> >     When a VMA is expanded is is marked soft-dirty to ensure it would be
> >     scanned for the dirty pages.
> >     Each page has its own marking that says whether the page is dirty or
> not.
> >     The per-page information is available at /proc/PID/pagemap and CRIU
> uses it
> >     to actually decide what page should be dumped.
> >
> >     > Thanks again for the wonderful tool and your help!
> >     >
> >     > Best,
> >     > Diyu
> >     >
> >     >
> >     >
> >
> >     > _______________________________________________
> >     > CRIU mailing list
> >     > CRIU at openvz.org
> >     > https://lists.openvz.org/mailman/listinfo/criu
> >
> >
> >     --
> >     Sincerely yours,
> >     Mike.
> >
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openvz.org/pipermail/criu/attachments/20190528/80cf4ae0/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: test.c
Type: text/x-csrc
Size: 499 bytes
Desc: not available
URL: <http://lists.openvz.org/pipermail/criu/attachments/20190528/80cf4ae0/attachment-0002.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: test.sh
Type: application/x-shellscript
Size: 456 bytes
Desc: not available
URL: <http://lists.openvz.org/pipermail/criu/attachments/20190528/80cf4ae0/attachment-0003.bin>