<div dir="ltr">Hey Mike, <br><br>Sorry for the late reply. It took me a while to understand what's really going<br>on. I have written a simple program to illustrate my point. <br><br>The program first creates a large heap and then block.<br><br>//40MB heap<br>#define TOT_HEAP_SIZE (40 * 1024 * 1024)<br><br>char * start = sbrk(0);<br>sbrk(TOT_HEAP_SIZE);<br>memset(start, 0, TOT_HEAP_SIZE);<br>getchar(); //CRIU dump here!<br><br>I invoke CRIU to checkpoint the process when it is blocked at getchar(). <br><br>Then the program increase the heap size and then block again:<br><br>sbrk(4096);<br>getchar(); //CRIU dump again here. <br><br><br>I invoke CRIU to checkpoint the process again when the process is blocked at<br>getchar(). What I observe is that the second dump from CRIU will copy all the<br>pages in the heap (40MB). I have attached the testing program and the scripts<br>to invoke CRIU in case someone wants to play with it. <br><br>This seems to me can be further optimized. I found that the cause of this<br>behaviour is that the kernel code will mark a VMA as soft-dirty when the VMA is<br>expanded. As a result, all the pages within the VMA will be marked as<br>soft-dirty when reading from /proc/pid/pagemap. Is there a reason for that? <br><br>Thanks in advance for your help!<br><br>Best,<br>Diyu<br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, May 15, 2019 at 4:27 AM Mike Rapoport <<a href="mailto:rppt@linux.ibm.com">rppt@linux.ibm.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi,<br>
<br>
On Tue, May 14, 2019 at 12:10:26PM -0700, Diyu Zhou wrote:<br>
> Hey Mike,<br>
> <br>
> Thanks for your reply. I understand what you wrote. However, what I observe is<br>
> that some of the "clean" pages are copied by the CRIU.<br>
> <br>
> For example, let's say CRIU performs iterative checkpoints (let's say using<br>
> predump) on a dumpee process. After one checkpoint is completed, the dumpee<br>
> process calls sbrk to increase the size of the heap. What I observe is that<br>
> even the dumpee program didn't touch any pages in the heap, all the present<br>
> pages in the heap will be marked as dirty and copied in the next checkpoint<br>
> operation. Is there an explanation for such a behavior? <br>
<br>
Are you running with '--track-mem'?<br>
<br>
If I understand correctly, the size of pages*img increases for each<br>
iteration?<br>
<br>
Can you check what's the contents of pagemap*img with<br>
'crit show /path/to/pagemap-<pid>.img'<br>
and see if the second and the next iterations contain areas marked as<br>
'PE_PRESENT'?<br>
<br>
> Thanks in advance for your help.<br>
> <br>
> Best,<br>
> Diyu<br>
<br>
--<br>
Sincerely yours,<br>
Mike.<br>
<br>
> On Tue, May 14, 2019 at 9:04 AM Mike Rapoport <<a href="mailto:rppt@linux.ibm.com" target="_blank">rppt@linux.ibm.com</a>> wrote:<br>
> <br>
> On Mon, May 13, 2019 at 04:10:54PM -0700, Diyu Zhou wrote:<br>
> > Hey, <br>
> ><br>
> > Firstly, thanks for the great tool! It is very very helpful. I can see<br>
> the<br>
> > development of such a tool <br>
> > is very challenging and you guys do several clever tricks to make it<br>
> possible.<br>
> > I enjoy a lot using <br>
> > the tool as well as reading the source code and have learned a lot from<br>
> it. <br>
> ><br>
> > I have a question regarding the implementation of the soft dirty pages.<br>
> From<br>
> > what I can tell, if <br>
> > a VMA is expanded, all the pages within it will be marked as soft dirty.<br>
> This<br>
> > seems to me will <br>
> > result in unnecessarily transferring clean pages when iterative dumps are<br>
> > performed. <br>
> > Is there any reason not just mark the pages in the expanded part of the<br>
> VMA<br>
> > soft dirty? <br>
> <br>
> When a VMA is expanded is is marked soft-dirty to ensure it would be<br>
> scanned for the dirty pages.<br>
> Each page has its own marking that says whether the page is dirty or not.<br>
> The per-page information is available at /proc/PID/pagemap and CRIU uses it<br>
> to actually decide what page should be dumped.<br>
> <br>
> > Thanks again for the wonderful tool and your help! <br>
> ><br>
> > Best,<br>
> > Diyu<br>
> ><br>
> ><br>
> > <br>
> <br>
> > _______________________________________________<br>
> > CRIU mailing list<br>
> > <a href="mailto:CRIU@openvz.org" target="_blank">CRIU@openvz.org</a><br>
> > <a href="https://lists.openvz.org/mailman/listinfo/criu" rel="noreferrer" target="_blank">https://lists.openvz.org/mailman/listinfo/criu</a><br>
> <br>
> <br>
> --<br>
> Sincerely yours,<br>
> Mike.<br>
> <br>
> <br>
<br>
</blockquote></div>