[CRIU] [PATCH RFC 8/8] criu: lazy-pages: enable remoting of lazy pages

[Pavel Emelyanov]

On 05/30/2016 06:23 PM, Mike Rapoport wrote:
> On Mon, May 30, 2016 at 06:21:53PM +0300, Pavel Emelyanov wrote:
>> On 05/30/2016 06:16 PM, Mike Rapoport wrote:
>>> On Mon, May 30, 2016 at 05:59:28PM +0300, Pavel Emelyanov wrote:
>>>> On 05/30/2016 03:33 PM, Mike Rapoport wrote:
>>>>> On Mon, May 30, 2016 at 01:58:53PM +0300, Pavel Emelyanov wrote:
>>>>>> On 05/29/2016 09:58 AM, Mike Rapoport wrote:
>>>>>>> On Fri, May 27, 2016 at 10:38:00PM +0300, Pavel Emelyanov wrote:
>>>>>>>> On 05/21/2016 01:49 PM, Mike Rapoport wrote:
>>>>>>>>> The remote lazy pages variant can be run as follows:
>>>>>>>>>
>>>>>>>>> src# criu dump -t <pid> --lazy-pages --port 9876 -D /tmp/1 &
>>>>>>>>
>>>>>>>> This thing starts dump and lazy page server that flushes pages remotely.
>>>>>>>>
>>>>>>>>> src# while ! sudo fuser 9876/tcp ; do sleep 1; done
>>>>>>>>> src# scp -r /tmp/1/ dst:/tmp/
>>>>>>>>>
>>>>>>>>> dst# criu lazy-pages --lazy-addr /tmp/uffd.sock --page-server \
>>>>>>>>>                      --address dst --port 9876 -D /tmp/1 &
>>>>>>>>
>>>>>>>> This will start lazy pages client that would connect to dump side
>>>>>>>> and ... request for pages?
>>>>>>>
>>>>>>> Yep, lazy-pages daemon will connect to the dump side and forward page fault
>>>>>>> requests there.
>>>>>>>  
>>>>>>>>> dst# criu restore --lazy-pages --lazy-addr /tmp/uffd.sock -D /tmp/1
>>>>>>>>
>>>>>>>> One more comment inline :)
>>>>>>>>
>>>>>>>>> Signed-off-by: Mike Rapoport <rppt at linux.vnet.ibm.com>
>>>>>>>>> ---
>>>>>>>>>  criu/cr-dump.c   | 54 ++++++++++++++++++++++++++++++++++++++++++++++++++++--
>>>>>>>>>  criu/page-read.c |  2 +-
>>>>>>>>>  criu/uffd.c      |  9 ++++++++-
>>>>>>>>>  3 files changed, 61 insertions(+), 4 deletions(-)
>>>>>>>>>
>>>>>>>>> diff --git a/criu/cr-dump.c b/criu/cr-dump.c
>>>>>>>>> index 1a551b4..faca73b 100644
>>>>>>>>> --- a/criu/cr-dump.c
>>>>>>>>> +++ b/criu/cr-dump.c
>>>>>>>>> @@ -1298,7 +1298,7 @@ static int dump_one_task(struct pstree_item *item)
>>>>>>>>>  		}
>>>>>>>>>  	}
>>>>>>>>>  
>>>>>>>>> -	ret = parasite_dump_pages_seized(parasite_ctl, &vmas, false);
>>>>>>>>> +	ret = parasite_dump_pages_seized(parasite_ctl, &vmas, opts.lazy_pages);
>>>>>>>>>  	if (ret)
>>>>>>>>>  		goto err_cure;
>>>>>>>>>  
>>>>>>>>> @@ -1338,7 +1338,10 @@ static int dump_one_task(struct pstree_item *item)
>>>>>>>>>  		goto err;
>>>>>>>>>  	}
>>>>>>>>>  
>>>>>>>>> -	ret = parasite_cure_seized(parasite_ctl);
>>>>>>>>> +	if (opts.lazy_pages)
>>>>>>>>> +		ret = parasite_cure_remote(parasite_ctl);
>>>>>>>>> +	else
>>>>>>>>> +		ret = parasite_cure_seized(parasite_ctl);
>>>>>>>>>  	if (ret) {
>>>>>>>>>  		pr_err("Can't cure (pid: %d) from parasite\n", pid);
>>>>>>>>>  		goto err;
>>>>>>>>> @@ -1525,6 +1528,49 @@ err:
>>>>>>>>>  	return cr_pre_dump_finish(ret);
>>>>>>>>>  }
>>>>>>>>>  
>>>>>>>>> +static int cr_lazy_mem_dump(void)
>>>>>>>>> +{
>>>>>>>>> +	struct pstree_item *item;
>>>>>>>>> +	int ret = 0;
>>>>>>>>> +
>>>>>>>>> +	pr_info("Lazy pages: pre-dumping memory\n");
>>>>>>>>> +	for_each_pstree_item(item) {
>>>>>>>>> +		struct parasite_ctl *ctl = item->parasite_ctl;
>>>>>>>>> +		struct page_xfer xfer;
>>>>>>>>> +
>>>>>>>>> +		timing_start(TIME_MEMWRITE);
>>>>>>>>> +		ret = open_page_xfer(&xfer, CR_FD_PAGEMAP, ctl->pid.virt);
>>>>>>>>> +		if (ret < 0)
>>>>>>>>> +			goto err;
>>>>>>>>> +
>>>>>>>>> +		ret = page_xfer_dump_pages(&xfer, ctl->mem_pp, 0, false);
>>>>>>>>
>>>>>>>> If I got the set right :) 
>>>>>>>
>>>>>>> Hmm, not quite ;-)
>>>>>>>
>>>>>>>> this place just flushes all the lazy memory into
>>>>>>>> socket ignoring the requests from restore side and that's it. No?
>>>>>>>
>>>>>>> This place flushes all the memory *except* lazy pages into the images.
>>>>>>> The buffers that are marked with PPB_LAZY are not  written.
>>>>>>
>>>>>> Ah! Indeed. The last 'false', yes. So here we flush all the memory __but__ the
>>>>>> lazy one. OK, the question is -- why not write one using existing --page-server
>>>>>> dump?
>>>>>
>>>>> Haven't got this. Write what? The PPB_LAZY buffer?
>>>>
>>>> No. You say that at this point you send into the network all the pages that
>>>> are not lazy. My question is -- why we pull pages till this point, instead
>>>> of sending them into socket immediately, as it's done during regular dump?
>>>
>>> Well, I've started from "delayed" dump and then added remoting of the
>>> pages, so that's what I've got :)
>>> Anyway, there should be no problem to dump non-lazy pages immediately like
>>> in regular dump, it just makes __parasite_dump_pages_seized too ugly
>>> because of the three cases it needs to handle :)
>>
>> Yes :( But this is somehow unavoidable. The problem with pulling page-pipes
>> out of parasite_dump_pages_seized is that 
>>
>> a) all the memory in pipes is unswappable
>> b) there's a limit on the number of pipes per process and on the pipe size
>>    itself. This limits the amount of mem we can carry.
>>
>> so the less page-pipes we have in criu the better :)
>  
> Then maybe we should consider keeping parasite in the dumpee and making it
> handle the remote #PFs instead of collecting dumpee memory to pipes?

Yup, that would be much better. But that's in the long term, I suppose :)

-- Pavel