[CRIU] Remote lazy-restore design discussion

[Pavel Emelyanov]

On 04/03/2016 06:13 PM, Mike Rapoport wrote:
> On Thu, Mar 31, 2016 at 04:25:42PM +0200, Adrian Reber wrote:
>> Hello Pavel,
>>
>> after Mike asked if there have been any design discussions and after I
>> am not 100% sure how the page-server fits into the remote restore, it
>> seems to be a good idea to get a common understanding what the right
>> implementation for remote lazy-restore should look like.
>>
>> I am using my implementation as a starting point for the discussion.
>>
>> I think we need three different process for remote lazy restore
>> independent of how they are started. 'destination system' is the system
>> the process should be migrated to and 'source system' is the system the
>> original process was running on before the migration.
>>
>>  1. The actual restore process (destination system):
>>      This is a 'normal' restore with the difference that memory pages
>>      (MAP_ANONYMOUS and MAP_PRIVATE) are not copied to their place
>>      but they are marked as being handled by userfaultfd. Therefore
>>      a userfaultfd FD (UFFD) is opened and passed to a second process.
>>
>>  2. The local lazy restore UFFD handler (destination system):
>>      This process listens on the UFFD for userfault requests and tries to
>>      handle the userfault requests. Either by reading the required pages
>>      from a local checkpoint (rather unlikely use case) or it is requesting
>>      the pages from a remote system (source system) via the network.
>>
>>  3. The remote lazy restore page request handler (source system):
>>      This process opens a network port and listens for page requests
>>      and reads the requested pages from a local checkpoint (or even
>>      better, directly from a stopped process).
>>
>>
>> As this describes the solution I have implemented it all sounds correct
>> to me. In addition to handling request for pages (processes 2. and 3.)
>> both page handlers need to know how to push unrequested pages at some
>> point in time to make sure the migration can finish.
>  
> I'd say we need 2a that transfers pages that haven't faulted yet in the
> background and then 2 and 3 may end after background transfer is finished.

I've proposed the "ideal process 3" in my response to Adrian :) Please,
share your thought on it.

>> Looking at the page-server it is currently not clear to me how it fits
>> into this scenario. Currently it listens on a network port (like process
>> 3. from above) and writes the received pages to the local disk.
>>
>> To serve as the process mention as process 3. it would need to learn all
>> the functionality as it has currently been implemented.
>>
>> Instead of receiving pages and writing it to disk it needs to
>> receive page requests and read the from disk to the network.
>>
>> This sounds like the opposite of what it is currently doing and,
>> from my point of view, it is either a complete separate process,
>> like my implementation, or all the functionality needs to be added.
>> Also the logic to handle unrequested pages does not seem like
>> something which the page-server can currently do or is designed to do.
>>
>> So, from my point of view, page-server and remote page request handler
>> seem rather different in their functionality (besides being a TCP
>> server). I suppose there are some points I am not seeing so I hope to
>> understand the situation better from the answers to this email. Thanks.
>>
>> 		Adrian
> 
> In the current CRIU implementation, the page-server is indeed have
> different functionality from that required to serve remote page fault
> requests. Still, this functionality needs to be integrated with other parts
> of CRIU and adding it to the page server is as good as adding it elsewhere.

Yes. By page-server I meant the page-xfer, page-read and protocol components
rather than the process that ties it all together.

> And, extending page-read to handle remote pages as well as pages read from
> a file seems quite logical.

Not sure whether we'll be able to fully re-use the page_server_serve() :) but
smth like this is required.

> My two kopeyka's :)
> 
> --
> Sincerely yours,
> Mike.
> .
>