[CRIU] Lazy-restore design discussion - round 3

[Adrian Reber]

On Tue, Apr 19, 2016 at 05:03:57PM +0200, Adrian Reber wrote:
> > >>> Maybe we really should implement it like Mike said. First try to get the
> > >>> current locally on my and on Mike's system existing patches into shape and
> > >>> then we can decide if we want to move the page handling logic to the
> > >>> dump side on the destination system.
> > >>
> > >> OK, let's see how it goes.
> > >>
> > >> But I have one concern about having brains on restore side. Look, the uffd can request
> > >> for two kinds (or types) of pages -- those that task are blocked on in #PF (i.e. -- 
> > >> explicit uffd requests) and those that task hasn't yet touched (i.e. -- request them
> > >> in advance). With the former pages the situation is clear, it's uffd who knows what
> > >> these pages are. It can even know something about the latter pages, e.g. with #PF-ed
> > >> pages request for adjacent pages as Adrian proposed. That's clear. But what to do
> > >> with other "in advance" pages. It seems that it's better to request those pages in
> > >> LRU manner, i.e. -- request for recent pages before those that were used long ago. But
> > >> the problem I see is that this LRU information can only be obtained from the dump
> > >> side -- all this LRU statistics sits _there_. And what would be the way to share
> > >> this knowledge with the restore side (as we plan to make it "smart" or "active")?
> > >>
> > >> Had we the "brain" (or "active part") on dump side we could just scan this info and
> > >> make decision. But what to do when we have "brain" on restore side and all the LRU
> > >> info on the dump side?
> > > 
> > >>From where do we have the LRU information? Does CRIU collect this during
> > > dump? Or can this be queried from the kernel?
> > 
> > Right now we don't collect it by CRIU, it's only present in CRIU (and somehow can be
> > collected using reference-d bit from the proc pagemap file, but we don't do it either).
> > My point is that this information is only present on the dump side.
> 
> Good to know, that this information exists. Your proposal to insert
> additional pages based on LRU basis makes sense. I need to think about
> this a bit...

Knowing about the LRU data on the dump side this means that either the
dump side decides which pages are transmitted or with have to transmit
the LRU data to the uffd daemon on the restore side. Transferring the
LRU data to uffd daemon sounds like it will make the whole thing
unnecessarily complicated. Having the logic which pages are transferred
when on the dump side means, that the uffd daemon on the restore side
does nothing more than forwarding pages or userfault requests. This also
means it doesn't need to know anything about the restored process and
therefore it does not need to access the checkpoint directory. From how
I see it this would lead to a similar solution like the one implemented
in my first remote lazy restore patchset. Only based on the
page-server protocol.

Thinking more about how the protocol needs to be implemented between
dump side and uffd daemon it also sounds like the uffd daemon, which is
now only forwarding pages and userfault requests, will/should lose the
ability to read local checkpoint directories. It will only be used for
forwarding pages. A lazy-restore with source and destination on the same
machine will then probably also require to forward the pages through the
local uffd-page-forwarding-daemon.

So the above is the result of my thoughts about what it means that the
LRU data only exists on the dump side. This is not (yet) what I am
proposing to do, it is just what I think this might lead to.

		Adrian