[CRIU] problem dumping some kinds of lxc containers

[Tycho Andersen]

Hi Pavel,

On Wed, Aug 27, 2014 at 06:27:25PM +0400, Pavel Emelyanov wrote:
> On 08/27/2014 05:43 PM, Tycho Andersen wrote:
> > On Wed, Aug 27, 2014 at 05:11:39PM +0400, Pavel Emelyanov wrote:
> >> On 08/27/2014 05:08 PM, Tycho Andersen wrote:
> >>> Hi Pavel,
> >>>
> >>> On Wed, Aug 27, 2014 at 12:35:07PM +0400, Pavel Emelyanov wrote:
> >>>> On 08/27/2014 03:18 AM, Tycho Andersen wrote:
> >>>>> Hi all,
> >>>>>
> >>>>> I'm trying to dump an lxc container (created with the ubuntu-cloud
> >>>>> template). I get:
> >>>>>
> >>>>> (00.563988) Error (files-reg.c:457): Can't link remap to /proc/20/mountinfo: No such file or directory
> >>>>>
> >>>>> /proc/20 doesn't exist, and when this happens there is no pid in the
> >>>>> container with pid 20. This is a little confusing, though, since
> >>>>> fill_fdlink() takes a struct fd_parms with a pid in the host pid ns,
> >>>>> but gives back the path in the container pid ns.
> >>>>>
> >>>>> After a bit of debugging, I found that the process that is causing
> >>>>> this problem is:
> >>>>>
> >>>>> root     17593  0.3  0.0  26052  1340 ?        S    17:49   0:00 \_ mountall --daemon
> >>>>>
> >>>>> If I try to checkpoint the container after mountall has exited, it all
> >>>>> works fine.
> >>>>>
> >>>>> Any ideas what is going on here?
> >>>>
> >>>> Yes. CRIU finds an open file, that cannot be opened by the path kernel provides.
> >>>> In your case this is because task 20 has died. At the same time stat() reports
> >>>> that the link count on that file is not 0 (this is due to how proc works), which
> >>>> in case of disk file would mean, that file "should exist" and we just have to
> >>>> create some other name for it. This is called "link remap". For disk files CRIU
> >>>> handles it by creating a hard link on the file. For proc this will obviously not
> >>>> work, we have to invent something else.
> >>>
> >>> Thanks for the explanation. Any ideas on what the proper solution is?
> >>
> >> I was thinking that when we meet an opened file of a dead task, we could create
> >> a "fake" one on restore with desired pid (it can be a light-weight task with FS,
> >> VM, FILES, etc. shared with parent), wait for its /proc/pid/smth to get opened,
> >> then kill one.
> >>
> >> We have TASK_HELPER state for that in CRIU, they help us restore orphaned pgrps
> >> and sessions. Probably these helpers can help here too :)
> > 
> > Just to repeat it in my own words so I understand: if we see a
> > /proc/$pid that doesn't exist, we write it down somewhere (a new
> > protobuf, or would it fit somewhere currently?), and then on restore
> > we create a task helper for each pid we found. The rest of the restore
> > process opens /proc/$pid/whatever, and then it is ok for the task
> > helper to exit immediately once the restore completes and we are
> > running the actual processes.
> 
> Exactly.
> 
> > Does that sound about right? Any ideas where the "fake" pid list would
> > go?
> 
> We have a (not extremely elegant, but) way of telling to files
> restoring engine, that "the path of a file doesn't exit, need to
> take additional steps before opening one" called "remap".
> 
> Currently we have 2 types of remaps -- ghost and link. Ghost remap
> is a file, that doesn't have any names on disk and is only alive
> due to some process has one opened. Ghosts are taken with us into
> images. Link remaps are files with the path we see it by being
> unlinked, but with some other name. For this files we create hard
> links on disks.
> 
> I think it's worth trying to introduce the 3rd type of remaps
> which correspond to proc files that correspond to dead tasks.

Sounds good. I will work on a patch. Thanks!

Tycho

> Thanks,
> Pavel
>