<div dir="ltr">Thanks for the patch Andrew. I did a quick test and verified that it fixes the problem we see in /proc/<pid>/map_files. C/R still worked with the existing AUFS support code in CRIU although I didn't test extensively. When this kernel patch has made it to all distributions, we can revert the AUFS patches in CRIU.<div><br></div><div>Since it will take some time for the kernel patch to be in all distros, what do you and Pavel think about the mnt_id patch that I sent earlier?<br><div><div><br></div><div>--Saied</div><div><br></div></div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Sep 8, 2014 at 10:33 PM, Andrew Vagin <span dir="ltr"><<a href="mailto:avagin@parallels.com" target="_blank">avagin@parallels.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">On Mon, Sep 08, 2014 at 09:00:53AM -0700, Saied Kazemi wrote:<br>
> Sure, and that would be the right solution. As I haven't worked with J. R.<br>
> Okajima, please feel free to let him know about the issues since you seem to be<br>
> working with him already. Alternatively, I can let him know too.<br>
<br>
</span>The patch from J. R. Okajima is attached.<br>
<div class="HOEnZb"><div class="h5"><br>
><br>
> That said, please note that we'd still need logic in CRIU while the AUFS fixed<br>
> isn't available.<br>
><br>
> --Saied<br>
><br>
><br>
><br>
> On Mon, Sep 8, 2014 at 8:49 AM, Andrew Vagin <<a href="mailto:avagin@parallels.com">avagin@parallels.com</a>> wrote:<br>
><br>
> On Sun, Sep 07, 2014 at 11:36:15PM -0700, Saied Kazemi wrote:<br>
> > Here is an explanation of the problem. As you know, the AUFS<br>
> implementation in<br>
> > the kernel has a bug. Specifically, by pointing inside AUFS branches,<br>
> the file<br>
><br>
> Saied, can we ask AUFS developers to fix this bug? I was talking with J. R.<br>
> Okajima about inotify issues and he fixed all this issues very fast.<br>
><br>
> > descriptors of links in /proc/<pid>/map_files create two problems (these<br>
> file<br>
> > descriptors are saved by ma_get_mapfile() in vma->vm_file_fd):<br>
> ><br>
> > 1. When doing readlink() on them, the path inside the branch is not<br>
> accessible<br>
> > in the mount namespace.<br>
> ><br>
> > 2. When reading their /proc/<pid>/fdinfo, the mnt_id value corresponds to<br>
> a<br>
> > mountpoint that is not visible in the mount namespace.<br>
> ><br>
> > The patch that we applied earlier addressed issue 1 above. Attached is a<br>
> patch<br>
> > for issue 2 if we need a quick workaround but I would like a better<br>
> solution.<br>
> > The patch has a special provision for AUFS to return the root mountpoint<br>
> if<br>
> > mnt_id cannot be found.<br>
> ><br>
> > What do you guys think?<br>
> ><br>
> > --Saied<br>
> ><br>
> ><br>
> ><br>
> > On Wed, Sep 3, 2014 at 12:32 AM, Andrew Vagin <<a href="mailto:avagin@parallels.com">avagin@parallels.com</a>><br>
> wrote:<br>
> ><br>
> > On Tue, Sep 02, 2014 at 03:14:17PM -0700, Saied Kazemi wrote:<br>
> > > Pertaining to my previous email, I noticed that when I upgraded my<br>
> Ubuntu<br>
> > > 14.04's default 3.13 kernel to 3.15 using instructions at http://<br>
> > > <a href="http://ubuntuhandbook.org/index.php/2014/06/" target="_blank">ubuntuhandbook.org/index.php/2014/06/</a><br>
> install-upgrade-linux-kernel-3-15,<br>
> > it<br>
> > > lacks AUFS support causing Docker to use VFS.<br>
> > ><br>
> > > I need to test the same system as yours with 3.15 kernel and AUFS<br>
> > support.<br>
> > > What are you using?<br>
> ><br>
> > I use the aufs3.16 branch of <a href="https://github.com/sfjro/aufs3-linux" target="_blank">https://github.com/sfjro/aufs3-linux</a><br>
> ><br>
> > ><br>
> > > --Saied<br>
> > ><br>
> > ><br>
> > ><br>
> > > On Tue, Sep 2, 2014 at 11:36 AM, Saied Kazemi <<a href="mailto:saied@google.com">saied@google.com</a>><br>
> wrote:<br>
> > ><br>
> > > Andrew,<br>
> > ><br>
> > > I installed 3.15 and ran the env00 test but it succeeded for me<br>
> > (please see<br>
> > > below). I also verified that checkpointing and restoring<br>
> Docker<br>
> > containers<br>
> > > using AUFS still works.<br>
> > ><br>
> > > I noticed that you're specifying --aufs which was removed in<br>
> commit<br>
> > > 102a6778.<br>
> > ><br>
> > > Can you please try again with CRIU source from the head? If it<br>
> still<br>
> > > fails, please send me specific information on how to reproduce<br>
> the<br>
> > problem.<br>
> > ><br>
> > > Thanks,<br>
> > ><br>
> > > --Saied<br>
> > ><br>
> > > $ uname -a<br>
> > > Linux saied-docker 3.15.0-031500-generic #201406131105 SMP Fri<br>
> Jun 13<br>
> > > 15:06:46 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux<br>
> > > $ sudo bash test/zdtm.sh ns/static/env00<br>
> > > ================================= CRIU CHECK ==================<br>
> ======<br>
> > ======<br>
> > > ===<br>
> > > Error (timerfd.c:56): timerfd: No timerfd support for c/r:<br>
> > Inappropriate<br>
> > > ioctl for device<br>
</div></div><span class="im HOEnZb">> > > ============================= WARNING =========================<br>
> ====<br>
> > > Not all features needed for CRIU are merged to upstream kernel<br>
> yet,<br>
> > > so for now we maintain our own branch which can be cloned from:<br>
> > > git://<a href="http://git.kernel.org/pub/scm/linux/kernel/git/gorcunov/" target="_blank">git.kernel.org/pub/scm/linux/kernel/git/gorcunov/</a><br>
> linux-cr.git<br>
> > > ===============================================================<br>
> ====<br>
> > > Execute zdtm/live/static/env00<br>
> > > ./env00 --pidfile=env00.pid --outfile=env00.out --envname=<br>
> ENV_00_TEST<br>
</span><div class="HOEnZb"><div class="h5">> > > /home/saied/work/crtools.upstream/test<br>
> > > Dump 6558<br>
> > > Restore<br>
> > > Check results 6587<br>
> > > 18:26:23.831: 5: PASS<br>
> > > Test: zdtm/live/static/env00, Result: PASS<br>
> > > ZDTM tests PASS.<br>
> > ><br>
> > > $<br>
> > ><br>
> > ><br>
> > > On Tue, Sep 2, 2014 at 9:07 AM, Saied Kazemi <<a href="mailto:saied@google.com">saied@google.com</a>><br>
> > wrote:<br>
> > ><br>
> > > Hi Andrew,<br>
> > ><br>
> > > I was kind of afraid of this because none of the kernels<br>
> that I<br>
> > had<br>
> > > supported mnt_id. I will install 3.15 to investigate the<br>
> issue<br>
> > and get<br>
> > > back to you soon.<br>
> > ><br>
> > > --Saied<br>
> > ><br>
> > ><br>
> > ><br>
> > > On Tue, Sep 2, 2014 at 7:05 AM, Andrew Vagin <<br>
> > <a href="mailto:avagin@parallels.com">avagin@parallels.com</a>><br>
> > > wrote:<br>
> > ><br>
> > > Hi Saied,<br>
> > ><br>
> > > I found an issue, which blocks testing of aufs.<br>
> > > We can't dump nothing on a new kernel (3.15), which<br>
> shows<br>
> > mnt_id in<br>
> > > /proc/PID/fdinfo/FD. Looks like we need to fixup mnt_id<br>
> for<br>
> > aufs<br>
> > > files<br>
> > > too.<br>
> > ><br>
> > > Said, could you look at this issue?<br>
> > ><br>
> > ><br>
> > > [root@avagin-fc19-cr criu]# bash test/zdtm.sh --aufs<br>
> ns/<br>
> > static/<br>
> > > env00<br>
> > > ================================= CRIU CHECK ==========<br>
> ======<br>
> > ======<br>
> > > ===========<br>
> > > Error (timerfd.c:56): timerfd: No timerfd support for c<br>
> /r:<br>
</div></div><span class="im HOEnZb">> > > Inappropriate ioctl for device<br>
> > > ============================= WARNING =================<br>
> ======<br>
</span><div class="HOEnZb"><div class="h5">> > ======<br>
> > > Not all features needed for CRIU are merged to upstream<br>
> > kernel yet,<br>
> > > so for now we maintain our own branch which can be<br>
> cloned<br>
> > from:<br>
> > > git://<a href="http://git.kernel.org/pub/scm/linux/kernel/git/gorcunov/" target="_blank">git.kernel.org/pub/scm/linux/kernel/git/gorcunov/</a><br>
> > linux-cr.git<br>
> > > =======================================================<br>
> ======<br>
> > ======<br>
> > > Execute zdtm/live/static/env00<br>
> > > ./env00 --pidfile=env00.pid --outfile=env00.out<br>
> --envname=<br>
> > > ENV_00_TEST<br>
> > > /root/git/orig/criu/test<br>
> > > Dump 30437<br>
> > > WARNING: env00 returned 1 and left running for debug<br>
> needs<br>
> > > Test: zdtm/live/static/env00, Result: FAIL<br>
> > > ==================================== ERROR ============<br>
> ======<br>
> > ======<br>
> > > ============<br>
> > > Test: zdtm/live/static/env00, Namespace: 1<br>
> > > Dump log : /root/git/orig/criu/test/dump/static/env00<br>
> /30437<br>
> > /1/<br>
> > > dump.log<br>
> > > --------------------------------- grep Error<br>
> > > ---------------------------------<br>
> > > (00.035741) Error (files-reg.c:629): Unable to look up<br>
> the 33<br>
> > mount<br>
> > > (00.035771) Error (cr-dump.c:1665): Dump mappings (pid:<br>
> > 30437)<br>
> > > failed with -1<br>
> > > (00.036042) Error (cr-dump.c:1953): Dumping FAILED.<br>
> > > ------------------------------------- END<br>
> > > -------------------------------------<br>
> > > ================================= ERROR OVER ==========<br>
> ======<br>
> > ======<br>
> > > ===========<br>
> > ><br>
> > > [root@avagin-fc19-cr criu]# cat /proc/30437/mountinfo<br>
> > > 68 42 0:33 / / rw,relatime - aufs none rw,si=<br>
> 46777ba2737567db<br>
> > > 43 68 0:35 / /proc rw,relatime - proc proc rw<br>
> > > 44 68 0:36 / /dev/pts rw,relatime - devpts pts rw,mode=<br>
> > 666,ptmxmode<br>
> > > =666<br>
> > ><br>
> > > [root@avagin-fc19-cr criu]# cat /proc/self/mountinfo |<br>
> grep ^<br>
> > 33<br>
> > > 33 0 8:3 / / rw,relatime shared:1 - ext4 /dev/sda3<br>
> rw,data=<br>
> > ordered<br>
> > ><br>
> > ><br>
> > ><br>
> > > On Sat, Aug 30, 2014 at 02:49:22PM +0400, Andrew Vagin<br>
> wrote:<br>
> > > > And here are a few issues:<br>
> > > ><br>
> > > > + cat /tmp/criu-root.7GvnTZ/dump/static/cwd01/6970/1/<br>
> > dump.log<br>
> > > > (00.012328) Error (cr-dump.c:303): Can't open 6974/<br>
> cwd on<br>
> > procfs:<br>
> > > Stale file handle<br>
> > > > (00.012350) Error (cr-dump.c:1671): Dump fs (pid:<br>
> 6974)<br>
> > failed<br>
> > > with -1<br>
> > > > (00.012528) Error (cr-dump.c:1953): Dumping FAILED.<br>
> > > ><br>
> > > > + cat /tmp/criu-root.HTTKBE/dump/static/<br>
> unlink_fstat03/<br>
> > 14801/1/<br>
> > > dump.log<br>
> > > > (00.009870) Error (files-reg.c:455): Can't link remap<br>
> to /<br>
> > zdtm/<br>
> > > live/static/unlink_fstat03.test (deleted): No such file<br>
> or<br>
> > > directory<br>
> > > > (00.009906) Error (cr-dump.c:1603): Dump files (pid:<br>
> 14805)<br>
> > > failed with -1<br>
> > > > (00.010533) Error (cr-dump.c:1953): Dumping FAILED.<br>
> > > ><br>
> > > > + cat /tmp/criu-root.HDf6Nn/dump/static/unlink_mmap00<br>
> /23065<br>
> > /1/<br>
> > > dump.log<br>
> > > > (00.008996) Error (proc_parse.c:472): Failed stat on<br>
> > 23069's map<br>
> > > 47071697305600 (/tmp/criu-root.HDf6Nn/zdtm/live/static/<br>
> > > unlink_mmap00.test (deleted)): No such file or<br>
> directory<br>
> > > > (00.009021) Error (cr-dump.c:1515): Collect mappings<br>
> (pid:<br>
> > 23069)<br>
> > > failed with -1<br>
> > > > (00.009228) Error (cr-dump.c:1953): Dumping FAILED.<br>
> > > ><br>
> > > > + cat /tmp/criu-root.XULszc/dump/static/inotify00/<br>
> 30616/1/<br>
> > > dump.log<br>
> > > > (00.010422) Error (proc_parse.c:1345):<br>
> parse_fdinfo_pid_s:<br>
> > error<br>
> > > parsing [inotify wd:2 ino:1202 sdev:24 mask:800afff<br>
> > ignored_mask:0<br>
> > > > (00.010465) Error (cr-dump.c:1603): Dump files (pid:<br>
> 30620)<br>
> > > failed with -1<br>
> > > > (00.011124) Error (cr-dump.c:1953): Dumping FAILED.<br>
> > > ><br>
> > > > We have problems with unlinked files and fsnotifies.<br>
> > > ><br>
> > > > Cc: Saied Kazemi <<a href="mailto:saied@google.com">saied@google.com</a>><br>
> > > > Signed-off-by: Andrew Vagin <<a href="mailto:avagin@openvz.org">avagin@openvz.org</a>><br>
> > > > ---<br>
> > > > test/zdtm.sh | 11 ++++++++++-<br>
> > > > 1 file changed, 10 insertions(+), 1 deletion(-)<br>
> > > ><br>
> > > > diff --git a/test/zdtm.sh b/test/zdtm.sh<br>
> > > > index eee3342..f92a63f 100755<br>
> > > > --- a/test/zdtm.sh<br>
> > > > +++ b/test/zdtm.sh<br>
> > > > @@ -412,7 +412,12 @@ start_test()<br>
> > > > mkdir -p dump<br>
> > > > ZDTM_ROOT=`mktemp -d /tmp/<br>
> > criu-root.XXXXXX`<br>
> > > > ZDTM_ROOT=`readlink -f<br>
> $ZDTM_ROOT`<br>
> > > > - mount --make-private --bind .<br>
> > $ZDTM_ROOT ||<br>
> > > return 1<br>
> > > > + if [ -z "$ZDTM_AUFS" ]; then<br>
> > > > + mount --make-private<br>
> --bind .<br>
> > > $ZDTM_ROOT || return 1<br>
> > > > + else<br>
> > > > + mount -t aufs -o br=.<br>
> none<br>
> > > $ZDTM_ROOT || return 1<br>
> > > > + cd $ZDTM_ROOT<br>
> > > > + fi<br>
> > > > fi<br>
> > > > construct_root $ZDTM_ROOT $test ||<br>
> return 1<br>
> > > > export ZDTM_NEWNS=1<br>
> > > > @@ -901,6 +906,10 @@ while :; do<br>
> > > > }<br>
> > > > shift<br>
> > > > ;;<br>
> > > > + --aufs)<br>
> > > > + export ZDTM_AUFS=1<br>
> > > > + shift<br>
> > > > + ;;<br>
> > > > -*)<br>
> > > > echo "Unrecognized option $1, aborting!<br>
> " 1>&2<br>
> > > > usage<br>
> > > > --<br>
> > > > 1.9.3<br>
> > > ><br>
> > ><br>
> > ><br>
> > ><br>
> > ><br>
> > ><br>
> > ><br>
> ><br>
> ><br>
><br>
><br>
><br>
><br>
</div></div></blockquote></div><br></div>