<div dir="ltr">Here is an explanation of the problem. As you know, the AUFS implementation in the kernel has a bug. Specifically, by pointing inside AUFS branches, the file descriptors of links in /proc/<pid>/map_files create two problems (these file descriptors are saved by ma_get_mapfile() in vma->vm_file_fd):<div><div><div><br></div><div>1. When doing readlink() on them, the path inside the branch is not accessible in the mount namespace.</div><div><br></div><div>2. When reading their /proc/<pid>/fdinfo, the mnt_id value corresponds to a mountpoint that is not visible in the mount namespace.</div><div><br></div><div>The patch that we applied earlier addressed issue 1 above. Attached is a patch for issue 2 if we need a quick workaround but I would like a better solution. The patch has a special provision for AUFS to return the root mountpoint if mnt_id cannot be found.<br></div><div><br></div><div>What do you guys think?</div><div><br></div><div>--Saied</div></div><div><br></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Wed, Sep 3, 2014 at 12:32 AM, Andrew Vagin <span dir="ltr"><<a href="mailto:avagin@parallels.com" target="_blank">avagin@parallels.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span>On Tue, Sep 02, 2014 at 03:14:17PM -0700, Saied Kazemi wrote:<br>
> Pertaining to my previous email, I noticed that when I upgraded my Ubuntu<br>
> 14.04's default 3.13 kernel to 3.15 using instructions at http://<br>
> <a href="http://ubuntuhandbook.org/index.php/2014/06/install-upgrade-linux-kernel-3-15" target="_blank">ubuntuhandbook.org/index.php/2014/06/install-upgrade-linux-kernel-3-15</a>, it<br>
> lacks AUFS support causing Docker to use VFS.<br>
><br>
> I need to test the same system as yours with 3.15 kernel and AUFS support.<br>
> What are you using?<br>
<br>
</span>I use the aufs3.16 branch of <a href="https://github.com/sfjro/aufs3-linux" target="_blank">https://github.com/sfjro/aufs3-linux</a><br>
<div><div><br>
><br>
> --Saied<br>
><br>
><br>
><br>
> On Tue, Sep 2, 2014 at 11:36 AM, Saied Kazemi <<a href="mailto:saied@google.com" target="_blank">saied@google.com</a>> wrote:<br>
><br>
> Andrew,<br>
><br>
> I installed 3.15 and ran the env00 test but it succeeded for me (please see<br>
> below). I also verified that checkpointing and restoring Docker containers<br>
> using AUFS still works.<br>
><br>
> I noticed that you're specifying --aufs which was removed in commit<br>
> 102a6778.<br>
><br>
> Can you please try again with CRIU source from the head? If it still<br>
> fails, please send me specific information on how to reproduce the problem.<br>
><br>
> Thanks,<br>
><br>
> --Saied<br>
><br>
> $ uname -a<br>
> Linux saied-docker 3.15.0-031500-generic #201406131105 SMP Fri Jun 13<br>
> 15:06:46 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux<br>
> $ sudo bash test/zdtm.sh ns/static/env00<br>
> ================================= CRIU CHECK ==============================<br>
> ===<br>
> Error (timerfd.c:56): timerfd: No timerfd support for c/r: Inappropriate<br>
> ioctl for device<br>
> ============================= WARNING =============================<br>
> Not all features needed for CRIU are merged to upstream kernel yet,<br>
> so for now we maintain our own branch which can be cloned from:<br>
> git://<a href="http://git.kernel.org/pub/scm/linux/kernel/git/gorcunov/linux-cr.git" target="_blank">git.kernel.org/pub/scm/linux/kernel/git/gorcunov/linux-cr.git</a><br>
> ===================================================================<br>
> Execute zdtm/live/static/env00<br>
> ./env00 --pidfile=env00.pid --outfile=env00.out --envname=ENV_00_TEST<br>
> /home/saied/work/crtools.upstream/test<br>
> Dump 6558<br>
> Restore<br>
> Check results 6587<br>
> 18:26:23.831: 5: PASS<br>
> Test: zdtm/live/static/env00, Result: PASS<br>
> ZDTM tests PASS.<br>
><br>
> $<br>
><br>
><br>
> On Tue, Sep 2, 2014 at 9:07 AM, Saied Kazemi <<a href="mailto:saied@google.com" target="_blank">saied@google.com</a>> wrote:<br>
><br>
> Hi Andrew,<br>
><br>
> I was kind of afraid of this because none of the kernels that I had<br>
> supported mnt_id. I will install 3.15 to investigate the issue and get<br>
> back to you soon.<br>
><br>
> --Saied<br>
><br>
><br>
><br>
> On Tue, Sep 2, 2014 at 7:05 AM, Andrew Vagin <<a href="mailto:avagin@parallels.com" target="_blank">avagin@parallels.com</a>><br>
> wrote:<br>
><br>
> Hi Saied,<br>
><br>
> I found an issue, which blocks testing of aufs.<br>
> We can't dump nothing on a new kernel (3.15), which shows mnt_id in<br>
> /proc/PID/fdinfo/FD. Looks like we need to fixup mnt_id for aufs<br>
> files<br>
> too.<br>
><br>
> Said, could you look at this issue?<br>
><br>
><br>
> [root@avagin-fc19-cr criu]# bash test/zdtm.sh --aufs ns/static/<br>
> env00<br>
> ================================= CRIU CHECK ======================<br>
> ===========<br>
> Error (timerfd.c:56): timerfd: No timerfd support for c/r:<br>
> Inappropriate ioctl for device<br>
> ============================= WARNING =============================<br>
> Not all features needed for CRIU are merged to upstream kernel yet,<br>
> so for now we maintain our own branch which can be cloned from:<br>
> git://<a href="http://git.kernel.org/pub/scm/linux/kernel/git/gorcunov/linux-cr.git" target="_blank">git.kernel.org/pub/scm/linux/kernel/git/gorcunov/linux-cr.git</a><br>
> ===================================================================<br>
> Execute zdtm/live/static/env00<br>
> ./env00 --pidfile=env00.pid --outfile=env00.out --envname=<br>
> ENV_00_TEST<br>
</div></div><div><div>> /root/git/orig/criu/test<br>
> Dump 30437<br>
> WARNING: env00 returned 1 and left running for debug needs<br>
> Test: zdtm/live/static/env00, Result: FAIL<br>
> ==================================== ERROR ========================<br>
> ============<br>
> Test: zdtm/live/static/env00, Namespace: 1<br>
> Dump log : /root/git/orig/criu/test/dump/static/env00/30437/1/<br>
> dump.log<br>
> --------------------------------- grep Error<br>
> ---------------------------------<br>
> (00.035741) Error (files-reg.c:629): Unable to look up the 33 mount<br>
> (00.035771) Error (cr-dump.c:1665): Dump mappings (pid: 30437)<br>
> failed with -1<br>
> (00.036042) Error (cr-dump.c:1953): Dumping FAILED.<br>
> ------------------------------------- END<br>
> -------------------------------------<br>
> ================================= ERROR OVER ======================<br>
> ===========<br>
><br>
> [root@avagin-fc19-cr criu]# cat /proc/30437/mountinfo<br>
> 68 42 0:33 / / rw,relatime - aufs none rw,si=46777ba2737567db<br>
> 43 68 0:35 / /proc rw,relatime - proc proc rw<br>
> 44 68 0:36 / /dev/pts rw,relatime - devpts pts rw,mode=666,ptmxmode<br>
> =666<br>
><br>
> [root@avagin-fc19-cr criu]# cat /proc/self/mountinfo | grep ^33<br>
> 33 0 8:3 / / rw,relatime shared:1 - ext4 /dev/sda3 rw,data=ordered<br>
><br>
><br>
><br>
> On Sat, Aug 30, 2014 at 02:49:22PM +0400, Andrew Vagin wrote:<br>
> > And here are a few issues:<br>
> ><br>
> > + cat /tmp/criu-root.7GvnTZ/dump/static/cwd01/6970/1/dump.log<br>
> > (00.012328) Error (cr-dump.c:303): Can't open 6974/cwd on procfs:<br>
> Stale file handle<br>
> > (00.012350) Error (cr-dump.c:1671): Dump fs (pid: 6974) failed<br>
> with -1<br>
> > (00.012528) Error (cr-dump.c:1953): Dumping FAILED.<br>
> ><br>
> > + cat /tmp/criu-root.HTTKBE/dump/static/unlink_fstat03/14801/1/<br>
> dump.log<br>
> > (00.009870) Error (files-reg.c:455): Can't link remap to /zdtm/<br>
> live/static/unlink_fstat03.test (deleted): No such file or<br>
> directory<br>
> > (00.009906) Error (cr-dump.c:1603): Dump files (pid: 14805)<br>
> failed with -1<br>
> > (00.010533) Error (cr-dump.c:1953): Dumping FAILED.<br>
> ><br>
> > + cat /tmp/criu-root.HDf6Nn/dump/static/unlink_mmap00/23065/1/<br>
> dump.log<br>
> > (00.008996) Error (proc_parse.c:472): Failed stat on 23069's map<br>
> 47071697305600 (/tmp/criu-root.HDf6Nn/zdtm/live/static/<br>
> unlink_mmap00.test (deleted)): No such file or directory<br>
> > (00.009021) Error (cr-dump.c:1515): Collect mappings (pid: 23069)<br>
> failed with -1<br>
> > (00.009228) Error (cr-dump.c:1953): Dumping FAILED.<br>
> ><br>
> > + cat /tmp/criu-root.XULszc/dump/static/inotify00/30616/1/<br>
> dump.log<br>
> > (00.010422) Error (proc_parse.c:1345): parse_fdinfo_pid_s: error<br>
> parsing [inotify wd:2 ino:1202 sdev:24 mask:800afff ignored_mask:0<br>
> > (00.010465) Error (cr-dump.c:1603): Dump files (pid: 30620)<br>
> failed with -1<br>
> > (00.011124) Error (cr-dump.c:1953): Dumping FAILED.<br>
> ><br>
> > We have problems with unlinked files and fsnotifies.<br>
> ><br>
> > Cc: Saied Kazemi <<a href="mailto:saied@google.com" target="_blank">saied@google.com</a>><br>
> > Signed-off-by: Andrew Vagin <<a href="mailto:avagin@openvz.org" target="_blank">avagin@openvz.org</a>><br>
> > ---<br>
> > test/zdtm.sh | 11 ++++++++++-<br>
> > 1 file changed, 10 insertions(+), 1 deletion(-)<br>
> ><br>
> > diff --git a/test/zdtm.sh b/test/zdtm.sh<br>
> > index eee3342..f92a63f 100755<br>
> > --- a/test/zdtm.sh<br>
> > +++ b/test/zdtm.sh<br>
> > @@ -412,7 +412,12 @@ start_test()<br>
> > mkdir -p dump<br>
> > ZDTM_ROOT=`mktemp -d /tmp/criu-root.XXXXXX`<br>
> > ZDTM_ROOT=`readlink -f $ZDTM_ROOT`<br>
> > - mount --make-private --bind . $ZDTM_ROOT ||<br>
> return 1<br>
> > + if [ -z "$ZDTM_AUFS" ]; then<br>
> > + mount --make-private --bind .<br>
> $ZDTM_ROOT || return 1<br>
> > + else<br>
> > + mount -t aufs -o br=. none<br>
> $ZDTM_ROOT || return 1<br>
> > + cd $ZDTM_ROOT<br>
> > + fi<br>
> > fi<br>
> > construct_root $ZDTM_ROOT $test || return 1<br>
> > export ZDTM_NEWNS=1<br>
> > @@ -901,6 +906,10 @@ while :; do<br>
> > }<br>
> > shift<br>
> > ;;<br>
> > + --aufs)<br>
> > + export ZDTM_AUFS=1<br>
> > + shift<br>
> > + ;;<br>
> > -*)<br>
> > echo "Unrecognized option $1, aborting!" 1>&2<br>
> > usage<br>
> > --<br>
> > 1.9.3<br>
> ><br>
><br>
><br>
><br>
><br>
><br>
><br>
</div></div></blockquote></div><br></div></div>