<div dir="ltr">Sure, and that would be the right solution. As I haven't worked with J. R. Okajima, please feel free to let him know about the issues since you seem to be working with him already. Alternatively, I can let him know too.<div><br></div><div>That said, please note that we'd still need logic in CRIU while the AUFS fixed isn't available.</div><div><br></div><div>--Saied</div><div><br></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Mon, Sep 8, 2014 at 8:49 AM, Andrew Vagin <span dir="ltr"><<a href="mailto:avagin@parallels.com" target="_blank">avagin@parallels.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">On Sun, Sep 07, 2014 at 11:36:15PM -0700, Saied Kazemi wrote:<br>
> Here is an explanation of the problem. As you know, the AUFS implementation in<br>
> the kernel has a bug. Specifically, by pointing inside AUFS branches, the file<br>
<br>
</span>Saied, can we ask AUFS developers to fix this bug? I was talking with J. R.<br>
Okajima about inotify issues and he fixed all this issues very fast.<br>
<div class="HOEnZb"><div class="h5"><br>
> descriptors of links in /proc/<pid>/map_files create two problems (these file<br>
> descriptors are saved by ma_get_mapfile() in vma->vm_file_fd):<br>
><br>
> 1. When doing readlink() on them, the path inside the branch is not accessible<br>
> in the mount namespace.<br>
><br>
> 2. When reading their /proc/<pid>/fdinfo, the mnt_id value corresponds to a<br>
> mountpoint that is not visible in the mount namespace.<br>
><br>
> The patch that we applied earlier addressed issue 1 above. Attached is a patch<br>
> for issue 2 if we need a quick workaround but I would like a better solution.<br>
> The patch has a special provision for AUFS to return the root mountpoint if<br>
> mnt_id cannot be found.<br>
><br>
> What do you guys think?<br>
><br>
> --Saied<br>
><br>
><br>
><br>
> On Wed, Sep 3, 2014 at 12:32 AM, Andrew Vagin <<a href="mailto:avagin@parallels.com">avagin@parallels.com</a>> wrote:<br>
><br>
> On Tue, Sep 02, 2014 at 03:14:17PM -0700, Saied Kazemi wrote:<br>
> > Pertaining to my previous email, I noticed that when I upgraded my Ubuntu<br>
> > 14.04's default 3.13 kernel to 3.15 using instructions at http://<br>
> > <a href="http://ubuntuhandbook.org/index.php/2014/06/install-upgrade-linux-kernel-3-15" target="_blank">ubuntuhandbook.org/index.php/2014/06/install-upgrade-linux-kernel-3-15</a>,<br>
> it<br>
> > lacks AUFS support causing Docker to use VFS.<br>
> ><br>
> > I need to test the same system as yours with 3.15 kernel and AUFS<br>
> support.<br>
> > What are you using?<br>
><br>
> I use the aufs3.16 branch of <a href="https://github.com/sfjro/aufs3-linux" target="_blank">https://github.com/sfjro/aufs3-linux</a><br>
><br>
> ><br>
> > --Saied<br>
> ><br>
> ><br>
> ><br>
> > On Tue, Sep 2, 2014 at 11:36 AM, Saied Kazemi <<a href="mailto:saied@google.com">saied@google.com</a>> wrote:<br>
> ><br>
> > Andrew,<br>
> ><br>
> > I installed 3.15 and ran the env00 test but it succeeded for me<br>
> (please see<br>
> > below). I also verified that checkpointing and restoring Docker<br>
> containers<br>
> > using AUFS still works.<br>
> ><br>
> > I noticed that you're specifying --aufs which was removed in commit<br>
> > 102a6778.<br>
> ><br>
> > Can you please try again with CRIU source from the head? If it still<br>
> > fails, please send me specific information on how to reproduce the<br>
> problem.<br>
> ><br>
> > Thanks,<br>
> ><br>
> > --Saied<br>
> ><br>
> > $ uname -a<br>
> > Linux saied-docker 3.15.0-031500-generic #201406131105 SMP Fri Jun 13<br>
> > 15:06:46 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux<br>
> > $ sudo bash test/zdtm.sh ns/static/env00<br>
> > ================================= CRIU CHECK ========================<br>
> ======<br>
> > ===<br>
> > Error (timerfd.c:56): timerfd: No timerfd support for c/r:<br>
> Inappropriate<br>
> > ioctl for device<br>
> > ============================= WARNING =============================<br>
> > Not all features needed for CRIU are merged to upstream kernel yet,<br>
> > so for now we maintain our own branch which can be cloned from:<br>
> > git://<a href="http://git.kernel.org/pub/scm/linux/kernel/git/gorcunov/linux-cr.git" target="_blank">git.kernel.org/pub/scm/linux/kernel/git/gorcunov/linux-cr.git</a><br>
> > ===================================================================<br>
> > Execute zdtm/live/static/env00<br>
> > ./env00 --pidfile=env00.pid --outfile=env00.out --envname=ENV_00_TEST<br>
> > /home/saied/work/crtools.upstream/test<br>
> > Dump 6558<br>
> > Restore<br>
> > Check results 6587<br>
> > 18:26:23.831: 5: PASS<br>
> > Test: zdtm/live/static/env00, Result: PASS<br>
> > ZDTM tests PASS.<br>
> ><br>
> > $<br>
> ><br>
> ><br>
> > On Tue, Sep 2, 2014 at 9:07 AM, Saied Kazemi <<a href="mailto:saied@google.com">saied@google.com</a>><br>
> wrote:<br>
> ><br>
> > Hi Andrew,<br>
> ><br>
> > I was kind of afraid of this because none of the kernels that I<br>
> had<br>
> > supported mnt_id. I will install 3.15 to investigate the issue<br>
> and get<br>
> > back to you soon.<br>
> ><br>
> > --Saied<br>
> ><br>
> ><br>
> ><br>
> > On Tue, Sep 2, 2014 at 7:05 AM, Andrew Vagin <<br>
> <a href="mailto:avagin@parallels.com">avagin@parallels.com</a>><br>
> > wrote:<br>
> ><br>
> > Hi Saied,<br>
> ><br>
> > I found an issue, which blocks testing of aufs.<br>
> > We can't dump nothing on a new kernel (3.15), which shows<br>
> mnt_id in<br>
> > /proc/PID/fdinfo/FD. Looks like we need to fixup mnt_id for<br>
> aufs<br>
> > files<br>
> > too.<br>
> ><br>
> > Said, could you look at this issue?<br>
> ><br>
> ><br>
> > [root@avagin-fc19-cr criu]# bash test/zdtm.sh --aufs ns/<br>
> static/<br>
> > env00<br>
> > ================================= CRIU CHECK ================<br>
> ======<br>
> > ===========<br>
> > Error (timerfd.c:56): timerfd: No timerfd support for c/r:<br>
> > Inappropriate ioctl for device<br>
> > ============================= WARNING =======================<br>
> ======<br>
> > Not all features needed for CRIU are merged to upstream<br>
> kernel yet,<br>
> > so for now we maintain our own branch which can be cloned<br>
> from:<br>
> > git://<a href="http://git.kernel.org/pub/scm/linux/kernel/git/gorcunov/" target="_blank">git.kernel.org/pub/scm/linux/kernel/git/gorcunov/</a><br>
> linux-cr.git<br>
> > =============================================================<br>
> ======<br>
</div></div><div class="HOEnZb"><div class="h5">> > Execute zdtm/live/static/env00<br>
> > ./env00 --pidfile=env00.pid --outfile=env00.out --envname=<br>
> > ENV_00_TEST<br>
> > /root/git/orig/criu/test<br>
> > Dump 30437<br>
> > WARNING: env00 returned 1 and left running for debug needs<br>
> > Test: zdtm/live/static/env00, Result: FAIL<br>
> > ==================================== ERROR ==================<br>
> ======<br>
> > ============<br>
> > Test: zdtm/live/static/env00, Namespace: 1<br>
> > Dump log : /root/git/orig/criu/test/dump/static/env00/30437<br>
> /1/<br>
> > dump.log<br>
> > --------------------------------- grep Error<br>
> > ---------------------------------<br>
> > (00.035741) Error (files-reg.c:629): Unable to look up the 33<br>
> mount<br>
> > (00.035771) Error (cr-dump.c:1665): Dump mappings (pid:<br>
> 30437)<br>
> > failed with -1<br>
> > (00.036042) Error (cr-dump.c:1953): Dumping FAILED.<br>
> > ------------------------------------- END<br>
> > -------------------------------------<br>
> > ================================= ERROR OVER ================<br>
> ======<br>
> > ===========<br>
> ><br>
> > [root@avagin-fc19-cr criu]# cat /proc/30437/mountinfo<br>
> > 68 42 0:33 / / rw,relatime - aufs none rw,si=46777ba2737567db<br>
> > 43 68 0:35 / /proc rw,relatime - proc proc rw<br>
> > 44 68 0:36 / /dev/pts rw,relatime - devpts pts rw,mode=<br>
> 666,ptmxmode<br>
> > =666<br>
> ><br>
> > [root@avagin-fc19-cr criu]# cat /proc/self/mountinfo | grep ^<br>
> 33<br>
> > 33 0 8:3 / / rw,relatime shared:1 - ext4 /dev/sda3 rw,data=<br>
> ordered<br>
> ><br>
> ><br>
> ><br>
> > On Sat, Aug 30, 2014 at 02:49:22PM +0400, Andrew Vagin wrote:<br>
> > > And here are a few issues:<br>
> > ><br>
> > > + cat /tmp/criu-root.7GvnTZ/dump/static/cwd01/6970/1/<br>
> dump.log<br>
> > > (00.012328) Error (cr-dump.c:303): Can't open 6974/cwd on<br>
> procfs:<br>
> > Stale file handle<br>
> > > (00.012350) Error (cr-dump.c:1671): Dump fs (pid: 6974)<br>
> failed<br>
> > with -1<br>
> > > (00.012528) Error (cr-dump.c:1953): Dumping FAILED.<br>
> > ><br>
> > > + cat /tmp/criu-root.HTTKBE/dump/static/unlink_fstat03/<br>
> 14801/1/<br>
> > dump.log<br>
> > > (00.009870) Error (files-reg.c:455): Can't link remap to /<br>
> zdtm/<br>
> > live/static/unlink_fstat03.test (deleted): No such file or<br>
> > directory<br>
> > > (00.009906) Error (cr-dump.c:1603): Dump files (pid: 14805)<br>
> > failed with -1<br>
> > > (00.010533) Error (cr-dump.c:1953): Dumping FAILED.<br>
> > ><br>
> > > + cat /tmp/criu-root.HDf6Nn/dump/static/unlink_mmap00/23065<br>
> /1/<br>
> > dump.log<br>
> > > (00.008996) Error (proc_parse.c:472): Failed stat on<br>
> 23069's map<br>
> > 47071697305600 (/tmp/criu-root.HDf6Nn/zdtm/live/static/<br>
> > unlink_mmap00.test (deleted)): No such file or directory<br>
> > > (00.009021) Error (cr-dump.c:1515): Collect mappings (pid:<br>
> 23069)<br>
> > failed with -1<br>
> > > (00.009228) Error (cr-dump.c:1953): Dumping FAILED.<br>
> > ><br>
> > > + cat /tmp/criu-root.XULszc/dump/static/inotify00/30616/1/<br>
> > dump.log<br>
> > > (00.010422) Error (proc_parse.c:1345): parse_fdinfo_pid_s:<br>
> error<br>
> > parsing [inotify wd:2 ino:1202 sdev:24 mask:800afff<br>
> ignored_mask:0<br>
> > > (00.010465) Error (cr-dump.c:1603): Dump files (pid: 30620)<br>
> > failed with -1<br>
> > > (00.011124) Error (cr-dump.c:1953): Dumping FAILED.<br>
> > ><br>
> > > We have problems with unlinked files and fsnotifies.<br>
> > ><br>
> > > Cc: Saied Kazemi <<a href="mailto:saied@google.com">saied@google.com</a>><br>
> > > Signed-off-by: Andrew Vagin <<a href="mailto:avagin@openvz.org">avagin@openvz.org</a>><br>
> > > ---<br>
> > > test/zdtm.sh | 11 ++++++++++-<br>
> > > 1 file changed, 10 insertions(+), 1 deletion(-)<br>
> > ><br>
> > > diff --git a/test/zdtm.sh b/test/zdtm.sh<br>
> > > index eee3342..f92a63f 100755<br>
> > > --- a/test/zdtm.sh<br>
> > > +++ b/test/zdtm.sh<br>
> > > @@ -412,7 +412,12 @@ start_test()<br>
> > > mkdir -p dump<br>
> > > ZDTM_ROOT=`mktemp -d /tmp/<br>
> criu-root.XXXXXX`<br>
> > > ZDTM_ROOT=`readlink -f $ZDTM_ROOT`<br>
> > > - mount --make-private --bind .<br>
> $ZDTM_ROOT ||<br>
> > return 1<br>
> > > + if [ -z "$ZDTM_AUFS" ]; then<br>
> > > + mount --make-private --bind .<br>
> > $ZDTM_ROOT || return 1<br>
> > > + else<br>
> > > + mount -t aufs -o br=. none<br>
> > $ZDTM_ROOT || return 1<br>
> > > + cd $ZDTM_ROOT<br>
> > > + fi<br>
> > > fi<br>
> > > construct_root $ZDTM_ROOT $test || return 1<br>
> > > export ZDTM_NEWNS=1<br>
> > > @@ -901,6 +906,10 @@ while :; do<br>
> > > }<br>
> > > shift<br>
> > > ;;<br>
> > > + --aufs)<br>
> > > + export ZDTM_AUFS=1<br>
> > > + shift<br>
> > > + ;;<br>
> > > -*)<br>
> > > echo "Unrecognized option $1, aborting!" 1>&2<br>
> > > usage<br>
> > > --<br>
> > > 1.9.3<br>
> > ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
><br>
><br>
<br>
<br>
</div></div></blockquote></div><br></div>