<div dir="ltr">Here is an explanation of the problem.  As you know, the AUFS implementation in the kernel has a bug.  Specifically, by pointing inside AUFS branches, the file descriptors of links in /proc/&lt;pid&gt;/map_files create two problems (these file descriptors are saved by ma_get_mapfile() in vma-&gt;vm_file_fd):<div><div><div><br></div><div>1. When doing readlink() on them, the path inside the branch is not accessible in the mount namespace.</div><div><br></div><div>2. When reading their /proc/&lt;pid&gt;/fdinfo, the mnt_id value corresponds to a mountpoint that is not visible in the mount namespace.</div><div><br></div><div>The patch that we applied earlier addressed issue 1 above.  Attached is a patch for issue 2 if we need a quick workaround but I would like a better solution.  The patch has a special provision for AUFS to return the root mountpoint if mnt_id cannot be found.<br></div><div><br></div><div>What do you guys think?</div><div><br></div><div>--Saied</div></div><div><br></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Wed, Sep 3, 2014 at 12:32 AM, Andrew Vagin <span dir="ltr">&lt;<a href="mailto:avagin@parallels.com" target="_blank">avagin@parallels.com</a>&gt;</span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span>On Tue, Sep 02, 2014 at 03:14:17PM -0700, Saied Kazemi wrote:<br>
&gt; Pertaining to my previous email, I noticed that when I upgraded my Ubuntu<br>
&gt; 14.04&#39;s default 3.13 kernel to 3.15 using instructions at http://<br>
&gt; <a href="http://ubuntuhandbook.org/index.php/2014/06/install-upgrade-linux-kernel-3-15" target="_blank">ubuntuhandbook.org/index.php/2014/06/install-upgrade-linux-kernel-3-15</a>, it<br>
&gt; lacks AUFS support causing Docker to use VFS.<br>
&gt;<br>
&gt; I need to test the same system as yours with 3.15 kernel and AUFS support.<br>
&gt;  What are you using?<br>
<br>
</span>I use the aufs3.16 branch of <a href="https://github.com/sfjro/aufs3-linux" target="_blank">https://github.com/sfjro/aufs3-linux</a><br>
<div><div><br>
&gt;<br>
&gt; --Saied<br>
&gt;<br>
&gt;<br>
&gt;<br>
&gt; On Tue, Sep 2, 2014 at 11:36 AM, Saied Kazemi &lt;<a href="mailto:saied@google.com" target="_blank">saied@google.com</a>&gt; wrote:<br>
&gt;<br>
&gt;     Andrew,<br>
&gt;<br>
&gt;     I installed 3.15 and ran the env00 test but it succeeded for me (please see<br>
&gt;     below).  I also verified that checkpointing and restoring Docker containers<br>
&gt;     using AUFS still works.<br>
&gt;<br>
&gt;     I noticed that you&#39;re specifying --aufs which was removed in commit<br>
&gt;     102a6778.<br>
&gt;<br>
&gt;     Can you please try again with CRIU source from the head?  If it still<br>
&gt;     fails, please send me specific information on how to reproduce the problem.<br>
&gt;<br>
&gt;     Thanks,<br>
&gt;<br>
&gt;     --Saied<br>
&gt;<br>
&gt;     $ uname -a<br>
&gt;     Linux saied-docker 3.15.0-031500-generic #201406131105 SMP Fri Jun 13<br>
&gt;     15:06:46 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux<br>
&gt;     $ sudo bash test/zdtm.sh ns/static/env00<br>
&gt;     ================================= CRIU CHECK ==============================<br>
&gt;     ===<br>
&gt;     Error (timerfd.c:56): timerfd: No timerfd support for c/r: Inappropriate<br>
&gt;     ioctl for device<br>
&gt;     ============================= WARNING =============================<br>
&gt;     Not all features needed for CRIU are merged to upstream kernel yet,<br>
&gt;     so for now we maintain our own branch which can be cloned from:<br>
&gt;     git://<a href="http://git.kernel.org/pub/scm/linux/kernel/git/gorcunov/linux-cr.git" target="_blank">git.kernel.org/pub/scm/linux/kernel/git/gorcunov/linux-cr.git</a><br>
&gt;     ===================================================================<br>
&gt;     Execute zdtm/live/static/env00<br>
&gt;     ./env00 --pidfile=env00.pid --outfile=env00.out --envname=ENV_00_TEST<br>
&gt;     /home/saied/work/crtools.upstream/test<br>
&gt;     Dump 6558<br>
&gt;     Restore<br>
&gt;     Check results 6587<br>
&gt;     18:26:23.831:     5: PASS<br>
&gt;     Test: zdtm/live/static/env00, Result: PASS<br>
&gt;     ZDTM tests PASS.<br>
&gt;<br>
&gt;     $<br>
&gt;<br>
&gt;<br>
&gt;     On Tue, Sep 2, 2014 at 9:07 AM, Saied Kazemi &lt;<a href="mailto:saied@google.com" target="_blank">saied@google.com</a>&gt; wrote:<br>
&gt;<br>
&gt;         Hi Andrew,<br>
&gt;<br>
&gt;         I was kind of afraid of this because none of the kernels that I had<br>
&gt;         supported mnt_id.  I will install 3.15 to investigate the issue and get<br>
&gt;         back to you soon.<br>
&gt;<br>
&gt;         --Saied<br>
&gt;<br>
&gt;<br>
&gt;<br>
&gt;         On Tue, Sep 2, 2014 at 7:05 AM, Andrew Vagin &lt;<a href="mailto:avagin@parallels.com" target="_blank">avagin@parallels.com</a>&gt;<br>
&gt;         wrote:<br>
&gt;<br>
&gt;             Hi Saied,<br>
&gt;<br>
&gt;             I found an issue, which blocks testing of aufs.<br>
&gt;             We can&#39;t dump nothing on a new kernel (3.15), which shows mnt_id in<br>
&gt;             /proc/PID/fdinfo/FD. Looks like we need to fixup mnt_id for aufs<br>
&gt;             files<br>
&gt;             too.<br>
&gt;<br>
&gt;             Said, could you look at this issue?<br>
&gt;<br>
&gt;<br>
&gt;             [root@avagin-fc19-cr criu]# bash  test/zdtm.sh --aufs  ns/static/<br>
&gt;             env00<br>
&gt;             ================================= CRIU CHECK ======================<br>
&gt;             ===========<br>
&gt;             Error (timerfd.c:56): timerfd: No timerfd support for c/r:<br>
&gt;             Inappropriate ioctl for device<br>
&gt;             ============================= WARNING =============================<br>
&gt;             Not all features needed for CRIU are merged to upstream kernel yet,<br>
&gt;             so for now we maintain our own branch which can be cloned from:<br>
&gt;             git://<a href="http://git.kernel.org/pub/scm/linux/kernel/git/gorcunov/linux-cr.git" target="_blank">git.kernel.org/pub/scm/linux/kernel/git/gorcunov/linux-cr.git</a><br>
&gt;             ===================================================================<br>
&gt;             Execute zdtm/live/static/env00<br>
&gt;             ./env00 --pidfile=env00.pid --outfile=env00.out --envname=<br>
&gt;             ENV_00_TEST<br>
</div></div><div><div>&gt;             /root/git/orig/criu/test<br>
&gt;             Dump 30437<br>
&gt;             WARNING: env00 returned 1 and left running for debug needs<br>
&gt;             Test: zdtm/live/static/env00, Result: FAIL<br>
&gt;             ==================================== ERROR ========================<br>
&gt;             ============<br>
&gt;             Test: zdtm/live/static/env00, Namespace: 1<br>
&gt;             Dump log   : /root/git/orig/criu/test/dump/static/env00/30437/1/<br>
&gt;             dump.log<br>
&gt;             --------------------------------- grep Error<br>
&gt;             ---------------------------------<br>
&gt;             (00.035741) Error (files-reg.c:629): Unable to look up the 33 mount<br>
&gt;             (00.035771) Error (cr-dump.c:1665): Dump mappings (pid: 30437)<br>
&gt;             failed with -1<br>
&gt;             (00.036042) Error (cr-dump.c:1953): Dumping FAILED.<br>
&gt;             ------------------------------------- END<br>
&gt;             -------------------------------------<br>
&gt;             ================================= ERROR OVER ======================<br>
&gt;             ===========<br>
&gt;<br>
&gt;             [root@avagin-fc19-cr criu]# cat /proc/30437/mountinfo<br>
&gt;             68 42 0:33 / / rw,relatime - aufs none rw,si=46777ba2737567db<br>
&gt;             43 68 0:35 / /proc rw,relatime - proc proc rw<br>
&gt;             44 68 0:36 / /dev/pts rw,relatime - devpts pts rw,mode=666,ptmxmode<br>
&gt;             =666<br>
&gt;<br>
&gt;             [root@avagin-fc19-cr criu]# cat /proc/self/mountinfo | grep ^33<br>
&gt;             33 0 8:3 / / rw,relatime shared:1 - ext4 /dev/sda3 rw,data=ordered<br>
&gt;<br>
&gt;<br>
&gt;<br>
&gt;             On Sat, Aug 30, 2014 at 02:49:22PM +0400, Andrew Vagin wrote:<br>
&gt;             &gt; And here are a few issues:<br>
&gt;             &gt;<br>
&gt;             &gt; + cat /tmp/criu-root.7GvnTZ/dump/static/cwd01/6970/1/dump.log<br>
&gt;             &gt; (00.012328) Error (cr-dump.c:303): Can&#39;t open 6974/cwd on procfs:<br>
&gt;             Stale file handle<br>
&gt;             &gt; (00.012350) Error (cr-dump.c:1671): Dump fs (pid: 6974) failed<br>
&gt;             with -1<br>
&gt;             &gt; (00.012528) Error (cr-dump.c:1953): Dumping FAILED.<br>
&gt;             &gt;<br>
&gt;             &gt; + cat /tmp/criu-root.HTTKBE/dump/static/unlink_fstat03/14801/1/<br>
&gt;             dump.log<br>
&gt;             &gt; (00.009870) Error (files-reg.c:455): Can&#39;t link remap to /zdtm/<br>
&gt;             live/static/unlink_fstat03.test (deleted): No such file or<br>
&gt;             directory<br>
&gt;             &gt; (00.009906) Error (cr-dump.c:1603): Dump files (pid: 14805)<br>
&gt;             failed with -1<br>
&gt;             &gt; (00.010533) Error (cr-dump.c:1953): Dumping FAILED.<br>
&gt;             &gt;<br>
&gt;             &gt; + cat /tmp/criu-root.HDf6Nn/dump/static/unlink_mmap00/23065/1/<br>
&gt;             dump.log<br>
&gt;             &gt; (00.008996) Error (proc_parse.c:472): Failed stat on 23069&#39;s map<br>
&gt;             47071697305600 (/tmp/criu-root.HDf6Nn/zdtm/live/static/<br>
&gt;             unlink_mmap00.test (deleted)): No such file or directory<br>
&gt;             &gt; (00.009021) Error (cr-dump.c:1515): Collect mappings (pid: 23069)<br>
&gt;             failed with -1<br>
&gt;             &gt; (00.009228) Error (cr-dump.c:1953): Dumping FAILED.<br>
&gt;             &gt;<br>
&gt;             &gt; + cat /tmp/criu-root.XULszc/dump/static/inotify00/30616/1/<br>
&gt;             dump.log<br>
&gt;             &gt; (00.010422) Error (proc_parse.c:1345): parse_fdinfo_pid_s: error<br>
&gt;             parsing [inotify wd:2 ino:1202 sdev:24 mask:800afff ignored_mask:0<br>
&gt;             &gt; (00.010465) Error (cr-dump.c:1603): Dump files (pid: 30620)<br>
&gt;             failed with -1<br>
&gt;             &gt; (00.011124) Error (cr-dump.c:1953): Dumping FAILED.<br>
&gt;             &gt;<br>
&gt;             &gt; We have problems with unlinked files and fsnotifies.<br>
&gt;             &gt;<br>
&gt;             &gt; Cc: Saied Kazemi &lt;<a href="mailto:saied@google.com" target="_blank">saied@google.com</a>&gt;<br>
&gt;             &gt; Signed-off-by: Andrew Vagin &lt;<a href="mailto:avagin@openvz.org" target="_blank">avagin@openvz.org</a>&gt;<br>
&gt;             &gt; ---<br>
&gt;             &gt;  test/zdtm.sh | 11 ++++++++++-<br>
&gt;             &gt;  1 file changed, 10 insertions(+), 1 deletion(-)<br>
&gt;             &gt;<br>
&gt;             &gt; diff --git a/test/zdtm.sh b/test/zdtm.sh<br>
&gt;             &gt; index eee3342..f92a63f 100755<br>
&gt;             &gt; --- a/test/zdtm.sh<br>
&gt;             &gt; +++ b/test/zdtm.sh<br>
&gt;             &gt; @@ -412,7 +412,12 @@ start_test()<br>
&gt;             &gt;                       mkdir -p dump<br>
&gt;             &gt;                       ZDTM_ROOT=`mktemp -d /tmp/criu-root.XXXXXX`<br>
&gt;             &gt;                       ZDTM_ROOT=`readlink -f $ZDTM_ROOT`<br>
&gt;             &gt; -                     mount --make-private --bind . $ZDTM_ROOT ||<br>
&gt;             return 1<br>
&gt;             &gt; +                     if [ -z &quot;$ZDTM_AUFS&quot; ]; then<br>
&gt;             &gt; +                             mount --make-private --bind .<br>
&gt;             $ZDTM_ROOT || return 1<br>
&gt;             &gt; +                     else<br>
&gt;             &gt; +                             mount -t aufs -o br=. none<br>
&gt;             $ZDTM_ROOT || return 1<br>
&gt;             &gt; +                             cd $ZDTM_ROOT<br>
&gt;             &gt; +                     fi<br>
&gt;             &gt;               fi<br>
&gt;             &gt;               construct_root $ZDTM_ROOT $test || return 1<br>
&gt;             &gt;               export ZDTM_NEWNS=1<br>
&gt;             &gt; @@ -901,6 +906,10 @@ while :; do<br>
&gt;             &gt;               }<br>
&gt;             &gt;               shift<br>
&gt;             &gt;               ;;<br>
&gt;             &gt; +     --aufs)<br>
&gt;             &gt; +             export ZDTM_AUFS=1<br>
&gt;             &gt; +             shift<br>
&gt;             &gt; +             ;;<br>
&gt;             &gt;         -*)<br>
&gt;             &gt;               echo &quot;Unrecognized option $1, aborting!&quot; 1&gt;&amp;2<br>
&gt;             &gt;               usage<br>
&gt;             &gt; --<br>
&gt;             &gt; 1.9.3<br>
&gt;             &gt;<br>
&gt;<br>
&gt;<br>
&gt;<br>
&gt;<br>
&gt;<br>
&gt;<br>
</div></div></blockquote></div><br></div></div>