<div dir="ltr"><div>I have edited mount.c. Patch is attached with the mail.<br><br><span class="im"><span style="font-size:13px">1. Where really is your container's root? Is it node's / or is it /usr/local/lib/lxc/rootfs/L</span><br style="font-size:13px"><span style="font-size:13px"> or w/o "L" at the end of previous path? Or where? And how does it get there -- with the</span><br style="font-size:13px"><span style="font-size:13px"> chroot() call or with the pivot_root() one?</span><br style="font-size:13px"><br></span>----- host root: "/" container's root: "/sdcard/L/rootfs" -> for pivot root it is mounted at "/usr/local/lib/lxc/rootfs"<span class="im"><br><br style="font-size:13px"><span style="font-size:13px">2. How did you manage to get several rootfs mountpoints inside container? Do you use</span><br style="font-size:13px"><span style="font-size:13px"> somehow patched kernel?</span></span><br>not sure. may be host's mounpoints are visible inside the container? There was a discussion about this a while back on CRIU<br><a href="https://lists.linuxcontainers.org/pipermail/lxc-devel/2014-October/010571.html" target="_blank">https://lists.linuxcontainers.org/pipermail/lxc-devel/2014-October/010571.html</a><br><br></div><div><br>/sdcard # ls -ld /proc/self/root/ <br>drwxrwxr-x 15 1001 1001 340 Feb 27 13:15 /proc/self/root/<br><br>/sdcard # lxc-attach -n L -- ls -ld /proc/self/root/ <br>drwxrwxr-x 15 1001 1001 340 Feb 27 13:15 /proc/self/root/</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Sat, Feb 28, 2015 at 2:13 AM, Pavel Emelyanov <span dir="ltr"><<a href="mailto:xemul@parallels.com" target="_blank">xemul@parallels.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">On 02/27/2015 11:09 PM, beproject criu wrote:<br>
> Is this ok?<br>
<br>
Yes, that's OK and this ... is even more strange than I expected %)<br>
But anyway, let's try to debug that.<br>
<span class=""><br>
> /sdcard # lxc-attach -n L -- cat /proc/self/mountinfo<br>
> 16 16 0:2 / / rw - rootfs rootfs rw,size=372900k,nr_inodes=93225<br>
> 17 16 0:4 / /proc rw,relatime - proc proc rw<br>
> 18 16 0:11 / /sys rw,relatime - sysfs sysfs rw<br>
> 19 18 0:12 / /sys/fs/cgroup rw,relatime - cgroup none rw,cpuset,debug,cpu,cpuacct,memory,devices,freezer,blkio,perf_event,clone_children<br>
> 20 18 0:6 / /sys/kernel/debug rw,relatime - debugfs none rw<br>
> 21 16 0:10 / /dev/pts rw,relatime - devpts devpts rw,mode=600,ptmxmode=000<br>
> 22 16 179:0 / /sdcard rw,relatime - ext2 /dev/mmcblk0 rw<br>
> 24 16 0:2 /usr/local/lib/lxc/rootfs /usr/local/lib/lxc/rootfs rw - rootfs rootfs rw,size=372900k,nr_inodes=93225<br>
> 25 24 0:14 / /usr/local/lib/lxc/rootfs rw,relatime - tmpfs none rw,size=12k,mode=755<br>
> 35 25 179:0 /L/rootfs /usr/local/lib/lxc/rootfs/root rw,relatime shared:1 - ext2 /dev/mmcblk0 rw<br>
> 36 35 0:13 / /usr/local/lib/lxc/rootfs/root/proc rw,relatime shared:2 - proc none rw<br>
> 37 35 0:15 / /usr/local/lib/lxc/rootfs/root/sys rw,relatime shared:3 - sysfs none rw<br>
> 38 35 0:2 /dev /usr/local/lib/lxc/rootfs/root/dev rw,relatime shared:6 - rootfs rootfs rw,size=372900k,nr_inodes=93225<br>
> 39 38 0:10 /1 /usr/local/lib/lxc/rootfs/root/dev/console rw,relatime shared:7 - devpts devpts rw,mode=600,ptmxmode=000<br>
> 26 37 0:6 / /usr/local/lib/lxc/rootfs/root/sys/kernel/debug rw,relatime shared:4 - debugfs debugfs rw<br>
> 27 37 0:16 / /usr/local/lib/lxc/rootfs/root/sys/fs/cgroup rw,relatime shared:5 - tmpfs none rw,mode=750,gid=1000<br>
> 28 35 0:17 / /usr/local/lib/lxc/rootfs/root/mnt/asec rw,relatime shared:8 - tmpfs tmpfs rw,mode=755,gid=1000<br>
> 29 35 0:18 / /usr/local/lib/lxc/rootfs/root/mnt/obb rw,relatime shared:9 - tmpfs tmpfs rw,mode=755,gid=1000<br>
> 30 35 31:0 / /usr/local/lib/lxc/rootfs/root/system ro,relatime shared:10 - ext4 /dev/block/mtdblock0 ro,data=ordered<br>
> 31 35 31:1 / /usr/local/lib/lxc/rootfs/root/data rw,nosuid,nodev,noatime shared:11 - ext4 /dev/block/mtdblock1 rw,data=ordered<br>
> 32 35 31:2 / /usr/local/lib/lxc/rootfs/root/cache rw,nosuid,nodev,noatime shared:12 - ext4 /dev/block/mtdblock2 rw,data=ordered<br>
<br>
</span>OK, so if we put all bits together I see that some process inside container has a file<br>
named /usr/local/lib/lxc/rootfs/root/dev/__properties__ opened and this file is really<br>
present inside container. According to the mountpoints this path should exist.<br>
<br>
Next, CRIU tries to stat this file relative to what it thinks is your container's root<br>
(the 1017-th descriptor). And gets the ENOENT error, which in turn means, that the<br>
1017 points not to container's root.<br>
<br>
Need to find out why. But things that I don't understand and need your help (since I<br>
don't have your box's console at hands) are:<br>
<br>
1. Where really is your container's root? Is it node's / or is it /usr/local/lib/lxc/rootfs/L<br>
or w/o "L" at the end of previous path? Or where? And how does it get there -- with the<br>
chroot() call or with the pivot_root() one?<br>
<br>
2. How did you manage to get several rootfs mountpoints inside container? Do you use<br>
somehow patched kernel?<br>
<br>
3. In theory this fd should point to container's namespace's root, which _should_<br>
be invisible in criu's namespace and, thus, have the path "/". But it's not such.<br>
What if you find your container's root task as seen from host and do this:<br>
<br>
ls -ld /proc/$the-pid-in-question/root<br>
<br>
? What path would you see?<br>
<br>
Thanks,<br>
Pavel<br>
<span class="im HOEnZb"><br>
> /sdcard # cat /proc/self/mountinfo<br>
> 1 1 0:2 / / rw - rootfs rootfs rw,size=372900k,nr_inodes=93225<br>
> 10 1 0:4 / /proc rw,relatime - proc proc rw<br>
> 11 1 0:11 / /sys rw,relatime - sysfs sysfs rw<br>
> 12 11 0:12 / /sys/fs/cgroup rw,relatime - cgroup none rw,cpuset,debug,cpu,cpuacct,memory,devices,freezer,blkio,perf_event,clone_children<br>
> 13 11 0:6 / /sys/kernel/debug rw,relatime - debugfs none rw<br>
> 14 1 0:10 / /dev/pts rw,relatime - devpts devpts rw,mode=600,ptmxmode=000<br>
> 15 1 179:0 / /sdcard rw,relatime - ext2 /dev/mmcblk0 rw<br>
> /sdcard #<br>
><br>
><br>
> I have this line in my lxc.conf<br>
> lxc.mount.entry=/dev /sdcard/L/rootfs/dev none defaults,bind 0 0<br>
><br>
><br>
> Thanks.<br>
><br>
</span><span class="im HOEnZb">> On Sat, Feb 28, 2015 at 1:20 AM, Pavel Emelyanov <<a href="mailto:xemul@parallels.com">xemul@parallels.com</a> <mailto:<a href="mailto:xemul@parallels.com">xemul@parallels.com</a>>> wrote:<br>
><br>
> On 02/27/2015 08:05 PM, beproject criu wrote:<br>
> > path [/usr/local/lib/lxc/rootfs/root] read_fd_link() return : 30<br>
> > I have mounted the /dev of host on /dev inside my container.<br>
><br>
> That's interesting. How did you do it? Can you show me the mount namespaces<br>
> layout on your node and inside the container? It's just /proc/$pid/mountinfo<br>
> file contents on some process on host and some process in CT.<br>
><br>
> > (03.347637) Dumping path for 4 fd via self 43 [/usr/local/lib/lxc/rootfs/root/dev/__properties__]<br>
><br>
> OK, this is quite strange. The /usr/local/lib/lxc/rootfs is the path on HOST.<br>
> How did this path become visible inside container?<br>
><br>
> > (03.347847) [nyc_fd] The required root is already opened. get_service_fd() returns : 1017<br>
> > (03.348197) [nyc_fd] Going into fstatat()-> [mntns_root : 1017],[rpath : ./usr/local/lib/lxc/rootfs/root/dev/__properties__]<br>
> > (03.348393) [nyc_fd] Out of fstatat()-> [ret : -1],[rpath : ./usr/local/lib/lxc/rootfs/root/dev/__properties__]<br>
> > (03.348583) [nyc_fd] Going into dump_linked_remap()<br>
> > (03.349177) [nyc_fd] The required root is already opened. get_service_fd() returns : 1017<br>
> > (03.349432) [nyc_fd] Doing linkat() [mntns_root : 1017],[link_name : ./usr/local/lib/lxc/rootfs/root/dev/link_remap.4]<br>
> ><br>
> > (03.349756) [nyc_fd] bad_path [/usr/local/lib/lxc/rootfs/root] ,read_fd_link() return : 30<br>
> ><br>
> > (03.350027) Error (files-reg.c:515): Can't link remap to /usr/local/lib/lxc/rootfs/root/dev/__properties__: No such file or directory<br>
> > (03.351097) [nyc_fd] Error in dump_one_reg_file->check_path_remap()<br>
> > (03.351266) [nyc_fd] Going into dump_task_files_seized() : dump_one_file()<br>
> ><br>
> ><br>
</span><div class="HOEnZb"><div class="h5">> > On Fri, Feb 27, 2015 at 8:10 PM, Pavel Emelyanov <<a href="mailto:xemul@parallels.com">xemul@parallels.com</a> <mailto:<a href="mailto:xemul@parallels.com">xemul@parallels.com</a>> <mailto:<a href="mailto:xemul@parallels.com">xemul@parallels.com</a> <mailto:<a href="mailto:xemul@parallels.com">xemul@parallels.com</a>>>> wrote:<br>
> ><br>
> > On 02/27/2015 05:25 PM, beproject criu wrote:<br>
> > > This is the flow before error,could you get what's wrong :<br>
> > ><br>
> > > (02.352922) Dumping path for 4 fd via self 43 [/usr/local/lib/lxc/rootfs/root/dev/__properties__]<br>
> > > (02.353079) [nyc_fd] The required root is already opened. get_service_fd() returns : 1017<br>
> > > (02.353216) [nyc_fd] Going into fstatat()-> [mntns_root : 1017],[rpath : ./usr/local/lib/lxc/rootfs/root/dev/__properties__]<br>
> > > (02.353398) [nyc_fd] Out of fstatat()-> [ret : -1],[rpath : ./usr/local/lib/lxc/rootfs/root/dev/__properties__]<br>
> > > (02.353574) [nyc_fd] Going into dump_linked_remap()<br>
> > > (02.354191) [nyc_fd] The required root is already opened. get_service_fd() returns : 1017<br>
> > > (02.354431) [nyc_fd] Doing linkat() [mntns_root : 1017],[link_name : ./usr/local/lib/lxc/rootfs/root/dev/link_remap.4]<br>
> > > (02.354747) Error (files-reg.c:510): Can't link remap to /usr/local/lib/lxc/rootfs/root/dev/__properties__: No such file or directory<br>
> > > (02.355915) [nyc_fd] Error in dump_one_reg_file->check_path_remap()<br>
> > > (02.356066) [nyc_fd] Going into dump_task_files_seized() : dump_one_file()<br>
> ><br>
> > OK, so in both cases we access file af fd 1017 and sub-path "./user/local/lib/lxc/roootfs/root/dev/__properties__"<br>
> > and you tell that this file actually exists in container, right?<br>
> ><br>
> > The failing fstatat then means, that the 1017 descriptor points to some bad path. Can you check where? We<br>
> > have a helper called read_fd_link() helper for that, see the fsnotify.c line 455 for code example.<br>
> ><br>
> > And one more question -- does your container live in another mount namespace, or shared one with host?<br>
> ><br>
> > Thanks,<br>
> > Pavel<br>
> ><br>
> ><br>
><br>
><br>
<br>
</div></div></blockquote></div><br></div>