[CRIU] Restoring lxc-1.1.5 centos 7 container with httpd fails

[Adrian Reber]

On Fri, May 06, 2016 at 01:04:47PM -0700, Andrey Vagin wrote:
> On Fri, May 06, 2016 at 02:09:47PM +0200, Adrian Reber wrote:
> > On Thu, May 05, 2016 at 06:15:39PM -0700, Andrey Vagin wrote:
> > > On Thu, May 05, 2016 at 02:03:17PM +0300, Pavel Emelyanov wrote:
> > > > On 05/04/2016 03:41 PM, Adrian Reber wrote:
> > > > > I am able to restore a lxc-1.1.5 container running centos 7. But if I
> > > > > start apache's httpd in the container I am no longer able to restore the
> > > > > container. I get:
> > > > > 
> > > > > (00.028105)      1: mnt: 148:./ private 0 shared 0 slave 1
> > > > > (00.028206)      1: mnt: Start with 148:./
> > > > > (00.028209)      1: mnt: 	Mounting unsupported @./ (0)
> > > > > (00.028212)      1: mnt: 	Mounting unsupported @.criu.mntns.8pI85T/13/ (0)
> > > > > (00.028215)      1: mnt: 	Bind /tmp/cr-tmpfs.JF8cRh/ to .criu.mntns.8pI85T/13/
> > > > > (00.028218)      1: mnt: 93:.criu.mntns.8pI85T/13/ private 0 shared 1 slave 0
> > > > > (00.028225)      1: mnt: 	Mounting tmpfs @.criu.mntns.8pI85T/13/dev (0)
> > > > > (00.030633)      1: mnt: 94:.criu.mntns.8pI85T/13/dev private 0 shared 1 slave 0
> > > > > (00.030645)      1: mnt: 	Mounting devpts @.criu.mntns.8pI85T/13/dev/pts (0)
> > > > > (00.030725)      1: mnt: 95:.criu.mntns.8pI85T/13/dev/pts private 0 shared 1 slave 0
> > > > > (00.030731)      1: mnt: 	Mounting hugetlbfs @.criu.mntns.8pI85T/13/dev/hugepages (0)
> > > > > (00.030753)      1: mnt: 98:.criu.mntns.8pI85T/13/dev/hugepages private 0 shared 1 slave 0
> > > > > (00.030758)      1: mnt: 	Mounting mqueue @.criu.mntns.8pI85T/13/dev/mqueue (0)
> > > > > (00.030769)      1: mnt: 97:.criu.mntns.8pI85T/13/dev/mqueue private 0 shared 1 slave 0
> > > > > (00.030773)      1: mnt: 	Mounting tmpfs @.criu.mntns.8pI85T/13/dev/shm (0)
> > > > > (00.033125)      1: mnt: 96:.criu.mntns.8pI85T/13/dev/shm private 0 shared 1 slave 0
> > > > > (00.033140)      1: mnt: 	Mounting unsupported @.criu.mntns.8pI85T/13/var/tmp (0)
> > > > > (00.033143)      1: mnt: 	Bind //var/lib/lxc/c7/rootfs/var/tmp/systemd-private-953f46ab8f04414db2ca1b71ed6212c2-httpd.service-yl6pWp/tmp to .criu.mntns.8pI85T/13/var/tmp
> > > > > (00.033165)      1: Error (mount.c:2464): mnt: Can't mount at .criu.mntns.8pI85T/13/var/tmp: No such file or directory
> > > > > (00.033542) Error (cr-restore.c:1384): 18067 exited, status=1
> > > > > (00.033610) Switching to new ns to clean ghosts
> > > > > (00.033743) Error (cr-restore.c:2223): Restoring FAILED.
> > > > > 
> > > > > Same error with today's master and criu-dev branch. Full log at:
> > > > > 
> > > > > https://lisas.de/~adrian/restore.log
> > > > > 
> > > > > The host is centos 7 as well. And ideas?
> > > > 
> > > > Can you try recent patches from Andrey:
> > > > mount: remove an extra condition from mounts_equal()
> > > > mount: handle a case when parent and child mounts in the same directory
> > > > 
> > > > ?
> > > >
> > > 
> > > I think these patches don't affect this issue.
> > 
> > Those patches did not help. The following is all with those patches
> > applied.
> > 
> > > We've discussed with Adrian in irc and he promised to give more
> > > info about this issue.
> > > 
> > > To investiage this sort of bugs I add sleep(1000) after pr_err() to
> > > freeze processes in a moment of the error and try to find what is wrong
> > > here via /proc/PID/root.
> > 
> > With the sleep after the last pr_err() I see two criu processes:
> > 
> > # ls -la /proc/10183/root
> > lrwxrwxrwx. 1 root root 0 May  6 08:47 /proc/10183/root -> /
> > # ls -la /proc/10188/root
> > lrwxrwxrwx. 1 root root 0 May  6 08:47 /proc/10188/root -> /
> 
> I mean that you need to try to resolve source and target argumnets of a
> mount syscall which returns an error.

I am not really sure what to do. The current restore fails with:

(00.080566)      1: mnt: 	Bind /tmp/cr-tmpfs.KD4sxa/hugetlb to .criu.mntns.07hSc4/13/sys/fs/cgroup/hugetlb
(00.080585)      1: Error (mount.c:2479): mnt: Can't mount at .criu.mntns.07hSc4/13/sys/fs/cgroup/hugetlb: No such file or directory

The directory /proc/4745/root/var/lib/lxc/c7/rootfs/.criu.mntns.07hSc4
is empty. So that seems to explain why the mount is not working.

The other directory exists:

# ls  /proc/4745/root/tmp/cr-tmpfs.KD4sxa/hugetlb/
lxc

So is the first empty directory the problem? As the path contains
'mntns'... Does this need some special mount namespace support. This is
running on the CentOS 7 kernel. Which might be missing some features.


		Adrian