[CRIU] Podman container migration

[Andrei Vagin]

On Thu, Jan 24, 2019 at 05:06:46PM +0100, Adrian Reber wrote:
> > CRIU still fails with:
> > 
> > (00.014590)      1: mnt: Start with 0:/tmp/.criu.mntns.Eeg3El
> > (00.014689)      1: mnt: Start with 0:/tmp/.criu.mntns.Eeg3El
> > (00.014693)      1: mnt: 	Mounting overlay @/tmp/.criu.mntns.Eeg3El/12-0000000000/ (0)
> > (00.014701)      1: mnt: 281:/tmp/.criu.mntns.Eeg3El/12-0000000000/ private 1 shared 0 slave 0
> > (30.014993)      1: mnt: 	Mounting tmpfs @/tmp/.criu.mntns.Eeg3El/12-0000000000/etc/resolv.conf (0)
> > (30.015011)      1: mnt: 	Bind /var/run/containers/storage/overlay-containers/0aa9817999ecb991ca5f458398dace103c3f142ad6ba60da2d18bf3d1bc2e76e/userdata/resolv.conf to /tmp/.criu.mntns.Eeg3El/12-0000000000/etc/resolv.conf
> > (30.019715)      1: Error (criu/mount.c:2282): mnt: Can't mount at /tmp/.criu.mntns.Eeg3El/12-0000000000: Not a directory
> > (30.019739)      1: mnt: Start with 0:/tmp/.criu.mntns.Eeg3El
> > (30.042875) Error (criu/mount.c:3280): mnt: Can't remove the directory /tmp/.criu.mntns.Eeg3El: Device or resource busy
> > (30.042927) Error (criu/cr-restore.c:2294): Restoring FAILED.
> > 
> > Not sure how the files in /tmp and /runc/runc/.../criu-root are connected.
> > 
> > Any ideas what I am doing wrong here?

ENOTDIR reported when you try to bind-moyunt a directory on a file or a file on a directory.

https://github.com/torvalds/linux/blob/master/fs/namespace.c#L2080

In your case,
/var/run/containers/storage/overlay-containers/4317a2a4c7816ecc23c8fb694c856f60ef6d2dcc39617924867c84635bf415fe/userdata/.containerenv
is probably a file, but
/tmp/.criu.mntns.lDH0Au/12-0000000000/run/.containerenv is a directory.

Linux allow to bind-mount file on file and directory on directory.

> 
> Adding some more debug output to CRIU I see, the old error that
> it cannot mount:
> 
> (00.017252)      1: mnt: 	Mounting tmpfs @/tmp/.criu.mntns.lDH0Au/12-0000000000/run/.containerenv (0)
> (00.017256)      1: mnt: 	Bind /var/run/containers/storage/overlay-containers/4317a2a4c7816ecc23c8fb694c856f60ef6d2dcc39617924867c84635bf415fe/userdata/.containerenv to /tmp/.criu.mntns.lDH0Au/12-0000000000/run/.containerenv
> (00.021660)      1: Error (criu/mount.c:2284): mnt: Can't mount at /tmp/.criu.mntns.lDH0Au/12-0000000000: Not a directory
> 
> Running stat() immediately after that error I see the following:
> 
> (00.021671)      1: mnt: ID of containing device:  [0,2b]
> (00.021674)      1: mnt: File type:                (00.021676)      1: mnt: directory
> (00.021679)      1: mnt: I-node number:            12417025
> (00.021681)      1: mnt: Mode:                     40755 (octal)
> (00.021684)      1: mnt: Link count:               1
> (00.021686)      1: mnt: Ownership:                UID=0   GID=0
> (00.021688)      1: mnt: Preferred I/O block size: 4096 bytes
> (00.021690)      1: mnt: File size:                6 bytes
> (00.021692)      1: mnt: Blocks allocated:         0
> (00.021740)      1: mnt: Last status change:       Thu Jan 24 14:11:47 2019
> (00.021747)      1: mnt: Last file access:         Thu Jan 24 14:11:47 2019
> (00.021751)      1: mnt: Last file modification:   Thu Jan 24 14:11:47 2019
> 
> Not sure why I get the error 'Not a directory' if stat() confirms
> that it is a directory. Not sure if mount() complains about the
> source or the destination of the mount. The source is not a directory.
> That would be correct.
> 
> The following is mounted at /tmp/.criu.mntns.lDH0Au/12-0000000000
> 
> overlay /tmp/.criu.mntns.lDH0Au/12-0000000000 overlay rw,relatime,lowerdir=/var/lib/containers/storage/overlay/l/2S375XKZQPDN4PUEXDF5O6UOHR:/var/lib/containers/storage/overlay/l/BZ4VT4FMA3V5R3B5OQCCTI4EIK:/var/lib/containers/storage/overlay/l/PPTBJ3ILFMWC2I7GTG4MW5LFJS:/var/lib/containers/storage/overlay/l/7MYJABNPHL5WGKWQZUQZIAZAAK:/var/lib/containers/storage/overlay/l/PR33ERQ7UP22DCFFUD7T22TE2L:/var/lib/containers/storage/overlay/l/3JGMFT6UEEGG3ENP6ZU4A6WVRX:/var/lib/containers/storage/overlay/l/ZOZ7KNBDLJ22DO6YYFSRHX2BXJ:/var/lib/containers/storage/overlay/l/UX2SILYJEP6X63BN3J2KOJD2VI:/var/lib/containers/storage/overlay/l/LA5B6O3237C5XBRVMWX2F353VJ:/var/lib/containers/storage/overlay/l/7UCRZZJIKLTNFMJWKMXUSXY542:/var/lib/containers/storage/overlay/l/VYA4EVBDXX2YFRVI2WQW2YIDWP:/var/lib/containers/storage/overlay/l/LKKTET5Q6LOPVNF4SUMH5DTLFM:/var/lib/containers/storage/overlay/l/7K3UZVA54OYTUGJYW7VOYAZZXP:/var/lib/containers/storage/overlay/l/3OWJO6K4ZJY2UQPDDN5ZXWZBHC,upperdir=/var/lib/containers/storage/overlay/37ed4b95b7f68fba70a0d6f3519cc5bd549a27b45bdd2f751a9022a365fbfb64/diff,workdir=/var/lib/containers/storage/overlay/37ed4b95b7f68fba70a0d6f3519cc5bd549a27b45bdd2f751a9022a365fbfb64/work 0 0
> 
> So the overlay mounting works, but the bind mount fails. It seems
> that the overlay is mounted, but empty... or wrong. Not sure why...
> 
> 		Adrian