[CRIU] Podman container migration
Adrian Reber
adrian at lisas.de
Thu Jan 24 19:06:46 MSK 2019
On Thu, Jan 24, 2019 at 11:29:36AM +0100, Adrian Reber wrote:
> On Thu, Jan 24, 2019 at 11:20:21AM +0100, Adrian Reber wrote:
> > On Thu, Jan 10, 2019 at 10:08:16PM -0800, Andrei Vagin wrote:
> > > On Wed, Jan 09, 2019 at 03:16:38PM +0100, Adrian Reber wrote:
> > > > I started to look into migrating containers with Podman and I am
> > > > currently stuck at files which are bind-mounted into the container:
> > > >
> > > > If I checkpoint a container and restore it, it works.
> > > >
> > > > If I checkpoint a container and transfer the checkpoint to another
> > > > container it only works if the previous container is running, as
> > > > CRIU fails to mount the file into the container.
> > > >
> > > > Following is from the checkpointed container:
> > > >
> > > > {
> > > > "fstype": 5,
> > > > "mnt_id": 368,
> > > > "root_dev": "0:20",
> > > > "parent_mnt_id": 367,
> > > > "flags": "0x1000006",
> > > > "root": "/containers/storage/overlay-containers/43d02fb390d236ec42c5119b6655be44157e4a61103bc827db3b0718feecc898/userdata/run/secrets",
> > > > "mountpoint": "/run/secrets",
> > > > "source": "tmpfs",
> > > > "options": "size=2899748k,nr_inodes=724937,mode=755",
> > > > "shared_id": 0,
> > > > "master_id": 0,
> > > > "sb_flags": "0x0",
> > > > "ext_key": "/run/secrets"
> > > > },
> > > >
> > > >
> > > > The main problem I see is that the storage location of the checkpointed
> > > > container is part of "root".
> > > >
> > > > Or do I have to make sure that the migrated container is using the same
> > > > paths as the restored container.
> > > >
> > > > I get the following error during restore, if I copy the checkpoint to another
> > > > container.
> > > >
> > > > (00.009274) 1: mnt: Start with 0:/tmp/.criu.mntns.d6UW28
> > > > (00.009279) 1: mnt: Mounting overlay @/tmp/.criu.mntns.d6UW28/12-0000000000/ (0)
> > > > (00.009285) 1: mnt: 367:/tmp/.criu.mntns.d6UW28/12-0000000000/ private 1 shared 0 slave 0
> > > > (00.009295) 1: mnt: Mounting tmpfs @/tmp/.criu.mntns.d6UW28/12-0000000000/run/secrets (0)
> > > > (00.009297) 1: mnt: Bind /var/run/containers/storage/overlay-containers/43d02fb390d236ec42c5119b6655be44157e4a61103bc827db3b0718feecc898/userdata/run/secrets to /tmp/.criu.mntns.d6UW28/12-0000000000/run/secrets
> > > > (00.009326) 1: Error (criu/mount.c:2279): mnt: Can't mount at /tmp/.criu.mntns.d6UW28/12-0000000000/run/secrets: No such file or directory
> > >
> > > runsc creates a destination directory for external mounts:
> > > https://github.com/opencontainers/runc/blob/master/libcontainer/rootfs_linux.go#L278
> > >
> > > when a container is restored, destination directories for external
> > > mounts are not created, because we suppose that the container rootfs
> > > will be restored or migrated too. CRIU and runsc doesn't handle a
> > > container rootfs, so you need to do this in podman.
> >
> > I am doing this now. It still does not work, so I am probably doing it
> > wrong.
> >
> > CRIU has the following information about the container in its
> > mountpoints-*img:
> >
> > {
> > "fstype": 17,
> > "mnt_id": 252,
> > "root_dev": "0:43",
> > "parent_mnt_id": 153,
> > "flags": "0x200000",
> > "root": "/",
> > "mountpoint": "/",
> > "source": "overlay",
> > "options": "lowerdir=/var/lib/containers/storage/overlay/l/2S375XKZQPDN4PUEXDF5O6UOHR:/var/lib/containers/storage/overlay/l/BZ4VT4FMA3V5R3B5OQCCTI4EIK:/var/lib/containers/storage/overlay/l/PPTBJ3ILFMWC2I7GTG4MW5LFJS:/var/lib/containers/storage/overlay/l/7MYJABNPHL5WGKWQZUQZIAZAAK:/var/lib/containers/storage/overlay/l/PR33ERQ7UP22DCFFUD7T22TE2L:/var/lib/containers/storage/overlay/l/3JGMFT6UEEGG3ENP6ZU4A6WVRX:/var/lib/containers/storage/overlay/l/ZOZ7KNBDLJ22DO6YYFSRHX2BXJ:/var/lib/containers/storage/overlay/l/UX2SILYJEP6X63BN3J2KOJD2VI:/var/lib/containers/storage/overlay/l/LA5B6O3237C5XBRVMWX2F353VJ:/var/lib/containers/storage/overlay/l/7UCRZZJIKLTNFMJWKMXUSXY542:/var/lib/containers/storage/overlay/l/VYA4EVBDXX2YFRVI2WQW2YIDWP:/var/lib/containers/storage/overlay/l/LKKTET5Q6LOPVNF4SUMH5DTLFM:/var/lib/containers/storage/overlay/l/7K3UZVA54OYTUGJYW7VOYAZZXP:/var/lib/containers/storage/overlay/l/3OWJO6K4ZJY2UQPDDN5ZXWZBHC,upperdir=/var/lib/containers/storage/overlay!
> /761a5c37a39a0436741a35fed374c8212b37b053ebe826d40096008413960659/diff,workdir=/var/lib/containers/storage/overlay/761a5c37a39a0436741a35fed374c8212b37b053ebe826d40096008413960659/work",
> > "shared_id": 0,
> > "master_id": 0,
> > "sb_flags": "0x0"
> > }
> >
> > If I am restoring the container the options of the overlay filesystem
> > are different. 'upperdir' and 'workdir' changes. 'lowerdir' stays the
> > same. As an ugly hack I am now changing the mountpoint.img before
> > restoring the container. I still see the message:
> >
> > 1: mnt: Mounting overlay @/tmp/.criu.mntns.d6UW28/12-0000000000/ (0)
> >
> > But I do not see anything mounted at /tmp/.criu.mntns.d6UW28/12-0000000000/
> >
> > Is there some mechanism in CRIU with which I can mount another overlay
> > as the root file system? Somehow ignore or skip the one from
> > mountpoints-*.img?
>
> Some more information:
>
> Podman mounts the root file system here:
>
> overlay on /var/lib/containers/storage/overlay/702652b225e27aec3346470b1474adb177db599ff9c200629114a73367c60557/merged type overlay (rw,relatime,lowerdir=/var/lib/containers/storage/overlay/l/2S375XKZQPDN4PUEXDF5O6UOHR:/var/lib/containers/storage/overlay/l/BZ4VT4FMA3V5R3B5OQCCTI4EIK:/var/lib/containers/storage/overlay/l/PPTBJ3ILFMWC2I7GTG4MW5LFJS:/var/lib/containers/storage/overlay/l/7MYJABNPHL5WGKWQZUQZIAZAAK:/var/lib/containers/storage/overlay/l/PR33ERQ7UP22DCFFUD7T22TE2L:/var/lib/containers/storage/overlay/l/3JGMFT6UEEGG3ENP6ZU4A6WVRX:/var/lib/containers/storage/overlay/l/ZOZ7KNBDLJ22DO6YYFSRHX2BXJ:/var/lib/containers/storage/overlay/l/UX2SILYJEP6X63BN3J2KOJD2VI:/var/lib/containers/storage/overlay/l/LA5B6O3237C5XBRVMWX2F353VJ:/var/lib/containers/storage/overlay/l/7UCRZZJIKLTNFMJWKMXUSXY542:/var/lib/containers/storage/overlay/l/VYA4EVBDXX2YFRVI2WQW2YIDWP:/var/lib/containers/storage/overlay/l/LKKTET5Q6LOPVNF4SUMH5DTLFM:/var/lib/containers/storage/overlay/l/7K3UZVA54OYTUGJYW!
> 7VOYAZZXP:/var/lib/containers/storage/overlay/l/3OWJO6K4ZJY2UQPDDN5ZXWZBHC,upperdir=/var/lib/containers/storage/overlay/702652b225e27aec3346470b1474adb177db599ff9c200629114a73367c60557/diff,workdir=/var/lib/containers/storage/overlay/702652b225e27aec3346470b1474adb177db599ff9c200629114a73367c60557/work)
>
> And CRIU during restore does this:
>
> overlay on /run/runc/0aa9817999ecb991ca5f458398dace103c3f142ad6ba60da2d18bf3d1bc2e76e/criu-root type overlay (rw,relatime,lowerdir=/var/lib/containers/storage/overlay/l/2S375XKZQPDN4PUEXDF5O6UOHR:/var/lib/containers/storage/overlay/l/BZ4VT4FMA3V5R3B5OQCCTI4EIK:/var/lib/containers/storage/overlay/l/PPTBJ3ILFMWC2I7GTG4MW5LFJS:/var/lib/containers/storage/overlay/l/7MYJABNPHL5WGKWQZUQZIAZAAK:/var/lib/containers/storage/overlay/l/PR33ERQ7UP22DCFFUD7T22TE2L:/var/lib/containers/storage/overlay/l/3JGMFT6UEEGG3ENP6ZU4A6WVRX:/var/lib/containers/storage/overlay/l/ZOZ7KNBDLJ22DO6YYFSRHX2BXJ:/var/lib/containers/storage/overlay/l/UX2SILYJEP6X63BN3J2KOJD2VI:/var/lib/containers/storage/overlay/l/LA5B6O3237C5XBRVMWX2F353VJ:/var/lib/containers/storage/overlay/l/7UCRZZJIKLTNFMJWKMXUSXY542:/var/lib/containers/storage/overlay/l/VYA4EVBDXX2YFRVI2WQW2YIDWP:/var/lib/containers/storage/overlay/l/LKKTET5Q6LOPVNF4SUMH5DTLFM:/var/lib/containers/storage/overlay/l/7K3UZVA54OYTUGJYW7VOYAZZXP:/var/lib/cont!
> ainers/storage/overlay/l/3OWJO6K4ZJY2UQPDDN5ZXWZBHC,upperdir=/var/lib/containers/storage/overlay/702652b225e27aec3346470b1474adb177db599ff9c200629114a73367c60557/diff,workdir=/var/lib/containers/storage/overlay/702652b225e27aec3346470b1474adb177db599ff9c200629114a73367c60557/work)
>
>
> CRIU still fails with:
>
> (00.014590) 1: mnt: Start with 0:/tmp/.criu.mntns.Eeg3El
> (00.014689) 1: mnt: Start with 0:/tmp/.criu.mntns.Eeg3El
> (00.014693) 1: mnt: Mounting overlay @/tmp/.criu.mntns.Eeg3El/12-0000000000/ (0)
> (00.014701) 1: mnt: 281:/tmp/.criu.mntns.Eeg3El/12-0000000000/ private 1 shared 0 slave 0
> (30.014993) 1: mnt: Mounting tmpfs @/tmp/.criu.mntns.Eeg3El/12-0000000000/etc/resolv.conf (0)
> (30.015011) 1: mnt: Bind /var/run/containers/storage/overlay-containers/0aa9817999ecb991ca5f458398dace103c3f142ad6ba60da2d18bf3d1bc2e76e/userdata/resolv.conf to /tmp/.criu.mntns.Eeg3El/12-0000000000/etc/resolv.conf
> (30.019715) 1: Error (criu/mount.c:2282): mnt: Can't mount at /tmp/.criu.mntns.Eeg3El/12-0000000000: Not a directory
> (30.019739) 1: mnt: Start with 0:/tmp/.criu.mntns.Eeg3El
> (30.042875) Error (criu/mount.c:3280): mnt: Can't remove the directory /tmp/.criu.mntns.Eeg3El: Device or resource busy
> (30.042927) Error (criu/cr-restore.c:2294): Restoring FAILED.
>
> Not sure how the files in /tmp and /runc/runc/.../criu-root are connected.
>
> Any ideas what I am doing wrong here?
Adding some more debug output to CRIU I see, the old error that
it cannot mount:
(00.017252) 1: mnt: Mounting tmpfs @/tmp/.criu.mntns.lDH0Au/12-0000000000/run/.containerenv (0)
(00.017256) 1: mnt: Bind /var/run/containers/storage/overlay-containers/4317a2a4c7816ecc23c8fb694c856f60ef6d2dcc39617924867c84635bf415fe/userdata/.containerenv to /tmp/.criu.mntns.lDH0Au/12-0000000000/run/.containerenv
(00.021660) 1: Error (criu/mount.c:2284): mnt: Can't mount at /tmp/.criu.mntns.lDH0Au/12-0000000000: Not a directory
Running stat() immediately after that error I see the following:
(00.021671) 1: mnt: ID of containing device: [0,2b]
(00.021674) 1: mnt: File type: (00.021676) 1: mnt: directory
(00.021679) 1: mnt: I-node number: 12417025
(00.021681) 1: mnt: Mode: 40755 (octal)
(00.021684) 1: mnt: Link count: 1
(00.021686) 1: mnt: Ownership: UID=0 GID=0
(00.021688) 1: mnt: Preferred I/O block size: 4096 bytes
(00.021690) 1: mnt: File size: 6 bytes
(00.021692) 1: mnt: Blocks allocated: 0
(00.021740) 1: mnt: Last status change: Thu Jan 24 14:11:47 2019
(00.021747) 1: mnt: Last file access: Thu Jan 24 14:11:47 2019
(00.021751) 1: mnt: Last file modification: Thu Jan 24 14:11:47 2019
Not sure why I get the error 'Not a directory' if stat() confirms
that it is a directory. Not sure if mount() complains about the
source or the destination of the mount. The source is not a directory.
That would be correct.
The following is mounted at /tmp/.criu.mntns.lDH0Au/12-0000000000
overlay /tmp/.criu.mntns.lDH0Au/12-0000000000 overlay rw,relatime,lowerdir=/var/lib/containers/storage/overlay/l/2S375XKZQPDN4PUEXDF5O6UOHR:/var/lib/containers/storage/overlay/l/BZ4VT4FMA3V5R3B5OQCCTI4EIK:/var/lib/containers/storage/overlay/l/PPTBJ3ILFMWC2I7GTG4MW5LFJS:/var/lib/containers/storage/overlay/l/7MYJABNPHL5WGKWQZUQZIAZAAK:/var/lib/containers/storage/overlay/l/PR33ERQ7UP22DCFFUD7T22TE2L:/var/lib/containers/storage/overlay/l/3JGMFT6UEEGG3ENP6ZU4A6WVRX:/var/lib/containers/storage/overlay/l/ZOZ7KNBDLJ22DO6YYFSRHX2BXJ:/var/lib/containers/storage/overlay/l/UX2SILYJEP6X63BN3J2KOJD2VI:/var/lib/containers/storage/overlay/l/LA5B6O3237C5XBRVMWX2F353VJ:/var/lib/containers/storage/overlay/l/7UCRZZJIKLTNFMJWKMXUSXY542:/var/lib/containers/storage/overlay/l/VYA4EVBDXX2YFRVI2WQW2YIDWP:/var/lib/containers/storage/overlay/l/LKKTET5Q6LOPVNF4SUMH5DTLFM:/var/lib/containers/storage/overlay/l/7K3UZVA54OYTUGJYW7VOYAZZXP:/var/lib/containers/storage/overlay/l/3OWJO6K4ZJY2UQPDDN5ZXWZBHC,upperdir=/var/lib/containers/storage/overlay/37ed4b95b7f68fba70a0d6f3519cc5bd549a27b45bdd2f751a9022a365fbfb64/diff,workdir=/var/lib/containers/storage/overlay/37ed4b95b7f68fba70a0d6f3519cc5bd549a27b45bdd2f751a9022a365fbfb64/work 0 0
So the overlay mounting works, but the bind mount fails. It seems
that the overlay is mounted, but empty... or wrong. Not sure why...
Adrian
More information about the CRIU
mailing list