[CRIU] Podman container migration
Adrian Reber
adrian at lisas.de
Thu Jan 24 13:29:36 MSK 2019
On Thu, Jan 24, 2019 at 11:20:21AM +0100, Adrian Reber wrote:
> On Thu, Jan 10, 2019 at 10:08:16PM -0800, Andrei Vagin wrote:
> > On Wed, Jan 09, 2019 at 03:16:38PM +0100, Adrian Reber wrote:
> > > I started to look into migrating containers with Podman and I am
> > > currently stuck at files which are bind-mounted into the container:
> > >
> > > If I checkpoint a container and restore it, it works.
> > >
> > > If I checkpoint a container and transfer the checkpoint to another
> > > container it only works if the previous container is running, as
> > > CRIU fails to mount the file into the container.
> > >
> > > Following is from the checkpointed container:
> > >
> > > {
> > > "fstype": 5,
> > > "mnt_id": 368,
> > > "root_dev": "0:20",
> > > "parent_mnt_id": 367,
> > > "flags": "0x1000006",
> > > "root": "/containers/storage/overlay-containers/43d02fb390d236ec42c5119b6655be44157e4a61103bc827db3b0718feecc898/userdata/run/secrets",
> > > "mountpoint": "/run/secrets",
> > > "source": "tmpfs",
> > > "options": "size=2899748k,nr_inodes=724937,mode=755",
> > > "shared_id": 0,
> > > "master_id": 0,
> > > "sb_flags": "0x0",
> > > "ext_key": "/run/secrets"
> > > },
> > >
> > >
> > > The main problem I see is that the storage location of the checkpointed
> > > container is part of "root".
> > >
> > > Or do I have to make sure that the migrated container is using the same
> > > paths as the restored container.
> > >
> > > I get the following error during restore, if I copy the checkpoint to another
> > > container.
> > >
> > > (00.009274) 1: mnt: Start with 0:/tmp/.criu.mntns.d6UW28
> > > (00.009279) 1: mnt: Mounting overlay @/tmp/.criu.mntns.d6UW28/12-0000000000/ (0)
> > > (00.009285) 1: mnt: 367:/tmp/.criu.mntns.d6UW28/12-0000000000/ private 1 shared 0 slave 0
> > > (00.009295) 1: mnt: Mounting tmpfs @/tmp/.criu.mntns.d6UW28/12-0000000000/run/secrets (0)
> > > (00.009297) 1: mnt: Bind /var/run/containers/storage/overlay-containers/43d02fb390d236ec42c5119b6655be44157e4a61103bc827db3b0718feecc898/userdata/run/secrets to /tmp/.criu.mntns.d6UW28/12-0000000000/run/secrets
> > > (00.009326) 1: Error (criu/mount.c:2279): mnt: Can't mount at /tmp/.criu.mntns.d6UW28/12-0000000000/run/secrets: No such file or directory
> >
> > runsc creates a destination directory for external mounts:
> > https://github.com/opencontainers/runc/blob/master/libcontainer/rootfs_linux.go#L278
> >
> > when a container is restored, destination directories for external
> > mounts are not created, because we suppose that the container rootfs
> > will be restored or migrated too. CRIU and runsc doesn't handle a
> > container rootfs, so you need to do this in podman.
>
> I am doing this now. It still does not work, so I am probably doing it
> wrong.
>
> CRIU has the following information about the container in its
> mountpoints-*img:
>
> {
> "fstype": 17,
> "mnt_id": 252,
> "root_dev": "0:43",
> "parent_mnt_id": 153,
> "flags": "0x200000",
> "root": "/",
> "mountpoint": "/",
> "source": "overlay",
> "options": "lowerdir=/var/lib/containers/storage/overlay/l/2S375XKZQPDN4PUEXDF5O6UOHR:/var/lib/containers/storage/overlay/l/BZ4VT4FMA3V5R3B5OQCCTI4EIK:/var/lib/containers/storage/overlay/l/PPTBJ3ILFMWC2I7GTG4MW5LFJS:/var/lib/containers/storage/overlay/l/7MYJABNPHL5WGKWQZUQZIAZAAK:/var/lib/containers/storage/overlay/l/PR33ERQ7UP22DCFFUD7T22TE2L:/var/lib/containers/storage/overlay/l/3JGMFT6UEEGG3ENP6ZU4A6WVRX:/var/lib/containers/storage/overlay/l/ZOZ7KNBDLJ22DO6YYFSRHX2BXJ:/var/lib/containers/storage/overlay/l/UX2SILYJEP6X63BN3J2KOJD2VI:/var/lib/containers/storage/overlay/l/LA5B6O3237C5XBRVMWX2F353VJ:/var/lib/containers/storage/overlay/l/7UCRZZJIKLTNFMJWKMXUSXY542:/var/lib/containers/storage/overlay/l/VYA4EVBDXX2YFRVI2WQW2YIDWP:/var/lib/containers/storage/overlay/l/LKKTET5Q6LOPVNF4SUMH5DTLFM:/var/lib/containers/storage/overlay/l/7K3UZVA54OYTUGJYW7VOYAZZXP:/var/lib/containers/storage/overlay/l/3OWJO6K4ZJY2UQPDDN5ZXWZBHC,upperdir=/var/lib/containers/storage/overlay/761a5c37a39a0436741a35fed374c8212b37b053ebe826d40096008413960659/diff,workdir=/var/lib/containers/storage/overlay/761a5c37a39a0436741a35fed374c8212b37b053ebe826d40096008413960659/work",
> "shared_id": 0,
> "master_id": 0,
> "sb_flags": "0x0"
> }
>
> If I am restoring the container the options of the overlay filesystem
> are different. 'upperdir' and 'workdir' changes. 'lowerdir' stays the
> same. As an ugly hack I am now changing the mountpoint.img before
> restoring the container. I still see the message:
>
> 1: mnt: Mounting overlay @/tmp/.criu.mntns.d6UW28/12-0000000000/ (0)
>
> But I do not see anything mounted at /tmp/.criu.mntns.d6UW28/12-0000000000/
>
> Is there some mechanism in CRIU with which I can mount another overlay
> as the root file system? Somehow ignore or skip the one from
> mountpoints-*.img?
Some more information:
Podman mounts the root file system here:
overlay on /var/lib/containers/storage/overlay/702652b225e27aec3346470b1474adb177db599ff9c200629114a73367c60557/merged type overlay (rw,relatime,lowerdir=/var/lib/containers/storage/overlay/l/2S375XKZQPDN4PUEXDF5O6UOHR:/var/lib/containers/storage/overlay/l/BZ4VT4FMA3V5R3B5OQCCTI4EIK:/var/lib/containers/storage/overlay/l/PPTBJ3ILFMWC2I7GTG4MW5LFJS:/var/lib/containers/storage/overlay/l/7MYJABNPHL5WGKWQZUQZIAZAAK:/var/lib/containers/storage/overlay/l/PR33ERQ7UP22DCFFUD7T22TE2L:/var/lib/containers/storage/overlay/l/3JGMFT6UEEGG3ENP6ZU4A6WVRX:/var/lib/containers/storage/overlay/l/ZOZ7KNBDLJ22DO6YYFSRHX2BXJ:/var/lib/containers/storage/overlay/l/UX2SILYJEP6X63BN3J2KOJD2VI:/var/lib/containers/storage/overlay/l/LA5B6O3237C5XBRVMWX2F353VJ:/var/lib/containers/storage/overlay/l/7UCRZZJIKLTNFMJWKMXUSXY542:/var/lib/containers/storage/overlay/l/VYA4EVBDXX2YFRVI2WQW2YIDWP:/var/lib/containers/storage/overlay/l/LKKTET5Q6LOPVNF4SUMH5DTLFM:/var/lib/containers/storage/overlay/l/7K3UZVA54OYTUGJYW7VOYAZZXP:/var/lib/containers/storage/overlay/l/3OWJO6K4ZJY2UQPDDN5ZXWZBHC,upperdir=/var/lib/containers/storage/overlay/702652b225e27aec3346470b1474adb177db599ff9c200629114a73367c60557/diff,workdir=/var/lib/containers/storage/overlay/702652b225e27aec3346470b1474adb177db599ff9c200629114a73367c60557/work)
And CRIU during restore does this:
overlay on /run/runc/0aa9817999ecb991ca5f458398dace103c3f142ad6ba60da2d18bf3d1bc2e76e/criu-root type overlay (rw,relatime,lowerdir=/var/lib/containers/storage/overlay/l/2S375XKZQPDN4PUEXDF5O6UOHR:/var/lib/containers/storage/overlay/l/BZ4VT4FMA3V5R3B5OQCCTI4EIK:/var/lib/containers/storage/overlay/l/PPTBJ3ILFMWC2I7GTG4MW5LFJS:/var/lib/containers/storage/overlay/l/7MYJABNPHL5WGKWQZUQZIAZAAK:/var/lib/containers/storage/overlay/l/PR33ERQ7UP22DCFFUD7T22TE2L:/var/lib/containers/storage/overlay/l/3JGMFT6UEEGG3ENP6ZU4A6WVRX:/var/lib/containers/storage/overlay/l/ZOZ7KNBDLJ22DO6YYFSRHX2BXJ:/var/lib/containers/storage/overlay/l/UX2SILYJEP6X63BN3J2KOJD2VI:/var/lib/containers/storage/overlay/l/LA5B6O3237C5XBRVMWX2F353VJ:/var/lib/containers/storage/overlay/l/7UCRZZJIKLTNFMJWKMXUSXY542:/var/lib/containers/storage/overlay/l/VYA4EVBDXX2YFRVI2WQW2YIDWP:/var/lib/containers/storage/overlay/l/LKKTET5Q6LOPVNF4SUMH5DTLFM:/var/lib/containers/storage/overlay/l/7K3UZVA54OYTUGJYW7VOYAZZXP:/var/lib/containers/storage/overlay/l/3OWJO6K4ZJY2UQPDDN5ZXWZBHC,upperdir=/var/lib/containers/storage/overlay/702652b225e27aec3346470b1474adb177db599ff9c200629114a73367c60557/diff,workdir=/var/lib/containers/storage/overlay/702652b225e27aec3346470b1474adb177db599ff9c200629114a73367c60557/work)
CRIU still fails with:
(00.014590) 1: mnt: Start with 0:/tmp/.criu.mntns.Eeg3El
(00.014689) 1: mnt: Start with 0:/tmp/.criu.mntns.Eeg3El
(00.014693) 1: mnt: Mounting overlay @/tmp/.criu.mntns.Eeg3El/12-0000000000/ (0)
(00.014701) 1: mnt: 281:/tmp/.criu.mntns.Eeg3El/12-0000000000/ private 1 shared 0 slave 0
(30.014993) 1: mnt: Mounting tmpfs @/tmp/.criu.mntns.Eeg3El/12-0000000000/etc/resolv.conf (0)
(30.015011) 1: mnt: Bind /var/run/containers/storage/overlay-containers/0aa9817999ecb991ca5f458398dace103c3f142ad6ba60da2d18bf3d1bc2e76e/userdata/resolv.conf to /tmp/.criu.mntns.Eeg3El/12-0000000000/etc/resolv.conf
(30.019715) 1: Error (criu/mount.c:2282): mnt: Can't mount at /tmp/.criu.mntns.Eeg3El/12-0000000000: Not a directory
(30.019739) 1: mnt: Start with 0:/tmp/.criu.mntns.Eeg3El
(30.042875) Error (criu/mount.c:3280): mnt: Can't remove the directory /tmp/.criu.mntns.Eeg3El: Device or resource busy
(30.042927) Error (criu/cr-restore.c:2294): Restoring FAILED.
Not sure how the files in /tmp and /runc/runc/.../criu-root are connected.
Any ideas what I am doing wrong here?
Adrian
More information about the CRIU
mailing list