[CRIU] Podman container migration

[Adrian Reber]

On Fri, Feb 01, 2019 at 09:47:10AM -0800, Andrei Vagin wrote:
> On Mon, Jan 28, 2019 at 06:52:38PM -0800, Andrei Vagin wrote:
> > On Thu, Jan 24, 2019 at 11:20:21AM +0100, Adrian Reber wrote:
> > > On Thu, Jan 10, 2019 at 10:08:16PM -0800, Andrei Vagin wrote:
> > > > On Wed, Jan 09, 2019 at 03:16:38PM +0100, Adrian Reber wrote:
> > > > > I started to look into migrating containers with Podman and I am
> > > > > currently stuck at files which are bind-mounted into the container:
> > > > > 
> > > > > If I checkpoint a container and restore it, it works.
> > > > > 
> > > > > If I checkpoint a container and transfer the checkpoint to another
> > > > > container it only works if the previous container is running, as
> > > > > CRIU fails to mount the file into the container.
> > > > > 
> > > > > Following is from the checkpointed container:
> > > > > 
> > > > >         {
> > > > >             "fstype": 5, 
> > > > >             "mnt_id": 368, 
> > > > >             "root_dev": "0:20", 
> > > > >             "parent_mnt_id": 367, 
> > > > >             "flags": "0x1000006", 
> > > > >             "root": "/containers/storage/overlay-containers/43d02fb390d236ec42c5119b6655be44157e4a61103bc827db3b0718feecc898/userdata/run/secrets", 
> > > > >             "mountpoint": "/run/secrets", 
> > > > >             "source": "tmpfs", 
> > > > >             "options": "size=2899748k,nr_inodes=724937,mode=755", 
> > > > >             "shared_id": 0, 
> > > > >             "master_id": 0, 
> > > > >             "sb_flags": "0x0", 
> > > > >             "ext_key": "/run/secrets"
> > > > >         }, 
> > > > > 
> > > > > 
> > > > > The main problem I see is that the storage location of the checkpointed
> > > > > container is part of "root".
> > > > > 
> > > > > Or do I have to make sure that the migrated container is using the same
> > > > > paths as the restored container.
> > > > > 
> > > > > I get the following error during restore, if I copy the checkpoint to another
> > > > > container.
> > > > > 
> > > > > (00.009274)      1: mnt: Start with 0:/tmp/.criu.mntns.d6UW28
> > > > > (00.009279)      1: mnt: 	Mounting overlay @/tmp/.criu.mntns.d6UW28/12-0000000000/ (0)
> > > > > (00.009285)      1: mnt: 367:/tmp/.criu.mntns.d6UW28/12-0000000000/ private 1 shared 0 slave 0
> > > > > (00.009295)      1: mnt: 	Mounting tmpfs @/tmp/.criu.mntns.d6UW28/12-0000000000/run/secrets (0)
> > > > > (00.009297)      1: mnt: 	Bind /var/run/containers/storage/overlay-containers/43d02fb390d236ec42c5119b6655be44157e4a61103bc827db3b0718feecc898/userdata/run/secrets to /tmp/.criu.mntns.d6UW28/12-0000000000/run/secrets
> > > > > (00.009326)      1: Error (criu/mount.c:2279): mnt: Can't mount at /tmp/.criu.mntns.d6UW28/12-0000000000/run/secrets: No such file or directory
> > > > 
> > > > runsc creates a destination directory for external mounts:
> > > > https://github.com/opencontainers/runc/blob/master/libcontainer/rootfs_linux.go#L278
> > > > 
> > > > when a container is restored, destination directories for external
> > > > mounts are not created, because we suppose that the container rootfs
> > > > will be restored or migrated too. CRIU and runsc doesn't handle a
> > > > container rootfs, so you need to do this in podman.
> > > 
> > > I am doing this now. It still does not work, so I am probably doing it
> > > wrong.
> > 
> > Do you do this in CRIU? If it is yes, can you share a patch?
> > 
> > It looks like you always create a directory, but you have to look at a
> > source path. If a source is a file, you need to create a file, if a
> > source if a directory, you need to create a directory.
> 
> Hi Adrian,
> 
> Do you have any news about this issue?

I solved it with the following runc PR:

https://github.com/opencontainers/runc/pull/1968

runc does not create the bind mount mountpoints during restore as it
does during container creation. So if I try to migrate a container by
restoring it into a newly created container image (same image as during
initial container creation), CRIU is now able to restore all those bind
mounts as I added the same runc code path to the restore code path as
during container creation. runc basically calls createIfNotExists()
during container creation and now it also calls it during restore.

		Adrian