[CRIU] Podman container migration

[Adrian Reber]

On Thu, Jan 10, 2019 at 11:42:57AM +0000, Radostin Stoyanov wrote:
> On 10/01/2019 11:02, Radostin Stoyanov wrote:
> > On 09/01/2019 18:53, Adrian Reber wrote:
> >> On Wed, Jan 09, 2019 at 02:34:31PM +0000, Radostin Stoyanov wrote:
> >>> On 09/01/2019 14:16, Adrian Reber wrote:
> >>>> I started to look into migrating containers with Podman and I am
> >>>> currently stuck at files which are bind-mounted into the container:
> >>>>
> >>>> If I checkpoint a container and restore it, it works.
> >>>>
> >>>> If I checkpoint a container and transfer the checkpoint to another
> >>>> container it only works if the previous container is running, as
> >>>> CRIU fails to mount the file into the container.
> >>>>
> >>>> Following is from the checkpointed container:
> >>>>
> >>>>         {
> >>>>             "fstype": 5, 
> >>>>             "mnt_id": 368, 
> >>>>             "root_dev": "0:20", 
> >>>>             "parent_mnt_id": 367, 
> >>>>             "flags": "0x1000006", 
> >>>>             "root": "/containers/storage/overlay-containers/43d02fb390d236ec42c5119b6655be44157e4a61103bc827db3b0718feecc898/userdata/run/secrets", 
> >>>>             "mountpoint": "/run/secrets", 
> >>>>             "source": "tmpfs", 
> >>>>             "options": "size=2899748k,nr_inodes=724937,mode=755", 
> >>>>             "shared_id": 0, 
> >>>>             "master_id": 0, 
> >>>>             "sb_flags": "0x0", 
> >>>>             "ext_key": "/run/secrets"
> >>>>         }, 
> >>>>
> >>>>
> >>>> The main problem I see is that the storage location of the checkpointed
> >>>> container is part of "root".
> >>>>
> >>>> Or do I have to make sure that the migrated container is using the same
> >>>> paths as the restored container.
> >>>>
> >>>> I get the following error during restore, if I copy the checkpoint to another
> >>>> container.
> >>>>
> >>>> (00.009274)      1: mnt: Start with 0:/tmp/.criu.mntns.d6UW28
> >>>> (00.009279)      1: mnt: 	Mounting overlay @/tmp/.criu.mntns.d6UW28/12-0000000000/ (0)
> >>>> (00.009285)      1: mnt: 367:/tmp/.criu.mntns.d6UW28/12-0000000000/ private 1 shared 0 slave 0
> >>>> (00.009295)      1: mnt: 	Mounting tmpfs @/tmp/.criu.mntns.d6UW28/12-0000000000/run/secrets (0)
> >>>> (00.009297)      1: mnt: 	Bind /var/run/containers/storage/overlay-containers/43d02fb390d236ec42c5119b6655be44157e4a61103bc827db3b0718feecc898/userdata/run/secrets to /tmp/.criu.mntns.d6UW28/12-0000000000/run/secrets
> >>>> (00.009326)      1: Error (criu/mount.c:2279): mnt: Can't mount at /tmp/.criu.mntns.d6UW28/12-0000000000/run/secrets: No such file or directory
> >>>> (00.009331)      1: mnt: Start with 0:/tmp/.criu.mntns.d6UW28
> >>>> (00.027848) Error (criu/mount.c:3274): mnt: Can't remove the directory /tmp/.criu.mntns.d6UW28: No such file or directory
> >>>> (00.027860) Error (criu/cr-restore.c:2300): Restoring FAILED.
> >>>>
> >>>> The checkpointed container has the ID 43d02fb390d236ec42c5119b6655be44157e4a61103bc827db3b0718feecc898
> >>>> and the restored container has the ID 5ff14ac246ec6624c9667767532436d6fa5e3abda2b4437dc4b0db2ff1ff661a
> >>>>
> >>>> How do I tell CRIU to use another directory for the bind mount?
> >>> What about "--external mnt[KEY]:VAL" ?
> >> This is not exported as a runc interface.
> > In the rpc.proto file we have:
> >
> > message ext_mount_map {
> >     required string        key    = 1;
> >     required string        val    = 2;
> > };
> >
> > It is also added in libcriu as "criu_add_ext_mount()".
> Sorry, my previous message was not very useful.

;)

> I think we should expose the ext-mount-map functionality as a runc
> interface because
> we would need this to migrate volumes that are bind-mounted in a container.

That sounds correct, but looking at runc, I think it already does that
automatically.

> >> And runc already does some magic with the mounts:
> >>
> >> func (c *linuxContainer) addCriuDumpMount(req *criurpc.CriuReq, m *configs.Mount) {
> >>         mountDest := m.Destination
> >>         if strings.HasPrefix(mountDest, c.config.Rootfs) {
> >>                 mountDest = mountDest[len(c.config.Rootfs):]
> >>         }
> >>
> >>         extMnt := &criurpc.ExtMountMap{
> >>                 Key: proto.String(mountDest),
> >>                 Val: proto.String(mountDest),
> >>         }
> >>         req.Opts.ExtMnt = append(req.Opts.ExtMnt, extMnt)
> >> }

This function is called for all bind mounts already. So what else is
necessary?

> >> There is also the option to mask paths, not sure if that helps. So
> >> actually this is more a runc/criu question. As Andrei wrote the code in
> >> 2015 I am hoping he still remembers how this should be handled (if it
> >> can be handled at all...).

		Adrian