[CRIU] [PATCH 00/11] tty: Add c/r for system console
Tycho Andersen
tycho.andersen at canonical.com
Tue Sep 30 14:25:47 PDT 2014
On Tue, Sep 30, 2014 at 10:03:33PM +0400, Cyrill Gorcunov wrote:
> On Tue, Sep 30, 2014 at 08:32:57AM -0500, Tycho Andersen wrote:
> > > >
> > > > Thanks for your work on this set. This patch does fix the aforementioned bug
> > > > about not being able to checkpoint containers that have been restored. (I'm not
> > > > sure which patch does it, but with the entire series applied it works just
> > > > fine. I can go back and find out if this information would be useful.)
> > >
> > > Hi Tycho! Thanks a lot for giving it a shot (and btw I remember about the
> > > problem with sequential c/r where second dump failed due to tty layer, simply
> > > didn't manage yet to investigate, I suspect I need to achieve same error on
> > > my testing node, once I manage to I'll ping you, hopefully on the week).
> >
> > What I meant by the above is that this patchset actually fixes that
> > error for me (somehow, not sure which patch does it :).
>
> Actually this patchset only adds support of /dev/console, so in your
> report we've had pty pair in first dump, then after restore (for some
> unknown reason) instead of ptys we got /dev/console reference and that's
> the question who is changed it, I suspect it might be systemd or something
> like that.
Yes, I just saw it again today with this patchset applied, so it looks
like I spoke too soon :(
> > > >
> > > > However, when I try to checkpoint a container with /dev/console and some ttys, I
> > > > get:
> > > >
> > > > Error (mount.c:453): 113:./dev/lxc/tty4 doesn't have a proper root mount
> > > >
> > > > On an LXC in my system with ttys enabled, the /proc/self/mountinfo has:
> > > >
> > > > ubuntu at u1:~$ cat /proc/self/mountinfo
> > > > 75 76 253:1 /usr/local/var/lib/lxc/u1/rootfs / rw,relatime - ext4 /dev/disk/by-uuid/7fd0b676-46f3-46d6-94a6-5c620e72e7b0 rw,data=ordered
> > > > 103 75 0:35 / /proc rw,nosuid,nodev,noexec,relatime - proc proc rw
> > > > 104 75 0:33 / /sys rw,relatime - sysfs sysfs rw
> > > > 105 104 0:18 / /sys/fs/fuse/connections rw,relatime - fusectl none rw
> > > > 106 104 0:6 / /sys/kernel/debug rw,relatime - debugfs none rw
> > > > 107 104 0:10 / /sys/kernel/security rw,relatime - securityfs none rw
> > > > 108 104 0:22 / /sys/fs/pstore rw,relatime - pstore none rw
> > > > 109 75 0:12 /7 /dev/lxc/console rw,nosuid,noexec,relatime - devpts devpts rw,gid=5,mode=620,ptmxmode=000
> > > > 110 75 0:12 /1 /dev/lxc/tty1 rw,nosuid,noexec,relatime - devpts devpts rw,gid=5,mode=620,ptmxmode=000
> > > > 111 75 0:12 /3 /dev/lxc/tty2 rw,nosuid,noexec,relatime - devpts devpts rw,gid=5,mode=620,ptmxmode=000
> > > > 112 75 0:12 /5 /dev/lxc/tty3 rw,nosuid,noexec,relatime - devpts devpts rw,gid=5,mode=620,ptmxmode=000
> > > > 113 75 0:12 /6 /dev/lxc/tty4 rw,nosuid,noexec,relatime - devpts devpts rw,gid=5,mode=620,ptmxmode=000
> > > > 77 75 0:36 / /dev/pts rw,relatime - devpts devpts rw,gid=5,mode=620,ptmxmode=666
> > > > 78 75 0:36 /ptmx /dev/ptmx rw,relatime - devpts devpts rw,gid=5,mode=620,ptmxmode=666
> > > > 79 104 0:37 / /sys/fs/cgroup rw,relatime - tmpfs none rw,size=4k,mode=755
> > > > 80 75 0:38 / /run rw,nosuid,noexec,relatime - tmpfs none rw,size=200060k,mode=755
> > > > 81 80 0:39 / /run/lock rw,nosuid,nodev,noexec,relatime - tmpfs none rw,size=5120k
> > > > 82 80 0:40 / /run/shm rw,nosuid,nodev,relatime - tmpfs none rw
> > > > 83 80 0:41 / /run/user rw,nosuid,nodev,noexec,relatime - tmpfs none rw,size=102400k,mode=755
> > > >
>
> Looks like this are external moutns which might require --ext-mount-map keys, how this mountinfo
> represented on pid 1 of container (ie do "ps afx" on node and the choose pid for init of container,
> then show /proc/$pid/mountinfo)
It looks like the same sort of confusing thing to me:
$ cat /proc/24351/mountinfo
45 46 253:1 /usr/local/var/lib/lxc/u1/rootfs / rw,relatime - ext4 /dev/disk/by-uuid/86de4daf-7810-4422-804f-d1ab23ffd9fa rw,data=ordered
89 45 0:35 / /proc rw,nosuid,nodev,noexec,relatime - proc proc rw
90 45 0:33 / /sys rw,relatime - sysfs sysfs rw
91 90 0:18 / /sys/fs/fuse/connections rw,relatime - fusectl none rw
92 90 0:6 / /sys/kernel/debug rw,relatime - debugfs none rw
93 90 0:10 / /sys/kernel/security rw,relatime - securityfs none rw
94 90 0:23 / /sys/fs/pstore rw,relatime - pstore none rw
95 45 0:12 /14 /dev/lxc/console rw,nosuid,noexec,relatime - devpts devpts rw,gid=5,mode=620,ptmxmode=000
96 45 0:12 /10 /dev/lxc/tty1 rw,nosuid,noexec,relatime - devpts devpts rw,gid=5,mode=620,ptmxmode=000
97 45 0:12 /11 /dev/lxc/tty2 rw,nosuid,noexec,relatime - devpts devpts rw,gid=5,mode=620,ptmxmode=000
98 45 0:12 /12 /dev/lxc/tty3 rw,nosuid,noexec,relatime - devpts devpts rw,gid=5,mode=620,ptmxmode=000
99 45 0:12 /13 /dev/lxc/tty4 rw,nosuid,noexec,relatime - devpts devpts rw,gid=5,mode=620,ptmxmode=000
47 45 0:36 / /dev/pts rw,relatime - devpts devpts rw,gid=5,mode=620,ptmxmode=666
48 45 0:36 /ptmx /dev/ptmx rw,relatime - devpts devpts rw,gid=5,mode=620,ptmxmode=666
49 90 0:37 / /sys/fs/cgroup rw,relatime - tmpfs none rw,size=4k,mode=755
54 45 0:38 / /run rw,nosuid,noexec,relatime - tmpfs none rw,size=200060k,mode=755
55 54 0:39 / /run/lock rw,nosuid,nodev,noexec,relatime - tmpfs none rw,size=5120k
56 54 0:40 / /run/shm rw,nosuid,nodev,relatime - tmpfs none rw
57 54 0:41 / /run/user rw,nosuid,nodev,noexec,relatime - tmpfs none rw,size=102400k,mode=755
$ ssh ubuntu@$(sudo lxc-info -n u1 -H -i)
ubuntu at 10.0.3.166's password:
Welcome to Ubuntu 14.04.1 LTS (GNU/Linux 3.13.0-35-generic x86_64)
* Documentation: https://help.ubuntu.com/
Last login: Tue Sep 30 15:44:48 2014 from 10.0.3.1
ubuntu at u1:~$ cat /proc/self/mountinfo
45 46 253:1 /usr/local/var/lib/lxc/u1/rootfs / rw,relatime - ext4 /dev/disk/by-uuid/86de4daf-7810-4422-804f-d1ab23ffd9fa rw,data=ordered
89 45 0:35 / /proc rw,nosuid,nodev,noexec,relatime - proc proc rw
90 45 0:33 / /sys rw,relatime - sysfs sysfs rw
91 90 0:18 / /sys/fs/fuse/connections rw,relatime - fusectl none rw
92 90 0:6 / /sys/kernel/debug rw,relatime - debugfs none rw
93 90 0:10 / /sys/kernel/security rw,relatime - securityfs none rw
94 90 0:23 / /sys/fs/pstore rw,relatime - pstore none rw
95 45 0:12 /14 /dev/lxc/console rw,nosuid,noexec,relatime - devpts devpts rw,gid=5,mode=620,ptmxmode=000
96 45 0:12 /10 /dev/lxc/tty1 rw,nosuid,noexec,relatime - devpts devpts rw,gid=5,mode=620,ptmxmode=000
97 45 0:12 /11 /dev/lxc/tty2 rw,nosuid,noexec,relatime - devpts devpts rw,gid=5,mode=620,ptmxmode=000
98 45 0:12 /12 /dev/lxc/tty3 rw,nosuid,noexec,relatime - devpts devpts rw,gid=5,mode=620,ptmxmode=000
99 45 0:12 /13 /dev/lxc/tty4 rw,nosuid,noexec,relatime - devpts devpts rw,gid=5,mode=620,ptmxmode=000
47 45 0:36 / /dev/pts rw,relatime - devpts devpts rw,gid=5,mode=620,ptmxmode=666
48 45 0:36 /ptmx /dev/ptmx rw,relatime - devpts devpts rw,gid=5,mode=620,ptmxmode=666
49 90 0:37 / /sys/fs/cgroup rw,relatime - tmpfs none rw,size=4k,mode=755
54 45 0:38 / /run rw,nosuid,noexec,relatime - tmpfs none rw,size=200060k,mode=755
55 54 0:39 / /run/lock rw,nosuid,nodev,noexec,relatime - tmpfs none rw,size=5120k
56 54 0:40 / /run/shm rw,nosuid,nodev,relatime - tmpfs none rw
57 54 0:41 / /run/user rw,nosuid,nodev,noexec,relatime - tmpfs none rw,size=102400k,mode=755
> > >
> > > Mind to share your lxc conatiner details and how you set it up?
> >
> > Sure, the config I used is:
> >
> > # Template used to create this container:
> > # /usr/local/share/lxc/templates/lxc-download
> > # Parameters passed to the template: -d ubuntu -r trusty -a amd64
> > # For additional config options, please look at lxc.container.conf(5)
> >
> > # Distribution configuration
> > lxc.include = /usr/local/share/lxc/config/ubuntu.common.conf
> > lxc.arch = x86_64
> >
> > # Container specific configuration
> > lxc.rootfs = /usr/local/var/lib/lxc/u1/rootfs
> > lxc.utsname = u1
> >
> > # Network configuration
> > lxc.network.type = veth
> > lxc.network.flags = up
> > lxc.network.link = lxcbr0
> > lxc.network.hwaddr = 00:16:3e:0e:08:34
> >
> > # hax for criu
> > #lxc.console = none
> > #lxc.tty = 0
> > lxc.cgroup.devices.deny = c 5:1 rwm
> >
> > And I created it with the command line:
> >
> > sudo lxc-create -t download -n u1 -- -d ubuntu -r trusty -a amd64
> >
> > > > When I try to work around that with something like,
> > > >
> > > > diff --git a/mount.c b/mount.c
> > > > index 8c60693..a28e041 100644
> > > > --- a/mount.c
> > > > +++ b/mount.c
> > > > @@ -433,8 +433,19 @@ static int validate_mounts(struct mount_info *info, bool for_dump)
> > > > ret = run_plugins(DUMP_EXT_MOUNT, m->mountpoint, m->mnt_id);
> > > > if (ret == 0)
> > > > m->need_plugin = true;
> > > > - else if (ret == -ENOTSUP)
> > > > - ret = try_resolve_ext_mount(m);
> > > > + else if (ret == -ENOTSUP) {
> > > > + /* inside containers
> > > > + * /dev/console and /dev/tty*
> > > > + * are mounted from paths like
> > > > + * /5 or /6, so we ignore
> > > > + * their path here if things
> > > > + * go bad.
> > > > + */
> > > > + if (m->fstype->code == FSTYPE__DEVPTS)
> > > > + ret = 0;
> > > > + else
> > > > + ret = try_resolve_ext_mount(m);
> > > > + }
> > > > } else {
> > > > if (m->need_plugin || m->external)
> > > > /*
> > > >
> > > >
> > > > I hit:
> > > >
> > > > (00.247031) Error (tty.c:233): tty: Unexpected format on path /dev/lxc/tty4
> > > >
> > > > which looks like it might be a bigger problem :)
>
> Aha! We're expecting ptys to have a canonical forms, ie /dev/pts/N, where N is
> pty number, need to thhink how to workaround it (you can fix up criu by now
> changing parse_index() helper to parse not PTS_FMT but say "/dev/lxc/tty%d"
Yes, I was trying to think of a nice workaround too. I guess a hack
would be to just parse an int at the end of the string, I'm not sure
if there is a better way.
>
> > > >
> > > > Any thoughts are appreciated.
> > >
> > > Show me the stat output on this paths please? (stat /dev/lxc/tty4)
> >
> > ubuntu at u1:~$ stat /dev/lxc/tty4
> > File: ‘/dev/lxc/tty4’
> > Size: 0 Blocks: 0 IO Block: 1024 character special file
> > Device: ch/12d Inode: 10 Links: 1 Device type: 88,7
> > Access: (0660/crw-rw----) Uid: ( 0/ root) Gid: ( 5/ tty)
> > Access: 2014-09-30 13:28:09.359281000 +0000
> > Modify: 2014-09-30 13:28:09.359281000 +0000
> > Change: 2014-09-30 13:28:13.239281000 +0000
> > Birth: -
>
> Thanks, this is indeed unix98 pty device, so need to fix this
> problem somehow...
Ok, let me know if I can help.
Tycho
More information about the CRIU
mailing list