<div dir="ltr">Hi Tycho, <div><br></div><div>Unfortunately, looks like I don't have any <a href="http://tty.info">tty.info</a> in my checkpoint directory...</div><div>Hard to give you the output then :)</div><div><br></div><div>Thanks, <br>Alex </div></div><div class="gmail_extra"><br><div class="gmail_quote">2016-06-14 16:50 GMT+02:00 Tycho Andersen <span dir="ltr"><<a href="mailto:tycho.andersen@canonical.com" target="_blank">tycho.andersen@canonical.com</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi alex,<br>
<span class=""><br>
On Thu, Jun 09, 2016 at 05:53:26PM +0200, alex vk wrote:<br>
> Hi everyone,<br>
><br>
> I'm now trying to checkpoint and restore an LXC container on a raspberry PI<br>
> 2. (Criu 2.0 is working fine with the kernel 3.18, and LXC 2.0 works as<br>
> well on the RPI2)<br>
> Checkpointing a container apparently succeeds after disabling seccomp and<br>
> adding these lines to the container config (according to the wiki):<br>
><br>
> lxc.seccomp =<br>
> # hax for criu<br>
> lxc.console = none<br>
> lxc.tty = 0<br>
> lxc.cgroup.devices.deny = c 5:1 rwm<br>
><br>
> However restoring fails with this message : "lxc-checkpoint: criu.c:<br>
> exec_criu: 305 lxc.console configured on source host but not target"<br>
<br>
</span>Looks like you have tried a lot of stuff, let's sort out the liblxc<br>
issue first.<br>
<br>
Can you post the output of `cat <a href="http://tty.info" rel="noreferrer" target="_blank">tty.info</a>` in your checkpoint<br>
directory?<br>
<br>
Thanks,<br>
<br>
Tycho<br>
<div class="HOEnZb"><div class="h5"><br>
> I'read on this post (<br>
> <a href="https://lists.linuxcontainers.org/pipermail/lxc-devel/2016-March/013856.html" rel="noreferrer" target="_blank">https://lists.linuxcontainers.org/pipermail/lxc-devel/2016-March/013856.html</a>)<br>
> not to pass the --ext-mount-map option when console=none.<br>
> In this case, I have another error : "1: Error (mount.c:3381): mnt: New<br>
> root and old root are the same"<br>
><br>
> Interestingly, I have another error when trying to restore, if i run all<br>
> the container creation, start, dump and restore with sudo (not directly in<br>
> root):<br>
><br>
> pie: 1: Error (pie/restorer.c:888): Can't restore ticks/time for timer><br>
> pie: 1: fd - 0<br>
> pie: 1: Error (pie/restorer.c:1408): Can't restore timerfd -22<br>
><br>
> I've put the full log of dumping and restoring, as well as the container<br>
> config attached below, if anyone has any ideas ?<br>
><br>
> Thanks for your help,<br>
> Alex<br>
><br>
><br>
> ----------------------------------------------------------------------------------------------------<br>
> Here is what i get from the restore :<br>
><br>
> root@pi-desktop:/home/pi/checkpoint# lxc-checkpoint -r -D<br>
> /home/pi/checkpoint/ -n test-container --logfile=debug_restore<br>
> --logpriority=DEBUG<br>
> lxc-checkpoint: criu.c: do_restore: 705 criu process exited 1, output:<br>
> lxc-checkpoint: criu.c: exec_criu: 305 lxc.console configured on source<br>
> host but not target<br>
><br>
><br>
> lxc-checkpoint: cgfsng.c: recursive_destroy: 983 Error destroying<br>
> /sys/fs/cgroup/systemd//lxc/test-container<br>
> lxc-checkpoint: cgfsng.c: recursive_destroy: 983 Error destroying<br>
> /sys/fs/cgroup/blkio//lxc/test-container<br>
> lxc-checkpoint: cgfsng.c: recursive_destroy: 983 Error destroying<br>
> /sys/fs/cgroup/freezer//lxc/test-container<br>
> lxc-checkpoint: cgfsng.c: recursive_destroy: 983 Error destroying<br>
> /sys/fs/cgroup/debug//lxc/test-container<br>
> lxc-checkpoint: cgfsng.c: recursive_destroy: 983 Error destroying<br>
> /sys/fs/cgroup/cpu//lxc/test-container<br>
> lxc-checkpoint: cgfsng.c: recursive_destroy: 983 Error destroying<br>
> /sys/fs/cgroup/devices//lxc/test-container<br>
> lxc-checkpoint: cgfsng.c: recursive_destroy: 983 Error destroying<br>
> /sys/fs/cgroup/perf_event//lxc/test-container<br>
> lxc-checkpoint: cgfsng.c: recursive_destroy: 983 Error destroying<br>
> /sys/fs/cgroup/cpuset//lxc/test-container<br>
> lxc-checkpoint: cgfsng.c: recursive_destroy: 983 Error destroying<br>
> /sys/fs/cgroup/net_cls//lxc/test-container<br>
> lxc-checkpoint: criu.c: __criu_restore: 953 restore process died<br>
> Restoring test-container failed.<br>
><br>
> ----------------------------------------------------------------------------------------------------<br>
> Here is the log of the dumping created with the lxc-checkpoint -s<br>
> --logfile=debug_dump :<br>
> root@pi-desktop:/home/pi/checkpoint# lxc-checkpoint -s -D<br>
> /home/pi/checkpoint/ -n test-container --logfile=debug_dump<br>
> --logpriority=DEBUG<br>
> root@pi-desktop:/home/pi/checkpoint# cat debug_dump<br>
> lxc-checkpoint 20160609165234.776 WARN lxc_confile -<br>
> confile.c:config_pivotdir:1879 - lxc.pivotdir is ignored. It will soon<br>
> become an error.<br>
> lxc-checkpoint 20160609165234.777 WARN lxc_confile -<br>
> confile.c:config_personality:1117 - unsupported personality 'armhf'<br>
> lxc-checkpoint 20160609165234.784 DEBUG lxc_commands -<br>
> commands.c:lxc_cmd_get_state:579 - 'test-container' is in 'RUNNING' state<br>
> lxc-checkpoint 20160609165234.799 INFO lxc_criu - criu.c:exec_criu:387<br>
> - execing: /usr/local/sbin/criu dump --tcp-established --file-locks<br>
> --link-remap --manage-cgroups --ext-mount-map auto<br>
> --enable-external-sharing --enable-external-masters --enable-fs hugetlbfs<br>
> --enable-fs tracefs -D /home/pi/checkpoint/ -o<br>
> /home/pi/checkpoint//dump.log -t 4617 --freeze-cgroup<br>
> /sys/fs/cgroup/freezer///lxc/test-container --force-irmap<br>
> -------------------------------------------------------------------------------------------<br>
><br>
> And here is the dump.log :<br>
> root@pi-desktop:/home/pi/checkpoint# cat dump.log<br>
> Warn (autofs.c:77): Failed to find pipe_ino option (old kernel?)<br>
> Detected FS_EVENT_ON_CHILD bit in mask (will be ignored on restore)<br>
> tar: ./systemd/journal/socket: socket ignored<br>
> tar: ./systemd/journal/stdout: socket ignored<br>
> tar: ./systemd/journal/syslog: socket ignored<br>
> tar: ./systemd/journal/dev-log: socket ignored<br>
> tar: ./systemd/private: socket ignored<br>
> tar: ./systemd/notify: socket ignored<br>
><br>
> ----------------------------------------------------------------------------------------------------<br>
><br>
> Here is the log created with lxc-checkpoint -r --logfile :<br>
><br>
> root@pi-desktop:/home/pi/checkpoint# cat debug_restore<br>
> lxc-checkpoint 20160609165308.055 WARN lxc_confile -<br>
> confile.c:config_pivotdir:1879 - lxc.pivotdir is ignored. It will soon<br>
> become an error.<br>
> lxc-checkpoint 20160609165308.056 WARN lxc_confile -<br>
> confile.c:config_personality:1117 - unsupported personality 'armhf'<br>
> lxc-checkpoint 20160609165308.075 INFO lxc_lsm - lsm/lsm.c:lsm_init:48<br>
> - LSM security driver nop<br>
> lxc-checkpoint 20160609165308.076 DEBUG lxc_start -<br>
> start.c:setup_signal_fd:289 - sigchild handler set<br>
> lxc-checkpoint 20160609165308.076 INFO lxc_start -<br>
> start.c:lxc_init:488 - 'test-container' is initialized<br>
> lxc-checkpoint 20160609165308.076 INFO lxc_cgroup -<br>
> cgroup.c:cgroup_init:68 - cgroup driver cgroupfs-ng initing for<br>
> test-container<br>
> lxc-checkpoint 20160609165308.087 ERROR lxc_criu - criu.c:exec_criu:305<br>
> - lxc.console configured on source host but not target<br>
> lxc-checkpoint 20160609165308.115 INFO lxc_conf -<br>
> conf.c:run_script_argv:367 - Executing script<br>
> '/usr/share/lxcfs/lxc.reboot.hook' for container 'test-container', config<br>
> section 'lxc'<br>
> lxc-checkpoint 20160609165308.686 ERROR lxc_criu -<br>
> criu.c:do_restore:705 - criu process exited 1, output:<br>
> lxc-checkpoint: criu.c: exec_criu: 305 lxc.console configured on source<br>
> host but not target<br>
><br>
><br>
> lxc-checkpoint 20160609165308.689 INFO lxc_conf -<br>
> conf.c:run_script_argv:367 - Executing script<br>
> '/usr/share/lxcfs/lxc.reboot.hook' for container 'test-container', config<br>
> section 'lxc'<br>
> lxc-checkpoint 20160609165309.206 ERROR lxc_cgfsng -<br>
> cgfsng.c:recursive_destroy:983 - Error destroying<br>
> /sys/fs/cgroup/systemd//lxc/test-container<br>
> lxc-checkpoint 20160609165309.208 ERROR lxc_cgfsng -<br>
> cgfsng.c:recursive_destroy:983 - Error destroying<br>
> /sys/fs/cgroup/blkio//lxc/test-container<br>
> lxc-checkpoint 20160609165309.209 ERROR lxc_cgfsng -<br>
> cgfsng.c:recursive_destroy:983 - Error destroying<br>
> /sys/fs/cgroup/freezer//lxc/test-container<br>
> lxc-checkpoint 20160609165309.211 ERROR lxc_cgfsng -<br>
> cgfsng.c:recursive_destroy:983 - Error destroying<br>
> /sys/fs/cgroup/debug//lxc/test-container<br>
> lxc-checkpoint 20160609165309.212 ERROR lxc_cgfsng -<br>
> cgfsng.c:recursive_destroy:983 - Error destroying<br>
> /sys/fs/cgroup/cpu//lxc/test-container<br>
> lxc-checkpoint 20160609165309.213 ERROR lxc_cgfsng -<br>
> cgfsng.c:recursive_destroy:983 - Error destroying<br>
> /sys/fs/cgroup/devices//lxc/test-container<br>
> lxc-checkpoint 20160609165309.214 ERROR lxc_cgfsng -<br>
> cgfsng.c:recursive_destroy:983 - Error destroying<br>
> /sys/fs/cgroup/perf_event//lxc/test-container<br>
> lxc-checkpoint 20160609165309.215 ERROR lxc_cgfsng -<br>
> cgfsng.c:recursive_destroy:983 - Error destroying<br>
> /sys/fs/cgroup/cpuset//lxc/test-container<br>
> lxc-checkpoint 20160609165309.217 ERROR lxc_cgfsng -<br>
> cgfsng.c:recursive_destroy:983 - Error destroying<br>
> /sys/fs/cgroup/net_cls//lxc/test-container<br>
> lxc-checkpoint 20160609165309.220 ERROR lxc_criu -<br>
> criu.c:__criu_restore:953 - restore process died<br>
><br>
><br>
> ----------------------------------------------------------------------------------------------------<br>
><br>
> The log without --ext-mount-map<br>
><br>
> /usr/local/sbin/criu restore --tcp-established --file-locks --link-remap<br>
> --manage-cgroups --ext-mount-map auto --enable-external-sharing<br>
> --enable-external-masters --enable-fs hugetlbfs --enable-fs tracefs -D<br>
> /home/pi/checkpoint -o /home/pi/checkpoint/restore.log --root<br>
> /usr/lib/arm-linux-gnueabihf/lxc --restore-detached --restore-sibling<br>
> --pidfile /tmp/filet0XGku --cgroup-root /lxc/test-container console:<br>
> --veth-pair eth0=vethINVTTV@lxcbr0<br>
><br>
> root@pi-desktop:/home/pi/checkpoint# cat restore.log<br>
> Warn (cr-restore.c:812): Set CLONE_PARENT | CLONE_NEWPID but it might<br>
> cause restore problem,because not all kernels support such clone flags<br>
> combinations!<br>
> RTNETLINK answers: File exists<br>
> RTNETLINK answers: File exists<br>
> RTNETLINK answers: File exists<br>
> RTNETLINK answers: File exists<br>
> 1: Warn (autofs.c:77): Failed to find pipe_ino option (old kernel?)<br>
> 1: Error (mount.c:3381): mnt: New root and old root are the same<br>
> Error (cr-restore.c:988): 5521 killed by signal 9: Killed<br>
> Error (cr-restore.c:1867): Restoring FAILED.<br>
><br>
> ----------------------------------------------------------------------------------------------------<br>
><br>
> As a normal user, with sudo :<br>
><br>
> pi@pi-desktop:~/checkpoint$ cat debug_restore<br>
> lxc-checkpoint 20160609171247.434 WARN lxc_confile -<br>
> confile.c:config_pivotdir:1879 - lxc.pivotdir is ignored. It will soon<br>
> become an error.<br>
> lxc-checkpoint 20160609171247.435 WARN lxc_confile -<br>
> confile.c:config_personality:1117 - unsupported personality 'armhf'<br>
> lxc-checkpoint 20160609171247.454 INFO lxc_lsm - lsm/lsm.c:lsm_init:48<br>
> - LSM security driver nop<br>
> lxc-checkpoint 20160609171247.455 DEBUG lxc_start -<br>
> start.c:setup_signal_fd:289 - sigchild handler set<br>
> lxc-checkpoint 20160609171247.455 INFO lxc_start -<br>
> start.c:lxc_init:488 - 'test-container' is initialized<br>
> lxc-checkpoint 20160609171247.455 INFO lxc_cgroup -<br>
> cgroup.c:cgroup_init:68 - cgroup driver cgroupfs-ng initing for<br>
> test-container<br>
> lxc-checkpoint 20160609171247.456 ERROR lxc_cgfsng -<br>
> cgfsng.c:cgfsng_create:1072 - No such file or directory - Failed to create<br>
> /sys/fs/cgroup/systemd//lxc/test-container: No such file or directory<br>
> lxc-checkpoint 20160609171247.466 INFO lxc_criu - criu.c:exec_criu:387<br>
> - execing: /usr/local/sbin/criu restore --tcp-established --file-locks<br>
> --link-remap --manage-cgroups --ext-mount-map auto<br>
> --enable-external-sharing --enable-external-masters --enable-fs hugetlbfs<br>
> --enable-fs tracefs -D /home/pi/checkpoint -o<br>
> /home/pi/checkpoint/restore.log --root /usr/lib/arm-linux-gnueabihf/lxc<br>
> --restore-detached --restore-sibling --pidfile /tmp/file8x1eFF<br>
> --cgroup-root /lxc/test-container-1 --ext-mount-map console: --veth-pair<br>
> eth0=veth6AHNY2@lxcbr0<br>
> lxc-checkpoint 20160609171248.456 ERROR lxc_criu -<br>
> criu.c:do_restore:705 - criu process exited 1, output:<br>
><br>
><br>
> lxc-checkpoint 20160609171248.458 INFO lxc_conf -<br>
> conf.c:run_script_argv:367 - Executing script<br>
> '/usr/share/lxcfs/lxc.reboot.hook' for container 'test-container', config<br>
> section 'lxc'<br>
> lxc-checkpoint 20160609171248.984 ERROR lxc_criu -<br>
> criu.c:__criu_restore:953 - restore process died<br>
><br>
> ----------------------------------<br>
><br>
> The restore.log as a normal user, with sudo :<br>
><br>
> pi@pi-desktop:~/checkpoint$ sudo cat restore.log<br>
> Warn (cr-restore.c:812): Set CLONE_PARENT | CLONE_NEWPID but it might<br>
> cause restore problem,because not all kernels support such clone flags<br>
> combinations!<br>
> RTNETLINK answers: File exists<br>
> RTNETLINK answers: File exists<br>
> RTNETLINK answers: File exists<br>
> RTNETLINK answers: File exists<br>
> 1: Warn (autofs.c:77): Failed to find pipe_ino option (old kernel?)<br>
> 1: Warn (sk-unix.c:1292): sk unix: Can't unlink stale socket 0x88b4<br>
> peer 0 (name /run/systemd/notify dir -)<br>
> 1: Warn (sk-unix.c:1292): sk unix: Can't unlink stale socket 0x88b6<br>
> peer 0 (name /run/systemd/private dir -)<br>
> 1: Warn (sk-unix.c:1292): sk unix: Can't unlink stale socket 0x88ba<br>
> peer 0 (name /run/systemd/journal/dev-log dir -)<br>
> 1: Warn (sk-unix.c:1292): sk unix: Can't unlink stale socket 0x8285<br>
> peer 0 (name /run/systemd/journal/stdout dir -)<br>
> 1: Warn (sk-unix.c:1292): sk unix: Can't unlink stale socket 0x8287<br>
> peer 0 (name /run/systemd/journal/socket dir -)<br>
> 1: Warn (sk-unix.c:1292): sk unix: Can't unlink stale socket 0x88d0<br>
> peer 0 (name /run/systemd/journal/syslog dir -)<br>
> 1: Warn (sk-unix.c:1292): sk unix: Can't unlink stale socket 0x83cc<br>
> peer 0x83cb (name /run/systemd/journal/stdout dir -)<br>
> 1: Warn (sk-unix.c:1292): sk unix: Can't unlink stale socket 0x83fc<br>
> peer 0x8dfa (name /run/systemd/journal/stdout dir -)<br>
> pie: 1: Error (pie/restorer.c:888): Can't restore ticks/time for timer><br>
> pie: 1: fd - 0<br>
> pie: 1: Error (pie/restorer.c:1408): Can't restore timerfd -22<br>
> pie: 1: Error (pie/restorer.c:1506): Restorer fail 1<br>
> Error (cr-restore.c:985): 6401 exited, status=1<br>
> Error (cr-restore.c:1867): Restoring FAILED.<br>
><br>
><br>
> -------------------------------<br>
><br>
> Here is the config of the container : /var/lib/lxc/test-container/config<br>
><br>
> # Template used to create this container:<br>
> /usr/share/lxc/templates/lxc-ubuntu<br>
> # Parameters passed to the template:<br>
> # For additional config options, please look at lxc.container.conf(5)<br>
><br>
> # Uncomment the following line to support nesting containers:<br>
> #lxc.include = /usr/share/lxc/config/nesting.conf<br>
> # (Be aware this has security implications)<br>
><br>
><br>
> # Common configuration<br>
> lxc.include = /usr/share/lxc/config/ubuntu.common.conf<br>
><br>
> # Container specific configuration<br>
> lxc.rootfs = /var/lib/lxc/test-container/rootfs<br>
> lxc.rootfs.backend = dir<br>
> lxc.utsname = test-container<br>
> lxc.arch = armhf<br>
><br>
> # Network configuration<br>
> lxc.network.type = veth<br>
> lxc.network.link = lxcbr0<br>
> lxc.network.flags = up<br>
> lxc.network.hwaddr = 00:16:3e:d5:62:a5<br>
> lxc.seccomp =<br>
> lxc.console = none<br>
> lxc.tty = 0<br>
> lxc.cgroup.devices.deny = c 5:1 rwm<br>
<br>
</div></div><div class="HOEnZb"><div class="h5">> _______________________________________________<br>
> CRIU mailing list<br>
> <a href="mailto:CRIU@openvz.org">CRIU@openvz.org</a><br>
> <a href="https://lists.openvz.org/mailman/listinfo/criu" rel="noreferrer" target="_blank">https://lists.openvz.org/mailman/listinfo/criu</a><br>
<br>
</div></div></blockquote></div><br></div>