[CRIU] Not able to checkpoint docker container with criu-1.3-rc2
Pradeep Padala
ppadala at gmail.com
Wed Aug 6 12:49:27 PDT 2014
Thanks for the detailed commands. It doesn't give an error right away, but
when I check the dump.log, I see an error. I used the command below.
$./criu dump -D /tmp/docker -o dump.log -v4 --ext-mount-map
/etc/resolv.conf:/etc/resolv.conf --ext-mount-map
/etc/hostname:/etc/hostname --ext-mount-map /etc/hosts:/etc/hosts
--evasive-devices -t 2140
$cat /tmp/docker/dump.log
(00.000219) ========================================
(00.000270) Dumping processes (pid: 2140)
(00.000275) ========================================
(00.003150) Found anon-shmem device at 4
(00.003198) Reset 6112's dirty tracking
(00.004367) ... done
(00.004441) Dirty track supported on kernel
(00.004483) irmap: Searching irmap cache in work dir
(00.004514) irmap: Searching irmap cache in parent
(00.004535) irmap: No irmap cache
(00.004821) cpu: fpu:1 fxsr:1 xsave:1
(00.005014) vdso: Parsing at 7fff65bad000 7fff65baf000
(00.005027) vdso: PT_LOAD p_vaddr: 0
(00.005030) vdso: DT_HASH: 0x120
(00.005033) vdso: DT_STRTAB: 0x268
(00.005035) vdso: DT_SYMTAB: 0x160
(00.005037) vdso: DT_STRSZ: 94
(00.005040) vdso: DT_SYMENT: 24
(00.005042) vdso: nbucket 3 nchain 11 bucket 0x7fff65bad128 chain
0x7fff65bad134
(00.005050) vdso: rt [vdso] 7fff65bad000-7fff65baf000 [vvar]
ffffffffffffffff-ffffffffffffffff
(00.005238) Writing image inventory (version 1)
(00.005434) Collected 1.pid namespace
(00.005459) Collected 2.net namespace
(00.005471) Collected 3.ipc namespace
(00.005483) Collected 4.uts namespace
(00.005498) Collected 5.mnt namespace
(00.005503) cg: Dumping cgroups for 6112
(00.005585) cg: `- New css ID 1
(00.005593) cg: `- [cpuset] -> [/]
(00.005596) cg: `- [devices] -> [/]
(00.005598) cg: `- [freezer] -> [/]
(00.005600) cg: `- [hugetlb] -> [/]
(00.005602) cg: `- [memory] -> [/]
(00.005605) cg: `- [net_cls,net_prio] -> [/]
(00.005607) cg: `- [perf_event] -> [/]
(00.005609) cg: `- [blkio] -> [/user.slice]
(00.005612) cg: `- [cpu,cpuacct] -> [/user.slice]
(00.005614) cg: `- [name=systemd] ->
[/user.slice/user-0.slice/session-18.scope]
(00.005617) cg: Set 1 is criu one
(00.007028) Seized task 2140, state 1
(00.007213) Collected 2140 in 1 state
(00.007493) Will take pid namespace in the image
(00.007500) Collected 6.pid namespace
(00.007510) Will take net namespace in the image
(00.007513) Collected 7.net namespace
(00.007521) Will take ipc namespace in the image
(00.007524) Collected 8.ipc namespace
(00.007532) Will take uts namespace in the image
(00.007535) Collected 9.uts namespace
(00.007542) Will take mnt namespace in the image
(00.007545) Collected 10.mnt namespace
(00.007548) Lock network
(00.007551) Running network-lock scripts
(00.007684) Dump MNT namespace (mountpoints) 10 via 2140
(00.007827) type unsupported source
/dev/mapper/fedora_wdc--drm--vm--dhcp423-root fd00000
/var/lib/docker/vfs/dir/0895cf3a6f52123e5ef6cf81db662607ebd90c9c872b816bd7cf97188cfbcf5a
@ ./ flags 200000 options seclabel,data=ordered,
(00.007847) type proc source proc 25 / @ ./proc flags 20000e options
(00.007859) type sysfs source sysfs 26 / @ ./sys flags 200001 options
seclabel,
(00.007873) type tmpfs source tmpfs 27 / @ ./dev flags 2 options
context="system_u:object_r:svirt_sandbox_file_t:s0:c275,c327",mode=755,
(00.007886) type tmpfs source shm 28 / @ ./dev/shm flags 20000e options
context="system_u:object_r:svirt_sandbox_file_t:s0:c275,c327",size=65536k,
(00.007905) type devpts source devpts 29 / @ ./dev/pts flags 20000a
options
context="system_u:object_r:svirt_sandbox_file_t:s0:c275,c327",gid=5,mode=620,ptmxmode=666,
(00.008187) type unsupported source
/dev/mapper/fedora_wdc--drm--vm--dhcp423-root fd00000
/var/lib/docker/init/dockerinit-1.0.0 @ ./.dockerinit flags 200001 options
seclabel,data=ordered,
(00.008252) type unsupported source
/dev/mapper/fedora_wdc--drm--vm--dhcp423-root fd00000 /etc/resolv.conf @
./etc/resolv.conf flags 200001 options seclabel,data=ordered,
(00.008267) type unsupported source
/dev/mapper/fedora_wdc--drm--vm--dhcp423-root fd00000
/var/lib/docker/containers/0895cf3a6f52123e5ef6cf81db662607ebd90c9c872b816bd7cf97188cfbcf5a/hostname
@ ./etc/hostname flags 200001 options seclabel,data=ordered,
(00.008290) type unsupported source
/dev/mapper/fedora_wdc--drm--vm--dhcp423-root fd00000
/var/lib/docker/containers/0895cf3a6f52123e5ef6cf81db662607ebd90c9c872b816bd7cf97188cfbcf5a/hosts
@ ./etc/hosts flags 200001 options seclabel,data=ordered,
(00.008305) type devpts source devpts b /1 @ ./dev/console flags 20000a
options seclabel,gid=5,mode=620,ptmxmode=000,
(00.008331) type proc source proc 25 /sys @ ./proc/sys flags 20000f
options
(00.008344) type proc source proc 25 /sysrq-trigger @
./proc/sysrq-trigger flags 20000f options
(00.008356) type proc source proc 25 /irq @ ./proc/irq flags 20000f
options
(00.008366) type proc source proc 25 /bus @ ./proc/bus flags 20000f
options
(00.008378) type tmpfs source tmpfs 27 /null @ ./proc/kcore flags 2
options
context="system_u:object_r:svirt_sandbox_file_t:s0:c275,c327",mode=755,
(00.008409) Building mountpoints tree
(00.008416) Building plain mount tree
(00.008419) Working on 113->138
(00.008422) Working on 112->138
(00.008424) Working on 111->138
(00.008426) Working on 109->138
(00.008428) Working on 108->138
(00.008431) Working on 147->140
(00.008433) Working on 146->137
(00.008435) Working on 145->137
(00.008438) Working on 144->137
(00.008440) Working on 143->137
(00.008442) Working on 142->140
(00.008445) Working on 141->140
(00.008447) Working on 140->137
(00.008449) Working on 139->137
(00.008451) Working on 138->137
(00.008454) Working on 137->106
(00.008456) Resorting siblings on 137
(00.008459) Resorting siblings on 146
(00.008461) Resorting siblings on 145
(00.008463) Resorting siblings on 144
(00.008466) Resorting siblings on 143
(00.008468) Resorting siblings on 140
(00.008470) Resorting siblings on 147
(00.008473) Resorting siblings on 142
(00.008475) Resorting siblings on 141
(00.008477) Resorting siblings on 139
(00.008479) Resorting siblings on 138
(00.008482) Resorting siblings on 113
(00.008484) Resorting siblings on 112
(00.008486) Resorting siblings on 111
(00.008488) Resorting siblings on 109
(00.008490) Resorting siblings on 108
(00.008496) Done:
(00.008498) [./](137->106)
(00.008501) [./.dockerinit](143->137)
(00.008504) <--
(00.008507) [./proc](138->137)
(00.008510) [./proc/kcore](113->138)
(00.008512) <--
(00.008514) [./proc/sys](108->138)
(00.008517) <--
(00.008519) [./proc/sysrq-trigger](109->138)
(00.008522) <--
(00.008524) [./proc/irq](111->138)
(00.008527) <--
(00.008529) [./proc/bus](112->138)
(00.008531) <--
(00.008534) <--
(00.008536) [./sys](139->137)
(00.008538) <--
(00.008540) [./dev](140->137)
(00.008543) [./dev/console](147->140)
(00.008546) <--
(00.008548) [./dev/shm](141->140)
(00.008550) <--
(00.008552) [./dev/pts](142->140)
(00.008555) <--
(00.008557) <--
(00.008559) [./etc/hosts](146->137)
(00.008562) <--
(00.008564) [./etc/resolv.conf](144->137)
(00.008567) <--
(00.008569) [./etc/hostname](145->137)
(00.008572) <--
(00.008574) <--
(00.008584) Error (mount.c:449): 147:./dev/console doesn't have a proper
root mount
(00.008590) Unlock network
(00.008597) Running network-unlock scripts
(00.008606) Unfreezing tasks into 1
(00.008610) Unseizing 2140 into 1
(00.008972) Error (cr-dump.c:1914): Dumping FAILED.
On Wed, Aug 6, 2014 at 10:24 AM, Saied Kazemi <saied at google.com> wrote:
> That's because your command line is incomplete. Also, note that you need
> to bind mount the root before doing "criu restore". Below is how I do it.
> You can optionally run "sudo tail -f
> /var/lib/docker/vfs/dir/b3438bf5d19d241058659d3e56b7b516100a7db9720c6869a04b36528d297fe3/foo"
> and actually see the output stop after dump and restart after restore.
>
> --Saied
>
> # docker run -d busybox:latest /bin/sh -c 'i=1; while true; do echo $i >>
> /foo; i=`expr $i + 1`; sleep 3; done'
> b3438bf5d19d241058659d3e56b7b516100a7db9720c6869a04b36528d297fe3
> #
> # ps -efl | grep /bin/sh
> 4 S root 30210 29160 0 80 0 - 791 wait 10:08 ? 00:00:00
> /bin/sh -c i=0; while true; do echo $i >> /foo; i=`expr $i + 1`; sleep 3;
> done
> 0 S root 30247 26797 0 80 0 - 2936 pipe_w 10:08 pts/2 00:00:00
> grep --color=auto /bin/sh
> #
> # criu dump -D /tmp/img.vfs -o dump.log -v4 --ext-mount-map
> /etc/resolv.conf:/etc/resolv.conf --ext-mount-map
> /etc/hostname:/etc/hostname --ext-mount-map /etc/hosts:/etc/hosts
> --evasive-devices -t 30210
> #
> # echo $?
> 0
> #
> # ps -efl | grep /bin/sh
> 0 S root 30425 26797 0 80 0 - 2936 pipe_w 10:11 pts/2 00:00:00
> grep --color=auto /bin/sh
> #
> # mount --rbind
> /var/lib/docker/vfs/dir/b3438bf5d19d241058659d3e56b7b516100a7db9720c6869a04b36528d297fe3
> /var/lib/docker/vfs/dir/b3438bf5d19d241058659d3e56b7b516100a7db9720c6869a04b36528d297fe3
> #
> # criu restore -D /tmp/img.vfs -o restore.log -v4 -d --pidfile
> /tmp/img.vfs/restore.pid \
> --root
> /var/lib/docker/vfs/dir/b3438bf5d19d241058659d3e56b7b516100a7db9720c6869a04b36528d297fe3
> \
> --ext-mount-map /etc/resolv.conf:/etc/resolv.conf \
> --ext-mount-map
> /etc/hostname:/var/lib/docker/containers/b3438bf5d19d241058659d3e56b7b516100a7db9720c6869a04b36528d297fe3/hostname
> \
> --ext-mount-map
> /etc/hosts:/var/lib/docker/containers/b3438bf5d19d241058659d3e56b7b516100a7db9720c6869a04b36528d297fe3/hosts
> #
> # echo $?
> 0
> #
> # umount
> /var/lib/docker/vfs/dir/b3438bf5d19d241058659d3e56b7b516100a7db9720c6869a04b36528d297fe3
> #
> #
> # ps -efl | grep /bin/sh
> 5 S root 30478 1 0 80 0 - 791 wait 10:12 ? 00:00:00
> /bin/sh -c i=0; while true; do echo $i >> /foo; i=`expr $i + 1`; sleep 3;
> done
> 0 S root 30524 26797 0 80 0 - 2936 pipe_w 10:12 pts/2 00:00:00
> grep --color=auto /bin/sh
> #
>
>
>
> On Tue, Aug 5, 2014 at 10:47 PM, Pradeep Padala <ppadala at gmail.com> wrote:
>
>> Hi Saied,
>>
>> Thanks for your detailed reply. I changed the storage driver to vfs, but
>> still getting the same error. Am I missing something here?
>>
>> $ ./criu --version
>> Version: 1.3-rc2
>> GitID: 5cb0c0d
>>
>> $ docker info
>> Containers: 2
>> Images: 3
>> Storage Driver: vfs
>> Execution Driver: native-0.2
>> Kernel Version: 3.15.7-200.fc20.x86_64
>>
>> $ ./criu dump -t 1473 -D /tmp/docker
>> Error (mount.c:449): 147:./dev/console doesn't have a proper root mount
>> Error (cr-dump.c:1914): Dumping FAILED.
>>
>> Regards,
>> Pradeep
>>
>>
>> On Tue, Aug 5, 2014 at 9:35 PM, Saied Kazemi <saied at google.com> wrote:
>>
>>> Hi Pradeep,
>>>
>>> The official response should come from the CRIU development team but
>>> since I have been looking at this issue I can provide some info.
>>>
>>> Dumping and restoring Docker containers using CRIU as-is is not possible
>>> at the moment.
>>>
>>> As you noted, work is in progress, but it will take some time to finish.
>>> So far, support for external bind mounts (e.g., /etc/hosts for Docker
>>> containers) and restoration of control groups has been added and is already
>>> available in 1.3-rc2.
>>>
>>> If you use the VFS graph driver (as opposed to the default AUFS), it's
>>> possible to dump and restore a container with CRIU using the external bind
>>> mount options. However, currently there is no way to tell Docker that a
>>> container was restored, so "docker ps" would show its status as exited.
>>>
>>> Work is underway to add AUFS support to CRIU as well as
>>> checkpoint/restore support in libcontainer and Docker. Ultimately
>>> checkpointing and restoring Docker containers should be done by Docker
>>> itself (e.g., docker checkpoint, docker restore) with a call to CRIU for
>>> doing the actual checkpoint and restore operation.
>>>
>>> Hope this helps...
>>>
>>> --Saied
>>>
>>>
>>>
>>>
>>> On Tue, Aug 5, 2014 at 3:19 PM, Pradeep Padala <ppadala at gmail.com>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> I tried checkpointing the docker container with the latest version of
>>>> criu and got the following error.
>>>>
>>>> ./criu dump -t 15814 --images-dir /tmp/docker
>>>> Error (mount.c:449): 225:./dev/console doesn't have a proper root mount
>>>> Error (cr-dump.c:1882): Dumping FAILED.
>>>>
>>>> I saw that there's been some progress on this lately and was not sure,
>>>> if this has been fixed yet. Is there a timeframe on when the docker
>>>> container c/r will be supported?
>>>>
>>>> Let me know.
>>>>
>>>> Thanks,
>>>> Pradeep
>>>>
>>>> _______________________________________________
>>>> CRIU mailing list
>>>> CRIU at openvz.org
>>>> https://lists.openvz.org/mailman/listinfo/criu
>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openvz.org/pipermail/criu/attachments/20140806/8b3b159a/attachment-0001.html>
More information about the CRIU
mailing list