[CRIU] Not able to checkpoint docker container with criu-1.3-rc2
Pradeep Padala
ppadala at gmail.com
Fri Aug 8 13:42:09 PDT 2014
Quick update. I tried latest code from git master branch, and it seems like
it's dumping a few files. But, the command hangs and never returns. Don't
see much in the log.
I am guessing there are still bugs in this, will check back later to see
how this works.
Regards,
P
On Wed, Aug 6, 2014 at 12:49 PM, Pradeep Padala <ppadala at gmail.com> wrote:
> Thanks for the detailed commands. It doesn't give an error right away, but
> when I check the dump.log, I see an error. I used the command below.
>
> $./criu dump -D /tmp/docker -o dump.log -v4 --ext-mount-map
> /etc/resolv.conf:/etc/resolv.conf --ext-mount-map
> /etc/hostname:/etc/hostname --ext-mount-map /etc/hosts:/etc/hosts
> --evasive-devices -t 2140
>
> $cat /tmp/docker/dump.log
> (00.000219) ========================================
> (00.000270) Dumping processes (pid: 2140)
> (00.000275) ========================================
> (00.003150) Found anon-shmem device at 4
> (00.003198) Reset 6112's dirty tracking
> (00.004367) ... done
> (00.004441) Dirty track supported on kernel
> (00.004483) irmap: Searching irmap cache in work dir
> (00.004514) irmap: Searching irmap cache in parent
> (00.004535) irmap: No irmap cache
> (00.004821) cpu: fpu:1 fxsr:1 xsave:1
> (00.005014) vdso: Parsing at 7fff65bad000 7fff65baf000
> (00.005027) vdso: PT_LOAD p_vaddr: 0
> (00.005030) vdso: DT_HASH: 0x120
> (00.005033) vdso: DT_STRTAB: 0x268
> (00.005035) vdso: DT_SYMTAB: 0x160
> (00.005037) vdso: DT_STRSZ: 94
> (00.005040) vdso: DT_SYMENT: 24
> (00.005042) vdso: nbucket 3 nchain 11 bucket 0x7fff65bad128 chain
> 0x7fff65bad134
> (00.005050) vdso: rt [vdso] 7fff65bad000-7fff65baf000 [vvar]
> ffffffffffffffff-ffffffffffffffff
> (00.005238) Writing image inventory (version 1)
> (00.005434) Collected 1.pid namespace
> (00.005459) Collected 2.net namespace
> (00.005471) Collected 3.ipc namespace
> (00.005483) Collected 4.uts namespace
> (00.005498) Collected 5.mnt namespace
> (00.005503) cg: Dumping cgroups for 6112
> (00.005585) cg: `- New css ID 1
> (00.005593) cg: `- [cpuset] -> [/]
> (00.005596) cg: `- [devices] -> [/]
> (00.005598) cg: `- [freezer] -> [/]
> (00.005600) cg: `- [hugetlb] -> [/]
> (00.005602) cg: `- [memory] -> [/]
> (00.005605) cg: `- [net_cls,net_prio] -> [/]
> (00.005607) cg: `- [perf_event] -> [/]
> (00.005609) cg: `- [blkio] -> [/user.slice]
> (00.005612) cg: `- [cpu,cpuacct] -> [/user.slice]
> (00.005614) cg: `- [name=systemd] ->
> [/user.slice/user-0.slice/session-18.scope]
> (00.005617) cg: Set 1 is criu one
> (00.007028) Seized task 2140, state 1
> (00.007213) Collected 2140 in 1 state
> (00.007493) Will take pid namespace in the image
> (00.007500) Collected 6.pid namespace
> (00.007510) Will take net namespace in the image
> (00.007513) Collected 7.net namespace
> (00.007521) Will take ipc namespace in the image
> (00.007524) Collected 8.ipc namespace
> (00.007532) Will take uts namespace in the image
> (00.007535) Collected 9.uts namespace
> (00.007542) Will take mnt namespace in the image
> (00.007545) Collected 10.mnt namespace
> (00.007548) Lock network
> (00.007551) Running network-lock scripts
> (00.007684) Dump MNT namespace (mountpoints) 10 via 2140
> (00.007827) type unsupported source
> /dev/mapper/fedora_wdc--drm--vm--dhcp423-root fd00000
> /var/lib/docker/vfs/dir/0895cf3a6f52123e5ef6cf81db662607ebd90c9c872b816bd7cf97188cfbcf5a
> @ ./ flags 200000 options seclabel,data=ordered,
> (00.007847) type proc source proc 25 / @ ./proc flags 20000e options
> (00.007859) type sysfs source sysfs 26 / @ ./sys flags 200001 options
> seclabel,
> (00.007873) type tmpfs source tmpfs 27 / @ ./dev flags 2 options
> context="system_u:object_r:svirt_sandbox_file_t:s0:c275,c327",mode=755,
> (00.007886) type tmpfs source shm 28 / @ ./dev/shm flags 20000e
> options
> context="system_u:object_r:svirt_sandbox_file_t:s0:c275,c327",size=65536k,
> (00.007905) type devpts source devpts 29 / @ ./dev/pts flags 20000a
> options
> context="system_u:object_r:svirt_sandbox_file_t:s0:c275,c327",gid=5,mode=620,ptmxmode=666,
> (00.008187) type unsupported source
> /dev/mapper/fedora_wdc--drm--vm--dhcp423-root fd00000
> /var/lib/docker/init/dockerinit-1.0.0 @ ./.dockerinit flags 200001 options
> seclabel,data=ordered,
> (00.008252) type unsupported source
> /dev/mapper/fedora_wdc--drm--vm--dhcp423-root fd00000 /etc/resolv.conf @
> ./etc/resolv.conf flags 200001 options seclabel,data=ordered,
> (00.008267) type unsupported source
> /dev/mapper/fedora_wdc--drm--vm--dhcp423-root fd00000
> /var/lib/docker/containers/0895cf3a6f52123e5ef6cf81db662607ebd90c9c872b816bd7cf97188cfbcf5a/hostname
> @ ./etc/hostname flags 200001 options seclabel,data=ordered,
> (00.008290) type unsupported source
> /dev/mapper/fedora_wdc--drm--vm--dhcp423-root fd00000
> /var/lib/docker/containers/0895cf3a6f52123e5ef6cf81db662607ebd90c9c872b816bd7cf97188cfbcf5a/hosts
> @ ./etc/hosts flags 200001 options seclabel,data=ordered,
> (00.008305) type devpts source devpts b /1 @ ./dev/console flags
> 20000a options seclabel,gid=5,mode=620,ptmxmode=000,
> (00.008331) type proc source proc 25 /sys @ ./proc/sys flags 20000f
> options
> (00.008344) type proc source proc 25 /sysrq-trigger @
> ./proc/sysrq-trigger flags 20000f options
> (00.008356) type proc source proc 25 /irq @ ./proc/irq flags 20000f
> options
> (00.008366) type proc source proc 25 /bus @ ./proc/bus flags 20000f
> options
> (00.008378) type tmpfs source tmpfs 27 /null @ ./proc/kcore flags 2
> options
> context="system_u:object_r:svirt_sandbox_file_t:s0:c275,c327",mode=755,
> (00.008409) Building mountpoints tree
> (00.008416) Building plain mount tree
> (00.008419) Working on 113->138
> (00.008422) Working on 112->138
> (00.008424) Working on 111->138
> (00.008426) Working on 109->138
> (00.008428) Working on 108->138
> (00.008431) Working on 147->140
> (00.008433) Working on 146->137
> (00.008435) Working on 145->137
> (00.008438) Working on 144->137
> (00.008440) Working on 143->137
> (00.008442) Working on 142->140
> (00.008445) Working on 141->140
> (00.008447) Working on 140->137
> (00.008449) Working on 139->137
> (00.008451) Working on 138->137
> (00.008454) Working on 137->106
> (00.008456) Resorting siblings on 137
> (00.008459) Resorting siblings on 146
> (00.008461) Resorting siblings on 145
> (00.008463) Resorting siblings on 144
> (00.008466) Resorting siblings on 143
> (00.008468) Resorting siblings on 140
> (00.008470) Resorting siblings on 147
> (00.008473) Resorting siblings on 142
> (00.008475) Resorting siblings on 141
> (00.008477) Resorting siblings on 139
> (00.008479) Resorting siblings on 138
> (00.008482) Resorting siblings on 113
> (00.008484) Resorting siblings on 112
> (00.008486) Resorting siblings on 111
> (00.008488) Resorting siblings on 109
> (00.008490) Resorting siblings on 108
> (00.008496) Done:
> (00.008498) [./](137->106)
> (00.008501) [./.dockerinit](143->137)
> (00.008504) <--
> (00.008507) [./proc](138->137)
> (00.008510) [./proc/kcore](113->138)
> (00.008512) <--
> (00.008514) [./proc/sys](108->138)
> (00.008517) <--
> (00.008519) [./proc/sysrq-trigger](109->138)
> (00.008522) <--
> (00.008524) [./proc/irq](111->138)
> (00.008527) <--
> (00.008529) [./proc/bus](112->138)
> (00.008531) <--
> (00.008534) <--
> (00.008536) [./sys](139->137)
> (00.008538) <--
> (00.008540) [./dev](140->137)
> (00.008543) [./dev/console](147->140)
> (00.008546) <--
> (00.008548) [./dev/shm](141->140)
> (00.008550) <--
> (00.008552) [./dev/pts](142->140)
> (00.008555) <--
> (00.008557) <--
> (00.008559) [./etc/hosts](146->137)
> (00.008562) <--
> (00.008564) [./etc/resolv.conf](144->137)
> (00.008567) <--
> (00.008569) [./etc/hostname](145->137)
> (00.008572) <--
> (00.008574) <--
> (00.008584) Error (mount.c:449): 147:./dev/console doesn't have a proper
> root mount
> (00.008590) Unlock network
> (00.008597) Running network-unlock scripts
> (00.008606) Unfreezing tasks into 1
> (00.008610) Unseizing 2140 into 1
> (00.008972) Error (cr-dump.c:1914): Dumping FAILED.
>
>
>
> On Wed, Aug 6, 2014 at 10:24 AM, Saied Kazemi <saied at google.com> wrote:
>
>> That's because your command line is incomplete. Also, note that you need
>> to bind mount the root before doing "criu restore". Below is how I do it.
>> You can optionally run "sudo tail -f
>> /var/lib/docker/vfs/dir/b3438bf5d19d241058659d3e56b7b516100a7db9720c6869a04b36528d297fe3/foo"
>> and actually see the output stop after dump and restart after restore.
>>
>> --Saied
>>
>> # docker run -d busybox:latest /bin/sh -c 'i=1; while true; do echo $i >>
>> /foo; i=`expr $i + 1`; sleep 3; done'
>> b3438bf5d19d241058659d3e56b7b516100a7db9720c6869a04b36528d297fe3
>> #
>> # ps -efl | grep /bin/sh
>> 4 S root 30210 29160 0 80 0 - 791 wait 10:08 ?
>> 00:00:00 /bin/sh -c i=0; while true; do echo $i >> /foo; i=`expr $i + 1`;
>> sleep 3; done
>> 0 S root 30247 26797 0 80 0 - 2936 pipe_w 10:08 pts/2
>> 00:00:00 grep --color=auto /bin/sh
>> #
>> # criu dump -D /tmp/img.vfs -o dump.log -v4 --ext-mount-map
>> /etc/resolv.conf:/etc/resolv.conf --ext-mount-map
>> /etc/hostname:/etc/hostname --ext-mount-map /etc/hosts:/etc/hosts
>> --evasive-devices -t 30210
>> #
>> # echo $?
>> 0
>> #
>> # ps -efl | grep /bin/sh
>> 0 S root 30425 26797 0 80 0 - 2936 pipe_w 10:11 pts/2
>> 00:00:00 grep --color=auto /bin/sh
>> #
>> # mount --rbind
>> /var/lib/docker/vfs/dir/b3438bf5d19d241058659d3e56b7b516100a7db9720c6869a04b36528d297fe3
>> /var/lib/docker/vfs/dir/b3438bf5d19d241058659d3e56b7b516100a7db9720c6869a04b36528d297fe3
>> #
>> # criu restore -D /tmp/img.vfs -o restore.log -v4 -d --pidfile
>> /tmp/img.vfs/restore.pid \
>> --root
>> /var/lib/docker/vfs/dir/b3438bf5d19d241058659d3e56b7b516100a7db9720c6869a04b36528d297fe3
>> \
>> --ext-mount-map /etc/resolv.conf:/etc/resolv.conf \
>> --ext-mount-map
>> /etc/hostname:/var/lib/docker/containers/b3438bf5d19d241058659d3e56b7b516100a7db9720c6869a04b36528d297fe3/hostname
>> \
>> --ext-mount-map
>> /etc/hosts:/var/lib/docker/containers/b3438bf5d19d241058659d3e56b7b516100a7db9720c6869a04b36528d297fe3/hosts
>> #
>> # echo $?
>> 0
>> #
>> # umount
>> /var/lib/docker/vfs/dir/b3438bf5d19d241058659d3e56b7b516100a7db9720c6869a04b36528d297fe3
>> #
>> #
>> # ps -efl | grep /bin/sh
>> 5 S root 30478 1 0 80 0 - 791 wait 10:12 ?
>> 00:00:00 /bin/sh -c i=0; while true; do echo $i >> /foo; i=`expr $i + 1`;
>> sleep 3; done
>> 0 S root 30524 26797 0 80 0 - 2936 pipe_w 10:12 pts/2
>> 00:00:00 grep --color=auto /bin/sh
>> #
>>
>>
>>
>> On Tue, Aug 5, 2014 at 10:47 PM, Pradeep Padala <ppadala at gmail.com>
>> wrote:
>>
>>> Hi Saied,
>>>
>>> Thanks for your detailed reply. I changed the storage driver to vfs, but
>>> still getting the same error. Am I missing something here?
>>>
>>> $ ./criu --version
>>> Version: 1.3-rc2
>>> GitID: 5cb0c0d
>>>
>>> $ docker info
>>> Containers: 2
>>> Images: 3
>>> Storage Driver: vfs
>>> Execution Driver: native-0.2
>>> Kernel Version: 3.15.7-200.fc20.x86_64
>>>
>>> $ ./criu dump -t 1473 -D /tmp/docker
>>> Error (mount.c:449): 147:./dev/console doesn't have a proper root mount
>>> Error (cr-dump.c:1914): Dumping FAILED.
>>>
>>> Regards,
>>> Pradeep
>>>
>>>
>>> On Tue, Aug 5, 2014 at 9:35 PM, Saied Kazemi <saied at google.com> wrote:
>>>
>>>> Hi Pradeep,
>>>>
>>>> The official response should come from the CRIU development team but
>>>> since I have been looking at this issue I can provide some info.
>>>>
>>>> Dumping and restoring Docker containers using CRIU as-is is not
>>>> possible at the moment.
>>>>
>>>> As you noted, work is in progress, but it will take some time to
>>>> finish. So far, support for external bind mounts (e.g., /etc/hosts for
>>>> Docker containers) and restoration of control groups has been added and is
>>>> already available in 1.3-rc2.
>>>>
>>>> If you use the VFS graph driver (as opposed to the default AUFS), it's
>>>> possible to dump and restore a container with CRIU using the external bind
>>>> mount options. However, currently there is no way to tell Docker that a
>>>> container was restored, so "docker ps" would show its status as exited.
>>>>
>>>> Work is underway to add AUFS support to CRIU as well as
>>>> checkpoint/restore support in libcontainer and Docker. Ultimately
>>>> checkpointing and restoring Docker containers should be done by Docker
>>>> itself (e.g., docker checkpoint, docker restore) with a call to CRIU for
>>>> doing the actual checkpoint and restore operation.
>>>>
>>>> Hope this helps...
>>>>
>>>> --Saied
>>>>
>>>>
>>>>
>>>>
>>>> On Tue, Aug 5, 2014 at 3:19 PM, Pradeep Padala <ppadala at gmail.com>
>>>> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I tried checkpointing the docker container with the latest version of
>>>>> criu and got the following error.
>>>>>
>>>>> ./criu dump -t 15814 --images-dir /tmp/docker
>>>>> Error (mount.c:449): 225:./dev/console doesn't have a proper root mount
>>>>> Error (cr-dump.c:1882): Dumping FAILED.
>>>>>
>>>>> I saw that there's been some progress on this lately and was not sure,
>>>>> if this has been fixed yet. Is there a timeframe on when the docker
>>>>> container c/r will be supported?
>>>>>
>>>>> Let me know.
>>>>>
>>>>> Thanks,
>>>>> Pradeep
>>>>>
>>>>> _______________________________________________
>>>>> CRIU mailing list
>>>>> CRIU at openvz.org
>>>>> https://lists.openvz.org/mailman/listinfo/criu
>>>>>
>>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openvz.org/pipermail/criu/attachments/20140808/6a791cbe/attachment-0001.html>
More information about the CRIU
mailing list