<div dir="ltr">Quick update. I tried latest code from git master branch, and it seems like it's dumping a few files. But, the command hangs and never returns. Don't see much in the log.<div><br></div><div>I am guessing there are still bugs in this, will check back later to see how this works.</div>
<div><br>Regards,</div><div>P</div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Wed, Aug 6, 2014 at 12:49 PM, Pradeep Padala <span dir="ltr"><<a href="mailto:ppadala@gmail.com" target="_blank">ppadala@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Thanks for the detailed commands. It doesn't give an error right away, but when I check the dump.log, I see an error. I used the command below.<div>
<br></div><div>$./criu dump -D /tmp/docker -o dump.log -v4 --ext-mount-map /etc/resolv.conf:/etc/resolv.conf --ext-mount-map /etc/hostname:/etc/hostname --ext-mount-map /etc/hosts:/etc/hosts --evasive-devices -t 2140</div>
<div><br><div>$cat /tmp/docker/dump.log</div><div><div>(00.000219) ========================================</div><div>(00.000270) Dumping processes (pid: 2140)</div><div>(00.000275) ========================================</div>
<div>(00.003150) Found anon-shmem device at 4</div><div>(00.003198) Reset 6112's dirty tracking</div><div>(00.004367) ... done</div><div>(00.004441) Dirty track supported on kernel</div><div>(00.004483) irmap: Searching irmap cache in work dir</div>
<div>(00.004514) irmap: Searching irmap cache in parent</div><div>(00.004535) irmap: No irmap cache</div><div>(00.004821) cpu: fpu:1 fxsr:1 xsave:1</div><div>(00.005014) vdso: Parsing at 7fff65bad000 7fff65baf000</div><div>
(00.005027) vdso: PT_LOAD p_vaddr: 0</div><div>(00.005030) vdso: DT_HASH: 0x120</div><div>(00.005033) vdso: DT_STRTAB: 0x268</div><div>(00.005035) vdso: DT_SYMTAB: 0x160</div><div>(00.005037) vdso: DT_STRSZ: 94</div><div>
(00.005040) vdso: DT_SYMENT: 24</div><div>(00.005042) vdso: nbucket 3 nchain 11 bucket 0x7fff65bad128 chain 0x7fff65bad134</div><div>(00.005050) vdso: rt [vdso] 7fff65bad000-7fff65baf000 [vvar] ffffffffffffffff-ffffffffffffffff</div>
<div>(00.005238) Writing image inventory (version 1)</div><div>(00.005434) Collected 1.pid namespace</div><div>(00.005459) Collected <a href="http://2.net" target="_blank">2.net</a> namespace</div><div>(00.005471) Collected 3.ipc namespace</div>
<div>(00.005483) Collected 4.uts namespace</div><div>(00.005498) Collected 5.mnt namespace</div><div>(00.005503) cg: Dumping cgroups for 6112</div><div>(00.005585) cg: `- New css ID 1</div><div>(00.005593) cg: `- [cpuset] -> [/]</div>
<div>(00.005596) cg: `- [devices] -> [/]</div><div>(00.005598) cg: `- [freezer] -> [/]</div><div>(00.005600) cg: `- [hugetlb] -> [/]</div><div>(00.005602) cg: `- [memory] -> [/]</div><div>(00.005605) cg: `- [net_cls,net_prio] -> [/]</div>
<div>(00.005607) cg: `- [perf_event] -> [/]</div><div>(00.005609) cg: `- [blkio] -> [/user.slice]</div><div>(00.005612) cg: `- [cpu,cpuacct] -> [/user.slice]</div><div>(00.005614) cg: `- [name=systemd] -> [/user.slice/user-0.slice/session-18.scope]</div>
<div>(00.005617) cg: Set 1 is criu one</div><div>(00.007028) Seized task 2140, state 1</div><div>(00.007213) Collected 2140 in 1 state</div><div>(00.007493) Will take pid namespace in the image</div><div>(00.007500) Collected 6.pid namespace</div>
<div>(00.007510) Will take net namespace in the image</div><div>(00.007513) Collected <a href="http://7.net" target="_blank">7.net</a> namespace</div><div>(00.007521) Will take ipc namespace in the image</div><div>(00.007524) Collected 8.ipc namespace</div>
<div>(00.007532) Will take uts namespace in the image</div><div>(00.007535) Collected 9.uts namespace</div><div>(00.007542) Will take mnt namespace in the image</div><div>(00.007545) Collected 10.mnt namespace</div><div>
(00.007548) Lock network</div>
<div>(00.007551) Running network-lock scripts</div><div>(00.007684) Dump MNT namespace (mountpoints) 10 via 2140</div><div>(00.007827) type unsupported source /dev/mapper/fedora_wdc--drm--vm--dhcp423-root fd00000 /var/lib/docker/vfs/dir/0895cf3a6f52123e5ef6cf81db662607ebd90c9c872b816bd7cf97188cfbcf5a @ ./ flags 200000 options seclabel,data=ordered,</div>
<div>(00.007847) type proc source proc 25 / @ ./proc flags 20000e options</div><div>(00.007859) type sysfs source sysfs 26 / @ ./sys flags 200001 options seclabel,</div><div>(00.007873) type tmpfs source tmpfs 27 / @ ./dev flags 2 options context="system_u:object_r:svirt_sandbox_file_t:s0:c275,c327",mode=755,</div>
<div>(00.007886) type tmpfs source shm 28 / @ ./dev/shm flags 20000e options context="system_u:object_r:svirt_sandbox_file_t:s0:c275,c327",size=65536k,</div><div>(00.007905) type devpts source devpts 29 / @ ./dev/pts flags 20000a options context="system_u:object_r:svirt_sandbox_file_t:s0:c275,c327",gid=5,mode=620,ptmxmode=666,</div>
<div>(00.008187) type unsupported source /dev/mapper/fedora_wdc--drm--vm--dhcp423-root fd00000 /var/lib/docker/init/dockerinit-1.0.0 @ ./.dockerinit flags 200001 options seclabel,data=ordered,</div><div>(00.008252) type unsupported source /dev/mapper/fedora_wdc--drm--vm--dhcp423-root fd00000 /etc/resolv.conf @ ./etc/resolv.conf flags 200001 options seclabel,data=ordered,</div>
<div>(00.008267) type unsupported source /dev/mapper/fedora_wdc--drm--vm--dhcp423-root fd00000 /var/lib/docker/containers/0895cf3a6f52123e5ef6cf81db662607ebd90c9c872b816bd7cf97188cfbcf5a/hostname @ ./etc/hostname flags 200001 options seclabel,data=ordered,</div>
<div>(00.008290) type unsupported source /dev/mapper/fedora_wdc--drm--vm--dhcp423-root fd00000 /var/lib/docker/containers/0895cf3a6f52123e5ef6cf81db662607ebd90c9c872b816bd7cf97188cfbcf5a/hosts @ ./etc/hosts flags 200001 options seclabel,data=ordered,</div>
<div>(00.008305) type devpts source devpts b /1 @ ./dev/console flags 20000a options seclabel,gid=5,mode=620,ptmxmode=000,</div><div>(00.008331) type proc source proc 25 /sys @ ./proc/sys flags 20000f options</div>
<div>(00.008344) type proc source proc 25 /sysrq-trigger @ ./proc/sysrq-trigger flags 20000f options</div><div>(00.008356) type proc source proc 25 /irq @ ./proc/irq flags 20000f options</div><div>(00.008366) type proc source proc 25 /bus @ ./proc/bus flags 20000f options</div>
<div>(00.008378) type tmpfs source tmpfs 27 /null @ ./proc/kcore flags 2 options context="system_u:object_r:svirt_sandbox_file_t:s0:c275,c327",mode=755,</div><div>(00.008409) Building mountpoints tree</div>
<div>
(00.008416) Building plain mount tree</div><div>(00.008419) Working on 113->138</div><div>(00.008422) Working on 112->138</div><div>(00.008424) Working on 111->138</div><div>
(00.008426) Working on 109->138</div><div>(00.008428) Working on 108->138</div><div>(00.008431) Working on 147->140</div><div>(00.008433) Working on 146->137</div>
<div>(00.008435) Working on 145->137</div><div>(00.008438) Working on 144->137</div><div>(00.008440) Working on 143->137</div><div>(00.008442) Working on 142->140</div>
<div>(00.008445) Working on 141->140</div><div>(00.008447) Working on 140->137</div><div>(00.008449) Working on 139->137</div><div>(00.008451) Working on 138->137</div>
<div>(00.008454) Working on 137->106</div><div>(00.008456) Resorting siblings on 137</div><div>(00.008459) Resorting siblings on 146</div><div>(00.008461) Resorting siblings on 145</div><div>(00.008463) Resorting siblings on 144</div>
<div>(00.008466) Resorting siblings on 143</div><div>(00.008468) Resorting siblings on 140</div><div>(00.008470) Resorting siblings on 147</div><div>(00.008473) Resorting siblings on 142</div><div>(00.008475) Resorting siblings on 141</div>
<div>(00.008477) Resorting siblings on 139</div><div>(00.008479) Resorting siblings on 138</div><div>(00.008482) Resorting siblings on 113</div><div>(00.008484) Resorting siblings on 112</div><div>(00.008486) Resorting siblings on 111</div>
<div>(00.008488) Resorting siblings on 109</div><div>(00.008490) Resorting siblings on 108</div><div>(00.008496) Done:</div><div>(00.008498) [./](137->106)</div><div>(00.008501) [./.dockerinit](143->137)</div>
<div>(00.008504) <--</div><div>(00.008507) [./proc](138->137)</div><div>(00.008510) [./proc/kcore](113->138)</div><div>(00.008512) <--</div><div>(00.008514) [./proc/sys](108->138)</div><div>(00.008517) <--</div>
<div>(00.008519) [./proc/sysrq-trigger](109->138)</div><div>(00.008522) <--</div><div>(00.008524) [./proc/irq](111->138)</div><div>(00.008527) <--</div><div>(00.008529) [./proc/bus](112->138)</div>
<div>(00.008531) <--</div><div>(00.008534) <--</div><div>(00.008536) [./sys](139->137)</div><div>(00.008538) <--</div><div>(00.008540) [./dev](140->137)</div><div>(00.008543) [./dev/console](147->140)</div>
<div>(00.008546) <--</div><div>(00.008548) [./dev/shm](141->140)</div><div>(00.008550) <--</div><div>(00.008552) [./dev/pts](142->140)</div><div>(00.008555) <--</div><div>(00.008557) <--</div>
<div>(00.008559) [./etc/hosts](146->137)</div><div>(00.008562) <--</div><div>(00.008564) [./etc/resolv.conf](144->137)</div><div>(00.008567) <--</div><div>(00.008569) [./etc/hostname](145->137)</div><div>
(00.008572) <--</div><div>(00.008574) <--</div><div>(00.008584) Error (mount.c:449): 147:./dev/console doesn't have a proper root mount</div><div>(00.008590) Unlock network</div><div>(00.008597) Running network-unlock scripts</div>
<div>(00.008606) Unfreezing tasks into 1</div><div>(00.008610) Unseizing 2140 into 1</div><div>(00.008972) Error (cr-dump.c:1914): Dumping FAILED.</div></div><div><br></div></div></div><div class="HOEnZb"><div class="h5">
<div class="gmail_extra"><br><br>
<div class="gmail_quote">On Wed, Aug 6, 2014 at 10:24 AM, Saied Kazemi <span dir="ltr"><<a href="mailto:saied@google.com" target="_blank">saied@google.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">That's because your command line is incomplete. Also, note that you need to bind mount the root before doing "criu restore". Below is how I do it. You can optionally run "sudo tail -f /var/lib/docker/vfs/dir/b3438bf5d19d241058659d3e56b7b516100a7db9720c6869a04b36528d297fe3/foo" and actually see the output stop after dump and restart after restore.<div>
<br></div><div>--Saied<br><div><br></div><div><div># docker run -d busybox:latest /bin/sh -c 'i=1; while true; do echo $i >> /foo; i=`expr $i + 1`; sleep 3; done'</div><div>b3438bf5d19d241058659d3e56b7b516100a7db9720c6869a04b36528d297fe3</div>
<div>#</div><div># ps -efl | grep /bin/sh</div><div>4 S root 30210 29160 0 80 0 - 791 wait 10:08 ? 00:00:00 /bin/sh -c i=0; while true; do echo $i >> /foo; i=`expr $i + 1`; sleep 3; done</div><div>
0 S root 30247 26797 0 80 0 - 2936 pipe_w 10:08 pts/2 00:00:00 grep --color=auto /bin/sh</div><div># </div><div># criu dump -D /tmp/img.vfs -o dump.log -v4 --ext-mount-map /etc/resolv.conf:/etc/resolv.conf --ext-mount-map /etc/hostname:/etc/hostname --ext-mount-map /etc/hosts:/etc/hosts --evasive-devices -t 30210</div>
<div>#</div><div># echo $?</div><div>0</div><div>#</div><div># ps -efl | grep /bin/sh</div><div>0 S root 30425 26797 0 80 0 - 2936 pipe_w 10:11 pts/2 00:00:00 grep --color=auto /bin/sh</div><div>#</div><div># mount --rbind /var/lib/docker/vfs/dir/b3438bf5d19d241058659d3e56b7b516100a7db9720c6869a04b36528d297fe3 /var/lib/docker/vfs/dir/b3438bf5d19d241058659d3e56b7b516100a7db9720c6869a04b36528d297fe3</div>
<div>#</div><div># criu restore -D /tmp/img.vfs -o restore.log -v4 -d --pidfile /tmp/img.vfs/restore.pid \</div><div> --root /var/lib/docker/vfs/dir/b3438bf5d19d241058659d3e56b7b516100a7db9720c6869a04b36528d297fe3 \</div>
<div> --ext-mount-map /etc/resolv.conf:/etc/resolv.conf \</div><div> --ext-mount-map /etc/hostname:/var/lib/docker/containers/b3438bf5d19d241058659d3e56b7b516100a7db9720c6869a04b36528d297fe3/hostname \</div>
<div> --ext-mount-map /etc/hosts:/var/lib/docker/containers/b3438bf5d19d241058659d3e56b7b516100a7db9720c6869a04b36528d297fe3/hosts</div><div>#</div><div># echo $?</div><div>0</div><div>#</div><div># umount /var/lib/docker/vfs/dir/b3438bf5d19d241058659d3e56b7b516100a7db9720c6869a04b36528d297fe3</div>
<div>#</div><div>#</div><div># ps -efl | grep /bin/sh</div><div>5 S root 30478 1 0 80 0 - 791 wait 10:12 ? 00:00:00 /bin/sh -c i=0; while true; do echo $i >> /foo; i=`expr $i + 1`; sleep 3; done</div>
<div>0 S root 30524 26797 0 80 0 - 2936 pipe_w 10:12 pts/2 00:00:00 grep --color=auto /bin/sh</div><div>#</div></div><div><br></div></div></div><div><div><div class="gmail_extra"><br>
<br><div class="gmail_quote">On Tue, Aug 5, 2014 at 10:47 PM, Pradeep Padala <span dir="ltr"><<a href="mailto:ppadala@gmail.com" target="_blank">ppadala@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div dir="ltr"><span style="font-family:arial,sans-serif;font-size:12.800000190734863px">Hi Saied,</span><div style="font-family:arial,sans-serif;font-size:12.800000190734863px">
<br></div><div style="font-family:arial,sans-serif;font-size:12.800000190734863px">
Thanks for your detailed reply. I changed the storage driver to vfs, but still getting the same error. Am I missing something here?</div><div style="font-family:arial,sans-serif;font-size:12.800000190734863px"><br></div>
<div style="font-family:arial,sans-serif;font-size:12.800000190734863px">
<div>$ ./criu --version</div><div>Version: 1.3-rc2</div><div>GitID: 5cb0c0d</div><div><br></div><div>$ docker info</div><div>Containers: 2</div><div>Images: 3</div><div>Storage Driver: vfs</div><div>Execution Driver: native-0.2</div>
<div>Kernel Version: 3.15.7-200.fc20.x86_64</div><div><br></div><div>$ ./criu dump -t 1473 -D /tmp/docker</div><div>Error (mount.c:449): 147:./dev/console doesn't have a proper root mount</div><div>Error (cr-dump.c:1914): Dumping FAILED.</div>
</div><div style="font-family:arial,sans-serif;font-size:12.800000190734863px"><br></div><div style="font-family:arial,sans-serif;font-size:12.800000190734863px">Regards,</div><div style="font-family:arial,sans-serif;font-size:12.800000190734863px">
Pradeep</div></div></div><div class="gmail_extra"><br><br><div class="gmail_quote"><div>On Tue, Aug 5, 2014 at 9:35 PM, Saied Kazemi <span dir="ltr"><<a href="mailto:saied@google.com" target="_blank">saied@google.com</a>></span> wrote:<br>
</div><div><div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi Pradeep,<div><br></div><div>The official response should come from the CRIU development team but since I have been looking at this issue I can provide some info.</div>
<div><br></div><div>Dumping and restoring Docker containers using CRIU as-is is not possible at the moment.</div>
<div><br></div><div>As you noted, work is in progress, but it will take some time to finish. So far, support for external bind mounts (e.g., /etc/hosts for Docker containers) and restoration of control groups has been added and is already available in 1.3-rc2.</div>
<div><br></div><div>If you use the VFS graph driver (as opposed to the default AUFS), it's possible to dump and restore a container with CRIU using the external bind mount options. However, currently there is no way to tell Docker that a container was restored, so "docker ps" would show its status as exited.</div>
<div><br></div><div>Work is underway to add AUFS support to CRIU as well as checkpoint/restore support in libcontainer and Docker. Ultimately checkpointing and restoring Docker containers should be done by Docker itself (e.g., docker checkpoint, docker restore) with a call to CRIU for doing the actual checkpoint and restore operation.</div>
<div><br></div><div>Hope this helps...</div><div><br></div><div>--Saied</div><div><br></div><div><br></div></div><div class="gmail_extra"><br><br><div class="gmail_quote"><div><div>On Tue, Aug 5, 2014 at 3:19 PM, Pradeep Padala <span dir="ltr"><<a href="mailto:ppadala@gmail.com" target="_blank">ppadala@gmail.com</a>></span> wrote:<br>
</div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div><div dir="ltr">Hi,<div><br></div><div>I tried checkpointing the docker container with the latest version of criu and got the following error.</div>
<div><br></div><div><div>./criu dump -t 15814 --images-dir /tmp/docker</div>
<div>Error (mount.c:449): 225:./dev/console doesn't have a proper root mount</div><div>Error (cr-dump.c:1882): Dumping FAILED.</div></div><div><br></div><div>I saw that there's been some progress on this lately and was not sure, if this has been fixed yet. Is there a timeframe on when the docker container c/r will be supported?</div>
<div><br></div><div>Let me know.</div><div><br>Thanks,</div><div>Pradeep</div></div>
<br></div></div>_______________________________________________<br>
CRIU mailing list<br>
<a href="mailto:CRIU@openvz.org" target="_blank">CRIU@openvz.org</a><br>
<a href="https://lists.openvz.org/mailman/listinfo/criu" target="_blank">https://lists.openvz.org/mailman/listinfo/criu</a><br>
<br></blockquote></div><br></div>
</blockquote></div></div></div><br></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>