<div dir="ltr">Here is a new patch. I rebased to the head (commit ded04267f8) and cleaned up the code a bit more. Please use the attached patch instead of the one I sent yesterday.<div><br></div><div>Note that I had to delete 3 lines from cgroup.c for properties that I don't have on my Ubuntu 14.04. Also, please note that you have to add --manage-cgroups to criu command line for both dump and restore.</div>
<div><br></div><div>--Saied</div><div><br></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Mon, Aug 11, 2014 at 4:28 PM, Saied Kazemi <span dir="ltr"><<a href="mailto:saied@google.com" target="_blank">saied@google.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi Pavel,<div><br></div><div>Attached please find a patch file for supporting AUFS in CRIU. The main reason that we need the patch is to compensate for missing and/or erroneous information we get from the kernel in /proc/mountinfo and /proc/<PID>/map_files (more details below). Once these are corrected, we can revert the patch.</div>
<div><br></div><div>I tried to minimize the changes to CRIU's code, so the majority of the code is in a new file, sysfs_parse.c. Also, please consider this as a quick workaround to get CRIU to dump and restore Docker containers that use AUFS (default graph driver). Feel free to make any changes.</div>
<div><br></div><div>Here are mountinfo and map_files issues that I ran into:</div>
<div><br></div><div>1) /proc/mountinfo</div><div><br></div><div>The issue is that for AUFS the root entry looks something like: </div><div><br></div><div>90 61 0:33 / / rw,relatime - aufs none rw,si=4476a910a24617e6</div>
<div><br>
</div><div>Compared to VFS (which CRIU supports with no issues), the root field is missing the pathname (it's only /) and its device, source, and options do not match the underlying device/fs values causing external bind mounts like /etc/hosts not to be mounted upon restore (mounts_equal() fails). To compensate for these, I specify the root pathname with --aufs-root option and a "reference" file with the --aufs-ref option to "fix up" the fields. There must be a better way to do this, but as I hope this will be temporary code I didn't spend too much time investigating.</div>
<div><br></div><div>2) /proc/<PID>/map_files</div><div><br></div><div>The issue is that symbolic links point to the internals of AUFS, namely the absolute pathnames in AUFS branches (versus pathnames from the root of the mount namespace). Below is an example:</div>
<div><br></div><div>
<p>lr-------- 1 root root 64 Jul 23 17:15 400000-489000 -> /var/lib/docker/aufs/diff/<ID>/bin/busybox</p><p>where /var/lib/docker/aufs/diff/<ID> is an AUFS branch. The link should point to /bin/busybox instead. To compensate for this, I specify the --aufs option to parse branch pathnames through /sys/fs/aufs and replace them with pathnames from the root during lookups (i.e., stat(), fstat()).<br>
</p><p>The good news is that when we do this during dump, we won't have to specify any AUFS options during restore.</p><p>The patch is against commit 7a203afe0a4 from August 7th. When I rebased to the head today, I got fatal error messages from the recently added cgroups code so I reverted.</p>
<p>Below is the log of my test. Please let me know what you think.</p><p>Thanks,</p><p>--Saied</p><p><br></p><p># docker run -d busybox:latest /bin/sh -c 'i=0; while true; do echo $i >> /foo; i=`expr $i + 1`; sleep 3; done'</p>
<p><ID></p><p># ps -efl | grep /bin/sh<br></p><p>4 S root 4423 27064 0 80 0 - 791 wait 16:00 ? 00:00:00 /bin/sh -c i=0; while true; do echo $i >> /foo; i=`expr $i + 1`; sleep 3; done</p>
<p>0 S root 4475 27195 0 80 0 - 2936 pipe_w 16:01 pts/4 00:00:00 grep --color=auto /bin/sh</p><p># criu dump -D /tmp/img.aufs -o dump.log -v4 --evasive-devices --ext-mount-map /etc/resolv.conf:/etc/resolv.conf --ext-mount-map /etc/hostname:/etc/hostname --ext-mount-map /etc/hosts:/etc/hosts --aufs --aufs-root /var/lib/docker/aufs/mnt/<ID> --aufs-ref /etc/hosts -t 4423<br>
</p><p># grep "finished successfully" /tmp/img.aufs/dump.log<br></p><p>(00.029719) Dumping finished successfully</p><p># ps -efl | grep /bin/sh<br></p><p>0 S root 4532 27195 0 80 0 - 2936 pipe_w 16:01 pts/4 00:00:00 grep --color=auto /bin/sh</p>
<p># mount -t aufs -o br=/var/lib/docker/aufs/diff/<ID> \<br></p><p>/var/lib/docker/aufs/diff/<ID>-init \</p><p>/var/lib/docker/aufs/diff/<BRID1> \</p><p>/var/lib/docker/aufs/diff/<BRID2> \</p>
<p>/var/lib/docker/aufs/diff/<BRID3> \</p><p>/var/lib/docker/aufs/diff/<BDID4> \</p><p>none /var/lib/docker/aufs/mnt/<ID></p><p># criu restore -D /tmp/img.aufs -o restore.log -v4 -d --root /var/lib/docker/aufs/mnt/<ID> --pidfile /tmp/img.aufs/restore.pid --ext-mount-map /etc/resolv.conf:/etc/resolv.conf --ext-mount-map /etc/hostname:/var/lib/docker/containers/<ID>/hostname --ext-mount-map /etc/hosts:/var/lib/docker/containers/<ID>/hosts <br>
</p><p># grep "finished successfully" /tmp/img.aufs/restore.log<br></p><p>(00.451924) Restore finished successfully. Resuming tasks.</p><p># ps -efl | grep /bin/sh<br></p><p>5 S root 4569 1 0 80 0 - 791 wait 16:01 ? 00:00:00 /bin/sh -c i=0; while true; do echo $i >> /foo; i=`expr $i + 1`; sleep 3; done</p>
<p>0 S root 4594 27195 0 80 0 - 2936 pipe_w 16:01 pts/4 00:00:00 grep --color=auto /bin/sh</p><p><br></p>
</div></div>
</blockquote></div><br></div>