<div dir="ltr"><div><div> criu/Makefile.crtools        |   4 +<br></div><div> criu/cr-dump.c               |  16 +++</div><div> criu/cr-restore.c            |   6 +</div><div> criu/crtools.c               |  20 +++-</div><div> criu/image-desc.c            |   4 +-</div><div> criu/image.c                 |  51 +++++++-</div><div> criu/img-remote.c            | 275 +++++++++++++++++++++++++++++++++++++++++++</div><div> criu/include/cr_options.h    |   1 +</div><div> criu/include/image.h         |   1 +</div><div> criu/include/img-remote.h    |  83 +++++++++++++</div><div> criu/include/protobuf-desc.h |   4 +</div><div> criu/page-xfer.c             |  25 +++-</div><div> criu/pagemap.c               |  53 +++++++--</div><div> criu/protobuf-desc.c         |   1 +</div><div> criu/sk-queue.c              |  33 ++----</div><div> criu/util.c                  |  31 +++--</div><div> images/Makefile              |   1 +</div><div> images/remote-image.proto    |  20 ++++</div><div> 18 files changed, 577 insertions(+), 52 deletions(-)</div><div><br></div><div>    This patch introduces the --remote option and the necessary code changes to</div><div>    support it. This leaves user the option to decide if the checkpoint data is to</div><div>    be stored on disk or sent through the network (through the image-proxy).</div><div>    The latter forwards the data to the destination node where image-cache</div><div>    receives it.</div><div>    </div><div>    The overall communication is performed as follows:</div><div>    src_node CRIU dump -&gt; (sends images through UNIX sockets) -&gt;       image-proxy</div><div>                                                                               |</div><div>                                                                               V</div><div>    dst_node: CRIU restore &lt;- (receives images through UNIX sockets)&lt;- image-cache</div><div>    </div><div>    Communication between image-proxy and image-cache is done through a single</div><div>    TCP connection.</div><div>    </div><div>    Running criu with --remote option is like this:</div><div>    </div><div>    dst_node# criu image-cache -d --port &lt;port&gt; -o /tmp/image-cache.log</div><div>    --local-cache-path &lt;local_cache_path&gt; ...</div><div>    dst_node# criu restore --remote -o /tmp/image-cache.log</div><div>    --local-cache-path &lt;local_cache_path&gt; ...</div><div>    src_node# criu image-proxy -d --port &lt;port&gt; --address &lt;dst_node&gt; -o /tmp/image-proxy.log</div><div>    --local-proxy-path &lt;local_proxy_path&gt; ...</div><div>    src_node# criu dump -t &lt;pid&gt; --remote -o /tmp/dump.log</div><div>    --local-proxy-path &lt;local_proxy_path&gt; ...</div><div>    </div><div>    Signed-off-by: Rodrigo Bruno &lt;<a href="mailto:rbruno@gsd.inesc-id.pt">rbruno@gsd.inesc-id.pt</a>&gt;</div><div>    Signed-off-by: Katerina Koukiou &lt;<a href="mailto:k.koukiou@gmail.com">k.koukiou@gmail.com</a>&gt;</div><div><br></div><div>diff --git a/criu/Makefile.crtools b/criu/Makefile.crtools</div><div>index e740798..4d21199 100644</div><div>--- a/criu/Makefile.crtools</div><div>+++ b/criu/Makefile.crtools</div><div>@@ -26,6 +26,10 @@ obj-y<span class="gmail-Apple-tab-span" style="white-space:pre">                        </span>+= files-reg.o</div><div> obj-y<span class="gmail-Apple-tab-span" style="white-space:pre">                        </span>+= fsnotify.o</div><div> obj-y<span class="gmail-Apple-tab-span" style="white-space:pre">                        </span>+= image-desc.o</div><div> obj-y<span class="gmail-Apple-tab-span" style="white-space:pre">                        </span>+= image.o</div><div>+obj-y<span class="gmail-Apple-tab-span" style="white-space:pre">                        </span>+= img-remote.o</div><div>+obj-y<span class="gmail-Apple-tab-span" style="white-space:pre">                        </span>+= img-proxy.o</div><div>+obj-y<span class="gmail-Apple-tab-span" style="white-space:pre">                        </span>+= img-cache.o</div><div>+obj-y<span class="gmail-Apple-tab-span" style="white-space:pre">                        </span>+= img-remote-proto.o</div><div> obj-y<span class="gmail-Apple-tab-span" style="white-space:pre">                        </span>+= ipc_ns.o</div><div> obj-y<span class="gmail-Apple-tab-span" style="white-space:pre">                        </span>+= irmap.o</div><div> obj-y<span class="gmail-Apple-tab-span" style="white-space:pre">                        </span>+= kcmp-ids.o</div><div>diff --git a/criu/cr-dump.c b/criu/cr-dump.c</div><div>index 5e708a3..f7a4efb 100644</div><div>--- a/criu/cr-dump.c</div><div>+++ b/criu/cr-dump.c</div><div>@@ -83,6 +83,7 @@</div><div> #include &quot;seize.h&quot;</div><div> #include &quot;fault-injection.h&quot;</div><div> #include &quot;dump.h&quot;</div><div>+#include &quot;img-remote.h&quot;</div><div> </div><div> static char loc_buf[PAGE_SIZE];</div><div> </div><div>@@ -1497,6 +1498,11 @@ int cr_pre_dump_tasks(pid_t pid)</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">        </span>struct pstree_item *item;</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">        </span>int ret = -1;</div><div> </div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>if (opts.remote &amp;&amp; push_snapshot_id() &lt; 0) {</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>pr_err(&quot;Failed to push image namespace.\n&quot;);</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>goto err;</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>}</div><div>+</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">        </span>root_item = alloc_pstree_item();</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">        </span>if (!root_item)</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">                </span>goto err;</div><div>@@ -1632,6 +1638,11 @@ static int cr_dump_finish(int ret)</div><div> </div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">        </span>close_service_fd(CR_PROC_FD_OFF);</div><div> </div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>if (opts.remote &amp;&amp; (finish_remote_dump() &lt; 0)) {</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>pr_err(&quot;Finish remote dump failed.\n&quot;);</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>return post_dump_ret ? : 1;</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>}</div><div>+</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">        </span>if (ret) {</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">                </span>pr_err(&quot;Dumping FAILED.\n&quot;);</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">        </span>} else {</div><div>@@ -1652,6 +1663,11 @@ int cr_dump_tasks(pid_t pid)</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">        </span>pr_info(&quot;Dumping processes (pid: %d)\n&quot;, pid);</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">        </span>pr_info(&quot;========================================\n&quot;);</div><div> </div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>if (opts.remote &amp;&amp; push_snapshot_id() &lt; 0) {</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>pr_err(&quot;Failed to push image namespace.\n&quot;);</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>goto err;</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>}</div><div>+</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">        </span>root_item = alloc_pstree_item();</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">        </span>if (!root_item)</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">                </span>goto err;</div><div>diff --git a/criu/cr-restore.c b/criu/cr-restore.c</div><div>index dc7bef0..4c1b419 100644</div><div>--- a/criu/cr-restore.c</div><div>+++ b/criu/cr-restore.c</div><div>@@ -30,6 +30,7 @@</div><div> #include &quot;cr_options.h&quot;</div><div> #include &quot;servicefd.h&quot;</div><div> #include &quot;image.h&quot;</div><div>+#include &quot;img-remote.h&quot;</div><div> #include &quot;util.h&quot;</div><div> #include &quot;util-pie.h&quot;</div><div> #include &quot;criu-log.h&quot;</div><div>@@ -1954,6 +1955,11 @@ int cr_restore_tasks(void)</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">                </span>goto err;</div><div> </div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">        </span>ret = restore_root_task(root_item);</div><div>+</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>if (opts.remote &amp;&amp; (finish_remote_restore() &lt; 0)) {</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>pr_err(&quot;Finish remote restore failed.\n&quot;);</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>goto err;</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>}</div><div> err:</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">        </span>cr_plugin_fini(CR_PLUGIN_STAGE__RESTORE, ret);</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">        </span>return ret;</div><div>diff --git a/criu/crtools.c b/criu/crtools.c</div><div>index 1b5cbd1..622fc6c 100644</div><div>--- a/criu/crtools.c</div><div>+++ b/criu/crtools.c</div><div>@@ -52,6 +52,7 @@</div><div> </div><div> #include &quot;setproctitle.h&quot;</div><div> #include &quot;sysctl.h&quot;</div><div>+#include &quot;img-remote.h&quot;</div><div> </div><div> struct cr_options opts;</div><div> </div><div>@@ -74,6 +75,8 @@ void init_opts(void)</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">        </span>opts.ghost_limit = DEFAULT_GHOST_LIMIT;</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">        </span>opts.timeout = DEFAULT_TIMEOUT;</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">        </span>opts.empty_ns = 0;</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>opts.addr = DEFAULT_CACHE_HOST;</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>opts.port = DEFAULT_CACHE_PORT;</div><div> }</div><div> </div><div> static int parse_join_ns(const char *ptr)</div><div>@@ -280,6 +283,7 @@ int main(int argc, char *argv[], char *envp[])</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">                </span>{ &quot;cgroup-dump-controller&quot;,<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>required_argument,<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>0, 1082<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>},</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">                </span>{ SK_INFLIGHT_PARAM,<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>no_argument,<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>0, 1083<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>},</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">                </span>{ &quot;deprecated&quot;,<span class="gmail-Apple-tab-span" style="white-space:pre">                        </span>no_argument,<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>0, 1084 },</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>{ &quot;remote&quot;,<span class="gmail-Apple-tab-span" style="white-space:pre">                        </span>no_argument,<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>0, 1085 },</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">                </span>{ &quot;display-stats&quot;,<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>no_argument,<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>0, 1086 },</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">                </span>{ &quot;weak-sysctls&quot;,<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>no_argument,<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>0, 1087 },</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">                </span>{ },</div><div>@@ -594,6 +598,9 @@ int main(int argc, char *argv[], char *envp[])</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">                        </span>pr_msg(&quot;Turn deprecated stuff ON\n&quot;);</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">                        </span>opts.deprecated_ok = true;</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">                        </span>break;</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>case 1085:</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                        </span>opts.remote = true;</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                        </span>break;</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">                </span>case 1086:</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">                        </span>opts.display_stats = true;</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">                        </span>break;</div><div>@@ -755,6 +762,12 @@ int main(int argc, char *argv[], char *envp[])</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">        </span>if (!strcmp(argv[optind], &quot;page-server&quot;))</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">                </span>return cr_page_server(opts.daemon_mode, -1) &gt; 0 ? 0 : 1;</div><div> </div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>if (!strcmp(argv[optind], &quot;image-cache&quot;))</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>return image_cache(opts.daemon_mode, DEFAULT_CACHE_SOCKET, opts.port);</div><div>+</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>if (!strcmp(argv[optind], &quot;image-proxy&quot;))</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>return image_proxy(opts.daemon_mode, DEFAULT_PROXY_SOCKET, opts.addr, opts.port);</div><div>+</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">        </span>if (!strcmp(argv[optind], &quot;service&quot;))</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">                </span>return cr_service(opts.daemon_mode);</div><div> </div><div>@@ -781,6 +794,8 @@ usage:</div><div> &quot;  criu page-server\n&quot;</div><div> &quot;  criu service [&lt;options&gt;]\n&quot;</div><div> &quot;  criu dedup\n&quot;</div><div>+&quot;  criu image-cache [&lt;options&gt;]\n&quot;</div><div>+&quot;  criu image-proxy [&lt;options&gt;]\n&quot;</div><div> &quot;\n&quot;</div><div> &quot;Commands:\n&quot;</div><div> &quot;  dump           checkpoint a process/tree identified by pid\n&quot;</div><div>@@ -793,6 +808,8 @@ usage:</div><div> &quot;  dedup          remove duplicates in memory dump\n&quot;</div><div> &quot;  cpuinfo dump   writes cpu information into image file\n&quot;</div><div> &quot;  cpuinfo check  validates cpu information read from image file\n&quot;</div><div>+&quot;  image-cache    launch destination-side cache for images sent from the source-side\n&quot;</div><div>+&quot;  image-proxy    launch source-side proxy to sent images to the destination-side\n&quot;</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">        </span>);</div><div> </div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">        </span>if (usage_error) {</div><div>@@ -836,6 +853,7 @@ usage:</div><div> &quot;                            macvlan[IFNAME]:OUTNAME\n&quot;</div><div> &quot;                            mnt[COOKIE]:ROOT\n&quot;</div><div> &quot;\n&quot;</div><div>+&quot;  --remote              dump/restore images directly to/from remote node using image-proxy/image-cache\n&quot;</div><div> &quot;* Special resources support:\n&quot;</div><div> &quot;     --&quot; SK_EST_PARAM &quot;  checkpoint/restore established TCP connections\n&quot;</div><div> &quot;     --&quot; SK_INFLIGHT_PARAM &quot;   skip (ignore) in-flight TCP connections\n&quot;</div><div>@@ -922,7 +940,7 @@ usage:</div><div> &quot;\n&quot;</div><div> &quot;Page/Service server options:\n&quot;</div><div> &quot;  --address ADDR        address of server or service\n&quot;</div><div>-&quot;  --port PORT           port of page server\n&quot;</div><div>+&quot;  --port PORT           port of page serve or service\n&quot;</div><div> &quot;  -d|--daemon           run in the background after creating socket\n&quot;</div><div> &quot;\n&quot;</div><div> &quot;Other options:\n&quot;</div><div>diff --git a/criu/image-desc.c b/criu/image-desc.c</div><div>index bac7ca2..fffaf5a 100644</div><div>--- a/criu/image-desc.c</div><div>+++ b/criu/image-desc.c</div><div>@@ -103,13 +103,13 @@ struct cr_fd_desc_tmpl imgset_template[CR_FD_MAX] = {</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">        </span>[CR_FD_STATS] = {</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">                </span>.fmt<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>= &quot;stats-%s&quot;,</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">                </span>.magic<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>= STATS_MAGIC,</div><div>-<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>.oflags = O_SERVICE,</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>.oflags = O_SERVICE | O_FORCE_LOCAL,</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">        </span>},</div><div> </div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">        </span>[CR_FD_IRMAP_CACHE] = {</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">                </span>.fmt<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>= &quot;irmap-cache&quot;,</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">                </span>.magic<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>= IRMAP_CACHE_MAGIC,</div><div>-<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>.oflags = O_SERVICE,</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>.oflags = O_SERVICE | O_FORCE_LOCAL,</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">        </span>},</div><div> </div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">        </span>[CR_FD_FILE_LOCKS_PID] = {</div><div>diff --git a/criu/image.c b/criu/image.c</div><div>index 11f1e7b..05c2652 100644</div><div>--- a/criu/image.c</div><div>+++ b/criu/image.c</div><div>@@ -16,6 +16,7 @@</div><div> #include &quot;xmalloc.h&quot;</div><div> #include &quot;images/inventory.pb-c.h&quot;</div><div> #include &quot;images/pagemap.pb-c.h&quot;</div><div>+#include &quot;img-remote.h&quot;</div><div> </div><div> bool ns_per_id = false;</div><div> bool img_common_magic = true;</div><div>@@ -312,15 +313,55 @@ static int img_write_magic(struct cr_img *img, int oflags, int type)</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">        </span>return write_img(img, &amp;imgset_template[type].magic);</div><div> }</div><div> </div><div>+int do_open_remote_image(int dfd, char *path, int flags)</div><div>+{</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>char *snapshot_id = NULL;</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>int ret;</div><div>+</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>/* When using namespaces, the current dir is changed so we need to</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span> * change to previous working dir and back to correctly open the image</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span> * proxy and cache sockets. */</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>int save = dirfd(opendir(&quot;.&quot;));</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>if (fchdir(dfd) &lt; 0) {</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>pr_debug(&quot;fchdir to dfd failed!\n&quot;);</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>return -1;</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>}</div><div>+</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>snapshot_id = get_snapshot_id_from_idx(dfd);</div><div>+</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>if (snapshot_id == NULL)</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>ret = -1;</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>else if (flags == O_RDONLY) {</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>pr_debug(&quot;do_open_remote_image RDONLY path=%s snapshot_id=%s\n&quot;,</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                                </span>  path, snapshot_id);</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>ret = read_remote_image_connection(snapshot_id, path);</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>} else {</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>pr_debug(&quot;do_open_remote_image WDONLY path=%s snapshot_id=%s\n&quot;,</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                                </span>  path, snapshot_id);</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>ret = write_remote_image_connection(snapshot_id, path, O_WRONLY);</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>}</div><div>+</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>if (fchdir(save) &lt; 0) {</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>pr_debug(&quot;fchdir to save failed!\n&quot;);</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>return -1;</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>}</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>close(save);</div><div>+</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>return ret;</div><div>+}</div><div>+</div><div> static int do_open_image(struct cr_img *img, int dfd, int type, unsigned long oflags, char *path)</div><div> {</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">        </span>int ret, flags;</div><div> </div><div>-<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>flags = oflags &amp; ~(O_NOBUF | O_SERVICE);</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>flags = oflags &amp; ~(O_NOBUF | O_SERVICE | O_FORCE_LOCAL);</div><div> </div><div>-<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>ret = openat(dfd, path, flags, CR_FD_PERM);</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>if (opts.remote &amp;&amp; !(oflags &amp; O_FORCE_LOCAL))</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>ret = do_open_remote_image(dfd, path, flags);</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>else</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>ret = openat(dfd, path, flags, CR_FD_PERM);</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">        </span>if (ret &lt; 0) {</div><div>-<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>if (!(flags &amp; O_CREAT) &amp;&amp; (errno == ENOENT)) {</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>if (!(flags &amp; O_CREAT) &amp;&amp; (errno == ENOENT || ret == -ENOENT)) {</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">                        </span>pr_info(&quot;No %s image\n&quot;, path);</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">                        </span>img-&gt;_x.fd = EMPTY_IMG_FD;</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">                        </span>goto skip_magic;</div><div>@@ -420,7 +461,9 @@ int open_image_dir(char *dir)</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">        </span>close(fd);</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">        </span>fd = ret;</div><div> </div><div>-<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>if (opts.img_parent) {</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>if (opts.remote) {</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>init_snapshot_id(dir);</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>} else if (opts.img_parent) {</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">                </span>ret = symlinkat(opts.img_parent, fd, CR_PARENT_LINK);</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">                </span>if (ret &lt; 0 &amp;&amp; errno != EEXIST) {</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">                        </span>pr_perror(&quot;Can&#39;t link parent snapshot&quot;);</div><div>diff --git a/criu/img-remote.c b/criu/img-remote.c</div><div>new file mode 100644</div><div>index 0000000..337cb4a</div><div>--- /dev/null</div><div>+++ b/criu/img-remote.c</div><div>@@ -0,0 +1,275 @@</div><div>+#include &lt;unistd.h&gt;</div><div>+#include &lt;stdlib.h&gt;</div><div>+#include &lt;stdio.h&gt;</div><div>+#include &lt;sys/types.h&gt;</div><div>+#include &lt;sys/socket.h&gt;</div><div>+#include &lt;netinet/in.h&gt;</div><div>+#include &lt;netdb.h&gt;</div><div>+#include &quot;xmalloc.h&quot;</div><div>+#include &quot;criu-log.h&quot;</div><div>+#include &quot;img-remote.h&quot;</div><div>+#include &quot;img-remote-proto.h&quot;</div><div>+#include &quot;images/remote-image.pb-c.h&quot;</div><div>+#include &quot;protobuf-desc.h&quot;</div><div>+#include &lt;fcntl.h&gt;</div><div>+#include &quot;servicefd.h&quot;</div><div>+#include &quot;common/compiler.h&quot;</div><div>+#include &quot;cr_options.h&quot;</div><div>+</div><div>+#define PB_LOCAL_IMAGE_SIZE PATHLEN</div><div>+</div><div>+static char *snapshot_id;</div><div>+bool restoring = true;</div><div>+</div><div>+LIST_HEAD(snapshot_head);</div><div>+</div><div>+/* A snapshot is a dump or pre-dump operation. Each snapshot is identified by an</div><div>+ * ID which corresponds to the working directory specefied by the user.</div><div>+ */</div><div>+struct snapshot {</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>char snapshot_id[PATHLEN];</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>struct list_head l;</div><div>+};</div><div>+</div><div>+struct snapshot *new_snapshot(char *snapshot_id)</div><div>+{</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>struct snapshot *s = malloc(sizeof(struct snapshot));</div><div>+</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>if (!s) {</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>pr_perror(&quot;Failed to allocate snapshot structure&quot;);</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>return NULL;</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>}</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>strncpy(s-&gt;snapshot_id, snapshot_id, PATHLEN);</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>return s;</div><div>+}</div><div>+</div><div>+void add_snapshot(struct snapshot *snapshot)</div><div>+{</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>list_add_tail(&amp;(snapshot-&gt;l), &amp;snapshot_head);</div><div>+}</div><div>+</div><div>+int read_remote_image_connection(char *snapshot_id, char *path)</div><div>+{</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>int error;</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>int sockfd = setup_UNIX_client_socket(restoring ? DEFAULT_CACHE_SOCKET: DEFAULT_PROXY_SOCKET);</div><div>+</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>if (sockfd &lt; 0) {</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>pr_perror(&quot;Error opening local connection for %s:%s&quot;, path, snapshot_id);</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>return -1;</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>}</div><div>+</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>if (write_header(sockfd, snapshot_id, path, O_RDONLY) &lt; 0) {</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>pr_perror(&quot;Error writing header for %s:%s&quot;, path, snapshot_id);</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>return -1;</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>}</div><div>+</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>if (read_reply_header(sockfd, &amp;error) &lt; 0) {</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>pr_perror(&quot;Error reading reply header for %s:%s&quot;, path, snapshot_id);</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>return -1;</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>}</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>if (!error || !strncmp(path, RESTORE_FINISH, sizeof(RESTORE_FINISH)))</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>return sockfd;</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>else if (error == ENOENT) {</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>pr_info(&quot;Image does not exist (%s:%s)\n&quot;, path, snapshot_id);</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>close(sockfd);</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>return -ENOENT;</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>}</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>pr_perror(&quot;Unexpected error returned: %d (%s:%s)\n&quot;, error, path, snapshot_id);</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>close(sockfd);</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>return -1;</div><div>+}</div><div>+</div><div>+int write_remote_image_connection(char *snapshot_id, char *path, int flags)</div><div>+{</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>int sockfd = setup_UNIX_client_socket(DEFAULT_PROXY_SOCKET);</div><div>+</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>if (sockfd &lt; 0)</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>return -1;</div><div>+</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>if (write_header(sockfd, snapshot_id, path, flags) &lt; 0) {</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>pr_perror(&quot;Error writing header for %s:%s&quot;, path, snapshot_id);</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>return -1;</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>}</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>return sockfd;</div><div>+}</div><div>+</div><div>+int finish_remote_dump(void)</div><div>+{</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>pr_info(&quot;Dump side is calling finish\n&quot;);</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>int fd = write_remote_image_connection(NULL_SNAPSHOT_ID, DUMP_FINISH, O_WRONLY);</div><div>+</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>if (fd == -1) {</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>pr_perror(&quot;Unable to open finish dump connection&quot;);</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>return -1;</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>}</div><div>+</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>close(fd);</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>return 0;</div><div>+}</div><div>+</div><div>+int finish_remote_restore(void)</div><div>+{</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>pr_info(&quot;Restore side is calling finish\n&quot;);</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>int fd = read_remote_image_connection(NULL_SNAPSHOT_ID, RESTORE_FINISH);</div><div>+</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>if (fd == -1) {</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>pr_perror(&quot;Unable to open finish restore connection&quot;);</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>return -1;</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>}</div><div>+</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>close(fd);</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>return 0;</div><div>+}</div><div>+</div><div>+int skip_remote_bytes(int fd, unsigned long len)</div><div>+{</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>static char buf[4096];</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>int n = 0;</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>unsigned long curr = 0;</div><div>+</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>for (; curr &lt; len; ) {</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>n = read(fd, buf, min(len - curr, (unsigned long)4096));</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>if (n == 0) {</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                        </span>pr_perror(&quot;Unexpected end of stream (skipping %lx/%lx bytes)&quot;,</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                                </span>curr, len);</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                        </span>return -1;</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>} else if (n &gt; 0) {</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                        </span>curr += n;</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>} else {</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                        </span>pr_perror(&quot;Error while skipping bytes from stream (%lx/%lx)&quot;,</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                                </span>curr, len);</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                        </span>return -1;</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>}</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>}</div><div>+</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>if (curr != len) {</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>pr_perror(&quot;Unable to skip the current number of bytes: %lx instead of %lx&quot;,</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                        </span>curr, len);</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>return -1;</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>}</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>return 0;</div><div>+}</div><div>+</div><div>+static int pull_snapshot_ids(void)</div><div>+{</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>int n, sockfd;</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>SnapshotIdEntry *ls;</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>struct snapshot *s = NULL;</div><div>+</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>sockfd = read_remote_image_connection(NULL_SNAPSHOT_ID, PARENT_IMG);</div><div>+</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>/* The connection was successful but there is not file. */</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>if (sockfd &lt; 0 &amp;&amp; errno == ENOENT)</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>return 0;</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>else if (sockfd &lt; 0) {</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>pr_perror(&quot;Unable to open snapshot id read connection&quot;);</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>return -1;</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>}</div><div>+</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>while (1) {</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>n = pb_read_obj(sockfd, (void **)&amp;ls, PB_SNAPSHOT_ID);</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>if (!n) {</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                        </span>close(sockfd);</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                        </span>return n;</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>} else if (n &lt; 0) {</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                        </span>pr_perror(&quot;Unable to read remote snapshot ids&quot;);</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                        </span>close(sockfd);</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                        </span>return n;</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>}</div><div>+</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>s = new_snapshot(ls-&gt;snapshot_id);</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>if (!s) {</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                        </span>pr_perror(&quot;Unable create new snapshot structure&quot;);</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                        </span>close(sockfd);</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                        </span>return -1;</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>}</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>add_snapshot(s);</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>pr_info(&quot;[read_snapshot ids] parent = %s\n&quot;, ls-&gt;snapshot_id);</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>}</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>free(ls);</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>close(sockfd);</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>return n;</div><div>+}</div><div>+</div><div>+int push_snapshot_id(void)</div><div>+{</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>int n;</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>restoring = false;</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>SnapshotIdEntry rn = SNAPSHOT_ID_ENTRY__INIT;</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>int sockfd = write_remote_image_connection(NULL_SNAPSHOT_ID, PARENT_IMG, O_APPEND);</div><div>+</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>if (sockfd &lt; 0) {</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>pr_perror(&quot;Unable to open snapshot id push connection&quot;);</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>return -1;</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>}</div><div>+</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>rn.snapshot_id = xmalloc(sizeof(char) * PATHLEN);</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>if (!rn.snapshot_id) {</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>pr_perror(&quot;Unable to allocate snapshot id buffer&quot;);</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>close(sockfd);</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>return -1;</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>}</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>strncpy(rn.snapshot_id, snapshot_id, PATHLEN);</div><div>+</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>n = pb_write_obj(sockfd, &amp;rn, PB_SNAPSHOT_ID);</div><div>+</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>xfree(rn.snapshot_id);</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>close(sockfd);</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>return n;</div><div>+}</div><div>+</div><div>+void init_snapshot_id(char *si)</div><div>+{</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>snapshot_id = si;</div><div>+}</div><div>+</div><div>+char *get_curr_snapshot_id(void)</div><div>+{</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>return snapshot_id;</div><div>+}</div><div>+</div><div>+int get_curr_snapshot_id_idx(void)</div><div>+{</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>struct snapshot *si;</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>int idx = 0;</div><div>+</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>if (list_empty(&amp;snapshot_head))</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>pull_snapshot_ids();</div><div>+</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>list_for_each_entry(si, &amp;snapshot_head, l) {</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>if (!strncmp(si-&gt;snapshot_id, snapshot_id, PATHLEN))</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                        </span>return idx;</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>idx++;</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>}</div><div>+</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>pr_perror(&quot;Error, could not find current snapshot id (%s) fd&quot;, snapshot_id);</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>return -1;</div><div>+}</div><div>+</div><div>+char *get_snapshot_id_from_idx(int idx)</div><div>+{</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>struct snapshot *si;</div><div>+</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>if (list_empty(&amp;snapshot_head))</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>pull_snapshot_ids();</div><div>+</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>/* Note: if idx is the service fd then we need the current</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span> * snapshot_id idx. Else we need a parent snapshot_id idx.</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span> */</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>if (idx == get_service_fd(IMG_FD_OFF))</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>idx = get_curr_snapshot_id_idx();</div><div>+</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>list_for_each_entry(si, &amp;snapshot_head, l) {</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>if (!idx)</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                        </span>return si-&gt;snapshot_id;</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>idx--;</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>}</div><div>+</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>pr_perror(&quot;Error, could not find snapshot id for idx %d&quot;, idx);</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>return NULL;</div><div>+}</div><div>+</div><div>+int get_curr_parent_snapshot_id_idx(void)</div><div>+{</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>return get_curr_snapshot_id_idx() - 1;</div><div>+}</div><div>diff --git a/criu/include/cr_options.h b/criu/include/cr_options.h</div><div>index c2f4b10..16e9243 100644</div><div>--- a/criu/include/cr_options.h</div><div>+++ b/criu/include/cr_options.h</div><div>@@ -115,6 +115,7 @@ struct cr_options {</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">        </span> * to turn one ON while the code is in.</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">        </span> */</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">        </span>bool<span class="gmail-Apple-tab-span" style="white-space:pre">                        </span>deprecated_ok;</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>bool<span class="gmail-Apple-tab-span" style="white-space:pre">                        </span>remote;</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">        </span>bool<span class="gmail-Apple-tab-span" style="white-space:pre">                        </span>display_stats;</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">        </span>bool<span class="gmail-Apple-tab-span" style="white-space:pre">                        </span>weak_sysctls;</div><div> };</div><div>diff --git a/criu/include/image.h b/criu/include/image.h</div><div>index 1c22a4f..c2b612e 100644</div><div>--- a/criu/include/image.h</div><div>+++ b/criu/include/image.h</div><div>@@ -105,6 +105,7 @@ extern bool img_common_magic;</div><div> #define O_DUMP<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>(O_WRONLY | O_CREAT | O_TRUNC)</div><div> #define O_SHOW<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>(O_RDONLY | O_NOBUF)</div><div> #define O_RSTR<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>(O_RDONLY)</div><div>+#define O_FORCE_LOCAL<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>(O_SYNC)</div><div> </div><div> struct cr_img {</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">        </span>union {</div><div>diff --git a/criu/include/img-remote.h b/criu/include/img-remote.h</div><div>new file mode 100644</div><div>index 0000000..6ffc429</div><div>--- /dev/null</div><div>+++ b/criu/include/img-remote.h</div><div>@@ -0,0 +1,83 @@</div><div>+#include &lt;limits.h&gt;</div><div>+#include &lt;stdbool.h&gt;</div><div>+</div><div>+#ifndef IMAGE_REMOTE_H</div><div>+#define<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>IMAGE_REMOTE_H</div><div>+</div><div>+#define PATHLEN PATH_MAX</div><div>+#define DUMP_FINISH &quot;DUMP_FINISH&quot;</div><div>+#define RESTORE_FINISH &quot;RESTORE_FINISH&quot;</div><div>+#define PARENT_IMG &quot;parent&quot;</div><div>+#define NULL_SNAPSHOT_ID &quot;null&quot;</div><div>+#define DEFAULT_CACHE_SOCKET &quot;img-cache.sock&quot;</div><div>+#define DEFAULT_PROXY_SOCKET &quot;img-proxy.sock&quot;</div><div>+#define DEFAULT_CACHE_PORT 9996</div><div>+#define DEFAULT_CACHE_HOST &quot;localhost&quot;</div><div>+</div><div>+/* Called by restore to get the fd correspondent to a particular path. This call</div><div>+ * will block until the connection is received.</div><div>+ */</div><div>+int read_remote_image_connection(char *snapshot_id, char *path);</div><div>+</div><div>+/* Called by dump to create a socket connection to the restore side. The socket</div><div>+ * fd is returned for further writing operations.</div><div>+ */</div><div>+int write_remote_image_connection(char *snapshot_id, char *path, int flags);</div><div>+</div><div>+/* Called by dump/restore when everything is dumped/restored. This function </div><div>+ * creates a new connection with a special control name. The receiver side uses</div><div>+ * it to ack that no more files are coming.</div><div>+ */</div><div>+int finish_remote_dump();</div><div>+int finish_remote_restore();</div><div>+</div><div>+/* Starts an image proxy daemon (dump side). It receives image files through</div><div>+ * socket connections and forwards them to the image cache (restore side).</div><div>+ */</div><div>+int image_proxy(bool background, char *local_proxy_path, char *cache_host, unsigned short cache_port);</div><div>+</div><div>+/* Starts an image cache daemon (restore side). It receives image files through</div><div>+ * socket connections and caches them until they are requested by the restore</div><div>+ * process.</div><div>+ */</div><div>+int image_cache(bool background, char *local_cache_path, unsigned short cache_port);</div><div>+</div><div>+/* Reads (discards) &#39;len&#39; bytes from fd. This is used to emulate the function</div><div>+ * lseek, which is used to advance the file needle.</div><div>+ */</div><div>+int skip_remote_bytes(int fd, unsigned long len);</div><div>+</div><div>+/* To support iterative migration, the concept of snapshot_id is introduced</div><div>+ * (only when remote migration is enabled). Each image is tagged with one</div><div>+ * snapshot_id. The snapshot_id is the image directory used for the operation</div><div>+ * that creates the image (either predump or dump). Images stored in memory</div><div>+ * (both in Image Proxy and Image Cache) are identified by their name and</div><div>+ * snapshot_id. Snapshot_ids are ordered so that we can find parent pagemaps</div><div>+ * (that will be used when restoring the process).</div><div>+ */</div><div>+</div><div>+/* Sets the current snapshot_id */</div><div>+void init_snapshot_id(char *ns);</div><div>+</div><div>+/* Returns the current snapshot_id. */</div><div>+char *get_curr_snapshot_id();</div><div>+</div><div>+/* Returns the snapshot_id index representing the current snapshot_id. This</div><div>+ * index represents the hierarchy position. For example: images tagged with</div><div>+ * the snapshot_id with index 1 are more recent than the images tagged with</div><div>+ * the snapshot_id with index 0.</div><div>+ */</div><div>+int get_curr_snapshot_id_idx();</div><div>+</div><div>+/* Returns the snapshot_id associated with the snapshot_id index. */</div><div>+char *get_snapshot_id_from_idx(int idx);</div><div>+</div><div>+/* Pushes the current snapshot_id into the snapshot_id hierarchy (into the Image</div><div>+ * Proxy and Image Cache).</div><div>+ */</div><div>+int push_snapshot_id();</div><div>+</div><div>+/* Returns the snapshot id index that preceeds the current snapshot_id. */</div><div>+int get_curr_parent_snapshot_id_idx();</div><div>+</div><div>+#endif</div><div>diff --git a/criu/include/protobuf-desc.h b/criu/include/protobuf-desc.h</div><div>index 6c76b49..43ac534 100644</div><div>--- a/criu/include/protobuf-desc.h</div><div>+++ b/criu/include/protobuf-desc.h</div><div>@@ -59,6 +59,10 @@ enum {</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">        </span>PB_BINFMT_MISC,<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>/* 50 */</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">        </span>PB_TTY_DATA,</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">        </span>PB_AUTOFS,</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>PB_REMOTE_IMAGE,        /* Header for images sent from proxy to cache.*/</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>PB_LOCAL_IMAGE,         /* Header for reading/writing images from/to proxy or cache. */</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>PB_LOCAL_IMAGE_REPLY,<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>/* Header for reading/writing images reply. */</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>PB_SNAPSHOT_ID,         /* Contains a single id. Used for reading/writing ids from proxy or cache. */</div><div> </div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">        </span>/* PB_AUTOGEN_STOP */</div><div> </div><div>diff --git a/criu/page-xfer.c b/criu/page-xfer.c</div><div>index 4def13a..d933bb7 100644</div><div>--- a/criu/page-xfer.c</div><div>+++ b/criu/page-xfer.c</div><div>@@ -17,6 +17,7 @@</div><div> #include &quot;protobuf.h&quot;</div><div> #include &quot;images/pagemap.pb-c.h&quot;</div><div> #include &quot;fcntl.h&quot;</div><div>+#include &quot;img-remote.h&quot;</div><div> </div><div> static int page_server_sk = -1;</div><div> </div><div>@@ -294,7 +295,8 @@ static int open_page_local_xfer(struct page_xfer *xfer, int fd_type, long id)</div><div> </div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">                </span>xfer-&gt;parent = xmalloc(sizeof(*xfer-&gt;parent));</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">                </span>if (!xfer-&gt;parent) {</div><div>-<span class="gmail-Apple-tab-span" style="white-space:pre">                        </span>close(pfd);</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                        </span>if (!opts.remote)</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                                </span>close(pfd);</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">                        </span>return -1;</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">                </span>}</div><div> </div><div>@@ -303,10 +305,12 @@ static int open_page_local_xfer(struct page_xfer *xfer, int fd_type, long id)</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">                        </span>pr_perror(&quot;No parent image found, though parent directory is set&quot;);</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">                        </span>xfree(xfer-&gt;parent);</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">                        </span>xfer-&gt;parent = NULL;</div><div>-<span class="gmail-Apple-tab-span" style="white-space:pre">                        </span>close(pfd);</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                        </span>if (!opts.remote)</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                                </span>close(pfd);</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">                        </span>goto out;</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">                </span>}</div><div>-<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>close(pfd);</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>if (!opts.remote)</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                        </span>close(pfd);</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">        </span>}</div><div> </div><div> out:</div><div>@@ -411,9 +415,16 @@ int check_parent_local_xfer(int fd_type, int id)</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">        </span>struct stat st;</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">        </span>int ret, pfd;</div><div> </div><div>-<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>pfd = openat(get_service_fd(IMG_FD_OFF), CR_PARENT_LINK, O_RDONLY);</div><div>-<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>if (pfd &lt; 0 &amp;&amp; errno == ENOENT)</div><div>-<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>return 0;</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>if (opts.remote) {</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>pfd = get_curr_parent_snapshot_id_idx();</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>pr_err(&quot;Unable to get parent snapsgot id&quot;);</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>if (pfd == -1)</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                        </span>return -1;</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>} else {</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>pfd = openat(get_service_fd(IMG_FD_OFF), CR_PARENT_LINK, O_RDONLY);</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>if (pfd &lt; 0 &amp;&amp; errno == ENOENT)</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                        </span>return 0;</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>}</div><div> </div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">        </span>snprintf(path, sizeof(path), imgset_template[fd_type].fmt, id);</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">        </span>ret = fstatat(pfd, path, &amp;st, 0);</div><div>@@ -475,6 +486,8 @@ int check_parent_page_xfer(int fd_type, long id)</div><div> {</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">        </span>if (opts.use_page_server)</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">                </span>return check_parent_server_xfer(fd_type, id);</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>else if (opts.remote)</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>return get_curr_parent_snapshot_id_idx() == -1 ? 0 : 1;</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">        </span>else</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">                </span>return check_parent_local_xfer(fd_type, id);</div><div> }</div><div>diff --git a/criu/pagemap.c b/criu/pagemap.c</div><div>index 8e854c9..be5b144 100644</div><div>--- a/criu/pagemap.c</div><div>+++ b/criu/pagemap.c</div><div>@@ -14,6 +14,7 @@</div><div> #include &quot;xmalloc.h&quot;</div><div> #include &quot;protobuf.h&quot;</div><div> #include &quot;images/pagemap.pb-c.h&quot;</div><div>+#include &quot;img-remote.h&quot;</div><div> </div><div> #ifndef SEEK_DATA</div><div> #define SEEK_DATA<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>3</div><div>@@ -137,6 +138,9 @@ static void skip_pagemap_pages(struct page_read *pr, unsigned long len)</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">                </span>return;</div><div> </div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">        </span>if (!pr-&gt;pe-&gt;in_parent)</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>if (opts.remote)</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                        </span>if (! skip_remote_bytes(img_raw_fd(pr-&gt;pi), len))</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                                </span>pr_perror(&quot;Error skipping remote bytes&quot;);</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">                </span>pr-&gt;pi_off += len;</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">        </span>pr-&gt;cvaddr += len;</div><div> }</div><div>@@ -248,6 +252,7 @@ static int read_local_page(struct page_read *pr, unsigned long vaddr,</div><div> {</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">        </span>int fd = img_raw_fd(pr-&gt;pi);</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">        </span>int ret;</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>size_t curr = 0;</div><div> </div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">        </span>/*</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">        </span> * Flush any pending async requests if any not to break the</div><div>@@ -257,10 +262,15 @@ static int read_local_page(struct page_read *pr, unsigned long vaddr,</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">                </span>return -1;</div><div> </div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">        </span>pr_debug(&quot;\tpr%u Read page from self %lx/%&quot;PRIx64&quot;\n&quot;, pr-&gt;id, pr-&gt;cvaddr, pr-&gt;pi_off);</div><div>-<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>ret = pread(fd, buf, len, pr-&gt;pi_off);</div><div>-<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>if (ret != len) {</div><div>-<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>pr_perror(&quot;Can&#39;t read mapping page %d&quot;, ret);</div><div>-<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>return -1;</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>while (1) {</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>ret = read(fd, buf + curr, len - curr);</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>if (ret &lt; 1) {</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                        </span>pr_perror(&quot;Can&#39;t read mapping page %d&quot;, ret);</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                        </span>return -1;</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>}</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>curr += ret;</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>if (curr == len)</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                        </span>break;</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">        </span>}</div><div> </div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">        </span>if (opts.auto_dedup) {</div><div>@@ -358,7 +368,7 @@ static int maybe_read_page(struct page_read *pr, unsigned long vaddr,</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">        </span>int ret;</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">        </span>unsigned long len = nr * PAGE_SIZE;</div><div> </div><div>-<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>if (flags &amp; PR_ASYNC)</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>if (flags &amp; PR_ASYNC &amp;&amp; !opts.remote)</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">                </span>ret = enqueue_async_page(pr, vaddr, len, buf);</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">        </span>else</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">                </span>ret = read_local_page(pr, vaddr, len, buf);</div><div>@@ -459,9 +469,24 @@ static int try_open_parent(int dfd, int pid, struct page_read *pr, int pr_flags)</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">        </span>int pfd, ret;</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">        </span>struct page_read *parent = NULL;</div><div> </div><div>-<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>pfd = openat(dfd, CR_PARENT_LINK, O_RDONLY);</div><div>-<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>if (pfd &lt; 0 &amp;&amp; errno == ENOENT)</div><div>-<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>goto out;</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>if (opts.remote) {</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>/* Note: we are replacing a real directory FD for a snapshot_id</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                </span> * index. Since we need the parent of the current snapshot_id,</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                </span> * we want the current snapshot_id index minus one. It is</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                </span> * possible that dfd is already a snapshot_id index. We test it</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                </span> * by comparing it to the service FD. When opening an image (see</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                </span> * do_open_image) we convert the snapshot_id index into a real</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                </span> * snapshot_id.</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                </span> */</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>pfd = dfd == get_service_fd(IMG_FD_OFF) ?</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                        </span>get_curr_snapshot_id_idx() - 1 : dfd - 1;</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>if (pfd &lt; 0)</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                        </span>goto out;</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>} else {</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>pfd = openat(dfd, CR_PARENT_LINK, O_RDONLY);</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>if (pfd &lt; 0 &amp;&amp; errno == ENOENT)</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                        </span>goto out;</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>}</div><div> </div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">        </span>parent = xmalloc(sizeof(*parent));</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">        </span>if (!parent)</div><div>@@ -476,7 +501,8 @@ static int try_open_parent(int dfd, int pid, struct page_read *pr, int pr_flags)</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">                </span>parent = NULL;</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">        </span>}</div><div> </div><div>-<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>close(pfd);</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>if (!opts.remote)</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>close(pfd);</div><div> out:</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">        </span>pr-&gt;parent = parent;</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">        </span>return 0;</div><div>@@ -484,7 +510,8 @@ out:</div><div> err_free:</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">        </span>xfree(parent);</div><div> err_cl:</div><div>-<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>close(pfd);</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>if (!opts.remote)</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>close(pfd);</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">        </span>return -1;</div><div> }</div><div> </div><div>@@ -501,7 +528,11 @@ static int init_pagemaps(struct page_read *pr)</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">        </span>off_t fsize;</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">        </span>int nr_pmes, nr_realloc;</div><div> </div><div>-<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>fsize = img_raw_size(pr-&gt;pmi);</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>if (!opts.remote)</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>fsize = img_raw_size(pr-&gt;pmi);</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>else</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>fsize = 1024; /*FIXME*/</div><div>+</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">        </span>if (fsize &lt; 0)</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">                </span>return -1;</div><div> </div><div>diff --git a/criu/protobuf-desc.c b/criu/protobuf-desc.c</div><div>index 41c2080..bfe00c5 100644</div><div>--- a/criu/protobuf-desc.c</div><div>+++ b/criu/protobuf-desc.c</div><div>@@ -62,6 +62,7 @@</div><div> #include &quot;images/seccomp.pb-c.h&quot;</div><div> #include &quot;images/binfmt-misc.pb-c.h&quot;</div><div> #include &quot;images/autofs.pb-c.h&quot;</div><div>+#include &quot;images/remote-image.pb-c.h&quot;</div><div> </div><div> struct cr_pb_message_desc cr_pb_descs[PB_MAX];</div><div> </div><div>diff --git a/criu/sk-queue.c b/criu/sk-queue.c</div><div>index 7b1da2d..f92d020 100644</div><div>--- a/criu/sk-queue.c</div><div>+++ b/criu/sk-queue.c</div><div>@@ -27,7 +27,7 @@</div><div> struct sk_packet {</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">        </span>struct list_head<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>list;</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">        </span>SkPacketEntry<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>*entry;</div><div>-<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>off_t<span class="gmail-Apple-tab-span" style="white-space:pre">                        </span>img_off;</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>char        <span class="gmail-Apple-tab-span" style="white-space:pre">                </span>*data;</div><div> };</div><div> </div><div> static LIST_HEAD(packets_list);</div><div>@@ -37,14 +37,20 @@ static int collect_one_packet(void *obj, ProtobufCMessage *msg, struct cr_img *i</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">        </span>struct sk_packet *pkt = obj;</div><div> </div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">        </span>pkt-&gt;entry = pb_msg(msg, SkPacketEntry);</div><div>-<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>pkt-&gt;img_off = lseek(img_raw_fd(img), 0, SEEK_CUR);</div><div>+</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>pkt-&gt;data = xmalloc(pkt-&gt;entry-&gt;length);</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>if (pkt-&gt;data ==NULL)</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>return -1;</div><div>+</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">        </span>/*</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">        </span> * NOTE: packet must be added to the tail. Otherwise sequence</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">        </span> * will be broken.</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">        </span> */</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">        </span>list_add_tail(&amp;pkt-&gt;list, &amp;packets_list);</div><div>-<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>if (lseek(img_raw_fd(img), pkt-&gt;entry-&gt;length, SEEK_CUR) &lt; 0) {</div><div>-<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>pr_perror(&quot;Unable to change an image offset&quot;);</div><div>+</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>if (read_img_buf(img, pkt-&gt;data, pkt-&gt;entry-&gt;length) != 1) {</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>xfree(pkt-&gt;data);</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>pr_perror(&quot;Unable to read packet data&quot;);</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">                </span>return -1;</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">        </span>}</div><div> </div><div>@@ -208,7 +214,6 @@ int restore_sk_queue(int fd, unsigned int peer_id)</div><div> </div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">        </span>list_for_each_entry_safe(pkt, tmp, &amp;packets_list, list) {</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">                </span>SkPacketEntry *entry = pkt-&gt;entry;</div><div>-<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>char *buf;</div><div> </div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">                </span>if (entry-&gt;id_for != peer_id)</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">                        </span>continue;</div><div>@@ -224,22 +229,8 @@ int restore_sk_queue(int fd, unsigned int peer_id)</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">                </span> * boundaries messages should be saved.</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">                </span> */</div><div> </div><div>-<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>buf = xmalloc(entry-&gt;length);</div><div>-<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>if (buf ==NULL)</div><div>-<span class="gmail-Apple-tab-span" style="white-space:pre">                        </span>goto err;</div><div>-</div><div>-<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>if (lseek(img_raw_fd(img), pkt-&gt;img_off, SEEK_SET) == -1) {</div><div>-<span class="gmail-Apple-tab-span" style="white-space:pre">                        </span>pr_perror(&quot;lseek() failed&quot;);</div><div>-<span class="gmail-Apple-tab-span" style="white-space:pre">                        </span>xfree(buf);</div><div>-<span class="gmail-Apple-tab-span" style="white-space:pre">                        </span>goto err;</div><div>-<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>}</div><div>-<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>if (read_img_buf(img, buf, entry-&gt;length) != 1) {</div><div>-<span class="gmail-Apple-tab-span" style="white-space:pre">                        </span>xfree(buf);</div><div>-<span class="gmail-Apple-tab-span" style="white-space:pre">                        </span>goto err;</div><div>-<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>}</div><div>-</div><div>-<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>ret = write(fd, buf, entry-&gt;length);</div><div>-<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>xfree(buf);</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>ret = write(fd, pkt-&gt;data, entry-&gt;length);</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>xfree(pkt-&gt;data);</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">                </span>if (ret &lt; 0) {</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">                        </span>pr_perror(&quot;Failed to send packet&quot;);</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">                        </span>goto err;</div><div>diff --git a/criu/util.c b/criu/util.c</div><div>index 4b0bcfd..d826732 100644</div><div>--- a/criu/util.c</div><div>+++ b/criu/util.c</div><div>@@ -516,29 +516,46 @@ int copy_file(int fd_in, int fd_out, size_t bytes)</div><div> {</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">        </span>ssize_t written = 0;</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">        </span>size_t chunk = bytes ? bytes : 4096;</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>char *buffer = (char*) malloc(chunk);</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>ssize_t ret;</div><div> </div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">        </span>while (1) {</div><div>-<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>ssize_t ret;</div><div>-</div><div>-<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>ret = sendfile(fd_out, fd_in, NULL, chunk);</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>if (opts.remote) {</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                        </span>ret = read(fd_in, buffer, chunk);</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                        </span>if (ret &lt; 0) {</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                                </span>pr_perror(&quot;Can&#39;t read from fd_in\n&quot;);</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                                </span>ret = -1;</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                                </span>goto err;</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                        </span>}</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                        </span>if (write(fd_out, buffer, ret) != ret) {</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                                </span>pr_perror(&quot;Couldn&#39;t write all read bytes\n&quot;);</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                                </span>ret = -1;</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                                </span>goto err;</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                        </span>}</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>}</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>else</div><div>+                        ret = sendfile(fd_out, fd_in, NULL, chunk);</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">                </span>if (ret &lt; 0) {</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">                        </span>pr_perror(&quot;Can&#39;t send data to ghost file&quot;);</div><div>-<span class="gmail-Apple-tab-span" style="white-space:pre">                        </span>return -1;</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                        </span>ret = -1;</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                        </span>goto err;</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">                </span>}</div><div> </div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">                </span>if (ret == 0) {</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">                        </span>if (bytes &amp;&amp; (written != bytes)) {</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">                                </span>pr_err(&quot;Ghost file size mismatch %zu/%zu\n&quot;,</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">                                                </span>written, bytes);</div><div>-<span class="gmail-Apple-tab-span" style="white-space:pre">                                </span>return -1;</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                                </span>ret = -1;</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">                                </span>goto err;</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">                        </span>}</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">                        </span>break;</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">                </span>}</div><div> </div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">                </span>written += ret;</div><div> <span class="gmail-Apple-tab-span" style="white-space:pre">        </span>}</div><div>-</div><div>-<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>return 0;</div><div>+err:</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>free(buffer);</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>return ret;</div><div> }</div><div> </div><div> int read_fd_link(int lfd, char *buf, size_t size)</div><div>diff --git a/images/Makefile b/images/Makefile</div><div>index eb18526..5b60948 100644</div><div>--- a/images/Makefile</div><div>+++ b/images/Makefile</div><div>@@ -61,6 +61,7 @@ proto-obj-y<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>+= time.o</div><div> proto-obj-y<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>+= sysctl.o</div><div> proto-obj-y<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>+= autofs.o</div><div> proto-obj-y<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>+= macvlan.o</div><div>+proto-obj-y<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>+= remote-image.o</div><div> </div><div> CFLAGS<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>+= -iquote $(obj)/</div><div> </div><div>diff --git a/images/remote-image.proto b/images/remote-image.proto</div><div>new file mode 100644</div><div>index 0000000..1212627</div><div>--- /dev/null</div><div>+++ b/images/remote-image.proto</div><div>@@ -0,0 +1,20 @@</div><div>+message local_image_entry {</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>required string name<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>= 1;</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>required string snapshot_id<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>= 2;</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>required uint32 open_mode<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>= 3;</div><div>+}</div><div>+</div><div>+message remote_image_entry {</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>required string name<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>= 1;</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>required string snapshot_id<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>= 2;</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>required uint32 open_mode<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>= 3;</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>required uint64 size<span class="gmail-Apple-tab-span" style="white-space:pre">                </span>= 4;</div><div>+}</div><div>+</div><div>+message local_image_reply_entry {</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>required uint32 error           = 1;</div><div>+}</div><div>+</div><div>+message snapshot_id_entry {</div><div>+<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>required string snapshot_id<span class="gmail-Apple-tab-span" style="white-space:pre">        </span>= 1;</div><div>+}</div></div><div><br></div></div>