[CRIU] [PATCH v4 00/12] lazy-pages: add support for remote pages

[Mike Rapoport]

On Mon, Jun 06, 2016 at 10:36:36PM +0200, Adrian Reber wrote:
> I also tested this version (v4) and I can now migrate small processes
> and larger processes (200MB) without errors. I have not tried to migrate
> more than once process with this patchset.
> 
> What is the status of the kernel patches for non cooperative uffd usage.
> Have they been merged?

Not yet. Andrea said they are on his list, but there's no progress yet :(
 
> To migrate a 200MB process between two VMs on my system I need about 3
> seconds with dump, rsync and restore. The process just loops over its
> malloc'd memory and writes one byte in each page.
> 
> Using the remote lazy-restore options the restore requires more than
> three hours. So again, as with all optimizations, it can make the
> situation worse. Looking at the debug log I see about 4 to 5 pages
> transferred per second. About 50000 pages results in over 10000 seconds.
> Which matches the three hours pretty good. Just reporting this without
> any intent to argument that lazy-restore might be a bad idea. I have
> already seen cases where pre-copy migration takes longer than
> the unoptimized case. I guess the combination of pre-copy and post-copy
> migration is the final goal.
 
We have a lot of room for optimization. Even simple change of socket
parameters give significant speed up (see below). 
Another thing I thought of is using IOV lists instead of page lists in
uffd.c. Then we can transfer up to 64K at #PF time, and entire IOV when we
are transferring the remaining pages.
So, there's still a lot of work ahead to make post-copy really working :)

--
Sincerely yours,
Mike.

> So, despite having found a case where it is slower I am acking this
> series. Mike, thanks for these patches!
> 
> Acked-by: Adrian Reber <areber at redhat.com>
> 
> 		Adrian
> 

>From 7956dd5ad6cbe1ce7ac5f60728137cfef66317e5 Mon Sep 17 00:00:00 2001
From: Mike Rapoport <rppt at linux.vnet.ibm.com>
Date: Mon, 6 Jun 2016 14:18:54 +0300
Subject: [PATCH] criu: page-xfer: change TCP socket options for lazy-server

Signed-off-by: Mike Rapoport <rppt at linux.vnet.ibm.com>
---
 criu/page-xfer.c | 20 +++++++++++++-------
 1 file changed, 13 insertions(+), 7 deletions(-)

diff --git a/criu/page-xfer.c b/criu/page-xfer.c
index 5f9dda6..09b780c 100644
--- a/criu/page-xfer.c
+++ b/criu/page-xfer.c
@@ -611,6 +611,8 @@ static int page_server_get_pages(int sk, struct page_server_iov *pi)
 	if (ret != iov->iov_len)
 		return -1;
 
+	tcp_nodelay(sk, true);
+
 	page_pipe_destroy_ppb(ppb);
 
 	return 0;
@@ -621,14 +623,14 @@ static int page_server_serve(int sk)
 	int ret = -1;
 	bool flushed = false;
 
-	/*
-	 * This socket only accepts data except one thing -- it
-	 * writes back the has_parent bit from time to time, so
-	 * make it NODELAY all the time.
-	 */
-	tcp_nodelay(sk, true);
-
 	if (!opts.lazy_pages) {
+		/*
+		 * This socket only accepts data except one thing -- it
+		 * writes back the has_parent bit from time to time, so
+		 * make it NODELAY all the time.
+		 */
+		tcp_nodelay(sk, true);
+
 		if (pipe(cxfer.p)) {
 			pr_perror("Can't make pipe for xfer");
 			close(sk);
@@ -637,6 +639,8 @@ static int page_server_serve(int sk)
 
 		cxfer.pipe_size = fcntl(cxfer.p[0], F_GETPIPE_SZ, 0);
 		pr_debug("Created xfer pipe size %u\n", cxfer.pipe_size);
+	} else {
+		tcp_cork(sk, true);
 	}
 
 	while (1) {
@@ -835,6 +839,8 @@ int get_remote_pages(int pid, unsigned long addr, int nr_pages, void *dest)
 	if (send_psi(page_server_sk, PS_IOV_GET, nr_pages, addr, pid))
 		return -1;
 
+	tcp_nodelay(page_server_sk, true);
+
 	ret = recv(page_server_sk, &pi, sizeof(pi), MSG_WAITALL);
 	if (ret != sizeof(pi))
 		return -1;
-- 
1.9.1