[CRIU] [PATCH 7/7] travis: enable ccache for docker/qemu builds

[Dmitry Safonov]

On 03/16/2017 02:17 AM, Kir Kolyshkin wrote:
> As we compile-test non-x86_64 architectures under qemu emulation,
> it works pretty slow.
> Dmitry Safonov suggested, and Andrey Vagin initially implemented
> supporting ccache for such builds. This patch is based heavily
> on Andrey's work -- all the bugs added are purely mine though.
>
> Performance results: in an ideal environment (two builds of the same
> code, one with cold (empty) ccache, another with the hot one)
> I saw compile time improvements of 4x to 5x, and total test run time
> improvement up to 2x to 2.5x. In layman terms, the complete test run
> that was taking more than 50 minutes now takes about 25!
>
> Notes on handling .ccache directory:
>
> 1. Before running docker build, .ccache directory (saved in between
>    runs by Travis) is moved to criu source code root, from where it
>    is copied by docker together with criu sources.
>
> 2. In Dockerfile, .ccache gets moved to /tmp, and CCACHE_DIR
>    is set accordingly.
>
> 3. After running docker build, .ccache is copied out from docker
>    container back to the host (from where it is saved by Travis).
>
> Ccache envorinment notes:
>
> 1. CCACHE_NOCOMPRESS is needed because otherwise tons of time is spent
>    on compression/decompression (big performance hit under qemu).
>
> 2. CCACHE_CPP2 is required with clang, see detailed explanation at
>    http://petereisentraut.blogspot.com/2011/09/ccache-and-clang-part-2.html
>    The logic of setting CCACHE_CPP2 in Dockerfile is somewhat fancy;
>    unfortunately I was not able to come up with a simpler approach.
>
> Misc:
>
> 1. Travis runs "ccache -s" after the build is completed. Also, ccache -s
>    is called by Dockerfile before make, so you can compare before/after
>    numbers.
>
> 2. make invocations are surrounded by "date" calls so you can see the
>    timings. For the sake of accurate timings, a separate compilation
>    of criu/parasite-syscall.o (see commit 48f680df) is commented out.

Hmm, so this separate compilation is a test that building not whole
criu, but a object file works - it was broken some time ago and I added
this to check that any changes to makefiles does not break it.

I expected that building an object that anyway is needed by criu
wouldn't enlarge compilation time much. Did you test how much time is
saved here?

The other questions:
- is exact build time needed? I mean, patches that speed up compilation
can provide this information, but is it always needed? (I don't object,
just questioning)
- is building some object file needed? Usally I don't use it and
recompile all criu, but maybe there are users? Should we keep the test
then to ensure that that ability will not broke?

>
> Signed-off-by: Kir Kolyshkin <kir at openvz.org>
> ---
>  scripts/build/Dockerfile.alpine | 13 +++++++++++--
>  scripts/build/Dockerfile.tmpl   | 14 +++++++++++---
>  scripts/build/Makefile          |  6 +++++-
>  3 files changed, 27 insertions(+), 6 deletions(-)
>
> diff --git a/scripts/build/Dockerfile.alpine b/scripts/build/Dockerfile.alpine
> index 86738a74..51da014 100644
> --- a/scripts/build/Dockerfile.alpine
> +++ b/scripts/build/Dockerfile.alpine
> @@ -1,5 +1,7 @@
>  FROM alpine:3.5
>  ARG CC=gcc
> +ARG ENV1=FOOBAR
> +
>  RUN apk update && apk add \
>                  build-base \
>                  coreutils \
> @@ -12,11 +14,18 @@ RUN apk update && apk add \
>                  libnl3-dev \
>                  pkgconfig \
>                  libnet-dev \
> +                ccache \
>                  $CC
>
>  COPY . /criu
>  WORKDIR /criu
> -RUN make mrproper && make -j $(nproc) CC=$CC
> +RUN mv .ccache /tmp
> +ENV CC="ccache $CC"
> +ENV CCACHE_DIR=/tmp/.ccache
> +ENV CCACHE_NOCOMPRESS=1
> +ENV $ENV1 yes
> +RUN ccache -s
> +RUN date && make mrproper && make -j $(nproc) CC="$CC" && date
>
>  # to run tests
>  RUN apk add	py-yaml \
> @@ -26,4 +35,4 @@ RUN apk add	py-yaml \
>  		iproute2
>
>  RUN pip install protobuf
> -RUN make -C test/zdtm/static env00
> +RUN date && make -C test/zdtm/static env00 && date
> diff --git a/scripts/build/Dockerfile.tmpl b/scripts/build/Dockerfile.tmpl
> index 6e35f87..bab8aad 100644
> --- a/scripts/build/Dockerfile.tmpl
> +++ b/scripts/build/Dockerfile.tmpl
> @@ -1,4 +1,5 @@
>  ARG CC=gcc
> +ARG ENV1=FOOBAR
>
>  RUN apt-get update && apt-get install -y \
>                  build-essential \
> @@ -16,15 +17,22 @@ RUN apt-get update && apt-get install -y \
>                  pkg-config \
>                  git-core \
>  		libnet-dev \
> +		ccache \
>                  $CC
>
>  COPY . /criu
>  WORKDIR /criu
> +RUN mv .ccache /tmp
> +ENV CC="ccache $CC"
> +ENV CCACHE_DIR=/tmp/.ccache
> +ENV CCACHE_NOCOMPRESS=1
> +ENV $ENV1 yes
>
>  RUN make mrproper
> -RUN make -j $(nproc) CC=$CC V=1 criu/parasite-syscall.o
> -RUN make -j $(nproc) CC=$CC V=1
> +RUN ccache -s
> +#RUN make -j $(nproc) CC=$CC V=1 criu/parasite-syscall.o
> +RUN date; make -j $(nproc) CC="$CC" V=1; date
>  RUN make mrproper
>  RUN bash -c 'CLEAN="$(git clean -ndx --exclude=scripts/build --exclude=.config)"; echo "${CLEAN}"; test -z "${CLEAN}"; exit $?'
> -RUN make -j $(nproc) CC=$CC -C test/zdtm
> +RUN date; make -j $(nproc) CC="$CC" -C test/zdtm; date
>  #RUN make test/compel/handle_binary && ./test/compel/handle_binary
> diff --git a/scripts/build/Makefile b/scripts/build/Makefile
> index fc8317b..b17bf10 100644
> --- a/scripts/build/Makefile
> +++ b/scripts/build/Makefile
> @@ -25,7 +25,10 @@ binfmt_misc:
>  $(QEMU_ARCHES): qemu-user-static binfmt_misc
>
>  $(TARGETS):
> -	docker build  -t criu-$@ -f Dockerfile.$@ $(DB_CC) ../..
> +	mkdir -p $(HOME)/.ccache
> +	mv $(HOME)/.ccache ../../
> +	docker build  -t criu-$@ -f Dockerfile.$@ $(DB_CC) $(DB_ENV) ../..
> +	docker run criu-$@ tar c -C /tmp .ccache | tar x -C $(HOME)
>  .PHONY: $(TARGETS)
>
>  # Clang builds add some Docker build env
> @@ -35,6 +38,7 @@ endef
>  $(foreach t,$(TARGETS),$(eval $(call CLANG_DEP,$(t))))
>
>  %-clang:	DB_CC=--build-arg CC=clang-3.8
> +%-clang:	DB_ENV=--build-arg ENV1=CCACHE_CPP2
>  alpine-clang:	DB_CC=--build-arg CC=clang
>  aarch64-clang:	DB_CC=--build-arg CC=clang-3.6
>  .PHONY: $(TARGETS_CLANG)
>


-- 
              Dmitry