[CRIU] [PATCH 7/7 v2] travis: enable ccache for docker/qemu builds

[Dmitry Safonov]

On 03/17/2017 12:27 AM, Kir Kolyshkin wrote:
> As we compile-test non-x86_64 architectures under qemu emulation,
> it works pretty slow.
> Dmitry Safonov suggested, and Andrey Vagin initially implemented
> supporting ccache for such builds. This patch is based heavily
> on Andrey's work -- all the bugs added are purely mine though.
>
> Performance results: in an ideal environment (two builds of the same
> code, one with cold (empty) ccache, another with the hot one)
> I saw compile time improvements of 4x to 5x, and total test run time
> improvement up to 2x to 2.5x. In layman terms, the complete test run
> that was taking more than 50 minutes now takes about 25!
>
> Notes on handling .ccache directory:
>
> 1. Before running docker build, .ccache directory (saved in between
>    runs by Travis) is moved to criu source code root, from where it
>    is copied by docker together with criu sources.
>
> 2. In Dockerfile, .ccache gets moved to /tmp, and CCACHE_DIR
>    is set accordingly.
>
> 3. After running docker build, .ccache is copied out from docker
>    container back to the host (from where it is saved by Travis).
>
> Ccache envorinment notes:
>
> 1. CCACHE_NOCOMPRESS is needed because otherwise tons of time is spent
>    on compression/decompression (big performance hit under qemu).
>
> 2. CCACHE_CPP2 is required with clang, see detailed explanation at
>    http://petereisentraut.blogspot.com/2011/09/ccache-and-clang-part-2.html
>    The logic of setting CCACHE_CPP2 in Dockerfile is somewhat fancy;
>    unfortunately I was not able to come up with a simpler approach.
>
> Misc:
>
> 1. Travis runs "ccache -s" after the build is completed. A call to
>    "ccache -s" is called to Dockerfile before make, so one can
>    compare before/after numbers.
>
> 2. make invocations are surrounded by "date" calls so one can get the
>    compilation times.
>
> Changes in v2:
>  - consolidate Dockerfile statements (ENV, RUN)
>  - single object make test is no longer commented out
>  - simplify "make mrproper" test
>
> Signed-off-by: Kir Kolyshkin <kir at openvz.org>

Thanks,
Reviewed-by: Dmitry Safonov <dsafonov at virtuozzo.com>

> ---
>  scripts/build/Dockerfile.alpine | 16 ++++++++++------
>  scripts/build/Dockerfile.tmpl   | 23 +++++++++++++++++------
>  scripts/build/Makefile          |  6 +++++-
>  3 files changed, 32 insertions(+), 13 deletions(-)
>
> diff --git a/scripts/build/Dockerfile.alpine b/scripts/build/Dockerfile.alpine
> index 86738a74..26c39ac 100644
> --- a/scripts/build/Dockerfile.alpine
> +++ b/scripts/build/Dockerfile.alpine
> @@ -1,5 +1,7 @@
>  FROM alpine:3.5
>  ARG CC=gcc
> +ARG ENV1=FOOBAR
> +
>  RUN apk update && apk add \
>                  build-base \
>                  coreutils \
> @@ -12,18 +14,20 @@ RUN apk update && apk add \
>                  libnl3-dev \
>                  pkgconfig \
>                  libnet-dev \
> +                ccache \
>                  $CC
>
>  COPY . /criu
>  WORKDIR /criu
> -RUN make mrproper && make -j $(nproc) CC=$CC
> +ENV CC="ccache $CC" CCACHE_DIR=/tmp/.ccache CCACHE_NOCOMPRESS=1 $ENV1=yes
> +RUN mv .ccache /tmp; make mrproper; ccache -s; \
> +	date; make -j $(nproc) CC="$CC"; date
>
> -# to run tests
> +# Run a test
>  RUN apk add	py-yaml \
>  		py-pip	\
>  		ip6tables \
>  		iptables \
> -		iproute2
> -
> -RUN pip install protobuf
> -RUN make -C test/zdtm/static env00
> +		iproute2 \
> +	&& pip install protobuf \
> +	&& make -C test/zdtm/static env00
> diff --git a/scripts/build/Dockerfile.tmpl b/scripts/build/Dockerfile.tmpl
> index 6e35f87..60e0e45 100644
> --- a/scripts/build/Dockerfile.tmpl
> +++ b/scripts/build/Dockerfile.tmpl
> @@ -1,4 +1,5 @@
>  ARG CC=gcc
> +ARG ENV1=FOOBAR
>
>  RUN apt-get update && apt-get install -y \
>                  build-essential \
> @@ -16,15 +17,25 @@ RUN apt-get update && apt-get install -y \
>                  pkg-config \
>                  git-core \
>  		libnet-dev \
> +		ccache \
>                  $CC
>
>  COPY . /criu
>  WORKDIR /criu
> +ENV CC="ccache $CC" CCACHE_DIR=/tmp/.ccache CCACHE_NOCOMPRESS=1 $ENV1=yes
> +
> +RUN mv .ccache /tmp && make mrproper; ccache -s; \
> +	date; \
> +# Check single object build
> +	make -j $(nproc) CC="$CC" criu/parasite-syscall.o; \
> +# Compile criu
> +	make -j $(nproc) CC="$CC"; \
> +	date; \
> +# Check that "make mrproper" works
> +	make mrproper && ! git clean -ndx --exclude=scripts/build \
> +	--exclude=.config --exclude=test | grep .
> +
> +# Compile tests
> +RUN date; make -j $(nproc) CC="$CC" -C test/zdtm; date
>
> -RUN make mrproper
> -RUN make -j $(nproc) CC=$CC V=1 criu/parasite-syscall.o
> -RUN make -j $(nproc) CC=$CC V=1
> -RUN make mrproper
> -RUN bash -c 'CLEAN="$(git clean -ndx --exclude=scripts/build --exclude=.config)"; echo "${CLEAN}"; test -z "${CLEAN}"; exit $?'
> -RUN make -j $(nproc) CC=$CC -C test/zdtm
>  #RUN make test/compel/handle_binary && ./test/compel/handle_binary
> diff --git a/scripts/build/Makefile b/scripts/build/Makefile
> index fc8317b..b17bf10 100644
> --- a/scripts/build/Makefile
> +++ b/scripts/build/Makefile
> @@ -25,7 +25,10 @@ binfmt_misc:
>  $(QEMU_ARCHES): qemu-user-static binfmt_misc
>
>  $(TARGETS):
> -	docker build  -t criu-$@ -f Dockerfile.$@ $(DB_CC) ../..
> +	mkdir -p $(HOME)/.ccache
> +	mv $(HOME)/.ccache ../../
> +	docker build  -t criu-$@ -f Dockerfile.$@ $(DB_CC) $(DB_ENV) ../..
> +	docker run criu-$@ tar c -C /tmp .ccache | tar x -C $(HOME)
>  .PHONY: $(TARGETS)
>
>  # Clang builds add some Docker build env
> @@ -35,6 +38,7 @@ endef
>  $(foreach t,$(TARGETS),$(eval $(call CLANG_DEP,$(t))))
>
>  %-clang:	DB_CC=--build-arg CC=clang-3.8
> +%-clang:	DB_ENV=--build-arg ENV1=CCACHE_CPP2
>  alpine-clang:	DB_CC=--build-arg CC=clang
>  aarch64-clang:	DB_CC=--build-arg CC=clang-3.6
>  .PHONY: $(TARGETS_CLANG)
>


-- 
              Dmitry