[CRIU] [PATCH 7/7] travis: enable ccache for docker/qemu builds

Thu Mar 16 09:41:20 PDT 2017

On 03/16/2017 07:54 AM, Dmitry Safonov wrote:
> On 03/16/2017 02:17 AM, Kir Kolyshkin wrote:
>> As we compile-test non-x86_64 architectures under qemu emulation,
>> it works pretty slow.
>> Dmitry Safonov suggested, and Andrey Vagin initially implemented
>> supporting ccache for such builds. This patch is based heavily
>> on Andrey's work -- all the bugs added are purely mine though.
>>
>> Performance results: in an ideal environment (two builds of the same
>> code, one with cold (empty) ccache, another with the hot one)
>> I saw compile time improvements of 4x to 5x, and total test run time
>> improvement up to 2x to 2.5x. In layman terms, the complete test run
>> that was taking more than 50 minutes now takes about 25!
>>
>> Notes on handling .ccache directory:
>>
>> 1. Before running docker build, .ccache directory (saved in between
>>    runs by Travis) is moved to criu source code root, from where it
>>    is copied by docker together with criu sources.
>>
>> 2. In Dockerfile, .ccache gets moved to /tmp, and CCACHE_DIR
>>    is set accordingly.
>>
>> 3. After running docker build, .ccache is copied out from docker
>>    container back to the host (from where it is saved by Travis).
>>
>> Ccache envorinment notes:
>>
>> 1. CCACHE_NOCOMPRESS is needed because otherwise tons of time is spent
>>    on compression/decompression (big performance hit under qemu).
>>
>> 2. CCACHE_CPP2 is required with clang, see detailed explanation at
>> http://petereisentraut.blogspot.com/2011/09/ccache-and-clang-part-2.html
>>    The logic of setting CCACHE_CPP2 in Dockerfile is somewhat fancy;
>>    unfortunately I was not able to come up with a simpler approach.
>>
>> Misc:
>>
>> 1. Travis runs "ccache -s" after the build is completed. Also, ccache -s
>>    is called by Dockerfile before make, so you can compare before/after
>>    numbers.
>>
>> 2. make invocations are surrounded by "date" calls so you can see the
>>    timings. For the sake of accurate timings, a separate compilation
>>    of criu/parasite-syscall.o (see commit 48f680df) is commented out.
>
> Hmm, so this separate compilation is a test that building not whole
> criu, but a object file works - it was broken some time ago and I added
> this to check that any changes to makefiles does not break it.
>
> I expected that building an object that anyway is needed by criu
> wouldn't enlarge compilation time much. Did you test how much time is
> saved here?

It's not about saving compile time, it's about more accurate measurements
of a compile time. Since this is clean build, "make criu/parasite-syscall.o"
leads to building images/, compel/, libsoccr/, and criu/pie/ -- that takes
about 1/2 of all criu compile time.

>
> The other questions:
> - is exact build time needed? I mean, patches that speed up compilation
> can provide this information, but is it always needed? (I don't object,
> just questioning)

Surely it's only needed when you try to optimize it.

> - is building some object file needed? Usally I don't use it and
> recompile all criu, but maybe there are users? Should we keep the test
> then to ensure that that ability will not broke?

I think it is needed.

OK, I have an idea of how to add the test back, will send a patch soon.

>
>>
>> Signed-off-by: Kir Kolyshkin <kir at openvz.org>
>> ---
>>  scripts/build/Dockerfile.alpine | 13 +++++++++++--
>>  scripts/build/Dockerfile.tmpl   | 14 +++++++++++---
>>  scripts/build/Makefile          |  6 +++++-
>>  3 files changed, 27 insertions(+), 6 deletions(-)
>>
>> diff --git a/scripts/build/Dockerfile.alpine 
>> b/scripts/build/Dockerfile.alpine
>> index 86738a74..51da014 100644
>> --- a/scripts/build/Dockerfile.alpine
>> +++ b/scripts/build/Dockerfile.alpine
>> @@ -1,5 +1,7 @@
>>  FROM alpine:3.5
>>  ARG CC=gcc
>> +ARG ENV1=FOOBAR
>> +
>>  RUN apk update && apk add \
>>                  build-base \
>>                  coreutils \
>> @@ -12,11 +14,18 @@ RUN apk update && apk add \
>>                  libnl3-dev \
>>                  pkgconfig \
>>                  libnet-dev \
>> +                ccache \
>>                  $CC
>>
>>  COPY . /criu
>>  WORKDIR /criu
>> -RUN make mrproper && make -j $(nproc) CC=$CC
>> +RUN mv .ccache /tmp
>> +ENV CC="ccache $CC"
>> +ENV CCACHE_DIR=/tmp/.ccache
>> +ENV CCACHE_NOCOMPRESS=1
>> +ENV $ENV1 yes
>> +RUN ccache -s
>> +RUN date && make mrproper && make -j $(nproc) CC="$CC" && date
>>
>>  # to run tests
>>  RUN apk add    py-yaml \
>> @@ -26,4 +35,4 @@ RUN apk add    py-yaml \
>>          iproute2
>>
>>  RUN pip install protobuf
>> -RUN make -C test/zdtm/static env00
>> +RUN date && make -C test/zdtm/static env00 && date
>> diff --git a/scripts/build/Dockerfile.tmpl 
>> b/scripts/build/Dockerfile.tmpl
>> index 6e35f87..bab8aad 100644
>> --- a/scripts/build/Dockerfile.tmpl
>> +++ b/scripts/build/Dockerfile.tmpl
>> @@ -1,4 +1,5 @@
>>  ARG CC=gcc
>> +ARG ENV1=FOOBAR
>>
>>  RUN apt-get update && apt-get install -y \
>>                  build-essential \
>> @@ -16,15 +17,22 @@ RUN apt-get update && apt-get install -y \
>>                  pkg-config \
>>                  git-core \
>>          libnet-dev \
>> +        ccache \
>>                  $CC
>>
>>  COPY . /criu
>>  WORKDIR /criu
>> +RUN mv .ccache /tmp
>> +ENV CC="ccache $CC"
>> +ENV CCACHE_DIR=/tmp/.ccache
>> +ENV CCACHE_NOCOMPRESS=1
>> +ENV $ENV1 yes
>>
>>  RUN make mrproper
>> -RUN make -j $(nproc) CC=$CC V=1 criu/parasite-syscall.o
>> -RUN make -j $(nproc) CC=$CC V=1
>> +RUN ccache -s
>> +#RUN make -j $(nproc) CC=$CC V=1 criu/parasite-syscall.o
>> +RUN date; make -j $(nproc) CC="$CC" V=1; date
>>  RUN make mrproper
>>  RUN bash -c 'CLEAN="$(git clean -ndx --exclude=scripts/build 
>> --exclude=.config)"; echo "${CLEAN}"; test -z "${CLEAN}"; exit $?'
>> -RUN make -j $(nproc) CC=$CC -C test/zdtm
>> +RUN date; make -j $(nproc) CC="$CC" -C test/zdtm; date
>>  #RUN make test/compel/handle_binary && ./test/compel/handle_binary
>> diff --git a/scripts/build/Makefile b/scripts/build/Makefile
>> index fc8317b..b17bf10 100644
>> --- a/scripts/build/Makefile
>> +++ b/scripts/build/Makefile
>> @@ -25,7 +25,10 @@ binfmt_misc:
>>  $(QEMU_ARCHES): qemu-user-static binfmt_misc
>>
>>  $(TARGETS):
>> -    docker build  -t criu-$@ -f Dockerfile.$@ $(DB_CC) ../..
>> +    mkdir -p $(HOME)/.ccache
>> +    mv $(HOME)/.ccache ../../
>> +    docker build  -t criu-$@ -f Dockerfile.$@ $(DB_CC) $(DB_ENV) ../..
>> +    docker run criu-$@ tar c -C /tmp .ccache | tar x -C $(HOME)
>>  .PHONY: $(TARGETS)
>>
>>  # Clang builds add some Docker build env
>> @@ -35,6 +38,7 @@ endef
>>  $(foreach t,$(TARGETS),$(eval $(call CLANG_DEP,$(t))))
>>
>>  %-clang:    DB_CC=--build-arg CC=clang-3.8
>> +%-clang:    DB_ENV=--build-arg ENV1=CCACHE_CPP2
>>  alpine-clang:    DB_CC=--build-arg CC=clang
>>  aarch64-clang:    DB_CC=--build-arg CC=clang-3.6
>>  .PHONY: $(TARGETS_CLANG)
>>
>
>