[CRIU] [PATCH 7/7] travis: enable ccache for docker/qemu builds

[Dmitry Safonov]

On 03/16/2017 07:41 PM, Kir Kolyshkin wrote:
> On 03/16/2017 07:54 AM, Dmitry Safonov wrote:
>> On 03/16/2017 02:17 AM, Kir Kolyshkin wrote:
>>> As we compile-test non-x86_64 architectures under qemu emulation,
>>> it works pretty slow.
>>> Dmitry Safonov suggested, and Andrey Vagin initially implemented
>>> supporting ccache for such builds. This patch is based heavily
>>> on Andrey's work -- all the bugs added are purely mine though.
>>>
>>> Performance results: in an ideal environment (two builds of the same
>>> code, one with cold (empty) ccache, another with the hot one)
>>> I saw compile time improvements of 4x to 5x, and total test run time
>>> improvement up to 2x to 2.5x. In layman terms, the complete test run
>>> that was taking more than 50 minutes now takes about 25!
>>>
>>> Notes on handling .ccache directory:
>>>
>>> 1. Before running docker build, .ccache directory (saved in between
>>>    runs by Travis) is moved to criu source code root, from where it
>>>    is copied by docker together with criu sources.
>>>
>>> 2. In Dockerfile, .ccache gets moved to /tmp, and CCACHE_DIR
>>>    is set accordingly.
>>>
>>> 3. After running docker build, .ccache is copied out from docker
>>>    container back to the host (from where it is saved by Travis).
>>>
>>> Ccache envorinment notes:
>>>
>>> 1. CCACHE_NOCOMPRESS is needed because otherwise tons of time is spent
>>>    on compression/decompression (big performance hit under qemu).
>>>
>>> 2. CCACHE_CPP2 is required with clang, see detailed explanation at
>>> http://petereisentraut.blogspot.com/2011/09/ccache-and-clang-part-2.html
>>>    The logic of setting CCACHE_CPP2 in Dockerfile is somewhat fancy;
>>>    unfortunately I was not able to come up with a simpler approach.
>>>
>>> Misc:
>>>
>>> 1. Travis runs "ccache -s" after the build is completed. Also, ccache -s
>>>    is called by Dockerfile before make, so you can compare before/after
>>>    numbers.
>>>
>>> 2. make invocations are surrounded by "date" calls so you can see the
>>>    timings. For the sake of accurate timings, a separate compilation
>>>    of criu/parasite-syscall.o (see commit 48f680df) is commented out.
>>
>> Hmm, so this separate compilation is a test that building not whole
>> criu, but a object file works - it was broken some time ago and I added
>> this to check that any changes to makefiles does not break it.
>>
>> I expected that building an object that anyway is needed by criu
>> wouldn't enlarge compilation time much. Did you test how much time is
>> saved here?
>
> It's not about saving compile time, it's about more accurate measurements
> of a compile time. Since this is clean build, "make
> criu/parasite-syscall.o"
> leads to building images/, compel/, libsoccr/, and criu/pie/ -- that takes
> about 1/2 of all criu compile time.

Yes, but those built objects will be used on the next `make`, no need
to rebuild them again. There is only overhead of twice-make processing,
which I expect to be low.

>
>>
>> The other questions:
>> - is exact build time needed? I mean, patches that speed up compilation
>> can provide this information, but is it always needed? (I don't object,
>> just questioning)
>
> Surely it's only needed when you try to optimize it.
>
>> - is building some object file needed? Usally I don't use it and
>> recompile all criu, but maybe there are users? Should we keep the test
>> then to ensure that that ability will not broke?
>
> I think it is needed.
>
> OK, I have an idea of how to add the test back, will send a patch soon.

Ok, good, thanks!

>
>>
>>>
>>> Signed-off-by: Kir Kolyshkin <kir at openvz.org>
>>> ---
>>>  scripts/build/Dockerfile.alpine | 13 +++++++++++--
>>>  scripts/build/Dockerfile.tmpl   | 14 +++++++++++---
>>>  scripts/build/Makefile          |  6 +++++-
>>>  3 files changed, 27 insertions(+), 6 deletions(-)
>>>
>>> diff --git a/scripts/build/Dockerfile.alpine
>>> b/scripts/build/Dockerfile.alpine
>>> index 86738a74..51da014 100644
>>> --- a/scripts/build/Dockerfile.alpine
>>> +++ b/scripts/build/Dockerfile.alpine
>>> @@ -1,5 +1,7 @@
>>>  FROM alpine:3.5
>>>  ARG CC=gcc
>>> +ARG ENV1=FOOBAR
>>> +
>>>  RUN apk update && apk add \
>>>                  build-base \
>>>                  coreutils \
>>> @@ -12,11 +14,18 @@ RUN apk update && apk add \
>>>                  libnl3-dev \
>>>                  pkgconfig \
>>>                  libnet-dev \
>>> +                ccache \
>>>                  $CC
>>>
>>>  COPY . /criu
>>>  WORKDIR /criu
>>> -RUN make mrproper && make -j $(nproc) CC=$CC
>>> +RUN mv .ccache /tmp
>>> +ENV CC="ccache $CC"
>>> +ENV CCACHE_DIR=/tmp/.ccache
>>> +ENV CCACHE_NOCOMPRESS=1
>>> +ENV $ENV1 yes
>>> +RUN ccache -s
>>> +RUN date && make mrproper && make -j $(nproc) CC="$CC" && date
>>>
>>>  # to run tests
>>>  RUN apk add    py-yaml \
>>> @@ -26,4 +35,4 @@ RUN apk add    py-yaml \
>>>          iproute2
>>>
>>>  RUN pip install protobuf
>>> -RUN make -C test/zdtm/static env00
>>> +RUN date && make -C test/zdtm/static env00 && date
>>> diff --git a/scripts/build/Dockerfile.tmpl
>>> b/scripts/build/Dockerfile.tmpl
>>> index 6e35f87..bab8aad 100644
>>> --- a/scripts/build/Dockerfile.tmpl
>>> +++ b/scripts/build/Dockerfile.tmpl
>>> @@ -1,4 +1,5 @@
>>>  ARG CC=gcc
>>> +ARG ENV1=FOOBAR
>>>
>>>  RUN apt-get update && apt-get install -y \
>>>                  build-essential \
>>> @@ -16,15 +17,22 @@ RUN apt-get update && apt-get install -y \
>>>                  pkg-config \
>>>                  git-core \
>>>          libnet-dev \
>>> +        ccache \
>>>                  $CC
>>>
>>>  COPY . /criu
>>>  WORKDIR /criu
>>> +RUN mv .ccache /tmp
>>> +ENV CC="ccache $CC"
>>> +ENV CCACHE_DIR=/tmp/.ccache
>>> +ENV CCACHE_NOCOMPRESS=1
>>> +ENV $ENV1 yes
>>>
>>>  RUN make mrproper
>>> -RUN make -j $(nproc) CC=$CC V=1 criu/parasite-syscall.o
>>> -RUN make -j $(nproc) CC=$CC V=1
>>> +RUN ccache -s
>>> +#RUN make -j $(nproc) CC=$CC V=1 criu/parasite-syscall.o
>>> +RUN date; make -j $(nproc) CC="$CC" V=1; date
>>>  RUN make mrproper
>>>  RUN bash -c 'CLEAN="$(git clean -ndx --exclude=scripts/build
>>> --exclude=.config)"; echo "${CLEAN}"; test -z "${CLEAN}"; exit $?'
>>> -RUN make -j $(nproc) CC=$CC -C test/zdtm
>>> +RUN date; make -j $(nproc) CC="$CC" -C test/zdtm; date
>>>  #RUN make test/compel/handle_binary && ./test/compel/handle_binary
>>> diff --git a/scripts/build/Makefile b/scripts/build/Makefile
>>> index fc8317b..b17bf10 100644
>>> --- a/scripts/build/Makefile
>>> +++ b/scripts/build/Makefile
>>> @@ -25,7 +25,10 @@ binfmt_misc:
>>>  $(QEMU_ARCHES): qemu-user-static binfmt_misc
>>>
>>>  $(TARGETS):
>>> -    docker build  -t criu-$@ -f Dockerfile.$@ $(DB_CC) ../..
>>> +    mkdir -p $(HOME)/.ccache
>>> +    mv $(HOME)/.ccache ../../
>>> +    docker build  -t criu-$@ -f Dockerfile.$@ $(DB_CC) $(DB_ENV) ../..
>>> +    docker run criu-$@ tar c -C /tmp .ccache | tar x -C $(HOME)
>>>  .PHONY: $(TARGETS)
>>>
>>>  # Clang builds add some Docker build env
>>> @@ -35,6 +38,7 @@ endef
>>>  $(foreach t,$(TARGETS),$(eval $(call CLANG_DEP,$(t))))
>>>
>>>  %-clang:    DB_CC=--build-arg CC=clang-3.8
>>> +%-clang:    DB_ENV=--build-arg ENV1=CCACHE_CPP2
>>>  alpine-clang:    DB_CC=--build-arg CC=clang
>>>  aarch64-clang:    DB_CC=--build-arg CC=clang-3.6
>>>  .PHONY: $(TARGETS_CLANG)
>>>
>>
>>
>


-- 
              Dmitry