[CRIU] [RFC] travis: add coveralls.io support

[Dmitry Safonov]

On 06/10/2016 01:31 PM, Pavel Emelyanov wrote:
> On 06/09/2016 08:33 PM, Andrey Vagin wrote:
>> On Thu, Jun 9, 2016 at 9:13 AM, Pavel Emelyanov <xemul at virtuozzo.com> wrote:
>>> On 06/09/2016 06:57 PM, Andrey Vagin wrote:
>>>> On Thu, Jun 9, 2016 at 5:46 AM, Pavel Emelyanov <xemul at virtuozzo.com> wrote:
>>>>> Applied, thanks
>>>>
>>>> Now could you revert it? We collect a coverage report in Jenkins,
>>>> because we can't execute all tests in Travis
>>>
>>> I liked Dima's arguments "for" this patch. Although not full, but
>>> one still can get some coverage on coveralls. Are they invalid?
>>
>> It isn't invalid, but how are you going to use it. We execute travis
>> to validate patch series. Is it ok to report coverage for them? How
>> often will travis fail due to this? How is it affect a time which is
>> required to validate a build?
>
> Dima?
>

Well, my arguments for this:
- Jenkins coverage reports are rare (by now) here: 
https://coveralls.io/github/xemul/criu?branch=HEAD
- They don't intersect with coverage from Travis, which places them
on topic branches master/criu-dev: 
https://coveralls.io/github/xemul/criu?branch=criu-dev
- Feature owner can see on coveralls if kernel is too old and no path
for that new feature was tested (and it's ok for him to wait for Jenkins
coverage  or make it by hands)
- As CRIU supports v3.11 kernel, Travis's kernel should be fine to see
basic coverage for old features. We may also run fail injection tests
if this patch will be accepted
- You can see coverage right after you push and travis has built CRIU,
no need to wait for this patches got accepted & Jenkins tests have run.
By that time you may forgot to see all your code pathes.
I find it kinda easier to use than `make coverage`.

It can be used after patches got applied to criu-dev like:
"look, after your patches coverage decreased on <n> percent, could you
provide a test for this?" Of course, if Travis can test that feature
with it's kernel.

To the points:
 > How often will travis fail due to this?
I don't see the reason why it would fail due to this. If it can't build
CRIU with GCOV=1 - that's a new patch's failure. If you mean coverage
report generation - it's done "on_success", so if that package will fail
to do it's job to prepare/send results to coveralls.io, build will still
success.

 > How is it affect a time which is required to validate a build?
As I didn't see time changes on my github branches, you may want to
check with that patch now on top, time to build criu-dev:
https://travis-ci.org/xemul/criu/builds
It's near statistical error. At least, I do not see time increases,
do you?

Well, to resume I may say that I have send it as RFC, so I wanted to
hear if this approach is suitable for criu development.
If Andrey minds about it - I'm fine about revert this patch and drop
it on next criu-dev rebase.
I can always see coverage for my codepath with this patch on top on my
topic-branch, so that's totaly fine for me.

-- 
Regards,
Dmitry Safonov