[CRIU] [RFC] travis: add coveralls.io support

[Andrew Vagin]

Hi,

I get 62.57% in Travis :)

https://coveralls.io/builds/6556578

Ok, I think we can live with this patch. I'm going to send a patch to
execute more tests in Travis to maximise code coverage for the minimum
execution time.

On Fri, Jun 10, 2016 at 04:01:22PM -0700, Andrew Vagin wrote:
> On Fri, Jun 10, 2016 at 01:56:16PM +0300, Dmitry Safonov wrote:
> > On 06/10/2016 01:31 PM, Pavel Emelyanov wrote:
> > > On 06/09/2016 08:33 PM, Andrey Vagin wrote:
> > > > On Thu, Jun 9, 2016 at 9:13 AM, Pavel Emelyanov <xemul at virtuozzo.com> wrote:
> > > > > On 06/09/2016 06:57 PM, Andrey Vagin wrote:
> > > > > > On Thu, Jun 9, 2016 at 5:46 AM, Pavel Emelyanov <xemul at virtuozzo.com> wrote:
> > > > > > > Applied, thanks
> > > > > > 
> > > > > > Now could you revert it? We collect a coverage report in Jenkins,
> > > > > > because we can't execute all tests in Travis
> > > > > 
> > > > > I liked Dima's arguments "for" this patch. Although not full, but
> > > > > one still can get some coverage on coveralls. Are they invalid?
> > > > 
> > > > It isn't invalid, but how are you going to use it. We execute travis
> > > > to validate patch series. Is it ok to report coverage for them? How
> > > > often will travis fail due to this? How is it affect a time which is
> > > > required to validate a build?
> > > 
> > > Dima?
> > > 
> > 
> > Well, my arguments for this:
> > - Jenkins coverage reports are rare (by now) here:
> > https://coveralls.io/github/xemul/criu?branch=HEAD
> 
> It isn't a problem to generate these reports more often. And the branch
> name should be fixed for these reports.
> 
> > - They don't intersect with coverage from Travis, which places them
> > on topic branches master/criu-dev:
> > https://coveralls.io/github/xemul/criu?branch=criu-dev
> 
> coveralls.io is a public place for these reports and I would prefer to
> have full reports here.
> 
> I want to say that users have to understnad where is our full coverage
> report and where is out coverage report from travis.
> 
> Other than that, I don't have objections.
> 
> Thanks,
> Andrew
> 
> > - Feature owner can see on coveralls if kernel is too old and no path
> > for that new feature was tested (and it's ok for him to wait for Jenkins
> > coverage  or make it by hands)
> > - As CRIU supports v3.11 kernel, Travis's kernel should be fine to see
> > basic coverage for old features. We may also run fail injection tests
> > if this patch will be accepted
> > - You can see coverage right after you push and travis has built CRIU,
> > no need to wait for this patches got accepted & Jenkins tests have run.
> > By that time you may forgot to see all your code pathes.
> > I find it kinda easier to use than `make coverage`.
> > 
> > It can be used after patches got applied to criu-dev like:
> > "look, after your patches coverage decreased on <n> percent, could you
> > provide a test for this?" Of course, if Travis can test that feature
> > with it's kernel.
> > 
> > To the points:
> > > How often will travis fail due to this?
> > I don't see the reason why it would fail due to this. If it can't build
> > CRIU with GCOV=1 - that's a new patch's failure. If you mean coverage
> > report generation - it's done "on_success", so if that package will fail
> > to do it's job to prepare/send results to coveralls.io, build will still
> > success.
> > 
> > > How is it affect a time which is required to validate a build?
> > As I didn't see time changes on my github branches, you may want to
> > check with that patch now on top, time to build criu-dev:
> > https://travis-ci.org/xemul/criu/builds
> > It's near statistical error. At least, I do not see time increases,
> > do you?
> > 
> > Well, to resume I may say that I have send it as RFC, so I wanted to
> > hear if this approach is suitable for criu development.
> > If Andrey minds about it - I'm fine about revert this patch and drop
> > it on next criu-dev rebase.
> > I can always see coverage for my codepath with this patch on top on my
> > topic-branch, so that's totaly fine for me.
> > 
> > -- 
> > Regards,
> > Dmitry Safonov