[CRIU] [RFC] travis: add coveralls.io support

[Andrew Vagin]

On Fri, Jun 10, 2016 at 01:56:16PM +0300, Dmitry Safonov wrote:
> On 06/10/2016 01:31 PM, Pavel Emelyanov wrote:
> > On 06/09/2016 08:33 PM, Andrey Vagin wrote:
> > > On Thu, Jun 9, 2016 at 9:13 AM, Pavel Emelyanov <xemul at virtuozzo.com> wrote:
> > > > On 06/09/2016 06:57 PM, Andrey Vagin wrote:
> > > > > On Thu, Jun 9, 2016 at 5:46 AM, Pavel Emelyanov <xemul at virtuozzo.com> wrote:
> > > > > > Applied, thanks
> > > > > 
> > > > > Now could you revert it? We collect a coverage report in Jenkins,
> > > > > because we can't execute all tests in Travis
> > > > 
> > > > I liked Dima's arguments "for" this patch. Although not full, but
> > > > one still can get some coverage on coveralls. Are they invalid?
> > > 
> > > It isn't invalid, but how are you going to use it. We execute travis
> > > to validate patch series. Is it ok to report coverage for them? How
> > > often will travis fail due to this? How is it affect a time which is
> > > required to validate a build?
> > 
> > Dima?
> > 
> 
> Well, my arguments for this:
> - Jenkins coverage reports are rare (by now) here:
> https://coveralls.io/github/xemul/criu?branch=HEAD

It isn't a problem to generate these reports more often. And the branch
name should be fixed for these reports.

> - They don't intersect with coverage from Travis, which places them
> on topic branches master/criu-dev:
> https://coveralls.io/github/xemul/criu?branch=criu-dev

coveralls.io is a public place for these reports and I would prefer to
have full reports here.

I want to say that users have to understnad where is our full coverage
report and where is out coverage report from travis.

Other than that, I don't have objections.

Thanks,
Andrew

> - Feature owner can see on coveralls if kernel is too old and no path
> for that new feature was tested (and it's ok for him to wait for Jenkins
> coverage  or make it by hands)
> - As CRIU supports v3.11 kernel, Travis's kernel should be fine to see
> basic coverage for old features. We may also run fail injection tests
> if this patch will be accepted
> - You can see coverage right after you push and travis has built CRIU,
> no need to wait for this patches got accepted & Jenkins tests have run.
> By that time you may forgot to see all your code pathes.
> I find it kinda easier to use than `make coverage`.
> 
> It can be used after patches got applied to criu-dev like:
> "look, after your patches coverage decreased on <n> percent, could you
> provide a test for this?" Of course, if Travis can test that feature
> with it's kernel.
> 
> To the points:
> > How often will travis fail due to this?
> I don't see the reason why it would fail due to this. If it can't build
> CRIU with GCOV=1 - that's a new patch's failure. If you mean coverage
> report generation - it's done "on_success", so if that package will fail
> to do it's job to prepare/send results to coveralls.io, build will still
> success.
> 
> > How is it affect a time which is required to validate a build?
> As I didn't see time changes on my github branches, you may want to
> check with that patch now on top, time to build criu-dev:
> https://travis-ci.org/xemul/criu/builds
> It's near statistical error. At least, I do not see time increases,
> do you?
> 
> Well, to resume I may say that I have send it as RFC, so I wanted to
> hear if this approach is suitable for criu development.
> If Andrey minds about it - I'm fine about revert this patch and drop
> it on next criu-dev rebase.
> I can always see coverage for my codepath with this patch on top on my
> topic-branch, so that's totaly fine for me.
> 
> -- 
> Regards,
> Dmitry Safonov