[Vserver] Re: [Devel] Container Test Campaign

[Kirill Korotaev]

Gerrit,

>>>>I assuming you are doing your tests on the same system (i.e. same 
>>>>compiler/libs/whatever else), and you do not change that system over 
>>>>time (i.e. you do not upgrade gcc on it in between the tests).
>>>
>>>
>>>I hope! :)
>>
>>All binaries should be built statically to work the same way inside host/guest or
>>you need to make sure that you have exactly the same versions of glibc and other
>>system libraries. At least glibc can affect perforamnce very much :/
> 
> 
> Ick - no one builds binaries statically in the real world.  And,
> when you build binaries statically, you lose all ability to fix
> security problems in base libraries by doing an update of that library.
> Instead, all applications need to be rebuilt.
> 
> Performance tests should reflect real end user usage - not contrived
> situations that make a particular solution look better or worse.
> If glibc can affect performance, that should be demonstrated in the
> real performance results - it is part of the impact of the solution and
> may need an additional solution or discussion.
What I tried to say is that performance results done in different
environments are not comparable so have no much meaning. I don't want us
to waste our time digging in why one environment is a bif faster or slower than another.
I hope you don't want too.

Now, to have the same environment there are at least 2 ways:
- make static binaries (not that good, but easiest way)
- have exactly the same packages in host/VPS for all test cases.

BTW, I also prefer 2nd way, but it is harder.

Thanks,
Kirill