[Users] I've noticed a new 'readykernel' package

[Konstantin Khorenko]

Hi Frank,

let me try to answer some question, please see inline.

On 11/10/2016 06:34 PM, Frank Myhr wrote:

>> all the intermediate updates are delivered to
>> our commercial customers in rebootless (kpatch) format
>>
>> As for the contributions to the community product, we still build these
>> kernels for the "factory" repository.
>
> I confess I am confused by the Virtuozzo & OpenVZ kernel development process. I
> wonder if you could answer some questions and/or point to resources that address
> them:
>
> 1) How (generally) are the commercial kpatches related to the OpenVZ kernel
> code? Is it a case of select OpenVZ commits being cherry-picked and tested for
> inclusion in the commercial kpatches? Or is it the other way around, with all
> development being done on the commercial product, and some subset of patches
> being contributed to the community project? Certainly there's no need to divulge
> all details of your development process(es), but it would be very helpful to
> have a better idea of the relationship between the Virtuozzo & OpenVZ kernels.
> (I was under the possibly mistaken impression that they were practically
> identical for OpenVZ 6.)

Virtuozzo kernels == OpenVZ kernels. On binary level.
There were no exceptions, the first one - recently release 3.10.0-327.36.1.vz7.18.8 with the fix for Dirty Cow,
this kernel was released for OpenVZ only,
Virtuozzo users do not need this kernel, Dirty Cow fix (and others) are provided to Virtuozzo via ReadyKernel live patches.

Basically the kernel development process is the following:
imagine we have a kernel branch, we plan to release a kernel as a Virtuozzo kernel in Update N.
We develop features, fix bugs, at some point we have a feature freeze, so the branch gets only fixes,
at some point we decide start commiting only fixes for critical issues to the branch
and finally we decide we are ready to release the Update N (and thus the kernel).

At this moment we write announcements, push packages (including vzkernel) to Virtuozzo yum repos
and to https://download.openvz.org/virtuozzo/

So this becomes the stable kernel both for Virtuozzo and OpenVZ.

Then we start (in fact - continue, work is started earlier) to develop next kernel branch for next Update N+1.

Virtuozzo updates which include vzkernel are quite rare now, we target at 1 vzkernel per quarter:
people do not like to reboot nodes but on the other hand want nodes to be timely updated with important fixes,
so all other important security/stability kernel fixes are delivered now via ReadyKernel to Virtuozzo users.

There are physically no full vzkernels which equals to stable vzkernel of previous update + fixes which are provided via ReadyKernel.
They are not commited, built, tested.

OpenVZ factory repo (https://download.openvz.org/virtuozzo/factory/)
contains nightly snapshot of OpenVZ build we are working on.

Related to vzkernel: at the moment we work on next Virtuozzo 7 Update,
we plan to release a kernel from branch branch-rh7-3.10.0-327.36.1.vz7.19.x-ovz,
so factory OpenVZ repo contains vzkernel from that branch.

> 2) Why are factory builds apparently newer than available source code? I.e:
> http://mirror.yandex.ru/mirrors/download.openvz.org/virtuozzo/factory/x86_64/os/Packages/v/
> has 7.19.7 (and had 7.19.4, 7.19.5, and 7.19.6 before that)
>
> while newest source code tag is still 19.3?:
> https://src.openvz.org/projects/OVZ/repos/vzkernel/browse

This is a matter of security: from time to time we find security issues internally,
sometimes there are security issues which are still under embargo,
but we don't want OpenVZ users to be affected by them on one hand - thus we build vzkernel binaries and they are pushed to factory repo.
On the other hand we don't want bad guys to monitor our vzkernel git and use vulnerabilities right after commit is published but before
users have rebooted their nodes.
And in case of issue under embargo - we just cannot publish fixes for them until embargo is over.

=> we made a delay of git sources appeared but do publish binaries first so people can defend themselves.

> 3) Are older factory builds archived online somewhere?

Nope. Feel free to create a grabber and store them if needed.

> 4) I really like how the wiki is organized for OpenVZ 6 kernels. For instance:
> https://wiki.openvz.org/Download/kernel/rhel6-testing/042stab120.7
>
> I know your distribution model has changed significantly for OpenVZ 7, so it's
> probably impossible to keep exactly the same wiki format for the OpenVZ 7
> kernels. But it would be VERY helpful if for each factory kernel you could
> publish in the wiki:
>
> a) Link to factory kernel -- preferably a link that will keep working when a
> newer factory build becomes available.

Well, as currently we don't have a repo with old factory vzkernels, there is only one link:
https://download.openvz.org/virtuozzo/factory/x86_64/os/Packages/v/

> b) List of changes. The human-readable changes for the OpenVZ 6 kernels are
> exemplary. But even an automatically-generated list of changes would be helpful.

May be "git log" will work for you?
There are vzkernels tags, so "git log --oneline tag1..tag2" gives you the generated short info.

This is again a matter of security.
We cannot create that (full) description on kernel build stage - because there may be some security fixes
which we don't want/are not allowed to publish yet.
To skip security-related issues? => the changelog will be incomplete.
To fix it adding skipped descriptions later manually? Well, not that good way, too many builds.

>> We... invite
>> anyone from the OpenVZ community to contribute resources into testing of
>> these intermediate kernel updates - we will be more than happy to cooperate
>> and publish these kernels appropriately tagged as stable/community tested.
>>
>> So please consider it as open invitation and volunteering opportunity to
>> contribute into the project
>
> I do want to contribute to the community project.
>
> With regard to testing kernels: as Volker Janzen has remarked, it is difficult
> to know how to start. I'm not a kernel developer, but am certainly willing to
> contribute some cpu time to testing, IF this will actually help OpenVZ kernel
> developers (and eventually myself and the rest of the community) and not simply
> waste everybody's time.
>
> For now I will contribute with the wiki and with (modest) financial donations.

We do appreciate this! We do know the wiki requires a lot of (re)work, so it would be really great if you help us with it.

Thank you.

--
Best regards,

Konstantin Khorenko,
Virtuozzo Linux Kernel Team