<div dir="ltr"><div><div><div>Hi Vasily,<br><br></div>I've upgraded two nodes last week from 113.12 to 113.21 and it seems better. Backups last weekend took the same time as it was on <=108.8. I'll still keep an eye on this and also on the development of 115 in OpenVZ Jira.<br><br></div>Thanks!<br><br></div>Karl<br></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Mar 31, 2016 at 4:13 AM, Vasily Averin <span dir="ltr"><<a href="mailto:vvs@virtuozzo.com" target="_blank">vvs@virtuozzo.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">On 30.03.2016 18:38, Karl Johnson wrote:<br>
> Hi Vasily,<br>
><br>
</span><span class="">> I do indeed use simfs / ext4 / cfq. Only a backup of each containers<br>
> private areas is done with vzdump and then transferred to a backup<br>
> server with ncftpput. Compressing the data is OK while transferring<br>
> the dump over local network peak the load so the issue is with (read)<br>
> IO. I’m trying to find out why it was fine before and cause problem<br>
> now. Those nodes are in heavy production so it’s hard to do testing<br>
> (including downgrading kernel).<br>
<br>
</span>Few lists of blocked processes taken on alt+sysrq+W "magic sysrq" key can be useful,<br>
it allows to see who is blocked, and it allows to see dynamic of process,<br>
but it does not explain who causes this traffic jam.<br>
<br>
I'm sorry, but another ways of troubleshooting are much more destuctive.<br>
Moreover even kernel crash dump does not guarantee success in your case.<br>
It allows to see whole picture with all details,<br>
but it does not allow to understand the dynamic of process.<br>
<span class=""><br>
> Thanks for all the information on futur roadmap. I’m glad that the<br>
> work as already begun on RHEL 6.8 rebase. I read the beta technical<br>
> notes last week and some upgrades seem great. Do you consider<br>
> 042stab114.5 stable even if it’s in the testing repo? I might try it<br>
> tomorrow and see how it goes.<br>
<br>
</span>In fact we do not know yet.<br>
<br>
114.x kernels includes ~30 new patches from Red Hat and ~10 our ones,<br>
and we had few minor rejects only during re-base.<br>
At the first glance it should not cause problems,<br>
but first 114.x kernel was crashed on boot,<br>
and 114.4 was crashed after CT suspend-resume.<br>
In both cases we was need to re-work our patches.<br>
<br>
042stab114.5 kernel work well on my test node right now,<br>
but it is not ready for production yet and requires careful re-testing.<br>
So if you have some specific workload, we would be very grateful<br>
for any testing and bugreports.<br>
It allows us to know about hidden bugs before release.<br>
<br>
thank you,<br>
Vasily Averin<br>
<br>
> Regards,<br>
><br>
> Karl<br>
<div><div class="h5">><br>
> On Wed, Mar 30, 2016 at 5:48 AM, Vasily Averin <<a href="mailto:vvs@virtuozzo.com">vvs@virtuozzo.com</a> <mailto:<a href="mailto:vvs@virtuozzo.com">vvs@virtuozzo.com</a>>> wrote:<br>
><br>
> Dear Karl,<br>
><br>
> thank you for explanation.<br>
> however some details are still not it clear.<br>
><br>
> I believe you use simfs containers (otherwise you can do not worry about PSBM-34244,<br>
> using of 113.12 kernels also confirms it)<br>
> but it isn't clear how exactly you backup your nodes.<br>
> Do you dump whole partition with containers or just copy containers private areas somehow?<br>
> What filesystem you have on partition with containers.<br>
> What is backup storage in your case?<br>
><br>
> Anyway seems you do not freeze filesystem with containers before backup.<br>
> This functionality was broken in RHEL6 kernels quite long time,<br>
> and Red Hat fixed it in 2.6.32-504.x and 573.x kernels.<br>
><br>
> <a href="https://access.redhat.com/solutions/1506563" rel="noreferrer" target="_blank">https://access.redhat.com/solutions/1506563</a><br>
><br>
> Probably these fixes affect your testcase.<br>
><br>
> I'm not sure of course,<br>
> may be it isn't and some other fixes are guilty:<br>
> Red Hat added >7000 new patches into 2.6.32-573.x kernels<br>
> many our patches was changed during re-base,<br>
> and many new patches was added.<br>
> There was to many changes between 108.x and 113.x kernels.<br>
><br>
> Our tests did not detected significant performance degradation,<br>
> but it means nothing, most likely we just did not measured your testcase.<br>
><br>
> I do not expect that situation will be changed on 113.21 kernel,<br>
> seems we did not fixed similar issues last time.<br>
><br>
> Yes, you-re right, our 042stab114.x kernels will be based<br>
> on last released RHEL6.7 kernel 2.6.32-573.22.1.el6.<br>
> its validation is in progress at present,<br>
> and I hope we'll publish it in nearest future.<br>
><br>
> However I did not found any related bugfixes in new RHEL6 kernels,<br>
> and doubt that it helps you.<br>
><br>
> Also we're going to make 115.x kernel based on RHEL6 update8 beta kernel 2.6.32-621.el6,<br>
> it have no chances to be released in stable branch but its testing helps us to speed-up<br>
> our rebase to RHEL6.8 release kernel (we expect RHEL6u8 will be released in end of May).<br>
><br>
> The work on 115.x kernel is in progress, and I hope it should be done in next few days.<br>
><br>
> So I would like to propose you following plan:<br>
> please check how works 113.21, 114.x and 115.x kernels, (may be it works already)<br>
> if issue will be still present, please reproduce the problem once again, crash affected host,<br>
> create new bug in jira and push me again. I'll send you private link for vmcore uploading.<br>
> Investigation of kernel crash dump file probably allows me to find bottleneck in your case.<br>
><br>
> Thank you,<br>
> Vasily Averin<br>
><br>
> On 29.03.2016 21:03, Karl Johnson wrote:<br>
> > Hi Vasily,<br>
> ><br>
> > Every weekend I do backups of all CT which take a lot of IO. It<br>
> > didn't affect much load average before 108 but as soon as I upgraded<br>
> > to 113, load got very high and nodes became sluggish during backups.<br>
> > It might be something else but I was looking for feedback if someone<br>
> > else had the same issue. I will continue to troubleshoot this issue.<br>
> > Meanwhile, I will upgrade them from 113.12 to 113.21 and see how it<br>
> > goes even if there's nothing related to this in the changelog.<br>
> ><br>
> > Thanks for the reply,<br>
> ><br>
> > Karl<br>
> ><br>
</div></div><div><div class="h5">> > On Tue, Mar 29, 2016 at 5:21 AM, Vasily Averin <<a href="mailto:vvs@virtuozzo.com">vvs@virtuozzo.com</a> <mailto:<a href="mailto:vvs@virtuozzo.com">vvs@virtuozzo.com</a>> <mailto:<a href="mailto:vvs@virtuozzo.com">vvs@virtuozzo.com</a> <mailto:<a href="mailto:vvs@virtuozzo.com">vvs@virtuozzo.com</a>>>> wrote:<br>
> ><br>
> > Dear Karl,<br>
> ><br>
> > no, we know nothing about possible performance degradation between<br>
> > 042stab108.x and 042stab113.x kernels.<br>
> > High load average and CPU peaks are not a problems per se,<br>
> > it can be caused by increased activity on your nodes.<br>
> ><br>
> > Could you please explain in more details,<br>
> > why you believe you have a problem on your nodes?<br>
> ><br>
> > Thank you,<br>
> > Vasily Averin<br>
> ><br>
> > On 28.03.2016 20:28, Karl Johnson wrote:<br>
> > > Hello,<br>
> > ><br>
> > > Did anyone notice performance degradation after upgrading vzkernel to<br>
> > > 042stab113.X? I’ve been running 042stab108.5 on few nodes for a while<br>
> > > with no issue and upgraded to 042stab113.12 few weeks ago to fix an<br>
> > > important CVE and rebase to latest rhel6 kernel.<br>
> > ><br>
> > > Since the upgrade from 108.5 to 113.12, I noticed much higher load<br>
> > > average on those upgraded OpenVZ nodes, mostly when IO is heavily<br>
> > > used. High CPU peaks are much more frequent. I would be curious to<br>
> > > know if someone else has the same issue. I wouldn’t downgrade because<br>
> > > of security fix PSBM-34244.<br>
> > ><br>
> > > Regards,<br>
> > ><br>
> > > Karl<br>
> > ><br>
> > ><br>
> > > _______________________________________________<br>
> > > Users mailing list<br>
</div></div>> > > <a href="mailto:Users@openvz.org">Users@openvz.org</a> <mailto:<a href="mailto:Users@openvz.org">Users@openvz.org</a>> <mailto:<a href="mailto:Users@openvz.org">Users@openvz.org</a> <mailto:<a href="mailto:Users@openvz.org">Users@openvz.org</a>>><br>
<span class="">> > > <a href="https://lists.openvz.org/mailman/listinfo/users" rel="noreferrer" target="_blank">https://lists.openvz.org/mailman/listinfo/users</a><br>
> > ><br>
> > _______________________________________________<br>
> > Users mailing list<br>
</span>> > <a href="mailto:Users@openvz.org">Users@openvz.org</a> <mailto:<a href="mailto:Users@openvz.org">Users@openvz.org</a>> <mailto:<a href="mailto:Users@openvz.org">Users@openvz.org</a> <mailto:<a href="mailto:Users@openvz.org">Users@openvz.org</a>>><br>
<div class="HOEnZb"><div class="h5">> > <a href="https://lists.openvz.org/mailman/listinfo/users" rel="noreferrer" target="_blank">https://lists.openvz.org/mailman/listinfo/users</a><br>
> ><br>
> ><br>
> ><br>
> ><br>
> > _______________________________________________<br>
> > Users mailing list<br>
> > <a href="mailto:Users@openvz.org">Users@openvz.org</a> <mailto:<a href="mailto:Users@openvz.org">Users@openvz.org</a>><br>
> > <a href="https://lists.openvz.org/mailman/listinfo/users" rel="noreferrer" target="_blank">https://lists.openvz.org/mailman/listinfo/users</a><br>
> ><br>
> _______________________________________________<br>
> Users mailing list<br>
> <a href="mailto:Users@openvz.org">Users@openvz.org</a> <mailto:<a href="mailto:Users@openvz.org">Users@openvz.org</a>><br>
> <a href="https://lists.openvz.org/mailman/listinfo/users" rel="noreferrer" target="_blank">https://lists.openvz.org/mailman/listinfo/users</a><br>
><br>
><br>
><br>
><br>
> _______________________________________________<br>
> Users mailing list<br>
> <a href="mailto:Users@openvz.org">Users@openvz.org</a><br>
> <a href="https://lists.openvz.org/mailman/listinfo/users" rel="noreferrer" target="_blank">https://lists.openvz.org/mailman/listinfo/users</a><br>
><br>
_______________________________________________<br>
Users mailing list<br>
<a href="mailto:Users@openvz.org">Users@openvz.org</a><br>
<a href="https://lists.openvz.org/mailman/listinfo/users" rel="noreferrer" target="_blank">https://lists.openvz.org/mailman/listinfo/users</a><br>
</div></div></blockquote></div><br></div>