[Users] OOM Killer invoked even that are available RAM and swap on the machine

CoolCold coolthecold at gmail.com
Fri Sep 29 18:02:29 MSK 2017


it's actually 1st time I see limits for _host_ itself, is it somehow
default installation or you modified it for some reason?

On Fri, Sep 29, 2017 at 7:27 PM, Danny Gurman <DannyG at radware.com> wrote:
> Vasily - you are the man
> you were correct  ! :-)
>
>
> Setting /etc/vz/conf/0.conf with
> "PHYSPAGES="0:unlimited"
> SWAPPAGES="0:unlimited"
>
> Indeed solve the issue :-)
>
> Thanks so much for your assistant and effort,
> Danny
>
>
> -----Original Message-----
> From: Vasily Averin [mailto:vvs at virtuozzo.com]
> Sent: Friday, September 29, 2017 2:03 PM
> To: Danny Gurman <DannyG at Radware.com>
> Subject: Re: [Users] OOM Killer invoked even that are available RAM and swap on the machine
>
> On 2017-09-29 13:32, Danny Gurman wrote:
>> Thanks again Vasily,
>>
>> Here are the values from   /etc/vz/conf/0.conf:
>> # UBC parameters (in form of barrier:limit)
>> OOMGUARPAGES="unlimited:unlimited"
>> CPUS="4"
>> CPUMASK="0-3"
>> #CPUUNITS="400000"
>> PHYSPAGES="0:8G"
>> SWAPPAGES="0:4G"
>>
>> So , what the  physical pages limitation  on  VE0 (host)  actually mean ?
>> Host is limited to 8 GB RAM?
>> Note that we reach ~ 13 GB.
>
> yes, I was wrong in previous letter,
> I  forget that PAGES paraments are showed in 4kb pages so RAM =  2097152 means not 2 Gb but 8Gb, and SWAP = 1048576 means 4Gb virtual swap..
> so it correspond to your config.
>
> These parameters limits host processes by the same way as containers.
> It means that host processes cannot use more that 12 Gb accounted memory, if they overuse this limit -- OOM-killer will be called to decrease the usage.
>
> I would note -- it isn't default settings, so you can safely remove it.
>
>
>> Of course we could try to change to unlimited:unlimited  for check if it solve the issue.
>> I'll update
>>
>> Regards,
>> Danny
>>
>> -----Original Message-----
>> From: Vasily Averin [mailto:vvs at virtuozzo.com]
>> Sent: Friday, September 29, 2017 12:29 PM
>> To: Danny Gurman <DannyG at Radware.com>; OpenVZ users <users at openvz.org>
>> Cc: Liat Vaknin <LiatV at Radware.com>; Dima Paikin <DimaP at Radware.com>;
>> Lior Komanski <LiorK at Radware.com>; Yohai Liebman <YohaiL at Radware.com>;
>> Rotem Inbar <RotemI at Radware.com>; Alexander Lurye
>> <alexanderl at 3vium.com>; Gal Yahel <GalY at radware.com>
>> Subject: Re: [Users] OOM Killer invoked even that are available RAM
>> and swap on the machine
>>
>> Dear Danny,
>> the following message means that it was not global memory shortage, [
>> 841.805656] 6716 (mysqld) invoked oom-killer in ub 0 generation 0 gfp
>> 0x200d2
>>
>> It is caused by per-ct memory management.| However "ub 0" means that memory shortage was happen not in real container but in so-called VE0, i.e. on host itself.
>> It looks like some misconfiguration for me.
>> I expect you have assigned some strong memory limits for VE0, and OOM-killer was executed when host processes overuse this limit.
>>
>> Please check  do you probably have /etc/vz/conf/0.conf file on affected node?
>>
>> Thank you,
>>       Vasily Averin
>>
>> On 2017-09-29 09:14, Danny Gurman wrote:
>>> Hello Vasily ,
>>>
>>> First - thanks for your ultra-fast response :-)
>>>
>>> Note that for debugging  purpose the container (there is a single
>>> one) was manually stopped (vzctl stop 101)immediately  as kernel started.
>>>
>>> I Just want to clarify that we did not observe global memory shortage on the host (see attached "free" command output and meminfo).
>>> We first notice the issue on 24 RAM and then we increased physical RAM to 32 - exactly same results.
>>> The issue happen always when RAM usage get to 13 GB.
>>>
>>> Regarding forcing kernel for to get memory strongly from specified memory zone - not familiar with those setting.
>>> Is there any command to check this ?
>>>
>>> - Note also that machine was deployed on VMWare.
>>>
>>> I am attaching:
>>> 1. dmesg output.
>>> 2. sysctl -a output
>>> 3. Output of script that run every minute and output "free" and meminfo (the issue reproduced in ~ 10 minutes so it's not long).
>>> 4. Short version of the previous - the state before OOM killer invocation.
>>>
>>> Thanks,
>>> Danny
>>>
>>>
>>>
>>> -----Original Message-----
>>> From: Vasily Averin [mailto:vvs at virtuozzo.com]
>>> Sent: Friday, September 29, 2017 7:46 AM
>>> To: OpenVZ users <users at openvz.org>; Danny Gurman
>>> <DannyG at Radware.com>
>>> Cc: Liat Vaknin <LiatV at Radware.com>; Dima Paikin <DimaP at Radware.com>;
>>> Lior Komanski <LiorK at Radware.com>; Yohai Liebman
>>> <YohaiL at Radware.com>; Rotem Inbar <RotemI at Radware.com>
>>> Subject: Re: [Users] OOM Killer invoked even that are available RAM
>>> and swap on the machine
>>>
>>> Dear Danny,
>>>
>>> 1) Are you sure that it was global memory shortage?
>>>
>>> On Virtuozzo kernels there is per-container OOM-killer, it is executed in case memory shortage inside affected container and it kills tasks related to this container only.
>>> It is normal behaviour,and you should not worry about such cases.
>>>
>>> Please read related kernel messages, it explain the reason of OOM.
>>>
>>> 2) if it was global OOM -- need to clarify in which memory zone it was happen.
>>>
>>> as you probably know memory in linux kernels is divided into few memory zones.
>>> Global OOM can happen when kernel was forced to get memory strongly from specified memory zone.
>>> The memory zone can have no pages that can be moved to swap, and when memory management cannot free memory in this zone by other ways it can call OOM-killer.
>>> This can happen, but it is quite rare case. Need to find how who had submitted such memory request.
>>>
>>> Frankly speaking I think you observe first of described scenarious.
>>>
>>> Thank you,
>>>     Vasily Averin
>>>
>>> On 2017-09-29 02:17, Danny Gurman wrote:
>>>>
>>>>
>>>> Hello all,
>>>>
>>>> I will really appreciate assistance regarding the following issue:
>>>>
>>>>
>>>>
>>>> We have a product deployed on CentosOS 6.9 with OpenVZ.
>>>>
>>>> Machine RAM size – *32* GB + Swap file (16 GB).
>>>>
>>>> The following described issue was reproduced on the OpenVZ kernels:
>>>>
>>>> 2.6.32-042stab125.1 , 2.6.32-042stab123.9 , 2.6.32-042stab123.2,
>>>> 2.6.32-042stab120.11
>>>>
>>>>
>>>>
>>>> For debugging  purpose the single *container* on the host*was stopped*.
>>>>
>>>>
>>>>
>>>> We observed that OOM Killer is invoked whenever host RAM usage without buffers reach ~13 GB:
>>>>
>>>> Free command output (few seconds before OOM killer invoked):
>>>>
>>>> total       used       free     shared    buffers     cached
>>>>
>>>> Mem:      32793080   23197272    9595808        920     135992   10276712
>>>>
>>>>     -/+ buffers/cache:   *12784568*   20008512
>>>>
>>>> Swap:     16465916          *0*   16465916
>>>>
>>>>
>>>>
>>>> On the message buffer, we see that the machine memory state just before the OOM killer invocation is:
>>>>
>>>> RAM: *2097074* / 2097152 [1] SWAP: *1048576* / *1048576* [1] KMEM:
>>>> 227942400 /
>>>>
>>>>
>>>>
>>>> Again , machine has 32 GB, only 13 GB is used, no swap usage – so
>>>> where those wrong values came
>>>>
>>>> from and why OOM killer is invoked in this stage?
>>>>
>>>>
>>>>
>>>> The issue is reproducible with similar results every time.
>>>>
>>>>
>>>>
>>>> Is it a known issue ?
>>>>
>>>> Should I open a defect for this?
>>>>
>>>> Also – is there a  way to totally disable the OOM killer?
>>>>
>>>>
>>>>
>>>> I’ll be glad to provide any required detail (like sysctl –a output)
>>>>
>>>>
>>>>
>>>> Regards,
>>>>
>>>> Danny
>>>>
>>>>
>>>>
>>>> Radware
>>>>
>>>>
>>>>
>>>> Danny Gurman
>>>> Dev Lead, APSolute Vision
>>>> Email: danny.gurman at gmail.com <mailto:danny.gurman at gmail.com>
>>>>
>>>>
>>>>
>>>> T:+972 723917088
>>>> M:+972 526111008
>>>> F:
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Check out the latest and greatest from Radware
>>>> <https://www.radware.com/Resources/CampaignRedirector.html>
>>>>
>>>>
>>>>
>>>> *www.radware.com <https://www.radware.com>*
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Blog <https://blog.radware.com/>
>>>> https://www.radware.com/images/signature/linkedin.jpg
>>>> <https://www.linkedin.com/companies/165642>https://www.radware.com/i
>>>> m a ges/signature/twitter.jpg <file://twitter.com/radware>  youtube
>>>> <https://www.youtube.com/user/radwareinc>
>>>>
>>>> Confidentiality note: This message, and any attachments to it, contains privileged/confidential information of RADWARE Ltd./RADWARE Inc. and may not be disclosed, used, copied, or transmitted in any form or by any means without prior written permission from RADWARE. If you are not the intended recipient, delete the message and any attachments from your system without reading or copying it, and kindly notify the sender by e-mail. Thank you.
>>>>
>>>> *P****Please consider your environmental responsibility before
>>>> printing this e-mail***
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Users mailing list
>>>> Users at openvz.org
>>>> https://lists.openvz.org/mailman/listinfo/users
>>>
>
> _______________________________________________
> Users mailing list
> Users at openvz.org
> https://lists.openvz.org/mailman/listinfo/users



-- 
Best regards,
[COOLCOLD-RIPN]



More information about the Users mailing list