[Users] OOM Killer invoked even that are available RAM and swap on the machine
Vasily Averin
vvs at virtuozzo.com
Fri Sep 29 13:14:02 MSK 2017
On 2017-09-29 12:29, Vasily Averin wrote:
> Dear Danny,
> the following message means that it was not global memory shortage,
> [ 841.805656] 6716 (mysqld) invoked oom-killer in ub 0 generation 0 gfp 0x200d2
>
> It is caused by per-ct memory management.|
> However "ub 0" means that memory shortage was happen not in real container but in so-called VE0, i.e. on host itself.
> It looks like some misconfiguration for me.
> I expect you have assigned some strong memory limits for VE0,
> and OOM-killer was executed when host processes overuse this limit.
>
> Please check do you probably have /etc/vz/conf/0.conf file on affected node?
the following message shows current useage and assigned limits
[ 841.806102] RAM: 2097074 / 2097152 [1] SWAP: 1048576 / 1048576 [1] KMEM: 227942400 / inf [0] DCSZ: 137965068 / inf [0] OOMG: 1812712 / inf [0] Dirty 44005 Wback 260 Dche 139729 Prnd 0
RAM -- PHYSPAGES
SWAP -- SWAPPAGES
KMEM -- KMEMSIZE
DCSZ -- DCACHESIZE
OOMG -- OOMGUARPAGES
as we can see RAM is assigned to 2 Gb SWAP -- into 1Gb, and usage counters are near to limits
(SWAP here is not real swap on host but virtual swap for VE0, i.e. for all host processes)
The problem was caused by these assigned limits.
Kernel protects containers from memory overuse on host and kills host processes.
> Thank you,
> Vasily Averin
>
> On 2017-09-29 09:14, Danny Gurman wrote:
>> Hello Vasily ,
>>
>> First - thanks for your ultra-fast response :-)
>>
>> Note that for debugging purpose the container (there is a single one) was manually
>> stopped (vzctl stop 101)immediately as kernel started.
>>
>> I Just want to clarify that we did not observe global memory shortage on the host (see attached "free" command output and meminfo).
>> We first notice the issue on 24 RAM and then we increased physical RAM to 32 - exactly same results.
>> The issue happen always when RAM usage get to 13 GB.
>>
>> Regarding forcing kernel for to get memory strongly from specified memory zone - not familiar with those setting.
>> Is there any command to check this ?
>>
>> - Note also that machine was deployed on VMWare.
>>
>> I am attaching:
>> 1. dmesg output.
>> 2. sysctl -a output
>> 3. Output of script that run every minute and output "free" and meminfo (the issue reproduced in ~ 10 minutes so it's not long).
>> 4. Short version of the previous - the state before OOM killer invocation.
>>
>> Thanks,
>> Danny
>>
>>
>>
>> -----Original Message-----
>> From: Vasily Averin [mailto:vvs at virtuozzo.com]
>> Sent: Friday, September 29, 2017 7:46 AM
>> To: OpenVZ users <users at openvz.org>; Danny Gurman <DannyG at Radware.com>
>> Cc: Liat Vaknin <LiatV at Radware.com>; Dima Paikin <DimaP at Radware.com>; Lior Komanski <LiorK at Radware.com>; Yohai Liebman <YohaiL at Radware.com>; Rotem Inbar <RotemI at Radware.com>
>> Subject: Re: [Users] OOM Killer invoked even that are available RAM and swap on the machine
>>
>> Dear Danny,
>>
>> 1) Are you sure that it was global memory shortage?
>>
>> On Virtuozzo kernels there is per-container OOM-killer, it is executed in case memory shortage inside affected container and it kills tasks related to this container only.
>> It is normal behaviour,and you should not worry about such cases.
>>
>> Please read related kernel messages, it explain the reason of OOM.
>>
>> 2) if it was global OOM -- need to clarify in which memory zone it was happen.
>>
>> as you probably know memory in linux kernels is divided into few memory zones.
>> Global OOM can happen when kernel was forced to get memory strongly from specified memory zone.
>> The memory zone can have no pages that can be moved to swap, and when memory management cannot free memory in this zone by other ways it can call OOM-killer.
>> This can happen, but it is quite rare case. Need to find how who had submitted such memory request.
>>
>> Frankly speaking I think you observe first of described scenarious.
>>
>> Thank you,
>> Vasily Averin
>>
>> On 2017-09-29 02:17, Danny Gurman wrote:
>>>
>>>
>>> Hello all,
>>>
>>> I will really appreciate assistance regarding the following issue:
>>>
>>>
>>>
>>> We have a product deployed on CentosOS 6.9 with OpenVZ.
>>>
>>> Machine RAM size – *32* GB + Swap file (16 GB).
>>>
>>> The following described issue was reproduced on the OpenVZ kernels:
>>>
>>> 2.6.32-042stab125.1 , 2.6.32-042stab123.9 , 2.6.32-042stab123.2,
>>> 2.6.32-042stab120.11
>>>
>>>
>>>
>>> For debugging purpose the single *container* on the host*was stopped*.
>>>
>>>
>>>
>>> We observed that OOM Killer is invoked whenever host RAM usage without buffers reach ~13 GB:
>>>
>>> Free command output (few seconds before OOM killer invoked):
>>>
>>> total used free shared buffers cached
>>>
>>> Mem: 32793080 23197272 9595808 920 135992 10276712
>>>
>>> -/+ buffers/cache: *12784568* 20008512
>>>
>>> Swap: 16465916 *0* 16465916
>>>
>>>
>>>
>>> On the message buffer, we see that the machine memory state just before the OOM killer invocation is:
>>>
>>> RAM: *2097074* / 2097152 [1] SWAP: *1048576* / *1048576* [1] KMEM:
>>> 227942400 /
>>>
>>>
>>>
>>> Again , machine has 32 GB, only 13 GB is used, no swap usage – so
>>> where those wrong values came
>>>
>>> from and why OOM killer is invoked in this stage?
>>>
>>>
>>>
>>> The issue is reproducible with similar results every time.
>>>
>>>
>>>
>>> Is it a known issue ?
>>>
>>> Should I open a defect for this?
>>>
>>> Also – is there a way to totally disable the OOM killer?
>>>
>>>
>>>
>>> I’ll be glad to provide any required detail (like sysctl –a output)
>>>
>>>
>>>
>>> Regards,
>>>
>>> Danny
>>>
>>>
>>>
>>> Radware
>>>
>>>
>>>
>>> Danny Gurman
>>> Dev Lead, APSolute Vision
>>> Email: danny.gurman at gmail.com <mailto:danny.gurman at gmail.com>
>>>
>>>
>>>
>>> T:+972 723917088
>>> M:+972 526111008
>>> F:
>>>
>>>
>>>
>>>
>>>
>>> Check out the latest and greatest from Radware
>>> <https://www.radware.com/Resources/CampaignRedirector.html>
>>>
>>>
>>>
>>> *www.radware.com <https://www.radware.com>*
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> Blog <https://blog.radware.com/>
>>> https://www.radware.com/images/signature/linkedin.jpg
>>> <https://www.linkedin.com/companies/165642>https://www.radware.com/ima
>>> ges/signature/twitter.jpg <file://twitter.com/radware> youtube
>>> <https://www.youtube.com/user/radwareinc>
>>>
>>> Confidentiality note: This message, and any attachments to it, contains privileged/confidential information of RADWARE Ltd./RADWARE Inc. and may not be disclosed, used, copied, or transmitted in any form or by any means without prior written permission from RADWARE. If you are not the intended recipient, delete the message and any attachments from your system without reading or copying it, and kindly notify the sender by e-mail. Thank you.
>>>
>>> *P****Please consider your environmental responsibility before
>>> printing this e-mail***
>>>
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> Users mailing list
>>> Users at openvz.org
>>> https://lists.openvz.org/mailman/listinfo/users
>>
More information about the Users
mailing list