[Users] OOM Killer invoked even that are available RAM and swap on the machine
Danny Gurman
DannyG at Radware.com
Fri Sep 29 13:32:19 MSK 2017
Thanks again Vasily,
Here are the values from /etc/vz/conf/0.conf:
# UBC parameters (in form of barrier:limit)
OOMGUARPAGES="unlimited:unlimited"
CPUS="4"
CPUMASK="0-3"
#CPUUNITS="400000"
PHYSPAGES="0:8G"
SWAPPAGES="0:4G"
So , what the physical pages limitation on VE0 (host) actually mean ?
Host is limited to 8 GB RAM?
Note that we reach ~ 13 GB.
Of course we could try to change to unlimited:unlimited for check if it solve the issue.
I'll update
Regards,
Danny
-----Original Message-----
From: Vasily Averin [mailto:vvs at virtuozzo.com]
Sent: Friday, September 29, 2017 12:29 PM
To: Danny Gurman <DannyG at Radware.com>; OpenVZ users <users at openvz.org>
Cc: Liat Vaknin <LiatV at Radware.com>; Dima Paikin <DimaP at Radware.com>; Lior Komanski <LiorK at Radware.com>; Yohai Liebman <YohaiL at Radware.com>; Rotem Inbar <RotemI at Radware.com>; Alexander Lurye <alexanderl at 3vium.com>; Gal Yahel <GalY at radware.com>
Subject: Re: [Users] OOM Killer invoked even that are available RAM and swap on the machine
Dear Danny,
the following message means that it was not global memory shortage, [ 841.805656] 6716 (mysqld) invoked oom-killer in ub 0 generation 0 gfp 0x200d2
It is caused by per-ct memory management.| However "ub 0" means that memory shortage was happen not in real container but in so-called VE0, i.e. on host itself.
It looks like some misconfiguration for me.
I expect you have assigned some strong memory limits for VE0, and OOM-killer was executed when host processes overuse this limit.
Please check do you probably have /etc/vz/conf/0.conf file on affected node?
Thank you,
Vasily Averin
On 2017-09-29 09:14, Danny Gurman wrote:
> Hello Vasily ,
>
> First - thanks for your ultra-fast response :-)
>
> Note that for debugging purpose the container (there is a single one)
> was manually stopped (vzctl stop 101)immediately as kernel started.
>
> I Just want to clarify that we did not observe global memory shortage on the host (see attached "free" command output and meminfo).
> We first notice the issue on 24 RAM and then we increased physical RAM to 32 - exactly same results.
> The issue happen always when RAM usage get to 13 GB.
>
> Regarding forcing kernel for to get memory strongly from specified memory zone - not familiar with those setting.
> Is there any command to check this ?
>
> - Note also that machine was deployed on VMWare.
>
> I am attaching:
> 1. dmesg output.
> 2. sysctl -a output
> 3. Output of script that run every minute and output "free" and meminfo (the issue reproduced in ~ 10 minutes so it's not long).
> 4. Short version of the previous - the state before OOM killer invocation.
>
> Thanks,
> Danny
>
>
>
> -----Original Message-----
> From: Vasily Averin [mailto:vvs at virtuozzo.com]
> Sent: Friday, September 29, 2017 7:46 AM
> To: OpenVZ users <users at openvz.org>; Danny Gurman <DannyG at Radware.com>
> Cc: Liat Vaknin <LiatV at Radware.com>; Dima Paikin <DimaP at Radware.com>;
> Lior Komanski <LiorK at Radware.com>; Yohai Liebman <YohaiL at Radware.com>;
> Rotem Inbar <RotemI at Radware.com>
> Subject: Re: [Users] OOM Killer invoked even that are available RAM
> and swap on the machine
>
> Dear Danny,
>
> 1) Are you sure that it was global memory shortage?
>
> On Virtuozzo kernels there is per-container OOM-killer, it is executed in case memory shortage inside affected container and it kills tasks related to this container only.
> It is normal behaviour,and you should not worry about such cases.
>
> Please read related kernel messages, it explain the reason of OOM.
>
> 2) if it was global OOM -- need to clarify in which memory zone it was happen.
>
> as you probably know memory in linux kernels is divided into few memory zones.
> Global OOM can happen when kernel was forced to get memory strongly from specified memory zone.
> The memory zone can have no pages that can be moved to swap, and when memory management cannot free memory in this zone by other ways it can call OOM-killer.
> This can happen, but it is quite rare case. Need to find how who had submitted such memory request.
>
> Frankly speaking I think you observe first of described scenarious.
>
> Thank you,
> Vasily Averin
>
> On 2017-09-29 02:17, Danny Gurman wrote:
>>
>>
>> Hello all,
>>
>> I will really appreciate assistance regarding the following issue:
>>
>>
>>
>> We have a product deployed on CentosOS 6.9 with OpenVZ.
>>
>> Machine RAM size – *32* GB + Swap file (16 GB).
>>
>> The following described issue was reproduced on the OpenVZ kernels:
>>
>> 2.6.32-042stab125.1 , 2.6.32-042stab123.9 , 2.6.32-042stab123.2,
>> 2.6.32-042stab120.11
>>
>>
>>
>> For debugging purpose the single *container* on the host*was stopped*.
>>
>>
>>
>> We observed that OOM Killer is invoked whenever host RAM usage without buffers reach ~13 GB:
>>
>> Free command output (few seconds before OOM killer invoked):
>>
>> total used free shared buffers cached
>>
>> Mem: 32793080 23197272 9595808 920 135992 10276712
>>
>> -/+ buffers/cache: *12784568* 20008512
>>
>> Swap: 16465916 *0* 16465916
>>
>>
>>
>> On the message buffer, we see that the machine memory state just before the OOM killer invocation is:
>>
>> RAM: *2097074* / 2097152 [1] SWAP: *1048576* / *1048576* [1] KMEM:
>> 227942400 /
>>
>>
>>
>> Again , machine has 32 GB, only 13 GB is used, no swap usage – so
>> where those wrong values came
>>
>> from and why OOM killer is invoked in this stage?
>>
>>
>>
>> The issue is reproducible with similar results every time.
>>
>>
>>
>> Is it a known issue ?
>>
>> Should I open a defect for this?
>>
>> Also – is there a way to totally disable the OOM killer?
>>
>>
>>
>> I’ll be glad to provide any required detail (like sysctl –a output)
>>
>>
>>
>> Regards,
>>
>> Danny
>>
>>
>>
>> Radware
>>
>>
>>
>> Danny Gurman
>> Dev Lead, APSolute Vision
>> Email: danny.gurman at gmail.com <mailto:danny.gurman at gmail.com>
>>
>>
>>
>> T:+972 723917088
>> M:+972 526111008
>> F:
>>
>>
>>
>>
>>
>> Check out the latest and greatest from Radware
>> <https://www.radware.com/Resources/CampaignRedirector.html>
>>
>>
>>
>> *www.radware.com <https://www.radware.com>*
>>
>>
>>
>>
>>
>>
>>
>> Blog <https://blog.radware.com/>
>> https://www.radware.com/images/signature/linkedin.jpg
>> <https://www.linkedin.com/companies/165642>https://www.radware.com/im
>> a ges/signature/twitter.jpg <file://twitter.com/radware> youtube
>> <https://www.youtube.com/user/radwareinc>
>>
>> Confidentiality note: This message, and any attachments to it, contains privileged/confidential information of RADWARE Ltd./RADWARE Inc. and may not be disclosed, used, copied, or transmitted in any form or by any means without prior written permission from RADWARE. If you are not the intended recipient, delete the message and any attachments from your system without reading or copying it, and kindly notify the sender by e-mail. Thank you.
>>
>> *P****Please consider your environmental responsibility before
>> printing this e-mail***
>>
>>
>>
>>
>>
>> _______________________________________________
>> Users mailing list
>> Users at openvz.org
>> https://lists.openvz.org/mailman/listinfo/users
>
More information about the Users
mailing list