[Users] OOM Killer invoked even that are available RAM and swap on the machine

Fri Sep 29 09:14:27 MSK 2017

Hello Vasily ,

First - thanks for your ultra-fast response :-)

Note that for debugging  purpose the container (there is a single one) was manually 
stopped (vzctl stop 101)immediately  as kernel started.

I Just want to clarify that we did not observe global memory shortage on the host (see attached "free" command output and meminfo).
We first notice the issue on 24 RAM and then we increased physical RAM to 32 - exactly same results.
The issue happen always when RAM usage get to 13 GB.

Regarding forcing kernel for to get memory strongly from specified memory zone - not familiar with those setting.
Is there any command to check this ?

- Note also that machine was deployed on VMWare.

I am attaching:
1. dmesg output.
2. sysctl -a output
3. Output of script that run every minute and output "free" and meminfo (the issue reproduced in ~ 10 minutes so it's not long).
4. Short version of the previous - the state before OOM killer invocation.

Thanks,
Danny 

-----Original Message-----
From: Vasily Averin [mailto:vvs at virtuozzo.com] 
Sent: Friday, September 29, 2017 7:46 AM
To: OpenVZ users <users at openvz.org>; Danny Gurman <DannyG at Radware.com>
Cc: Liat Vaknin <LiatV at Radware.com>; Dima Paikin <DimaP at Radware.com>; Lior Komanski <LiorK at Radware.com>; Yohai Liebman <YohaiL at Radware.com>; Rotem Inbar <RotemI at Radware.com>
Subject: Re: [Users] OOM Killer invoked even that are available RAM and swap on the machine

Dear Danny,

1) Are you sure that it was global memory shortage?

On Virtuozzo kernels there is per-container OOM-killer, it is executed in case memory shortage inside affected container and it kills tasks related to this container only.
It is normal behaviour,and you should not worry about such cases.

Please read related kernel messages, it explain the reason of OOM.

2) if it was global OOM -- need to clarify in which memory zone it was happen.

as you probably know memory in linux kernels is divided into few memory zones.
Global OOM can happen when kernel was forced to get memory strongly from specified memory zone.
The memory zone can have no pages that can be moved to swap, and when memory management cannot free memory in this zone by other ways it can call OOM-killer.
This can happen, but it is quite rare case. Need to find how who had submitted such memory request.

Frankly speaking I think you observe first of described scenarious.

Thank you,
    Vasily Averin

On 2017-09-29 02:17, Danny Gurman wrote:
>
>
> Hello all,
>
> I will really appreciate assistance regarding the following issue:
>
>  
>
> We have a product deployed on CentosOS 6.9 with OpenVZ.
>
> Machine RAM size – *32* GB + Swap file (16 GB).
>
> The following described issue was reproduced on the OpenVZ kernels:
>
> 2.6.32-042stab125.1 , 2.6.32-042stab123.9 , 2.6.32-042stab123.2, 
> 2.6.32-042stab120.11
>
>  
>
> For debugging  purpose the single *container* on the host*was stopped*.
>
>  
>
> We observed that OOM Killer is invoked whenever host RAM usage without buffers reach ~13 GB:
>
> Free command output (few seconds before OOM killer invoked):
>
> total       used       free     shared    buffers     cached
>
> Mem:      32793080   23197272    9595808        920     135992   10276712
>
>     -/+ buffers/cache:   *12784568*   20008512
>
> Swap:     16465916          *0*   16465916
>
>  
>
> On the message buffer, we see that the machine memory state just before the OOM killer invocation is:
>
> RAM: *2097074* / 2097152 [1] SWAP: *1048576* / *1048576* [1] KMEM: 
> 227942400 /
>
>  
>
> Again , machine has 32 GB, only 13 GB is used, no swap usage – so 
> where those wrong values came
>
> from and why OOM killer is invoked in this stage?
>
>  
>
> The issue is reproducible with similar results every time.
>
>  
>
> Is it a known issue ?
>
> Should I open a defect for this?
>
> Also – is there a  way to totally disable the OOM killer?
>
>  
>
> I’ll be glad to provide any required detail (like sysctl –a output)
>
>  
>
> Regards,
>
> Danny
>
>  
>
> Radware
>
> 	
>
> Danny Gurman
> Dev Lead, APSolute Vision
> Email: danny.gurman at gmail.com <mailto:danny.gurman at gmail.com>
>
> 	
>
> T:+972 723917088
> M:+972 526111008
> F:
>
>
> 	
> 	
>
> Check out the latest and greatest from Radware 
> <https://www.radware.com/Resources/CampaignRedirector.html>
>
>  
>
> *www.radware.com <https://www.radware.com>*
>
> 	
>
>  
>
> 	
>
> Blog <https://blog.radware.com/>  
> https://www.radware.com/images/signature/linkedin.jpg 
> <https://www.linkedin.com/companies/165642>https://www.radware.com/ima
> ges/signature/twitter.jpg <file://twitter.com/radware>  youtube 
> <https://www.youtube.com/user/radwareinc>
>
> Confidentiality note: This message, and any attachments to it, contains privileged/confidential information of RADWARE Ltd./RADWARE Inc. and may not be disclosed, used, copied, or transmitted in any form or by any means without prior written permission from RADWARE. If you are not the intended recipient, delete the message and any attachments from your system without reading or copying it, and kindly notify the sender by e-mail. Thank you.
>
> *P****Please consider your environmental responsibility before 
> printing this e-mail***
>
>  
>
>
>
> _______________________________________________
> Users mailing list
> Users at openvz.org
> https://lists.openvz.org/mailman/listinfo/users

-------------- next part --------------
A non-text attachment was scrubbed...
Name: dmesg.log
Type: application/octet-stream
Size: 106683 bytes
Desc: dmesg.log
URL: <http://lists.openvz.org/pipermail/users/attachments/20170929/7de97766/attachment-0003.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: oom_debug_before_invoke.log
Type: application/octet-stream
Size: 3114 bytes
Desc: oom_debug_before_invoke.log
URL: <http://lists.openvz.org/pipermail/users/attachments/20170929/7de97766/attachment-0004.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: oom_debug_full.log
Type: application/octet-stream
Size: 42904 bytes
Desc: oom_debug_full.log
URL: <http://lists.openvz.org/pipermail/users/attachments/20170929/7de97766/attachment-0005.obj>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: sysctl.out.txt
URL: <http://lists.openvz.org/pipermail/users/attachments/20170929/7de97766/attachment-0001.txt>