[Users] OOM and /etc/vz/oom-groups.conf

[Kir Kolyshkin]

On 10/01/2013 03:54 PM, Mark J. wrote:
> I knew I was missing something in there, kernel is:
>
> 2.6.32-042stab068.8
>
> We also use ksplice, so its currently patched to:
>
> Effective kernel version is 2.6.32-042stab081.3

I'd suggest you to reproduce the bug on 081.3 kernel booted for real,
and if you succeed, file a kernel bug to bugzilla.openvz.org.

Also, it would be helpful to see output of pstree or ps axf inside a 
container.
Note that you have specified oom adjustment for mysqld processes
which parent is init. I guess that OOM killer kills a mysqld process
which parent is not init, so oom adj is not applied.

Generally speaking, I think you are doing something wrong here.

First, mysqld is a service which controls its own memory consumption
pretty well, you just need to set a few parameters in its /etc/my.cnf,
and that would be the preferred way of managing it.

Second, OOM should not happen in general. In case you are using
vswap, the sum of ram and swap configured for all containers
should not exceed real ram and swap on the system. Worst case
scenario is swap will be used heavily, resulting in a slowdown
(which is still way better than killing processes). In case you are
not using VSwap -- well, you should, see http://openvz.org/VSwap.

Kir.

>
> On Tue, Oct 01, 2013 at 03:25:58PM -0700, Roman Haefeli wrote:
>> It's strongly advised to report the exact kernel version along with a
>> problem report. It's not unlikely your problem is specific to the kernel
>> version, especially if you use an OpenVZ kernel from Debian Squeeze.
>>
>> Roman
>>
>> On Tue, 2013-10-01 at 14:45 -0400, Mark J. wrote:
>>> Hello,
>>>
>>> I have been trying to tweak the OOM killer to prevent some database corruption thats been happening in the VM's when the oom killer kills off mysql. Based on what i have read online and in the conf file itself I am a bit confused on why its not killing other things off before mysql.
>>>
>>> Based on the following (taken directly form the conf file):
>>>
>>> OOM-score adjustment, a number from -1000 to 1000. A positive number increases the chances of killing, while a negative number decreases it (more protection). -1000 makes a process unkillable till any other process exists.
>>>
>>> I made the following changes to the conf file:
>>>
>>> mysqld          init            -500    -1000
>>> httpd           init            -500    100
>>> httpsd          init            -500    100
>>> apache          init            -500    100
>>> php             init            -500    100
>>>
>>>
>>> Saved and then made the changes live:
>>>
>>> cat /etc/vz/oom-groups.conf > /proc/vz/oom_score_adj
>>>
>>>
>>> Verified the changes were in fact live:
>>>
>>> root ~ #cat /proc/vz/oom_score_adj | grep mysql
>>> mysqld init -500 -1000
>>> root ~ #cat /proc/vz/oom_score_adj | grep http
>>> httpd init -500 100
>>> httpsd init -500 100
>>>
>>>
>>> However I am still watching mysql get killed off before anything else:
>>>
>>> Oct  1 14:38:18 server kernel: [2456651.082410] >>> 3679 oom generation 62 starts
>>> Oct  1 14:38:18 server kernel: [2456651.082418] 133199 (mysqld) invoked loc oom-killer: gfp 0x200da order 0 oomkilladj=0
>>> Oct  1 14:38:18 server kernel: [2456651.084483] Out of memory in UB: Kill process 126607 (mysqld) score -903 or sacrifice child
>>> Oct  1 14:38:18 server kernel: [2456651.084772] OOM killed process 126607 (mysqld) vm:3829812kB, rss:166344kB, swap:36760kB
>>> Oct  1 14:38:18 server kernel: [2456651.115460] OOM killed process mysqld (pid=157305, ve=3679) exited, free=519316.
>>> Oct  1 14:38:18 server kernel: [2456651.115465] <<< 3679 oom generation 62 ends (task died)
>>>
>>> Anyone have any suggestions or been able to make these adjustments that worked? Any help would be of great assistance.
>>> _______________________________________________
>>> Users mailing list
>>> Users at openvz.org
>>> https://lists.openvz.org/mailman/listinfo/users
>>
>> _______________________________________________
>> Users mailing list
>> Users at openvz.org
>> https://lists.openvz.org/mailman/listinfo/users
> _______________________________________________
> Users mailing list
> Users at openvz.org
> https://lists.openvz.org/mailman/listinfo/users