<div dir="ltr"><div>/sys/fs/cgroup/memory was checked and not show this groups - <br></div><div><br></div><div> find /sys/fs/cgroup/memory -type d | wc -l<br>2</div><div><br></div><div>grep -E "memory|num" /proc/cgroups <br>#subsys_name hierarchy num_cgroups enabled<br>memory 2 63744 1</div><div><br></div><div>And more strange - uptime was only about 3 days when the problem reproduced.<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, 12 Feb 2021 at 04:06, Kir Kolyshkin <<a href="mailto:kir@openvz.org">kir@openvz.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On Thu, Feb 11, 2021 at 12:59 AM Сергей Мамонов <<a href="mailto:mrqwer88@gmail.com" target="_blank">mrqwer88@gmail.com</a>> wrote:<br>
><br>
> And after migrate all containers to another node it still shows 63745 cgroups -<br>
><br>
> cat /proc/cgroups<br>
> #subsys_name hierarchy num_cgroups enabled<br>
> cpuset 7 2 1<br>
> cpu 10 2 1<br>
> cpuacct 10 2 1<br>
> memory 2 63745 1<br>
<br>
Looks like a leakage (or a bug in memory accounting which prevents<br>
cgroup from being released).<br>
You can check the number of memory cgroups with something like<br>
<br>
find /sys/fs/cgroup/memory -type d | wc -l<br>
<br>
If you see a large number, go explore those cgroups (check<br>
cgroup.procs, memory.usage_in_bytes).<br>
<br>
> devices 11 2 1<br>
> freezer 17 2 1<br>
> net_cls 12 2 1<br>
> blkio 1 4 1<br>
> perf_event 13 2 1<br>
> hugetlb 14 2 1<br>
> pids 3 68 1<br>
> ve 6 1 1<br>
> beancounter 4 3 1<br>
> net_prio 12 2 1<br>
><br>
> On Wed, 10 Feb 2021 at 18:47, Сергей Мамонов <<a href="mailto:mrqwer88@gmail.com" target="_blank">mrqwer88@gmail.com</a>> wrote:<br>
>><br>
>> And it is definitely it -<br>
>> grep -E "memory|num_cgroups" /proc/cgroups<br>
>> #subsys_name hierarchy num_cgroups enabled<br>
>> memory 2 65534 1<br>
>><br>
>> After migration some of containers to another node num_cgroups goes down to 65365 and it allowed to start stopped container without `<br>
>> Can't create directory /sys/fs/cgroup/memory/machine.slice/1000133882: Cannot allocate memory` error.<br>
>><br>
>> But I don't understand why num_cgroups for memory so big, yet.<br>
>><br>
>> Like ~460 per container instead of 60 and less per container on other nodes (with the same kernel version too).<br>
>><br>
>> On Wed, 10 Feb 2021 at 17:48, Сергей Мамонов <<a href="mailto:mrqwer88@gmail.com" target="_blank">mrqwer88@gmail.com</a>> wrote:<br>
>>><br>
>>> Hello!<br>
>>><br>
>>> Looks like we reproduced this problem too.<br>
>>><br>
>>> kernel - 3.10.0-1127.18.2.vz7.163.46<br>
>>><br>
>>> Same error -<br>
>>> Can't create directory /sys/fs/cgroup/memory/machine.slice/1000133882: Cannot allocate memory<br>
>>><br>
>>> Same ok output for<br>
>>> /sys/fs/cgroup/memory/*limit_in_bytes<br>
>>> /sys/fs/cgroup/memory/machine.slice/*limit_in_bytes<br>
>>><br>
>>> Have a lot of free memory on node (per numa too).<br>
>>><br>
>>> Only that looks really strange -<br>
>>> grep -E "memory|num_cgroups" /proc/cgroups<br>
>>> #subsys_name hierarchy num_cgroups enabled<br>
>>> memory 2 65534 1<br>
>>><br>
>>> huge nub_cgroups only on this node<br>
>>><br>
>>> cat /proc/cgroups<br>
>>> #subsys_name hierarchy num_cgroups enabled<br>
>>> cpuset 7 144 1<br>
>>> cpu 10 263 1<br>
>>> cpuacct 10 263 1<br>
>>> memory 2 65534 1<br>
>>> devices 11 1787 1<br>
>>> freezer 17 144 1<br>
>>> net_cls 12 144 1<br>
>>> blkio 1 257 1<br>
>>> perf_event 13 144 1<br>
>>> hugetlb 14 144 1<br>
>>> pids 3 2955 1<br>
>>> ve 6 143 1<br>
>>> beancounter 4 143 1<br>
>>> net_prio 12 144 1<br>
>>><br>
>>> On Thu, 28 Jan 2021 at 14:22, Konstantin Khorenko <<a href="mailto:khorenko@virtuozzo.com" target="_blank">khorenko@virtuozzo.com</a>> wrote:<br>
>>>><br>
>>>> May be you hit memory shortage in a particular NUMA node only, for example.<br>
>>>><br>
>>>> # numactl --hardware<br>
>>>> # numastat -m<br>
>>>><br>
>>>><br>
>>>> Or go hard way - trace kernel where exactly do we get -ENOMEM:<br>
>>>><br>
>>>> trace the kernel function cgroup_mkdir() using /sys/kernel/debug/tracing/<br>
>>>> with function_graph tracer.<br>
>>>><br>
>>>><br>
>>>> <a href="https://lwn.net/Articles/370423/" rel="noreferrer" target="_blank">https://lwn.net/Articles/370423/</a><br>
>>>><br>
>>>> --<br>
>>>> Best regards,<br>
>>>><br>
>>>> Konstantin Khorenko,<br>
>>>> Virtuozzo Linux Kernel Team<br>
>>>><br>
>>>> On 01/28/2021 12:43 PM, Joe Dougherty wrote:<br>
>>>><br>
>>>> I checked that, doesn't appear to be the case.<br>
>>>><br>
>>>> # pwd<br>
>>>> /sys/fs/cgroup/memory<br>
>>>> # cat *limit_in_bytes<br>
>>>> 9223372036854771712<br>
>>>> 9223372036854767616<br>
>>>> 2251799813685247<br>
>>>> 2251799813685247<br>
>>>> 9223372036854771712<br>
>>>> 9223372036854771712<br>
>>>> 9223372036854771712<br>
>>>> # cat *failcnt<br>
>>>> 0<br>
>>>> 0<br>
>>>> 0<br>
>>>> 0<br>
>>>> 0<br>
>>>><br>
>>>> # pwd<br>
>>>> /sys/fs/cgroup/memory/machine.slice<br>
>>>> # cat *limit_in_bytes<br>
>>>> 9223372036854771712<br>
>>>> 9223372036854767616<br>
>>>> 9223372036854771712<br>
>>>> 9223372036854771712<br>
>>>> 9223372036854771712<br>
>>>> 9223372036854771712<br>
>>>> 9223372036854771712<br>
>>>> # cat *failcnt<br>
>>>> 0<br>
>>>> 0<br>
>>>> 0<br>
>>>> 0<br>
>>>> 0<br>
>>>><br>
>>>><br>
>>>><br>
>>>> On Thu, Jan 28, 2021 at 2:47 AM Konstantin Khorenko <<a href="mailto:khorenko@virtuozzo.com" target="_blank">khorenko@virtuozzo.com</a>> wrote:<br>
>>>>><br>
>>>>> Hi Joe,<br>
>>>>><br>
>>>>> i'd suggest to check memory limits for root and "machine.slice" memory cgroups<br>
>>>>><br>
>>>>> /sys/fs/cgroup/memory/*limit_in_bytes<br>
>>>>> /sys/fs/cgroup/memory/machine.slice/*limit_in_bytes<br>
>>>>><br>
>>>>> All of them should be unlimited.<br>
>>>>><br>
>>>>> If not - search who limit them.<br>
>>>>><br>
>>>>> --<br>
>>>>> Best regards,<br>
>>>>><br>
>>>>> Konstantin Khorenko,<br>
>>>>> Virtuozzo Linux Kernel Team<br>
>>>>><br>
>>>>> On 01/27/2021 10:28 PM, Joe Dougherty wrote:<br>
>>>>><br>
>>>>> I'm running into an issue on only 1 of my OpenVZ 7 nodes where it's unable to create a directory on /sys/fs/cgroup/memory/machine.slice due to "Cannot allocate memory" whenever I try to start a new container or restart and existing one. I've been trying to research this but I'm unable to find any concrete info on what could cause this. It appears to be memory related because sometimes if I issue "echo 1 /proc/sys/vm/drop_caches" it allows me to start a container (this only works sometimes) but my RAM usage is extremely low with no swapping (swappiness even set to 0 for testing). Thank you in advance for your help.<br>
>>>>><br>
>>>>><br>
>>>>> Example:<br>
>>>>> # vzctl start 9499<br>
>>>>> Starting Container ...<br>
>>>>> Mount image: /vz/private/9499/root.hdd<br>
>>>>> Container is mounted<br>
>>>>> Can't create directory /sys/fs/cgroup/memory/machine.slice/9499: Cannot allocate memory<br>
>>>>> Unmount image: /vz/private/9499/root.hdd (190)<br>
>>>>> Container is unmounted<br>
>>>>> Failed to start the Container<br>
>>>>><br>
>>>>><br>
>>>>> Node Info:<br>
>>>>> Uptime: 10 days<br>
>>>>> OS: Virtuozzo 7.0.15<br>
>>>>> Kernel: 3.10.0-1127.18.2.vz7.163.46 GNU/Linux<br>
>>>>> System Load: 3.1<br>
>>>>> /vz Usage: 56% of 37T<br>
>>>>> Swap Usage: 0%<br>
>>>>> RAM Free: 84% of 94.2GB<br>
>>>>><br>
>>>>> # free -m<br>
>>>>> total used free shared buff/cache available<br>
>>>>> Mem: 96502 14259 49940 413 32303 80990<br>
>>>>> Swap: 32767 93 32674<br>
>>>>><br>
>>>>><br>
>>>>> _______________________________________________<br>
>>>>> Users mailing list<br>
>>>>> <a href="mailto:Users@openvz.org" target="_blank">Users@openvz.org</a><br>
>>>>> <a href="https://lists.openvz.org/mailman/listinfo/users" rel="noreferrer" target="_blank">https://lists.openvz.org/mailman/listinfo/users</a><br>
>>>>><br>
>>>>><br>
>>>>> _______________________________________________<br>
>>>>> Users mailing list<br>
>>>>> <a href="mailto:Users@openvz.org" target="_blank">Users@openvz.org</a><br>
>>>>> <a href="https://lists.openvz.org/mailman/listinfo/users" rel="noreferrer" target="_blank">https://lists.openvz.org/mailman/listinfo/users</a><br>
>>>><br>
>>>><br>
>>>><br>
>>>> --<br>
>>>> -Joe Dougherty<br>
>>>> Chief Operating Officer<br>
>>>> Secure Dragon LLC<br>
>>>> <a href="http://www.SecureDragon.net" rel="noreferrer" target="_blank">www.SecureDragon.net</a><br>
>>>><br>
>>>><br>
>>>> _______________________________________________<br>
>>>> Users mailing list<br>
>>>> <a href="mailto:Users@openvz.org" target="_blank">Users@openvz.org</a><br>
>>>> <a href="https://lists.openvz.org/mailman/listinfo/users" rel="noreferrer" target="_blank">https://lists.openvz.org/mailman/listinfo/users</a><br>
>>>><br>
>>>><br>
>>>> _______________________________________________<br>
>>>> Users mailing list<br>
>>>> <a href="mailto:Users@openvz.org" target="_blank">Users@openvz.org</a><br>
>>>> <a href="https://lists.openvz.org/mailman/listinfo/users" rel="noreferrer" target="_blank">https://lists.openvz.org/mailman/listinfo/users</a><br>
>>><br>
>>><br>
>>><br>
>>> --<br>
>>> Best Regards,<br>
>>> Sergei Mamonov<br>
>><br>
>><br>
>><br>
>> --<br>
>> Best Regards,<br>
>> Sergei Mamonov<br>
><br>
><br>
><br>
> --<br>
> Best Regards,<br>
> Sergei Mamonov<br>
> _______________________________________________<br>
> Users mailing list<br>
> <a href="mailto:Users@openvz.org" target="_blank">Users@openvz.org</a><br>
> <a href="https://lists.openvz.org/mailman/listinfo/users" rel="noreferrer" target="_blank">https://lists.openvz.org/mailman/listinfo/users</a><br>
<br>
_______________________________________________<br>
Users mailing list<br>
<a href="mailto:Users@openvz.org" target="_blank">Users@openvz.org</a><br>
<a href="https://lists.openvz.org/mailman/listinfo/users" rel="noreferrer" target="_blank">https://lists.openvz.org/mailman/listinfo/users</a><br>
</blockquote></div><br clear="all"><br>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><span style="font-size:12.8px">Best Regards,</span><br></div><div>Sergei Mamonov</div></div></div></div></div>