<div dir="ltr"><div>And after migrate all containers to another node it still shows 63745 cgroups - <br></div><div><br></div><div>cat /proc/cgroups <br>#subsys_name hierarchy num_cgroups enabled<br>cpuset 7 2 1<br>cpu 10 2 1<br>cpuacct 10 2 1<br>memory 2 63745 1<br>devices 11 2 1<br>freezer 17 2 1<br>net_cls 12 2 1<br>blkio 1 4 1<br>perf_event 13 2 1<br>hugetlb 14 2 1<br>pids 3 68 1<br>ve 6 1 1<br>beancounter 4 3 1<br>net_prio 12 2 1<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, 10 Feb 2021 at 18:47, Сергей Мамонов <<a href="mailto:mrqwer88@gmail.com">mrqwer88@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>And it is definitely it - <br></div><div>grep -E "memory|num_cgroups" /proc/cgroups </div>#subsys_name hierarchy num_cgroups enabled<br><div>memory 2 65534 1</div><div><br></div><div>After migration some of containers to another node num_cgroups goes down to 65365 and it allowed to start stopped container without `<br><div>Can't create directory /sys/fs/cgroup/memory/machine.slice/1000133882: Cannot allocate memory` error.</div><div><br></div><div>But I don't understand why num_cgroups for memory so big, yet.</div><div><br></div><div>Like ~460 per container instead of 60 and less per container on other nodes (with the same kernel version too).<br></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, 10 Feb 2021 at 17:48, Сергей Мамонов <<a href="mailto:mrqwer88@gmail.com" target="_blank">mrqwer88@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Hello!</div><div><br></div><div>Looks like we reproduced this problem too.</div><div><br></div><div>kernel - 3.10.0-1127.18.2.vz7.163.46</div><div><br></div><div>Same error - <br></div><div>Can't create directory /sys/fs/cgroup/memory/machine.slice/1000133882: Cannot allocate memory</div><div><br></div><div>Same ok output for<br></div><div>/sys/fs/cgroup/<span>memory</span>/*limit_in_bytes<br>
/sys/fs/cgroup/<span>memory</span>/machine.slice/*limit_in_bytes</div><div><br></div><div>Have a lot of free memory on node (per numa too).</div><div><br></div><div>Only that looks really strange - <br></div><div>grep -E "memory|num_cgroups" /proc/cgroups <br>#subsys_name hierarchy num_cgroups enabled<br>memory 2 65534 1</div><div><br></div><div>huge nub_cgroups only on this node</div><div><br></div><div>cat /proc/cgroups <br>#subsys_name hierarchy num_cgroups enabled<br>cpuset 7 144 1<br>cpu 10 263 1<br>cpuacct 10 263 1<br>memory 2 65534 1<br>devices 11 1787 1<br>freezer 17 144 1<br>net_cls 12 144 1<br>blkio 1 257 1<br>perf_event 13 144 1<br>hugetlb 14 144 1<br>pids 3 2955 1<br>ve 6 143 1<br>beancounter 4 143 1<br>net_prio 12 144 1<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, 28 Jan 2021 at 14:22, Konstantin Khorenko <<a href="mailto:khorenko@virtuozzo.com" target="_blank">khorenko@virtuozzo.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p><tt>May be you hit memory shortage in a particular NUMA node
only, for example.<br>
<br>
# numactl --hardware<br>
# numastat -m</tt></p>
<p><tt><br>
</tt></p>
<p><tt>Or go hard way - trace kernel where exactly do we get -ENOMEM:</tt></p>
<p><tt>trace the kernel function cgroup_mkdir() using /sys/kernel/debug/tracing/<br>
with function_graph tracer.<br>
</tt></p>
<p><br>
<a href="https://lwn.net/Articles/370423/" target="_blank">https://lwn.net/Articles/370423/</a></p>
<pre cols="179">--
Best regards,
Konstantin Khorenko,
Virtuozzo Linux Kernel Team
</pre>
<div>On 01/28/2021 12:43 PM, Joe Dougherty
wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">I checked that, doesn't appear to be
the case.
<div><br>
</div>
<div>
<div><b># pwd</b></div>
<div><b>/sys/fs/cgroup/memory</b></div>
</div>
<div>
<div><b># cat *limit_in_bytes</b></div>
<div><b>9223372036854771712</b></div>
<div><b>9223372036854767616</b></div>
<div><b>2251799813685247</b></div>
<div><b>2251799813685247</b></div>
<div><b>9223372036854771712</b></div>
<div><b>9223372036854771712</b></div>
<div><b>9223372036854771712</b></div>
</div>
<div>
<div><b># cat *failcnt</b></div>
<div><b>0</b></div>
<div><b>0</b></div>
<div><b>0</b></div>
<div><b>0</b></div>
<div><b>0</b></div>
<div style="font-weight:bold"><br>
</div>
</div>
<div>
<div style="font-weight:bold"># pwd</div>
<div style="font-weight:bold">/sys/fs/cgroup/memory/machine.slice</div>
<div>
<div><b># cat *limit_in_bytes</b></div>
<div><b>9223372036854771712</b></div>
<div><b>9223372036854767616</b></div>
<div><b>9223372036854771712</b></div>
<div><b>9223372036854771712</b></div>
<div><b>9223372036854771712</b></div>
<div><b>9223372036854771712</b></div>
<div><b>9223372036854771712</b></div>
</div>
<div>
<div><b># cat *failcnt</b></div>
<div><b>0</b></div>
<div><b>0</b></div>
<div><b>0</b></div>
<div><b>0</b></div>
<div><b>0</b></div>
<div style="font-weight:bold"><br>
</div>
</div>
<div style="font-weight:bold"><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Thu, Jan 28, 2021 at 2:47
AM Konstantin Khorenko <<a href="mailto:khorenko@virtuozzo.com" target="_blank">khorenko@virtuozzo.com</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>Hi Joe,</p>
<p>i'd suggest to check memory limits for root and
"machine.slice" memory cgroups<br>
</p>
<p>/sys/fs/cgroup/memory/*limit_in_bytes<br>
/sys/fs/cgroup/memory/machine.slice/*limit_in_bytes<br>
</p>
<p>All of them should be unlimited.</p>
<p>If not - search who limit them.<br>
</p>
<pre cols="179">--
Best regards,
Konstantin Khorenko,
Virtuozzo Linux Kernel Team
</pre>
<div>On 01/27/2021 10:28 PM, Joe Dougherty wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">I'm running into an issue on only 1
of my OpenVZ 7 nodes where it's unable to create
a directory on
/sys/fs/cgroup/memory/machine.slice due to
"Cannot allocate memory" whenever I try to start
a new container or restart and existing one.
I've been trying to research this but I'm unable
to find any concrete info on what could cause
this. It appears to be memory related because
sometimes if I issue "echo 1
/proc/sys/vm/drop_caches" it allows me to start
a container (this only works sometimes) but my
RAM usage is extremely low with no swapping
(swappiness even set to 0 for testing). Thank
you in advance for your help.</div>
<div dir="ltr"><br>
</div>
<div dir="ltr"><br>
</div>
<div dir="ltr">Example:</div>
<div dir="ltr">
<div dir="ltr"><b># vzctl start 9499</b></div>
<div dir="ltr"><b>Starting Container ...</b></div>
<div dir="ltr"><b>Mount image:
/vz/private/9499/root.hdd</b></div>
<div dir="ltr"><b>Container is mounted</b></div>
<div dir="ltr"><b>Can't create directory
/sys/fs/cgroup/memory/machine.slice/9499:
Cannot allocate memory</b></div>
<div dir="ltr"><b>Unmount image:
/vz/private/9499/root.hdd (190)</b></div>
<div dir="ltr"><b>Container is unmounted</b></div>
<div dir="ltr"><b>Failed to start the Container</b></div>
<div><br>
</div>
<div><br>
</div>
<div>Node Info:<br>
</div>
<div>
<div><b>Uptime: 10 days</b></div>
<div><b>OS: Virtuozzo 7.0.15</b></div>
<div><b>Kernel:
3.10.0-1127.18.2.vz7.163.46 GNU/Linux</b></div>
<div><b>System Load: 3.1</b></div>
<div><b>/vz Usage: 56% of 37T</b></div>
<div><b>Swap Usage: 0%</b></div>
<div><b>RAM Free: 84% of 94.2GB</b></div>
</div>
<div><b><br>
</b></div>
<div>
<div><b># free -m</b></div>
<div><b> total used
free shared buff/cache
available</b></div>
<div><b>Mem: 96502 14259
49940 413 32303
80990</b></div>
<div><b>Swap: 32767 93
32674</b></div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
<fieldset></fieldset>
<br>
<pre>_______________________________________________
Users mailing list
<a href="mailto:Users@openvz.org" target="_blank">Users@openvz.org</a>
<a href="https://lists.openvz.org/mailman/listinfo/users" target="_blank">https://lists.openvz.org/mailman/listinfo/users</a>
</pre>
</blockquote>
<br>
</div>
_______________________________________________<br>
Users mailing list<br>
<a href="mailto:Users@openvz.org" target="_blank">Users@openvz.org</a><br>
<a href="https://lists.openvz.org/mailman/listinfo/users" rel="noreferrer" target="_blank">https://lists.openvz.org/mailman/listinfo/users</a><br>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr"><b><i><font face="verdana,
sans-serif">-Joe Dougherty</font></i></b>
<div><b><i><font face="verdana, sans-serif">Chief Operating
Officer</font></i></b></div>
<div>
<div><i><font face="verdana, sans-serif"><b><span style="font-size:13px;line-height:18px"></span></b><b>Secure
Dragon LLC </b></font></i></div>
<div><b><i><a href="http://www.SecureDragon.net" target="_blank"><font face="verdana, sans-serif">www.SecureDragon.net</font></a></i></b></div>
</div>
</div>
<br>
<fieldset></fieldset>
<br>
<pre>_______________________________________________
Users mailing list
<a href="mailto:Users@openvz.org" target="_blank">Users@openvz.org</a>
<a href="https://lists.openvz.org/mailman/listinfo/users" target="_blank">https://lists.openvz.org/mailman/listinfo/users</a>
</pre>
</blockquote>
<br>
</div>
_______________________________________________<br>
Users mailing list<br>
<a href="mailto:Users@openvz.org" target="_blank">Users@openvz.org</a><br>
<a href="https://lists.openvz.org/mailman/listinfo/users" rel="noreferrer" target="_blank">https://lists.openvz.org/mailman/listinfo/users</a><br>
</blockquote></div><br clear="all"><br>-- <br><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div><span style="font-size:12.8px">Best Regards,</span><br></div><div>Sergei Mamonov</div></div></div></div></div>
</blockquote></div><br clear="all"><br>-- <br><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div><span style="font-size:12.8px">Best Regards,</span><br></div><div>Sergei Mamonov</div></div></div></div></div>
</blockquote></div><br clear="all"><br>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><span style="font-size:12.8px">Best Regards,</span><br></div><div>Sergei Mamonov</div></div></div></div></div>