<html><head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p><tt>May be you hit memory shortage in a particular NUMA node
only, for example.<br>
<br>
# numactl --hardware<br>
# numastat -m</tt></p>
<p><tt><br>
</tt></p>
<p><tt>Or go hard way - trace kernel where exactly do we get -ENOMEM:</tt></p>
<p><tt>trace the kernel function cgroup_mkdir() using /sys/kernel/debug/tracing/<br>
with function_graph tracer.<br>
</tt></p>
<p><br>
<a href="https://lwn.net/Articles/370423/">https://lwn.net/Articles/370423/</a></p>
<pre class="moz-signature" cols="179">--
Best regards,
Konstantin Khorenko,
Virtuozzo Linux Kernel Team
</pre>
<div class="moz-cite-prefix">On 01/28/2021 12:43 PM, Joe Dougherty
wrote:<br>
</div>
<blockquote cite="mid:CAPoKwR3oMjLLD75CPYRNVPcEeg7oPC6middXVF+j__385ZwN8A@mail.gmail.com" type="cite">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">I checked that, doesn't appear to be
the case.
<div><br>
</div>
<div>
<div><b># pwd</b></div>
<div><b>/sys/fs/cgroup/memory</b></div>
</div>
<div>
<div><b># cat *limit_in_bytes</b></div>
<div><b>9223372036854771712</b></div>
<div><b>9223372036854767616</b></div>
<div><b>2251799813685247</b></div>
<div><b>2251799813685247</b></div>
<div><b>9223372036854771712</b></div>
<div><b>9223372036854771712</b></div>
<div><b>9223372036854771712</b></div>
</div>
<div>
<div><b># cat *failcnt</b></div>
<div><b>0</b></div>
<div><b>0</b></div>
<div><b>0</b></div>
<div><b>0</b></div>
<div><b>0</b></div>
<div style="font-weight:bold"><br>
</div>
</div>
<div>
<div style="font-weight:bold"># pwd</div>
<div style="font-weight:bold">/sys/fs/cgroup/memory/machine.slice</div>
<div>
<div><b># cat *limit_in_bytes</b></div>
<div><b>9223372036854771712</b></div>
<div><b>9223372036854767616</b></div>
<div><b>9223372036854771712</b></div>
<div><b>9223372036854771712</b></div>
<div><b>9223372036854771712</b></div>
<div><b>9223372036854771712</b></div>
<div><b>9223372036854771712</b></div>
</div>
<div>
<div><b># cat *failcnt</b></div>
<div><b>0</b></div>
<div><b>0</b></div>
<div><b>0</b></div>
<div><b>0</b></div>
<div><b>0</b></div>
<div style="font-weight:bold"><br>
</div>
</div>
<div style="font-weight:bold"><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Thu, Jan 28, 2021 at 2:47
AM Konstantin Khorenko <<a moz-do-not-send="true" href="mailto:khorenko@virtuozzo.com">khorenko@virtuozzo.com</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>Hi Joe,</p>
<p>i'd suggest to check memory limits for root and
"machine.slice" memory cgroups<br>
</p>
<p>/sys/fs/cgroup/memory/*limit_in_bytes<br>
/sys/fs/cgroup/memory/machine.slice/*limit_in_bytes<br>
</p>
<p>All of them should be unlimited.</p>
<p>If not - search who limit them.<br>
</p>
<pre cols="179">--
Best regards,
Konstantin Khorenko,
Virtuozzo Linux Kernel Team
</pre>
<div>On 01/27/2021 10:28 PM, Joe Dougherty wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">I'm running into an issue on only 1
of my OpenVZ 7 nodes where it's unable to create
a directory on
/sys/fs/cgroup/memory/machine.slice due to
"Cannot allocate memory" whenever I try to start
a new container or restart and existing one.
I've been trying to research this but I'm unable
to find any concrete info on what could cause
this. It appears to be memory related because
sometimes if I issue "echo 1
/proc/sys/vm/drop_caches" it allows me to start
a container (this only works sometimes) but my
RAM usage is extremely low with no swapping
(swappiness even set to 0 for testing). Thank
you in advance for your help.</div>
<div dir="ltr"><br>
</div>
<div dir="ltr"><br>
</div>
<div dir="ltr">Example:</div>
<div dir="ltr">
<div dir="ltr"><b># vzctl start 9499</b></div>
<div dir="ltr"><b>Starting Container ...</b></div>
<div dir="ltr"><b>Mount image:
/vz/private/9499/root.hdd</b></div>
<div dir="ltr"><b>Container is mounted</b></div>
<div dir="ltr"><b>Can't create directory
/sys/fs/cgroup/memory/machine.slice/9499:
Cannot allocate memory</b></div>
<div dir="ltr"><b>Unmount image:
/vz/private/9499/root.hdd (190)</b></div>
<div dir="ltr"><b>Container is unmounted</b></div>
<div dir="ltr"><b>Failed to start the Container</b></div>
<div><br>
</div>
<div><br>
</div>
<div>Node Info:<br>
</div>
<div>
<div><b>Uptime: 10 days</b></div>
<div><b>OS: Virtuozzo 7.0.15</b></div>
<div><b>Kernel:
3.10.0-1127.18.2.vz7.163.46 GNU/Linux</b></div>
<div><b>System Load: 3.1</b></div>
<div><b>/vz Usage: 56% of 37T</b></div>
<div><b>Swap Usage: 0%</b></div>
<div><b>RAM Free: 84% of 94.2GB</b></div>
</div>
<div><b><br>
</b></div>
<div>
<div><b># free -m</b></div>
<div><b> total used
free shared buff/cache
available</b></div>
<div><b>Mem: 96502 14259
49940 413 32303
80990</b></div>
<div><b>Swap: 32767 93
32674</b></div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
<fieldset></fieldset>
<br>
<pre>_______________________________________________
Users mailing list
<a moz-do-not-send="true" href="mailto:Users@openvz.org" target="_blank">Users@openvz.org</a>
<a moz-do-not-send="true" href="https://lists.openvz.org/mailman/listinfo/users" target="_blank">https://lists.openvz.org/mailman/listinfo/users</a>
</pre>
</blockquote>
<br>
</div>
_______________________________________________<br>
Users mailing list<br>
<a moz-do-not-send="true" href="mailto:Users@openvz.org" target="_blank">Users@openvz.org</a><br>
<a moz-do-not-send="true" href="https://lists.openvz.org/mailman/listinfo/users" rel="noreferrer" target="_blank">https://lists.openvz.org/mailman/listinfo/users</a><br>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr" class="gmail_signature"><b><i><font face="verdana,
sans-serif">-Joe Dougherty</font></i></b>
<div><b><i><font face="verdana, sans-serif">Chief Operating
Officer</font></i></b></div>
<div>
<div><i><font face="verdana, sans-serif"><b><span style="font-size:13px;line-height:18px"></span></b><b>Secure
Dragon LLC </b></font></i></div>
<div><b><i><a moz-do-not-send="true" href="http://www.SecureDragon.net" target="_blank"><font face="verdana, sans-serif">www.SecureDragon.net</font></a></i></b></div>
</div>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
Users mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Users@openvz.org">Users@openvz.org</a>
<a class="moz-txt-link-freetext" href="https://lists.openvz.org/mailman/listinfo/users">https://lists.openvz.org/mailman/listinfo/users</a>
</pre>
</blockquote>
<br>
</body>
</html>