[Users] Can't create directory /sys/fs/cgroup/memory/machine.slice/ Cannot allocate memory
Сергей Мамонов
mrqwer88 at gmail.com
Thu Feb 11 11:55:57 MSK 2021
And after migrate all containers to another node it still shows 63745
cgroups -
cat /proc/cgroups
#subsys_name hierarchy num_cgroups enabled
cpuset 7 2 1
cpu 10 2 1
cpuacct 10 2 1
memory 2 63745 1
devices 11 2 1
freezer 17 2 1
net_cls 12 2 1
blkio 1 4 1
perf_event 13 2 1
hugetlb 14 2 1
pids 3 68 1
ve 6 1 1
beancounter 4 3 1
net_prio 12 2 1
On Wed, 10 Feb 2021 at 18:47, Сергей Мамонов <mrqwer88 at gmail.com> wrote:
> And it is definitely it -
> grep -E "memory|num_cgroups" /proc/cgroups
> #subsys_name hierarchy num_cgroups enabled
> memory 2 65534 1
>
> After migration some of containers to another node num_cgroups goes down
> to 65365 and it allowed to start stopped container without `
> Can't create directory /sys/fs/cgroup/memory/machine.slice/1000133882:
> Cannot allocate memory` error.
>
> But I don't understand why num_cgroups for memory so big, yet.
>
> Like ~460 per container instead of 60 and less per container on other
> nodes (with the same kernel version too).
>
> On Wed, 10 Feb 2021 at 17:48, Сергей Мамонов <mrqwer88 at gmail.com> wrote:
>
>> Hello!
>>
>> Looks like we reproduced this problem too.
>>
>> kernel - 3.10.0-1127.18.2.vz7.163.46
>>
>> Same error -
>> Can't create directory /sys/fs/cgroup/memory/machine.slice/1000133882:
>> Cannot allocate memory
>>
>> Same ok output for
>> /sys/fs/cgroup/memory/*limit_in_bytes
>> /sys/fs/cgroup/memory/machine.slice/*limit_in_bytes
>>
>> Have a lot of free memory on node (per numa too).
>>
>> Only that looks really strange -
>> grep -E "memory|num_cgroups" /proc/cgroups
>> #subsys_name hierarchy num_cgroups enabled
>> memory 2 65534 1
>>
>> huge nub_cgroups only on this node
>>
>> cat /proc/cgroups
>> #subsys_name hierarchy num_cgroups enabled
>> cpuset 7 144 1
>> cpu 10 263 1
>> cpuacct 10 263 1
>> memory 2 65534 1
>> devices 11 1787 1
>> freezer 17 144 1
>> net_cls 12 144 1
>> blkio 1 257 1
>> perf_event 13 144 1
>> hugetlb 14 144 1
>> pids 3 2955 1
>> ve 6 143 1
>> beancounter 4 143 1
>> net_prio 12 144 1
>>
>> On Thu, 28 Jan 2021 at 14:22, Konstantin Khorenko <khorenko at virtuozzo.com>
>> wrote:
>>
>>> May be you hit memory shortage in a particular NUMA node only, for
>>> example.
>>>
>>> # numactl --hardware
>>> # numastat -m
>>>
>>>
>>> Or go hard way - trace kernel where exactly do we get -ENOMEM:
>>>
>>> trace the kernel function cgroup_mkdir() using /sys/kernel/debug/tracing/
>>> with function_graph tracer.
>>>
>>>
>>> https://lwn.net/Articles/370423/
>>>
>>> --
>>> Best regards,
>>>
>>> Konstantin Khorenko,
>>> Virtuozzo Linux Kernel Team
>>>
>>> On 01/28/2021 12:43 PM, Joe Dougherty wrote:
>>>
>>> I checked that, doesn't appear to be the case.
>>>
>>> *# pwd*
>>> */sys/fs/cgroup/memory*
>>> *# cat *limit_in_bytes*
>>> *9223372036854771712*
>>> *9223372036854767616*
>>> *2251799813685247*
>>> *2251799813685247*
>>> *9223372036854771712*
>>> *9223372036854771712*
>>> *9223372036854771712*
>>> *# cat *failcnt*
>>> *0*
>>> *0*
>>> *0*
>>> *0*
>>> *0*
>>>
>>> # pwd
>>> /sys/fs/cgroup/memory/machine.slice
>>> *# cat *limit_in_bytes*
>>> *9223372036854771712*
>>> *9223372036854767616*
>>> *9223372036854771712*
>>> *9223372036854771712*
>>> *9223372036854771712*
>>> *9223372036854771712*
>>> *9223372036854771712*
>>> *# cat *failcnt*
>>> *0*
>>> *0*
>>> *0*
>>> *0*
>>> *0*
>>>
>>>
>>>
>>> On Thu, Jan 28, 2021 at 2:47 AM Konstantin Khorenko <
>>> khorenko at virtuozzo.com> wrote:
>>>
>>>> Hi Joe,
>>>>
>>>> i'd suggest to check memory limits for root and "machine.slice" memory
>>>> cgroups
>>>>
>>>> /sys/fs/cgroup/memory/*limit_in_bytes
>>>> /sys/fs/cgroup/memory/machine.slice/*limit_in_bytes
>>>>
>>>> All of them should be unlimited.
>>>>
>>>> If not - search who limit them.
>>>>
>>>> --
>>>> Best regards,
>>>>
>>>> Konstantin Khorenko,
>>>> Virtuozzo Linux Kernel Team
>>>>
>>>> On 01/27/2021 10:28 PM, Joe Dougherty wrote:
>>>>
>>>> I'm running into an issue on only 1 of my OpenVZ 7 nodes where it's
>>>> unable to create a directory on /sys/fs/cgroup/memory/machine.slice due to
>>>> "Cannot allocate memory" whenever I try to start a new container or restart
>>>> and existing one. I've been trying to research this but I'm unable to find
>>>> any concrete info on what could cause this. It appears to be memory related
>>>> because sometimes if I issue "echo 1 /proc/sys/vm/drop_caches" it allows me
>>>> to start a container (this only works sometimes) but my RAM usage is
>>>> extremely low with no swapping (swappiness even set to 0 for testing).
>>>> Thank you in advance for your help.
>>>>
>>>>
>>>> Example:
>>>> *# vzctl start 9499*
>>>> *Starting Container ...*
>>>> *Mount image: /vz/private/9499/root.hdd*
>>>> *Container is mounted*
>>>> *Can't create directory /sys/fs/cgroup/memory/machine.slice/9499:
>>>> Cannot allocate memory*
>>>> *Unmount image: /vz/private/9499/root.hdd (190)*
>>>> *Container is unmounted*
>>>> *Failed to start the Container*
>>>>
>>>>
>>>> Node Info:
>>>> *Uptime: 10 days*
>>>> *OS: Virtuozzo 7.0.15*
>>>> *Kernel: 3.10.0-1127.18.2.vz7.163.46 GNU/Linux*
>>>> *System Load: 3.1*
>>>> */vz Usage: 56% of 37T*
>>>> *Swap Usage: 0%*
>>>> *RAM Free: 84% of 94.2GB*
>>>>
>>>> *# free -m*
>>>> * total used free shared
>>>> buff/cache available*
>>>> *Mem: 96502 14259 49940 413 32303
>>>> 80990*
>>>> *Swap: 32767 93 32674*
>>>>
>>>>
>>>> _______________________________________________
>>>> Users mailing listUsers at openvz.orghttps://lists.openvz.org/mailman/listinfo/users
>>>>
>>>>
>>>> _______________________________________________
>>>> Users mailing list
>>>> Users at openvz.org
>>>> https://lists.openvz.org/mailman/listinfo/users
>>>>
>>>
>>>
>>> --
>>> *-Joe Dougherty*
>>> *Chief Operating Officer*
>>> *Secure Dragon LLC *
>>> *www.SecureDragon.net <http://www.SecureDragon.net>*
>>>
>>>
>>> _______________________________________________
>>> Users mailing listUsers at openvz.orghttps://lists.openvz.org/mailman/listinfo/users
>>>
>>>
>>> _______________________________________________
>>> Users mailing list
>>> Users at openvz.org
>>> https://lists.openvz.org/mailman/listinfo/users
>>>
>>
>>
>> --
>> Best Regards,
>> Sergei Mamonov
>>
>
>
> --
> Best Regards,
> Sergei Mamonov
>
--
Best Regards,
Sergei Mamonov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openvz.org/pipermail/users/attachments/20210211/7aacf870/attachment-0001.html>
More information about the Users
mailing list