[Users] Can't create directory /sys/fs/cgroup/memory/machine.slice/ Cannot allocate memory
Сергей Мамонов
mrqwer88 at gmail.com
Fri Feb 12 04:45:53 MSK 2021
/sys/fs/cgroup/memory was checked and not show this groups -
find /sys/fs/cgroup/memory -type d | wc -l
2
grep -E "memory|num" /proc/cgroups
#subsys_name hierarchy num_cgroups enabled
memory 2 63744 1
And more strange - uptime was only about 3 days when the problem reproduced.
On Fri, 12 Feb 2021 at 04:06, Kir Kolyshkin <kir at openvz.org> wrote:
> On Thu, Feb 11, 2021 at 12:59 AM Сергей Мамонов <mrqwer88 at gmail.com>
> wrote:
> >
> > And after migrate all containers to another node it still shows 63745
> cgroups -
> >
> > cat /proc/cgroups
> > #subsys_name hierarchy num_cgroups enabled
> > cpuset 7 2 1
> > cpu 10 2 1
> > cpuacct 10 2 1
> > memory 2 63745 1
>
> Looks like a leakage (or a bug in memory accounting which prevents
> cgroup from being released).
> You can check the number of memory cgroups with something like
>
> find /sys/fs/cgroup/memory -type d | wc -l
>
> If you see a large number, go explore those cgroups (check
> cgroup.procs, memory.usage_in_bytes).
>
> > devices 11 2 1
> > freezer 17 2 1
> > net_cls 12 2 1
> > blkio 1 4 1
> > perf_event 13 2 1
> > hugetlb 14 2 1
> > pids 3 68 1
> > ve 6 1 1
> > beancounter 4 3 1
> > net_prio 12 2 1
> >
> > On Wed, 10 Feb 2021 at 18:47, Сергей Мамонов <mrqwer88 at gmail.com> wrote:
> >>
> >> And it is definitely it -
> >> grep -E "memory|num_cgroups" /proc/cgroups
> >> #subsys_name hierarchy num_cgroups enabled
> >> memory 2 65534 1
> >>
> >> After migration some of containers to another node num_cgroups goes
> down to 65365 and it allowed to start stopped container without `
> >> Can't create directory /sys/fs/cgroup/memory/machine.slice/1000133882:
> Cannot allocate memory` error.
> >>
> >> But I don't understand why num_cgroups for memory so big, yet.
> >>
> >> Like ~460 per container instead of 60 and less per container on other
> nodes (with the same kernel version too).
> >>
> >> On Wed, 10 Feb 2021 at 17:48, Сергей Мамонов <mrqwer88 at gmail.com>
> wrote:
> >>>
> >>> Hello!
> >>>
> >>> Looks like we reproduced this problem too.
> >>>
> >>> kernel - 3.10.0-1127.18.2.vz7.163.46
> >>>
> >>> Same error -
> >>> Can't create directory /sys/fs/cgroup/memory/machine.slice/1000133882:
> Cannot allocate memory
> >>>
> >>> Same ok output for
> >>> /sys/fs/cgroup/memory/*limit_in_bytes
> >>> /sys/fs/cgroup/memory/machine.slice/*limit_in_bytes
> >>>
> >>> Have a lot of free memory on node (per numa too).
> >>>
> >>> Only that looks really strange -
> >>> grep -E "memory|num_cgroups" /proc/cgroups
> >>> #subsys_name hierarchy num_cgroups enabled
> >>> memory 2 65534 1
> >>>
> >>> huge nub_cgroups only on this node
> >>>
> >>> cat /proc/cgroups
> >>> #subsys_name hierarchy num_cgroups enabled
> >>> cpuset 7 144 1
> >>> cpu 10 263 1
> >>> cpuacct 10 263 1
> >>> memory 2 65534 1
> >>> devices 11 1787 1
> >>> freezer 17 144 1
> >>> net_cls 12 144 1
> >>> blkio 1 257 1
> >>> perf_event 13 144 1
> >>> hugetlb 14 144 1
> >>> pids 3 2955 1
> >>> ve 6 143 1
> >>> beancounter 4 143 1
> >>> net_prio 12 144 1
> >>>
> >>> On Thu, 28 Jan 2021 at 14:22, Konstantin Khorenko <
> khorenko at virtuozzo.com> wrote:
> >>>>
> >>>> May be you hit memory shortage in a particular NUMA node only, for
> example.
> >>>>
> >>>> # numactl --hardware
> >>>> # numastat -m
> >>>>
> >>>>
> >>>> Or go hard way - trace kernel where exactly do we get -ENOMEM:
> >>>>
> >>>> trace the kernel function cgroup_mkdir() using
> /sys/kernel/debug/tracing/
> >>>> with function_graph tracer.
> >>>>
> >>>>
> >>>> https://lwn.net/Articles/370423/
> >>>>
> >>>> --
> >>>> Best regards,
> >>>>
> >>>> Konstantin Khorenko,
> >>>> Virtuozzo Linux Kernel Team
> >>>>
> >>>> On 01/28/2021 12:43 PM, Joe Dougherty wrote:
> >>>>
> >>>> I checked that, doesn't appear to be the case.
> >>>>
> >>>> # pwd
> >>>> /sys/fs/cgroup/memory
> >>>> # cat *limit_in_bytes
> >>>> 9223372036854771712
> >>>> 9223372036854767616
> >>>> 2251799813685247
> >>>> 2251799813685247
> >>>> 9223372036854771712
> >>>> 9223372036854771712
> >>>> 9223372036854771712
> >>>> # cat *failcnt
> >>>> 0
> >>>> 0
> >>>> 0
> >>>> 0
> >>>> 0
> >>>>
> >>>> # pwd
> >>>> /sys/fs/cgroup/memory/machine.slice
> >>>> # cat *limit_in_bytes
> >>>> 9223372036854771712
> >>>> 9223372036854767616
> >>>> 9223372036854771712
> >>>> 9223372036854771712
> >>>> 9223372036854771712
> >>>> 9223372036854771712
> >>>> 9223372036854771712
> >>>> # cat *failcnt
> >>>> 0
> >>>> 0
> >>>> 0
> >>>> 0
> >>>> 0
> >>>>
> >>>>
> >>>>
> >>>> On Thu, Jan 28, 2021 at 2:47 AM Konstantin Khorenko <
> khorenko at virtuozzo.com> wrote:
> >>>>>
> >>>>> Hi Joe,
> >>>>>
> >>>>> i'd suggest to check memory limits for root and "machine.slice"
> memory cgroups
> >>>>>
> >>>>> /sys/fs/cgroup/memory/*limit_in_bytes
> >>>>> /sys/fs/cgroup/memory/machine.slice/*limit_in_bytes
> >>>>>
> >>>>> All of them should be unlimited.
> >>>>>
> >>>>> If not - search who limit them.
> >>>>>
> >>>>> --
> >>>>> Best regards,
> >>>>>
> >>>>> Konstantin Khorenko,
> >>>>> Virtuozzo Linux Kernel Team
> >>>>>
> >>>>> On 01/27/2021 10:28 PM, Joe Dougherty wrote:
> >>>>>
> >>>>> I'm running into an issue on only 1 of my OpenVZ 7 nodes where it's
> unable to create a directory on /sys/fs/cgroup/memory/machine.slice due to
> "Cannot allocate memory" whenever I try to start a new container or restart
> and existing one. I've been trying to research this but I'm unable to find
> any concrete info on what could cause this. It appears to be memory related
> because sometimes if I issue "echo 1 /proc/sys/vm/drop_caches" it allows me
> to start a container (this only works sometimes) but my RAM usage is
> extremely low with no swapping (swappiness even set to 0 for testing).
> Thank you in advance for your help.
> >>>>>
> >>>>>
> >>>>> Example:
> >>>>> # vzctl start 9499
> >>>>> Starting Container ...
> >>>>> Mount image: /vz/private/9499/root.hdd
> >>>>> Container is mounted
> >>>>> Can't create directory /sys/fs/cgroup/memory/machine.slice/9499:
> Cannot allocate memory
> >>>>> Unmount image: /vz/private/9499/root.hdd (190)
> >>>>> Container is unmounted
> >>>>> Failed to start the Container
> >>>>>
> >>>>>
> >>>>> Node Info:
> >>>>> Uptime: 10 days
> >>>>> OS: Virtuozzo 7.0.15
> >>>>> Kernel: 3.10.0-1127.18.2.vz7.163.46 GNU/Linux
> >>>>> System Load: 3.1
> >>>>> /vz Usage: 56% of 37T
> >>>>> Swap Usage: 0%
> >>>>> RAM Free: 84% of 94.2GB
> >>>>>
> >>>>> # free -m
> >>>>> total used free shared
> buff/cache available
> >>>>> Mem: 96502 14259 49940 413 32303
> 80990
> >>>>> Swap: 32767 93 32674
> >>>>>
> >>>>>
> >>>>> _______________________________________________
> >>>>> Users mailing list
> >>>>> Users at openvz.org
> >>>>> https://lists.openvz.org/mailman/listinfo/users
> >>>>>
> >>>>>
> >>>>> _______________________________________________
> >>>>> Users mailing list
> >>>>> Users at openvz.org
> >>>>> https://lists.openvz.org/mailman/listinfo/users
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> -Joe Dougherty
> >>>> Chief Operating Officer
> >>>> Secure Dragon LLC
> >>>> www.SecureDragon.net
> >>>>
> >>>>
> >>>> _______________________________________________
> >>>> Users mailing list
> >>>> Users at openvz.org
> >>>> https://lists.openvz.org/mailman/listinfo/users
> >>>>
> >>>>
> >>>> _______________________________________________
> >>>> Users mailing list
> >>>> Users at openvz.org
> >>>> https://lists.openvz.org/mailman/listinfo/users
> >>>
> >>>
> >>>
> >>> --
> >>> Best Regards,
> >>> Sergei Mamonov
> >>
> >>
> >>
> >> --
> >> Best Regards,
> >> Sergei Mamonov
> >
> >
> >
> > --
> > Best Regards,
> > Sergei Mamonov
> > _______________________________________________
> > Users mailing list
> > Users at openvz.org
> > https://lists.openvz.org/mailman/listinfo/users
>
> _______________________________________________
> Users mailing list
> Users at openvz.org
> https://lists.openvz.org/mailman/listinfo/users
>
--
Best Regards,
Sergei Mamonov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openvz.org/pipermail/users/attachments/20210212/161944cb/attachment-0001.html>
More information about the Users
mailing list