[Devel] BUG: Overflow in mem_cgroup_apply_beancounter leads to ENOMEM

[Vladimir Meshkov]

When unlimited UBC is created next code is performed in
mem_cgroup_apply_beancounter:

        mem = ub->ub_parms[UB_PHYSPAGES].limit;
        memsw = ub->ub_parms[UB_SWAPPAGES].limit;

        if (memsw < PAGE_COUNTER_MAX - mem)
                memsw += mem;
    else
                memsw = PAGE_COUNTER_MAX;

If mem == memsw == UB_MAXVALUE == 9223372036854775807, the condition if
(memsw < PAGE_COUNTER_MAX - mem) will be true,
due to (LONG_MAX/PAGE_SIZE - LONG_MAX) > LONG_MAX and memsw is unsigned
long long.
Then we add 9223372036854775807 + 9223372036854775807, and this leads to
overflow, and as a result we have unsigned long
memcg->memsw.limit==18446744073709551614, or -2 for long type variable.

The value memsw==18446744073709551614 is used in mem_cgroup_enough_memory
to calculate variable free:

    long free = memcg->memsw.limit - page_counter_read(&memcg->memsw);

Due to type 'long' memcg->memsw.limit == -2 (18446744073709551614), and the
function returns -ENOMEM despite of unlimited UBC.

Could you please check the condition if (memsw < PAGE_COUNTER_MAX - mem) ?
It's pretty strange to compare LONG_MAX/PAGE_SIZE and LONG_MAX values.

--
Best regards,
Vladimir Meshkov,
CloudLinux kernel team
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openvz.org/pipermail/devel/attachments/20170531/4e8c8430/attachment.html>