[Devel] [PATCH rh7 2/3] net: core: use atomic high-order allocations

[Debabrata Banerjee]

On Fri, Oct 21, 2016 at 7:36 AM, Anatoly Stepanov
<astepanov at cloudlinux.com> wrote:
> As we detected intensive direct reclaim activity in
> sk_page_frag_refill()
> it's reasonable to prevent it from trying so hard to allocate high-order
> blocks, just do it when it's effortless.
>
> This is a port of upstream (vanilla) change.
> Original commit: fb05e7a89f500cfc06ae277bdc911b281928995d
>
> We saw excessive direct memory compaction triggered by
> skb_page_frag_refill.
> This causes performance issues and add latency. Commit 5640f7685831e0
> introduces the order-3 allocation. According to the changelog, the
> order-3
> allocation isn't a must-have but to improve performance. But direct
> memory
> compaction has high overhead. The benefit of order-3 allocation can't
> compensate the overhead of direct memory compaction.
>
> This patch makes the order-3 page allocation atomic. If there is no
> memory
> pressure and memory isn't fragmented, the alloction will still success,
> so we
> don't sacrifice the order-3 benefit here. If the atomic allocation
> fails,
> direct memory compaction will not be triggered, skb_page_frag_refill
> will
> fallback to order-0 immediately, hence the direct memory compaction
> overhead is
>
> avoided. In the allocation failure case, kswapd is waken up and doing
> compaction, so chances are allocation could success next time.
>

So while you do avoid direct reclaim, you can still wake up non-direct
reclaim for callers than can wait, and this can happen very very
often. We've had the allocation order forced to 0 to workaround the
problem since the change was introduced, it doesn't really have much
impact for us since it's almost an impossibility to fill en-masse
order-3 allocations on one of our real-world machines. So if it ever
goes through the __GFP_WAIT path, compact/reclaim cycles happen
indirectly and still take up CPU cycles, without making enough forward
progress for it to matter. This seems to me to be an intractable
problem, unless we find a way to make sure all kernel pages are
movable/compactible.

-Deb