[Devel] [PATCH rh7 00/19] netfilter/conntrack: use a single hashtable for all namespaces

[Konstantin Khorenko]

Committed to 3.10.0-1127.8.2.vz7.151.4 kernel.

--
Best regards,

Konstantin Khorenko,
Virtuozzo Linux Kernel Team

On 05/22/2020 11:10 AM, Konstantin Khorenko wrote:
> There is some software which creates new net namespaces often,
> for example:
>  - "very secure FTP daemon" (vsftpd) - be default create a new
>     network namespace per connection.
>  - "phpsessionclean.service" (ionclean) - native php mechanism for old sessions
>     garbage collection, if "PrivateNetwork=true" config is set (also default).
>
> Currently a netns creation triggers new conntrack hashtable allocation
> (65K), which is heavy.
>
> The current patchset makes all namespaces to use a single hash for
> conntracks (and similarly nat bysrc hash).
>
> Testing results:
>
>    // unfixed kernel
>    # free; time for i in `seq 1 1000`; do ip netns add a$i; done; free
>                  total        used        free      shared  buff/cache   available
>    Mem:        8008232      328524     7456876         776      222832     7443024
>    Swap:       8127484           0     8127484
>    real    0m32.014s
>    user    0m0.227s
>    sys     0m30.453s
>                  total        used        free      shared  buff/cache   available
>    Mem:        8008232     1469060     6282088         776      257084     6287440
>    Swap:       8127484           0     8127484
>
> creating 1000 netns via "ip" utility.
> It takes 32 sec and eats > 1Gb RAM.
>
>    // the kernel with current patchset
>
>    # free; time for i in `seq 1 1000`; do ip netns add a$i; done; free
>                  total        used        free      shared  buff/cache   available
>    Mem:        8008080      260508     6905880         744      841692     7507772
>    Swap:       8127484           0     8127484
>    real    0m3.589s
>    user    0m0.180s
>    sys     0m3.157s
>                  total        used        free      shared  buff/cache   available
>    Mem:        8008080      619232     6530296         744      858552     7140864
>    Swap:       8127484           0     8127484
>
> 3(!) sec and only 350 Mb used!
>
> Test was run on a VM (just rebooted) with 8Gb RAM, so the memory is not
> fragmented and not close to the limit.
>
> https://jira.sw.ru/browse/PSBM-103515
>
> Andrey Ryabinin (1):
>   ms/netfilter: nf_conntrack: Fix possible possible crash on module
>     loading.
>
> Florian Westphal (14):
>   ms/netfilter: conntrack: don't attempt to iterate over empty table
>   ms/netfilter: conntrack: use nf_ct_key_equal() in more places
>   ms/netfilter: conntrack: small refactoring of conntrack seq_printf
>   ms/netfilter: conntrack: check netns when comparing conntrack objects
>   ms/netfilter: conntrack: make netns address part of hash
>   ms/netfilter: conntrack: use a single hashtable for all namespaces
>   ms/netfilter: conntrack: consider ct netns in early_drop logic
>   ms/netfilter: conntrack: check netns when walking expect hash
>   ms/netfilter: conntrack: use get_random_once for nat and expectations
>   ms/netfilter: conntrack: make netns address part of expect hash
>   ms/netfilter: conntrack: use a single expectation table for all
>     namespaces
>   ms/netfilter: conntrack: make netns address part of nat bysrc hash
>   ms/netfilter: conntrack: use a single nat bysource table for all
>     namespaces
>   ms/netfilter: conntrack: use single slab cache
>
> Konstantin Khorenko (1):
>   revert RH patch
>     9837-net-netfilter-nf_conntrack-don-t-resize-NULL-or-free.patch
>
> Liping Zhang (1):
>   ms/netfilter: conntrack: do not dump other netns's conntrack entries
>     via proc
>
> Vasily Khoruzhick (1):
>   ms/netfilter: conntrack: fix calculation of next bucket number in
>     early_drop
>
> Zhang Chunyu (1):
>   netfilter: xt_MARK: Add ARP support
>
>  include/net/netfilter/nf_conntrack_core.h          |   2 +
>  include/net/netfilter/nf_conntrack_expect.h        |   1 +
>  include/net/netns/conntrack.h                      |   9 -
>  include/net/netns/hash.h                           |   2 +-
>  net/ipv4/netfilter/nf_conntrack_l3proto_ipv4.c     |   2 +-
>  .../netfilter/nf_conntrack_l3proto_ipv4_compat.c   |  47 ++--
>  net/netfilter/nf_conntrack_core.c                  | 255 ++++++++++-----------
>  net/netfilter/nf_conntrack_expect.c                |  83 ++++---
>  net/netfilter/nf_conntrack_helper.c                |   8 +-
>  net/netfilter/nf_conntrack_netlink.c               |  29 ++-
>  net/netfilter/nf_conntrack_standalone.c            |  16 +-
>  net/netfilter/nf_nat_core.c                        |  44 ++--
>  net/netfilter/nfnetlink_cttimeout.c                |   6 +-
>  net/netfilter/xt_mark.c                            |   1 +
>  14 files changed, 265 insertions(+), 240 deletions(-)
>