[Devel] [PATCH rh7 00/19] netfilter/conntrack: use a single hashtable for all namespaces

[Konstantin Khorenko]

There is some software which creates new net namespaces often,
for example:
 - "very secure FTP daemon" (vsftpd) - be default create a new
    network namespace per connection.
 - "phpsessionclean.service" (ionclean) - native php mechanism for old sessions
    garbage collection, if "PrivateNetwork=true" config is set (also default).

Currently a netns creation triggers new conntrack hashtable allocation
(65K), which is heavy.

The current patchset makes all namespaces to use a single hash for
conntracks (and similarly nat bysrc hash).

Testing results:

   // unfixed kernel
   # free; time for i in `seq 1 1000`; do ip netns add a$i; done; free
                 total        used        free      shared  buff/cache   available
   Mem:        8008232      328524     7456876         776      222832     7443024
   Swap:       8127484           0     8127484
   real    0m32.014s
   user    0m0.227s
   sys     0m30.453s
                 total        used        free      shared  buff/cache   available
   Mem:        8008232     1469060     6282088         776      257084     6287440
   Swap:       8127484           0     8127484

creating 1000 netns via "ip" utility.
It takes 32 sec and eats > 1Gb RAM.

   // the kernel with current patchset

   # free; time for i in `seq 1 1000`; do ip netns add a$i; done; free
                 total        used        free      shared  buff/cache   available
   Mem:        8008080      260508     6905880         744      841692     7507772
   Swap:       8127484           0     8127484
   real    0m3.589s
   user    0m0.180s
   sys     0m3.157s
                 total        used        free      shared  buff/cache   available
   Mem:        8008080      619232     6530296         744      858552     7140864
   Swap:       8127484           0     8127484

3(!) sec and only 350 Mb used!

Test was run on a VM (just rebooted) with 8Gb RAM, so the memory is not
fragmented and not close to the limit.

https://jira.sw.ru/browse/PSBM-103515

Andrey Ryabinin (1):
  ms/netfilter: nf_conntrack: Fix possible possible crash on module
    loading.

Florian Westphal (14):
  ms/netfilter: conntrack: don't attempt to iterate over empty table
  ms/netfilter: conntrack: use nf_ct_key_equal() in more places
  ms/netfilter: conntrack: small refactoring of conntrack seq_printf
  ms/netfilter: conntrack: check netns when comparing conntrack objects
  ms/netfilter: conntrack: make netns address part of hash
  ms/netfilter: conntrack: use a single hashtable for all namespaces
  ms/netfilter: conntrack: consider ct netns in early_drop logic
  ms/netfilter: conntrack: check netns when walking expect hash
  ms/netfilter: conntrack: use get_random_once for nat and expectations
  ms/netfilter: conntrack: make netns address part of expect hash
  ms/netfilter: conntrack: use a single expectation table for all
    namespaces
  ms/netfilter: conntrack: make netns address part of nat bysrc hash
  ms/netfilter: conntrack: use a single nat bysource table for all
    namespaces
  ms/netfilter: conntrack: use single slab cache

Konstantin Khorenko (1):
  revert RH patch
    9837-net-netfilter-nf_conntrack-don-t-resize-NULL-or-free.patch

Liping Zhang (1):
  ms/netfilter: conntrack: do not dump other netns's conntrack entries
    via proc

Vasily Khoruzhick (1):
  ms/netfilter: conntrack: fix calculation of next bucket number in
    early_drop

Zhang Chunyu (1):
  netfilter: xt_MARK: Add ARP support

 include/net/netfilter/nf_conntrack_core.h          |   2 +
 include/net/netfilter/nf_conntrack_expect.h        |   1 +
 include/net/netns/conntrack.h                      |   9 -
 include/net/netns/hash.h                           |   2 +-
 net/ipv4/netfilter/nf_conntrack_l3proto_ipv4.c     |   2 +-
 .../netfilter/nf_conntrack_l3proto_ipv4_compat.c   |  47 ++--
 net/netfilter/nf_conntrack_core.c                  | 255 ++++++++++-----------
 net/netfilter/nf_conntrack_expect.c                |  83 ++++---
 net/netfilter/nf_conntrack_helper.c                |   8 +-
 net/netfilter/nf_conntrack_netlink.c               |  29 ++-
 net/netfilter/nf_conntrack_standalone.c            |  16 +-
 net/netfilter/nf_nat_core.c                        |  44 ++--
 net/netfilter/nfnetlink_cttimeout.c                |   6 +-
 net/netfilter/xt_mark.c                            |   1 +
 14 files changed, 265 insertions(+), 240 deletions(-)

-- 
2.15.1