[Users] Feeble large OpenVZ VPS linked apparently to common SYN_RECV floods

[Andrei Banu]

Hello,

I am having big problems with a fairly large VPS (32 threads x Intel(R) 
Xeon(R) CPU E5-2667 v2 @ 3.30GHz, 32GB RAM, RAID 10 HW SSDs) that is 
quite underloaded from a user perspective but very fragile. Any / many 
services start timing out under the smallest amount of SYN_RECV floods. 
I've checked this issue on a lot smaller metal servers and one KVM VPS 
that cope with similar or greater amounts of SYN_RECV floods much much 
better (with no apparent problems). Also this VM is the least used we 
have in terms of users and user load.

The VM is provisioned by SolusVM. I have checked their support but they 
said it's not a Solus problem and that I should ask for community 
support so here I am.

Environment:
Hardware: Dual Intel(R) Xeon(R) CPU E5-2667 v2 @ 3.30GHz, all allocated 
to this one VM that we also manage, 32GB RAM, solid RAID 10 SSD matrix.
Kernel: 2.6.32-042stab134.3
Node base distro: CentOS release 6.10 (Final)
VM Control panel: cPanel

What I've done (CSF):
CT_LIMIT: 300
SYNFLOOD_RATE: 50/s
SYNFLOOD_BURST: 50

I need to mention that this VM has always manifested a significant 
fragility even before the above mentioned CSF settings. If I recall 
correctly even around 10 SYN_RECV simultaneous connections (not 10/s) 
used to bring down (or render it inaccessible rather) the FTP server 
even in the somewhat distant past.

What I've checked:

1. /proc/user_beancounters: all the failcnt are on 0:
101:  kmemsize               4435446093 8822018048          17179869184 
17179869184                    0
             lockedpages                     1 52              
4194304              4194304                    0
             privvmpages               4993265             50280121 
9223372036854775807  9223372036854775807                    0
             shmpages                     1051              1212827 
9223372036854775807  9223372036854775807                    0
             dummy                           0                    0 
9223372036854775807  9223372036854775807                    0
             numproc                       482                 1446 
9223372036854775807  9223372036854775807                    0
             physpages                 7911397 
8409746                    0              8388608 0
             vmguarpages                     0                    0 
9223372036854775807  9223372036854775807                    0
             oomguarpages              3474241              8135512 
9223372036854775807  9223372036854775807                    0
             numtcpsock                    274                 1735 
9223372036854775807  9223372036854775807                    0
             numflock                     7747                 8307 
9223372036854775807  9223372036854775807                    0
             numpty                          1                    4 
9223372036854775807  9223372036854775807                    0
             numsiginfo                      1                  534 
9223372036854775807  9223372036854775807                    0
             tcpsndbuf                13382416            391330448 
9223372036854775807  9223372036854775807                    0
             tcprcvbuf                 4497936             61408352 
9223372036854775807  9223372036854775807                    0
             othersockbuf              1269256             32944336 
9223372036854775807  9223372036854775807                    0
             dgramrcvbuf                     0               582176 
9223372036854775807  9223372036854775807                    0
             numothersock                  619                 1526 
9223372036854775807  9223372036854775807                    0
             dcachesize             4314373277 8589934592           
8589934592 8589934592                    0
             numfile                     21753                35410 
9223372036854775807  9223372036854775807                    0
             dummy                           0                    0 
9223372036854775807  9223372036854775807                    0
             dummy                           0                    0 
9223372036854775807  9223372036854775807                    0
             dummy                           0                    0 
9223372036854775807  9223372036854775807                    0
             numiptent                    1576                 4575 
9223372036854775807  9223372036854775807

2. The port range is normal:
net.ipv4.ip_local_port_range = 32768    60999

3. SYN COOKIES were enabled some time ago:
net.ipv4.tcp_syncookies = 1

4. The connection count and limit:
net.netfilter.nf_conntrack_count = 902 (this does get to around 5000 
under SYN flood but this is way lower than what other much smaller 
servers can cope with)
net.netfilter.nf_conntrack_max = 65536

5. Other settings:
net.core.netdev_max_backlog = 1000
net.ipv4.tcp_max_syn_backlog = 2048

Please help me with some tips on what else to check. I have not been 
able to locate a single error in any log so I'm looking in the dark here.

Thanks,
Kind regards!