[Users] Re: TCP packets lost?

[Max Deineko]

Vitaliy Gusev <vgusev at sw.ru> wrote:
> On 7 January 2008 13:01:24 Max Deineko wrote:
>> > On Mon, Jan 07, 2008 at 07:55:43AM +0000, Max Deineko wrote:
>> >> My first guess would have been that when the server is busy
>> >> serving an expensive request, the tcp rcv buffer would get full
>> >> and packets would get dropped.  But wouldn't tcpdump not be able
>> >> to see them then?

>> I already [...] tried increasing tcp_rmem size, which hung the
>> webserver completely (maybe it was a litte too big ;), but I'm still
>> experimenting with the settings.
>>
>> Of course there might be a problem with other server components, and
>> I'm not really a networking expert, it's just that the tcp flow
>> doesn't look like what I'd expect it to.

> Please show files: /proc/net/netstat, /proc/net/snmp ( inside VE)

See below.  Currently running with

  net.ipv4.tcp_rmem = 4096        87380   184762
  net.core.rmem_max = 184761

> Have you seen any suspicious messages in dmesg related to this issue?

Indeed, dmesg shows

  TCP: Treason uncloaked! Peer 91.4.192.15:52306/80 shrinks window 204385904:204399829. Repaired.
  TCP: Treason uncloaked! Peer 91.4.192.15:52306/80 shrinks window 204385904:204399829. Repaired.
  TCP: Treason uncloaked! Peer 91.4.192.15:52306/80 shrinks window 204385904:204399829. Repaired.
  TCP: Treason uncloaked! Peer 62.158.79.38:1883/80 shrinks window 1835842591:1835855659. Repaired.
  TCP: Treason uncloaked! Peer 91.12.168.104:2918/80 shrinks window 2946649572:2946651606. Repaired.
  TCP: Treason uncloaked! Peer 217.238.252.24:49584/80 shrinks window 3063558791:3063563147. Repaired.
  TCP: Treason uncloaked! Peer 87.167.76.47:52766/80 shrinks window 867053615:867090259. Repaired.
  TCP: Treason uncloaked! Peer 91.11.116.53:2475/80 shrinks window 3699395483:3699409070. Repaired.
  TCP: Treason uncloaked! Peer 91.11.116.53:2476/80 shrinks window 3699833370:3699846438. Repaired.
  TCP: Treason uncloaked! Peer 91.11.116.53:2475/80 shrinks window 3699403648:3699409070. Repaired.
  TCP: Treason uncloaked! Peer 91.11.116.53:2476/80 shrinks window 3699841562:3699846438. Repaired.
  TCP: Treason uncloaked! Peer 79.212.215.179:2859/80 shrinks window 400801346:400802798. Repaired.
  TCP: Treason uncloaked! Peer 80.139.188.46:3646/80 shrinks window 156197829:156205177. Repaired.
  TCP: Treason uncloaked! Peer 80.139.188.46:3646/80 shrinks window 156197829:156205177. Repaired.
  TCP: Treason uncloaked! Peer 80.140.219.65:49793/80 shrinks window 4253417104:4253432263. Repaired.
  TCP: Treason uncloaked! Peer 80.140.219.65:49793/80 shrinks window 4253417104:4253432263. Repaired.
  TCP: Treason uncloaked! Peer 80.140.219.65:49793/80 shrinks window 4253417104:4253432263. Repaired.

That's from the hardware node after running for three days.  I'm sorry I
cannot relate the messages to the dropout times - the messages do not
show up in the syslogs (yet - I'm new to the server).

Thanks & regards - Max.


# cat /proc/net/netstat

TcpExt: SyncookiesSent SyncookiesRecv SyncookiesFailed EmbryonicRsts PruneCalled RcvPruned OfoPruned OutOfWindowIcmps LockDroppedIcmps ArpFilter TW TWRecycled TWKilled PAWSPassive PAWSActive PAWSEstab DelayedACKs DelayedACKLocked DelayedACKLost ListenOverflows ListenDrops TCPPrequeued TCPDirectCopyFromBacklog TCPDirectCopyFromPrequeue TCPPrequeueDropped TCPHPHits TCPHPHitsToUser TCPPureAcks TCPHPAcks TCPRenoRecovery TCPSackRecovery TCPSACKReneging TCPFACKReorder TCPSACKReorder TCPRenoReorder TCPTSReorder TCPFullUndo TCPPartialUndo TCPDSACKUndo TCPLossUndo TCPLoss TCPLostRetransmit TCPRenoFailures TCPSackFailures TCPLossFailures TCPFastRetrans TCPForwardRetrans TCPSlowStartRetrans TCPTimeouts TCPRenoRecoveryFail TCPSackRecoveryFail TCPSchedulerFailed TCPRcvCollapsed TCPDSACKOldSent TCPDSACKOfoSent TCPDSACKRecv TCPDSACKOfoRecv TCPAbortOnSyn TCPAbortOnData TCPAbortOnClose TCPAbortOnMemory TCPAbortOnTimeout TCPAbortOnLinger TCPAbortFailed TCPMemoryPressures
TcpExt: 0 0 0 15666 390 7 0 3 0 0 0 0 0 0 0 350 518998 283 5722 6044 6044 13385260 860306 145918454 0 13263422 87351 9290224 12159160 1576 60817 31 631 240 92 198 367 2815 1284 1383 34266 20 369 13804 1501 90841 8390 24116 28574 309 7175 2 12934 5823 2 14577 291 0 751 5077 0 2285 0 0 0

# cat /proc/net/snmp

Ip: Forwarding DefaultTTL InReceives InHdrErrors InAddrErrors ForwDatagrams InUnknownProtos InDiscards InDelivers OutRequests OutDiscards OutNoRoutes ReasmTimeout ReasmReqds ReasmOKs ReasmFails FragOKs FragFails FragCreates
Ip: 2 64 55086729 0 0 0 0 0 55079918 59160345 0 0 22 64 0 22 0 0 0
Icmp: InMsgs InErrors InDestUnreachs InTimeExcds InParmProbs InSrcQuenchs InRedirects InEchos InEchoReps InTimestamps InTimestampReps InAddrMasks InAddrMaskReps OutMsgs OutErrors OutDestUnreachs OutTimeExcds OutParmProbs OutSrcQuenchs OutRedirects OutEchos OutEchoReps OutTimestamps OutTimestampReps OutAddrMasks OutAddrMaskReps
Icmp: 10852 99 2358 0 0 2 0 8401 0 0 0 0 0 8472 0 71 0 0 0 0 0 8401 0 0 0 0
Tcp: RtoAlgorithm RtoMin RtoMax MaxConn ActiveOpens PassiveOpens AttemptFails EstabResets CurrEstab InSegs OutSegs RetransSegs InErrs OutRsts
Tcp: 0 0 0 0 487897 1752675 1326 25489 45 55038781 59121501 195520 397 7118
Udp: InDatagrams NoPorts InErrors OutDatagrams
Udp: 30214 71 0 30372