[Users] Strange hiccups for new socket connections in Debian Lenny using 2.6.26-2-openvz-amd64

[Pavel Shevaev]

Hi!

On a moderately busy server I've experienced a strange behavior - new
socket connections from remote hosts to the server running in VE were
established with large delays(3-5 seconds) from time to time. I've
noticed this in php-fpm slow logs - mysqli_connect popped very
frequently.

At first I thought it was MySQL to blame but it turned not to be the
case. I also tested remote connections to Redis and some other daemons
- and faced the same problem. Then I thought maybe I had too strict
limits for the container but /proc/user_beancounters showed no
failcounts. The host machine's dmesg and /var/log/messages also
revealed nothing. Just to be on the safe side I set almost unlimited
networking sysctl settings for the host server but nothing changed.

I wrote a simple PHP script which measured new connection time delays
and run it on the remote server. Here it is:

<?php
$last = 0;
while(true) {
 $t1 = microtime(true);
 $link = mysqli_connect('92.x.x.x', 'root', 'xxxxx');
 $ts = microtime(true)-$t1;
 if($ts > 0.5) {
   echo  $ts . "(" . (time()-$last) . ")\n";
   $last = time();
 }
 mysqli_close($link);
 usleep(1000);
}

Here's its output after running it about a minute on the remote server:

$ php conn.php
5.001708984375(7)
5.0034439563751(6)
4.9977340698242(5)
5.0048727989197(8)
5.0043451786041(6)
5.00546002388(6)
5.0057151317596(14)
5.0037190914154(16)
5.0062158107758(8)
5.0030419826508(7)

As you can see connection slow time is almost identical in each case -
about 5 seconds(the number in parenthesis shows how much time passed
in seconds after the previous hiccup).

I had to do something ASAP with this situation and I started crazy
experiments.... First I moved all VE daemons in question to the
hardware node, nothing changed. Second, I...changed the kernel back to
2.6.26-2-amd64 and all hiccups are now gone!

I really love OpenVZ and I really want to have it running on the
server. What would you recommend doing? Try another distribution with
more recent version of OpenVZ kernel?

-- 
Best regards, Pavel