[Users] occasional high loadavg without any noticeable cpu/memory/io load

Kirill Korotaev dev at parallels.com
Tue Jul 10 12:34:23 EDT 2012


I can take a look if you give me access to node.
If agree - send it privately, w/o users@ on CC.

Kirill


On Jul 10, 2012, at 18:40 , Rene C. wrote:

No takers for this one?

If I missed to provide any important information please let me know.  The issue happens regularly on several hardware nodes so if I missed anything I can check it next time it happens.

On Wed, Jul 4, 2012 at 4:16 PM, Rene C. <openvz at dokbua.com<mailto:openvz at dokbua.com>> wrote:
Today I again had a VE that went up to a relative high load for no apparent reason.

Below are the details for the hardware node, followed by the high-load container.

I realize it's not the latest kernel, but a reboot takes half an hour (from first VE goes down to last VE is back up, assuming everything goes well and no FSCK is forced) so we only reboot into new kernels when there is a really serious reason for it or the server crashes - but I don't see anything in the kernel updates since our current kernel that would address this issue anyway.

Why does the load in this container suddenly go up like that?  Websites hosted by the container becomes very sluggish, so it is a real problem.

It isn't just a problem with this container - or even this hardware node for that reason, I occasionally see it with containers on other hardware nodes as well.  One idea I brought up before was that perhaps it's the file system journal, as suggested in http://wiki.openvz.org/Ploop/Why - but I think that would affect all containers on that file system, not just a single container?

--- HARDWARE NODE ---

# uname -a
Linux server15.hardwarenode.com<http://server15.hardwarenode.com/> 2.6.32-042stab049.6 #1 SMP Mon Feb 6 19:17:43 MSK 2012 x86_64 x86_64 x86_64 GNU/Linux

# rpm -q sl-release
sl-release-6.1-2.x86_64

# top -cbn1 | head -17
top - 21:00:02 up 123 days, 15:31,  1 user,  load average: 0.97, 2.70, 2.37
Tasks: 886 total,   6 running, 880 sleeping,   0 stopped,   0 zombie
Cpu(s):  8.4%us,  1.7%sy,  0.0%ni, 86.3%id,  3.5%wa,  0.0%hi,  0.1%si,  0.0%st
Mem:  16420716k total, 15566264k used,   854452k free,  1477372k buffers
Swap: 16777184k total,   623672k used, 16153512k free,  4578176k cached

    PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
  94153 27        20   0  164m  41m 3392 S 150.9  0.3  50575:37 /usr/libexec/mys
   9178 27        20   0  159m  29m 3000 S 72.6  0.2   1284:50 /usr/libexec/mysq
 567031 apache    20   0 40296  15m 3588 S 17.2  0.1   0:00.09 /usr/sbin/httpd
 567382 root      20   0 15672 1820  864 R  5.7  0.0   0:00.04 top -cbn1
     38 root      20   0     0    0    0 S  1.9  0.0   2:55.25 [events/3]
     41 root      20   0     0    0    0 S  1.9  0.0   0:29.00 [events/6]
 566362 apache    20   0 43240  19m 4448 R  1.9  0.1   0:01.04 /usr/sbin/httpd
 566857 apache    20   0 55248  11m 3456 R  1.9  0.1   0:00.05 /usr/sbin/httpd
 566918 apache    20   0 42596  17m 3704 S  1.9  0.1   0:00.15 /usr/sbin/httpd
 567033 apache    20   0 39784  14m 3468 S  1.9  0.1   0:00.01 /usr/sbin/httpd

# vzlist -o ctid,laverage
      CTID       LAVERAGE
      1501 0.00/0.05/0.02
      1502 0.00/0.00/0.00
      1503 0.08/0.03/0.01
      1504 0.00/0.00/0.00
      1505 8.29/6.04/3.67
      1506 27.11/16.97/7.89
      1507 0.00/0.00/0.00
      1508 0.19/0.06/0.01
      1509 0.07/0.03/0.00
      1510 0.02/0.02/0.00
      1512 0.00/0.00/0.00
      1514 0.00/0.00/0.00

# iostat -xN
Linux 2.6.32-042stab049.6 (server15.hardwarenode.com<http://server15.hardwarenode.com/>)    07/03/12        _x86_64_        (8 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           8.41    0.04    1.75    3.51    0.00   86.28

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sdd               0.76    56.58    0.59    0.59    20.27   457.28   402.66     0.25  211.66   4.03   0.48
sdc               1.72    27.94   17.20   16.16   887.30   336.18    36.68     0.02   12.71   5.23  17.45
sdb               1.65    27.79   19.48   12.95   975.43   318.64    39.91     0.09   15.22   3.77  12.23
sda               0.01     0.16    0.10    0.24     1.95     2.79    13.79     0.00    7.06   4.16   0.14
vg01-swap         0.00     0.00    0.00    0.00     0.00     0.00     8.00     0.00    3.68   2.22   0.00
vg01-root         0.00     0.00    0.11    0.35     1.94     2.78    10.30     0.02   38.30   3.12   0.14
vg04-swap         0.00     0.00    1.30    0.22    10.41     1.80     8.00     0.01    9.28   1.44   0.22
vg04-vz           0.00     0.00    0.05   56.94     9.86   455.49     8.17     0.01    0.18   0.05   0.27
vg03-swap         0.00     0.00    0.00    0.00     0.00     0.00     8.00     0.00    6.72   1.10   0.00
vg03-vz           0.00     0.00   18.98   42.41   887.30   336.18    19.93     0.39    6.33   2.84  17.45
vg02-swap         0.00     0.00    0.00    0.00     0.00     0.00     8.00     0.00    7.03   0.89   0.00
vg02-vz           0.00     0.00   21.19   39.91   975.43   318.64    21.18     0.15    8.99   2.00  12.23
vg01-vz           0.00     0.00    0.00    0.00     0.00     0.00     7.98     0.00   17.73  17.73   0.00

--- CONTAINER ---

# top -cbn1 | head -100
top - 21:00:04 up 123 days, 15:25,  0 users,  load average: 27.11, 16.97, 7.89
Tasks:  86 total,   2 running,  84 sleeping,   0 stopped,   0 zombie
Cpu(s):  1.4%us,  0.2%sy,  0.0%ni, 98.1%id,  0.1%wa,  0.0%hi,  0.0%si,  0.2%st
Mem:    655360k total,   316328k used,   339032k free,        0k buffers
Swap:  1310720k total,    68380k used,  1242340k free,    58268k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
  916 mysql     20   0  159m  29m 3000 S 79.3  4.6   1284:51 /usr/libexec/mysqld
    1 root      20   0  2156   92   64 S  0.0  0.0   0:36.50 init [3]
    2 root      20   0     0    0    0 S  0.0  0.0   0:00.00 [kthreadd/1506]
    3 root      20   0     0    0    0 S  0.0  0.0   0:00.00 [khelper/1506]
   97 root      16  -4  2244    8    4 S  0.0  0.0   0:00.00 /sbin/udevd -d
  634 root      20   0  1812  212  136 S  0.0  0.0   2:39.88 syslogd -m 0
  667 root      20   0  7180  268  168 S  0.0  0.0   1:01.55 /usr/sbin/sshd
  676 root      20   0  2832  392  304 S  0.0  0.1   0:15.13 xinetd -stayalive -
  690 root      20   0  6040  124   72 S  0.0  0.0   0:02.45 /usr/lib/courier-im
  693 root      20   0  4872  252  200 S  0.0  0.0   0:01.94 /usr/sbin/courierlo
  701 root      20   0  6040  124   72 S  0.0  0.0   0:06.34 /usr/lib/courier-im
  703 root      20   0  4872  256  200 S  0.0  0.0   0:03.09 /usr/sbin/courierlo
  709 root      20   0  6040  128   72 S  0.0  0.0   0:18.15 /usr/lib/courier-im
  711 root      20   0  4872  256  200 S  0.0  0.0   0:09.15 /usr/sbin/courierlo
  718 root      20   0  6040  124   72 S  0.0  0.0   0:05.68 /usr/lib/courier-im
  720 root      20   0  4872  252  200 S  0.0  0.0   0:02.54 /usr/sbin/courierlo
  730 qmails    20   0  1796  224  144 S  0.0  0.0   1:27.21 qmail-send
  732 qmaill    20   0  1752  244  192 S  0.0  0.0   0:22.64 splogger qmail
  733 root      20   0  1780  140   64 S  0.0  0.0   0:07.85 qmail-lspawn | /usr
  734 qmailr    20   0  1776  148   76 S  0.0  0.0   0:14.07 qmail-rspawn
  735 qmailq    20   0  1748  104   68 S  0.0  0.0   0:14.01 qmail-clean
  781 root      20   0 51880 4364  196 S  0.0  0.7   1:35.02 /usr/sbin/httpd
  828 named     20   0 44104 5708 1112 S  0.0  0.9  10:10.53 /usr/sbin/named -u
  866 root      20   0  3708    8    4 S  0.0  0.0   0:00.00 /bin/sh /usr/bin/my
  981 root      20   0 33912 3756  916 S  0.0  0.6  10:55.30 /usr/bin/spamd --us
 1107 xfs       20   0  3392   72   40 S  0.0  0.0   0:00.09 xfs -droppriv -daem
 1115 root      20   0  5672    8    4 S  0.0  0.0   0:00.00 /usr/sbin/saslauthd
 1116 root      20   0  5672    8    4 S  0.0  0.0   0:00.00 /usr/sbin/saslauthd
 1122 root      20   0 22992 1868 1084 S  0.0  0.3   2:09.79 /usr/bin/sw-engine
 1123 root      20   0 27328 1508 1160 S  0.0  0.2   6:06.30 /usr/local/psa/admi
 7251 root      20   0  4488  192  136 S  0.0  0.0   0:22.85 crond
 9463 apache    20   0 59184  14m 4356 S  0.0  2.3   0:05.10 /usr/sbin/httpd
10512 apache    20   0 42316 2504   84 S  0.0  0.4   0:00.91 /usr/sbin/httpd
12090 apache    20   0 56964  14m 4492 S  0.0  2.2   0:04.48 /usr/sbin/httpd
12682 apache    20   0 61060  17m 4516 S  0.0  2.7   0:02.45 /usr/sbin/httpd
13870 sw-cp-se  20   0  7852 1932   16 S  0.0  0.3   1:19.03 /usr/sbin/sw-cp-ser
17443 apache    20   0 62416  17m 4436 S  0.0  2.7   0:05.27 /usr/sbin/httpd
17461 apache    20   0 52788  10m 4480 S  0.0  1.6   0:02.24 /usr/sbin/httpd
20430 apache    20   0 62164  17m 4356 S  0.0  2.7   0:04.25 /usr/sbin/httpd
23539 popuser   20   0 37612  25m 2328 S  0.0  3.9   0:01.50 spamd child
23924 apache    20   0 58004  15m 5536 S  0.0  2.4   0:01.56 /usr/sbin/httpd
26361 apache    20   0 54496  11m 3864 S  0.0  1.8   0:01.35 /usr/sbin/httpd
26366 apache    20   0 52944 9.8m 3892 S  0.0  1.5   0:01.45 /usr/sbin/httpd
26964 apache    20   0 59184  14m 4316 S  0.0  2.3   0:07.26 /usr/sbin/httpd
27096 apache    20   0 53728  10m 3868 S  0.0  1.6   0:00.33 /usr/sbin/httpd
27102 apache    20   0 54736  11m 3780 S  0.0  1.8   0:00.15 /usr/sbin/httpd
27103 apache    20   0 54480  11m 3784 S  0.0  1.7   0:00.11 /usr/sbin/httpd
27115 apache    20   0 57064  12m 3816 S  0.0  2.0   0:00.32 /usr/sbin/httpd
27118 apache    20   0 53728  10m 3884 S  0.0  1.6   0:01.21 /usr/sbin/httpd
27120 apache    20   0 52184 8376 3120 S  0.0  1.3   0:00.00 /usr/sbin/httpd
27129 apache    20   0 52168 8072 2960 S  0.0  1.2   0:00.00 /usr/sbin/httpd
27139 apache    20   0 53304 9840 3744 S  0.0  1.5   0:01.08 /usr/sbin/httpd
27140 apache    20   0 53000 9.8m 3832 S  0.0  1.5   0:00.66 /usr/sbin/httpd
27144 apache    20   0 52168 8072 2960 S  0.0  1.2   0:00.00 /usr/sbin/httpd
27147 apache    20   0 53252  12m 5536 S  0.0  1.9   0:00.50 /usr/sbin/httpd
27149 apache    20   0 52980 9924 3740 S  0.0  1.5   0:00.17 /usr/sbin/httpd
27153 apache    20   0 53728  10m 3836 S  0.0  1.6   0:00.49 /usr/sbin/httpd
27164 apache    20   0 55224  11m 3812 S  0.0  1.9   0:00.47 /usr/sbin/httpd
27171 apache    20   0 52916 9776 3708 S  0.0  1.5   0:00.16 /usr/sbin/httpd
27172 apache    20   0 52916 9452 3436 S  0.0  1.4   0:00.17 /usr/sbin/httpd
27173 apache    20   0 55340  11m 3720 S  0.0  1.8   0:00.08 /usr/sbin/httpd
27179 apache    20   0 52020 7764 2716 S  0.0  1.2   0:00.00 /usr/sbin/httpd
27182 apache    20   0 52020 7764 2716 S  0.0  1.2   0:00.00 /usr/sbin/httpd
27185 apache    20   0 55224  11m 3824 S  0.0  1.9   0:00.30 /usr/sbin/httpd
27186 apache    20   0 53788  10m 3840 S  0.0  1.7   0:00.11 /usr/sbin/httpd
27187 apache    20   0 52916 9448 3436 S  0.0  1.4   0:00.08 /usr/sbin/httpd
27188 apache    20   0 54628  10m 3504 S  0.0  1.7   0:00.05 /usr/sbin/httpd
27196 apache    20   0 53728  10m 3572 S  0.0  1.6   0:00.36 /usr/sbin/httpd
27200 apache    20   0 54628  11m 3796 S  0.0  1.7   0:00.05 /usr/sbin/httpd
27202 apache    20   0 54480  11m 3796 S  0.0  1.7   0:00.10 /usr/sbin/httpd
27204 apache    20   0 53992  10m 3544 S  0.0  1.6   0:00.09 /usr/sbin/httpd
27207 apache    20   0 52168 8084 2960 S  0.0  1.2   0:00.00 /usr/sbin/httpd
27213 apache    20   0 52020 6464 1788 S  0.0  1.0   0:00.00 /usr/sbin/httpd
27214 apache    20   0 54216  10m 3516 S  0.0  1.6   0:00.05 /usr/sbin/httpd
27215 apache    20   0 52020 6456 1788 S  0.0  1.0   0:00.00 /usr/sbin/httpd
27216 apache    20   0 52020 7860 2804 S  0.0  1.2   0:00.00 /usr/sbin/httpd
27218 root      20   0  9400 1900 1408 S  0.0  0.3   0:00.00 crond
27219 root      20   0  2492  956  848 S  0.0  0.1   0:00.00 /bin/sh -c /usr/loc
27220 root      20   0  2496 1052  920 S  0.0  0.2   0:00.00 /bin/sh /usr/local/
27233 root      20   0  2540 1016  892 S  0.0  0.2   0:00.00 /bin/bash -c top -c
27234 root      20   0  2284  952  724 R  0.0  0.1   0:00.00 top -cbn1
27235 root      20   0  1756  420  352 S  0.0  0.1   0:00.00 head -100
27247 root      20   0  2496  452  320 S  0.0  0.1   0:00.00 /bin/sh /usr/local/
27248 root      20   0  8280 1504 1120 R  0.0  0.2   0:00.00 /usr/bin/mysql -uad
27249 root      20   0  1800  448  376 S  0.0  0.1   0:00.00 sed -e 1d
27250 root      20   0  2240  640  540 S  0.0  0.1   0:00.00 awk {printf("%s", $

# netstat -ptan | grep ESTABLISHED
tcp        0      0 ::ffff:xx.xx.xx.xx:80   ::ffff:77.87.207.166:21863<http://77.87.207.166:21863/>  ESTABLISHED 23924/httpd
tcp        0      0 ::ffff:xx.xx.xx.xx:80   ::ffff:95.165.204.26:62259<http://95.165.204.26:62259/>  ESTABLISHED 27144/httpd
tcp        0      0 ::ffff:xx.xx.xx.xx:80   ::ffff:193.151.105.100:4059<http://193.151.105.100:4059/> ESTABLISHED 27200/httpd
tcp        0      0 ::ffff:xx.xx.xx.xx:80   ::ffff:109.169.207.68:50087<http://109.169.207.68:50087/> ESTABLISHED 27185/httpd
tcp        0      0 ::ffff:xx.xx.xx.xx:80   ::ffff:31.131.70.135:57017<http://31.131.70.135:57017/>  ESTABLISHED 27179/httpd
tcp        0      0 ::ffff:xx.xx.xx.xx:80   ::ffff:95.165.204.26:62220<http://95.165.204.26:62220/>  ESTABLISHED 27103/httpd
tcp        0      0 ::ffff:xx.xx.xx.xx:80   ::ffff:188.134.61.1:60732<http://188.134.61.1:60732/>   ESTABLISHED 27215/httpd
tcp        0      0 ::ffff:xx.xx.xx.xx:80   ::ffff:193.151.105.100:4112<http://193.151.105.100:4112/> ESTABLISHED 26964/httpd
tcp        0      0 ::ffff:xx.xx.xx.xx:80   ::ffff:109.169.207.68:50043<http://109.169.207.68:50043/> ESTABLISHED 27164/httpd
tcp        0      0 ::ffff:xx.xx.xx.xx:80   ::ffff:31.131.70.135:56976<http://31.131.70.135:56976/>  ESTABLISHED 27153/httpd

# cat /proc/user_beancounters
Version: 2.5
       uid  resource                     held              maxheld              barrier                limit              failcnt
     1506:  kmemsize                 27735306            179081216            304087040            335544320                    0
            lockedpages                     0                    0                81920                81920                    0
            privvmpages                393683               430195  9223372036854775807  9223372036854775807                    0
            shmpages                      823                21639  9223372036854775807  9223372036854775807                    0
            dummy                           0                    0                    0                    0                    0
            numproc                       128                  204  9223372036854775807  9223372036854775807                    0
            physpages                   79702               163840                    0               163840                    0
            vmguarpages                     0                    0                    0  9223372036854775807                    0
            oomguarpages                74734                75707                    0  9223372036854775807                    0
            numtcpsock                     59                  153  9223372036854775807  9223372036854775807                    0
            numflock                       46                   62  9223372036854775807  9223372036854775807                    0
            numpty                          0                    1  9223372036854775807  9223372036854775807                    0
            numsiginfo                      0                   33  9223372036854775807  9223372036854775807                    0
            tcpsndbuf                 1037680             11426176  9223372036854775807  9223372036854775807                    0
            tcprcvbuf                  966656              2867584  9223372036854775807  9223372036854775807                    0
            othersockbuf                53824               838688  9223372036854775807  9223372036854775807                    0
            dgramrcvbuf                     0               502224  9223372036854775807  9223372036854775807                    0
            numothersock                  114                  273  9223372036854775807  9223372036854775807                    0
            dcachesize               10070617            167772160            150994944            167772160                    0
            numfile                      1634                 1865  9223372036854775807  9223372036854775807                    0
            dummy                           0                    0                    0                    0                    0
            dummy                           0                    0                    0                    0                    0
            dummy                           0                    0                    0                    0                    0
            numiptent                      20                   20  9223372036854775807  9223372036854775807                    0

<ATT00001.c>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://openvz.org/pipermail/users/attachments/20120710/5ab8cccf/attachment-0001.html


More information about the Users mailing list