[Users] occasional high loadavg without any noticeable cpu/memory/io load

Rene C. openvz at dokbua.com
Tue Jul 10 10:40:17 EDT 2012


No takers for this one?

If I missed to provide any important information please let me know.  The
issue happens regularly on several hardware nodes so if I missed anything I
can check it next time it happens.

On Wed, Jul 4, 2012 at 4:16 PM, Rene C. <openvz at dokbua.com> wrote:

> Today I again had a VE that went up to a relative high load for no
> apparent reason.
>
> Below are the details for the hardware node, followed by the high-load
> container.
>
> I realize it's not the latest kernel, but a reboot takes half an hour
> (from first VE goes down to last VE is back up, assuming everything goes
> well and no FSCK is forced) so we only reboot into new kernels when there
> is a really serious reason for it or the server crashes - but I don't see
> anything in the kernel updates since our current kernel that would address
> this issue anyway.
>
> Why does the load in this container suddenly go up like that?  Websites
> hosted by the container becomes very sluggish, so it is a real problem.
>
> It isn't just a problem with this container - or even this hardware node
> for that reason, I occasionally see it with containers on other hardware
> nodes as well.  One idea I brought up before was that perhaps it's the file
> system journal, as suggested in http://wiki.openvz.org/Ploop/Why - but I
> think that would affect all containers on that file system, not just a
> single container?
>
> --- HARDWARE NODE ---
>
> # uname -a
> Linux server15.hardwarenode.com 2.6.32-042stab049.6 #1 SMP Mon Feb 6
> 19:17:43 MSK 2012 x86_64 x86_64 x86_64 GNU/Linux
>
> # rpm -q sl-release
> sl-release-6.1-2.x86_64
>
> # top -cbn1 | head -17
> top - 21:00:02 up 123 days, 15:31,  1 user,  load average: 0.97, 2.70, 2.37
> Tasks: 886 total,   6 running, 880 sleeping,   0 stopped,   0 zombie
> Cpu(s):  8.4%us,  1.7%sy,  0.0%ni, 86.3%id,  3.5%wa,  0.0%hi,  0.1%si,
>  0.0%st
> Mem:  16420716k total, 15566264k used,   854452k free,  1477372k buffers
> Swap: 16777184k total,   623672k used, 16153512k free,  4578176k cached
>
>     PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
>   94153 27        20   0  164m  41m 3392 S 150.9  0.3  50575:37
> /usr/libexec/mys
>    9178 27        20   0  159m  29m 3000 S 72.6  0.2   1284:50
> /usr/libexec/mysq
>  567031 apache    20   0 40296  15m 3588 S 17.2  0.1   0:00.09
> /usr/sbin/httpd
>  567382 root      20   0 15672 1820  864 R  5.7  0.0   0:00.04 top -cbn1
>      38 root      20   0     0    0    0 S  1.9  0.0   2:55.25 [events/3]
>      41 root      20   0     0    0    0 S  1.9  0.0   0:29.00 [events/6]
>  566362 apache    20   0 43240  19m 4448 R  1.9  0.1   0:01.04
> /usr/sbin/httpd
>  566857 apache    20   0 55248  11m 3456 R  1.9  0.1   0:00.05
> /usr/sbin/httpd
>  566918 apache    20   0 42596  17m 3704 S  1.9  0.1   0:00.15
> /usr/sbin/httpd
>  567033 apache    20   0 39784  14m 3468 S  1.9  0.1   0:00.01
> /usr/sbin/httpd
>
> # vzlist -o ctid,laverage
>       CTID       LAVERAGE
>       1501 0.00/0.05/0.02
>       1502 0.00/0.00/0.00
>       1503 0.08/0.03/0.01
>       1504 0.00/0.00/0.00
>       1505 8.29/6.04/3.67
>       1506 27.11/16.97/7.89
>       1507 0.00/0.00/0.00
>       1508 0.19/0.06/0.01
>       1509 0.07/0.03/0.00
>       1510 0.02/0.02/0.00
>       1512 0.00/0.00/0.00
>       1514 0.00/0.00/0.00
>
> # iostat -xN
> Linux 2.6.32-042stab049.6 (server15.hardwarenode.com)    07/03/12
>  _x86_64_        (8 CPU)
>
> avg-cpu:  %user   %nice %system %iowait  %steal   %idle
>            8.41    0.04    1.75    3.51    0.00   86.28
>
> Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz
> avgqu-sz   await  svctm  %util
> sdd               0.76    56.58    0.59    0.59    20.27   457.28   402.66
>     0.25  211.66   4.03   0.48
> sdc               1.72    27.94   17.20   16.16   887.30   336.18    36.68
>     0.02   12.71   5.23  17.45
> sdb               1.65    27.79   19.48   12.95   975.43   318.64    39.91
>     0.09   15.22   3.77  12.23
> sda               0.01     0.16    0.10    0.24     1.95     2.79    13.79
>     0.00    7.06   4.16   0.14
> vg01-swap         0.00     0.00    0.00    0.00     0.00     0.00     8.00
>     0.00    3.68   2.22   0.00
> vg01-root         0.00     0.00    0.11    0.35     1.94     2.78    10.30
>     0.02   38.30   3.12   0.14
> vg04-swap         0.00     0.00    1.30    0.22    10.41     1.80     8.00
>     0.01    9.28   1.44   0.22
> vg04-vz           0.00     0.00    0.05   56.94     9.86   455.49     8.17
>     0.01    0.18   0.05   0.27
> vg03-swap         0.00     0.00    0.00    0.00     0.00     0.00     8.00
>     0.00    6.72   1.10   0.00
> vg03-vz           0.00     0.00   18.98   42.41   887.30   336.18    19.93
>     0.39    6.33   2.84  17.45
> vg02-swap         0.00     0.00    0.00    0.00     0.00     0.00     8.00
>     0.00    7.03   0.89   0.00
> vg02-vz           0.00     0.00   21.19   39.91   975.43   318.64    21.18
>     0.15    8.99   2.00  12.23
> vg01-vz           0.00     0.00    0.00    0.00     0.00     0.00     7.98
>     0.00   17.73  17.73   0.00
>
> --- CONTAINER ---
>
> # top -cbn1 | head -100
> top - 21:00:04 up 123 days, 15:25,  0 users,  load average: 27.11, 16.97,
> 7.89
> Tasks:  86 total,   2 running,  84 sleeping,   0 stopped,   0 zombie
> Cpu(s):  1.4%us,  0.2%sy,  0.0%ni, 98.1%id,  0.1%wa,  0.0%hi,  0.0%si,
>  0.2%st
> Mem:    655360k total,   316328k used,   339032k free,        0k buffers
> Swap:  1310720k total,    68380k used,  1242340k free,    58268k cached
>
>   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
>   916 mysql     20   0  159m  29m 3000 S 79.3  4.6   1284:51
> /usr/libexec/mysqld
>     1 root      20   0  2156   92   64 S  0.0  0.0   0:36.50 init [3]
>     2 root      20   0     0    0    0 S  0.0  0.0   0:00.00
> [kthreadd/1506]
>     3 root      20   0     0    0    0 S  0.0  0.0   0:00.00 [khelper/1506]
>    97 root      16  -4  2244    8    4 S  0.0  0.0   0:00.00 /sbin/udevd -d
>   634 root      20   0  1812  212  136 S  0.0  0.0   2:39.88 syslogd -m 0
>   667 root      20   0  7180  268  168 S  0.0  0.0   1:01.55 /usr/sbin/sshd
>   676 root      20   0  2832  392  304 S  0.0  0.1   0:15.13 xinetd
> -stayalive -
>   690 root      20   0  6040  124   72 S  0.0  0.0   0:02.45
> /usr/lib/courier-im
>   693 root      20   0  4872  252  200 S  0.0  0.0   0:01.94
> /usr/sbin/courierlo
>   701 root      20   0  6040  124   72 S  0.0  0.0   0:06.34
> /usr/lib/courier-im
>   703 root      20   0  4872  256  200 S  0.0  0.0   0:03.09
> /usr/sbin/courierlo
>   709 root      20   0  6040  128   72 S  0.0  0.0   0:18.15
> /usr/lib/courier-im
>   711 root      20   0  4872  256  200 S  0.0  0.0   0:09.15
> /usr/sbin/courierlo
>   718 root      20   0  6040  124   72 S  0.0  0.0   0:05.68
> /usr/lib/courier-im
>   720 root      20   0  4872  252  200 S  0.0  0.0   0:02.54
> /usr/sbin/courierlo
>   730 qmails    20   0  1796  224  144 S  0.0  0.0   1:27.21 qmail-send
>   732 qmaill    20   0  1752  244  192 S  0.0  0.0   0:22.64 splogger qmail
>   733 root      20   0  1780  140   64 S  0.0  0.0   0:07.85 qmail-lspawn
> | /usr
>   734 qmailr    20   0  1776  148   76 S  0.0  0.0   0:14.07 qmail-rspawn
>   735 qmailq    20   0  1748  104   68 S  0.0  0.0   0:14.01 qmail-clean
>   781 root      20   0 51880 4364  196 S  0.0  0.7   1:35.02
> /usr/sbin/httpd
>   828 named     20   0 44104 5708 1112 S  0.0  0.9  10:10.53
> /usr/sbin/named -u
>   866 root      20   0  3708    8    4 S  0.0  0.0   0:00.00 /bin/sh
> /usr/bin/my
>   981 root      20   0 33912 3756  916 S  0.0  0.6  10:55.30
> /usr/bin/spamd --us
>  1107 xfs       20   0  3392   72   40 S  0.0  0.0   0:00.09 xfs -droppriv
> -daem
>  1115 root      20   0  5672    8    4 S  0.0  0.0   0:00.00
> /usr/sbin/saslauthd
>  1116 root      20   0  5672    8    4 S  0.0  0.0   0:00.00
> /usr/sbin/saslauthd
>  1122 root      20   0 22992 1868 1084 S  0.0  0.3   2:09.79
> /usr/bin/sw-engine
>  1123 root      20   0 27328 1508 1160 S  0.0  0.2   6:06.30
> /usr/local/psa/admi
>  7251 root      20   0  4488  192  136 S  0.0  0.0   0:22.85 crond
>  9463 apache    20   0 59184  14m 4356 S  0.0  2.3   0:05.10
> /usr/sbin/httpd
> 10512 apache    20   0 42316 2504   84 S  0.0  0.4   0:00.91
> /usr/sbin/httpd
> 12090 apache    20   0 56964  14m 4492 S  0.0  2.2   0:04.48
> /usr/sbin/httpd
> 12682 apache    20   0 61060  17m 4516 S  0.0  2.7   0:02.45
> /usr/sbin/httpd
> 13870 sw-cp-se  20   0  7852 1932   16 S  0.0  0.3   1:19.03
> /usr/sbin/sw-cp-ser
> 17443 apache    20   0 62416  17m 4436 S  0.0  2.7   0:05.27
> /usr/sbin/httpd
> 17461 apache    20   0 52788  10m 4480 S  0.0  1.6   0:02.24
> /usr/sbin/httpd
> 20430 apache    20   0 62164  17m 4356 S  0.0  2.7   0:04.25
> /usr/sbin/httpd
> 23539 popuser   20   0 37612  25m 2328 S  0.0  3.9   0:01.50 spamd child
> 23924 apache    20   0 58004  15m 5536 S  0.0  2.4   0:01.56
> /usr/sbin/httpd
> 26361 apache    20   0 54496  11m 3864 S  0.0  1.8   0:01.35
> /usr/sbin/httpd
> 26366 apache    20   0 52944 9.8m 3892 S  0.0  1.5   0:01.45
> /usr/sbin/httpd
> 26964 apache    20   0 59184  14m 4316 S  0.0  2.3   0:07.26
> /usr/sbin/httpd
> 27096 apache    20   0 53728  10m 3868 S  0.0  1.6   0:00.33
> /usr/sbin/httpd
> 27102 apache    20   0 54736  11m 3780 S  0.0  1.8   0:00.15
> /usr/sbin/httpd
> 27103 apache    20   0 54480  11m 3784 S  0.0  1.7   0:00.11
> /usr/sbin/httpd
> 27115 apache    20   0 57064  12m 3816 S  0.0  2.0   0:00.32
> /usr/sbin/httpd
> 27118 apache    20   0 53728  10m 3884 S  0.0  1.6   0:01.21
> /usr/sbin/httpd
> 27120 apache    20   0 52184 8376 3120 S  0.0  1.3   0:00.00
> /usr/sbin/httpd
> 27129 apache    20   0 52168 8072 2960 S  0.0  1.2   0:00.00
> /usr/sbin/httpd
> 27139 apache    20   0 53304 9840 3744 S  0.0  1.5   0:01.08
> /usr/sbin/httpd
> 27140 apache    20   0 53000 9.8m 3832 S  0.0  1.5   0:00.66
> /usr/sbin/httpd
> 27144 apache    20   0 52168 8072 2960 S  0.0  1.2   0:00.00
> /usr/sbin/httpd
> 27147 apache    20   0 53252  12m 5536 S  0.0  1.9   0:00.50
> /usr/sbin/httpd
> 27149 apache    20   0 52980 9924 3740 S  0.0  1.5   0:00.17
> /usr/sbin/httpd
> 27153 apache    20   0 53728  10m 3836 S  0.0  1.6   0:00.49
> /usr/sbin/httpd
> 27164 apache    20   0 55224  11m 3812 S  0.0  1.9   0:00.47
> /usr/sbin/httpd
> 27171 apache    20   0 52916 9776 3708 S  0.0  1.5   0:00.16
> /usr/sbin/httpd
> 27172 apache    20   0 52916 9452 3436 S  0.0  1.4   0:00.17
> /usr/sbin/httpd
> 27173 apache    20   0 55340  11m 3720 S  0.0  1.8   0:00.08
> /usr/sbin/httpd
> 27179 apache    20   0 52020 7764 2716 S  0.0  1.2   0:00.00
> /usr/sbin/httpd
> 27182 apache    20   0 52020 7764 2716 S  0.0  1.2   0:00.00
> /usr/sbin/httpd
> 27185 apache    20   0 55224  11m 3824 S  0.0  1.9   0:00.30
> /usr/sbin/httpd
> 27186 apache    20   0 53788  10m 3840 S  0.0  1.7   0:00.11
> /usr/sbin/httpd
> 27187 apache    20   0 52916 9448 3436 S  0.0  1.4   0:00.08
> /usr/sbin/httpd
> 27188 apache    20   0 54628  10m 3504 S  0.0  1.7   0:00.05
> /usr/sbin/httpd
> 27196 apache    20   0 53728  10m 3572 S  0.0  1.6   0:00.36
> /usr/sbin/httpd
> 27200 apache    20   0 54628  11m 3796 S  0.0  1.7   0:00.05
> /usr/sbin/httpd
> 27202 apache    20   0 54480  11m 3796 S  0.0  1.7   0:00.10
> /usr/sbin/httpd
> 27204 apache    20   0 53992  10m 3544 S  0.0  1.6   0:00.09
> /usr/sbin/httpd
> 27207 apache    20   0 52168 8084 2960 S  0.0  1.2   0:00.00
> /usr/sbin/httpd
> 27213 apache    20   0 52020 6464 1788 S  0.0  1.0   0:00.00
> /usr/sbin/httpd
> 27214 apache    20   0 54216  10m 3516 S  0.0  1.6   0:00.05
> /usr/sbin/httpd
> 27215 apache    20   0 52020 6456 1788 S  0.0  1.0   0:00.00
> /usr/sbin/httpd
> 27216 apache    20   0 52020 7860 2804 S  0.0  1.2   0:00.00
> /usr/sbin/httpd
> 27218 root      20   0  9400 1900 1408 S  0.0  0.3   0:00.00 crond
> 27219 root      20   0  2492  956  848 S  0.0  0.1   0:00.00 /bin/sh -c
> /usr/loc
> 27220 root      20   0  2496 1052  920 S  0.0  0.2   0:00.00 /bin/sh
> /usr/local/
> 27233 root      20   0  2540 1016  892 S  0.0  0.2   0:00.00 /bin/bash -c
> top -c
> 27234 root      20   0  2284  952  724 R  0.0  0.1   0:00.00 top -cbn1
> 27235 root      20   0  1756  420  352 S  0.0  0.1   0:00.00 head -100
> 27247 root      20   0  2496  452  320 S  0.0  0.1   0:00.00 /bin/sh
> /usr/local/
> 27248 root      20   0  8280 1504 1120 R  0.0  0.2   0:00.00
> /usr/bin/mysql -uad
> 27249 root      20   0  1800  448  376 S  0.0  0.1   0:00.00 sed -e 1d
> 27250 root      20   0  2240  640  540 S  0.0  0.1   0:00.00 awk
> {printf("%s", $
>
> # netstat -ptan | grep ESTABLISHED
> tcp        0      0 ::ffff:xx.xx.xx.xx:80   ::ffff:77.87.207.166:21863 ESTABLISHED 23924/httpd
> tcp        0      0 ::ffff:xx.xx.xx.xx:80   ::ffff:95.165.204.26:62259 ESTABLISHED 27144/httpd
> tcp        0      0 ::ffff:xx.xx.xx.xx:80   ::ffff:193.151.105.100:4059ESTABLISHED 27200/httpd
> tcp        0      0 ::ffff:xx.xx.xx.xx:80   ::ffff:109.169.207.68:50087ESTABLISHED 27185/httpd
> tcp        0      0 ::ffff:xx.xx.xx.xx:80   ::ffff:31.131.70.135:57017 ESTABLISHED 27179/httpd
> tcp        0      0 ::ffff:xx.xx.xx.xx:80   ::ffff:95.165.204.26:62220 ESTABLISHED 27103/httpd
> tcp        0      0 ::ffff:xx.xx.xx.xx:80   ::ffff:188.134.61.1:60732
> ESTABLISHED 27215/httpd
> tcp        0      0 ::ffff:xx.xx.xx.xx:80   ::ffff:193.151.105.100:4112ESTABLISHED 26964/httpd
> tcp        0      0 ::ffff:xx.xx.xx.xx:80   ::ffff:109.169.207.68:50043ESTABLISHED 27164/httpd
> tcp        0      0 ::ffff:xx.xx.xx.xx:80   ::ffff:31.131.70.135:56976 ESTABLISHED 27153/httpd
>
> # cat /proc/user_beancounters
> Version: 2.5
>        uid  resource                     held              maxheld
>      barrier                limit              failcnt
>      1506:  kmemsize                 27735306            179081216
>    304087040            335544320                    0
>             lockedpages                     0                    0
>        81920                81920                    0
>             privvmpages                393683               430195
>  9223372036854775807  9223372036854775807                    0
>             shmpages                      823                21639
>  9223372036854775807  9223372036854775807                    0
>             dummy                           0                    0
>            0                    0                    0
>             numproc                       128                  204
>  9223372036854775807  9223372036854775807                    0
>             physpages                   79702               163840
>            0               163840                    0
>             vmguarpages                     0                    0
>            0  9223372036854775807                    0
>             oomguarpages                74734                75707
>            0  9223372036854775807                    0
>             numtcpsock                     59                  153
>  9223372036854775807  9223372036854775807                    0
>             numflock                       46                   62
>  9223372036854775807  9223372036854775807                    0
>             numpty                          0                    1
>  9223372036854775807  9223372036854775807                    0
>             numsiginfo                      0                   33
>  9223372036854775807  9223372036854775807                    0
>             tcpsndbuf                 1037680             11426176
>  9223372036854775807  9223372036854775807                    0
>             tcprcvbuf                  966656              2867584
>  9223372036854775807  9223372036854775807                    0
>             othersockbuf                53824               838688
>  9223372036854775807  9223372036854775807                    0
>             dgramrcvbuf                     0               502224
>  9223372036854775807  9223372036854775807                    0
>             numothersock                  114                  273
>  9223372036854775807  9223372036854775807                    0
>             dcachesize               10070617            167772160
>    150994944            167772160                    0
>             numfile                      1634                 1865
>  9223372036854775807  9223372036854775807                    0
>             dummy                           0                    0
>            0                    0                    0
>             dummy                           0                    0
>            0                    0                    0
>             dummy                           0                    0
>            0                    0                    0
>             numiptent                      20                   20
>  9223372036854775807  9223372036854775807                    0
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://openvz.org/pipermail/users/attachments/20120710/fb892dc0/attachment-0001.html


More information about the Users mailing list