[Users] occasional high loadavg without any noticeable
cpu/memory/io load
Rene C.
openvz at dokbua.com
Tue Jul 10 10:40:17 EDT 2012
No takers for this one?
If I missed to provide any important information please let me know. The
issue happens regularly on several hardware nodes so if I missed anything I
can check it next time it happens.
On Wed, Jul 4, 2012 at 4:16 PM, Rene C. <openvz at dokbua.com> wrote:
> Today I again had a VE that went up to a relative high load for no
> apparent reason.
>
> Below are the details for the hardware node, followed by the high-load
> container.
>
> I realize it's not the latest kernel, but a reboot takes half an hour
> (from first VE goes down to last VE is back up, assuming everything goes
> well and no FSCK is forced) so we only reboot into new kernels when there
> is a really serious reason for it or the server crashes - but I don't see
> anything in the kernel updates since our current kernel that would address
> this issue anyway.
>
> Why does the load in this container suddenly go up like that? Websites
> hosted by the container becomes very sluggish, so it is a real problem.
>
> It isn't just a problem with this container - or even this hardware node
> for that reason, I occasionally see it with containers on other hardware
> nodes as well. One idea I brought up before was that perhaps it's the file
> system journal, as suggested in http://wiki.openvz.org/Ploop/Why - but I
> think that would affect all containers on that file system, not just a
> single container?
>
> --- HARDWARE NODE ---
>
> # uname -a
> Linux server15.hardwarenode.com 2.6.32-042stab049.6 #1 SMP Mon Feb 6
> 19:17:43 MSK 2012 x86_64 x86_64 x86_64 GNU/Linux
>
> # rpm -q sl-release
> sl-release-6.1-2.x86_64
>
> # top -cbn1 | head -17
> top - 21:00:02 up 123 days, 15:31, 1 user, load average: 0.97, 2.70, 2.37
> Tasks: 886 total, 6 running, 880 sleeping, 0 stopped, 0 zombie
> Cpu(s): 8.4%us, 1.7%sy, 0.0%ni, 86.3%id, 3.5%wa, 0.0%hi, 0.1%si,
> 0.0%st
> Mem: 16420716k total, 15566264k used, 854452k free, 1477372k buffers
> Swap: 16777184k total, 623672k used, 16153512k free, 4578176k cached
>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 94153 27 20 0 164m 41m 3392 S 150.9 0.3 50575:37
> /usr/libexec/mys
> 9178 27 20 0 159m 29m 3000 S 72.6 0.2 1284:50
> /usr/libexec/mysq
> 567031 apache 20 0 40296 15m 3588 S 17.2 0.1 0:00.09
> /usr/sbin/httpd
> 567382 root 20 0 15672 1820 864 R 5.7 0.0 0:00.04 top -cbn1
> 38 root 20 0 0 0 0 S 1.9 0.0 2:55.25 [events/3]
> 41 root 20 0 0 0 0 S 1.9 0.0 0:29.00 [events/6]
> 566362 apache 20 0 43240 19m 4448 R 1.9 0.1 0:01.04
> /usr/sbin/httpd
> 566857 apache 20 0 55248 11m 3456 R 1.9 0.1 0:00.05
> /usr/sbin/httpd
> 566918 apache 20 0 42596 17m 3704 S 1.9 0.1 0:00.15
> /usr/sbin/httpd
> 567033 apache 20 0 39784 14m 3468 S 1.9 0.1 0:00.01
> /usr/sbin/httpd
>
> # vzlist -o ctid,laverage
> CTID LAVERAGE
> 1501 0.00/0.05/0.02
> 1502 0.00/0.00/0.00
> 1503 0.08/0.03/0.01
> 1504 0.00/0.00/0.00
> 1505 8.29/6.04/3.67
> 1506 27.11/16.97/7.89
> 1507 0.00/0.00/0.00
> 1508 0.19/0.06/0.01
> 1509 0.07/0.03/0.00
> 1510 0.02/0.02/0.00
> 1512 0.00/0.00/0.00
> 1514 0.00/0.00/0.00
>
> # iostat -xN
> Linux 2.6.32-042stab049.6 (server15.hardwarenode.com) 07/03/12
> _x86_64_ (8 CPU)
>
> avg-cpu: %user %nice %system %iowait %steal %idle
> 8.41 0.04 1.75 3.51 0.00 86.28
>
> Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz
> avgqu-sz await svctm %util
> sdd 0.76 56.58 0.59 0.59 20.27 457.28 402.66
> 0.25 211.66 4.03 0.48
> sdc 1.72 27.94 17.20 16.16 887.30 336.18 36.68
> 0.02 12.71 5.23 17.45
> sdb 1.65 27.79 19.48 12.95 975.43 318.64 39.91
> 0.09 15.22 3.77 12.23
> sda 0.01 0.16 0.10 0.24 1.95 2.79 13.79
> 0.00 7.06 4.16 0.14
> vg01-swap 0.00 0.00 0.00 0.00 0.00 0.00 8.00
> 0.00 3.68 2.22 0.00
> vg01-root 0.00 0.00 0.11 0.35 1.94 2.78 10.30
> 0.02 38.30 3.12 0.14
> vg04-swap 0.00 0.00 1.30 0.22 10.41 1.80 8.00
> 0.01 9.28 1.44 0.22
> vg04-vz 0.00 0.00 0.05 56.94 9.86 455.49 8.17
> 0.01 0.18 0.05 0.27
> vg03-swap 0.00 0.00 0.00 0.00 0.00 0.00 8.00
> 0.00 6.72 1.10 0.00
> vg03-vz 0.00 0.00 18.98 42.41 887.30 336.18 19.93
> 0.39 6.33 2.84 17.45
> vg02-swap 0.00 0.00 0.00 0.00 0.00 0.00 8.00
> 0.00 7.03 0.89 0.00
> vg02-vz 0.00 0.00 21.19 39.91 975.43 318.64 21.18
> 0.15 8.99 2.00 12.23
> vg01-vz 0.00 0.00 0.00 0.00 0.00 0.00 7.98
> 0.00 17.73 17.73 0.00
>
> --- CONTAINER ---
>
> # top -cbn1 | head -100
> top - 21:00:04 up 123 days, 15:25, 0 users, load average: 27.11, 16.97,
> 7.89
> Tasks: 86 total, 2 running, 84 sleeping, 0 stopped, 0 zombie
> Cpu(s): 1.4%us, 0.2%sy, 0.0%ni, 98.1%id, 0.1%wa, 0.0%hi, 0.0%si,
> 0.2%st
> Mem: 655360k total, 316328k used, 339032k free, 0k buffers
> Swap: 1310720k total, 68380k used, 1242340k free, 58268k cached
>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 916 mysql 20 0 159m 29m 3000 S 79.3 4.6 1284:51
> /usr/libexec/mysqld
> 1 root 20 0 2156 92 64 S 0.0 0.0 0:36.50 init [3]
> 2 root 20 0 0 0 0 S 0.0 0.0 0:00.00
> [kthreadd/1506]
> 3 root 20 0 0 0 0 S 0.0 0.0 0:00.00 [khelper/1506]
> 97 root 16 -4 2244 8 4 S 0.0 0.0 0:00.00 /sbin/udevd -d
> 634 root 20 0 1812 212 136 S 0.0 0.0 2:39.88 syslogd -m 0
> 667 root 20 0 7180 268 168 S 0.0 0.0 1:01.55 /usr/sbin/sshd
> 676 root 20 0 2832 392 304 S 0.0 0.1 0:15.13 xinetd
> -stayalive -
> 690 root 20 0 6040 124 72 S 0.0 0.0 0:02.45
> /usr/lib/courier-im
> 693 root 20 0 4872 252 200 S 0.0 0.0 0:01.94
> /usr/sbin/courierlo
> 701 root 20 0 6040 124 72 S 0.0 0.0 0:06.34
> /usr/lib/courier-im
> 703 root 20 0 4872 256 200 S 0.0 0.0 0:03.09
> /usr/sbin/courierlo
> 709 root 20 0 6040 128 72 S 0.0 0.0 0:18.15
> /usr/lib/courier-im
> 711 root 20 0 4872 256 200 S 0.0 0.0 0:09.15
> /usr/sbin/courierlo
> 718 root 20 0 6040 124 72 S 0.0 0.0 0:05.68
> /usr/lib/courier-im
> 720 root 20 0 4872 252 200 S 0.0 0.0 0:02.54
> /usr/sbin/courierlo
> 730 qmails 20 0 1796 224 144 S 0.0 0.0 1:27.21 qmail-send
> 732 qmaill 20 0 1752 244 192 S 0.0 0.0 0:22.64 splogger qmail
> 733 root 20 0 1780 140 64 S 0.0 0.0 0:07.85 qmail-lspawn
> | /usr
> 734 qmailr 20 0 1776 148 76 S 0.0 0.0 0:14.07 qmail-rspawn
> 735 qmailq 20 0 1748 104 68 S 0.0 0.0 0:14.01 qmail-clean
> 781 root 20 0 51880 4364 196 S 0.0 0.7 1:35.02
> /usr/sbin/httpd
> 828 named 20 0 44104 5708 1112 S 0.0 0.9 10:10.53
> /usr/sbin/named -u
> 866 root 20 0 3708 8 4 S 0.0 0.0 0:00.00 /bin/sh
> /usr/bin/my
> 981 root 20 0 33912 3756 916 S 0.0 0.6 10:55.30
> /usr/bin/spamd --us
> 1107 xfs 20 0 3392 72 40 S 0.0 0.0 0:00.09 xfs -droppriv
> -daem
> 1115 root 20 0 5672 8 4 S 0.0 0.0 0:00.00
> /usr/sbin/saslauthd
> 1116 root 20 0 5672 8 4 S 0.0 0.0 0:00.00
> /usr/sbin/saslauthd
> 1122 root 20 0 22992 1868 1084 S 0.0 0.3 2:09.79
> /usr/bin/sw-engine
> 1123 root 20 0 27328 1508 1160 S 0.0 0.2 6:06.30
> /usr/local/psa/admi
> 7251 root 20 0 4488 192 136 S 0.0 0.0 0:22.85 crond
> 9463 apache 20 0 59184 14m 4356 S 0.0 2.3 0:05.10
> /usr/sbin/httpd
> 10512 apache 20 0 42316 2504 84 S 0.0 0.4 0:00.91
> /usr/sbin/httpd
> 12090 apache 20 0 56964 14m 4492 S 0.0 2.2 0:04.48
> /usr/sbin/httpd
> 12682 apache 20 0 61060 17m 4516 S 0.0 2.7 0:02.45
> /usr/sbin/httpd
> 13870 sw-cp-se 20 0 7852 1932 16 S 0.0 0.3 1:19.03
> /usr/sbin/sw-cp-ser
> 17443 apache 20 0 62416 17m 4436 S 0.0 2.7 0:05.27
> /usr/sbin/httpd
> 17461 apache 20 0 52788 10m 4480 S 0.0 1.6 0:02.24
> /usr/sbin/httpd
> 20430 apache 20 0 62164 17m 4356 S 0.0 2.7 0:04.25
> /usr/sbin/httpd
> 23539 popuser 20 0 37612 25m 2328 S 0.0 3.9 0:01.50 spamd child
> 23924 apache 20 0 58004 15m 5536 S 0.0 2.4 0:01.56
> /usr/sbin/httpd
> 26361 apache 20 0 54496 11m 3864 S 0.0 1.8 0:01.35
> /usr/sbin/httpd
> 26366 apache 20 0 52944 9.8m 3892 S 0.0 1.5 0:01.45
> /usr/sbin/httpd
> 26964 apache 20 0 59184 14m 4316 S 0.0 2.3 0:07.26
> /usr/sbin/httpd
> 27096 apache 20 0 53728 10m 3868 S 0.0 1.6 0:00.33
> /usr/sbin/httpd
> 27102 apache 20 0 54736 11m 3780 S 0.0 1.8 0:00.15
> /usr/sbin/httpd
> 27103 apache 20 0 54480 11m 3784 S 0.0 1.7 0:00.11
> /usr/sbin/httpd
> 27115 apache 20 0 57064 12m 3816 S 0.0 2.0 0:00.32
> /usr/sbin/httpd
> 27118 apache 20 0 53728 10m 3884 S 0.0 1.6 0:01.21
> /usr/sbin/httpd
> 27120 apache 20 0 52184 8376 3120 S 0.0 1.3 0:00.00
> /usr/sbin/httpd
> 27129 apache 20 0 52168 8072 2960 S 0.0 1.2 0:00.00
> /usr/sbin/httpd
> 27139 apache 20 0 53304 9840 3744 S 0.0 1.5 0:01.08
> /usr/sbin/httpd
> 27140 apache 20 0 53000 9.8m 3832 S 0.0 1.5 0:00.66
> /usr/sbin/httpd
> 27144 apache 20 0 52168 8072 2960 S 0.0 1.2 0:00.00
> /usr/sbin/httpd
> 27147 apache 20 0 53252 12m 5536 S 0.0 1.9 0:00.50
> /usr/sbin/httpd
> 27149 apache 20 0 52980 9924 3740 S 0.0 1.5 0:00.17
> /usr/sbin/httpd
> 27153 apache 20 0 53728 10m 3836 S 0.0 1.6 0:00.49
> /usr/sbin/httpd
> 27164 apache 20 0 55224 11m 3812 S 0.0 1.9 0:00.47
> /usr/sbin/httpd
> 27171 apache 20 0 52916 9776 3708 S 0.0 1.5 0:00.16
> /usr/sbin/httpd
> 27172 apache 20 0 52916 9452 3436 S 0.0 1.4 0:00.17
> /usr/sbin/httpd
> 27173 apache 20 0 55340 11m 3720 S 0.0 1.8 0:00.08
> /usr/sbin/httpd
> 27179 apache 20 0 52020 7764 2716 S 0.0 1.2 0:00.00
> /usr/sbin/httpd
> 27182 apache 20 0 52020 7764 2716 S 0.0 1.2 0:00.00
> /usr/sbin/httpd
> 27185 apache 20 0 55224 11m 3824 S 0.0 1.9 0:00.30
> /usr/sbin/httpd
> 27186 apache 20 0 53788 10m 3840 S 0.0 1.7 0:00.11
> /usr/sbin/httpd
> 27187 apache 20 0 52916 9448 3436 S 0.0 1.4 0:00.08
> /usr/sbin/httpd
> 27188 apache 20 0 54628 10m 3504 S 0.0 1.7 0:00.05
> /usr/sbin/httpd
> 27196 apache 20 0 53728 10m 3572 S 0.0 1.6 0:00.36
> /usr/sbin/httpd
> 27200 apache 20 0 54628 11m 3796 S 0.0 1.7 0:00.05
> /usr/sbin/httpd
> 27202 apache 20 0 54480 11m 3796 S 0.0 1.7 0:00.10
> /usr/sbin/httpd
> 27204 apache 20 0 53992 10m 3544 S 0.0 1.6 0:00.09
> /usr/sbin/httpd
> 27207 apache 20 0 52168 8084 2960 S 0.0 1.2 0:00.00
> /usr/sbin/httpd
> 27213 apache 20 0 52020 6464 1788 S 0.0 1.0 0:00.00
> /usr/sbin/httpd
> 27214 apache 20 0 54216 10m 3516 S 0.0 1.6 0:00.05
> /usr/sbin/httpd
> 27215 apache 20 0 52020 6456 1788 S 0.0 1.0 0:00.00
> /usr/sbin/httpd
> 27216 apache 20 0 52020 7860 2804 S 0.0 1.2 0:00.00
> /usr/sbin/httpd
> 27218 root 20 0 9400 1900 1408 S 0.0 0.3 0:00.00 crond
> 27219 root 20 0 2492 956 848 S 0.0 0.1 0:00.00 /bin/sh -c
> /usr/loc
> 27220 root 20 0 2496 1052 920 S 0.0 0.2 0:00.00 /bin/sh
> /usr/local/
> 27233 root 20 0 2540 1016 892 S 0.0 0.2 0:00.00 /bin/bash -c
> top -c
> 27234 root 20 0 2284 952 724 R 0.0 0.1 0:00.00 top -cbn1
> 27235 root 20 0 1756 420 352 S 0.0 0.1 0:00.00 head -100
> 27247 root 20 0 2496 452 320 S 0.0 0.1 0:00.00 /bin/sh
> /usr/local/
> 27248 root 20 0 8280 1504 1120 R 0.0 0.2 0:00.00
> /usr/bin/mysql -uad
> 27249 root 20 0 1800 448 376 S 0.0 0.1 0:00.00 sed -e 1d
> 27250 root 20 0 2240 640 540 S 0.0 0.1 0:00.00 awk
> {printf("%s", $
>
> # netstat -ptan | grep ESTABLISHED
> tcp 0 0 ::ffff:xx.xx.xx.xx:80 ::ffff:77.87.207.166:21863 ESTABLISHED 23924/httpd
> tcp 0 0 ::ffff:xx.xx.xx.xx:80 ::ffff:95.165.204.26:62259 ESTABLISHED 27144/httpd
> tcp 0 0 ::ffff:xx.xx.xx.xx:80 ::ffff:193.151.105.100:4059ESTABLISHED 27200/httpd
> tcp 0 0 ::ffff:xx.xx.xx.xx:80 ::ffff:109.169.207.68:50087ESTABLISHED 27185/httpd
> tcp 0 0 ::ffff:xx.xx.xx.xx:80 ::ffff:31.131.70.135:57017 ESTABLISHED 27179/httpd
> tcp 0 0 ::ffff:xx.xx.xx.xx:80 ::ffff:95.165.204.26:62220 ESTABLISHED 27103/httpd
> tcp 0 0 ::ffff:xx.xx.xx.xx:80 ::ffff:188.134.61.1:60732
> ESTABLISHED 27215/httpd
> tcp 0 0 ::ffff:xx.xx.xx.xx:80 ::ffff:193.151.105.100:4112ESTABLISHED 26964/httpd
> tcp 0 0 ::ffff:xx.xx.xx.xx:80 ::ffff:109.169.207.68:50043ESTABLISHED 27164/httpd
> tcp 0 0 ::ffff:xx.xx.xx.xx:80 ::ffff:31.131.70.135:56976 ESTABLISHED 27153/httpd
>
> # cat /proc/user_beancounters
> Version: 2.5
> uid resource held maxheld
> barrier limit failcnt
> 1506: kmemsize 27735306 179081216
> 304087040 335544320 0
> lockedpages 0 0
> 81920 81920 0
> privvmpages 393683 430195
> 9223372036854775807 9223372036854775807 0
> shmpages 823 21639
> 9223372036854775807 9223372036854775807 0
> dummy 0 0
> 0 0 0
> numproc 128 204
> 9223372036854775807 9223372036854775807 0
> physpages 79702 163840
> 0 163840 0
> vmguarpages 0 0
> 0 9223372036854775807 0
> oomguarpages 74734 75707
> 0 9223372036854775807 0
> numtcpsock 59 153
> 9223372036854775807 9223372036854775807 0
> numflock 46 62
> 9223372036854775807 9223372036854775807 0
> numpty 0 1
> 9223372036854775807 9223372036854775807 0
> numsiginfo 0 33
> 9223372036854775807 9223372036854775807 0
> tcpsndbuf 1037680 11426176
> 9223372036854775807 9223372036854775807 0
> tcprcvbuf 966656 2867584
> 9223372036854775807 9223372036854775807 0
> othersockbuf 53824 838688
> 9223372036854775807 9223372036854775807 0
> dgramrcvbuf 0 502224
> 9223372036854775807 9223372036854775807 0
> numothersock 114 273
> 9223372036854775807 9223372036854775807 0
> dcachesize 10070617 167772160
> 150994944 167772160 0
> numfile 1634 1865
> 9223372036854775807 9223372036854775807 0
> dummy 0 0
> 0 0 0
> dummy 0 0
> 0 0 0
> dummy 0 0
> 0 0 0
> numiptent 20 20
> 9223372036854775807 9223372036854775807 0
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://openvz.org/pipermail/users/attachments/20120710/fb892dc0/attachment-0001.html
More information about the Users
mailing list