[Users] occasional high loadavg without any noticeable
cpu/memory/io load
Kirill Korotaev
dev at parallels.com
Tue Jul 10 12:34:23 EDT 2012
I can take a look if you give me access to node.
If agree - send it privately, w/o users@ on CC.
Kirill
On Jul 10, 2012, at 18:40 , Rene C. wrote:
No takers for this one?
If I missed to provide any important information please let me know. The issue happens regularly on several hardware nodes so if I missed anything I can check it next time it happens.
On Wed, Jul 4, 2012 at 4:16 PM, Rene C. <openvz at dokbua.com<mailto:openvz at dokbua.com>> wrote:
Today I again had a VE that went up to a relative high load for no apparent reason.
Below are the details for the hardware node, followed by the high-load container.
I realize it's not the latest kernel, but a reboot takes half an hour (from first VE goes down to last VE is back up, assuming everything goes well and no FSCK is forced) so we only reboot into new kernels when there is a really serious reason for it or the server crashes - but I don't see anything in the kernel updates since our current kernel that would address this issue anyway.
Why does the load in this container suddenly go up like that? Websites hosted by the container becomes very sluggish, so it is a real problem.
It isn't just a problem with this container - or even this hardware node for that reason, I occasionally see it with containers on other hardware nodes as well. One idea I brought up before was that perhaps it's the file system journal, as suggested in http://wiki.openvz.org/Ploop/Why - but I think that would affect all containers on that file system, not just a single container?
--- HARDWARE NODE ---
# uname -a
Linux server15.hardwarenode.com<http://server15.hardwarenode.com/> 2.6.32-042stab049.6 #1 SMP Mon Feb 6 19:17:43 MSK 2012 x86_64 x86_64 x86_64 GNU/Linux
# rpm -q sl-release
sl-release-6.1-2.x86_64
# top -cbn1 | head -17
top - 21:00:02 up 123 days, 15:31, 1 user, load average: 0.97, 2.70, 2.37
Tasks: 886 total, 6 running, 880 sleeping, 0 stopped, 0 zombie
Cpu(s): 8.4%us, 1.7%sy, 0.0%ni, 86.3%id, 3.5%wa, 0.0%hi, 0.1%si, 0.0%st
Mem: 16420716k total, 15566264k used, 854452k free, 1477372k buffers
Swap: 16777184k total, 623672k used, 16153512k free, 4578176k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
94153 27 20 0 164m 41m 3392 S 150.9 0.3 50575:37 /usr/libexec/mys
9178 27 20 0 159m 29m 3000 S 72.6 0.2 1284:50 /usr/libexec/mysq
567031 apache 20 0 40296 15m 3588 S 17.2 0.1 0:00.09 /usr/sbin/httpd
567382 root 20 0 15672 1820 864 R 5.7 0.0 0:00.04 top -cbn1
38 root 20 0 0 0 0 S 1.9 0.0 2:55.25 [events/3]
41 root 20 0 0 0 0 S 1.9 0.0 0:29.00 [events/6]
566362 apache 20 0 43240 19m 4448 R 1.9 0.1 0:01.04 /usr/sbin/httpd
566857 apache 20 0 55248 11m 3456 R 1.9 0.1 0:00.05 /usr/sbin/httpd
566918 apache 20 0 42596 17m 3704 S 1.9 0.1 0:00.15 /usr/sbin/httpd
567033 apache 20 0 39784 14m 3468 S 1.9 0.1 0:00.01 /usr/sbin/httpd
# vzlist -o ctid,laverage
CTID LAVERAGE
1501 0.00/0.05/0.02
1502 0.00/0.00/0.00
1503 0.08/0.03/0.01
1504 0.00/0.00/0.00
1505 8.29/6.04/3.67
1506 27.11/16.97/7.89
1507 0.00/0.00/0.00
1508 0.19/0.06/0.01
1509 0.07/0.03/0.00
1510 0.02/0.02/0.00
1512 0.00/0.00/0.00
1514 0.00/0.00/0.00
# iostat -xN
Linux 2.6.32-042stab049.6 (server15.hardwarenode.com<http://server15.hardwarenode.com/>) 07/03/12 _x86_64_ (8 CPU)
avg-cpu: %user %nice %system %iowait %steal %idle
8.41 0.04 1.75 3.51 0.00 86.28
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
sdd 0.76 56.58 0.59 0.59 20.27 457.28 402.66 0.25 211.66 4.03 0.48
sdc 1.72 27.94 17.20 16.16 887.30 336.18 36.68 0.02 12.71 5.23 17.45
sdb 1.65 27.79 19.48 12.95 975.43 318.64 39.91 0.09 15.22 3.77 12.23
sda 0.01 0.16 0.10 0.24 1.95 2.79 13.79 0.00 7.06 4.16 0.14
vg01-swap 0.00 0.00 0.00 0.00 0.00 0.00 8.00 0.00 3.68 2.22 0.00
vg01-root 0.00 0.00 0.11 0.35 1.94 2.78 10.30 0.02 38.30 3.12 0.14
vg04-swap 0.00 0.00 1.30 0.22 10.41 1.80 8.00 0.01 9.28 1.44 0.22
vg04-vz 0.00 0.00 0.05 56.94 9.86 455.49 8.17 0.01 0.18 0.05 0.27
vg03-swap 0.00 0.00 0.00 0.00 0.00 0.00 8.00 0.00 6.72 1.10 0.00
vg03-vz 0.00 0.00 18.98 42.41 887.30 336.18 19.93 0.39 6.33 2.84 17.45
vg02-swap 0.00 0.00 0.00 0.00 0.00 0.00 8.00 0.00 7.03 0.89 0.00
vg02-vz 0.00 0.00 21.19 39.91 975.43 318.64 21.18 0.15 8.99 2.00 12.23
vg01-vz 0.00 0.00 0.00 0.00 0.00 0.00 7.98 0.00 17.73 17.73 0.00
--- CONTAINER ---
# top -cbn1 | head -100
top - 21:00:04 up 123 days, 15:25, 0 users, load average: 27.11, 16.97, 7.89
Tasks: 86 total, 2 running, 84 sleeping, 0 stopped, 0 zombie
Cpu(s): 1.4%us, 0.2%sy, 0.0%ni, 98.1%id, 0.1%wa, 0.0%hi, 0.0%si, 0.2%st
Mem: 655360k total, 316328k used, 339032k free, 0k buffers
Swap: 1310720k total, 68380k used, 1242340k free, 58268k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
916 mysql 20 0 159m 29m 3000 S 79.3 4.6 1284:51 /usr/libexec/mysqld
1 root 20 0 2156 92 64 S 0.0 0.0 0:36.50 init [3]
2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 [kthreadd/1506]
3 root 20 0 0 0 0 S 0.0 0.0 0:00.00 [khelper/1506]
97 root 16 -4 2244 8 4 S 0.0 0.0 0:00.00 /sbin/udevd -d
634 root 20 0 1812 212 136 S 0.0 0.0 2:39.88 syslogd -m 0
667 root 20 0 7180 268 168 S 0.0 0.0 1:01.55 /usr/sbin/sshd
676 root 20 0 2832 392 304 S 0.0 0.1 0:15.13 xinetd -stayalive -
690 root 20 0 6040 124 72 S 0.0 0.0 0:02.45 /usr/lib/courier-im
693 root 20 0 4872 252 200 S 0.0 0.0 0:01.94 /usr/sbin/courierlo
701 root 20 0 6040 124 72 S 0.0 0.0 0:06.34 /usr/lib/courier-im
703 root 20 0 4872 256 200 S 0.0 0.0 0:03.09 /usr/sbin/courierlo
709 root 20 0 6040 128 72 S 0.0 0.0 0:18.15 /usr/lib/courier-im
711 root 20 0 4872 256 200 S 0.0 0.0 0:09.15 /usr/sbin/courierlo
718 root 20 0 6040 124 72 S 0.0 0.0 0:05.68 /usr/lib/courier-im
720 root 20 0 4872 252 200 S 0.0 0.0 0:02.54 /usr/sbin/courierlo
730 qmails 20 0 1796 224 144 S 0.0 0.0 1:27.21 qmail-send
732 qmaill 20 0 1752 244 192 S 0.0 0.0 0:22.64 splogger qmail
733 root 20 0 1780 140 64 S 0.0 0.0 0:07.85 qmail-lspawn | /usr
734 qmailr 20 0 1776 148 76 S 0.0 0.0 0:14.07 qmail-rspawn
735 qmailq 20 0 1748 104 68 S 0.0 0.0 0:14.01 qmail-clean
781 root 20 0 51880 4364 196 S 0.0 0.7 1:35.02 /usr/sbin/httpd
828 named 20 0 44104 5708 1112 S 0.0 0.9 10:10.53 /usr/sbin/named -u
866 root 20 0 3708 8 4 S 0.0 0.0 0:00.00 /bin/sh /usr/bin/my
981 root 20 0 33912 3756 916 S 0.0 0.6 10:55.30 /usr/bin/spamd --us
1107 xfs 20 0 3392 72 40 S 0.0 0.0 0:00.09 xfs -droppriv -daem
1115 root 20 0 5672 8 4 S 0.0 0.0 0:00.00 /usr/sbin/saslauthd
1116 root 20 0 5672 8 4 S 0.0 0.0 0:00.00 /usr/sbin/saslauthd
1122 root 20 0 22992 1868 1084 S 0.0 0.3 2:09.79 /usr/bin/sw-engine
1123 root 20 0 27328 1508 1160 S 0.0 0.2 6:06.30 /usr/local/psa/admi
7251 root 20 0 4488 192 136 S 0.0 0.0 0:22.85 crond
9463 apache 20 0 59184 14m 4356 S 0.0 2.3 0:05.10 /usr/sbin/httpd
10512 apache 20 0 42316 2504 84 S 0.0 0.4 0:00.91 /usr/sbin/httpd
12090 apache 20 0 56964 14m 4492 S 0.0 2.2 0:04.48 /usr/sbin/httpd
12682 apache 20 0 61060 17m 4516 S 0.0 2.7 0:02.45 /usr/sbin/httpd
13870 sw-cp-se 20 0 7852 1932 16 S 0.0 0.3 1:19.03 /usr/sbin/sw-cp-ser
17443 apache 20 0 62416 17m 4436 S 0.0 2.7 0:05.27 /usr/sbin/httpd
17461 apache 20 0 52788 10m 4480 S 0.0 1.6 0:02.24 /usr/sbin/httpd
20430 apache 20 0 62164 17m 4356 S 0.0 2.7 0:04.25 /usr/sbin/httpd
23539 popuser 20 0 37612 25m 2328 S 0.0 3.9 0:01.50 spamd child
23924 apache 20 0 58004 15m 5536 S 0.0 2.4 0:01.56 /usr/sbin/httpd
26361 apache 20 0 54496 11m 3864 S 0.0 1.8 0:01.35 /usr/sbin/httpd
26366 apache 20 0 52944 9.8m 3892 S 0.0 1.5 0:01.45 /usr/sbin/httpd
26964 apache 20 0 59184 14m 4316 S 0.0 2.3 0:07.26 /usr/sbin/httpd
27096 apache 20 0 53728 10m 3868 S 0.0 1.6 0:00.33 /usr/sbin/httpd
27102 apache 20 0 54736 11m 3780 S 0.0 1.8 0:00.15 /usr/sbin/httpd
27103 apache 20 0 54480 11m 3784 S 0.0 1.7 0:00.11 /usr/sbin/httpd
27115 apache 20 0 57064 12m 3816 S 0.0 2.0 0:00.32 /usr/sbin/httpd
27118 apache 20 0 53728 10m 3884 S 0.0 1.6 0:01.21 /usr/sbin/httpd
27120 apache 20 0 52184 8376 3120 S 0.0 1.3 0:00.00 /usr/sbin/httpd
27129 apache 20 0 52168 8072 2960 S 0.0 1.2 0:00.00 /usr/sbin/httpd
27139 apache 20 0 53304 9840 3744 S 0.0 1.5 0:01.08 /usr/sbin/httpd
27140 apache 20 0 53000 9.8m 3832 S 0.0 1.5 0:00.66 /usr/sbin/httpd
27144 apache 20 0 52168 8072 2960 S 0.0 1.2 0:00.00 /usr/sbin/httpd
27147 apache 20 0 53252 12m 5536 S 0.0 1.9 0:00.50 /usr/sbin/httpd
27149 apache 20 0 52980 9924 3740 S 0.0 1.5 0:00.17 /usr/sbin/httpd
27153 apache 20 0 53728 10m 3836 S 0.0 1.6 0:00.49 /usr/sbin/httpd
27164 apache 20 0 55224 11m 3812 S 0.0 1.9 0:00.47 /usr/sbin/httpd
27171 apache 20 0 52916 9776 3708 S 0.0 1.5 0:00.16 /usr/sbin/httpd
27172 apache 20 0 52916 9452 3436 S 0.0 1.4 0:00.17 /usr/sbin/httpd
27173 apache 20 0 55340 11m 3720 S 0.0 1.8 0:00.08 /usr/sbin/httpd
27179 apache 20 0 52020 7764 2716 S 0.0 1.2 0:00.00 /usr/sbin/httpd
27182 apache 20 0 52020 7764 2716 S 0.0 1.2 0:00.00 /usr/sbin/httpd
27185 apache 20 0 55224 11m 3824 S 0.0 1.9 0:00.30 /usr/sbin/httpd
27186 apache 20 0 53788 10m 3840 S 0.0 1.7 0:00.11 /usr/sbin/httpd
27187 apache 20 0 52916 9448 3436 S 0.0 1.4 0:00.08 /usr/sbin/httpd
27188 apache 20 0 54628 10m 3504 S 0.0 1.7 0:00.05 /usr/sbin/httpd
27196 apache 20 0 53728 10m 3572 S 0.0 1.6 0:00.36 /usr/sbin/httpd
27200 apache 20 0 54628 11m 3796 S 0.0 1.7 0:00.05 /usr/sbin/httpd
27202 apache 20 0 54480 11m 3796 S 0.0 1.7 0:00.10 /usr/sbin/httpd
27204 apache 20 0 53992 10m 3544 S 0.0 1.6 0:00.09 /usr/sbin/httpd
27207 apache 20 0 52168 8084 2960 S 0.0 1.2 0:00.00 /usr/sbin/httpd
27213 apache 20 0 52020 6464 1788 S 0.0 1.0 0:00.00 /usr/sbin/httpd
27214 apache 20 0 54216 10m 3516 S 0.0 1.6 0:00.05 /usr/sbin/httpd
27215 apache 20 0 52020 6456 1788 S 0.0 1.0 0:00.00 /usr/sbin/httpd
27216 apache 20 0 52020 7860 2804 S 0.0 1.2 0:00.00 /usr/sbin/httpd
27218 root 20 0 9400 1900 1408 S 0.0 0.3 0:00.00 crond
27219 root 20 0 2492 956 848 S 0.0 0.1 0:00.00 /bin/sh -c /usr/loc
27220 root 20 0 2496 1052 920 S 0.0 0.2 0:00.00 /bin/sh /usr/local/
27233 root 20 0 2540 1016 892 S 0.0 0.2 0:00.00 /bin/bash -c top -c
27234 root 20 0 2284 952 724 R 0.0 0.1 0:00.00 top -cbn1
27235 root 20 0 1756 420 352 S 0.0 0.1 0:00.00 head -100
27247 root 20 0 2496 452 320 S 0.0 0.1 0:00.00 /bin/sh /usr/local/
27248 root 20 0 8280 1504 1120 R 0.0 0.2 0:00.00 /usr/bin/mysql -uad
27249 root 20 0 1800 448 376 S 0.0 0.1 0:00.00 sed -e 1d
27250 root 20 0 2240 640 540 S 0.0 0.1 0:00.00 awk {printf("%s", $
# netstat -ptan | grep ESTABLISHED
tcp 0 0 ::ffff:xx.xx.xx.xx:80 ::ffff:77.87.207.166:21863<http://77.87.207.166:21863/> ESTABLISHED 23924/httpd
tcp 0 0 ::ffff:xx.xx.xx.xx:80 ::ffff:95.165.204.26:62259<http://95.165.204.26:62259/> ESTABLISHED 27144/httpd
tcp 0 0 ::ffff:xx.xx.xx.xx:80 ::ffff:193.151.105.100:4059<http://193.151.105.100:4059/> ESTABLISHED 27200/httpd
tcp 0 0 ::ffff:xx.xx.xx.xx:80 ::ffff:109.169.207.68:50087<http://109.169.207.68:50087/> ESTABLISHED 27185/httpd
tcp 0 0 ::ffff:xx.xx.xx.xx:80 ::ffff:31.131.70.135:57017<http://31.131.70.135:57017/> ESTABLISHED 27179/httpd
tcp 0 0 ::ffff:xx.xx.xx.xx:80 ::ffff:95.165.204.26:62220<http://95.165.204.26:62220/> ESTABLISHED 27103/httpd
tcp 0 0 ::ffff:xx.xx.xx.xx:80 ::ffff:188.134.61.1:60732<http://188.134.61.1:60732/> ESTABLISHED 27215/httpd
tcp 0 0 ::ffff:xx.xx.xx.xx:80 ::ffff:193.151.105.100:4112<http://193.151.105.100:4112/> ESTABLISHED 26964/httpd
tcp 0 0 ::ffff:xx.xx.xx.xx:80 ::ffff:109.169.207.68:50043<http://109.169.207.68:50043/> ESTABLISHED 27164/httpd
tcp 0 0 ::ffff:xx.xx.xx.xx:80 ::ffff:31.131.70.135:56976<http://31.131.70.135:56976/> ESTABLISHED 27153/httpd
# cat /proc/user_beancounters
Version: 2.5
uid resource held maxheld barrier limit failcnt
1506: kmemsize 27735306 179081216 304087040 335544320 0
lockedpages 0 0 81920 81920 0
privvmpages 393683 430195 9223372036854775807 9223372036854775807 0
shmpages 823 21639 9223372036854775807 9223372036854775807 0
dummy 0 0 0 0 0
numproc 128 204 9223372036854775807 9223372036854775807 0
physpages 79702 163840 0 163840 0
vmguarpages 0 0 0 9223372036854775807 0
oomguarpages 74734 75707 0 9223372036854775807 0
numtcpsock 59 153 9223372036854775807 9223372036854775807 0
numflock 46 62 9223372036854775807 9223372036854775807 0
numpty 0 1 9223372036854775807 9223372036854775807 0
numsiginfo 0 33 9223372036854775807 9223372036854775807 0
tcpsndbuf 1037680 11426176 9223372036854775807 9223372036854775807 0
tcprcvbuf 966656 2867584 9223372036854775807 9223372036854775807 0
othersockbuf 53824 838688 9223372036854775807 9223372036854775807 0
dgramrcvbuf 0 502224 9223372036854775807 9223372036854775807 0
numothersock 114 273 9223372036854775807 9223372036854775807 0
dcachesize 10070617 167772160 150994944 167772160 0
numfile 1634 1865 9223372036854775807 9223372036854775807 0
dummy 0 0 0 0 0
dummy 0 0 0 0 0
dummy 0 0 0 0 0
numiptent 20 20 9223372036854775807 9223372036854775807 0
<ATT00001.c>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://openvz.org/pipermail/users/attachments/20120710/5ab8cccf/attachment-0001.html
More information about the Users
mailing list