[Users] occasional high loadavg without any noticeable
cpu/memory/io load
Rene C.
openvz at dokbua.com
Tue Jul 10 14:36:29 EDT 2012
Thanks, that'd be very cool. Access to the hardware node is limited by IP
but if you send me (privately if you prefer) the IP address you will use to
access I'll add that to allowed hosts and reply with the login coordinates.
Rene
On Tue, Jul 10, 2012 at 11:34 PM, Kirill Korotaev <dev at parallels.com> wrote:
> I can take a look if you give me access to node.
> If agree - send it privately, w/o users@ on CC.
>
> Kirill
>
>
> On Jul 10, 2012, at 18:40 , Rene C. wrote:
>
> No takers for this one?
>
> If I missed to provide any important information please let me know. The
> issue happens regularly on several hardware nodes so if I missed anything I
> can check it next time it happens.
>
> On Wed, Jul 4, 2012 at 4:16 PM, Rene C. <openvz at dokbua.com> wrote:
>
>> Today I again had a VE that went up to a relative high load for no
>> apparent reason.
>>
>> Below are the details for the hardware node, followed by the high-load
>> container.
>>
>> I realize it's not the latest kernel, but a reboot takes half an hour
>> (from first VE goes down to last VE is back up, assuming everything goes
>> well and no FSCK is forced) so we only reboot into new kernels when there
>> is a really serious reason for it or the server crashes - but I don't see
>> anything in the kernel updates since our current kernel that would address
>> this issue anyway.
>>
>> Why does the load in this container suddenly go up like that? Websites
>> hosted by the container becomes very sluggish, so it is a real problem.
>>
>> It isn't just a problem with this container - or even this hardware node
>> for that reason, I occasionally see it with containers on other hardware
>> nodes as well. One idea I brought up before was that perhaps it's the file
>> system journal, as suggested in http://wiki.openvz.org/Ploop/Why - but I
>> think that would affect all containers on that file system, not just a
>> single container?
>>
>> --- HARDWARE NODE ---
>>
>> # uname -a
>> Linux server15.hardwarenode.com 2.6.32-042stab049.6 #1 SMP Mon Feb 6
>> 19:17:43 MSK 2012 x86_64 x86_64 x86_64 GNU/Linux
>>
>> # rpm -q sl-release
>> sl-release-6.1-2.x86_64
>>
>> # top -cbn1 | head -17
>> top - 21:00:02 up 123 days, 15:31, 1 user, load average: 0.97, 2.70,
>> 2.37
>> Tasks: 886 total, 6 running, 880 sleeping, 0 stopped, 0 zombie
>> Cpu(s): 8.4%us, 1.7%sy, 0.0%ni, 86.3%id, 3.5%wa, 0.0%hi, 0.1%si,
>> 0.0%st
>> Mem: 16420716k total, 15566264k used, 854452k free, 1477372k buffers
>> Swap: 16777184k total, 623672k used, 16153512k free, 4578176k cached
>>
>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
>> 94153 27 20 0 164m 41m 3392 S 150.9 0.3 50575:37
>> /usr/libexec/mys
>> 9178 27 20 0 159m 29m 3000 S 72.6 0.2 1284:50
>> /usr/libexec/mysq
>> 567031 apache 20 0 40296 15m 3588 S 17.2 0.1 0:00.09
>> /usr/sbin/httpd
>> 567382 root 20 0 15672 1820 864 R 5.7 0.0 0:00.04 top -cbn1
>> 38 root 20 0 0 0 0 S 1.9 0.0 2:55.25 [events/3]
>> 41 root 20 0 0 0 0 S 1.9 0.0 0:29.00 [events/6]
>> 566362 apache 20 0 43240 19m 4448 R 1.9 0.1 0:01.04
>> /usr/sbin/httpd
>> 566857 apache 20 0 55248 11m 3456 R 1.9 0.1 0:00.05
>> /usr/sbin/httpd
>> 566918 apache 20 0 42596 17m 3704 S 1.9 0.1 0:00.15
>> /usr/sbin/httpd
>> 567033 apache 20 0 39784 14m 3468 S 1.9 0.1 0:00.01
>> /usr/sbin/httpd
>>
>> # vzlist -o ctid,laverage
>> CTID LAVERAGE
>> 1501 0.00/0.05/0.02
>> 1502 0.00/0.00/0.00
>> 1503 0.08/0.03/0.01
>> 1504 0.00/0.00/0.00
>> 1505 8.29/6.04/3.67
>> 1506 27.11/16.97/7.89
>> 1507 0.00/0.00/0.00
>> 1508 0.19/0.06/0.01
>> 1509 0.07/0.03/0.00
>> 1510 0.02/0.02/0.00
>> 1512 0.00/0.00/0.00
>> 1514 0.00/0.00/0.00
>>
>> # iostat -xN
>> Linux 2.6.32-042stab049.6 (server15.hardwarenode.com) 07/03/12
>> _x86_64_ (8 CPU)
>>
>> avg-cpu: %user %nice %system %iowait %steal %idle
>> 8.41 0.04 1.75 3.51 0.00 86.28
>>
>> Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s
>> avgrq-sz avgqu-sz await svctm %util
>> sdd 0.76 56.58 0.59 0.59 20.27 457.28
>> 402.66 0.25 211.66 4.03 0.48
>> sdc 1.72 27.94 17.20 16.16 887.30 336.18
>> 36.68 0.02 12.71 5.23 17.45
>> sdb 1.65 27.79 19.48 12.95 975.43 318.64
>> 39.91 0.09 15.22 3.77 12.23
>> sda 0.01 0.16 0.10 0.24 1.95 2.79
>> 13.79 0.00 7.06 4.16 0.14
>> vg01-swap 0.00 0.00 0.00 0.00 0.00 0.00
>> 8.00 0.00 3.68 2.22 0.00
>> vg01-root 0.00 0.00 0.11 0.35 1.94 2.78
>> 10.30 0.02 38.30 3.12 0.14
>> vg04-swap 0.00 0.00 1.30 0.22 10.41 1.80
>> 8.00 0.01 9.28 1.44 0.22
>> vg04-vz 0.00 0.00 0.05 56.94 9.86 455.49
>> 8.17 0.01 0.18 0.05 0.27
>> vg03-swap 0.00 0.00 0.00 0.00 0.00 0.00
>> 8.00 0.00 6.72 1.10 0.00
>> vg03-vz 0.00 0.00 18.98 42.41 887.30 336.18
>> 19.93 0.39 6.33 2.84 17.45
>> vg02-swap 0.00 0.00 0.00 0.00 0.00 0.00
>> 8.00 0.00 7.03 0.89 0.00
>> vg02-vz 0.00 0.00 21.19 39.91 975.43 318.64
>> 21.18 0.15 8.99 2.00 12.23
>> vg01-vz 0.00 0.00 0.00 0.00 0.00 0.00
>> 7.98 0.00 17.73 17.73 0.00
>>
>> --- CONTAINER ---
>>
>> # top -cbn1 | head -100
>> top - 21:00:04 up 123 days, 15:25, 0 users, load average: 27.11, 16.97,
>> 7.89
>> Tasks: 86 total, 2 running, 84 sleeping, 0 stopped, 0 zombie
>> Cpu(s): 1.4%us, 0.2%sy, 0.0%ni, 98.1%id, 0.1%wa, 0.0%hi, 0.0%si,
>> 0.2%st
>> Mem: 655360k total, 316328k used, 339032k free, 0k buffers
>> Swap: 1310720k total, 68380k used, 1242340k free, 58268k cached
>>
>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
>> 916 mysql 20 0 159m 29m 3000 S 79.3 4.6 1284:51
>> /usr/libexec/mysqld
>> 1 root 20 0 2156 92 64 S 0.0 0.0 0:36.50 init [3]
>> 2 root 20 0 0 0 0 S 0.0 0.0 0:00.00
>> [kthreadd/1506]
>> 3 root 20 0 0 0 0 S 0.0 0.0 0:00.00
>> [khelper/1506]
>> 97 root 16 -4 2244 8 4 S 0.0 0.0 0:00.00 /sbin/udevd
>> -d
>> 634 root 20 0 1812 212 136 S 0.0 0.0 2:39.88 syslogd -m 0
>> 667 root 20 0 7180 268 168 S 0.0 0.0 1:01.55
>> /usr/sbin/sshd
>> 676 root 20 0 2832 392 304 S 0.0 0.1 0:15.13 xinetd
>> -stayalive -
>> 690 root 20 0 6040 124 72 S 0.0 0.0 0:02.45
>> /usr/lib/courier-im
>> 693 root 20 0 4872 252 200 S 0.0 0.0 0:01.94
>> /usr/sbin/courierlo
>> 701 root 20 0 6040 124 72 S 0.0 0.0 0:06.34
>> /usr/lib/courier-im
>> 703 root 20 0 4872 256 200 S 0.0 0.0 0:03.09
>> /usr/sbin/courierlo
>> 709 root 20 0 6040 128 72 S 0.0 0.0 0:18.15
>> /usr/lib/courier-im
>> 711 root 20 0 4872 256 200 S 0.0 0.0 0:09.15
>> /usr/sbin/courierlo
>> 718 root 20 0 6040 124 72 S 0.0 0.0 0:05.68
>> /usr/lib/courier-im
>> 720 root 20 0 4872 252 200 S 0.0 0.0 0:02.54
>> /usr/sbin/courierlo
>> 730 qmails 20 0 1796 224 144 S 0.0 0.0 1:27.21 qmail-send
>> 732 qmaill 20 0 1752 244 192 S 0.0 0.0 0:22.64 splogger
>> qmail
>> 733 root 20 0 1780 140 64 S 0.0 0.0 0:07.85 qmail-lspawn
>> | /usr
>> 734 qmailr 20 0 1776 148 76 S 0.0 0.0 0:14.07 qmail-rspawn
>> 735 qmailq 20 0 1748 104 68 S 0.0 0.0 0:14.01 qmail-clean
>> 781 root 20 0 51880 4364 196 S 0.0 0.7 1:35.02
>> /usr/sbin/httpd
>> 828 named 20 0 44104 5708 1112 S 0.0 0.9 10:10.53
>> /usr/sbin/named -u
>> 866 root 20 0 3708 8 4 S 0.0 0.0 0:00.00 /bin/sh
>> /usr/bin/my
>> 981 root 20 0 33912 3756 916 S 0.0 0.6 10:55.30
>> /usr/bin/spamd --us
>> 1107 xfs 20 0 3392 72 40 S 0.0 0.0 0:00.09 xfs
>> -droppriv -daem
>> 1115 root 20 0 5672 8 4 S 0.0 0.0 0:00.00
>> /usr/sbin/saslauthd
>> 1116 root 20 0 5672 8 4 S 0.0 0.0 0:00.00
>> /usr/sbin/saslauthd
>> 1122 root 20 0 22992 1868 1084 S 0.0 0.3 2:09.79
>> /usr/bin/sw-engine
>> 1123 root 20 0 27328 1508 1160 S 0.0 0.2 6:06.30
>> /usr/local/psa/admi
>> 7251 root 20 0 4488 192 136 S 0.0 0.0 0:22.85 crond
>> 9463 apache 20 0 59184 14m 4356 S 0.0 2.3 0:05.10
>> /usr/sbin/httpd
>> 10512 apache 20 0 42316 2504 84 S 0.0 0.4 0:00.91
>> /usr/sbin/httpd
>> 12090 apache 20 0 56964 14m 4492 S 0.0 2.2 0:04.48
>> /usr/sbin/httpd
>> 12682 apache 20 0 61060 17m 4516 S 0.0 2.7 0:02.45
>> /usr/sbin/httpd
>> 13870 sw-cp-se 20 0 7852 1932 16 S 0.0 0.3 1:19.03
>> /usr/sbin/sw-cp-ser
>> 17443 apache 20 0 62416 17m 4436 S 0.0 2.7 0:05.27
>> /usr/sbin/httpd
>> 17461 apache 20 0 52788 10m 4480 S 0.0 1.6 0:02.24
>> /usr/sbin/httpd
>> 20430 apache 20 0 62164 17m 4356 S 0.0 2.7 0:04.25
>> /usr/sbin/httpd
>> 23539 popuser 20 0 37612 25m 2328 S 0.0 3.9 0:01.50 spamd child
>> 23924 apache 20 0 58004 15m 5536 S 0.0 2.4 0:01.56
>> /usr/sbin/httpd
>> 26361 apache 20 0 54496 11m 3864 S 0.0 1.8 0:01.35
>> /usr/sbin/httpd
>> 26366 apache 20 0 52944 9.8m 3892 S 0.0 1.5 0:01.45
>> /usr/sbin/httpd
>> 26964 apache 20 0 59184 14m 4316 S 0.0 2.3 0:07.26
>> /usr/sbin/httpd
>> 27096 apache 20 0 53728 10m 3868 S 0.0 1.6 0:00.33
>> /usr/sbin/httpd
>> 27102 apache 20 0 54736 11m 3780 S 0.0 1.8 0:00.15
>> /usr/sbin/httpd
>> 27103 apache 20 0 54480 11m 3784 S 0.0 1.7 0:00.11
>> /usr/sbin/httpd
>> 27115 apache 20 0 57064 12m 3816 S 0.0 2.0 0:00.32
>> /usr/sbin/httpd
>> 27118 apache 20 0 53728 10m 3884 S 0.0 1.6 0:01.21
>> /usr/sbin/httpd
>> 27120 apache 20 0 52184 8376 3120 S 0.0 1.3 0:00.00
>> /usr/sbin/httpd
>> 27129 apache 20 0 52168 8072 2960 S 0.0 1.2 0:00.00
>> /usr/sbin/httpd
>> 27139 apache 20 0 53304 9840 3744 S 0.0 1.5 0:01.08
>> /usr/sbin/httpd
>> 27140 apache 20 0 53000 9.8m 3832 S 0.0 1.5 0:00.66
>> /usr/sbin/httpd
>> 27144 apache 20 0 52168 8072 2960 S 0.0 1.2 0:00.00
>> /usr/sbin/httpd
>> 27147 apache 20 0 53252 12m 5536 S 0.0 1.9 0:00.50
>> /usr/sbin/httpd
>> 27149 apache 20 0 52980 9924 3740 S 0.0 1.5 0:00.17
>> /usr/sbin/httpd
>> 27153 apache 20 0 53728 10m 3836 S 0.0 1.6 0:00.49
>> /usr/sbin/httpd
>> 27164 apache 20 0 55224 11m 3812 S 0.0 1.9 0:00.47
>> /usr/sbin/httpd
>> 27171 apache 20 0 52916 9776 3708 S 0.0 1.5 0:00.16
>> /usr/sbin/httpd
>> 27172 apache 20 0 52916 9452 3436 S 0.0 1.4 0:00.17
>> /usr/sbin/httpd
>> 27173 apache 20 0 55340 11m 3720 S 0.0 1.8 0:00.08
>> /usr/sbin/httpd
>> 27179 apache 20 0 52020 7764 2716 S 0.0 1.2 0:00.00
>> /usr/sbin/httpd
>> 27182 apache 20 0 52020 7764 2716 S 0.0 1.2 0:00.00
>> /usr/sbin/httpd
>> 27185 apache 20 0 55224 11m 3824 S 0.0 1.9 0:00.30
>> /usr/sbin/httpd
>> 27186 apache 20 0 53788 10m 3840 S 0.0 1.7 0:00.11
>> /usr/sbin/httpd
>> 27187 apache 20 0 52916 9448 3436 S 0.0 1.4 0:00.08
>> /usr/sbin/httpd
>> 27188 apache 20 0 54628 10m 3504 S 0.0 1.7 0:00.05
>> /usr/sbin/httpd
>> 27196 apache 20 0 53728 10m 3572 S 0.0 1.6 0:00.36
>> /usr/sbin/httpd
>> 27200 apache 20 0 54628 11m 3796 S 0.0 1.7 0:00.05
>> /usr/sbin/httpd
>> 27202 apache 20 0 54480 11m 3796 S 0.0 1.7 0:00.10
>> /usr/sbin/httpd
>> 27204 apache 20 0 53992 10m 3544 S 0.0 1.6 0:00.09
>> /usr/sbin/httpd
>> 27207 apache 20 0 52168 8084 2960 S 0.0 1.2 0:00.00
>> /usr/sbin/httpd
>> 27213 apache 20 0 52020 6464 1788 S 0.0 1.0 0:00.00
>> /usr/sbin/httpd
>> 27214 apache 20 0 54216 10m 3516 S 0.0 1.6 0:00.05
>> /usr/sbin/httpd
>> 27215 apache 20 0 52020 6456 1788 S 0.0 1.0 0:00.00
>> /usr/sbin/httpd
>> 27216 apache 20 0 52020 7860 2804 S 0.0 1.2 0:00.00
>> /usr/sbin/httpd
>> 27218 root 20 0 9400 1900 1408 S 0.0 0.3 0:00.00 crond
>> 27219 root 20 0 2492 956 848 S 0.0 0.1 0:00.00 /bin/sh -c
>> /usr/loc
>> 27220 root 20 0 2496 1052 920 S 0.0 0.2 0:00.00 /bin/sh
>> /usr/local/
>> 27233 root 20 0 2540 1016 892 S 0.0 0.2 0:00.00 /bin/bash -c
>> top -c
>> 27234 root 20 0 2284 952 724 R 0.0 0.1 0:00.00 top -cbn1
>> 27235 root 20 0 1756 420 352 S 0.0 0.1 0:00.00 head -100
>> 27247 root 20 0 2496 452 320 S 0.0 0.1 0:00.00 /bin/sh
>> /usr/local/
>> 27248 root 20 0 8280 1504 1120 R 0.0 0.2 0:00.00
>> /usr/bin/mysql -uad
>> 27249 root 20 0 1800 448 376 S 0.0 0.1 0:00.00 sed -e 1d
>> 27250 root 20 0 2240 640 540 S 0.0 0.1 0:00.00 awk
>> {printf("%s", $
>>
>> # netstat -ptan | grep ESTABLISHED
>> tcp 0 0 ::ffff:xx.xx.xx.xx:80 ::ffff:77.87.207.166:21863 ESTABLISHED 23924/httpd
>> tcp 0 0 ::ffff:xx.xx.xx.xx:80 ::ffff:95.165.204.26:62259 ESTABLISHED 27144/httpd
>> tcp 0 0 ::ffff:xx.xx.xx.xx:80 ::ffff:193.151.105.100:4059ESTABLISHED 27200/httpd
>> tcp 0 0 ::ffff:xx.xx.xx.xx:80 ::ffff:109.169.207.68:50087ESTABLISHED 27185/httpd
>> tcp 0 0 ::ffff:xx.xx.xx.xx:80 ::ffff:31.131.70.135:57017 ESTABLISHED 27179/httpd
>> tcp 0 0 ::ffff:xx.xx.xx.xx:80 ::ffff:95.165.204.26:62220 ESTABLISHED 27103/httpd
>> tcp 0 0 ::ffff:xx.xx.xx.xx:80 ::ffff:188.134.61.1:60732
>> ESTABLISHED 27215/httpd
>> tcp 0 0 ::ffff:xx.xx.xx.xx:80 ::ffff:193.151.105.100:4112ESTABLISHED 26964/httpd
>> tcp 0 0 ::ffff:xx.xx.xx.xx:80 ::ffff:109.169.207.68:50043ESTABLISHED 27164/httpd
>> tcp 0 0 ::ffff:xx.xx.xx.xx:80 ::ffff:31.131.70.135:56976 ESTABLISHED 27153/httpd
>>
>> # cat /proc/user_beancounters
>> Version: 2.5
>> uid resource held maxheld
>> barrier limit failcnt
>> 1506: kmemsize 27735306 179081216
>> 304087040 335544320 0
>> lockedpages 0 0
>> 81920 81920 0
>> privvmpages 393683 430195
>> 9223372036854775807 9223372036854775807 0
>> shmpages 823 21639
>> 9223372036854775807 9223372036854775807 0
>> dummy 0 0
>> 0 0 0
>> numproc 128 204
>> 9223372036854775807 9223372036854775807 0
>> physpages 79702 163840
>> 0 163840 0
>> vmguarpages 0 0
>> 0 9223372036854775807 0
>> oomguarpages 74734 75707
>> 0 9223372036854775807 0
>> numtcpsock 59 153
>> 9223372036854775807 9223372036854775807 0
>> numflock 46 62
>> 9223372036854775807 9223372036854775807 0
>> numpty 0 1
>> 9223372036854775807 9223372036854775807 0
>> numsiginfo 0 33
>> 9223372036854775807 9223372036854775807 0
>> tcpsndbuf 1037680 11426176
>> 9223372036854775807 9223372036854775807 0
>> tcprcvbuf 966656 2867584
>> 9223372036854775807 9223372036854775807 0
>> othersockbuf 53824 838688
>> 9223372036854775807 9223372036854775807 0
>> dgramrcvbuf 0 502224
>> 9223372036854775807 9223372036854775807 0
>> numothersock 114 273
>> 9223372036854775807 9223372036854775807 0
>> dcachesize 10070617 167772160
>> 150994944 167772160 0
>> numfile 1634 1865
>> 9223372036854775807 9223372036854775807 0
>> dummy 0 0
>> 0 0 0
>> dummy 0 0
>> 0 0 0
>> dummy 0 0
>> 0 0 0
>> numiptent 20 20
>> 9223372036854775807 9223372036854775807 0
>>
>
> <ATT00001.c>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://openvz.org/pipermail/users/attachments/20120711/bed73b5c/attachment-0001.html
More information about the Users
mailing list