<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
On 05/22/2012 01:06 PM, Rene C. wrote:
<blockquote
cite="mid:CA+ZVM8v5iuD6qM_SMwc+Dn05h3cdxtRSwx=2pYjydAxhgwdNDw@mail.gmail.com"
type="cite">
<div><br>
</div>
Actually I made a small shell script that loops through the list
of active containers and outputs the content of each containers
/proc/loadavg. It started out as a bit more elaborate script that
was intended to provide some of the functionality of a script
vzstat, that I used to use with Virtuozzo.
<div>
<br>
</div>
<div>You can download both scripts from <a moz-do-not-send="true"
href="https://www.ourhelpdesk.net/downloads/z.tgz">https://www.ourhelpdesk.net/downloads/z.tgz</a></div>
</blockquote>
<br>
vzlist have laverage field that might be of use. I.e.<br>
<br>
vzlist -o ctid,laverage<br>
<br>
<blockquote
cite="mid:CA+ZVM8v5iuD6qM_SMwc+Dn05h3cdxtRSwx=2pYjydAxhgwdNDw@mail.gmail.com"
type="cite">
<div><br>
</div>
<div><br>
</div>
<div><br>
<div class="gmail_quote">
On Tue, May 22, 2012 at 3:15 PM, Steffan <span dir="ltr"><<a
moz-do-not-send="true" href="mailto:general@ziggo.nl"
target="_blank">general@ziggo.nl</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div link="blue" vlink="purple" lang="NL">
<div>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"
lang="EN-US">Sorry dont have the answer for you</span></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"
lang="EN-US">But can you tell me what command you
used to see all loads on your node ?</span></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"
lang="EN-US"> </span></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"
lang="EN-US">Thanxs Steffan</span></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"
lang="EN-US"> </span></p>
<div style="border:none;border-top:solid #b5c4df
1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal" style="margin-left:35.4pt"><b><span
style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">Van:</span></b><span
style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">
<a moz-do-not-send="true"
href="mailto:users-bounces@openvz.org"
target="_blank">users-bounces@openvz.org</a>
[mailto:<a moz-do-not-send="true"
href="mailto:users-bounces@openvz.org"
target="_blank">users-bounces@openvz.org</a>] <b>Namens
</b>Rene Dokbua<br>
<b>Verzonden:</b> maandag 21 mei 2012 20:07<br>
<b>Aan:</b> <a moz-do-not-send="true"
href="mailto:users@openvz.org" target="_blank">users@openvz.org</a><br>
<b>Onderwerp:</b> [Users] occasional high loadavg
without any noticeable cpu/memory/io load</span></p>
</div>
<div>
<div class="h5">
<p class="MsoNormal" style="margin-left:35.4pt"> </p>
<p class="MsoNormal" style="margin-left:35.4pt">Hello,</p>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"> </p>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">I occasionally get
this extreme load on one of our VPS servers.
It is quite large, 4 full E31230 cores, 4 GB
RAM and hosting ca. 400 websites +
parked/addon/subdomains.</p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"> </p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">The
hardware node has 12 active VPS servers and
most of the time things are chugging along
just fine, something like this.</p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"> </p>
</div>
<div>
<div>
<p class="MsoNormal"
style="margin-left:35.4pt">1401: 0.00 0.00
0.00 1/23 4561</p>
</div>
<div>
<p class="MsoNormal"
style="margin-left:35.4pt">
1402: 0.02 0.05 0.05 1/57 16991</p>
</div>
<div>
<p class="MsoNormal"
style="margin-left:35.4pt">1404: 0.01 0.02
0.00 1/73 18863</p>
</div>
<div>
<p class="MsoNormal"
style="margin-left:35.4pt">1406: 0.07 0.13
0.06 1/39 31189</p>
</div>
<div>
<p class="MsoNormal"
style="margin-left:35.4pt">1407: 0.86 1.03
1.14 1/113 31460</p>
</div>
<div>
<p class="MsoNormal"
style="margin-left:35.4pt">1408: 0.17 0.17
0.18 1/79 32579</p>
</div>
<div>
<p class="MsoNormal"
style="margin-left:35.4pt">1409: 0.00 0.00
0.02 1/77 21784</p>
</div>
<div>
<p class="MsoNormal"
style="margin-left:35.4pt">1410: 0.01 0.02
0.00 1/60 7454</p>
</div>
<div>
<p class="MsoNormal"
style="margin-left:35.4pt">
1413: 0.00 0.00 0.00 1/46 18579</p>
</div>
<div>
<p class="MsoNormal"
style="margin-left:35.4pt">1414: 0.00 0.00
0.00 1/41 23812</p>
</div>
<div>
<p class="MsoNormal"
style="margin-left:35.4pt">1415: 0.00 0.00
0.00 1/45 9831</p>
</div>
<div>
<p class="MsoNormal"
style="margin-left:35.4pt">1416: 0.05 0.02
0.00 1/59 11332</p>
</div>
<div>
<p class="MsoNormal"
style="margin-left:35.4pt">12 active</p>
</div>
<div>
<p class="MsoNormal"
style="margin-left:35.4pt">
</p>
</div>
<div>
<p class="MsoNormal"
style="margin-left:35.4pt">The problem VPS
is 1407. As you can see below it only uses a
bit of the cpu and memory. </p>
</div>
<div>
<p class="MsoNormal"
style="margin-left:35.4pt">
</p>
</div>
<div>
<p class="MsoNormal"
style="margin-left:35.4pt">top - 17:34:12 up
32 days, 12:21, 0 users, load average:
0.78, 0.95, 1.09</p>
</div>
<div>
<p class="MsoNormal"
style="margin-left:35.4pt">
Tasks: 102 total, 4 running, 90 sleeping,
0 stopped, 8 zombie</p>
</div>
<div>
<p class="MsoNormal"
style="margin-left:35.4pt">Cpu(s): 16.3%us,
2.9%sy, 0.4%ni, 78.5%id, 1.8%wa, 0.0%hi,
0.0%si, 0.1%st</p>
</div>
<div>
<p class="MsoNormal"
style="margin-left:35.4pt">Mem: 4194304k
total, 2550572k used, 1643732k free,
0k buffers</p>
</div>
<div>
<p class="MsoNormal"
style="margin-left:35.4pt">Swap: 8388608k
total, 105344k used, 8283264k free,
1793828k cached</p>
</div>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"> </p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">Also
iostat and vmstat shows no particular io or
swap activity.</p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"> </p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">Now
for the problem. Every once in a while the
loadavg of this particular VPS shoots up to
like crazy values, 30 or more and it becomes
completely sluggish. The odd thing is load
goes up for the VPS server, and starts
spilling into other VPS serers on the same
hardware node - but there are still no
particular cpu/memory/io usage going on that I
can se. No particular network activity. In
this example load has fallen back to around 10
but it was much higher earlier.</p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"> </p>
</div>
<div>
<div>
<div>
<p class="MsoNormal"
style="margin-left:35.4pt"> 16:19:44 up 32
days, 11:19, 3 users, load average:
12.87, 19.11, 18.87</p>
</div>
<div>
<p class="MsoNormal"
style="margin-left:35.4pt"> </p>
</div>
<div>
<p class="MsoNormal"
style="margin-left:35.4pt">1401: 0.01 0.03
0.00 1/23 2876</p>
</div>
<div>
<p class="MsoNormal"
style="margin-left:35.4pt">
1402: 0.00 0.11 0.13 1/57 15334</p>
</div>
<div>
<p class="MsoNormal"
style="margin-left:35.4pt">1404: 0.02 0.20
0.16 1/77 14918</p>
</div>
<div>
<p class="MsoNormal"
style="margin-left:35.4pt">1406: 0.01 0.13
0.10 1/39 29595</p>
</div>
<div>
<p class="MsoNormal"
style="margin-left:35.4pt">1407: 10.95
15.71 15.05 1/128 13950</p>
</div>
<div>
<p class="MsoNormal"
style="margin-left:35.4pt">1408: 0.36 0.52
0.57 1/81 27167</p>
</div>
<div>
<p class="MsoNormal"
style="margin-left:35.4pt">1409: 0.09 0.26
0.43 1/78 17851</p>
</div>
<div>
<p class="MsoNormal"
style="margin-left:35.4pt">1410: 0.09 0.17
0.18 1/61 4344</p>
</div>
<div>
<p class="MsoNormal"
style="margin-left:35.4pt">
1413: 0.00 0.03 0.00 1/46 16539</p>
</div>
<div>
<p class="MsoNormal"
style="margin-left:35.4pt">1414: 0.01 0.01
0.00 1/41 22372</p>
</div>
<div>
<p class="MsoNormal"
style="margin-left:35.4pt">1415: 0.00 0.01
0.00 1/45 8404</p>
</div>
<div>
<p class="MsoNormal"
style="margin-left:35.4pt">1416: 0.05 0.10
0.11 1/58 9292</p>
</div>
<div>
<p class="MsoNormal"
style="margin-left:35.4pt">12 active</p>
</div>
<div>
<p class="MsoNormal"
style="margin-left:35.4pt">
</p>
</div>
<div>
<p class="MsoNormal"
style="margin-left:35.4pt">top - 16:20:02
up 32 days, 11:07, 0 users, load
average: 9.14, 14.97, 14.82</p>
</div>
<div>
<p class="MsoNormal"
style="margin-left:35.4pt">
Tasks: 135 total, 1 running, 122
sleeping, 0 stopped, 12 zombie</p>
</div>
<div>
<p class="MsoNormal"
style="margin-left:35.4pt">Cpu(s):
16.3%us, 2.9%sy, 0.4%ni, 78.5%id,
1.8%wa, 0.0%hi, 0.0%si, 0.1%st</p>
</div>
<div>
<p class="MsoNormal"
style="margin-left:35.4pt">Mem: 4194304k
total, 1173844k used, 3020460k free,
0k buffers</p>
</div>
<div>
<p class="MsoNormal"
style="margin-left:35.4pt">Swap: 8388608k
total, 115576k used, 8273032k free,
725144k cache</p>
</div>
</div>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"> </p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">Notice
how cpu is plenty idle, and only 1/4 of the
available memory is being used.</p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"> </p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><a
moz-do-not-send="true"
href="http://wiki.openvz.org/Ploop/Why"
target="_blank">http://wiki.openvz.org/Ploop/Why</a> explains "One
such property that deserves a special item in
this list is file system journal. While
journal is a good thing to have, because it
helps to maintain file system integrity and
improve reboot times (by eliminating fsck in
many cases), it is also a bottleneck for
containers. If one container will fill up
in-memory journal (with lots of small
operations leading to file metadata updates,
e.g. file truncates), all the other containers
I/O will block waiting for the journal to be
written to disk. In some extreme cases we saw
up to 15 seconds of such blockage.". The
problem I noticed last much longer than 15
seconds though - typically 15-30 minutes, then
load goes back where it should be.</p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"> </p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">Any
suggestions where I could look for the cause
of this? It's not like it happens everyday,
maybe once or twice per month, but it's enough
to cause customers to complain.</p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"> </p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">Regards,<br>
Rene</p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">
</p>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
_______________________________________________<br>
Users mailing list<br>
<a moz-do-not-send="true" href="mailto:Users@openvz.org">Users@openvz.org</a><br>
<a moz-do-not-send="true"
href="https://openvz.org/mailman/listinfo/users"
target="_blank">https://openvz.org/mailman/listinfo/users</a><br>
<br>
</blockquote>
</div>
<br>
</div>
</blockquote>
<br>
</body>
</html>