Hello,<div><br><div>I occasionally get this extreme load on one of our VPS servers. It is quite large, 4 full E31230 cores, 4 GB RAM and hosting ca. 400 websites + parked/addon/subdomains.</div><div><br></div><div>The hardware node has 12 active VPS servers and most of the time things are chugging along just fine, something like this.</div>
<div><br></div><div><div>1401: 0.00 0.00 0.00 1/23 4561</div><div>1402: 0.02 0.05 0.05 1/57 16991</div><div>1404: 0.01 0.02 0.00 1/73 18863</div><div>1406: 0.07 0.13 0.06 1/39 31189</div><div>1407: 0.86 1.03 1.14 1/113 31460</div>
<div>1408: 0.17 0.17 0.18 1/79 32579</div><div>1409: 0.00 0.00 0.02 1/77 21784</div><div>1410: 0.01 0.02 0.00 1/60 7454</div><div>1413: 0.00 0.00 0.00 1/46 18579</div><div>1414: 0.00 0.00 0.00 1/41 23812</div><div>1415: 0.00 0.00 0.00 1/45 9831</div>
<div>1416: 0.05 0.02 0.00 1/59 11332</div><div>12 active</div><div><br></div><div>The problem VPS is 1407. As you can see below it only uses a bit of the cpu and memory. </div><div><br></div><div>top - 17:34:12 up 32 days, 12:21, 0 users, load average: 0.78, 0.95, 1.09</div>
<div>Tasks: 102 total, 4 running, 90 sleeping, 0 stopped, 8 zombie</div><div>Cpu(s): 16.3%us, 2.9%sy, 0.4%ni, 78.5%id, 1.8%wa, 0.0%hi, 0.0%si, 0.1%st</div><div>Mem: 4194304k total, 2550572k used, 1643732k free, 0k buffers</div>
<div>Swap: 8388608k total, 105344k used, 8283264k free, 1793828k cached</div></div><div><br></div><div>Also iostat and vmstat shows no particular io or swap activity.</div><div><br></div><div>Now for the problem. Every once in a while the loadavg of this particular VPS shoots up to like crazy values, 30 or more and it becomes completely sluggish. The odd thing is load goes up for the VPS server, and starts spilling into other VPS serers on the same hardware node - but there are still no particular cpu/memory/io usage going on that I can se. No particular network activity. In this example load has fallen back to around 10 but it was much higher earlier.</div>
<div><br></div><div><div><div> 16:19:44 up 32 days, 11:19, 3 users, load average: 12.87, 19.11, 18.87</div><div><br></div><div>1401: 0.01 0.03 0.00 1/23 2876</div><div>1402: 0.00 0.11 0.13 1/57 15334</div><div>1404: 0.02 0.20 0.16 1/77 14918</div>
<div>1406: 0.01 0.13 0.10 1/39 29595</div><div>1407: 10.95 15.71 15.05 1/128 13950</div><div>1408: 0.36 0.52 0.57 1/81 27167</div><div>1409: 0.09 0.26 0.43 1/78 17851</div><div>1410: 0.09 0.17 0.18 1/61 4344</div><div>1413: 0.00 0.03 0.00 1/46 16539</div>
<div>1414: 0.01 0.01 0.00 1/41 22372</div><div>1415: 0.00 0.01 0.00 1/45 8404</div><div>1416: 0.05 0.10 0.11 1/58 9292</div><div>12 active</div><div><br></div><div>top - 16:20:02 up 32 days, 11:07, 0 users, load average: 9.14, 14.97, 14.82</div>
<div>Tasks: 135 total, 1 running, 122 sleeping, 0 stopped, 12 zombie</div><div>Cpu(s): 16.3%us, 2.9%sy, 0.4%ni, 78.5%id, 1.8%wa, 0.0%hi, 0.0%si, 0.1%st</div><div>Mem: 4194304k total, 1173844k used, 3020460k free, 0k buffers</div>
<div>Swap: 8388608k total, 115576k used, 8273032k free, 725144k cache</div></div></div><div><br></div><div>Notice how cpu is plenty idle, and only 1/4 of the available memory is being used.</div><div><br></div><div>
<a href="http://wiki.openvz.org/Ploop/Why">http://wiki.openvz.org/Ploop/Why</a> explains "One such property that deserves a special item in this list is file system journal. While journal is a good thing to have, because it helps to maintain file system integrity and improve reboot times (by eliminating fsck in many cases), it is also a bottleneck for containers. If one container will fill up in-memory journal (with lots of small operations leading to file metadata updates, e.g. file truncates), all the other containers I/O will block waiting for the journal to be written to disk. In some extreme cases we saw up to 15 seconds of such blockage.". The problem I noticed last much longer than 15 seconds though - typically 15-30 minutes, then load goes back where it should be.</div>
<div><br></div><div>Any suggestions where I could look for the cause of this? It's not like it happens everyday, maybe once or twice per month, but it's enough to cause customers to complain.</div><div><br></div>
<div>Regards,<br>Rene</div><div><br></div></div>