[Users] Wrong load values on openvz kernels?

[Solar Designer]

On Mon, Jan 17, 2011 at 02:31:27PM +0100, Benjamin Henrion wrote:
> I am trying to fix thresholds for monitoring the load on our openvz
> servers, is it normal that a machine without load reports 22.0 (on a 8
> cores, 16 cores in HT).

No, a machine completely without load should report 0 for the load
averages.  Perhaps you did not literally mean "without load"?

In case you're running multiple containers (maybe 22?) that are not in
use yet, you could want to check if they possibly contain processes
running busy loops on the CPU, which could indicate a template vs. host
incompatibility (some program in the template failing to work right).

> Any clue how to measure the load?

In our experience with servers under real load, we found load averages
to be a rather poor measure.  It is possible to have no idle CPU time
with load averages matching the number of logical CPUs (e.g., 8.00 on
8-core without SMT) with one kind of load (lengthy computation), but it
is also possible to have plenty of idle CPU time with much higher load
average numbers (like 20.0 or more) with another kind of load (frequent
but quick-to-handle requests to a server).  I/O load plays an important
role too (often more important than CPU), but as it relates to CPU load,
I wrote the following idlewatch.pl script, which we run on OpenVZ hosts.

The output (log file) from this script may look like:

$ tail idlewatch.log
Mon Jan 17 05:07:59 2011: 577.6% 393.4%
Mon Jan 17 05:17:59 2011: 589.0% 393.4%
Mon Jan 17 05:27:59 2011: 635.3% 393.4%
Mon Jan 17 05:37:59 2011: 559.5% 393.4%
Mon Jan 17 05:47:59 2011: 526.9% 393.4%
Mon Jan 17 05:57:59 2011: 570.7% 393.5%
Mon Jan 17 06:07:59 2011: 492.7% 393.5%
Mon Jan 17 06:17:59 2011: 548.8% 393.5%
Mon Jan 17 06:27:59 2011: 596.1% 393.5%
Mon Jan 17 06:37:59 2011: 507.0% 393.5%

This is on an 8-core (no SMT) system with $delay in the script set to
600 (10-minute intervals).  The first column is average idle CPU time
percentage (max 800% for this system) during the 10-minute intervals.
The second column is long-term average idle CPU time since the script
was started.

In the script below, $delay is set to 10 seconds instead - this value is
useful for quick tests (does the script work right? is the server
overloaded right now? are there any 10-second intervals with no idle CPU
time, which could cause user-visible web page generation delays?)  For
long runs, I recommend setting $delay to 600 seconds instead since it's
inconvenient to review a log with too many entries (as well as to
minimize extra disk seeks caused by logging).

On your 16 logical CPUs system, you should see "1600% 1600%" (or very
similar) on idlewatch.log lines.  As you start to load the system, you
may decide that you consider the server "full" (it's time to stop adding
containers to it), say, when there are any 10-minute intervals with less
than 400% idle or when the average idle drops below 800%.  This is just
an example; you may choose different thresholds.

Note that with two logical CPUs per core, 800% idle may actually be much
closer to 0% idle than it is to 1600% idle.  That is, when you get from
1600% down to 800% idle on average, you have used way more than 50% of
your total CPU power on average.

---
#!/usr/bin/perl

use Fcntl qw(SEEK_SET);

$delay = 10;

$SIG{CHLD} = 'IGNORE';
$SIG{HUP} = 'IGNORE';

open(S, "< /proc/stat");

close(STDOUT);
open(STDOUT, ">> idlewatch.log");
$| = 1;

$sum = $dur = 0;
$ip = 0;

while (1) {
	seek(S, 0, SEEK_SET);
	$_ = <S>;
	chop;
	($name, $u, $n, $s, $i) = split(/ +/, $_);
	next unless ($name eq 'cpu' && $i > 0);
	$t = localtime();
	if ($ip) {
		$cur = $i - $ip;
		$sum += $cur;
		$dur += $delay;
# Assume "visible HZ" of 100 (thus, we get percentages right away).
# This may slightly exceed 100% per logical CPU because our actual delay
# between the measurements is slightly higher than $delay.
		printf "$t: %.1f%% %.1f%%\n", $cur / $delay, $sum / $dur;
	} else {
		printf "$t: Started\n";
	}
	$ip = $i;
	sleep($delay);
}
---

Alexander Peslyak <solar at openwall.com>
GPG key ID: 5B341F15  fp: B3FB 63F4 D7A3 BCCC 6F6E  FC55 A2FC 027C 5B34 1F15
http://www.openwall.com - bringing security into open computing environments