[Users] ploop performance

[Rene C.]

Disk IO on one of our hardware nodes has an unreasonably high IO use
according to iostat and iotop:

This is the output of "iostat -xzN 30"

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s
avgrq-sz avgqu-sz   await  svctm  %util
vg0-root          0.00     0.00    0.00    0.43     0.00     3.47
8.00     0.01   26.00  18.92   0.82
vg0-vz            0.00     0.00    0.70  300.50    39.47  2604.53
8.78    42.93  142.51   3.17  95.47

Here is the output from iotop:

Total DISK READ: 39.04 K/s | Total DISK WRITE: 1206.61 K/s
  TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND
 1717 be/3 root        0.00 B/s  205.55 K/s  0.00 % 51.06 % [jbd2/ploop30888]
 2736 be/4 root        0.00 B/s   51.39 K/s  0.00 % 10.68 % syslogd -m 0
597591 be/4 566        13.54 K/s    0.00 B/s  0.00 %  2.83 % authProg
 2015 be/3 root        0.00 B/s   12.35 K/s  0.00 %  1.11 % [jbd2/ploop45556]
 3446 be/3 root        0.00 B/s    0.00 B/s  0.00 %  0.71 % [jbd2/ploop24055]
  610 be/3 root        0.00 B/s    0.00 B/s  0.00 %  0.38 % [jbd2/dm-0-8]
639011 be/4 47          0.00 B/s  407.91 B/s  0.00 %  0.00 % exim -bd -q60m
 2343 be/4 root        0.00 B/s 1631.66 B/s  0.00 %  0.00 % [flush-182:72889]
567695 be/4 nobody      0.00 B/s  815.83 B/s  0.00 %  0.00 % httpd -k
start -DSSL
563274 be/4 nobody      0.00 B/s  815.83 B/s  0.00 %  0.00 % httpd -k
start -DSSL

The hardware node is a rather beefy Intel(R) Core(TM) i7 CPU X 990  @
3.47GHz with 24 GB RAM and 4 SATA drives in (software) RAID 0+1

mdadm doesn't show any unusual activity:
[root at server16 ~]# cat /proc/mdstat
Personalities : [raid1] [raid10]
md0 : active raid1 sda1[0] sdd1[4] sdb1[1] sdc1[5]
      524276 blocks super 1.0 [4/4] [UUUU]

md1 : active raid10 sda2[0] sdd2[4] sdb2[1] sdc2[5]
      2929223680 blocks super 1.0 512K chunks 2 near-copies [4/4] [UUUU]

unused devices: <none>

The server only has 3 containers, two of which are fairly quiet, the
busiest is a cpanel server with 170 websites. Loadavg is typically
around 1.xx, cpu use minimal, the same with memory use.

top - 07:14:25 up 9 days,  2:10,  3 users,  load average: 3.43, 3.02, 3.02
Tasks: 553 total,   4 running, 542 sleeping,   0 stopped,   7 zombie
Cpu(s):  6.6%us,  0.8%sy,  0.2%ni, 88.8%id,  3.5%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  24544856k total, 23808120k used,   736736k free,  1840000k buffers
Swap:  6291448k total,   336684k used,  5954764k free, 18275304k cached


      CTID       LAVERAGE
      1602 1.90/1.85/1.83
      1603 0.52/0.28/0.19
      1609 0.00/0.01/0.01

So the questions are

1)  Why does iostat show such a high %util when iotop doesn't show any
particular load?
2)  I note that the highest processes in iotop are [jbd2/ploopXXXX]
processes - what actually happens in there? I presume this would be
the internal IO activity in containers but why is it shown as the
ploop's IO and not the IO of the individual container processes?
3) Can this be improved somehow?