<div dir="ltr">Interesting - I'll check out FIO.<div><br></div><div>I wondered about the possibility of disk placement, and created the 3rd and 4th CTs in the reverse order of the 1st two, but regardless, the 2 ploopfs-based CTs performed better than the 2 simfs-based CTs. (they are all ext4 underneath) The CTs are contained in a partition at the end of the disk, which somewhat limits the effect of disk placement. I suppose a better test would be to eliminate the variables of rotational media by using SSDs for future testing, </div>
<div><br></div><div>J J </div><div><br></div><div><br></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Sat, May 3, 2014 at 11:50 PM, Kirill Korotaev <span dir="ltr"><<a href="mailto:dev@parallels.com" target="_blank">dev@parallels.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="auto">
<div>Forget about iozone - it benchmarks cached i/o and small data sets, so essentially it measures memory / syscall speeds. On larger data sets it measures mix of ram and real i/o and a lot depends on previous cache state.</div>
<div><br>
</div>
<div>Fio is a better tool.</div>
<div><br>
</div>
<div>Actually the most likely explanation for your effect is non-uniform disk performance across the plate. You can find that for rotational media performance at beginning of block device is almost about twice faster than at the end (rotational speed is the
same, but velocity is obviously faster on inner tracks).</div>
<div>So you can verify that by dumping extent info by dumpfs. Accurate benchmarking would require a small localized partition for both tests to make sure performance can't vary due to this effect.</div>
<div><br>
Sent from my iPhone</div><div class="">
<div><br>
On 04 мая 2014 г., at 1:06, "jjs - mainphrame" <<a href="mailto:jjs@mainphrame.com" target="_blank">jjs@mainphrame.com</a>> wrote:<br>
<br>
</div>
</div><blockquote type="cite">
<div><div class="">
<div dir="ltr">I did some benchmarks on newly created CTs with iozone, and the results were probably more in line with what you'd expect.
<div><br>
</div>
<div>The simfs-based CT was about 5% faster on write, and the ploop-based CT was about 5% faster on re-write, read, and re-read. The results are repeatable.</div>
<div><br>
</div>
<div><br>
</div>
<div>Regards,</div>
<div><br>
</div>
<div>J J </div>
</div>
</div><div class="gmail_extra"><br>
<br>
<div class="gmail_quote"><div class="">On Sat, May 3, 2014 at 11:53 AM, jjs - mainphrame <span dir="ltr">
<<a href="mailto:jjs@mainphrame.com" target="_blank">jjs@mainphrame.com</a>></span> wrote:<br>
</div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr"><div class="">I am continuing to do testing as time allows. Lat night I ran sysbench fileio tests, and again, the ploop CT yielded better performance then either then simfs CT or the vzhost. It wasn't as drastic a difference as the dbench results, but the
difference was there. I'll continue in this vein with freshly created CTs. The machine was just built a few days ago, it's quiescent, it's doing nothing except hosting a few vanilla CTs.
<div><br>
</div>
</div><div><div class="">As for the rules of thumb, I can tell you that the results are 100% repeatable. But explainable, ah, that's the thing. still working on that.
<div><br>
</div>
<div>Regards,</div>
<div><br>
</div>
</div><div>J J
<div>
<div><br>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote"><div class="">On Sat, May 3, 2014 at 11:31 AM, Kir Kolyshkin <span dir="ltr">
<<a href="mailto:kir@openvz.org" target="_blank">kir@openvz.org</a>></span> wrote:<br>
</div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"><div class="">
<div>
<div>On 05/02/2014 04:38 PM, jjs - mainphrame wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Thanks Kir, the /dev/zero makes sense I suppose. I tried with /dev/random but that blocks pretty quickly - /dev/urandom is better, but still seems to be a bottleneck.</div>
</blockquote>
<br>
</div>
You can use a real file on tmpfs.<br>
<br>
Also, in general, there are very many factors that influence test results. Starting from the cron jobs and other stuff (say, network activity) that runs periodically or sporadically and spoils your results, to the cache state (you need to use vm_drop_caches,
or yet better, reboot between tests), to the physical place on disk where your data is placed (rotating hdds tend to be faster at the first sectors compared to the last sectors, so ideally you need to do this on a clean freshly formatted filesystem). There
is much more to it, can be some other factors, too. The rule of thumb is results need to be reproducible and explainable.
<br>
<span><font color="#888888"><br>
Kir.</font></span>
</div><div>
<div><br>
<br>
<blockquote type="cite"><div class="">
<div dir="ltr">
<div><br>
</div>
<div>As for the dbench results, I'd love to hear what results others obtain from the same test, and/or any other testing approaches that would give a more "acceptable" answer.</div>
<div><br>
</div>
<div>Regards,</div>
<div><br>
</div>
<div>J J<br>
<div><br>
</div>
</div>
</div>
</div><div class="gmail_extra"><br>
<br>
<div class="gmail_quote"><div class="">On Fri, May 2, 2014 at 4:01 PM, Kir Kolyshkin <span dir="ltr">
<<a href="mailto:kir@openvz.org" target="_blank">kir@openvz.org</a>></span> wrote:<br>
</div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<div><div class="">
<div>On 05/02/2014 03:00 PM, jjs - mainphrame wrote:<br>
</div>
</div><blockquote type="cite">
<div dir="ltr"><div class="">Just for kicks, here are the data from the tests. (these were run on a rather modest old machine)
<div><br>
</div>
</div><div><mime-attachment.png></div><div><div class="h5">
<div><br>
</div>
<div><br>
</div>
<div>Here are the raw dbench data:</div>
<div><br>
</div>
<div><br>
</div>
<div>
<div><font face="courier new, monospace">#clients vzhost simfs CT ploop CT</font></div>
<div><font face="courier new, monospace">---------------------------------------------------------------------</font></div>
<div><font face="courier new, monospace">1 11.1297MB/sec 9.96657MB/sec 19.7214MB/sec</font></div>
<div><font face="courier new, monospace">2 12.2936MB/sec 14.3138MB/sec 23.5628MB/sec</font></div>
<div><font face="courier new, monospace">4 17.8909MB/sec 16.0859MB/sec 45.1936MB/sec</font></div>
<div><font face="courier new, monospace">8 25.8332MB/sec 22.9195MB/sec 84.2607MB/sec</font></div>
<div><font face="courier new, monospace">16 32.1436MB/sec 28.921MB/sec 155.207MB/sec</font></div>
<div><font face="courier new, monospace">32 35.5809MB/sec 32.1429MB/sec 206.571MB/sec</font></div>
<div><font face="courier new, monospace">64 34.3609MB/sec 29.9307MB/sec 221.119MB/sec</font></div>
</div>
</div></div></div>
</blockquote>
<br>
</div><div><div class="h5">
Well, I can't explain this, but there's probably something wrong with the test.
<div>
<div><br>
<br>
<blockquote type="cite">
<div dir="ltr">
<div>
<div><br>
</div>
<div>Here is the script used to invoke dbench:</div>
<div><br>
</div>
<div>
<div><font face="courier new, monospace">HOST=`uname -n`</font></div>
<div><font face="courier new, monospace">WD=/tmp</font></div>
<div><font face="courier new, monospace">FILE=/usr/share/dbench/client.txt</font></div>
<div><font face="courier new, monospace"><br>
</font></div>
<div><font face="courier new, monospace">for i in 1 2 4 8 16 32 64</font></div>
<div><font face="courier new, monospace">do</font></div>
<div><font face="courier new, monospace"> dbench -D $WD -c $FILE $i &>dbench-${HOST}-${i}</font></div>
<div><font face="courier new, monospace">done</font></div>
</div>
<div><br>
</div>
<div>Here are the dd commands and outputs:</div>
<div><br>
</div>
<div>
<div><font face="courier new, monospace">OPENVZ HOST</font></div>
<div><font face="courier new, monospace">----------------</font></div>
<div><font face="courier new, monospace">[root@vzhost ~]# dd bs=1M count=512 if=/dev/zero of=test conv=fdatasync</font></div>
<div><font face="courier new, monospace">512+0 records in</font></div>
<div><font face="courier new, monospace">512+0 records out</font></div>
<div><font face="courier new, monospace">536870912 bytes (537 MB) copied, 11.813 s, 45.4 MB/s</font></div>
<div><font face="courier new, monospace">[root@vzhost ~]# df -T</font></div>
<div><font face="courier new, monospace">Filesystem Type 1K-blocks Used Available Use% Mounted on</font></div>
<div><font face="courier new, monospace">/dev/sda2 ext4 20642428 2390620 17203232 13% /</font></div>
<div><font face="courier new, monospace">tmpfs tmpfs 952008 0 952008 0% /dev/shm</font></div>
<div><font face="courier new, monospace">/dev/sda1 ext2 482922 68436 389552 15% /boot</font></div>
<div><font face="courier new, monospace">/dev/sda4 ext4 51633780 3631524 45379332 8% /local</font></div>
<div><font face="courier new, monospace">[root@vzhost ~]# </font></div>
<div><font face="courier new, monospace"><br>
</font></div>
<div><font face="courier new, monospace"><br>
</font></div>
<div><font face="courier new, monospace">PLOOP CT</font></div>
<div><font face="courier new, monospace">----------------</font></div>
<div><font face="courier new, monospace">root@vz101:~# dd bs=1M count=512 if=/dev/zero of=test conv=fdatasync</font></div>
<div><font face="courier new, monospace">512+0 records in</font></div>
<div><font face="courier new, monospace">512+0 records out</font></div>
<div><font face="courier new, monospace">536870912 bytes (537 MB) copied, 2.50071 s, 215 MB/s</font></div>
</div>
</div>
</div>
</blockquote>
<br>
</div>
</div>
This one I can explain :)<br>
<br>
This is caused by ploop optimization that was enabled in the kernel recently.<br>
If data block is all zeroes, it is not written to the disk (same thing as sparse files,<br>
just for ploop).<br>
<br>
So you need to test it with some real data (anything but not all zeroes).<br>
I am not sure how fast is /dev/urandom but this is one of the options.<br>
<br>
<blockquote type="cite">
<div>
<div>
<div dir="ltr">
<div>
<div>
<div><font face="courier new, monospace"><br>
</font></div>
<div><font face="courier new, monospace">root@vz101:~# df -T</font></div>
<div><font face="courier new, monospace">Filesystem Type 1K-blocks Used Available Use% Mounted on</font></div>
<div><font face="courier new, monospace">/dev/ploop11054p1 ext4 4539600 1529316 2804928 36% /</font></div>
<div><font face="courier new, monospace">none devtmpfs 262144 4 262140 1% /dev</font></div>
<div><font face="courier new, monospace">none tmpfs 52432 52 52380 1% /run</font></div>
<div><font face="courier new, monospace">none tmpfs 5120 0 5120 0% /run/lock</font></div>
<div><font face="courier new, monospace">none tmpfs 262144 0 262144 0% /run/shm</font></div>
<div><font face="courier new, monospace">root@vz101:~#</font></div>
<div><font face="courier new, monospace"><br>
</font></div>
<div><font face="courier new, monospace"><br>
</font></div>
<div><font face="courier new, monospace">SIMFS CT</font></div>
<div><font face="courier new, monospace">----------------</font></div>
<div><font face="courier new, monospace">root@vz102:~# dd bs=1M count=512 if=/dev/zero of=test conv=fdatasync</font></div>
<div><font face="courier new, monospace">512+0 records in</font></div>
<div><font face="courier new, monospace">512+0 records out</font></div>
<div><font face="courier new, monospace">536870912 bytes (537 MB) copied, 12.6913 s, 42.3 MB/s</font></div>
<div><font face="courier new, monospace">root@vz102:~# df -T</font></div>
<div><font face="courier new, monospace">Filesystem Type 1K-blocks Used Available Use% Mounted on</font></div>
<div><font face="courier new, monospace">/dev/simfs simfs 4194304 1365500 2828804 33% /</font></div>
<div><font face="courier new, monospace">none devtmpfs 262144 4 262140 1% /dev</font></div>
<div><font face="courier new, monospace">none tmpfs 52432 52 52380 1% /run</font></div>
<div><font face="courier new, monospace">none tmpfs 5120 0 5120 0% /run/lock</font></div>
<div><font face="courier new, monospace">none tmpfs 262144 0 262144 0% /run/shm</font></div>
<div><font face="courier new, monospace">root@vz102:~# </font></div>
</div>
<div><font face="courier new, monospace"><br>
</font></div>
<div><font face="arial, helvetica, sans-serif">Regards,</font></div>
<div><font face="arial, helvetica, sans-serif"><br>
</font></div>
<div><font face="arial, helvetica, sans-serif">J J</font></div>
<div><font face="arial, helvetica, sans-serif"><br>
</font></div>
<font face="arial, helvetica, sans-serif"></font><br>
</div>
</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On Fri, May 2, 2014 at 2:10 PM, jjs - mainphrame <span dir="ltr">
<<a href="mailto:jjs@mainphrame.com" target="_blank">jjs@mainphrame.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">You know the saying, "when something seems too good to be true"...
<div><br>
</div>
<div>I just installed centos 6.5 and openvz on an older machine, and when I built an ubuntu 12.04 CT I noticed that ploop is now the default layout. Cool. So I built another ubuntu12.04 CT, identical in every way except that I specified smifs, so I could do
a quick performance comparison.</div>
<div><br>
</div>
<div>First I did a quick timed dd run, then I ran dbench with varying numbers of clients.</div>
<div><br>
</div>
<div>The simfs CT showed performance roughly similar to the host, which was not too surprising.</div>
<div>What did surprise me was that the ploop CT showed performance which was significantly better than the host, in both the dd test and the dbench tests.</div>
<div><br>
</div>
<div>I know someone will tell me "dbench is a terrible benchmark" but it's also a standard. Of course, if anyone knows a "better" benchmark, I'd love to try it.</div>
<div><br>
</div>
<div>Regards,</div>
<div><br>
</div>
<div>J J</div>
</div>
</blockquote>
</div>
<br>
</div>
<br>
<fieldset></fieldset> <br>
</div>
</div>
<pre>_______________________________________________
Users mailing list
<a href="mailto:Users@openvz.org" target="_blank">Users@openvz.org</a>
<a href="https://lists.openvz.org/mailman/listinfo/users" target="_blank">https://lists.openvz.org/mailman/listinfo/users</a>
</pre>
</blockquote>
<br>
</div></div></div><div><div class="h5">
<br>
_______________________________________________<br>
Users mailing list<br>
<a href="mailto:Users@openvz.org" target="_blank">Users@openvz.org</a><br>
<a href="https://lists.openvz.org/mailman/listinfo/users" target="_blank">https://lists.openvz.org/mailman/listinfo/users</a><br>
<br>
</div></div></blockquote>
</div>
<br>
</div><div><div class="h5">
<br>
<fieldset></fieldset> <br>
<pre>_______________________________________________
Users mailing list
<a href="mailto:Users@openvz.org" target="_blank">Users@openvz.org</a>
<a href="https://lists.openvz.org/mailman/listinfo/users" target="_blank">https://lists.openvz.org/mailman/listinfo/users</a>
</pre>
</div></div></blockquote>
<br>
</div>
</div>
</div><div><div class="h5">
<br>
_______________________________________________<br>
Users mailing list<br>
<a href="mailto:Users@openvz.org" target="_blank">Users@openvz.org</a><br>
<a href="https://lists.openvz.org/mailman/listinfo/users" target="_blank">https://lists.openvz.org/mailman/listinfo/users</a><br>
<br>
</div></div></blockquote>
</div>
<br>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote><div><div class="h5">
<blockquote type="cite">
<div><span>_______________________________________________</span><br>
<span>Users mailing list</span><br>
<span><a href="mailto:Users@openvz.org" target="_blank">Users@openvz.org</a></span><br>
<span><a href="https://lists.openvz.org/mailman/listinfo/users" target="_blank">https://lists.openvz.org/mailman/listinfo/users</a></span><br>
</div>
</blockquote>
</div></div></div>
<br>_______________________________________________<br>
Users mailing list<br>
<a href="mailto:Users@openvz.org">Users@openvz.org</a><br>
<a href="https://lists.openvz.org/mailman/listinfo/users" target="_blank">https://lists.openvz.org/mailman/listinfo/users</a><br>
<br></blockquote></div><br></div>