[Users] RE: slow fsync rate
Dmitrijs Jerihovs
de at balticom.lv
Wed Sep 22 03:17:27 EDT 2010
Hello guys,
I have the same problem with XFS(without LVM), I tested
newest(2.6.32-openvz-bykovsky.1) kernel with patch provided in Bugzilla, and
2.6.27-openvz-briullov kernel the results are almost the same is where work
ground ?
1) With2.6.32-openvz-bykovsky.1 kernel and patch:
Command: uname -a
Linux vps1_public 2.6.32-openvz-bykovsky.1 #1 SMP Mon Sep 13 18:12:07 EEST
2010 i686 Pentium(R) Dual-Core CPU E5200 @ 2.50GHz GenuineIntel GNU/Linux
Command: sysbench --test=fileio --file-num=1 --file-total-size=50G
--file-fsync-all=on --file-test-mode=seqrewr --max-time=100
--file-block-size=4096 --max-requests=0 run
sysbench 0.4.10: multi-threaded system evaluation benchmark
Running the test with following options:
Number of threads: 1
Extra file open flags: 0
1 files, 50Gb each
50Gb total file size
Block size 4Kb
Calling fsync() after each write operation.
Using synchronous I/O mode
Doing sequential rewrite test
Threads started!
Time limit exceeded, exiting...
Done.
Operations performed: 0 Read, 2372 Write, 2372 Other = 4744 Total
Read 0b Written 9.2656Mb Total transferred 9.2656Mb (94.86Kb/sec)
23.72 Requests/sec executed
Test execution summary:
total time: 100.0207s
total number of events: 2372
total time taken by event execution: 100.0155
per-request statistics:
min: 24.82ms
avg: 42.17ms
max: 132.44ms
approx. 95 percentile: 66.20ms
Threads fairness:
events (avg/stddev): 2372.0000/0.00
execution time (avg/stddev): 100.0155/0.00
2) With 2.6.27-openvz-briullov.1 kernel:
Command: uname -a
Linux vps1_public 2.6.27-openvz-briullov.1-r4ovz-i7 #1 SMP Thu Dec 17
06:37:57 EET 2009 i686 Pentium(R) Dual-Core CPU E5200 @ 2.50GHz GenuineIntel
GNU/Linux
Command: sysbench --test=fileio --file-num=1 --file-total-size=50G
--file-fsync-all=on --file-test-mode=seqrewr --max-time=100
--file-block-size=4096 --max-requests=0 run
sysbench 0.4.10: multi-threaded system evaluation benchmark
Running the test with following options:
Number of threads: 1
Extra file open flags: 0
1 files, 50Gb each
50Gb total file size
Block size 4Kb
Calling fsync() after each write operation.
Using synchronous I/O mode
Doing sequential rewrite test
Threads started!
Time limit exceeded, exiting...
Done.
Operations performed: 0 Read, 2245 Write, 2245 Other = 4490 Total
Read 0b Written 8.7695Mb Total transferred 8.7695Mb (89.757Kb/sec)
22.44 Requests/sec executed
Test execution summary:
total time: 100.0481s
total number of events: 2245
total time taken by event execution: 100.0430
per-request statistics:
min: 24.82ms
avg: 44.56ms
max: 149.03ms
approx. 95 percentile: 74.47ms
Threads fairness:
events (avg/stddev): 2245.0000/0.00
execution time (avg/stddev): 100.0430/0.00
From: users-bounces at openvz.org [mailto:users-bounces at openvz.org] On Behalf
Of Martin Maurer
Sent: Thursday, September 02, 2010 17:21
To: users at openvz.org
Cc: Kirill Korotaev
Subject: RE: [Users] RE: slow fsync rate
Hi Kir,
Yes, we already see the fix and I can confirm that CFQ scheduler is working
again.
We are already using it in our Proxmox VE, now also with OpenVZ in the
2.6.32 kernel released today.
(my post was from 25.8, was held back due to size issues).
Thanks,
Br, martin
From: Kirill Korotaev [mailto:dev at parallels.com]
Sent: Donnerstag, 02. September 2010 14:27
To: users at openvz.org
Cc: Martin Maurer
Subject: Re: [Users] RE: slow fsync rate
Pavel has already confirmed that this is a problem of mainstream group CFS
scheduler, see bug in bugzilla for fix.
Will be applied soon. Use deadline scheduler for some time until it is fixed
in the tree.
Thanks,
Kirill
On Aug 25, 2010, at 10:33 , Martin Maurer wrote:
Hi Kir,
Yes, the performance number shows that it does not use the cache so if
anyone want to run high load like a database on such a system the
performance loss will be significant and the system will be more or less
unusable (compared to current stable). I got this behavior ONLY with 2.6.32
OpenVZ kernels. E.g. also the standard Debian Squeeze Kernel give the
expected performance, the OpenVZ enabled kernel from Squeeze doesn´t. (see
also <http://lists.debian.org/debian-kernel/2010/08/msg00217.html>
http://lists.debian.org/debian-kernel/2010/08/msg00217.html).
We tested on several hardware and distributions, e.g. with single drives,
hardware raid controllers with BBU e.g. the Intel Modular Server but also
on single drives on ICH7/8/9. The logs always shows that the disk has read
and write cache enabled, but the values are just bad.
Can you reproduce the issue (just run the sysbench)?
Br, Martin
From: users-bounces at openvz.org [mailto:users-bounces at openvz.org] On Behalf
Of Kirill Korotaev
Sent: Dienstag, 24. August 2010 21:42
To: users at openvz.org
Subject: Re: [Users] RE: slow fsync rate
These numbers very much resemble fsync() rate with write cache enabled
(~1000/sec) and disabled (50-70/sec).
check write cache settings with hdparm + check whether you have barrier
mount option on ext3.
I believe, 2.6.32 is just more honest on fsync and really forces drive to
save data. While earlier kernels
did this only with BARRIER mount option (or ignored this problem at all)...
Thanks,
Kirill
On Aug 24, 2010, at 22:53 , Martin Maurer wrote:
Hi all,
I just tested the latest OpenVZ kernel 2.6.32 (bykovsky) from today with
sysbench, I just got 63.12 Requests/sec executed (see below).
2.6.18 and 2.6.24 OpenVZ Kernels performs well, I got 1074.81 Requests/sec
executed. What's wrong here, why is the 2.6.32 branch so slow regarding
fsyns/sec? (I am using ext3)
____
OpenVZ 2.6.24:~# sysbench --test=fileio --file-num=1 --file-total-size=50G
--file-fsync-all=on --file-test-mode=seqrewr --max-time=100
--file-block-size=4096 --max-requests=0 run
sysbench 0.4.12: multi-threaded system evaluation benchmark
Running the test with following options:
Number of threads: 1
Extra file open flags: 0
1 files, 50Gb each
50Gb total file size
Block size 4Kb
Calling fsync() after each write operation.
Using synchronous I/O mode
Doing sequential rewrite test
Threads started!
Time limit exceeded, exiting...
Done.
Operations performed: 0 Read, 107485 Write, 107485 Other = 214970 Total
Read 0b Written 419.86Mb Total transferred 419.86Mb (4.1985Mb/sec)
1074.81 Requests/sec executed
Test execution summary:
total time: 100.0033s
total number of events: 107485
total time taken by event execution: 99.9460
per-request statistics:
min: 0.54ms
avg: 0.93ms
max: 97.34ms
approx. 95 percentile: 0.87ms
Threads fairness:
events (avg/stddev): 107485.0000/0.00
execution time (avg/stddev): 99.9460/0.00
_____
Testing newest Kernel: 2.6.32-bykovsky.1 #1 SMP Mon Aug 23 19:59:54 MSD 2010
x86_64
I just got 63.12 Requests/sec executed. Here are the details, can someone
reproduce this?
_____
OpenVZ 2.6.32:~# sysbench --test=fileio --file-num=1 --file-total-size=50G
--file-fsync-all=on --file-test-mode=seqrewr --max-time=100
--file-block-size=4096 --max-requests=0 run
sysbench 0.4.12: multi-threaded system evaluation benchmark
Running the test with following options:
Number of threads: 1
Extra file open flags: 0
1 files, 50Gb each
50Gb total file size
Block size 4Kb
Calling fsync() after each write operation.
Using synchronous I/O mode
Doing sequential rewrite test
Threads started!
Time limit exceeded, exiting...
Done.
Operations performed: 0 Read, 6312 Write, 6312 Other = 12624 Total
Read 0b Written 24.656Mb Total transferred 24.656Mb (252.46Kb/sec)
63.12 Requests/sec executed
Test execution summary:
total time: 100.0070s
total number of events: 6312
total time taken by event execution: 99.9838
per-request statistics:
min: 7.70ms
avg: 15.84ms
max: 260.40ms
approx. 95 percentile: 16.70ms
Threads fairness:
events (avg/stddev): 6312.0000/0.00
execution time (avg/stddev): 99.9838/0.00
___
Br, Martin
From: users-bounces at openvz.org [mailto:users-bounces at openvz.org] On Behalf
Of Dietmar Maurer
Sent: Dienstag, 24. August 2010 10:36
To: users at openvz.org
Subject: [Users] slow fsync rate
Hi all,
we observe very slow fsync rates on newer 2.6.32 kernel with OpenVZ:
It is possible to reproduce the problem with sysbench:
# sysbench --test=fileio --file-num=1 --file-total-size=50G
--file-fsync-all=on --file-test-mode=seqrewr --max-time=100
--file-block-size=4096 --max-requests=0 run
Requests/sec executed is considerable slower on OpenVZ kernel (factor 20 on
Intel Modular Server).
Can someone reproduce that problem?
- Dietmar
<ATT00001..c>
<ATT00001..c>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://openvz.org/pipermail/users/attachments/20100922/391153ad/attachment-0001.html
More information about the Users
mailing list