[Users] RE: slow fsync rate

Kirill Korotaev dev at parallels.com
Thu Sep 2 08:26:58 EDT 2010


Pavel has already confirmed  that this is a problem of mainstream group CFS scheduler, see bug in bugzilla for fix.
Will be applied soon. Use deadline scheduler for some time until it is fixed in the tree.

Thanks,
Kirill


On Aug 25, 2010, at 10:33 , Martin Maurer wrote:

Hi Kir,

Yes, the performance number shows that it does not use the cache so if anyone want to run high load like a database on such a system the performance loss will be significant and the system will be more or less unusable (compared to current stable). I got this behavior ONLY with 2.6.32 OpenVZ kernels. E.g. also the standard Debian Squeeze Kernel give the expected performance, the OpenVZ enabled kernel from Squeeze doesn´t. (see also http://lists.debian.org/debian-kernel/2010/08/msg00217.html).

We tested on several hardware and distributions, e.g. with single drives, hardware raid controllers with BBU – e.g. the Intel Modular Server but also on single drives on ICH7/8/9. The logs always shows that the disk has read and write cache enabled, but the values are just bad.

Can you reproduce the issue (just run the sysbench)?

Br, Martin

From: users-bounces at openvz.org<mailto:users-bounces at openvz.org> [mailto:users-bounces at openvz.org] On Behalf Of Kirill Korotaev
Sent: Dienstag, 24. August 2010 21:42
To: users at openvz.org<mailto:users at openvz.org>
Subject: Re: [Users] RE: slow fsync rate

These numbers very much resemble fsync() rate with write cache enabled (~1000/sec) and disabled (50-70/sec).
check write cache settings with hdparm + check whether you have barrier mount option on ext3.

I believe, 2.6.32 is just more honest on fsync and really forces drive to save data. While earlier kernels
did this only with BARRIER mount option (or ignored this problem at all)...

Thanks,
Kirill


On Aug 24, 2010, at 22:53 , Martin Maurer wrote:


Hi all,

I just tested the latest OpenVZ kernel 2.6.32 (bykovsky) from today with sysbench, I just got 63.12 Requests/sec executed (see below).
2.6.18 and 2.6.24 OpenVZ Kernels performs well,   I got 1074.81 Requests/sec executed. What's wrong here, why is the 2.6.32 branch so slow regarding fsyns/sec? (I am using ext3)
____
OpenVZ 2.6.24:~# sysbench --test=fileio --file-num=1 --file-total-size=50G --file-fsync-all=on --file-test-mode=seqrewr --max-time=100 --file-block-size=4096 --max-requests=0 run
sysbench 0.4.12:  multi-threaded system evaluation benchmark

Running the test with following options:
Number of threads: 1

Extra file open flags: 0
1 files, 50Gb each
50Gb total file size
Block size 4Kb
Calling fsync() after each write operation.
Using synchronous I/O mode
Doing sequential rewrite test
Threads started!
Time limit exceeded, exiting...
Done.

Operations performed:  0 Read, 107485 Write, 107485 Other = 214970 Total
Read 0b  Written 419.86Mb  Total transferred 419.86Mb  (4.1985Mb/sec)
1074.81 Requests/sec executed

Test execution summary:
    total time:                          100.0033s
    total number of events:              107485
    total time taken by event execution: 99.9460
    per-request statistics:
         min:                                  0.54ms
         avg:                                  0.93ms
         max:                                 97.34ms
         approx.  95 percentile:               0.87ms

Threads fairness:
    events (avg/stddev):           107485.0000/0.00
    execution time (avg/stddev):   99.9460/0.00

_____


Testing newest Kernel: 2.6.32-bykovsky.1 #1 SMP Mon Aug 23 19:59:54 MSD 2010 x86_64

I just got 63.12 Requests/sec executed. Here are the details, can someone reproduce this?
_____
OpenVZ 2.6.32:~#  sysbench --test=fileio --file-num=1 --file-total-size=50G --file-fsync-all=on --file-test-mode=seqrewr --max-time=100 --file-block-size=4096 --max-requests=0 run

sysbench 0.4.12:  multi-threaded system evaluation benchmark

Running the test with following options:
Number of threads: 1

Extra file open flags: 0
1 files, 50Gb each
50Gb total file size
Block size 4Kb
Calling fsync() after each write operation.
Using synchronous I/O mode
Doing sequential rewrite test
Threads started!
Time limit exceeded, exiting...
Done.

Operations performed:  0 Read, 6312 Write, 6312 Other = 12624 Total
Read 0b  Written 24.656Mb  Total transferred 24.656Mb  (252.46Kb/sec)
   63.12 Requests/sec executed

Test execution summary:
    total time:                          100.0070s
    total number of events:              6312
    total time taken by event execution: 99.9838
    per-request statistics:
         min:                                  7.70ms
         avg:                                 15.84ms
         max:                                260.40ms
         approx.  95 percentile:              16.70ms

Threads fairness:
    events (avg/stddev):           6312.0000/0.00
    execution time (avg/stddev):   99.9838/0.00
___

Br, Martin

From: users-bounces at openvz.org<mailto:users-bounces at openvz.org> [mailto:users-bounces at openvz.org] On Behalf Of Dietmar Maurer
Sent: Dienstag, 24. August 2010 10:36
To: users at openvz.org<mailto:users at openvz.org>
Subject: [Users] slow fsync rate

Hi all,

we observe very slow fsync rates on newer 2.6.32 kernel with OpenVZ:

It is possible to reproduce the problem with sysbench:

# sysbench --test=fileio --file-num=1 --file-total-size=50G --file-fsync-all=on --file-test-mode=seqrewr --max-time=100 --file-block-size=4096 --max-requests=0 run

Requests/sec executed is considerable slower on OpenVZ kernel (factor 20 on Intel Modular Server).

Can someone reproduce that problem?

- Dietmar


<ATT00001..c>

<ATT00001..c>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://openvz.org/pipermail/users/attachments/20100902/45fde43c/attachment-0001.html


More information about the Users mailing list