[Users] RE: slow fsync rate

Wed Sep 22 03:21:24 EDT 2010

Hello guys,

I have the same problem with XFS(without LVM), I tested
newest(2.6.32-openvz-bykovsky.1) kernel with patch provided in Bugzilla, and
2.6.27-openvz-briullov kernel the  results are almost the same is where work
ground ? 

1)      With2.6.32-openvz-bykovsky.1  kernel and patch:

Command: uname -a

Linux vps1_public 2.6.32-openvz-bykovsky.1 #1 SMP Mon Sep 13 18:12:07 EEST
2010 i686 Pentium(R) Dual-Core CPU E5200 @ 2.50GHz GenuineIntel GNU/Linux

Command:  sysbench --test=fileio --file-num=1 --file-total-size=50G
--file-fsync-all=on --file-test-mode=seqrewr --max-time=100
--file-block-size=4096 --max-requests=0 run

sysbench 0.4.10:  multi-threaded system evaluation benchmark

Running the test with following options:

Number of threads: 1

Extra file open flags: 0

1 files, 50Gb each

50Gb total file size

Block size 4Kb

Calling fsync() after each write operation.

Using synchronous I/O mode

Doing sequential rewrite test

Threads started!

Time limit exceeded, exiting...

Done.

Operations performed:  0 Read, 2372 Write, 2372 Other = 4744 Total

Read 0b  Written 9.2656Mb  Total transferred 9.2656Mb  (94.86Kb/sec)

   23.72 Requests/sec executed

Test execution summary:

    total time:                          100.0207s

    total number of events:              2372

    total time taken by event execution: 100.0155

    per-request statistics:

         min:                                 24.82ms

         avg:                                 42.17ms

         max:                                132.44ms

         approx.  95 percentile:              66.20ms

Threads fairness:

    events (avg/stddev):           2372.0000/0.00

    execution time (avg/stddev):   100.0155/0.00

2)      With 2.6.27-openvz-briullov.1 kernel:

Command: uname -a

Linux vps1_public 2.6.27-openvz-briullov.1-r4ovz-i7 #1 SMP Thu Dec 17
06:37:57 EET 2009 i686 Pentium(R) Dual-Core CPU E5200 @ 2.50GHz GenuineIntel
GNU/Linux

Command:  sysbench --test=fileio --file-num=1 --file-total-size=50G
--file-fsync-all=on --file-test-mode=seqrewr --max-time=100
--file-block-size=4096 --max-requests=0 run

sysbench 0.4.10:  multi-threaded system evaluation benchmark

Running the test with following options:

Number of threads: 1

Extra file open flags: 0

1 files, 50Gb each

50Gb total file size

Block size 4Kb

Calling fsync() after each write operation.

Using synchronous I/O mode

Doing sequential rewrite test

Threads started!

Time limit exceeded, exiting...

Done.

Operations performed:  0 Read, 2245 Write, 2245 Other = 4490 Total

Read 0b  Written 8.7695Mb  Total transferred 8.7695Mb  (89.757Kb/sec)

   22.44 Requests/sec executed

Test execution summary:

    total time:                          100.0481s

    total number of events:              2245

    total time taken by event execution: 100.0430

    per-request statistics:

         min:                                 24.82ms

         avg:                                 44.56ms

         max:                                149.03ms

         approx.  95 percentile:              74.47ms

Threads fairness:

    events (avg/stddev):           2245.0000/0.00

    execution time (avg/stddev):   100.0430/0.00

From: users-bounces at openvz.org [mailto:users-bounces at openvz.org] On Behalf
Of Martin Maurer
Sent: Thursday, September 02, 2010 17:21
To: users at openvz.org
Cc: Kirill Korotaev
Subject: RE: [Users] RE: slow fsync rate

Hi Kir,

Yes, we already see the fix and I can confirm that CFQ scheduler is working
again.

We are already using it in our Proxmox VE, now also with OpenVZ in the
2.6.32 kernel – released today.

(my post was from 25.8, was held back due to size issues).

Thanks,

Br, martin

From: Kirill Korotaev [mailto:dev at parallels.com] 
Sent: Donnerstag, 02. September 2010 14:27
To: users at openvz.org
Cc: Martin Maurer
Subject: Re: [Users] RE: slow fsync rate

Pavel has already confirmed  that this is a problem of mainstream group CFS
scheduler, see bug in bugzilla for fix.

Will be applied soon. Use deadline scheduler for some time until it is fixed
in the tree.

Thanks,

Kirill

On Aug 25, 2010, at 10:33 , Martin Maurer wrote:

Hi Kir,

Yes, the performance number shows that it does not use the cache so if
anyone want to run high load like a database on such a system the
performance loss will be significant and the system will be more or less
unusable (compared to current stable). I got this behavior ONLY with 2.6.32
OpenVZ kernels. E.g. also the standard Debian Squeeze Kernel give the
expected performance, the OpenVZ enabled kernel from Squeeze doesn´t. (see
also  <http://lists.debian.org/debian-kernel/2010/08/msg00217.html>
http://lists.debian.org/debian-kernel/2010/08/msg00217.html).

We tested on several hardware and distributions, e.g. with single drives,
hardware raid controllers with BBU – e.g. the Intel Modular Server but also
on single drives on ICH7/8/9. The logs always shows that the disk has read
and write cache enabled, but the values are just bad.

Can you reproduce the issue (just run the sysbench)?

Br, Martin

From: users-bounces at openvz.org [mailto:users-bounces at openvz.org] On Behalf
Of Kirill Korotaev
Sent: Dienstag, 24. August 2010 21:42
To: users at openvz.org
Subject: Re: [Users] RE: slow fsync rate

These numbers very much resemble fsync() rate with write cache enabled
(~1000/sec) and disabled (50-70/sec).

check write cache settings with hdparm + check whether you have barrier
mount option on ext3.

I believe, 2.6.32 is just more honest on fsync and really forces drive to
save data. While earlier kernels

did this only with BARRIER mount option (or ignored this problem at all)...

Thanks,

Kirill

On Aug 24, 2010, at 22:53 , Martin Maurer wrote:

Hi all,

I just tested the latest OpenVZ kernel 2.6.32 (bykovsky) from today with
sysbench, I just got 63.12 Requests/sec executed (see below).

2.6.18 and 2.6.24 OpenVZ Kernels performs well,   I got 1074.81 Requests/sec
executed. What's wrong here, why is the 2.6.32 branch so slow regarding
fsyns/sec? (I am using ext3)

____

OpenVZ 2.6.24:~# sysbench --test=fileio --file-num=1 --file-total-size=50G
--file-fsync-all=on --file-test-mode=seqrewr --max-time=100
--file-block-size=4096 --max-requests=0 run

sysbench 0.4.12:  multi-threaded system evaluation benchmark

Running the test with following options:

Number of threads: 1

Extra file open flags: 0

1 files, 50Gb each

50Gb total file size

Block size 4Kb

Calling fsync() after each write operation.

Using synchronous I/O mode

Doing sequential rewrite test

Threads started!

Time limit exceeded, exiting...

Done.

Operations performed:  0 Read, 107485 Write, 107485 Other = 214970 Total

Read 0b  Written 419.86Mb  Total transferred 419.86Mb  (4.1985Mb/sec)

1074.81 Requests/sec executed

Test execution summary:

    total time:                          100.0033s

    total number of events:              107485

    total time taken by event execution: 99.9460

    per-request statistics:

         min:                                  0.54ms

         avg:                                  0.93ms

         max:                                 97.34ms

         approx.  95 percentile:               0.87ms

Threads fairness:

    events (avg/stddev):           107485.0000/0.00

    execution time (avg/stddev):   99.9460/0.00

_____

Testing newest Kernel: 2.6.32-bykovsky.1 #1 SMP Mon Aug 23 19:59:54 MSD 2010
x86_64

I just got 63.12 Requests/sec executed. Here are the details, can someone
reproduce this?

_____

OpenVZ 2.6.32:~#  sysbench --test=fileio --file-num=1 --file-total-size=50G
--file-fsync-all=on --file-test-mode=seqrewr --max-time=100
--file-block-size=4096 --max-requests=0 run

sysbench 0.4.12:  multi-threaded system evaluation benchmark

Running the test with following options:

Number of threads: 1

Extra file open flags: 0

1 files, 50Gb each

50Gb total file size

Block size 4Kb

Calling fsync() after each write operation.

Using synchronous I/O mode

Doing sequential rewrite test

Threads started!

Time limit exceeded, exiting...

Done.

Operations performed:  0 Read, 6312 Write, 6312 Other = 12624 Total

Read 0b  Written 24.656Mb  Total transferred 24.656Mb  (252.46Kb/sec)

   63.12 Requests/sec executed

Test execution summary:

    total time:                          100.0070s

    total number of events:              6312

    total time taken by event execution: 99.9838

    per-request statistics:

         min:                                  7.70ms

         avg:                                 15.84ms

         max:                                260.40ms

         approx.  95 percentile:              16.70ms

Threads fairness:

    events (avg/stddev):           6312.0000/0.00

    execution time (avg/stddev):   99.9838/0.00

___

Br, Martin

From: users-bounces at openvz.org [mailto:users-bounces at openvz.org] On Behalf
Of Dietmar Maurer
Sent: Dienstag, 24. August 2010 10:36
To: users at openvz.org
Subject: [Users] slow fsync rate

Hi all,

we observe very slow fsync rates on newer 2.6.32 kernel with OpenVZ:

It is possible to reproduce the problem with sysbench:

# sysbench --test=fileio --file-num=1 --file-total-size=50G
--file-fsync-all=on --file-test-mode=seqrewr --max-time=100
--file-block-size=4096 --max-requests=0 run

Requests/sec executed is considerable slower on OpenVZ kernel (factor 20 on
Intel Modular Server).

Can someone reproduce that problem?

- Dietmar

<ATT00001..c>

<ATT00001..c>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://openvz.org/pipermail/users/attachments/20100922/079a1029/attachment-0001.html