Strange delete performance using XFS

Strange delete performance using XFS

[Thomas Kaehn]

Hi,

I've got a strange problem on one machine using XFS. Deleting large
directories (containing about 100000 files, 20k each) using "rm -rf"
lasts nearly as long as creating the the files using a bash loop.

The machine is running Debian Sarge with a vanilla 2.6.20.3 kernel.
CPU: Dual Xeon(TM) CPU 3.20GHz
RAM: 4 GB
RAID10: 4x 320 GB disks connected to 3ware 9550SXU-8LP 
 (Firmware Version = FE9X 3.08.00.004)

The XFS was first created using default options and later on with
"-d su=64k,sw=2 -l su=64k" which improved overall performance
but not delete performance.

Has anyone realized similar effects? On a different server (Dell 6850)
the directory can be deleted within seconds. What could be the reason
for the huge difference in delete performance?

Please see below for "time" output. 

| # time for i in `seq 1 100000`; do dd if=/dev/zero of=$i bs=1k count=20 >/dev/null 2>&1; done
| 
| real    6m6.814s
| user    0m30.290s
| sys     2m42.562s
| # time rm -rf y
| 
| real    5m18.034s
| user    0m0.036s
| sys     0m8.169s

In contrast to this the result on the Dell machine looks more
reasonable:

| # time for i in `seq 1 100000`; do dd if=/dev/zero of=$i bs=1k count=20 >/dev/null 2>&1; done
| 
| real    9m26.658s
| user    0m24.134s
| sys     3m3.623s
| # time rm -rf x
| 
| real    0m10.254s
| user    0m0.124s
| sys     0m10.105s

Ciao,
Thomas

PS: Using JFS and ext3 it is also possible to delete the above directory
in a couple of seconds. Only XFS seems problematic in this regard on
this system.
-- 
Thomas Kähn                   WESTEND GmbH  |  Internet-Business-Provider
Technik                       CISCO Systems Partner - Authorized Reseller
                              Im Süsterfeld 6          Tel 0241/701333-18
tk@westend.com                D-52072 Aachen              Fax 0241/911879
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Die Gesellschaft ist eingetragen im Handelsregister Aachen unter HRB 7608
Geschäftsführer:           Thomas Neugebauer, Thomas Heller, Michael Kolb

Re: Strange delete performance using XFS

[Justin Piszcz]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 2459 bytes --]



On Wed, 4 Apr 2007, Thomas Kaehn wrote:

> Hi,
>
> I've got a strange problem on one machine using XFS. Deleting large
> directories (containing about 100000 files, 20k each) using "rm -rf"
> lasts nearly as long as creating the the files using a bash loop.
>
> The machine is running Debian Sarge with a vanilla 2.6.20.3 kernel.
> CPU: Dual Xeon(TM) CPU 3.20GHz
> RAM: 4 GB
> RAID10: 4x 320 GB disks connected to 3ware 9550SXU-8LP
> (Firmware Version = FE9X 3.08.00.004)
>
> The XFS was first created using default options and later on with
> "-d su=64k,sw=2 -l su=64k" which improved overall performance
> but not delete performance.
>
> Has anyone realized similar effects? On a different server (Dell 6850)
> the directory can be deleted within seconds. What could be the reason
> for the huge difference in delete performance?
>
> Please see below for "time" output.
>
> | # time for i in `seq 1 100000`; do dd if=/dev/zero of=$i bs=1k count=20 >/dev/null 2>&1; done
> |
> | real    6m6.814s
> | user    0m30.290s
> | sys     2m42.562s
> | # time rm -rf y
> |
> | real    5m18.034s
> | user    0m0.036s
> | sys     0m8.169s
>
> In contrast to this the result on the Dell machine looks more
> reasonable:
>
> | # time for i in `seq 1 100000`; do dd if=/dev/zero of=$i bs=1k count=20 >/dev/null 2>&1; done
> |
> | real    9m26.658s
> | user    0m24.134s
> | sys     3m3.623s
> | # time rm -rf x
> |
> | real    0m10.254s
> | user    0m0.124s
> | sys     0m10.105s
>
> Ciao,
> Thomas
>
> PS: Using JFS and ext3 it is also possible to delete the above directory
> in a couple of seconds. Only XFS seems problematic in this regard on
> this system.
> -- 
> Thomas Kähn                   WESTEND GmbH  |  Internet-Business-Provider
> Technik                       CISCO Systems Partner - Authorized Reseller
>                              Im Süsterfeld 6          Tel 0241/701333-18
> tk@westend.com                D-52072 Aachen              Fax 0241/911879
> - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
> Die Gesellschaft ist eingetragen im Handelsregister Aachen unter HRB 7608
> Geschäftsführer:           Thomas Neugebauer, Thomas Heller, Michael Kolb
>
>

Deletes on XFS is one area that is a little slower than other filesystems. 
You can increase the log size during the creation of the filesystem and 
also increase logbufs to 8 and that might help.

Re: Strange delete performance using XFS

[Thomas Kaehn]

Hi Justin,

On Wed, Apr 04, 2007 at 09:29:46AM -0400, Justin Piszcz wrote:
> >Please see below for "time" output.
> >
> >| # time for i in `seq 1 100000`; do dd if=/dev/zero of=$i bs=1k count=20 
> >>/dev/null 2>&1; done
> >|
> >| real    6m6.814s
> >| user    0m30.290s
> >| sys     2m42.562s
> >| # time rm -rf y
> >|
> >| real    5m18.034s
> >| user    0m0.036s
> >| sys     0m8.169s

> Deletes on XFS is one area that is a little slower than other filesystems. 
> You can increase the log size during the creation of the filesystem and 
> also increase logbufs to 8 and that might help.

Thanks for your suggestions.

I also tried to increase the log size and logbufs mount option. This
optimizes create and delete times to the above values (with default options
both are around 9-10 minutes).

The strange thing is that on a similar Dell machines using XFS, too,
deletes take only ten seconds which would match user and system time.

More than five minutes for deleting 100000 files where ext3 needs
3 seconds on the same machine is actually more than a little bit slower
- to my mind there must be something wrong. JFS needs around 18 seconds.

However I am not sure if the problem is hardware or software related.
I've also tried to use the newest 3ware firmware - but this did not lead
to an improvement.

Ciao,
Thomas
-- 
Thomas Kähn                   WESTEND GmbH  |  Internet-Business-Provider
Technik                       CISCO Systems Partner - Authorized Reseller
                              Im Süsterfeld 6          Tel 0241/701333-18
tk@westend.com                D-52072 Aachen              Fax 0241/911879
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Die Gesellschaft ist eingetragen im Handelsregister Aachen unter HRB 7608
Geschäftsführer:           Thomas Neugebauer, Thomas Heller, Michael Kolb

Re: Strange delete performance using XFS

[Justin Piszcz]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 2100 bytes --]



On Wed, 4 Apr 2007, Thomas Kaehn wrote:

> Hi Justin,
>
> On Wed, Apr 04, 2007 at 09:29:46AM -0400, Justin Piszcz wrote:
>>> Please see below for "time" output.
>>>
>>> | # time for i in `seq 1 100000`; do dd if=/dev/zero of=$i bs=1k count=20
>>>> /dev/null 2>&1; done
>>> |
>>> | real    6m6.814s
>>> | user    0m30.290s
>>> | sys     2m42.562s
>>> | # time rm -rf y
>>> |
>>> | real    5m18.034s
>>> | user    0m0.036s
>>> | sys     0m8.169s
>
>> Deletes on XFS is one area that is a little slower than other filesystems.
>> You can increase the log size during the creation of the filesystem and
>> also increase logbufs to 8 and that might help.
>
> Thanks for your suggestions.
>
> I also tried to increase the log size and logbufs mount option. This
> optimizes create and delete times to the above values (with default options
> both are around 9-10 minutes).
>
> The strange thing is that on a similar Dell machines using XFS, too,
> deletes take only ten seconds which would match user and system time.
>
> More than five minutes for deleting 100000 files where ext3 needs
> 3 seconds on the same machine is actually more than a little bit slower
> - to my mind there must be something wrong. JFS needs around 18 seconds.
>
> However I am not sure if the problem is hardware or software related.
> I've also tried to use the newest 3ware firmware - but this did not lead
> to an improvement.
>
> Ciao,
> Thomas
> -- 
> Thomas Kähn                   WESTEND GmbH  |  Internet-Business-Provider
> Technik                       CISCO Systems Partner - Authorized Reseller
>                              Im Süsterfeld 6          Tel 0241/701333-18
> tk@westend.com                D-52072 Aachen              Fax 0241/911879
> - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
> Die Gesellschaft ist eingetragen im Handelsregister Aachen unter HRB 7608
> Geschäftsführer:           Thomas Neugebauer, Thomas Heller, Michael Kolb
>
>

I am running some benchmarks with SW raid and will prevent my findings 
shortly.

Re: Strange delete performance using XFS

[Justin Piszcz]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 2448 bytes --]



On Wed, 4 Apr 2007, Thomas Kaehn wrote:

> Hi Justin,
>
> On Wed, Apr 04, 2007 at 09:29:46AM -0400, Justin Piszcz wrote:
>>> Please see below for "time" output.
>>>
>>> | # time for i in `seq 1 100000`; do dd if=/dev/zero of=$i bs=1k count=20
>>>> /dev/null 2>&1; done
>>> |
>>> | real    6m6.814s
>>> | user    0m30.290s
>>> | sys     2m42.562s
>>> | # time rm -rf y
>>> |
>>> | real    5m18.034s
>>> | user    0m0.036s
>>> | sys     0m8.169s
>
>> Deletes on XFS is one area that is a little slower than other filesystems.
>> You can increase the log size during the creation of the filesystem and
>> also increase logbufs to 8 and that might help.
>
> Thanks for your suggestions.
>
> I also tried to increase the log size and logbufs mount option. This
> optimizes create and delete times to the above values (with default options
> both are around 9-10 minutes).
>
> The strange thing is that on a similar Dell machines using XFS, too,
> deletes take only ten seconds which would match user and system time.
>
> More than five minutes for deleting 100000 files where ext3 needs
> 3 seconds on the same machine is actually more than a little bit slower
> - to my mind there must be something wrong. JFS needs around 18 seconds.
>
> However I am not sure if the problem is hardware or software related.
> I've also tried to use the newest 3ware firmware - but this did not lead
> to an improvement.
>
> Ciao,
> Thomas
> -- 
> Thomas Kähn                   WESTEND GmbH  |  Internet-Business-Provider
> Technik                       CISCO Systems Partner - Authorized Reseller
>                              Im Süsterfeld 6          Tel 0241/701333-18
> tk@westend.com                D-52072 Aachen              Fax 0241/911879
> - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
> Die Gesellschaft ist eingetragen im Handelsregister Aachen unter HRB 7608
> Geschäftsführer:           Thomas Neugebauer, Thomas Heller, Michael Kolb
>
>

The benchmark:
$ time for i in `seq 1 100000`; do dd if=/dev/zero of=$i bs=1k count=20 
>/dev/null 2>&1; done

1. Six 400GB SATA drives using SW RAID5:
real    6m24.411s
user    0m43.097s
sys     2m17.350s

2. Four Raptor 150 ADFD drives using SW RAID5:
real    3m16.962s
user    0m42.899s
sys     2m15.420s

3. Two Raptor 74GB *GD drives using SW RAID1:
real    3m19.241s
user    0m41.731s
sys     2m15.873s

Re: Strange delete performance using XFS

[Justin Piszcz]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 2288 bytes --]



On Wed, 4 Apr 2007, Justin Piszcz wrote:

>
>
> On Wed, 4 Apr 2007, Thomas Kaehn wrote:
>
>> Hi Justin,
>> 
>> On Wed, Apr 04, 2007 at 09:29:46AM -0400, Justin Piszcz wrote:
>>>> Please see below for "time" output.
>>>> 
>>>> | # time for i in `seq 1 100000`; do dd if=/dev/zero of=$i bs=1k count=20
>>>>> /dev/null 2>&1; done
>>>> |
>>>> | real    6m6.814s
>>>> | user    0m30.290s
>>>> | sys     2m42.562s
>>>> | # time rm -rf y
>>>> |
>>>> | real    5m18.034s
>>>> | user    0m0.036s
>>>> | sys     0m8.169s
>> 
>>> Deletes on XFS is one area that is a little slower than other filesystems.
>>> You can increase the log size during the creation of the filesystem and
>>> also increase logbufs to 8 and that might help.
>> 
>> Thanks for your suggestions.
>> 
>> I also tried to increase the log size and logbufs mount option. This
>> optimizes create and delete times to the above values (with default options
>> both are around 9-10 minutes).
>> 
>> The strange thing is that on a similar Dell machines using XFS, too,
>> deletes take only ten seconds which would match user and system time.
>> 
>> More than five minutes for deleting 100000 files where ext3 needs
>> 3 seconds on the same machine is actually more than a little bit slower
>> - to my mind there must be something wrong. JFS needs around 18 seconds.
>> 
>> However I am not sure if the problem is hardware or software related.
>> I've also tried to use the newest 3ware firmware - but this did not lead
>> to an improvement.
>> 
>> Ciao,
>> Thomas
>> -- 
>> Thomas Kähn                   WESTEND GmbH  |  Internet-Business-Provider
>> Technik                       CISCO Systems Partner - Authorized Reseller
>>                              Im Süsterfeld 6          Tel 0241/701333-18
>> tk@westend.com                D-52072 Aachen              Fax 0241/911879
>> - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
>> Die Gesellschaft ist eingetragen im Handelsregister Aachen unter HRB 7608
>> Geschäftsführer:           Thomas Neugebauer, Thomas Heller, Michael Kolb
>> 
>> 
>
> I am running some benchmarks with SW raid and will prevent my findings 
> shortly.

Removal tests coming shortly, benchmarking is always interesting.

Re: Strange delete performance using XFS

[Justin Piszcz]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 2965 bytes --]



On Wed, 4 Apr 2007, Justin Piszcz wrote:

>
>
> On Wed, 4 Apr 2007, Thomas Kaehn wrote:
>
>> Hi Justin,
>> 
>> On Wed, Apr 04, 2007 at 09:29:46AM -0400, Justin Piszcz wrote:
>>>> Please see below for "time" output.
>>>> 
>>>> | # time for i in `seq 1 100000`; do dd if=/dev/zero of=$i bs=1k count=20
>>>>> /dev/null 2>&1; done
>>>> |
>>>> | real    6m6.814s
>>>> | user    0m30.290s
>>>> | sys     2m42.562s
>>>> | # time rm -rf y
>>>> |
>>>> | real    5m18.034s
>>>> | user    0m0.036s
>>>> | sys     0m8.169s
>> 
>>> Deletes on XFS is one area that is a little slower than other filesystems.
>>> You can increase the log size during the creation of the filesystem and
>>> also increase logbufs to 8 and that might help.
>> 
>> Thanks for your suggestions.
>> 
>> I also tried to increase the log size and logbufs mount option. This
>> optimizes create and delete times to the above values (with default options
>> both are around 9-10 minutes).
>> 
>> The strange thing is that on a similar Dell machines using XFS, too,
>> deletes take only ten seconds which would match user and system time.
>> 
>> More than five minutes for deleting 100000 files where ext3 needs
>> 3 seconds on the same machine is actually more than a little bit slower
>> - to my mind there must be something wrong. JFS needs around 18 seconds.
>> 
>> However I am not sure if the problem is hardware or software related.
>> I've also tried to use the newest 3ware firmware - but this did not lead
>> to an improvement.
>> 
>> Ciao,
>> Thomas
>> -- 
>> Thomas Kähn                   WESTEND GmbH  |  Internet-Business-Provider
>> Technik                       CISCO Systems Partner - Authorized Reseller
>>                              Im Süsterfeld 6          Tel 0241/701333-18
>> tk@westend.com                D-52072 Aachen              Fax 0241/911879
>> - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
>> Die Gesellschaft ist eingetragen im Handelsregister Aachen unter HRB 7608
>> Geschäftsführer:           Thomas Neugebauer, Thomas Heller, Michael Kolb
>> 
>> 
>
> The benchmark:
> $ time for i in `seq 1 100000`; do dd if=/dev/zero of=$i bs=1k count=20 
>> /dev/null 2>&1; done
>
> 1. Six 400GB SATA drives using SW RAID5:
> real    6m24.411s
> user    0m43.097s
> sys     2m17.350s
>
> 2. Four Raptor 150 ADFD drives using SW RAID5:
> real    3m16.962s
> user    0m42.899s
> sys     2m15.420s
>
> 3. Two Raptor 74GB *GD drives using SW RAID1:
> real    3m19.241s
> user    0m41.731s
> sys     2m15.873s
>
>

The removals:

The benchmark:
$ time rm -rf test

1. Six 400GB SATA drives using SW RAID5:
real    0m33.996s
user    0m0.057s
sys     0m8.101s

2. Four Raptor 150 ADFD drives using SW RAID5:
real    0m43.967s
user    0m0.071s
sys     0m8.340s

3. Two Raptor 74GB *GD drives using SW RAID1:
real    0m32.965s
user    0m0.049s
sys     0m6.307s

Re: Strange delete performance using XFS

[Justin Piszcz]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 2968 bytes --]



On Wed, 4 Apr 2007, Justin Piszcz wrote:

>
>
> On Wed, 4 Apr 2007, Thomas Kaehn wrote:
>
>> Hi Justin,
>> 
>> On Wed, Apr 04, 2007 at 09:29:46AM -0400, Justin Piszcz wrote:
>>>> Please see below for "time" output.
>>>> 
>>>> | # time for i in `seq 1 100000`; do dd if=/dev/zero of=$i bs=1k count=20
>>>>> /dev/null 2>&1; done
>>>> |
>>>> | real    6m6.814s
>>>> | user    0m30.290s
>>>> | sys     2m42.562s
>>>> | # time rm -rf y
>>>> |
>>>> | real    5m18.034s
>>>> | user    0m0.036s
>>>> | sys     0m8.169s
>> 
>>> Deletes on XFS is one area that is a little slower than other filesystems.
>>> You can increase the log size during the creation of the filesystem and
>>> also increase logbufs to 8 and that might help.
>> 
>> Thanks for your suggestions.
>> 
>> I also tried to increase the log size and logbufs mount option. This
>> optimizes create and delete times to the above values (with default options
>> both are around 9-10 minutes).
>> 
>> The strange thing is that on a similar Dell machines using XFS, too,
>> deletes take only ten seconds which would match user and system time.
>> 
>> More than five minutes for deleting 100000 files where ext3 needs
>> 3 seconds on the same machine is actually more than a little bit slower
>> - to my mind there must be something wrong. JFS needs around 18 seconds.
>> 
>> However I am not sure if the problem is hardware or software related.
>> I've also tried to use the newest 3ware firmware - but this did not lead
>> to an improvement.
>> 
>> Ciao,
>> Thomas
>> -- 
>> Thomas Kähn                   WESTEND GmbH  |  Internet-Business-Provider
>> Technik                       CISCO Systems Partner - Authorized Reseller
>>                              Im Süsterfeld 6          Tel 0241/701333-18
>> tk@westend.com                D-52072 Aachen              Fax 0241/911879
>> - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
>> Die Gesellschaft ist eingetragen im Handelsregister Aachen unter HRB 7608
>> Geschäftsführer:           Thomas Neugebauer, Thomas Heller, Michael Kolb
>> 
>> 
>
> The benchmark:
> $ time for i in `seq 1 100000`; do dd if=/dev/zero of=$i bs=1k count=20 
>> /dev/null 2>&1; done
>
> 1. Six 400GB SATA drives using SW RAID5:
> real    6m24.411s
> user    0m43.097s
> sys     2m17.350s
>
> 2. Four Raptor 150 ADFD drives using SW RAID5:
> real    3m16.962s
> user    0m42.899s
> sys     2m15.420s
>
> 3. Two Raptor 74GB *GD drives using SW RAID1:
> real    3m19.241s
> user    0m41.731s
> sys     2m15.873s
>
>

I used the DEFAULT create options for XFS as I find it highly optimizes 
itself (at least with SW raid) with the exception of the ROOT FS, I had 
that optimized awhile ago and I kept it:

/dev/md2       /            xfs 
logbufs=8,logbsize=262144,biosize=16,noatime,nodiratime,nobarrier  0  1


For my regular RAID5s though I use defaults,noatime.

Justin.

Re: Strange delete performance using XFS

[Thomas Kaehn]

Hi Justin,

On Wed, Apr 04, 2007 at 10:12:48AM -0400, Justin Piszcz wrote:
> On Wed, 4 Apr 2007, Justin Piszcz wrote:
> >On Wed, 4 Apr 2007, Thomas Kaehn wrote:
> >$ time for i in `seq 1 100000`; do dd if=/dev/zero of=$i bs=1k count=20 
> >>/dev/null 2>&1; done
> >
> >1. Six 400GB SATA drives using SW RAID5:
> >real    6m24.411s
> >user    0m43.097s
> >sys     2m17.350s
> >
> 
> The removals:
> The benchmark:
> $ time rm -rf test
> 
> 1. Six 400GB SATA drives using SW RAID5:
> real    0m33.996s
> user    0m0.057s
> sys     0m8.101s

thanks for your bechmark. To my mind this clearly shows that my
setup is wrong at some point. I'll try again with your mount options.

Ciao,
Thomas
-- 
Thomas Kähn                   WESTEND GmbH  |  Internet-Business-Provider
Technik                       CISCO Systems Partner - Authorized Reseller
                              Im Süsterfeld 6          Tel 0241/701333-18
tk@westend.com                D-52072 Aachen              Fax 0241/911879
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Die Gesellschaft ist eingetragen im Handelsregister Aachen unter HRB 7608
Geschäftsführer:           Thomas Neugebauer, Thomas Heller, Michael Kolb

Re: Strange delete performance using XFS

[Justin Piszcz]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1588 bytes --]



On Wed, 4 Apr 2007, Thomas Kaehn wrote:

> Hi Justin,
>
> On Wed, Apr 04, 2007 at 10:12:48AM -0400, Justin Piszcz wrote:
>> On Wed, 4 Apr 2007, Justin Piszcz wrote:
>>> On Wed, 4 Apr 2007, Thomas Kaehn wrote:
>>> $ time for i in `seq 1 100000`; do dd if=/dev/zero of=$i bs=1k count=20
>>>> /dev/null 2>&1; done
>>>
>>> 1. Six 400GB SATA drives using SW RAID5:
>>> real    6m24.411s
>>> user    0m43.097s
>>> sys     2m17.350s
>>>
>>
>> The removals:
>> The benchmark:
>> $ time rm -rf test
>>
>> 1. Six 400GB SATA drives using SW RAID5:
>> real    0m33.996s
>> user    0m0.057s
>> sys     0m8.101s
>
> thanks for your bechmark. To my mind this clearly shows that my
> setup is wrong at some point. I'll try again with your mount options.
>
> Ciao,
> Thomas
> -- 
> Thomas Kähn                   WESTEND GmbH  |  Internet-Business-Provider
> Technik                       CISCO Systems Partner - Authorized Reseller
>                              Im Süsterfeld 6          Tel 0241/701333-18
> tk@westend.com                D-52072 Aachen              Fax 0241/911879
> - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
> Die Gesellschaft ist eingetragen im Handelsregister Aachen unter HRB 7608
> Geschäftsführer:           Thomas Neugebauer, Thomas Heller, Michael Kolb
>
>

My guess is mkfs.xfs cannot optimzie for your array like it can with a SW 
RAID device because it cannot see what is undereath it.  Have you tried 
making a SW RAID?  I also use optimized parameters for my SW RAID1/5 as 
well FYI.

Justin.

Re: Strange delete performance using XFS

[Thomas Kaehn]

Hi Justin,

On Wed, Apr 04, 2007 at 10:24:42AM -0400, Justin Piszcz wrote:
> >On Wed, Apr 04, 2007 at 10:12:48AM -0400, Justin Piszcz wrote:
> >>On Wed, 4 Apr 2007, Justin Piszcz wrote:
> >>>On Wed, 4 Apr 2007, Thomas Kaehn wrote:
> >>>$ time for i in `seq 1 100000`; do dd if=/dev/zero of=$i bs=1k count=20
> >>>>/dev/null 2>&1; done
> >>>
> My guess is mkfs.xfs cannot optimzie for your array like it can with a SW 
> RAID device because it cannot see what is undereath it.  Have you tried 
> making a SW RAID?  I also use optimized parameters for my SW RAID1/5 as 
> well FYI.

I guess this might be the problem. I've already tried to alter
the stripe unit to match the RAID stripe size: "-d su=64k,sw=2 -l su=64k".

Maybe the 3ware controller can't deal with the kind of read and write
patterns needed by XFS. But in this case other people should have
realized similar problems.

On a different system with a 3ware 9500S-4LP using 4 disks as RAID5
setup I get a better (but not really good) result for delete
performance (I've taken only 50000 files in this case as the system's
CPU is much slower):

| # time for i in `seq 1 50000`; do dd if=/dev/zero of=$i
| bs=1k count=20 >/dev/null 2>&1; done
| 
| real    18m21.643s
| user    0m55.727s
| sys     3m12.140s
| backup:/srv/x# cd ..
| backup:/srv# rm -rf x
| 
| # time rm -rf x
| 
| real    5m7.845s
| user    0m0.160s
| sys     0m11.369s


Ciao,
Thomas
-- 
Thomas Kähn                   WESTEND GmbH  |  Internet-Business-Provider
Technik                       CISCO Systems Partner - Authorized Reseller
                              Im Süsterfeld 6          Tel 0241/701333-18
tk@westend.com                D-52072 Aachen              Fax 0241/911879
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Die Gesellschaft ist eingetragen im Handelsregister Aachen unter HRB 7608
Geschäftsführer:           Thomas Neugebauer, Thomas Heller, Michael Kolb

Re: Strange delete performance using XFS

[Chris Wedgwood]

On Wed, Apr 04, 2007 at 03:05:35PM +0200, Thomas Kaehn wrote:

> I've got a strange problem on one machine using XFS. Deleting large
> directories (containing about 100000 files, 20k each) using "rm -rf"
> lasts nearly as long as creating the the files using a bash loop.

quite possible

> RAM: 4 GB
> RAID10: 4x 320 GB disks connected to 3ware 9550SXU-8LP
>  (Firmware Version = FE9X 3.08.00.004)


> The XFS was first created using default options and later on with
> "-d su=64k,sw=2 -l su=64k" which improved overall performance
> but not delete performance.

have you tried w/o using the hw raid?

> Has anyone realized similar effects? On a different server (Dell
> 6850) the directory can be deleted within seconds. What could be the
> reason for the huge difference in delete performance?

a lot of log updates; does the other server have a battery-backed
write-cache like many cards to these days?

> | # time for i in `seq 1 100000`; do dd if=/dev/zero of=$i bs=1k count=20 >/dev/null 2>&1; done
> |
> | real    6m6.814s
> | user    0m30.290s
> | sys     2m42.562s

that's about the same as my quick single-spindle cheap-desktop test
here

> | # time rm -rf y
> |
> | real    5m18.034s
> | user    0m0.036s
> | sys     0m8.169s

v2 logs? what logbufs & logbsize is used?

testing with my cheap crappy desktop workstation thing with a
single disk I get "1m25.004s" for the delete

Re: Strange delete performance using XFS

[Justin Piszcz]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 2068 bytes --]



On Wed, 4 Apr 2007, Thomas Kaehn wrote:

> Hi Justin,
>
> On Wed, Apr 04, 2007 at 09:29:46AM -0400, Justin Piszcz wrote:
>>> Please see below for "time" output.
>>>
>>> | # time for i in `seq 1 100000`; do dd if=/dev/zero of=$i bs=1k count=20
>>>> /dev/null 2>&1; done
>>> |
>>> | real    6m6.814s
>>> | user    0m30.290s
>>> | sys     2m42.562s
>>> | # time rm -rf y
>>> |
>>> | real    5m18.034s
>>> | user    0m0.036s
>>> | sys     0m8.169s
>
>> Deletes on XFS is one area that is a little slower than other filesystems.
>> You can increase the log size during the creation of the filesystem and
>> also increase logbufs to 8 and that might help.
>
> Thanks for your suggestions.
>
> I also tried to increase the log size and logbufs mount option. This
> optimizes create and delete times to the above values (with default options
> both are around 9-10 minutes).
>
> The strange thing is that on a similar Dell machines using XFS, too,
> deletes take only ten seconds which would match user and system time.
>
> More than five minutes for deleting 100000 files where ext3 needs
> 3 seconds on the same machine is actually more than a little bit slower
> - to my mind there must be something wrong. JFS needs around 18 seconds.
>
> However I am not sure if the problem is hardware or software related.
> I've also tried to use the newest 3ware firmware - but this did not lead
> to an improvement.
>
> Ciao,
> Thomas
> -- 
> Thomas Kähn                   WESTEND GmbH  |  Internet-Business-Provider
> Technik                       CISCO Systems Partner - Authorized Reseller
>                              Im Süsterfeld 6          Tel 0241/701333-18
> tk@westend.com                D-52072 Aachen              Fax 0241/911879
> - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
> Die Gesellschaft ist eingetragen im Handelsregister Aachen unter HRB 7608
> Geschäftsführer:           Thomas Neugebauer, Thomas Heller, Michael Kolb
>

For the ext3, try time bash -c 'rm -rf test; sync'

Re: Strange delete performance using XFS

[Justin Piszcz]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 2183 bytes --]



On Wed, 4 Apr 2007, Thomas Kaehn wrote:

> Hi Justin,
>
> On Wed, Apr 04, 2007 at 10:24:42AM -0400, Justin Piszcz wrote:
>>> On Wed, Apr 04, 2007 at 10:12:48AM -0400, Justin Piszcz wrote:
>>>> On Wed, 4 Apr 2007, Justin Piszcz wrote:
>>>>> On Wed, 4 Apr 2007, Thomas Kaehn wrote:
>>>>> $ time for i in `seq 1 100000`; do dd if=/dev/zero of=$i bs=1k count=20
>>>>>> /dev/null 2>&1; done
>>>>>
>> My guess is mkfs.xfs cannot optimzie for your array like it can with a SW
>> RAID device because it cannot see what is undereath it.  Have you tried
>> making a SW RAID?  I also use optimized parameters for my SW RAID1/5 as
>> well FYI.
>
> I guess this might be the problem. I've already tried to alter
> the stripe unit to match the RAID stripe size: "-d su=64k,sw=2 -l su=64k".
>
> Maybe the 3ware controller can't deal with the kind of read and write
> patterns needed by XFS. But in this case other people should have
> realized similar problems.
>
> On a different system with a 3ware 9500S-4LP using 4 disks as RAID5
> setup I get a better (but not really good) result for delete
> performance (I've taken only 50000 files in this case as the system's
> CPU is much slower):
>
> | # time for i in `seq 1 50000`; do dd if=/dev/zero of=$i
> | bs=1k count=20 >/dev/null 2>&1; done
> |
> | real    18m21.643s
> | user    0m55.727s
> | sys     3m12.140s
> | backup:/srv/x# cd ..
> | backup:/srv# rm -rf x
> |
> | # time rm -rf x
> |
> | real    5m7.845s
> | user    0m0.160s
> | sys     0m11.369s
>
>
> Ciao,
> Thomas
> -- 
> Thomas Kähn                   WESTEND GmbH  |  Internet-Business-Provider
> Technik                       CISCO Systems Partner - Authorized Reseller
>                              Im Süsterfeld 6          Tel 0241/701333-18
> tk@westend.com                D-52072 Aachen              Fax 0241/911879
> - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
> Die Gesellschaft ist eingetragen im Handelsregister Aachen unter HRB 7608
> Geschäftsführer:           Thomas Neugebauer, Thomas Heller, Michael Kolb
>

What do you get with ext3 when using time bash -c 'rm -f file; sync'

Re: Strange delete performance using XFS

[Thomas Kaehn]

Hi Chris,

On Wed, Apr 04, 2007 at 08:45:23AM -0700, Chris Wedgwood wrote:
> On Wed, Apr 04, 2007 at 03:05:35PM +0200, Thomas Kaehn wrote:
> > The XFS was first created using default options and later on with
> > "-d su=64k,sw=2 -l su=64k" which improved overall performance
> > but not delete performance.
> 
> have you tried w/o using the hw raid?

I'am going to test with a single disk in the same machine. 

> > Has anyone realized similar effects? On a different server (Dell
> > 6850) the directory can be deleted within seconds. What could be the
> > reason for the huge difference in delete performance?
> 
> a lot of log updates; does the other server have a battery-backed
> write-cache like many cards to these days?

The Dell system has got a battery-backed write-cache. The 3ware system
has no battery unit. However it's supposed to provide write cache, too.
At least I've enabled it in the RAID's configuration. The controller
has got more than 100MB memory.

> > | # time for i in `seq 1 100000`; do dd if=/dev/zero of=$i bs=1k count=20 >/dev/null 2>&1; done
> > |
> > | real    6m6.814s
> > | user    0m30.290s
> > | sys     2m42.562s
> 
> that's about the same as my quick single-spindle cheap-desktop test
> here
> 
> > | # time rm -rf y
> > |
> > | real    5m18.034s
> > | user    0m0.036s
> > | sys     0m8.169s
> 
> v2 logs? what logbufs & logbsize is used?
> 
> testing with my cheap crappy desktop workstation thing with a
> single disk I get "1m25.004s" for the delete

Your delete time sounds more sensible to me. The file system was
created first with default options - later on I tried to match
the RAID's stripe size, increased the log size and mounted
with logbufs=8:

log stripe unit specified, using v2 logs
meta-data=/dev/sda1              isize=256    agcount=8, agsize=125008 blks
         =                       sectsz=512  
data     =                       bsize=4096   blocks=1000032, imaxpct=25
         =                       sunit=16     swidth=32 blks, unwritten=1
naming   =version 2              bsize=4096  
log      =internal log           bsize=4096   blocks=16384, version=2
         =                       sectsz=512   sunit=16 blks
realtime =none                   extsz=65536  blocks=0, rtextents=0


Ciao,
Thomas
-- 
Thomas Kähn                   WESTEND GmbH  |  Internet-Business-Provider
Technik                       CISCO Systems Partner - Authorized Reseller
                              Im Süsterfeld 6          Tel 0241/701333-18
tk@westend.com                D-52072 Aachen              Fax 0241/911879
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Die Gesellschaft ist eingetragen im Handelsregister Aachen unter HRB 7608
Geschäftsführer:           Thomas Neugebauer, Thomas Heller, Michael Kolb

Re: Strange delete performance using XFS

[Thomas Kaehn]

Hi Justin,

On Wed, Apr 04, 2007 at 02:36:37PM -0400, Justin Piszcz wrote:
> For the ext3, try time bash -c 'rm -rf test; sync'

# time bash -c 'rm -rf y; sync'

real    0m1.592s
user    0m0.032s
sys     0m1.408s


Ciao,
Thomas
-- 
Thomas Kähn                   WESTEND GmbH  |  Internet-Business-Provider
Technik                       CISCO Systems Partner - Authorized Reseller
                              Im Süsterfeld 6          Tel 0241/701333-18
tk@westend.com                D-52072 Aachen              Fax 0241/911879
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Die Gesellschaft ist eingetragen im Handelsregister Aachen unter HRB 7608
Geschäftsführer:           Thomas Neugebauer, Thomas Heller, Michael Kolb

Re: Strange delete performance using XFS

[Thomas Kaehn]

Hi Justin,

On Wed, Apr 04, 2007 at 10:13:36AM -0400, Justin Piszcz wrote:
> I used the DEFAULT create options for XFS as I find it highly optimizes 
> itself (at least with SW raid) with the exception of the ROOT FS, I had 
> that optimized awhile ago and I kept it:
> 
> /dev/md2       /            xfs 
> logbufs=8,logbsize=262144,biosize=16,noatime,nodiratime,nobarrier  0  1
> 
> 
> For my regular RAID5s though I use defaults,noatime.

I've disabled barriers, too, and performance increased dramatically.

However I am not aware of the consequences of disabling write barriers.
The FAQ generally recommends using write barriers except when having
a battery-backed cache (this 3ware has not).

| # time for i in `seq 1 100000`; do dd if=/dev/zero of=$i bs=1k count=20 >/dev/null 2>&1; done
| 
| real    3m52.182s
| user    0m30.482s
| sys     3m16.152s
| 
| # time \rm -rf y
| 
| real    0m16.327s
| user    0m0.052s
| sys     0m8.305s

So I am unsure if disabling is an option for me. I could imagine
that write barriers are not properly supported by 3ware or have
to be fine tuned at the kernel or SCSI level.

Ciao,
Thomas
-- 
Thomas Kähn                   WESTEND GmbH  |  Internet-Business-Provider
Technik                       CISCO Systems Partner - Authorized Reseller
                              Im Süsterfeld 6          Tel 0241/701333-18
tk@westend.com                D-52072 Aachen              Fax 0241/911879
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Die Gesellschaft ist eingetragen im Handelsregister Aachen unter HRB 7608
Geschäftsführer:           Thomas Neugebauer, Thomas Heller, Michael Kolb

Re: Strange delete performance using XFS

[Thomas Kaehn]

Hi Chris,

On Thu, Apr 05, 2007 at 09:28:03AM +0200, Thomas Kaehn wrote:
> On Wed, Apr 04, 2007 at 08:45:23AM -0700, Chris Wedgwood wrote:
> > On Wed, Apr 04, 2007 at 03:05:35PM +0200, Thomas Kaehn wrote:
> > > The XFS was first created using default options and later on with
> > > "-d su=64k,sw=2 -l su=64k" which improved overall performance
> > > but not delete performance.
> > 
> > have you tried w/o using the hw raid?
> 
> I'am going to test with a single disk in the same machine. 

this is what I got with a single disk (defaults for mkfs.xfs, logbufs=8 for
mount) in the same machine:

| # time for i in `seq 1 100000`; do dd if=/dev/zero of=$i bs=1k count=20 >/dev/null 2>&1; done
| 
| real    11m22.487s
| user    0m30.278s
| sys     2m33.762s
| # time \rm -rf y
| 
| real    8m20.963s
| user    0m0.056s
| sys     0m7.968s

So there is no improvement for a single disk.

Ciao,
Thomas
-- 
Thomas Kähn                   WESTEND GmbH  |  Internet-Business-Provider
Technik                       CISCO Systems Partner - Authorized Reseller
                              Im Süsterfeld 6          Tel 0241/701333-18
tk@westend.com                D-52072 Aachen              Fax 0241/911879
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Die Gesellschaft ist eingetragen im Handelsregister Aachen unter HRB 7608
Geschäftsführer:           Thomas Neugebauer, Thomas Heller, Michael Kolb

Re: Strange delete performance using XFS

[Justin Piszcz]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1620 bytes --]



On Thu, 5 Apr 2007, Thomas Kaehn wrote:

> Hi Chris,
>
> On Thu, Apr 05, 2007 at 09:28:03AM +0200, Thomas Kaehn wrote:
>> On Wed, Apr 04, 2007 at 08:45:23AM -0700, Chris Wedgwood wrote:
>>> On Wed, Apr 04, 2007 at 03:05:35PM +0200, Thomas Kaehn wrote:
>>>> The XFS was first created using default options and later on with
>>>> "-d su=64k,sw=2 -l su=64k" which improved overall performance
>>>> but not delete performance.
>>>
>>> have you tried w/o using the hw raid?
>>
>> I'am going to test with a single disk in the same machine.
>
> this is what I got with a single disk (defaults for mkfs.xfs, logbufs=8 for
> mount) in the same machine:
>
> | # time for i in `seq 1 100000`; do dd if=/dev/zero of=$i bs=1k count=20 >/dev/null 2>&1; done
> |
> | real    11m22.487s
> | user    0m30.278s
> | sys     2m33.762s
> | # time \rm -rf y
> |
> | real    8m20.963s
> | user    0m0.056s
> | sys     0m7.968s
>
> So there is no improvement for a single disk.
>
> Ciao,
> Thomas
> -- 
> Thomas Kähn                   WESTEND GmbH  |  Internet-Business-Provider
> Technik                       CISCO Systems Partner - Authorized Reseller
>                              Im Süsterfeld 6          Tel 0241/701333-18
> tk@westend.com                D-52072 Aachen              Fax 0241/911879
> - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
> Die Gesellschaft ist eingetragen im Handelsregister Aachen unter HRB 7608
> Geschäftsführer:           Thomas Neugebauer, Thomas Heller, Michael Kolb
>
>

What kind of disks are you using? Maybe just slow disks??

Re: Strange delete performance using XFS

[Thomas Kaehn]

Hi Justin,

On Thu, Apr 05, 2007 at 06:21:43AM -0400, Justin Piszcz wrote:
> >So there is no improvement for a single disk.
> >
> What kind of disks are you using? Maybe just slow disks??

I am using the following disks:

http://www.westerndigital.com/en/products/products.asp?DriveID=233

When disabling write barriers delete times are OK. I think that
the 3ware RAID controller could have a problem with it. I'll try to
contact 3ware in order to come to know if this feature is supported or
not.

Additionally I am going to try out some advices presented in the
3ware knowledge base.

Ciao,
Thomas
-- 
Thomas Kähn                   WESTEND GmbH  |  Internet-Business-Provider
Technik                       CISCO Systems Partner - Authorized Reseller
                              Im Süsterfeld 6          Tel 0241/701333-18
tk@westend.com                D-52072 Aachen              Fax 0241/911879
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Die Gesellschaft ist eingetragen im Handelsregister Aachen unter HRB 7608
Geschäftsführer:           Thomas Neugebauer, Thomas Heller, Michael Kolb

Re: Strange delete performance using XFS

[Justin Piszcz]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1461 bytes --]



On Thu, 5 Apr 2007, Thomas Kaehn wrote:

> Hi Justin,
>
> On Thu, Apr 05, 2007 at 06:21:43AM -0400, Justin Piszcz wrote:
>>> So there is no improvement for a single disk.
>>>
>> What kind of disks are you using? Maybe just slow disks??
>
> I am using the following disks:
>
> http://www.westerndigital.com/en/products/products.asp?DriveID=233
>
> When disabling write barriers delete times are OK. I think that
> the 3ware RAID controller could have a problem with it. I'll try to
> contact 3ware in order to come to know if this feature is supported or
> not.
>
> Additionally I am going to try out some advices presented in the
> 3ware knowledge base.
>
> Ciao,
> Thomas
> -- 
> Thomas Kähn                   WESTEND GmbH  |  Internet-Business-Provider
> Technik                       CISCO Systems Partner - Authorized Reseller
>                              Im Süsterfeld 6          Tel 0241/701333-18
> tk@westend.com                D-52072 Aachen              Fax 0241/911879
> - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
> Die Gesellschaft ist eingetragen im Handelsregister Aachen unter HRB 7608
> Geschäftsführer:           Thomas Neugebauer, Thomas Heller, Michael Kolb
>
>

Ah, ok-- Keep us updated/let us know if you get any new findings/etc.

Something else you can try as well is turning off NCQ, that gave me a 
10-35% performance boost depending on the benchmark.

Justin.

Re: Strange delete performance using XFS

[Chris Wedgwood]

On Thu, Apr 05, 2007 at 09:28:03AM +0200, Thomas Kaehn wrote:

> The Dell system has got a battery-backed write-cache. The 3ware
> system has no battery unit. However it's supposed to provide write
> cache, too.

That sounds like the main reason for the difference.  The Dell's raid
system can safely buffer outstanding writes and flsuh them, the 3ware
can't so it stalls waiting fot the disks to catch up.

You could run blktrace and watch what's going on in both cases to
verify this.

The numbers do seem a little low for a raid array all the same, I'd be
tempted to just use the 3ware as a JBOD and use sw, but I'm arguably
biased, I've had so many reliability and performance problems with hw
raid over the years I will almost always use sw raid given the choice.

Re: Strange delete performance using XFS

[Peter Grandi]

[ ... slowness deleting a lot of inodes ... ]

>> The Dell system has got a battery-backed write-cache. The
>> 3ware system has no battery unit. However it's supposed to
>> provide write cache, too.

Whatever, but you cannot have both metadata consistency and high
speed without fully reliable hw...

> [ ... ] The Dell's raid system can safely buffer outstanding
> writes and flsuh them, the 3ware can't so it stalls waiting
> fot the disks to catch up. [ ... ] I'd be tempted to just use
> the 3ware as a JBOD and use sw, but I'm arguably biased, I've
> had so many reliability and performance problems with hw raid
> over the years

Uhm, I had a friend that worked for a middling storage system
vendor and he was telling me horror stories about bugs and
misdesigns in their quite popular RAID products.

3ware seem to me one of the more reliable RAID brands, their
cautious approach may be why they are slower above.

> I will almost always use sw raid given the choice.

Does not buy you a lot over well designed RAID host adapter.
it is also a lot less convenient.

Re: Strange delete performance using XFS

[Thomas Kaehn]

Hi Peter,

On Fri, Apr 06, 2007 at 08:02:28PM +0100, Peter Grandi wrote:
> > [ ... ] The Dell's raid system can safely buffer outstanding
> > writes and flsuh them, the 3ware can't so it stalls waiting
> > fot the disks to catch up. [ ... ] I'd be tempted to just use
> > the 3ware as a JBOD and use sw, but I'm arguably biased, I've
> > had so many reliability and performance problems with hw raid
> > over the years
> 
> Uhm, I had a friend that worked for a middling storage system
> vendor and he was telling me horror stories about bugs and
> misdesigns in their quite popular RAID products.
> 
> 3ware seem to me one of the more reliable RAID brands, their
> cautious approach may be why they are slower above.

your are probably right. 3ware didn't answer yet. However I've
found an option in the controller to set the "storsave" policy.

In the default profile FUA (force unit access) commands are only
acknowledged directly in case a BBU is present. Otherwise the controller
waits until the data is written to disk.

When selecting the "performance" profile FUA commands are ignored and
delete time lowers to a couple of seconds.

So the behaviour of the controller should be considered a feature. But
I am still astonished how slow deletes were in the first place.
This might be a bug or incompatibility anyhow.

Thanks to all others for your suggestions. I'll inform you in case
3ware has news for me.

Ciao,
Thomas
-- 
Thomas Kähn                   WESTEND GmbH  |  Internet-Business-Provider
Technik                       CISCO Systems Partner - Authorized Reseller
                              Im Süsterfeld 6          Tel 0241/701333-18
tk@westend.com                D-52072 Aachen              Fax 0241/911879
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Die Gesellschaft ist eingetragen im Handelsregister Aachen unter HRB 7608
Geschäftsführer:           Thomas Neugebauer, Thomas Heller, Michael Kolb