LWN.net article: creating 1 billion files -> XFS looses

LWN.net article: creating 1 billion files -> XFS looses

[Michael Monnerie]

[-- Attachment #1.1: Type: text/plain, Size: 597 bytes --]

The subject is a bit harsh, but overall the article says:
XFS is slowest on creating and deleting a billion files
XFS fsck needs 30GB RAM to fsck that 100TB filesystem.

http://lwn.net/SubscriberLink/400629/3fb4bc34d6223b32/

-- 
mit freundlichen Grüssen,
Michael Monnerie, Ing. BSc

it-management Internet Services
http://proteger.at [gesprochen: Prot-e-schee]
Tel: 0660 / 415 65 31

****** Aktuelles Radiointerview! ******
http://www.it-podcast.at/aktuelle-sendung.html

// Wir haben im Moment zwei Häuser zu verkaufen:
// http://zmi.at/langegg/
// http://zmi.at/haus2009/

[-- Attachment #1.2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: LWN.net article: creating 1 billion files -> XFS looses

[Christoph Hellwig]

On Thu, Aug 19, 2010 at 01:12:45PM +0200, Michael Monnerie wrote:
> The subject is a bit harsh, but overall the article says:
> XFS is slowest on creating and deleting a billion files
> XFS fsck needs 30GB RAM to fsck that 100TB filesystem.
> 
> http://lwn.net/SubscriberLink/400629/3fb4bc34d6223b32/

The creation and deletion performance is a known issue, and too a large
extent fixes by the new delaylog code.  We're not quite as fast as ext4
yet, but it's getting close.

The repair result looks a lot like the pre-3.1.0 xfsprogs repair.

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: LWN.net article: creating 1 billion files -> XFS looses

[Michael Monnerie]

[-- Attachment #1.1: Type: Text/Plain, Size: 1064 bytes --]

On Donnerstag, 19. August 2010 Christoph Hellwig wrote:
> The creation and deletion performance is a known issue, and too a
>  large extent fixes by the new delaylog code.  We're not quite as
>  fast as ext4 yet, but it's getting close.
> 
> The repair result looks a lot like the pre-3.1.0 xfsprogs repair.
 
Yes I know. I thought some XFS dev might contact the Author to do some 
re-testing, as a reputation is quickly destroyed by such articles and 
takes long to be returned. Just this week I had a friend in a FS 
discussion saying "ins't XFS destroying/zeroing files on power 
failure?". That information is ancient, but things like that stay in 
peoples brain for(almost)ever.


-- 
mit freundlichen Grüssen,
Michael Monnerie, Ing. BSc

it-management Internet Services
http://proteger.at [gesprochen: Prot-e-schee]
Tel: 0660 / 415 65 31

****** Aktuelles Radiointerview! ******
http://www.it-podcast.at/aktuelle-sendung.html

// Wir haben im Moment zwei Häuser zu verkaufen:
// http://zmi.at/langegg/
// http://zmi.at/haus2009/

[-- Attachment #1.2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: LWN.net article: creating 1 billion files -> XFS looses

[Emmanuel Florac]

[-- Attachment #1.1: Type: text/plain, Size: 876 bytes --]

Le Thu, 19 Aug 2010 13:12:45 +0200
Michael Monnerie <michael.monnerie@is.it-management.at> écrivait:

> The subject is a bit harsh, but overall the article says:
> XFS is slowest on creating and deleting a billion files
> XFS fsck needs 30GB RAM to fsck that 100TB filesystem.

Too bad I haven't got a 100 TB machine at hand. However I have a 24TB
system dedicated to tests. I'm pretty sure we can do much better with
XFS and the proper mount options :)

In fact, I have an unused 40 TB array too. See this one later...

Stay tuned :)

-- 
------------------------------------------------------------------------
Emmanuel Florac     |   Direction technique
                    |   Intellique
                    |	<eflorac@intellique.com>
                    |   +33 1 78 94 84 02
------------------------------------------------------------------------

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: LWN.net article: creating 1 billion files -> XFS looses

[Stan Hoeppner]

Michael Monnerie put forth on 8/19/2010 7:45 AM:
> Just this week I had a friend in a FS 
> discussion saying "ins't XFS destroying/zeroing files on power 
> failure?". That information is ancient, but things like that stay in 
> peoples brain for(almost)ever.

Had a similar lengthy discussion over on debian-users not more than a month or
so ago.  Same thing there.  Of the 10 or so people active in the thread, I'd
say 8 of them were anti-XFS because of the "corruption  due to power failure"
issue that they'd "read about" years before.  Not a single one of them had
ever used XFS.  A couple of them considered it a "hobbyist quality" filesystem
that might be ready for production use in a few years.  Ahh the ignorance
abounds in our world...

I did my best to educate them, sending them to the Wikipedia page on XFS and
to the xfs.org site and specifically the relevant sections of the FAQ.
Unfortunately there are some people who simply refuse to be educated.  But
those type of people aren't candidates for XFS anyway, thankfully. ;)

-- 
Stan

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: LWN.net article: creating 1 billion files -> XFS looses

[Dave Chinner]

On Thu, Aug 19, 2010 at 02:45:22PM +0200, Michael Monnerie wrote:
> On Donnerstag, 19. August 2010 Christoph Hellwig wrote:
> > The creation and deletion performance is a known issue, and too a
> >  large extent fixes by the new delaylog code.  We're not quite as
> >  fast as ext4 yet, but it's getting close.
> > 
> > The repair result looks a lot like the pre-3.1.0 xfsprogs repair.
>  
> Yes I know. I thought some XFS dev might contact the Author to do some 
> re-testing, as a reputation is quickly destroyed by such articles and 
> takes long to be returned. Just this week I had a friend in a FS 
> discussion saying "ins't XFS destroying/zeroing files on power 
> failure?". That information is ancient, but things like that stay in 
> peoples brain for(almost)ever.

Don't worry too much - I have the details of the test that was run
and alredy know why XFS appeared so slow: it was single threaded.
Immediately, that means XFS will be slower to create 1b files
regardless of any other detail.

Look at it this way - the initial numbers I'm seeing on my test rig
are sustained create rates of about 8,000/s with default mkfs/mount
options (i.e. no tuning, no delayed logging, 32k logbsize, etc) and
it is burning exactly one of 8 CPUs in the VM. I know I can get an
order of magnitude better performance out of XFS on this VM....

It'll take me a few days to run the numbers to be able to write a
solid reply, but I have every confidence that a "create 1b inodes"
benchmark tuned to XFS's strengths rather than one designed to avoid
ext4's weaknesses will show very, very different results.

In the meantime, there is no need to start a flamewar. ;)

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: LWN.net article: creating 1 billion files -> XFS looses

[Emmanuel Florac]

Le Thu, 19 Aug 2010 13:12:45 +0200
Michael Monnerie <michael.monnerie@is.it-management.at> écrivait:

> The subject is a bit harsh, but overall the article says:
> XFS is slowest on creating and deleting a billion files
> XFS fsck needs 30GB RAM to fsck that 100TB filesystem.

Just to go on this subject : a colleague (following my suggestion :)
tried to create 1 billion files in the same XFS directory.
Unfortunately the directories themselves don't scale well that far :
after 1 million files in the first 30 minutes, file creation slows down
gradually, so after 100 hours we had about 230 million files. The
directory size at that point was 5,3 GB.

Now we're starting afresh with 1000 directories with 1 million files
each :)

(Kernel version used : vanilla 2.6.32.11 x86_64 smp)

-- 
------------------------------------------------------------------------
Emmanuel Florac     |   Direction technique
                    |   Intellique
                    |	<eflorac@intellique.com>
                    |   +33 1 78 94 84 02
------------------------------------------------------------------------

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: LWN.net article: creating 1 billion files -> XFS looses

[Dave Chinner]

On Mon, Sep 06, 2010 at 03:42:54PM +0200, Emmanuel Florac wrote:
> Le Thu, 19 Aug 2010 13:12:45 +0200
> Michael Monnerie <michael.monnerie@is.it-management.at> écrivait:
> 
> > The subject is a bit harsh, but overall the article says:
> > XFS is slowest on creating and deleting a billion files
> > XFS fsck needs 30GB RAM to fsck that 100TB filesystem.
> 
> Just to go on this subject : a colleague (following my suggestion :)
> tried to create 1 billion files in the same XFS directory.
> Unfortunately the directories themselves don't scale well that far :
> after 1 million files in the first 30 minutes, file creation slows down
> gradually, so after 100 hours we had about 230 million files. The
> directory size at that point was 5,3 GB.

Oh, that's larger than I've every run before ;)

Try using:

# mkfs.xfs -d size=64k

Will speed up large directory operations by at least an order of
magnitude.

> Now we're starting afresh with 1000 directories with 1 million files
> each :)

Which is exactly the test that was used to generate the numbers that
were published.

> (Kernel version used : vanilla 2.6.32.11 x86_64 smp)

Not much point in testing that kernel - delayed logging is where the
future is for this sort of workload, which is what I'm testing.

FWIW, I'm able to create 50 million inodes in under 14 minutes with
delayed logging and 8 threads using directories of 100k entries.

The run to 1 billion inodes that I started late last night (10 hours
in) has just passed 700M inodes on a 16TB filesystem.  It's running
at about 25,000 creates/s, but it is limited by bad shrinker
behaviour causing the dentry cache to be completely trashed causing
~3000 read iops to reload dentries that are still necessary for
operation. It should be running about 3-4x faster than that.

FYI, The reason I'm taking a while to get numbers is that parallel
create workloads of this scale are showing significant problems (VM
livelocks, shrinker misbehaviour, lock contention in IO completion
processing, buffer cache hash scaling issues, etc) and I'm trying to
fix them as I go - these metadata workloads are completely
unexplored territory....

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: LWN.net article: creating 1 billion files -> XFS looses

[Michael Monnerie]

[-- Attachment #1.1: Type: Text/Plain, Size: 1231 bytes --]

On Dienstag, 7. September 2010 Dave Chinner wrote:
> # mkfs.xfs -d size=64k
> 
> Will speed up large directory operations by at least an order of
> magnitude.
 
I've read the man page for mkfs.xfs, but I couldn't find out if using 
mkfs -d su=64k,sw=2
would be a redundant (and superior) option for that? I'd guess so, 
reading the description of sunit:

sunit=value
This  is  used  to specify the stripe unit for a RAID device or a 
logical volume. The value has to be specified in 512-byte block units. 
Use the su suboption to specify the stripe unit size in bytes. This
suboption ensures that data allocations will be stripe unit aligned when 
the current end of file is being extended and the file size is larger 
than 512KiB. Also inode allocations and the internal log will
be stripe unit aligned.

Or would I still need to use size=64k?

-- 
mit freundlichen Grüssen,
Michael Monnerie, Ing. BSc

it-management Internet Services
http://proteger.at [gesprochen: Prot-e-schee]
Tel: 0660 / 415 65 31

****** Aktuelles Radiointerview! ******
http://www.it-podcast.at/aktuelle-sendung.html

// Wir haben im Moment zwei Häuser zu verkaufen:
// http://zmi.at/langegg/
// http://zmi.at/haus2009/

[-- Attachment #1.2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: LWN.net article: creating 1 billion files -> XFS looses

[Dave Chinner]

On Tue, Sep 07, 2010 at 12:58:40AM +0200, Michael Monnerie wrote:
> On Dienstag, 7. September 2010 Dave Chinner wrote:
> > # mkfs.xfs -d size=64k
> > 
> > Will speed up large directory operations by at least an order of
> > magnitude.
>  
> I've read the man page for mkfs.xfs, but I couldn't find out if using 
> mkfs -d su=64k,sw=2

Sorry, I screwed that up, it should have read:

# mkfs.xfs -n size=64k

(-n = naming = directories. -d = data != directories)

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: LWN.net article: creating 1 billion files -> XFS looses

[Michael Monnerie]

[-- Attachment #1.1: Type: Text/Plain, Size: 1251 bytes --]

On Dienstag, 7. September 2010 Dave Chinner wrote:
> # mkfs.xfs -n size=64k
> (-n = naming = directories. -d = data != directories)

Thank you, Dave. Do I interpret that parameter right:

When a new directory is created, per default it would occupy only 4KB, 
with -n size=64k would be reserved. As the directory fills, space within 
that block will be used, so in the default case after 4KB (how many 
inodes would that be roughly? 256 Bytes/Inode, so 16 entries?) XFS would 
reserve the next block, but in your case 256 entries would fit.

That would keep dir fragmentation lower, and with todays disks, take a 
minimal more space, so it sounds very good to use that option. 
Especially with RAIDs, where stripes usually are 64KB or bigger. Or 
would the waste of space be so big that it could hurt?

Last question: Is there a way to set that option on a given XFS?

-- 
mit freundlichen Grüssen,
Michael Monnerie, Ing. BSc

it-management Internet Services
http://proteger.at [gesprochen: Prot-e-schee]
Tel: 0660 / 415 65 31

****** Aktuelles Radiointerview! ******
http://www.it-podcast.at/aktuelle-sendung.html

// Wir haben im Moment zwei Häuser zu verkaufen:
// http://zmi.at/langegg/
// http://zmi.at/haus2009/

[-- Attachment #1.2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: LWN.net article: creating 1 billion files -> XFS looses

[Emmanuel Florac]

Le Tue, 7 Sep 2010 08:04:10 +1000 vous écriviez:

> Oh, that's larger than I've ever run before ;)

Excellent :) Still works fine afterwards; mount, umount, etc works
flawlessly. Memory consumption though is huge :)
> 
> Try using:
> 
> # mkfs.xfs -d size=64k
> 
> Will speed up large directory operations by at least an order of
> magnitude.

OK, we'll try that too :)
 
> > Now we're starting afresh with 1000 directories with 1 million files
> > each :)
> 
> Which is exactly the test that was used to generate the numbers that
> were published.
> 
> > (Kernel version used : vanilla 2.6.32.11 x86_64 smp)
> 
> Not much point in testing that kernel - delayed logging is where the
> future is for this sort of workload, which is what I'm testing.

I'll compile a 2.6.36rc for comparison.

-- 
------------------------------------------------------------------------
Emmanuel Florac     |   Direction technique
                    |   Intellique
                    |	<eflorac@intellique.com>
                    |   +33 1 78 94 84 02
------------------------------------------------------------------------

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: LWN.net article: creating 1 billion files -> XFS looses

[Dave Chinner]

On Tue, Sep 07, 2010 at 08:20:07AM +0200, Michael Monnerie wrote:
> On Dienstag, 7. September 2010 Dave Chinner wrote:
> > # mkfs.xfs -n size=64k
> > (-n = naming = directories. -d = data != directories)
> 
> Thank you, Dave. Do I interpret that parameter right:
> 
> When a new directory is created, per default it would occupy only 4KB, 
> with -n size=64k would be reserved.

No, it allocates 64k blocks for the directory instead of 4k blocks.

> As the directory fills, space within 
> that block will be used, so in the default case after 4KB (how many 
> inodes would that be roughly? 256 Bytes/Inode, so 16 entries?) XFS would 
> reserve the next block, but in your case 256 entries would fit.

Inodes are not stored in the dirctory structure, only the directory
entry name and the inode number. Hence the amount of space used by a
directory entry is determined by the length of the name.

> That would keep dir fragmentation lower, and with todays disks, take a 
> minimal more space, so it sounds very good to use that option. 
> Especially with RAIDs, where stripes usually are 64KB or bigger. Or 
> would the waste of space be so big that it could hurt?

Well, there is extra overhead to allocate large directory blocks (16
pages instead of one, to begin with, then there's the vmap overhead,
etc), so for small directories smaller block sizes are faster for
create and unlink operations.

For empty directorys, operations on 4k block sized directories
consume roughly 50% less CPU that 64k block size directories. The
4k block size directoeies consume less CPU out to roughly 1.5
million entries where the two are roughly equal. At directory sizes
of 10 million entries, 64k directory block operations are consuming
about 15% of the CPU that 4k directory block operations consume.

In terms of lookups, the 64k block directory will take less IO but
consume more CPU for a given lookup. Hence it depends on your IO
latency and whether directory readahead can hide that latency as to
which will be faster. e.g. For SSDs, CPU usage might be the limiting
factor, not the IO. Right now I don't have any numbers on what
the difference might be - I'm getting 1B inode population issues worked
out first before I start on measuring cold cache lookup times on 1B
files....

> Last question: Is there a way to set that option on a given XFS?

No, it is a mkfs time parameter, though we have been discussing the
possibility of being able to set it per-directory (at mkdir time
when no blocks have been allocated).

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: LWN.net article: creating 1 billion files -> XFS looses

[Michael Monnerie]

[-- Attachment #1.1: Type: Text/Plain, Size: 612 bytes --]

On Dienstag, 7. September 2010 Dave Chinner wrote:
> # mkfs.xfs -n size=64k
 
That explanation was worth a FAQ entry to keep it as a reference.

http://xfs.org/index.php/XFS_FAQ#Q:_Performance:_mkfs.xfs_-n_size.3D64k_option

Thanks, Dave.

-- 
mit freundlichen Grüssen,
Michael Monnerie, Ing. BSc

it-management Internet Services
http://proteger.at [gesprochen: Prot-e-schee]
Tel: 0660 / 415 65 31

****** Aktuelles Radiointerview! ******
http://www.it-podcast.at/aktuelle-sendung.html

// Wir haben im Moment zwei Häuser zu verkaufen:
// http://zmi.at/langegg/
// http://zmi.at/haus2009/

[-- Attachment #1.2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: LWN.net article: creating 1 billion files -> Tests we did

[Emmanuel Florac]

Le Thu, 19 Aug 2010 13:12:45 +0200
Michael Monnerie <michael.monnerie@is.it-management.at> écrivait:

> The subject is a bit harsh, but overall the article says:
> XFS is slowest on creating and deleting a billion files
> XFS fsck needs 30GB RAM to fsck that 100TB filesystem.
> 
> http://lwn.net/SubscriberLink/400629/3fb4bc34d6223b32/

So We've made a test with 1KB files (space, space...) and a production
kernel : 2.6.32.11 (yeah I know, 2.6.38 should be faster but you know,
we upgrade our production kernels prudently :). 

mk1BFiles will create and delete 1000000000 files with 32 threads
Version: v0.2.4-10-gf6decd3, build: Sep  7 2010 13:39:34

Creating 1000000000 files, started at 2010-09-07 13:45:16...
Done, time spent: 89:35:12.262

Doing `ls -R`, started at 2010-09-11 07:20:28...
Stat: ls (pid: 18844) status: ok, returned value: 0
Cpu usage: user: 1:27:47.242, system: 20:18:21.689
Max rss: 229.01 MBytes, page fault: major: 4, minor: 58694

Compute size used by 1000000000 files, started at 2010-09-12 09:30:52...
Size used by files: 11.1759 TBytes
Size used by directory: 32.897 GBytes
Size used (total): 11.2080 TBytes
Done, time spent: 25:50:32.355

Deleting 1000000000 files, started at 2010-09-13 11:21:24...
Done, time spent: 68:37:38.117

Test run on a dual Opteron quad core, 16 GB RAM, kernel 2.6.32.11
x86_64...

-- 
------------------------------------------------------------------------
Emmanuel Florac     |   Direction technique
                    |   Intellique
                    |	<eflorac@intellique.com>
                    |   +33 1 78 94 84 02
------------------------------------------------------------------------

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: LWN.net article: creating 1 billion files -> Tests we did

[Stan Hoeppner]

Emmanuel Florac put forth on 9/16/2010 5:13 AM:

> Test run on a dual Opteron quad core, 16 GB RAM, kernel 2.6.32.11
> x86_64...

This is a test of storage system performance, and you left out the
storage array specs?  By doing so it seems you're stating the underlying
storage is not relevant to the results.

So, are you saying I should be able to duplicate your results with that
dual Opty system, but using an md RAID0 stripe over 8x2TB SATA disks
connected to two $60 4 port SiI 3124 PCIe x1 cards?

-- 
Stan



_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: LWN.net article: creating 1 billion files -> Tests we did

[Michael Monnerie]

[-- Attachment #1.1: Type: Text/Plain, Size: 1328 bytes --]

On Donnerstag, 16. September 2010 Stan Hoeppner wrote:
> So, are you saying I should be able to duplicate your results with
>  that dual Opty system, but using an md RAID0 stripe over 8x2TB SATA
>  disks connected to two $60 4 port SiI 3124 PCIe x1 cards?
 
According to Dave, with his patches you should even outperform that if 
you got faster CPUs :-)

Emmanuel, where is the "mk1BFiles" Benchmark? We're planning for a new 
hardware this year, so this would be a good time to run it.  Could I 
have the script?

The output misses the time "ls" took, but I can one can calculate that 
from the next test start, so 9:30-7:20 = 2:10, is that correct? Two 
hours to just see all files, ugh. I guess it will take some years until 
we want to have such a filesystem. Either hardware must become quicker, 
or another wonderful new patch is needed.

-- 
// Michael Monnerie, Ing.BSc.
----------------------------------
Sorcerers have their magic wands:
  powerful, potentially dangerous tools with a life of their own.
Witches have their familiars:
  creatures disguised as household beasts that could,
  if they choose, wreak the witches' havoc.
Mystics have their golems:
  beings built of wood and tin brought to life to do their
  masters' bidding.
I have Linux.
----------------------------------

[-- Attachment #1.2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: LWN.net article: creating 1 billion files -> Tests we did

[Peter Grandi]

[ ... useless run of something misrepresented as a test ... ]

>> Test run on a dual Opteron quad core, 16 GB RAM, kernel
>> 2.6.32.11 x86_64...

> This is a test of storage system performance, and you left out
> the storage array specs?  By doing so it seems you're stating
> the underlying storage is not relevant to the results. [ ... ]

This is only one of the several aspects of the waste of time
that was misrepresented as a storage test, even if it is one of
the funniest.

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: LWN.net article: creating 1 billion files -> Tests we did

[Peter Grandi]

>> The subject is a bit harsh, but overall the article says: XFS
>> is slowest on creating and deleting a billion files XFS fsck
>> needs 30GB RAM to fsck that 100TB filesystem.

Hahahaha. Very funny. So what?

>> http://lwn.net/SubscriberLink/400629/3fb4bc34d6223b32/

LWN is usually fairly decent, but I have noticed it does
occasionally waste pixels on/bits things that the author(s)
misrepresent as storage or file system tests.

However in this case the main takeaway of the presentation
reported is that it is just a bad idea to assume that file systems
can scale to large collections of small files as DBMSes designed
for that purpose. So what?

> So We've made a test with 1KB files (space, space...) and a
> production kernel : 2.6.32.11 (yeah I know, 2.6.38 should be
> faster but you know, we upgrade our production kernels prudently :).

Why is this a test of anything other than how to waste time?

> mk1BFiles will create and delete 1000000000 files with 32
> threads Version: v0.2.4-10-gf6decd3, build: Sep 7 2010
> 13:39:34

> Creating 1000000000 files, started at 2010-09-07 13:45:16...
> Done, time spent: 89:35:12.262

Was there any intervening cache flush?

> Doing `ls -R`, started at 2010-09-11 07:20:28...
> Stat: ls (pid: 18844) status: ok, returned value: 0
> Cpu usage: user: 1:27:47.242, system: 20:18:21.689
> Max rss: 229.01 MBytes, page fault: major: 4, minor: 58694

Was there any intervening cache flush?

> Compute size used by 1000000000 files, started at 2010-09-12 09:30:52...
> Size used by files: 11.1759 TBytes
> Size used by directory: 32.897 GBytes
> Size used (total): 11.2080 TBytes
> Done, time spent: 25:50:32.355

Was there any intervening cache flush?

> Deleting 1000000000 files, started at 2010-09-13 11:21:24...
> Done, time spent: 68:37:38.117

Was there any intervening cache flush?

Why would anybody with even a little knowledge of computers and
systems want to use a filesystem as database for small records?

> Test run on a dual Opteron quad core, 16 GB RAM, kernel 2.6.32.11
> x86_64...

So what?

Some of the most amusing quotes from the LWN article are from the
comments.

 "Recently I did similiar tests for determining how well PostgreSQL
  would be able to deal with databases with potentially hundreds of
  thousands of tables. From what I found out, it's only limited by
  the file system's ability to work with that many files in a
  single directory."

HHAHAHAHAHAHAHA.

 "> But in what situations will it make more sense to not group a
  > billion of file items into logical groups?
 
  Things like squid cache directories, git object directories,
  ccache cache directories, that hidden thumbnails directory in
  your $HOME... They all have in common that the files are named
  by a hash or something similar. There is no logical grouping at
  all here; it is a completely flat namespace."

AAAAAAGGGGGHHHHHHHHHHH.

But the original presentation has absolutely the funniest bit:

 "Why Not Use a Database?
  ● Users and system administrators are familiar
    with file systems
      Backup, creation, etc are all well understood
  ● File systems handle partial failures pretty well
      Being able to recover part of the stored data is
      useful for some applications
  ● File systems are “cheap” since they come with
    your operating system!"

My evil translation of that is "because so many sysadms and
programmers are incompetent and stupid and wish for ponies".

Of course the best bit is where someone :-) was quoted making
sense:

  “Millions of files may work; but 1 billion is an utter
  absurdity. A filesystem that can store reasonably 1 billion
  small files in 7TB is an unsolved research issue...,”

The stupidest bit of the presentation was part of the quoted
reply:

  “Strangely enough, I have been testing ext4 and stopped filling
  it at a bit over 1 billion 20KB files on Monday (with 60TB of
  storage). Running fsck on it took only 2.4 hours.”

Where the idea that the 'fsck' time that matters is that of a
freshly created (and was the page cache flushed?), uncorrupted
filesystem is intensely comical. "Possible" does not mean
"reasonably". Just delirious.

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: LWN.net article: creating 1 billion files -> Tests we did

[Emmanuel Florac]

Le Thu, 16 Sep 2010 16:53:07 -0500 vous écriviez:

> This is a test of storage system performance, and you left out the
> storage array specs?  By doing so it seems you're stating the
> underlying storage is not relevant to the results.

Sorry :) The storage is a 24 2TB disks RAID-6 array on a 3W&re 9650. Not
exactly stellar at IOPS performance.

> So, are you saying I should be able to duplicate your results with
> that dual Opty system, but using an md RAID0 stripe over 8x2TB SATA
> disks connected to two $60 4 port SiI 3124 PCIe x1 cards?
> 

Your setup may be slightly slower.

-- 
------------------------------------------------------------------------
Emmanuel Florac     |   Direction technique
                    |   Intellique
                    |	<eflorac@intellique.com>
                    |   +33 1 78 94 84 02
------------------------------------------------------------------------

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: LWN.net article: creating 1 billion files -> Tests we did

[Emmanuel Florac]

Le Fri, 17 Sep 2010 20:29:01 +0100 vous écriviez:

> [ ... useless run of something misrepresented as a test ... ]

I won't comment on the usefulness of your rant, though, because I'd
rather stay amiable.

If you don't mind, I'll however post complementary results of the same
test running on different filesystems on the very same hardware,
because apparently you missed the fact that this was about tracking
comparative xfs progress in some metadata intensive workloads. 

-- 
------------------------------------------------------------------------
Emmanuel Florac     |   Direction technique
                    |   Intellique
                    |	<eflorac@intellique.com>
                    |   +33 1 78 94 84 02
------------------------------------------------------------------------

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: LWN.net article: creating 1 billion files -> Tests we did

[Emmanuel Florac]

Le Fri, 17 Sep 2010 20:57:48 +0100 vous écriviez:

> LWN is usually fairly decent, but I have noticed it does
> occasionally waste pixels on/bits things that the author(s)
> misrepresent as storage or file system tests.

How unfortunate we missed your precious stance on that matter. Everybody
knows that benchmarks are mostly useless /per se/, however it often
occurs that comparative benchmarks may easily reveal some interesting
differences.

As for the interest of an experiment of pushing something to the limit
for the sake of it, it may equally reveal interesting bugs. The fact
that all filesystems in this tests didn't simply *fail* under the load
is by itself revealing of the overall robustness and stability of
these, the VFS and Linux kernel. 

As a side note, were you using a slightly less harsh tone, you'd
probably be help people being less reluctant to discuss those points
more deeply.

-- 
------------------------------------------------------------------------
Emmanuel Florac     |   Direction technique
                    |   Intellique
                    |	<eflorac@intellique.com>
                    |   +33 1 78 94 84 02
------------------------------------------------------------------------

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs