Re: Updated performance results

[Steven Pratt <slpratt@austin.ibm.com>]

Chris Mason wrote:
> On Fri, Sep 11, 2009 at 04:35:50PM -0500, Steven Pratt wrote:
>   
>> Chris Mason wrote:
>>     
>>> On Mon, Aug 31, 2009 at 12:49:13PM -0500, Steven Pratt wrote:
>>>       
>>>> Better late than never. Finally got this finished up.  Mixed bag on
>>>> this one.  BTRFS lags significantly on single threaded.  Seems
>>>> unable to keep IO outstanding to the device.  Less that 60% busy on
>>>> the DM device, compared to 97%+ for all other filesystems.
>>>> nodatacow helps out, increasing utilization to about 70%, but still
>>>> trails by a large margin.
>>>>         
>>> Hi Steve,
>>>
>>> Jens Axboe did some profiling on his big test rig and I think we found
>>> the biggest CPU problems.  The end result is now setting in the master
>>> branch of the btrfs-unstable repo.
>>>
>>> On his boxes, btrfs went from around 400MB/s streaming writes to 1GB/s
>>> limit, and we're now tied with XFS while using less CPU time.
>>>
>>> Hopefully you will see similar results ;)
>>>       
>> Hmmm, well no I didn't.  Throughputs at 1 and 128 threads are pretty
>> much unchanged, although I do see a good CPU savings on the 128
>> thread case (with cow).  For 16 threads we actually regressed with
>> cow enabled.
>>
>> Results  are here:
>>
>> http://btrfs.boxacle.net/repository/raid/large_create_test/write-test/1M_odirect_create.html
>>
>> I'll try to look more into this next week.
>>
>>     
>
> Hmmm, Jens was benchmarking buffered writes, but he was also testing on
> his new per-bdi write back code.  If your next run could be buffered
> instead of O_DIRECT, I'd be curious to see the results.
>
>   
Buffered does look a lot better. I don't have a btrfs baseline before 
these latest changes for this exact workload, but these results are not 
bad at all.  With cow, beats just about everything except XFS, and with 
nocow simply screams.  CPU consumption looks good as well.  I'll 
probably give the full set of tests a run tonight.

Results are here:
http://btrfs.boxacle.net/repository/raid/buffered-creates/buffered-create/buffered-create.html

Only bit of bad news is I did get one error that crashed the system on 
single threaded nocow run. So that data point is missing.
Output below:

btrfs1 kernel: [251789.525886] ------------[ cut here ]------------
Message from syslogd@ at Mon Sep 14 13:13:04 2009 ...
btrfs1 kernel: [251789.526574] invalid opcode: 0000 [#1] SMP
Message from syslogd@ at Mon Sep 14 13:13:04 2009 ...
btrfs1 kernel: [251789.526654] last sysfs file: 
/sys/devices/pci0000:0c/0000:0c:01.0/local_cpus
Message from syslogd@ at Mon Sep 14 13:13:05 2009 ...
btrfs1 kernel: [251789.526654] Stack:
Message from syslogd@ at Mon Sep 14 13:13:05 2009 ...
btrfs1 kernel: [251789.526654]  ffff88013fc234c0 ffff88013fbcf400 
0000000000000000 ffff88013fc01080
Message from syslogd@ at Mon Sep 14 13:13:05 2009 ...
btrfs1 kernel: [251789.526654]  ffff880132e11d38 ffffffff802a5392 
0000000000000001 ffff88013fbcf400
Message from syslogd@ at Mon Sep 14 13:13:05 2009 ...
btrfs1 kernel: [251789.526654] Call Trace:
Message from syslogd@ at Mon Sep 14 13:13:05 2009 ...
btrfs1 kernel: [251789.526654]  [<ffffffff802a5392>] 
cache_flusharray+0x7d/0xae
Message from syslogd@ at Mon Sep 14 13:13:05 2009 ...
btrfs1 kernel: [251789.526654]  [<ffffffff802a5629>] kfree+0x192/0x1b1
Message from syslogd@ at Mon Sep 14 13:13:05 2009 ...
btrfs1 kernel: [251789.526654]  [<ffffffffa0378c9f>] 
put_worker+0x14/0x16 [btrfs]
Message from syslogd@ at Mon Sep 14 13:13:05 2009 ...
btrfs1 kernel: [251789.526654]  [<ffffffffa0378d55>] 
btrfs_stop_workers+0xb4/0xc9 [btrfs]
Message from syslogd@ at Mon Sep 14 13:13:05 2009 ...
btrfs1 kernel: [251789.526654]  [<ffffffffa0355cbe>] 
close_ctree+0x210/0x288 [btrfs]
Message from syslogd@ at Mon Sep 14 13:13:05 2009 ...
btrfs1 kernel: [251789.526654]  [<ffffffff802bd1a1>] ? 
invalidate_inodes+0x100/0x112
Message from syslogd@ at Mon Sep 14 13:13:05 2009 ...
btrfs1 kernel: [251789.526654]  [<ffffffffa033f4cb>] 
btrfs_put_super+0x18/0x27 [btrfs]
Message from syslogd@ at Mon Sep 14 13:13:05 2009 ...
btrfs1 kernel: [251789.526654]  [<ffffffff802ad12b>] 
generic_shutdown_super+0x73/0xe2
Message from syslogd@ at Mon Sep 14 13:13:05 2009 ...
btrfs1 kernel: [251789.526654]  [<ffffffff802ad1e5>] 
kill_anon_super+0x11/0x3b
Message from syslogd@ at Mon Sep 14 13:13:05 2009 ...
btrfs1 kernel: [251789.526654]  [<ffffffff802ad51d>] 
deactivate_super+0x62/0x77
Message from syslogd@ at Mon Sep 14 13:13:05 2009 ...
btrfs1 kernel: [251789.526654]  [<ffffffff802bf9eb>] 
mntput_no_expire+0xec/0x12c
Message from syslogd@ at Mon Sep 14 13:13:05 2009 ...
btrfs1 kernel: [251789.526654]  [<ffffffff802bff3a>] sys_umount+0x2c5/0x31c
Message from syslogd@ at Mon Sep 14 13:13:05 2009 ...
btrfs1 kernel: [251789.526654]  [<ffffffff8020ba2b>] 
system_call_fastpath+0x16/0x1b
Message from syslogd@ at Mon Sep 14 13:13:05 2009 ...
btrfs1 kernel: [251789.526654] Code: 89 f7 e8 48 07 f8 ff 48 c1 e8 0c 48 
ba 00 00 00 00 00 e2 ff ff 48 6b c0 38 48 01 d0 66 83 38 00 79 04 48 8b 
40 10 80 38 00 78 04 <0f> 0b eb fe 48 8b 58 30 48 63 45 c8 48 89 df 4d 
8b a4 c5 60 08

Steve


> Thanks,
> Chris
>