Performance issues (XFS-bcache-P400-S3700)

Performance issues (XFS-bcache-P400-S3700)

[Erik Hjelmås]

Hi, I have a HP 380 G5 server (with P400 raid controller without battery
backup so not using write cache) where c0d0 is Intel S3500 SSD, c0d1 is Intel
S3700 SSD and c0d2 is a slow RAID5 array:

root@storage01:~# fdisk -l /dev/cciss/c0d{0,1,2} | grep 'Disk /dev'
Disk /dev/cciss/c0d0: 80.0 GB, 79990456320 bytes
Disk /dev/cciss/c0d1: 100.0 GB, 99994337280 bytes
Disk /dev/cciss/c0d2: 2500.4 GB, 2500371439616 bytes

c0d1 is the caching device and c0d2 is the backing device, and everything
seems to look ok. Ive enabled drive write caches (since SSD's performance
depend on it), running ubuntu 14.04 upgraded today (in case my problem was
the uninterruptable sleep status of bcache_writebac, but it wasn't), and
Ive mounted XFS with nobarrier since I also read something about that:

root@storage01:~# mount | grep bcache
/dev/bcache0 on /brick0 type xfs (rw,nobarrier)

status of bcache seems ok:

root@storage01:~# dmesg | grep bcac
[    7.453145] bcache: bch_journal_replay() journal replay done, 1017
keys in 462 entries, seq 76745658
[    7.523539] bcache: register_cache() registered cache device cciss/c0d1
[    7.561245] bcache: register_bdev() registered backing device cciss/c0d2
[    7.684531] bcache: bch_cached_dev_attach() Caching cciss/c0d2 as
bcache0 on set 6c7ad3b1-388e-47df-904a-94da0a092669
[    8.781182] XFS (bcache0): Mounting Filesystem
[    8.980111] XFS (bcache0): Ending clean mount
root@storage01:~# cat /sys/block/bcache0/bcache/writeback_running
1
root@storage01:~# cat /sys/block/bcache0/bcache/cache_mode
writethrough [writeback] writearound none
root@storage01:~# cat /sys/block/bcache0/bcache/dirty_data
9.3G
root@storage01:~# cat /sys/block/bcache0/bcache/sequential_cutoff
4.0M
root@storage01:~# cat /sys/block/bcache0/bcache/state
dirty
root@storage01:~# cat /sys/block/bcache0/bcache/writeback_rate_debug
rate:           512/sec
dirty:          9.2G
target:         9.3G
proportional:   -20.5k
derivative:     0
change:         -20.5k/sec
next io:        482ms

Unfortunately performance is not very good (random write IOPS is about 200
when I would expect it to be about 5000-10000):

(the fio.bash script I run is available at
http://www.ansatt.hig.no/erikh/sysadm/fio.bash)

root@storage01:~# ./fio.bash /brick0/tmp/
Baseline reads with hdparm
 /dev/bcache0:
  Timing cached reads:   5870 MB in  2.00 seconds = 2936.42 MB/sec
  Timing buffered disk reads: 252 MB in  3.01 seconds =  83.83 MB/sec
Sequential read
  read : io=359784KB, bw=35622KB/s, iops=8905, runt= 10100msec
Sequential write
  write: io=18508KB, bw=1745.8KB/s, iops=436, runt= 10602msec
Random read
  read : io=43716KB, bw=4244.3KB/s, iops=1061, runt= 10300msec
Random write
  write: io=10288KB, bw=883208B/s, iops=215, runt= 11928msec
Mixed 70/30 random read and write with 8K block size
  read : io=23272KB, bw=2128.2KB/s, iops=266, runt= 10936msec
  write: io=10376KB, bw=971564B/s, iops=118, runt= 10936msec

If I run the same test on the root filesystem on c0d0 (which is EXT4-LVM on
Intel S3500 SSD, and the S3500 is supposed to have much worse random write
performance compared to the much more expensive S3700 which my bcache uses)
random write performance (and in general) is 50x better:

root@storage01:~# ./fio.bash /root/
Baseline reads with hdparm
 /dev/mapper/storage01--vg-root:
  Timing cached reads:   6370 MB in  2.00 seconds = 3186.45 MB/sec
  Timing buffered disk reads: 392 MB in  3.00 seconds = 130.66 MB/sec
Sequential read
  read : io=490832KB, bw=48985KB/s, iops=12246, runt= 10020msec
Sequential write
  write: io=492388KB, bw=49180KB/s, iops=12294, runt= 10012msec
Random read
  read : io=262656KB, bw=26203KB/s, iops=6550, runt= 10024msec
Random write
  write: io=498980KB, bw=49848KB/s, iops=12462, runt= 10010msec
Mixed 70/30 random read and write with 8K block size
  read : io=288480KB, bw=28773KB/s, iops=3596, runt= 10026msec
  write: io=121216KB, bw=12090KB/s, iops=1511, runt= 10026msec

When I have IOPS-numbers around 200 that sounds very much like spinning
drives, but my setup is as far as I can tell correct with a Intel S3700
high performance SSD as caching device on c0d1, so I can't make sense of
this.

any help greatly appreciated :)

/Erik

Re: Performance issues (XFS-bcache-P400-S3700)

[Erik Hjelmås]

thanks Matthias, I just tried to upgrade to Ubunty 14.10 since it has
the 3.16 kernel and mentions bcache support in the release notes, but
didn't change anything,

I'll look into the change logs of of 3.17 and 3.18 to see if its worth
trying an even newer kernel

/Erik

2014-11-08 11:23 GMT+01:00 Matthias Ferdinand <bcache@mfedv.net>:
> Hi,
>
> sounds similar to the performance problems I had
> (http://comments.gmane.org/gmane.linux.kernel.bcache.devel/2542) with an
> Intel 530 SSD. The SSD would perform great with linear io, but with
> random io it was great only for a short time and then slowing down. The
> entire bcache+backing combo became slower than just the rotating disks
> alone.
>
> In my case I found it to be the SSD, as performance was much better when
> I switched the caching device to rotating disk (!) on an old hardware
> raid-1 controller (w/128MB battery backed cache). In your case it should
> be different as the S3700 seems to be an enterprise model.
>
> Also I do not use the 14.04 stock kernel, it was missing a lot of bcache
> patches, currently my system runs 3.18.0-rc2.
>
> I don't think the nobarrier option has something to do with it, as the
> cache device will only see the random data + meta data updates of the
> underlying block device and that should still be an order of magnitude
> faster than on the rotating disk.
>
> Regards
> Matthias
>
>
> On Sat, Nov 08, 2014 at 07:18:35AM +0100, Erik Hjelmås wrote:
>> Hi, I have a HP 380 G5 server (with P400 raid controller without battery
>> backup so not using write cache) where c0d0 is Intel S3500 SSD, c0d1 is Intel
>> S3700 SSD and c0d2 is a slow RAID5 array:
>>
>> root@storage01:~# fdisk -l /dev/cciss/c0d{0,1,2} | grep 'Disk /dev'
>> Disk /dev/cciss/c0d0: 80.0 GB, 79990456320 bytes
>> Disk /dev/cciss/c0d1: 100.0 GB, 99994337280 bytes
>> Disk /dev/cciss/c0d2: 2500.4 GB, 2500371439616 bytes
>>
>> c0d1 is the caching device and c0d2 is the backing device, and everything
>> seems to look ok. Ive enabled drive write caches (since SSD's performance
>> depend on it), running ubuntu 14.04 upgraded today (in case my problem was
>> the uninterruptable sleep status of bcache_writebac, but it wasn't), and
>> Ive mounted XFS with nobarrier since I also read something about that:
>>
>> root@storage01:~# mount | grep bcache
>> /dev/bcache0 on /brick0 type xfs (rw,nobarrier)
>>
>> status of bcache seems ok:
>>
>> root@storage01:~# dmesg | grep bcac
>> [    7.453145] bcache: bch_journal_replay() journal replay done, 1017
>> keys in 462 entries, seq 76745658
>> [    7.523539] bcache: register_cache() registered cache device cciss/c0d1
>> [    7.561245] bcache: register_bdev() registered backing device cciss/c0d2
>> [    7.684531] bcache: bch_cached_dev_attach() Caching cciss/c0d2 as
>> bcache0 on set 6c7ad3b1-388e-47df-904a-94da0a092669
>> [    8.781182] XFS (bcache0): Mounting Filesystem
>> [    8.980111] XFS (bcache0): Ending clean mount
>> root@storage01:~# cat /sys/block/bcache0/bcache/writeback_running
>> 1
>> root@storage01:~# cat /sys/block/bcache0/bcache/cache_mode
>> writethrough [writeback] writearound none
>> root@storage01:~# cat /sys/block/bcache0/bcache/dirty_data
>> 9.3G
>> root@storage01:~# cat /sys/block/bcache0/bcache/sequential_cutoff
>> 4.0M
>> root@storage01:~# cat /sys/block/bcache0/bcache/state
>> dirty
>> root@storage01:~# cat /sys/block/bcache0/bcache/writeback_rate_debug
>> rate:           512/sec
>> dirty:          9.2G
>> target:         9.3G
>> proportional:   -20.5k
>> derivative:     0
>> change:         -20.5k/sec
>> next io:        482ms
>>
>> Unfortunately performance is not very good (random write IOPS is about 200
>> when I would expect it to be about 5000-10000):
>>
>> (the fio.bash script I run is available at
>> http://www.ansatt.hig.no/erikh/sysadm/fio.bash)
>>
>> root@storage01:~# ./fio.bash /brick0/tmp/
>> Baseline reads with hdparm
>>  /dev/bcache0:
>>   Timing cached reads:   5870 MB in  2.00 seconds = 2936.42 MB/sec
>>   Timing buffered disk reads: 252 MB in  3.01 seconds =  83.83 MB/sec
>> Sequential read
>>   read : io=359784KB, bw=35622KB/s, iops=8905, runt= 10100msec
>> Sequential write
>>   write: io=18508KB, bw=1745.8KB/s, iops=436, runt= 10602msec
>> Random read
>>   read : io=43716KB, bw=4244.3KB/s, iops=1061, runt= 10300msec
>> Random write
>>   write: io=10288KB, bw=883208B/s, iops=215, runt= 11928msec
>> Mixed 70/30 random read and write with 8K block size
>>   read : io=23272KB, bw=2128.2KB/s, iops=266, runt= 10936msec
>>   write: io=10376KB, bw=971564B/s, iops=118, runt= 10936msec
>>
>> If I run the same test on the root filesystem on c0d0 (which is EXT4-LVM on
>> Intel S3500 SSD, and the S3500 is supposed to have much worse random write
>> performance compared to the much more expensive S3700 which my bcache uses)
>> random write performance (and in general) is 50x better:
>>
>> root@storage01:~# ./fio.bash /root/
>> Baseline reads with hdparm
>>  /dev/mapper/storage01--vg-root:
>>   Timing cached reads:   6370 MB in  2.00 seconds = 3186.45 MB/sec
>>   Timing buffered disk reads: 392 MB in  3.00 seconds = 130.66 MB/sec
>> Sequential read
>>   read : io=490832KB, bw=48985KB/s, iops=12246, runt= 10020msec
>> Sequential write
>>   write: io=492388KB, bw=49180KB/s, iops=12294, runt= 10012msec
>> Random read
>>   read : io=262656KB, bw=26203KB/s, iops=6550, runt= 10024msec
>> Random write
>>   write: io=498980KB, bw=49848KB/s, iops=12462, runt= 10010msec
>> Mixed 70/30 random read and write with 8K block size
>>   read : io=288480KB, bw=28773KB/s, iops=3596, runt= 10026msec
>>   write: io=121216KB, bw=12090KB/s, iops=1511, runt= 10026msec
>>
>> When I have IOPS-numbers around 200 that sounds very much like spinning
>> drives, but my setup is as far as I can tell correct with a Intel S3700
>> high performance SSD as caching device on c0d1, so I can't make sense of
>> this.
>>
>> any help greatly appreciated :)
>>
>> /Erik
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-bcache" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html