* Preliminary performance benchmarks.
@ 2012-03-03 1:35 Joseph Glanville
0 siblings, 0 replies; 5+ messages in thread
From: Joseph Glanville @ 2012-03-03 1:35 UTC (permalink / raw)
To: Kent Overstreet, linux-bcache-u79uwXL29TY76Z2rM5mHXA
Carrying out some scalability tests on high I/O systems. More tests to
come using the Phoronix Test Suite.
Fio summary:
24 jobs
Direct IO
Randwrite test
Total of about 80k IOPs at 3.5k IOPS per thread.
Test rig specs:
2x Intel(R) Xeon(R) CPU X5650 @ 2.67GHz (12 physical cores, 24 logical)
12x 2TB Seagate nearline-SAS in RAID6 on LSI Logic / Symbios Logic LSI
MegaSAS 9260
2x Intel 520 SSDs 120GB in RAID0
The 2 520s are striped with md raid as /dev/md0 whch is formatted as a
bcache cache device using 1M buckets and 8k hard block size
Backing device is the big old raid6.
Random IO performance of the native raid0 is about 96K IOPs, backing
device in the realm of 1600ish.
However the backing device has a sequential IO performance of about 1.5GB/s.
Below are some quick findings using fio - showing very good
scalability of bcache even with very very fast SSDs.
*Note: The SSDs are connected via a SATA2 interface being somewhat of
a bottleneck.
dev05 ~ # fio --filename=/dev/bcache0 --direct=1 --rw=randwrite
--bs=4096 --size=10001078620 --runtime=60 --name=file1 --iodepth=32
--numjobs=24 --ioengine=libaio
file1: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=32
...
file1: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=32
fio 2.0.3
Starting 24 processes
Jobs: 10 (f=10): [______ww_wwww__E__www_w_] [7.8% done] [0K/322.4M /s]
[0 /78.7K iops] [eta 12m:15s]
file1: (groupid=0, jobs=1): err= 0: pid=3993
write: io=821604KB, bw=13645KB/s, iops=3411 , runt= 60214msec
slat (usec): min=4 , max=113487 , avg=114.47, stdev=2376.82
clat (usec): min=460 , max=1617.5K, avg=9223.06, stdev=19736.65
lat (usec): min=476 , max=1617.5K, avg=9337.33, stdev=20514.45
clat percentiles (usec):
| 1.00th=[ 1288], 5.00th=[ 1896], 10.00th=[ 2416], 20.00th=[ 3120],
| 30.00th=[ 3728], 40.00th=[ 4384], 50.00th=[ 5088], 60.00th=[ 5728],
| 70.00th=[ 6432], 80.00th=[ 7264], 90.00th=[ 9152], 95.00th=[37120],
| 99.00th=[94720], 99.50th=[138240], 99.90th=[216064]
bw (KB/s) : min= 5769, max=50211, per=4.33%, avg=13718.03, stdev=5147.65
lat (usec) : 500=0.01%, 750=0.01%, 1000=0.16%
lat (msec) : 2=5.69%, 4=28.76%, 10=56.24%, 20=1.62%, 50=3.89%
lat (msec) : 100=2.72%, 250=0.87%, 500=0.02%, 750=0.01%, 1000=0.01%
lat (msec) : 2000=0.01%
cpu : usr=0.94%, sys=3.05%, ctx=39039, majf=0, minf=23
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=205401/d=0, short=r=0/w=0/d=0
file1: (groupid=0, jobs=1): err= 0: pid=3994
write: io=773564KB, bw=12864KB/s, iops=3216 , runt= 60134msec
slat (usec): min=4 , max=121438 , avg=123.82, stdev=2487.05
clat (usec): min=473 , max=1676.1K, avg=9799.12, stdev=22553.15
lat (usec): min=489 , max=1760.2K, avg=9922.61, stdev=23345.78
clat percentiles (usec):
| 1.00th=[ 1448], 5.00th=[ 2160], 10.00th=[ 2672], 20.00th=[ 3280],
| 30.00th=[ 3856], 40.00th=[ 4576], 50.00th=[ 5280], 60.00th=[ 5856],
| 70.00th=[ 6560], 80.00th=[ 7392], 90.00th=[ 9408], 95.00th=[38656],
| 99.00th=[99840], 99.50th=[156672], 99.90th=[228352]
bw (KB/s) : min= 5158, max=37016, per=4.07%, avg=12898.20, stdev=4203.06
lat (usec) : 500=0.01%, 750=0.01%, 1000=0.12%
lat (msec) : 2=3.54%, 4=28.30%, 10=58.52%, 20=1.64%, 50=4.03%
lat (msec) : 100=2.84%, 250=0.95%, 500=0.03%, 750=0.01%, 1000=0.01%
lat (msec) : 2000=0.01%
cpu : usr=0.89%, sys=2.95%, ctx=38309, majf=0, minf=24
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=193391/d=0, short=r=0/w=0/d=0
file1: (groupid=0, jobs=1): err= 0: pid=3995
write: io=783660KB, bw=13060KB/s, iops=3265 , runt= 60004msec
slat (usec): min=4 , max=118055 , avg=120.56, stdev=2423.94
clat (usec): min=530 , max=1082.8K, avg=9675.59, stdev=20228.23
lat (usec): min=551 , max=1142.8K, avg=9796.35, stdev=21045.95
clat percentiles (usec):
| 1.00th=[ 1480], 5.00th=[ 2256], 10.00th=[ 2736], 20.00th=[ 3408],
| 30.00th=[ 3952], 40.00th=[ 4640], 50.00th=[ 5280], 60.00th=[ 5856],
| 70.00th=[ 6560], 80.00th=[ 7392], 90.00th=[ 9536], 95.00th=[38144],
| 99.00th=[94720], 99.50th=[150528], 99.90th=[222208]
bw (KB/s) : min= 4756, max=21700, per=4.11%, avg=13022.62, stdev=3325.00
lat (usec) : 750=0.01%, 1000=0.10%
lat (msec) : 2=3.14%, 4=27.40%, 10=59.65%, 20=1.81%, 50=4.03%
lat (msec) : 100=2.92%, 250=0.90%, 500=0.03%, 750=0.01%, 2000=0.01%
cpu : usr=0.91%, sys=3.06%, ctx=38590, majf=0, minf=25
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=195915/d=0, short=r=0/w=0/d=0
file1: (groupid=0, jobs=1): err= 0: pid=3996
write: io=799344KB, bw=13233KB/s, iops=3308 , runt= 60405msec
slat (usec): min=4 , max=113409 , avg=115.84, stdev=2365.77
clat (usec): min=648 , max=1313.9K, avg=9487.45, stdev=21360.00
lat (usec): min=661 , max=1313.9K, avg=9603.37, stdev=22116.87
clat percentiles (usec):
| 1.00th=[ 1464], 5.00th=[ 2096], 10.00th=[ 2544], 20.00th=[ 3184],
| 30.00th=[ 3792], 40.00th=[ 4448], 50.00th=[ 5152], 60.00th=[ 5792],
| 70.00th=[ 6432], 80.00th=[ 7328], 90.00th=[ 9280], 95.00th=[37120],
| 99.00th=[96768], 99.50th=[152576], 99.90th=[222208]
bw (KB/s) : min= 5483, max=34091, per=4.20%, avg=13305.74, stdev=3850.52
lat (usec) : 750=0.01%, 1000=0.07%
lat (msec) : 2=4.13%, 4=29.36%, 10=57.07%, 20=1.78%, 50=4.05%
lat (msec) : 100=2.56%, 250=0.93%, 500=0.03%, 750=0.01%, 1000=0.01%
lat (msec) : 2000=0.01%
cpu : usr=0.94%, sys=2.95%, ctx=38593, majf=0, minf=24
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=199836/d=0, short=r=0/w=0/d=0
file1: (groupid=0, jobs=1): err= 0: pid=3997
write: io=821316KB, bw=13643KB/s, iops=3410 , runt= 60200msec
slat (usec): min=4 , max=117990 , avg=110.12, stdev=2280.98
clat (usec): min=568 , max=1796.8K, avg=9237.17, stdev=20452.93
lat (usec): min=585 , max=1796.8K, avg=9347.04, stdev=21191.36
clat percentiles (usec):
| 1.00th=[ 1416], 5.00th=[ 2128], 10.00th=[ 2544], 20.00th=[ 3152],
| 30.00th=[ 3728], 40.00th=[ 4384], 50.00th=[ 5152], 60.00th=[ 5728],
| 70.00th=[ 6432], 80.00th=[ 7264], 90.00th=[ 9024], 95.00th=[36608],
| 99.00th=[94720], 99.50th=[148480], 99.90th=[214016]
bw (KB/s) : min= 5928, max=43330, per=4.31%, avg=13668.87, stdev=4570.80
lat (usec) : 750=0.01%, 1000=0.17%
lat (msec) : 2=3.79%, 4=30.50%, 10=56.62%, 20=1.52%, 50=4.01%
lat (msec) : 100=2.45%, 250=0.90%, 500=0.02%, 750=0.01%, 1000=0.01%
lat (msec) : 2000=0.01%
cpu : usr=0.97%, sys=3.02%, ctx=39689, majf=0, minf=25
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=205329/d=0, short=r=0/w=0/d=0
file1: (groupid=0, jobs=1): err= 0: pid=3999
write: io=849880KB, bw=14154KB/s, iops=3538 , runt= 60047msec
slat (usec): min=4 , max=123486 , avg=105.83, stdev=2216.28
clat (usec): min=446 , max=1383.1K, avg=8926.80, stdev=19692.96
lat (usec): min=461 , max=1384.2K, avg=9032.43, stdev=20445.23
clat percentiles (usec):
| 1.00th=[ 1368], 5.00th=[ 1944], 10.00th=[ 2352], 20.00th=[ 3024],
| 30.00th=[ 3632], 40.00th=[ 4256], 50.00th=[ 5024], 60.00th=[ 5664],
| 70.00th=[ 6368], 80.00th=[ 7264], 90.00th=[ 9024], 95.00th=[33536],
| 99.00th=[91648], 99.50th=[146432], 99.90th=[220160]
bw (KB/s) : min= 5807, max=48232, per=4.48%, avg=14192.07, stdev=5047.52
lat (usec) : 500=0.01%, 750=0.01%, 1000=0.18%
lat (msec) : 2=5.43%, 4=30.44%, 10=55.07%, 20=1.70%, 50=3.99%
lat (msec) : 100=2.27%, 250=0.86%, 500=0.02%, 750=0.01%, 1000=0.01%
lat (msec) : 2000=0.01%
cpu : usr=1.03%, sys=3.21%, ctx=40302, majf=0, minf=24
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=212470/d=0, short=r=0/w=0/d=0
file1: (groupid=0, jobs=1): err= 0: pid=4000
write: io=773688KB, bw=12894KB/s, iops=3223 , runt= 60003msec
slat (usec): min=4 , max=126488 , avg=119.41, stdev=2415.30
clat (usec): min=694 , max=949429 , avg=9803.35, stdev=20713.09
lat (usec): min=721 , max=949436 , avg=9922.96, stdev=21497.37
clat percentiles (usec):
| 1.00th=[ 1496], 5.00th=[ 2256], 10.00th=[ 2736], 20.00th=[ 3408],
| 30.00th=[ 3984], 40.00th=[ 4640], 50.00th=[ 5280], 60.00th=[ 5856],
| 70.00th=[ 6560], 80.00th=[ 7392], 90.00th=[ 9536], 95.00th=[38656],
| 99.00th=[104960], 99.50th=[154624], 99.90th=[228352]
bw (KB/s) : min= 5880, max=19437, per=4.04%, avg=12796.90, stdev=3142.95
lat (usec) : 750=0.01%, 1000=0.09%
lat (msec) : 2=2.99%, 4=26.97%, 10=60.34%, 20=1.76%, 50=4.06%
lat (msec) : 100=2.70%, 250=1.04%, 500=0.04%, 750=0.01%, 1000=0.01%
cpu : usr=0.89%, sys=2.93%, ctx=39334, majf=0, minf=24
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=193422/d=0, short=r=0/w=0/d=0
file1: (groupid=0, jobs=1): err= 0: pid=4001
write: io=769892KB, bw=12831KB/s, iops=3207 , runt= 60004msec
slat (usec): min=4 , max=128937 , avg=125.40, stdev=2497.02
clat (usec): min=543 , max=1641.2K, avg=9845.70, stdev=21517.57
lat (usec): min=557 , max=1641.2K, avg=9971.31, stdev=22291.20
clat percentiles (usec):
| 1.00th=[ 1432], 5.00th=[ 2096], 10.00th=[ 2576], 20.00th=[ 3280],
| 30.00th=[ 3888], 40.00th=[ 4576], 50.00th=[ 5280], 60.00th=[ 5856],
| 70.00th=[ 6560], 80.00th=[ 7392], 90.00th=[ 9920], 95.00th=[38144],
| 99.00th=[104960], 99.50th=[154624], 99.90th=[232448]
bw (KB/s) : min= 6854, max=23338, per=4.03%, avg=12774.62, stdev=3244.76
lat (usec) : 750=0.01%, 1000=0.17%
lat (msec) : 2=3.97%, 4=27.74%, 10=58.14%, 20=1.71%, 50=4.53%
lat (msec) : 100=2.68%, 250=1.00%, 500=0.04%, 750=0.01%, 1000=0.01%
lat (msec) : 2000=0.01%
cpu : usr=0.89%, sys=3.00%, ctx=38235, majf=0, minf=26
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=192473/d=0, short=r=0/w=0/d=0
file1: (groupid=0, jobs=1): err= 0: pid=4002
write: io=828440KB, bw=13807KB/s, iops=3451 , runt= 60003msec
slat (usec): min=4 , max=113456 , avg=107.59, stdev=2251.36
clat (usec): min=717 , max=1012.1K, avg=9159.59, stdev=19972.65
lat (usec): min=731 , max=1013.3K, avg=9267.38, stdev=20704.40
clat percentiles (usec):
| 1.00th=[ 1416], 5.00th=[ 2024], 10.00th=[ 2480], 20.00th=[ 3120],
| 30.00th=[ 3696], 40.00th=[ 4384], 50.00th=[ 5088], 60.00th=[ 5664],
| 70.00th=[ 6432], 80.00th=[ 7264], 90.00th=[ 9024], 95.00th=[35584],
| 99.00th=[98816], 99.50th=[154624], 99.90th=[226304]
bw (KB/s) : min= 4280, max=25462, per=4.35%, avg=13769.89, stdev=4158.74
lat (usec) : 750=0.01%, 1000=0.12%
lat (msec) : 2=4.67%, 4=29.90%, 10=56.57%, 20=1.58%, 50=3.82%
lat (msec) : 100=2.35%, 250=0.94%, 500=0.03%, 750=0.01%, 1000=0.01%
lat (msec) : 2000=0.01%
cpu : usr=0.99%, sys=3.13%, ctx=40961, majf=0, minf=24
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=207110/d=0, short=r=0/w=0/d=0
file1: (groupid=0, jobs=1): err= 0: pid=4003
write: io=788452KB, bw=13090KB/s, iops=3272 , runt= 60231msec
slat (usec): min=4 , max=135424 , avg=119.03, stdev=2426.72
clat (usec): min=514 , max=1681.9K, avg=9614.94, stdev=21243.36
lat (usec): min=527 , max=1681.9K, avg=9734.19, stdev=22016.46
clat percentiles (usec):
| 1.00th=[ 1336], 5.00th=[ 2064], 10.00th=[ 2512], 20.00th=[ 3216],
| 30.00th=[ 3792], 40.00th=[ 4448], 50.00th=[ 5152], 60.00th=[ 5728],
| 70.00th=[ 6432], 80.00th=[ 7328], 90.00th=[ 9280], 95.00th=[38144],
| 99.00th=[104960], 99.50th=[158720], 99.90th=[228352]
bw (KB/s) : min= 4976, max=27612, per=4.13%, avg=13082.37, stdev=3858.63
lat (usec) : 750=0.01%, 1000=0.14%
lat (msec) : 2=4.44%, 4=28.95%, 10=57.10%, 20=1.67%, 50=3.93%
lat (msec) : 100=2.66%, 250=1.04%, 500=0.05%, 750=0.01%, 1000=0.01%
lat (msec) : 2000=0.01%
cpu : usr=0.96%, sys=3.04%, ctx=39625, majf=0, minf=25
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=197113/d=0, short=r=0/w=0/d=0
file1: (groupid=0, jobs=1): err= 0: pid=4004
write: io=790536KB, bw=13175KB/s, iops=3293 , runt= 60002msec
slat (usec): min=4 , max=114963 , avg=119.88, stdev=2419.63
clat (usec): min=537 , max=1023.4K, avg=9591.03, stdev=20883.64
lat (usec): min=552 , max=1023.4K, avg=9711.11, stdev=21661.57
clat percentiles (usec):
| 1.00th=[ 1400], 5.00th=[ 2024], 10.00th=[ 2480], 20.00th=[ 3216],
| 30.00th=[ 3824], 40.00th=[ 4512], 50.00th=[ 5216], 60.00th=[ 5792],
| 70.00th=[ 6496], 80.00th=[ 7328], 90.00th=[ 9280], 95.00th=[38144],
| 99.00th=[101888], 99.50th=[158720], 99.90th=[228352]
bw (KB/s) : min= 4088, max=30406, per=4.13%, avg=13071.40, stdev=4094.62
lat (usec) : 750=0.01%, 1000=0.13%
lat (msec) : 2=4.63%, 4=28.15%, 10=57.73%, 20=1.64%, 50=4.03%
lat (msec) : 100=2.63%, 250=0.99%, 500=0.04%, 750=0.01%, 1000=0.01%
lat (msec) : 2000=0.01%
cpu : usr=0.97%, sys=2.95%, ctx=39111, majf=0, minf=25
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=197634/d=0, short=r=0/w=0/d=0
file1: (groupid=0, jobs=1): err= 0: pid=4005
write: io=877548KB, bw=14626KB/s, iops=3656 , runt= 60001msec
slat (usec): min=4 , max=113739 , avg=101.26, stdev=2181.30
clat (usec): min=500 , max=1509.9K, avg=8646.60, stdev=18853.03
lat (usec): min=517 , max=1509.9K, avg=8748.06, stdev=19575.58
clat percentiles (usec):
| 1.00th=[ 1272], 5.00th=[ 1848], 10.00th=[ 2288], 20.00th=[ 2992],
| 30.00th=[ 3568], 40.00th=[ 4192], 50.00th=[ 4896], 60.00th=[ 5536],
| 70.00th=[ 6240], 80.00th=[ 7136], 90.00th=[ 8640], 95.00th=[34048],
| 99.00th=[87552], 99.50th=[132096], 99.90th=[218112]
bw (KB/s) : min= 6671, max=30083, per=4.54%, avg=14369.59, stdev=4168.17
lat (usec) : 750=0.01%, 1000=0.16%
lat (msec) : 2=6.39%, 4=30.90%, 10=54.36%, 20=1.32%, 50=3.77%
lat (msec) : 100=2.31%, 250=0.73%, 500=0.03%, 750=0.01%, 1000=0.01%
lat (msec) : 2000=0.01%
cpu : usr=0.99%, sys=3.31%, ctx=40531, majf=0, minf=24
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=219387/d=0, short=r=0/w=0/d=0
file1: (groupid=0, jobs=1): err= 0: pid=4006
write: io=770940KB, bw=12849KB/s, iops=3212 , runt= 60002msec
slat (usec): min=4 , max=117996 , avg=124.61, stdev=2459.97
clat (usec): min=446 , max=1537.3K, avg=9834.30, stdev=22022.92
lat (usec): min=458 , max=1537.7K, avg=9959.11, stdev=22800.07
clat percentiles (usec):
| 1.00th=[ 1304], 5.00th=[ 1880], 10.00th=[ 2320], 20.00th=[ 3088],
| 30.00th=[ 3728], 40.00th=[ 4384], 50.00th=[ 5152], 60.00th=[ 5728],
| 70.00th=[ 6496], 80.00th=[ 7392], 90.00th=[ 9792], 95.00th=[39168],
| 99.00th=[115200], 99.50th=[164864], 99.90th=[228352]
bw (KB/s) : min= 3659, max=40632, per=4.00%, avg=12683.36, stdev=4691.41
lat (usec) : 500=0.01%, 750=0.01%, 1000=0.21%
lat (msec) : 2=6.05%, 4=28.06%, 10=55.83%, 20=1.69%, 50=4.23%
lat (msec) : 100=2.72%, 250=1.12%, 500=0.05%, 750=0.01%, 1000=0.01%
lat (msec) : 2000=0.01%
cpu : usr=0.93%, sys=2.93%, ctx=38130, majf=0, minf=25
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=192735/d=0, short=r=0/w=0/d=0
file1: (groupid=0, jobs=1): err= 0: pid=4007
write: io=784360KB, bw=13005KB/s, iops=3251 , runt= 60314msec
slat (usec): min=4 , max=128950 , avg=119.95, stdev=2394.53
clat (usec): min=527 , max=1082.5K, avg=9666.92, stdev=20835.35
lat (usec): min=553 , max=1082.5K, avg=9787.07, stdev=21616.62
clat percentiles (usec):
| 1.00th=[ 1368], 5.00th=[ 2040], 10.00th=[ 2480], 20.00th=[ 3248],
| 30.00th=[ 3856], 40.00th=[ 4512], 50.00th=[ 5216], 60.00th=[ 5856],
| 70.00th=[ 6496], 80.00th=[ 7328], 90.00th=[ 9536], 95.00th=[37632],
| 99.00th=[101888], 99.50th=[156672], 99.90th=[226304]
bw (KB/s) : min= 4433, max=26743, per=4.13%, avg=13096.11, stdev=3884.65
lat (usec) : 750=0.01%, 1000=0.17%
lat (msec) : 2=4.41%, 4=27.75%, 10=58.12%, 20=1.63%, 50=4.21%
lat (msec) : 100=2.66%, 250=0.99%, 500=0.04%, 750=0.01%, 1000=0.01%
lat (msec) : 2000=0.01%
cpu : usr=0.89%, sys=3.05%, ctx=39152, majf=0, minf=25
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=196090/d=0, short=r=0/w=0/d=0
file1: (groupid=0, jobs=1): err= 0: pid=4008
write: io=784736KB, bw=13061KB/s, iops=3265 , runt= 60084msec
slat (usec): min=4 , max=112108 , avg=121.25, stdev=2421.10
clat (usec): min=583 , max=1580.3K, avg=9660.47, stdev=21404.93
lat (usec): min=595 , max=1603.4K, avg=9781.91, stdev=22175.06
clat percentiles (usec):
| 1.00th=[ 1464], 5.00th=[ 2064], 10.00th=[ 2512], 20.00th=[ 3216],
| 30.00th=[ 3856], 40.00th=[ 4576], 50.00th=[ 5280], 60.00th=[ 5856],
| 70.00th=[ 6496], 80.00th=[ 7392], 90.00th=[ 9408], 95.00th=[38144],
| 99.00th=[95744], 99.50th=[152576], 99.90th=[226304]
bw (KB/s) : min= 6019, max=31992, per=4.13%, avg=13095.62, stdev=4157.33
lat (usec) : 750=0.01%, 1000=0.12%
lat (msec) : 2=4.19%, 4=27.99%, 10=58.12%, 20=1.54%, 50=4.28%
lat (msec) : 100=2.79%, 250=0.90%, 500=0.04%, 750=0.01%, 1000=0.01%
lat (msec) : 2000=0.01%
cpu : usr=0.94%, sys=2.99%, ctx=38544, majf=0, minf=24
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=196184/d=0, short=r=0/w=0/d=0
file1: (groupid=0, jobs=1): err= 0: pid=4010
write: io=815684KB, bw=13594KB/s, iops=3398 , runt= 60003msec
slat (usec): min=4 , max=141599 , avg=111.80, stdev=2314.86
clat (usec): min=600 , max=1217.5K, avg=9300.28, stdev=20275.17
lat (usec): min=610 , max=1217.5K, avg=9412.28, stdev=21008.51
clat percentiles (usec):
| 1.00th=[ 1480], 5.00th=[ 2128], 10.00th=[ 2544], 20.00th=[ 3152],
| 30.00th=[ 3760], 40.00th=[ 4448], 50.00th=[ 5152], 60.00th=[ 5728],
| 70.00th=[ 6432], 80.00th=[ 7264], 90.00th=[ 8896], 95.00th=[36096],
| 99.00th=[97792], 99.50th=[152576], 99.90th=[222208]
bw (KB/s) : min= 5160, max=24846, per=4.29%, avg=13584.51, stdev=3625.42
lat (usec) : 750=0.01%, 1000=0.09%
lat (msec) : 2=3.81%, 4=29.84%, 10=57.34%, 20=1.53%, 50=3.84%
lat (msec) : 100=2.57%, 250=0.93%, 500=0.03%, 750=0.01%, 1000=0.01%
lat (msec) : 2000=0.01%
cpu : usr=0.92%, sys=3.20%, ctx=39886, majf=0, minf=25
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=203921/d=0, short=r=0/w=0/d=0
file1: (groupid=0, jobs=1): err= 0: pid=4011
write: io=784100KB, bw=13055KB/s, iops=3263 , runt= 60062msec
slat (usec): min=4 , max=99300 , avg=116.71, stdev=2345.60
clat (usec): min=591 , max=838633 , avg=9673.18, stdev=21239.11
lat (usec): min=605 , max=838655 , avg=9789.84, stdev=21997.58
clat percentiles (usec):
| 1.00th=[ 1384], 5.00th=[ 1960], 10.00th=[ 2448], 20.00th=[ 3216],
| 30.00th=[ 3792], 40.00th=[ 4512], 50.00th=[ 5216], 60.00th=[ 5792],
| 70.00th=[ 6496], 80.00th=[ 7328], 90.00th=[ 9408], 95.00th=[37120],
| 99.00th=[108032], 99.50th=[162816], 99.90th=[230400]
bw (KB/s) : min= 5988, max=28728, per=4.13%, avg=13098.09, stdev=3959.17
lat (usec) : 750=0.01%, 1000=0.11%
lat (msec) : 2=5.19%, 4=27.85%, 10=57.38%, 20=1.66%, 50=4.18%
lat (msec) : 100=2.52%, 250=1.05%, 500=0.04%, 750=0.01%, 1000=0.01%
cpu : usr=0.97%, sys=2.94%, ctx=40022, majf=0, minf=24
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=196025/d=0, short=r=0/w=0/d=0
file1: (groupid=0, jobs=1): err= 0: pid=4012
write: io=784720KB, bw=13078KB/s, iops=3269 , runt= 60005msec
slat (usec): min=4 , max=112089 , avg=118.79, stdev=2392.98
clat (usec): min=451 , max=1475.5K, avg=9663.55, stdev=21206.38
lat (usec): min=464 , max=1475.6K, avg=9782.54, stdev=21983.94
clat percentiles (usec):
| 1.00th=[ 1288], 5.00th=[ 1992], 10.00th=[ 2480], 20.00th=[ 3248],
| 30.00th=[ 3888], 40.00th=[ 4576], 50.00th=[ 5280], 60.00th=[ 5856],
| 70.00th=[ 6496], 80.00th=[ 7392], 90.00th=[ 9408], 95.00th=[37120],
| 99.00th=[100864], 99.50th=[158720], 99.90th=[224256]
bw (KB/s) : min= 5946, max=23496, per=4.12%, avg=13059.17, stdev=3462.91
lat (usec) : 500=0.01%, 750=0.01%, 1000=0.18%
lat (msec) : 2=4.84%, 4=26.82%, 10=58.59%, 20=1.76%, 50=4.07%
lat (msec) : 100=2.70%, 250=0.98%, 500=0.04%, 750=0.01%, 1000=0.01%
lat (msec) : 2000=0.01%
cpu : usr=0.95%, sys=2.99%, ctx=39607, majf=0, minf=25
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=196180/d=0, short=r=0/w=0/d=0
file1: (groupid=0, jobs=1): err= 0: pid=4013
write: io=793884KB, bw=13231KB/s, iops=3307 , runt= 60003msec
slat (usec): min=4 , max=115547 , avg=116.91, stdev=2359.21
clat (usec): min=628 , max=1535.7K, avg=9553.22, stdev=20747.17
lat (usec): min=634 , max=1535.8K, avg=9670.33, stdev=21533.58
clat percentiles (usec):
| 1.00th=[ 1512], 5.00th=[ 2160], 10.00th=[ 2640], 20.00th=[ 3280],
| 30.00th=[ 3856], 40.00th=[ 4512], 50.00th=[ 5216], 60.00th=[ 5792],
| 70.00th=[ 6496], 80.00th=[ 7328], 90.00th=[ 9536], 95.00th=[37632],
| 99.00th=[99840], 99.50th=[152576], 99.90th=[224256]
bw (KB/s) : min= 6772, max=27640, per=4.15%, avg=13157.26, stdev=3657.51
lat (usec) : 750=0.01%, 1000=0.07%
lat (msec) : 2=3.52%, 4=28.95%, 10=57.77%, 20=1.86%, 50=4.24%
lat (msec) : 100=2.57%, 250=0.96%, 500=0.03%, 750=0.01%, 1000=0.01%
lat (msec) : 2000=0.01%
cpu : usr=0.95%, sys=2.97%, ctx=38933, majf=0, minf=25
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=198471/d=0, short=r=0/w=0/d=0
file1: (groupid=0, jobs=1): err= 0: pid=4014
write: io=826984KB, bw=13782KB/s, iops=3445 , runt= 60003msec
slat (usec): min=4 , max=126500 , avg=110.53, stdev=2316.49
clat (usec): min=605 , max=1631.9K, avg=9172.40, stdev=20560.58
lat (usec): min=619 , max=1653.4K, avg=9283.13, stdev=21293.23
clat percentiles (usec):
| 1.00th=[ 1352], 5.00th=[ 2040], 10.00th=[ 2480], 20.00th=[ 3120],
| 30.00th=[ 3696], 40.00th=[ 4320], 50.00th=[ 5024], 60.00th=[ 5664],
| 70.00th=[ 6368], 80.00th=[ 7264], 90.00th=[ 9024], 95.00th=[36096],
| 99.00th=[93696], 99.50th=[148480], 99.90th=[222208]
bw (KB/s) : min= 4413, max=27808, per=4.28%, avg=13551.74, stdev=3870.91
lat (usec) : 750=0.01%, 1000=0.12%
lat (msec) : 2=4.57%, 4=30.57%, 10=55.76%, 20=1.70%, 50=3.95%
lat (msec) : 100=2.39%, 250=0.90%, 500=0.03%, 750=0.01%, 1000=0.01%
lat (msec) : 2000=0.01%
cpu : usr=1.01%, sys=3.06%, ctx=39314, majf=0, minf=24
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=206746/d=0, short=r=0/w=0/d=0
file1: (groupid=0, jobs=1): err= 0: pid=4015
write: io=826752KB, bw=13779KB/s, iops=3444 , runt= 60003msec
slat (usec): min=4 , max=116274 , avg=108.34, stdev=2274.99
clat (usec): min=610 , max=1268.9K, avg=9177.07, stdev=20181.03
lat (usec): min=622 , max=1268.9K, avg=9285.61, stdev=20895.62
clat percentiles (usec):
| 1.00th=[ 1416], 5.00th=[ 2128], 10.00th=[ 2608], 20.00th=[ 3248],
| 30.00th=[ 3792], 40.00th=[ 4448], 50.00th=[ 5152], 60.00th=[ 5728],
| 70.00th=[ 6432], 80.00th=[ 7264], 90.00th=[ 9024], 95.00th=[35072],
| 99.00th=[94720], 99.50th=[144384], 99.90th=[224256]
bw (KB/s) : min= 6374, max=30040, per=4.35%, avg=13772.31, stdev=4065.37
lat (usec) : 750=0.01%, 1000=0.12%
lat (msec) : 2=3.91%, 4=29.33%, 10=57.75%, 20=1.68%, 50=3.86%
lat (msec) : 100=2.45%, 250=0.84%, 500=0.04%, 750=0.01%, 1000=0.01%
lat (msec) : 2000=0.01%
cpu : usr=0.91%, sys=3.16%, ctx=40150, majf=0, minf=25
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=206688/d=0, short=r=0/w=0/d=0
file1: (groupid=0, jobs=1): err= 0: pid=4016
write: io=774808KB, bw=12899KB/s, iops=3224 , runt= 60066msec
slat (usec): min=4 , max=116276 , avg=122.45, stdev=2450.71
clat (usec): min=492 , max=1488.5K, avg=9785.13, stdev=21698.70
lat (usec): min=505 , max=1488.5K, avg=9907.78, stdev=22471.59
clat percentiles (usec):
| 1.00th=[ 1400], 5.00th=[ 2096], 10.00th=[ 2544], 20.00th=[ 3280],
| 30.00th=[ 3888], 40.00th=[ 4576], 50.00th=[ 5280], 60.00th=[ 5856],
| 70.00th=[ 6560], 80.00th=[ 7392], 90.00th=[ 9664], 95.00th=[38144],
| 99.00th=[101888], 99.50th=[154624], 99.90th=[228352]
bw (KB/s) : min= 5100, max=24714, per=4.08%, avg=12937.68, stdev=3431.60
lat (usec) : 500=0.01%, 750=0.01%, 1000=0.13%
lat (msec) : 2=4.14%, 4=27.53%, 10=58.41%, 20=1.78%, 50=4.26%
lat (msec) : 100=2.68%, 250=0.97%, 500=0.05%, 750=0.01%, 1000=0.01%
lat (msec) : 2000=0.01%
cpu : usr=0.95%, sys=2.97%, ctx=39454, majf=0, minf=25
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=193702/d=0, short=r=0/w=0/d=0
file1: (groupid=0, jobs=1): err= 0: pid=4018
write: io=802892KB, bw=13245KB/s, iops=3311 , runt= 60617msec
slat (usec): min=4 , max=115549 , avg=114.45, stdev=2351.84
clat (usec): min=443 , max=1541.4K, avg=9448.96, stdev=20788.49
lat (usec): min=460 , max=1596.1K, avg=9563.61, stdev=21560.78
clat percentiles (usec):
| 1.00th=[ 1240], 5.00th=[ 2008], 10.00th=[ 2544], 20.00th=[ 3248],
| 30.00th=[ 3856], 40.00th=[ 4512], 50.00th=[ 5216], 60.00th=[ 5792],
| 70.00th=[ 6496], 80.00th=[ 7328], 90.00th=[ 9152], 95.00th=[36608],
| 99.00th=[98816], 99.50th=[152576], 99.90th=[224256]
bw (KB/s) : min= 1731, max=24256, per=4.20%, avg=13307.98, stdev=3643.41
lat (usec) : 500=0.01%, 750=0.01%, 1000=0.25%
lat (msec) : 2=4.71%, 4=27.47%, 10=58.34%, 20=1.62%, 50=4.11%
lat (msec) : 100=2.48%, 250=0.95%, 500=0.04%, 750=0.01%, 1000=0.01%
lat (msec) : 2000=0.01%
cpu : usr=0.94%, sys=3.06%, ctx=40190, majf=0, minf=24
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=200723/d=0, short=r=0/w=0/d=0
file1: (groupid=0, jobs=1): err= 0: pid=4019
write: io=778136KB, bw=12945KB/s, iops=3236 , runt= 60111msec
slat (usec): min=4 , max=135424 , avg=117.66, stdev=2386.05
clat (usec): min=452 , max=1677.3K, avg=9747.29, stdev=22049.42
lat (usec): min=469 , max=1677.4K, avg=9864.91, stdev=22787.60
clat percentiles (usec):
| 1.00th=[ 1384], 5.00th=[ 2128], 10.00th=[ 2576], 20.00th=[ 3248],
| 30.00th=[ 3824], 40.00th=[ 4512], 50.00th=[ 5216], 60.00th=[ 5792],
| 70.00th=[ 6496], 80.00th=[ 7328], 90.00th=[ 9408], 95.00th=[37120],
| 99.00th=[113152], 99.50th=[162816], 99.90th=[224256]
bw (KB/s) : min= 6296, max=38856, per=4.10%, avg=12994.39, stdev=4142.39
lat (usec) : 500=0.01%, 750=0.01%, 1000=0.14%
lat (msec) : 2=3.85%, 4=28.97%, 10=57.45%, 20=1.83%, 50=4.00%
lat (msec) : 100=2.63%, 250=1.06%, 500=0.04%, 750=0.01%, 1000=0.01%
lat (msec) : 2000=0.01%
cpu : usr=0.85%, sys=3.06%, ctx=40263, majf=0, minf=25
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=194534/d=0, short=r=0/w=0/d=0
Run status group 0 (all jobs):
WRITE: io=18756MB, aggrb=316840KB/s, minb=13138KB/s, maxb=14976KB/s,
mint=60001msec, maxt=60617msec
Disk stats (read/write):
bcache0: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
--
Founder | Director | VP Research
Orion Virtualisation Solutions | www.orionvm.com.au | Phone: 1300 56
99 52 | Mobile: 0428 754 846
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Preliminary performance benchmarks.
[not found] ` <CAOzFzEgYeru29S8BnPr+y=C1=ess1d8F4v3wRFqGwiTHiWdGRQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
@ 2012-04-13 11:40 ` Leen Besselink
0 siblings, 0 replies; 5+ messages in thread
From: Leen Besselink @ 2012-04-13 11:40 UTC (permalink / raw)
To: linux-bcache-u79uwXL29TY76Z2rM5mHXA
On Fri, Apr 13, 2012 at 11:43:14AM +1000, Joseph Glanville wrote:
> On 13 April 2012 10:05, Leen Besselink <leen-RaxvPIb8m1QxPGQnf4u0eF6hYfS7NtTn@public.gmane.org> wrote:
> > (I've been lurking the mailinglist archive for a few months already, I think
> > bcache sounds really interresting)
> >
> > Hello Joseph,
> >
> >> Carrying out some scalability tests on high I/O systems. More tests to
> >> come using the Phoronix Test Suite.
> >>
> >> Fio summary:
> >> 24 jobs
> >> Direct IO
> >> Randwrite test
> >> Total of about 80k IOPs at 3.5k IOPS per thread.
> >>
> >> Test rig specs:
> >> 2x Intel(R) Xeon(R) CPU X5650 @ 2.67GHz (12 physical cores, 24 logical)
> >> 12x 2TB Seagate nearline-SAS in RAID6 on LSI Logic / Symbios Logic LSI
> >> MegaSAS 9260
> >> 2x Intel 520 SSDs 120GB in RAID0
> >>
> >> The 2 520s are striped with md raid as /dev/md0 whch is formatted as a
> >> bcache cache device using 1M buckets and 8k hard block size
> >> Backing device is the big old raid6.
> >>
> >> Random IO performance of the native raid0 is about 96K IOPs, backing
> >> device in the realm of 1600ish.
> >> However the backing device has a sequential IO performance of about 1.5GB/s.
> >>
> >
> > So why do you put your SSDs in RAID0 ?
> >
> > If you are using RAID6 for your HDD you obviously care about your data, shouldn't
> > you be using RAID1 or let bcache do some RAID1-like behaviour* ?
> >
> > Because when you use RAID0 your data will only be written to one SSD.
> >
> > * I think bcache had some ability to handle that sort of automatically.
>
> I used RAID0 to demonstrate bcache scaling to 80k IOPs, this was not a
> production machine nor a production configuration.
>
OK, that explains it.
> Yes, atleast somewhat. Bcache cache devices are structured into
> cache-sets, which are groups of SSDs that form one logical cache which
> you can attach multiple backing devices to.
> This is different from RAID1 because only dirty data is mirrored. You
> don't need redundant copies of your read cache but you care very much
> that you don't lose writeback cache data.
> This isn't fully fleshed out in bcache yet but all the infrastructure
> is there, I am looking to doing some work on it soonish I have just
> been busy with other things.
Yes, only mirror the dirty data. That is what it was, now I remember.
Good to know what the state of development is also.
Thanks.
> >
> >> Below are some quick findings using fio - showing very good
> >> scalability of bcache even with very very fast SSDs.
> >> *Note: The SSDs are connected via a SATA2 interface being somewhat of
> >> a bottleneck.
> >
> > Have a nice day,
> > Leen.
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-bcache" in
> > the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> > More majordomo info at http://vger.kernel.org/majordomo-info.html
>
> Joseph.
>
> --
> Founder | Director | VP Research
> Orion Virtualisation Solutions | www.orionvm.com.au | Phone: 1300 56
> 99 52 | Mobile: 0428 754 846
> --
> To unsubscribe from this list: send the line "unsubscribe linux-bcache" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Preliminary performance benchmarks.
[not found] ` <20120413000546.GB2941-4L8T6chqF5cLUefWDxyFbw@public.gmane.org>
2012-04-13 0:38 ` Kent Overstreet
@ 2012-04-13 1:43 ` Joseph Glanville
[not found] ` <CAOzFzEgYeru29S8BnPr+y=C1=ess1d8F4v3wRFqGwiTHiWdGRQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
1 sibling, 1 reply; 5+ messages in thread
From: Joseph Glanville @ 2012-04-13 1:43 UTC (permalink / raw)
To: leen-RaxvPIb8m1QxPGQnf4u0eF6hYfS7NtTn; +Cc: linux-bcache-u79uwXL29TY76Z2rM5mHXA
On 13 April 2012 10:05, Leen Besselink <leen-RaxvPIb8m1QxPGQnf4u0eF6hYfS7NtTn@public.gmane.org> wrote:
> (I've been lurking the mailinglist archive for a few months already, I think
> bcache sounds really interresting)
>
> Hello Joseph,
>
>> Carrying out some scalability tests on high I/O systems. More tests to
>> come using the Phoronix Test Suite.
>>
>> Fio summary:
>> 24 jobs
>> Direct IO
>> Randwrite test
>> Total of about 80k IOPs at 3.5k IOPS per thread.
>>
>> Test rig specs:
>> 2x Intel(R) Xeon(R) CPU X5650 @ 2.67GHz (12 physical cores, 24 logical)
>> 12x 2TB Seagate nearline-SAS in RAID6 on LSI Logic / Symbios Logic LSI
>> MegaSAS 9260
>> 2x Intel 520 SSDs 120GB in RAID0
>>
>> The 2 520s are striped with md raid as /dev/md0 whch is formatted as a
>> bcache cache device using 1M buckets and 8k hard block size
>> Backing device is the big old raid6.
>>
>> Random IO performance of the native raid0 is about 96K IOPs, backing
>> device in the realm of 1600ish.
>> However the backing device has a sequential IO performance of about 1.5GB/s.
>>
>
> So why do you put your SSDs in RAID0 ?
>
> If you are using RAID6 for your HDD you obviously care about your data, shouldn't
> you be using RAID1 or let bcache do some RAID1-like behaviour* ?
>
> Because when you use RAID0 your data will only be written to one SSD.
>
> * I think bcache had some ability to handle that sort of automatically.
I used RAID0 to demonstrate bcache scaling to 80k IOPs, this was not a
production machine nor a production configuration.
Yes, atleast somewhat. Bcache cache devices are structured into
cache-sets, which are groups of SSDs that form one logical cache which
you can attach multiple backing devices to.
This is different from RAID1 because only dirty data is mirrored. You
don't need redundant copies of your read cache but you care very much
that you don't lose writeback cache data.
This isn't fully fleshed out in bcache yet but all the infrastructure
is there, I am looking to doing some work on it soonish I have just
been busy with other things.
>
>> Below are some quick findings using fio - showing very good
>> scalability of bcache even with very very fast SSDs.
>> *Note: The SSDs are connected via a SATA2 interface being somewhat of
>> a bottleneck.
>
> Have a nice day,
> Leen.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-bcache" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
Joseph.
--
Founder | Director | VP Research
Orion Virtualisation Solutions | www.orionvm.com.au | Phone: 1300 56
99 52 | Mobile: 0428 754 846
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Preliminary performance benchmarks.
[not found] ` <20120413000546.GB2941-4L8T6chqF5cLUefWDxyFbw@public.gmane.org>
@ 2012-04-13 0:38 ` Kent Overstreet
2012-04-13 1:43 ` Joseph Glanville
1 sibling, 0 replies; 5+ messages in thread
From: Kent Overstreet @ 2012-04-13 0:38 UTC (permalink / raw)
To: leen-RaxvPIb8m1QxPGQnf4u0eF6hYfS7NtTn; +Cc: linux-bcache-u79uwXL29TY76Z2rM5mHXA
On Thu, Apr 12, 2012 at 5:05 PM, Leen Besselink <leen-RaxvPIb8m1QxPGQnf4u0eF6hYfS7NtTn@public.gmane.org> wrote:
> (I've been lurking the mailinglist archive for a few months already, I think
> bcache sounds really interresting)
>
> Hello Joseph,
>
>> Carrying out some scalability tests on high I/O systems. More tests to
>> come using the Phoronix Test Suite.
>>
>> Fio summary:
>> 24 jobs
>> Direct IO
>> Randwrite test
>> Total of about 80k IOPs at 3.5k IOPS per thread.
>>
>> Test rig specs:
>> 2x Intel(R) Xeon(R) CPU X5650 @ 2.67GHz (12 physical cores, 24 logical)
>> 12x 2TB Seagate nearline-SAS in RAID6 on LSI Logic / Symbios Logic LSI
>> MegaSAS 9260
>> 2x Intel 520 SSDs 120GB in RAID0
>>
>> The 2 520s are striped with md raid as /dev/md0 whch is formatted as a
>> bcache cache device using 1M buckets and 8k hard block size
>> Backing device is the big old raid6.
>>
>> Random IO performance of the native raid0 is about 96K IOPs, backing
>> device in the realm of 1600ish.
>> However the backing device has a sequential IO performance of about 1.5GB/s.
>>
>
> So why do you put your SSDs in RAID0 ?
>
> If you are using RAID6 for your HDD you obviously care about your data, shouldn't
> you be using RAID1 or let bcache do some RAID1-like behaviour* ?
>
> Because when you use RAID0 your data will only be written to one SSD.
>
> * I think bcache had some ability to handle that sort of automatically.
That's been in the design since forever, but I haven't gotten around
to finishing it since we haven't needed it internally.
(But yes, the idea is to have multile SSDs in a cache set, with only
dirty data and metadata mirrored. Wouldn't be a _huge_ amount of work
to finish.)
>
>> Below are some quick findings using fio - showing very good
>> scalability of bcache even with very very fast SSDs.
>> *Note: The SSDs are connected via a SATA2 interface being somewhat of
>> a bottleneck.
>
> Have a nice day,
> Leen.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-bcache" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Preliminary performance benchmarks.
@ 2012-04-13 0:05 Leen Besselink
[not found] ` <20120413000546.GB2941-4L8T6chqF5cLUefWDxyFbw@public.gmane.org>
0 siblings, 1 reply; 5+ messages in thread
From: Leen Besselink @ 2012-04-13 0:05 UTC (permalink / raw)
To: linux-bcache-u79uwXL29TY76Z2rM5mHXA
(I've been lurking the mailinglist archive for a few months already, I think
bcache sounds really interresting)
Hello Joseph,
> Carrying out some scalability tests on high I/O systems. More tests to
> come using the Phoronix Test Suite.
>
> Fio summary:
> 24 jobs
> Direct IO
> Randwrite test
> Total of about 80k IOPs at 3.5k IOPS per thread.
>
> Test rig specs:
> 2x Intel(R) Xeon(R) CPU X5650 @ 2.67GHz (12 physical cores, 24 logical)
> 12x 2TB Seagate nearline-SAS in RAID6 on LSI Logic / Symbios Logic LSI
> MegaSAS 9260
> 2x Intel 520 SSDs 120GB in RAID0
>
> The 2 520s are striped with md raid as /dev/md0 whch is formatted as a
> bcache cache device using 1M buckets and 8k hard block size
> Backing device is the big old raid6.
>
> Random IO performance of the native raid0 is about 96K IOPs, backing
> device in the realm of 1600ish.
> However the backing device has a sequential IO performance of about 1.5GB/s.
>
So why do you put your SSDs in RAID0 ?
If you are using RAID6 for your HDD you obviously care about your data, shouldn't
you be using RAID1 or let bcache do some RAID1-like behaviour* ?
Because when you use RAID0 your data will only be written to one SSD.
* I think bcache had some ability to handle that sort of automatically.
> Below are some quick findings using fio - showing very good
> scalability of bcache even with very very fast SSDs.
> *Note: The SSDs are connected via a SATA2 interface being somewhat of
> a bottleneck.
Have a nice day,
Leen.
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2012-04-13 11:40 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2012-03-03 1:35 Preliminary performance benchmarks Joseph Glanville
2012-04-13 0:05 Leen Besselink
[not found] ` <20120413000546.GB2941-4L8T6chqF5cLUefWDxyFbw@public.gmane.org>
2012-04-13 0:38 ` Kent Overstreet
2012-04-13 1:43 ` Joseph Glanville
[not found] ` <CAOzFzEgYeru29S8BnPr+y=C1=ess1d8F4v3wRFqGwiTHiWdGRQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2012-04-13 11:40 ` Leen Besselink
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).