* speed decrease since firefly,giant,hammer the 2nd try
@ 2015-02-10 18:55 Stefan Priebe
2015-02-10 19:05 ` Gregory Farnum
` (2 more replies)
0 siblings, 3 replies; 32+ messages in thread
From: Stefan Priebe @ 2015-02-10 18:55 UTC (permalink / raw)
To: ceph-devel
Hello,
last year in june i already reported this but there was no real result.
(http://lists.ceph.com/pipermail/ceph-users-ceph.com/2014-July/041070.html)
I then had the hope that this will be fixed itself when hammer is
released. Now i tried hammer an the results are bad as before.
Since firefly librbd1 / librados2 are 20% slower for 4k random iop/s
than dumpling - this is also the reason why i still stick to dumpling.
I've now modified my test again to be a bit more clear.
Ceph cluster itself completely dumpling.
librbd1 / librados from dumpling (fio inside qemu): 23k iop/s for random
4k writes
- stopped qemu
- cp -ra firefly_0.80.8/usr/lib/librados.so.2.0.0 /usr/lib/
- cp -ra firefly_0.80.8/usr/lib/librbd.so.1.0.0 /usr/lib/
- start qemu
same fio, same qemu, same vm, same host, same ceph dumpling storage,
different librados / librbd: 16k iop/s for random 4k writes
What's wrong with librbd / librados2 since firefly?
Greets,
Stefan
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: speed decrease since firefly,giant,hammer the 2nd try
2015-02-10 18:55 speed decrease since firefly,giant,hammer the 2nd try Stefan Priebe
@ 2015-02-10 19:05 ` Gregory Farnum
2015-02-10 19:12 ` Stefan Priebe
2015-02-10 19:10 ` Mark Nelson
[not found] ` <1971513819.362434.1423640637237.JavaMail.zimbra@oxygem.tv>
2 siblings, 1 reply; 32+ messages in thread
From: Gregory Farnum @ 2015-02-10 19:05 UTC (permalink / raw)
To: Stefan Priebe; +Cc: ceph-devel
On Tue, Feb 10, 2015 at 10:55 AM, Stefan Priebe <s.priebe@profihost.ag> wrote:
> Hello,
>
> last year in june i already reported this but there was no real result.
> (http://lists.ceph.com/pipermail/ceph-users-ceph.com/2014-July/041070.html)
>
> I then had the hope that this will be fixed itself when hammer is released.
> Now i tried hammer an the results are bad as before.
>
> Since firefly librbd1 / librados2 are 20% slower for 4k random iop/s than
> dumpling - this is also the reason why i still stick to dumpling.
>
> I've now modified my test again to be a bit more clear.
>
> Ceph cluster itself completely dumpling.
>
> librbd1 / librados from dumpling (fio inside qemu): 23k iop/s for random 4k
> writes
>
> - stopped qemu
> - cp -ra firefly_0.80.8/usr/lib/librados.so.2.0.0 /usr/lib/
> - cp -ra firefly_0.80.8/usr/lib/librbd.so.1.0.0 /usr/lib/
> - start qemu
>
> same fio, same qemu, same vm, same host, same ceph dumpling storage,
> different librados / librbd: 16k iop/s for random 4k writes
>
> What's wrong with librbd / librados2 since firefly?
We're all going to have the same questions now as we did last time,
about what the cluster looks like, what the perfcounters are reporting
on both versions of librados, etc.
Also, please give us the results from Giant rather than Firefly, for
the reasons I mentioned previously.
-Greg
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: speed decrease since firefly,giant,hammer the 2nd try
2015-02-10 18:55 speed decrease since firefly,giant,hammer the 2nd try Stefan Priebe
2015-02-10 19:05 ` Gregory Farnum
@ 2015-02-10 19:10 ` Mark Nelson
2015-02-10 19:13 ` Stefan Priebe
[not found] ` <1971513819.362434.1423640637237.JavaMail.zimbra@oxygem.tv>
2 siblings, 1 reply; 32+ messages in thread
From: Mark Nelson @ 2015-02-10 19:10 UTC (permalink / raw)
To: Stefan Priebe, ceph-devel
On 02/10/2015 12:55 PM, Stefan Priebe wrote:
> Hello,
>
> last year in june i already reported this but there was no real result.
> (http://lists.ceph.com/pipermail/ceph-users-ceph.com/2014-July/041070.html)
>
> I then had the hope that this will be fixed itself when hammer is
> released. Now i tried hammer an the results are bad as before.
>
> Since firefly librbd1 / librados2 are 20% slower for 4k random iop/s
> than dumpling - this is also the reason why i still stick to dumpling.
>
> I've now modified my test again to be a bit more clear.
>
> Ceph cluster itself completely dumpling.
>
> librbd1 / librados from dumpling (fio inside qemu): 23k iop/s for random
> 4k writes
>
> - stopped qemu
> - cp -ra firefly_0.80.8/usr/lib/librados.so.2.0.0 /usr/lib/
> - cp -ra firefly_0.80.8/usr/lib/librbd.so.1.0.0 /usr/lib/
> - start qemu
>
> same fio, same qemu, same vm, same host, same ceph dumpling storage,
> different librados / librbd: 16k iop/s for random 4k writes
>
> What's wrong with librbd / librados2 since firefly?
Hi Stephen,
Just off the top of my head, some questions to investigate:
What happens to single op latencies?
Does enabling/disabling RBD cache have any effect?
How's CPU usage? (Does perf report show anything useful?)
Can you get trace data?
Mark
>
> Greets,
> Stefan
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: speed decrease since firefly,giant,hammer the 2nd try
2015-02-10 19:05 ` Gregory Farnum
@ 2015-02-10 19:12 ` Stefan Priebe
0 siblings, 0 replies; 32+ messages in thread
From: Stefan Priebe @ 2015-02-10 19:12 UTC (permalink / raw)
To: Gregory Farnum; +Cc: ceph-devel
Am 10.02.2015 um 20:05 schrieb Gregory Farnum:
> On Tue, Feb 10, 2015 at 10:55 AM, Stefan Priebe <s.priebe@profihost.ag> wrote:
>> Hello,
>>
>> last year in june i already reported this but there was no real result.
>> (http://lists.ceph.com/pipermail/ceph-users-ceph.com/2014-July/041070.html)
>>
>> I then had the hope that this will be fixed itself when hammer is released.
>> Now i tried hammer an the results are bad as before.
>>
>> Since firefly librbd1 / librados2 are 20% slower for 4k random iop/s than
>> dumpling - this is also the reason why i still stick to dumpling.
>>
>> I've now modified my test again to be a bit more clear.
>>
>> Ceph cluster itself completely dumpling.
>>
>> librbd1 / librados from dumpling (fio inside qemu): 23k iop/s for random 4k
>> writes
>>
>> - stopped qemu
>> - cp -ra firefly_0.80.8/usr/lib/librados.so.2.0.0 /usr/lib/
>> - cp -ra firefly_0.80.8/usr/lib/librbd.so.1.0.0 /usr/lib/
>> - start qemu
>>
>> same fio, same qemu, same vm, same host, same ceph dumpling storage,
>> different librados / librbd: 16k iop/s for random 4k writes
>>
>> What's wrong with librbd / librados2 since firefly?
>
> We're all going to have the same questions now as we did last time,
> about what the cluster looks like, what the perfcounters are reporting
> on both versions of librados, etc.
I try to answer all your questions - not sue how easy this is.
6 Nodes each with:
- Single Intel E5-1650 v3
- 48GB RAM
- 4x 800GB Samsung SSD
- 2x 10Gbit/s bonded storage network
- Client side
- Dual Xeon E5
- 256GB RAM
- 2x 10Gbit/s bonded storage network
Regarding perf counters - i'm willing to make tests. Just tell me how.
> Also, please give us the results from Giant rather than Firefly, for
> the reasons I mentioned previously.
As giant is not a long term release and we have a support contract it's
not an option to me. Even tough i tried hammer git master three days
ago. Same results.
Stefan
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: speed decrease since firefly,giant,hammer the 2nd try
2015-02-10 19:10 ` Mark Nelson
@ 2015-02-10 19:13 ` Stefan Priebe
2015-02-10 19:40 ` Mark Nelson
0 siblings, 1 reply; 32+ messages in thread
From: Stefan Priebe @ 2015-02-10 19:13 UTC (permalink / raw)
To: Mark Nelson, ceph-devel
Am 10.02.2015 um 20:10 schrieb Mark Nelson:
>
>
> On 02/10/2015 12:55 PM, Stefan Priebe wrote:
>> Hello,
>>
>> last year in june i already reported this but there was no real result.
>> (http://lists.ceph.com/pipermail/ceph-users-ceph.com/2014-July/041070.html)
>>
>>
>> I then had the hope that this will be fixed itself when hammer is
>> released. Now i tried hammer an the results are bad as before.
>>
>> Since firefly librbd1 / librados2 are 20% slower for 4k random iop/s
>> than dumpling - this is also the reason why i still stick to dumpling.
>>
>> I've now modified my test again to be a bit more clear.
>>
>> Ceph cluster itself completely dumpling.
>>
>> librbd1 / librados from dumpling (fio inside qemu): 23k iop/s for random
>> 4k writes
>>
>> - stopped qemu
>> - cp -ra firefly_0.80.8/usr/lib/librados.so.2.0.0 /usr/lib/
>> - cp -ra firefly_0.80.8/usr/lib/librbd.so.1.0.0 /usr/lib/
>> - start qemu
>>
>> same fio, same qemu, same vm, same host, same ceph dumpling storage,
>> different librados / librbd: 16k iop/s for random 4k writes
>>
>> What's wrong with librbd / librados2 since firefly?
>
> Hi Stephen,
>
> Just off the top of my head, some questions to investigate:
>
> What happens to single op latencies?
How to test this?
> Does enabling/disabling RBD cache have any effect?
I've it enabled on both through qemu write back setting.
> How's CPU usage? (Does perf report show anything useful?)
> Can you get trace data?
I'm not familiar with trace or perf - what should do exactly?
Stefan
> Mark
>
>>
>> Greets,
>> Stefan
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: speed decrease since firefly,giant,hammer the 2nd try
2015-02-10 19:13 ` Stefan Priebe
@ 2015-02-10 19:40 ` Mark Nelson
2015-02-10 20:24 ` Stefan Priebe
0 siblings, 1 reply; 32+ messages in thread
From: Mark Nelson @ 2015-02-10 19:40 UTC (permalink / raw)
To: Stefan Priebe, ceph-devel
On 02/10/2015 01:13 PM, Stefan Priebe wrote:
>
> Am 10.02.2015 um 20:10 schrieb Mark Nelson:
>>
>>
>> On 02/10/2015 12:55 PM, Stefan Priebe wrote:
>>> Hello,
>>>
>>> last year in june i already reported this but there was no real result.
>>> (http://lists.ceph.com/pipermail/ceph-users-ceph.com/2014-July/041070.html)
>>>
>>>
>>>
>>> I then had the hope that this will be fixed itself when hammer is
>>> released. Now i tried hammer an the results are bad as before.
>>>
>>> Since firefly librbd1 / librados2 are 20% slower for 4k random iop/s
>>> than dumpling - this is also the reason why i still stick to dumpling.
>>>
>>> I've now modified my test again to be a bit more clear.
>>>
>>> Ceph cluster itself completely dumpling.
>>>
>>> librbd1 / librados from dumpling (fio inside qemu): 23k iop/s for random
>>> 4k writes
>>>
>>> - stopped qemu
>>> - cp -ra firefly_0.80.8/usr/lib/librados.so.2.0.0 /usr/lib/
>>> - cp -ra firefly_0.80.8/usr/lib/librbd.so.1.0.0 /usr/lib/
>>> - start qemu
>>>
>>> same fio, same qemu, same vm, same host, same ceph dumpling storage,
>>> different librados / librbd: 16k iop/s for random 4k writes
>>>
>>> What's wrong with librbd / librados2 since firefly?
>>
>> Hi Stephen,
>>
>> Just off the top of my head, some questions to investigate:
>>
>> What happens to single op latencies?
>
> How to test this?
try your random 4k write test using libaio, direct IO, and iodepth=1.
Actually it would be interesting to know how it is with higher IO depths
as well (I assume this is what you are doing now?) Basically I want to
know if single-op latency changes and whether or not it gets hidden or
exaggerated with lots of concurrent IO.
>
>> Does enabling/disabling RBD cache have any effect?
>
> I've it enabled on both through qemu write back setting.
It'd be great if you could do the above test both with WB RBD cache and
with it turned off.
>
>> How's CPU usage? (Does perf report show anything useful?)
>> Can you get trace data?
>
> I'm not familiar with trace or perf - what should do exactly?
you may need extra packages. Basically on VM host, during the test with
each library you'd do:
sudo perf record -a -g dwarf -F 99
(ctrl+c after a while)
sudo perf report --stdio > foo.txt
if you are on a kernel that doesn't have libunwind support:
sudo perf record -a -g
(ctrl+c after a while)
sudo perf report --stdio > foo.txt
Then look and see what's different. This may not catch anything though.
You should also try Greg's suggestion looking at the performance
counters to see if any interesting differences show up between the runs.
>
> Stefan
>
>> Mark
>>
>>>
>>> Greets,
>>> Stefan
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: speed decrease since firefly,giant,hammer the 2nd try
2015-02-10 19:40 ` Mark Nelson
@ 2015-02-10 20:24 ` Stefan Priebe
2015-02-10 20:36 ` Mark Nelson
0 siblings, 1 reply; 32+ messages in thread
From: Stefan Priebe @ 2015-02-10 20:24 UTC (permalink / raw)
To: Mark Nelson, ceph-devel
Am 10.02.2015 um 20:40 schrieb Mark Nelson:
> On 02/10/2015 01:13 PM, Stefan Priebe wrote:
>> Am 10.02.2015 um 20:10 schrieb Mark Nelson:
>>> On 02/10/2015 12:55 PM, Stefan Priebe wrote:
>>>> Hello,
>>>>
>>>> last year in june i already reported this but there was no real result.
>>>> (http://lists.ceph.com/pipermail/ceph-users-ceph.com/2014-July/041070.html)
>>>>
>>>> I then had the hope that this will be fixed itself when hammer is
>>>> released. Now i tried hammer an the results are bad as before.
>>>>
>>>> Since firefly librbd1 / librados2 are 20% slower for 4k random iop/s
>>>> than dumpling - this is also the reason why i still stick to dumpling.
>>>>
>>>> I've now modified my test again to be a bit more clear.
>>>>
>>>> Ceph cluster itself completely dumpling.
>>>>
>>>> librbd1 / librados from dumpling (fio inside qemu): 23k iop/s for
>>>> random
>>>> 4k writes
>>>>
>>>> - stopped qemu
>>>> - cp -ra firefly_0.80.8/usr/lib/librados.so.2.0.0 /usr/lib/
>>>> - cp -ra firefly_0.80.8/usr/lib/librbd.so.1.0.0 /usr/lib/
>>>> - start qemu
>>>>
>>>> same fio, same qemu, same vm, same host, same ceph dumpling storage,
>>>> different librados / librbd: 16k iop/s for random 4k writes
>>>>
>>>> What's wrong with librbd / librados2 since firefly?
>>>
>>> Hi Stephen,
>>>
>>> Just off the top of my head, some questions to investigate:
>>>
>>> What happens to single op latencies?
>>
>> How to test this?
>
> try your random 4k write test using libaio, direct IO, and iodepth=1.
> Actually it would be interesting to know how it is with higher IO depths
> as well (I assume this is what you are doing now?) Basically I want to
> know if single-op latency changes and whether or not it gets hidden or
> exaggerated with lots of concurrent IO.
dumpling:
ioengine=libaio and iodepth=32 with 32 threads:
Jobs: 32 (f=32): [wwwwwwwwwwwwwwwwwwwwwwwwwwwwwwww] [100.0% done]
[0K/85224K /s] [0 /21.4K iops] [eta 00m:00s]
ioengine=libaio and iodepth=1 with 32 threads:
Jobs: 32 (f=32): [wwwwwwwwwwwwwwwwwwwwwwwwwwwwwwww] [100.0% done]
[0K/79064K /s] [0 /19.8K iops] [eta 00m:00s]
firefly:
ioengine=libaio and iodepth=32 with 32 threads:
Jobs: 32 (f=32): [wwwwwwwwwwwwwwwwwwwwwwwwwwwwwwww] [100.0% done]
[0K/55781K /s] [0 /15.4K iops] [eta 00m:00s]
ioengine=libaio and iodepth=1 with 32 threads:
Jobs: 32 (f=32): [wwwwwwwwwwwwwwwwwwwwwwwwwwwwwwww] [100.0% done]
[0K/46055K /s] [0 /11.6K iops] [eta 00m:00s]
>>> Does enabling/disabling RBD cache have any effect?
>>
>> I've it enabled on both through qemu write back setting.
>
> It'd be great if you could do the above test both with WB RBD cache and
> with it turned off.
Test with cache off:
dumpling:
ioengine=libaio and iodepth=32 with 32 threads:
Jobs: 32 (f=32): [wwwwwwwwwwwwwwwwwwwwwwwwwwwwwwww] [100.0% done]
[0K/85111K /s] [0 /21.3K iops] [eta 00m:00s]
ioengine=libaio and iodepth=1 with 32 threads:
Jobs: 32 (f=32): [wwwwwwwwwwwwwwwwwwwwwwwwwwwwwwww] [100.0% done]
[0K/88984K /s] [0 /22.3K iops] [eta 00m:00s]
firefly:
ioengine=libaio and iodepth=32 with 32 threads:
Jobs: 32 (f=32): [wwwwwwwwwwwwwwwwwwwwwwwwwwwwwwww] [100.0% done]
[0K/46479K /s] [0 /11.7K iops] [eta 00m:00s]
ioengine=libaio and iodepth=1 with 32 threads:
Jobs: 32 (f=32): [wwwwwwwwwwwwwwwwwwwwwwwwwwwwwwww] [100.0% done]
[0K/46019K /s] [0 /11.6K iops] [eta 00m:00s]
>>> How's CPU usage? (Does perf report show anything useful?)
>>> Can you get trace data?
>>
>> I'm not familiar with trace or perf - what should do exactly?
>
> you may need extra packages. Basically on VM host, during the test with
> each library you'd do:
>
> sudo perf record -a -g dwarf -F 99
> (ctrl+c after a while)
> sudo perf report --stdio > foo.txt
>
> if you are on a kernel that doesn't have libunwind support:
>
> sudo perf record -a -g
> (ctrl+c after a while)
> sudo perf report --stdio > foo.txt
>
> Then look and see what's different. This may not catch anything though.
Don't have unwind.
Output is only full of hex values.
Stefan
> You should also try Greg's suggestion looking at the performance
> counters to see if any interesting differences show up between the runs.
Where / how to check?
Stefan
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: speed decrease since firefly,giant,hammer the 2nd try
2015-02-10 20:24 ` Stefan Priebe
@ 2015-02-10 20:36 ` Mark Nelson
2015-02-10 21:11 ` Stefan Priebe
0 siblings, 1 reply; 32+ messages in thread
From: Mark Nelson @ 2015-02-10 20:36 UTC (permalink / raw)
To: Stefan Priebe, ceph-devel
On 02/10/2015 02:24 PM, Stefan Priebe wrote:
> Am 10.02.2015 um 20:40 schrieb Mark Nelson:
>> On 02/10/2015 01:13 PM, Stefan Priebe wrote:
>>> Am 10.02.2015 um 20:10 schrieb Mark Nelson:
>>>> On 02/10/2015 12:55 PM, Stefan Priebe wrote:
>>>>> Hello,
>>>>>
>>>>> last year in june i already reported this but there was no real
>>>>> result.
>>>>> (http://lists.ceph.com/pipermail/ceph-users-ceph.com/2014-July/041070.html)
>>>>>
>>>>>
>>>>> I then had the hope that this will be fixed itself when hammer is
>>>>> released. Now i tried hammer an the results are bad as before.
>>>>>
>>>>> Since firefly librbd1 / librados2 are 20% slower for 4k random iop/s
>>>>> than dumpling - this is also the reason why i still stick to dumpling.
>>>>>
>>>>> I've now modified my test again to be a bit more clear.
>>>>>
>>>>> Ceph cluster itself completely dumpling.
>>>>>
>>>>> librbd1 / librados from dumpling (fio inside qemu): 23k iop/s for
>>>>> random
>>>>> 4k writes
>>>>>
>>>>> - stopped qemu
>>>>> - cp -ra firefly_0.80.8/usr/lib/librados.so.2.0.0 /usr/lib/
>>>>> - cp -ra firefly_0.80.8/usr/lib/librbd.so.1.0.0 /usr/lib/
>>>>> - start qemu
>>>>>
>>>>> same fio, same qemu, same vm, same host, same ceph dumpling storage,
>>>>> different librados / librbd: 16k iop/s for random 4k writes
>>>>>
>>>>> What's wrong with librbd / librados2 since firefly?
>>>>
>>>> Hi Stephen,
>>>>
>>>> Just off the top of my head, some questions to investigate:
>>>>
>>>> What happens to single op latencies?
>>>
>>> How to test this?
>>
>> try your random 4k write test using libaio, direct IO, and iodepth=1.
>> Actually it would be interesting to know how it is with higher IO depths
>> as well (I assume this is what you are doing now?) Basically I want to
>> know if single-op latency changes and whether or not it gets hidden or
>> exaggerated with lots of concurrent IO.
>
> dumpling:
> ioengine=libaio and iodepth=32 with 32 threads:
>
> Jobs: 32 (f=32): [wwwwwwwwwwwwwwwwwwwwwwwwwwwwwwww] [100.0% done]
> [0K/85224K /s] [0 /21.4K iops] [eta 00m:00s]
>
> ioengine=libaio and iodepth=1 with 32 threads:
>
> Jobs: 32 (f=32): [wwwwwwwwwwwwwwwwwwwwwwwwwwwwwwww] [100.0% done]
> [0K/79064K /s] [0 /19.8K iops] [eta 00m:00s]
>
> firefly:
> ioengine=libaio and iodepth=32 with 32 threads:
>
> Jobs: 32 (f=32): [wwwwwwwwwwwwwwwwwwwwwwwwwwwwwwww] [100.0% done]
> [0K/55781K /s] [0 /15.4K iops] [eta 00m:00s]
>
> ioengine=libaio and iodepth=1 with 32 threads:
>
> Jobs: 32 (f=32): [wwwwwwwwwwwwwwwwwwwwwwwwwwwwwwww] [100.0% done]
> [0K/46055K /s] [0 /11.6K iops] [eta 00m:00s]
>
Sorry, please do this with only 1 thread. If you can include the
latency results too that would be great.
>>>> Does enabling/disabling RBD cache have any effect?
>>>
>>> I've it enabled on both through qemu write back setting.
>>
>> It'd be great if you could do the above test both with WB RBD cache and
>> with it turned off.
>
> Test with cache off:
>
> dumpling:
> ioengine=libaio and iodepth=32 with 32 threads:
>
> Jobs: 32 (f=32): [wwwwwwwwwwwwwwwwwwwwwwwwwwwwwwww] [100.0% done]
> [0K/85111K /s] [0 /21.3K iops] [eta 00m:00s]
>
> ioengine=libaio and iodepth=1 with 32 threads:
>
> Jobs: 32 (f=32): [wwwwwwwwwwwwwwwwwwwwwwwwwwwwwwww] [100.0% done]
> [0K/88984K /s] [0 /22.3K iops] [eta 00m:00s]
>
> firefly:
> ioengine=libaio and iodepth=32 with 32 threads:
>
> Jobs: 32 (f=32): [wwwwwwwwwwwwwwwwwwwwwwwwwwwwwwww] [100.0% done]
> [0K/46479K /s] [0 /11.7K iops] [eta 00m:00s]
>
> ioengine=libaio and iodepth=1 with 32 threads:
>
> Jobs: 32 (f=32): [wwwwwwwwwwwwwwwwwwwwwwwwwwwwwwww] [100.0% done]
> [0K/46019K /s] [0 /11.6K iops] [eta 00m:00s]
So at least so far it appears that this may not be RBD cache related.
>
>>>> How's CPU usage? (Does perf report show anything useful?)
>>>> Can you get trace data?
>>>
>>> I'm not familiar with trace or perf - what should do exactly?
>>
>> you may need extra packages. Basically on VM host, during the test with
>> each library you'd do:
>>
>> sudo perf record -a -g dwarf -F 99
>> (ctrl+c after a while)
>> sudo perf report --stdio > foo.txt
>>
>> if you are on a kernel that doesn't have libunwind support:
>>
>> sudo perf record -a -g
>> (ctrl+c after a while)
>> sudo perf report --stdio > foo.txt
>>
>> Then look and see what's different. This may not catch anything though.
>
> Don't have unwind.
Too bad, oh well.
> Output is only full of hex values.
You'll at least need the correct debug symbols either compiled into the
library or available wherever your OS puts them. Sometimes the perf
cache needs to be manually edited so they point to the right place, it's
super annoying.
>
> Stefan
>
>> You should also try Greg's suggestion looking at the performance
>> counters to see if any interesting differences show up between the runs.
>
> Where / how to check?
>
> Stefan
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: speed decrease since firefly,giant,hammer the 2nd try
2015-02-10 20:36 ` Mark Nelson
@ 2015-02-10 21:11 ` Stefan Priebe
2015-02-10 21:38 ` Mark Nelson
0 siblings, 1 reply; 32+ messages in thread
From: Stefan Priebe @ 2015-02-10 21:11 UTC (permalink / raw)
To: Mark Nelson, ceph-devel
Am 10.02.2015 um 21:36 schrieb Mark Nelson:
>
>
> On 02/10/2015 02:24 PM, Stefan Priebe wrote:
>> Am 10.02.2015 um 20:40 schrieb Mark Nelson:
>>> On 02/10/2015 01:13 PM, Stefan Priebe wrote:
>>>> Am 10.02.2015 um 20:10 schrieb Mark Nelson:
>>>>> On 02/10/2015 12:55 PM, Stefan Priebe wrote:
>>>>>> Hello,
>>>>>>
>>>>>> last year in june i already reported this but there was no real
>>>>>> result.
>>>>>> (http://lists.ceph.com/pipermail/ceph-users-ceph.com/2014-July/041070.html)
>>>>>>
>>>>>>
>>>>>>
>>>>>> I then had the hope that this will be fixed itself when hammer is
>>>>>> released. Now i tried hammer an the results are bad as before.
>>>>>>
>>>>>> Since firefly librbd1 / librados2 are 20% slower for 4k random iop/s
>>>>>> than dumpling - this is also the reason why i still stick to
>>>>>> dumpling.
>>>>>>
>>>>>> I've now modified my test again to be a bit more clear.
>>>>>>
>>>>>> Ceph cluster itself completely dumpling.
>>>>>>
>>>>>> librbd1 / librados from dumpling (fio inside qemu): 23k iop/s for
>>>>>> random
>>>>>> 4k writes
>>>>>>
>>>>>> - stopped qemu
>>>>>> - cp -ra firefly_0.80.8/usr/lib/librados.so.2.0.0 /usr/lib/
>>>>>> - cp -ra firefly_0.80.8/usr/lib/librbd.so.1.0.0 /usr/lib/
>>>>>> - start qemu
>>>>>>
>>>>>> same fio, same qemu, same vm, same host, same ceph dumpling storage,
>>>>>> different librados / librbd: 16k iop/s for random 4k writes
>>>>>>
>>>>>> What's wrong with librbd / librados2 since firefly?
>>>>>
>>>>> Hi Stephen,
>>>>>
>>>>> Just off the top of my head, some questions to investigate:
>>>>>
>>>>> What happens to single op latencies?
>>>>
>>>> How to test this?
>>>
>>> try your random 4k write test using libaio, direct IO, and iodepth=1.
>>> Actually it would be interesting to know how it is with higher IO depths
>>> as well (I assume this is what you are doing now?) Basically I want to
>>> know if single-op latency changes and whether or not it gets hidden or
>>> exaggerated with lots of concurrent IO.
>>
>> dumpling:
>> ioengine=libaio and iodepth=32 with 32 threads:
>>
>> Jobs: 32 (f=32): [wwwwwwwwwwwwwwwwwwwwwwwwwwwwwwww] [100.0% done]
>> [0K/85224K /s] [0 /21.4K iops] [eta 00m:00s]
>>
>> ioengine=libaio and iodepth=1 with 32 threads:
>>
>> Jobs: 32 (f=32): [wwwwwwwwwwwwwwwwwwwwwwwwwwwwwwww] [100.0% done]
>> [0K/79064K /s] [0 /19.8K iops] [eta 00m:00s]
>>
>> firefly:
>> ioengine=libaio and iodepth=32 with 32 threads:
>>
>> Jobs: 32 (f=32): [wwwwwwwwwwwwwwwwwwwwwwwwwwwwwwww] [100.0% done]
>> [0K/55781K /s] [0 /15.4K iops] [eta 00m:00s]
>>
>> ioengine=libaio and iodepth=1 with 32 threads:
>>
>> Jobs: 32 (f=32): [wwwwwwwwwwwwwwwwwwwwwwwwwwwwwwww] [100.0% done]
>> [0K/46055K /s] [0 /11.6K iops] [eta 00m:00s]
>>
>
> Sorry, please do this with only 1 thread. If you can include the
> latency results too that would be great.
Sorry here again.
Cache on:
dumpling:
file1: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=1
2.0.8
Starting 1 thread
Jobs: 1 (f=1): [w] [100.0% done] [0K/42892K /s] [0 /10.8K iops] [eta
00m:00s]
file1: (groupid=0, jobs=1): err= 0: pid=3203
write: io=1273.1MB, bw=43483KB/s, iops=10870 , runt= 30001msec
slat (usec): min=5 , max=183 , avg= 8.99, stdev= 1.78
clat (usec): min=0 , max=6378 , avg=81.15, stdev=44.09
lat (usec): min=59 , max=6390 , avg=90.35, stdev=44.22
clat percentiles (usec):
| 1.00th=[ 59], 5.00th=[ 62], 10.00th=[ 64], 20.00th=[ 66],
| 30.00th=[ 69], 40.00th=[ 71], 50.00th=[ 74], 60.00th=[ 80],
| 70.00th=[ 87], 80.00th=[ 95], 90.00th=[ 105], 95.00th=[ 114],
| 99.00th=[ 135], 99.50th=[ 145], 99.90th=[ 179], 99.95th=[ 237],
| 99.99th=[ 2320]
bw (KB/s) : min=36176, max=46816, per=99.96%, avg=43465.49,
stdev=2169.33
lat (usec) : 2=0.01%, 4=0.01%, 20=0.01%, 50=0.01%, 100=85.24%
lat (usec) : 250=14.71%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%
cpu : usr=2.95%, sys=12.29%, ctx=329519, majf=0, minf=133
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
>=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
issued : total=r=0/w=326130/d=0, short=r=0/w=0/d=0
Run status group 0 (all jobs):
WRITE: io=1273.1MB, aggrb=43482KB/s, minb=43482KB/s, maxb=43482KB/s,
mint=30001msec, maxt=30001msec
Disk stats (read/write):
sdb: ios=166/325241, merge=0/0, ticks=8/24624, in_queue=24492,
util=81.64%
firefly:
file1: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=1
2.0.8
Starting 1 thread
Jobs: 1 (f=1): [w] [100.0% done] [0K/44588K /s] [0 /11.2K iops] [eta
00m:00s]
file1: (groupid=0, jobs=1): err= 0: pid=2904
write: io=1212.1MB, bw=41401KB/s, iops=10350 , runt= 30001msec
slat (usec): min=5 , max=464 , avg= 8.95, stdev= 2.34
clat (usec): min=0 , max=4410 , avg=85.81, stdev=41.82
lat (usec): min=59 , max=4418 , avg=94.96, stdev=41.97
clat percentiles (usec):
| 1.00th=[ 59], 5.00th=[ 63], 10.00th=[ 65], 20.00th=[ 68],
| 30.00th=[ 72], 40.00th=[ 76], 50.00th=[ 80], 60.00th=[ 85],
| 70.00th=[ 94], 80.00th=[ 102], 90.00th=[ 112], 95.00th=[ 122],
| 99.00th=[ 145], 99.50th=[ 155], 99.90th=[ 189], 99.95th=[ 239],
| 99.99th=[ 2192]
bw (KB/s) : min=34352, max=45992, per=99.91%, avg=41363.80,
stdev=3314.63
lat (usec) : 2=0.01%, 50=0.01%, 100=77.26%, 250=22.69%, 500=0.02%
lat (usec) : 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%
cpu : usr=2.92%, sys=11.63%, ctx=313889, majf=0, minf=133
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
>=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
issued : total=r=0/w=310521/d=0, short=r=0/w=0/d=0
Run status group 0 (all jobs):
WRITE: io=1212.1MB, aggrb=41401KB/s, minb=41401KB/s, maxb=41401KB/s,
mint=30001msec, maxt=30001msec
Disk stats (read/write):
sdb: ios=166/309726, merge=0/0, ticks=16/24772, in_queue=24688,
util=82.30%
sorry the old cache off results were wrong here are new ones also with
only one thread:
dumpling:
Jobs: 1 (f=1): [w] [100.0% done] [0K/2140K /s] [0 /535 iops] [eta 00m:00s]
file1: (groupid=0, jobs=1): err= 0: pid=2906
write: io=64424KB, bw=2147.5KB/s, iops=536 , runt= 30001msec
slat (usec): min=14 , max=848 , avg=30.27, stdev= 8.70
clat (usec): min=914 , max=74110 , avg=1826.25, stdev=604.49
lat (usec): min=1395 , max=74138 , avg=1857.26, stdev=604.48
clat percentiles (usec):
| 1.00th=[ 1592], 5.00th=[ 1656], 10.00th=[ 1688], 20.00th=[ 1736],
| 30.00th=[ 1768], 40.00th=[ 1784], 50.00th=[ 1816], 60.00th=[ 1832],
| 70.00th=[ 1864], 80.00th=[ 1896], 90.00th=[ 1928], 95.00th=[ 1960],
| 99.00th=[ 2064], 99.50th=[ 2384], 99.90th=[ 5024], 99.95th=[ 5600],
| 99.99th=[11712]
bw (KB/s) : min= 1768, max= 2240, per=100.00%, avg=2149.53,
stdev=56.30
lat (usec) : 1000=0.01%
lat (msec) : 2=97.67%, 4=2.11%, 10=0.20%, 20=0.01%, 100=0.01%
cpu : usr=0.72%, sys=2.12%, ctx=19289, majf=0, minf=133
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
>=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
issued : total=r=0/w=16106/d=0, short=r=0/w=0/d=0
Run status group 0 (all jobs):
WRITE: io=64424KB, aggrb=2147KB/s, minb=2147KB/s, maxb=2147KB/s,
mint=30001msec, maxt=30001msec
Disk stats (read/write):
sdb: ios=168/16067, merge=0/0, ticks=164/29004, in_queue=29144,
util=96.93%
firefly:
file1: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=1
2.0.8
Starting 1 thread
Jobs: 1 (f=1): [w] [100.0% done] [0K/2141K /s] [0 /535 iops] [eta 00m:00s]
file1: (groupid=0, jobs=1): err= 0: pid=3034
write: io=63100KB, bw=2103.3KB/s, iops=525 , runt= 30001msec
slat (usec): min=17 , max=388 , avg=33.26, stdev= 6.55
clat (usec): min=1392 , max=54698 , avg=1862.25, stdev=478.79
lat (usec): min=1507 , max=54716 , avg=1896.28, stdev=478.69
clat percentiles (usec):
| 1.00th=[ 1624], 5.00th=[ 1688], 10.00th=[ 1720], 20.00th=[ 1768],
| 30.00th=[ 1800], 40.00th=[ 1832], 50.00th=[ 1848], 60.00th=[ 1880],
| 70.00th=[ 1896], 80.00th=[ 1928], 90.00th=[ 1960], 95.00th=[ 1992],
| 99.00th=[ 2096], 99.50th=[ 2512], 99.90th=[ 5344], 99.95th=[ 5984],
| 99.99th=[11840]
bw (KB/s) : min= 1760, max= 2160, per=100.00%, avg=2104.68,
stdev=49.36
lat (msec) : 2=95.18%, 4=4.57%, 10=0.23%, 20=0.01%, 100=0.01%
cpu : usr=0.88%, sys=2.09%, ctx=18938, majf=0, minf=133
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
>=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
issued : total=r=0/w=15775/d=0, short=r=0/w=0/d=0
Run status group 0 (all jobs):
WRITE: io=63100KB, aggrb=2103KB/s, minb=2103KB/s, maxb=2103KB/s,
mint=30001msec, maxt=30001msec
Disk stats (read/write):
sdb: ios=170/15740, merge=0/0, ticks=268/28856, in_queue=29116,
util=96.45%
>>>>> How's CPU usage? (Does perf report show anything useful?)
>>>>> Can you get trace data?
>>>>
>>>> I'm not familiar with trace or perf - what should do exactly?
>>>
>>> you may need extra packages. Basically on VM host, during the test with
>>> each library you'd do:
>>>
>>> sudo perf record -a -g dwarf -F 99
>>> (ctrl+c after a while)
>>> sudo perf report --stdio > foo.txt
>>>
>>> if you are on a kernel that doesn't have libunwind support:
>>>
>>> sudo perf record -a -g
>>> (ctrl+c after a while)
>>> sudo perf report --stdio > foo.txt
>>>
>>> Then look and see what's different. This may not catch anything though.
>>
>> Don't have unwind.
>
> Too bad, oh well.
>
>> Output is only full of hex values.
>
> You'll at least need the correct debug symbols either compiled into the
> library or available wherever your OS puts them. Sometimes the perf
> cache needs to be manually edited so they point to the right place, it's
> super annoying.
mhm i installed librbd1-dbg and librados2-dbg - but the output still
looks useless to me. Should i upload it somewhere?
Stefan
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: speed decrease since firefly,giant,hammer the 2nd try
2015-02-10 21:11 ` Stefan Priebe
@ 2015-02-10 21:38 ` Mark Nelson
2015-02-10 22:18 ` Stefan Priebe
[not found] ` <764258391.358931.1423634637318.JavaMail.zimbra@oxygem.tv>
0 siblings, 2 replies; 32+ messages in thread
From: Mark Nelson @ 2015-02-10 21:38 UTC (permalink / raw)
To: Stefan Priebe, ceph-devel
On 02/10/2015 03:11 PM, Stefan Priebe wrote:
>
> mhm i installed librbd1-dbg and librados2-dbg - but the output still
> looks useless to me. Should i upload it somewhere?
Meh, if it's all just symbols it's probably not that helpful.
I've summarized your results here:
1 concurrent 4k write (libaio, direct=1, iodepth=1)
IOPS Latency
wb on wb off wb on wb off
dumpling 10870 536 ~100us ~2ms
firefly 10350 525 ~100us ~2ms
So in single op tests dumpling and firefly are far closer. Now let's
see each of these cases with iodepth=32 (still 1 thread for now).
Mark
>
> Stefan
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: speed decrease since firefly,giant,hammer the 2nd try
2015-02-10 21:38 ` Mark Nelson
@ 2015-02-10 22:18 ` Stefan Priebe
2015-02-11 4:45 ` Mark Nelson
[not found] ` <764258391.358931.1423634637318.JavaMail.zimbra@oxygem.tv>
1 sibling, 1 reply; 32+ messages in thread
From: Stefan Priebe @ 2015-02-10 22:18 UTC (permalink / raw)
To: Mark Nelson, ceph-devel
Am 10.02.2015 um 22:38 schrieb Mark Nelson:
> On 02/10/2015 03:11 PM, Stefan Priebe wrote:
>>
>> mhm i installed librbd1-dbg and librados2-dbg - but the output still
>> looks useless to me. Should i upload it somewhere?
>
> Meh, if it's all just symbols it's probably not that helpful.
>
> I've summarized your results here:
>
> 1 concurrent 4k write (libaio, direct=1, iodepth=1)
>
> IOPS Latency
> wb on wb off wb on wb off
> dumpling 10870 536 ~100us ~2ms
> firefly 10350 525 ~100us ~2ms
>
> So in single op tests dumpling and firefly are far closer. Now let's
> see each of these cases with iodepth=32 (still 1 thread for now).
dumpling:
file1: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=32
2.0.8
Starting 1 thread
Jobs: 1 (f=1): [w] [100.0% done] [0K/72812K /s] [0 /18.3K iops] [eta
00m:00s]
file1: (groupid=0, jobs=1): err= 0: pid=3011
write: io=2060.6MB, bw=70329KB/s, iops=17582 , runt= 30001msec
slat (usec): min=1 , max=3517 , avg= 3.42, stdev= 7.30
clat (usec): min=93 , max=7475 , avg=1815.72, stdev=233.43
lat (usec): min=219 , max=7477 , avg=1819.27, stdev=233.52
clat percentiles (usec):
| 1.00th=[ 1480], 5.00th=[ 1576], 10.00th=[ 1608], 20.00th=[ 1672],
| 30.00th=[ 1704], 40.00th=[ 1752], 50.00th=[ 1800], 60.00th=[ 1832],
| 70.00th=[ 1896], 80.00th=[ 1960], 90.00th=[ 2064], 95.00th=[ 2128],
| 99.00th=[ 2352], 99.50th=[ 2448], 99.90th=[ 4704], 99.95th=[ 5344],
| 99.99th=[ 7072]
bw (KB/s) : min=59696, max=77840, per=100.00%, avg=70351.27,
stdev=4783.25
lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.53%
lat (msec) : 2=85.02%, 4=14.31%, 10=0.13%
cpu : usr=1.96%, sys=6.71%, ctx=22791, majf=0, minf=133
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%,
>=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
>=64=0.0%
issued : total=r=0/w=527487/d=0, short=r=0/w=0/d=0
Run status group 0 (all jobs):
WRITE: io=2060.6MB, aggrb=70329KB/s, minb=70329KB/s, maxb=70329KB/s,
mint=30001msec, maxt=30001msec
Disk stats (read/write):
sdb: ios=166/526079, merge=0/0, ticks=24/890120, in_queue=890064,
util=98.73%
firefly:
file1: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=32
2.0.8
Starting 1 thread
Jobs: 1 (f=1): [w] [100.0% done] [0K/69096K /s] [0 /17.3K iops] [eta
00m:00s]
file1: (groupid=0, jobs=1): err= 0: pid=2982
write: io=1784.9MB, bw=60918KB/s, iops=15229 , runt= 30002msec
slat (usec): min=1 , max=1389 , avg= 3.43, stdev= 5.32
clat (usec): min=117 , max=8235 , avg=2096.88, stdev=396.30
lat (usec): min=540 , max=8258 , avg=2100.43, stdev=396.61
clat percentiles (usec):
| 1.00th=[ 1608], 5.00th=[ 1720], 10.00th=[ 1768], 20.00th=[ 1832],
| 30.00th=[ 1896], 40.00th=[ 1944], 50.00th=[ 2008], 60.00th=[ 2064],
| 70.00th=[ 2160], 80.00th=[ 2256], 90.00th=[ 2512], 95.00th=[ 2896],
| 99.00th=[ 3600], 99.50th=[ 3792], 99.90th=[ 5088], 99.95th=[ 6304],
| 99.99th=[ 6752]
bw (KB/s) : min=36717, max=73712, per=99.94%, avg=60879.92,
stdev=8302.27
lat (usec) : 250=0.01%, 750=0.01%
lat (msec) : 2=48.56%, 4=51.18%, 10=0.26%
cpu : usr=2.03%, sys=5.48%, ctx=20440, majf=0, minf=133
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%,
>=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
>=64=0.0%
issued : total=r=0/w=456918/d=0, short=r=0/w=0/d=0
Run status group 0 (all jobs):
WRITE: io=1784.9MB, aggrb=60918KB/s, minb=60918KB/s, maxb=60918KB/s,
mint=30002msec, maxt=30002msec
Disk stats (read/write):
sdb: ios=166/455574, merge=0/0, ticks=12/897748, in_queue=897696,
util=98.96%
Stefan
> Mark
>
>>
>> Stefan
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: speed decrease since firefly,giant,hammer the 2nd try
2015-02-10 22:18 ` Stefan Priebe
@ 2015-02-11 4:45 ` Mark Nelson
2015-02-11 5:42 ` Stefan Priebe
0 siblings, 1 reply; 32+ messages in thread
From: Mark Nelson @ 2015-02-11 4:45 UTC (permalink / raw)
To: Stefan Priebe, ceph-devel
On 02/10/2015 04:18 PM, Stefan Priebe wrote:
>
> Am 10.02.2015 um 22:38 schrieb Mark Nelson:
>> On 02/10/2015 03:11 PM, Stefan Priebe wrote:
>>>
>>> mhm i installed librbd1-dbg and librados2-dbg - but the output still
>>> looks useless to me. Should i upload it somewhere?
>>
>> Meh, if it's all just symbols it's probably not that helpful.
>>
>> I've summarized your results here:
>>
>> 1 concurrent 4k write (libaio, direct=1, iodepth=1)
>>
>> IOPS Latency
>> wb on wb off wb on wb off
>> dumpling 10870 536 ~100us ~2ms
>> firefly 10350 525 ~100us ~2ms
>>
>> So in single op tests dumpling and firefly are far closer. Now let's
>> see each of these cases with iodepth=32 (still 1 thread for now).
>
>
> dumpling:
>
> file1: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=32
> 2.0.8
> Starting 1 thread
> Jobs: 1 (f=1): [w] [100.0% done] [0K/72812K /s] [0 /18.3K iops] [eta
> 00m:00s]
> file1: (groupid=0, jobs=1): err= 0: pid=3011
> write: io=2060.6MB, bw=70329KB/s, iops=17582 , runt= 30001msec
> slat (usec): min=1 , max=3517 , avg= 3.42, stdev= 7.30
> clat (usec): min=93 , max=7475 , avg=1815.72, stdev=233.43
> lat (usec): min=219 , max=7477 , avg=1819.27, stdev=233.52
> clat percentiles (usec):
> | 1.00th=[ 1480], 5.00th=[ 1576], 10.00th=[ 1608], 20.00th=[ 1672],
> | 30.00th=[ 1704], 40.00th=[ 1752], 50.00th=[ 1800], 60.00th=[ 1832],
> | 70.00th=[ 1896], 80.00th=[ 1960], 90.00th=[ 2064], 95.00th=[ 2128],
> | 99.00th=[ 2352], 99.50th=[ 2448], 99.90th=[ 4704], 99.95th=[ 5344],
> | 99.99th=[ 7072]
> bw (KB/s) : min=59696, max=77840, per=100.00%, avg=70351.27,
> stdev=4783.25
> lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.53%
> lat (msec) : 2=85.02%, 4=14.31%, 10=0.13%
> cpu : usr=1.96%, sys=6.71%, ctx=22791, majf=0, minf=133
> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%,
> >=64=0.0%
> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
> >=64=0.0%
> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
> >=64=0.0%
> issued : total=r=0/w=527487/d=0, short=r=0/w=0/d=0
>
> Run status group 0 (all jobs):
> WRITE: io=2060.6MB, aggrb=70329KB/s, minb=70329KB/s, maxb=70329KB/s,
> mint=30001msec, maxt=30001msec
>
> Disk stats (read/write):
> sdb: ios=166/526079, merge=0/0, ticks=24/890120, in_queue=890064,
> util=98.73%
>
> firefly:
>
> file1: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=32
> 2.0.8
> Starting 1 thread
> Jobs: 1 (f=1): [w] [100.0% done] [0K/69096K /s] [0 /17.3K iops] [eta
> 00m:00s]
> file1: (groupid=0, jobs=1): err= 0: pid=2982
> write: io=1784.9MB, bw=60918KB/s, iops=15229 , runt= 30002msec
> slat (usec): min=1 , max=1389 , avg= 3.43, stdev= 5.32
> clat (usec): min=117 , max=8235 , avg=2096.88, stdev=396.30
> lat (usec): min=540 , max=8258 , avg=2100.43, stdev=396.61
> clat percentiles (usec):
> | 1.00th=[ 1608], 5.00th=[ 1720], 10.00th=[ 1768], 20.00th=[ 1832],
> | 30.00th=[ 1896], 40.00th=[ 1944], 50.00th=[ 2008], 60.00th=[ 2064],
> | 70.00th=[ 2160], 80.00th=[ 2256], 90.00th=[ 2512], 95.00th=[ 2896],
> | 99.00th=[ 3600], 99.50th=[ 3792], 99.90th=[ 5088], 99.95th=[ 6304],
> | 99.99th=[ 6752]
> bw (KB/s) : min=36717, max=73712, per=99.94%, avg=60879.92,
> stdev=8302.27
> lat (usec) : 250=0.01%, 750=0.01%
> lat (msec) : 2=48.56%, 4=51.18%, 10=0.26%
> cpu : usr=2.03%, sys=5.48%, ctx=20440, majf=0, minf=133
> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%,
> >=64=0.0%
> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
> >=64=0.0%
> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
> >=64=0.0%
> issued : total=r=0/w=456918/d=0, short=r=0/w=0/d=0
>
> Run status group 0 (all jobs):
> WRITE: io=1784.9MB, aggrb=60918KB/s, minb=60918KB/s, maxb=60918KB/s,
> mint=30002msec, maxt=30002msec
>
> Disk stats (read/write):
> sdb: ios=166/455574, merge=0/0, ticks=12/897748, in_queue=897696,
> util=98.96%
>
Ok, so it looks like as you increase concurrency the effect increases
(ie contention?). Does the same thing happen without cache enabled?
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: speed decrease since firefly,giant,hammer the 2nd try
2015-02-11 4:45 ` Mark Nelson
@ 2015-02-11 5:42 ` Stefan Priebe
[not found] ` <1032317804.358821.1423634390142.JavaMail.zimbra@oxygem.tv>
2015-02-15 18:40 ` Stefan Priebe
0 siblings, 2 replies; 32+ messages in thread
From: Stefan Priebe @ 2015-02-11 5:42 UTC (permalink / raw)
To: Mark Nelson, ceph-devel
Am 11.02.2015 um 05:45 schrieb Mark Nelson:
> On 02/10/2015 04:18 PM, Stefan Priebe wrote:
>>
>> Am 10.02.2015 um 22:38 schrieb Mark Nelson:
>>> On 02/10/2015 03:11 PM, Stefan Priebe wrote:
>>>>
>>>> mhm i installed librbd1-dbg and librados2-dbg - but the output still
>>>> looks useless to me. Should i upload it somewhere?
>>>
>>> Meh, if it's all just symbols it's probably not that helpful.
>>>
>>> I've summarized your results here:
>>>
>>> 1 concurrent 4k write (libaio, direct=1, iodepth=1)
>>>
>>> IOPS Latency
>>> wb on wb off wb on wb off
>>> dumpling 10870 536 ~100us ~2ms
>>> firefly 10350 525 ~100us ~2ms
>>>
>>> So in single op tests dumpling and firefly are far closer. Now let's
>>> see each of these cases with iodepth=32 (still 1 thread for now).
>>
>>
>> dumpling:
>>
>> file1: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=32
>> 2.0.8
>> Starting 1 thread
>> Jobs: 1 (f=1): [w] [100.0% done] [0K/72812K /s] [0 /18.3K iops] [eta
>> 00m:00s]
>> file1: (groupid=0, jobs=1): err= 0: pid=3011
>> write: io=2060.6MB, bw=70329KB/s, iops=17582 , runt= 30001msec
>> slat (usec): min=1 , max=3517 , avg= 3.42, stdev= 7.30
>> clat (usec): min=93 , max=7475 , avg=1815.72, stdev=233.43
>> lat (usec): min=219 , max=7477 , avg=1819.27, stdev=233.52
>> clat percentiles (usec):
>> | 1.00th=[ 1480], 5.00th=[ 1576], 10.00th=[ 1608], 20.00th=[
>> 1672],
>> | 30.00th=[ 1704], 40.00th=[ 1752], 50.00th=[ 1800], 60.00th=[
>> 1832],
>> | 70.00th=[ 1896], 80.00th=[ 1960], 90.00th=[ 2064], 95.00th=[
>> 2128],
>> | 99.00th=[ 2352], 99.50th=[ 2448], 99.90th=[ 4704], 99.95th=[
>> 5344],
>> | 99.99th=[ 7072]
>> bw (KB/s) : min=59696, max=77840, per=100.00%, avg=70351.27,
>> stdev=4783.25
>> lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.53%
>> lat (msec) : 2=85.02%, 4=14.31%, 10=0.13%
>> cpu : usr=1.96%, sys=6.71%, ctx=22791, majf=0, minf=133
>> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%,
>> >=64=0.0%
>> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>> >=64=0.0%
>> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
>> >=64=0.0%
>> issued : total=r=0/w=527487/d=0, short=r=0/w=0/d=0
>>
>> Run status group 0 (all jobs):
>> WRITE: io=2060.6MB, aggrb=70329KB/s, minb=70329KB/s, maxb=70329KB/s,
>> mint=30001msec, maxt=30001msec
>>
>> Disk stats (read/write):
>> sdb: ios=166/526079, merge=0/0, ticks=24/890120, in_queue=890064,
>> util=98.73%
>>
>> firefly:
>>
>> file1: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=32
>> 2.0.8
>> Starting 1 thread
>> Jobs: 1 (f=1): [w] [100.0% done] [0K/69096K /s] [0 /17.3K iops] [eta
>> 00m:00s]
>> file1: (groupid=0, jobs=1): err= 0: pid=2982
>> write: io=1784.9MB, bw=60918KB/s, iops=15229 , runt= 30002msec
>> slat (usec): min=1 , max=1389 , avg= 3.43, stdev= 5.32
>> clat (usec): min=117 , max=8235 , avg=2096.88, stdev=396.30
>> lat (usec): min=540 , max=8258 , avg=2100.43, stdev=396.61
>> clat percentiles (usec):
>> | 1.00th=[ 1608], 5.00th=[ 1720], 10.00th=[ 1768], 20.00th=[
>> 1832],
>> | 30.00th=[ 1896], 40.00th=[ 1944], 50.00th=[ 2008], 60.00th=[
>> 2064],
>> | 70.00th=[ 2160], 80.00th=[ 2256], 90.00th=[ 2512], 95.00th=[
>> 2896],
>> | 99.00th=[ 3600], 99.50th=[ 3792], 99.90th=[ 5088], 99.95th=[
>> 6304],
>> | 99.99th=[ 6752]
>> bw (KB/s) : min=36717, max=73712, per=99.94%, avg=60879.92,
>> stdev=8302.27
>> lat (usec) : 250=0.01%, 750=0.01%
>> lat (msec) : 2=48.56%, 4=51.18%, 10=0.26%
>> cpu : usr=2.03%, sys=5.48%, ctx=20440, majf=0, minf=133
>> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%,
>> >=64=0.0%
>> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>> >=64=0.0%
>> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
>> >=64=0.0%
>> issued : total=r=0/w=456918/d=0, short=r=0/w=0/d=0
>>
>> Run status group 0 (all jobs):
>> WRITE: io=1784.9MB, aggrb=60918KB/s, minb=60918KB/s, maxb=60918KB/s,
>> mint=30002msec, maxt=30002msec
>>
>> Disk stats (read/write):
>> sdb: ios=166/455574, merge=0/0, ticks=12/897748, in_queue=897696,
>> util=98.96%
>>
>
> Ok, so it looks like as you increase concurrency the effect increases
> (ie contention?). Does the same thing happen without cache enabled?
here again without rbd cache:
dumpling:
file1: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=32
2.0.8
Starting 1 thread
Jobs: 1 (f=1): [w] [100.0% done] [0K/83488K /s] [0 /20.9K iops] [eta
00m:00s]
file1: (groupid=0, jobs=1): err= 0: pid=3000
write: io=2449.2MB, bw=83583KB/s, iops=20895 , runt= 30005msec
slat (usec): min=1 , max=975 , avg= 4.50, stdev= 5.25
clat (usec): min=364 , max=80566 , avg=1525.87, stdev=1194.57
lat (usec): min=519 , max=80568 , avg=1530.51, stdev=1194.44
clat percentiles (usec):
| 1.00th=[ 660], 5.00th=[ 780], 10.00th=[ 876], 20.00th=[ 1032],
| 30.00th=[ 1144], 40.00th=[ 1240], 50.00th=[ 1304], 60.00th=[ 1384],
| 70.00th=[ 1480], 80.00th=[ 1640], 90.00th=[ 2096], 95.00th=[ 2960],
| 99.00th=[ 6816], 99.50th=[ 7840], 99.90th=[11712], 99.95th=[13888],
| 99.99th=[18816]
bw (KB/s) : min=47184, max=95432, per=100.00%, avg=83639.19,
stdev=7973.92
lat (usec) : 500=0.01%, 750=3.82%, 1000=14.40%
lat (msec) : 2=70.57%, 4=7.91%, 10=3.11%, 20=0.17%, 50=0.01%
lat (msec) : 100=0.01%
cpu : usr=3.12%, sys=11.49%, ctx=74951, majf=0, minf=133
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%,
>=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
>=64=0.0%
issued : total=r=0/w=626979/d=0, short=r=0/w=0/d=0
Run status group 0 (all jobs):
WRITE: io=2449.2MB, aggrb=83583KB/s, minb=83583KB/s, maxb=83583KB/s,
mint=30005msec, maxt=30005msec
Disk stats (read/write):
sdb: ios=168/625292, merge=0/0, ticks=144/916096, in_queue=916128,
util=99.93%
firefly:
fio --filename=/dev/sdb --direct=1 --rw=randwrite --bs=4k --numjobs=1
--thread --iodepth=32 --ioengine=libaio --runtime=30 --group_reporting
--name=file1
file1: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=32
2.0.8
Starting 1 thread
Jobs: 1 (f=1): [w] [100.0% done] [0K/90044K /s] [0 /22.6K iops] [eta
00m:00s]
file1: (groupid=0, jobs=1): err= 0: pid=2970
write: io=2372.9MB, bw=80976KB/s, iops=20244 , runt= 30006msec
slat (usec): min=1 , max=4047 , avg= 4.36, stdev= 7.17
clat (usec): min=197 , max=76656 , avg=1575.29, stdev=1165.74
lat (usec): min=523 , max=76660 , avg=1579.79, stdev=1165.59
clat percentiles (usec):
| 1.00th=[ 676], 5.00th=[ 804], 10.00th=[ 916], 20.00th=[ 1096],
| 30.00th=[ 1224], 40.00th=[ 1304], 50.00th=[ 1384], 60.00th=[ 1448],
| 70.00th=[ 1544], 80.00th=[ 1704], 90.00th=[ 2128], 95.00th=[ 2736],
| 99.00th=[ 6752], 99.50th=[ 7904], 99.90th=[12096], 99.95th=[14656],
| 99.99th=[18560]
bw (KB/s) : min=47800, max=91952, per=99.91%, avg=80900.88,
stdev=7234.98
lat (usec) : 250=0.01%, 500=0.01%, 750=2.95%, 1000=11.38%
lat (msec) : 2=73.81%, 4=8.81%, 10=2.85%, 20=0.19%, 50=0.01%
lat (msec) : 100=0.01%
cpu : usr=2.99%, sys=10.60%, ctx=66549, majf=0, minf=133
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%,
>=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
>=64=0.0%
issued : total=r=0/w=607445/d=0, short=r=0/w=0/d=0
Run status group 0 (all jobs):
WRITE: io=2372.9MB, aggrb=80976KB/s, minb=80976KB/s, maxb=80976KB/s,
mint=30006msec, maxt=30006msec
Disk stats (read/write):
sdb: ios=170/605440, merge=0/0, ticks=156/916492, in_queue=916560,
util=99.93%
Stefan
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: speed decrease since firefly,giant,hammer the 2nd try
[not found] ` <1032317804.358821.1423634390142.JavaMail.zimbra@oxygem.tv>
@ 2015-02-11 5:59 ` Alexandre DERUMIER
0 siblings, 0 replies; 32+ messages in thread
From: Alexandre DERUMIER @ 2015-02-11 5:59 UTC (permalink / raw)
To: Stefan Priebe; +Cc: Mark Nelson, ceph-devel
Hi All,
do we have client cpu benchmark between dumpling vs firefly.
As qemu only use 1 thread, It's quite possible that cpu increase inside librbd in firefly,
give use lower iops.
(See my other discussion about librbd using 4x cpu than krbd)
----- Mail original -----
De: "Stefan Priebe" <s.priebe@profihost.ag>
À: "Mark Nelson" <mnelson@redhat.com>, "ceph-devel" <ceph-devel@vger.kernel.org>
Envoyé: Mercredi 11 Février 2015 06:42:21
Objet: Re: speed decrease since firefly,giant,hammer the 2nd try
Am 11.02.2015 um 05:45 schrieb Mark Nelson:
> On 02/10/2015 04:18 PM, Stefan Priebe wrote:
>>
>> Am 10.02.2015 um 22:38 schrieb Mark Nelson:
>>> On 02/10/2015 03:11 PM, Stefan Priebe wrote:
>>>>
>>>> mhm i installed librbd1-dbg and librados2-dbg - but the output still
>>>> looks useless to me. Should i upload it somewhere?
>>>
>>> Meh, if it's all just symbols it's probably not that helpful.
>>>
>>> I've summarized your results here:
>>>
>>> 1 concurrent 4k write (libaio, direct=1, iodepth=1)
>>>
>>> IOPS Latency
>>> wb on wb off wb on wb off
>>> dumpling 10870 536 ~100us ~2ms
>>> firefly 10350 525 ~100us ~2ms
>>>
>>> So in single op tests dumpling and firefly are far closer. Now let's
>>> see each of these cases with iodepth=32 (still 1 thread for now).
>>
>>
>> dumpling:
>>
>> file1: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=32
>> 2.0.8
>> Starting 1 thread
>> Jobs: 1 (f=1): [w] [100.0% done] [0K/72812K /s] [0 /18.3K iops] [eta
>> 00m:00s]
>> file1: (groupid=0, jobs=1): err= 0: pid=3011
>> write: io=2060.6MB, bw=70329KB/s, iops=17582 , runt= 30001msec
>> slat (usec): min=1 , max=3517 , avg= 3.42, stdev= 7.30
>> clat (usec): min=93 , max=7475 , avg=1815.72, stdev=233.43
>> lat (usec): min=219 , max=7477 , avg=1819.27, stdev=233.52
>> clat percentiles (usec):
>> | 1.00th=[ 1480], 5.00th=[ 1576], 10.00th=[ 1608], 20.00th=[
>> 1672],
>> | 30.00th=[ 1704], 40.00th=[ 1752], 50.00th=[ 1800], 60.00th=[
>> 1832],
>> | 70.00th=[ 1896], 80.00th=[ 1960], 90.00th=[ 2064], 95.00th=[
>> 2128],
>> | 99.00th=[ 2352], 99.50th=[ 2448], 99.90th=[ 4704], 99.95th=[
>> 5344],
>> | 99.99th=[ 7072]
>> bw (KB/s) : min=59696, max=77840, per=100.00%, avg=70351.27,
>> stdev=4783.25
>> lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.53%
>> lat (msec) : 2=85.02%, 4=14.31%, 10=0.13%
>> cpu : usr=1.96%, sys=6.71%, ctx=22791, majf=0, minf=133
>> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%,
>> >=64=0.0%
>> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>> >=64=0.0%
>> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
>> >=64=0.0%
>> issued : total=r=0/w=527487/d=0, short=r=0/w=0/d=0
>>
>> Run status group 0 (all jobs):
>> WRITE: io=2060.6MB, aggrb=70329KB/s, minb=70329KB/s, maxb=70329KB/s,
>> mint=30001msec, maxt=30001msec
>>
>> Disk stats (read/write):
>> sdb: ios=166/526079, merge=0/0, ticks=24/890120, in_queue=890064,
>> util=98.73%
>>
>> firefly:
>>
>> file1: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=32
>> 2.0.8
>> Starting 1 thread
>> Jobs: 1 (f=1): [w] [100.0% done] [0K/69096K /s] [0 /17.3K iops] [eta
>> 00m:00s]
>> file1: (groupid=0, jobs=1): err= 0: pid=2982
>> write: io=1784.9MB, bw=60918KB/s, iops=15229 , runt= 30002msec
>> slat (usec): min=1 , max=1389 , avg= 3.43, stdev= 5.32
>> clat (usec): min=117 , max=8235 , avg=2096.88, stdev=396.30
>> lat (usec): min=540 , max=8258 , avg=2100.43, stdev=396.61
>> clat percentiles (usec):
>> | 1.00th=[ 1608], 5.00th=[ 1720], 10.00th=[ 1768], 20.00th=[
>> 1832],
>> | 30.00th=[ 1896], 40.00th=[ 1944], 50.00th=[ 2008], 60.00th=[
>> 2064],
>> | 70.00th=[ 2160], 80.00th=[ 2256], 90.00th=[ 2512], 95.00th=[
>> 2896],
>> | 99.00th=[ 3600], 99.50th=[ 3792], 99.90th=[ 5088], 99.95th=[
>> 6304],
>> | 99.99th=[ 6752]
>> bw (KB/s) : min=36717, max=73712, per=99.94%, avg=60879.92,
>> stdev=8302.27
>> lat (usec) : 250=0.01%, 750=0.01%
>> lat (msec) : 2=48.56%, 4=51.18%, 10=0.26%
>> cpu : usr=2.03%, sys=5.48%, ctx=20440, majf=0, minf=133
>> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%,
>> >=64=0.0%
>> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>> >=64=0.0%
>> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
>> >=64=0.0%
>> issued : total=r=0/w=456918/d=0, short=r=0/w=0/d=0
>>
>> Run status group 0 (all jobs):
>> WRITE: io=1784.9MB, aggrb=60918KB/s, minb=60918KB/s, maxb=60918KB/s,
>> mint=30002msec, maxt=30002msec
>>
>> Disk stats (read/write):
>> sdb: ios=166/455574, merge=0/0, ticks=12/897748, in_queue=897696,
>> util=98.96%
>>
>
> Ok, so it looks like as you increase concurrency the effect increases
> (ie contention?). Does the same thing happen without cache enabled?
here again without rbd cache:
dumpling:
file1: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=32
2.0.8
Starting 1 thread
Jobs: 1 (f=1): [w] [100.0% done] [0K/83488K /s] [0 /20.9K iops] [eta
00m:00s]
file1: (groupid=0, jobs=1): err= 0: pid=3000
write: io=2449.2MB, bw=83583KB/s, iops=20895 , runt= 30005msec
slat (usec): min=1 , max=975 , avg= 4.50, stdev= 5.25
clat (usec): min=364 , max=80566 , avg=1525.87, stdev=1194.57
lat (usec): min=519 , max=80568 , avg=1530.51, stdev=1194.44
clat percentiles (usec):
| 1.00th=[ 660], 5.00th=[ 780], 10.00th=[ 876], 20.00th=[ 1032],
| 30.00th=[ 1144], 40.00th=[ 1240], 50.00th=[ 1304], 60.00th=[ 1384],
| 70.00th=[ 1480], 80.00th=[ 1640], 90.00th=[ 2096], 95.00th=[ 2960],
| 99.00th=[ 6816], 99.50th=[ 7840], 99.90th=[11712], 99.95th=[13888],
| 99.99th=[18816]
bw (KB/s) : min=47184, max=95432, per=100.00%, avg=83639.19,
stdev=7973.92
lat (usec) : 500=0.01%, 750=3.82%, 1000=14.40%
lat (msec) : 2=70.57%, 4=7.91%, 10=3.11%, 20=0.17%, 50=0.01%
lat (msec) : 100=0.01%
cpu : usr=3.12%, sys=11.49%, ctx=74951, majf=0, minf=133
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%,
>=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
>=64=0.0%
issued : total=r=0/w=626979/d=0, short=r=0/w=0/d=0
Run status group 0 (all jobs):
WRITE: io=2449.2MB, aggrb=83583KB/s, minb=83583KB/s, maxb=83583KB/s,
mint=30005msec, maxt=30005msec
Disk stats (read/write):
sdb: ios=168/625292, merge=0/0, ticks=144/916096, in_queue=916128,
util=99.93%
firefly:
fio --filename=/dev/sdb --direct=1 --rw=randwrite --bs=4k --numjobs=1
--thread --iodepth=32 --ioengine=libaio --runtime=30 --group_reporting
--name=file1
file1: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=32
2.0.8
Starting 1 thread
Jobs: 1 (f=1): [w] [100.0% done] [0K/90044K /s] [0 /22.6K iops] [eta
00m:00s]
file1: (groupid=0, jobs=1): err= 0: pid=2970
write: io=2372.9MB, bw=80976KB/s, iops=20244 , runt= 30006msec
slat (usec): min=1 , max=4047 , avg= 4.36, stdev= 7.17
clat (usec): min=197 , max=76656 , avg=1575.29, stdev=1165.74
lat (usec): min=523 , max=76660 , avg=1579.79, stdev=1165.59
clat percentiles (usec):
| 1.00th=[ 676], 5.00th=[ 804], 10.00th=[ 916], 20.00th=[ 1096],
| 30.00th=[ 1224], 40.00th=[ 1304], 50.00th=[ 1384], 60.00th=[ 1448],
| 70.00th=[ 1544], 80.00th=[ 1704], 90.00th=[ 2128], 95.00th=[ 2736],
| 99.00th=[ 6752], 99.50th=[ 7904], 99.90th=[12096], 99.95th=[14656],
| 99.99th=[18560]
bw (KB/s) : min=47800, max=91952, per=99.91%, avg=80900.88,
stdev=7234.98
lat (usec) : 250=0.01%, 500=0.01%, 750=2.95%, 1000=11.38%
lat (msec) : 2=73.81%, 4=8.81%, 10=2.85%, 20=0.19%, 50=0.01%
lat (msec) : 100=0.01%
cpu : usr=2.99%, sys=10.60%, ctx=66549, majf=0, minf=133
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%,
>=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
>=64=0.0%
issued : total=r=0/w=607445/d=0, short=r=0/w=0/d=0
Run status group 0 (all jobs):
WRITE: io=2372.9MB, aggrb=80976KB/s, minb=80976KB/s, maxb=80976KB/s,
mint=30006msec, maxt=30006msec
Disk stats (read/write):
sdb: ios=170/605440, merge=0/0, ticks=156/916492, in_queue=916560,
util=99.93%
Stefan
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: speed decrease since firefly,giant,hammer the 2nd try
[not found] ` <764258391.358931.1423634637318.JavaMail.zimbra@oxygem.tv>
@ 2015-02-11 6:04 ` Alexandre DERUMIER
0 siblings, 0 replies; 32+ messages in thread
From: Alexandre DERUMIER @ 2015-02-11 6:04 UTC (permalink / raw)
To: Mark Nelson; +Cc: Stefan Priebe, ceph-devel
>
> mhm i installed librbd1-dbg and librados2-dbg - but the output still
> looks useless to me. Should i upload it somewhere?
>>Meh, if it's all just symbols it's probably not that helpful.
Last Time I have done perf, even with all debug packages (ceph debug package, kernel debug, libc6 debug package,..)
I have remaining unresolved symbols
Sage Have run my perf report through c++filt, to have missing symbols.
see here:
http://article.gmane.org/gmane.comp.file-systems.ceph.devel/22089/match=client+cpu+usage+kbrd+vs+librbd+perf+report
----- Mail original -----
De: "Mark Nelson" <mnelson@redhat.com>
À: "Stefan Priebe" <s.priebe@profihost.ag>, "ceph-devel" <ceph-devel@vger.kernel.org>
Envoyé: Mardi 10 Février 2015 22:38:07
Objet: Re: speed decrease since firefly,giant,hammer the 2nd try
On 02/10/2015 03:11 PM, Stefan Priebe wrote:
>
> mhm i installed librbd1-dbg and librados2-dbg - but the output still
> looks useless to me. Should i upload it somewhere?
Meh, if it's all just symbols it's probably not that helpful.
I've summarized your results here:
1 concurrent 4k write (libaio, direct=1, iodepth=1)
IOPS Latency
wb on wb off wb on wb off
dumpling 10870 536 ~100us ~2ms
firefly 10350 525 ~100us ~2ms
So in single op tests dumpling and firefly are far closer. Now let's
see each of these cases with iodepth=32 (still 1 thread for now).
Mark
>
> Stefan
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: speed decrease since firefly,giant,hammer the 2nd try
[not found] ` <1971513819.362434.1423640637237.JavaMail.zimbra@oxygem.tv>
@ 2015-02-11 7:44 ` Alexandre DERUMIER
2015-02-11 8:32 ` Stefan Priebe - Profihost AG
0 siblings, 1 reply; 32+ messages in thread
From: Alexandre DERUMIER @ 2015-02-11 7:44 UTC (permalink / raw)
To: Stefan Priebe; +Cc: ceph-devel
>>same fio, same qemu, same vm, same host, same ceph dumpling storage,
>>different librados / librbd: 16k iop/s for random 4k writes
>>
>>What's wrong with librbd / librados2 since firefly?
Maybe could we bissect this ?
Maybe testing intermediate librbd releases between dumpling and firefly,
http://gitbuilder.ceph.com/ceph-deb-wheezy-x86_64-basic/ref/
could we give us an hint.
----- Mail original -----
De: "Stefan Priebe" <s.priebe@profihost.ag>
À: "ceph-devel" <ceph-devel@vger.kernel.org>
Envoyé: Mardi 10 Février 2015 19:55:26
Objet: speed decrease since firefly,giant,hammer the 2nd try
Hello,
last year in june i already reported this but there was no real result.
(http://lists.ceph.com/pipermail/ceph-users-ceph.com/2014-July/041070.html)
I then had the hope that this will be fixed itself when hammer is
released. Now i tried hammer an the results are bad as before.
Since firefly librbd1 / librados2 are 20% slower for 4k random iop/s
than dumpling - this is also the reason why i still stick to dumpling.
I've now modified my test again to be a bit more clear.
Ceph cluster itself completely dumpling.
librbd1 / librados from dumpling (fio inside qemu): 23k iop/s for random
4k writes
- stopped qemu
- cp -ra firefly_0.80.8/usr/lib/librados.so.2.0.0 /usr/lib/
- cp -ra firefly_0.80.8/usr/lib/librbd.so.1.0.0 /usr/lib/
- start qemu
same fio, same qemu, same vm, same host, same ceph dumpling storage,
different librados / librbd: 16k iop/s for random 4k writes
What's wrong with librbd / librados2 since firefly?
Greets,
Stefan
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: speed decrease since firefly,giant,hammer the 2nd try
2015-02-11 7:44 ` Alexandre DERUMIER
@ 2015-02-11 8:32 ` Stefan Priebe - Profihost AG
0 siblings, 0 replies; 32+ messages in thread
From: Stefan Priebe - Profihost AG @ 2015-02-11 8:32 UTC (permalink / raw)
To: Alexandre DERUMIER; +Cc: ceph-devel
Am 11.02.2015 um 08:44 schrieb Alexandre DERUMIER:
>>> same fio, same qemu, same vm, same host, same ceph dumpling storage,
>>> different librados / librbd: 16k iop/s for random 4k writes
>>>
>>> What's wrong with librbd / librados2 since firefly?
>
> Maybe could we bissect this ?
>
> Maybe testing intermediate librbd releases between dumpling and firefly,
>
> http://gitbuilder.ceph.com/ceph-deb-wheezy-x86_64-basic/ref/
Yes may be. Sadly i've currently another problem on my newest cluster
having strange kworker workload - i've never noticed before on any ceph
system. All writes are hanging - while using the same kernel as everywhere.
Stefan
>
> could we give us an hint.
>
>
> ----- Mail original -----
> De: "Stefan Priebe" <s.priebe@profihost.ag>
> À: "ceph-devel" <ceph-devel@vger.kernel.org>
> Envoyé: Mardi 10 Février 2015 19:55:26
> Objet: speed decrease since firefly,giant,hammer the 2nd try
>
> Hello,
>
> last year in june i already reported this but there was no real result.
> (http://lists.ceph.com/pipermail/ceph-users-ceph.com/2014-July/041070.html)
>
> I then had the hope that this will be fixed itself when hammer is
> released. Now i tried hammer an the results are bad as before.
>
> Since firefly librbd1 / librados2 are 20% slower for 4k random iop/s
> than dumpling - this is also the reason why i still stick to dumpling.
>
> I've now modified my test again to be a bit more clear.
>
> Ceph cluster itself completely dumpling.
>
> librbd1 / librados from dumpling (fio inside qemu): 23k iop/s for random
> 4k writes
>
> - stopped qemu
> - cp -ra firefly_0.80.8/usr/lib/librados.so.2.0.0 /usr/lib/
> - cp -ra firefly_0.80.8/usr/lib/librbd.so.1.0.0 /usr/lib/
> - start qemu
>
> same fio, same qemu, same vm, same host, same ceph dumpling storage,
> different librados / librbd: 16k iop/s for random 4k writes
>
> What's wrong with librbd / librados2 since firefly?
>
> Greets,
> Stefan
>
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: speed decrease since firefly,giant,hammer the 2nd try
2015-02-11 5:42 ` Stefan Priebe
[not found] ` <1032317804.358821.1423634390142.JavaMail.zimbra@oxygem.tv>
@ 2015-02-15 18:40 ` Stefan Priebe
[not found] ` <1880785650.892634.1424077856500.JavaMail.zimbra@oxygem.tv>
2015-02-16 21:22 ` Stefan Priebe
1 sibling, 2 replies; 32+ messages in thread
From: Stefan Priebe @ 2015-02-15 18:40 UTC (permalink / raw)
To: Mark Nelson, ceph-devel
Hi Mark,
what's next?
I've this test cluster only for 2 more days.
Here some perf Details:
dumpling:
12,65% libc-2.13.so [.] 0x79000
2,86% libc-2.13.so [.] malloc
2,80% kvm [.] 0xb59c5
2,59% libc-2.13.so [.] free
2,35% [kernel] [k] __schedule
2,16% [kernel] [k] _raw_spin_lock
1,92% [kernel] [k] __switch_to
1,58% [kernel] [k] lapic_next_deadline
1,09% [kernel] [k] update_sd_lb_stats
1,08% [kernel] [k] _raw_spin_lock_irqsave
0,91% librados.so.2.0.0 [.] ceph_crc32c_le_intel
0,91% libpthread-2.13.so [.] pthread_mutex_trylock
0,87% [kernel] [k] resched_task
0,72% [kernel] [k] cpu_startup_entry
0,71% librados.so.2.0.0 [.] crush_hash32_3
0,66% [kernel] [k] leave_mm
0,65% librados.so.2.0.0 [.] Mutex::Lock(bool)
0,64% [kernel] [k] idle_cpu
0,62% libpthread-2.13.so [.] __pthread_mutex_unlock_usercnt
0,59% [kernel] [k] try_to_wake_up
0,56% [kernel] [k] wake_futex
0,50% librados.so.2.0.0 [.] ceph::buffer::ptr::release()
firefly:
12,56% libc-2.13.so [.] 0x7905d
2,82% libc-2.13.so [.] malloc
2,64% libc-2.13.so [.] free
2,61% kvm [.] 0x34322f
2,33% [kernel] [k] __schedule
2,14% [kernel] [k] _raw_spin_lock
1,83% [kernel] [k] __switch_to
1,62% [kernel] [k] lapic_next_deadline
1,17% [kernel] [k] _raw_spin_lock_irqsave
1,09% [kernel] [k] update_sd_lb_stats
1,08% libpthread-2.13.so [.] pthread_mutex_trylock
0,85% libpthread-2.13.so [.] __pthread_mutex_unlock_usercnt
0,77% [kernel] [k] resched_task
0,74% librbd.so.1.0.0 [.] 0x71b73
0,72% librados.so.2.0.0 [.] Mutex::Lock(bool)
0,68% librados.so.2.0.0 [.] crush_hash32_3
0,67% [kernel] [k] idle_cpu
0,65% [kernel] [k] leave_mm
0,65% [kernel] [k] cpu_startup_entry
0,59% [kernel] [k] try_to_wake_up
0,51% librados.so.2.0.0 [.] ceph::buffer::ptr::release()
0,51% [kernel] [k] wake_futex
Stefan
Am 11.02.2015 um 06:42 schrieb Stefan Priebe:
>
> Am 11.02.2015 um 05:45 schrieb Mark Nelson:
>> On 02/10/2015 04:18 PM, Stefan Priebe wrote:
>>>
>>> Am 10.02.2015 um 22:38 schrieb Mark Nelson:
>>>> On 02/10/2015 03:11 PM, Stefan Priebe wrote:
>>>>>
>>>>> mhm i installed librbd1-dbg and librados2-dbg - but the output still
>>>>> looks useless to me. Should i upload it somewhere?
>>>>
>>>> Meh, if it's all just symbols it's probably not that helpful.
>>>>
>>>> I've summarized your results here:
>>>>
>>>> 1 concurrent 4k write (libaio, direct=1, iodepth=1)
>>>>
>>>> IOPS Latency
>>>> wb on wb off wb on wb off
>>>> dumpling 10870 536 ~100us ~2ms
>>>> firefly 10350 525 ~100us ~2ms
>>>>
>>>> So in single op tests dumpling and firefly are far closer. Now let's
>>>> see each of these cases with iodepth=32 (still 1 thread for now).
>>>
>>>
>>> dumpling:
>>>
>>> file1: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=32
>>> 2.0.8
>>> Starting 1 thread
>>> Jobs: 1 (f=1): [w] [100.0% done] [0K/72812K /s] [0 /18.3K iops] [eta
>>> 00m:00s]
>>> file1: (groupid=0, jobs=1): err= 0: pid=3011
>>> write: io=2060.6MB, bw=70329KB/s, iops=17582 , runt= 30001msec
>>> slat (usec): min=1 , max=3517 , avg= 3.42, stdev= 7.30
>>> clat (usec): min=93 , max=7475 , avg=1815.72, stdev=233.43
>>> lat (usec): min=219 , max=7477 , avg=1819.27, stdev=233.52
>>> clat percentiles (usec):
>>> | 1.00th=[ 1480], 5.00th=[ 1576], 10.00th=[ 1608], 20.00th=[
>>> 1672],
>>> | 30.00th=[ 1704], 40.00th=[ 1752], 50.00th=[ 1800], 60.00th=[
>>> 1832],
>>> | 70.00th=[ 1896], 80.00th=[ 1960], 90.00th=[ 2064], 95.00th=[
>>> 2128],
>>> | 99.00th=[ 2352], 99.50th=[ 2448], 99.90th=[ 4704], 99.95th=[
>>> 5344],
>>> | 99.99th=[ 7072]
>>> bw (KB/s) : min=59696, max=77840, per=100.00%, avg=70351.27,
>>> stdev=4783.25
>>> lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.53%
>>> lat (msec) : 2=85.02%, 4=14.31%, 10=0.13%
>>> cpu : usr=1.96%, sys=6.71%, ctx=22791, majf=0, minf=133
>>> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%,
>>> >=64=0.0%
>>> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>>> >=64=0.0%
>>> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
>>> >=64=0.0%
>>> issued : total=r=0/w=527487/d=0, short=r=0/w=0/d=0
>>>
>>> Run status group 0 (all jobs):
>>> WRITE: io=2060.6MB, aggrb=70329KB/s, minb=70329KB/s, maxb=70329KB/s,
>>> mint=30001msec, maxt=30001msec
>>>
>>> Disk stats (read/write):
>>> sdb: ios=166/526079, merge=0/0, ticks=24/890120, in_queue=890064,
>>> util=98.73%
>>>
>>> firefly:
>>>
>>> file1: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=32
>>> 2.0.8
>>> Starting 1 thread
>>> Jobs: 1 (f=1): [w] [100.0% done] [0K/69096K /s] [0 /17.3K iops] [eta
>>> 00m:00s]
>>> file1: (groupid=0, jobs=1): err= 0: pid=2982
>>> write: io=1784.9MB, bw=60918KB/s, iops=15229 , runt= 30002msec
>>> slat (usec): min=1 , max=1389 , avg= 3.43, stdev= 5.32
>>> clat (usec): min=117 , max=8235 , avg=2096.88, stdev=396.30
>>> lat (usec): min=540 , max=8258 , avg=2100.43, stdev=396.61
>>> clat percentiles (usec):
>>> | 1.00th=[ 1608], 5.00th=[ 1720], 10.00th=[ 1768], 20.00th=[
>>> 1832],
>>> | 30.00th=[ 1896], 40.00th=[ 1944], 50.00th=[ 2008], 60.00th=[
>>> 2064],
>>> | 70.00th=[ 2160], 80.00th=[ 2256], 90.00th=[ 2512], 95.00th=[
>>> 2896],
>>> | 99.00th=[ 3600], 99.50th=[ 3792], 99.90th=[ 5088], 99.95th=[
>>> 6304],
>>> | 99.99th=[ 6752]
>>> bw (KB/s) : min=36717, max=73712, per=99.94%, avg=60879.92,
>>> stdev=8302.27
>>> lat (usec) : 250=0.01%, 750=0.01%
>>> lat (msec) : 2=48.56%, 4=51.18%, 10=0.26%
>>> cpu : usr=2.03%, sys=5.48%, ctx=20440, majf=0, minf=133
>>> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%,
>>> >=64=0.0%
>>> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>>> >=64=0.0%
>>> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
>>> >=64=0.0%
>>> issued : total=r=0/w=456918/d=0, short=r=0/w=0/d=0
>>>
>>> Run status group 0 (all jobs):
>>> WRITE: io=1784.9MB, aggrb=60918KB/s, minb=60918KB/s, maxb=60918KB/s,
>>> mint=30002msec, maxt=30002msec
>>>
>>> Disk stats (read/write):
>>> sdb: ios=166/455574, merge=0/0, ticks=12/897748, in_queue=897696,
>>> util=98.96%
>>>
>>
>> Ok, so it looks like as you increase concurrency the effect increases
>> (ie contention?). Does the same thing happen without cache enabled?
>
> here again without rbd cache:
>
> dumpling:
> file1: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=32
> 2.0.8
> Starting 1 thread
> Jobs: 1 (f=1): [w] [100.0% done] [0K/83488K /s] [0 /20.9K iops] [eta
> 00m:00s]
> file1: (groupid=0, jobs=1): err= 0: pid=3000
> write: io=2449.2MB, bw=83583KB/s, iops=20895 , runt= 30005msec
> slat (usec): min=1 , max=975 , avg= 4.50, stdev= 5.25
> clat (usec): min=364 , max=80566 , avg=1525.87, stdev=1194.57
> lat (usec): min=519 , max=80568 , avg=1530.51, stdev=1194.44
> clat percentiles (usec):
> | 1.00th=[ 660], 5.00th=[ 780], 10.00th=[ 876], 20.00th=[ 1032],
> | 30.00th=[ 1144], 40.00th=[ 1240], 50.00th=[ 1304], 60.00th=[ 1384],
> | 70.00th=[ 1480], 80.00th=[ 1640], 90.00th=[ 2096], 95.00th=[ 2960],
> | 99.00th=[ 6816], 99.50th=[ 7840], 99.90th=[11712], 99.95th=[13888],
> | 99.99th=[18816]
> bw (KB/s) : min=47184, max=95432, per=100.00%, avg=83639.19,
> stdev=7973.92
> lat (usec) : 500=0.01%, 750=3.82%, 1000=14.40%
> lat (msec) : 2=70.57%, 4=7.91%, 10=3.11%, 20=0.17%, 50=0.01%
> lat (msec) : 100=0.01%
> cpu : usr=3.12%, sys=11.49%, ctx=74951, majf=0, minf=133
> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%,
> >=64=0.0%
> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
> >=64=0.0%
> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
> >=64=0.0%
> issued : total=r=0/w=626979/d=0, short=r=0/w=0/d=0
>
> Run status group 0 (all jobs):
> WRITE: io=2449.2MB, aggrb=83583KB/s, minb=83583KB/s, maxb=83583KB/s,
> mint=30005msec, maxt=30005msec
>
> Disk stats (read/write):
> sdb: ios=168/625292, merge=0/0, ticks=144/916096, in_queue=916128,
> util=99.93%
>
>
> firefly:
>
> fio --filename=/dev/sdb --direct=1 --rw=randwrite --bs=4k --numjobs=1
> --thread --iodepth=32 --ioengine=libaio --runtime=30 --group_reporting
> --name=file1
> file1: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=32
> 2.0.8
> Starting 1 thread
> Jobs: 1 (f=1): [w] [100.0% done] [0K/90044K /s] [0 /22.6K iops] [eta
> 00m:00s]
> file1: (groupid=0, jobs=1): err= 0: pid=2970
> write: io=2372.9MB, bw=80976KB/s, iops=20244 , runt= 30006msec
> slat (usec): min=1 , max=4047 , avg= 4.36, stdev= 7.17
> clat (usec): min=197 , max=76656 , avg=1575.29, stdev=1165.74
> lat (usec): min=523 , max=76660 , avg=1579.79, stdev=1165.59
> clat percentiles (usec):
> | 1.00th=[ 676], 5.00th=[ 804], 10.00th=[ 916], 20.00th=[ 1096],
> | 30.00th=[ 1224], 40.00th=[ 1304], 50.00th=[ 1384], 60.00th=[ 1448],
> | 70.00th=[ 1544], 80.00th=[ 1704], 90.00th=[ 2128], 95.00th=[ 2736],
> | 99.00th=[ 6752], 99.50th=[ 7904], 99.90th=[12096], 99.95th=[14656],
> | 99.99th=[18560]
> bw (KB/s) : min=47800, max=91952, per=99.91%, avg=80900.88,
> stdev=7234.98
> lat (usec) : 250=0.01%, 500=0.01%, 750=2.95%, 1000=11.38%
> lat (msec) : 2=73.81%, 4=8.81%, 10=2.85%, 20=0.19%, 50=0.01%
> lat (msec) : 100=0.01%
> cpu : usr=2.99%, sys=10.60%, ctx=66549, majf=0, minf=133
> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%,
> >=64=0.0%
> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
> >=64=0.0%
> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
> >=64=0.0%
> issued : total=r=0/w=607445/d=0, short=r=0/w=0/d=0
>
> Run status group 0 (all jobs):
> WRITE: io=2372.9MB, aggrb=80976KB/s, minb=80976KB/s, maxb=80976KB/s,
> mint=30006msec, maxt=30006msec
>
> Disk stats (read/write):
> sdb: ios=170/605440, merge=0/0, ticks=156/916492, in_queue=916560,
> util=99.93%
>
> Stefan
>
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: speed decrease since firefly,giant,hammer the 2nd try
[not found] ` <1880785650.892634.1424077856500.JavaMail.zimbra@oxygem.tv>
@ 2015-02-16 9:11 ` Alexandre DERUMIER
2015-02-16 14:50 ` Stefan Priebe - Profihost AG
0 siblings, 1 reply; 32+ messages in thread
From: Alexandre DERUMIER @ 2015-02-16 9:11 UTC (permalink / raw)
To: Stefan Priebe; +Cc: Mark Nelson, ceph-devel
Hi Stefan,
I could be interesting to see if you have the same speed decrease with fio-librbd on the host,
without the qemu layer.
the perf reports don't seem to be too much different.
do you have the same cpu usage ? (check qemu process usage)
----- Mail original -----
De: "Stefan Priebe" <s.priebe@profihost.ag>
À: "Mark Nelson" <mnelson@redhat.com>, "ceph-devel" <ceph-devel@vger.kernel.org>
Envoyé: Dimanche 15 Février 2015 19:40:45
Objet: Re: speed decrease since firefly,giant,hammer the 2nd try
Hi Mark,
what's next?
I've this test cluster only for 2 more days.
Here some perf Details:
dumpling:
12,65% libc-2.13.so [.] 0x79000
2,86% libc-2.13.so [.] malloc
2,80% kvm [.] 0xb59c5
2,59% libc-2.13.so [.] free
2,35% [kernel] [k] __schedule
2,16% [kernel] [k] _raw_spin_lock
1,92% [kernel] [k] __switch_to
1,58% [kernel] [k] lapic_next_deadline
1,09% [kernel] [k] update_sd_lb_stats
1,08% [kernel] [k] _raw_spin_lock_irqsave
0,91% librados.so.2.0.0 [.] ceph_crc32c_le_intel
0,91% libpthread-2.13.so [.] pthread_mutex_trylock
0,87% [kernel] [k] resched_task
0,72% [kernel] [k] cpu_startup_entry
0,71% librados.so.2.0.0 [.] crush_hash32_3
0,66% [kernel] [k] leave_mm
0,65% librados.so.2.0.0 [.] Mutex::Lock(bool)
0,64% [kernel] [k] idle_cpu
0,62% libpthread-2.13.so [.] __pthread_mutex_unlock_usercnt
0,59% [kernel] [k] try_to_wake_up
0,56% [kernel] [k] wake_futex
0,50% librados.so.2.0.0 [.] ceph::buffer::ptr::release()
firefly:
12,56% libc-2.13.so [.] 0x7905d
2,82% libc-2.13.so [.] malloc
2,64% libc-2.13.so [.] free
2,61% kvm [.] 0x34322f
2,33% [kernel] [k] __schedule
2,14% [kernel] [k] _raw_spin_lock
1,83% [kernel] [k] __switch_to
1,62% [kernel] [k] lapic_next_deadline
1,17% [kernel] [k] _raw_spin_lock_irqsave
1,09% [kernel] [k] update_sd_lb_stats
1,08% libpthread-2.13.so [.] pthread_mutex_trylock
0,85% libpthread-2.13.so [.] __pthread_mutex_unlock_usercnt
0,77% [kernel] [k] resched_task
0,74% librbd.so.1.0.0 [.] 0x71b73
0,72% librados.so.2.0.0 [.] Mutex::Lock(bool)
0,68% librados.so.2.0.0 [.] crush_hash32_3
0,67% [kernel] [k] idle_cpu
0,65% [kernel] [k] leave_mm
0,65% [kernel] [k] cpu_startup_entry
0,59% [kernel] [k] try_to_wake_up
0,51% librados.so.2.0.0 [.] ceph::buffer::ptr::release()
0,51% [kernel] [k] wake_futex
Stefan
Am 11.02.2015 um 06:42 schrieb Stefan Priebe:
>
> Am 11.02.2015 um 05:45 schrieb Mark Nelson:
>> On 02/10/2015 04:18 PM, Stefan Priebe wrote:
>>>
>>> Am 10.02.2015 um 22:38 schrieb Mark Nelson:
>>>> On 02/10/2015 03:11 PM, Stefan Priebe wrote:
>>>>>
>>>>> mhm i installed librbd1-dbg and librados2-dbg - but the output still
>>>>> looks useless to me. Should i upload it somewhere?
>>>>
>>>> Meh, if it's all just symbols it's probably not that helpful.
>>>>
>>>> I've summarized your results here:
>>>>
>>>> 1 concurrent 4k write (libaio, direct=1, iodepth=1)
>>>>
>>>> IOPS Latency
>>>> wb on wb off wb on wb off
>>>> dumpling 10870 536 ~100us ~2ms
>>>> firefly 10350 525 ~100us ~2ms
>>>>
>>>> So in single op tests dumpling and firefly are far closer. Now let's
>>>> see each of these cases with iodepth=32 (still 1 thread for now).
>>>
>>>
>>> dumpling:
>>>
>>> file1: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=32
>>> 2.0.8
>>> Starting 1 thread
>>> Jobs: 1 (f=1): [w] [100.0% done] [0K/72812K /s] [0 /18.3K iops] [eta
>>> 00m:00s]
>>> file1: (groupid=0, jobs=1): err= 0: pid=3011
>>> write: io=2060.6MB, bw=70329KB/s, iops=17582 , runt= 30001msec
>>> slat (usec): min=1 , max=3517 , avg= 3.42, stdev= 7.30
>>> clat (usec): min=93 , max=7475 , avg=1815.72, stdev=233.43
>>> lat (usec): min=219 , max=7477 , avg=1819.27, stdev=233.52
>>> clat percentiles (usec):
>>> | 1.00th=[ 1480], 5.00th=[ 1576], 10.00th=[ 1608], 20.00th=[
>>> 1672],
>>> | 30.00th=[ 1704], 40.00th=[ 1752], 50.00th=[ 1800], 60.00th=[
>>> 1832],
>>> | 70.00th=[ 1896], 80.00th=[ 1960], 90.00th=[ 2064], 95.00th=[
>>> 2128],
>>> | 99.00th=[ 2352], 99.50th=[ 2448], 99.90th=[ 4704], 99.95th=[
>>> 5344],
>>> | 99.99th=[ 7072]
>>> bw (KB/s) : min=59696, max=77840, per=100.00%, avg=70351.27,
>>> stdev=4783.25
>>> lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.53%
>>> lat (msec) : 2=85.02%, 4=14.31%, 10=0.13%
>>> cpu : usr=1.96%, sys=6.71%, ctx=22791, majf=0, minf=133
>>> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%,
>>> >=64=0.0%
>>> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>>> >=64=0.0%
>>> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
>>> >=64=0.0%
>>> issued : total=r=0/w=527487/d=0, short=r=0/w=0/d=0
>>>
>>> Run status group 0 (all jobs):
>>> WRITE: io=2060.6MB, aggrb=70329KB/s, minb=70329KB/s, maxb=70329KB/s,
>>> mint=30001msec, maxt=30001msec
>>>
>>> Disk stats (read/write):
>>> sdb: ios=166/526079, merge=0/0, ticks=24/890120, in_queue=890064,
>>> util=98.73%
>>>
>>> firefly:
>>>
>>> file1: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=32
>>> 2.0.8
>>> Starting 1 thread
>>> Jobs: 1 (f=1): [w] [100.0% done] [0K/69096K /s] [0 /17.3K iops] [eta
>>> 00m:00s]
>>> file1: (groupid=0, jobs=1): err= 0: pid=2982
>>> write: io=1784.9MB, bw=60918KB/s, iops=15229 , runt= 30002msec
>>> slat (usec): min=1 , max=1389 , avg= 3.43, stdev= 5.32
>>> clat (usec): min=117 , max=8235 , avg=2096.88, stdev=396.30
>>> lat (usec): min=540 , max=8258 , avg=2100.43, stdev=396.61
>>> clat percentiles (usec):
>>> | 1.00th=[ 1608], 5.00th=[ 1720], 10.00th=[ 1768], 20.00th=[
>>> 1832],
>>> | 30.00th=[ 1896], 40.00th=[ 1944], 50.00th=[ 2008], 60.00th=[
>>> 2064],
>>> | 70.00th=[ 2160], 80.00th=[ 2256], 90.00th=[ 2512], 95.00th=[
>>> 2896],
>>> | 99.00th=[ 3600], 99.50th=[ 3792], 99.90th=[ 5088], 99.95th=[
>>> 6304],
>>> | 99.99th=[ 6752]
>>> bw (KB/s) : min=36717, max=73712, per=99.94%, avg=60879.92,
>>> stdev=8302.27
>>> lat (usec) : 250=0.01%, 750=0.01%
>>> lat (msec) : 2=48.56%, 4=51.18%, 10=0.26%
>>> cpu : usr=2.03%, sys=5.48%, ctx=20440, majf=0, minf=133
>>> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%,
>>> >=64=0.0%
>>> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>>> >=64=0.0%
>>> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
>>> >=64=0.0%
>>> issued : total=r=0/w=456918/d=0, short=r=0/w=0/d=0
>>>
>>> Run status group 0 (all jobs):
>>> WRITE: io=1784.9MB, aggrb=60918KB/s, minb=60918KB/s, maxb=60918KB/s,
>>> mint=30002msec, maxt=30002msec
>>>
>>> Disk stats (read/write):
>>> sdb: ios=166/455574, merge=0/0, ticks=12/897748, in_queue=897696,
>>> util=98.96%
>>>
>>
>> Ok, so it looks like as you increase concurrency the effect increases
>> (ie contention?). Does the same thing happen without cache enabled?
>
> here again without rbd cache:
>
> dumpling:
> file1: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=32
> 2.0.8
> Starting 1 thread
> Jobs: 1 (f=1): [w] [100.0% done] [0K/83488K /s] [0 /20.9K iops] [eta
> 00m:00s]
> file1: (groupid=0, jobs=1): err= 0: pid=3000
> write: io=2449.2MB, bw=83583KB/s, iops=20895 , runt= 30005msec
> slat (usec): min=1 , max=975 , avg= 4.50, stdev= 5.25
> clat (usec): min=364 , max=80566 , avg=1525.87, stdev=1194.57
> lat (usec): min=519 , max=80568 , avg=1530.51, stdev=1194.44
> clat percentiles (usec):
> | 1.00th=[ 660], 5.00th=[ 780], 10.00th=[ 876], 20.00th=[ 1032],
> | 30.00th=[ 1144], 40.00th=[ 1240], 50.00th=[ 1304], 60.00th=[ 1384],
> | 70.00th=[ 1480], 80.00th=[ 1640], 90.00th=[ 2096], 95.00th=[ 2960],
> | 99.00th=[ 6816], 99.50th=[ 7840], 99.90th=[11712], 99.95th=[13888],
> | 99.99th=[18816]
> bw (KB/s) : min=47184, max=95432, per=100.00%, avg=83639.19,
> stdev=7973.92
> lat (usec) : 500=0.01%, 750=3.82%, 1000=14.40%
> lat (msec) : 2=70.57%, 4=7.91%, 10=3.11%, 20=0.17%, 50=0.01%
> lat (msec) : 100=0.01%
> cpu : usr=3.12%, sys=11.49%, ctx=74951, majf=0, minf=133
> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%,
> >=64=0.0%
> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
> >=64=0.0%
> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
> >=64=0.0%
> issued : total=r=0/w=626979/d=0, short=r=0/w=0/d=0
>
> Run status group 0 (all jobs):
> WRITE: io=2449.2MB, aggrb=83583KB/s, minb=83583KB/s, maxb=83583KB/s,
> mint=30005msec, maxt=30005msec
>
> Disk stats (read/write):
> sdb: ios=168/625292, merge=0/0, ticks=144/916096, in_queue=916128,
> util=99.93%
>
>
> firefly:
>
> fio --filename=/dev/sdb --direct=1 --rw=randwrite --bs=4k --numjobs=1
> --thread --iodepth=32 --ioengine=libaio --runtime=30 --group_reporting
> --name=file1
> file1: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=32
> 2.0.8
> Starting 1 thread
> Jobs: 1 (f=1): [w] [100.0% done] [0K/90044K /s] [0 /22.6K iops] [eta
> 00m:00s]
> file1: (groupid=0, jobs=1): err= 0: pid=2970
> write: io=2372.9MB, bw=80976KB/s, iops=20244 , runt= 30006msec
> slat (usec): min=1 , max=4047 , avg= 4.36, stdev= 7.17
> clat (usec): min=197 , max=76656 , avg=1575.29, stdev=1165.74
> lat (usec): min=523 , max=76660 , avg=1579.79, stdev=1165.59
> clat percentiles (usec):
> | 1.00th=[ 676], 5.00th=[ 804], 10.00th=[ 916], 20.00th=[ 1096],
> | 30.00th=[ 1224], 40.00th=[ 1304], 50.00th=[ 1384], 60.00th=[ 1448],
> | 70.00th=[ 1544], 80.00th=[ 1704], 90.00th=[ 2128], 95.00th=[ 2736],
> | 99.00th=[ 6752], 99.50th=[ 7904], 99.90th=[12096], 99.95th=[14656],
> | 99.99th=[18560]
> bw (KB/s) : min=47800, max=91952, per=99.91%, avg=80900.88,
> stdev=7234.98
> lat (usec) : 250=0.01%, 500=0.01%, 750=2.95%, 1000=11.38%
> lat (msec) : 2=73.81%, 4=8.81%, 10=2.85%, 20=0.19%, 50=0.01%
> lat (msec) : 100=0.01%
> cpu : usr=2.99%, sys=10.60%, ctx=66549, majf=0, minf=133
> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%,
> >=64=0.0%
> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
> >=64=0.0%
> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
> >=64=0.0%
> issued : total=r=0/w=607445/d=0, short=r=0/w=0/d=0
>
> Run status group 0 (all jobs):
> WRITE: io=2372.9MB, aggrb=80976KB/s, minb=80976KB/s, maxb=80976KB/s,
> mint=30006msec, maxt=30006msec
>
> Disk stats (read/write):
> sdb: ios=170/605440, merge=0/0, ticks=156/916492, in_queue=916560,
> util=99.93%
>
> Stefan
>
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: speed decrease since firefly,giant,hammer the 2nd try
2015-02-16 9:11 ` Alexandre DERUMIER
@ 2015-02-16 14:50 ` Stefan Priebe - Profihost AG
2015-02-16 15:35 ` Mark Nelson
` (2 more replies)
0 siblings, 3 replies; 32+ messages in thread
From: Stefan Priebe - Profihost AG @ 2015-02-16 14:50 UTC (permalink / raw)
To: Alexandre DERUMIER; +Cc: Mark Nelson, ceph-devel
Hi Mark,
Hi Alexandre,
Am 16.02.2015 um 10:11 schrieb Alexandre DERUMIER:
> Hi Stefan,
>
> I could be interesting to see if you have the same speed decrease with fio-librbd on the host,
> without the qemu layer.
>
> the perf reports don't seem to be too much different.
> do you have the same cpu usage ? (check qemu process usage)
the idea to use fio-librbd was very good.
I cannot reproduce the behaviour using fio-rbd. I can just reproduce it
with qemu.
Very strange. So please ignore me for the moment. I'll try to dig deeper
into it.
Greets,
Stefan
> ----- Mail original -----
> De: "Stefan Priebe" <s.priebe@profihost.ag>
> À: "Mark Nelson" <mnelson@redhat.com>, "ceph-devel" <ceph-devel@vger.kernel.org>
> Envoyé: Dimanche 15 Février 2015 19:40:45
> Objet: Re: speed decrease since firefly,giant,hammer the 2nd try
>
> Hi Mark,
>
> what's next?
>
> I've this test cluster only for 2 more days.
>
> Here some perf Details:
>
> dumpling:
> 12,65% libc-2.13.so [.] 0x79000
> 2,86% libc-2.13.so [.] malloc
> 2,80% kvm [.] 0xb59c5
> 2,59% libc-2.13.so [.] free
> 2,35% [kernel] [k] __schedule
> 2,16% [kernel] [k] _raw_spin_lock
> 1,92% [kernel] [k] __switch_to
> 1,58% [kernel] [k] lapic_next_deadline
> 1,09% [kernel] [k] update_sd_lb_stats
> 1,08% [kernel] [k] _raw_spin_lock_irqsave
> 0,91% librados.so.2.0.0 [.] ceph_crc32c_le_intel
> 0,91% libpthread-2.13.so [.] pthread_mutex_trylock
> 0,87% [kernel] [k] resched_task
> 0,72% [kernel] [k] cpu_startup_entry
> 0,71% librados.so.2.0.0 [.] crush_hash32_3
> 0,66% [kernel] [k] leave_mm
> 0,65% librados.so.2.0.0 [.] Mutex::Lock(bool)
> 0,64% [kernel] [k] idle_cpu
> 0,62% libpthread-2.13.so [.] __pthread_mutex_unlock_usercnt
> 0,59% [kernel] [k] try_to_wake_up
> 0,56% [kernel] [k] wake_futex
> 0,50% librados.so.2.0.0 [.] ceph::buffer::ptr::release()
>
> firefly:
> 12,56% libc-2.13.so [.] 0x7905d
> 2,82% libc-2.13.so [.] malloc
> 2,64% libc-2.13.so [.] free
> 2,61% kvm [.] 0x34322f
> 2,33% [kernel] [k] __schedule
> 2,14% [kernel] [k] _raw_spin_lock
> 1,83% [kernel] [k] __switch_to
> 1,62% [kernel] [k] lapic_next_deadline
> 1,17% [kernel] [k] _raw_spin_lock_irqsave
> 1,09% [kernel] [k] update_sd_lb_stats
> 1,08% libpthread-2.13.so [.] pthread_mutex_trylock
> 0,85% libpthread-2.13.so [.] __pthread_mutex_unlock_usercnt
> 0,77% [kernel] [k] resched_task
> 0,74% librbd.so.1.0.0 [.] 0x71b73
> 0,72% librados.so.2.0.0 [.] Mutex::Lock(bool)
> 0,68% librados.so.2.0.0 [.] crush_hash32_3
> 0,67% [kernel] [k] idle_cpu
> 0,65% [kernel] [k] leave_mm
> 0,65% [kernel] [k] cpu_startup_entry
> 0,59% [kernel] [k] try_to_wake_up
> 0,51% librados.so.2.0.0 [.] ceph::buffer::ptr::release()
> 0,51% [kernel] [k] wake_futex
>
> Stefan
>
> Am 11.02.2015 um 06:42 schrieb Stefan Priebe:
>>
>> Am 11.02.2015 um 05:45 schrieb Mark Nelson:
>>> On 02/10/2015 04:18 PM, Stefan Priebe wrote:
>>>>
>>>> Am 10.02.2015 um 22:38 schrieb Mark Nelson:
>>>>> On 02/10/2015 03:11 PM, Stefan Priebe wrote:
>>>>>>
>>>>>> mhm i installed librbd1-dbg and librados2-dbg - but the output still
>>>>>> looks useless to me. Should i upload it somewhere?
>>>>>
>>>>> Meh, if it's all just symbols it's probably not that helpful.
>>>>>
>>>>> I've summarized your results here:
>>>>>
>>>>> 1 concurrent 4k write (libaio, direct=1, iodepth=1)
>>>>>
>>>>> IOPS Latency
>>>>> wb on wb off wb on wb off
>>>>> dumpling 10870 536 ~100us ~2ms
>>>>> firefly 10350 525 ~100us ~2ms
>>>>>
>>>>> So in single op tests dumpling and firefly are far closer. Now let's
>>>>> see each of these cases with iodepth=32 (still 1 thread for now).
>>>>
>>>>
>>>> dumpling:
>>>>
>>>> file1: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=32
>>>> 2.0.8
>>>> Starting 1 thread
>>>> Jobs: 1 (f=1): [w] [100.0% done] [0K/72812K /s] [0 /18.3K iops] [eta
>>>> 00m:00s]
>>>> file1: (groupid=0, jobs=1): err= 0: pid=3011
>>>> write: io=2060.6MB, bw=70329KB/s, iops=17582 , runt= 30001msec
>>>> slat (usec): min=1 , max=3517 , avg= 3.42, stdev= 7.30
>>>> clat (usec): min=93 , max=7475 , avg=1815.72, stdev=233.43
>>>> lat (usec): min=219 , max=7477 , avg=1819.27, stdev=233.52
>>>> clat percentiles (usec):
>>>> | 1.00th=[ 1480], 5.00th=[ 1576], 10.00th=[ 1608], 20.00th=[
>>>> 1672],
>>>> | 30.00th=[ 1704], 40.00th=[ 1752], 50.00th=[ 1800], 60.00th=[
>>>> 1832],
>>>> | 70.00th=[ 1896], 80.00th=[ 1960], 90.00th=[ 2064], 95.00th=[
>>>> 2128],
>>>> | 99.00th=[ 2352], 99.50th=[ 2448], 99.90th=[ 4704], 99.95th=[
>>>> 5344],
>>>> | 99.99th=[ 7072]
>>>> bw (KB/s) : min=59696, max=77840, per=100.00%, avg=70351.27,
>>>> stdev=4783.25
>>>> lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.53%
>>>> lat (msec) : 2=85.02%, 4=14.31%, 10=0.13%
>>>> cpu : usr=1.96%, sys=6.71%, ctx=22791, majf=0, minf=133
>>>> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%,
>>>>> =64=0.0%
>>>> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>>>>> =64=0.0%
>>>> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
>>>>> =64=0.0%
>>>> issued : total=r=0/w=527487/d=0, short=r=0/w=0/d=0
>>>>
>>>> Run status group 0 (all jobs):
>>>> WRITE: io=2060.6MB, aggrb=70329KB/s, minb=70329KB/s, maxb=70329KB/s,
>>>> mint=30001msec, maxt=30001msec
>>>>
>>>> Disk stats (read/write):
>>>> sdb: ios=166/526079, merge=0/0, ticks=24/890120, in_queue=890064,
>>>> util=98.73%
>>>>
>>>> firefly:
>>>>
>>>> file1: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=32
>>>> 2.0.8
>>>> Starting 1 thread
>>>> Jobs: 1 (f=1): [w] [100.0% done] [0K/69096K /s] [0 /17.3K iops] [eta
>>>> 00m:00s]
>>>> file1: (groupid=0, jobs=1): err= 0: pid=2982
>>>> write: io=1784.9MB, bw=60918KB/s, iops=15229 , runt= 30002msec
>>>> slat (usec): min=1 , max=1389 , avg= 3.43, stdev= 5.32
>>>> clat (usec): min=117 , max=8235 , avg=2096.88, stdev=396.30
>>>> lat (usec): min=540 , max=8258 , avg=2100.43, stdev=396.61
>>>> clat percentiles (usec):
>>>> | 1.00th=[ 1608], 5.00th=[ 1720], 10.00th=[ 1768], 20.00th=[
>>>> 1832],
>>>> | 30.00th=[ 1896], 40.00th=[ 1944], 50.00th=[ 2008], 60.00th=[
>>>> 2064],
>>>> | 70.00th=[ 2160], 80.00th=[ 2256], 90.00th=[ 2512], 95.00th=[
>>>> 2896],
>>>> | 99.00th=[ 3600], 99.50th=[ 3792], 99.90th=[ 5088], 99.95th=[
>>>> 6304],
>>>> | 99.99th=[ 6752]
>>>> bw (KB/s) : min=36717, max=73712, per=99.94%, avg=60879.92,
>>>> stdev=8302.27
>>>> lat (usec) : 250=0.01%, 750=0.01%
>>>> lat (msec) : 2=48.56%, 4=51.18%, 10=0.26%
>>>> cpu : usr=2.03%, sys=5.48%, ctx=20440, majf=0, minf=133
>>>> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%,
>>>>> =64=0.0%
>>>> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>>>>> =64=0.0%
>>>> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
>>>>> =64=0.0%
>>>> issued : total=r=0/w=456918/d=0, short=r=0/w=0/d=0
>>>>
>>>> Run status group 0 (all jobs):
>>>> WRITE: io=1784.9MB, aggrb=60918KB/s, minb=60918KB/s, maxb=60918KB/s,
>>>> mint=30002msec, maxt=30002msec
>>>>
>>>> Disk stats (read/write):
>>>> sdb: ios=166/455574, merge=0/0, ticks=12/897748, in_queue=897696,
>>>> util=98.96%
>>>>
>>>
>>> Ok, so it looks like as you increase concurrency the effect increases
>>> (ie contention?). Does the same thing happen without cache enabled?
>>
>> here again without rbd cache:
>>
>> dumpling:
>> file1: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=32
>> 2.0.8
>> Starting 1 thread
>> Jobs: 1 (f=1): [w] [100.0% done] [0K/83488K /s] [0 /20.9K iops] [eta
>> 00m:00s]
>> file1: (groupid=0, jobs=1): err= 0: pid=3000
>> write: io=2449.2MB, bw=83583KB/s, iops=20895 , runt= 30005msec
>> slat (usec): min=1 , max=975 , avg= 4.50, stdev= 5.25
>> clat (usec): min=364 , max=80566 , avg=1525.87, stdev=1194.57
>> lat (usec): min=519 , max=80568 , avg=1530.51, stdev=1194.44
>> clat percentiles (usec):
>> | 1.00th=[ 660], 5.00th=[ 780], 10.00th=[ 876], 20.00th=[ 1032],
>> | 30.00th=[ 1144], 40.00th=[ 1240], 50.00th=[ 1304], 60.00th=[ 1384],
>> | 70.00th=[ 1480], 80.00th=[ 1640], 90.00th=[ 2096], 95.00th=[ 2960],
>> | 99.00th=[ 6816], 99.50th=[ 7840], 99.90th=[11712], 99.95th=[13888],
>> | 99.99th=[18816]
>> bw (KB/s) : min=47184, max=95432, per=100.00%, avg=83639.19,
>> stdev=7973.92
>> lat (usec) : 500=0.01%, 750=3.82%, 1000=14.40%
>> lat (msec) : 2=70.57%, 4=7.91%, 10=3.11%, 20=0.17%, 50=0.01%
>> lat (msec) : 100=0.01%
>> cpu : usr=3.12%, sys=11.49%, ctx=74951, majf=0, minf=133
>> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%,
>>> =64=0.0%
>> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>>> =64=0.0%
>> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
>>> =64=0.0%
>> issued : total=r=0/w=626979/d=0, short=r=0/w=0/d=0
>>
>> Run status group 0 (all jobs):
>> WRITE: io=2449.2MB, aggrb=83583KB/s, minb=83583KB/s, maxb=83583KB/s,
>> mint=30005msec, maxt=30005msec
>>
>> Disk stats (read/write):
>> sdb: ios=168/625292, merge=0/0, ticks=144/916096, in_queue=916128,
>> util=99.93%
>>
>>
>> firefly:
>>
>> fio --filename=/dev/sdb --direct=1 --rw=randwrite --bs=4k --numjobs=1
>> --thread --iodepth=32 --ioengine=libaio --runtime=30 --group_reporting
>> --name=file1
>> file1: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=32
>> 2.0.8
>> Starting 1 thread
>> Jobs: 1 (f=1): [w] [100.0% done] [0K/90044K /s] [0 /22.6K iops] [eta
>> 00m:00s]
>> file1: (groupid=0, jobs=1): err= 0: pid=2970
>> write: io=2372.9MB, bw=80976KB/s, iops=20244 , runt= 30006msec
>> slat (usec): min=1 , max=4047 , avg= 4.36, stdev= 7.17
>> clat (usec): min=197 , max=76656 , avg=1575.29, stdev=1165.74
>> lat (usec): min=523 , max=76660 , avg=1579.79, stdev=1165.59
>> clat percentiles (usec):
>> | 1.00th=[ 676], 5.00th=[ 804], 10.00th=[ 916], 20.00th=[ 1096],
>> | 30.00th=[ 1224], 40.00th=[ 1304], 50.00th=[ 1384], 60.00th=[ 1448],
>> | 70.00th=[ 1544], 80.00th=[ 1704], 90.00th=[ 2128], 95.00th=[ 2736],
>> | 99.00th=[ 6752], 99.50th=[ 7904], 99.90th=[12096], 99.95th=[14656],
>> | 99.99th=[18560]
>> bw (KB/s) : min=47800, max=91952, per=99.91%, avg=80900.88,
>> stdev=7234.98
>> lat (usec) : 250=0.01%, 500=0.01%, 750=2.95%, 1000=11.38%
>> lat (msec) : 2=73.81%, 4=8.81%, 10=2.85%, 20=0.19%, 50=0.01%
>> lat (msec) : 100=0.01%
>> cpu : usr=2.99%, sys=10.60%, ctx=66549, majf=0, minf=133
>> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%,
>>> =64=0.0%
>> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>>> =64=0.0%
>> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
>>> =64=0.0%
>> issued : total=r=0/w=607445/d=0, short=r=0/w=0/d=0
>>
>> Run status group 0 (all jobs):
>> WRITE: io=2372.9MB, aggrb=80976KB/s, minb=80976KB/s, maxb=80976KB/s,
>> mint=30006msec, maxt=30006msec
>>
>> Disk stats (read/write):
>> sdb: ios=170/605440, merge=0/0, ticks=156/916492, in_queue=916560,
>> util=99.93%
>>
>> Stefan
>>
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: speed decrease since firefly,giant,hammer the 2nd try
2015-02-16 14:50 ` Stefan Priebe - Profihost AG
@ 2015-02-16 15:35 ` Mark Nelson
2015-02-16 15:36 ` Alexandre DERUMIER
[not found] ` <1760409866.980984.1424105112443.JavaMail.zimbra@oxygem.tv>
2 siblings, 0 replies; 32+ messages in thread
From: Mark Nelson @ 2015-02-16 15:35 UTC (permalink / raw)
To: Stefan Priebe - Profihost AG, Alexandre DERUMIER; +Cc: ceph-devel
On 02/16/2015 08:50 AM, Stefan Priebe - Profihost AG wrote:
> Hi Mark,
> Hi Alexandre,
>
> Am 16.02.2015 um 10:11 schrieb Alexandre DERUMIER:
>> Hi Stefan,
>>
>> I could be interesting to see if you have the same speed decrease with fio-librbd on the host,
>> without the qemu layer.
>>
>> the perf reports don't seem to be too much different.
>> do you have the same cpu usage ? (check qemu process usage)
>
> the idea to use fio-librbd was very good.
>
> I cannot reproduce the behaviour using fio-rbd. I can just reproduce it
> with qemu.
>
> Very strange. So please ignore me for the moment. I'll try to dig deeper
> into it.
Interesting development! FWIW, I've been doing some testing with
librados on dumpling, firefly, and hammer. Not RBD numbers, but adds a
bit of info to the greater puzzle.
Hoping I should have something written up this week.
>
> Greets,
> Stefan
>
>> ----- Mail original -----
>> De: "Stefan Priebe" <s.priebe@profihost.ag>
>> À: "Mark Nelson" <mnelson@redhat.com>, "ceph-devel" <ceph-devel@vger.kernel.org>
>> Envoyé: Dimanche 15 Février 2015 19:40:45
>> Objet: Re: speed decrease since firefly,giant,hammer the 2nd try
>>
>> Hi Mark,
>>
>> what's next?
>>
>> I've this test cluster only for 2 more days.
>>
>> Here some perf Details:
>>
>> dumpling:
>> 12,65% libc-2.13.so [.] 0x79000
>> 2,86% libc-2.13.so [.] malloc
>> 2,80% kvm [.] 0xb59c5
>> 2,59% libc-2.13.so [.] free
>> 2,35% [kernel] [k] __schedule
>> 2,16% [kernel] [k] _raw_spin_lock
>> 1,92% [kernel] [k] __switch_to
>> 1,58% [kernel] [k] lapic_next_deadline
>> 1,09% [kernel] [k] update_sd_lb_stats
>> 1,08% [kernel] [k] _raw_spin_lock_irqsave
>> 0,91% librados.so.2.0.0 [.] ceph_crc32c_le_intel
>> 0,91% libpthread-2.13.so [.] pthread_mutex_trylock
>> 0,87% [kernel] [k] resched_task
>> 0,72% [kernel] [k] cpu_startup_entry
>> 0,71% librados.so.2.0.0 [.] crush_hash32_3
>> 0,66% [kernel] [k] leave_mm
>> 0,65% librados.so.2.0.0 [.] Mutex::Lock(bool)
>> 0,64% [kernel] [k] idle_cpu
>> 0,62% libpthread-2.13.so [.] __pthread_mutex_unlock_usercnt
>> 0,59% [kernel] [k] try_to_wake_up
>> 0,56% [kernel] [k] wake_futex
>> 0,50% librados.so.2.0.0 [.] ceph::buffer::ptr::release()
>>
>> firefly:
>> 12,56% libc-2.13.so [.] 0x7905d
>> 2,82% libc-2.13.so [.] malloc
>> 2,64% libc-2.13.so [.] free
>> 2,61% kvm [.] 0x34322f
>> 2,33% [kernel] [k] __schedule
>> 2,14% [kernel] [k] _raw_spin_lock
>> 1,83% [kernel] [k] __switch_to
>> 1,62% [kernel] [k] lapic_next_deadline
>> 1,17% [kernel] [k] _raw_spin_lock_irqsave
>> 1,09% [kernel] [k] update_sd_lb_stats
>> 1,08% libpthread-2.13.so [.] pthread_mutex_trylock
>> 0,85% libpthread-2.13.so [.] __pthread_mutex_unlock_usercnt
>> 0,77% [kernel] [k] resched_task
>> 0,74% librbd.so.1.0.0 [.] 0x71b73
>> 0,72% librados.so.2.0.0 [.] Mutex::Lock(bool)
>> 0,68% librados.so.2.0.0 [.] crush_hash32_3
>> 0,67% [kernel] [k] idle_cpu
>> 0,65% [kernel] [k] leave_mm
>> 0,65% [kernel] [k] cpu_startup_entry
>> 0,59% [kernel] [k] try_to_wake_up
>> 0,51% librados.so.2.0.0 [.] ceph::buffer::ptr::release()
>> 0,51% [kernel] [k] wake_futex
>>
>> Stefan
>>
>> Am 11.02.2015 um 06:42 schrieb Stefan Priebe:
>>>
>>> Am 11.02.2015 um 05:45 schrieb Mark Nelson:
>>>> On 02/10/2015 04:18 PM, Stefan Priebe wrote:
>>>>>
>>>>> Am 10.02.2015 um 22:38 schrieb Mark Nelson:
>>>>>> On 02/10/2015 03:11 PM, Stefan Priebe wrote:
>>>>>>>
>>>>>>> mhm i installed librbd1-dbg and librados2-dbg - but the output still
>>>>>>> looks useless to me. Should i upload it somewhere?
>>>>>>
>>>>>> Meh, if it's all just symbols it's probably not that helpful.
>>>>>>
>>>>>> I've summarized your results here:
>>>>>>
>>>>>> 1 concurrent 4k write (libaio, direct=1, iodepth=1)
>>>>>>
>>>>>> IOPS Latency
>>>>>> wb on wb off wb on wb off
>>>>>> dumpling 10870 536 ~100us ~2ms
>>>>>> firefly 10350 525 ~100us ~2ms
>>>>>>
>>>>>> So in single op tests dumpling and firefly are far closer. Now let's
>>>>>> see each of these cases with iodepth=32 (still 1 thread for now).
>>>>>
>>>>>
>>>>> dumpling:
>>>>>
>>>>> file1: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=32
>>>>> 2.0.8
>>>>> Starting 1 thread
>>>>> Jobs: 1 (f=1): [w] [100.0% done] [0K/72812K /s] [0 /18.3K iops] [eta
>>>>> 00m:00s]
>>>>> file1: (groupid=0, jobs=1): err= 0: pid=3011
>>>>> write: io=2060.6MB, bw=70329KB/s, iops=17582 , runt= 30001msec
>>>>> slat (usec): min=1 , max=3517 , avg= 3.42, stdev= 7.30
>>>>> clat (usec): min=93 , max=7475 , avg=1815.72, stdev=233.43
>>>>> lat (usec): min=219 , max=7477 , avg=1819.27, stdev=233.52
>>>>> clat percentiles (usec):
>>>>> | 1.00th=[ 1480], 5.00th=[ 1576], 10.00th=[ 1608], 20.00th=[
>>>>> 1672],
>>>>> | 30.00th=[ 1704], 40.00th=[ 1752], 50.00th=[ 1800], 60.00th=[
>>>>> 1832],
>>>>> | 70.00th=[ 1896], 80.00th=[ 1960], 90.00th=[ 2064], 95.00th=[
>>>>> 2128],
>>>>> | 99.00th=[ 2352], 99.50th=[ 2448], 99.90th=[ 4704], 99.95th=[
>>>>> 5344],
>>>>> | 99.99th=[ 7072]
>>>>> bw (KB/s) : min=59696, max=77840, per=100.00%, avg=70351.27,
>>>>> stdev=4783.25
>>>>> lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.53%
>>>>> lat (msec) : 2=85.02%, 4=14.31%, 10=0.13%
>>>>> cpu : usr=1.96%, sys=6.71%, ctx=22791, majf=0, minf=133
>>>>> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%,
>>>>>> =64=0.0%
>>>>> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>>>>>> =64=0.0%
>>>>> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
>>>>>> =64=0.0%
>>>>> issued : total=r=0/w=527487/d=0, short=r=0/w=0/d=0
>>>>>
>>>>> Run status group 0 (all jobs):
>>>>> WRITE: io=2060.6MB, aggrb=70329KB/s, minb=70329KB/s, maxb=70329KB/s,
>>>>> mint=30001msec, maxt=30001msec
>>>>>
>>>>> Disk stats (read/write):
>>>>> sdb: ios=166/526079, merge=0/0, ticks=24/890120, in_queue=890064,
>>>>> util=98.73%
>>>>>
>>>>> firefly:
>>>>>
>>>>> file1: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=32
>>>>> 2.0.8
>>>>> Starting 1 thread
>>>>> Jobs: 1 (f=1): [w] [100.0% done] [0K/69096K /s] [0 /17.3K iops] [eta
>>>>> 00m:00s]
>>>>> file1: (groupid=0, jobs=1): err= 0: pid=2982
>>>>> write: io=1784.9MB, bw=60918KB/s, iops=15229 , runt= 30002msec
>>>>> slat (usec): min=1 , max=1389 , avg= 3.43, stdev= 5.32
>>>>> clat (usec): min=117 , max=8235 , avg=2096.88, stdev=396.30
>>>>> lat (usec): min=540 , max=8258 , avg=2100.43, stdev=396.61
>>>>> clat percentiles (usec):
>>>>> | 1.00th=[ 1608], 5.00th=[ 1720], 10.00th=[ 1768], 20.00th=[
>>>>> 1832],
>>>>> | 30.00th=[ 1896], 40.00th=[ 1944], 50.00th=[ 2008], 60.00th=[
>>>>> 2064],
>>>>> | 70.00th=[ 2160], 80.00th=[ 2256], 90.00th=[ 2512], 95.00th=[
>>>>> 2896],
>>>>> | 99.00th=[ 3600], 99.50th=[ 3792], 99.90th=[ 5088], 99.95th=[
>>>>> 6304],
>>>>> | 99.99th=[ 6752]
>>>>> bw (KB/s) : min=36717, max=73712, per=99.94%, avg=60879.92,
>>>>> stdev=8302.27
>>>>> lat (usec) : 250=0.01%, 750=0.01%
>>>>> lat (msec) : 2=48.56%, 4=51.18%, 10=0.26%
>>>>> cpu : usr=2.03%, sys=5.48%, ctx=20440, majf=0, minf=133
>>>>> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%,
>>>>>> =64=0.0%
>>>>> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>>>>>> =64=0.0%
>>>>> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
>>>>>> =64=0.0%
>>>>> issued : total=r=0/w=456918/d=0, short=r=0/w=0/d=0
>>>>>
>>>>> Run status group 0 (all jobs):
>>>>> WRITE: io=1784.9MB, aggrb=60918KB/s, minb=60918KB/s, maxb=60918KB/s,
>>>>> mint=30002msec, maxt=30002msec
>>>>>
>>>>> Disk stats (read/write):
>>>>> sdb: ios=166/455574, merge=0/0, ticks=12/897748, in_queue=897696,
>>>>> util=98.96%
>>>>>
>>>>
>>>> Ok, so it looks like as you increase concurrency the effect increases
>>>> (ie contention?). Does the same thing happen without cache enabled?
>>>
>>> here again without rbd cache:
>>>
>>> dumpling:
>>> file1: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=32
>>> 2.0.8
>>> Starting 1 thread
>>> Jobs: 1 (f=1): [w] [100.0% done] [0K/83488K /s] [0 /20.9K iops] [eta
>>> 00m:00s]
>>> file1: (groupid=0, jobs=1): err= 0: pid=3000
>>> write: io=2449.2MB, bw=83583KB/s, iops=20895 , runt= 30005msec
>>> slat (usec): min=1 , max=975 , avg= 4.50, stdev= 5.25
>>> clat (usec): min=364 , max=80566 , avg=1525.87, stdev=1194.57
>>> lat (usec): min=519 , max=80568 , avg=1530.51, stdev=1194.44
>>> clat percentiles (usec):
>>> | 1.00th=[ 660], 5.00th=[ 780], 10.00th=[ 876], 20.00th=[ 1032],
>>> | 30.00th=[ 1144], 40.00th=[ 1240], 50.00th=[ 1304], 60.00th=[ 1384],
>>> | 70.00th=[ 1480], 80.00th=[ 1640], 90.00th=[ 2096], 95.00th=[ 2960],
>>> | 99.00th=[ 6816], 99.50th=[ 7840], 99.90th=[11712], 99.95th=[13888],
>>> | 99.99th=[18816]
>>> bw (KB/s) : min=47184, max=95432, per=100.00%, avg=83639.19,
>>> stdev=7973.92
>>> lat (usec) : 500=0.01%, 750=3.82%, 1000=14.40%
>>> lat (msec) : 2=70.57%, 4=7.91%, 10=3.11%, 20=0.17%, 50=0.01%
>>> lat (msec) : 100=0.01%
>>> cpu : usr=3.12%, sys=11.49%, ctx=74951, majf=0, minf=133
>>> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%,
>>>> =64=0.0%
>>> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>>>> =64=0.0%
>>> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
>>>> =64=0.0%
>>> issued : total=r=0/w=626979/d=0, short=r=0/w=0/d=0
>>>
>>> Run status group 0 (all jobs):
>>> WRITE: io=2449.2MB, aggrb=83583KB/s, minb=83583KB/s, maxb=83583KB/s,
>>> mint=30005msec, maxt=30005msec
>>>
>>> Disk stats (read/write):
>>> sdb: ios=168/625292, merge=0/0, ticks=144/916096, in_queue=916128,
>>> util=99.93%
>>>
>>>
>>> firefly:
>>>
>>> fio --filename=/dev/sdb --direct=1 --rw=randwrite --bs=4k --numjobs=1
>>> --thread --iodepth=32 --ioengine=libaio --runtime=30 --group_reporting
>>> --name=file1
>>> file1: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=32
>>> 2.0.8
>>> Starting 1 thread
>>> Jobs: 1 (f=1): [w] [100.0% done] [0K/90044K /s] [0 /22.6K iops] [eta
>>> 00m:00s]
>>> file1: (groupid=0, jobs=1): err= 0: pid=2970
>>> write: io=2372.9MB, bw=80976KB/s, iops=20244 , runt= 30006msec
>>> slat (usec): min=1 , max=4047 , avg= 4.36, stdev= 7.17
>>> clat (usec): min=197 , max=76656 , avg=1575.29, stdev=1165.74
>>> lat (usec): min=523 , max=76660 , avg=1579.79, stdev=1165.59
>>> clat percentiles (usec):
>>> | 1.00th=[ 676], 5.00th=[ 804], 10.00th=[ 916], 20.00th=[ 1096],
>>> | 30.00th=[ 1224], 40.00th=[ 1304], 50.00th=[ 1384], 60.00th=[ 1448],
>>> | 70.00th=[ 1544], 80.00th=[ 1704], 90.00th=[ 2128], 95.00th=[ 2736],
>>> | 99.00th=[ 6752], 99.50th=[ 7904], 99.90th=[12096], 99.95th=[14656],
>>> | 99.99th=[18560]
>>> bw (KB/s) : min=47800, max=91952, per=99.91%, avg=80900.88,
>>> stdev=7234.98
>>> lat (usec) : 250=0.01%, 500=0.01%, 750=2.95%, 1000=11.38%
>>> lat (msec) : 2=73.81%, 4=8.81%, 10=2.85%, 20=0.19%, 50=0.01%
>>> lat (msec) : 100=0.01%
>>> cpu : usr=2.99%, sys=10.60%, ctx=66549, majf=0, minf=133
>>> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%,
>>>> =64=0.0%
>>> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>>>> =64=0.0%
>>> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
>>>> =64=0.0%
>>> issued : total=r=0/w=607445/d=0, short=r=0/w=0/d=0
>>>
>>> Run status group 0 (all jobs):
>>> WRITE: io=2372.9MB, aggrb=80976KB/s, minb=80976KB/s, maxb=80976KB/s,
>>> mint=30006msec, maxt=30006msec
>>>
>>> Disk stats (read/write):
>>> sdb: ios=170/605440, merge=0/0, ticks=156/916492, in_queue=916560,
>>> util=99.93%
>>>
>>> Stefan
>>>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: speed decrease since firefly,giant,hammer the 2nd try
2015-02-16 14:50 ` Stefan Priebe - Profihost AG
2015-02-16 15:35 ` Mark Nelson
@ 2015-02-16 15:36 ` Alexandre DERUMIER
2015-02-16 19:10 ` Stefan Priebe
[not found] ` <1760409866.980984.1424105112443.JavaMail.zimbra@oxygem.tv>
2 siblings, 1 reply; 32+ messages in thread
From: Alexandre DERUMIER @ 2015-02-16 15:36 UTC (permalink / raw)
To: Stefan Priebe; +Cc: Mark Nelson, ceph-devel
What is you fio command line ?
do you test with numjobs > 1 ?
(I think under qemu, you can use any numjobs value, as it's use only 1 thread, is equal to numjobs=1)
----- Mail original -----
De: "Stefan Priebe" <s.priebe@profihost.ag>
À: "aderumier" <aderumier@odiso.com>
Cc: "Mark Nelson" <mnelson@redhat.com>, "ceph-devel" <ceph-devel@vger.kernel.org>
Envoyé: Lundi 16 Février 2015 15:50:56
Objet: Re: speed decrease since firefly,giant,hammer the 2nd try
Hi Mark,
Hi Alexandre,
Am 16.02.2015 um 10:11 schrieb Alexandre DERUMIER:
> Hi Stefan,
>
> I could be interesting to see if you have the same speed decrease with fio-librbd on the host,
> without the qemu layer.
>
> the perf reports don't seem to be too much different.
> do you have the same cpu usage ? (check qemu process usage)
the idea to use fio-librbd was very good.
I cannot reproduce the behaviour using fio-rbd. I can just reproduce it
with qemu.
Very strange. So please ignore me for the moment. I'll try to dig deeper
into it.
Greets,
Stefan
> ----- Mail original -----
> De: "Stefan Priebe" <s.priebe@profihost.ag>
> À: "Mark Nelson" <mnelson@redhat.com>, "ceph-devel" <ceph-devel@vger.kernel.org>
> Envoyé: Dimanche 15 Février 2015 19:40:45
> Objet: Re: speed decrease since firefly,giant,hammer the 2nd try
>
> Hi Mark,
>
> what's next?
>
> I've this test cluster only for 2 more days.
>
> Here some perf Details:
>
> dumpling:
> 12,65% libc-2.13.so [.] 0x79000
> 2,86% libc-2.13.so [.] malloc
> 2,80% kvm [.] 0xb59c5
> 2,59% libc-2.13.so [.] free
> 2,35% [kernel] [k] __schedule
> 2,16% [kernel] [k] _raw_spin_lock
> 1,92% [kernel] [k] __switch_to
> 1,58% [kernel] [k] lapic_next_deadline
> 1,09% [kernel] [k] update_sd_lb_stats
> 1,08% [kernel] [k] _raw_spin_lock_irqsave
> 0,91% librados.so.2.0.0 [.] ceph_crc32c_le_intel
> 0,91% libpthread-2.13.so [.] pthread_mutex_trylock
> 0,87% [kernel] [k] resched_task
> 0,72% [kernel] [k] cpu_startup_entry
> 0,71% librados.so.2.0.0 [.] crush_hash32_3
> 0,66% [kernel] [k] leave_mm
> 0,65% librados.so.2.0.0 [.] Mutex::Lock(bool)
> 0,64% [kernel] [k] idle_cpu
> 0,62% libpthread-2.13.so [.] __pthread_mutex_unlock_usercnt
> 0,59% [kernel] [k] try_to_wake_up
> 0,56% [kernel] [k] wake_futex
> 0,50% librados.so.2.0.0 [.] ceph::buffer::ptr::release()
>
> firefly:
> 12,56% libc-2.13.so [.] 0x7905d
> 2,82% libc-2.13.so [.] malloc
> 2,64% libc-2.13.so [.] free
> 2,61% kvm [.] 0x34322f
> 2,33% [kernel] [k] __schedule
> 2,14% [kernel] [k] _raw_spin_lock
> 1,83% [kernel] [k] __switch_to
> 1,62% [kernel] [k] lapic_next_deadline
> 1,17% [kernel] [k] _raw_spin_lock_irqsave
> 1,09% [kernel] [k] update_sd_lb_stats
> 1,08% libpthread-2.13.so [.] pthread_mutex_trylock
> 0,85% libpthread-2.13.so [.] __pthread_mutex_unlock_usercnt
> 0,77% [kernel] [k] resched_task
> 0,74% librbd.so.1.0.0 [.] 0x71b73
> 0,72% librados.so.2.0.0 [.] Mutex::Lock(bool)
> 0,68% librados.so.2.0.0 [.] crush_hash32_3
> 0,67% [kernel] [k] idle_cpu
> 0,65% [kernel] [k] leave_mm
> 0,65% [kernel] [k] cpu_startup_entry
> 0,59% [kernel] [k] try_to_wake_up
> 0,51% librados.so.2.0.0 [.] ceph::buffer::ptr::release()
> 0,51% [kernel] [k] wake_futex
>
> Stefan
>
> Am 11.02.2015 um 06:42 schrieb Stefan Priebe:
>>
>> Am 11.02.2015 um 05:45 schrieb Mark Nelson:
>>> On 02/10/2015 04:18 PM, Stefan Priebe wrote:
>>>>
>>>> Am 10.02.2015 um 22:38 schrieb Mark Nelson:
>>>>> On 02/10/2015 03:11 PM, Stefan Priebe wrote:
>>>>>>
>>>>>> mhm i installed librbd1-dbg and librados2-dbg - but the output still
>>>>>> looks useless to me. Should i upload it somewhere?
>>>>>
>>>>> Meh, if it's all just symbols it's probably not that helpful.
>>>>>
>>>>> I've summarized your results here:
>>>>>
>>>>> 1 concurrent 4k write (libaio, direct=1, iodepth=1)
>>>>>
>>>>> IOPS Latency
>>>>> wb on wb off wb on wb off
>>>>> dumpling 10870 536 ~100us ~2ms
>>>>> firefly 10350 525 ~100us ~2ms
>>>>>
>>>>> So in single op tests dumpling and firefly are far closer. Now let's
>>>>> see each of these cases with iodepth=32 (still 1 thread for now).
>>>>
>>>>
>>>> dumpling:
>>>>
>>>> file1: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=32
>>>> 2.0.8
>>>> Starting 1 thread
>>>> Jobs: 1 (f=1): [w] [100.0% done] [0K/72812K /s] [0 /18.3K iops] [eta
>>>> 00m:00s]
>>>> file1: (groupid=0, jobs=1): err= 0: pid=3011
>>>> write: io=2060.6MB, bw=70329KB/s, iops=17582 , runt= 30001msec
>>>> slat (usec): min=1 , max=3517 , avg= 3.42, stdev= 7.30
>>>> clat (usec): min=93 , max=7475 , avg=1815.72, stdev=233.43
>>>> lat (usec): min=219 , max=7477 , avg=1819.27, stdev=233.52
>>>> clat percentiles (usec):
>>>> | 1.00th=[ 1480], 5.00th=[ 1576], 10.00th=[ 1608], 20.00th=[
>>>> 1672],
>>>> | 30.00th=[ 1704], 40.00th=[ 1752], 50.00th=[ 1800], 60.00th=[
>>>> 1832],
>>>> | 70.00th=[ 1896], 80.00th=[ 1960], 90.00th=[ 2064], 95.00th=[
>>>> 2128],
>>>> | 99.00th=[ 2352], 99.50th=[ 2448], 99.90th=[ 4704], 99.95th=[
>>>> 5344],
>>>> | 99.99th=[ 7072]
>>>> bw (KB/s) : min=59696, max=77840, per=100.00%, avg=70351.27,
>>>> stdev=4783.25
>>>> lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.53%
>>>> lat (msec) : 2=85.02%, 4=14.31%, 10=0.13%
>>>> cpu : usr=1.96%, sys=6.71%, ctx=22791, majf=0, minf=133
>>>> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%,
>>>>> =64=0.0%
>>>> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>>>>> =64=0.0%
>>>> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
>>>>> =64=0.0%
>>>> issued : total=r=0/w=527487/d=0, short=r=0/w=0/d=0
>>>>
>>>> Run status group 0 (all jobs):
>>>> WRITE: io=2060.6MB, aggrb=70329KB/s, minb=70329KB/s, maxb=70329KB/s,
>>>> mint=30001msec, maxt=30001msec
>>>>
>>>> Disk stats (read/write):
>>>> sdb: ios=166/526079, merge=0/0, ticks=24/890120, in_queue=890064,
>>>> util=98.73%
>>>>
>>>> firefly:
>>>>
>>>> file1: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=32
>>>> 2.0.8
>>>> Starting 1 thread
>>>> Jobs: 1 (f=1): [w] [100.0% done] [0K/69096K /s] [0 /17.3K iops] [eta
>>>> 00m:00s]
>>>> file1: (groupid=0, jobs=1): err= 0: pid=2982
>>>> write: io=1784.9MB, bw=60918KB/s, iops=15229 , runt= 30002msec
>>>> slat (usec): min=1 , max=1389 , avg= 3.43, stdev= 5.32
>>>> clat (usec): min=117 , max=8235 , avg=2096.88, stdev=396.30
>>>> lat (usec): min=540 , max=8258 , avg=2100.43, stdev=396.61
>>>> clat percentiles (usec):
>>>> | 1.00th=[ 1608], 5.00th=[ 1720], 10.00th=[ 1768], 20.00th=[
>>>> 1832],
>>>> | 30.00th=[ 1896], 40.00th=[ 1944], 50.00th=[ 2008], 60.00th=[
>>>> 2064],
>>>> | 70.00th=[ 2160], 80.00th=[ 2256], 90.00th=[ 2512], 95.00th=[
>>>> 2896],
>>>> | 99.00th=[ 3600], 99.50th=[ 3792], 99.90th=[ 5088], 99.95th=[
>>>> 6304],
>>>> | 99.99th=[ 6752]
>>>> bw (KB/s) : min=36717, max=73712, per=99.94%, avg=60879.92,
>>>> stdev=8302.27
>>>> lat (usec) : 250=0.01%, 750=0.01%
>>>> lat (msec) : 2=48.56%, 4=51.18%, 10=0.26%
>>>> cpu : usr=2.03%, sys=5.48%, ctx=20440, majf=0, minf=133
>>>> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%,
>>>>> =64=0.0%
>>>> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>>>>> =64=0.0%
>>>> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
>>>>> =64=0.0%
>>>> issued : total=r=0/w=456918/d=0, short=r=0/w=0/d=0
>>>>
>>>> Run status group 0 (all jobs):
>>>> WRITE: io=1784.9MB, aggrb=60918KB/s, minb=60918KB/s, maxb=60918KB/s,
>>>> mint=30002msec, maxt=30002msec
>>>>
>>>> Disk stats (read/write):
>>>> sdb: ios=166/455574, merge=0/0, ticks=12/897748, in_queue=897696,
>>>> util=98.96%
>>>>
>>>
>>> Ok, so it looks like as you increase concurrency the effect increases
>>> (ie contention?). Does the same thing happen without cache enabled?
>>
>> here again without rbd cache:
>>
>> dumpling:
>> file1: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=32
>> 2.0.8
>> Starting 1 thread
>> Jobs: 1 (f=1): [w] [100.0% done] [0K/83488K /s] [0 /20.9K iops] [eta
>> 00m:00s]
>> file1: (groupid=0, jobs=1): err= 0: pid=3000
>> write: io=2449.2MB, bw=83583KB/s, iops=20895 , runt= 30005msec
>> slat (usec): min=1 , max=975 , avg= 4.50, stdev= 5.25
>> clat (usec): min=364 , max=80566 , avg=1525.87, stdev=1194.57
>> lat (usec): min=519 , max=80568 , avg=1530.51, stdev=1194.44
>> clat percentiles (usec):
>> | 1.00th=[ 660], 5.00th=[ 780], 10.00th=[ 876], 20.00th=[ 1032],
>> | 30.00th=[ 1144], 40.00th=[ 1240], 50.00th=[ 1304], 60.00th=[ 1384],
>> | 70.00th=[ 1480], 80.00th=[ 1640], 90.00th=[ 2096], 95.00th=[ 2960],
>> | 99.00th=[ 6816], 99.50th=[ 7840], 99.90th=[11712], 99.95th=[13888],
>> | 99.99th=[18816]
>> bw (KB/s) : min=47184, max=95432, per=100.00%, avg=83639.19,
>> stdev=7973.92
>> lat (usec) : 500=0.01%, 750=3.82%, 1000=14.40%
>> lat (msec) : 2=70.57%, 4=7.91%, 10=3.11%, 20=0.17%, 50=0.01%
>> lat (msec) : 100=0.01%
>> cpu : usr=3.12%, sys=11.49%, ctx=74951, majf=0, minf=133
>> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%,
>>> =64=0.0%
>> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>>> =64=0.0%
>> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
>>> =64=0.0%
>> issued : total=r=0/w=626979/d=0, short=r=0/w=0/d=0
>>
>> Run status group 0 (all jobs):
>> WRITE: io=2449.2MB, aggrb=83583KB/s, minb=83583KB/s, maxb=83583KB/s,
>> mint=30005msec, maxt=30005msec
>>
>> Disk stats (read/write):
>> sdb: ios=168/625292, merge=0/0, ticks=144/916096, in_queue=916128,
>> util=99.93%
>>
>>
>> firefly:
>>
>> fio --filename=/dev/sdb --direct=1 --rw=randwrite --bs=4k --numjobs=1
>> --thread --iodepth=32 --ioengine=libaio --runtime=30 --group_reporting
>> --name=file1
>> file1: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=32
>> 2.0.8
>> Starting 1 thread
>> Jobs: 1 (f=1): [w] [100.0% done] [0K/90044K /s] [0 /22.6K iops] [eta
>> 00m:00s]
>> file1: (groupid=0, jobs=1): err= 0: pid=2970
>> write: io=2372.9MB, bw=80976KB/s, iops=20244 , runt= 30006msec
>> slat (usec): min=1 , max=4047 , avg= 4.36, stdev= 7.17
>> clat (usec): min=197 , max=76656 , avg=1575.29, stdev=1165.74
>> lat (usec): min=523 , max=76660 , avg=1579.79, stdev=1165.59
>> clat percentiles (usec):
>> | 1.00th=[ 676], 5.00th=[ 804], 10.00th=[ 916], 20.00th=[ 1096],
>> | 30.00th=[ 1224], 40.00th=[ 1304], 50.00th=[ 1384], 60.00th=[ 1448],
>> | 70.00th=[ 1544], 80.00th=[ 1704], 90.00th=[ 2128], 95.00th=[ 2736],
>> | 99.00th=[ 6752], 99.50th=[ 7904], 99.90th=[12096], 99.95th=[14656],
>> | 99.99th=[18560]
>> bw (KB/s) : min=47800, max=91952, per=99.91%, avg=80900.88,
>> stdev=7234.98
>> lat (usec) : 250=0.01%, 500=0.01%, 750=2.95%, 1000=11.38%
>> lat (msec) : 2=73.81%, 4=8.81%, 10=2.85%, 20=0.19%, 50=0.01%
>> lat (msec) : 100=0.01%
>> cpu : usr=2.99%, sys=10.60%, ctx=66549, majf=0, minf=133
>> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%,
>>> =64=0.0%
>> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>>> =64=0.0%
>> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
>>> =64=0.0%
>> issued : total=r=0/w=607445/d=0, short=r=0/w=0/d=0
>>
>> Run status group 0 (all jobs):
>> WRITE: io=2372.9MB, aggrb=80976KB/s, minb=80976KB/s, maxb=80976KB/s,
>> mint=30006msec, maxt=30006msec
>>
>> Disk stats (read/write):
>> sdb: ios=170/605440, merge=0/0, ticks=156/916492, in_queue=916560,
>> util=99.93%
>>
>> Stefan
>>
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: speed decrease since firefly,giant,hammer the 2nd try
[not found] ` <1760409866.980984.1424105112443.JavaMail.zimbra@oxygem.tv>
@ 2015-02-16 16:45 ` Alexandre DERUMIER
2015-02-16 19:40 ` Stefan Priebe
0 siblings, 1 reply; 32+ messages in thread
From: Alexandre DERUMIER @ 2015-02-16 16:45 UTC (permalink / raw)
To: Stefan Priebe; +Cc: Mark Nelson, ceph-devel
I also thinked about 1 thing
fio-lirbd use the rbd_cache value from ceph.conf.
and qemu change the value if cache=none or cache=writeback in qemu conf.
So, verify that too.
I'm thinked of this old bug with cache
http://tracker.ceph.com/issues/9513
It was a bug in giant, but tracker said also dumpling and firefly (but no commit for them)
But the original bug was
http://tracker.ceph.com/issues/9854
and I'm not sure it's already released
----- Mail original -----
De: "Stefan Priebe" <s.priebe@profihost.ag>
À: "aderumier" <aderumier@odiso.com>
Cc: "Mark Nelson" <mnelson@redhat.com>, "ceph-devel" <ceph-devel@vger.kernel.org>
Envoyé: Lundi 16 Février 2015 15:50:56
Objet: Re: speed decrease since firefly,giant,hammer the 2nd try
Hi Mark,
Hi Alexandre,
Am 16.02.2015 um 10:11 schrieb Alexandre DERUMIER:
> Hi Stefan,
>
> I could be interesting to see if you have the same speed decrease with fio-librbd on the host,
> without the qemu layer.
>
> the perf reports don't seem to be too much different.
> do you have the same cpu usage ? (check qemu process usage)
the idea to use fio-librbd was very good.
I cannot reproduce the behaviour using fio-rbd. I can just reproduce it
with qemu.
Very strange. So please ignore me for the moment. I'll try to dig deeper
into it.
Greets,
Stefan
> ----- Mail original -----
> De: "Stefan Priebe" <s.priebe@profihost.ag>
> À: "Mark Nelson" <mnelson@redhat.com>, "ceph-devel" <ceph-devel@vger.kernel.org>
> Envoyé: Dimanche 15 Février 2015 19:40:45
> Objet: Re: speed decrease since firefly,giant,hammer the 2nd try
>
> Hi Mark,
>
> what's next?
>
> I've this test cluster only for 2 more days.
>
> Here some perf Details:
>
> dumpling:
> 12,65% libc-2.13.so [.] 0x79000
> 2,86% libc-2.13.so [.] malloc
> 2,80% kvm [.] 0xb59c5
> 2,59% libc-2.13.so [.] free
> 2,35% [kernel] [k] __schedule
> 2,16% [kernel] [k] _raw_spin_lock
> 1,92% [kernel] [k] __switch_to
> 1,58% [kernel] [k] lapic_next_deadline
> 1,09% [kernel] [k] update_sd_lb_stats
> 1,08% [kernel] [k] _raw_spin_lock_irqsave
> 0,91% librados.so.2.0.0 [.] ceph_crc32c_le_intel
> 0,91% libpthread-2.13.so [.] pthread_mutex_trylock
> 0,87% [kernel] [k] resched_task
> 0,72% [kernel] [k] cpu_startup_entry
> 0,71% librados.so.2.0.0 [.] crush_hash32_3
> 0,66% [kernel] [k] leave_mm
> 0,65% librados.so.2.0.0 [.] Mutex::Lock(bool)
> 0,64% [kernel] [k] idle_cpu
> 0,62% libpthread-2.13.so [.] __pthread_mutex_unlock_usercnt
> 0,59% [kernel] [k] try_to_wake_up
> 0,56% [kernel] [k] wake_futex
> 0,50% librados.so.2.0.0 [.] ceph::buffer::ptr::release()
>
> firefly:
> 12,56% libc-2.13.so [.] 0x7905d
> 2,82% libc-2.13.so [.] malloc
> 2,64% libc-2.13.so [.] free
> 2,61% kvm [.] 0x34322f
> 2,33% [kernel] [k] __schedule
> 2,14% [kernel] [k] _raw_spin_lock
> 1,83% [kernel] [k] __switch_to
> 1,62% [kernel] [k] lapic_next_deadline
> 1,17% [kernel] [k] _raw_spin_lock_irqsave
> 1,09% [kernel] [k] update_sd_lb_stats
> 1,08% libpthread-2.13.so [.] pthread_mutex_trylock
> 0,85% libpthread-2.13.so [.] __pthread_mutex_unlock_usercnt
> 0,77% [kernel] [k] resched_task
> 0,74% librbd.so.1.0.0 [.] 0x71b73
> 0,72% librados.so.2.0.0 [.] Mutex::Lock(bool)
> 0,68% librados.so.2.0.0 [.] crush_hash32_3
> 0,67% [kernel] [k] idle_cpu
> 0,65% [kernel] [k] leave_mm
> 0,65% [kernel] [k] cpu_startup_entry
> 0,59% [kernel] [k] try_to_wake_up
> 0,51% librados.so.2.0.0 [.] ceph::buffer::ptr::release()
> 0,51% [kernel] [k] wake_futex
>
> Stefan
>
> Am 11.02.2015 um 06:42 schrieb Stefan Priebe:
>>
>> Am 11.02.2015 um 05:45 schrieb Mark Nelson:
>>> On 02/10/2015 04:18 PM, Stefan Priebe wrote:
>>>>
>>>> Am 10.02.2015 um 22:38 schrieb Mark Nelson:
>>>>> On 02/10/2015 03:11 PM, Stefan Priebe wrote:
>>>>>>
>>>>>> mhm i installed librbd1-dbg and librados2-dbg - but the output still
>>>>>> looks useless to me. Should i upload it somewhere?
>>>>>
>>>>> Meh, if it's all just symbols it's probably not that helpful.
>>>>>
>>>>> I've summarized your results here:
>>>>>
>>>>> 1 concurrent 4k write (libaio, direct=1, iodepth=1)
>>>>>
>>>>> IOPS Latency
>>>>> wb on wb off wb on wb off
>>>>> dumpling 10870 536 ~100us ~2ms
>>>>> firefly 10350 525 ~100us ~2ms
>>>>>
>>>>> So in single op tests dumpling and firefly are far closer. Now let's
>>>>> see each of these cases with iodepth=32 (still 1 thread for now).
>>>>
>>>>
>>>> dumpling:
>>>>
>>>> file1: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=32
>>>> 2.0.8
>>>> Starting 1 thread
>>>> Jobs: 1 (f=1): [w] [100.0% done] [0K/72812K /s] [0 /18.3K iops] [eta
>>>> 00m:00s]
>>>> file1: (groupid=0, jobs=1): err= 0: pid=3011
>>>> write: io=2060.6MB, bw=70329KB/s, iops=17582 , runt= 30001msec
>>>> slat (usec): min=1 , max=3517 , avg= 3.42, stdev= 7.30
>>>> clat (usec): min=93 , max=7475 , avg=1815.72, stdev=233.43
>>>> lat (usec): min=219 , max=7477 , avg=1819.27, stdev=233.52
>>>> clat percentiles (usec):
>>>> | 1.00th=[ 1480], 5.00th=[ 1576], 10.00th=[ 1608], 20.00th=[
>>>> 1672],
>>>> | 30.00th=[ 1704], 40.00th=[ 1752], 50.00th=[ 1800], 60.00th=[
>>>> 1832],
>>>> | 70.00th=[ 1896], 80.00th=[ 1960], 90.00th=[ 2064], 95.00th=[
>>>> 2128],
>>>> | 99.00th=[ 2352], 99.50th=[ 2448], 99.90th=[ 4704], 99.95th=[
>>>> 5344],
>>>> | 99.99th=[ 7072]
>>>> bw (KB/s) : min=59696, max=77840, per=100.00%, avg=70351.27,
>>>> stdev=4783.25
>>>> lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.53%
>>>> lat (msec) : 2=85.02%, 4=14.31%, 10=0.13%
>>>> cpu : usr=1.96%, sys=6.71%, ctx=22791, majf=0, minf=133
>>>> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%,
>>>>> =64=0.0%
>>>> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>>>>> =64=0.0%
>>>> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
>>>>> =64=0.0%
>>>> issued : total=r=0/w=527487/d=0, short=r=0/w=0/d=0
>>>>
>>>> Run status group 0 (all jobs):
>>>> WRITE: io=2060.6MB, aggrb=70329KB/s, minb=70329KB/s, maxb=70329KB/s,
>>>> mint=30001msec, maxt=30001msec
>>>>
>>>> Disk stats (read/write):
>>>> sdb: ios=166/526079, merge=0/0, ticks=24/890120, in_queue=890064,
>>>> util=98.73%
>>>>
>>>> firefly:
>>>>
>>>> file1: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=32
>>>> 2.0.8
>>>> Starting 1 thread
>>>> Jobs: 1 (f=1): [w] [100.0% done] [0K/69096K /s] [0 /17.3K iops] [eta
>>>> 00m:00s]
>>>> file1: (groupid=0, jobs=1): err= 0: pid=2982
>>>> write: io=1784.9MB, bw=60918KB/s, iops=15229 , runt= 30002msec
>>>> slat (usec): min=1 , max=1389 , avg= 3.43, stdev= 5.32
>>>> clat (usec): min=117 , max=8235 , avg=2096.88, stdev=396.30
>>>> lat (usec): min=540 , max=8258 , avg=2100.43, stdev=396.61
>>>> clat percentiles (usec):
>>>> | 1.00th=[ 1608], 5.00th=[ 1720], 10.00th=[ 1768], 20.00th=[
>>>> 1832],
>>>> | 30.00th=[ 1896], 40.00th=[ 1944], 50.00th=[ 2008], 60.00th=[
>>>> 2064],
>>>> | 70.00th=[ 2160], 80.00th=[ 2256], 90.00th=[ 2512], 95.00th=[
>>>> 2896],
>>>> | 99.00th=[ 3600], 99.50th=[ 3792], 99.90th=[ 5088], 99.95th=[
>>>> 6304],
>>>> | 99.99th=[ 6752]
>>>> bw (KB/s) : min=36717, max=73712, per=99.94%, avg=60879.92,
>>>> stdev=8302.27
>>>> lat (usec) : 250=0.01%, 750=0.01%
>>>> lat (msec) : 2=48.56%, 4=51.18%, 10=0.26%
>>>> cpu : usr=2.03%, sys=5.48%, ctx=20440, majf=0, minf=133
>>>> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%,
>>>>> =64=0.0%
>>>> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>>>>> =64=0.0%
>>>> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
>>>>> =64=0.0%
>>>> issued : total=r=0/w=456918/d=0, short=r=0/w=0/d=0
>>>>
>>>> Run status group 0 (all jobs):
>>>> WRITE: io=1784.9MB, aggrb=60918KB/s, minb=60918KB/s, maxb=60918KB/s,
>>>> mint=30002msec, maxt=30002msec
>>>>
>>>> Disk stats (read/write):
>>>> sdb: ios=166/455574, merge=0/0, ticks=12/897748, in_queue=897696,
>>>> util=98.96%
>>>>
>>>
>>> Ok, so it looks like as you increase concurrency the effect increases
>>> (ie contention?). Does the same thing happen without cache enabled?
>>
>> here again without rbd cache:
>>
>> dumpling:
>> file1: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=32
>> 2.0.8
>> Starting 1 thread
>> Jobs: 1 (f=1): [w] [100.0% done] [0K/83488K /s] [0 /20.9K iops] [eta
>> 00m:00s]
>> file1: (groupid=0, jobs=1): err= 0: pid=3000
>> write: io=2449.2MB, bw=83583KB/s, iops=20895 , runt= 30005msec
>> slat (usec): min=1 , max=975 , avg= 4.50, stdev= 5.25
>> clat (usec): min=364 , max=80566 , avg=1525.87, stdev=1194.57
>> lat (usec): min=519 , max=80568 , avg=1530.51, stdev=1194.44
>> clat percentiles (usec):
>> | 1.00th=[ 660], 5.00th=[ 780], 10.00th=[ 876], 20.00th=[ 1032],
>> | 30.00th=[ 1144], 40.00th=[ 1240], 50.00th=[ 1304], 60.00th=[ 1384],
>> | 70.00th=[ 1480], 80.00th=[ 1640], 90.00th=[ 2096], 95.00th=[ 2960],
>> | 99.00th=[ 6816], 99.50th=[ 7840], 99.90th=[11712], 99.95th=[13888],
>> | 99.99th=[18816]
>> bw (KB/s) : min=47184, max=95432, per=100.00%, avg=83639.19,
>> stdev=7973.92
>> lat (usec) : 500=0.01%, 750=3.82%, 1000=14.40%
>> lat (msec) : 2=70.57%, 4=7.91%, 10=3.11%, 20=0.17%, 50=0.01%
>> lat (msec) : 100=0.01%
>> cpu : usr=3.12%, sys=11.49%, ctx=74951, majf=0, minf=133
>> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%,
>>> =64=0.0%
>> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>>> =64=0.0%
>> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
>>> =64=0.0%
>> issued : total=r=0/w=626979/d=0, short=r=0/w=0/d=0
>>
>> Run status group 0 (all jobs):
>> WRITE: io=2449.2MB, aggrb=83583KB/s, minb=83583KB/s, maxb=83583KB/s,
>> mint=30005msec, maxt=30005msec
>>
>> Disk stats (read/write):
>> sdb: ios=168/625292, merge=0/0, ticks=144/916096, in_queue=916128,
>> util=99.93%
>>
>>
>> firefly:
>>
>> fio --filename=/dev/sdb --direct=1 --rw=randwrite --bs=4k --numjobs=1
>> --thread --iodepth=32 --ioengine=libaio --runtime=30 --group_reporting
>> --name=file1
>> file1: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=32
>> 2.0.8
>> Starting 1 thread
>> Jobs: 1 (f=1): [w] [100.0% done] [0K/90044K /s] [0 /22.6K iops] [eta
>> 00m:00s]
>> file1: (groupid=0, jobs=1): err= 0: pid=2970
>> write: io=2372.9MB, bw=80976KB/s, iops=20244 , runt= 30006msec
>> slat (usec): min=1 , max=4047 , avg= 4.36, stdev= 7.17
>> clat (usec): min=197 , max=76656 , avg=1575.29, stdev=1165.74
>> lat (usec): min=523 , max=76660 , avg=1579.79, stdev=1165.59
>> clat percentiles (usec):
>> | 1.00th=[ 676], 5.00th=[ 804], 10.00th=[ 916], 20.00th=[ 1096],
>> | 30.00th=[ 1224], 40.00th=[ 1304], 50.00th=[ 1384], 60.00th=[ 1448],
>> | 70.00th=[ 1544], 80.00th=[ 1704], 90.00th=[ 2128], 95.00th=[ 2736],
>> | 99.00th=[ 6752], 99.50th=[ 7904], 99.90th=[12096], 99.95th=[14656],
>> | 99.99th=[18560]
>> bw (KB/s) : min=47800, max=91952, per=99.91%, avg=80900.88,
>> stdev=7234.98
>> lat (usec) : 250=0.01%, 500=0.01%, 750=2.95%, 1000=11.38%
>> lat (msec) : 2=73.81%, 4=8.81%, 10=2.85%, 20=0.19%, 50=0.01%
>> lat (msec) : 100=0.01%
>> cpu : usr=2.99%, sys=10.60%, ctx=66549, majf=0, minf=133
>> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%,
>>> =64=0.0%
>> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>>> =64=0.0%
>> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
>>> =64=0.0%
>> issued : total=r=0/w=607445/d=0, short=r=0/w=0/d=0
>>
>> Run status group 0 (all jobs):
>> WRITE: io=2372.9MB, aggrb=80976KB/s, minb=80976KB/s, maxb=80976KB/s,
>> mint=30006msec, maxt=30006msec
>>
>> Disk stats (read/write):
>> sdb: ios=170/605440, merge=0/0, ticks=156/916492, in_queue=916560,
>> util=99.93%
>>
>> Stefan
>>
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: speed decrease since firefly,giant,hammer the 2nd try
2015-02-16 15:36 ` Alexandre DERUMIER
@ 2015-02-16 19:10 ` Stefan Priebe
[not found] ` <198700848.1004998.1424124086231.JavaMail.zimbra@oxygem.tv>
0 siblings, 1 reply; 32+ messages in thread
From: Stefan Priebe @ 2015-02-16 19:10 UTC (permalink / raw)
To: Alexandre DERUMIER; +Cc: Mark Nelson, ceph-devel
Am 16.02.2015 um 16:36 schrieb Alexandre DERUMIER:
> What is you fio command line ?
fio rbd or fio under qemu?
> do you test with numjobs > 1 ?
Both.
> (I think under qemu, you can use any numjobs value, as it's use only 1 thread, is equal to numjobs=1)
numjobs > 1 gives also under qemu better results.
Stefan
>
> ----- Mail original -----
> De: "Stefan Priebe" <s.priebe@profihost.ag>
> À: "aderumier" <aderumier@odiso.com>
> Cc: "Mark Nelson" <mnelson@redhat.com>, "ceph-devel" <ceph-devel@vger.kernel.org>
> Envoyé: Lundi 16 Février 2015 15:50:56
> Objet: Re: speed decrease since firefly,giant,hammer the 2nd try
>
> Hi Mark,
> Hi Alexandre,
>
> Am 16.02.2015 um 10:11 schrieb Alexandre DERUMIER:
>> Hi Stefan,
>>
>> I could be interesting to see if you have the same speed decrease with fio-librbd on the host,
>> without the qemu layer.
>>
>> the perf reports don't seem to be too much different.
>> do you have the same cpu usage ? (check qemu process usage)
>
> the idea to use fio-librbd was very good.
>
> I cannot reproduce the behaviour using fio-rbd. I can just reproduce it
> with qemu.
>
> Very strange. So please ignore me for the moment. I'll try to dig deeper
> into it.
>
> Greets,
> Stefan
>
>> ----- Mail original -----
>> De: "Stefan Priebe" <s.priebe@profihost.ag>
>> À: "Mark Nelson" <mnelson@redhat.com>, "ceph-devel" <ceph-devel@vger.kernel.org>
>> Envoyé: Dimanche 15 Février 2015 19:40:45
>> Objet: Re: speed decrease since firefly,giant,hammer the 2nd try
>>
>> Hi Mark,
>>
>> what's next?
>>
>> I've this test cluster only for 2 more days.
>>
>> Here some perf Details:
>>
>> dumpling:
>> 12,65% libc-2.13.so [.] 0x79000
>> 2,86% libc-2.13.so [.] malloc
>> 2,80% kvm [.] 0xb59c5
>> 2,59% libc-2.13.so [.] free
>> 2,35% [kernel] [k] __schedule
>> 2,16% [kernel] [k] _raw_spin_lock
>> 1,92% [kernel] [k] __switch_to
>> 1,58% [kernel] [k] lapic_next_deadline
>> 1,09% [kernel] [k] update_sd_lb_stats
>> 1,08% [kernel] [k] _raw_spin_lock_irqsave
>> 0,91% librados.so.2.0.0 [.] ceph_crc32c_le_intel
>> 0,91% libpthread-2.13.so [.] pthread_mutex_trylock
>> 0,87% [kernel] [k] resched_task
>> 0,72% [kernel] [k] cpu_startup_entry
>> 0,71% librados.so.2.0.0 [.] crush_hash32_3
>> 0,66% [kernel] [k] leave_mm
>> 0,65% librados.so.2.0.0 [.] Mutex::Lock(bool)
>> 0,64% [kernel] [k] idle_cpu
>> 0,62% libpthread-2.13.so [.] __pthread_mutex_unlock_usercnt
>> 0,59% [kernel] [k] try_to_wake_up
>> 0,56% [kernel] [k] wake_futex
>> 0,50% librados.so.2.0.0 [.] ceph::buffer::ptr::release()
>>
>> firefly:
>> 12,56% libc-2.13.so [.] 0x7905d
>> 2,82% libc-2.13.so [.] malloc
>> 2,64% libc-2.13.so [.] free
>> 2,61% kvm [.] 0x34322f
>> 2,33% [kernel] [k] __schedule
>> 2,14% [kernel] [k] _raw_spin_lock
>> 1,83% [kernel] [k] __switch_to
>> 1,62% [kernel] [k] lapic_next_deadline
>> 1,17% [kernel] [k] _raw_spin_lock_irqsave
>> 1,09% [kernel] [k] update_sd_lb_stats
>> 1,08% libpthread-2.13.so [.] pthread_mutex_trylock
>> 0,85% libpthread-2.13.so [.] __pthread_mutex_unlock_usercnt
>> 0,77% [kernel] [k] resched_task
>> 0,74% librbd.so.1.0.0 [.] 0x71b73
>> 0,72% librados.so.2.0.0 [.] Mutex::Lock(bool)
>> 0,68% librados.so.2.0.0 [.] crush_hash32_3
>> 0,67% [kernel] [k] idle_cpu
>> 0,65% [kernel] [k] leave_mm
>> 0,65% [kernel] [k] cpu_startup_entry
>> 0,59% [kernel] [k] try_to_wake_up
>> 0,51% librados.so.2.0.0 [.] ceph::buffer::ptr::release()
>> 0,51% [kernel] [k] wake_futex
>>
>> Stefan
>>
>> Am 11.02.2015 um 06:42 schrieb Stefan Priebe:
>>>
>>> Am 11.02.2015 um 05:45 schrieb Mark Nelson:
>>>> On 02/10/2015 04:18 PM, Stefan Priebe wrote:
>>>>>
>>>>> Am 10.02.2015 um 22:38 schrieb Mark Nelson:
>>>>>> On 02/10/2015 03:11 PM, Stefan Priebe wrote:
>>>>>>>
>>>>>>> mhm i installed librbd1-dbg and librados2-dbg - but the output still
>>>>>>> looks useless to me. Should i upload it somewhere?
>>>>>>
>>>>>> Meh, if it's all just symbols it's probably not that helpful.
>>>>>>
>>>>>> I've summarized your results here:
>>>>>>
>>>>>> 1 concurrent 4k write (libaio, direct=1, iodepth=1)
>>>>>>
>>>>>> IOPS Latency
>>>>>> wb on wb off wb on wb off
>>>>>> dumpling 10870 536 ~100us ~2ms
>>>>>> firefly 10350 525 ~100us ~2ms
>>>>>>
>>>>>> So in single op tests dumpling and firefly are far closer. Now let's
>>>>>> see each of these cases with iodepth=32 (still 1 thread for now).
>>>>>
>>>>>
>>>>> dumpling:
>>>>>
>>>>> file1: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=32
>>>>> 2.0.8
>>>>> Starting 1 thread
>>>>> Jobs: 1 (f=1): [w] [100.0% done] [0K/72812K /s] [0 /18.3K iops] [eta
>>>>> 00m:00s]
>>>>> file1: (groupid=0, jobs=1): err= 0: pid=3011
>>>>> write: io=2060.6MB, bw=70329KB/s, iops=17582 , runt= 30001msec
>>>>> slat (usec): min=1 , max=3517 , avg= 3.42, stdev= 7.30
>>>>> clat (usec): min=93 , max=7475 , avg=1815.72, stdev=233.43
>>>>> lat (usec): min=219 , max=7477 , avg=1819.27, stdev=233.52
>>>>> clat percentiles (usec):
>>>>> | 1.00th=[ 1480], 5.00th=[ 1576], 10.00th=[ 1608], 20.00th=[
>>>>> 1672],
>>>>> | 30.00th=[ 1704], 40.00th=[ 1752], 50.00th=[ 1800], 60.00th=[
>>>>> 1832],
>>>>> | 70.00th=[ 1896], 80.00th=[ 1960], 90.00th=[ 2064], 95.00th=[
>>>>> 2128],
>>>>> | 99.00th=[ 2352], 99.50th=[ 2448], 99.90th=[ 4704], 99.95th=[
>>>>> 5344],
>>>>> | 99.99th=[ 7072]
>>>>> bw (KB/s) : min=59696, max=77840, per=100.00%, avg=70351.27,
>>>>> stdev=4783.25
>>>>> lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.53%
>>>>> lat (msec) : 2=85.02%, 4=14.31%, 10=0.13%
>>>>> cpu : usr=1.96%, sys=6.71%, ctx=22791, majf=0, minf=133
>>>>> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%,
>>>>>> =64=0.0%
>>>>> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>>>>>> =64=0.0%
>>>>> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
>>>>>> =64=0.0%
>>>>> issued : total=r=0/w=527487/d=0, short=r=0/w=0/d=0
>>>>>
>>>>> Run status group 0 (all jobs):
>>>>> WRITE: io=2060.6MB, aggrb=70329KB/s, minb=70329KB/s, maxb=70329KB/s,
>>>>> mint=30001msec, maxt=30001msec
>>>>>
>>>>> Disk stats (read/write):
>>>>> sdb: ios=166/526079, merge=0/0, ticks=24/890120, in_queue=890064,
>>>>> util=98.73%
>>>>>
>>>>> firefly:
>>>>>
>>>>> file1: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=32
>>>>> 2.0.8
>>>>> Starting 1 thread
>>>>> Jobs: 1 (f=1): [w] [100.0% done] [0K/69096K /s] [0 /17.3K iops] [eta
>>>>> 00m:00s]
>>>>> file1: (groupid=0, jobs=1): err= 0: pid=2982
>>>>> write: io=1784.9MB, bw=60918KB/s, iops=15229 , runt= 30002msec
>>>>> slat (usec): min=1 , max=1389 , avg= 3.43, stdev= 5.32
>>>>> clat (usec): min=117 , max=8235 , avg=2096.88, stdev=396.30
>>>>> lat (usec): min=540 , max=8258 , avg=2100.43, stdev=396.61
>>>>> clat percentiles (usec):
>>>>> | 1.00th=[ 1608], 5.00th=[ 1720], 10.00th=[ 1768], 20.00th=[
>>>>> 1832],
>>>>> | 30.00th=[ 1896], 40.00th=[ 1944], 50.00th=[ 2008], 60.00th=[
>>>>> 2064],
>>>>> | 70.00th=[ 2160], 80.00th=[ 2256], 90.00th=[ 2512], 95.00th=[
>>>>> 2896],
>>>>> | 99.00th=[ 3600], 99.50th=[ 3792], 99.90th=[ 5088], 99.95th=[
>>>>> 6304],
>>>>> | 99.99th=[ 6752]
>>>>> bw (KB/s) : min=36717, max=73712, per=99.94%, avg=60879.92,
>>>>> stdev=8302.27
>>>>> lat (usec) : 250=0.01%, 750=0.01%
>>>>> lat (msec) : 2=48.56%, 4=51.18%, 10=0.26%
>>>>> cpu : usr=2.03%, sys=5.48%, ctx=20440, majf=0, minf=133
>>>>> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%,
>>>>>> =64=0.0%
>>>>> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>>>>>> =64=0.0%
>>>>> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
>>>>>> =64=0.0%
>>>>> issued : total=r=0/w=456918/d=0, short=r=0/w=0/d=0
>>>>>
>>>>> Run status group 0 (all jobs):
>>>>> WRITE: io=1784.9MB, aggrb=60918KB/s, minb=60918KB/s, maxb=60918KB/s,
>>>>> mint=30002msec, maxt=30002msec
>>>>>
>>>>> Disk stats (read/write):
>>>>> sdb: ios=166/455574, merge=0/0, ticks=12/897748, in_queue=897696,
>>>>> util=98.96%
>>>>>
>>>>
>>>> Ok, so it looks like as you increase concurrency the effect increases
>>>> (ie contention?). Does the same thing happen without cache enabled?
>>>
>>> here again without rbd cache:
>>>
>>> dumpling:
>>> file1: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=32
>>> 2.0.8
>>> Starting 1 thread
>>> Jobs: 1 (f=1): [w] [100.0% done] [0K/83488K /s] [0 /20.9K iops] [eta
>>> 00m:00s]
>>> file1: (groupid=0, jobs=1): err= 0: pid=3000
>>> write: io=2449.2MB, bw=83583KB/s, iops=20895 , runt= 30005msec
>>> slat (usec): min=1 , max=975 , avg= 4.50, stdev= 5.25
>>> clat (usec): min=364 , max=80566 , avg=1525.87, stdev=1194.57
>>> lat (usec): min=519 , max=80568 , avg=1530.51, stdev=1194.44
>>> clat percentiles (usec):
>>> | 1.00th=[ 660], 5.00th=[ 780], 10.00th=[ 876], 20.00th=[ 1032],
>>> | 30.00th=[ 1144], 40.00th=[ 1240], 50.00th=[ 1304], 60.00th=[ 1384],
>>> | 70.00th=[ 1480], 80.00th=[ 1640], 90.00th=[ 2096], 95.00th=[ 2960],
>>> | 99.00th=[ 6816], 99.50th=[ 7840], 99.90th=[11712], 99.95th=[13888],
>>> | 99.99th=[18816]
>>> bw (KB/s) : min=47184, max=95432, per=100.00%, avg=83639.19,
>>> stdev=7973.92
>>> lat (usec) : 500=0.01%, 750=3.82%, 1000=14.40%
>>> lat (msec) : 2=70.57%, 4=7.91%, 10=3.11%, 20=0.17%, 50=0.01%
>>> lat (msec) : 100=0.01%
>>> cpu : usr=3.12%, sys=11.49%, ctx=74951, majf=0, minf=133
>>> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%,
>>>> =64=0.0%
>>> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>>>> =64=0.0%
>>> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
>>>> =64=0.0%
>>> issued : total=r=0/w=626979/d=0, short=r=0/w=0/d=0
>>>
>>> Run status group 0 (all jobs):
>>> WRITE: io=2449.2MB, aggrb=83583KB/s, minb=83583KB/s, maxb=83583KB/s,
>>> mint=30005msec, maxt=30005msec
>>>
>>> Disk stats (read/write):
>>> sdb: ios=168/625292, merge=0/0, ticks=144/916096, in_queue=916128,
>>> util=99.93%
>>>
>>>
>>> firefly:
>>>
>>> fio --filename=/dev/sdb --direct=1 --rw=randwrite --bs=4k --numjobs=1
>>> --thread --iodepth=32 --ioengine=libaio --runtime=30 --group_reporting
>>> --name=file1
>>> file1: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=32
>>> 2.0.8
>>> Starting 1 thread
>>> Jobs: 1 (f=1): [w] [100.0% done] [0K/90044K /s] [0 /22.6K iops] [eta
>>> 00m:00s]
>>> file1: (groupid=0, jobs=1): err= 0: pid=2970
>>> write: io=2372.9MB, bw=80976KB/s, iops=20244 , runt= 30006msec
>>> slat (usec): min=1 , max=4047 , avg= 4.36, stdev= 7.17
>>> clat (usec): min=197 , max=76656 , avg=1575.29, stdev=1165.74
>>> lat (usec): min=523 , max=76660 , avg=1579.79, stdev=1165.59
>>> clat percentiles (usec):
>>> | 1.00th=[ 676], 5.00th=[ 804], 10.00th=[ 916], 20.00th=[ 1096],
>>> | 30.00th=[ 1224], 40.00th=[ 1304], 50.00th=[ 1384], 60.00th=[ 1448],
>>> | 70.00th=[ 1544], 80.00th=[ 1704], 90.00th=[ 2128], 95.00th=[ 2736],
>>> | 99.00th=[ 6752], 99.50th=[ 7904], 99.90th=[12096], 99.95th=[14656],
>>> | 99.99th=[18560]
>>> bw (KB/s) : min=47800, max=91952, per=99.91%, avg=80900.88,
>>> stdev=7234.98
>>> lat (usec) : 250=0.01%, 500=0.01%, 750=2.95%, 1000=11.38%
>>> lat (msec) : 2=73.81%, 4=8.81%, 10=2.85%, 20=0.19%, 50=0.01%
>>> lat (msec) : 100=0.01%
>>> cpu : usr=2.99%, sys=10.60%, ctx=66549, majf=0, minf=133
>>> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%,
>>>> =64=0.0%
>>> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>>>> =64=0.0%
>>> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
>>>> =64=0.0%
>>> issued : total=r=0/w=607445/d=0, short=r=0/w=0/d=0
>>>
>>> Run status group 0 (all jobs):
>>> WRITE: io=2372.9MB, aggrb=80976KB/s, minb=80976KB/s, maxb=80976KB/s,
>>> mint=30006msec, maxt=30006msec
>>>
>>> Disk stats (read/write):
>>> sdb: ios=170/605440, merge=0/0, ticks=156/916492, in_queue=916560,
>>> util=99.93%
>>>
>>> Stefan
>>>
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: speed decrease since firefly,giant,hammer the 2nd try
2015-02-16 16:45 ` Alexandre DERUMIER
@ 2015-02-16 19:40 ` Stefan Priebe
0 siblings, 0 replies; 32+ messages in thread
From: Stefan Priebe @ 2015-02-16 19:40 UTC (permalink / raw)
To: Alexandre DERUMIER; +Cc: Mark Nelson, ceph-devel
Am 16.02.2015 um 17:45 schrieb Alexandre DERUMIER:
> I also thinked about 1 thing
>
> fio-lirbd use the rbd_cache value from ceph.conf.
>
> and qemu change the value if cache=none or cache=writeback in qemu conf.
>
> So, verify that too.
>
>
>
> I'm thinked of this old bug with cache
>
> http://tracker.ceph.com/issues/9513
> It was a bug in giant, but tracker said also dumpling and firefly (but no commit for them)
>
>
> But the original bug was
>
> http://tracker.ceph.com/issues/9854
> and I'm not sure it's already released
>
No it's not in latest firefly nor in latest dumpling. But it's in latest
git for both.
But it looks read related not write related - isn't it?
Stefan
>
>
>
>
> ----- Mail original -----
> De: "Stefan Priebe" <s.priebe@profihost.ag>
> À: "aderumier" <aderumier@odiso.com>
> Cc: "Mark Nelson" <mnelson@redhat.com>, "ceph-devel" <ceph-devel@vger.kernel.org>
> Envoyé: Lundi 16 Février 2015 15:50:56
> Objet: Re: speed decrease since firefly,giant,hammer the 2nd try
>
> Hi Mark,
> Hi Alexandre,
>
> Am 16.02.2015 um 10:11 schrieb Alexandre DERUMIER:
>> Hi Stefan,
>>
>> I could be interesting to see if you have the same speed decrease with fio-librbd on the host,
>> without the qemu layer.
>>
>> the perf reports don't seem to be too much different.
>> do you have the same cpu usage ? (check qemu process usage)
>
> the idea to use fio-librbd was very good.
>
> I cannot reproduce the behaviour using fio-rbd. I can just reproduce it
> with qemu.
>
> Very strange. So please ignore me for the moment. I'll try to dig deeper
> into it.
>
> Greets,
> Stefan
>
>> ----- Mail original -----
>> De: "Stefan Priebe" <s.priebe@profihost.ag>
>> À: "Mark Nelson" <mnelson@redhat.com>, "ceph-devel" <ceph-devel@vger.kernel.org>
>> Envoyé: Dimanche 15 Février 2015 19:40:45
>> Objet: Re: speed decrease since firefly,giant,hammer the 2nd try
>>
>> Hi Mark,
>>
>> what's next?
>>
>> I've this test cluster only for 2 more days.
>>
>> Here some perf Details:
>>
>> dumpling:
>> 12,65% libc-2.13.so [.] 0x79000
>> 2,86% libc-2.13.so [.] malloc
>> 2,80% kvm [.] 0xb59c5
>> 2,59% libc-2.13.so [.] free
>> 2,35% [kernel] [k] __schedule
>> 2,16% [kernel] [k] _raw_spin_lock
>> 1,92% [kernel] [k] __switch_to
>> 1,58% [kernel] [k] lapic_next_deadline
>> 1,09% [kernel] [k] update_sd_lb_stats
>> 1,08% [kernel] [k] _raw_spin_lock_irqsave
>> 0,91% librados.so.2.0.0 [.] ceph_crc32c_le_intel
>> 0,91% libpthread-2.13.so [.] pthread_mutex_trylock
>> 0,87% [kernel] [k] resched_task
>> 0,72% [kernel] [k] cpu_startup_entry
>> 0,71% librados.so.2.0.0 [.] crush_hash32_3
>> 0,66% [kernel] [k] leave_mm
>> 0,65% librados.so.2.0.0 [.] Mutex::Lock(bool)
>> 0,64% [kernel] [k] idle_cpu
>> 0,62% libpthread-2.13.so [.] __pthread_mutex_unlock_usercnt
>> 0,59% [kernel] [k] try_to_wake_up
>> 0,56% [kernel] [k] wake_futex
>> 0,50% librados.so.2.0.0 [.] ceph::buffer::ptr::release()
>>
>> firefly:
>> 12,56% libc-2.13.so [.] 0x7905d
>> 2,82% libc-2.13.so [.] malloc
>> 2,64% libc-2.13.so [.] free
>> 2,61% kvm [.] 0x34322f
>> 2,33% [kernel] [k] __schedule
>> 2,14% [kernel] [k] _raw_spin_lock
>> 1,83% [kernel] [k] __switch_to
>> 1,62% [kernel] [k] lapic_next_deadline
>> 1,17% [kernel] [k] _raw_spin_lock_irqsave
>> 1,09% [kernel] [k] update_sd_lb_stats
>> 1,08% libpthread-2.13.so [.] pthread_mutex_trylock
>> 0,85% libpthread-2.13.so [.] __pthread_mutex_unlock_usercnt
>> 0,77% [kernel] [k] resched_task
>> 0,74% librbd.so.1.0.0 [.] 0x71b73
>> 0,72% librados.so.2.0.0 [.] Mutex::Lock(bool)
>> 0,68% librados.so.2.0.0 [.] crush_hash32_3
>> 0,67% [kernel] [k] idle_cpu
>> 0,65% [kernel] [k] leave_mm
>> 0,65% [kernel] [k] cpu_startup_entry
>> 0,59% [kernel] [k] try_to_wake_up
>> 0,51% librados.so.2.0.0 [.] ceph::buffer::ptr::release()
>> 0,51% [kernel] [k] wake_futex
>>
>> Stefan
>>
>> Am 11.02.2015 um 06:42 schrieb Stefan Priebe:
>>>
>>> Am 11.02.2015 um 05:45 schrieb Mark Nelson:
>>>> On 02/10/2015 04:18 PM, Stefan Priebe wrote:
>>>>>
>>>>> Am 10.02.2015 um 22:38 schrieb Mark Nelson:
>>>>>> On 02/10/2015 03:11 PM, Stefan Priebe wrote:
>>>>>>>
>>>>>>> mhm i installed librbd1-dbg and librados2-dbg - but the output still
>>>>>>> looks useless to me. Should i upload it somewhere?
>>>>>>
>>>>>> Meh, if it's all just symbols it's probably not that helpful.
>>>>>>
>>>>>> I've summarized your results here:
>>>>>>
>>>>>> 1 concurrent 4k write (libaio, direct=1, iodepth=1)
>>>>>>
>>>>>> IOPS Latency
>>>>>> wb on wb off wb on wb off
>>>>>> dumpling 10870 536 ~100us ~2ms
>>>>>> firefly 10350 525 ~100us ~2ms
>>>>>>
>>>>>> So in single op tests dumpling and firefly are far closer. Now let's
>>>>>> see each of these cases with iodepth=32 (still 1 thread for now).
>>>>>
>>>>>
>>>>> dumpling:
>>>>>
>>>>> file1: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=32
>>>>> 2.0.8
>>>>> Starting 1 thread
>>>>> Jobs: 1 (f=1): [w] [100.0% done] [0K/72812K /s] [0 /18.3K iops] [eta
>>>>> 00m:00s]
>>>>> file1: (groupid=0, jobs=1): err= 0: pid=3011
>>>>> write: io=2060.6MB, bw=70329KB/s, iops=17582 , runt= 30001msec
>>>>> slat (usec): min=1 , max=3517 , avg= 3.42, stdev= 7.30
>>>>> clat (usec): min=93 , max=7475 , avg=1815.72, stdev=233.43
>>>>> lat (usec): min=219 , max=7477 , avg=1819.27, stdev=233.52
>>>>> clat percentiles (usec):
>>>>> | 1.00th=[ 1480], 5.00th=[ 1576], 10.00th=[ 1608], 20.00th=[
>>>>> 1672],
>>>>> | 30.00th=[ 1704], 40.00th=[ 1752], 50.00th=[ 1800], 60.00th=[
>>>>> 1832],
>>>>> | 70.00th=[ 1896], 80.00th=[ 1960], 90.00th=[ 2064], 95.00th=[
>>>>> 2128],
>>>>> | 99.00th=[ 2352], 99.50th=[ 2448], 99.90th=[ 4704], 99.95th=[
>>>>> 5344],
>>>>> | 99.99th=[ 7072]
>>>>> bw (KB/s) : min=59696, max=77840, per=100.00%, avg=70351.27,
>>>>> stdev=4783.25
>>>>> lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.53%
>>>>> lat (msec) : 2=85.02%, 4=14.31%, 10=0.13%
>>>>> cpu : usr=1.96%, sys=6.71%, ctx=22791, majf=0, minf=133
>>>>> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%,
>>>>>> =64=0.0%
>>>>> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>>>>>> =64=0.0%
>>>>> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
>>>>>> =64=0.0%
>>>>> issued : total=r=0/w=527487/d=0, short=r=0/w=0/d=0
>>>>>
>>>>> Run status group 0 (all jobs):
>>>>> WRITE: io=2060.6MB, aggrb=70329KB/s, minb=70329KB/s, maxb=70329KB/s,
>>>>> mint=30001msec, maxt=30001msec
>>>>>
>>>>> Disk stats (read/write):
>>>>> sdb: ios=166/526079, merge=0/0, ticks=24/890120, in_queue=890064,
>>>>> util=98.73%
>>>>>
>>>>> firefly:
>>>>>
>>>>> file1: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=32
>>>>> 2.0.8
>>>>> Starting 1 thread
>>>>> Jobs: 1 (f=1): [w] [100.0% done] [0K/69096K /s] [0 /17.3K iops] [eta
>>>>> 00m:00s]
>>>>> file1: (groupid=0, jobs=1): err= 0: pid=2982
>>>>> write: io=1784.9MB, bw=60918KB/s, iops=15229 , runt= 30002msec
>>>>> slat (usec): min=1 , max=1389 , avg= 3.43, stdev= 5.32
>>>>> clat (usec): min=117 , max=8235 , avg=2096.88, stdev=396.30
>>>>> lat (usec): min=540 , max=8258 , avg=2100.43, stdev=396.61
>>>>> clat percentiles (usec):
>>>>> | 1.00th=[ 1608], 5.00th=[ 1720], 10.00th=[ 1768], 20.00th=[
>>>>> 1832],
>>>>> | 30.00th=[ 1896], 40.00th=[ 1944], 50.00th=[ 2008], 60.00th=[
>>>>> 2064],
>>>>> | 70.00th=[ 2160], 80.00th=[ 2256], 90.00th=[ 2512], 95.00th=[
>>>>> 2896],
>>>>> | 99.00th=[ 3600], 99.50th=[ 3792], 99.90th=[ 5088], 99.95th=[
>>>>> 6304],
>>>>> | 99.99th=[ 6752]
>>>>> bw (KB/s) : min=36717, max=73712, per=99.94%, avg=60879.92,
>>>>> stdev=8302.27
>>>>> lat (usec) : 250=0.01%, 750=0.01%
>>>>> lat (msec) : 2=48.56%, 4=51.18%, 10=0.26%
>>>>> cpu : usr=2.03%, sys=5.48%, ctx=20440, majf=0, minf=133
>>>>> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%,
>>>>>> =64=0.0%
>>>>> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>>>>>> =64=0.0%
>>>>> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
>>>>>> =64=0.0%
>>>>> issued : total=r=0/w=456918/d=0, short=r=0/w=0/d=0
>>>>>
>>>>> Run status group 0 (all jobs):
>>>>> WRITE: io=1784.9MB, aggrb=60918KB/s, minb=60918KB/s, maxb=60918KB/s,
>>>>> mint=30002msec, maxt=30002msec
>>>>>
>>>>> Disk stats (read/write):
>>>>> sdb: ios=166/455574, merge=0/0, ticks=12/897748, in_queue=897696,
>>>>> util=98.96%
>>>>>
>>>>
>>>> Ok, so it looks like as you increase concurrency the effect increases
>>>> (ie contention?). Does the same thing happen without cache enabled?
>>>
>>> here again without rbd cache:
>>>
>>> dumpling:
>>> file1: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=32
>>> 2.0.8
>>> Starting 1 thread
>>> Jobs: 1 (f=1): [w] [100.0% done] [0K/83488K /s] [0 /20.9K iops] [eta
>>> 00m:00s]
>>> file1: (groupid=0, jobs=1): err= 0: pid=3000
>>> write: io=2449.2MB, bw=83583KB/s, iops=20895 , runt= 30005msec
>>> slat (usec): min=1 , max=975 , avg= 4.50, stdev= 5.25
>>> clat (usec): min=364 , max=80566 , avg=1525.87, stdev=1194.57
>>> lat (usec): min=519 , max=80568 , avg=1530.51, stdev=1194.44
>>> clat percentiles (usec):
>>> | 1.00th=[ 660], 5.00th=[ 780], 10.00th=[ 876], 20.00th=[ 1032],
>>> | 30.00th=[ 1144], 40.00th=[ 1240], 50.00th=[ 1304], 60.00th=[ 1384],
>>> | 70.00th=[ 1480], 80.00th=[ 1640], 90.00th=[ 2096], 95.00th=[ 2960],
>>> | 99.00th=[ 6816], 99.50th=[ 7840], 99.90th=[11712], 99.95th=[13888],
>>> | 99.99th=[18816]
>>> bw (KB/s) : min=47184, max=95432, per=100.00%, avg=83639.19,
>>> stdev=7973.92
>>> lat (usec) : 500=0.01%, 750=3.82%, 1000=14.40%
>>> lat (msec) : 2=70.57%, 4=7.91%, 10=3.11%, 20=0.17%, 50=0.01%
>>> lat (msec) : 100=0.01%
>>> cpu : usr=3.12%, sys=11.49%, ctx=74951, majf=0, minf=133
>>> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%,
>>>> =64=0.0%
>>> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>>>> =64=0.0%
>>> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
>>>> =64=0.0%
>>> issued : total=r=0/w=626979/d=0, short=r=0/w=0/d=0
>>>
>>> Run status group 0 (all jobs):
>>> WRITE: io=2449.2MB, aggrb=83583KB/s, minb=83583KB/s, maxb=83583KB/s,
>>> mint=30005msec, maxt=30005msec
>>>
>>> Disk stats (read/write):
>>> sdb: ios=168/625292, merge=0/0, ticks=144/916096, in_queue=916128,
>>> util=99.93%
>>>
>>>
>>> firefly:
>>>
>>> fio --filename=/dev/sdb --direct=1 --rw=randwrite --bs=4k --numjobs=1
>>> --thread --iodepth=32 --ioengine=libaio --runtime=30 --group_reporting
>>> --name=file1
>>> file1: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=32
>>> 2.0.8
>>> Starting 1 thread
>>> Jobs: 1 (f=1): [w] [100.0% done] [0K/90044K /s] [0 /22.6K iops] [eta
>>> 00m:00s]
>>> file1: (groupid=0, jobs=1): err= 0: pid=2970
>>> write: io=2372.9MB, bw=80976KB/s, iops=20244 , runt= 30006msec
>>> slat (usec): min=1 , max=4047 , avg= 4.36, stdev= 7.17
>>> clat (usec): min=197 , max=76656 , avg=1575.29, stdev=1165.74
>>> lat (usec): min=523 , max=76660 , avg=1579.79, stdev=1165.59
>>> clat percentiles (usec):
>>> | 1.00th=[ 676], 5.00th=[ 804], 10.00th=[ 916], 20.00th=[ 1096],
>>> | 30.00th=[ 1224], 40.00th=[ 1304], 50.00th=[ 1384], 60.00th=[ 1448],
>>> | 70.00th=[ 1544], 80.00th=[ 1704], 90.00th=[ 2128], 95.00th=[ 2736],
>>> | 99.00th=[ 6752], 99.50th=[ 7904], 99.90th=[12096], 99.95th=[14656],
>>> | 99.99th=[18560]
>>> bw (KB/s) : min=47800, max=91952, per=99.91%, avg=80900.88,
>>> stdev=7234.98
>>> lat (usec) : 250=0.01%, 500=0.01%, 750=2.95%, 1000=11.38%
>>> lat (msec) : 2=73.81%, 4=8.81%, 10=2.85%, 20=0.19%, 50=0.01%
>>> lat (msec) : 100=0.01%
>>> cpu : usr=2.99%, sys=10.60%, ctx=66549, majf=0, minf=133
>>> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%,
>>>> =64=0.0%
>>> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>>>> =64=0.0%
>>> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
>>>> =64=0.0%
>>> issued : total=r=0/w=607445/d=0, short=r=0/w=0/d=0
>>>
>>> Run status group 0 (all jobs):
>>> WRITE: io=2372.9MB, aggrb=80976KB/s, minb=80976KB/s, maxb=80976KB/s,
>>> mint=30006msec, maxt=30006msec
>>>
>>> Disk stats (read/write):
>>> sdb: ios=170/605440, merge=0/0, ticks=156/916492, in_queue=916560,
>>> util=99.93%
>>>
>>> Stefan
>>>
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: speed decrease since firefly,giant,hammer the 2nd try
2015-02-15 18:40 ` Stefan Priebe
[not found] ` <1880785650.892634.1424077856500.JavaMail.zimbra@oxygem.tv>
@ 2015-02-16 21:22 ` Stefan Priebe
[not found] ` <2015376321.1005004.1424124122884.JavaMail.zimbra@oxygem.tv>
1 sibling, 1 reply; 32+ messages in thread
From: Stefan Priebe @ 2015-02-16 21:22 UTC (permalink / raw)
To: Mark Nelson, ceph-devel
I've now upgraded server side and client side to latest upstream/firefly.
This results in fio-rbd showing avg 26000 iop/s instead of 30500 iop/s
while running dumpling...
Greets,
Stefan
Am 15.02.2015 um 19:40 schrieb Stefan Priebe:
> Hi Mark,
>
> what's next?
>
> I've this test cluster only for 2 more days.
>
> Here some perf Details:
>
> dumpling:
> 12,65% libc-2.13.so [.] 0x79000
> 2,86% libc-2.13.so [.] malloc
> 2,80% kvm [.] 0xb59c5
> 2,59% libc-2.13.so [.] free
> 2,35% [kernel] [k] __schedule
> 2,16% [kernel] [k] _raw_spin_lock
> 1,92% [kernel] [k] __switch_to
> 1,58% [kernel] [k] lapic_next_deadline
> 1,09% [kernel] [k] update_sd_lb_stats
> 1,08% [kernel] [k] _raw_spin_lock_irqsave
> 0,91% librados.so.2.0.0 [.] ceph_crc32c_le_intel
> 0,91% libpthread-2.13.so [.] pthread_mutex_trylock
> 0,87% [kernel] [k] resched_task
> 0,72% [kernel] [k] cpu_startup_entry
> 0,71% librados.so.2.0.0 [.] crush_hash32_3
> 0,66% [kernel] [k] leave_mm
> 0,65% librados.so.2.0.0 [.] Mutex::Lock(bool)
> 0,64% [kernel] [k] idle_cpu
> 0,62% libpthread-2.13.so [.] __pthread_mutex_unlock_usercnt
> 0,59% [kernel] [k] try_to_wake_up
> 0,56% [kernel] [k] wake_futex
> 0,50% librados.so.2.0.0 [.] ceph::buffer::ptr::release()
>
> firefly:
> 12,56% libc-2.13.so [.] 0x7905d
> 2,82% libc-2.13.so [.] malloc
> 2,64% libc-2.13.so [.] free
> 2,61% kvm [.] 0x34322f
> 2,33% [kernel] [k] __schedule
> 2,14% [kernel] [k] _raw_spin_lock
> 1,83% [kernel] [k] __switch_to
> 1,62% [kernel] [k] lapic_next_deadline
> 1,17% [kernel] [k] _raw_spin_lock_irqsave
> 1,09% [kernel] [k] update_sd_lb_stats
> 1,08% libpthread-2.13.so [.] pthread_mutex_trylock
> 0,85% libpthread-2.13.so [.] __pthread_mutex_unlock_usercnt
> 0,77% [kernel] [k] resched_task
> 0,74% librbd.so.1.0.0 [.] 0x71b73
> 0,72% librados.so.2.0.0 [.] Mutex::Lock(bool)
> 0,68% librados.so.2.0.0 [.] crush_hash32_3
> 0,67% [kernel] [k] idle_cpu
> 0,65% [kernel] [k] leave_mm
> 0,65% [kernel] [k] cpu_startup_entry
> 0,59% [kernel] [k] try_to_wake_up
> 0,51% librados.so.2.0.0 [.] ceph::buffer::ptr::release()
> 0,51% [kernel] [k] wake_futex
>
> Stefan
>
> Am 11.02.2015 um 06:42 schrieb Stefan Priebe:
>>
>> Am 11.02.2015 um 05:45 schrieb Mark Nelson:
>>> On 02/10/2015 04:18 PM, Stefan Priebe wrote:
>>>>
>>>> Am 10.02.2015 um 22:38 schrieb Mark Nelson:
>>>>> On 02/10/2015 03:11 PM, Stefan Priebe wrote:
>>>>>>
>>>>>> mhm i installed librbd1-dbg and librados2-dbg - but the output still
>>>>>> looks useless to me. Should i upload it somewhere?
>>>>>
>>>>> Meh, if it's all just symbols it's probably not that helpful.
>>>>>
>>>>> I've summarized your results here:
>>>>>
>>>>> 1 concurrent 4k write (libaio, direct=1, iodepth=1)
>>>>>
>>>>> IOPS Latency
>>>>> wb on wb off wb on wb off
>>>>> dumpling 10870 536 ~100us ~2ms
>>>>> firefly 10350 525 ~100us ~2ms
>>>>>
>>>>> So in single op tests dumpling and firefly are far closer. Now let's
>>>>> see each of these cases with iodepth=32 (still 1 thread for now).
>>>>
>>>>
>>>> dumpling:
>>>>
>>>> file1: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=32
>>>> 2.0.8
>>>> Starting 1 thread
>>>> Jobs: 1 (f=1): [w] [100.0% done] [0K/72812K /s] [0 /18.3K iops] [eta
>>>> 00m:00s]
>>>> file1: (groupid=0, jobs=1): err= 0: pid=3011
>>>> write: io=2060.6MB, bw=70329KB/s, iops=17582 , runt= 30001msec
>>>> slat (usec): min=1 , max=3517 , avg= 3.42, stdev= 7.30
>>>> clat (usec): min=93 , max=7475 , avg=1815.72, stdev=233.43
>>>> lat (usec): min=219 , max=7477 , avg=1819.27, stdev=233.52
>>>> clat percentiles (usec):
>>>> | 1.00th=[ 1480], 5.00th=[ 1576], 10.00th=[ 1608], 20.00th=[
>>>> 1672],
>>>> | 30.00th=[ 1704], 40.00th=[ 1752], 50.00th=[ 1800], 60.00th=[
>>>> 1832],
>>>> | 70.00th=[ 1896], 80.00th=[ 1960], 90.00th=[ 2064], 95.00th=[
>>>> 2128],
>>>> | 99.00th=[ 2352], 99.50th=[ 2448], 99.90th=[ 4704], 99.95th=[
>>>> 5344],
>>>> | 99.99th=[ 7072]
>>>> bw (KB/s) : min=59696, max=77840, per=100.00%, avg=70351.27,
>>>> stdev=4783.25
>>>> lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%,
>>>> 1000=0.53%
>>>> lat (msec) : 2=85.02%, 4=14.31%, 10=0.13%
>>>> cpu : usr=1.96%, sys=6.71%, ctx=22791, majf=0, minf=133
>>>> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%,
>>>> >=64=0.0%
>>>> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>>>> >=64=0.0%
>>>> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
>>>> >=64=0.0%
>>>> issued : total=r=0/w=527487/d=0, short=r=0/w=0/d=0
>>>>
>>>> Run status group 0 (all jobs):
>>>> WRITE: io=2060.6MB, aggrb=70329KB/s, minb=70329KB/s, maxb=70329KB/s,
>>>> mint=30001msec, maxt=30001msec
>>>>
>>>> Disk stats (read/write):
>>>> sdb: ios=166/526079, merge=0/0, ticks=24/890120, in_queue=890064,
>>>> util=98.73%
>>>>
>>>> firefly:
>>>>
>>>> file1: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=32
>>>> 2.0.8
>>>> Starting 1 thread
>>>> Jobs: 1 (f=1): [w] [100.0% done] [0K/69096K /s] [0 /17.3K iops] [eta
>>>> 00m:00s]
>>>> file1: (groupid=0, jobs=1): err= 0: pid=2982
>>>> write: io=1784.9MB, bw=60918KB/s, iops=15229 , runt= 30002msec
>>>> slat (usec): min=1 , max=1389 , avg= 3.43, stdev= 5.32
>>>> clat (usec): min=117 , max=8235 , avg=2096.88, stdev=396.30
>>>> lat (usec): min=540 , max=8258 , avg=2100.43, stdev=396.61
>>>> clat percentiles (usec):
>>>> | 1.00th=[ 1608], 5.00th=[ 1720], 10.00th=[ 1768], 20.00th=[
>>>> 1832],
>>>> | 30.00th=[ 1896], 40.00th=[ 1944], 50.00th=[ 2008], 60.00th=[
>>>> 2064],
>>>> | 70.00th=[ 2160], 80.00th=[ 2256], 90.00th=[ 2512], 95.00th=[
>>>> 2896],
>>>> | 99.00th=[ 3600], 99.50th=[ 3792], 99.90th=[ 5088], 99.95th=[
>>>> 6304],
>>>> | 99.99th=[ 6752]
>>>> bw (KB/s) : min=36717, max=73712, per=99.94%, avg=60879.92,
>>>> stdev=8302.27
>>>> lat (usec) : 250=0.01%, 750=0.01%
>>>> lat (msec) : 2=48.56%, 4=51.18%, 10=0.26%
>>>> cpu : usr=2.03%, sys=5.48%, ctx=20440, majf=0, minf=133
>>>> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%,
>>>> >=64=0.0%
>>>> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>>>> >=64=0.0%
>>>> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
>>>> >=64=0.0%
>>>> issued : total=r=0/w=456918/d=0, short=r=0/w=0/d=0
>>>>
>>>> Run status group 0 (all jobs):
>>>> WRITE: io=1784.9MB, aggrb=60918KB/s, minb=60918KB/s, maxb=60918KB/s,
>>>> mint=30002msec, maxt=30002msec
>>>>
>>>> Disk stats (read/write):
>>>> sdb: ios=166/455574, merge=0/0, ticks=12/897748, in_queue=897696,
>>>> util=98.96%
>>>>
>>>
>>> Ok, so it looks like as you increase concurrency the effect increases
>>> (ie contention?). Does the same thing happen without cache enabled?
>>
>> here again without rbd cache:
>>
>> dumpling:
>> file1: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=32
>> 2.0.8
>> Starting 1 thread
>> Jobs: 1 (f=1): [w] [100.0% done] [0K/83488K /s] [0 /20.9K iops] [eta
>> 00m:00s]
>> file1: (groupid=0, jobs=1): err= 0: pid=3000
>> write: io=2449.2MB, bw=83583KB/s, iops=20895 , runt= 30005msec
>> slat (usec): min=1 , max=975 , avg= 4.50, stdev= 5.25
>> clat (usec): min=364 , max=80566 , avg=1525.87, stdev=1194.57
>> lat (usec): min=519 , max=80568 , avg=1530.51, stdev=1194.44
>> clat percentiles (usec):
>> | 1.00th=[ 660], 5.00th=[ 780], 10.00th=[ 876], 20.00th=[
>> 1032],
>> | 30.00th=[ 1144], 40.00th=[ 1240], 50.00th=[ 1304], 60.00th=[
>> 1384],
>> | 70.00th=[ 1480], 80.00th=[ 1640], 90.00th=[ 2096], 95.00th=[
>> 2960],
>> | 99.00th=[ 6816], 99.50th=[ 7840], 99.90th=[11712],
>> 99.95th=[13888],
>> | 99.99th=[18816]
>> bw (KB/s) : min=47184, max=95432, per=100.00%, avg=83639.19,
>> stdev=7973.92
>> lat (usec) : 500=0.01%, 750=3.82%, 1000=14.40%
>> lat (msec) : 2=70.57%, 4=7.91%, 10=3.11%, 20=0.17%, 50=0.01%
>> lat (msec) : 100=0.01%
>> cpu : usr=3.12%, sys=11.49%, ctx=74951, majf=0, minf=133
>> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%,
>> >=64=0.0%
>> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>> >=64=0.0%
>> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
>> >=64=0.0%
>> issued : total=r=0/w=626979/d=0, short=r=0/w=0/d=0
>>
>> Run status group 0 (all jobs):
>> WRITE: io=2449.2MB, aggrb=83583KB/s, minb=83583KB/s, maxb=83583KB/s,
>> mint=30005msec, maxt=30005msec
>>
>> Disk stats (read/write):
>> sdb: ios=168/625292, merge=0/0, ticks=144/916096, in_queue=916128,
>> util=99.93%
>>
>>
>> firefly:
>>
>> fio --filename=/dev/sdb --direct=1 --rw=randwrite --bs=4k --numjobs=1
>> --thread --iodepth=32 --ioengine=libaio --runtime=30 --group_reporting
>> --name=file1
>> file1: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=32
>> 2.0.8
>> Starting 1 thread
>> Jobs: 1 (f=1): [w] [100.0% done] [0K/90044K /s] [0 /22.6K iops] [eta
>> 00m:00s]
>> file1: (groupid=0, jobs=1): err= 0: pid=2970
>> write: io=2372.9MB, bw=80976KB/s, iops=20244 , runt= 30006msec
>> slat (usec): min=1 , max=4047 , avg= 4.36, stdev= 7.17
>> clat (usec): min=197 , max=76656 , avg=1575.29, stdev=1165.74
>> lat (usec): min=523 , max=76660 , avg=1579.79, stdev=1165.59
>> clat percentiles (usec):
>> | 1.00th=[ 676], 5.00th=[ 804], 10.00th=[ 916], 20.00th=[
>> 1096],
>> | 30.00th=[ 1224], 40.00th=[ 1304], 50.00th=[ 1384], 60.00th=[
>> 1448],
>> | 70.00th=[ 1544], 80.00th=[ 1704], 90.00th=[ 2128], 95.00th=[
>> 2736],
>> | 99.00th=[ 6752], 99.50th=[ 7904], 99.90th=[12096],
>> 99.95th=[14656],
>> | 99.99th=[18560]
>> bw (KB/s) : min=47800, max=91952, per=99.91%, avg=80900.88,
>> stdev=7234.98
>> lat (usec) : 250=0.01%, 500=0.01%, 750=2.95%, 1000=11.38%
>> lat (msec) : 2=73.81%, 4=8.81%, 10=2.85%, 20=0.19%, 50=0.01%
>> lat (msec) : 100=0.01%
>> cpu : usr=2.99%, sys=10.60%, ctx=66549, majf=0, minf=133
>> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%,
>> >=64=0.0%
>> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>> >=64=0.0%
>> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
>> >=64=0.0%
>> issued : total=r=0/w=607445/d=0, short=r=0/w=0/d=0
>>
>> Run status group 0 (all jobs):
>> WRITE: io=2372.9MB, aggrb=80976KB/s, minb=80976KB/s, maxb=80976KB/s,
>> mint=30006msec, maxt=30006msec
>>
>> Disk stats (read/write):
>> sdb: ios=170/605440, merge=0/0, ticks=156/916492, in_queue=916560,
>> util=99.93%
>>
>> Stefan
>>
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: speed decrease since firefly,giant,hammer the 2nd try
[not found] ` <198700848.1004998.1424124086231.JavaMail.zimbra@oxygem.tv>
@ 2015-02-16 22:01 ` Alexandre DERUMIER
0 siblings, 0 replies; 32+ messages in thread
From: Alexandre DERUMIER @ 2015-02-16 22:01 UTC (permalink / raw)
To: Stefan Priebe; +Cc: Mark Nelson, ceph-devel
>>fio rbd or fio under qemu?
both ;)
>> (I think under qemu, you can use any numjobs value, as it's use only 1 thread, is equal to numjobs=1)
>
>numjobs > 1 gives also under qemu better results.
I think under qemu, the fio process can scale across vcpus, but not the io thread itself.
So maybe numjobs > 1 under qemu help a little.
how many iops with
fio-rbd vs qemu fio with numjob=1
and
fio-rbd vs qemu fio with numjob=4
for example ?
(I think it should scale better with fio-rbd)
----- Mail original -----
De: "Stefan Priebe" <s.priebe@profihost.ag>
À: "aderumier" <aderumier@odiso.com>
Cc: "Mark Nelson" <mnelson@redhat.com>, "ceph-devel" <ceph-devel@vger.kernel.org>
Envoyé: Lundi 16 Février 2015 20:10:16
Objet: Re: speed decrease since firefly,giant,hammer the 2nd try
Am 16.02.2015 um 16:36 schrieb Alexandre DERUMIER:
> What is you fio command line ?
fio rbd or fio under qemu?
> do you test with numjobs > 1 ?
Both.
> (I think under qemu, you can use any numjobs value, as it's use only 1 thread, is equal to numjobs=1)
numjobs > 1 gives also under qemu better results.
Stefan
>
> ----- Mail original -----
> De: "Stefan Priebe" <s.priebe@profihost.ag>
> À: "aderumier" <aderumier@odiso.com>
> Cc: "Mark Nelson" <mnelson@redhat.com>, "ceph-devel" <ceph-devel@vger.kernel.org>
> Envoyé: Lundi 16 Février 2015 15:50:56
> Objet: Re: speed decrease since firefly,giant,hammer the 2nd try
>
> Hi Mark,
> Hi Alexandre,
>
> Am 16.02.2015 um 10:11 schrieb Alexandre DERUMIER:
>> Hi Stefan,
>>
>> I could be interesting to see if you have the same speed decrease with fio-librbd on the host,
>> without the qemu layer.
>>
>> the perf reports don't seem to be too much different.
>> do you have the same cpu usage ? (check qemu process usage)
>
> the idea to use fio-librbd was very good.
>
> I cannot reproduce the behaviour using fio-rbd. I can just reproduce it
> with qemu.
>
> Very strange. So please ignore me for the moment. I'll try to dig deeper
> into it.
>
> Greets,
> Stefan
>
>> ----- Mail original -----
>> De: "Stefan Priebe" <s.priebe@profihost.ag>
>> À: "Mark Nelson" <mnelson@redhat.com>, "ceph-devel" <ceph-devel@vger.kernel.org>
>> Envoyé: Dimanche 15 Février 2015 19:40:45
>> Objet: Re: speed decrease since firefly,giant,hammer the 2nd try
>>
>> Hi Mark,
>>
>> what's next?
>>
>> I've this test cluster only for 2 more days.
>>
>> Here some perf Details:
>>
>> dumpling:
>> 12,65% libc-2.13.so [.] 0x79000
>> 2,86% libc-2.13.so [.] malloc
>> 2,80% kvm [.] 0xb59c5
>> 2,59% libc-2.13.so [.] free
>> 2,35% [kernel] [k] __schedule
>> 2,16% [kernel] [k] _raw_spin_lock
>> 1,92% [kernel] [k] __switch_to
>> 1,58% [kernel] [k] lapic_next_deadline
>> 1,09% [kernel] [k] update_sd_lb_stats
>> 1,08% [kernel] [k] _raw_spin_lock_irqsave
>> 0,91% librados.so.2.0.0 [.] ceph_crc32c_le_intel
>> 0,91% libpthread-2.13.so [.] pthread_mutex_trylock
>> 0,87% [kernel] [k] resched_task
>> 0,72% [kernel] [k] cpu_startup_entry
>> 0,71% librados.so.2.0.0 [.] crush_hash32_3
>> 0,66% [kernel] [k] leave_mm
>> 0,65% librados.so.2.0.0 [.] Mutex::Lock(bool)
>> 0,64% [kernel] [k] idle_cpu
>> 0,62% libpthread-2.13.so [.] __pthread_mutex_unlock_usercnt
>> 0,59% [kernel] [k] try_to_wake_up
>> 0,56% [kernel] [k] wake_futex
>> 0,50% librados.so.2.0.0 [.] ceph::buffer::ptr::release()
>>
>> firefly:
>> 12,56% libc-2.13.so [.] 0x7905d
>> 2,82% libc-2.13.so [.] malloc
>> 2,64% libc-2.13.so [.] free
>> 2,61% kvm [.] 0x34322f
>> 2,33% [kernel] [k] __schedule
>> 2,14% [kernel] [k] _raw_spin_lock
>> 1,83% [kernel] [k] __switch_to
>> 1,62% [kernel] [k] lapic_next_deadline
>> 1,17% [kernel] [k] _raw_spin_lock_irqsave
>> 1,09% [kernel] [k] update_sd_lb_stats
>> 1,08% libpthread-2.13.so [.] pthread_mutex_trylock
>> 0,85% libpthread-2.13.so [.] __pthread_mutex_unlock_usercnt
>> 0,77% [kernel] [k] resched_task
>> 0,74% librbd.so.1.0.0 [.] 0x71b73
>> 0,72% librados.so.2.0.0 [.] Mutex::Lock(bool)
>> 0,68% librados.so.2.0.0 [.] crush_hash32_3
>> 0,67% [kernel] [k] idle_cpu
>> 0,65% [kernel] [k] leave_mm
>> 0,65% [kernel] [k] cpu_startup_entry
>> 0,59% [kernel] [k] try_to_wake_up
>> 0,51% librados.so.2.0.0 [.] ceph::buffer::ptr::release()
>> 0,51% [kernel] [k] wake_futex
>>
>> Stefan
>>
>> Am 11.02.2015 um 06:42 schrieb Stefan Priebe:
>>>
>>> Am 11.02.2015 um 05:45 schrieb Mark Nelson:
>>>> On 02/10/2015 04:18 PM, Stefan Priebe wrote:
>>>>>
>>>>> Am 10.02.2015 um 22:38 schrieb Mark Nelson:
>>>>>> On 02/10/2015 03:11 PM, Stefan Priebe wrote:
>>>>>>>
>>>>>>> mhm i installed librbd1-dbg and librados2-dbg - but the output still
>>>>>>> looks useless to me. Should i upload it somewhere?
>>>>>>
>>>>>> Meh, if it's all just symbols it's probably not that helpful.
>>>>>>
>>>>>> I've summarized your results here:
>>>>>>
>>>>>> 1 concurrent 4k write (libaio, direct=1, iodepth=1)
>>>>>>
>>>>>> IOPS Latency
>>>>>> wb on wb off wb on wb off
>>>>>> dumpling 10870 536 ~100us ~2ms
>>>>>> firefly 10350 525 ~100us ~2ms
>>>>>>
>>>>>> So in single op tests dumpling and firefly are far closer. Now let's
>>>>>> see each of these cases with iodepth=32 (still 1 thread for now).
>>>>>
>>>>>
>>>>> dumpling:
>>>>>
>>>>> file1: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=32
>>>>> 2.0.8
>>>>> Starting 1 thread
>>>>> Jobs: 1 (f=1): [w] [100.0% done] [0K/72812K /s] [0 /18.3K iops] [eta
>>>>> 00m:00s]
>>>>> file1: (groupid=0, jobs=1): err= 0: pid=3011
>>>>> write: io=2060.6MB, bw=70329KB/s, iops=17582 , runt= 30001msec
>>>>> slat (usec): min=1 , max=3517 , avg= 3.42, stdev= 7.30
>>>>> clat (usec): min=93 , max=7475 , avg=1815.72, stdev=233.43
>>>>> lat (usec): min=219 , max=7477 , avg=1819.27, stdev=233.52
>>>>> clat percentiles (usec):
>>>>> | 1.00th=[ 1480], 5.00th=[ 1576], 10.00th=[ 1608], 20.00th=[
>>>>> 1672],
>>>>> | 30.00th=[ 1704], 40.00th=[ 1752], 50.00th=[ 1800], 60.00th=[
>>>>> 1832],
>>>>> | 70.00th=[ 1896], 80.00th=[ 1960], 90.00th=[ 2064], 95.00th=[
>>>>> 2128],
>>>>> | 99.00th=[ 2352], 99.50th=[ 2448], 99.90th=[ 4704], 99.95th=[
>>>>> 5344],
>>>>> | 99.99th=[ 7072]
>>>>> bw (KB/s) : min=59696, max=77840, per=100.00%, avg=70351.27,
>>>>> stdev=4783.25
>>>>> lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.53%
>>>>> lat (msec) : 2=85.02%, 4=14.31%, 10=0.13%
>>>>> cpu : usr=1.96%, sys=6.71%, ctx=22791, majf=0, minf=133
>>>>> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%,
>>>>>> =64=0.0%
>>>>> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>>>>>> =64=0.0%
>>>>> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
>>>>>> =64=0.0%
>>>>> issued : total=r=0/w=527487/d=0, short=r=0/w=0/d=0
>>>>>
>>>>> Run status group 0 (all jobs):
>>>>> WRITE: io=2060.6MB, aggrb=70329KB/s, minb=70329KB/s, maxb=70329KB/s,
>>>>> mint=30001msec, maxt=30001msec
>>>>>
>>>>> Disk stats (read/write):
>>>>> sdb: ios=166/526079, merge=0/0, ticks=24/890120, in_queue=890064,
>>>>> util=98.73%
>>>>>
>>>>> firefly:
>>>>>
>>>>> file1: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=32
>>>>> 2.0.8
>>>>> Starting 1 thread
>>>>> Jobs: 1 (f=1): [w] [100.0% done] [0K/69096K /s] [0 /17.3K iops] [eta
>>>>> 00m:00s]
>>>>> file1: (groupid=0, jobs=1): err= 0: pid=2982
>>>>> write: io=1784.9MB, bw=60918KB/s, iops=15229 , runt= 30002msec
>>>>> slat (usec): min=1 , max=1389 , avg= 3.43, stdev= 5.32
>>>>> clat (usec): min=117 , max=8235 , avg=2096.88, stdev=396.30
>>>>> lat (usec): min=540 , max=8258 , avg=2100.43, stdev=396.61
>>>>> clat percentiles (usec):
>>>>> | 1.00th=[ 1608], 5.00th=[ 1720], 10.00th=[ 1768], 20.00th=[
>>>>> 1832],
>>>>> | 30.00th=[ 1896], 40.00th=[ 1944], 50.00th=[ 2008], 60.00th=[
>>>>> 2064],
>>>>> | 70.00th=[ 2160], 80.00th=[ 2256], 90.00th=[ 2512], 95.00th=[
>>>>> 2896],
>>>>> | 99.00th=[ 3600], 99.50th=[ 3792], 99.90th=[ 5088], 99.95th=[
>>>>> 6304],
>>>>> | 99.99th=[ 6752]
>>>>> bw (KB/s) : min=36717, max=73712, per=99.94%, avg=60879.92,
>>>>> stdev=8302.27
>>>>> lat (usec) : 250=0.01%, 750=0.01%
>>>>> lat (msec) : 2=48.56%, 4=51.18%, 10=0.26%
>>>>> cpu : usr=2.03%, sys=5.48%, ctx=20440, majf=0, minf=133
>>>>> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%,
>>>>>> =64=0.0%
>>>>> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>>>>>> =64=0.0%
>>>>> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
>>>>>> =64=0.0%
>>>>> issued : total=r=0/w=456918/d=0, short=r=0/w=0/d=0
>>>>>
>>>>> Run status group 0 (all jobs):
>>>>> WRITE: io=1784.9MB, aggrb=60918KB/s, minb=60918KB/s, maxb=60918KB/s,
>>>>> mint=30002msec, maxt=30002msec
>>>>>
>>>>> Disk stats (read/write):
>>>>> sdb: ios=166/455574, merge=0/0, ticks=12/897748, in_queue=897696,
>>>>> util=98.96%
>>>>>
>>>>
>>>> Ok, so it looks like as you increase concurrency the effect increases
>>>> (ie contention?). Does the same thing happen without cache enabled?
>>>
>>> here again without rbd cache:
>>>
>>> dumpling:
>>> file1: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=32
>>> 2.0.8
>>> Starting 1 thread
>>> Jobs: 1 (f=1): [w] [100.0% done] [0K/83488K /s] [0 /20.9K iops] [eta
>>> 00m:00s]
>>> file1: (groupid=0, jobs=1): err= 0: pid=3000
>>> write: io=2449.2MB, bw=83583KB/s, iops=20895 , runt= 30005msec
>>> slat (usec): min=1 , max=975 , avg= 4.50, stdev= 5.25
>>> clat (usec): min=364 , max=80566 , avg=1525.87, stdev=1194.57
>>> lat (usec): min=519 , max=80568 , avg=1530.51, stdev=1194.44
>>> clat percentiles (usec):
>>> | 1.00th=[ 660], 5.00th=[ 780], 10.00th=[ 876], 20.00th=[ 1032],
>>> | 30.00th=[ 1144], 40.00th=[ 1240], 50.00th=[ 1304], 60.00th=[ 1384],
>>> | 70.00th=[ 1480], 80.00th=[ 1640], 90.00th=[ 2096], 95.00th=[ 2960],
>>> | 99.00th=[ 6816], 99.50th=[ 7840], 99.90th=[11712], 99.95th=[13888],
>>> | 99.99th=[18816]
>>> bw (KB/s) : min=47184, max=95432, per=100.00%, avg=83639.19,
>>> stdev=7973.92
>>> lat (usec) : 500=0.01%, 750=3.82%, 1000=14.40%
>>> lat (msec) : 2=70.57%, 4=7.91%, 10=3.11%, 20=0.17%, 50=0.01%
>>> lat (msec) : 100=0.01%
>>> cpu : usr=3.12%, sys=11.49%, ctx=74951, majf=0, minf=133
>>> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%,
>>>> =64=0.0%
>>> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>>>> =64=0.0%
>>> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
>>>> =64=0.0%
>>> issued : total=r=0/w=626979/d=0, short=r=0/w=0/d=0
>>>
>>> Run status group 0 (all jobs):
>>> WRITE: io=2449.2MB, aggrb=83583KB/s, minb=83583KB/s, maxb=83583KB/s,
>>> mint=30005msec, maxt=30005msec
>>>
>>> Disk stats (read/write):
>>> sdb: ios=168/625292, merge=0/0, ticks=144/916096, in_queue=916128,
>>> util=99.93%
>>>
>>>
>>> firefly:
>>>
>>> fio --filename=/dev/sdb --direct=1 --rw=randwrite --bs=4k --numjobs=1
>>> --thread --iodepth=32 --ioengine=libaio --runtime=30 --group_reporting
>>> --name=file1
>>> file1: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=32
>>> 2.0.8
>>> Starting 1 thread
>>> Jobs: 1 (f=1): [w] [100.0% done] [0K/90044K /s] [0 /22.6K iops] [eta
>>> 00m:00s]
>>> file1: (groupid=0, jobs=1): err= 0: pid=2970
>>> write: io=2372.9MB, bw=80976KB/s, iops=20244 , runt= 30006msec
>>> slat (usec): min=1 , max=4047 , avg= 4.36, stdev= 7.17
>>> clat (usec): min=197 , max=76656 , avg=1575.29, stdev=1165.74
>>> lat (usec): min=523 , max=76660 , avg=1579.79, stdev=1165.59
>>> clat percentiles (usec):
>>> | 1.00th=[ 676], 5.00th=[ 804], 10.00th=[ 916], 20.00th=[ 1096],
>>> | 30.00th=[ 1224], 40.00th=[ 1304], 50.00th=[ 1384], 60.00th=[ 1448],
>>> | 70.00th=[ 1544], 80.00th=[ 1704], 90.00th=[ 2128], 95.00th=[ 2736],
>>> | 99.00th=[ 6752], 99.50th=[ 7904], 99.90th=[12096], 99.95th=[14656],
>>> | 99.99th=[18560]
>>> bw (KB/s) : min=47800, max=91952, per=99.91%, avg=80900.88,
>>> stdev=7234.98
>>> lat (usec) : 250=0.01%, 500=0.01%, 750=2.95%, 1000=11.38%
>>> lat (msec) : 2=73.81%, 4=8.81%, 10=2.85%, 20=0.19%, 50=0.01%
>>> lat (msec) : 100=0.01%
>>> cpu : usr=2.99%, sys=10.60%, ctx=66549, majf=0, minf=133
>>> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%,
>>>> =64=0.0%
>>> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>>>> =64=0.0%
>>> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
>>>> =64=0.0%
>>> issued : total=r=0/w=607445/d=0, short=r=0/w=0/d=0
>>>
>>> Run status group 0 (all jobs):
>>> WRITE: io=2372.9MB, aggrb=80976KB/s, minb=80976KB/s, maxb=80976KB/s,
>>> mint=30006msec, maxt=30006msec
>>>
>>> Disk stats (read/write):
>>> sdb: ios=170/605440, merge=0/0, ticks=156/916492, in_queue=916560,
>>> util=99.93%
>>>
>>> Stefan
>>>
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: speed decrease since firefly,giant,hammer the 2nd try
[not found] ` <2015376321.1005004.1424124122884.JavaMail.zimbra@oxygem.tv>
@ 2015-02-16 22:02 ` Alexandre DERUMIER
2015-02-16 22:08 ` Stefan Priebe - Profihost AG
0 siblings, 1 reply; 32+ messages in thread
From: Alexandre DERUMIER @ 2015-02-16 22:02 UTC (permalink / raw)
To: Stefan Priebe; +Cc: Mark Nelson, ceph-devel
>>This results in fio-rbd showing avg 26000 iop/s instead of 30500 iop/s
>>while running dumpling...
Is it for write only ?
or do you see same decrease for read too ?
----- Mail original -----
De: "Stefan Priebe" <s.priebe@profihost.ag>
À: "Mark Nelson" <mnelson@redhat.com>, "ceph-devel" <ceph-devel@vger.kernel.org>
Envoyé: Lundi 16 Février 2015 22:22:01
Objet: Re: speed decrease since firefly,giant,hammer the 2nd try
I've now upgraded server side and client side to latest upstream/firefly.
This results in fio-rbd showing avg 26000 iop/s instead of 30500 iop/s
while running dumpling...
Greets,
Stefan
Am 15.02.2015 um 19:40 schrieb Stefan Priebe:
> Hi Mark,
>
> what's next?
>
> I've this test cluster only for 2 more days.
>
> Here some perf Details:
>
> dumpling:
> 12,65% libc-2.13.so [.] 0x79000
> 2,86% libc-2.13.so [.] malloc
> 2,80% kvm [.] 0xb59c5
> 2,59% libc-2.13.so [.] free
> 2,35% [kernel] [k] __schedule
> 2,16% [kernel] [k] _raw_spin_lock
> 1,92% [kernel] [k] __switch_to
> 1,58% [kernel] [k] lapic_next_deadline
> 1,09% [kernel] [k] update_sd_lb_stats
> 1,08% [kernel] [k] _raw_spin_lock_irqsave
> 0,91% librados.so.2.0.0 [.] ceph_crc32c_le_intel
> 0,91% libpthread-2.13.so [.] pthread_mutex_trylock
> 0,87% [kernel] [k] resched_task
> 0,72% [kernel] [k] cpu_startup_entry
> 0,71% librados.so.2.0.0 [.] crush_hash32_3
> 0,66% [kernel] [k] leave_mm
> 0,65% librados.so.2.0.0 [.] Mutex::Lock(bool)
> 0,64% [kernel] [k] idle_cpu
> 0,62% libpthread-2.13.so [.] __pthread_mutex_unlock_usercnt
> 0,59% [kernel] [k] try_to_wake_up
> 0,56% [kernel] [k] wake_futex
> 0,50% librados.so.2.0.0 [.] ceph::buffer::ptr::release()
>
> firefly:
> 12,56% libc-2.13.so [.] 0x7905d
> 2,82% libc-2.13.so [.] malloc
> 2,64% libc-2.13.so [.] free
> 2,61% kvm [.] 0x34322f
> 2,33% [kernel] [k] __schedule
> 2,14% [kernel] [k] _raw_spin_lock
> 1,83% [kernel] [k] __switch_to
> 1,62% [kernel] [k] lapic_next_deadline
> 1,17% [kernel] [k] _raw_spin_lock_irqsave
> 1,09% [kernel] [k] update_sd_lb_stats
> 1,08% libpthread-2.13.so [.] pthread_mutex_trylock
> 0,85% libpthread-2.13.so [.] __pthread_mutex_unlock_usercnt
> 0,77% [kernel] [k] resched_task
> 0,74% librbd.so.1.0.0 [.] 0x71b73
> 0,72% librados.so.2.0.0 [.] Mutex::Lock(bool)
> 0,68% librados.so.2.0.0 [.] crush_hash32_3
> 0,67% [kernel] [k] idle_cpu
> 0,65% [kernel] [k] leave_mm
> 0,65% [kernel] [k] cpu_startup_entry
> 0,59% [kernel] [k] try_to_wake_up
> 0,51% librados.so.2.0.0 [.] ceph::buffer::ptr::release()
> 0,51% [kernel] [k] wake_futex
>
> Stefan
>
> Am 11.02.2015 um 06:42 schrieb Stefan Priebe:
>>
>> Am 11.02.2015 um 05:45 schrieb Mark Nelson:
>>> On 02/10/2015 04:18 PM, Stefan Priebe wrote:
>>>>
>>>> Am 10.02.2015 um 22:38 schrieb Mark Nelson:
>>>>> On 02/10/2015 03:11 PM, Stefan Priebe wrote:
>>>>>>
>>>>>> mhm i installed librbd1-dbg and librados2-dbg - but the output still
>>>>>> looks useless to me. Should i upload it somewhere?
>>>>>
>>>>> Meh, if it's all just symbols it's probably not that helpful.
>>>>>
>>>>> I've summarized your results here:
>>>>>
>>>>> 1 concurrent 4k write (libaio, direct=1, iodepth=1)
>>>>>
>>>>> IOPS Latency
>>>>> wb on wb off wb on wb off
>>>>> dumpling 10870 536 ~100us ~2ms
>>>>> firefly 10350 525 ~100us ~2ms
>>>>>
>>>>> So in single op tests dumpling and firefly are far closer. Now let's
>>>>> see each of these cases with iodepth=32 (still 1 thread for now).
>>>>
>>>>
>>>> dumpling:
>>>>
>>>> file1: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=32
>>>> 2.0.8
>>>> Starting 1 thread
>>>> Jobs: 1 (f=1): [w] [100.0% done] [0K/72812K /s] [0 /18.3K iops] [eta
>>>> 00m:00s]
>>>> file1: (groupid=0, jobs=1): err= 0: pid=3011
>>>> write: io=2060.6MB, bw=70329KB/s, iops=17582 , runt= 30001msec
>>>> slat (usec): min=1 , max=3517 , avg= 3.42, stdev= 7.30
>>>> clat (usec): min=93 , max=7475 , avg=1815.72, stdev=233.43
>>>> lat (usec): min=219 , max=7477 , avg=1819.27, stdev=233.52
>>>> clat percentiles (usec):
>>>> | 1.00th=[ 1480], 5.00th=[ 1576], 10.00th=[ 1608], 20.00th=[
>>>> 1672],
>>>> | 30.00th=[ 1704], 40.00th=[ 1752], 50.00th=[ 1800], 60.00th=[
>>>> 1832],
>>>> | 70.00th=[ 1896], 80.00th=[ 1960], 90.00th=[ 2064], 95.00th=[
>>>> 2128],
>>>> | 99.00th=[ 2352], 99.50th=[ 2448], 99.90th=[ 4704], 99.95th=[
>>>> 5344],
>>>> | 99.99th=[ 7072]
>>>> bw (KB/s) : min=59696, max=77840, per=100.00%, avg=70351.27,
>>>> stdev=4783.25
>>>> lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%,
>>>> 1000=0.53%
>>>> lat (msec) : 2=85.02%, 4=14.31%, 10=0.13%
>>>> cpu : usr=1.96%, sys=6.71%, ctx=22791, majf=0, minf=133
>>>> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%,
>>>> >=64=0.0%
>>>> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>>>> >=64=0.0%
>>>> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
>>>> >=64=0.0%
>>>> issued : total=r=0/w=527487/d=0, short=r=0/w=0/d=0
>>>>
>>>> Run status group 0 (all jobs):
>>>> WRITE: io=2060.6MB, aggrb=70329KB/s, minb=70329KB/s, maxb=70329KB/s,
>>>> mint=30001msec, maxt=30001msec
>>>>
>>>> Disk stats (read/write):
>>>> sdb: ios=166/526079, merge=0/0, ticks=24/890120, in_queue=890064,
>>>> util=98.73%
>>>>
>>>> firefly:
>>>>
>>>> file1: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=32
>>>> 2.0.8
>>>> Starting 1 thread
>>>> Jobs: 1 (f=1): [w] [100.0% done] [0K/69096K /s] [0 /17.3K iops] [eta
>>>> 00m:00s]
>>>> file1: (groupid=0, jobs=1): err= 0: pid=2982
>>>> write: io=1784.9MB, bw=60918KB/s, iops=15229 , runt= 30002msec
>>>> slat (usec): min=1 , max=1389 , avg= 3.43, stdev= 5.32
>>>> clat (usec): min=117 , max=8235 , avg=2096.88, stdev=396.30
>>>> lat (usec): min=540 , max=8258 , avg=2100.43, stdev=396.61
>>>> clat percentiles (usec):
>>>> | 1.00th=[ 1608], 5.00th=[ 1720], 10.00th=[ 1768], 20.00th=[
>>>> 1832],
>>>> | 30.00th=[ 1896], 40.00th=[ 1944], 50.00th=[ 2008], 60.00th=[
>>>> 2064],
>>>> | 70.00th=[ 2160], 80.00th=[ 2256], 90.00th=[ 2512], 95.00th=[
>>>> 2896],
>>>> | 99.00th=[ 3600], 99.50th=[ 3792], 99.90th=[ 5088], 99.95th=[
>>>> 6304],
>>>> | 99.99th=[ 6752]
>>>> bw (KB/s) : min=36717, max=73712, per=99.94%, avg=60879.92,
>>>> stdev=8302.27
>>>> lat (usec) : 250=0.01%, 750=0.01%
>>>> lat (msec) : 2=48.56%, 4=51.18%, 10=0.26%
>>>> cpu : usr=2.03%, sys=5.48%, ctx=20440, majf=0, minf=133
>>>> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%,
>>>> >=64=0.0%
>>>> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>>>> >=64=0.0%
>>>> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
>>>> >=64=0.0%
>>>> issued : total=r=0/w=456918/d=0, short=r=0/w=0/d=0
>>>>
>>>> Run status group 0 (all jobs):
>>>> WRITE: io=1784.9MB, aggrb=60918KB/s, minb=60918KB/s, maxb=60918KB/s,
>>>> mint=30002msec, maxt=30002msec
>>>>
>>>> Disk stats (read/write):
>>>> sdb: ios=166/455574, merge=0/0, ticks=12/897748, in_queue=897696,
>>>> util=98.96%
>>>>
>>>
>>> Ok, so it looks like as you increase concurrency the effect increases
>>> (ie contention?). Does the same thing happen without cache enabled?
>>
>> here again without rbd cache:
>>
>> dumpling:
>> file1: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=32
>> 2.0.8
>> Starting 1 thread
>> Jobs: 1 (f=1): [w] [100.0% done] [0K/83488K /s] [0 /20.9K iops] [eta
>> 00m:00s]
>> file1: (groupid=0, jobs=1): err= 0: pid=3000
>> write: io=2449.2MB, bw=83583KB/s, iops=20895 , runt= 30005msec
>> slat (usec): min=1 , max=975 , avg= 4.50, stdev= 5.25
>> clat (usec): min=364 , max=80566 , avg=1525.87, stdev=1194.57
>> lat (usec): min=519 , max=80568 , avg=1530.51, stdev=1194.44
>> clat percentiles (usec):
>> | 1.00th=[ 660], 5.00th=[ 780], 10.00th=[ 876], 20.00th=[
>> 1032],
>> | 30.00th=[ 1144], 40.00th=[ 1240], 50.00th=[ 1304], 60.00th=[
>> 1384],
>> | 70.00th=[ 1480], 80.00th=[ 1640], 90.00th=[ 2096], 95.00th=[
>> 2960],
>> | 99.00th=[ 6816], 99.50th=[ 7840], 99.90th=[11712],
>> 99.95th=[13888],
>> | 99.99th=[18816]
>> bw (KB/s) : min=47184, max=95432, per=100.00%, avg=83639.19,
>> stdev=7973.92
>> lat (usec) : 500=0.01%, 750=3.82%, 1000=14.40%
>> lat (msec) : 2=70.57%, 4=7.91%, 10=3.11%, 20=0.17%, 50=0.01%
>> lat (msec) : 100=0.01%
>> cpu : usr=3.12%, sys=11.49%, ctx=74951, majf=0, minf=133
>> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%,
>> >=64=0.0%
>> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>> >=64=0.0%
>> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
>> >=64=0.0%
>> issued : total=r=0/w=626979/d=0, short=r=0/w=0/d=0
>>
>> Run status group 0 (all jobs):
>> WRITE: io=2449.2MB, aggrb=83583KB/s, minb=83583KB/s, maxb=83583KB/s,
>> mint=30005msec, maxt=30005msec
>>
>> Disk stats (read/write):
>> sdb: ios=168/625292, merge=0/0, ticks=144/916096, in_queue=916128,
>> util=99.93%
>>
>>
>> firefly:
>>
>> fio --filename=/dev/sdb --direct=1 --rw=randwrite --bs=4k --numjobs=1
>> --thread --iodepth=32 --ioengine=libaio --runtime=30 --group_reporting
>> --name=file1
>> file1: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=32
>> 2.0.8
>> Starting 1 thread
>> Jobs: 1 (f=1): [w] [100.0% done] [0K/90044K /s] [0 /22.6K iops] [eta
>> 00m:00s]
>> file1: (groupid=0, jobs=1): err= 0: pid=2970
>> write: io=2372.9MB, bw=80976KB/s, iops=20244 , runt= 30006msec
>> slat (usec): min=1 , max=4047 , avg= 4.36, stdev= 7.17
>> clat (usec): min=197 , max=76656 , avg=1575.29, stdev=1165.74
>> lat (usec): min=523 , max=76660 , avg=1579.79, stdev=1165.59
>> clat percentiles (usec):
>> | 1.00th=[ 676], 5.00th=[ 804], 10.00th=[ 916], 20.00th=[
>> 1096],
>> | 30.00th=[ 1224], 40.00th=[ 1304], 50.00th=[ 1384], 60.00th=[
>> 1448],
>> | 70.00th=[ 1544], 80.00th=[ 1704], 90.00th=[ 2128], 95.00th=[
>> 2736],
>> | 99.00th=[ 6752], 99.50th=[ 7904], 99.90th=[12096],
>> 99.95th=[14656],
>> | 99.99th=[18560]
>> bw (KB/s) : min=47800, max=91952, per=99.91%, avg=80900.88,
>> stdev=7234.98
>> lat (usec) : 250=0.01%, 500=0.01%, 750=2.95%, 1000=11.38%
>> lat (msec) : 2=73.81%, 4=8.81%, 10=2.85%, 20=0.19%, 50=0.01%
>> lat (msec) : 100=0.01%
>> cpu : usr=2.99%, sys=10.60%, ctx=66549, majf=0, minf=133
>> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%,
>> >=64=0.0%
>> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>> >=64=0.0%
>> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
>> >=64=0.0%
>> issued : total=r=0/w=607445/d=0, short=r=0/w=0/d=0
>>
>> Run status group 0 (all jobs):
>> WRITE: io=2372.9MB, aggrb=80976KB/s, minb=80976KB/s, maxb=80976KB/s,
>> mint=30006msec, maxt=30006msec
>>
>> Disk stats (read/write):
>> sdb: ios=170/605440, merge=0/0, ticks=156/916492, in_queue=916560,
>> util=99.93%
>>
>> Stefan
>>
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: speed decrease since firefly,giant,hammer the 2nd try
2015-02-16 22:02 ` Alexandre DERUMIER
@ 2015-02-16 22:08 ` Stefan Priebe - Profihost AG
[not found] ` <1975903570.1005549.1424125095286.JavaMail.zimbra@oxygem.tv>
0 siblings, 1 reply; 32+ messages in thread
From: Stefan Priebe - Profihost AG @ 2015-02-16 22:08 UTC (permalink / raw)
To: Alexandre DERUMIER; +Cc: Mark Nelson, ceph-devel
Am 16.02.2015 um 23:02 schrieb Alexandre DERUMIER <aderumier@odiso.com>:
>>> This results in fio-rbd showing avg 26000 iop/s instead of 30500 iop/s
>>> while running dumpling...
>
> Is it for write only ?
> or do you see same decrease for read too
Just tested write. This might be the result of higher CPU load of the ceph-osd processes under firefly.
Dumpling 180% per process vs. firefly 220%
Stefan
> ?
>
>
> ----- Mail original -----
> De: "Stefan Priebe" <s.priebe@profihost.ag>
> À: "Mark Nelson" <mnelson@redhat.com>, "ceph-devel" <ceph-devel@vger.kernel.org>
> Envoyé: Lundi 16 Février 2015 22:22:01
> Objet: Re: speed decrease since firefly,giant,hammer the 2nd try
>
> I've now upgraded server side and client side to latest upstream/firefly.
>
> This results in fio-rbd showing avg 26000 iop/s instead of 30500 iop/s
> while running dumpling...
>
> Greets,
> Stefan
>> Am 15.02.2015 um 19:40 schrieb Stefan Priebe:
>> Hi Mark,
>>
>> what's next?
>>
>> I've this test cluster only for 2 more days.
>>
>> Here some perf Details:
>>
>> dumpling:
>> 12,65% libc-2.13.so [.] 0x79000
>> 2,86% libc-2.13.so [.] malloc
>> 2,80% kvm [.] 0xb59c5
>> 2,59% libc-2.13.so [.] free
>> 2,35% [kernel] [k] __schedule
>> 2,16% [kernel] [k] _raw_spin_lock
>> 1,92% [kernel] [k] __switch_to
>> 1,58% [kernel] [k] lapic_next_deadline
>> 1,09% [kernel] [k] update_sd_lb_stats
>> 1,08% [kernel] [k] _raw_spin_lock_irqsave
>> 0,91% librados.so.2.0.0 [.] ceph_crc32c_le_intel
>> 0,91% libpthread-2.13.so [.] pthread_mutex_trylock
>> 0,87% [kernel] [k] resched_task
>> 0,72% [kernel] [k] cpu_startup_entry
>> 0,71% librados.so.2.0.0 [.] crush_hash32_3
>> 0,66% [kernel] [k] leave_mm
>> 0,65% librados.so.2.0.0 [.] Mutex::Lock(bool)
>> 0,64% [kernel] [k] idle_cpu
>> 0,62% libpthread-2.13.so [.] __pthread_mutex_unlock_usercnt
>> 0,59% [kernel] [k] try_to_wake_up
>> 0,56% [kernel] [k] wake_futex
>> 0,50% librados.so.2.0.0 [.] ceph::buffer::ptr::release()
>>
>> firefly:
>> 12,56% libc-2.13.so [.] 0x7905d
>> 2,82% libc-2.13.so [.] malloc
>> 2,64% libc-2.13.so [.] free
>> 2,61% kvm [.] 0x34322f
>> 2,33% [kernel] [k] __schedule
>> 2,14% [kernel] [k] _raw_spin_lock
>> 1,83% [kernel] [k] __switch_to
>> 1,62% [kernel] [k] lapic_next_deadline
>> 1,17% [kernel] [k] _raw_spin_lock_irqsave
>> 1,09% [kernel] [k] update_sd_lb_stats
>> 1,08% libpthread-2.13.so [.] pthread_mutex_trylock
>> 0,85% libpthread-2.13.so [.] __pthread_mutex_unlock_usercnt
>> 0,77% [kernel] [k] resched_task
>> 0,74% librbd.so.1.0.0 [.] 0x71b73
>> 0,72% librados.so.2.0.0 [.] Mutex::Lock(bool)
>> 0,68% librados.so.2.0.0 [.] crush_hash32_3
>> 0,67% [kernel] [k] idle_cpu
>> 0,65% [kernel] [k] leave_mm
>> 0,65% [kernel] [k] cpu_startup_entry
>> 0,59% [kernel] [k] try_to_wake_up
>> 0,51% librados.so.2.0.0 [.] ceph::buffer::ptr::release()
>> 0,51% [kernel] [k] wake_futex
>>
>> Stefan
>>
>>> Am 11.02.2015 um 06:42 schrieb Stefan Priebe:
>>>
>>>> Am 11.02.2015 um 05:45 schrieb Mark Nelson:
>>>>> On 02/10/2015 04:18 PM, Stefan Priebe wrote:
>>>>>
>>>>>> Am 10.02.2015 um 22:38 schrieb Mark Nelson:
>>>>>>> On 02/10/2015 03:11 PM, Stefan Priebe wrote:
>>>>>>>
>>>>>>> mhm i installed librbd1-dbg and librados2-dbg - but the output still
>>>>>>> looks useless to me. Should i upload it somewhere?
>>>>>>
>>>>>> Meh, if it's all just symbols it's probably not that helpful.
>>>>>>
>>>>>> I've summarized your results here:
>>>>>>
>>>>>> 1 concurrent 4k write (libaio, direct=1, iodepth=1)
>>>>>>
>>>>>> IOPS Latency
>>>>>> wb on wb off wb on wb off
>>>>>> dumpling 10870 536 ~100us ~2ms
>>>>>> firefly 10350 525 ~100us ~2ms
>>>>>>
>>>>>> So in single op tests dumpling and firefly are far closer. Now let's
>>>>>> see each of these cases with iodepth=32 (still 1 thread for now).
>>>>>
>>>>>
>>>>> dumpling:
>>>>>
>>>>> file1: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=32
>>>>> 2.0.8
>>>>> Starting 1 thread
>>>>> Jobs: 1 (f=1): [w] [100.0% done] [0K/72812K /s] [0 /18.3K iops] [eta
>>>>> 00m:00s]
>>>>> file1: (groupid=0, jobs=1): err= 0: pid=3011
>>>>> write: io=2060.6MB, bw=70329KB/s, iops=17582 , runt= 30001msec
>>>>> slat (usec): min=1 , max=3517 , avg= 3.42, stdev= 7.30
>>>>> clat (usec): min=93 , max=7475 , avg=1815.72, stdev=233.43
>>>>> lat (usec): min=219 , max=7477 , avg=1819.27, stdev=233.52
>>>>> clat percentiles (usec):
>>>>> | 1.00th=[ 1480], 5.00th=[ 1576], 10.00th=[ 1608], 20.00th=[
>>>>> 1672],
>>>>> | 30.00th=[ 1704], 40.00th=[ 1752], 50.00th=[ 1800], 60.00th=[
>>>>> 1832],
>>>>> | 70.00th=[ 1896], 80.00th=[ 1960], 90.00th=[ 2064], 95.00th=[
>>>>> 2128],
>>>>> | 99.00th=[ 2352], 99.50th=[ 2448], 99.90th=[ 4704], 99.95th=[
>>>>> 5344],
>>>>> | 99.99th=[ 7072]
>>>>> bw (KB/s) : min=59696, max=77840, per=100.00%, avg=70351.27,
>>>>> stdev=4783.25
>>>>> lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%,
>>>>> 1000=0.53%
>>>>> lat (msec) : 2=85.02%, 4=14.31%, 10=0.13%
>>>>> cpu : usr=1.96%, sys=6.71%, ctx=22791, majf=0, minf=133
>>>>> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%,
>>>>>> =64=0.0%
>>>>> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>>>>>> =64=0.0%
>>>>> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
>>>>>> =64=0.0%
>>>>> issued : total=r=0/w=527487/d=0, short=r=0/w=0/d=0
>>>>>
>>>>> Run status group 0 (all jobs):
>>>>> WRITE: io=2060.6MB, aggrb=70329KB/s, minb=70329KB/s, maxb=70329KB/s,
>>>>> mint=30001msec, maxt=30001msec
>>>>>
>>>>> Disk stats (read/write):
>>>>> sdb: ios=166/526079, merge=0/0, ticks=24/890120, in_queue=890064,
>>>>> util=98.73%
>>>>>
>>>>> firefly:
>>>>>
>>>>> file1: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=32
>>>>> 2.0.8
>>>>> Starting 1 thread
>>>>> Jobs: 1 (f=1): [w] [100.0% done] [0K/69096K /s] [0 /17.3K iops] [eta
>>>>> 00m:00s]
>>>>> file1: (groupid=0, jobs=1): err= 0: pid=2982
>>>>> write: io=1784.9MB, bw=60918KB/s, iops=15229 , runt= 30002msec
>>>>> slat (usec): min=1 , max=1389 , avg= 3.43, stdev= 5.32
>>>>> clat (usec): min=117 , max=8235 , avg=2096.88, stdev=396.30
>>>>> lat (usec): min=540 , max=8258 , avg=2100.43, stdev=396.61
>>>>> clat percentiles (usec):
>>>>> | 1.00th=[ 1608], 5.00th=[ 1720], 10.00th=[ 1768], 20.00th=[
>>>>> 1832],
>>>>> | 30.00th=[ 1896], 40.00th=[ 1944], 50.00th=[ 2008], 60.00th=[
>>>>> 2064],
>>>>> | 70.00th=[ 2160], 80.00th=[ 2256], 90.00th=[ 2512], 95.00th=[
>>>>> 2896],
>>>>> | 99.00th=[ 3600], 99.50th=[ 3792], 99.90th=[ 5088], 99.95th=[
>>>>> 6304],
>>>>> | 99.99th=[ 6752]
>>>>> bw (KB/s) : min=36717, max=73712, per=99.94%, avg=60879.92,
>>>>> stdev=8302.27
>>>>> lat (usec) : 250=0.01%, 750=0.01%
>>>>> lat (msec) : 2=48.56%, 4=51.18%, 10=0.26%
>>>>> cpu : usr=2.03%, sys=5.48%, ctx=20440, majf=0, minf=133
>>>>> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%,
>>>>>> =64=0.0%
>>>>> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>>>>>> =64=0.0%
>>>>> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
>>>>>> =64=0.0%
>>>>> issued : total=r=0/w=456918/d=0, short=r=0/w=0/d=0
>>>>>
>>>>> Run status group 0 (all jobs):
>>>>> WRITE: io=1784.9MB, aggrb=60918KB/s, minb=60918KB/s, maxb=60918KB/s,
>>>>> mint=30002msec, maxt=30002msec
>>>>>
>>>>> Disk stats (read/write):
>>>>> sdb: ios=166/455574, merge=0/0, ticks=12/897748, in_queue=897696,
>>>>> util=98.96%
>>>>
>>>> Ok, so it looks like as you increase concurrency the effect increases
>>>> (ie contention?). Does the same thing happen without cache enabled?
>>>
>>> here again without rbd cache:
>>>
>>> dumpling:
>>> file1: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=32
>>> 2.0.8
>>> Starting 1 thread
>>> Jobs: 1 (f=1): [w] [100.0% done] [0K/83488K /s] [0 /20.9K iops] [eta
>>> 00m:00s]
>>> file1: (groupid=0, jobs=1): err= 0: pid=3000
>>> write: io=2449.2MB, bw=83583KB/s, iops=20895 , runt= 30005msec
>>> slat (usec): min=1 , max=975 , avg= 4.50, stdev= 5.25
>>> clat (usec): min=364 , max=80566 , avg=1525.87, stdev=1194.57
>>> lat (usec): min=519 , max=80568 , avg=1530.51, stdev=1194.44
>>> clat percentiles (usec):
>>> | 1.00th=[ 660], 5.00th=[ 780], 10.00th=[ 876], 20.00th=[
>>> 1032],
>>> | 30.00th=[ 1144], 40.00th=[ 1240], 50.00th=[ 1304], 60.00th=[
>>> 1384],
>>> | 70.00th=[ 1480], 80.00th=[ 1640], 90.00th=[ 2096], 95.00th=[
>>> 2960],
>>> | 99.00th=[ 6816], 99.50th=[ 7840], 99.90th=[11712],
>>> 99.95th=[13888],
>>> | 99.99th=[18816]
>>> bw (KB/s) : min=47184, max=95432, per=100.00%, avg=83639.19,
>>> stdev=7973.92
>>> lat (usec) : 500=0.01%, 750=3.82%, 1000=14.40%
>>> lat (msec) : 2=70.57%, 4=7.91%, 10=3.11%, 20=0.17%, 50=0.01%
>>> lat (msec) : 100=0.01%
>>> cpu : usr=3.12%, sys=11.49%, ctx=74951, majf=0, minf=133
>>> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%,
>>>> =64=0.0%
>>> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>>>> =64=0.0%
>>> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
>>>> =64=0.0%
>>> issued : total=r=0/w=626979/d=0, short=r=0/w=0/d=0
>>>
>>> Run status group 0 (all jobs):
>>> WRITE: io=2449.2MB, aggrb=83583KB/s, minb=83583KB/s, maxb=83583KB/s,
>>> mint=30005msec, maxt=30005msec
>>>
>>> Disk stats (read/write):
>>> sdb: ios=168/625292, merge=0/0, ticks=144/916096, in_queue=916128,
>>> util=99.93%
>>>
>>>
>>> firefly:
>>>
>>> fio --filename=/dev/sdb --direct=1 --rw=randwrite --bs=4k --numjobs=1
>>> --thread --iodepth=32 --ioengine=libaio --runtime=30 --group_reporting
>>> --name=file1
>>> file1: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=32
>>> 2.0.8
>>> Starting 1 thread
>>> Jobs: 1 (f=1): [w] [100.0% done] [0K/90044K /s] [0 /22.6K iops] [eta
>>> 00m:00s]
>>> file1: (groupid=0, jobs=1): err= 0: pid=2970
>>> write: io=2372.9MB, bw=80976KB/s, iops=20244 , runt= 30006msec
>>> slat (usec): min=1 , max=4047 , avg= 4.36, stdev= 7.17
>>> clat (usec): min=197 , max=76656 , avg=1575.29, stdev=1165.74
>>> lat (usec): min=523 , max=76660 , avg=1579.79, stdev=1165.59
>>> clat percentiles (usec):
>>> | 1.00th=[ 676], 5.00th=[ 804], 10.00th=[ 916], 20.00th=[
>>> 1096],
>>> | 30.00th=[ 1224], 40.00th=[ 1304], 50.00th=[ 1384], 60.00th=[
>>> 1448],
>>> | 70.00th=[ 1544], 80.00th=[ 1704], 90.00th=[ 2128], 95.00th=[
>>> 2736],
>>> | 99.00th=[ 6752], 99.50th=[ 7904], 99.90th=[12096],
>>> 99.95th=[14656],
>>> | 99.99th=[18560]
>>> bw (KB/s) : min=47800, max=91952, per=99.91%, avg=80900.88,
>>> stdev=7234.98
>>> lat (usec) : 250=0.01%, 500=0.01%, 750=2.95%, 1000=11.38%
>>> lat (msec) : 2=73.81%, 4=8.81%, 10=2.85%, 20=0.19%, 50=0.01%
>>> lat (msec) : 100=0.01%
>>> cpu : usr=2.99%, sys=10.60%, ctx=66549, majf=0, minf=133
>>> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%,
>>>> =64=0.0%
>>> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>>>> =64=0.0%
>>> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
>>>> =64=0.0%
>>> issued : total=r=0/w=607445/d=0, short=r=0/w=0/d=0
>>>
>>> Run status group 0 (all jobs):
>>> WRITE: io=2372.9MB, aggrb=80976KB/s, minb=80976KB/s, maxb=80976KB/s,
>>> mint=30006msec, maxt=30006msec
>>>
>>> Disk stats (read/write):
>>> sdb: ios=170/605440, merge=0/0, ticks=156/916492, in_queue=916560,
>>> util=99.93%
>>>
>>> Stefan
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: speed decrease since firefly,giant,hammer the 2nd try
[not found] ` <1975903570.1005549.1424125095286.JavaMail.zimbra@oxygem.tv>
@ 2015-02-16 22:18 ` Alexandre DERUMIER
2015-02-17 9:15 ` Stefan Priebe - Profihost AG
0 siblings, 1 reply; 32+ messages in thread
From: Alexandre DERUMIER @ 2015-02-16 22:18 UTC (permalink / raw)
To: Stefan Priebe; +Cc: Mark Nelson, ceph-devel
>>Just tested write. This might be the result of higher CPU load of the ceph-osd processes under firefly.
>>
>>Dumpling 180% per process vs. firefly 220%
Oh yes, indeed, that's what I think too. (and more cpu -> less ios in qemu because of single iothread)
I think that perf report should show the difference, this is strange that you have almost same result in your perf reports.
----- Mail original -----
De: "Stefan Priebe" <s.priebe@profihost.ag>
À: "aderumier" <aderumier@odiso.com>
Cc: "Mark Nelson" <mnelson@redhat.com>, "ceph-devel" <ceph-devel@vger.kernel.org>
Envoyé: Lundi 16 Février 2015 23:08:37
Objet: Re: speed decrease since firefly,giant,hammer the 2nd try
Am 16.02.2015 um 23:02 schrieb Alexandre DERUMIER <aderumier@odiso.com>:
>>> This results in fio-rbd showing avg 26000 iop/s instead of 30500 iop/s
>>> while running dumpling...
>
> Is it for write only ?
> or do you see same decrease for read too
Just tested write. This might be the result of higher CPU load of the ceph-osd processes under firefly.
Dumpling 180% per process vs. firefly 220%
Stefan
> ?
>
>
> ----- Mail original -----
> De: "Stefan Priebe" <s.priebe@profihost.ag>
> À: "Mark Nelson" <mnelson@redhat.com>, "ceph-devel" <ceph-devel@vger.kernel.org>
> Envoyé: Lundi 16 Février 2015 22:22:01
> Objet: Re: speed decrease since firefly,giant,hammer the 2nd try
>
> I've now upgraded server side and client side to latest upstream/firefly.
>
> This results in fio-rbd showing avg 26000 iop/s instead of 30500 iop/s
> while running dumpling...
>
> Greets,
> Stefan
>> Am 15.02.2015 um 19:40 schrieb Stefan Priebe:
>> Hi Mark,
>>
>> what's next?
>>
>> I've this test cluster only for 2 more days.
>>
>> Here some perf Details:
>>
>> dumpling:
>> 12,65% libc-2.13.so [.] 0x79000
>> 2,86% libc-2.13.so [.] malloc
>> 2,80% kvm [.] 0xb59c5
>> 2,59% libc-2.13.so [.] free
>> 2,35% [kernel] [k] __schedule
>> 2,16% [kernel] [k] _raw_spin_lock
>> 1,92% [kernel] [k] __switch_to
>> 1,58% [kernel] [k] lapic_next_deadline
>> 1,09% [kernel] [k] update_sd_lb_stats
>> 1,08% [kernel] [k] _raw_spin_lock_irqsave
>> 0,91% librados.so.2.0.0 [.] ceph_crc32c_le_intel
>> 0,91% libpthread-2.13.so [.] pthread_mutex_trylock
>> 0,87% [kernel] [k] resched_task
>> 0,72% [kernel] [k] cpu_startup_entry
>> 0,71% librados.so.2.0.0 [.] crush_hash32_3
>> 0,66% [kernel] [k] leave_mm
>> 0,65% librados.so.2.0.0 [.] Mutex::Lock(bool)
>> 0,64% [kernel] [k] idle_cpu
>> 0,62% libpthread-2.13.so [.] __pthread_mutex_unlock_usercnt
>> 0,59% [kernel] [k] try_to_wake_up
>> 0,56% [kernel] [k] wake_futex
>> 0,50% librados.so.2.0.0 [.] ceph::buffer::ptr::release()
>>
>> firefly:
>> 12,56% libc-2.13.so [.] 0x7905d
>> 2,82% libc-2.13.so [.] malloc
>> 2,64% libc-2.13.so [.] free
>> 2,61% kvm [.] 0x34322f
>> 2,33% [kernel] [k] __schedule
>> 2,14% [kernel] [k] _raw_spin_lock
>> 1,83% [kernel] [k] __switch_to
>> 1,62% [kernel] [k] lapic_next_deadline
>> 1,17% [kernel] [k] _raw_spin_lock_irqsave
>> 1,09% [kernel] [k] update_sd_lb_stats
>> 1,08% libpthread-2.13.so [.] pthread_mutex_trylock
>> 0,85% libpthread-2.13.so [.] __pthread_mutex_unlock_usercnt
>> 0,77% [kernel] [k] resched_task
>> 0,74% librbd.so.1.0.0 [.] 0x71b73
>> 0,72% librados.so.2.0.0 [.] Mutex::Lock(bool)
>> 0,68% librados.so.2.0.0 [.] crush_hash32_3
>> 0,67% [kernel] [k] idle_cpu
>> 0,65% [kernel] [k] leave_mm
>> 0,65% [kernel] [k] cpu_startup_entry
>> 0,59% [kernel] [k] try_to_wake_up
>> 0,51% librados.so.2.0.0 [.] ceph::buffer::ptr::release()
>> 0,51% [kernel] [k] wake_futex
>>
>> Stefan
>>
>>> Am 11.02.2015 um 06:42 schrieb Stefan Priebe:
>>>
>>>> Am 11.02.2015 um 05:45 schrieb Mark Nelson:
>>>>> On 02/10/2015 04:18 PM, Stefan Priebe wrote:
>>>>>
>>>>>> Am 10.02.2015 um 22:38 schrieb Mark Nelson:
>>>>>>> On 02/10/2015 03:11 PM, Stefan Priebe wrote:
>>>>>>>
>>>>>>> mhm i installed librbd1-dbg and librados2-dbg - but the output still
>>>>>>> looks useless to me. Should i upload it somewhere?
>>>>>>
>>>>>> Meh, if it's all just symbols it's probably not that helpful.
>>>>>>
>>>>>> I've summarized your results here:
>>>>>>
>>>>>> 1 concurrent 4k write (libaio, direct=1, iodepth=1)
>>>>>>
>>>>>> IOPS Latency
>>>>>> wb on wb off wb on wb off
>>>>>> dumpling 10870 536 ~100us ~2ms
>>>>>> firefly 10350 525 ~100us ~2ms
>>>>>>
>>>>>> So in single op tests dumpling and firefly are far closer. Now let's
>>>>>> see each of these cases with iodepth=32 (still 1 thread for now).
>>>>>
>>>>>
>>>>> dumpling:
>>>>>
>>>>> file1: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=32
>>>>> 2.0.8
>>>>> Starting 1 thread
>>>>> Jobs: 1 (f=1): [w] [100.0% done] [0K/72812K /s] [0 /18.3K iops] [eta
>>>>> 00m:00s]
>>>>> file1: (groupid=0, jobs=1): err= 0: pid=3011
>>>>> write: io=2060.6MB, bw=70329KB/s, iops=17582 , runt= 30001msec
>>>>> slat (usec): min=1 , max=3517 , avg= 3.42, stdev= 7.30
>>>>> clat (usec): min=93 , max=7475 , avg=1815.72, stdev=233.43
>>>>> lat (usec): min=219 , max=7477 , avg=1819.27, stdev=233.52
>>>>> clat percentiles (usec):
>>>>> | 1.00th=[ 1480], 5.00th=[ 1576], 10.00th=[ 1608], 20.00th=[
>>>>> 1672],
>>>>> | 30.00th=[ 1704], 40.00th=[ 1752], 50.00th=[ 1800], 60.00th=[
>>>>> 1832],
>>>>> | 70.00th=[ 1896], 80.00th=[ 1960], 90.00th=[ 2064], 95.00th=[
>>>>> 2128],
>>>>> | 99.00th=[ 2352], 99.50th=[ 2448], 99.90th=[ 4704], 99.95th=[
>>>>> 5344],
>>>>> | 99.99th=[ 7072]
>>>>> bw (KB/s) : min=59696, max=77840, per=100.00%, avg=70351.27,
>>>>> stdev=4783.25
>>>>> lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%,
>>>>> 1000=0.53%
>>>>> lat (msec) : 2=85.02%, 4=14.31%, 10=0.13%
>>>>> cpu : usr=1.96%, sys=6.71%, ctx=22791, majf=0, minf=133
>>>>> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%,
>>>>>> =64=0.0%
>>>>> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>>>>>> =64=0.0%
>>>>> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
>>>>>> =64=0.0%
>>>>> issued : total=r=0/w=527487/d=0, short=r=0/w=0/d=0
>>>>>
>>>>> Run status group 0 (all jobs):
>>>>> WRITE: io=2060.6MB, aggrb=70329KB/s, minb=70329KB/s, maxb=70329KB/s,
>>>>> mint=30001msec, maxt=30001msec
>>>>>
>>>>> Disk stats (read/write):
>>>>> sdb: ios=166/526079, merge=0/0, ticks=24/890120, in_queue=890064,
>>>>> util=98.73%
>>>>>
>>>>> firefly:
>>>>>
>>>>> file1: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=32
>>>>> 2.0.8
>>>>> Starting 1 thread
>>>>> Jobs: 1 (f=1): [w] [100.0% done] [0K/69096K /s] [0 /17.3K iops] [eta
>>>>> 00m:00s]
>>>>> file1: (groupid=0, jobs=1): err= 0: pid=2982
>>>>> write: io=1784.9MB, bw=60918KB/s, iops=15229 , runt= 30002msec
>>>>> slat (usec): min=1 , max=1389 , avg= 3.43, stdev= 5.32
>>>>> clat (usec): min=117 , max=8235 , avg=2096.88, stdev=396.30
>>>>> lat (usec): min=540 , max=8258 , avg=2100.43, stdev=396.61
>>>>> clat percentiles (usec):
>>>>> | 1.00th=[ 1608], 5.00th=[ 1720], 10.00th=[ 1768], 20.00th=[
>>>>> 1832],
>>>>> | 30.00th=[ 1896], 40.00th=[ 1944], 50.00th=[ 2008], 60.00th=[
>>>>> 2064],
>>>>> | 70.00th=[ 2160], 80.00th=[ 2256], 90.00th=[ 2512], 95.00th=[
>>>>> 2896],
>>>>> | 99.00th=[ 3600], 99.50th=[ 3792], 99.90th=[ 5088], 99.95th=[
>>>>> 6304],
>>>>> | 99.99th=[ 6752]
>>>>> bw (KB/s) : min=36717, max=73712, per=99.94%, avg=60879.92,
>>>>> stdev=8302.27
>>>>> lat (usec) : 250=0.01%, 750=0.01%
>>>>> lat (msec) : 2=48.56%, 4=51.18%, 10=0.26%
>>>>> cpu : usr=2.03%, sys=5.48%, ctx=20440, majf=0, minf=133
>>>>> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%,
>>>>>> =64=0.0%
>>>>> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>>>>>> =64=0.0%
>>>>> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
>>>>>> =64=0.0%
>>>>> issued : total=r=0/w=456918/d=0, short=r=0/w=0/d=0
>>>>>
>>>>> Run status group 0 (all jobs):
>>>>> WRITE: io=1784.9MB, aggrb=60918KB/s, minb=60918KB/s, maxb=60918KB/s,
>>>>> mint=30002msec, maxt=30002msec
>>>>>
>>>>> Disk stats (read/write):
>>>>> sdb: ios=166/455574, merge=0/0, ticks=12/897748, in_queue=897696,
>>>>> util=98.96%
>>>>
>>>> Ok, so it looks like as you increase concurrency the effect increases
>>>> (ie contention?). Does the same thing happen without cache enabled?
>>>
>>> here again without rbd cache:
>>>
>>> dumpling:
>>> file1: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=32
>>> 2.0.8
>>> Starting 1 thread
>>> Jobs: 1 (f=1): [w] [100.0% done] [0K/83488K /s] [0 /20.9K iops] [eta
>>> 00m:00s]
>>> file1: (groupid=0, jobs=1): err= 0: pid=3000
>>> write: io=2449.2MB, bw=83583KB/s, iops=20895 , runt= 30005msec
>>> slat (usec): min=1 , max=975 , avg= 4.50, stdev= 5.25
>>> clat (usec): min=364 , max=80566 , avg=1525.87, stdev=1194.57
>>> lat (usec): min=519 , max=80568 , avg=1530.51, stdev=1194.44
>>> clat percentiles (usec):
>>> | 1.00th=[ 660], 5.00th=[ 780], 10.00th=[ 876], 20.00th=[
>>> 1032],
>>> | 30.00th=[ 1144], 40.00th=[ 1240], 50.00th=[ 1304], 60.00th=[
>>> 1384],
>>> | 70.00th=[ 1480], 80.00th=[ 1640], 90.00th=[ 2096], 95.00th=[
>>> 2960],
>>> | 99.00th=[ 6816], 99.50th=[ 7840], 99.90th=[11712],
>>> 99.95th=[13888],
>>> | 99.99th=[18816]
>>> bw (KB/s) : min=47184, max=95432, per=100.00%, avg=83639.19,
>>> stdev=7973.92
>>> lat (usec) : 500=0.01%, 750=3.82%, 1000=14.40%
>>> lat (msec) : 2=70.57%, 4=7.91%, 10=3.11%, 20=0.17%, 50=0.01%
>>> lat (msec) : 100=0.01%
>>> cpu : usr=3.12%, sys=11.49%, ctx=74951, majf=0, minf=133
>>> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%,
>>>> =64=0.0%
>>> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>>>> =64=0.0%
>>> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
>>>> =64=0.0%
>>> issued : total=r=0/w=626979/d=0, short=r=0/w=0/d=0
>>>
>>> Run status group 0 (all jobs):
>>> WRITE: io=2449.2MB, aggrb=83583KB/s, minb=83583KB/s, maxb=83583KB/s,
>>> mint=30005msec, maxt=30005msec
>>>
>>> Disk stats (read/write):
>>> sdb: ios=168/625292, merge=0/0, ticks=144/916096, in_queue=916128,
>>> util=99.93%
>>>
>>>
>>> firefly:
>>>
>>> fio --filename=/dev/sdb --direct=1 --rw=randwrite --bs=4k --numjobs=1
>>> --thread --iodepth=32 --ioengine=libaio --runtime=30 --group_reporting
>>> --name=file1
>>> file1: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=32
>>> 2.0.8
>>> Starting 1 thread
>>> Jobs: 1 (f=1): [w] [100.0% done] [0K/90044K /s] [0 /22.6K iops] [eta
>>> 00m:00s]
>>> file1: (groupid=0, jobs=1): err= 0: pid=2970
>>> write: io=2372.9MB, bw=80976KB/s, iops=20244 , runt= 30006msec
>>> slat (usec): min=1 , max=4047 , avg= 4.36, stdev= 7.17
>>> clat (usec): min=197 , max=76656 , avg=1575.29, stdev=1165.74
>>> lat (usec): min=523 , max=76660 , avg=1579.79, stdev=1165.59
>>> clat percentiles (usec):
>>> | 1.00th=[ 676], 5.00th=[ 804], 10.00th=[ 916], 20.00th=[
>>> 1096],
>>> | 30.00th=[ 1224], 40.00th=[ 1304], 50.00th=[ 1384], 60.00th=[
>>> 1448],
>>> | 70.00th=[ 1544], 80.00th=[ 1704], 90.00th=[ 2128], 95.00th=[
>>> 2736],
>>> | 99.00th=[ 6752], 99.50th=[ 7904], 99.90th=[12096],
>>> 99.95th=[14656],
>>> | 99.99th=[18560]
>>> bw (KB/s) : min=47800, max=91952, per=99.91%, avg=80900.88,
>>> stdev=7234.98
>>> lat (usec) : 250=0.01%, 500=0.01%, 750=2.95%, 1000=11.38%
>>> lat (msec) : 2=73.81%, 4=8.81%, 10=2.85%, 20=0.19%, 50=0.01%
>>> lat (msec) : 100=0.01%
>>> cpu : usr=2.99%, sys=10.60%, ctx=66549, majf=0, minf=133
>>> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%,
>>>> =64=0.0%
>>> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>>>> =64=0.0%
>>> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
>>>> =64=0.0%
>>> issued : total=r=0/w=607445/d=0, short=r=0/w=0/d=0
>>>
>>> Run status group 0 (all jobs):
>>> WRITE: io=2372.9MB, aggrb=80976KB/s, minb=80976KB/s, maxb=80976KB/s,
>>> mint=30006msec, maxt=30006msec
>>>
>>> Disk stats (read/write):
>>> sdb: ios=170/605440, merge=0/0, ticks=156/916492, in_queue=916560,
>>> util=99.93%
>>>
>>> Stefan
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: speed decrease since firefly,giant,hammer the 2nd try
2015-02-16 22:18 ` Alexandre DERUMIER
@ 2015-02-17 9:15 ` Stefan Priebe - Profihost AG
[not found] ` <1999473015.1036573.1424165503094.JavaMail.zimbra@oxygem.tv>
0 siblings, 1 reply; 32+ messages in thread
From: Stefan Priebe - Profihost AG @ 2015-02-17 9:15 UTC (permalink / raw)
To: Alexandre DERUMIER; +Cc: Mark Nelson, ceph-devel
Am 16.02.2015 um 23:18 schrieb Alexandre DERUMIER:
>>> Just tested write. This might be the result of higher CPU load of the ceph-osd processes under firefly.
>>>
>>> Dumpling 180% per process vs. firefly 220%
>
> Oh yes, indeed, that's what I think too. (and more cpu -> less ios in qemu because of single iothread)
>
>
> I think that perf report should show the difference, this is strange that you have almost same result in your perf reports.
Not that easy as you can't downgrade from firefly to dumpling. Here is
perf output of firefly vs giant. While they have a difference of about
6k iops i do not see any differences in the perf output.
This is the perf output of the osd side with firefly (27k iops):
5,44% libc-2.13.so [.] 0x1370b0
4,02% libtcmalloc.so.4.1.0 [.] operator new(unsigned long)
2,84% libtcmalloc.so.4.1.0 [.] operator delete(void*)
2,50% ceph-osd [.] 0x5aed30
1,40% libstdc++.so.6.0.17 [.] 0xbeacc
1,39% [kernel] [k] intel_idle
1,34% libleveldb.so.1.12 [.] 0x24673
1,30% libc-2.13.so [.] vfprintf
1,20% [kernel] [k] __schedule
1,00% [kernel] [k] __switch_to
0,99% libleveldb.so.1.12 [.] leveldb::SkipList<char const*,
leveldb::MemTable::KeyComparator>::FindGreaterOrEqual(char const*
0,96% libpthread-2.13.so [.] pthread_mutex_trylock
0,86% [kernel] [k] _raw_spin_lock
0,83% libstdc++.so.6.0.17 [.] std::basic_string<char,
std::char_traits<char>, std::allocator<char> >::basic_string(std::string
0,82% ceph-osd [.] ceph::buffer::list::append(char
const*, unsigned int)
0,81% libsnappy.so.1.1.3 [.]
snappy::internal::CompressFragment(char const*, unsigned long, char*,
unsigned short*, int)
0,74% libpthread-2.13.so [.] __pthread_mutex_unlock_usercnt
0,73% ceph-osd [.] Mutex::Lock(bool)
0,72% [kernel] [k] __d_lookup_rcu
0,65% libtcmalloc.so.4.1.0 [.]
tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*,
unsigned long, int
0,65% ceph-osd [.] ceph::buffer::ptr::release()
0,64% libleveldb.so.1.12 [.] leveldb::crc32c::Extend(unsigned int,
char const*, unsigned long)
0,62% [kernel] [k] page_fault
0,58% ceph-osd [.]
ceph::buffer::list::iterator::copy(unsigned int, char*)
0,55% [kernel] [k] link_path_walk
0,53% libstdc++.so.6.0.17 [.] std::string::append(char const*,
unsigned long)
0,51% ceph-osd [.]
ceph::buffer::list::iterator::advance(int)
0,46% ceph-osd [.] ceph::buffer::ptr::append(char
const*, unsigned int)
0,45% libpthread-2.13.so [.] pthread_cond_wait@@GLIBC_2.3.2
0,43% libsnappy.so.1.1.3 [.]
snappy::RawUncompress(snappy::Source*, char*)
0,43% libstdc++.so.6.0.17 [.] std::string::_M_mutate(unsigned long,
unsigned long, unsigned long)
0,43% ceph-osd [.]
ceph::buffer::list::append(ceph::buffer::ptr const&, unsigned int,
unsigned int)
0,42% [kernel] [k] copy_user_enhanced_fast_string
and this is giant having (33k iops):
5,65% libc-2.13.so [.] 0x1376a4
4,36% libtcmalloc.so.4.1.0 [.] operator new(unsigned long)
2,74% ceph-osd [.] 0x48011d
2,49% libtcmalloc.so.4.1.0 [.] operator delete(void*)
1,42% libleveldb.so.1.12 [.] 0x24673
1,27% libleveldb.so.1.12 [.] leveldb::SkipList<char const*,
leveldb::MemTable::KeyComparator>::FindGreaterOrEqual(char cons
1,19% libstdc++.so.6.0.17 [.] 0xbfacb
0,99% libc-2.13.so [.] vfprintf
0,91% libsnappy.so.1.1.3 [.]
snappy::internal::CompressFragment(char const*, unsigned long, char*,
unsigned short*, int)
0,88% [kernel] [k] __schedule
0,88% libpthread-2.13.so [.] pthread_mutex_trylock
0,81% ceph-osd [.] ceph::buffer::list::append(char
const*, unsigned int)
0,76% [kernel] [k] __d_lookup_rcu
0,76% [kernel] [k] __switch_to
0,74% ceph-osd [.] Mutex::Lock(bool)
0,74% libtcmalloc.so.4.1.0 [.]
tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*,
unsigned long,
0,72% [kernel] [k] intel_idle
0,72% ceph-osd [.] ceph::buffer::ptr::release()
0,70% [kernel] [k] _raw_spin_lock
0,66% libleveldb.so.1.12 [.] leveldb::crc32c::Extend(unsigned
int, char const*, unsigned long)
0,66% ceph-osd [.]
ceph::buffer::list::iterator::copy(unsigned int, char*)
0,64% libstdc++.so.6.0.17 [.] std::basic_string<char,
std::char_traits<char>, std::allocator<char> >::basic_string(std::stri
0,64% [kernel] [k] page_fault
0,59% [kernel] [k] link_path_walk
0,58% libstdc++.so.6.0.17 [.] std::string::append(char const*,
unsigned long)
0,57% libpthread-2.13.so [.] __pthread_mutex_unlock_usercnt
0,56% ceph-osd [.]
ceph::buffer::list::iterator::advance(int)
0,44% ceph-osd [.] ceph::buffer::ptr::append(char
const*, unsigned int)
Stefan
>
>
>
> ----- Mail original -----
> De: "Stefan Priebe" <s.priebe@profihost.ag>
> À: "aderumier" <aderumier@odiso.com>
> Cc: "Mark Nelson" <mnelson@redhat.com>, "ceph-devel" <ceph-devel@vger.kernel.org>
> Envoyé: Lundi 16 Février 2015 23:08:37
> Objet: Re: speed decrease since firefly,giant,hammer the 2nd try
>
> Am 16.02.2015 um 23:02 schrieb Alexandre DERUMIER <aderumier@odiso.com>:
>
>>>> This results in fio-rbd showing avg 26000 iop/s instead of 30500 iop/s
>>>> while running dumpling...
>>
>> Is it for write only ?
>> or do you see same decrease for read too
>
> Just tested write. This might be the result of higher CPU load of the ceph-osd processes under firefly.
>
> Dumpling 180% per process vs. firefly 220%
>
> Stefan
>
>> ?
>>
>>
>> ----- Mail original -----
>> De: "Stefan Priebe" <s.priebe@profihost.ag>
>> À: "Mark Nelson" <mnelson@redhat.com>, "ceph-devel" <ceph-devel@vger.kernel.org>
>> Envoyé: Lundi 16 Février 2015 22:22:01
>> Objet: Re: speed decrease since firefly,giant,hammer the 2nd try
>>
>> I've now upgraded server side and client side to latest upstream/firefly.
>>
>> This results in fio-rbd showing avg 26000 iop/s instead of 30500 iop/s
>> while running dumpling...
>>
>> Greets,
>> Stefan
>>> Am 15.02.2015 um 19:40 schrieb Stefan Priebe:
>>> Hi Mark,
>>>
>>> what's next?
>>>
>>> I've this test cluster only for 2 more days.
>>>
>>> Here some perf Details:
>>>
>>> dumpling:
>>> 12,65% libc-2.13.so [.] 0x79000
>>> 2,86% libc-2.13.so [.] malloc
>>> 2,80% kvm [.] 0xb59c5
>>> 2,59% libc-2.13.so [.] free
>>> 2,35% [kernel] [k] __schedule
>>> 2,16% [kernel] [k] _raw_spin_lock
>>> 1,92% [kernel] [k] __switch_to
>>> 1,58% [kernel] [k] lapic_next_deadline
>>> 1,09% [kernel] [k] update_sd_lb_stats
>>> 1,08% [kernel] [k] _raw_spin_lock_irqsave
>>> 0,91% librados.so.2.0.0 [.] ceph_crc32c_le_intel
>>> 0,91% libpthread-2.13.so [.] pthread_mutex_trylock
>>> 0,87% [kernel] [k] resched_task
>>> 0,72% [kernel] [k] cpu_startup_entry
>>> 0,71% librados.so.2.0.0 [.] crush_hash32_3
>>> 0,66% [kernel] [k] leave_mm
>>> 0,65% librados.so.2.0.0 [.] Mutex::Lock(bool)
>>> 0,64% [kernel] [k] idle_cpu
>>> 0,62% libpthread-2.13.so [.] __pthread_mutex_unlock_usercnt
>>> 0,59% [kernel] [k] try_to_wake_up
>>> 0,56% [kernel] [k] wake_futex
>>> 0,50% librados.so.2.0.0 [.] ceph::buffer::ptr::release()
>>>
>>> firefly:
>>> 12,56% libc-2.13.so [.] 0x7905d
>>> 2,82% libc-2.13.so [.] malloc
>>> 2,64% libc-2.13.so [.] free
>>> 2,61% kvm [.] 0x34322f
>>> 2,33% [kernel] [k] __schedule
>>> 2,14% [kernel] [k] _raw_spin_lock
>>> 1,83% [kernel] [k] __switch_to
>>> 1,62% [kernel] [k] lapic_next_deadline
>>> 1,17% [kernel] [k] _raw_spin_lock_irqsave
>>> 1,09% [kernel] [k] update_sd_lb_stats
>>> 1,08% libpthread-2.13.so [.] pthread_mutex_trylock
>>> 0,85% libpthread-2.13.so [.] __pthread_mutex_unlock_usercnt
>>> 0,77% [kernel] [k] resched_task
>>> 0,74% librbd.so.1.0.0 [.] 0x71b73
>>> 0,72% librados.so.2.0.0 [.] Mutex::Lock(bool)
>>> 0,68% librados.so.2.0.0 [.] crush_hash32_3
>>> 0,67% [kernel] [k] idle_cpu
>>> 0,65% [kernel] [k] leave_mm
>>> 0,65% [kernel] [k] cpu_startup_entry
>>> 0,59% [kernel] [k] try_to_wake_up
>>> 0,51% librados.so.2.0.0 [.] ceph::buffer::ptr::release()
>>> 0,51% [kernel] [k] wake_futex
>>>
>>> Stefan
>>>
>>>> Am 11.02.2015 um 06:42 schrieb Stefan Priebe:
>>>>
>>>>> Am 11.02.2015 um 05:45 schrieb Mark Nelson:
>>>>>> On 02/10/2015 04:18 PM, Stefan Priebe wrote:
>>>>>>
>>>>>>> Am 10.02.2015 um 22:38 schrieb Mark Nelson:
>>>>>>>> On 02/10/2015 03:11 PM, Stefan Priebe wrote:
>>>>>>>>
>>>>>>>> mhm i installed librbd1-dbg and librados2-dbg - but the output still
>>>>>>>> looks useless to me. Should i upload it somewhere?
>>>>>>>
>>>>>>> Meh, if it's all just symbols it's probably not that helpful.
>>>>>>>
>>>>>>> I've summarized your results here:
>>>>>>>
>>>>>>> 1 concurrent 4k write (libaio, direct=1, iodepth=1)
>>>>>>>
>>>>>>> IOPS Latency
>>>>>>> wb on wb off wb on wb off
>>>>>>> dumpling 10870 536 ~100us ~2ms
>>>>>>> firefly 10350 525 ~100us ~2ms
>>>>>>>
>>>>>>> So in single op tests dumpling and firefly are far closer. Now let's
>>>>>>> see each of these cases with iodepth=32 (still 1 thread for now).
>>>>>>
>>>>>>
>>>>>> dumpling:
>>>>>>
>>>>>> file1: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=32
>>>>>> 2.0.8
>>>>>> Starting 1 thread
>>>>>> Jobs: 1 (f=1): [w] [100.0% done] [0K/72812K /s] [0 /18.3K iops] [eta
>>>>>> 00m:00s]
>>>>>> file1: (groupid=0, jobs=1): err= 0: pid=3011
>>>>>> write: io=2060.6MB, bw=70329KB/s, iops=17582 , runt= 30001msec
>>>>>> slat (usec): min=1 , max=3517 , avg= 3.42, stdev= 7.30
>>>>>> clat (usec): min=93 , max=7475 , avg=1815.72, stdev=233.43
>>>>>> lat (usec): min=219 , max=7477 , avg=1819.27, stdev=233.52
>>>>>> clat percentiles (usec):
>>>>>> | 1.00th=[ 1480], 5.00th=[ 1576], 10.00th=[ 1608], 20.00th=[
>>>>>> 1672],
>>>>>> | 30.00th=[ 1704], 40.00th=[ 1752], 50.00th=[ 1800], 60.00th=[
>>>>>> 1832],
>>>>>> | 70.00th=[ 1896], 80.00th=[ 1960], 90.00th=[ 2064], 95.00th=[
>>>>>> 2128],
>>>>>> | 99.00th=[ 2352], 99.50th=[ 2448], 99.90th=[ 4704], 99.95th=[
>>>>>> 5344],
>>>>>> | 99.99th=[ 7072]
>>>>>> bw (KB/s) : min=59696, max=77840, per=100.00%, avg=70351.27,
>>>>>> stdev=4783.25
>>>>>> lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%,
>>>>>> 1000=0.53%
>>>>>> lat (msec) : 2=85.02%, 4=14.31%, 10=0.13%
>>>>>> cpu : usr=1.96%, sys=6.71%, ctx=22791, majf=0, minf=133
>>>>>> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%,
>>>>>>> =64=0.0%
>>>>>> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>>>>>>> =64=0.0%
>>>>>> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
>>>>>>> =64=0.0%
>>>>>> issued : total=r=0/w=527487/d=0, short=r=0/w=0/d=0
>>>>>>
>>>>>> Run status group 0 (all jobs):
>>>>>> WRITE: io=2060.6MB, aggrb=70329KB/s, minb=70329KB/s, maxb=70329KB/s,
>>>>>> mint=30001msec, maxt=30001msec
>>>>>>
>>>>>> Disk stats (read/write):
>>>>>> sdb: ios=166/526079, merge=0/0, ticks=24/890120, in_queue=890064,
>>>>>> util=98.73%
>>>>>>
>>>>>> firefly:
>>>>>>
>>>>>> file1: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=32
>>>>>> 2.0.8
>>>>>> Starting 1 thread
>>>>>> Jobs: 1 (f=1): [w] [100.0% done] [0K/69096K /s] [0 /17.3K iops] [eta
>>>>>> 00m:00s]
>>>>>> file1: (groupid=0, jobs=1): err= 0: pid=2982
>>>>>> write: io=1784.9MB, bw=60918KB/s, iops=15229 , runt= 30002msec
>>>>>> slat (usec): min=1 , max=1389 , avg= 3.43, stdev= 5.32
>>>>>> clat (usec): min=117 , max=8235 , avg=2096.88, stdev=396.30
>>>>>> lat (usec): min=540 , max=8258 , avg=2100.43, stdev=396.61
>>>>>> clat percentiles (usec):
>>>>>> | 1.00th=[ 1608], 5.00th=[ 1720], 10.00th=[ 1768], 20.00th=[
>>>>>> 1832],
>>>>>> | 30.00th=[ 1896], 40.00th=[ 1944], 50.00th=[ 2008], 60.00th=[
>>>>>> 2064],
>>>>>> | 70.00th=[ 2160], 80.00th=[ 2256], 90.00th=[ 2512], 95.00th=[
>>>>>> 2896],
>>>>>> | 99.00th=[ 3600], 99.50th=[ 3792], 99.90th=[ 5088], 99.95th=[
>>>>>> 6304],
>>>>>> | 99.99th=[ 6752]
>>>>>> bw (KB/s) : min=36717, max=73712, per=99.94%, avg=60879.92,
>>>>>> stdev=8302.27
>>>>>> lat (usec) : 250=0.01%, 750=0.01%
>>>>>> lat (msec) : 2=48.56%, 4=51.18%, 10=0.26%
>>>>>> cpu : usr=2.03%, sys=5.48%, ctx=20440, majf=0, minf=133
>>>>>> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%,
>>>>>>> =64=0.0%
>>>>>> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>>>>>>> =64=0.0%
>>>>>> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
>>>>>>> =64=0.0%
>>>>>> issued : total=r=0/w=456918/d=0, short=r=0/w=0/d=0
>>>>>>
>>>>>> Run status group 0 (all jobs):
>>>>>> WRITE: io=1784.9MB, aggrb=60918KB/s, minb=60918KB/s, maxb=60918KB/s,
>>>>>> mint=30002msec, maxt=30002msec
>>>>>>
>>>>>> Disk stats (read/write):
>>>>>> sdb: ios=166/455574, merge=0/0, ticks=12/897748, in_queue=897696,
>>>>>> util=98.96%
>>>>>
>>>>> Ok, so it looks like as you increase concurrency the effect increases
>>>>> (ie contention?). Does the same thing happen without cache enabled?
>>>>
>>>> here again without rbd cache:
>>>>
>>>> dumpling:
>>>> file1: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=32
>>>> 2.0.8
>>>> Starting 1 thread
>>>> Jobs: 1 (f=1): [w] [100.0% done] [0K/83488K /s] [0 /20.9K iops] [eta
>>>> 00m:00s]
>>>> file1: (groupid=0, jobs=1): err= 0: pid=3000
>>>> write: io=2449.2MB, bw=83583KB/s, iops=20895 , runt= 30005msec
>>>> slat (usec): min=1 , max=975 , avg= 4.50, stdev= 5.25
>>>> clat (usec): min=364 , max=80566 , avg=1525.87, stdev=1194.57
>>>> lat (usec): min=519 , max=80568 , avg=1530.51, stdev=1194.44
>>>> clat percentiles (usec):
>>>> | 1.00th=[ 660], 5.00th=[ 780], 10.00th=[ 876], 20.00th=[
>>>> 1032],
>>>> | 30.00th=[ 1144], 40.00th=[ 1240], 50.00th=[ 1304], 60.00th=[
>>>> 1384],
>>>> | 70.00th=[ 1480], 80.00th=[ 1640], 90.00th=[ 2096], 95.00th=[
>>>> 2960],
>>>> | 99.00th=[ 6816], 99.50th=[ 7840], 99.90th=[11712],
>>>> 99.95th=[13888],
>>>> | 99.99th=[18816]
>>>> bw (KB/s) : min=47184, max=95432, per=100.00%, avg=83639.19,
>>>> stdev=7973.92
>>>> lat (usec) : 500=0.01%, 750=3.82%, 1000=14.40%
>>>> lat (msec) : 2=70.57%, 4=7.91%, 10=3.11%, 20=0.17%, 50=0.01%
>>>> lat (msec) : 100=0.01%
>>>> cpu : usr=3.12%, sys=11.49%, ctx=74951, majf=0, minf=133
>>>> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%,
>>>>> =64=0.0%
>>>> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>>>>> =64=0.0%
>>>> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
>>>>> =64=0.0%
>>>> issued : total=r=0/w=626979/d=0, short=r=0/w=0/d=0
>>>>
>>>> Run status group 0 (all jobs):
>>>> WRITE: io=2449.2MB, aggrb=83583KB/s, minb=83583KB/s, maxb=83583KB/s,
>>>> mint=30005msec, maxt=30005msec
>>>>
>>>> Disk stats (read/write):
>>>> sdb: ios=168/625292, merge=0/0, ticks=144/916096, in_queue=916128,
>>>> util=99.93%
>>>>
>>>>
>>>> firefly:
>>>>
>>>> fio --filename=/dev/sdb --direct=1 --rw=randwrite --bs=4k --numjobs=1
>>>> --thread --iodepth=32 --ioengine=libaio --runtime=30 --group_reporting
>>>> --name=file1
>>>> file1: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=32
>>>> 2.0.8
>>>> Starting 1 thread
>>>> Jobs: 1 (f=1): [w] [100.0% done] [0K/90044K /s] [0 /22.6K iops] [eta
>>>> 00m:00s]
>>>> file1: (groupid=0, jobs=1): err= 0: pid=2970
>>>> write: io=2372.9MB, bw=80976KB/s, iops=20244 , runt= 30006msec
>>>> slat (usec): min=1 , max=4047 , avg= 4.36, stdev= 7.17
>>>> clat (usec): min=197 , max=76656 , avg=1575.29, stdev=1165.74
>>>> lat (usec): min=523 , max=76660 , avg=1579.79, stdev=1165.59
>>>> clat percentiles (usec):
>>>> | 1.00th=[ 676], 5.00th=[ 804], 10.00th=[ 916], 20.00th=[
>>>> 1096],
>>>> | 30.00th=[ 1224], 40.00th=[ 1304], 50.00th=[ 1384], 60.00th=[
>>>> 1448],
>>>> | 70.00th=[ 1544], 80.00th=[ 1704], 90.00th=[ 2128], 95.00th=[
>>>> 2736],
>>>> | 99.00th=[ 6752], 99.50th=[ 7904], 99.90th=[12096],
>>>> 99.95th=[14656],
>>>> | 99.99th=[18560]
>>>> bw (KB/s) : min=47800, max=91952, per=99.91%, avg=80900.88,
>>>> stdev=7234.98
>>>> lat (usec) : 250=0.01%, 500=0.01%, 750=2.95%, 1000=11.38%
>>>> lat (msec) : 2=73.81%, 4=8.81%, 10=2.85%, 20=0.19%, 50=0.01%
>>>> lat (msec) : 100=0.01%
>>>> cpu : usr=2.99%, sys=10.60%, ctx=66549, majf=0, minf=133
>>>> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%,
>>>>> =64=0.0%
>>>> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>>>>> =64=0.0%
>>>> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
>>>>> =64=0.0%
>>>> issued : total=r=0/w=607445/d=0, short=r=0/w=0/d=0
>>>>
>>>> Run status group 0 (all jobs):
>>>> WRITE: io=2372.9MB, aggrb=80976KB/s, minb=80976KB/s, maxb=80976KB/s,
>>>> mint=30006msec, maxt=30006msec
>>>>
>>>> Disk stats (read/write):
>>>> sdb: ios=170/605440, merge=0/0, ticks=156/916492, in_queue=916560,
>>>> util=99.93%
>>>>
>>>> Stefan
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: speed decrease since firefly,giant,hammer the 2nd try
[not found] ` <1999473015.1036573.1424165503094.JavaMail.zimbra@oxygem.tv>
@ 2015-02-17 9:31 ` Alexandre DERUMIER
0 siblings, 0 replies; 32+ messages in thread
From: Alexandre DERUMIER @ 2015-02-17 9:31 UTC (permalink / raw)
To: Stefan Priebe; +Cc: Mark Nelson, ceph-devel
>>Not that easy as you can't downgrade from firefly to dumpling. Here is
>>perf output of firefly vs giant. While they have a difference of about
>>6k iops i do not see any differences in the perf output.
>>
>>This is the perf output of the osd side with firefly (27k iops):
I don't known for osd side (because they are a lot of optimisations in giant for ssd).
I'll try to redone big client side benchmark next month when my new ssd cluster will be ready.
(I'll try dumpling,firefly,giant,hammer , fio-krbd,fio-librbd, qemu, virtio,virtio-scsi,iothreads,....)
I have planned to take 1month for benchmark, so I hope I'll help.
----- Mail original -----
De: "Stefan Priebe" <s.priebe@profihost.ag>
À: "aderumier" <aderumier@odiso.com>
Cc: "Mark Nelson" <mnelson@redhat.com>, "ceph-devel" <ceph-devel@vger.kernel.org>
Envoyé: Mardi 17 Février 2015 10:15:09
Objet: Re: speed decrease since firefly,giant,hammer the 2nd try
Am 16.02.2015 um 23:18 schrieb Alexandre DERUMIER:
>>> Just tested write. This might be the result of higher CPU load of the ceph-osd processes under firefly.
>>>
>>> Dumpling 180% per process vs. firefly 220%
>
> Oh yes, indeed, that's what I think too. (and more cpu -> less ios in qemu because of single iothread)
>
>
> I think that perf report should show the difference, this is strange that you have almost same result in your perf reports.
Not that easy as you can't downgrade from firefly to dumpling. Here is
perf output of firefly vs giant. While they have a difference of about
6k iops i do not see any differences in the perf output.
This is the perf output of the osd side with firefly (27k iops):
5,44% libc-2.13.so [.] 0x1370b0
4,02% libtcmalloc.so.4.1.0 [.] operator new(unsigned long)
2,84% libtcmalloc.so.4.1.0 [.] operator delete(void*)
2,50% ceph-osd [.] 0x5aed30
1,40% libstdc++.so.6.0.17 [.] 0xbeacc
1,39% [kernel] [k] intel_idle
1,34% libleveldb.so.1.12 [.] 0x24673
1,30% libc-2.13.so [.] vfprintf
1,20% [kernel] [k] __schedule
1,00% [kernel] [k] __switch_to
0,99% libleveldb.so.1.12 [.] leveldb::SkipList<char const*,
leveldb::MemTable::KeyComparator>::FindGreaterOrEqual(char const*
0,96% libpthread-2.13.so [.] pthread_mutex_trylock
0,86% [kernel] [k] _raw_spin_lock
0,83% libstdc++.so.6.0.17 [.] std::basic_string<char,
std::char_traits<char>, std::allocator<char> >::basic_string(std::string
0,82% ceph-osd [.] ceph::buffer::list::append(char
const*, unsigned int)
0,81% libsnappy.so.1.1.3 [.]
snappy::internal::CompressFragment(char const*, unsigned long, char*,
unsigned short*, int)
0,74% libpthread-2.13.so [.] __pthread_mutex_unlock_usercnt
0,73% ceph-osd [.] Mutex::Lock(bool)
0,72% [kernel] [k] __d_lookup_rcu
0,65% libtcmalloc.so.4.1.0 [.]
tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*,
unsigned long, int
0,65% ceph-osd [.] ceph::buffer::ptr::release()
0,64% libleveldb.so.1.12 [.] leveldb::crc32c::Extend(unsigned int,
char const*, unsigned long)
0,62% [kernel] [k] page_fault
0,58% ceph-osd [.]
ceph::buffer::list::iterator::copy(unsigned int, char*)
0,55% [kernel] [k] link_path_walk
0,53% libstdc++.so.6.0.17 [.] std::string::append(char const*,
unsigned long)
0,51% ceph-osd [.]
ceph::buffer::list::iterator::advance(int)
0,46% ceph-osd [.] ceph::buffer::ptr::append(char
const*, unsigned int)
0,45% libpthread-2.13.so [.] pthread_cond_wait@@GLIBC_2.3.2
0,43% libsnappy.so.1.1.3 [.]
snappy::RawUncompress(snappy::Source*, char*)
0,43% libstdc++.so.6.0.17 [.] std::string::_M_mutate(unsigned long,
unsigned long, unsigned long)
0,43% ceph-osd [.]
ceph::buffer::list::append(ceph::buffer::ptr const&, unsigned int,
unsigned int)
0,42% [kernel] [k] copy_user_enhanced_fast_string
and this is giant having (33k iops):
5,65% libc-2.13.so [.] 0x1376a4
4,36% libtcmalloc.so.4.1.0 [.] operator new(unsigned long)
2,74% ceph-osd [.] 0x48011d
2,49% libtcmalloc.so.4.1.0 [.] operator delete(void*)
1,42% libleveldb.so.1.12 [.] 0x24673
1,27% libleveldb.so.1.12 [.] leveldb::SkipList<char const*,
leveldb::MemTable::KeyComparator>::FindGreaterOrEqual(char cons
1,19% libstdc++.so.6.0.17 [.] 0xbfacb
0,99% libc-2.13.so [.] vfprintf
0,91% libsnappy.so.1.1.3 [.]
snappy::internal::CompressFragment(char const*, unsigned long, char*,
unsigned short*, int)
0,88% [kernel] [k] __schedule
0,88% libpthread-2.13.so [.] pthread_mutex_trylock
0,81% ceph-osd [.] ceph::buffer::list::append(char
const*, unsigned int)
0,76% [kernel] [k] __d_lookup_rcu
0,76% [kernel] [k] __switch_to
0,74% ceph-osd [.] Mutex::Lock(bool)
0,74% libtcmalloc.so.4.1.0 [.]
tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*,
unsigned long,
0,72% [kernel] [k] intel_idle
0,72% ceph-osd [.] ceph::buffer::ptr::release()
0,70% [kernel] [k] _raw_spin_lock
0,66% libleveldb.so.1.12 [.] leveldb::crc32c::Extend(unsigned
int, char const*, unsigned long)
0,66% ceph-osd [.]
ceph::buffer::list::iterator::copy(unsigned int, char*)
0,64% libstdc++.so.6.0.17 [.] std::basic_string<char,
std::char_traits<char>, std::allocator<char> >::basic_string(std::stri
0,64% [kernel] [k] page_fault
0,59% [kernel] [k] link_path_walk
0,58% libstdc++.so.6.0.17 [.] std::string::append(char const*,
unsigned long)
0,57% libpthread-2.13.so [.] __pthread_mutex_unlock_usercnt
0,56% ceph-osd [.]
ceph::buffer::list::iterator::advance(int)
0,44% ceph-osd [.] ceph::buffer::ptr::append(char
const*, unsigned int)
Stefan
>
>
>
> ----- Mail original -----
> De: "Stefan Priebe" <s.priebe@profihost.ag>
> À: "aderumier" <aderumier@odiso.com>
> Cc: "Mark Nelson" <mnelson@redhat.com>, "ceph-devel" <ceph-devel@vger.kernel.org>
> Envoyé: Lundi 16 Février 2015 23:08:37
> Objet: Re: speed decrease since firefly,giant,hammer the 2nd try
>
> Am 16.02.2015 um 23:02 schrieb Alexandre DERUMIER <aderumier@odiso.com>:
>
>>>> This results in fio-rbd showing avg 26000 iop/s instead of 30500 iop/s
>>>> while running dumpling...
>>
>> Is it for write only ?
>> or do you see same decrease for read too
>
> Just tested write. This might be the result of higher CPU load of the ceph-osd processes under firefly.
>
> Dumpling 180% per process vs. firefly 220%
>
> Stefan
>
>> ?
>>
>>
>> ----- Mail original -----
>> De: "Stefan Priebe" <s.priebe@profihost.ag>
>> À: "Mark Nelson" <mnelson@redhat.com>, "ceph-devel" <ceph-devel@vger.kernel.org>
>> Envoyé: Lundi 16 Février 2015 22:22:01
>> Objet: Re: speed decrease since firefly,giant,hammer the 2nd try
>>
>> I've now upgraded server side and client side to latest upstream/firefly.
>>
>> This results in fio-rbd showing avg 26000 iop/s instead of 30500 iop/s
>> while running dumpling...
>>
>> Greets,
>> Stefan
>>> Am 15.02.2015 um 19:40 schrieb Stefan Priebe:
>>> Hi Mark,
>>>
>>> what's next?
>>>
>>> I've this test cluster only for 2 more days.
>>>
>>> Here some perf Details:
>>>
>>> dumpling:
>>> 12,65% libc-2.13.so [.] 0x79000
>>> 2,86% libc-2.13.so [.] malloc
>>> 2,80% kvm [.] 0xb59c5
>>> 2,59% libc-2.13.so [.] free
>>> 2,35% [kernel] [k] __schedule
>>> 2,16% [kernel] [k] _raw_spin_lock
>>> 1,92% [kernel] [k] __switch_to
>>> 1,58% [kernel] [k] lapic_next_deadline
>>> 1,09% [kernel] [k] update_sd_lb_stats
>>> 1,08% [kernel] [k] _raw_spin_lock_irqsave
>>> 0,91% librados.so.2.0.0 [.] ceph_crc32c_le_intel
>>> 0,91% libpthread-2.13.so [.] pthread_mutex_trylock
>>> 0,87% [kernel] [k] resched_task
>>> 0,72% [kernel] [k] cpu_startup_entry
>>> 0,71% librados.so.2.0.0 [.] crush_hash32_3
>>> 0,66% [kernel] [k] leave_mm
>>> 0,65% librados.so.2.0.0 [.] Mutex::Lock(bool)
>>> 0,64% [kernel] [k] idle_cpu
>>> 0,62% libpthread-2.13.so [.] __pthread_mutex_unlock_usercnt
>>> 0,59% [kernel] [k] try_to_wake_up
>>> 0,56% [kernel] [k] wake_futex
>>> 0,50% librados.so.2.0.0 [.] ceph::buffer::ptr::release()
>>>
>>> firefly:
>>> 12,56% libc-2.13.so [.] 0x7905d
>>> 2,82% libc-2.13.so [.] malloc
>>> 2,64% libc-2.13.so [.] free
>>> 2,61% kvm [.] 0x34322f
>>> 2,33% [kernel] [k] __schedule
>>> 2,14% [kernel] [k] _raw_spin_lock
>>> 1,83% [kernel] [k] __switch_to
>>> 1,62% [kernel] [k] lapic_next_deadline
>>> 1,17% [kernel] [k] _raw_spin_lock_irqsave
>>> 1,09% [kernel] [k] update_sd_lb_stats
>>> 1,08% libpthread-2.13.so [.] pthread_mutex_trylock
>>> 0,85% libpthread-2.13.so [.] __pthread_mutex_unlock_usercnt
>>> 0,77% [kernel] [k] resched_task
>>> 0,74% librbd.so.1.0.0 [.] 0x71b73
>>> 0,72% librados.so.2.0.0 [.] Mutex::Lock(bool)
>>> 0,68% librados.so.2.0.0 [.] crush_hash32_3
>>> 0,67% [kernel] [k] idle_cpu
>>> 0,65% [kernel] [k] leave_mm
>>> 0,65% [kernel] [k] cpu_startup_entry
>>> 0,59% [kernel] [k] try_to_wake_up
>>> 0,51% librados.so.2.0.0 [.] ceph::buffer::ptr::release()
>>> 0,51% [kernel] [k] wake_futex
>>>
>>> Stefan
>>>
>>>> Am 11.02.2015 um 06:42 schrieb Stefan Priebe:
>>>>
>>>>> Am 11.02.2015 um 05:45 schrieb Mark Nelson:
>>>>>> On 02/10/2015 04:18 PM, Stefan Priebe wrote:
>>>>>>
>>>>>>> Am 10.02.2015 um 22:38 schrieb Mark Nelson:
>>>>>>>> On 02/10/2015 03:11 PM, Stefan Priebe wrote:
>>>>>>>>
>>>>>>>> mhm i installed librbd1-dbg and librados2-dbg - but the output still
>>>>>>>> looks useless to me. Should i upload it somewhere?
>>>>>>>
>>>>>>> Meh, if it's all just symbols it's probably not that helpful.
>>>>>>>
>>>>>>> I've summarized your results here:
>>>>>>>
>>>>>>> 1 concurrent 4k write (libaio, direct=1, iodepth=1)
>>>>>>>
>>>>>>> IOPS Latency
>>>>>>> wb on wb off wb on wb off
>>>>>>> dumpling 10870 536 ~100us ~2ms
>>>>>>> firefly 10350 525 ~100us ~2ms
>>>>>>>
>>>>>>> So in single op tests dumpling and firefly are far closer. Now let's
>>>>>>> see each of these cases with iodepth=32 (still 1 thread for now).
>>>>>>
>>>>>>
>>>>>> dumpling:
>>>>>>
>>>>>> file1: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=32
>>>>>> 2.0.8
>>>>>> Starting 1 thread
>>>>>> Jobs: 1 (f=1): [w] [100.0% done] [0K/72812K /s] [0 /18.3K iops] [eta
>>>>>> 00m:00s]
>>>>>> file1: (groupid=0, jobs=1): err= 0: pid=3011
>>>>>> write: io=2060.6MB, bw=70329KB/s, iops=17582 , runt= 30001msec
>>>>>> slat (usec): min=1 , max=3517 , avg= 3.42, stdev= 7.30
>>>>>> clat (usec): min=93 , max=7475 , avg=1815.72, stdev=233.43
>>>>>> lat (usec): min=219 , max=7477 , avg=1819.27, stdev=233.52
>>>>>> clat percentiles (usec):
>>>>>> | 1.00th=[ 1480], 5.00th=[ 1576], 10.00th=[ 1608], 20.00th=[
>>>>>> 1672],
>>>>>> | 30.00th=[ 1704], 40.00th=[ 1752], 50.00th=[ 1800], 60.00th=[
>>>>>> 1832],
>>>>>> | 70.00th=[ 1896], 80.00th=[ 1960], 90.00th=[ 2064], 95.00th=[
>>>>>> 2128],
>>>>>> | 99.00th=[ 2352], 99.50th=[ 2448], 99.90th=[ 4704], 99.95th=[
>>>>>> 5344],
>>>>>> | 99.99th=[ 7072]
>>>>>> bw (KB/s) : min=59696, max=77840, per=100.00%, avg=70351.27,
>>>>>> stdev=4783.25
>>>>>> lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%,
>>>>>> 1000=0.53%
>>>>>> lat (msec) : 2=85.02%, 4=14.31%, 10=0.13%
>>>>>> cpu : usr=1.96%, sys=6.71%, ctx=22791, majf=0, minf=133
>>>>>> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%,
>>>>>>> =64=0.0%
>>>>>> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>>>>>>> =64=0.0%
>>>>>> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
>>>>>>> =64=0.0%
>>>>>> issued : total=r=0/w=527487/d=0, short=r=0/w=0/d=0
>>>>>>
>>>>>> Run status group 0 (all jobs):
>>>>>> WRITE: io=2060.6MB, aggrb=70329KB/s, minb=70329KB/s, maxb=70329KB/s,
>>>>>> mint=30001msec, maxt=30001msec
>>>>>>
>>>>>> Disk stats (read/write):
>>>>>> sdb: ios=166/526079, merge=0/0, ticks=24/890120, in_queue=890064,
>>>>>> util=98.73%
>>>>>>
>>>>>> firefly:
>>>>>>
>>>>>> file1: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=32
>>>>>> 2.0.8
>>>>>> Starting 1 thread
>>>>>> Jobs: 1 (f=1): [w] [100.0% done] [0K/69096K /s] [0 /17.3K iops] [eta
>>>>>> 00m:00s]
>>>>>> file1: (groupid=0, jobs=1): err= 0: pid=2982
>>>>>> write: io=1784.9MB, bw=60918KB/s, iops=15229 , runt= 30002msec
>>>>>> slat (usec): min=1 , max=1389 , avg= 3.43, stdev= 5.32
>>>>>> clat (usec): min=117 , max=8235 , avg=2096.88, stdev=396.30
>>>>>> lat (usec): min=540 , max=8258 , avg=2100.43, stdev=396.61
>>>>>> clat percentiles (usec):
>>>>>> | 1.00th=[ 1608], 5.00th=[ 1720], 10.00th=[ 1768], 20.00th=[
>>>>>> 1832],
>>>>>> | 30.00th=[ 1896], 40.00th=[ 1944], 50.00th=[ 2008], 60.00th=[
>>>>>> 2064],
>>>>>> | 70.00th=[ 2160], 80.00th=[ 2256], 90.00th=[ 2512], 95.00th=[
>>>>>> 2896],
>>>>>> | 99.00th=[ 3600], 99.50th=[ 3792], 99.90th=[ 5088], 99.95th=[
>>>>>> 6304],
>>>>>> | 99.99th=[ 6752]
>>>>>> bw (KB/s) : min=36717, max=73712, per=99.94%, avg=60879.92,
>>>>>> stdev=8302.27
>>>>>> lat (usec) : 250=0.01%, 750=0.01%
>>>>>> lat (msec) : 2=48.56%, 4=51.18%, 10=0.26%
>>>>>> cpu : usr=2.03%, sys=5.48%, ctx=20440, majf=0, minf=133
>>>>>> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%,
>>>>>>> =64=0.0%
>>>>>> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>>>>>>> =64=0.0%
>>>>>> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
>>>>>>> =64=0.0%
>>>>>> issued : total=r=0/w=456918/d=0, short=r=0/w=0/d=0
>>>>>>
>>>>>> Run status group 0 (all jobs):
>>>>>> WRITE: io=1784.9MB, aggrb=60918KB/s, minb=60918KB/s, maxb=60918KB/s,
>>>>>> mint=30002msec, maxt=30002msec
>>>>>>
>>>>>> Disk stats (read/write):
>>>>>> sdb: ios=166/455574, merge=0/0, ticks=12/897748, in_queue=897696,
>>>>>> util=98.96%
>>>>>
>>>>> Ok, so it looks like as you increase concurrency the effect increases
>>>>> (ie contention?). Does the same thing happen without cache enabled?
>>>>
>>>> here again without rbd cache:
>>>>
>>>> dumpling:
>>>> file1: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=32
>>>> 2.0.8
>>>> Starting 1 thread
>>>> Jobs: 1 (f=1): [w] [100.0% done] [0K/83488K /s] [0 /20.9K iops] [eta
>>>> 00m:00s]
>>>> file1: (groupid=0, jobs=1): err= 0: pid=3000
>>>> write: io=2449.2MB, bw=83583KB/s, iops=20895 , runt= 30005msec
>>>> slat (usec): min=1 , max=975 , avg= 4.50, stdev= 5.25
>>>> clat (usec): min=364 , max=80566 , avg=1525.87, stdev=1194.57
>>>> lat (usec): min=519 , max=80568 , avg=1530.51, stdev=1194.44
>>>> clat percentiles (usec):
>>>> | 1.00th=[ 660], 5.00th=[ 780], 10.00th=[ 876], 20.00th=[
>>>> 1032],
>>>> | 30.00th=[ 1144], 40.00th=[ 1240], 50.00th=[ 1304], 60.00th=[
>>>> 1384],
>>>> | 70.00th=[ 1480], 80.00th=[ 1640], 90.00th=[ 2096], 95.00th=[
>>>> 2960],
>>>> | 99.00th=[ 6816], 99.50th=[ 7840], 99.90th=[11712],
>>>> 99.95th=[13888],
>>>> | 99.99th=[18816]
>>>> bw (KB/s) : min=47184, max=95432, per=100.00%, avg=83639.19,
>>>> stdev=7973.92
>>>> lat (usec) : 500=0.01%, 750=3.82%, 1000=14.40%
>>>> lat (msec) : 2=70.57%, 4=7.91%, 10=3.11%, 20=0.17%, 50=0.01%
>>>> lat (msec) : 100=0.01%
>>>> cpu : usr=3.12%, sys=11.49%, ctx=74951, majf=0, minf=133
>>>> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%,
>>>>> =64=0.0%
>>>> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>>>>> =64=0.0%
>>>> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
>>>>> =64=0.0%
>>>> issued : total=r=0/w=626979/d=0, short=r=0/w=0/d=0
>>>>
>>>> Run status group 0 (all jobs):
>>>> WRITE: io=2449.2MB, aggrb=83583KB/s, minb=83583KB/s, maxb=83583KB/s,
>>>> mint=30005msec, maxt=30005msec
>>>>
>>>> Disk stats (read/write):
>>>> sdb: ios=168/625292, merge=0/0, ticks=144/916096, in_queue=916128,
>>>> util=99.93%
>>>>
>>>>
>>>> firefly:
>>>>
>>>> fio --filename=/dev/sdb --direct=1 --rw=randwrite --bs=4k --numjobs=1
>>>> --thread --iodepth=32 --ioengine=libaio --runtime=30 --group_reporting
>>>> --name=file1
>>>> file1: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=32
>>>> 2.0.8
>>>> Starting 1 thread
>>>> Jobs: 1 (f=1): [w] [100.0% done] [0K/90044K /s] [0 /22.6K iops] [eta
>>>> 00m:00s]
>>>> file1: (groupid=0, jobs=1): err= 0: pid=2970
>>>> write: io=2372.9MB, bw=80976KB/s, iops=20244 , runt= 30006msec
>>>> slat (usec): min=1 , max=4047 , avg= 4.36, stdev= 7.17
>>>> clat (usec): min=197 , max=76656 , avg=1575.29, stdev=1165.74
>>>> lat (usec): min=523 , max=76660 , avg=1579.79, stdev=1165.59
>>>> clat percentiles (usec):
>>>> | 1.00th=[ 676], 5.00th=[ 804], 10.00th=[ 916], 20.00th=[
>>>> 1096],
>>>> | 30.00th=[ 1224], 40.00th=[ 1304], 50.00th=[ 1384], 60.00th=[
>>>> 1448],
>>>> | 70.00th=[ 1544], 80.00th=[ 1704], 90.00th=[ 2128], 95.00th=[
>>>> 2736],
>>>> | 99.00th=[ 6752], 99.50th=[ 7904], 99.90th=[12096],
>>>> 99.95th=[14656],
>>>> | 99.99th=[18560]
>>>> bw (KB/s) : min=47800, max=91952, per=99.91%, avg=80900.88,
>>>> stdev=7234.98
>>>> lat (usec) : 250=0.01%, 500=0.01%, 750=2.95%, 1000=11.38%
>>>> lat (msec) : 2=73.81%, 4=8.81%, 10=2.85%, 20=0.19%, 50=0.01%
>>>> lat (msec) : 100=0.01%
>>>> cpu : usr=2.99%, sys=10.60%, ctx=66549, majf=0, minf=133
>>>> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%,
>>>>> =64=0.0%
>>>> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>>>>> =64=0.0%
>>>> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
>>>>> =64=0.0%
>>>> issued : total=r=0/w=607445/d=0, short=r=0/w=0/d=0
>>>>
>>>> Run status group 0 (all jobs):
>>>> WRITE: io=2372.9MB, aggrb=80976KB/s, minb=80976KB/s, maxb=80976KB/s,
>>>> mint=30006msec, maxt=30006msec
>>>>
>>>> Disk stats (read/write):
>>>> sdb: ios=170/605440, merge=0/0, ticks=156/916492, in_queue=916560,
>>>> util=99.93%
>>>>
>>>> Stefan
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 32+ messages in thread
end of thread, other threads:[~2015-02-17 9:31 UTC | newest]
Thread overview: 32+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2015-02-10 18:55 speed decrease since firefly,giant,hammer the 2nd try Stefan Priebe
2015-02-10 19:05 ` Gregory Farnum
2015-02-10 19:12 ` Stefan Priebe
2015-02-10 19:10 ` Mark Nelson
2015-02-10 19:13 ` Stefan Priebe
2015-02-10 19:40 ` Mark Nelson
2015-02-10 20:24 ` Stefan Priebe
2015-02-10 20:36 ` Mark Nelson
2015-02-10 21:11 ` Stefan Priebe
2015-02-10 21:38 ` Mark Nelson
2015-02-10 22:18 ` Stefan Priebe
2015-02-11 4:45 ` Mark Nelson
2015-02-11 5:42 ` Stefan Priebe
[not found] ` <1032317804.358821.1423634390142.JavaMail.zimbra@oxygem.tv>
2015-02-11 5:59 ` Alexandre DERUMIER
2015-02-15 18:40 ` Stefan Priebe
[not found] ` <1880785650.892634.1424077856500.JavaMail.zimbra@oxygem.tv>
2015-02-16 9:11 ` Alexandre DERUMIER
2015-02-16 14:50 ` Stefan Priebe - Profihost AG
2015-02-16 15:35 ` Mark Nelson
2015-02-16 15:36 ` Alexandre DERUMIER
2015-02-16 19:10 ` Stefan Priebe
[not found] ` <198700848.1004998.1424124086231.JavaMail.zimbra@oxygem.tv>
2015-02-16 22:01 ` Alexandre DERUMIER
[not found] ` <1760409866.980984.1424105112443.JavaMail.zimbra@oxygem.tv>
2015-02-16 16:45 ` Alexandre DERUMIER
2015-02-16 19:40 ` Stefan Priebe
2015-02-16 21:22 ` Stefan Priebe
[not found] ` <2015376321.1005004.1424124122884.JavaMail.zimbra@oxygem.tv>
2015-02-16 22:02 ` Alexandre DERUMIER
2015-02-16 22:08 ` Stefan Priebe - Profihost AG
[not found] ` <1975903570.1005549.1424125095286.JavaMail.zimbra@oxygem.tv>
2015-02-16 22:18 ` Alexandre DERUMIER
2015-02-17 9:15 ` Stefan Priebe - Profihost AG
[not found] ` <1999473015.1036573.1424165503094.JavaMail.zimbra@oxygem.tv>
2015-02-17 9:31 ` Alexandre DERUMIER
[not found] ` <764258391.358931.1423634637318.JavaMail.zimbra@oxygem.tv>
2015-02-11 6:04 ` Alexandre DERUMIER
[not found] ` <1971513819.362434.1423640637237.JavaMail.zimbra@oxygem.tv>
2015-02-11 7:44 ` Alexandre DERUMIER
2015-02-11 8:32 ` Stefan Priebe - Profihost AG
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.