Ceph Dumpling/Firefly/Hammer SSD/Memstore performance comparison

Ceph Dumpling/Firefly/Hammer SSD/Memstore performance comparison

[Mark Nelson]

[-- Attachment #1: Type: text/plain, Size: 475 bytes --]

Hi All,

I wrote up a short document describing some tests I ran recently to look 
at how SSD backed OSD performance has changed across our LTS releases. 
This is just looking at RADOS performance and not RBD or RGW.  It also 
doesn't offer any real explanations regarding the results.  It's just a 
first high level step toward understanding some of the behaviors folks 
on the mailing list have reported over the last couple of releases.  I 
hope you find it useful.

Mark

[-- Attachment #2: Ceph_SSD_OSD_Performance.pdf --]
[-- Type: application/pdf, Size: 116024 bytes --]

[-- Attachment #3: Type: text/plain, Size: 178 bytes --]

_______________________________________________
ceph-users mailing list
ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: Ceph Dumpling/Firefly/Hammer SSD/Memstore performance comparison

[Irek Fasikhov]

[-- Attachment #1.1: Type: text/plain, Size: 920 bytes --]

Mark, very very good!

2015-02-17 20:37 GMT+03:00 Mark Nelson <mnelson-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>:

> Hi All,
>
> I wrote up a short document describing some tests I ran recently to look
> at how SSD backed OSD performance has changed across our LTS releases. This
> is just looking at RADOS performance and not RBD or RGW.  It also doesn't
> offer any real explanations regarding the results.  It's just a first high
> level step toward understanding some of the behaviors folks on the mailing
> list have reported over the last couple of releases.  I hope you find it
> useful.
>
> Mark
>
> _______________________________________________
> ceph-users mailing list
> ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>


-- 
С уважением, Фасихов Ирек Нургаязович
Моб.: +79229045757

[-- Attachment #1.2: Type: text/html, Size: 1529 bytes --]

[-- Attachment #2: Type: text/plain, Size: 178 bytes --]

_______________________________________________
ceph-users mailing list
ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: Ceph Dumpling/Firefly/Hammer SSD/Memstore performance comparison

[Andrei Mikhailovsky]

[-- Attachment #1.1: Type: text/plain, Size: 1925 bytes --]

Mark, many thanks for your effort and ceph performance tests. This puts things in perspective. 

Looking at the results, I was a bit concerned that the IOPs performance in niether releases come even marginally close to the capabilities of the underlying ssd device. Even the fastest PCI ssds have only managed to achieve about the 1/6th IOPs of the raw device. 

I guess there is a great deal more optimisations to be done in the upcoming LTS releases to make the IOPs rate close to the raw device performance. 

I have done some testing in the past and noticed that despite the server having a lot of unused resources (about 40-50% server idle and about 60-70% ssd idle) the ceph would not perform well when used with ssds. I was testing with Firefly + auth and my IOPs rate was around the 3K mark. Something is holding ceph back from performing well with ssds ((( 

Andrei 

----- Original Message -----

> From: "Mark Nelson" <mnelson-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
> To: "ceph-devel" <ceph-devel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org>
> Cc: ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
> Sent: Tuesday, 17 February, 2015 5:37:01 PM
> Subject: [ceph-users] Ceph Dumpling/Firefly/Hammer SSD/Memstore
> performance comparison

> Hi All,

> I wrote up a short document describing some tests I ran recently to
> look
> at how SSD backed OSD performance has changed across our LTS
> releases.
> This is just looking at RADOS performance and not RBD or RGW. It also
> doesn't offer any real explanations regarding the results. It's just
> a
> first high level step toward understanding some of the behaviors
> folks
> on the mailing list have reported over the last couple of releases. I
> hope you find it useful.

> Mark

> _______________________________________________
> ceph-users mailing list
> ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[-- Attachment #1.2: Type: text/html, Size: 2547 bytes --]

[-- Attachment #2: Type: text/plain, Size: 178 bytes --]

_______________________________________________
ceph-users mailing list
ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph Dumpling/Firefly/Hammer SSD/Memstore performance comparison

[Mark Nelson]

Hi Andrei,

On 02/18/2015 09:08 AM, Andrei Mikhailovsky wrote:
>
> Mark, many thanks for your effort and ceph performance tests. This puts
> things in perspective.
>
> Looking at the results, I was a bit concerned that the IOPs performance
> in niether releases come even marginally close to the capabilities of
> the underlying ssd device. Even the fastest PCI ssds have only managed
> to achieve about the 1/6th IOPs of the raw device.

Perspective is definitely good!  Any time you are dealing with latency 
sensitive workloads, there are a lot of bottlenecks that can limit your 
performance.  There's a world of difference between streaming data to a 
raw SSD as fast as possible and writing data out to a distributed 
storage system that is calculating data placement, invoking the TCP 
stack, doing CRC checks, journaling writes, invoking the VM layer to 
cache data in case it's hot (which in this case it's not).

>
> I guess there is a great deal more optimisations to be done in the
> upcoming LTS releases to make the IOPs rate close to the raw device
> performance.

There is definitely still room for improvement!  It's important to 
remember though that there is always going to be a trade off between 
flexibility, data integrity, and performance.  If low latency is your 
number one need before anything else, you are probably best off 
eliminating as much software as possible between you and the device 
(except possibly if you can make clever use of caching).  While Ceph 
itself is some times the bottleneck, in many cases we've found that 
bottlenecks in the software that surrounds Ceph are just as big 
obstacles (filesystem, VM layer, TCP stack, leveldb, etc).  If you need 
a distributed storage system that can universally maintain native SSD 
levels of performance, the entire stack has to be highly tuned.

>
> I have done some testing in the past and noticed that despite the server
> having a lot of unused resources (about 40-50% server idle and about
> 60-70% ssd idle) the ceph would not perform well when used with ssds. I
> was testing with Firefly + auth and my IOPs rate was around the 3K mark.
> Something is holding ceph back from performing well with ssds (((

Out of curiosity, did you try the same tests directly on the SSD?

>
> Andrei
>
> ------------------------------------------------------------------------
>
>     *From: *"Mark Nelson" <mnelson@redhat.com>
>     *To: *"ceph-devel" <ceph-devel@vger.kernel.org>
>     *Cc: *ceph-users@lists.ceph.com
>     *Sent: *Tuesday, 17 February, 2015 5:37:01 PM
>     *Subject: *[ceph-users] Ceph Dumpling/Firefly/Hammer SSD/Memstore
>     performance        comparison
>
>     Hi All,
>
>     I wrote up a short document describing some tests I ran recently to
>     look
>     at how SSD backed OSD performance has changed across our LTS releases.
>     This is just looking at RADOS performance and not RBD or RGW.  It also
>     doesn't offer any real explanations regarding the results.  It's just a
>     first high level step toward understanding some of the behaviors folks
>     on the mailing list have reported over the last couple of releases.  I
>     hope you find it useful.
>
>     Mark
>
>     _______________________________________________
>     ceph-users mailing list
>     ceph-users@lists.ceph.com
>     http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>

Re: [ceph-users] Ceph Dumpling/Firefly/Hammer SSD/Memstore performance comparison

[Stephen Hindle]

I was wondering what the 'CBT' tool is ?  Google is useless for that acronym...

Thanks!
Steve

On Tue, Feb 17, 2015 at 10:37 AM, Mark Nelson <mnelson@redhat.com> wrote:
> Hi All,
>
> I wrote up a short document describing some tests I ran recently to look at
> how SSD backed OSD performance has changed across our LTS releases. This is
> just looking at RADOS performance and not RBD or RGW.  It also doesn't offer
> any real explanations regarding the results.  It's just a first high level
> step toward understanding some of the behaviors folks on the mailing list
> have reported over the last couple of releases.  I hope you find it useful.
>
> Mark
>
> _______________________________________________
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>

-- 
The information in this message may be confidential.  It is intended solely 
for
the addressee(s).  If you are not the intended recipient, any disclosure,
copying or distribution of the message, or any action or omission taken by 
you
in reliance on it, is prohibited and may be unlawful.  Please immediately
contact the sender if you have received this message in error.

Re: Ceph Dumpling/Firefly/Hammer SSD/Memstore performance comparison

[Tyler Brekke]

[-- Attachment #1.1: Type: text/plain, Size: 1699 bytes --]

https://github.com/ceph/ceph-tools/tree/master/cbt

On Tue, Feb 17, 2015 at 12:16 PM, Stephen Hindle <shindle-zIyXdg6N6CA@public.gmane.org> wrote:

> I was wondering what the 'CBT' tool is ?  Google is useless for that
> acronym...
>
> Thanks!
> Steve
>
> On Tue, Feb 17, 2015 at 10:37 AM, Mark Nelson <mnelson-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> wrote:
> > Hi All,
> >
> > I wrote up a short document describing some tests I ran recently to look
> at
> > how SSD backed OSD performance has changed across our LTS releases. This
> is
> > just looking at RADOS performance and not RBD or RGW.  It also doesn't
> offer
> > any real explanations regarding the results.  It's just a first high
> level
> > step toward understanding some of the behaviors folks on the mailing list
> > have reported over the last couple of releases.  I hope you find it
> useful.
> >
> > Mark
> >
> > _______________________________________________
> > ceph-users mailing list
> > ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >
>
> --
> The information in this message may be confidential.  It is intended solely
> for
> the addressee(s).  If you are not the intended recipient, any disclosure,
> copying or distribution of the message, or any action or omission taken by
> you
> in reliance on it, is prohibited and may be unlawful.  Please immediately
> contact the sender if you have received this message in error.
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

[-- Attachment #1.2: Type: text/html, Size: 2641 bytes --]

[-- Attachment #2: Type: text/plain, Size: 178 bytes --]

_______________________________________________
ceph-users mailing list
ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: Ceph Dumpling/Firefly/Hammer SSD/Memstore performance comparison

[Karan Singh]

[-- Attachment #1.1.1: Type: text/plain, Size: 2056 bytes --]

Thanks Mark , for a superb explanation . This is indeed very useful.

****************************************************************
Karan Singh 
Systems Specialist , Storage Platforms
CSC - IT Center for Science,
Keilaranta 14, P. O. Box 405, FIN-02101 Espoo, Finland
mobile: +358 503 812758
tel. +358 9 4572001
fax +358 9 4572302
http://www.csc.fi/
****************************************************************

> On 17 Feb 2015, at 22:16, Stephen Hindle <shindle-zIyXdg6N6CA@public.gmane.org> wrote:
> 
> I was wondering what the 'CBT' tool is ?  Google is useless for that acronym...
> 
> Thanks!
> Steve
> 
> On Tue, Feb 17, 2015 at 10:37 AM, Mark Nelson <mnelson-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> wrote:
>> Hi All,
>> 
>> I wrote up a short document describing some tests I ran recently to look at
>> how SSD backed OSD performance has changed across our LTS releases. This is
>> just looking at RADOS performance and not RBD or RGW.  It also doesn't offer
>> any real explanations regarding the results.  It's just a first high level
>> step toward understanding some of the behaviors folks on the mailing list
>> have reported over the last couple of releases.  I hope you find it useful.
>> 
>> Mark
>> 
>> _______________________________________________
>> ceph-users mailing list
>> ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>> 
> 
> -- 
> The information in this message may be confidential.  It is intended solely 
> for
> the addressee(s).  If you are not the intended recipient, any disclosure,
> copying or distribution of the message, or any action or omission taken by 
> you
> in reliance on it, is prohibited and may be unlawful.  Please immediately
> contact the sender if you have received this message in error.
> 
> _______________________________________________
> ceph-users mailing list
> ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


[-- Attachment #1.1.2: Type: text/html, Size: 6636 bytes --]

[-- Attachment #1.2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 4436 bytes --]

[-- Attachment #2: Type: text/plain, Size: 178 bytes --]

_______________________________________________
ceph-users mailing list
ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph Dumpling/Firefly/Hammer SSD/Memstore performance comparison

[Mark Nelson]

Hi Stephen,

It's a benchmark automation tool we wrote that builds a ceph cluster and 
then runs benchmarks against it.  It's still pretty rough (no real error 
checking,  no documentation, etc).  We have some partners that are 
interested in using it too and I'd like to make it useful for the 
community so we're going to try to make it a bit more accessible.

cbt is here:

https://github.com/ceph/ceph-tools/tree/master/cbt

We've also been using it to prototype nightly performance testing of 
firefly and master for the last month or two on some of our lab nodes. 
The cron job and test suites are here:

https://github.com/ceph/ceph-tools/tree/master/regression

Mark


On 02/17/2015 02:16 PM, Stephen Hindle wrote:
> I was wondering what the 'CBT' tool is ?  Google is useless for that acronym...
>
> Thanks!
> Steve
>
> On Tue, Feb 17, 2015 at 10:37 AM, Mark Nelson <mnelson@redhat.com> wrote:
>> Hi All,
>>
>> I wrote up a short document describing some tests I ran recently to look at
>> how SSD backed OSD performance has changed across our LTS releases. This is
>> just looking at RADOS performance and not RBD or RGW.  It also doesn't offer
>> any real explanations regarding the results.  It's just a first high level
>> step toward understanding some of the behaviors folks on the mailing list
>> have reported over the last couple of releases.  I hope you find it useful.
>>
>> Mark
>>
>> _______________________________________________
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>

Re: [ceph-users] Ceph Dumpling/Firefly/Hammer SSD/Memstore performance comparison

[Stephen Hindle]

Awesome!  Thanks Much!

On Tue, Feb 17, 2015 at 1:28 PM, Mark Nelson <mnelson@redhat.com> wrote:
> Hi Stephen,
>
> It's a benchmark automation tool we wrote that builds a ceph cluster and
> then runs benchmarks against it.  It's still pretty rough (no real error
> checking,  no documentation, etc).  We have some partners that are
> interested in using it too and I'd like to make it useful for the community
> so we're going to try to make it a bit more accessible.
>
> cbt is here:
>
> https://github.com/ceph/ceph-tools/tree/master/cbt
>
> We've also been using it to prototype nightly performance testing of firefly
> and master for the last month or two on some of our lab nodes. The cron job
> and test suites are here:
>
> https://github.com/ceph/ceph-tools/tree/master/regression
>
> Mark
>
>
>
> On 02/17/2015 02:16 PM, Stephen Hindle wrote:
>>
>> I was wondering what the 'CBT' tool is ?  Google is useless for that
>> acronym...
>>
>> Thanks!
>> Steve
>>
>> On Tue, Feb 17, 2015 at 10:37 AM, Mark Nelson <mnelson@redhat.com> wrote:
>>>
>>> Hi All,
>>>
>>> I wrote up a short document describing some tests I ran recently to look
>>> at
>>> how SSD backed OSD performance has changed across our LTS releases. This
>>> is
>>> just looking at RADOS performance and not RBD or RGW.  It also doesn't
>>> offer
>>> any real explanations regarding the results.  It's just a first high
>>> level
>>> step toward understanding some of the behaviors folks on the mailing list
>>> have reported over the last couple of releases.  I hope you find it
>>> useful.
>>>
>>> Mark
>>>
>>> _______________________________________________
>>> ceph-users mailing list
>>> ceph-users@lists.ceph.com
>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>>
>>
>

-- 
The information in this message may be confidential.  It is intended solely 
for
the addressee(s).  If you are not the intended recipient, any disclosure,
copying or distribution of the message, or any action or omission taken by 
you
in reliance on it, is prohibited and may be unlawful.  Please immediately
contact the sender if you have received this message in error.

Re: [ceph-users] Ceph Dumpling/Firefly/Hammer SSD/Memstore performance comparison

[Alexandre DERUMIER]

Nice Work Mark !

I don't see any tuning about sharding in the config file sample

(osd_op_num_threads_per_shard,osd_op_num_shards,...)

as you only use 1 ssd for the bench, I think it should improve results for hammer ?



----- Mail original -----
De: "Mark Nelson" <mnelson@redhat.com>
À: "ceph-devel" <ceph-devel@vger.kernel.org>
Cc: "ceph-users" <ceph-users@lists.ceph.com>
Envoyé: Mardi 17 Février 2015 18:37:01
Objet: [ceph-users] Ceph Dumpling/Firefly/Hammer SSD/Memstore performance	comparison

Hi All, 

I wrote up a short document describing some tests I ran recently to look 
at how SSD backed OSD performance has changed across our LTS releases. 
This is just looking at RADOS performance and not RBD or RGW. It also 
doesn't offer any real explanations regarding the results. It's just a 
first high level step toward understanding some of the behaviors folks 
on the mailing list have reported over the last couple of releases. I 
hope you find it useful. 

Mark 

_______________________________________________ 
ceph-users mailing list 
ceph-users@lists.ceph.com 
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com 
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Ceph Dumpling/Firefly/Hammer SSD/Memstore performance comparison

[Mark Nelson]

Hi Alex,

Thanks!  I didn't tweak the sharding settings at all, so they are just 
at the default values:

OPTION(osd_op_num_threads_per_shard, OPT_INT, 2)
OPTION(osd_op_num_shards, OPT_INT, 5)

I don't have really good insight yet into how tweaking these would 
affect single-osd performance.  I know the PCIe SSDs do have multiple 
controllers on-board so perhaps increasing the number of shards would 
improve things, but I suspect that going too high could maybe start 
hurting performance as well.  Have you done any testing here?  It could 
be an interesting follow-up paper.

Mark

On 02/18/2015 02:34 AM, Alexandre DERUMIER wrote:
> Nice Work Mark !
>
> I don't see any tuning about sharding in the config file sample
>
> (osd_op_num_threads_per_shard,osd_op_num_shards,...)
>
> as you only use 1 ssd for the bench, I think it should improve results for hammer ?
>
>
>
> ----- Mail original -----
> De: "Mark Nelson" <mnelson@redhat.com>
> À: "ceph-devel" <ceph-devel@vger.kernel.org>
> Cc: "ceph-users" <ceph-users@lists.ceph.com>
> Envoyé: Mardi 17 Février 2015 18:37:01
> Objet: [ceph-users] Ceph Dumpling/Firefly/Hammer SSD/Memstore performance	comparison
>
> Hi All,
>
> I wrote up a short document describing some tests I ran recently to look
> at how SSD backed OSD performance has changed across our LTS releases.
> This is just looking at RADOS performance and not RBD or RGW. It also
> doesn't offer any real explanations regarding the results. It's just a
> first high level step toward understanding some of the behaviors folks
> on the mailing list have reported over the last couple of releases. I
> hope you find it useful.
>
> Mark
>
> _______________________________________________
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: Ceph Dumpling/Firefly/Hammer SSD/Memstore performance comparison

[Alexandre DERUMIER]

>>I don't have really good insight yet into how tweaking these would 
>>affect single-osd performance. I know the PCIe SSDs do have multiple 
>>controllers on-board so perhaps increasing the number of shards would 
>>improve things, but I suspect that going too high could maybe start 
>>hurting performance as well. Have you done any testing here? It could 
>>be an interesting follow-up paper. 

I think it should be tunned regarding number of osds and number of cores you have.
I have done test in past with sommath values

osd_op_num_threads_per_shard = 1
osd_op_num_shards = 25
filestore_fd_cache_size = 64
filestore_fd_cache_shards = 32

But don't have take time to try differents values.
But I was to be able to reach 120000iops 4k read with 3osd if I remember. (But I was limited by client cpu)

I'm going to do big benchmark next month (3 nodes (20cores) with 6ssd each),
So I'll try to test different sharding values, with different number of osd.


----- Mail original -----
De: "Mark Nelson" <mnelson@redhat.com>
À: "aderumier" <aderumier@odiso.com>
Cc: "ceph-devel" <ceph-devel@vger.kernel.org>, "ceph-users" <ceph-users@lists.ceph.com>
Envoyé: Mercredi 18 Février 2015 15:56:44
Objet: Re: [ceph-users] Ceph Dumpling/Firefly/Hammer SSD/Memstore performance comparison

Hi Alex, 

Thanks! I didn't tweak the sharding settings at all, so they are just 
at the default values: 

OPTION(osd_op_num_threads_per_shard, OPT_INT, 2) 
OPTION(osd_op_num_shards, OPT_INT, 5) 

I don't have really good insight yet into how tweaking these would 
affect single-osd performance. I know the PCIe SSDs do have multiple 
controllers on-board so perhaps increasing the number of shards would 
improve things, but I suspect that going too high could maybe start 
hurting performance as well. Have you done any testing here? It could 
be an interesting follow-up paper. 

Mark 

On 02/18/2015 02:34 AM, Alexandre DERUMIER wrote: 
> Nice Work Mark ! 
> 
> I don't see any tuning about sharding in the config file sample 
> 
> (osd_op_num_threads_per_shard,osd_op_num_shards,...) 
> 
> as you only use 1 ssd for the bench, I think it should improve results for hammer ? 
> 
> 
> 
> ----- Mail original ----- 
> De: "Mark Nelson" <mnelson@redhat.com> 
> À: "ceph-devel" <ceph-devel@vger.kernel.org> 
> Cc: "ceph-users" <ceph-users@lists.ceph.com> 
> Envoyé: Mardi 17 Février 2015 18:37:01 
> Objet: [ceph-users] Ceph Dumpling/Firefly/Hammer SSD/Memstore performance comparison 
> 
> Hi All, 
> 
> I wrote up a short document describing some tests I ran recently to look 
> at how SSD backed OSD performance has changed across our LTS releases. 
> This is just looking at RADOS performance and not RBD or RGW. It also 
> doesn't offer any real explanations regarding the results. It's just a 
> first high level step toward understanding some of the behaviors folks 
> on the mailing list have reported over the last couple of releases. I 
> hope you find it useful. 
> 
> Mark 
> 
> _______________________________________________ 
> ceph-users mailing list 
> ceph-users@lists.ceph.com 
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com 
> -- 
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in 
> the body of a message to majordomo@vger.kernel.org 
> More majordomo info at http://vger.kernel.org/majordomo-info.html 
> 

_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph Dumpling/Firefly/Hammer SSD/Memstore performance comparison

[Gregory Farnum]

On Tue, Feb 17, 2015 at 9:37 AM, Mark Nelson <mnelson@redhat.com> wrote:
> Hi All,
>
> I wrote up a short document describing some tests I ran recently to look at
> how SSD backed OSD performance has changed across our LTS releases. This is
> just looking at RADOS performance and not RBD or RGW.  It also doesn't offer
> any real explanations regarding the results.  It's just a first high level
> step toward understanding some of the behaviors folks on the mailing list
> have reported over the last couple of releases.  I hope you find it useful.

Do you have any work scheduled to examine the synchronous IO latency
changes across versions? I suspect those are involved with the loss of
performance some users have reported, and I've not heard any
believable theories as to the cause. Since this is the first set of
results pointing that way on hardware available for detailed tests I
hope we can dig into it. And those per-op latencies are the next thing
we'll need to cut down on, since they correspond pretty directly with
CPU costs that we want to scale down! :)
-Greg

Re: [ceph-users] Ceph Dumpling/Firefly/Hammer SSD/Memstore performance comparison

[Haomai Wang]

I don't have detail perf number for sync io latency now.

But a few days ago I did single OSD single io depth benchmark. In
short, Firefly > Dumpling > Hammer per op latency.

It's great to see Mark's benchmark result! As for pcie ssd, I think
ceph can't make full use of it currently for one OSD. We may need to
mainly focus on sata-ssd improvments.

On Mon, Feb 23, 2015 at 1:09 PM, Gregory Farnum <greg@gregs42.com> wrote:
> On Tue, Feb 17, 2015 at 9:37 AM, Mark Nelson <mnelson@redhat.com> wrote:
>> Hi All,
>>
>> I wrote up a short document describing some tests I ran recently to look at
>> how SSD backed OSD performance has changed across our LTS releases. This is
>> just looking at RADOS performance and not RBD or RGW.  It also doesn't offer
>> any real explanations regarding the results.  It's just a first high level
>> step toward understanding some of the behaviors folks on the mailing list
>> have reported over the last couple of releases.  I hope you find it useful.
>
> Do you have any work scheduled to examine the synchronous IO latency
> changes across versions? I suspect those are involved with the loss of
> performance some users have reported, and I've not heard any
> believable theories as to the cause. Since this is the first set of
> results pointing that way on hardware available for detailed tests I
> hope we can dig into it. And those per-op latencies are the next thing
> we'll need to cut down on, since they correspond pretty directly with
> CPU costs that we want to scale down! :)
> -Greg
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



-- 
Best Regards,

Wheat

Re: [ceph-users] Ceph Dumpling/Firefly/Hammer SSD/Memstore performance comparison

[Mark Nelson]

On 02/22/2015 11:09 PM, Gregory Farnum wrote:
> On Tue, Feb 17, 2015 at 9:37 AM, Mark Nelson <mnelson@redhat.com> wrote:
>> Hi All,
>>
>> I wrote up a short document describing some tests I ran recently to look at
>> how SSD backed OSD performance has changed across our LTS releases. This is
>> just looking at RADOS performance and not RBD or RGW.  It also doesn't offer
>> any real explanations regarding the results.  It's just a first high level
>> step toward understanding some of the behaviors folks on the mailing list
>> have reported over the last couple of releases.  I hope you find it useful.
>
> Do you have any work scheduled to examine the synchronous IO latency
> changes across versions? I suspect those are involved with the loss of
> performance some users have reported, and I've not heard any
> believable theories as to the cause. Since this is the first set of
> results pointing that way on hardware available for detailed tests I
> hope we can dig into it. And those per-op latencies are the next thing
> we'll need to cut down on, since they correspond pretty directly with
> CPU costs that we want to scale down! :)

I kind of suspect that some of the user issues might be RBD on the 
client side (or even QEMU).  Certainly possible it's OSD too though. 
With so many different kinds of hardware and so many ways to tune things 
it can be tough to narrow down.  I'm really looking forward to the work 
that's being done on LTTNG tracing.  That will give us much easier 
insight into these kinds of things than we've had in the past.

> -Greg
>