From mboxrd@z Thu Jan 1 00:00:00 1970 From: shiva rkreddy Subject: Re: v0.80.8 and librbd performance Date: Tue, 14 Apr 2015 22:01:18 -0500 Message-ID: References: Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============1586584730==" Return-path: In-Reply-To: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ceph-users-bounces-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org Sender: "ceph-users" To: Josh Durgin Cc: Sage Weil , Ceph Development , "ceph-users-Qp0mS5GaXlQ@public.gmane.org" List-Id: ceph-devel.vger.kernel.org --===============1586584730== Content-Type: multipart/alternative; boundary=20cf303dd3eaa34dca0513ba8f37 --20cf303dd3eaa34dca0513ba8f37 Content-Type: text/plain; charset=UTF-8 The clusters are in test environment, so its a new deployment of 0.80.9. OS on the cluster nodes is reinstalled as well, so there shouldn't be any fs aging unless the disks are slowing down. The perf measurement is done initiating multiple cinder create/delete commands and tracking the volume to be in available or completely gone from "cinder list" output. Even running "rbd rm " command from cinder node results in similar behaviour. I'll try with increasing rbd_concurrent_management in ceph.conf. Is the param name rbd_concurrent_management or rbd-concurrent-management ? On Tue, Apr 14, 2015 at 12:36 PM, Josh Durgin wrote: > I don't see any commits that would be likely to affect that between 0.80.7 > and 0.80.9. > > Is this after upgrading an existing cluster? > Could this be due to fs aging beneath your osds? > > How are you measuring create/delete performance? > > You can try increasing rbd concurrent management ops in ceph.conf on the > cinder node. This affects delete speed, since rbd tries to delete each > object in a volume. > > Josh > > *From:* shiva rkreddy > *Sent:* Apr 14, 2015 5:53 AM > *To:* Josh Durgin > *Cc:* Ken Dreyer; Sage Weil; Ceph Development; ceph-users-Qp0mS5GaXlQ@public.gmane.org > *Subject:* Re: v0.80.8 and librbd performance > > Hi Josh, > > We are using firefly 0.80.9 and see both cinder create/delete numbers slow > down compared 0.80.7. > I don't see any specific tuning requirements and our cluster is run pretty > much on default configuration. > Do you recommend any tuning or can you please suggest some log signatures > we need to be looking at? > > Thanks > shiva > > On Wed, Mar 4, 2015 at 1:53 PM, Josh Durgin wrote: > >> On 03/03/2015 03:28 PM, Ken Dreyer wrote: >> >>> On 03/03/2015 04:19 PM, Sage Weil wrote: >>> >>>> Hi, >>>> >>>> This is just a heads up that we've identified a performance regression >>>> in >>>> v0.80.8 from previous firefly releases. A v0.80.9 is working it's way >>>> through QA and should be out in a few days. If you haven't upgraded yet >>>> you may want to wait. >>>> >>>> Thanks! >>>> sage >>>> >>> >>> Hi Sage, >>> >>> I've seen a couple Redmine tickets on this (eg >>> http://tracker.ceph.com/issues/9854 , >>> http://tracker.ceph.com/issues/10956). It's not totally clear to me >>> which of the 70+ unreleased commits on the firefly branch fix this >>> librbd issue. Is it only the three commits in >>> https://github.com/ceph/ceph/pull/3410 , or are there more? >>> >> >> Those are the only ones needed to fix the librbd performance >> regression, yes. >> >> Josh >> >> -- >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in >> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html >> > > --20cf303dd3eaa34dca0513ba8f37 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
The clusters are in test environm= ent, so its a new deployment of 0.80.9. OS on the cluster nodes is reinstal= led as well, so there shouldn't be any fs aging unless the disks are sl= owing down.

The perf measurement is done initiating multiple c= inder create/delete commands and tracking the volume to be in available or = completely gone from "cinder list" output.

Eve= n running=C2=A0 "rbd rm " command from cinder node results in sim= ilar behaviour.

I'll try with=C2=A0 increasing=C2=A0 = rbd_concurrent_management in ceph.conf.
=C2=A0Is the param name rbd_conc= urrent_management or rbd-concurrent-management ?
<= br>

On Tue, Apr 14, 2015 at 12:36 PM, Josh Durgin = <jdurgin@redhat.= com> wrote:

I don't see any commits that woul= d be likely to affect that between 0.80.7 and 0.80.9.

Is this after upgrading an existing cluster?
Could this be due to fs aging beneath your osds?

How are you measuring create/delete performance?

You can try increasing rbd concurrent management ops in ceph= .conf on the cinder node. This affects delete speed, since rbd tries to del= ete each object in a volume.

Josh


From: shiva rk= reddy <shiv= a.rkreddy-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
Sent: Apr 14, 2015 5:53 AM
To:<= /b> Josh Durgin
Cc: Ken Dreyer; Sage Weil; Ceph Development; ceph-users-Qp0mS5GaXlQ@public.gmane.org
Subject: Re: v0.80.8 and librbd performance

Hi Josh,
<= br>
We are using firefly 0.80.9 and see both cinder create/delete= numbers slow down compared 0.80.7.
I don't see any specific = tuning requirements and our cluster is run pretty much on default configura= tion.
Do you recommend any tuning or can you please suggest some = log signatures we need to be looking at?

Thanks
shiva

On Wed, Mar 4, 2015 at 1:53 PM, Josh Durgin <<= a href=3D"mailto:jdurgin-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org" target=3D"_blank">jdurgin-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> wrote:
<= span>On 03/03/2015 03:28 PM, Ken Dreyer wrote:
On 03/03/2015 04:19 PM, Sage Weil wrote:
Hi,

This is just a heads up that we've identified a performance regression = in
v0.80.8 from previous firefly releases.=C2=A0 A v0.80.9 is working it's= way
through QA and should be out in a few days.=C2=A0 If you haven't upgrad= ed yet
you may want to wait.

Thanks!
sage

Hi Sage,

I've seen a couple Redmine tickets on this (eg
http://tr= acker.ceph.com/issues/9854 ,
http://t= racker.ceph.com/issues/10956). It's not totally clear to me<= br> which of the 70+ unreleased commits on the firefly branch fix this
librbd issue.=C2=A0 Is it only the three commits in
https:= //github.com/ceph/ceph/pull/3410 , or are there more?

Those are the only ones needed to fix the librbd performance
regression, yes.

Josh

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel&q= uot; in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at=C2=A0 http://vger.kernel.org/majordomo-info.htm= l


--20cf303dd3eaa34dca0513ba8f37-- --===============1586584730== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ ceph-users mailing list ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com --===============1586584730==--