From mboxrd@z Thu Jan 1 00:00:00 1970 From: David Turner Subject: Re: Ceph cluster stability Date: Fri, 22 Feb 2019 06:10:19 -0500 Message-ID: References: Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============7896716898528985112==" Return-path: In-Reply-To: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ceph-users-bounces-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org Sender: "ceph-users" To: M Ranga Swami Reddy Cc: ceph-users , ceph-devel List-Id: ceph-devel.vger.kernel.org --===============7896716898528985112== Content-Type: multipart/alternative; boundary="0000000000009b79ff058279a388" --0000000000009b79ff058279a388 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable What about the system stats on your mons during recovery? If they are having a hard time keeping up with requests during a recovery, I could see that impacting client io. What disks are they running on? CPU? Etc. On Fri, Feb 22, 2019, 6:01 AM M Ranga Swami Reddy wrote: > Debug setting defaults are using..like 1/5 and 0/5 for almost.. > Shall I try with 0 for all debug settings? > > On Wed, Feb 20, 2019 at 9:17 PM Darius Kasparavi=C4=8Dius > wrote: > > > > Hello, > > > > > > Check your CPU usage when you are doing those kind of operations. We > > had a similar issue where our CPU monitoring was reporting fine < 40% > > usage, but our load on the nodes was high mid 60-80. If it's possible > > try disabling ht and see the actual cpu usage. > > If you are hitting CPU limits you can try disabling crc on messages. > > ms_nocrc > > ms_crc_data > > ms_crc_header > > > > And setting all your debug messages to 0. > > If you haven't done you can also lower your recovery settings a little. > > osd recovery max active > > osd max backfills > > > > You can also lower your file store threads. > > filestore op threads > > > > > > If you can also switch to bluestore from filestore. This will also > > lower your CPU usage. I'm not sure that this is bluestore that does > > it, but I'm seeing lower cpu usage when moving to bluestore + rocksdb > > compared to filestore + leveldb . > > > > > > On Wed, Feb 20, 2019 at 4:27 PM M Ranga Swami Reddy > > wrote: > > > > > > Thats expected from Ceph by design. But in our case, we are using all > > > recommendation like rack failure domain, replication n/w,etc, still > > > face client IO performance issues during one OSD down.. > > > > > > On Tue, Feb 19, 2019 at 10:56 PM David Turner > wrote: > > > > > > > > With a RACK failure domain, you should be able to have an entire > rack powered down without noticing any major impact on the clients. I > regularly take down OSDs and nodes for maintenance and upgrades without > seeing any problems with client IO. > > > > > > > > On Tue, Feb 12, 2019 at 5:01 AM M Ranga Swami Reddy < > swamireddy-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote: > > > >> > > > >> Hello - I have a couple of questions on ceph cluster stability, ev= en > > > >> we follow all recommendations as below: > > > >> - Having separate replication n/w and data n/w > > > >> - RACK is the failure domain > > > >> - Using SSDs for journals (1:4ratio) > > > >> > > > >> Q1 - If one OSD down, cluster IO down drastically and customer App= s > impacted. > > > >> Q2 - what is stability ratio, like with above, is ceph cluster > > > >> workable condition, if one osd down or one node down,etc. > > > >> > > > >> Thanks > > > >> Swami > > > >> _______________________________________________ > > > >> ceph-users mailing list > > > >> ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org > > > >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > _______________________________________________ > > > ceph-users mailing list > > > ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > --0000000000009b79ff058279a388 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
What about the system stats on your mons during recovery?= If they are having a hard time keeping up with requests during a recovery,= I could see that impacting client io. What disks are they running on? CPU?= Etc.

On Fri, Feb 22, 2019, 6:01 AM M Ranga Swami Reddy <swamireddy-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
Debug setting defaults are using..like 1/5 and 0/5= for almost..
Shall I try with 0 for all debug settings?

On Wed, Feb 20, 2019 at 9:17 PM Darius Kasparavi=C4=8Dius <daznis-Re5JQEeQqe8@public.gmane.org= m> wrote:
>
> Hello,
>
>
> Check your CPU usage when you are doing those kind of operations. We > had a similar issue where our CPU monitoring was reporting fine < 4= 0%
> usage, but our load on the nodes was high mid 60-80. If it's possi= ble
> try disabling ht and see the actual cpu usage.
> If you are hitting CPU limits you can try disabling crc on messages. > ms_nocrc
> ms_crc_data
> ms_crc_header
>
> And setting all your debug messages to 0.
> If you haven't done you can also lower your recovery settings a li= ttle.
> osd recovery max active
> osd max backfills
>
> You can also lower your file store threads.
> filestore op threads
>
>
> If you can also switch to bluestore from filestore. This will also
> lower your CPU usage. I'm not sure that this is bluestore that doe= s
> it, but I'm seeing lower cpu usage when moving to bluestore + rock= sdb
> compared to filestore + leveldb .
>
>
> On Wed, Feb 20, 2019 at 4:27 PM M Ranga Swami Reddy
> <swamireddy-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> >
> > Thats expected from Ceph by design. But in our case, we are using= all
> > recommendation like rack failure domain, replication n/w,etc, sti= ll
> > face client IO performance issues during one OSD down..
> >
> > On Tue, Feb 19, 2019 at 10:56 PM David Turner <drakonstein@= gmail.com> wrote:
> > >
> > > With a RACK failure domain, you should be able to have an en= tire rack powered down without noticing any major impact on the clients.=C2= =A0 I regularly take down OSDs and nodes for maintenance and upgrades witho= ut seeing any problems with client IO.
> > >
> > > On Tue, Feb 12, 2019 at 5:01 AM M Ranga Swami Reddy <sw= amireddy-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> > >>
> > >> Hello - I have a couple of questions on ceph cluster sta= bility, even
> > >> we follow all recommendations as below:
> > >> - Having separate replication n/w and data n/w
> > >> - RACK is the failure domain
> > >> - Using SSDs for journals (1:4ratio)
> > >>
> > >> Q1 - If one OSD down, cluster IO down drastically and cu= stomer Apps impacted.
> > >> Q2 - what is stability ratio, like with above, is ceph c= luster
> > >> workable condition, if one osd down or one node down,etc= .
> > >>
> > >> Thanks
> > >> Swami
> > >> _______________________________________________
> > >> ceph-users mailing list
> > >> ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
> > >> http://lists.cep= h.com/listinfo.cgi/ceph-users-ceph.com
> > _______________________________________________
> > ceph-users mailing list
> > ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
> > http://lists.ceph.com/lis= tinfo.cgi/ceph-users-ceph.com
--0000000000009b79ff058279a388-- --===============7896716898528985112== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ ceph-users mailing list ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com --===============7896716898528985112==--