From mboxrd@z Thu Jan 1 00:00:00 1970 From: David Turner Subject: Re: Ceph cluster stability Date: Fri, 22 Feb 2019 06:43:42 -0500 Message-ID: References: Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============1902477026652385573==" Return-path: In-Reply-To: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ceph-users-bounces-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org Sender: "ceph-users" To: M Ranga Swami Reddy Cc: ceph-users , ceph-devel List-Id: ceph-devel.vger.kernel.org --===============1902477026652385573== Content-Type: multipart/alternative; boundary="000000000000fcf8a805827a1a1d" --000000000000fcf8a805827a1a1d Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Mon disks don't have journals, they're just a folder on a filesystem on a disk. On Fri, Feb 22, 2019, 6:40 AM M Ranga Swami Reddy wrote: > ceph mons looks fine during the recovery. Using HDD with SSD > journals. with recommeded CPU and RAM numbers. > > On Fri, Feb 22, 2019 at 4:40 PM David Turner > wrote: > > > > What about the system stats on your mons during recovery? If they are > having a hard time keeping up with requests during a recovery, I could se= e > that impacting client io. What disks are they running on? CPU? Etc. > > > > On Fri, Feb 22, 2019, 6:01 AM M Ranga Swami Reddy > wrote: > >> > >> Debug setting defaults are using..like 1/5 and 0/5 for almost.. > >> Shall I try with 0 for all debug settings? > >> > >> On Wed, Feb 20, 2019 at 9:17 PM Darius Kasparavi=C4=8Dius > wrote: > >> > > >> > Hello, > >> > > >> > > >> > Check your CPU usage when you are doing those kind of operations. We > >> > had a similar issue where our CPU monitoring was reporting fine < 40= % > >> > usage, but our load on the nodes was high mid 60-80. If it's possibl= e > >> > try disabling ht and see the actual cpu usage. > >> > If you are hitting CPU limits you can try disabling crc on messages. > >> > ms_nocrc > >> > ms_crc_data > >> > ms_crc_header > >> > > >> > And setting all your debug messages to 0. > >> > If you haven't done you can also lower your recovery settings a > little. > >> > osd recovery max active > >> > osd max backfills > >> > > >> > You can also lower your file store threads. > >> > filestore op threads > >> > > >> > > >> > If you can also switch to bluestore from filestore. This will also > >> > lower your CPU usage. I'm not sure that this is bluestore that does > >> > it, but I'm seeing lower cpu usage when moving to bluestore + rocksd= b > >> > compared to filestore + leveldb . > >> > > >> > > >> > On Wed, Feb 20, 2019 at 4:27 PM M Ranga Swami Reddy > >> > wrote: > >> > > > >> > > Thats expected from Ceph by design. But in our case, we are using > all > >> > > recommendation like rack failure domain, replication n/w,etc, stil= l > >> > > face client IO performance issues during one OSD down.. > >> > > > >> > > On Tue, Feb 19, 2019 at 10:56 PM David Turner < > drakonstein-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote: > >> > > > > >> > > > With a RACK failure domain, you should be able to have an entire > rack powered down without noticing any major impact on the clients. I > regularly take down OSDs and nodes for maintenance and upgrades without > seeing any problems with client IO. > >> > > > > >> > > > On Tue, Feb 12, 2019 at 5:01 AM M Ranga Swami Reddy < > swamireddy-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote: > >> > > >> > >> > > >> Hello - I have a couple of questions on ceph cluster stability, > even > >> > > >> we follow all recommendations as below: > >> > > >> - Having separate replication n/w and data n/w > >> > > >> - RACK is the failure domain > >> > > >> - Using SSDs for journals (1:4ratio) > >> > > >> > >> > > >> Q1 - If one OSD down, cluster IO down drastically and customer > Apps impacted. > >> > > >> Q2 - what is stability ratio, like with above, is ceph cluster > >> > > >> workable condition, if one osd down or one node down,etc. > >> > > >> > >> > > >> Thanks > >> > > >> Swami > >> > > >> _______________________________________________ > >> > > >> ceph-users mailing list > >> > > >> ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org > >> > > >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > >> > > _______________________________________________ > >> > > ceph-users mailing list > >> > > ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org > >> > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > --000000000000fcf8a805827a1a1d Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Mon disks don't have journals, they're just a fol= der on a filesystem on a disk.

On Fri, Feb 22, 2019, 6:40 AM M Ranga Swami R= eddy <swamireddy-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org&g= t; wrote:
ceph mons looks fine duri= ng the recovery.=C2=A0 Using=C2=A0 HDD with SSD
journals. with recommeded CPU and RAM numbers.

On Fri, Feb 22, 2019 at 4:40 PM David Turner <drakonstein-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>
> What about the system stats on your mons during recovery? If they are = having a hard time keeping up with requests during a recovery, I could see = that impacting client io. What disks are they running on? CPU? Etc.
>
> On Fri, Feb 22, 2019, 6:01 AM M Ranga Swami Reddy <swamireddy@gma= il.com> wrote:
>>
>> Debug setting defaults are using..like 1/5 and 0/5 for almost.. >> Shall I try with 0 for all debug settings?
>>
>> On Wed, Feb 20, 2019 at 9:17 PM Darius Kasparavi=C4=8Dius <daznis= @gmail.com> wrote:
>> >
>> > Hello,
>> >
>> >
>> > Check your CPU usage when you are doing those kind of operati= ons. We
>> > had a similar issue where our CPU monitoring was reporting fi= ne < 40%
>> > usage, but our load on the nodes was high mid 60-80. If it= 9;s possible
>> > try disabling ht and see the actual cpu usage.
>> > If you are hitting CPU limits you can try disabling crc on me= ssages.
>> > ms_nocrc
>> > ms_crc_data
>> > ms_crc_header
>> >
>> > And setting all your debug messages to 0.
>> > If you haven't done you can also lower your recovery sett= ings a little.
>> > osd recovery max active
>> > osd max backfills
>> >
>> > You can also lower your file store threads.
>> > filestore op threads
>> >
>> >
>> > If you can also switch to bluestore from filestore. This will= also
>> > lower your CPU usage. I'm not sure that this is bluestore= that does
>> > it, but I'm seeing lower cpu usage when moving to bluesto= re + rocksdb
>> > compared to filestore + leveldb .
>> >
>> >
>> > On Wed, Feb 20, 2019 at 4:27 PM M Ranga Swami Reddy
>> > <swamireddy-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>> > >
>> > > Thats expected from Ceph by design. But in our case, we = are using all
>> > > recommendation like rack failure domain, replication n/w= ,etc, still
>> > > face client IO performance issues during one OSD down..<= br> >> > >
>> > > On Tue, Feb 19, 2019 at 10:56 PM David Turner <dra= konstein-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>> > > >
>> > > > With a RACK failure domain, you should be able to h= ave an entire rack powered down without noticing any major impact on the cl= ients.=C2=A0 I regularly take down OSDs and nodes for maintenance and upgra= des without seeing any problems with client IO.
>> > > >
>> > > > On Tue, Feb 12, 2019 at 5:01 AM M Ranga Swami Reddy= <swamireddy-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>> > > >>
>> > > >> Hello - I have a couple of questions on ceph cl= uster stability, even
>> > > >> we follow all recommendations as below:
>> > > >> - Having separate replication n/w and data n/w<= br> >> > > >> - RACK is the failure domain
>> > > >> - Using SSDs for journals (1:4ratio)
>> > > >>
>> > > >> Q1 - If one OSD down, cluster IO down drastical= ly and customer Apps impacted.
>> > > >> Q2 - what is stability ratio, like with above, = is ceph cluster
>> > > >> workable condition, if one osd down or one node= down,etc.
>> > > >>
>> > > >> Thanks
>> > > >> Swami
>> > > >> _______________________________________________=
>> > > >> ceph-users mailing list
>> > > >> ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
>> > > >> http://= lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>> > > _______________________________________________
>> > > ceph-users mailing list
>> > > ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
>> > > http://lists.cep= h.com/listinfo.cgi/ceph-users-ceph.com
--000000000000fcf8a805827a1a1d-- --===============1902477026652385573== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ ceph-users mailing list ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com --===============1902477026652385573==--