I see the same behavior with the threshold of around 20M objects for 4 nodes, 16 OSDs, 32TB, hdd-based cluster. The issue dates back to hammer.

Sent from my Windows 10 phone

From: Blair Bethwaite
Sent: Thursday, June 16, 2016 2:48 PM
To: Wade Holler
Cc: Ceph Development; ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
Subject: Re: [ceph-users] Dramatic performance drop at certain number ofobjects in pool

Hi Wade,

What IO are you seeing on the OSD devices when this happens (see e.g.

iostat), are there short periods of high read IOPS where (almost) no

writes occur? What does your memory usage look like (including slab)?

Cheers,

On 16 June 2016 at 22:14, Wade Holler <wade.holler-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:

> Hi All,

>

> I have a repeatable condition when the object count in a pool gets to

> 320-330 million the object write time dramatically and almost

> instantly increases as much as 10X, exhibited by fs_apply_latency

> going from 10ms to 100s of ms.

>

> Can someone point me in a direction / have an explanation ?

>

> I can add a new pool and it performs normally.

>

> Config is generally

> 3 Nodes 24 physical core each, 768GB Ram each, 16 OSD / node , all SSD

> with NVME for journals. Centos 7.2, XFS

>

> Jewell is the release; inserting objects with librados via some Python

> test code.

>

> Best Regards

> Wade

> --

> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in

> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org

> More majordomo info at  http://vger.kernel.org/majordomo-info.html

--

Cheers,

~Blairo

_______________________________________________

ceph-users mailing list

ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org

http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com