[CEPH] OSD daemons running with a large number of threads

[CEPH] OSD daemons running with a large number of threads

[ghislain.chevalier]

Hi,

Context:
Firefly 0.80.9
Ubuntu 14.04.1
Almost a production platform  in an openstack environment
176 OSD (SAS and SSD), 2 crushmap-oriented storage classes , 8 servers in 2 rooms, 3 monitors on openstack controllers
Usage: Rados Gateway for object service and RBD as back-end for Cinder and Glance

Issue:
We are currently running performances tests on this cluster before turning it to production.
We created cinder volumes (attached to Ceph Back End) on virtual machines and we use FIO to stress the cluster.
A very large number of threads are created per OSD daemon (about 1000).

I would like to identify the cause of this behavior and to be guided in tuning the environment (ceph, linux, network,...).

Best regards




_________________________________________________________________________________________________________________________

Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
France Telecom - Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci

This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorization.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, France Telecom - Orange shall not be liable if this message was modified, changed or falsified.
Thank you.

Re: [CEPH] OSD daemons running with a large number of threads

[Sage Weil]

On Tue, 17 Nov 2015, ghislain.chevalier@orange.com wrote:
> Hi,
> 
> Context:
> Firefly 0.80.9
> Ubuntu 14.04.1
> Almost a production platform  in an openstack environment
> 176 OSD (SAS and SSD), 2 crushmap-oriented storage classes , 8 servers in 2 rooms, 3 monitors on openstack controllers
> Usage: Rados Gateway for object service and RBD as back-end for Cinder and Glance
> 
> Issue:
> We are currently running performances tests on this cluster before turning it to production.
> We created cinder volumes (attached to Ceph Back End) on virtual machines and we use FIO to stress the cluster.
> A very large number of threads are created per OSD daemon (about 1000).

This is normal.  The init scripts set the max open files ulimit to a high 
value (usually 4194304 to avoid any possibility of hitting it) but you may 
need to set /proc/sys/kernel/pid_max to something big if your cluster is 
large.

sage

RE: [CEPH] OSD daemons running with a large number of threads

[ghislain.chevalier]

Hi,

Thanks for the advice

Brgds


-----Message d'origine-----
De : Sage Weil [mailto:sage@newdream.net] 
Envoyé : mardi 17 novembre 2015 14:41
À : CHEVALIER Ghislain IMT/OLPS
Cc : ceph-devel@vger.kernel.org
Objet : Re: [CEPH] OSD daemons running with a large number of threads

On Tue, 17 Nov 2015, ghislain.chevalier@orange.com wrote:
> Hi,
> 
> Context:
> Firefly 0.80.9
> Ubuntu 14.04.1
> Almost a production platform  in an openstack environment
> 176 OSD (SAS and SSD), 2 crushmap-oriented storage classes , 8 servers 
> in 2 rooms, 3 monitors on openstack controllers
> Usage: Rados Gateway for object service and RBD as back-end for Cinder 
> and Glance
> 
> Issue:
> We are currently running performances tests on this cluster before turning it to production.
> We created cinder volumes (attached to Ceph Back End) on virtual machines and we use FIO to stress the cluster.
> A very large number of threads are created per OSD daemon (about 1000).

This is normal.  The init scripts set the max open files ulimit to a high value (usually 4194304 to avoid any possibility of hitting it) but you may need to set /proc/sys/kernel/pid_max to something big if your cluster is large.

sage

_________________________________________________________________________________________________________________________

Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html