All of lore.kernel.org
 help / color / mirror / Atom feed
* [CEPH][Crush][Tunables] issue when updating tunables
@ 2015-11-10  8:32 ghislain.chevalier
  2015-11-10 10:23 ` Sage Weil
  0 siblings, 1 reply; 5+ messages in thread
From: ghislain.chevalier @ 2015-11-10  8:32 UTC (permalink / raw)
  To: ceph-devel

Hi all,

Context:
Firefly 0.80.9
Ubuntu 14.04.1
Almost a production platform  in an openstack environment
176 OSD (SAS and SSD), 2 crushmap-oriented storage classes , 8 servers in 2 rooms, 3 monitors on openstack controllers
Usage: Rados Gateway for object service and RBD as back-end for Cinder and Glance

The Ceph cluster was installed by Mirantis procedures (puppet/fuel/ceph-deploy):

I noticed that tunables were curiously set.
ceph  osd crush show-tunables ==>
{ "choose_local_tries": 0,
  "choose_local_fallback_tries": 0,
  "choose_total_tries": 50,
  "chooseleaf_descend_once": 1,
  "chooseleaf_vary_r": 1,
  "straw_calc_version": 1,
  "profile": "unknown",
  "optimal_tunables": 0,
  "legacy_tunables": 0,
  "require_feature_tunables": 1,
  "require_feature_tunables2": 1,
  "require_feature_tunables3": 1,
  "has_v2_rules": 0,
  "has_v3_rules": 0}

I tried to update them
ceph  osd crush tunables optimal ==>
adjusted tunables profile to optimal

But when checking
ceph  osd crush show-tunables ==>
{ "choose_local_tries": 0,
  "choose_local_fallback_tries": 0,
  "choose_total_tries": 50,
  "chooseleaf_descend_once": 1,
  "chooseleaf_vary_r": 1,
  "straw_calc_version": 1,
  "profile": "unknown",
  "optimal_tunables": 0,
  "legacy_tunables": 0,
  "require_feature_tunables": 1,
  "require_feature_tunables2": 1,
  "require_feature_tunables3": 1,
  "has_v2_rules": 0,
  "has_v3_rules": 0}

Nothing has changed.

I finally did
ceph osd crush set-tunable straw_calc_version 0

and
ceph  osd crush show-tunables ==>
{ "choose_local_tries": 0,
  "choose_local_fallback_tries": 0,
  "choose_total_tries": 50,
  "chooseleaf_descend_once": 1,
  "chooseleaf_vary_r": 1,
  "straw_calc_version": 0,
  "profile": "firefly",
  "optimal_tunables": 1,
  "legacy_tunables": 0,
  "require_feature_tunables": 1,
  "require_feature_tunables2": 1,
  "require_feature_tunables3": 1,
  "has_v2_rules": 0,
  "has_v3_rules": 0}

It's OK

My question:
Does the "ceph osd crush tunables <profile>" command change all the requested parameters in order to set the tunables to the right profile?

Brgds

_________________________________________________________________________________________________________________________

Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
France Telecom - Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci

This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorization.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, France Telecom - Orange shall not be liable if this message was modified, changed or falsified.
Thank you.


^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: [CEPH][Crush][Tunables] issue when updating tunables
  2015-11-10  8:32 [CEPH][Crush][Tunables] issue when updating tunables ghislain.chevalier
@ 2015-11-10 10:23 ` Sage Weil
  2015-11-12 14:11   ` ghislain.chevalier
  0 siblings, 1 reply; 5+ messages in thread
From: Sage Weil @ 2015-11-10 10:23 UTC (permalink / raw)
  To: ghislain.chevalier; +Cc: ceph-devel

On Tue, 10 Nov 2015, ghislain.chevalier@orange.com wrote:
> Hi all,
> 
> Context:
> Firefly 0.80.9
> Ubuntu 14.04.1
> Almost a production platform  in an openstack environment
> 176 OSD (SAS and SSD), 2 crushmap-oriented storage classes , 8 servers in 2 rooms, 3 monitors on openstack controllers
> Usage: Rados Gateway for object service and RBD as back-end for Cinder and Glance
> 
> The Ceph cluster was installed by Mirantis procedures (puppet/fuel/ceph-deploy):
> 
> I noticed that tunables were curiously set.
> ceph  osd crush show-tunables ==>
> { "choose_local_tries": 0,
>   "choose_local_fallback_tries": 0,
>   "choose_total_tries": 50,
>   "chooseleaf_descend_once": 1,
>   "chooseleaf_vary_r": 1,
>   "straw_calc_version": 1,
>   "profile": "unknown",
>   "optimal_tunables": 0,
>   "legacy_tunables": 0,
>   "require_feature_tunables": 1,
>   "require_feature_tunables2": 1,
>   "require_feature_tunables3": 1,
>   "has_v2_rules": 0,
>   "has_v3_rules": 0}
> 
> I tried to update them
> ceph  osd crush tunables optimal ==>
> adjusted tunables profile to optimal
> 
> But when checking
> ceph  osd crush show-tunables ==>
> { "choose_local_tries": 0,
>   "choose_local_fallback_tries": 0,
>   "choose_total_tries": 50,
>   "chooseleaf_descend_once": 1,
>   "chooseleaf_vary_r": 1,
>   "straw_calc_version": 1,
>   "profile": "unknown",
>   "optimal_tunables": 0,
>   "legacy_tunables": 0,
>   "require_feature_tunables": 1,
>   "require_feature_tunables2": 1,
>   "require_feature_tunables3": 1,
>   "has_v2_rules": 0,
>   "has_v3_rules": 0}
> 
> Nothing has changed.
> 
> I finally did
> ceph osd crush set-tunable straw_calc_version 0

You actually want straw_calc_version 1.  This is just confusing output 
from the 'firefly' tunable detection... the straw_calc_version does not 
have any client dependencies.

sage


> 
> and
> ceph  osd crush show-tunables ==>
> { "choose_local_tries": 0,
>   "choose_local_fallback_tries": 0,
>   "choose_total_tries": 50,
>   "chooseleaf_descend_once": 1,
>   "chooseleaf_vary_r": 1,
>   "straw_calc_version": 0,
>   "profile": "firefly",
>   "optimal_tunables": 1,
>   "legacy_tunables": 0,
>   "require_feature_tunables": 1,
>   "require_feature_tunables2": 1,
>   "require_feature_tunables3": 1,
>   "has_v2_rules": 0,
>   "has_v3_rules": 0}
> 
> It's OK
> 
> My question:
> Does the "ceph osd crush tunables <profile>" command change all the requested parameters in order to set the tunables to the right profile?
> 
> Brgds
> 
> _________________________________________________________________________________________________________________________
> 
> Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
> pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
> a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
> France Telecom - Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci
> 
> This message and its attachments may contain confidential or privileged information that may be protected by law;
> they should not be distributed, used or copied without authorization.
> If you have received this email in error, please notify the sender and delete this message and its attachments.
> As emails may be altered, France Telecom - Orange shall not be liable if this message was modified, changed or falsified.
> Thank you.
> 
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 

^ permalink raw reply	[flat|nested] 5+ messages in thread

* RE: [CEPH][Crush][Tunables] issue when updating tunables
  2015-11-10 10:23 ` Sage Weil
@ 2015-11-12 14:11   ` ghislain.chevalier
  2015-11-12 15:01     ` Sage Weil
  0 siblings, 1 reply; 5+ messages in thread
From: ghislain.chevalier @ 2015-11-12 14:11 UTC (permalink / raw)
  To: Sage Weil; +Cc: ceph-devel

Hi Sage,
Thanks for the reply

You said 
" You actually want straw_calc_version 1.  This is just confusing output from the 'firefly' tunable detection... the straw_calc_version does not have any client dependencies."

My objective is to have the most relevant tunables for a firefly platform.

I didn't understand if :
- it's better to have straw_calc_version set to 1 but  tunables_optimal will be automatically set to 0.
In other words are the following tunables OK ?
{ "choose_local_tries": 0,
>   "choose_local_fallback_tries": 0,
>   "choose_total_tries": 50,
>   "chooseleaf_descend_once": 1,
>   "chooseleaf_vary_r": 1,
>   "straw_calc_version": 0,
>   "profile": "firefly",
>   "optimal_tunables": 1,
>   "legacy_tunables": 0,
>   "require_feature_tunables": 1,
>   "require_feature_tunables2": 1,
>   "require_feature_tunables3": 1,
>   "has_v2_rules": 0,
>   "has_v3_rules": 0} 

- there's an issue with the tunables detection and update?

Best regards 

-----Message d'origine-----
De : Sage Weil [mailto:sage@newdream.net] 
Envoyé : mardi 10 novembre 2015 11:23
À : CHEVALIER Ghislain IMT/OLPS
Cc : ceph-devel@vger.kernel.org
Objet : Re: [CEPH][Crush][Tunables] issue when updating tunables

On Tue, 10 Nov 2015, ghislain.chevalier@orange.com wrote:
> Hi all,
> 
> Context:
> Firefly 0.80.9
> Ubuntu 14.04.1
> Almost a production platform  in an openstack environment
> 176 OSD (SAS and SSD), 2 crushmap-oriented storage classes , 8 servers 
> in 2 rooms, 3 monitors on openstack controllers
> Usage: Rados Gateway for object service and RBD as back-end for Cinder 
> and Glance
> 
> The Ceph cluster was installed by Mirantis procedures (puppet/fuel/ceph-deploy):
> 
> I noticed that tunables were curiously set.
> ceph  osd crush show-tunables ==>
> { "choose_local_tries": 0,
>   "choose_local_fallback_tries": 0,
>   "choose_total_tries": 50,
>   "chooseleaf_descend_once": 1,
>   "chooseleaf_vary_r": 1,
>   "straw_calc_version": 1,
>   "profile": "unknown",
>   "optimal_tunables": 0,
>   "legacy_tunables": 0,
>   "require_feature_tunables": 1,
>   "require_feature_tunables2": 1,
>   "require_feature_tunables3": 1,
>   "has_v2_rules": 0,
>   "has_v3_rules": 0}
> 
> I tried to update them
> ceph  osd crush tunables optimal ==>
> adjusted tunables profile to optimal
> 
> But when checking
> ceph  osd crush show-tunables ==>
> { "choose_local_tries": 0,
>   "choose_local_fallback_tries": 0,
>   "choose_total_tries": 50,
>   "chooseleaf_descend_once": 1,
>   "chooseleaf_vary_r": 1,
>   "straw_calc_version": 1,
>   "profile": "unknown",
>   "optimal_tunables": 0,
>   "legacy_tunables": 0,
>   "require_feature_tunables": 1,
>   "require_feature_tunables2": 1,
>   "require_feature_tunables3": 1,
>   "has_v2_rules": 0,
>   "has_v3_rules": 0}
> 
> Nothing has changed.
> 
> I finally did
> ceph osd crush set-tunable straw_calc_version 0

You actually want straw_calc_version 1.  This is just confusing output from the 'firefly' tunable detection... the straw_calc_version does not have any client dependencies.

sage


> 
> and
> ceph  osd crush show-tunables ==>
> { "choose_local_tries": 0,
>   "choose_local_fallback_tries": 0,
>   "choose_total_tries": 50,
>   "chooseleaf_descend_once": 1,
>   "chooseleaf_vary_r": 1,
>   "straw_calc_version": 0,
>   "profile": "firefly",
>   "optimal_tunables": 1,
>   "legacy_tunables": 0,
>   "require_feature_tunables": 1,
>   "require_feature_tunables2": 1,
>   "require_feature_tunables3": 1,
>   "has_v2_rules": 0,
>   "has_v3_rules": 0}
> 
> It's OK
> 
> My question:
> Does the "ceph osd crush tunables <profile>" command change all the requested parameters in order to set the tunables to the right profile?
> 
> Brgds
> 
> ______________________________________________________________________
> ___________________________________________________
> 
> Ce message et ses pieces jointes peuvent contenir des informations 
> confidentielles ou privilegiees et ne doivent donc pas etre diffuses, 
> exploites ou copies sans autorisation. Si vous avez recu ce message 
> par erreur, veuillez le signaler a l'expediteur et le detruire ainsi 
> que les pieces jointes. Les messages electroniques etant susceptibles 
> d'alteration, France Telecom - Orange decline toute responsabilite si 
> ce message a ete altere, deforme ou falsifie. Merci
> 
> This message and its attachments may contain confidential or 
> privileged information that may be protected by law; they should not be distributed, used or copied without authorization.
> If you have received this email in error, please notify the sender and delete this message and its attachments.
> As emails may be altered, France Telecom - Orange shall not be liable if this message was modified, changed or falsified.
> Thank you.
> 
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" 
> in the body of a message to majordomo@vger.kernel.org More majordomo 
> info at  http://vger.kernel.org/majordomo-info.html
> 
> 

_________________________________________________________________________________________________________________________

Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 5+ messages in thread

* RE: [CEPH][Crush][Tunables] issue when updating tunables
  2015-11-12 14:11   ` ghislain.chevalier
@ 2015-11-12 15:01     ` Sage Weil
  2015-11-12 17:40       ` ghislain.chevalier
  0 siblings, 1 reply; 5+ messages in thread
From: Sage Weil @ 2015-11-12 15:01 UTC (permalink / raw)
  To: ghislain.chevalier; +Cc: ceph-devel

[-- Attachment #1: Type: TEXT/PLAIN, Size: 7376 bytes --]

On Thu, 12 Nov 2015, ghislain.chevalier@orange.com wrote:
> Hi Sage,
> Thanks for the reply
> 
> You said 
> " You actually want straw_calc_version 1.  This is just confusing output from the 'firefly' tunable detection... the straw_calc_version does not have any client dependencies."
> 
> My objective is to have the most relevant tunables for a firefly platform.
> 
> I didn't understand if :
> - it's better to have straw_calc_version set to 1 but  tunables_optimal will be automatically set to 0.
> In other words are the following tunables OK ?
> { "choose_local_tries": 0,
> >   "choose_local_fallback_tries": 0,
> >   "choose_total_tries": 50,
> >   "chooseleaf_descend_once": 1,
> >   "chooseleaf_vary_r": 1,
> >   "straw_calc_version": 0,
> >   "profile": "firefly",
> >   "optimal_tunables": 1,
> >   "legacy_tunables": 0,
> >   "require_feature_tunables": 1,
> >   "require_feature_tunables2": 1,
> >   "require_feature_tunables3": 1,
> >   "has_v2_rules": 0,
> >   "has_v3_rules": 0} 

It is best to also manually set straw_calc_version = 1.  It won't say 
firefly, but it will still be compatible with firefly clients (that 
tunable only affects the mon behavior when updating the crush map).

> - there's an issue with the tunables detection and update?

Yes and no.  We want 'ceph osd crush tunables firefly' to set options 
supported by all firefly deployments, and the initial firefly releases did 
not have the straw_calc_version = 1 support.  In some cases switching it 
on can trigger some data movement the next time the crush map is adjusted, 
so we leave it off to be conservative.  And we want the profile to match 
exactly what setting the profile sets.  But it's confusing since it isn't 
1:1 with what clients support.  And if it is a fresh cluster you are 
better off with straw_calc_version = 1.  (Same goes for old clusters, if 
you can tolerate a bit of initial rebalancing.)

sage



> 
> Best regards 
> 
> -----Message d'origine-----
> De : Sage Weil [mailto:sage@newdream.net] 
> Envoyé : mardi 10 novembre 2015 11:23
> À : CHEVALIER Ghislain IMT/OLPS
> Cc : ceph-devel@vger.kernel.org
> Objet : Re: [CEPH][Crush][Tunables] issue when updating tunables
> 
> On Tue, 10 Nov 2015, ghislain.chevalier@orange.com wrote:
> > Hi all,
> > 
> > Context:
> > Firefly 0.80.9
> > Ubuntu 14.04.1
> > Almost a production platform  in an openstack environment
> > 176 OSD (SAS and SSD), 2 crushmap-oriented storage classes , 8 servers 
> > in 2 rooms, 3 monitors on openstack controllers
> > Usage: Rados Gateway for object service and RBD as back-end for Cinder 
> > and Glance
> > 
> > The Ceph cluster was installed by Mirantis procedures (puppet/fuel/ceph-deploy):
> > 
> > I noticed that tunables were curiously set.
> > ceph  osd crush show-tunables ==>
> > { "choose_local_tries": 0,
> >   "choose_local_fallback_tries": 0,
> >   "choose_total_tries": 50,
> >   "chooseleaf_descend_once": 1,
> >   "chooseleaf_vary_r": 1,
> >   "straw_calc_version": 1,
> >   "profile": "unknown",
> >   "optimal_tunables": 0,
> >   "legacy_tunables": 0,
> >   "require_feature_tunables": 1,
> >   "require_feature_tunables2": 1,
> >   "require_feature_tunables3": 1,
> >   "has_v2_rules": 0,
> >   "has_v3_rules": 0}
> > 
> > I tried to update them
> > ceph  osd crush tunables optimal ==>
> > adjusted tunables profile to optimal
> > 
> > But when checking
> > ceph  osd crush show-tunables ==>
> > { "choose_local_tries": 0,
> >   "choose_local_fallback_tries": 0,
> >   "choose_total_tries": 50,
> >   "chooseleaf_descend_once": 1,
> >   "chooseleaf_vary_r": 1,
> >   "straw_calc_version": 1,
> >   "profile": "unknown",
> >   "optimal_tunables": 0,
> >   "legacy_tunables": 0,
> >   "require_feature_tunables": 1,
> >   "require_feature_tunables2": 1,
> >   "require_feature_tunables3": 1,
> >   "has_v2_rules": 0,
> >   "has_v3_rules": 0}
> > 
> > Nothing has changed.
> > 
> > I finally did
> > ceph osd crush set-tunable straw_calc_version 0
> 
> You actually want straw_calc_version 1.  This is just confusing output from the 'firefly' tunable detection... the straw_calc_version does not have any client dependencies.
> 
> sage
> 
> 
> > 
> > and
> > ceph  osd crush show-tunables ==>
> > { "choose_local_tries": 0,
> >   "choose_local_fallback_tries": 0,
> >   "choose_total_tries": 50,
> >   "chooseleaf_descend_once": 1,
> >   "chooseleaf_vary_r": 1,
> >   "straw_calc_version": 0,
> >   "profile": "firefly",
> >   "optimal_tunables": 1,
> >   "legacy_tunables": 0,
> >   "require_feature_tunables": 1,
> >   "require_feature_tunables2": 1,
> >   "require_feature_tunables3": 1,
> >   "has_v2_rules": 0,
> >   "has_v3_rules": 0}
> > 
> > It's OK
> > 
> > My question:
> > Does the "ceph osd crush tunables <profile>" command change all the requested parameters in order to set the tunables to the right profile?
> > 
> > Brgds
> > 
> > ______________________________________________________________________
> > ___________________________________________________
> > 
> > Ce message et ses pieces jointes peuvent contenir des informations 
> > confidentielles ou privilegiees et ne doivent donc pas etre diffuses, 
> > exploites ou copies sans autorisation. Si vous avez recu ce message 
> > par erreur, veuillez le signaler a l'expediteur et le detruire ainsi 
> > que les pieces jointes. Les messages electroniques etant susceptibles 
> > d'alteration, France Telecom - Orange decline toute responsabilite si 
> > ce message a ete altere, deforme ou falsifie. Merci
> > 
> > This message and its attachments may contain confidential or 
> > privileged information that may be protected by law; they should not be distributed, used or copied without authorization.
> > If you have received this email in error, please notify the sender and delete this message and its attachments.
> > As emails may be altered, France Telecom - Orange shall not be liable if this message was modified, changed or falsified.
> > Thank you.
> > 
> > --
> > To unsubscribe from this list: send the line "unsubscribe ceph-devel" 
> > in the body of a message to majordomo@vger.kernel.org More majordomo 
> > info at  http://vger.kernel.org/majordomo-info.html
> > 
> > 
> 
> _________________________________________________________________________________________________________________________
> 
> Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
> pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
> a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
> Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.
> 
> This message and its attachments may contain confidential or privileged information that may be protected by law;
> they should not be distributed, used or copied without authorisation.
> If you have received this email in error, please notify the sender and delete this message and its attachments.
> As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
> Thank you.
> 
> 

^ permalink raw reply	[flat|nested] 5+ messages in thread

* RE: [CEPH][Crush][Tunables] issue when updating tunables
  2015-11-12 15:01     ` Sage Weil
@ 2015-11-12 17:40       ` ghislain.chevalier
  0 siblings, 0 replies; 5+ messages in thread
From: ghislain.chevalier @ 2015-11-12 17:40 UTC (permalink / raw)
  To: Sage Weil; +Cc: ceph-devel

Thx Sage
It's clear now

Best regards

-----Message d'origine-----
De : Sage Weil [mailto:sage@newdream.net] 
Envoyé : jeudi 12 novembre 2015 16:01
À : CHEVALIER Ghislain IMT/OLPS
Cc : ceph-devel@vger.kernel.org
Objet : RE: [CEPH][Crush][Tunables] issue when updating tunables

On Thu, 12 Nov 2015, ghislain.chevalier@orange.com wrote:
> Hi Sage,
> Thanks for the reply
> 
> You said
> " You actually want straw_calc_version 1.  This is just confusing output from the 'firefly' tunable detection... the straw_calc_version does not have any client dependencies."
> 
> My objective is to have the most relevant tunables for a firefly platform.
> 
> I didn't understand if :
> - it's better to have straw_calc_version set to 1 but  tunables_optimal will be automatically set to 0.
> In other words are the following tunables OK ?
> { "choose_local_tries": 0,
> >   "choose_local_fallback_tries": 0,
> >   "choose_total_tries": 50,
> >   "chooseleaf_descend_once": 1,
> >   "chooseleaf_vary_r": 1,
> >   "straw_calc_version": 0,
> >   "profile": "firefly",
> >   "optimal_tunables": 1,
> >   "legacy_tunables": 0,
> >   "require_feature_tunables": 1,
> >   "require_feature_tunables2": 1,
> >   "require_feature_tunables3": 1,
> >   "has_v2_rules": 0,
> >   "has_v3_rules": 0}

It is best to also manually set straw_calc_version = 1.  It won't say firefly, but it will still be compatible with firefly clients (that tunable only affects the mon behavior when updating the crush map).

> - there's an issue with the tunables detection and update?

Yes and no.  We want 'ceph osd crush tunables firefly' to set options supported by all firefly deployments, and the initial firefly releases did not have the straw_calc_version = 1 support.  In some cases switching it on can trigger some data movement the next time the crush map is adjusted, so we leave it off to be conservative.  And we want the profile to match exactly what setting the profile sets.  But it's confusing since it isn't
1:1 with what clients support.  And if it is a fresh cluster you are better off with straw_calc_version = 1.  (Same goes for old clusters, if you can tolerate a bit of initial rebalancing.)

sage



> 
> Best regards
> 
> -----Message d'origine-----
> De : Sage Weil [mailto:sage@newdream.net] Envoyé : mardi 10 novembre 
> 2015 11:23 À : CHEVALIER Ghislain IMT/OLPS Cc : 
> ceph-devel@vger.kernel.org Objet : Re: [CEPH][Crush][Tunables] issue 
> when updating tunables
> 
> On Tue, 10 Nov 2015, ghislain.chevalier@orange.com wrote:
> > Hi all,
> > 
> > Context:
> > Firefly 0.80.9
> > Ubuntu 14.04.1
> > Almost a production platform  in an openstack environment
> > 176 OSD (SAS and SSD), 2 crushmap-oriented storage classes , 8 
> > servers in 2 rooms, 3 monitors on openstack controllers
> > Usage: Rados Gateway for object service and RBD as back-end for 
> > Cinder and Glance
> > 
> > The Ceph cluster was installed by Mirantis procedures (puppet/fuel/ceph-deploy):
> > 
> > I noticed that tunables were curiously set.
> > ceph  osd crush show-tunables ==>
> > { "choose_local_tries": 0,
> >   "choose_local_fallback_tries": 0,
> >   "choose_total_tries": 50,
> >   "chooseleaf_descend_once": 1,
> >   "chooseleaf_vary_r": 1,
> >   "straw_calc_version": 1,
> >   "profile": "unknown",
> >   "optimal_tunables": 0,
> >   "legacy_tunables": 0,
> >   "require_feature_tunables": 1,
> >   "require_feature_tunables2": 1,
> >   "require_feature_tunables3": 1,
> >   "has_v2_rules": 0,
> >   "has_v3_rules": 0}
> > 
> > I tried to update them
> > ceph  osd crush tunables optimal ==> adjusted tunables profile to 
> > optimal
> > 
> > But when checking
> > ceph  osd crush show-tunables ==>
> > { "choose_local_tries": 0,
> >   "choose_local_fallback_tries": 0,
> >   "choose_total_tries": 50,
> >   "chooseleaf_descend_once": 1,
> >   "chooseleaf_vary_r": 1,
> >   "straw_calc_version": 1,
> >   "profile": "unknown",
> >   "optimal_tunables": 0,
> >   "legacy_tunables": 0,
> >   "require_feature_tunables": 1,
> >   "require_feature_tunables2": 1,
> >   "require_feature_tunables3": 1,
> >   "has_v2_rules": 0,
> >   "has_v3_rules": 0}
> > 
> > Nothing has changed.
> > 
> > I finally did
> > ceph osd crush set-tunable straw_calc_version 0
> 
> You actually want straw_calc_version 1.  This is just confusing output from the 'firefly' tunable detection... the straw_calc_version does not have any client dependencies.
> 
> sage
> 
> 
> > 
> > and
> > ceph  osd crush show-tunables ==>
> > { "choose_local_tries": 0,
> >   "choose_local_fallback_tries": 0,
> >   "choose_total_tries": 50,
> >   "chooseleaf_descend_once": 1,
> >   "chooseleaf_vary_r": 1,
> >   "straw_calc_version": 0,
> >   "profile": "firefly",
> >   "optimal_tunables": 1,
> >   "legacy_tunables": 0,
> >   "require_feature_tunables": 1,
> >   "require_feature_tunables2": 1,
> >   "require_feature_tunables3": 1,
> >   "has_v2_rules": 0,
> >   "has_v3_rules": 0}
> > 
> > It's OK
> > 
> > My question:
> > Does the "ceph osd crush tunables <profile>" command change all the requested parameters in order to set the tunables to the right profile?
> > 
> > Brgds
> > 
> > ____________________________________________________________________
> > __ ___________________________________________________
> > 
> > Ce message et ses pieces jointes peuvent contenir des informations 
> > confidentielles ou privilegiees et ne doivent donc pas etre 
> > diffuses, exploites ou copies sans autorisation. Si vous avez recu 
> > ce message par erreur, veuillez le signaler a l'expediteur et le 
> > detruire ainsi que les pieces jointes. Les messages electroniques 
> > etant susceptibles d'alteration, France Telecom - Orange decline 
> > toute responsabilite si ce message a ete altere, deforme ou 
> > falsifie. Merci
> > 
> > This message and its attachments may contain confidential or 
> > privileged information that may be protected by law; they should not be distributed, used or copied without authorization.
> > If you have received this email in error, please notify the sender and delete this message and its attachments.
> > As emails may be altered, France Telecom - Orange shall not be liable if this message was modified, changed or falsified.
> > Thank you.
> > 
> > --
> > To unsubscribe from this list: send the line "unsubscribe ceph-devel" 
> > in the body of a message to majordomo@vger.kernel.org More majordomo 
> > info at  http://vger.kernel.org/majordomo-info.html
> > 
> > 
> 
> ______________________________________________________________________
> ___________________________________________________
> 
> Ce message et ses pieces jointes peuvent contenir des informations 
> confidentielles ou privilegiees et ne doivent donc pas etre diffuses, 
> exploites ou copies sans autorisation. Si vous avez recu ce message 
> par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.
> 
> This message and its attachments may contain confidential or 
> privileged information that may be protected by law; they should not be distributed, used or copied without authorisation.
> If you have received this email in error, please notify the sender and delete this message and its attachments.
> As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
> Thank you.
> 
> 

_________________________________________________________________________________________________________________________

Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 5+ messages in thread

end of thread, other threads:[~2015-11-12 17:47 UTC | newest]

Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2015-11-10  8:32 [CEPH][Crush][Tunables] issue when updating tunables ghislain.chevalier
2015-11-10 10:23 ` Sage Weil
2015-11-12 14:11   ` ghislain.chevalier
2015-11-12 15:01     ` Sage Weil
2015-11-12 17:40       ` ghislain.chevalier

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.