ceph mgr balancer bad distribution

ceph mgr balancer bad distribution

[Stefan Priebe - Profihost AG]

Hello,

with jewel we always used the python crush optimizer which gave us a
pretty good distribution fo the used space.

Since luminous we're using the included ceph mgr balancer but the
distribution is far from perfect and much worse than the old method.

Is there any way to tune the mgr balancer?

Currently after a balance we still have:
75% to 92% disk usage which is pretty unfair

Greets,
Stefan

Re: ceph mgr balancer bad distribution

[Dan van der Ster]

Hi Stefan,

Which balancer mode are you using? crush-compat scores using a mix of
nobjects, npgs, and size. It's doing pretty well over here as long as
you have a relatively small number of empty PGs.
I believe that upmap uses nPGs only, and I haven't tested it enough
yet to know if it actually improves things.

Also, did you only run one iteration of the balancer? It only moves up
to 5% of objects each iteration, so it can take several to fully
balance things.

-- dan


On Wed, Feb 28, 2018 at 1:47 PM, Stefan Priebe - Profihost AG
<s.priebe-2Lf/h1ldwEHR5kwTpVNS9A@public.gmane.org> wrote:
> Hello,
>
> with jewel we always used the python crush optimizer which gave us a
> pretty good distribution fo the used space.
>
> Since luminous we're using the included ceph mgr balancer but the
> distribution is far from perfect and much worse than the old method.
>
> Is there any way to tune the mgr balancer?
>
> Currently after a balance we still have:
> 75% to 92% disk usage which is pretty unfair
>
> Greets,
> Stefan
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: ceph mgr balancer bad distribution

[Stefan Priebe - Profihost AG]

Am 28.02.2018 um 13:58 schrieb Dan van der Ster:
> Hi Stefan,
> 
> Which balancer mode are you using? crush-compat scores using a mix of
> nobjects, npgs, and size. It's doing pretty well over here as long as
> you have a relatively small number of empty PGs.
>
> I believe that upmap uses nPGs only, and I haven't tested it enough
> yet to know if it actually improves things.
> 
> Also, did you only run one iteration of the balancer? It only moves up
> to 5% of objects each iteration, so it can take several to fully
> balance things.

crush-compat mode

Yes only one iteration but i set max_misplaced to 20%:
    "mgr/balancer/max_misplaced": "20.00",

> 
> -- dan
> 
> 
> On Wed, Feb 28, 2018 at 1:47 PM, Stefan Priebe - Profihost AG
> <s.priebe-2Lf/h1ldwEHR5kwTpVNS9A@public.gmane.org> wrote:
>> Hello,
>>
>> with jewel we always used the python crush optimizer which gave us a
>> pretty good distribution fo the used space.
>>
>> Since luminous we're using the included ceph mgr balancer but the
>> distribution is far from perfect and much worse than the old method.
>>
>> Is there any way to tune the mgr balancer?
>>
>> Currently after a balance we still have:
>> 75% to 92% disk usage which is pretty unfair
>>
>> Greets,
>> Stefan
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: ceph mgr balancer bad distribution

[Stefan Priebe - Profihost AG]

Does anybody have some more input?

I keeped the balancer active for 24h now and it is rebalancing 1-3%
every 30 minutes but the distribution is still bad.

It seems to balance from left to right and than back from right to left...

Greets,
Stefan

Am 28.02.2018 um 13:47 schrieb Stefan Priebe - Profihost AG:
> Hello,
> 
> with jewel we always used the python crush optimizer which gave us a
> pretty good distribution fo the used space.
> 
> Since luminous we're using the included ceph mgr balancer but the
> distribution is far from perfect and much worse than the old method.
> 
> Is there any way to tune the mgr balancer?
> 
> Currently after a balance we still have:
> 75% to 92% disk usage which is pretty unfair
> 
> Greets,
> Stefan
> 

Re: ceph mgr balancer bad distribution

[Dan van der Ster]

Is the score improving?

    ceph balancer eval

It should be decreasing over time as the variances drop toward zero.

You mentioned a crush optimize code at the beginning... how did that
leave your cluster? The mgr balancer assumes that the crush weight of
each OSD is equal to its size in TB.
Do you have any osd reweights? crush-compat will gradually adjust
those back to 1.0.

Cheers, Dan



On Thu, Mar 1, 2018 at 8:27 AM, Stefan Priebe - Profihost AG
<s.priebe-2Lf/h1ldwEHR5kwTpVNS9A@public.gmane.org> wrote:
> Does anybody have some more input?
>
> I keeped the balancer active for 24h now and it is rebalancing 1-3%
> every 30 minutes but the distribution is still bad.
>
> It seems to balance from left to right and than back from right to left...
>
> Greets,
> Stefan
>
> Am 28.02.2018 um 13:47 schrieb Stefan Priebe - Profihost AG:
>> Hello,
>>
>> with jewel we always used the python crush optimizer which gave us a
>> pretty good distribution fo the used space.
>>
>> Since luminous we're using the included ceph mgr balancer but the
>> distribution is far from perfect and much worse than the old method.
>>
>> Is there any way to tune the mgr balancer?
>>
>> Currently after a balance we still have:
>> 75% to 92% disk usage which is pretty unfair
>>
>> Greets,
>> Stefan
>>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: ceph mgr balancer bad distribution

[Stefan Priebe - Profihost AG]

Hi,
Am 01.03.2018 um 09:03 schrieb Dan van der Ster:
> Is the score improving?
> 
>     ceph balancer eval
> 
> It should be decreasing over time as the variances drop toward zero.
> 
> You mentioned a crush optimize code at the beginning... how did that
> leave your cluster? The mgr balancer assumes that the crush weight of
> each OSD is equal to its size in TB.
> Do you have any osd reweights? crush-compat will gradually adjust
> those back to 1.0.

I reweighted them all back to their correct weight.

Now the mgr balancer module says:
mgr[balancer] Failed to find further optimization, score 0.010646

But as you can see it's heavily imbalanced:


Example:
49   ssd 0.84000  1.00000   864G   546G   317G 63.26 1.13  49

vs:

48   ssd 0.84000  1.00000   864G   397G   467G 45.96 0.82  49

45% usage vs. 63%

Greets,
Stefan

> 
> Cheers, Dan
> 
> 
> 
> On Thu, Mar 1, 2018 at 8:27 AM, Stefan Priebe - Profihost AG
> <s.priebe-2Lf/h1ldwEHR5kwTpVNS9A@public.gmane.org> wrote:
>> Does anybody have some more input?
>>
>> I keeped the balancer active for 24h now and it is rebalancing 1-3%
>> every 30 minutes but the distribution is still bad.
>>
>> It seems to balance from left to right and than back from right to left...
>>
>> Greets,
>> Stefan
>>
>> Am 28.02.2018 um 13:47 schrieb Stefan Priebe - Profihost AG:
>>> Hello,
>>>
>>> with jewel we always used the python crush optimizer which gave us a
>>> pretty good distribution fo the used space.
>>>
>>> Since luminous we're using the included ceph mgr balancer but the
>>> distribution is far from perfect and much worse than the old method.
>>>
>>> Is there any way to tune the mgr balancer?
>>>
>>> Currently after a balance we still have:
>>> 75% to 92% disk usage which is pretty unfair
>>>
>>> Greets,
>>> Stefan
>>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: ceph mgr balancer bad distribution

[Dan van der Ster]

On Thu, Mar 1, 2018 at 9:31 AM, Stefan Priebe - Profihost AG
<s.priebe-2Lf/h1ldwEHR5kwTpVNS9A@public.gmane.org> wrote:
> Hi,
> Am 01.03.2018 um 09:03 schrieb Dan van der Ster:
>> Is the score improving?
>>
>>     ceph balancer eval
>>
>> It should be decreasing over time as the variances drop toward zero.
>>
>> You mentioned a crush optimize code at the beginning... how did that
>> leave your cluster? The mgr balancer assumes that the crush weight of
>> each OSD is equal to its size in TB.
>> Do you have any osd reweights? crush-compat will gradually adjust
>> those back to 1.0.
>
> I reweighted them all back to their correct weight.
>
> Now the mgr balancer module says:
> mgr[balancer] Failed to find further optimization, score 0.010646
>
> But as you can see it's heavily imbalanced:
>
>
> Example:
> 49   ssd 0.84000  1.00000   864G   546G   317G 63.26 1.13  49
>
> vs:
>
> 48   ssd 0.84000  1.00000   864G   397G   467G 45.96 0.82  49
>
> 45% usage vs. 63%

Ahh... but look, the num PGs are perfectly balanced, which implies
that you have a relatively large number of empty PGs.

But regardless, this is annoying and I expect lots of operators to get
this result. (I've also observed that the num PGs is gets balanced
perfectly at the expense of the other score metrics.)

I was thinking of a patch around here [1] that lets operators add a
score weight on pgs, objects, bytes so we can balance how we like.

Spandan: you were the last to look at this function. Do you think it
can be improved as I suggested?

Cheers, Dan

[1] https://github.com/ceph/ceph/blob/luminous/src/pybind/mgr/balancer/module.py#L558

Re: ceph mgr balancer bad distribution

[Stefan Priebe - Profihost AG]

Hi,

Am 01.03.2018 um 09:42 schrieb Dan van der Ster:
> On Thu, Mar 1, 2018 at 9:31 AM, Stefan Priebe - Profihost AG
> <s.priebe-2Lf/h1ldwEHR5kwTpVNS9A@public.gmane.org> wrote:
>> Hi,
>> Am 01.03.2018 um 09:03 schrieb Dan van der Ster:
>>> Is the score improving?
>>>
>>>     ceph balancer eval
>>>
>>> It should be decreasing over time as the variances drop toward zero.
>>>
>>> You mentioned a crush optimize code at the beginning... how did that
>>> leave your cluster? The mgr balancer assumes that the crush weight of
>>> each OSD is equal to its size in TB.
>>> Do you have any osd reweights? crush-compat will gradually adjust
>>> those back to 1.0.
>>
>> I reweighted them all back to their correct weight.
>>
>> Now the mgr balancer module says:
>> mgr[balancer] Failed to find further optimization, score 0.010646
>>
>> But as you can see it's heavily imbalanced:
>>
>>
>> Example:
>> 49   ssd 0.84000  1.00000   864G   546G   317G 63.26 1.13  49
>>
>> vs:
>>
>> 48   ssd 0.84000  1.00000   864G   397G   467G 45.96 0.82  49
>>
>> 45% usage vs. 63%
> 
> Ahh... but look, the num PGs are perfectly balanced, which implies
> that you have a relatively large number of empty PGs.
> 
> But regardless, this is annoying and I expect lots of operators to get
> this result. (I've also observed that the num PGs is gets balanced
> perfectly at the expense of the other score metrics.)
> 
> I was thinking of a patch around here [1] that lets operators add a
> score weight on pgs, objects, bytes so we can balance how we like.
> 
> Spandan: you were the last to look at this function. Do you think it
> can be improved as I suggested?

Yes the PGs are perfectly distributed - but i think most of the people
would like to have a dsitribution by bytes and not pgs.

Is this possible? I mean in the code there is already a dict for pgs,
objects and bytes - but i don't know how to change the logic. Just
remove the pgs and objects from the dict?

> Cheers, Dan
> 
> [1] https://github.com/ceph/ceph/blob/luminous/src/pybind/mgr/balancer/module.py#L558
> 

Re: ceph mgr balancer bad distribution

[Dan van der Ster]

On Thu, Mar 1, 2018 at 9:52 AM, Stefan Priebe - Profihost AG
<s.priebe-2Lf/h1ldwEHR5kwTpVNS9A@public.gmane.org> wrote:
> Hi,
>
> Am 01.03.2018 um 09:42 schrieb Dan van der Ster:
>> On Thu, Mar 1, 2018 at 9:31 AM, Stefan Priebe - Profihost AG
>> <s.priebe-2Lf/h1ldwEHR5kwTpVNS9A@public.gmane.org> wrote:
>>> Hi,
>>> Am 01.03.2018 um 09:03 schrieb Dan van der Ster:
>>>> Is the score improving?
>>>>
>>>>     ceph balancer eval
>>>>
>>>> It should be decreasing over time as the variances drop toward zero.
>>>>
>>>> You mentioned a crush optimize code at the beginning... how did that
>>>> leave your cluster? The mgr balancer assumes that the crush weight of
>>>> each OSD is equal to its size in TB.
>>>> Do you have any osd reweights? crush-compat will gradually adjust
>>>> those back to 1.0.
>>>
>>> I reweighted them all back to their correct weight.
>>>
>>> Now the mgr balancer module says:
>>> mgr[balancer] Failed to find further optimization, score 0.010646
>>>
>>> But as you can see it's heavily imbalanced:
>>>
>>>
>>> Example:
>>> 49   ssd 0.84000  1.00000   864G   546G   317G 63.26 1.13  49
>>>
>>> vs:
>>>
>>> 48   ssd 0.84000  1.00000   864G   397G   467G 45.96 0.82  49
>>>
>>> 45% usage vs. 63%
>>
>> Ahh... but look, the num PGs are perfectly balanced, which implies
>> that you have a relatively large number of empty PGs.
>>
>> But regardless, this is annoying and I expect lots of operators to get
>> this result. (I've also observed that the num PGs is gets balanced
>> perfectly at the expense of the other score metrics.)
>>
>> I was thinking of a patch around here [1] that lets operators add a
>> score weight on pgs, objects, bytes so we can balance how we like.
>>
>> Spandan: you were the last to look at this function. Do you think it
>> can be improved as I suggested?
>
> Yes the PGs are perfectly distributed - but i think most of the people
> would like to have a dsitribution by bytes and not pgs.
>
> Is this possible? I mean in the code there is already a dict for pgs,
> objects and bytes - but i don't know how to change the logic. Just
> remove the pgs and objects from the dict?

It's worth a try to remove the pgs and objects from this dict:

https://github.com/ceph/ceph/blob/luminous/src/pybind/mgr/balancer/module.py#L552

You can update that directly in the python code on your mgr's. Turn
the ceph balancer off then failover to the next mgr so it reloads the
module. Then:

ceph balancer eval
ceph balancer optimize myplan
ceph balancer eval myplan

Does it move in the right direction?

-- dan

Re: ceph mgr balancer bad distribution

[Stefan Priebe - Profihost AG]

Am 01.03.2018 um 09:58 schrieb Dan van der Ster:
> On Thu, Mar 1, 2018 at 9:52 AM, Stefan Priebe - Profihost AG
> <s.priebe-2Lf/h1ldwEHR5kwTpVNS9A@public.gmane.org> wrote:
>> Hi,
>>
>> Am 01.03.2018 um 09:42 schrieb Dan van der Ster:
>>> On Thu, Mar 1, 2018 at 9:31 AM, Stefan Priebe - Profihost AG
>>> <s.priebe-2Lf/h1ldwEHR5kwTpVNS9A@public.gmane.org> wrote:
>>>> Hi,
>>>> Am 01.03.2018 um 09:03 schrieb Dan van der Ster:
>>>>> Is the score improving?
>>>>>
>>>>>     ceph balancer eval
>>>>>
>>>>> It should be decreasing over time as the variances drop toward zero.
>>>>>
>>>>> You mentioned a crush optimize code at the beginning... how did that
>>>>> leave your cluster? The mgr balancer assumes that the crush weight of
>>>>> each OSD is equal to its size in TB.
>>>>> Do you have any osd reweights? crush-compat will gradually adjust
>>>>> those back to 1.0.
>>>>
>>>> I reweighted them all back to their correct weight.
>>>>
>>>> Now the mgr balancer module says:
>>>> mgr[balancer] Failed to find further optimization, score 0.010646
>>>>
>>>> But as you can see it's heavily imbalanced:
>>>>
>>>>
>>>> Example:
>>>> 49   ssd 0.84000  1.00000   864G   546G   317G 63.26 1.13  49
>>>>
>>>> vs:
>>>>
>>>> 48   ssd 0.84000  1.00000   864G   397G   467G 45.96 0.82  49
>>>>
>>>> 45% usage vs. 63%
>>>
>>> Ahh... but look, the num PGs are perfectly balanced, which implies
>>> that you have a relatively large number of empty PGs.
>>>
>>> But regardless, this is annoying and I expect lots of operators to get
>>> this result. (I've also observed that the num PGs is gets balanced
>>> perfectly at the expense of the other score metrics.)
>>>
>>> I was thinking of a patch around here [1] that lets operators add a
>>> score weight on pgs, objects, bytes so we can balance how we like.
>>>
>>> Spandan: you were the last to look at this function. Do you think it
>>> can be improved as I suggested?
>>
>> Yes the PGs are perfectly distributed - but i think most of the people
>> would like to have a dsitribution by bytes and not pgs.
>>
>> Is this possible? I mean in the code there is already a dict for pgs,
>> objects and bytes - but i don't know how to change the logic. Just
>> remove the pgs and objects from the dict?
> 
> It's worth a try to remove the pgs and objects from this dict:
> https://github.com/ceph/ceph/blob/luminous/src/pybind/mgr/balancer/module.py#L552

Do i have to change this 3 to 1 cause we have only one item in the dict?
I'm not sure where the 3 comes from.
        pe.score /= 3 * len(roots)


> You can update that directly in the python code on your mgr's. Turn
> the ceph balancer off then failover to the next mgr so it reloads the
> module. Then:
> 
> ceph balancer eval
> ceph balancer optimize myplan
> ceph balancer eval myplan
> 
> Does it move in the right direction?
> 
> -- dan
> 

Re: ceph mgr balancer bad distribution

[Dan van der Ster]

On Thu, Mar 1, 2018 at 10:24 AM, Stefan Priebe - Profihost AG
<s.priebe-2Lf/h1ldwEHR5kwTpVNS9A@public.gmane.org> wrote:
>
> Am 01.03.2018 um 09:58 schrieb Dan van der Ster:
>> On Thu, Mar 1, 2018 at 9:52 AM, Stefan Priebe - Profihost AG
>> <s.priebe-2Lf/h1ldwEHR5kwTpVNS9A@public.gmane.org> wrote:
>>> Hi,
>>>
>>> Am 01.03.2018 um 09:42 schrieb Dan van der Ster:
>>>> On Thu, Mar 1, 2018 at 9:31 AM, Stefan Priebe - Profihost AG
>>>> <s.priebe-2Lf/h1ldwEHR5kwTpVNS9A@public.gmane.org> wrote:
>>>>> Hi,
>>>>> Am 01.03.2018 um 09:03 schrieb Dan van der Ster:
>>>>>> Is the score improving?
>>>>>>
>>>>>>     ceph balancer eval
>>>>>>
>>>>>> It should be decreasing over time as the variances drop toward zero.
>>>>>>
>>>>>> You mentioned a crush optimize code at the beginning... how did that
>>>>>> leave your cluster? The mgr balancer assumes that the crush weight of
>>>>>> each OSD is equal to its size in TB.
>>>>>> Do you have any osd reweights? crush-compat will gradually adjust
>>>>>> those back to 1.0.
>>>>>
>>>>> I reweighted them all back to their correct weight.
>>>>>
>>>>> Now the mgr balancer module says:
>>>>> mgr[balancer] Failed to find further optimization, score 0.010646
>>>>>
>>>>> But as you can see it's heavily imbalanced:
>>>>>
>>>>>
>>>>> Example:
>>>>> 49   ssd 0.84000  1.00000   864G   546G   317G 63.26 1.13  49
>>>>>
>>>>> vs:
>>>>>
>>>>> 48   ssd 0.84000  1.00000   864G   397G   467G 45.96 0.82  49
>>>>>
>>>>> 45% usage vs. 63%
>>>>
>>>> Ahh... but look, the num PGs are perfectly balanced, which implies
>>>> that you have a relatively large number of empty PGs.
>>>>
>>>> But regardless, this is annoying and I expect lots of operators to get
>>>> this result. (I've also observed that the num PGs is gets balanced
>>>> perfectly at the expense of the other score metrics.)
>>>>
>>>> I was thinking of a patch around here [1] that lets operators add a
>>>> score weight on pgs, objects, bytes so we can balance how we like.
>>>>
>>>> Spandan: you were the last to look at this function. Do you think it
>>>> can be improved as I suggested?
>>>
>>> Yes the PGs are perfectly distributed - but i think most of the people
>>> would like to have a dsitribution by bytes and not pgs.
>>>
>>> Is this possible? I mean in the code there is already a dict for pgs,
>>> objects and bytes - but i don't know how to change the logic. Just
>>> remove the pgs and objects from the dict?
>>
>> It's worth a try to remove the pgs and objects from this dict:
>> https://github.com/ceph/ceph/blob/luminous/src/pybind/mgr/balancer/module.py#L552
>
> Do i have to change this 3 to 1 cause we have only one item in the dict?
> I'm not sure where the 3 comes from.
>         pe.score /= 3 * len(roots)
>

I'm pretty sure that 3 is just for our 3 metrics. Indeed you can
change that to 1.

I'm trying this on our test cluster here too. The last few lines of
output from `ceph balancer eval-verbose` will confirm that the score
is based only on bytes.

But I'm not sure this is going to work -- indeed the score here went
from ~0.02 to 0.08, but the do_crush_compat doesn't manage to find a
better score.

-- Dan

Re: ceph mgr balancer bad distribution

[Dan van der Ster]

On Thu, Mar 1, 2018 at 10:38 AM, Dan van der Ster <dan-EOCVfBHj35C+XT7JhA+gdA@public.gmane.org> wrote:
> On Thu, Mar 1, 2018 at 10:24 AM, Stefan Priebe - Profihost AG
> <s.priebe-2Lf/h1ldwEHR5kwTpVNS9A@public.gmane.org> wrote:
>>
>> Am 01.03.2018 um 09:58 schrieb Dan van der Ster:
>>> On Thu, Mar 1, 2018 at 9:52 AM, Stefan Priebe - Profihost AG
>>> <s.priebe-2Lf/h1ldwEHR5kwTpVNS9A@public.gmane.org> wrote:
>>>> Hi,
>>>>
>>>> Am 01.03.2018 um 09:42 schrieb Dan van der Ster:
>>>>> On Thu, Mar 1, 2018 at 9:31 AM, Stefan Priebe - Profihost AG
>>>>> <s.priebe-2Lf/h1ldwEHR5kwTpVNS9A@public.gmane.org> wrote:
>>>>>> Hi,
>>>>>> Am 01.03.2018 um 09:03 schrieb Dan van der Ster:
>>>>>>> Is the score improving?
>>>>>>>
>>>>>>>     ceph balancer eval
>>>>>>>
>>>>>>> It should be decreasing over time as the variances drop toward zero.
>>>>>>>
>>>>>>> You mentioned a crush optimize code at the beginning... how did that
>>>>>>> leave your cluster? The mgr balancer assumes that the crush weight of
>>>>>>> each OSD is equal to its size in TB.
>>>>>>> Do you have any osd reweights? crush-compat will gradually adjust
>>>>>>> those back to 1.0.
>>>>>>
>>>>>> I reweighted them all back to their correct weight.
>>>>>>
>>>>>> Now the mgr balancer module says:
>>>>>> mgr[balancer] Failed to find further optimization, score 0.010646
>>>>>>
>>>>>> But as you can see it's heavily imbalanced:
>>>>>>
>>>>>>
>>>>>> Example:
>>>>>> 49   ssd 0.84000  1.00000   864G   546G   317G 63.26 1.13  49
>>>>>>
>>>>>> vs:
>>>>>>
>>>>>> 48   ssd 0.84000  1.00000   864G   397G   467G 45.96 0.82  49
>>>>>>
>>>>>> 45% usage vs. 63%
>>>>>
>>>>> Ahh... but look, the num PGs are perfectly balanced, which implies
>>>>> that you have a relatively large number of empty PGs.
>>>>>
>>>>> But regardless, this is annoying and I expect lots of operators to get
>>>>> this result. (I've also observed that the num PGs is gets balanced
>>>>> perfectly at the expense of the other score metrics.)
>>>>>
>>>>> I was thinking of a patch around here [1] that lets operators add a
>>>>> score weight on pgs, objects, bytes so we can balance how we like.
>>>>>
>>>>> Spandan: you were the last to look at this function. Do you think it
>>>>> can be improved as I suggested?
>>>>
>>>> Yes the PGs are perfectly distributed - but i think most of the people
>>>> would like to have a dsitribution by bytes and not pgs.
>>>>
>>>> Is this possible? I mean in the code there is already a dict for pgs,
>>>> objects and bytes - but i don't know how to change the logic. Just
>>>> remove the pgs and objects from the dict?
>>>
>>> It's worth a try to remove the pgs and objects from this dict:
>>> https://github.com/ceph/ceph/blob/luminous/src/pybind/mgr/balancer/module.py#L552
>>
>> Do i have to change this 3 to 1 cause we have only one item in the dict?
>> I'm not sure where the 3 comes from.
>>         pe.score /= 3 * len(roots)
>>
>
> I'm pretty sure that 3 is just for our 3 metrics. Indeed you can
> change that to 1.
>
> I'm trying this on our test cluster here too. The last few lines of
> output from `ceph balancer eval-verbose` will confirm that the score
> is based only on bytes.
>
> But I'm not sure this is going to work -- indeed the score here went
> from ~0.02 to 0.08, but the do_crush_compat doesn't manage to find a
> better score.

Maybe this:

https://github.com/ceph/ceph/blob/luminous/src/pybind/mgr/balancer/module.py#L682

I'm trying with that = 'bytes'

-- dan

Re: ceph mgr balancer bad distribution

[Dan van der Ster]

On Thu, Mar 1, 2018 at 10:40 AM, Dan van der Ster <dan-EOCVfBHj35C+XT7JhA+gdA@public.gmane.org> wrote:
> On Thu, Mar 1, 2018 at 10:38 AM, Dan van der Ster <dan-EOCVfBHj35C+XT7JhA+gdA@public.gmane.org> wrote:
>> On Thu, Mar 1, 2018 at 10:24 AM, Stefan Priebe - Profihost AG
>> <s.priebe-2Lf/h1ldwEHR5kwTpVNS9A@public.gmane.org> wrote:
>>>
>>> Am 01.03.2018 um 09:58 schrieb Dan van der Ster:
>>>> On Thu, Mar 1, 2018 at 9:52 AM, Stefan Priebe - Profihost AG
>>>> <s.priebe-2Lf/h1ldwEHR5kwTpVNS9A@public.gmane.org> wrote:
>>>>> Hi,
>>>>>
>>>>> Am 01.03.2018 um 09:42 schrieb Dan van der Ster:
>>>>>> On Thu, Mar 1, 2018 at 9:31 AM, Stefan Priebe - Profihost AG
>>>>>> <s.priebe-2Lf/h1ldwEHR5kwTpVNS9A@public.gmane.org> wrote:
>>>>>>> Hi,
>>>>>>> Am 01.03.2018 um 09:03 schrieb Dan van der Ster:
>>>>>>>> Is the score improving?
>>>>>>>>
>>>>>>>>     ceph balancer eval
>>>>>>>>
>>>>>>>> It should be decreasing over time as the variances drop toward zero.
>>>>>>>>
>>>>>>>> You mentioned a crush optimize code at the beginning... how did that
>>>>>>>> leave your cluster? The mgr balancer assumes that the crush weight of
>>>>>>>> each OSD is equal to its size in TB.
>>>>>>>> Do you have any osd reweights? crush-compat will gradually adjust
>>>>>>>> those back to 1.0.
>>>>>>>
>>>>>>> I reweighted them all back to their correct weight.
>>>>>>>
>>>>>>> Now the mgr balancer module says:
>>>>>>> mgr[balancer] Failed to find further optimization, score 0.010646
>>>>>>>
>>>>>>> But as you can see it's heavily imbalanced:
>>>>>>>
>>>>>>>
>>>>>>> Example:
>>>>>>> 49   ssd 0.84000  1.00000   864G   546G   317G 63.26 1.13  49
>>>>>>>
>>>>>>> vs:
>>>>>>>
>>>>>>> 48   ssd 0.84000  1.00000   864G   397G   467G 45.96 0.82  49
>>>>>>>
>>>>>>> 45% usage vs. 63%
>>>>>>
>>>>>> Ahh... but look, the num PGs are perfectly balanced, which implies
>>>>>> that you have a relatively large number of empty PGs.
>>>>>>
>>>>>> But regardless, this is annoying and I expect lots of operators to get
>>>>>> this result. (I've also observed that the num PGs is gets balanced
>>>>>> perfectly at the expense of the other score metrics.)
>>>>>>
>>>>>> I was thinking of a patch around here [1] that lets operators add a
>>>>>> score weight on pgs, objects, bytes so we can balance how we like.
>>>>>>
>>>>>> Spandan: you were the last to look at this function. Do you think it
>>>>>> can be improved as I suggested?
>>>>>
>>>>> Yes the PGs are perfectly distributed - but i think most of the people
>>>>> would like to have a dsitribution by bytes and not pgs.
>>>>>
>>>>> Is this possible? I mean in the code there is already a dict for pgs,
>>>>> objects and bytes - but i don't know how to change the logic. Just
>>>>> remove the pgs and objects from the dict?
>>>>
>>>> It's worth a try to remove the pgs and objects from this dict:
>>>> https://github.com/ceph/ceph/blob/luminous/src/pybind/mgr/balancer/module.py#L552
>>>
>>> Do i have to change this 3 to 1 cause we have only one item in the dict?
>>> I'm not sure where the 3 comes from.
>>>         pe.score /= 3 * len(roots)
>>>
>>
>> I'm pretty sure that 3 is just for our 3 metrics. Indeed you can
>> change that to 1.
>>
>> I'm trying this on our test cluster here too. The last few lines of
>> output from `ceph balancer eval-verbose` will confirm that the score
>> is based only on bytes.
>>
>> But I'm not sure this is going to work -- indeed the score here went
>> from ~0.02 to 0.08, but the do_crush_compat doesn't manage to find a
>> better score.
>
> Maybe this:
>
> https://github.com/ceph/ceph/blob/luminous/src/pybind/mgr/balancer/module.py#L682
>
> I'm trying with that = 'bytes'

That seems to be working. I sent this PR as a start
https://github.com/ceph/ceph/pull/20665

I'm not sure we need to mess with the score function, on second thought.

-- dan

Re: ceph mgr balancer bad distribution

[Stefan Priebe - Profihost AG]

nice thanks will try that soon.

Can you tell me how to change the log lever to info for the balancer module?

Am 01.03.2018 um 11:30 schrieb Dan van der Ster:
> On Thu, Mar 1, 2018 at 10:40 AM, Dan van der Ster <dan-EOCVfBHj35C+XT7JhA+gdA@public.gmane.org> wrote:
>> On Thu, Mar 1, 2018 at 10:38 AM, Dan van der Ster <dan-EOCVfBHj35C+XT7JhA+gdA@public.gmane.org> wrote:
>>> On Thu, Mar 1, 2018 at 10:24 AM, Stefan Priebe - Profihost AG
>>> <s.priebe-2Lf/h1ldwEHR5kwTpVNS9A@public.gmane.org> wrote:
>>>>
>>>> Am 01.03.2018 um 09:58 schrieb Dan van der Ster:
>>>>> On Thu, Mar 1, 2018 at 9:52 AM, Stefan Priebe - Profihost AG
>>>>> <s.priebe-2Lf/h1ldwEHR5kwTpVNS9A@public.gmane.org> wrote:
>>>>>> Hi,
>>>>>>
>>>>>> Am 01.03.2018 um 09:42 schrieb Dan van der Ster:
>>>>>>> On Thu, Mar 1, 2018 at 9:31 AM, Stefan Priebe - Profihost AG
>>>>>>> <s.priebe-2Lf/h1ldwEHR5kwTpVNS9A@public.gmane.org> wrote:
>>>>>>>> Hi,
>>>>>>>> Am 01.03.2018 um 09:03 schrieb Dan van der Ster:
>>>>>>>>> Is the score improving?
>>>>>>>>>
>>>>>>>>>     ceph balancer eval
>>>>>>>>>
>>>>>>>>> It should be decreasing over time as the variances drop toward zero.
>>>>>>>>>
>>>>>>>>> You mentioned a crush optimize code at the beginning... how did that
>>>>>>>>> leave your cluster? The mgr balancer assumes that the crush weight of
>>>>>>>>> each OSD is equal to its size in TB.
>>>>>>>>> Do you have any osd reweights? crush-compat will gradually adjust
>>>>>>>>> those back to 1.0.
>>>>>>>>
>>>>>>>> I reweighted them all back to their correct weight.
>>>>>>>>
>>>>>>>> Now the mgr balancer module says:
>>>>>>>> mgr[balancer] Failed to find further optimization, score 0.010646
>>>>>>>>
>>>>>>>> But as you can see it's heavily imbalanced:
>>>>>>>>
>>>>>>>>
>>>>>>>> Example:
>>>>>>>> 49   ssd 0.84000  1.00000   864G   546G   317G 63.26 1.13  49
>>>>>>>>
>>>>>>>> vs:
>>>>>>>>
>>>>>>>> 48   ssd 0.84000  1.00000   864G   397G   467G 45.96 0.82  49
>>>>>>>>
>>>>>>>> 45% usage vs. 63%
>>>>>>>
>>>>>>> Ahh... but look, the num PGs are perfectly balanced, which implies
>>>>>>> that you have a relatively large number of empty PGs.
>>>>>>>
>>>>>>> But regardless, this is annoying and I expect lots of operators to get
>>>>>>> this result. (I've also observed that the num PGs is gets balanced
>>>>>>> perfectly at the expense of the other score metrics.)
>>>>>>>
>>>>>>> I was thinking of a patch around here [1] that lets operators add a
>>>>>>> score weight on pgs, objects, bytes so we can balance how we like.
>>>>>>>
>>>>>>> Spandan: you were the last to look at this function. Do you think it
>>>>>>> can be improved as I suggested?
>>>>>>
>>>>>> Yes the PGs are perfectly distributed - but i think most of the people
>>>>>> would like to have a dsitribution by bytes and not pgs.
>>>>>>
>>>>>> Is this possible? I mean in the code there is already a dict for pgs,
>>>>>> objects and bytes - but i don't know how to change the logic. Just
>>>>>> remove the pgs and objects from the dict?
>>>>>
>>>>> It's worth a try to remove the pgs and objects from this dict:
>>>>> https://github.com/ceph/ceph/blob/luminous/src/pybind/mgr/balancer/module.py#L552
>>>>
>>>> Do i have to change this 3 to 1 cause we have only one item in the dict?
>>>> I'm not sure where the 3 comes from.
>>>>         pe.score /= 3 * len(roots)
>>>>
>>>
>>> I'm pretty sure that 3 is just for our 3 metrics. Indeed you can
>>> change that to 1.
>>>
>>> I'm trying this on our test cluster here too. The last few lines of
>>> output from `ceph balancer eval-verbose` will confirm that the score
>>> is based only on bytes.
>>>
>>> But I'm not sure this is going to work -- indeed the score here went
>>> from ~0.02 to 0.08, but the do_crush_compat doesn't manage to find a
>>> better score.
>>
>> Maybe this:
>>
>> https://github.com/ceph/ceph/blob/luminous/src/pybind/mgr/balancer/module.py#L682
>>
>> I'm trying with that = 'bytes'
> 
> That seems to be working. I sent this PR as a start
> https://github.com/ceph/ceph/pull/20665
> 
> I'm not sure we need to mess with the score function, on second thought.
> 
> -- dan
> 

Re: ceph mgr balancer bad distribution

[Dan van der Ster]

On Thu, Mar 1, 2018 at 1:08 PM, Stefan Priebe - Profihost AG
<s.priebe-2Lf/h1ldwEHR5kwTpVNS9A@public.gmane.org> wrote:
> nice thanks will try that soon.
>
> Can you tell me how to change the log lever to info for the balancer module?

debug mgr = 4/5

-- dan

Re: ceph mgr balancer bad distribution

[Stefan Priebe - Profihost AG]

[-- Attachment #1.1: Type: text/plain, Size: 790 bytes --]

Thanks! Your patch works great! The only problem I still see is that the balancer kicks in even when the old optimize has not finished. It seems it only evaluated the degraded of value. But while remapping it can happen that none are degraded but a lot are still misplaced.

I think the balancer should evaluate the ceph health status as well.

Stefan

Excuse my typo sent from my mobile phone.

> Am 01.03.2018 um 13:12 schrieb Dan van der Ster <dan-EOCVfBHj35C+XT7JhA+gdA@public.gmane.org>:
> 
> On Thu, Mar 1, 2018 at 1:08 PM, Stefan Priebe - Profihost AG
> <s.priebe-2Lf/h1ldwEHR5kwTpVNS9A@public.gmane.org> wrote:
>> nice thanks will try that soon.
>> 
>> Can you tell me how to change the log lever to info for the balancer module?
> 
> debug mgr = 4/5
> 
> -- dan

[-- Attachment #1.2: Type: text/html, Size: 1432 bytes --]

[-- Attachment #2: Type: text/plain, Size: 178 bytes --]

_______________________________________________
ceph-users mailing list
ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: ceph mgr balancer bad distribution

[Stefan Priebe - Profihost AG]

Thanks! Your patch works great! The only problem I still see is that the
balancer kicks in even when the old optimize has not finished. It seems
it only evaluated the degraded of value. But while remapping it can
happen that none are degraded but a lot are still misplaced.

I think the balancer should evaluate the ceph health status as well.

Stefan

Excuse my typo sent from my mobile phone.

Am 01.03.2018 um 13:12 schrieb Dan van der Ster <dan-EOCVfBHj35C+XT7JhA+gdA@public.gmane.org
<mailto:dan-EOCVfBHj35C+XT7JhA+gdA@public.gmane.org>>:

> On Thu, Mar 1, 2018 at 1:08 PM, Stefan Priebe - Profihost AG
> <s.priebe-2Lf/h1ldwEHR5kwTpVNS9A@public.gmane.org <mailto:s.priebe-2Lf/h1ldwEHR5kwTpVNS9A@public.gmane.org>> wrote:
>> nice thanks will try that soon.
>>
>> Can you tell me how to change the log lever to info for the balancer
>> module?
>
> debug mgr = 4/5
>
> -- dan

Re: ceph mgr balancer bad distribution

[Dan van der Ster]

On Fri, Mar 2, 2018 at 10:12 AM, Stefan Priebe - Profihost AG
<s.priebe-2Lf/h1ldwEHR5kwTpVNS9A@public.gmane.org> wrote:
> Thanks! Your patch works great!

Cool! I plan to add one more feature to allow operators to switch off
components of the score function. Currently, by only changing the key
to 'bytes', we aren't able to fully balance things because at some
point the pgs score gets too suboptimal and the overall score reaches
a min value.

> The only problem I still see is that the
> balancer kicks in even when the old optimize has not finished. It seems it
> only evaluated the degraded of value. But while remapping it can happen that
> none are degraded but a lot are still misplaced.
>
> I think the balancer should evaluate the ceph health status as well.

I guess this point is debatable. On our clusters we use max_misplaced
= 0.01 and set the begin_hour end_hour to daytime hours so that by the
late evening, every day, the cluster is back to HEALTH_OK.

Cheers, Dan

> Stefan
>
> Excuse my typo sent from my mobile phone.
>
> Am 01.03.2018 um 13:12 schrieb Dan van der Ster <dan-EOCVfBHj35C+XT7JhA+gdA@public.gmane.org>:
>
> On Thu, Mar 1, 2018 at 1:08 PM, Stefan Priebe - Profihost AG
> <s.priebe-2Lf/h1ldwEHR5kwTpVNS9A@public.gmane.org> wrote:
>
> nice thanks will try that soon.
>
>
> Can you tell me how to change the log lever to info for the balancer module?
>
>
> debug mgr = 4/5
>
> -- dan

Re: ceph mgr balancer bad distribution

[Stefan Priebe - Profihost AG]

Hi,

Am 02.03.2018 um 14:29 schrieb Dan van der Ster:
> On Fri, Mar 2, 2018 at 10:12 AM, Stefan Priebe - Profihost AG
> <s.priebe-2Lf/h1ldwEHR5kwTpVNS9A@public.gmane.org> wrote:
>> Thanks! Your patch works great!
> 
> Cool! I plan to add one more feature to allow operators to switch off
> components of the score function. Currently, by only changing the key
> to 'bytes', we aren't able to fully balance things because at some
> point the pgs score gets too suboptimal and the overall score reaches
> a min value.

OK great but it still works perfectly for my clusters. 2%-3% difference
for all OSDs.

>> The only problem I still see is that the
>> balancer kicks in even when the old optimize has not finished. It seems it
>> only evaluated the degraded of value. But while remapping it can happen that
>> none are degraded but a lot are still misplaced.
>>
>> I think the balancer should evaluate the ceph health status as well.
> 
> I guess this point is debatable. On our clusters we use max_misplaced
> = 0.01 and set the begin_hour end_hour to daytime hours so that by the
> late evening, every day, the cluster is back to HEALTH_OK.

ah OK. Is it true that the begin_time and end_time is GMT and not local
time? Can we change this as it makes configuration of monitoring systems
impossible - time changes also with summer and winter time.

Greets,
Stefan

> Cheers, Dan
> 
>> Stefan
>>
>> Excuse my typo sent from my mobile phone.
>>
>> Am 01.03.2018 um 13:12 schrieb Dan van der Ster <dan-EOCVfBHj35C+XT7JhA+gdA@public.gmane.org>:
>>
>> On Thu, Mar 1, 2018 at 1:08 PM, Stefan Priebe - Profihost AG
>> <s.priebe-2Lf/h1ldwEHR5kwTpVNS9A@public.gmane.org> wrote:
>>
>> nice thanks will try that soon.
>>
>>
>> Can you tell me how to change the log lever to info for the balancer module?
>>
>>
>> debug mgr = 4/5
>>
>> -- dan

Re: ceph mgr balancer bad distribution

[Stefan Priebe - Profihost AG]

Hi,

Am 02.03.2018 um 21:21 schrieb Stefan Priebe - Profihost AG:
> Hi,
> 
> Am 02.03.2018 um 14:29 schrieb Dan van der Ster:
>> On Fri, Mar 2, 2018 at 10:12 AM, Stefan Priebe - Profihost AG
>> <s.priebe-2Lf/h1ldwEHR5kwTpVNS9A@public.gmane.org> wrote:
>>> Thanks! Your patch works great!
>>
>> Cool! I plan to add one more feature to allow operators to switch off
>> components of the score function. Currently, by only changing the key
>> to 'bytes', we aren't able to fully balance things because at some
>> point the pgs score gets too suboptimal and the overall score reaches
>> a min value.
> 
> OK great but it still works perfectly for my clusters. 2%-3% difference
> for all OSDs.

ok i had another cluster - where this seems to happen. It just says
Failed to find further optimization but the bytes difference is 12%.

>>> The only problem I still see is that the
>>> balancer kicks in even when the old optimize has not finished. It seems it
>>> only evaluated the degraded of value. But while remapping it can happen that
>>> none are degraded but a lot are still misplaced.
>>>
>>> I think the balancer should evaluate the ceph health status as well.
>>
>> I guess this point is debatable. On our clusters we use max_misplaced
>> = 0.01 and set the begin_hour end_hour to daytime hours so that by the
>> late evening, every day, the cluster is back to HEALTH_OK.
> 
> ah OK. Is it true that the begin_time and end_time is GMT and not local
> time? Can we change this as it makes configuration of monitoring systems
> impossible - time changes also with summer and winter time.

Sorry at least in my case this works fine.

> 
> Greets,
> Stefan
> 
>> Cheers, Dan
>>
>>> Stefan
>>>
>>> Excuse my typo sent from my mobile phone.
>>>
>>> Am 01.03.2018 um 13:12 schrieb Dan van der Ster <dan-EOCVfBHj35C+XT7JhA+gdA@public.gmane.org>:
>>>
>>> On Thu, Mar 1, 2018 at 1:08 PM, Stefan Priebe - Profihost AG
>>> <s.priebe-2Lf/h1ldwEHR5kwTpVNS9A@public.gmane.org> wrote:
>>>
>>> nice thanks will try that soon.
>>>
>>>
>>> Can you tell me how to change the log lever to info for the balancer module?
>>>
>>>
>>> debug mgr = 4/5
>>>
>>> -- dan