reweight-by-utilization warning

reweight-by-utilization warning

[Blair Bethwaite]

Hi all,

IMHO reweight-by-utilization should come with some sort of warning, it
just suddenly reweights everything - no dry run, no confirmation,
apparently no option to see what it's going to do. It also doesn't
appear to consider pools and hence crush rulesets, which I imagine
could result in it making some poor reweighting decisions.

We ran it on a cluster this evening and promptly had over 70% of
objects misplaced - even at 5-7 GB/s that's quite a lot of data
movement when there are half a billion objects in the cluster! I think
we'll stick with Dan's scripts
(https://github.com/cernceph/ceph-scripts/blob/master/tools/crush-reweight-by-utilization.py)
for the moment (thanks Dan!).

-- 
Cheers,
~Blairo

Re: [ceph-users] reweight-by-utilization warning

[Dan van der Ster]

Hi Blaire!

The good news is that the functionality of that python script is now
available natively in jewel and has been backported to hammer 0.96.7.

Now you can use

  ceph osd test-reweight-by-(pg|utilization)

in order to see how the weights would change if you were to run
reweight-by-(pg|utilization). Also there are some new options on the
(test-)reweight-by-* functions which allow you to adjust the max
weight changed per run and num osds changed per run.

Hope that helps!

Dan

On 15 May 2016 4:09 pm, "Blair Bethwaite" <blair.bethwaite@gmail.com> wrote:
>
> Hi all,
>
> IMHO reweight-by-utilization should come with some sort of warning, it
> just suddenly reweights everything - no dry run, no confirmation,
> apparently no option to see what it's going to do. It also doesn't
> appear to consider pools and hence crush rulesets, which I imagine
> could result in it making some poor reweighting decisions.
>
> We ran it on a cluster this evening and promptly had over 70% of
> objects misplaced - even at 5-7 GB/s that's quite a lot of data
> movement when there are half a billion objects in the cluster! I think
> we'll stick with Dan's scripts
> (https://github.com/cernceph/ceph-scripts/blob/master/tools/crush-reweight-by-utilization.py)
> for the moment (thanks Dan!).
>
> --
> Cheers,
> ~Blairo
> _______________________________________________
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] reweight-by-utilization warning

[Dan van der Ster]

Hi Blaire! (re-copying to list)

The good news is that the functionality of that python script is now
available natively in jewel and has been backported to hammer 0.96.7.

Now you can use

  ceph osd test-reweight-by-(pg|utilization)

in order to see how the weights would change if you were to run
reweight-by-(pg|utilization). Also there are some new options on the
(test-)reweight-by-* functions which allow you to adjust the max
weight changed per run and num osds changed per run.

Hope that helps!

Dan


On Sun, May 15, 2016 at 4:09 PM, Blair Bethwaite
<blair.bethwaite@gmail.com> wrote:
> Hi all,
>
> IMHO reweight-by-utilization should come with some sort of warning, it
> just suddenly reweights everything - no dry run, no confirmation,
> apparently no option to see what it's going to do. It also doesn't
> appear to consider pools and hence crush rulesets, which I imagine
> could result in it making some poor reweighting decisions.
>
> We ran it on a cluster this evening and promptly had over 70% of
> objects misplaced - even at 5-7 GB/s that's quite a lot of data
> movement when there are half a billion objects in the cluster! I think
> we'll stick with Dan's scripts
> (https://github.com/cernceph/ceph-scripts/blob/master/tools/crush-reweight-by-utilization.py)
> for the moment (thanks Dan!).
>
> --
> Cheers,
> ~Blairo
> _______________________________________________
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com