ceph-mgr is in master

ceph-mgr is in master

[John Spray]

Hi all,

A big thank you to everyone who helped with the reviews, packaging and
design discussions.  This will be included in forthcoming Kraken
(11.2.x) release.

There is a new section in the documentation:
http://docs.ceph.com/docs/master/mgr/

There is a new sub-project on tracker.ceph.com, populated with some
issues by me:
http://tracker.ceph.com/projects/mgr/issues

Pull requests are of course welcome.

At some point before Kraken I'll try to remember to send out a message
to ceph-users to give people a heads up about this mysterious new
daemon that's appearing on their systems.

Thanks,
John

Re: ceph-mgr is in master

[Gregory Farnum]

On Fri, Sep 30, 2016 at 4:24 AM, John Spray <jspray@redhat.com> wrote:
> Hi all,
>
> A big thank you to everyone who helped with the reviews, packaging and
> design discussions.  This will be included in forthcoming Kraken
> (11.2.x) release.
>
> There is a new section in the documentation:
> http://docs.ceph.com/docs/master/mgr/
>
> There is a new sub-project on tracker.ceph.com, populated with some
> issues by me:
> http://tracker.ceph.com/projects/mgr/issues
>
> Pull requests are of course welcome.
>
> At some point before Kraken I'll try to remember to send out a message
> to ceph-users to give people a heads up about this mysterious new
> daemon that's appearing on their systems.

Awesome, this is going to be great. :) A few questions just from going
over the admin guide:

1) It says to run a ceph-mgr on each monitor host. Does this mean
admins need to do this manually when setting up a cluster?

2) I gather that the way HA works is that traffic just gets directed
to whichever one the monitors pick. It seems like we'd want that to
migrate to the same host as the leader monitor when possible; is that
possible? Is it planned? Is it actually undesirable because it
increases CPU usage and network savings isn't worth that?

3) It uses "ceph tell" syntax. I thought we were moving away from
top-level "tell" commands and towards making them part of the
sub-command (eg "ceph osd tell", "ceph mgr tell"); is this a
deliberate new lease on life for that? Unintended? Something I just
made up?

-Greg

Re: ceph-mgr is in master

[Dan Mick]

On 10/03/2016 10:45 AM, Gregory Farnum wrote:

> 3) It uses "ceph tell" syntax. I thought we were moving away from
> top-level "tell" commands and towards making them part of the
> sub-command (eg "ceph osd tell", "ceph mgr tell"); is this a
> deliberate new lease on life for that? Unintended? Something I just
> made up?

Actually it's the other way around; 'osd tell', for example, says:

"osd tell" is deprecated; try "tell osd.<id> <command> [options...]"
instead (id can be "*")

Re: ceph-mgr is in master

[Gregory Farnum]

On Mon, Oct 3, 2016 at 8:16 PM, Dan Mick <dmick@redhat.com> wrote:
> On 10/03/2016 10:45 AM, Gregory Farnum wrote:
>
>> 3) It uses "ceph tell" syntax. I thought we were moving away from
>> top-level "tell" commands and towards making them part of the
>> sub-command (eg "ceph osd tell", "ceph mgr tell"); is this a
>> deliberate new lease on life for that? Unintended? Something I just
>> made up?
>
> Actually it's the other way around; 'osd tell', for example, says:
>
> "osd tell" is deprecated; try "tell osd.<id> <command> [options...]"
> instead (id can be "*")

o_0. Thanks Dan.

In other news we've also acquired some valgrind errors in the
MgrMonitor; not sure how serious they are but it's getting flagged in
the tests. http://tracker.ceph.com/issues/17492

Re: ceph-mgr is in master

[Sage Weil]

On Mon, 3 Oct 2016, Gregory Farnum wrote:
> On Mon, Oct 3, 2016 at 8:16 PM, Dan Mick <dmick@redhat.com> wrote:
> > On 10/03/2016 10:45 AM, Gregory Farnum wrote:
> >
> >> 3) It uses "ceph tell" syntax. I thought we were moving away from
> >> top-level "tell" commands and towards making them part of the
> >> sub-command (eg "ceph osd tell", "ceph mgr tell"); is this a
> >> deliberate new lease on life for that? Unintended? Something I just
> >> made up?
> >
> > Actually it's the other way around; 'osd tell', for example, says:
> >
> > "osd tell" is deprecated; try "tell osd.<id> <command> [options...]"
> > instead (id can be "*")
> 
> o_0. Thanks Dan.
> 
> In other news we've also acquired some valgrind errors in the
> MgrMonitor; not sure how serious they are but it's getting flagged in
> the tests. http://tracker.ceph.com/issues/17492

https://github.com/ceph/ceph/pull/11308

Re: ceph-mgr is in master

[John Spray]

On Mon, Oct 3, 2016 at 6:45 PM, Gregory Farnum <gfarnum@redhat.com> wrote:
> On Fri, Sep 30, 2016 at 4:24 AM, John Spray <jspray@redhat.com> wrote:
>> Hi all,
>>
>> A big thank you to everyone who helped with the reviews, packaging and
>> design discussions.  This will be included in forthcoming Kraken
>> (11.2.x) release.
>>
>> There is a new section in the documentation:
>> http://docs.ceph.com/docs/master/mgr/
>>
>> There is a new sub-project on tracker.ceph.com, populated with some
>> issues by me:
>> http://tracker.ceph.com/projects/mgr/issues
>>
>> Pull requests are of course welcome.
>>
>> At some point before Kraken I'll try to remember to send out a message
>> to ceph-users to give people a heads up about this mysterious new
>> daemon that's appearing on their systems.
>
> Awesome, this is going to be great. :) A few questions just from going
> over the admin guide:
>
> 1) It says to run a ceph-mgr on each monitor host. Does this mean
> admins need to do this manually when setting up a cluster?

The systemd units will automatically set up a mgr instance in Kraken.
I haven't personally tested that though.

> 2) I gather that the way HA works is that traffic just gets directed
> to whichever one the monitors pick. It seems like we'd want that to
> migrate to the same host as the leader monitor when possible; is that
> possible? Is it planned? Is it actually undesirable because it
> increases CPU usage and network savings isn't worth that?

The obvious advantage would be to avoid a network hop by running on
the same host as the leader.  However, unless it's always going to be
the case, that extra efficiency could give a false sense of security
if things are fast enough in that case, but become too slow when the
service runs somewhere else.  It would be interesting to try.

> 3) It uses "ceph tell" syntax. I thought we were moving away from
> top-level "tell" commands and towards making them part of the
> sub-command (eg "ceph osd tell", "ceph mgr tell"); is this a
> deliberate new lease on life for that? Unintended? Something I just
> made up?

As Dan says, `ceph tell` is the new one, the one that sends commands
directly from the CLI to the named service (not via the mon).
However, this is a bit of a placeholder, as we'll eventually be
switching to some commands (especially PG stats ones) to silently go
to the mgr instead of the mon.  There are various ways of
accomplishing that, the simplest one is to have the CLI fetch both mgr
and mon command descriptions, and preferentially send commands to
matching entries from the mgr.

John

>
> -Greg

Re: ceph-mgr is in master

[Willem Jan Withagen]

On 4-10-2016 08:07, John Spray wrote:
> As Dan says, `ceph tell` is the new one, the one that sends commands
> directly from the CLI to the named service (not via the mon).
> However, this is a bit of a placeholder, as we'll eventually be
> switching to some commands (especially PG stats ones) to silently go
> to the mgr instead of the mon.  There are various ways of
> accomplishing that, the simplest one is to have the CLI fetch both mgr
> and mon command descriptions, and preferentially send commands to
> matching entries from the mgr.

ATM I had to disable building MGR, because Clang/Python/FreeBSD throw a
fit when having to compile on of the python files...
Something to do with function-signatures not matching.
I'll look into it after I've completed most of my other blocking
problems. Unless you'd really want to have the compiler errors now, I'll
fireup a build with MGR on.

Just hope that this switch is not enforced before that...
And I'd then expect the WITH_MGR flag to go away?

--WjW

Re: ceph-mgr is in master

[John Spray]

On Tue, Oct 4, 2016 at 1:03 PM, Willem Jan Withagen <wjw@digiware.nl> wrote:
> On 4-10-2016 08:07, John Spray wrote:
>> As Dan says, `ceph tell` is the new one, the one that sends commands
>> directly from the CLI to the named service (not via the mon).
>> However, this is a bit of a placeholder, as we'll eventually be
>> switching to some commands (especially PG stats ones) to silently go
>> to the mgr instead of the mon.  There are various ways of
>> accomplishing that, the simplest one is to have the CLI fetch both mgr
>> and mon command descriptions, and preferentially send commands to
>> matching entries from the mgr.
>
> ATM I had to disable building MGR, because Clang/Python/FreeBSD throw a
> fit when having to compile on of the python files...
> Something to do with function-signatures not matching.
> I'll look into it after I've completed most of my other blocking
> problems. Unless you'd really want to have the compiler errors now, I'll
> fireup a build with MGR on.

Ah... yeah, I do seem to recall some places the compiler warned about
(perfectly safe) const char* casts.  I'll look into cleaning that up,
assuming it's what's making clang unhappy.

> Just hope that this switch is not enforced before that...
> And I'd then expect the WITH_MGR flag to go away?

Yeah, I think it would be reasonable to remove it some time before
Luminous, or whenever we move essential user-facing stuff (like pg
dump commands) into mgr so that it ceases to be optional.

John

>
> --WjW
>

Re: ceph-mgr is in master

[Willem Jan Withagen]

On 4-10-2016 14:16, John Spray wrote:
> On Tue, Oct 4, 2016 at 1:03 PM, Willem Jan Withagen <wjw@digiware.nl> wrote:
>> On 4-10-2016 08:07, John Spray wrote:
>>> As Dan says, `ceph tell` is the new one, the one that sends commands
>>> directly from the CLI to the named service (not via the mon).
>>> However, this is a bit of a placeholder, as we'll eventually be
>>> switching to some commands (especially PG stats ones) to silently go
>>> to the mgr instead of the mon.  There are various ways of
>>> accomplishing that, the simplest one is to have the CLI fetch both mgr
>>> and mon command descriptions, and preferentially send commands to
>>> matching entries from the mgr.
>>
>> ATM I had to disable building MGR, because Clang/Python/FreeBSD throw a
>> fit when having to compile on of the python files...
>> Something to do with function-signatures not matching.
>> I'll look into it after I've completed most of my other blocking
>> problems. Unless you'd really want to have the compiler errors now, I'll
>> fireup a build with MGR on.
> 
> Ah... yeah, I do seem to recall some places the compiler warned about
> (perfectly safe) const char* casts.  I'll look into cleaning that up,
> assuming it's what's making clang unhappy.

Clang is a lot more picky at times... But usually I get away with a
warning, this time it throws errors.

>> Just hope that this switch is not enforced before that...
>> And I'd then expect the WITH_MGR flag to go away?
> 
> Yeah, I think it would be reasonable to remove it some time before
> Luminous, or whenever we move essential user-facing stuff (like pg
> dump commands) into mgr so that it ceases to be optional.

Right, hope that buys me enough time. :)

--WjW

Re: ceph-mgr is in master

[John Spray]

On Tue, Oct 4, 2016 at 2:28 PM, Willem Jan Withagen <wjw@digiware.nl> wrote:
> On 4-10-2016 14:16, John Spray wrote:
>> On Tue, Oct 4, 2016 at 1:03 PM, Willem Jan Withagen <wjw@digiware.nl> wrote:
>>> On 4-10-2016 08:07, John Spray wrote:
>>>> As Dan says, `ceph tell` is the new one, the one that sends commands
>>>> directly from the CLI to the named service (not via the mon).
>>>> However, this is a bit of a placeholder, as we'll eventually be
>>>> switching to some commands (especially PG stats ones) to silently go
>>>> to the mgr instead of the mon.  There are various ways of
>>>> accomplishing that, the simplest one is to have the CLI fetch both mgr
>>>> and mon command descriptions, and preferentially send commands to
>>>> matching entries from the mgr.
>>>
>>> ATM I had to disable building MGR, because Clang/Python/FreeBSD throw a
>>> fit when having to compile on of the python files...
>>> Something to do with function-signatures not matching.
>>> I'll look into it after I've completed most of my other blocking
>>> problems. Unless you'd really want to have the compiler errors now, I'll
>>> fireup a build with MGR on.
>>
>> Ah... yeah, I do seem to recall some places the compiler warned about
>> (perfectly safe) const char* casts.  I'll look into cleaning that up,
>> assuming it's what's making clang unhappy.
>
> Clang is a lot more picky at times... But usually I get away with a
> warning, this time it throws errors.
>

Can you try this please?
https://github.com/ceph/ceph/pull/11307

John

>>> Just hope that this switch is not enforced before that...
>>> And I'd then expect the WITH_MGR flag to go away?
>>
>> Yeah, I think it would be reasonable to remove it some time before
>> Luminous, or whenever we move essential user-facing stuff (like pg
>> dump commands) into mgr so that it ceases to be optional.
>
> Right, hope that buys me enough time. :)
>
> --WjW
>

Re: ceph-mgr is in master

[Willem Jan Withagen]

On 4-10-2016 15:15, John Spray wrote:
> On Tue, Oct 4, 2016 at 2:28 PM, Willem Jan Withagen <wjw@digiware.nl> wrote:
>> On 4-10-2016 14:16, John Spray wrote:
>>> On Tue, Oct 4, 2016 at 1:03 PM, Willem Jan Withagen <wjw@digiware.nl> wrote:
>>>> On 4-10-2016 08:07, John Spray wrote:
>>>>> As Dan says, `ceph tell` is the new one, the one that sends commands
>>>>> directly from the CLI to the named service (not via the mon).
>>>>> However, this is a bit of a placeholder, as we'll eventually be
>>>>> switching to some commands (especially PG stats ones) to silently go
>>>>> to the mgr instead of the mon.  There are various ways of
>>>>> accomplishing that, the simplest one is to have the CLI fetch both mgr
>>>>> and mon command descriptions, and preferentially send commands to
>>>>> matching entries from the mgr.
>>>>
>>>> ATM I had to disable building MGR, because Clang/Python/FreeBSD throw a
>>>> fit when having to compile on of the python files...
>>>> Something to do with function-signatures not matching.
>>>> I'll look into it after I've completed most of my other blocking
>>>> problems. Unless you'd really want to have the compiler errors now, I'll
>>>> fireup a build with MGR on.
>>>
>>> Ah... yeah, I do seem to recall some places the compiler warned about
>>> (perfectly safe) const char* casts.  I'll look into cleaning that up,
>>> assuming it's what's making clang unhappy.
>>
>> Clang is a lot more picky at times... But usually I get away with a
>> warning, this time it throws errors.
>>
> 
> Can you try this please?
> https://github.com/ceph/ceph/pull/11307

Seems we are talking about the same errors.
At least PyObject_CallMethod is what I remember giving the errors.

I'll put this in and give it a spinf for its money...

--WjW

Re: ceph-mgr is in master

[Willem Jan Withagen]

On 4-10-2016 16:42, Willem Jan Withagen wrote:
> On 4-10-2016 15:15, John Spray wrote:
>> On Tue, Oct 4, 2016 at 2:28 PM, Willem Jan Withagen <wjw@digiware.nl> wrote:
>>> On 4-10-2016 14:16, John Spray wrote:
>>>> On Tue, Oct 4, 2016 at 1:03 PM, Willem Jan Withagen <wjw@digiware.nl> wrote:
>>>>> On 4-10-2016 08:07, John Spray wrote:
>>>>>> As Dan says, `ceph tell` is the new one, the one that sends commands
>>>>>> directly from the CLI to the named service (not via the mon).
>>>>>> However, this is a bit of a placeholder, as we'll eventually be
>>>>>> switching to some commands (especially PG stats ones) to silently go
>>>>>> to the mgr instead of the mon.  There are various ways of
>>>>>> accomplishing that, the simplest one is to have the CLI fetch both mgr
>>>>>> and mon command descriptions, and preferentially send commands to
>>>>>> matching entries from the mgr.
>>>>>
>>>>> ATM I had to disable building MGR, because Clang/Python/FreeBSD throw a
>>>>> fit when having to compile on of the python files...
>>>>> Something to do with function-signatures not matching.
>>>>> I'll look into it after I've completed most of my other blocking
>>>>> problems. Unless you'd really want to have the compiler errors now, I'll
>>>>> fireup a build with MGR on.
>>>>
>>>> Ah... yeah, I do seem to recall some places the compiler warned about
>>>> (perfectly safe) const char* casts.  I'll look into cleaning that up,
>>>> assuming it's what's making clang unhappy.
>>>
>>> Clang is a lot more picky at times... But usually I get away with a
>>> warning, this time it throws errors.
>>>
>>
>> Can you try this please?
>> https://github.com/ceph/ceph/pull/11307
> 
> Seems we are talking about the same errors.
> At least PyObject_CallMethod is what I remember giving the errors.
> 
> I'll put this in and give it a spinf for its money...

This is another Clang warning, I guess...
But as I read it, it sounds rather true...

/usr/srcs/Ceph/work/ceph/src/messages/MMgrReport.h:39:17: warning:
comparison of constant 256 with expression of type 'const enum
perfcounter_type_d' is always true
      [-Wtautological-constant-out-of-range-compare]
    assert(type < 256);
           ~~~~ ^ ~~~
atleast if enum perfcounter_type_d is a byte.

--WjW

Re: ceph-mgr is in master

[John Spray]

On Wed, Oct 5, 2016 at 6:25 PM, Willem Jan Withagen <wjw@digiware.nl> wrote:
> On 4-10-2016 16:42, Willem Jan Withagen wrote:
>> On 4-10-2016 15:15, John Spray wrote:
>>> On Tue, Oct 4, 2016 at 2:28 PM, Willem Jan Withagen <wjw@digiware.nl> wrote:
>>>> On 4-10-2016 14:16, John Spray wrote:
>>>>> On Tue, Oct 4, 2016 at 1:03 PM, Willem Jan Withagen <wjw@digiware.nl> wrote:
>>>>>> On 4-10-2016 08:07, John Spray wrote:
>>>>>>> As Dan says, `ceph tell` is the new one, the one that sends commands
>>>>>>> directly from the CLI to the named service (not via the mon).
>>>>>>> However, this is a bit of a placeholder, as we'll eventually be
>>>>>>> switching to some commands (especially PG stats ones) to silently go
>>>>>>> to the mgr instead of the mon.  There are various ways of
>>>>>>> accomplishing that, the simplest one is to have the CLI fetch both mgr
>>>>>>> and mon command descriptions, and preferentially send commands to
>>>>>>> matching entries from the mgr.
>>>>>>
>>>>>> ATM I had to disable building MGR, because Clang/Python/FreeBSD throw a
>>>>>> fit when having to compile on of the python files...
>>>>>> Something to do with function-signatures not matching.
>>>>>> I'll look into it after I've completed most of my other blocking
>>>>>> problems. Unless you'd really want to have the compiler errors now, I'll
>>>>>> fireup a build with MGR on.
>>>>>
>>>>> Ah... yeah, I do seem to recall some places the compiler warned about
>>>>> (perfectly safe) const char* casts.  I'll look into cleaning that up,
>>>>> assuming it's what's making clang unhappy.
>>>>
>>>> Clang is a lot more picky at times... But usually I get away with a
>>>> warning, this time it throws errors.
>>>>
>>>
>>> Can you try this please?
>>> https://github.com/ceph/ceph/pull/11307
>>
>> Seems we are talking about the same errors.
>> At least PyObject_CallMethod is what I remember giving the errors.
>>
>> I'll put this in and give it a spinf for its money...
>
> This is another Clang warning, I guess...
> But as I read it, it sounds rather true...
>
> /usr/srcs/Ceph/work/ceph/src/messages/MMgrReport.h:39:17: warning:
> comparison of constant 256 with expression of type 'const enum
> perfcounter_type_d' is always true
>       [-Wtautological-constant-out-of-range-compare]
>     assert(type < 256);
>            ~~~~ ^ ~~~
> atleast if enum perfcounter_type_d is a byte.

Fix here: https://github.com/ceph/ceph/pull/11345

John

>
> --WjW
>

Re: ceph-mgr is in master

[Tim Serong]

On 10/04/2016 08:07 AM, John Spray wrote:
> On Mon, Oct 3, 2016 at 6:45 PM, Gregory Farnum <gfarnum@redhat.com> wrote:
>> On Fri, Sep 30, 2016 at 4:24 AM, John Spray <jspray@redhat.com> wrote:
>>> Hi all,
>>>
>>> A big thank you to everyone who helped with the reviews, packaging and
>>> design discussions.  This will be included in forthcoming Kraken
>>> (11.2.x) release.
>>>
>>> There is a new section in the documentation:
>>> http://docs.ceph.com/docs/master/mgr/
>>>
>>> There is a new sub-project on tracker.ceph.com, populated with some
>>> issues by me:
>>> http://tracker.ceph.com/projects/mgr/issues
>>>
>>> Pull requests are of course welcome.
>>>
>>> At some point before Kraken I'll try to remember to send out a message
>>> to ceph-users to give people a heads up about this mysterious new
>>> daemon that's appearing on their systems.
>>
>> Awesome, this is going to be great. :) A few questions just from going
>> over the admin guide:
>>
>> 1) It says to run a ceph-mgr on each monitor host. Does this mean
>> admins need to do this manually when setting up a cluster?
> 
> The systemd units will automatically set up a mgr instance in Kraken.
> I haven't personally tested that though.

If you want a little more detail:

https://github.com/ceph/ceph/commit/61d7793 added
Wants=ceph-mgr@%i.service to ceph-mon@.service, so when the mon starts,
it will try to start mgr with the same instance ID, but because it's
"Wants" (not "Requires"), if mgr for some reason fails to start, the mon
will still run.

Additionally, ceph-mgr@.service will autogenerate a key for mgr if you
didn't already create one manually (see
https://github.com/ceph/ceph/commit/082199f).

This seemed the safest way to get the mgr daemon *running* automatically
with zero setup effort on the part of the user.

One kink ATM -- ceph-mgr will run, but if you want to *use* the included
REST API module, you additionally need to `pip2 install django==1.8
python-dateutil gevent djangorestframework`, because that module has
those dependencies.

Regards,

Tim
-- 
Tim Serong
Senior Clustering Engineer
SUSE
tserong@suse.com