ceph-mgr setup magic

ceph-mgr setup magic

[Sage Weil]

There is some goofy behavior with the systemd unit files that tries to 
make a new ceph-mgr daemon automagically appear next to the mons during a 
cluster upgrade:

- The ceph-mon@.service triggers ceph-mgr@ to start
- The ceph-mgr@.service tries to create a ceph-mgr key using whatever 
mon key it can find

This was intended to ease the upgrade path by auto-creating ceph-mgr 
daemons along the way.

Do we really want this?  I was reminded this behavior is here because of 
#19994, which points out that this isn't implemented for upstart (still 
used by trusty 14.04).

The jewel -> luminous upgrade process is currently documented here

	http://docs.ceph.com/docs/master/release-notes/#upgrade-from-jewel-or-kraken

and basically has a step that says 'verify mgr daemons are running and, if 
not, create some'.  Is that sufficient?

sage

Re: ceph-mgr setup magic

[Tim Serong]

On 06/29/2017 05:04 AM, Sage Weil wrote:
> There is some goofy behavior with the systemd unit files that tries to 
> make a new ceph-mgr daemon automagically appear next to the mons during a 
> cluster upgrade:
> 
> - The ceph-mon@.service triggers ceph-mgr@ to start
> - The ceph-mgr@.service tries to create a ceph-mgr key using whatever 
> mon key it can find
> 
> This was intended to ease the upgrade path by auto-creating ceph-mgr 
> daemons along the way.

It's also helpful for new installs; it means you don't actually have to
explicitly configure mgr daemons in order to get a running cluster.  You
just end up with as many mgrs as mons automatically.

That said, it is somewhat obscure, and also can make it slightly
irritating if you want to run mgrs on separate nodes (then you have to
explicitly deploy mgr, and also either uninstall the ceph-mgr package
from the mon nodes, or at least `systemctl mask ceph-mgr@$(hostname)` on
those nodes.

> Do we really want this?  I was reminded this behavior is here because of 
> #19994, which points out that this isn't implemented for upstart (still 
> used by trusty 14.04).
> 
> The jewel -> luminous upgrade process is currently documented here
> 
> 	http://docs.ceph.com/docs/master/release-notes/#upgrade-from-jewel-or-kraken
> 
> and basically has a step that says 'verify mgr daemons are running and, if 
> not, create some'.  Is that sufficient?

Probably.  Especially given that now ceph-mgr isn't optional (you'll end
up with "HEATH_ERR; no active mgr"), there's incentive for everyone to
configure some mgr daemons ;-)

More seriously: the systemd magic was able to ensure mgr was running
with Kraken, prior to the various deployment tools growing support for
deploying mgr, and (possibly?) prior to mgr appearing in the docs.

That magic still works fine and I personally don't mind keeping it, but
if it's going to result in inconsistent automatic behaviour across
distros, that's probably bad.  Maybe then for Luminous it's best now to
drop the magic and require everyone to explicitly configure mgr, as for
all other daemons?

Regards,

Tim
-- 
Tim Serong
Senior Clustering Engineer
SUSE
tserong@suse.com

Re: ceph-mgr setup magic

[Jens Rosenboom]

2017-06-29 5:14 GMT+00:00 Tim Serong <tserong@suse.com>:
> On 06/29/2017 05:04 AM, Sage Weil wrote:
>> There is some goofy behavior with the systemd unit files that tries to
>> make a new ceph-mgr daemon automagically appear next to the mons during a
>> cluster upgrade:
>>
>> - The ceph-mon@.service triggers ceph-mgr@ to start
>> - The ceph-mgr@.service tries to create a ceph-mgr key using whatever
>> mon key it can find
>>
>> This was intended to ease the upgrade path by auto-creating ceph-mgr
>> daemons along the way.
>
> It's also helpful for new installs; it means you don't actually have to
> explicitly configure mgr daemons in order to get a running cluster.  You
> just end up with as many mgrs as mons automatically.
>
> That said, it is somewhat obscure, and also can make it slightly
> irritating if you want to run mgrs on separate nodes (then you have to
> explicitly deploy mgr, and also either uninstall the ceph-mgr package
> from the mon nodes, or at least `systemctl mask ceph-mgr@$(hostname)` on
> those nodes.

IMO this is a good point, please remove the interdependency between
mon and mgr packages.

>> Do we really want this?  I was reminded this behavior is here because of
>> #19994, which points out that this isn't implemented for upstart (still
>> used by trusty 14.04).
>>
>> The jewel -> luminous upgrade process is currently documented here
>>
>>       http://docs.ceph.com/docs/master/release-notes/#upgrade-from-jewel-or-kraken
>>
>> and basically has a step that says 'verify mgr daemons are running and, if
>> not, create some'.  Is that sufficient?
>
> Probably.  Especially given that now ceph-mgr isn't optional (you'll end
> up with "HEATH_ERR; no active mgr"), there's incentive for everyone to
> configure some mgr daemons ;-)
>
> More seriously: the systemd magic was able to ensure mgr was running
> with Kraken, prior to the various deployment tools growing support for
> deploying mgr, and (possibly?) prior to mgr appearing in the docs.
>
> That magic still works fine and I personally don't mind keeping it, but
> if it's going to result in inconsistent automatic behaviour across
> distros, that's probably bad.  Maybe then for Luminous it's best now to
> drop the magic and require everyone to explicitly configure mgr, as for
> all other daemons?

+1 to removing magic, it is making things unnecessarily complicated
for people who want to deploy their cluster with some kind of
automation tool. I was happy that the ceph-create-keys service has
finally been dropped, please don't let it return through the backdoor.

Re: ceph-mgr setup magic

[John Spray]

On Wed, Jun 28, 2017 at 8:04 PM, Sage Weil <sage@newdream.net> wrote:
> There is some goofy behavior with the systemd unit files that tries to
> make a new ceph-mgr daemon automagically appear next to the mons during a
> cluster upgrade:
>
> - The ceph-mon@.service triggers ceph-mgr@ to start
> - The ceph-mgr@.service tries to create a ceph-mgr key using whatever
> mon key it can find
>
> This was intended to ease the upgrade path by auto-creating ceph-mgr
> daemons along the way.
>
> Do we really want this?  I was reminded this behavior is here because of
> #19994, which points out that this isn't implemented for upstart (still
> used by trusty 14.04).

I was thinking of it as a temporary thing for Kraken, because mgr
didn't really do anything so it seemed unreasonable to make people
jump through manual hoops at the time, and the tools support wasn't
there.

For luminous I'm kind of ambivalent about whether to remove it in the
name of cleanliness or leave it in in the name of convenience.

John

> The jewel -> luminous upgrade process is currently documented here
>
>         http://docs.ceph.com/docs/master/release-notes/#upgrade-from-jewel-or-kraken
>
> and basically has a step that says 'verify mgr daemons are running and, if
> not, create some'.  Is that sufficient?
>
> sage

Re: ceph-mgr setup magic

[Sage Weil]

On Thu, 29 Jun 2017, John Spray wrote:
> On Wed, Jun 28, 2017 at 8:04 PM, Sage Weil <sage@newdream.net> wrote:
> > There is some goofy behavior with the systemd unit files that tries to
> > make a new ceph-mgr daemon automagically appear next to the mons during a
> > cluster upgrade:
> >
> > - The ceph-mon@.service triggers ceph-mgr@ to start
> > - The ceph-mgr@.service tries to create a ceph-mgr key using whatever
> > mon key it can find
> >
> > This was intended to ease the upgrade path by auto-creating ceph-mgr
> > daemons along the way.
> >
> > Do we really want this?  I was reminded this behavior is here because of
> > #19994, which points out that this isn't implemented for upstart (still
> > used by trusty 14.04).
> 
> I was thinking of it as a temporary thing for Kraken, because mgr
> didn't really do anything so it seemed unreasonable to make people
> jump through manual hoops at the time, and the tools support wasn't
> there.
> 
> For luminous I'm kind of ambivalent about whether to remove it in the
> name of cleanliness or leave it in in the name of convenience.

I lean toward cleanliness and transparency. Major upgrades are a rare 
opportunity have the admin's full attention; let's take advantage of it.

	https://github.com/ceph/ceph/pull/16023

sage