subpackages for mgr modules

subpackages for mgr modules

[Tim Serong]

Hi All,

At some point, we're going to want to have (some of) mgr's python
modules shipped not as part of the ceph-mgr package, but as subpackages
that can be optionally installed (ceph-mgr-foo, ceph-mgr-bar, whatever).
 That's easy to do.  What I'm wondering about is having installation of
an mgr module subpackage automatically enable the loading of that module.

Ordinarily, to enable an mgr module that isn't enabled by default at
compile time, you'd set "mgr modules = foo bar" in the [mgr] section of
ceph.conf.  But that will get irritating.

We can't have mgr just load everything that exists (someone might want
to disable some modules, without necessarily uninstalling them).

A couple of approaches I thought of are:

1) Add a command line argument (or arguments) to ceph-mgr to list
modules to enable (or disable).  This could in turn be set by some
variable(s) in /etc/sysconfig/ceph.  Subpackages then tweak those
variables at install time.

2) Do something with symlinks; set up a directory structure something
like this:

/usr/lib64/ceph/mgr/enabled/
/usr/lib64/ceph/mgr/modules/
/usr/lib64/ceph/mgr/modules/fsstatus
/usr/lib64/ceph/mgr/modules/rest
/usr/lib64/ceph/mgr/modules/...

Then, any module you want to enable, create a symlink to it from the
"enabled" directory.  Subpackages could do this at install time.  mgr
iterates through that directory and loads anything present.

(Personally I rather prefer approach 2 to approach 1).

I'm sure there's other possibilities.  Thoughts, anyone?

Thanks,

Tim
-- 
Tim Serong
Senior Clustering Engineer
SUSE
tserong@suse.com

Re: subpackages for mgr modules

[Sage Weil]

On Fri, 19 May 2017, Tim Serong wrote:
> Hi All,
> 
> At some point, we're going to want to have (some of) mgr's python
> modules shipped not as part of the ceph-mgr package, but as subpackages
> that can be optionally installed (ceph-mgr-foo, ceph-mgr-bar, whatever).
>  That's easy to do.  What I'm wondering about is having installation of
> an mgr module subpackage automatically enable the loading of that module.
> 
> Ordinarily, to enable an mgr module that isn't enabled by default at
> compile time, you'd set "mgr modules = foo bar" in the [mgr] section of
> ceph.conf.  But that will get irritating.
> 
> We can't have mgr just load everything that exists (someone might want
> to disable some modules, without necessarily uninstalling them).
> 
> A couple of approaches I thought of are:
> 
> 1) Add a command line argument (or arguments) to ceph-mgr to list
> modules to enable (or disable).  This could in turn be set by some
> variable(s) in /etc/sysconfig/ceph.  Subpackages then tweak those
> variables at install time.
> 
> 2) Do something with symlinks; set up a directory structure something
> like this:
> 
> /usr/lib64/ceph/mgr/enabled/
> /usr/lib64/ceph/mgr/modules/
> /usr/lib64/ceph/mgr/modules/fsstatus
> /usr/lib64/ceph/mgr/modules/rest
> /usr/lib64/ceph/mgr/modules/...
> 
> Then, any module you want to enable, create a symlink to it from the
> "enabled" directory.  Subpackages could do this at install time.  mgr
> iterates through that directory and loads anything present.
> 
> (Personally I rather prefer approach 2 to approach 1).
> 
> I'm sure there's other possibilities.  Thoughts, anyone?

I like #2, too!  This is similar to the sites-enabled thing apache does.

We can make the config option blank (or something like *?) to indicate 
using the directory, otherwise the specified list?

sage

Re: subpackages for mgr modules

[Ken Dreyer]

On Fri, May 19, 2017 at 12:19 AM, Tim Serong <tserong@suse.com> wrote:
> We can't have mgr just load everything that exists (someone might want
> to disable some modules, without necessarily uninstalling them).

I'm curious why not?

Ceph is already hard to configure, and "install to enable / uninstall
to disable" would make it simpler to use.

- Ken

Re: subpackages for mgr modules

[kefu chai]

On Sat, May 20, 2017 at 2:25 AM, Ken Dreyer <kdreyer@redhat.com> wrote:
> On Fri, May 19, 2017 at 12:19 AM, Tim Serong <tserong@suse.com> wrote:
>> We can't have mgr just load everything that exists (someone might want
>> to disable some modules, without necessarily uninstalling them).

it does not. user can always enable/disable a plugin using "mgr_modules", right?

>
> I'm curious why not?
>
> Ceph is already hard to configure, and "install to enable / uninstall
> to disable" would make it simpler to use.


i agree, if he/she wants to disable it, he/she could just uninstall
the plugin. actually, i am planning to split the dashboard into a
separated package. so for example, mgr does not need to depend on
cherrypy.

>
> - Ken
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



-- 
Regards
Kefu Chai

Re: subpackages for mgr modules

[John Spray]

On Fri, May 19, 2017 at 7:25 PM, Ken Dreyer <kdreyer@redhat.com> wrote:
> On Fri, May 19, 2017 at 12:19 AM, Tim Serong <tserong@suse.com> wrote:
>> We can't have mgr just load everything that exists (someone might want
>> to disable some modules, without necessarily uninstalling them).
>
> I'm curious why not?

Perhaps "can't" is too strong a word, but the way it only loads
explicitly enabled modules is intentional.

The thinking was...
1. It's not a given that every module will be packaged independently.
That's not necessarily going to be useful in all cases -- if/when we
reach a point where some modules are so ubiquitous that we want them
everywhere, then having lots of separate packages may not be
desirable.  I'm thinking back to things like the
gstreamer-plugins-good/bad/ugly packaging, and the bajillion apache
sub-packages -- as a user I usually find this stuff really frustrating
because I have a big root partition and my life is simpler if I just
pre-emptively have all the modules (but that doesn't mean I want them
all running).
2. Which packages are installed on a system is usually something
controlled by an external tool, which tends to be different on
different sites/in diffferent products.  Consequently, the exact steps
required to turn off a module are not going to be consistent across
sites.  It is not a given that a support/admin person will know how to
turn off an individual package, unless they're also the person who
configured whatever orchestration tool was installing packages.
3. The "install to enable" convention is more welcome on some distros
than others.  For example, debian (I think) historically did/does this
for services, RPM distros historically didn't.

>
> Ceph is already hard to configure, and "install to enable / uninstall
> to disable" would make it simpler to use.

My preferred solution would be mon commands for toggling modules "ceph
mgr module [enable|disable] <module>", and a record in the MgrMap that
says which are enabled.  There would be a bit of extra data in the mgr
beacon message for the daemons to tell the MgrMonitor which modules
are available.  The MgrMonitor could also raise a health warning if
the mgr daemons have inconsistent plugins.

The major advantage of doing that is that all the mgr daemons see the
same setting.  That way, if someone enables a module on one of their
mgr nodes, they don't get a nasty shock when a failover happens and
the module is gone on the node they failed over to.

It would also let them do it without having to go and directly log
into each of their mgr nodes to make a change (as they currently have
to do with config files, or would also have to do with package
install/uninstall or symlinks).

John

>
> - Ken
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: subpackages for mgr modules

[Tim Serong]

On 05/22/2017 08:31 AM, John Spray wrote:
> On Fri, May 19, 2017 at 7:25 PM, Ken Dreyer <kdreyer@redhat.com> wrote:
>> On Fri, May 19, 2017 at 12:19 AM, Tim Serong <tserong@suse.com> wrote:
>>> We can't have mgr just load everything that exists (someone might want
>>> to disable some modules, without necessarily uninstalling them).
>>
>> I'm curious why not?
> 
> Perhaps "can't" is too strong a word, but the way it only loads
> explicitly enabled modules is intentional.
> 
> The thinking was...
> 1. It's not a given that every module will be packaged independently.
> That's not necessarily going to be useful in all cases -- if/when we
> reach a point where some modules are so ubiquitous that we want them
> everywhere, then having lots of separate packages may not be
> desirable.  I'm thinking back to things like the
> gstreamer-plugins-good/bad/ugly packaging, and the bajillion apache
> sub-packages -- as a user I usually find this stuff really frustrating
> because I have a big root partition and my life is simpler if I just
> pre-emptively have all the modules (but that doesn't mean I want them
> all running).
> 2. Which packages are installed on a system is usually something
> controlled by an external tool, which tends to be different on
> different sites/in diffferent products.  Consequently, the exact steps
> required to turn off a module are not going to be consistent across
> sites.  It is not a given that a support/admin person will know how to
> turn off an individual package, unless they're also the person who
> configured whatever orchestration tool was installing packages.

What John said.  I might not mind having installation of a package
trigger default enablement of something (depends on the service), but I
really don't like the idea of being required to uninstall something in
order to disable it.  In addition to the above, even if I *am* the
person immediately responsible for installing packages on a system and I
know what they all are, I can't necessarily guarantee that package repos
are always available, or that if available their content won't change.
This causes problems if I uninstall something to disable it, then later
want to re-enable it.

Here's a fun possible edge case: install a module, then later decide to
disable it temporarily by uninstalling it.  Later still, I want to
re-enable it, but meanwhile updates to ceph have been released in my
distro's update repo.  I can no longer install the module which matches
the version of ceph I have installed, and so need to upgrade my base
ceph packages too in order to install the module I want to enable.

> 3. The "install to enable" convention is more welcome on some distros
> than others.  For example, debian (I think) historically did/does this
> for services, RPM distros historically didn't.

Some services will be enabled by default if installed on openSUSE and
SLE now.  But not very many :)  See
https://build.opensuse.org/package/view_file/openSUSE:Factory/systemd-presets-branding-openSUSE/default-openSUSE.preset?expand=1

>>
>> Ceph is already hard to configure, and "install to enable / uninstall
>> to disable" would make it simpler to use.
> 
> My preferred solution would be mon commands for toggling modules "ceph
> mgr module [enable|disable] <module>", and a record in the MgrMap that
> says which are enabled.  There would be a bit of extra data in the mgr
> beacon message for the daemons to tell the MgrMonitor which modules
> are available.  The MgrMonitor could also raise a health warning if
> the mgr daemons have inconsistent plugins.
> 
> The major advantage of doing that is that all the mgr daemons see the
> same setting.  That way, if someone enables a module on one of their
> mgr nodes, they don't get a nasty shock when a failover happens and
> the module is gone on the node they failed over to.
> 
> It would also let them do it without having to go and directly log
> into each of their mgr nodes to make a change (as they currently have
> to do with config files, or would also have to do with package
> install/uninstall or symlinks).

Good idea, that's much better than my symlink thing.

Regards,

Tim
-- 
Tim Serong
Senior Clustering Engineer
SUSE
tserong@suse.com

Re: subpackages for mgr modules

[Willem Jan Withagen]

On 22-05-17 10:22, Tim Serong wrote:
> On 05/22/2017 08:31 AM, John Spray wrote:
>> On Fri, May 19, 2017 at 7:25 PM, Ken Dreyer <kdreyer@redhat.com> wrote:
>>> On Fri, May 19, 2017 at 12:19 AM, Tim Serong <tserong@suse.com> wrote:
>>>> We can't have mgr just load everything that exists (someone might want
>>>> to disable some modules, without necessarily uninstalling them).
>>>
>>> I'm curious why not?
>>
>> Perhaps "can't" is too strong a word, but the way it only loads
>> explicitly enabled modules is intentional.
>>
>> The thinking was...
>> 1. It's not a given that every module will be packaged independently.
>> That's not necessarily going to be useful in all cases -- if/when we
>> reach a point where some modules are so ubiquitous that we want them
>> everywhere, then having lots of separate packages may not be
>> desirable.  I'm thinking back to things like the
>> gstreamer-plugins-good/bad/ugly packaging, and the bajillion apache
>> sub-packages -- as a user I usually find this stuff really frustrating
>> because I have a big root partition and my life is simpler if I just
>> pre-emptively have all the modules (but that doesn't mean I want them
>> all running).
>> 2. Which packages are installed on a system is usually something
>> controlled by an external tool, which tends to be different on
>> different sites/in diffferent products.  Consequently, the exact steps
>> required to turn off a module are not going to be consistent across
>> sites.  It is not a given that a support/admin person will know how to
>> turn off an individual package, unless they're also the person who
>> configured whatever orchestration tool was installing packages.
> 
> What John said.  I might not mind having installation of a package
> trigger default enablement of something (depends on the service), but I
> really don't like the idea of being required to uninstall something in
> order to disable it.  In addition to the above, even if I *am* the
> person immediately responsible for installing packages on a system and I
> know what they all are, I can't necessarily guarantee that package repos
> are always available, or that if available their content won't change.
> This causes problems if I uninstall something to disable it, then later
> want to re-enable it.
> 
> Here's a fun possible edge case: install a module, then later decide to
> disable it temporarily by uninstalling it.  Later still, I want to
> re-enable it, but meanwhile updates to ceph have been released in my
> distro's update repo.  I can no longer install the module which matches
> the version of ceph I have installed, and so need to upgrade my base
> ceph packages too in order to install the module I want to enable.
> 
>> 3. The "install to enable" convention is more welcome on some distros
>> than others.  For example, debian (I think) historically did/does this
>> for services, RPM distros historically didn't.
> 
> Some services will be enabled by default if installed on openSUSE and
> SLE now.  But not very many :)  See
> https://build.opensuse.org/package/view_file/openSUSE:Factory/systemd-presets-branding-openSUSE/default-openSUSE.preset?expand=1
> 
>>>
>>> Ceph is already hard to configure, and "install to enable / uninstall
>>> to disable" would make it simpler to use.
>>
>> My preferred solution would be mon commands for toggling modules "ceph
>> mgr module [enable|disable] <module>", and a record in the MgrMap that
>> says which are enabled.  There would be a bit of extra data in the mgr
>> beacon message for the daemons to tell the MgrMonitor which modules
>> are available.  The MgrMonitor could also raise a health warning if
>> the mgr daemons have inconsistent plugins.
>>
>> The major advantage of doing that is that all the mgr daemons see the
>> same setting.  That way, if someone enables a module on one of their
>> mgr nodes, they don't get a nasty shock when a failover happens and
>> the module is gone on the node they failed over to.
>>
>> It would also let them do it without having to go and directly log
>> into each of their mgr nodes to make a change (as they currently have
>> to do with config files, or would also have to do with package
>> install/uninstall or symlinks).

Just 2 FreeBSD cts. It is the modus there that installation is just a
"get the software step", nothing is automagically enabled. As not to
surprise the sysadmin with servers running that he/she has not
configured to their liking.

Another valid point is versioning. FreeBSD has quarterly package
building releases. Once you cross that line for a new release. It could
be that you need to upgrade quite some tools to actually have a new and
compatible set of applications.
So unstall to disable will very likely get you into trouble.

Next to the fact that I have been caught in the module-jungle that John
references to more that once. Especially Apache/php/php-modules and
security issue were/are a notorious pain.

--WjW

Re: subpackages for mgr modules

[Ken Dreyer]

On Sun, May 21, 2017 at 4:31 PM, John Spray <jspray@redhat.com> wrote:
> My preferred solution would be mon commands for toggling modules "ceph
> mgr module [enable|disable] <module>", and a record in the MgrMap that
> says which are enabled.

Sure. It's just another command users have to do to get to a preferred state.

The classic response on these things has been "ceph setup is hard,
let's wrap it in ceph-ansible" (or whatever higher-level tool will do
all these tricky steps). Then ceph-ansible becomes more and more
complex, and I wonder whether the complexity could be addressed
elsewhere by simply scaling down the flexibility.

- Ken

Re: subpackages for mgr modules

[Alfredo Deza]

On Mon, May 22, 2017 at 11:45 AM, Ken Dreyer <kdreyer@redhat.com> wrote:
> On Sun, May 21, 2017 at 4:31 PM, John Spray <jspray@redhat.com> wrote:
>> My preferred solution would be mon commands for toggling modules "ceph
>> mgr module [enable|disable] <module>", and a record in the MgrMap that
>> says which are enabled.
>
> Sure. It's just another command users have to do to get to a preferred state.
>
> The classic response on these things has been "ceph setup is hard,
> let's wrap it in ceph-ansible" (or whatever higher-level tool will do
> all these tricky steps). Then ceph-ansible becomes more and more
> complex, and I wonder whether the complexity could be addressed
> elsewhere by simply scaling down the flexibility.

I can agree here with how these helpful commands tend to add
complexity on the deployment systems.

Maybe a good compromise would be the allow both? That is: installation
would enable a module, but that
could also be disabled (and re-enabled) by the CLI. That way, we
aren't forcing users to uninstall always and keeps
other systems like ceph-ansible lean.

>
> - Ken
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: subpackages for mgr modules

[John Spray]

On Wed, May 24, 2017 at 9:15 PM, Alfredo Deza <adeza@redhat.com> wrote:
> On Mon, May 22, 2017 at 11:45 AM, Ken Dreyer <kdreyer@redhat.com> wrote:
>> On Sun, May 21, 2017 at 4:31 PM, John Spray <jspray@redhat.com> wrote:
>>> My preferred solution would be mon commands for toggling modules "ceph
>>> mgr module [enable|disable] <module>", and a record in the MgrMap that
>>> says which are enabled.
>>
>> Sure. It's just another command users have to do to get to a preferred state.
>>
>> The classic response on these things has been "ceph setup is hard,
>> let's wrap it in ceph-ansible" (or whatever higher-level tool will do
>> all these tricky steps). Then ceph-ansible becomes more and more
>> complex, and I wonder whether the complexity could be addressed
>> elsewhere by simply scaling down the flexibility.
>
> I can agree here with how these helpful commands tend to add
> complexity on the deployment systems.
>
> Maybe a good compromise would be the allow both? That is: installation
> would enable a module, but that
> could also be disabled (and re-enabled) by the CLI. That way, we
> aren't forcing users to uninstall always and keeps
> other systems like ceph-ansible lean.

Oops, I wrote a reply to Ken but then failed to click send on it... it
was wordy so I'll be briefer now:

We do need a way to define what the default set of enabled modules
should be, so it probably makes sense to do that via a config setting
(perhaps just the existing config setting), and then let installation
tools give their preference via that interface too, killing two birds
with one stone.  I still think none of this should be triggered by
packaging.

If the user has never used the CLI to enable/disable things, then
updates to the config setting would continue to be authoritative.  If
a user has explicitly disabled a module using the CLI, then that would
take precedence over what was in the config setting.

That would mean that if an ansible-like tool wanted to definitely
enabled a module (even if the user had used the CLI to disable it)
then it would need to understand the CLI too.  However, if a user was
configuring everything using ansible, then ansible would never need to
touch the CLI.

John





>
>>
>> - Ken
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html