mon config storage (config-key vs config)

mon config storage (config-key vs config)

[Sage Weil]

I forgot to bring this up in the other thread, so staring a new one.

Currently the configs are stored in the config-key table, which means you 
can also see the source values with 'ceph config-key dump':

{
    "config/crush_chooseleaf_type": "0",
    "config/mds/debug_mds": "20",
    "config/mds/debug_mgrc": "20",
    "config/mds/debug_monc": "20",
...
    "config/osd/osd_scrub_load_threshold": "2000",
    "config/osd_pool_default_min_size": "1",
    "config/osd_pool_default_size": "1",
    "mgr/dashboard/x/server_port": "41879",
    "mgr/restful/keys/admin": "c28a8d35-282a-4330-aa61-c57e3d7f0ae8",
...
}

This was handy during development before 'ceph config *' were implemented.  
How, I'm not so sure it's the best path.

The first issue is that it's not easy to detect changes directly to 
config-key from the ConfigMonitor code.  (This is an implementation 
artifact; ConfigKeyService is not a normal PaxosService for reasons lost 
of the mists of time.)

More generally, I'm not sure if it *should* be visible and modifiable in a 
separate, different way.  A simple option is to simply move the 
config/* keys into a separate namespace.

On the other hand, config-key so far is only lightly used, by two in-tree 
things:

 - mgr module configs, which look like mgr/$module[/$mgrid]/$option
 - ceph-disk (and soon ceph-volume) put dmcrypt keys in dm-crypt/osd/*

(It's also been around forever, so users might be sticking all kinds of 
other stuff in there, who knows.)

Should we try to unify the general-purpose config-key storage with the 
well-structured config options in config/*?  Or are they simply two 
different things?  It *is* kind of nice that 'ceph config-key dump' shows 
you both mgr module configs and the daemon configs.  

Is the 'config/' prefix for the native ceph config options the part that 
is out of place?  For example, we could have something like

 mgr/dashboard/server_port = 7000
 mgr/debug_mgr = 20
 mgr.x/ms_type = simple

that squishes the ceph configs into the same key space as the module 
options.

This brings up another longstanding issue: most of our config options have 
a daemon type prefix in the name, e.g. osd_objectstore, mds_cache_size, or 
(more confusingly) things like mon_osd_full_ratio and 
osd_crush_chooseleaf_type (both mon options that affect the mon's initial 
OSDMap creation only).  Is there a way we can realistically rename some of 
these options, perhaps as part of the ceph.conf -> mon config transition?  
Then we would end up with configs more like

 mon/crush_chooseleaf_type = foo
 mon/initial_osd_full_ratio = ...
 mds/cache_size = ...
 osd/objectstore = bluestore

and so on.

sage

Re: mon config storage (config-key vs config)

[Jesse Williamson]

On Fri, 5 Jan 2018, Sage Weil wrote:

> mgr/dashboard/server_port = 7000

I like this style. There could be an "all" or "shared" magic root for 
items that are shared everywhere, and (though I admit this feels a bit 
half-baked, I'm thowing it out there) maybe something like:

shared/osd,mon/max_threads = foo

> OSDMap creation only).  Is there a way we can realistically rename some of
> these options, perhaps as part of the ceph.conf -> mon config transition?

I feel like if there are going to be visible configuration changes, then 
it's a good time to break things. Maybe it's possible to make the "old" 
names work for the next release, but give a deprecation warning? A script 
that can do migration?

-Jesse

Re: mon config storage (config-key vs config)

[Sage Weil]

On Fri, 5 Jan 2018, Jesse Williamson wrote:
> On Fri, 5 Jan 2018, Sage Weil wrote:
> 
> > mgr/dashboard/server_port = 7000
> 
> I like this style. There could be an "all" or "shared" magic root for items
> that are shared everywhere, and (though I admit this feels a bit half-baked,

The thing to keep in mind here is that the module options are pretty much 
freeform.  The normal config options have a pretty strict schema.

> I'm thowing it out there) maybe something like:
>
> shared/osd,mon/max_threads = foo

Maaaybe... this makes me nervous, and I'm not sure this has big benefits 
over just setting 2 separate options.

> > OSDMap creation only).  Is there a way we can realistically rename some of
> > these options, perhaps as part of the ceph.conf -> mon config transition?
> 
> I feel like if there are going to be visible configuration changes, then it's
> a good time to break things. Maybe it's possible to make the "old" names work
> for the next release, but give a deprecation warning? A script that can do
> migration?

I think we could annotate the options with "old" names so they get parsed 
properly.  And deprecate them.  And the process to assimilate 
ceph.conf settings into the mon config can do the remapping for you.  And 
once options are in the mon in the future we can rename them transparently 
(if we choose).  We could even raise health warnings if deprecated options 
are in use (in config files) that are about to be killed off.

So I think the question(s) are (1) is it worth investing the time to go 
through the options and rename them, and (2) assuming we did that, is 
there a more sane/intuitive way to mix the mgr module options with 
the traditional options?

sage

Re: mon config storage (config-key vs config)

[Piotr Dałek]

On 18-01-05 10:13 PM, Sage Weil wrote:
> This brings up another longstanding issue: most of our config options have
> a daemon type prefix in the name, e.g. osd_objectstore, mds_cache_size, or
> (more confusingly) things like mon_osd_full_ratio and
> osd_crush_chooseleaf_type (both mon options that affect the mon's initial
> OSDMap creation only).  Is there a way we can realistically rename some of
> these options, perhaps as part of the ceph.conf -> mon config transition?
> Then we would end up with configs more like
> 
>   mon/crush_chooseleaf_type = foo
>   mon/initial_osd_full_ratio = ...
>   mds/cache_size = ...
>   osd/objectstore = bluestore
> 
> and so on.

Maybe it's worth implementing some sort of automatic renaming and outputting 
deprecation warnings? I imagine it'd be easier for everyone if both forms 
would work just fine, but the old form would spew a bunch of warnings, like

* WARNING: "osd objectstore" is deprecated, you should change your config to 
use "osd/objectstore" instead.

Warnings always annoy admins and they'll clean them up eventually, the other 
benefit is that the sky won't fall for anybody upgrading to new config.

-- 
Piotr Dałek
piotr.dalek@corp.ovh.com
https://www.ovh.com/us/

Re: mon config storage (config-key vs config)

[John Spray]

On Fri, Jan 5, 2018 at 9:13 PM, Sage Weil <sweil@redhat.com> wrote:
> I forgot to bring this up in the other thread, so staring a new one.
>
> Currently the configs are stored in the config-key table, which means you
> can also see the source values with 'ceph config-key dump':
>
> {
>     "config/crush_chooseleaf_type": "0",
>     "config/mds/debug_mds": "20",
>     "config/mds/debug_mgrc": "20",
>     "config/mds/debug_monc": "20",
> ...
>     "config/osd/osd_scrub_load_threshold": "2000",
>     "config/osd_pool_default_min_size": "1",
>     "config/osd_pool_default_size": "1",
>     "mgr/dashboard/x/server_port": "41879",
>     "mgr/restful/keys/admin": "c28a8d35-282a-4330-aa61-c57e3d7f0ae8",
> ...
> }
>
> This was handy during development before 'ceph config *' were implemented.
> How, I'm not so sure it's the best path.
>
> The first issue is that it's not easy to detect changes directly to
> config-key from the ConfigMonitor code.  (This is an implementation
> artifact; ConfigKeyService is not a normal PaxosService for reasons lost
> of the mists of time.)
>
> More generally, I'm not sure if it *should* be visible and modifiable in a
> separate, different way.  A simple option is to simply move the
> config/* keys into a separate namespace.
>
> On the other hand, config-key so far is only lightly used, by two in-tree
> things:
>
>  - mgr module configs, which look like mgr/$module[/$mgrid]/$option
>  - ceph-disk (and soon ceph-volume) put dmcrypt keys in dm-crypt/osd/*
>
> (It's also been around forever, so users might be sticking all kinds of
> other stuff in there, who knows.)

I like the idea of keeping a schema-less KV store exposed by the mon,
but perhaps the config-key command name was always a bit awkward: it's
not really config keys, it's just whatever data the user is shoving in
there :-)  Not that I have an immediate thought on a better naming.

We do need some separation though, so that we can complain to the user
if they try to set a config value that doesn't exist in the schema,
rather than accepting typos on the assumption that they weren't meant
to be config settings.  Strict validation would mean either
requiring/expecting that the mons are >= the version of the other
daemons, or having a force flag that enables setting config options
that the mon doesn't.

> Should we try to unify the general-purpose config-key storage with the
> well-structured config options in config/*?  Or are they simply two
> different things?  It *is* kind of nice that 'ceph config-key dump' shows
> you both mgr module configs and the daemon configs.

I was hoping to unify the storage and transmission parts, so that the
mgr could subscribe for config updates using the same wiring as normal
config, even if the values in question are not part of the main config
schema.  At the moment the mgrs have some awkward caveats around when
they will actually see modified config options if they weren't the
active mgr that did the modification.

> Is the 'config/' prefix for the native ceph config options the part that
> is out of place?  For example, we could have something like
>
>  mgr/dashboard/server_port = 7000
>  mgr/debug_mgr = 20
>  mgr.x/ms_type = simple
>
> that squishes the ceph configs into the same key space as the module
> options.

Perhaps all the flexible module config should be under a prefix like
"mgr/modules/" so that we can still strictly validate everything
outside that prefix?

> This brings up another longstanding issue: most of our config options have
> a daemon type prefix in the name, e.g. osd_objectstore, mds_cache_size, or
> (more confusingly) things like mon_osd_full_ratio and
> osd_crush_chooseleaf_type (both mon options that affect the mon's initial
> OSDMap creation only).  Is there a way we can realistically rename some of
> these options, perhaps as part of the ceph.conf -> mon config transition?
> Then we would end up with configs more like
>
>  mon/crush_chooseleaf_type = foo
>  mon/initial_osd_full_ratio = ...
>  mds/cache_size = ...
>  osd/objectstore = bluestore
>
> and so on.

In principle, the config metadata is meant to remove the need to try
and express this kind of thing via the variable name.  There are quite
a few things that only make sense for a subset of daemons (e.g. the
osdc_* stuff), so there's not really any way to express that in the
name.

For those things that do have a clear mapping to a particular daemon,
I'm not sure "mds/cache_size" buys us much over "mds_cache_size".

John

>
> sage
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: mon config storage (config-key vs config)

[Jesse Williamson]

[-- Attachment #1: Type: text/plain, Size: 228 bytes --]

On Mon, 8 Jan 2018, Piotr Dałek wrote:

> * WARNING: "osd objectstore" is deprecated, you should change your config to 
> use "osd/objectstore" instead.

If there is a 1:1 correspondence between names, I like this idea.

-Jesse

Re: mon config storage (config-key vs config)

[Sage Weil]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 2097 bytes --]

On Wed, 10 Jan 2018, Jesse Williamson wrote:
> On Mon, 8 Jan 2018, Piotr Dałek wrote:
> 
> > * WARNING: "osd objectstore" is deprecated, you should change your config to
> > use "osd/objectstore" instead.
> 
> If there is a 1:1 correspondence between names, I like this idea.

I think it comes down to this.  We have these two options (at least):

1. Current branch

mgr/$module/$option = $value        # mgr module settings, pretty freeform
dm-crypt/osd/$uuid/... = ...        # dm-crypt secrets
config/$option = $value             # global
config/$type/$option = $value       # by type
config/$type.$id/$option = $value   # by daemon
config/.../$type:$val/$option = ..  # filtered by crush, osd class

- config stuff is very clearly separate
- everything not under mgr/ dm-crypt/ config/ above is available for other 
uses


2. Combined schema

mgr/$module/$option = $value        # mgr module settings, pretty freeform
dm-crypt/osd/$uuid/... = ...        # dm-crypt secrets
global/$option = $value             # global
$type/$option = $value              # by type
$type.$id/$option = $value          # by daemon
$type[.$id]/$ctype:$val/$option =   # filtered by crush, osd class

This would result in keys like

mgr/dashboard/...
mgr/debug_mgr
mgr.x/debug_mgr
global/*
osd/*
mds/*
osd.123/

and so on.

- everything exists closer to the same level, looks more unified
- config stuff *still* has distinct prefixes, so we can tell what is a 
traditional config option and what is not
- everything outside of $daemontype and 'global' prefixes (and dm-crypt) 
is still available for other uses.


I'm ignoring the part where we start renaming options (osd_data -> 
osd[.123]/data) -- that is an independent problem to solve IMO.

The only real technical downside I see to #2 above is that the 
laod_config() method in ConfigMonitor has to scan more config-key keys 
than it did before, and be smart enough to ignore mgr/$string/$option keys 
where $string doesn't have a '$this:$that' mask format (should be okay 
since mgr modules can't contain : in the name).

What does everyone think?

sage