mon daemon makes authentication side effects on startup

mon daemon makes authentication side effects on startup

[Owen Synge]

Dear all,

This is in my opinion is clearly a bug, but I raise it in the mailing
list as I expect all admins of ceph will strongly agree, that this makes
ceph simpler, but developers may feel that since it requires changes to
more than one repo its not worth doing.

When ever you start the mon demon as a side effect the admin, osd, rgw
and mds keys are created as a side effect if the mds keyring is not
existing.

In the systemV and systemd init scripts (at least) we have a side
effect, that should be removed in my opinion, (or worse in my
alternatively correctly documented.)

This is a deployment layer violation, in my opinion, and it requires
considerably more documentation, (and on my part also code) to keep this
side effect than remove it.

usecases for removing this are:

(0) A ceph cluster should be able to be installed in any order. With the
current behavior if the mds, rgw, or osd nodes are deployed first (along
with the boot strap keyrings), the mon created must have all keys for
the admin, mds-bootstrap, rgw-bootstrap, and osd-boostrap deployed in
the correct path before the mon can safely be started, even if the
cluster does not need the mds or rgw service's.

(1) It is unfriendly to configuration being stored on the configuration
server as the server needs to be updated with the values from the
configured node keys, when people might want to store these keys centrally.

(2) Assuming the admin, rgw-bootstrap, mds-bootstrap and osd-boostrap
keys are always installed on all mon nodes is clearly increasing the
distribution of keys where they might not be needed. Hence reducing
security.

(3) Using the current model adds an extra complication that these keys
then need to be distributed to each node from the configured node, if
generated by starting the mon, and not from the configuration server.

(4) If you wish to use a more devops approach, and generate keys
explicitly all the keys must be installed on all mon nodes before the
mon nodes are started.

(4.1) As a side effect we need to document why admins need the
mds-bootstrap keyring when they dont want this service it is confusing,
and requires an unnecessary process of migrating all keys to the
explicitly desired keys.

(5) I am developing a simple python library to configure ceph on each
node independently of all others, (think of it as a parallelism version
of ceph-deploy that can be called by any config management system) but
with the current side effect behavior starting the mon needs to fail if
the mds-bootstrap keyring is not created on the mon nodes before
starting the mon, otherwise we get ordering complications.

(5) The side effect is confusing, as no one expects this side effect,
hence this leads to ceph seeming complex to a first time user.

(6) I feel it is the responsibility of configuration management not the
mon demon to request creating these keys.

(7) I dont think this is clearly documented, hence this leads to ceph
seeming complex to a first time user.

(8) As more services like mds and rgw get added to ceph the problem gets
multiplied.

(9) Adding one more step to the by hand installation will clarify the
authentication process. This extra step would simply be:

   /usr/sbin/ceph-create-keys --cluster ${CLUSTER} --id ${MON_NAME}

This is simpler and clearer than documenting the side effect.

consequences:

(1) Each configuration system which depends upon this behavior will need
to be modified to call the single command on each mon:

   /usr/sbin/ceph-create-keys --cluster ${CLUSTER} --id ${MON_NAME}

Here is a fix for ceph-deploy:

https://github.com/SUSE/ceph-deploy/commit/58b030dbe0a964b32f1fbc9a3762e64dd74bf50c

I assume other solutions will be easy to fix too.

The systemd file in question, is
"/usr/lib/systemd/system/ceph-create-keys@.service" and should be removed.

This will simplify the salt configuration module documentation
considerably, and if this is not done the salt module will need to add a
requirement on the mds keyring being created before the mon can be created.

the systemd file looks as follows:

    [Unit]
    Description=Ceph cluster key creator task

    # the last key created is the mds bootstrap key -- look for that.
    ConditionPathExists=!/var/lib/ceph/bootstrap-mds/ceph.keyring

    [Service]
    EnvironmentFile=-/etc/sysconfig/ceph
    Environment=CLUSTER=ceph
    ExecStart=/usr/sbin/ceph-create-keys --cluster ${CLUSTER} --id %i

as you can see the side effect is blocked if the file

    /var/lib/ceph/bootstrap-mds/ceph.keyring

already exists, which is just more to document.

Hoping that you all agree

Owen Synge

Re: mon daemon makes authentication side effects on startup

[John Spray]

On Tue, Apr 5, 2016 at 12:56 PM, Owen Synge <osynge@suse.com> wrote:
> Dear all,
>
> This is in my opinion is clearly a bug, but I raise it in the mailing
> list as I expect all admins of ceph will strongly agree, that this makes
> ceph simpler, but developers may feel that since it requires changes to
> more than one repo its not worth doing.
>
> When ever you start the mon demon as a side effect the admin, osd, rgw
> and mds keys are created as a side effect if the mds keyring is not
> existing.
>
> In the systemV and systemd init scripts (at least) we have a side
> effect, that should be removed in my opinion, (or worse in my
> alternatively correctly documented.)
>
> This is a deployment layer violation, in my opinion, and it requires
> considerably more documentation, (and on my part also code) to keep this
> side effect than remove it.
>
> usecases for removing this are:
>
> (0) A ceph cluster should be able to be installed in any order. With the
> current behavior if the mds, rgw, or osd nodes are deployed first (along
> with the boot strap keyrings), the mon created must have all keys for
> the admin, mds-bootstrap, rgw-bootstrap, and osd-boostrap deployed in
> the correct path before the mon can safely be started, even if the
> cluster does not need the mds or rgw service's.
>
> (1) It is unfriendly to configuration being stored on the configuration
> server as the server needs to be updated with the values from the
> configured node keys, when people might want to store these keys centrally.

Is there anything stopping you from pre-generating all your keys
(including bootstrap etc) using ceph-authtool, and then including
those keys in the keyring file that you pass into mons when you create
them?  That way, you could create your keys and store them in your
config management tool, all before starting any mons.

> (2) Assuming the admin, rgw-bootstrap, mds-bootstrap and osd-boostrap
> keys are always installed on all mon nodes is clearly increasing the
> distribution of keys where they might not be needed. Hence reducing
> security.

I think this is a bit of a subjective balance between security and
convenience.  Currently we don't have to explain to folks how to get
their admin key into /etc/ceph, because it just pops out when the mon
starts.  The security sacrifice is small because the keys are
remaining in the same domain (same host, same UID) as the mon's
/var/lib, which is already a master key.

For sites/systems that don't want to write out key files, it would be
straightforward to add a configuration option to ceph-create-keys so
that it would not write these out to disk.

Once you had that option, and were also pre-generating all your keys
with ceph-authtool, I think you'd have the behaviour you wanted
without anyone else having to update for a change to the mon's default
behaviour.

John

> (3) Using the current model adds an extra complication that these keys
> then need to be distributed to each node from the configured node, if
> generated by starting the mon, and not from the configuration server.
>
> (4) If you wish to use a more devops approach, and generate keys
> explicitly all the keys must be installed on all mon nodes before the
> mon nodes are started.
>
> (4.1) As a side effect we need to document why admins need the
> mds-bootstrap keyring when they dont want this service it is confusing,
> and requires an unnecessary process of migrating all keys to the
> explicitly desired keys.
>
> (5) I am developing a simple python library to configure ceph on each
> node independently of all others, (think of it as a parallelism version
> of ceph-deploy that can be called by any config management system) but
> with the current side effect behavior starting the mon needs to fail if
> the mds-bootstrap keyring is not created on the mon nodes before
> starting the mon, otherwise we get ordering complications.
>
> (5) The side effect is confusing, as no one expects this side effect,
> hence this leads to ceph seeming complex to a first time user.
>
> (6) I feel it is the responsibility of configuration management not the
> mon demon to request creating these keys.
>
> (7) I dont think this is clearly documented, hence this leads to ceph
> seeming complex to a first time user.
>
> (8) As more services like mds and rgw get added to ceph the problem gets
> multiplied.
>
> (9) Adding one more step to the by hand installation will clarify the
> authentication process. This extra step would simply be:
>
>    /usr/sbin/ceph-create-keys --cluster ${CLUSTER} --id ${MON_NAME}
>
> This is simpler and clearer than documenting the side effect.
>
> consequences:
>
> (1) Each configuration system which depends upon this behavior will need
> to be modified to call the single command on each mon:
>
>    /usr/sbin/ceph-create-keys --cluster ${CLUSTER} --id ${MON_NAME}
>
> Here is a fix for ceph-deploy:
>
> https://github.com/SUSE/ceph-deploy/commit/58b030dbe0a964b32f1fbc9a3762e64dd74bf50c
>
> I assume other solutions will be easy to fix too.
>
> The systemd file in question, is
> "/usr/lib/systemd/system/ceph-create-keys@.service" and should be removed.
>
> This will simplify the salt configuration module documentation
> considerably, and if this is not done the salt module will need to add a
> requirement on the mds keyring being created before the mon can be created.
>
> the systemd file looks as follows:
>
>     [Unit]
>     Description=Ceph cluster key creator task
>
>     # the last key created is the mds bootstrap key -- look for that.
>     ConditionPathExists=!/var/lib/ceph/bootstrap-mds/ceph.keyring
>
>     [Service]
>     EnvironmentFile=-/etc/sysconfig/ceph
>     Environment=CLUSTER=ceph
>     ExecStart=/usr/sbin/ceph-create-keys --cluster ${CLUSTER} --id %i
>
> as you can see the side effect is blocked if the file
>
>     /var/lib/ceph/bootstrap-mds/ceph.keyring
>
> already exists, which is just more to document.
>
> Hoping that you all agree
>
> Owen Synge
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: mon daemon makes authentication side effects on startup

[Gregory Farnum]

I think you're fundamentally understanding how these keys come into
existence. They aren't generated randomly on the local monitor; it
uses get-or-create in order to fetch them (and create them if they
don't already exist).

So maybe it's difficult to pre-generate your own keys and plug them
into the system (I don't remember where the initial values come from
in standard deployment scenarios), but once they're set up you don't
need to carefully install your values on all the monitor nodes — they
will fetch the correct values from the monitor cluster.
The coordination problem here is not really any different than that of
making sure your monitors are all part of the "mon initial members"
config option, btw. Which you need to solve or else you're liable to
have them coming up and creating independent monitor clusters and
going haywire.
-Greg

On Tue, Apr 5, 2016 at 4:56 AM, Owen Synge <osynge@suse.com> wrote:
> Dear all,
>
> This is in my opinion is clearly a bug, but I raise it in the mailing
> list as I expect all admins of ceph will strongly agree, that this makes
> ceph simpler, but developers may feel that since it requires changes to
> more than one repo its not worth doing.
>
> When ever you start the mon demon as a side effect the admin, osd, rgw
> and mds keys are created as a side effect if the mds keyring is not
> existing.
>
> In the systemV and systemd init scripts (at least) we have a side
> effect, that should be removed in my opinion, (or worse in my
> alternatively correctly documented.)
>
> This is a deployment layer violation, in my opinion, and it requires
> considerably more documentation, (and on my part also code) to keep this
> side effect than remove it.
>
> usecases for removing this are:
>
> (0) A ceph cluster should be able to be installed in any order. With the
> current behavior if the mds, rgw, or osd nodes are deployed first (along
> with the boot strap keyrings), the mon created must have all keys for
> the admin, mds-bootstrap, rgw-bootstrap, and osd-boostrap deployed in
> the correct path before the mon can safely be started, even if the
> cluster does not need the mds or rgw service's.
>
> (1) It is unfriendly to configuration being stored on the configuration
> server as the server needs to be updated with the values from the
> configured node keys, when people might want to store these keys centrally.
>
> (2) Assuming the admin, rgw-bootstrap, mds-bootstrap and osd-boostrap
> keys are always installed on all mon nodes is clearly increasing the
> distribution of keys where they might not be needed. Hence reducing
> security.
>
> (3) Using the current model adds an extra complication that these keys
> then need to be distributed to each node from the configured node, if
> generated by starting the mon, and not from the configuration server.
>
> (4) If you wish to use a more devops approach, and generate keys
> explicitly all the keys must be installed on all mon nodes before the
> mon nodes are started.
>
> (4.1) As a side effect we need to document why admins need the
> mds-bootstrap keyring when they dont want this service it is confusing,
> and requires an unnecessary process of migrating all keys to the
> explicitly desired keys.
>
> (5) I am developing a simple python library to configure ceph on each
> node independently of all others, (think of it as a parallelism version
> of ceph-deploy that can be called by any config management system) but
> with the current side effect behavior starting the mon needs to fail if
> the mds-bootstrap keyring is not created on the mon nodes before
> starting the mon, otherwise we get ordering complications.
>
> (5) The side effect is confusing, as no one expects this side effect,
> hence this leads to ceph seeming complex to a first time user.
>
> (6) I feel it is the responsibility of configuration management not the
> mon demon to request creating these keys.
>
> (7) I dont think this is clearly documented, hence this leads to ceph
> seeming complex to a first time user.
>
> (8) As more services like mds and rgw get added to ceph the problem gets
> multiplied.
>
> (9) Adding one more step to the by hand installation will clarify the
> authentication process. This extra step would simply be:
>
>    /usr/sbin/ceph-create-keys --cluster ${CLUSTER} --id ${MON_NAME}
>
> This is simpler and clearer than documenting the side effect.
>
> consequences:
>
> (1) Each configuration system which depends upon this behavior will need
> to be modified to call the single command on each mon:
>
>    /usr/sbin/ceph-create-keys --cluster ${CLUSTER} --id ${MON_NAME}
>
> Here is a fix for ceph-deploy:
>
> https://github.com/SUSE/ceph-deploy/commit/58b030dbe0a964b32f1fbc9a3762e64dd74bf50c
>
> I assume other solutions will be easy to fix too.
>
> The systemd file in question, is
> "/usr/lib/systemd/system/ceph-create-keys@.service" and should be removed.
>
> This will simplify the salt configuration module documentation
> considerably, and if this is not done the salt module will need to add a
> requirement on the mds keyring being created before the mon can be created.
>
> the systemd file looks as follows:
>
>     [Unit]
>     Description=Ceph cluster key creator task
>
>     # the last key created is the mds bootstrap key -- look for that.
>     ConditionPathExists=!/var/lib/ceph/bootstrap-mds/ceph.keyring
>
>     [Service]
>     EnvironmentFile=-/etc/sysconfig/ceph
>     Environment=CLUSTER=ceph
>     ExecStart=/usr/sbin/ceph-create-keys --cluster ${CLUSTER} --id %i
>
> as you can see the side effect is blocked if the file
>
>     /var/lib/ceph/bootstrap-mds/ceph.keyring
>
> already exists, which is just more to document.
>
> Hoping that you all agree
>
> Owen Synge
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: mon daemon makes authentication side effects on startup

[Owen Synge]

Dear John,

Thanks for the reply, I hope my answer does not sound to abrupt.

On 04/05/2016 08:35 PM, John Spray wrote:
> On Tue, Apr 5, 2016 at 12:56 PM, Owen Synge <osynge@suse.com> wrote:
>> Dear all,
>>
>> This is in my opinion is clearly a bug, but I raise it in the mailing
>> list as I expect all admins of ceph will strongly agree, that this makes
>> ceph simpler, but developers may feel that since it requires changes to
>> more than one repo its not worth doing.
>>
>> When ever you start the mon demon as a side effect the admin, osd, rgw
>> and mds keys are created as a side effect if the mds keyring is not
>> existing.
>>
>> In the systemV and systemd init scripts (at least) we have a side
>> effect, that should be removed in my opinion, (or worse in my
>> alternatively correctly documented.)
>>
>> This is a deployment layer violation, in my opinion, and it requires
>> considerably more documentation, (and on my part also code) to keep this
>> side effect than remove it.
>>
>> usecases for removing this are:
>>
>> (0) A ceph cluster should be able to be installed in any order. With the
>> current behavior if the mds, rgw, or osd nodes are deployed first (along
>> with the boot strap keyrings), the mon created must have all keys for
>> the admin, mds-bootstrap, rgw-bootstrap, and osd-boostrap deployed in
>> the correct path before the mon can safely be started, even if the
>> cluster does not need the mds or rgw service's.
>>
>> (1) It is unfriendly to configuration being stored on the configuration
>> server as the server needs to be updated with the values from the
>> configured node keys, when people might want to store these keys centrally.
> 
> Is there anything stopping you from pre-generating all your keys
> (including bootstrap etc) using ceph-authtool, and then including
> those keys in the keyring file that you pass into mons when you create
> them?  That way, you could create your keys and store them in your
> config management tool, all before starting any mons.

Currently you can not use pre-generated keys and configure nodes in any
order (point 0) using secrets from the configuration server (point 1)
for brevity I skip other use cases. We can remove this restraint very
simply, if we prevent ceph-create-keys being run on mon nodes at the
booting stage of mon's by our choice of methods, which I propose is best
done by removing ceph-create-keys from the init scripts.

To answer your question, yes, you can have a fully functioning cluster
with pre-generated keys, without using ceph-create-keys at any stage.
pre-generating all your keys and making a running cluster is possible
and not very hard.

What is needed is to create a "booting the mon keyring", by importing
the mon and the admin keys. Then use this when creating the mon.

For reference the two steps to set up the mon nodes are:

(1) creating the boot key:

ceph-authtool \
  --create-keyring ${path_booting_keyring} \
  --import-keyring ${path_mon_keyring}

ceph-authtool \
  ${path_booting_keyring} \
  --import-keyring ${path_admin_keyring}

(it seems you can also just concatenate both keyrings, but I expect
using ceph-authtool is safer in the long term)

(2) creating the mon:

ceph-mon \
 --mkfs \
 -i ${hostname} \
 --monmap ${path_monmap}
 --keyring ${booting_key_path}

For reference I also include the step of generating the admin and mon
keyrings which can be done before configuring any service:

(0) Creating admin and mon keys:

ceph-authtool \
-n client.admin \
--create-keyring ${path_admin_keyring} \
--gen-key \
--cap mds 'allow *' \
--cap mon 'allow *' \
--cap osd 'allow *'

ceph-authtool \
-n mon. \
--create-keyring ${path_mon_keyring} \
--gen-key \
--set-uid=0 \
--cap mon 'allow *'

>> (2) Assuming the admin, rgw-bootstrap, mds-bootstrap and osd-boostrap
>> keys are always installed on all mon nodes is clearly increasing the
>> distribution of keys where they might not be needed. Hence reducing
>> security.
> 
> I think this is a bit of a subjective balance between security and
> convenience. 

I do not agree the current behavior is convenient for the admin who runs
a ceph cluster.

For the admin points (0), (1), (2), (3), (4), (4.1), (5), (6), (7) are
currently an issue, and lead to ceph being both mysterious and magical,
with all the fear that gives an admin.

Only if you are a developer of ceph, is the current behavior more
convenient, but the convenience only extends to avoiding running one
command in setting up your test environment namely running:

 ceph-create-keys --cluster ${CLUSTER} --id ${MON_NAME}

Having this as a separate step to booting a mon adds clarity to the
"magic", and is a simple one liner command line operation, which can be
skipped when not wanted, rather than running under some conditions
whenever you boot a mon as is the current behavior.

> Currently we don't have to explain to folks how to get
> their admin key into /etc/ceph, because it just pops out when the mon
> starts.

Yes, I object to the side effect "magic" and the consequences of the
"magic", and this "magic" is completely undocumented, and that is far
worse than not documenting a path.

   ceph-create-keys --cluster ${CLUSTER} --id ${MON_NAME}

You still would not have to explain the paths, if the above was a
separate step, ie creating the keys, I would have no problem with this
being a separate step, since their is an alternative as detailed above.

Ideally I also feel the process I outlined should also be documented as
an alternative to running ceph-create-keys.

> The security sacrifice is small because the keys are
> remaining in the same domain (same host, same UID) as the mon's
> /var/lib, which is already a master key.

I agree that the enhancement is security is small.

In a high security environment, we could have the mon keyring deleted
after startup, by the mon.

I am not advocating this for anyone but the very security conscious, but
it is definitely possible that some ceph users with highly secure
environments could find this desirable, assuming you can not run
ceph-create-keys without access to the mon keyring, which I expect is
true (but have not tested).

Such an approach would be even more work with the current side effect of
creating all the missing keys each time the mon boots.

> For sites/systems that don't want to write out key files, it would be
> straightforward to add a configuration option to ceph-create-keys so
> that it would not write these out to disk.

Agreed.

But my objection is the side effect, of running ceph-create-keys on mon
boot up, not that ceph-create-keys as a tool exists, or that the tool
ceph-create-keys writes out all the keys, although having this as
optional may be desirable for some deployments.

> Once you had that option, and were also pre-generating all your keys
> with ceph-authtool, I think you'd have the behaviour you wanted
> without anyone else having to update for a change to the mon's default
> behaviour.

Thank you for trying hard to preserve the current behavior and address
this issue, but no I don’t have the behavior I want for all the other
points I have raised.

I accept only for point (2) your proposal of not writing the keys to
disk, is a solution, but I caution that this potential solution is even
more confusing than the current behavior, and my proposed solution of
removing this step irrespective of writing to disk or not, from the boot
up of a mon is much more intuitive for an admin coming to ceph.

What I want is that booting a mon does not always run or under some
conditions run, ceph-create-keys after booting in an automagical way.

We still have a lot of use cases I have raised, as to why the current
behavior is magical, and hence creates fear in system admins.

> 
> John

Thank you for your reply.

(I will get to Greg's kind reply tomorrow is its now 01:00 AM)

Best wishes.

Owen

>> (3) Using the current model adds an extra complication that these keys
>> then need to be distributed to each node from the configured node, if
>> generated by starting the mon, and not from the configuration server.
>>
>> (4) If you wish to use a more devops approach, and generate keys
>> explicitly all the keys must be installed on all mon nodes before the
>> mon nodes are started.
>>
>> (4.1) As a side effect we need to document why admins need the
>> mds-bootstrap keyring when they dont want this service it is confusing,
>> and requires an unnecessary process of migrating all keys to the
>> explicitly desired keys.
>>
>> (5) I am developing a simple python library to configure ceph on each
>> node independently of all others, (think of it as a parallelism version
>> of ceph-deploy that can be called by any config management system) but
>> with the current side effect behavior starting the mon needs to fail if
>> the mds-bootstrap keyring is not created on the mon nodes before
>> starting the mon, otherwise we get ordering complications.
>>
>> (5) The side effect is confusing, as no one expects this side effect,
>> hence this leads to ceph seeming complex to a first time user.
>>
>> (6) I feel it is the responsibility of configuration management not the
>> mon demon to request creating these keys.
>>
>> (7) I dont think this is clearly documented, hence this leads to ceph
>> seeming complex to a first time user.
>>
>> (8) As more services like mds and rgw get added to ceph the problem gets
>> multiplied.
>>
>> (9) Adding one more step to the by hand installation will clarify the
>> authentication process. This extra step would simply be:
>>
>>    /usr/sbin/ceph-create-keys --cluster ${CLUSTER} --id ${MON_NAME}
>>
>> This is simpler and clearer than documenting the side effect.
>>
>> consequences:
>>
>> (1) Each configuration system which depends upon this behavior will need
>> to be modified to call the single command on each mon:
>>
>>    /usr/sbin/ceph-create-keys --cluster ${CLUSTER} --id ${MON_NAME}
>>
>> Here is a fix for ceph-deploy:
>>
>> https://github.com/SUSE/ceph-deploy/commit/58b030dbe0a964b32f1fbc9a3762e64dd74bf50c
>>
>> I assume other solutions will be easy to fix too.
>>
>> The systemd file in question, is
>> "/usr/lib/systemd/system/ceph-create-keys@.service" and should be removed.
>>
>> This will simplify the salt configuration module documentation
>> considerably, and if this is not done the salt module will need to add a
>> requirement on the mds keyring being created before the mon can be created.
>>
>> the systemd file looks as follows:
>>
>>     [Unit]
>>     Description=Ceph cluster key creator task
>>
>>     # the last key created is the mds bootstrap key -- look for that.
>>     ConditionPathExists=!/var/lib/ceph/bootstrap-mds/ceph.keyring
>>
>>     [Service]
>>     EnvironmentFile=-/etc/sysconfig/ceph
>>     Environment=CLUSTER=ceph
>>     ExecStart=/usr/sbin/ceph-create-keys --cluster ${CLUSTER} --id %i
>>
>> as you can see the side effect is blocked if the file
>>
>>     /var/lib/ceph/bootstrap-mds/ceph.keyring
>>
>> already exists, which is just more to document.
>>
>> Hoping that you all agree
>>
>> Owen Synge
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

-- 
SUSE LINUX GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB
21284 (AG
Nürnberg)

Maxfeldstraße 5

90409 Nürnberg

Germany
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

The fundamental evil of "magic" in computing systems -> Was: mon daemon makes authentication side effects on startup

[Owen Synge]

Dear Greg and others,

Thankyou for your very helpful email, as it completely misses my point,
and that illustrate why this point is so important to be addressed.

I am sure Greg has a deep understanding of this area. But I am pleased
Greg missed my points from 0-9, Greg's assumption that it is lack of
understanding on my part (which I am sure is common), clearly
illustrates where this "magic" of the side effect of starting a mon
demon becomes becomes "dark magic".

If you object to "magic" and "dark magic" in this email please
substitute them with "side effect" and "negative consequences of side
effects" respectively, and you get a more serious reply :)

On 04/05/2016 10:14 PM, Gregory Farnum wrote:
> I think you're fundamentally understanding how these keys come into
> existence. They aren't generated randomly on the local monitor; it
> uses get-or-create in order to fetch them (and create them if they
> don't already exist).

I have looked at this issue in depth, and general confusion in this area
is indeed very common, so it is reasonable to expect everyone is
confused by the same thing.

In my experience it is "magic" that causes admins fear, as good admins,
need to understand, because they need to understand the side effects of
any "magic", in case the "magic" is "dark", and in this case it is with
points (0) to (8) showing is indeed "dark magic".

Lets be specific:

Fetch and create are fundamentally different in side effects when doing
deployment. Lets be clear, when ceph does a "fetch" of a key, is not I
believe and issue, but when ceph uses magic to "create" keys, it can
often cause side effects. Hence the process to "create" a key should
only occur when its asked to be done.

The current get-or-create keyrings as a side effect of booting a mon
makes many issues (points 0-8 may not be all the issues, just ones that
spring to my mind). If the booting of a mon only did a fetch I would
feel we could resolve all my point except (2) and (9) sadly a boot of a
mon will also do a create keys where this "magic" starts to become very
"dark" indeed.

> So maybe it's difficult to pre-generate your own keys and plug them
> into the system (I don't remember where the initial values come from
> in standard deployment scenarios),

See my reply to John as to how you can deploy ceph without ceph-create-keys.

> but once they're set up you don't
> need to carefully install your values on all the monitor nodes — they
> will fetch the correct values from the monitor cluster.

I am objecting to the side effect of booting the mon and that process
creating keys that where not asked for, potentially causing valid
osd-bootstrap, rgw-bootstrap or mds-bootstrap to fail authentication as
invalid ones have been created as a side effect of starting the mon daemons.

> The coordination problem here is not really any different than that of
> making sure your monitors are all part of the "mon initial members"
> config option, 

You are forgetting that we also have osd-bootstrap, rgw-bootstrap or
mds-bootstrap keys and these may be generated by some other tool than
the mon, this is made much much harder to do by the mon init scripts
without being asked explicitly to do so.

> btw. Which you need to solve or else you're liable to
> have them coming up and creating independent monitor clusters and
> going haywire.
> -Greg

Not knowing what is happening is the enemy of understanding, and hence
the creator of "magic". Often giving the "magic" a name, or making it
explicit, causes enough understanding to remove it's "magic" properties.
Hence making all occurrences of key "create" (not "fetch") an explicit
step rather than a side effect will go a great deal to address this issue.

So if creating keys was not a side effect of booting mons, we would have
not issue here, as anyone who is used to cluster automation, has good
tools. These tools include chef, puppet, salt, and ansible, for cluster
management ideally, but more manually we have tools to copy files such
as rsync, scp, and tools to diagnose such issues such as checksums.

  ceph-create-keys --cluster ${CLUSTER} --id ${MON_NAME}

Having the above command separated from booting a mon actually avoids
osd's rgw's and mds's going haywire if they are configured in parallel
to the mon with keys from a source external to the mon, unless you
either (a) build in a layer of cluster synchronization above ceph, such
as ceph-deploy has done with its single threaded operation across a
complete cluster, or (b) do lots of dirty "magic" to remove
inconsistencies. Solution (a) is not good due to issue (0) amongst
others, and (b) creates more "magic" which has to be very carefully
designed to avoid it being "dark".

Another way to remove this "magic" is to document "magic" in detail, and
documenting this in this email is long and detailed, although Greg made
a start, he missed out the very important part of why the mds-bootstrap
keyring, is more important than is documented when if comes to deploying
your cluster the first time. I will skip it for now, but I am happy to
expand if needed.

In this case I argue the "magic" can be removed by making the process of
creating keys explicit. I would propose separating the "create" of  keys
from booting a mon is the least confusing and "magical" solution, with
the least chance of causing trouble for admins.

Thank you Greg for taking the time to reply, and please forgive me for
using your reply to illustrate that the real problem is the "magic", and
that "magic" removes understanding, hence knowledge of the "magic"
having "dark" issues, as this is a fear inducing thing for an admin new
to ceph.

Best wishes,

Owen

> On Tue, Apr 5, 2016 at 4:56 AM, Owen Synge <osynge@suse.com> wrote:
>> Dear all,
>>
>> This is in my opinion is clearly a bug, but I raise it in the mailing
>> list as I expect all admins of ceph will strongly agree, that this makes
>> ceph simpler, but developers may feel that since it requires changes to
>> more than one repo its not worth doing.
>>
>> When ever you start the mon demon as a side effect the admin, osd, rgw
>> and mds keys are created as a side effect if the mds keyring is not
>> existing.
>>
>> In the systemV and systemd init scripts (at least) we have a side
>> effect, that should be removed in my opinion, (or worse in my
>> alternatively correctly documented.)
>>
>> This is a deployment layer violation, in my opinion, and it requires
>> considerably more documentation, (and on my part also code) to keep this
>> side effect than remove it.
>>
>> usecases for removing this are:
>>
>> (0) A ceph cluster should be able to be installed in any order. With the
>> current behavior if the mds, rgw, or osd nodes are deployed first (along
>> with the boot strap keyrings), the mon created must have all keys for
>> the admin, mds-bootstrap, rgw-bootstrap, and osd-boostrap deployed in
>> the correct path before the mon can safely be started, even if the
>> cluster does not need the mds or rgw service's.
>>
>> (1) It is unfriendly to configuration being stored on the configuration
>> server as the server needs to be updated with the values from the
>> configured node keys, when people might want to store these keys centrally.
>>
>> (2) Assuming the admin, rgw-bootstrap, mds-bootstrap and osd-boostrap
>> keys are always installed on all mon nodes is clearly increasing the
>> distribution of keys where they might not be needed. Hence reducing
>> security.
>>
>> (3) Using the current model adds an extra complication that these keys
>> then need to be distributed to each node from the configured node, if
>> generated by starting the mon, and not from the configuration server.
>>
>> (4) If you wish to use a more devops approach, and generate keys
>> explicitly all the keys must be installed on all mon nodes before the
>> mon nodes are started.
>>
>> (4.1) As a side effect we need to document why admins need the
>> mds-bootstrap keyring when they dont want this service it is confusing,
>> and requires an unnecessary process of migrating all keys to the
>> explicitly desired keys.
>>
>> (5) I am developing a simple python library to configure ceph on each
>> node independently of all others, (think of it as a parallelism version
>> of ceph-deploy that can be called by any config management system) but
>> with the current side effect behavior starting the mon needs to fail if
>> the mds-bootstrap keyring is not created on the mon nodes before
>> starting the mon, otherwise we get ordering complications.
>>
>> (5) The side effect is confusing, as no one expects this side effect,
>> hence this leads to ceph seeming complex to a first time user.
>>
>> (6) I feel it is the responsibility of configuration management not the
>> mon demon to request creating these keys.
>>
>> (7) I dont think this is clearly documented, hence this leads to ceph
>> seeming complex to a first time user.
>>
>> (8) As more services like mds and rgw get added to ceph the problem gets
>> multiplied.
>>
>> (9) Adding one more step to the by hand installation will clarify the
>> authentication process. This extra step would simply be:
>>
>>    /usr/sbin/ceph-create-keys --cluster ${CLUSTER} --id ${MON_NAME}
>>
>> This is simpler and clearer than documenting the side effect.
>>
>> consequences:
>>
>> (1) Each configuration system which depends upon this behavior will need
>> to be modified to call the single command on each mon:
>>
>>    /usr/sbin/ceph-create-keys --cluster ${CLUSTER} --id ${MON_NAME}
>>
>> Here is a fix for ceph-deploy:
>>
>> https://github.com/SUSE/ceph-deploy/commit/58b030dbe0a964b32f1fbc9a3762e64dd74bf50c
>>
>> I assume other solutions will be easy to fix too.
>>
>> The systemd file in question, is
>> "/usr/lib/systemd/system/ceph-create-keys@.service" and should be removed.
>>
>> This will simplify the salt configuration module documentation
>> considerably, and if this is not done the salt module will need to add a
>> requirement on the mds keyring being created before the mon can be created.
>>
>> the systemd file looks as follows:
>>
>>     [Unit]
>>     Description=Ceph cluster key creator task
>>
>>     # the last key created is the mds bootstrap key -- look for that.
>>     ConditionPathExists=!/var/lib/ceph/bootstrap-mds/ceph.keyring
>>
>>     [Service]
>>     EnvironmentFile=-/etc/sysconfig/ceph
>>     Environment=CLUSTER=ceph
>>     ExecStart=/usr/sbin/ceph-create-keys --cluster ${CLUSTER} --id %i
>>
>> as you can see the side effect is blocked if the file
>>
>>     /var/lib/ceph/bootstrap-mds/ceph.keyring
>>
>> already exists, which is just more to document.
>>
>> Hoping that you all agree
>>
>> Owen Synge
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

-- 
SUSE LINUX GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB
21284 (AG
Nürnberg)

Maxfeldstraße 5

90409 Nürnberg

Germany
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: The fundamental evil of "magic" in computing systems -> Was: mon daemon makes authentication side effects on startup

[Jens Rosenboom]

2016-04-06 10:23 GMT+02:00 Owen Synge <osynge@suse.com>:
> Dear Greg and others,
>
> Thankyou for your very helpful email, as it completely misses my point,
> and that illustrate why this point is so important to be addressed.
>
> I am sure Greg has a deep understanding of this area. But I am pleased
> Greg missed my points from 0-9, Greg's assumption that it is lack of
> understanding on my part (which I am sure is common), clearly
> illustrates where this "magic" of the side effect of starting a mon
> demon becomes becomes "dark magic".
>
> If you object to "magic" and "dark magic" in this email please
> substitute them with "side effect" and "negative consequences of side
> effects" respectively, and you get a more serious reply :)
>
> On 04/05/2016 10:14 PM, Gregory Farnum wrote:
>> I think you're fundamentally understanding how these keys come into
>> existence. They aren't generated randomly on the local monitor; it
>> uses get-or-create in order to fetch them (and create them if they
>> don't already exist).
>
> I have looked at this issue in depth, and general confusion in this area
> is indeed very common, so it is reasonable to expect everyone is
> confused by the same thing.
>
> In my experience it is "magic" that causes admins fear, as good admins,
> need to understand, because they need to understand the side effects of
> any "magic", in case the "magic" is "dark", and in this case it is with
> points (0) to (8) showing is indeed "dark magic".
>
> Lets be specific:
>
> Fetch and create are fundamentally different in side effects when doing
> deployment. Lets be clear, when ceph does a "fetch" of a key, is not I
> believe and issue, but when ceph uses magic to "create" keys, it can
> often cause side effects. Hence the process to "create" a key should
> only occur when its asked to be done.
>
> The current get-or-create keyrings as a side effect of booting a mon
> makes many issues (points 0-8 may not be all the issues, just ones that
> spring to my mind). If the booting of a mon only did a fetch I would
> feel we could resolve all my point except (2) and (9) sadly a boot of a
> mon will also do a create keys where this "magic" starts to become very
> "dark" indeed.
>
>> So maybe it's difficult to pre-generate your own keys and plug them
>> into the system (I don't remember where the initial values come from
>> in standard deployment scenarios),
>
> See my reply to John as to how you can deploy ceph without ceph-create-keys.
>
>> but once they're set up you don't
>> need to carefully install your values on all the monitor nodes — they
>> will fetch the correct values from the monitor cluster.
>
> I am objecting to the side effect of booting the mon and that process
> creating keys that where not asked for, potentially causing valid
> osd-bootstrap, rgw-bootstrap or mds-bootstrap to fail authentication as
> invalid ones have been created as a side effect of starting the mon daemons.
>
>> The coordination problem here is not really any different than that of
>> making sure your monitors are all part of the "mon initial members"
>> config option,
>
> You are forgetting that we also have osd-bootstrap, rgw-bootstrap or
> mds-bootstrap keys and these may be generated by some other tool than
> the mon, this is made much much harder to do by the mon init scripts
> without being asked explicitly to do so.
>
>> btw. Which you need to solve or else you're liable to
>> have them coming up and creating independent monitor clusters and
>> going haywire.
>> -Greg
>
> Not knowing what is happening is the enemy of understanding, and hence
> the creator of "magic". Often giving the "magic" a name, or making it
> explicit, causes enough understanding to remove it's "magic" properties.
> Hence making all occurrences of key "create" (not "fetch") an explicit
> step rather than a side effect will go a great deal to address this issue.
>
> So if creating keys was not a side effect of booting mons, we would have
> not issue here, as anyone who is used to cluster automation, has good
> tools. These tools include chef, puppet, salt, and ansible, for cluster
> management ideally, but more manually we have tools to copy files such
> as rsync, scp, and tools to diagnose such issues such as checksums.
>
>   ceph-create-keys --cluster ${CLUSTER} --id ${MON_NAME}
>
> Having the above command separated from booting a mon actually avoids
> osd's rgw's and mds's going haywire if they are configured in parallel
> to the mon with keys from a source external to the mon, unless you
> either (a) build in a layer of cluster synchronization above ceph, such
> as ceph-deploy has done with its single threaded operation across a
> complete cluster, or (b) do lots of dirty "magic" to remove
> inconsistencies. Solution (a) is not good due to issue (0) amongst
> others, and (b) creates more "magic" which has to be very carefully
> designed to avoid it being "dark".
>
> Another way to remove this "magic" is to document "magic" in detail, and
> documenting this in this email is long and detailed, although Greg made
> a start, he missed out the very important part of why the mds-bootstrap
> keyring, is more important than is documented when if comes to deploying
> your cluster the first time. I will skip it for now, but I am happy to
> expand if needed.
>
> In this case I argue the "magic" can be removed by making the process of
> creating keys explicit. I would propose separating the "create" of  keys
> from booting a mon is the least confusing and "magical" solution, with
> the least chance of causing trouble for admins.
>
> Thank you Greg for taking the time to reply, and please forgive me for
> using your reply to illustrate that the real problem is the "magic", and
> that "magic" removes understanding, hence knowledge of the "magic"
> having "dark" issues, as this is a fear inducing thing for an admin new
> to ceph.

Hi Owen,

thanks for taking this up, I have been trying to get this issue fixed
in Ubuntu, not been aware that this was actually present in the
upstream code already, see

https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/1563330

So I strongly support your wish of being able to automatically deploy
a cluster, including all necessary credentials, without any
well-intended dark magic intervening.

Yours,
Jens
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: The fundamental evil of "magic" in computing systems -> Was: mon daemon makes authentication side effects on startup

[Owen Synge]

On 04/07/2016 09:49 AM, Jens Rosenboom wrote:
> 2016-04-06 10:23 GMT+02:00 Owen Synge <osynge@suse.com>:
>> Dear Greg and others,
>>
>> Thankyou for your very helpful email, as it completely misses my point,
>> and that illustrate why this point is so important to be addressed.
>>
>> I am sure Greg has a deep understanding of this area. But I am pleased
>> Greg missed my points from 0-9, Greg's assumption that it is lack of
>> understanding on my part (which I am sure is common), clearly
>> illustrates where this "magic" of the side effect of starting a mon
>> demon becomes becomes "dark magic".
>>
>> If you object to "magic" and "dark magic" in this email please
>> substitute them with "side effect" and "negative consequences of side
>> effects" respectively, and you get a more serious reply :)
>>
>> On 04/05/2016 10:14 PM, Gregory Farnum wrote:
>>> I think you're fundamentally understanding how these keys come into
>>> existence. They aren't generated randomly on the local monitor; it
>>> uses get-or-create in order to fetch them (and create them if they
>>> don't already exist).
>>
>> I have looked at this issue in depth, and general confusion in this area
>> is indeed very common, so it is reasonable to expect everyone is
>> confused by the same thing.
>>
>> In my experience it is "magic" that causes admins fear, as good admins,
>> need to understand, because they need to understand the side effects of
>> any "magic", in case the "magic" is "dark", and in this case it is with
>> points (0) to (8) showing is indeed "dark magic".
>>
>> Lets be specific:
>>
>> Fetch and create are fundamentally different in side effects when doing
>> deployment. Lets be clear, when ceph does a "fetch" of a key, is not I
>> believe and issue, but when ceph uses magic to "create" keys, it can
>> often cause side effects. Hence the process to "create" a key should
>> only occur when its asked to be done.
>>
>> The current get-or-create keyrings as a side effect of booting a mon
>> makes many issues (points 0-8 may not be all the issues, just ones that
>> spring to my mind). If the booting of a mon only did a fetch I would
>> feel we could resolve all my point except (2) and (9) sadly a boot of a
>> mon will also do a create keys where this "magic" starts to become very
>> "dark" indeed.
>>
>>> So maybe it's difficult to pre-generate your own keys and plug them
>>> into the system (I don't remember where the initial values come from
>>> in standard deployment scenarios),
>>
>> See my reply to John as to how you can deploy ceph without ceph-create-keys.
>>
>>> but once they're set up you don't
>>> need to carefully install your values on all the monitor nodes — they
>>> will fetch the correct values from the monitor cluster.
>>
>> I am objecting to the side effect of booting the mon and that process
>> creating keys that where not asked for, potentially causing valid
>> osd-bootstrap, rgw-bootstrap or mds-bootstrap to fail authentication as
>> invalid ones have been created as a side effect of starting the mon daemons.
>>
>>> The coordination problem here is not really any different than that of
>>> making sure your monitors are all part of the "mon initial members"
>>> config option,
>>
>> You are forgetting that we also have osd-bootstrap, rgw-bootstrap or
>> mds-bootstrap keys and these may be generated by some other tool than
>> the mon, this is made much much harder to do by the mon init scripts
>> without being asked explicitly to do so.
>>
>>> btw. Which you need to solve or else you're liable to
>>> have them coming up and creating independent monitor clusters and
>>> going haywire.
>>> -Greg
>>
>> Not knowing what is happening is the enemy of understanding, and hence
>> the creator of "magic". Often giving the "magic" a name, or making it
>> explicit, causes enough understanding to remove it's "magic" properties.
>> Hence making all occurrences of key "create" (not "fetch") an explicit
>> step rather than a side effect will go a great deal to address this issue.
>>
>> So if creating keys was not a side effect of booting mons, we would have
>> not issue here, as anyone who is used to cluster automation, has good
>> tools. These tools include chef, puppet, salt, and ansible, for cluster
>> management ideally, but more manually we have tools to copy files such
>> as rsync, scp, and tools to diagnose such issues such as checksums.
>>
>>   ceph-create-keys --cluster ${CLUSTER} --id ${MON_NAME}
>>
>> Having the above command separated from booting a mon actually avoids
>> osd's rgw's and mds's going haywire if they are configured in parallel
>> to the mon with keys from a source external to the mon, unless you
>> either (a) build in a layer of cluster synchronization above ceph, such
>> as ceph-deploy has done with its single threaded operation across a
>> complete cluster, or (b) do lots of dirty "magic" to remove
>> inconsistencies. Solution (a) is not good due to issue (0) amongst
>> others, and (b) creates more "magic" which has to be very carefully
>> designed to avoid it being "dark".
>>
>> Another way to remove this "magic" is to document "magic" in detail, and
>> documenting this in this email is long and detailed, although Greg made
>> a start, he missed out the very important part of why the mds-bootstrap
>> keyring, is more important than is documented when if comes to deploying
>> your cluster the first time. I will skip it for now, but I am happy to
>> expand if needed.
>>
>> In this case I argue the "magic" can be removed by making the process of
>> creating keys explicit. I would propose separating the "create" of  keys
>> from booting a mon is the least confusing and "magical" solution, with
>> the least chance of causing trouble for admins.
>>
>> Thank you Greg for taking the time to reply, and please forgive me for
>> using your reply to illustrate that the real problem is the "magic", and
>> that "magic" removes understanding, hence knowledge of the "magic"
>> having "dark" issues, as this is a fear inducing thing for an admin new
>> to ceph.
> 
> Hi Owen,
> 
> thanks for taking this up, I have been trying to get this issue fixed
> in Ubuntu, not been aware that this was actually present in the
> upstream code already, see
> 
> https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/1563330
> 
> So I strongly support your wish of being able to automatically deploy
> a cluster, including all necessary credentials, without any
> well-intended dark magic intervening.
> 
> Yours,
>
> --> Jens

Thanks Jens,

Your support in this is really valuable to know other people agree
well-intended dark magic is an issue for automation.

Best regards

Owen


-- 
SUSE LINUX GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB
21284 (AG
Nürnberg)

Maxfeldstraße 5

90409 Nürnberg

Germany
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: The fundamental evil of "magic" in computing systems -> Was: mon daemon makes authentication side effects on startup

[Sage Weil]

Hi Owen,

I never really liked ceph-create-keys either, but it simplified the 
deployment process.  I have no problem with removing it as long as we make 
sure the deployment process doesn't too much harder for ceph-deploy users.

Right now, you have to do something like

 ceph-deploy create MON1 MON2 MON3
 ceph-deploy install MON1 MON2 MON3

then manually

 ceph-deploy create MON1 MON2 MON3
 ceph-deploy gatherkeys MON1

or, using the simplified step that was added later,

 ceph-deploy mon create-initial

How about we change the manual process to do

 ceph-deploy create MON1 MON2 MON3
 ceph-deploy createkeys MON1    # ceph-deploy bootstrap-keys create HOST ?
 ceph-deploy gatherkeys MON1    # ceph-deploy bootstrap-keys gather HOST ?

and the create-initial one to run ceph-create-keys manually?

We'll need chef, ansible, puppet, and other recipes/playbooks/manifests to 
be updated as well...

sage

Re: The fundamental evil of "magic" in computing systems -> Was: mon daemon makes authentication side effects on startup

[Alfredo Deza]

On Wed, Apr 6, 2016 at 4:23 AM, Owen Synge <osynge@suse.com> wrote:
> Dear Greg and others,
>
> Thankyou for your very helpful email, as it completely misses my point,
> and that illustrate why this point is so important to be addressed.
>
> I am sure Greg has a deep understanding of this area. But I am pleased
> Greg missed my points from 0-9, Greg's assumption that it is lack of
> understanding on my part (which I am sure is common), clearly
> illustrates where this "magic" of the side effect of starting a mon
> demon becomes becomes "dark magic".
>
> If you object to "magic" and "dark magic" in this email please
> substitute them with "side effect" and "negative consequences of side
> effects" respectively, and you get a more serious reply :)
>
> On 04/05/2016 10:14 PM, Gregory Farnum wrote:
>> I think you're fundamentally understanding how these keys come into
>> existence. They aren't generated randomly on the local monitor; it
>> uses get-or-create in order to fetch them (and create them if they
>> don't already exist).
>
> I have looked at this issue in depth, and general confusion in this area
> is indeed very common, so it is reasonable to expect everyone is
> confused by the same thing.
>
> In my experience it is "magic" that causes admins fear, as good admins,
> need to understand, because they need to understand the side effects of
> any "magic", in case the "magic" is "dark", and in this case it is with
> points (0) to (8) showing is indeed "dark magic".
>
> Lets be specific:
>
> Fetch and create are fundamentally different in side effects when doing
> deployment. Lets be clear, when ceph does a "fetch" of a key, is not I
> believe and issue, but when ceph uses magic to "create" keys, it can
> often cause side effects. Hence the process to "create" a key should
> only occur when its asked to be done.
>
> The current get-or-create keyrings as a side effect of booting a mon
> makes many issues (points 0-8 may not be all the issues, just ones that
> spring to my mind). If the booting of a mon only did a fetch I would
> feel we could resolve all my point except (2) and (9) sadly a boot of a
> mon will also do a create keys where this "magic" starts to become very
> "dark" indeed.
>
>> So maybe it's difficult to pre-generate your own keys and plug them
>> into the system (I don't remember where the initial values come from
>> in standard deployment scenarios),
>
> See my reply to John as to how you can deploy ceph without ceph-create-keys.
>
>> but once they're set up you don't
>> need to carefully install your values on all the monitor nodes — they
>> will fetch the correct values from the monitor cluster.
>
> I am objecting to the side effect of booting the mon and that process
> creating keys that where not asked for, potentially causing valid
> osd-bootstrap, rgw-bootstrap or mds-bootstrap to fail authentication as
> invalid ones have been created as a side effect of starting the mon daemons.

That has been a *major* pain point in all deployment strategies
(ceph-deploy, ceph-ansible, ceph-installer, manual deployment)
I've tried: at some point a monitor is created and started and the
whole thing hangs forever because the keys are being helpfully
get-or-created for you but for $reasons it is unable to do so and
waits indefinitely.

This loop here:
https://github.com/ceph/ceph/blob/master/src/ceph-create-keys#L89-L120

Not even the log output is helpful because it is used as a side effect
process of starting a monitor, muting all output:

https://github.com/ceph/ceph/blob/jewel/src/init-ceph.in#L443

>
>> The coordination problem here is not really any different than that of
>> making sure your monitors are all part of the "mon initial members"
>> config option,
>
> You are forgetting that we also have osd-bootstrap, rgw-bootstrap or
> mds-bootstrap keys and these may be generated by some other tool than
> the mon, this is made much much harder to do by the mon init scripts
> without being asked explicitly to do so.
>
>> btw. Which you need to solve or else you're liable to
>> have them coming up and creating independent monitor clusters and
>> going haywire.
>> -Greg
>
> Not knowing what is happening is the enemy of understanding, and hence
> the creator of "magic". Often giving the "magic" a name, or making it
> explicit, causes enough understanding to remove it's "magic" properties.
> Hence making all occurrences of key "create" (not "fetch") an explicit
> step rather than a side effect will go a great deal to address this issue.
>
> So if creating keys was not a side effect of booting mons, we would have
> not issue here, as anyone who is used to cluster automation, has good
> tools. These tools include chef, puppet, salt, and ansible, for cluster
> management ideally, but more manually we have tools to copy files such
> as rsync, scp, and tools to diagnose such issues such as checksums.
>
>   ceph-create-keys --cluster ${CLUSTER} --id ${MON_NAME}
>
> Having the above command separated from booting a mon actually avoids
> osd's rgw's and mds's going haywire if they are configured in parallel
> to the mon with keys from a source external to the mon, unless you
> either (a) build in a layer of cluster synchronization above ceph, such
> as ceph-deploy has done with its single threaded operation across a
> complete cluster, or (b) do lots of dirty "magic" to remove
> inconsistencies. Solution (a) is not good due to issue (0) amongst
> others, and (b) creates more "magic" which has to be very carefully
> designed to avoid it being "dark".

Putting the "magic" definition aside,  being explicit about the
creation and management of keys would
be fantastic to have. Having an extra explicit step where a user/admin
needs to "create a key or distribute the keys you already have for
your cluster"
would be a big win here.

>
> Another way to remove this "magic" is to document "magic" in detail, and
> documenting this in this email is long and detailed, although Greg made
> a start, he missed out the very important part of why the mds-bootstrap
> keyring, is more important than is documented when if comes to deploying
> your cluster the first time. I will skip it for now, but I am happy to
> expand if needed.
>
> In this case I argue the "magic" can be removed by making the process of
> creating keys explicit. I would propose separating the "create" of  keys
> from booting a mon is the least confusing and "magical" solution, with
> the least chance of causing trouble for admins.
>
> Thank you Greg for taking the time to reply, and please forgive me for
> using your reply to illustrate that the real problem is the "magic", and
> that "magic" removes understanding, hence knowledge of the "magic"
> having "dark" issues, as this is a fear inducing thing for an admin new
> to ceph.
>
> Best wishes,
>
> Owen
>
>> On Tue, Apr 5, 2016 at 4:56 AM, Owen Synge <osynge@suse.com> wrote:
>>> Dear all,
>>>
>>> This is in my opinion is clearly a bug, but I raise it in the mailing
>>> list as I expect all admins of ceph will strongly agree, that this makes
>>> ceph simpler, but developers may feel that since it requires changes to
>>> more than one repo its not worth doing.
>>>
>>> When ever you start the mon demon as a side effect the admin, osd, rgw
>>> and mds keys are created as a side effect if the mds keyring is not
>>> existing.
>>>
>>> In the systemV and systemd init scripts (at least) we have a side
>>> effect, that should be removed in my opinion, (or worse in my
>>> alternatively correctly documented.)
>>>
>>> This is a deployment layer violation, in my opinion, and it requires
>>> considerably more documentation, (and on my part also code) to keep this
>>> side effect than remove it.
>>>
>>> usecases for removing this are:
>>>
>>> (0) A ceph cluster should be able to be installed in any order. With the
>>> current behavior if the mds, rgw, or osd nodes are deployed first (along
>>> with the boot strap keyrings), the mon created must have all keys for
>>> the admin, mds-bootstrap, rgw-bootstrap, and osd-boostrap deployed in
>>> the correct path before the mon can safely be started, even if the
>>> cluster does not need the mds or rgw service's.
>>>
>>> (1) It is unfriendly to configuration being stored on the configuration
>>> server as the server needs to be updated with the values from the
>>> configured node keys, when people might want to store these keys centrally.
>>>
>>> (2) Assuming the admin, rgw-bootstrap, mds-bootstrap and osd-boostrap
>>> keys are always installed on all mon nodes is clearly increasing the
>>> distribution of keys where they might not be needed. Hence reducing
>>> security.
>>>
>>> (3) Using the current model adds an extra complication that these keys
>>> then need to be distributed to each node from the configured node, if
>>> generated by starting the mon, and not from the configuration server.
>>>
>>> (4) If you wish to use a more devops approach, and generate keys
>>> explicitly all the keys must be installed on all mon nodes before the
>>> mon nodes are started.
>>>
>>> (4.1) As a side effect we need to document why admins need the
>>> mds-bootstrap keyring when they dont want this service it is confusing,
>>> and requires an unnecessary process of migrating all keys to the
>>> explicitly desired keys.
>>>
>>> (5) I am developing a simple python library to configure ceph on each
>>> node independently of all others, (think of it as a parallelism version
>>> of ceph-deploy that can be called by any config management system) but
>>> with the current side effect behavior starting the mon needs to fail if
>>> the mds-bootstrap keyring is not created on the mon nodes before
>>> starting the mon, otherwise we get ordering complications.
>>>
>>> (5) The side effect is confusing, as no one expects this side effect,
>>> hence this leads to ceph seeming complex to a first time user.
>>>
>>> (6) I feel it is the responsibility of configuration management not the
>>> mon demon to request creating these keys.
>>>
>>> (7) I dont think this is clearly documented, hence this leads to ceph
>>> seeming complex to a first time user.
>>>
>>> (8) As more services like mds and rgw get added to ceph the problem gets
>>> multiplied.
>>>
>>> (9) Adding one more step to the by hand installation will clarify the
>>> authentication process. This extra step would simply be:
>>>
>>>    /usr/sbin/ceph-create-keys --cluster ${CLUSTER} --id ${MON_NAME}
>>>
>>> This is simpler and clearer than documenting the side effect.
>>>
>>> consequences:
>>>
>>> (1) Each configuration system which depends upon this behavior will need
>>> to be modified to call the single command on each mon:
>>>
>>>    /usr/sbin/ceph-create-keys --cluster ${CLUSTER} --id ${MON_NAME}
>>>
>>> Here is a fix for ceph-deploy:
>>>
>>> https://github.com/SUSE/ceph-deploy/commit/58b030dbe0a964b32f1fbc9a3762e64dd74bf50c
>>>
>>> I assume other solutions will be easy to fix too.
>>>
>>> The systemd file in question, is
>>> "/usr/lib/systemd/system/ceph-create-keys@.service" and should be removed.
>>>
>>> This will simplify the salt configuration module documentation
>>> considerably, and if this is not done the salt module will need to add a
>>> requirement on the mds keyring being created before the mon can be created.
>>>
>>> the systemd file looks as follows:
>>>
>>>     [Unit]
>>>     Description=Ceph cluster key creator task
>>>
>>>     # the last key created is the mds bootstrap key -- look for that.
>>>     ConditionPathExists=!/var/lib/ceph/bootstrap-mds/ceph.keyring
>>>
>>>     [Service]
>>>     EnvironmentFile=-/etc/sysconfig/ceph
>>>     Environment=CLUSTER=ceph
>>>     ExecStart=/usr/sbin/ceph-create-keys --cluster ${CLUSTER} --id %i
>>>
>>> as you can see the side effect is blocked if the file
>>>
>>>     /var/lib/ceph/bootstrap-mds/ceph.keyring
>>>
>>> already exists, which is just more to document.
>>>
>>> Hoping that you all agree
>>>
>>> Owen Synge
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>
> --
> SUSE LINUX GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB
> 21284 (AG
> Nürnberg)
>
> Maxfeldstraße 5
>
> 90409 Nürnberg
>
> Germany
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: The fundamental evil of "magic" in computing systems -> Was: mon daemon makes authentication side effects on startup

[Owen Synge]

On 04/07/2016 02:33 PM, Alfredo Deza wrote:
> On Wed, Apr 6, 2016 at 4:23 AM, Owen Synge <osynge@suse.com> wrote:
>> Dear Greg and others,
>>
>> Thankyou for your very helpful email, as it completely misses my point,
>> and that illustrate why this point is so important to be addressed.
>>
>> I am sure Greg has a deep understanding of this area. But I am pleased
>> Greg missed my points from 0-9, Greg's assumption that it is lack of
>> understanding on my part (which I am sure is common), clearly
>> illustrates where this "magic" of the side effect of starting a mon
>> demon becomes becomes "dark magic".
>>
>> If you object to "magic" and "dark magic" in this email please
>> substitute them with "side effect" and "negative consequences of side
>> effects" respectively, and you get a more serious reply :)
>>
>> On 04/05/2016 10:14 PM, Gregory Farnum wrote:
>>> I think you're fundamentally understanding how these keys come into
>>> existence. They aren't generated randomly on the local monitor; it
>>> uses get-or-create in order to fetch them (and create them if they
>>> don't already exist).
>>
>> I have looked at this issue in depth, and general confusion in this area
>> is indeed very common, so it is reasonable to expect everyone is
>> confused by the same thing.
>>
>> In my experience it is "magic" that causes admins fear, as good admins,
>> need to understand, because they need to understand the side effects of
>> any "magic", in case the "magic" is "dark", and in this case it is with
>> points (0) to (8) showing is indeed "dark magic".
>>
>> Lets be specific:
>>
>> Fetch and create are fundamentally different in side effects when doing
>> deployment. Lets be clear, when ceph does a "fetch" of a key, is not I
>> believe and issue, but when ceph uses magic to "create" keys, it can
>> often cause side effects. Hence the process to "create" a key should
>> only occur when its asked to be done.
>>
>> The current get-or-create keyrings as a side effect of booting a mon
>> makes many issues (points 0-8 may not be all the issues, just ones that
>> spring to my mind). If the booting of a mon only did a fetch I would
>> feel we could resolve all my point except (2) and (9) sadly a boot of a
>> mon will also do a create keys where this "magic" starts to become very
>> "dark" indeed.
>>
>>> So maybe it's difficult to pre-generate your own keys and plug them
>>> into the system (I don't remember where the initial values come from
>>> in standard deployment scenarios),
>>
>> See my reply to John as to how you can deploy ceph without ceph-create-keys.
>>
>>> but once they're set up you don't
>>> need to carefully install your values on all the monitor nodes — they
>>> will fetch the correct values from the monitor cluster.
>>
>> I am objecting to the side effect of booting the mon and that process
>> creating keys that where not asked for, potentially causing valid
>> osd-bootstrap, rgw-bootstrap or mds-bootstrap to fail authentication as
>> invalid ones have been created as a side effect of starting the mon daemons.
> 
> That has been a *major* pain point in all deployment strategies
> (ceph-deploy, ceph-ansible, ceph-installer, manual deployment)
> I've tried: at some point a monitor is created and started and the
> whole thing hangs forever because the keys are being helpfully
> get-or-created for you but for $reasons it is unable to do so and
> waits indefinitely.

Thank you for the conformation, that this has effected you too.

I have had this problem with trying to make "ceph-salt" truly idiot
proof and without any timing issues.

> This loop here:
> https://github.com/ceph/ceph/blob/master/src/ceph-create-keys#L89-L120

Fortunately I have not yet seen this loop waiting indefinitely.

I (or some one else) should I guess get around to writing a time out
patch, if some one does not get there first.

> Not even the log output is helpful because it is used as a side effect
> process of starting a monitor, muting all output:
> 
> https://github.com/ceph/ceph/blob/jewel/src/init-ceph.in#L443

Oh yes that is probably the most horrid consequence of the "magic" I
have yet seen. I think this now deserves being referenced as point (10)

>>> The coordination problem here is not really any different than that of
>>> making sure your monitors are all part of the "mon initial members"
>>> config option,
>>
>> You are forgetting that we also have osd-bootstrap, rgw-bootstrap or
>> mds-bootstrap keys and these may be generated by some other tool than
>> the mon, this is made much much harder to do by the mon init scripts
>> without being asked explicitly to do so.
>>
>>> btw. Which you need to solve or else you're liable to
>>> have them coming up and creating independent monitor clusters and
>>> going haywire.
>>> -Greg
>>
>> Not knowing what is happening is the enemy of understanding, and hence
>> the creator of "magic". Often giving the "magic" a name, or making it
>> explicit, causes enough understanding to remove it's "magic" properties.
>> Hence making all occurrences of key "create" (not "fetch") an explicit
>> step rather than a side effect will go a great deal to address this issue.
>>
>> So if creating keys was not a side effect of booting mons, we would have
>> not issue here, as anyone who is used to cluster automation, has good
>> tools. These tools include chef, puppet, salt, and ansible, for cluster
>> management ideally, but more manually we have tools to copy files such
>> as rsync, scp, and tools to diagnose such issues such as checksums.
>>
>>   ceph-create-keys --cluster ${CLUSTER} --id ${MON_NAME}
>>
>> Having the above command separated from booting a mon actually avoids
>> osd's rgw's and mds's going haywire if they are configured in parallel
>> to the mon with keys from a source external to the mon, unless you
>> either (a) build in a layer of cluster synchronization above ceph, such
>> as ceph-deploy has done with its single threaded operation across a
>> complete cluster, or (b) do lots of dirty "magic" to remove
>> inconsistencies. Solution (a) is not good due to issue (0) amongst
>> others, and (b) creates more "magic" which has to be very carefully
>> designed to avoid it being "dark".
> 
> Putting the "magic" definition aside,  being explicit about the
> creation and management of keys would
> be fantastic to have. Having an extra explicit step where a user/admin
> needs to "create a key or distribute the keys you already have for
> your cluster"
> would be a big win here.

This is wonderful that we are coming to consensus here :)

So I will raise a bug, and site this thread.

Best regards

Owen

> 
>>
>> Another way to remove this "magic" is to document "magic" in detail, and
>> documenting this in this email is long and detailed, although Greg made
>> a start, he missed out the very important part of why the mds-bootstrap
>> keyring, is more important than is documented when if comes to deploying
>> your cluster the first time. I will skip it for now, but I am happy to
>> expand if needed.
>>
>> In this case I argue the "magic" can be removed by making the process of
>> creating keys explicit. I would propose separating the "create" of  keys
>> from booting a mon is the least confusing and "magical" solution, with
>> the least chance of causing trouble for admins.
>>
>> Thank you Greg for taking the time to reply, and please forgive me for
>> using your reply to illustrate that the real problem is the "magic", and
>> that "magic" removes understanding, hence knowledge of the "magic"
>> having "dark" issues, as this is a fear inducing thing for an admin new
>> to ceph.
>>
>> Best wishes,
>>
>> Owen
>>
>>> On Tue, Apr 5, 2016 at 4:56 AM, Owen Synge <osynge@suse.com> wrote:
>>>> Dear all,
>>>>
>>>> This is in my opinion is clearly a bug, but I raise it in the mailing
>>>> list as I expect all admins of ceph will strongly agree, that this makes
>>>> ceph simpler, but developers may feel that since it requires changes to
>>>> more than one repo its not worth doing.
>>>>
>>>> When ever you start the mon demon as a side effect the admin, osd, rgw
>>>> and mds keys are created as a side effect if the mds keyring is not
>>>> existing.
>>>>
>>>> In the systemV and systemd init scripts (at least) we have a side
>>>> effect, that should be removed in my opinion, (or worse in my
>>>> alternatively correctly documented.)
>>>>
>>>> This is a deployment layer violation, in my opinion, and it requires
>>>> considerably more documentation, (and on my part also code) to keep this
>>>> side effect than remove it.
>>>>
>>>> usecases for removing this are:
>>>>
>>>> (0) A ceph cluster should be able to be installed in any order. With the
>>>> current behavior if the mds, rgw, or osd nodes are deployed first (along
>>>> with the boot strap keyrings), the mon created must have all keys for
>>>> the admin, mds-bootstrap, rgw-bootstrap, and osd-boostrap deployed in
>>>> the correct path before the mon can safely be started, even if the
>>>> cluster does not need the mds or rgw service's.
>>>>
>>>> (1) It is unfriendly to configuration being stored on the configuration
>>>> server as the server needs to be updated with the values from the
>>>> configured node keys, when people might want to store these keys centrally.
>>>>
>>>> (2) Assuming the admin, rgw-bootstrap, mds-bootstrap and osd-boostrap
>>>> keys are always installed on all mon nodes is clearly increasing the
>>>> distribution of keys where they might not be needed. Hence reducing
>>>> security.
>>>>
>>>> (3) Using the current model adds an extra complication that these keys
>>>> then need to be distributed to each node from the configured node, if
>>>> generated by starting the mon, and not from the configuration server.
>>>>
>>>> (4) If you wish to use a more devops approach, and generate keys
>>>> explicitly all the keys must be installed on all mon nodes before the
>>>> mon nodes are started.
>>>>
>>>> (4.1) As a side effect we need to document why admins need the
>>>> mds-bootstrap keyring when they dont want this service it is confusing,
>>>> and requires an unnecessary process of migrating all keys to the
>>>> explicitly desired keys.
>>>>
>>>> (5) I am developing a simple python library to configure ceph on each
>>>> node independently of all others, (think of it as a parallelism version
>>>> of ceph-deploy that can be called by any config management system) but
>>>> with the current side effect behavior starting the mon needs to fail if
>>>> the mds-bootstrap keyring is not created on the mon nodes before
>>>> starting the mon, otherwise we get ordering complications.
>>>>
>>>> (5) The side effect is confusing, as no one expects this side effect,
>>>> hence this leads to ceph seeming complex to a first time user.
>>>>
>>>> (6) I feel it is the responsibility of configuration management not the
>>>> mon demon to request creating these keys.
>>>>
>>>> (7) I dont think this is clearly documented, hence this leads to ceph
>>>> seeming complex to a first time user.
>>>>
>>>> (8) As more services like mds and rgw get added to ceph the problem gets
>>>> multiplied.
>>>>
>>>> (9) Adding one more step to the by hand installation will clarify the
>>>> authentication process. This extra step would simply be:
>>>>
>>>>    /usr/sbin/ceph-create-keys --cluster ${CLUSTER} --id ${MON_NAME}
>>>>
>>>> This is simpler and clearer than documenting the side effect.
>>>>
>>>> consequences:
>>>>
>>>> (1) Each configuration system which depends upon this behavior will need
>>>> to be modified to call the single command on each mon:
>>>>
>>>>    /usr/sbin/ceph-create-keys --cluster ${CLUSTER} --id ${MON_NAME}
>>>>
>>>> Here is a fix for ceph-deploy:
>>>>
>>>> https://github.com/SUSE/ceph-deploy/commit/58b030dbe0a964b32f1fbc9a3762e64dd74bf50c
>>>>
>>>> I assume other solutions will be easy to fix too.
>>>>
>>>> The systemd file in question, is
>>>> "/usr/lib/systemd/system/ceph-create-keys@.service" and should be removed.
>>>>
>>>> This will simplify the salt configuration module documentation
>>>> considerably, and if this is not done the salt module will need to add a
>>>> requirement on the mds keyring being created before the mon can be created.
>>>>
>>>> the systemd file looks as follows:
>>>>
>>>>     [Unit]
>>>>     Description=Ceph cluster key creator task
>>>>
>>>>     # the last key created is the mds bootstrap key -- look for that.
>>>>     ConditionPathExists=!/var/lib/ceph/bootstrap-mds/ceph.keyring
>>>>
>>>>     [Service]
>>>>     EnvironmentFile=-/etc/sysconfig/ceph
>>>>     Environment=CLUSTER=ceph
>>>>     ExecStart=/usr/sbin/ceph-create-keys --cluster ${CLUSTER} --id %i
>>>>
>>>> as you can see the side effect is blocked if the file
>>>>
>>>>     /var/lib/ceph/bootstrap-mds/ceph.keyring
>>>>
>>>> already exists, which is just more to document.
>>>>
>>>> Hoping that you all agree
>>>>
>>>> Owen Synge
>>>> --
>>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>>>> the body of a message to majordomo@vger.kernel.org
>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>
>>
>> --
>> SUSE LINUX GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB
>> 21284 (AG
>> Nürnberg)
>>
>> Maxfeldstraße 5
>>
>> 90409 Nürnberg
>>
>> Germany
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

-- 
SUSE LINUX GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB
21284 (AG
Nürnberg)

Maxfeldstraße 5

90409 Nürnberg

Germany
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: The fundamental evil of "magic" in computing systems -> Was: mon daemon makes authentication side effects on startup

[Owen Synge]

Hi Sage,

On 04/07/2016 02:26 PM, Sage Weil wrote:
> Hi Owen,
> 
> I never really liked ceph-create-keys either, but it simplified the 
> deployment process.  

I would propose we do this in two stages.

(A) Remove calling the command from the init scripts as a side effect of
starting the mon.

This allows us to get most of the issues solved.

(B) Remove the command.

This is the long term goal, which is not as urgent in my opinion but
others may disagree.

> I have no problem with removing it as long as we make
> sure the deployment process doesn't too much harder for ceph-deploy users.

The documentation for the manual process without using ceph-deploy will
need to be changed if we remove calling ceph-create-keys from the boot
scripts.

For ceph-deploy users I think we should see if any changes to the
process are needed, the next question is will any be wanted?

While this change still depends on ceph-create-keys it can work without
this code being a side effect of mon booting.

This change here solves the issue for the new way but uses ceph-create-keys:

https://github.com/SUSE/ceph-deploy/commit/58b030dbe0a964b32f1fbc9a3762e64dd74bf50c

Thank you for mentioning the old manual way, I will see if this fix also
solves this, and make a PR to get us to be able to remove the command
ceph-create-keys from the boot scripts of the mon under both ways of
using ceph-deploy you mention below.

(I had forgotten about the old way)

> Right now, you have to do something like
> 
>  ceph-deploy create MON1 MON2 MON3
>  ceph-deploy install MON1 MON2 MON3
>
> then manually
> 
>  ceph-deploy create MON1 MON2 MON3
>  ceph-deploy gatherkeys MON1
> 
> or, using the simplified step that was added later,
> 
>  ceph-deploy mon create-initial
> 
> How about we change the manual process to do
> 
>  ceph-deploy create MON1 MON2 MON3
>  ceph-deploy createkeys MON1    # ceph-deploy bootstrap-keys create HOST ?
>  ceph-deploy gatherkeys MON1    # ceph-deploy bootstrap-keys gather HOST ?
> 
> and the create-initial one to run ceph-create-keys manually?

I think for ceph-deploy we should target no changes to use.

Manually installing ceph without ceph-deploy their will need to be
documentation changes.

> We'll need chef, ansible, puppet, and other recipes/playbooks/manifests to 
> be updated as well...

Yes this is why I had interstitially worried that this would never be
accepted.

Best regards

Owen

Re: The fundamental evil of "magic" in computing systems -> Was: mon daemon makes authentication side effects on startup

[Sage Weil]

On Thu, 7 Apr 2016, Owen Synge wrote:
> Hi Sage,
> 
> On 04/07/2016 02:26 PM, Sage Weil wrote:
> > Hi Owen,
> > 
> > I never really liked ceph-create-keys either, but it simplified the 
> > deployment process.  
> 
> I would propose we do this in two stages.
> 
> (A) Remove calling the command from the init scripts as a side effect of
> starting the mon.
> 
> This allows us to get most of the issues solved.
> 
> (B) Remove the command.
> 
> This is the long term goal, which is not as urgent in my opinion but
> others may disagree.

Works for me.  We just need to change ceph-deploy and get the other 
install/deploy tool folks on board before A.

> > I have no problem with removing it as long as we make
> > sure the deployment process doesn't too much harder for ceph-deploy users.
> 
> The documentation for the manual process without using ceph-deploy will
> need to be changed if we remove calling ceph-create-keys from the boot
> scripts.

Yeah.

> For ceph-deploy users I think we should see if any changes to the
> process are needed, the next question is will any be wanted?

Actually, thinking about it a bit more, I don't think ceph-deploy usage 
has to change at all.  The old way was

 1. ceph-create-keys creates and installs the keys on the mons
 2. ceph-deploy gatherkeys or create-initial slurps them up

We can just change ceph-deploy so it creates and stores them locally, and 
doesn't store them on the mons at all.  Users don't get the side-effect 
that the mons have the keys installed, but that is arguably better anyway.

> https://github.com/SUSE/ceph-deploy/commit/58b030dbe0a964b32f1fbc9a3762e64dd74bf50c

Instead of this, it should do the same steps as ceph-create-keys (use the 
mon. internal key to authenticate to create the admin and bootstrap 
keys), but just write them to the local ceph-dpeloy work dir.

sage

Re: The fundamental evil of "magic" in computing systems -> Was: mon daemon makes authentication side effects on startup

[Gregory Farnum]

On Thu, Apr 7, 2016 at 5:33 AM, Alfredo Deza <adeza@redhat.com> wrote:
> On Wed, Apr 6, 2016 at 4:23 AM, Owen Synge <osynge@suse.com> wrote:
>> Dear Greg and others,
>>
>> Thankyou for your very helpful email, as it completely misses my point,
>> and that illustrate why this point is so important to be addressed.
>>
>> I am sure Greg has a deep understanding of this area. But I am pleased
>> Greg missed my points from 0-9, Greg's assumption that it is lack of
>> understanding on my part (which I am sure is common), clearly
>> illustrates where this "magic" of the side effect of starting a mon
>> demon becomes becomes "dark magic".
>>
>> If you object to "magic" and "dark magic" in this email please
>> substitute them with "side effect" and "negative consequences of side
>> effects" respectively, and you get a more serious reply :)
>>
>> On 04/05/2016 10:14 PM, Gregory Farnum wrote:
>>> I think you're fundamentally understanding how these keys come into
>>> existence. They aren't generated randomly on the local monitor; it
>>> uses get-or-create in order to fetch them (and create them if they
>>> don't already exist).
>>
>> I have looked at this issue in depth, and general confusion in this area
>> is indeed very common, so it is reasonable to expect everyone is
>> confused by the same thing.
>>
>> In my experience it is "magic" that causes admins fear, as good admins,
>> need to understand, because they need to understand the side effects of
>> any "magic", in case the "magic" is "dark", and in this case it is with
>> points (0) to (8) showing is indeed "dark magic".
>>
>> Lets be specific:
>>
>> Fetch and create are fundamentally different in side effects when doing
>> deployment. Lets be clear, when ceph does a "fetch" of a key, is not I
>> believe and issue, but when ceph uses magic to "create" keys, it can
>> often cause side effects. Hence the process to "create" a key should
>> only occur when its asked to be done.
>>
>> The current get-or-create keyrings as a side effect of booting a mon
>> makes many issues (points 0-8 may not be all the issues, just ones that
>> spring to my mind). If the booting of a mon only did a fetch I would
>> feel we could resolve all my point except (2) and (9) sadly a boot of a
>> mon will also do a create keys where this "magic" starts to become very
>> "dark" indeed.
>>
>>> So maybe it's difficult to pre-generate your own keys and plug them
>>> into the system (I don't remember where the initial values come from
>>> in standard deployment scenarios),
>>
>> See my reply to John as to how you can deploy ceph without ceph-create-keys.
>>
>>> but once they're set up you don't
>>> need to carefully install your values on all the monitor nodes — they
>>> will fetch the correct values from the monitor cluster.
>>
>> I am objecting to the side effect of booting the mon and that process
>> creating keys that where not asked for, potentially causing valid
>> osd-bootstrap, rgw-bootstrap or mds-bootstrap to fail authentication as
>> invalid ones have been created as a side effect of starting the mon daemons.
>
> That has been a *major* pain point in all deployment strategies
> (ceph-deploy, ceph-ansible, ceph-installer, manual deployment)
> I've tried: at some point a monitor is created and started and the
> whole thing hangs forever because the keys are being helpfully
> get-or-created for you but for $reasons it is unable to do so and
> waits indefinitely.

I don't have any attachment to ceph-create-keys and the magic itself,
but this right here is my main concern about changing things: the
$reasons for failure being cited here were never problems with
ceph-create-keys itself; it was just the first part of the process
which required a functioning monitor quorum, and so it functioned as a
(not very good) break point. I have very little personal interest in
the configuration systems and would be vaguely more content if that
"test your monitors are speaking to each other" step was both distinct
from the keys stage and more explicit/debuggable.

(But I am noticing that so far the monitor test has either been moved
to the "create new OSDs" phase or else to the "user runs post-setup
ceph -s and gets nothing" unsupervised phase; I'm not sure those are
better since they don't even point directly to the monitors.)

So I think it would be fine to not automatically create keys on
monitor startup. But Owen is saying, if you create your own bootstrap
keys, you need to make sure you write them out to the monitor node
filesystems before startup or they won't match. That's not true! You
need to make sure the monitor cluster has them, internally, yes. But
once you've accomplished that, all of your monitors are going to grab
the bootstrap keys from the cluster and write them out locally — if
you've got different bootstrap keys on different monitors, you've
*created multiple monitor clusters*.
That's why I'm worried about this specific thread of complaint. :)
-Greg
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: The fundamental evil of "magic" in computing systems -> Was: mon daemon makes authentication side effects on startup

[Gregory Farnum]

On Thu, Apr 7, 2016 at 7:03 AM, Sage Weil <sweil@redhat.com> wrote:
> On Thu, 7 Apr 2016, Owen Synge wrote:
>> Hi Sage,
>>
>> On 04/07/2016 02:26 PM, Sage Weil wrote:
>> > Hi Owen,
>> >
>> > I never really liked ceph-create-keys either, but it simplified the
>> > deployment process.
>>
>> I would propose we do this in two stages.
>>
>> (A) Remove calling the command from the init scripts as a side effect of
>> starting the mon.
>>
>> This allows us to get most of the issues solved.
>>
>> (B) Remove the command.
>>
>> This is the long term goal, which is not as urgent in my opinion but
>> others may disagree.
>
> Works for me.  We just need to change ceph-deploy and get the other
> install/deploy tool folks on board before A.
>
>> > I have no problem with removing it as long as we make
>> > sure the deployment process doesn't too much harder for ceph-deploy users.
>>
>> The documentation for the manual process without using ceph-deploy will
>> need to be changed if we remove calling ceph-create-keys from the boot
>> scripts.
>
> Yeah.
>
>> For ceph-deploy users I think we should see if any changes to the
>> process are needed, the next question is will any be wanted?
>
> Actually, thinking about it a bit more, I don't think ceph-deploy usage
> has to change at all.  The old way was
>
>  1. ceph-create-keys creates and installs the keys on the mons
>  2. ceph-deploy gatherkeys or create-initial slurps them up
>
> We can just change ceph-deploy so it creates and stores them locally, and
> doesn't store them on the mons at all.  Users don't get the side-effect
> that the mons have the keys installed, but that is arguably better anyway.

Note that this would mean users can only run "ceph -s" from a node
that has explicitly installed the keys — if you are debugging a
problem and ssh into the monitor node, you'll have to run fugly
command lines pointing to the monitor's data directory and using a
"mon." entity name in order to get any cluster debug info. :/
-Greg
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: The fundamental evil of "magic" in computing systems -> Was: mon daemon makes authentication side effects on startup

[Sage Weil]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 2356 bytes --]

On Thu, 7 Apr 2016, Gregory Farnum wrote:
> On Thu, Apr 7, 2016 at 7:03 AM, Sage Weil <sweil@redhat.com> wrote:
> > On Thu, 7 Apr 2016, Owen Synge wrote:
> >> Hi Sage,
> >>
> >> On 04/07/2016 02:26 PM, Sage Weil wrote:
> >> > Hi Owen,
> >> >
> >> > I never really liked ceph-create-keys either, but it simplified the
> >> > deployment process.
> >>
> >> I would propose we do this in two stages.
> >>
> >> (A) Remove calling the command from the init scripts as a side effect of
> >> starting the mon.
> >>
> >> This allows us to get most of the issues solved.
> >>
> >> (B) Remove the command.
> >>
> >> This is the long term goal, which is not as urgent in my opinion but
> >> others may disagree.
> >
> > Works for me.  We just need to change ceph-deploy and get the other
> > install/deploy tool folks on board before A.
> >
> >> > I have no problem with removing it as long as we make
> >> > sure the deployment process doesn't too much harder for ceph-deploy users.
> >>
> >> The documentation for the manual process without using ceph-deploy will
> >> need to be changed if we remove calling ceph-create-keys from the boot
> >> scripts.
> >
> > Yeah.
> >
> >> For ceph-deploy users I think we should see if any changes to the
> >> process are needed, the next question is will any be wanted?
> >
> > Actually, thinking about it a bit more, I don't think ceph-deploy usage
> > has to change at all.  The old way was
> >
> >  1. ceph-create-keys creates and installs the keys on the mons
> >  2. ceph-deploy gatherkeys or create-initial slurps them up
> >
> > We can just change ceph-deploy so it creates and stores them locally, and
> > doesn't store them on the mons at all.  Users don't get the side-effect
> > that the mons have the keys installed, but that is arguably better anyway.
> 
> Note that this would mean users can only run "ceph -s" from a node
> that has explicitly installed the keys — if you are debugging a
> problem and ssh into the monitor node, you'll have to run fugly
> command lines pointing to the monitor's data directory and using a
> "mon." entity name in order to get any cluster debug info. :/

Yeah.  We could make 'mon create-initial' also install the admin (and 
maybe bootstrap) keys on the mon, and make a note in the manual 
ceph-deploy sequence about this (add 'ceph-deploy admin HOST' to the 
sequence).

sage

Re: The fundamental evil of "magic" in computing systems -> Was: mon daemon makes authentication side effects on startup

[Owen Synge]

On 04/07/2016 04:03 PM, Sage Weil wrote:
> On Thu, 7 Apr 2016, Owen Synge wrote:
>> Hi Sage,
>>
>> On 04/07/2016 02:26 PM, Sage Weil wrote:
>>> Hi Owen,
>>>
>>> I never really liked ceph-create-keys either, but it simplified the 
>>> deployment process.  
>>
>> I would propose we do this in two stages.
>>
>> (A) Remove calling the command from the init scripts as a side effect of
>> starting the mon.
>>
>> This allows us to get most of the issues solved.
>>
>> (B) Remove the command.
>>
>> This is the long term goal, which is not as urgent in my opinion but
>> others may disagree.
> 
> Works for me.  We just need to change ceph-deploy and get the other 
> install/deploy tool folks on board before A.

Are you intending to get this into Jewel?

I had assumed this would only be done on master, and only come into the
next release.

As a change to master I felt that we could just do (A) as soon as
ceph-deploy works without the mon boot up scripts calling
ceph-create-keys, ideally without having  ceph-create-keys in
ceph-deploy's process.

We can then file bugs as needed against other install processes that
depend on ceph-create-keys, and they can test against master.

>>> I have no problem with removing it as long as we make
>>> sure the deployment process doesn't too much harder for ceph-deploy users.
>>
>> The documentation for the manual process without using ceph-deploy will
>> need to be changed if we remove calling ceph-create-keys from the boot
>> scripts.
> 
> Yeah.
> 
>> For ceph-deploy users I think we should see if any changes to the
>> process are needed, the next question is will any be wanted?
> 
> Actually, thinking about it a bit more, I don't think ceph-deploy usage 
> has to change at all.  The old way was
> 
>  1. ceph-create-keys creates and installs the keys on the mons
>  2. ceph-deploy gatherkeys or create-initial slurps them up
> 
> We can just change ceph-deploy so it creates and stores them locally, and 
> doesn't store them on the mons at all.  Users don't get the side-effect 
> that the mons have the keys installed, but that is arguably better anyway.
> 
>> https://github.com/SUSE/ceph-deploy/commit/58b030dbe0a964b32f1fbc9a3762e64dd74bf50c
> 
> Instead of this, it should do the same steps as ceph-create-keys (use the 
> mon. internal key to authenticate to create the admin and bootstrap 
> keys), but just write them to the local ceph-dpeloy work dir.

That sounds like a better plan than my idea of just using
ceph-create-keys in ceph-deploy to me.

Best regards

Owen

-- 
SUSE LINUX GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB
21284 (AG
Nürnberg)

Maxfeldstraße 5

90409 Nürnberg

Germany
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: The fundamental evil of "magic" in computing systems -> Was: mon daemon makes authentication side effects on startup

[Sage Weil]

On Thu, 7 Apr 2016, Owen Synge wrote:
> On 04/07/2016 04:03 PM, Sage Weil wrote:
> > On Thu, 7 Apr 2016, Owen Synge wrote:
> >> Hi Sage,
> >>
> >> On 04/07/2016 02:26 PM, Sage Weil wrote:
> >>> Hi Owen,
> >>>
> >>> I never really liked ceph-create-keys either, but it simplified the 
> >>> deployment process.  
> >>
> >> I would propose we do this in two stages.
> >>
> >> (A) Remove calling the command from the init scripts as a side effect of
> >> starting the mon.
> >>
> >> This allows us to get most of the issues solved.
> >>
> >> (B) Remove the command.
> >>
> >> This is the long term goal, which is not as urgent in my opinion but
> >> others may disagree.
> > 
> > Works for me.  We just need to change ceph-deploy and get the other 
> > install/deploy tool folks on board before A.
> 
> Are you intending to get this into Jewel?
> 
> I had assumed this would only be done on master, and only come into the
> next release.

Yeah, too late for jewel.

> As a change to master I felt that we could just do (A) as soon as
> ceph-deploy works without the mon boot up scripts calling
> ceph-create-keys, ideally without having  ceph-create-keys in
> ceph-deploy's process.
> 
> We can then file bugs as needed against other install processes that
> depend on ceph-create-keys, and they can test against master.

That works for me.

sage

Re: The fundamental evil of "magic" in computing systems -> Was: mon daemon makes authentication side effects on startup

[Owen Synge]

On 04/07/2016 04:22 PM, Gregory Farnum wrote:
> On Thu, Apr 7, 2016 at 5:33 AM, Alfredo Deza <adeza@redhat.com> wrote:
>> On Wed, Apr 6, 2016 at 4:23 AM, Owen Synge <osynge@suse.com> wrote:
>>> Dear Greg and others,
>>>
>>> Thankyou for your very helpful email, as it completely misses my point,
>>> and that illustrate why this point is so important to be addressed.
>>>
>>> I am sure Greg has a deep understanding of this area. But I am pleased
>>> Greg missed my points from 0-9, Greg's assumption that it is lack of
>>> understanding on my part (which I am sure is common), clearly
>>> illustrates where this "magic" of the side effect of starting a mon
>>> demon becomes becomes "dark magic".
>>>
>>> If you object to "magic" and "dark magic" in this email please
>>> substitute them with "side effect" and "negative consequences of side
>>> effects" respectively, and you get a more serious reply :)
>>>
>>> On 04/05/2016 10:14 PM, Gregory Farnum wrote:
>>>> I think you're fundamentally understanding how these keys come into
>>>> existence. They aren't generated randomly on the local monitor; it
>>>> uses get-or-create in order to fetch them (and create them if they
>>>> don't already exist).
>>>
>>> I have looked at this issue in depth, and general confusion in this area
>>> is indeed very common, so it is reasonable to expect everyone is
>>> confused by the same thing.
>>>
>>> In my experience it is "magic" that causes admins fear, as good admins,
>>> need to understand, because they need to understand the side effects of
>>> any "magic", in case the "magic" is "dark", and in this case it is with
>>> points (0) to (8) showing is indeed "dark magic".
>>>
>>> Lets be specific:
>>>
>>> Fetch and create are fundamentally different in side effects when doing
>>> deployment. Lets be clear, when ceph does a "fetch" of a key, is not I
>>> believe and issue, but when ceph uses magic to "create" keys, it can
>>> often cause side effects. Hence the process to "create" a key should
>>> only occur when its asked to be done.
>>>
>>> The current get-or-create keyrings as a side effect of booting a mon
>>> makes many issues (points 0-8 may not be all the issues, just ones that
>>> spring to my mind). If the booting of a mon only did a fetch I would
>>> feel we could resolve all my point except (2) and (9) sadly a boot of a
>>> mon will also do a create keys where this "magic" starts to become very
>>> "dark" indeed.
>>>
>>>> So maybe it's difficult to pre-generate your own keys and plug them
>>>> into the system (I don't remember where the initial values come from
>>>> in standard deployment scenarios),
>>>
>>> See my reply to John as to how you can deploy ceph without ceph-create-keys.
>>>
>>>> but once they're set up you don't
>>>> need to carefully install your values on all the monitor nodes — they
>>>> will fetch the correct values from the monitor cluster.
>>>
>>> I am objecting to the side effect of booting the mon and that process
>>> creating keys that where not asked for, potentially causing valid
>>> osd-bootstrap, rgw-bootstrap or mds-bootstrap to fail authentication as
>>> invalid ones have been created as a side effect of starting the mon daemons.
>>
>> That has been a *major* pain point in all deployment strategies
>> (ceph-deploy, ceph-ansible, ceph-installer, manual deployment)
>> I've tried: at some point a monitor is created and started and the
>> whole thing hangs forever because the keys are being helpfully
>> get-or-created for you but for $reasons it is unable to do so and
>> waits indefinitely.
> 
> I don't have any attachment to ceph-create-keys and the magic itself,
> but this right here is my main concern about changing things: the
> $reasons for failure being cited here were never problems with
> ceph-create-keys itself; it was just the first part of the process
> which required a functioning monitor quorum, and so it functioned as a
> (not very good) break point. I have very little personal interest in
> the configuration systems and would be vaguely more content if that
> "test your monitors are speaking to each other" step was both distinct
> from the keys stage and more explicit/debuggable.

That is a good point.

Fortunately we already have a command that does not require authentication:

  ceph --connect-timeout 25 \
    --cluster=ceph \
    --admin-daemon /var/run/ceph/${CLUSTER}-mon.${MON_NAME}.asok \
    mon_status

and the follwoing command can be used to create the keys:

  ceph --connect-timeout 25 \
    --cluster ceph \
    --name', ${KEY_ID} \
    --keyring ${KEYPATH} \
    auth get-or-create \
    ....
    ....

So for by hand and for ceph-deploy installs we have no issue here as far
as I can see.

> (But I am noticing that so far the monitor test has either been moved
> to the "create new OSDs" phase or else to the "user runs post-setup
> ceph -s and gets nothing" unsupervised phase; I'm not sure those are
> better since they don't even point directly to the monitors.)

Is this true given my answer above?

> So I think it would be fine to not automatically create keys on
> monitor startup. But Owen is saying, if you create your own bootstrap
> keys, you need to make sure you write them out to the monitor node
> filesystems before startup or they won't match. That's not true! 

Tiz so :)

> You
> need to make sure the monitor cluster has them, internally, yes. But
> once you've accomplished that, all of your monitors are going to grab
> the bootstrap keys from the cluster and write them out locally — if
> you've got different bootstrap keys on different monitors, you've
> *created multiple monitor clusters*.

Your missing a few points  here:

(A) ceph-create-keys is started on each mon currently as a side effect
of booting the mon, unless a file exists at location

  /var/lib/ceph/bootstrap-mds/ceph.keyring

At least for systemd. ceph-create-keys is looping and waiting for the
mons to reach quorum, as stated by Alfredo, so you have no time window
to add your keys before ceph-create-keys does its thing.

(B) Sure you can create mds-bootstrap, rgw-bootstrap, osd-bootstrap, and
admin keys and add them all to the mon before you start the mon's and
then ceph-create-keys will not cause confusion, but if you dont think
you want one of these keys and change your mind later, then confusion
will happen.

> That's why I'm worried about this specific thread of complaint. :)
> -Greg

Have I removed your worries yet?

Best regards

Owen

-- 
SUSE LINUX GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB
21284 (AG
Nürnberg)

Maxfeldstraße 5

90409 Nürnberg

Germany
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: The fundamental evil of "magic" in computing systems -> Was: mon daemon makes authentication side effects on startup

[Gregory Farnum]

On Thu, Apr 7, 2016 at 9:08 AM, Owen Synge <osynge@suse.com> wrote:
> On 04/07/2016 04:22 PM, Gregory Farnum wrote:
>> On Thu, Apr 7, 2016 at 5:33 AM, Alfredo Deza <adeza@redhat.com> wrote:
>>> On Wed, Apr 6, 2016 at 4:23 AM, Owen Synge <osynge@suse.com> wrote:
>>>> Dear Greg and others,
>>>>
>>>> Thankyou for your very helpful email, as it completely misses my point,
>>>> and that illustrate why this point is so important to be addressed.
>>>>
>>>> I am sure Greg has a deep understanding of this area. But I am pleased
>>>> Greg missed my points from 0-9, Greg's assumption that it is lack of
>>>> understanding on my part (which I am sure is common), clearly
>>>> illustrates where this "magic" of the side effect of starting a mon
>>>> demon becomes becomes "dark magic".
>>>>
>>>> If you object to "magic" and "dark magic" in this email please
>>>> substitute them with "side effect" and "negative consequences of side
>>>> effects" respectively, and you get a more serious reply :)
>>>>
>>>> On 04/05/2016 10:14 PM, Gregory Farnum wrote:
>>>>> I think you're fundamentally understanding how these keys come into
>>>>> existence. They aren't generated randomly on the local monitor; it
>>>>> uses get-or-create in order to fetch them (and create them if they
>>>>> don't already exist).
>>>>
>>>> I have looked at this issue in depth, and general confusion in this area
>>>> is indeed very common, so it is reasonable to expect everyone is
>>>> confused by the same thing.
>>>>
>>>> In my experience it is "magic" that causes admins fear, as good admins,
>>>> need to understand, because they need to understand the side effects of
>>>> any "magic", in case the "magic" is "dark", and in this case it is with
>>>> points (0) to (8) showing is indeed "dark magic".
>>>>
>>>> Lets be specific:
>>>>
>>>> Fetch and create are fundamentally different in side effects when doing
>>>> deployment. Lets be clear, when ceph does a "fetch" of a key, is not I
>>>> believe and issue, but when ceph uses magic to "create" keys, it can
>>>> often cause side effects. Hence the process to "create" a key should
>>>> only occur when its asked to be done.
>>>>
>>>> The current get-or-create keyrings as a side effect of booting a mon
>>>> makes many issues (points 0-8 may not be all the issues, just ones that
>>>> spring to my mind). If the booting of a mon only did a fetch I would
>>>> feel we could resolve all my point except (2) and (9) sadly a boot of a
>>>> mon will also do a create keys where this "magic" starts to become very
>>>> "dark" indeed.
>>>>
>>>>> So maybe it's difficult to pre-generate your own keys and plug them
>>>>> into the system (I don't remember where the initial values come from
>>>>> in standard deployment scenarios),
>>>>
>>>> See my reply to John as to how you can deploy ceph without ceph-create-keys.
>>>>
>>>>> but once they're set up you don't
>>>>> need to carefully install your values on all the monitor nodes — they
>>>>> will fetch the correct values from the monitor cluster.
>>>>
>>>> I am objecting to the side effect of booting the mon and that process
>>>> creating keys that where not asked for, potentially causing valid
>>>> osd-bootstrap, rgw-bootstrap or mds-bootstrap to fail authentication as
>>>> invalid ones have been created as a side effect of starting the mon daemons.
>>>
>>> That has been a *major* pain point in all deployment strategies
>>> (ceph-deploy, ceph-ansible, ceph-installer, manual deployment)
>>> I've tried: at some point a monitor is created and started and the
>>> whole thing hangs forever because the keys are being helpfully
>>> get-or-created for you but for $reasons it is unable to do so and
>>> waits indefinitely.
>>
>> I don't have any attachment to ceph-create-keys and the magic itself,
>> but this right here is my main concern about changing things: the
>> $reasons for failure being cited here were never problems with
>> ceph-create-keys itself; it was just the first part of the process
>> which required a functioning monitor quorum, and so it functioned as a
>> (not very good) break point. I have very little personal interest in
>> the configuration systems and would be vaguely more content if that
>> "test your monitors are speaking to each other" step was both distinct
>> from the keys stage and more explicit/debuggable.
>
> That is a good point.
>
> Fortunately we already have a command that does not require authentication:
>
>   ceph --connect-timeout 25 \
>     --cluster=ceph \
>     --admin-daemon /var/run/ceph/${CLUSTER}-mon.${MON_NAME}.asok \
>     mon_status
>
> and the follwoing command can be used to create the keys:
>
>   ceph --connect-timeout 25 \
>     --cluster ceph \
>     --name', ${KEY_ID} \
>     --keyring ${KEYPATH} \
>     auth get-or-create \
>     ....
>     ....
>
> So for by hand and for ceph-deploy installs we have no issue here as far
> as I can see.
>
>> (But I am noticing that so far the monitor test has either been moved
>> to the "create new OSDs" phase or else to the "user runs post-setup
>> ceph -s and gets nothing" unsupervised phase; I'm not sure those are
>> better since they don't even point directly to the monitors.)
>
> Is this true given my answer above?
>
>> So I think it would be fine to not automatically create keys on
>> monitor startup. But Owen is saying, if you create your own bootstrap
>> keys, you need to make sure you write them out to the monitor node
>> filesystems before startup or they won't match. That's not true!
>
> Tiz so :)
>
>> You
>> need to make sure the monitor cluster has them, internally, yes. But
>> once you've accomplished that, all of your monitors are going to grab
>> the bootstrap keys from the cluster and write them out locally — if
>> you've got different bootstrap keys on different monitors, you've
>> *created multiple monitor clusters*.
>
> Your missing a few points  here:
>
> (A) ceph-create-keys is started on each mon currently as a side effect
> of booting the mon, unless a file exists at location
>
>   /var/lib/ceph/bootstrap-mds/ceph.keyring
>
> At least for systemd. ceph-create-keys is looping and waiting for the
> mons to reach quorum, as stated by Alfredo, so you have no time window
> to add your keys before ceph-create-keys does its thing.

Ah, yeah, if we don't have an injection step that would be annoying. I
guess I was thinking back to last time I was dealing with Chef
cookbooks, but we didn't have the create-keys stuff then.

>
> (B) Sure you can create mds-bootstrap, rgw-bootstrap, osd-bootstrap, and
> admin keys and add them all to the mon before you start the mon's and
> then ceph-create-keys will not cause confusion, but if you dont think
> you want one of these keys and change your mind later, then confusion
> will happen.
>
>> That's why I'm worried about this specific thread of complaint. :)
>> -Greg
>
> Have I removed your worries yet?

Yeah, I was just not certain about the way it was being phrased. Sounds good!
-Greg
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: The fundamental evil of "magic" in computing systems -> Was: mon daemon makes authentication side effects on startup

[Mark Nelson]

On 04/06/2016 03:23 AM, Owen Synge wrote:
> Dear Greg and others,
>
> Thankyou for your very helpful email, as it completely misses my point,
> and that illustrate why this point is so important to be addressed.
>
> I am sure Greg has a deep understanding of this area. But I am pleased
> Greg missed my points from 0-9, Greg's assumption that it is lack of
> understanding on my part (which I am sure is common), clearly
> illustrates where this "magic" of the side effect of starting a mon
> demon becomes becomes "dark magic".
>
> If you object to "magic" and "dark magic" in this email please
> substitute them with "side effect" and "negative consequences of side
> effects" respectively, and you get a more serious reply :)
>

FWIW, I wanted to chime in and say that anything we can do to generally 
reduce instances of "dark magic" like this would be fantastic.

Back when mkcephfs was retired a couple of years ago I had to decide 
what I should replace it with in CBT.  Ultimately it was concern over 
issues like this that lead me to utilize the underlying key/mon/osd 
creation tools directly.  Your point about confusion is totally valid. 
I use ceph-authtool and had only a vague idea that ceph-create-keys even 
existed (and certainly didn't realize the behavior your describing).  I 
create ceph clusters (using CBT) to test performance pretty much daily!

Looking at the documentation, it's pretty easy to miss what's going on:

http://docs.ceph.com/docs/master/man/8/ceph-create-keys/

ceph-authtool is a little better documented:

http://docs.ceph.com/docs/hammer/man/8/ceph-authtool/

It *is* scary when software behaves in mysterious ways.  It doesn't 
invoke trust and it's not the kind of first impression to make with 
already paranoid sysadmins (Being a paranoid ex-sysadmin myself).  I 
think our heart was in the right place to try to reduce the number of 
steps required in ceph-deploy, but it can't come at the expense of 
introducing ambiguity and complexity like this.

Anyway, that's my 2C.

Mark

Re: The fundamental evil of "magic" in computing systems -> Was: mon daemon makes authentication side effects on startup

[Owen Synge]

On 04/07/2016 05:43 PM, Sage Weil wrote:
> On Thu, 7 Apr 2016, Owen Synge wrote:
>> On 04/07/2016 04:03 PM, Sage Weil wrote:
>>> On Thu, 7 Apr 2016, Owen Synge wrote:
>>>> Hi Sage,
>>>>
>>>> On 04/07/2016 02:26 PM, Sage Weil wrote:
>>>>> Hi Owen,
>>>>>
>>>>> I never really liked ceph-create-keys either, but it simplified the 
>>>>> deployment process.  
>>>>
>>>> I would propose we do this in two stages.
>>>>
>>>> (A) Remove calling the command from the init scripts as a side effect of
>>>> starting the mon.
>>>>
>>>> This allows us to get most of the issues solved.
>>>>
>>>> (B) Remove the command.
>>>>
>>>> This is the long term goal, which is not as urgent in my opinion but
>>>> others may disagree.
>>>
>>> Works for me.  We just need to change ceph-deploy and get the other 
>>> install/deploy tool folks on board before A.
>>
>> Are you intending to get this into Jewel?
>>
>> I had assumed this would only be done on master, and only come into the
>> next release.
> 
> Yeah, too late for jewel.
> 
>> As a change to master I felt that we could just do (A) as soon as
>> ceph-deploy works without the mon boot up scripts calling
>> ceph-create-keys, ideally without having  ceph-create-keys in
>> ceph-deploy's process.
>>
>> We can then file bugs as needed against other install processes that
>> depend on ceph-create-keys, and they can test against master.
> 
> That works for me.
> 
> sage

Great,

I have a fix, that is tested and working for ceph-deploy without
depending upon ceph-create-keys based upon a rewrite of the method

  ceph-deploy gatherkeys mon-node-01 mon-node-02 mon-node-03

Works nicely for the old and new methods, and seems to have little
impact apart from no new keys are wrote to disk on the mon nodes. OSD's
and rgw can be deployed without change, (I haven’t tested mds)

Previous behavior with the admin keys being deployed can be achieved
simply by executing:

  ceph-deploy admin mon-node-01 mon-node-02 mon-node-03

If we definitely what to enforce the admin code being persisted on all
mon nodes can be changed later, but I think its cleaner if we do not.

I will submit a PR on Monday.

Best wishes

Owen
-- 
SUSE LINUX GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB
21284 (AG
Nürnberg)

Maxfeldstraße 5

90409 Nürnberg

Germany
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: The fundamental evil of "magic" in computing systems -> Was: mon daemon makes authentication side effects on startup

[Owen Synge]

On 04/08/2016 10:57 PM, Owen Synge wrote:
> On 04/07/2016 05:43 PM, Sage Weil wrote:
>> On Thu, 7 Apr 2016, Owen Synge wrote:
>>> On 04/07/2016 04:03 PM, Sage Weil wrote:
>>>> On Thu, 7 Apr 2016, Owen Synge wrote:
>>>>> Hi Sage,
>>>>>
>>>>> On 04/07/2016 02:26 PM, Sage Weil wrote:
>>>>>> Hi Owen,
>>>>>>
>>>>>> I never really liked ceph-create-keys either, but it simplified the 
>>>>>> deployment process.  
>>>>>
>>>>> I would propose we do this in two stages.
>>>>>
>>>>> (A) Remove calling the command from the init scripts as a side effect of
>>>>> starting the mon.
>>>>>
>>>>> This allows us to get most of the issues solved.
>>>>>
>>>>> (B) Remove the command.
>>>>>
>>>>> This is the long term goal, which is not as urgent in my opinion but
>>>>> others may disagree.
>>>>
>>>> Works for me.  We just need to change ceph-deploy and get the other 
>>>> install/deploy tool folks on board before A.
>>>
>>> Are you intending to get this into Jewel?
>>>
>>> I had assumed this would only be done on master, and only come into the
>>> next release.
>>
>> Yeah, too late for jewel.
>>
>>> As a change to master I felt that we could just do (A) as soon as
>>> ceph-deploy works without the mon boot up scripts calling
>>> ceph-create-keys, ideally without having  ceph-create-keys in
>>> ceph-deploy's process.
>>>
>>> We can then file bugs as needed against other install processes that
>>> depend on ceph-create-keys, and they can test against master.
>>
>> That works for me.
>>
>> sage
> 
> Great,
> 
> I have a fix, that is tested and working for ceph-deploy without
> depending upon ceph-create-keys based upon a rewrite of the method
> 
>   ceph-deploy gatherkeys mon-node-01 mon-node-02 mon-node-03
> 
> Works nicely for the old and new methods, and seems to have little
> impact apart from no new keys are wrote to disk on the mon nodes. OSD's
> and rgw can be deployed without change, (I haven’t tested mds)
> 
> Previous behavior with the admin keys being deployed can be achieved
> simply by executing:
> 
>   ceph-deploy admin mon-node-01 mon-node-02 mon-node-03
> 
> If we definitely what to enforce the admin code being persisted on all
> mon nodes can be changed later, but I think its cleaner if we do not.
> 
> I will submit a PR on Monday.

ceph-deploy bug raised:

http://tracker.ceph.com/issues/15451

PR submitted:

https://github.com/ceph/ceph-deploy/pull/393

Best regards

Owen

-- 
SUSE LINUX GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB
21284 (AG
Nürnberg)

Maxfeldstraße 5

90409 Nürnberg

Germany
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: The fundamental evil of "magic" in computing systems -> Was: mon daemon makes authentication side effects on startup

[Sage Weil]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 2886 bytes --]

On Mon, 11 Apr 2016, Owen Synge wrote:
> On 04/08/2016 10:57 PM, Owen Synge wrote:
> > On 04/07/2016 05:43 PM, Sage Weil wrote:
> >> On Thu, 7 Apr 2016, Owen Synge wrote:
> >>> On 04/07/2016 04:03 PM, Sage Weil wrote:
> >>>> On Thu, 7 Apr 2016, Owen Synge wrote:
> >>>>> Hi Sage,
> >>>>>
> >>>>> On 04/07/2016 02:26 PM, Sage Weil wrote:
> >>>>>> Hi Owen,
> >>>>>>
> >>>>>> I never really liked ceph-create-keys either, but it simplified the 
> >>>>>> deployment process.  
> >>>>>
> >>>>> I would propose we do this in two stages.
> >>>>>
> >>>>> (A) Remove calling the command from the init scripts as a side effect of
> >>>>> starting the mon.
> >>>>>
> >>>>> This allows us to get most of the issues solved.
> >>>>>
> >>>>> (B) Remove the command.
> >>>>>
> >>>>> This is the long term goal, which is not as urgent in my opinion but
> >>>>> others may disagree.
> >>>>
> >>>> Works for me.  We just need to change ceph-deploy and get the other 
> >>>> install/deploy tool folks on board before A.
> >>>
> >>> Are you intending to get this into Jewel?
> >>>
> >>> I had assumed this would only be done on master, and only come into the
> >>> next release.
> >>
> >> Yeah, too late for jewel.
> >>
> >>> As a change to master I felt that we could just do (A) as soon as
> >>> ceph-deploy works without the mon boot up scripts calling
> >>> ceph-create-keys, ideally without having  ceph-create-keys in
> >>> ceph-deploy's process.
> >>>
> >>> We can then file bugs as needed against other install processes that
> >>> depend on ceph-create-keys, and they can test against master.
> >>
> >> That works for me.
> >>
> >> sage
> > 
> > Great,
> > 
> > I have a fix, that is tested and working for ceph-deploy without
> > depending upon ceph-create-keys based upon a rewrite of the method
> > 
> >   ceph-deploy gatherkeys mon-node-01 mon-node-02 mon-node-03
> > 
> > Works nicely for the old and new methods, and seems to have little
> > impact apart from no new keys are wrote to disk on the mon nodes. OSD's
> > and rgw can be deployed without change, (I haven’t tested mds)
> > 
> > Previous behavior with the admin keys being deployed can be achieved
> > simply by executing:
> > 
> >   ceph-deploy admin mon-node-01 mon-node-02 mon-node-03
> > 
> > If we definitely what to enforce the admin code being persisted on all
> > mon nodes can be changed later, but I think its cleaner if we do not.
> > 
> > I will submit a PR on Monday.
> 
> ceph-deploy bug raised:
> 
> http://tracker.ceph.com/issues/15451
> 
> PR submitted:
> 
> https://github.com/ceph/ceph-deploy/pull/393

Hey Owen-

Now that jewel is out, now would be a good time to make this change.  The 
ceph-deploy pr looks basically ready to go, minus a doc piece and a run 
through the ceph-deploy suite.  Yuri can probably handle the 
latter.

Then we can do the ceph.git changes to kill the ceph-create-keys task...

sage

Re: The fundamental evil of "magic" in computing systems -> Was: mon daemon makes authentication side effects on startup

[Owen Synge]

On 05/12/2016 03:06 PM, Sage Weil wrote:
> On Mon, 11 Apr 2016, Owen Synge wrote:
>> On 04/08/2016 10:57 PM, Owen Synge wrote:
>>> On 04/07/2016 05:43 PM, Sage Weil wrote:
>>>> On Thu, 7 Apr 2016, Owen Synge wrote:
>>>>> On 04/07/2016 04:03 PM, Sage Weil wrote:
>>>>>> On Thu, 7 Apr 2016, Owen Synge wrote:
>>>>>>> Hi Sage,
>>>>>>>
>>>>>>> On 04/07/2016 02:26 PM, Sage Weil wrote:
>>>>>>>> Hi Owen,
>>>>>>>>
>>>>>>>> I never really liked ceph-create-keys either, but it simplified the 
>>>>>>>> deployment process.  
>>>>>>>
>>>>>>> I would propose we do this in two stages.
>>>>>>>
>>>>>>> (A) Remove calling the command from the init scripts as a side effect of
>>>>>>> starting the mon.
>>>>>>>
>>>>>>> This allows us to get most of the issues solved.
>>>>>>>
>>>>>>> (B) Remove the command.
>>>>>>>
>>>>>>> This is the long term goal, which is not as urgent in my opinion but
>>>>>>> others may disagree.
>>>>>>
>>>>>> Works for me.  We just need to change ceph-deploy and get the other 
>>>>>> install/deploy tool folks on board before A.
>>>>>
>>>>> Are you intending to get this into Jewel?
>>>>>
>>>>> I had assumed this would only be done on master, and only come into the
>>>>> next release.
>>>>
>>>> Yeah, too late for jewel.
>>>>
>>>>> As a change to master I felt that we could just do (A) as soon as
>>>>> ceph-deploy works without the mon boot up scripts calling
>>>>> ceph-create-keys, ideally without having  ceph-create-keys in
>>>>> ceph-deploy's process.
>>>>>
>>>>> We can then file bugs as needed against other install processes that
>>>>> depend on ceph-create-keys, and they can test against master.
>>>>
>>>> That works for me.
>>>>
>>>> sage
>>>
>>> Great,
>>>
>>> I have a fix, that is tested and working for ceph-deploy without
>>> depending upon ceph-create-keys based upon a rewrite of the method
>>>
>>>   ceph-deploy gatherkeys mon-node-01 mon-node-02 mon-node-03
>>>
>>> Works nicely for the old and new methods, and seems to have little
>>> impact apart from no new keys are wrote to disk on the mon nodes. OSD's
>>> and rgw can be deployed without change, (I haven’t tested mds)
>>>
>>> Previous behavior with the admin keys being deployed can be achieved
>>> simply by executing:
>>>
>>>   ceph-deploy admin mon-node-01 mon-node-02 mon-node-03
>>>
>>> If we definitely what to enforce the admin code being persisted on all
>>> mon nodes can be changed later, but I think its cleaner if we do not.
>>>
>>> I will submit a PR on Monday.
>>
>> ceph-deploy bug raised:
>>
>> http://tracker.ceph.com/issues/15451
>>
>> PR submitted:
>>
>> https://github.com/ceph/ceph-deploy/pull/393
> 
> Hey Owen-
> 
> Now that jewel is out, now would be a good time to make this change.  The 
> ceph-deploy pr looks basically ready to go, minus a doc piece and a run 
> through the ceph-deploy suite.  Yuri can probably handle the 
> latter.
> 
> Then we can do the ceph.git changes to kill the ceph-create-keys task...

Dear Sage,

Sorry for the delay, I had a big pile of downstream work and test suite
development to do for my salt work, I have now added some documentation.

I hope Yuri can do the latter as I really dont know "the ceph-deploy suite".

Best wishes

Owen


--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: The fundamental evil of "magic" in computing systems -> Was: mon daemon makes authentication side effects on startup

[Owen Synge]

On 05/20/2016 03:01 PM, Owen Synge wrote:
> On 05/12/2016 03:06 PM, Sage Weil wrote:
>> On Mon, 11 Apr 2016, Owen Synge wrote:
>>> On 04/08/2016 10:57 PM, Owen Synge wrote:
>>>> On 04/07/2016 05:43 PM, Sage Weil wrote:
>>>>> On Thu, 7 Apr 2016, Owen Synge wrote:
>>>>>> On 04/07/2016 04:03 PM, Sage Weil wrote:
>>>>>>> On Thu, 7 Apr 2016, Owen Synge wrote:
>>>>>>>> Hi Sage,
>>>>>>>>
>>>>>>>> On 04/07/2016 02:26 PM, Sage Weil wrote:
>>>>>>>>> Hi Owen,
>>>>>>>>>
>>>>>>>>> I never really liked ceph-create-keys either, but it simplified the 
>>>>>>>>> deployment process.  
>>>>>>>>
>>>>>>>> I would propose we do this in two stages.
>>>>>>>>
>>>>>>>> (A) Remove calling the command from the init scripts as a side effect of
>>>>>>>> starting the mon.
>>>>>>>>
>>>>>>>> This allows us to get most of the issues solved.
>>>>>>>>
>>>>>>>> (B) Remove the command.
>>>>>>>>
>>>>>>>> This is the long term goal, which is not as urgent in my opinion but
>>>>>>>> others may disagree.
>>>>>>>
>>>>>>> Works for me.  We just need to change ceph-deploy and get the other 
>>>>>>> install/deploy tool folks on board before A.
>>>>>>
>>>>>> Are you intending to get this into Jewel?
>>>>>>
>>>>>> I had assumed this would only be done on master, and only come into the
>>>>>> next release.
>>>>>
>>>>> Yeah, too late for jewel.
>>>>>
>>>>>> As a change to master I felt that we could just do (A) as soon as
>>>>>> ceph-deploy works without the mon boot up scripts calling
>>>>>> ceph-create-keys, ideally without having  ceph-create-keys in
>>>>>> ceph-deploy's process.
>>>>>>
>>>>>> We can then file bugs as needed against other install processes that
>>>>>> depend on ceph-create-keys, and they can test against master.
>>>>>
>>>>> That works for me.
>>>>>
>>>>> sage
>>>>
>>>> Great,
>>>>
>>>> I have a fix, that is tested and working for ceph-deploy without
>>>> depending upon ceph-create-keys based upon a rewrite of the method
>>>>
>>>>   ceph-deploy gatherkeys mon-node-01 mon-node-02 mon-node-03
>>>>
>>>> Works nicely for the old and new methods, and seems to have little
>>>> impact apart from no new keys are wrote to disk on the mon nodes. OSD's
>>>> and rgw can be deployed without change, (I haven’t tested mds)
>>>>
>>>> Previous behavior with the admin keys being deployed can be achieved
>>>> simply by executing:
>>>>
>>>>   ceph-deploy admin mon-node-01 mon-node-02 mon-node-03
>>>>
>>>> If we definitely what to enforce the admin code being persisted on all
>>>> mon nodes can be changed later, but I think its cleaner if we do not.
>>>>
>>>> I will submit a PR on Monday.
>>>
>>> ceph-deploy bug raised:
>>>
>>> http://tracker.ceph.com/issues/15451
>>>
>>> PR submitted:
>>>
>>> https://github.com/ceph/ceph-deploy/pull/393
>>
>> Hey Owen-
>>
>> Now that jewel is out, now would be a good time to make this change.  The 
>> ceph-deploy pr looks basically ready to go, minus a doc piece and a run 
>> through the ceph-deploy suite.  Yuri can probably handle the 
>> latter.
>>
>> Then we can do the ceph.git changes to kill the ceph-create-keys task...
> 
> Dear Sage,
> 
> Sorry for the delay, I had a big pile of downstream work and test suite
> development to do for my salt work, I have now added some documentation.
> 
> I hope Yuri can do the latter as I really dont know "the ceph-deploy suite".

Great news ceph-deploy master now has the PR merged in so that
ceph-deploy can now work without any magic involved with ceph-create-keys.

I will now make a patch to ceph proper to remove ceph-create-keys from
the init scripts. Hopefully this will move things forward and we can get
this patch merged relatively quickly.

Please let me know if I am doing anything wrong.

Best regards

Owen


--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: The fundamental evil of "magic" in computing systems -> Was: mon daemon makes authentication side effects on startup

[Sage Weil]

On Wed, 25 May 2016, Owen Synge wrote:
> Great news ceph-deploy master now has the PR merged in so that
> ceph-deploy can now work without any magic involved with ceph-create-keys.
> 
> I will now make a patch to ceph proper to remove ceph-create-keys from
> the init scripts. Hopefully this will move things forward and we can get
> this patch merged relatively quickly.
> 
> Please let me know if I am doing anything wrong.

Great!

Alfredo, are there changes we need to ceph-ansible for this too?  Puppet?  
Chef?

(This is post-jewel, so it can wait a bit if necessary.)

sage

Re: The fundamental evil of "magic" in computing systems -> Was: mon daemon makes authentication side effects on startup

[Owen Synge]

On 05/25/2016 02:45 PM, Sage Weil wrote:
> On Wed, 25 May 2016, Owen Synge wrote:
>> Great news ceph-deploy master now has the PR merged in so that
>> ceph-deploy can now work without any magic involved with ceph-create-keys.
>>
>> I will now make a patch to ceph proper to remove ceph-create-keys from
>> the init scripts. Hopefully this will move things forward and we can get
>> this patch merged relatively quickly.
>>
>> Please let me know if I am doing anything wrong.
> 
> Great!

I created a bug report against ceph:

    http://tracker.ceph.com/issues/16036

I have a patch now against ceph:

    https://github.com/ceph/ceph/pull/9345

But it seems to be failing on a test, I suspect the issue is in the test
suite as the failing command is:

    ceph osd setcrushmap -i /tmp/cephtool26589/map

Now the mon servers will not definitely have admin keyrings on them this
could cause the above command to fail.

Any help in the right direction appreciated.

> Alfredo, are there changes we need to ceph-ansible for this too?  Puppet?  
> Chef?
> 
> (This is post-jewel, so it can wait a bit if necessary.)

I hope we wont have to block merging this patch until they are all fixed
too?

Best regards

Owen

Re: The fundamental evil of "magic" in computing systems -> Was: mon daemon makes authentication side effects on startup

[Alfredo Deza]

On Wed, May 25, 2016 at 8:45 AM, Sage Weil <sweil@redhat.com> wrote:
> On Wed, 25 May 2016, Owen Synge wrote:
>> Great news ceph-deploy master now has the PR merged in so that
>> ceph-deploy can now work without any magic involved with ceph-create-keys.
>>
>> I will now make a patch to ceph proper to remove ceph-create-keys from
>> the init scripts. Hopefully this will move things forward and we can get
>> this patch merged relatively quickly.
>>
>> Please let me know if I am doing anything wrong.
>
> Great!
>
> Alfredo, are there changes we need to ceph-ansible for this too?  Puppet?
> Chef?

We don't have anything to deal with this in ceph-ansible (yet). We
should make some noise about this change when the PR is up
so that Chef/Puppet users+maintainers are aware of the required changes.


>
> (This is post-jewel, so it can wait a bit if necessary.)
>
> sage