Understanding encrypted OSD's and cephx

Understanding encrypted OSD's and cephx

[Owen Synge]

Dear all,

I have been trying to get to grips with understanding cephx and
encrypted OSD's. WRT this bug:

    http://tracker.ceph.com/issues/17833

Please correct me here if I am wrong:

Installing a encrypted OSD with ceph-deploy as:

  ceph-deploy osd create --dmcrypt ceph-node3:vdc

Will fail unless you first run:

  ceph-deploy osd admin ceph-node3

I have been digging further:

(1) Encrypted OSD's are mounted using a key.
(2) This key is retrieved from the mon, using a cephx key.
(3) This cephx key is generated with the capability to retrieve the key
from the mon.

The reason being the admin key is needed is to make keys with correct
capabilities.

To set this up I have played with the command.

With the admin key you can:

    /usr/bin/ceph --connect-timeout 20  config-key list
    /usr/bin/ceph --connect-timeout 20  config-key put  key value
    /usr/bin/ceph --connect-timeout 20  config-key get key value

So we dont want to put the admin keyring on the OSD's with encrypted
OSD's as all encrypted OSD keys can be retrived using the admin keyring.

So I then looked at locking down the capabilities:

I can generate a key with only the ability to get a single key:

   /usr/bin/ceph auth get-or-create \
       client.osd-lockbox.d967ec85-4bd5-44c5-b20c-fc6864f6c7c0 \
       mon 'allow command "config-key get" with key="Key_name"' \
       > /tmp/foo

I can then use this key to get the key's value:

    /usr/bin/ceph --keyring /tmp/foo  --name \
        client.osd-lockbox.d967ec85-4bd5-44c5-b20c-fc6864f6c7c0 \
       config-key get Key_name

I can also create a key that can only place values for a key:

    /usr/bin/ceph auth get-or-create client.bar mon \
        'allow command "config-key put"' > /tmp/bar

And this can upload to that value:

    /usr/bin/ceph auth get-or-create client.nting mon \
        'allow command "config-key put" with \
        key="Key_name"'

And it can only set this value:

    /usr/bin/ceph --connect-timeout 20 --keyring /tmp/bar --name \
        client.bar   config-key put Key_name Key_value

I can also use the mon keyrign ratehr than the admin key to create these
keys:

    /usr/bin/ceph --connect-timeout 20 --keyring \
         /var/lib/ceph/mon/ceph-ceph-node1/keyring \
        --name mon. auth get-or-create client.jam mon \
        'allow command "config-key put" with \
        key="Key_name"'

So it seems to me, that we can potentially resolve bug:

    http://tracker.ceph.com/issues/17833

Without ever putting the admin key or similar high privileged key on an
encrypted OSD node.

Now where does this leave us:

(1) We either continue to expect the admin keyring to exist.
(2) We use locked down keys on the OSD nodes.

Following option (2) We can take 1 of 2 approaches:

(A) ceph-deploy shoudl set up the encryption key and value on the mon
node, generate a value readonly key to the OSD node then deploy the
encrypted OSD.
(B) ceph-deploy should set up a read and write keys for the OSD
encryption value, then ship these to OSD node to deploy the OSD, then
remove the write key.

In both these cases ceph-deploy will need to contact the osd node, get
the partion UUID, process this on the mon node, then finish the process
of deploying an encrypted OSD on the OSD node.

Option (A) seems simpler here.

Does anyone see a better way assuming (2)?

Best regards

Owen

Re: Understanding encrypted OSD's and cephx

[Sage Weil]

On Mon, 14 Nov 2016, Owen Synge wrote:
> Dear all,
> 
> I have been trying to get to grips with understanding cephx and
> encrypted OSD's. WRT this bug:
> 
>     http://tracker.ceph.com/issues/17833
> 
> Please correct me here if I am wrong:
> 
> Installing a encrypted OSD with ceph-deploy as:
> 
>   ceph-deploy osd create --dmcrypt ceph-node3:vdc
> 
> Will fail unless you first run:
> 
>   ceph-deploy osd admin ceph-node3
> 
> I have been digging further:
> 
> (1) Encrypted OSD's are mounted using a key.
> (2) This key is retrieved from the mon, using a cephx key.
> (3) This cephx key is generated with the capability to retrieve the key
> from the mon.
> 
> The reason being the admin key is needed is to make keys with correct
> capabilities.
> 
> To set this up I have played with the command.
> 
> With the admin key you can:
> 
>     /usr/bin/ceph --connect-timeout 20  config-key list
>     /usr/bin/ceph --connect-timeout 20  config-key put  key value
>     /usr/bin/ceph --connect-timeout 20  config-key get key value
> 
> So we dont want to put the admin keyring on the OSD's with encrypted
> OSD's as all encrypted OSD keys can be retrived using the admin keyring.
> 
> So I then looked at locking down the capabilities:
> 
> I can generate a key with only the ability to get a single key:
> 
>    /usr/bin/ceph auth get-or-create \
>        client.osd-lockbox.d967ec85-4bd5-44c5-b20c-fc6864f6c7c0 \
>        mon 'allow command "config-key get" with key="Key_name"' \
>        > /tmp/foo
> 
> I can then use this key to get the key's value:
> 
>     /usr/bin/ceph --keyring /tmp/foo  --name \
>         client.osd-lockbox.d967ec85-4bd5-44c5-b20c-fc6864f6c7c0 \
>        config-key get Key_name
> 
> I can also create a key that can only place values for a key:
> 
>     /usr/bin/ceph auth get-or-create client.bar mon \
>         'allow command "config-key put"' > /tmp/bar
> 
> And this can upload to that value:
> 
>     /usr/bin/ceph auth get-or-create client.nting mon \
>         'allow command "config-key put" with \
>         key="Key_name"'
> 
> And it can only set this value:
> 
>     /usr/bin/ceph --connect-timeout 20 --keyring /tmp/bar --name \
>         client.bar   config-key put Key_name Key_value
> 
> I can also use the mon keyrign ratehr than the admin key to create these
> keys:
> 
>     /usr/bin/ceph --connect-timeout 20 --keyring \
>          /var/lib/ceph/mon/ceph-ceph-node1/keyring \
>         --name mon. auth get-or-create client.jam mon \
>         'allow command "config-key put" with \
>         key="Key_name"'
> 
> So it seems to me, that we can potentially resolve bug:
> 
>     http://tracker.ceph.com/issues/17833
> 
> Without ever putting the admin key or similar high privileged key on an
> encrypted OSD node.
> 
> Now where does this leave us:
> 
> (1) We either continue to expect the admin keyring to exist.
> (2) We use locked down keys on the OSD nodes.
> 
> Following option (2) We can take 1 of 2 approaches:
> 
> (A) ceph-deploy shoudl set up the encryption key and value on the mon
> node, generate a value readonly key to the OSD node then deploy the
> encrypted OSD.
> (B) ceph-deploy should set up a read and write keys for the OSD
> encryption value, then ship these to OSD node to deploy the OSD, then
> remove the write key.
> 
> In both these cases ceph-deploy will need to contact the osd node, get
> the partion UUID, process this on the mon node, then finish the process
> of deploying an encrypted OSD on the OSD node.
> 
> Option (A) seems simpler here.
> 
> Does anyone see a better way assuming (2)?

I think ideally we want this to be permitted as part of the osd-bootstrap 
key.  Then we can hopefully improve things to make provisioning tools 
temporarily issue a bootstrap key, create the osds, then remove the 
bootstrep key.

However, it isn't really possible to sufficiently restrict the commands to 
do the bootstrap with a generic key.  You need to know things like the 
uuid to restrict the key properly.  And it's tedious.  And even the 
current bootstrap process isn't fully secure, see MonCap.cc:

  if (profile == "bootstrap-mds") {
...
    profile_grants.push_back(MonCapGrant("auth get-or-create"));  // FIXME: this can expose other mds keys

I think the fix for this is to give up on trying to write a capability 
that lets you do the necessary steps to set up an OSD or MDS or whatever, 
and instead define a new monitor command to do all of this in one go.  
For example, 'osd bootstrap' could take the necessary arguments (e.g., the 
uuid and local key) and in one step set up the various keys for you.  I 
think that's going to be the most maintainable and sane solution going 
forward.

In the meantime, though... let's just change the ceph deploy test to 
install the admin key so that we can make the tests pass?  And set teh bug 
severity on this so that we make sure it's addressed in a complete way 
soon.

sage

Re: Understanding encrypted OSD's and cephx

[Owen Synge]

On 11/14/2016 05:17 PM, Sage Weil wrote:
> On Mon, 14 Nov 2016, Owen Synge wrote:
>> Dear all,
>>
>> I have been trying to get to grips with understanding cephx and
>> encrypted OSD's. WRT this bug:
>>
>>     http://tracker.ceph.com/issues/17833
>>
>> Please correct me here if I am wrong:
>>
>> Installing a encrypted OSD with ceph-deploy as:
>>
>>   ceph-deploy osd create --dmcrypt ceph-node3:vdc
>>
>> Will fail unless you first run:
>>
>>   ceph-deploy osd admin ceph-node3
>>
>> I have been digging further:
>>
>> (1) Encrypted OSD's are mounted using a key.
>> (2) This key is retrieved from the mon, using a cephx key.
>> (3) This cephx key is generated with the capability to retrieve the key
>> from the mon.
>>
>> The reason being the admin key is needed is to make keys with correct
>> capabilities.
>>
>> To set this up I have played with the command.
>>
>> With the admin key you can:
>>
>>     /usr/bin/ceph --connect-timeout 20  config-key list
>>     /usr/bin/ceph --connect-timeout 20  config-key put  key value
>>     /usr/bin/ceph --connect-timeout 20  config-key get key value
>>
>> So we dont want to put the admin keyring on the OSD's with encrypted
>> OSD's as all encrypted OSD keys can be retrieved using the admin keyring.
>>
>> So I then looked at locking down the capabilities:
>>
>> I can generate a key with only the ability to get a single key:
>>
>>    /usr/bin/ceph auth get-or-create \
>>        client.osd-lockbox.d967ec85-4bd5-44c5-b20c-fc6864f6c7c0 \
>>        mon 'allow command "config-key get" with key="Key_name"' \
>>        > /tmp/foo
>>
>> I can then use this key to get the key's value:
>>
>>     /usr/bin/ceph --keyring /tmp/foo  --name \
>>         client.osd-lockbox.d967ec85-4bd5-44c5-b20c-fc6864f6c7c0 \
>>        config-key get Key_name
>>
>> I can also create a key that can only place values for a key:
>>
>>     /usr/bin/ceph auth get-or-create client.bar mon \
>>         'allow command "config-key put"' > /tmp/bar
>>
>> And this can upload to that value:
>>
>>     /usr/bin/ceph auth get-or-create client.nting mon \
>>         'allow command "config-key put" with \
>>         key="Key_name"'
>>
>> And it can only set this value:
>>
>>     /usr/bin/ceph --connect-timeout 20 --keyring /tmp/bar --name \
>>         client.bar   config-key put Key_name Key_value
>>
>> I can also use the mon keyrign ratehr than the admin key to create these
>> keys:
>>
>>     /usr/bin/ceph --connect-timeout 20 --keyring \
>>          /var/lib/ceph/mon/ceph-ceph-node1/keyring \
>>         --name mon. auth get-or-create client.jam mon \
>>         'allow command "config-key put" with \
>>         key="Key_name"'
>>
>> So it seems to me, that we can potentially resolve bug:
>>
>>     http://tracker.ceph.com/issues/17833
>>
>> Without ever putting the admin key or similar high privileged key on an
>> encrypted OSD node.
>>
>> Now where does this leave us:
>>
>> (1) We either continue to expect the admin keyring to exist.
>> (2) We use locked down keys on the OSD nodes.
>>
>> Following option (2) We can take 1 of 2 approaches:
>>
>> (A) ceph-deploy shoudl set up the encryption key and value on the mon
>> node, generate a value readonly key to the OSD node then deploy the
>> encrypted OSD.
>> (B) ceph-deploy should set up a read and write keys for the OSD
>> encryption value, then ship these to OSD node to deploy the OSD, then
>> remove the write key.
>>
>> In both these cases ceph-deploy will need to contact the osd node, get
>> the partion UUID, process this on the mon node, then finish the process
>> of deploying an encrypted OSD on the OSD node.
>>
>> Option (A) seems simpler here.
>>
>> Does anyone see a better way assuming (2)?
> 
> I think ideally we want this to be permitted as part of the osd-bootstrap 
> key.  Then we can hopefully improve things to make provisioning tools 
> temporarily issue a bootstrap key, create the osds, then remove the 
> bootstrep key.

That sounds reasonable.

> However, it isn't really possible to sufficiently restrict the commands to 
> do the bootstrap with a generic key. 

I would agree.

I have always considered security a compromise between usability and
security as if some thing is secure and unusable, then we have no point
in making some thing secure as it will never be used. The approaches I
outlined with (A) and (B) are significantly more secure, but they are
impacting usability considerably, and possibly unusable.

> You need to know things like the
> uuid to restrict the key properly.  And it's tedious.  And even the 
> current bootstrap process isn't fully secure, see MonCap.cc:
> 
>   if (profile == "bootstrap-mds") {
> ...
>     profile_grants.push_back(MonCapGrant("auth get-or-create"));  // FIXME: this can expose other mds keys

I agree that this approach is not fully secure for exactly the above reason.

> I think the fix for this is to give up on trying to write a capability 
> that lets you do the necessary steps to set up an OSD or MDS or whatever, 

I agree, I had a look at this briefly and decided I needed to bring the
mailing list on board before I even considered trying.

> and instead define a new monitor command to do all of this in one go.  
> For example, 'osd bootstrap' could take the necessary arguments (e.g., the 
> uuid and local key) and in one step set up the various keys for you.  I 
> think that's going to be the most maintainable and sane solution going 
> forward.

Having a single command to do this sounds a good long term solution.

I would add that a replay attack is still a danger, with a one command
approach.

Using the trigger of OSD registering to prevent replay attacks, seems
like a better compromise than making each command only work once, so
setup can be repeated until success, and then since replay is prevented
other keys cannot be exposed.

> In the meantime, though... let's just change the ceph deploy test to 
> install the admin key so that we can make the tests pass?  And set teh bug 
> severity on this so that we make sure it's addressed in a complete way 
> soon.

I fully agree.

Please can you (Sage) raise the bug so this plan is executed with
appropriate timeliness, as I feel if you raise the bug it will enhance
the importance. I would suggest you also include the replay attack issue
in the bug report.

I will need help to understand how to change the functional test suite.
The change will be just to issue:

    ceph-deploy admin $TARGET_NODE

before issuing:

    ceph-deploy osd create --dmcrypt $TARGET_NODE:sdb:sdc

Best wishes

Owen

Re: Understanding encrypted OSD's and cephx

[Sage Weil]

On Tue, 15 Nov 2016, Owen Synge wrote:
> > and instead define a new monitor command to do all of this in one go.  
> > For example, 'osd bootstrap' could take the necessary arguments (e.g., the 
> > uuid and local key) and in one step set up the various keys for you.  I 
> > think that's going to be the most maintainable and sane solution going 
> > forward.
> 
> Having a single command to do this sounds a good long term solution.
> 
> I would add that a replay attack is still a danger, with a one command
> approach.
> 
> Using the trigger of OSD registering to prevent replay attacks, seems
> like a better compromise than making each command only work once, so
> setup can be repeated until success, and then since replay is prevented
> other keys cannot be exposed.

There are two options:

 (1) Make the command always instantiate a new osd id and set of keys, 
such that you get leaked/garbage state if there is a command resend, 
or

 (2) make the command properly idempotent.  The trick here is that we need 
to uniquely identify the bootstrap command by some piece of state that is 
private so that the command cannot be used to fetch secrets for the 
created osd.  This shouldn't really be a problem.

In the end, this will be significantly more secure than what we were doing 
before.  The recommended workflow will become

 (1) push bootstrap-{osd,mon,mds} key to host
 (2) create daemon (via new bootstrap command that is tailor-made for the 
purpose)
 (3) remove bootstrap key

Before we commit to this, there is an alternative model that does all key 
creation on some other trusted host, but it would mean a more significant 
rewrite of the ceph-disk tooling...

> > In the meantime, though... let's just change the ceph deploy test to 
> > install the admin key so that we can make the tests pass?  And set teh bug 
> > severity on this so that we make sure it's addressed in a complete way 
> > soon.
> 
> I fully agree.
> 
> Please can you (Sage) raise the bug so this plan is executed with
> appropriate timeliness, as I feel if you raise the bug it will enhance
> the importance. I would suggest you also include the replay attack issue
> in the bug report.
> 
> I will need help to understand how to change the functional test suite.
> The change will be just to issue:
> 
>     ceph-deploy admin $TARGET_NODE
> 
> before issuing:
> 
>     ceph-deploy osd create --dmcrypt $TARGET_NODE:sdb:sdc

Yep, that's it.  Should just be a few lines of code in 

https://github.com/ceph/ceph-qa-suite/blob/master/tasks/ceph_deploy.py

Thanks!
sage

Re: Understanding encrypted OSD's and cephx

[Owen Synge]

On 11/15/2016 03:27 PM, Sage Weil wrote:
> On Tue, 15 Nov 2016, Owen Synge wrote:
>>> and instead define a new monitor command to do all of this in one go.  
>>> For example, 'osd bootstrap' could take the necessary arguments (e.g., the 
>>> uuid and local key) and in one step set up the various keys for you.  I 
>>> think that's going to be the most maintainable and sane solution going 
>>> forward.
>>
>> Having a single command to do this sounds a good long term solution.
>>
>> I would add that a replay attack is still a danger, with a one command
>> approach.
>>
>> Using the trigger of OSD registering to prevent replay attacks, seems
>> like a better compromise than making each command only work once, so
>> setup can be repeated until success, and then since replay is prevented
>> other keys cannot be exposed.
> 
> There are two options:
> 
>  (1) Make the command always instantiate a new osd id and set of keys, 
> such that you get leaked/garbage state if there is a command resend, 
> or

Understood.

>  (2) make the command properly idempotent.  The trick here is that we need 
> to uniquely identify the bootstrap command by some piece of state that is 
> private so that the command cannot be used to fetch secrets for the 
> created osd.  This shouldn't really be a problem.

Not sure I follow this point. Can you explain.

My difficulty is idempotent commands are suitable for replay attacks,
and cant understand how we get around this with a "private" data hook, I
am probably just being dumb here, please enlighten me if you have time.

Personally I like the idea of using the partition uuid to identify
things as is done currently.

> In the end, this will be significantly more secure than what we were doing 
> before.  The recommended workflow will become
> 
>  (1) push bootstrap-{osd,mon,mds} key to host
>  (2) create daemon (via new bootstrap command that is tailor-made for the 
> purpose)
>  (3) remove bootstrap key

Agreed, for my close the replay window when OSD is registered approach,
or your point (1) and probably when I understand it (2).

> Before we commit to this, there is an alternative model that does all key 
> creation on some other trusted host, but it would mean a more significant 
> rewrite of the ceph-disk tooling...

Yes,

Forgive me for going in circles here, I think if we substitute "mon
node" with "trusted host" this is what I proposed first in this thread
but I dont like it for the below reasons.

The negative of the "trusted host" approach is:

* We would still need to get the partition UUID and present it to the
"trusted host". Then take the output from the "trusted host" back to the
OSD with the encrypted disks. This makes for inter node communication
which makes for usability issues.

* Also new ceph-disk tooling as you pointed out. (ceph-deploy also)

The positive of the "trusted host" approach:

* It is the most secure approach.

* Requires no new tooling in the mon.

* No worries about replay attacks.

>>> In the meantime, though... let's just change the ceph deploy test to 
>>> install the admin key so that we can make the tests pass?  And set teh bug 
>>> severity on this so that we make sure it's addressed in a complete way 
>>> soon.
>>
>> I fully agree.
>>
>> Please can you (Sage) raise the bug so this plan is executed with
>> appropriate timeliness, as I feel if you raise the bug it will enhance
>> the importance. I would suggest you also include the replay attack issue
>> in the bug report.
>>
>> I will need help to understand how to change the functional test suite.
>> The change will be just to issue:
>>
>>     ceph-deploy admin $TARGET_NODE
>>
>> before issuing:
>>
>>     ceph-deploy osd create --dmcrypt $TARGET_NODE:sdb:sdc
> 
> Yep, that's it.  Should just be a few lines of code in 
> 
> https://github.com/ceph/ceph-qa-suite/blob/master/tasks/ceph_deploy.py

I think this might do it,

    https://github.com/ceph/ceph-qa-suite/pull/1259

Sadly I haven’t had the opportunity to test this locally.

Best wishes

Owen

Re: Understanding encrypted OSD's and cephx

[Sage Weil]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 4532 bytes --]

On Tue, 15 Nov 2016, Owen Synge wrote:
> On 11/15/2016 03:27 PM, Sage Weil wrote:
> > On Tue, 15 Nov 2016, Owen Synge wrote:
> >>> and instead define a new monitor command to do all of this in one go.  
> >>> For example, 'osd bootstrap' could take the necessary arguments (e.g., the 
> >>> uuid and local key) and in one step set up the various keys for you.  I 
> >>> think that's going to be the most maintainable and sane solution going 
> >>> forward.
> >>
> >> Having a single command to do this sounds a good long term solution.
> >>
> >> I would add that a replay attack is still a danger, with a one command
> >> approach.
> >>
> >> Using the trigger of OSD registering to prevent replay attacks, seems
> >> like a better compromise than making each command only work once, so
> >> setup can be repeated until success, and then since replay is prevented
> >> other keys cannot be exposed.
> > 
> > There are two options:
> > 
> >  (1) Make the command always instantiate a new osd id and set of keys, 
> > such that you get leaked/garbage state if there is a command resend, 
> > or
> 
> Understood.
> 
> >  (2) make the command properly idempotent.  The trick here is that we need 
> > to uniquely identify the bootstrap command by some piece of state that is 
> > private so that the command cannot be used to fetch secrets for the 
> > created osd.  This shouldn't really be a problem.
> 
> Not sure I follow this point. Can you explain.
> 
> My difficulty is idempotent commands are suitable for replay attacks,
> and cant understand how we get around this with a "private" data hook, I
> am probably just being dumb here, please enlighten me if you have time.
> 
> Personally I like the idea of using the partition uuid to identify
> things as is done currently.

The problem is that if an attacker gets ahold of a partition uuid and an 
bootstrap key then we don't want them to be able to fetch all of the 
generated secrets of the new osd.  I suggest we narrow the window by (1) 
make the bootstrap command fail after the osd has been started for the 
first time and (2) using some private nonce that is not exposed in any 
other way to identify the request.  Probably in combination with the 
partition uuid so that if the command is resent with a different nonce you 
get EPERM instead of the result.

(We're way down in a weeds here.  Suffice to say all mon commands are 
written to be idempotent because of the nature of the client->mon 
protocol, and we'll do a few special things here to improve security.  A 
traditional "replay" attack isn't really the issue here because of the 
nature of the session authentication that wraps these commands.)

> > In the end, this will be significantly more secure than what we were doing 
> > before.  The recommended workflow will become
> > 
> >  (1) push bootstrap-{osd,mon,mds} key to host
> >  (2) create daemon (via new bootstrap command that is tailor-made for the 
> > purpose)
> >  (3) remove bootstrap key
> 
> Agreed, for my close the replay window when OSD is registered approach,
> or your point (1) and probably when I understand it (2).
> 
> > Before we commit to this, there is an alternative model that does all key 
> > creation on some other trusted host, but it would mean a more significant 
> > rewrite of the ceph-disk tooling...
> 
> Yes,
> 
> Forgive me for going in circles here, I think if we substitute "mon
> node" with "trusted host" this is what I proposed first in this thread
> but I dont like it for the below reasons.
> 
> The negative of the "trusted host" approach is:
> 
> * We would still need to get the partition UUID and present it to the
> "trusted host". Then take the output from the "trusted host" back to the
> OSD with the encrypted disks. This makes for inter node communication
> which makes for usability issues.
> 
> * Also new ceph-disk tooling as you pointed out. (ceph-deploy also)
> 
> The positive of the "trusted host" approach:
> 
> * It is the most secure approach.
> 
> * Requires no new tooling in the mon.
> 
> * No worries about replay attacks.

Yeah.  Perhaps the best path is to ensure that hte bootstrap process can 
be driven both ways... eigher by a boostrap-osd key present on the osd 
host or by the trusted host, with all of the resulting bits of information 
passed into ceph-disk explicitly on the command line.

> I think this might do it,
> 
>     https://github.com/ceph/ceph-qa-suite/pull/1259
> 
> Sadly I haven’t had the opportunity to test this locally.

Will test this today!  Thanks-
sage