LUKS encryption in OSDs (ceph-volume)

LUKS encryption in OSDs (ceph-volume)

[Alfredo Deza]

We have started looking into encryption support in ceph-volume, and
one of the unclear paths is if we really want to support both "plain"
and "LUKS".

According to the cryptsetup docs [0] :

    (LUKS) is now the preferred way to set up disk encryption with
dm-crypt using the cryptsetup utility


ceph-disk supports both plain and LUKS, but moving forward, I was
interested in understanding if anyone is really expecting the "plain"
type to be a choice?

The legacy support will mean that ceph-volume will have to deal with
"plain", but moving forward it might be easier if we are supporting a
single type of encryption with LUKS.

Re: LUKS encryption in OSDs (ceph-volume)

[Wyllys Ingersoll]

Its useful for legacy systems that installed with "plain" back when
that was the only option. Since there is no easy migration path for
re-keying an encrypted OSD to use a new encryption scheme, keeping
support for legacy "plain" is still very useful and desirable.

On Tue, Dec 12, 2017 at 2:27 PM, Alfredo Deza <adeza@redhat.com> wrote:
> We have started looking into encryption support in ceph-volume, and
> one of the unclear paths is if we really want to support both "plain"
> and "LUKS".
>
> According to the cryptsetup docs [0] :
>
>     (LUKS) is now the preferred way to set up disk encryption with
> dm-crypt using the cryptsetup utility
>
>
> ceph-disk supports both plain and LUKS, but moving forward, I was
> interested in understanding if anyone is really expecting the "plain"
> type to be a choice?
>
> The legacy support will mean that ceph-volume will have to deal with
> "plain", but moving forward it might be easier if we are supporting a
> single type of encryption with LUKS.
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: LUKS encryption in OSDs (ceph-volume)

[Alfredo Deza]

On Tue, Dec 12, 2017 at 2:38 PM, Wyllys Ingersoll
<wyllys.ingersoll@keepertech.com> wrote:
> Its useful for legacy systems that installed with "plain" back when
> that was the only option. Since there is no easy migration path for
> re-keying an encrypted OSD to use a new encryption scheme, keeping
> support for legacy "plain" is still very useful and desirable.

Yes, for sure we are going to support that legacy option. But for
*newly* created OSDs, I was looking forward to follow
the preferred way with LUKS only.

>
> On Tue, Dec 12, 2017 at 2:27 PM, Alfredo Deza <adeza@redhat.com> wrote:
>> We have started looking into encryption support in ceph-volume, and
>> one of the unclear paths is if we really want to support both "plain"
>> and "LUKS".
>>
>> According to the cryptsetup docs [0] :
>>
>>     (LUKS) is now the preferred way to set up disk encryption with
>> dm-crypt using the cryptsetup utility
>>
>>
>> ceph-disk supports both plain and LUKS, but moving forward, I was
>> interested in understanding if anyone is really expecting the "plain"
>> type to be a choice?
>>
>> The legacy support will mean that ceph-volume will have to deal with
>> "plain", but moving forward it might be easier if we are supporting a
>> single type of encryption with LUKS.
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: LUKS encryption in OSDs (ceph-volume)

[Sage Weil]

On Tue, 12 Dec 2017, Alfredo Deza wrote:
> On Tue, Dec 12, 2017 at 2:38 PM, Wyllys Ingersoll
> <wyllys.ingersoll@keepertech.com> wrote:
> > Its useful for legacy systems that installed with "plain" back when
> > that was the only option. Since there is no easy migration path for
> > re-keying an encrypted OSD to use a new encryption scheme, keeping
> > support for legacy "plain" is still very useful and desirable.
> 
> Yes, for sure we are going to support that legacy option. But for
> *newly* created OSDs, I was looking forward to follow
> the preferred way with LUKS only.

It's worth mentioning that the "new" way for new ceph-volume OSD 
deployments will also be using LVM, and (presumably?) allow layering 
dm-crypt on top of an LV--not just a PV or raw device.  So this is more a 
question of what, clean slate, we want to do to deploy dm-crypt when the 
end result that we're after is an LV to feed to bluestore or filestore.  
I'm not sure how/where LUKS fits in in the LVM world...

Copying Milan, as I expect he has an opinion here?  :)

sage


> 
> >
> > On Tue, Dec 12, 2017 at 2:27 PM, Alfredo Deza <adeza@redhat.com> wrote:
> >> We have started looking into encryption support in ceph-volume, and
> >> one of the unclear paths is if we really want to support both "plain"
> >> and "LUKS".
> >>
> >> According to the cryptsetup docs [0] :
> >>
> >>     (LUKS) is now the preferred way to set up disk encryption with
> >> dm-crypt using the cryptsetup utility
> >>
> >>
> >> ceph-disk supports both plain and LUKS, but moving forward, I was
> >> interested in understanding if anyone is really expecting the "plain"
> >> type to be a choice?
> >>
> >> The legacy support will mean that ceph-volume will have to deal with
> >> "plain", but moving forward it might be easier if we are supporting a
> >> single type of encryption with LUKS.
> >> --
> >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> >> the body of a message to majordomo@vger.kernel.org
> >> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 

Re: LUKS encryption in OSDs (ceph-volume)

[Milan Broz]

On 12/12/2017 09:47 PM, Sage Weil wrote:
> On Tue, 12 Dec 2017, Alfredo Deza wrote:
>> On Tue, Dec 12, 2017 at 2:38 PM, Wyllys Ingersoll
>> <wyllys.ingersoll@keepertech.com> wrote:
>>> Its useful for legacy systems that installed with "plain" back when
>>> that was the only option. Since there is no easy migration path for
>>> re-keying an encrypted OSD to use a new encryption scheme, keeping
>>> support for legacy "plain" is still very useful and desirable.
>>
>> Yes, for sure we are going to support that legacy option. But for
>> *newly* created OSDs, I was looking forward to follow
>> the preferred way with LUKS only.
> 
> It's worth mentioning that the "new" way for new ceph-volume OSD 
> deployments will also be using LVM, and (presumably?) allow layering 
> dm-crypt on top of an LV--not just a PV or raw device.  So this is more a 
> question of what, clean slate, we want to do to deploy dm-crypt when the 
> end result that we're after is an LV to feed to bluestore or filestore.  
> I'm not sure how/where LUKS fits in in the LVM world...

I think LUKS fits in LVM world quite well.

Standard Fedora (and most other distors as well) install stacks LVM over LUKS
(so you activate only one encrypted device and then the partitioning is up to LVM.
Also LVM metadata are then encrypted.)

You can of course stack LUKS over LV as well, but for example LV resize
will be two-step operation (well, fsadm can automate it but it is still two-steps).

Milan

Re: LUKS encryption in OSDs (ceph-volume)

[Sage Weil]

On Wed, 13 Dec 2017, Milan Broz wrote:
> On 12/12/2017 09:47 PM, Sage Weil wrote:
> > On Tue, 12 Dec 2017, Alfredo Deza wrote:
> >> On Tue, Dec 12, 2017 at 2:38 PM, Wyllys Ingersoll
> >> <wyllys.ingersoll@keepertech.com> wrote:
> >>> Its useful for legacy systems that installed with "plain" back when
> >>> that was the only option. Since there is no easy migration path for
> >>> re-keying an encrypted OSD to use a new encryption scheme, keeping
> >>> support for legacy "plain" is still very useful and desirable.
> >>
> >> Yes, for sure we are going to support that legacy option. But for
> >> *newly* created OSDs, I was looking forward to follow
> >> the preferred way with LUKS only.
> > 
> > It's worth mentioning that the "new" way for new ceph-volume OSD 
> > deployments will also be using LVM, and (presumably?) allow layering 
> > dm-crypt on top of an LV--not just a PV or raw device.  So this is more a 
> > question of what, clean slate, we want to do to deploy dm-crypt when the 
> > end result that we're after is an LV to feed to bluestore or filestore.  
> > I'm not sure how/where LUKS fits in in the LVM world...
> 
> I think LUKS fits in LVM world quite well.
> 
> Standard Fedora (and most other distors as well) install stacks LVM over LUKS
> (so you activate only one encrypted device and then the partitioning is up to LVM.
> Also LVM metadata are then encrypted.)
> 
> You can of course stack LUKS over LV as well, but for example LV resize
> will be two-step operation (well, fsadm can automate it but it is still two-steps).

The problem I see is that we frequently carve up a single phsycial device 
across lots of OSDs (usually for the journal or wal/db), and the key 
management is currently structured around OSDs, not devices.  We can 
either (1) switch it all around so the key management is based on physical 
devices instead of OSDs, or (2) carve the physical device into raw LVs, 
dm-crypt those, and then consume the result.

We don't really have a need to resize devices, generally, so that's 
probably not an issue... but I wonder if (1) is going to be an 
easier/smaller change to make this work.  It maps more directly onto the 
threat model (a lost *device*) and probably also avoids some DM layers, 
and so ought to be slightly faster?

sage

Re: LUKS encryption in OSDs (ceph-volume)

[Alfredo Deza]

On Wed, Dec 13, 2017 at 12:36 PM, Sage Weil <sweil@redhat.com> wrote:
> On Wed, 13 Dec 2017, Milan Broz wrote:
>> On 12/12/2017 09:47 PM, Sage Weil wrote:
>> > On Tue, 12 Dec 2017, Alfredo Deza wrote:
>> >> On Tue, Dec 12, 2017 at 2:38 PM, Wyllys Ingersoll
>> >> <wyllys.ingersoll@keepertech.com> wrote:
>> >>> Its useful for legacy systems that installed with "plain" back when
>> >>> that was the only option. Since there is no easy migration path for
>> >>> re-keying an encrypted OSD to use a new encryption scheme, keeping
>> >>> support for legacy "plain" is still very useful and desirable.
>> >>
>> >> Yes, for sure we are going to support that legacy option. But for
>> >> *newly* created OSDs, I was looking forward to follow
>> >> the preferred way with LUKS only.
>> >
>> > It's worth mentioning that the "new" way for new ceph-volume OSD
>> > deployments will also be using LVM, and (presumably?) allow layering
>> > dm-crypt on top of an LV--not just a PV or raw device.  So this is more a
>> > question of what, clean slate, we want to do to deploy dm-crypt when the
>> > end result that we're after is an LV to feed to bluestore or filestore.
>> > I'm not sure how/where LUKS fits in in the LVM world...
>>
>> I think LUKS fits in LVM world quite well.
>>
>> Standard Fedora (and most other distors as well) install stacks LVM over LUKS
>> (so you activate only one encrypted device and then the partitioning is up to LVM.
>> Also LVM metadata are then encrypted.)

That is a very important distinction (encrypted LVM metadata) and why
we don't want to go that route: we rely heavily on LVM metadata where
we store the
information about the OSD to be later queried. That metadata must be
available without decryption.

>>
>> You can of course stack LUKS over LV as well, but for example LV resize
>> will be two-step operation (well, fsadm can automate it but it is still two-steps).

Right, this is what we are thinking to do.

>
> The problem I see is that we frequently carve up a single phsycial device
> across lots of OSDs (usually for the journal or wal/db), and the key
> management is currently structured around OSDs, not devices.  We can
> either (1) switch it all around so the key management is based on physical
> devices instead of OSDs, or (2) carve the physical device into raw LVs,
> dm-crypt those, and then consume the result.

#2 here is my preference. Lets not switch key management around. I
don't know how things like dmcache
(which is made up of several physical devices) would work for option #1.

>
> We don't really have a need to resize devices, generally, so that's
> probably not an issue... but I wonder if (1) is going to be an
> easier/smaller change to make this work.  It maps more directly onto the
> threat model (a lost *device*) and probably also avoids some DM layers,
> and so ought to be slightly faster?

#1 would involve extra work for anything that is not a single device
though right?

On the ceph-volume side of things, there wouldn't be any overhead for
#2 other than implementing the encryption.

If #1 means we encrypt the physical device, then that makes LVM
metadata encyrpted which wouldn't work for ceph-volume.

LVM would be fine with #2 with lost disks too, I don't think that model changes.

>
> sage

Re: LUKS encryption in OSDs (ceph-volume)

[Milan Broz]

On 12/13/2017 06:36 PM, Sage Weil wrote:
> We don't really have a need to resize devices, generally, so that's 
> probably not an issue... but I wonder if (1) is going to be an 
> easier/smaller change to make this work.  It maps more directly onto the 
> threat model (a lost *device*) and probably also avoids some DM layers, 
> and so ought to be slightly faster?

DM is based on stacking device model, linear mapping (normal LV in LVM)
cost you almost nothing - it is just offset overwrite inside kernel.
So if you think about layering linear devices, the speed difference should be negligible.

With dm-crypt it is something different - it need to clone bio requests and
then encrypt/decrypt them.

Milan

Re: LUKS encryption in OSDs (ceph-volume)

[Milan Broz]

On 12/14/2017 02:17 PM, Alfredo Deza wrote:
>>> Standard Fedora (and most other distors as well) install stacks LVM over LUKS
>>> (so you activate only one encrypted device and then the partitioning is up to LVM.
>>> Also LVM metadata are then encrypted.)
> 
> That is a very important distinction (encrypted LVM metadata) and why
> we don't want to go that route: we rely heavily on LVM metadata where
> we store the
> information about the OSD to be later queried. That metadata must be
> available without decryption.

LUKS2 allows you to store store your JSON data to LUKS header.
For the identification of OSD or whatever you need it is even simpler.

>> The problem I see is that we frequently carve up a single phsycial device
>> across lots of OSDs (usually for the journal or wal/db), and the key
>> management is currently structured around OSDs, not devices.  We can
>> either (1) switch it all around so the key management is based on physical
>> devices instead of OSDs, or (2) carve the physical device into raw LVs,
>> dm-crypt those, and then consume the result.
> 
> #2 here is my preference. Lets not switch key management around. I
> don't know how things like dmcache
> (which is made up of several physical devices) would work for option #1.

I currently lost overview what is Ceph doing, but for unlocking
of crypt device - LUKS2 allows you to store passphrase to kernel keyring
(in advance) and then device is unlocked automatically.
(It needs to configure a "token" for metadata - in the case of keyring
it is just name in keyring.)

Maybe it can simplify your key management in future with LUKS.

(LUKS2 is already in Fedora rawhide and in some experimental repos of other distros.
It need some time to be widely used and I know it will need some fixes - but in principle
things mentioned above already work.)

Milan

Re: LUKS encryption in OSDs (ceph-volume)

[Alfredo Deza]

On Thu, Dec 14, 2017 at 8:49 AM, Milan Broz <mbroz@redhat.com> wrote:
> On 12/14/2017 02:17 PM, Alfredo Deza wrote:
>>>> Standard Fedora (and most other distors as well) install stacks LVM over LUKS
>>>> (so you activate only one encrypted device and then the partitioning is up to LVM.
>>>> Also LVM metadata are then encrypted.)
>>
>> That is a very important distinction (encrypted LVM metadata) and why
>> we don't want to go that route: we rely heavily on LVM metadata where
>> we store the
>> information about the OSD to be later queried. That metadata must be
>> available without decryption.
>
> LUKS2 allows you to store store your JSON data to LUKS header.
> For the identification of OSD or whatever you need it is even simpler.
>
>>> The problem I see is that we frequently carve up a single phsycial device
>>> across lots of OSDs (usually for the journal or wal/db), and the key
>>> management is currently structured around OSDs, not devices.  We can
>>> either (1) switch it all around so the key management is based on physical
>>> devices instead of OSDs, or (2) carve the physical device into raw LVs,
>>> dm-crypt those, and then consume the result.
>>
>> #2 here is my preference. Lets not switch key management around. I
>> don't know how things like dmcache
>> (which is made up of several physical devices) would work for option #1.
>
> I currently lost overview what is Ceph doing, but for unlocking
> of crypt device - LUKS2 allows you to store passphrase to kernel keyring
> (in advance) and then device is unlocked automatically.
> (It needs to configure a "token" for metadata - in the case of keyring
> it is just name in keyring.)

Do you have docs that explain this bit in detail? We were just going
to do what was done already by ceph-disk, that differs a lot from the
kernel keyring
workflow.

>
> Maybe it can simplify your key management in future with LUKS.
>
> (LUKS2 is already in Fedora rawhide and in some experimental repos of other distros.
> It need some time to be widely used and I know it will need some fixes - but in principle
> things mentioned above already work.)

That is a big gotcha though, we would need 100% availability in the
distros/versions we support at release time (around April-May 2018)

Are you leaning towards encrypting the physical devices over
encrypting LVs ? I'm not seeing any risks so far in just going the lv
encryption route (yet)
>
> Milan

Re: LUKS encryption in OSDs (ceph-volume)

[Andrew Schoen]

>> It's worth mentioning that the "new" way for new ceph-volume OSD
>> deployments will also be using LVM, and (presumably?) allow layering
>> dm-crypt on top of an LV--not just a PV or raw device.  So this is more a
>> question of what, clean slate, we want to do to deploy dm-crypt when the
>> end result that we're after is an LV to feed to bluestore or filestore.
>> I'm not sure how/where LUKS fits in in the LVM world...
>
> I think LUKS fits in LVM world quite well.
>
> Standard Fedora (and most other distors as well) install stacks LVM over LUKS
> (so you activate only one encrypted device and then the partitioning is up to LVM.
> Also LVM metadata are then encrypted.)
>
> You can of course stack LUKS over LV as well, but for example LV resize
> will be two-step operation (well, fsadm can automate it but it is still two-steps).

Would this be the only downside to LUKS on LVM? This approach is nice
for ceph-volume
because we need to be able to encrypt anything given to us, which is
often times a LV.

The LVM on LUKS approach also makes it more difficult to expand the
underlying vgs and span
lvs across many disks. If I'm understanding correctly.


Thanks,
Andrew

Re: LUKS encryption in OSDs (ceph-volume)

[Milan Broz]

On 12/14/2017 03:58 PM, Andrew Schoen wrote:
>>> It's worth mentioning that the "new" way for new ceph-volume OSD
>>> deployments will also be using LVM, and (presumably?) allow layering
>>> dm-crypt on top of an LV--not just a PV or raw device.  So this is more a
>>> question of what, clean slate, we want to do to deploy dm-crypt when the
>>> end result that we're after is an LV to feed to bluestore or filestore.
>>> I'm not sure how/where LUKS fits in in the LVM world...
>>
>> I think LUKS fits in LVM world quite well.
>>
>> Standard Fedora (and most other distors as well) install stacks LVM over LUKS
>> (so you activate only one encrypted device and then the partitioning is up to LVM.
>> Also LVM metadata are then encrypted.)
>>
>> You can of course stack LUKS over LV as well, but for example LV resize
>> will be two-step operation (well, fsadm can automate it but it is still two-steps).
> 
> Would this be the only downside to LUKS on LVM? This approach is nice
> for ceph-volume
> because we need to be able to encrypt anything given to us, which is
> often times a LV.
> 
> The LVM on LUKS approach also makes it more difficult to expand the
> underlying vgs and span
> lvs across many disks. If I'm understanding correctly.

LUKS over LV should work perfectly fine, so if you have more arguments for this approach,
than use it.
(I think you can still have more LUKS devices that contains more PVs to one VG in LVM,
byt it could be more complicated to maintain.)

So I am probably just a little bit confused from the approach that there can be more OSDs
spanning one physical device. If it is common now, then I guess LV is just an equivalent
of OSD on a physical hotplugged device and you solution makes perfect sense.

Milan

Re: LUKS encryption in OSDs (ceph-volume)

[Milan Broz]

On 12/14/2017 03:00 PM, Alfredo Deza wrote:
> On Thu, Dec 14, 2017 at 8:49 AM, Milan Broz <mbroz@redhat.com> wrote:
>> On 12/14/2017 02:17 PM, Alfredo Deza wrote:

>> I currently lost overview what is Ceph doing, but for unlocking
>> of crypt device - LUKS2 allows you to store passphrase to kernel keyring
>> (in advance) and then device is unlocked automatically.
>> (It needs to configure a "token" for metadata - in the case of keyring
>> it is just name in keyring.)
> 
> Do you have docs that explain this bit in detail? We were just going
> to do what was done already by ceph-disk, that differs a lot from the
> kernel keyring
> workflow.

We have some example and I plan to write some short blog regarding keyring use.
But the best would be to talk directly to Ondra Kozina (cc) who wrote this keyring
part of the code and can help you to understand what is possible.

>> Maybe it can simplify your key management in future with LUKS.
>>
>> (LUKS2 is already in Fedora rawhide and in some experimental repos of other distros.
>> It need some time to be widely used and I know it will need some fixes - but in principle
>> things mentioned above already work.)
> 
> That is a big gotcha though, we would need 100% availability in the
> distros/versions we support at release time (around April-May 2018)

Yes, I think we should stick with LUKS1 for now.
But we can plan something more simple with LUKS2 in the future.
 
> Are you leaning towards encrypting the physical devices over
> encrypting LVs ? I'm not seeing any risks so far in just going the lv
> encryption route (yet)

See other mail - both works, if it makes more sense to use LUKS on top, that's ok.

Milan

Re: LUKS encryption in OSDs (ceph-volume)

[Ondrej Kozina]

On 12/15/2017 03:31 PM, Milan Broz wrote:
> On 12/14/2017 03:00 PM, Alfredo Deza wrote:
>> On Thu, Dec 14, 2017 at 8:49 AM, Milan Broz <mbroz@redhat.com> wrote:
>>> On 12/14/2017 02:17 PM, Alfredo Deza wrote:
> 
>>> I currently lost overview what is Ceph doing, but for unlocking
>>> of crypt device - LUKS2 allows you to store passphrase to kernel keyring
>>> (in advance) and then device is unlocked automatically.
>>> (It needs to configure a "token" for metadata - in the case of keyring
>>> it is just name in keyring.)
>>
>> Do you have docs that explain this bit in detail? We were just going
>> to do what was done already by ceph-disk, that differs a lot from the
>> kernel keyring
>> workflow.

I'll describe briefly the process for cryptsetup cli only, but you may 
achieve same using cryptsetup library as well. For reference here's the 
api docs: https://gitlab.com/cryptsetup/cryptsetup/wikis/API/index.html

In general kernel keyring activation's an extension for easier LUKS2 
device unlocking.

Let's say you have LUKS2 device with keyslot 1 on /dev/sdx.

Add keyring token to LUKS2 header:
cryptsetup token add /dev/sdx --key-description my_key_name -S 1

(The token itself will contain only the key description and reference to 
keyslot(s) it may be used with. It's stored in jsom metadata area of 
LUKS2 header in plain text)

When you activate LUKS2 with such token it does following:

1) It'll try to activate the device using passphrase retrieved from 
kernel key described by "my_key_name" (kernel key type "user")

2) Only if step 1) fails (i.e. no such key in keyring or passphrase is 
wrong), the command will ask for passphrase the usual way.

So as long as "my_key_name" is reachable from calling process, user 
doesn't have to input passphrase every time he wants to unlock the device.

In fact kernel keyring in this particular case is just another source 
for passphrase just like key files stored in fs.

Regards
O.