on disk encryption

on disk encryption

[Sage Weil]

Hey,

A common requirement that's come up in conversation a few times now is 
on-disk, at-rest encryption.  Usually, this is really just about making 
sure the bits on an individual disk are useless in isolation, so that 
drives can be safely discarded or RMAed without compromising customer 
data.

I suspect the simplest way to accomplish this would be through something 
like dm-crypt.  The trick would be keeping the keys for the osd's block 
device and journal elsewhere.

One option would be to use the monitor as a lock box to securely store the 
disk encryption key, secured by the osd's existing cephx key is provided.  
The startup scripts (triggered via upstart, sysvinit, whatever) would need 
to get the keyring off the disk (separate, unencrypted partition?), get 
the disk key from the monitor, set up the dm-crypt devices, mount the 
osd's fs, and then start ceph-osd.  An attacker in possession of a 
recovered disk would be need network connectivity to the cluster (prior to 
the keys getting revoked/destroyed) in order to decrypt it.

Looking forward, another option might be to implement encryption inside 
btrfs (placeholder fields are there in the disk format, introduced along 
with the compression code way back when).  This would let ceph-osd handle 
more of the key handling internally and do something like, say, only 
encrypt the current/ and snap_*/ subdirectories.  

Other ideas?  Thoughts?

sage

Re: on disk encryption

[Mark Nelson]

On 09/15/2012 06:54 AM, Sage Weil wrote:
> Hey,
>
> A common requirement that's come up in conversation a few times now is
> on-disk, at-rest encryption.  Usually, this is really just about making
> sure the bits on an individual disk are useless in isolation, so that
> drives can be safely discarded or RMAed without compromising customer
> data.
>
> I suspect the simplest way to accomplish this would be through something
> like dm-crypt.  The trick would be keeping the keys for the osd's block
> device and journal elsewhere.
>

Sounds good to me.  dm-crypt can use AES-NI which seems to help out a 
lot of the CPU usage front.  It'd be nice to make sure it works.

> One option would be to use the monitor as a lock box to securely store the
> disk encryption key, secured by the osd's existing cephx key is provided.
> The startup scripts (triggered via upstart, sysvinit, whatever) would need
> to get the keyring off the disk (separate, unencrypted partition?), get
> the disk key from the monitor, set up the dm-crypt devices, mount the
> osd's fs, and then start ceph-osd.  An attacker in possession of a
> recovered disk would be need network connectivity to the cluster (prior to
> the keys getting revoked/destroyed) in order to decrypt it.
>
> Looking forward, another option might be to implement encryption inside
> btrfs (placeholder fields are there in the disk format, introduced along
> with the compression code way back when).  This would let ceph-osd handle
> more of the key handling internally and do something like, say, only
> encrypt the current/ and snap_*/ subdirectories.
>

Sounds like this would make our code paths for btrfs and other 
filesystems diverge more rather than less.  I like the idea of not 
making our startup scripts that much more complicated, but what we end 
up doing for xfs and others?  Would we end up having to do the startup 
script path anyway or not support encryption on those FS?

> Other ideas?  Thoughts?
>
> sage
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: on disk encryption

[Dustin Kirkland]

On Sat, Sep 15, 2012 at 6:54 AM, Sage Weil <sage@inktank.com> wrote:
> Hey,
>
> A common requirement that's come up in conversation a few times now is
> on-disk, at-rest encryption.  Usually, this is really just about making
> sure the bits on an individual disk are useless in isolation, so that
> drives can be safely discarded or RMAed without compromising customer
> data.

Hi Sage,

Thanks for bringing this up on your devel list.  This is certainly
something that I experience on a daily basis around Linux storage
systems in general in the enterprise.

> I suspect the simplest way to accomplish this would be through something
> like dm-crypt.  The trick would be keeping the keys for the osd's block
> device and journal elsewhere.

Right, so using established technologies within the Linux kernel, I'd
think you should consider at least dm-crypt and eCryptfs, each of
which have their own costs and benefits.  (Full disclosure: I'm one of
the maintainers of eCryptfs).

dm-crypt is a block level encryption solution, whereas eCryptfs
handles its encryption on a per-file basis.

With dm-crypt, you must pre-allocate a disk, partition, or loopback
device, configure it using cryptsetup, LUKS, and optionally LVM,
format the partition with your filesystem of choice, and then perform
the mount of the device mapped encrypted block device to your mount
point.  The entire block device is encrypted using the same set of
keys, which cannot be easily changed or rotated.  I/O performance is
maximized if using AES-NI cryptographic acceleration on CPUs
supporting the instruction set.  Keys are loaded into LUKS and only
necessary at mount time.

eCryptfs is a layered filesystem, where one directory is mounted on
"top" of another.  The upper directory is just a virtual
representation of the files/directories, with reads and writes being
passed through the kernel and handling any decryption or encryption
before actually dealing with the real files which are written to disk
in the lower directory.  There's no need to "pre-allocate" space in a
partition, disk, or loop device, as you just use a directory in your
current filesystem, which can be any of the Linux filesystems.
Because the encryption happens on a per-file basis, you could actually
use unique or distinct keys for different files.  Some users or
processes on the local system may or may not have access to the
cryptographic keys necessary to mount, read, and write these files.
Also, incremental backups of the encrypted data is possible by using a
simple rsync or similar utility.  There is a little more overhead than
dmcrypt here, since each fread and fwrite has to perform its own
encryption, but honestly, the performance is usually quite tolerable
when encryption is essential to a given workload.  While eCryptfs does
not yet correctly leverage AES-NI, there is an patchset from Colin
King, pending approval by Tyler Hicks on the docket for the next merge
window.  (This does improve performance in some cases on modern
hardware, though there were some initial concerns about costing
performance on some older platforms that don't support AES-NI.)

> One option would be to use the monitor as a lock box to securely store the
> disk encryption key, secured by the osd's existing cephx key is provided.
> The startup scripts (triggered via upstart, sysvinit, whatever) would need
> to get the keyring off the disk (separate, unencrypted partition?), get
> the disk key from the monitor, set up the dm-crypt devices, mount the
> osd's fs, and then start ceph-osd.  An attacker in possession of a
> recovered disk would be need network connectivity to the cluster (prior to
> the keys getting revoked/destroyed) in order to decrypt it.

Yeah, so that's always the trick with a crypto solution...key
management.  You have to store the key somewhere other than on the
encrypted device to ensure the protection and integrity of the data.

Without turning this into an advertisement, my company has a
Linux-based (though proprietary) key server product in this space.
Fundamentally, you would need, as you say, a startup script with
network access to authenticate to a trusted, remote server, request
the necessary keys, have the server apply a flexible set of policies
to ensure that the key request is legitimate, and if so, return the
key material to the requesting server.  When a server is known to be
compromised, subsequent access to that key is revoked at the server,
thereby protecting access to the encrypted data.

There is an OASIS standard for such a protocol called KMIP, but it's
some 200+ pages long and there are no open implementations of KMIP
that I know of.

> Looking forward, another option might be to implement encryption inside
> btrfs (placeholder fields are there in the disk format, introduced along
> with the compression code way back when).  This would let ceph-osd handle
> more of the key handling internally and do something like, say, only
> encrypt the current/ and snap_*/ subdirectories.
>
> Other ideas?  Thoughts?
>
> sage

I love the idea of btrfs supporting encryption natively much like it
does compression.  It may be some time before that happens, so in the
meantime, I'd love to see Ceph support dm-crypt and/or eCryptfs
beneath.


Cheers,
-- 
Dustin Kirkland  |  Chief Technical Officer
Gazzang, Inc.

Re: on disk encryption

[James Page]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 19/09/12 02:53, Dustin Kirkland wrote:
>>> Looking forward, another option might be to implement
>>> encryption inside btrfs (placeholder fields are there in the
>>> disk format, introduced along with the compression code way
>>> back when).  This would let ceph-osd handle more of the key
>>> handling internally and do something like, say, only encrypt
>>> the current/ and snap_*/ subdirectories.
>>> 
>>> Other ideas?  Thoughts?
>>> 
>>> sage
> I love the idea of btrfs supporting encryption natively much like
> it does compression.  It may be some time before that happens, so
> in the meantime, I'd love to see Ceph support dm-crypt and/or
> eCryptfs beneath.

Has this discussion progressed into any sort of implementation yet?
It sounds like this is going to be a key feature for users who want
top-to-bottom encryption of data right down to the block level.

- -- 
James Page
Ubuntu Core Developer
Debian Maintainer
james.page@ubuntu.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with undefined - http://www.enigmail.net/

iQIcBAEBCAAGBQJQxaiUAAoJEL/srsug59jDULUP/2pbVVNJC/Dt6S7A+uUGMGQJ
/jgFqu6SVHplTGs3cKqDYH22W9b34Gr/kcga9qj00lo844drRNRo/AVfdaA+j7Ge
gkqc4ZiwNgZSHmu+I9/4fDpSRJf19i2le1/qtIToAXsxZJyefM4clPrWblK24bRd
T7yWbVJBxjiYv7FziHZohDEJ/jz2OMk4THZYVkB+yuUPLbDnbFxqK17gRtKPuS/K
EeuFBw1kFgB0OKQ4LGy/GSOK1xM4NiGKpdV9beeSfu1L5f1ClW0Drl221gnhZ4qe
g6HXAdCK1xhDU2xUhrrPSp0iVFGjxjnvoQz7PikX6Hn5lhqjbAHVaoQ9dJpshAsY
86XDVFJJF2ca9FjzBGo+Cx7Ap0ahI4eK1NTiNc/zPEb8TgM9q1OtIlAb9A6pyC/E
l0WQ/0WzhbbnjeByXloLkTG2K0WkaJYovemc959VUrdpP5Di2vsEhhFsVFlFUlTC
i8xQaQZmoXXp8mhzNwdSLIcoUb9Y5MnghNO3mdz6WfM2KtyrTobi5lKZyFxZJfhA
oGt5It6AF/fRHi2Xu9yLyfVYrnf/oDJn1vjzJ0BkJLZ8rUANLVGYrpiKAECY1EF3
Nb2kXnhBVs1426TgvcAlchDUACPNUR2YVx9s12gHVTZURgrSr0+QMPHJL9uRJPxE
5T4wqmJNV2Caponla/fr
=wHrw
-----END PGP SIGNATURE-----

Re: on disk encryption

[Gregory Farnum]

On Monday, December 10, 2012 at 1:17 AM, James Page wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>  
> On 19/09/12 02:53, Dustin Kirkland wrote:
> > > > Looking forward, another option might be to implement
> > > > encryption inside btrfs (placeholder fields are there in the
> > > > disk format, introduced along with the compression code way
> > > > back when). This would let ceph-osd handle more of the key
> > > > handling internally and do something like, say, only encrypt
> > > > the current/ and snap_*/ subdirectories.
> > > >  
> > > > Other ideas? Thoughts?
> > > >  
> > > > sage
> > I love the idea of btrfs supporting encryption natively much like
> > it does compression. It may be some time before that happens, so
> > in the meantime, I'd love to see Ceph support dm-crypt and/or
> > eCryptfs beneath.
>  
>  
>  
> Has this discussion progressed into any sort of implementation yet?
> It sounds like this is going to be a key feature for users who want
> top-to-bottom encryption of data right down to the block level.


Peter is working on this now — I'll let him discuss the details. :)
-Greg

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: on disk encryption

[James Page]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 10/12/12 09:53, Gregory Farnum wrote:
[...]
>>>>> I love the idea of btrfs supporting encryption natively
>>>>> much like it does compression. It may be some time before
>>>>> that happens, so in the meantime, I'd love to see Ceph
>>>>> support dm-crypt and/or eCryptfs beneath.
>>> 
>>> 
>>> 
>>> Has this discussion progressed into any sort of implementation
>>> yet? It sounds like this is going to be a key feature for users
>>> who want top-to-bottom encryption of data right down to the
>>> block level.
> 
> Peter is working on this now — I'll let him discuss the details.
> :)

Hey Peter - any update on the on-disk encryption work for Ceph?

Cheers

James

- -- 
James Page
Ubuntu Core Developer
Debian Maintainer
james.page@ubuntu.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)
Comment: Using GnuPG with undefined - http://www.enigmail.net/

iQIcBAEBCAAGBQJQ/wR4AAoJEL/srsug59jD4jAQAIByoFQ3rrbon/BsxqD+KUMZ
xlGbviVxGIiHtLyUIwaXPerrEqnpuQCKbg/ZBXH0F9NUCRw3SZN74YuOjNz8c0Tr
aAy1Wkx+lFCwt2FtiwC3pXx5++GO2qTbK7jsOeqJazxUN1J8EmoUv73jq3u+MmMo
NV5k4e04g7leap3o5f13ONyJmTZC48XDZWdpa2HoYO7h1Er04y2tqOVTHwAd4PS5
26NaT2Cz4c+GMnDoTu608WrUJPv+pbi/WWf3RotRqXC3YX9VIDu6UxEc/tZHA+VP
PcbfgtKGhzj7ooxdHsanhPtUtHv9o9Q2DZFbzvATDC0s3K5Rpav8C1vnC2ODq6fr
LXCiRmVcjXz8e9TIQvSeQZLpK7Sy+WN4PTFdGsQqiVtw+iakw9qSn3EermAsCNIj
EEeHlt6GcWgFF4oVxeZ5EDJHUobz/vyl+R0ZjJgNK3aYv0zDw4w249ARpvjmoIPS
FHYrukgSIHxv1CFSh4AxA4mgRseGM4B7H69+jdzp+3LNaCnHQBnT5cfsVrpoqCam
te5tytclC4gQ3xJh5L2lMH8D/ikSSZZjO+7cJ4ZEW5ebu7ChuonWMj0TQc2gPpUG
qqI0aV4QxRYaE5oRJlxoSlylKd6tWvHc/44TDqUPFWVnqLB1c8WEEZnDviTz5BCC
NYqJJb+2p+pzt2bK0p4r
=+Uvt
-----END PGP SIGNATURE-----
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: on disk encryption

[Sage Weil]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 2864 bytes --]

On Tue, 22 Jan 2013, Asghar Riahi wrote:
> Are you familiar with Seagate's Self Encrypting Disk (SED)?
> 
> Here are some links which might be usefull:
> 
> http://smb.media.seagate.com/tag/seagate-sed/
> 
> http://csrc.nist.gov/groups/STM/cmvp/documents/140-1/140sp/140sp1299.pdf

Yeah!  It would be great to use this.

My understanding is that there isn't yet any support in Linux to manage 
the keys.  That would be the first step before we can make the ceph 
tooling configure the keys prior to mounting the osd volume(s)...

sage

> 
> --
> Regards,
> Asghar Riahi
> Seagate Technology
> Principal Cloud Architect 
> asghar.riahi@seagate.com
> (925) 683-3104
> 
> 
> 
> On Tue, Jan 22, 2013 at 1:28 PM, James Page <james.page@ubuntu.com> wrote:
>       -----BEGIN PGP SIGNED MESSAGE-----
>       Hash: SHA256
> 
> On 10/12/12 09:53, Gregory Farnum wrote:
> [...]
> >>>>> I love the idea of btrfs supporting encryption natively
> >>>>> much like it does compression. It may be some time before
> >>>>> that happens, so in the meantime, I'd love to see Ceph
> >>>>> support dm-crypt and/or eCryptfs beneath.
> >>>
> >>>
> >>>
> >>> Has this discussion progressed into any sort of implementation
> >>> yet? It sounds like this is going to be a key feature for users
> >>> who want top-to-bottom encryption of data right down to the
> >>> block level.
> >
> > Peter is working on this now ? I'll let him discuss the details.
> > :)
> 
> Hey Peter - any update on the on-disk encryption work for Ceph?
> 
> Cheers
> 
> James
> 
> - --
> James Page
> Ubuntu Core Developer
> Debian Maintainer
> james.page@ubuntu.com
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.12 (GNU/Linux)
> Comment: Using GnuPG with undefined - http://www.enigmail.net/
> 
> iQIcBAEBCAAGBQJQ/wR4AAoJEL/srsug59jD4jAQAIByoFQ3rrbon/BsxqD+KUMZ
> xlGbviVxGIiHtLyUIwaXPerrEqnpuQCKbg/ZBXH0F9NUCRw3SZN74YuOjNz8c0Tr
> aAy1Wkx+lFCwt2FtiwC3pXx5++GO2qTbK7jsOeqJazxUN1J8EmoUv73jq3u+MmMo
> NV5k4e04g7leap3o5f13ONyJmTZC48XDZWdpa2HoYO7h1Er04y2tqOVTHwAd4PS5
> 26NaT2Cz4c+GMnDoTu608WrUJPv+pbi/WWf3RotRqXC3YX9VIDu6UxEc/tZHA+VP
> PcbfgtKGhzj7ooxdHsanhPtUtHv9o9Q2DZFbzvATDC0s3K5Rpav8C1vnC2ODq6fr
> LXCiRmVcjXz8e9TIQvSeQZLpK7Sy+WN4PTFdGsQqiVtw+iakw9qSn3EermAsCNIj
> EEeHlt6GcWgFF4oVxeZ5EDJHUobz/vyl+R0ZjJgNK3aYv0zDw4w249ARpvjmoIPS
> FHYrukgSIHxv1CFSh4AxA4mgRseGM4B7H69+jdzp+3LNaCnHQBnT5cfsVrpoqCam
> te5tytclC4gQ3xJh5L2lMH8D/ikSSZZjO+7cJ4ZEW5ebu7ChuonWMj0TQc2gPpUG
> qqI0aV4QxRYaE5oRJlxoSlylKd6tWvHc/44TDqUPFWVnqLB1c8WEEZnDviTz5BCC
> NYqJJb+2p+pzt2bK0p4r
> =+Uvt
> -----END PGP SIGNATURE-----
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel"
> in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 
> 
> 
> 
> 
> 

Re: on disk encryption

[Sage Weil]

On Tue, 22 Jan 2013, James Page wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
> 
> On 10/12/12 09:53, Gregory Farnum wrote:
> [...]
> >>>>> I love the idea of btrfs supporting encryption natively
> >>>>> much like it does compression. It may be some time before
> >>>>> that happens, so in the meantime, I'd love to see Ceph
> >>>>> support dm-crypt and/or eCryptfs beneath.
> >>> 
> >>> 
> >>> 
> >>> Has this discussion progressed into any sort of implementation
> >>> yet? It sounds like this is going to be a key feature for users
> >>> who want top-to-bottom encryption of data right down to the
> >>> block level.
> > 
> > Peter is working on this now ? I'll let him discuss the details.
> > :)
> 
> Hey Peter - any update on the on-disk encryption work for Ceph?

This was put on hold for now.

At this point we're mostly just envisioning a very simple key storage 
service via the ceph montiors (e.g., ceph key get <name>, ceph key put 
<name>), and hooks in the startup scripts (sysvinit and/or upstart) to 
configure dm-crypt.

sage


> 
> Cheers
> 
> James
> 
> - -- 
> James Page
> Ubuntu Core Developer
> Debian Maintainer
> james.page@ubuntu.com
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.12 (GNU/Linux)
> Comment: Using GnuPG with undefined - http://www.enigmail.net/
> 
> iQIcBAEBCAAGBQJQ/wR4AAoJEL/srsug59jD4jAQAIByoFQ3rrbon/BsxqD+KUMZ
> xlGbviVxGIiHtLyUIwaXPerrEqnpuQCKbg/ZBXH0F9NUCRw3SZN74YuOjNz8c0Tr
> aAy1Wkx+lFCwt2FtiwC3pXx5++GO2qTbK7jsOeqJazxUN1J8EmoUv73jq3u+MmMo
> NV5k4e04g7leap3o5f13ONyJmTZC48XDZWdpa2HoYO7h1Er04y2tqOVTHwAd4PS5
> 26NaT2Cz4c+GMnDoTu608WrUJPv+pbi/WWf3RotRqXC3YX9VIDu6UxEc/tZHA+VP
> PcbfgtKGhzj7ooxdHsanhPtUtHv9o9Q2DZFbzvATDC0s3K5Rpav8C1vnC2ODq6fr
> LXCiRmVcjXz8e9TIQvSeQZLpK7Sy+WN4PTFdGsQqiVtw+iakw9qSn3EermAsCNIj
> EEeHlt6GcWgFF4oVxeZ5EDJHUobz/vyl+R0ZjJgNK3aYv0zDw4w249ARpvjmoIPS
> FHYrukgSIHxv1CFSh4AxA4mgRseGM4B7H69+jdzp+3LNaCnHQBnT5cfsVrpoqCam
> te5tytclC4gQ3xJh5L2lMH8D/ikSSZZjO+7cJ4ZEW5ebu7ChuonWMj0TQc2gPpUG
> qqI0aV4QxRYaE5oRJlxoSlylKd6tWvHc/44TDqUPFWVnqLB1c8WEEZnDviTz5BCC
> NYqJJb+2p+pzt2bK0p4r
> =+Uvt
> -----END PGP SIGNATURE-----
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 

Re: on disk encryption

[Marcus Sorensen]

Yes, anyone could do this now by setting up the OSDs on top of
dm-crypted disks, correct? This would just automate the process, and
manage keys for us?

On Tue, Jan 22, 2013 at 5:04 PM, Sage Weil <sage@inktank.com> wrote:
> On Tue, 22 Jan 2013, James Page wrote:
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA256
>>
>> On 10/12/12 09:53, Gregory Farnum wrote:
>> [...]
>> >>>>> I love the idea of btrfs supporting encryption natively
>> >>>>> much like it does compression. It may be some time before
>> >>>>> that happens, so in the meantime, I'd love to see Ceph
>> >>>>> support dm-crypt and/or eCryptfs beneath.
>> >>>
>> >>>
>> >>>
>> >>> Has this discussion progressed into any sort of implementation
>> >>> yet? It sounds like this is going to be a key feature for users
>> >>> who want top-to-bottom encryption of data right down to the
>> >>> block level.
>> >
>> > Peter is working on this now ? I'll let him discuss the details.
>> > :)
>>
>> Hey Peter - any update on the on-disk encryption work for Ceph?
>
> This was put on hold for now.
>
> At this point we're mostly just envisioning a very simple key storage
> service via the ceph montiors (e.g., ceph key get <name>, ceph key put
> <name>), and hooks in the startup scripts (sysvinit and/or upstart) to
> configure dm-crypt.
>
> sage
>
>
>>
>> Cheers
>>
>> James
>>
>> - --
>> James Page
>> Ubuntu Core Developer
>> Debian Maintainer
>> james.page@ubuntu.com
>> -----BEGIN PGP SIGNATURE-----
>> Version: GnuPG v1.4.12 (GNU/Linux)
>> Comment: Using GnuPG with undefined - http://www.enigmail.net/
>>
>> iQIcBAEBCAAGBQJQ/wR4AAoJEL/srsug59jD4jAQAIByoFQ3rrbon/BsxqD+KUMZ
>> xlGbviVxGIiHtLyUIwaXPerrEqnpuQCKbg/ZBXH0F9NUCRw3SZN74YuOjNz8c0Tr
>> aAy1Wkx+lFCwt2FtiwC3pXx5++GO2qTbK7jsOeqJazxUN1J8EmoUv73jq3u+MmMo
>> NV5k4e04g7leap3o5f13ONyJmTZC48XDZWdpa2HoYO7h1Er04y2tqOVTHwAd4PS5
>> 26NaT2Cz4c+GMnDoTu608WrUJPv+pbi/WWf3RotRqXC3YX9VIDu6UxEc/tZHA+VP
>> PcbfgtKGhzj7ooxdHsanhPtUtHv9o9Q2DZFbzvATDC0s3K5Rpav8C1vnC2ODq6fr
>> LXCiRmVcjXz8e9TIQvSeQZLpK7Sy+WN4PTFdGsQqiVtw+iakw9qSn3EermAsCNIj
>> EEeHlt6GcWgFF4oVxeZ5EDJHUobz/vyl+R0ZjJgNK3aYv0zDw4w249ARpvjmoIPS
>> FHYrukgSIHxv1CFSh4AxA4mgRseGM4B7H69+jdzp+3LNaCnHQBnT5cfsVrpoqCam
>> te5tytclC4gQ3xJh5L2lMH8D/ikSSZZjO+7cJ4ZEW5ebu7ChuonWMj0TQc2gPpUG
>> qqI0aV4QxRYaE5oRJlxoSlylKd6tWvHc/44TDqUPFWVnqLB1c8WEEZnDviTz5BCC
>> NYqJJb+2p+pzt2bK0p4r
>> =+Uvt
>> -----END PGP SIGNATURE-----
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: on disk encryption

[Mark Kampe]

Correct.

I wasn't actually involved in this (or any other real) work,
but as I recall the only real trick is how much key management
you want:

   Do we want to be able to recover the key if a good disk
   is rescued from a destroyed server and added to a new
   server?

   Do we want to ensure that the keys are not persisted on
   the server, so that an entire server can be decommissioned
   without having to worry about the data being recovered
   by somebody who knows where to look?

If you are willing to keep the key on the server and lose
the data when the server fails, this is trivial.  If you
are unwilling to keep the key on the server, or if you need
the disk to remain readable after the server is lost, we
need some third party (like the monitors) to maintain the
keys.

We thought these might be important, so we were looking
at how to get the monitors to keep track of the encryption
keys.

On 01/31/2013 03:42 PM, Marcus Sorensen wrote:
> Yes, anyone could do this now by setting up the OSDs on top of
> dm-crypted disks, correct? This would just automate the process, and
> manage keys for us?

Re: on disk encryption

[Marcus Sorensen]

That's a great idea, it simplifies things for the admin and doesn't
sound too hairy on the monitor side. One alternative option is to do
it in one of the ways that ZFS does; that is you can provide a config
option that says 'get the keys from here'. For example you could set
up an https server and have ceph fetch the keys from it at start up.
Users could provide a master key for the whole pool, or per OSD, or
different keys for groups of OSDs, per the config file.

 I still like the monitor idea better, but different users may prefer
different methods, and this might be an easier solution to implement
first.

Here's some of the zfs documentation in case it sparks some other ideas.

# zfs create -o encryption=on -o
keysource=raw,https://keys.example.com/mykey tank/project/R

http://www.oracle.com/technetwork/articles/servers-storage-admin/manage-zfs-encryption-1715034.html

On Thu, Jan 31, 2013 at 5:04 PM, Mark Kampe <mark.kampe@inktank.com> wrote:
> Correct.
>
> I wasn't actually involved in this (or any other real) work,
> but as I recall the only real trick is how much key management
> you want:
>
>   Do we want to be able to recover the key if a good disk
>   is rescued from a destroyed server and added to a new
>   server?
>
>   Do we want to ensure that the keys are not persisted on
>   the server, so that an entire server can be decommissioned
>   without having to worry about the data being recovered
>   by somebody who knows where to look?
>
> If you are willing to keep the key on the server and lose
> the data when the server fails, this is trivial.  If you
> are unwilling to keep the key on the server, or if you need
> the disk to remain readable after the server is lost, we
> need some third party (like the monitors) to maintain the
> keys.
>
> We thought these might be important, so we were looking
> at how to get the monitors to keep track of the encryption
> keys.
>
>
> On 01/31/2013 03:42 PM, Marcus Sorensen wrote:
>>
>> Yes, anyone could do this now by setting up the OSDs on top of
>> dm-crypted disks, correct? This would just automate the process, and
>> manage keys for us?
>
>

Re: on disk encryption

[Sage Weil]

On Thu, 31 Jan 2013, Marcus Sorensen wrote:
> Yes, anyone could do this now by setting up the OSDs on top of
> dm-crypted disks, correct? This would just automate the process, and
> manage keys for us?

That is the idea.

sage

> 
> On Tue, Jan 22, 2013 at 5:04 PM, Sage Weil <sage@inktank.com> wrote:
> > On Tue, 22 Jan 2013, James Page wrote:
> >> -----BEGIN PGP SIGNED MESSAGE-----
> >> Hash: SHA256
> >>
> >> On 10/12/12 09:53, Gregory Farnum wrote:
> >> [...]
> >> >>>>> I love the idea of btrfs supporting encryption natively
> >> >>>>> much like it does compression. It may be some time before
> >> >>>>> that happens, so in the meantime, I'd love to see Ceph
> >> >>>>> support dm-crypt and/or eCryptfs beneath.
> >> >>>
> >> >>>
> >> >>>
> >> >>> Has this discussion progressed into any sort of implementation
> >> >>> yet? It sounds like this is going to be a key feature for users
> >> >>> who want top-to-bottom encryption of data right down to the
> >> >>> block level.
> >> >
> >> > Peter is working on this now ? I'll let him discuss the details.
> >> > :)
> >>
> >> Hey Peter - any update on the on-disk encryption work for Ceph?
> >
> > This was put on hold for now.
> >
> > At this point we're mostly just envisioning a very simple key storage
> > service via the ceph montiors (e.g., ceph key get <name>, ceph key put
> > <name>), and hooks in the startup scripts (sysvinit and/or upstart) to
> > configure dm-crypt.
> >
> > sage
> >
> >
> >>
> >> Cheers
> >>
> >> James
> >>
> >> - --
> >> James Page
> >> Ubuntu Core Developer
> >> Debian Maintainer
> >> james.page@ubuntu.com
> >> -----BEGIN PGP SIGNATURE-----
> >> Version: GnuPG v1.4.12 (GNU/Linux)
> >> Comment: Using GnuPG with undefined - http://www.enigmail.net/
> >>
> >> iQIcBAEBCAAGBQJQ/wR4AAoJEL/srsug59jD4jAQAIByoFQ3rrbon/BsxqD+KUMZ
> >> xlGbviVxGIiHtLyUIwaXPerrEqnpuQCKbg/ZBXH0F9NUCRw3SZN74YuOjNz8c0Tr
> >> aAy1Wkx+lFCwt2FtiwC3pXx5++GO2qTbK7jsOeqJazxUN1J8EmoUv73jq3u+MmMo
> >> NV5k4e04g7leap3o5f13ONyJmTZC48XDZWdpa2HoYO7h1Er04y2tqOVTHwAd4PS5
> >> 26NaT2Cz4c+GMnDoTu608WrUJPv+pbi/WWf3RotRqXC3YX9VIDu6UxEc/tZHA+VP
> >> PcbfgtKGhzj7ooxdHsanhPtUtHv9o9Q2DZFbzvATDC0s3K5Rpav8C1vnC2ODq6fr
> >> LXCiRmVcjXz8e9TIQvSeQZLpK7Sy+WN4PTFdGsQqiVtw+iakw9qSn3EermAsCNIj
> >> EEeHlt6GcWgFF4oVxeZ5EDJHUobz/vyl+R0ZjJgNK3aYv0zDw4w249ARpvjmoIPS
> >> FHYrukgSIHxv1CFSh4AxA4mgRseGM4B7H69+jdzp+3LNaCnHQBnT5cfsVrpoqCam
> >> te5tytclC4gQ3xJh5L2lMH8D/ikSSZZjO+7cJ4ZEW5ebu7ChuonWMj0TQc2gPpUG
> >> qqI0aV4QxRYaE5oRJlxoSlylKd6tWvHc/44TDqUPFWVnqLB1c8WEEZnDviTz5BCC
> >> NYqJJb+2p+pzt2bK0p4r
> >> =+Uvt
> >> -----END PGP SIGNATURE-----
> >> --
> >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> >> the body of a message to majordomo@vger.kernel.org
> >> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >>
> >>
> > --
> > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 

Re: on disk encryption

[Neil Levine]

If there are any users work for organizations which have strict
encryption or key management policies, I'd be interested to learn a
bit more about your needs to drive some of the roadmap around the
encryption features longer-term.

Neil

Re: on disk encryption

[Christian Brunner]

As a special use-case I would propose to do the encryption inside the
qemu-rbd driver (similar to the qcow2 driver). This would also encrypt the
network traffic from the KVM host to the osd.

I know that this is probably not the generic aproach for on disk
encryption, but it would provide some extra secuity for the "EBS" user.

Is there someone working on this / do others have interest in this solution?

Christian