Ceph connections with TLS

Ceph connections with TLS

[Amon Ott]

Hello Ceph,

for a customer we are currently designing a ceph cluster, which shall be
spread over two data centers. Data in the Ceph cluster is slightly
confidential, so we would like to encrypt at least all Ceph traffic over
the fast data center connection link.

AFAICS, Ceph does not support data encryption at connection level yet,
so we would have to setup VPN links between the two cluster networks.
This means extra configuration, maintenance and overhead.

How far away is TLS support or something similar for the Ceph
connections? AFAIK, TLS support should not be hard to implement, but I
am not too familiar with Ceph internals.

Thoughts?

Amon Ott
-- 
Dr. Amon Ott
m-privacy GmbH           Tel: +49 30 24342334
Werner-Voß-Damm 62       Fax: +49 30 99296856
12101 Berlin             http://www.m-privacy.de

Amtsgericht Charlottenburg, HRB 84946

Geschäftsführer:
 Dipl.-Kfm. Holger Maczkowsky,
 Roman Maczkowsky

GnuPG-Key-ID: 0x2DD3A649

Re: Ceph connections with TLS

[Wido den Hollander]

> Op 16 december 2016 om 11:15 schreef Amon Ott <a.ott@m-privacy.de>:
> 
> 
> Hello Ceph,
> 
> for a customer we are currently designing a ceph cluster, which shall be
> spread over two data centers. Data in the Ceph cluster is slightly
> confidential, so we would like to encrypt at least all Ceph traffic over
> the fast data center connection link.
> 

Fast as in low latency? Writes in Ceph are synchronous, so keep in mind that any increase in latency will decrease the write performance/latency on your Ceph cluster.

> AFAICS, Ceph does not support data encryption at connection level yet,
> so we would have to setup VPN links between the two cluster networks.
> This means extra configuration, maintenance and overhead.
> 
> How far away is TLS support or something similar for the Ceph
> connections? AFAIK, TLS support should not be hard to implement, but I
> am not too familiar with Ceph internals.
> 

Afaik there is no work currently in progress.

You might implement IPSec on the nodes itself. You might call that a VPN obviously, but IPSec can also just encrypt packets between nodes.

Wido

> Thoughts?
> 
> Amon Ott
> -- 
> Dr. Amon Ott
> m-privacy GmbH           Tel: +49 30 24342334
> Werner-Voß-Damm 62       Fax: +49 30 99296856
> 12101 Berlin             http://www.m-privacy.de
> 
> Amtsgericht Charlottenburg, HRB 84946
> 
> Geschäftsführer:
>  Dipl.-Kfm. Holger Maczkowsky,
>  Roman Maczkowsky
> 
> GnuPG-Key-ID: 0x2DD3A649
> 
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Ceph connections with TLS

[Amon Ott]

Am 16.12.2016 um 12:10 schrieb Wido den Hollander:
> 
>> Op 16 december 2016 om 11:15 schreef Amon Ott <a.ott@m-privacy.de>:
>> for a customer we are currently designing a ceph cluster, which shall be
>> spread over two data centers. Data in the Ceph cluster is slightly
>> confidential, so we would like to encrypt at least all Ceph traffic over
>> the fast data center connection link.
>>
> 
> Fast as in low latency? Writes in Ceph are synchronous, so keep in mind that any increase in latency will decrease the write performance/latency on your Ceph cluster.

Yes, we expect low latency and high bandwidth, but a connection shared
with other services. Having locality for accesses by clients would of
course help, too.

>> AFAICS, Ceph does not support data encryption at connection level yet,
>> so we would have to setup VPN links between the two cluster networks.
>> This means extra configuration, maintenance and overhead.
>>
>> How far away is TLS support or something similar for the Ceph
>> connections? AFAIK, TLS support should not be hard to implement, but I
>> am not too familiar with Ceph internals.
>>
> 
> Afaik there is no work currently in progress.

I feared so.

> You might implement IPSec on the nodes itself. You might call that a VPN obviously, but IPSec can also just encrypt packets between nodes.

Having quite a bit of experience with OpenVPN, we would rather use that
than IPSec, and we would do it right on the Ceph nodes. Still, TLS in
the connections would be better for lower latency.

Also, we then need VPN connections between all nodes as well as from all
clients to all nodes in the other location. The latter could be routed
through local Ceph nodes, but that would give even more latency.

Thanks for your answer,

Amon.
-- 
Dr. Amon Ott
m-privacy GmbH           Tel: +49 30 24342334
Werner-Voß-Damm 62       Fax: +49 30 99296856
12101 Berlin             http://www.m-privacy.de

Amtsgericht Charlottenburg, HRB 84946

Geschäftsführer:
 Dipl.-Kfm. Holger Maczkowsky,
 Roman Maczkowsky

GnuPG-Key-ID: 0x2DD3A649

Re: Ceph connections with TLS

[Wido den Hollander]

> Op 16 december 2016 om 12:39 schreef Amon Ott <a.ott@m-privacy.de>:
> 
> 
> Am 16.12.2016 um 12:10 schrieb Wido den Hollander:
> > 
> >> Op 16 december 2016 om 11:15 schreef Amon Ott <a.ott@m-privacy.de>:
> >> for a customer we are currently designing a ceph cluster, which shall be
> >> spread over two data centers. Data in the Ceph cluster is slightly
> >> confidential, so we would like to encrypt at least all Ceph traffic over
> >> the fast data center connection link.
> >>
> > 
> > Fast as in low latency? Writes in Ceph are synchronous, so keep in mind that any increase in latency will decrease the write performance/latency on your Ceph cluster.
> 
> Yes, we expect low latency and high bandwidth, but a connection shared
> with other services. Having locality for accesses by clients would of
> course help, too.
> 

You can play a bit with primary affinity or modify CRUSH in such a way that the primary OSDs are always in the local DC.

> >> AFAICS, Ceph does not support data encryption at connection level yet,
> >> so we would have to setup VPN links between the two cluster networks.
> >> This means extra configuration, maintenance and overhead.
> >>
> >> How far away is TLS support or something similar for the Ceph
> >> connections? AFAIK, TLS support should not be hard to implement, but I
> >> am not too familiar with Ceph internals.
> >>
> > 
> > Afaik there is no work currently in progress.
> 
> I feared so.
> 
> > You might implement IPSec on the nodes itself. You might call that a VPN obviously, but IPSec can also just encrypt packets between nodes.
> 
> Having quite a bit of experience with OpenVPN, we would rather use that
> than IPSec, and we would do it right on the Ceph nodes. Still, TLS in
> the connections would be better for lower latency.
> 
> Also, we then need VPN connections between all nodes as well as from all
> clients to all nodes in the other location. The latter could be routed
> through local Ceph nodes, but that would give even more latency.
> 

True. Isn't the a possibility to encrypt the data before it goes into Ceph? I don't know what the use-case is, but if it is RBD why not encrypt the whole RBD device with LUKS / dm-crypt?

Just an idea though.

Wido

> Thanks for your answer,
> 
> Amon.
> -- 
> Dr. Amon Ott
> m-privacy GmbH           Tel: +49 30 24342334
> Werner-Voß-Damm 62       Fax: +49 30 99296856
> 12101 Berlin             http://www.m-privacy.de
> 
> Amtsgericht Charlottenburg, HRB 84946
> 
> Geschäftsführer:
>  Dipl.-Kfm. Holger Maczkowsky,
>  Roman Maczkowsky
> 
> GnuPG-Key-ID: 0x2DD3A649
> 
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Ceph connections with TLS

[David Byte]

There are also NICS and switches that do the encryption at the hardware level. That is probably worth investigating. 

David Byte
Sr. Technical Strategist
IHV Alliances and Embedded
SUSE

Sent from my iPhone. Typos are Apple's fault. 

> On Dec 16, 2016, at 6:31 AM, Wido den Hollander <wido@42on.com> wrote:
> 
> 
>> Op 16 december 2016 om 12:39 schreef Amon Ott <a.ott@m-privacy.de>:
>> 
>> 
>>> Am 16.12.2016 um 12:10 schrieb Wido den Hollander:
>>> 
>>>> Op 16 december 2016 om 11:15 schreef Amon Ott <a.ott@m-privacy.de>:
>>>> for a customer we are currently designing a ceph cluster, which shall be
>>>> spread over two data centers. Data in the Ceph cluster is slightly
>>>> confidential, so we would like to encrypt at least all Ceph traffic over
>>>> the fast data center connection link.
>>>> 
>>> 
>>> Fast as in low latency? Writes in Ceph are synchronous, so keep in mind that any increase in latency will decrease the write performance/latency on your Ceph cluster.
>> 
>> Yes, we expect low latency and high bandwidth, but a connection shared
>> with other services. Having locality for accesses by clients would of
>> course help, too.
>> 
> 
> You can play a bit with primary affinity or modify CRUSH in such a way that the primary OSDs are always in the local DC.
> 
>>>> AFAICS, Ceph does not support data encryption at connection level yet,
>>>> so we would have to setup VPN links between the two cluster networks.
>>>> This means extra configuration, maintenance and overhead.
>>>> 
>>>> How far away is TLS support or something similar for the Ceph
>>>> connections? AFAIK, TLS support should not be hard to implement, but I
>>>> am not too familiar with Ceph internals.
>>>> 
>>> 
>>> Afaik there is no work currently in progress.
>> 
>> I feared so.
>> 
>>> You might implement IPSec on the nodes itself. You might call that a VPN obviously, but IPSec can also just encrypt packets between nodes.
>> 
>> Having quite a bit of experience with OpenVPN, we would rather use that
>> than IPSec, and we would do it right on the Ceph nodes. Still, TLS in
>> the connections would be better for lower latency.
>> 
>> Also, we then need VPN connections between all nodes as well as from all
>> clients to all nodes in the other location. The latter could be routed
>> through local Ceph nodes, but that would give even more latency.
>> 
> 
> True. Isn't the a possibility to encrypt the data before it goes into Ceph? I don't know what the use-case is, but if it is RBD why not encrypt the whole RBD device with LUKS / dm-crypt?
> 
> Just an idea though.
> 
> Wido
> 
>> Thanks for your answer,
>> 
>> Amon.
>> -- 
>> Dr. Amon Ott
>> m-privacy GmbH           Tel: +49 30 24342334
>> Werner-Voß-Damm 62       Fax: +49 30 99296856
>> 12101 Berlin             http://www.m-privacy.de
>> 
>> Amtsgericht Charlottenburg, HRB 84946
>> 
>> Geschäftsführer:
>> Dipl.-Kfm. Holger Maczkowsky,
>> Roman Maczkowsky
>> 
>> GnuPG-Key-ID: 0x2DD3A649
>> 
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: Ceph connections with TLS

[Sage Weil]

On Fri, 16 Dec 2016, Amon Ott wrote:
> Hello Ceph,
> 
> for a customer we are currently designing a ceph cluster, which shall be
> spread over two data centers. Data in the Ceph cluster is slightly
> confidential, so we would like to encrypt at least all Ceph traffic over
> the fast data center connection link.
> 
> AFAICS, Ceph does not support data encryption at connection level yet,
> so we would have to setup VPN links between the two cluster networks.
> This means extra configuration, maintenance and overhead.
> 
> How far away is TLS support or something similar for the Ceph
> connections? AFAIK, TLS support should not be hard to implement, but I
> am not too familiar with Ceph internals.

I hope to work on the msgr2 protocol change (which will enable encryption 
on the wire) during the next cycle, but I definitely can't promise it'll 
happen by luminous.  In the meantime, you'll need to this in the network 
layer.

Also, note that a stretch cluster will (1) increase latency and that (2) 
two is a bad number of datacenters because you won't be able to establish 
a quorum if the one with the majority of mons goes down.  You'll probably 
want to put one or more mons in a third data center to act as an arbiter.  
But in general these stretch clusters are tricky get set up in a way that 
doesn't break in a failure situation so proceed with extreme caution!

sage

Re: Ceph connections with TLS

[Amon Ott]

Am 16.12.2016 um 15:28 schrieb Sage Weil:
> On Fri, 16 Dec 2016, Amon Ott wrote:
>> How far away is TLS support or something similar for the Ceph
>> connections? AFAIK, TLS support should not be hard to implement, but I
>> am not too familiar with Ceph internals.
> 
> I hope to work on the msgr2 protocol change (which will enable encryption 
> on the wire) during the next cycle, but I definitely can't promise it'll 
> happen by luminous.  In the meantime, you'll need to this in the network 
> layer.

Ok, looking forward to those changes. And we will setup a VPN
infrastructure for now.

> Also, note that a stretch cluster will (1) increase latency and that (2) 
> two is a bad number of datacenters because you won't be able to establish 
> a quorum if the one with the majority of mons goes down.  You'll probably 
> want to put one or more mons in a third data center to act as an arbiter.  
> But in general these stretch clusters are tricky get set up in a way that 
> doesn't break in a failure situation so proceed with extreme caution!

For the quorum we plan to have an extra mon node in a separate building
containing the switches that connect everything. There is just enough
space for this extra node.

Thanks!

Amon.
-- 
Dr. Amon Ott
m-privacy GmbH           Tel: +49 30 24342334
Werner-Voß-Damm 62       Fax: +49 30 99296856
12101 Berlin             http://www.m-privacy.de

Amtsgericht Charlottenburg, HRB 84946

Geschäftsführer:
 Dipl.-Kfm. Holger Maczkowsky,
 Roman Maczkowsky

GnuPG-Key-ID: 0x2DD3A649