[LSF/MM/BPF TOPIC][LSF/MM/BPF ATTEND] TLS handshake for in-kernel consumers

[LSF/MM/BPF TOPIC][LSF/MM/BPF ATTEND] TLS handshake for in-kernel consumers

[Hannes Reinecke]

Hi all,

nvme-over-tcp has the option to utilize TLS for encrypted traffic, but 
due to the internal design of the nvme-over-fabrics stack we cannot 
initiate the TLS connection from userspace (as the current in-kernel TLS 
implementation is designed).

This leaves us with two options:
1) Put TLS handshake into the kernel (which will be quite some
   discussion as it's arguably a userspace configuration)
2) Pass an in-kernel socket to userspace and have a userspace
   application to run the TLS handshake.

None of these options are quiet clear cut, as we will be have to put
quite some complexity into the kernel to do full TLS handshake (if we
were to go with option 1) or will have to design a mechanism to pass
an in-kernel socket to userspace as we don't do that currently (if we 
were going with option 2).

We have been discussing some ideas on how to implement option 2 
(together with Chuck Lever and the NFS crowd), but so far haven't been 
able to come up with a decent design.

So I would like to discuss with interested parties on how TLS handshake 
could be facilitated, and what would be the best design options here.

The proposed configd would be an option, but then we don't have that, 
either :-)

Required attendees:

Chuck Lever
James Bottomley
Sagi Grimberg
Keith Busch
Christoph Hellwig
David Howells

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		           Kernel Storage Architect
hare@suse.de			                  +49 911 74053 688
SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), GF: Felix Imendörffer

Re: [LSF/MM/BPF TOPIC] [LSF/MM/BPF ATTEND] TLS handshake for in-kernel consumers

[Benjamin Coddington]

On 2 Feb 2022, at 9:12, Hannes Reinecke wrote:

> Hi all,
>
> nvme-over-tcp has the option to utilize TLS for encrypted traffic, but 
> due to the internal design of the nvme-over-fabrics stack we cannot 
> initiate the TLS connection from userspace (as the current in-kernel 
> TLS implementation is designed).
>
> This leaves us with two options:
> 1) Put TLS handshake into the kernel (which will be quite some
>   discussion as it's arguably a userspace configuration)
> 2) Pass an in-kernel socket to userspace and have a userspace
>   application to run the TLS handshake.
>
> None of these options are quiet clear cut, as we will be have to put
> quite some complexity into the kernel to do full TLS handshake (if we
> were to go with option 1) or will have to design a mechanism to pass
> an in-kernel socket to userspace as we don't do that currently (if we 
> were going with option 2).
>
> We have been discussing some ideas on how to implement option 2 
> (together with Chuck Lever and the NFS crowd), but so far haven't been 
> able to come up with a decent design.
>
> So I would like to discuss with interested parties on how TLS 
> handshake could be facilitated, and what would be the best design 
> options here.
>
> The proposed configd would be an option, but then we don't have that, 
> either :-)
>
> Required attendees:
>
> Chuck Lever
> James Bottomley
> Sagi Grimberg
> Keith Busch
> Christoph Hellwig
> David Howells

I'm interested in this as well, and have gone down quite a rabbit-hole 
of
experimental implementation for NFS.  I've found the keys API to be 
useful
to implement a tls_session keytype that allows kernel subsystems to 
request
that userspace do the heavy lifting of establishing a TLS session and
installing the kTLS bits on the socket by passing an existing socket fd 
to
userspace.

However, something more persistent than call_usermode_helper is needed,
since userspace helpers may not be able to be looked up from a 
filesystem
when establishing or re-keying (or doing a number of other things) on a 
TLS
session for NFS or NVMe-over-tcp.

I've got a rough idea to create a thing called a "keyagent" which would 
be
an alternative to using call_usermode_helper to service request-key.
Keyagents would be persistent processes in userspace themselves 
represented
by keys, and if a keyagent key is linked into a process' keyrings and
matches the requested type it is consulted to service request-key and 
update
for those keys.

Ben

Re: [LSF/MM/BPF TOPIC][LSF/MM/BPF ATTEND] TLS handshake for in-kernel consumers

[Chuck Lever III]

[ ... adding NFS and CIFS ... ]

> On Feb 2, 2022, at 9:12 AM, Hannes Reinecke <hare@suse.de> wrote:
> 
> Hi all,
> 
> nvme-over-tcp has the option to utilize TLS for encrypted traffic, but due to the internal design of the nvme-over-fabrics stack we cannot initiate the TLS connection from userspace (as the current in-kernel TLS implementation is designed).
> 
> This leaves us with two options:
> 1) Put TLS handshake into the kernel (which will be quite some
>  discussion as it's arguably a userspace configuration)
> 2) Pass an in-kernel socket to userspace and have a userspace
>  application to run the TLS handshake.
> 
> None of these options are quiet clear cut, as we will be have to put
> quite some complexity into the kernel to do full TLS handshake (if we
> were to go with option 1) or will have to design a mechanism to pass
> an in-kernel socket to userspace as we don't do that currently (if we were going with option 2).
> 
> We have been discussing some ideas on how to implement option 2 (together with Chuck Lever and the NFS crowd), but so far haven't been able to come up with a decent design.
> 
> So I would like to discuss with interested parties on how TLS handshake could be facilitated, and what would be the best design options here.

IMO we are a bit farther along than Hannes suggests, and I had
the impression that we have already agreed on a "decent design"
(see Ben Coddington's earlier post).

We currently have several prototypes we can discuss, and there
are some important issues on the table.

First, from the start we have recognized that we have a range
of potential in-kernel TLS consumers. To name a few: NVMe/TLS,
RPC-with-TLS (for in-transit NFS encryption), CIFS/SMB, and,
when it arrives, the QUICv1 transport. We don't intend to
build something that works for only one of these, thus it
will not be based on existing security infrastructure like
rpc.gssd.

Second, we believe in-kernel consumers will hitch onto the
existing kTLS infrastructure to handle payload encryption and
decryption. This transparently enables both software-based
and offload, and in the latter case, we hope for quite
reasonable performance.

As Hannes said, the missing piece is support for the TLS
handshake protocol to boot strap each TLS session.

The security community has demanded that we stick with user
space handshake implementations because they view the TLS
handshake as complex and a broad attack surface. I question
those assumptions, but even so...

We will need to have in-kernel handshake to support NFSROOT
and NVMe/TLS with a root filesystem, which are requirements
for the storage community.

We have an in-kernel prototype based on Tempesta's TLSv1.2
offload in the works. See the "topic-rpc-with-tls" branch:

 https://git.kernel.org/pub/scm/linux/kernel/git/cel/linux.git

We also have three promising user space upcall implementations
that are helping us with architectural choices. The main issue
here is how to set the correct peer authentication parameters
for each handshake. As Ben said, key rings can play a critical
part (as might netlink, but perhaps that can be avoided). We
are sensitive to containerization requirements as well.

One (not-yet-working) user space prototype is published in
the "topic-rpc-with-tls-upcall" branch in the repo above.


> The proposed configd would be an option, but then we don't have that, either :-)
> 
> Required attendees:
> 
> Chuck Lever
> James Bottomley
> Sagi Grimberg
> Keith Busch
> Christoph Hellwig
> David Howells

Anyone from the CIFS team? Enzo? How about Dave Miller?

Actually I think we need to have security and network folks
at the table. LSF/MM might not be the right venue for a
full-scale discussion of alternatives. We have been waiting
for an opportunity to bring this to a broad community event
such as Plumbers but the pandemic has interfered.

However, I am happy to discuss alternative upcall mechanisms,
new requirements, and anything related to securing an
in-kernel handshake against remote attack.

--
Chuck Lever

Re: [LSF/MM/BPF TOPIC][LSF/MM/BPF ATTEND] TLS handshake for in-kernel consumers

[Enzo Matsumiya]

On 02/02, Chuck Lever III wrote:
>[ ... adding NFS and CIFS ... ]
>
>> Required attendees:
>>
>> Chuck Lever
>> James Bottomley
>> Sagi Grimberg
>> Keith Busch
>> Christoph Hellwig
>> David Howells
>
>Anyone from the CIFS team? Enzo? How about Dave Miller?

I'll be attending representing the CIFS side. I hope to have my
prototype more solid and public by then.


Cheers,

Enzo

Re: [LSF/MM/BPF TOPIC][LSF/MM/BPF ATTEND] TLS handshake for in-kernel consumers

[Sagi Grimberg]

> Hi all,
> 
> nvme-over-tcp has the option to utilize TLS for encrypted traffic, but 
> due to the internal design of the nvme-over-fabrics stack we cannot 
> initiate the TLS connection from userspace (as the current in-kernel TLS 
> implementation is designed).
> 
> This leaves us with two options:
> 1) Put TLS handshake into the kernel (which will be quite some
>    discussion as it's arguably a userspace configuration)
> 2) Pass an in-kernel socket to userspace and have a userspace
>    application to run the TLS handshake.
> 
> None of these options are quiet clear cut, as we will be have to put
> quite some complexity into the kernel to do full TLS handshake (if we
> were to go with option 1) or will have to design a mechanism to pass
> an in-kernel socket to userspace as we don't do that currently (if we 
> were going with option 2).
> 
> We have been discussing some ideas on how to implement option 2 
> (together with Chuck Lever and the NFS crowd), but so far haven't been 
> able to come up with a decent design.
> 
> So I would like to discuss with interested parties on how TLS handshake 
> could be facilitated, and what would be the best design options here.
> 
> The proposed configd would be an option, but then we don't have that, 
> either :-)
> 
> Required attendees:
> 
> Chuck Lever
> James Bottomley
> Sagi Grimberg

I'd be interested to discuss this.

One other item with TLS besides the handshake part is that
nfs/cifs/nvme-tcp are all tcp ulps like tls itself, which at
the currently cannot be stacked IIRC (all use sk callbacks,
including tls).

Is anyone looking into enabling stacking tcp ulps on top of tls?

Re: [LSF/MM/BPF TOPIC][LSF/MM/BPF ATTEND] TLS handshake for in-kernel consumers

[Chaitanya Kulkarni]

>> The proposed configd would be an option, but then we don't have that, 
>> either :-)
>>
>> Required attendees:
>>
>> Chuck Lever
>> James Bottomley
>> Sagi Grimberg
> 
> I'd be interested to discuss this.
> 
> One other item with TLS besides the handshake part is that
> nfs/cifs/nvme-tcp are all tcp ulps like tls itself, which at
> the currently cannot be stacked IIRC (all use sk callbacks,
> including tls).
> 
> Is anyone looking into enabling stacking tcp ulps on top of tls?

I'm interested in attending this topic.

-ck

Re: [LSF/MM/BPF TOPIC][LSF/MM/BPF ATTEND] TLS handshake for in-kernel consumers

[Steve French]

This is an interesting question - and got me thinking about whether
also could be helpful to the critical question we have been asked
about multiple times for SMB, about how to get QUIC in the kernel (see
the various Storage Developer Conference presentations etc), while
putting as much as possible of the session establishment in userspace
accessed via upcalls.

I don't mind discussing this in detail at LSF (or at SambaXP a few
weeks after LSF), and since there are some SMB servers that support
QUIC already we could probably do at least some prototyping if there
is overlap between these two efforts.

On Wed, Feb 2, 2022 at 10:04 AM Chuck Lever III <chuck.lever@oracle.com> wrote:
>
> [ ... adding NFS and CIFS ... ]
>
> > On Feb 2, 2022, at 9:12 AM, Hannes Reinecke <hare@suse.de> wrote:
> >
> > Hi all,
> >
> > nvme-over-tcp has the option to utilize TLS for encrypted traffic, but due to the internal design of the nvme-over-fabrics stack we cannot initiate the TLS connection from userspace (as the current in-kernel TLS implementation is designed).
> >
> > This leaves us with two options:
> > 1) Put TLS handshake into the kernel (which will be quite some
> >  discussion as it's arguably a userspace configuration)
> > 2) Pass an in-kernel socket to userspace and have a userspace
> >  application to run the TLS handshake.
> >
> > None of these options are quiet clear cut, as we will be have to put
> > quite some complexity into the kernel to do full TLS handshake (if we
> > were to go with option 1) or will have to design a mechanism to pass
> > an in-kernel socket to userspace as we don't do that currently (if we were going with option 2).
> >
> > We have been discussing some ideas on how to implement option 2 (together with Chuck Lever and the NFS crowd), but so far haven't been able to come up with a decent design.
> >
> > So I would like to discuss with interested parties on how TLS handshake could be facilitated, and what would be the best design options here.
>
> IMO we are a bit farther along than Hannes suggests, and I had
> the impression that we have already agreed on a "decent design"
> (see Ben Coddington's earlier post).
>
> We currently have several prototypes we can discuss, and there
> are some important issues on the table.
>
> First, from the start we have recognized that we have a range
> of potential in-kernel TLS consumers. To name a few: NVMe/TLS,
> RPC-with-TLS (for in-transit NFS encryption), CIFS/SMB, and,
> when it arrives, the QUICv1 transport. We don't intend to
> build something that works for only one of these, thus it
> will not be based on existing security infrastructure like
> rpc.gssd.
>
> Second, we believe in-kernel consumers will hitch onto the
> existing kTLS infrastructure to handle payload encryption and
> decryption. This transparently enables both software-based
> and offload, and in the latter case, we hope for quite
> reasonable performance.
>
> As Hannes said, the missing piece is support for the TLS
> handshake protocol to boot strap each TLS session.
>
> The security community has demanded that we stick with user
> space handshake implementations because they view the TLS
> handshake as complex and a broad attack surface. I question
> those assumptions, but even so...
>
> We will need to have in-kernel handshake to support NFSROOT
> and NVMe/TLS with a root filesystem, which are requirements
> for the storage community.
>
> We have an in-kernel prototype based on Tempesta's TLSv1.2
> offload in the works. See the "topic-rpc-with-tls" branch:
>
>  https://git.kernel.org/pub/scm/linux/kernel/git/cel/linux.git
>
> We also have three promising user space upcall implementations
> that are helping us with architectural choices. The main issue
> here is how to set the correct peer authentication parameters
> for each handshake. As Ben said, key rings can play a critical
> part (as might netlink, but perhaps that can be avoided). We
> are sensitive to containerization requirements as well.
>
> One (not-yet-working) user space prototype is published in
> the "topic-rpc-with-tls-upcall" branch in the repo above.
>
>
> > The proposed configd would be an option, but then we don't have that, either :-)
> >
> > Required attendees:
> >
> > Chuck Lever
> > James Bottomley
> > Sagi Grimberg
> > Keith Busch
> > Christoph Hellwig
> > David Howells
>
> Anyone from the CIFS team? Enzo? How about Dave Miller?
>
> Actually I think we need to have security and network folks
> at the table. LSF/MM might not be the right venue for a
> full-scale discussion of alternatives. We have been waiting
> for an opportunity to bring this to a broad community event
> such as Plumbers but the pandemic has interfered.
>
> However, I am happy to discuss alternative upcall mechanisms,
> new requirements, and anything related to securing an
> in-kernel handshake against remote attack.
>
> --
> Chuck Lever
>
>
>


-- 
Thanks,

Steve