quic in-kernel implementation?

quic in-kernel implementation?

[Alexander Ahring Oder Aring]

Hi,

as I notice there exists several quic user space implementations, is
there any interest or process of doing an in-kernel implementation? I
am asking because I would like to try out quic with an in-kernel
application protocol like DLM. Besides DLM I've heard that the SMB
community is also interested into such implementation.

- Alex

Re: quic in-kernel implementation?

[Aurélien Aptel]

Alexander Ahring Oder Aring <aahringo@redhat.com> writes:
> as I notice there exists several quic user space implementations, is
> there any interest or process of doing an in-kernel implementation? I
> am asking because I would like to try out quic with an in-kernel
> application protocol like DLM. Besides DLM I've heard that the SMB
> community is also interested into such implementation.

Yes SMB can work over QUIC. It would be nice if there was an in-kernel
implementation that cifs.ko could use. Many firewall block port 445
(SMB) despite the newer version of the protocol now having encryption,
signing, etc. Using QUIC (UDP port 443) would allow for more reliable
connectivity to cloud storage like azure.

There are already multiple well-tested C QUIC implementation out there
(Microsoft one for example, has a lot of extra code annotation to allow
for deep static analysis) but I'm not sure how we would go about porting
it to linux.

https://github.com/microsoft/msquic

Cheers,
-- 
Aurélien Aptel / SUSE Labs Samba Team
GPG: 1839 CB5F 9F5B FB9B AA97  8C99 03C8 A49B 521B D5D3
SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg, DE
GF: Felix Imendörffer, Mary Higgins, Sri Rasiah HRB 247165 (AG München)

Re: quic in-kernel implementation?

[Steve French]

On Mon, Jun 7, 2021 at 11:45 AM Aurélien Aptel <aaptel@suse.com> wrote:
>
> Alexander Ahring Oder Aring <aahringo@redhat.com> writes:
> > as I notice there exists several quic user space implementations, is
> > there any interest or process of doing an in-kernel implementation? I
> > am asking because I would like to try out quic with an in-kernel
> > application protocol like DLM. Besides DLM I've heard that the SMB
> > community is also interested into such implementation.
>
> Yes SMB can work over QUIC. It would be nice if there was an in-kernel
> implementation that cifs.ko could use. Many firewall block port 445
> (SMB) despite the newer version of the protocol now having encryption,
> signing, etc. Using QUIC (UDP port 443) would allow for more reliable
> connectivity to cloud storage like azure.
>
> There are already multiple well-tested C QUIC implementation out there
> (Microsoft one for example, has a lot of extra code annotation to allow
> for deep static analysis) but I'm not sure how we would go about porting
> it to linux.
>
> https://github.com/microsoft/msquic

Since the Windows implementation of SMB3.1.1 over QUIC appears stable
(for quite a while now) and well tested, and even wireshark can now decode it, a
possible sequence of steps has been discussed similar to the below:

1) using a userspace port of QUIC (e.g. msquic since is one of the more tested
ports, and apparently similar to what already works well for QUIC on Windows
with SMB3.1.1) finish up the SMB3.1.1 kernel pieces needed for running over
QUIC
2) then switch focus to porting a smaller C userspace implementation of
QUIC to Linux (probably not msquic since it is larger and doesn't
follow kernel style)
to kernel in fs/cifs  (since currently SMB3.1.1 is the only protocol
that uses QUIC,
and the Windows server target is quite stable and can be used to test against)
3) use the userspace upcall example from step 1 for
comparison/testing/debugging etc.
since we know the userspace version is stable
4) Once SMB3.1.1 over QUIC is no longer experimental, remove, and
we are convinced it (kernel QUIC port) works well with SMB3.1.1
to servers which support QUIC, then move the quic code from fs/cifs to the /net
tree




-- 
Thanks,

Steve

Re: quic in-kernel implementation?

[Stefan Metzmacher]

Am 08.06.21 um 05:04 schrieb Steve French:
> On Mon, Jun 7, 2021 at 11:45 AM Aurélien Aptel <aaptel@suse.com> wrote:
>>
>> Alexander Ahring Oder Aring <aahringo@redhat.com> writes:
>>> as I notice there exists several quic user space implementations, is
>>> there any interest or process of doing an in-kernel implementation? I
>>> am asking because I would like to try out quic with an in-kernel
>>> application protocol like DLM. Besides DLM I've heard that the SMB
>>> community is also interested into such implementation.
>>
>> Yes SMB can work over QUIC. It would be nice if there was an in-kernel
>> implementation that cifs.ko could use. Many firewall block port 445
>> (SMB) despite the newer version of the protocol now having encryption,
>> signing, etc. Using QUIC (UDP port 443) would allow for more reliable
>> connectivity to cloud storage like azure.
>>
>> There are already multiple well-tested C QUIC implementation out there
>> (Microsoft one for example, has a lot of extra code annotation to allow
>> for deep static analysis) but I'm not sure how we would go about porting
>> it to linux.
>>
>> https://github.com/microsoft/msquic
> 
> Since the Windows implementation of SMB3.1.1 over QUIC appears stable
> (for quite a while now) and well tested, and even wireshark can now decode it, a
> possible sequence of steps has been discussed similar to the below:
> 
> 1) using a userspace port of QUIC (e.g. msquic since is one of the more tested
> ports, and apparently similar to what already works well for QUIC on Windows
> with SMB3.1.1) finish up the SMB3.1.1 kernel pieces needed for running over
> QUIC

Instead of using userspace upcalls directly, it would be great if we could hide
behind a fuse-like socket type, in order to keep the kernel changes in fs/cifs (and other parts)
tiny and just replace the socket(AF_INET) call, but continue to use a
stream socket (likely with a few QUIC specific getsockopt/setsockopt calls).

It would also allow userspace applications like Samba's smbclient and smbd
to use it that way too.

> 2) then switch focus to porting a smaller C userspace implementation of
> QUIC to Linux (probably not msquic since it is larger and doesn't
> follow kernel style)
> to kernel in fs/cifs  (since currently SMB3.1.1 is the only protocol
> that uses QUIC,
> and the Windows server target is quite stable and can be used to test against)> 3) use the userspace upcall example from step 1 for
> comparison/testing/debugging etc.
> since we know the userspace version is stable

With having the fuse-like socket before it should be trivial to switch
between the implementations.

> 4) Once SMB3.1.1 over QUIC is no longer experimental, remove, and
> we are convinced it (kernel QUIC port) works well with SMB3.1.1
> to servers which support QUIC, then move the quic code from fs/cifs to the /net
> tree

The 4th step would then finally allocate a stable PF_QUIC which would be
ABI stable.

metze

Re: quic in-kernel implementation?

[Alexander Aring]

Hi,

On Tue, Jun 8, 2021 at 3:36 AM Stefan Metzmacher <metze@samba.org> wrote:
...
>
> > 2) then switch focus to porting a smaller C userspace implementation of
> > QUIC to Linux (probably not msquic since it is larger and doesn't
> > follow kernel style)
> > to kernel in fs/cifs  (since currently SMB3.1.1 is the only protocol
> > that uses QUIC,
> > and the Windows server target is quite stable and can be used to test against)> 3) use the userspace upcall example from step 1 for
> > comparison/testing/debugging etc.
> > since we know the userspace version is stable
>
> With having the fuse-like socket before it should be trivial to switch
> between the implementations.

So a good starting point would be to have such a "fuse-like socket"
component? What about having a simple example for that at first
without having quic involved. The kernel calls some POSIX-like socket
interface which triggers a communication to a user space application.
This user space application will then map everything to a user space
generated socket. This would be a map from socket struct
"proto/proto_ops" to user space and vice versa. The kernel application
probably can use the kernel_FOO() (e.g. kernel_recvmsg()) socket api
directly then. Exactly like "fuse" as you mentioned just for sockets.

I think two veth interfaces can help to test something like that,
either with a "fuse-like socket" on the other end or an user space
application. Just doing a ping-pong example.

Afterwards we can look at how to replace the user generated socket
application with any $LIBQUIC e.g. msquic implementation as second
step.

- Alex

Re: quic in-kernel implementation?

[Stephen Hemminger]

On Tue, 8 Jun 2021 17:03:16 -0400
Alexander Aring <aahringo@redhat.com> wrote:

> Hi,
> 
> On Tue, Jun 8, 2021 at 3:36 AM Stefan Metzmacher <metze@samba.org> wrote:
> ...
> >  
> > > 2) then switch focus to porting a smaller C userspace implementation of
> > > QUIC to Linux (probably not msquic since it is larger and doesn't
> > > follow kernel style)
> > > to kernel in fs/cifs  (since currently SMB3.1.1 is the only protocol
> > > that uses QUIC,
> > > and the Windows server target is quite stable and can be used to test against)> 3) use the userspace upcall example from step 1 for
> > > comparison/testing/debugging etc.
> > > since we know the userspace version is stable  
> >
> > With having the fuse-like socket before it should be trivial to switch
> > between the implementations.  
> 
> So a good starting point would be to have such a "fuse-like socket"
> component? What about having a simple example for that at first
> without having quic involved. The kernel calls some POSIX-like socket
> interface which triggers a communication to a user space application.
> This user space application will then map everything to a user space
> generated socket. This would be a map from socket struct
> "proto/proto_ops" to user space and vice versa. The kernel application
> probably can use the kernel_FOO() (e.g. kernel_recvmsg()) socket api
> directly then. Exactly like "fuse" as you mentioned just for sockets.
> 
> I think two veth interfaces can help to test something like that,
> either with a "fuse-like socket" on the other end or an user space
> application. Just doing a ping-pong example.
> 
> Afterwards we can look at how to replace the user generated socket
> application with any $LIBQUIC e.g. msquic implementation as second
> step.
> 
> - Alex
> 

Socket state management is complex and timers etc in userspace are hard.

Re: quic in-kernel implementation?

[Jakub Kicinski]

On Tue, 8 Jun 2021 15:33:49 -0700 Stephen Hemminger wrote:
> On Tue, 8 Jun 2021 17:03:16 -0400
> > > With having the fuse-like socket before it should be trivial to switch
> > > between the implementations.    
> > 
> > So a good starting point would be to have such a "fuse-like socket"
> > component? What about having a simple example for that at first
> > without having quic involved. The kernel calls some POSIX-like socket
> > interface which triggers a communication to a user space application.
> > This user space application will then map everything to a user space
> > generated socket. This would be a map from socket struct
> > "proto/proto_ops" to user space and vice versa. The kernel application
> > probably can use the kernel_FOO() (e.g. kernel_recvmsg()) socket api
> > directly then. Exactly like "fuse" as you mentioned just for sockets.
> > 
> > I think two veth interfaces can help to test something like that,
> > either with a "fuse-like socket" on the other end or an user space
> > application. Just doing a ping-pong example.
> > 
> > Afterwards we can look at how to replace the user generated socket
> > application with any $LIBQUIC e.g. msquic implementation as second
> > step.
> 
> Socket state management is complex and timers etc in userspace are hard.

+1 seeing the struggles fuse causes in storage land "fuse for sockets"
is not an exciting temporary solution IMHO..

Re: quic in-kernel implementation?

[Alexander Aring]

Hi,

On Wed, Jun 9, 2021 at 12:48 PM Jakub Kicinski <kuba@kernel.org> wrote:
>
> On Tue, 8 Jun 2021 15:33:49 -0700 Stephen Hemminger wrote:
> > On Tue, 8 Jun 2021 17:03:16 -0400
> > > > With having the fuse-like socket before it should be trivial to switch
> > > > between the implementations.
> > >
> > > So a good starting point would be to have such a "fuse-like socket"
> > > component? What about having a simple example for that at first
> > > without having quic involved. The kernel calls some POSIX-like socket
> > > interface which triggers a communication to a user space application.
> > > This user space application will then map everything to a user space
> > > generated socket. This would be a map from socket struct
> > > "proto/proto_ops" to user space and vice versa. The kernel application
> > > probably can use the kernel_FOO() (e.g. kernel_recvmsg()) socket api
> > > directly then. Exactly like "fuse" as you mentioned just for sockets.
> > >
> > > I think two veth interfaces can help to test something like that,
> > > either with a "fuse-like socket" on the other end or an user space
> > > application. Just doing a ping-pong example.
> > >
> > > Afterwards we can look at how to replace the user generated socket
> > > application with any $LIBQUIC e.g. msquic implementation as second
> > > step.
> >
> > Socket state management is complex and timers etc in userspace are hard.
>
> +1 seeing the struggles fuse causes in storage land "fuse for sockets"
> is not an exciting temporary solution IMHO..

What about an in-kernel sunrpc client which forwards "in-kernel proxy
socket syscall functions" to a user server who executes those on a
user socket? Does this sound like a better approach?
Sure there may be more problems, but maybe we could try it with
something simple at first to discover all those problems.

- Alex

RE: quic in-kernel implementation?

[David Laight]

From: Jakub Kicinski
> Sent: 09 June 2021 17:48
...
> > > I think two veth interfaces can help to test something like that,
> > > either with a "fuse-like socket" on the other end or an user space
> > > application. Just doing a ping-pong example.
> > >
> > > Afterwards we can look at how to replace the user generated socket
> > > application with any $LIBQUIC e.g. msquic implementation as second
> > > step.
> >
> > Socket state management is complex and timers etc in userspace are hard.
> 
> +1 seeing the struggles fuse causes in storage land "fuse for sockets"
> is not an exciting temporary solution IMHO..

Especially since you'd want reasonable performance for quic.

Fuse is normally used to access obscure filesystems where
you just need access, rather than something that really
needs to be quick.

	David

-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK
Registration No: 1397386 (Wales)

Re: quic in-kernel implementation?

[Alexander Aring]

Hi,

On Sun, Jun 13, 2021 at 8:17 AM David Laight <David.Laight@aculab.com> wrote:
>
> From: Jakub Kicinski
> > Sent: 09 June 2021 17:48
> ...
> > > > I think two veth interfaces can help to test something like that,
> > > > either with a "fuse-like socket" on the other end or an user space
> > > > application. Just doing a ping-pong example.
> > > >
> > > > Afterwards we can look at how to replace the user generated socket
> > > > application with any $LIBQUIC e.g. msquic implementation as second
> > > > step.
> > >
> > > Socket state management is complex and timers etc in userspace are hard.
> >
> > +1 seeing the struggles fuse causes in storage land "fuse for sockets"
> > is not an exciting temporary solution IMHO..
>
> Especially since you'd want reasonable performance for quic.
>
> Fuse is normally used to access obscure filesystems where
> you just need access, rather than something that really
> needs to be quick.
>

or you have library dependencies like sshfs. That is the case in quic
for some parts of TLS (see TLS socket API). Sure it will not be the
final solution, that was never the intention. It is to establish a
kernel-API which will be replaced for a final in-kernel solution later
and not trying to solve all problems at once.

- Alex

Re: quic in-kernel implementation?

[Chuck Lever III]

> On Jun 8, 2021, at 3:36 AM, Stefan Metzmacher <metze@samba.org> wrote:
> 
> Am 08.06.21 um 05:04 schrieb Steve French:
>> On Mon, Jun 7, 2021 at 11:45 AM Aurélien Aptel <aaptel@suse.com> wrote:
>>> 
>>> Alexander Ahring Oder Aring <aahringo@redhat.com> writes:
>>>> as I notice there exists several quic user space implementations, is
>>>> there any interest or process of doing an in-kernel implementation? I
>>>> am asking because I would like to try out quic with an in-kernel
>>>> application protocol like DLM. Besides DLM I've heard that the SMB
>>>> community is also interested into such implementation.
>>> 
>>> Yes SMB can work over QUIC. It would be nice if there was an in-kernel
>>> implementation that cifs.ko could use. Many firewall block port 445
>>> (SMB) despite the newer version of the protocol now having encryption,
>>> signing, etc. Using QUIC (UDP port 443) would allow for more reliable
>>> connectivity to cloud storage like azure.
>>> 
>>> There are already multiple well-tested C QUIC implementation out there
>>> (Microsoft one for example, has a lot of extra code annotation to allow
>>> for deep static analysis) but I'm not sure how we would go about porting
>>> it to linux.
>>> 
>>> https://github.com/microsoft/msquic
>> 
>> Since the Windows implementation of SMB3.1.1 over QUIC appears stable
>> (for quite a while now) and well tested, and even wireshark can now decode it, a
>> possible sequence of steps has been discussed similar to the below:
>> 
>> 1) using a userspace port of QUIC (e.g. msquic since is one of the more tested
>> ports, and apparently similar to what already works well for QUIC on Windows
>> with SMB3.1.1) finish up the SMB3.1.1 kernel pieces needed for running over
>> QUIC
> 
> Instead of using userspace upcalls directly, it would be great if we could hide
> behind a fuse-like socket type, in order to keep the kernel changes in fs/cifs (and other parts)
> tiny and just replace the socket(AF_INET) call, but continue to use a
> stream socket (likely with a few QUIC specific getsockopt/setsockopt calls).
> 
> It would also allow userspace applications like Samba's smbclient and smbd
> to use it that way too.

That's interesting as a development scaffold.

However, IMO the interesting part of QUIC for us is transport
layer security. NFS already has TLS via RPC-over-TLS, and we
intend to have a full in-kernel implementation soon. Using a
user-space transport protocol implementation is likely to be
an unacceptable step backwards in terms of performance for us.
NFS connections are long-lived, no benefit at all from the
special 0-RTT mechanisms.

I hope the end goal is to have a full in-kernel implementation
of QUIC at some point, otherwise I don't see Linux QUIC ever
being on par with current TCP performance for a kernel
consumer.


>> 2) then switch focus to porting a smaller C userspace implementation of
>> QUIC to Linux (probably not msquic since it is larger and doesn't
>> follow kernel style)
>> to kernel in fs/cifs  (since currently SMB3.1.1 is the only protocol
>> that uses QUIC,
>> and the Windows server target is quite stable and can be used to test against)> 3) use the userspace upcall example from step 1 for
>> comparison/testing/debugging etc.
>> since we know the userspace version is stable
> 
> With having the fuse-like socket before it should be trivial to switch
> between the implementations.

Although switching QUIC implementations is a cool trick for
rapid prototyping, I'm unclear on the eventual user benefit
of it.


>> 4) Once SMB3.1.1 over QUIC is no longer experimental, remove, and
>> we are convinced it (kernel QUIC port) works well with SMB3.1.1
>> to servers which support QUIC, then move the quic code from fs/cifs to the /net
>> tree
> 
> The 4th step would then finally allocate a stable PF_QUIC which would be
> ABI stable.
> 
> metze

--
Chuck Lever

Re: quic in-kernel implementation?

[Chuck Lever III]

> On Jun 7, 2021, at 11:25 AM, Alexander Ahring Oder Aring <aahringo@redhat.com> wrote:
> 
> Hi,
> 
> as I notice there exists several quic user space implementations, is
> there any interest or process of doing an in-kernel implementation? I
> am asking because I would like to try out quic with an in-kernel
> application protocol like DLM. Besides DLM I've heard that the SMB
> community is also interested into such implementation.

The NFS community has standardized RPC-over-TLS as part of an
effort to prepare for running NFS on QUIC transports.

https://datatracker.ietf.org/doc/draft-ietf-nfsv4-rpc-tls/

Towards that end, an in-kernel QUIC would need a way to perform
transport security handshakes. Tempesta has one we are looking
closely at -- it can support both TLS and QUIC.

There's more work to be done to address the ways in which QUIC
is different than existing transports (eg. its support for
streams and transactions); however if Linux were to gain an
in-kernel QUIC implementation, there is interest in seeing NFS
use it.


--
Chuck Lever

Re: quic in-kernel implementation?

[Vadim Fedorenko]

On 07.06.2021 16:25, Alexander Ahring Oder Aring wrote:
> Hi,
> 
> as I notice there exists several quic user space implementations, is
> there any interest or process of doing an in-kernel implementation? I
> am asking because I would like to try out quic with an in-kernel
> application protocol like DLM. Besides DLM I've heard that the SMB
> community is also interested into such implementation.
> 
> - Alex
> 

Hi!
I'm working on test in-kernel implementation of quic. It's based on the
kernel-tls work and uses the same ULP approach to setup connection
configuration. It's mostly about offload crypto operations of short header
to kernel and use user-space implementation to deal with any other types
of packets. Hope to test it till the end of June with some help from
Jakub.
Vadim

Re: quic in-kernel implementation?

[Alexander Aring]

Hi Vadim,

On Tue, Jun 8, 2021 at 4:59 PM Vadim Fedorenko <vfedorenko@novek.ru> wrote:
>
> On 07.06.2021 16:25, Alexander Ahring Oder Aring wrote:
> > Hi,
> >
> > as I notice there exists several quic user space implementations, is
> > there any interest or process of doing an in-kernel implementation? I
> > am asking because I would like to try out quic with an in-kernel
> > application protocol like DLM. Besides DLM I've heard that the SMB
> > community is also interested into such implementation.
> >
> > - Alex
> >
>
> Hi!
> I'm working on test in-kernel implementation of quic. It's based on the
> kernel-tls work and uses the same ULP approach to setup connection
> configuration. It's mostly about offload crypto operations of short header
> to kernel and use user-space implementation to deal with any other types
> of packets. Hope to test it till the end of June with some help from
> Jakub.

Thanks, sounds interesting. Does this allow the kernel to create a quic socket?

- Alex

Re: quic in-kernel implementation?

[Vadim Fedorenko]

On 08.06.2021 22:06, Alexander Aring wrote:
> Hi Vadim,
> 
> On Tue, Jun 8, 2021 at 4:59 PM Vadim Fedorenko <vfedorenko@novek.ru> wrote:
>>
>> On 07.06.2021 16:25, Alexander Ahring Oder Aring wrote:
>>> Hi,
>>>
>>> as I notice there exists several quic user space implementations, is
>>> there any interest or process of doing an in-kernel implementation? I
>>> am asking because I would like to try out quic with an in-kernel
>>> application protocol like DLM. Besides DLM I've heard that the SMB
>>> community is also interested into such implementation.
>>>
>>> - Alex
>>>
>>
>> Hi!
>> I'm working on test in-kernel implementation of quic. It's based on the
>> kernel-tls work and uses the same ULP approach to setup connection
>> configuration. It's mostly about offload crypto operations of short header
>> to kernel and use user-space implementation to deal with any other types
>> of packets. Hope to test it till the end of June with some help from
>> Jakub.
> 
> Thanks, sounds interesting. Does this allow the kernel to create a quic socket?
> 

Not exactly. It's based on top of UDP socket and is configured by setsockopt
like it's done for Kernel TLS implementation. The main point of this work is to
offload cryptography only without implementing special address family.

Re: quic in-kernel implementation?

[Eric Curtin]

Hi Guys,

Great idea, something I have been hoping to see in the kernel for a
while. How has your implementation been going @Vadim? I'd be interested
in a non-encrypted version of QUIC also in the kernel (may not be
supported in the spec but possible and I think worth having), would be
useful for cases where you don't care about network ossification
protection or data-in-transit encryption, say a trusted local network
where you would prefer the performance and reliability advantages of
going plaintext and you don't want to figure out how to deploy
certififcates. Something that could be used as a straight swap for a
TCP socket.

Re: quic in-kernel implementation?

[Steve French]

I am interested in this as well (encrypted and non-encrypted QUIC
cases in kernel)

Short term given that Windows is the only server than currently
support it - testing probably needs to be done via upcall with
SMB3.1.1 Linux mounts to Windows, and once that is verified to work
(as a baseline for comparison) - start work on the kernel QUIC driver

But for the initial point of comparison - would be helpful to have
example code that exposes a kernel "socket like" ("sock_sendmsg") API
for upcall ... and once we verify that that works start the work on
the kernel driver

On Sun, Sep 5, 2021 at 9:10 AM Eric Curtin <ericcurtin17@gmail.com> wrote:
>
> Hi Guys,
>
> Great idea, something I have been hoping to see in the kernel for a
> while. How has your implementation been going @Vadim? I'd be interested
> in a non-encrypted version of QUIC also in the kernel (may not be
> supported in the spec but possible and I think worth having), would be
> useful for cases where you don't care about network ossification
> protection or data-in-transit encryption, say a trusted local network
> where you would prefer the performance and reliability advantages of
> going plaintext and you don't want to figure out how to deploy
> certififcates. Something that could be used as a straight swap for a
> TCP socket.
>


-- 
Thanks,

Steve