[SPDK] Re: SPDK socket abstraction layer

[Sasha Kotchubievsky <sashakot at dev.mellanox.co.il>]

[-- Attachment #1: Type: text/plain, Size: 7335 bytes --]

Hi Ben, 

Great list of patches,
This work will , definitely, take NVME-OF TCP at the next level.

We, also, work on zero-copy (TX) in initiator side based on MSG_ZEROCOPY. Preliminary results are great. In the target side, we still investigate the solution.

We run a lot of tests and see great potential for zero-copy. It look like a real bottleneck. On the send side, it can be removed with POSIX interface (MSG_ZEROCOPY), in receive side, it needs deep integration between TCP stack and SPDK. Next week we will have internal brain-storming, and, I hope, on dev meetup, I'll be ready for discussions.

We will invest in both directions: user-space path and in Linux Kernel path.  But, in user-space area, at this stage, VPP is out of our interest. 

Best regards
Sasha  

-----Original Message-----
From: Walker, Benjamin <benjamin.walker(a)intel.com> 
Sent: Wednesday, October 30, 2019 8:47 PM
To: spdk(a)lists.01.org
Subject: [SPDK] Re: SPDK socket abstraction layer

On Wed, 2019-10-30 at 17:54 +0000, Harris, James R wrote:
> Hi Sasha,
> 
> Tomek is only talking about the VPP implementation.  There are no 
> plans to remove the socket abstraction layer.  If anything, the 
> project needs to look at extending it in ways as you suggested.

To expand on this, there's a lot of activity right now in the SPDK sock abstraction layer to begin to implement asynchronous operations, zero copy operations, etc. For example, see:

Asynchronous writev
https://review.gerrithub.io/c/spdk/spdk/+/470523

MSG_ZEROCOPY use in the posix implementation
https://review.gerrithub.io/c/spdk/spdk/+/471752

A new sock implementation based on io_uring/libaio:
https://review.gerrithub.io/c/spdk/spdk/+/471314

And a new sock implementation based on Seastar:
https://review.gerrithub.io/c/spdk/spdk/+/466629

So not only is the sock abstraction layer sticking around, but it's getting a lot of focus going forward. There is a lot of innovation happening in the Linux kernel around networking at all layers that we need to keep up with.

One thing I would like community feedback on is what to do about the current VPP implementation. As we make improvements and additions to the sock abstraction, it will necessarily require updates to the VPP implementation. We can of course continue to make those, but does the community see value in maintaining support here? I'd really love to see someone take up the mantle on VPP if they believe there is value that we just haven't been able to unlock yet, but absent that it's just a maintenance burden.

Personally speaking, it would be easier for me, as someone trying to evolve the sock abstraction layer, to drop VPP. That's one less implementation that I then have to go update and test each time. But I'm very open to opinions and feedback here if anyone has something to say. SPDK obviously can't just drop support without strong consensus and a considerable amount of forewarning.

Thanks,
Ben

> 
> -Jim
> 
> 
> On 10/30/19, 10:50 AM, "Sasha Kotchubievsky" 
> <sashakot(a)dev.mellanox.co.il>
> wrote:
> 
>     Hi Tomek,
>     
>     Are you looking for community feedback regarding VPP implementation of TCP
>     stack, or about having socket abstraction layer in SPDK?
>     I think, socket abstraction layer is critical for future 
> integration between
>     SPDK and user-space stacks. In Mellanox, we're evaluating integration
>     between VMA (https://github.com/Mellanox/libvma) and SPDK. 
> Although, VMA can
>     be used as replacement for Kernel implementation of Posix socket 
> interface,
>     we see great potential in "deep" integration, which definitely needs keep
>     existing abstraction layer. For example, one of potential improvements can
>     be zero-copy in RX (receive) flow. I don't see how that can be implemented
>     on top of Linux Kernel stack. 
>     
>     Best regards
>     Sasha
>     
>     -----Original Message-----
>     From: Zawadzki, Tomasz <tomasz.zawadzki(a)intel.com> 
>     Sent: Monday, October 21, 2019 3:01 PM
>     To: Storage Performance Development Kit <spdk(a)lists.01.org>
>     Subject: [SPDK] SPDK socket abstraction layer
>     
>     Hello everyone,
>     
>     Summary:
>     
>     With this message I wanted to update SPDK community on state of VPP socket
>     abstraction as of SPDK 19.07 release.
>     At this time there does not seem to be a clear efficiency 
> improvements with
>     VPP. There is no further work planned on SPDK and VPP integration.
>     
>     Details:
>     
>     As some of you may remember, SPDK 18.04 release introduced support for
>     alternative socket types. Along with that release, Vector Packet 
> Processing
>     (VPP)<https://wiki.fd.io/view/VPP> 18.01 was integrated with SPDK, by
>     expanding socket abstraction to use VPP Communications Library (VCL).
> TCP/IP
>     stack in VPP<https://wiki.fd.io/view/VPP/HostStack> was in early 
> stages back
>     then and has seen improvements throughout the last year.
>     
>     To better use VPP capabilities, following fruitful collaboration with VPP
>     team, in SPDK 19.07, this implementation was changed from VCL to 
> VPP Session
>     API from VPP 19.04.2.
>     
>     VPP socket abstraction has met some challenges due to inherent design of
>     both projects, in particular related to running separate processes and
>     memory copies.
>     Seeing improvements from original implementation was encouraging, yet
>     measuring against posix socket abstraction (taking into 
> consideration entire
>     system, i.e. both processes), results are comparable. In other words,  at
>     this time there does not seem to be a clear benefit of either socket
>     abstraction from standpoint of CPU efficiency or IOPS.
>     
>     With this message I just wanted to update SPDK community on state 
> of socket
>     abstraction layers as of SPDK 19.07 release. Each SPDK release 
> always brings
>     improvements to the abstraction and its implementations, with 
> exciting work
>     on more efficient use of kernel TCP stack - changes in SPDK 19.10 and SPDK
>     20.01.
>     
>     However there is no active involvement at this point around VPP
>     implementation of socket abstraction in SPDK. Contributions in 
> this area are
>     always welcome. In case you're interested in implementing further
>     enhancements of VPP and SPDK integration feel free to reply, or to use one
>     of the many SPDK community communications
>     channels<https://spdk.io/community/>;.
>     
>     Thanks,
>     Tomek
>     
>     _______________________________________________
>     SPDK mailing list -- spdk(a)lists.01.org
>     To unsubscribe send an email to spdk-leave(a)lists.01.org
>     _______________________________________________
>     SPDK mailing list -- spdk(a)lists.01.org
>     To unsubscribe send an email to spdk-leave(a)lists.01.org
>     
> 
> _______________________________________________
> SPDK mailing list -- spdk(a)lists.01.org To unsubscribe send an email to 
> spdk-leave(a)lists.01.org

_______________________________________________
SPDK mailing list -- spdk(a)lists.01.org
To unsubscribe send an email to spdk-leave(a)lists.01.org