Re: [SPDK] NBD with SPDK

[Mittal, Rishabh <rimittal at ebay.com>]

[-- Attachment #1: Type: text/plain, Size: 2791 bytes --]

<<As I’m sure you’re aware, SPDK apps use spdk_alloc() with the SPDK_MALLOC_DMA which is backed by huge pages that are effectively pinned already. SPDK does virt to phy transition on memory allocated this <<way very efficiently using spdk_vtophys().  It would be an interesting experiment though.  Your app is not in a VM right?>>

I am thinking of passing the physical address of the buffers in bio to spdk.  I don’t know if it is already pinned by the kernel or do we need to explicitly do it. And also, spdk has some requirements on the alignment of physical address. I don’t know if address in bio conforms to those requirements.

SPDK won’t be running in VM.

From: "Luse, Paul E" <paul.e.luse(a)intel.com>
Date: Sunday, August 11, 2019 at 12:53 PM
To: "Mittal, Rishabh" <rimittal(a)ebay.com>, "spdk(a)lists.01.org" <spdk(a)lists.01.org>
Cc: "Kadayam, Hari" <hkadayam(a)ebay.com>, "Chen, Xiaoxi" <xiaoxchen(a)ebay.com>, "Szmyd, Brian" <bszmyd(a)ebay.com>
Subject: RE: NBD with SPDK

Hi Rishabh,

Thanks for the question. I was talking to Jim and Ben about this a bit, one of them may want to elaborate but we’re thinking the cost of mmap and also making sure the memory is pinned is probably prohibitive. As I’m sure you’re aware, SPDK apps use spdk_alloc() with the SPDK_MALLOC_DMA which is backed by huge pages that are effectively pinned already. SPDK does virt to phy transition on memory allocated this way very efficiently using spdk_vtophys().  It would be an interesting experiment though.  Your app is not in a VM right?

Thx
Paul

From: Mittal, Rishabh [mailto:rimittal(a)ebay.com]
Sent: Saturday, August 10, 2019 6:09 PM
To: spdk(a)lists.01.org
Cc: Luse, Paul E <paul.e.luse(a)intel.com>; Kadayam, Hari <hkadayam(a)ebay.com>; Chen, Xiaoxi <xiaoxchen(a)ebay.com>; Szmyd, Brian <bszmyd(a)ebay.com>
Subject: NBD with SPDK

Hi,

We are trying to use NBD and SPDK on client side.  Data path looks like this

File System ----> NBD client ------>SPDK------->NVMEoF


Currently we are seeing a high latency in the order of 50 us by using this path. It seems like there is data buffer copy happening for write commands from kernel to user space when spdk nbd read data from the nbd socket.

I think that there could be two ways to prevent data copy .


  1.  Memory mapped the kernel buffers to spdk virtual space.  I am not sure if it is possible to mmap a buffer. And what is the impact to call mmap for each IO.
  2.  If NBD kernel give the physical address of a buffer and SPDK use that to DMA it to NVMEoF. I think spdk must also be changing a virtual address to physical address before sending it to nvmeof.

Option 2 makes more sense to me. Please let me know if option 2 is feasible in spdk

Thanks
Rishabh Mittal