From mboxrd@z Thu Jan  1 00:00:00 1970
Content-Type: multipart/mixed; boundary="===============0566649879347951140=="
MIME-Version: 1.0
From: Mittal, Rishabh <rimittal at ebay.com>
Subject: Re: [SPDK] NBD with SPDK
Date: Sun, 11 Aug 2019 23:33:02 +0000
Message-ID: <2042D875-9E3C-4375-9F38-58D3731BEF56@ebay.com>
In-Reply-To: 99DFD8AF-0CEB-4058-AB41-EE823A284D31@ebay.com
List-ID: <spdk@lists.01.org>
To: spdk@lists.01.org

--===============0566649879347951140==
Content-Type: text/plain; charset="utf-8"
MIME-Version: 1.0
Content-Transfer-Encoding: quoted-printable

By looking at the structure it seems that it is already pinned. Now only qu=
estion remains that what is the behavior if it doesn=E2=80=99t conform to t=
he alignment requirements of NVME ?


struct bio_vec {
        /* pointer to the physical page on which this buffer resides */
        struct page     *bv_page;

        /* the length in bytes of this buffer */
        unsigned int    bv_len;

        /* the byte offset within the page where the buffer resides */
        unsigned int    bv_offset;
};


From: "Mittal, Rishabh" <rimittal(a)ebay.com>
Date: Sunday, August 11, 2019 at 3:51 PM
To: "Luse, Paul E" <paul.e.luse(a)intel.com>, "spdk(a)lists.01.org" <spdk(a=
)lists.01.org>
Cc: "Kadayam, Hari" <hkadayam(a)ebay.com>, "Chen, Xiaoxi" <xiaoxchen(a)ebay=
.com>, "Szmyd, Brian" <bszmyd(a)ebay.com>
Subject: Re: NBD with SPDK

<<As I=E2=80=99m sure you=E2=80=99re aware, SPDK apps use spdk_alloc() with=
 the SPDK_MALLOC_DMA which is backed by huge pages that are effectively pin=
ned already. SPDK does virt to phy transition on memory allocated this <<wa=
y very efficiently using spdk_vtophys().  It would be an interesting experi=
ment though.  Your app is not in a VM right?>>

I am thinking of passing the physical address of the buffers in bio to spdk=
.  I don=E2=80=99t know if it is already pinned by the kernel or do we need=
 to explicitly do it. And also, spdk has some requirements on the alignment=
 of physical address. I don=E2=80=99t know if address in bio conforms to th=
ose requirements.

SPDK won=E2=80=99t be running in VM.

From: "Luse, Paul E" <paul.e.luse(a)intel.com>
Date: Sunday, August 11, 2019 at 12:53 PM
To: "Mittal, Rishabh" <rimittal(a)ebay.com>, "spdk(a)lists.01.org" <spdk(a)=
lists.01.org>
Cc: "Kadayam, Hari" <hkadayam(a)ebay.com>, "Chen, Xiaoxi" <xiaoxchen(a)ebay=
.com>, "Szmyd, Brian" <bszmyd(a)ebay.com>
Subject: RE: NBD with SPDK

Hi Rishabh,

Thanks for the question. I was talking to Jim and Ben about this a bit, one=
 of them may want to elaborate but we=E2=80=99re thinking the cost of mmap =
and also making sure the memory is pinned is probably prohibitive. As I=E2=
=80=99m sure you=E2=80=99re aware, SPDK apps use spdk_alloc() with the SPDK=
_MALLOC_DMA which is backed by huge pages that are effectively pinned alrea=
dy. SPDK does virt to phy transition on memory allocated this way very effi=
ciently using spdk_vtophys().  It would be an interesting experiment though=
.  Your app is not in a VM right?

Thx
Paul

From: Mittal, Rishabh [mailto:rimittal(a)ebay.com]
Sent: Saturday, August 10, 2019 6:09 PM
To: spdk(a)lists.01.org
Cc: Luse, Paul E <paul.e.luse(a)intel.com>; Kadayam, Hari <hkadayam(a)ebay.=
com>; Chen, Xiaoxi <xiaoxchen(a)ebay.com>; Szmyd, Brian <bszmyd(a)ebay.com>
Subject: NBD with SPDK

Hi,

We are trying to use NBD and SPDK on client side.  Data path looks like this

File System ----> NBD client ------>SPDK------->NVMEoF


Currently we are seeing a high latency in the order of 50 us by using this =
path. It seems like there is data buffer copy happening for write commands =
from kernel to user space when spdk nbd read data from the nbd socket.

I think that there could be two ways to prevent data copy .


  1.  Memory mapped the kernel buffers to spdk virtual space.  I am not sur=
e if it is possible to mmap a buffer. And what is the impact to call mmap f=
or each IO.
  2.  If NBD kernel give the physical address of a buffer and SPDK use that=
 to DMA it to NVMEoF. I think spdk must also be changing a virtual address =
to physical address before sending it to nvmeof.

Option 2 makes more sense to me. Please let me know if option 2 is feasible=
 in spdk

Thanks
Rishabh Mittal


--===============0566649879347951140==--