[SPDK] Re: How can SPDK process copy data to target process?

[SPDK] Re: How can SPDK process copy data to target process?

[Walker, Benjamin]

[-- Attachment #1: Type: text/plain, Size: 2580 bytes --]

> -----Original Message-----
> From: yaoxinjing517(a)gmail.com <yaoxinjing517(a)gmail.com>
> Sent: Wednesday, March 23, 2022 1:16 AM
> To: spdk(a)lists.01.org
> Subject: [SPDK] How can SPDK process copy data to target process?
> 
> Currently, I want to build an SPDK backend for our app, which has multiple
> processes created by fork.  To this end, we would like to create a process
> dedicated to SPDK for doing the IO work, for example, retrieving data from
> NVME and sending the data to other processes, or collect requests from
> other processes and sending data to NVME. However, SPDK doesn't support
> shared memory between native processes, so we intend to use the ring on
> shared memory (shm_open, etc.) for sending the request from other
> processes to the SPDK process.  The difficulty is how the process running
> SPDK can achieve the zero-copy, which means, instead of copying the data
> from NVME disk to the process running SPDK and then copying it to shared
> memory, then copy it to the target process, we want the process running
> SPDK copy the data from NVME directly to the target process. Do you have
> any idea how we can achieve such stuff?

For posterity, we discussed this on slack (https://spdk-team.slack.com/archives/CJE4C98G1/p1648024391999489)

Ben Walker 
Allocate the memory in the non-spdk process in one of the ways that associates the memory with a file descriptor. The standard process-shared memory stuff. Then send that file descriptor over a Unix domain socket to the SPDK process. SPDK can then mmap it, spdk_mem_register() and then you can do DMA to it directly

xin yao 
Thanks for your reply, But I am still not clear, do you mean that I need to do the following steps for the direct DMA  from SPDK process to non SPDK process?

xin yao 
1.In the non-spdk process: use shm_open to create a shared object and map it to the address space 
2. Pass the file descriptor we got from shm_open to spdk-process via socket
3. Use mmap to map the shared object referred by the file descriptor to spdk-process
4. use spdk_mem_register to turn the shared memory to DMA capable

Ben Walker
yep exactly - the shared memory does need to be either backed by huge pages OR you need to have your iommu enabled or spdk_mem_register will fail

Ben Walker
note that this is how virtio-blk/scsi and nvmf/vfio-user work today, so there are examples to go look at

> _______________________________________________
> SPDK mailing list -- spdk(a)lists.01.org
> To unsubscribe send an email to spdk-leave(a)lists.01.org