[SPDK] memory region requirement for spdk_mem_register

[SPDK] memory region requirement for spdk_mem_register

[Feng Li]

[-- Attachment #1: Type: text/plain, Size: 1632 bytes --]

Hi,

I was trying to share DMA memory between two processes with SPDK in this 
way but failed:
* Process A creates a memory region ( 2M aligned pinned memory from 
anonymous mmap with MAP_HUGETLB)
* Process B then registers the above memory region  from A, so that SPDK 
can use it as DMA memory for nvme driver.

My current approach is (if there are better ways please let me know~):
1. Use a kernel module to store the starting physical address of this 
region shared from process A.
2. process B then also uses the kernel module to map this range of 
physical region to its virtual space (phys_addr->virt_addr_2) (using 
remap_pfn_range)
3. process B uses spdk_mem_register to register virt_addr_2.

My questions are:
i. The remap_pfn_range in step 2 only gives me 4K mappings(but they are 
actually backed by hugepages allocated in process A). Does this violate 
"The memory region must map to pinned huge pages (2MB or greater" 
requirement in spdk_mem_register? (since the mappings are in 4K). This 
thread(https://lists.01.org/pipermail/spdk/2018-July/002143.html) might 
be related.

ii. Currently the registration failed with -14 in process B. I traced 
into spdk and found out that the VFIO_IOMMU_MAP_DMA ioctrol failed in 
vtophys_iommu_map_dma (note that I had vfio enabled). Am i missing some 
steps (like in 
https://lists.01.org/pipermail/spdk/2018-May/001884.html)? Or the 
failure might just be caused by the 4k mapping i had?

iii. if memory from process A is mmaped from a dax device, are there any 
pitfalls i need to be aware of(in the presence of vfio)?

Thanks!

Feng

Re: [SPDK] memory region requirement for spdk_mem_register

[Feng Li]

[-- Attachment #1: Type: text/plain, Size: 3892 bytes --]

Hi Benjamin thanks for your reply, I also answered inline below.

On 7/1/19 4:51 AM, Walker, Benjamin wrote:
>
>> -----Original Message-----
>> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Feng Li
>> Sent: Thursday, June 27, 2019 10:07 PM
>> To: spdk(a)lists.01.org
>> Subject: [SPDK] memory region requirement for spdk_mem_register
>>
>> Hi,
>>
>> I was trying to share DMA memory between two processes with SPDK in this way
>> but failed:
>> * Process A creates a memory region ( 2M aligned pinned memory from
>> anonymous mmap with MAP_HUGETLB)
>> * Process B then registers the above memory region  from A, so that SPDK can
>> use it as DMA memory for nvme driver.
>>
>> My current approach is (if there are better ways please let me know~):
>> 1. Use a kernel module to store the starting physical address of this region
>> shared from process A.
>> 2. process B then also uses the kernel module to map this range of physical
>> region to its virtual space (phys_addr->virt_addr_2) (using
>> remap_pfn_range)
>> 3. process B uses spdk_mem_register to register virt_addr_2.
> Why not just create files on a hugetlbfs filesystem? That's what DPDK does, and then any process can mmap them. You don't need your own kernel module then.

That can be a good idea. I did see something similar in systemV IPC 
implementation with hugepage support in the kernel tree 
(https://elixir.bootlin.com/linux/v4.15.18/source/ipc/shm.c), where they 
share the file descriptor across the processes to share the hugepage 
segment.

But in the case of dax device mapped memory( NOT mapped from 
hugetlbfs),  i am not sure how to share it through hugetlbfs though..

Perhaps i need sth like hugetlbfs_unlinked_fd(), and I  will also look 
into how libpmem did this.

>
>> My questions are:
>> i. The remap_pfn_range in step 2 only gives me 4K mappings(but they are
>> actually backed by hugepages allocated in process A). Does this violate "The
>> memory region must map to pinned huge pages (2MB or greater"
>> requirement in spdk_mem_register? (since the mappings are in 4K). This
>> thread(https://lists.01.org/pipermail/spdk/2018-July/002143.html) might be
>> related.
> SPDK's memory maps only go down to 2MB granularity today, mostly for space reasons. But if you have 2MB chunks registered in 4K space, just register a 2MB aligned chunk and it will be fine.
I see
>
>> ii. Currently the registration failed with -14 in process B. I traced into spdk and
>> found out that the VFIO_IOMMU_MAP_DMA ioctrol failed in
>> vtophys_iommu_map_dma (note that I had vfio enabled). Am i missing some
>> steps (like in https://lists.01.org/pipermail/spdk/2018-May/001884.html)? Or the
>> failure might just be caused by the 4k mapping i had?
> I'm not familiar enough with the ways in which VFIO_IOMMU_MAP_DMA can fail - you'd have to dig into that driver I think to find what specifically -14 means.
it lead me to the 
vfio_pin_pages_remote(https://elixir.bootlin.com/linux/v4.15.18/source/drivers/vfio/vfio_iommu_type1.c#L948). 
it seems like it's trying to pin a page already pinned from another 
process. I will update once I can get more details.
>
>> iii. if memory from process A is mmaped from a dax device, are there any pitfalls
>> i need to be aware of(in the presence of vfio)?
> You can use memory allocated on a DAX device in SPDK - that definitely works. I recommend you use libpmem to manage the persistent memory. That's how we do it in SPDK when we use persistent memory.
Sure, good suggestion!
>
>> Thanks!
>>
>> Feng
>>
>> _______________________________________________
>> SPDK mailing list
>> SPDK(a)lists.01.org
>> https://lists.01.org/mailman/listinfo/spdk
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk

Re: [SPDK] memory region requirement for spdk_mem_register

[Walker, Benjamin]

[-- Attachment #1: Type: text/plain, Size: 2798 bytes --]



> -----Original Message-----
> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Feng Li
> Sent: Thursday, June 27, 2019 10:07 PM
> To: spdk(a)lists.01.org
> Subject: [SPDK] memory region requirement for spdk_mem_register
> 
> Hi,
> 
> I was trying to share DMA memory between two processes with SPDK in this way
> but failed:
> * Process A creates a memory region ( 2M aligned pinned memory from
> anonymous mmap with MAP_HUGETLB)
> * Process B then registers the above memory region  from A, so that SPDK can
> use it as DMA memory for nvme driver.
> 
> My current approach is (if there are better ways please let me know~):
> 1. Use a kernel module to store the starting physical address of this region
> shared from process A.
> 2. process B then also uses the kernel module to map this range of physical
> region to its virtual space (phys_addr->virt_addr_2) (using
> remap_pfn_range)
> 3. process B uses spdk_mem_register to register virt_addr_2.

Why not just create files on a hugetlbfs filesystem? That's what DPDK does, and then any process can mmap them. You don't need your own kernel module then.

> 
> My questions are:
> i. The remap_pfn_range in step 2 only gives me 4K mappings(but they are
> actually backed by hugepages allocated in process A). Does this violate "The
> memory region must map to pinned huge pages (2MB or greater"
> requirement in spdk_mem_register? (since the mappings are in 4K). This
> thread(https://lists.01.org/pipermail/spdk/2018-July/002143.html) might be
> related.

SPDK's memory maps only go down to 2MB granularity today, mostly for space reasons. But if you have 2MB chunks registered in 4K space, just register a 2MB aligned chunk and it will be fine.

> 
> ii. Currently the registration failed with -14 in process B. I traced into spdk and
> found out that the VFIO_IOMMU_MAP_DMA ioctrol failed in
> vtophys_iommu_map_dma (note that I had vfio enabled). Am i missing some
> steps (like in https://lists.01.org/pipermail/spdk/2018-May/001884.html)? Or the
> failure might just be caused by the 4k mapping i had?

I'm not familiar enough with the ways in which VFIO_IOMMU_MAP_DMA can fail - you'd have to dig into that driver I think to find what specifically -14 means.

> 
> iii. if memory from process A is mmaped from a dax device, are there any pitfalls
> i need to be aware of(in the presence of vfio)?

You can use memory allocated on a DAX device in SPDK - that definitely works. I recommend you use libpmem to manage the persistent memory. That's how we do it in SPDK when we use persistent memory.

> 
> Thanks!
> 
> Feng
> 
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk