Where are iommu region setup in dma_alloc_from_dev_coherent() track?

Where are iommu region setup in dma_alloc_from_dev_coherent() track?

[Сергей Пыптев]

Hi!

I try to port our 3D scanner driver to kernel v4.9 and I have a trouble. 
I can not find a correct way to allocate DMA memory with correct iommu 
setup. Could you help me?

HW: Tegra X1 and Tegra X2 (arm64), our PCIe bus master device uses about 
200M of system memory to send data from sensors. The memory is organized 
in circular buffer (8 frames) and one info page (only 8 bytes are used now).
Old driver works at kernel v3.10 and some memory access rules are new in 
v4.9, such as iommu.

I try to use reserved memory region attached to the device in device 
tree and dma_alloc_coherent() to allocate the buffer. (Then I am going 
to find correct flags to turn off coherency and to sync data to CPU 
manually when full frame is arrived.)

Unfortunately, iommu does not set the area.

I've tried to find why. I see iommu calls in general dma memory 
allocation where arch-dependent dma_ops are used (and it sets iommu 
area, I can see it in /sys/kernel/debug), but I can not find the calls 
in per-device allocation track. So I do not see if my code is not 
working because of my bugs or I use wrong concept or it is bug in kernel.

Is there iommu setup in per-device reserved memory allocator (or may be 
earlier in memory reservation)? If no, why? Is it a feature or a bug?

While there are some significant changes in the code from v4.9 to v5.8, 
I have not found any related to iommu in the area, so I think it may be 
the same problem with latest kernel version.

Thanks,
Sergei Pyptev

_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: Where are iommu region setup in dma_alloc_from_dev_coherent() track?

[Robin Murphy]

On 2020-08-24 08:01, Сергей Пыптев wrote:
> Hi!
> 
> I try to port our 3D scanner driver to kernel v4.9 and I have a trouble. 
> I can not find a correct way to allocate DMA memory with correct iommu 
> setup. Could you help me?
> 
> HW: Tegra X1 and Tegra X2 (arm64), our PCIe bus master device uses about 
> 200M of system memory to send data from sensors. The memory is organized 
> in circular buffer (8 frames) and one info page (only 8 bytes are used 
> now).
> Old driver works at kernel v3.10 and some memory access rules are new in 
> v4.9, such as iommu.
> 
> I try to use reserved memory region attached to the device in device 
> tree and dma_alloc_coherent() to allocate the buffer. (Then I am going 
> to find correct flags to turn off coherency and to sync data to CPU 
> manually when full frame is arrived.)
> 
> Unfortunately, iommu does not set the area.
> 
> I've tried to find why. I see iommu calls in general dma memory 
> allocation where arch-dependent dma_ops are used (and it sets iommu 
> area, I can see it in /sys/kernel/debug), but I can not find the calls 
> in per-device allocation track. So I do not see if my code is not 
> working because of my bugs or I use wrong concept or it is bug in kernel.
> 
> Is there iommu setup in per-device reserved memory allocator (or may be 
> earlier in memory reservation)? If no, why? Is it a feature or a bug?

There is not. It's not a bug, just that there's no support for this 
combination because it doesn't really make sense. The expectation is 
that an IOMMU can address all system memory at page granularity, 
therefore there's no need for reserved regions because there are no 
constraints on the *physical* address space. The IOMMU-based DMA ops can 
allocate 200MB worth of pages from all over the place and still present 
them to the device (and driver) as one large contiguous buffer.

Robin.
_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu