Re: IOVA allocation dependency between firmware buffer and remaining buffers

[Shaik Ameer Basha <shaik.samsung@gmail.com>]

Hi Robin and Marek,


On Thu, Sep 24, 2020 at 4:36 PM Robin Murphy <robin.murphy@arm.com> wrote:
>
> On 2020-09-24 11:47, Marek Szyprowski wrote:
> > Hi Robin,
> >
> > On 24.09.2020 12:40, Robin Murphy wrote:
> >> On 2020-09-24 11:16, Thierry Reding wrote:
> >>> On Thu, Sep 24, 2020 at 10:46:46AM +0200, Marek Szyprowski wrote:
> >>>> On 24.09.2020 10:28, Joerg Roedel wrote:
> >>>>> On Wed, Sep 23, 2020 at 08:48:26AM +0200, Marek Szyprowski wrote:
> >>>>>> It allows to remap given buffer at the specific IOVA address,
> >>>>>> although
> >>>>>> it doesn't guarantee that those specific addresses won't be later
> >>>>>> used
> >>>>>> by the IOVA allocator. Probably it would make sense to add an API for
> >>>>>> generic IOMMU-DMA framework to mark the given IOVA range as
> >>>>>> reserved/unused to protect them.
> >>>>> There is an API for that, the IOMMU driver can return IOVA reserved
> >>>>> regions per device and the IOMMU core code will take care of mapping
> >>>>> these regions and reserving them in the IOVA allocator, so that
> >>>>> DMA-IOMMU code will not use it for allocations.
> >>>>>
> >>>>> Have a look at the iommu_ops->get_resv_regions() and
> >>>>> iommu_ops->put_resv_regions().
> >>>>
> >>>> I know about the reserved regions IOMMU API, but the main problem here,
> >>>> in case of Exynos, is that those reserved regions won't be created by
> >>>> the IOMMU driver but by the IOMMU client device. It is just a result
> >>>> how
> >>>> the media drivers manages their IOVA space. They simply have to load
> >>>> firmware at the IOVA address lower than the any address of the used
> >>>> buffers.
> >>>
> >>> I've been working on adding a way to automatically add direct mappings
> >>> using reserved-memory regions parsed from device tree, see:
> >>>
> >>> https://lore.kernel.org/lkml/20200904130000.691933-1-thierry.reding@gmail.com/
> >>>
> >>> Perhaps this can be of use? With that you should be able to add a
> >>> reserved-memory region somewhere in the lower range that you need for
> >>> firmware images and have that automatically added as a direct mapping
> >>> so that it won't be reused later on for dynamic allocations.
> >>
> >> It can't easily be a *direct* mapping though - if the driver has to
> >> use the DMA masks to ensure that everything stays within the
> >> addressable range, then (as far as I'm aware) there's no physical
> >> memory that low down to equal the DMA addresses.
> >>
> >> TBH I'm not convinced that this is a sufficiently common concern to
> >> justify new APIs, or even to try to make overly generic. I think just
> >> implementing a new DMA attribute to say "please allocate/map this
> >> particular request at the lowest DMA address possible" would be good
> >> enough. Such a thing could also serve PCI drivers that actually care
> >> about SAC/DAC to give us more of a chance of removing the "try a
> >> 32-bit mask first" trick from everyone's hotpath...
> >
> > Hmm, I like the idea of such DMA attribute! It should make things really
> > simple, especially in the drivers. Thanks for the great idea! I will try
> > to implement it then instead of the workarounds I've proposed in
> > s5p-mfc/exynos4-is drivers.
>
> Right, I think it's fair to draw a line and say that anyone who wants a
> *specific* address needs to manage their own IOMMU domain.
>
> In the backend I suspect it's going to be cleanest to implement a
> dedicated iova_alloc_low() (or similar) function in the IOVA API that
> sidesteps all of the existing allocation paths and goes straight to the
> rbtree.

This is the place we started with..

But our solution was to provide an API which limits the allocation
range per device (dynamically) based on the driver request..
Something like,
    limit_iova_alloc_range(dev, low_iova, high_iova); /* via helpers */

When multiple devices use the same IOVA space, how we can handle api's
like " iova_alloc_low()" ?
And providing APIs like " limit_iova_alloc_range()" may cater similar
future requirements from drivers instead of worrying about
high/low/mid etc.

Again, flexibility should be there with user drivers to request the
range they want at any point of time...

Please let us know your inputs.

>
> Robin.