[RFC PATCH] iommu/dma: Allow drivers to reserve an iova range

[RFC PATCH] iommu/dma: Allow drivers to reserve an iova range

[Isaac J. Manjarres]

From: Liam Mark <lmark@codeaurora.org>

Some devices have a memory map which contains gaps or holes.
In order for the device to have as much IOVA space as possible,
allow its driver to inform the DMA-IOMMU layer that it should
not allocate addresses from these holes.

Change-Id: I15bd1d313d889c2572d0eb2adecf6bebde3267f7
Signed-off-by: Isaac J. Manjarres <isaacm@codeaurora.org>
---
 drivers/iommu/dma-iommu.c | 28 ++++++++++++++++++++++++++++
 include/linux/dma-iommu.h |  9 +++++++++
 2 files changed, 37 insertions(+)

diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c
index a2e96a5..3b83e1a 100644
--- a/drivers/iommu/dma-iommu.c
+++ b/drivers/iommu/dma-iommu.c
@@ -368,6 +368,34 @@ static int iommu_dma_deferred_attach(struct device *dev,
 	return 0;
 }
 
+/*
+ * Should be called prior to using dma-apis
+ */
+int iommu_dma_reserve_iova(struct device *dev, dma_addr_t base,
+			   u64 size)
+{
+	struct iommu_domain *domain;
+	struct iommu_dma_cookie *cookie;
+	struct iova_domain *iovad;
+	unsigned long pfn_lo, pfn_hi;
+
+	domain = iommu_get_domain_for_dev(dev);
+	if (!domain || !domain->iova_cookie)
+		return -EINVAL;
+
+	cookie = domain->iova_cookie;
+	iovad = &cookie->iovad;
+
+	/* iova will be freed automatically by put_iova_domain() */
+	pfn_lo = iova_pfn(iovad, base);
+	pfn_hi = iova_pfn(iovad, base + size - 1);
+	if (!reserve_iova(iovad, pfn_lo, pfn_hi))
+		return -EINVAL;
+
+	return 0;
+}
+EXPORT_SYMBOL(iommu_dma_reserve_iova);
+
 /**
  * dma_info_to_prot - Translate DMA API directions and attributes to IOMMU API
  *                    page flags.
diff --git a/include/linux/dma-iommu.h b/include/linux/dma-iommu.h
index 2112f21..79eef7c 100644
--- a/include/linux/dma-iommu.h
+++ b/include/linux/dma-iommu.h
@@ -37,6 +37,9 @@ void iommu_dma_compose_msi_msg(struct msi_desc *desc,
 
 void iommu_dma_get_resv_regions(struct device *dev, struct list_head *list);
 
+int iommu_dma_reserve_iova(struct device *dev, dma_addr_t base,
+			   u64 size);
+
 #else /* CONFIG_IOMMU_DMA */
 
 struct iommu_domain;
@@ -78,5 +81,11 @@ static inline void iommu_dma_get_resv_regions(struct device *dev, struct list_he
 {
 }
 
+static inline int iommu_dma_reserve_iova(struct device *dev, dma_addr_t base,
+					 u64 size)
+{
+	return -ENODEV;
+}
+
 #endif	/* CONFIG_IOMMU_DMA */
 #endif	/* __DMA_IOMMU_H */
-- 
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
a Linux Foundation Collaborative Project

Re: [RFC PATCH] iommu/dma: Allow drivers to reserve an iova range

[Christoph Hellwig]

On Fri, Feb 14, 2020 at 02:58:16PM -0800, Isaac J. Manjarres wrote:
> From: Liam Mark <lmark@codeaurora.org>
> 
> Some devices have a memory map which contains gaps or holes.
> In order for the device to have as much IOVA space as possible,
> allow its driver to inform the DMA-IOMMU layer that it should
> not allocate addresses from these holes.

Layering violation.  dma-iommu is the translation layer between the
DMA API and the IOMMU API.  And calls into it from drivers performing
DMA mappings need to go through the DMA API (and be documented there).

Re: [RFC PATCH] iommu/dma: Allow drivers to reserve an iova range

[Robin Murphy]

On 17/02/2020 8:01 am, Christoph Hellwig wrote:
> On Fri, Feb 14, 2020 at 02:58:16PM -0800, Isaac J. Manjarres wrote:
>> From: Liam Mark <lmark@codeaurora.org>
>>
>> Some devices have a memory map which contains gaps or holes.
>> In order for the device to have as much IOVA space as possible,
>> allow its driver to inform the DMA-IOMMU layer that it should
>> not allocate addresses from these holes.
> 
> Layering violation.  dma-iommu is the translation layer between the
> DMA API and the IOMMU API.  And calls into it from drivers performing
> DMA mappings need to go through the DMA API (and be documented there).

+1

More than that, though, we already have "holes in the address space" 
support for the sake of PCI host bridge windows - assuming this is the 
same kind of thing (i.e. the holes are between memory regions and other 
resources in PA space, so are only relevant once address translation 
comes into the picture), then this is IOMMU API level stuff, so even a 
DMA API level interface would be inappropriate.

Robin.

Re: [RFC PATCH] iommu/dma: Allow drivers to reserve an iova range

[isaacm]

On 2020-02-17 07:50, Robin Murphy wrote:
> On 17/02/2020 8:01 am, Christoph Hellwig wrote:
>> On Fri, Feb 14, 2020 at 02:58:16PM -0800, Isaac J. Manjarres wrote:
>>> From: Liam Mark <lmark@codeaurora.org>
>>> 
>>> Some devices have a memory map which contains gaps or holes.
>>> In order for the device to have as much IOVA space as possible,
>>> allow its driver to inform the DMA-IOMMU layer that it should
>>> not allocate addresses from these holes.
>> 
>> Layering violation.  dma-iommu is the translation layer between the
>> DMA API and the IOMMU API.  And calls into it from drivers performing
>> DMA mappings need to go through the DMA API (and be documented there).
> 
> +1
> 
> More than that, though, we already have "holes in the address space"
> support for the sake of PCI host bridge windows - assuming this is the
> same kind of thing (i.e. the holes are between memory regions and
> other resources in PA space, so are only relevant once address
> translation comes into the picture), then this is IOMMU API level
To make sure that we're on the same page, this support alludes to the 
handling in
dma-iommu.c that reserves portions of the IOVA space for the PCI host 
bridge windows,
correct? If so, then yes, this is similar.
> stuff, so even a DMA API level interface would be inappropriate.
Does this mean that the driver should be managing the IOVA space and 
mappings for this device using the IOMMU API? If so, is the rationale 
for this because the device driver can have the information of what IOVA 
ranges can and cannot be used? Shouldn't there be a generic way of 
informing an IOMMU driver about these reserved ranges? Perhaps through a 
device tree property, instead of deferring this type of management to 
the driver?
> 
> Robin.

Re: [RFC PATCH] iommu/dma: Allow drivers to reserve an iova range

[Joerg Roedel]

On Fri, Feb 14, 2020 at 02:58:16PM -0800, Isaac J. Manjarres wrote:
> From: Liam Mark <lmark@codeaurora.org>
> 
> Some devices have a memory map which contains gaps or holes.
> In order for the device to have as much IOVA space as possible,
> allow its driver to inform the DMA-IOMMU layer that it should
> not allocate addresses from these holes.
> 
> Change-Id: I15bd1d313d889c2572d0eb2adecf6bebde3267f7
> Signed-off-by: Isaac J. Manjarres <isaacm@codeaurora.org>

Ideally this is something put into the IOMMU firmware table by the
platform firmware. If its not there, a quirk is the best way to handle
this.

Regards,

	Joerg

Re: [RFC PATCH] iommu/dma: Allow drivers to reserve an iova range

[Will Deacon]

On Tue, Feb 18, 2020 at 05:57:18PM -0800, isaacm@codeaurora.org wrote:
> On 2020-02-17 07:50, Robin Murphy wrote:
> > On 17/02/2020 8:01 am, Christoph Hellwig wrote:
> > > On Fri, Feb 14, 2020 at 02:58:16PM -0800, Isaac J. Manjarres wrote:
> > > > From: Liam Mark <lmark@codeaurora.org>
> > > > 
> > > > Some devices have a memory map which contains gaps or holes.
> > > > In order for the device to have as much IOVA space as possible,
> > > > allow its driver to inform the DMA-IOMMU layer that it should
> > > > not allocate addresses from these holes.
> > > 
> > > Layering violation.  dma-iommu is the translation layer between the
> > > DMA API and the IOMMU API.  And calls into it from drivers performing
> > > DMA mappings need to go through the DMA API (and be documented there).
> > 
> > +1
> > 
> > More than that, though, we already have "holes in the address space"
> > support for the sake of PCI host bridge windows - assuming this is the
> > same kind of thing (i.e. the holes are between memory regions and
> > other resources in PA space, so are only relevant once address
> > translation comes into the picture), then this is IOMMU API level
> To make sure that we're on the same page, this support alludes to the
> handling in
> dma-iommu.c that reserves portions of the IOVA space for the PCI host bridge
> windows,
> correct? If so, then yes, this is similar.
> > stuff, so even a DMA API level interface would be inappropriate.
> Does this mean that the driver should be managing the IOVA space and
> mappings for this device using the IOMMU API? If so, is the rationale for
> this because the device driver can have the information of what IOVA ranges
> can and cannot be used? Shouldn't there be a generic way of informing an
> IOMMU driver about these reserved ranges? Perhaps through a device tree
> property, instead of deferring this type of management to the driver?

Before we dive into designing that, can you please clarify whether the
reserved IOVA range applies to all DMA masters mastering through a
particular SMMU, or whether it's just about one specific master? I was
assuming the former, but wanted to be sure.

Thanks,

Will

Re: [RFC PATCH] iommu/dma: Allow drivers to reserve an iova range

[isaacm]

On 2020-02-19 03:15, Will Deacon wrote:
> On Tue, Feb 18, 2020 at 05:57:18PM -0800, isaacm@codeaurora.org wrote:
>> On 2020-02-17 07:50, Robin Murphy wrote:
>> > On 17/02/2020 8:01 am, Christoph Hellwig wrote:
>> > > On Fri, Feb 14, 2020 at 02:58:16PM -0800, Isaac J. Manjarres wrote:
>> > > > From: Liam Mark <lmark@codeaurora.org>
>> > > >
>> > > > Some devices have a memory map which contains gaps or holes.
>> > > > In order for the device to have as much IOVA space as possible,
>> > > > allow its driver to inform the DMA-IOMMU layer that it should
>> > > > not allocate addresses from these holes.
>> > >
>> > > Layering violation.  dma-iommu is the translation layer between the
>> > > DMA API and the IOMMU API.  And calls into it from drivers performing
>> > > DMA mappings need to go through the DMA API (and be documented there).
>> >
>> > +1
>> >
>> > More than that, though, we already have "holes in the address space"
>> > support for the sake of PCI host bridge windows - assuming this is the
>> > same kind of thing (i.e. the holes are between memory regions and
>> > other resources in PA space, so are only relevant once address
>> > translation comes into the picture), then this is IOMMU API level
>> To make sure that we're on the same page, this support alludes to the
>> handling in
>> dma-iommu.c that reserves portions of the IOVA space for the PCI host 
>> bridge
>> windows,
>> correct? If so, then yes, this is similar.
>> > stuff, so even a DMA API level interface would be inappropriate.
>> Does this mean that the driver should be managing the IOVA space and
>> mappings for this device using the IOMMU API? If so, is the rationale 
>> for
>> this because the device driver can have the information of what IOVA 
>> ranges
>> can and cannot be used? Shouldn't there be a generic way of informing 
>> an
>> IOMMU driver about these reserved ranges? Perhaps through a device 
>> tree
>> property, instead of deferring this type of management to the driver?
> 
> Before we dive into designing that, can you please clarify whether the
> reserved IOVA range applies to all DMA masters mastering through a
> particular SMMU, or whether it's just about one specific master? I was
> assuming the former, but wanted to be sure.
> 
This situation currently applies to one master.
> Thanks,
> 
> Will

Thanks,
Isaac

Re: [RFC PATCH] iommu/dma: Allow drivers to reserve an iova range

[Will Deacon]

On Wed, Feb 19, 2020 at 12:06:28PM -0800, isaacm@codeaurora.org wrote:
> On 2020-02-19 03:15, Will Deacon wrote:
> > On Tue, Feb 18, 2020 at 05:57:18PM -0800, isaacm@codeaurora.org wrote:
> > > Does this mean that the driver should be managing the IOVA space and
> > > mappings for this device using the IOMMU API? If so, is the
> > > rationale for
> > > this because the device driver can have the information of what IOVA
> > > ranges
> > > can and cannot be used? Shouldn't there be a generic way of
> > > informing an
> > > IOMMU driver about these reserved ranges? Perhaps through a device
> > > tree
> > > property, instead of deferring this type of management to the driver?
> > 
> > Before we dive into designing that, can you please clarify whether the
> > reserved IOVA range applies to all DMA masters mastering through a
> > particular SMMU, or whether it's just about one specific master? I was
> > assuming the former, but wanted to be sure.
> > 
> This situation currently applies to one master.

Interesting. Is it problematic if the range is reserved for all masters
sharing that SMMU?

Will

Re: [RFC PATCH] iommu/dma: Allow drivers to reserve an iova range

[Pratik Patel]

On 02/20  08:45 am, Will Deacon wrote:
> On Wed, Feb 19, 2020 at 12:06:28PM -0800, isaacm@codeaurora.org wrote:
> > On 2020-02-19 03:15, Will Deacon wrote:
> > > On Tue, Feb 18, 2020 at 05:57:18PM -0800, isaacm@codeaurora.org wrote:
> > > > Does this mean that the driver should be managing the IOVA space and
> > > > mappings for this device using the IOMMU API? If so, is the
> > > > rationale for
> > > > this because the device driver can have the information of what IOVA
> > > > ranges
> > > > can and cannot be used? Shouldn't there be a generic way of
> > > > informing an
> > > > IOMMU driver about these reserved ranges? Perhaps through a device
> > > > tree
> > > > property, instead of deferring this type of management to the driver?
> > > 
> > > Before we dive into designing that, can you please clarify whether the
> > > reserved IOVA range applies to all DMA masters mastering through a
> > > particular SMMU, or whether it's just about one specific master? I was
> > > assuming the former, but wanted to be sure.
> > > 
> > This situation currently applies to one master.
> 
> Interesting. Is it problematic if the range is reserved for all masters
> sharing that SMMU?
> 
Yes, that would be an overkill. It certainly is useful and in
some cases quite helpful to not waste that range of IOVA space for
other masters on the same SMMU that can use it.

-- 
The Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
a Linux Foundation Collaborative Project