[PATCH v4] devicetree: Add generic IOMMU device tree bindings

[PATCH v4] devicetree: Add generic IOMMU device tree bindings

[Thierry Reding]

From: Thierry Reding <treding@nvidia.com>

This commit introduces a generic device tree binding for IOMMU devices.
Only a very minimal subset is described here, but it is enough to cover
the requirements of both the Exynos System MMU and Tegra SMMU as
discussed here:

    https://lkml.org/lkml/2014/4/27/346

Signed-off-by: Thierry Reding <treding@nvidia.com>
---
Changes in v4:
- clarify that disabling an IOMMU DT node may not disable translation
- be more explicit that examples are only examples
- add multi-ID master example

Changes in v3:
- use #iommu-cells instead of #address-cells/#size-cells
- drop optional iommu-names property

Changes in v2:
- add notes about "dma-ranges" property (drop note from commit message)
- document priorities of "iommus" property vs. "dma-ranges" property
- drop #iommu-cells in favour of #address-cells and #size-cells
- remove multiple-master device example

 Documentation/devicetree/bindings/iommu/iommu.txt | 172 ++++++++++++++++++++++
 1 file changed, 172 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/iommu/iommu.txt

diff --git a/Documentation/devicetree/bindings/iommu/iommu.txt b/Documentation/devicetree/bindings/iommu/iommu.txt
new file mode 100644
index 000000000000..464a81eaaf61
--- /dev/null
+++ b/Documentation/devicetree/bindings/iommu/iommu.txt
@@ -0,0 +1,172 @@
+This document describes the generic device tree binding for IOMMUs and their
+master(s).
+
+
+IOMMU device node:
+==================
+
+An IOMMU can provide the following services:
+
+* Remap address space to allow devices to access physical memory ranges that
+  they otherwise wouldn't be capable of accessing.
+
+  Example: 32-bit DMA to 64-bit physical addresses
+
+* Implement scatter-gather at page level granularity so that the device does
+  not have to.
+
+* Provide system protection against "rogue" DMA by forcing all accesses to go
+  through the IOMMU and faulting when encountering accesses to unmapped
+  address regions.
+
+* Provide address space isolation between multiple contexts.
+
+  Example: Virtualization
+
+Device nodes compatible with this binding represent hardware with some of the
+above capabilities.
+
+IOMMUs can be single-master or multiple-master. Single-master IOMMU devices
+typically have a fixed association to the master device, whereas multiple-
+master IOMMU devices can translate accesses from more than one master.
+
+The device tree node of the IOMMU device's parent bus must contain a valid
+"dma-ranges" property that describes how the physical address space of the
+IOMMU maps to memory. An empty "dma-ranges" property means that there is a
+1:1 mapping from IOMMU to memory.
+
+Required properties:
+--------------------
+- #iommu-cells: The number of cells in an IOMMU specifier needed to encode an
+  address.
+
+The meaning of the IOMMU specifier is defined by the device tree binding of
+the specific IOMMU. Below are a few examples of typical use-cases:
+
+- #iommu-cells = <0>: Single master IOMMU devices are not configurable and
+  therefore no additional information needs to be encoded in the specifier.
+  This may also apply to multiple master IOMMU devices that do not allow the
+  association of masters to be configured. Note that an IOMMU can by design
+  be multi-master yet only expose a single master in a given configuration.
+  In such cases the number of cells will usually be 1 as in the next case.
+- #iommu-cells = <1>: Multiple master IOMMU devices may need to be configured
+  in order to enable translation for a given master. In such cases the single
+  address cell corresponds to the master device's ID. In some cases more than
+  one cell can be required to represent a single master ID.
+- #iommu-cells = <4>: Some IOMMU devices allow the DMA window for masters to
+  be configured. The first cell of the address in this may contain the master
+  device's ID for example, while the second cell could contain the start of
+  the DMA window for the given device. The length of the DMA window is given
+  by the third and fourth cells.
+
+Note that these are merely examples and real-world use-cases may use different
+definitions to represent their individual needs. Always refer to the specific
+IOMMU binding for the exact meaning of the cells that make up the specifier.
+
+
+IOMMU master node:
+==================
+
+Devices that access memory through an IOMMU are called masters. A device can
+have multiple master interfaces (to one or more IOMMU devices).
+
+Required properties:
+--------------------
+- iommus: A list of phandle and IOMMU specifier pairs that describe the IOMMU
+  master interfaces of the device. One entry in the list describes one master
+  interface of the device.
+
+When an "iommus" property is specified in a device tree node, the IOMMU will
+be used for address translation. If a "dma-ranges" property exists in the
+device's parent node it will be ignored. An exception to this rule is if the
+referenced IOMMU is disabled, in which case the "dma-ranges" property of the
+parent shall take effect. Note that merely disabling a device tree node does
+not guarantee that the IOMMU is really disabled since the hardware may not
+have a means to turn off translation.
+
+
+Notes:
+======
+
+One possible extension to the above is to use an "iommus" property along with
+a "dma-ranges" property in a bus device node (such as PCI host bridges). This
+can be useful to describe how children on the bus relate to the IOMMU if they
+are not explicitly listed in the device tree (e.g. PCI devices). However, the
+requirements of that use-case haven't been fully determined yet. Implementing
+this is therefore not recommended without further discussion and extension of
+this binding.
+
+
+Examples:
+=========
+
+Single-master IOMMU:
+--------------------
+
+	iommu {
+		#iommu-cells = <0>;
+	};
+
+	master {
+		iommus = <&/iommu>;
+	};
+
+Multiple-master IOMMU with fixed associations:
+----------------------------------------------
+
+	/* multiple-master IOMMU */
+	iommu {
+		/*
+		 * Masters are statically associated with this IOMMU and
+		 * address translation is always enabled.
+		 */
+		#iommu-cells = <0>;
+	};
+
+	/* static association with IOMMU */
+	master@1 {
+		reg = <1>;
+		iommus = <&/iommu>;
+	};
+
+	/* static association with IOMMU */
+	master@2 {
+		reg = <2>;
+		iommus = <&/iommu>;
+	};
+
+Multiple-master IOMMU:
+----------------------
+
+	iommu {
+		/* the specifier represents the ID of the master */
+		#iommu-cells = <1>;
+	};
+
+	master@1 {
+		/* device has master ID 42 in the IOMMU */
+		iommus = <&/iommu 42>;
+	};
+
+	master@2 {
+		/* device has master IDs 23 and 24 in the IOMMU */
+		iommus = <&/iommu 23>, <&/iommu 24>;
+	};
+
+Multiple-master IOMMU with configurable DMA window:
+---------------------------------------------------
+
+	/ {
+		#address-cells = <1>;
+		#size-cells = <1>;
+
+		iommu {
+			/* master ID, address and length of DMA window */
+			#iommu-cells = <4>;
+		};
+
+		master {
+			/* master ID 42, 4 GiB DMA window starting at 0 */
+			iommus = <&/iommu  42  0  0x1 0x0>;
+		};
+	};
-- 
2.0.1

Re: [PATCH v4] devicetree: Add generic IOMMU device tree bindings

[Will Deacon]

On Fri, Jul 04, 2014 at 04:29:17PM +0100, Thierry Reding wrote:
> From: Thierry Reding <treding@nvidia.com>
> 
> This commit introduces a generic device tree binding for IOMMU devices.
> Only a very minimal subset is described here, but it is enough to cover
> the requirements of both the Exynos System MMU and Tegra SMMU as
> discussed here:
> 
>     https://lkml.org/lkml/2014/4/27/346
> 
> Signed-off-by: Thierry Reding <treding@nvidia.com>

  Acked-by: Will Deacon <will.deacon@arm.com>

I would like to move the ARM SMMU driver over to this for 3.18, if possible.
One use-case there is the ability to describe groups of masters behind a
multi-master IOMMU but which must be part of the same domain (i.e. an
iommu_group). This is useful for presenting devices to a guest with a
virtual SMMU, where the physical devices share a stage-2 context.

With your binding, does this simply mean determining the set of master IDs
in the group, then describing the complete set for each master?

Will

Re: [PATCH v4] devicetree: Add generic IOMMU device tree bindings

[Thierry Reding]

[-- Attachment #1: Type: text/plain, Size: 2149 bytes --]

On Wed, Jul 09, 2014 at 02:40:50PM +0100, Will Deacon wrote:
> On Fri, Jul 04, 2014 at 04:29:17PM +0100, Thierry Reding wrote:
> > From: Thierry Reding <treding@nvidia.com>
> > 
> > This commit introduces a generic device tree binding for IOMMU devices.
> > Only a very minimal subset is described here, but it is enough to cover
> > the requirements of both the Exynos System MMU and Tegra SMMU as
> > discussed here:
> > 
> >     https://lkml.org/lkml/2014/4/27/346
> > 
> > Signed-off-by: Thierry Reding <treding@nvidia.com>
> 
>   Acked-by: Will Deacon <will.deacon@arm.com>
> 
> I would like to move the ARM SMMU driver over to this for 3.18, if possible.
> One use-case there is the ability to describe groups of masters behind a
> multi-master IOMMU but which must be part of the same domain (i.e. an
> iommu_group). This is useful for presenting devices to a guest with a
> virtual SMMU, where the physical devices share a stage-2 context.
> 
> With your binding, does this simply mean determining the set of master IDs
> in the group, then describing the complete set for each master?

I'm not sure I properly understand what you're trying to do, but I don't
think the binding is designed to cover that. Rather the goal was to
describe the IDs belonging to each master, so that an IOMMU can be
properly configured.

Anything beyond that (e.g. logical grouping of masters) isn't directly
within the scope of the binding (it doesn't describe hardware but some
policy pertaining to some specific use-case).

That said, the IOMMU driver that I prototyped for Tegra did some similar
grouping of devices, although in a much more restricted way. The goal of
that was to add a known set of devices into one group, "peripherals", so
that they could share one IOMMU domain. This was meant to separate them
from devices with more advanced needs (such as a GPU driver). Devices in
the "peripherals" group would be using the DMA mapping API integration
whereas other devices would have to explicitly allocate an IOMMU domain.

I'm not sure how much that helps for the use-case that you have in mind.

Thierry

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: [PATCH v4] devicetree: Add generic IOMMU device tree bindings

[Will Deacon]

Hi Thierry,

On Wed, Jul 09, 2014 at 03:21:27PM +0100, Thierry Reding wrote:
> On Wed, Jul 09, 2014 at 02:40:50PM +0100, Will Deacon wrote:
> > I would like to move the ARM SMMU driver over to this for 3.18, if possible.
> > One use-case there is the ability to describe groups of masters behind a
> > multi-master IOMMU but which must be part of the same domain (i.e. an
> > iommu_group). This is useful for presenting devices to a guest with a
> > virtual SMMU, where the physical devices share a stage-2 context.
> > 
> > With your binding, does this simply mean determining the set of master IDs
> > in the group, then describing the complete set for each master?
> 
> I'm not sure I properly understand what you're trying to do, but I don't
> think the binding is designed to cover that. Rather the goal was to
> describe the IDs belonging to each master, so that an IOMMU can be
> properly configured.

This is directly related to that problem, see below.

> Anything beyond that (e.g. logical grouping of masters) isn't directly
> within the scope of the binding (it doesn't describe hardware but some
> policy pertaining to some specific use-case).

This *is* for hardware. I can use PCI as an example, but this could equally
apply to other types of bus. If you have a bunch of PCI master devices
sitting being a non-transparent bridge, they can end up sharing the same
master device ID (requester ID). This means that there is no way in the
IOMMU to initialise a translation for one of these devices without also
affecting the others. We currently have iommu_groups to deal with this, but
it *is* a property of the hardware and we absolutely need a way to describe
it. I'm happy to add it later, but we need to think about it now to avoid
merging something that can't easily be extended.

For PCI, the topology is probable but even then, we need this information to
describe the resulting master device ID emitted by the bridge for the
upstream group. One way to do this with your binding would be to treat all
of the upstream masters as having the same device ID.

With virtualisation, we may want to assign a group of devices to a guest but
without emulating the bridge. This would need something the device-tree to
describe that they are grouped together.

> That said, the IOMMU driver that I prototyped for Tegra did some similar
> grouping of devices, although in a much more restricted way. The goal of
> that was to add a known set of devices into one group, "peripherals", so
> that they could share one IOMMU domain. This was meant to separate them
> from devices with more advanced needs (such as a GPU driver). Devices in
> the "peripherals" group would be using the DMA mapping API integration
> whereas other devices would have to explicitly allocate an IOMMU domain.
> 
> I'm not sure how much that helps for the use-case that you have in mind.

That sounds more like a software decision, which I agree doesn't need to be
described.

Will

Re: [PATCH v4] devicetree: Add generic IOMMU device tree bindings

[Thierry Reding]

[-- Attachment #1: Type: text/plain, Size: 3147 bytes --]

On Wed, Jul 09, 2014 at 07:10:48PM +0100, Will Deacon wrote:
> Hi Thierry,
> 
> On Wed, Jul 09, 2014 at 03:21:27PM +0100, Thierry Reding wrote:
> > On Wed, Jul 09, 2014 at 02:40:50PM +0100, Will Deacon wrote:
> > > I would like to move the ARM SMMU driver over to this for 3.18, if possible.
> > > One use-case there is the ability to describe groups of masters behind a
> > > multi-master IOMMU but which must be part of the same domain (i.e. an
> > > iommu_group). This is useful for presenting devices to a guest with a
> > > virtual SMMU, where the physical devices share a stage-2 context.
> > > 
> > > With your binding, does this simply mean determining the set of master IDs
> > > in the group, then describing the complete set for each master?
> > 
> > I'm not sure I properly understand what you're trying to do, but I don't
> > think the binding is designed to cover that. Rather the goal was to
> > describe the IDs belonging to each master, so that an IOMMU can be
> > properly configured.
> 
> This is directly related to that problem, see below.
> 
> > Anything beyond that (e.g. logical grouping of masters) isn't directly
> > within the scope of the binding (it doesn't describe hardware but some
> > policy pertaining to some specific use-case).
> 
> This *is* for hardware. I can use PCI as an example, but this could equally
> apply to other types of bus. If you have a bunch of PCI master devices
> sitting being a non-transparent bridge, they can end up sharing the same
> master device ID (requester ID). This means that there is no way in the
> IOMMU to initialise a translation for one of these devices without also
> affecting the others. We currently have iommu_groups to deal with this, but
> it *is* a property of the hardware and we absolutely need a way to describe
> it. I'm happy to add it later, but we need to think about it now to avoid
> merging something that can't easily be extended.
> 
> For PCI, the topology is probable but even then, we need this information to
> describe the resulting master device ID emitted by the bridge for the
> upstream group. One way to do this with your binding would be to treat all
> of the upstream masters as having the same device ID.

Yes, I think that makes most sense. After all from the IOMMU's point of
view requests from all devices behind the bridge will originate from the
same ID.

So technically it's not really correct to encode the master ID within
each of the devices, but rather they should be inheriting the ID from
the non-transparent bridge.

> With virtualisation, we may want to assign a group of devices to a guest but
> without emulating the bridge. This would need something the device-tree to
> describe that they are grouped together.

But that's also a software decision, isn't it? Virtualization doesn't
have anything to do with the hardware description. Or am I missing
something? Of course I guess you could generate a DTB for the guest and
group device together, in which case you're pretty much free to do what
you want since you're essentially defining your own hardware.

Thierry

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: [PATCH v4] devicetree: Add generic IOMMU device tree bindings

[Will Deacon]

On Thu, Jul 10, 2014 at 10:49:10AM +0100, Thierry Reding wrote:
> On Wed, Jul 09, 2014 at 07:10:48PM +0100, Will Deacon wrote:
> > On Wed, Jul 09, 2014 at 03:21:27PM +0100, Thierry Reding wrote:
> > > Anything beyond that (e.g. logical grouping of masters) isn't directly
> > > within the scope of the binding (it doesn't describe hardware but some
> > > policy pertaining to some specific use-case).
> > 
> > This *is* for hardware. I can use PCI as an example, but this could equally
> > apply to other types of bus. If you have a bunch of PCI master devices
> > sitting being a non-transparent bridge, they can end up sharing the same
> > master device ID (requester ID). This means that there is no way in the
> > IOMMU to initialise a translation for one of these devices without also
> > affecting the others. We currently have iommu_groups to deal with this, but
> > it *is* a property of the hardware and we absolutely need a way to describe
> > it. I'm happy to add it later, but we need to think about it now to avoid
> > merging something that can't easily be extended.
> > 
> > For PCI, the topology is probable but even then, we need this information to
> > describe the resulting master device ID emitted by the bridge for the
> > upstream group. One way to do this with your binding would be to treat all
> > of the upstream masters as having the same device ID.
> 
> Yes, I think that makes most sense. After all from the IOMMU's point of
> view requests from all devices behind the bridge will originate from the
> same ID.
> 
> So technically it's not really correct to encode the master ID within
> each of the devices, but rather they should be inheriting the ID from
> the non-transparent bridge.

Indeed. Is that possible with your binding, or would we just duplicate the
IDs between the masters?

> > With virtualisation, we may want to assign a group of devices to a guest but
> > without emulating the bridge. This would need something the device-tree to
> > describe that they are grouped together.
> 
> But that's also a software decision, isn't it? Virtualization doesn't
> have anything to do with the hardware description. Or am I missing
> something? Of course I guess you could generate a DTB for the guest and
> group device together, in which case you're pretty much free to do what
> you want since you're essentially defining your own hardware.

If you're doing device passthrough and you want to allow the guest to
program the IOMMU, I think that virtualisation is directly related to the
hardware description, since the guest will be bound by physical properties
of the system.

Will

Re: [PATCH v4] devicetree: Add generic IOMMU device tree bindings

[Thierry Reding]

[-- Attachment #1: Type: text/plain, Size: 4231 bytes --]

On Thu, Jul 10, 2014 at 11:23:34AM +0100, Will Deacon wrote:
> On Thu, Jul 10, 2014 at 10:49:10AM +0100, Thierry Reding wrote:
> > On Wed, Jul 09, 2014 at 07:10:48PM +0100, Will Deacon wrote:
> > > On Wed, Jul 09, 2014 at 03:21:27PM +0100, Thierry Reding wrote:
> > > > Anything beyond that (e.g. logical grouping of masters) isn't directly
> > > > within the scope of the binding (it doesn't describe hardware but some
> > > > policy pertaining to some specific use-case).
> > > 
> > > This *is* for hardware. I can use PCI as an example, but this could equally
> > > apply to other types of bus. If you have a bunch of PCI master devices
> > > sitting being a non-transparent bridge, they can end up sharing the same
> > > master device ID (requester ID). This means that there is no way in the
> > > IOMMU to initialise a translation for one of these devices without also
> > > affecting the others. We currently have iommu_groups to deal with this, but
> > > it *is* a property of the hardware and we absolutely need a way to describe
> > > it. I'm happy to add it later, but we need to think about it now to avoid
> > > merging something that can't easily be extended.
> > > 
> > > For PCI, the topology is probable but even then, we need this information to
> > > describe the resulting master device ID emitted by the bridge for the
> > > upstream group. One way to do this with your binding would be to treat all
> > > of the upstream masters as having the same device ID.
> > 
> > Yes, I think that makes most sense. After all from the IOMMU's point of
> > view requests from all devices behind the bridge will originate from the
> > same ID.
> > 
> > So technically it's not really correct to encode the master ID within
> > each of the devices, but rather they should be inheriting the ID from
> > the non-transparent bridge.
> 
> Indeed. Is that possible with your binding, or would we just duplicate the
> IDs between the masters?

No, the binding only describes direct relationships between the IOMMU
and masters. There's no way to translate them inbetween or inherit them.

I'm wondering how this could be described in device tree, though.
Perhaps something like this:

	iommu {
		#iommu-cells = <1>;
	};

	bridge {
		iommus = <&/iommu 42>;
		#iommu-cells = <0>;

		device@0 {
			iommus = <&/bridge>;
		};

		device@1 {
			iommus = <&/bridge>;
		};

		...
	};

? That way some code could walk up the IOMMU tree to resolve this. Or
perhaps even easier:

	iommu {
		#iommu-cells = <1>;
	};

	bridge {
		iommus = <&/iommu 42>;

		device@0 {
			...
		};

		device@1 {
			...
		};

		...
	};

And we could enhance the binding by defining that the iommus node is
inherited by devices on a bus, which by what you're saying would be the
sensible thing to do anyway.

In the second example above, the presence of an iommus property in the
bridge would indicate that it's non-transparent regarding IOMMU
translation and therefore the master ID should be inherited. Devices
could still override by providing their own iommus property, though I'd
be a little surprised if there ever was hardware like that.

> > > With virtualisation, we may want to assign a group of devices to a guest but
> > > without emulating the bridge. This would need something the device-tree to
> > > describe that they are grouped together.
> > 
> > But that's also a software decision, isn't it? Virtualization doesn't
> > have anything to do with the hardware description. Or am I missing
> > something? Of course I guess you could generate a DTB for the guest and
> > group device together, in which case you're pretty much free to do what
> > you want since you're essentially defining your own hardware.
> 
> If you're doing device passthrough and you want to allow the guest to
> program the IOMMU, I think that virtualisation is directly related to the
> hardware description, since the guest will be bound by physical properties
> of the system.

Evidently you know much better what the requirements are here and what
will actually be required. I guess we'll need to have more discussions
along with examples of use-cases.

Thierry

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: [PATCH v4] devicetree: Add generic IOMMU device tree bindings

[Will Deacon]

On Thu, Jul 10, 2014 at 11:57:38AM +0100, Thierry Reding wrote:
> On Thu, Jul 10, 2014 at 11:23:34AM +0100, Will Deacon wrote:
> > On Thu, Jul 10, 2014 at 10:49:10AM +0100, Thierry Reding wrote:
> > > On Wed, Jul 09, 2014 at 07:10:48PM +0100, Will Deacon wrote:
> > > > On Wed, Jul 09, 2014 at 03:21:27PM +0100, Thierry Reding wrote:
> > > > > Anything beyond that (e.g. logical grouping of masters) isn't directly
> > > > > within the scope of the binding (it doesn't describe hardware but some
> > > > > policy pertaining to some specific use-case).
> > > > 
> > > > This *is* for hardware. I can use PCI as an example, but this could equally
> > > > apply to other types of bus. If you have a bunch of PCI master devices
> > > > sitting being a non-transparent bridge, they can end up sharing the same
> > > > master device ID (requester ID). This means that there is no way in the
> > > > IOMMU to initialise a translation for one of these devices without also
> > > > affecting the others. We currently have iommu_groups to deal with this, but
> > > > it *is* a property of the hardware and we absolutely need a way to describe
> > > > it. I'm happy to add it later, but we need to think about it now to avoid
> > > > merging something that can't easily be extended.
> > > > 
> > > > For PCI, the topology is probable but even then, we need this information to
> > > > describe the resulting master device ID emitted by the bridge for the
> > > > upstream group. One way to do this with your binding would be to treat all
> > > > of the upstream masters as having the same device ID.
> > > 
> > > Yes, I think that makes most sense. After all from the IOMMU's point of
> > > view requests from all devices behind the bridge will originate from the
> > > same ID.
> > > 
> > > So technically it's not really correct to encode the master ID within
> > > each of the devices, but rather they should be inheriting the ID from
> > > the non-transparent bridge.
> > 
> > Indeed. Is that possible with your binding, or would we just duplicate the
> > IDs between the masters?
> 
> No, the binding only describes direct relationships between the IOMMU
> and masters. There's no way to translate them inbetween or inherit them.

[...]

> ? That way some code could walk up the IOMMU tree to resolve this. Or
> perhaps even easier:
> 
> 	iommu {
> 		#iommu-cells = <1>;
> 	};
> 
> 	bridge {
> 		iommus = <&/iommu 42>;
> 
> 		device@0 {
> 			...
> 		};
> 
> 		device@1 {
> 			...
> 		};
> 
> 		...
> 	};

Yes, I like that. Good thinking!

> And we could enhance the binding by defining that the iommus node is
> inherited by devices on a bus, which by what you're saying would be the
> sensible thing to do anyway.
> 
> In the second example above, the presence of an iommus property in the
> bridge would indicate that it's non-transparent regarding IOMMU
> translation and therefore the master ID should be inherited. Devices
> could still override by providing their own iommus property, though I'd
> be a little surprised if there ever was hardware like that.
> 
> > > > With virtualisation, we may want to assign a group of devices to a guest but
> > > > without emulating the bridge. This would need something the device-tree to
> > > > describe that they are grouped together.
> > > 
> > > But that's also a software decision, isn't it? Virtualization doesn't
> > > have anything to do with the hardware description. Or am I missing
> > > something? Of course I guess you could generate a DTB for the guest and
> > > group device together, in which case you're pretty much free to do what
> > > you want since you're essentially defining your own hardware.
> > 
> > If you're doing device passthrough and you want to allow the guest to
> > program the IOMMU, I think that virtualisation is directly related to the
> > hardware description, since the guest will be bound by physical properties
> > of the system.
> 
> Evidently you know much better what the requirements are here and what
> will actually be required. I guess we'll need to have more discussions
> along with examples of use-cases.

It's still early days for getting this stuff up and running on ARM, so I
agree that we'll have to come back to it a few times once we've got concrete
examples and code.

Will

Re: [PATCH v4] devicetree: Add generic IOMMU device tree bindings

[Rob Clark]

On Fri, Jul 4, 2014 at 11:29 AM, Thierry Reding
<thierry.reding@gmail.com> wrote:
> From: Thierry Reding <treding@nvidia.com>
>
> This commit introduces a generic device tree binding for IOMMU devices.
> Only a very minimal subset is described here, but it is enough to cover
> the requirements of both the Exynos System MMU and Tegra SMMU as
> discussed here:
>
>     https://lkml.org/lkml/2014/4/27/346
>
> Signed-off-by: Thierry Reding <treding@nvidia.com>
> ---
> Changes in v4:
> - clarify that disabling an IOMMU DT node may not disable translation
> - be more explicit that examples are only examples
> - add multi-ID master example
>
> Changes in v3:
> - use #iommu-cells instead of #address-cells/#size-cells
> - drop optional iommu-names property
>
> Changes in v2:
> - add notes about "dma-ranges" property (drop note from commit message)
> - document priorities of "iommus" property vs. "dma-ranges" property
> - drop #iommu-cells in favour of #address-cells and #size-cells
> - remove multiple-master device example
>
>  Documentation/devicetree/bindings/iommu/iommu.txt | 172 ++++++++++++++++++++++
>  1 file changed, 172 insertions(+)
>  create mode 100644 Documentation/devicetree/bindings/iommu/iommu.txt
>
> diff --git a/Documentation/devicetree/bindings/iommu/iommu.txt b/Documentation/devicetree/bindings/iommu/iommu.txt
> new file mode 100644
> index 000000000000..464a81eaaf61
> --- /dev/null
> +++ b/Documentation/devicetree/bindings/iommu/iommu.txt
> @@ -0,0 +1,172 @@
> +This document describes the generic device tree binding for IOMMUs and their
> +master(s).
> +
> +
> +IOMMU device node:
> +==================
> +
> +An IOMMU can provide the following services:
> +
> +* Remap address space to allow devices to access physical memory ranges that
> +  they otherwise wouldn't be capable of accessing.
> +
> +  Example: 32-bit DMA to 64-bit physical addresses
> +
> +* Implement scatter-gather at page level granularity so that the device does
> +  not have to.
> +
> +* Provide system protection against "rogue" DMA by forcing all accesses to go
> +  through the IOMMU and faulting when encountering accesses to unmapped
> +  address regions.
> +
> +* Provide address space isolation between multiple contexts.
> +
> +  Example: Virtualization
> +
> +Device nodes compatible with this binding represent hardware with some of the
> +above capabilities.
> +
> +IOMMUs can be single-master or multiple-master. Single-master IOMMU devices
> +typically have a fixed association to the master device, whereas multiple-
> +master IOMMU devices can translate accesses from more than one master.
> +
> +The device tree node of the IOMMU device's parent bus must contain a valid
> +"dma-ranges" property that describes how the physical address space of the
> +IOMMU maps to memory. An empty "dma-ranges" property means that there is a
> +1:1 mapping from IOMMU to memory.
> +
> +Required properties:
> +--------------------
> +- #iommu-cells: The number of cells in an IOMMU specifier needed to encode an
> +  address.
> +
> +The meaning of the IOMMU specifier is defined by the device tree binding of
> +the specific IOMMU. Below are a few examples of typical use-cases:
> +
> +- #iommu-cells = <0>: Single master IOMMU devices are not configurable and
> +  therefore no additional information needs to be encoded in the specifier.
> +  This may also apply to multiple master IOMMU devices that do not allow the
> +  association of masters to be configured. Note that an IOMMU can by design
> +  be multi-master yet only expose a single master in a given configuration.
> +  In such cases the number of cells will usually be 1 as in the next case.
> +- #iommu-cells = <1>: Multiple master IOMMU devices may need to be configured
> +  in order to enable translation for a given master. In such cases the single
> +  address cell corresponds to the master device's ID. In some cases more than
> +  one cell can be required to represent a single master ID.
> +- #iommu-cells = <4>: Some IOMMU devices allow the DMA window for masters to
> +  be configured. The first cell of the address in this may contain the master
> +  device's ID for example, while the second cell could contain the start of
> +  the DMA window for the given device. The length of the DMA window is given
> +  by the third and fourth cells.
> +
> +Note that these are merely examples and real-world use-cases may use different
> +definitions to represent their individual needs. Always refer to the specific
> +IOMMU binding for the exact meaning of the cells that make up the specifier.
> +
> +
> +IOMMU master node:
> +==================
> +
> +Devices that access memory through an IOMMU are called masters. A device can
> +have multiple master interfaces (to one or more IOMMU devices).
> +
> +Required properties:
> +--------------------
> +- iommus: A list of phandle and IOMMU specifier pairs that describe the IOMMU
> +  master interfaces of the device. One entry in the list describes one master
> +  interface of the device.
> +
> +When an "iommus" property is specified in a device tree node, the IOMMU will
> +be used for address translation. If a "dma-ranges" property exists in the
> +device's parent node it will be ignored. An exception to this rule is if the
> +referenced IOMMU is disabled, in which case the "dma-ranges" property of the
> +parent shall take effect. Note that merely disabling a device tree node does
> +not guarantee that the IOMMU is really disabled since the hardware may not
> +have a means to turn off translation.
> +
> +
> +Notes:
> +======
> +
> +One possible extension to the above is to use an "iommus" property along with
> +a "dma-ranges" property in a bus device node (such as PCI host bridges). This
> +can be useful to describe how children on the bus relate to the IOMMU if they
> +are not explicitly listed in the device tree (e.g. PCI devices). However, the
> +requirements of that use-case haven't been fully determined yet. Implementing
> +this is therefore not recommended without further discussion and extension of
> +this binding.
> +
> +
> +Examples:
> +=========
> +
> +Single-master IOMMU:
> +--------------------
> +
> +       iommu {
> +               #iommu-cells = <0>;
> +       };
> +
> +       master {
> +               iommus = <&/iommu>;
> +       };
> +
> +Multiple-master IOMMU with fixed associations:
> +----------------------------------------------
> +
> +       /* multiple-master IOMMU */
> +       iommu {
> +               /*
> +                * Masters are statically associated with this IOMMU and
> +                * address translation is always enabled.
> +                */
> +               #iommu-cells = <0>;
> +       };
> +
> +       /* static association with IOMMU */
> +       master@1 {
> +               reg = <1>;
> +               iommus = <&/iommu>;
> +       };
> +
> +       /* static association with IOMMU */
> +       master@2 {
> +               reg = <2>;
> +               iommus = <&/iommu>;
> +       };
> +
> +Multiple-master IOMMU:
> +----------------------
> +
> +       iommu {
> +               /* the specifier represents the ID of the master */
> +               #iommu-cells = <1>;
> +       };
> +
> +       master@1 {
> +               /* device has master ID 42 in the IOMMU */
> +               iommus = <&/iommu 42>;
> +       };
> +
> +       master@2 {
> +               /* device has master IDs 23 and 24 in the IOMMU */
> +               iommus = <&/iommu 23>, <&/iommu 24>;
> +       };
> +
> +Multiple-master IOMMU with configurable DMA window:
> +---------------------------------------------------
> +
> +       / {
> +               #address-cells = <1>;
> +               #size-cells = <1>;
> +
> +               iommu {
> +                       /* master ID, address and length of DMA window */
> +                       #iommu-cells = <4>;
> +               };
> +
> +               master {
> +                       /* master ID 42, 4 GiB DMA window starting at 0 */
> +                       iommus = <&/iommu  42  0  0x1 0x0>;
> +               };
> +       };


ok, so I was working through this to try to convert my
{qcom,msm}-iommu-v0 RFC over to using these bindings.  For background,
I was initially using something that looked a bit more like the
current arm-smmu bindings:

        gpu {
            #stream-id-cells = <16>;
            ...
        };

        gfx3d: qcom,iommu@7c00000 {
            compatible = "qcom,iommu-v0";
            ...
            mmu-masters =
                /* gfx3d_user: */
                <&gpu 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15>,
                /* gfx3d_priv: */
                <&gpu 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31>;
        };

        gfx3d1: qcom,iommu@7d00000 {
            compatible = "qcom,iommu-v0";
            ...
            mmu-masters =
                /* gfx3d_user: */
                <&gpu 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15>,
                /* gfx3d_priv: */
                <&gpu 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31>;
        };

With my current arrangement, I have everything I need when the iommu
device is probed to set up each of the context banks

This proposal inverts that relationship.  Which forces me to do a lot
more (including DT parsing) on device attach.  Which I'm not a huge
fan of.  Ie. if I even wanted to try to implement per-process
pagetables for gpu without completely going behind the IOMMU API's
back, I would want attach/detach to be as lightweight as possible.

Was there actually a good reason for having the device link to the
iommu rather than the other way around?  How much would people hate it
if I just ignore the generic bindings and use something that works for
me instead.  I mean, it isn't exactly like there is going to be .dts
re-use across different SoC's..  and at least with current IOMMU API
some sort of of_get_named_iommu() API doesn't really make sense.

Maybe I'm missing something, I kinda jumped into the discussion late.

BR,
-R


> --
> 2.0.1
>
>
> _______________________________________________
> linux-arm-kernel mailing list
> linux-arm-kernel@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v4] devicetree: Add generic IOMMU device tree bindings

[Will Deacon]

Hi Rob,

On Fri, Jul 11, 2014 at 09:55:14PM +0100, Rob Clark wrote:
> On Fri, Jul 4, 2014 at 11:29 AM, Thierry Reding
> <thierry.reding@gmail.com> wrote:
> > From: Thierry Reding <treding@nvidia.com>
> ok, so I was working through this to try to convert my
> {qcom,msm}-iommu-v0 RFC over to using these bindings.  For background,
> I was initially using something that looked a bit more like the
> current arm-smmu bindings:
> 
>         gpu {
>             #stream-id-cells = <16>;
>             ...
>         };
> 
>         gfx3d: qcom,iommu@7c00000 {
>             compatible = "qcom,iommu-v0";
>             ...
>             mmu-masters =
>                 /* gfx3d_user: */
>                 <&gpu 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15>,
>                 /* gfx3d_priv: */
>                 <&gpu 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31>;
>         };
> 
>         gfx3d1: qcom,iommu@7d00000 {
>             compatible = "qcom,iommu-v0";
>             ...
>             mmu-masters =
>                 /* gfx3d_user: */
>                 <&gpu 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15>,
>                 /* gfx3d_priv: */
>                 <&gpu 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31>;
>         };
> 
> With my current arrangement, I have everything I need when the iommu
> device is probed to set up each of the context banks
> 
> This proposal inverts that relationship.  Which forces me to do a lot
> more (including DT parsing) on device attach.  Which I'm not a huge
> fan of.  Ie. if I even wanted to try to implement per-process
> pagetables for gpu without completely going behind the IOMMU API's
> back, I would want attach/detach to be as lightweight as possible.

I think we'd have to walk the entire device-tree at IOMMU probe time in
order to enumerate the masters, which sounds horrible... If we want to do
clever SMR allocation on the ARM SMMU (as I've been discussing with Olav),
we're going to need knowledge about *all* the valid Stream IDs in the system
before we can program *any* translations.

> Was there actually a good reason for having the device link to the
> iommu rather than the other way around?  How much would people hate it
> if I just ignore the generic bindings and use something that works for
> me instead.  I mean, it isn't exactly like there is going to be .dts
> re-use across different SoC's..  and at least with current IOMMU API
> some sort of of_get_named_iommu() API doesn't really make sense.

The thing is, if you end up ignoring the generic binding then we have two
IOMMUs using the same (ARM SMMU) binding and it begs the question as to
which is the more generic! I know we're keen to get this merged, but merging
something that people won't use and calling it generic doesn't seem ideal
either. We do, however, desperately need a generic binding.

Turning the question around; Thierry -- what are the issue in using
something like the ARM SMMU binding (master device IDs held in the IOMMU
node) for the nvidia IOMMU?

Will

Re: [PATCH v4] devicetree: Add generic IOMMU device tree bindings

[Rob Clark]

On Sat, Jul 12, 2014 at 5:39 AM, Will Deacon <will.deacon@arm.com> wrote:
> Hi Rob,
>
> On Fri, Jul 11, 2014 at 09:55:14PM +0100, Rob Clark wrote:
>> On Fri, Jul 4, 2014 at 11:29 AM, Thierry Reding
>> <thierry.reding@gmail.com> wrote:
>> > From: Thierry Reding <treding@nvidia.com>
>> ok, so I was working through this to try to convert my
>> {qcom,msm}-iommu-v0 RFC over to using these bindings.  For background,
>> I was initially using something that looked a bit more like the
>> current arm-smmu bindings:
>>
>>         gpu {
>>             #stream-id-cells = <16>;
>>             ...
>>         };
>>
>>         gfx3d: qcom,iommu@7c00000 {
>>             compatible = "qcom,iommu-v0";
>>             ...
>>             mmu-masters =
>>                 /* gfx3d_user: */
>>                 <&gpu 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15>,
>>                 /* gfx3d_priv: */
>>                 <&gpu 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31>;
>>         };
>>
>>         gfx3d1: qcom,iommu@7d00000 {
>>             compatible = "qcom,iommu-v0";
>>             ...
>>             mmu-masters =
>>                 /* gfx3d_user: */
>>                 <&gpu 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15>,
>>                 /* gfx3d_priv: */
>>                 <&gpu 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31>;
>>         };
>>
>> With my current arrangement, I have everything I need when the iommu
>> device is probed to set up each of the context banks
>>
>> This proposal inverts that relationship.  Which forces me to do a lot
>> more (including DT parsing) on device attach.  Which I'm not a huge
>> fan of.  Ie. if I even wanted to try to implement per-process
>> pagetables for gpu without completely going behind the IOMMU API's
>> back, I would want attach/detach to be as lightweight as possible.
>
> I think we'd have to walk the entire device-tree at IOMMU probe time in
> order to enumerate the masters, which sounds horrible... If we want to do
> clever SMR allocation on the ARM SMMU (as I've been discussing with Olav),
> we're going to need knowledge about *all* the valid Stream IDs in the system
> before we can program *any* translations.

I guess walking the whole tree would work.. but it really doesn't
really sound like a nice solution ;-)

also, I'm not really sure to what extent it is bad form to assume the
tree is static.

>> Was there actually a good reason for having the device link to the
>> iommu rather than the other way around?  How much would people hate it
>> if I just ignore the generic bindings and use something that works for
>> me instead.  I mean, it isn't exactly like there is going to be .dts
>> re-use across different SoC's..  and at least with current IOMMU API
>> some sort of of_get_named_iommu() API doesn't really make sense.
>
> The thing is, if you end up ignoring the generic binding then we have two
> IOMMUs using the same (ARM SMMU) binding and it begs the question as to
> which is the more generic! I know we're keen to get this merged, but merging
> something that people won't use and calling it generic doesn't seem ideal
> either. We do, however, desperately need a generic binding.

yeah, ignoring the generic binding is not my first choice.  I'd rather
have something that works well for everyone.  But I wasn't really sure
if the current proposal was arbitrary, or if there are some
conflicting requirements between different platforms.

> Turning the question around; Thierry -- what are the issue in using
> something like the ARM SMMU binding (master device IDs held in the IOMMU
> node) for the nvidia IOMMU?

+1 for doing it more like how arm-smmu is currently.. that works much
better for me :-)

BR,
-R

> Will

Re: [PATCH v4] devicetree: Add generic IOMMU device tree bindings

[Arnd Bergmann]

On Saturday 12 July 2014, Rob Clark wrote:
> >> Was there actually a good reason for having the device link to the
> >> iommu rather than the other way around?  How much would people hate it
> >> if I just ignore the generic bindings and use something that works for
> >> me instead.  I mean, it isn't exactly like there is going to be .dts
> >> re-use across different SoC's..  and at least with current IOMMU API
> >> some sort of of_get_named_iommu() API doesn't really make sense.
> >
> > The thing is, if you end up ignoring the generic binding then we have two
> > IOMMUs using the same (ARM SMMU) binding and it begs the question as to
> > which is the more generic! I know we're keen to get this merged, but merging
> > something that people won't use and calling it generic doesn't seem ideal
> > either. We do, however, desperately need a generic binding.
> 
> yeah, ignoring the generic binding is not my first choice.  I'd rather
> have something that works well for everyone.  But I wasn't really sure
> if the current proposal was arbitrary, or if there are some
> conflicting requirements between different platforms.

The common case that needs to be simple is attaching one (master) device
to an IOMMU using the shared global context for the purposes of implementing
the dma-mapping API.

The way that Thierry's binding does that is the obvious solution to this,
and it mirrors what we do in practically every other subsystem. I definitely
want the SMMU to change before anybody starts using it in a real system,
which we fortunately do not have yet.

	Arnd

Re: [PATCH v4] devicetree: Add generic IOMMU device tree bindings

[Rob Clark]

On Sat, Jul 12, 2014 at 8:22 AM, Arnd Bergmann <arnd@arndb.de> wrote:
> On Saturday 12 July 2014, Rob Clark wrote:
>> >> Was there actually a good reason for having the device link to the
>> >> iommu rather than the other way around?  How much would people hate it
>> >> if I just ignore the generic bindings and use something that works for
>> >> me instead.  I mean, it isn't exactly like there is going to be .dts
>> >> re-use across different SoC's..  and at least with current IOMMU API
>> >> some sort of of_get_named_iommu() API doesn't really make sense.
>> >
>> > The thing is, if you end up ignoring the generic binding then we have two
>> > IOMMUs using the same (ARM SMMU) binding and it begs the question as to
>> > which is the more generic! I know we're keen to get this merged, but merging
>> > something that people won't use and calling it generic doesn't seem ideal
>> > either. We do, however, desperately need a generic binding.
>>
>> yeah, ignoring the generic binding is not my first choice.  I'd rather
>> have something that works well for everyone.  But I wasn't really sure
>> if the current proposal was arbitrary, or if there are some
>> conflicting requirements between different platforms.
>
> The common case that needs to be simple is attaching one (master) device
> to an IOMMU using the shared global context for the purposes of implementing
> the dma-mapping API.

well, I don't disagree that IOMMU API has some problems.  It is too
tied to the bus type, which doesn't really seem to make sense for
platform devices.  (Unless we start having multiple platform busses?)

But at least given the current IOMMU API I'm not really sure how it
makes a difference which way the link goes.  But if there has already
been some discussion about how you want to handle the tie in with
dma-mapping, if you could point me at that then maybe your point will
make more sense to me.

> The way that Thierry's binding does that is the obvious solution to this,
> and it mirrors what we do in practically every other subsystem. I definitely
> want the SMMU to change before anybody starts using it in a real system,
> which we fortunately do not have yet.

hmm, well if some of the things I need for (like this or batching
mappings) are too weird and gpu specific, I'm willing to duplicate the
IOMMU driver in drm/msm.  It really isn't so much code, and that gives
me a lot more more flexibility to do crazy things... at some point I'm
probably going to want to do context switches by banging the IOMMU
registers directly from the gpu.

But given what Will said, I don't think what I need here is too far
out of line.  But if it is really a problem for dma-mapping, I suppose
we could have links in both directions?  However, I think the link
which contains the stream-id's really needs to be in the IOMMU, not
the device using the IOMMU.

BR,
-R

Re: [PATCH v4] devicetree: Add generic IOMMU device tree bindings

[Will Deacon]

On Sat, Jul 12, 2014 at 01:57:31PM +0100, Rob Clark wrote:
> On Sat, Jul 12, 2014 at 8:22 AM, Arnd Bergmann <arnd@arndb.de> wrote:
> > On Saturday 12 July 2014, Rob Clark wrote:
> >> >> Was there actually a good reason for having the device link to the
> >> >> iommu rather than the other way around?  How much would people hate it
> >> >> if I just ignore the generic bindings and use something that works for
> >> >> me instead.  I mean, it isn't exactly like there is going to be .dts
> >> >> re-use across different SoC's..  and at least with current IOMMU API
> >> >> some sort of of_get_named_iommu() API doesn't really make sense.
> >> >
> >> > The thing is, if you end up ignoring the generic binding then we have two
> >> > IOMMUs using the same (ARM SMMU) binding and it begs the question as to
> >> > which is the more generic! I know we're keen to get this merged, but merging
> >> > something that people won't use and calling it generic doesn't seem ideal
> >> > either. We do, however, desperately need a generic binding.
> >>
> >> yeah, ignoring the generic binding is not my first choice.  I'd rather
> >> have something that works well for everyone.  But I wasn't really sure
> >> if the current proposal was arbitrary, or if there are some
> >> conflicting requirements between different platforms.
> >
> > The common case that needs to be simple is attaching one (master) device
> > to an IOMMU using the shared global context for the purposes of implementing
> > the dma-mapping API.
> 
> well, I don't disagree that IOMMU API has some problems.  It is too
> tied to the bus type, which doesn't really seem to make sense for
> platform devices.  (Unless we start having multiple platform busses?)
> 
> But at least given the current IOMMU API I'm not really sure how it
> makes a difference which way the link goes.  But if there has already
> been some discussion about how you want to handle the tie in with
> dma-mapping, if you could point me at that then maybe your point will
> make more sense to me.

If you look at the proposed binding in isolation, I think it *is* cleaner
than the ARM SMMU binding (I've acked it...) and I believe it's more
consistent with the way we describe linkages elsewhere.

However, the issue you're raising is that it's more difficult to make use of
in a Linux IOMMU driver. The reward you'll get for using it will come
eventually when the DMA ops are automatically swizzled for devices using the
generic binding.

My plan for the ARM SMMU driver is:

  (1) Change ->probe() to walk the device-tree looking for all masters with
      phandles back to the SMMU instance being probed

  (2) For each master, extract the Stream IDs and add them to the internal
      SMMU driver data structures (an rbtree per SMMU instance). For
      hotpluggable buses, we'll need a way for the bus controller to
      reserve a range of IDs -- this will likely be a later extension to
      the binding.

  (3) When we get an ->add() call, warn if it's a device we haven't seen
      and reject the addition.

That way, ->attach() should be the same as it is now, I think.

Have you tried implementing something like that? We agreed that (1) isn't
pretty, but I don't have a good alternative and it's only done at
probe-time.

Will

BTW: Is the msm-v0 IOMMU compatible with the ARM SMMU driver, or is it a
completely different design requiring a different driver?

Re: [PATCH v4] devicetree: Add generic IOMMU device tree bindings

[Rob Clark]

On Sun, Jul 13, 2014 at 5:43 AM, Will Deacon <will.deacon@arm.com> wrote:
> On Sat, Jul 12, 2014 at 01:57:31PM +0100, Rob Clark wrote:
>> On Sat, Jul 12, 2014 at 8:22 AM, Arnd Bergmann <arnd@arndb.de> wrote:
>> > On Saturday 12 July 2014, Rob Clark wrote:
>> >> >> Was there actually a good reason for having the device link to the
>> >> >> iommu rather than the other way around?  How much would people hate it
>> >> >> if I just ignore the generic bindings and use something that works for
>> >> >> me instead.  I mean, it isn't exactly like there is going to be .dts
>> >> >> re-use across different SoC's..  and at least with current IOMMU API
>> >> >> some sort of of_get_named_iommu() API doesn't really make sense.
>> >> >
>> >> > The thing is, if you end up ignoring the generic binding then we have two
>> >> > IOMMUs using the same (ARM SMMU) binding and it begs the question as to
>> >> > which is the more generic! I know we're keen to get this merged, but merging
>> >> > something that people won't use and calling it generic doesn't seem ideal
>> >> > either. We do, however, desperately need a generic binding.
>> >>
>> >> yeah, ignoring the generic binding is not my first choice.  I'd rather
>> >> have something that works well for everyone.  But I wasn't really sure
>> >> if the current proposal was arbitrary, or if there are some
>> >> conflicting requirements between different platforms.
>> >
>> > The common case that needs to be simple is attaching one (master) device
>> > to an IOMMU using the shared global context for the purposes of implementing
>> > the dma-mapping API.
>>
>> well, I don't disagree that IOMMU API has some problems.  It is too
>> tied to the bus type, which doesn't really seem to make sense for
>> platform devices.  (Unless we start having multiple platform busses?)
>>
>> But at least given the current IOMMU API I'm not really sure how it
>> makes a difference which way the link goes.  But if there has already
>> been some discussion about how you want to handle the tie in with
>> dma-mapping, if you could point me at that then maybe your point will
>> make more sense to me.
>
> If you look at the proposed binding in isolation, I think it *is* cleaner
> than the ARM SMMU binding (I've acked it...) and I believe it's more
> consistent with the way we describe linkages elsewhere.
>
> However, the issue you're raising is that it's more difficult to make use of
> in a Linux IOMMU driver. The reward you'll get for using it will come
> eventually when the DMA ops are automatically swizzled for devices using the
> generic binding.
>
> My plan for the ARM SMMU driver is:
>
>   (1) Change ->probe() to walk the device-tree looking for all masters with
>       phandles back to the SMMU instance being probed
>
>   (2) For each master, extract the Stream IDs and add them to the internal
>       SMMU driver data structures (an rbtree per SMMU instance). For
>       hotpluggable buses, we'll need a way for the bus controller to
>       reserve a range of IDs -- this will likely be a later extension to
>       the binding.
>
>   (3) When we get an ->add() call, warn if it's a device we haven't seen
>       and reject the addition.
>
> That way, ->attach() should be the same as it is now, I think.
>
> Have you tried implementing something like that? We agreed that (1) isn't
> pretty, but I don't have a good alternative and it's only done at
> probe-time.

I haven't tried implementing that yet, but I'm sure it would work.  I
was just hoping to avoid having to do that ;-)

I suppose perhaps there is room for a shared helper here, to at least
avoid duplicating that in each IOMMU driver which needs the
stream-id's up front.

> Will
>
> BTW: Is the msm-v0 IOMMU compatible with the ARM SMMU driver, or is it a
> completely different design requiring a different driver?

My understanding is that it is different from msm v1 IOMMU, although I
think it shares the same pagetable format with v1.  Not sure if that
is the same as arm-smmu?   If so it might be nice to try to extract
out some shared helper fxns for map/unmap as well.

I expect Olav knows better the similarities/differences.

BR,
-R

Re: [PATCH v4] devicetree: Add generic IOMMU device tree bindings

[Olav Haugan]

On 7/13/2014 4:43 AM, Rob Clark wrote:
> On Sun, Jul 13, 2014 at 5:43 AM, Will Deacon <will.deacon@arm.com> wrote:
>> On Sat, Jul 12, 2014 at 01:57:31PM +0100, Rob Clark wrote:
>>> On Sat, Jul 12, 2014 at 8:22 AM, Arnd Bergmann <arnd@arndb.de> wrote:
>>>> On Saturday 12 July 2014, Rob Clark wrote:
>>>>>>> Was there actually a good reason for having the device link to the
>>>>>>> iommu rather than the other way around?  How much would people hate it
>>>>>>> if I just ignore the generic bindings and use something that works for
>>>>>>> me instead.  I mean, it isn't exactly like there is going to be .dts
>>>>>>> re-use across different SoC's..  and at least with current IOMMU API
>>>>>>> some sort of of_get_named_iommu() API doesn't really make sense.
>>>>>>
>>>>>> The thing is, if you end up ignoring the generic binding then we have two
>>>>>> IOMMUs using the same (ARM SMMU) binding and it begs the question as to
>>>>>> which is the more generic! I know we're keen to get this merged, but merging
>>>>>> something that people won't use and calling it generic doesn't seem ideal
>>>>>> either. We do, however, desperately need a generic binding.
>>>>>
>>>>> yeah, ignoring the generic binding is not my first choice.  I'd rather
>>>>> have something that works well for everyone.  But I wasn't really sure
>>>>> if the current proposal was arbitrary, or if there are some
>>>>> conflicting requirements between different platforms.
>>>>
>>>> The common case that needs to be simple is attaching one (master) device
>>>> to an IOMMU using the shared global context for the purposes of implementing
>>>> the dma-mapping API.
>>>
>>> well, I don't disagree that IOMMU API has some problems.  It is too
>>> tied to the bus type, which doesn't really seem to make sense for
>>> platform devices.  (Unless we start having multiple platform busses?)
>>>
>>> But at least given the current IOMMU API I'm not really sure how it
>>> makes a difference which way the link goes.  But if there has already
>>> been some discussion about how you want to handle the tie in with
>>> dma-mapping, if you could point me at that then maybe your point will
>>> make more sense to me.
>>
>> If you look at the proposed binding in isolation, I think it *is* cleaner
>> than the ARM SMMU binding (I've acked it...) and I believe it's more
>> consistent with the way we describe linkages elsewhere.
>>
>> However, the issue you're raising is that it's more difficult to make use of
>> in a Linux IOMMU driver. The reward you'll get for using it will come
>> eventually when the DMA ops are automatically swizzled for devices using the
>> generic binding.
>>
>> My plan for the ARM SMMU driver is:
>>
>>   (1) Change ->probe() to walk the device-tree looking for all masters with
>>       phandles back to the SMMU instance being probed
>>
>>   (2) For each master, extract the Stream IDs and add them to the internal
>>       SMMU driver data structures (an rbtree per SMMU instance). For
>>       hotpluggable buses, we'll need a way for the bus controller to
>>       reserve a range of IDs -- this will likely be a later extension to
>>       the binding.
>>
>>   (3) When we get an ->add() call, warn if it's a device we haven't seen
>>       and reject the addition.
>>
>> That way, ->attach() should be the same as it is now, I think.
>>
>> Have you tried implementing something like that? We agreed that (1) isn't
>> pretty, but I don't have a good alternative and it's only done at
>> probe-time.
> 
> I haven't tried implementing that yet, but I'm sure it would work.  I
> was just hoping to avoid having to do that ;-)

Is the reason you want to do it this way because you want to guarantee
that all masters (and stream IDs) have been identified before the first
attach call? I am just wondering why you cannot continue doing the
master/streamID discovery during add_device() callback?

>>
>> BTW: Is the msm-v0 IOMMU compatible with the ARM SMMU driver, or is it a
>> completely different design requiring a different driver?
> 
> My understanding is that it is different from msm v1 IOMMU, although I
> think it shares the same pagetable format with v1.  Not sure if that
> is the same as arm-smmu?   If so it might be nice to try to extract
> out some shared helper fxns for map/unmap as well.
> 
> I expect Olav knows better the similarities/differences.
> 

The msm-v0 IOMMU is not compatible with ARM SMMUv1 specification.
However, it is a close cousin. The hardware was designed before the ARM
SMMUv1 specification was available I believe. But it shares many of the
same concepts as the ARM SMMUv1.

msm-v0 IOMMU supports V7S page table format only. The ARM SMMU driver
does not support V7S at this time. However, I believe we need to support
this.

Will, this reminds me. We definitely have a need to use different page
tables in the ARM SMMU driver vs. the ARM CPU. We have an SoC with ARMv8
cores (and thus ARMv8 page tables) but the SMMUs (SMMUv1) on this SoC
only have support for V7S/V7L page table format. So we cannot use the
same page table format as the CPU.

Thanks,

Olav

-- 
The Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
hosted by The Linux Foundation

Re: [PATCH v4] devicetree: Add generic IOMMU device tree bindings

[Will Deacon]

On Wed, Jul 16, 2014 at 02:25:20AM +0100, Olav Haugan wrote:
> On 7/13/2014 4:43 AM, Rob Clark wrote:
> > On Sun, Jul 13, 2014 at 5:43 AM, Will Deacon <will.deacon@arm.com> wrote:
> >> My plan for the ARM SMMU driver is:
> >>
> >>   (1) Change ->probe() to walk the device-tree looking for all masters with
> >>       phandles back to the SMMU instance being probed
> >>
> >>   (2) For each master, extract the Stream IDs and add them to the internal
> >>       SMMU driver data structures (an rbtree per SMMU instance). For
> >>       hotpluggable buses, we'll need a way for the bus controller to
> >>       reserve a range of IDs -- this will likely be a later extension to
> >>       the binding.
> >>
> >>   (3) When we get an ->add() call, warn if it's a device we haven't seen
> >>       and reject the addition.
> >>
> >> That way, ->attach() should be the same as it is now, I think.
> >>
> >> Have you tried implementing something like that? We agreed that (1) isn't
> >> pretty, but I don't have a good alternative and it's only done at
> >> probe-time.
> > 
> > I haven't tried implementing that yet, but I'm sure it would work.  I
> > was just hoping to avoid having to do that ;-)
> 
> Is the reason you want to do it this way because you want to guarantee
> that all masters (and stream IDs) have been identified before the first
> attach call? I am just wondering why you cannot continue doing the
> master/streamID discovery during add_device() callback?

That's fine if we have one SMR per ID, but the moment we want to do more
involved matching, we're going to need complete system knowledge prior to
the first attach. I don't think we can safely reconfigure live streams on
the fly.

> >> BTW: Is the msm-v0 IOMMU compatible with the ARM SMMU driver, or is it a
> >> completely different design requiring a different driver?
> > 
> > My understanding is that it is different from msm v1 IOMMU, although I
> > think it shares the same pagetable format with v1.  Not sure if that
> > is the same as arm-smmu?   If so it might be nice to try to extract
> > out some shared helper fxns for map/unmap as well.
> > 
> > I expect Olav knows better the similarities/differences.
> > 
> 
> The msm-v0 IOMMU is not compatible with ARM SMMUv1 specification.
> However, it is a close cousin. The hardware was designed before the ARM
> SMMUv1 specification was available I believe. But it shares many of the
> same concepts as the ARM SMMUv1.
> 
> msm-v0 IOMMU supports V7S page table format only. The ARM SMMU driver
> does not support V7S at this time. However, I believe we need to support
> this.
> 
> Will, this reminds me. We definitely have a need to use different page
> tables in the ARM SMMU driver vs. the ARM CPU. We have an SoC with ARMv8
> cores (and thus ARMv8 page tables) but the SMMUs (SMMUv1) on this SoC
> only have support for V7S/V7L page table format. So we cannot use the
> same page table format as the CPU.

That sounds like a sane use-case. The best thing to do would be to add
some ARM page-table code as a library under drivers/iommu/, then we can
move the ARM IOMMUs over to using that. Since we'd be moving away from the
CPU page table helpers, we could also take the opportunity to use the
coherent DMA API instead of the hack I currently use for flushing out tables
to non-coherent walkers.

Will

Re: [PATCH v4] devicetree: Add generic IOMMU device tree bindings

[Rob Clark]

On Tue, Jul 15, 2014 at 9:25 PM, Olav Haugan <ohaugan@codeaurora.org> wrote:
> On 7/13/2014 4:43 AM, Rob Clark wrote:
>> On Sun, Jul 13, 2014 at 5:43 AM, Will Deacon <will.deacon@arm.com> wrote:
>>> On Sat, Jul 12, 2014 at 01:57:31PM +0100, Rob Clark wrote:
>>>> On Sat, Jul 12, 2014 at 8:22 AM, Arnd Bergmann <arnd@arndb.de> wrote:
>>>>> On Saturday 12 July 2014, Rob Clark wrote:
>>>>>>>> Was there actually a good reason for having the device link to the
>>>>>>>> iommu rather than the other way around?  How much would people hate it
>>>>>>>> if I just ignore the generic bindings and use something that works for
>>>>>>>> me instead.  I mean, it isn't exactly like there is going to be .dts
>>>>>>>> re-use across different SoC's..  and at least with current IOMMU API
>>>>>>>> some sort of of_get_named_iommu() API doesn't really make sense.
>>>>>>>
>>>>>>> The thing is, if you end up ignoring the generic binding then we have two
>>>>>>> IOMMUs using the same (ARM SMMU) binding and it begs the question as to
>>>>>>> which is the more generic! I know we're keen to get this merged, but merging
>>>>>>> something that people won't use and calling it generic doesn't seem ideal
>>>>>>> either. We do, however, desperately need a generic binding.
>>>>>>
>>>>>> yeah, ignoring the generic binding is not my first choice.  I'd rather
>>>>>> have something that works well for everyone.  But I wasn't really sure
>>>>>> if the current proposal was arbitrary, or if there are some
>>>>>> conflicting requirements between different platforms.
>>>>>
>>>>> The common case that needs to be simple is attaching one (master) device
>>>>> to an IOMMU using the shared global context for the purposes of implementing
>>>>> the dma-mapping API.
>>>>
>>>> well, I don't disagree that IOMMU API has some problems.  It is too
>>>> tied to the bus type, which doesn't really seem to make sense for
>>>> platform devices.  (Unless we start having multiple platform busses?)
>>>>
>>>> But at least given the current IOMMU API I'm not really sure how it
>>>> makes a difference which way the link goes.  But if there has already
>>>> been some discussion about how you want to handle the tie in with
>>>> dma-mapping, if you could point me at that then maybe your point will
>>>> make more sense to me.
>>>
>>> If you look at the proposed binding in isolation, I think it *is* cleaner
>>> than the ARM SMMU binding (I've acked it...) and I believe it's more
>>> consistent with the way we describe linkages elsewhere.
>>>
>>> However, the issue you're raising is that it's more difficult to make use of
>>> in a Linux IOMMU driver. The reward you'll get for using it will come
>>> eventually when the DMA ops are automatically swizzled for devices using the
>>> generic binding.
>>>
>>> My plan for the ARM SMMU driver is:
>>>
>>>   (1) Change ->probe() to walk the device-tree looking for all masters with
>>>       phandles back to the SMMU instance being probed
>>>
>>>   (2) For each master, extract the Stream IDs and add them to the internal
>>>       SMMU driver data structures (an rbtree per SMMU instance). For
>>>       hotpluggable buses, we'll need a way for the bus controller to
>>>       reserve a range of IDs -- this will likely be a later extension to
>>>       the binding.
>>>
>>>   (3) When we get an ->add() call, warn if it's a device we haven't seen
>>>       and reject the addition.
>>>
>>> That way, ->attach() should be the same as it is now, I think.
>>>
>>> Have you tried implementing something like that? We agreed that (1) isn't
>>> pretty, but I don't have a good alternative and it's only done at
>>> probe-time.
>>
>> I haven't tried implementing that yet, but I'm sure it would work.  I
>> was just hoping to avoid having to do that ;-)
>
> Is the reason you want to do it this way because you want to guarantee
> that all masters (and stream IDs) have been identified before the first
> attach call? I am just wondering why you cannot continue doing the
> master/streamID discovery during add_device() callback?

it was mostly because I couldn't think of a sane way to differentiate
between first and second time a device attaches (without keeping a
reference to the device).  But I guess it is ok to assume no hotplug
(since walking the device tree also seems acceptable)

BR,
-R

>>>
>>> BTW: Is the msm-v0 IOMMU compatible with the ARM SMMU driver, or is it a
>>> completely different design requiring a different driver?
>>
>> My understanding is that it is different from msm v1 IOMMU, although I
>> think it shares the same pagetable format with v1.  Not sure if that
>> is the same as arm-smmu?   If so it might be nice to try to extract
>> out some shared helper fxns for map/unmap as well.
>>
>> I expect Olav knows better the similarities/differences.
>>
>
> The msm-v0 IOMMU is not compatible with ARM SMMUv1 specification.
> However, it is a close cousin. The hardware was designed before the ARM
> SMMUv1 specification was available I believe. But it shares many of the
> same concepts as the ARM SMMUv1.
>
> msm-v0 IOMMU supports V7S page table format only. The ARM SMMU driver
> does not support V7S at this time. However, I believe we need to support
> this.
>
> Will, this reminds me. We definitely have a need to use different page
> tables in the ARM SMMU driver vs. the ARM CPU. We have an SoC with ARMv8
> cores (and thus ARMv8 page tables) but the SMMUs (SMMUv1) on this SoC
> only have support for V7S/V7L page table format. So we cannot use the
> same page table format as the CPU.
>
> Thanks,
>
> Olav
>
> --
> The Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
> hosted by The Linux Foundation

Re: [PATCH v4] devicetree: Add generic IOMMU device tree bindings

[Thierry Reding]

[-- Attachment #1: Type: text/plain, Size: 5187 bytes --]

On Sun, Jul 13, 2014 at 10:43:41AM +0100, Will Deacon wrote:
> On Sat, Jul 12, 2014 at 01:57:31PM +0100, Rob Clark wrote:
> > On Sat, Jul 12, 2014 at 8:22 AM, Arnd Bergmann <arnd@arndb.de> wrote:
> > > On Saturday 12 July 2014, Rob Clark wrote:
> > >> >> Was there actually a good reason for having the device link to the
> > >> >> iommu rather than the other way around?  How much would people hate it
> > >> >> if I just ignore the generic bindings and use something that works for
> > >> >> me instead.  I mean, it isn't exactly like there is going to be .dts
> > >> >> re-use across different SoC's..  and at least with current IOMMU API
> > >> >> some sort of of_get_named_iommu() API doesn't really make sense.
> > >> >
> > >> > The thing is, if you end up ignoring the generic binding then we have two
> > >> > IOMMUs using the same (ARM SMMU) binding and it begs the question as to
> > >> > which is the more generic! I know we're keen to get this merged, but merging
> > >> > something that people won't use and calling it generic doesn't seem ideal
> > >> > either. We do, however, desperately need a generic binding.
> > >>
> > >> yeah, ignoring the generic binding is not my first choice.  I'd rather
> > >> have something that works well for everyone.  But I wasn't really sure
> > >> if the current proposal was arbitrary, or if there are some
> > >> conflicting requirements between different platforms.
> > >
> > > The common case that needs to be simple is attaching one (master) device
> > > to an IOMMU using the shared global context for the purposes of implementing
> > > the dma-mapping API.
> > 
> > well, I don't disagree that IOMMU API has some problems.  It is too
> > tied to the bus type, which doesn't really seem to make sense for
> > platform devices.  (Unless we start having multiple platform busses?)
> > 
> > But at least given the current IOMMU API I'm not really sure how it
> > makes a difference which way the link goes.  But if there has already
> > been some discussion about how you want to handle the tie in with
> > dma-mapping, if you could point me at that then maybe your point will
> > make more sense to me.
> 
> If you look at the proposed binding in isolation, I think it *is* cleaner
> than the ARM SMMU binding (I've acked it...) and I believe it's more
> consistent with the way we describe linkages elsewhere.
> 
> However, the issue you're raising is that it's more difficult to make use of
> in a Linux IOMMU driver.

Like I said, if it's difficult to make use of this in a Linux IOMMU
driver then the right thing that we should be focusing on is enhancing
the IOMMU framework to better cope with this.

> The reward you'll get for using it will come eventually when the DMA
> ops are automatically swizzled for devices using the generic binding.

Note that for the case that Rob mentioned (and similarly for the primary
use-case that I have) the DMA integration layer isn't a good fit. So the
goal should be to make this easier to work with when using the DMA layer
*and* when using the IOMMU API directly.

> My plan for the ARM SMMU driver is:
> 
>   (1) Change ->probe() to walk the device-tree looking for all masters with
>       phandles back to the SMMU instance being probed

You and Rob mentioned this several times and I don't understand why the
SMMU needs to know all masters up front. Is this necessary because it
needs to program all registers at .probe() time and they can't be
reprogrammed subsequently? Or is this just some kind of optimization?

>   (2) For each master, extract the Stream IDs and add them to the internal
>       SMMU driver data structures (an rbtree per SMMU instance). For
>       hotpluggable buses, we'll need a way for the bus controller to
>       reserve a range of IDs -- this will likely be a later extension to
>       the binding.
> 
>   (3) When we get an ->add() call, warn if it's a device we haven't seen
>       and reject the addition.

It seems to me like this would be the logical place to parse stream IDs.
You could for example have a case where some device tree contains a node
for which no driver will ever be loaded (for example because it hasn't
been built-in, or the device is never used and the module is therefore
never loaded). That's a situation that you cannot determine by simply
walking the device tree in the IOMMU's .probe().

I've always thought about IOMMU masters much in the same way as other
types of resources, such as memory or interrupts. In the rest of the
kernel we do carefully try to postpone allocation of these resources
until they are required, specifically so we don't waste resources when
they're unused.

That's also one of the reasons why I think associating an IOMMU with the
bus type is bad. Currently if an IOMMU driver thinks it should enable
translation for a given device, then there's no way for that device's
driver to opt out again. There may be reasons (performance, hardware
bugs, ...) for the driver to decide against using the IOMMU for
translation, but there's currently no way to do that if the IOMMU driver
disagrees.

Thierry

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: [PATCH v4] devicetree: Add generic IOMMU device tree bindings

[Will Deacon]

Hi Thierry,

On Mon, Jul 14, 2014 at 07:44:53AM +0100, Thierry Reding wrote:
> On Sun, Jul 13, 2014 at 10:43:41AM +0100, Will Deacon wrote:
> > My plan for the ARM SMMU driver is:
> > 
> >   (1) Change ->probe() to walk the device-tree looking for all masters with
> >       phandles back to the SMMU instance being probed
> 
> You and Rob mentioned this several times and I don't understand why the
> SMMU needs to know all masters up front. Is this necessary because it
> needs to program all registers at .probe() time and they can't be
> reprogrammed subsequently? Or is this just some kind of optimization?

It's an optimization to reduce resource usage in the IOMMU, but one that
is *required* by certain platforms (e.g. Olav mentioned his 43-ID master
and Calxeda had something similar).

Basically, in order to perform an ->attach() for a master to a domain, we
need complete knowledge of the system so that we can avoid accidentally
attaching other masters to the same domain. The programming is done using
a form of StreamID wildcarding, so at this point we would need to have
parsed the entire DT to ensure our wildcard doesn't match other masters.

> >   (2) For each master, extract the Stream IDs and add them to the internal
> >       SMMU driver data structures (an rbtree per SMMU instance). For
> >       hotpluggable buses, we'll need a way for the bus controller to
> >       reserve a range of IDs -- this will likely be a later extension to
> >       the binding.
> > 
> >   (3) When we get an ->add() call, warn if it's a device we haven't seen
> >       and reject the addition.
> 
> It seems to me like this would be the logical place to parse stream IDs.

We could do that only if we were guaranteed to have an ->add() call for
*every* master before an ->attach() call for *any* master. I don't think
that is necessarily true.

> You could for example have a case where some device tree contains a node
> for which no driver will ever be loaded (for example because it hasn't
> been built-in, or the device is never used and the module is therefore
> never loaded). That's a situation that you cannot determine by simply
> walking the device tree in the IOMMU's .probe().

Why not? If we're simply searching for phandles to the IOMMU, why does it
matter whether a driver is bound to the master?

> I've always thought about IOMMU masters much in the same way as other
> types of resources, such as memory or interrupts. In the rest of the
> kernel we do carefully try to postpone allocation of these resources
> until they are required, specifically so we don't waste resources when
> they're unused.

See above, they are all required the moment anybody tries an ->attach().

> That's also one of the reasons why I think associating an IOMMU with the
> bus type is bad. Currently if an IOMMU driver thinks it should enable
> translation for a given device, then there's no way for that device's
> driver to opt out again. There may be reasons (performance, hardware
> bugs, ...) for the driver to decide against using the IOMMU for
> translation, but there's currently no way to do that if the IOMMU driver
> disagrees.

Yes, we need a way to associate an IOMMU with a bus instance, but I think
that's a separate topic, no?

Will

Re: [PATCH v4] devicetree: Add generic IOMMU device tree bindings

[Thierry Reding]

[-- Attachment #1: Type: text/plain, Size: 1609 bytes --]

On Sat, Jul 12, 2014 at 08:57:31AM -0400, Rob Clark wrote:
> On Sat, Jul 12, 2014 at 8:22 AM, Arnd Bergmann <arnd@arndb.de> wrote:
[...]
> > The way that Thierry's binding does that is the obvious solution to this,
> > and it mirrors what we do in practically every other subsystem. I definitely
> > want the SMMU to change before anybody starts using it in a real system,
> > which we fortunately do not have yet.
> 
> hmm, well if some of the things I need for (like this or batching
> mappings) are too weird and gpu specific, I'm willing to duplicate the
> IOMMU driver in drm/msm.  It really isn't so much code, and that gives
> me a lot more more flexibility to do crazy things... at some point I'm
> probably going to want to do context switches by banging the IOMMU
> registers directly from the gpu.

If the IOMMU API doesn't provide for what you need, then perhaps it's
time to enhance it? We do that all the time in other parts of the
kernel, why should IOMMU be special?

It seems to me like context switching for per-process address space
isolation is one of the important features of an IOMMU. If the current
API doesn't let you do that then we should think of ways how it can be
improved. And if it doesn't do it fast enough, then we should equally
find ways to speed it up.

This is part of why I think it would be good to have explicit objects
associated with IOMMU contexts. That would give us a good place to add
caching for this kind of situation. Currently we're required to handle
most of this in drivers (map from struct device to context, ...).

Thierry

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: [PATCH v4] devicetree: Add generic IOMMU device tree bindings

[Rob Clark]

On Mon, Jul 14, 2014 at 2:24 AM, Thierry Reding
<thierry.reding@gmail.com> wrote:
> On Sat, Jul 12, 2014 at 08:57:31AM -0400, Rob Clark wrote:
>> On Sat, Jul 12, 2014 at 8:22 AM, Arnd Bergmann <arnd@arndb.de> wrote:
> [...]
>> > The way that Thierry's binding does that is the obvious solution to this,
>> > and it mirrors what we do in practically every other subsystem. I definitely
>> > want the SMMU to change before anybody starts using it in a real system,
>> > which we fortunately do not have yet.
>>
>> hmm, well if some of the things I need for (like this or batching
>> mappings) are too weird and gpu specific, I'm willing to duplicate the
>> IOMMU driver in drm/msm.  It really isn't so much code, and that gives
>> me a lot more more flexibility to do crazy things... at some point I'm
>> probably going to want to do context switches by banging the IOMMU
>> registers directly from the gpu.
>
> If the IOMMU API doesn't provide for what you need, then perhaps it's
> time to enhance it? We do that all the time in other parts of the
> kernel, why should IOMMU be special?

sure.. and my comment was also about the map/unmap batching.

Bypassing IOMMU wouldn't be my first choice.  (Especially because I'd
then get to implement it twice.)  But if some of the things I need are
too specific to one driver (or worse, problematic for other IOMMU
use-cases which I don't know about), then it is an option I'd be
willing to consider.  If nothing else, it would get me out of
allocating sglists for every buffer..  I wonder how much memory
scatterlists take up for 500M of gfx buffers?

> It seems to me like context switching for per-process address space
> isolation is one of the important features of an IOMMU. If the current
> API doesn't let you do that then we should think of ways how it can be
> improved. And if it doesn't do it fast enough, then we should equally
> find ways to speed it up.
>
> This is part of why I think it would be good to have explicit objects
> associated with IOMMU contexts. That would give us a good place to add
> caching for this kind of situation. Currently we're required to handle
> most of this in drivers (map from struct device to context, ...).

well, it is at least awkward that the current api conflates attaching
device and attaching context.  I think we could get some use out of an
iommu_swap() API which conceptually acts as:

  iommu_swap(olddomain, newdomain, dev)
  {
     iommu_detach_device(olddomain, dev);
     iommu_attach_device(newdomain, dev);
  }

BR,
-R

> Thierry

Re: [PATCH v4] devicetree: Add generic IOMMU device tree bindings

[Thierry Reding]

[-- Attachment #1: Type: text/plain, Size: 2018 bytes --]

On Sat, Jul 12, 2014 at 02:22:01PM +0200, Arnd Bergmann wrote:
> On Saturday 12 July 2014, Rob Clark wrote:
> > >> Was there actually a good reason for having the device link to the
> > >> iommu rather than the other way around?  How much would people hate it
> > >> if I just ignore the generic bindings and use something that works for
> > >> me instead.  I mean, it isn't exactly like there is going to be .dts
> > >> re-use across different SoC's..  and at least with current IOMMU API
> > >> some sort of of_get_named_iommu() API doesn't really make sense.
> > >
> > > The thing is, if you end up ignoring the generic binding then we have two
> > > IOMMUs using the same (ARM SMMU) binding and it begs the question as to
> > > which is the more generic! I know we're keen to get this merged, but merging
> > > something that people won't use and calling it generic doesn't seem ideal
> > > either. We do, however, desperately need a generic binding.
> > 
> > yeah, ignoring the generic binding is not my first choice.  I'd rather
> > have something that works well for everyone.  But I wasn't really sure
> > if the current proposal was arbitrary, or if there are some
> > conflicting requirements between different platforms.
> 
> The common case that needs to be simple is attaching one (master) device
> to an IOMMU using the shared global context for the purposes of implementing
> the dma-mapping API.
> 
> The way that Thierry's binding does that is the obvious solution to this,
> and it mirrors what we do in practically every other subsystem.

That wasn't really the intention, though. We shouldn't be designing
bindings to work well in one use-case or another. My motivation for
doing it this way was that I think it naturally models the flow of
master IDs. They originate within the masters and flow towards the
IOMMU device. In other words, they are a property of the masters so
quite literally should be described in the device tree nodes of the
masters.

Thierry

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: [PATCH v4] devicetree: Add generic IOMMU device tree bindings

[Will Deacon]

Hi all,

On Fri, Jul 04, 2014 at 04:29:17PM +0100, Thierry Reding wrote:
> From: Thierry Reding <treding@nvidia.com>
> 
> This commit introduces a generic device tree binding for IOMMU devices.
> Only a very minimal subset is described here, but it is enough to cover
> the requirements of both the Exynos System MMU and Tegra SMMU as
> discussed here:
> 
>     https://lkml.org/lkml/2014/4/27/346
> 
> Signed-off-by: Thierry Reding <treding@nvidia.com>
> ---
> Changes in v4:
> - clarify that disabling an IOMMU DT node may not disable translation
> - be more explicit that examples are only examples
> - add multi-ID master example
> 
> Changes in v3:
> - use #iommu-cells instead of #address-cells/#size-cells
> - drop optional iommu-names property
> 
> Changes in v2:
> - add notes about "dma-ranges" property (drop note from commit message)
> - document priorities of "iommus" property vs. "dma-ranges" property
> - drop #iommu-cells in favour of #address-cells and #size-cells
> - remove multiple-master device example
> 
>  Documentation/devicetree/bindings/iommu/iommu.txt | 172 ++++++++++++++++++++++
>  1 file changed, 172 insertions(+)
>  create mode 100644 Documentation/devicetree/bindings/iommu/iommu.txt

I'm concerned that this patch hasn't been picked up for 3.17 (I can't see it
in -next). If we want to move the ARM SMMU driver over to this new binding,
we can't keep dragging our feet for much longer as I *really* don't plan to
support two bindings in parallel (one is complicated enough already).

Any chance we can see this merged, please?

Will

Re: [PATCH v4] devicetree: Add generic IOMMU device tree bindings

[Thierry Reding]

[-- Attachment #1: Type: text/plain, Size: 2164 bytes --]

On Wed, Jul 30, 2014 at 12:04:25PM +0100, Will Deacon wrote:
> Hi all,
> 
> On Fri, Jul 04, 2014 at 04:29:17PM +0100, Thierry Reding wrote:
> > From: Thierry Reding <treding@nvidia.com>
> > 
> > This commit introduces a generic device tree binding for IOMMU devices.
> > Only a very minimal subset is described here, but it is enough to cover
> > the requirements of both the Exynos System MMU and Tegra SMMU as
> > discussed here:
> > 
> >     https://lkml.org/lkml/2014/4/27/346
> > 
> > Signed-off-by: Thierry Reding <treding@nvidia.com>
> > ---
> > Changes in v4:
> > - clarify that disabling an IOMMU DT node may not disable translation
> > - be more explicit that examples are only examples
> > - add multi-ID master example
> > 
> > Changes in v3:
> > - use #iommu-cells instead of #address-cells/#size-cells
> > - drop optional iommu-names property
> > 
> > Changes in v2:
> > - add notes about "dma-ranges" property (drop note from commit message)
> > - document priorities of "iommus" property vs. "dma-ranges" property
> > - drop #iommu-cells in favour of #address-cells and #size-cells
> > - remove multiple-master device example
> > 
> >  Documentation/devicetree/bindings/iommu/iommu.txt | 172 ++++++++++++++++++++++
> >  1 file changed, 172 insertions(+)
> >  create mode 100644 Documentation/devicetree/bindings/iommu/iommu.txt
> 
> I'm concerned that this patch hasn't been picked up for 3.17 (I can't see it
> in -next). If we want to move the ARM SMMU driver over to this new binding,
> we can't keep dragging our feet for much longer as I *really* don't plan to
> support two bindings in parallel (one is complicated enough already).
> 
> Any chance we can see this merged, please?

I think there weren't any comments left for me to address and I've
mostly been waiting for Joerg to pick it up.

Joerg, can you take this through the iommu tree for 3.17? Will acked
this, but perhaps you were waiting for an ACK from the device tree
bindings maintainers?

Will, perhaps you can get Pawel or Mark to look at this?

Arnd, I'm sure if we had your Acked-by that would go a long way too.

Thierry

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: [PATCH v4] devicetree: Add generic IOMMU device tree bindings

[Joerg Roedel]

On Wed, Jul 30, 2014 at 03:23:50PM +0200, Thierry Reding wrote:
> I think there weren't any comments left for me to address and I've
> mostly been waiting for Joerg to pick it up.
> 
> Joerg, can you take this through the iommu tree for 3.17? Will acked
> this, but perhaps you were waiting for an ACK from the device tree
> bindings maintainers?
> 
> Will, perhaps you can get Pawel or Mark to look at this?
> 
> Arnd, I'm sure if we had your Acked-by that would go a long way too.

Yes, as Arnd requested this generic binding it would be good to have his
Acked-by before proceeding. Arnd?


	Joerg

Re: [PATCH v4] devicetree: Add generic IOMMU device tree bindings

[Olof Johansson]

On Wed, Jul 30, 2014 at 6:33 AM, Joerg Roedel <joro@8bytes.org> wrote:
> On Wed, Jul 30, 2014 at 03:23:50PM +0200, Thierry Reding wrote:
>> I think there weren't any comments left for me to address and I've
>> mostly been waiting for Joerg to pick it up.
>>
>> Joerg, can you take this through the iommu tree for 3.17? Will acked
>> this, but perhaps you were waiting for an ACK from the device tree
>> bindings maintainers?
>>
>> Will, perhaps you can get Pawel or Mark to look at this?
>>
>> Arnd, I'm sure if we had your Acked-by that would go a long way too.
>
> Yes, as Arnd requested this generic binding it would be good to have his
> Acked-by before proceeding. Arnd?

Arnd is on vacation now, unfortunately. I've read up on the history
and the current proposal from Thierry looks sane to me. As discussed,
there might be a need for some common helpers to walk the tree and
figure out things.

There's also the possibility that the IOMMU at init time actually
modifies and configures the stream IDs for the clients, but that again
is an implementation detail and not something that affects the binding
per se.

I'm not giving my Ack now since I want to make sure there are no
disagreements on my separate reply from a minute ago, but I expect
you'll have it once we've had that round trip of comments. :)


-Olof

Re: [PATCH v4] devicetree: Add generic IOMMU device tree bindings

[Will Deacon]

On Wed, Jul 30, 2014 at 02:23:50PM +0100, Thierry Reding wrote:
> On Wed, Jul 30, 2014 at 12:04:25PM +0100, Will Deacon wrote:
> > On Fri, Jul 04, 2014 at 04:29:17PM +0100, Thierry Reding wrote:
> > > From: Thierry Reding <treding@nvidia.com>
> > > 
> > > This commit introduces a generic device tree binding for IOMMU devices.
> > > Only a very minimal subset is described here, but it is enough to cover
> > > the requirements of both the Exynos System MMU and Tegra SMMU as
> > > discussed here:
> > > 
> > >     https://lkml.org/lkml/2014/4/27/346
> > > 
> > > Signed-off-by: Thierry Reding <treding@nvidia.com>
> > > ---
> > > Changes in v4:
> > > - clarify that disabling an IOMMU DT node may not disable translation
> > > - be more explicit that examples are only examples
> > > - add multi-ID master example
> > > 
> > > Changes in v3:
> > > - use #iommu-cells instead of #address-cells/#size-cells
> > > - drop optional iommu-names property
> > > 
> > > Changes in v2:
> > > - add notes about "dma-ranges" property (drop note from commit message)
> > > - document priorities of "iommus" property vs. "dma-ranges" property
> > > - drop #iommu-cells in favour of #address-cells and #size-cells
> > > - remove multiple-master device example
> > > 
> > >  Documentation/devicetree/bindings/iommu/iommu.txt | 172 ++++++++++++++++++++++
> > >  1 file changed, 172 insertions(+)
> > >  create mode 100644 Documentation/devicetree/bindings/iommu/iommu.txt
> > 
> > I'm concerned that this patch hasn't been picked up for 3.17 (I can't see it
> > in -next). If we want to move the ARM SMMU driver over to this new binding,
> > we can't keep dragging our feet for much longer as I *really* don't plan to
> > support two bindings in parallel (one is complicated enough already).
> > 
> > Any chance we can see this merged, please?
> 
> I think there weren't any comments left for me to address and I've
> mostly been waiting for Joerg to pick it up.
> 
> Joerg, can you take this through the iommu tree for 3.17? Will acked
> this, but perhaps you were waiting for an ACK from the device tree
> bindings maintainers?

Rob, Mark: can one or both of you take a look at this please?

Cheers,

Will

Re: [PATCH v4] devicetree: Add generic IOMMU device tree bindings

[Rob Herring]

On Wed, Jul 30, 2014 at 9:30 AM, Will Deacon <will.deacon@arm.com> wrote:
> On Wed, Jul 30, 2014 at 02:23:50PM +0100, Thierry Reding wrote:
>> On Wed, Jul 30, 2014 at 12:04:25PM +0100, Will Deacon wrote:
>> > On Fri, Jul 04, 2014 at 04:29:17PM +0100, Thierry Reding wrote:
>> > > From: Thierry Reding <treding@nvidia.com>
>> > >
>> > > This commit introduces a generic device tree binding for IOMMU devices.
>> > > Only a very minimal subset is described here, but it is enough to cover
>> > > the requirements of both the Exynos System MMU and Tegra SMMU as
>> > > discussed here:
>> > >
>> > >     https://lkml.org/lkml/2014/4/27/346
>> > >
>> > > Signed-off-by: Thierry Reding <treding@nvidia.com>
>> > > ---
>> > > Changes in v4:
>> > > - clarify that disabling an IOMMU DT node may not disable translation
>> > > - be more explicit that examples are only examples
>> > > - add multi-ID master example
>> > >
>> > > Changes in v3:
>> > > - use #iommu-cells instead of #address-cells/#size-cells
>> > > - drop optional iommu-names property
>> > >
>> > > Changes in v2:
>> > > - add notes about "dma-ranges" property (drop note from commit message)
>> > > - document priorities of "iommus" property vs. "dma-ranges" property
>> > > - drop #iommu-cells in favour of #address-cells and #size-cells
>> > > - remove multiple-master device example
>> > >
>> > >  Documentation/devicetree/bindings/iommu/iommu.txt | 172 ++++++++++++++++++++++
>> > >  1 file changed, 172 insertions(+)
>> > >  create mode 100644 Documentation/devicetree/bindings/iommu/iommu.txt
>> >
>> > I'm concerned that this patch hasn't been picked up for 3.17 (I can't see it
>> > in -next). If we want to move the ARM SMMU driver over to this new binding,
>> > we can't keep dragging our feet for much longer as I *really* don't plan to
>> > support two bindings in parallel (one is complicated enough already).
>> >
>> > Any chance we can see this merged, please?
>>
>> I think there weren't any comments left for me to address and I've
>> mostly been waiting for Joerg to pick it up.
>>
>> Joerg, can you take this through the iommu tree for 3.17? Will acked
>> this, but perhaps you were waiting for an ACK from the device tree
>> bindings maintainers?
>
> Rob, Mark: can one or both of you take a look at this please?

I've been quiet on this round, but I think prior input I've had has
been addressed. If we believe this will work for ARM SMMU and MSM
IOMMU and some of the crazy chaining scenarios, then I'm fine with the
binding.

Acked-by: Rob Herring <robh@kernel.org>

Rob

P.S. Thankfully, there are no Calxeda systems with the SMMU enabled,
so a binding change should not cause much pain.

Re: [PATCH v4] devicetree: Add generic IOMMU device tree bindings

[Arnd Bergmann]

On Wednesday 30 July 2014, Thierry Reding wrote:
se?
> 
> I think there weren't any comments left for me to address and I've
> mostly been waiting for Joerg to pick it up.
> 
> Joerg, can you take this through the iommu tree for 3.17? Will acked
> this, but perhaps you were waiting for an ACK from the device tree
> bindings maintainers?
> 
> Will, perhaps you can get Pawel or Mark to look at this?
> 
> Arnd, I'm sure if we had your Acked-by that would go a long way too.


Sorry for missing this before my vacation.

Reviewed-by: Arnd Bergmann <arnd@arndb.de>

Olof, please merge it into arm-soc so we can finally build on top
of this!

	Arnd

Re: [PATCH v4] devicetree: Add generic IOMMU device tree bindings

[Mark Rutland]

Hi Thierry,

This looks sane to me.

I just have a few questions below which are hopefully simple/stupid.

On Fri, Jul 04, 2014 at 04:29:17PM +0100, Thierry Reding wrote:
> From: Thierry Reding <treding@nvidia.com>
> 
> This commit introduces a generic device tree binding for IOMMU devices.
> Only a very minimal subset is described here, but it is enough to cover
> the requirements of both the Exynos System MMU and Tegra SMMU as
> discussed here:
> 
>     https://lkml.org/lkml/2014/4/27/346
> 
> Signed-off-by: Thierry Reding <treding@nvidia.com>
> ---
> Changes in v4:
> - clarify that disabling an IOMMU DT node may not disable translation
> - be more explicit that examples are only examples
> - add multi-ID master example
> 
> Changes in v3:
> - use #iommu-cells instead of #address-cells/#size-cells
> - drop optional iommu-names property
> 
> Changes in v2:
> - add notes about "dma-ranges" property (drop note from commit message)
> - document priorities of "iommus" property vs. "dma-ranges" property
> - drop #iommu-cells in favour of #address-cells and #size-cells
> - remove multiple-master device example
> 
>  Documentation/devicetree/bindings/iommu/iommu.txt | 172 ++++++++++++++++++++++
>  1 file changed, 172 insertions(+)
>  create mode 100644 Documentation/devicetree/bindings/iommu/iommu.txt
> 
> diff --git a/Documentation/devicetree/bindings/iommu/iommu.txt b/Documentation/devicetree/bindings/iommu/iommu.txt
> new file mode 100644
> index 000000000000..464a81eaaf61
> --- /dev/null
> +++ b/Documentation/devicetree/bindings/iommu/iommu.txt
> @@ -0,0 +1,172 @@
> +This document describes the generic device tree binding for IOMMUs and their
> +master(s).
> +
> +
> +IOMMU device node:
> +==================
> +
> +An IOMMU can provide the following services:
> +
> +* Remap address space to allow devices to access physical memory ranges that
> +  they otherwise wouldn't be capable of accessing.
> +
> +  Example: 32-bit DMA to 64-bit physical addresses
> +
> +* Implement scatter-gather at page level granularity so that the device does
> +  not have to.
> +
> +* Provide system protection against "rogue" DMA by forcing all accesses to go
> +  through the IOMMU and faulting when encountering accesses to unmapped
> +  address regions.
> +
> +* Provide address space isolation between multiple contexts.
> +
> +  Example: Virtualization
> +
> +Device nodes compatible with this binding represent hardware with some of the
> +above capabilities.
> +
> +IOMMUs can be single-master or multiple-master. Single-master IOMMU devices
> +typically have a fixed association to the master device, whereas multiple-
> +master IOMMU devices can translate accesses from more than one master.
> +
> +The device tree node of the IOMMU device's parent bus must contain a valid
> +"dma-ranges" property that describes how the physical address space of the
> +IOMMU maps to memory. An empty "dma-ranges" property means that there is a
> +1:1 mapping from IOMMU to memory.
> +
> +Required properties:
> +--------------------
> +- #iommu-cells: The number of cells in an IOMMU specifier needed to encode an
> +  address.
> +
> +The meaning of the IOMMU specifier is defined by the device tree binding of
> +the specific IOMMU. Below are a few examples of typical use-cases:
> +
> +- #iommu-cells = <0>: Single master IOMMU devices are not configurable and
> +  therefore no additional information needs to be encoded in the specifier.
> +  This may also apply to multiple master IOMMU devices that do not allow the
> +  association of masters to be configured. Note that an IOMMU can by design
> +  be multi-master yet only expose a single master in a given configuration.
> +  In such cases the number of cells will usually be 1 as in the next case.
> +- #iommu-cells = <1>: Multiple master IOMMU devices may need to be configured
> +  in order to enable translation for a given master. In such cases the single
> +  address cell corresponds to the master device's ID. In some cases more than
> +  one cell can be required to represent a single master ID.
> +- #iommu-cells = <4>: Some IOMMU devices allow the DMA window for masters to
> +  be configured. The first cell of the address in this may contain the master
> +  device's ID for example, while the second cell could contain the start of
> +  the DMA window for the given device. The length of the DMA window is given
> +  by the third and fourth cells.
> +
> +Note that these are merely examples and real-world use-cases may use different
> +definitions to represent their individual needs. Always refer to the specific
> +IOMMU binding for the exact meaning of the cells that make up the specifier.
> +
> +
> +IOMMU master node:
> +==================
> +
> +Devices that access memory through an IOMMU are called masters. A device can
> +have multiple master interfaces (to one or more IOMMU devices).
> +
> +Required properties:
> +--------------------
> +- iommus: A list of phandle and IOMMU specifier pairs that describe the IOMMU
> +  master interfaces of the device. One entry in the list describes one master
> +  interface of the device.
> +
> +When an "iommus" property is specified in a device tree node, the IOMMU will
> +be used for address translation. If a "dma-ranges" property exists in the
> +device's parent node it will be ignored. An exception to this rule is if the
> +referenced IOMMU is disabled, in which case the "dma-ranges" property of the
> +parent shall take effect. Note that merely disabling a device tree node does
> +not guarantee that the IOMMU is really disabled since the hardware may not
> +have a means to turn off translation.
> +
> +
> +Notes:
> +======
> +
> +One possible extension to the above is to use an "iommus" property along with
> +a "dma-ranges" property in a bus device node (such as PCI host bridges). This
> +can be useful to describe how children on the bus relate to the IOMMU if they
> +are not explicitly listed in the device tree (e.g. PCI devices). However, the
> +requirements of that use-case haven't been fully determined yet. Implementing
> +this is therefore not recommended without further discussion and extension of
> +this binding.
> +
> +
> +Examples:
> +=========
> +
> +Single-master IOMMU:
> +--------------------
> +
> +	iommu {
> +		#iommu-cells = <0>;
> +	};
> +
> +	master {
> +		iommus = <&/iommu>;

Nit: this should be iommus = <&{/iommu}>, or it's not valid dts syntax.

> +	};
> +
> +Multiple-master IOMMU with fixed associations:
> +----------------------------------------------
> +
> +	/* multiple-master IOMMU */
> +	iommu {
> +		/*
> +		 * Masters are statically associated with this IOMMU and
> +		 * address translation is always enabled.
> +		 */
> +		#iommu-cells = <0>;

I don't follow why translation being always enabled is relevant to the
example; that would seem to be independent from the binding.

Surely the key point is that with no way to distinguish devices, they
presumably share the same translations?

> +	};
> +
> +	/* static association with IOMMU */
> +	master@1 {
> +		reg = <1>;
> +		iommus = <&/iommu>;
> +	};
> +
> +	/* static association with IOMMU */
> +	master@2 {
> +		reg = <2>;
> +		iommus = <&/iommu>;
> +	};
> +
> +Multiple-master IOMMU:
> +----------------------
> +
> +	iommu {
> +		/* the specifier represents the ID of the master */
> +		#iommu-cells = <1>;
> +	};
> +
> +	master@1 {
> +		/* device has master ID 42 in the IOMMU */
> +		iommus = <&/iommu 42>;
> +	};
> +
> +	master@2 {
> +		/* device has master IDs 23 and 24 in the IOMMU */
> +		iommus = <&/iommu 23>, <&/iommu 24>;
> +	};

In future I suspect master will need to be able to identify which master
IDs correspond to which of their master ports (where each port might
have an arbitrary number of master IDs).

While we don't need that for the first run, it would be nice to have
that looked into so master bindings don't come up with arbitrarily
different ways of doing that.

> +
> +Multiple-master IOMMU with configurable DMA window:
> +---------------------------------------------------
> +
> +	/ {
> +		#address-cells = <1>;
> +		#size-cells = <1>;
> +
> +		iommu {
> +			/* master ID, address and length of DMA window */
> +			#iommu-cells = <4>;
> +		};
> +
> +		master {
> +			/* master ID 42, 4 GiB DMA window starting at 0 */
> +			iommus = <&/iommu  42  0  0x1 0x0>;

Is this that window is from the POV of the master, i.e. the master can
address 0x0 to 0xffffffff when generating transactions, and these get
translated somehow?

Or is this the physical addresses to allocate to the master?

Cheers,
Mark.

Re: [PATCH v4] devicetree: Add generic IOMMU device tree bindings

[Olof Johansson]

Hi,

On Wed, Jul 30, 2014 at 8:26 AM, Mark Rutland <mark.rutland@arm.com> wrote:
> Hi Thierry,
>
> This looks sane to me.
>
> I just have a few questions below which are hopefully simple/stupid.
>
> On Fri, Jul 04, 2014 at 04:29:17PM +0100, Thierry Reding wrote:
>> From: Thierry Reding <treding@nvidia.com>
>>
>> This commit introduces a generic device tree binding for IOMMU devices.
>> Only a very minimal subset is described here, but it is enough to cover
>> the requirements of both the Exynos System MMU and Tegra SMMU as
>> discussed here:
>>
>>     https://lkml.org/lkml/2014/4/27/346
>>
>> Signed-off-by: Thierry Reding <treding@nvidia.com>
>> ---
>> Changes in v4:
>> - clarify that disabling an IOMMU DT node may not disable translation
>> - be more explicit that examples are only examples
>> - add multi-ID master example
>>
>> Changes in v3:
>> - use #iommu-cells instead of #address-cells/#size-cells
>> - drop optional iommu-names property
>>
>> Changes in v2:
>> - add notes about "dma-ranges" property (drop note from commit message)
>> - document priorities of "iommus" property vs. "dma-ranges" property
>> - drop #iommu-cells in favour of #address-cells and #size-cells
>> - remove multiple-master device example
>>
>>  Documentation/devicetree/bindings/iommu/iommu.txt | 172 ++++++++++++++++++++++
>>  1 file changed, 172 insertions(+)
>>  create mode 100644 Documentation/devicetree/bindings/iommu/iommu.txt
>>
>> diff --git a/Documentation/devicetree/bindings/iommu/iommu.txt b/Documentation/devicetree/bindings/iommu/iommu.txt
>> new file mode 100644
>> index 000000000000..464a81eaaf61
>> --- /dev/null
>> +++ b/Documentation/devicetree/bindings/iommu/iommu.txt
>> @@ -0,0 +1,172 @@
>> +This document describes the generic device tree binding for IOMMUs and their
>> +master(s).
>> +
>> +
>> +IOMMU device node:
>> +==================
>> +
>> +An IOMMU can provide the following services:
>> +
>> +* Remap address space to allow devices to access physical memory ranges that
>> +  they otherwise wouldn't be capable of accessing.
>> +
>> +  Example: 32-bit DMA to 64-bit physical addresses
>> +
>> +* Implement scatter-gather at page level granularity so that the device does
>> +  not have to.
>> +
>> +* Provide system protection against "rogue" DMA by forcing all accesses to go
>> +  through the IOMMU and faulting when encountering accesses to unmapped
>> +  address regions.
>> +
>> +* Provide address space isolation between multiple contexts.
>> +
>> +  Example: Virtualization
>> +
>> +Device nodes compatible with this binding represent hardware with some of the
>> +above capabilities.
>> +
>> +IOMMUs can be single-master or multiple-master. Single-master IOMMU devices
>> +typically have a fixed association to the master device, whereas multiple-
>> +master IOMMU devices can translate accesses from more than one master.
>> +
>> +The device tree node of the IOMMU device's parent bus must contain a valid
>> +"dma-ranges" property that describes how the physical address space of the
>> +IOMMU maps to memory. An empty "dma-ranges" property means that there is a
>> +1:1 mapping from IOMMU to memory.
>> +
>> +Required properties:
>> +--------------------
>> +- #iommu-cells: The number of cells in an IOMMU specifier needed to encode an
>> +  address.
>> +
>> +The meaning of the IOMMU specifier is defined by the device tree binding of
>> +the specific IOMMU. Below are a few examples of typical use-cases:
>> +
>> +- #iommu-cells = <0>: Single master IOMMU devices are not configurable and
>> +  therefore no additional information needs to be encoded in the specifier.
>> +  This may also apply to multiple master IOMMU devices that do not allow the
>> +  association of masters to be configured. Note that an IOMMU can by design
>> +  be multi-master yet only expose a single master in a given configuration.
>> +  In such cases the number of cells will usually be 1 as in the next case.
>> +- #iommu-cells = <1>: Multiple master IOMMU devices may need to be configured
>> +  in order to enable translation for a given master. In such cases the single
>> +  address cell corresponds to the master device's ID. In some cases more than
>> +  one cell can be required to represent a single master ID.
>> +- #iommu-cells = <4>: Some IOMMU devices allow the DMA window for masters to
>> +  be configured. The first cell of the address in this may contain the master
>> +  device's ID for example, while the second cell could contain the start of
>> +  the DMA window for the given device. The length of the DMA window is given
>> +  by the third and fourth cells.
>> +
>> +Note that these are merely examples and real-world use-cases may use different
>> +definitions to represent their individual needs. Always refer to the specific
>> +IOMMU binding for the exact meaning of the cells that make up the specifier.
>> +
>> +
>> +IOMMU master node:
>> +==================
>> +
>> +Devices that access memory through an IOMMU are called masters. A device can
>> +have multiple master interfaces (to one or more IOMMU devices).
>> +
>> +Required properties:
>> +--------------------
>> +- iommus: A list of phandle and IOMMU specifier pairs that describe the IOMMU
>> +  master interfaces of the device. One entry in the list describes one master
>> +  interface of the device.
>> +
>> +When an "iommus" property is specified in a device tree node, the IOMMU will
>> +be used for address translation. If a "dma-ranges" property exists in the
>> +device's parent node it will be ignored. An exception to this rule is if the
>> +referenced IOMMU is disabled, in which case the "dma-ranges" property of the
>> +parent shall take effect. Note that merely disabling a device tree node does
>> +not guarantee that the IOMMU is really disabled since the hardware may not
>> +have a means to turn off translation.

How do you know if the IOMMU is disabled then? Runtime by checking
status of the iommu?

>> +
>> +
>> +Notes:
>> +======
>> +
>> +One possible extension to the above is to use an "iommus" property along with
>> +a "dma-ranges" property in a bus device node (such as PCI host bridges). This
>> +can be useful to describe how children on the bus relate to the IOMMU if they
>> +are not explicitly listed in the device tree (e.g. PCI devices). However, the
>> +requirements of that use-case haven't been fully determined yet. Implementing
>> +this is therefore not recommended without further discussion and extension of
>> +this binding.
>> +
>> +
>> +Examples:
>> +=========
>> +
>> +Single-master IOMMU:
>> +--------------------
>> +
>> +     iommu {
>> +             #iommu-cells = <0>;
>> +     };
>> +
>> +     master {
>> +             iommus = <&/iommu>;
>
> Nit: this should be iommus = <&{/iommu}>, or it's not valid dts syntax.
>
>> +     };
>> +
>> +Multiple-master IOMMU with fixed associations:
>> +----------------------------------------------
>> +
>> +     /* multiple-master IOMMU */
>> +     iommu {
>> +             /*
>> +              * Masters are statically associated with this IOMMU and
>> +              * address translation is always enabled.
>> +              */
>> +             #iommu-cells = <0>;
>
> I don't follow why translation being always enabled is relevant to the
> example; that would seem to be independent from the binding.
>
> Surely the key point is that with no way to distinguish devices, they
> presumably share the same translations?
>
>> +     };
>> +
>> +     /* static association with IOMMU */
>> +     master@1 {
>> +             reg = <1>;
>> +             iommus = <&/iommu>;
>> +     };
>> +
>> +     /* static association with IOMMU */
>> +     master@2 {
>> +             reg = <2>;
>> +             iommus = <&/iommu>;
>> +     };
>> +
>> +Multiple-master IOMMU:
>> +----------------------
>> +
>> +     iommu {
>> +             /* the specifier represents the ID of the master */
>> +             #iommu-cells = <1>;
>> +     };
>> +
>> +     master@1 {
>> +             /* device has master ID 42 in the IOMMU */
>> +             iommus = <&/iommu 42>;
>> +     };
>> +
>> +     master@2 {
>> +             /* device has master IDs 23 and 24 in the IOMMU */
>> +             iommus = <&/iommu 23>, <&/iommu 24>;
>> +     };
>
> In future I suspect master will need to be able to identify which master
> IDs correspond to which of their master ports (where each port might
> have an arbitrary number of master IDs).
>
> While we don't need that for the first run, it would be nice to have
> that looked into so master bindings don't come up with arbitrarily
> different ways of doing that.

iommu-names would be the logical extension to handle that, just like
we do with other resources, right?

>> +
>> +Multiple-master IOMMU with configurable DMA window:
>> +---------------------------------------------------
>> +
>> +     / {
>> +             #address-cells = <1>;
>> +             #size-cells = <1>;
>> +
>> +             iommu {
>> +                     /* master ID, address and length of DMA window */
>> +                     #iommu-cells = <4>;
>> +             };
>> +
>> +             master {
>> +                     /* master ID 42, 4 GiB DMA window starting at 0 */
>> +                     iommus = <&/iommu  42  0  0x1 0x0>;
>
> Is this that window is from the POV of the master, i.e. the master can
> address 0x0 to 0xffffffff when generating transactions, and these get
> translated somehow?
>
> Or is this the physical addresses to allocate to the master?

It needs to be clarified in the documentation, but as far as I know it
is the DMA address space that is used.

It is somewhat confusing to have size-cells = 1 and then use 2 cells
for size in the iommu property. It's legal and expected, but having
size-cells in the example adds a little confusion.

Either way, I'm OK with fixing the above with an incremental patch,
assuming there is no disagreements on what's said above.


-Olof

Re: [PATCH v4] devicetree: Add generic IOMMU device tree bindings

[Mark Rutland]

[...]

> >> +Multiple-master IOMMU:
> >> +----------------------
> >> +
> >> +     iommu {
> >> +             /* the specifier represents the ID of the master */
> >> +             #iommu-cells = <1>;
> >> +     };
> >> +
> >> +     master@1 {
> >> +             /* device has master ID 42 in the IOMMU */
> >> +             iommus = <&/iommu 42>;
> >> +     };
> >> +
> >> +     master@2 {
> >> +             /* device has master IDs 23 and 24 in the IOMMU */
> >> +             iommus = <&/iommu 23>, <&/iommu 24>;
> >> +     };
> >
> > In future I suspect master will need to be able to identify which master
> > IDs correspond to which of their master ports (where each port might
> > have an arbitrary number of master IDs).
> >
> > While we don't need that for the first run, it would be nice to have
> > that looked into so master bindings don't come up with arbitrarily
> > different ways of doing that.
> 
> iommu-names would be the logical extension to handle that, just like
> we do with other resources, right?

Possibly. If the master has multiple IDs assigned to transactions from a
single master port then it depends on how the master wants to group
those for the sake of the binding. If it's per-port then you'd need the
same name multiple times:

iommus = <&iommu 0>, <&iommu 4>, <&iommu 17>, <&iommu 25>;
iommu-names = "video", "video", "dram", dram";

This is really specific to a given master, so we can table that until
the first master appears which needs to distinguish between IDs.

> >> +
> >> +Multiple-master IOMMU with configurable DMA window:
> >> +---------------------------------------------------
> >> +
> >> +     / {
> >> +             #address-cells = <1>;
> >> +             #size-cells = <1>;
> >> +
> >> +             iommu {
> >> +                     /* master ID, address and length of DMA window */
> >> +                     #iommu-cells = <4>;
> >> +             };
> >> +
> >> +             master {
> >> +                     /* master ID 42, 4 GiB DMA window starting at 0 */
> >> +                     iommus = <&/iommu  42  0  0x1 0x0>;
> >
> > Is this that window is from the POV of the master, i.e. the master can
> > address 0x0 to 0xffffffff when generating transactions, and these get
> > translated somehow?
> >
> > Or is this the physical addresses to allocate to the master?
> 
> It needs to be clarified in the documentation, but as far as I know it
> is the DMA address space that is used.

Ok. So that's pre-translation, from the POV of the master?

If we don't have that knowledge about the master already (e.g. based on
the compatible string), surely we always need that information in a
given iommu-specifier format? Otherwise certain iommus won't be able to
handle masters with limited addressing only due to limitations of their
binding.

> It is somewhat confusing to have size-cells = 1 and then use 2 cells
> for size in the iommu property. It's legal and expected, but having
> size-cells in the example adds a little confusion.
> 
> Either way, I'm OK with fixing the above with an incremental patch,
> assuming there is no disagreements on what's said above.

I like the general idea.

Given my concerns are to do with implementation details I'm happy to
have this go through and fix it up as the first implementations of the
binding take shape.

Thanks,
Mark.

Re: [PATCH v4] devicetree: Add generic IOMMU device tree bindings

[Thierry Reding]

[-- Attachment #1: Type: text/plain, Size: 2212 bytes --]

On Wed, Jul 30, 2014 at 07:18:42PM +0100, Mark Rutland wrote:
[...]
> > >> +
> > >> +Multiple-master IOMMU with configurable DMA window:
> > >> +---------------------------------------------------
> > >> +
> > >> +     / {
> > >> +             #address-cells = <1>;
> > >> +             #size-cells = <1>;
> > >> +
> > >> +             iommu {
> > >> +                     /* master ID, address and length of DMA window */
> > >> +                     #iommu-cells = <4>;
> > >> +             };
> > >> +
> > >> +             master {
> > >> +                     /* master ID 42, 4 GiB DMA window starting at 0 */
> > >> +                     iommus = <&/iommu  42  0  0x1 0x0>;
> > >
> > > Is this that window is from the POV of the master, i.e. the master can
> > > address 0x0 to 0xffffffff when generating transactions, and these get
> > > translated somehow?
> > >
> > > Or is this the physical addresses to allocate to the master?
> > 
> > It needs to be clarified in the documentation, but as far as I know it
> > is the DMA address space that is used.
> 
> Ok. So that's pre-translation, from the POV of the master?

Correct. It represents the window of the IOMMU's addressable I/O virtual
address space that should be assigned to this particular master.

> If we don't have that knowledge about the master already (e.g. based on
> the compatible string), surely we always need that information in a
> given iommu-specifier format? Otherwise certain iommus won't be able to
> handle masters with limited addressing only due to limitations of their
> binding.

This is only used for what's often called a windowed IOMMU. Many IOMMUs
(non-windowed) typically allow only a complete address space to be
assigned to a master without additional control over subregions. So this
is really a property/capability of the IOMMU rather than the masters
themselves.

There are already other means to respect the addressing limitations of
masters. We typcially use a device's DMA mask for this, and it's natural
to reuse that for I/O virtual addresses since they will in fact take the
place of physical addresses for the master when translation is enabled.

Thierry

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: [PATCH v4] devicetree: Add generic IOMMU device tree bindings

[Mark Rutland]

On Thu, Jul 31, 2014 at 11:09:06AM +0100, Thierry Reding wrote:
> On Wed, Jul 30, 2014 at 07:18:42PM +0100, Mark Rutland wrote:
> [...]
> > > >> +
> > > >> +Multiple-master IOMMU with configurable DMA window:
> > > >> +---------------------------------------------------
> > > >> +
> > > >> +     / {
> > > >> +             #address-cells = <1>;
> > > >> +             #size-cells = <1>;
> > > >> +
> > > >> +             iommu {
> > > >> +                     /* master ID, address and length of DMA window */
> > > >> +                     #iommu-cells = <4>;
> > > >> +             };
> > > >> +
> > > >> +             master {
> > > >> +                     /* master ID 42, 4 GiB DMA window starting at 0 */
> > > >> +                     iommus = <&/iommu  42  0  0x1 0x0>;
> > > >
> > > > Is this that window is from the POV of the master, i.e. the master can
> > > > address 0x0 to 0xffffffff when generating transactions, and these get
> > > > translated somehow?
> > > >
> > > > Or is this the physical addresses to allocate to the master?
> > > 
> > > It needs to be clarified in the documentation, but as far as I know it
> > > is the DMA address space that is used.
> > 
> > Ok. So that's pre-translation, from the POV of the master?
> 
> Correct. It represents the window of the IOMMU's addressable I/O virtual
> address space that should be assigned to this particular master.
> 
> > If we don't have that knowledge about the master already (e.g. based on
> > the compatible string), surely we always need that information in a
> > given iommu-specifier format? Otherwise certain iommus won't be able to
> > handle masters with limited addressing only due to limitations of their
> > binding.
> 
> This is only used for what's often called a windowed IOMMU. Many IOMMUs
> (non-windowed) typically allow only a complete address space to be
> assigned to a master without additional control over subregions. So this
> is really a property/capability of the IOMMU rather than the masters
> themselves.

I'm not sure I follow, but I'm happy to wait until we have the first
windowed IOMMU using this binding. I'll try to get myself up to speed in
the mean time.

> There are already other means to respect the addressing limitations of
> masters. We typcially use a device's DMA mask for this, and it's natural
> to reuse that for I/O virtual addresses since they will in fact take the
> place of physical addresses for the master when translation is enabled.

Ok, that covers my worry then.

Cheers,
Mark.

Re: [PATCH v4] devicetree: Add generic IOMMU device tree bindings

[Thierry Reding]

[-- Attachment #1: Type: text/plain, Size: 3842 bytes --]

On Thu, Jul 31, 2014 at 11:50:17AM +0100, Mark Rutland wrote:
> On Thu, Jul 31, 2014 at 11:09:06AM +0100, Thierry Reding wrote:
> > On Wed, Jul 30, 2014 at 07:18:42PM +0100, Mark Rutland wrote:
> > [...]
> > > > >> +
> > > > >> +Multiple-master IOMMU with configurable DMA window:
> > > > >> +---------------------------------------------------
> > > > >> +
> > > > >> +     / {
> > > > >> +             #address-cells = <1>;
> > > > >> +             #size-cells = <1>;
> > > > >> +
> > > > >> +             iommu {
> > > > >> +                     /* master ID, address and length of DMA window */
> > > > >> +                     #iommu-cells = <4>;
> > > > >> +             };
> > > > >> +
> > > > >> +             master {
> > > > >> +                     /* master ID 42, 4 GiB DMA window starting at 0 */
> > > > >> +                     iommus = <&/iommu  42  0  0x1 0x0>;
> > > > >
> > > > > Is this that window is from the POV of the master, i.e. the master can
> > > > > address 0x0 to 0xffffffff when generating transactions, and these get
> > > > > translated somehow?
> > > > >
> > > > > Or is this the physical addresses to allocate to the master?
> > > > 
> > > > It needs to be clarified in the documentation, but as far as I know it
> > > > is the DMA address space that is used.
> > > 
> > > Ok. So that's pre-translation, from the POV of the master?
> > 
> > Correct. It represents the window of the IOMMU's addressable I/O virtual
> > address space that should be assigned to this particular master.
> > 
> > > If we don't have that knowledge about the master already (e.g. based on
> > > the compatible string), surely we always need that information in a
> > > given iommu-specifier format? Otherwise certain iommus won't be able to
> > > handle masters with limited addressing only due to limitations of their
> > > binding.
> > 
> > This is only used for what's often called a windowed IOMMU. Many IOMMUs
> > (non-windowed) typically allow only a complete address space to be
> > assigned to a master without additional control over subregions. So this
> > is really a property/capability of the IOMMU rather than the masters
> > themselves.
> 
> I'm not sure I follow, but I'm happy to wait until we have the first
> windowed IOMMU using this binding. I'll try to get myself up to speed in
> the mean time.

As I understand it, a windowed IOMMU manages a given I/O virtual address
space (only one or perhaps even several). Each such address space is the
complete range that the IOMMU can take as inputs from any master. For
purposes of virtualization and process separation it can subdivide this
address space into subranges, so that each context can only access that
given range of virtual I/O addresses. I suspect that this works by
setting up a mapping between that range and the context's master ID(s).
And I also suppose it could be possible for the DMA windows to be truly
configurable within the IOMMU or for specific devices to be assigned a
fixed window.

Simpler IOMMUs (Tegra uses one of those for example) know only address
spaces. That is each address space can be assigned to one or more
masters. But each master can always access the whole address space and
accesses cannot be restricted to subregions thereof.

So from a memory protection point of view the difference is that for
non-windowed IOMMUs translations will fault only if no mapping has been
set up for the I/O virtual address being accessed, whereas for windowed
IOMMUs translations can in addition also fault if they access an I/O
virtual addresses outside of the range that they've been assigned.

Does that help? Note that I've never dealt with windowed IOMMUs myself,
so this is largely based on what I scooped up in previous discussions
with Arnd.

Thierry

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: [PATCH v4] devicetree: Add generic IOMMU device tree bindings

[Thierry Reding]

[-- Attachment #1: Type: text/plain, Size: 11177 bytes --]

On Wed, Jul 30, 2014 at 10:35:06AM -0700, Olof Johansson wrote:
> Hi,
> 
> On Wed, Jul 30, 2014 at 8:26 AM, Mark Rutland <mark.rutland@arm.com> wrote:
> > Hi Thierry,
> >
> > This looks sane to me.
> >
> > I just have a few questions below which are hopefully simple/stupid.
> >
> > On Fri, Jul 04, 2014 at 04:29:17PM +0100, Thierry Reding wrote:
> >> From: Thierry Reding <treding@nvidia.com>
> >>
> >> This commit introduces a generic device tree binding for IOMMU devices.
> >> Only a very minimal subset is described here, but it is enough to cover
> >> the requirements of both the Exynos System MMU and Tegra SMMU as
> >> discussed here:
> >>
> >>     https://lkml.org/lkml/2014/4/27/346
> >>
> >> Signed-off-by: Thierry Reding <treding@nvidia.com>
> >> ---
> >> Changes in v4:
> >> - clarify that disabling an IOMMU DT node may not disable translation
> >> - be more explicit that examples are only examples
> >> - add multi-ID master example
> >>
> >> Changes in v3:
> >> - use #iommu-cells instead of #address-cells/#size-cells
> >> - drop optional iommu-names property
> >>
> >> Changes in v2:
> >> - add notes about "dma-ranges" property (drop note from commit message)
> >> - document priorities of "iommus" property vs. "dma-ranges" property
> >> - drop #iommu-cells in favour of #address-cells and #size-cells
> >> - remove multiple-master device example
> >>
> >>  Documentation/devicetree/bindings/iommu/iommu.txt | 172 ++++++++++++++++++++++
> >>  1 file changed, 172 insertions(+)
> >>  create mode 100644 Documentation/devicetree/bindings/iommu/iommu.txt
> >>
> >> diff --git a/Documentation/devicetree/bindings/iommu/iommu.txt b/Documentation/devicetree/bindings/iommu/iommu.txt
> >> new file mode 100644
> >> index 000000000000..464a81eaaf61
> >> --- /dev/null
> >> +++ b/Documentation/devicetree/bindings/iommu/iommu.txt
> >> @@ -0,0 +1,172 @@
> >> +This document describes the generic device tree binding for IOMMUs and their
> >> +master(s).
> >> +
> >> +
> >> +IOMMU device node:
> >> +==================
> >> +
> >> +An IOMMU can provide the following services:
> >> +
> >> +* Remap address space to allow devices to access physical memory ranges that
> >> +  they otherwise wouldn't be capable of accessing.
> >> +
> >> +  Example: 32-bit DMA to 64-bit physical addresses
> >> +
> >> +* Implement scatter-gather at page level granularity so that the device does
> >> +  not have to.
> >> +
> >> +* Provide system protection against "rogue" DMA by forcing all accesses to go
> >> +  through the IOMMU and faulting when encountering accesses to unmapped
> >> +  address regions.
> >> +
> >> +* Provide address space isolation between multiple contexts.
> >> +
> >> +  Example: Virtualization
> >> +
> >> +Device nodes compatible with this binding represent hardware with some of the
> >> +above capabilities.
> >> +
> >> +IOMMUs can be single-master or multiple-master. Single-master IOMMU devices
> >> +typically have a fixed association to the master device, whereas multiple-
> >> +master IOMMU devices can translate accesses from more than one master.
> >> +
> >> +The device tree node of the IOMMU device's parent bus must contain a valid
> >> +"dma-ranges" property that describes how the physical address space of the
> >> +IOMMU maps to memory. An empty "dma-ranges" property means that there is a
> >> +1:1 mapping from IOMMU to memory.
> >> +
> >> +Required properties:
> >> +--------------------
> >> +- #iommu-cells: The number of cells in an IOMMU specifier needed to encode an
> >> +  address.
> >> +
> >> +The meaning of the IOMMU specifier is defined by the device tree binding of
> >> +the specific IOMMU. Below are a few examples of typical use-cases:
> >> +
> >> +- #iommu-cells = <0>: Single master IOMMU devices are not configurable and
> >> +  therefore no additional information needs to be encoded in the specifier.
> >> +  This may also apply to multiple master IOMMU devices that do not allow the
> >> +  association of masters to be configured. Note that an IOMMU can by design
> >> +  be multi-master yet only expose a single master in a given configuration.
> >> +  In such cases the number of cells will usually be 1 as in the next case.
> >> +- #iommu-cells = <1>: Multiple master IOMMU devices may need to be configured
> >> +  in order to enable translation for a given master. In such cases the single
> >> +  address cell corresponds to the master device's ID. In some cases more than
> >> +  one cell can be required to represent a single master ID.
> >> +- #iommu-cells = <4>: Some IOMMU devices allow the DMA window for masters to
> >> +  be configured. The first cell of the address in this may contain the master
> >> +  device's ID for example, while the second cell could contain the start of
> >> +  the DMA window for the given device. The length of the DMA window is given
> >> +  by the third and fourth cells.
> >> +
> >> +Note that these are merely examples and real-world use-cases may use different
> >> +definitions to represent their individual needs. Always refer to the specific
> >> +IOMMU binding for the exact meaning of the cells that make up the specifier.
> >> +
> >> +
> >> +IOMMU master node:
> >> +==================
> >> +
> >> +Devices that access memory through an IOMMU are called masters. A device can
> >> +have multiple master interfaces (to one or more IOMMU devices).
> >> +
> >> +Required properties:
> >> +--------------------
> >> +- iommus: A list of phandle and IOMMU specifier pairs that describe the IOMMU
> >> +  master interfaces of the device. One entry in the list describes one master
> >> +  interface of the device.
> >> +
> >> +When an "iommus" property is specified in a device tree node, the IOMMU will
> >> +be used for address translation. If a "dma-ranges" property exists in the
> >> +device's parent node it will be ignored. An exception to this rule is if the
> >> +referenced IOMMU is disabled, in which case the "dma-ranges" property of the
> >> +parent shall take effect. Note that merely disabling a device tree node does
> >> +not guarantee that the IOMMU is really disabled since the hardware may not
> >> +have a means to turn off translation.
> 
> How do you know if the IOMMU is disabled then? Runtime by checking
> status of the iommu?

I don't think you can really know that it is disabled. If it's disabled
in device tree then you don't have a way of querying it either. This is
probably an issue that needs to be resolved at integration time.

Generally I'd think that if an IOMMU is capable of enable or disabling
translation it will be disabled by default for all masters. Therefore if
the device tree node is status = "disabled", then there's no driver that
will set it up and enable translation, so the IOMMU can be considered to
be disabled. If an IOMMU has translations always enabled with no means
to disable them, then does it make sense to set the status = "disabled"
on the IOMMU? I think not. No driver would be setting up the mappings
within the IOMMU and translations would always fail. For IOMMUs that
have no way to enable translations... well... they wouldn't be IOMMUs.

So I think we should either drop that last sentence or be more explicit
in that disabling an IOMMU device (via device tree) that can't turn off
address translation is invalid:

	Note that merely disabling a device tree node does not guarantee
	that the IOMMU is really disabled since the hardware may not
	have a means to turn off translation. But it is invalid in such
	cases to disable the IOMMU's device tree node in the first place
	because doing so would prevent any driver from properly setting
	up the translations.

> >> +Multiple-master IOMMU:
> >> +----------------------
> >> +
> >> +     iommu {
> >> +             /* the specifier represents the ID of the master */
> >> +             #iommu-cells = <1>;
> >> +     };
> >> +
> >> +     master@1 {
> >> +             /* device has master ID 42 in the IOMMU */
> >> +             iommus = <&/iommu 42>;
> >> +     };
> >> +
> >> +     master@2 {
> >> +             /* device has master IDs 23 and 24 in the IOMMU */
> >> +             iommus = <&/iommu 23>, <&/iommu 24>;
> >> +     };
> >
> > In future I suspect master will need to be able to identify which master
> > IDs correspond to which of their master ports (where each port might
> > have an arbitrary number of master IDs).
> >
> > While we don't need that for the first run, it would be nice to have
> > that looked into so master bindings don't come up with arbitrarily
> > different ways of doing that.
> 
> iommu-names would be the logical extension to handle that, just like
> we do with other resources, right?

iommu-names used to be in the bindings as an optional property until v3
where it was dropped upon request by Rob and Arnd on the grounds of
being overengineered by modeling the binding after that of other
subsystems.

But I still think that if we ever want to support multiple-master IOMMUs
then an iommu-names property will be required to tell them apart.

Either that or bindings will have to specify a fixed ordering.

> >> +Multiple-master IOMMU with configurable DMA window:
> >> +---------------------------------------------------
> >> +
> >> +     / {
> >> +             #address-cells = <1>;
> >> +             #size-cells = <1>;
> >> +
> >> +             iommu {
> >> +                     /* master ID, address and length of DMA window */
> >> +                     #iommu-cells = <4>;
> >> +             };
> >> +
> >> +             master {
> >> +                     /* master ID 42, 4 GiB DMA window starting at 0 */
> >> +                     iommus = <&/iommu  42  0  0x1 0x0>;
> >
> > Is this that window is from the POV of the master, i.e. the master can
> > address 0x0 to 0xffffffff when generating transactions, and these get
> > translated somehow?
> >
> > Or is this the physical addresses to allocate to the master?
> 
> It needs to be clarified in the documentation, but as far as I know it
> is the DMA address space that is used.
> 
> It is somewhat confusing to have size-cells = 1 and then use 2 cells
> for size in the iommu property. It's legal and expected, but having
> size-cells in the example adds a little confusion.

I've changed the above to say:

	/ {
		iommu {
			/*
			 * One cell for the master ID and one cell for the
			 * address of the DMA window. The length of the DMA
			 * window is encoded in two cells.
			 *
			 * The DMA window is the range addressable by the
			 * master (i.e. the I/O virtual address space).
			 */
			#iommu-cells = <4>;
		};

		master {
			/* master ID 42, 4 GiB DMA window starting at 0 */
			iommus = <&{/iommu}  42  0  0x1 0x0>;
		};
	};

The #size-cells = <1> was a remainder of an earlier example that didn't
use #iommu-cells. #address-cells and #size-cells are irrelevant for the
example, so removing them makes sense (and reduces confusion).

Does that look better?

Thierry

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: [PATCH v4] devicetree: Add generic IOMMU device tree bindings

[Thierry Reding]

[-- Attachment #1: Type: text/plain, Size: 8331 bytes --]

On Wed, Jul 30, 2014 at 04:26:47PM +0100, Mark Rutland wrote:
> Hi Thierry,
> 
> This looks sane to me.
> 
> I just have a few questions below which are hopefully simple/stupid.
> 
> On Fri, Jul 04, 2014 at 04:29:17PM +0100, Thierry Reding wrote:
> > From: Thierry Reding <treding@nvidia.com>
> > 
> > This commit introduces a generic device tree binding for IOMMU devices.
> > Only a very minimal subset is described here, but it is enough to cover
> > the requirements of both the Exynos System MMU and Tegra SMMU as
> > discussed here:
> > 
> >     https://lkml.org/lkml/2014/4/27/346
> > 
> > Signed-off-by: Thierry Reding <treding@nvidia.com>
> > ---
> > Changes in v4:
> > - clarify that disabling an IOMMU DT node may not disable translation
> > - be more explicit that examples are only examples
> > - add multi-ID master example
> > 
> > Changes in v3:
> > - use #iommu-cells instead of #address-cells/#size-cells
> > - drop optional iommu-names property
> > 
> > Changes in v2:
> > - add notes about "dma-ranges" property (drop note from commit message)
> > - document priorities of "iommus" property vs. "dma-ranges" property
> > - drop #iommu-cells in favour of #address-cells and #size-cells
> > - remove multiple-master device example
> > 
> >  Documentation/devicetree/bindings/iommu/iommu.txt | 172 ++++++++++++++++++++++
> >  1 file changed, 172 insertions(+)
> >  create mode 100644 Documentation/devicetree/bindings/iommu/iommu.txt
> > 
> > diff --git a/Documentation/devicetree/bindings/iommu/iommu.txt b/Documentation/devicetree/bindings/iommu/iommu.txt
> > new file mode 100644
> > index 000000000000..464a81eaaf61
> > --- /dev/null
> > +++ b/Documentation/devicetree/bindings/iommu/iommu.txt
> > @@ -0,0 +1,172 @@
> > +This document describes the generic device tree binding for IOMMUs and their
> > +master(s).
> > +
> > +
> > +IOMMU device node:
> > +==================
> > +
> > +An IOMMU can provide the following services:
> > +
> > +* Remap address space to allow devices to access physical memory ranges that
> > +  they otherwise wouldn't be capable of accessing.
> > +
> > +  Example: 32-bit DMA to 64-bit physical addresses
> > +
> > +* Implement scatter-gather at page level granularity so that the device does
> > +  not have to.
> > +
> > +* Provide system protection against "rogue" DMA by forcing all accesses to go
> > +  through the IOMMU and faulting when encountering accesses to unmapped
> > +  address regions.
> > +
> > +* Provide address space isolation between multiple contexts.
> > +
> > +  Example: Virtualization
> > +
> > +Device nodes compatible with this binding represent hardware with some of the
> > +above capabilities.
> > +
> > +IOMMUs can be single-master or multiple-master. Single-master IOMMU devices
> > +typically have a fixed association to the master device, whereas multiple-
> > +master IOMMU devices can translate accesses from more than one master.
> > +
> > +The device tree node of the IOMMU device's parent bus must contain a valid
> > +"dma-ranges" property that describes how the physical address space of the
> > +IOMMU maps to memory. An empty "dma-ranges" property means that there is a
> > +1:1 mapping from IOMMU to memory.
> > +
> > +Required properties:
> > +--------------------
> > +- #iommu-cells: The number of cells in an IOMMU specifier needed to encode an
> > +  address.
> > +
> > +The meaning of the IOMMU specifier is defined by the device tree binding of
> > +the specific IOMMU. Below are a few examples of typical use-cases:
> > +
> > +- #iommu-cells = <0>: Single master IOMMU devices are not configurable and
> > +  therefore no additional information needs to be encoded in the specifier.
> > +  This may also apply to multiple master IOMMU devices that do not allow the
> > +  association of masters to be configured. Note that an IOMMU can by design
> > +  be multi-master yet only expose a single master in a given configuration.
> > +  In such cases the number of cells will usually be 1 as in the next case.
> > +- #iommu-cells = <1>: Multiple master IOMMU devices may need to be configured
> > +  in order to enable translation for a given master. In such cases the single
> > +  address cell corresponds to the master device's ID. In some cases more than
> > +  one cell can be required to represent a single master ID.
> > +- #iommu-cells = <4>: Some IOMMU devices allow the DMA window for masters to
> > +  be configured. The first cell of the address in this may contain the master
> > +  device's ID for example, while the second cell could contain the start of
> > +  the DMA window for the given device. The length of the DMA window is given
> > +  by the third and fourth cells.
> > +
> > +Note that these are merely examples and real-world use-cases may use different
> > +definitions to represent their individual needs. Always refer to the specific
> > +IOMMU binding for the exact meaning of the cells that make up the specifier.
> > +
> > +
> > +IOMMU master node:
> > +==================
> > +
> > +Devices that access memory through an IOMMU are called masters. A device can
> > +have multiple master interfaces (to one or more IOMMU devices).
> > +
> > +Required properties:
> > +--------------------
> > +- iommus: A list of phandle and IOMMU specifier pairs that describe the IOMMU
> > +  master interfaces of the device. One entry in the list describes one master
> > +  interface of the device.
> > +
> > +When an "iommus" property is specified in a device tree node, the IOMMU will
> > +be used for address translation. If a "dma-ranges" property exists in the
> > +device's parent node it will be ignored. An exception to this rule is if the
> > +referenced IOMMU is disabled, in which case the "dma-ranges" property of the
> > +parent shall take effect. Note that merely disabling a device tree node does
> > +not guarantee that the IOMMU is really disabled since the hardware may not
> > +have a means to turn off translation.
> > +
> > +
> > +Notes:
> > +======
> > +
> > +One possible extension to the above is to use an "iommus" property along with
> > +a "dma-ranges" property in a bus device node (such as PCI host bridges). This
> > +can be useful to describe how children on the bus relate to the IOMMU if they
> > +are not explicitly listed in the device tree (e.g. PCI devices). However, the
> > +requirements of that use-case haven't been fully determined yet. Implementing
> > +this is therefore not recommended without further discussion and extension of
> > +this binding.
> > +
> > +
> > +Examples:
> > +=========
> > +
> > +Single-master IOMMU:
> > +--------------------
> > +
> > +	iommu {
> > +		#iommu-cells = <0>;
> > +	};
> > +
> > +	master {
> > +		iommus = <&/iommu>;
> 
> Nit: this should be iommus = <&{/iommu}>, or it's not valid dts syntax.

Done.

> > +	};
> > +
> > +Multiple-master IOMMU with fixed associations:
> > +----------------------------------------------
> > +
> > +	/* multiple-master IOMMU */
> > +	iommu {
> > +		/*
> > +		 * Masters are statically associated with this IOMMU and
> > +		 * address translation is always enabled.
> > +		 */
> > +		#iommu-cells = <0>;
> 
> I don't follow why translation being always enabled is relevant to the
> example; that would seem to be independent from the binding.
> 
> Surely the key point is that with no way to distinguish devices, they
> presumably share the same translations?

Both aspects are important I think. For #iommu-cells = <0> there is no
way for the IOMMU driver to know how to enable translation for a given
device. So it must be either always on or always off.

I guess one could say that this is implicit if all masters share the
same translations. And I guess translations don't always have to be on
or off technically. Let me try to rephrase this:

		/*
		 * Masters are statically associated with this IOMMU and share
		 * the same address translations because the IOMMU does not
		 * have sufficient information to distinguish between masters.
		 *
		 * Consequently address translation is always on or off for
		 * all masters at any given point in time.
		 */

Does that sound better?

Thierry

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: [PATCH v4] devicetree: Add generic IOMMU device tree bindings

[Mark Rutland]

[...]

> > > +Examples:
> > > +=========
> > > +
> > > +Single-master IOMMU:
> > > +--------------------
> > > +
> > > +	iommu {
> > > +		#iommu-cells = <0>;
> > > +	};
> > > +
> > > +	master {
> > > +		iommus = <&/iommu>;
> > 
> > Nit: this should be iommus = <&{/iommu}>, or it's not valid dts syntax.
> 
> Done.

Cheers. I take it that was done for the other occurrences too?

> 
> > > +	};
> > > +
> > > +Multiple-master IOMMU with fixed associations:
> > > +----------------------------------------------
> > > +
> > > +	/* multiple-master IOMMU */
> > > +	iommu {
> > > +		/*
> > > +		 * Masters are statically associated with this IOMMU and
> > > +		 * address translation is always enabled.
> > > +		 */
> > > +		#iommu-cells = <0>;
> > 
> > I don't follow why translation being always enabled is relevant to the
> > example; that would seem to be independent from the binding.
> > 
> > Surely the key point is that with no way to distinguish devices, they
> > presumably share the same translations?
> 
> Both aspects are important I think. For #iommu-cells = <0> there is no
> way for the IOMMU driver to know how to enable translation for a given
> device. So it must be either always on or always off.

Sure. But "always on or off" is not the same as "always enabled", which
was what confused me.

> I guess one could say that this is implicit if all masters share the
> same translations. And I guess translations don't always have to be on
> or off technically. Let me try to rephrase this:
> 
> 		/*
> 		 * Masters are statically associated with this IOMMU and share
> 		 * the same address translations because the IOMMU does not
> 		 * have sufficient information to distinguish between masters.
> 		 *
> 		 * Consequently address translation is always on or off for
> 		 * all masters at any given point in time.
> 		 */
> 
> Does that sound better?

That addresses my concern, so yes.

Given these are minor and everyone wants this in now, I'm happy for
these to go through in a fixup patch later.

Cheers,
Mark.

Re: [PATCH v4] devicetree: Add generic IOMMU device tree bindings

[Thierry Reding]

[-- Attachment #1: Type: text/plain, Size: 2733 bytes --]

On Thu, Jul 31, 2014 at 10:22:41AM +0100, Mark Rutland wrote:
> [...]
> 
> > > > +Examples:
> > > > +=========
> > > > +
> > > > +Single-master IOMMU:
> > > > +--------------------
> > > > +
> > > > +	iommu {
> > > > +		#iommu-cells = <0>;
> > > > +	};
> > > > +
> > > > +	master {
> > > > +		iommus = <&/iommu>;
> > > 
> > > Nit: this should be iommus = <&{/iommu}>, or it's not valid dts syntax.
> > 
> > Done.
> 
> Cheers. I take it that was done for the other occurrences too?

Of course. =)

> > > > +	};
> > > > +
> > > > +Multiple-master IOMMU with fixed associations:
> > > > +----------------------------------------------
> > > > +
> > > > +	/* multiple-master IOMMU */
> > > > +	iommu {
> > > > +		/*
> > > > +		 * Masters are statically associated with this IOMMU and
> > > > +		 * address translation is always enabled.
> > > > +		 */
> > > > +		#iommu-cells = <0>;
> > > 
> > > I don't follow why translation being always enabled is relevant to the
> > > example; that would seem to be independent from the binding.
> > > 
> > > Surely the key point is that with no way to distinguish devices, they
> > > presumably share the same translations?
> > 
> > Both aspects are important I think. For #iommu-cells = <0> there is no
> > way for the IOMMU driver to know how to enable translation for a given
> > device. So it must be either always on or always off.
> 
> Sure. But "always on or off" is not the same as "always enabled", which
> was what confused me.

Yes, this was indeed awkwardly formulated. I think the point that I was
trying to get across was that there could be IOMMUs that are always on,
with no means to disable translations at all. But since that's now
mentioned in the "Notes:" section that Olof commented on I think we have
that covered as well.

> 
> > I guess one could say that this is implicit if all masters share the
> > same translations. And I guess translations don't always have to be on
> > or off technically. Let me try to rephrase this:
> > 
> > 		/*
> > 		 * Masters are statically associated with this IOMMU and share
> > 		 * the same address translations because the IOMMU does not
> > 		 * have sufficient information to distinguish between masters.
> > 		 *
> > 		 * Consequently address translation is always on or off for
> > 		 * all masters at any given point in time.
> > 		 */
> > 
> > Does that sound better?
> 
> That addresses my concern, so yes.
> 
> Given these are minor and everyone wants this in now, I'm happy for
> these to go through in a fixup patch later.

It looks like this hasn't been applied yet, so I can send out a v5
shortly with the requested changes addressed.

Thierry

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: [PATCH v4] devicetree: Add generic IOMMU device tree bindings

[Joerg Roedel]

On Thu, Jul 31, 2014 at 12:18:08PM +0200, Thierry Reding wrote:
> It looks like this hasn't been applied yet, so I can send out a v5
> shortly with the requested changes addressed.

Yes, please send a v5 with the requested changes and all Reviewed-bys
and Acked-bys this got so far. I'll take it into my tree then if nobody
else objects.


	Joerg

Re: [PATCH v4] devicetree: Add generic IOMMU device tree bindings

[Thierry Reding]

[-- Attachment #1.1: Type: text/plain, Size: 654 bytes --]

On Thu, Jul 31, 2014 at 12:23:51PM +0200, Joerg Roedel wrote:
> On Thu, Jul 31, 2014 at 12:18:08PM +0200, Thierry Reding wrote:
> > It looks like this hasn't been applied yet, so I can send out a v5
> > shortly with the requested changes addressed.
> 
> Yes, please send a v5 with the requested changes and all Reviewed-bys
> and Acked-bys this got so far. I'll take it into my tree then if nobody
> else objects.

I just sent out v5, although I trimmed the Cc list a little since it was
growing somewhat long. But I've kept everyone that gave feedback on v4
as well as the lists.

For reference I've attached the diff since v4.

Thierry

[-- Attachment #1.2: v5.patch --]
[-- Type: text/x-diff, Size: 3033 bytes --]

diff --git a/Documentation/devicetree/bindings/iommu/iommu.txt b/Documentation/devicetree/bindings/iommu/iommu.txt
index 464a81eaaf61..5a8b4624defc 100644
--- a/Documentation/devicetree/bindings/iommu/iommu.txt
+++ b/Documentation/devicetree/bindings/iommu/iommu.txt
@@ -82,7 +82,9 @@ device's parent node it will be ignored. An exception to this rule is if the
 referenced IOMMU is disabled, in which case the "dma-ranges" property of the
 parent shall take effect. Note that merely disabling a device tree node does
 not guarantee that the IOMMU is really disabled since the hardware may not
-have a means to turn off translation.
+have a means to turn off translation. But it is invalid in such cases to
+disable the IOMMU's device tree node in the first place because it would
+prevent any driver from properly setting up the translations.
 
 
 Notes:
@@ -108,7 +110,7 @@ Single-master IOMMU:
 	};
 
 	master {
-		iommus = <&/iommu>;
+		iommus = <&{/iommu}>;
 	};
 
 Multiple-master IOMMU with fixed associations:
@@ -117,8 +119,12 @@ Multiple-master IOMMU with fixed associations:
 	/* multiple-master IOMMU */
 	iommu {
 		/*
-		 * Masters are statically associated with this IOMMU and
-		 * address translation is always enabled.
+		 * Masters are statically associated with this IOMMU and share
+		 * the same address translations because the IOMMU does not
+		 * have sufficient information to distinguish between masters.
+		 *
+		 * Consequently address translation is always on or off for
+		 * all masters at any given point in time.
 		 */
 		#iommu-cells = <0>;
 	};
@@ -126,13 +132,13 @@ Multiple-master IOMMU with fixed associations:
 	/* static association with IOMMU */
 	master@1 {
 		reg = <1>;
-		iommus = <&/iommu>;
+		iommus = <&{/iommu}>;
 	};
 
 	/* static association with IOMMU */
 	master@2 {
 		reg = <2>;
-		iommus = <&/iommu>;
+		iommus = <&{/iommu}>;
 	};
 
 Multiple-master IOMMU:
@@ -145,28 +151,32 @@ Multiple-master IOMMU:
 
 	master@1 {
 		/* device has master ID 42 in the IOMMU */
-		iommus = <&/iommu 42>;
+		iommus = <&{/iommu} 42>;
 	};
 
 	master@2 {
 		/* device has master IDs 23 and 24 in the IOMMU */
-		iommus = <&/iommu 23>, <&/iommu 24>;
+		iommus = <&{/iommu} 23>, <&{/iommu} 24>;
 	};
 
 Multiple-master IOMMU with configurable DMA window:
 ---------------------------------------------------
 
 	/ {
-		#address-cells = <1>;
-		#size-cells = <1>;
-
 		iommu {
-			/* master ID, address and length of DMA window */
+			/*
+			 * One cell for the master ID and one cell for the
+			 * address of the DMA window. The length of the DMA
+			 * window is encoded in two cells.
+			 *
+			 * The DMA window is the range addressable by the
+			 * master (i.e. the I/O virtual address space).
+			 */
 			#iommu-cells = <4>;
 		};
 
 		master {
 			/* master ID 42, 4 GiB DMA window starting at 0 */
-			iommus = <&/iommu  42  0  0x1 0x0>;
+			iommus = <&{/iommu}  42  0  0x1 0x0>;
 		};
 	};

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]