[PATCH v2] devicetree: Add generic IOMMU device tree bindings

[PATCH v2] devicetree: Add generic IOMMU device tree bindings

[Arnd Bergmann]

On Saturday 07 June 2014 00:45:45 Thierry Reding wrote:
> This is somewhat off-topic, but given the various concepts discussed in
> this thread I'm beginning to wonder how they will be implemented.

I think it's good you raised the question.

> The
> current implementation hooks the IOMMU API into the DMA mapping API, and
> the way this is done is by setting a single IOMMU (or rather a set of
> IOMMU operations) globally per bus type.

I hadn't realized that we have a per-bus iommu_ops pointer. I agree
this will become a limitation as soon as we have a soc with two different
IOMMUs that have platform devices attached, and it has to be moved into
the device or a structure related to that.

If that turns out controversial, we can probably have a set of pseudo
iommu ops that just call into dev->archdata->iommu_ops->function()
for ARM.

> There are two issues that I can see with that: one is that we can't
> support multiple IOMMUs in the system at all, and the other is that
> there is no context associated with the IOMMU ops, and therefore there
> is no way to differentiate between two instances of the same IOMMU. A
> few drivers use global variables to keep track of context information
> but that won't work with multiple instances, unless they keep a global
> list of all instances and then require explicit lookup in each of the
> IOMMU operations. That sounds more like a workaround rather than a
> proper solution to me.

Supporting multiple iommus that share one iommu driver should work
without such hacks, as you can put the per-device information into
dev->device_dma_parameters (this works only for very simple IOMMUs)
or dev->archdata->iommu (we may want to generalize that, I think
someone just posted patches for it).

> Discussion in this thread indicates that both of the above will be
> required at some point. Have I completely missed something or will we
> have to rework (parts of) the IOMMU API to make this work?
> 
> One other thing that I have some difficulty understanding is how we can
> support things like process isolation using the current IOMMU API. Since
> a device will be statically assigned to one IOMMU domain at probe time
> there is no way we can change the address space upon a context switch.

We have just introduced a way to parse dma-ranges in of_dma_configure().

The only way I see this done for platform devices is to do the IOMMU
configuration in the same place: if an iommus property is found there,
we call out to the iommu driver that matches the respective iommu device
and let it configure the master device.

The device already has multiple properties related to iommus:
'struct device_dma_parameters', 'archdata', 'iommu_group', and
pdev_archdata for platform devices. This should be enough to set up
the default iommu dma_map_ops so we can have non-isolated DMA using
dma_map_* and dma_alloc_coherent.

I haven't given much thought to devices that want to use the IOMMU
API directly so they can have multiple domains rather than rely on
the dma-mapping abstraction.

	Arnd

[PATCH v2] devicetree: Add generic IOMMU device tree bindings

[Thierry Reding]

On Sat, Jun 07, 2014 at 03:22:13PM +0200, Arnd Bergmann wrote:
> On Saturday 07 June 2014 00:45:45 Thierry Reding wrote:
> > This is somewhat off-topic, but given the various concepts discussed in
> > this thread I'm beginning to wonder how they will be implemented.
> 
> I think it's good you raised the question.
> 
> > The
> > current implementation hooks the IOMMU API into the DMA mapping API, and
> > the way this is done is by setting a single IOMMU (or rather a set of
> > IOMMU operations) globally per bus type.
> 
> I hadn't realized that we have a per-bus iommu_ops pointer. I agree
> this will become a limitation as soon as we have a soc with two different
> IOMMUs that have platform devices attached, and it has to be moved into
> the device or a structure related to that.
> 
> If that turns out controversial, we can probably have a set of pseudo
> iommu ops that just call into dev->archdata->iommu_ops->function()
> for ARM.
> 
> > There are two issues that I can see with that: one is that we can't
> > support multiple IOMMUs in the system at all, and the other is that
> > there is no context associated with the IOMMU ops, and therefore there
> > is no way to differentiate between two instances of the same IOMMU. A
> > few drivers use global variables to keep track of context information
> > but that won't work with multiple instances, unless they keep a global
> > list of all instances and then require explicit lookup in each of the
> > IOMMU operations. That sounds more like a workaround rather than a
> > proper solution to me.
> 
> Supporting multiple iommus that share one iommu driver should work
> without such hacks, as you can put the per-device information into
> dev->device_dma_parameters (this works only for very simple IOMMUs)
> or dev->archdata->iommu

I was talking about the lack of a place to store context for the IOMMU
itself. Currently none of the functions in iommu_ops have a way to get
access to the IOMMU context itself. In fact there's not even a common
structure that could be used for this purpose. I have a couple of local
patches that try to add something like that, along with functions to
more explicitly hook up a device with it's IOMMU(s). It looks somewhat
like this:

	struct iommu {
		struct device *dev;

		struct list_head list;

		const struct iommu_ops *ops;
	};

	/* register and unregister IOMMU device with core */
	int iommu_add(struct iommu *iommu);
	void iommu_remove(struct iommu *iommu);

	/*
	 * Attach a device to one or more IOMMUs (according to the
	 * iommus property).
	 */
	int iommu_attach(struct device *dev);
	int iommu_detach(struct device *dev);

Does that look like a direction that we would want to pursue?

> (we may want to generalize that, I think someone just posted patches
> for it).

Perhaps you mean this:

	[PATCHv3 1/3] device.h: arm dma-iommu: Move out dma_iommu_mapping struct

? From a quick glance that indeed looks like a promising step towards
unifying this across architectures.

> > Discussion in this thread indicates that both of the above will be
> > required at some point. Have I completely missed something or will we
> > have to rework (parts of) the IOMMU API to make this work?
> > 
> > One other thing that I have some difficulty understanding is how we can
> > support things like process isolation using the current IOMMU API. Since
> > a device will be statically assigned to one IOMMU domain at probe time
> > there is no way we can change the address space upon a context switch.
> 
> We have just introduced a way to parse dma-ranges in of_dma_configure().
> 
> The only way I see this done for platform devices is to do the IOMMU
> configuration in the same place: if an iommus property is found there,
> we call out to the iommu driver that matches the respective iommu device
> and let it configure the master device.
> 
> The device already has multiple properties related to iommus:
> 'struct device_dma_parameters', 'archdata', 'iommu_group', and
> pdev_archdata for platform devices. This should be enough to set up
> the default iommu dma_map_ops so we can have non-isolated DMA using
> dma_map_* and dma_alloc_coherent.

Right, I think up to that point things should be fine with the existing
IOMMU API and using only DMA mapping functions.

> I haven't given much thought to devices that want to use the IOMMU
> API directly so they can have multiple domains rather than rely on
> the dma-mapping abstraction.

I'm specifically thinking about cases where we want to use the IOMMU to
isolate processes from each other. This is probably most relevant for
GPUs, since they are driven to a large degree from userspace. Other
peripherals are mostly services in kernel space exclusively, so I don't
think the issue is as relevant there.

On Tegra there's two IOMMUs, one system-wide and another one directly
used by the GPU (and programmed by the GPU driver (nouveau)). For the
latter it probably doesn't make sense to expose it via the IOMMU API,
and we may be able to get sufficient process isolation using only it
rather than in combination with the SMMU. However some of the graphics
components don't master through the GPU's IOMMU, so if we want any kind
of process isolation there we may need to control the IOMMU more
explicitly. I'm still trying to get a more full understanding of this,
but it would seem to me that we'd need some way to use different domains
(which I think is the proper abstraction for what the Tegra SMMU calls
an Address Space ID (ASID)) in one driver, depending on the current
process.

Thierry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20140609/7c73be3a/attachment.sig>

[PATCH v2] devicetree: Add generic IOMMU device tree bindings

[Will Deacon]

On Thu, Jul 10, 2014 at 11:32:16PM +0100, Olav Haugan wrote:
> On 7/9/2014 3:54 AM, Will Deacon wrote:
> > On Wed, Jul 09, 2014 at 02:07:38AM +0100, Olav Haugan wrote:
> >> So how does an algorithm figure this out in both my examples? The
> >> algorithm would have to know about both (all) bus masters and their
> >> stream IDs for a specific SMMU. If the algorithm operates on the set of
> >> stream IDs for one bus master at a time the algorithm has no way of
> >> knowing which bits can be ignored since it doesn't know the value of the
> >> other stream IDs for the other bus masters and thus could potentially
> >> create a mask that could cause a stream ID to match in two different
> >> entries.
> > 
> > Complete knowledge of the system topology (i.e. all bus masters) is a
> > requirement for being able to configure the SMMU correctly if you want to
> > guarantee that you don't have SMR aliasing issues.
> 
> So you agree that an algorithm needs to know about all the bus
> masters/stream IDs for a specific IOMMU before it can figure out the
> StreamID masks and how many SMRs can be allocated to a specific bus
> master? Andreas's algorithm does not know about the other bus
> masters/stream IDs. It operates on one bus master at a time.

Right, but it can certainly be improved. There are certain things you can do
without complete knowledge, as I mentioned previously (if you can have
densely packed power-of-2 aligned/sized regions).

> >>>> I am not familiar with Andreas's proposal. Do you have a link?
> >>>
> >>>   http://marc.info/?l=linux-arm-kernel&m=139110598005846&w=2
> >>
> >> Unless I am mistaken the algorithm works on one bus master at a time. I
> >> don't think that will work.
> > 
> > IIRC, it works for densely packed SIDs on the master, so it tries to build
> > up power-of-2 sized groups for that master then mops up the rest with
> > individual entries.
> 
> I ran the algorithm through a few trivial cases:
> 
> 1)
> Stream IDs: 0x21, 0x22, 0x23, 0x24, 0x25, 0x26, 0x27, 0x28
> Number of SMRs: 9
> 
> In this case the algorithm decided to set mask to 0 for all entries
> using up 8 of the SMRs.

Well think about what it's doing... we don't know about SID 0x20, for
example so there's not much it can do.

> 2) Same Stream IDs but only 2 SMRs.
> The algorithm gave an error saying I did not have enough SMRs.

There's a reason this didn't get merged, and it would be great if you could
try to improve the situation ;).

Will

[PATCH v2] devicetree: Add generic IOMMU device tree bindings

[Olav Haugan]

On 7/9/2014 3:54 AM, Will Deacon wrote:
> On Wed, Jul 09, 2014 at 02:07:38AM +0100, Olav Haugan wrote:
>> On 6/30/2014 2:52 AM, Will Deacon wrote:
>>> On Fri, Jun 27, 2014 at 11:23:27PM +0100, Olav Haugan wrote:
>>>> Lets say I have an IOMMU with 2 masters and 2 SMRn slots with the
>>>> following stream IDs coming from the masters:
>>>>
>>>> Master 1: 0x21, 0x22, 0x23, 0x24, 0x25, 0x26, 0x27, 0x28
>>>> Master 2: 0x30
>>>>
>>>> To make this work I would program SMR[0] with StreamID 0x20 and mask 0xF
>>>> to ignore lower 4 bits. SMR[1] would just be StreamID 0x30 with mask 0x0.
>>>>
>>>> However, I could also have an IOMMU with 2 masters and 9 SMRn slots with
>>>> the following stream IDs:
>>>>
>>>> Master 1: 0x21, 0x22, 0x23, 0x24, 0x25, 0x26, 0x27, 0x28
>>>> Master 2: 0x29
>>>>
>>>> Here I would program all SMRn and leave the mask to be 0 for all SMRn's.
>>>> So how do I detect when to apply a mask or not?
>>>
>>> You would aim to use the smallest number of SMRs per master possible.
>>> You could probably use:
>>>
>>>   Master 1: SMR[0].id == 0x20, SMR[0].mask = 0x07
>>>             SMR[1].id == 0x28, SMR[1].mask = 0x00
>>>
>>>   Master 2: SMR[2].id == 0x29, SMR[2].mask = 0x00
>>
>> So how does an algorithm figure this out in both my examples? The
>> algorithm would have to know about both (all) bus masters and their
>> stream IDs for a specific SMMU. If the algorithm operates on the set of
>> stream IDs for one bus master at a time the algorithm has no way of
>> knowing which bits can be ignored since it doesn't know the value of the
>> other stream IDs for the other bus masters and thus could potentially
>> create a mask that could cause a stream ID to match in two different
>> entries.
> 
> Complete knowledge of the system topology (i.e. all bus masters) is a
> requirement for being able to configure the SMMU correctly if you want to
> guarantee that you don't have SMR aliasing issues.

So you agree that an algorithm needs to know about all the bus
masters/stream IDs for a specific IOMMU before it can figure out the
StreamID masks and how many SMRs can be allocated to a specific bus
master? Andreas's algorithm does not know about the other bus
masters/stream IDs. It operates on one bus master at a time.

>>>> I am not familiar with Andreas's proposal. Do you have a link?
>>>
>>>   http://marc.info/?l=linux-arm-kernel&m=139110598005846&w=2
>>
>> Unless I am mistaken the algorithm works on one bus master at a time. I
>> don't think that will work.
> 
> IIRC, it works for densely packed SIDs on the master, so it tries to build
> up power-of-2 sized groups for that master then mops up the rest with
> individual entries.

I ran the algorithm through a few trivial cases:

1)
Stream IDs: 0x21, 0x22, 0x23, 0x24, 0x25, 0x26, 0x27, 0x28
Number of SMRs: 9

In this case the algorithm decided to set mask to 0 for all entries
using up 8 of the SMRs.

2) Same Stream IDs but only 2 SMRs.
The algorithm gave an error saying I did not have enough SMRs.

Thanks,

Olav

-- 
The Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
hosted by The Linux Foundation

[PATCH v2] devicetree: Add generic IOMMU device tree bindings

[Will Deacon]

On Wed, Jul 09, 2014 at 02:07:38AM +0100, Olav Haugan wrote:
> On 6/30/2014 2:52 AM, Will Deacon wrote:
> > On Fri, Jun 27, 2014 at 11:23:27PM +0100, Olav Haugan wrote:
> >> Lets say I have an IOMMU with 2 masters and 2 SMRn slots with the
> >> following stream IDs coming from the masters:
> >>
> >> Master 1: 0x21, 0x22, 0x23, 0x24, 0x25, 0x26, 0x27, 0x28
> >> Master 2: 0x30
> >>
> >> To make this work I would program SMR[0] with StreamID 0x20 and mask 0xF
> >> to ignore lower 4 bits. SMR[1] would just be StreamID 0x30 with mask 0x0.
> >>
> >> However, I could also have an IOMMU with 2 masters and 9 SMRn slots with
> >> the following stream IDs:
> >>
> >> Master 1: 0x21, 0x22, 0x23, 0x24, 0x25, 0x26, 0x27, 0x28
> >> Master 2: 0x29
> >>
> >> Here I would program all SMRn and leave the mask to be 0 for all SMRn's.
> >> So how do I detect when to apply a mask or not?
> > 
> > You would aim to use the smallest number of SMRs per master possible.
> > You could probably use:
> > 
> >   Master 1: SMR[0].id == 0x20, SMR[0].mask = 0x07
> >             SMR[1].id == 0x28, SMR[1].mask = 0x00
> > 
> >   Master 2: SMR[2].id == 0x29, SMR[2].mask = 0x00
> 
> So how does an algorithm figure this out in both my examples? The
> algorithm would have to know about both (all) bus masters and their
> stream IDs for a specific SMMU. If the algorithm operates on the set of
> stream IDs for one bus master at a time the algorithm has no way of
> knowing which bits can be ignored since it doesn't know the value of the
> other stream IDs for the other bus masters and thus could potentially
> create a mask that could cause a stream ID to match in two different
> entries.

Complete knowledge of the system topology (i.e. all bus masters) is a
requirement for being able to configure the SMMU correctly if you want to
guarantee that you don't have SMR aliasing issues.

> >> I am not familiar with Andreas's proposal. Do you have a link?
> > 
> >   http://marc.info/?l=linux-arm-kernel&m=139110598005846&w=2
> 
> Unless I am mistaken the algorithm works on one bus master at a time. I
> don't think that will work.

IIRC, it works for densely packed SIDs on the master, so it tries to build
up power-of-2 sized groups for that master then mops up the rest with
individual entries.

Will

[PATCH v2] devicetree: Add generic IOMMU device tree bindings

[Olav Haugan]

On 6/30/2014 2:52 AM, Will Deacon wrote:
> Hi Olav,
> 
> On Fri, Jun 27, 2014 at 11:23:27PM +0100, Olav Haugan wrote:
>> On 6/25/2014 2:18 AM, Will Deacon wrote:
>>> Why can't it be dynamically detected? Whilst the StreamIDs are fixed in
>>> hardware (from the SMMU architecture perspective), the SMRs are completely
>>> programmable. Why doesn't something like Andreas's proposal work for you?
>>> The idea there was to find the constant bits among the StreamIDs for a
>>> master and create the mask accordingly.
>>>
>> Lets say I have an IOMMU with 2 masters and 2 SMRn slots with the
>> following stream IDs coming from the masters:
>>
>> Master 1: 0x21, 0x22, 0x23, 0x24, 0x25, 0x26, 0x27, 0x28
>> Master 2: 0x30
>>
>> To make this work I would program SMR[0] with StreamID 0x20 and mask 0xF
>> to ignore lower 4 bits. SMR[1] would just be StreamID 0x30 with mask 0x0.
>>
>> However, I could also have an IOMMU with 2 masters and 9 SMRn slots with
>> the following stream IDs:
>>
>> Master 1: 0x21, 0x22, 0x23, 0x24, 0x25, 0x26, 0x27, 0x28
>> Master 2: 0x29
>>
>> Here I would program all SMRn and leave the mask to be 0 for all SMRn's.
>> So how do I detect when to apply a mask or not?
> 
> You would aim to use the smallest number of SMRs per master possible.
> You could probably use:
> 
>   Master 1: SMR[0].id == 0x20, SMR[0].mask = 0x07
>             SMR[1].id == 0x28, SMR[1].mask = 0x00
> 
>   Master 2: SMR[2].id == 0x29, SMR[2].mask = 0x00

So how does an algorithm figure this out in both my examples? The
algorithm would have to know about both (all) bus masters and their
stream IDs for a specific SMMU. If the algorithm operates on the set of
stream IDs for one bus master at a time the algorithm has no way of
knowing which bits can be ignored since it doesn't know the value of the
other stream IDs for the other bus masters and thus could potentially
create a mask that could cause a stream ID to match in two different
entries.

>> I am not familiar with Andreas's proposal. Do you have a link?
> 
>   http://marc.info/?l=linux-arm-kernel&m=139110598005846&w=2

Unless I am mistaken the algorithm works on one bus master at a time. I
don't think that will work.

Thanks,

Olav Haugan

-- 
The Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
hosted by The Linux Foundation

[PATCH v2] devicetree: Add generic IOMMU device tree bindings

[Will Deacon]

Hi Olav,

On Fri, Jun 27, 2014 at 11:23:27PM +0100, Olav Haugan wrote:
> On 6/25/2014 2:18 AM, Will Deacon wrote:
> > Why can't it be dynamically detected? Whilst the StreamIDs are fixed in
> > hardware (from the SMMU architecture perspective), the SMRs are completely
> > programmable. Why doesn't something like Andreas's proposal work for you?
> > The idea there was to find the constant bits among the StreamIDs for a
> > master and create the mask accordingly.
> > 
> Lets say I have an IOMMU with 2 masters and 2 SMRn slots with the
> following stream IDs coming from the masters:
> 
> Master 1: 0x21, 0x22, 0x23, 0x24, 0x25, 0x26, 0x27, 0x28
> Master 2: 0x30
> 
> To make this work I would program SMR[0] with StreamID 0x20 and mask 0xF
> to ignore lower 4 bits. SMR[1] would just be StreamID 0x30 with mask 0x0.
> 
> However, I could also have an IOMMU with 2 masters and 9 SMRn slots with
> the following stream IDs:
> 
> Master 1: 0x21, 0x22, 0x23, 0x24, 0x25, 0x26, 0x27, 0x28
> Master 2: 0x29
> 
> Here I would program all SMRn and leave the mask to be 0 for all SMRn's.
> So how do I detect when to apply a mask or not?

You would aim to use the smallest number of SMRs per master possible.
You could probably use:

  Master 1: SMR[0].id == 0x20, SMR[0].mask = 0x07
            SMR[1].id == 0x28, SMR[1].mask = 0x00

  Master 2: SMR[2].id == 0x29, SMR[2].mask = 0x00

> I am not familiar with Andreas's proposal. Do you have a link?

  http://marc.info/?l=linux-arm-kernel&m=139110598005846&w=2

Note that since Calxeda went under, Andreas is unfortunately no longer
working on this.

Will

[PATCH v2] devicetree: Add generic IOMMU device tree bindings

[Olav Haugan]

On 6/25/2014 2:18 AM, Will Deacon wrote:
> On Tue, Jun 24, 2014 at 10:35:54PM +0100, Olav Haugan wrote:
>> On 6/24/2014 11:11 AM, Will Deacon wrote:
>>> On Tue, Jun 24, 2014 at 06:57:44PM +0100, Olav Haugan wrote:
>>>> On 6/24/2014 2:18 AM, Will Deacon wrote:
>>>>> On Sat, Jun 21, 2014 at 12:16:25AM +0100, Olav Haugan wrote:
>>>>>> We have multiple-master SMMUs and each master emits a variable number of
>>>>>> StreamIDs. However, we have to apply a mask (the ARM SMMU spec allows
>>>>>> for this) to the StreamIDs due to limited number of StreamID 2 Context
>>>>>> Bank entries in the SMMU. If my understanding is correct we would
>>>>>> represent this in the DT like this:
>>>>>>
>>>>>> 	iommu {
>>>>>> 		#address-cells = <2>;
>>>>>> 		#size-cells = <0>;
>>>>>> 	};
>>>>>>
>>>>>> 	master at a {
>>>>>> 		...
>>>>>> 		iommus = <&iommu StreamID0 MASK0>,
>>>>>> 			 <&iommu StreamID1 MASK1>,
>>>>>> 			 <&iommu StreamID2 MASK2>;
>>>>>> 	};
>>>>>
>>>>> Stupid question, but why not simply describe the masked IDs? What use does
>>>>> the `raw' ID have to Linux?
>>>>
>>>> We do describe the masked StreamID (SID) but we need to specify the mask
>>>> that the SMMU should apply to the incoming SIDs, right?
>>>>
>>>> We have a bus master that emits 43 unique SIDs. However, we have only 40
>>>> SMMU_SMRn registers in the SMMU. So we need to mask out some of the
>>>> incoming SID bits so that the 43 SIDs can match one of 40 entries in the
>>>> SMR.
>>>
>>> Hmm, so you're talking about stream matching, right? That doesn't belong in
>>> the device-tree. I appreciate that the current driver does a terrible job at
>>> allocating the SMRs (it's bloody difficult!), but we should try to improve
>>> the dynamic behaviour instead of moving configuration of the SMMU out into
>>> device-tree, where it's inflexible at best.
>>
>> I am talking about SMMU_SMRn[MASK] register bits. This is not something
>> that can be dynamically detected at run-time. It is configuration at the
>> same level as the actual StreamIDs.
> 
> Why can't it be dynamically detected? Whilst the StreamIDs are fixed in
> hardware (from the SMMU architecture perspective), the SMRs are completely
> programmable. Why doesn't something like Andreas's proposal work for you?
> The idea there was to find the constant bits among the StreamIDs for a
> master and create the mask accordingly.
> 
Lets say I have an IOMMU with 2 masters and 2 SMRn slots with the
following stream IDs coming from the masters:

Master 1: 0x21, 0x22, 0x23, 0x24, 0x25, 0x26, 0x27, 0x28
Master 2: 0x30

To make this work I would program SMR[0] with StreamID 0x20 and mask 0xF
to ignore lower 4 bits. SMR[1] would just be StreamID 0x30 with mask 0x0.

However, I could also have an IOMMU with 2 masters and 9 SMRn slots with
the following stream IDs:

Master 1: 0x21, 0x22, 0x23, 0x24, 0x25, 0x26, 0x27, 0x28
Master 2: 0x29

Here I would program all SMRn and leave the mask to be 0 for all SMRn's.
So how do I detect when to apply a mask or not?

I am not familiar with Andreas's proposal. Do you have a link?

Thanks,

Olav Haugan

-- 
The Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
hosted by The Linux Foundation

[PATCH v2] devicetree: Add generic IOMMU device tree bindings

[Will Deacon]

On Wed, Jun 25, 2014 at 11:12:13AM +0100, Arnd Bergmann wrote:
> On Wednesday 25 June 2014 10:57:36 Will Deacon wrote:
> > So far, I've been avoiding the hardcoding. However, you could potentially
> > build a system with a small number of SMRs (compared to the number of
> > StreamIDs) and allocate the StreamIDs in such a way that I think the dynamic
> > configuration would be NP complete if we require an optimal SMR allocation.
> > 
> > However:
> > 
> >   (1) I don't know of a system where this is the case
> >   (2) Not much work has been done on improving the dynamic allocator yet
> > 
> > which is why I'm still favouring dynamic configuration in the driver.
> > 
> > The other thing I forgot to mention earlier is that we need to support
> > device hotplug in the future, so some level of dynamic configuration
> > will always be required.
> 
> Ok, got it. So we just hope that we can make dynamic configuration
> work all the time, but if it all fails, then we come up with a
> hardcoded configuration method.
> 
> I guess this could be done similarly to how we handle clocks on
> a lot of systems: generally these are dynamic, but you have the
> option to provide hints in the clock controller node about how
> you expect things to be configured.
> 
> For the SMMU that could mean that (if we get into the situation you
> describe), we add optional properties to the SMMU node itself
> describing how we expect the SMRs to be used.

That sounds good to me! Thanks for the discussion.

Will

[PATCH v2] devicetree: Add generic IOMMU device tree bindings

[Arnd Bergmann]

On Wednesday 25 June 2014 10:57:36 Will Deacon wrote:
> So far, I've been avoiding the hardcoding. However, you could potentially
> build a system with a small number of SMRs (compared to the number of
> StreamIDs) and allocate the StreamIDs in such a way that I think the dynamic
> configuration would be NP complete if we require an optimal SMR allocation.
> 
> However:
> 
>   (1) I don't know of a system where this is the case
>   (2) Not much work has been done on improving the dynamic allocator yet
> 
> which is why I'm still favouring dynamic configuration in the driver.
> 
> The other thing I forgot to mention earlier is that we need to support
> device hotplug in the future, so some level of dynamic configuration
> will always be required.

Ok, got it. So we just hope that we can make dynamic configuration
work all the time, but if it all fails, then we come up with a
hardcoded configuration method.

I guess this could be done similarly to how we handle clocks on
a lot of systems: generally these are dynamic, but you have the
option to provide hints in the clock controller node about how
you expect things to be configured.

For the SMMU that could mean that (if we get into the situation you
describe), we add optional properties to the SMMU node itself
describing how we expect the SMRs to be used.

	Arnd

[PATCH v2] devicetree: Add generic IOMMU device tree bindings

[Will Deacon]

On Wed, Jun 25, 2014 at 10:48:31AM +0100, Arnd Bergmann wrote:
> On Wednesday 25 June 2014 10:38:25 Will Deacon wrote:
> > On Wed, Jun 25, 2014 at 10:27:50AM +0100, Arnd Bergmann wrote:
> > > I think the situation is a bit different here: It's less about the corner
> > > cases for the SMMU, but about the question whether it makes more sense to
> > > have the kernel figure out the settings, or have them come from DT
> > > all the time.
> > 
> > But, as far as I can tell, this setting is basically `which bits are
> > constant among this set of IDs'.
> > 
> > > As I said, I can't tell which approach is best here, but it sounds to
> > > me we should either do dynamic configuration and get it right, or
> > > hardcode the configuration it all the time if we can't.
> > 
> > I disagree. If you have `sensible' StreamID allocations, doing this
> > dynamically should be straight-forward and gives the driver more flexibility
> > (e.g. we then have the option of combining SMR entries for different masters
> > if they are in the same domain). The dynamic approach also lends itself to
> > sanity-checking (it is IMPLEMENTATION DEFINED whether the SMMU detects SMR
> > aliases) and helps with virtualisation (forcing QEMU to generate these masks
> > in a device-tree for a guest using a virtual SMMU interface is very painful).
> 
> In which case do you think hardcoding is needed then? I'm still confused
> why you think we need both, as your arguments seem to all be in favor of
> dynamic configuration.

So far, I've been avoiding the hardcoding. However, you could potentially
build a system with a small number of SMRs (compared to the number of
StreamIDs) and allocate the StreamIDs in such a way that I think the dynamic
configuration would be NP complete if we require an optimal SMR allocation.

However:

  (1) I don't know of a system where this is the case
  (2) Not much work has been done on improving the dynamic allocator yet

which is why I'm still favouring dynamic configuration in the driver.

The other thing I forgot to mention earlier is that we need to support
device hotplug in the future, so some level of dynamic configuration
will always be required.

Will

[PATCH v2] devicetree: Add generic IOMMU device tree bindings

[Arnd Bergmann]

On Wednesday 25 June 2014 10:38:25 Will Deacon wrote:
> On Wed, Jun 25, 2014 at 10:27:50AM +0100, Arnd Bergmann wrote:
> > On Wednesday 25 June 2014 10:17:02 Will Deacon wrote:
> > > On Tue, Jun 24, 2014 at 07:20:56PM +0100, Arnd Bergmann wrote:
> > > > On Tuesday 24 June 2014 19:11:50 Will Deacon wrote:
> > > > > On Tue, Jun 24, 2014 at 06:57:44PM +0100, Olav Haugan wrote:
> > > > > > We do describe the masked StreamID (SID) but we need to specify the mask
> > > > > > that the SMMU should apply to the incoming SIDs, right?
> > > > > > 
> > > > > > We have a bus master that emits 43 unique SIDs. However, we have only 40
> > > > > > SMMU_SMRn registers in the SMMU. So we need to mask out some of the
> > > > > > incoming SID bits so that the 43 SIDs can match one of 40 entries in the
> > > > > > SMR.
> > > > > 
> > > > > Hmm, so you're talking about stream matching, right? That doesn't belong in
> > > > > the device-tree. I appreciate that the current driver does a terrible job at
> > > > > allocating the SMRs (it's bloody difficult!), but we should try to improve
> > > > > the dynamic behaviour instead of moving configuration of the SMMU out into
> > > > > device-tree, where it's inflexible at best.
> > > > > 
> > > > > There have been patches previously posted by Andreas Herrmann helping here.
> > > > > I'd be glad to see them revived.
> > > > 
> > > > Note that there are areas where we have in the past decided that dynamic
> > > > configuration is just too hard for the kernel to do and that we're better
> > > > off putting the configuration into DT. Pinctrl and clocks are at least
> > > > partially in that category.
> > > > 
> > > > It's always best if you can get the kernel to do things in the ideal
> > > > way where that is possible, but getting there may be just not worth it.
> > > > 
> > > > I have no idea where it should be for SMMU, but it's something to consider:
> > > > if you can take reasonable shortcuts by reading parts of the configuration
> > > > from DT, you may just as well do that.
> > > 
> > > I treat this in the same manner as the topology bindings we discussed
> > > previously; we should do a best-effort to configure things dynamically and
> > > solve corner-cases and quirks as special cases.
> > 
> > I think the situation is a bit different here: It's less about the corner
> > cases for the SMMU, but about the question whether it makes more sense to
> > have the kernel figure out the settings, or have them come from DT
> > all the time.
> 
> But, as far as I can tell, this setting is basically `which bits are
> constant among this set of IDs'.
> 
> > As I said, I can't tell which approach is best here, but it sounds to
> > me we should either do dynamic configuration and get it right, or
> > hardcode the configuration it all the time if we can't.
> 
> I disagree. If you have `sensible' StreamID allocations, doing this
> dynamically should be straight-forward and gives the driver more flexibility
> (e.g. we then have the option of combining SMR entries for different masters
> if they are in the same domain). The dynamic approach also lends itself to
> sanity-checking (it is IMPLEMENTATION DEFINED whether the SMMU detects SMR
> aliases) and helps with virtualisation (forcing QEMU to generate these masks
> in a device-tree for a guest using a virtual SMMU interface is very painful).

In which case do you think hardcoding is needed then? I'm still confused
why you think we need both, as your arguments seem to all be in favor of
dynamic configuration.

	Arnd

[PATCH v2] devicetree: Add generic IOMMU device tree bindings

[Will Deacon]

On Wed, Jun 25, 2014 at 10:27:50AM +0100, Arnd Bergmann wrote:
> On Wednesday 25 June 2014 10:17:02 Will Deacon wrote:
> > On Tue, Jun 24, 2014 at 07:20:56PM +0100, Arnd Bergmann wrote:
> > > On Tuesday 24 June 2014 19:11:50 Will Deacon wrote:
> > > > On Tue, Jun 24, 2014 at 06:57:44PM +0100, Olav Haugan wrote:
> > > > > We do describe the masked StreamID (SID) but we need to specify the mask
> > > > > that the SMMU should apply to the incoming SIDs, right?
> > > > > 
> > > > > We have a bus master that emits 43 unique SIDs. However, we have only 40
> > > > > SMMU_SMRn registers in the SMMU. So we need to mask out some of the
> > > > > incoming SID bits so that the 43 SIDs can match one of 40 entries in the
> > > > > SMR.
> > > > 
> > > > Hmm, so you're talking about stream matching, right? That doesn't belong in
> > > > the device-tree. I appreciate that the current driver does a terrible job at
> > > > allocating the SMRs (it's bloody difficult!), but we should try to improve
> > > > the dynamic behaviour instead of moving configuration of the SMMU out into
> > > > device-tree, where it's inflexible at best.
> > > > 
> > > > There have been patches previously posted by Andreas Herrmann helping here.
> > > > I'd be glad to see them revived.
> > > 
> > > Note that there are areas where we have in the past decided that dynamic
> > > configuration is just too hard for the kernel to do and that we're better
> > > off putting the configuration into DT. Pinctrl and clocks are at least
> > > partially in that category.
> > > 
> > > It's always best if you can get the kernel to do things in the ideal
> > > way where that is possible, but getting there may be just not worth it.
> > > 
> > > I have no idea where it should be for SMMU, but it's something to consider:
> > > if you can take reasonable shortcuts by reading parts of the configuration
> > > from DT, you may just as well do that.
> > 
> > I treat this in the same manner as the topology bindings we discussed
> > previously; we should do a best-effort to configure things dynamically and
> > solve corner-cases and quirks as special cases.
> 
> I think the situation is a bit different here: It's less about the corner
> cases for the SMMU, but about the question whether it makes more sense to
> have the kernel figure out the settings, or have them come from DT
> all the time.

But, as far as I can tell, this setting is basically `which bits are
constant among this set of IDs'.

> As I said, I can't tell which approach is best here, but it sounds to
> me we should either do dynamic configuration and get it right, or
> hardcode the configuration it all the time if we can't.

I disagree. If you have `sensible' StreamID allocations, doing this
dynamically should be straight-forward and gives the driver more flexibility
(e.g. we then have the option of combining SMR entries for different masters
if they are in the same domain). The dynamic approach also lends itself to
sanity-checking (it is IMPLEMENTATION DEFINED whether the SMMU detects SMR
aliases) and helps with virtualisation (forcing QEMU to generate these masks
in a device-tree for a guest using a virtual SMMU interface is very painful).

Will

[PATCH v2] devicetree: Add generic IOMMU device tree bindings

[Arnd Bergmann]

On Wednesday 25 June 2014 10:17:02 Will Deacon wrote:
> On Tue, Jun 24, 2014 at 07:20:56PM +0100, Arnd Bergmann wrote:
> > On Tuesday 24 June 2014 19:11:50 Will Deacon wrote:
> > > On Tue, Jun 24, 2014 at 06:57:44PM +0100, Olav Haugan wrote:
> > > > We do describe the masked StreamID (SID) but we need to specify the mask
> > > > that the SMMU should apply to the incoming SIDs, right?
> > > > 
> > > > We have a bus master that emits 43 unique SIDs. However, we have only 40
> > > > SMMU_SMRn registers in the SMMU. So we need to mask out some of the
> > > > incoming SID bits so that the 43 SIDs can match one of 40 entries in the
> > > > SMR.
> > > 
> > > Hmm, so you're talking about stream matching, right? That doesn't belong in
> > > the device-tree. I appreciate that the current driver does a terrible job at
> > > allocating the SMRs (it's bloody difficult!), but we should try to improve
> > > the dynamic behaviour instead of moving configuration of the SMMU out into
> > > device-tree, where it's inflexible at best.
> > > 
> > > There have been patches previously posted by Andreas Herrmann helping here.
> > > I'd be glad to see them revived.
> > 
> > Note that there are areas where we have in the past decided that dynamic
> > configuration is just too hard for the kernel to do and that we're better
> > off putting the configuration into DT. Pinctrl and clocks are at least
> > partially in that category.
> > 
> > It's always best if you can get the kernel to do things in the ideal
> > way where that is possible, but getting there may be just not worth it.
> > 
> > I have no idea where it should be for SMMU, but it's something to consider:
> > if you can take reasonable shortcuts by reading parts of the configuration
> > from DT, you may just as well do that.
> 
> I treat this in the same manner as the topology bindings we discussed
> previously; we should do a best-effort to configure things dynamically and
> solve corner-cases and quirks as special cases.

I think the situation is a bit different here: It's less about the corner
cases for the SMMU, but about the question whether it makes more sense to
have the kernel figure out the settings, or have them come from DT
all the time.

As I said, I can't tell which approach is best here, but it sounds to
me we should either do dynamic configuration and get it right, or
hardcode the configuration it all the time if we can't.

	Arnd

[PATCH v2] devicetree: Add generic IOMMU device tree bindings

[Will Deacon]

On Tue, Jun 24, 2014 at 10:35:54PM +0100, Olav Haugan wrote:
> On 6/24/2014 11:11 AM, Will Deacon wrote:
> > On Tue, Jun 24, 2014 at 06:57:44PM +0100, Olav Haugan wrote:
> >> On 6/24/2014 2:18 AM, Will Deacon wrote:
> >>> On Sat, Jun 21, 2014 at 12:16:25AM +0100, Olav Haugan wrote:
> >>>> We have multiple-master SMMUs and each master emits a variable number of
> >>>> StreamIDs. However, we have to apply a mask (the ARM SMMU spec allows
> >>>> for this) to the StreamIDs due to limited number of StreamID 2 Context
> >>>> Bank entries in the SMMU. If my understanding is correct we would
> >>>> represent this in the DT like this:
> >>>>
> >>>> 	iommu {
> >>>> 		#address-cells = <2>;
> >>>> 		#size-cells = <0>;
> >>>> 	};
> >>>>
> >>>> 	master at a {
> >>>> 		...
> >>>> 		iommus = <&iommu StreamID0 MASK0>,
> >>>> 			 <&iommu StreamID1 MASK1>,
> >>>> 			 <&iommu StreamID2 MASK2>;
> >>>> 	};
> >>>
> >>> Stupid question, but why not simply describe the masked IDs? What use does
> >>> the `raw' ID have to Linux?
> >>
> >> We do describe the masked StreamID (SID) but we need to specify the mask
> >> that the SMMU should apply to the incoming SIDs, right?
> >>
> >> We have a bus master that emits 43 unique SIDs. However, we have only 40
> >> SMMU_SMRn registers in the SMMU. So we need to mask out some of the
> >> incoming SID bits so that the 43 SIDs can match one of 40 entries in the
> >> SMR.
> > 
> > Hmm, so you're talking about stream matching, right? That doesn't belong in
> > the device-tree. I appreciate that the current driver does a terrible job at
> > allocating the SMRs (it's bloody difficult!), but we should try to improve
> > the dynamic behaviour instead of moving configuration of the SMMU out into
> > device-tree, where it's inflexible at best.
> 
> I am talking about SMMU_SMRn[MASK] register bits. This is not something
> that can be dynamically detected at run-time. It is configuration at the
> same level as the actual StreamIDs.

Why can't it be dynamically detected? Whilst the StreamIDs are fixed in
hardware (from the SMMU architecture perspective), the SMRs are completely
programmable. Why doesn't something like Andreas's proposal work for you?
The idea there was to find the constant bits among the StreamIDs for a
master and create the mask accordingly.

Will

[PATCH v2] devicetree: Add generic IOMMU device tree bindings

[Will Deacon]

On Tue, Jun 24, 2014 at 07:20:56PM +0100, Arnd Bergmann wrote:
> On Tuesday 24 June 2014 19:11:50 Will Deacon wrote:
> > On Tue, Jun 24, 2014 at 06:57:44PM +0100, Olav Haugan wrote:
> > > We do describe the masked StreamID (SID) but we need to specify the mask
> > > that the SMMU should apply to the incoming SIDs, right?
> > > 
> > > We have a bus master that emits 43 unique SIDs. However, we have only 40
> > > SMMU_SMRn registers in the SMMU. So we need to mask out some of the
> > > incoming SID bits so that the 43 SIDs can match one of 40 entries in the
> > > SMR.
> > 
> > Hmm, so you're talking about stream matching, right? That doesn't belong in
> > the device-tree. I appreciate that the current driver does a terrible job at
> > allocating the SMRs (it's bloody difficult!), but we should try to improve
> > the dynamic behaviour instead of moving configuration of the SMMU out into
> > device-tree, where it's inflexible at best.
> > 
> > There have been patches previously posted by Andreas Herrmann helping here.
> > I'd be glad to see them revived.
> 
> Note that there are areas where we have in the past decided that dynamic
> configuration is just too hard for the kernel to do and that we're better
> off putting the configuration into DT. Pinctrl and clocks are at least
> partially in that category.
> 
> It's always best if you can get the kernel to do things in the ideal
> way where that is possible, but getting there may be just not worth it.
> 
> I have no idea where it should be for SMMU, but it's something to consider:
> if you can take reasonable shortcuts by reading parts of the configuration
> from DT, you may just as well do that.

I treat this in the same manner as the topology bindings we discussed
previously; we should do a best-effort to configure things dynamically and
solve corner-cases and quirks as special cases.

Will

[PATCH v2] devicetree: Add generic IOMMU device tree bindings

[Olav Haugan]

On 6/24/2014 11:11 AM, Will Deacon wrote:
> On Tue, Jun 24, 2014 at 06:57:44PM +0100, Olav Haugan wrote:
>> On 6/24/2014 2:18 AM, Will Deacon wrote:
>>> On Sat, Jun 21, 2014 at 12:16:25AM +0100, Olav Haugan wrote:
>>>> We have multiple-master SMMUs and each master emits a variable number of
>>>> StreamIDs. However, we have to apply a mask (the ARM SMMU spec allows
>>>> for this) to the StreamIDs due to limited number of StreamID 2 Context
>>>> Bank entries in the SMMU. If my understanding is correct we would
>>>> represent this in the DT like this:
>>>>
>>>> 	iommu {
>>>> 		#address-cells = <2>;
>>>> 		#size-cells = <0>;
>>>> 	};
>>>>
>>>> 	master at a {
>>>> 		...
>>>> 		iommus = <&iommu StreamID0 MASK0>,
>>>> 			 <&iommu StreamID1 MASK1>,
>>>> 			 <&iommu StreamID2 MASK2>;
>>>> 	};
>>>
>>> Stupid question, but why not simply describe the masked IDs? What use does
>>> the `raw' ID have to Linux?
>>
>> We do describe the masked StreamID (SID) but we need to specify the mask
>> that the SMMU should apply to the incoming SIDs, right?
>>
>> We have a bus master that emits 43 unique SIDs. However, we have only 40
>> SMMU_SMRn registers in the SMMU. So we need to mask out some of the
>> incoming SID bits so that the 43 SIDs can match one of 40 entries in the
>> SMR.
> 
> Hmm, so you're talking about stream matching, right? That doesn't belong in
> the device-tree. I appreciate that the current driver does a terrible job at
> allocating the SMRs (it's bloody difficult!), but we should try to improve
> the dynamic behaviour instead of moving configuration of the SMMU out into
> device-tree, where it's inflexible at best.

I am talking about SMMU_SMRn[MASK] register bits. This is not something
that can be dynamically detected at run-time. It is configuration at the
same level as the actual StreamIDs.

Thanks,


Olav Haugan

-- 
The Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
hosted by The Linux Foundation

[PATCH v2] devicetree: Add generic IOMMU device tree bindings

[Arnd Bergmann]

On Tuesday 24 June 2014 19:11:50 Will Deacon wrote:
> On Tue, Jun 24, 2014 at 06:57:44PM +0100, Olav Haugan wrote:
> > On 6/24/2014 2:18 AM, Will Deacon wrote:
> > > On Sat, Jun 21, 2014 at 12:16:25AM +0100, Olav Haugan wrote:
> > >> We have multiple-master SMMUs and each master emits a variable number of
> > >> StreamIDs. However, we have to apply a mask (the ARM SMMU spec allows
> > >> for this) to the StreamIDs due to limited number of StreamID 2 Context
> > >> Bank entries in the SMMU. If my understanding is correct we would
> > >> represent this in the DT like this:
> > >>
> > >>    iommu {
> > >>            #address-cells = <2>;
> > >>            #size-cells = <0>;
> > >>    };
> > >>
> > >>    master at a {
> > >>            ...
> > >>            iommus = <&iommu StreamID0 MASK0>,
> > >>                     <&iommu StreamID1 MASK1>,
> > >>                     <&iommu StreamID2 MASK2>;
> > >>    };
> > > 
> > > Stupid question, but why not simply describe the masked IDs? What use does
> > > the `raw' ID have to Linux?
> > 
> > We do describe the masked StreamID (SID) but we need to specify the mask
> > that the SMMU should apply to the incoming SIDs, right?
> > 
> > We have a bus master that emits 43 unique SIDs. However, we have only 40
> > SMMU_SMRn registers in the SMMU. So we need to mask out some of the
> > incoming SID bits so that the 43 SIDs can match one of 40 entries in the
> > SMR.
> 
> Hmm, so you're talking about stream matching, right? That doesn't belong in
> the device-tree. I appreciate that the current driver does a terrible job at
> allocating the SMRs (it's bloody difficult!), but we should try to improve
> the dynamic behaviour instead of moving configuration of the SMMU out into
> device-tree, where it's inflexible at best.
> 
> There have been patches previously posted by Andreas Herrmann helping here.
> I'd be glad to see them revived.

Note that there are areas where we have in the past decided that dynamic
configuration is just too hard for the kernel to do and that we're better
off putting the configuration into DT. Pinctrl and clocks are at least
partially in that category.

It's always best if you can get the kernel to do things in the ideal
way where that is possible, but getting there may be just not worth it.

I have no idea where it should be for SMMU, but it's something to consider:
if you can take reasonable shortcuts by reading parts of the configuration
from DT, you may just as well do that.

	Arnd

[PATCH v2] devicetree: Add generic IOMMU device tree bindings

[Will Deacon]

On Tue, Jun 24, 2014 at 06:57:44PM +0100, Olav Haugan wrote:
> On 6/24/2014 2:18 AM, Will Deacon wrote:
> > On Sat, Jun 21, 2014 at 12:16:25AM +0100, Olav Haugan wrote:
> >> We have multiple-master SMMUs and each master emits a variable number of
> >> StreamIDs. However, we have to apply a mask (the ARM SMMU spec allows
> >> for this) to the StreamIDs due to limited number of StreamID 2 Context
> >> Bank entries in the SMMU. If my understanding is correct we would
> >> represent this in the DT like this:
> >>
> >> 	iommu {
> >> 		#address-cells = <2>;
> >> 		#size-cells = <0>;
> >> 	};
> >>
> >> 	master at a {
> >> 		...
> >> 		iommus = <&iommu StreamID0 MASK0>,
> >> 			 <&iommu StreamID1 MASK1>,
> >> 			 <&iommu StreamID2 MASK2>;
> >> 	};
> > 
> > Stupid question, but why not simply describe the masked IDs? What use does
> > the `raw' ID have to Linux?
> 
> We do describe the masked StreamID (SID) but we need to specify the mask
> that the SMMU should apply to the incoming SIDs, right?
> 
> We have a bus master that emits 43 unique SIDs. However, we have only 40
> SMMU_SMRn registers in the SMMU. So we need to mask out some of the
> incoming SID bits so that the 43 SIDs can match one of 40 entries in the
> SMR.

Hmm, so you're talking about stream matching, right? That doesn't belong in
the device-tree. I appreciate that the current driver does a terrible job at
allocating the SMRs (it's bloody difficult!), but we should try to improve
the dynamic behaviour instead of moving configuration of the SMMU out into
device-tree, where it's inflexible at best.

There have been patches previously posted by Andreas Herrmann helping here.
I'd be glad to see them revived.

Will

[PATCH v2] devicetree: Add generic IOMMU device tree bindings

[Olav Haugan]

On 6/24/2014 2:18 AM, Will Deacon wrote:
> On Sat, Jun 21, 2014 at 12:16:25AM +0100, Olav Haugan wrote:
>> On 5/30/2014 12:06 PM, Arnd Bergmann wrote:
>>> On Friday 30 May 2014 08:16:05 Rob Herring wrote:
>>>> Presumably the ID would be the streamID on ARM's SMMU. How would a
>>>> master with 8 streamIDs be described? This is what Calxeda midway has
>>>> for SATA and I would expect that to be somewhat common. Either you
>>>> need some ID masking or you'll have lots of duplication when you have
>>>> windows.
>>>
>>> I don't understand the problem. If you have stream IDs 0 through 7,
>>> you would have
>>>
>>> 	master at a {
>>> 		...
>>> 		iommus = <&smmu 0>;
>>> 	};
>>>
>>> 	master at b {
>>> 		...
>>> 		iommus = <&smmu 1;
>>> 	};
>>>
>>> 	...
>>>
>>> 	master at 12 {
>>> 		...
>>> 		iommus = <&smmu 7;
>>> 	};
>>>
>>> and you don't need a window at all. Why would you need a mask of
>>> some sort?
>>
>> We have multiple-master SMMUs and each master emits a variable number of
>> StreamIDs. However, we have to apply a mask (the ARM SMMU spec allows
>> for this) to the StreamIDs due to limited number of StreamID 2 Context
>> Bank entries in the SMMU. If my understanding is correct we would
>> represent this in the DT like this:
>>
>> 	iommu {
>> 		#address-cells = <2>;
>> 		#size-cells = <0>;
>> 	};
>>
>> 	master at a {
>> 		...
>> 		iommus = <&iommu StreamID0 MASK0>,
>> 			 <&iommu StreamID1 MASK1>,
>> 			 <&iommu StreamID2 MASK2>;
>> 	};
> 
> Stupid question, but why not simply describe the masked IDs? What use does
> the `raw' ID have to Linux?

We do describe the masked StreamID (SID) but we need to specify the mask
that the SMMU should apply to the incoming SIDs, right?

We have a bus master that emits 43 unique SIDs. However, we have only 40
SMMU_SMRn registers in the SMMU. So we need to mask out some of the
incoming SID bits so that the 43 SIDs can match one of 40 entries in the
SMR.

Thanks,

Olav Haugan

-- 
The Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
hosted by The Linux Foundation

[PATCH v2] devicetree: Add generic IOMMU device tree bindings

[Will Deacon]

On Sat, Jun 21, 2014 at 12:16:25AM +0100, Olav Haugan wrote:
> On 5/30/2014 12:06 PM, Arnd Bergmann wrote:
> > On Friday 30 May 2014 08:16:05 Rob Herring wrote:
> >> Presumably the ID would be the streamID on ARM's SMMU. How would a
> >> master with 8 streamIDs be described? This is what Calxeda midway has
> >> for SATA and I would expect that to be somewhat common. Either you
> >> need some ID masking or you'll have lots of duplication when you have
> >> windows.
> > 
> > I don't understand the problem. If you have stream IDs 0 through 7,
> > you would have
> > 
> > 	master at a {
> > 		...
> > 		iommus = <&smmu 0>;
> > 	};
> > 
> > 	master at b {
> > 		...
> > 		iommus = <&smmu 1;
> > 	};
> > 
> > 	...
> > 
> > 	master at 12 {
> > 		...
> > 		iommus = <&smmu 7;
> > 	};
> > 
> > and you don't need a window at all. Why would you need a mask of
> > some sort?
> 
> We have multiple-master SMMUs and each master emits a variable number of
> StreamIDs. However, we have to apply a mask (the ARM SMMU spec allows
> for this) to the StreamIDs due to limited number of StreamID 2 Context
> Bank entries in the SMMU. If my understanding is correct we would
> represent this in the DT like this:
> 
> 	iommu {
> 		#address-cells = <2>;
> 		#size-cells = <0>;
> 	};
> 
> 	master at a {
> 		...
> 		iommus = <&iommu StreamID0 MASK0>,
> 			 <&iommu StreamID1 MASK1>,
> 			 <&iommu StreamID2 MASK2>;
> 	};

Stupid question, but why not simply describe the masked IDs? What use does
the `raw' ID have to Linux?

Will

[PATCH v2] devicetree: Add generic IOMMU device tree bindings

[Olav Haugan]

On 5/30/2014 12:06 PM, Arnd Bergmann wrote:
> On Friday 30 May 2014 08:16:05 Rob Herring wrote:
>> On Fri, May 23, 2014 at 3:33 PM, Thierry Reding
>> <thierry.reding@gmail.com> wrote:
>>> From: Thierry Reding <treding@nvidia.com>
>>> +IOMMU master node:
>>> +==================
>>> +
>>> +Devices that access memory through an IOMMU are called masters. A device can
>>> +have multiple master interfaces (to one or more IOMMU devices).
>>> +
>>> +Required properties:
>>> +--------------------
>>> +- iommus: A list of phandle and IOMMU specifier pairs that describe the IOMMU
>>> +  master interfaces of the device. One entry in the list describes one master
>>> +  interface of the device.
>>> +
>>> +When an "iommus" property is specified in a device tree node, the IOMMU will
>>> +be used for address translation. If a "dma-ranges" property exists in the
>>> +device's parent node it will be ignored. An exception to this rule is if the
>>> +referenced IOMMU is disabled, in which case the "dma-ranges" property of the
>>> +parent shall take effect.
>>
>> Just thinking out loud, could you have dma-ranges in the iommu node
>> for the case when the iommu is enabled rather than putting the DMA
>> window information into the iommus property?
>>
>> This would probably mean that you need both #iommu-cells and #address-cells.
> 
> The reason for doing like this was that you may need a different window
> for each device, while there can only be one dma-ranges property in 
> an iommu node.
> 
>>> +
>>> +Optional properties:
>>> +--------------------
>>> +- iommu-names: A list of names identifying each entry in the "iommus"
>>> +  property.
>>
>> Do we really need a name here? I would not expect that you have
>> clearly documented names here from the datasheet like you would for
>> interrupts or clocks, so you'd just be making up names. Sorry, but I'm
>> not a fan of names properties in general.
> 
> Good point, this was really overdesign by modeling it after other
> subsystems that can have a use for names.
>  
>>> +Multiple-master IOMMU:
>>> +----------------------
>>> +
>>> +       iommu {
>>> +               /* the specifier represents the ID of the master */
>>> +               #address-cells = <1>;
>>> +               #size-cells = <0>;
>>> +       };
>>> +
>>> +       master {
>>> +               /* device has master ID 42 in the IOMMU */
>>> +               iommus = <&/iommu 42>;
>>> +       };
>>
>> Presumably the ID would be the streamID on ARM's SMMU. How would a
>> master with 8 streamIDs be described? This is what Calxeda midway has
>> for SATA and I would expect that to be somewhat common. Either you
>> need some ID masking or you'll have lots of duplication when you have
>> windows.
> 
> I don't understand the problem. If you have stream IDs 0 through 7,
> you would have
> 
> 	master at a {
> 		...
> 		iommus = <&smmu 0>;
> 	};
> 
> 	master at b {
> 		...
> 		iommus = <&smmu 1;
> 	};
> 
> 	...
> 
> 	master at 12 {
> 		...
> 		iommus = <&smmu 7;
> 	};
> 
> and you don't need a window at all. Why would you need a mask of
> some sort?

We have multiple-master SMMUs and each master emits a variable number of
StreamIDs. However, we have to apply a mask (the ARM SMMU spec allows
for this) to the StreamIDs due to limited number of StreamID 2 Context
Bank entries in the SMMU. If my understanding is correct we would
represent this in the DT like this:

	iommu {
		#address-cells = <2>;
		#size-cells = <0>;
	};

	master at a {
		...
		iommus = <&iommu StreamID0 MASK0>,
			 <&iommu StreamID1 MASK1>,
			 <&iommu StreamID2 MASK2>;
	};

	master at b {
		...
		iommus = <&iommu StreamID3 MASK3>,
			 <&iommu StreamID4 MASK4>;
	};


Thanks,

Olav Haugan

-- 
The Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
hosted by The Linux Foundation

[PATCH v2] devicetree: Add generic IOMMU device tree bindings

[Arnd Bergmann]

On Friday 20 June 2014 18:50:51 Will Deacon wrote:
> On Fri, Jun 20, 2014 at 04:53:08PM +0100, Arnd Bergmann wrote:
> > On Wednesday 18 June 2014 11:14:39 Will Deacon wrote:
> > > On Wed, Jun 18, 2014 at 12:37:16AM +0100, Thierry Reding wrote:
> > >   - Each master has a set of fixed StreamIDs
> > >   - StreamIDs can be remastered by adding a constant offset (this could also
> > >     be used to describe RequesterID -> StreamID mapping)
> > > 
> > > I'd hope this would be sufficient for most people. Dynamic ID assignment can
> > > be worked out later (I'm not even sure it belongs in this binding) and any
> > > mappings other than `add a constant offset' can be treated on a case-by-case
> > > basis. We don't want to throw the kitchen sink at a language for describing
> > > arbitrary transformations!
> > > 
> > > > We've had similar discussions before (power sequences anyone?) where we
> > > > tried to come up with a generic way to describe something in device tree
> > > > that just didn't work out too well. Some things are better done in code,
> > > > so I think we should at least consider that possibility rather than
> > > > blindly try and force everything into device tree.
> > > 
> > > If we can support 90% of SoCs with a simple DT-based description, we can
> > > address the corner cases as they arise. I'm not ruling our hardcoding
> > > topology if we have no choice, but I don't think that's a healthy place to
> > > start from.
> > 
> > So we could use the "arm,gicv3" comaptible string for all those that
> > have a relatively simple mapping, and describe that mapping entirely
> > in DT properties, but use a different compatible string for those
> > SoCs that have a mapping which we can't easily describe, and then
> > put that into code?
> 
> That doesn't sound unreasonable, but I don't think we should commit to
> putting things into code until they come along and we can't describe them.

Yes, that makes sense. But it's good to know that we have the option so
we don't have to cover all corner cases with the binding.

	Arnd

[PATCH v2] devicetree: Add generic IOMMU device tree bindings

[Will Deacon]

On Fri, Jun 20, 2014 at 04:53:08PM +0100, Arnd Bergmann wrote:
> On Wednesday 18 June 2014 11:14:39 Will Deacon wrote:
> > On Wed, Jun 18, 2014 at 12:37:16AM +0100, Thierry Reding wrote:
> >   - Each master has a set of fixed StreamIDs
> >   - StreamIDs can be remastered by adding a constant offset (this could also
> >     be used to describe RequesterID -> StreamID mapping)
> > 
> > I'd hope this would be sufficient for most people. Dynamic ID assignment can
> > be worked out later (I'm not even sure it belongs in this binding) and any
> > mappings other than `add a constant offset' can be treated on a case-by-case
> > basis. We don't want to throw the kitchen sink at a language for describing
> > arbitrary transformations!
> > 
> > > We've had similar discussions before (power sequences anyone?) where we
> > > tried to come up with a generic way to describe something in device tree
> > > that just didn't work out too well. Some things are better done in code,
> > > so I think we should at least consider that possibility rather than
> > > blindly try and force everything into device tree.
> > 
> > If we can support 90% of SoCs with a simple DT-based description, we can
> > address the corner cases as they arise. I'm not ruling our hardcoding
> > topology if we have no choice, but I don't think that's a healthy place to
> > start from.
> 
> So we could use the "arm,gicv3" comaptible string for all those that
> have a relatively simple mapping, and describe that mapping entirely
> in DT properties, but use a different compatible string for those
> SoCs that have a mapping which we can't easily describe, and then
> put that into code?

That doesn't sound unreasonable, but I don't think we should commit to
putting things into code until they come along and we can't describe them.

Will

[PATCH v2] devicetree: Add generic IOMMU device tree bindings

[Arnd Bergmann]

On Wednesday 18 June 2014 11:14:39 Will Deacon wrote:
> On Wed, Jun 18, 2014 at 12:37:16AM +0100, Thierry Reding wrote:
>   - Each master has a set of fixed StreamIDs
>   - StreamIDs can be remastered by adding a constant offset (this could also
>     be used to describe RequesterID -> StreamID mapping)
> 
> I'd hope this would be sufficient for most people. Dynamic ID assignment can
> be worked out later (I'm not even sure it belongs in this binding) and any
> mappings other than `add a constant offset' can be treated on a case-by-case
> basis. We don't want to throw the kitchen sink at a language for describing
> arbitrary transformations!
> 
> > We've had similar discussions before (power sequences anyone?) where we
> > tried to come up with a generic way to describe something in device tree
> > that just didn't work out too well. Some things are better done in code,
> > so I think we should at least consider that possibility rather than
> > blindly try and force everything into device tree.
> 
> If we can support 90% of SoCs with a simple DT-based description, we can
> address the corner cases as they arise. I'm not ruling our hardcoding
> topology if we have no choice, but I don't think that's a healthy place to
> start from.

So we could use the "arm,gicv3" comaptible string for all those that
have a relatively simple mapping, and describe that mapping entirely
in DT properties, but use a different compatible string for those
SoCs that have a mapping which we can't easily describe, and then
put that into code?

	Arnd

[PATCH v2] devicetree: Add generic IOMMU device tree bindings

[Will Deacon]

On Wed, Jun 18, 2014 at 12:37:16AM +0100, Thierry Reding wrote:
> On Tue, Jun 17, 2014 at 01:18:11PM +0100, Will Deacon wrote:
> > On Tue, Jun 17, 2014 at 12:58:30PM +0100, Thierry Reding wrote:
> > > On Mon, Jun 16, 2014 at 01:57:04PM +0100, Will Deacon wrote:
> > > > On Wed, Jun 04, 2014 at 10:12:38PM +0100, Thierry Reding wrote:
> > > > > It can easily be argued that if the algorithm used to remap the ID
> > > > > varies, the compatibility of the device changes. Therefore I would
> > > > > expect any variant of the GICv3 that deviates from the "standard"
> > > > > mapping (if there is such a thing) to have its own compatible string.
> > > > 
> > > > There is no standard mapping; it's a property defined at system integration
> > > > time. I fully expect different SoCs to do different things here.
> > > 
> > > My point was that the mapping itself seems to be fundamental enough to
> > > make devices with different mappings "incompatible". Therefore I think
> > > this could probably be handled by using different compatible values,
> > > something along the lines of this:
> > > 
> > > 	compatible = "vendor,soc-gicv3", "arm,gicv3";
> > > 
> > > Then the mapping can be described in code, which should be a whole lot
> > > easier and more flexible than a more or less generic notation in device
> > > tree.
> > 
> > I don't think that scales well beyond a handful of unique mappings, and I
> > really anticipate everybody doing something different based on their
> > integration constraints.
> > 
> > You'd very quickly end up with sets of tables for each SoC, describing the
> > topology and associated IDs in the kernel source, which feels like a giant
> > step backwards from where we are today with device tree.
> 
> Well, today we don't have a generic binding at all, so anything will
> really be a giant step forward in my opinion.

Oh, I'm not disputing that at all. I just think it's worth considering how
we can extend the binding in future to describe some of the ID routing and
remapping that we've discussed.

> But seriously, from what you said earlier I got the impression that some
> of the mappings may not be easy or possible to represent in DT, which is
> why I proposed to encode it into the compatible property so that it can
> be handled in code instead.

For any old arbitrary mappings, we're going to have a rough time, but that
doesn't mean we need to support that until we're forced to (and then we can
consider our options). What I *do* think we need to describe is:

  - Each master has a set of fixed StreamIDs
  - StreamIDs can be remastered by adding a constant offset (this could also
    be used to describe RequesterID -> StreamID mapping)

I'd hope this would be sufficient for most people. Dynamic ID assignment can
be worked out later (I'm not even sure it belongs in this binding) and any
mappings other than `add a constant offset' can be treated on a case-by-case
basis. We don't want to throw the kitchen sink at a language for describing
arbitrary transformations!

> We've had similar discussions before (power sequences anyone?) where we
> tried to come up with a generic way to describe something in device tree
> that just didn't work out too well. Some things are better done in code,
> so I think we should at least consider that possibility rather than
> blindly try and force everything into device tree.

If we can support 90% of SoCs with a simple DT-based description, we can
address the corner cases as they arise. I'm not ruling our hardcoding
topology if we have no choice, but I don't think that's a healthy place to
start from.

Will

[PATCH v2] devicetree: Add generic IOMMU device tree bindings

[Will Deacon]

On Tue, Jun 17, 2014 at 03:50:25PM +0100, Stuart Yoder wrote:
> > > I think for simple masters (i.e. those that have all their StreamIDs
> > > under control of one driver), then setting something during attach (or
> > > add?) based on the DT could work pretty well. The other case is when we
> > > have masters behind a bridge (such as a PCI RC). In this case, it might
> > > actually be better to ask the bridge for the IDs and let it sort out
> > the
> > > allocation itself. That would also move the RequesterID -> StreamID
> > > mapping out of the SMMU code.
> > >
> > > What do you think?
> > The PCI bus iommu group creation code would be part of the SMMU driver
> > (it is handled by other IOMMU drivers as well). My understanding is that
> > there would be one is to one correspondence between the requestor ID and
> > the iommu group. May be, we can have an API provided by the PCI bridge
> > (architecture specific) for setting the stream ID.
> 
> I think Will is suggesting something along those lines-- I think it's a
> question of where the streamID allocation happens.  You could
> either do something like the following in the SMMU driver when attaching
> a PCI device:
> 
>     id = alloc_stream_id(...);
>     pci_set_streamid(pci-dev, id);
> 
>     or
> 
>     id = pci_get_streamid(pci-dev);
> 
> ...i.e the PCI RC could allocate (from some TBD
> allocator) and set the stream ID itself.
> 
> Not sure how big a deal it is to extend PCI RC interfaces for 
> something like that.

I think both interfaces have their place; the latter for hotpluggable buses
and the former for static allocation (i.e. set once). The only way to see
how well it works is to try implementing it, I guess.

Will

[PATCH v2] devicetree: Add generic IOMMU device tree bindings

[Thierry Reding]

On Tue, Jun 17, 2014 at 01:18:11PM +0100, Will Deacon wrote:
> On Tue, Jun 17, 2014 at 12:58:30PM +0100, Thierry Reding wrote:
> > On Mon, Jun 16, 2014 at 01:57:04PM +0100, Will Deacon wrote:
> > > On Wed, Jun 04, 2014 at 10:12:38PM +0100, Thierry Reding wrote:
> > > > It can easily be argued that if the algorithm used to remap the ID
> > > > varies, the compatibility of the device changes. Therefore I would
> > > > expect any variant of the GICv3 that deviates from the "standard"
> > > > mapping (if there is such a thing) to have its own compatible string.
> > > 
> > > There is no standard mapping; it's a property defined at system integration
> > > time. I fully expect different SoCs to do different things here.
> > 
> > My point was that the mapping itself seems to be fundamental enough to
> > make devices with different mappings "incompatible". Therefore I think
> > this could probably be handled by using different compatible values,
> > something along the lines of this:
> > 
> > 	compatible = "vendor,soc-gicv3", "arm,gicv3";
> > 
> > Then the mapping can be described in code, which should be a whole lot
> > easier and more flexible than a more or less generic notation in device
> > tree.
> 
> I don't think that scales well beyond a handful of unique mappings, and I
> really anticipate everybody doing something different based on their
> integration constraints.
> 
> You'd very quickly end up with sets of tables for each SoC, describing the
> topology and associated IDs in the kernel source, which feels like a giant
> step backwards from where we are today with device tree.

Well, today we don't have a generic binding at all, so anything will
really be a giant step forward in my opinion.

But seriously, from what you said earlier I got the impression that some
of the mappings may not be easy or possible to represent in DT, which is
why I proposed to encode it into the compatible property so that it can
be handled in code instead.

We've had similar discussions before (power sequences anyone?) where we
tried to come up with a generic way to describe something in device tree
that just didn't work out too well. Some things are better done in code,
so I think we should at least consider that possibility rather than
blindly try and force everything into device tree.

Thierry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20140618/1a18178b/attachment.sig>

[PATCH v2] devicetree: Add generic IOMMU device tree bindings

[Stuart Yoder]

> -----Original Message-----
> From: Sethi Varun-B16395
> Sent: Tuesday, June 17, 2014 6:22 AM
> To: Will Deacon
> Cc: Mark Rutland; devicetree at vger.kernel.org; linux-samsung-
> soc at vger.kernel.org; Arnd Bergmann; Pawel Moll; Ian Campbell; Grant
> Grundler; Stephen Warren; Yoder Stuart-B08248; Rob Herring; linux-
> kernel at vger.kernel.org; Marc Zyngier; Linux IOMMU; Thierry Reding; Kumar
> Gala; linux-tegra at vger.kernel.org; Cho KyongHo; Dave P Martin; linux-arm-
> kernel at lists.infradead.org
> Subject: RE: [PATCH v2] devicetree: Add generic IOMMU device tree
> bindings
> 
> 
> 
> > -----Original Message-----
> > From: iommu-bounces at lists.linux-foundation.org [mailto:iommu-
> > bounces at lists.linux-foundation.org] On Behalf Of Will Deacon
> > Sent: Tuesday, June 17, 2014 4:13 PM
> > To: Sethi Varun-B16395
> > Cc: Mark Rutland; devicetree at vger.kernel.org; linux-samsung-
> > soc at vger.kernel.org; Arnd Bergmann; Pawel Moll; Ian Campbell; Grant
> > Grundler; Stephen Warren; Yoder Stuart-B08248; Rob Herring; linux-
> > kernel at vger.kernel.org; Marc Zyngier; Linux IOMMU; Thierry Reding;
> Kumar
> > Gala; linux-tegra at vger.kernel.org; Cho KyongHo; Dave P Martin; linux-
> arm-
> > kernel at lists.infradead.org
> > Subject: Re: [PATCH v2] devicetree: Add generic IOMMU device tree
> > bindings
> >
> > On Tue, Jun 17, 2014 at 11:26:48AM +0100, Varun Sethi wrote:
> > > > The way we generally thought it would work was something like
> > > > this:
> > > >    -u-boot/bootloader makes any static streamID allocation if
> needed,
> > > >     sets a default streamID  (e.g. 0x0) in device and expresses
> > > >     that in the device tree
> > > >    -device tree would express relationship between devices
> > > >     (including bus controllers) and the SMMU through mmu-masters
> > > >     property
> > > >    -u-boot would express the range of unused (or used) streamIDs
> via
> > > > a new
> > > >     device tree property so the kernel SMMU driver knows what
> > > > streamIDs are
> > > >     free
> > > >    -in the SMMU driver a different vendor specific 'add_device'
> > callback
> > > >     could be used to handle our special cases where we need to
> > set/change
> > > >     the stream ID for devices added to a domain
> > >
> > > Another possibility, could be to program the stream Id in the device
> > > registers (reference for the stream ID register can be obtained from
> > > the device tree) during device attach. This could be relevant in case
> > > of VFIO, when we are assigning multiple devices to a single VM. All
> > > the devices can share the same stream ID.
> >
> > I think for simple masters (i.e. those that have all their StreamIDs
> > under control of one driver), then setting something during attach (or
> > add?) based on the DT could work pretty well. The other case is when we
> > have masters behind a bridge (such as a PCI RC). In this case, it might
> > actually be better to ask the bridge for the IDs and let it sort out
> the
> > allocation itself. That would also move the RequesterID -> StreamID
> > mapping out of the SMMU code.
> >
> > What do you think?
> The PCI bus iommu group creation code would be part of the SMMU driver
> (it is handled by other IOMMU drivers as well). My understanding is that
> there would be one is to one correspondence between the requestor ID and
> the iommu group. May be, we can have an API provided by the PCI bridge
> (architecture specific) for setting the stream ID.

I think Will is suggesting something along those lines-- I think it's a
question of where the streamID allocation happens.  You could
either do something like the following in the SMMU driver when attaching
a PCI device:

    id = alloc_stream_id(...);
    pci_set_streamid(pci-dev, id);

    or

    id = pci_get_streamid(pci-dev);

...i.e the PCI RC could allocate (from some TBD
allocator) and set the stream ID itself.

Not sure how big a deal it is to extend PCI RC interfaces for 
something like that.

Stuart

[PATCH v2] devicetree: Add generic IOMMU device tree bindings

[Stuart Yoder]

> -----Original Message-----
> From: Sethi Varun-B16395
> Sent: Tuesday, June 17, 2014 5:27 AM
> To: Yoder Stuart-B08248; Will Deacon
> Cc: Thierry Reding; Mark Rutland; devicetree at vger.kernel.org; linux-
> samsung-soc at vger.kernel.org; Pawel Moll; Arnd Bergmann; Ian Campbell;
> Grant Grundler; Stephen Warren; linux-kernel at vger.kernel.org; Marc
> Zyngier; Linux IOMMU; Rob Herring; Kumar Gala; linux-
> tegra at vger.kernel.org; Cho KyongHo; Dave P Martin; linux-arm-
> kernel at lists.infradead.org
> Subject: RE: [PATCH v2] devicetree: Add generic IOMMU device tree
> bindings
> 
> 
> 
> > -----Original Message-----
> > From: Yoder Stuart-B08248
> > Sent: Tuesday, June 17, 2014 12:24 AM
> > To: Will Deacon
> > Cc: Sethi Varun-B16395; Thierry Reding; Mark Rutland;
> > devicetree at vger.kernel.org; linux-samsung-soc at vger.kernel.org; Pawel
> > Moll; Arnd Bergmann; Ian Campbell; Grant Grundler; Stephen Warren;
> linux-
> > kernel at vger.kernel.org; Marc Zyngier; Linux IOMMU; Rob Herring; Kumar
> > Gala; linux-tegra at vger.kernel.org; Cho KyongHo; Dave P Martin; linux-
> arm-
> > kernel at lists.infradead.org
> > Subject: RE: [PATCH v2] devicetree: Add generic IOMMU device tree
> > bindings
> >
> >
> >
> > > -----Original Message-----
> > > From: Will Deacon [mailto:will.deacon at arm.com]
> > > Sent: Monday, June 16, 2014 12:04 PM
> > > To: Yoder Stuart-B08248
> > > Cc: Sethi Varun-B16395; Thierry Reding; Mark Rutland;
> > > devicetree at vger.kernel.org; linux-samsung-soc at vger.kernel.org; Pawel
> > > Moll; Arnd Bergmann; Ian Campbell; Grant Grundler; Stephen Warren;
> > > linux- kernel at vger.kernel.org; Marc Zyngier; Linux IOMMU; Rob
> Herring;
> > > Kumar Gala; linux-tegra at vger.kernel.org; Cho KyongHo; Dave P Martin;
> > > linux-arm- kernel at lists.infradead.org
> > > Subject: Re: [PATCH v2] devicetree: Add generic IOMMU device tree
> > > bindings
> > >
> > > Hi Stuart,
> > >
> > > On Mon, Jun 16, 2014 at 05:56:32PM +0100, Stuart Yoder wrote:
> > > > > Do you have use-cases where you really need to change these
> > > > > mappings dynamically?
> > > >
> > > > Yes.  In the case of a PCI bus-- you may not know in advance how
> many
> > > > PCI devices there are until you probe the bus.   We have another
> FSL
> > > > proprietary bus we call the "fsl-mc" bus that is similar.
> > >
> > > For that case, though, you could still describe an algorithmic
> > > transformation from RequesterID to StreamID which corresponds to a
> > > fixed mapping.
> > >
> > > > Another thing to consider-- starting with SMMUv2, as you know,
> there
> > > > is a new distributed architecture with multiple TBUs and a
> > > > centralized TCU that walks the SMMU page tables.  So instead of
> > > > sprinkling multiple SMMUs all over an SoC you now have the option a
> > > > 1 central TCU and
> > > sprinkling
> > > > multiple TBUs around.   However, this means that the stream ID
> > > namespace
> > > > is now global and can be pretty limited.  In the SMMU
> implementation
> > > > we have there are only 64 stream ID total for our Soc.  But we have
> > > > many
> > > more
> > > > masters than that.
> > > >
> > > > So we look at stream IDs as really corresponding to an 'isolation
> > > context'
> > > > and not to a bus master.  An isolation context is the domain you
> are
> > > > trying to isolate with the SMMU.  Devices that all belong to the
> > > > same 'isolation context' can share the same stream ID, since they
> > > > share the same domain and page tables.
> > >
> > > Ok, this is more compelling.
> > >
> > > > So, perhaps by default some/most SMMU masters may have a default
> > > > stream
> > > ID
> > > > of 0x0 that is used by the host...and that could be represented
> > > > statically in the device tree.
> > > >
> > > > But, we absolutely will need to dynamically set new stream IDs into
> > > > masters when a new IOMMU 'domain' is created and devices
> > > > are added to it.   All the devices in a domain will share
> > > > the same stream ID.
> > > >
> > > > So whatever we do, let's please have an architecture flexible
> enough
> > > > to allow for this.
> > >
> > > What is the software interface to the logic that assigns the
> StreamIDs?
> > > Is
> > > it part of the SMMU, or a separate device (or set of devices)?
> >
> > For us at the hardware level there are a few different ways that the
> > streamIDs can be set.  It is not part of the SMMU.  In the cases where
> > there is simply
> > 1 device to 1 streamID (e.g. USB controller) there is an SoC register
> > where
> > you just program the stream ID.   In the case of PCI, our PCI
> controller
> > has a RequesterID-to-streamID table that you set via some PCI
> controller
> > registers.
> >
> > The way we generally thought it would work was something like
> > this:
> >    -u-boot/bootloader makes any static streamID allocation if needed,
> >     sets a default streamID  (e.g. 0x0) in device and expresses
> >     that in the device tree
> >    -device tree would express relationship between devices
> >     (including bus controllers) and the SMMU through mmu-masters
> >     property
> >    -u-boot would express the range of unused (or used) streamIDs via a
> > new
> >     device tree property so the kernel SMMU driver knows what streamIDs
> > are
> >     free
> >    -in the SMMU driver a different vendor specific 'add_device'
> callback
> >     could be used to handle our special cases where we need to
> set/change
> >     the stream ID for devices added to a domain
> 
> Another possibility, could be to program the stream Id in the device
> registers (reference for the stream ID register can be obtained from the
> device tree) during device attach. This could be relevant in case of
> VFIO, when we are assigning multiple devices to a single VM. All the
> devices can share the same stream ID.

Actually, that is what I meant-- do the special case handling during
device "attach" (not add).

Stuart

[PATCH v2] devicetree: Add generic IOMMU device tree bindings

[Will Deacon]

On Tue, Jun 17, 2014 at 12:58:30PM +0100, Thierry Reding wrote:
> On Mon, Jun 16, 2014 at 01:57:04PM +0100, Will Deacon wrote:
> > On Wed, Jun 04, 2014 at 10:12:38PM +0100, Thierry Reding wrote:
> > > It can easily be argued that if the algorithm used to remap the ID
> > > varies, the compatibility of the device changes. Therefore I would
> > > expect any variant of the GICv3 that deviates from the "standard"
> > > mapping (if there is such a thing) to have its own compatible string.
> > 
> > There is no standard mapping; it's a property defined at system integration
> > time. I fully expect different SoCs to do different things here.
> 
> My point was that the mapping itself seems to be fundamental enough to
> make devices with different mappings "incompatible". Therefore I think
> this could probably be handled by using different compatible values,
> something along the lines of this:
> 
> 	compatible = "vendor,soc-gicv3", "arm,gicv3";
> 
> Then the mapping can be described in code, which should be a whole lot
> easier and more flexible than a more or less generic notation in device
> tree.

I don't think that scales well beyond a handful of unique mappings, and I
really anticipate everybody doing something different based on their
integration constraints.

You'd very quickly end up with sets of tables for each SoC, describing the
topology and associated IDs in the kernel source, which feels like a giant
step backwards from where we are today with device tree.

If, for example, the GIC architecture prescribed a fixed set of Device IDs
and an algorithm for converting from Stream IDs then your approach may have
some merits, but that's not where we are today (and I also don't think it's
practical to try and enforce such system-wide properties into an interrupt
controller architecture).

Will

[PATCH v2] devicetree: Add generic IOMMU device tree bindings

[Thierry Reding]

On Mon, Jun 16, 2014 at 01:57:04PM +0100, Will Deacon wrote:
> On Wed, Jun 04, 2014 at 10:12:38PM +0100, Thierry Reding wrote:
> > On Fri, May 30, 2014 at 12:27:28PM +0100, Dave Martin wrote:
> > > On Fri, May 30, 2014 at 08:30:08AM +0100, Thierry Reding wrote:
> > [...]
> > > > Arnd, can you take another look at this binding and see if there's
> > > > anything else missing? If not I'll go through the document again and
> > > > update all #address-cells/#size-cells references with #iommu-cells as
> > > > appropriate and submit v3.
> > > 
> > > How do you envisage propagation of the master ID bits downstream of the
> > > IOMMU would be described?
> > > 
> > > We will definitely need a way to describe this for GICv3.  How those
> > > values are propagated is likely to vary between related SoCs and doesn't
> > > feel like it should be baked into a driver, especially for the ARM SMMU
> > > which may get reused in radically different SoC families from different
> > > vendors.
> > 
> > Well, we've had cases like these in the past (power sequences come to
> > mind). Some concepts are just too difficult or unwieldy to be put into
> > device tree. I think that this is one of them.
> > 
> > > The most likely types of remapping are the adding of a base offset or
> > > some extra bits to the ID -- because not all MSIs to the GIC will
> > > necessarily pass through the IOMMU.  It's also possible that we might
> > > see ID squashing or folding in some systems.
> > 
> > It can easily be argued that if the algorithm used to remap the ID
> > varies, the compatibility of the device changes. Therefore I would
> > expect any variant of the GICv3 that deviates from the "standard"
> > mapping (if there is such a thing) to have its own compatible string.
> 
> There is no standard mapping; it's a property defined at system integration
> time. I fully expect different SoCs to do different things here.

My point was that the mapping itself seems to be fundamental enough to
make devices with different mappings "incompatible". Therefore I think
this could probably be handled by using different compatible values,
something along the lines of this:

	compatible = "vendor,soc-gicv3", "arm,gicv3";

Then the mapping can be described in code, which should be a whole lot
easier and more flexible than a more or less generic notation in device
tree.

Thierry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20140617/957f757f/attachment.sig>

[PATCH v2] devicetree: Add generic IOMMU device tree bindings

[Varun Sethi]

> -----Original Message-----
> From: iommu-bounces at lists.linux-foundation.org [mailto:iommu-
> bounces at lists.linux-foundation.org] On Behalf Of Will Deacon
> Sent: Tuesday, June 17, 2014 4:13 PM
> To: Sethi Varun-B16395
> Cc: Mark Rutland; devicetree at vger.kernel.org; linux-samsung-
> soc at vger.kernel.org; Arnd Bergmann; Pawel Moll; Ian Campbell; Grant
> Grundler; Stephen Warren; Yoder Stuart-B08248; Rob Herring; linux-
> kernel at vger.kernel.org; Marc Zyngier; Linux IOMMU; Thierry Reding; Kumar
> Gala; linux-tegra at vger.kernel.org; Cho KyongHo; Dave P Martin; linux-arm-
> kernel at lists.infradead.org
> Subject: Re: [PATCH v2] devicetree: Add generic IOMMU device tree
> bindings
> 
> On Tue, Jun 17, 2014 at 11:26:48AM +0100, Varun Sethi wrote:
> > > The way we generally thought it would work was something like
> > > this:
> > >    -u-boot/bootloader makes any static streamID allocation if needed,
> > >     sets a default streamID  (e.g. 0x0) in device and expresses
> > >     that in the device tree
> > >    -device tree would express relationship between devices
> > >     (including bus controllers) and the SMMU through mmu-masters
> > >     property
> > >    -u-boot would express the range of unused (or used) streamIDs via
> > > a new
> > >     device tree property so the kernel SMMU driver knows what
> > > streamIDs are
> > >     free
> > >    -in the SMMU driver a different vendor specific 'add_device'
> callback
> > >     could be used to handle our special cases where we need to
> set/change
> > >     the stream ID for devices added to a domain
> >
> > Another possibility, could be to program the stream Id in the device
> > registers (reference for the stream ID register can be obtained from
> > the device tree) during device attach. This could be relevant in case
> > of VFIO, when we are assigning multiple devices to a single VM. All
> > the devices can share the same stream ID.
> 
> I think for simple masters (i.e. those that have all their StreamIDs
> under control of one driver), then setting something during attach (or
> add?) based on the DT could work pretty well. The other case is when we
> have masters behind a bridge (such as a PCI RC). In this case, it might
> actually be better to ask the bridge for the IDs and let it sort out the
> allocation itself. That would also move the RequesterID -> StreamID
> mapping out of the SMMU code.
> 
> What do you think?
The PCI bus iommu group creation code would be part of the SMMU driver (it is handled by other IOMMU drivers as well). My understanding is that there would be one is to one correspondence between the requestor ID and the iommu group. May be, we can have an API provided by the PCI bridge (architecture specific) for setting the stream ID.

-Varun

[PATCH v2] devicetree: Add generic IOMMU device tree bindings

[Will Deacon]

On Tue, Jun 17, 2014 at 11:26:48AM +0100, Varun Sethi wrote:
> > The way we generally thought it would work was something like
> > this:
> >    -u-boot/bootloader makes any static streamID allocation if needed,
> >     sets a default streamID  (e.g. 0x0) in device and expresses
> >     that in the device tree
> >    -device tree would express relationship between devices
> >     (including bus controllers) and the SMMU through mmu-masters
> >     property
> >    -u-boot would express the range of unused (or used) streamIDs via a
> > new
> >     device tree property so the kernel SMMU driver knows what streamIDs
> > are
> >     free
> >    -in the SMMU driver a different vendor specific 'add_device' callback
> >     could be used to handle our special cases where we need to set/change
> >     the stream ID for devices added to a domain
> 
> Another possibility, could be to program the stream Id in the device
> registers (reference for the stream ID register can be obtained from the
> device tree) during device attach. This could be relevant in case of VFIO,
> when we are assigning multiple devices to a single VM. All the devices can
> share the same stream ID.

I think for simple masters (i.e. those that have all their StreamIDs under
control of one driver), then setting something during attach (or add?)
based on the DT could work pretty well. The other case is when we have
masters behind a bridge (such as a PCI RC). In this case, it might actually
be better to ask the bridge for the IDs and let it sort out the allocation
itself. That would also move the RequesterID -> StreamID mapping out of
the SMMU code.

What do you think?

Will

[PATCH v2] devicetree: Add generic IOMMU device tree bindings

[Varun Sethi]

> -----Original Message-----
> From: Yoder Stuart-B08248
> Sent: Tuesday, June 17, 2014 12:24 AM
> To: Will Deacon
> Cc: Sethi Varun-B16395; Thierry Reding; Mark Rutland;
> devicetree at vger.kernel.org; linux-samsung-soc at vger.kernel.org; Pawel
> Moll; Arnd Bergmann; Ian Campbell; Grant Grundler; Stephen Warren; linux-
> kernel at vger.kernel.org; Marc Zyngier; Linux IOMMU; Rob Herring; Kumar
> Gala; linux-tegra at vger.kernel.org; Cho KyongHo; Dave P Martin; linux-arm-
> kernel at lists.infradead.org
> Subject: RE: [PATCH v2] devicetree: Add generic IOMMU device tree
> bindings
> 
> 
> 
> > -----Original Message-----
> > From: Will Deacon [mailto:will.deacon at arm.com]
> > Sent: Monday, June 16, 2014 12:04 PM
> > To: Yoder Stuart-B08248
> > Cc: Sethi Varun-B16395; Thierry Reding; Mark Rutland;
> > devicetree at vger.kernel.org; linux-samsung-soc at vger.kernel.org; Pawel
> > Moll; Arnd Bergmann; Ian Campbell; Grant Grundler; Stephen Warren;
> > linux- kernel at vger.kernel.org; Marc Zyngier; Linux IOMMU; Rob Herring;
> > Kumar Gala; linux-tegra at vger.kernel.org; Cho KyongHo; Dave P Martin;
> > linux-arm- kernel at lists.infradead.org
> > Subject: Re: [PATCH v2] devicetree: Add generic IOMMU device tree
> > bindings
> >
> > Hi Stuart,
> >
> > On Mon, Jun 16, 2014 at 05:56:32PM +0100, Stuart Yoder wrote:
> > > > Do you have use-cases where you really need to change these
> > > > mappings dynamically?
> > >
> > > Yes.  In the case of a PCI bus-- you may not know in advance how many
> > > PCI devices there are until you probe the bus.   We have another FSL
> > > proprietary bus we call the "fsl-mc" bus that is similar.
> >
> > For that case, though, you could still describe an algorithmic
> > transformation from RequesterID to StreamID which corresponds to a
> > fixed mapping.
> >
> > > Another thing to consider-- starting with SMMUv2, as you know, there
> > > is a new distributed architecture with multiple TBUs and a
> > > centralized TCU that walks the SMMU page tables.  So instead of
> > > sprinkling multiple SMMUs all over an SoC you now have the option a
> > > 1 central TCU and
> > sprinkling
> > > multiple TBUs around.   However, this means that the stream ID
> > namespace
> > > is now global and can be pretty limited.  In the SMMU implementation
> > > we have there are only 64 stream ID total for our Soc.  But we have
> > > many
> > more
> > > masters than that.
> > >
> > > So we look at stream IDs as really corresponding to an 'isolation
> > context'
> > > and not to a bus master.  An isolation context is the domain you are
> > > trying to isolate with the SMMU.  Devices that all belong to the
> > > same 'isolation context' can share the same stream ID, since they
> > > share the same domain and page tables.
> >
> > Ok, this is more compelling.
> >
> > > So, perhaps by default some/most SMMU masters may have a default
> > > stream
> > ID
> > > of 0x0 that is used by the host...and that could be represented
> > > statically in the device tree.
> > >
> > > But, we absolutely will need to dynamically set new stream IDs into
> > > masters when a new IOMMU 'domain' is created and devices
> > > are added to it.   All the devices in a domain will share
> > > the same stream ID.
> > >
> > > So whatever we do, let's please have an architecture flexible enough
> > > to allow for this.
> >
> > What is the software interface to the logic that assigns the StreamIDs?
> > Is
> > it part of the SMMU, or a separate device (or set of devices)?
> 
> For us at the hardware level there are a few different ways that the
> streamIDs can be set.  It is not part of the SMMU.  In the cases where
> there is simply
> 1 device to 1 streamID (e.g. USB controller) there is an SoC register
> where
> you just program the stream ID.   In the case of PCI, our PCI controller
> has a RequesterID-to-streamID table that you set via some PCI controller
> registers.
> 
> The way we generally thought it would work was something like
> this:
>    -u-boot/bootloader makes any static streamID allocation if needed,
>     sets a default streamID  (e.g. 0x0) in device and expresses
>     that in the device tree
>    -device tree would express relationship between devices
>     (including bus controllers) and the SMMU through mmu-masters
>     property
>    -u-boot would express the range of unused (or used) streamIDs via a
> new
>     device tree property so the kernel SMMU driver knows what streamIDs
> are
>     free
>    -in the SMMU driver a different vendor specific 'add_device' callback
>     could be used to handle our special cases where we need to set/change
>     the stream ID for devices added to a domain

Another possibility, could be to program the stream Id in the device registers (reference for the stream ID register can be obtained from the device tree) during device attach. This could be relevant in case of VFIO, when we are assigning multiple devices to a single VM. All the devices can share the same stream ID.

-Varun

[PATCH v2] devicetree: Add generic IOMMU device tree bindings

[Stuart Yoder]

> -----Original Message-----
> From: Will Deacon [mailto:will.deacon at arm.com]
> Sent: Monday, June 16, 2014 12:04 PM
> To: Yoder Stuart-B08248
> Cc: Sethi Varun-B16395; Thierry Reding; Mark Rutland;
> devicetree at vger.kernel.org; linux-samsung-soc at vger.kernel.org; Pawel
> Moll; Arnd Bergmann; Ian Campbell; Grant Grundler; Stephen Warren; linux-
> kernel at vger.kernel.org; Marc Zyngier; Linux IOMMU; Rob Herring; Kumar
> Gala; linux-tegra at vger.kernel.org; Cho KyongHo; Dave P Martin; linux-arm-
> kernel at lists.infradead.org
> Subject: Re: [PATCH v2] devicetree: Add generic IOMMU device tree
> bindings
> 
> Hi Stuart,
> 
> On Mon, Jun 16, 2014 at 05:56:32PM +0100, Stuart Yoder wrote:
> > > Do you have use-cases where you really need to change these mappings
> > > dynamically?
> >
> > Yes.  In the case of a PCI bus-- you may not know in advance how many
> > PCI devices there are until you probe the bus.   We have another FSL
> > proprietary bus we call the "fsl-mc" bus that is similar.
> 
> For that case, though, you could still describe an algorithmic
> transformation from RequesterID to StreamID which corresponds to a fixed
> mapping.
> 
> > Another thing to consider-- starting with SMMUv2, as you know, there
> > is a new distributed architecture with multiple TBUs and a centralized
> > TCU that walks the SMMU page tables.  So instead of sprinkling multiple
> > SMMUs all over an SoC you now have the option a 1 central TCU and
> sprinkling
> > multiple TBUs around.   However, this means that the stream ID
> namespace
> > is now global and can be pretty limited.  In the SMMU implementation we
> > have there are only 64 stream ID total for our Soc.  But we have many
> more
> > masters than that.
> >
> > So we look at stream IDs as really corresponding to an 'isolation
> context'
> > and not to a bus master.  An isolation context is the domain you are
> > trying to isolate with the SMMU.  Devices that all belong to the same
> > 'isolation context' can share the same stream ID, since they share
> > the same domain and page tables.
> 
> Ok, this is more compelling.
> 
> > So, perhaps by default some/most SMMU masters may have a default stream
> ID
> > of 0x0 that is used by the host...and that could be represented
> > statically in the device tree.
> >
> > But, we absolutely will need to dynamically set new stream IDs
> > into masters when a new IOMMU 'domain' is created and devices
> > are added to it.   All the devices in a domain will share
> > the same stream ID.
> >
> > So whatever we do, let's please have an architecture flexible enough
> > to allow for this.
> 
> What is the software interface to the logic that assigns the StreamIDs?
> Is
> it part of the SMMU, or a separate device (or set of devices)?

For us at the hardware level there are a few different ways that the streamIDs
can be set.  It is not part of the SMMU.  In the cases where there is simply
1 device to 1 streamID (e.g. USB controller) there is an SoC register where
you just program the stream ID.   In the case of PCI, our PCI controller
has a RequesterID-to-streamID table that you set via some PCI controller
registers.

The way we generally thought it would work was something like
this:
   -u-boot/bootloader makes any static streamID allocation if needed,
    sets a default streamID  (e.g. 0x0) in device and expresses
    that in the device tree
   -device tree would express relationship between devices
    (including bus controllers) and the SMMU through mmu-masters
    property
   -u-boot would express the range of unused (or used) streamIDs via a new
    device tree property so the kernel SMMU driver knows what streamIDs are
    free
   -in the SMMU driver a different vendor specific 'add_device' callback
    could be used to handle our special cases where we need to set/change
    the stream ID for devices added to a domain

Thanks,
Stuart

[PATCH v2] devicetree: Add generic IOMMU device tree bindings

[Arnd Bergmann]

On Monday 16 June 2014 18:04:16 Will Deacon wrote:
> 
> On Mon, Jun 16, 2014 at 05:56:32PM +0100, Stuart Yoder wrote:
> > > Do you have use-cases where you really need to change these mappings
> > > dynamically?
> > 
> > Yes.  In the case of a PCI bus-- you may not know in advance how many
> > PCI devices there are until you probe the bus.   We have another FSL
> > proprietary bus we call the "fsl-mc" bus that is similar.
> 
> For that case, though, you could still describe an algorithmic
> transformation from RequesterID to StreamID which corresponds to a fixed
> mapping.

It sounds to me like the best option here would be to have only the
RequesterID passed in the "iommus" property and have the StreamID
dynamically assigned. This would mean we treat the StreamID as
the context.

> > Another thing to consider-- starting with SMMUv2, as you know, there
> > is a new distributed architecture with multiple TBUs and a centralized
> > TCU that walks the SMMU page tables.  So instead of sprinkling multiple
> > SMMUs all over an SoC you now have the option a 1 central TCU and sprinkling
> > multiple TBUs around.   However, this means that the stream ID namespace
> > is now global and can be pretty limited.  In the SMMU implementation we 
> > have there are only 64 stream ID total for our Soc.  But we have many more
> > masters than that.
> > 
> > So we look at stream IDs as really corresponding to an 'isolation context'
> > and not to a bus master.  An isolation context is the domain you are
> > trying to isolate with the SMMU.  Devices that all belong to the same
> > 'isolation context' can share the same stream ID, since they share
> > the same domain and page tables.
> 
> Ok, this is more compelling.

This also makes sense from the kernel's perspective: The shared Stream ID
is the one that is used by the dma-mapping API here, while all other
Stream IDs would only be used if you have things like PCI device assignment
or GPU contexts that map into IOMMU contexts using the Linux IOMMU
abstraction, which is a lower-level interface than the dma-mapping
abstraction.

	Arnd

[PATCH v2] devicetree: Add generic IOMMU device tree bindings

[Will Deacon]

Hi Stuart,

On Mon, Jun 16, 2014 at 05:56:32PM +0100, Stuart Yoder wrote:
> > Do you have use-cases where you really need to change these mappings
> > dynamically?
> 
> Yes.  In the case of a PCI bus-- you may not know in advance how many
> PCI devices there are until you probe the bus.   We have another FSL
> proprietary bus we call the "fsl-mc" bus that is similar.

For that case, though, you could still describe an algorithmic
transformation from RequesterID to StreamID which corresponds to a fixed
mapping.

> Another thing to consider-- starting with SMMUv2, as you know, there
> is a new distributed architecture with multiple TBUs and a centralized
> TCU that walks the SMMU page tables.  So instead of sprinkling multiple
> SMMUs all over an SoC you now have the option a 1 central TCU and sprinkling
> multiple TBUs around.   However, this means that the stream ID namespace
> is now global and can be pretty limited.  In the SMMU implementation we 
> have there are only 64 stream ID total for our Soc.  But we have many more
> masters than that.
> 
> So we look at stream IDs as really corresponding to an 'isolation context'
> and not to a bus master.  An isolation context is the domain you are
> trying to isolate with the SMMU.  Devices that all belong to the same
> 'isolation context' can share the same stream ID, since they share
> the same domain and page tables.

Ok, this is more compelling.

> So, perhaps by default some/most SMMU masters may have a default stream ID
> of 0x0 that is used by the host...and that could be represented
> statically in the device tree.
> 
> But, we absolutely will need to dynamically set new stream IDs
> into masters when a new IOMMU 'domain' is created and devices
> are added to it.   All the devices in a domain will share
> the same stream ID.
> 
> So whatever we do, let's please have an architecture flexible enough
> to allow for this.

What is the software interface to the logic that assigns the StreamIDs? Is
it part of the SMMU, or a separate device (or set of devices)?

Will

[PATCH v2] devicetree: Add generic IOMMU device tree bindings

[Stuart Yoder]

> -----Original Message-----
> From: Will Deacon [mailto:will.deacon at arm.com]
> Sent: Monday, June 16, 2014 10:28 AM
> To: Sethi Varun-B16395
> Cc: Thierry Reding; Mark Rutland; devicetree at vger.kernel.org; linux-
> samsung-soc at vger.kernel.org; Pawel Moll; Arnd Bergmann; Ian Campbell;
> Grant Grundler; Stephen Warren; linux-kernel at vger.kernel.org; Marc
> Zyngier; Linux IOMMU; Rob Herring; Kumar Gala; linux-
> tegra at vger.kernel.org; Cho KyongHo; Dave P Martin; linux-arm-
> kernel at lists.infradead.org; Yoder Stuart-B08248
> Subject: Re: [PATCH v2] devicetree: Add generic IOMMU device tree
> bindings
> 
> Hi Varun,
> 
> On Thu, Jun 05, 2014 at 08:10:19PM +0100, Varun Sethi wrote:
> > > The set of StreamIDs that can be generated by a master is fixed in
> the
> > > hardware. The SMMU can then be programmed to map these incoming IDs
> onto
> > > a context ID (or a set of context IDs), which are the IDs used
> internally
> > > by the SMMU to find the page tables etc.
> > >
> > > The StreamID -> ContextID mapping is dynamic and controlled by
> software.
> > > The Master -> StreamIDs mapping is fixed in the hardware.
> > The Master -> StreamIDs mapping may not always be fixed in the
> hardware.
> > At, least in our case we plan to program these via software. PCI
> devices
> > is one place where this mapping would have to be dynamic (based on the
> > topology i.e. if the devices are connected to a bridge etc). Also, we
> have
> > a hot plug device architecture where the stream ID is software
> > programmable.
> >
> > Other than that, based on the isolation requirements (iommu domain)
> > software programmability offers greater flexibility.
> 
> For the sake of consistency, I'd really like to treat the master ->
> streamIDs mapping as fixed. If that means setting it once during boot in
> your case, then so be it (ideally in the firmware). That way, we just
> describe the fixed mapping to the kernel and the driver doesn't have to
> worry about changing the mapping, especially given that that's likely to
> be
> highly error-prone once the SMMU is in us.
> 
> Do you have use-cases where you really need to change these mappings
> dynamically?

Yes.  In the case of a PCI bus-- you may not know in advance how many
PCI devices there are until you probe the bus.   We have another FSL
proprietary bus we call the "fsl-mc" bus that is similar.

Another thing to consider-- starting with SMMUv2, as you know, there
is a new distributed architecture with multiple TBUs and a centralized
TCU that walks the SMMU page tables.  So instead of sprinkling multiple
SMMUs all over an SoC you now have the option a 1 central TCU and sprinkling
multiple TBUs around.   However, this means that the stream ID namespace
is now global and can be pretty limited.  In the SMMU implementation we 
have there are only 64 stream ID total for our Soc.  But we have many more
masters than that.

So we look at stream IDs as really corresponding to an 'isolation context'
and not to a bus master.  An isolation context is the domain you are
trying to isolate with the SMMU.  Devices that all belong to the same
'isolation context' can share the same stream ID, since they share
the same domain and page tables.

So, perhaps by default some/most SMMU masters may have a default stream ID
of 0x0 that is used by the host...and that could be represented
statically in the device tree.

But, we absolutely will need to dynamically set new stream IDs
into masters when a new IOMMU 'domain' is created and devices
are added to it.   All the devices in a domain will share
the same stream ID.

So whatever we do, let's please have an architecture flexible enough
to allow for this.

Thanks,
Stuart

[PATCH v2] devicetree: Add generic IOMMU device tree bindings

[Will Deacon]

Hi Varun,

On Thu, Jun 05, 2014 at 08:10:19PM +0100, Varun Sethi wrote:
> > The set of StreamIDs that can be generated by a master is fixed in the
> > hardware. The SMMU can then be programmed to map these incoming IDs onto
> > a context ID (or a set of context IDs), which are the IDs used internally
> > by the SMMU to find the page tables etc.
> > 
> > The StreamID -> ContextID mapping is dynamic and controlled by software.
> > The Master -> StreamIDs mapping is fixed in the hardware.
> The Master -> StreamIDs mapping may not always be fixed in the hardware.
> At, least in our case we plan to program these via software. PCI devices
> is one place where this mapping would have to be dynamic (based on the
> topology i.e. if the devices are connected to a bridge etc). Also, we have
> a hot plug device architecture where the stream ID is software
> programmable.
> 
> Other than that, based on the isolation requirements (iommu domain)
> software programmability offers greater flexibility.

For the sake of consistency, I'd really like to treat the master ->
streamIDs mapping as fixed. If that means setting it once during boot in
your case, then so be it (ideally in the firmware). That way, we just
describe the fixed mapping to the kernel and the driver doesn't have to
worry about changing the mapping, especially given that that's likely to be
highly error-prone once the SMMU is in us.

Do you have use-cases where you really need to change these mappings
dynamically?

Will

[PATCH v2] devicetree: Add generic IOMMU device tree bindings

[Will Deacon]

On Wed, Jun 04, 2014 at 10:12:38PM +0100, Thierry Reding wrote:
> On Fri, May 30, 2014 at 12:27:28PM +0100, Dave Martin wrote:
> > On Fri, May 30, 2014 at 08:30:08AM +0100, Thierry Reding wrote:
> [...]
> > > Arnd, can you take another look at this binding and see if there's
> > > anything else missing? If not I'll go through the document again and
> > > update all #address-cells/#size-cells references with #iommu-cells as
> > > appropriate and submit v3.
> > 
> > How do you envisage propagation of the master ID bits downstream of the
> > IOMMU would be described?
> > 
> > We will definitely need a way to describe this for GICv3.  How those
> > values are propagated is likely to vary between related SoCs and doesn't
> > feel like it should be baked into a driver, especially for the ARM SMMU
> > which may get reused in radically different SoC families from different
> > vendors.
> 
> Well, we've had cases like these in the past (power sequences come to
> mind). Some concepts are just too difficult or unwieldy to be put into
> device tree. I think that this is one of them.
> 
> > The most likely types of remapping are the adding of a base offset or
> > some extra bits to the ID -- because not all MSIs to the GIC will
> > necessarily pass through the IOMMU.  It's also possible that we might
> > see ID squashing or folding in some systems.
> 
> It can easily be argued that if the algorithm used to remap the ID
> varies, the compatibility of the device changes. Therefore I would
> expect any variant of the GICv3 that deviates from the "standard"
> mapping (if there is such a thing) to have its own compatible string.

There is no standard mapping; it's a property defined at system integration
time. I fully expect different SoCs to do different things here.

Will

[PATCH v2] devicetree: Add generic IOMMU device tree bindings

[Varun Sethi]

> -----Original Message-----
> From: iommu-bounces at lists.linux-foundation.org [mailto:iommu-
> bounces at lists.linux-foundation.org] On Behalf Of Will Deacon
> Sent: Wednesday, June 04, 2014 10:12 PM
> To: Thierry Reding
> Cc: Mark Rutland; devicetree at vger.kernel.org; linux-samsung-
> soc at vger.kernel.org; Pawel Moll; Arnd Bergmann; Ian Campbell; Grant
> Grundler; Stephen Warren; linux-kernel at vger.kernel.org; Marc Zyngier;
> Linux IOMMU; Rob Herring; Kumar Gala; linux-tegra at vger.kernel.org; Cho
> KyongHo; Dave P Martin; linux-arm-kernel at lists.infradead.org
> Subject: Re: [PATCH v2] devicetree: Add generic IOMMU device tree
> bindings
> 
> On Wed, Jun 04, 2014 at 03:35:10PM +0100, Thierry Reding wrote:
> > On Mon, Jun 02, 2014 at 11:41:04AM +0100, Dave Martin wrote:
> > > In the strictest sense, no.
> > >
> > > But for a large set of sane configurations, this probably works.
> > >
> > > Small sets of randomly-assigned IDs can just be enumerated one by
> one.
> > >
> > > We wouldn't be able to describe folding and bit shuffling, but we
> > > probably don't want to encourage that anyway.
> >
> > I'm having some difficulty understanding this. You make it sound like
> > there's a fairly arbitrary number of IDs that the SMMU can handle. So
> > how is the mapping to devices defined? If you say encourage that does
> > make it sound like the assignment of IDs is purely defined by some
> > mechanism in software rather than in hardware. Or they are more or
> > less randomly picked by someone. If that's the case, is that not
> > something that should be dynamically allocated by the kernel rather
> > than put into the device tree?
> 
> The set of StreamIDs that can be generated by a master is fixed in the
> hardware. The SMMU can then be programmed to map these incoming IDs onto
> a context ID (or a set of context IDs), which are the IDs used internally
> by the SMMU to find the page tables etc.
> 
> The StreamID -> ContextID mapping is dynamic and controlled by software.
> The Master -> StreamIDs mapping is fixed in the hardware.
The Master -> StreamIDs mapping may not always be fixed in the hardware. At, least in our case we plan to program these via software. PCI devices is one place where this mapping would have to be dynamic (based on the topology i.e. if the devices are connected to a bridge etc). Also, we have a hot plug device architecture where the stream ID is software programmable.

Other than that, based on the isolation requirements (iommu domain) software programmability offers greater flexibility.

-Varun

[PATCH v2] devicetree: Add generic IOMMU device tree bindings

[Arnd Bergmann]

On Wednesday 04 June 2014 23:32:00 Thierry Reding wrote:
> On Fri, May 30, 2014 at 09:01:19PM +0200, Arnd Bergmann wrote:
> > On Friday 30 May 2014 12:22:32 Dave Martin wrote:
> > > > +
> > > > +Examples:
> > > > +=========
> > > > +
> > > > +Single-master IOMMU:
> > > > +--------------------
> > > > +
> > > > + iommu {
> > > > +         #address-cells = <0>;
> > > > +         #size-cells = <0>;
> > > > + };
> > > > +
> > > > + master {
> > > > +         iommus = <&/iommu>;
> > > > + };
> > > > +
> > > > +Multiple-master IOMMU with fixed associations:
> > > > +----------------------------------------------
> > > > +
> > > > + /* multiple-master IOMMU */
> > > > + iommu {
> > > > +         /*
> > > > +          * Masters are statically associated with this IOMMU and
> > > > +          * address translation is always enabled.
> > > > +          */
> > > > +         #address-cells = <0>;
> > > > +         #size-cells = <0>;
> > > 
> > > In this example, can different translations be set up for the different
> > > masters?
> > > 
> > > With no cells available to contain any sort of ID, it looks like this
> > > is not possible.
> > 
> > Correct, this example is for an IOMMU that does not use IDs but has a
> > shared address space for all devices.
> 
> Couldn't these device all still have separate address spaces?

No. If they had separate address spaces, they would require a more
sophisticated IOMMU. A simple IOMMU without IDs can only be used
for overcoming address space limits (e.g. for 32-bit DMA masters on
systems with more than 4GB RAM) but not for strict isolation.

You basically have one page table shared across all devices connected
to the IOMMU, and every call to dma_alloc_coherent or dma_map_*
allocates a new IOVA that isn't used by any of the other devices
already, but you can't prevent a malicious user from getting a device
to do DMA to an IOVA that has been set up for another device.

You could have one such IOMMU per device of course, but I guess that's
not what you mean.

	Arnd

[PATCH v2] devicetree: Add generic IOMMU device tree bindings

[Thierry Reding]

On Fri, May 30, 2014 at 09:01:19PM +0200, Arnd Bergmann wrote:
> On Friday 30 May 2014 12:22:32 Dave Martin wrote:
> > > +
> > > +Examples:
> > > +=========
> > > +
> > > +Single-master IOMMU:
> > > +--------------------
> > > +
> > > +	iommu {
> > > +		#address-cells = <0>;
> > > +		#size-cells = <0>;
> > > +	};
> > > +
> > > +	master {
> > > +		iommus = <&/iommu>;
> > > +	};
> > > +
> > > +Multiple-master IOMMU with fixed associations:
> > > +----------------------------------------------
> > > +
> > > +	/* multiple-master IOMMU */
> > > +	iommu {
> > > +		/*
> > > +		 * Masters are statically associated with this IOMMU and
> > > +		 * address translation is always enabled.
> > > +		 */
> > > +		#address-cells = <0>;
> > > +		#size-cells = <0>;
> > 
> > In this example, can different translations be set up for the different
> > masters?
> > 
> > With no cells available to contain any sort of ID, it looks like this
> > is not possible.
> 
> Correct, this example is for an IOMMU that does not use IDs but has a
> shared address space for all devices.

Couldn't these device all still have separate address spaces?

Thierry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20140604/bc24c4e6/attachment.sig>

[PATCH v2] devicetree: Add generic IOMMU device tree bindings

[Thierry Reding]

On Fri, May 30, 2014 at 12:27:28PM +0100, Dave Martin wrote:
> On Fri, May 30, 2014 at 08:30:08AM +0100, Thierry Reding wrote:
[...]
> > Arnd, can you take another look at this binding and see if there's
> > anything else missing? If not I'll go through the document again and
> > update all #address-cells/#size-cells references with #iommu-cells as
> > appropriate and submit v3.
> 
> How do you envisage propagation of the master ID bits downstream of the
> IOMMU would be described?
> 
> We will definitely need a way to describe this for GICv3.  How those
> values are propagated is likely to vary between related SoCs and doesn't
> feel like it should be baked into a driver, especially for the ARM SMMU
> which may get reused in radically different SoC families from different
> vendors.

Well, we've had cases like these in the past (power sequences come to
mind). Some concepts are just too difficult or unwieldy to be put into
device tree. I think that this is one of them.

> The most likely types of remapping are the adding of a base offset or
> some extra bits to the ID -- because not all MSIs to the GIC will
> necessarily pass through the IOMMU.  It's also possible that we might
> see ID squashing or folding in some systems.

It can easily be argued that if the algorithm used to remap the ID
varies, the compatibility of the device changes. Therefore I would
expect any variant of the GICv3 that deviates from the "standard"
mapping (if there is such a thing) to have its own compatible string.

At that point there are two possibilities: a) handle the remapping in
code hooked into the driver via matching of the compatible string or
b) specifying a different specifier for each of these and keep the
description in DT. From an implementation point of view, variant b) is
likely going to be very similar to a), though (I'm thinking .of_xlate
type of functions here).

Thierry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20140604/d7163784/attachment.sig>

[PATCH v2] devicetree: Add generic IOMMU device tree bindings

[Thierry Reding]

On Wed, Jun 04, 2014 at 05:41:32PM +0100, Will Deacon wrote:
> On Wed, Jun 04, 2014 at 03:35:10PM +0100, Thierry Reding wrote:
> > On Mon, Jun 02, 2014 at 11:41:04AM +0100, Dave Martin wrote:
> > > In the strictest sense, no.
> > > 
> > > But for a large set of sane configurations, this probably works.
> > > 
> > > Small sets of randomly-assigned IDs can just be enumerated one by one.
> > > 
> > > We wouldn't be able to describe folding and bit shuffling, but we
> > > probably don't want to encourage that anyway.
> > 
> > I'm having some difficulty understanding this. You make it sound like
> > there's a fairly arbitrary number of IDs that the SMMU can handle. So
> > how is the mapping to devices defined? If you say encourage that does
> > make it sound like the assignment of IDs is purely defined by some
> > mechanism in software rather than in hardware. Or they are more or less
> > randomly picked by someone. If that's the case, is that not something
> > that should be dynamically allocated by the kernel rather than put into
> > the device tree?
> 
> The set of StreamIDs that can be generated by a master is fixed in the
> hardware. The SMMU can then be programmed to map these incoming IDs onto
> a context ID (or a set of context IDs), which are the IDs used internally
> by the SMMU to find the page tables etc.
> 
> The StreamID -> ContextID mapping is dynamic and controlled by software.
> The Master -> StreamIDs mapping is fixed in the hardware.

Okay, that sounds similar to what the Tegra SMMU does. The naming is
slightly differently. "Master" is called "memory client", "StreamID"
would be "SWGROUP" and "ContextID" I guess would map to what's called an
"ASID" (Address Space ID) on Tegra.

I'm not sure about the amount of leverage that we have to encourage the
mappings that we think are easy to describe in DT and discourage those
that we don't want. Last time I checked we were still playing catch up
rather than being able to give recommendations to hardware engineers
about what will result in sane DT content and what won't.

Irrespective of the above, I still think that a generic binding based on
an #iommu-cells and iommus property gives us the most flexibility. For
easy cases, most of which are listed in the current proposal could
easily be dealt with in convenience helpers. For the more complex cases
drivers for those IOMMUs can define a specifier that make sense for the
mappings they need to be able to describe.

Also we're talking about a generic IOMMU binding here. That means that
if some hardware comes along that's not "generic" and doesn't fit into
the constraints set by this binding, then we still have the option of
defining a completely different one for that particular case.

Thierry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20140604/ef17f0b9/attachment.sig>

[PATCH v2] devicetree: Add generic IOMMU device tree bindings

[Will Deacon]

On Wed, Jun 04, 2014 at 03:35:10PM +0100, Thierry Reding wrote:
> On Mon, Jun 02, 2014 at 11:41:04AM +0100, Dave Martin wrote:
> > In the strictest sense, no.
> > 
> > But for a large set of sane configurations, this probably works.
> > 
> > Small sets of randomly-assigned IDs can just be enumerated one by one.
> > 
> > We wouldn't be able to describe folding and bit shuffling, but we
> > probably don't want to encourage that anyway.
> 
> I'm having some difficulty understanding this. You make it sound like
> there's a fairly arbitrary number of IDs that the SMMU can handle. So
> how is the mapping to devices defined? If you say encourage that does
> make it sound like the assignment of IDs is purely defined by some
> mechanism in software rather than in hardware. Or they are more or less
> randomly picked by someone. If that's the case, is that not something
> that should be dynamically allocated by the kernel rather than put into
> the device tree?

The set of StreamIDs that can be generated by a master is fixed in the
hardware. The SMMU can then be programmed to map these incoming IDs onto
a context ID (or a set of context IDs), which are the IDs used internally
by the SMMU to find the page tables etc.

The StreamID -> ContextID mapping is dynamic and controlled by software.
The Master -> StreamIDs mapping is fixed in the hardware.

Will

[PATCH v2] devicetree: Add generic IOMMU device tree bindings

[Will Deacon]

On Wed, Jun 04, 2014 at 03:01:15PM +0100, Arnd Bergmann wrote:
> On Wednesday 04 June 2014 14:56:01 Will Deacon wrote:
> > On Wed, Jun 04, 2014 at 02:44:03PM +0100, Thierry Reding wrote:
> > > On Fri, May 30, 2014 at 09:54:37PM +0200, Arnd Bergmann wrote:
> > > > On Friday 30 May 2014 22:29:13 Hiroshi Doyu wrote:
> > > > > The disadvantage of this is that this limits the max number of streamIDs
> > > > > to support. If # of streamID is increased later more than 64, this
> > > > > format cannot cover any more. You have to predict the max # of streamIDs
> > > > > in advance if steamID is statically assigned.
> > > > > 
> > > > 
> > > > Well, the iommu specific binding could allow a variable #address-cells.
> > > > That way, you just need to know the number of stream IDs for that instance
> > > > of the iommu.
> > > 
> > > That sounds fairly complicated to me. I don't see what that buys us over
> > > the clarity and simplicity that the above explicit notation gives us. Is
> > > it not more common for a device to have a single master rather than a
> > > whole bunch of them?
> > 
> > I've never seen a device upstream of an ARM SMMU with a single stream-id;
> > they always seem to have a whole bunch of them. Calxeda's SATA controller
> > had 10 IDs, for example, and a PL330 DMA controller tends to have at least
> > 3.
> 
> What are those good for? Would we just always use the first one?

I think It's just an artifact of how these systems tend to be built. The
StreamID is constructed out of bits on the bus (AXI and sideband), so a
DMA controller could easily have one ID for reads, one for writes,
one for execute. Masters capable of multiple outstanding transactions
could also have sets of IDs too.

The SMMU driver would point them all at the same context bank, as far as I
can tell, but that means the driver does need to be aware of the incoming
IDs.

Rob might know how they constructed the IDs on the Calxeda SATA controller?

Will

[PATCH v2] devicetree: Add generic IOMMU device tree bindings

[Thierry Reding]

On Mon, Jun 02, 2014 at 11:41:04AM +0100, Dave Martin wrote:
> On Fri, May 30, 2014 at 09:49:59PM +0200, Arnd Bergmann wrote:
> > On Friday 30 May 2014 14:31:55 Rob Herring wrote:
> > > On Fri, May 30, 2014 at 2:06 PM, Arnd Bergmann <arnd@arndb.de> wrote:
> > > > On Friday 30 May 2014 08:16:05 Rob Herring wrote:
> > > >> On Fri, May 23, 2014 at 3:33 PM, Thierry Reding
> > > >> <thierry.reding@gmail.com> wrote:
> > > >> > From: Thierry Reding <treding@nvidia.com>
> > > >> > +IOMMU master node:
> > > >> > +==================
> > > >> > +
> > > >> > +Devices that access memory through an IOMMU are called masters. A device can
> > > >> > +have multiple master interfaces (to one or more IOMMU devices).
> > > >> > +
> > > >> > +Required properties:
> > > >> > +--------------------
> > > >> > +- iommus: A list of phandle and IOMMU specifier pairs that describe the IOMMU
> > > >> > +  master interfaces of the device. One entry in the list describes one master
> > > >> > +  interface of the device.
> > > >> > +
> > > >> > +When an "iommus" property is specified in a device tree node, the IOMMU will
> > > >> > +be used for address translation. If a "dma-ranges" property exists in the
> > > >> > +device's parent node it will be ignored. An exception to this rule is if the
> > > >> > +referenced IOMMU is disabled, in which case the "dma-ranges" property of the
> > > >> > +parent shall take effect.
> > > >>
> > > >> Just thinking out loud, could you have dma-ranges in the iommu node
> > > >> for the case when the iommu is enabled rather than putting the DMA
> > > >> window information into the iommus property?
> > > >>
> > > >> This would probably mean that you need both #iommu-cells and #address-cells.
> > > >
> > > > The reason for doing like this was that you may need a different window
> > > > for each device, while there can only be one dma-ranges property in
> > > > an iommu node.
> > > 
> > > My suggestion was that you also put the IDs in the dma-ranges as the
> > > first cell much as ranges for PCI encodes other information in the
> > > first cell. Then you can have an entry for each ID. The downside is
> > > another special case like PCI.
> > > 
> > > The argument for using #address-cells and #size-cells seems to be to
> > > align with how ranges work. If that's the route we want to go, then I
> > > say we should not stop there and actually use dma-ranges as well.
> > > Otherwise, I don't see the advantage over #iommu-cells.
> > 
> > I can see how dma-ranges in bus nodes work, it just doesn't seem to
> > have any reasonable meaning in the iommu node itself.
> 
> dma-ranges defines a static mapping for mastering through the bus node.
> 
> The whole point of an IOMMU is that it maps dynamically, so I agree:
> I'm unclear on what dma-ranges should mean in the IOMMU node itself
> (if anything).
> 
> > 
> > > > I don't understand the problem. If you have stream IDs 0 through 7,
> > > > you would have
> > > >
> > > >         master at a {
> > > >                 ...
> > > >                 iommus = <&smmu 0>;
> > > >         };
> > > >
> > > >         master at b {
> > > >                 ...
> > > >                 iommus = <&smmu 1;
> > > >         };
> > > >
> > > >         ...
> > > >
> > > >         master at 12 {
> > > >                 ...
> > > >                 iommus = <&smmu 7;
> > > >         };
> > > >
> > > > and you don't need a window at all. Why would you need a mask of
> > > > some sort?
> > > 
> > > 1 master with 7 IDs like this:
> > > 
> > >          master at a {
> > >                  ...
> > >                  iommus = <&smmu 0> <&smmu 1> <&smmu 2> <&smmu 3>
> > > <&smmu 4> <&smmu 5> <&smmu 6> <&smmu 7>;
> > >          };
> > > 
> > > If there was any sort of window, then it is almost certainly the same
> > > window for each ID.
> 
> Do we have real examples of using a window *and* an ID?  I thought the
> windowed-IOMMU concept was partly a way of encoding the ID in some
> real address bits on the bus.  If you're doing that, it seems less likely
> that there is a true "ID" as such (though it is possible).
> 
> > Ok, I see. In that case you'd probably want to have #address-cells = <1>
> > and #size-cells = <1> and give a range of IDs like
> > 
> > 	iommus = <&smmu 0 8>;
> > 
> > Do you think that ranges can have a meaningful definition with the ARM
> > SMMU stream IDs?
> 
> In the strictest sense, no.
> 
> But for a large set of sane configurations, this probably works.
> 
> Small sets of randomly-assigned IDs can just be enumerated one by one.
> 
> We wouldn't be able to describe folding and bit shuffling, but we
> probably don't want to encourage that anyway.

I'm having some difficulty understanding this. You make it sound like
there's a fairly arbitrary number of IDs that the SMMU can handle. So
how is the mapping to devices defined? If you say encourage that does
make it sound like the assignment of IDs is purely defined by some
mechanism in software rather than in hardware. Or they are more or less
randomly picked by someone. If that's the case, is that not something
that should be dynamically allocated by the kernel rather than put into
the device tree?

Maybe in the above case all you really need to know is how many IDs a
device needs to be assigned so that they can be properly allocated,
rather than the device exactly specifying which ones.

Thierry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20140604/266cc7e2/attachment-0001.sig>

[PATCH v2] devicetree: Add generic IOMMU device tree bindings

[Arnd Bergmann]

On Wednesday 04 June 2014 14:56:01 Will Deacon wrote:
> On Wed, Jun 04, 2014 at 02:44:03PM +0100, Thierry Reding wrote:
> > On Fri, May 30, 2014 at 09:54:37PM +0200, Arnd Bergmann wrote:
> > > On Friday 30 May 2014 22:29:13 Hiroshi Doyu wrote:
> > > > The disadvantage of this is that this limits the max number of streamIDs
> > > > to support. If # of streamID is increased later more than 64, this
> > > > format cannot cover any more. You have to predict the max # of streamIDs
> > > > in advance if steamID is statically assigned.
> > > > 
> > > 
> > > Well, the iommu specific binding could allow a variable #address-cells.
> > > That way, you just need to know the number of stream IDs for that instance
> > > of the iommu.
> > 
> > That sounds fairly complicated to me. I don't see what that buys us over
> > the clarity and simplicity that the above explicit notation gives us. Is
> > it not more common for a device to have a single master rather than a
> > whole bunch of them?
> 
> I've never seen a device upstream of an ARM SMMU with a single stream-id;
> they always seem to have a whole bunch of them. Calxeda's SATA controller
> had 10 IDs, for example, and a PL330 DMA controller tends to have at least
> 3.

What are those good for? Would we just always use the first one?

	Arnd

[PATCH v2] devicetree: Add generic IOMMU device tree bindings

[Will Deacon]

On Wed, Jun 04, 2014 at 02:44:03PM +0100, Thierry Reding wrote:
> On Fri, May 30, 2014 at 09:54:37PM +0200, Arnd Bergmann wrote:
> > On Friday 30 May 2014 22:29:13 Hiroshi Doyu wrote:
> > > The disadvantage of this is that this limits the max number of streamIDs
> > > to support. If # of streamID is increased later more than 64, this
> > > format cannot cover any more. You have to predict the max # of streamIDs
> > > in advance if steamID is statically assigned.
> > > 
> > 
> > Well, the iommu specific binding could allow a variable #address-cells.
> > That way, you just need to know the number of stream IDs for that instance
> > of the iommu.
> 
> That sounds fairly complicated to me. I don't see what that buys us over
> the clarity and simplicity that the above explicit notation gives us. Is
> it not more common for a device to have a single master rather than a
> whole bunch of them?

I've never seen a device upstream of an ARM SMMU with a single stream-id;
they always seem to have a whole bunch of them. Calxeda's SATA controller
had 10 IDs, for example, and a PL330 DMA controller tends to have at least
3.

Will

[PATCH v2] devicetree: Add generic IOMMU device tree bindings

[Arnd Bergmann]

On Wednesday 04 June 2014 15:44:03 Thierry Reding wrote:
> > Well, the iommu specific binding could allow a variable #address-cells.
> > That way, you just need to know the number of stream IDs for that instance
> > of the iommu.
> 
> That sounds fairly complicated to me. I don't see what that buys us over
> the clarity and simplicity that the above explicit notation gives us. Is
> it not more common for a device to have a single master rather than a
> whole bunch of them?

If I understood the problem right, the case that people want to handle
is not actually multiple masters but instead multiple IOMMU contexts, which
you can have in a GPU to handle multiple processes drawing on different
parts of the screen, or in a NIC with virtual functions assigned to KVM
guests or user processes. I don't know what the number of contexts per
device would be here, but I assume it may be large enough to become
a nuisance to list them individually.

Then again, I also don't know how IOMMUs would show those. In the cases
I've seen before, you actually have an ID for the master that is separate
from the context ID, but apparently SMMU isn't that smart.

	Arnd

[PATCH v2] devicetree: Add generic IOMMU device tree bindings

[Thierry Reding]

On Fri, May 30, 2014 at 09:54:37PM +0200, Arnd Bergmann wrote:
> On Friday 30 May 2014 22:29:13 Hiroshi Doyu wrote:
> > 
> > IIUC the original problem, "a master with 8 streamIDs" means something
> > like below, where some devices have multiple IDs but some have a
> > single. A sinle #address-cells cannot afford those 2 masters at once.
> > 
> >        iommu {
> >                /* the specifier represents the ID of the master */
> >                #address-cells = <1>;
> >                #size-cells = <0>;
> >        };
> > 
> >         master at a {
> >                 ...
> >                 iommus = <&smmu 1 2 3>; # 3 IDs
> >         };
> > 
> >         master at b {
> >                 ...
> >                 iommus = <&smmu 4>;     # 1 ID
> >         };
> 
> This would not be the usual format really. It should instead be
> 
> 		iommus = <&smmu 1>, <&smmu 2>, <&smmu 3>;
> 
> which can be tedious to type.

"Tedious to type" doesn't sound like a good argument to me. I don't see
why the above would necessarily be a bad notation. It's very much up to
the point and very explicit. This very obviously translates to:

	This device has three master interfaces, one for smmu ID 1, one
	for smmu ID 2 and one for smmu ID 3.

> > Tegra,SMMU has a similar problem and we have used a fixed size bitmap(64
> > bit) to afford 64 stream IDs so that a single device can hold multiple
> > IDs. If we apply the same bitmap to the above exmaple:
> > 
> >        iommu {
> >                /* the specifier represents the ID of the master */
> >                #address-cells = <1>;
> >                #size-cells = <0>;
> >        };
> > 
> >         master at a {
> >                 ...
> >                 iommus = <&smmu (BIT(1) | BIT(2) | BIT(3))>; # IDs 1 2 3
> >         };
> > 
> >         master at b {
> >                 ...
> >                 iommus = <&smmu BIT(4)>;     # ID 4
> >         };
> > 
> > The disadvantage of this is that this limits the max number of streamIDs
> > to support. If # of streamID is increased later more than 64, this
> > format cannot cover any more. You have to predict the max # of streamIDs
> > in advance if steamID is statically assigned.
> > 
> 
> Well, the iommu specific binding could allow a variable #address-cells.
> That way, you just need to know the number of stream IDs for that instance
> of the iommu.

That sounds fairly complicated to me. I don't see what that buys us over
the clarity and simplicity that the above explicit notation gives us. Is
it not more common for a device to have a single master rather than a
whole bunch of them?

Thierry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20140604/8714474f/attachment.sig>

[PATCH v2] devicetree: Add generic IOMMU device tree bindings

[Thierry Reding]

On Sun, Jun 01, 2014 at 10:55:46AM +0100, Will Deacon wrote:
> On Fri, May 30, 2014 at 08:54:37PM +0100, Arnd Bergmann wrote:
> > On Friday 30 May 2014 22:29:13 Hiroshi Doyu wrote:
> > > Tegra,SMMU has a similar problem and we have used a fixed size bitmap(64
> > > bit) to afford 64 stream IDs so that a single device can hold multiple
> > > IDs. If we apply the same bitmap to the above exmaple:
> > > 
> > >        iommu {
> > >                /* the specifier represents the ID of the master */
> > >                #address-cells = <1>;
> > >                #size-cells = <0>;
> > >        };
> > > 
> > >         master at a {
> > >                 ...
> > >                 iommus = <&smmu (BIT(1) | BIT(2) | BIT(3))>; # IDs 1 2 3
> > >         };
> > > 
> > >         master at b {
> > >                 ...
> > >                 iommus = <&smmu BIT(4)>;     # ID 4
> > >         };
> > > 
> > > The disadvantage of this is that this limits the max number of streamIDs
> > > to support. If # of streamID is increased later more than 64, this
> > > format cannot cover any more. You have to predict the max # of streamIDs
> > > in advance if steamID is statically assigned.
> > > 
> > 
> > Well, the iommu specific binding could allow a variable #address-cells.
> > That way, you just need to know the number of stream IDs for that instance
> > of the iommu.
> 
> In general, though, the SMMU will be able to support a large number of
> stream IDs (say a 16-bit space). The restriction we're interested in here is
> how many different stream IDs can be emitted by a single master device
> coming into the SMMU. *That* is a property of the master, not the SMMU.
> 
> In the current arm,smmu binding I have a #stream-id-cells property in each
> master. I can then feed that straight into of_parse_phandle_with_args to
> enumerate the IDs for that master. The problem with that is we're
> artificially restricted by MAX_PHANDLE_ARGS.

Maybe I don't fully understand, but since we leave it up to the IOMMU
binding to define the exact meaning of #iommu-cells, can't you simply
use that to your advantage and define something like:

	iommus = <&smmu 0 7>;

to mean IDs 0 to 7 for this particular IOMMU type?

Thierry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20140604/a7470b3b/attachment.sig>

[PATCH v2] devicetree: Add generic IOMMU device tree bindings

[Dave Martin]

On Fri, May 30, 2014 at 09:01:19PM +0200, Arnd Bergmann wrote:
> On Friday 30 May 2014 12:22:32 Dave Martin wrote:
> > > +
> > > +Examples:
> > > +=========
> > > +
> > > +Single-master IOMMU:
> > > +--------------------
> > > +
> > > +	iommu {
> > > +		#address-cells = <0>;
> > > +		#size-cells = <0>;
> > > +	};
> > > +
> > > +	master {
> > > +		iommus = <&/iommu>;
> > > +	};
> > > +
> > > +Multiple-master IOMMU with fixed associations:
> > > +----------------------------------------------
> > > +
> > > +	/* multiple-master IOMMU */
> > > +	iommu {
> > > +		/*
> > > +		 * Masters are statically associated with this IOMMU and
> > > +		 * address translation is always enabled.
> > > +		 */
> > > +		#address-cells = <0>;
> > > +		#size-cells = <0>;
> > 
> > In this example, can different translations be set up for the different
> > masters?
> > 
> > With no cells available to contain any sort of ID, it looks like this
> > is not possible.
> 
> Correct, this example is for an IOMMU that does not use IDs but has a
> shared address space for all devices.
> 
> > > +Multiple-master IOMMU with configurable DMA window:
> > > +---------------------------------------------------
> > > +
> > > +	/ {
> > > +		#address-cells = <1>;
> > > +		#size-cells = <1>;
> > > +
> > > +		iommu {
> > > +			/* master ID, address of DMA window */
> > > +			#address-cells = <2>;
> > > +			#size-cells = <2>;
> > > +		};
> > > +
> > > +		master {
> > > +			/* master ID 42, 4 GiB DMA window starting at 0 */
> > > +			iommus = <&/iommu  42 0  0x1 0x0>;
> > 
> > I'm still concerned that in order to deal with future cases we will have
> > to invent multiple ways to parse the "iommus" property.  For example, if
> > we have a PCEe RC mastering through an IOMMU, it will pass a huge set
> > of possible master IDs to the IOMMU, not just noe or two.
> > 
> > Do you have a solution in mind for that which doesn't break backwards
> > compatibility?
> 
> I think we can treat PCI as a special case here and have an interface
> that gets used by the PCI core code to talk to the IOMMU core code
> when setting up a the dma_map_ops for a PCI function. As long as the
> IOMMU driver understands what PCI is, we don't have to describe the
> mapping in detail.

PCI is only an example, but I admit it's likely to be the most important
example of a peripheral using a huge ID space.

There may still be integration-specific parameters which would need to
be fed in via DT regarding how IDs coming out of the RC map onto the
IOMMU and GIC.

> 
> > One option is to include an extra cell to the IOMMUs property
> > that indicates how to parse it.  For now, only a single value would
> > be defined.  For example:
> > 
> > 	iommus = <&/iommu IOMMU_SIMPLE 42>;
> > 
> > Then maybe later
> > 
> > 	iommus = <&/iommu IOMMU_RANGE 0x10000 0x10000>;
> > 
> > (I'm not suggesting what IOMMU_RANGE might mean.)
> > 
> 
> This can really be left up to the specific IOMMU driver itself.
> We can have drivers that support both #address-cells=<1>
> and #address-cells=<2> and behave differently based on that.
> I don't see a reason to define that across IOMMU implementations.

There's a risk we have to come up with multiple solutions to the same
problem there -- i.e., one solution per IOMMU implementation and then
another solution for the GIC.

Ideally that would be avoided, but the number of times this problem
would need to be solved is probably not that large.

Cheers
---Dave

[PATCH v2] devicetree: Add generic IOMMU device tree bindings

[Dave Martin]

On Fri, May 30, 2014 at 09:11:07PM +0200, Arnd Bergmann wrote:
> On Friday 30 May 2014 12:27:28 Dave Martin wrote:
> > On Fri, May 30, 2014 at 08:30:08AM +0100, Thierry Reding wrote:
> > > On Thu, May 29, 2014 at 09:52:22AM -0600, Stephen Warren wrote:
> > > > On 05/23/2014 02:36 PM, Thierry Reding wrote:
> > > > I think this is a mistake. address-cells/size-cells are for transactions
> > > > flowing down the bus (from the CPU to date). Describing a connection
> > > > from a device to an IOMMU is something completely different, and should
> > > > therefore simply use an iommu-cells property to describe any necessary
> > > > information. If we start re-using properties for different things in
> > > > different contexts, how is anyone going to know what they mean, and how
> > > > will conflicts be resolved. For example, what if there's a single HW
> > > > module that both acts as a regular register bus with children (where
> > > > address-cells/size-cells defines how transactions reach the children
> > > > from the parent), and is also an IOMMU (where according to this binding
> > > > proposal, address-cells/size-cells represent some aspect of the IOMMU
> > > > feature). Using different properties for different things is the only
> > > > sane way to keep different concepts separate. Another alternative would
> > > > be to represent the single HW module as separate nodes in DT, but I
> > > > think that will only make our lives harder, and where I've done that in
> > > > the past, I've regretted it.
> > > 
> > > There was some back-and-forth on this topic and the latest concensus
> > > when I wrote the second version was that #address-cells and #size-cells
> > > were to be used.
> > > 
> > > But there was some bore back-and-forth after that, and it seems like
> > > Arnd no longer thinks that using #address-cells and #size-cells is a
> > > good idea either[0].
> > 
> > Mistake or not, ePAPR already (ab)uses the address concept for PCI.
> > Unless ePAPR is wrong, I don't think it makes sense to argue that
> > the address concept cannot be repurposed.
> > 
> > The reason why this is abused for PCI is the same as our reason here:
> > different masters really are treated as distinct even when accessing
> > the same destination address, and DT has no general native way to
> > describe that.
> > 
> > One clear advantage of using #address-cells etc. is that ePAPR already
> > has very clear and well-defined ways of how to specify range mappings.
> > This gives us a ready-made way to describe windowed IOMMUs and 1:1
> > remappings of whole blocks of master IDs, which are the most obvious
> > cases other than having a small-integer set of explicit IDs.
> > 
> > That said, the PCI pseudo-address thing is not something we _necessarily_
> > want to repeat.
> 
> I'm really impartial to the question of whether to use #address-cells
> or #iommus now. It's possible that we can use the addresses in some
> meaningful way, but it's not clear to me that this will help make the
> representation cleaner or that we will actually want it for SMMU.
> 
> > > Arnd, can you take another look at this binding and see if there's
> > > anything else missing? If not I'll go through the document again and
> > > update all #address-cells/#size-cells references with #iommu-cells as
> > > appropriate and submit v3.
> > 
> > How do you envisage propagation of the master ID bits downstream of the
> > IOMMU would be described?
> > 
> > We will definitely need a way to describe this for GICv3.  How those
> > values are propagated is likely to vary between related SoCs and doesn't
> > feel like it should be baked into a driver, especially for the ARM SMMU
> > which may get reused in radically different SoC families from different
> > vendors.
> > 
> > The most likely types of remapping are the adding of a base offset or
> > some extra bits to the ID -- because not all MSIs to the GIC will
> > necessarily pass through the IOMMU.  It's also possible that we might
> > see ID squashing or folding in some systems.
> > 
> > 
> > For types of remapping which mix the ID and address together, I now
> > do tend to agree that any flexibility arising from describing that
> > in a general way that is unlikely to repay the cost of trying to
> > interpret and analyse the DT.  Defining a few sterotypical kinds
> > of mapping explicitly, as needed, looks more sensible for now.  The
> > windowed-IOMMU case is one example.
> 
> For MSI, my feeling is that we'd just be best off doing an end-to-end
> description. Any device using an MSI would have an interrupt specifier
> that is sufficient for the interrupt controller to understand where
> the IRQ comes from, and return an address/value pair back to the driver
> to program into some register.

This is OK so long as the number of endpoint pairings is trivial.

If there are multiple GICv3 ITSes addressable by a master, then we
might have to describe each one per master, or impose policy on the
DT whereby we choose one or two arbitrary or preferred ITSes for a
particular master and describe just those.

It's worth nothing that ePAPR strenuously avoids such an approach
for describing DMA: device-device DMA would require information about
every possible destination in every master node.

Of course, device-device DMA turns out not to be a hot topic in
practice most of the time, and likewise with this master ID stuff the
number of pairings that are really relevant in useful situations is
much smaller than the number of possible pairings.

The element of gamble here, and the design of DT around Linux's
present-day limitations makes me feel uneasy, but there's no
knockdown argument that I'm aware of.

_If_ we end up with a combinatorial explosion of pairings in the
future, do you have a solution in mind?

Cheers
---Dave

[PATCH v2] devicetree: Add generic IOMMU device tree bindings

[Dave Martin]

On Fri, May 30, 2014 at 09:49:59PM +0200, Arnd Bergmann wrote:
> On Friday 30 May 2014 14:31:55 Rob Herring wrote:
> > On Fri, May 30, 2014 at 2:06 PM, Arnd Bergmann <arnd@arndb.de> wrote:
> > > On Friday 30 May 2014 08:16:05 Rob Herring wrote:
> > >> On Fri, May 23, 2014 at 3:33 PM, Thierry Reding
> > >> <thierry.reding@gmail.com> wrote:
> > >> > From: Thierry Reding <treding@nvidia.com>
> > >> > +IOMMU master node:
> > >> > +==================
> > >> > +
> > >> > +Devices that access memory through an IOMMU are called masters. A device can
> > >> > +have multiple master interfaces (to one or more IOMMU devices).
> > >> > +
> > >> > +Required properties:
> > >> > +--------------------
> > >> > +- iommus: A list of phandle and IOMMU specifier pairs that describe the IOMMU
> > >> > +  master interfaces of the device. One entry in the list describes one master
> > >> > +  interface of the device.
> > >> > +
> > >> > +When an "iommus" property is specified in a device tree node, the IOMMU will
> > >> > +be used for address translation. If a "dma-ranges" property exists in the
> > >> > +device's parent node it will be ignored. An exception to this rule is if the
> > >> > +referenced IOMMU is disabled, in which case the "dma-ranges" property of the
> > >> > +parent shall take effect.
> > >>
> > >> Just thinking out loud, could you have dma-ranges in the iommu node
> > >> for the case when the iommu is enabled rather than putting the DMA
> > >> window information into the iommus property?
> > >>
> > >> This would probably mean that you need both #iommu-cells and #address-cells.
> > >
> > > The reason for doing like this was that you may need a different window
> > > for each device, while there can only be one dma-ranges property in
> > > an iommu node.
> > 
> > My suggestion was that you also put the IDs in the dma-ranges as the
> > first cell much as ranges for PCI encodes other information in the
> > first cell. Then you can have an entry for each ID. The downside is
> > another special case like PCI.
> > 
> > The argument for using #address-cells and #size-cells seems to be to
> > align with how ranges work. If that's the route we want to go, then I
> > say we should not stop there and actually use dma-ranges as well.
> > Otherwise, I don't see the advantage over #iommu-cells.
> 
> I can see how dma-ranges in bus nodes work, it just doesn't seem to
> have any reasonable meaning in the iommu node itself.

dma-ranges defines a static mapping for mastering through the bus node.

The whole point of an IOMMU is that it maps dynamically, so I agree:
I'm unclear on what dma-ranges should mean in the IOMMU node itself
(if anything).

> 
> > > I don't understand the problem. If you have stream IDs 0 through 7,
> > > you would have
> > >
> > >         master at a {
> > >                 ...
> > >                 iommus = <&smmu 0>;
> > >         };
> > >
> > >         master at b {
> > >                 ...
> > >                 iommus = <&smmu 1;
> > >         };
> > >
> > >         ...
> > >
> > >         master at 12 {
> > >                 ...
> > >                 iommus = <&smmu 7;
> > >         };
> > >
> > > and you don't need a window at all. Why would you need a mask of
> > > some sort?
> > 
> > 1 master with 7 IDs like this:
> > 
> >          master at a {
> >                  ...
> >                  iommus = <&smmu 0> <&smmu 1> <&smmu 2> <&smmu 3>
> > <&smmu 4> <&smmu 5> <&smmu 6> <&smmu 7>;
> >          };
> > 
> > If there was any sort of window, then it is almost certainly the same
> > window for each ID.

Do we have real examples of using a window *and* an ID?  I thought the
windowed-IOMMU concept was partly a way of encoding the ID in some
real address bits on the bus.  If you're doing that, it seems less likely
that there is a true "ID" as such (though it is possible).

> Ok, I see. In that case you'd probably want to have #address-cells = <1>
> and #size-cells = <1> and give a range of IDs like
> 
> 	iommus = <&smmu 0 8>;
> 
> Do you think that ranges can have a meaningful definition with the ARM
> SMMU stream IDs?

In the strictest sense, no.

But for a large set of sane configurations, this probably works.

Small sets of randomly-assigned IDs can just be enumerated one by one.

We wouldn't be able to describe folding and bit shuffling, but we
probably don't want to encourage that anyway.

Cheers
---Dave

[PATCH v2] devicetree: Add generic IOMMU device tree bindings

[Will Deacon]

On Fri, May 30, 2014 at 08:54:37PM +0100, Arnd Bergmann wrote:
> On Friday 30 May 2014 22:29:13 Hiroshi Doyu wrote:
> > Tegra,SMMU has a similar problem and we have used a fixed size bitmap(64
> > bit) to afford 64 stream IDs so that a single device can hold multiple
> > IDs. If we apply the same bitmap to the above exmaple:
> > 
> >        iommu {
> >                /* the specifier represents the ID of the master */
> >                #address-cells = <1>;
> >                #size-cells = <0>;
> >        };
> > 
> >         master at a {
> >                 ...
> >                 iommus = <&smmu (BIT(1) | BIT(2) | BIT(3))>; # IDs 1 2 3
> >         };
> > 
> >         master at b {
> >                 ...
> >                 iommus = <&smmu BIT(4)>;     # ID 4
> >         };
> > 
> > The disadvantage of this is that this limits the max number of streamIDs
> > to support. If # of streamID is increased later more than 64, this
> > format cannot cover any more. You have to predict the max # of streamIDs
> > in advance if steamID is statically assigned.
> > 
> 
> Well, the iommu specific binding could allow a variable #address-cells.
> That way, you just need to know the number of stream IDs for that instance
> of the iommu.

In general, though, the SMMU will be able to support a large number of
stream IDs (say a 16-bit space). The restriction we're interested in here is
how many different stream IDs can be emitted by a single master device
coming into the SMMU. *That* is a property of the master, not the SMMU.

In the current arm,smmu binding I have a #stream-id-cells property in each
master. I can then feed that straight into of_parse_phandle_with_args to
enumerate the IDs for that master. The problem with that is we're
artificially restricted by MAX_PHANDLE_ARGS.

Will

[PATCH v2] devicetree: Add generic IOMMU device tree bindings

[Arnd Bergmann]

On Friday 30 May 2014 22:29:13 Hiroshi Doyu wrote:
> 
> IIUC the original problem, "a master with 8 streamIDs" means something
> like below, where some devices have multiple IDs but some have a
> single. A sinle #address-cells cannot afford those 2 masters at once.
> 
>        iommu {
>                /* the specifier represents the ID of the master */
>                #address-cells = <1>;
>                #size-cells = <0>;
>        };
> 
>         master at a {
>                 ...
>                 iommus = <&smmu 1 2 3>; # 3 IDs
>         };
> 
>         master at b {
>                 ...
>                 iommus = <&smmu 4>;     # 1 ID
>         };

This would not be the usual format really. It should instead be

		iommus = <&smmu 1>, <&smmu 2>, <&smmu 3>;

which can be tedious to type.

> Tegra,SMMU has a similar problem and we have used a fixed size bitmap(64
> bit) to afford 64 stream IDs so that a single device can hold multiple
> IDs. If we apply the same bitmap to the above exmaple:
> 
>        iommu {
>                /* the specifier represents the ID of the master */
>                #address-cells = <1>;
>                #size-cells = <0>;
>        };
> 
>         master at a {
>                 ...
>                 iommus = <&smmu (BIT(1) | BIT(2) | BIT(3))>; # IDs 1 2 3
>         };
> 
>         master at b {
>                 ...
>                 iommus = <&smmu BIT(4)>;     # ID 4
>         };
> 
> The disadvantage of this is that this limits the max number of streamIDs
> to support. If # of streamID is increased later more than 64, this
> format cannot cover any more. You have to predict the max # of streamIDs
> in advance if steamID is statically assigned.
> 

Well, the iommu specific binding could allow a variable #address-cells.
That way, you just need to know the number of stream IDs for that instance
of the iommu.

	Arnd

[PATCH v2] devicetree: Add generic IOMMU device tree bindings

[Arnd Bergmann]

On Friday 30 May 2014 14:31:55 Rob Herring wrote:
> On Fri, May 30, 2014 at 2:06 PM, Arnd Bergmann <arnd@arndb.de> wrote:
> > On Friday 30 May 2014 08:16:05 Rob Herring wrote:
> >> On Fri, May 23, 2014 at 3:33 PM, Thierry Reding
> >> <thierry.reding@gmail.com> wrote:
> >> > From: Thierry Reding <treding@nvidia.com>
> >> > +IOMMU master node:
> >> > +==================
> >> > +
> >> > +Devices that access memory through an IOMMU are called masters. A device can
> >> > +have multiple master interfaces (to one or more IOMMU devices).
> >> > +
> >> > +Required properties:
> >> > +--------------------
> >> > +- iommus: A list of phandle and IOMMU specifier pairs that describe the IOMMU
> >> > +  master interfaces of the device. One entry in the list describes one master
> >> > +  interface of the device.
> >> > +
> >> > +When an "iommus" property is specified in a device tree node, the IOMMU will
> >> > +be used for address translation. If a "dma-ranges" property exists in the
> >> > +device's parent node it will be ignored. An exception to this rule is if the
> >> > +referenced IOMMU is disabled, in which case the "dma-ranges" property of the
> >> > +parent shall take effect.
> >>
> >> Just thinking out loud, could you have dma-ranges in the iommu node
> >> for the case when the iommu is enabled rather than putting the DMA
> >> window information into the iommus property?
> >>
> >> This would probably mean that you need both #iommu-cells and #address-cells.
> >
> > The reason for doing like this was that you may need a different window
> > for each device, while there can only be one dma-ranges property in
> > an iommu node.
> 
> My suggestion was that you also put the IDs in the dma-ranges as the
> first cell much as ranges for PCI encodes other information in the
> first cell. Then you can have an entry for each ID. The downside is
> another special case like PCI.
> 
> The argument for using #address-cells and #size-cells seems to be to
> align with how ranges work. If that's the route we want to go, then I
> say we should not stop there and actually use dma-ranges as well.
> Otherwise, I don't see the advantage over #iommu-cells.

I can see how dma-ranges in bus nodes work, it just doesn't seem to
have any reasonable meaning in the iommu node itself.

> > I don't understand the problem. If you have stream IDs 0 through 7,
> > you would have
> >
> >         master at a {
> >                 ...
> >                 iommus = <&smmu 0>;
> >         };
> >
> >         master at b {
> >                 ...
> >                 iommus = <&smmu 1;
> >         };
> >
> >         ...
> >
> >         master at 12 {
> >                 ...
> >                 iommus = <&smmu 7;
> >         };
> >
> > and you don't need a window at all. Why would you need a mask of
> > some sort?
> 
> 1 master with 7 IDs like this:
> 
>          master at a {
>                  ...
>                  iommus = <&smmu 0> <&smmu 1> <&smmu 2> <&smmu 3>
> <&smmu 4> <&smmu 5> <&smmu 6> <&smmu 7>;
>          };
> 
> If there was any sort of window, then it is almost certainly the same
> window for each ID.

Ok, I see. In that case you'd probably want to have #address-cells = <1>
and #size-cells = <1> and give a range of IDs like

	iommus = <&smmu 0 8>;

Do you think that ranges can have a meaningful definition with the ARM
SMMU stream IDs?

	Arnd

[PATCH v2] devicetree: Add generic IOMMU device tree bindings

[Rob Herring]

On Fri, May 30, 2014 at 2:06 PM, Arnd Bergmann <arnd@arndb.de> wrote:
> On Friday 30 May 2014 08:16:05 Rob Herring wrote:
>> On Fri, May 23, 2014 at 3:33 PM, Thierry Reding
>> <thierry.reding@gmail.com> wrote:
>> > From: Thierry Reding <treding@nvidia.com>
>> > +IOMMU master node:
>> > +==================
>> > +
>> > +Devices that access memory through an IOMMU are called masters. A device can
>> > +have multiple master interfaces (to one or more IOMMU devices).
>> > +
>> > +Required properties:
>> > +--------------------
>> > +- iommus: A list of phandle and IOMMU specifier pairs that describe the IOMMU
>> > +  master interfaces of the device. One entry in the list describes one master
>> > +  interface of the device.
>> > +
>> > +When an "iommus" property is specified in a device tree node, the IOMMU will
>> > +be used for address translation. If a "dma-ranges" property exists in the
>> > +device's parent node it will be ignored. An exception to this rule is if the
>> > +referenced IOMMU is disabled, in which case the "dma-ranges" property of the
>> > +parent shall take effect.
>>
>> Just thinking out loud, could you have dma-ranges in the iommu node
>> for the case when the iommu is enabled rather than putting the DMA
>> window information into the iommus property?
>>
>> This would probably mean that you need both #iommu-cells and #address-cells.
>
> The reason for doing like this was that you may need a different window
> for each device, while there can only be one dma-ranges property in
> an iommu node.

My suggestion was that you also put the IDs in the dma-ranges as the
first cell much as ranges for PCI encodes other information in the
first cell. Then you can have an entry for each ID. The downside is
another special case like PCI.

The argument for using #address-cells and #size-cells seems to be to
align with how ranges work. If that's the route we want to go, then I
say we should not stop there and actually use dma-ranges as well.
Otherwise, I don't see the advantage over #iommu-cells.

>> > +
>> > +Optional properties:
>> > +--------------------
>> > +- iommu-names: A list of names identifying each entry in the "iommus"
>> > +  property.
>>
>> Do we really need a name here? I would not expect that you have
>> clearly documented names here from the datasheet like you would for
>> interrupts or clocks, so you'd just be making up names. Sorry, but I'm
>> not a fan of names properties in general.
>
> Good point, this was really overdesign by modeling it after other
> subsystems that can have a use for names.
>
>> > +Multiple-master IOMMU:
>> > +----------------------
>> > +
>> > +       iommu {
>> > +               /* the specifier represents the ID of the master */
>> > +               #address-cells = <1>;
>> > +               #size-cells = <0>;
>> > +       };
>> > +
>> > +       master {
>> > +               /* device has master ID 42 in the IOMMU */
>> > +               iommus = <&/iommu 42>;
>> > +       };
>>
>> Presumably the ID would be the streamID on ARM's SMMU. How would a
>> master with 8 streamIDs be described? This is what Calxeda midway has
>> for SATA and I would expect that to be somewhat common. Either you
>> need some ID masking or you'll have lots of duplication when you have
>> windows.
>
> I don't understand the problem. If you have stream IDs 0 through 7,
> you would have
>
>         master at a {
>                 ...
>                 iommus = <&smmu 0>;
>         };
>
>         master at b {
>                 ...
>                 iommus = <&smmu 1;
>         };
>
>         ...
>
>         master at 12 {
>                 ...
>                 iommus = <&smmu 7;
>         };
>
> and you don't need a window at all. Why would you need a mask of
> some sort?

1 master with 7 IDs like this:

         master at a {
                 ...
                 iommus = <&smmu 0> <&smmu 1> <&smmu 2> <&smmu 3>
<&smmu 4> <&smmu 5> <&smmu 6> <&smmu 7>;
         };

If there was any sort of window, then it is almost certainly the same
window for each ID.

Rob

[PATCH v2] devicetree: Add generic IOMMU device tree bindings

[Hiroshi Doyu]

Arnd Bergmann <arnd@arndb.de> writes:

>> > +Multiple-master IOMMU:
>> > +----------------------
>> > +
>> > +       iommu {
>> > +               /* the specifier represents the ID of the master */
>> > +               #address-cells = <1>;
>> > +               #size-cells = <0>;
>> > +       };
>> > +
>> > +       master {
>> > +               /* device has master ID 42 in the IOMMU */
>> > +               iommus = <&/iommu 42>;
>> > +       };
>> 
>> Presumably the ID would be the streamID on ARM's SMMU. How would a
>> master with 8 streamIDs be described? This is what Calxeda midway has
>> for SATA and I would expect that to be somewhat common. Either you
>> need some ID masking or you'll have lots of duplication when you have
>> windows.
>
> I don't understand the problem. If you have stream IDs 0 through 7,
> you would have
>
> 	master at a {
> 		...
> 		iommus = <&smmu 0>;
> 	};
>
> 	master at b {
> 		...
> 		iommus = <&smmu 1;
> 	};
>
> 	...
>
> 	master at 12 {
> 		...
> 		iommus = <&smmu 7;
> 	};
>
> and you don't need a window at all. Why would you need a mask of
> some sort?

IIUC the original problem, "a master with 8 streamIDs" means something
like below, where some devices have multiple IDs but some have a
single. A sinle #address-cells cannot afford those 2 masters at once.

       iommu {
               /* the specifier represents the ID of the master */
               #address-cells = <1>;
               #size-cells = <0>;
       };

 	master at a {
 		...
 		iommus = <&smmu 1 2 3>; # 3 IDs
 	};

 	master at b {
 		...
 		iommus = <&smmu 4>;     # 1 ID
 	};

Tegra,SMMU has a similar problem and we have used a fixed size bitmap(64
bit) to afford 64 stream IDs so that a single device can hold multiple
IDs. If we apply the same bitmap to the above exmaple:

       iommu {
               /* the specifier represents the ID of the master */
               #address-cells = <1>;
               #size-cells = <0>;
       };

 	master at a {
 		...
 		iommus = <&smmu (BIT(1) | BIT(2) | BIT(3))>; # IDs 1 2 3
 	};

 	master at b {
 		...
 		iommus = <&smmu BIT(4)>;     # ID 4
 	};

The disadvantage of this is that this limits the max number of streamIDs
to support. If # of streamID is increased later more than 64, this
format cannot cover any more. You have to predict the max # of streamIDs
in advance if steamID is statically assigned.

[PATCH v2] devicetree: Add generic IOMMU device tree bindings

[Arnd Bergmann]

On Friday 30 May 2014 12:27:28 Dave Martin wrote:
> On Fri, May 30, 2014 at 08:30:08AM +0100, Thierry Reding wrote:
> > On Thu, May 29, 2014 at 09:52:22AM -0600, Stephen Warren wrote:
> > > On 05/23/2014 02:36 PM, Thierry Reding wrote:
> > > I think this is a mistake. address-cells/size-cells are for transactions
> > > flowing down the bus (from the CPU to date). Describing a connection
> > > from a device to an IOMMU is something completely different, and should
> > > therefore simply use an iommu-cells property to describe any necessary
> > > information. If we start re-using properties for different things in
> > > different contexts, how is anyone going to know what they mean, and how
> > > will conflicts be resolved. For example, what if there's a single HW
> > > module that both acts as a regular register bus with children (where
> > > address-cells/size-cells defines how transactions reach the children
> > > from the parent), and is also an IOMMU (where according to this binding
> > > proposal, address-cells/size-cells represent some aspect of the IOMMU
> > > feature). Using different properties for different things is the only
> > > sane way to keep different concepts separate. Another alternative would
> > > be to represent the single HW module as separate nodes in DT, but I
> > > think that will only make our lives harder, and where I've done that in
> > > the past, I've regretted it.
> > 
> > There was some back-and-forth on this topic and the latest concensus
> > when I wrote the second version was that #address-cells and #size-cells
> > were to be used.
> > 
> > But there was some bore back-and-forth after that, and it seems like
> > Arnd no longer thinks that using #address-cells and #size-cells is a
> > good idea either[0].
> 
> Mistake or not, ePAPR already (ab)uses the address concept for PCI.
> Unless ePAPR is wrong, I don't think it makes sense to argue that
> the address concept cannot be repurposed.
> 
> The reason why this is abused for PCI is the same as our reason here:
> different masters really are treated as distinct even when accessing
> the same destination address, and DT has no general native way to
> describe that.
> 
> One clear advantage of using #address-cells etc. is that ePAPR already
> has very clear and well-defined ways of how to specify range mappings.
> This gives us a ready-made way to describe windowed IOMMUs and 1:1
> remappings of whole blocks of master IDs, which are the most obvious
> cases other than having a small-integer set of explicit IDs.
> 
> That said, the PCI pseudo-address thing is not something we _necessarily_
> want to repeat.

I'm really impartial to the question of whether to use #address-cells
or #iommus now. It's possible that we can use the addresses in some
meaningful way, but it's not clear to me that this will help make the
representation cleaner or that we will actually want it for SMMU.

> > Arnd, can you take another look at this binding and see if there's
> > anything else missing? If not I'll go through the document again and
> > update all #address-cells/#size-cells references with #iommu-cells as
> > appropriate and submit v3.
> 
> How do you envisage propagation of the master ID bits downstream of the
> IOMMU would be described?
> 
> We will definitely need a way to describe this for GICv3.  How those
> values are propagated is likely to vary between related SoCs and doesn't
> feel like it should be baked into a driver, especially for the ARM SMMU
> which may get reused in radically different SoC families from different
> vendors.
> 
> The most likely types of remapping are the adding of a base offset or
> some extra bits to the ID -- because not all MSIs to the GIC will
> necessarily pass through the IOMMU.  It's also possible that we might
> see ID squashing or folding in some systems.
> 
> 
> For types of remapping which mix the ID and address together, I now
> do tend to agree that any flexibility arising from describing that
> in a general way that is unlikely to repay the cost of trying to
> interpret and analyse the DT.  Defining a few sterotypical kinds
> of mapping explicitly, as needed, looks more sensible for now.  The
> windowed-IOMMU case is one example.

For MSI, my feeling is that we'd just be best off doing an end-to-end
description. Any device using an MSI would have an interrupt specifier
that is sufficient for the interrupt controller to understand where
the IRQ comes from, and return an address/value pair back to the driver
to program into some register.

	Arnd

[PATCH v2] devicetree: Add generic IOMMU device tree bindings

[Arnd Bergmann]

On Friday 30 May 2014 08:16:05 Rob Herring wrote:
> On Fri, May 23, 2014 at 3:33 PM, Thierry Reding
> <thierry.reding@gmail.com> wrote:
> > From: Thierry Reding <treding@nvidia.com>
> > +IOMMU master node:
> > +==================
> > +
> > +Devices that access memory through an IOMMU are called masters. A device can
> > +have multiple master interfaces (to one or more IOMMU devices).
> > +
> > +Required properties:
> > +--------------------
> > +- iommus: A list of phandle and IOMMU specifier pairs that describe the IOMMU
> > +  master interfaces of the device. One entry in the list describes one master
> > +  interface of the device.
> > +
> > +When an "iommus" property is specified in a device tree node, the IOMMU will
> > +be used for address translation. If a "dma-ranges" property exists in the
> > +device's parent node it will be ignored. An exception to this rule is if the
> > +referenced IOMMU is disabled, in which case the "dma-ranges" property of the
> > +parent shall take effect.
> 
> Just thinking out loud, could you have dma-ranges in the iommu node
> for the case when the iommu is enabled rather than putting the DMA
> window information into the iommus property?
> 
> This would probably mean that you need both #iommu-cells and #address-cells.

The reason for doing like this was that you may need a different window
for each device, while there can only be one dma-ranges property in 
an iommu node.

> > +
> > +Optional properties:
> > +--------------------
> > +- iommu-names: A list of names identifying each entry in the "iommus"
> > +  property.
> 
> Do we really need a name here? I would not expect that you have
> clearly documented names here from the datasheet like you would for
> interrupts or clocks, so you'd just be making up names. Sorry, but I'm
> not a fan of names properties in general.

Good point, this was really overdesign by modeling it after other
subsystems that can have a use for names.
 
> > +Multiple-master IOMMU:
> > +----------------------
> > +
> > +       iommu {
> > +               /* the specifier represents the ID of the master */
> > +               #address-cells = <1>;
> > +               #size-cells = <0>;
> > +       };
> > +
> > +       master {
> > +               /* device has master ID 42 in the IOMMU */
> > +               iommus = <&/iommu 42>;
> > +       };
> 
> Presumably the ID would be the streamID on ARM's SMMU. How would a
> master with 8 streamIDs be described? This is what Calxeda midway has
> for SATA and I would expect that to be somewhat common. Either you
> need some ID masking or you'll have lots of duplication when you have
> windows.

I don't understand the problem. If you have stream IDs 0 through 7,
you would have

	master at a {
		...
		iommus = <&smmu 0>;
	};

	master at b {
		...
		iommus = <&smmu 1;
	};

	...

	master at 12 {
		...
		iommus = <&smmu 7;
	};

and you don't need a window@all. Why would you need a mask of
some sort?

	Arnd

[PATCH v2] devicetree: Add generic IOMMU device tree bindings

[Arnd Bergmann]

On Friday 30 May 2014 12:22:32 Dave Martin wrote:
> > +
> > +Examples:
> > +=========
> > +
> > +Single-master IOMMU:
> > +--------------------
> > +
> > +	iommu {
> > +		#address-cells = <0>;
> > +		#size-cells = <0>;
> > +	};
> > +
> > +	master {
> > +		iommus = <&/iommu>;
> > +	};
> > +
> > +Multiple-master IOMMU with fixed associations:
> > +----------------------------------------------
> > +
> > +	/* multiple-master IOMMU */
> > +	iommu {
> > +		/*
> > +		 * Masters are statically associated with this IOMMU and
> > +		 * address translation is always enabled.
> > +		 */
> > +		#address-cells = <0>;
> > +		#size-cells = <0>;
> 
> In this example, can different translations be set up for the different
> masters?
> 
> With no cells available to contain any sort of ID, it looks like this
> is not possible.

Correct, this example is for an IOMMU that does not use IDs but has a
shared address space for all devices.

> > +Multiple-master IOMMU with configurable DMA window:
> > +---------------------------------------------------
> > +
> > +	/ {
> > +		#address-cells = <1>;
> > +		#size-cells = <1>;
> > +
> > +		iommu {
> > +			/* master ID, address of DMA window */
> > +			#address-cells = <2>;
> > +			#size-cells = <2>;
> > +		};
> > +
> > +		master {
> > +			/* master ID 42, 4 GiB DMA window starting at 0 */
> > +			iommus = <&/iommu  42 0  0x1 0x0>;
> 
> I'm still concerned that in order to deal with future cases we will have
> to invent multiple ways to parse the "iommus" property.  For example, if
> we have a PCEe RC mastering through an IOMMU, it will pass a huge set
> of possible master IDs to the IOMMU, not just noe or two.
> 
> Do you have a solution in mind for that which doesn't break backwards
> compatibility?

I think we can treat PCI as a special case here and have an interface
that gets used by the PCI core code to talk to the IOMMU core code
when setting up a the dma_map_ops for a PCI function. As long as the
IOMMU driver understands what PCI is, we don't have to describe the
mapping in detail.

> One option is to include an extra cell to the IOMMUs property
> that indicates how to parse it.  For now, only a single value would
> be defined.  For example:
> 
> 	iommus = <&/iommu IOMMU_SIMPLE 42>;
> 
> Then maybe later
> 
> 	iommus = <&/iommu IOMMU_RANGE 0x10000 0x10000>;
> 
> (I'm not suggesting what IOMMU_RANGE might mean.)
> 

This can really be left up to the specific IOMMU driver itself.
We can have drivers that support both #address-cells=<1>
and #address-cells=<2> and behave differently based on that.
I don't see a reason to define that across IOMMU implementations.

> Other options are to introduce a new property name
> 
> 	range-iommus = <&/iommu 0x10000 0x10000>;
> 
> or control the parsing of incompatible iommus properties via a compatible
> string somewhere.

Introducing a new compatible string is always an option as the last resort.

	Arnd

[PATCH v2] devicetree: Add generic IOMMU device tree bindings

[Rob Herring]

On Fri, May 23, 2014 at 3:33 PM, Thierry Reding
<thierry.reding@gmail.com> wrote:
> From: Thierry Reding <treding@nvidia.com>
>
> This commit introduces a generic device tree binding for IOMMU devices.
> Only a very minimal subset is described here, but it is enough to cover
> the requirements of both the Exynos System MMU and Tegra SMMU as
> discussed here:
>
>     https://lkml.org/lkml/2014/4/27/346
>
> Signed-off-by: Thierry Reding <treding@nvidia.com>
> ---
> Changes in v2:
> - add notes about "dma-ranges" property (drop note from commit message)
> - document priorities of "iommus" property vs. "dma-ranges" property
> - drop #iommu-cells in favour of #address-cells and #size-cells
> - remove multiple-master device example
>
>  Documentation/devicetree/bindings/iommu/iommu.txt | 167 ++++++++++++++++++++++
>  1 file changed, 167 insertions(+)
>  create mode 100644 Documentation/devicetree/bindings/iommu/iommu.txt
>
> diff --git a/Documentation/devicetree/bindings/iommu/iommu.txt b/Documentation/devicetree/bindings/iommu/iommu.txt
> new file mode 100644
> index 000000000000..6ce759afcc94
> --- /dev/null
> +++ b/Documentation/devicetree/bindings/iommu/iommu.txt
> @@ -0,0 +1,167 @@
> +This document describes the generic device tree binding for IOMMUs and their
> +master(s).
> +
> +
> +IOMMU device node:
> +==================
> +
> +An IOMMU can provide the following services:
> +
> +* Remap address space to allow devices to access physical memory ranges that
> +  they otherwise wouldn't be capable of accessing.
> +
> +  Example: 32-bit DMA to 64-bit physical addresses
> +
> +* Implement scatter-gather at page level granularity so that the device does
> +  not have to.
> +
> +* Provide system protection against "rogue" DMA by forcing all accesses to go
> +  through the IOMMU and faulting when encountering accesses to unmapped
> +  address regions.
> +
> +* Provide address space isolation between multiple contexts.
> +
> +  Example: Virtualization
> +
> +Device nodes compatible with this binding represent hardware with some of the
> +above capabilities.
> +
> +IOMMUs can be single-master or multiple-master. Single-master IOMMU devices
> +typically have a fixed association to the master device, whereas multiple-
> +master IOMMU devices can translate accesses from more than one master.
> +
> +The device tree node of the IOMMU device's parent bus must contain a valid
> +"dma-ranges" property that describes how the physical address space of the
> +IOMMU maps to memory. An empty "dma-ranges" property means that there is a
> +1:1 mapping from IOMMU to memory.
> +
> +Required properties:
> +--------------------
> +- #address-cells: The number of cells in an IOMMU specifier needed to encode
> +  an address.
> +- #size-cells: The number of cells in an IOMMU specifier needed to represent
> +  the length of an address range.
> +
> +Typical values for the above include:
> +- #address-cells = <0>, size-cells = <0>: Single master IOMMU devices are not
> +  configurable and therefore no additional information needs to be encoded in
> +  the specifier. This may also apply to multiple master IOMMU devices that do
> +  not allow the association of masters to be configured.
> +- #address-cells = <1>, size-cells = <0>: Multiple master IOMMU devices may
> +  need to be configured in order to enable translation for a given master. In
> +  such cases the single address cell corresponds to the master device's ID.
> +- #address-cells = <2>, size-cells = <2>: Some IOMMU devices allow the DMA
> +  window for masters to be configured. The first cell of the address in this
> +  may contain the master device's ID for example, while the second cell could
> +  contain the start of the DMA window for the given device. The length of the
> +  DMA window is specified by two additional cells.
> +
> +
> +IOMMU master node:
> +==================
> +
> +Devices that access memory through an IOMMU are called masters. A device can
> +have multiple master interfaces (to one or more IOMMU devices).
> +
> +Required properties:
> +--------------------
> +- iommus: A list of phandle and IOMMU specifier pairs that describe the IOMMU
> +  master interfaces of the device. One entry in the list describes one master
> +  interface of the device.
> +
> +When an "iommus" property is specified in a device tree node, the IOMMU will
> +be used for address translation. If a "dma-ranges" property exists in the
> +device's parent node it will be ignored. An exception to this rule is if the
> +referenced IOMMU is disabled, in which case the "dma-ranges" property of the
> +parent shall take effect.

Just thinking out loud, could you have dma-ranges in the iommu node
for the case when the iommu is enabled rather than putting the DMA
window information into the iommus property?

This would probably mean that you need both #iommu-cells and #address-cells.

> +
> +Optional properties:
> +--------------------
> +- iommu-names: A list of names identifying each entry in the "iommus"
> +  property.

Do we really need a name here? I would not expect that you have
clearly documented names here from the datasheet like you would for
interrupts or clocks, so you'd just be making up names. Sorry, but I'm
not a fan of names properties in general.

> +
> +
> +Notes:
> +======
> +
> +One possible extension to the above is to use an "iommus" property along with
> +a "dma-ranges" property in a bus device node (such as PCI host bridges). This
> +can be useful to describe how children on the bus relate to the IOMMU if they
> +are not explicitly listed in the device tree (e.g. PCI devices). However, the
> +requirements of that use-case haven't been fully determined yet. Implementing
> +this is therefore not recommended without further discussion and extension of
> +this binding.
> +
> +

[...]

> +Multiple-master IOMMU:
> +----------------------
> +
> +       iommu {
> +               /* the specifier represents the ID of the master */
> +               #address-cells = <1>;
> +               #size-cells = <0>;
> +       };
> +
> +       master {
> +               /* device has master ID 42 in the IOMMU */
> +               iommus = <&/iommu 42>;
> +       };

Presumably the ID would be the streamID on ARM's SMMU. How would a
master with 8 streamIDs be described? This is what Calxeda midway has
for SATA and I would expect that to be somewhat common. Either you
need some ID masking or you'll have lots of duplication when you have
windows.

Rob

[PATCH v2] devicetree: Add generic IOMMU device tree bindings

[Dave Martin]

On Fri, May 30, 2014 at 08:30:08AM +0100, Thierry Reding wrote:
> On Thu, May 29, 2014 at 09:52:22AM -0600, Stephen Warren wrote:
> > On 05/23/2014 02:36 PM, Thierry Reding wrote:
> > > From: Thierry Reding <treding@nvidia.com>
> > > 
> > > This commit introduces a generic device tree binding for IOMMU devices.
> > > Only a very minimal subset is described here, but it is enough to cover
> > > the requirements of both the Exynos System MMU and Tegra SMMU as
> > > discussed here:
> > > 
> > >     https://lkml.org/lkml/2014/4/27/346
> > > 
> > > Signed-off-by: Thierry Reding <treding@nvidia.com>
> > > ---
> > > Apologies for the noise, but apparently I mistyped one of the email
> > > addresses, should be fixed now.
> > > 
> > > Changes in v2:
> > > - add notes about "dma-ranges" property (drop note from commit message)
> > > - document priorities of "iommus" property vs. "dma-ranges" property
> > > - drop #iommu-cells in favour of #address-cells and #size-cells
> > 
> > I think this is a mistake. address-cells/size-cells are for transactions
> > flowing down the bus (from the CPU to date). Describing a connection
> > from a device to an IOMMU is something completely different, and should
> > therefore simply use an iommu-cells property to describe any necessary
> > information. If we start re-using properties for different things in
> > different contexts, how is anyone going to know what they mean, and how
> > will conflicts be resolved. For example, what if there's a single HW
> > module that both acts as a regular register bus with children (where
> > address-cells/size-cells defines how transactions reach the children
> > from the parent), and is also an IOMMU (where according to this binding
> > proposal, address-cells/size-cells represent some aspect of the IOMMU
> > feature). Using different properties for different things is the only
> > sane way to keep different concepts separate. Another alternative would
> > be to represent the single HW module as separate nodes in DT, but I
> > think that will only make our lives harder, and where I've done that in
> > the past, I've regretted it.
> 
> There was some back-and-forth on this topic and the latest concensus
> when I wrote the second version was that #address-cells and #size-cells
> were to be used.
> 
> But there was some bore back-and-forth after that, and it seems like
> Arnd no longer thinks that using #address-cells and #size-cells is a
> good idea either[0].

Mistake or not, ePAPR already (ab)uses the address concept for PCI.
Unless ePAPR is wrong, I don't think it makes sense to argue that
the address concept cannot be repurposed.

The reason why this is abused for PCI is the same as our reason here:
different masters really are treated as distinct even when accessing
the same destination address, and DT has no general native way to
describe that.

One clear advantage of using #address-cells etc. is that ePAPR already
has very clear and well-defined ways of how to specify range mappings.
This gives us a ready-made way to describe windowed IOMMUs and 1:1
remappings of whole blocks of master IDs, which are the most obvious
cases other than having a small-integer set of explicit IDs.

That said, the PCI pseudo-address thing is not something we _necessarily_
want to repeat.

> Arnd, can you take another look at this binding and see if there's
> anything else missing? If not I'll go through the document again and
> update all #address-cells/#size-cells references with #iommu-cells as
> appropriate and submit v3.

How do you envisage propagation of the master ID bits downstream of the
IOMMU would be described?

We will definitely need a way to describe this for GICv3.  How those
values are propagated is likely to vary between related SoCs and doesn't
feel like it should be baked into a driver, especially for the ARM SMMU
which may get reused in radically different SoC families from different
vendors.

The most likely types of remapping are the adding of a base offset or
some extra bits to the ID -- because not all MSIs to the GIC will
necessarily pass through the IOMMU.  It's also possible that we might
see ID squashing or folding in some systems.


For types of remapping which mix the ID and address together, I now
do tend to agree that any flexibility arising from describing that
in a general way that is unlikely to repay the cost of trying to
interpret and analyse the DT.  Defining a few sterotypical kinds
of mapping explicitly, as needed, looks more sensible for now.  The
windowed-IOMMU case is one example.

Cheers
---Dave

[PATCH v2] devicetree: Add generic IOMMU device tree bindings

[Dave Martin]

On Fri, May 23, 2014 at 10:36:35PM +0200, Thierry Reding wrote:
> From: Thierry Reding <treding@nvidia.com>
> 
> This commit introduces a generic device tree binding for IOMMU devices.
> Only a very minimal subset is described here, but it is enough to cover
> the requirements of both the Exynos System MMU and Tegra SMMU as
> discussed here:
> 
>     https://lkml.org/lkml/2014/4/27/346
> 
> Signed-off-by: Thierry Reding <treding@nvidia.com>
> ---
> Apologies for the noise, but apparently I mistyped one of the email
> addresses, should be fixed now.
> 
> Changes in v2:
> - add notes about "dma-ranges" property (drop note from commit message)
> - document priorities of "iommus" property vs. "dma-ranges" property
> - drop #iommu-cells in favour of #address-cells and #size-cells
> - remove multiple-master device example
> 
>  Documentation/devicetree/bindings/iommu/iommu.txt | 167 ++++++++++++++++++++++
>  1 file changed, 167 insertions(+)
>  create mode 100644 Documentation/devicetree/bindings/iommu/iommu.txt
> 
> diff --git a/Documentation/devicetree/bindings/iommu/iommu.txt b/Documentation/devicetree/bindings/iommu/iommu.txt
> new file mode 100644
> index 000000000000..6ce759afcc94
> --- /dev/null
> +++ b/Documentation/devicetree/bindings/iommu/iommu.txt
> @@ -0,0 +1,167 @@
> +This document describes the generic device tree binding for IOMMUs and their
> +master(s).
> +
> +
> +IOMMU device node:
> +==================
> +
> +An IOMMU can provide the following services:
> +
> +* Remap address space to allow devices to access physical memory ranges that
> +  they otherwise wouldn't be capable of accessing.
> +
> +  Example: 32-bit DMA to 64-bit physical addresses
> +
> +* Implement scatter-gather at page level granularity so that the device does
> +  not have to.
> +
> +* Provide system protection against "rogue" DMA by forcing all accesses to go
> +  through the IOMMU and faulting when encountering accesses to unmapped
> +  address regions.
> +
> +* Provide address space isolation between multiple contexts.
> +
> +  Example: Virtualization
> +
> +Device nodes compatible with this binding represent hardware with some of the
> +above capabilities.
> +
> +IOMMUs can be single-master or multiple-master. Single-master IOMMU devices
> +typically have a fixed association to the master device, whereas multiple-
> +master IOMMU devices can translate accesses from more than one master.
> +
> +The device tree node of the IOMMU device's parent bus must contain a valid
> +"dma-ranges" property that describes how the physical address space of the
> +IOMMU maps to memory. An empty "dma-ranges" property means that there is a
> +1:1 mapping from IOMMU to memory.
> +
> +Required properties:
> +--------------------
> +- #address-cells: The number of cells in an IOMMU specifier needed to encode
> +  an address.
> +- #size-cells: The number of cells in an IOMMU specifier needed to represent
> +  the length of an address range.
> +
> +Typical values for the above include:
> +- #address-cells = <0>, size-cells = <0>: Single master IOMMU devices are not
> +  configurable and therefore no additional information needs to be encoded in
> +  the specifier. This may also apply to multiple master IOMMU devices that do
> +  not allow the association of masters to be configured.
> +- #address-cells = <1>, size-cells = <0>: Multiple master IOMMU devices may
> +  need to be configured in order to enable translation for a given master. In
> +  such cases the single address cell corresponds to the master device's ID.
> +- #address-cells = <2>, size-cells = <2>: Some IOMMU devices allow the DMA
> +  window for masters to be configured. The first cell of the address in this
> +  may contain the master device's ID for example, while the second cell could
> +  contain the start of the DMA window for the given device. The length of the
> +  DMA window is specified by two additional cells.
> +
> +
> +IOMMU master node:
> +==================
> +
> +Devices that access memory through an IOMMU are called masters. A device can
> +have multiple master interfaces (to one or more IOMMU devices).
> +
> +Required properties:
> +--------------------
> +- iommus: A list of phandle and IOMMU specifier pairs that describe the IOMMU
> +  master interfaces of the device. One entry in the list describes one master
> +  interface of the device.
> +
> +When an "iommus" property is specified in a device tree node, the IOMMU will
> +be used for address translation. If a "dma-ranges" property exists in the
> +device's parent node it will be ignored. An exception to this rule is if the
> +referenced IOMMU is disabled, in which case the "dma-ranges" property of the
> +parent shall take effect.
> +
> +Optional properties:
> +--------------------
> +- iommu-names: A list of names identifying each entry in the "iommus"
> +  property.
> +
> +
> +Notes:
> +======
> +
> +One possible extension to the above is to use an "iommus" property along with
> +a "dma-ranges" property in a bus device node (such as PCI host bridges). This
> +can be useful to describe how children on the bus relate to the IOMMU if they
> +are not explicitly listed in the device tree (e.g. PCI devices). However, the
> +requirements of that use-case haven't been fully determined yet. Implementing
> +this is therefore not recommended without further discussion and extension of
> +this binding.

Sounds sensible.

> +
> +
> +Examples:
> +=========
> +
> +Single-master IOMMU:
> +--------------------
> +
> +	iommu {
> +		#address-cells = <0>;
> +		#size-cells = <0>;
> +	};
> +
> +	master {
> +		iommus = <&/iommu>;
> +	};
> +
> +Multiple-master IOMMU with fixed associations:
> +----------------------------------------------
> +
> +	/* multiple-master IOMMU */
> +	iommu {
> +		/*
> +		 * Masters are statically associated with this IOMMU and
> +		 * address translation is always enabled.
> +		 */
> +		#address-cells = <0>;
> +		#size-cells = <0>;

In this example, can different translations be set up for the different
masters?

With no cells available to contain any sort of ID, it looks like this
is not possible.

> +	};
> +
> +	/* static association with IOMMU */
> +	master at 1 {
> +		reg = <1>;
> +		iommus = <&/iommu>;
> +	};
> +
> +	/* static association with IOMMU */
> +	master at 2 {
> +		reg = <2>;
> +		iommus = <&/iommu>;
> +	};
> +
> +Multiple-master IOMMU:
> +----------------------
> +
> +	iommu {
> +		/* the specifier represents the ID of the master */
> +		#address-cells = <1>;
> +		#size-cells = <0>;
> +	};
> +
> +	master {
> +		/* device has master ID 42 in the IOMMU */
> +		iommus = <&/iommu 42>;
> +	};
> +
> +Multiple-master IOMMU with configurable DMA window:
> +---------------------------------------------------
> +
> +	/ {
> +		#address-cells = <1>;
> +		#size-cells = <1>;
> +
> +		iommu {
> +			/* master ID, address of DMA window */
> +			#address-cells = <2>;
> +			#size-cells = <2>;
> +		};
> +
> +		master {
> +			/* master ID 42, 4 GiB DMA window starting at 0 */
> +			iommus = <&/iommu  42 0  0x1 0x0>;

I'm still concerned that in order to deal with future cases we will have
to invent multiple ways to parse the "iommus" property.  For example, if
we have a PCEe RC mastering through an IOMMU, it will pass a huge set
of possible master IDs to the IOMMU, not just noe or two.

Do you have a solution in mind for that which doesn't break backwards
compatibility?

One option is to include an extra cell to the IOMMUs property
that indicates how to parse it.  For now, only a single value would
be defined.  For example:

	iommus = <&/iommu IOMMU_SIMPLE 42>;

Then maybe later

	iommus = <&/iommu IOMMU_RANGE 0x10000 0x10000>;

(I'm not suggesting what IOMMU_RANGE might mean.)


Other options are to introduce a new property name

	range-iommus = <&/iommu 0x10000 0x10000>;

or control the parsing of incompatible iommus properties via a compatible
string somewhere.

Cheers
---Dave

[PATCH v2] devicetree: Add generic IOMMU device tree bindings

[Thierry Reding]

On Thu, May 29, 2014 at 09:52:22AM -0600, Stephen Warren wrote:
> On 05/23/2014 02:36 PM, Thierry Reding wrote:
> > From: Thierry Reding <treding@nvidia.com>
> > 
> > This commit introduces a generic device tree binding for IOMMU devices.
> > Only a very minimal subset is described here, but it is enough to cover
> > the requirements of both the Exynos System MMU and Tegra SMMU as
> > discussed here:
> > 
> >     https://lkml.org/lkml/2014/4/27/346
> > 
> > Signed-off-by: Thierry Reding <treding@nvidia.com>
> > ---
> > Apologies for the noise, but apparently I mistyped one of the email
> > addresses, should be fixed now.
> > 
> > Changes in v2:
> > - add notes about "dma-ranges" property (drop note from commit message)
> > - document priorities of "iommus" property vs. "dma-ranges" property
> > - drop #iommu-cells in favour of #address-cells and #size-cells
> 
> I think this is a mistake. address-cells/size-cells are for transactions
> flowing down the bus (from the CPU to date). Describing a connection
> from a device to an IOMMU is something completely different, and should
> therefore simply use an iommu-cells property to describe any necessary
> information. If we start re-using properties for different things in
> different contexts, how is anyone going to know what they mean, and how
> will conflicts be resolved. For example, what if there's a single HW
> module that both acts as a regular register bus with children (where
> address-cells/size-cells defines how transactions reach the children
> from the parent), and is also an IOMMU (where according to this binding
> proposal, address-cells/size-cells represent some aspect of the IOMMU
> feature). Using different properties for different things is the only
> sane way to keep different concepts separate. Another alternative would
> be to represent the single HW module as separate nodes in DT, but I
> think that will only make our lives harder, and where I've done that in
> the past, I've regretted it.

There was some back-and-forth on this topic and the latest concensus
when I wrote the second version was that #address-cells and #size-cells
were to be used.

But there was some bore back-and-forth after that, and it seems like
Arnd no longer thinks that using #address-cells and #size-cells is a
good idea either[0].

Arnd, can you take another look at this binding and see if there's
anything else missing? If not I'll go through the document again and
update all #address-cells/#size-cells references with #iommu-cells as
appropriate and submit v3.

Thanks,
Thierry

[0]: https://lkml.org/lkml/2014/5/20/609
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20140530/6cf2a62b/attachment.sig>

[PATCH v2] devicetree: Add generic IOMMU device tree bindings

[Stephen Warren]

On 05/23/2014 02:36 PM, Thierry Reding wrote:
> From: Thierry Reding <treding@nvidia.com>
> 
> This commit introduces a generic device tree binding for IOMMU devices.
> Only a very minimal subset is described here, but it is enough to cover
> the requirements of both the Exynos System MMU and Tegra SMMU as
> discussed here:
> 
>     https://lkml.org/lkml/2014/4/27/346
> 
> Signed-off-by: Thierry Reding <treding@nvidia.com>
> ---
> Apologies for the noise, but apparently I mistyped one of the email
> addresses, should be fixed now.
> 
> Changes in v2:
> - add notes about "dma-ranges" property (drop note from commit message)
> - document priorities of "iommus" property vs. "dma-ranges" property
> - drop #iommu-cells in favour of #address-cells and #size-cells

I think this is a mistake. address-cells/size-cells are for transactions
flowing down the bus (from the CPU to date). Describing a connection
from a device to an IOMMU is something completely different, and should
therefore simply use an iommu-cells property to describe any necessary
information. If we start re-using properties for different things in
different contexts, how is anyone going to know what they mean, and how
will conflicts be resolved. For example, what if there's a single HW
module that both acts as a regular register bus with children (where
address-cells/size-cells defines how transactions reach the children
from the parent), and is also an IOMMU (where according to this binding
proposal, address-cells/size-cells represent some aspect of the IOMMU
feature). Using different properties for different things is the only
sane way to keep different concepts separate. Another alternative would
be to represent the single HW module as separate nodes in DT, but I
think that will only make our lives harder, and where I've done that in
the past, I've regretted it.

[PATCH v2] devicetree: Add generic IOMMU device tree bindings

[Thierry Reding]

From: Thierry Reding <treding@nvidia.com>

This commit introduces a generic device tree binding for IOMMU devices.
Only a very minimal subset is described here, but it is enough to cover
the requirements of both the Exynos System MMU and Tegra SMMU as
discussed here:

    https://lkml.org/lkml/2014/4/27/346

Signed-off-by: Thierry Reding <treding@nvidia.com>
---
Apologies for the noise, but apparently I mistyped one of the email
addresses, should be fixed now.

Changes in v2:
- add notes about "dma-ranges" property (drop note from commit message)
- document priorities of "iommus" property vs. "dma-ranges" property
- drop #iommu-cells in favour of #address-cells and #size-cells
- remove multiple-master device example

 Documentation/devicetree/bindings/iommu/iommu.txt | 167 ++++++++++++++++++++++
 1 file changed, 167 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/iommu/iommu.txt

diff --git a/Documentation/devicetree/bindings/iommu/iommu.txt b/Documentation/devicetree/bindings/iommu/iommu.txt
new file mode 100644
index 000000000000..6ce759afcc94
--- /dev/null
+++ b/Documentation/devicetree/bindings/iommu/iommu.txt
@@ -0,0 +1,167 @@
+This document describes the generic device tree binding for IOMMUs and their
+master(s).
+
+
+IOMMU device node:
+==================
+
+An IOMMU can provide the following services:
+
+* Remap address space to allow devices to access physical memory ranges that
+  they otherwise wouldn't be capable of accessing.
+
+  Example: 32-bit DMA to 64-bit physical addresses
+
+* Implement scatter-gather at page level granularity so that the device does
+  not have to.
+
+* Provide system protection against "rogue" DMA by forcing all accesses to go
+  through the IOMMU and faulting when encountering accesses to unmapped
+  address regions.
+
+* Provide address space isolation between multiple contexts.
+
+  Example: Virtualization
+
+Device nodes compatible with this binding represent hardware with some of the
+above capabilities.
+
+IOMMUs can be single-master or multiple-master. Single-master IOMMU devices
+typically have a fixed association to the master device, whereas multiple-
+master IOMMU devices can translate accesses from more than one master.
+
+The device tree node of the IOMMU device's parent bus must contain a valid
+"dma-ranges" property that describes how the physical address space of the
+IOMMU maps to memory. An empty "dma-ranges" property means that there is a
+1:1 mapping from IOMMU to memory.
+
+Required properties:
+--------------------
+- #address-cells: The number of cells in an IOMMU specifier needed to encode
+  an address.
+- #size-cells: The number of cells in an IOMMU specifier needed to represent
+  the length of an address range.
+
+Typical values for the above include:
+- #address-cells = <0>, size-cells = <0>: Single master IOMMU devices are not
+  configurable and therefore no additional information needs to be encoded in
+  the specifier. This may also apply to multiple master IOMMU devices that do
+  not allow the association of masters to be configured.
+- #address-cells = <1>, size-cells = <0>: Multiple master IOMMU devices may
+  need to be configured in order to enable translation for a given master. In
+  such cases the single address cell corresponds to the master device's ID.
+- #address-cells = <2>, size-cells = <2>: Some IOMMU devices allow the DMA
+  window for masters to be configured. The first cell of the address in this
+  may contain the master device's ID for example, while the second cell could
+  contain the start of the DMA window for the given device. The length of the
+  DMA window is specified by two additional cells.
+
+
+IOMMU master node:
+==================
+
+Devices that access memory through an IOMMU are called masters. A device can
+have multiple master interfaces (to one or more IOMMU devices).
+
+Required properties:
+--------------------
+- iommus: A list of phandle and IOMMU specifier pairs that describe the IOMMU
+  master interfaces of the device. One entry in the list describes one master
+  interface of the device.
+
+When an "iommus" property is specified in a device tree node, the IOMMU will
+be used for address translation. If a "dma-ranges" property exists in the
+device's parent node it will be ignored. An exception to this rule is if the
+referenced IOMMU is disabled, in which case the "dma-ranges" property of the
+parent shall take effect.
+
+Optional properties:
+--------------------
+- iommu-names: A list of names identifying each entry in the "iommus"
+  property.
+
+
+Notes:
+======
+
+One possible extension to the above is to use an "iommus" property along with
+a "dma-ranges" property in a bus device node (such as PCI host bridges). This
+can be useful to describe how children on the bus relate to the IOMMU if they
+are not explicitly listed in the device tree (e.g. PCI devices). However, the
+requirements of that use-case haven't been fully determined yet. Implementing
+this is therefore not recommended without further discussion and extension of
+this binding.
+
+
+Examples:
+=========
+
+Single-master IOMMU:
+--------------------
+
+	iommu {
+		#address-cells = <0>;
+		#size-cells = <0>;
+	};
+
+	master {
+		iommus = <&/iommu>;
+	};
+
+Multiple-master IOMMU with fixed associations:
+----------------------------------------------
+
+	/* multiple-master IOMMU */
+	iommu {
+		/*
+		 * Masters are statically associated with this IOMMU and
+		 * address translation is always enabled.
+		 */
+		#address-cells = <0>;
+		#size-cells = <0>;
+	};
+
+	/* static association with IOMMU */
+	master at 1 {
+		reg = <1>;
+		iommus = <&/iommu>;
+	};
+
+	/* static association with IOMMU */
+	master at 2 {
+		reg = <2>;
+		iommus = <&/iommu>;
+	};
+
+Multiple-master IOMMU:
+----------------------
+
+	iommu {
+		/* the specifier represents the ID of the master */
+		#address-cells = <1>;
+		#size-cells = <0>;
+	};
+
+	master {
+		/* device has master ID 42 in the IOMMU */
+		iommus = <&/iommu 42>;
+	};
+
+Multiple-master IOMMU with configurable DMA window:
+---------------------------------------------------
+
+	/ {
+		#address-cells = <1>;
+		#size-cells = <1>;
+
+		iommu {
+			/* master ID, address of DMA window */
+			#address-cells = <2>;
+			#size-cells = <2>;
+		};
+
+		master {
+			/* master ID 42, 4 GiB DMA window starting at 0 */
+			iommus = <&/iommu  42 0  0x1 0x0>;
+		};
+	};
-- 
1.9.2