[PATCH] arm64: dts: juno: Enable some SMMUs

[PATCH] arm64: dts: juno: Enable some SMMUs

[Robin Murphy]

The IOMMU-backed DMA API support has now been in place for a while and
proven stable, so there's no real need to keep most of Juno's SMMUs
disabled. The USB, HDLCDs, and CoreSight ETR all just need to map RAM
buffers for DMA - enabling their SMMUs obviates CPU bounce buffering for
USB's streaming DMA to the upper memory bank, and lets the other two
allocate their relatively large coherent buffers without pressuring CMA.

Some more software work is still needed for the DMA-330 and PCIe before
those can accommodate SMMU translation correctly in all cases, so we
leave those alone for now.

Signed-off-by: Robin Murphy <robin.murphy@arm.com>
---
 arch/arm64/boot/dts/arm/juno-base.dtsi | 4 ----
 1 file changed, 4 deletions(-)

diff --git a/arch/arm64/boot/dts/arm/juno-base.dtsi b/arch/arm64/boot/dts/arm/juno-base.dtsi
index bfe7d683a42e..02aea43238fc 100644
--- a/arch/arm64/boot/dts/arm/juno-base.dtsi
+++ b/arch/arm64/boot/dts/arm/juno-base.dtsi
@@ -53,7 +53,6 @@
 		#global-interrupts = <1>;
 		dma-coherent;
 		power-domains = <&scpi_devpd 0>;
-		status = "disabled";
 	};
 
 	gic: interrupt-controller at 2c010000 {
@@ -546,7 +545,6 @@
 			     <GIC_SPI 99 IRQ_TYPE_LEVEL_HIGH>;
 		#iommu-cells = <1>;
 		#global-interrupts = <1>;
-		status = "disabled";
 	};
 
 	smmu_hdlcd0: iommu at 7fb20000 {
@@ -556,7 +554,6 @@
 			     <GIC_SPI 97 IRQ_TYPE_LEVEL_HIGH>;
 		#iommu-cells = <1>;
 		#global-interrupts = <1>;
-		status = "disabled";
 	};
 
 	smmu_usb: iommu at 7fb30000 {
@@ -567,7 +564,6 @@
 		#iommu-cells = <1>;
 		#global-interrupts = <1>;
 		dma-coherent;
-		status = "disabled";
 	};
 
 	dma at 7ff00000 {
-- 
2.12.2.dirty

[PATCH] arm64: dts: juno: Enable some SMMUs

[Jon Medhurst (Tixy)]

On Thu, 2017-05-18 at 13:23 +0100, Robin Murphy wrote:
> The IOMMU-backed DMA API support has now been in place for a while and
> proven stable, so there's no real need to keep most of Juno's SMMUs
> disabled. The USB, HDLCDs, and CoreSight ETR all just need to map RAM
> buffers for DMA - enabling their SMMUs obviates CPU bounce buffering for
> USB's streaming DMA to the upper memory bank, and lets the other two
> allocate their relatively large coherent buffers without pressuring CMA.
> 
> Some more software work is still needed for the DMA-330 and PCIe before
> those can accommodate SMMU translation correctly in all cases, so we
> leave those alone for now.
> 
> Signed-off-by: Robin Murphy <robin.murphy@arm.com>
> ---

For those that don't already know, status = "disabled" for smmu is
broken in Linux 4.12-rc1, and this fixes it...
https://lkml.org/lkml/2017/5/17/344

>  arch/arm64/boot/dts/arm/juno-base.dtsi | 4 ----
>  1 file changed, 4 deletions(-)
> 
> diff --git a/arch/arm64/boot/dts/arm/juno-base.dtsi b/arch/arm64/boot/dts/arm/juno-base.dtsi
> index bfe7d683a42e..02aea43238fc 100644
> --- a/arch/arm64/boot/dts/arm/juno-base.dtsi
> +++ b/arch/arm64/boot/dts/arm/juno-base.dtsi
> @@ -53,7 +53,6 @@
>  		#global-interrupts = <1>;
>  		dma-coherent;
>  		power-domains = <&scpi_devpd 0>;
> -		status = "disabled";
>  	};
>  
>  	gic: interrupt-controller at 2c010000 {
> @@ -546,7 +545,6 @@
>  			     <GIC_SPI 99 IRQ_TYPE_LEVEL_HIGH>;
>  		#iommu-cells = <1>;
>  		#global-interrupts = <1>;
> -		status = "disabled";
>  	};
>  
>  	smmu_hdlcd0: iommu at 7fb20000 {
> @@ -556,7 +554,6 @@
>  			     <GIC_SPI 97 IRQ_TYPE_LEVEL_HIGH>;
>  		#iommu-cells = <1>;
>  		#global-interrupts = <1>;
> -		status = "disabled";
>  	};
>  
>  	smmu_usb: iommu at 7fb30000 {
> @@ -567,7 +564,6 @@
>  		#iommu-cells = <1>;
>  		#global-interrupts = <1>;
>  		dma-coherent;
> -		status = "disabled";
>  	};
>  
>  	dma at 7ff00000 {

[PATCH] arm64: dts: juno: Enable some SMMUs

[Liviu Dudau]

On Thu, May 18, 2017 at 02:13:59PM +0100, Jon Medhurst (Tixy) wrote:
> On Thu, 2017-05-18 at 13:23 +0100, Robin Murphy wrote:
> > The IOMMU-backed DMA API support has now been in place for a while and
> > proven stable, so there's no real need to keep most of Juno's SMMUs
> > disabled. The USB, HDLCDs, and CoreSight ETR all just need to map RAM
> > buffers for DMA - enabling their SMMUs obviates CPU bounce buffering for
> > USB's streaming DMA to the upper memory bank, and lets the other two
> > allocate their relatively large coherent buffers without pressuring CMA.
> > 
> > Some more software work is still needed for the DMA-330 and PCIe before
> > those can accommodate SMMU translation correctly in all cases, so we
> > leave those alone for now.
> > 
> > Signed-off-by: Robin Murphy <robin.murphy@arm.com>
> > ---
> 
> For those that don't already know, status = "disabled" for smmu is
> broken in Linux 4.12-rc1, and this fixes it...
> https://lkml.org/lkml/2017/5/17/344

Hey, fancy that!! I was just starting to wonder why some of my "disabled" nodes
in the juno-base.dts were showing as "okay". :)

Thanks Tixy for the tip!
Liviu

> 
> >  arch/arm64/boot/dts/arm/juno-base.dtsi | 4 ----
> >  1 file changed, 4 deletions(-)
> > 
> > diff --git a/arch/arm64/boot/dts/arm/juno-base.dtsi b/arch/arm64/boot/dts/arm/juno-base.dtsi
> > index bfe7d683a42e..02aea43238fc 100644
> > --- a/arch/arm64/boot/dts/arm/juno-base.dtsi
> > +++ b/arch/arm64/boot/dts/arm/juno-base.dtsi
> > @@ -53,7 +53,6 @@
> >  		#global-interrupts = <1>;
> >  		dma-coherent;
> >  		power-domains = <&scpi_devpd 0>;
> > -		status = "disabled";
> >  	};
> >  
> >  	gic: interrupt-controller at 2c010000 {
> > @@ -546,7 +545,6 @@
> >  			     <GIC_SPI 99 IRQ_TYPE_LEVEL_HIGH>;
> >  		#iommu-cells = <1>;
> >  		#global-interrupts = <1>;
> > -		status = "disabled";
> >  	};
> >  
> >  	smmu_hdlcd0: iommu at 7fb20000 {
> > @@ -556,7 +554,6 @@
> >  			     <GIC_SPI 97 IRQ_TYPE_LEVEL_HIGH>;
> >  		#iommu-cells = <1>;
> >  		#global-interrupts = <1>;
> > -		status = "disabled";
> >  	};
> >  
> >  	smmu_usb: iommu at 7fb30000 {
> > @@ -567,7 +564,6 @@
> >  		#iommu-cells = <1>;
> >  		#global-interrupts = <1>;
> >  		dma-coherent;
> > -		status = "disabled";
> >  	};
> >  
> >  	dma at 7ff00000 {

-- 
====================
| I would like to |
| fix the world,  |
| but they're not |
| giving me the   |
 \ source code!  /
  ---------------
    ?\_(?)_/?

[PATCH] arm64: dts: juno: Enable some SMMUs

[Liviu Dudau]

On Thu, May 18, 2017 at 01:23:36PM +0100, Robin Murphy wrote:
> The IOMMU-backed DMA API support has now been in place for a while and
> proven stable, so there's no real need to keep most of Juno's SMMUs
> disabled. The USB, HDLCDs, and CoreSight ETR all just need to map RAM
> buffers for DMA - enabling their SMMUs obviates CPU bounce buffering for
> USB's streaming DMA to the upper memory bank, and lets the other two
> allocate their relatively large coherent buffers without pressuring CMA.
> 
> Some more software work is still needed for the DMA-330 and PCIe before
> those can accommodate SMMU translation correctly in all cases, so we
> leave those alone for now.
> 
> Signed-off-by: Robin Murphy <robin.murphy@arm.com>

For the HDLCDs' ones, you can have my Tested-by: Liviu Dudau <Liviu.Dudau@arm.com>.

For the whole thing:

Acked-by: Liviu Dudau <Liviu.Dudau@arm.com>

Best regards,
Liviu

> ---
>  arch/arm64/boot/dts/arm/juno-base.dtsi | 4 ----
>  1 file changed, 4 deletions(-)
> 
> diff --git a/arch/arm64/boot/dts/arm/juno-base.dtsi b/arch/arm64/boot/dts/arm/juno-base.dtsi
> index bfe7d683a42e..02aea43238fc 100644
> --- a/arch/arm64/boot/dts/arm/juno-base.dtsi
> +++ b/arch/arm64/boot/dts/arm/juno-base.dtsi
> @@ -53,7 +53,6 @@
>  		#global-interrupts = <1>;
>  		dma-coherent;
>  		power-domains = <&scpi_devpd 0>;
> -		status = "disabled";
>  	};
>  
>  	gic: interrupt-controller at 2c010000 {
> @@ -546,7 +545,6 @@
>  			     <GIC_SPI 99 IRQ_TYPE_LEVEL_HIGH>;
>  		#iommu-cells = <1>;
>  		#global-interrupts = <1>;
> -		status = "disabled";
>  	};
>  
>  	smmu_hdlcd0: iommu at 7fb20000 {
> @@ -556,7 +554,6 @@
>  			     <GIC_SPI 97 IRQ_TYPE_LEVEL_HIGH>;
>  		#iommu-cells = <1>;
>  		#global-interrupts = <1>;
> -		status = "disabled";
>  	};
>  
>  	smmu_usb: iommu at 7fb30000 {
> @@ -567,7 +564,6 @@
>  		#iommu-cells = <1>;
>  		#global-interrupts = <1>;
>  		dma-coherent;
> -		status = "disabled";
>  	};
>  
>  	dma at 7ff00000 {
> -- 
> 2.12.2.dirty
> 

-- 
====================
| I would like to |
| fix the world,  |
| but they're not |
| giving me the   |
 \ source code!  /
  ---------------
    ?\_(?)_/?

[PATCH] arm64: dts: juno: Enable some SMMUs

[Robin Murphy]

On 18/05/17 14:13, Jon Medhurst (Tixy) wrote:
> On Thu, 2017-05-18 at 13:23 +0100, Robin Murphy wrote:
>> The IOMMU-backed DMA API support has now been in place for a while and
>> proven stable, so there's no real need to keep most of Juno's SMMUs
>> disabled. The USB, HDLCDs, and CoreSight ETR all just need to map RAM
>> buffers for DMA - enabling their SMMUs obviates CPU bounce buffering for
>> USB's streaming DMA to the upper memory bank, and lets the other two
>> allocate their relatively large coherent buffers without pressuring CMA.
>>
>> Some more software work is still needed for the DMA-330 and PCIe before
>> those can accommodate SMMU translation correctly in all cases, so we
>> leave those alone for now.
>>
>> Signed-off-by: Robin Murphy <robin.murphy@arm.com>
>> ---
> 
> For those that don't already know, status = "disabled" for smmu is
> broken in Linux 4.12-rc1, and this fixes it...
> https://lkml.org/lkml/2017/5/17/344

Well, it's not *technically* broken - we do stop waiting for disabled
IOMMUs that never appear, you just need to get to an initramfs or other
workable userspace to load another driver such that the deferred probe
queue gets kicked again after that point :P

But yeah, the whole business reminded me that it was about time to do
this anyway. I should dig up that pl330 patch I started, too...

Robin.

>>  arch/arm64/boot/dts/arm/juno-base.dtsi | 4 ----
>>  1 file changed, 4 deletions(-)
>>
>> diff --git a/arch/arm64/boot/dts/arm/juno-base.dtsi b/arch/arm64/boot/dts/arm/juno-base.dtsi
>> index bfe7d683a42e..02aea43238fc 100644
>> --- a/arch/arm64/boot/dts/arm/juno-base.dtsi
>> +++ b/arch/arm64/boot/dts/arm/juno-base.dtsi
>> @@ -53,7 +53,6 @@
>>  		#global-interrupts = <1>;
>>  		dma-coherent;
>>  		power-domains = <&scpi_devpd 0>;
>> -		status = "disabled";
>>  	};
>>  
>>  	gic: interrupt-controller at 2c010000 {
>> @@ -546,7 +545,6 @@
>>  			     <GIC_SPI 99 IRQ_TYPE_LEVEL_HIGH>;
>>  		#iommu-cells = <1>;
>>  		#global-interrupts = <1>;
>> -		status = "disabled";
>>  	};
>>  
>>  	smmu_hdlcd0: iommu at 7fb20000 {
>> @@ -556,7 +554,6 @@
>>  			     <GIC_SPI 97 IRQ_TYPE_LEVEL_HIGH>;
>>  		#iommu-cells = <1>;
>>  		#global-interrupts = <1>;
>> -		status = "disabled";
>>  	};
>>  
>>  	smmu_usb: iommu at 7fb30000 {
>> @@ -567,7 +564,6 @@
>>  		#iommu-cells = <1>;
>>  		#global-interrupts = <1>;
>>  		dma-coherent;
>> -		status = "disabled";
>>  	};
>>  
>>  	dma at 7ff00000 {

[PATCH] arm64: dts: juno: Enable some SMMUs

[Sudeep Holla]

On 18/05/17 13:23, Robin Murphy wrote:
> The IOMMU-backed DMA API support has now been in place for a while and
> proven stable, so there's no real need to keep most of Juno's SMMUs
> disabled. The USB, HDLCDs, and CoreSight ETR all just need to map RAM
> buffers for DMA - enabling their SMMUs obviates CPU bounce buffering for
> USB's streaming DMA to the upper memory bank, and lets the other two
> allocate their relatively large coherent buffers without pressuring CMA.
> 
> Some more software work is still needed for the DMA-330 and PCIe before
> those can accommodate SMMU translation correctly in all cases, so we
> leave those alone for now.
> 

Queued for v4.13 [1]

IIUC, it's not a fix and can wait for v4.13. Let me know if that's not
the case.

-- 
Regards,
Sudeep

[1] https://git.kernel.org/sudeep.holla/linux/h/for-next/juno-updates

[PATCH] arm64: dts: juno: Enable some SMMUs

[Jean-Philippe Brucker]

Hello,

On 18/05/17 13:23, Robin Murphy wrote:
> The IOMMU-backed DMA API support has now been in place for a while and
> proven stable, so there's no real need to keep most of Juno's SMMUs
> disabled. The USB, HDLCDs, and CoreSight ETR all just need to map RAM
> buffers for DMA - enabling their SMMUs obviates CPU bounce buffering for
> USB's streaming DMA to the upper memory bank, and lets the other two
> allocate their relatively large coherent buffers without pressuring CMA.
> 
> Some more software work is still needed for the DMA-330 and PCIe before
> those can accommodate SMMU translation correctly in all cases, so we
> leave those alone for now.

Out of curiosity, what is missing for DMA-330? I can use dmatest over SMMU
on my juno-r1 by enabling the node, but I don't have any complex workload
yet. Guest pass-through is really unreliable and I'm trying to figure out why.

Thanks,
Jean

[PATCH] arm64: dts: juno: Enable some SMMUs

[Robin Murphy]

On 06/06/17 16:45, Jean-Philippe Brucker wrote:
> Hello,
> 
> On 18/05/17 13:23, Robin Murphy wrote:
>> The IOMMU-backed DMA API support has now been in place for a while and
>> proven stable, so there's no real need to keep most of Juno's SMMUs
>> disabled. The USB, HDLCDs, and CoreSight ETR all just need to map RAM
>> buffers for DMA - enabling their SMMUs obviates CPU bounce buffering for
>> USB's streaming DMA to the upper memory bank, and lets the other two
>> allocate their relatively large coherent buffers without pressuring CMA.
>>
>> Some more software work is still needed for the DMA-330 and PCIe before
>> those can accommodate SMMU translation correctly in all cases, so we
>> leave those alone for now.
> 
> Out of curiosity, what is missing for DMA-330? I can use dmatest over SMMU
> on my juno-r1 by enabling the node, but I don't have any complex workload
> yet. Guest pass-through is really unreliable and I'm trying to figure out why.

Yes, dmatest has always been fine* since both ends get mapped for DMA
appropriately by the client. The missing bit is, or should I say was,
4d6d74e22096 currently in linux-next.

Robin.

* For certain values of fine, anyway. There are a number of catastrophic
races in the driver that make it generally a bad idea to use more than 1
thread per channel.

> 
> Thanks,
> Jean
> 

[PATCH] arm64: dts: juno: Enable some SMMUs

[Jean-Philippe Brucker]

On 06/06/17 16:59, Robin Murphy wrote:
> On 06/06/17 16:45, Jean-Philippe Brucker wrote:
>> Hello,
>>
>> On 18/05/17 13:23, Robin Murphy wrote:
>>> The IOMMU-backed DMA API support has now been in place for a while and
>>> proven stable, so there's no real need to keep most of Juno's SMMUs
>>> disabled. The USB, HDLCDs, and CoreSight ETR all just need to map RAM
>>> buffers for DMA - enabling their SMMUs obviates CPU bounce buffering for
>>> USB's streaming DMA to the upper memory bank, and lets the other two
>>> allocate their relatively large coherent buffers without pressuring CMA.
>>>
>>> Some more software work is still needed for the DMA-330 and PCIe before
>>> those can accommodate SMMU translation correctly in all cases, so we
>>> leave those alone for now.
>>
>> Out of curiosity, what is missing for DMA-330? I can use dmatest over SMMU
>> on my juno-r1 by enabling the node, but I don't have any complex workload
>> yet. Guest pass-through is really unreliable and I'm trying to figure out why.
> 
> Yes, dmatest has always been fine* since both ends get mapped for DMA
> appropriately by the client. The missing bit is, or should I say was,
> 4d6d74e22096 currently in linux-next.
> 
> Robin.
> 
> * For certain values of fine, anyway. There are a number of catastrophic
> races in the driver that make it generally a bad idea to use more than 1
> thread per channel.

Ah I see, thanks. The issue I was seeing was mismatched dma-coherency,
which I fixed. But my guest does sometimes hang when running 1024
threads/chan in dmatest, so that might be it.

Thanks,
Jean