[PATCH] perf/smmuv3: Allow sharing MMIO registers with the SMMU driver

[PATCH] perf/smmuv3: Allow sharing MMIO registers with the SMMU driver

[Jean-Philippe Brucker]

Some Arm SMMUv3 implementations, for example Arm CoreLink MMU-600, embed
the PMCG registers into the SMMU MMIO regions. It currently causes probe
failure because the PMU and SMMU drivers request overlapping resources.

Avoid the conflict by calling devm_ioremap() directly from the PMU
driver. We loose some sanity-checking of the memory map provided by
firmware, which doesn't seem catastrophic.

Signed-off-by: Jean-Philippe Brucker <jean-philippe@linaro.org>
---

So this is the simplest solution, and I don't think we're missing much
by skipping the resource reservation. I've also been exploring a more
complex approach [1] which has the SMMU driver perform resource
reservation on behalf of the PMU driver, but I'm not sure it's
necessary.

Please test, I've only tried the RevC FastModel using devicetree so far.

[1] https://jpbrucker.net/git/linux/log/?h=smmu/pmu
---
 drivers/perf/arm_smmuv3_pmu.c | 28 +++++++++++++++++++++-------
 1 file changed, 21 insertions(+), 7 deletions(-)

diff --git a/drivers/perf/arm_smmuv3_pmu.c b/drivers/perf/arm_smmuv3_pmu.c
index ca183a53a7f10..ad63d1e73333f 100644
--- a/drivers/perf/arm_smmuv3_pmu.c
+++ b/drivers/perf/arm_smmuv3_pmu.c
@@ -730,8 +730,8 @@ static void smmu_pmu_get_acpi_options(struct smmu_pmu *smmu_pmu)
 
 static int smmu_pmu_probe(struct platform_device *pdev)
 {
+	struct resource *res_0, *res_1;
 	struct smmu_pmu *smmu_pmu;
-	struct resource *res_0;
 	u32 cfgr, reg_size;
 	u64 ceid_64[2];
 	int irq, err;
@@ -759,18 +759,32 @@ static int smmu_pmu_probe(struct platform_device *pdev)
 		.capabilities	= PERF_PMU_CAP_NO_EXCLUDE,
 	};
 
+	/*
+	 * If the PMCG registers are embedded into the SMMU regions, the
+	 * resources have to be shared with the SMMU driver. Use ioremap()
+	 * rather than ioremap_resource() to avoid conflicts.
+	 */
 	res_0 = platform_get_resource(pdev, IORESOURCE_MEM, 0);
-	smmu_pmu->reg_base = devm_ioremap_resource(dev, res_0);
-	if (IS_ERR(smmu_pmu->reg_base))
-		return PTR_ERR(smmu_pmu->reg_base);
+	if (!res_0)
+		return -ENXIO;
+
+	smmu_pmu->reg_base = devm_ioremap(dev, res_0->start,
+					  resource_size(res_0));
+	if (!smmu_pmu->reg_base)
+		return -ENOMEM;
 
 	cfgr = readl_relaxed(smmu_pmu->reg_base + SMMU_PMCG_CFGR);
 
 	/* Determine if page 1 is present */
 	if (cfgr & SMMU_PMCG_CFGR_RELOC_CTRS) {
-		smmu_pmu->reloc_base = devm_platform_ioremap_resource(pdev, 1);
-		if (IS_ERR(smmu_pmu->reloc_base))
-			return PTR_ERR(smmu_pmu->reloc_base);
+		res_1 = platform_get_resource(pdev, IORESOURCE_MEM, 1);
+		if (!res_1)
+			return -ENXIO;
+
+		smmu_pmu->reloc_base = devm_ioremap(dev, res_1->start,
+						    resource_size(res_1));
+		if (!smmu_pmu->reloc_base)
+			return -ENOMEM;
 	} else {
 		smmu_pmu->reloc_base = smmu_pmu->reg_base;
 	}
-- 
2.26.0

_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [PATCH] perf/smmuv3: Allow sharing MMIO registers with the SMMU driver

[Robin Murphy]

On 2020-04-21 4:57 pm, Jean-Philippe Brucker wrote:
> Some Arm SMMUv3 implementations, for example Arm CoreLink MMU-600, embed
> the PMCG registers into the SMMU MMIO regions. It currently causes probe
> failure because the PMU and SMMU drivers request overlapping resources.
> 
> Avoid the conflict by calling devm_ioremap() directly from the PMU
> driver. We loose some sanity-checking of the memory map provided by
> firmware, which doesn't seem catastrophic.
> 
> Signed-off-by: Jean-Philippe Brucker <jean-philippe@linaro.org>
> ---
> 
> So this is the simplest solution, and I don't think we're missing much
> by skipping the resource reservation. I've also been exploring a more
> complex approach [1] which has the SMMU driver perform resource
> reservation on behalf of the PMU driver, but I'm not sure it's
> necessary.

Now try it for potential future PMCGs on DTI masters in any old device 
or root complex ;)

If we did want to go down the resource-claiming route, rather than new 
inter-driver APIs it would probably be more sensible to just resolve the 
associated device and rifle through its resource list directly within 
the PMCG driver. Of course that probably leads to a whole bunch of probe 
ordering and dependency issues, and if the end result is just to make 
/proc/iomem look slightly nicer then I'd agree it's not worth the bother.

> Please test, I've only tried the RevC FastModel using devicetree so far.

For ACPI there's the additional fun that all the resources may already 
have been claimed at least once more, by companion devices, but I guess 
SMMU and PMCG at least escape that by virtue of not being namespace objects.

Robin.

> [1] https://jpbrucker.net/git/linux/log/?h=smmu/pmu
> ---
>   drivers/perf/arm_smmuv3_pmu.c | 28 +++++++++++++++++++++-------
>   1 file changed, 21 insertions(+), 7 deletions(-)
> 
> diff --git a/drivers/perf/arm_smmuv3_pmu.c b/drivers/perf/arm_smmuv3_pmu.c
> index ca183a53a7f10..ad63d1e73333f 100644
> --- a/drivers/perf/arm_smmuv3_pmu.c
> +++ b/drivers/perf/arm_smmuv3_pmu.c
> @@ -730,8 +730,8 @@ static void smmu_pmu_get_acpi_options(struct smmu_pmu *smmu_pmu)
>   
>   static int smmu_pmu_probe(struct platform_device *pdev)
>   {
> +	struct resource *res_0, *res_1;
>   	struct smmu_pmu *smmu_pmu;
> -	struct resource *res_0;
>   	u32 cfgr, reg_size;
>   	u64 ceid_64[2];
>   	int irq, err;
> @@ -759,18 +759,32 @@ static int smmu_pmu_probe(struct platform_device *pdev)
>   		.capabilities	= PERF_PMU_CAP_NO_EXCLUDE,
>   	};
>   
> +	/*
> +	 * If the PMCG registers are embedded into the SMMU regions, the
> +	 * resources have to be shared with the SMMU driver. Use ioremap()
> +	 * rather than ioremap_resource() to avoid conflicts.
> +	 */
>   	res_0 = platform_get_resource(pdev, IORESOURCE_MEM, 0);
> -	smmu_pmu->reg_base = devm_ioremap_resource(dev, res_0);
> -	if (IS_ERR(smmu_pmu->reg_base))
> -		return PTR_ERR(smmu_pmu->reg_base);
> +	if (!res_0)
> +		return -ENXIO;
> +
> +	smmu_pmu->reg_base = devm_ioremap(dev, res_0->start,
> +					  resource_size(res_0));
> +	if (!smmu_pmu->reg_base)
> +		return -ENOMEM;
>   
>   	cfgr = readl_relaxed(smmu_pmu->reg_base + SMMU_PMCG_CFGR);
>   
>   	/* Determine if page 1 is present */
>   	if (cfgr & SMMU_PMCG_CFGR_RELOC_CTRS) {
> -		smmu_pmu->reloc_base = devm_platform_ioremap_resource(pdev, 1);
> -		if (IS_ERR(smmu_pmu->reloc_base))
> -			return PTR_ERR(smmu_pmu->reloc_base);
> +		res_1 = platform_get_resource(pdev, IORESOURCE_MEM, 1);
> +		if (!res_1)
> +			return -ENXIO;
> +
> +		smmu_pmu->reloc_base = devm_ioremap(dev, res_1->start,
> +						    resource_size(res_1));
> +		if (!smmu_pmu->reloc_base)
> +			return -ENOMEM;
>   	} else {
>   		smmu_pmu->reloc_base = smmu_pmu->reg_base;
>   	}
> 
_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [PATCH] perf/smmuv3: Allow sharing MMIO registers with the SMMU driver

[Tuan Phan]

> On Apr 21, 2020, at 8:57 AM, Jean-Philippe Brucker <jean-philippe@linaro.org> wrote:
> 
> Some Arm SMMUv3 implementations, for example Arm CoreLink MMU-600, embed
> the PMCG registers into the SMMU MMIO regions. It currently causes probe
> failure because the PMU and SMMU drivers request overlapping resources.
> 
> Avoid the conflict by calling devm_ioremap() directly from the PMU
> driver. We loose some sanity-checking of the memory map provided by
> firmware, which doesn't seem catastrophic.
> 
> Signed-off-by: Jean-Philippe Brucker <jean-philippe@linaro.org>
> ---
> 
> So this is the simplest solution, and I don't think we're missing much
> by skipping the resource reservation. I've also been exploring a more
> complex approach [1] which has the SMMU driver perform resource
> reservation on behalf of the PMU driver, but I'm not sure it's
> necessary.
> 
> Please test, I've only tried the RevC FastModel using devicetree so far.
> 
> [1] https://jpbrucker.net/git/linux/log/?h=smmu/pmu
> ---
> drivers/perf/arm_smmuv3_pmu.c | 28 +++++++++++++++++++++-------
> 1 file changed, 21 insertions(+), 7 deletions(-)
> 
> diff --git a/drivers/perf/arm_smmuv3_pmu.c b/drivers/perf/arm_smmuv3_pmu.c
> index ca183a53a7f10..ad63d1e73333f 100644
> --- a/drivers/perf/arm_smmuv3_pmu.c
> +++ b/drivers/perf/arm_smmuv3_pmu.c
> @@ -730,8 +730,8 @@ static void smmu_pmu_get_acpi_options(struct smmu_pmu *smmu_pmu)
> 
> static int smmu_pmu_probe(struct platform_device *pdev)
> {
> +	struct resource *res_0, *res_1;
> 	struct smmu_pmu *smmu_pmu;
> -	struct resource *res_0;
> 	u32 cfgr, reg_size;
> 	u64 ceid_64[2];
> 	int irq, err;
> @@ -759,18 +759,32 @@ static int smmu_pmu_probe(struct platform_device *pdev)
> 		.capabilities	= PERF_PMU_CAP_NO_EXCLUDE,
> 	};
> 
> +	/*
> +	 * If the PMCG registers are embedded into the SMMU regions, the
> +	 * resources have to be shared with the SMMU driver. Use ioremap()
> +	 * rather than ioremap_resource() to avoid conflicts.
> +	 */
> 	res_0 = platform_get_resource(pdev, IORESOURCE_MEM, 0);
> -	smmu_pmu->reg_base = devm_ioremap_resource(dev, res_0);
> -	if (IS_ERR(smmu_pmu->reg_base))
> -		return PTR_ERR(smmu_pmu->reg_base);
> +	if (!res_0)
> +		return -ENXIO;
> +
> +	smmu_pmu->reg_base = devm_ioremap(dev, res_0->start,
> +					  resource_size(res_0));
> +	if (!smmu_pmu->reg_base)
> +		return -ENOMEM;
> 
> 	cfgr = readl_relaxed(smmu_pmu->reg_base + SMMU_PMCG_CFGR);
> 
> 	/* Determine if page 1 is present */
> 	if (cfgr & SMMU_PMCG_CFGR_RELOC_CTRS) {
> -		smmu_pmu->reloc_base = devm_platform_ioremap_resource(pdev, 1);
> -		if (IS_ERR(smmu_pmu->reloc_base))
> -			return PTR_ERR(smmu_pmu->reloc_base);
> +		res_1 = platform_get_resource(pdev, IORESOURCE_MEM, 1);
> +		if (!res_1)
> +			return -ENXIO;
> +
> +		smmu_pmu->reloc_base = devm_ioremap(dev, res_1->start,
> +						    resource_size(res_1));
> +		if (!smmu_pmu->reloc_base)
> +			return -ENOMEM;

I tested this patch on HW, however I need to add one more following change to make it works

@@ -2854,7 +2854,7 @@ static int arm_smmu_device_probe(struct platform_device *pdev)
        }
        ioaddr = res->start;
 
-       smmu->base = devm_ioremap_resource(dev, res);
+       smmu->base = devm_ioremap(dev, res->start, resource_size(res));
        if (IS_ERR(smmu->base))
                return PTR_ERR(smmu->base);


> 	} else {
> 		smmu_pmu->reloc_base = smmu_pmu->reg_base;
> 	}
> -- 
> 2.26.0
> 
> 
> _______________________________________________
> linux-arm-kernel mailing list
> linux-arm-kernel@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
> 

_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [PATCH] perf/smmuv3: Allow sharing MMIO registers with the SMMU driver

[Jean-Philippe Brucker]

On Tue, Apr 28, 2020 at 11:10:09AM -0700, Tuan Phan wrote:
> I tested this patch on HW, however I need to add one more following change to make it works

Thanks for testing. I don't understand why you need the change below
though, do you know which other region is conflicting with the SMMU?
It should be displayed in the error message and /proc/iomem.

Thanks,
Jean

> @@ -2854,7 +2854,7 @@ static int arm_smmu_device_probe(struct platform_device *pdev)
>         }
>         ioaddr = res->start;
>  
> -       smmu->base = devm_ioremap_resource(dev, res);
> +       smmu->base = devm_ioremap(dev, res->start, resource_size(res));
>         if (IS_ERR(smmu->base))
>                 return PTR_ERR(smmu->base);
> 
> 
> > 	} else {
> > 		smmu_pmu->reloc_base = smmu_pmu->reg_base;
> > 	}
> > -- 
> > 2.26.0
> > 
> > 
> > _______________________________________________
> > linux-arm-kernel mailing list
> > linux-arm-kernel@lists.infradead.org
> > http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
> > 
> 
_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [PATCH] perf/smmuv3: Allow sharing MMIO registers with the SMMU driver

[Tuan Phan]

> On Apr 29, 2020, at 12:21 AM, Jean-Philippe Brucker <jean-philippe@linaro.org> wrote:
> 
> On Tue, Apr 28, 2020 at 11:10:09AM -0700, Tuan Phan wrote:
>> I tested this patch on HW, however I need to add one more following change to make it works
> 
> Thanks for testing. I don't understand why you need the change below
> though, do you know which other region is conflicting with the SMMU?
> It should be displayed in the error message and /proc/iomem.
> 
> Thanks,
> Jean

The error if I don’t apply that patch:
[    4.943655] arm-smmu-v3 arm-smmu-v3.0.auto: can't request region for resource [mem 0x3bffe0000000-0x3bffe001ffff]

The output of /proc/iomem for that region:
3bffe0000000-3bffe001ffff : arm-smmu-v3.0.auto                                                                                                                
  3bffe0002000-3bffe0002fff : arm-smmu-v3-pmcg.17.auto                                                                                                        
  3bffe0012000-3bffe0012fff : arm-smmu-v3-pmcg.17.auto                                                                                                        
3bffe0042000-3bffe0042fff : arm-smmu-v3-pmcg.11.auto                                                                                                          
3bffe0052000-3bffe0052fff : arm-smmu-v3-pmcg.11.auto                                                                                                          
3bffe0062000-3bffe0062fff : arm-smmu-v3-pmcg.12.auto                                                                                                          
3bffe0072000-3bffe0072fff : arm-smmu-v3-pmcg.12.auto                                                                                                          
3bffe00a2000-3bffe00a2fff : arm-smmu-v3-pmcg.13.auto                                                                                                          
3bffe00b2000-3bffe00b2fff : arm-smmu-v3-pmcg.13.auto                                                                                                          
3bffe00e2000-3bffe00e2fff : arm-smmu-v3-pmcg.14.auto                                                                                                          
3bffe00f2000-3bffe00f2fff : arm-smmu-v3-pmcg.14.auto                                                                                                          
3bffe0102000-3bffe0102fff : arm-smmu-v3-pmcg.15.auto                                                                                                          
3bffe0112000-3bffe0112fff : arm-smmu-v3-pmcg.15.auto                                                                                                          
3bffe0142000-3bffe0142fff : arm-smmu-v3-pmcg.16.auto                                                                                                          
3bffe0152000-3bffe0152fff : arm-smmu-v3-pmcg.16.auto

> 
>> @@ -2854,7 +2854,7 @@ static int arm_smmu_device_probe(struct platform_device *pdev)
>>        }
>>        ioaddr = res->start;
>> 
>> -       smmu->base = devm_ioremap_resource(dev, res);
>> +       smmu->base = devm_ioremap(dev, res->start, resource_size(res));
>>        if (IS_ERR(smmu->base))
>>                return PTR_ERR(smmu->base);
>> 
>> 
>>> 	} else {
>>> 		smmu_pmu->reloc_base = smmu_pmu->reg_base;
>>> 	}
>>> -- 
>>> 2.26.0
>>> 
>>> 
>>> _______________________________________________
>>> linux-arm-kernel mailing list
>>> linux-arm-kernel@lists.infradead.org
>>> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
>>> 
>> 

_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [PATCH] perf/smmuv3: Allow sharing MMIO registers with the SMMU driver

[Jean-Philippe Brucker]

On Wed, Apr 29, 2020 at 11:01:05AM -0700, Tuan Phan wrote:
> 
> 
> > On Apr 29, 2020, at 12:21 AM, Jean-Philippe Brucker <jean-philippe@linaro.org> wrote:
> > 
> > On Tue, Apr 28, 2020 at 11:10:09AM -0700, Tuan Phan wrote:
> >> I tested this patch on HW, however I need to add one more following change to make it works
> > 
> > Thanks for testing. I don't understand why you need the change below
> > though, do you know which other region is conflicting with the SMMU?
> > It should be displayed in the error message and /proc/iomem.
> > 
> > Thanks,
> > Jean
> 
> The error if I don’t apply that patch:
> [    4.943655] arm-smmu-v3 arm-smmu-v3.0.auto: can't request region for resource [mem 0x3bffe0000000-0x3bffe001ffff]
> 
> The output of /proc/iomem for that region:
> 3bffe0000000-3bffe001ffff : arm-smmu-v3.0.auto                                                                                                                
>   3bffe0002000-3bffe0002fff : arm-smmu-v3-pmcg.17.auto                                                                                                        
>   3bffe0012000-3bffe0012fff : arm-smmu-v3-pmcg.17.auto                                                                                                        

Thanks for this. It looks like the regions are added to the resource tree
twice, when booting with ACPI:

When the IORT creates the platform devices, platform_device_add() inserts
these resources but doesn't mark them busy. As I was testing with
devicetree I was missing this step.

Then the SMMU probe calls devm_ioremap_resource(), which tries to reserve
the resources and mark them busy this time. If there only was the non-busy
SMMU region inserted above, then __request_region() would adds the
reserved region as a child. However since there are smaller PMCG regions
as children of the SMMU region, __request_region() fails.

Another idea is to avoid mapping the IMPDEF registers in the SMMU driver.
I think it's nicer, and I'll post that patch shortly.

Thanks,
Jean

_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [PATCH] perf/smmuv3: Allow sharing MMIO registers with the SMMU driver

[Will Deacon]

On Tue, Apr 21, 2020 at 05:57:46PM +0200, Jean-Philippe Brucker wrote:
> Some Arm SMMUv3 implementations, for example Arm CoreLink MMU-600, embed
> the PMCG registers into the SMMU MMIO regions. It currently causes probe
> failure because the PMU and SMMU drivers request overlapping resources.
> 
> Avoid the conflict by calling devm_ioremap() directly from the PMU
> driver. We loose some sanity-checking of the memory map provided by
> firmware, which doesn't seem catastrophic.
> 
> Signed-off-by: Jean-Philippe Brucker <jean-philippe@linaro.org>
> ---
> 
> So this is the simplest solution, and I don't think we're missing much
> by skipping the resource reservation. I've also been exploring a more
> complex approach [1] which has the SMMU driver perform resource
> reservation on behalf of the PMU driver, but I'm not sure it's
> necessary.
> 
> Please test, I've only tried the RevC FastModel using devicetree so far.
> 
> [1] https://jpbrucker.net/git/linux/log/?h=smmu/pmu
> ---
>  drivers/perf/arm_smmuv3_pmu.c | 28 +++++++++++++++++++++-------
>  1 file changed, 21 insertions(+), 7 deletions(-)

Is this patch still needed? I can't quite follow from the discussion.

Will
_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [PATCH] perf/smmuv3: Allow sharing MMIO registers with the SMMU driver

[Jean-Philippe Brucker]

On Wed, May 20, 2020 at 01:24:53PM +0100, Will Deacon wrote:
> On Tue, Apr 21, 2020 at 05:57:46PM +0200, Jean-Philippe Brucker wrote:
> > Some Arm SMMUv3 implementations, for example Arm CoreLink MMU-600, embed
> > the PMCG registers into the SMMU MMIO regions. It currently causes probe
> > failure because the PMU and SMMU drivers request overlapping resources.
> > 
> > Avoid the conflict by calling devm_ioremap() directly from the PMU
> > driver. We loose some sanity-checking of the memory map provided by
> > firmware, which doesn't seem catastrophic.
> > 
> > Signed-off-by: Jean-Philippe Brucker <jean-philippe@linaro.org>
> > ---
> > 
> > So this is the simplest solution, and I don't think we're missing much
> > by skipping the resource reservation. I've also been exploring a more
> > complex approach [1] which has the SMMU driver perform resource
> > reservation on behalf of the PMU driver, but I'm not sure it's
> > necessary.
> > 
> > Please test, I've only tried the RevC FastModel using devicetree so far.
> > 
> > [1] https://jpbrucker.net/git/linux/log/?h=smmu/pmu
> > ---
> >  drivers/perf/arm_smmuv3_pmu.c | 28 +++++++++++++++++++++-------
> >  1 file changed, 21 insertions(+), 7 deletions(-)
> 
> Is this patch still needed? I can't quite follow from the discussion.

No, it is superseded by "[PATCH v2] iommu/arm-smmu-v3: Don't reserve
implementation defined register space" that you applied yesterday
(Thanks!)

Jean
_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu