mpt2sas DMA mask

mpt2sas DMA mask

[Brian King]

The mpt2sas driver was changed late last year in that it now requests a 64 bit DMA
mask, then requests a 32 bit coherent DMA mask, then later requests a 64 bit coherent
DMA mask. This was 5fb1bf8aaa832e1e9ca3198de7bbecb8eff7db9c. This breaks 64 bit DMA
support for mpt2sas on Power and we always fall back to using 32 bit DMA. Looking
at the commit log, it looks like this was an intentional change.

Ben - you had a patch set you had posted to the list back in Feb of this year, but it
doesn't look like it got merged. 

https://lists.ozlabs.org/pipermail/linuxppc-dev/2015-February/125087.html

This would fix the issue I'm seeing on mpt2sas. Do you plan to dust that patch set
off and upstream it? Were there issues with it that still need to be resolved?

Thanks,

Brian

-- 
Brian King
Power Linux I/O
IBM Linux Technology Center

mpt2sas DMA mask

[Brian King]

The mpt2sas driver was changed late last year in that it now requests a 64 bit DMA
mask, then requests a 32 bit coherent DMA mask, then later requests a 64 bit coherent
DMA mask. This was 5fb1bf8aaa832e1e9ca3198de7bbecb8eff7db9c. This breaks 64 bit DMA
support for mpt2sas on Power and we always fall back to using 32 bit DMA. Looking
at the commit log, it looks like this was an intentional change.

Ben - you had a patch set you had posted to the list back in Feb of this year, but it
doesn't look like it got merged. 

https://lists.ozlabs.org/pipermail/linuxppc-dev/2015-February/125087.html

This would fix the issue I'm seeing on mpt2sas. Do you plan to dust that patch set
off and upstream it? Were there issues with it that still need to be resolved?

Thanks,

Brian

-- 
Brian King
Power Linux I/O
IBM Linux Technology Center

Re: mpt2sas DMA mask

[Benjamin Herrenschmidt]

On Tue, 2015-05-12 at 17:07 -0500, Brian King wrote:
> The mpt2sas driver was changed late last year in that it now requests a 64 bit DMA
> mask, then requests a 32 bit coherent DMA mask, then later requests a 64 bit coherent
> DMA mask. This was 5fb1bf8aaa832e1e9ca3198de7bbecb8eff7db9c. This breaks 64 bit DMA
> support for mpt2sas on Power and we always fall back to using 32 bit DMA. Looking
> at the commit log, it looks like this was an intentional change.
> 
> Ben - you had a patch set you had posted to the list back in Feb of this year, but it
> doesn't look like it got merged. 

Right, it was broken, we can't do that unfortunately. It's a hard
problem to fix.

> https://lists.ozlabs.org/pipermail/linuxppc-dev/2015-February/125087.html
> 
> This would fix the issue I'm seeing on mpt2sas. Do you plan to dust that patch set
> off and upstream it? Were there issues with it that still need to be resolved?

No that patch doesn't actually work.

What we need is essentially a new set of DMA ops that can route an
individual map request via the iommu or the bypass window depending on
what mask applies.

Ben.

Re: mpt2sas DMA mask

[Benjamin Herrenschmidt]

On Tue, 2015-05-12 at 17:07 -0500, Brian King wrote:
> The mpt2sas driver was changed late last year in that it now requests a 64 bit DMA
> mask, then requests a 32 bit coherent DMA mask, then later requests a 64 bit coherent
> DMA mask. This was 5fb1bf8aaa832e1e9ca3198de7bbecb8eff7db9c. This breaks 64 bit DMA
> support for mpt2sas on Power and we always fall back to using 32 bit DMA. Looking
> at the commit log, it looks like this was an intentional change.
> 
> Ben - you had a patch set you had posted to the list back in Feb of this year, but it
> doesn't look like it got merged. 

Right, it was broken, we can't do that unfortunately. It's a hard
problem to fix.

> https://lists.ozlabs.org/pipermail/linuxppc-dev/2015-February/125087.html
> 
> This would fix the issue I'm seeing on mpt2sas. Do you plan to dust that patch set
> off and upstream it? Were there issues with it that still need to be resolved?

No that patch doesn't actually work.

What we need is essentially a new set of DMA ops that can route an
individual map request via the iommu or the bypass window depending on
what mask applies.

Ben.

[PATCH] mpt2sas: Fall back to 64 bit coherent mask if 64 bit DMA / 32 bit coherent mask not supported

[Brian King]

On 05/12/2015 05:10 PM, Benjamin Herrenschmidt wrote:
> On Tue, 2015-05-12 at 17:07 -0500, Brian King wrote:
>> The mpt2sas driver was changed late last year in that it now requests a 64 bit DMA
>> mask, then requests a 32 bit coherent DMA mask, then later requests a 64 bit coherent
>> DMA mask. This was 5fb1bf8aaa832e1e9ca3198de7bbecb8eff7db9c. This breaks 64 bit DMA
>> support for mpt2sas on Power and we always fall back to using 32 bit DMA. Looking
>> at the commit log, it looks like this was an intentional change.
>>
>> Ben - you had a patch set you had posted to the list back in Feb of this year, but it
>> doesn't look like it got merged. 
> 
> Right, it was broken, we can't do that unfortunately. It's a hard
> problem to fix.
> 
>> https://lists.ozlabs.org/pipermail/linuxppc-dev/2015-February/125087.html
>>
>> This would fix the issue I'm seeing on mpt2sas. Do you plan to dust that patch set
>> off and upstream it? Were there issues with it that still need to be resolved?
> 
> No that patch doesn't actually work.
> 
> What we need is essentially a new set of DMA ops that can route an
> individual map request via the iommu or the bypass window depending on
> what mask applies.

Ok. For the interim, then, Sreekanth, would you consider a patch such as the one
below to allow mpt2sas to support 64 bit DMA on architectures that don't support
a 32 bit coherent mask and a 64 bit dma mask at the same time? Only compile tested
at this point. Will run it on a Power machine shortly to confirm it doesn't
break anything...


-- 
Brian King
Power Linux I/O
IBM Linux Technology Center


8<-------


Commit 5fb1bf8aaa832e1e9ca3198de7bbecb8eff7db9c broke 64 bit DMA for mpt2sas on Power.
That commit changed the sequence for setting up the DMA and coherent DMA masks so
that during initialization the driver requests a 64 bit DMA mask and a 32 bit consistent
DMA mask, then later requests a 64 bit consistent DMA mask. The Power architecture does
not currently support this, which results in always falling back to a 32 bit DMA window,
which has a negative impact on performance. Tweak this algorithm slightly so that
if requesting a 32 bit consistent mask fails after we've successfully set a 64 bit
DMA mask, just try to get a 64 bit consistent mask. This should preserve existing
behavior on platforms that support mixed mask setting and restore previous functionality
to those that do not.

Signed-off-by: Brian King <brking@linux.vnet.ibm.com>
---

 drivers/scsi/mpt2sas/mpt2sas_base.c |   14 ++++++++------
 1 file changed, 8 insertions(+), 6 deletions(-)

diff -puN drivers/scsi/mpt2sas/mpt2sas_base.c~mpt2sas_dma_mask drivers/scsi/mpt2sas/mpt2sas_base.c
--- linux/drivers/scsi/mpt2sas/mpt2sas_base.c~mpt2sas_dma_mask	2015-05-12 16:09:58.228687852 -0500
+++ linux-bjking1/drivers/scsi/mpt2sas/mpt2sas_base.c	2015-05-12 17:38:31.588060903 -0500
@@ -1200,12 +1200,14 @@ _base_config_dma_addressing(struct MPT2S
 		const uint64_t required_mask =
 		    dma_get_required_mask(&pdev->dev);
 		if ((required_mask > DMA_BIT_MASK(32)) &&
-		    !pci_set_dma_mask(pdev, DMA_BIT_MASK(64)) &&
-		    !pci_set_consistent_dma_mask(pdev, consistent_dma_mask)) {
-			ioc->base_add_sg_single = &_base_add_sg_single_64;
-			ioc->sge_size = sizeof(Mpi2SGESimple64_t);
-			ioc->dma_mask = 64;
-			goto out;
+		    !pci_set_dma_mask(pdev, DMA_BIT_MASK(64))) {
+			if (!pci_set_consistent_dma_mask(pdev, consistent_dma_mask) ||
+			    !pci_set_consistent_dma_mask(pdev, DMA_BIT_MASK(64))) {
+				ioc->base_add_sg_single = &_base_add_sg_single_64;
+				ioc->sge_size = sizeof(Mpi2SGESimple64_t);
+				ioc->dma_mask = 64;
+				goto out;
+			}
 		}
 	}
 
_

[PATCH] mpt2sas: Fall back to 64 bit coherent mask if 64 bit DMA / 32 bit coherent mask not supported

[Brian King]

On 05/12/2015 05:10 PM, Benjamin Herrenschmidt wrote:
> On Tue, 2015-05-12 at 17:07 -0500, Brian King wrote:
>> The mpt2sas driver was changed late last year in that it now requests a 64 bit DMA
>> mask, then requests a 32 bit coherent DMA mask, then later requests a 64 bit coherent
>> DMA mask. This was 5fb1bf8aaa832e1e9ca3198de7bbecb8eff7db9c. This breaks 64 bit DMA
>> support for mpt2sas on Power and we always fall back to using 32 bit DMA. Looking
>> at the commit log, it looks like this was an intentional change.
>>
>> Ben - you had a patch set you had posted to the list back in Feb of this year, but it
>> doesn't look like it got merged. 
> 
> Right, it was broken, we can't do that unfortunately. It's a hard
> problem to fix.
> 
>> https://lists.ozlabs.org/pipermail/linuxppc-dev/2015-February/125087.html
>>
>> This would fix the issue I'm seeing on mpt2sas. Do you plan to dust that patch set
>> off and upstream it? Were there issues with it that still need to be resolved?
> 
> No that patch doesn't actually work.
> 
> What we need is essentially a new set of DMA ops that can route an
> individual map request via the iommu or the bypass window depending on
> what mask applies.

Ok. For the interim, then, Sreekanth, would you consider a patch such as the one
below to allow mpt2sas to support 64 bit DMA on architectures that don't support
a 32 bit coherent mask and a 64 bit dma mask at the same time? Only compile tested
at this point. Will run it on a Power machine shortly to confirm it doesn't
break anything...


-- 
Brian King
Power Linux I/O
IBM Linux Technology Center


8<-------


Commit 5fb1bf8aaa832e1e9ca3198de7bbecb8eff7db9c broke 64 bit DMA for mpt2sas on Power.
That commit changed the sequence for setting up the DMA and coherent DMA masks so
that during initialization the driver requests a 64 bit DMA mask and a 32 bit consistent
DMA mask, then later requests a 64 bit consistent DMA mask. The Power architecture does
not currently support this, which results in always falling back to a 32 bit DMA window,
which has a negative impact on performance. Tweak this algorithm slightly so that
if requesting a 32 bit consistent mask fails after we've successfully set a 64 bit
DMA mask, just try to get a 64 bit consistent mask. This should preserve existing
behavior on platforms that support mixed mask setting and restore previous functionality
to those that do not.

Signed-off-by: Brian King <brking@linux.vnet.ibm.com>
---

 drivers/scsi/mpt2sas/mpt2sas_base.c |   14 ++++++++------
 1 file changed, 8 insertions(+), 6 deletions(-)

diff -puN drivers/scsi/mpt2sas/mpt2sas_base.c~mpt2sas_dma_mask drivers/scsi/mpt2sas/mpt2sas_base.c
--- linux/drivers/scsi/mpt2sas/mpt2sas_base.c~mpt2sas_dma_mask	2015-05-12 16:09:58.228687852 -0500
+++ linux-bjking1/drivers/scsi/mpt2sas/mpt2sas_base.c	2015-05-12 17:38:31.588060903 -0500
@@ -1200,12 +1200,14 @@ _base_config_dma_addressing(struct MPT2S
 		const uint64_t required_mask =
 		    dma_get_required_mask(&pdev->dev);
 		if ((required_mask > DMA_BIT_MASK(32)) &&
-		    !pci_set_dma_mask(pdev, DMA_BIT_MASK(64)) &&
-		    !pci_set_consistent_dma_mask(pdev, consistent_dma_mask)) {
-			ioc->base_add_sg_single = &_base_add_sg_single_64;
-			ioc->sge_size = sizeof(Mpi2SGESimple64_t);
-			ioc->dma_mask = 64;
-			goto out;
+		    !pci_set_dma_mask(pdev, DMA_BIT_MASK(64))) {
+			if (!pci_set_consistent_dma_mask(pdev, consistent_dma_mask) ||
+			    !pci_set_consistent_dma_mask(pdev, DMA_BIT_MASK(64))) {
+				ioc->base_add_sg_single = &_base_add_sg_single_64;
+				ioc->sge_size = sizeof(Mpi2SGESimple64_t);
+				ioc->dma_mask = 64;
+				goto out;
+			}
 		}
 	}
 
_

Re: [PATCH] mpt2sas: Fall back to 64 bit coherent mask if 64 bit DMA / 32 bit coherent mask not supported

[Arnd Bergmann]

On Tuesday 12 May 2015 18:24:43 Brian King wrote:
> 
> Commit 5fb1bf8aaa832e1e9ca3198de7bbecb8eff7db9c broke 64 bit DMA for mpt2sas on Power.
> That commit changed the sequence for setting up the DMA and coherent DMA masks so
> that during initialization the driver requests a 64 bit DMA mask and a 32 bit consistent
> DMA mask, then later requests a 64 bit consistent DMA mask. The Power architecture does
> not currently support this, which results in always falling back to a 32 bit DMA window,
> which has a negative impact on performance. Tweak this algorithm slightly so that
> if requesting a 32 bit consistent mask fails after we've successfully set a 64 bit
> DMA mask, just try to get a 64 bit consistent mask. This should preserve existing
> behavior on platforms that support mixed mask setting and restore previous functionality
> to those that do not.
> 
> Signed-off-by: Brian King <brking@linux.vnet.ibm.com>

I believe the way the API is designed, it should guarantee that after dma_set_mask()
succeeds for a device, dma_set_coherent_mask() with the same mask will also succeed.

Could you just call dma_set_mask_and_coherent() here to avoid that complex logic?

	Arnd

Re: [PATCH] mpt2sas: Fall back to 64 bit coherent mask if 64 bit DMA / 32 bit coherent mask not supported

[Arnd Bergmann]

On Tuesday 12 May 2015 18:24:43 Brian King wrote:
> 
> Commit 5fb1bf8aaa832e1e9ca3198de7bbecb8eff7db9c broke 64 bit DMA for mpt2sas on Power.
> That commit changed the sequence for setting up the DMA and coherent DMA masks so
> that during initialization the driver requests a 64 bit DMA mask and a 32 bit consistent
> DMA mask, then later requests a 64 bit consistent DMA mask. The Power architecture does
> not currently support this, which results in always falling back to a 32 bit DMA window,
> which has a negative impact on performance. Tweak this algorithm slightly so that
> if requesting a 32 bit consistent mask fails after we've successfully set a 64 bit
> DMA mask, just try to get a 64 bit consistent mask. This should preserve existing
> behavior on platforms that support mixed mask setting and restore previous functionality
> to those that do not.
> 
> Signed-off-by: Brian King <brking@linux.vnet.ibm.com>

I believe the way the API is designed, it should guarantee that after dma_set_mask()
succeeds for a device, dma_set_coherent_mask() with the same mask will also succeed.

Could you just call dma_set_mask_and_coherent() here to avoid that complex logic?

	Arnd

Re: [PATCH] mpt2sas: Fall back to 64 bit coherent mask if 64 bit DMA / 32 bit coherent mask not supported

[Brian King]

On 05/13/2015 03:10 AM, Arnd Bergmann wrote:
> On Tuesday 12 May 2015 18:24:43 Brian King wrote:
>>
>> Commit 5fb1bf8aaa832e1e9ca3198de7bbecb8eff7db9c broke 64 bit DMA for mpt2sas on Power.
>> That commit changed the sequence for setting up the DMA and coherent DMA masks so
>> that during initialization the driver requests a 64 bit DMA mask and a 32 bit consistent
>> DMA mask, then later requests a 64 bit consistent DMA mask. The Power architecture does
>> not currently support this, which results in always falling back to a 32 bit DMA window,
>> which has a negative impact on performance. Tweak this algorithm slightly so that
>> if requesting a 32 bit consistent mask fails after we've successfully set a 64 bit
>> DMA mask, just try to get a 64 bit consistent mask. This should preserve existing
>> behavior on platforms that support mixed mask setting and restore previous functionality
>> to those that do not.
>>
>> Signed-off-by: Brian King <brking@linux.vnet.ibm.com>
> 
> I believe the way the API is designed, it should guarantee that after dma_set_mask()
> succeeds for a device, dma_set_coherent_mask() with the same mask will also succeed.
> 
> Could you just call dma_set_mask_and_coherent() here to avoid that complex logic?

I don't think that would work. The mpt2sas driver wants to set the dma mask to 64bits
but set the coherent mask to 32 bits, then my change is to set the coherent mask to
64bits if setting it to 32bit fails. I'm not seeing how using dma_set_mask_and_coherent
would simplify anything here.

Thanks,

Brian

-- 
Brian King
Power Linux I/O
IBM Linux Technology Center

Re: [PATCH] mpt2sas: Fall back to 64 bit coherent mask if 64 bit DMA / 32 bit coherent mask not supported

[Brian King]

On 05/13/2015 03:10 AM, Arnd Bergmann wrote:
> On Tuesday 12 May 2015 18:24:43 Brian King wrote:
>>
>> Commit 5fb1bf8aaa832e1e9ca3198de7bbecb8eff7db9c broke 64 bit DMA for mpt2sas on Power.
>> That commit changed the sequence for setting up the DMA and coherent DMA masks so
>> that during initialization the driver requests a 64 bit DMA mask and a 32 bit consistent
>> DMA mask, then later requests a 64 bit consistent DMA mask. The Power architecture does
>> not currently support this, which results in always falling back to a 32 bit DMA window,
>> which has a negative impact on performance. Tweak this algorithm slightly so that
>> if requesting a 32 bit consistent mask fails after we've successfully set a 64 bit
>> DMA mask, just try to get a 64 bit consistent mask. This should preserve existing
>> behavior on platforms that support mixed mask setting and restore previous functionality
>> to those that do not.
>>
>> Signed-off-by: Brian King <brking@linux.vnet.ibm.com>
> 
> I believe the way the API is designed, it should guarantee that after dma_set_mask()
> succeeds for a device, dma_set_coherent_mask() with the same mask will also succeed.
> 
> Could you just call dma_set_mask_and_coherent() here to avoid that complex logic?

I don't think that would work. The mpt2sas driver wants to set the dma mask to 64bits
but set the coherent mask to 32 bits, then my change is to set the coherent mask to
64bits if setting it to 32bit fails. I'm not seeing how using dma_set_mask_and_coherent
would simplify anything here.

Thanks,

Brian

-- 
Brian King
Power Linux I/O
IBM Linux Technology Center

Re: [PATCH] mpt2sas: Fall back to 64 bit coherent mask if 64 bit DMA / 32 bit coherent mask not supported

[Arnd Bergmann]

On Wednesday 13 May 2015 08:23:41 Brian King wrote:
> On 05/13/2015 03:10 AM, Arnd Bergmann wrote:
> > On Tuesday 12 May 2015 18:24:43 Brian King wrote:
> >>
> >> Commit 5fb1bf8aaa832e1e9ca3198de7bbecb8eff7db9c broke 64 bit DMA for mpt2sas on Power.
> >> That commit changed the sequence for setting up the DMA and coherent DMA masks so
> >> that during initialization the driver requests a 64 bit DMA mask and a 32 bit consistent
> >> DMA mask, then later requests a 64 bit consistent DMA mask. The Power architecture does
> >> not currently support this, which results in always falling back to a 32 bit DMA window,
> >> which has a negative impact on performance. Tweak this algorithm slightly so that
> >> if requesting a 32 bit consistent mask fails after we've successfully set a 64 bit
> >> DMA mask, just try to get a 64 bit consistent mask. This should preserve existing
> >> behavior on platforms that support mixed mask setting and restore previous functionality
> >> to those that do not.
> >>
> >> Signed-off-by: Brian King <brking@linux.vnet.ibm.com>
> > 
> > I believe the way the API is designed, it should guarantee that after dma_set_mask()
> > succeeds for a device, dma_set_coherent_mask() with the same mask will also succeed.
> > 
> > Could you just call dma_set_mask_and_coherent() here to avoid that complex logic?
> 
> I don't think that would work. The mpt2sas driver wants to set the dma mask to 64bits
> but set the coherent mask to 32 bits, then my change is to set the coherent mask to
> 64bits if setting it to 32bit fails. I'm not seeing how using dma_set_mask_and_coherent
> would simplify anything here.

My question was more about why the driver would ask for a 32-bit coherent
mask to start with. Is this a limitation in the mpt2sas device that can
only have descriptors at low addresses, or is it trying to work around some
bug in a particular host system?

	Arnd

Re: [PATCH] mpt2sas: Fall back to 64 bit coherent mask if 64 bit DMA / 32 bit coherent mask not supported

[Arnd Bergmann]

On Wednesday 13 May 2015 08:23:41 Brian King wrote:
> On 05/13/2015 03:10 AM, Arnd Bergmann wrote:
> > On Tuesday 12 May 2015 18:24:43 Brian King wrote:
> >>
> >> Commit 5fb1bf8aaa832e1e9ca3198de7bbecb8eff7db9c broke 64 bit DMA for mpt2sas on Power.
> >> That commit changed the sequence for setting up the DMA and coherent DMA masks so
> >> that during initialization the driver requests a 64 bit DMA mask and a 32 bit consistent
> >> DMA mask, then later requests a 64 bit consistent DMA mask. The Power architecture does
> >> not currently support this, which results in always falling back to a 32 bit DMA window,
> >> which has a negative impact on performance. Tweak this algorithm slightly so that
> >> if requesting a 32 bit consistent mask fails after we've successfully set a 64 bit
> >> DMA mask, just try to get a 64 bit consistent mask. This should preserve existing
> >> behavior on platforms that support mixed mask setting and restore previous functionality
> >> to those that do not.
> >>
> >> Signed-off-by: Brian King <brking@linux.vnet.ibm.com>
> > 
> > I believe the way the API is designed, it should guarantee that after dma_set_mask()
> > succeeds for a device, dma_set_coherent_mask() with the same mask will also succeed.
> > 
> > Could you just call dma_set_mask_and_coherent() here to avoid that complex logic?
> 
> I don't think that would work. The mpt2sas driver wants to set the dma mask to 64bits
> but set the coherent mask to 32 bits, then my change is to set the coherent mask to
> 64bits if setting it to 32bit fails. I'm not seeing how using dma_set_mask_and_coherent
> would simplify anything here.

My question was more about why the driver would ask for a 32-bit coherent
mask to start with. Is this a limitation in the mpt2sas device that can
only have descriptors at low addresses, or is it trying to work around some
bug in a particular host system?

	Arnd

Re: [PATCH] mpt2sas: Fall back to 64 bit coherent mask if 64 bit DMA / 32 bit coherent mask not supported

[James Bottomley]

On Wed, 2015-05-13 at 15:31 +0200, Arnd Bergmann wrote:
> On Wednesday 13 May 2015 08:23:41 Brian King wrote:
> > On 05/13/2015 03:10 AM, Arnd Bergmann wrote:
> > > On Tuesday 12 May 2015 18:24:43 Brian King wrote:
> > >>
> > >> Commit 5fb1bf8aaa832e1e9ca3198de7bbecb8eff7db9c broke 64 bit DMA for mpt2sas on Power.
> > >> That commit changed the sequence for setting up the DMA and coherent DMA masks so
> > >> that during initialization the driver requests a 64 bit DMA mask and a 32 bit consistent
> > >> DMA mask, then later requests a 64 bit consistent DMA mask. The Power architecture does
> > >> not currently support this, which results in always falling back to a 32 bit DMA window,
> > >> which has a negative impact on performance. Tweak this algorithm slightly so that
> > >> if requesting a 32 bit consistent mask fails after we've successfully set a 64 bit
> > >> DMA mask, just try to get a 64 bit consistent mask. This should preserve existing
> > >> behavior on platforms that support mixed mask setting and restore previous functionality
> > >> to those that do not.
> > >>
> > >> Signed-off-by: Brian King <brking@linux.vnet.ibm.com>
> > > 
> > > I believe the way the API is designed, it should guarantee that after dma_set_mask()
> > > succeeds for a device, dma_set_coherent_mask() with the same mask will also succeed.
> > > 
> > > Could you just call dma_set_mask_and_coherent() here to avoid that complex logic?
> > 
> > I don't think that would work. The mpt2sas driver wants to set the dma mask to 64bits
> > but set the coherent mask to 32 bits, then my change is to set the coherent mask to
> > 64bits if setting it to 32bit fails. I'm not seeing how using dma_set_mask_and_coherent
> > would simplify anything here.
> 
> My question was more about why the driver would ask for a 32-bit coherent
> mask to start with. Is this a limitation in the mpt2sas device that can
> only have descriptors at low addresses, or is it trying to work around some
> bug in a particular host system?

Can't speak for mp2sas, but it's a standard pattern for a lot of script
based (internal microcode) devices.  The microcode often gets extended
for external 64 bit addressing just by adding extra descriptor types, so
the device itself can do DMA to/from the full 64 bit range.  However,
the actual bit width of the microcode processor is usually left at 32
bits.  This means that if microcode scripts are executed out of system
memory, they have to be located in the lower 32 bits of physical memory
otherwise the internal microcode pointer references wrap (the aic79xx
has exactly this problem).  Hence they need a 64 bit mask but a 32 bit
coherent mask (coherent memory is where the scripts are located).

James

Re: [PATCH] mpt2sas: Fall back to 64 bit coherent mask if 64 bit DMA / 32 bit coherent mask not supported

[James Bottomley]

On Wed, 2015-05-13 at 15:31 +0200, Arnd Bergmann wrote:
> On Wednesday 13 May 2015 08:23:41 Brian King wrote:
> > On 05/13/2015 03:10 AM, Arnd Bergmann wrote:
> > > On Tuesday 12 May 2015 18:24:43 Brian King wrote:
> > >>
> > >> Commit 5fb1bf8aaa832e1e9ca3198de7bbecb8eff7db9c broke 64 bit DMA for mpt2sas on Power.
> > >> That commit changed the sequence for setting up the DMA and coherent DMA masks so
> > >> that during initialization the driver requests a 64 bit DMA mask and a 32 bit consistent
> > >> DMA mask, then later requests a 64 bit consistent DMA mask. The Power architecture does
> > >> not currently support this, which results in always falling back to a 32 bit DMA window,
> > >> which has a negative impact on performance. Tweak this algorithm slightly so that
> > >> if requesting a 32 bit consistent mask fails after we've successfully set a 64 bit
> > >> DMA mask, just try to get a 64 bit consistent mask. This should preserve existing
> > >> behavior on platforms that support mixed mask setting and restore previous functionality
> > >> to those that do not.
> > >>
> > >> Signed-off-by: Brian King <brking@linux.vnet.ibm.com>
> > > 
> > > I believe the way the API is designed, it should guarantee that after dma_set_mask()
> > > succeeds for a device, dma_set_coherent_mask() with the same mask will also succeed.
> > > 
> > > Could you just call dma_set_mask_and_coherent() here to avoid that complex logic?
> > 
> > I don't think that would work. The mpt2sas driver wants to set the dma mask to 64bits
> > but set the coherent mask to 32 bits, then my change is to set the coherent mask to
> > 64bits if setting it to 32bit fails. I'm not seeing how using dma_set_mask_and_coherent
> > would simplify anything here.
> 
> My question was more about why the driver would ask for a 32-bit coherent
> mask to start with. Is this a limitation in the mpt2sas device that can
> only have descriptors at low addresses, or is it trying to work around some
> bug in a particular host system?

Can't speak for mp2sas, but it's a standard pattern for a lot of script
based (internal microcode) devices.  The microcode often gets extended
for external 64 bit addressing just by adding extra descriptor types, so
the device itself can do DMA to/from the full 64 bit range.  However,
the actual bit width of the microcode processor is usually left at 32
bits.  This means that if microcode scripts are executed out of system
memory, they have to be located in the lower 32 bits of physical memory
otherwise the internal microcode pointer references wrap (the aic79xx
has exactly this problem).  Hence they need a 64 bit mask but a 32 bit
coherent mask (coherent memory is where the scripts are located).

James

Re: [PATCH] mpt2sas: Fall back to 64 bit coherent mask if 64 bit DMA / 32 bit coherent mask not supported

[Brian King]

On 05/13/2015 08:31 AM, Arnd Bergmann wrote:
> On Wednesday 13 May 2015 08:23:41 Brian King wrote:
>> On 05/13/2015 03:10 AM, Arnd Bergmann wrote:
>>> On Tuesday 12 May 2015 18:24:43 Brian King wrote:
>>>>
>>>> Commit 5fb1bf8aaa832e1e9ca3198de7bbecb8eff7db9c broke 64 bit DMA for mpt2sas on Power.
>>>> That commit changed the sequence for setting up the DMA and coherent DMA masks so
>>>> that during initialization the driver requests a 64 bit DMA mask and a 32 bit consistent
>>>> DMA mask, then later requests a 64 bit consistent DMA mask. The Power architecture does
>>>> not currently support this, which results in always falling back to a 32 bit DMA window,
>>>> which has a negative impact on performance. Tweak this algorithm slightly so that
>>>> if requesting a 32 bit consistent mask fails after we've successfully set a 64 bit
>>>> DMA mask, just try to get a 64 bit consistent mask. This should preserve existing
>>>> behavior on platforms that support mixed mask setting and restore previous functionality
>>>> to those that do not.
>>>>
>>>> Signed-off-by: Brian King <brking@linux.vnet.ibm.com>
>>>
>>> I believe the way the API is designed, it should guarantee that after dma_set_mask()
>>> succeeds for a device, dma_set_coherent_mask() with the same mask will also succeed.
>>>
>>> Could you just call dma_set_mask_and_coherent() here to avoid that complex logic?
>>
>> I don't think that would work. The mpt2sas driver wants to set the dma mask to 64bits
>> but set the coherent mask to 32 bits, then my change is to set the coherent mask to
>> 64bits if setting it to 32bit fails. I'm not seeing how using dma_set_mask_and_coherent
>> would simplify anything here.
> 
> My question was more about why the driver would ask for a 32-bit coherent
> mask to start with. Is this a limitation in the mpt2sas device that can
> only have descriptors at low addresses, or is it trying to work around some
> bug in a particular host system?

I'll need help from Sreekanth in answering that. All that is in the git commit log is:

    2. Initially set the consistent DMA mask to 32 bit and then change it to 64 bit mask
    after allocating RDPQ pools by calling the function _base_change_consistent_dma_mask.
    This is to ensure that all the upper 32 bits of RDPQ entries's base address to be same.

So, I'm not sure my patch won't break something with mpt2sas... This commit also changed
how the RDPO pools are allocated, so its possible this won't work. However, I did load
it on Power box and I'm not seeing any major problems so far. I am seeing the following message
get logged on a regular basis, not sure if its related to this change or not:

mpt2sas3: log_info(0x31120303): originator(IOP), code(0x03), sub_code(0x3112)

-Brian


-- 
Brian King
Power Linux I/O
IBM Linux Technology Center

Re: [PATCH] mpt2sas: Fall back to 64 bit coherent mask if 64 bit DMA / 32 bit coherent mask not supported

[Brian King]

On 05/13/2015 08:31 AM, Arnd Bergmann wrote:
> On Wednesday 13 May 2015 08:23:41 Brian King wrote:
>> On 05/13/2015 03:10 AM, Arnd Bergmann wrote:
>>> On Tuesday 12 May 2015 18:24:43 Brian King wrote:
>>>>
>>>> Commit 5fb1bf8aaa832e1e9ca3198de7bbecb8eff7db9c broke 64 bit DMA for mpt2sas on Power.
>>>> That commit changed the sequence for setting up the DMA and coherent DMA masks so
>>>> that during initialization the driver requests a 64 bit DMA mask and a 32 bit consistent
>>>> DMA mask, then later requests a 64 bit consistent DMA mask. The Power architecture does
>>>> not currently support this, which results in always falling back to a 32 bit DMA window,
>>>> which has a negative impact on performance. Tweak this algorithm slightly so that
>>>> if requesting a 32 bit consistent mask fails after we've successfully set a 64 bit
>>>> DMA mask, just try to get a 64 bit consistent mask. This should preserve existing
>>>> behavior on platforms that support mixed mask setting and restore previous functionality
>>>> to those that do not.
>>>>
>>>> Signed-off-by: Brian King <brking@linux.vnet.ibm.com>
>>>
>>> I believe the way the API is designed, it should guarantee that after dma_set_mask()
>>> succeeds for a device, dma_set_coherent_mask() with the same mask will also succeed.
>>>
>>> Could you just call dma_set_mask_and_coherent() here to avoid that complex logic?
>>
>> I don't think that would work. The mpt2sas driver wants to set the dma mask to 64bits
>> but set the coherent mask to 32 bits, then my change is to set the coherent mask to
>> 64bits if setting it to 32bit fails. I'm not seeing how using dma_set_mask_and_coherent
>> would simplify anything here.
> 
> My question was more about why the driver would ask for a 32-bit coherent
> mask to start with. Is this a limitation in the mpt2sas device that can
> only have descriptors at low addresses, or is it trying to work around some
> bug in a particular host system?

I'll need help from Sreekanth in answering that. All that is in the git commit log is:

    2. Initially set the consistent DMA mask to 32 bit and then change it to 64 bit mask
    after allocating RDPQ pools by calling the function _base_change_consistent_dma_mask.
    This is to ensure that all the upper 32 bits of RDPQ entries's base address to be same.

So, I'm not sure my patch won't break something with mpt2sas... This commit also changed
how the RDPO pools are allocated, so its possible this won't work. However, I did load
it on Power box and I'm not seeing any major problems so far. I am seeing the following message
get logged on a regular basis, not sure if its related to this change or not:

mpt2sas3: log_info(0x31120303): originator(IOP), code(0x03), sub_code(0x3112)

-Brian


-- 
Brian King
Power Linux I/O
IBM Linux Technology Center

Re: [PATCH] mpt2sas: Fall back to 64 bit coherent mask if 64 bit DMA / 32 bit coherent mask not supported

[Sreekanth Reddy]

Hi Brain,

We had a restriction from the firmware that the upper 32 bits of the
RDPQ pool entries must be same. So to limit this restriction we
initially set the consistent DMA mask to 32 and then after allocating
the RDPQ pools we changed the consistent DMA mask to 64 before
allocating remaining pools.

Regards,
Sreekanth

On Wed, May 13, 2015 at 7:42 PM, Brian King <brking@linux.vnet.ibm.com> wrote:
> On 05/13/2015 08:31 AM, Arnd Bergmann wrote:
>> On Wednesday 13 May 2015 08:23:41 Brian King wrote:
>>> On 05/13/2015 03:10 AM, Arnd Bergmann wrote:
>>>> On Tuesday 12 May 2015 18:24:43 Brian King wrote:
>>>>>
>>>>> Commit 5fb1bf8aaa832e1e9ca3198de7bbecb8eff7db9c broke 64 bit DMA for mpt2sas on Power.
>>>>> That commit changed the sequence for setting up the DMA and coherent DMA masks so
>>>>> that during initialization the driver requests a 64 bit DMA mask and a 32 bit consistent
>>>>> DMA mask, then later requests a 64 bit consistent DMA mask. The Power architecture does
>>>>> not currently support this, which results in always falling back to a 32 bit DMA window,
>>>>> which has a negative impact on performance. Tweak this algorithm slightly so that
>>>>> if requesting a 32 bit consistent mask fails after we've successfully set a 64 bit
>>>>> DMA mask, just try to get a 64 bit consistent mask. This should preserve existing
>>>>> behavior on platforms that support mixed mask setting and restore previous functionality
>>>>> to those that do not.
>>>>>
>>>>> Signed-off-by: Brian King <brking@linux.vnet.ibm.com>
>>>>
>>>> I believe the way the API is designed, it should guarantee that after dma_set_mask()
>>>> succeeds for a device, dma_set_coherent_mask() with the same mask will also succeed.
>>>>
>>>> Could you just call dma_set_mask_and_coherent() here to avoid that complex logic?
>>>
>>> I don't think that would work. The mpt2sas driver wants to set the dma mask to 64bits
>>> but set the coherent mask to 32 bits, then my change is to set the coherent mask to
>>> 64bits if setting it to 32bit fails. I'm not seeing how using dma_set_mask_and_coherent
>>> would simplify anything here.
>>
>> My question was more about why the driver would ask for a 32-bit coherent
>> mask to start with. Is this a limitation in the mpt2sas device that can
>> only have descriptors at low addresses, or is it trying to work around some
>> bug in a particular host system?
>
> I'll need help from Sreekanth in answering that. All that is in the git commit log is:
>
>     2. Initially set the consistent DMA mask to 32 bit and then change it to 64 bit mask
>     after allocating RDPQ pools by calling the function _base_change_consistent_dma_mask.
>     This is to ensure that all the upper 32 bits of RDPQ entries's base address to be same.
>
> So, I'm not sure my patch won't break something with mpt2sas... This commit also changed
> how the RDPO pools are allocated, so its possible this won't work. However, I did load
> it on Power box and I'm not seeing any major problems so far. I am seeing the following message
> get logged on a regular basis, not sure if its related to this change or not:
>
> mpt2sas3: log_info(0x31120303): originator(IOP), code(0x03), sub_code(0x3112)
>
> -Brian
>
>
> --
> Brian King
> Power Linux I/O
> IBM Linux Technology Center
>
>

Re: [PATCH] mpt2sas: Fall back to 64 bit coherent mask if 64 bit DMA / 32 bit coherent mask not supported

[Sreekanth Reddy]

Hi Brain,

We had a restriction from the firmware that the upper 32 bits of the
RDPQ pool entries must be same. So to limit this restriction we
initially set the consistent DMA mask to 32 and then after allocating
the RDPQ pools we changed the consistent DMA mask to 64 before
allocating remaining pools.

Regards,
Sreekanth

On Wed, May 13, 2015 at 7:42 PM, Brian King <brking@linux.vnet.ibm.com> wrote:
> On 05/13/2015 08:31 AM, Arnd Bergmann wrote:
>> On Wednesday 13 May 2015 08:23:41 Brian King wrote:
>>> On 05/13/2015 03:10 AM, Arnd Bergmann wrote:
>>>> On Tuesday 12 May 2015 18:24:43 Brian King wrote:
>>>>>
>>>>> Commit 5fb1bf8aaa832e1e9ca3198de7bbecb8eff7db9c broke 64 bit DMA for mpt2sas on Power.
>>>>> That commit changed the sequence for setting up the DMA and coherent DMA masks so
>>>>> that during initialization the driver requests a 64 bit DMA mask and a 32 bit consistent
>>>>> DMA mask, then later requests a 64 bit consistent DMA mask. The Power architecture does
>>>>> not currently support this, which results in always falling back to a 32 bit DMA window,
>>>>> which has a negative impact on performance. Tweak this algorithm slightly so that
>>>>> if requesting a 32 bit consistent mask fails after we've successfully set a 64 bit
>>>>> DMA mask, just try to get a 64 bit consistent mask. This should preserve existing
>>>>> behavior on platforms that support mixed mask setting and restore previous functionality
>>>>> to those that do not.
>>>>>
>>>>> Signed-off-by: Brian King <brking@linux.vnet.ibm.com>
>>>>
>>>> I believe the way the API is designed, it should guarantee that after dma_set_mask()
>>>> succeeds for a device, dma_set_coherent_mask() with the same mask will also succeed.
>>>>
>>>> Could you just call dma_set_mask_and_coherent() here to avoid that complex logic?
>>>
>>> I don't think that would work. The mpt2sas driver wants to set the dma mask to 64bits
>>> but set the coherent mask to 32 bits, then my change is to set the coherent mask to
>>> 64bits if setting it to 32bit fails. I'm not seeing how using dma_set_mask_and_coherent
>>> would simplify anything here.
>>
>> My question was more about why the driver would ask for a 32-bit coherent
>> mask to start with. Is this a limitation in the mpt2sas device that can
>> only have descriptors at low addresses, or is it trying to work around some
>> bug in a particular host system?
>
> I'll need help from Sreekanth in answering that. All that is in the git commit log is:
>
>     2. Initially set the consistent DMA mask to 32 bit and then change it to 64 bit mask
>     after allocating RDPQ pools by calling the function _base_change_consistent_dma_mask.
>     This is to ensure that all the upper 32 bits of RDPQ entries's base address to be same.
>
> So, I'm not sure my patch won't break something with mpt2sas... This commit also changed
> how the RDPO pools are allocated, so its possible this won't work. However, I did load
> it on Power box and I'm not seeing any major problems so far. I am seeing the following message
> get logged on a regular basis, not sure if its related to this change or not:
>
> mpt2sas3: log_info(0x31120303): originator(IOP), code(0x03), sub_code(0x3112)
>
> -Brian
>
>
> --
> Brian King
> Power Linux I/O
> IBM Linux Technology Center
>
>

Re: [PATCH] mpt2sas: Fall back to 64 bit coherent mask if 64 bit DMA / 32 bit coherent mask not supported

[Benjamin Herrenschmidt]

On Wed, 2015-05-13 at 22:14 +0530, Sreekanth Reddy wrote:
> Hi Brain,
> 
> We had a restriction from the firmware that the upper 32 bits of the
> RDPQ pool entries must be same. So to limit this restriction we
> initially set the consistent DMA mask to 32 and then after allocating
> the RDPQ pools we changed the consistent DMA mask to 64 before
> allocating remaining pools.

Brian, maybe we should just bite that bullet and implement that hybrid
DMA ops I mentioned. I'll see if I can spare some time today to look
at it. It would work as long as we never remove the legacy window using
the DDW APIs (removing the legacy window is broken anyway for other
reasons I think we discussed already).

As long as we have both the legacy window and the DDW available (or the
legacy and bypass on "nv"), we can then route individual DMAs according
to the corresponding applicable mask.

I'll try to come up with a patch but I'll need you to test it.

Cheers,
Ben.

> Regards,
> Sreekanth
> 
> On Wed, May 13, 2015 at 7:42 PM, Brian King <brking@linux.vnet.ibm.com> wrote:
> > On 05/13/2015 08:31 AM, Arnd Bergmann wrote:
> >> On Wednesday 13 May 2015 08:23:41 Brian King wrote:
> >>> On 05/13/2015 03:10 AM, Arnd Bergmann wrote:
> >>>> On Tuesday 12 May 2015 18:24:43 Brian King wrote:
> >>>>>
> >>>>> Commit 5fb1bf8aaa832e1e9ca3198de7bbecb8eff7db9c broke 64 bit DMA for mpt2sas on Power.
> >>>>> That commit changed the sequence for setting up the DMA and coherent DMA masks so
> >>>>> that during initialization the driver requests a 64 bit DMA mask and a 32 bit consistent
> >>>>> DMA mask, then later requests a 64 bit consistent DMA mask. The Power architecture does
> >>>>> not currently support this, which results in always falling back to a 32 bit DMA window,
> >>>>> which has a negative impact on performance. Tweak this algorithm slightly so that
> >>>>> if requesting a 32 bit consistent mask fails after we've successfully set a 64 bit
> >>>>> DMA mask, just try to get a 64 bit consistent mask. This should preserve existing
> >>>>> behavior on platforms that support mixed mask setting and restore previous functionality
> >>>>> to those that do not.
> >>>>>
> >>>>> Signed-off-by: Brian King <brking@linux.vnet.ibm.com>
> >>>>
> >>>> I believe the way the API is designed, it should guarantee that after dma_set_mask()
> >>>> succeeds for a device, dma_set_coherent_mask() with the same mask will also succeed.
> >>>>
> >>>> Could you just call dma_set_mask_and_coherent() here to avoid that complex logic?
> >>>
> >>> I don't think that would work. The mpt2sas driver wants to set the dma mask to 64bits
> >>> but set the coherent mask to 32 bits, then my change is to set the coherent mask to
> >>> 64bits if setting it to 32bit fails. I'm not seeing how using dma_set_mask_and_coherent
> >>> would simplify anything here.
> >>
> >> My question was more about why the driver would ask for a 32-bit coherent
> >> mask to start with. Is this a limitation in the mpt2sas device that can
> >> only have descriptors at low addresses, or is it trying to work around some
> >> bug in a particular host system?
> >
> > I'll need help from Sreekanth in answering that. All that is in the git commit log is:
> >
> >     2. Initially set the consistent DMA mask to 32 bit and then change it to 64 bit mask
> >     after allocating RDPQ pools by calling the function _base_change_consistent_dma_mask.
> >     This is to ensure that all the upper 32 bits of RDPQ entries's base address to be same.
> >
> > So, I'm not sure my patch won't break something with mpt2sas... This commit also changed
> > how the RDPO pools are allocated, so its possible this won't work. However, I did load
> > it on Power box and I'm not seeing any major problems so far. I am seeing the following message
> > get logged on a regular basis, not sure if its related to this change or not:
> >
> > mpt2sas3: log_info(0x31120303): originator(IOP), code(0x03), sub_code(0x3112)
> >
> > -Brian
> >
> >
> > --
> > Brian King
> > Power Linux I/O
> > IBM Linux Technology Center
> >
> >

Re: [PATCH] mpt2sas: Fall back to 64 bit coherent mask if 64 bit DMA / 32 bit coherent mask not supported

[Benjamin Herrenschmidt]

On Wed, 2015-05-13 at 22:14 +0530, Sreekanth Reddy wrote:
> Hi Brain,
> 
> We had a restriction from the firmware that the upper 32 bits of the
> RDPQ pool entries must be same. So to limit this restriction we
> initially set the consistent DMA mask to 32 and then after allocating
> the RDPQ pools we changed the consistent DMA mask to 64 before
> allocating remaining pools.

Brian, maybe we should just bite that bullet and implement that hybrid
DMA ops I mentioned. I'll see if I can spare some time today to look
at it. It would work as long as we never remove the legacy window using
the DDW APIs (removing the legacy window is broken anyway for other
reasons I think we discussed already).

As long as we have both the legacy window and the DDW available (or the
legacy and bypass on "nv"), we can then route individual DMAs according
to the corresponding applicable mask.

I'll try to come up with a patch but I'll need you to test it.

Cheers,
Ben.

> Regards,
> Sreekanth
> 
> On Wed, May 13, 2015 at 7:42 PM, Brian King <brking@linux.vnet.ibm.com> wrote:
> > On 05/13/2015 08:31 AM, Arnd Bergmann wrote:
> >> On Wednesday 13 May 2015 08:23:41 Brian King wrote:
> >>> On 05/13/2015 03:10 AM, Arnd Bergmann wrote:
> >>>> On Tuesday 12 May 2015 18:24:43 Brian King wrote:
> >>>>>
> >>>>> Commit 5fb1bf8aaa832e1e9ca3198de7bbecb8eff7db9c broke 64 bit DMA for mpt2sas on Power.
> >>>>> That commit changed the sequence for setting up the DMA and coherent DMA masks so
> >>>>> that during initialization the driver requests a 64 bit DMA mask and a 32 bit consistent
> >>>>> DMA mask, then later requests a 64 bit consistent DMA mask. The Power architecture does
> >>>>> not currently support this, which results in always falling back to a 32 bit DMA window,
> >>>>> which has a negative impact on performance. Tweak this algorithm slightly so that
> >>>>> if requesting a 32 bit consistent mask fails after we've successfully set a 64 bit
> >>>>> DMA mask, just try to get a 64 bit consistent mask. This should preserve existing
> >>>>> behavior on platforms that support mixed mask setting and restore previous functionality
> >>>>> to those that do not.
> >>>>>
> >>>>> Signed-off-by: Brian King <brking@linux.vnet.ibm.com>
> >>>>
> >>>> I believe the way the API is designed, it should guarantee that after dma_set_mask()
> >>>> succeeds for a device, dma_set_coherent_mask() with the same mask will also succeed.
> >>>>
> >>>> Could you just call dma_set_mask_and_coherent() here to avoid that complex logic?
> >>>
> >>> I don't think that would work. The mpt2sas driver wants to set the dma mask to 64bits
> >>> but set the coherent mask to 32 bits, then my change is to set the coherent mask to
> >>> 64bits if setting it to 32bit fails. I'm not seeing how using dma_set_mask_and_coherent
> >>> would simplify anything here.
> >>
> >> My question was more about why the driver would ask for a 32-bit coherent
> >> mask to start with. Is this a limitation in the mpt2sas device that can
> >> only have descriptors at low addresses, or is it trying to work around some
> >> bug in a particular host system?
> >
> > I'll need help from Sreekanth in answering that. All that is in the git commit log is:
> >
> >     2. Initially set the consistent DMA mask to 32 bit and then change it to 64 bit mask
> >     after allocating RDPQ pools by calling the function _base_change_consistent_dma_mask.
> >     This is to ensure that all the upper 32 bits of RDPQ entries's base address to be same.
> >
> > So, I'm not sure my patch won't break something with mpt2sas... This commit also changed
> > how the RDPO pools are allocated, so its possible this won't work. However, I did load
> > it on Power box and I'm not seeing any major problems so far. I am seeing the following message
> > get logged on a regular basis, not sure if its related to this change or not:
> >
> > mpt2sas3: log_info(0x31120303): originator(IOP), code(0x03), sub_code(0x3112)
> >
> > -Brian
> >
> >
> > --
> > Brian King
> > Power Linux I/O
> > IBM Linux Technology Center
> >
> >

Re: [PATCH] mpt2sas: Fall back to 64 bit coherent mask if 64 bit DMA / 32 bit coherent mask not supported

[Brian King]

On 05/13/2015 03:56 PM, Benjamin Herrenschmidt wrote:
> On Wed, 2015-05-13 at 22:14 +0530, Sreekanth Reddy wrote:
>> Hi Brain,
>>
>> We had a restriction from the firmware that the upper 32 bits of the
>> RDPQ pool entries must be same. So to limit this restriction we
>> initially set the consistent DMA mask to 32 and then after allocating
>> the RDPQ pools we changed the consistent DMA mask to 64 before
>> allocating remaining pools.
> 
> Brian, maybe we should just bite that bullet and implement that hybrid
> DMA ops I mentioned. I'll see if I can spare some time today to look
> at it. It would work as long as we never remove the legacy window using
> the DDW APIs (removing the legacy window is broken anyway for other
> reasons I think we discussed already).
> 
> As long as we have both the legacy window and the DDW available (or the
> legacy and bypass on "nv"), we can then route individual DMAs according
> to the corresponding applicable mask.
> 
> I'll try to come up with a patch but I'll need you to test it.

Sure. I can help with that.

Thanks,

Brian

> 
> Cheers,
> Ben.
> 
>> Regards,
>> Sreekanth
>>
>> On Wed, May 13, 2015 at 7:42 PM, Brian King <brking@linux.vnet.ibm.com> wrote:
>>> On 05/13/2015 08:31 AM, Arnd Bergmann wrote:
>>>> On Wednesday 13 May 2015 08:23:41 Brian King wrote:
>>>>> On 05/13/2015 03:10 AM, Arnd Bergmann wrote:
>>>>>> On Tuesday 12 May 2015 18:24:43 Brian King wrote:
>>>>>>>
>>>>>>> Commit 5fb1bf8aaa832e1e9ca3198de7bbecb8eff7db9c broke 64 bit DMA for mpt2sas on Power.
>>>>>>> That commit changed the sequence for setting up the DMA and coherent DMA masks so
>>>>>>> that during initialization the driver requests a 64 bit DMA mask and a 32 bit consistent
>>>>>>> DMA mask, then later requests a 64 bit consistent DMA mask. The Power architecture does
>>>>>>> not currently support this, which results in always falling back to a 32 bit DMA window,
>>>>>>> which has a negative impact on performance. Tweak this algorithm slightly so that
>>>>>>> if requesting a 32 bit consistent mask fails after we've successfully set a 64 bit
>>>>>>> DMA mask, just try to get a 64 bit consistent mask. This should preserve existing
>>>>>>> behavior on platforms that support mixed mask setting and restore previous functionality
>>>>>>> to those that do not.
>>>>>>>
>>>>>>> Signed-off-by: Brian King <brking@linux.vnet.ibm.com>
>>>>>>
>>>>>> I believe the way the API is designed, it should guarantee that after dma_set_mask()
>>>>>> succeeds for a device, dma_set_coherent_mask() with the same mask will also succeed.
>>>>>>
>>>>>> Could you just call dma_set_mask_and_coherent() here to avoid that complex logic?
>>>>>
>>>>> I don't think that would work. The mpt2sas driver wants to set the dma mask to 64bits
>>>>> but set the coherent mask to 32 bits, then my change is to set the coherent mask to
>>>>> 64bits if setting it to 32bit fails. I'm not seeing how using dma_set_mask_and_coherent
>>>>> would simplify anything here.
>>>>
>>>> My question was more about why the driver would ask for a 32-bit coherent
>>>> mask to start with. Is this a limitation in the mpt2sas device that can
>>>> only have descriptors at low addresses, or is it trying to work around some
>>>> bug in a particular host system?
>>>
>>> I'll need help from Sreekanth in answering that. All that is in the git commit log is:
>>>
>>>     2. Initially set the consistent DMA mask to 32 bit and then change it to 64 bit mask
>>>     after allocating RDPQ pools by calling the function _base_change_consistent_dma_mask.
>>>     This is to ensure that all the upper 32 bits of RDPQ entries's base address to be same.
>>>
>>> So, I'm not sure my patch won't break something with mpt2sas... This commit also changed
>>> how the RDPO pools are allocated, so its possible this won't work. However, I did load
>>> it on Power box and I'm not seeing any major problems so far. I am seeing the following message
>>> get logged on a regular basis, not sure if its related to this change or not:
>>>
>>> mpt2sas3: log_info(0x31120303): originator(IOP), code(0x03), sub_code(0x3112)
>>>
>>> -Brian
>>>
>>>
>>> --
>>> Brian King
>>> Power Linux I/O
>>> IBM Linux Technology Center
>>>
>>>
> 
> 


-- 
Brian King
Power Linux I/O
IBM Linux Technology Center

Re: [PATCH] mpt2sas: Fall back to 64 bit coherent mask if 64 bit DMA / 32 bit coherent mask not supported

[Brian King]

On 05/13/2015 03:56 PM, Benjamin Herrenschmidt wrote:
> On Wed, 2015-05-13 at 22:14 +0530, Sreekanth Reddy wrote:
>> Hi Brain,
>>
>> We had a restriction from the firmware that the upper 32 bits of the
>> RDPQ pool entries must be same. So to limit this restriction we
>> initially set the consistent DMA mask to 32 and then after allocating
>> the RDPQ pools we changed the consistent DMA mask to 64 before
>> allocating remaining pools.
> 
> Brian, maybe we should just bite that bullet and implement that hybrid
> DMA ops I mentioned. I'll see if I can spare some time today to look
> at it. It would work as long as we never remove the legacy window using
> the DDW APIs (removing the legacy window is broken anyway for other
> reasons I think we discussed already).
> 
> As long as we have both the legacy window and the DDW available (or the
> legacy and bypass on "nv"), we can then route individual DMAs according
> to the corresponding applicable mask.
> 
> I'll try to come up with a patch but I'll need you to test it.

Sure. I can help with that.

Thanks,

Brian

> 
> Cheers,
> Ben.
> 
>> Regards,
>> Sreekanth
>>
>> On Wed, May 13, 2015 at 7:42 PM, Brian King <brking@linux.vnet.ibm.com> wrote:
>>> On 05/13/2015 08:31 AM, Arnd Bergmann wrote:
>>>> On Wednesday 13 May 2015 08:23:41 Brian King wrote:
>>>>> On 05/13/2015 03:10 AM, Arnd Bergmann wrote:
>>>>>> On Tuesday 12 May 2015 18:24:43 Brian King wrote:
>>>>>>>
>>>>>>> Commit 5fb1bf8aaa832e1e9ca3198de7bbecb8eff7db9c broke 64 bit DMA for mpt2sas on Power.
>>>>>>> That commit changed the sequence for setting up the DMA and coherent DMA masks so
>>>>>>> that during initialization the driver requests a 64 bit DMA mask and a 32 bit consistent
>>>>>>> DMA mask, then later requests a 64 bit consistent DMA mask. The Power architecture does
>>>>>>> not currently support this, which results in always falling back to a 32 bit DMA window,
>>>>>>> which has a negative impact on performance. Tweak this algorithm slightly so that
>>>>>>> if requesting a 32 bit consistent mask fails after we've successfully set a 64 bit
>>>>>>> DMA mask, just try to get a 64 bit consistent mask. This should preserve existing
>>>>>>> behavior on platforms that support mixed mask setting and restore previous functionality
>>>>>>> to those that do not.
>>>>>>>
>>>>>>> Signed-off-by: Brian King <brking@linux.vnet.ibm.com>
>>>>>>
>>>>>> I believe the way the API is designed, it should guarantee that after dma_set_mask()
>>>>>> succeeds for a device, dma_set_coherent_mask() with the same mask will also succeed.
>>>>>>
>>>>>> Could you just call dma_set_mask_and_coherent() here to avoid that complex logic?
>>>>>
>>>>> I don't think that would work. The mpt2sas driver wants to set the dma mask to 64bits
>>>>> but set the coherent mask to 32 bits, then my change is to set the coherent mask to
>>>>> 64bits if setting it to 32bit fails. I'm not seeing how using dma_set_mask_and_coherent
>>>>> would simplify anything here.
>>>>
>>>> My question was more about why the driver would ask for a 32-bit coherent
>>>> mask to start with. Is this a limitation in the mpt2sas device that can
>>>> only have descriptors at low addresses, or is it trying to work around some
>>>> bug in a particular host system?
>>>
>>> I'll need help from Sreekanth in answering that. All that is in the git commit log is:
>>>
>>>     2. Initially set the consistent DMA mask to 32 bit and then change it to 64 bit mask
>>>     after allocating RDPQ pools by calling the function _base_change_consistent_dma_mask.
>>>     This is to ensure that all the upper 32 bits of RDPQ entries's base address to be same.
>>>
>>> So, I'm not sure my patch won't break something with mpt2sas... This commit also changed
>>> how the RDPO pools are allocated, so its possible this won't work. However, I did load
>>> it on Power box and I'm not seeing any major problems so far. I am seeing the following message
>>> get logged on a regular basis, not sure if its related to this change or not:
>>>
>>> mpt2sas3: log_info(0x31120303): originator(IOP), code(0x03), sub_code(0x3112)
>>>
>>> -Brian
>>>
>>>
>>> --
>>> Brian King
>>> Power Linux I/O
>>> IBM Linux Technology Center
>>>
>>>
> 
> 


-- 
Brian King
Power Linux I/O
IBM Linux Technology Center

[RFC/PATCH] powerpc/iommu: Support "hybrid" iommu/direct DMA ops for coherent_mask < dma_mask

[Benjamin Herrenschmidt]

This patch adds the ability to the DMA direct ops to fallback to the IOMMU
ops for coherent alloc/free if the coherent mask of the device isn't
suitable for accessing the direct DMA space and the device also happens
to have an active IOMMU table.

Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
---

diff --git a/arch/powerpc/include/asm/device.h b/arch/powerpc/include/asm/device.h
index 9f1371b..9ce2199 100644
--- a/arch/powerpc/include/asm/device.h
+++ b/arch/powerpc/include/asm/device.h
@@ -10,6 +10,7 @@ struct dma_map_ops;
 struct device_node;
 #ifdef CONFIG_PPC64
 struct pci_dn;
+struct iommu_table;
 #endif
 
 /*
@@ -23,13 +24,15 @@ struct dev_archdata {
 	struct dma_map_ops	*dma_ops;
 
 	/*
-	 * When an iommu is in use, dma_data is used as a ptr to the base of the
-	 * iommu_table.  Otherwise, it is a simple numerical offset.
+	 * These two used to be a union. However, with the hybrid ops we need
+	 * both so here we store both a DMA offset for direct mappings and
+	 * an iommu_table for remapped DMA.
 	 */
-	union {
-		dma_addr_t	dma_offset;
-		void		*iommu_table_base;
-	} dma_data;
+	dma_addr_t		dma_offset;
+
+#ifdef CONFIG_PPC64
+	struct iommu_table	*iommu_table_base;
+#endif
 
 #ifdef CONFIG_IOMMU_API
 	void			*iommu_domain;
diff --git a/arch/powerpc/include/asm/dma-mapping.h b/arch/powerpc/include/asm/dma-mapping.h
index 894d538..2e12366 100644
--- a/arch/powerpc/include/asm/dma-mapping.h
+++ b/arch/powerpc/include/asm/dma-mapping.h
@@ -21,12 +21,12 @@
 #define DMA_ERROR_CODE		(~(dma_addr_t)0x0)
 
 /* Some dma direct funcs must be visible for use in other dma_ops */
-extern void *dma_direct_alloc_coherent(struct device *dev, size_t size,
-				       dma_addr_t *dma_handle, gfp_t flag,
+extern void *__dma_direct_alloc_coherent(struct device *dev, size_t size,
+					 dma_addr_t *dma_handle, gfp_t flag,
+					 struct dma_attrs *attrs);
+extern void __dma_direct_free_coherent(struct device *dev, size_t size,
+				       void *vaddr, dma_addr_t dma_handle,
 				       struct dma_attrs *attrs);
-extern void dma_direct_free_coherent(struct device *dev, size_t size,
-				     void *vaddr, dma_addr_t dma_handle,
-				     struct dma_attrs *attrs);
 extern int dma_direct_mmap_coherent(struct device *dev,
 				    struct vm_area_struct *vma,
 				    void *cpu_addr, dma_addr_t handle,
@@ -106,7 +106,7 @@ static inline void set_dma_ops(struct device *dev, struct dma_map_ops *ops)
 static inline dma_addr_t get_dma_offset(struct device *dev)
 {
 	if (dev)
-		return dev->archdata.dma_data.dma_offset;
+		return dev->archdata.dma_offset;
 
 	return PCI_DRAM_OFFSET;
 }
@@ -114,7 +114,7 @@ static inline dma_addr_t get_dma_offset(struct device *dev)
 static inline void set_dma_offset(struct device *dev, dma_addr_t off)
 {
 	if (dev)
-		dev->archdata.dma_data.dma_offset = off;
+		dev->archdata.dma_offset = off;
 }
 
 /* this will be removed soon */
diff --git a/arch/powerpc/include/asm/iommu.h b/arch/powerpc/include/asm/iommu.h
index e2abbe8..157cfdd 100644
--- a/arch/powerpc/include/asm/iommu.h
+++ b/arch/powerpc/include/asm/iommu.h
@@ -2,17 +2,17 @@
  * Copyright (C) 2001 Mike Corrigan & Dave Engebretsen, IBM Corporation
  * Rewrite, cleanup:
  * Copyright (C) 2004 Olof Johansson <olof@lixom.net>, IBM Corporation
- * 
+ *
  * This program is free software; you can redistribute it and/or modify
  * it under the terms of the GNU General Public License as published by
  * the Free Software Foundation; either version 2 of the License, or
  * (at your option) any later version.
- * 
+ *
  * This program is distributed in the hope that it will be useful,
  * but WITHOUT ANY WARRANTY; without even the implied warranty of
  * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
  * GNU General Public License for more details.
- * 
+ *
  * You should have received a copy of the GNU General Public License
  * along with this program; if not, write to the Free Software
  * Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307 USA
@@ -93,14 +93,17 @@ int get_iommu_order(unsigned long size, struct iommu_table *tbl)
 
 struct scatterlist;
 
-static inline void set_iommu_table_base(struct device *dev, void *base)
+#ifdef CONFIG_PPC64
+
+static inline void set_iommu_table_base(struct device *dev,
+					struct iommu_table *base)
 {
-	dev->archdata.dma_data.iommu_table_base = base;
+	dev->archdata.iommu_table_base = base;
 }
 
 static inline void *get_iommu_table_base(struct device *dev)
 {
-	return dev->archdata.dma_data.iommu_table_base;
+	return dev->archdata.iommu_table_base;
 }
 
 /* Frees table for an individual device node */
@@ -146,6 +149,15 @@ static inline void set_iommu_table_base_and_group(struct device *dev,
 	iommu_add_device(dev);
 }
 
+#else
+
+static inline void *get_iommu_table_base(struct device *dev)
+{
+	return NULL;
+}
+
+#endif /* CONFIG_PPC64 */
+
 extern int ppc_iommu_map_sg(struct device *dev, struct iommu_table *tbl,
 			    struct scatterlist *sglist, int nelems,
 			    unsigned long mask,
diff --git a/arch/powerpc/kernel/dma-swiotlb.c b/arch/powerpc/kernel/dma-swiotlb.c
index 7359797..81b81cd 100644
--- a/arch/powerpc/kernel/dma-swiotlb.c
+++ b/arch/powerpc/kernel/dma-swiotlb.c
@@ -47,8 +47,8 @@ static u64 swiotlb_powerpc_get_required(struct device *dev)
  * for everything else.
  */
 struct dma_map_ops swiotlb_dma_ops = {
-	.alloc = dma_direct_alloc_coherent,
-	.free = dma_direct_free_coherent,
+	.alloc = __dma_direct_alloc_coherent,
+	.free = __dma_direct_free_coherent,
 	.mmap = dma_direct_mmap_coherent,
 	.map_sg = swiotlb_map_sg_attrs,
 	.unmap_sg = swiotlb_unmap_sg_attrs,
diff --git a/arch/powerpc/kernel/dma.c b/arch/powerpc/kernel/dma.c
index 484b2d4..08b0724 100644
--- a/arch/powerpc/kernel/dma.c
+++ b/arch/powerpc/kernel/dma.c
@@ -16,6 +16,7 @@
 #include <asm/bug.h>
 #include <asm/machdep.h>
 #include <asm/swiotlb.h>
+#include <asm/iommu.h>
 
 /*
  * Generic direct DMA implementation
@@ -39,9 +40,31 @@ static u64 __maybe_unused get_pfn_limit(struct device *dev)
 	return pfn;
 }
 
-void *dma_direct_alloc_coherent(struct device *dev, size_t size,
-				dma_addr_t *dma_handle, gfp_t flag,
-				struct dma_attrs *attrs)
+static int dma_direct_dma_supported(struct device *dev, u64 mask)
+{
+#ifdef CONFIG_PPC64
+	u64 limit = get_dma_offset(dev) + (memblock_end_of_DRAM() - 1);
+
+	/* Limit fits in the mask, we are good */
+	if (mask >= limit)
+		return 1;
+
+#ifdef CONFIG_FSL_SOC
+	/* Freescale gets another chance via ZONE_DMA/ZONE_DMA32, however
+	 * that will have to be refined if/when they support iommus
+	 */
+	return 1;
+#endif
+	/* Sorry ... */
+	return 0;
+#else
+	return 1;
+#endif
+}
+
+void *__dma_direct_alloc_coherent(struct device *dev, size_t size,
+				  dma_addr_t *dma_handle, gfp_t flag,
+				  struct dma_attrs *attrs)
 {
 	void *ret;
 #ifdef CONFIG_NOT_COHERENT_CACHE
@@ -96,9 +119,9 @@ void *dma_direct_alloc_coherent(struct device *dev, size_t size,
 #endif
 }
 
-void dma_direct_free_coherent(struct device *dev, size_t size,
-			      void *vaddr, dma_addr_t dma_handle,
-			      struct dma_attrs *attrs)
+void __dma_direct_free_coherent(struct device *dev, size_t size,
+				void *vaddr, dma_addr_t dma_handle,
+				struct dma_attrs *attrs)
 {
 #ifdef CONFIG_NOT_COHERENT_CACHE
 	__dma_free_coherent(size, vaddr);
@@ -107,6 +130,51 @@ void dma_direct_free_coherent(struct device *dev, size_t size,
 #endif
 }
 
+static void *dma_direct_alloc_coherent(struct device *dev, size_t size,
+				       dma_addr_t *dma_handle, gfp_t flag,
+				       struct dma_attrs *attrs)
+{
+	struct iommu_table *iommu;
+
+	/* The coherent mask may be smaller than the real mask, check if
+	 * we can really use the direct ops
+	 */
+	if (dma_direct_dma_supported(dev, dev->coherent_dma_mask))
+		return __dma_direct_alloc_coherent(dev, size, dma_handle,
+						   flag, attrs);
+
+	/* Ok we can't ... do we have an iommu ? If not, fail */
+	iommu = get_iommu_table_base(dev);
+	if (!iommu)
+		return NULL;
+
+	/* Try to use the iommu */
+	return iommu_alloc_coherent(dev, iommu, size, dma_handle,
+				    dev->coherent_dma_mask, flag,
+				    dev_to_node(dev));
+}
+
+static void dma_direct_free_coherent(struct device *dev, size_t size,
+				     void *vaddr, dma_addr_t dma_handle,
+				     struct dma_attrs *attrs)
+{
+	struct iommu_table *iommu;
+
+	/* See comments in dma_direct_alloc_coherent() */
+	if (dma_direct_dma_supported(dev, dev->coherent_dma_mask))
+		return __dma_direct_free_coherent(dev, size, vaddr, dma_handle,
+						  attrs);
+	/* Maybe we used an iommu ... */
+	iommu = get_iommu_table_base(dev);
+
+	/* If we hit that we should have never allocated in the first
+	 * place so how come we are freeing ?
+	 */
+	if (WARN_ON(!iommu))
+		return;
+	iommu_free_coherent(iommu, size, vaddr, dma_handle);
+}
+
 int dma_direct_mmap_coherent(struct device *dev, struct vm_area_struct *vma,
 			     void *cpu_addr, dma_addr_t handle, size_t size,
 			     struct dma_attrs *attrs)
@@ -147,18 +215,6 @@ static void dma_direct_unmap_sg(struct device *dev, struct scatterlist *sg,
 {
 }
 
-static int dma_direct_dma_supported(struct device *dev, u64 mask)
-{
-#ifdef CONFIG_PPC64
-	/* Could be improved so platforms can set the limit in case
-	 * they have limited DMA windows
-	 */
-	return mask >= get_dma_offset(dev) + (memblock_end_of_DRAM() - 1);
-#else
-	return 1;
-#endif
-}
-
 static u64 dma_direct_get_required_mask(struct device *dev)
 {
 	u64 end, mask;
diff --git a/arch/powerpc/platforms/powernv/pci-ioda.c b/arch/powerpc/platforms/powernv/pci-ioda.c
index 5ac7c60..fcff6fa 100644
--- a/arch/powerpc/platforms/powernv/pci-ioda.c
+++ b/arch/powerpc/platforms/powernv/pci-ioda.c
@@ -1625,7 +1625,6 @@ static int pnv_pci_ioda_dma_set_mask(struct pnv_phb *phb,
 	} else {
 		dev_info(&pdev->dev, "Using 32-bit DMA via iommu\n");
 		set_dma_ops(&pdev->dev, &dma_iommu_ops);
-		set_iommu_table_base(&pdev->dev, pe->tce32_table);
 	}
 	*pdev->dev.dma_mask = dma_mask;
 	return 0;
diff --git a/arch/powerpc/platforms/pseries/iommu.c b/arch/powerpc/platforms/pseries/iommu.c
index 7803a19..0acab86 100644
--- a/arch/powerpc/platforms/pseries/iommu.c
+++ b/arch/powerpc/platforms/pseries/iommu.c
@@ -1161,11 +1161,10 @@ static int dma_set_mask_pSeriesLP(struct device *dev, u64 dma_mask)
 		}
 	}
 
-	/* fall back on iommu ops, restore table pointer with ops */
+	/* fall back on iommu ops */
 	if (!ddw_enabled && get_dma_ops(dev) != &dma_iommu_ops) {
 		dev_info(dev, "Restoring 32-bit DMA via iommu\n");
 		set_dma_ops(dev, &dma_iommu_ops);
-		pci_dma_dev_setup_pSeriesLP(pdev);
 	}
 
 check_mask:
diff --git a/arch/powerpc/sysdev/dart_iommu.c b/arch/powerpc/sysdev/dart_iommu.c
index 9e5353f..6cf033d 100644
--- a/arch/powerpc/sysdev/dart_iommu.c
+++ b/arch/powerpc/sysdev/dart_iommu.c
@@ -306,20 +306,11 @@ static void iommu_table_dart_setup(void)
 	set_bit(iommu_table_dart.it_size - 1, iommu_table_dart.it_map);
 }
 
-static void dma_dev_setup_dart(struct device *dev)
-{
-	/* We only have one iommu table on the mac for now, which makes
-	 * things simple. Setup all PCI devices to point to this table
-	 */
-	if (get_dma_ops(dev) == &dma_direct_ops)
-		set_dma_offset(dev, DART_U4_BYPASS_BASE);
-	else
-		set_iommu_table_base(dev, &iommu_table_dart);
-}
-
 static void pci_dma_dev_setup_dart(struct pci_dev *dev)
 {
-	dma_dev_setup_dart(&dev->dev);
+	if (dart_is_u4)
+		set_dma_offset(&dev->dev, DART_U4_BYPASS_BASE);
+	set_iommu_table_base(&dev->dev, &iommu_table_dart);
 }
 
 static void pci_dma_bus_setup_dart(struct pci_bus *bus)
@@ -363,7 +354,6 @@ static int dart_dma_set_mask(struct device *dev, u64 dma_mask)
 		dev_info(dev, "Using 32-bit DMA via iommu\n");
 		set_dma_ops(dev, &dma_iommu_ops);
 	}
-	dma_dev_setup_dart(dev);
 
 	*dev->dma_mask = dma_mask;
 	return 0;

[RFC/PATCH] powerpc/iommu: Support "hybrid" iommu/direct DMA ops for coherent_mask < dma_mask

[Benjamin Herrenschmidt]

This patch adds the ability to the DMA direct ops to fallback to the IOMMU
ops for coherent alloc/free if the coherent mask of the device isn't
suitable for accessing the direct DMA space and the device also happens
to have an active IOMMU table.

Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
---

diff --git a/arch/powerpc/include/asm/device.h b/arch/powerpc/include/asm/device.h
index 9f1371b..9ce2199 100644
--- a/arch/powerpc/include/asm/device.h
+++ b/arch/powerpc/include/asm/device.h
@@ -10,6 +10,7 @@ struct dma_map_ops;
 struct device_node;
 #ifdef CONFIG_PPC64
 struct pci_dn;
+struct iommu_table;
 #endif
 
 /*
@@ -23,13 +24,15 @@ struct dev_archdata {
 	struct dma_map_ops	*dma_ops;
 
 	/*
-	 * When an iommu is in use, dma_data is used as a ptr to the base of the
-	 * iommu_table.  Otherwise, it is a simple numerical offset.
+	 * These two used to be a union. However, with the hybrid ops we need
+	 * both so here we store both a DMA offset for direct mappings and
+	 * an iommu_table for remapped DMA.
 	 */
-	union {
-		dma_addr_t	dma_offset;
-		void		*iommu_table_base;
-	} dma_data;
+	dma_addr_t		dma_offset;
+
+#ifdef CONFIG_PPC64
+	struct iommu_table	*iommu_table_base;
+#endif
 
 #ifdef CONFIG_IOMMU_API
 	void			*iommu_domain;
diff --git a/arch/powerpc/include/asm/dma-mapping.h b/arch/powerpc/include/asm/dma-mapping.h
index 894d538..2e12366 100644
--- a/arch/powerpc/include/asm/dma-mapping.h
+++ b/arch/powerpc/include/asm/dma-mapping.h
@@ -21,12 +21,12 @@
 #define DMA_ERROR_CODE		(~(dma_addr_t)0x0)
 
 /* Some dma direct funcs must be visible for use in other dma_ops */
-extern void *dma_direct_alloc_coherent(struct device *dev, size_t size,
-				       dma_addr_t *dma_handle, gfp_t flag,
+extern void *__dma_direct_alloc_coherent(struct device *dev, size_t size,
+					 dma_addr_t *dma_handle, gfp_t flag,
+					 struct dma_attrs *attrs);
+extern void __dma_direct_free_coherent(struct device *dev, size_t size,
+				       void *vaddr, dma_addr_t dma_handle,
 				       struct dma_attrs *attrs);
-extern void dma_direct_free_coherent(struct device *dev, size_t size,
-				     void *vaddr, dma_addr_t dma_handle,
-				     struct dma_attrs *attrs);
 extern int dma_direct_mmap_coherent(struct device *dev,
 				    struct vm_area_struct *vma,
 				    void *cpu_addr, dma_addr_t handle,
@@ -106,7 +106,7 @@ static inline void set_dma_ops(struct device *dev, struct dma_map_ops *ops)
 static inline dma_addr_t get_dma_offset(struct device *dev)
 {
 	if (dev)
-		return dev->archdata.dma_data.dma_offset;
+		return dev->archdata.dma_offset;
 
 	return PCI_DRAM_OFFSET;
 }
@@ -114,7 +114,7 @@ static inline dma_addr_t get_dma_offset(struct device *dev)
 static inline void set_dma_offset(struct device *dev, dma_addr_t off)
 {
 	if (dev)
-		dev->archdata.dma_data.dma_offset = off;
+		dev->archdata.dma_offset = off;
 }
 
 /* this will be removed soon */
diff --git a/arch/powerpc/include/asm/iommu.h b/arch/powerpc/include/asm/iommu.h
index e2abbe8..157cfdd 100644
--- a/arch/powerpc/include/asm/iommu.h
+++ b/arch/powerpc/include/asm/iommu.h
@@ -2,17 +2,17 @@
  * Copyright (C) 2001 Mike Corrigan & Dave Engebretsen, IBM Corporation
  * Rewrite, cleanup:
  * Copyright (C) 2004 Olof Johansson <olof@lixom.net>, IBM Corporation
- * 
+ *
  * This program is free software; you can redistribute it and/or modify
  * it under the terms of the GNU General Public License as published by
  * the Free Software Foundation; either version 2 of the License, or
  * (at your option) any later version.
- * 
+ *
  * This program is distributed in the hope that it will be useful,
  * but WITHOUT ANY WARRANTY; without even the implied warranty of
  * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
  * GNU General Public License for more details.
- * 
+ *
  * You should have received a copy of the GNU General Public License
  * along with this program; if not, write to the Free Software
  * Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307 USA
@@ -93,14 +93,17 @@ int get_iommu_order(unsigned long size, struct iommu_table *tbl)
 
 struct scatterlist;
 
-static inline void set_iommu_table_base(struct device *dev, void *base)
+#ifdef CONFIG_PPC64
+
+static inline void set_iommu_table_base(struct device *dev,
+					struct iommu_table *base)
 {
-	dev->archdata.dma_data.iommu_table_base = base;
+	dev->archdata.iommu_table_base = base;
 }
 
 static inline void *get_iommu_table_base(struct device *dev)
 {
-	return dev->archdata.dma_data.iommu_table_base;
+	return dev->archdata.iommu_table_base;
 }
 
 /* Frees table for an individual device node */
@@ -146,6 +149,15 @@ static inline void set_iommu_table_base_and_group(struct device *dev,
 	iommu_add_device(dev);
 }
 
+#else
+
+static inline void *get_iommu_table_base(struct device *dev)
+{
+	return NULL;
+}
+
+#endif /* CONFIG_PPC64 */
+
 extern int ppc_iommu_map_sg(struct device *dev, struct iommu_table *tbl,
 			    struct scatterlist *sglist, int nelems,
 			    unsigned long mask,
diff --git a/arch/powerpc/kernel/dma-swiotlb.c b/arch/powerpc/kernel/dma-swiotlb.c
index 7359797..81b81cd 100644
--- a/arch/powerpc/kernel/dma-swiotlb.c
+++ b/arch/powerpc/kernel/dma-swiotlb.c
@@ -47,8 +47,8 @@ static u64 swiotlb_powerpc_get_required(struct device *dev)
  * for everything else.
  */
 struct dma_map_ops swiotlb_dma_ops = {
-	.alloc = dma_direct_alloc_coherent,
-	.free = dma_direct_free_coherent,
+	.alloc = __dma_direct_alloc_coherent,
+	.free = __dma_direct_free_coherent,
 	.mmap = dma_direct_mmap_coherent,
 	.map_sg = swiotlb_map_sg_attrs,
 	.unmap_sg = swiotlb_unmap_sg_attrs,
diff --git a/arch/powerpc/kernel/dma.c b/arch/powerpc/kernel/dma.c
index 484b2d4..08b0724 100644
--- a/arch/powerpc/kernel/dma.c
+++ b/arch/powerpc/kernel/dma.c
@@ -16,6 +16,7 @@
 #include <asm/bug.h>
 #include <asm/machdep.h>
 #include <asm/swiotlb.h>
+#include <asm/iommu.h>
 
 /*
  * Generic direct DMA implementation
@@ -39,9 +40,31 @@ static u64 __maybe_unused get_pfn_limit(struct device *dev)
 	return pfn;
 }
 
-void *dma_direct_alloc_coherent(struct device *dev, size_t size,
-				dma_addr_t *dma_handle, gfp_t flag,
-				struct dma_attrs *attrs)
+static int dma_direct_dma_supported(struct device *dev, u64 mask)
+{
+#ifdef CONFIG_PPC64
+	u64 limit = get_dma_offset(dev) + (memblock_end_of_DRAM() - 1);
+
+	/* Limit fits in the mask, we are good */
+	if (mask >= limit)
+		return 1;
+
+#ifdef CONFIG_FSL_SOC
+	/* Freescale gets another chance via ZONE_DMA/ZONE_DMA32, however
+	 * that will have to be refined if/when they support iommus
+	 */
+	return 1;
+#endif
+	/* Sorry ... */
+	return 0;
+#else
+	return 1;
+#endif
+}
+
+void *__dma_direct_alloc_coherent(struct device *dev, size_t size,
+				  dma_addr_t *dma_handle, gfp_t flag,
+				  struct dma_attrs *attrs)
 {
 	void *ret;
 #ifdef CONFIG_NOT_COHERENT_CACHE
@@ -96,9 +119,9 @@ void *dma_direct_alloc_coherent(struct device *dev, size_t size,
 #endif
 }
 
-void dma_direct_free_coherent(struct device *dev, size_t size,
-			      void *vaddr, dma_addr_t dma_handle,
-			      struct dma_attrs *attrs)
+void __dma_direct_free_coherent(struct device *dev, size_t size,
+				void *vaddr, dma_addr_t dma_handle,
+				struct dma_attrs *attrs)
 {
 #ifdef CONFIG_NOT_COHERENT_CACHE
 	__dma_free_coherent(size, vaddr);
@@ -107,6 +130,51 @@ void dma_direct_free_coherent(struct device *dev, size_t size,
 #endif
 }
 
+static void *dma_direct_alloc_coherent(struct device *dev, size_t size,
+				       dma_addr_t *dma_handle, gfp_t flag,
+				       struct dma_attrs *attrs)
+{
+	struct iommu_table *iommu;
+
+	/* The coherent mask may be smaller than the real mask, check if
+	 * we can really use the direct ops
+	 */
+	if (dma_direct_dma_supported(dev, dev->coherent_dma_mask))
+		return __dma_direct_alloc_coherent(dev, size, dma_handle,
+						   flag, attrs);
+
+	/* Ok we can't ... do we have an iommu ? If not, fail */
+	iommu = get_iommu_table_base(dev);
+	if (!iommu)
+		return NULL;
+
+	/* Try to use the iommu */
+	return iommu_alloc_coherent(dev, iommu, size, dma_handle,
+				    dev->coherent_dma_mask, flag,
+				    dev_to_node(dev));
+}
+
+static void dma_direct_free_coherent(struct device *dev, size_t size,
+				     void *vaddr, dma_addr_t dma_handle,
+				     struct dma_attrs *attrs)
+{
+	struct iommu_table *iommu;
+
+	/* See comments in dma_direct_alloc_coherent() */
+	if (dma_direct_dma_supported(dev, dev->coherent_dma_mask))
+		return __dma_direct_free_coherent(dev, size, vaddr, dma_handle,
+						  attrs);
+	/* Maybe we used an iommu ... */
+	iommu = get_iommu_table_base(dev);
+
+	/* If we hit that we should have never allocated in the first
+	 * place so how come we are freeing ?
+	 */
+	if (WARN_ON(!iommu))
+		return;
+	iommu_free_coherent(iommu, size, vaddr, dma_handle);
+}
+
 int dma_direct_mmap_coherent(struct device *dev, struct vm_area_struct *vma,
 			     void *cpu_addr, dma_addr_t handle, size_t size,
 			     struct dma_attrs *attrs)
@@ -147,18 +215,6 @@ static void dma_direct_unmap_sg(struct device *dev, struct scatterlist *sg,
 {
 }
 
-static int dma_direct_dma_supported(struct device *dev, u64 mask)
-{
-#ifdef CONFIG_PPC64
-	/* Could be improved so platforms can set the limit in case
-	 * they have limited DMA windows
-	 */
-	return mask >= get_dma_offset(dev) + (memblock_end_of_DRAM() - 1);
-#else
-	return 1;
-#endif
-}
-
 static u64 dma_direct_get_required_mask(struct device *dev)
 {
 	u64 end, mask;
diff --git a/arch/powerpc/platforms/powernv/pci-ioda.c b/arch/powerpc/platforms/powernv/pci-ioda.c
index 5ac7c60..fcff6fa 100644
--- a/arch/powerpc/platforms/powernv/pci-ioda.c
+++ b/arch/powerpc/platforms/powernv/pci-ioda.c
@@ -1625,7 +1625,6 @@ static int pnv_pci_ioda_dma_set_mask(struct pnv_phb *phb,
 	} else {
 		dev_info(&pdev->dev, "Using 32-bit DMA via iommu\n");
 		set_dma_ops(&pdev->dev, &dma_iommu_ops);
-		set_iommu_table_base(&pdev->dev, pe->tce32_table);
 	}
 	*pdev->dev.dma_mask = dma_mask;
 	return 0;
diff --git a/arch/powerpc/platforms/pseries/iommu.c b/arch/powerpc/platforms/pseries/iommu.c
index 7803a19..0acab86 100644
--- a/arch/powerpc/platforms/pseries/iommu.c
+++ b/arch/powerpc/platforms/pseries/iommu.c
@@ -1161,11 +1161,10 @@ static int dma_set_mask_pSeriesLP(struct device *dev, u64 dma_mask)
 		}
 	}
 
-	/* fall back on iommu ops, restore table pointer with ops */
+	/* fall back on iommu ops */
 	if (!ddw_enabled && get_dma_ops(dev) != &dma_iommu_ops) {
 		dev_info(dev, "Restoring 32-bit DMA via iommu\n");
 		set_dma_ops(dev, &dma_iommu_ops);
-		pci_dma_dev_setup_pSeriesLP(pdev);
 	}
 
 check_mask:
diff --git a/arch/powerpc/sysdev/dart_iommu.c b/arch/powerpc/sysdev/dart_iommu.c
index 9e5353f..6cf033d 100644
--- a/arch/powerpc/sysdev/dart_iommu.c
+++ b/arch/powerpc/sysdev/dart_iommu.c
@@ -306,20 +306,11 @@ static void iommu_table_dart_setup(void)
 	set_bit(iommu_table_dart.it_size - 1, iommu_table_dart.it_map);
 }
 
-static void dma_dev_setup_dart(struct device *dev)
-{
-	/* We only have one iommu table on the mac for now, which makes
-	 * things simple. Setup all PCI devices to point to this table
-	 */
-	if (get_dma_ops(dev) == &dma_direct_ops)
-		set_dma_offset(dev, DART_U4_BYPASS_BASE);
-	else
-		set_iommu_table_base(dev, &iommu_table_dart);
-}
-
 static void pci_dma_dev_setup_dart(struct pci_dev *dev)
 {
-	dma_dev_setup_dart(&dev->dev);
+	if (dart_is_u4)
+		set_dma_offset(&dev->dev, DART_U4_BYPASS_BASE);
+	set_iommu_table_base(&dev->dev, &iommu_table_dart);
 }
 
 static void pci_dma_bus_setup_dart(struct pci_bus *bus)
@@ -363,7 +354,6 @@ static int dart_dma_set_mask(struct device *dev, u64 dma_mask)
 		dev_info(dev, "Using 32-bit DMA via iommu\n");
 		set_dma_ops(dev, &dma_iommu_ops);
 	}
-	dma_dev_setup_dart(dev);
 
 	*dev->dma_mask = dma_mask;
 	return 0;

Re: [RFC/PATCH] powerpc/iommu: Support "hybrid" iommu/direct DMA ops for coherent_mask < dma_mask

[Brian King]

On 05/14/2015 02:43 AM, Benjamin Herrenschmidt wrote:
> This patch adds the ability to the DMA direct ops to fallback to the IOMMU
> ops for coherent alloc/free if the coherent mask of the device isn't
> suitable for accessing the direct DMA space and the device also happens
> to have an active IOMMU table.
> 
> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>

Thanks Ben. This doesn't quite work, though, since pci_set_consistent_dma_mask
ends up calling dma_supported, which returns 0 here for a 32 bit mask if
pci_set_consistent_dma_mask was already called with a 64 bit mask. I worked
around it with the hack below and saw that I was able to get 64 bit DMA
and the card showed up all the drives. We'll need a better a better fix
for upstream obviously, but I wanted to be able to check out the rest of the patch
a bit...

Thanks,

Brian


-- 
Brian King
Power Linux I/O
IBM Linux Technology Center




Signed-off-by: Brian King <brking@linux.vnet.ibm.com>
---

 arch/powerpc/kernel/dma.c |   12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)

diff -puN arch/powerpc/kernel/dma.c~powerpc_iommu_hybrid_fixup arch/powerpc/kernel/dma.c
--- linux/arch/powerpc/kernel/dma.c~powerpc_iommu_hybrid_fixup	2015-05-14 10:28:38.361416590 -0500
+++ linux-bjking1/arch/powerpc/kernel/dma.c	2015-05-14 10:54:39.367292683 -0500
@@ -62,6 +62,16 @@ static int dma_direct_dma_supported(stru
 #endif
 }
 
+static int dma_direct_is_dma_supported(struct device *dev, u64 mask)
+{
+	if (!dma_direct_dma_supported(dev, mask)) {
+		if (mask < 0xfffffff)
+			return 0;
+	}
+
+	return 1;
+}
+
 void *__dma_direct_alloc_coherent(struct device *dev, size_t size,
 				  dma_addr_t *dma_handle, gfp_t flag,
 				  struct dma_attrs *attrs)
@@ -273,7 +283,7 @@ struct dma_map_ops dma_direct_ops = {
 	.mmap				= dma_direct_mmap_coherent,
 	.map_sg				= dma_direct_map_sg,
 	.unmap_sg			= dma_direct_unmap_sg,
-	.dma_supported			= dma_direct_dma_supported,
+	.dma_supported			= dma_direct_is_dma_supported,
 	.map_page			= dma_direct_map_page,
 	.unmap_page			= dma_direct_unmap_page,
 	.get_required_mask		= dma_direct_get_required_mask,
_

Re: [RFC/PATCH] powerpc/iommu: Support "hybrid" iommu/direct DMA ops for coherent_mask < dma_mask

[Brian King]

On 05/14/2015 02:43 AM, Benjamin Herrenschmidt wrote:
> This patch adds the ability to the DMA direct ops to fallback to the IOMMU
> ops for coherent alloc/free if the coherent mask of the device isn't
> suitable for accessing the direct DMA space and the device also happens
> to have an active IOMMU table.
> 
> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>

Thanks Ben. This doesn't quite work, though, since pci_set_consistent_dma_mask
ends up calling dma_supported, which returns 0 here for a 32 bit mask if
pci_set_consistent_dma_mask was already called with a 64 bit mask. I worked
around it with the hack below and saw that I was able to get 64 bit DMA
and the card showed up all the drives. We'll need a better a better fix
for upstream obviously, but I wanted to be able to check out the rest of the patch
a bit...

Thanks,

Brian


-- 
Brian King
Power Linux I/O
IBM Linux Technology Center




Signed-off-by: Brian King <brking@linux.vnet.ibm.com>
---

 arch/powerpc/kernel/dma.c |   12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)

diff -puN arch/powerpc/kernel/dma.c~powerpc_iommu_hybrid_fixup arch/powerpc/kernel/dma.c
--- linux/arch/powerpc/kernel/dma.c~powerpc_iommu_hybrid_fixup	2015-05-14 10:28:38.361416590 -0500
+++ linux-bjking1/arch/powerpc/kernel/dma.c	2015-05-14 10:54:39.367292683 -0500
@@ -62,6 +62,16 @@ static int dma_direct_dma_supported(stru
 #endif
 }
 
+static int dma_direct_is_dma_supported(struct device *dev, u64 mask)
+{
+	if (!dma_direct_dma_supported(dev, mask)) {
+		if (mask < 0xfffffff)
+			return 0;
+	}
+
+	return 1;
+}
+
 void *__dma_direct_alloc_coherent(struct device *dev, size_t size,
 				  dma_addr_t *dma_handle, gfp_t flag,
 				  struct dma_attrs *attrs)
@@ -273,7 +283,7 @@ struct dma_map_ops dma_direct_ops = {
 	.mmap				= dma_direct_mmap_coherent,
 	.map_sg				= dma_direct_map_sg,
 	.unmap_sg			= dma_direct_unmap_sg,
-	.dma_supported			= dma_direct_dma_supported,
+	.dma_supported			= dma_direct_is_dma_supported,
 	.map_page			= dma_direct_map_page,
 	.unmap_page			= dma_direct_unmap_page,
 	.get_required_mask		= dma_direct_get_required_mask,
_

[RFC/PATCH v2] powerpc/iommu: Support "hybrid" iommu/direct DMA ops for coherent_mask < dma_mask

[Benjamin Herrenschmidt]

This patch adds the ability to the DMA direct ops to fallback to the IOMMU
ops for coherent alloc/free if the coherent mask of the device isn't
suitable for accessing the direct DMA space and the device also happens
to have an active IOMMU table.

Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
---

v2: Implement a custom dma_set_coherent_mask() to work around the fact
that dma_supported() will fail otherwise.

diff --git a/arch/powerpc/Kconfig b/arch/powerpc/Kconfig
index 22b0940..cd46474 100644
--- a/arch/powerpc/Kconfig
+++ b/arch/powerpc/Kconfig
@@ -82,6 +82,9 @@ config GENERIC_HWEIGHT
 	bool
 	default y
 
+config ARCH_HAS_DMA_SET_COHERENT_MASK
+        bool
+
 config PPC
 	bool
 	default y
@@ -152,6 +155,7 @@ config PPC
 	select DCACHE_WORD_ACCESS if PPC64 && CPU_LITTLE_ENDIAN
 	select NO_BOOTMEM
 	select HAVE_GENERIC_RCU_GUP
+	select ARCH_HAS_DMA_SET_COHERENT_MASK
 
 config GENERIC_CSUM
 	def_bool CPU_LITTLE_ENDIAN
diff --git a/arch/powerpc/include/asm/device.h b/arch/powerpc/include/asm/device.h
index 9f1371b..9ce2199 100644
--- a/arch/powerpc/include/asm/device.h
+++ b/arch/powerpc/include/asm/device.h
@@ -10,6 +10,7 @@ struct dma_map_ops;
 struct device_node;
 #ifdef CONFIG_PPC64
 struct pci_dn;
+struct iommu_table;
 #endif
 
 /*
@@ -23,13 +24,15 @@ struct dev_archdata {
 	struct dma_map_ops	*dma_ops;
 
 	/*
-	 * When an iommu is in use, dma_data is used as a ptr to the base of the
-	 * iommu_table.  Otherwise, it is a simple numerical offset.
+	 * These two used to be a union. However, with the hybrid ops we need
+	 * both so here we store both a DMA offset for direct mappings and
+	 * an iommu_table for remapped DMA.
 	 */
-	union {
-		dma_addr_t	dma_offset;
-		void		*iommu_table_base;
-	} dma_data;
+	dma_addr_t		dma_offset;
+
+#ifdef CONFIG_PPC64
+	struct iommu_table	*iommu_table_base;
+#endif
 
 #ifdef CONFIG_IOMMU_API
 	void			*iommu_domain;
diff --git a/arch/powerpc/include/asm/dma-mapping.h b/arch/powerpc/include/asm/dma-mapping.h
index 894d538..2e12366 100644
--- a/arch/powerpc/include/asm/dma-mapping.h
+++ b/arch/powerpc/include/asm/dma-mapping.h
@@ -21,12 +21,12 @@
 #define DMA_ERROR_CODE		(~(dma_addr_t)0x0)
 
 /* Some dma direct funcs must be visible for use in other dma_ops */
-extern void *dma_direct_alloc_coherent(struct device *dev, size_t size,
-				       dma_addr_t *dma_handle, gfp_t flag,
+extern void *__dma_direct_alloc_coherent(struct device *dev, size_t size,
+					 dma_addr_t *dma_handle, gfp_t flag,
+					 struct dma_attrs *attrs);
+extern void __dma_direct_free_coherent(struct device *dev, size_t size,
+				       void *vaddr, dma_addr_t dma_handle,
 				       struct dma_attrs *attrs);
-extern void dma_direct_free_coherent(struct device *dev, size_t size,
-				     void *vaddr, dma_addr_t dma_handle,
-				     struct dma_attrs *attrs);
 extern int dma_direct_mmap_coherent(struct device *dev,
 				    struct vm_area_struct *vma,
 				    void *cpu_addr, dma_addr_t handle,
@@ -106,7 +106,7 @@ static inline void set_dma_ops(struct device *dev, struct dma_map_ops *ops)
 static inline dma_addr_t get_dma_offset(struct device *dev)
 {
 	if (dev)
-		return dev->archdata.dma_data.dma_offset;
+		return dev->archdata.dma_offset;
 
 	return PCI_DRAM_OFFSET;
 }
@@ -114,7 +114,7 @@ static inline dma_addr_t get_dma_offset(struct device *dev)
 static inline void set_dma_offset(struct device *dev, dma_addr_t off)
 {
 	if (dev)
-		dev->archdata.dma_data.dma_offset = off;
+		dev->archdata.dma_offset = off;
 }
 
 /* this will be removed soon */
diff --git a/arch/powerpc/include/asm/iommu.h b/arch/powerpc/include/asm/iommu.h
index e2abbe8..e67e678 100644
--- a/arch/powerpc/include/asm/iommu.h
+++ b/arch/powerpc/include/asm/iommu.h
@@ -2,17 +2,17 @@
  * Copyright (C) 2001 Mike Corrigan & Dave Engebretsen, IBM Corporation
  * Rewrite, cleanup:
  * Copyright (C) 2004 Olof Johansson <olof@lixom.net>, IBM Corporation
- * 
+ *
  * This program is free software; you can redistribute it and/or modify
  * it under the terms of the GNU General Public License as published by
  * the Free Software Foundation; either version 2 of the License, or
  * (at your option) any later version.
- * 
+ *
  * This program is distributed in the hope that it will be useful,
  * but WITHOUT ANY WARRANTY; without even the implied warranty of
  * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
  * GNU General Public License for more details.
- * 
+ *
  * You should have received a copy of the GNU General Public License
  * along with this program; if not, write to the Free Software
  * Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307 USA
@@ -93,16 +93,21 @@ int get_iommu_order(unsigned long size, struct iommu_table *tbl)
 
 struct scatterlist;
 
-static inline void set_iommu_table_base(struct device *dev, void *base)
+#ifdef CONFIG_PPC64
+
+static inline void set_iommu_table_base(struct device *dev,
+					struct iommu_table *base)
 {
-	dev->archdata.dma_data.iommu_table_base = base;
+	dev->archdata.iommu_table_base = base;
 }
 
 static inline void *get_iommu_table_base(struct device *dev)
 {
-	return dev->archdata.dma_data.iommu_table_base;
+	return dev->archdata.iommu_table_base;
 }
 
+extern int dma_iommu_dma_supported(struct device *dev, u64 mask);
+
 /* Frees table for an individual device node */
 extern void iommu_free_table(struct iommu_table *tbl, const char *node_name);
 
@@ -146,6 +151,20 @@ static inline void set_iommu_table_base_and_group(struct device *dev,
 	iommu_add_device(dev);
 }
 
+#else
+
+static inline void *get_iommu_table_base(struct device *dev)
+{
+	return NULL;
+}
+
+static inline int dma_iommu_dma_supported(struct device *dev, u64 mask)
+{
+	return 0;
+}
+
+#endif /* CONFIG_PPC64 */
+
 extern int ppc_iommu_map_sg(struct device *dev, struct iommu_table *tbl,
 			    struct scatterlist *sglist, int nelems,
 			    unsigned long mask,
diff --git a/arch/powerpc/kernel/dma-iommu.c b/arch/powerpc/kernel/dma-iommu.c
index 4c68bfe..41a7d9d 100644
--- a/arch/powerpc/kernel/dma-iommu.c
+++ b/arch/powerpc/kernel/dma-iommu.c
@@ -73,7 +73,7 @@ static void dma_iommu_unmap_sg(struct device *dev, struct scatterlist *sglist,
 }
 
 /* We support DMA to/from any memory page via the iommu */
-static int dma_iommu_dma_supported(struct device *dev, u64 mask)
+int dma_iommu_dma_supported(struct device *dev, u64 mask)
 {
 	struct iommu_table *tbl = get_iommu_table_base(dev);
 
diff --git a/arch/powerpc/kernel/dma-swiotlb.c b/arch/powerpc/kernel/dma-swiotlb.c
index 7359797..81b81cd 100644
--- a/arch/powerpc/kernel/dma-swiotlb.c
+++ b/arch/powerpc/kernel/dma-swiotlb.c
@@ -47,8 +47,8 @@ static u64 swiotlb_powerpc_get_required(struct device *dev)
  * for everything else.
  */
 struct dma_map_ops swiotlb_dma_ops = {
-	.alloc = dma_direct_alloc_coherent,
-	.free = dma_direct_free_coherent,
+	.alloc = __dma_direct_alloc_coherent,
+	.free = __dma_direct_free_coherent,
 	.mmap = dma_direct_mmap_coherent,
 	.map_sg = swiotlb_map_sg_attrs,
 	.unmap_sg = swiotlb_unmap_sg_attrs,
diff --git a/arch/powerpc/kernel/dma.c b/arch/powerpc/kernel/dma.c
index 484b2d4..19941d5 100644
--- a/arch/powerpc/kernel/dma.c
+++ b/arch/powerpc/kernel/dma.c
@@ -16,6 +16,7 @@
 #include <asm/bug.h>
 #include <asm/machdep.h>
 #include <asm/swiotlb.h>
+#include <asm/iommu.h>
 
 /*
  * Generic direct DMA implementation
@@ -39,9 +40,31 @@ static u64 __maybe_unused get_pfn_limit(struct device *dev)
 	return pfn;
 }
 
-void *dma_direct_alloc_coherent(struct device *dev, size_t size,
-				dma_addr_t *dma_handle, gfp_t flag,
-				struct dma_attrs *attrs)
+static int dma_direct_dma_supported(struct device *dev, u64 mask)
+{
+#ifdef CONFIG_PPC64
+	u64 limit = get_dma_offset(dev) + (memblock_end_of_DRAM() - 1);
+
+	/* Limit fits in the mask, we are good */
+	if (mask >= limit)
+		return 1;
+
+#ifdef CONFIG_FSL_SOC
+	/* Freescale gets another chance via ZONE_DMA/ZONE_DMA32, however
+	 * that will have to be refined if/when they support iommus
+	 */
+	return 1;
+#endif
+	/* Sorry ... */
+	return 0;
+#else
+	return 1;
+#endif
+}
+
+void *__dma_direct_alloc_coherent(struct device *dev, size_t size,
+				  dma_addr_t *dma_handle, gfp_t flag,
+				  struct dma_attrs *attrs)
 {
 	void *ret;
 #ifdef CONFIG_NOT_COHERENT_CACHE
@@ -96,9 +119,9 @@ void *dma_direct_alloc_coherent(struct device *dev, size_t size,
 #endif
 }
 
-void dma_direct_free_coherent(struct device *dev, size_t size,
-			      void *vaddr, dma_addr_t dma_handle,
-			      struct dma_attrs *attrs)
+void __dma_direct_free_coherent(struct device *dev, size_t size,
+				void *vaddr, dma_addr_t dma_handle,
+				struct dma_attrs *attrs)
 {
 #ifdef CONFIG_NOT_COHERENT_CACHE
 	__dma_free_coherent(size, vaddr);
@@ -107,6 +130,51 @@ void dma_direct_free_coherent(struct device *dev, size_t size,
 #endif
 }
 
+static void *dma_direct_alloc_coherent(struct device *dev, size_t size,
+				       dma_addr_t *dma_handle, gfp_t flag,
+				       struct dma_attrs *attrs)
+{
+	struct iommu_table *iommu;
+
+	/* The coherent mask may be smaller than the real mask, check if
+	 * we can really use the direct ops
+	 */
+	if (dma_direct_dma_supported(dev, dev->coherent_dma_mask))
+		return __dma_direct_alloc_coherent(dev, size, dma_handle,
+						   flag, attrs);
+
+	/* Ok we can't ... do we have an iommu ? If not, fail */
+	iommu = get_iommu_table_base(dev);
+	if (!iommu)
+		return NULL;
+
+	/* Try to use the iommu */
+	return iommu_alloc_coherent(dev, iommu, size, dma_handle,
+				    dev->coherent_dma_mask, flag,
+				    dev_to_node(dev));
+}
+
+static void dma_direct_free_coherent(struct device *dev, size_t size,
+				     void *vaddr, dma_addr_t dma_handle,
+				     struct dma_attrs *attrs)
+{
+	struct iommu_table *iommu;
+
+	/* See comments in dma_direct_alloc_coherent() */
+	if (dma_direct_dma_supported(dev, dev->coherent_dma_mask))
+		return __dma_direct_free_coherent(dev, size, vaddr, dma_handle,
+						  attrs);
+	/* Maybe we used an iommu ... */
+	iommu = get_iommu_table_base(dev);
+
+	/* If we hit that we should have never allocated in the first
+	 * place so how come we are freeing ?
+	 */
+	if (WARN_ON(!iommu))
+		return;
+	iommu_free_coherent(iommu, size, vaddr, dma_handle);
+}
+
 int dma_direct_mmap_coherent(struct device *dev, struct vm_area_struct *vma,
 			     void *cpu_addr, dma_addr_t handle, size_t size,
 			     struct dma_attrs *attrs)
@@ -147,18 +215,6 @@ static void dma_direct_unmap_sg(struct device *dev, struct scatterlist *sg,
 {
 }
 
-static int dma_direct_dma_supported(struct device *dev, u64 mask)
-{
-#ifdef CONFIG_PPC64
-	/* Could be improved so platforms can set the limit in case
-	 * they have limited DMA windows
-	 */
-	return mask >= get_dma_offset(dev) + (memblock_end_of_DRAM() - 1);
-#else
-	return 1;
-#endif
-}
-
 static u64 dma_direct_get_required_mask(struct device *dev)
 {
 	u64 end, mask;
@@ -230,6 +286,24 @@ struct dma_map_ops dma_direct_ops = {
 };
 EXPORT_SYMBOL(dma_direct_ops);
 
+int dma_set_coherent_mask(struct device *dev, u64 mask)
+{
+	if (!dma_supported(dev, mask)) {
+		/*
+		 * We need to special case the direct DMA ops which can
+		 * support a fallback for coherent allocations. There
+		 * is no dma_op->set_coherent_mask() so we have to do
+		 * things the hard way:
+		 */
+		if (get_dma_ops(dev) != &dma_direct_ops ||
+		    get_iommu_table_base(dev) == NULL ||
+		    !dma_iommu_dma_supported(dev, mask))
+			return -EIO;
+	}
+	dev->coherent_dma_mask = mask;
+	return 0;
+}
+
 #define PREALLOC_DMA_DEBUG_ENTRIES (1 << 16)
 
 int __dma_set_mask(struct device *dev, u64 dma_mask)
diff --git a/arch/powerpc/platforms/powernv/pci-ioda.c b/arch/powerpc/platforms/powernv/pci-ioda.c
index 5ac7c60..fcff6fa 100644
--- a/arch/powerpc/platforms/powernv/pci-ioda.c
+++ b/arch/powerpc/platforms/powernv/pci-ioda.c
@@ -1625,7 +1625,6 @@ static int pnv_pci_ioda_dma_set_mask(struct pnv_phb *phb,
 	} else {
 		dev_info(&pdev->dev, "Using 32-bit DMA via iommu\n");
 		set_dma_ops(&pdev->dev, &dma_iommu_ops);
-		set_iommu_table_base(&pdev->dev, pe->tce32_table);
 	}
 	*pdev->dev.dma_mask = dma_mask;
 	return 0;
diff --git a/arch/powerpc/platforms/pseries/iommu.c b/arch/powerpc/platforms/pseries/iommu.c
index 7803a19..0acab86 100644
--- a/arch/powerpc/platforms/pseries/iommu.c
+++ b/arch/powerpc/platforms/pseries/iommu.c
@@ -1161,11 +1161,10 @@ static int dma_set_mask_pSeriesLP(struct device *dev, u64 dma_mask)
 		}
 	}
 
-	/* fall back on iommu ops, restore table pointer with ops */
+	/* fall back on iommu ops */
 	if (!ddw_enabled && get_dma_ops(dev) != &dma_iommu_ops) {
 		dev_info(dev, "Restoring 32-bit DMA via iommu\n");
 		set_dma_ops(dev, &dma_iommu_ops);
-		pci_dma_dev_setup_pSeriesLP(pdev);
 	}
 
 check_mask:
diff --git a/arch/powerpc/sysdev/dart_iommu.c b/arch/powerpc/sysdev/dart_iommu.c
index 9e5353f..6cf033d 100644
--- a/arch/powerpc/sysdev/dart_iommu.c
+++ b/arch/powerpc/sysdev/dart_iommu.c
@@ -306,20 +306,11 @@ static void iommu_table_dart_setup(void)
 	set_bit(iommu_table_dart.it_size - 1, iommu_table_dart.it_map);
 }
 
-static void dma_dev_setup_dart(struct device *dev)
-{
-	/* We only have one iommu table on the mac for now, which makes
-	 * things simple. Setup all PCI devices to point to this table
-	 */
-	if (get_dma_ops(dev) == &dma_direct_ops)
-		set_dma_offset(dev, DART_U4_BYPASS_BASE);
-	else
-		set_iommu_table_base(dev, &iommu_table_dart);
-}
-
 static void pci_dma_dev_setup_dart(struct pci_dev *dev)
 {
-	dma_dev_setup_dart(&dev->dev);
+	if (dart_is_u4)
+		set_dma_offset(&dev->dev, DART_U4_BYPASS_BASE);
+	set_iommu_table_base(&dev->dev, &iommu_table_dart);
 }
 
 static void pci_dma_bus_setup_dart(struct pci_bus *bus)
@@ -363,7 +354,6 @@ static int dart_dma_set_mask(struct device *dev, u64 dma_mask)
 		dev_info(dev, "Using 32-bit DMA via iommu\n");
 		set_dma_ops(dev, &dma_iommu_ops);
 	}
-	dma_dev_setup_dart(dev);
 
 	*dev->dma_mask = dma_mask;
 	return 0;

[RFC/PATCH v2] powerpc/iommu: Support "hybrid" iommu/direct DMA ops for coherent_mask < dma_mask

[Benjamin Herrenschmidt]

This patch adds the ability to the DMA direct ops to fallback to the IOMMU
ops for coherent alloc/free if the coherent mask of the device isn't
suitable for accessing the direct DMA space and the device also happens
to have an active IOMMU table.

Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
---

v2: Implement a custom dma_set_coherent_mask() to work around the fact
that dma_supported() will fail otherwise.

diff --git a/arch/powerpc/Kconfig b/arch/powerpc/Kconfig
index 22b0940..cd46474 100644
--- a/arch/powerpc/Kconfig
+++ b/arch/powerpc/Kconfig
@@ -82,6 +82,9 @@ config GENERIC_HWEIGHT
 	bool
 	default y
 
+config ARCH_HAS_DMA_SET_COHERENT_MASK
+        bool
+
 config PPC
 	bool
 	default y
@@ -152,6 +155,7 @@ config PPC
 	select DCACHE_WORD_ACCESS if PPC64 && CPU_LITTLE_ENDIAN
 	select NO_BOOTMEM
 	select HAVE_GENERIC_RCU_GUP
+	select ARCH_HAS_DMA_SET_COHERENT_MASK
 
 config GENERIC_CSUM
 	def_bool CPU_LITTLE_ENDIAN
diff --git a/arch/powerpc/include/asm/device.h b/arch/powerpc/include/asm/device.h
index 9f1371b..9ce2199 100644
--- a/arch/powerpc/include/asm/device.h
+++ b/arch/powerpc/include/asm/device.h
@@ -10,6 +10,7 @@ struct dma_map_ops;
 struct device_node;
 #ifdef CONFIG_PPC64
 struct pci_dn;
+struct iommu_table;
 #endif
 
 /*
@@ -23,13 +24,15 @@ struct dev_archdata {
 	struct dma_map_ops	*dma_ops;
 
 	/*
-	 * When an iommu is in use, dma_data is used as a ptr to the base of the
-	 * iommu_table.  Otherwise, it is a simple numerical offset.
+	 * These two used to be a union. However, with the hybrid ops we need
+	 * both so here we store both a DMA offset for direct mappings and
+	 * an iommu_table for remapped DMA.
 	 */
-	union {
-		dma_addr_t	dma_offset;
-		void		*iommu_table_base;
-	} dma_data;
+	dma_addr_t		dma_offset;
+
+#ifdef CONFIG_PPC64
+	struct iommu_table	*iommu_table_base;
+#endif
 
 #ifdef CONFIG_IOMMU_API
 	void			*iommu_domain;
diff --git a/arch/powerpc/include/asm/dma-mapping.h b/arch/powerpc/include/asm/dma-mapping.h
index 894d538..2e12366 100644
--- a/arch/powerpc/include/asm/dma-mapping.h
+++ b/arch/powerpc/include/asm/dma-mapping.h
@@ -21,12 +21,12 @@
 #define DMA_ERROR_CODE		(~(dma_addr_t)0x0)
 
 /* Some dma direct funcs must be visible for use in other dma_ops */
-extern void *dma_direct_alloc_coherent(struct device *dev, size_t size,
-				       dma_addr_t *dma_handle, gfp_t flag,
+extern void *__dma_direct_alloc_coherent(struct device *dev, size_t size,
+					 dma_addr_t *dma_handle, gfp_t flag,
+					 struct dma_attrs *attrs);
+extern void __dma_direct_free_coherent(struct device *dev, size_t size,
+				       void *vaddr, dma_addr_t dma_handle,
 				       struct dma_attrs *attrs);
-extern void dma_direct_free_coherent(struct device *dev, size_t size,
-				     void *vaddr, dma_addr_t dma_handle,
-				     struct dma_attrs *attrs);
 extern int dma_direct_mmap_coherent(struct device *dev,
 				    struct vm_area_struct *vma,
 				    void *cpu_addr, dma_addr_t handle,
@@ -106,7 +106,7 @@ static inline void set_dma_ops(struct device *dev, struct dma_map_ops *ops)
 static inline dma_addr_t get_dma_offset(struct device *dev)
 {
 	if (dev)
-		return dev->archdata.dma_data.dma_offset;
+		return dev->archdata.dma_offset;
 
 	return PCI_DRAM_OFFSET;
 }
@@ -114,7 +114,7 @@ static inline dma_addr_t get_dma_offset(struct device *dev)
 static inline void set_dma_offset(struct device *dev, dma_addr_t off)
 {
 	if (dev)
-		dev->archdata.dma_data.dma_offset = off;
+		dev->archdata.dma_offset = off;
 }
 
 /* this will be removed soon */
diff --git a/arch/powerpc/include/asm/iommu.h b/arch/powerpc/include/asm/iommu.h
index e2abbe8..e67e678 100644
--- a/arch/powerpc/include/asm/iommu.h
+++ b/arch/powerpc/include/asm/iommu.h
@@ -2,17 +2,17 @@
  * Copyright (C) 2001 Mike Corrigan & Dave Engebretsen, IBM Corporation
  * Rewrite, cleanup:
  * Copyright (C) 2004 Olof Johansson <olof@lixom.net>, IBM Corporation
- * 
+ *
  * This program is free software; you can redistribute it and/or modify
  * it under the terms of the GNU General Public License as published by
  * the Free Software Foundation; either version 2 of the License, or
  * (at your option) any later version.
- * 
+ *
  * This program is distributed in the hope that it will be useful,
  * but WITHOUT ANY WARRANTY; without even the implied warranty of
  * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
  * GNU General Public License for more details.
- * 
+ *
  * You should have received a copy of the GNU General Public License
  * along with this program; if not, write to the Free Software
  * Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307 USA
@@ -93,16 +93,21 @@ int get_iommu_order(unsigned long size, struct iommu_table *tbl)
 
 struct scatterlist;
 
-static inline void set_iommu_table_base(struct device *dev, void *base)
+#ifdef CONFIG_PPC64
+
+static inline void set_iommu_table_base(struct device *dev,
+					struct iommu_table *base)
 {
-	dev->archdata.dma_data.iommu_table_base = base;
+	dev->archdata.iommu_table_base = base;
 }
 
 static inline void *get_iommu_table_base(struct device *dev)
 {
-	return dev->archdata.dma_data.iommu_table_base;
+	return dev->archdata.iommu_table_base;
 }
 
+extern int dma_iommu_dma_supported(struct device *dev, u64 mask);
+
 /* Frees table for an individual device node */
 extern void iommu_free_table(struct iommu_table *tbl, const char *node_name);
 
@@ -146,6 +151,20 @@ static inline void set_iommu_table_base_and_group(struct device *dev,
 	iommu_add_device(dev);
 }
 
+#else
+
+static inline void *get_iommu_table_base(struct device *dev)
+{
+	return NULL;
+}
+
+static inline int dma_iommu_dma_supported(struct device *dev, u64 mask)
+{
+	return 0;
+}
+
+#endif /* CONFIG_PPC64 */
+
 extern int ppc_iommu_map_sg(struct device *dev, struct iommu_table *tbl,
 			    struct scatterlist *sglist, int nelems,
 			    unsigned long mask,
diff --git a/arch/powerpc/kernel/dma-iommu.c b/arch/powerpc/kernel/dma-iommu.c
index 4c68bfe..41a7d9d 100644
--- a/arch/powerpc/kernel/dma-iommu.c
+++ b/arch/powerpc/kernel/dma-iommu.c
@@ -73,7 +73,7 @@ static void dma_iommu_unmap_sg(struct device *dev, struct scatterlist *sglist,
 }
 
 /* We support DMA to/from any memory page via the iommu */
-static int dma_iommu_dma_supported(struct device *dev, u64 mask)
+int dma_iommu_dma_supported(struct device *dev, u64 mask)
 {
 	struct iommu_table *tbl = get_iommu_table_base(dev);
 
diff --git a/arch/powerpc/kernel/dma-swiotlb.c b/arch/powerpc/kernel/dma-swiotlb.c
index 7359797..81b81cd 100644
--- a/arch/powerpc/kernel/dma-swiotlb.c
+++ b/arch/powerpc/kernel/dma-swiotlb.c
@@ -47,8 +47,8 @@ static u64 swiotlb_powerpc_get_required(struct device *dev)
  * for everything else.
  */
 struct dma_map_ops swiotlb_dma_ops = {
-	.alloc = dma_direct_alloc_coherent,
-	.free = dma_direct_free_coherent,
+	.alloc = __dma_direct_alloc_coherent,
+	.free = __dma_direct_free_coherent,
 	.mmap = dma_direct_mmap_coherent,
 	.map_sg = swiotlb_map_sg_attrs,
 	.unmap_sg = swiotlb_unmap_sg_attrs,
diff --git a/arch/powerpc/kernel/dma.c b/arch/powerpc/kernel/dma.c
index 484b2d4..19941d5 100644
--- a/arch/powerpc/kernel/dma.c
+++ b/arch/powerpc/kernel/dma.c
@@ -16,6 +16,7 @@
 #include <asm/bug.h>
 #include <asm/machdep.h>
 #include <asm/swiotlb.h>
+#include <asm/iommu.h>
 
 /*
  * Generic direct DMA implementation
@@ -39,9 +40,31 @@ static u64 __maybe_unused get_pfn_limit(struct device *dev)
 	return pfn;
 }
 
-void *dma_direct_alloc_coherent(struct device *dev, size_t size,
-				dma_addr_t *dma_handle, gfp_t flag,
-				struct dma_attrs *attrs)
+static int dma_direct_dma_supported(struct device *dev, u64 mask)
+{
+#ifdef CONFIG_PPC64
+	u64 limit = get_dma_offset(dev) + (memblock_end_of_DRAM() - 1);
+
+	/* Limit fits in the mask, we are good */
+	if (mask >= limit)
+		return 1;
+
+#ifdef CONFIG_FSL_SOC
+	/* Freescale gets another chance via ZONE_DMA/ZONE_DMA32, however
+	 * that will have to be refined if/when they support iommus
+	 */
+	return 1;
+#endif
+	/* Sorry ... */
+	return 0;
+#else
+	return 1;
+#endif
+}
+
+void *__dma_direct_alloc_coherent(struct device *dev, size_t size,
+				  dma_addr_t *dma_handle, gfp_t flag,
+				  struct dma_attrs *attrs)
 {
 	void *ret;
 #ifdef CONFIG_NOT_COHERENT_CACHE
@@ -96,9 +119,9 @@ void *dma_direct_alloc_coherent(struct device *dev, size_t size,
 #endif
 }
 
-void dma_direct_free_coherent(struct device *dev, size_t size,
-			      void *vaddr, dma_addr_t dma_handle,
-			      struct dma_attrs *attrs)
+void __dma_direct_free_coherent(struct device *dev, size_t size,
+				void *vaddr, dma_addr_t dma_handle,
+				struct dma_attrs *attrs)
 {
 #ifdef CONFIG_NOT_COHERENT_CACHE
 	__dma_free_coherent(size, vaddr);
@@ -107,6 +130,51 @@ void dma_direct_free_coherent(struct device *dev, size_t size,
 #endif
 }
 
+static void *dma_direct_alloc_coherent(struct device *dev, size_t size,
+				       dma_addr_t *dma_handle, gfp_t flag,
+				       struct dma_attrs *attrs)
+{
+	struct iommu_table *iommu;
+
+	/* The coherent mask may be smaller than the real mask, check if
+	 * we can really use the direct ops
+	 */
+	if (dma_direct_dma_supported(dev, dev->coherent_dma_mask))
+		return __dma_direct_alloc_coherent(dev, size, dma_handle,
+						   flag, attrs);
+
+	/* Ok we can't ... do we have an iommu ? If not, fail */
+	iommu = get_iommu_table_base(dev);
+	if (!iommu)
+		return NULL;
+
+	/* Try to use the iommu */
+	return iommu_alloc_coherent(dev, iommu, size, dma_handle,
+				    dev->coherent_dma_mask, flag,
+				    dev_to_node(dev));
+}
+
+static void dma_direct_free_coherent(struct device *dev, size_t size,
+				     void *vaddr, dma_addr_t dma_handle,
+				     struct dma_attrs *attrs)
+{
+	struct iommu_table *iommu;
+
+	/* See comments in dma_direct_alloc_coherent() */
+	if (dma_direct_dma_supported(dev, dev->coherent_dma_mask))
+		return __dma_direct_free_coherent(dev, size, vaddr, dma_handle,
+						  attrs);
+	/* Maybe we used an iommu ... */
+	iommu = get_iommu_table_base(dev);
+
+	/* If we hit that we should have never allocated in the first
+	 * place so how come we are freeing ?
+	 */
+	if (WARN_ON(!iommu))
+		return;
+	iommu_free_coherent(iommu, size, vaddr, dma_handle);
+}
+
 int dma_direct_mmap_coherent(struct device *dev, struct vm_area_struct *vma,
 			     void *cpu_addr, dma_addr_t handle, size_t size,
 			     struct dma_attrs *attrs)
@@ -147,18 +215,6 @@ static void dma_direct_unmap_sg(struct device *dev, struct scatterlist *sg,
 {
 }
 
-static int dma_direct_dma_supported(struct device *dev, u64 mask)
-{
-#ifdef CONFIG_PPC64
-	/* Could be improved so platforms can set the limit in case
-	 * they have limited DMA windows
-	 */
-	return mask >= get_dma_offset(dev) + (memblock_end_of_DRAM() - 1);
-#else
-	return 1;
-#endif
-}
-
 static u64 dma_direct_get_required_mask(struct device *dev)
 {
 	u64 end, mask;
@@ -230,6 +286,24 @@ struct dma_map_ops dma_direct_ops = {
 };
 EXPORT_SYMBOL(dma_direct_ops);
 
+int dma_set_coherent_mask(struct device *dev, u64 mask)
+{
+	if (!dma_supported(dev, mask)) {
+		/*
+		 * We need to special case the direct DMA ops which can
+		 * support a fallback for coherent allocations. There
+		 * is no dma_op->set_coherent_mask() so we have to do
+		 * things the hard way:
+		 */
+		if (get_dma_ops(dev) != &dma_direct_ops ||
+		    get_iommu_table_base(dev) == NULL ||
+		    !dma_iommu_dma_supported(dev, mask))
+			return -EIO;
+	}
+	dev->coherent_dma_mask = mask;
+	return 0;
+}
+
 #define PREALLOC_DMA_DEBUG_ENTRIES (1 << 16)
 
 int __dma_set_mask(struct device *dev, u64 dma_mask)
diff --git a/arch/powerpc/platforms/powernv/pci-ioda.c b/arch/powerpc/platforms/powernv/pci-ioda.c
index 5ac7c60..fcff6fa 100644
--- a/arch/powerpc/platforms/powernv/pci-ioda.c
+++ b/arch/powerpc/platforms/powernv/pci-ioda.c
@@ -1625,7 +1625,6 @@ static int pnv_pci_ioda_dma_set_mask(struct pnv_phb *phb,
 	} else {
 		dev_info(&pdev->dev, "Using 32-bit DMA via iommu\n");
 		set_dma_ops(&pdev->dev, &dma_iommu_ops);
-		set_iommu_table_base(&pdev->dev, pe->tce32_table);
 	}
 	*pdev->dev.dma_mask = dma_mask;
 	return 0;
diff --git a/arch/powerpc/platforms/pseries/iommu.c b/arch/powerpc/platforms/pseries/iommu.c
index 7803a19..0acab86 100644
--- a/arch/powerpc/platforms/pseries/iommu.c
+++ b/arch/powerpc/platforms/pseries/iommu.c
@@ -1161,11 +1161,10 @@ static int dma_set_mask_pSeriesLP(struct device *dev, u64 dma_mask)
 		}
 	}
 
-	/* fall back on iommu ops, restore table pointer with ops */
+	/* fall back on iommu ops */
 	if (!ddw_enabled && get_dma_ops(dev) != &dma_iommu_ops) {
 		dev_info(dev, "Restoring 32-bit DMA via iommu\n");
 		set_dma_ops(dev, &dma_iommu_ops);
-		pci_dma_dev_setup_pSeriesLP(pdev);
 	}
 
 check_mask:
diff --git a/arch/powerpc/sysdev/dart_iommu.c b/arch/powerpc/sysdev/dart_iommu.c
index 9e5353f..6cf033d 100644
--- a/arch/powerpc/sysdev/dart_iommu.c
+++ b/arch/powerpc/sysdev/dart_iommu.c
@@ -306,20 +306,11 @@ static void iommu_table_dart_setup(void)
 	set_bit(iommu_table_dart.it_size - 1, iommu_table_dart.it_map);
 }
 
-static void dma_dev_setup_dart(struct device *dev)
-{
-	/* We only have one iommu table on the mac for now, which makes
-	 * things simple. Setup all PCI devices to point to this table
-	 */
-	if (get_dma_ops(dev) == &dma_direct_ops)
-		set_dma_offset(dev, DART_U4_BYPASS_BASE);
-	else
-		set_iommu_table_base(dev, &iommu_table_dart);
-}
-
 static void pci_dma_dev_setup_dart(struct pci_dev *dev)
 {
-	dma_dev_setup_dart(&dev->dev);
+	if (dart_is_u4)
+		set_dma_offset(&dev->dev, DART_U4_BYPASS_BASE);
+	set_iommu_table_base(&dev->dev, &iommu_table_dart);
 }
 
 static void pci_dma_bus_setup_dart(struct pci_bus *bus)
@@ -363,7 +354,6 @@ static int dart_dma_set_mask(struct device *dev, u64 dma_mask)
 		dev_info(dev, "Using 32-bit DMA via iommu\n");
 		set_dma_ops(dev, &dma_iommu_ops);
 	}
-	dma_dev_setup_dart(dev);
 
 	*dev->dma_mask = dma_mask;
 	return 0;

[RFC/PATCH v3] powerpc/iommu: Support "hybrid" iommu/direct DMA ops for coherent_mask < dma_mask

[Benjamin Herrenschmidt]

This patch adds the ability to the DMA direct ops to fallback to the IOMMU
ops for coherent alloc/free if the coherent mask of the device isn't
suitable for accessing the direct DMA space and the device also happens
to have an active IOMMU table.

Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
---

v2: Implement a custom dma_set_coherent_mask() to work around the fact
that dma_supported() will fail otherwise.

v3: Add missing symbol export

diff --git a/arch/powerpc/Kconfig b/arch/powerpc/Kconfig
index 22b0940..cd46474 100644
--- a/arch/powerpc/Kconfig
+++ b/arch/powerpc/Kconfig
@@ -82,6 +82,9 @@ config GENERIC_HWEIGHT
 	bool
 	default y
 
+config ARCH_HAS_DMA_SET_COHERENT_MASK
+        bool
+
 config PPC
 	bool
 	default y
@@ -152,6 +155,7 @@ config PPC
 	select DCACHE_WORD_ACCESS if PPC64 && CPU_LITTLE_ENDIAN
 	select NO_BOOTMEM
 	select HAVE_GENERIC_RCU_GUP
+	select ARCH_HAS_DMA_SET_COHERENT_MASK
 
 config GENERIC_CSUM
 	def_bool CPU_LITTLE_ENDIAN
diff --git a/arch/powerpc/include/asm/device.h b/arch/powerpc/include/asm/device.h
index 9f1371b..9ce2199 100644
--- a/arch/powerpc/include/asm/device.h
+++ b/arch/powerpc/include/asm/device.h
@@ -10,6 +10,7 @@ struct dma_map_ops;
 struct device_node;
 #ifdef CONFIG_PPC64
 struct pci_dn;
+struct iommu_table;
 #endif
 
 /*
@@ -23,13 +24,15 @@ struct dev_archdata {
 	struct dma_map_ops	*dma_ops;
 
 	/*
-	 * When an iommu is in use, dma_data is used as a ptr to the base of the
-	 * iommu_table.  Otherwise, it is a simple numerical offset.
+	 * These two used to be a union. However, with the hybrid ops we need
+	 * both so here we store both a DMA offset for direct mappings and
+	 * an iommu_table for remapped DMA.
 	 */
-	union {
-		dma_addr_t	dma_offset;
-		void		*iommu_table_base;
-	} dma_data;
+	dma_addr_t		dma_offset;
+
+#ifdef CONFIG_PPC64
+	struct iommu_table	*iommu_table_base;
+#endif
 
 #ifdef CONFIG_IOMMU_API
 	void			*iommu_domain;
diff --git a/arch/powerpc/include/asm/dma-mapping.h b/arch/powerpc/include/asm/dma-mapping.h
index 894d538..2e12366 100644
--- a/arch/powerpc/include/asm/dma-mapping.h
+++ b/arch/powerpc/include/asm/dma-mapping.h
@@ -21,12 +21,12 @@
 #define DMA_ERROR_CODE		(~(dma_addr_t)0x0)
 
 /* Some dma direct funcs must be visible for use in other dma_ops */
-extern void *dma_direct_alloc_coherent(struct device *dev, size_t size,
-				       dma_addr_t *dma_handle, gfp_t flag,
+extern void *__dma_direct_alloc_coherent(struct device *dev, size_t size,
+					 dma_addr_t *dma_handle, gfp_t flag,
+					 struct dma_attrs *attrs);
+extern void __dma_direct_free_coherent(struct device *dev, size_t size,
+				       void *vaddr, dma_addr_t dma_handle,
 				       struct dma_attrs *attrs);
-extern void dma_direct_free_coherent(struct device *dev, size_t size,
-				     void *vaddr, dma_addr_t dma_handle,
-				     struct dma_attrs *attrs);
 extern int dma_direct_mmap_coherent(struct device *dev,
 				    struct vm_area_struct *vma,
 				    void *cpu_addr, dma_addr_t handle,
@@ -106,7 +106,7 @@ static inline void set_dma_ops(struct device *dev, struct dma_map_ops *ops)
 static inline dma_addr_t get_dma_offset(struct device *dev)
 {
 	if (dev)
-		return dev->archdata.dma_data.dma_offset;
+		return dev->archdata.dma_offset;
 
 	return PCI_DRAM_OFFSET;
 }
@@ -114,7 +114,7 @@ static inline dma_addr_t get_dma_offset(struct device *dev)
 static inline void set_dma_offset(struct device *dev, dma_addr_t off)
 {
 	if (dev)
-		dev->archdata.dma_data.dma_offset = off;
+		dev->archdata.dma_offset = off;
 }
 
 /* this will be removed soon */
diff --git a/arch/powerpc/include/asm/iommu.h b/arch/powerpc/include/asm/iommu.h
index e2abbe8..e67e678 100644
--- a/arch/powerpc/include/asm/iommu.h
+++ b/arch/powerpc/include/asm/iommu.h
@@ -2,17 +2,17 @@
  * Copyright (C) 2001 Mike Corrigan & Dave Engebretsen, IBM Corporation
  * Rewrite, cleanup:
  * Copyright (C) 2004 Olof Johansson <olof@lixom.net>, IBM Corporation
- * 
+ *
  * This program is free software; you can redistribute it and/or modify
  * it under the terms of the GNU General Public License as published by
  * the Free Software Foundation; either version 2 of the License, or
  * (at your option) any later version.
- * 
+ *
  * This program is distributed in the hope that it will be useful,
  * but WITHOUT ANY WARRANTY; without even the implied warranty of
  * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
  * GNU General Public License for more details.
- * 
+ *
  * You should have received a copy of the GNU General Public License
  * along with this program; if not, write to the Free Software
  * Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307 USA
@@ -93,16 +93,21 @@ int get_iommu_order(unsigned long size, struct iommu_table *tbl)
 
 struct scatterlist;
 
-static inline void set_iommu_table_base(struct device *dev, void *base)
+#ifdef CONFIG_PPC64
+
+static inline void set_iommu_table_base(struct device *dev,
+					struct iommu_table *base)
 {
-	dev->archdata.dma_data.iommu_table_base = base;
+	dev->archdata.iommu_table_base = base;
 }
 
 static inline void *get_iommu_table_base(struct device *dev)
 {
-	return dev->archdata.dma_data.iommu_table_base;
+	return dev->archdata.iommu_table_base;
 }
 
+extern int dma_iommu_dma_supported(struct device *dev, u64 mask);
+
 /* Frees table for an individual device node */
 extern void iommu_free_table(struct iommu_table *tbl, const char *node_name);
 
@@ -146,6 +151,20 @@ static inline void set_iommu_table_base_and_group(struct device *dev,
 	iommu_add_device(dev);
 }
 
+#else
+
+static inline void *get_iommu_table_base(struct device *dev)
+{
+	return NULL;
+}
+
+static inline int dma_iommu_dma_supported(struct device *dev, u64 mask)
+{
+	return 0;
+}
+
+#endif /* CONFIG_PPC64 */
+
 extern int ppc_iommu_map_sg(struct device *dev, struct iommu_table *tbl,
 			    struct scatterlist *sglist, int nelems,
 			    unsigned long mask,
diff --git a/arch/powerpc/kernel/dma-iommu.c b/arch/powerpc/kernel/dma-iommu.c
index 4c68bfe..41a7d9d 100644
--- a/arch/powerpc/kernel/dma-iommu.c
+++ b/arch/powerpc/kernel/dma-iommu.c
@@ -73,7 +73,7 @@ static void dma_iommu_unmap_sg(struct device *dev, struct scatterlist *sglist,
 }
 
 /* We support DMA to/from any memory page via the iommu */
-static int dma_iommu_dma_supported(struct device *dev, u64 mask)
+int dma_iommu_dma_supported(struct device *dev, u64 mask)
 {
 	struct iommu_table *tbl = get_iommu_table_base(dev);
 
diff --git a/arch/powerpc/kernel/dma-swiotlb.c b/arch/powerpc/kernel/dma-swiotlb.c
index 7359797..81b81cd 100644
--- a/arch/powerpc/kernel/dma-swiotlb.c
+++ b/arch/powerpc/kernel/dma-swiotlb.c
@@ -47,8 +47,8 @@ static u64 swiotlb_powerpc_get_required(struct device *dev)
  * for everything else.
  */
 struct dma_map_ops swiotlb_dma_ops = {
-	.alloc = dma_direct_alloc_coherent,
-	.free = dma_direct_free_coherent,
+	.alloc = __dma_direct_alloc_coherent,
+	.free = __dma_direct_free_coherent,
 	.mmap = dma_direct_mmap_coherent,
 	.map_sg = swiotlb_map_sg_attrs,
 	.unmap_sg = swiotlb_unmap_sg_attrs,
diff --git a/arch/powerpc/kernel/dma.c b/arch/powerpc/kernel/dma.c
index 484b2d4..b0c4faa 100644
--- a/arch/powerpc/kernel/dma.c
+++ b/arch/powerpc/kernel/dma.c
@@ -16,6 +16,7 @@
 #include <asm/bug.h>
 #include <asm/machdep.h>
 #include <asm/swiotlb.h>
+#include <asm/iommu.h>
 
 /*
  * Generic direct DMA implementation
@@ -39,9 +40,31 @@ static u64 __maybe_unused get_pfn_limit(struct device *dev)
 	return pfn;
 }
 
-void *dma_direct_alloc_coherent(struct device *dev, size_t size,
-				dma_addr_t *dma_handle, gfp_t flag,
-				struct dma_attrs *attrs)
+static int dma_direct_dma_supported(struct device *dev, u64 mask)
+{
+#ifdef CONFIG_PPC64
+	u64 limit = get_dma_offset(dev) + (memblock_end_of_DRAM() - 1);
+
+	/* Limit fits in the mask, we are good */
+	if (mask >= limit)
+		return 1;
+
+#ifdef CONFIG_FSL_SOC
+	/* Freescale gets another chance via ZONE_DMA/ZONE_DMA32, however
+	 * that will have to be refined if/when they support iommus
+	 */
+	return 1;
+#endif
+	/* Sorry ... */
+	return 0;
+#else
+	return 1;
+#endif
+}
+
+void *__dma_direct_alloc_coherent(struct device *dev, size_t size,
+				  dma_addr_t *dma_handle, gfp_t flag,
+				  struct dma_attrs *attrs)
 {
 	void *ret;
 #ifdef CONFIG_NOT_COHERENT_CACHE
@@ -96,9 +119,9 @@ void *dma_direct_alloc_coherent(struct device *dev, size_t size,
 #endif
 }
 
-void dma_direct_free_coherent(struct device *dev, size_t size,
-			      void *vaddr, dma_addr_t dma_handle,
-			      struct dma_attrs *attrs)
+void __dma_direct_free_coherent(struct device *dev, size_t size,
+				void *vaddr, dma_addr_t dma_handle,
+				struct dma_attrs *attrs)
 {
 #ifdef CONFIG_NOT_COHERENT_CACHE
 	__dma_free_coherent(size, vaddr);
@@ -107,6 +130,51 @@ void dma_direct_free_coherent(struct device *dev, size_t size,
 #endif
 }
 
+static void *dma_direct_alloc_coherent(struct device *dev, size_t size,
+				       dma_addr_t *dma_handle, gfp_t flag,
+				       struct dma_attrs *attrs)
+{
+	struct iommu_table *iommu;
+
+	/* The coherent mask may be smaller than the real mask, check if
+	 * we can really use the direct ops
+	 */
+	if (dma_direct_dma_supported(dev, dev->coherent_dma_mask))
+		return __dma_direct_alloc_coherent(dev, size, dma_handle,
+						   flag, attrs);
+
+	/* Ok we can't ... do we have an iommu ? If not, fail */
+	iommu = get_iommu_table_base(dev);
+	if (!iommu)
+		return NULL;
+
+	/* Try to use the iommu */
+	return iommu_alloc_coherent(dev, iommu, size, dma_handle,
+				    dev->coherent_dma_mask, flag,
+				    dev_to_node(dev));
+}
+
+static void dma_direct_free_coherent(struct device *dev, size_t size,
+				     void *vaddr, dma_addr_t dma_handle,
+				     struct dma_attrs *attrs)
+{
+	struct iommu_table *iommu;
+
+	/* See comments in dma_direct_alloc_coherent() */
+	if (dma_direct_dma_supported(dev, dev->coherent_dma_mask))
+		return __dma_direct_free_coherent(dev, size, vaddr, dma_handle,
+						  attrs);
+	/* Maybe we used an iommu ... */
+	iommu = get_iommu_table_base(dev);
+
+	/* If we hit that we should have never allocated in the first
+	 * place so how come we are freeing ?
+	 */
+	if (WARN_ON(!iommu))
+		return;
+	iommu_free_coherent(iommu, size, vaddr, dma_handle);
+}
+
 int dma_direct_mmap_coherent(struct device *dev, struct vm_area_struct *vma,
 			     void *cpu_addr, dma_addr_t handle, size_t size,
 			     struct dma_attrs *attrs)
@@ -147,18 +215,6 @@ static void dma_direct_unmap_sg(struct device *dev, struct scatterlist *sg,
 {
 }
 
-static int dma_direct_dma_supported(struct device *dev, u64 mask)
-{
-#ifdef CONFIG_PPC64
-	/* Could be improved so platforms can set the limit in case
-	 * they have limited DMA windows
-	 */
-	return mask >= get_dma_offset(dev) + (memblock_end_of_DRAM() - 1);
-#else
-	return 1;
-#endif
-}
-
 static u64 dma_direct_get_required_mask(struct device *dev)
 {
 	u64 end, mask;
@@ -230,6 +286,25 @@ struct dma_map_ops dma_direct_ops = {
 };
 EXPORT_SYMBOL(dma_direct_ops);
 
+int dma_set_coherent_mask(struct device *dev, u64 mask)
+{
+	if (!dma_supported(dev, mask)) {
+		/*
+		 * We need to special case the direct DMA ops which can
+		 * support a fallback for coherent allocations. There
+		 * is no dma_op->set_coherent_mask() so we have to do
+		 * things the hard way:
+		 */
+		if (get_dma_ops(dev) != &dma_direct_ops ||
+		    get_iommu_table_base(dev) == NULL ||
+		    !dma_iommu_dma_supported(dev, mask))
+			return -EIO;
+	}
+	dev->coherent_dma_mask = mask;
+	return 0;
+}
+EXPORT_SYMBOL_GPL(dma_set_coherent_mask);
+
 #define PREALLOC_DMA_DEBUG_ENTRIES (1 << 16)
 
 int __dma_set_mask(struct device *dev, u64 dma_mask)
diff --git a/arch/powerpc/platforms/powernv/pci-ioda.c b/arch/powerpc/platforms/powernv/pci-ioda.c
index 5ac7c60..fcff6fa 100644
--- a/arch/powerpc/platforms/powernv/pci-ioda.c
+++ b/arch/powerpc/platforms/powernv/pci-ioda.c
@@ -1625,7 +1625,6 @@ static int pnv_pci_ioda_dma_set_mask(struct pnv_phb *phb,
 	} else {
 		dev_info(&pdev->dev, "Using 32-bit DMA via iommu\n");
 		set_dma_ops(&pdev->dev, &dma_iommu_ops);
-		set_iommu_table_base(&pdev->dev, pe->tce32_table);
 	}
 	*pdev->dev.dma_mask = dma_mask;
 	return 0;
diff --git a/arch/powerpc/platforms/pseries/iommu.c b/arch/powerpc/platforms/pseries/iommu.c
index 7803a19..0acab86 100644
--- a/arch/powerpc/platforms/pseries/iommu.c
+++ b/arch/powerpc/platforms/pseries/iommu.c
@@ -1161,11 +1161,10 @@ static int dma_set_mask_pSeriesLP(struct device *dev, u64 dma_mask)
 		}
 	}
 
-	/* fall back on iommu ops, restore table pointer with ops */
+	/* fall back on iommu ops */
 	if (!ddw_enabled && get_dma_ops(dev) != &dma_iommu_ops) {
 		dev_info(dev, "Restoring 32-bit DMA via iommu\n");
 		set_dma_ops(dev, &dma_iommu_ops);
-		pci_dma_dev_setup_pSeriesLP(pdev);
 	}
 
 check_mask:
diff --git a/arch/powerpc/sysdev/dart_iommu.c b/arch/powerpc/sysdev/dart_iommu.c
index 9e5353f..6cf033d 100644
--- a/arch/powerpc/sysdev/dart_iommu.c
+++ b/arch/powerpc/sysdev/dart_iommu.c
@@ -306,20 +306,11 @@ static void iommu_table_dart_setup(void)
 	set_bit(iommu_table_dart.it_size - 1, iommu_table_dart.it_map);
 }
 
-static void dma_dev_setup_dart(struct device *dev)
-{
-	/* We only have one iommu table on the mac for now, which makes
-	 * things simple. Setup all PCI devices to point to this table
-	 */
-	if (get_dma_ops(dev) == &dma_direct_ops)
-		set_dma_offset(dev, DART_U4_BYPASS_BASE);
-	else
-		set_iommu_table_base(dev, &iommu_table_dart);
-}
-
 static void pci_dma_dev_setup_dart(struct pci_dev *dev)
 {
-	dma_dev_setup_dart(&dev->dev);
+	if (dart_is_u4)
+		set_dma_offset(&dev->dev, DART_U4_BYPASS_BASE);
+	set_iommu_table_base(&dev->dev, &iommu_table_dart);
 }
 
 static void pci_dma_bus_setup_dart(struct pci_bus *bus)
@@ -363,7 +354,6 @@ static int dart_dma_set_mask(struct device *dev, u64 dma_mask)
 		dev_info(dev, "Using 32-bit DMA via iommu\n");
 		set_dma_ops(dev, &dma_iommu_ops);
 	}
-	dma_dev_setup_dart(dev);
 
 	*dev->dma_mask = dma_mask;
 	return 0;

[RFC/PATCH v3] powerpc/iommu: Support "hybrid" iommu/direct DMA ops for coherent_mask < dma_mask

[Benjamin Herrenschmidt]

This patch adds the ability to the DMA direct ops to fallback to the IOMMU
ops for coherent alloc/free if the coherent mask of the device isn't
suitable for accessing the direct DMA space and the device also happens
to have an active IOMMU table.

Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
---

v2: Implement a custom dma_set_coherent_mask() to work around the fact
that dma_supported() will fail otherwise.

v3: Add missing symbol export

diff --git a/arch/powerpc/Kconfig b/arch/powerpc/Kconfig
index 22b0940..cd46474 100644
--- a/arch/powerpc/Kconfig
+++ b/arch/powerpc/Kconfig
@@ -82,6 +82,9 @@ config GENERIC_HWEIGHT
 	bool
 	default y
 
+config ARCH_HAS_DMA_SET_COHERENT_MASK
+        bool
+
 config PPC
 	bool
 	default y
@@ -152,6 +155,7 @@ config PPC
 	select DCACHE_WORD_ACCESS if PPC64 && CPU_LITTLE_ENDIAN
 	select NO_BOOTMEM
 	select HAVE_GENERIC_RCU_GUP
+	select ARCH_HAS_DMA_SET_COHERENT_MASK
 
 config GENERIC_CSUM
 	def_bool CPU_LITTLE_ENDIAN
diff --git a/arch/powerpc/include/asm/device.h b/arch/powerpc/include/asm/device.h
index 9f1371b..9ce2199 100644
--- a/arch/powerpc/include/asm/device.h
+++ b/arch/powerpc/include/asm/device.h
@@ -10,6 +10,7 @@ struct dma_map_ops;
 struct device_node;
 #ifdef CONFIG_PPC64
 struct pci_dn;
+struct iommu_table;
 #endif
 
 /*
@@ -23,13 +24,15 @@ struct dev_archdata {
 	struct dma_map_ops	*dma_ops;
 
 	/*
-	 * When an iommu is in use, dma_data is used as a ptr to the base of the
-	 * iommu_table.  Otherwise, it is a simple numerical offset.
+	 * These two used to be a union. However, with the hybrid ops we need
+	 * both so here we store both a DMA offset for direct mappings and
+	 * an iommu_table for remapped DMA.
 	 */
-	union {
-		dma_addr_t	dma_offset;
-		void		*iommu_table_base;
-	} dma_data;
+	dma_addr_t		dma_offset;
+
+#ifdef CONFIG_PPC64
+	struct iommu_table	*iommu_table_base;
+#endif
 
 #ifdef CONFIG_IOMMU_API
 	void			*iommu_domain;
diff --git a/arch/powerpc/include/asm/dma-mapping.h b/arch/powerpc/include/asm/dma-mapping.h
index 894d538..2e12366 100644
--- a/arch/powerpc/include/asm/dma-mapping.h
+++ b/arch/powerpc/include/asm/dma-mapping.h
@@ -21,12 +21,12 @@
 #define DMA_ERROR_CODE		(~(dma_addr_t)0x0)
 
 /* Some dma direct funcs must be visible for use in other dma_ops */
-extern void *dma_direct_alloc_coherent(struct device *dev, size_t size,
-				       dma_addr_t *dma_handle, gfp_t flag,
+extern void *__dma_direct_alloc_coherent(struct device *dev, size_t size,
+					 dma_addr_t *dma_handle, gfp_t flag,
+					 struct dma_attrs *attrs);
+extern void __dma_direct_free_coherent(struct device *dev, size_t size,
+				       void *vaddr, dma_addr_t dma_handle,
 				       struct dma_attrs *attrs);
-extern void dma_direct_free_coherent(struct device *dev, size_t size,
-				     void *vaddr, dma_addr_t dma_handle,
-				     struct dma_attrs *attrs);
 extern int dma_direct_mmap_coherent(struct device *dev,
 				    struct vm_area_struct *vma,
 				    void *cpu_addr, dma_addr_t handle,
@@ -106,7 +106,7 @@ static inline void set_dma_ops(struct device *dev, struct dma_map_ops *ops)
 static inline dma_addr_t get_dma_offset(struct device *dev)
 {
 	if (dev)
-		return dev->archdata.dma_data.dma_offset;
+		return dev->archdata.dma_offset;
 
 	return PCI_DRAM_OFFSET;
 }
@@ -114,7 +114,7 @@ static inline dma_addr_t get_dma_offset(struct device *dev)
 static inline void set_dma_offset(struct device *dev, dma_addr_t off)
 {
 	if (dev)
-		dev->archdata.dma_data.dma_offset = off;
+		dev->archdata.dma_offset = off;
 }
 
 /* this will be removed soon */
diff --git a/arch/powerpc/include/asm/iommu.h b/arch/powerpc/include/asm/iommu.h
index e2abbe8..e67e678 100644
--- a/arch/powerpc/include/asm/iommu.h
+++ b/arch/powerpc/include/asm/iommu.h
@@ -2,17 +2,17 @@
  * Copyright (C) 2001 Mike Corrigan & Dave Engebretsen, IBM Corporation
  * Rewrite, cleanup:
  * Copyright (C) 2004 Olof Johansson <olof@lixom.net>, IBM Corporation
- * 
+ *
  * This program is free software; you can redistribute it and/or modify
  * it under the terms of the GNU General Public License as published by
  * the Free Software Foundation; either version 2 of the License, or
  * (at your option) any later version.
- * 
+ *
  * This program is distributed in the hope that it will be useful,
  * but WITHOUT ANY WARRANTY; without even the implied warranty of
  * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
  * GNU General Public License for more details.
- * 
+ *
  * You should have received a copy of the GNU General Public License
  * along with this program; if not, write to the Free Software
  * Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307 USA
@@ -93,16 +93,21 @@ int get_iommu_order(unsigned long size, struct iommu_table *tbl)
 
 struct scatterlist;
 
-static inline void set_iommu_table_base(struct device *dev, void *base)
+#ifdef CONFIG_PPC64
+
+static inline void set_iommu_table_base(struct device *dev,
+					struct iommu_table *base)
 {
-	dev->archdata.dma_data.iommu_table_base = base;
+	dev->archdata.iommu_table_base = base;
 }
 
 static inline void *get_iommu_table_base(struct device *dev)
 {
-	return dev->archdata.dma_data.iommu_table_base;
+	return dev->archdata.iommu_table_base;
 }
 
+extern int dma_iommu_dma_supported(struct device *dev, u64 mask);
+
 /* Frees table for an individual device node */
 extern void iommu_free_table(struct iommu_table *tbl, const char *node_name);
 
@@ -146,6 +151,20 @@ static inline void set_iommu_table_base_and_group(struct device *dev,
 	iommu_add_device(dev);
 }
 
+#else
+
+static inline void *get_iommu_table_base(struct device *dev)
+{
+	return NULL;
+}
+
+static inline int dma_iommu_dma_supported(struct device *dev, u64 mask)
+{
+	return 0;
+}
+
+#endif /* CONFIG_PPC64 */
+
 extern int ppc_iommu_map_sg(struct device *dev, struct iommu_table *tbl,
 			    struct scatterlist *sglist, int nelems,
 			    unsigned long mask,
diff --git a/arch/powerpc/kernel/dma-iommu.c b/arch/powerpc/kernel/dma-iommu.c
index 4c68bfe..41a7d9d 100644
--- a/arch/powerpc/kernel/dma-iommu.c
+++ b/arch/powerpc/kernel/dma-iommu.c
@@ -73,7 +73,7 @@ static void dma_iommu_unmap_sg(struct device *dev, struct scatterlist *sglist,
 }
 
 /* We support DMA to/from any memory page via the iommu */
-static int dma_iommu_dma_supported(struct device *dev, u64 mask)
+int dma_iommu_dma_supported(struct device *dev, u64 mask)
 {
 	struct iommu_table *tbl = get_iommu_table_base(dev);
 
diff --git a/arch/powerpc/kernel/dma-swiotlb.c b/arch/powerpc/kernel/dma-swiotlb.c
index 7359797..81b81cd 100644
--- a/arch/powerpc/kernel/dma-swiotlb.c
+++ b/arch/powerpc/kernel/dma-swiotlb.c
@@ -47,8 +47,8 @@ static u64 swiotlb_powerpc_get_required(struct device *dev)
  * for everything else.
  */
 struct dma_map_ops swiotlb_dma_ops = {
-	.alloc = dma_direct_alloc_coherent,
-	.free = dma_direct_free_coherent,
+	.alloc = __dma_direct_alloc_coherent,
+	.free = __dma_direct_free_coherent,
 	.mmap = dma_direct_mmap_coherent,
 	.map_sg = swiotlb_map_sg_attrs,
 	.unmap_sg = swiotlb_unmap_sg_attrs,
diff --git a/arch/powerpc/kernel/dma.c b/arch/powerpc/kernel/dma.c
index 484b2d4..b0c4faa 100644
--- a/arch/powerpc/kernel/dma.c
+++ b/arch/powerpc/kernel/dma.c
@@ -16,6 +16,7 @@
 #include <asm/bug.h>
 #include <asm/machdep.h>
 #include <asm/swiotlb.h>
+#include <asm/iommu.h>
 
 /*
  * Generic direct DMA implementation
@@ -39,9 +40,31 @@ static u64 __maybe_unused get_pfn_limit(struct device *dev)
 	return pfn;
 }
 
-void *dma_direct_alloc_coherent(struct device *dev, size_t size,
-				dma_addr_t *dma_handle, gfp_t flag,
-				struct dma_attrs *attrs)
+static int dma_direct_dma_supported(struct device *dev, u64 mask)
+{
+#ifdef CONFIG_PPC64
+	u64 limit = get_dma_offset(dev) + (memblock_end_of_DRAM() - 1);
+
+	/* Limit fits in the mask, we are good */
+	if (mask >= limit)
+		return 1;
+
+#ifdef CONFIG_FSL_SOC
+	/* Freescale gets another chance via ZONE_DMA/ZONE_DMA32, however
+	 * that will have to be refined if/when they support iommus
+	 */
+	return 1;
+#endif
+	/* Sorry ... */
+	return 0;
+#else
+	return 1;
+#endif
+}
+
+void *__dma_direct_alloc_coherent(struct device *dev, size_t size,
+				  dma_addr_t *dma_handle, gfp_t flag,
+				  struct dma_attrs *attrs)
 {
 	void *ret;
 #ifdef CONFIG_NOT_COHERENT_CACHE
@@ -96,9 +119,9 @@ void *dma_direct_alloc_coherent(struct device *dev, size_t size,
 #endif
 }
 
-void dma_direct_free_coherent(struct device *dev, size_t size,
-			      void *vaddr, dma_addr_t dma_handle,
-			      struct dma_attrs *attrs)
+void __dma_direct_free_coherent(struct device *dev, size_t size,
+				void *vaddr, dma_addr_t dma_handle,
+				struct dma_attrs *attrs)
 {
 #ifdef CONFIG_NOT_COHERENT_CACHE
 	__dma_free_coherent(size, vaddr);
@@ -107,6 +130,51 @@ void dma_direct_free_coherent(struct device *dev, size_t size,
 #endif
 }
 
+static void *dma_direct_alloc_coherent(struct device *dev, size_t size,
+				       dma_addr_t *dma_handle, gfp_t flag,
+				       struct dma_attrs *attrs)
+{
+	struct iommu_table *iommu;
+
+	/* The coherent mask may be smaller than the real mask, check if
+	 * we can really use the direct ops
+	 */
+	if (dma_direct_dma_supported(dev, dev->coherent_dma_mask))
+		return __dma_direct_alloc_coherent(dev, size, dma_handle,
+						   flag, attrs);
+
+	/* Ok we can't ... do we have an iommu ? If not, fail */
+	iommu = get_iommu_table_base(dev);
+	if (!iommu)
+		return NULL;
+
+	/* Try to use the iommu */
+	return iommu_alloc_coherent(dev, iommu, size, dma_handle,
+				    dev->coherent_dma_mask, flag,
+				    dev_to_node(dev));
+}
+
+static void dma_direct_free_coherent(struct device *dev, size_t size,
+				     void *vaddr, dma_addr_t dma_handle,
+				     struct dma_attrs *attrs)
+{
+	struct iommu_table *iommu;
+
+	/* See comments in dma_direct_alloc_coherent() */
+	if (dma_direct_dma_supported(dev, dev->coherent_dma_mask))
+		return __dma_direct_free_coherent(dev, size, vaddr, dma_handle,
+						  attrs);
+	/* Maybe we used an iommu ... */
+	iommu = get_iommu_table_base(dev);
+
+	/* If we hit that we should have never allocated in the first
+	 * place so how come we are freeing ?
+	 */
+	if (WARN_ON(!iommu))
+		return;
+	iommu_free_coherent(iommu, size, vaddr, dma_handle);
+}
+
 int dma_direct_mmap_coherent(struct device *dev, struct vm_area_struct *vma,
 			     void *cpu_addr, dma_addr_t handle, size_t size,
 			     struct dma_attrs *attrs)
@@ -147,18 +215,6 @@ static void dma_direct_unmap_sg(struct device *dev, struct scatterlist *sg,
 {
 }
 
-static int dma_direct_dma_supported(struct device *dev, u64 mask)
-{
-#ifdef CONFIG_PPC64
-	/* Could be improved so platforms can set the limit in case
-	 * they have limited DMA windows
-	 */
-	return mask >= get_dma_offset(dev) + (memblock_end_of_DRAM() - 1);
-#else
-	return 1;
-#endif
-}
-
 static u64 dma_direct_get_required_mask(struct device *dev)
 {
 	u64 end, mask;
@@ -230,6 +286,25 @@ struct dma_map_ops dma_direct_ops = {
 };
 EXPORT_SYMBOL(dma_direct_ops);
 
+int dma_set_coherent_mask(struct device *dev, u64 mask)
+{
+	if (!dma_supported(dev, mask)) {
+		/*
+		 * We need to special case the direct DMA ops which can
+		 * support a fallback for coherent allocations. There
+		 * is no dma_op->set_coherent_mask() so we have to do
+		 * things the hard way:
+		 */
+		if (get_dma_ops(dev) != &dma_direct_ops ||
+		    get_iommu_table_base(dev) == NULL ||
+		    !dma_iommu_dma_supported(dev, mask))
+			return -EIO;
+	}
+	dev->coherent_dma_mask = mask;
+	return 0;
+}
+EXPORT_SYMBOL_GPL(dma_set_coherent_mask);
+
 #define PREALLOC_DMA_DEBUG_ENTRIES (1 << 16)
 
 int __dma_set_mask(struct device *dev, u64 dma_mask)
diff --git a/arch/powerpc/platforms/powernv/pci-ioda.c b/arch/powerpc/platforms/powernv/pci-ioda.c
index 5ac7c60..fcff6fa 100644
--- a/arch/powerpc/platforms/powernv/pci-ioda.c
+++ b/arch/powerpc/platforms/powernv/pci-ioda.c
@@ -1625,7 +1625,6 @@ static int pnv_pci_ioda_dma_set_mask(struct pnv_phb *phb,
 	} else {
 		dev_info(&pdev->dev, "Using 32-bit DMA via iommu\n");
 		set_dma_ops(&pdev->dev, &dma_iommu_ops);
-		set_iommu_table_base(&pdev->dev, pe->tce32_table);
 	}
 	*pdev->dev.dma_mask = dma_mask;
 	return 0;
diff --git a/arch/powerpc/platforms/pseries/iommu.c b/arch/powerpc/platforms/pseries/iommu.c
index 7803a19..0acab86 100644
--- a/arch/powerpc/platforms/pseries/iommu.c
+++ b/arch/powerpc/platforms/pseries/iommu.c
@@ -1161,11 +1161,10 @@ static int dma_set_mask_pSeriesLP(struct device *dev, u64 dma_mask)
 		}
 	}
 
-	/* fall back on iommu ops, restore table pointer with ops */
+	/* fall back on iommu ops */
 	if (!ddw_enabled && get_dma_ops(dev) != &dma_iommu_ops) {
 		dev_info(dev, "Restoring 32-bit DMA via iommu\n");
 		set_dma_ops(dev, &dma_iommu_ops);
-		pci_dma_dev_setup_pSeriesLP(pdev);
 	}
 
 check_mask:
diff --git a/arch/powerpc/sysdev/dart_iommu.c b/arch/powerpc/sysdev/dart_iommu.c
index 9e5353f..6cf033d 100644
--- a/arch/powerpc/sysdev/dart_iommu.c
+++ b/arch/powerpc/sysdev/dart_iommu.c
@@ -306,20 +306,11 @@ static void iommu_table_dart_setup(void)
 	set_bit(iommu_table_dart.it_size - 1, iommu_table_dart.it_map);
 }
 
-static void dma_dev_setup_dart(struct device *dev)
-{
-	/* We only have one iommu table on the mac for now, which makes
-	 * things simple. Setup all PCI devices to point to this table
-	 */
-	if (get_dma_ops(dev) == &dma_direct_ops)
-		set_dma_offset(dev, DART_U4_BYPASS_BASE);
-	else
-		set_iommu_table_base(dev, &iommu_table_dart);
-}
-
 static void pci_dma_dev_setup_dart(struct pci_dev *dev)
 {
-	dma_dev_setup_dart(&dev->dev);
+	if (dart_is_u4)
+		set_dma_offset(&dev->dev, DART_U4_BYPASS_BASE);
+	set_iommu_table_base(&dev->dev, &iommu_table_dart);
 }
 
 static void pci_dma_bus_setup_dart(struct pci_bus *bus)
@@ -363,7 +354,6 @@ static int dart_dma_set_mask(struct device *dev, u64 dma_mask)
 		dev_info(dev, "Using 32-bit DMA via iommu\n");
 		set_dma_ops(dev, &dma_iommu_ops);
 	}
-	dma_dev_setup_dart(dev);
 
 	*dev->dma_mask = dma_mask;
 	return 0;

Re: [RFC/PATCH v3] powerpc/iommu: Support "hybrid" iommu/direct DMA ops for coherent_mask < dma_mask

[Brian King]

On 05/14/2015 05:03 PM, Benjamin Herrenschmidt wrote:
> This patch adds the ability to the DMA direct ops to fallback to the IOMMU
> ops for coherent alloc/free if the coherent mask of the device isn't
> suitable for accessing the direct DMA space and the device also happens
> to have an active IOMMU table.
> 
> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
> ---
> 

Thanks. This one works fine for me with mpt2sas.

Tested-by: Brian King <brking@linux.vnet.ibm.com>

-Brian

-- 
Brian King
Power Linux I/O
IBM Linux Technology Center

Re: [RFC/PATCH v3] powerpc/iommu: Support "hybrid" iommu/direct DMA ops for coherent_mask < dma_mask

[Brian King]

On 05/14/2015 05:03 PM, Benjamin Herrenschmidt wrote:
> This patch adds the ability to the DMA direct ops to fallback to the IOMMU
> ops for coherent alloc/free if the coherent mask of the device isn't
> suitable for accessing the direct DMA space and the device also happens
> to have an active IOMMU table.
> 
> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
> ---
> 

Thanks. This one works fine for me with mpt2sas.

Tested-by: Brian King <brking@linux.vnet.ibm.com>

-Brian

-- 
Brian King
Power Linux I/O
IBM Linux Technology Center

[PATCH] powerpc/iommu: Support "hybrid" iommu/direct DMA ops for coherent_mask < dma_mask

[Benjamin Herrenschmidt]

This patch adds the ability to the DMA direct ops to fallback to the IOMMU
ops for coherent alloc/free if the coherent mask of the device isn't
suitable for accessing the direct DMA space and the device also happens
to have an active IOMMU table.

Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Tested-by: Brian King <brking@linux.vnet.ibm.com>
---

Identical to RFC v3

diff --git a/arch/powerpc/Kconfig b/arch/powerpc/Kconfig
index 22b0940..cd46474 100644
--- a/arch/powerpc/Kconfig
+++ b/arch/powerpc/Kconfig
@@ -82,6 +82,9 @@ config GENERIC_HWEIGHT
 	bool
 	default y
 
+config ARCH_HAS_DMA_SET_COHERENT_MASK
+        bool
+
 config PPC
 	bool
 	default y
@@ -152,6 +155,7 @@ config PPC
 	select DCACHE_WORD_ACCESS if PPC64 && CPU_LITTLE_ENDIAN
 	select NO_BOOTMEM
 	select HAVE_GENERIC_RCU_GUP
+	select ARCH_HAS_DMA_SET_COHERENT_MASK
 
 config GENERIC_CSUM
 	def_bool CPU_LITTLE_ENDIAN
diff --git a/arch/powerpc/include/asm/device.h b/arch/powerpc/include/asm/device.h
index 9f1371b..9ce2199 100644
--- a/arch/powerpc/include/asm/device.h
+++ b/arch/powerpc/include/asm/device.h
@@ -10,6 +10,7 @@ struct dma_map_ops;
 struct device_node;
 #ifdef CONFIG_PPC64
 struct pci_dn;
+struct iommu_table;
 #endif
 
 /*
@@ -23,13 +24,15 @@ struct dev_archdata {
 	struct dma_map_ops	*dma_ops;
 
 	/*
-	 * When an iommu is in use, dma_data is used as a ptr to the base of the
-	 * iommu_table.  Otherwise, it is a simple numerical offset.
+	 * These two used to be a union. However, with the hybrid ops we need
+	 * both so here we store both a DMA offset for direct mappings and
+	 * an iommu_table for remapped DMA.
 	 */
-	union {
-		dma_addr_t	dma_offset;
-		void		*iommu_table_base;
-	} dma_data;
+	dma_addr_t		dma_offset;
+
+#ifdef CONFIG_PPC64
+	struct iommu_table	*iommu_table_base;
+#endif
 
 #ifdef CONFIG_IOMMU_API
 	void			*iommu_domain;
diff --git a/arch/powerpc/include/asm/dma-mapping.h b/arch/powerpc/include/asm/dma-mapping.h
index 894d538..2e12366 100644
--- a/arch/powerpc/include/asm/dma-mapping.h
+++ b/arch/powerpc/include/asm/dma-mapping.h
@@ -21,12 +21,12 @@
 #define DMA_ERROR_CODE		(~(dma_addr_t)0x0)
 
 /* Some dma direct funcs must be visible for use in other dma_ops */
-extern void *dma_direct_alloc_coherent(struct device *dev, size_t size,
-				       dma_addr_t *dma_handle, gfp_t flag,
+extern void *__dma_direct_alloc_coherent(struct device *dev, size_t size,
+					 dma_addr_t *dma_handle, gfp_t flag,
+					 struct dma_attrs *attrs);
+extern void __dma_direct_free_coherent(struct device *dev, size_t size,
+				       void *vaddr, dma_addr_t dma_handle,
 				       struct dma_attrs *attrs);
-extern void dma_direct_free_coherent(struct device *dev, size_t size,
-				     void *vaddr, dma_addr_t dma_handle,
-				     struct dma_attrs *attrs);
 extern int dma_direct_mmap_coherent(struct device *dev,
 				    struct vm_area_struct *vma,
 				    void *cpu_addr, dma_addr_t handle,
@@ -106,7 +106,7 @@ static inline void set_dma_ops(struct device *dev, struct dma_map_ops *ops)
 static inline dma_addr_t get_dma_offset(struct device *dev)
 {
 	if (dev)
-		return dev->archdata.dma_data.dma_offset;
+		return dev->archdata.dma_offset;
 
 	return PCI_DRAM_OFFSET;
 }
@@ -114,7 +114,7 @@ static inline dma_addr_t get_dma_offset(struct device *dev)
 static inline void set_dma_offset(struct device *dev, dma_addr_t off)
 {
 	if (dev)
-		dev->archdata.dma_data.dma_offset = off;
+		dev->archdata.dma_offset = off;
 }
 
 /* this will be removed soon */
diff --git a/arch/powerpc/include/asm/iommu.h b/arch/powerpc/include/asm/iommu.h
index e2abbe8..e67e678 100644
--- a/arch/powerpc/include/asm/iommu.h
+++ b/arch/powerpc/include/asm/iommu.h
@@ -2,17 +2,17 @@
  * Copyright (C) 2001 Mike Corrigan & Dave Engebretsen, IBM Corporation
  * Rewrite, cleanup:
  * Copyright (C) 2004 Olof Johansson <olof@lixom.net>, IBM Corporation
- * 
+ *
  * This program is free software; you can redistribute it and/or modify
  * it under the terms of the GNU General Public License as published by
  * the Free Software Foundation; either version 2 of the License, or
  * (at your option) any later version.
- * 
+ *
  * This program is distributed in the hope that it will be useful,
  * but WITHOUT ANY WARRANTY; without even the implied warranty of
  * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
  * GNU General Public License for more details.
- * 
+ *
  * You should have received a copy of the GNU General Public License
  * along with this program; if not, write to the Free Software
  * Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307 USA
@@ -93,16 +93,21 @@ int get_iommu_order(unsigned long size, struct iommu_table *tbl)
 
 struct scatterlist;
 
-static inline void set_iommu_table_base(struct device *dev, void *base)
+#ifdef CONFIG_PPC64
+
+static inline void set_iommu_table_base(struct device *dev,
+					struct iommu_table *base)
 {
-	dev->archdata.dma_data.iommu_table_base = base;
+	dev->archdata.iommu_table_base = base;
 }
 
 static inline void *get_iommu_table_base(struct device *dev)
 {
-	return dev->archdata.dma_data.iommu_table_base;
+	return dev->archdata.iommu_table_base;
 }
 
+extern int dma_iommu_dma_supported(struct device *dev, u64 mask);
+
 /* Frees table for an individual device node */
 extern void iommu_free_table(struct iommu_table *tbl, const char *node_name);
 
@@ -146,6 +151,20 @@ static inline void set_iommu_table_base_and_group(struct device *dev,
 	iommu_add_device(dev);
 }
 
+#else
+
+static inline void *get_iommu_table_base(struct device *dev)
+{
+	return NULL;
+}
+
+static inline int dma_iommu_dma_supported(struct device *dev, u64 mask)
+{
+	return 0;
+}
+
+#endif /* CONFIG_PPC64 */
+
 extern int ppc_iommu_map_sg(struct device *dev, struct iommu_table *tbl,
 			    struct scatterlist *sglist, int nelems,
 			    unsigned long mask,
diff --git a/arch/powerpc/kernel/dma-iommu.c b/arch/powerpc/kernel/dma-iommu.c
index 4c68bfe..41a7d9d 100644
--- a/arch/powerpc/kernel/dma-iommu.c
+++ b/arch/powerpc/kernel/dma-iommu.c
@@ -73,7 +73,7 @@ static void dma_iommu_unmap_sg(struct device *dev, struct scatterlist *sglist,
 }
 
 /* We support DMA to/from any memory page via the iommu */
-static int dma_iommu_dma_supported(struct device *dev, u64 mask)
+int dma_iommu_dma_supported(struct device *dev, u64 mask)
 {
 	struct iommu_table *tbl = get_iommu_table_base(dev);
 
diff --git a/arch/powerpc/kernel/dma-swiotlb.c b/arch/powerpc/kernel/dma-swiotlb.c
index 7359797..81b81cd 100644
--- a/arch/powerpc/kernel/dma-swiotlb.c
+++ b/arch/powerpc/kernel/dma-swiotlb.c
@@ -47,8 +47,8 @@ static u64 swiotlb_powerpc_get_required(struct device *dev)
  * for everything else.
  */
 struct dma_map_ops swiotlb_dma_ops = {
-	.alloc = dma_direct_alloc_coherent,
-	.free = dma_direct_free_coherent,
+	.alloc = __dma_direct_alloc_coherent,
+	.free = __dma_direct_free_coherent,
 	.mmap = dma_direct_mmap_coherent,
 	.map_sg = swiotlb_map_sg_attrs,
 	.unmap_sg = swiotlb_unmap_sg_attrs,
diff --git a/arch/powerpc/kernel/dma.c b/arch/powerpc/kernel/dma.c
index 484b2d4..b0c4faa 100644
--- a/arch/powerpc/kernel/dma.c
+++ b/arch/powerpc/kernel/dma.c
@@ -16,6 +16,7 @@
 #include <asm/bug.h>
 #include <asm/machdep.h>
 #include <asm/swiotlb.h>
+#include <asm/iommu.h>
 
 /*
  * Generic direct DMA implementation
@@ -39,9 +40,31 @@ static u64 __maybe_unused get_pfn_limit(struct device *dev)
 	return pfn;
 }
 
-void *dma_direct_alloc_coherent(struct device *dev, size_t size,
-				dma_addr_t *dma_handle, gfp_t flag,
-				struct dma_attrs *attrs)
+static int dma_direct_dma_supported(struct device *dev, u64 mask)
+{
+#ifdef CONFIG_PPC64
+	u64 limit = get_dma_offset(dev) + (memblock_end_of_DRAM() - 1);
+
+	/* Limit fits in the mask, we are good */
+	if (mask >= limit)
+		return 1;
+
+#ifdef CONFIG_FSL_SOC
+	/* Freescale gets another chance via ZONE_DMA/ZONE_DMA32, however
+	 * that will have to be refined if/when they support iommus
+	 */
+	return 1;
+#endif
+	/* Sorry ... */
+	return 0;
+#else
+	return 1;
+#endif
+}
+
+void *__dma_direct_alloc_coherent(struct device *dev, size_t size,
+				  dma_addr_t *dma_handle, gfp_t flag,
+				  struct dma_attrs *attrs)
 {
 	void *ret;
 #ifdef CONFIG_NOT_COHERENT_CACHE
@@ -96,9 +119,9 @@ void *dma_direct_alloc_coherent(struct device *dev, size_t size,
 #endif
 }
 
-void dma_direct_free_coherent(struct device *dev, size_t size,
-			      void *vaddr, dma_addr_t dma_handle,
-			      struct dma_attrs *attrs)
+void __dma_direct_free_coherent(struct device *dev, size_t size,
+				void *vaddr, dma_addr_t dma_handle,
+				struct dma_attrs *attrs)
 {
 #ifdef CONFIG_NOT_COHERENT_CACHE
 	__dma_free_coherent(size, vaddr);
@@ -107,6 +130,51 @@ void dma_direct_free_coherent(struct device *dev, size_t size,
 #endif
 }
 
+static void *dma_direct_alloc_coherent(struct device *dev, size_t size,
+				       dma_addr_t *dma_handle, gfp_t flag,
+				       struct dma_attrs *attrs)
+{
+	struct iommu_table *iommu;
+
+	/* The coherent mask may be smaller than the real mask, check if
+	 * we can really use the direct ops
+	 */
+	if (dma_direct_dma_supported(dev, dev->coherent_dma_mask))
+		return __dma_direct_alloc_coherent(dev, size, dma_handle,
+						   flag, attrs);
+
+	/* Ok we can't ... do we have an iommu ? If not, fail */
+	iommu = get_iommu_table_base(dev);
+	if (!iommu)
+		return NULL;
+
+	/* Try to use the iommu */
+	return iommu_alloc_coherent(dev, iommu, size, dma_handle,
+				    dev->coherent_dma_mask, flag,
+				    dev_to_node(dev));
+}
+
+static void dma_direct_free_coherent(struct device *dev, size_t size,
+				     void *vaddr, dma_addr_t dma_handle,
+				     struct dma_attrs *attrs)
+{
+	struct iommu_table *iommu;
+
+	/* See comments in dma_direct_alloc_coherent() */
+	if (dma_direct_dma_supported(dev, dev->coherent_dma_mask))
+		return __dma_direct_free_coherent(dev, size, vaddr, dma_handle,
+						  attrs);
+	/* Maybe we used an iommu ... */
+	iommu = get_iommu_table_base(dev);
+
+	/* If we hit that we should have never allocated in the first
+	 * place so how come we are freeing ?
+	 */
+	if (WARN_ON(!iommu))
+		return;
+	iommu_free_coherent(iommu, size, vaddr, dma_handle);
+}
+
 int dma_direct_mmap_coherent(struct device *dev, struct vm_area_struct *vma,
 			     void *cpu_addr, dma_addr_t handle, size_t size,
 			     struct dma_attrs *attrs)
@@ -147,18 +215,6 @@ static void dma_direct_unmap_sg(struct device *dev, struct scatterlist *sg,
 {
 }
 
-static int dma_direct_dma_supported(struct device *dev, u64 mask)
-{
-#ifdef CONFIG_PPC64
-	/* Could be improved so platforms can set the limit in case
-	 * they have limited DMA windows
-	 */
-	return mask >= get_dma_offset(dev) + (memblock_end_of_DRAM() - 1);
-#else
-	return 1;
-#endif
-}
-
 static u64 dma_direct_get_required_mask(struct device *dev)
 {
 	u64 end, mask;
@@ -230,6 +286,25 @@ struct dma_map_ops dma_direct_ops = {
 };
 EXPORT_SYMBOL(dma_direct_ops);
 
+int dma_set_coherent_mask(struct device *dev, u64 mask)
+{
+	if (!dma_supported(dev, mask)) {
+		/*
+		 * We need to special case the direct DMA ops which can
+		 * support a fallback for coherent allocations. There
+		 * is no dma_op->set_coherent_mask() so we have to do
+		 * things the hard way:
+		 */
+		if (get_dma_ops(dev) != &dma_direct_ops ||
+		    get_iommu_table_base(dev) == NULL ||
+		    !dma_iommu_dma_supported(dev, mask))
+			return -EIO;
+	}
+	dev->coherent_dma_mask = mask;
+	return 0;
+}
+EXPORT_SYMBOL_GPL(dma_set_coherent_mask);
+
 #define PREALLOC_DMA_DEBUG_ENTRIES (1 << 16)
 
 int __dma_set_mask(struct device *dev, u64 dma_mask)
diff --git a/arch/powerpc/platforms/powernv/pci-ioda.c b/arch/powerpc/platforms/powernv/pci-ioda.c
index 5ac7c60..fcff6fa 100644
--- a/arch/powerpc/platforms/powernv/pci-ioda.c
+++ b/arch/powerpc/platforms/powernv/pci-ioda.c
@@ -1625,7 +1625,6 @@ static int pnv_pci_ioda_dma_set_mask(struct pnv_phb *phb,
 	} else {
 		dev_info(&pdev->dev, "Using 32-bit DMA via iommu\n");
 		set_dma_ops(&pdev->dev, &dma_iommu_ops);
-		set_iommu_table_base(&pdev->dev, pe->tce32_table);
 	}
 	*pdev->dev.dma_mask = dma_mask;
 	return 0;
diff --git a/arch/powerpc/platforms/pseries/iommu.c b/arch/powerpc/platforms/pseries/iommu.c
index 7803a19..0acab86 100644
--- a/arch/powerpc/platforms/pseries/iommu.c
+++ b/arch/powerpc/platforms/pseries/iommu.c
@@ -1161,11 +1161,10 @@ static int dma_set_mask_pSeriesLP(struct device *dev, u64 dma_mask)
 		}
 	}
 
-	/* fall back on iommu ops, restore table pointer with ops */
+	/* fall back on iommu ops */
 	if (!ddw_enabled && get_dma_ops(dev) != &dma_iommu_ops) {
 		dev_info(dev, "Restoring 32-bit DMA via iommu\n");
 		set_dma_ops(dev, &dma_iommu_ops);
-		pci_dma_dev_setup_pSeriesLP(pdev);
 	}
 
 check_mask:
diff --git a/arch/powerpc/sysdev/dart_iommu.c b/arch/powerpc/sysdev/dart_iommu.c
index 9e5353f..6cf033d 100644
--- a/arch/powerpc/sysdev/dart_iommu.c
+++ b/arch/powerpc/sysdev/dart_iommu.c
@@ -306,20 +306,11 @@ static void iommu_table_dart_setup(void)
 	set_bit(iommu_table_dart.it_size - 1, iommu_table_dart.it_map);
 }
 
-static void dma_dev_setup_dart(struct device *dev)
-{
-	/* We only have one iommu table on the mac for now, which makes
-	 * things simple. Setup all PCI devices to point to this table
-	 */
-	if (get_dma_ops(dev) == &dma_direct_ops)
-		set_dma_offset(dev, DART_U4_BYPASS_BASE);
-	else
-		set_iommu_table_base(dev, &iommu_table_dart);
-}
-
 static void pci_dma_dev_setup_dart(struct pci_dev *dev)
 {
-	dma_dev_setup_dart(&dev->dev);
+	if (dart_is_u4)
+		set_dma_offset(&dev->dev, DART_U4_BYPASS_BASE);
+	set_iommu_table_base(&dev->dev, &iommu_table_dart);
 }
 
 static void pci_dma_bus_setup_dart(struct pci_bus *bus)
@@ -363,7 +354,6 @@ static int dart_dma_set_mask(struct device *dev, u64 dma_mask)
 		dev_info(dev, "Using 32-bit DMA via iommu\n");
 		set_dma_ops(dev, &dma_iommu_ops);
 	}
-	dma_dev_setup_dart(dev);
 
 	*dev->dma_mask = dma_mask;
 	return 0;

Re: powerpc/iommu: Support "hybrid" iommu/direct DMA ops for coherent_mask < dma_mask

[Michael Ellerman]

On Mon, 2015-18-05 at 03:56:51 UTC, Benjamin Herrenschmidt wrote:
> This patch adds the ability to the DMA direct ops to fallback to the IOMMU
> ops for coherent alloc/free if the coherent mask of the device isn't
> suitable for accessing the direct DMA space and the device also happens
> to have an active IOMMU table.

Can you do the removal of the union, the #ifdef PPC64 and the static inlines as
a precursor patch, that would remove some of the noise in the diff.

And can you explain the changes to dart, pseries and powernv.

There's also some whitespace changes in iommu.h that I assume you didn't want?

cheers

Re: powerpc/iommu: Support "hybrid" iommu/direct DMA ops for coherent_mask < dma_mask

[Brian King]

On 05/18/2015 01:40 AM, Michael Ellerman wrote:
> On Mon, 2015-18-05 at 03:56:51 UTC, Benjamin Herrenschmidt wrote:
>> This patch adds the ability to the DMA direct ops to fallback to the IOMMU
>> ops for coherent alloc/free if the coherent mask of the device isn't
>> suitable for accessing the direct DMA space and the device also happens
>> to have an active IOMMU table.
> 
> Can you do the removal of the union, the #ifdef PPC64 and the static inlines as
> a precursor patch, that would remove some of the noise in the diff.
> 
> And can you explain the changes to dart, pseries and powernv.
> 
> There's also some whitespace changes in iommu.h that I assume you didn't want?

Hi Ben,

Do you plan to resubmit with these changes? The patch seems to work fine in
my testing. Would be nice to get this merged.

Thanks,

Brian

-- 
Brian King
Power Linux I/O
IBM Linux Technology Center

Re: powerpc/iommu: Support "hybrid" iommu/direct DMA ops for coherent_mask < dma_mask

[Benjamin Herrenschmidt]

On Fri, 2015-06-19 at 16:19 -0500, Brian King wrote:
> On 05/18/2015 01:40 AM, Michael Ellerman wrote:
> > On Mon, 2015-18-05 at 03:56:51 UTC, Benjamin Herrenschmidt wrote:
> >> This patch adds the ability to the DMA direct ops to fallback to the IOMMU
> >> ops for coherent alloc/free if the coherent mask of the device isn't
> >> suitable for accessing the direct DMA space and the device also happens
> >> to have an active IOMMU table.
> > 
> > Can you do the removal of the union, the #ifdef PPC64 and the static inlines as
> > a precursor patch, that would remove some of the noise in the diff.
> > 
> > And can you explain the changes to dart, pseries and powernv.
> > 
> > There's also some whitespace changes in iommu.h that I assume you didn't want?
> 
> Hi Ben,
> 
> Do you plan to resubmit with these changes? The patch seems to work fine in
> my testing. Would be nice to get this merged.

Argh, completely forgot about it. I'll respin it next week.

> Thanks,
> 
> Brian
> 

Re: powerpc/iommu: Support "hybrid" iommu/direct DMA ops for coherent_mask < dma_mask

[Brian King]

On 06/19/2015 06:01 PM, Benjamin Herrenschmidt wrote:
> On Fri, 2015-06-19 at 16:19 -0500, Brian King wrote:
>> On 05/18/2015 01:40 AM, Michael Ellerman wrote:
>>> On Mon, 2015-18-05 at 03:56:51 UTC, Benjamin Herrenschmidt wrote:
>>>> This patch adds the ability to the DMA direct ops to fallback to the IOMMU
>>>> ops for coherent alloc/free if the coherent mask of the device isn't
>>>> suitable for accessing the direct DMA space and the device also happens
>>>> to have an active IOMMU table.
>>>
>>> Can you do the removal of the union, the #ifdef PPC64 and the static inlines as
>>> a precursor patch, that would remove some of the noise in the diff.
>>>
>>> And can you explain the changes to dart, pseries and powernv.
>>>
>>> There's also some whitespace changes in iommu.h that I assume you didn't want?
>>
>> Hi Ben,
>>
>> Do you plan to resubmit with these changes? The patch seems to work fine in
>> my testing. Would be nice to get this merged.
> 
> Argh, completely forgot about it. I'll respin it next week.

Anything I can do to help on this one?

Thanks,

Brian

-- 
Brian King
Power Linux I/O
IBM Linux Technology Center