[PATCH kernel] powerpc/powernv: Fix stale iommu table base after VFIO

[PATCH kernel] powerpc/powernv: Fix stale iommu table base after VFIO

[Alexey Kardashevskiy]

The powernv platform uses @dma_iommu_ops for non-bypass DMA. These ops
need an iommu_table pointer which is stored in
dev->archdata.iommu_table_base. It is initialized during
pcibios_setup_device() which handles boot time devices. However when
a device is taken from the system in order to pass it through, the default
IOMMU table is destroyed but the pointer in a device is not updated;
also when a device is returned back to the system, a new table pointer
is not stored in dev->archdata.iommu_table_base either. So when a just
returned device tries using IOMMU, it crashes on accessing stale
iommu_table or its members.

This calls set_iommu_table_base() when the default window is created.
Note it used to be there before but was wrongly removed (see "fixes").
It did not appear before as these days most devices simply use bypass.

This adds set_iommu_table_base(NULL) when a device is taken from
the system to make it clear that IOMMU DMA cannot be used past that point.

Fixes: c4e9d3c1e65a ("powerpc/powernv/pseries: Rework device adding to IOMMU groups")
Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
---

This needs to be applied before
https://patchwork.ozlabs.org/project/linuxppc-dev/list/?series=110810
(discovered while testing this on POWER8)
---
 arch/powerpc/platforms/powernv/pci-ioda.c | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/arch/powerpc/platforms/powernv/pci-ioda.c b/arch/powerpc/platforms/powernv/pci-ioda.c
index 10cc42b9e541..0f72c7484824 100644
--- a/arch/powerpc/platforms/powernv/pci-ioda.c
+++ b/arch/powerpc/platforms/powernv/pci-ioda.c
@@ -2456,6 +2456,14 @@ static long pnv_pci_ioda2_setup_default_config(struct pnv_ioda_pe *pe)
 	if (!pnv_iommu_bypass_disabled)
 		pnv_pci_ioda2_set_bypass(pe, true);
 
+	/*
+	 * Set table base for the case of IOMMU DMA use. Usually this is done
+	 * from dma_dev_setup() which is not called when a device is returned
+	 * from VFIO so do it here.
+	 */
+	if (pe->pdev)
+		set_iommu_table_base(&pe->pdev->dev, tbl);
+
 	return 0;
 }
 
@@ -2543,6 +2551,8 @@ static void pnv_ioda2_take_ownership(struct iommu_table_group *table_group)
 	pnv_pci_ioda2_unset_window(&pe->table_group, 0);
 	if (pe->pbus)
 		pnv_ioda_setup_bus_dma(pe, pe->pbus);
+	else if (pe->pdev)
+		set_iommu_table_base(&pe->pdev->dev, NULL);
 	iommu_tce_table_put(tbl);
 }
 
-- 
2.17.1

Re: [PATCH kernel] powerpc/powernv: Fix stale iommu table base after VFIO

[Michael Ellerman]

On Fri, 2019-06-28 at 06:53:00 UTC, Alexey Kardashevskiy wrote:
> The powernv platform uses @dma_iommu_ops for non-bypass DMA. These ops
> need an iommu_table pointer which is stored in
> dev->archdata.iommu_table_base. It is initialized during
> pcibios_setup_device() which handles boot time devices. However when
> a device is taken from the system in order to pass it through, the default
> IOMMU table is destroyed but the pointer in a device is not updated;
> also when a device is returned back to the system, a new table pointer
> is not stored in dev->archdata.iommu_table_base either. So when a just
> returned device tries using IOMMU, it crashes on accessing stale
> iommu_table or its members.
> 
> This calls set_iommu_table_base() when the default window is created.
> Note it used to be there before but was wrongly removed (see "fixes").
> It did not appear before as these days most devices simply use bypass.
> 
> This adds set_iommu_table_base(NULL) when a device is taken from
> the system to make it clear that IOMMU DMA cannot be used past that point.
> 
> Fixes: c4e9d3c1e65a ("powerpc/powernv/pseries: Rework device adding to IOMMU groups")
> Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>

Applied to powerpc next, thanks.

https://git.kernel.org/powerpc/c/5636427d087a55842c1a199dfb839e6545d30e5d

cheers