[PATCH kernel 0/2] pseries: Enable SWIOTLB

[PATCH kernel 0/2] pseries: Enable SWIOTLB

[Alexey Kardashevskiy]

This is an attempt to allow PCI pass through to a secure guest when
hardware can only access insecure memory. This allows SWIOTLB use
for passed through devices.

Later on secure VMs will unsecure SWIOTLB bounce buffers for DMA
and the rest of the guest RAM will be unavailable to the hardware
by default.


This is based on sha1
e93c9c99a629 Linus Torvalds "Linux 5.1".

Please comment. Thanks.



Alexey Kardashevskiy (2):
  powerpc/pseries/dma: Allow swiotlb
  powerpc/pseries/dma: Enable swiotlb

 arch/powerpc/kernel/dma-iommu.c        | 36 ++++++++++++++++++++++++++
 arch/powerpc/platforms/pseries/setup.c |  5 ++++
 arch/powerpc/platforms/pseries/Kconfig |  1 +
 3 files changed, 42 insertions(+)

-- 
2.17.1

[PATCH kernel 1/2] powerpc/pseries/dma: Allow swiotlb

[Alexey Kardashevskiy]

The commit 8617a5c5bc00 ("powerpc/dma: handle iommu bypass in
dma_iommu_ops") merged direct DMA ops into the IOMMU DMA ops allowing
SWIOTLB as well but only for mapping; the unmapping and bouncing parts
were left unmodified.

This adds missing direct unmapping calls to .unmap_page() and .unmap_sg().

This adds missing sync callbacks and directs them to the direct DMA hooks.

Fixes: 8617a5c5bc00 (powerpc/dma: handle iommu bypass in dma_iommu_ops)
Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
---
 arch/powerpc/kernel/dma-iommu.c | 36 +++++++++++++++++++++++++++++++++
 1 file changed, 36 insertions(+)

diff --git a/arch/powerpc/kernel/dma-iommu.c b/arch/powerpc/kernel/dma-iommu.c
index 09231ef06d01..92b318df1aa1 100644
--- a/arch/powerpc/kernel/dma-iommu.c
+++ b/arch/powerpc/kernel/dma-iommu.c
@@ -82,6 +82,8 @@ static void dma_iommu_unmap_page(struct device *dev, dma_addr_t dma_handle,
 	if (!dma_iommu_map_bypass(dev, attrs))
 		iommu_unmap_page(get_iommu_table_base(dev), dma_handle, size,
 				direction,  attrs);
+	else
+		dma_direct_unmap_page(dev, dma_handle, size, direction, attrs);
 }
 
 
@@ -102,6 +104,8 @@ static void dma_iommu_unmap_sg(struct device *dev, struct scatterlist *sglist,
 	if (!dma_iommu_map_bypass(dev, attrs))
 		ppc_iommu_unmap_sg(get_iommu_table_base(dev), sglist, nelems,
 			   direction, attrs);
+	else
+		dma_direct_unmap_sg(dev, sglist, nelems, direction, attrs);
 }
 
 static bool dma_iommu_bypass_supported(struct device *dev, u64 mask)
@@ -163,6 +167,34 @@ u64 dma_iommu_get_required_mask(struct device *dev)
 	return mask;
 }
 
+static void dma_iommu_sync_for_cpu(struct device *dev, dma_addr_t addr,
+		size_t size, enum dma_data_direction dir)
+{
+	if (dma_iommu_alloc_bypass(dev))
+		dma_direct_sync_single_for_cpu(dev, addr, size, dir);
+}
+
+static void dma_iommu_sync_for_device(struct device *dev, dma_addr_t addr,
+		size_t sz, enum dma_data_direction dir)
+{
+	if (dma_iommu_alloc_bypass(dev))
+		dma_direct_sync_single_for_device(dev, addr, sz, dir);
+}
+
+extern void dma_iommu_sync_sg_for_cpu(struct device *dev,
+		struct scatterlist *sgl, int nents, enum dma_data_direction dir)
+{
+	if (dma_iommu_alloc_bypass(dev))
+		dma_direct_sync_sg_for_cpu(dev, sgl, nents, dir);
+}
+
+extern void dma_iommu_sync_sg_for_device(struct device *dev,
+		struct scatterlist *sgl, int nents, enum dma_data_direction dir)
+{
+	if (dma_iommu_alloc_bypass(dev))
+		dma_direct_sync_sg_for_device(dev, sgl, nents, dir);
+}
+
 const struct dma_map_ops dma_iommu_ops = {
 	.alloc			= dma_iommu_alloc_coherent,
 	.free			= dma_iommu_free_coherent,
@@ -172,4 +204,8 @@ const struct dma_map_ops dma_iommu_ops = {
 	.map_page		= dma_iommu_map_page,
 	.unmap_page		= dma_iommu_unmap_page,
 	.get_required_mask	= dma_iommu_get_required_mask,
+	.sync_single_for_cpu	= dma_iommu_sync_for_cpu,
+	.sync_single_for_device	= dma_iommu_sync_for_device,
+	.sync_sg_for_cpu	= dma_iommu_sync_sg_for_cpu,
+	.sync_sg_for_device	= dma_iommu_sync_sg_for_device,
 };
-- 
2.17.1

[PATCH kernel 2/2] powerpc/pseries/dma: Enable swiotlb

[Alexey Kardashevskiy]

So far the pseries platforms has always been using IOMMU making SWIOTLB
unnecessary. Now we want secure guests which means devices can only
access certain areas of guest physical memory; we are going to use
SWIOTLB for this purpose.

This allows SWIOTLB for pseries. By default there is no change in behavior.

This enables SWIOTLB when the "swiotlb" kernel parameter is set to "force".

With the SWIOTLB enabled, the kernel creates a directly mapped DMA window
(using the usual DDW mechanism) and implements SWIOTLB on top of that.

Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
---
 arch/powerpc/platforms/pseries/setup.c | 5 +++++
 arch/powerpc/platforms/pseries/Kconfig | 1 +
 2 files changed, 6 insertions(+)

diff --git a/arch/powerpc/platforms/pseries/setup.c b/arch/powerpc/platforms/pseries/setup.c
index e4f0dfd4ae33..30d72b587ac5 100644
--- a/arch/powerpc/platforms/pseries/setup.c
+++ b/arch/powerpc/platforms/pseries/setup.c
@@ -42,6 +42,7 @@
 #include <linux/of.h>
 #include <linux/of_pci.h>
 #include <linux/memblock.h>
+#include <linux/swiotlb.h>
 
 #include <asm/mmu.h>
 #include <asm/processor.h>
@@ -71,6 +72,7 @@
 #include <asm/isa-bridge.h>
 #include <asm/security_features.h>
 #include <asm/asm-const.h>
+#include <asm/swiotlb.h>
 
 #include "pseries.h"
 #include "../../../../drivers/pci/pci.h"
@@ -797,6 +799,9 @@ static void __init pSeries_setup_arch(void)
 	}
 
 	ppc_md.pcibios_root_bridge_prepare = pseries_root_bridge_prepare;
+
+	if (swiotlb_force == SWIOTLB_FORCE)
+		ppc_swiotlb_enable = 1;
 }
 
 static void pseries_panic(char *str)
diff --git a/arch/powerpc/platforms/pseries/Kconfig b/arch/powerpc/platforms/pseries/Kconfig
index 9c6b3d860518..b9e8b608de01 100644
--- a/arch/powerpc/platforms/pseries/Kconfig
+++ b/arch/powerpc/platforms/pseries/Kconfig
@@ -23,6 +23,7 @@ config PPC_PSERIES
 	select ARCH_RANDOM
 	select PPC_DOORBELL
 	select FORCE_SMP
+	select SWIOTLB
 	default y
 
 config PPC_SPLPAR
-- 
2.17.1

Re: [PATCH kernel 1/2] powerpc/pseries/dma: Allow swiotlb

[Thiago Jung Bauermann]

Alexey Kardashevskiy <aik@ozlabs.ru> writes:

> The commit 8617a5c5bc00 ("powerpc/dma: handle iommu bypass in
> dma_iommu_ops") merged direct DMA ops into the IOMMU DMA ops allowing
> SWIOTLB as well but only for mapping; the unmapping and bouncing parts
> were left unmodified.
>
> This adds missing direct unmapping calls to .unmap_page() and .unmap_sg().
>
> This adds missing sync callbacks and directs them to the direct DMA hooks.
>
> Fixes: 8617a5c5bc00 (powerpc/dma: handle iommu bypass in dma_iommu_ops)
> Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>

Nice! Thanks for working on this. I have the patch at the end of this
email to get virtio-scsi-pci and virtio-blk-pci working in a secure
guest.

I applied your patch and reverted my patch and unfortunately the guest
hangs right after mounting the disk:

[    0.185659] virtio-pci 0000:00:04.0: enabling device (0100 -> 0102)
[    0.187082] virtio-pci 0000:00:04.0: ibm,query-pe-dma-windows(2026) 2000 8000000 20000000 returned 0
[    0.187497] virtio-pci 0000:00:04.0: ibm,create-pe-dma-window(2027) 2000 8000000 20000000 10 20 returned 0 (liobn = 0x80000001 startin
g addr = 8000000 0)
[    0.226654] Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
[    0.227094] Non-volatile memory driver v1.3
[    0.228950] brd: module loaded
[    0.230666] loop: module loaded
[    0.230773] ipr: IBM Power RAID SCSI Device Driver version: 2.6.4 (March 14, 2017)
[    0.233323] scsi host0: Virtio SCSI HBA
[    0.235439] scsi 0:0:0:0: Direct-Access     QEMU     QEMU HARDDISK    2.5+ PQ: 0 ANSI: 5
[    0.369009] random: fast init done
[    0.370819] sd 0:0:0:0: Attached scsi generic sg0 type 0
[    0.371320] sd 0:0:0:0: Power-on or device reset occurred

<snip>

[    0.380378] sd 0:0:0:0: [sda] 31457280 512-byte logical blocks: (16.1 GB/15.0 GiB)
[    0.381102] sd 0:0:0:0: [sda] Write Protect is off
[    0.381195] sd 0:0:0:0: [sda] Mode Sense: 63 00 00 08
[    0.382436] sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[    0.383630] sd 0:0:0:0: [sda] Optimal transfer size 0 bytes < PAGE_SIZE (65536 bytes)
[    0.391562]  sda: sda1 sda2
[    0.398101] sd 0:0:0:0: [sda] Attached SCSI disk
[    0.398205] md: Waiting for all devices to be available before autodetect
[    0.398318] md: If you don't use raid, use raid=noautodetect
[    0.398515] md: Autodetecting RAID arrays.
[    0.398585] md: autorun ...
[    0.398631] md: ... autorun DONE.
[    0.403552] EXT4-fs (sda2): mounted filesystem with ordered data mode. Opts: (null)
[    0.403700] VFS: Mounted root (ext4 filesystem) readonly on device 8:2.
[    0.405258] devtmpfs: mounted
[    0.406427] Freeing unused kernel memory: 4224K
[    0.406519] This architecture does not have kernel memory protection.
[    0.406633] Run /sbin/init as init process

Sorry, I don't have any information on where the guest is stuck. I tried
<sysrq>+l, <sysrq>+t and <sysrq>+w but nothing out of the ordinary
showed up. Will try something else later.

--
Thiago Jung Bauermann
IBM Linux Technology Center



From 70d2fba809119ae2d35c9ca4269405bb5c28413a Mon Sep 17 00:00:00 2001
From: Thiago Jung Bauermann <bauerman@linux.ibm.com>
Date: Thu, 24 Jan 2019 22:40:16 -0200
Subject: [PATCH 1/1] powerpc/pseries/iommu: Don't use dma_iommu_ops on secure
 guests

Secure guest memory is inacessible to devices so regular DMA isn't
possible.

In that case set devices' dma_map_ops to NULL so that the generic
DMA code path will use SWIOTLB and DMA to bounce buffers.

Signed-off-by: Thiago Jung Bauermann <bauerman@linux.ibm.com>
---
 arch/powerpc/platforms/pseries/iommu.c | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/arch/powerpc/platforms/pseries/iommu.c b/arch/powerpc/platforms/pseries/iommu.c
index 36eb1ddbac69..1636306007eb 100644
--- a/arch/powerpc/platforms/pseries/iommu.c
+++ b/arch/powerpc/platforms/pseries/iommu.c
@@ -50,6 +50,7 @@
 #include <asm/udbg.h>
 #include <asm/mmzone.h>
 #include <asm/plpar_wrappers.h>
+#include <asm/svm.h>

 #include "pseries.h"

@@ -1335,7 +1336,10 @@ void iommu_init_early_pSeries(void)
 	of_reconfig_notifier_register(&iommu_reconfig_nb);
 	register_memory_notifier(&iommu_mem_nb);

-	set_pci_dma_ops(&dma_iommu_ops);
+	if (is_secure_guest())
+		set_pci_dma_ops(NULL);
+	else
+		set_pci_dma_ops(&dma_iommu_ops);
 }

 static int __init disable_multitce(char *str)

Re: [PATCH kernel 2/2] powerpc/pseries/dma: Enable swiotlb

[Thiago Jung Bauermann]

Hello Alexey,

Thanks!

I have similar changes in my "Secure Virtual Machine Enablement"
patches, which I am currently preparing for posting again real soon now.

This is the last version:

https://lore.kernel.org/linuxppc-dev/20180824162535.22798-1-bauerman@linux.ibm.com/

Alexey Kardashevskiy <aik@ozlabs.ru> writes:

> So far the pseries platforms has always been using IOMMU making SWIOTLB
> unnecessary. Now we want secure guests which means devices can only
> access certain areas of guest physical memory; we are going to use
> SWIOTLB for this purpose.
>
> This allows SWIOTLB for pseries. By default there is no change in behavior.
>
> This enables SWIOTLB when the "swiotlb" kernel parameter is set to "force".
>
> With the SWIOTLB enabled, the kernel creates a directly mapped DMA window
> (using the usual DDW mechanism) and implements SWIOTLB on top of that.
>
> Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
> ---
>  arch/powerpc/platforms/pseries/setup.c | 5 +++++
>  arch/powerpc/platforms/pseries/Kconfig | 1 +
>  2 files changed, 6 insertions(+)
>
> diff --git a/arch/powerpc/platforms/pseries/setup.c b/arch/powerpc/platforms/pseries/setup.c
> index e4f0dfd4ae33..30d72b587ac5 100644
> --- a/arch/powerpc/platforms/pseries/setup.c
> +++ b/arch/powerpc/platforms/pseries/setup.c
> @@ -42,6 +42,7 @@
>  #include <linux/of.h>
>  #include <linux/of_pci.h>
>  #include <linux/memblock.h>
> +#include <linux/swiotlb.h>
>
>  #include <asm/mmu.h>
>  #include <asm/processor.h>
> @@ -71,6 +72,7 @@
>  #include <asm/isa-bridge.h>
>  #include <asm/security_features.h>
>  #include <asm/asm-const.h>
> +#include <asm/swiotlb.h>
>
>  #include "pseries.h"
>  #include "../../../../drivers/pci/pci.h"
> @@ -797,6 +799,9 @@ static void __init pSeries_setup_arch(void)
>  	}
>
>  	ppc_md.pcibios_root_bridge_prepare = pseries_root_bridge_prepare;
> +
> +	if (swiotlb_force == SWIOTLB_FORCE)
> +		ppc_swiotlb_enable = 1;
>  }

Yep! I have this here, enabled when booting as a secure guest:

https://lore.kernel.org/linuxppc-dev/20180824162535.22798-6-bauerman@linux.ibm.com/

And also another patch which makes it so that if booting as a secure
guest it acts as if the swiotlb kernel parameter was set to force:

https://lore.kernel.org/linuxppc-dev/20180824162535.22798-11-bauerman@linux.ibm.com/

>  static void pseries_panic(char *str)
> diff --git a/arch/powerpc/platforms/pseries/Kconfig b/arch/powerpc/platforms/pseries/Kconfig
> index 9c6b3d860518..b9e8b608de01 100644
> --- a/arch/powerpc/platforms/pseries/Kconfig
> +++ b/arch/powerpc/platforms/pseries/Kconfig
> @@ -23,6 +23,7 @@ config PPC_PSERIES
>  	select ARCH_RANDOM
>  	select PPC_DOORBELL
>  	select FORCE_SMP
> +	select SWIOTLB
>  	default y
>
>  config PPC_SPLPAR

I put this in a PPC_SVM config option:

https://lore.kernel.org/linuxppc-dev/20180824162535.22798-3-bauerman@linux.ibm.com/

--
Thiago Jung Bauermann
IBM Linux Technology Center

Re: [PATCH kernel 1/2] powerpc/pseries/dma: Allow swiotlb

[Alexey Kardashevskiy]

On 11/05/2019 08:36, Thiago Jung Bauermann wrote:
> 
> Alexey Kardashevskiy <aik@ozlabs.ru> writes:
> 
>> The commit 8617a5c5bc00 ("powerpc/dma: handle iommu bypass in
>> dma_iommu_ops") merged direct DMA ops into the IOMMU DMA ops allowing
>> SWIOTLB as well but only for mapping; the unmapping and bouncing parts
>> were left unmodified.
>>
>> This adds missing direct unmapping calls to .unmap_page() and .unmap_sg().
>>
>> This adds missing sync callbacks and directs them to the direct DMA hooks.
>>
>> Fixes: 8617a5c5bc00 (powerpc/dma: handle iommu bypass in dma_iommu_ops)
>> Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
> 
> Nice! Thanks for working on this. I have the patch at the end of this
> email to get virtio-scsi-pci and virtio-blk-pci working in a secure
> guest.

I saw the set_pci_dma_ops(NULL) patch but could not figure out how pass
NULL there sets the DMA ops to direct.

> 
> I applied your patch and reverted my patch and unfortunately the guest
> hangs right after mounting the disk:

Have you applied it on upstream kernel? I cannot see how it affects
current guests as it is...


> 
> [    0.185659] virtio-pci 0000:00:04.0: enabling device (0100 -> 0102)
> [    0.187082] virtio-pci 0000:00:04.0: ibm,query-pe-dma-windows(2026) 2000 8000000 20000000 returned 0
> [    0.187497] virtio-pci 0000:00:04.0: ibm,create-pe-dma-window(2027) 2000 8000000 20000000 10 20 returned 0 (liobn = 0x80000001 startin
> g addr = 8000000 0)
> [    0.226654] Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
> [    0.227094] Non-volatile memory driver v1.3
> [    0.228950] brd: module loaded
> [    0.230666] loop: module loaded
> [    0.230773] ipr: IBM Power RAID SCSI Device Driver version: 2.6.4 (March 14, 2017)
> [    0.233323] scsi host0: Virtio SCSI HBA
> [    0.235439] scsi 0:0:0:0: Direct-Access     QEMU     QEMU HARDDISK    2.5+ PQ: 0 ANSI: 5
> [    0.369009] random: fast init done
> [    0.370819] sd 0:0:0:0: Attached scsi generic sg0 type 0
> [    0.371320] sd 0:0:0:0: Power-on or device reset occurred
> 
> <snip>
> 
> [    0.380378] sd 0:0:0:0: [sda] 31457280 512-byte logical blocks: (16.1 GB/15.0 GiB)
> [    0.381102] sd 0:0:0:0: [sda] Write Protect is off
> [    0.381195] sd 0:0:0:0: [sda] Mode Sense: 63 00 00 08
> [    0.382436] sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
> [    0.383630] sd 0:0:0:0: [sda] Optimal transfer size 0 bytes < PAGE_SIZE (65536 bytes)
> [    0.391562]  sda: sda1 sda2
> [    0.398101] sd 0:0:0:0: [sda] Attached SCSI disk
> [    0.398205] md: Waiting for all devices to be available before autodetect
> [    0.398318] md: If you don't use raid, use raid=noautodetect
> [    0.398515] md: Autodetecting RAID arrays.
> [    0.398585] md: autorun ...
> [    0.398631] md: ... autorun DONE.
> [    0.403552] EXT4-fs (sda2): mounted filesystem with ordered data mode. Opts: (null)
> [    0.403700] VFS: Mounted root (ext4 filesystem) readonly on device 8:2.
> [    0.405258] devtmpfs: mounted
> [    0.406427] Freeing unused kernel memory: 4224K
> [    0.406519] This architecture does not have kernel memory protection.
> [    0.406633] Run /sbin/init as init process
> 
> Sorry, I don't have any information on where the guest is stuck. I tried
> <sysrq>+l, <sysrq>+t and <sysrq>+w but nothing out of the ordinary
> showed up. Will try something else later.
> 
> --
> Thiago Jung Bauermann
> IBM Linux Technology Center
> 
> 
> 
> From 70d2fba809119ae2d35c9ca4269405bb5c28413a Mon Sep 17 00:00:00 2001
> From: Thiago Jung Bauermann <bauerman@linux.ibm.com>
> Date: Thu, 24 Jan 2019 22:40:16 -0200
> Subject: [PATCH 1/1] powerpc/pseries/iommu: Don't use dma_iommu_ops on secure
>  guests
> 
> Secure guest memory is inacessible to devices so regular DMA isn't
> possible.
> 
> In that case set devices' dma_map_ops to NULL so that the generic
> DMA code path will use SWIOTLB and DMA to bounce buffers.
> 
> Signed-off-by: Thiago Jung Bauermann <bauerman@linux.ibm.com>
> ---
>  arch/powerpc/platforms/pseries/iommu.c | 6 +++++-
>  1 file changed, 5 insertions(+), 1 deletion(-)
> 
> diff --git a/arch/powerpc/platforms/pseries/iommu.c b/arch/powerpc/platforms/pseries/iommu.c
> index 36eb1ddbac69..1636306007eb 100644
> --- a/arch/powerpc/platforms/pseries/iommu.c
> +++ b/arch/powerpc/platforms/pseries/iommu.c
> @@ -50,6 +50,7 @@
>  #include <asm/udbg.h>
>  #include <asm/mmzone.h>
>  #include <asm/plpar_wrappers.h>
> +#include <asm/svm.h>
> 
>  #include "pseries.h"
> 
> @@ -1335,7 +1336,10 @@ void iommu_init_early_pSeries(void)
>  	of_reconfig_notifier_register(&iommu_reconfig_nb);
>  	register_memory_notifier(&iommu_mem_nb);
> 
> -	set_pci_dma_ops(&dma_iommu_ops);
> +	if (is_secure_guest())
> +		set_pci_dma_ops(NULL);
> +	else
> +		set_pci_dma_ops(&dma_iommu_ops);
>  }
> 
>  static int __init disable_multitce(char *str)
> 

-- 
Alexey

Re: [PATCH kernel 2/2] powerpc/pseries/dma: Enable swiotlb

[Alexey Kardashevskiy]

On 11/05/2019 08:41, Thiago Jung Bauermann wrote:
> 
> Hello Alexey,
> 
> Thanks!
> 
> I have similar changes in my "Secure Virtual Machine Enablement"
> patches, which I am currently preparing for posting again real soon now.
> 
> This is the last version:
> 
> https://lore.kernel.org/linuxppc-dev/20180824162535.22798-1-bauerman@linux.ibm.com/
> 
> Alexey Kardashevskiy <aik@ozlabs.ru> writes:
> 
>> So far the pseries platforms has always been using IOMMU making SWIOTLB
>> unnecessary. Now we want secure guests which means devices can only
>> access certain areas of guest physical memory; we are going to use
>> SWIOTLB for this purpose.
>>
>> This allows SWIOTLB for pseries. By default there is no change in behavior.
>>
>> This enables SWIOTLB when the "swiotlb" kernel parameter is set to "force".
>>
>> With the SWIOTLB enabled, the kernel creates a directly mapped DMA window
>> (using the usual DDW mechanism) and implements SWIOTLB on top of that.
>>
>> Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
>> ---
>>  arch/powerpc/platforms/pseries/setup.c | 5 +++++
>>  arch/powerpc/platforms/pseries/Kconfig | 1 +
>>  2 files changed, 6 insertions(+)
>>
>> diff --git a/arch/powerpc/platforms/pseries/setup.c b/arch/powerpc/platforms/pseries/setup.c
>> index e4f0dfd4ae33..30d72b587ac5 100644
>> --- a/arch/powerpc/platforms/pseries/setup.c
>> +++ b/arch/powerpc/platforms/pseries/setup.c
>> @@ -42,6 +42,7 @@
>>  #include <linux/of.h>
>>  #include <linux/of_pci.h>
>>  #include <linux/memblock.h>
>> +#include <linux/swiotlb.h>
>>
>>  #include <asm/mmu.h>
>>  #include <asm/processor.h>
>> @@ -71,6 +72,7 @@
>>  #include <asm/isa-bridge.h>
>>  #include <asm/security_features.h>
>>  #include <asm/asm-const.h>
>> +#include <asm/swiotlb.h>
>>
>>  #include "pseries.h"
>>  #include "../../../../drivers/pci/pci.h"
>> @@ -797,6 +799,9 @@ static void __init pSeries_setup_arch(void)
>>  	}
>>
>>  	ppc_md.pcibios_root_bridge_prepare = pseries_root_bridge_prepare;
>> +
>> +	if (swiotlb_force == SWIOTLB_FORCE)
>> +		ppc_swiotlb_enable = 1;
>>  }
> 
> Yep! I have this here, enabled when booting as a secure guest:
> 
> https://lore.kernel.org/linuxppc-dev/20180824162535.22798-6-bauerman@linux.ibm.com/
> 
> And also another patch which makes it so that if booting as a secure
> guest it acts as if the swiotlb kernel parameter was set to force:
> 
> https://lore.kernel.org/linuxppc-dev/20180824162535.22798-11-bauerman@linux.ibm.com/
> 
>>  static void pseries_panic(char *str)
>> diff --git a/arch/powerpc/platforms/pseries/Kconfig b/arch/powerpc/platforms/pseries/Kconfig
>> index 9c6b3d860518..b9e8b608de01 100644
>> --- a/arch/powerpc/platforms/pseries/Kconfig
>> +++ b/arch/powerpc/platforms/pseries/Kconfig
>> @@ -23,6 +23,7 @@ config PPC_PSERIES
>>  	select ARCH_RANDOM
>>  	select PPC_DOORBELL
>>  	select FORCE_SMP
>> +	select SWIOTLB
>>  	default y
>>
>>  config PPC_SPLPAR
> 
> I put this in a PPC_SVM config option:
> 
> https://lore.kernel.org/linuxppc-dev/20180824162535.22798-3-bauerman@linux.ibm.com/


Well, my intention is to make it work regardless SVM, just to see if it
works and where the problems are if it does not (right now the NVIDIA
driver does not like SWIOTLB, debugging).



-- 
Alexey

Re: [PATCH kernel 0/2] pseries: Enable SWIOTLB

[Alexey Kardashevskiy]

Ping, anyone?

On 07/05/2019 16:25, Alexey Kardashevskiy wrote:
> This is an attempt to allow PCI pass through to a secure guest when
> hardware can only access insecure memory. This allows SWIOTLB use
> for passed through devices.
> 
> Later on secure VMs will unsecure SWIOTLB bounce buffers for DMA
> and the rest of the guest RAM will be unavailable to the hardware
> by default.
> 
> 
> This is based on sha1
> e93c9c99a629 Linus Torvalds "Linux 5.1".
> 
> Please comment. Thanks.
> 
> 
> 
> Alexey Kardashevskiy (2):
>   powerpc/pseries/dma: Allow swiotlb
>   powerpc/pseries/dma: Enable swiotlb
> 
>  arch/powerpc/kernel/dma-iommu.c        | 36 ++++++++++++++++++++++++++
>  arch/powerpc/platforms/pseries/setup.c |  5 ++++
>  arch/powerpc/platforms/pseries/Kconfig |  1 +
>  3 files changed, 42 insertions(+)
> 

-- 
Alexey

Re: [PATCH kernel 1/2] powerpc/pseries/dma: Allow swiotlb

[Thiago Jung Bauermann]

Alexey Kardashevskiy <aik@ozlabs.ru> writes:

> On 11/05/2019 08:36, Thiago Jung Bauermann wrote:
>>
>> Alexey Kardashevskiy <aik@ozlabs.ru> writes:
>>
>>> The commit 8617a5c5bc00 ("powerpc/dma: handle iommu bypass in
>>> dma_iommu_ops") merged direct DMA ops into the IOMMU DMA ops allowing
>>> SWIOTLB as well but only for mapping; the unmapping and bouncing parts
>>> were left unmodified.
>>>
>>> This adds missing direct unmapping calls to .unmap_page() and .unmap_sg().
>>>
>>> This adds missing sync callbacks and directs them to the direct DMA hooks.
>>>
>>> Fixes: 8617a5c5bc00 (powerpc/dma: handle iommu bypass in dma_iommu_ops)
>>> Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
>>
>> Nice! Thanks for working on this. I have the patch at the end of this
>> email to get virtio-scsi-pci and virtio-blk-pci working in a secure
>> guest.
>
> I saw the set_pci_dma_ops(NULL) patch but could not figure out how pass
> NULL there sets the DMA ops to direct.

That causes pcibios_setup_device() to call set_dma_ops(&dev->dev, NULL),
which in turn causes dma_is_direct(get_dma_ops(dev)) to return true.

>> I applied your patch and reverted my patch and unfortunately the guest
>> hangs right after mounting the disk:
>
> Have you applied it on upstream kernel? I cannot see how it affects
> current guests as it is...

I applied it on a branch containing both Claudio Carvalho's "kvmppc:
Paravirtualize KVM to support ultravisor" series as well as my "Secure
Virtual Machine Enablement" patch series.

https://lore.kernel.org/linuxppc-dev/20190518142524.28528-1-cclaudio@linux.ibm.com/
https://lore.kernel.org/linuxppc-dev/20190521044912.1375-1-bauerman@linux.ibm.com/

--
Thiago Jung Bauermann
IBM Linux Technology Center

Re: [PATCH kernel 1/2] powerpc/pseries/dma: Allow swiotlb

[Michael Ellerman]

On Tue, 2019-05-07 at 06:25:58 UTC, Alexey Kardashevskiy wrote:
> The commit 8617a5c5bc00 ("powerpc/dma: handle iommu bypass in
> dma_iommu_ops") merged direct DMA ops into the IOMMU DMA ops allowing
> SWIOTLB as well but only for mapping; the unmapping and bouncing parts
> were left unmodified.
> 
> This adds missing direct unmapping calls to .unmap_page() and .unmap_sg().
> 
> This adds missing sync callbacks and directs them to the direct DMA hooks.
> 
> Fixes: 8617a5c5bc00 (powerpc/dma: handle iommu bypass in dma_iommu_ops)
> Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
> Signed-off-by: Thiago Jung Bauermann <bauerman@linux.ibm.com>

Series applied to powerpc next, thanks.

https://git.kernel.org/powerpc/c/efd176a04bef41aab5b3087e977fea2b69915174

cheers