IOMMU Archive on lore.kernel.org
 help / color / Atom feed
* [RFC PATCH 00/11] Secure Virtual Machine Enablement
@ 2018-08-24 16:25 Thiago Jung Bauermann
  2018-08-24 16:25 ` [RFC PATCH 01/11] powerpc/svm: Detect Secure Virtual Machine (SVM) platform Thiago Jung Bauermann
                   ` (12 more replies)
  0 siblings, 13 replies; 20+ messages in thread
From: Thiago Jung Bauermann @ 2018-08-24 16:25 UTC (permalink / raw)
  To: linuxppc-dev
  Cc: iommu, linux-kernel, Alexey Kardashevskiy, Anshuman Khandual,
	Benjamin Herrenschmidt, Christoph Hellwig, Michael Ellerman,
	Mike Anderson, Paul Mackerras, Ram Pai, Thiago Jung Bauermann

[ Some people didn't receive all the patches in this series, even though
  the linuxppc-dev list did so trying to send again. This is exactly the
  same series I posted yesterday. Sorry for the clutter. ]

This series contains preliminary work to enable Secure Virtual Machines
(SVM) on powerpc. SVMs request to be migrated to secure memory very early in
the boot process (in prom_init()), so by default all of their memory is
inaccessible to the hypervisor. There is an ultravisor call that the VM can
use to request certain pages to be made accessible (aka shared).

The objective of these patches is to have the guest perform this request for
buffers that need to be shared with the hypervisor, such as the LPPACAs, the
SWIOTLB buffer and the Debug Trace Log. This work is incomplete: there are
surely other memory regions that need to be made accessible, but I'm posting
it early to get comments on whether the approach being taken is appropriate.

It should be applied on top of the generic virtio DMA API rework series
posted earlier, which adds a platform hook to override any arch based DMA
API operations for any virtio device:

https://lists.ozlabs.org/pipermail/linuxppc-dev/2018-July/175994.html

I'm aware that changes need to be made to the patch series above, but IIUC
it depends on upcoming virtio cleanup from Christoph Hellwig so for now the
patch series above will be used as a stepping stone for this series.

This code has been tested with virtio block, net and scsi devices with and
without VIRTIO_F_IOMMU_PLATFORM flag. Please let me know what you think.

For now I am testing on a regular guest with a couple of patches on top
forcing is_svm_platform() to always return true and adding debug messages to
confirm that mem_convert_shared() is being called in the expected places.

These are the commands I'm using to start up the guest:

Without VIRTIO_F_IOMMU_PLATFORM:

qemu-system-ppc64 \
  -enable-kvm \
  -kernel /home/bauermann/src/linux/arch/powerpc/boot/zImage \
  -append "root=PARTUUID=e550ad6f-05 ro" \
  -machine pseries-2.6 \
  -m 8G \
  -smp 2 \
  -serial mon:stdio \
  -nographic \
  -nodefaults \
  -device virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x4 \
  -drive file=/home/bauermann/VMs/svm.qcow2,format=qcow2,if=none,id=drive-scsi0-0-0-0 \
  -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0 \
  -drive file=/home/bauermann/VMs/svm-blk.qcow2,format=qcow2,if=none,id=drive-virtio-disk0 \
  -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0 \
  -device virtio-net,netdev=hostnet0,id=net0,mac=52:54:00:96:70:1f \
  -netdev user,id=hostnet0 \
  -set netdev.hostnet0.hostfwd=tcp::42022-:22

With VIRTIO_F_IOMMU_PLATFORM. Same as above plus some -global options so
that the virtio devices use the modern interface rather than the
transitional one:

qemu-system-ppc64 \
  -enable-kvm \
  -kernel /home/bauermann/src/linux/arch/powerpc/boot/zImage \
  -append "root=PARTUUID=e550ad6f-05 ro" \
  -machine pseries-2.6 \
  -m 8G \
  -smp 2 \
  -serial mon:stdio \
  -nographic \
  -nodefaults \
  -device virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x4 \
  -drive file=/home/bauermann/VMs/svm.qcow2,format=qcow2,if=none,id=drive-scsi0-0-0-0 \
  -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0 \
  -drive file=/home/bauermann/VMs/svm-blk.qcow2,format=qcow2,if=none,id=drive-virtio-disk0 \
  -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0 \
  -device virtio-net,netdev=hostnet0,id=net0,mac=52:54:00:96:70:1f \
  -netdev user,id=hostnet0 \
  -set netdev.hostnet0.hostfwd=tcp::42022-:22 \
  -global virtio-blk-pci.iommu_platform=true \
  -global virtio-blk-pci.disable-legacy=on \
  -global virtio-blk-pci.disable-modern=off \
  -global virtio-net-pci.iommu_platform=true \
  -global virtio-net-pci.disable-legacy=on \
  -global virtio-net-pci.disable-modern=off \
  -global virtio-scsi-pci.iommu_platform=true \
  -global virtio-scsi-pci.disable-legacy=on \
  -global virtio-scsi-pci.disable-modern=off

The code was tested with a couple of other permutations where one virtio
device has the flag VIRTIO_F_IOMMU_PLATFORM and others don't. Please suggest
some other scenarios which need to be tested as well.

Anshuman Khandual (10):
  powerpc/svm: Detect Secure Virtual Machine (SVM) platform
  powerpc/svm: Select CONFIG_DMA_DIRECT_OPS and CONFIG_SWIOTLB
  powerpc/svm: Add memory conversion (shared/secure) helper functions
  powerpc/svm: Convert SWIOTLB buffers to shared memory
  powerpc/svm: Don't release SWIOTLB buffers on secure guests
  powerpc/svm: Use SWIOTLB DMA API for all virtio devices
  powerpc/svm: Use shared memory for Debug Trace Log (DTL)
  powerpc/svm: Use shared memory for LPPACA structures
  powerpc/svm: Force the use of bounce buffers
  powerpc/svm: Increase SWIOTLB buffer size

Thiago Jung Bauermann (1):
  powerpc: Add and use LPPACA_SIZE constant

 arch/powerpc/Kconfig                   | 22 ++++++++
 arch/powerpc/include/asm/mem_encrypt.h | 19 +++++++
 arch/powerpc/include/asm/reg.h         |  3 ++
 arch/powerpc/include/asm/svm.h         | 26 +++++++++
 arch/powerpc/kernel/Makefile           |  1 +
 arch/powerpc/kernel/paca.c             | 41 ++++++++++++--
 arch/powerpc/kernel/svm.c              | 99 ++++++++++++++++++++++++++++++++++
 arch/powerpc/platforms/pseries/iommu.c |  6 ++-
 arch/powerpc/platforms/pseries/setup.c |  5 +-
 kernel/dma/swiotlb.c                   |  5 ++
 10 files changed, 221 insertions(+), 6 deletions(-)
 create mode 100644 arch/powerpc/include/asm/mem_encrypt.h
 create mode 100644 arch/powerpc/include/asm/svm.h
 create mode 100644 arch/powerpc/kernel/svm.c

^ permalink raw reply	[flat|nested] 20+ messages in thread

* [RFC PATCH 01/11] powerpc/svm: Detect Secure Virtual Machine (SVM) platform
  2018-08-24 16:25 [RFC PATCH 00/11] Secure Virtual Machine Enablement Thiago Jung Bauermann
@ 2018-08-24 16:25 ` Thiago Jung Bauermann
  2018-08-24 16:25 ` [RFC PATCH 02/11] powerpc/svm: Select CONFIG_DMA_DIRECT_OPS and CONFIG_SWIOTLB Thiago Jung Bauermann
                   ` (11 subsequent siblings)
  12 siblings, 0 replies; 20+ messages in thread
From: Thiago Jung Bauermann @ 2018-08-24 16:25 UTC (permalink / raw)
  To: linuxppc-dev
  Cc: iommu, linux-kernel, Alexey Kardashevskiy, Anshuman Khandual,
	Benjamin Herrenschmidt, Christoph Hellwig, Michael Ellerman,
	Mike Anderson, Paul Mackerras, Ram Pai, Anshuman Khandual,
	Sukadev Bhattiprolu, Thiago Jung Bauermann

From: Anshuman Khandual <khandual@linux.vnet.ibm.com>

A guest requests to be moved to secure memory early at the kernel
startup (in prom_init). Define a flag that can be easily checked by other
parts of the kernel so that they can set things up accordingly. This is
done by checking the MSR(S) bit, which is always set for secure VMs.

Also add a new config option CONFIG_PPC_SVM to wrap all these code to
prevent it from being executed from non subscribing platforms. This SVM
platform detection is applicable only to guest kernels that will run under
an Ultravisor as a secure guest.

Signed-off-by: Anshuman Khandual <khandual@linux.vnet.ibm.com>
Signed-off-by: Ram Pai <linuxram@us.ibm.com>
Signed-off-by: Sukadev Bhattiprolu <sukadev@linux.vnet.ibm.com>
Signed-off-by: Thiago Jung Bauermann <bauerman@linux.ibm.com>
---
 arch/powerpc/Kconfig           | 11 +++++++++++
 arch/powerpc/include/asm/reg.h |  3 +++
 arch/powerpc/include/asm/svm.h | 22 ++++++++++++++++++++++
 3 files changed, 36 insertions(+)

diff --git a/arch/powerpc/Kconfig b/arch/powerpc/Kconfig
index 9f2b75fe2c2d..f786c962edf0 100644
--- a/arch/powerpc/Kconfig
+++ b/arch/powerpc/Kconfig
@@ -440,6 +440,17 @@ config MATH_EMULATION_HW_UNIMPLEMENTED
 
 endchoice
 
+config PPC_SVM
+       bool "Secure virtual machine (SVM) support for POWERPC"
+       default n
+       depends on PPC_PSERIES
+       help
+         Support secure guests on POWERPC. There are certain POWER platforms
+	 which support secure guests with the help of an Ultravisor executing
+	 below the hypervisor layer. This enables the support for those guests.
+
+	 If unsure, say "N".
+
 config PPC_TRANSACTIONAL_MEM
        bool "Transactional Memory support for POWERPC"
        depends on PPC_BOOK3S_64
diff --git a/arch/powerpc/include/asm/reg.h b/arch/powerpc/include/asm/reg.h
index 562568414cf4..fcf7b79356d0 100644
--- a/arch/powerpc/include/asm/reg.h
+++ b/arch/powerpc/include/asm/reg.h
@@ -36,6 +36,7 @@
 #define MSR_TM_LG	32		/* Trans Mem Available */
 #define MSR_VEC_LG	25	        /* Enable AltiVec */
 #define MSR_VSX_LG	23		/* Enable VSX */
+#define MSR_S_LG	22		/* Secure VM bit */
 #define MSR_POW_LG	18		/* Enable Power Management */
 #define MSR_WE_LG	18		/* Wait State Enable */
 #define MSR_TGPR_LG	17		/* TLB Update registers in use */
@@ -69,11 +70,13 @@
 #define MSR_SF		__MASK(MSR_SF_LG)	/* Enable 64 bit mode */
 #define MSR_ISF		__MASK(MSR_ISF_LG)	/* Interrupt 64b mode valid on 630 */
 #define MSR_HV 		__MASK(MSR_HV_LG)	/* Hypervisor state */
+#define MSR_S 		__MASK(MSR_S_LG)	/* Secure state */
 #else
 /* so tests for these bits fail on 32-bit */
 #define MSR_SF		0
 #define MSR_ISF		0
 #define MSR_HV		0
+#define MSR_S		0
 #endif
 
 /*
diff --git a/arch/powerpc/include/asm/svm.h b/arch/powerpc/include/asm/svm.h
new file mode 100644
index 000000000000..6f89e5d6d37f
--- /dev/null
+++ b/arch/powerpc/include/asm/svm.h
@@ -0,0 +1,22 @@
+/* SPDX-License-Identifier: GPL-2.0+ */
+/*
+ * SVM helper functions
+ *
+ * Copyright 2018 Anshuman Khandual, IBM Corporation.
+ */
+
+#ifndef _ASM_POWERPC_SVM_H
+#define _ASM_POWERPC_SVM_H
+
+#ifdef CONFIG_PPC_SVM
+static bool is_svm_platform(void)
+{
+	return mfmsr() & MSR_S;
+}
+#else
+static inline bool is_svm_platform(void)
+{
+	return false;
+}
+#endif
+#endif /* _ASM_POWERPC_SVM_H */

^ permalink raw reply	[flat|nested] 20+ messages in thread

* [RFC PATCH 02/11] powerpc/svm: Select CONFIG_DMA_DIRECT_OPS and CONFIG_SWIOTLB
  2018-08-24 16:25 [RFC PATCH 00/11] Secure Virtual Machine Enablement Thiago Jung Bauermann
  2018-08-24 16:25 ` [RFC PATCH 01/11] powerpc/svm: Detect Secure Virtual Machine (SVM) platform Thiago Jung Bauermann
@ 2018-08-24 16:25 ` Thiago Jung Bauermann
  2018-08-24 16:25 ` [RFC PATCH 03/11] powerpc/svm: Add memory conversion (shared/secure) helper functions Thiago Jung Bauermann
                   ` (10 subsequent siblings)
  12 siblings, 0 replies; 20+ messages in thread
From: Thiago Jung Bauermann @ 2018-08-24 16:25 UTC (permalink / raw)
  To: linuxppc-dev
  Cc: iommu, linux-kernel, Alexey Kardashevskiy, Anshuman Khandual,
	Benjamin Herrenschmidt, Christoph Hellwig, Michael Ellerman,
	Mike Anderson, Paul Mackerras, Ram Pai, Anshuman Khandual,
	Thiago Jung Bauermann

From: Anshuman Khandual <khandual@linux.vnet.ibm.com>

Ultravisor based secure guest platforms will use generic SWIOTLB DMA API
(i.e swiotlb_dma_ops structure) available under the CONFIG_DMA_DIRECT_OPS
config. Also base CONFIG_SWIOTLB needs to be explicitly selected as well.
Select both these config options on powerpc server platforms with secure
guest support.

Signed-off-by: Anshuman Khandual <khandual@linux.vnet.ibm.com>
Signed-off-by: Thiago Jung Bauermann <bauerman@linux.ibm.com>
---
 arch/powerpc/Kconfig | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/arch/powerpc/Kconfig b/arch/powerpc/Kconfig
index f786c962edf0..90f73d15f58a 100644
--- a/arch/powerpc/Kconfig
+++ b/arch/powerpc/Kconfig
@@ -444,6 +444,8 @@ config PPC_SVM
        bool "Secure virtual machine (SVM) support for POWERPC"
        default n
        depends on PPC_PSERIES
+       select DMA_DIRECT_OPS
+       select SWIOTLB
        help
          Support secure guests on POWERPC. There are certain POWER platforms
 	 which support secure guests with the help of an Ultravisor executing

^ permalink raw reply	[flat|nested] 20+ messages in thread

* [RFC PATCH 03/11] powerpc/svm: Add memory conversion (shared/secure) helper functions
  2018-08-24 16:25 [RFC PATCH 00/11] Secure Virtual Machine Enablement Thiago Jung Bauermann
  2018-08-24 16:25 ` [RFC PATCH 01/11] powerpc/svm: Detect Secure Virtual Machine (SVM) platform Thiago Jung Bauermann
  2018-08-24 16:25 ` [RFC PATCH 02/11] powerpc/svm: Select CONFIG_DMA_DIRECT_OPS and CONFIG_SWIOTLB Thiago Jung Bauermann
@ 2018-08-24 16:25 ` Thiago Jung Bauermann
  2018-08-24 16:25 ` [RFC PATCH 04/11] powerpc/svm: Convert SWIOTLB buffers to shared memory Thiago Jung Bauermann
                   ` (9 subsequent siblings)
  12 siblings, 0 replies; 20+ messages in thread
From: Thiago Jung Bauermann @ 2018-08-24 16:25 UTC (permalink / raw)
  To: linuxppc-dev
  Cc: iommu, linux-kernel, Alexey Kardashevskiy, Anshuman Khandual,
	Benjamin Herrenschmidt, Christoph Hellwig, Michael Ellerman,
	Mike Anderson, Paul Mackerras, Ram Pai, Anshuman Khandual,
	Thiago Jung Bauermann

From: Anshuman Khandual <khandual@linux.vnet.ibm.com>

Add the following helper functions to convert PAGE_SIZE aligned memory
buffers as shared or secure (i.e., accessible to the hypervisor or not) via
Ultravisor calls.

1. mem_convert_shared(unsigned long pfn, unsigned long npages)
2. mem_convert_secure(unsigned long pfn, unsigned long npages)

Signed-off-by: Anshuman Khandual <khandual@linux.vnet.ibm.com>
Signed-off-by: Thiago Jung Bauermann <bauerman@linux.ibm.com>
---
 arch/powerpc/include/asm/svm.h |  3 +++
 arch/powerpc/kernel/Makefile   |  1 +
 arch/powerpc/kernel/svm.c      | 33 +++++++++++++++++++++++++++++++++
 3 files changed, 37 insertions(+)

diff --git a/arch/powerpc/include/asm/svm.h b/arch/powerpc/include/asm/svm.h
index 6f89e5d6d37f..95d69e472e52 100644
--- a/arch/powerpc/include/asm/svm.h
+++ b/arch/powerpc/include/asm/svm.h
@@ -13,6 +13,9 @@ static bool is_svm_platform(void)
 {
 	return mfmsr() & MSR_S;
 }
+
+extern void mem_convert_shared(unsigned long pfn, unsigned long npages);
+extern void mem_convert_secure(unsigned long pfn, unsigned long npages);
 #else
 static inline bool is_svm_platform(void)
 {
diff --git a/arch/powerpc/kernel/Makefile b/arch/powerpc/kernel/Makefile
index 2b4c40b255e4..98780b4e924c 100644
--- a/arch/powerpc/kernel/Makefile
+++ b/arch/powerpc/kernel/Makefile
@@ -113,6 +113,7 @@ ifeq ($(CONFIG_HAVE_IMA_KEXEC)$(CONFIG_IMA),yy)
 obj-y				+= ima_kexec.o
 endif
 
+obj-$(CONFIG_PPC_SVM)		+= svm.o
 obj-$(CONFIG_AUDIT)		+= audit.o
 obj64-$(CONFIG_AUDIT)		+= compat_audit.o
 
diff --git a/arch/powerpc/kernel/svm.c b/arch/powerpc/kernel/svm.c
new file mode 100644
index 000000000000..37437cf92df5
--- /dev/null
+++ b/arch/powerpc/kernel/svm.c
@@ -0,0 +1,33 @@
+// SPDX-License-Identifier: GPL-2.0+
+/*
+ * Secure VM platform
+ *
+ * Copyright 2018 IBM Corporation
+ * Author: Anshuman Khandual <khandual@linux.vnet.ibm.com>
+ */
+
+#include <linux/mm.h>
+
+void mem_convert_shared(unsigned long pfn, unsigned long npages)
+{
+	if (!pfn_valid(pfn))
+		return;
+
+	/*
+	 * FIXME: Insert real UV call when ready
+	 *
+	 * ucall_convert_shared(paddr, size)
+	 */
+}
+
+void mem_convert_secure(unsigned long pfn, unsigned long npages)
+{
+	if (!pfn_valid(pfn))
+		return;
+
+	/*
+	 * FIXME: Insert real UV call when ready
+	 *
+	 * ucall_convert_secure(paddr, size)
+	 */
+}

^ permalink raw reply	[flat|nested] 20+ messages in thread

* [RFC PATCH 04/11] powerpc/svm: Convert SWIOTLB buffers to shared memory
  2018-08-24 16:25 [RFC PATCH 00/11] Secure Virtual Machine Enablement Thiago Jung Bauermann
                   ` (2 preceding siblings ...)
  2018-08-24 16:25 ` [RFC PATCH 03/11] powerpc/svm: Add memory conversion (shared/secure) helper functions Thiago Jung Bauermann
@ 2018-08-24 16:25 ` Thiago Jung Bauermann
  2018-08-24 16:25 ` [RFC PATCH 05/11] powerpc/svm: Don't release SWIOTLB buffers on secure guests Thiago Jung Bauermann
                   ` (8 subsequent siblings)
  12 siblings, 0 replies; 20+ messages in thread
From: Thiago Jung Bauermann @ 2018-08-24 16:25 UTC (permalink / raw)
  To: linuxppc-dev
  Cc: iommu, linux-kernel, Alexey Kardashevskiy, Anshuman Khandual,
	Benjamin Herrenschmidt, Christoph Hellwig, Michael Ellerman,
	Mike Anderson, Paul Mackerras, Ram Pai, Anshuman Khandual,
	Thiago Jung Bauermann

From: Anshuman Khandual <khandual@linux.vnet.ibm.com>

Hook the shared memory conversion functions into the ARCH_HAS_MEM_ENCRYPT
framework and call swiotlb_update_mem_attributes() to convert SWIOTLB's
buffers to shared memory.

Signed-off-by: Anshuman Khandual <khandual@linux.vnet.ibm.com>
Signed-off-by: Thiago Jung Bauermann <bauerman@linux.ibm.com>
---
 arch/powerpc/Kconfig                   |  4 ++++
 arch/powerpc/include/asm/mem_encrypt.h | 19 +++++++++++++++++++
 arch/powerpc/kernel/svm.c              | 34 ++++++++++++++++++++++++++++++++++
 3 files changed, 57 insertions(+)

diff --git a/arch/powerpc/Kconfig b/arch/powerpc/Kconfig
index 90f73d15f58a..1466d1234723 100644
--- a/arch/powerpc/Kconfig
+++ b/arch/powerpc/Kconfig
@@ -440,12 +440,16 @@ config MATH_EMULATION_HW_UNIMPLEMENTED
 
 endchoice
 
+config ARCH_HAS_MEM_ENCRYPT
+	def_bool n
+
 config PPC_SVM
        bool "Secure virtual machine (SVM) support for POWERPC"
        default n
        depends on PPC_PSERIES
        select DMA_DIRECT_OPS
        select SWIOTLB
+       select ARCH_HAS_MEM_ENCRYPT
        help
          Support secure guests on POWERPC. There are certain POWER platforms
 	 which support secure guests with the help of an Ultravisor executing
diff --git a/arch/powerpc/include/asm/mem_encrypt.h b/arch/powerpc/include/asm/mem_encrypt.h
new file mode 100644
index 000000000000..2b6e37ea446c
--- /dev/null
+++ b/arch/powerpc/include/asm/mem_encrypt.h
@@ -0,0 +1,19 @@
+/* SPDX-License-Identifier: GPL-2.0+ */
+/*
+ * SVM helper functions
+ *
+ * Copyright 2018 IBM Corporation
+ */
+
+#ifndef _ASM_POWERPC_MEM_ENCRYPT_H
+#define _ASM_POWERPC_MEM_ENCRYPT_H
+
+#define sme_me_mask	0ULL
+
+static inline bool sme_active(void) { return false; }
+static inline bool sev_active(void) { return false; }
+
+int set_memory_encrypted(unsigned long addr, int numpages);
+int set_memory_decrypted(unsigned long addr, int numpages);
+
+#endif /* _ASM_POWERPC_MEM_ENCRYPT_H */
diff --git a/arch/powerpc/kernel/svm.c b/arch/powerpc/kernel/svm.c
index 37437cf92df5..eab2a64d8643 100644
--- a/arch/powerpc/kernel/svm.c
+++ b/arch/powerpc/kernel/svm.c
@@ -7,6 +7,20 @@
  */
 
 #include <linux/mm.h>
+#include <linux/swiotlb.h>
+#include <asm/machdep.h>
+#include <asm/svm.h>
+
+static int __init init_svm(void)
+{
+	if (!is_svm_platform())
+		return 0;
+
+	swiotlb_update_mem_attributes();
+
+	return 0;
+}
+machine_early_initcall(pseries, init_svm);
 
 void mem_convert_shared(unsigned long pfn, unsigned long npages)
 {
@@ -31,3 +45,23 @@ void mem_convert_secure(unsigned long pfn, unsigned long npages)
 	 * ucall_convert_secure(paddr, size)
 	 */
 }
+
+int set_memory_encrypted(unsigned long addr, int numpages)
+{
+	if (!PAGE_ALIGNED(addr))
+		return -EINVAL;
+
+	mem_convert_secure(PHYS_PFN(__pa(addr)), numpages);
+
+	return 0;
+}
+
+int set_memory_decrypted(unsigned long addr, int numpages)
+{
+	if (!PAGE_ALIGNED(addr))
+		return -EINVAL;
+
+	mem_convert_shared(PHYS_PFN(__pa(addr)), numpages);
+
+	return 0;
+}

^ permalink raw reply	[flat|nested] 20+ messages in thread

* [RFC PATCH 05/11] powerpc/svm: Don't release SWIOTLB buffers on secure guests
  2018-08-24 16:25 [RFC PATCH 00/11] Secure Virtual Machine Enablement Thiago Jung Bauermann
                   ` (3 preceding siblings ...)
  2018-08-24 16:25 ` [RFC PATCH 04/11] powerpc/svm: Convert SWIOTLB buffers to shared memory Thiago Jung Bauermann
@ 2018-08-24 16:25 ` Thiago Jung Bauermann
  2018-08-24 16:25 ` [RFC PATCH 06/11] powerpc/svm: Use SWIOTLB DMA API for all virtio devices Thiago Jung Bauermann
                   ` (7 subsequent siblings)
  12 siblings, 0 replies; 20+ messages in thread
From: Thiago Jung Bauermann @ 2018-08-24 16:25 UTC (permalink / raw)
  To: linuxppc-dev
  Cc: iommu, linux-kernel, Alexey Kardashevskiy, Anshuman Khandual,
	Benjamin Herrenschmidt, Christoph Hellwig, Michael Ellerman,
	Mike Anderson, Paul Mackerras, Ram Pai, Anshuman Khandual,
	Thiago Jung Bauermann

From: Anshuman Khandual <khandual@linux.vnet.ibm.com>

Even though SWIOTLB slab gets allocated and initialized on each powerpc
platform with swiotlb_init(), it gets released away again on all server
platforms because ppc_swiotlb_enable variable never gets set. Secure
guests would require SWIOTLB DMA API support for virtio bounce buffering
purpose. Hence retain the allocated buffer by setting ppc_swiotlb_enable
variable for secure guests on Ultravisor platforms.

Signed-off-by: Anshuman Khandual <khandual@linux.vnet.ibm.com>
Signed-off-by: Thiago Jung Bauermann <bauerman@linux.ibm.com>
---
 arch/powerpc/kernel/svm.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/arch/powerpc/kernel/svm.c b/arch/powerpc/kernel/svm.c
index eab2a64d8643..891db2de8c04 100644
--- a/arch/powerpc/kernel/svm.c
+++ b/arch/powerpc/kernel/svm.c
@@ -16,6 +16,7 @@ static int __init init_svm(void)
 	if (!is_svm_platform())
 		return 0;
 
+	ppc_swiotlb_enable = 1;
 	swiotlb_update_mem_attributes();
 
 	return 0;

^ permalink raw reply	[flat|nested] 20+ messages in thread

* [RFC PATCH 06/11] powerpc/svm: Use SWIOTLB DMA API for all virtio devices
  2018-08-24 16:25 [RFC PATCH 00/11] Secure Virtual Machine Enablement Thiago Jung Bauermann
                   ` (4 preceding siblings ...)
  2018-08-24 16:25 ` [RFC PATCH 05/11] powerpc/svm: Don't release SWIOTLB buffers on secure guests Thiago Jung Bauermann
@ 2018-08-24 16:25 ` Thiago Jung Bauermann
  2018-08-24 16:25 ` [RFC PATCH 07/11] powerpc/svm: Use shared memory for Debug Trace Log (DTL) Thiago Jung Bauermann
                   ` (6 subsequent siblings)
  12 siblings, 0 replies; 20+ messages in thread
From: Thiago Jung Bauermann @ 2018-08-24 16:25 UTC (permalink / raw)
  To: linuxppc-dev
  Cc: iommu, linux-kernel, Alexey Kardashevskiy, Anshuman Khandual,
	Benjamin Herrenschmidt, Christoph Hellwig, Michael Ellerman,
	Mike Anderson, Paul Mackerras, Ram Pai, Anshuman Khandual,
	Thiago Jung Bauermann

From: Anshuman Khandual <khandual@linux.vnet.ibm.com>

Secure guest memory (GPA range) is isolated and inaccessible from the host.
But virtio ring transactions require the back end drivers to process
incoming scatter gather buffers which require their access in host. Hence a
portion of guest memory can be converted to shared memory and all buffers
need to be bounced into the pool before being passed on to the host.

SWIOTLB buffers can be pre-allocated and converted as shared memory during
early boot. Generic SWIOTLB DMA API (swiotlb_dma_ops) callbacks can be used
to bounce each incoming scatter gather I/O buffer addresses into this pool
of shared memory before being passed on to the host. All virtio devices on
secure guest platform need to use generic SWIOTLB DMA API. Utilize the new
virtio core platform hook platform_override_dma_ops() to achieve this.

Signed-off-by: Anshuman Khandual <khandual@linux.vnet.ibm.com>
Signed-off-by: Thiago Jung Bauermann <bauerman@linux.ibm.com>
---
 arch/powerpc/platforms/pseries/iommu.c | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/arch/powerpc/platforms/pseries/iommu.c b/arch/powerpc/platforms/pseries/iommu.c
index 5773bc7eb4bd..56b894d65dba 100644
--- a/arch/powerpc/platforms/pseries/iommu.c
+++ b/arch/powerpc/platforms/pseries/iommu.c
@@ -39,6 +39,7 @@
 #include <linux/iommu.h>
 #include <linux/rculist.h>
 #include <linux/virtio.h>
+#include <linux/virtio_config.h>
 #include <asm/io.h>
 #include <asm/prom.h>
 #include <asm/rtas.h>
@@ -51,6 +52,7 @@
 #include <asm/udbg.h>
 #include <asm/mmzone.h>
 #include <asm/plpar_wrappers.h>
+#include <asm/svm.h>
 
 #include "pseries.h"
 
@@ -1400,5 +1402,7 @@ machine_subsys_initcall_sync(pseries, tce_iommu_bus_notifier_init);
 
 void platform_override_dma_ops(struct virtio_device *vdev)
 {
-	/* Override vdev->parent.dma_ops if required */
+	if (is_svm_platform() &&
+			!virtio_has_feature(vdev, VIRTIO_F_IOMMU_PLATFORM))
+		set_dma_ops(vdev->dev.parent, &swiotlb_dma_ops);
 }

^ permalink raw reply	[flat|nested] 20+ messages in thread

* [RFC PATCH 07/11] powerpc/svm: Use shared memory for Debug Trace Log (DTL)
  2018-08-24 16:25 [RFC PATCH 00/11] Secure Virtual Machine Enablement Thiago Jung Bauermann
                   ` (5 preceding siblings ...)
  2018-08-24 16:25 ` [RFC PATCH 06/11] powerpc/svm: Use SWIOTLB DMA API for all virtio devices Thiago Jung Bauermann
@ 2018-08-24 16:25 ` Thiago Jung Bauermann
  2018-08-24 16:25 ` [RFC PATCH 08/11] powerpc: Add and use LPPACA_SIZE constant Thiago Jung Bauermann
                   ` (5 subsequent siblings)
  12 siblings, 0 replies; 20+ messages in thread
From: Thiago Jung Bauermann @ 2018-08-24 16:25 UTC (permalink / raw)
  To: linuxppc-dev
  Cc: iommu, linux-kernel, Alexey Kardashevskiy, Anshuman Khandual,
	Benjamin Herrenschmidt, Christoph Hellwig, Michael Ellerman,
	Mike Anderson, Paul Mackerras, Ram Pai, Anshuman Khandual,
	Thiago Jung Bauermann

From: Anshuman Khandual <khandual@linux.vnet.ibm.com>

On Ultravisor platform kmem_cache for DTL buffers must use a constructor
function which converts the underlying buddy allocated SLUB cache pages
into shared memory so that they are accessible to the hypervisor.

Signed-off-by: Anshuman Khandual <khandual@linux.vnet.ibm.com>
Signed-off-by: Thiago Jung Bauermann <bauerman@linux.ibm.com>
---
 arch/powerpc/include/asm/svm.h         |  1 +
 arch/powerpc/kernel/svm.c              | 30 ++++++++++++++++++++++++++++++
 arch/powerpc/platforms/pseries/setup.c |  5 ++++-
 3 files changed, 35 insertions(+), 1 deletion(-)

diff --git a/arch/powerpc/include/asm/svm.h b/arch/powerpc/include/asm/svm.h
index 95d69e472e52..e00688761704 100644
--- a/arch/powerpc/include/asm/svm.h
+++ b/arch/powerpc/include/asm/svm.h
@@ -16,6 +16,7 @@ static bool is_svm_platform(void)
 
 extern void mem_convert_shared(unsigned long pfn, unsigned long npages);
 extern void mem_convert_secure(unsigned long pfn, unsigned long npages);
+extern void dtl_cache_ctor(void *addr);
 #else
 static inline bool is_svm_platform(void)
 {
diff --git a/arch/powerpc/kernel/svm.c b/arch/powerpc/kernel/svm.c
index 891db2de8c04..1af5caa955f5 100644
--- a/arch/powerpc/kernel/svm.c
+++ b/arch/powerpc/kernel/svm.c
@@ -66,3 +66,33 @@ int set_memory_decrypted(unsigned long addr, int numpages)
 
 	return 0;
 }
+
+/* There's one dispatch log per CPU. */
+#define NR_DTL_PAGE (DISPATCH_LOG_BYTES * CONFIG_NR_CPUS / PAGE_SIZE)
+
+static struct page *dtl_page_store[NR_DTL_PAGE];
+static long dtl_nr_pages;
+
+static bool is_dtl_page_shared(struct page *page)
+{
+	long i;
+
+	for (i = 0; i < dtl_nr_pages; i++)
+		if (dtl_page_store[i] == page)
+			return true;
+
+	return false;
+}
+
+void dtl_cache_ctor(void *addr)
+{
+	unsigned long pfn = PHYS_PFN(__pa(addr));
+	struct page *page = pfn_to_page(pfn);
+
+	if (!is_dtl_page_shared(page)) {
+		dtl_page_store[dtl_nr_pages] = page;
+		dtl_nr_pages++;
+		WARN_ON(dtl_nr_pages >= NR_DTL_PAGE);
+		mem_convert_shared(pfn, PAGE_SIZE);
+	}
+}
diff --git a/arch/powerpc/platforms/pseries/setup.c b/arch/powerpc/platforms/pseries/setup.c
index 139f0af6c3d9..50ba77c802d2 100644
--- a/arch/powerpc/platforms/pseries/setup.c
+++ b/arch/powerpc/platforms/pseries/setup.c
@@ -69,6 +69,7 @@
 #include <asm/kexec.h>
 #include <asm/isa-bridge.h>
 #include <asm/security_features.h>
+#include <asm/svm.h>
 
 #include "pseries.h"
 
@@ -288,8 +289,10 @@ static inline int alloc_dispatch_logs(void)
 
 static int alloc_dispatch_log_kmem_cache(void)
 {
+	void (*ctor)(void *) = is_svm_platform() ? dtl_cache_ctor : NULL;
+
 	dtl_cache = kmem_cache_create("dtl", DISPATCH_LOG_BYTES,
-						DISPATCH_LOG_BYTES, 0, NULL);
+						DISPATCH_LOG_BYTES, 0, ctor);
 	if (!dtl_cache) {
 		pr_warn("Failed to create dispatch trace log buffer cache\n");
 		pr_warn("Stolen time statistics will be unreliable\n");

^ permalink raw reply	[flat|nested] 20+ messages in thread

* [RFC PATCH 08/11] powerpc: Add and use LPPACA_SIZE constant
  2018-08-24 16:25 [RFC PATCH 00/11] Secure Virtual Machine Enablement Thiago Jung Bauermann
                   ` (6 preceding siblings ...)
  2018-08-24 16:25 ` [RFC PATCH 07/11] powerpc/svm: Use shared memory for Debug Trace Log (DTL) Thiago Jung Bauermann
@ 2018-08-24 16:25 ` Thiago Jung Bauermann
  2018-08-24 16:25 ` [RFC PATCH 09/11] powerpc/svm: Use shared memory for LPPACA structures Thiago Jung Bauermann
                   ` (4 subsequent siblings)
  12 siblings, 0 replies; 20+ messages in thread
From: Thiago Jung Bauermann @ 2018-08-24 16:25 UTC (permalink / raw)
  To: linuxppc-dev
  Cc: iommu, linux-kernel, Alexey Kardashevskiy, Anshuman Khandual,
	Benjamin Herrenschmidt, Christoph Hellwig, Michael Ellerman,
	Mike Anderson, Paul Mackerras, Ram Pai, Thiago Jung Bauermann

Helps document what the hard-coded number means.

Suggested-by: Alexey Kardashevskiy <aik@linux.ibm.com>
Signed-off-by: Thiago Jung Bauermann <bauerman@linux.ibm.com>
---
 arch/powerpc/kernel/paca.c | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)

diff --git a/arch/powerpc/kernel/paca.c b/arch/powerpc/kernel/paca.c
index 0ee3e6d50f28..1edf8695019d 100644
--- a/arch/powerpc/kernel/paca.c
+++ b/arch/powerpc/kernel/paca.c
@@ -24,6 +24,8 @@
 #define boot_cpuid 0
 #endif
 
+#define LPPACA_SIZE 0x400
+
 static void *__init alloc_paca_data(unsigned long size, unsigned long align,
 				unsigned long limit, int cpu)
 {
@@ -70,7 +72,7 @@ static inline void init_lppaca(struct lppaca *lppaca)
 
 	*lppaca = (struct lppaca) {
 		.desc = cpu_to_be32(0xd397d781),	/* "LpPa" */
-		.size = cpu_to_be16(0x400),
+		.size = cpu_to_be16(LPPACA_SIZE),
 		.fpregs_in_use = 1,
 		.slb_count = cpu_to_be16(64),
 		.vmxregs_in_use = 0,
@@ -80,14 +82,13 @@ static inline void init_lppaca(struct lppaca *lppaca)
 static struct lppaca * __init new_lppaca(int cpu, unsigned long limit)
 {
 	struct lppaca *lp;
-	size_t size = 0x400;
 
-	BUILD_BUG_ON(size < sizeof(struct lppaca));
+	BUILD_BUG_ON(LPPACA_SIZE < sizeof(struct lppaca));
 
 	if (early_cpu_has_feature(CPU_FTR_HVMODE))
 		return NULL;
 
-	lp = alloc_paca_data(size, 0x400, limit, cpu);
+	lp = alloc_paca_data(LPPACA_SIZE, 0x400, limit, cpu);
 	init_lppaca(lp);
 
 	return lp;

^ permalink raw reply	[flat|nested] 20+ messages in thread

* [RFC PATCH 09/11] powerpc/svm: Use shared memory for LPPACA structures
  2018-08-24 16:25 [RFC PATCH 00/11] Secure Virtual Machine Enablement Thiago Jung Bauermann
                   ` (7 preceding siblings ...)
  2018-08-24 16:25 ` [RFC PATCH 08/11] powerpc: Add and use LPPACA_SIZE constant Thiago Jung Bauermann
@ 2018-08-24 16:25 ` Thiago Jung Bauermann
  2018-08-24 16:25 ` [RFC PATCH 10/11] powerpc/svm: Force the use of bounce buffers Thiago Jung Bauermann
                   ` (3 subsequent siblings)
  12 siblings, 0 replies; 20+ messages in thread
From: Thiago Jung Bauermann @ 2018-08-24 16:25 UTC (permalink / raw)
  To: linuxppc-dev
  Cc: iommu, linux-kernel, Alexey Kardashevskiy, Anshuman Khandual,
	Benjamin Herrenschmidt, Christoph Hellwig, Michael Ellerman,
	Mike Anderson, Paul Mackerras, Ram Pai, Anshuman Khandual,
	Thiago Jung Bauermann

From: Anshuman Khandual <khandual@linux.vnet.ibm.com>

LPPACA structures need to be shared with the host. Hence they need to be on
shared memory. Instead of allocating individual chunks of memory for given
structure from memblock, a contiguous chunk of memory is allocated and then
converted into shared memory. Subsequent allocation requests will come from
the contiguous chunk which will be always shared memory for all structures.

While we were able to use a kmem_cache constructor for the Debug Trace Log,
LPPACAs are allocated very early in the boot process (before SLUB is
available) so we need to use a simpler scheme here.

Signed-off-by: Anshuman Khandual <khandual@linux.vnet.ibm.com>
Signed-off-by: Thiago Jung Bauermann <bauerman@linux.ibm.com>
---
 arch/powerpc/kernel/paca.c | 34 +++++++++++++++++++++++++++++++++-
 1 file changed, 33 insertions(+), 1 deletion(-)

diff --git a/arch/powerpc/kernel/paca.c b/arch/powerpc/kernel/paca.c
index 1edf8695019d..3e2aca150ad2 100644
--- a/arch/powerpc/kernel/paca.c
+++ b/arch/powerpc/kernel/paca.c
@@ -17,6 +17,7 @@
 #include <asm/sections.h>
 #include <asm/pgtable.h>
 #include <asm/kexec.h>
+#include <asm/svm.h>
 
 #include "setup.h"
 
@@ -25,6 +26,33 @@
 #endif
 
 #define LPPACA_SIZE 0x400
+#define SHARED_LPPACA_SIZE PAGE_ALIGN(LPPACA_SIZE * CONFIG_NR_CPUS)
+
+static phys_addr_t shared_lppaca_pa;
+static unsigned long shared_lppaca_size;
+
+static void *__init alloc_shared_lppaca(unsigned long size, unsigned long align,
+				unsigned long limit, int cpu)
+{
+	unsigned long pa;
+
+	if (!shared_lppaca_pa) {
+		memblock_set_bottom_up(true);
+		shared_lppaca_pa = memblock_alloc_base_nid(SHARED_LPPACA_SIZE,
+					PAGE_SIZE, limit, -1, MEMBLOCK_NONE);
+		if (!shared_lppaca_pa)
+			panic("cannot allocate shared data");
+		memblock_set_bottom_up(false);
+		mem_convert_shared(PHYS_PFN(shared_lppaca_pa),
+				       SHARED_LPPACA_SIZE / PAGE_SIZE);
+	}
+
+	pa = shared_lppaca_pa + shared_lppaca_size;
+	shared_lppaca_size += size;
+
+	BUG_ON(shared_lppaca_size >= SHARED_LPPACA_SIZE);
+	return __va(pa);
+}
 
 static void *__init alloc_paca_data(unsigned long size, unsigned long align,
 				unsigned long limit, int cpu)
@@ -88,7 +116,11 @@ static struct lppaca * __init new_lppaca(int cpu, unsigned long limit)
 	if (early_cpu_has_feature(CPU_FTR_HVMODE))
 		return NULL;
 
-	lp = alloc_paca_data(LPPACA_SIZE, 0x400, limit, cpu);
+	if (is_svm_platform())
+		lp = alloc_shared_lppaca(LPPACA_SIZE, 0x400, limit, cpu);
+	else
+		lp = alloc_paca_data(LPPACA_SIZE, 0x400, limit, cpu);
+
 	init_lppaca(lp);
 
 	return lp;

^ permalink raw reply	[flat|nested] 20+ messages in thread

* [RFC PATCH 10/11] powerpc/svm: Force the use of bounce buffers
  2018-08-24 16:25 [RFC PATCH 00/11] Secure Virtual Machine Enablement Thiago Jung Bauermann
                   ` (8 preceding siblings ...)
  2018-08-24 16:25 ` [RFC PATCH 09/11] powerpc/svm: Use shared memory for LPPACA structures Thiago Jung Bauermann
@ 2018-08-24 16:25 ` Thiago Jung Bauermann
  2018-08-24 16:25 ` [RFC PATCH 11/11] powerpc/svm: Increase SWIOTLB buffer size Thiago Jung Bauermann
                   ` (2 subsequent siblings)
  12 siblings, 0 replies; 20+ messages in thread
From: Thiago Jung Bauermann @ 2018-08-24 16:25 UTC (permalink / raw)
  To: linuxppc-dev
  Cc: iommu, linux-kernel, Alexey Kardashevskiy, Anshuman Khandual,
	Benjamin Herrenschmidt, Christoph Hellwig, Michael Ellerman,
	Mike Anderson, Paul Mackerras, Ram Pai, Anshuman Khandual,
	Thiago Jung Bauermann

From: Anshuman Khandual <khandual@linux.vnet.ibm.com>

SWIOTLB checks range of incoming CPU addresses to be bounced and see if the
device can access it through it's DMA window without requiring bouncing. In
such cases it just chooses to skip bouncing. But for cases like secure
guests on powerpc platform all addresses need to be bounced into the shared
pool of memory because the host cannot access it otherwise. Hence the need
to do the bouncing is not related to device's DMA window. Hence force the
use of bouncing by setting the swiotlb_force variable on secure guests.

Signed-off-by: Anshuman Khandual <khandual@linux.vnet.ibm.com>
Signed-off-by: Thiago Jung Bauermann <bauerman@linux.ibm.com>
---
 arch/powerpc/kernel/svm.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/arch/powerpc/kernel/svm.c b/arch/powerpc/kernel/svm.c
index 1af5caa955f5..f0576ad65cd0 100644
--- a/arch/powerpc/kernel/svm.c
+++ b/arch/powerpc/kernel/svm.c
@@ -17,6 +17,7 @@ static int __init init_svm(void)
 		return 0;
 
 	ppc_swiotlb_enable = 1;
+	swiotlb_force = SWIOTLB_FORCE;
 	swiotlb_update_mem_attributes();
 
 	return 0;

^ permalink raw reply	[flat|nested] 20+ messages in thread

* [RFC PATCH 11/11] powerpc/svm: Increase SWIOTLB buffer size
  2018-08-24 16:25 [RFC PATCH 00/11] Secure Virtual Machine Enablement Thiago Jung Bauermann
                   ` (9 preceding siblings ...)
  2018-08-24 16:25 ` [RFC PATCH 10/11] powerpc/svm: Force the use of bounce buffers Thiago Jung Bauermann
@ 2018-08-24 16:25 ` Thiago Jung Bauermann
  2018-08-24 17:16   ` Randy Dunlap
  2018-08-27 18:18   ` Konrad Rzeszutek Wilk
  2018-08-24 16:33 ` [RFC PATCH 00/11] Secure Virtual Machine Enablement Christoph Hellwig
  2019-09-04  2:48 ` Sukadev Bhattiprolu
  12 siblings, 2 replies; 20+ messages in thread
From: Thiago Jung Bauermann @ 2018-08-24 16:25 UTC (permalink / raw)
  To: linuxppc-dev
  Cc: iommu, linux-kernel, Alexey Kardashevskiy, Anshuman Khandual,
	Benjamin Herrenschmidt, Christoph Hellwig, Michael Ellerman,
	Mike Anderson, Paul Mackerras, Ram Pai, Anshuman Khandual,
	Thiago Jung Bauermann

From: Anshuman Khandual <khandual@linux.vnet.ibm.com>

SWIOTLB buffer default size (64MB) is not enough for large sequential write
operations which eventually leads to kernel crash like here.

virtio-pci 0000:00:05.0: swiotlb buffer is full (sz: 327680 bytes)
virtio-pci 0000:00:05.0: DMA: Out of SW-IOMMU space for 327680 bytes
Kernel panic - not syncing: DMA: Random memory could be DMA read
CPU: 12 PID: 3985 Comm: mkfs.ext4 Not tainted 4.18.0-rc4+ #285
Call Trace:
[c0000007d2a27020] [c000000000cfdffc] dump_stack+0xb0/0xf4 (unreliable)
[c0000007d2a27060] [c000000000112a98] panic+0x140/0x328
[c0000007d2a270f0] [c0000000001b4f88] swiotlb_full+0x108/0x130
[c0000007d2a27180] [c0000000001b5f6c] swiotlb_map_page+0x25c/0x2c0
[c0000007d2a271e0] [c0000000007bfaf8] vring_map_one_sg.isra.0+0x58/0x70
[c0000007d2a27200] [c0000000007c08dc] virtqueue_add_sgs+0x1bc/0x690
[c0000007d2a272f0] [d0000000042a1280] virtio_queue_rq+0x358/0x4a0 [virtio_blk]
[c0000007d2a273d0] [c0000000006b5d68] blk_mq_dispatch_rq_list+0x1f8/0x6d0
..................

Increase the SWIOTLB size to 1GB on Ultravisor based secure guests.

Signed-off-by: Anshuman Khandual <khandual@linux.vnet.ibm.com>
Signed-off-by: Thiago Jung Bauermann <bauerman@linux.ibm.com>
---
 arch/powerpc/Kconfig | 5 +++++
 kernel/dma/swiotlb.c | 5 +++++
 2 files changed, 10 insertions(+)

diff --git a/arch/powerpc/Kconfig b/arch/powerpc/Kconfig
index 1466d1234723..fee7194ce9e4 100644
--- a/arch/powerpc/Kconfig
+++ b/arch/powerpc/Kconfig
@@ -457,6 +457,11 @@ config PPC_SVM
 
 	 If unsure, say "N".
 
+config SWIOTLB_DEFAULT_SIZE
+       int "Size of Software I/O TLB buffer (in MiB)"
+       default "1024"
+       depends on PPC_SVM
+
 config PPC_TRANSACTIONAL_MEM
        bool "Transactional Memory support for POWERPC"
        depends on PPC_BOOK3S_64
diff --git a/kernel/dma/swiotlb.c b/kernel/dma/swiotlb.c
index 04b68d9dffac..32dc67422d8a 100644
--- a/kernel/dma/swiotlb.c
+++ b/kernel/dma/swiotlb.c
@@ -146,8 +146,13 @@ void swiotlb_set_max_segment(unsigned int val)
 		max_segment = rounddown(val, PAGE_SIZE);
 }
 
+#ifdef CONFIG_SWIOTLB_DEFAULT_SIZE
+#define IO_TLB_DEFAULT_SIZE ((unsigned long) CONFIG_SWIOTLB_DEFAULT_SIZE << 20)
+#else
 /* default to 64MB */
 #define IO_TLB_DEFAULT_SIZE (64UL<<20)
+#endif
+
 unsigned long swiotlb_size_or_default(void)
 {
 	unsigned long size;

^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: [RFC PATCH 00/11] Secure Virtual Machine Enablement
  2018-08-24 16:25 [RFC PATCH 00/11] Secure Virtual Machine Enablement Thiago Jung Bauermann
                   ` (10 preceding siblings ...)
  2018-08-24 16:25 ` [RFC PATCH 11/11] powerpc/svm: Increase SWIOTLB buffer size Thiago Jung Bauermann
@ 2018-08-24 16:33 ` Christoph Hellwig
  2018-08-24 18:16   ` Ram Pai
  2019-09-04  2:48 ` Sukadev Bhattiprolu
  12 siblings, 1 reply; 20+ messages in thread
From: Christoph Hellwig @ 2018-08-24 16:33 UTC (permalink / raw)
  To: Thiago Jung Bauermann
  Cc: linuxppc-dev, iommu, linux-kernel, Alexey Kardashevskiy,
	Anshuman Khandual, Benjamin Herrenschmidt, Christoph Hellwig,
	Michael Ellerman, Mike Anderson, Paul Mackerras, Ram Pai

On Fri, Aug 24, 2018 at 01:25:24PM -0300, Thiago Jung Bauermann wrote:
> [ Some people didn't receive all the patches in this series, even though
>   the linuxppc-dev list did so trying to send again. This is exactly the
>   same series I posted yesterday. Sorry for the clutter. ]

I'm still only getting six of the patches.

^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: [RFC PATCH 11/11] powerpc/svm: Increase SWIOTLB buffer size
  2018-08-24 16:25 ` [RFC PATCH 11/11] powerpc/svm: Increase SWIOTLB buffer size Thiago Jung Bauermann
@ 2018-08-24 17:16   ` Randy Dunlap
       [not found]     ` <45561478-ee75-ee62-52d6-a96d60132669-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org>
  2018-08-27 18:18   ` Konrad Rzeszutek Wilk
  1 sibling, 1 reply; 20+ messages in thread
From: Randy Dunlap @ 2018-08-24 17:16 UTC (permalink / raw)
  To: Thiago Jung Bauermann, linuxppc-dev
  Cc: iommu, linux-kernel, Alexey Kardashevskiy, Anshuman Khandual,
	Benjamin Herrenschmidt, Christoph Hellwig, Michael Ellerman,
	Mike Anderson, Paul Mackerras, Ram Pai, Anshuman Khandual

On 08/24/2018 09:25 AM, Thiago Jung Bauermann wrote:
> From: Anshuman Khandual <khandual@linux.vnet.ibm.com>
> 
> SWIOTLB buffer default size (64MB) is not enough for large sequential write
> operations which eventually leads to kernel crash like here.
> 
> virtio-pci 0000:00:05.0: swiotlb buffer is full (sz: 327680 bytes)
> virtio-pci 0000:00:05.0: DMA: Out of SW-IOMMU space for 327680 bytes
> Kernel panic - not syncing: DMA: Random memory could be DMA read
> CPU: 12 PID: 3985 Comm: mkfs.ext4 Not tainted 4.18.0-rc4+ #285
> Call Trace:
> [c0000007d2a27020] [c000000000cfdffc] dump_stack+0xb0/0xf4 (unreliable)
> [c0000007d2a27060] [c000000000112a98] panic+0x140/0x328
> [c0000007d2a270f0] [c0000000001b4f88] swiotlb_full+0x108/0x130
> [c0000007d2a27180] [c0000000001b5f6c] swiotlb_map_page+0x25c/0x2c0
> [c0000007d2a271e0] [c0000000007bfaf8] vring_map_one_sg.isra.0+0x58/0x70
> [c0000007d2a27200] [c0000000007c08dc] virtqueue_add_sgs+0x1bc/0x690
> [c0000007d2a272f0] [d0000000042a1280] virtio_queue_rq+0x358/0x4a0 [virtio_blk]
> [c0000007d2a273d0] [c0000000006b5d68] blk_mq_dispatch_rq_list+0x1f8/0x6d0
> ..................
> 
> Increase the SWIOTLB size to 1GB on Ultravisor based secure guests.
> 
> Signed-off-by: Anshuman Khandual <khandual@linux.vnet.ibm.com>
> Signed-off-by: Thiago Jung Bauermann <bauerman@linux.ibm.com>
> ---
>  arch/powerpc/Kconfig | 5 +++++
>  kernel/dma/swiotlb.c | 5 +++++
>  2 files changed, 10 insertions(+)
> 
> diff --git a/arch/powerpc/Kconfig b/arch/powerpc/Kconfig
> index 1466d1234723..fee7194ce9e4 100644
> --- a/arch/powerpc/Kconfig
> +++ b/arch/powerpc/Kconfig
> @@ -457,6 +457,11 @@ config PPC_SVM
>  
>  	 If unsure, say "N".
>  
> +config SWIOTLB_DEFAULT_SIZE
> +       int "Size of Software I/O TLB buffer (in MiB)"
> +       default "1024"

I would add a "range" to limit (restrict) how small or large that can be.  E.g.:

	range 16 102400

or even smaller for the maximum value...

> +       depends on PPC_SVM
> +
>  config PPC_TRANSACTIONAL_MEM
>         bool "Transactional Memory support for POWERPC"
>         depends on PPC_BOOK3S_64
> diff --git a/kernel/dma/swiotlb.c b/kernel/dma/swiotlb.c
> index 04b68d9dffac..32dc67422d8a 100644
> --- a/kernel/dma/swiotlb.c
> +++ b/kernel/dma/swiotlb.c
> @@ -146,8 +146,13 @@ void swiotlb_set_max_segment(unsigned int val)
>  		max_segment = rounddown(val, PAGE_SIZE);
>  }
>  
> +#ifdef CONFIG_SWIOTLB_DEFAULT_SIZE
> +#define IO_TLB_DEFAULT_SIZE ((unsigned long) CONFIG_SWIOTLB_DEFAULT_SIZE << 20)
> +#else
>  /* default to 64MB */
>  #define IO_TLB_DEFAULT_SIZE (64UL<<20)
> +#endif
> +
>  unsigned long swiotlb_size_or_default(void)
>  {
>  	unsigned long size;
> 


-- 
~Randy

^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: [RFC PATCH 00/11] Secure Virtual Machine Enablement
  2018-08-24 16:33 ` [RFC PATCH 00/11] Secure Virtual Machine Enablement Christoph Hellwig
@ 2018-08-24 18:16   ` Ram Pai
  0 siblings, 0 replies; 20+ messages in thread
From: Ram Pai @ 2018-08-24 18:16 UTC (permalink / raw)
  To: Christoph Hellwig
  Cc: Thiago Jung Bauermann, linuxppc-dev, iommu, linux-kernel,
	Alexey Kardashevskiy, Anshuman Khandual, Benjamin Herrenschmidt,
	Michael Ellerman, Mike Anderson, Paul Mackerras

On Fri, Aug 24, 2018 at 06:33:30PM +0200, Christoph Hellwig wrote:
> On Fri, Aug 24, 2018 at 01:25:24PM -0300, Thiago Jung Bauermann wrote:
> > [ Some people didn't receive all the patches in this series, even though
> >   the linuxppc-dev list did so trying to send again. This is exactly the
> >   same series I posted yesterday. Sorry for the clutter. ]
> 
> I'm still only getting six of the patches.

We are all receving the entire series.  And I see that the mails have
also landed on the mailing lists.  Some filter on your side
is dropping the mails?

-- 
Ram Pai

^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: [RFC PATCH 11/11] powerpc/svm: Increase SWIOTLB buffer size
       [not found]     ` <45561478-ee75-ee62-52d6-a96d60132669-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org>
@ 2018-08-25  0:38       ` Thiago Jung Bauermann
  0 siblings, 0 replies; 20+ messages in thread
From: Thiago Jung Bauermann @ 2018-08-25  0:38 UTC (permalink / raw)
  To: Randy Dunlap
  Cc: Anshuman Khandual, Benjamin Herrenschmidt, Mike Anderson,
	Ram Pai, linux-kernel-u79uwXL29TY76Z2rM5mHXA,
	iommu-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA,
	Paul Mackerras, Michael Ellerman,
	linuxppc-dev-uLR06cmDAlY/bJ5BZ2RsiQ, Christoph Hellwig,
	Anshuman Khandual


Hello Randy,

Thanks for your quick review.

Randy Dunlap <rdunlap-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org> writes:

> On 08/24/2018 09:25 AM, Thiago Jung Bauermann wrote:
>> From: Anshuman Khandual <khandual-23VcF4HTsmIX0ybBhKVfKdBPR1lH4CV8@public.gmane.org>
>>
>> SWIOTLB buffer default size (64MB) is not enough for large sequential write
>> operations which eventually leads to kernel crash like here.
>>
>> virtio-pci 0000:00:05.0: swiotlb buffer is full (sz: 327680 bytes)
>> virtio-pci 0000:00:05.0: DMA: Out of SW-IOMMU space for 327680 bytes
>> Kernel panic - not syncing: DMA: Random memory could be DMA read
>> CPU: 12 PID: 3985 Comm: mkfs.ext4 Not tainted 4.18.0-rc4+ #285
>> Call Trace:
>> [c0000007d2a27020] [c000000000cfdffc] dump_stack+0xb0/0xf4 (unreliable)
>> [c0000007d2a27060] [c000000000112a98] panic+0x140/0x328
>> [c0000007d2a270f0] [c0000000001b4f88] swiotlb_full+0x108/0x130
>> [c0000007d2a27180] [c0000000001b5f6c] swiotlb_map_page+0x25c/0x2c0
>> [c0000007d2a271e0] [c0000000007bfaf8] vring_map_one_sg.isra.0+0x58/0x70
>> [c0000007d2a27200] [c0000000007c08dc] virtqueue_add_sgs+0x1bc/0x690
>> [c0000007d2a272f0] [d0000000042a1280] virtio_queue_rq+0x358/0x4a0 [virtio_blk]
>> [c0000007d2a273d0] [c0000000006b5d68] blk_mq_dispatch_rq_list+0x1f8/0x6d0
>> ..................
>>
>> Increase the SWIOTLB size to 1GB on Ultravisor based secure guests.
>>
>> Signed-off-by: Anshuman Khandual <khandual-23VcF4HTsmIX0ybBhKVfKdBPR1lH4CV8@public.gmane.org>
>> Signed-off-by: Thiago Jung Bauermann <bauerman-tEXmvtCZX7AybS5Ee8rs3A@public.gmane.org>
>> ---
>>  arch/powerpc/Kconfig | 5 +++++
>>  kernel/dma/swiotlb.c | 5 +++++
>>  2 files changed, 10 insertions(+)
>>
>> diff --git a/arch/powerpc/Kconfig b/arch/powerpc/Kconfig
>> index 1466d1234723..fee7194ce9e4 100644
>> --- a/arch/powerpc/Kconfig
>> +++ b/arch/powerpc/Kconfig
>> @@ -457,6 +457,11 @@ config PPC_SVM
>>
>>  	 If unsure, say "N".
>>
>> +config SWIOTLB_DEFAULT_SIZE
>> +       int "Size of Software I/O TLB buffer (in MiB)"
>> +       default "1024"
>
> I would add a "range" to limit (restrict) how small or large that can be.  E.g.:
>
> 	range 16 102400
>
> or even smaller for the maximum value...

That's an interesting idea. Since this config option is restricted to
PPC_SVM it may be possible to find out what reasonable limits would be.
I'll have to experiment a bit to find out. Though I can say that 16 MB
is too small since even with 64 MB the kernel panics with the simple
task of formatting a filesystem...

--
Thiago Jung Bauermann
IBM Linux Technology Center

^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: [RFC PATCH 11/11] powerpc/svm: Increase SWIOTLB buffer size
  2018-08-24 16:25 ` [RFC PATCH 11/11] powerpc/svm: Increase SWIOTLB buffer size Thiago Jung Bauermann
  2018-08-24 17:16   ` Randy Dunlap
@ 2018-08-27 18:18   ` Konrad Rzeszutek Wilk
  1 sibling, 0 replies; 20+ messages in thread
From: Konrad Rzeszutek Wilk @ 2018-08-27 18:18 UTC (permalink / raw)
  To: Thiago Jung Bauermann
  Cc: linuxppc-dev, iommu, linux-kernel, Alexey Kardashevskiy,
	Anshuman Khandual, Benjamin Herrenschmidt, Christoph Hellwig,
	Michael Ellerman, Mike Anderson, Paul Mackerras, Ram Pai,
	Anshuman Khandual

On Fri, Aug 24, 2018 at 01:25:35PM -0300, Thiago Jung Bauermann wrote:
> From: Anshuman Khandual <khandual@linux.vnet.ibm.com>
> 
> SWIOTLB buffer default size (64MB) is not enough for large sequential write
> operations which eventually leads to kernel crash like here.
> 
> virtio-pci 0000:00:05.0: swiotlb buffer is full (sz: 327680 bytes)
> virtio-pci 0000:00:05.0: DMA: Out of SW-IOMMU space for 327680 bytes
> Kernel panic - not syncing: DMA: Random memory could be DMA read
> CPU: 12 PID: 3985 Comm: mkfs.ext4 Not tainted 4.18.0-rc4+ #285
> Call Trace:
> [c0000007d2a27020] [c000000000cfdffc] dump_stack+0xb0/0xf4 (unreliable)
> [c0000007d2a27060] [c000000000112a98] panic+0x140/0x328
> [c0000007d2a270f0] [c0000000001b4f88] swiotlb_full+0x108/0x130
> [c0000007d2a27180] [c0000000001b5f6c] swiotlb_map_page+0x25c/0x2c0
> [c0000007d2a271e0] [c0000000007bfaf8] vring_map_one_sg.isra.0+0x58/0x70
> [c0000007d2a27200] [c0000000007c08dc] virtqueue_add_sgs+0x1bc/0x690
> [c0000007d2a272f0] [d0000000042a1280] virtio_queue_rq+0x358/0x4a0 [virtio_blk]
> [c0000007d2a273d0] [c0000000006b5d68] blk_mq_dispatch_rq_list+0x1f8/0x6d0
> ..................
> 
> Increase the SWIOTLB size to 1GB on Ultravisor based secure guests.

Gosh, that is huge.

What about making the SWIOTLB be more dynamic? That is expand it's size
dynamically if it can?

> 
> Signed-off-by: Anshuman Khandual <khandual@linux.vnet.ibm.com>
> Signed-off-by: Thiago Jung Bauermann <bauerman@linux.ibm.com>
> ---
>  arch/powerpc/Kconfig | 5 +++++
>  kernel/dma/swiotlb.c | 5 +++++
>  2 files changed, 10 insertions(+)
> 
> diff --git a/arch/powerpc/Kconfig b/arch/powerpc/Kconfig
> index 1466d1234723..fee7194ce9e4 100644
> --- a/arch/powerpc/Kconfig
> +++ b/arch/powerpc/Kconfig
> @@ -457,6 +457,11 @@ config PPC_SVM
>  
>  	 If unsure, say "N".
>  
> +config SWIOTLB_DEFAULT_SIZE
> +       int "Size of Software I/O TLB buffer (in MiB)"
> +       default "1024"
> +       depends on PPC_SVM
> +
>  config PPC_TRANSACTIONAL_MEM
>         bool "Transactional Memory support for POWERPC"
>         depends on PPC_BOOK3S_64
> diff --git a/kernel/dma/swiotlb.c b/kernel/dma/swiotlb.c
> index 04b68d9dffac..32dc67422d8a 100644
> --- a/kernel/dma/swiotlb.c
> +++ b/kernel/dma/swiotlb.c
> @@ -146,8 +146,13 @@ void swiotlb_set_max_segment(unsigned int val)
>  		max_segment = rounddown(val, PAGE_SIZE);
>  }
>  
> +#ifdef CONFIG_SWIOTLB_DEFAULT_SIZE
> +#define IO_TLB_DEFAULT_SIZE ((unsigned long) CONFIG_SWIOTLB_DEFAULT_SIZE << 20)
> +#else
>  /* default to 64MB */
>  #define IO_TLB_DEFAULT_SIZE (64UL<<20)
> +#endif
> +
>  unsigned long swiotlb_size_or_default(void)
>  {
>  	unsigned long size;
> 

^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: [RFC PATCH 00/11] Secure Virtual Machine Enablement
  2018-08-24 16:25 [RFC PATCH 00/11] Secure Virtual Machine Enablement Thiago Jung Bauermann
                   ` (11 preceding siblings ...)
  2018-08-24 16:33 ` [RFC PATCH 00/11] Secure Virtual Machine Enablement Christoph Hellwig
@ 2019-09-04  2:48 ` Sukadev Bhattiprolu
  12 siblings, 0 replies; 20+ messages in thread
From: Sukadev Bhattiprolu @ 2019-09-04  2:48 UTC (permalink / raw)
  To: Thiago Jung Bauermann
  Cc: Anshuman Khandual, Mike Anderson, Ram Pai, linux-kernel, iommu,
	Paul Mackerras, linuxppc-dev, Christoph Hellwig

Thiago Jung Bauermann [bauerman@linux.ibm.com] wrote:
> [ Some people didn't receive all the patches in this series, even though
>   the linuxppc-dev list did so trying to send again. This is exactly the
>   same series I posted yesterday. Sorry for the clutter. ]
> 
> This series contains preliminary work to enable Secure Virtual Machines
> (SVM) on powerpc. SVMs request to be migrated to secure memory very early in
> the boot process (in prom_init()), so by default all of their memory is
> inaccessible to the hypervisor. There is an ultravisor call that the VM can
> use to request certain pages to be made accessible (aka shared).

I would like to piggy-back on this series (since it provides the
context) to add another patch we need for SVMs :-) 

Appreciate any comments. 

---

From ed93a0e36ec886483a72fdb8d99380bbe6607f37 Mon Sep 17 00:00:00 2001
From: Sukadev Bhattiprolu <sukadev@linux.vnet.ibm.com>
Date: Thu, 16 May 2019 20:57:12 -0500
Subject: [PATCH 1/1] powerpc/pseries/svm: don't access some SPRs

Ultravisor disables some CPU features like EBB and BHRB in the HFSCR
for secure virtual machines (SVMs). If the SVMs attempt to access
related registers, they will get a Program Interrupt.

Use macros/wrappers to skip accessing EBB and BHRB related registers
for secure VMs.

Signed-off-by: Sukadev Bhattiprolu <sukadev@linux.ibm.com>
---
 arch/powerpc/include/asm/reg.h          | 35 ++++++++++++++++++++++++
 arch/powerpc/kernel/process.c           | 12 ++++-----
 arch/powerpc/kvm/book3s_hv.c            | 24 ++++++++---------
 arch/powerpc/kvm/book3s_hv_rmhandlers.S | 48 ++++++++++++++++++++++++---------
 arch/powerpc/kvm/book3s_hv_tm_builtin.c |  6 ++---
 arch/powerpc/perf/core-book3s.c         |  4 +--
 arch/powerpc/xmon/xmon.c                |  2 +-
 7 files changed, 95 insertions(+), 36 deletions(-)

diff --git a/arch/powerpc/include/asm/reg.h b/arch/powerpc/include/asm/reg.h
index ec3714c..1397cb3 100644
--- a/arch/powerpc/include/asm/reg.h
+++ b/arch/powerpc/include/asm/reg.h
@@ -1376,6 +1376,41 @@ static inline void msr_check_and_clear(unsigned long bits)
 		__msr_check_and_clear(bits);
 }
 
+#ifdef CONFIG_PPC_SVM
+/*
+ * Move from some "restricted" sprs.
+ * Secure VMs should not access some registers as the related features
+ * are disabled in the CPU. If an SVM is attempting read from the given
+ * SPR, return 0. Otherwise behave like a normal mfspr.
+ */
+#define mfspr_r(rn)						\
+({								\
+	unsigned long rval = 0ULL;				\
+								\
+	if (!(mfmsr() & MSR_S))					\
+		asm volatile("mfspr %0," __stringify(rn)	\
+				: "=r" (rval));			\
+	rval;							\
+})
+
+/*
+ * Move to some "restricted" sprs.
+ * Secure VMs should not access some registers as the related features
+ * are disabled in the CPU. If an SVM is attempting write to the given
+ * SPR, ignore the write. Otherwise behave like a normal mtspr.
+ */
+#define mtspr_r(rn, v)					\
+({								\
+	if (!(mfmsr() & MSR_S))					\
+		asm volatile("mtspr " __stringify(rn) ",%0" :	\
+				     : "r" ((unsigned long)(v)) \
+				     : "memory");		\
+})
+#else
+#define mfspr_r		mfspr
+#define mtspr_r		mtspr
+#endif
+
 #ifdef __powerpc64__
 #if defined(CONFIG_PPC_CELL) || defined(CONFIG_PPC_FSL_BOOK3E)
 #define mftb()		({unsigned long rval;				\
diff --git a/arch/powerpc/kernel/process.c b/arch/powerpc/kernel/process.c
index 8fc4de0..d5e7386 100644
--- a/arch/powerpc/kernel/process.c
+++ b/arch/powerpc/kernel/process.c
@@ -1072,9 +1072,9 @@ static inline void save_sprs(struct thread_struct *t)
 		t->dscr = mfspr(SPRN_DSCR);
 
 	if (cpu_has_feature(CPU_FTR_ARCH_207S)) {
-		t->bescr = mfspr(SPRN_BESCR);
-		t->ebbhr = mfspr(SPRN_EBBHR);
-		t->ebbrr = mfspr(SPRN_EBBRR);
+		t->bescr = mfspr_r(SPRN_BESCR);
+		t->ebbhr = mfspr_r(SPRN_EBBHR);
+		t->ebbrr = mfspr_r(SPRN_EBBRR);
 
 		t->fscr = mfspr(SPRN_FSCR);
 
@@ -1111,11 +1111,11 @@ static inline void restore_sprs(struct thread_struct *old_thread,
 
 	if (cpu_has_feature(CPU_FTR_ARCH_207S)) {
 		if (old_thread->bescr != new_thread->bescr)
-			mtspr(SPRN_BESCR, new_thread->bescr);
+			mtspr_r(SPRN_BESCR, new_thread->bescr);
 		if (old_thread->ebbhr != new_thread->ebbhr)
-			mtspr(SPRN_EBBHR, new_thread->ebbhr);
+			mtspr_r(SPRN_EBBHR, new_thread->ebbhr);
 		if (old_thread->ebbrr != new_thread->ebbrr)
-			mtspr(SPRN_EBBRR, new_thread->ebbrr);
+			mtspr_r(SPRN_EBBRR, new_thread->ebbrr);
 
 		if (old_thread->fscr != new_thread->fscr)
 			mtspr(SPRN_FSCR, new_thread->fscr);
diff --git a/arch/powerpc/kvm/book3s_hv.c b/arch/powerpc/kvm/book3s_hv.c
index 8304ee2..91f4db2 100644
--- a/arch/powerpc/kvm/book3s_hv.c
+++ b/arch/powerpc/kvm/book3s_hv.c
@@ -3584,9 +3584,9 @@ int kvmhv_p9_guest_entry(struct kvm_vcpu *vcpu, u64 time_limit,
 	mtspr(SPRN_PSPB, vcpu->arch.pspb);
 	mtspr(SPRN_FSCR, vcpu->arch.fscr);
 	mtspr(SPRN_TAR, vcpu->arch.tar);
-	mtspr(SPRN_EBBHR, vcpu->arch.ebbhr);
-	mtspr(SPRN_EBBRR, vcpu->arch.ebbrr);
-	mtspr(SPRN_BESCR, vcpu->arch.bescr);
+	mtspr_r(SPRN_EBBHR, vcpu->arch.ebbhr);
+	mtspr_r(SPRN_EBBRR, vcpu->arch.ebbrr);
+	mtspr_r(SPRN_BESCR, vcpu->arch.bescr);
 	mtspr(SPRN_WORT, vcpu->arch.wort);
 	mtspr(SPRN_TIDR, vcpu->arch.tid);
 	mtspr(SPRN_DAR, vcpu->arch.shregs.dar);
@@ -3657,9 +3657,9 @@ int kvmhv_p9_guest_entry(struct kvm_vcpu *vcpu, u64 time_limit,
 	vcpu->arch.pspb = mfspr(SPRN_PSPB);
 	vcpu->arch.fscr = mfspr(SPRN_FSCR);
 	vcpu->arch.tar = mfspr(SPRN_TAR);
-	vcpu->arch.ebbhr = mfspr(SPRN_EBBHR);
-	vcpu->arch.ebbrr = mfspr(SPRN_EBBRR);
-	vcpu->arch.bescr = mfspr(SPRN_BESCR);
+	vcpu->arch.ebbhr = mfspr_r(SPRN_EBBHR);
+	vcpu->arch.ebbrr = mfspr_r(SPRN_EBBRR);
+	vcpu->arch.bescr = mfspr_r(SPRN_BESCR);
 	vcpu->arch.wort = mfspr(SPRN_WORT);
 	vcpu->arch.tid = mfspr(SPRN_TIDR);
 	vcpu->arch.amr = mfspr(SPRN_AMR);
@@ -4288,9 +4288,9 @@ static int kvmppc_vcpu_run_hv(struct kvm_run *run, struct kvm_vcpu *vcpu)
 
 	/* Save userspace EBB and other register values */
 	if (cpu_has_feature(CPU_FTR_ARCH_207S)) {
-		ebb_regs[0] = mfspr(SPRN_EBBHR);
-		ebb_regs[1] = mfspr(SPRN_EBBRR);
-		ebb_regs[2] = mfspr(SPRN_BESCR);
+		ebb_regs[0] = mfspr_r(SPRN_EBBHR);
+		ebb_regs[1] = mfspr_r(SPRN_EBBRR);
+		ebb_regs[2] = mfspr_r(SPRN_BESCR);
 		user_tar = mfspr(SPRN_TAR);
 	}
 	user_vrsave = mfspr(SPRN_VRSAVE);
@@ -4336,9 +4336,9 @@ static int kvmppc_vcpu_run_hv(struct kvm_run *run, struct kvm_vcpu *vcpu)
 
 	/* Restore userspace EBB and other register values */
 	if (cpu_has_feature(CPU_FTR_ARCH_207S)) {
-		mtspr(SPRN_EBBHR, ebb_regs[0]);
-		mtspr(SPRN_EBBRR, ebb_regs[1]);
-		mtspr(SPRN_BESCR, ebb_regs[2]);
+		mtspr_r(SPRN_EBBHR, ebb_regs[0]);
+		mtspr_r(SPRN_EBBRR, ebb_regs[1]);
+		mtspr_r(SPRN_BESCR, ebb_regs[2]);
 		mtspr(SPRN_TAR, user_tar);
 		mtspr(SPRN_FSCR, current->thread.fscr);
 	}
diff --git a/arch/powerpc/kvm/book3s_hv_rmhandlers.S b/arch/powerpc/kvm/book3s_hv_rmhandlers.S
index f4399b0..4cbf8ca 100644
--- a/arch/powerpc/kvm/book3s_hv_rmhandlers.S
+++ b/arch/powerpc/kvm/book3s_hv_rmhandlers.S
@@ -808,15 +808,27 @@ END_FTR_SECTION_IFCLR(CPU_FTR_ARCH_207S)
 	mtspr	SPRN_CIABR, r7
 	mtspr	SPRN_TAR, r8
 	ld	r5, VCPU_IC(r4)
-	ld	r8, VCPU_EBBHR(r4)
 	mtspr	SPRN_IC, r5
-	mtspr	SPRN_EBBHR, r8
-	ld	r5, VCPU_EBBRR(r4)
-	ld	r6, VCPU_BESCR(r4)
+
+	/* EBB and TM are disabled for secure VMs so skip them */
+	mfmsr   r8
+	andis.  r8, r8, MSR_S@high
+	bne     clear_ebb0
+	ld      r5, VCPU_EBBHR(r4)
+	ld      r6, VCPU_EBBRR(r4)
+	ld      r7, VCPU_BESCR(r4)
+	b       store_ebb0
+clear_ebb0:
+	li      r5, 0
+	li      r6, 0
+	li      r7, 0
+store_ebb0:
+	mtspr   SPRN_EBBHR, r5
+	mtspr   SPRN_EBBRR, r6
+	mtspr   SPRN_BESCR, r7
+
 	lwz	r7, VCPU_GUEST_PID(r4)
 	ld	r8, VCPU_WORT(r4)
-	mtspr	SPRN_EBBRR, r5
-	mtspr	SPRN_BESCR, r6
 	mtspr	SPRN_PID, r7
 	mtspr	SPRN_WORT, r8
 BEGIN_FTR_SECTION
@@ -1611,14 +1623,26 @@ END_FTR_SECTION_IFCLR(CPU_FTR_ARCH_207S)
 	mfspr	r7, SPRN_TAR
 	std	r5, VCPU_IC(r9)
 	std	r7, VCPU_TAR(r9)
-	mfspr	r8, SPRN_EBBHR
-	std	r8, VCPU_EBBHR(r9)
-	mfspr	r5, SPRN_EBBRR
-	mfspr	r6, SPRN_BESCR
+
+	/* EBB and TM are disabled for secure VMs so skip them */
+	mfmsr   r8
+	andis.  r8, r8, MSR_S@high
+	bne     clear_ebb1
+	mfspr   r5, SPRN_EBBHR
+	mfspr   r6, SPRN_EBBRR
+	mfspr   r7, SPRN_BESCR
+	b       store_ebb1
+clear_ebb1:
+	li      r5, 0
+	li      r6, 0
+	li      r7, 0
+store_ebb1:
+	std     r5, VCPU_EBBHR(r9)
+	std     r6, VCPU_EBBRR(r9)
+	std     r7, VCPU_BESCR(r9)
+
 	mfspr	r7, SPRN_PID
 	mfspr	r8, SPRN_WORT
-	std	r5, VCPU_EBBRR(r9)
-	std	r6, VCPU_BESCR(r9)
 	stw	r7, VCPU_GUEST_PID(r9)
 	std	r8, VCPU_WORT(r9)
 BEGIN_FTR_SECTION
diff --git a/arch/powerpc/kvm/book3s_hv_tm_builtin.c b/arch/powerpc/kvm/book3s_hv_tm_builtin.c
index 2172462..bc3071c 100644
--- a/arch/powerpc/kvm/book3s_hv_tm_builtin.c
+++ b/arch/powerpc/kvm/book3s_hv_tm_builtin.c
@@ -45,18 +45,18 @@ int kvmhv_p9_tm_emulation_early(struct kvm_vcpu *vcpu)
 		if (!(vcpu->arch.hfscr & HFSCR_EBB) ||
 		    ((msr & MSR_PR) && !(mfspr(SPRN_FSCR) & FSCR_EBB)))
 			return 0;
-		bescr = mfspr(SPRN_BESCR);
+		bescr = mfspr_r(SPRN_BESCR);
 		/* expect to see a S->T transition requested */
 		if (((bescr >> 30) & 3) != 2)
 			return 0;
 		bescr &= ~BESCR_GE;
 		if (instr & (1 << 11))
 			bescr |= BESCR_GE;
-		mtspr(SPRN_BESCR, bescr);
+		mtspr_r(SPRN_BESCR, bescr);
 		msr = (msr & ~MSR_TS_MASK) | MSR_TS_T;
 		vcpu->arch.shregs.msr = msr;
 		vcpu->arch.cfar = vcpu->arch.regs.nip - 4;
-		vcpu->arch.regs.nip = mfspr(SPRN_EBBRR);
+		vcpu->arch.regs.nip = mfspr_r(SPRN_EBBRR);
 		return 1;
 
 	case PPC_INST_MTMSRD:
diff --git a/arch/powerpc/perf/core-book3s.c b/arch/powerpc/perf/core-book3s.c
index ca92e01..e51b2c9 100644
--- a/arch/powerpc/perf/core-book3s.c
+++ b/arch/powerpc/perf/core-book3s.c
@@ -846,9 +846,9 @@ void perf_event_print_debug(void)
 
 	if (ppmu->flags & PPMU_ARCH_207S) {
 		pr_info("MMCR2: %016lx EBBHR: %016lx\n",
-			mfspr(SPRN_MMCR2), mfspr(SPRN_EBBHR));
+			mfspr(SPRN_MMCR2), mfspr_r(SPRN_EBBHR));
 		pr_info("EBBRR: %016lx BESCR: %016lx\n",
-			mfspr(SPRN_EBBRR), mfspr(SPRN_BESCR));
+			mfspr_r(SPRN_EBBRR), mfspr_r(SPRN_BESCR));
 	}
 #endif
 	pr_info("SIAR:  %016lx SDAR:  %016lx SIER:  %016lx\n",
diff --git a/arch/powerpc/xmon/xmon.c b/arch/powerpc/xmon/xmon.c
index 14e56c2..20b3431 100644
--- a/arch/powerpc/xmon/xmon.c
+++ b/arch/powerpc/xmon/xmon.c
@@ -1871,7 +1871,7 @@ static void dump_207_sprs(void)
 	printf("sdar   = %.16lx   sier = %.16lx pmc6   = %.8lx\n",
 		mfspr(SPRN_SDAR), mfspr(SPRN_SIER), mfspr(SPRN_PMC6));
 	printf("ebbhr  = %.16lx  ebbrr = %.16lx bescr  = %.16lx\n",
-		mfspr(SPRN_EBBHR), mfspr(SPRN_EBBRR), mfspr(SPRN_BESCR));
+		mfspr_r(SPRN_EBBHR), mfspr_r(SPRN_EBBRR), mfspr_r(SPRN_BESCR));
 	printf("iamr   = %.16lx\n", mfspr(SPRN_IAMR));
 
 	if (!(msr & MSR_HV))
-- 
1.8.3.1

_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: [RFC PATCH 00/11] Secure Virtual Machine Enablement
       [not found] ` <20180824025933.24319-1-bauerman-tEXmvtCZX7AybS5Ee8rs3A@public.gmane.org>
@ 2018-08-24  6:00   ` Christoph Hellwig
  0 siblings, 0 replies; 20+ messages in thread
From: Christoph Hellwig @ 2018-08-24  6:00 UTC (permalink / raw)
  To: Thiago Jung Bauermann
  Cc: Anshuman Khandual, Benjamin Herrenschmidt, Mike Anderson,
	Ram Pai, linux-kernel-u79uwXL29TY76Z2rM5mHXA,
	iommu-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA,
	Paul Mackerras, Michael Ellerman,
	linuxppc-dev-uLR06cmDAlY/bJ5BZ2RsiQ, Christoph Hellwig

It seems like you only sent out 6 our of the actual 11 patches according
to the numbering, please resend the full series.

^ permalink raw reply	[flat|nested] 20+ messages in thread

* [RFC PATCH 00/11] Secure Virtual Machine Enablement
@ 2018-08-24  2:59 Thiago Jung Bauermann
       [not found] ` <20180824025933.24319-1-bauerman-tEXmvtCZX7AybS5Ee8rs3A@public.gmane.org>
  0 siblings, 1 reply; 20+ messages in thread
From: Thiago Jung Bauermann @ 2018-08-24  2:59 UTC (permalink / raw)
  To: linuxppc-dev
  Cc: iommu, linux-kernel, Alexey Kardashevskiy, Anshuman Khandual,
	Benjamin Herrenschmidt, Christoph Hellwig, Michael Ellerman,
	Mike Anderson, Paul Mackerras, Ram Pai, Thiago Jung Bauermann

This series contains preliminary work to enable Secure Virtual Machines
(SVM) on powerpc. SVMs request to be migrated to secure memory very early in
the boot process (in prom_init()), so by default all of their memory is
inaccessible to the hypervisor. There is an ultravisor call that the VM can
use to request certain pages to be made accessible (aka shared).

The objective of these patches is to have the guest perform this request for
buffers that need to be shared with the hypervisor, such as the LPPACAs, the
SWIOTLB buffer and the Debug Trace Log. This work is incomplete: there are
surely other memory regions that need to be made accessible, but I'm posting
it early to get comments on whether the approach being taken is appropriate.

It should be applied on top of the generic virtio DMA API rework series
posted earlier, which adds a platform hook to override any arch based DMA
API operations for any virtio device:

https://lists.ozlabs.org/pipermail/linuxppc-dev/2018-July/175994.html

I'm aware that changes need to be made to the patch series above, but IIUC
it depends on upcoming virtio cleanup from Christoph Hellwig so for now the
patch series above will be used as a stepping stone for this series.

This code has been tested with virtio block, net and scsi devices with and
without VIRTIO_F_IOMMU_PLATFORM flag. Please let me know what you think.

For now I am testing on a regular guest with a couple of patches on top
forcing is_svm_platform() to always return true and adding debug messages to
confirm that mem_convert_shared() is being called in the expected places.

These are the commands I'm using to start up the guest:

Without VIRTIO_F_IOMMU_PLATFORM:

qemu-system-ppc64 \
  -enable-kvm \
  -kernel /home/bauermann/src/linux/arch/powerpc/boot/zImage \
  -append "root=PARTUUID=e550ad6f-05 ro" \
  -machine pseries-2.6 \
  -m 8G \
  -smp 2 \
  -serial mon:stdio \
  -nographic \
  -nodefaults \
  -device virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x4 \
  -drive file=/home/bauermann/VMs/svm.qcow2,format=qcow2,if=none,id=drive-scsi0-0-0-0 \
  -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0 \
  -drive file=/home/bauermann/VMs/svm-blk.qcow2,format=qcow2,if=none,id=drive-virtio-disk0 \
  -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0 \
  -device virtio-net,netdev=hostnet0,id=net0,mac=52:54:00:96:70:1f \
  -netdev user,id=hostnet0 \
  -set netdev.hostnet0.hostfwd=tcp::42022-:22

With VIRTIO_F_IOMMU_PLATFORM. Same as above plus some -global options so
that the virtio devices use the modern interface rather than the
transitional one:

qemu-system-ppc64 \
  -enable-kvm \
  -kernel /home/bauermann/src/linux/arch/powerpc/boot/zImage \
  -append "root=PARTUUID=e550ad6f-05 ro" \
  -machine pseries-2.6 \
  -m 8G \
  -smp 2 \
  -serial mon:stdio \
  -nographic \
  -nodefaults \
  -device virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x4 \
  -drive file=/home/bauermann/VMs/svm.qcow2,format=qcow2,if=none,id=drive-scsi0-0-0-0 \
  -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0 \
  -drive file=/home/bauermann/VMs/svm-blk.qcow2,format=qcow2,if=none,id=drive-virtio-disk0 \
  -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0 \
  -device virtio-net,netdev=hostnet0,id=net0,mac=52:54:00:96:70:1f \
  -netdev user,id=hostnet0 \
  -set netdev.hostnet0.hostfwd=tcp::42022-:22 \
  -global virtio-blk-pci.iommu_platform=true \
  -global virtio-blk-pci.disable-legacy=on \
  -global virtio-blk-pci.disable-modern=off \
  -global virtio-net-pci.iommu_platform=true \
  -global virtio-net-pci.disable-legacy=on \
  -global virtio-net-pci.disable-modern=off \
  -global virtio-scsi-pci.iommu_platform=true \
  -global virtio-scsi-pci.disable-legacy=on \
  -global virtio-scsi-pci.disable-modern=off

The code was tested with a couple of other permutations where one virtio
device has the flag VIRTIO_F_IOMMU_PLATFORM and others don't. Please suggest
some other scenarios which need to be tested as well.

Anshuman Khandual (10):
  powerpc/svm: Detect Secure Virtual Machine (SVM) platform
  powerpc/svm: Select CONFIG_DMA_DIRECT_OPS and CONFIG_SWIOTLB
  powerpc/svm: Add memory conversion (shared/secure) helper functions
  powerpc/svm: Convert SWIOTLB buffers to shared memory
  powerpc/svm: Don't release SWIOTLB buffers on secure guests
  powerpc/svm: Use SWIOTLB DMA API for all virtio devices
  powerpc/svm: Use shared memory for Debug Trace Log (DTL)
  powerpc/svm: Use shared memory for LPPACA structures
  powerpc/svm: Force the use of bounce buffers
  powerpc/svm: Increase SWIOTLB buffer size

Thiago Jung Bauermann (1):
  powerpc: Add and use LPPACA_SIZE constant

 arch/powerpc/Kconfig                   | 22 ++++++++
 arch/powerpc/include/asm/mem_encrypt.h | 19 +++++++
 arch/powerpc/include/asm/reg.h         |  3 ++
 arch/powerpc/include/asm/svm.h         | 26 +++++++++
 arch/powerpc/kernel/Makefile           |  1 +
 arch/powerpc/kernel/paca.c             | 41 ++++++++++++--
 arch/powerpc/kernel/svm.c              | 99 ++++++++++++++++++++++++++++++++++
 arch/powerpc/platforms/pseries/iommu.c |  6 ++-
 arch/powerpc/platforms/pseries/setup.c |  5 +-
 kernel/dma/swiotlb.c                   |  5 ++
 10 files changed, 221 insertions(+), 6 deletions(-)
 create mode 100644 arch/powerpc/include/asm/mem_encrypt.h
 create mode 100644 arch/powerpc/include/asm/svm.h
 create mode 100644 arch/powerpc/kernel/svm.c

^ permalink raw reply	[flat|nested] 20+ messages in thread

end of thread, back to index

Thread overview: 20+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-08-24 16:25 [RFC PATCH 00/11] Secure Virtual Machine Enablement Thiago Jung Bauermann
2018-08-24 16:25 ` [RFC PATCH 01/11] powerpc/svm: Detect Secure Virtual Machine (SVM) platform Thiago Jung Bauermann
2018-08-24 16:25 ` [RFC PATCH 02/11] powerpc/svm: Select CONFIG_DMA_DIRECT_OPS and CONFIG_SWIOTLB Thiago Jung Bauermann
2018-08-24 16:25 ` [RFC PATCH 03/11] powerpc/svm: Add memory conversion (shared/secure) helper functions Thiago Jung Bauermann
2018-08-24 16:25 ` [RFC PATCH 04/11] powerpc/svm: Convert SWIOTLB buffers to shared memory Thiago Jung Bauermann
2018-08-24 16:25 ` [RFC PATCH 05/11] powerpc/svm: Don't release SWIOTLB buffers on secure guests Thiago Jung Bauermann
2018-08-24 16:25 ` [RFC PATCH 06/11] powerpc/svm: Use SWIOTLB DMA API for all virtio devices Thiago Jung Bauermann
2018-08-24 16:25 ` [RFC PATCH 07/11] powerpc/svm: Use shared memory for Debug Trace Log (DTL) Thiago Jung Bauermann
2018-08-24 16:25 ` [RFC PATCH 08/11] powerpc: Add and use LPPACA_SIZE constant Thiago Jung Bauermann
2018-08-24 16:25 ` [RFC PATCH 09/11] powerpc/svm: Use shared memory for LPPACA structures Thiago Jung Bauermann
2018-08-24 16:25 ` [RFC PATCH 10/11] powerpc/svm: Force the use of bounce buffers Thiago Jung Bauermann
2018-08-24 16:25 ` [RFC PATCH 11/11] powerpc/svm: Increase SWIOTLB buffer size Thiago Jung Bauermann
2018-08-24 17:16   ` Randy Dunlap
     [not found]     ` <45561478-ee75-ee62-52d6-a96d60132669-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org>
2018-08-25  0:38       ` Thiago Jung Bauermann
2018-08-27 18:18   ` Konrad Rzeszutek Wilk
2018-08-24 16:33 ` [RFC PATCH 00/11] Secure Virtual Machine Enablement Christoph Hellwig
2018-08-24 18:16   ` Ram Pai
2019-09-04  2:48 ` Sukadev Bhattiprolu
  -- strict thread matches above, loose matches on Subject: below --
2018-08-24  2:59 Thiago Jung Bauermann
     [not found] ` <20180824025933.24319-1-bauerman-tEXmvtCZX7AybS5Ee8rs3A@public.gmane.org>
2018-08-24  6:00   ` Christoph Hellwig

IOMMU Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/linux-iommu/0 linux-iommu/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 linux-iommu linux-iommu/ https://lore.kernel.org/linux-iommu \
		iommu@lists.linux-foundation.org iommu@archiver.kernel.org
	public-inbox-index linux-iommu


Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.linux-foundation.lists.iommu


AGPL code for this site: git clone https://public-inbox.org/ public-inbox