[RFCv2 00/37] KVM: s390: Add support for protected VMs

[RFCv2 00/37] KVM: s390: Add support for protected VMs

[Christian Borntraeger]

Protected VMs (PVM) are KVM VMs, where KVM can't access the VM's state
like guest memory and guest registers anymore. Instead the PVMs are
mostly managed by a new entity called Ultravisor (UV), which provides
an API, so KVM and the PV can request management actions.

PVMs are encrypted at rest and protected from hypervisor access while
running. They switch from a normal operation into protected mode, so
we can still use the standard boot process to load a encrypted blob
and then move it into protected mode.

Rebooting is only possible by passing through the unprotected/normal
mode and switching to protected again.

All patches are in the protvirtv2 branch of the korg s390 kvm git
(on top of Janoschs reset rework).

Claudio presented the technology at his presentation at KVM Forum
2019.

This contains a "pretty small" common code memory management change that
will allow paging, guest backing with files etc almost just like normal
VMs. Please note that the memory management part will still see some
changes to deal with a corner case for the adapter interrupt indicator
pages. So please focus on the non-mm parts (which hopefully has
everthing addressed in the next version). Claudio will work with Andrea
regarding this.

Christian Borntraeger (3):
  KVM: s390/mm: Make pages accessible before destroying the guest
  KVM: s390: protvirt: Add SCLP interrupt handling
  KVM: s390: protvirt: do not inject interrupts after start

Claudio Imbrenda (3):
  mm:gup/writeback: add callbacks for inaccessible pages
  s390/mm: provide memory management functions for protected KVM guests
  KVM: s390/mm: handle guest unpin events

Janosch Frank (24):
  DOCUMENTATION: protvirt: Protected virtual machine introduction
  KVM: s390: add new variants of UV CALL
  KVM: s390: protvirt: Add initial lifecycle handling
  KVM: s390: protvirt: Add KVM api documentation
  KVM: s390: protvirt: Secure memory is not mergeable
  KVM: s390: protvirt: Handle SE notification interceptions
  KVM: s390: protvirt: Instruction emulation
  KVM: s390: protvirt: Handle spec exception loops
  KVM: s390: protvirt: Add new gprs location handling
  KVM: S390: protvirt: Introduce instruction data area bounce buffer
  KVM: s390: protvirt: handle secure guest prefix pages
  KVM: s390: protvirt: Write sthyi data to instruction data area
  KVM: s390: protvirt: STSI handling
  KVM: s390: protvirt: disallow one_reg
  KVM: s390: protvirt: Only sync fmt4 registers
  KVM: s390: protvirt: Add program exception injection
  DOCUMENTATION: protvirt: Diag 308 IPL
  KVM: s390: protvirt: Add diag 308 subcode 8 - 10 handling
  KVM: s390: protvirt: UV calls diag308 0, 1
  KVM: s390: protvirt: Report CPU state to Ultravisor
  KVM: s390: protvirt: Support cmd 5 operation state
  KVM: s390: protvirt: Add UV debug trace
  KVM: s390: protvirt: Mask PSW interrupt bits for interception 104 and
    112
  KVM: s390: protvirt: Add UV cpu reset calls

Michael Mueller (4):
  KVM: s390: protvirt: Add interruption injection controls
  KVM: s390: protvirt: Implement interruption injection
  KVM: s390: protvirt: Add machine-check interruption injection controls
  KVM: s390: protvirt: Implement machine-check interruption injection

Vasily Gorbik (3):
  s390/protvirt: introduce host side setup
  s390/protvirt: add ultravisor initialization
  s390: add (non)secure page access exceptions handlers

 .../admin-guide/kernel-parameters.txt         |   5 +
 Documentation/virt/kvm/api.txt                |  62 +++
 Documentation/virt/kvm/s390-pv-boot.rst       |  64 +++
 Documentation/virt/kvm/s390-pv.rst            | 103 ++++
 MAINTAINERS                                   |   1 +
 arch/s390/boot/Makefile                       |   2 +-
 arch/s390/boot/uv.c                           |  20 +-
 arch/s390/include/asm/gmap.h                  |   3 +
 arch/s390/include/asm/kvm_host.h              | 112 +++-
 arch/s390/include/asm/mmu.h                   |   2 +
 arch/s390/include/asm/mmu_context.h           |   1 +
 arch/s390/include/asm/page.h                  |   5 +
 arch/s390/include/asm/pgtable.h               |  35 +-
 arch/s390/include/asm/uv.h                    | 265 +++++++++-
 arch/s390/kernel/Makefile                     |   1 +
 arch/s390/kernel/pgm_check.S                  |   4 +-
 arch/s390/kernel/setup.c                      |   7 +-
 arch/s390/kernel/uv.c                         | 271 ++++++++++
 arch/s390/kvm/Kconfig                         |  19 +
 arch/s390/kvm/Makefile                        |   2 +-
 arch/s390/kvm/diag.c                          |   7 +
 arch/s390/kvm/intercept.c                     | 107 +++-
 arch/s390/kvm/interrupt.c                     | 211 ++++++--
 arch/s390/kvm/kvm-s390.c                      | 486 ++++++++++++++++--
 arch/s390/kvm/kvm-s390.h                      |  56 ++
 arch/s390/kvm/priv.c                          |   9 +-
 arch/s390/kvm/pv.c                            | 293 +++++++++++
 arch/s390/mm/fault.c                          |  87 ++++
 arch/s390/mm/gmap.c                           |  63 ++-
 include/linux/gfp.h                           |   6 +
 include/uapi/linux/kvm.h                      |  47 +-
 mm/gup.c                                      |   2 +
 mm/page-writeback.c                           |   1 +
 33 files changed, 2217 insertions(+), 142 deletions(-)
 create mode 100644 Documentation/virt/kvm/s390-pv-boot.rst
 create mode 100644 Documentation/virt/kvm/s390-pv.rst
 create mode 100644 arch/s390/kernel/uv.c
 create mode 100644 arch/s390/kvm/pv.c

-- 
2.24.0

[RFCv2 01/37] DOCUMENTATION: protvirt: Protected virtual machine introduction

[Christian Borntraeger]

From: Janosch Frank <frankja@linux.ibm.com>

Add documentation about protected KVM guests.

Signed-off-by: Janosch Frank <frankja@linux.ibm.com>
---
 Documentation/virt/kvm/s390-pv.rst | 103 +++++++++++++++++++++++++++++
 MAINTAINERS                        |   1 +
 2 files changed, 104 insertions(+)
 create mode 100644 Documentation/virt/kvm/s390-pv.rst

diff --git a/Documentation/virt/kvm/s390-pv.rst b/Documentation/virt/kvm/s390-pv.rst
new file mode 100644
index 000000000000..5ef7e6cc2180
--- /dev/null
+++ b/Documentation/virt/kvm/s390-pv.rst
@@ -0,0 +1,103 @@
+.. SPDX-License-Identifier: GPL-2.0
+============================
+Ultravisor and Protected VMs
+============================
+
+Summary
+-------
+Protected virtual machines (PVM) are KVM VMs, where KVM can't access
+the VM's state like guest memory and guest registers anymore. Instead,
+the PVMs are mostly managed by a new entity called Ultravisor
+(UV). The UV provides an API that can be used by PVMs and KVM to
+request management actions.
+
+Each guest starts in the non-protected mode and then may make a
+request to transition into protected mode. On transition, KVM
+registers the guest and its VCPUs with the Ultravisor and prepares
+everything for running it.
+
+The Ultravisor will secure and decrypt the guest's boot memory
+(i.e. kernel/initrd). It will safeguard state changes like VCPU
+starts/stops and injected interrupts while the guest is running.
+
+As access to the guest's state, such as the SIE state description, is
+normally needed to be able to run a VM, some changes have been made in
+SIE behavior. A new format 4 state description has been introduced,
+where some fields have different meanings for a PVM. SIE exits are
+minimized as much as possible to improve speed and reduce exposed
+guest state.
+
+
+Interrupt injection
+-------------------
+Interrupt injection is safeguarded by the Ultravisor. As KVM doesn't
+have access to the VCPUs' lowcores, injection is handled via the
+format 4 state description.
+
+Machine check, external, IO and restart interruptions each can be
+injected on SIE entry via a bit in the interrupt injection control
+field (offset 0x54). If the guest cpu is not enabled for the interrupt
+at the time of injection, a validity interception is recognized. The
+format 4 state description contains fields in the interception data
+block where data associated with the interrupt can be transported.
+
+Program and Service Call exceptions have another layer of
+safeguarding; they can only be injected for instructions that have
+been intercepted into KVM. The exceptions need to be a valid outcome
+of an instruction emulation by KVM, e.g. we can never inject a
+addressing exception as they are reported by SIE since KVM has no
+access to the guest memory.
+
+
+Mask notification interceptions
+-------------------------------
+As a replacement for the lctl(g) and lpsw(e) instruction
+interceptions, two new interception codes have been introduced. One
+indicating that the contents of CRs 0, 6 or 14 have been changed. And
+one indicating PSW bit 13 changes.
+
+Instruction emulation
+---------------------
+With the format 4 state description for PVMs, the SIE instruction already
+interprets more instructions than it does with format 2. As it is not
+able to interpret every instruction, the SIE and the UV safeguard KVM's
+emulation inputs and outputs.
+
+Guest GRs and most of the instruction data, such as I/O data structures,
+are filtered. Instruction data is copied to and from the Secure
+Instruction Data Area. Guest GRs are put into / retrieved from the
+Interception-Data block.
+
+The Interception-Data block from the state description's offset 0x380
+contains GRs 0 - 16. Only GR values needed to emulate an instruction
+will be copied into this area.
+
+The Interception Parameters state description field still contains the
+the bytes of the instruction text, but with pre-set register values
+instead of the actual ones. I.e. each instruction always uses the same
+instruction text, in order not to leak guest instruction text.
+
+The Secure Instruction Data Area contains instruction storage
+data. Instruction data, i.e. data being referenced by an instruction
+like the SCCB for sclp, is moved over the SIDA When an instruction is
+intercepted, the SIE will only allow data and program interrupts for
+this instruction to be moved to the guest via the two data areas
+discussed before. Other data is either ignored or results in validity
+interceptions.
+
+
+Instruction emulation interceptions
+-----------------------------------
+There are two types of SIE secure instruction intercepts: the normal
+and the notification type. Normal secure instruction intercepts will
+make the guest pending for instruction completion of the intercepted
+instruction type, i.e. on SIE entry it is attempted to complete
+emulation of the instruction with the data provided by KVM. That might
+be a program exception or instruction completion.
+
+The notification type intercepts inform KVM about guest environment
+changes due to guest instruction interpretation. Such an interception
+is recognized for example for the store prefix instruction to provide
+the new lowcore location. On SIE reentry, any KVM data in the data
+areas is ignored, program exceptions are not injected and execution
+continues, as if no intercept had happened.
diff --git a/MAINTAINERS b/MAINTAINERS
index 56765f542244..90da412bebd9 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -9106,6 +9106,7 @@ L:	kvm@vger.kernel.org
 W:	http://www.ibm.com/developerworks/linux/linux390/
 T:	git git://git.kernel.org/pub/scm/linux/kernel/git/kvms390/linux.git
 S:	Supported
+F:	Documentation/virt/kvm/s390*
 F:	arch/s390/include/uapi/asm/kvm*
 F:	arch/s390/include/asm/gmap.h
 F:	arch/s390/include/asm/kvm*
-- 
2.24.0

[RFCv2 02/37] s390/protvirt: introduce host side setup

[Christian Borntraeger]

From: Vasily Gorbik <gor@linux.ibm.com>

Introduce KVM_S390_PROTECTED_VIRTUALIZATION_HOST kbuild option for
protected virtual machines hosting support code.

Add "prot_virt" command line option which controls if the kernel
protected VMs support is enabled at early boot time. This has to be
done early, because it needs large amounts of memory and will disable
some features like STP time sync for the lpar.

Extend ultravisor info definitions and expose it via uv_info struct
filled in during startup.

Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
---
 .../admin-guide/kernel-parameters.txt         |  5 ++
 arch/s390/boot/Makefile                       |  2 +-
 arch/s390/boot/uv.c                           | 20 +++++++-
 arch/s390/include/asm/uv.h                    | 46 ++++++++++++++++--
 arch/s390/kernel/Makefile                     |  1 +
 arch/s390/kernel/setup.c                      |  4 --
 arch/s390/kernel/uv.c                         | 48 +++++++++++++++++++
 arch/s390/kvm/Kconfig                         | 19 ++++++++
 8 files changed, 136 insertions(+), 9 deletions(-)
 create mode 100644 arch/s390/kernel/uv.c

diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
index ade4e6ec23e0..327af96f9528 100644
--- a/Documentation/admin-guide/kernel-parameters.txt
+++ b/Documentation/admin-guide/kernel-parameters.txt
@@ -3750,6 +3750,11 @@
 			before loading.
 			See Documentation/admin-guide/blockdev/ramdisk.rst.
 
+	prot_virt=	[S390] enable hosting protected virtual machines
+			isolated from the hypervisor (if hardware supports
+			that).
+			Format: <bool>
+
 	psi=		[KNL] Enable or disable pressure stall information
 			tracking.
 			Format: <bool>
diff --git a/arch/s390/boot/Makefile b/arch/s390/boot/Makefile
index e2c47d3a1c89..82247e71617a 100644
--- a/arch/s390/boot/Makefile
+++ b/arch/s390/boot/Makefile
@@ -37,7 +37,7 @@ CFLAGS_sclp_early_core.o += -I$(srctree)/drivers/s390/char
 obj-y	:= head.o als.o startup.o mem_detect.o ipl_parm.o ipl_report.o
 obj-y	+= string.o ebcdic.o sclp_early_core.o mem.o ipl_vmparm.o cmdline.o
 obj-y	+= version.o pgm_check_info.o ctype.o text_dma.o
-obj-$(CONFIG_PROTECTED_VIRTUALIZATION_GUEST)	+= uv.o
+obj-$(findstring y, $(CONFIG_PROTECTED_VIRTUALIZATION_GUEST) $(CONFIG_KVM_S390_PROTECTED_VIRTUALIZATION_HOST))	+= uv.o
 obj-$(CONFIG_RELOCATABLE)	+= machine_kexec_reloc.o
 obj-$(CONFIG_RANDOMIZE_BASE)	+= kaslr.o
 targets	:= bzImage startup.a section_cmp.boot.data section_cmp.boot.preserved.data $(obj-y)
diff --git a/arch/s390/boot/uv.c b/arch/s390/boot/uv.c
index ed007f4a6444..88cf8825d169 100644
--- a/arch/s390/boot/uv.c
+++ b/arch/s390/boot/uv.c
@@ -3,7 +3,12 @@
 #include <asm/facility.h>
 #include <asm/sections.h>
 
+#ifdef CONFIG_PROTECTED_VIRTUALIZATION_GUEST
 int __bootdata_preserved(prot_virt_guest);
+#endif
+#ifdef CONFIG_KVM_S390_PROTECTED_VIRTUALIZATION_HOST
+struct uv_info __bootdata_preserved(uv_info);
+#endif
 
 void uv_query_info(void)
 {
@@ -18,7 +23,20 @@ void uv_query_info(void)
 	if (uv_call(0, (uint64_t)&uvcb))
 		return;
 
-	if (test_bit_inv(BIT_UVC_CMD_SET_SHARED_ACCESS, (unsigned long *)uvcb.inst_calls_list) &&
+	if (IS_ENABLED(CONFIG_KVM_S390_PROTECTED_VIRTUALIZATION_HOST)) {
+		memcpy(uv_info.inst_calls_list, uvcb.inst_calls_list, sizeof(uv_info.inst_calls_list));
+		uv_info.uv_base_stor_len = uvcb.uv_base_stor_len;
+		uv_info.guest_base_stor_len = uvcb.conf_base_phys_stor_len;
+		uv_info.guest_virt_base_stor_len = uvcb.conf_base_virt_stor_len;
+		uv_info.guest_virt_var_stor_len = uvcb.conf_virt_var_stor_len;
+		uv_info.guest_cpu_stor_len = uvcb.cpu_stor_len;
+		uv_info.max_sec_stor_addr = ALIGN(uvcb.max_guest_stor_addr, PAGE_SIZE);
+		uv_info.max_num_sec_conf = uvcb.max_num_sec_conf;
+		uv_info.max_guest_cpus = uvcb.max_guest_cpus;
+	}
+
+	if (IS_ENABLED(CONFIG_PROTECTED_VIRTUALIZATION_GUEST) &&
+	    test_bit_inv(BIT_UVC_CMD_SET_SHARED_ACCESS, (unsigned long *)uvcb.inst_calls_list) &&
 	    test_bit_inv(BIT_UVC_CMD_REMOVE_SHARED_ACCESS, (unsigned long *)uvcb.inst_calls_list))
 		prot_virt_guest = 1;
 }
diff --git a/arch/s390/include/asm/uv.h b/arch/s390/include/asm/uv.h
index 4093a2856929..32eac3ab2d3b 100644
--- a/arch/s390/include/asm/uv.h
+++ b/arch/s390/include/asm/uv.h
@@ -44,7 +44,19 @@ struct uv_cb_qui {
 	struct uv_cb_header header;
 	u64 reserved08;
 	u64 inst_calls_list[4];
-	u64 reserved30[15];
+	u64 reserved30[2];
+	u64 uv_base_stor_len;
+	u64 reserved48;
+	u64 conf_base_phys_stor_len;
+	u64 conf_base_virt_stor_len;
+	u64 conf_virt_var_stor_len;
+	u64 cpu_stor_len;
+	u32 reserved68[3];
+	u32 max_num_sec_conf;
+	u64 max_guest_stor_addr;
+	u8  reserved80[150-128];
+	u16 max_guest_cpus;
+	u64 reserved98;
 } __packed __aligned(8);
 
 struct uv_cb_share {
@@ -69,9 +81,21 @@ static inline int uv_call(unsigned long r1, unsigned long r2)
 	return cc;
 }
 
-#ifdef CONFIG_PROTECTED_VIRTUALIZATION_GUEST
+struct uv_info {
+	unsigned long inst_calls_list[4];
+	unsigned long uv_base_stor_len;
+	unsigned long guest_base_stor_len;
+	unsigned long guest_virt_base_stor_len;
+	unsigned long guest_virt_var_stor_len;
+	unsigned long guest_cpu_stor_len;
+	unsigned long max_sec_stor_addr;
+	unsigned int max_num_sec_conf;
+	unsigned short max_guest_cpus;
+};
+extern struct uv_info uv_info;
 extern int prot_virt_guest;
 
+#ifdef CONFIG_PROTECTED_VIRTUALIZATION_GUEST
 static inline int is_prot_virt_guest(void)
 {
 	return prot_virt_guest;
@@ -121,11 +145,27 @@ static inline int uv_remove_shared(unsigned long addr)
 	return share(addr, UVC_CMD_REMOVE_SHARED_ACCESS);
 }
 
-void uv_query_info(void);
 #else
 #define is_prot_virt_guest() 0
 static inline int uv_set_shared(unsigned long addr) { return 0; }
 static inline int uv_remove_shared(unsigned long addr) { return 0; }
+#endif
+
+#ifdef CONFIG_KVM_S390_PROTECTED_VIRTUALIZATION_HOST
+extern int prot_virt_host;
+
+static inline int is_prot_virt_host(void)
+{
+	return prot_virt_host;
+}
+#else
+#define is_prot_virt_host() 0
+#endif
+
+#if defined(CONFIG_PROTECTED_VIRTUALIZATION_GUEST) ||                          \
+	defined(CONFIG_KVM_S390_PROTECTED_VIRTUALIZATION_HOST)
+void uv_query_info(void);
+#else
 static inline void uv_query_info(void) {}
 #endif
 
diff --git a/arch/s390/kernel/Makefile b/arch/s390/kernel/Makefile
index 2b1203cf7be6..1b958a453e37 100644
--- a/arch/s390/kernel/Makefile
+++ b/arch/s390/kernel/Makefile
@@ -78,6 +78,7 @@ obj-$(CONFIG_PERF_EVENTS)	+= perf_cpum_cf_events.o perf_regs.o
 obj-$(CONFIG_PERF_EVENTS)	+= perf_cpum_cf_diag.o
 
 obj-$(CONFIG_TRACEPOINTS)	+= trace.o
+obj-$(findstring y, $(CONFIG_PROTECTED_VIRTUALIZATION_GUEST) $(CONFIG_KVM_S390_PROTECTED_VIRTUALIZATION_HOST))	+= uv.o
 
 # vdso
 obj-y				+= vdso64/
diff --git a/arch/s390/kernel/setup.c b/arch/s390/kernel/setup.c
index d5fbd754f41a..f2ab2528859f 100644
--- a/arch/s390/kernel/setup.c
+++ b/arch/s390/kernel/setup.c
@@ -92,10 +92,6 @@ char elf_platform[ELF_PLATFORM_SIZE];
 
 unsigned long int_hwcap = 0;
 
-#ifdef CONFIG_PROTECTED_VIRTUALIZATION_GUEST
-int __bootdata_preserved(prot_virt_guest);
-#endif
-
 int __bootdata(noexec_disabled);
 int __bootdata(memory_end_set);
 unsigned long __bootdata(memory_end);
diff --git a/arch/s390/kernel/uv.c b/arch/s390/kernel/uv.c
new file mode 100644
index 000000000000..35ce89695509
--- /dev/null
+++ b/arch/s390/kernel/uv.c
@@ -0,0 +1,48 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Common Ultravisor functions and initialization
+ *
+ * Copyright IBM Corp. 2019
+ */
+#include <linux/kernel.h>
+#include <linux/types.h>
+#include <linux/sizes.h>
+#include <linux/bitmap.h>
+#include <linux/memblock.h>
+#include <asm/facility.h>
+#include <asm/sections.h>
+#include <asm/uv.h>
+
+#ifdef CONFIG_PROTECTED_VIRTUALIZATION_GUEST
+int __bootdata_preserved(prot_virt_guest);
+#endif
+
+#ifdef CONFIG_KVM_S390_PROTECTED_VIRTUALIZATION_HOST
+int prot_virt_host;
+EXPORT_SYMBOL(prot_virt_host);
+struct uv_info __bootdata_preserved(uv_info);
+EXPORT_SYMBOL(uv_info);
+
+static int __init prot_virt_setup(char *val)
+{
+	bool enabled;
+	int rc;
+
+	rc = kstrtobool(val, &enabled);
+	if (!rc && enabled)
+		prot_virt_host = 1;
+
+	if (is_prot_virt_guest() && prot_virt_host) {
+		prot_virt_host = 0;
+		pr_info("Running as protected virtualization guest.");
+	}
+
+	if (prot_virt_host && !test_facility(158)) {
+		prot_virt_host = 0;
+		pr_info("The ultravisor call facility is not available.");
+	}
+
+	return rc;
+}
+early_param("prot_virt", prot_virt_setup);
+#endif
diff --git a/arch/s390/kvm/Kconfig b/arch/s390/kvm/Kconfig
index d3db3d7ed077..9ce9c7295e16 100644
--- a/arch/s390/kvm/Kconfig
+++ b/arch/s390/kvm/Kconfig
@@ -55,6 +55,25 @@ config KVM_S390_UCONTROL
 
 	  If unsure, say N.
 
+config KVM_S390_PROTECTED_VIRTUALIZATION_HOST
+	bool "Protected guests execution support"
+	depends on KVM
+	help
+	Support hosting protected virtual machines in KVM. The state
+	of these machines like memory content or register content is
+	protected from the host or host administrators.
+
+	Enabling this option will enable extra code that allows a
+	Linux hypervisor to talk to a new firmware instance called
+	ultravisor that will take care of protecting the guest while
+	also enabling KVM to run this guest.
+
+	This feature must be enable via the kernel command line option
+	prot_virt.
+
+	If unsure, say Y.
+
+
 # OK, it's a little counter-intuitive to do this, but it puts it neatly under
 # the virtualization menu.
 source "drivers/vhost/Kconfig"
-- 
2.24.0

[RFCv2 03/37] s390/protvirt: add ultravisor initialization

[Christian Borntraeger]

From: Vasily Gorbik <gor@linux.ibm.com>

Before being able to host protected virtual machines, donate some of
the memory to the ultravisor. Besides that the ultravisor might impose
addressing limitations for memory used to back protected VM storage. Treat
that limit as protected virtualization host's virtual memory limit.

Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
Reviewed-by: Christian Borntraeger <borntraeger@de.ibm.com>
---
 arch/s390/include/asm/uv.h | 16 ++++++++++++
 arch/s390/kernel/setup.c   |  3 +++
 arch/s390/kernel/uv.c      | 53 ++++++++++++++++++++++++++++++++++++++
 3 files changed, 72 insertions(+)

diff --git a/arch/s390/include/asm/uv.h b/arch/s390/include/asm/uv.h
index 32eac3ab2d3b..cdf2fd71d7ab 100644
--- a/arch/s390/include/asm/uv.h
+++ b/arch/s390/include/asm/uv.h
@@ -23,12 +23,14 @@
 #define UVC_RC_NO_RESUME	0x0007
 
 #define UVC_CMD_QUI			0x0001
+#define UVC_CMD_INIT_UV			0x000f
 #define UVC_CMD_SET_SHARED_ACCESS	0x1000
 #define UVC_CMD_REMOVE_SHARED_ACCESS	0x1001
 
 /* Bits in installed uv calls */
 enum uv_cmds_inst {
 	BIT_UVC_CMD_QUI = 0,
+	BIT_UVC_CMD_INIT_UV = 1,
 	BIT_UVC_CMD_SET_SHARED_ACCESS = 8,
 	BIT_UVC_CMD_REMOVE_SHARED_ACCESS = 9,
 };
@@ -59,6 +61,15 @@ struct uv_cb_qui {
 	u64 reserved98;
 } __packed __aligned(8);
 
+struct uv_cb_init {
+	struct uv_cb_header header;
+	u64 reserved08[2];
+	u64 stor_origin;
+	u64 stor_len;
+	u64 reserved28[4];
+
+} __packed __aligned(8);
+
 struct uv_cb_share {
 	struct uv_cb_header header;
 	u64 reserved08[3];
@@ -158,8 +169,13 @@ static inline int is_prot_virt_host(void)
 {
 	return prot_virt_host;
 }
+
+void setup_uv(void);
+void adjust_to_uv_max(unsigned long *vmax);
 #else
 #define is_prot_virt_host() 0
+static inline void setup_uv(void) {}
+static inline void adjust_to_uv_max(unsigned long *vmax) {}
 #endif
 
 #if defined(CONFIG_PROTECTED_VIRTUALIZATION_GUEST) ||                          \
diff --git a/arch/s390/kernel/setup.c b/arch/s390/kernel/setup.c
index f2ab2528859f..5f178d557cc8 100644
--- a/arch/s390/kernel/setup.c
+++ b/arch/s390/kernel/setup.c
@@ -560,6 +560,8 @@ static void __init setup_memory_end(void)
 			vmax = _REGION1_SIZE; /* 4-level kernel page table */
 	}
 
+	adjust_to_uv_max(&vmax);
+
 	/* module area is at the end of the kernel address space. */
 	MODULES_END = vmax;
 	MODULES_VADDR = MODULES_END - MODULES_LEN;
@@ -1140,6 +1142,7 @@ void __init setup_arch(char **cmdline_p)
 	 */
 	memblock_trim_memory(1UL << (MAX_ORDER - 1 + PAGE_SHIFT));
 
+	setup_uv();
 	setup_memory_end();
 	setup_memory();
 	dma_contiguous_reserve(memory_end);
diff --git a/arch/s390/kernel/uv.c b/arch/s390/kernel/uv.c
index 35ce89695509..f7778493e829 100644
--- a/arch/s390/kernel/uv.c
+++ b/arch/s390/kernel/uv.c
@@ -45,4 +45,57 @@ static int __init prot_virt_setup(char *val)
 	return rc;
 }
 early_param("prot_virt", prot_virt_setup);
+
+static int __init uv_init(unsigned long stor_base, unsigned long stor_len)
+{
+	struct uv_cb_init uvcb = {
+		.header.cmd = UVC_CMD_INIT_UV,
+		.header.len = sizeof(uvcb),
+		.stor_origin = stor_base,
+		.stor_len = stor_len,
+	};
+	int cc;
+
+	cc = uv_call(0, (uint64_t)&uvcb);
+	if (cc || uvcb.header.rc != UVC_RC_EXECUTED) {
+		pr_err("Ultravisor init failed with cc: %d rc: 0x%hx\n", cc,
+		       uvcb.header.rc);
+		return -1;
+	}
+	return 0;
+}
+
+void __init setup_uv(void)
+{
+	unsigned long uv_stor_base;
+
+	if (!prot_virt_host)
+		return;
+
+	uv_stor_base = (unsigned long)memblock_alloc_try_nid(
+		uv_info.uv_base_stor_len, SZ_1M, SZ_2G,
+		MEMBLOCK_ALLOC_ACCESSIBLE, NUMA_NO_NODE);
+	if (!uv_stor_base) {
+		pr_info("Failed to reserve %lu bytes for ultravisor base storage\n",
+			uv_info.uv_base_stor_len);
+		goto fail;
+	}
+
+	if (uv_init(uv_stor_base, uv_info.uv_base_stor_len)) {
+		memblock_free(uv_stor_base, uv_info.uv_base_stor_len);
+		goto fail;
+	}
+
+	pr_info("Reserving %luMB as ultravisor base storage\n",
+		uv_info.uv_base_stor_len >> 20);
+	return;
+fail:
+	prot_virt_host = 0;
+}
+
+void adjust_to_uv_max(unsigned long *vmax)
+{
+	if (prot_virt_host && *vmax > uv_info.max_sec_stor_addr)
+		*vmax = uv_info.max_sec_stor_addr;
+}
 #endif
-- 
2.24.0

[RFCv2 04/37] mm:gup/writeback: add callbacks for inaccessible pages

[Christian Borntraeger]

From: Claudio Imbrenda <imbrenda@linux.ibm.com>

With the introduction of protected KVM guests on s390 there is now a
concept of inaccessible pages. These pages need to be made accessible
before the host can access them.

While cpu accesses will trigger a fault that can be resolved, I/O
accesses will just fail.  We need to add a callback into architecture
code for places that will do I/Oi, namely when writeback is started or
when a page reference is taken.

Signed-off-by: Claudio Imbrenda <imbrenda@linux.ibm.com>
---
 include/linux/gfp.h | 6 ++++++
 mm/gup.c            | 2 ++
 mm/page-writeback.c | 1 +
 3 files changed, 9 insertions(+)

diff --git a/include/linux/gfp.h b/include/linux/gfp.h
index e5b817cb86e7..be2754841369 100644
--- a/include/linux/gfp.h
+++ b/include/linux/gfp.h
@@ -485,6 +485,12 @@ static inline void arch_free_page(struct page *page, int order) { }
 #ifndef HAVE_ARCH_ALLOC_PAGE
 static inline void arch_alloc_page(struct page *page, int order) { }
 #endif
+#ifndef HAVE_ARCH_MAKE_PAGE_ACCESSIBLE
+static inline int arch_make_page_accessible(struct page *page)
+{
+	return 0;
+}
+#endif
 
 struct page *
 __alloc_pages_nodemask(gfp_t gfp_mask, unsigned int order, int preferred_nid,
diff --git a/mm/gup.c b/mm/gup.c
index 7646bf993b25..a01262cd2821 100644
--- a/mm/gup.c
+++ b/mm/gup.c
@@ -257,6 +257,7 @@ static struct page *follow_page_pte(struct vm_area_struct *vma,
 			page = ERR_PTR(-ENOMEM);
 			goto out;
 		}
+		arch_make_page_accessible(page);
 	}
 	if (flags & FOLL_TOUCH) {
 		if ((flags & FOLL_WRITE) &&
@@ -1870,6 +1871,7 @@ static int gup_pte_range(pmd_t pmd, unsigned long addr, unsigned long end,
 
 		VM_BUG_ON_PAGE(compound_head(page) != head, page);
 
+		arch_make_page_accessible(page);
 		SetPageReferenced(page);
 		pages[*nr] = page;
 		(*nr)++;
diff --git a/mm/page-writeback.c b/mm/page-writeback.c
index 2caf780a42e7..0f0bd14571b1 100644
--- a/mm/page-writeback.c
+++ b/mm/page-writeback.c
@@ -2806,6 +2806,7 @@ int __test_set_page_writeback(struct page *page, bool keep_write)
 		inc_lruvec_page_state(page, NR_WRITEBACK);
 		inc_zone_page_state(page, NR_ZONE_WRITE_PENDING);
 	}
+	arch_make_page_accessible(page);
 	unlock_page_memcg(page);
 	return ret;
 
-- 
2.24.0

[RFCv2 05/37] s390/mm: provide memory management functions for protected KVM guests

[Christian Borntraeger]

From: Claudio Imbrenda <imbrenda@linux.ibm.com>

This provides the basic ultravisor calls and page table handling to cope
with secure guests.

Co-authored-by: Ulrich Weigand <Ulrich.Weigand@de.ibm.com>
Signed-off-by: Claudio Imbrenda <imbrenda@linux.ibm.com>
---
 arch/s390/include/asm/gmap.h        |   2 +
 arch/s390/include/asm/mmu.h         |   2 +
 arch/s390/include/asm/mmu_context.h |   1 +
 arch/s390/include/asm/page.h        |   5 +
 arch/s390/include/asm/pgtable.h     |  34 +++++-
 arch/s390/include/asm/uv.h          |  59 ++++++++++
 arch/s390/kernel/uv.c               | 170 ++++++++++++++++++++++++++++
 7 files changed, 268 insertions(+), 5 deletions(-)

diff --git a/arch/s390/include/asm/gmap.h b/arch/s390/include/asm/gmap.h
index 37f96b6f0e61..f2ab8b6d4b57 100644
--- a/arch/s390/include/asm/gmap.h
+++ b/arch/s390/include/asm/gmap.h
@@ -9,6 +9,7 @@
 #ifndef _ASM_S390_GMAP_H
 #define _ASM_S390_GMAP_H
 
+#include <linux/radix-tree.h>
 #include <linux/refcount.h>
 
 /* Generic bits for GMAP notification on DAT table entry changes. */
@@ -61,6 +62,7 @@ struct gmap {
 	spinlock_t shadow_lock;
 	struct gmap *parent;
 	unsigned long orig_asce;
+	unsigned long se_handle;
 	int edat_level;
 	bool removed;
 	bool initialized;
diff --git a/arch/s390/include/asm/mmu.h b/arch/s390/include/asm/mmu.h
index bcfb6371086f..984026cb3608 100644
--- a/arch/s390/include/asm/mmu.h
+++ b/arch/s390/include/asm/mmu.h
@@ -16,6 +16,8 @@ typedef struct {
 	unsigned long asce;
 	unsigned long asce_limit;
 	unsigned long vdso_base;
+	/* The mmu context belongs to a secure guest. */
+	atomic_t is_se;
 	/*
 	 * The following bitfields need a down_write on the mm
 	 * semaphore when they are written to. As they are only
diff --git a/arch/s390/include/asm/mmu_context.h b/arch/s390/include/asm/mmu_context.h
index 8d04e6f3f796..0e5e67ecdaf8 100644
--- a/arch/s390/include/asm/mmu_context.h
+++ b/arch/s390/include/asm/mmu_context.h
@@ -23,6 +23,7 @@ static inline int init_new_context(struct task_struct *tsk,
 	INIT_LIST_HEAD(&mm->context.gmap_list);
 	cpumask_clear(&mm->context.cpu_attach_mask);
 	atomic_set(&mm->context.flush_count, 0);
+	atomic_set(&mm->context.is_se, 0);
 	mm->context.gmap_asce = 0;
 	mm->context.flush_mm = 0;
 	mm->context.compat_mm = test_thread_flag(TIF_31BIT);
diff --git a/arch/s390/include/asm/page.h b/arch/s390/include/asm/page.h
index a4d38092530a..eb209416c45b 100644
--- a/arch/s390/include/asm/page.h
+++ b/arch/s390/include/asm/page.h
@@ -151,6 +151,11 @@ static inline int devmem_is_allowed(unsigned long pfn)
 #define HAVE_ARCH_FREE_PAGE
 #define HAVE_ARCH_ALLOC_PAGE
 
+#ifdef CONFIG_KVM_S390_PROTECTED_VIRTUALIZATION_HOST
+int arch_make_page_accessible(struct page *page);
+#define HAVE_ARCH_MAKE_PAGE_ACCESSIBLE
+#endif
+
 #endif /* !__ASSEMBLY__ */
 
 #define __PAGE_OFFSET		0x0UL
diff --git a/arch/s390/include/asm/pgtable.h b/arch/s390/include/asm/pgtable.h
index 7b03037a8475..65b6bb47af0a 100644
--- a/arch/s390/include/asm/pgtable.h
+++ b/arch/s390/include/asm/pgtable.h
@@ -19,6 +19,7 @@
 #include <linux/atomic.h>
 #include <asm/bug.h>
 #include <asm/page.h>
+#include <asm/uv.h>
 
 extern pgd_t swapper_pg_dir[];
 extern void paging_init(void);
@@ -520,6 +521,15 @@ static inline int mm_has_pgste(struct mm_struct *mm)
 	return 0;
 }
 
+static inline int mm_is_se(struct mm_struct *mm)
+{
+#ifdef CONFIG_KVM_S390_PROTECTED_VIRTUALIZATION_HOST
+	if (unlikely(atomic_read(&mm->context.is_se)))
+		return 1;
+#endif
+	return 0;
+}
+
 static inline int mm_alloc_pgste(struct mm_struct *mm)
 {
 #ifdef CONFIG_PGSTE
@@ -1059,7 +1069,12 @@ static inline int ptep_clear_flush_young(struct vm_area_struct *vma,
 static inline pte_t ptep_get_and_clear(struct mm_struct *mm,
 				       unsigned long addr, pte_t *ptep)
 {
-	return ptep_xchg_lazy(mm, addr, ptep, __pte(_PAGE_INVALID));
+	pte_t res;
+
+	res = ptep_xchg_lazy(mm, addr, ptep, __pte(_PAGE_INVALID));
+	if (mm_is_se(mm) && pte_present(res))
+		uv_convert_from_secure(pte_val(res) & PAGE_MASK);
+	return res;
 }
 
 #define __HAVE_ARCH_PTEP_MODIFY_PROT_TRANSACTION
@@ -1071,7 +1086,12 @@ void ptep_modify_prot_commit(struct vm_area_struct *, unsigned long,
 static inline pte_t ptep_clear_flush(struct vm_area_struct *vma,
 				     unsigned long addr, pte_t *ptep)
 {
-	return ptep_xchg_direct(vma->vm_mm, addr, ptep, __pte(_PAGE_INVALID));
+	pte_t res;
+
+	res = ptep_xchg_direct(vma->vm_mm, addr, ptep, __pte(_PAGE_INVALID));
+	if (mm_is_se(vma->vm_mm) && pte_present(res))
+		uv_convert_from_secure(pte_val(res) & PAGE_MASK);
+	return res;
 }
 
 /*
@@ -1086,12 +1106,16 @@ static inline pte_t ptep_get_and_clear_full(struct mm_struct *mm,
 					    unsigned long addr,
 					    pte_t *ptep, int full)
 {
+	pte_t res;
 	if (full) {
-		pte_t pte = *ptep;
+		res = *ptep;
 		*ptep = __pte(_PAGE_INVALID);
-		return pte;
+	} else {
+		res = ptep_xchg_lazy(mm, addr, ptep, __pte(_PAGE_INVALID));
 	}
-	return ptep_xchg_lazy(mm, addr, ptep, __pte(_PAGE_INVALID));
+	if (mm_is_se(mm) && pte_present(res))
+		uv_convert_from_secure(pte_val(res) & PAGE_MASK);
+	return res;
 }
 
 #define __HAVE_ARCH_PTEP_SET_WRPROTECT
diff --git a/arch/s390/include/asm/uv.h b/arch/s390/include/asm/uv.h
index cdf2fd71d7ab..4eaea95f5c64 100644
--- a/arch/s390/include/asm/uv.h
+++ b/arch/s390/include/asm/uv.h
@@ -15,6 +15,7 @@
 #include <linux/errno.h>
 #include <linux/bug.h>
 #include <asm/page.h>
+#include <asm/gmap.h>
 
 #define UVC_RC_EXECUTED		0x0001
 #define UVC_RC_INV_CMD		0x0002
@@ -24,6 +25,10 @@
 
 #define UVC_CMD_QUI			0x0001
 #define UVC_CMD_INIT_UV			0x000f
+#define UVC_CMD_CONV_TO_SEC_STOR	0x0200
+#define UVC_CMD_CONV_FROM_SEC_STOR	0x0201
+#define UVC_CMD_PIN_PAGE_SHARED		0x0341
+#define UVC_CMD_UNPIN_PAGE_SHARED	0x0342
 #define UVC_CMD_SET_SHARED_ACCESS	0x1000
 #define UVC_CMD_REMOVE_SHARED_ACCESS	0x1001
 
@@ -31,8 +36,12 @@
 enum uv_cmds_inst {
 	BIT_UVC_CMD_QUI = 0,
 	BIT_UVC_CMD_INIT_UV = 1,
+	BIT_UVC_CMD_CONV_TO_SEC_STOR = 6,
+	BIT_UVC_CMD_CONV_FROM_SEC_STOR = 7,
 	BIT_UVC_CMD_SET_SHARED_ACCESS = 8,
 	BIT_UVC_CMD_REMOVE_SHARED_ACCESS = 9,
+	BIT_UVC_CMD_PIN_PAGE_SHARED = 21,
+	BIT_UVC_CMD_UNPIN_PAGE_SHARED = 22,
 };
 
 struct uv_cb_header {
@@ -70,6 +79,19 @@ struct uv_cb_init {
 
 } __packed __aligned(8);
 
+struct uv_cb_cts {
+	struct uv_cb_header header;
+	u64 reserved08[2];
+	u64 guest_handle;
+	u64 gaddr;
+} __packed __aligned(8);
+
+struct uv_cb_cfs {
+	struct uv_cb_header header;
+	u64 reserved08[2];
+	u64 paddr;
+} __packed __aligned(8);
+
 struct uv_cb_share {
 	struct uv_cb_header header;
 	u64 reserved08[3];
@@ -170,12 +192,49 @@ static inline int is_prot_virt_host(void)
 	return prot_virt_host;
 }
 
+int uv_make_secure(struct gmap *gmap, unsigned long gaddr, void *uvcb, int pins);
+int uv_convert_from_secure(unsigned long paddr);
+
+static inline int uv_convert_to_secure_pinned(struct gmap *gmap,
+					      unsigned long gaddr,
+					      int pins)
+{
+	struct uv_cb_cts uvcb = {
+		.header.cmd = UVC_CMD_CONV_TO_SEC_STOR,
+		.header.len = sizeof(uvcb),
+		.guest_handle = gmap->se_handle,
+		.gaddr = gaddr,
+	};
+
+	return uv_make_secure(gmap, gaddr, &uvcb, pins);
+}
+
+static inline int uv_convert_to_secure(struct gmap *gmap, unsigned long gaddr)
+{
+	return uv_convert_to_secure_pinned(gmap, gaddr, 0);
+}
+
 void setup_uv(void);
 void adjust_to_uv_max(unsigned long *vmax);
 #else
 #define is_prot_virt_host() 0
 static inline void setup_uv(void) {}
 static inline void adjust_to_uv_max(unsigned long *vmax) {}
+
+static inline int uv_make_secure(struct gmap *gmap, unsigned long gaddr, void *uvcb, int pins)
+{
+	return 0;
+}
+
+static inline int uv_convert_from_secure(unsigned long paddr)
+{
+	return 0;
+}
+
+static inline int uv_convert_to_secure(struct gmap *gmap, unsigned long gaddr)
+{
+	return 0;
+}
 #endif
 
 #if defined(CONFIG_PROTECTED_VIRTUALIZATION_GUEST) ||                          \
diff --git a/arch/s390/kernel/uv.c b/arch/s390/kernel/uv.c
index f7778493e829..136c60a8e3ca 100644
--- a/arch/s390/kernel/uv.c
+++ b/arch/s390/kernel/uv.c
@@ -9,6 +9,8 @@
 #include <linux/sizes.h>
 #include <linux/bitmap.h>
 #include <linux/memblock.h>
+#include <linux/pagemap.h>
+#include <linux/swap.h>
 #include <asm/facility.h>
 #include <asm/sections.h>
 #include <asm/uv.h>
@@ -98,4 +100,172 @@ void adjust_to_uv_max(unsigned long *vmax)
 	if (prot_virt_host && *vmax > uv_info.max_sec_stor_addr)
 		*vmax = uv_info.max_sec_stor_addr;
 }
+
+static int __uv_pin_shared(unsigned long paddr)
+{
+	struct uv_cb_cfs uvcb = {
+		.header.cmd	= UVC_CMD_PIN_PAGE_SHARED,
+		.header.len	= sizeof(uvcb),
+		.paddr		= paddr,
+	};
+
+	if (uv_call(0, (u64)&uvcb))
+		return -EINVAL;
+	return 0;
+}
+
+/*
+ * Requests the Ultravisor to encrypt a guest page and make it
+ * accessible to the host for paging (export).
+ *
+ * @paddr: Absolute host address of page to be exported
+ */
+int uv_convert_from_secure(unsigned long paddr)
+{
+	struct uv_cb_cfs uvcb = {
+		.header.cmd = UVC_CMD_CONV_FROM_SEC_STOR,
+		.header.len = sizeof(uvcb),
+		.paddr = paddr
+	};
+
+	uv_call(0, (u64)&uvcb);
+
+	if (uvcb.header.rc == 1 || uvcb.header.rc == 0x107)
+		return 0;
+	return -EINVAL;
+}
+
+static int expected_page_refs(struct page *page)
+{
+	int res;
+
+	res = page_mapcount(page);
+	if (PageSwapCache(page))
+		res++;
+	else if (page_mapping(page)) {
+		res++;
+		if (page_has_private(page))
+			res++;
+	}
+	return res;
+}
+
+struct conv_params {
+	struct uv_cb_header *uvcb;
+	struct page *page;
+	int extra_pins;
+};
+
+static int make_secure_pte(pte_t *ptep, unsigned long addr, void *data)
+{
+	struct conv_params *params = data;
+	pte_t entry = READ_ONCE(*ptep);
+	struct page *page;
+	int expected, rc = 0;
+
+	if (!pte_present(entry))
+		return -ENXIO;
+	if (pte_val(entry) & (_PAGE_INVALID | _PAGE_PROTECT))
+		return -ENXIO;
+
+	page = pte_page(entry);
+	if (page != params->page)
+		return -ENXIO;
+
+	if (PageWriteback(page))
+		return -EAGAIN;
+	expected = expected_page_refs(page) + params->extra_pins;
+	if (!page_ref_freeze(page, expected))
+		return -EBUSY;
+	set_bit(PG_arch_1, &page->flags);
+	rc = uv_call(0, (u64)params->uvcb);
+	page_ref_unfreeze(page, expected);
+	if (rc)
+		rc = (params->uvcb->rc == 0x10a) ? -ENXIO : -EINVAL;
+	return rc;
+}
+
+/*
+ * Requests the Ultravisor to make a page accessible to a guest.
+ * If it's brought in the first time, it will be cleared. If
+ * it has been exported before, it will be decrypted and integrity
+ * checked.
+ *
+ * @gmap: Guest mapping
+ * @gaddr: Guest 2 absolute address to be imported
+ */
+int uv_make_secure(struct gmap *gmap, unsigned long gaddr, void *uvcb, int pins)
+{
+	struct conv_params params = { .uvcb = uvcb, .extra_pins = pins };
+	struct vm_area_struct *vma;
+	unsigned long uaddr;
+	int rc, local_drain = 0;
+
+again:
+	rc = -EFAULT;
+	down_read(&gmap->mm->mmap_sem);
+
+	uaddr = __gmap_translate(gmap, gaddr);
+	if (IS_ERR_VALUE(uaddr))
+		goto out;
+	vma = find_vma(gmap->mm, uaddr);
+	if (!vma)
+		goto out;
+	if (is_vm_hugetlb_page(vma))
+		goto out;
+
+	rc = -ENXIO;
+	params.page = follow_page(vma, uaddr, FOLL_WRITE | FOLL_NOWAIT);
+	if (IS_ERR_OR_NULL(params.page))
+		goto out;
+
+	lock_page(params.page);
+	rc = apply_to_page_range(gmap->mm, uaddr, PAGE_SIZE, make_secure_pte, &params);
+	unlock_page(params.page);
+out:
+	up_read(&gmap->mm->mmap_sem);
+
+	if (rc == -EBUSY) {
+		if (local_drain) {
+			lru_add_drain_all();
+			return -EAGAIN;
+		}
+		lru_add_drain();
+		local_drain = 1;
+		goto again;
+	} else if (rc == -ENXIO) {
+		if (gmap_fault(gmap, gaddr, FAULT_FLAG_WRITE))
+			return -EFAULT;
+		return -EAGAIN;
+	}
+	return rc;
+}
+EXPORT_SYMBOL_GPL(uv_make_secure);
+
+/**
+ * To be called with the page locked or with an extra reference!
+ */
+int arch_make_page_accessible(struct page *page)
+{
+	int rc = 0;
+
+	if (!test_bit(PG_arch_1, &page->flags))
+		return 0;
+
+	rc = __uv_pin_shared(page_to_phys(page));
+	if (!rc) {
+		clear_bit(PG_arch_1, &page->flags);
+		return 0;
+	}
+
+	rc = uv_convert_from_secure(page_to_phys(page));
+	if (!rc) {
+		clear_bit(PG_arch_1, &page->flags);
+		return 0;
+	}
+
+	return rc;
+}
+EXPORT_SYMBOL_GPL(arch_make_page_accessible);
+
 #endif
-- 
2.24.0

[RFCv2 06/37] s390: add (non)secure page access exceptions handlers

[Christian Borntraeger]

From: Vasily Gorbik <gor@linux.ibm.com>

Add exceptions handlers performing transparent transition of non-secure
pages to secure (import) upon guest access and secure pages to
non-secure (export) upon hypervisor access.

Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
Signed-off-by: Janosch Frank <frankja@linux.ibm.com>
[adding checks for failures]
Signed-off-by: Claudio Imbrenda <imbrenda@linux.ibm.com>
[further changes like adding a check for gmap fault]
---
 arch/s390/kernel/pgm_check.S |  4 +-
 arch/s390/mm/fault.c         | 87 ++++++++++++++++++++++++++++++++++++
 2 files changed, 89 insertions(+), 2 deletions(-)

diff --git a/arch/s390/kernel/pgm_check.S b/arch/s390/kernel/pgm_check.S
index 59dee9d3bebf..27ac4f324c70 100644
--- a/arch/s390/kernel/pgm_check.S
+++ b/arch/s390/kernel/pgm_check.S
@@ -78,8 +78,8 @@ PGM_CHECK(do_dat_exception)		/* 39 */
 PGM_CHECK(do_dat_exception)		/* 3a */
 PGM_CHECK(do_dat_exception)		/* 3b */
 PGM_CHECK_DEFAULT			/* 3c */
-PGM_CHECK_DEFAULT			/* 3d */
-PGM_CHECK_DEFAULT			/* 3e */
+PGM_CHECK(do_secure_storage_access)	/* 3d */
+PGM_CHECK(do_non_secure_storage_access)	/* 3e */
 PGM_CHECK_DEFAULT			/* 3f */
 PGM_CHECK_DEFAULT			/* 40 */
 PGM_CHECK_DEFAULT			/* 41 */
diff --git a/arch/s390/mm/fault.c b/arch/s390/mm/fault.c
index 7b0bb475c166..bd75b0765cf1 100644
--- a/arch/s390/mm/fault.c
+++ b/arch/s390/mm/fault.c
@@ -38,6 +38,7 @@
 #include <asm/irq.h>
 #include <asm/mmu_context.h>
 #include <asm/facility.h>
+#include <asm/uv.h>
 #include "../kernel/entry.h"
 
 #define __FAIL_ADDR_MASK -4096L
@@ -816,3 +817,89 @@ static int __init pfault_irq_init(void)
 early_initcall(pfault_irq_init);
 
 #endif /* CONFIG_PFAULT */
+
+#ifdef CONFIG_KVM_S390_PROTECTED_VIRTUALIZATION_HOST
+
+void do_secure_storage_access(struct pt_regs *regs)
+{
+	unsigned long addr = regs->int_parm_long & __FAIL_ADDR_MASK;
+	struct vm_area_struct *vma;
+	struct mm_struct *mm;
+	struct page *page;
+	int rc;
+
+	switch (get_fault_type(regs)) {
+	case USER_FAULT:
+		mm = current->mm;
+		down_read(&mm->mmap_sem);
+		vma = find_vma(mm, addr);
+		if (!vma) {
+			up_read(&mm->mmap_sem);
+			do_fault_error(regs, VM_READ | VM_WRITE, VM_FAULT_BADMAP);
+			break;
+		}
+		page = follow_page(vma, addr, FOLL_WRITE | FOLL_GET);
+		if (IS_ERR_OR_NULL(page)) {
+			up_read(&mm->mmap_sem);
+			break;
+		}
+		if (arch_make_page_accessible(page))
+			send_sig(SIGSEGV, current, 0);
+		put_page(page);
+		up_read(&mm->mmap_sem);
+		break;
+	case KERNEL_FAULT:
+		page = phys_to_page(addr);
+		if (unlikely(!try_get_page(page)))
+			break;
+		rc = arch_make_page_accessible(page);
+		put_page(page);
+		if (rc)
+			BUG();
+		break;
+	case VDSO_FAULT:
+		/* fallthrough */
+	case GMAP_FAULT:
+		/* fallthrough */
+	default:
+		do_fault_error(regs, VM_READ | VM_WRITE, VM_FAULT_BADMAP);
+		WARN_ON_ONCE(1);
+	}
+}
+NOKPROBE_SYMBOL(do_secure_storage_access);
+
+void do_non_secure_storage_access(struct pt_regs *regs)
+{
+	unsigned long gaddr = regs->int_parm_long & __FAIL_ADDR_MASK;
+	struct gmap *gmap = (struct gmap *)S390_lowcore.gmap;
+	struct uv_cb_cts uvcb = {
+		.header.cmd = UVC_CMD_CONV_TO_SEC_STOR,
+		.header.len = sizeof(uvcb),
+		.guest_handle = gmap->se_handle,
+		.gaddr = gaddr,
+	};
+	int rc;
+
+	if (get_fault_type(regs) != GMAP_FAULT) {
+		do_fault_error(regs, VM_READ | VM_WRITE, VM_FAULT_BADMAP);
+		WARN_ON_ONCE(1);
+		return;
+	}
+
+	rc = uv_make_secure(gmap, gaddr, &uvcb, 0);
+	if (rc == -EINVAL && uvcb.header.rc != 0x104)
+		send_sig(SIGSEGV, current, 0);
+}
+NOKPROBE_SYMBOL(do_non_secure_storage_access);
+
+#else
+void do_secure_storage_access(struct pt_regs *regs)
+{
+	default_trap_handler(regs);
+}
+
+void do_non_secure_storage_access(struct pt_regs *regs)
+{
+	default_trap_handler(regs);
+}
+#endif
-- 
2.24.0

[RFCv2 07/37] KVM: s390: add new variants of UV CALL

[Christian Borntraeger]

From: Janosch Frank <frankja@linux.ibm.com>

This add 2 new variants of the UV CALL.

The first variant handles UV CALLs that might have longer busy
conditions or just need longer when doing partial completion. We should
schedule when necessary.

The second variant handles UV CALLs that only need the handle but have
no payload (e.g. destroying a VM). We can provide a simple wrapper for
those.

Signed-off-by: Janosch Frank <frankja@linux.ibm.com>
---
 arch/s390/include/asm/uv.h | 58 ++++++++++++++++++++++++++++++++++++++
 1 file changed, 58 insertions(+)

diff --git a/arch/s390/include/asm/uv.h b/arch/s390/include/asm/uv.h
index 4eaea95f5c64..3448f12ef57a 100644
--- a/arch/s390/include/asm/uv.h
+++ b/arch/s390/include/asm/uv.h
@@ -14,6 +14,7 @@
 #include <linux/types.h>
 #include <linux/errno.h>
 #include <linux/bug.h>
+#include <linux/sched.h>
 #include <asm/page.h>
 #include <asm/gmap.h>
 
@@ -92,6 +93,18 @@ struct uv_cb_cfs {
 	u64 paddr;
 } __packed __aligned(8);
 
+/*
+ * A common UV call struct for the following calls:
+ * Destroy cpu/config
+ * Verify
+ */
+struct uv_cb_nodata {
+	struct uv_cb_header header;
+	u64 reserved08[2];
+	u64 handle;
+	u64 reserved20[4];
+} __packed __aligned(8);
+
 struct uv_cb_share {
 	struct uv_cb_header header;
 	u64 reserved08[3];
@@ -99,6 +112,31 @@ struct uv_cb_share {
 	u64 reserved28;
 } __packed __aligned(8);
 
+/*
+ * Low level uv_call that takes r1 and r2 as parameter and avoids
+ * stalls for long running busy conditions by doing schedule
+ */
+static inline int uv_call_sched(unsigned long r1, unsigned long r2)
+{
+	int cc = 3;
+
+	while (cc > 1) {
+		asm volatile(
+			"0:	.insn rrf,0xB9A40000,%[r1],%[r2],0,0\n"
+			"		ipm	%[cc]\n"
+			"		srl	%[cc],28\n"
+			: [cc] "=d" (cc)
+			: [r1] "a" (r1), [r2] "a" (r2)
+			: "memory", "cc");
+		if (need_resched())
+			schedule();
+	}
+	return cc;
+}
+
+/*
+ * Low level uv_call that takes r1 and r2 as parameter
+ */
 static inline int uv_call(unsigned long r1, unsigned long r2)
 {
 	int cc;
@@ -114,6 +152,26 @@ static inline int uv_call(unsigned long r1, unsigned long r2)
 	return cc;
 }
 
+/*
+ * special variant of uv_call that only transport the cpu or guest
+ * handle and the command, like destroy or verify.
+ */
+static inline int uv_cmd_nodata(u64 handle, u16 cmd, u32 *ret)
+{
+	int rc;
+	struct uv_cb_nodata uvcb = {
+		.header.cmd = cmd,
+		.header.len = sizeof(uvcb),
+		.handle = handle,
+	};
+
+	WARN(!handle, "No handle provided to Ultravisor call cmd %x\n", cmd);
+	rc = uv_call_sched(0, (u64)&uvcb);
+	if (ret)
+		*ret = *(u32 *)&uvcb.header.rc;
+	return rc ? -EINVAL : 0;
+}
+
 struct uv_info {
 	unsigned long inst_calls_list[4];
 	unsigned long uv_base_stor_len;
-- 
2.24.0

[RFCv2 08/37] KVM: s390: protvirt: Add initial lifecycle handling

[Christian Borntraeger]

From: Janosch Frank <frankja@linux.ibm.com>

This contains 3 main changes:
1. changes in SIE control block handling for secure guests
2. helper functions for create/destroy/unpack secure guests
3. KVM_S390_PV_COMMAND ioctl to allow userspace dealing with secure
machines

Signed-off-by: Janosch Frank <frankja@linux.ibm.com>
---
 arch/s390/include/asm/kvm_host.h |  24 ++-
 arch/s390/include/asm/uv.h       |  60 ++++++++
 arch/s390/kvm/Makefile           |   2 +-
 arch/s390/kvm/kvm-s390.c         | 198 ++++++++++++++++++++++++-
 arch/s390/kvm/kvm-s390.h         |  45 ++++++
 arch/s390/kvm/pv.c               | 246 +++++++++++++++++++++++++++++++
 include/uapi/linux/kvm.h         |  33 +++++
 7 files changed, 604 insertions(+), 4 deletions(-)
 create mode 100644 arch/s390/kvm/pv.c

diff --git a/arch/s390/include/asm/kvm_host.h b/arch/s390/include/asm/kvm_host.h
index 73044545ecac..841690d05080 100644
--- a/arch/s390/include/asm/kvm_host.h
+++ b/arch/s390/include/asm/kvm_host.h
@@ -160,7 +160,13 @@ struct kvm_s390_sie_block {
 	__u8	reserved08[4];		/* 0x0008 */
 #define PROG_IN_SIE (1<<0)
 	__u32	prog0c;			/* 0x000c */
-	__u8	reserved10[16];		/* 0x0010 */
+	union {
+		__u8	reserved10[16];		/* 0x0010 */
+		struct {
+			__u64	pv_handle_cpu;
+			__u64	pv_handle_config;
+		};
+	};
 #define PROG_BLOCK_SIE	(1<<0)
 #define PROG_REQUEST	(1<<1)
 	atomic_t prog20;		/* 0x0020 */
@@ -233,7 +239,7 @@ struct kvm_s390_sie_block {
 #define ECB3_RI  0x01
 	__u8    ecb3;			/* 0x0063 */
 	__u32	scaol;			/* 0x0064 */
-	__u8	reserved68;		/* 0x0068 */
+	__u8	sdf;			/* 0x0068 */
 	__u8    epdx;			/* 0x0069 */
 	__u8    reserved6a[2];		/* 0x006a */
 	__u32	todpr;			/* 0x006c */
@@ -645,6 +651,11 @@ struct kvm_guestdbg_info_arch {
 	unsigned long last_bp;
 };
 
+struct kvm_s390_pv_vcpu {
+	u64 handle;
+	unsigned long stor_base;
+};
+
 struct kvm_vcpu_arch {
 	struct kvm_s390_sie_block *sie_block;
 	/* if vsie is active, currently executed shadow sie control block */
@@ -673,6 +684,7 @@ struct kvm_vcpu_arch {
 	__u64 cputm_start;
 	bool gs_enabled;
 	bool skey_enabled;
+	struct kvm_s390_pv_vcpu pv;
 };
 
 struct kvm_vm_stat {
@@ -846,6 +858,13 @@ struct kvm_s390_gisa_interrupt {
 	DECLARE_BITMAP(kicked_mask, KVM_MAX_VCPUS);
 };
 
+struct kvm_s390_pv {
+	u64 handle;
+	u64 guest_len;
+	unsigned long stor_base;
+	void *stor_var;
+};
+
 struct kvm_arch{
 	void *sca;
 	int use_esca;
@@ -881,6 +900,7 @@ struct kvm_arch{
 	DECLARE_BITMAP(cpu_feat, KVM_S390_VM_CPU_FEAT_NR_BITS);
 	DECLARE_BITMAP(idle_mask, KVM_MAX_VCPUS);
 	struct kvm_s390_gisa_interrupt gisa_int;
+	struct kvm_s390_pv pv;
 };
 
 #define KVM_HVA_ERR_BAD		(-1UL)
diff --git a/arch/s390/include/asm/uv.h b/arch/s390/include/asm/uv.h
index 3448f12ef57a..315ec15efe44 100644
--- a/arch/s390/include/asm/uv.h
+++ b/arch/s390/include/asm/uv.h
@@ -23,11 +23,19 @@
 #define UVC_RC_INV_STATE	0x0003
 #define UVC_RC_INV_LEN		0x0005
 #define UVC_RC_NO_RESUME	0x0007
+#define UVC_RC_NEED_DESTROY	0x8000
 
 #define UVC_CMD_QUI			0x0001
 #define UVC_CMD_INIT_UV			0x000f
+#define UVC_CMD_CREATE_SEC_CONF		0x0100
+#define UVC_CMD_DESTROY_SEC_CONF	0x0101
+#define UVC_CMD_CREATE_SEC_CPU		0x0120
+#define UVC_CMD_DESTROY_SEC_CPU		0x0121
 #define UVC_CMD_CONV_TO_SEC_STOR	0x0200
 #define UVC_CMD_CONV_FROM_SEC_STOR	0x0201
+#define UVC_CMD_SET_SEC_CONF_PARAMS	0x0300
+#define UVC_CMD_UNPACK_IMG		0x0301
+#define UVC_CMD_VERIFY_IMG		0x0302
 #define UVC_CMD_PIN_PAGE_SHARED		0x0341
 #define UVC_CMD_UNPIN_PAGE_SHARED	0x0342
 #define UVC_CMD_SET_SHARED_ACCESS	0x1000
@@ -37,10 +45,17 @@
 enum uv_cmds_inst {
 	BIT_UVC_CMD_QUI = 0,
 	BIT_UVC_CMD_INIT_UV = 1,
+	BIT_UVC_CMD_CREATE_SEC_CONF = 2,
+	BIT_UVC_CMD_DESTROY_SEC_CONF = 3,
+	BIT_UVC_CMD_CREATE_SEC_CPU = 4,
+	BIT_UVC_CMD_DESTROY_SEC_CPU = 5,
 	BIT_UVC_CMD_CONV_TO_SEC_STOR = 6,
 	BIT_UVC_CMD_CONV_FROM_SEC_STOR = 7,
 	BIT_UVC_CMD_SET_SHARED_ACCESS = 8,
 	BIT_UVC_CMD_REMOVE_SHARED_ACCESS = 9,
+	BIT_UVC_CMD_SET_SEC_PARMS = 11,
+	BIT_UVC_CMD_UNPACK_IMG = 13,
+	BIT_UVC_CMD_VERIFY_IMG = 14,
 	BIT_UVC_CMD_PIN_PAGE_SHARED = 21,
 	BIT_UVC_CMD_UNPIN_PAGE_SHARED = 22,
 };
@@ -80,6 +95,32 @@ struct uv_cb_init {
 
 } __packed __aligned(8);
 
+struct uv_cb_cgc {
+	struct uv_cb_header header;
+	u64 reserved08[2];
+	u64 guest_handle;
+	u64 conf_base_stor_origin;
+	u64 conf_var_stor_origin;
+	u64 reserved30;
+	u64 guest_stor_origin;
+	u64 guest_stor_len;
+	u64 guest_sca;
+	u64 guest_asce;
+	u64 reserved60[5];
+} __packed __aligned(8);
+
+struct uv_cb_csc {
+	struct uv_cb_header header;
+	u64 reserved08[2];
+	u64 cpu_handle;
+	u64 guest_handle;
+	u64 stor_origin;
+	u8  reserved30[6];
+	u16 num;
+	u64 state_origin;
+	u64 reserved[4];
+} __packed __aligned(8);
+
 struct uv_cb_cts {
 	struct uv_cb_header header;
 	u64 reserved08[2];
@@ -93,6 +134,25 @@ struct uv_cb_cfs {
 	u64 paddr;
 } __packed __aligned(8);
 
+struct uv_cb_ssc {
+	struct uv_cb_header header;
+	u64 reserved08[2];
+	u64 guest_handle;
+	u64 sec_header_origin;
+	u32 sec_header_len;
+	u32 reserved32;
+	u64 reserved38[4];
+} __packed __aligned(8);
+
+struct uv_cb_unp {
+	struct uv_cb_header header;
+	u64 reserved08[2];
+	u64 guest_handle;
+	u64 gaddr;
+	u64 tweak[2];
+	u64 reserved28[3];
+} __packed __aligned(8);
+
 /*
  * A common UV call struct for the following calls:
  * Destroy cpu/config
diff --git a/arch/s390/kvm/Makefile b/arch/s390/kvm/Makefile
index 05ee90a5ea08..eaaedf9e61a7 100644
--- a/arch/s390/kvm/Makefile
+++ b/arch/s390/kvm/Makefile
@@ -9,6 +9,6 @@ common-objs = $(KVM)/kvm_main.o $(KVM)/eventfd.o  $(KVM)/async_pf.o $(KVM)/irqch
 ccflags-y := -Ivirt/kvm -Iarch/s390/kvm
 
 kvm-objs := $(common-objs) kvm-s390.o intercept.o interrupt.o priv.o sigp.o
-kvm-objs += diag.o gaccess.o guestdbg.o vsie.o
+kvm-objs += diag.o gaccess.o guestdbg.o vsie.o $(if $(CONFIG_KVM_S390_PROTECTED_VIRTUALIZATION_HOST),pv.o)
 
 obj-$(CONFIG_KVM) += kvm.o
diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
index e39f6ef97b09..35f46404830f 100644
--- a/arch/s390/kvm/kvm-s390.c
+++ b/arch/s390/kvm/kvm-s390.c
@@ -44,6 +44,7 @@
 #include <asm/cpacf.h>
 #include <asm/timex.h>
 #include <asm/ap.h>
+#include <asm/uv.h>
 #include "kvm-s390.h"
 #include "gaccess.h"
 
@@ -235,6 +236,7 @@ int kvm_arch_check_processor_compat(void)
 
 static void kvm_gmap_notifier(struct gmap *gmap, unsigned long start,
 			      unsigned long end);
+static int sca_switch_to_extended(struct kvm *kvm);
 
 static void kvm_clock_sync_scb(struct kvm_s390_sie_block *scb, u64 delta)
 {
@@ -564,6 +566,11 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
 	case KVM_CAP_S390_BPB:
 		r = test_facility(82);
 		break;
+#ifdef CONFIG_KVM_S390_PROTECTED_VIRTUALIZATION_HOST
+	case KVM_CAP_S390_PROTECTED:
+		r = is_prot_virt_host();
+		break;
+#endif
 	default:
 		r = 0;
 	}
@@ -2158,6 +2165,112 @@ static int kvm_s390_set_cmma_bits(struct kvm *kvm,
 	return r;
 }
 
+#ifdef CONFIG_KVM_S390_PROTECTED_VIRTUALIZATION_HOST
+static int kvm_s390_handle_pv(struct kvm *kvm, struct kvm_pv_cmd *cmd)
+{
+	int r = 0;
+	void __user *argp = (void __user *)cmd->data;
+
+	switch (cmd->cmd) {
+	case KVM_PV_VM_CREATE: {
+		r = -EINVAL;
+		if (kvm_s390_pv_is_protected(kvm))
+			break;
+
+		r = kvm_s390_pv_alloc_vm(kvm);
+		if (r)
+			break;
+
+		mutex_lock(&kvm->lock);
+		kvm_s390_vcpu_block_all(kvm);
+		/* FMT 4 SIE needs esca */
+		r = sca_switch_to_extended(kvm);
+		if (!r)
+			r = kvm_s390_pv_create_vm(kvm);
+		kvm_s390_vcpu_unblock_all(kvm);
+		mutex_unlock(&kvm->lock);
+		break;
+	}
+	case KVM_PV_VM_DESTROY: {
+		r = -EINVAL;
+		if (!kvm_s390_pv_is_protected(kvm))
+			break;
+
+		/* All VCPUs have to be destroyed before this call. */
+		mutex_lock(&kvm->lock);
+		kvm_s390_vcpu_block_all(kvm);
+		r = kvm_s390_pv_destroy_vm(kvm);
+		if (!r)
+			kvm_s390_pv_dealloc_vm(kvm);
+		kvm_s390_vcpu_unblock_all(kvm);
+		mutex_unlock(&kvm->lock);
+		break;
+	}
+	case KVM_PV_VM_SET_SEC_PARMS: {
+		struct kvm_s390_pv_sec_parm parms = {};
+		void *hdr;
+
+		r = -EINVAL;
+		if (!kvm_s390_pv_is_protected(kvm))
+			break;
+
+		r = -EFAULT;
+		if (copy_from_user(&parms, argp, sizeof(parms)))
+			break;
+
+		/* Currently restricted to 8KB */
+		r = -EINVAL;
+		if (parms.length > PAGE_SIZE * 2)
+			break;
+
+		r = -ENOMEM;
+		hdr = vmalloc(parms.length);
+		if (!hdr)
+			break;
+
+		r = -EFAULT;
+		if (!copy_from_user(hdr, (void __user *)parms.origin,
+				   parms.length))
+			r = kvm_s390_pv_set_sec_parms(kvm, hdr, parms.length);
+
+		vfree(hdr);
+		break;
+	}
+	case KVM_PV_VM_UNPACK: {
+		struct kvm_s390_pv_unp unp = {};
+
+		r = -EINVAL;
+		if (!kvm_s390_pv_is_protected(kvm))
+			break;
+
+		r = -EFAULT;
+		if (copy_from_user(&unp, argp, sizeof(unp)))
+			break;
+
+		r = kvm_s390_pv_unpack(kvm, unp.addr, unp.size, unp.tweak);
+		break;
+	}
+	case KVM_PV_VM_VERIFY: {
+		u32 ret;
+
+		r = -EINVAL;
+		if (!kvm_s390_pv_is_protected(kvm))
+			break;
+
+		r = uv_cmd_nodata(kvm_s390_pv_handle(kvm),
+				  UVC_CMD_VERIFY_IMG,
+				  &ret);
+		VM_EVENT(kvm, 3, "PROTVIRT VERIFY: rc %x rrc %x",
+			 ret >> 16, ret & 0x0000ffff);
+		break;
+	}
+	default:
+		return -ENOTTY;
+	}
+	return r;
+}
+#endif
+
 long kvm_arch_vm_ioctl(struct file *filp,
 		       unsigned int ioctl, unsigned long arg)
 {
@@ -2255,6 +2368,22 @@ long kvm_arch_vm_ioctl(struct file *filp,
 		mutex_unlock(&kvm->slots_lock);
 		break;
 	}
+#ifdef CONFIG_KVM_S390_PROTECTED_VIRTUALIZATION_HOST
+	case KVM_S390_PV_COMMAND: {
+		struct kvm_pv_cmd args;
+
+		r = -EINVAL;
+		if (!is_prot_virt_host())
+			break;
+
+		r = -EFAULT;
+		if (copy_from_user(&args, argp, sizeof(args)))
+			break;
+
+		r = kvm_s390_handle_pv(kvm, &args);
+		break;
+	}
+#endif
 	default:
 		r = -ENOTTY;
 	}
@@ -2530,6 +2659,9 @@ void kvm_arch_vcpu_destroy(struct kvm_vcpu *vcpu)
 
 	if (vcpu->kvm->arch.use_cmma)
 		kvm_s390_vcpu_unsetup_cmma(vcpu);
+	if (IS_ENABLED(CONFIG_KVM_S390_PROTECTED_VIRTUALIZATION_HOST) &&
+	    kvm_s390_pv_handle_cpu(vcpu))
+		kvm_s390_pv_destroy_cpu(vcpu);
 	free_page((unsigned long)(vcpu->arch.sie_block));
 
 	kvm_vcpu_uninit(vcpu);
@@ -2556,8 +2688,13 @@ void kvm_arch_destroy_vm(struct kvm *kvm)
 {
 	kvm_free_vcpus(kvm);
 	sca_dispose(kvm);
-	debug_unregister(kvm->arch.dbf);
 	kvm_s390_gisa_destroy(kvm);
+	if (IS_ENABLED(CONFIG_KVM_S390_PROTECTED_VIRTUALIZATION_HOST) &&
+	    kvm_s390_pv_is_protected(kvm)) {
+		kvm_s390_pv_destroy_vm(kvm);
+		kvm_s390_pv_dealloc_vm(kvm);
+	}
+	debug_unregister(kvm->arch.dbf);
 	free_page((unsigned long)kvm->arch.sie_page2);
 	if (!kvm_is_ucontrol(kvm))
 		gmap_remove(kvm->arch.gmap);
@@ -2653,6 +2790,9 @@ static int sca_switch_to_extended(struct kvm *kvm)
 	unsigned int vcpu_idx;
 	u32 scaol, scaoh;
 
+	if (kvm->arch.use_esca)
+		return 0;
+
 	new_sca = alloc_pages_exact(sizeof(*new_sca), GFP_KERNEL|__GFP_ZERO);
 	if (!new_sca)
 		return -ENOMEM;
@@ -3045,6 +3185,15 @@ struct kvm_vcpu *kvm_arch_vcpu_create(struct kvm *kvm,
 	rc = kvm_vcpu_init(vcpu, kvm, id);
 	if (rc)
 		goto out_free_sie_block;
+
+	if (kvm_s390_pv_is_protected(kvm)) {
+		rc = kvm_s390_pv_create_cpu(vcpu);
+		if (rc) {
+			kvm_vcpu_uninit(vcpu);
+			goto out_free_sie_block;
+		}
+	}
+
 	VM_EVENT(kvm, 3, "create cpu %d at 0x%pK, sie block at 0x%pK", id, vcpu,
 		 vcpu->arch.sie_block);
 	trace_kvm_s390_create_vcpu(id, vcpu, vcpu->arch.sie_block);
@@ -4353,6 +4502,37 @@ long kvm_arch_vcpu_async_ioctl(struct file *filp,
 	return -ENOIOCTLCMD;
 }
 
+#ifdef CONFIG_KVM_S390_PROTECTED_VIRTUALIZATION_HOST
+static int kvm_s390_handle_pv_vcpu(struct kvm_vcpu *vcpu,
+				   struct kvm_pv_cmd *cmd)
+{
+	int r = 0;
+
+	if (!kvm_s390_pv_is_protected(vcpu->kvm))
+		return -EINVAL;
+
+	switch (cmd->cmd) {
+	case KVM_PV_VCPU_CREATE: {
+		if (kvm_s390_pv_handle_cpu(vcpu))
+			return -EINVAL;
+
+		r = kvm_s390_pv_create_cpu(vcpu);
+		break;
+	}
+	case KVM_PV_VCPU_DESTROY: {
+		if (!kvm_s390_pv_handle_cpu(vcpu))
+			return -EINVAL;
+
+		r = kvm_s390_pv_destroy_cpu(vcpu);
+		break;
+	}
+	default:
+		r = -ENOTTY;
+	}
+	return r;
+}
+#endif
+
 long kvm_arch_vcpu_ioctl(struct file *filp,
 			 unsigned int ioctl, unsigned long arg)
 {
@@ -4494,6 +4674,22 @@ long kvm_arch_vcpu_ioctl(struct file *filp,
 					   irq_state.len);
 		break;
 	}
+#ifdef CONFIG_KVM_S390_PROTECTED_VIRTUALIZATION_HOST
+	case KVM_S390_PV_COMMAND_VCPU: {
+		struct kvm_pv_cmd args;
+
+		r = -EINVAL;
+		if (!is_prot_virt_host())
+			break;
+
+		r = -EFAULT;
+		if (copy_from_user(&args, argp, sizeof(args)))
+			break;
+
+		r = kvm_s390_handle_pv_vcpu(vcpu, &args);
+		break;
+	}
+#endif
 	default:
 		r = -ENOTTY;
 	}
diff --git a/arch/s390/kvm/kvm-s390.h b/arch/s390/kvm/kvm-s390.h
index 6d9448dbd052..d77da15bebe5 100644
--- a/arch/s390/kvm/kvm-s390.h
+++ b/arch/s390/kvm/kvm-s390.h
@@ -196,6 +196,51 @@ static inline int kvm_s390_user_cpu_state_ctrl(struct kvm *kvm)
 	return kvm->arch.user_cpu_state_ctrl != 0;
 }
 
+#ifdef CONFIG_KVM_S390_PROTECTED_VIRTUALIZATION_HOST
+/* implemented in pv.c */
+void kvm_s390_pv_dealloc_vm(struct kvm *kvm);
+int kvm_s390_pv_alloc_vm(struct kvm *kvm);
+int kvm_s390_pv_create_vm(struct kvm *kvm);
+int kvm_s390_pv_create_cpu(struct kvm_vcpu *vcpu);
+int kvm_s390_pv_destroy_vm(struct kvm *kvm);
+int kvm_s390_pv_destroy_cpu(struct kvm_vcpu *vcpu);
+int kvm_s390_pv_set_sec_parms(struct kvm *kvm, void *hdr, u64 length);
+int kvm_s390_pv_unpack(struct kvm *kvm, unsigned long addr, unsigned long size,
+		       unsigned long tweak);
+int kvm_s390_pv_verify(struct kvm *kvm);
+
+static inline bool kvm_s390_pv_is_protected(struct kvm *kvm)
+{
+	return !!kvm->arch.pv.handle;
+}
+
+static inline u64 kvm_s390_pv_handle(struct kvm *kvm)
+{
+	return kvm->arch.pv.handle;
+}
+
+static inline u64 kvm_s390_pv_handle_cpu(struct kvm_vcpu *vcpu)
+{
+	return vcpu->arch.pv.handle;
+}
+#else
+static inline void kvm_s390_pv_dealloc_vm(struct kvm *kvm) {}
+static inline int kvm_s390_pv_alloc_vm(struct kvm *kvm) { return 0; }
+static inline int kvm_s390_pv_create_vm(struct kvm *kvm) { return 0; }
+static inline int kvm_s390_pv_create_cpu(struct kvm_vcpu *vcpu) { return 0; }
+static inline int kvm_s390_pv_destroy_vm(struct kvm *kvm) { return 0; }
+static inline int kvm_s390_pv_destroy_cpu(struct kvm_vcpu *vcpu) { return 0; }
+static inline int kvm_s390_pv_set_sec_parms(struct kvm *kvm,
+					    u64 origin, u64 length) { return 0; }
+static inline int kvm_s390_pv_unpack(struct kvm *kvm, unsigned long addr,
+				     unsigned long size,  unsigned long tweak)
+{ return 0; }
+static inline int kvm_s390_pv_verify(struct kvm *kvm) { return 0; }
+static inline bool kvm_s390_pv_is_protected(struct kvm *kvm) { return 0; }
+static inline u64 kvm_s390_pv_handle(struct kvm *kvm) { return 0; }
+static inline u64 kvm_s390_pv_handle_cpu(struct kvm_vcpu *vcpu) { return 0; }
+#endif
+
 /* implemented in interrupt.c */
 int kvm_s390_handle_wait(struct kvm_vcpu *vcpu);
 void kvm_s390_vcpu_wakeup(struct kvm_vcpu *vcpu);
diff --git a/arch/s390/kvm/pv.c b/arch/s390/kvm/pv.c
new file mode 100644
index 000000000000..a867b9e9c069
--- /dev/null
+++ b/arch/s390/kvm/pv.c
@@ -0,0 +1,246 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Hosting Secure Execution virtual machines
+ *
+ * Copyright IBM Corp. 2019
+ *    Author(s): Janosch Frank <frankja@linux.ibm.com>
+ */
+#include <linux/kvm.h>
+#include <linux/kvm_host.h>
+#include <linux/pagemap.h>
+#include <asm/pgalloc.h>
+#include <asm/gmap.h>
+#include <asm/uv.h>
+#include <asm/gmap.h>
+#include <asm/mman.h>
+#include "kvm-s390.h"
+
+void kvm_s390_pv_dealloc_vm(struct kvm *kvm)
+{
+	vfree(kvm->arch.pv.stor_var);
+	free_pages(kvm->arch.pv.stor_base,
+		   get_order(uv_info.guest_base_stor_len));
+	memset(&kvm->arch.pv, 0, sizeof(kvm->arch.pv));
+}
+
+int kvm_s390_pv_alloc_vm(struct kvm *kvm)
+{
+	unsigned long base = uv_info.guest_base_stor_len;
+	unsigned long virt = uv_info.guest_virt_var_stor_len;
+	unsigned long npages = 0, vlen = 0;
+	struct kvm_memory_slot *memslot;
+
+	kvm->arch.pv.stor_var = NULL;
+	kvm->arch.pv.stor_base = __get_free_pages(GFP_KERNEL, get_order(base));
+	if (!kvm->arch.pv.stor_base)
+		return -ENOMEM;
+
+	/*
+	 * Calculate current guest storage for allocation of the
+	 * variable storage, which is based on the length in MB.
+	 *
+	 * Slots are sorted by GFN
+	 */
+	mutex_lock(&kvm->slots_lock);
+	memslot = kvm_memslots(kvm)->memslots;
+	npages = memslot->base_gfn + memslot->npages;
+	mutex_unlock(&kvm->slots_lock);
+
+	kvm->arch.pv.guest_len = npages * PAGE_SIZE;
+
+	/* Allocate variable storage */
+	vlen = ALIGN(virt * ((npages * PAGE_SIZE) / HPAGE_SIZE), PAGE_SIZE);
+	vlen += uv_info.guest_virt_base_stor_len;
+	kvm->arch.pv.stor_var = vzalloc(vlen);
+	if (!kvm->arch.pv.stor_var)
+		goto out_err;
+	return 0;
+
+out_err:
+	kvm_s390_pv_dealloc_vm(kvm);
+	return -ENOMEM;
+}
+
+int kvm_s390_pv_destroy_vm(struct kvm *kvm)
+{
+	int rc;
+	u32 ret;
+
+	rc = uv_cmd_nodata(kvm_s390_pv_handle(kvm),
+			   UVC_CMD_DESTROY_SEC_CONF, &ret);
+	WRITE_ONCE(kvm->arch.gmap->se_handle, 0);
+	atomic_set(&kvm->mm->context.is_se, 0);
+	VM_EVENT(kvm, 3, "PROTVIRT DESTROY VM: rc %x rrc %x",
+		 ret >> 16, ret & 0x0000ffff);
+	return rc;
+}
+
+int kvm_s390_pv_destroy_cpu(struct kvm_vcpu *vcpu)
+{
+	int rc = 0;
+	u32 ret;
+
+	if (kvm_s390_pv_handle_cpu(vcpu)) {
+		rc = uv_cmd_nodata(kvm_s390_pv_handle_cpu(vcpu),
+				   UVC_CMD_DESTROY_SEC_CPU,
+				   &ret);
+
+		VCPU_EVENT(vcpu, 3, "PROTVIRT DESTROY VCPU: cpu %d rc %x rrc %x",
+			   vcpu->vcpu_id, ret >> 16, ret & 0x0000ffff);
+	}
+
+	free_pages(vcpu->arch.pv.stor_base,
+		   get_order(uv_info.guest_cpu_stor_len));
+	/* Clear cpu and vm handle */
+	memset(&vcpu->arch.sie_block->reserved10, 0,
+	       sizeof(vcpu->arch.sie_block->reserved10));
+	memset(&vcpu->arch.pv, 0, sizeof(vcpu->arch.pv));
+	vcpu->arch.sie_block->sdf = 0;
+	return rc;
+}
+
+int kvm_s390_pv_create_cpu(struct kvm_vcpu *vcpu)
+{
+	int rc;
+	struct uv_cb_csc uvcb = {
+		.header.cmd = UVC_CMD_CREATE_SEC_CPU,
+		.header.len = sizeof(uvcb),
+	};
+
+	/* EEXIST and ENOENT? */
+	if (kvm_s390_pv_handle_cpu(vcpu))
+		return -EINVAL;
+
+	vcpu->arch.pv.stor_base = __get_free_pages(GFP_KERNEL,
+						   get_order(uv_info.guest_cpu_stor_len));
+	if (!vcpu->arch.pv.stor_base)
+		return -ENOMEM;
+
+	/* Input */
+	uvcb.guest_handle = kvm_s390_pv_handle(vcpu->kvm);
+	uvcb.num = vcpu->arch.sie_block->icpua;
+	uvcb.state_origin = (u64)vcpu->arch.sie_block;
+	uvcb.stor_origin = (u64)vcpu->arch.pv.stor_base;
+
+	rc = uv_call(0, (u64)&uvcb);
+	VCPU_EVENT(vcpu, 3, "PROTVIRT CREATE VCPU: cpu %d handle %llx rc %x rrc %x",
+		   vcpu->vcpu_id, uvcb.cpu_handle, uvcb.header.rc,
+		   uvcb.header.rrc);
+
+	/* Output */
+	vcpu->arch.pv.handle = uvcb.cpu_handle;
+	vcpu->arch.sie_block->pv_handle_cpu = uvcb.cpu_handle;
+	vcpu->arch.sie_block->pv_handle_config = kvm_s390_pv_handle(vcpu->kvm);
+	vcpu->arch.sie_block->sdf = 2;
+	if (!rc)
+		return 0;
+
+	kvm_s390_pv_destroy_cpu(vcpu);
+	return -EINVAL;
+}
+
+int kvm_s390_pv_create_vm(struct kvm *kvm)
+{
+	int rc;
+
+	struct uv_cb_cgc uvcb = {
+		.header.cmd = UVC_CMD_CREATE_SEC_CONF,
+		.header.len = sizeof(uvcb)
+	};
+
+	if (kvm_s390_pv_handle(kvm))
+		return -EINVAL;
+
+	/* Inputs */
+	uvcb.guest_stor_origin = 0; /* MSO is 0 for KVM */
+	uvcb.guest_stor_len = kvm->arch.pv.guest_len;
+	uvcb.guest_asce = kvm->arch.gmap->asce;
+	uvcb.guest_sca = (unsigned long)kvm->arch.sca;
+	uvcb.conf_base_stor_origin = (u64)kvm->arch.pv.stor_base;
+	uvcb.conf_var_stor_origin = (u64)kvm->arch.pv.stor_var;
+
+	rc = uv_call(0, (u64)&uvcb);
+	VM_EVENT(kvm, 3, "PROTVIRT CREATE VM: handle %llx len %llx rc %x rrc %x",
+		 uvcb.guest_handle, uvcb.guest_stor_len, uvcb.header.rc,
+		 uvcb.header.rrc);
+
+	/* Outputs */
+	kvm->arch.pv.handle = uvcb.guest_handle;
+
+	if (rc && (uvcb.header.rc & 0x8000)) {
+		kvm_s390_pv_destroy_vm(kvm);
+		kvm_s390_pv_dealloc_vm(kvm);
+		return -EINVAL;
+	}
+	kvm->arch.gmap->se_handle = uvcb.guest_handle;
+	atomic_set(&kvm->mm->context.is_se, 1);
+	return rc;
+}
+
+int kvm_s390_pv_set_sec_parms(struct kvm *kvm,
+			      void *hdr, u64 length)
+{
+	int rc;
+	struct uv_cb_ssc uvcb = {
+		.header.cmd = UVC_CMD_SET_SEC_CONF_PARAMS,
+		.header.len = sizeof(uvcb),
+		.sec_header_origin = (u64)hdr,
+		.sec_header_len = length,
+		.guest_handle = kvm_s390_pv_handle(kvm),
+	};
+
+	if (!kvm_s390_pv_handle(kvm))
+		return -EINVAL;
+
+	rc = uv_call(0, (u64)&uvcb);
+	VM_EVENT(kvm, 3, "PROTVIRT VM SET PARMS: rc %x rrc %x",
+		 uvcb.header.rc, uvcb.header.rrc);
+	if (rc)
+		return -EINVAL;
+	return 0;
+}
+
+static int unpack_one(struct kvm *kvm, unsigned long addr, u64 tweak[2])
+{
+	struct uv_cb_unp uvcb = {
+		.header.cmd = UVC_CMD_UNPACK_IMG,
+		.header.len = sizeof(uvcb),
+		.guest_handle = kvm_s390_pv_handle(kvm),
+		.gaddr = addr,
+		.tweak[0] = tweak[0],
+		.tweak[1] = tweak[1],
+	};
+	int rc;
+
+	rc = uv_make_secure(kvm->arch.gmap, addr, &uvcb, 0);
+
+	if (rc)
+		VM_EVENT(kvm, 3, "PROTVIRT VM UNPACK: failed addr %llx rc %x rrc %x",
+			 uvcb.gaddr, uvcb.header.rc, uvcb.header.rrc);
+	return rc;
+}
+
+int kvm_s390_pv_unpack(struct kvm *kvm, unsigned long addr, unsigned long size,
+		       unsigned long tweak)
+{
+	int rc = 0;
+	u64 tw[2] = {tweak, 0};
+
+	if (addr & ~PAGE_MASK || !size || size & ~PAGE_MASK)
+		return -EINVAL;
+
+	VM_EVENT(kvm, 3, "PROTVIRT VM UNPACK: start addr %lx size %lx",
+		 addr, size);
+
+	while (tw[1] < size) {
+		rc = unpack_one(kvm, addr, tw);
+		if (rc == -EAGAIN)
+			continue;
+		if (rc)
+			break;
+		addr += PAGE_SIZE;
+		tw[1] += PAGE_SIZE;
+	}
+	VM_EVENT(kvm, 3, "PROTVIRT VM UNPACK: finished rc %x", rc);
+	return rc;
+}
diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
index 4b95f9a31a2f..eab741bc12c3 100644
--- a/include/uapi/linux/kvm.h
+++ b/include/uapi/linux/kvm.h
@@ -1010,6 +1010,7 @@ struct kvm_ppc_resize_hpt {
 #define KVM_CAP_ARM_NISV_TO_USER 177
 #define KVM_CAP_ARM_INJECT_EXT_DABT 178
 #define KVM_CAP_S390_VCPU_RESETS 179
+#define KVM_CAP_S390_PROTECTED 181
 
 #ifdef KVM_CAP_IRQ_ROUTING
 
@@ -1478,6 +1479,38 @@ struct kvm_enc_region {
 #define KVM_S390_NORMAL_RESET	_IO(KVMIO,   0xc3)
 #define KVM_S390_CLEAR_RESET	_IO(KVMIO,   0xc4)
 
+struct kvm_s390_pv_sec_parm {
+	__u64	origin;
+	__u64	length;
+};
+
+struct kvm_s390_pv_unp {
+	__u64 addr;
+	__u64 size;
+	__u64 tweak;
+};
+
+enum pv_cmd_id {
+	KVM_PV_VM_CREATE,
+	KVM_PV_VM_DESTROY,
+	KVM_PV_VM_SET_SEC_PARMS,
+	KVM_PV_VM_UNPACK,
+	KVM_PV_VM_VERIFY,
+	KVM_PV_VCPU_CREATE,
+	KVM_PV_VCPU_DESTROY,
+};
+
+struct kvm_pv_cmd {
+	__u32	cmd;	/* Command to be executed */
+	__u16	rc;	/* Ultravisor return code */
+	__u16	rrc;	/* Ultravisor return reason code */
+	__u64	data;	/* Data or address */
+};
+
+/* Available with KVM_CAP_S390_PROTECTED */
+#define KVM_S390_PV_COMMAND		_IOW(KVMIO, 0xc5, struct kvm_pv_cmd)
+#define KVM_S390_PV_COMMAND_VCPU	_IOW(KVMIO, 0xc6, struct kvm_pv_cmd)
+
 /* Secure Encrypted Virtualization command */
 enum sev_cmd_id {
 	/* Guest initialization commands */
-- 
2.24.0

[RFCv2 09/37] KVM: s390: protvirt: Add KVM api documentation

[Christian Borntraeger]

From: Janosch Frank <frankja@linux.ibm.com>

Add documentation for KVM_CAP_S390_PROTECTED capability and the
KVM_S390_PV_COMMAND and KVM_S390_PV_COMMAND_VCPU ioctls.

Signed-off-by: Janosch Frank <frankja@linux.ibm.com>
---
 Documentation/virt/kvm/api.txt | 62 ++++++++++++++++++++++++++++++++++
 1 file changed, 62 insertions(+)

diff --git a/Documentation/virt/kvm/api.txt b/Documentation/virt/kvm/api.txt
index 73448764f544..a73fdae40e26 100644
--- a/Documentation/virt/kvm/api.txt
+++ b/Documentation/virt/kvm/api.txt
@@ -4204,6 +4204,61 @@ the clear cpu reset definition in the POP. However, the cpu is not put
 into ESA mode. This reset is a superset of the initial reset.
 
 
+4.125 KVM_S390_PV_COMMAND
+
+Capability: KVM_CAP_S390_PROTECTED
+Architectures: s390
+Type: vm ioctl
+Parameters: struct kvm_pv_cmd
+Returns: 0 on success, < 0 on error
+
+struct kvm_pv_cmd {
+	__u32	cmd;	/* Command to be executed */
+	__u16	rc;	/* Ultravisor return code */
+	__u16	rrc;	/* Ultravisor return reason code */
+	__u64	data;	/* Data or address */
+};
+
+cmd values:
+KVM_PV_VM_CREATE
+Allocate memory and register the VM with the Ultravisor, thereby
+donating memory to the Ultravisor making it inaccessible to KVM.
+
+KVM_PV_VM_DESTROY
+Unregisters the VM from the Ultravisor and frees memory that was
+donated, so the kernel can use it again. All registered VCPUs have to
+be unregistered beforehand and all memory has to be exported or
+shared.
+
+KVM_PV_VM_SET_SEC_PARMS
+Pass the image header from VM memory to the Ultravisor in preparation
+of image unpacking and verification.
+
+KVM_PV_VM_UNPACK
+Unpack (protect and decrypt) a page of the encrypted boot image.
+
+KVM_PV_VM_VERIFY
+Verify the integrity of the unpacked image. Only if this succeeds, KVM
+
+is allowed to start protected VCPUs.
+
+4.126 KVM_S390_PV_COMMAND_VCPU
+
+Capability: KVM_CAP_S390_PROTECTED
+Architectures: s390
+Type: vcpu ioctl
+Parameters: struct kvm_pv_cmd
+Returns: 0 on success, < 0 on error
+
+cmd values:
+KVM_PV_VCPU_CREATE
+Allocate memory and register a VCPU with the Ultravisor, thereby
+donating memory to the Ultravisor making it inaccessible to KVM.
+
+KVM_PV_VCPU_DESTROY
+Unregisters the VCPU from the Ultravisor and frees memory that was
+donated, so the kernel can use it again.
+
 5. The kvm_run structure
 ------------------------
 
@@ -5439,3 +5494,10 @@ Architectures: s390
 
 This capability indicates that the KVM_S390_NORMAL_RESET and
 KVM_S390_CLEAR_RESET ioctls are available.
+
+8.23 KVM_CAP_S390_PROTECTED
+
+Architecture: s390
+
+This capability indicates that KVM can start protected VMs and the
+Ultravisor has therefore been initialized.
-- 
2.24.0

[RFCv2 10/37] KVM: s390: protvirt: Secure memory is not mergeable

[Christian Borntraeger]

From: Janosch Frank <frankja@linux.ibm.com>

KSM will not work on secure pages, because when the kernel reads a
secure page, it will be encrypted and hence no two pages will look the
same.

Let's mark the guest pages as unmergeable when we transition to secure
mode.

Signed-off-by: Janosch Frank <frankja@linux.ibm.com>
Reviewed-by: Thomas Huth <thuth@redhat.com>
---
 arch/s390/include/asm/gmap.h |  1 +
 arch/s390/kvm/kvm-s390.c     |  6 ++++++
 arch/s390/mm/gmap.c          | 28 ++++++++++++++++++----------
 3 files changed, 25 insertions(+), 10 deletions(-)

diff --git a/arch/s390/include/asm/gmap.h b/arch/s390/include/asm/gmap.h
index f2ab8b6d4b57..bf819c62abfe 100644
--- a/arch/s390/include/asm/gmap.h
+++ b/arch/s390/include/asm/gmap.h
@@ -146,4 +146,5 @@ int gmap_mprotect_notify(struct gmap *, unsigned long start,
 
 void gmap_sync_dirty_log_pmd(struct gmap *gmap, unsigned long dirty_bitmap[4],
 			     unsigned long gaddr, unsigned long vmaddr);
+int gmap_mark_unmergeable(void);
 #endif /* _ASM_S390_GMAP_H */
diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
index 35f46404830f..741d81f57c3c 100644
--- a/arch/s390/kvm/kvm-s390.c
+++ b/arch/s390/kvm/kvm-s390.c
@@ -2181,6 +2181,12 @@ static int kvm_s390_handle_pv(struct kvm *kvm, struct kvm_pv_cmd *cmd)
 		if (r)
 			break;
 
+		down_write(&current->mm->mmap_sem);
+		r = gmap_mark_unmergeable();
+		up_write(&current->mm->mmap_sem);
+		if (r)
+			break;
+
 		mutex_lock(&kvm->lock);
 		kvm_s390_vcpu_block_all(kvm);
 		/* FMT 4 SIE needs esca */
diff --git a/arch/s390/mm/gmap.c b/arch/s390/mm/gmap.c
index edcdca97e85e..bf365a09f900 100644
--- a/arch/s390/mm/gmap.c
+++ b/arch/s390/mm/gmap.c
@@ -2548,6 +2548,23 @@ int s390_enable_sie(void)
 }
 EXPORT_SYMBOL_GPL(s390_enable_sie);
 
+int gmap_mark_unmergeable(void)
+{
+	struct mm_struct *mm = current->mm;
+	struct vm_area_struct *vma;
+
+	for (vma = mm->mmap; vma; vma = vma->vm_next) {
+		if (ksm_madvise(vma, vma->vm_start, vma->vm_end,
+				MADV_UNMERGEABLE, &vma->vm_flags)) {
+			mm->context.uses_skeys = 0;
+			return -ENOMEM;
+		}
+	}
+	mm->def_flags &= ~VM_MERGEABLE;
+	return 0;
+}
+EXPORT_SYMBOL_GPL(gmap_mark_unmergeable);
+
 /*
  * Enable storage key handling from now on and initialize the storage
  * keys with the default key.
@@ -2593,7 +2610,6 @@ static const struct mm_walk_ops enable_skey_walk_ops = {
 int s390_enable_skey(void)
 {
 	struct mm_struct *mm = current->mm;
-	struct vm_area_struct *vma;
 	int rc = 0;
 
 	down_write(&mm->mmap_sem);
@@ -2601,15 +2617,7 @@ int s390_enable_skey(void)
 		goto out_up;
 
 	mm->context.uses_skeys = 1;
-	for (vma = mm->mmap; vma; vma = vma->vm_next) {
-		if (ksm_madvise(vma, vma->vm_start, vma->vm_end,
-				MADV_UNMERGEABLE, &vma->vm_flags)) {
-			mm->context.uses_skeys = 0;
-			rc = -ENOMEM;
-			goto out_up;
-		}
-	}
-	mm->def_flags &= ~VM_MERGEABLE;
+	gmap_mark_unmergeable();
 
 	walk_page_range(mm, 0, TASK_SIZE, &enable_skey_walk_ops, NULL);
 
-- 
2.24.0

[RFCv2 11/37] KVM: s390/mm: Make pages accessible before destroying the guest

[Christian Borntraeger]

Before we destroy the secure configuration, we better make all
pages accessible again. This also happens during reboot, where we reboot
into a non-secure guest that then can go again into a secure mode. As
this "new" secure guest will have a new ID we cannot reuse the old page
state.

Signed-off-by: Christian Borntraeger <borntraeger@de.ibm.com>
---
 arch/s390/include/asm/pgtable.h |  1 +
 arch/s390/kvm/pv.c              |  2 ++
 arch/s390/mm/gmap.c             | 35 +++++++++++++++++++++++++++++++++
 3 files changed, 38 insertions(+)

diff --git a/arch/s390/include/asm/pgtable.h b/arch/s390/include/asm/pgtable.h
index 65b6bb47af0a..6e167dcc35f1 100644
--- a/arch/s390/include/asm/pgtable.h
+++ b/arch/s390/include/asm/pgtable.h
@@ -1669,6 +1669,7 @@ extern int vmem_remove_mapping(unsigned long start, unsigned long size);
 extern int s390_enable_sie(void);
 extern int s390_enable_skey(void);
 extern void s390_reset_cmma(struct mm_struct *mm);
+extern void s390_reset_acc(struct mm_struct *mm);
 
 /* s390 has a private copy of get unmapped area to deal with cache synonyms */
 #define HAVE_ARCH_UNMAPPED_AREA
diff --git a/arch/s390/kvm/pv.c b/arch/s390/kvm/pv.c
index a867b9e9c069..24d802072ac7 100644
--- a/arch/s390/kvm/pv.c
+++ b/arch/s390/kvm/pv.c
@@ -66,6 +66,8 @@ int kvm_s390_pv_destroy_vm(struct kvm *kvm)
 	int rc;
 	u32 ret;
 
+	/* make all pages accessible before destroying the guest */
+	s390_reset_acc(kvm->mm);
 	rc = uv_cmd_nodata(kvm_s390_pv_handle(kvm),
 			   UVC_CMD_DESTROY_SEC_CONF, &ret);
 	WRITE_ONCE(kvm->arch.gmap->se_handle, 0);
diff --git a/arch/s390/mm/gmap.c b/arch/s390/mm/gmap.c
index bf365a09f900..0b00e8d5fa39 100644
--- a/arch/s390/mm/gmap.c
+++ b/arch/s390/mm/gmap.c
@@ -2648,3 +2648,38 @@ void s390_reset_cmma(struct mm_struct *mm)
 	up_write(&mm->mmap_sem);
 }
 EXPORT_SYMBOL_GPL(s390_reset_cmma);
+
+/*
+ * make inaccessible pages accessible again
+ */
+static int __s390_reset_acc(pte_t *ptep, unsigned long addr,
+			    unsigned long next, struct mm_walk *walk)
+{
+	pte_t pte = READ_ONCE(*ptep);
+
+	if (pte_present(pte))
+		uv_convert_from_secure(pte_val(pte) & PAGE_MASK);
+	return 0;
+}
+
+static const struct mm_walk_ops reset_acc_walk_ops = {
+	.pte_entry		= __s390_reset_acc,
+};
+
+#include <linux/sched/mm.h>
+void s390_reset_acc(struct mm_struct *mm)
+{
+	/*
+	 * we might be called during
+	 * reset:                            we walk the pages and clear
+	 * close of all kvm file descriptor: we walk the pages and clear
+	 * exit of process on fd closure:    vma already gone, do nothing
+	 */
+	if (!mmget_not_zero(mm))
+		return;
+	down_read(&mm->mmap_sem);
+	walk_page_range(mm, 0, TASK_SIZE, &reset_acc_walk_ops, NULL);
+	up_read(&mm->mmap_sem);
+	mmput(mm);
+}
+EXPORT_SYMBOL_GPL(s390_reset_acc);
-- 
2.24.0

[RFCv2 12/37] KVM: s390: protvirt: Handle SE notification interceptions

[Christian Borntraeger]

From: Janosch Frank <frankja@linux.ibm.com>

Since KVM doesn't emulate any form of load control and load psw
instructions anymore, we wouldn't get an interception if PSWs or CRs
are changed in the guest. That means we can't inject IRQs right after
the guest is enabled for them.

The new interception codes solve that problem by being a notification
for changes to IRQ enablement relevant bits in CRs 0, 6 and 14, as
well a the machine check mask bit in the PSW.

No special handling is needed for these interception codes, the KVM
pre-run code will consult all necessary CRs and PSW bits and inject
IRQs the guest is enabled for.

Signed-off-by: Janosch Frank <frankja@linux.ibm.com>
---
 arch/s390/include/asm/kvm_host.h |  2 ++
 arch/s390/kvm/intercept.c        | 10 ++++++++++
 2 files changed, 12 insertions(+)

diff --git a/arch/s390/include/asm/kvm_host.h b/arch/s390/include/asm/kvm_host.h
index 841690d05080..d63ed05272ec 100644
--- a/arch/s390/include/asm/kvm_host.h
+++ b/arch/s390/include/asm/kvm_host.h
@@ -215,6 +215,8 @@ struct kvm_s390_sie_block {
 #define ICPT_PARTEXEC	0x38
 #define ICPT_IOINST	0x40
 #define ICPT_KSS	0x5c
+#define ICPT_PV_MCHKR	0x60
+#define ICPT_PV_INT_EN	0x64
 	__u8	icptcode;		/* 0x0050 */
 	__u8	icptstatus;		/* 0x0051 */
 	__u16	ihcpu;			/* 0x0052 */
diff --git a/arch/s390/kvm/intercept.c b/arch/s390/kvm/intercept.c
index a389fa85cca2..eaa2a21c3170 100644
--- a/arch/s390/kvm/intercept.c
+++ b/arch/s390/kvm/intercept.c
@@ -480,6 +480,16 @@ int kvm_handle_sie_intercept(struct kvm_vcpu *vcpu)
 	case ICPT_KSS:
 		rc = kvm_s390_skey_check_enable(vcpu);
 		break;
+	case ICPT_PV_MCHKR:
+		/* fallthrough */
+	case ICPT_PV_INT_EN:
+		/*
+		 * PSW bit 13 or a CR (0, 6, 14) changed and we might
+		 * now be able to deliver interrupts. The pre-run code
+		 * will take care of this.
+		 */
+		rc = 0;
+		break;
 	default:
 		return -EOPNOTSUPP;
 	}
-- 
2.24.0

[RFCv2 13/37] KVM: s390: protvirt: Instruction emulation

[Christian Borntraeger]

From: Janosch Frank <frankja@linux.ibm.com>

We have two new SIE exit codes dealing with instructions.
104 (0x68) for a secure instruction interception, on which the SIE needs
hypervisor action to complete the instruction. We can piggy-back on the
existing instruction handlers.

108 which is merely a notification and provides data for tracking and
management. For example this is used to tell the host about a new value
for the prefix register. As there will be several special case handlers
in later patches, we handle this in a separate function.

Signed-off-by: Janosch Frank <frankja@linux.ibm.com>
---
 arch/s390/include/asm/kvm_host.h |  2 ++
 arch/s390/kvm/intercept.c        | 11 +++++++++++
 2 files changed, 13 insertions(+)

diff --git a/arch/s390/include/asm/kvm_host.h b/arch/s390/include/asm/kvm_host.h
index d63ed05272ec..58845b315be0 100644
--- a/arch/s390/include/asm/kvm_host.h
+++ b/arch/s390/include/asm/kvm_host.h
@@ -217,6 +217,8 @@ struct kvm_s390_sie_block {
 #define ICPT_KSS	0x5c
 #define ICPT_PV_MCHKR	0x60
 #define ICPT_PV_INT_EN	0x64
+#define ICPT_PV_INSTR	0x68
+#define ICPT_PV_NOTIF	0x6c
 	__u8	icptcode;		/* 0x0050 */
 	__u8	icptstatus;		/* 0x0051 */
 	__u16	ihcpu;			/* 0x0052 */
diff --git a/arch/s390/kvm/intercept.c b/arch/s390/kvm/intercept.c
index eaa2a21c3170..4b3fbbde1674 100644
--- a/arch/s390/kvm/intercept.c
+++ b/arch/s390/kvm/intercept.c
@@ -444,6 +444,11 @@ static int handle_operexc(struct kvm_vcpu *vcpu)
 	return kvm_s390_inject_program_int(vcpu, PGM_OPERATION);
 }
 
+static int handle_pv_not(struct kvm_vcpu *vcpu)
+{
+	return handle_instruction(vcpu);
+}
+
 int kvm_handle_sie_intercept(struct kvm_vcpu *vcpu)
 {
 	int rc, per_rc = 0;
@@ -490,6 +495,12 @@ int kvm_handle_sie_intercept(struct kvm_vcpu *vcpu)
 		 */
 		rc = 0;
 		break;
+	case ICPT_PV_INSTR:
+		rc = handle_instruction(vcpu);
+		break;
+	case ICPT_PV_NOTIF:
+		rc = handle_pv_not(vcpu);
+		break;
 	default:
 		return -EOPNOTSUPP;
 	}
-- 
2.24.0

[RFCv2 14/37] KVM: s390: protvirt: Add interruption injection controls

[Christian Borntraeger]

From: Michael Mueller <mimu@linux.ibm.com>

Define the interruption injection codes and the related fields in the
sie control block for PVM interruption injection.

Signed-off-by: Michael Mueller <mimu@linux.ibm.com>
---
 arch/s390/include/asm/kvm_host.h | 25 +++++++++++++++++++++----
 1 file changed, 21 insertions(+), 4 deletions(-)

diff --git a/arch/s390/include/asm/kvm_host.h b/arch/s390/include/asm/kvm_host.h
index 58845b315be0..a45d10d87a8a 100644
--- a/arch/s390/include/asm/kvm_host.h
+++ b/arch/s390/include/asm/kvm_host.h
@@ -222,7 +222,15 @@ struct kvm_s390_sie_block {
 	__u8	icptcode;		/* 0x0050 */
 	__u8	icptstatus;		/* 0x0051 */
 	__u16	ihcpu;			/* 0x0052 */
-	__u8	reserved54[2];		/* 0x0054 */
+	__u8	reserved54;		/* 0x0054 */
+#define IICTL_CODE_NONE		 0x00
+#define IICTL_CODE_MCHK		 0x01
+#define IICTL_CODE_EXT		 0x02
+#define IICTL_CODE_IO		 0x03
+#define IICTL_CODE_RESTART	 0x04
+#define IICTL_CODE_SPECIFICATION 0x10
+#define IICTL_CODE_OPERAND	 0x11
+	__u8	iictl;			/* 0x0055 */
 	__u16	ipa;			/* 0x0056 */
 	__u32	ipb;			/* 0x0058 */
 	__u32	scaoh;			/* 0x005c */
@@ -259,7 +267,8 @@ struct kvm_s390_sie_block {
 #define HPID_KVM	0x4
 #define HPID_VSIE	0x5
 	__u8	hpid;			/* 0x00b8 */
-	__u8	reservedb9[11];		/* 0x00b9 */
+	__u8	reservedb9[7];		/* 0x00b9 */
+	__u32	eiparams;		/* 0x00c0 */
 	__u16	extcpuaddr;		/* 0x00c4 */
 	__u16	eic;			/* 0x00c6 */
 	__u32	reservedc8;		/* 0x00c8 */
@@ -275,8 +284,16 @@ struct kvm_s390_sie_block {
 	__u8	oai;			/* 0x00e2 */
 	__u8	armid;			/* 0x00e3 */
 	__u8	reservede4[4];		/* 0x00e4 */
-	__u64	tecmc;			/* 0x00e8 */
-	__u8	reservedf0[12];		/* 0x00f0 */
+	union {
+		__u64	tecmc;		/* 0x00e8 */
+		struct {
+			__u16	subchannel_id;	/* 0x00e8 */
+			__u16	subchannel_nr;	/* 0x00ea */
+			__u32	io_int_parm;	/* 0x00ec */
+			__u32	io_int_word;	/* 0x00f0 */
+		};
+	} __packed;
+	__u8	reservedf4[8];		/* 0x00f4 */
 #define CRYCB_FORMAT_MASK 0x00000003
 #define CRYCB_FORMAT0 0x00000000
 #define CRYCB_FORMAT1 0x00000001
-- 
2.24.0

[RFCv2 15/37] KVM: s390: protvirt: Implement interruption injection

[Christian Borntraeger]

From: Michael Mueller <mimu@linux.ibm.com>

The patch implements interruption injection for the following
list of interruption types:

  - I/O
    __deliver_io (III)

  - External
    __deliver_cpu_timer (IEI)
    __deliver_ckc (IEI)
    __deliver_emergency_signal (IEI)
    __deliver_external_call (IEI)

  - cpu restart
    __deliver_restart (IRI)

The service interrupt is handled in a followup patch.

Signed-off-by: Michael Mueller <mimu@linux.ibm.com>
Signed-off-by: Christian Borntraeger <borntraeger@de.ibm.com>
[fixes]
---
 arch/s390/include/asm/kvm_host.h |  8 +++
 arch/s390/kvm/interrupt.c        | 93 ++++++++++++++++++++++----------
 2 files changed, 74 insertions(+), 27 deletions(-)

diff --git a/arch/s390/include/asm/kvm_host.h b/arch/s390/include/asm/kvm_host.h
index a45d10d87a8a..989cea7a5591 100644
--- a/arch/s390/include/asm/kvm_host.h
+++ b/arch/s390/include/asm/kvm_host.h
@@ -563,6 +563,14 @@ enum irq_types {
 #define IRQ_PEND_MCHK_MASK ((1UL << IRQ_PEND_MCHK_REP) | \
 			    (1UL << IRQ_PEND_MCHK_EX))
 
+#define IRQ_PEND_MCHK_REP_MASK (1UL << IRQ_PEND_MCHK_REP)
+
+#define IRQ_PEND_EXT_II_MASK ((1UL << IRQ_PEND_EXT_CPU_TIMER)  | \
+			      (1UL << IRQ_PEND_EXT_CLOCK_COMP) | \
+			      (1UL << IRQ_PEND_EXT_EMERGENCY)  | \
+			      (1UL << IRQ_PEND_EXT_EXTERNAL)   | \
+			      (1UL << IRQ_PEND_EXT_SERVICE))
+
 struct kvm_s390_interrupt_info {
 	struct list_head list;
 	u64	type;
diff --git a/arch/s390/kvm/interrupt.c b/arch/s390/kvm/interrupt.c
index c06c89d370a7..ecdec6960a60 100644
--- a/arch/s390/kvm/interrupt.c
+++ b/arch/s390/kvm/interrupt.c
@@ -387,6 +387,12 @@ static unsigned long deliverable_irqs(struct kvm_vcpu *vcpu)
 		__clear_bit(IRQ_PEND_EXT_SERVICE, &active_mask);
 	if (psw_mchk_disabled(vcpu))
 		active_mask &= ~IRQ_PEND_MCHK_MASK;
+	/* PV guest cpus can have a single interruption injected at a time. */
+	if (kvm_s390_pv_is_protected(vcpu->kvm) &&
+	    vcpu->arch.sie_block->iictl != IICTL_CODE_NONE)
+		active_mask &= ~(IRQ_PEND_EXT_II_MASK |
+				 IRQ_PEND_IO_MASK |
+				 IRQ_PEND_MCHK_REP_MASK);
 	/*
 	 * Check both floating and local interrupt's cr14 because
 	 * bit IRQ_PEND_MCHK_REP could be set in both cases.
@@ -479,19 +485,23 @@ static void set_intercept_indicators(struct kvm_vcpu *vcpu)
 static int __must_check __deliver_cpu_timer(struct kvm_vcpu *vcpu)
 {
 	struct kvm_s390_local_interrupt *li = &vcpu->arch.local_int;
-	int rc;
+	int rc = 0;
 
 	vcpu->stat.deliver_cputm++;
 	trace_kvm_s390_deliver_interrupt(vcpu->vcpu_id, KVM_S390_INT_CPU_TIMER,
 					 0, 0);
-
-	rc  = put_guest_lc(vcpu, EXT_IRQ_CPU_TIMER,
-			   (u16 *)__LC_EXT_INT_CODE);
-	rc |= put_guest_lc(vcpu, 0, (u16 *)__LC_EXT_CPU_ADDR);
-	rc |= write_guest_lc(vcpu, __LC_EXT_OLD_PSW,
-			     &vcpu->arch.sie_block->gpsw, sizeof(psw_t));
-	rc |= read_guest_lc(vcpu, __LC_EXT_NEW_PSW,
-			    &vcpu->arch.sie_block->gpsw, sizeof(psw_t));
+	if (kvm_s390_pv_is_protected(vcpu->kvm)) {
+		vcpu->arch.sie_block->iictl = IICTL_CODE_EXT;
+		vcpu->arch.sie_block->eic = EXT_IRQ_CPU_TIMER;
+	} else {
+		rc  = put_guest_lc(vcpu, EXT_IRQ_CPU_TIMER,
+				   (u16 *)__LC_EXT_INT_CODE);
+		rc |= put_guest_lc(vcpu, 0, (u16 *)__LC_EXT_CPU_ADDR);
+		rc |= write_guest_lc(vcpu, __LC_EXT_OLD_PSW,
+				     &vcpu->arch.sie_block->gpsw, sizeof(psw_t));
+		rc |= read_guest_lc(vcpu, __LC_EXT_NEW_PSW,
+				    &vcpu->arch.sie_block->gpsw, sizeof(psw_t));
+	}
 	clear_bit(IRQ_PEND_EXT_CPU_TIMER, &li->pending_irqs);
 	return rc ? -EFAULT : 0;
 }
@@ -499,19 +509,23 @@ static int __must_check __deliver_cpu_timer(struct kvm_vcpu *vcpu)
 static int __must_check __deliver_ckc(struct kvm_vcpu *vcpu)
 {
 	struct kvm_s390_local_interrupt *li = &vcpu->arch.local_int;
-	int rc;
+	int rc = 0;
 
 	vcpu->stat.deliver_ckc++;
 	trace_kvm_s390_deliver_interrupt(vcpu->vcpu_id, KVM_S390_INT_CLOCK_COMP,
 					 0, 0);
-
-	rc  = put_guest_lc(vcpu, EXT_IRQ_CLK_COMP,
-			   (u16 __user *)__LC_EXT_INT_CODE);
-	rc |= put_guest_lc(vcpu, 0, (u16 *)__LC_EXT_CPU_ADDR);
-	rc |= write_guest_lc(vcpu, __LC_EXT_OLD_PSW,
-			     &vcpu->arch.sie_block->gpsw, sizeof(psw_t));
-	rc |= read_guest_lc(vcpu, __LC_EXT_NEW_PSW,
-			    &vcpu->arch.sie_block->gpsw, sizeof(psw_t));
+	if (kvm_s390_pv_is_protected(vcpu->kvm)) {
+		vcpu->arch.sie_block->iictl = IICTL_CODE_EXT;
+		vcpu->arch.sie_block->eic = EXT_IRQ_CLK_COMP;
+	} else {
+		rc  = put_guest_lc(vcpu, EXT_IRQ_CLK_COMP,
+				   (u16 __user *)__LC_EXT_INT_CODE);
+		rc |= put_guest_lc(vcpu, 0, (u16 *)__LC_EXT_CPU_ADDR);
+		rc |= write_guest_lc(vcpu, __LC_EXT_OLD_PSW,
+				     &vcpu->arch.sie_block->gpsw, sizeof(psw_t));
+		rc |= read_guest_lc(vcpu, __LC_EXT_NEW_PSW,
+				    &vcpu->arch.sie_block->gpsw, sizeof(psw_t));
+	}
 	clear_bit(IRQ_PEND_EXT_CLOCK_COMP, &li->pending_irqs);
 	return rc ? -EFAULT : 0;
 }
@@ -533,7 +547,6 @@ static int __must_check __deliver_pfault_init(struct kvm_vcpu *vcpu)
 	trace_kvm_s390_deliver_interrupt(vcpu->vcpu_id,
 					 KVM_S390_INT_PFAULT_INIT,
 					 0, ext.ext_params2);
-
 	rc  = put_guest_lc(vcpu, EXT_IRQ_CP_SERVICE, (u16 *) __LC_EXT_INT_CODE);
 	rc |= put_guest_lc(vcpu, PFAULT_INIT, (u16 *) __LC_EXT_CPU_ADDR);
 	rc |= write_guest_lc(vcpu, __LC_EXT_OLD_PSW,
@@ -696,17 +709,21 @@ static int __must_check __deliver_machine_check(struct kvm_vcpu *vcpu)
 static int __must_check __deliver_restart(struct kvm_vcpu *vcpu)
 {
 	struct kvm_s390_local_interrupt *li = &vcpu->arch.local_int;
-	int rc;
+	int rc = 0;
 
 	VCPU_EVENT(vcpu, 3, "%s", "deliver: cpu restart");
 	vcpu->stat.deliver_restart_signal++;
 	trace_kvm_s390_deliver_interrupt(vcpu->vcpu_id, KVM_S390_RESTART, 0, 0);
 
-	rc  = write_guest_lc(vcpu,
-			     offsetof(struct lowcore, restart_old_psw),
-			     &vcpu->arch.sie_block->gpsw, sizeof(psw_t));
-	rc |= read_guest_lc(vcpu, offsetof(struct lowcore, restart_psw),
-			    &vcpu->arch.sie_block->gpsw, sizeof(psw_t));
+	if (kvm_s390_pv_is_protected(vcpu->kvm)) {
+		vcpu->arch.sie_block->iictl = IICTL_CODE_RESTART;
+	} else {
+		rc  = write_guest_lc(vcpu,
+				     offsetof(struct lowcore, restart_old_psw),
+				     &vcpu->arch.sie_block->gpsw, sizeof(psw_t));
+		rc |= read_guest_lc(vcpu, offsetof(struct lowcore, restart_psw),
+				    &vcpu->arch.sie_block->gpsw, sizeof(psw_t));
+	}
 	clear_bit(IRQ_PEND_RESTART, &li->pending_irqs);
 	return rc ? -EFAULT : 0;
 }
@@ -748,6 +765,12 @@ static int __must_check __deliver_emergency_signal(struct kvm_vcpu *vcpu)
 	vcpu->stat.deliver_emergency_signal++;
 	trace_kvm_s390_deliver_interrupt(vcpu->vcpu_id, KVM_S390_INT_EMERGENCY,
 					 cpu_addr, 0);
+	if (kvm_s390_pv_is_protected(vcpu->kvm)) {
+		vcpu->arch.sie_block->iictl = IICTL_CODE_EXT;
+		vcpu->arch.sie_block->eic = EXT_IRQ_EMERGENCY_SIG;
+		vcpu->arch.sie_block->extcpuaddr = cpu_addr;
+		return 0;
+	}
 
 	rc  = put_guest_lc(vcpu, EXT_IRQ_EMERGENCY_SIG,
 			   (u16 *)__LC_EXT_INT_CODE);
@@ -776,6 +799,12 @@ static int __must_check __deliver_external_call(struct kvm_vcpu *vcpu)
 	trace_kvm_s390_deliver_interrupt(vcpu->vcpu_id,
 					 KVM_S390_INT_EXTERNAL_CALL,
 					 extcall.code, 0);
+	if (kvm_s390_pv_is_protected(vcpu->kvm)) {
+		vcpu->arch.sie_block->iictl = IICTL_CODE_EXT;
+		vcpu->arch.sie_block->eic = EXT_IRQ_EXTERNAL_CALL;
+		vcpu->arch.sie_block->extcpuaddr = extcall.code;
+		return 0;
+	}
 
 	rc  = put_guest_lc(vcpu, EXT_IRQ_EXTERNAL_CALL,
 			   (u16 *)__LC_EXT_INT_CODE);
@@ -1028,6 +1057,15 @@ static int __do_deliver_io(struct kvm_vcpu *vcpu, struct kvm_s390_io_info *io)
 {
 	int rc;
 
+	if (kvm_s390_pv_is_protected(vcpu->kvm)) {
+		vcpu->arch.sie_block->iictl = IICTL_CODE_IO;
+		vcpu->arch.sie_block->subchannel_id = io->subchannel_id;
+		vcpu->arch.sie_block->subchannel_nr = io->subchannel_nr;
+		vcpu->arch.sie_block->io_int_parm = io->io_int_parm;
+		vcpu->arch.sie_block->io_int_word = io->io_int_word;
+		return 0;
+	}
+
 	rc  = put_guest_lc(vcpu, io->subchannel_id, (u16 *)__LC_SUBCHANNEL_ID);
 	rc |= put_guest_lc(vcpu, io->subchannel_nr, (u16 *)__LC_SUBCHANNEL_NR);
 	rc |= put_guest_lc(vcpu, io->io_int_parm, (u32 *)__LC_IO_INT_PARM);
@@ -1421,7 +1459,7 @@ static int __inject_extcall(struct kvm_vcpu *vcpu, struct kvm_s390_irq *irq)
 	if (kvm_get_vcpu_by_id(vcpu->kvm, src_id) == NULL)
 		return -EINVAL;
 
-	if (sclp.has_sigpif)
+	if (sclp.has_sigpif && !kvm_s390_pv_handle_cpu(vcpu))
 		return sca_inject_ext_call(vcpu, src_id);
 
 	if (test_and_set_bit(IRQ_PEND_EXT_EXTERNAL, &li->pending_irqs))
@@ -1834,7 +1872,8 @@ static void __floating_irq_kick(struct kvm *kvm, u64 type)
 		break;
 	case KVM_S390_INT_IO_MIN...KVM_S390_INT_IO_MAX:
 		if (!(type & KVM_S390_INT_IO_AI_MASK &&
-		      kvm->arch.gisa_int.origin))
+		      kvm->arch.gisa_int.origin) ||
+		      kvm_s390_pv_handle_cpu(dst_vcpu))
 			kvm_s390_set_cpuflags(dst_vcpu, CPUSTAT_IO_INT);
 		break;
 	default:
-- 
2.24.0

[RFCv2 16/37] KVM: s390: protvirt: Add SCLP interrupt handling

[Christian Borntraeger]

The sclp interrupt is kind of special. The ultravisor polices that we
do not inject and sclp interrupt with payload if no sccb is outstanding.
On the other hand we have "asynchronous" event interrupts, e.g. for
console input.
We separate both variants into sclp interrupt and sclp event interrupt.
The sclp interrupt is masked until a previous servc instruction has
finished (sie exit 108).

Signed-off-by: Christian Borntraeger <borntraeger@de.ibm.com>
---
 arch/s390/include/asm/kvm_host.h |  6 ++-
 arch/s390/kvm/intercept.c        | 25 +++++++++
 arch/s390/kvm/interrupt.c        | 92 ++++++++++++++++++++++++++------
 arch/s390/kvm/kvm-s390.c         |  2 +
 4 files changed, 108 insertions(+), 17 deletions(-)

diff --git a/arch/s390/include/asm/kvm_host.h b/arch/s390/include/asm/kvm_host.h
index 989cea7a5591..5e85358d9090 100644
--- a/arch/s390/include/asm/kvm_host.h
+++ b/arch/s390/include/asm/kvm_host.h
@@ -503,6 +503,7 @@ enum irq_types {
 	IRQ_PEND_PFAULT_INIT,
 	IRQ_PEND_EXT_HOST,
 	IRQ_PEND_EXT_SERVICE,
+	IRQ_PEND_EXT_SERVICE_EV,
 	IRQ_PEND_EXT_TIMING,
 	IRQ_PEND_EXT_CPU_TIMER,
 	IRQ_PEND_EXT_CLOCK_COMP,
@@ -547,6 +548,7 @@ enum irq_types {
 			   (1UL << IRQ_PEND_EXT_TIMING)     | \
 			   (1UL << IRQ_PEND_EXT_HOST)       | \
 			   (1UL << IRQ_PEND_EXT_SERVICE)    | \
+			   (1UL << IRQ_PEND_EXT_SERVICE_EV) | \
 			   (1UL << IRQ_PEND_VIRTIO)         | \
 			   (1UL << IRQ_PEND_PFAULT_INIT)    | \
 			   (1UL << IRQ_PEND_PFAULT_DONE))
@@ -569,7 +571,8 @@ enum irq_types {
 			      (1UL << IRQ_PEND_EXT_CLOCK_COMP) | \
 			      (1UL << IRQ_PEND_EXT_EMERGENCY)  | \
 			      (1UL << IRQ_PEND_EXT_EXTERNAL)   | \
-			      (1UL << IRQ_PEND_EXT_SERVICE))
+			      (1UL << IRQ_PEND_EXT_SERVICE)    | \
+			      (1UL << IRQ_PEND_EXT_SERVICE_EV))
 
 struct kvm_s390_interrupt_info {
 	struct list_head list;
@@ -629,6 +632,7 @@ struct kvm_s390_local_interrupt {
 
 struct kvm_s390_float_interrupt {
 	unsigned long pending_irqs;
+	unsigned long masked_irqs;
 	spinlock_t lock;
 	struct list_head lists[FIRQ_LIST_COUNT];
 	int counters[FIRQ_MAX_COUNT];
diff --git a/arch/s390/kvm/intercept.c b/arch/s390/kvm/intercept.c
index 4b3fbbde1674..c22214967214 100644
--- a/arch/s390/kvm/intercept.c
+++ b/arch/s390/kvm/intercept.c
@@ -444,8 +444,33 @@ static int handle_operexc(struct kvm_vcpu *vcpu)
 	return kvm_s390_inject_program_int(vcpu, PGM_OPERATION);
 }
 
+static int handle_pv_sclp(struct kvm_vcpu *vcpu)
+{
+	struct kvm_s390_float_interrupt *fi = &vcpu->kvm->arch.float_int;
+
+	spin_lock(&fi->lock);
+	/*
+	 * 2 cases:
+	 * a: an sccb answering interrupt was already pending or in flight.
+	 *    As the sccb value is not used we can simply set some more bits
+	 *    and make sure that we deliver something
+	 * b: an error sccb interrupt needs to be injected so we also inject
+	 *    something and let firmware do the right thing.
+	 * This makes sure, that both errors and real sccb returns will only
+	 * be delivered when we are unmasked.
+	 */
+	fi->srv_signal.ext_params |= 0x43000;
+	set_bit(IRQ_PEND_EXT_SERVICE, &fi->pending_irqs);
+	clear_bit(IRQ_PEND_EXT_SERVICE, &fi->masked_irqs);
+	spin_unlock(&fi->lock);
+	return 0;
+}
+
 static int handle_pv_not(struct kvm_vcpu *vcpu)
 {
+	if (vcpu->arch.sie_block->ipa == 0xb220)
+		return handle_pv_sclp(vcpu);
+
 	return handle_instruction(vcpu);
 }
 
diff --git a/arch/s390/kvm/interrupt.c b/arch/s390/kvm/interrupt.c
index ecdec6960a60..c707725e618b 100644
--- a/arch/s390/kvm/interrupt.c
+++ b/arch/s390/kvm/interrupt.c
@@ -324,8 +324,11 @@ static inline int gisa_tac_ipm_gisc(struct kvm_s390_gisa *gisa, u32 gisc)
 
 static inline unsigned long pending_irqs_no_gisa(struct kvm_vcpu *vcpu)
 {
-	return vcpu->kvm->arch.float_int.pending_irqs |
-		vcpu->arch.local_int.pending_irqs;
+	unsigned long pending = vcpu->kvm->arch.float_int.pending_irqs |
+				vcpu->arch.local_int.pending_irqs;
+
+	pending &= ~vcpu->kvm->arch.float_int.masked_irqs;
+	return pending;
 }
 
 static inline unsigned long pending_irqs(struct kvm_vcpu *vcpu)
@@ -383,8 +386,10 @@ static unsigned long deliverable_irqs(struct kvm_vcpu *vcpu)
 		__clear_bit(IRQ_PEND_EXT_CLOCK_COMP, &active_mask);
 	if (!(vcpu->arch.sie_block->gcr[0] & CR0_CPU_TIMER_SUBMASK))
 		__clear_bit(IRQ_PEND_EXT_CPU_TIMER, &active_mask);
-	if (!(vcpu->arch.sie_block->gcr[0] & CR0_SERVICE_SIGNAL_SUBMASK))
+	if (!(vcpu->arch.sie_block->gcr[0] & CR0_SERVICE_SIGNAL_SUBMASK)) {
 		__clear_bit(IRQ_PEND_EXT_SERVICE, &active_mask);
+		__clear_bit(IRQ_PEND_EXT_SERVICE_EV, &active_mask);
+	}
 	if (psw_mchk_disabled(vcpu))
 		active_mask &= ~IRQ_PEND_MCHK_MASK;
 	/* PV guest cpus can have a single interruption injected at a time. */
@@ -931,6 +936,31 @@ static int __must_check __deliver_prog(struct kvm_vcpu *vcpu)
 	return rc ? -EFAULT : 0;
 }
 
+#define SCCB_MASK 0xFFFFFFF8
+#define SCCB_EVENT_PENDING 0x3
+
+static int write_sclp(struct kvm_vcpu *vcpu, u32 parm)
+{
+	int rc;
+
+	if (kvm_s390_pv_handle_cpu(vcpu)) {
+		vcpu->arch.sie_block->iictl = IICTL_CODE_EXT;
+		vcpu->arch.sie_block->eic = EXT_IRQ_SERVICE_SIG;
+		vcpu->arch.sie_block->eiparams = parm;
+		return 0;
+	}
+
+	rc  = put_guest_lc(vcpu, EXT_IRQ_SERVICE_SIG, (u16 *)__LC_EXT_INT_CODE);
+	rc |= put_guest_lc(vcpu, 0, (u16 *)__LC_EXT_CPU_ADDR);
+	rc |= write_guest_lc(vcpu, __LC_EXT_OLD_PSW,
+			     &vcpu->arch.sie_block->gpsw, sizeof(psw_t));
+	rc |= read_guest_lc(vcpu, __LC_EXT_NEW_PSW,
+			    &vcpu->arch.sie_block->gpsw, sizeof(psw_t));
+	rc |= put_guest_lc(vcpu, parm,
+			   (u32 *)__LC_EXT_PARAMS);
+	return rc;
+}
+
 static int __must_check __deliver_service(struct kvm_vcpu *vcpu)
 {
 	struct kvm_s390_float_interrupt *fi = &vcpu->kvm->arch.float_int;
@@ -938,13 +968,17 @@ static int __must_check __deliver_service(struct kvm_vcpu *vcpu)
 	int rc = 0;
 
 	spin_lock(&fi->lock);
-	if (!(test_bit(IRQ_PEND_EXT_SERVICE, &fi->pending_irqs))) {
+	if (test_bit(IRQ_PEND_EXT_SERVICE, &fi->masked_irqs) ||
+	    !(test_bit(IRQ_PEND_EXT_SERVICE, &fi->pending_irqs))) {
 		spin_unlock(&fi->lock);
 		return 0;
 	}
 	ext = fi->srv_signal;
 	memset(&fi->srv_signal, 0, sizeof(ext));
 	clear_bit(IRQ_PEND_EXT_SERVICE, &fi->pending_irqs);
+	clear_bit(IRQ_PEND_EXT_SERVICE_EV, &fi->pending_irqs);
+	if (kvm_s390_pv_is_protected(vcpu->kvm))
+		set_bit(IRQ_PEND_EXT_SERVICE, &fi->masked_irqs);
 	spin_unlock(&fi->lock);
 
 	VCPU_EVENT(vcpu, 4, "deliver: sclp parameter 0x%x",
@@ -953,15 +987,33 @@ static int __must_check __deliver_service(struct kvm_vcpu *vcpu)
 	trace_kvm_s390_deliver_interrupt(vcpu->vcpu_id, KVM_S390_INT_SERVICE,
 					 ext.ext_params, 0);
 
-	rc  = put_guest_lc(vcpu, EXT_IRQ_SERVICE_SIG, (u16 *)__LC_EXT_INT_CODE);
-	rc |= put_guest_lc(vcpu, 0, (u16 *)__LC_EXT_CPU_ADDR);
-	rc |= write_guest_lc(vcpu, __LC_EXT_OLD_PSW,
-			     &vcpu->arch.sie_block->gpsw, sizeof(psw_t));
-	rc |= read_guest_lc(vcpu, __LC_EXT_NEW_PSW,
-			    &vcpu->arch.sie_block->gpsw, sizeof(psw_t));
-	rc |= put_guest_lc(vcpu, ext.ext_params,
-			   (u32 *)__LC_EXT_PARAMS);
+	rc = write_sclp(vcpu, ext.ext_params);
+	return rc ? -EFAULT : 0;
+}
+
+static int __must_check __deliver_service_ev(struct kvm_vcpu *vcpu)
+{
+	struct kvm_s390_float_interrupt *fi = &vcpu->kvm->arch.float_int;
+	struct kvm_s390_ext_info ext;
+	int rc = 0;
+
+	spin_lock(&fi->lock);
+	if (!(test_bit(IRQ_PEND_EXT_SERVICE_EV, &fi->pending_irqs))) {
+		spin_unlock(&fi->lock);
+		return 0;
+	}
+	ext = fi->srv_signal;
+	/* only clear the event bit */
+	fi->srv_signal.ext_params &= ~SCCB_EVENT_PENDING;
+	clear_bit(IRQ_PEND_EXT_SERVICE_EV, &fi->pending_irqs);
+	spin_unlock(&fi->lock);
+
+	VCPU_EVENT(vcpu, 4, "%s", "deliver: sclp parameter event");
+	vcpu->stat.deliver_service_signal++;
+	trace_kvm_s390_deliver_interrupt(vcpu->vcpu_id, KVM_S390_INT_SERVICE,
+					 ext.ext_params, 0);
 
+	rc = write_sclp(vcpu, SCCB_EVENT_PENDING);
 	return rc ? -EFAULT : 0;
 }
 
@@ -1367,6 +1419,9 @@ int __must_check kvm_s390_deliver_pending_interrupts(struct kvm_vcpu *vcpu)
 		case IRQ_PEND_EXT_SERVICE:
 			rc = __deliver_service(vcpu);
 			break;
+		case IRQ_PEND_EXT_SERVICE_EV:
+			rc = __deliver_service_ev(vcpu);
+			break;
 		case IRQ_PEND_PFAULT_DONE:
 			rc = __deliver_pfault_done(vcpu);
 			break;
@@ -1719,9 +1774,6 @@ struct kvm_s390_interrupt_info *kvm_s390_get_io_int(struct kvm *kvm,
 	return inti;
 }
 
-#define SCCB_MASK 0xFFFFFFF8
-#define SCCB_EVENT_PENDING 0x3
-
 static int __inject_service(struct kvm *kvm,
 			     struct kvm_s390_interrupt_info *inti)
 {
@@ -1730,6 +1782,11 @@ static int __inject_service(struct kvm *kvm,
 	kvm->stat.inject_service_signal++;
 	spin_lock(&fi->lock);
 	fi->srv_signal.ext_params |= inti->ext.ext_params & SCCB_EVENT_PENDING;
+
+	/* We always allow events, track them separately from the sccb ints */
+	if (fi->srv_signal.ext_params & SCCB_EVENT_PENDING)
+		set_bit(IRQ_PEND_EXT_SERVICE_EV, &fi->pending_irqs);
+
 	/*
 	 * Early versions of the QEMU s390 bios will inject several
 	 * service interrupts after another without handling a
@@ -2121,6 +2178,8 @@ void kvm_s390_clear_float_irqs(struct kvm *kvm)
 
 	spin_lock(&fi->lock);
 	fi->pending_irqs = 0;
+	if (!kvm_s390_pv_is_protected(kvm))
+		fi->masked_irqs = 0;
 	memset(&fi->srv_signal, 0, sizeof(fi->srv_signal));
 	memset(&fi->mchk, 0, sizeof(fi->mchk));
 	for (i = 0; i < FIRQ_LIST_COUNT; i++)
@@ -2185,7 +2244,8 @@ static int get_all_floating_irqs(struct kvm *kvm, u8 __user *usrbuf, u64 len)
 			n++;
 		}
 	}
-	if (test_bit(IRQ_PEND_EXT_SERVICE, &fi->pending_irqs)) {
+	if (test_bit(IRQ_PEND_EXT_SERVICE, &fi->pending_irqs) ||
+	    test_bit(IRQ_PEND_EXT_SERVICE_EV, &fi->pending_irqs)) {
 		if (n == max_irqs) {
 			/* signal userspace to try again */
 			ret = -ENOMEM;
diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
index 741d81f57c3c..39bf39a10cf2 100644
--- a/arch/s390/kvm/kvm-s390.c
+++ b/arch/s390/kvm/kvm-s390.c
@@ -2194,6 +2194,8 @@ static int kvm_s390_handle_pv(struct kvm *kvm, struct kvm_pv_cmd *cmd)
 		if (!r)
 			r = kvm_s390_pv_create_vm(kvm);
 		kvm_s390_vcpu_unblock_all(kvm);
+		/* we need to block service interrupts from now on */
+		set_bit(IRQ_PEND_EXT_SERVICE, &kvm->arch.float_int.masked_irqs);
 		mutex_unlock(&kvm->lock);
 		break;
 	}
-- 
2.24.0

[RFCv2 17/37] KVM: s390: protvirt: Add machine-check interruption injection controls

[Christian Borntraeger]

From: Michael Mueller <mimu@linux.ibm.com>

The following fields are added to the sie control block type 4:
     - Machine Check Interruption Code (mcic)
     - External Damage Code (edc)
     - Failing Storage Address (faddr)

Signed-off-by: Michael Mueller <mimu@linux.ibm.com>
---
 arch/s390/include/asm/kvm_host.h | 33 +++++++++++++++++++++++---------
 1 file changed, 24 insertions(+), 9 deletions(-)

diff --git a/arch/s390/include/asm/kvm_host.h b/arch/s390/include/asm/kvm_host.h
index 5e85358d9090..f5ca53574406 100644
--- a/arch/s390/include/asm/kvm_host.h
+++ b/arch/s390/include/asm/kvm_host.h
@@ -268,16 +268,31 @@ struct kvm_s390_sie_block {
 #define HPID_VSIE	0x5
 	__u8	hpid;			/* 0x00b8 */
 	__u8	reservedb9[7];		/* 0x00b9 */
-	__u32	eiparams;		/* 0x00c0 */
-	__u16	extcpuaddr;		/* 0x00c4 */
-	__u16	eic;			/* 0x00c6 */
+	union {
+		struct {
+			__u32	eiparams;	/* 0x00c0 */
+			__u16	extcpuaddr;	/* 0x00c4 */
+			__u16	eic;		/* 0x00c6 */
+		};
+		__u64	mcic;			/* 0x00c0 */
+	} __packed;
 	__u32	reservedc8;		/* 0x00c8 */
-	__u16	pgmilc;			/* 0x00cc */
-	__u16	iprcc;			/* 0x00ce */
-	__u32	dxc;			/* 0x00d0 */
-	__u16	mcn;			/* 0x00d4 */
-	__u8	perc;			/* 0x00d6 */
-	__u8	peratmid;		/* 0x00d7 */
+	union {
+		struct {
+			__u16	pgmilc;		/* 0x00cc */
+			__u16	iprcc;		/* 0x00ce */
+		};
+		__u32	edc;			/* 0x00cc */
+	} __packed;
+	union {
+		struct {
+			__u32	dxc;		/* 0x00d0 */
+			__u16	mcn;		/* 0x00d4 */
+			__u8	perc;		/* 0x00d6 */
+			__u8	peratmid;	/* 0x00d7 */
+		};
+		__u64	faddr;			/* 0x00d0 */
+	} __packed;
 	__u64	peraddr;		/* 0x00d8 */
 	__u8	eai;			/* 0x00e0 */
 	__u8	peraid;			/* 0x00e1 */
-- 
2.24.0

[RFCv2 18/37] KVM: s390: protvirt: Implement machine-check interruption injection

[Christian Borntraeger]

From: Michael Mueller <mimu@linux.ibm.com>

Similar to external interrupts, the hypervisor can inject machine
checks by providing the right data in the interrupt injection controls.

Signed-off-by: Michael Mueller <mimu@linux.ibm.com>
---
 arch/s390/kvm/interrupt.c | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/arch/s390/kvm/interrupt.c b/arch/s390/kvm/interrupt.c
index c707725e618b..a98f1dfde8de 100644
--- a/arch/s390/kvm/interrupt.c
+++ b/arch/s390/kvm/interrupt.c
@@ -571,6 +571,14 @@ static int __write_machine_check(struct kvm_vcpu *vcpu,
 	union mci mci;
 	int rc;
 
+	if (kvm_s390_pv_is_protected(vcpu->kvm)) {
+		vcpu->arch.sie_block->iictl = IICTL_CODE_MCHK;
+		vcpu->arch.sie_block->mcic = mchk->mcic;
+		vcpu->arch.sie_block->faddr = mchk->failing_storage_address;
+		vcpu->arch.sie_block->edc = mchk->ext_damage_code;
+		return 0;
+	}
+
 	mci.val = mchk->mcic;
 	/* take care of lazy register loading */
 	save_fpu_regs();
-- 
2.24.0

[RFCv2 19/37] KVM: s390: protvirt: Handle spec exception loops

[Christian Borntraeger]

From: Janosch Frank <frankja@linux.ibm.com>

SIE intercept code 8 is used only on exception loops for protected
guests. That means we need stop the guest when we see it.

Signed-off-by: Janosch Frank <frankja@linux.ibm.com>
Reviewed-by: Thomas Huth <thuth@redhat.com>
---
 arch/s390/kvm/intercept.c | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/arch/s390/kvm/intercept.c b/arch/s390/kvm/intercept.c
index c22214967214..d63f9cf10360 100644
--- a/arch/s390/kvm/intercept.c
+++ b/arch/s390/kvm/intercept.c
@@ -231,6 +231,13 @@ static int handle_prog(struct kvm_vcpu *vcpu)
 
 	vcpu->stat.exit_program_interruption++;
 
+	/*
+	 * Intercept 8 indicates a loop of specification exceptions
+	 * for protected guests
+	 */
+	if (kvm_s390_pv_is_protected(vcpu->kvm))
+		return -EOPNOTSUPP;
+
 	if (guestdbg_enabled(vcpu) && per_event(vcpu)) {
 		rc = kvm_s390_handle_per_event(vcpu);
 		if (rc)
-- 
2.24.0

[RFCv2 20/37] KVM: s390: protvirt: Add new gprs location handling

[Christian Borntraeger]

From: Janosch Frank <frankja@linux.ibm.com>

Guest registers for protected guests are stored at offset 0x380.

Signed-off-by: Janosch Frank <frankja@linux.ibm.com>
---
 arch/s390/include/asm/kvm_host.h |  4 +++-
 arch/s390/kvm/kvm-s390.c         | 11 +++++++++++
 2 files changed, 14 insertions(+), 1 deletion(-)

diff --git a/arch/s390/include/asm/kvm_host.h b/arch/s390/include/asm/kvm_host.h
index f5ca53574406..125511ec6eb0 100644
--- a/arch/s390/include/asm/kvm_host.h
+++ b/arch/s390/include/asm/kvm_host.h
@@ -343,7 +343,9 @@ struct kvm_s390_itdb {
 struct sie_page {
 	struct kvm_s390_sie_block sie_block;
 	struct mcck_volatile_info mcck_info;	/* 0x0200 */
-	__u8 reserved218[1000];		/* 0x0218 */
+	__u8 reserved218[360];		/* 0x0218 */
+	__u64 pv_grregs[16];		/* 0x380 */
+	__u8 reserved400[512];
 	struct kvm_s390_itdb itdb;	/* 0x0600 */
 	__u8 reserved700[2304];		/* 0x0700 */
 };
diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
index 39bf39a10cf2..1945180b857a 100644
--- a/arch/s390/kvm/kvm-s390.c
+++ b/arch/s390/kvm/kvm-s390.c
@@ -3996,6 +3996,7 @@ static int vcpu_post_run(struct kvm_vcpu *vcpu, int exit_reason)
 static int __vcpu_run(struct kvm_vcpu *vcpu)
 {
 	int rc, exit_reason;
+	struct sie_page *sie_page = (struct sie_page *)vcpu->arch.sie_block;
 
 	/*
 	 * We try to hold kvm->srcu during most of vcpu_run (except when run-
@@ -4017,8 +4018,18 @@ static int __vcpu_run(struct kvm_vcpu *vcpu)
 		guest_enter_irqoff();
 		__disable_cpu_timer_accounting(vcpu);
 		local_irq_enable();
+		if (kvm_s390_pv_is_protected(vcpu->kvm)) {
+			memcpy(sie_page->pv_grregs,
+			       vcpu->run->s.regs.gprs,
+			       sizeof(sie_page->pv_grregs));
+		}
 		exit_reason = sie64a(vcpu->arch.sie_block,
 				     vcpu->run->s.regs.gprs);
+		if (kvm_s390_pv_is_protected(vcpu->kvm)) {
+			memcpy(vcpu->run->s.regs.gprs,
+			       sie_page->pv_grregs,
+			       sizeof(sie_page->pv_grregs));
+		}
 		local_irq_disable();
 		__enable_cpu_timer_accounting(vcpu);
 		guest_exit_irqoff();
-- 
2.24.0

[RFCv2 21/37] KVM: S390: protvirt: Introduce instruction data area bounce buffer

[Christian Borntraeger]

From: Janosch Frank <frankja@linux.ibm.com>

Now that we can't access guest memory anymore, we have a dedicated
sattelite block that's a bounce buffer for instruction data.

We re-use the memop interface to copy the instruction data to / from
userspace. This lets us re-use a lot of QEMU code which used that
interface to make logical guest memory accesses which are not possible
anymore in protected mode anyway.

Signed-off-by: Janosch Frank <frankja@linux.ibm.com>
---
 arch/s390/include/asm/kvm_host.h | 11 ++++++-
 arch/s390/kvm/kvm-s390.c         | 49 ++++++++++++++++++++++++++++++++
 arch/s390/kvm/pv.c               |  9 ++++++
 include/uapi/linux/kvm.h         | 11 +++++--
 4 files changed, 77 insertions(+), 3 deletions(-)

diff --git a/arch/s390/include/asm/kvm_host.h b/arch/s390/include/asm/kvm_host.h
index 125511ec6eb0..48f382680755 100644
--- a/arch/s390/include/asm/kvm_host.h
+++ b/arch/s390/include/asm/kvm_host.h
@@ -127,6 +127,12 @@ struct mcck_volatile_info {
 #define CR14_INITIAL_MASK (CR14_UNUSED_32 | CR14_UNUSED_33 | \
 			   CR14_EXTERNAL_DAMAGE_SUBMASK)
 
+#define SIDAD_SIZE_MASK		0xff
+#define sida_origin(sie_block) \
+	(sie_block->sidad & PAGE_MASK)
+#define sida_size(sie_block) \
+	(((sie_block->sidad & SIDAD_SIZE_MASK) + 1) * PAGE_SIZE)
+
 #define CPUSTAT_STOPPED    0x80000000
 #define CPUSTAT_WAIT       0x10000000
 #define CPUSTAT_ECALL_PEND 0x08000000
@@ -315,7 +321,10 @@ struct kvm_s390_sie_block {
 #define CRYCB_FORMAT2 0x00000003
 	__u32	crycbd;			/* 0x00fc */
 	__u64	gcr[16];		/* 0x0100 */
-	__u64	gbea;			/* 0x0180 */
+	union {
+		__u64	gbea;			/* 0x0180 */
+		__u64	sidad;
+	};
 	__u8    reserved188[8];		/* 0x0188 */
 	__u64   sdnxo;			/* 0x0190 */
 	__u8    reserved198[8];		/* 0x0198 */
diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
index 1945180b857a..76303b0f1226 100644
--- a/arch/s390/kvm/kvm-s390.c
+++ b/arch/s390/kvm/kvm-s390.c
@@ -4441,6 +4441,8 @@ static long kvm_s390_guest_mem_op(struct kvm_vcpu *vcpu,
 	const u64 supported_flags = KVM_S390_MEMOP_F_INJECT_EXCEPTION
 				    | KVM_S390_MEMOP_F_CHECK_ONLY;
 
+
+	BUILD_BUG_ON(sizeof(*mop) != 64);
 	if (mop->flags & ~supported_flags || mop->ar >= NUM_ACRS || !mop->size)
 		return -EINVAL;
 
@@ -4522,6 +4524,39 @@ long kvm_arch_vcpu_async_ioctl(struct file *filp,
 }
 
 #ifdef CONFIG_KVM_S390_PROTECTED_VIRTUALIZATION_HOST
+static long kvm_s390_guest_sida_op(struct kvm_vcpu *vcpu,
+				   struct kvm_s390_mem_op *mop)
+{
+	int r = 0;
+	void __user *uaddr = (void __user *)mop->buf;
+
+	if (mop->flags || !mop->size)
+		return -EINVAL;
+
+	if (mop->size > sida_size(vcpu->arch.sie_block))
+		return -E2BIG;
+
+	if (mop->size + mop->offset > sida_size(vcpu->arch.sie_block))
+		return -E2BIG;
+
+	switch (mop->op) {
+	case KVM_S390_MEMOP_SIDA_READ:
+		r = 0;
+		if (copy_to_user(uaddr, (void *)sida_origin(vcpu->arch.sie_block) +
+				 mop->offset, mop->size))
+			r = -EFAULT;
+
+		break;
+	case KVM_S390_MEMOP_SIDA_WRITE:
+		r = 0;
+		if (copy_from_user((void *)vcpu->arch.sie_block->sidad +
+				   mop->offset, uaddr, mop->size))
+			r = -EFAULT;
+		break;
+	}
+	return r;
+}
+
 static int kvm_s390_handle_pv_vcpu(struct kvm_vcpu *vcpu,
 				   struct kvm_pv_cmd *cmd)
 {
@@ -4708,6 +4743,20 @@ long kvm_arch_vcpu_ioctl(struct file *filp,
 		r = kvm_s390_handle_pv_vcpu(vcpu, &args);
 		break;
 	}
+	case KVM_S390_SIDA_OP: {
+		struct kvm_s390_mem_op mem_op;
+
+		if (!kvm_s390_pv_is_protected(vcpu->kvm)) {
+			r = -EINVAL;
+			break;
+		}
+
+		if (copy_from_user(&mem_op, argp, sizeof(mem_op)) == 0)
+			r = kvm_s390_guest_sida_op(vcpu, &mem_op);
+		else
+			r = -EFAULT;
+		break;
+	}
 #endif
 	default:
 		r = -ENOTTY;
diff --git a/arch/s390/kvm/pv.c b/arch/s390/kvm/pv.c
index 24d802072ac7..50e1dc68d972 100644
--- a/arch/s390/kvm/pv.c
+++ b/arch/s390/kvm/pv.c
@@ -93,6 +93,7 @@ int kvm_s390_pv_destroy_cpu(struct kvm_vcpu *vcpu)
 
 	free_pages(vcpu->arch.pv.stor_base,
 		   get_order(uv_info.guest_cpu_stor_len));
+	free_page(sida_origin(vcpu->arch.sie_block));
 	/* Clear cpu and vm handle */
 	memset(&vcpu->arch.sie_block->reserved10, 0,
 	       sizeof(vcpu->arch.sie_block->reserved10));
@@ -124,6 +125,14 @@ int kvm_s390_pv_create_cpu(struct kvm_vcpu *vcpu)
 	uvcb.state_origin = (u64)vcpu->arch.sie_block;
 	uvcb.stor_origin = (u64)vcpu->arch.pv.stor_base;
 
+	/* Alloc Secure Instruction Data Area Designation */
+	vcpu->arch.sie_block->sidad = __get_free_page(GFP_KERNEL | __GFP_ZERO);
+	if (!vcpu->arch.sie_block->sidad) {
+		free_pages(vcpu->arch.pv.stor_base,
+			   get_order(uv_info.guest_cpu_stor_len));
+		return -ENOMEM;
+	}
+
 	rc = uv_call(0, (u64)&uvcb);
 	VCPU_EVENT(vcpu, 3, "PROTVIRT CREATE VCPU: cpu %d handle %llx rc %x rrc %x",
 		   vcpu->vcpu_id, uvcb.cpu_handle, uvcb.header.rc,
diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
index eab741bc12c3..20969ce12096 100644
--- a/include/uapi/linux/kvm.h
+++ b/include/uapi/linux/kvm.h
@@ -466,7 +466,7 @@ struct kvm_translation {
 	__u8  pad[5];
 };
 
-/* for KVM_S390_MEM_OP */
+/* for KVM_S390_MEM_OP and KVM_S390_SIDA_OP */
 struct kvm_s390_mem_op {
 	/* in */
 	__u64 gaddr;		/* the guest address */
@@ -475,11 +475,17 @@ struct kvm_s390_mem_op {
 	__u32 op;		/* type of operation */
 	__u64 buf;		/* buffer in userspace */
 	__u8 ar;		/* the access register number */
-	__u8 reserved[31];	/* should be set to 0 */
+	__u8 reserved21[3];	/* should be set to 0 */
+	__u32 offset;		/* offset into the sida */
+	__u8 reserved28[24];	/* should be set to 0 */
 };
+
+
 /* types for kvm_s390_mem_op->op */
 #define KVM_S390_MEMOP_LOGICAL_READ	0
 #define KVM_S390_MEMOP_LOGICAL_WRITE	1
+#define KVM_S390_MEMOP_SIDA_READ	2
+#define KVM_S390_MEMOP_SIDA_WRITE	3
 /* flags for kvm_s390_mem_op->flags */
 #define KVM_S390_MEMOP_F_CHECK_ONLY		(1ULL << 0)
 #define KVM_S390_MEMOP_F_INJECT_EXCEPTION	(1ULL << 1)
@@ -1510,6 +1516,7 @@ struct kvm_pv_cmd {
 /* Available with KVM_CAP_S390_PROTECTED */
 #define KVM_S390_PV_COMMAND		_IOW(KVMIO, 0xc5, struct kvm_pv_cmd)
 #define KVM_S390_PV_COMMAND_VCPU	_IOW(KVMIO, 0xc6, struct kvm_pv_cmd)
+#define KVM_S390_SIDA_OP		_IOW(KVMIO, 0xc7, struct kvm_s390_mem_op)
 
 /* Secure Encrypted Virtualization command */
 enum sev_cmd_id {
-- 
2.24.0

[RFCv2 22/37] KVM: s390: protvirt: handle secure guest prefix pages

[Christian Borntraeger]

From: Janosch Frank <frankja@linux.ibm.com>

The SPX instruction is handled by the ulravisor. We do get a
notification intercept, though. Let us update our internal view.

In addition to that, when the guest prefix page is not secure, an
intercept 112 (0x70) is indicated.  To avoid this for the most common
cases, we can make the guest prefix page protected whenever we pin it.
We have to deal with 112 nevertheless, e.g. when some host code triggers
an export (e.g. qemu dump guest memory). We can simply re-run the
pinning logic by doing a no-op prefix change.

Signed-off-by: Janosch Frank <frankja@linux.ibm.com>
---
 arch/s390/include/asm/kvm_host.h |  1 +
 arch/s390/kvm/intercept.c        | 15 +++++++++++++++
 arch/s390/kvm/kvm-s390.c         | 14 ++++++++++++++
 3 files changed, 30 insertions(+)

diff --git a/arch/s390/include/asm/kvm_host.h b/arch/s390/include/asm/kvm_host.h
index 48f382680755..686b00ced55b 100644
--- a/arch/s390/include/asm/kvm_host.h
+++ b/arch/s390/include/asm/kvm_host.h
@@ -225,6 +225,7 @@ struct kvm_s390_sie_block {
 #define ICPT_PV_INT_EN	0x64
 #define ICPT_PV_INSTR	0x68
 #define ICPT_PV_NOTIF	0x6c
+#define ICPT_PV_PREF	0x70
 	__u8	icptcode;		/* 0x0050 */
 	__u8	icptstatus;		/* 0x0051 */
 	__u16	ihcpu;			/* 0x0052 */
diff --git a/arch/s390/kvm/intercept.c b/arch/s390/kvm/intercept.c
index d63f9cf10360..ceba0abb1900 100644
--- a/arch/s390/kvm/intercept.c
+++ b/arch/s390/kvm/intercept.c
@@ -451,6 +451,15 @@ static int handle_operexc(struct kvm_vcpu *vcpu)
 	return kvm_s390_inject_program_int(vcpu, PGM_OPERATION);
 }
 
+static int handle_pv_spx(struct kvm_vcpu *vcpu)
+{
+	u32 pref = *(u32 *)vcpu->arch.sie_block->sidad;
+
+	kvm_s390_set_prefix(vcpu, pref);
+	trace_kvm_s390_handle_prefix(vcpu, 1, pref);
+	return 0;
+}
+
 static int handle_pv_sclp(struct kvm_vcpu *vcpu)
 {
 	struct kvm_s390_float_interrupt *fi = &vcpu->kvm->arch.float_int;
@@ -475,6 +484,8 @@ static int handle_pv_sclp(struct kvm_vcpu *vcpu)
 
 static int handle_pv_not(struct kvm_vcpu *vcpu)
 {
+	if (vcpu->arch.sie_block->ipa == 0xb210)
+		return handle_pv_spx(vcpu);
 	if (vcpu->arch.sie_block->ipa == 0xb220)
 		return handle_pv_sclp(vcpu);
 
@@ -533,6 +544,10 @@ int kvm_handle_sie_intercept(struct kvm_vcpu *vcpu)
 	case ICPT_PV_NOTIF:
 		rc = handle_pv_not(vcpu);
 		break;
+	case ICPT_PV_PREF:
+		rc = 0;
+		kvm_s390_set_prefix(vcpu, kvm_s390_get_prefix(vcpu));
+		break;
 	default:
 		return -EOPNOTSUPP;
 	}
diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
index 76303b0f1226..6e74c7afae3a 100644
--- a/arch/s390/kvm/kvm-s390.c
+++ b/arch/s390/kvm/kvm-s390.c
@@ -3675,6 +3675,20 @@ static int kvm_s390_handle_requests(struct kvm_vcpu *vcpu)
 		rc = gmap_mprotect_notify(vcpu->arch.gmap,
 					  kvm_s390_get_prefix(vcpu),
 					  PAGE_SIZE * 2, PROT_WRITE);
+		if (!rc && kvm_s390_pv_is_protected(vcpu->kvm)) {
+			do {
+				rc = uv_convert_to_secure(
+						vcpu->arch.gmap,
+						kvm_s390_get_prefix(vcpu));
+			} while (rc == -EAGAIN);
+			WARN_ONCE(rc, "Error while importing first prefix page. rc %d", rc);
+			do {
+				rc = uv_convert_to_secure(
+						vcpu->arch.gmap,
+						kvm_s390_get_prefix(vcpu) + PAGE_SIZE);
+			} while (rc == -EAGAIN);
+			WARN_ONCE(rc, "Error while importing second prefix page. rc %d", rc);
+		}
 		if (rc) {
 			kvm_make_request(KVM_REQ_MMU_RELOAD, vcpu);
 			return rc;
-- 
2.24.0

[RFCv2 23/37] KVM: s390/mm: handle guest unpin events

[Christian Borntraeger]

From: Claudio Imbrenda <imbrenda@linux.ibm.com>

When a guest makes a shared page unshared we must act accordingly.

Signed-off-by: Claudio Imbrenda <imbrenda@linux.ibm.com>
---
 arch/s390/kvm/intercept.c | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)

diff --git a/arch/s390/kvm/intercept.c b/arch/s390/kvm/intercept.c
index ceba0abb1900..4b1effa44e41 100644
--- a/arch/s390/kvm/intercept.c
+++ b/arch/s390/kvm/intercept.c
@@ -16,6 +16,7 @@
 #include <asm/asm-offsets.h>
 #include <asm/irq.h>
 #include <asm/sysinfo.h>
+#include <asm/uv.h>
 
 #include "kvm-s390.h"
 #include "gaccess.h"
@@ -482,12 +483,35 @@ static int handle_pv_sclp(struct kvm_vcpu *vcpu)
 	return 0;
 }
 
+static int handle_pv_uvc(struct kvm_vcpu *vcpu)
+{
+	struct uv_cb_share *guest_uvcb = (void *)vcpu->arch.sie_block->sidad;
+	struct uv_cb_cts uvcb = {
+		.header.cmd	= UVC_CMD_UNPIN_PAGE_SHARED,
+		.header.len	= sizeof(uvcb),
+		.guest_handle	= kvm_s390_pv_handle(vcpu->kvm),
+		.gaddr		= guest_uvcb->paddr,
+	};
+	int rc;
+
+	if (guest_uvcb->header.cmd != UVC_CMD_REMOVE_SHARED_ACCESS) {
+		WARN_ONCE(1, "Unexpected UVC 0x%x!\n", guest_uvcb->header.cmd);
+		return 0;
+	}
+	rc = uv_make_secure(vcpu->arch.gmap, uvcb.gaddr, &uvcb, 0);
+	if (rc == -EINVAL && uvcb.header.rc == 0x104)
+		return 0;
+	return rc;
+}
+
 static int handle_pv_not(struct kvm_vcpu *vcpu)
 {
 	if (vcpu->arch.sie_block->ipa == 0xb210)
 		return handle_pv_spx(vcpu);
 	if (vcpu->arch.sie_block->ipa == 0xb220)
 		return handle_pv_sclp(vcpu);
+	if (vcpu->arch.sie_block->ipa == 0xb9a4)
+		return handle_pv_uvc(vcpu);
 
 	return handle_instruction(vcpu);
 }
-- 
2.24.0

[RFCv2 24/37] KVM: s390: protvirt: Write sthyi data to instruction data area

[Christian Borntraeger]

From: Janosch Frank <frankja@linux.ibm.com>

STHYI data has to go through the bounce buffer.

Signed-off-by: Janosch Frank <frankja@linux.ibm.com>
---
 arch/s390/kvm/intercept.c | 15 ++++++++++-----
 1 file changed, 10 insertions(+), 5 deletions(-)

diff --git a/arch/s390/kvm/intercept.c b/arch/s390/kvm/intercept.c
index 4b1effa44e41..06d1fa83ef4c 100644
--- a/arch/s390/kvm/intercept.c
+++ b/arch/s390/kvm/intercept.c
@@ -392,7 +392,7 @@ int handle_sthyi(struct kvm_vcpu *vcpu)
 		goto out;
 	}
 
-	if (addr & ~PAGE_MASK)
+	if (!kvm_s390_pv_is_protected(vcpu->kvm) && (addr & ~PAGE_MASK))
 		return kvm_s390_inject_program_int(vcpu, PGM_SPECIFICATION);
 
 	sctns = (void *)get_zeroed_page(GFP_KERNEL);
@@ -403,10 +403,15 @@ int handle_sthyi(struct kvm_vcpu *vcpu)
 
 out:
 	if (!cc) {
-		r = write_guest(vcpu, addr, reg2, sctns, PAGE_SIZE);
-		if (r) {
-			free_page((unsigned long)sctns);
-			return kvm_s390_inject_prog_cond(vcpu, r);
+		if (kvm_s390_pv_is_protected(vcpu->kvm)) {
+			memcpy((void *)(vcpu->arch.sie_block->sidad & PAGE_MASK), sctns,
+			       PAGE_SIZE);
+		} else {
+			r = write_guest(vcpu, addr, reg2, sctns, PAGE_SIZE);
+			if (r) {
+				free_page((unsigned long)sctns);
+				return kvm_s390_inject_prog_cond(vcpu, r);
+			}
 		}
 	}
 
-- 
2.24.0

[RFCv2 25/37] KVM: s390: protvirt: STSI handling

[Christian Borntraeger]

From: Janosch Frank <frankja@linux.ibm.com>

Save response to sidad and disable address checking for protected
guests.

Signed-off-by: Janosch Frank <frankja@linux.ibm.com>
---
 arch/s390/kvm/priv.c | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)

diff --git a/arch/s390/kvm/priv.c b/arch/s390/kvm/priv.c
index ed52ffa8d5d4..06c7e7a10825 100644
--- a/arch/s390/kvm/priv.c
+++ b/arch/s390/kvm/priv.c
@@ -872,7 +872,7 @@ static int handle_stsi(struct kvm_vcpu *vcpu)
 
 	operand2 = kvm_s390_get_base_disp_s(vcpu, &ar);
 
-	if (operand2 & 0xfff)
+	if (!kvm_s390_pv_is_protected(vcpu->kvm) && (operand2 & 0xfff))
 		return kvm_s390_inject_program_int(vcpu, PGM_SPECIFICATION);
 
 	switch (fc) {
@@ -893,8 +893,13 @@ static int handle_stsi(struct kvm_vcpu *vcpu)
 		handle_stsi_3_2_2(vcpu, (void *) mem);
 		break;
 	}
+	if (kvm_s390_pv_is_protected(vcpu->kvm)) {
+		memcpy((void *)vcpu->arch.sie_block->sidad, (void *)mem,
+		       PAGE_SIZE);
+		rc = 0;
+	} else
+		rc = write_guest(vcpu, operand2, ar, (void *)mem, PAGE_SIZE);
 
-	rc = write_guest(vcpu, operand2, ar, (void *)mem, PAGE_SIZE);
 	if (rc) {
 		rc = kvm_s390_inject_prog_cond(vcpu, rc);
 		goto out;
-- 
2.24.0

[RFCv2 26/37] KVM: s390: protvirt: disallow one_reg

[Christian Borntraeger]

From: Janosch Frank <frankja@linux.ibm.com>

A lot of the registers are controlled by the Ultravisor and never
visible to KVM. Some fields in the sie control block are overlayed,
like gbea. As no userspace uses the ONE_REG interface on s390 it is safe
to disable this for protected guests.

Signed-off-by: Janosch Frank <frankja@linux.ibm.com>
---
 arch/s390/kvm/kvm-s390.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
index 6e74c7afae3a..b9692d722c1e 100644
--- a/arch/s390/kvm/kvm-s390.c
+++ b/arch/s390/kvm/kvm-s390.c
@@ -4641,6 +4641,9 @@ long kvm_arch_vcpu_ioctl(struct file *filp,
 	case KVM_SET_ONE_REG:
 	case KVM_GET_ONE_REG: {
 		struct kvm_one_reg reg;
+		r = -EINVAL;
+		if (kvm_s390_pv_is_protected(vcpu->kvm))
+			break;
 		r = -EFAULT;
 		if (copy_from_user(&reg, argp, sizeof(reg)))
 			break;
-- 
2.24.0

[RFCv2 27/37] KVM: s390: protvirt: Only sync fmt4 registers

[Christian Borntraeger]

From: Janosch Frank <frankja@linux.ibm.com>

A lot of the registers are controlled by the Ultravisor and never
visible to KVM. Also some registers are overlayed, like gbea is with
sidad, which might leak data to userspace.

Hence we sync a minimal set of registers for both SIE formats and then
check and sync format 2 registers if necessary.

Also we disable set/get one reg for the same reason. It's an old
interface anyway.

Signed-off-by: Janosch Frank <frankja@linux.ibm.com>
Signed-off-by: Christian Borntraeger <borntraeger@de.ibm.com>
[Fixes and patch splitting]
---
 arch/s390/kvm/kvm-s390.c | 116 ++++++++++++++++++++++++---------------
 1 file changed, 72 insertions(+), 44 deletions(-)

diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
index b9692d722c1e..00a0ce4a3d35 100644
--- a/arch/s390/kvm/kvm-s390.c
+++ b/arch/s390/kvm/kvm-s390.c
@@ -3444,9 +3444,11 @@ static void kvm_arch_vcpu_ioctl_initial_reset(struct kvm_vcpu *vcpu)
 	vcpu->arch.sie_block->gcr[0] = CR0_INITIAL_MASK;
 	vcpu->arch.sie_block->gcr[14] = CR14_INITIAL_MASK;
 	vcpu->run->s.regs.fpc = 0;
-	vcpu->arch.sie_block->gbea = 1;
-	vcpu->arch.sie_block->pp = 0;
-	vcpu->arch.sie_block->fpf &= ~FPF_BPBC;
+	if (!kvm_s390_pv_handle_cpu(vcpu)) {
+		vcpu->arch.sie_block->gbea = 1;
+		vcpu->arch.sie_block->pp = 0;
+		vcpu->arch.sie_block->fpf &= ~FPF_BPBC;
+	}
 }
 
 static void kvm_arch_vcpu_ioctl_clear_reset(struct kvm_vcpu *vcpu)
@@ -4057,25 +4059,16 @@ static int __vcpu_run(struct kvm_vcpu *vcpu)
 	return rc;
 }
 
-static void sync_regs(struct kvm_vcpu *vcpu, struct kvm_run *kvm_run)
+static void sync_regs_fmt2(struct kvm_vcpu *vcpu, struct kvm_run *kvm_run)
 {
 	struct runtime_instr_cb *riccb;
 	struct gs_cb *gscb;
 
-	riccb = (struct runtime_instr_cb *) &kvm_run->s.regs.riccb;
-	gscb = (struct gs_cb *) &kvm_run->s.regs.gscb;
 	vcpu->arch.sie_block->gpsw.mask = kvm_run->psw_mask;
 	vcpu->arch.sie_block->gpsw.addr = kvm_run->psw_addr;
-	if (kvm_run->kvm_dirty_regs & KVM_SYNC_PREFIX)
-		kvm_s390_set_prefix(vcpu, kvm_run->s.regs.prefix);
-	if (kvm_run->kvm_dirty_regs & KVM_SYNC_CRS) {
-		memcpy(&vcpu->arch.sie_block->gcr, &kvm_run->s.regs.crs, 128);
-		/* some control register changes require a tlb flush */
-		kvm_make_request(KVM_REQ_TLB_FLUSH, vcpu);
-	}
+	riccb = (struct runtime_instr_cb *) &kvm_run->s.regs.riccb;
+	gscb = (struct gs_cb *) &kvm_run->s.regs.gscb;
 	if (kvm_run->kvm_dirty_regs & KVM_SYNC_ARCH0) {
-		kvm_s390_set_cpu_timer(vcpu, kvm_run->s.regs.cputm);
-		vcpu->arch.sie_block->ckc = kvm_run->s.regs.ckc;
 		vcpu->arch.sie_block->todpr = kvm_run->s.regs.todpr;
 		vcpu->arch.sie_block->pp = kvm_run->s.regs.pp;
 		vcpu->arch.sie_block->gbea = kvm_run->s.regs.gbea;
@@ -4116,6 +4109,47 @@ static void sync_regs(struct kvm_vcpu *vcpu, struct kvm_run *kvm_run)
 		vcpu->arch.sie_block->fpf &= ~FPF_BPBC;
 		vcpu->arch.sie_block->fpf |= kvm_run->s.regs.bpbc ? FPF_BPBC : 0;
 	}
+	if (MACHINE_HAS_GS) {
+		preempt_disable();
+		__ctl_set_bit(2, 4);
+		if (current->thread.gs_cb) {
+			vcpu->arch.host_gscb = current->thread.gs_cb;
+			save_gs_cb(vcpu->arch.host_gscb);
+		}
+		if (vcpu->arch.gs_enabled) {
+			current->thread.gs_cb = (struct gs_cb *)
+						&vcpu->run->s.regs.gscb;
+			restore_gs_cb(current->thread.gs_cb);
+		}
+		preempt_enable();
+	}
+	/* SIE will load etoken directly from SDNX and therefore kvm_run */
+}
+
+static void sync_regs(struct kvm_vcpu *vcpu, struct kvm_run *kvm_run)
+{
+	/*
+	 * at several places we have to modify our internal view to not do
+	 * things that are disallowed by the ultravisor. For example we must
+	 * not inject interrupts after specific exits (e.g. 112). We do this
+	 * by turning off the MIE bits of our PSW copy. To avoid getting
+	 * validity intercepts, we do only accept the condition code from
+	 * userspace.
+	 */
+	vcpu->arch.sie_block->gpsw.mask &= ~PSW_MASK_CC;
+	vcpu->arch.sie_block->gpsw.mask |= kvm_run->psw_mask & PSW_MASK_CC;
+
+	if (kvm_run->kvm_dirty_regs & KVM_SYNC_PREFIX)
+		kvm_s390_set_prefix(vcpu, kvm_run->s.regs.prefix);
+	if (kvm_run->kvm_dirty_regs & KVM_SYNC_CRS) {
+		memcpy(&vcpu->arch.sie_block->gcr, &kvm_run->s.regs.crs, 128);
+		/* some control register changes require a tlb flush */
+		kvm_make_request(KVM_REQ_TLB_FLUSH, vcpu);
+	}
+	if (kvm_run->kvm_dirty_regs & KVM_SYNC_ARCH0) {
+		kvm_s390_set_cpu_timer(vcpu, kvm_run->s.regs.cputm);
+		vcpu->arch.sie_block->ckc = kvm_run->s.regs.ckc;
+	}
 	save_access_regs(vcpu->arch.host_acrs);
 	restore_access_regs(vcpu->run->s.regs.acrs);
 	/* save host (userspace) fprs/vrs */
@@ -4130,23 +4164,31 @@ static void sync_regs(struct kvm_vcpu *vcpu, struct kvm_run *kvm_run)
 	if (test_fp_ctl(current->thread.fpu.fpc))
 		/* User space provided an invalid FPC, let's clear it */
 		current->thread.fpu.fpc = 0;
+
+	/* Sync fmt2 only data */
+	if (likely(!kvm_s390_pv_is_protected(vcpu->kvm)))
+		sync_regs_fmt2(vcpu, kvm_run);
+	kvm_run->kvm_dirty_regs = 0;
+}
+
+static void store_regs_fmt2(struct kvm_vcpu *vcpu, struct kvm_run *kvm_run)
+{
+	kvm_run->s.regs.pp = vcpu->arch.sie_block->pp;
+	kvm_run->s.regs.gbea = vcpu->arch.sie_block->gbea;
+	kvm_run->s.regs.bpbc = (vcpu->arch.sie_block->fpf & FPF_BPBC) == FPF_BPBC;
 	if (MACHINE_HAS_GS) {
-		preempt_disable();
 		__ctl_set_bit(2, 4);
-		if (current->thread.gs_cb) {
-			vcpu->arch.host_gscb = current->thread.gs_cb;
-			save_gs_cb(vcpu->arch.host_gscb);
-		}
-		if (vcpu->arch.gs_enabled) {
-			current->thread.gs_cb = (struct gs_cb *)
-						&vcpu->run->s.regs.gscb;
-			restore_gs_cb(current->thread.gs_cb);
-		}
+		if (vcpu->arch.gs_enabled)
+			save_gs_cb(current->thread.gs_cb);
+		preempt_disable();
+		current->thread.gs_cb = vcpu->arch.host_gscb;
+		restore_gs_cb(vcpu->arch.host_gscb);
 		preempt_enable();
+		if (!vcpu->arch.host_gscb)
+			__ctl_clear_bit(2, 4);
+		vcpu->arch.host_gscb = NULL;
 	}
-	/* SIE will load etoken directly from SDNX and therefore kvm_run */
-
-	kvm_run->kvm_dirty_regs = 0;
+	/* SIE will save etoken directly into SDNX and therefore kvm_run */
 }
 
 static void store_regs(struct kvm_vcpu *vcpu, struct kvm_run *kvm_run)
@@ -4158,12 +4200,9 @@ static void store_regs(struct kvm_vcpu *vcpu, struct kvm_run *kvm_run)
 	kvm_run->s.regs.cputm = kvm_s390_get_cpu_timer(vcpu);
 	kvm_run->s.regs.ckc = vcpu->arch.sie_block->ckc;
 	kvm_run->s.regs.todpr = vcpu->arch.sie_block->todpr;
-	kvm_run->s.regs.pp = vcpu->arch.sie_block->pp;
-	kvm_run->s.regs.gbea = vcpu->arch.sie_block->gbea;
 	kvm_run->s.regs.pft = vcpu->arch.pfault_token;
 	kvm_run->s.regs.pfs = vcpu->arch.pfault_select;
 	kvm_run->s.regs.pfc = vcpu->arch.pfault_compare;
-	kvm_run->s.regs.bpbc = (vcpu->arch.sie_block->fpf & FPF_BPBC) == FPF_BPBC;
 	save_access_regs(vcpu->run->s.regs.acrs);
 	restore_access_regs(vcpu->arch.host_acrs);
 	/* Save guest register state */
@@ -4172,19 +4211,8 @@ static void store_regs(struct kvm_vcpu *vcpu, struct kvm_run *kvm_run)
 	/* Restore will be done lazily at return */
 	current->thread.fpu.fpc = vcpu->arch.host_fpregs.fpc;
 	current->thread.fpu.regs = vcpu->arch.host_fpregs.regs;
-	if (MACHINE_HAS_GS) {
-		__ctl_set_bit(2, 4);
-		if (vcpu->arch.gs_enabled)
-			save_gs_cb(current->thread.gs_cb);
-		preempt_disable();
-		current->thread.gs_cb = vcpu->arch.host_gscb;
-		restore_gs_cb(vcpu->arch.host_gscb);
-		preempt_enable();
-		if (!vcpu->arch.host_gscb)
-			__ctl_clear_bit(2, 4);
-		vcpu->arch.host_gscb = NULL;
-	}
-	/* SIE will save etoken directly into SDNX and therefore kvm_run */
+	if (likely(!kvm_s390_pv_is_protected(vcpu->kvm)))
+		store_regs_fmt2(vcpu, kvm_run);
 }
 
 int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu, struct kvm_run *kvm_run)
-- 
2.24.0

[RFCv2 28/37] KVM: s390: protvirt: Add program exception injection

[Christian Borntraeger]

From: Janosch Frank <frankja@linux.ibm.com>

Only two program exceptions can be injected for a protected guest:
specification and operand

Both have a code in offset 248 of the state description, as the lowcore
is not accessible by KVM for such guests.

Signed-off-by: Janosch Frank <frankja@linux.ibm.com>
---
 arch/s390/kvm/interrupt.c | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)

diff --git a/arch/s390/kvm/interrupt.c b/arch/s390/kvm/interrupt.c
index a98f1dfde8de..b9c6666818a6 100644
--- a/arch/s390/kvm/interrupt.c
+++ b/arch/s390/kvm/interrupt.c
@@ -829,6 +829,21 @@ static int __must_check __deliver_external_call(struct kvm_vcpu *vcpu)
 	return rc ? -EFAULT : 0;
 }
 
+static int __deliver_prog_pv(struct kvm_vcpu *vcpu, u16 code)
+{
+	switch (code) {
+	case PGM_SPECIFICATION:
+		vcpu->arch.sie_block->iictl = IICTL_CODE_SPECIFICATION;
+		break;
+	case PGM_OPERAND:
+		vcpu->arch.sie_block->iictl = IICTL_CODE_OPERAND;
+		break;
+	default:
+		return -EINVAL;
+	}
+	return 0;
+}
+
 static int __must_check __deliver_prog(struct kvm_vcpu *vcpu)
 {
 	struct kvm_s390_local_interrupt *li = &vcpu->arch.local_int;
@@ -849,6 +864,9 @@ static int __must_check __deliver_prog(struct kvm_vcpu *vcpu)
 	trace_kvm_s390_deliver_interrupt(vcpu->vcpu_id, KVM_S390_PROGRAM_INT,
 					 pgm_info.code, 0);
 
+	if (kvm_s390_pv_is_protected(vcpu->kvm))
+		return __deliver_prog_pv(vcpu, pgm_info.code & ~PGM_PER);
+
 	switch (pgm_info.code & ~PGM_PER) {
 	case PGM_AFX_TRANSLATION:
 	case PGM_ASX_TRANSLATION:
-- 
2.24.0

[RFCv2 29/37] DOCUMENTATION: protvirt: Diag 308 IPL

[Christian Borntraeger]

From: Janosch Frank <frankja@linux.ibm.com>

Description of changes that are necessary to move a KVM VM into
Protected Virtualization mode.

Signed-off-by: Janosch Frank <frankja@linux.ibm.com>
---
 Documentation/virt/kvm/s390-pv-boot.rst | 64 +++++++++++++++++++++++++
 1 file changed, 64 insertions(+)
 create mode 100644 Documentation/virt/kvm/s390-pv-boot.rst

diff --git a/Documentation/virt/kvm/s390-pv-boot.rst b/Documentation/virt/kvm/s390-pv-boot.rst
new file mode 100644
index 000000000000..431cd5d7f686
--- /dev/null
+++ b/Documentation/virt/kvm/s390-pv-boot.rst
@@ -0,0 +1,64 @@
+.. SPDX-License-Identifier: GPL-2.0
+=========================
+Boot/IPL of Protected VMs
+=========================
+
+Summary
+-------
+Protected VMs are encrypted while not running. On IPL a small
+plaintext bootloader is started which provides information about the
+encrypted components and necessary metadata to KVM to decrypt it.
+
+Based on this data, KVM will make the PV known to the Ultravisor and
+instruct it to secure its memory, decrypt the components and verify
+the data and address list hashes, to ensure integrity. Afterwards KVM
+can run the PV via SIE which the UV will intercept and execute on
+KVM's behalf.
+
+The switch into PV mode lets us load encrypted guest executables and
+data via every available method (network, dasd, scsi, direct kernel,
+...) without the need to change the boot process.
+
+
+Diag308
+-------
+This diagnose instruction is the basis for VM IPL. The VM can set and
+retrieve IPL information blocks, that specify the IPL method/devices
+and request VM memory and subsystem resets, as well as IPLs.
+
+For PVs this concept has been continued with new subcodes:
+
+Subcode 8: Set an IPL Information Block of type 5.
+Subcode 9: Store the saved block in guest memory
+Subcode 10: Move into Protected Virtualization mode
+
+The new PV load-device-specific-parameters field specifies all data,
+that is necessary to move into PV mode.
+
+* PV Header origin
+* PV Header length
+* List of Components composed of
+   * AES-XTS Tweak prefix
+   * Origin
+   * Size
+
+The PV header contains the keys and hashes, which the UV will use to
+decrypt and verify the PV, as well as control flags and a start PSW.
+
+The components are for instance an encrypted kernel, kernel cmd and
+initrd. The components are decrypted by the UV.
+
+All non-decrypted data of the non-PV guest instance are zero on first
+access of the PV.
+
+
+When running in a protected mode some subcodes will result in
+exceptions or return error codes.
+
+Subcodes 4 and 7 will result in specification exceptions.
+When removing a secure VM, the UV will clear all memory, so we can't
+have non-clearing IPL subcodes.
+
+Subcodes 8, 9, 10 will result in specification exceptions.
+Re-IPL into a protected mode is only possible via a detour into non
+protected mode.
-- 
2.24.0

[RFCv2 30/37] KVM: s390: protvirt: Add diag 308 subcode 8 - 10 handling

[Christian Borntraeger]

From: Janosch Frank <frankja@linux.ibm.com>

If the host initialized the Ultravisor, we can set stfle bit 161
(protected virtual IPL enhancements facility), which indicates, that
the IPL subcodes 8, 9 and are valid. These subcodes are used by a
normal guest to set/retrieve a IPIB of type 5 and transition into
protected mode.

Once in protected mode, the Ultravisor will conceal the facility
bit. Therefore each boot into protected mode has to go through
non-protected. There is no secure re-ipl with subcode 10 without a
previous subcode 3.

In protected mode, there is no subcode 4 available, as the VM has no
more access to its memory from non-protected mode. I.e. each IPL
clears.

Signed-off-by: Janosch Frank <frankja@linux.ibm.com>
---
 arch/s390/kvm/diag.c     | 6 ++++++
 arch/s390/kvm/kvm-s390.c | 5 +++++
 2 files changed, 11 insertions(+)

diff --git a/arch/s390/kvm/diag.c b/arch/s390/kvm/diag.c
index 3fb54ec2cf3e..b951dbdcb6a0 100644
--- a/arch/s390/kvm/diag.c
+++ b/arch/s390/kvm/diag.c
@@ -197,6 +197,12 @@ static int __diag_ipl_functions(struct kvm_vcpu *vcpu)
 	case 4:
 		vcpu->run->s390_reset_flags = 0;
 		break;
+	case 8:
+	case 9:
+	case 10:
+		if (!test_kvm_facility(vcpu->kvm, 161))
+			return kvm_s390_inject_program_int(vcpu, PGM_SPECIFICATION);
+		/* fall through */
 	default:
 		return -EOPNOTSUPP;
 	}
diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
index 00a0ce4a3d35..d786b34be244 100644
--- a/arch/s390/kvm/kvm-s390.c
+++ b/arch/s390/kvm/kvm-s390.c
@@ -2606,6 +2606,11 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type)
 	if (css_general_characteristics.aiv && test_facility(65))
 		set_kvm_facility(kvm->arch.model.fac_mask, 65);
 
+	if (is_prot_virt_host()) {
+		set_kvm_facility(kvm->arch.model.fac_mask, 161);
+		set_kvm_facility(kvm->arch.model.fac_list, 161);
+	}
+
 	kvm->arch.model.cpuid = kvm_s390_get_initial_cpuid();
 	kvm->arch.model.ibc = sclp.ibc & 0x0fff;
 
-- 
2.24.0

[RFCv2 31/37] KVM: s390: protvirt: UV calls diag308 0, 1

[Christian Borntraeger]

From: Janosch Frank <frankja@linux.ibm.com>

diag 308 subcode 0 and 1 require KVM and Ultravisor interaction, since
the cpus have to be set into multiple reset states.

* All cpus need to be stopped
* The "unshare all" UVC needs to be executed
* The "perform reset" UVC needs to be executed
* The cpus need to be reset via the "set cpu state" UVC
* The issuing cpu needs to set state 5 via "set cpu state"

Signed-off-by: Janosch Frank <frankja@linux.ibm.com>
---
 arch/s390/include/asm/uv.h | 25 +++++++++++++++++++++++++
 arch/s390/kvm/diag.c       |  1 +
 arch/s390/kvm/kvm-s390.c   | 28 ++++++++++++++++++++++++++++
 arch/s390/kvm/kvm-s390.h   |  2 ++
 arch/s390/kvm/pv.c         | 19 +++++++++++++++++++
 include/uapi/linux/kvm.h   |  2 ++
 6 files changed, 77 insertions(+)

diff --git a/arch/s390/include/asm/uv.h b/arch/s390/include/asm/uv.h
index 315ec15efe44..584ad82740b6 100644
--- a/arch/s390/include/asm/uv.h
+++ b/arch/s390/include/asm/uv.h
@@ -36,6 +36,12 @@
 #define UVC_CMD_SET_SEC_CONF_PARAMS	0x0300
 #define UVC_CMD_UNPACK_IMG		0x0301
 #define UVC_CMD_VERIFY_IMG		0x0302
+#define UVC_CMD_CPU_RESET		0x0310
+#define UVC_CMD_CPU_RESET_INITIAL	0x0311
+#define UVC_CMD_PERF_CONF_CLEAR_RESET	0x0320
+#define UVC_CMD_CPU_RESET_CLEAR		0x0321
+#define UVC_CMD_CPU_SET_STATE		0x0330
+#define UVC_CMD_SET_UNSHARED_ALL	0x0340
 #define UVC_CMD_PIN_PAGE_SHARED		0x0341
 #define UVC_CMD_UNPIN_PAGE_SHARED	0x0342
 #define UVC_CMD_SET_SHARED_ACCESS	0x1000
@@ -56,6 +62,12 @@ enum uv_cmds_inst {
 	BIT_UVC_CMD_SET_SEC_PARMS = 11,
 	BIT_UVC_CMD_UNPACK_IMG = 13,
 	BIT_UVC_CMD_VERIFY_IMG = 14,
+	BIT_UVC_CMD_CPU_RESET = 15,
+	BIT_UVC_CMD_CPU_RESET_INITIAL = 16,
+	BIT_UVC_CMD_CPU_SET_STATE = 17,
+	BIT_UVC_CMD_PREPARE_CLEAR_RESET = 18,
+	BIT_UVC_CMD_CPU_PERFORM_CLEAR_RESET = 19,
+	BIT_UVC_CMD_REMOVE_SHARED_ACCES = 20,
 	BIT_UVC_CMD_PIN_PAGE_SHARED = 21,
 	BIT_UVC_CMD_UNPIN_PAGE_SHARED = 22,
 };
@@ -153,6 +165,19 @@ struct uv_cb_unp {
 	u64 reserved28[3];
 } __packed __aligned(8);
 
+#define PV_CPU_STATE_OPR	1
+#define PV_CPU_STATE_STP	2
+#define PV_CPU_STATE_CHKSTP	3
+
+struct uv_cb_cpu_set_state {
+	struct uv_cb_header header;
+	u64 reserved08[2];
+	u64 cpu_handle;
+	u8  reserved20[7];
+	u8  state;
+	u64 reserved28[5];
+};
+
 /*
  * A common UV call struct for the following calls:
  * Destroy cpu/config
diff --git a/arch/s390/kvm/diag.c b/arch/s390/kvm/diag.c
index b951dbdcb6a0..1c53eb7ba152 100644
--- a/arch/s390/kvm/diag.c
+++ b/arch/s390/kvm/diag.c
@@ -13,6 +13,7 @@
 #include <asm/pgalloc.h>
 #include <asm/gmap.h>
 #include <asm/virtio-ccw.h>
+#include <asm/uv.h>
 #include "kvm-s390.h"
 #include "trace.h"
 #include "trace-s390.h"
diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
index d786b34be244..3ab5091ded6c 100644
--- a/arch/s390/kvm/kvm-s390.c
+++ b/arch/s390/kvm/kvm-s390.c
@@ -2272,6 +2272,34 @@ static int kvm_s390_handle_pv(struct kvm *kvm, struct kvm_pv_cmd *cmd)
 			 ret >> 16, ret & 0x0000ffff);
 		break;
 	}
+	case KVM_PV_VM_PERF_CLEAR_RESET: {
+		u32 ret;
+
+		r = -EINVAL;
+		if (!kvm_s390_pv_is_protected(kvm))
+			break;
+
+		r = uv_cmd_nodata(kvm_s390_pv_handle(kvm),
+				  UVC_CMD_PERF_CONF_CLEAR_RESET,
+				  &ret);
+		VM_EVENT(kvm, 3, "PROTVIRT PERF CLEAR: rc %x rrc %x",
+			 ret >> 16, ret & 0x0000ffff);
+		break;
+	}
+	case KVM_PV_VM_UNSHARE: {
+		u32 ret;
+
+		r = -EINVAL;
+		if (!kvm_s390_pv_is_protected(kvm))
+			break;
+
+		r = uv_cmd_nodata(kvm_s390_pv_handle(kvm),
+				  UVC_CMD_SET_UNSHARED_ALL,
+				  &ret);
+		VM_EVENT(kvm, 3, "PROTVIRT UNSHARE: %d rc %x rrc %x",
+			 r, ret >> 16, ret & 0x0000ffff);
+		break;
+	}
 	default:
 		return -ENOTTY;
 	}
diff --git a/arch/s390/kvm/kvm-s390.h b/arch/s390/kvm/kvm-s390.h
index d77da15bebe5..d890a4336690 100644
--- a/arch/s390/kvm/kvm-s390.h
+++ b/arch/s390/kvm/kvm-s390.h
@@ -208,6 +208,7 @@ int kvm_s390_pv_set_sec_parms(struct kvm *kvm, void *hdr, u64 length);
 int kvm_s390_pv_unpack(struct kvm *kvm, unsigned long addr, unsigned long size,
 		       unsigned long tweak);
 int kvm_s390_pv_verify(struct kvm *kvm);
+int kvm_s390_pv_set_cpu_state(struct kvm_vcpu *vcpu, u8 state);
 
 static inline bool kvm_s390_pv_is_protected(struct kvm *kvm)
 {
@@ -236,6 +237,7 @@ static inline int kvm_s390_pv_unpack(struct kvm *kvm, unsigned long addr,
 				     unsigned long size,  unsigned long tweak)
 { return 0; }
 static inline int kvm_s390_pv_verify(struct kvm *kvm) { return 0; }
+static inline int kvm_s390_pv_set_cpu_state(struct kvm_vcpu *vcpu, u8 state) { return 0; }
 static inline bool kvm_s390_pv_is_protected(struct kvm *kvm) { return 0; }
 static inline u64 kvm_s390_pv_handle(struct kvm *kvm) { return 0; }
 static inline u64 kvm_s390_pv_handle_cpu(struct kvm_vcpu *vcpu) { return 0; }
diff --git a/arch/s390/kvm/pv.c b/arch/s390/kvm/pv.c
index 50e1dc68d972..dade21e10bcc 100644
--- a/arch/s390/kvm/pv.c
+++ b/arch/s390/kvm/pv.c
@@ -255,3 +255,22 @@ int kvm_s390_pv_unpack(struct kvm *kvm, unsigned long addr, unsigned long size,
 	VM_EVENT(kvm, 3, "PROTVIRT VM UNPACK: finished rc %x", rc);
 	return rc;
 }
+
+int kvm_s390_pv_set_cpu_state(struct kvm_vcpu *vcpu, u8 state)
+{
+	int rc;
+	struct uv_cb_cpu_set_state uvcb = {
+		.header.cmd	= UVC_CMD_CPU_SET_STATE,
+		.header.len	= sizeof(uvcb),
+		.cpu_handle	= kvm_s390_pv_handle_cpu(vcpu),
+		.state		= state,
+	};
+
+	if (!kvm_s390_pv_handle_cpu(vcpu))
+		return -EINVAL;
+
+	rc = uv_call(0, (u64)&uvcb);
+	if (rc)
+		return -EINVAL;
+	return 0;
+}
diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
index 20969ce12096..8b77e92a6422 100644
--- a/include/uapi/linux/kvm.h
+++ b/include/uapi/linux/kvm.h
@@ -1502,6 +1502,8 @@ enum pv_cmd_id {
 	KVM_PV_VM_SET_SEC_PARMS,
 	KVM_PV_VM_UNPACK,
 	KVM_PV_VM_VERIFY,
+	KVM_PV_VM_PERF_CLEAR_RESET,
+	KVM_PV_VM_UNSHARE,
 	KVM_PV_VCPU_CREATE,
 	KVM_PV_VCPU_DESTROY,
 };
-- 
2.24.0

[RFCv2 32/37] KVM: s390: protvirt: Report CPU state to Ultravisor

[Christian Borntraeger]

From: Janosch Frank <frankja@linux.ibm.com>

VCPU states have to be reported to the ultravisor for SIGP
interpretation.

Signed-off-by: Janosch Frank <frankja@linux.ibm.com>
---
 arch/s390/kvm/kvm-s390.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
index 3ab5091ded6c..da8aeb24362a 100644
--- a/arch/s390/kvm/kvm-s390.c
+++ b/arch/s390/kvm/kvm-s390.c
@@ -4434,7 +4434,8 @@ void kvm_s390_vcpu_start(struct kvm_vcpu *vcpu)
 		 */
 		__disable_ibs_on_all_vcpus(vcpu->kvm);
 	}
-
+	/* Let's tell the UV that we want to start again */
+	kvm_s390_pv_set_cpu_state(vcpu, PV_CPU_STATE_OPR);
 	kvm_s390_clear_cpuflags(vcpu, CPUSTAT_STOPPED);
 	/*
 	 * Another VCPU might have used IBS while we were offline.
@@ -4462,6 +4463,8 @@ void kvm_s390_vcpu_stop(struct kvm_vcpu *vcpu)
 	kvm_s390_clear_stop_irq(vcpu);
 
 	kvm_s390_set_cpuflags(vcpu, CPUSTAT_STOPPED);
+	/* Let's tell the UV that we successfully stopped the vcpu */
+	kvm_s390_pv_set_cpu_state(vcpu, PV_CPU_STATE_STP);
 	__disable_ibs_on_vcpu(vcpu);
 
 	for (i = 0; i < online_vcpus; i++) {
-- 
2.24.0

[RFCv2 33/37] KVM: s390: protvirt: Support cmd 5 operation state

[Christian Borntraeger]

From: Janosch Frank <frankja@linux.ibm.com>

Code 5 for the set cpu state UV call tells the UV to load a PSW from
the SE header (first IPL) or from guest location 0x0 (diag 308 subcode
0/1). Also it sets the cpu into operating state afterwards, so we can
start it.

Signed-off-by: Janosch Frank <frankja@linux.ibm.com>
---
 arch/s390/include/asm/uv.h | 1 +
 arch/s390/kvm/kvm-s390.c   | 7 +++++++
 include/uapi/linux/kvm.h   | 1 +
 3 files changed, 9 insertions(+)

diff --git a/arch/s390/include/asm/uv.h b/arch/s390/include/asm/uv.h
index 584ad82740b6..7f5606f39d36 100644
--- a/arch/s390/include/asm/uv.h
+++ b/arch/s390/include/asm/uv.h
@@ -168,6 +168,7 @@ struct uv_cb_unp {
 #define PV_CPU_STATE_OPR	1
 #define PV_CPU_STATE_STP	2
 #define PV_CPU_STATE_CHKSTP	3
+#define PV_CPU_STATE_OPR_LOAD	5
 
 struct uv_cb_cpu_set_state {
 	struct uv_cb_header header;
diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
index da8aeb24362a..2beb93f0572f 100644
--- a/arch/s390/kvm/kvm-s390.c
+++ b/arch/s390/kvm/kvm-s390.c
@@ -4658,6 +4658,13 @@ static int kvm_s390_handle_pv_vcpu(struct kvm_vcpu *vcpu,
 		r = kvm_s390_pv_destroy_cpu(vcpu);
 		break;
 	}
+	case KVM_PV_VCPU_SET_IPL_PSW: {
+		if (!kvm_s390_pv_handle_cpu(vcpu))
+			return -EINVAL;
+
+		r = kvm_s390_pv_set_cpu_state(vcpu, PV_CPU_STATE_OPR_LOAD);
+		break;
+	}
 	default:
 		r = -ENOTTY;
 	}
diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
index 8b77e92a6422..69831f40519a 100644
--- a/include/uapi/linux/kvm.h
+++ b/include/uapi/linux/kvm.h
@@ -1506,6 +1506,7 @@ enum pv_cmd_id {
 	KVM_PV_VM_UNSHARE,
 	KVM_PV_VCPU_CREATE,
 	KVM_PV_VCPU_DESTROY,
+	KVM_PV_VCPU_SET_IPL_PSW,
 };
 
 struct kvm_pv_cmd {
-- 
2.24.0

[RFCv2 34/37] KVM: s390: protvirt: Add UV debug trace

[Christian Borntraeger]

From: Janosch Frank <frankja@linux.ibm.com>

Let's have some debug traces which stay around for longer than the
guest.

Signed-off-by: Janosch Frank <frankja@linux.ibm.com>
---
 arch/s390/kvm/kvm-s390.c | 10 +++++++++-
 arch/s390/kvm/kvm-s390.h |  9 +++++++++
 arch/s390/kvm/pv.c       | 21 +++++++++++++++++++--
 3 files changed, 37 insertions(+), 3 deletions(-)

diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
index 2beb93f0572f..d4dc156e2c3e 100644
--- a/arch/s390/kvm/kvm-s390.c
+++ b/arch/s390/kvm/kvm-s390.c
@@ -221,6 +221,7 @@ static struct kvm_s390_vm_cpu_subfunc kvm_s390_available_subfunc;
 static struct gmap_notifier gmap_notifier;
 static struct gmap_notifier vsie_gmap_notifier;
 debug_info_t *kvm_s390_dbf;
+debug_info_t *kvm_s390_dbf_uv;
 
 /* Section: not file related */
 int kvm_arch_hardware_enable(void)
@@ -462,7 +463,13 @@ int kvm_arch_init(void *opaque)
 	if (!kvm_s390_dbf)
 		return -ENOMEM;
 
-	if (debug_register_view(kvm_s390_dbf, &debug_sprintf_view))
+	kvm_s390_dbf_uv = debug_register("kvm-uv", 32, 1, 7 * sizeof(long));
+	if (!kvm_s390_dbf_uv)
+		return -ENOMEM;
+
+
+	if (debug_register_view(kvm_s390_dbf, &debug_sprintf_view) ||
+	    debug_register_view(kvm_s390_dbf_uv, &debug_sprintf_view))
 		goto out;
 
 	kvm_s390_cpu_feat_init();
@@ -489,6 +496,7 @@ void kvm_arch_exit(void)
 {
 	kvm_s390_gib_destroy();
 	debug_unregister(kvm_s390_dbf);
+	debug_unregister(kvm_s390_dbf_uv);
 }
 
 /* Section: device related */
diff --git a/arch/s390/kvm/kvm-s390.h b/arch/s390/kvm/kvm-s390.h
index d890a4336690..89221be1b379 100644
--- a/arch/s390/kvm/kvm-s390.h
+++ b/arch/s390/kvm/kvm-s390.h
@@ -25,6 +25,15 @@
 #define IS_ITDB_VALID(vcpu)	((*(char *)vcpu->arch.sie_block->itdba == TDB_FORMAT1))
 
 extern debug_info_t *kvm_s390_dbf;
+extern debug_info_t *kvm_s390_dbf_uv;
+
+#define KVM_UV_EVENT(d_kvm, d_loglevel, d_string, d_args...)\
+do { \
+	debug_sprintf_event(kvm_s390_dbf_uv, d_loglevel, \
+			    "%s: " d_string "\n", d_kvm->arch.dbf->name, \
+			    d_args); \
+} while (0)
+
 #define KVM_EVENT(d_loglevel, d_string, d_args...)\
 do { \
 	debug_sprintf_event(kvm_s390_dbf, d_loglevel, d_string "\n", \
diff --git a/arch/s390/kvm/pv.c b/arch/s390/kvm/pv.c
index dade21e10bcc..19a79c56a0a3 100644
--- a/arch/s390/kvm/pv.c
+++ b/arch/s390/kvm/pv.c
@@ -74,6 +74,8 @@ int kvm_s390_pv_destroy_vm(struct kvm *kvm)
 	atomic_set(&kvm->mm->context.is_se, 0);
 	VM_EVENT(kvm, 3, "PROTVIRT DESTROY VM: rc %x rrc %x",
 		 ret >> 16, ret & 0x0000ffff);
+	KVM_UV_EVENT(kvm, 3, "PROTVIRT DESTROY VM: rc %x rrc %x",
+		 ret >> 16, ret & 0x0000ffff);
 	return rc;
 }
 
@@ -89,6 +91,8 @@ int kvm_s390_pv_destroy_cpu(struct kvm_vcpu *vcpu)
 
 		VCPU_EVENT(vcpu, 3, "PROTVIRT DESTROY VCPU: cpu %d rc %x rrc %x",
 			   vcpu->vcpu_id, ret >> 16, ret & 0x0000ffff);
+		KVM_UV_EVENT(vcpu->kvm, 3, "PROTVIRT DESTROY VCPU: cpu %d rc %x rrc %x",
+			     vcpu->vcpu_id, ret >> 16, ret & 0x0000ffff);
 	}
 
 	free_pages(vcpu->arch.pv.stor_base,
@@ -137,6 +141,10 @@ int kvm_s390_pv_create_cpu(struct kvm_vcpu *vcpu)
 	VCPU_EVENT(vcpu, 3, "PROTVIRT CREATE VCPU: cpu %d handle %llx rc %x rrc %x",
 		   vcpu->vcpu_id, uvcb.cpu_handle, uvcb.header.rc,
 		   uvcb.header.rrc);
+	KVM_UV_EVENT(vcpu->kvm, 3,
+		     "PROTVIRT CREATE VCPU: cpu %d handle %llx rc %x rrc %x",
+		     vcpu->vcpu_id, uvcb.cpu_handle, uvcb.header.rc,
+		     uvcb.header.rrc);
 
 	/* Output */
 	vcpu->arch.pv.handle = uvcb.cpu_handle;
@@ -175,6 +183,10 @@ int kvm_s390_pv_create_vm(struct kvm *kvm)
 		 uvcb.guest_handle, uvcb.guest_stor_len, uvcb.header.rc,
 		 uvcb.header.rrc);
 
+	KVM_UV_EVENT(kvm, 3, "PROTVIRT CREATE VM: handle %llx len %llx rc %x rrc %x",
+		 uvcb.guest_handle, uvcb.guest_stor_len, uvcb.header.rc,
+		 uvcb.header.rrc);
+
 	/* Outputs */
 	kvm->arch.pv.handle = uvcb.guest_handle;
 
@@ -206,6 +218,8 @@ int kvm_s390_pv_set_sec_parms(struct kvm *kvm,
 	rc = uv_call(0, (u64)&uvcb);
 	VM_EVENT(kvm, 3, "PROTVIRT VM SET PARMS: rc %x rrc %x",
 		 uvcb.header.rc, uvcb.header.rrc);
+	KVM_UV_EVENT(kvm, 3, "PROTVIRT VM SET PARMS: rc %x rrc %x",
+		     uvcb.header.rc, uvcb.header.rrc);
 	if (rc)
 		return -EINVAL;
 	return 0;
@@ -225,9 +239,12 @@ static int unpack_one(struct kvm *kvm, unsigned long addr, u64 tweak[2])
 
 	rc = uv_make_secure(kvm->arch.gmap, addr, &uvcb, 0);
 
-	if (rc)
+	if (rc) {
 		VM_EVENT(kvm, 3, "PROTVIRT VM UNPACK: failed addr %llx rc %x rrc %x",
 			 uvcb.gaddr, uvcb.header.rc, uvcb.header.rrc);
+		KVM_UV_EVENT(kvm, 3, "PROTVIRT VM UNPACK: failed with rc %x rrc %x",
+			     uvcb.header.rc, uvcb.header.rrc);
+	}
 	return rc;
 }
 
@@ -252,7 +269,7 @@ int kvm_s390_pv_unpack(struct kvm *kvm, unsigned long addr, unsigned long size,
 		addr += PAGE_SIZE;
 		tw[1] += PAGE_SIZE;
 	}
-	VM_EVENT(kvm, 3, "PROTVIRT VM UNPACK: finished rc %x", rc);
+	VM_EVENT(kvm, 3, "PROTVIRT VM UNPACK: finished with rc %x", rc);
 	return rc;
 }
 
-- 
2.24.0

[RFCv2 35/37] KVM: s390: protvirt: Mask PSW interrupt bits for interception 104 and 112

[Christian Borntraeger]

From: Janosch Frank <frankja@linux.ibm.com>

We're not allowed to inject interrupts on those intercept codes. As our
PSW is just a copy of the real one that will be replaced on the next
exit, we can mask out the interrupt bits in the PSW to make sure that we
do not inject anything.

Signed-off-by: Janosch Frank <frankja@linux.ibm.com>
---
 arch/s390/kvm/kvm-s390.c | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
index d4dc156e2c3e..137ae5dc9101 100644
--- a/arch/s390/kvm/kvm-s390.c
+++ b/arch/s390/kvm/kvm-s390.c
@@ -4050,6 +4050,7 @@ static int vcpu_post_run(struct kvm_vcpu *vcpu, int exit_reason)
 	return vcpu_post_run_fault_in_sie(vcpu);
 }
 
+#define PSW_INT_MASK (PSW_MASK_EXT | PSW_MASK_IO | PSW_MASK_MCHECK)
 static int __vcpu_run(struct kvm_vcpu *vcpu)
 {
 	int rc, exit_reason;
@@ -4082,10 +4083,15 @@ static int __vcpu_run(struct kvm_vcpu *vcpu)
 		}
 		exit_reason = sie64a(vcpu->arch.sie_block,
 				     vcpu->run->s.regs.gprs);
+		/* This will likely be moved into a new function. */
 		if (kvm_s390_pv_is_protected(vcpu->kvm)) {
 			memcpy(vcpu->run->s.regs.gprs,
 			       sie_page->pv_grregs,
 			       sizeof(sie_page->pv_grregs));
+			if (vcpu->arch.sie_block->icptcode == ICPT_PV_INSTR ||
+			    vcpu->arch.sie_block->icptcode == ICPT_PV_PREF) {
+				vcpu->arch.sie_block->gpsw.mask &= ~PSW_INT_MASK;
+			}
 		}
 		local_irq_disable();
 		__enable_cpu_timer_accounting(vcpu);
-- 
2.24.0

[RFCv2 36/37] KVM: s390: protvirt: do not inject interrupts after start

[Christian Borntraeger]

As PSW restart is handled by the ultravisor (and we only get a start
notification) we must re-check the PSW after a start before injecting
interrupts.

Signed-off-by: Christian Borntraeger <borntraeger@de.ibm.com>
---
 arch/s390/kvm/kvm-s390.c | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
index 137ae5dc9101..3e4716b3fc02 100644
--- a/arch/s390/kvm/kvm-s390.c
+++ b/arch/s390/kvm/kvm-s390.c
@@ -4451,6 +4451,12 @@ void kvm_s390_vcpu_start(struct kvm_vcpu *vcpu)
 	/* Let's tell the UV that we want to start again */
 	kvm_s390_pv_set_cpu_state(vcpu, PV_CPU_STATE_OPR);
 	kvm_s390_clear_cpuflags(vcpu, CPUSTAT_STOPPED);
+	/*
+	 * The real PSW might have changed due to an interpreted RESTART.
+	 * We block all interrupts and let the next sie exit refresh our view.
+	 */
+	if (kvm_s390_pv_is_protected(vcpu->kvm))
+		vcpu->arch.sie_block->gpsw.mask &= ~PSW_INT_MASK;
 	/*
 	 * Another VCPU might have used IBS while we were offline.
 	 * Let's play safe and flush the VCPU at startup.
-- 
2.24.0

[RFCv2 37/37] KVM: s390: protvirt: Add UV cpu reset calls

[Christian Borntraeger]

From: Janosch Frank <frankja@linux.ibm.com>

For protected VMs, the VCPU resets are done by the Ultravisor, as KVM
has no access to the VCPU registers.

As the Ultravisor will only accept a call for the reset that is
needed, we need to fence the UV calls when chaining resets.

Signed-off-by: Janosch Frank <frankja@linux.ibm.com>
---
 arch/s390/kvm/kvm-s390.c | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)

diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
index 3e4716b3fc02..f7a3f84be064 100644
--- a/arch/s390/kvm/kvm-s390.c
+++ b/arch/s390/kvm/kvm-s390.c
@@ -4699,6 +4699,7 @@ long kvm_arch_vcpu_ioctl(struct file *filp,
 	void __user *argp = (void __user *)arg;
 	int idx;
 	long r;
+	u32 ret;
 
 	vcpu_load(vcpu);
 
@@ -4720,14 +4721,33 @@ long kvm_arch_vcpu_ioctl(struct file *filp,
 	case KVM_S390_CLEAR_RESET:
 		r = 0;
 		kvm_arch_vcpu_ioctl_clear_reset(vcpu);
+		if (kvm_s390_pv_handle_cpu(vcpu)) {
+			r = uv_cmd_nodata(kvm_s390_pv_handle_cpu(vcpu),
+					  UVC_CMD_CPU_RESET_CLEAR, &ret);
+			VCPU_EVENT(vcpu, 3, "PROTVIRT RESET CLEAR VCPU: cpu %d rc %x rrc %x",
+				   vcpu->vcpu_id, ret >> 16, ret & 0x0000ffff);
+		}
 		break;
 	case KVM_S390_INITIAL_RESET:
 		r = 0;
 		kvm_arch_vcpu_ioctl_initial_reset(vcpu);
+		if (kvm_s390_pv_handle_cpu(vcpu)) {
+			r = uv_cmd_nodata(kvm_s390_pv_handle_cpu(vcpu),
+					  UVC_CMD_CPU_RESET_INITIAL,
+					  &ret);
+			VCPU_EVENT(vcpu, 3, "PROTVIRT RESET INITIAL VCPU: cpu %d rc %x rrc %x",
+				   vcpu->vcpu_id, ret >> 16, ret & 0x0000ffff);
+		}
 		break;
 	case KVM_S390_NORMAL_RESET:
 		r = 0;
 		kvm_arch_vcpu_ioctl_normal_reset(vcpu);
+		if (kvm_s390_pv_handle_cpu(vcpu)) {
+			r = uv_cmd_nodata(kvm_s390_pv_handle_cpu(vcpu),
+					  UVC_CMD_CPU_RESET, &ret);
+			VCPU_EVENT(vcpu, 3, "PROTVIRT RESET NORMAL VCPU: cpu %d rc %x rrc %x",
+				   vcpu->vcpu_id, ret >> 16, ret & 0x0000ffff);
+		}
 		break;
 	case KVM_SET_ONE_REG:
 	case KVM_GET_ONE_REG: {
-- 
2.24.0

Re: [RFCv2 00/37] KVM: s390: Add support for protected VMs

[Cornelia Huck]

On Mon,  3 Feb 2020 08:19:20 -0500
Christian Borntraeger <borntraeger@de.ibm.com> wrote:

> Protected VMs (PVM) are KVM VMs, where KVM can't access the VM's state
> like guest memory and guest registers anymore. Instead the PVMs are
> mostly managed by a new entity called Ultravisor (UV), which provides
> an API, so KVM and the PV can request management actions.
> 
> PVMs are encrypted at rest and protected from hypervisor access while
> running. They switch from a normal operation into protected mode, so
> we can still use the standard boot process to load a encrypted blob
> and then move it into protected mode.
> 
> Rebooting is only possible by passing through the unprotected/normal
> mode and switching to protected again.
> 
> All patches are in the protvirtv2 branch of the korg s390 kvm git
> (on top of Janoschs reset rework).
> 
> Claudio presented the technology at his presentation at KVM Forum
> 2019.

Do you have a changelog from v1 somewhere?

> 
> This contains a "pretty small" common code memory management change that
> will allow paging, guest backing with files etc almost just like normal
> VMs. Please note that the memory management part will still see some
> changes to deal with a corner case for the adapter interrupt indicator
> pages. So please focus on the non-mm parts (which hopefully has
> everthing addressed in the next version). Claudio will work with Andrea
> regarding this.

Re: [RFCv2 00/37] KVM: s390: Add support for protected VMs

[Christian Borntraeger]

On 03.02.20 14:23, Cornelia Huck wrote:
> On Mon,  3 Feb 2020 08:19:20 -0500
> Christian Borntraeger <borntraeger@de.ibm.com> wrote:
> 
>> Protected VMs (PVM) are KVM VMs, where KVM can't access the VM's state
>> like guest memory and guest registers anymore. Instead the PVMs are
>> mostly managed by a new entity called Ultravisor (UV), which provides
>> an API, so KVM and the PV can request management actions.
>>
>> PVMs are encrypted at rest and protected from hypervisor access while
>> running. They switch from a normal operation into protected mode, so
>> we can still use the standard boot process to load a encrypted blob
>> and then move it into protected mode.
>>
>> Rebooting is only possible by passing through the unprotected/normal
>> mode and switching to protected again.
>>
>> All patches are in the protvirtv2 branch of the korg s390 kvm git
>> (on top of Janoschs reset rework).
>>
>> Claudio presented the technology at his presentation at KVM Forum
>> 2019.
> 
> Do you have a changelog from v1 somewhere?

Probably too many things have changed.

There is still the old branch protvirt that rebases almost fine on top of v5.5
so here are the differences that I can see
- docs as rst instead of txt
- memory management now with paging
- MEMOP interface now different (new code points instead of abusing the old ones)
- prefix page handling with intercept 112 (prefix not secure)
- interrupt refreshing exits reworked according to review
- fencing in several ioctls
- based on reset rework
- fixes fixes and fixes
> 
>>
>> This contains a "pretty small" common code memory management change that
>> will allow paging, guest backing with files etc almost just like normal
>> VMs. Please note that the memory management part will still see some
>> changes to deal with a corner case for the adapter interrupt indicator
>> pages. So please focus on the non-mm parts (which hopefully has
>> everthing addressed in the next version). Claudio will work with Andrea
>> regarding this.
> 

Re: [RFCv2 00/37] KVM: s390: Add support for protected VMs

[Cornelia Huck]

On Mon, 3 Feb 2020 14:32:52 +0100
Christian Borntraeger <borntraeger@de.ibm.com> wrote:

> On 03.02.20 14:23, Cornelia Huck wrote:
> > On Mon,  3 Feb 2020 08:19:20 -0500
> > Christian Borntraeger <borntraeger@de.ibm.com> wrote:
> >   
> >> Protected VMs (PVM) are KVM VMs, where KVM can't access the VM's state
> >> like guest memory and guest registers anymore. Instead the PVMs are
> >> mostly managed by a new entity called Ultravisor (UV), which provides
> >> an API, so KVM and the PV can request management actions.
> >>
> >> PVMs are encrypted at rest and protected from hypervisor access while
> >> running. They switch from a normal operation into protected mode, so
> >> we can still use the standard boot process to load a encrypted blob
> >> and then move it into protected mode.
> >>
> >> Rebooting is only possible by passing through the unprotected/normal
> >> mode and switching to protected again.
> >>
> >> All patches are in the protvirtv2 branch of the korg s390 kvm git
> >> (on top of Janoschs reset rework).
> >>
> >> Claudio presented the technology at his presentation at KVM Forum
> >> 2019.  
> > 
> > Do you have a changelog from v1 somewhere?  
> 
> Probably too many things have changed.
> 
> There is still the old branch protvirt that rebases almost fine on top of v5.5
> so here are the differences that I can see
> - docs as rst instead of txt
> - memory management now with paging
> - MEMOP interface now different (new code points instead of abusing the old ones)
> - prefix page handling with intercept 112 (prefix not secure)
> - interrupt refreshing exits reworked according to review
> - fencing in several ioctls
> - based on reset rework
> - fixes fixes and fixes

And also some fixes? :)

Ok, I think I'll be able to make my way through this.

Re: [RFCv2 01/37] DOCUMENTATION: protvirt: Protected virtual machine introduction

[Cornelia Huck]

On Mon,  3 Feb 2020 08:19:21 -0500
Christian Borntraeger <borntraeger@de.ibm.com> wrote:

> From: Janosch Frank <frankja@linux.ibm.com>
> 
> Add documentation about protected KVM guests.
> 
> Signed-off-by: Janosch Frank <frankja@linux.ibm.com>
> ---
>  Documentation/virt/kvm/s390-pv.rst | 103 +++++++++++++++++++++++++++++
>  MAINTAINERS                        |   1 +
>  2 files changed, 104 insertions(+)
>  create mode 100644 Documentation/virt/kvm/s390-pv.rst
> 
> diff --git a/Documentation/virt/kvm/s390-pv.rst b/Documentation/virt/kvm/s390-pv.rst
> new file mode 100644
> index 000000000000..5ef7e6cc2180
> --- /dev/null
> +++ b/Documentation/virt/kvm/s390-pv.rst
> @@ -0,0 +1,103 @@
> +.. SPDX-License-Identifier: GPL-2.0
> +============================
> +Ultravisor and Protected VMs
> +============================
> +
> +Summary
> +-------
> +Protected virtual machines (PVM) are KVM VMs, where KVM can't access
> +the VM's state like guest memory and guest registers anymore. Instead,
> +the PVMs are mostly managed by a new entity called Ultravisor
> +(UV). The UV provides an API that can be used by PVMs and KVM to
> +request management actions.
> +
> +Each guest starts in the non-protected mode and then may make a
> +request to transition into protected mode. On transition, KVM
> +registers the guest and its VCPUs with the Ultravisor and prepares
> +everything for running it.
> +
> +The Ultravisor will secure and decrypt the guest's boot memory
> +(i.e. kernel/initrd). It will safeguard state changes like VCPU
> +starts/stops and injected interrupts while the guest is running.
> +
> +As access to the guest's state, such as the SIE state description, is
> +normally needed to be able to run a VM, some changes have been made in
> +SIE behavior. A new format 4 state description has been introduced,
> +where some fields have different meanings for a PVM. SIE exits are
> +minimized as much as possible to improve speed and reduce exposed
> +guest state.

Suggestion: Can you include some ASCII art here describing the
relationship of KVM, PVMs, and the UV? I think there was something in
the KVM Forum talk.

> +
> +
> +Interrupt injection
> +-------------------
> +Interrupt injection is safeguarded by the Ultravisor. As KVM doesn't
> +have access to the VCPUs' lowcores, injection is handled via the
> +format 4 state description.
> +
> +Machine check, external, IO and restart interruptions each can be
> +injected on SIE entry via a bit in the interrupt injection control
> +field (offset 0x54). If the guest cpu is not enabled for the interrupt
> +at the time of injection, a validity interception is recognized. The
> +format 4 state description contains fields in the interception data
> +block where data associated with the interrupt can be transported.
> +
> +Program and Service Call exceptions have another layer of
> +safeguarding; they can only be injected for instructions that have
> +been intercepted into KVM. The exceptions need to be a valid outcome

s/valid/possible/ ?

> +of an instruction emulation by KVM, e.g. we can never inject a
> +addressing exception as they are reported by SIE since KVM has no
> +access to the guest memory.
> +
> +
> +Mask notification interceptions
> +-------------------------------
> +As a replacement for the lctl(g) and lpsw(e) instruction
> +interceptions, two new interception codes have been introduced. One
> +indicating that the contents of CRs 0, 6 or 14 have been changed. And
> +one indicating PSW bit 13 changes.

Hm, I think I already commented on this last time... here is my current
suggestion :)

"In order to be notified when a PVM enables a certain class of
interrupt, KVM cannot intercept lctl(g) and lpsw(e) anymore. As a
replacement, two new interception codes have been introduced: One
indicating that the contents of CRs 0, 6, or 14 have been changed,
indicating different interruption subclasses; and one indicating that
PSW bit 13 has been changed, indicating whether machine checks are
enabled."


> +
> +Instruction emulation
> +---------------------
> +With the format 4 state description for PVMs, the SIE instruction already
> +interprets more instructions than it does with format 2. As it is not
> +able to interpret every instruction, the SIE and the UV safeguard KVM's
> +emulation inputs and outputs.

"It is not able to interpret every instruction, but needs to hand some
tasks to KVM; therefore, the SIE and the UV safeguard..."

?

> +
> +Guest GRs and most of the instruction data, such as I/O data structures,
> +are filtered. Instruction data is copied to and from the Secure
> +Instruction Data Area. Guest GRs are put into / retrieved from the
> +Interception-Data block.

These areas are in the SIE control block, right?

> +
> +The Interception-Data block from the state description's offset 0x380
> +contains GRs 0 - 16. Only GR values needed to emulate an instruction
> +will be copied into this area.
> +
> +The Interception Parameters state description field still contains the
> +the bytes of the instruction text, but with pre-set register values
> +instead of the actual ones. I.e. each instruction always uses the same
> +instruction text, in order not to leak guest instruction text.

This also implies that the register content that a guest had in r<n>
may be in r<m> in the interception data block if <m> is the default
register used for that instruction?

> +
> +The Secure Instruction Data Area contains instruction storage
> +data. Instruction data, i.e. data being referenced by an instruction
> +like the SCCB for sclp, is moved over the SIDA When an instruction is

Maybe move the introduction of the 'SIDA' acronym up to the
introduction of the Secure Instruction Data Area?

Also, s/moved over the SIDA/moved over to the SIDA./ ?


> +intercepted, the SIE will only allow data and program interrupts for
> +this instruction to be moved to the guest via the two data areas
> +discussed before. Other data is either ignored or results in validity
> +interceptions.
> +
> +
> +Instruction emulation interceptions
> +-----------------------------------
> +There are two types of SIE secure instruction intercepts: the normal
> +and the notification type. Normal secure instruction intercepts will
> +make the guest pending for instruction completion of the intercepted
> +instruction type, i.e. on SIE entry it is attempted to complete
> +emulation of the instruction with the data provided by KVM. That might
> +be a program exception or instruction completion.
> +
> +The notification type intercepts inform KVM about guest environment
> +changes due to guest instruction interpretation. Such an interception
> +is recognized for example for the store prefix instruction to provide

s/ for example/, for example,/

> +the new lowcore location. On SIE reentry, any KVM data in the data
> +areas is ignored, program exceptions are not injected and execution
> +continues, as if no intercept had happened.

So, KVM putting stuff there does not cause any exception, it is simply
discarded?

> diff --git a/MAINTAINERS b/MAINTAINERS
> index 56765f542244..90da412bebd9 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -9106,6 +9106,7 @@ L:	kvm@vger.kernel.org
>  W:	http://www.ibm.com/developerworks/linux/linux390/
>  T:	git git://git.kernel.org/pub/scm/linux/kernel/git/kvms390/linux.git
>  S:	Supported
> +F:	Documentation/virt/kvm/s390*
>  F:	arch/s390/include/uapi/asm/kvm*
>  F:	arch/s390/include/asm/gmap.h
>  F:	arch/s390/include/asm/kvm*

Re: [RFCv2 29/37] DOCUMENTATION: protvirt: Diag 308 IPL

[Cornelia Huck]

On Mon,  3 Feb 2020 08:19:49 -0500
Christian Borntraeger <borntraeger@de.ibm.com> wrote:

> From: Janosch Frank <frankja@linux.ibm.com>
> 
> Description of changes that are necessary to move a KVM VM into
> Protected Virtualization mode.

Maybe move this up to the top of the series, so that new reviewers can
get a quick idea about the architecture as a whole? It might also make
sense to make the two documents link to each other...

> 
> Signed-off-by: Janosch Frank <frankja@linux.ibm.com>
> ---
>  Documentation/virt/kvm/s390-pv-boot.rst | 64 +++++++++++++++++++++++++
>  1 file changed, 64 insertions(+)
>  create mode 100644 Documentation/virt/kvm/s390-pv-boot.rst
> 
> diff --git a/Documentation/virt/kvm/s390-pv-boot.rst b/Documentation/virt/kvm/s390-pv-boot.rst
> new file mode 100644
> index 000000000000..431cd5d7f686
> --- /dev/null
> +++ b/Documentation/virt/kvm/s390-pv-boot.rst
> @@ -0,0 +1,64 @@
> +.. SPDX-License-Identifier: GPL-2.0
> +=========================
> +Boot/IPL of Protected VMs
> +=========================

...especially as the reader will have no idea what a "Protected VM" is,
unless they have read the other document before.


> +
> +Summary
> +-------
> +Protected VMs are encrypted while not running. On IPL a small
> +plaintext bootloader is started which provides information about the
> +encrypted components and necessary metadata to KVM to decrypt it.

s/it/the PVM/ ?

> +
> +Based on this data, KVM will make the PV known to the Ultravisor and

I think the other document uses 'PVM'... probably better to keep that
consistent.

> +instruct it to secure its memory, decrypt the components and verify

Too many it and its here... maybe use the abbreviations instead?

> +the data and address list hashes, to ensure integrity. Afterwards KVM
> +can run the PV via SIE which the UV will intercept and execute on
> +KVM's behalf.
> +
> +The switch into PV mode lets us load encrypted guest executables and
> +data via every available method (network, dasd, scsi, direct kernel,
> +...) without the need to change the boot process.
> +
> +
> +Diag308
> +-------
> +This diagnose instruction is the basis for VM IPL. The VM can set and
> +retrieve IPL information blocks, that specify the IPL method/devices
> +and request VM memory and subsystem resets, as well as IPLs.
> +
> +For PVs this concept has been continued with new subcodes:

s/continued/extended/ ?

> +
> +Subcode 8: Set an IPL Information Block of type 5.

"type 5" == information block for PVMs? Better spell that out.

> +Subcode 9: Store the saved block in guest memory
> +Subcode 10: Move into Protected Virtualization mode
> +
> +The new PV load-device-specific-parameters field specifies all data,
> +that is necessary to move into PV mode.
> +
> +* PV Header origin
> +* PV Header length
> +* List of Components composed of
> +   * AES-XTS Tweak prefix
> +   * Origin
> +   * Size
> +
> +The PV header contains the keys and hashes, which the UV will use to
> +decrypt and verify the PV, as well as control flags and a start PSW.
> +
> +The components are for instance an encrypted kernel, kernel cmd and

s/kernel cmd/kernel command line/ ?

> +initrd. The components are decrypted by the UV.
> +
> +All non-decrypted data of the non-PV guest instance are zero on first
> +access of the PV.

"non-PV guest" == "the guest before it switches to protected
virtualization mode" ?

> +
> +
> +When running in a protected mode some subcodes will result in

s/in a/in/

> +exceptions or return error codes.
> +
> +Subcodes 4 and 7 will result in specification exceptions.

"Subcodes 4 and 7, which would not clear the guest memory, ..." ?

> +When removing a secure VM, the UV will clear all memory, so we can't
> +have non-clearing IPL subcodes.
> +
> +Subcodes 8, 9, 10 will result in specification exceptions.
> +Re-IPL into a protected mode is only possible via a detour into non
> +protected mode.

In general, this looks like a good overview about how the guest can
move into protected virt mode.

Some information I'm missing in this doc: Where do the keys come from?
I assume from the machine... is there one key per CEC? Can keys be
transferred? Can an image be introspected to find out if it is possible
to run it on a given system?

(Not sure if there is a better resting place for that kind of
information.)

Re: [RFCv2 02/37] s390/protvirt: introduce host side setup

[Cornelia Huck]

On Mon,  3 Feb 2020 08:19:22 -0500
Christian Borntraeger <borntraeger@de.ibm.com> wrote:

> From: Vasily Gorbik <gor@linux.ibm.com>
> 
> Introduce KVM_S390_PROTECTED_VIRTUALIZATION_HOST kbuild option for
> protected virtual machines hosting support code.

Hm... I seem to remember that you wanted to drop this config option and
always build the code, in order to reduce complexity. Have you
reconsidered this?

> 
> Add "prot_virt" command line option which controls if the kernel
> protected VMs support is enabled at early boot time. This has to be
> done early, because it needs large amounts of memory and will disable
> some features like STP time sync for the lpar.
> 
> Extend ultravisor info definitions and expose it via uv_info struct
> filled in during startup.
> 
> Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
> ---
>  .../admin-guide/kernel-parameters.txt         |  5 ++
>  arch/s390/boot/Makefile                       |  2 +-
>  arch/s390/boot/uv.c                           | 20 +++++++-
>  arch/s390/include/asm/uv.h                    | 46 ++++++++++++++++--
>  arch/s390/kernel/Makefile                     |  1 +
>  arch/s390/kernel/setup.c                      |  4 --
>  arch/s390/kernel/uv.c                         | 48 +++++++++++++++++++
>  arch/s390/kvm/Kconfig                         | 19 ++++++++
>  8 files changed, 136 insertions(+), 9 deletions(-)
>  create mode 100644 arch/s390/kernel/uv.c

(...)

> diff --git a/arch/s390/kernel/uv.c b/arch/s390/kernel/uv.c
> new file mode 100644
> index 000000000000..35ce89695509
> --- /dev/null
> +++ b/arch/s390/kernel/uv.c
> @@ -0,0 +1,48 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * Common Ultravisor functions and initialization
> + *
> + * Copyright IBM Corp. 2019

Happy new year?

> + */
> +#include <linux/kernel.h>
> +#include <linux/types.h>
> +#include <linux/sizes.h>
> +#include <linux/bitmap.h>
> +#include <linux/memblock.h>
> +#include <asm/facility.h>
> +#include <asm/sections.h>
> +#include <asm/uv.h>
> +
> +#ifdef CONFIG_PROTECTED_VIRTUALIZATION_GUEST
> +int __bootdata_preserved(prot_virt_guest);

Confused. You have this and uv_info below both in this file and in
boot/uv.c. Is there some magic happening in __bootdata_preserved()?

> +#endif
> +
> +#ifdef CONFIG_KVM_S390_PROTECTED_VIRTUALIZATION_HOST
> +int prot_virt_host;
> +EXPORT_SYMBOL(prot_virt_host);
> +struct uv_info __bootdata_preserved(uv_info);
> +EXPORT_SYMBOL(uv_info);
> +
> +static int __init prot_virt_setup(char *val)
> +{
> +	bool enabled;
> +	int rc;
> +
> +	rc = kstrtobool(val, &enabled);
> +	if (!rc && enabled)
> +		prot_virt_host = 1;
> +
> +	if (is_prot_virt_guest() && prot_virt_host) {
> +		prot_virt_host = 0;
> +		pr_info("Running as protected virtualization guest.");

Trying to disentangle that a bit in my mind...

If we don't have facility 158, is_prot_virt_guest() will return 0. If
protected host support has been requested, we'll print a message below
(and turn it off).

If the hardware provides the facilities for running as a protected virt
guest, we turn off protected virt host support if requested and print a
messages that we're a guest.

Two questions:
- Can the hardware ever provide both host and guest interfaces at the
  same time? I guess not; maybe add a comment?
- Do we also want to print a message that we're running as a guest if
  the user didn't enable host support? If not, maybe prefix the message
  with "Cannot enable support for protected virtualization host:" or
  so? (Maybe also a good idea for the message below.)

> +	}
> +
> +	if (prot_virt_host && !test_facility(158)) {
> +		prot_virt_host = 0;
> +		pr_info("The ultravisor call facility is not available.");
> +	}
> +
> +	return rc;
> +}
> +early_param("prot_virt", prot_virt_setup);
> +#endif

(...)

Re: [RFCv2 01/37] DOCUMENTATION: protvirt: Protected virtual machine introduction

[Christian Borntraeger]

On 03.02.20 16:42, Cornelia Huck wrote:
[...]
>> +As access to the guest's state, such as the SIE state description, is
>> +normally needed to be able to run a VM, some changes have been made in
>> +SIE behavior. A new format 4 state description has been introduced,
>> +where some fields have different meanings for a PVM. SIE exits are
>> +minimized as much as possible to improve speed and reduce exposed
>> +guest state.
> 
> Suggestion: Can you include some ASCII art here describing the
> relationship of KVM, PVMs, and the UV? I think there was something in
> the KVM Forum talk.

Uh, maybe I find someone who is good at doing ASCII art - I am not.
I think I would prefer to have a link to the KVM forum talk?

I will add
+
+Links
+-----
+`KVM Forum 2019 presentation <https://static.sched.com/hosted_files/kvmforum2019/3b/ibm_protected_vms_s390x.pdf>`_

at the bottom, just in case.

[...]
>> +Program and Service Call exceptions have another layer of
>> +safeguarding; they can only be injected for instructions that have
>> +been intercepted into KVM. The exceptions need to be a valid outcome
> 
> s/valid/possible/ ?

hmm, this is bikeshedding, but I think valid is better because it refers to
the architecture. 

> 
>> +of an instruction emulation by KVM, e.g. we can never inject a
>> +addressing exception as they are reported by SIE since KVM has no
>> +access to the guest memory.
>> +
>> +
>> +Mask notification interceptions
>> +-------------------------------
>> +As a replacement for the lctl(g) and lpsw(e) instruction
>> +interceptions, two new interception codes have been introduced. One
>> +indicating that the contents of CRs 0, 6 or 14 have been changed. And
>> +one indicating PSW bit 13 changes.
> 
> Hm, I think I already commented on this last time... here is my current
> suggestion :)
> 
> "In order to be notified when a PVM enables a certain class of
> interrupt, KVM cannot intercept lctl(g) and lpsw(e) anymore. As a
> replacement, two new interception codes have been introduced: One
> indicating that the contents of CRs 0, 6, or 14 have been changed,
> indicating different interruption subclasses; and one indicating that
> PSW bit 13 has been changed, indicating whether machine checks are
> enabled."

I will use this with ... indicating that a machine check intervention was
requested and those are now enabled.

> 
>> +
>> +Instruction emulation
>> +---------------------
>> +With the format 4 state description for PVMs, the SIE instruction already
>> +interprets more instructions than it does with format 2. As it is not
>> +able to interpret every instruction, the SIE and the UV safeguard KVM's
>> +emulation inputs and outputs.
> 
> "It is not able to interpret every instruction, but needs to hand some
> tasks to KVM; therefore, the SIE and the UV safeguard..."

Will use this.


> 
> ?
> 
>> +
>> +Guest GRs and most of the instruction data, such as I/O data structures,
>> +are filtered. Instruction data is copied to and from the Secure
>> +Instruction Data Area. Guest GRs are put into / retrieved from the
>> +Interception-Data block.
> 
> These areas are in the SIE control block, right?

SIDA is a new block, linked from SIE control block. The register are stored in
the control block. I think this is really not relevant for such a document (too
much technical detail when explaining the big idea), but I will fix the name of
the location at 0x380 though.  (its now general register save area).
> 
>> +
>> +The Interception-Data block from the state description's offset 0x380
>> +contains GRs 0 - 16. Only GR values needed to emulate an instruction
>> +will be copied into this area.
>> +
>> +The Interception Parameters state description field still contains the
>> +the bytes of the instruction text, but with pre-set register values
>> +instead of the actual ones. I.e. each instruction always uses the same
>> +instruction text, in order not to leak guest instruction text.
> 
> This also implies that the register content that a guest had in r<n>
> may be in r<m> in the interception data block if <m> is the default
> register used for that instruction?

yes. I will do
---
...Guest GRs are put into / retrieved from the
General Register Save Area.

Only GR values needed to emulate an instruction will be copied into this 
area and the real register numbers will be hidden.

The Interception Parameters state description field still contains the
the bytes of the instruction text, but with pre-set register values
instead of the actual ones. I.e. each instruction always uses the same
instruction text, in order not to leak guest instruction text.
This also implies that the register content that a guest had in r<n>
may be in r<m> from the hypervisors point of view.

---

> 
>> +
>> +The Secure Instruction Data Area contains instruction storage
>> +data. Instruction data, i.e. data being referenced by an instruction
>> +like the SCCB for sclp, is moved over the SIDA When an instruction is
> 
> Maybe move the introduction of the 'SIDA' acronym up to the
> introduction of the Secure Instruction Data Area?
> 
> Also, s/moved over the SIDA/moved over to the SIDA./ ?

Fixed. 
> 
[...]
>> +The notification type intercepts inform KVM about guest environment
>> +changes due to guest instruction interpretation. Such an interception
>> +is recognized for example for the store prefix instruction to provide
> 
> s/ for example/, for example,/

fixed.

> 
>> +the new lowcore location. On SIE reentry, any KVM data in the data
>> +areas is ignored, program exceptions are not injected and execution
>> +continues, as if no intercept had happened.
> 
> So, KVM putting stuff there does not cause any exception, it is simply
> discarded?

Might be a bit ambigious. SIE will not inject program interrupts as the
instruction has already completed. What about

On SIE reentry, any KVM data in the data areas is ignored and execution
continues as if the guest instruction has completed. For that reasons
KVM is not allowed to inject a program interrupt. 

Re: [RFCv2 02/37] s390/protvirt: introduce host side setup

[Christian Borntraeger]

On 03.02.20 18:12, Cornelia Huck wrote:
> On Mon,  3 Feb 2020 08:19:22 -0500
> Christian Borntraeger <borntraeger@de.ibm.com> wrote:
> 
>> From: Vasily Gorbik <gor@linux.ibm.com>
>>
>> Introduce KVM_S390_PROTECTED_VIRTUALIZATION_HOST kbuild option for
>> protected virtual machines hosting support code.
> 
> Hm... I seem to remember that you wanted to drop this config option and
> always build the code, in order to reduce complexity. Have you
> reconsidered this?

I am still in favour of removing this, but I did not get an "yes, lets do
it" answer. Since removing is easier than re-adding its still in.

 [...]
>> + * Copyright IBM Corp. 2019
> 
> Happy new year?

yep :-)
[..]

>> +
>> +#ifdef CONFIG_PROTECTED_VIRTUALIZATION_GUEST
>> +int __bootdata_preserved(prot_virt_guest);
> 
> Confused. You have this and uv_info below both in this file and in
> boot/uv.c. Is there some magic happening in __bootdata_preserved()?

Yes, this is information that is transferred from the decompressor
to Linux. 
I think we discussed about this the last time as well?


> 
>> +#endif
>> +
>> +#ifdef CONFIG_KVM_S390_PROTECTED_VIRTUALIZATION_HOST
>> +int prot_virt_host;
>> +EXPORT_SYMBOL(prot_virt_host);
>> +struct uv_info __bootdata_preserved(uv_info);
>> +EXPORT_SYMBOL(uv_info);
>> +
>> +static int __init prot_virt_setup(char *val)
>> +{
>> +	bool enabled;
>> +	int rc;
>> +
>> +	rc = kstrtobool(val, &enabled);
>> +	if (!rc && enabled)
>> +		prot_virt_host = 1;
>> +
>> +	if (is_prot_virt_guest() && prot_virt_host) {
>> +		prot_virt_host = 0;
>> +		pr_info("Running as protected virtualization guest.");
> 
> Trying to disentangle that a bit in my mind...
> 
> If we don't have facility 158, is_prot_virt_guest() will return 0. If
> protected host support has been requested, we'll print a message below
> (and turn it off).

yes, a guest cannot be a host. 
> 
> If the hardware provides the facilities for running as a protected virt
> guest, we turn off protected virt host support if requested and print a
> messages that we're a guest.
> 
> Two questions:
> - Can the hardware ever provide both host and guest interfaces at the
>   same time? I guess not; maybe add a comment?

Right, you are either guest or host. 

> - Do we also want to print a message that we're running as a guest if
>   the user didn't enable host support? If not, maybe prefix the message
>   with "Cannot enable support for protected virtualization host:" or
>   so? (Maybe also a good idea for the message below.)

Line too long and I hate breaking string over multiple lines.
I can change if somebody comes up with a proper message that is not too long.

Re: [RFCv2 29/37] DOCUMENTATION: protvirt: Diag 308 IPL

[Christian Borntraeger]

On 03.02.20 17:13, Cornelia Huck wrote:
> On Mon,  3 Feb 2020 08:19:49 -0500
> Christian Borntraeger <borntraeger@de.ibm.com> wrote:
> 
>> From: Janosch Frank <frankja@linux.ibm.com>
>>
>> Description of changes that are necessary to move a KVM VM into
>> Protected Virtualization mode.
> 
> Maybe move this up to the top of the series, so that new reviewers can
> get a quick idea about the architecture as a whole? It might also make
Will do. 
> sense to make the two documents link to each other...
I added both files to the kvm index file and changed the title
to contain s390. I also added a link to the base doc.

> 
>>
>> Signed-off-by: Janosch Frank <frankja@linux.ibm.com>
>> ---
>>  Documentation/virt/kvm/s390-pv-boot.rst | 64 +++++++++++++++++++++++++
>>  1 file changed, 64 insertions(+)
>>  create mode 100644 Documentation/virt/kvm/s390-pv-boot.rst
>>
>> diff --git a/Documentation/virt/kvm/s390-pv-boot.rst b/Documentation/virt/kvm/s390-pv-boot.rst
>> new file mode 100644
>> index 000000000000..431cd5d7f686
>> --- /dev/null
>> +++ b/Documentation/virt/kvm/s390-pv-boot.rst
>> @@ -0,0 +1,64 @@
>> +.. SPDX-License-Identifier: GPL-2.0
>> +=========================
>> +Boot/IPL of Protected VMs
>> +=========================
> 
> ...especially as the reader will have no idea what a "Protected VM" is,
> unless they have read the other document before.

> 
> 
>> +
>> +Summary
>> +-------
>> +Protected VMs are encrypted while not running. On IPL a small
>> +plaintext bootloader is started which provides information about the
>> +encrypted components and necessary metadata to KVM to decrypt it.
> 
> s/it/the PVM/ ?

ack

This section looks now:

---
Protected Virtual Machines (PVM) are not accessible by I/O or the
hypervisor.  When the hypervisor wants to access the memory of PVMs
the memory needs to be made accessible. When doing so, the memory will
be encrypted.  See :doc:`s390-pv` for details.

On IPL a....
---

> 
>> +
>> +Based on this data, KVM will make the PV known to the Ultravisor and
> 
> I think the other document uses 'PVM'... probably better to keep that
> consistent.

The feature name might change to secure execution (SE). I will need to
go over this again. But I think we can continue to name the virtual
machines protected virtual machines as this is more a description than
a brand name.


> 
>> +instruct it to secure its memory, decrypt the components and verify
> 
> Too many it and its here... maybe use the abbreviations instead?

fixed

> 
>> +the data and address list hashes, to ensure integrity. Afterwards KVM
>> +can run the PV via SIE which the UV will intercept and execute on
>> +KVM's behalf.
>> +
>> +The switch into PV mode lets us load encrypted guest executables and
>> +data via every available method (network, dasd, scsi, direct kernel,
>> +...) without the need to change the boot process.
>> +
>> +
>> +Diag308
>> +-------
>> +This diagnose instruction is the basis for VM IPL. The VM can set and
>> +retrieve IPL information blocks, that specify the IPL method/devices
>> +and request VM memory and subsystem resets, as well as IPLs.
>> +
>> +For PVs this concept has been continued with new subcodes:
> 
> s/continued/extended/ ?

yes.

> 
>> +
>> +Subcode 8: Set an IPL Information Block of type 5.
> 
> "type 5" == information block for PVMs? Better spell that out.

ack

> 
>> +Subcode 9: Store the saved block in guest memory
>> +Subcode 10: Move into Protected Virtualization mode
>> +
>> +The new PV load-device-specific-parameters field specifies all data,
>> +that is necessary to move into PV mode.
>> +
>> +* PV Header origin
>> +* PV Header length
>> +* List of Components composed of
>> +   * AES-XTS Tweak prefix
>> +   * Origin
>> +   * Size
>> +
>> +The PV header contains the keys and hashes, which the UV will use to
>> +decrypt and verify the PV, as well as control flags and a start PSW.
>> +
>> +The components are for instance an encrypted kernel, kernel cmd and
> 
> s/kernel cmd/kernel command line/ ?

ack

> 
>> +initrd. The components are decrypted by the UV.
>> +
>> +All non-decrypted data of the non-PV guest instance are zero on first
>> +access of the PV.
> 
> "non-PV guest" == "the guest before it switches to protected
> virtualization mode" ?

ack
> 
>> +
>> +
>> +When running in a protected mode some subcodes will result in
> 
> s/in a/in/

ack
> 
>> +exceptions or return error codes.
>> +
>> +Subcodes 4 and 7 will result in specification exceptions.
> 
> "Subcodes 4 and 7, which would not clear the guest memory, ..." ?

Subcodes 4 and 7 will result in specification exceptions as they would
not clear out the guest memory.

> 
>> +When removing a secure VM, the UV will clear all memory, so we can't
>> +have non-clearing IPL subcodes.
>> +
>> +Subcodes 8, 9, 10 will result in specification exceptions.
>> +Re-IPL into a protected mode is only possible via a detour into non
>> +protected mode.
> 
> In general, this looks like a good overview about how the guest can
> move into protected virt mode.
> 
> Some information I'm missing in this doc: Where do the keys come from?
> I assume from the machine... is there one key per CEC? Can keys be
> transferred? Can an image be introspected to find out if it is possible
> to run it on a given system?
> 
> (Not sure if there is a better resting place for that kind of
> information.)

There will be tooling as part of the s390-tools. I will add 

---
Keys
----
Every CEC will have a unique public key to enable tooling to build
encrypted images.
See  `s390-tools <https://github.com/ibm-s390-tools/s390-tools/>`_
for the tooling.
---

The s390 tools part is not yet upstream but it will be soon.

Re: [RFCv2 02/37] s390/protvirt: introduce host side setup

[Thomas Huth]

On 03/02/2020 14.19, Christian Borntraeger wrote:
> From: Vasily Gorbik <gor@linux.ibm.com>
> 
> Introduce KVM_S390_PROTECTED_VIRTUALIZATION_HOST kbuild option for
> protected virtual machines hosting support code.
> 
> Add "prot_virt" command line option which controls if the kernel
> protected VMs support is enabled at early boot time. This has to be
> done early, because it needs large amounts of memory and will disable
> some features like STP time sync for the lpar.
> 
> Extend ultravisor info definitions and expose it via uv_info struct
> filled in during startup.
> 
> Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
> ---
[...]
> diff --git a/arch/s390/include/asm/uv.h b/arch/s390/include/asm/uv.h
> index 4093a2856929..32eac3ab2d3b 100644
> --- a/arch/s390/include/asm/uv.h
> +++ b/arch/s390/include/asm/uv.h
> @@ -44,7 +44,19 @@ struct uv_cb_qui {
>  	struct uv_cb_header header;
>  	u64 reserved08;
>  	u64 inst_calls_list[4];
> -	u64 reserved30[15];
> +	u64 reserved30[2];
> +	u64 uv_base_stor_len;
> +	u64 reserved48;
> +	u64 conf_base_phys_stor_len;
> +	u64 conf_base_virt_stor_len;
> +	u64 conf_virt_var_stor_len;
> +	u64 cpu_stor_len;
> +	u32 reserved68[3];

If I count right, that should be named reserved70 instead?

> +	u32 max_num_sec_conf;
> +	u64 max_guest_stor_addr;
> +	u8  reserved80[150-128];

And this one reserved88[158 - 136] ?

> +	u16 max_guest_cpus;
> +	u64 reserved98;

reservedA0 ?

>  } __packed __aligned(8);
>  

Apart from that, the patch looks ok to me.

 Thomas

Re: [RFCv2 02/37] s390/protvirt: introduce host side setup

[Christian Borntraeger]

On 04.02.20 09:40, Thomas Huth wrote:
> On 03/02/2020 14.19, Christian Borntraeger wrote:
>> From: Vasily Gorbik <gor@linux.ibm.com>
>>
>> Introduce KVM_S390_PROTECTED_VIRTUALIZATION_HOST kbuild option for
>> protected virtual machines hosting support code.
>>
>> Add "prot_virt" command line option which controls if the kernel
>> protected VMs support is enabled at early boot time. This has to be
>> done early, because it needs large amounts of memory and will disable
>> some features like STP time sync for the lpar.
>>
>> Extend ultravisor info definitions and expose it via uv_info struct
>> filled in during startup.
>>
>> Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
>> ---
> [...]
>> diff --git a/arch/s390/include/asm/uv.h b/arch/s390/include/asm/uv.h
>> index 4093a2856929..32eac3ab2d3b 100644
>> --- a/arch/s390/include/asm/uv.h
>> +++ b/arch/s390/include/asm/uv.h
>> @@ -44,7 +44,19 @@ struct uv_cb_qui {
>>  	struct uv_cb_header header;
>>  	u64 reserved08;
>>  	u64 inst_calls_list[4];
>> -	u64 reserved30[15];
>> +	u64 reserved30[2];
>> +	u64 uv_base_stor_len;
>> +	u64 reserved48;
>> +	u64 conf_base_phys_stor_len;
>> +	u64 conf_base_virt_stor_len;
>> +	u64 conf_virt_var_stor_len;
>> +	u64 cpu_stor_len;
>> +	u32 reserved68[3];
> 
> If I count right, that should be named reserved70 instead?

Right. Fixed. 
> 
>> +	u32 max_num_sec_conf;
>> +	u64 max_guest_stor_addr;
>> +	u8  reserved80[150-128];
> 
> And this one reserved88[158 - 136] ?

Right. Fixed
> 
>> +	u16 max_guest_cpus;
>> +	u64 reserved98;
> 
> reservedA0 ?

Fixed. (with lowercase a as in other s390 include files):
> 
>>  } __packed __aligned(8);
>>  
> 
> Apart from that, the patch looks ok to me.

Thanks

Re: [RFCv2 01/37] DOCUMENTATION: protvirt: Protected virtual machine introduction

[Cornelia Huck]

On Mon, 3 Feb 2020 22:41:40 +0100
Christian Borntraeger <borntraeger@de.ibm.com> wrote:

> On 03.02.20 16:42, Cornelia Huck wrote:
> [...]
> >> +As access to the guest's state, such as the SIE state description, is
> >> +normally needed to be able to run a VM, some changes have been made in
> >> +SIE behavior. A new format 4 state description has been introduced,
> >> +where some fields have different meanings for a PVM. SIE exits are
> >> +minimized as much as possible to improve speed and reduce exposed
> >> +guest state.  
> > 
> > Suggestion: Can you include some ASCII art here describing the
> > relationship of KVM, PVMs, and the UV? I think there was something in
> > the KVM Forum talk.  
> 
> Uh, maybe I find someone who is good at doing ASCII art - I am not.

That can easily be done later; I just find a diagram showing the
relationship between the components very helpful to figure out what is
going on.

> I think I would prefer to have a link to the KVM forum talk?
> 
> I will add
> +
> +Links
> +-----
> +`KVM Forum 2019 presentation <https://static.sched.com/hosted_files/kvmforum2019/3b/ibm_protected_vms_s390x.pdf>`_
> 
> at the bottom, just in case.

Good idea.

> 
> [...]
> >> +Program and Service Call exceptions have another layer of
> >> +safeguarding; they can only be injected for instructions that have
> >> +been intercepted into KVM. The exceptions need to be a valid outcome  
> > 
> > s/valid/possible/ ?  
> 
> hmm, this is bikeshedding, but I think valid is better because it refers to
> the architecture. 

ok

> 
> >   
> >> +of an instruction emulation by KVM, e.g. we can never inject a
> >> +addressing exception as they are reported by SIE since KVM has no
> >> +access to the guest memory.
> >> +
> >> +
> >> +Mask notification interceptions
> >> +-------------------------------
> >> +As a replacement for the lctl(g) and lpsw(e) instruction
> >> +interceptions, two new interception codes have been introduced. One
> >> +indicating that the contents of CRs 0, 6 or 14 have been changed. And
> >> +one indicating PSW bit 13 changes.  
> > 
> > Hm, I think I already commented on this last time... here is my current
> > suggestion :)
> > 
> > "In order to be notified when a PVM enables a certain class of
> > interrupt, KVM cannot intercept lctl(g) and lpsw(e) anymore. As a
> > replacement, two new interception codes have been introduced: One
> > indicating that the contents of CRs 0, 6, or 14 have been changed,
> > indicating different interruption subclasses; and one indicating that
> > PSW bit 13 has been changed, indicating whether machine checks are
> > enabled."  
> 
> I will use this with ... indicating that a machine check intervention was
> requested and those are now enabled.

ok

> 
> >   
> >> +
> >> +Instruction emulation
> >> +---------------------
> >> +With the format 4 state description for PVMs, the SIE instruction already
> >> +interprets more instructions than it does with format 2. As it is not
> >> +able to interpret every instruction, the SIE and the UV safeguard KVM's
> >> +emulation inputs and outputs.  
> > 
> > "It is not able to interpret every instruction, but needs to hand some
> > tasks to KVM; therefore, the SIE and the UV safeguard..."  
> 
> Will use this.
> 
> 
> > 
> > ?
> >   
> >> +
> >> +Guest GRs and most of the instruction data, such as I/O data structures,
> >> +are filtered. Instruction data is copied to and from the Secure
> >> +Instruction Data Area. Guest GRs are put into / retrieved from the
> >> +Interception-Data block.  
> > 
> > These areas are in the SIE control block, right?  
> 
> SIDA is a new block, linked from SIE control block. The register are stored in
> the control block. I think this is really not relevant for such a document (too
> much technical detail when explaining the big idea), but I will fix the name of
> the location at 0x380 though.  (its now general register save area).

It's mostly that the block makes an appearance here, and it's unclear
what it is and where it resides. Whether it is in the control block or
is a satellite block is not really relevant for this document, I agree;
but can we make it more obvious that it is another data structure
associated with SIE? Maybe something like,

"The control structures associated with SIE provide the Secure
Instruction Data Area (SIDA) and the Interception-Data block. [Does
that one have an acronym?] Instruction data is copied to and from the
SIDA. ..."

> >   
> >> +
> >> +The Interception-Data block from the state description's offset 0x380
> >> +contains GRs 0 - 16. Only GR values needed to emulate an instruction
> >> +will be copied into this area.
> >> +
> >> +The Interception Parameters state description field still contains the
> >> +the bytes of the instruction text, but with pre-set register values
> >> +instead of the actual ones. I.e. each instruction always uses the same
> >> +instruction text, in order not to leak guest instruction text.  
> > 
> > This also implies that the register content that a guest had in r<n>
> > may be in r<m> in the interception data block if <m> is the default
> > register used for that instruction?  
> 
> yes. I will do
> ---
> ...Guest GRs are put into / retrieved from the
> General Register Save Area.
> 
> Only GR values needed to emulate an instruction will be copied into this 
> area and the real register numbers will be hidden.
> 
> The Interception Parameters state description field still contains the
> the bytes of the instruction text, but with pre-set register values
> instead of the actual ones. I.e. each instruction always uses the same
> instruction text, in order not to leak guest instruction text.
> This also implies that the register content that a guest had in r<n>
> may be in r<m> from the hypervisors point of view.

ok

> 
> ---
> 
> >   
> >> +
> >> +The Secure Instruction Data Area contains instruction storage
> >> +data. Instruction data, i.e. data being referenced by an instruction
> >> +like the SCCB for sclp, is moved over the SIDA When an instruction is  
> > 
> > Maybe move the introduction of the 'SIDA' acronym up to the
> > introduction of the Secure Instruction Data Area?
> > 
> > Also, s/moved over the SIDA/moved over to the SIDA./ ?  
> 
> Fixed. 
> >   
> [...]
> >> +The notification type intercepts inform KVM about guest environment
> >> +changes due to guest instruction interpretation. Such an interception
> >> +is recognized for example for the store prefix instruction to provide  
> > 
> > s/ for example/, for example,/  
> 
> fixed.
> 
> >   
> >> +the new lowcore location. On SIE reentry, any KVM data in the data
> >> +areas is ignored, program exceptions are not injected and execution
> >> +continues, as if no intercept had happened.  
> > 
> > So, KVM putting stuff there does not cause any exception, it is simply
> > discarded?  
> 
> Might be a bit ambigious. SIE will not inject program interrupts as the
> instruction has already completed. What about
> 
> On SIE reentry, any KVM data in the data areas is ignored and execution
> continues as if the guest instruction has completed. For that reasons

s/has/had/
s/reasons/reason,/

> KVM is not allowed to inject a program interrupt. 
> 

Sounds good to me.

Re: [RFCv2 02/37] s390/protvirt: introduce host side setup

[Cornelia Huck]

On Mon, 3 Feb 2020 23:03:42 +0100
Christian Borntraeger <borntraeger@de.ibm.com> wrote:

> On 03.02.20 18:12, Cornelia Huck wrote:
> > On Mon,  3 Feb 2020 08:19:22 -0500
> > Christian Borntraeger <borntraeger@de.ibm.com> wrote:
> >   
> >> From: Vasily Gorbik <gor@linux.ibm.com>
> >>
> >> Introduce KVM_S390_PROTECTED_VIRTUALIZATION_HOST kbuild option for
> >> protected virtual machines hosting support code.  
> > 
> > Hm... I seem to remember that you wanted to drop this config option and
> > always build the code, in order to reduce complexity. Have you
> > reconsidered this?  
> 
> I am still in favour of removing this, but I did not get an "yes, lets do
> it" answer. Since removing is easier than re-adding its still in.

ok

> 
>  [...]
> >> + * Copyright IBM Corp. 2019  
> > 
> > Happy new year?  
> 
> yep :-)
> [..]
> 
> >> +
> >> +#ifdef CONFIG_PROTECTED_VIRTUALIZATION_GUEST
> >> +int __bootdata_preserved(prot_virt_guest);  
> > 
> > Confused. You have this and uv_info below both in this file and in
> > boot/uv.c. Is there some magic happening in __bootdata_preserved()?  
> 
> Yes, this is information that is transferred from the decompressor
> to Linux. 
> I think we discussed about this the last time as well?

I think I was confused about different things last time...

But that is probably a sign that this wants a comment :)

> 
> 
> >   
> >> +#endif
> >> +
> >> +#ifdef CONFIG_KVM_S390_PROTECTED_VIRTUALIZATION_HOST
> >> +int prot_virt_host;
> >> +EXPORT_SYMBOL(prot_virt_host);
> >> +struct uv_info __bootdata_preserved(uv_info);
> >> +EXPORT_SYMBOL(uv_info);
> >> +
> >> +static int __init prot_virt_setup(char *val)
> >> +{
> >> +	bool enabled;
> >> +	int rc;
> >> +
> >> +	rc = kstrtobool(val, &enabled);
> >> +	if (!rc && enabled)
> >> +		prot_virt_host = 1;
> >> +
> >> +	if (is_prot_virt_guest() && prot_virt_host) {
> >> +		prot_virt_host = 0;
> >> +		pr_info("Running as protected virtualization guest.");  
> > 
> > Trying to disentangle that a bit in my mind...
> > 
> > If we don't have facility 158, is_prot_virt_guest() will return 0. If
> > protected host support has been requested, we'll print a message below
> > (and turn it off).  
> 
> yes, a guest cannot be a host. 
> > 
> > If the hardware provides the facilities for running as a protected virt
> > guest, we turn off protected virt host support if requested and print a
> > messages that we're a guest.
> > 
> > Two questions:
> > - Can the hardware ever provide both host and guest interfaces at the
> >   same time? I guess not; maybe add a comment?  
> 
> Right, you are either guest or host. 
> 
> > - Do we also want to print a message that we're running as a guest if
> >   the user didn't enable host support? If not, maybe prefix the message
> >   with "Cannot enable support for protected virtualization host:" or
> >   so? (Maybe also a good idea for the message below.)  
> 
> Line too long and I hate breaking string over multiple lines.
> I can change if somebody comes up with a proper message that is not too long.
> 

Fair enough; it's just that it's not very clear from the messages in
the log what happened. Maybe

"prot_virt: Running as protected virtualization guest."
"prot_virt: The ultravisor call facility is not available."

That at least links back to the kernel parameter.

[Aside: Would prot_virt_host be a better name? But that probably moves
us into bikeshed territory.]

Re: [RFCv2 01/37] DOCUMENTATION: protvirt: Protected virtual machine introduction

[Christian Borntraeger]

On 04.02.20 10:19, Cornelia Huck wrote:
[...]
>>>> +Guest GRs and most of the instruction data, such as I/O data structures,
>>>> +are filtered. Instruction data is copied to and from the Secure
>>>> +Instruction Data Area. Guest GRs are put into / retrieved from the
>>>> +Interception-Data block.  
>>>
>>> These areas are in the SIE control block, right?  
>>
>> SIDA is a new block, linked from SIE control block. The register are stored in
>> the control block. I think this is really not relevant for such a document (too
>> much technical detail when explaining the big idea), but I will fix the name of
>> the location at 0x380 though.  (its now general register save area).
> 
> It's mostly that the block makes an appearance here, and it's unclear
> what it is and where it resides. Whether it is in the control block or
> is a satellite block is not really relevant for this document, I agree;
> but can we make it more obvious that it is another data structure
> associated with SIE? Maybe something like,
> 
> "The control structures associated with SIE provide the Secure
> Instruction Data Area (SIDA) and the Interception-Data block. [Does
> that one have an acronym?] Instruction data is copied to and from the
> SIDA. ..."

It is now:

----
The control structures associated with SIE provide the Secure
Instruction Data Area (SIDA), the Interception Parameters (IP) and the
Secure Interception General Register Save Area.  Guest GRs and most of
the instruction data, such as I/O data structures, are filtered.
Instruction data is copied to and from the Secure Instruction Data
Area (SIDA) when needed.  Guest GRs are put into / retrieved from the
Secure Interception General Register Save Area.
----

 

FWIW, I have a new branch pv_worktree on kernelorg where I have all review
feedback included.



truction has already completed. What about
>>
>> On SIE reentry, any KVM data in the data areas is ignored and execution
>> continues as if the guest instruction has completed. For that reasons
> 
> s/has/had/
> s/reasons/reason,/

ack.

Re: [RFCv2 29/37] DOCUMENTATION: protvirt: Diag 308 IPL

[Cornelia Huck]

On Tue, 4 Feb 2020 09:13:06 +0100
Christian Borntraeger <borntraeger@de.ibm.com> wrote:

> On 03.02.20 17:13, Cornelia Huck wrote:
> > On Mon,  3 Feb 2020 08:19:49 -0500
> > Christian Borntraeger <borntraeger@de.ibm.com> wrote:
> >   
> >> From: Janosch Frank <frankja@linux.ibm.com>
> >>
> >> Description of changes that are necessary to move a KVM VM into
> >> Protected Virtualization mode.  
> > 
> > Maybe move this up to the top of the series, so that new reviewers can
> > get a quick idea about the architecture as a whole? It might also make  
> Will do. 
> > sense to make the two documents link to each other...  
> I added both files to the kvm index file and changed the title
> to contain s390. I also added a link to the base doc.

Sounds good.

> 
> >   
> >>
> >> Signed-off-by: Janosch Frank <frankja@linux.ibm.com>
> >> ---
> >>  Documentation/virt/kvm/s390-pv-boot.rst | 64 +++++++++++++++++++++++++
> >>  1 file changed, 64 insertions(+)
> >>  create mode 100644 Documentation/virt/kvm/s390-pv-boot.rst
> >>
> >> diff --git a/Documentation/virt/kvm/s390-pv-boot.rst b/Documentation/virt/kvm/s390-pv-boot.rst
> >> new file mode 100644
> >> index 000000000000..431cd5d7f686
> >> --- /dev/null
> >> +++ b/Documentation/virt/kvm/s390-pv-boot.rst
> >> @@ -0,0 +1,64 @@
> >> +.. SPDX-License-Identifier: GPL-2.0
> >> +=========================
> >> +Boot/IPL of Protected VMs
> >> +=========================  
> > 
> > ...especially as the reader will have no idea what a "Protected VM" is,
> > unless they have read the other document before.  
> 
> > 
> >   
> >> +
> >> +Summary
> >> +-------
> >> +Protected VMs are encrypted while not running. On IPL a small
> >> +plaintext bootloader is started which provides information about the
> >> +encrypted components and necessary metadata to KVM to decrypt it.  
> > 
> > s/it/the PVM/ ?  
> 
> ack
> 
> This section looks now:
> 
> ---
> Protected Virtual Machines (PVM) are not accessible by I/O or the
> hypervisor.  When the hypervisor wants to access the memory of PVMs
> the memory needs to be made accessible. When doing so, the memory will
> be encrypted.  See :doc:`s390-pv` for details.
> 
> On IPL a....

ok

> ---
> 
> >   
> >> +
> >> +Based on this data, KVM will make the PV known to the Ultravisor and  
> > 
> > I think the other document uses 'PVM'... probably better to keep that
> > consistent.  
> 
> The feature name might change to secure execution (SE). I will need to
> go over this again. But I think we can continue to name the virtual
> machines protected virtual machines as this is more a description than
> a brand name.

I would stick with "protected virtual machines" as well; especially as
the kernel parameter is called "prot_virt", and people might wonder
why you're talking about SELinux :)

(...)

> >> +Subcodes 4 and 7 will result in specification exceptions.  
> > 
> > "Subcodes 4 and 7, which would not clear the guest memory, ..." ?  
> 
> Subcodes 4 and 7 will result in specification exceptions as they would
> not clear out the guest memory.

ok

> 
> >   
> >> +When removing a secure VM, the UV will clear all memory, so we can't
> >> +have non-clearing IPL subcodes.
> >> +
> >> +Subcodes 8, 9, 10 will result in specification exceptions.
> >> +Re-IPL into a protected mode is only possible via a detour into non
> >> +protected mode.  
> > 
> > In general, this looks like a good overview about how the guest can
> > move into protected virt mode.
> > 
> > Some information I'm missing in this doc: Where do the keys come from?
> > I assume from the machine... is there one key per CEC? Can keys be
> > transferred? Can an image be introspected to find out if it is possible
> > to run it on a given system?
> > 
> > (Not sure if there is a better resting place for that kind of
> > information.)  
> 
> There will be tooling as part of the s390-tools. I will add 
> 
> ---
> Keys
> ----
> Every CEC will have a unique public key to enable tooling to build
> encrypted images.
> See  `s390-tools <https://github.com/ibm-s390-tools/s390-tools/>`_
> for the tooling.

Sounds good.

> ---
> 
> The s390 tools part is not yet upstream but it will be soon.

I'll look forward to the s390 tools part for details on key handling,
then :)

Re: [RFCv2 02/37] s390/protvirt: introduce host side setup

[Christian Borntraeger]

On 04.02.20 10:28, Cornelia Huck wrote:
> On Mon, 3 Feb 2020 23:03:42 +0100
> Christian Borntraeger <borntraeger@de.ibm.com> wrote:
> 
>> On 03.02.20 18:12, Cornelia Huck wrote:
>>> On Mon,  3 Feb 2020 08:19:22 -0500
>>> Christian Borntraeger <borntraeger@de.ibm.com> wrote:
>>>   
>>>> From: Vasily Gorbik <gor@linux.ibm.com>
>>>>
>>>> Introduce KVM_S390_PROTECTED_VIRTUALIZATION_HOST kbuild option for
>>>> protected virtual machines hosting support code.  
>>>
>>> Hm... I seem to remember that you wanted to drop this config option and
>>> always build the code, in order to reduce complexity. Have you
>>> reconsidered this?  
>>
>> I am still in favour of removing this, but I did not get an "yes, lets do
>> it" answer. Since removing is easier than re-adding its still in.
> 
> ok

Any preference from you?

[...]
> 
> I think I was confused about different things last time...
> 
> But that is probably a sign that this wants a comment :)

Will add

to kernel/uv.c
/* the bootdata_preserved fields come from ones in arch/s390/boot/uv.c */

to boot/uv.c
/* will be used in arch/s390/kernel/uv.c */


[...]
> Fair enough; it's just that it's not very clear from the messages in
> the log what happened. Maybe
> 
> "prot_virt: Running as protected virtualization guest."
> "prot_virt: The ultravisor call facility is not available."
> 
> That at least links back to the kernel parameter.

I will defer this until the end, in the hope to have a final name by then.

Re: [RFCv2 02/37] s390/protvirt: introduce host side setup

[Cornelia Huck]

On Tue, 4 Feb 2020 10:38:55 +0100
Christian Borntraeger <borntraeger@de.ibm.com> wrote:

> On 04.02.20 10:28, Cornelia Huck wrote:
> > On Mon, 3 Feb 2020 23:03:42 +0100
> > Christian Borntraeger <borntraeger@de.ibm.com> wrote:
> >   
> >> On 03.02.20 18:12, Cornelia Huck wrote:  
> >>> On Mon,  3 Feb 2020 08:19:22 -0500
> >>> Christian Borntraeger <borntraeger@de.ibm.com> wrote:
> >>>     
> >>>> From: Vasily Gorbik <gor@linux.ibm.com>
> >>>>
> >>>> Introduce KVM_S390_PROTECTED_VIRTUALIZATION_HOST kbuild option for
> >>>> protected virtual machines hosting support code.    
> >>>
> >>> Hm... I seem to remember that you wanted to drop this config option and
> >>> always build the code, in order to reduce complexity. Have you
> >>> reconsidered this?    
> >>
> >> I am still in favour of removing this, but I did not get an "yes, lets do
> >> it" answer. Since removing is easier than re-adding its still in.  
> > 
> > ok  
> 
> Any preference from you?

Not at the moment, I still need to look at the following patches.

> 
> [...]
> > 
> > I think I was confused about different things last time...
> > 
> > But that is probably a sign that this wants a comment :)  
> 
> Will add
> 
> to kernel/uv.c
> /* the bootdata_preserved fields come from ones in arch/s390/boot/uv.c */
> 
> to boot/uv.c
> /* will be used in arch/s390/kernel/uv.c */

Thanks!

> 
> 
> [...]
> > Fair enough; it's just that it's not very clear from the messages in
> > the log what happened. Maybe
> > 
> > "prot_virt: Running as protected virtualization guest."
> > "prot_virt: The ultravisor call facility is not available."
> > 
> > That at least links back to the kernel parameter.  
> 
> I will defer this until the end, in the hope to have a final name by then.
> 
ok

Re: [RFCv2 03/37] s390/protvirt: add ultravisor initialization

[Thomas Huth]

On 03/02/2020 14.19, Christian Borntraeger wrote:
> From: Vasily Gorbik <gor@linux.ibm.com>
> 
> Before being able to host protected virtual machines, donate some of
> the memory to the ultravisor. Besides that the ultravisor might impose
> addressing limitations for memory used to back protected VM storage. Treat
> that limit as protected virtualization host's virtual memory limit.
> 
> Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
> Reviewed-by: Christian Borntraeger <borntraeger@de.ibm.com>
> ---
>  arch/s390/include/asm/uv.h | 16 ++++++++++++
>  arch/s390/kernel/setup.c   |  3 +++
>  arch/s390/kernel/uv.c      | 53 ++++++++++++++++++++++++++++++++++++++
>  3 files changed, 72 insertions(+)

Reviewed-by: Thomas Huth <thuth@redhat.com>

Re: [RFCv2 03/37] s390/protvirt: add ultravisor initialization

[Cornelia Huck]

On Mon,  3 Feb 2020 08:19:23 -0500
Christian Borntraeger <borntraeger@de.ibm.com> wrote:

> From: Vasily Gorbik <gor@linux.ibm.com>
> 
> Before being able to host protected virtual machines, donate some of
> the memory to the ultravisor. Besides that the ultravisor might impose
> addressing limitations for memory used to back protected VM storage. Treat
> that limit as protected virtualization host's virtual memory limit.
> 
> Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
> Reviewed-by: Christian Borntraeger <borntraeger@de.ibm.com>
> ---
>  arch/s390/include/asm/uv.h | 16 ++++++++++++
>  arch/s390/kernel/setup.c   |  3 +++
>  arch/s390/kernel/uv.c      | 53 ++++++++++++++++++++++++++++++++++++++
>  3 files changed, 72 insertions(+)
> 
> diff --git a/arch/s390/include/asm/uv.h b/arch/s390/include/asm/uv.h
> index 32eac3ab2d3b..cdf2fd71d7ab 100644
> --- a/arch/s390/include/asm/uv.h
> +++ b/arch/s390/include/asm/uv.h

(...)

> @@ -59,6 +61,15 @@ struct uv_cb_qui {
>  	u64 reserved98;
>  } __packed __aligned(8);
>  
> +struct uv_cb_init {
> +	struct uv_cb_header header;
> +	u64 reserved08[2];
> +	u64 stor_origin;
> +	u64 stor_len;
> +	u64 reserved28[4];
> +

Maybe drop the blank line?

> +} __packed __aligned(8);
> +
>  struct uv_cb_share {
>  	struct uv_cb_header header;
>  	u64 reserved08[3];

(...)

Reviewed-by: Cornelia Huck <cohuck@redhat.com>

Re: [RFCv2 06/37] s390: add (non)secure page access exceptions handlers

[Thomas Huth]

On 03/02/2020 14.19, Christian Borntraeger wrote:
> From: Vasily Gorbik <gor@linux.ibm.com>
> 
> Add exceptions handlers performing transparent transition of non-secure
> pages to secure (import) upon guest access and secure pages to
> non-secure (export) upon hypervisor access.
> 
> Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
> Signed-off-by: Janosch Frank <frankja@linux.ibm.com>
> [adding checks for failures]
> Signed-off-by: Claudio Imbrenda <imbrenda@linux.ibm.com>
> [further changes like adding a check for gmap fault]

Who did which modification here? According to
Documentation/process/submitting-patches.rst the square brackets should
go in front of the S-o-b of the person who did the change, e.g.:

  Signed-off-by: Random J Developer <random@developer.example.org>
  [lucky@maintainer.example.org: struct foo moved from foo.c to foo.h]
  Signed-off-by: Lucky K Maintainer <lucky@maintainer.example.org>

It would be nice if you could stick to that scheme.

> ---
>  arch/s390/kernel/pgm_check.S |  4 +-
>  arch/s390/mm/fault.c         | 87 ++++++++++++++++++++++++++++++++++++
>  2 files changed, 89 insertions(+), 2 deletions(-)
[...]
> +void do_non_secure_storage_access(struct pt_regs *regs)
> +{
> +	unsigned long gaddr = regs->int_parm_long & __FAIL_ADDR_MASK;
> +	struct gmap *gmap = (struct gmap *)S390_lowcore.gmap;
> +	struct uv_cb_cts uvcb = {
> +		.header.cmd = UVC_CMD_CONV_TO_SEC_STOR,
> +		.header.len = sizeof(uvcb),
> +		.guest_handle = gmap->se_handle,
> +		.gaddr = gaddr,
> +	};
> +	int rc;
> +
> +	if (get_fault_type(regs) != GMAP_FAULT) {
> +		do_fault_error(regs, VM_READ | VM_WRITE, VM_FAULT_BADMAP);
> +		WARN_ON_ONCE(1);
> +		return;
> +	}
> +
> +	rc = uv_make_secure(gmap, gaddr, &uvcb, 0);
> +	if (rc == -EINVAL && uvcb.header.rc != 0x104)
> +		send_sig(SIGSEGV, current, 0);
> +}

What about the other rc beside 0x104 that could happen here? They go
unnoticed?

 Thomas

Re: [RFCv2 05/37] s390/mm: provide memory management functions for protected KVM guests

[Cornelia Huck]

On Mon,  3 Feb 2020 08:19:25 -0500
Christian Borntraeger <borntraeger@de.ibm.com> wrote:

> From: Claudio Imbrenda <imbrenda@linux.ibm.com>
> 
> This provides the basic ultravisor calls and page table handling to cope
> with secure guests.

I'll leave reviewing the mm stuff to somebody actually familiar with
mm; only some other comments from me.

> 
> Co-authored-by: Ulrich Weigand <Ulrich.Weigand@de.ibm.com>
> Signed-off-by: Claudio Imbrenda <imbrenda@linux.ibm.com>
> ---
>  arch/s390/include/asm/gmap.h        |   2 +
>  arch/s390/include/asm/mmu.h         |   2 +
>  arch/s390/include/asm/mmu_context.h |   1 +
>  arch/s390/include/asm/page.h        |   5 +
>  arch/s390/include/asm/pgtable.h     |  34 +++++-
>  arch/s390/include/asm/uv.h          |  59 ++++++++++
>  arch/s390/kernel/uv.c               | 170 ++++++++++++++++++++++++++++
>  7 files changed, 268 insertions(+), 5 deletions(-)
> 
> diff --git a/arch/s390/include/asm/gmap.h b/arch/s390/include/asm/gmap.h
> index 37f96b6f0e61..f2ab8b6d4b57 100644
> --- a/arch/s390/include/asm/gmap.h
> +++ b/arch/s390/include/asm/gmap.h
> @@ -9,6 +9,7 @@
>  #ifndef _ASM_S390_GMAP_H
>  #define _ASM_S390_GMAP_H
>  
> +#include <linux/radix-tree.h>
>  #include <linux/refcount.h>
>  
>  /* Generic bits for GMAP notification on DAT table entry changes. */
> @@ -61,6 +62,7 @@ struct gmap {
>  	spinlock_t shadow_lock;
>  	struct gmap *parent;
>  	unsigned long orig_asce;
> +	unsigned long se_handle;

This is a deviation from the "protected virtualization" naming scheme
used in the previous patches, but I understand that the naming of this
whole feature is still in flux :) (Would still be nice to have it
consistent, though.)

However, I think I'd prefer something named based on protected
virtualization: the se_* stuff here just tends to make me think of
SELinux...

>  	int edat_level;
>  	bool removed;
>  	bool initialized;
> diff --git a/arch/s390/include/asm/mmu.h b/arch/s390/include/asm/mmu.h
> index bcfb6371086f..984026cb3608 100644
> --- a/arch/s390/include/asm/mmu.h
> +++ b/arch/s390/include/asm/mmu.h
> @@ -16,6 +16,8 @@ typedef struct {
>  	unsigned long asce;
>  	unsigned long asce_limit;
>  	unsigned long vdso_base;
> +	/* The mmu context belongs to a secure guest. */
> +	atomic_t is_se;

Here as well.

>  	/*
>  	 * The following bitfields need a down_write on the mm
>  	 * semaphore when they are written to. As they are only

(...)

> @@ -520,6 +521,15 @@ static inline int mm_has_pgste(struct mm_struct *mm)
>  	return 0;
>  }
>  
> +static inline int mm_is_se(struct mm_struct *mm)
> +{
> +#ifdef CONFIG_KVM_S390_PROTECTED_VIRTUALIZATION_HOST
> +	if (unlikely(atomic_read(&mm->context.is_se)))
> +		return 1;
> +#endif
> +	return 0;
> +}

I'm wondering how big of an overhead we actually have because we need
to check the flag here if the feature is enabled. We have an extra
check in a few functions anyway, even if protected virt is not
configured in. Given that distributions would likely want to enable the
feature in their kernels, I'm currently tending towards dropping the
extra config option.

> +
>  static inline int mm_alloc_pgste(struct mm_struct *mm)
>  {
>  #ifdef CONFIG_PGSTE

(...)

> diff --git a/arch/s390/kernel/uv.c b/arch/s390/kernel/uv.c
> index f7778493e829..136c60a8e3ca 100644
> --- a/arch/s390/kernel/uv.c
> +++ b/arch/s390/kernel/uv.c
> @@ -9,6 +9,8 @@
>  #include <linux/sizes.h>
>  #include <linux/bitmap.h>
>  #include <linux/memblock.h>
> +#include <linux/pagemap.h>
> +#include <linux/swap.h>
>  #include <asm/facility.h>
>  #include <asm/sections.h>
>  #include <asm/uv.h>
> @@ -98,4 +100,172 @@ void adjust_to_uv_max(unsigned long *vmax)
>  	if (prot_virt_host && *vmax > uv_info.max_sec_stor_addr)
>  		*vmax = uv_info.max_sec_stor_addr;
>  }
> +
> +static int __uv_pin_shared(unsigned long paddr)
> +{
> +	struct uv_cb_cfs uvcb = {
> +		.header.cmd	= UVC_CMD_PIN_PAGE_SHARED,
> +		.header.len	= sizeof(uvcb),
> +		.paddr		= paddr,
> +	};
> +
> +	if (uv_call(0, (u64)&uvcb))
> +		return -EINVAL;
> +	return 0;

I guess this call won't set an error rc in the control block, if it
does not set a condition code != 0?

(...)

Re: [RFCv2 06/37] s390: add (non)secure page access exceptions handlers

[Cornelia Huck]

On Mon,  3 Feb 2020 08:19:26 -0500
Christian Borntraeger <borntraeger@de.ibm.com> wrote:

> From: Vasily Gorbik <gor@linux.ibm.com>
> 
> Add exceptions handlers performing transparent transition of non-secure
> pages to secure (import) upon guest access and secure pages to
> non-secure (export) upon hypervisor access.
> 
> Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
> Signed-off-by: Janosch Frank <frankja@linux.ibm.com>
> [adding checks for failures]
> Signed-off-by: Claudio Imbrenda <imbrenda@linux.ibm.com>
> [further changes like adding a check for gmap fault]
> ---
>  arch/s390/kernel/pgm_check.S |  4 +-
>  arch/s390/mm/fault.c         | 87 ++++++++++++++++++++++++++++++++++++
>  2 files changed, 89 insertions(+), 2 deletions(-)
> 
> diff --git a/arch/s390/kernel/pgm_check.S b/arch/s390/kernel/pgm_check.S
> index 59dee9d3bebf..27ac4f324c70 100644
> --- a/arch/s390/kernel/pgm_check.S
> +++ b/arch/s390/kernel/pgm_check.S
> @@ -78,8 +78,8 @@ PGM_CHECK(do_dat_exception)		/* 39 */
>  PGM_CHECK(do_dat_exception)		/* 3a */
>  PGM_CHECK(do_dat_exception)		/* 3b */
>  PGM_CHECK_DEFAULT			/* 3c */
> -PGM_CHECK_DEFAULT			/* 3d */
> -PGM_CHECK_DEFAULT			/* 3e */
> +PGM_CHECK(do_secure_storage_access)	/* 3d */
> +PGM_CHECK(do_non_secure_storage_access)	/* 3e */

I suppose that these two can only happen when we actually run a
protected virt guest...

>  PGM_CHECK_DEFAULT			/* 3f */
>  PGM_CHECK_DEFAULT			/* 40 */
>  PGM_CHECK_DEFAULT			/* 41 */
> diff --git a/arch/s390/mm/fault.c b/arch/s390/mm/fault.c
> index 7b0bb475c166..bd75b0765cf1 100644
> --- a/arch/s390/mm/fault.c
> +++ b/arch/s390/mm/fault.c
> @@ -38,6 +38,7 @@
>  #include <asm/irq.h>
>  #include <asm/mmu_context.h>
>  #include <asm/facility.h>
> +#include <asm/uv.h>
>  #include "../kernel/entry.h"
>  
>  #define __FAIL_ADDR_MASK -4096L
> @@ -816,3 +817,89 @@ static int __init pfault_irq_init(void)
>  early_initcall(pfault_irq_init);
>  
>  #endif /* CONFIG_PFAULT */
> +
> +#ifdef CONFIG_KVM_S390_PROTECTED_VIRTUALIZATION_HOST
> +
> +void do_secure_storage_access(struct pt_regs *regs)
> +{
> +	unsigned long addr = regs->int_parm_long & __FAIL_ADDR_MASK;
> +	struct vm_area_struct *vma;
> +	struct mm_struct *mm;
> +	struct page *page;
> +	int rc;

...so this rather complex function will never be called if we don't act
as a host for protected virt guests?

> +
> +	switch (get_fault_type(regs)) {
> +	case USER_FAULT:
> +		mm = current->mm;
> +		down_read(&mm->mmap_sem);
> +		vma = find_vma(mm, addr);
> +		if (!vma) {
> +			up_read(&mm->mmap_sem);
> +			do_fault_error(regs, VM_READ | VM_WRITE, VM_FAULT_BADMAP);
> +			break;
> +		}
> +		page = follow_page(vma, addr, FOLL_WRITE | FOLL_GET);
> +		if (IS_ERR_OR_NULL(page)) {
> +			up_read(&mm->mmap_sem);
> +			break;
> +		}
> +		if (arch_make_page_accessible(page))
> +			send_sig(SIGSEGV, current, 0);
> +		put_page(page);
> +		up_read(&mm->mmap_sem);
> +		break;
> +	case KERNEL_FAULT:
> +		page = phys_to_page(addr);
> +		if (unlikely(!try_get_page(page)))
> +			break;
> +		rc = arch_make_page_accessible(page);
> +		put_page(page);
> +		if (rc)
> +			BUG();
> +		break;
> +	case VDSO_FAULT:
> +		/* fallthrough */
> +	case GMAP_FAULT:
> +		/* fallthrough */
> +	default:
> +		do_fault_error(regs, VM_READ | VM_WRITE, VM_FAULT_BADMAP);
> +		WARN_ON_ONCE(1);
> +	}
> +}
> +NOKPROBE_SYMBOL(do_secure_storage_access);
> +
> +void do_non_secure_storage_access(struct pt_regs *regs)
> +{
> +	unsigned long gaddr = regs->int_parm_long & __FAIL_ADDR_MASK;
> +	struct gmap *gmap = (struct gmap *)S390_lowcore.gmap;
> +	struct uv_cb_cts uvcb = {
> +		.header.cmd = UVC_CMD_CONV_TO_SEC_STOR,
> +		.header.len = sizeof(uvcb),
> +		.guest_handle = gmap->se_handle,
> +		.gaddr = gaddr,
> +	};
> +	int rc;

Same for this function, of course.

> +
> +	if (get_fault_type(regs) != GMAP_FAULT) {
> +		do_fault_error(regs, VM_READ | VM_WRITE, VM_FAULT_BADMAP);
> +		WARN_ON_ONCE(1);
> +		return;
> +	}
> +
> +	rc = uv_make_secure(gmap, gaddr, &uvcb, 0);
> +	if (rc == -EINVAL && uvcb.header.rc != 0x104)
> +		send_sig(SIGSEGV, current, 0);
> +}
> +NOKPROBE_SYMBOL(do_non_secure_storage_access);
> +
> +#else
> +void do_secure_storage_access(struct pt_regs *regs)
> +{
> +	default_trap_handler(regs);
> +}
> +
> +void do_non_secure_storage_access(struct pt_regs *regs)
> +{
> +	default_trap_handler(regs);
> +}

And these should not really be called at all, I guess?

> +#endif

IOW, we don't really introduce complex code paths for systems not
acting as protected virt hosts, even if we switch on the config option
or ditch it completely?

Re: [RFCv2 03/37] s390/protvirt: add ultravisor initialization

[Christian Borntraeger]

On 04.02.20 11:23, Cornelia Huck wrote:
> On Mon,  3 Feb 2020 08:19:23 -0500
> Christian Borntraeger <borntraeger@de.ibm.com> wrote:
> 
>> From: Vasily Gorbik <gor@linux.ibm.com>
>>
>> Before being able to host protected virtual machines, donate some of
>> the memory to the ultravisor. Besides that the ultravisor might impose
>> addressing limitations for memory used to back protected VM storage. Treat
>> that limit as protected virtualization host's virtual memory limit.
>>
>> Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
>> Reviewed-by: Christian Borntraeger <borntraeger@de.ibm.com>
>> ---
>>  arch/s390/include/asm/uv.h | 16 ++++++++++++
>>  arch/s390/kernel/setup.c   |  3 +++
>>  arch/s390/kernel/uv.c      | 53 ++++++++++++++++++++++++++++++++++++++
>>  3 files changed, 72 insertions(+)
>>
>> diff --git a/arch/s390/include/asm/uv.h b/arch/s390/include/asm/uv.h
>> index 32eac3ab2d3b..cdf2fd71d7ab 100644
>> --- a/arch/s390/include/asm/uv.h
>> +++ b/arch/s390/include/asm/uv.h
> 
> (...)
> 
>> @@ -59,6 +61,15 @@ struct uv_cb_qui {
>>  	u64 reserved98;
>>  } __packed __aligned(8);
>>  
>> +struct uv_cb_init {
>> +	struct uv_cb_header header;
>> +	u64 reserved08[2];
>> +	u64 stor_origin;
>> +	u64 stor_len;
>> +	u64 reserved28[4];
>> +
> 
> Maybe drop the blank line?

ack

> 
>> +} __packed __aligned(8);
>> +
>>  struct uv_cb_share {
>>  	struct uv_cb_header header;
>>  	u64 reserved08[3];
> 
> (...)
> 
> Reviewed-by: Cornelia Huck <cohuck@redhat.com>
> 

Re: [RFCv2 06/37] s390: add (non)secure page access exceptions handlers

[Claudio Imbrenda]

On Tue, 4 Feb 2020 11:37:42 +0100
Thomas Huth <thuth@redhat.com> wrote:

[...]

> > ---
> >  arch/s390/kernel/pgm_check.S |  4 +-
> >  arch/s390/mm/fault.c         | 87
> > ++++++++++++++++++++++++++++++++++++ 2 files changed, 89
> > insertions(+), 2 deletions(-)  
> [...]
> > +void do_non_secure_storage_access(struct pt_regs *regs)
> > +{
> > +	unsigned long gaddr = regs->int_parm_long &
> > __FAIL_ADDR_MASK;
> > +	struct gmap *gmap = (struct gmap *)S390_lowcore.gmap;
> > +	struct uv_cb_cts uvcb = {
> > +		.header.cmd = UVC_CMD_CONV_TO_SEC_STOR,
> > +		.header.len = sizeof(uvcb),
> > +		.guest_handle = gmap->se_handle,
> > +		.gaddr = gaddr,
> > +	};
> > +	int rc;
> > +
> > +	if (get_fault_type(regs) != GMAP_FAULT) {
> > +		do_fault_error(regs, VM_READ | VM_WRITE,
> > VM_FAULT_BADMAP);
> > +		WARN_ON_ONCE(1);
> > +		return;
> > +	}
> > +
> > +	rc = uv_make_secure(gmap, gaddr, &uvcb, 0);
> > +	if (rc == -EINVAL && uvcb.header.rc != 0x104)
> > +		send_sig(SIGSEGV, current, 0);
> > +}  
> 
> What about the other rc beside 0x104 that could happen here? They go
> unnoticed?

no, they are handled in the uv_make_secure, and return an appropriate
error code. 104 in this context still indicates that things were already
how we wanted, so we consider it a success here. e.g. for unpack it is
not considered a success. that's why we check for -EINVAL (which is the
generic error) and then check for the specific rc


a cleaner solution perhaps would have been to split uv_make_secure to
handle rc 0x104 differently, but I didn't want to duplicate the code
here.


Claudio

Re: [RFCv2 05/37] s390/mm: provide memory management functions for protected KVM guests

[Claudio Imbrenda]

On Tue, 4 Feb 2020 11:57:38 +0100
Cornelia Huck <cohuck@redhat.com> wrote:

> On Mon,  3 Feb 2020 08:19:25 -0500
> Christian Borntraeger <borntraeger@de.ibm.com> wrote:
> 
> > From: Claudio Imbrenda <imbrenda@linux.ibm.com>
> > 
> > This provides the basic ultravisor calls and page table handling to
> > cope with secure guests.  
> 
> I'll leave reviewing the mm stuff to somebody actually familiar with
> mm; only some other comments from me.
> 
> > 
> > Co-authored-by: Ulrich Weigand <Ulrich.Weigand@de.ibm.com>
> > Signed-off-by: Claudio Imbrenda <imbrenda@linux.ibm.com>
> > ---
> >  arch/s390/include/asm/gmap.h        |   2 +
> >  arch/s390/include/asm/mmu.h         |   2 +
> >  arch/s390/include/asm/mmu_context.h |   1 +
> >  arch/s390/include/asm/page.h        |   5 +
> >  arch/s390/include/asm/pgtable.h     |  34 +++++-
> >  arch/s390/include/asm/uv.h          |  59 ++++++++++
> >  arch/s390/kernel/uv.c               | 170
> > ++++++++++++++++++++++++++++ 7 files changed, 268 insertions(+), 5
> > deletions(-)
> > 
> > diff --git a/arch/s390/include/asm/gmap.h
> > b/arch/s390/include/asm/gmap.h index 37f96b6f0e61..f2ab8b6d4b57
> > 100644 --- a/arch/s390/include/asm/gmap.h
> > +++ b/arch/s390/include/asm/gmap.h
> > @@ -9,6 +9,7 @@
> >  #ifndef _ASM_S390_GMAP_H
> >  #define _ASM_S390_GMAP_H
> >  
> > +#include <linux/radix-tree.h>
> >  #include <linux/refcount.h>
> >  
> >  /* Generic bits for GMAP notification on DAT table entry changes.
> > */ @@ -61,6 +62,7 @@ struct gmap {
> >  	spinlock_t shadow_lock;
> >  	struct gmap *parent;
> >  	unsigned long orig_asce;
> > +	unsigned long se_handle;  
> 
> This is a deviation from the "protected virtualization" naming scheme
> used in the previous patches, but I understand that the naming of this
> whole feature is still in flux :) (Would still be nice to have it
> consistent, though.)
> 
> However, I think I'd prefer something named based on protected
> virtualization: the se_* stuff here just tends to make me think of
> SELinux...
> 
> >  	int edat_level;
> >  	bool removed;
> >  	bool initialized;
> > diff --git a/arch/s390/include/asm/mmu.h
> > b/arch/s390/include/asm/mmu.h index bcfb6371086f..984026cb3608
> > 100644 --- a/arch/s390/include/asm/mmu.h
> > +++ b/arch/s390/include/asm/mmu.h
> > @@ -16,6 +16,8 @@ typedef struct {
> >  	unsigned long asce;
> >  	unsigned long asce_limit;
> >  	unsigned long vdso_base;
> > +	/* The mmu context belongs to a secure guest. */
> > +	atomic_t is_se;  
> 
> Here as well.
> 
> >  	/*
> >  	 * The following bitfields need a down_write on the mm
> >  	 * semaphore when they are written to. As they are only  
> 
> (...)
> 
> > @@ -520,6 +521,15 @@ static inline int mm_has_pgste(struct
> > mm_struct *mm) return 0;
> >  }
> >  
> > +static inline int mm_is_se(struct mm_struct *mm)
> > +{
> > +#ifdef CONFIG_KVM_S390_PROTECTED_VIRTUALIZATION_HOST
> > +	if (unlikely(atomic_read(&mm->context.is_se)))
> > +		return 1;
> > +#endif
> > +	return 0;
> > +}  
> 
> I'm wondering how big of an overhead we actually have because we need
> to check the flag here if the feature is enabled. We have an extra
> check in a few functions anyway, even if protected virt is not
> configured in. Given that distributions would likely want to enable
> the feature in their kernels, I'm currently tending towards dropping
> the extra config option.
> 
> > +
> >  static inline int mm_alloc_pgste(struct mm_struct *mm)
> >  {
> >  #ifdef CONFIG_PGSTE  
> 
> (...)
> 
> > diff --git a/arch/s390/kernel/uv.c b/arch/s390/kernel/uv.c
> > index f7778493e829..136c60a8e3ca 100644
> > --- a/arch/s390/kernel/uv.c
> > +++ b/arch/s390/kernel/uv.c
> > @@ -9,6 +9,8 @@
> >  #include <linux/sizes.h>
> >  #include <linux/bitmap.h>
> >  #include <linux/memblock.h>
> > +#include <linux/pagemap.h>
> > +#include <linux/swap.h>
> >  #include <asm/facility.h>
> >  #include <asm/sections.h>
> >  #include <asm/uv.h>
> > @@ -98,4 +100,172 @@ void adjust_to_uv_max(unsigned long *vmax)
> >  	if (prot_virt_host && *vmax > uv_info.max_sec_stor_addr)
> >  		*vmax = uv_info.max_sec_stor_addr;
> >  }
> > +
> > +static int __uv_pin_shared(unsigned long paddr)
> > +{
> > +	struct uv_cb_cfs uvcb = {
> > +		.header.cmd	= UVC_CMD_PIN_PAGE_SHARED,
> > +		.header.len	= sizeof(uvcb),
> > +		.paddr		= paddr,
> > +	};
> > +
> > +	if (uv_call(0, (u64)&uvcb))
> > +		return -EINVAL;
> > +	return 0;  
> 
> I guess this call won't set an error rc in the control block, if it
> does not set a condition code != 0?

correct.
(cc == 0) ⇔ (rc == 1) which is success.
any rc other than 1 sets cc = 1

Re: [RFCv2 05/37] s390/mm: provide memory management functions for protected KVM guests

[Christian Borntraeger]

On 04.02.20 11:57, Cornelia Huck wrote:

>>  
>>  /* Generic bits for GMAP notification on DAT table entry changes. */
>> @@ -61,6 +62,7 @@ struct gmap {
>>  	spinlock_t shadow_lock;
>>  	struct gmap *parent;
>>  	unsigned long orig_asce;
>> +	unsigned long se_handle;
> 
> This is a deviation from the "protected virtualization" naming scheme
> used in the previous patches, but I understand that the naming of this
> whole feature is still in flux :) (Would still be nice to have it
> consistent, though.)
> 
> However, I think I'd prefer something named based on protected
> virtualization: the se_* stuff here just tends to make me think of
> SELinux...

I will just use guest_handle as this guests assigned to and from such:

arch/s390/include/asm/gmap.h:   unsigned long se_handle;
arch/s390/include/asm/uv.h:             .guest_handle = gmap->se_handle,
arch/s390/kvm/pv.c:     WRITE_ONCE(kvm->arch.gmap->se_handle, 0);
arch/s390/kvm/pv.c:     kvm->arch.gmap->se_handle = uvcb.guest_handle;
arch/s390/mm/fault.c:           .guest_handle = gmap->se_handle,


> 
>>  	int edat_level;
>>  	bool removed;
>>  	bool initialized;
>> diff --git a/arch/s390/include/asm/mmu.h b/arch/s390/include/asm/mmu.h
>> index bcfb6371086f..984026cb3608 100644
>> --- a/arch/s390/include/asm/mmu.h
>> +++ b/arch/s390/include/asm/mmu.h
>> @@ -16,6 +16,8 @@ typedef struct {
>>  	unsigned long asce;
>>  	unsigned long asce_limit;
>>  	unsigned long vdso_base;
>> +	/* The mmu context belongs to a secure guest. */
>> +	atomic_t is_se;
> 
> Here as well.

I will use is "is_protected"
> 
>>  	/*
>>  	 * The following bitfields need a down_write on the mm
>>  	 * semaphore when they are written to. As they are only
> 
> (...)
> 
>> @@ -520,6 +521,15 @@ static inline int mm_has_pgste(struct mm_struct *mm)
>>  	return 0;
>>  }
>>  
>> +static inline int mm_is_se(struct mm_struct *mm)
>> +{
>> +#ifdef CONFIG_KVM_S390_PROTECTED_VIRTUALIZATION_HOST
>> +	if (unlikely(atomic_read(&mm->context.is_se)))
>> +		return 1;
>> +#endif
>> +	return 0;
>> +}
> 
> I'm wondering how big of an overhead we actually have because we need
> to check the flag here if the feature is enabled. We have an extra
> check in a few functions anyway, even if protected virt is not
> configured in. Given that distributions would likely want to enable the
> feature in their kernels, I'm currently tending towards dropping the
> extra config option.

Makes sense. I will wait a bit more but if you do not disagree I will remove 
the extra host config and bind things to CONFIG_KVM or CONFIG_PGSTE depending
on context

> 
>> +
>>  static inline int mm_alloc_pgste(struct mm_struct *mm)
>>  {
>>  #ifdef CONFIG_PGSTE
> 
> (...)
> 
>> diff --git a/arch/s390/kernel/uv.c b/arch/s390/kernel/uv.c
>> index f7778493e829..136c60a8e3ca 100644
>> --- a/arch/s390/kernel/uv.c
>> +++ b/arch/s390/kernel/uv.c
>> @@ -9,6 +9,8 @@
>>  #include <linux/sizes.h>
>>  #include <linux/bitmap.h>
>>  #include <linux/memblock.h>
>> +#include <linux/pagemap.h>
>> +#include <linux/swap.h>
>>  #include <asm/facility.h>
>>  #include <asm/sections.h>
>>  #include <asm/uv.h>
>> @@ -98,4 +100,172 @@ void adjust_to_uv_max(unsigned long *vmax)
>>  	if (prot_virt_host && *vmax > uv_info.max_sec_stor_addr)
>>  		*vmax = uv_info.max_sec_stor_addr;
>>  }
>> +
>> +static int __uv_pin_shared(unsigned long paddr)
>> +{
>> +	struct uv_cb_cfs uvcb = {
>> +		.header.cmd	= UVC_CMD_PIN_PAGE_SHARED,
>> +		.header.len	= sizeof(uvcb),
>> +		.paddr		= paddr,
>> +	};
>> +
>> +	if (uv_call(0, (u64)&uvcb))
>> +		return -EINVAL;
>> +	return 0;
> 
> I guess this call won't set an error rc in the control block, if it
> does not set a condition code != 0?

Right. RC is only set for CC!=0

Re: [RFCv2 06/37] s390: add (non)secure page access exceptions handlers

[Christian Borntraeger]

On 04.02.20 11:37, Thomas Huth wrote:
> On 03/02/2020 14.19, Christian Borntraeger wrote:
>> From: Vasily Gorbik <gor@linux.ibm.com>
>>
>> Add exceptions handlers performing transparent transition of non-secure
>> pages to secure (import) upon guest access and secure pages to
>> non-secure (export) upon hypervisor access.
>>
>> Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
>> Signed-off-by: Janosch Frank <frankja@linux.ibm.com>
>> [adding checks for failures]
>> Signed-off-by: Claudio Imbrenda <imbrenda@linux.ibm.com>
>> [further changes like adding a check for gmap fault]
> 
> Who did which modification here? According to
> Documentation/process/submitting-patches.rst the square brackets should
> go in front of the S-o-b of the person who did the change, e.g.:
> 
>   Signed-off-by: Random J Developer <random@developer.example.org>
>   [lucky@maintainer.example.org: struct foo moved from foo.c to foo.h]
>   Signed-off-by: Lucky K Maintainer <lucky@maintainer.example.org>
> 
> It would be nice if you could stick to that scheme.

will fix

> 
>> ---
>>  arch/s390/kernel/pgm_check.S |  4 +-
>>  arch/s390/mm/fault.c         | 87 ++++++++++++++++++++++++++++++++++++
>>  2 files changed, 89 insertions(+), 2 deletions(-)
> [...]
>> +void do_non_secure_storage_access(struct pt_regs *regs)
>> +{
>> +	unsigned long gaddr = regs->int_parm_long & __FAIL_ADDR_MASK;
>> +	struct gmap *gmap = (struct gmap *)S390_lowcore.gmap;
>> +	struct uv_cb_cts uvcb = {
>> +		.header.cmd = UVC_CMD_CONV_TO_SEC_STOR,
>> +		.header.len = sizeof(uvcb),
>> +		.guest_handle = gmap->se_handle,
>> +		.gaddr = gaddr,
>> +	};
>> +	int rc;
>> +
>> +	if (get_fault_type(regs) != GMAP_FAULT) {
>> +		do_fault_error(regs, VM_READ | VM_WRITE, VM_FAULT_BADMAP);
>> +		WARN_ON_ONCE(1);
>> +		return;
>> +	}
>> +
>> +	rc = uv_make_secure(gmap, gaddr, &uvcb, 0);
>> +	if (rc == -EINVAL && uvcb.header.rc != 0x104)
>> +		send_sig(SIGSEGV, current, 0);
>> +}
> 
> What about the other rc beside 0x104 that could happen here? They go
> unnoticed?
> 
>  Thomas
> 

Re: [RFCv2 06/37] s390: add (non)secure page access exceptions handlers

[Christian Borntraeger]

On 04.02.20 12:18, Cornelia Huck wrote:
> On Mon,  3 Feb 2020 08:19:26 -0500
> Christian Borntraeger <borntraeger@de.ibm.com> wrote:
> 
>> From: Vasily Gorbik <gor@linux.ibm.com>
>>
>> Add exceptions handlers performing transparent transition of non-secure
>> pages to secure (import) upon guest access and secure pages to
>> non-secure (export) upon hypervisor access.
>>
>> Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
>> Signed-off-by: Janosch Frank <frankja@linux.ibm.com>
>> [adding checks for failures]
>> Signed-off-by: Claudio Imbrenda <imbrenda@linux.ibm.com>
>> [further changes like adding a check for gmap fault]
>> ---
>>  arch/s390/kernel/pgm_check.S |  4 +-
>>  arch/s390/mm/fault.c         | 87 ++++++++++++++++++++++++++++++++++++
>>  2 files changed, 89 insertions(+), 2 deletions(-)
>>
>> diff --git a/arch/s390/kernel/pgm_check.S b/arch/s390/kernel/pgm_check.S
>> index 59dee9d3bebf..27ac4f324c70 100644
>> --- a/arch/s390/kernel/pgm_check.S
>> +++ b/arch/s390/kernel/pgm_check.S
>> @@ -78,8 +78,8 @@ PGM_CHECK(do_dat_exception)		/* 39 */
>>  PGM_CHECK(do_dat_exception)		/* 3a */
>>  PGM_CHECK(do_dat_exception)		/* 3b */
>>  PGM_CHECK_DEFAULT			/* 3c */
>> -PGM_CHECK_DEFAULT			/* 3d */
>> -PGM_CHECK_DEFAULT			/* 3e */
>> +PGM_CHECK(do_secure_storage_access)	/* 3d */
>> +PGM_CHECK(do_non_secure_storage_access)	/* 3e */
> 
> I suppose that these two can only happen when we actually run a
> protected virt guest...

yes to this and all other questions.
[...]

> IOW, we don't really introduce complex code paths for systems not
> acting as protected virt hosts, even if we switch on the config option
> or ditch it completely?

Yes.

Re: [RFCv2 07/37] KVM: s390: add new variants of UV CALL

[Cornelia Huck]

On Mon,  3 Feb 2020 08:19:27 -0500
Christian Borntraeger <borntraeger@de.ibm.com> wrote:

> From: Janosch Frank <frankja@linux.ibm.com>
> 
> This add 2 new variants of the UV CALL.
> 
> The first variant handles UV CALLs that might have longer busy
> conditions or just need longer when doing partial completion. We should
> schedule when necessary.
> 
> The second variant handles UV CALLs that only need the handle but have
> no payload (e.g. destroying a VM). We can provide a simple wrapper for
> those.
> 
> Signed-off-by: Janosch Frank <frankja@linux.ibm.com>
> ---
>  arch/s390/include/asm/uv.h | 58 ++++++++++++++++++++++++++++++++++++++
>  1 file changed, 58 insertions(+)
> 
> diff --git a/arch/s390/include/asm/uv.h b/arch/s390/include/asm/uv.h
> index 4eaea95f5c64..3448f12ef57a 100644
> --- a/arch/s390/include/asm/uv.h
> +++ b/arch/s390/include/asm/uv.h
> @@ -14,6 +14,7 @@
>  #include <linux/types.h>
>  #include <linux/errno.h>
>  #include <linux/bug.h>
> +#include <linux/sched.h>
>  #include <asm/page.h>
>  #include <asm/gmap.h>
>  
> @@ -92,6 +93,18 @@ struct uv_cb_cfs {
>  	u64 paddr;
>  } __packed __aligned(8);
>  
> +/*
> + * A common UV call struct for the following calls:

"for calls that take no payload"?

In case there will be more of them in the future :)

> + * Destroy cpu/config
> + * Verify
> + */
> +struct uv_cb_nodata {
> +	struct uv_cb_header header;
> +	u64 reserved08[2];
> +	u64 handle;
> +	u64 reserved20[4];
> +} __packed __aligned(8);
> +
>  struct uv_cb_share {
>  	struct uv_cb_header header;
>  	u64 reserved08[3];
> @@ -99,6 +112,31 @@ struct uv_cb_share {
>  	u64 reserved28;
>  } __packed __aligned(8);
>  
> +/*
> + * Low level uv_call that takes r1 and r2 as parameter and avoids
> + * stalls for long running busy conditions by doing schedule

This can only ever return 0 or 1, right?

> + */
> +static inline int uv_call_sched(unsigned long r1, unsigned long r2)
> +{
> +	int cc = 3;
> +
> +	while (cc > 1) {
> +		asm volatile(
> +			"0:	.insn rrf,0xB9A40000,%[r1],%[r2],0,0\n"
> +			"		ipm	%[cc]\n"
> +			"		srl	%[cc],28\n"
> +			: [cc] "=d" (cc)
> +			: [r1] "a" (r1), [r2] "a" (r2)
> +			: "memory", "cc");
> +		if (need_resched())
> +			schedule();
> +	}
> +	return cc;
> +}
> +
> +/*
> + * Low level uv_call that takes r1 and r2 as parameter
> + */
>  static inline int uv_call(unsigned long r1, unsigned long r2)
>  {
>  	int cc;
> @@ -114,6 +152,26 @@ static inline int uv_call(unsigned long r1, unsigned long r2)
>  	return cc;
>  }
>  
> +/*
> + * special variant of uv_call that only transport the cpu or guest

s/transport/transports/

> + * handle and the command, like destroy or verify.
> + */
> +static inline int uv_cmd_nodata(u64 handle, u16 cmd, u32 *ret)
> +{
> +	int rc;
> +	struct uv_cb_nodata uvcb = {
> +		.header.cmd = cmd,
> +		.header.len = sizeof(uvcb),
> +		.handle = handle,
> +	};
> +
> +	WARN(!handle, "No handle provided to Ultravisor call cmd %x\n", cmd);

Just return -EINVAL for !handle?

> +	rc = uv_call_sched(0, (u64)&uvcb);
> +	if (ret)
> +		*ret = *(u32 *)&uvcb.header.rc;

Does that rc value in the block contain anything sensible if you didn't
get cc==0?

> +	return rc ? -EINVAL : 0;
> +}
> +
>  struct uv_info {
>  	unsigned long inst_calls_list[4];
>  	unsigned long uv_base_stor_len;

Re: [RFCv2 08/37] KVM: s390: protvirt: Add initial lifecycle handling

[David Hildenbrand]

[...]

> +#ifdef CONFIG_KVM_S390_PROTECTED_VIRTUALIZATION_HOST
> +static int kvm_s390_handle_pv(struct kvm *kvm, struct kvm_pv_cmd *cmd)
> +{
> +	int r = 0;
> +	void __user *argp = (void __user *)cmd->data;
> +
> +	switch (cmd->cmd) {
> +	case KVM_PV_VM_CREATE: {
> +		r = -EINVAL;
> +		if (kvm_s390_pv_is_protected(kvm))
> +			break;
> +
> +		r = kvm_s390_pv_alloc_vm(kvm);
> +		if (r)
> +			break;
> +
> +		mutex_lock(&kvm->lock);
> +		kvm_s390_vcpu_block_all(kvm);
> +		/* FMT 4 SIE needs esca */
> +		r = sca_switch_to_extended(kvm);
> +		if (!r)
> +			r = kvm_s390_pv_create_vm(kvm);
> +		kvm_s390_vcpu_unblock_all(kvm);
> +		mutex_unlock(&kvm->lock);
> +		break;
> +	}

I think KVM_PV_VM_ENABLE/KVM_PV_VM_DISABLE would be a better fit. You're
not creating/deleting VMs, aren't you? All you're doing is allocating
some data and performing some kind of a mode switch.
[...]

>  	VM_EVENT(kvm, 3, "create cpu %d at 0x%pK, sie block at 0x%pK", id, vcpu,
>  		 vcpu->arch.sie_block);
>  	trace_kvm_s390_create_vcpu(id, vcpu, vcpu->arch.sie_block);
> @@ -4353,6 +4502,37 @@ long kvm_arch_vcpu_async_ioctl(struct file *filp,
>  	return -ENOIOCTLCMD;
>  }
>  
> +#ifdef CONFIG_KVM_S390_PROTECTED_VIRTUALIZATION_HOST
> +static int kvm_s390_handle_pv_vcpu(struct kvm_vcpu *vcpu,
> +				   struct kvm_pv_cmd *cmd)
> +{
> +	int r = 0;
> +
> +	if (!kvm_s390_pv_is_protected(vcpu->kvm))
> +		return -EINVAL;
> +
> +	switch (cmd->cmd) {
> +	case KVM_PV_VCPU_CREATE: {
> +		if (kvm_s390_pv_handle_cpu(vcpu))
> +			return -EINVAL;
> +
> +		r = kvm_s390_pv_create_cpu(vcpu);
> +		break;
> +	}
> +	case KVM_PV_VCPU_DESTROY: {
> +		if (!kvm_s390_pv_handle_cpu(vcpu))
> +			return -EINVAL;
> +
> +		r = kvm_s390_pv_destroy_cpu(vcpu);
> +		break;
> +	}
> +	default:
> +		r = -ENOTTY;
> +	}
> +	return r;
> +}

I asked this already and didn't get an answer (lost in the flood of
comments :) )

Can't we simply convert all VCPUs via KVM_PV_VM_CREATE and destoy them
via KVM_PV_VM_DESTROY? Then you can easily handle hotplug as well in the
kernel when a new VCPU is created and PV is active - oh and I see you
are already doing that in kvm_arch_vcpu_create(). So that screams for
doing either this a) completely triggered by user space or b) completely
in the kernel. I prefer the latter. One interface less required.

I would assume that no VCPU is allowed to be running inside KVM while
performing the PV switch, which would make this even easier.

-- 
Thanks,

David / dhildenb

Re: [RFCv2 10/37] KVM: s390: protvirt: Secure memory is not mergeable

[David Hildenbrand]

>   * Enable storage key handling from now on and initialize the storage
>   * keys with the default key.
> @@ -2593,7 +2610,6 @@ static const struct mm_walk_ops enable_skey_walk_ops = {
>  int s390_enable_skey(void)
>  {
>  	struct mm_struct *mm = current->mm;
> -	struct vm_area_struct *vma;
>  	int rc = 0;
>  
>  	down_write(&mm->mmap_sem);
> @@ -2601,15 +2617,7 @@ int s390_enable_skey(void)
>  		goto out_up;
>  
>  	mm->context.uses_skeys = 1;
> -	for (vma = mm->mmap; vma; vma = vma->vm_next) {
> -		if (ksm_madvise(vma, vma->vm_start, vma->vm_end,
> -				MADV_UNMERGEABLE, &vma->vm_flags)) {
> -			mm->context.uses_skeys = 0;
> -			rc = -ENOMEM;
> -			goto out_up;
> -		}
> -	}
> -	mm->def_flags &= ~VM_MERGEABLE;
> +	gmap_mark_unmergeable();


I think I already complained about this patch *as is* and there was an
update. (ignoring errors, mm->context.uses_skeys handling in
gmap_mark_unmergeable() etc.)


-- 
Thanks,

David / dhildenb

Re: [RFCv2 05/37] s390/mm: provide memory management functions for protected KVM guests

[Cornelia Huck]

On Tue, 4 Feb 2020 12:56:10 +0100
Christian Borntraeger <borntraeger@de.ibm.com> wrote:

> On 04.02.20 11:57, Cornelia Huck wrote:
> 
> >>  
> >>  /* Generic bits for GMAP notification on DAT table entry changes. */
> >> @@ -61,6 +62,7 @@ struct gmap {
> >>  	spinlock_t shadow_lock;
> >>  	struct gmap *parent;
> >>  	unsigned long orig_asce;
> >> +	unsigned long se_handle;  
> > 
> > This is a deviation from the "protected virtualization" naming scheme
> > used in the previous patches, but I understand that the naming of this
> > whole feature is still in flux :) (Would still be nice to have it
> > consistent, though.)
> > 
> > However, I think I'd prefer something named based on protected
> > virtualization: the se_* stuff here just tends to make me think of
> > SELinux...  
> 
> I will just use guest_handle as this guests assigned to and from such:
> 
> arch/s390/include/asm/gmap.h:   unsigned long se_handle;
> arch/s390/include/asm/uv.h:             .guest_handle = gmap->se_handle,
> arch/s390/kvm/pv.c:     WRITE_ONCE(kvm->arch.gmap->se_handle, 0);
> arch/s390/kvm/pv.c:     kvm->arch.gmap->se_handle = uvcb.guest_handle;
> arch/s390/mm/fault.c:           .guest_handle = gmap->se_handle,

Even better.

> 
> 
> >   
> >>  	int edat_level;
> >>  	bool removed;
> >>  	bool initialized;
> >> diff --git a/arch/s390/include/asm/mmu.h b/arch/s390/include/asm/mmu.h
> >> index bcfb6371086f..984026cb3608 100644
> >> --- a/arch/s390/include/asm/mmu.h
> >> +++ b/arch/s390/include/asm/mmu.h
> >> @@ -16,6 +16,8 @@ typedef struct {
> >>  	unsigned long asce;
> >>  	unsigned long asce_limit;
> >>  	unsigned long vdso_base;
> >> +	/* The mmu context belongs to a secure guest. */
> >> +	atomic_t is_se;  
> > 
> > Here as well.  
> 
> I will use is "is_protected"

ok

> >   
> >>  	/*
> >>  	 * The following bitfields need a down_write on the mm
> >>  	 * semaphore when they are written to. As they are only  
> > 
> > (...)
> >   
> >> @@ -520,6 +521,15 @@ static inline int mm_has_pgste(struct mm_struct *mm)
> >>  	return 0;
> >>  }
> >>  
> >> +static inline int mm_is_se(struct mm_struct *mm)
> >> +{
> >> +#ifdef CONFIG_KVM_S390_PROTECTED_VIRTUALIZATION_HOST
> >> +	if (unlikely(atomic_read(&mm->context.is_se)))
> >> +		return 1;
> >> +#endif
> >> +	return 0;
> >> +}  
> > 
> > I'm wondering how big of an overhead we actually have because we need
> > to check the flag here if the feature is enabled. We have an extra
> > check in a few functions anyway, even if protected virt is not
> > configured in. Given that distributions would likely want to enable the
> > feature in their kernels, I'm currently tending towards dropping the
> > extra config option.  
> 
> Makes sense. I will wait a bit more but if you do not disagree I will remove 
> the extra host config and bind things to CONFIG_KVM or CONFIG_PGSTE depending
> on context

From the code I read so far, that makes sense.

Re: [RFCv2 12/37] KVM: s390: protvirt: Handle SE notification interceptions

[David Hildenbrand]

>  arch/s390/include/asm/kvm_host.h |  2 ++
>  arch/s390/kvm/intercept.c        | 10 ++++++++++
>  2 files changed, 12 insertions(+)
> 
> diff --git a/arch/s390/include/asm/kvm_host.h b/arch/s390/include/asm/kvm_host.h
> index 841690d05080..d63ed05272ec 100644
> --- a/arch/s390/include/asm/kvm_host.h
> +++ b/arch/s390/include/asm/kvm_host.h
> @@ -215,6 +215,8 @@ struct kvm_s390_sie_block {
>  #define ICPT_PARTEXEC	0x38
>  #define ICPT_IOINST	0x40
>  #define ICPT_KSS	0x5c
> +#define ICPT_PV_MCHKR	0x60
> +#define ICPT_PV_INT_EN	0x64
>  	__u8	icptcode;		/* 0x0050 */
>  	__u8	icptstatus;		/* 0x0051 */
>  	__u16	ihcpu;			/* 0x0052 */
> diff --git a/arch/s390/kvm/intercept.c b/arch/s390/kvm/intercept.c
> index a389fa85cca2..eaa2a21c3170 100644
> --- a/arch/s390/kvm/intercept.c
> +++ b/arch/s390/kvm/intercept.c
> @@ -480,6 +480,16 @@ int kvm_handle_sie_intercept(struct kvm_vcpu *vcpu)
>  	case ICPT_KSS:
>  		rc = kvm_s390_skey_check_enable(vcpu);
>  		break;
> +	case ICPT_PV_MCHKR:
> +		/* fallthrough */

No need for this annotation if there is no additional code in the case.

> +	case ICPT_PV_INT_EN:
> +		/*
> +		 * PSW bit 13 or a CR (0, 6, 14) changed and we might
> +		 * now be able to deliver interrupts. The pre-run code
> +		 * will take care of this.
> +		 */
> +		rc = 0;
> +		break;
>  	default:
>  		return -EOPNOTSUPP;
>  	}
> 

Reviewed-by: David Hildenbrand <david@redhat.com>

-- 
Thanks,

David / dhildenb

Re: [RFCv2 13/37] KVM: s390: protvirt: Instruction emulation

[David Hildenbrand]

On 03.02.20 14:19, Christian Borntraeger wrote:
> From: Janosch Frank <frankja@linux.ibm.com>
> 
> We have two new SIE exit codes dealing with instructions.
> 104 (0x68) for a secure instruction interception, on which the SIE needs
> hypervisor action to complete the instruction. We can piggy-back on the
> existing instruction handlers.
> 
> 108 which is merely a notification and provides data for tracking and
> management. For example this is used to tell the host about a new value
> for the prefix register. As there will be several special case handlers
> in later patches, we handle this in a separate function.
> 
> Signed-off-by: Janosch Frank <frankja@linux.ibm.com>
> ---
>  arch/s390/include/asm/kvm_host.h |  2 ++
>  arch/s390/kvm/intercept.c        | 11 +++++++++++
>  2 files changed, 13 insertions(+)
> 
> diff --git a/arch/s390/include/asm/kvm_host.h b/arch/s390/include/asm/kvm_host.h
> index d63ed05272ec..58845b315be0 100644
> --- a/arch/s390/include/asm/kvm_host.h
> +++ b/arch/s390/include/asm/kvm_host.h
> @@ -217,6 +217,8 @@ struct kvm_s390_sie_block {
>  #define ICPT_KSS	0x5c
>  #define ICPT_PV_MCHKR	0x60
>  #define ICPT_PV_INT_EN	0x64
> +#define ICPT_PV_INSTR	0x68
> +#define ICPT_PV_NOTIF	0x6c

NOTIFY? NOTIFICATION? NOTIF is weird.

>  	__u8	icptcode;		/* 0x0050 */
>  	__u8	icptstatus;		/* 0x0051 */
>  	__u16	ihcpu;			/* 0x0052 */
> diff --git a/arch/s390/kvm/intercept.c b/arch/s390/kvm/intercept.c
> index eaa2a21c3170..4b3fbbde1674 100644
> --- a/arch/s390/kvm/intercept.c
> +++ b/arch/s390/kvm/intercept.c
> @@ -444,6 +444,11 @@ static int handle_operexc(struct kvm_vcpu *vcpu)
>  	return kvm_s390_inject_program_int(vcpu, PGM_OPERATION);
>  }
>  
> +static int handle_pv_not(struct kvm_vcpu *vcpu)

"notification" please. not not. You see why ;)

> +{
> +	return handle_instruction(vcpu);
> +}
> +
>  int kvm_handle_sie_intercept(struct kvm_vcpu *vcpu)
>  {
>  	int rc, per_rc = 0;
> @@ -490,6 +495,12 @@ int kvm_handle_sie_intercept(struct kvm_vcpu *vcpu)
>  		 */
>  		rc = 0;
>  		break;
> +	case ICPT_PV_INSTR:
> +		rc = handle_instruction(vcpu);
> +		break;
> +	case ICPT_PV_NOTIF:
> +		rc = handle_pv_not(vcpu);
> +		break;
>  	default:
>  		return -EOPNOTSUPP;
>  	}

Reviewed-by: David Hildenbrand <david@redhat.com>

-- 
Thanks,

David / dhildenb

Re: [RFCv2 12/37] KVM: s390: protvirt: Handle SE notification interceptions

[Christian Borntraeger]

On 04.02.20 13:16, David Hildenbrand wrote:
> 
>>  arch/s390/include/asm/kvm_host.h |  2 ++
>>  arch/s390/kvm/intercept.c        | 10 ++++++++++
>>  2 files changed, 12 insertions(+)
>>
>> diff --git a/arch/s390/include/asm/kvm_host.h b/arch/s390/include/asm/kvm_host.h
>> index 841690d05080..d63ed05272ec 100644
>> --- a/arch/s390/include/asm/kvm_host.h
>> +++ b/arch/s390/include/asm/kvm_host.h
>> @@ -215,6 +215,8 @@ struct kvm_s390_sie_block {
>>  #define ICPT_PARTEXEC	0x38
>>  #define ICPT_IOINST	0x40
>>  #define ICPT_KSS	0x5c
>> +#define ICPT_PV_MCHKR	0x60
>> +#define ICPT_PV_INT_EN	0x64
>>  	__u8	icptcode;		/* 0x0050 */
>>  	__u8	icptstatus;		/* 0x0051 */
>>  	__u16	ihcpu;			/* 0x0052 */
>> diff --git a/arch/s390/kvm/intercept.c b/arch/s390/kvm/intercept.c
>> index a389fa85cca2..eaa2a21c3170 100644
>> --- a/arch/s390/kvm/intercept.c
>> +++ b/arch/s390/kvm/intercept.c
>> @@ -480,6 +480,16 @@ int kvm_handle_sie_intercept(struct kvm_vcpu *vcpu)
>>  	case ICPT_KSS:
>>  		rc = kvm_s390_skey_check_enable(vcpu);
>>  		break;
>> +	case ICPT_PV_MCHKR:
>> +		/* fallthrough */
> 
> No need for this annotation if there is no additional code in the case.

ack

Re: [RFCv2 13/37] KVM: s390: protvirt: Instruction emulation

[Christian Borntraeger]

On 04.02.20 13:20, David Hildenbrand wrote:
> On 03.02.20 14:19, Christian Borntraeger wrote:
>> From: Janosch Frank <frankja@linux.ibm.com>
>>
>> We have two new SIE exit codes dealing with instructions.
>> 104 (0x68) for a secure instruction interception, on which the SIE needs
>> hypervisor action to complete the instruction. We can piggy-back on the
>> existing instruction handlers.
>>
>> 108 which is merely a notification and provides data for tracking and
>> management. For example this is used to tell the host about a new value
>> for the prefix register. As there will be several special case handlers
>> in later patches, we handle this in a separate function.
>>
>> Signed-off-by: Janosch Frank <frankja@linux.ibm.com>
>> ---
>>  arch/s390/include/asm/kvm_host.h |  2 ++
>>  arch/s390/kvm/intercept.c        | 11 +++++++++++
>>  2 files changed, 13 insertions(+)
>>
>> diff --git a/arch/s390/include/asm/kvm_host.h b/arch/s390/include/asm/kvm_host.h
>> index d63ed05272ec..58845b315be0 100644
>> --- a/arch/s390/include/asm/kvm_host.h
>> +++ b/arch/s390/include/asm/kvm_host.h
>> @@ -217,6 +217,8 @@ struct kvm_s390_sie_block {
>>  #define ICPT_KSS	0x5c
>>  #define ICPT_PV_MCHKR	0x60
>>  #define ICPT_PV_INT_EN	0x64
>> +#define ICPT_PV_INSTR	0x68
>> +#define ICPT_PV_NOTIF	0x6c
> 
> NOTIFY? NOTIFICATION? NOTIF is weird.

ack. I used NOTIFY to keep the numbers aligned.

> 
>>  	__u8	icptcode;		/* 0x0050 */
>>  	__u8	icptstatus;		/* 0x0051 */
>>  	__u16	ihcpu;			/* 0x0052 */
>> diff --git a/arch/s390/kvm/intercept.c b/arch/s390/kvm/intercept.c
>> index eaa2a21c3170..4b3fbbde1674 100644
>> --- a/arch/s390/kvm/intercept.c
>> +++ b/arch/s390/kvm/intercept.c
>> @@ -444,6 +444,11 @@ static int handle_operexc(struct kvm_vcpu *vcpu)
>>  	return kvm_s390_inject_program_int(vcpu, PGM_OPERATION);
>>  }
>>  
>> +static int handle_pv_not(struct kvm_vcpu *vcpu)
> 
> "notification" please. not not. You see why ;)

ack. 

Re: [RFCv2 07/37] KVM: s390: add new variants of UV CALL

[Claudio Imbrenda]

On Tue, 4 Feb 2020 13:11:07 +0100
Cornelia Huck <cohuck@redhat.com> wrote:

[...]

> 
> > +	rc = uv_call_sched(0, (u64)&uvcb);
> > +	if (ret)
> > +		*ret = *(u32 *)&uvcb.header.rc;  
> 
> Does that rc value in the block contain anything sensible if you
> didn't get cc==0?

yes, RC is always meaningful. 

CC == 0 implies RC == 1, which means success.
CC == 1 implies RC != 1, which means something went wrong in some way

in theory you could always disregard CC and only check RC



Claudio

Re: [RFCv2 08/37] KVM: s390: protvirt: Add initial lifecycle handling

[Christian Borntraeger]

On 04.02.20 13:13, David Hildenbrand wrote:
> [...]
> 
>> +#ifdef CONFIG_KVM_S390_PROTECTED_VIRTUALIZATION_HOST
>> +static int kvm_s390_handle_pv(struct kvm *kvm, struct kvm_pv_cmd *cmd)
>> +{
>> +	int r = 0;
>> +	void __user *argp = (void __user *)cmd->data;
>> +
>> +	switch (cmd->cmd) {
>> +	case KVM_PV_VM_CREATE: {
>> +		r = -EINVAL;
>> +		if (kvm_s390_pv_is_protected(kvm))
>> +			break;
>> +
>> +		r = kvm_s390_pv_alloc_vm(kvm);
>> +		if (r)
>> +			break;
>> +
>> +		mutex_lock(&kvm->lock);
>> +		kvm_s390_vcpu_block_all(kvm);
>> +		/* FMT 4 SIE needs esca */
>> +		r = sca_switch_to_extended(kvm);
>> +		if (!r)
>> +			r = kvm_s390_pv_create_vm(kvm);
>> +		kvm_s390_vcpu_unblock_all(kvm);
>> +		mutex_unlock(&kvm->lock);
>> +		break;
>> +	}
> 
> I think KVM_PV_VM_ENABLE/KVM_PV_VM_DISABLE would be a better fit. You're
> not creating/deleting VMs, aren't you? All you're doing is allocating
> some data and performing some kind of a mode switch.

I kind of like the idea. Need to talk to Janosch about this. 
> [...]
> 
>>  	VM_EVENT(kvm, 3, "create cpu %d at 0x%pK, sie block at 0x%pK", id, vcpu,
>>  		 vcpu->arch.sie_block);
>>  	trace_kvm_s390_create_vcpu(id, vcpu, vcpu->arch.sie_block);
>> @@ -4353,6 +4502,37 @@ long kvm_arch_vcpu_async_ioctl(struct file *filp,
>>  	return -ENOIOCTLCMD;
>>  }
>>  
>> +#ifdef CONFIG_KVM_S390_PROTECTED_VIRTUALIZATION_HOST
>> +static int kvm_s390_handle_pv_vcpu(struct kvm_vcpu *vcpu,
>> +				   struct kvm_pv_cmd *cmd)
>> +{
>> +	int r = 0;
>> +
>> +	if (!kvm_s390_pv_is_protected(vcpu->kvm))
>> +		return -EINVAL;
>> +
>> +	switch (cmd->cmd) {
>> +	case KVM_PV_VCPU_CREATE: {
>> +		if (kvm_s390_pv_handle_cpu(vcpu))
>> +			return -EINVAL;
>> +
>> +		r = kvm_s390_pv_create_cpu(vcpu);
>> +		break;
>> +	}
>> +	case KVM_PV_VCPU_DESTROY: {
>> +		if (!kvm_s390_pv_handle_cpu(vcpu))
>> +			return -EINVAL;
>> +
>> +		r = kvm_s390_pv_destroy_cpu(vcpu);
>> +		break;
>> +	}
>> +	default:
>> +		r = -ENOTTY;
>> +	}
>> +	return r;
>> +}
> 
> I asked this already and didn't get an answer (lost in the flood of
> comments :) )
> 
> Can't we simply convert all VCPUs via KVM_PV_VM_CREATE and destoy them
> via KVM_PV_VM_DESTROY? Then you can easily handle hotplug as well in the
> kernel when a new VCPU is created and PV is active - oh and I see you
> are already doing that in kvm_arch_vcpu_create(). So that screams for
> doing either this a) completely triggered by user space or b) completely
> in the kernel. I prefer the latter. One interface less required.
> 
> I would assume that no VCPU is allowed to be running inside KVM while

right.

> performing the PV switch, which would make this even easier.

Same as above. I like the idea. Will need to talk to Janosch. 

Re: [RFCv2 07/37] KVM: s390: add new variants of UV CALL

[Cornelia Huck]

On Tue, 4 Feb 2020 13:30:02 +0100
Claudio Imbrenda <imbrenda@linux.ibm.com> wrote:

> On Tue, 4 Feb 2020 13:11:07 +0100
> Cornelia Huck <cohuck@redhat.com> wrote:
> 
> [...]
> 
> >   
> > > +	rc = uv_call_sched(0, (u64)&uvcb);
> > > +	if (ret)
> > > +		*ret = *(u32 *)&uvcb.header.rc;    
> > 
> > Does that rc value in the block contain anything sensible if you
> > didn't get cc==0?  
> 
> yes, RC is always meaningful. 
> 
> CC == 0 implies RC == 1, which means success.
> CC == 1 implies RC != 1, which means something went wrong in some way
> 
> in theory you could always disregard CC and only check RC

Ok, thx.

Re: [RFCv2 06/37] s390: add (non)secure page access exceptions handlers

[Thomas Huth]

On 04/02/2020 12.41, Claudio Imbrenda wrote:
> On Tue, 4 Feb 2020 11:37:42 +0100
> Thomas Huth <thuth@redhat.com> wrote:
> 
> [...]
> 
>>> ---
>>>  arch/s390/kernel/pgm_check.S |  4 +-
>>>  arch/s390/mm/fault.c         | 87
>>> ++++++++++++++++++++++++++++++++++++ 2 files changed, 89
>>> insertions(+), 2 deletions(-)  
>> [...]
>>> +void do_non_secure_storage_access(struct pt_regs *regs)
>>> +{
>>> +	unsigned long gaddr = regs->int_parm_long &
>>> __FAIL_ADDR_MASK;
>>> +	struct gmap *gmap = (struct gmap *)S390_lowcore.gmap;
>>> +	struct uv_cb_cts uvcb = {
>>> +		.header.cmd = UVC_CMD_CONV_TO_SEC_STOR,
>>> +		.header.len = sizeof(uvcb),
>>> +		.guest_handle = gmap->se_handle,
>>> +		.gaddr = gaddr,
>>> +	};
>>> +	int rc;
>>> +
>>> +	if (get_fault_type(regs) != GMAP_FAULT) {
>>> +		do_fault_error(regs, VM_READ | VM_WRITE,
>>> VM_FAULT_BADMAP);
>>> +		WARN_ON_ONCE(1);
>>> +		return;
>>> +	}
>>> +
>>> +	rc = uv_make_secure(gmap, gaddr, &uvcb, 0);
>>> +	if (rc == -EINVAL && uvcb.header.rc != 0x104)
>>> +		send_sig(SIGSEGV, current, 0);
>>> +}  
>>
>> What about the other rc beside 0x104 that could happen here? They go
>> unnoticed?
> 
> no, they are handled in the uv_make_secure, and return an appropriate
> error code. 
Hmm, in patch 05/37, I basically see:

+static int make_secure_pte(pte_t *ptep, unsigned long addr, void *data)
+{
[...]
+	rc = uv_call(0, (u64)params->uvcb);
+	page_ref_unfreeze(page, expected);
+	if (rc)
+		rc = (params->uvcb->rc == 0x10a) ? -ENXIO : -EINVAL;
+	return rc;
+}

+int uv_make_secure(struct gmap *gmap, unsigned long gaddr, void *uvcb,
int pins)
+{
[...]
+	lock_page(params.page);
+	rc = apply_to_page_range(gmap->mm, uaddr, PAGE_SIZE, make_secure_pte,
&params);
+	unlock_page(params.page);
+out:
+	up_read(&gmap->mm->mmap_sem);
+
+	if (rc == -EBUSY) {
+		if (local_drain) {
+			lru_add_drain_all();
+			return -EAGAIN;
+		}
+		lru_add_drain();
+		local_drain = 1;
+		goto again;
+	} else if (rc == -ENXIO) {
+		if (gmap_fault(gmap, gaddr, FAULT_FLAG_WRITE))
+			return -EFAULT;
+		return -EAGAIN;
+	}
+	return rc;
+}

So 0x10a result in -ENXIO and is handled ==> OK.
And 0x104 is handled in do_non_secure_storage_access ==> OK.

But what about the other possible error codes? make_secure_pte() returns
-EINVAL in that case, but uv_make_secure() does not care about that
error code, and do_non_secure_storage_access() only cares if
uvcb.header.rc was 0x104 ... what did I miss?

 Thomas

Re: [RFCv2 10/37] KVM: s390: protvirt: Secure memory is not mergeable

[Christian Borntraeger]

On 04.02.20 13:15, David Hildenbrand wrote:
> 
>>   * Enable storage key handling from now on and initialize the storage
>>   * keys with the default key.
>> @@ -2593,7 +2610,6 @@ static const struct mm_walk_ops enable_skey_walk_ops = {
>>  int s390_enable_skey(void)
>>  {
>>  	struct mm_struct *mm = current->mm;
>> -	struct vm_area_struct *vma;
>>  	int rc = 0;
>>  
>>  	down_write(&mm->mmap_sem);
>> @@ -2601,15 +2617,7 @@ int s390_enable_skey(void)
>>  		goto out_up;
>>  
>>  	mm->context.uses_skeys = 1;
>> -	for (vma = mm->mmap; vma; vma = vma->vm_next) {
>> -		if (ksm_madvise(vma, vma->vm_start, vma->vm_end,
>> -				MADV_UNMERGEABLE, &vma->vm_flags)) {
>> -			mm->context.uses_skeys = 0;
>> -			rc = -ENOMEM;
>> -			goto out_up;
>> -		}
>> -	}
>> -	mm->def_flags &= ~VM_MERGEABLE;
>> +	gmap_mark_unmergeable();
> 
> 
> I think I already complained about this patch *as is* and there was an
> update. (ignoring errors, mm->context.uses_skeys handling in
> gmap_mark_unmergeable() etc.)
> 
Right something like this on top:

diff --git a/arch/s390/mm/gmap.c b/arch/s390/mm/gmap.c
index bf365a09f900..b95bf40bb7a7 100644
--- a/arch/s390/mm/gmap.c
+++ b/arch/s390/mm/gmap.c
@@ -2556,7 +2556,6 @@ int gmap_mark_unmergeable(void)
        for (vma = mm->mmap; vma; vma = vma->vm_next) {
                if (ksm_madvise(vma, vma->vm_start, vma->vm_end,
                                MADV_UNMERGEABLE, &vma->vm_flags)) {
-                       mm->context.uses_skeys = 0;
                        return -ENOMEM;
                }
        }
@@ -2617,8 +2616,10 @@ int s390_enable_skey(void)
                goto out_up;
 
        mm->context.uses_skeys = 1;
-       gmap_mark_unmergeable();
-
+       if (gmap_mark_unmergeable()) {
+               mm->context.uses_skeys = 0;
+               goto out_up;
+       }
        walk_page_range(mm, 0, TASK_SIZE, &enable_skey_walk_ops, NULL);
 
 out_up:


?

Re: [RFCv2 10/37] KVM: s390: protvirt: Secure memory is not mergeable

[David Hildenbrand]

On 04.02.20 13:53, Christian Borntraeger wrote:
> 
> 
> On 04.02.20 13:15, David Hildenbrand wrote:
>>
>>>   * Enable storage key handling from now on and initialize the storage
>>>   * keys with the default key.
>>> @@ -2593,7 +2610,6 @@ static const struct mm_walk_ops enable_skey_walk_ops = {
>>>  int s390_enable_skey(void)
>>>  {
>>>  	struct mm_struct *mm = current->mm;
>>> -	struct vm_area_struct *vma;
>>>  	int rc = 0;
>>>  
>>>  	down_write(&mm->mmap_sem);
>>> @@ -2601,15 +2617,7 @@ int s390_enable_skey(void)
>>>  		goto out_up;
>>>  
>>>  	mm->context.uses_skeys = 1;
>>> -	for (vma = mm->mmap; vma; vma = vma->vm_next) {
>>> -		if (ksm_madvise(vma, vma->vm_start, vma->vm_end,
>>> -				MADV_UNMERGEABLE, &vma->vm_flags)) {
>>> -			mm->context.uses_skeys = 0;
>>> -			rc = -ENOMEM;
>>> -			goto out_up;
>>> -		}
>>> -	}
>>> -	mm->def_flags &= ~VM_MERGEABLE;
>>> +	gmap_mark_unmergeable();
>>
>>
>> I think I already complained about this patch *as is* and there was an
>> update. (ignoring errors, mm->context.uses_skeys handling in
>> gmap_mark_unmergeable() etc.)
>>
> Right something like this on top:
> 
> diff --git a/arch/s390/mm/gmap.c b/arch/s390/mm/gmap.c
> index bf365a09f900..b95bf40bb7a7 100644
> --- a/arch/s390/mm/gmap.c
> +++ b/arch/s390/mm/gmap.c
> @@ -2556,7 +2556,6 @@ int gmap_mark_unmergeable(void)
>         for (vma = mm->mmap; vma; vma = vma->vm_next) {
>                 if (ksm_madvise(vma, vma->vm_start, vma->vm_end,
>                                 MADV_UNMERGEABLE, &vma->vm_flags)) {
> -                       mm->context.uses_skeys = 0;
>                         return -ENOMEM;
>                 }
>         }
> @@ -2617,8 +2616,10 @@ int s390_enable_skey(void)
>                 goto out_up;
>  
>         mm->context.uses_skeys = 1;
> -       gmap_mark_unmergeable();
> -
> +       if (gmap_mark_unmergeable()) {
> +               mm->context.uses_skeys = 0;
> +               goto out_up;
> +       }

You have to set rc if I am not wrong.

-- 
Thanks,

David / dhildenb

[PATCH v2 10/37] KVM: s390: protvirt: Secure memory is not mergeable

[Christian Borntraeger]

From: Janosch Frank <frankja@linux.ibm.com>

KSM will not work on secure pages, because when the kernel reads a
secure page, it will be encrypted and hence no two pages will look the
same.

Let's mark the guest pages as unmergeable when we transition to secure
mode.

Signed-off-by: Janosch Frank <frankja@linux.ibm.com>
Reviewed-by: Thomas Huth <thuth@redhat.com>
---
 arch/s390/include/asm/gmap.h |  1 +
 arch/s390/kvm/kvm-s390.c     |  6 ++++++
 arch/s390/mm/gmap.c          | 30 ++++++++++++++++++++----------
 3 files changed, 27 insertions(+), 10 deletions(-)

diff --git a/arch/s390/include/asm/gmap.h b/arch/s390/include/asm/gmap.h
index e2d2f48c5c7c..e1f2cc0b2b00 100644
--- a/arch/s390/include/asm/gmap.h
+++ b/arch/s390/include/asm/gmap.h
@@ -146,4 +146,5 @@ int gmap_mprotect_notify(struct gmap *, unsigned long start,
 
 void gmap_sync_dirty_log_pmd(struct gmap *gmap, unsigned long dirty_bitmap[4],
 			     unsigned long gaddr, unsigned long vmaddr);
+int gmap_mark_unmergeable(void);
 #endif /* _ASM_S390_GMAP_H */
diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
index 35f46404830f..741d81f57c3c 100644
--- a/arch/s390/kvm/kvm-s390.c
+++ b/arch/s390/kvm/kvm-s390.c
@@ -2181,6 +2181,12 @@ static int kvm_s390_handle_pv(struct kvm *kvm, struct kvm_pv_cmd *cmd)
 		if (r)
 			break;
 
+		down_write(&current->mm->mmap_sem);
+		r = gmap_mark_unmergeable();
+		up_write(&current->mm->mmap_sem);
+		if (r)
+			break;
+
 		mutex_lock(&kvm->lock);
 		kvm_s390_vcpu_block_all(kvm);
 		/* FMT 4 SIE needs esca */
diff --git a/arch/s390/mm/gmap.c b/arch/s390/mm/gmap.c
index edcdca97e85e..7291452fe5f0 100644
--- a/arch/s390/mm/gmap.c
+++ b/arch/s390/mm/gmap.c
@@ -2548,6 +2548,22 @@ int s390_enable_sie(void)
 }
 EXPORT_SYMBOL_GPL(s390_enable_sie);
 
+int gmap_mark_unmergeable(void)
+{
+	struct mm_struct *mm = current->mm;
+	struct vm_area_struct *vma;
+
+	for (vma = mm->mmap; vma; vma = vma->vm_next) {
+		if (ksm_madvise(vma, vma->vm_start, vma->vm_end,
+				MADV_UNMERGEABLE, &vma->vm_flags)) {
+			return -ENOMEM;
+		}
+	}
+	mm->def_flags &= ~VM_MERGEABLE;
+	return 0;
+}
+EXPORT_SYMBOL_GPL(gmap_mark_unmergeable);
+
 /*
  * Enable storage key handling from now on and initialize the storage
  * keys with the default key.
@@ -2593,7 +2609,6 @@ static const struct mm_walk_ops enable_skey_walk_ops = {
 int s390_enable_skey(void)
 {
 	struct mm_struct *mm = current->mm;
-	struct vm_area_struct *vma;
 	int rc = 0;
 
 	down_write(&mm->mmap_sem);
@@ -2601,16 +2616,11 @@ int s390_enable_skey(void)
 		goto out_up;
 
 	mm->context.uses_skeys = 1;
-	for (vma = mm->mmap; vma; vma = vma->vm_next) {
-		if (ksm_madvise(vma, vma->vm_start, vma->vm_end,
-				MADV_UNMERGEABLE, &vma->vm_flags)) {
-			mm->context.uses_skeys = 0;
-			rc = -ENOMEM;
-			goto out_up;
-		}
+	rc = gmap_mark_unmergeable();
+	if (rc) {
+		mm->context.uses_skeys = 0;
+		goto out_up;
 	}
-	mm->def_flags &= ~VM_MERGEABLE;
-
 	walk_page_range(mm, 0, TASK_SIZE, &enable_skey_walk_ops, NULL);
 
 out_up:
-- 
2.24.0

Re: [PATCH v2 10/37] KVM: s390: protvirt: Secure memory is not mergeable

[David Hildenbrand]

On 04.02.20 14:04, Christian Borntraeger wrote:
> From: Janosch Frank <frankja@linux.ibm.com>
> 
> KSM will not work on secure pages, because when the kernel reads a
> secure page, it will be encrypted and hence no two pages will look the
> same.
> 
> Let's mark the guest pages as unmergeable when we transition to secure
> mode.
> 
> Signed-off-by: Janosch Frank <frankja@linux.ibm.com>
> Reviewed-by: Thomas Huth <thuth@redhat.com>
> ---
>  arch/s390/include/asm/gmap.h |  1 +
>  arch/s390/kvm/kvm-s390.c     |  6 ++++++
>  arch/s390/mm/gmap.c          | 30 ++++++++++++++++++++----------
>  3 files changed, 27 insertions(+), 10 deletions(-)
> 
> diff --git a/arch/s390/include/asm/gmap.h b/arch/s390/include/asm/gmap.h
> index e2d2f48c5c7c..e1f2cc0b2b00 100644
> --- a/arch/s390/include/asm/gmap.h
> +++ b/arch/s390/include/asm/gmap.h
> @@ -146,4 +146,5 @@ int gmap_mprotect_notify(struct gmap *, unsigned long start,
>  
>  void gmap_sync_dirty_log_pmd(struct gmap *gmap, unsigned long dirty_bitmap[4],
>  			     unsigned long gaddr, unsigned long vmaddr);
> +int gmap_mark_unmergeable(void);
>  #endif /* _ASM_S390_GMAP_H */
> diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
> index 35f46404830f..741d81f57c3c 100644
> --- a/arch/s390/kvm/kvm-s390.c
> +++ b/arch/s390/kvm/kvm-s390.c
> @@ -2181,6 +2181,12 @@ static int kvm_s390_handle_pv(struct kvm *kvm, struct kvm_pv_cmd *cmd)
>  		if (r)
>  			break;
>  
> +		down_write(&current->mm->mmap_sem);
> +		r = gmap_mark_unmergeable();
> +		up_write(&current->mm->mmap_sem);
> +		if (r)
> +			break;
> +
>  		mutex_lock(&kvm->lock);
>  		kvm_s390_vcpu_block_all(kvm);
>  		/* FMT 4 SIE needs esca */
> diff --git a/arch/s390/mm/gmap.c b/arch/s390/mm/gmap.c
> index edcdca97e85e..7291452fe5f0 100644
> --- a/arch/s390/mm/gmap.c
> +++ b/arch/s390/mm/gmap.c
> @@ -2548,6 +2548,22 @@ int s390_enable_sie(void)
>  }
>  EXPORT_SYMBOL_GPL(s390_enable_sie);
>  
> +int gmap_mark_unmergeable(void)
> +{
> +	struct mm_struct *mm = current->mm;
> +	struct vm_area_struct *vma;
> +
> +	for (vma = mm->mmap; vma; vma = vma->vm_next) {
> +		if (ksm_madvise(vma, vma->vm_start, vma->vm_end,
> +				MADV_UNMERGEABLE, &vma->vm_flags)) {
> +			return -ENOMEM;
> +		}
> +	}
> +	mm->def_flags &= ~VM_MERGEABLE;
> +	return 0;
> +}
> +EXPORT_SYMBOL_GPL(gmap_mark_unmergeable);
> +
>  /*
>   * Enable storage key handling from now on and initialize the storage
>   * keys with the default key.
> @@ -2593,7 +2609,6 @@ static const struct mm_walk_ops enable_skey_walk_ops = {
>  int s390_enable_skey(void)
>  {
>  	struct mm_struct *mm = current->mm;
> -	struct vm_area_struct *vma;
>  	int rc = 0;
>  
>  	down_write(&mm->mmap_sem);
> @@ -2601,16 +2616,11 @@ int s390_enable_skey(void)
>  		goto out_up;
>  
>  	mm->context.uses_skeys = 1;
> -	for (vma = mm->mmap; vma; vma = vma->vm_next) {
> -		if (ksm_madvise(vma, vma->vm_start, vma->vm_end,
> -				MADV_UNMERGEABLE, &vma->vm_flags)) {
> -			mm->context.uses_skeys = 0;
> -			rc = -ENOMEM;
> -			goto out_up;
> -		}
> +	rc = gmap_mark_unmergeable();
> +	if (rc) {
> +		mm->context.uses_skeys = 0;
> +		goto out_up;
>  	}
> -	mm->def_flags &= ~VM_MERGEABLE;
> -
>  	walk_page_range(mm, 0, TASK_SIZE, &enable_skey_walk_ops, NULL);
>  
>  out_up:
> 

Reviewed-by: David Hildenbrand <david@redhat.com>

-- 
Thanks,

David / dhildenb

Re: [RFCv2 06/37] s390: add (non)secure page access exceptions handlers

[Claudio Imbrenda]

On Tue, 4 Feb 2020 13:48:47 +0100
Thomas Huth <thuth@redhat.com> wrote:

> On 04/02/2020 12.41, Claudio Imbrenda wrote:
> > On Tue, 4 Feb 2020 11:37:42 +0100
> > Thomas Huth <thuth@redhat.com> wrote:
> > 
> > [...]
> >   
> >>> ---
> >>>  arch/s390/kernel/pgm_check.S |  4 +-
> >>>  arch/s390/mm/fault.c         | 87
> >>> ++++++++++++++++++++++++++++++++++++ 2 files changed, 89
> >>> insertions(+), 2 deletions(-)    
> >> [...]  
> >>> +void do_non_secure_storage_access(struct pt_regs *regs)
> >>> +{
> >>> +	unsigned long gaddr = regs->int_parm_long &
> >>> __FAIL_ADDR_MASK;
> >>> +	struct gmap *gmap = (struct gmap *)S390_lowcore.gmap;
> >>> +	struct uv_cb_cts uvcb = {
> >>> +		.header.cmd = UVC_CMD_CONV_TO_SEC_STOR,
> >>> +		.header.len = sizeof(uvcb),
> >>> +		.guest_handle = gmap->se_handle,
> >>> +		.gaddr = gaddr,
> >>> +	};
> >>> +	int rc;
> >>> +
> >>> +	if (get_fault_type(regs) != GMAP_FAULT) {
> >>> +		do_fault_error(regs, VM_READ | VM_WRITE,
> >>> VM_FAULT_BADMAP);
> >>> +		WARN_ON_ONCE(1);
> >>> +		return;
> >>> +	}
> >>> +
> >>> +	rc = uv_make_secure(gmap, gaddr, &uvcb, 0);
> >>> +	if (rc == -EINVAL && uvcb.header.rc != 0x104)
> >>> +		send_sig(SIGSEGV, current, 0);
> >>> +}    
> >>
> >> What about the other rc beside 0x104 that could happen here? They
> >> go unnoticed?  
> > 
> > no, they are handled in the uv_make_secure, and return an
> > appropriate error code.   
> Hmm, in patch 05/37, I basically see:
> 
> +static int make_secure_pte(pte_t *ptep, unsigned long addr, void
> *data) +{
> [...]
> +	rc = uv_call(0, (u64)params->uvcb);
> +	page_ref_unfreeze(page, expected);
> +	if (rc)
> +		rc = (params->uvcb->rc == 0x10a) ? -ENXIO : -EINVAL;
> +	return rc;
> +}
> 
> +int uv_make_secure(struct gmap *gmap, unsigned long gaddr, void
> *uvcb, int pins)
> +{
> [...]
> +	lock_page(params.page);
> +	rc = apply_to_page_range(gmap->mm, uaddr, PAGE_SIZE,
> make_secure_pte, &params);
> +	unlock_page(params.page);
> +out:
> +	up_read(&gmap->mm->mmap_sem);
> +
> +	if (rc == -EBUSY) {
> +		if (local_drain) {
> +			lru_add_drain_all();
> +			return -EAGAIN;
> +		}
> +		lru_add_drain();
> +		local_drain = 1;
> +		goto again;
> +	} else if (rc == -ENXIO) {
> +		if (gmap_fault(gmap, gaddr, FAULT_FLAG_WRITE))
> +			return -EFAULT;
> +		return -EAGAIN;
> +	}
> +	return rc;
> +}
> 
> So 0x10a result in -ENXIO and is handled ==> OK.
> And 0x104 is handled in do_non_secure_storage_access ==> OK.
> 
> But what about the other possible error codes? make_secure_pte()
> returns -EINVAL in that case, but uv_make_secure() does not care
> about that error code, and do_non_secure_storage_access() only cares
> if uvcb.header.rc was 0x104 ... what did I miss?

basically, any error value that is not handled by uv_make_secure is
passed as-is from make_secure_pte directly to the caller of
uv_make_secure .
any RC value that is not explicitly handled here will
result in -EINVAL. The caller has then to check for -EINVAL and check
the RC value, like do_non_secure_storage_access does.

so anything else that goes wrong is passed as-is to the caller.
for some things, like interrupt handlers, we simply don't care; if we
need to try again, we will try again, if we notice we can't continue,
the VM will get killed.

Re: [RFCv2 06/37] s390: add (non)secure page access exceptions handlers

[Thomas Huth]

On 04/02/2020 14.08, Claudio Imbrenda wrote:
> On Tue, 4 Feb 2020 13:48:47 +0100
> Thomas Huth <thuth@redhat.com> wrote:
> 
>> On 04/02/2020 12.41, Claudio Imbrenda wrote:
>>> On Tue, 4 Feb 2020 11:37:42 +0100
>>> Thomas Huth <thuth@redhat.com> wrote:
>>>
>>> [...]
>>>   
>>>>> ---
>>>>>  arch/s390/kernel/pgm_check.S |  4 +-
>>>>>  arch/s390/mm/fault.c         | 87
>>>>> ++++++++++++++++++++++++++++++++++++ 2 files changed, 89
>>>>> insertions(+), 2 deletions(-)    
>>>> [...]  
>>>>> +void do_non_secure_storage_access(struct pt_regs *regs)
>>>>> +{
>>>>> +	unsigned long gaddr = regs->int_parm_long &
>>>>> __FAIL_ADDR_MASK;
>>>>> +	struct gmap *gmap = (struct gmap *)S390_lowcore.gmap;
>>>>> +	struct uv_cb_cts uvcb = {
>>>>> +		.header.cmd = UVC_CMD_CONV_TO_SEC_STOR,
>>>>> +		.header.len = sizeof(uvcb),
>>>>> +		.guest_handle = gmap->se_handle,
>>>>> +		.gaddr = gaddr,
>>>>> +	};
>>>>> +	int rc;
>>>>> +
>>>>> +	if (get_fault_type(regs) != GMAP_FAULT) {
>>>>> +		do_fault_error(regs, VM_READ | VM_WRITE,
>>>>> VM_FAULT_BADMAP);
>>>>> +		WARN_ON_ONCE(1);
>>>>> +		return;
>>>>> +	}
>>>>> +
>>>>> +	rc = uv_make_secure(gmap, gaddr, &uvcb, 0);
>>>>> +	if (rc == -EINVAL && uvcb.header.rc != 0x104)
>>>>> +		send_sig(SIGSEGV, current, 0);
>>>>> +}    
>>>>
>>>> What about the other rc beside 0x104 that could happen here? They
>>>> go unnoticed?  
>>>
>>> no, they are handled in the uv_make_secure, and return an
>>> appropriate error code.   
>> Hmm, in patch 05/37, I basically see:
>>
>> +static int make_secure_pte(pte_t *ptep, unsigned long addr, void
>> *data) +{
>> [...]
>> +	rc = uv_call(0, (u64)params->uvcb);
>> +	page_ref_unfreeze(page, expected);
>> +	if (rc)
>> +		rc = (params->uvcb->rc == 0x10a) ? -ENXIO : -EINVAL;
>> +	return rc;
>> +}
>>
>> +int uv_make_secure(struct gmap *gmap, unsigned long gaddr, void
>> *uvcb, int pins)
>> +{
>> [...]
>> +	lock_page(params.page);
>> +	rc = apply_to_page_range(gmap->mm, uaddr, PAGE_SIZE,
>> make_secure_pte, &params);
>> +	unlock_page(params.page);
>> +out:
>> +	up_read(&gmap->mm->mmap_sem);
>> +
>> +	if (rc == -EBUSY) {
>> +		if (local_drain) {
>> +			lru_add_drain_all();
>> +			return -EAGAIN;
>> +		}
>> +		lru_add_drain();
>> +		local_drain = 1;
>> +		goto again;
>> +	} else if (rc == -ENXIO) {
>> +		if (gmap_fault(gmap, gaddr, FAULT_FLAG_WRITE))
>> +			return -EFAULT;
>> +		return -EAGAIN;
>> +	}
>> +	return rc;
>> +}
>>
>> So 0x10a result in -ENXIO and is handled ==> OK.
>> And 0x104 is handled in do_non_secure_storage_access ==> OK.
>>
>> But what about the other possible error codes? make_secure_pte()
>> returns -EINVAL in that case, but uv_make_secure() does not care
>> about that error code, and do_non_secure_storage_access() only cares
>> if uvcb.header.rc was 0x104 ... what did I miss?
> 
> basically, any error value that is not handled by uv_make_secure is
> passed as-is from make_secure_pte directly to the caller of
> uv_make_secure .
> any RC value that is not explicitly handled here will
> result in -EINVAL. The caller has then to check for -EINVAL and check
> the RC value, like do_non_secure_storage_access does.
> 
> so anything else that goes wrong is passed as-is to the caller.
> for some things, like interrupt handlers, we simply don't care; if we
> need to try again, we will try again, if we notice we can't continue,
> the VM will get killed.

In that case a comment in do_non_secure_storage_access() would be
helpful which states why we do not care about the other error codes in
uvcb.header.rc.

 Thomas

Re: [RFCv2 07/37] KVM: s390: add new variants of UV CALL

[Thomas Huth]

On 03/02/2020 14.19, Christian Borntraeger wrote:
> From: Janosch Frank <frankja@linux.ibm.com>
> 
> This add 2 new variants of the UV CALL.
> 
> The first variant handles UV CALLs that might have longer busy
> conditions or just need longer when doing partial completion. We should
> schedule when necessary.
> 
> The second variant handles UV CALLs that only need the handle but have
> no payload (e.g. destroying a VM). We can provide a simple wrapper for
> those.
> 
> Signed-off-by: Janosch Frank <frankja@linux.ibm.com>
> ---
>  arch/s390/include/asm/uv.h | 58 ++++++++++++++++++++++++++++++++++++++
>  1 file changed, 58 insertions(+)
> 
> diff --git a/arch/s390/include/asm/uv.h b/arch/s390/include/asm/uv.h
> index 4eaea95f5c64..3448f12ef57a 100644
> --- a/arch/s390/include/asm/uv.h
> +++ b/arch/s390/include/asm/uv.h
> @@ -14,6 +14,7 @@
>  #include <linux/types.h>
>  #include <linux/errno.h>
>  #include <linux/bug.h>
> +#include <linux/sched.h>
>  #include <asm/page.h>
>  #include <asm/gmap.h>
>  
> @@ -92,6 +93,18 @@ struct uv_cb_cfs {
>  	u64 paddr;
>  } __packed __aligned(8);
>  
> +/*
> + * A common UV call struct for the following calls:
> + * Destroy cpu/config
> + * Verify
> + */
> +struct uv_cb_nodata {
> +	struct uv_cb_header header;
> +	u64 reserved08[2];
> +	u64 handle;
> +	u64 reserved20[4];
> +} __packed __aligned(8);
> +
>  struct uv_cb_share {
>  	struct uv_cb_header header;
>  	u64 reserved08[3];
> @@ -99,6 +112,31 @@ struct uv_cb_share {
>  	u64 reserved28;
>  } __packed __aligned(8);
>  
> +/*
> + * Low level uv_call that takes r1 and r2 as parameter and avoids
> + * stalls for long running busy conditions by doing schedule
> + */
> +static inline int uv_call_sched(unsigned long r1, unsigned long r2)
> +{
> +	int cc = 3;
> +
> +	while (cc > 1) {
> +		asm volatile(
> +			"0:	.insn rrf,0xB9A40000,%[r1],%[r2],0,0\n"
> +			"		ipm	%[cc]\n"
> +			"		srl	%[cc],28\n"
> +			: [cc] "=d" (cc)
> +			: [r1] "a" (r1), [r2] "a" (r2)

You could use "d" instead of "a" for both, r1 and r2, here.

> +			: "memory", "cc");
> +		if (need_resched())
> +			schedule();
> +	}

It's a matter of taste, but I'd rather do:

   int cc;
   do {
       ...
   } while (cc > 1);

(i.e. no need to pre-initialize cc with 3)

> +	return cc;
> +}

 Thomas

Re: [RFCv2 07/37] KVM: s390: add new variants of UV CALL

[Christian Borntraeger]

On 04.02.20 13:11, Cornelia Huck wrote:
> On Mon,  3 Feb 2020 08:19:27 -0500
> Christian Borntraeger <borntraeger@de.ibm.com> wrote:
> 
>> From: Janosch Frank <frankja@linux.ibm.com>
>>
>> This add 2 new variants of the UV CALL.
>>
>> The first variant handles UV CALLs that might have longer busy
>> conditions or just need longer when doing partial completion. We should
>> schedule when necessary.
>>
>> The second variant handles UV CALLs that only need the handle but have
>> no payload (e.g. destroying a VM). We can provide a simple wrapper for
>> those.
>>
>> Signed-off-by: Janosch Frank <frankja@linux.ibm.com>
>> ---
>>  arch/s390/include/asm/uv.h | 58 ++++++++++++++++++++++++++++++++++++++
>>  1 file changed, 58 insertions(+)
>>
>> diff --git a/arch/s390/include/asm/uv.h b/arch/s390/include/asm/uv.h
>> index 4eaea95f5c64..3448f12ef57a 100644
>> --- a/arch/s390/include/asm/uv.h
>> +++ b/arch/s390/include/asm/uv.h
>> @@ -14,6 +14,7 @@
>>  #include <linux/types.h>
>>  #include <linux/errno.h>
>>  #include <linux/bug.h>
>> +#include <linux/sched.h>
>>  #include <asm/page.h>
>>  #include <asm/gmap.h>
>>  
>> @@ -92,6 +93,18 @@ struct uv_cb_cfs {
>>  	u64 paddr;
>>  } __packed __aligned(8);
>>  
>> +/*
>> + * A common UV call struct for the following calls:
> 
> "for calls that take no payload"?
> 
> In case there will be more of them in the future :)

ack. 

/*
 * A common UV call struct for calls that take no payload
 * Examples: 
 * Destroy cpu/config
 * Verify
 */




> 
>> + * Destroy cpu/config
>> + * Verify
>> + */
>> +struct uv_cb_nodata {
>> +	struct uv_cb_header header;
>> +	u64 reserved08[2];
>> +	u64 handle;
>> +	u64 reserved20[4];
>> +} __packed __aligned(8);
>> +
>>  struct uv_cb_share {
>>  	struct uv_cb_header header;
>>  	u64 reserved08[3];
>> @@ -99,6 +112,31 @@ struct uv_cb_share {
>>  	u64 reserved28;
>>  } __packed __aligned(8);
>>  
>> +/*
>> + * Low level uv_call that takes r1 and r2 as parameter and avoids
>> + * stalls for long running busy conditions by doing schedule
> 
> This can only ever return 0 or 1, right?

Right. But the r2 parameter is usually an uvcb so callers can fiddle out
more information (like rc) themselves. 
> 
>> + */
>> +static inline int uv_call_sched(unsigned long r1, unsigned long r2)
>> +{
>> +	int cc = 3;
>> +
>> +	while (cc > 1) {
>> +		asm volatile(
>> +			"0:	.insn rrf,0xB9A40000,%[r1],%[r2],0,0\n"
>> +			"		ipm	%[cc]\n"
>> +			"		srl	%[cc],28\n"
>> +			: [cc] "=d" (cc)
>> +			: [r1] "a" (r1), [r2] "a" (r2)
>> +			: "memory", "cc");
>> +		if (need_resched())
>> +			schedule();
>> +	}
>> +	return cc;
>> +}
>> +
>> +/*
>> + * Low level uv_call that takes r1 and r2 as parameter
>> + */
>>  static inline int uv_call(unsigned long r1, unsigned long r2)
>>  {
>>  	int cc;
>> @@ -114,6 +152,26 @@ static inline int uv_call(unsigned long r1, unsigned long r2)
>>  	return cc;
>>  }
>>  
>> +/*
>> + * special variant of uv_call that only transport the cpu or guest
> 
> s/transport/transports/

ack.

Re: [RFCv2 07/37] KVM: s390: add new variants of UV CALL

[Christian Borntraeger]

On 04.02.20 14:27, Thomas Huth wrote:
>> +/*
>> + * Low level uv_call that takes r1 and r2 as parameter and avoids
>> + * stalls for long running busy conditions by doing schedule
>> + */
>> +static inline int uv_call_sched(unsigned long r1, unsigned long r2)
>> +{
>> +	int cc = 3;
>> +
>> +	while (cc > 1) {
>> +		asm volatile(
>> +			"0:	.insn rrf,0xB9A40000,%[r1],%[r2],0,0\n"
>> +			"		ipm	%[cc]\n"
>> +			"		srl	%[cc],28\n"
>> +			: [cc] "=d" (cc)
>> +			: [r1] "a" (r1), [r2] "a" (r2)
> 
> You could use "d" instead of "a" for both, r1 and r2, here.

Indeed. Fixed.

> 
>> +			: "memory", "cc");
>> +		if (need_resched())
>> +			schedule();
>> +	}
> 
> It's a matter of taste, but I'd rather do:
> 
>    int cc;
>    do {
>        ...
>    } while (cc > 1);
> 
> (i.e. no need to pre-initialize cc with 3)

Yes, I like that better.
> 
>> +	return cc;
>> +}
> 
>  Thomas
> 

Re: [RFCv2 08/37] KVM: s390: protvirt: Add initial lifecycle handling

[Cornelia Huck]

On Mon,  3 Feb 2020 08:19:28 -0500
Christian Borntraeger <borntraeger@de.ibm.com> wrote:

> From: Janosch Frank <frankja@linux.ibm.com>
> 
> This contains 3 main changes:
> 1. changes in SIE control block handling for secure guests
> 2. helper functions for create/destroy/unpack secure guests
> 3. KVM_S390_PV_COMMAND ioctl to allow userspace dealing with secure
> machines
> 
> Signed-off-by: Janosch Frank <frankja@linux.ibm.com>
> ---
>  arch/s390/include/asm/kvm_host.h |  24 ++-
>  arch/s390/include/asm/uv.h       |  60 ++++++++
>  arch/s390/kvm/Makefile           |   2 +-
>  arch/s390/kvm/kvm-s390.c         | 198 ++++++++++++++++++++++++-
>  arch/s390/kvm/kvm-s390.h         |  45 ++++++
>  arch/s390/kvm/pv.c               | 246 +++++++++++++++++++++++++++++++
>  include/uapi/linux/kvm.h         |  33 +++++
>  7 files changed, 604 insertions(+), 4 deletions(-)
>  create mode 100644 arch/s390/kvm/pv.c
> 
(...)
> @@ -80,6 +95,32 @@ struct uv_cb_init {
>  
>  } __packed __aligned(8);
>  
> +struct uv_cb_cgc {

Given that we now have a bunch of structs of the form uv_cb_TLA, can we
add a comment to each for what uv call they are?

> +	struct uv_cb_header header;
> +	u64 reserved08[2];
> +	u64 guest_handle;
> +	u64 conf_base_stor_origin;
> +	u64 conf_var_stor_origin;
> +	u64 reserved30;
> +	u64 guest_stor_origin;
> +	u64 guest_stor_len;
> +	u64 guest_sca;
> +	u64 guest_asce;
> +	u64 reserved60[5];
> +} __packed __aligned(8);

(...)

> +#ifdef CONFIG_KVM_S390_PROTECTED_VIRTUALIZATION_HOST
> +static int kvm_s390_handle_pv(struct kvm *kvm, struct kvm_pv_cmd *cmd)
> +{
> +	int r = 0;
> +	void __user *argp = (void __user *)cmd->data;
> +
> +	switch (cmd->cmd) {
> +	case KVM_PV_VM_CREATE: {
> +		r = -EINVAL;
> +		if (kvm_s390_pv_is_protected(kvm))
> +			break;
> +
> +		r = kvm_s390_pv_alloc_vm(kvm);
> +		if (r)
> +			break;
> +
> +		mutex_lock(&kvm->lock);
> +		kvm_s390_vcpu_block_all(kvm);
> +		/* FMT 4 SIE needs esca */
> +		r = sca_switch_to_extended(kvm);
> +		if (!r)
> +			r = kvm_s390_pv_create_vm(kvm);

If sca_switch_to_extended() fails, you don't call
kvm_s390_pv_dealloc_vm(). Also, kvm_s390_pv_create_vm() _does_ call
_dealloc_vm() on failure, which seems a bit surprising. I'd probably
move the _dealloc_vm() out of the error path of _create_vm() and call
it here for r != 0.

> +		kvm_s390_vcpu_unblock_all(kvm);
> +		mutex_unlock(&kvm->lock);
> +		break;
> +	}

(...)

Re: [RFCv2 13/37] KVM: s390: protvirt: Instruction emulation

[Cornelia Huck]

On Tue, 4 Feb 2020 13:29:46 +0100
Christian Borntraeger <borntraeger@de.ibm.com> wrote:

> On 04.02.20 13:20, David Hildenbrand wrote:
> > On 03.02.20 14:19, Christian Borntraeger wrote:  
> >> From: Janosch Frank <frankja@linux.ibm.com>
> >>
> >> We have two new SIE exit codes dealing with instructions.
> >> 104 (0x68) for a secure instruction interception, on which the SIE needs
> >> hypervisor action to complete the instruction. We can piggy-back on the
> >> existing instruction handlers.
> >>
> >> 108 which is merely a notification and provides data for tracking and
> >> management. For example this is used to tell the host about a new value
> >> for the prefix register. As there will be several special case handlers
> >> in later patches, we handle this in a separate function.
> >>
> >> Signed-off-by: Janosch Frank <frankja@linux.ibm.com>
> >> ---
> >>  arch/s390/include/asm/kvm_host.h |  2 ++
> >>  arch/s390/kvm/intercept.c        | 11 +++++++++++
> >>  2 files changed, 13 insertions(+)
> >>
> >> diff --git a/arch/s390/include/asm/kvm_host.h b/arch/s390/include/asm/kvm_host.h
> >> index d63ed05272ec..58845b315be0 100644
> >> --- a/arch/s390/include/asm/kvm_host.h
> >> +++ b/arch/s390/include/asm/kvm_host.h
> >> @@ -217,6 +217,8 @@ struct kvm_s390_sie_block {
> >>  #define ICPT_KSS	0x5c
> >>  #define ICPT_PV_MCHKR	0x60
> >>  #define ICPT_PV_INT_EN	0x64
> >> +#define ICPT_PV_INSTR	0x68
> >> +#define ICPT_PV_NOTIF	0x6c  
> > 
> > NOTIFY? NOTIFICATION? NOTIF is weird.  
> 
> ack. I used NOTIFY to keep the numbers aligned.
> 
> >   
> >>  	__u8	icptcode;		/* 0x0050 */
> >>  	__u8	icptstatus;		/* 0x0051 */
> >>  	__u16	ihcpu;			/* 0x0052 */
> >> diff --git a/arch/s390/kvm/intercept.c b/arch/s390/kvm/intercept.c
> >> index eaa2a21c3170..4b3fbbde1674 100644
> >> --- a/arch/s390/kvm/intercept.c
> >> +++ b/arch/s390/kvm/intercept.c
> >> @@ -444,6 +444,11 @@ static int handle_operexc(struct kvm_vcpu *vcpu)
> >>  	return kvm_s390_inject_program_int(vcpu, PGM_OPERATION);
> >>  }
> >>  
> >> +static int handle_pv_not(struct kvm_vcpu *vcpu)  
> > 
> > "notification" please. not not. You see why ;)  
> 
> ack. 
> 

With these changes on top:

Reviewed-by: Cornelia Huck <cohuck@redhat.com>

Re: [RFCv2 27/37] KVM: s390: protvirt: Only sync fmt4 registers

[Cornelia Huck]

On Mon,  3 Feb 2020 08:19:47 -0500
Christian Borntraeger <borntraeger@de.ibm.com> wrote:

> From: Janosch Frank <frankja@linux.ibm.com>
> 
> A lot of the registers are controlled by the Ultravisor and never
> visible to KVM. Also some registers are overlayed, like gbea is with
> sidad, which might leak data to userspace.
> 
> Hence we sync a minimal set of registers for both SIE formats and then
> check and sync format 2 registers if necessary.
> 
> Also we disable set/get one reg for the same reason. It's an old
> interface anyway.

Didn't you already do that in the previous patch?

> 
> Signed-off-by: Janosch Frank <frankja@linux.ibm.com>
> Signed-off-by: Christian Borntraeger <borntraeger@de.ibm.com>
> [Fixes and patch splitting]
> ---
>  arch/s390/kvm/kvm-s390.c | 116 ++++++++++++++++++++++++---------------
>  1 file changed, 72 insertions(+), 44 deletions(-)

Re: [RFCv2 30/37] KVM: s390: protvirt: Add diag 308 subcode 8 - 10 handling

[Cornelia Huck]

On Mon,  3 Feb 2020 08:19:50 -0500
Christian Borntraeger <borntraeger@de.ibm.com> wrote:

> From: Janosch Frank <frankja@linux.ibm.com>
> 
> If the host initialized the Ultravisor, we can set stfle bit 161
> (protected virtual IPL enhancements facility), which indicates, that

s/indicates,/indicates/

> the IPL subcodes 8, 9 and are valid. These subcodes are used by a

s/9 and/9, and 10/

> normal guest to set/retrieve a IPIB of type 5 and transition into

"an IPL information block of type 5 (for PVMs)" ?

> protected mode.
> 
> Once in protected mode, the Ultravisor will conceal the facility
> bit. Therefore each boot into protected mode has to go through
> non-protected. There is no secure re-ipl with subcode 10 without a

"non-protected mode"

> previous subcode 3.
> 
> In protected mode, there is no subcode 4 available, as the VM has no
> more access to its memory from non-protected mode. I.e. each IPL
> clears.

"i.e., only IPL clear is possible" ?

> 
> Signed-off-by: Janosch Frank <frankja@linux.ibm.com>
> ---
>  arch/s390/kvm/diag.c     | 6 ++++++
>  arch/s390/kvm/kvm-s390.c | 5 +++++
>  2 files changed, 11 insertions(+)

Reviewed-by: Cornelia Huck <cohuck@redhat.com>

Re: [RFCv2 09/37] KVM: s390: protvirt: Add KVM api documentation

[Thomas Huth]

On 03/02/2020 14.19, Christian Borntraeger wrote:
> From: Janosch Frank <frankja@linux.ibm.com>
> 
> Add documentation for KVM_CAP_S390_PROTECTED capability and the
> KVM_S390_PV_COMMAND and KVM_S390_PV_COMMAND_VCPU ioctls.
> 
> Signed-off-by: Janosch Frank <frankja@linux.ibm.com>
> ---
>  Documentation/virt/kvm/api.txt | 62 ++++++++++++++++++++++++++++++++++
>  1 file changed, 62 insertions(+)
> 
> diff --git a/Documentation/virt/kvm/api.txt b/Documentation/virt/kvm/api.txt
> index 73448764f544..a73fdae40e26 100644
> --- a/Documentation/virt/kvm/api.txt
> +++ b/Documentation/virt/kvm/api.txt
> @@ -4204,6 +4204,61 @@ the clear cpu reset definition in the POP. However, the cpu is not put
>  into ESA mode. This reset is a superset of the initial reset.
>  
>  
> +4.125 KVM_S390_PV_COMMAND
> +
> +Capability: KVM_CAP_S390_PROTECTED
> +Architectures: s390
> +Type: vm ioctl
> +Parameters: struct kvm_pv_cmd
> +Returns: 0 on success, < 0 on error
> +
> +struct kvm_pv_cmd {
> +	__u32	cmd;	/* Command to be executed */
> +	__u16	rc;	/* Ultravisor return code */
> +	__u16	rrc;	/* Ultravisor return reason code */
> +	__u64	data;	/* Data or address */
> +};
> +
> +cmd values:
> +KVM_PV_VM_CREATE
> +Allocate memory and register the VM with the Ultravisor, thereby
> +donating memory to the Ultravisor making it inaccessible to KVM.
> +
> +KVM_PV_VM_DESTROY
> +Unregisters the VM from the Ultravisor and frees memory that was

s/Unregisters/Deregisters/ ?

> +donated, so the kernel can use it again. All registered VCPUs have to
> +be unregistered beforehand and all memory has to be exported or
> +shared.
> +
> +KVM_PV_VM_SET_SEC_PARMS
> +Pass the image header from VM memory to the Ultravisor in preparation
> +of image unpacking and verification.
> +
> +KVM_PV_VM_UNPACK
> +Unpack (protect and decrypt) a page of the encrypted boot image.
> +
> +KVM_PV_VM_VERIFY
> +Verify the integrity of the unpacked image. Only if this succeeds, KVM
> +
> +is allowed to start protected VCPUs.

Please remove the empty line between "KVM" and "is allowed".

Apart from the two nits, the patch looks fine to me.

 Thomas

Re: [RFCv2 11/37] KVM: s390/mm: Make pages accessible before destroying the guest

[Thomas Huth]

On 03/02/2020 14.19, Christian Borntraeger wrote:
> Before we destroy the secure configuration, we better make all
> pages accessible again. This also happens during reboot, where we reboot
> into a non-secure guest that then can go again into a secure mode. As
> this "new" secure guest will have a new ID we cannot reuse the old page
> state.
> 
> Signed-off-by: Christian Borntraeger <borntraeger@de.ibm.com>
> ---
>  arch/s390/include/asm/pgtable.h |  1 +
>  arch/s390/kvm/pv.c              |  2 ++
>  arch/s390/mm/gmap.c             | 35 +++++++++++++++++++++++++++++++++
>  3 files changed, 38 insertions(+)
> 
> diff --git a/arch/s390/include/asm/pgtable.h b/arch/s390/include/asm/pgtable.h
> index 65b6bb47af0a..6e167dcc35f1 100644
> --- a/arch/s390/include/asm/pgtable.h
> +++ b/arch/s390/include/asm/pgtable.h
> @@ -1669,6 +1669,7 @@ extern int vmem_remove_mapping(unsigned long start, unsigned long size);
>  extern int s390_enable_sie(void);
>  extern int s390_enable_skey(void);
>  extern void s390_reset_cmma(struct mm_struct *mm);
> +extern void s390_reset_acc(struct mm_struct *mm);
>  
>  /* s390 has a private copy of get unmapped area to deal with cache synonyms */
>  #define HAVE_ARCH_UNMAPPED_AREA
> diff --git a/arch/s390/kvm/pv.c b/arch/s390/kvm/pv.c
> index a867b9e9c069..24d802072ac7 100644
> --- a/arch/s390/kvm/pv.c
> +++ b/arch/s390/kvm/pv.c
> @@ -66,6 +66,8 @@ int kvm_s390_pv_destroy_vm(struct kvm *kvm)
>  	int rc;
>  	u32 ret;
>  
> +	/* make all pages accessible before destroying the guest */
> +	s390_reset_acc(kvm->mm);
>  	rc = uv_cmd_nodata(kvm_s390_pv_handle(kvm),
>  			   UVC_CMD_DESTROY_SEC_CONF, &ret);
>  	WRITE_ONCE(kvm->arch.gmap->se_handle, 0);
> diff --git a/arch/s390/mm/gmap.c b/arch/s390/mm/gmap.c
> index bf365a09f900..0b00e8d5fa39 100644
> --- a/arch/s390/mm/gmap.c
> +++ b/arch/s390/mm/gmap.c
> @@ -2648,3 +2648,38 @@ void s390_reset_cmma(struct mm_struct *mm)
>  	up_write(&mm->mmap_sem);
>  }
>  EXPORT_SYMBOL_GPL(s390_reset_cmma);
> +
> +/*
> + * make inaccessible pages accessible again
> + */
> +static int __s390_reset_acc(pte_t *ptep, unsigned long addr,
> +			    unsigned long next, struct mm_walk *walk)
> +{
> +	pte_t pte = READ_ONCE(*ptep);
> +
> +	if (pte_present(pte))
> +		uv_convert_from_secure(pte_val(pte) & PAGE_MASK);

Is it ok to ignore the return value from uv_convert_from_secure() ?
Problems might go unnoticed ... maybe use at least a WARN_ONCE ?

> +	return 0;
> +}
> +
> +static const struct mm_walk_ops reset_acc_walk_ops = {
> +	.pte_entry		= __s390_reset_acc,
> +};
> +
> +#include <linux/sched/mm.h>
> +void s390_reset_acc(struct mm_struct *mm)
> +{
> +	/*
> +	 * we might be called during
> +	 * reset:                            we walk the pages and clear
> +	 * close of all kvm file descriptor: we walk the pages and clear
> +	 * exit of process on fd closure:    vma already gone, do nothing
> +	 */
> +	if (!mmget_not_zero(mm))
> +		return;
> +	down_read(&mm->mmap_sem);
> +	walk_page_range(mm, 0, TASK_SIZE, &reset_acc_walk_ops, NULL);
> +	up_read(&mm->mmap_sem);
> +	mmput(mm);
> +}
> +EXPORT_SYMBOL_GPL(s390_reset_acc);
> 

Anyway,
Reviewed-by: Thomas Huth <thuth@redhat.com>

Re: [RFCv2 08/37] KVM: s390: protvirt: Add initial lifecycle handling

[Christian Borntraeger]

On 04.02.20 17:27, Cornelia Huck wrote:

>> +struct uv_cb_cgc {
> 
> Given that we now have a bunch of structs of the form uv_cb_TLA, can we
> add a comment to each for what uv call they are?

ack.

> 
>> +	struct uv_cb_header header;
>> +	u64 reserved08[2];
>> +	u64 guest_handle;
>> +	u64 conf_base_stor_origin;
>> +	u64 conf_var_stor_origin;
>> +	u64 reserved30;
>> +	u64 guest_stor_origin;
>> +	u64 guest_stor_len;
>> +	u64 guest_sca;
>> +	u64 guest_asce;
>> +	u64 reserved60[5];
>> +} __packed __aligned(8);
> 
> (...)
> 
>> +#ifdef CONFIG_KVM_S390_PROTECTED_VIRTUALIZATION_HOST
>> +static int kvm_s390_handle_pv(struct kvm *kvm, struct kvm_pv_cmd *cmd)
>> +{
>> +	int r = 0;
>> +	void __user *argp = (void __user *)cmd->data;
>> +
>> +	switch (cmd->cmd) {
>> +	case KVM_PV_VM_CREATE: {
>> +		r = -EINVAL;
>> +		if (kvm_s390_pv_is_protected(kvm))
>> +			break;
>> +
>> +		r = kvm_s390_pv_alloc_vm(kvm);
>> +		if (r)
>> +			break;
>> +
>> +		mutex_lock(&kvm->lock);
>> +		kvm_s390_vcpu_block_all(kvm);
>> +		/* FMT 4 SIE needs esca */
>> +		r = sca_switch_to_extended(kvm);
>> +		if (!r)
>> +			r = kvm_s390_pv_create_vm(kvm);
> 
> If sca_switch_to_extended() fails, you don't call
> kvm_s390_pv_dealloc_vm(). Also, kvm_s390_pv_create_vm() _does_ call
> _dealloc_vm() on failure, which seems a bit surprising. I'd probably
> move the _dealloc_vm() out of the error path of _create_vm() and call
> it here for r != 0.

Indeed this is fishy. Will fix. I might also want to split this patch into
an ultravisor related part and a kvm user interface part. then this would
have been more obvious and it could be easier to try Davids proposal regarding
enable/disable.   Its not that easyto split though, not sure if this works out. 
> 
>> +		kvm_s390_vcpu_unblock_all(kvm);
>> +		mutex_unlock(&kvm->lock);
>> +		break;
>> +	}
> 
> (...)
> 

Re: [RFCv2 12/37] KVM: s390: protvirt: Handle SE notification interceptions

[Thomas Huth]

On 03/02/2020 14.19, Christian Borntraeger wrote:
> From: Janosch Frank <frankja@linux.ibm.com>
> 
> Since KVM doesn't emulate any form of load control and load psw
> instructions anymore, we wouldn't get an interception if PSWs or CRs
> are changed in the guest. That means we can't inject IRQs right after
> the guest is enabled for them.

I had to read that twice to understand it. I'd suggest maybe rather
something like:

Since there is no interception for load control and load psw instruction
in the protected mode, we need a new way to get notified whether we have
to inject an IRQ right after the guest has just enabled the possibility
for receiving them.

> The new interception codes solve that problem by being a notification

maybe s/being/providing/

> for changes to IRQ enablement relevant bits in CRs 0, 6 and 14, as
> well a the machine check mask bit in the PSW.
> 
> No special handling is needed for these interception codes, the KVM
> pre-run code will consult all necessary CRs and PSW bits and inject
> IRQs the guest is enabled for.
> 
> Signed-off-by: Janosch Frank <frankja@linux.ibm.com>
> ---
>  arch/s390/include/asm/kvm_host.h |  2 ++
>  arch/s390/kvm/intercept.c        | 10 ++++++++++
>  2 files changed, 12 insertions(+)
> 
> diff --git a/arch/s390/include/asm/kvm_host.h b/arch/s390/include/asm/kvm_host.h
> index 841690d05080..d63ed05272ec 100644
> --- a/arch/s390/include/asm/kvm_host.h
> +++ b/arch/s390/include/asm/kvm_host.h
> @@ -215,6 +215,8 @@ struct kvm_s390_sie_block {
>  #define ICPT_PARTEXEC	0x38
>  #define ICPT_IOINST	0x40
>  #define ICPT_KSS	0x5c
> +#define ICPT_PV_MCHKR	0x60
> +#define ICPT_PV_INT_EN	0x64
>  	__u8	icptcode;		/* 0x0050 */
>  	__u8	icptstatus;		/* 0x0051 */
>  	__u16	ihcpu;			/* 0x0052 */
> diff --git a/arch/s390/kvm/intercept.c b/arch/s390/kvm/intercept.c
> index a389fa85cca2..eaa2a21c3170 100644
> --- a/arch/s390/kvm/intercept.c
> +++ b/arch/s390/kvm/intercept.c
> @@ -480,6 +480,16 @@ int kvm_handle_sie_intercept(struct kvm_vcpu *vcpu)
>  	case ICPT_KSS:
>  		rc = kvm_s390_skey_check_enable(vcpu);
>  		break;
> +	case ICPT_PV_MCHKR:
> +		/* fallthrough */
> +	case ICPT_PV_INT_EN:
> +		/*
> +		 * PSW bit 13 or a CR (0, 6, 14) changed and we might
> +		 * now be able to deliver interrupts. The pre-run code
> +		 * will take care of this.
> +		 */
> +		rc = 0;
> +		break;
>  	default:
>  		return -EOPNOTSUPP;
>  	}

With "fallthrough" removed and the commit message improved:

Reviewed-by: Thomas Huth <thuth@redhat.com>

Re: [RFCv2 13/37] KVM: s390: protvirt: Instruction emulation

[Thomas Huth]

On 04/02/2020 18.07, Cornelia Huck wrote:
> On Tue, 4 Feb 2020 13:29:46 +0100
> Christian Borntraeger <borntraeger@de.ibm.com> wrote:
> 
>> On 04.02.20 13:20, David Hildenbrand wrote:
>>> On 03.02.20 14:19, Christian Borntraeger wrote:  
>>>> From: Janosch Frank <frankja@linux.ibm.com>
>>>>
>>>> We have two new SIE exit codes dealing with instructions.
>>>> 104 (0x68) for a secure instruction interception, on which the SIE needs
>>>> hypervisor action to complete the instruction. We can piggy-back on the
>>>> existing instruction handlers.
>>>>
>>>> 108 which is merely a notification and provides data for tracking and
>>>> management. For example this is used to tell the host about a new value
>>>> for the prefix register. As there will be several special case handlers
>>>> in later patches, we handle this in a separate function.
>>>>
>>>> Signed-off-by: Janosch Frank <frankja@linux.ibm.com>
>>>> ---
>>>>  arch/s390/include/asm/kvm_host.h |  2 ++
>>>>  arch/s390/kvm/intercept.c        | 11 +++++++++++
>>>>  2 files changed, 13 insertions(+)
>>>>
>>>> diff --git a/arch/s390/include/asm/kvm_host.h b/arch/s390/include/asm/kvm_host.h
>>>> index d63ed05272ec..58845b315be0 100644
>>>> --- a/arch/s390/include/asm/kvm_host.h
>>>> +++ b/arch/s390/include/asm/kvm_host.h
>>>> @@ -217,6 +217,8 @@ struct kvm_s390_sie_block {
>>>>  #define ICPT_KSS	0x5c
>>>>  #define ICPT_PV_MCHKR	0x60
>>>>  #define ICPT_PV_INT_EN	0x64
>>>> +#define ICPT_PV_INSTR	0x68
>>>> +#define ICPT_PV_NOTIF	0x6c  
>>>
>>> NOTIFY? NOTIFICATION? NOTIF is weird.  
>>
>> ack. I used NOTIFY to keep the numbers aligned.
>>
>>>   
>>>>  	__u8	icptcode;		/* 0x0050 */
>>>>  	__u8	icptstatus;		/* 0x0051 */
>>>>  	__u16	ihcpu;			/* 0x0052 */
>>>> diff --git a/arch/s390/kvm/intercept.c b/arch/s390/kvm/intercept.c
>>>> index eaa2a21c3170..4b3fbbde1674 100644
>>>> --- a/arch/s390/kvm/intercept.c
>>>> +++ b/arch/s390/kvm/intercept.c
>>>> @@ -444,6 +444,11 @@ static int handle_operexc(struct kvm_vcpu *vcpu)
>>>>  	return kvm_s390_inject_program_int(vcpu, PGM_OPERATION);
>>>>  }
>>>>  
>>>> +static int handle_pv_not(struct kvm_vcpu *vcpu)  
>>>
>>> "notification" please. not not. You see why ;)  
>>
>> ack. 
>>
> 
> With these changes on top:

I concur, with the changes on top:

Reviewed-by: Thomas Huth <thuth@redhat.com>

Re: [RFCv2 09/37] KVM: s390: protvirt: Add KVM api documentation

[Christian Borntraeger]

On 04.02.20 21:19, Thomas Huth wrote:
[...]
>> +KVM_PV_VM_DESTROY
>> +Unregisters the VM from the Ultravisor and frees memory that was
> 
> s/Unregisters/Deregisters/ ?

ack

> 
>> +donated, so the kernel can use it again. All registered VCPUs have to
>> +be unregistered beforehand and all memory has to be exported or
>> +shared.
>> +
>> +KVM_PV_VM_SET_SEC_PARMS
>> +Pass the image header from VM memory to the Ultravisor in preparation
>> +of image unpacking and verification.
>> +
>> +KVM_PV_VM_UNPACK
>> +Unpack (protect and decrypt) a page of the encrypted boot image.
>> +
>> +KVM_PV_VM_VERIFY
>> +Verify the integrity of the unpacked image. Only if this succeeds, KVM
>> +
>> +is allowed to start protected VCPUs.
> 
> Please remove the empty line between "KVM" and "is allowed".

ack

Re: [RFCv2 12/37] KVM: s390: protvirt: Handle SE notification interceptions

[Christian Borntraeger]

On 04.02.20 22:16, Thomas Huth wrote:
> On 03/02/2020 14.19, Christian Borntraeger wrote:
>> From: Janosch Frank <frankja@linux.ibm.com>
>>
>> Since KVM doesn't emulate any form of load control and load psw
>> instructions anymore, we wouldn't get an interception if PSWs or CRs
>> are changed in the guest. That means we can't inject IRQs right after
>> the guest is enabled for them.
> 
> I had to read that twice to understand it. I'd suggest maybe rather
> something like:
> 
> Since there is no interception for load control and load psw instruction
> in the protected mode, we need a new way to get notified whether we have
> to inject an IRQ right after the guest has just enabled the possibility
> for receiving them.

I will use this with minor tweaks.
> 
>> The new interception codes solve that problem by being a notification
> 
> maybe s/being/providing/

ack

Re: [RFCv2 27/37] KVM: s390: protvirt: Only sync fmt4 registers

[Christian Borntraeger]

On 04.02.20 18:15, Cornelia Huck wrote:
> On Mon,  3 Feb 2020 08:19:47 -0500
> Christian Borntraeger <borntraeger@de.ibm.com> wrote:
> 
>> From: Janosch Frank <frankja@linux.ibm.com>
>>
>> A lot of the registers are controlled by the Ultravisor and never
>> visible to KVM. Also some registers are overlayed, like gbea is with
>> sidad, which might leak data to userspace.
>>
>> Hence we sync a minimal set of registers for both SIE formats and then
>> check and sync format 2 registers if necessary.
>>
>> Also we disable set/get one reg for the same reason. It's an old
>> interface anyway.
> 
> Didn't you already do that in the previous patch?

Yes, a leftover from my patch splitting.
Fixed. 

> 
>>
>> Signed-off-by: Janosch Frank <frankja@linux.ibm.com>
>> Signed-off-by: Christian Borntraeger <borntraeger@de.ibm.com>
>> [Fixes and patch splitting]
>> ---
>>  arch/s390/kvm/kvm-s390.c | 116 ++++++++++++++++++++++++---------------
>>  1 file changed, 72 insertions(+), 44 deletions(-)
> 

Re: [RFCv2 30/37] KVM: s390: protvirt: Add diag 308 subcode 8 - 10 handling

[Christian Borntraeger]

On 04.02.20 19:51, Cornelia Huck wrote:
> On Mon,  3 Feb 2020 08:19:50 -0500
> Christian Borntraeger <borntraeger@de.ibm.com> wrote:
> 
>> From: Janosch Frank <frankja@linux.ibm.com>
>>
>> If the host initialized the Ultravisor, we can set stfle bit 161
>> (protected virtual IPL enhancements facility), which indicates, that
> 
> s/indicates,/indicates/

ack

> 
>> the IPL subcodes 8, 9 and are valid. These subcodes are used by a
> 
> s/9 and/9, and 10/

ack
> 
>> normal guest to set/retrieve a IPIB of type 5 and transition into
> 
> "an IPL information block of type 5 (for PVMs)" ?

ack

> 
>> protected mode.
>>
>> Once in protected mode, the Ultravisor will conceal the facility
>> bit. Therefore each boot into protected mode has to go through
>> non-protected. There is no secure re-ipl with subcode 10 without a
> 
> "non-protected mode"

ack

 
>> previous subcode 3.
>>
>> In protected mode, there is no subcode 4 available, as the VM has no
>> more access to its memory from non-protected mode. I.e. each IPL
>> clears.
> 
> "i.e., only IPL clear is possible" ?

ack
> 
>>
>> Signed-off-by: Janosch Frank <frankja@linux.ibm.com>
>> ---
>>  arch/s390/kvm/diag.c     | 6 ++++++
>>  arch/s390/kvm/kvm-s390.c | 5 +++++
>>  2 files changed, 11 insertions(+)
> 
> Reviewed-by: Cornelia Huck <cohuck@redhat.com>
> 

Re: [RFCv2 14/37] KVM: s390: protvirt: Add interruption injection controls

[Thomas Huth]

On 03/02/2020 14.19, Christian Borntraeger wrote:
> From: Michael Mueller <mimu@linux.ibm.com>
> 
> Define the interruption injection codes and the related fields in the
> sie control block for PVM interruption injection.

You seem to only add the details for external interrupts and I/O
interrupts here? Maybe mention this in the description ... otherwise it
is confusing when you read patch 17 later ... or maybe merge this patch
here with patch 17 ?

> Signed-off-by: Michael Mueller <mimu@linux.ibm.com>
> ---
>  arch/s390/include/asm/kvm_host.h | 25 +++++++++++++++++++++----
>  1 file changed, 21 insertions(+), 4 deletions(-)
> 
> diff --git a/arch/s390/include/asm/kvm_host.h b/arch/s390/include/asm/kvm_host.h
> index 58845b315be0..a45d10d87a8a 100644
> --- a/arch/s390/include/asm/kvm_host.h
> +++ b/arch/s390/include/asm/kvm_host.h
> @@ -222,7 +222,15 @@ struct kvm_s390_sie_block {
>  	__u8	icptcode;		/* 0x0050 */
>  	__u8	icptstatus;		/* 0x0051 */
>  	__u16	ihcpu;			/* 0x0052 */
> -	__u8	reserved54[2];		/* 0x0054 */
> +	__u8	reserved54;		/* 0x0054 */
> +#define IICTL_CODE_NONE		 0x00
> +#define IICTL_CODE_MCHK		 0x01
> +#define IICTL_CODE_EXT		 0x02
> +#define IICTL_CODE_IO		 0x03
> +#define IICTL_CODE_RESTART	 0x04
> +#define IICTL_CODE_SPECIFICATION 0x10
> +#define IICTL_CODE_OPERAND	 0x11
> +	__u8	iictl;			/* 0x0055 */
>  	__u16	ipa;			/* 0x0056 */
>  	__u32	ipb;			/* 0x0058 */
>  	__u32	scaoh;			/* 0x005c */
> @@ -259,7 +267,8 @@ struct kvm_s390_sie_block {
>  #define HPID_KVM	0x4
>  #define HPID_VSIE	0x5
>  	__u8	hpid;			/* 0x00b8 */
> -	__u8	reservedb9[11];		/* 0x00b9 */
> +	__u8	reservedb9[7];		/* 0x00b9 */
> +	__u32	eiparams;		/* 0x00c0 */
>  	__u16	extcpuaddr;		/* 0x00c4 */
>  	__u16	eic;			/* 0x00c6 */
>  	__u32	reservedc8;		/* 0x00c8 */
> @@ -275,8 +284,16 @@ struct kvm_s390_sie_block {
>  	__u8	oai;			/* 0x00e2 */
>  	__u8	armid;			/* 0x00e3 */
>  	__u8	reservede4[4];		/* 0x00e4 */
> -	__u64	tecmc;			/* 0x00e8 */
> -	__u8	reservedf0[12];		/* 0x00f0 */
> +	union {
> +		__u64	tecmc;		/* 0x00e8 */
> +		struct {
> +			__u16	subchannel_id;	/* 0x00e8 */
> +			__u16	subchannel_nr;	/* 0x00ea */
> +			__u32	io_int_parm;	/* 0x00ec */
> +			__u32	io_int_word;	/* 0x00f0 */
> +		};
> +	} __packed;
> +	__u8	reservedf4[8];		/* 0x00f4 */

Maybe add a comment to the new struct for which injection type it is
good for ... otherwise this might get hard to understand in the future
(especially if more stuff gets added like in patch 17).

Anyway,
Reviewed-by: Thomas Huth <thuth@redhat.com>

Re: [RFCv2 17/37] KVM: s390: protvirt: Add machine-check interruption injection controls

[Thomas Huth]

On 03/02/2020 14.19, Christian Borntraeger wrote:
> From: Michael Mueller <mimu@linux.ibm.com>
> 
> The following fields are added to the sie control block type 4:
>      - Machine Check Interruption Code (mcic)
>      - External Damage Code (edc)
>      - Failing Storage Address (faddr)
> 
> Signed-off-by: Michael Mueller <mimu@linux.ibm.com>
> ---
>  arch/s390/include/asm/kvm_host.h | 33 +++++++++++++++++++++++---------
>  1 file changed, 24 insertions(+), 9 deletions(-)
> 
> diff --git a/arch/s390/include/asm/kvm_host.h b/arch/s390/include/asm/kvm_host.h
> index 5e85358d9090..f5ca53574406 100644
> --- a/arch/s390/include/asm/kvm_host.h
> +++ b/arch/s390/include/asm/kvm_host.h
> @@ -268,16 +268,31 @@ struct kvm_s390_sie_block {
>  #define HPID_VSIE	0x5
>  	__u8	hpid;			/* 0x00b8 */
>  	__u8	reservedb9[7];		/* 0x00b9 */
> -	__u32	eiparams;		/* 0x00c0 */
> -	__u16	extcpuaddr;		/* 0x00c4 */
> -	__u16	eic;			/* 0x00c6 */
> +	union {
> +		struct {
> +			__u32	eiparams;	/* 0x00c0 */
> +			__u16	extcpuaddr;	/* 0x00c4 */
> +			__u16	eic;		/* 0x00c6 */
> +		};
> +		__u64	mcic;			/* 0x00c0 */
> +	} __packed;
>  	__u32	reservedc8;		/* 0x00c8 */
> -	__u16	pgmilc;			/* 0x00cc */
> -	__u16	iprcc;			/* 0x00ce */
> -	__u32	dxc;			/* 0x00d0 */
> -	__u16	mcn;			/* 0x00d4 */
> -	__u8	perc;			/* 0x00d6 */
> -	__u8	peratmid;		/* 0x00d7 */
> +	union {
> +		struct {
> +			__u16	pgmilc;		/* 0x00cc */
> +			__u16	iprcc;		/* 0x00ce */
> +		};
> +		__u32	edc;			/* 0x00cc */
> +	} __packed;
> +	union {
> +		struct {
> +			__u32	dxc;		/* 0x00d0 */
> +			__u16	mcn;		/* 0x00d4 */
> +			__u8	perc;		/* 0x00d6 */
> +			__u8	peratmid;	/* 0x00d7 */
> +		};
> +		__u64	faddr;			/* 0x00d0 */
> +	} __packed;
>  	__u64	peraddr;		/* 0x00d8 */
>  	__u8	eai;			/* 0x00e0 */
>  	__u8	peraid;			/* 0x00e1 */
> 

Reviewed-by: Thomas Huth <thuth@redhat.com>

Re: [RFCv2 18/37] KVM: s390: protvirt: Implement machine-check interruption injection

[Thomas Huth]

On 03/02/2020 14.19, Christian Borntraeger wrote:
> From: Michael Mueller <mimu@linux.ibm.com>
> 
> Similar to external interrupts, the hypervisor can inject machine
> checks by providing the right data in the interrupt injection controls.
> 
> Signed-off-by: Michael Mueller <mimu@linux.ibm.com>
> ---
>  arch/s390/kvm/interrupt.c | 8 ++++++++
>  1 file changed, 8 insertions(+)
> 
> diff --git a/arch/s390/kvm/interrupt.c b/arch/s390/kvm/interrupt.c
> index c707725e618b..a98f1dfde8de 100644
> --- a/arch/s390/kvm/interrupt.c
> +++ b/arch/s390/kvm/interrupt.c
> @@ -571,6 +571,14 @@ static int __write_machine_check(struct kvm_vcpu *vcpu,
>  	union mci mci;
>  	int rc;
>  
> +	if (kvm_s390_pv_is_protected(vcpu->kvm)) {
> +		vcpu->arch.sie_block->iictl = IICTL_CODE_MCHK;
> +		vcpu->arch.sie_block->mcic = mchk->mcic;
> +		vcpu->arch.sie_block->faddr = mchk->failing_storage_address;
> +		vcpu->arch.sie_block->edc = mchk->ext_damage_code;
> +		return 0;
> +	}
> +
>  	mci.val = mchk->mcic;
>  	/* take care of lazy register loading */
>  	save_fpu_regs();
> 

Reviewed-by: Thomas Huth <thuth@redhat.com>

Re: [RFCv2 14/37] KVM: s390: protvirt: Add interruption injection controls

[Christian Borntraeger]

On 05.02.20 07:59, Thomas Huth wrote:
> On 03/02/2020 14.19, Christian Borntraeger wrote:
>> From: Michael Mueller <mimu@linux.ibm.com>
>>
>> Define the interruption injection codes and the related fields in the
>> sie control block for PVM interruption injection.
> 
> You seem to only add the details for external interrupts and I/O
> interrupts here? Maybe mention this in the description ... otherwise it
> is confusing when you read patch 17 later ... or maybe merge this patch
> here with patch 17 ?

What about the following:


This defines the necessary data structures in the SIE control block to
inject external and I/O interrupts. We first define the the interrupt
injection control, which defines the next interrupt to inject. Then we
define the fields that contain the payload for external and I/O
interrupts. The definitions for machine checks come in a later patch.

> 
>> Signed-off-by: Michael Mueller <mimu@linux.ibm.com>
>> ---
>>  arch/s390/include/asm/kvm_host.h | 25 +++++++++++++++++++++----
>>  1 file changed, 21 insertions(+), 4 deletions(-)
>>
>> diff --git a/arch/s390/include/asm/kvm_host.h b/arch/s390/include/asm/kvm_host.h
>> index 58845b315be0..a45d10d87a8a 100644
>> --- a/arch/s390/include/asm/kvm_host.h
>> +++ b/arch/s390/include/asm/kvm_host.h
>> @@ -222,7 +222,15 @@ struct kvm_s390_sie_block {
>>  	__u8	icptcode;		/* 0x0050 */
>>  	__u8	icptstatus;		/* 0x0051 */
>>  	__u16	ihcpu;			/* 0x0052 */
>> -	__u8	reserved54[2];		/* 0x0054 */
>> +	__u8	reserved54;		/* 0x0054 */
>> +#define IICTL_CODE_NONE		 0x00
>> +#define IICTL_CODE_MCHK		 0x01
>> +#define IICTL_CODE_EXT		 0x02
>> +#define IICTL_CODE_IO		 0x03
>> +#define IICTL_CODE_RESTART	 0x04
>> +#define IICTL_CODE_SPECIFICATION 0x10
>> +#define IICTL_CODE_OPERAND	 0x11
>> +	__u8	iictl;			/* 0x0055 */
>>  	__u16	ipa;			/* 0x0056 */
>>  	__u32	ipb;			/* 0x0058 */
>>  	__u32	scaoh;			/* 0x005c */
>> @@ -259,7 +267,8 @@ struct kvm_s390_sie_block {
>>  #define HPID_KVM	0x4
>>  #define HPID_VSIE	0x5
>>  	__u8	hpid;			/* 0x00b8 */
>> -	__u8	reservedb9[11];		/* 0x00b9 */
>> +	__u8	reservedb9[7];		/* 0x00b9 */
>> +	__u32	eiparams;		/* 0x00c0 */
>>  	__u16	extcpuaddr;		/* 0x00c4 */
>>  	__u16	eic;			/* 0x00c6 */
>>  	__u32	reservedc8;		/* 0x00c8 */
>> @@ -275,8 +284,16 @@ struct kvm_s390_sie_block {
>>  	__u8	oai;			/* 0x00e2 */
>>  	__u8	armid;			/* 0x00e3 */
>>  	__u8	reservede4[4];		/* 0x00e4 */
>> -	__u64	tecmc;			/* 0x00e8 */
>> -	__u8	reservedf0[12];		/* 0x00f0 */
>> +	union {
>> +		__u64	tecmc;		/* 0x00e8 */
>> +		struct {
>> +			__u16	subchannel_id;	/* 0x00e8 */
>> +			__u16	subchannel_nr;	/* 0x00ea */
>> +			__u32	io_int_parm;	/* 0x00ec */
>> +			__u32	io_int_word;	/* 0x00f0 */
>> +		};
>> +	} __packed;
>> +	__u8	reservedf4[8];		/* 0x00f4 */
> 
> Maybe add a comment to the new struct for which injection type it is
> good for ... otherwise this might get hard to understand in the future
> (especially if more stuff gets added like in patch 17).

Not sure. We usually do not have documentation inside this structure
for other things. 


> Anyway,
> Reviewed-by: Thomas Huth <thuth@redhat.com>

thanks

Re: [RFCv2 14/37] KVM: s390: protvirt: Add interruption injection controls

[Michael Mueller]

On 05.02.20 07:59, Thomas Huth wrote:
> On 03/02/2020 14.19, Christian Borntraeger wrote:
>> From: Michael Mueller <mimu@linux.ibm.com>
>>
>> Define the interruption injection codes and the related fields in the
>> sie control block for PVM interruption injection.
> 
> You seem to only add the details for external interrupts and I/O
> interrupts here? Maybe mention this in the description ... otherwise it
> is confusing when you read patch 17 later ... or maybe merge this patch
> here with patch 17 ?

In that case I will merge patch 17 into this one.

> 
>> Signed-off-by: Michael Mueller <mimu@linux.ibm.com>
>> ---
>>   arch/s390/include/asm/kvm_host.h | 25 +++++++++++++++++++++----
>>   1 file changed, 21 insertions(+), 4 deletions(-)
>>
>> diff --git a/arch/s390/include/asm/kvm_host.h b/arch/s390/include/asm/kvm_host.h
>> index 58845b315be0..a45d10d87a8a 100644
>> --- a/arch/s390/include/asm/kvm_host.h
>> +++ b/arch/s390/include/asm/kvm_host.h
>> @@ -222,7 +222,15 @@ struct kvm_s390_sie_block {
>>   	__u8	icptcode;		/* 0x0050 */
>>   	__u8	icptstatus;		/* 0x0051 */
>>   	__u16	ihcpu;			/* 0x0052 */
>> -	__u8	reserved54[2];		/* 0x0054 */
>> +	__u8	reserved54;		/* 0x0054 */
>> +#define IICTL_CODE_NONE		 0x00
>> +#define IICTL_CODE_MCHK		 0x01
>> +#define IICTL_CODE_EXT		 0x02
>> +#define IICTL_CODE_IO		 0x03
>> +#define IICTL_CODE_RESTART	 0x04
>> +#define IICTL_CODE_SPECIFICATION 0x10
>> +#define IICTL_CODE_OPERAND	 0x11
>> +	__u8	iictl;			/* 0x0055 */
>>   	__u16	ipa;			/* 0x0056 */
>>   	__u32	ipb;			/* 0x0058 */
>>   	__u32	scaoh;			/* 0x005c */
>> @@ -259,7 +267,8 @@ struct kvm_s390_sie_block {
>>   #define HPID_KVM	0x4
>>   #define HPID_VSIE	0x5
>>   	__u8	hpid;			/* 0x00b8 */
>> -	__u8	reservedb9[11];		/* 0x00b9 */
>> +	__u8	reservedb9[7];		/* 0x00b9 */
>> +	__u32	eiparams;		/* 0x00c0 */
>>   	__u16	extcpuaddr;		/* 0x00c4 */
>>   	__u16	eic;			/* 0x00c6 */
>>   	__u32	reservedc8;		/* 0x00c8 */
>> @@ -275,8 +284,16 @@ struct kvm_s390_sie_block {
>>   	__u8	oai;			/* 0x00e2 */
>>   	__u8	armid;			/* 0x00e3 */
>>   	__u8	reservede4[4];		/* 0x00e4 */
>> -	__u64	tecmc;			/* 0x00e8 */
>> -	__u8	reservedf0[12];		/* 0x00f0 */
>> +	union {
>> +		__u64	tecmc;		/* 0x00e8 */
>> +		struct {
>> +			__u16	subchannel_id;	/* 0x00e8 */
>> +			__u16	subchannel_nr;	/* 0x00ea */
>> +			__u32	io_int_parm;	/* 0x00ec */
>> +			__u32	io_int_word;	/* 0x00f0 */
>> +		};
>> +	} __packed;
>> +	__u8	reservedf4[8];		/* 0x00f4 */
> 
> Maybe add a comment to the new struct for which injection type it is
> good for ... otherwise this might get hard to understand in the future
> (especially if more stuff gets added like in patch 17).

Up to know we don't have comments in the the SIE control block struct at 
all as this is part of the documentation, also for protected virtualization.

Any other opinions on that are welcome.

> 
> Anyway,
> Reviewed-by: Thomas Huth <thuth@redhat.com>

thanks

> 

Michael

Re: [RFCv2 14/37] KVM: s390: protvirt: Add interruption injection controls

[Michael Mueller]

On 05.02.20 09:46, Christian Borntraeger wrote:
> 
> On 05.02.20 07:59, Thomas Huth wrote:
>> On 03/02/2020 14.19, Christian Borntraeger wrote:
>>> From: Michael Mueller <mimu@linux.ibm.com>
>>>
>>> Define the interruption injection codes and the related fields in the
>>> sie control block for PVM interruption injection.
>>
>> You seem to only add the details for external interrupts and I/O
>> interrupts here? Maybe mention this in the description ... otherwise it
>> is confusing when you read patch 17 later ... or maybe merge this patch
>> here with patch 17 ?
> 
> What about the following:
> 
> 
> This defines the necessary data structures in the SIE control block to
> inject external and I/O interrupts. We first define the the interrupt
> injection control, which defines the next interrupt to inject. Then we
> define the fields that contain the payload for external and I/O
> interrupts. The definitions for machine checks come in a later patch.

We just made a comment on that in parallel. Ok with me as well. :)

> 
>>
>>> Signed-off-by: Michael Mueller <mimu@linux.ibm.com>
>>> ---
>>>   arch/s390/include/asm/kvm_host.h | 25 +++++++++++++++++++++----
>>>   1 file changed, 21 insertions(+), 4 deletions(-)
>>>
>>> diff --git a/arch/s390/include/asm/kvm_host.h b/arch/s390/include/asm/kvm_host.h
>>> index 58845b315be0..a45d10d87a8a 100644
>>> --- a/arch/s390/include/asm/kvm_host.h
>>> +++ b/arch/s390/include/asm/kvm_host.h
>>> @@ -222,7 +222,15 @@ struct kvm_s390_sie_block {
>>>   	__u8	icptcode;		/* 0x0050 */
>>>   	__u8	icptstatus;		/* 0x0051 */
>>>   	__u16	ihcpu;			/* 0x0052 */
>>> -	__u8	reserved54[2];		/* 0x0054 */
>>> +	__u8	reserved54;		/* 0x0054 */
>>> +#define IICTL_CODE_NONE		 0x00
>>> +#define IICTL_CODE_MCHK		 0x01
>>> +#define IICTL_CODE_EXT		 0x02
>>> +#define IICTL_CODE_IO		 0x03
>>> +#define IICTL_CODE_RESTART	 0x04
>>> +#define IICTL_CODE_SPECIFICATION 0x10
>>> +#define IICTL_CODE_OPERAND	 0x11
>>> +	__u8	iictl;			/* 0x0055 */
>>>   	__u16	ipa;			/* 0x0056 */
>>>   	__u32	ipb;			/* 0x0058 */
>>>   	__u32	scaoh;			/* 0x005c */
>>> @@ -259,7 +267,8 @@ struct kvm_s390_sie_block {
>>>   #define HPID_KVM	0x4
>>>   #define HPID_VSIE	0x5
>>>   	__u8	hpid;			/* 0x00b8 */
>>> -	__u8	reservedb9[11];		/* 0x00b9 */
>>> +	__u8	reservedb9[7];		/* 0x00b9 */
>>> +	__u32	eiparams;		/* 0x00c0 */
>>>   	__u16	extcpuaddr;		/* 0x00c4 */
>>>   	__u16	eic;			/* 0x00c6 */
>>>   	__u32	reservedc8;		/* 0x00c8 */
>>> @@ -275,8 +284,16 @@ struct kvm_s390_sie_block {
>>>   	__u8	oai;			/* 0x00e2 */
>>>   	__u8	armid;			/* 0x00e3 */
>>>   	__u8	reservede4[4];		/* 0x00e4 */
>>> -	__u64	tecmc;			/* 0x00e8 */
>>> -	__u8	reservedf0[12];		/* 0x00f0 */
>>> +	union {
>>> +		__u64	tecmc;		/* 0x00e8 */
>>> +		struct {
>>> +			__u16	subchannel_id;	/* 0x00e8 */
>>> +			__u16	subchannel_nr;	/* 0x00ea */
>>> +			__u32	io_int_parm;	/* 0x00ec */
>>> +			__u32	io_int_word;	/* 0x00f0 */
>>> +		};
>>> +	} __packed;
>>> +	__u8	reservedf4[8];		/* 0x00f4 */
>>
>> Maybe add a comment to the new struct for which injection type it is
>> good for ... otherwise this might get hard to understand in the future
>> (especially if more stuff gets added like in patch 17).
> 
> Not sure. We usually do not have documentation inside this structure
> for other things.
> 
> 
>> Anyway,
>> Reviewed-by: Thomas Huth <thuth@redhat.com>
> 
> thanks
> 
> 

Michael

Re: [RFCv2 11/37] KVM: s390/mm: Make pages accessible before destroying the guest

[Christian Borntraeger]

On 04.02.20 21:52, Thomas Huth wrote:
> On 03/02/2020 14.19, Christian Borntraeger wrote:
>> Before we destroy the secure configuration, we better make all
>> pages accessible again. This also happens during reboot, where we reboot
>> into a non-secure guest that then can go again into a secure mode. As
>> this "new" secure guest will have a new ID we cannot reuse the old page
>> state.
>>
>> Signed-off-by: Christian Borntraeger <borntraeger@de.ibm.com>
>> ---
>>  arch/s390/include/asm/pgtable.h |  1 +
>>  arch/s390/kvm/pv.c              |  2 ++
>>  arch/s390/mm/gmap.c             | 35 +++++++++++++++++++++++++++++++++
>>  3 files changed, 38 insertions(+)
>>
>> diff --git a/arch/s390/include/asm/pgtable.h b/arch/s390/include/asm/pgtable.h
>> index 65b6bb47af0a..6e167dcc35f1 100644
>> --- a/arch/s390/include/asm/pgtable.h
>> +++ b/arch/s390/include/asm/pgtable.h
>> @@ -1669,6 +1669,7 @@ extern int vmem_remove_mapping(unsigned long start, unsigned long size);
>>  extern int s390_enable_sie(void);
>>  extern int s390_enable_skey(void);
>>  extern void s390_reset_cmma(struct mm_struct *mm);
>> +extern void s390_reset_acc(struct mm_struct *mm);
>>  
>>  /* s390 has a private copy of get unmapped area to deal with cache synonyms */
>>  #define HAVE_ARCH_UNMAPPED_AREA
>> diff --git a/arch/s390/kvm/pv.c b/arch/s390/kvm/pv.c
>> index a867b9e9c069..24d802072ac7 100644
>> --- a/arch/s390/kvm/pv.c
>> +++ b/arch/s390/kvm/pv.c
>> @@ -66,6 +66,8 @@ int kvm_s390_pv_destroy_vm(struct kvm *kvm)
>>  	int rc;
>>  	u32 ret;
>>  
>> +	/* make all pages accessible before destroying the guest */
>> +	s390_reset_acc(kvm->mm);
>>  	rc = uv_cmd_nodata(kvm_s390_pv_handle(kvm),
>>  			   UVC_CMD_DESTROY_SEC_CONF, &ret);
>>  	WRITE_ONCE(kvm->arch.gmap->se_handle, 0);
>> diff --git a/arch/s390/mm/gmap.c b/arch/s390/mm/gmap.c
>> index bf365a09f900..0b00e8d5fa39 100644
>> --- a/arch/s390/mm/gmap.c
>> +++ b/arch/s390/mm/gmap.c
>> @@ -2648,3 +2648,38 @@ void s390_reset_cmma(struct mm_struct *mm)
>>  	up_write(&mm->mmap_sem);
>>  }
>>  EXPORT_SYMBOL_GPL(s390_reset_cmma);
>> +
>> +/*
>> + * make inaccessible pages accessible again
>> + */
>> +static int __s390_reset_acc(pte_t *ptep, unsigned long addr,
>> +			    unsigned long next, struct mm_walk *walk)
>> +{
>> +	pte_t pte = READ_ONCE(*ptep);
>> +
>> +	if (pte_present(pte))
>> +		uv_convert_from_secure(pte_val(pte) & PAGE_MASK);
> 
> Is it ok to ignore the return value from uv_convert_from_secure() ?
> Problems might go unnoticed ... maybe use at least a WARN_ONCE ?

Hmm, it is certainly one of these "should not happen" cases.
So yes, I will add a WARN_ON_ONCE.



> 
>> +	return 0;
>> +}
>> +
>> +static const struct mm_walk_ops reset_acc_walk_ops = {
>> +	.pte_entry		= __s390_reset_acc,
>> +};
>> +
>> +#include <linux/sched/mm.h>
>> +void s390_reset_acc(struct mm_struct *mm)
>> +{
>> +	/*
>> +	 * we might be called during
>> +	 * reset:                            we walk the pages and clear
>> +	 * close of all kvm file descriptor: we walk the pages and clear
>> +	 * exit of process on fd closure:    vma already gone, do nothing
>> +	 */
>> +	if (!mmget_not_zero(mm))
>> +		return;
>> +	down_read(&mm->mmap_sem);
>> +	walk_page_range(mm, 0, TASK_SIZE, &reset_acc_walk_ops, NULL);
>> +	up_read(&mm->mmap_sem);
>> +	mmput(mm);
>> +}
>> +EXPORT_SYMBOL_GPL(s390_reset_acc);
>>
> 
> Anyway,
> Reviewed-by: Thomas Huth <thuth@redhat.com>
> 

thanks.

Re: [RFCv2 15/37] KVM: s390: protvirt: Implement interruption injection

[Thomas Huth]

On 03/02/2020 14.19, Christian Borntraeger wrote:
> From: Michael Mueller <mimu@linux.ibm.com>
> 
> The patch implements interruption injection for the following
> list of interruption types:
> 
>   - I/O
>     __deliver_io (III)
> 
>   - External
>     __deliver_cpu_timer (IEI)
>     __deliver_ckc (IEI)
>     __deliver_emergency_signal (IEI)
>     __deliver_external_call (IEI)
> 
>   - cpu restart
>     __deliver_restart (IRI)
> 
> The service interrupt is handled in a followup patch.
> 
> Signed-off-by: Michael Mueller <mimu@linux.ibm.com>
> Signed-off-by: Christian Borntraeger <borntraeger@de.ibm.com>
> [fixes]
> ---
>  arch/s390/include/asm/kvm_host.h |  8 +++
>  arch/s390/kvm/interrupt.c        | 93 ++++++++++++++++++++++----------
>  2 files changed, 74 insertions(+), 27 deletions(-)
> 
> diff --git a/arch/s390/include/asm/kvm_host.h b/arch/s390/include/asm/kvm_host.h
> index a45d10d87a8a..989cea7a5591 100644
> --- a/arch/s390/include/asm/kvm_host.h
> +++ b/arch/s390/include/asm/kvm_host.h
> @@ -563,6 +563,14 @@ enum irq_types {
>  #define IRQ_PEND_MCHK_MASK ((1UL << IRQ_PEND_MCHK_REP) | \
>  			    (1UL << IRQ_PEND_MCHK_EX))
>  
> +#define IRQ_PEND_MCHK_REP_MASK (1UL << IRQ_PEND_MCHK_REP)
> +
> +#define IRQ_PEND_EXT_II_MASK ((1UL << IRQ_PEND_EXT_CPU_TIMER)  | \
> +			      (1UL << IRQ_PEND_EXT_CLOCK_COMP) | \
> +			      (1UL << IRQ_PEND_EXT_EMERGENCY)  | \
> +			      (1UL << IRQ_PEND_EXT_EXTERNAL)   | \
> +			      (1UL << IRQ_PEND_EXT_SERVICE))
> +
>  struct kvm_s390_interrupt_info {
>  	struct list_head list;
>  	u64	type;
> diff --git a/arch/s390/kvm/interrupt.c b/arch/s390/kvm/interrupt.c
> index c06c89d370a7..ecdec6960a60 100644
> --- a/arch/s390/kvm/interrupt.c
> +++ b/arch/s390/kvm/interrupt.c
> @@ -387,6 +387,12 @@ static unsigned long deliverable_irqs(struct kvm_vcpu *vcpu)
>  		__clear_bit(IRQ_PEND_EXT_SERVICE, &active_mask);
>  	if (psw_mchk_disabled(vcpu))
>  		active_mask &= ~IRQ_PEND_MCHK_MASK;
> +	/* PV guest cpus can have a single interruption injected at a time. */
> +	if (kvm_s390_pv_is_protected(vcpu->kvm) &&
> +	    vcpu->arch.sie_block->iictl != IICTL_CODE_NONE)
> +		active_mask &= ~(IRQ_PEND_EXT_II_MASK |
> +				 IRQ_PEND_IO_MASK |
> +				 IRQ_PEND_MCHK_REP_MASK);

I don't quite understand why there is a difference between
IRQ_PEND_MCHK_REP and IRQ_PEND_MCHK_EX here? Why not simply use
IRQ_PEND_MCHK_MASK here? Could you elaborate? (and maybe add a sentence
to the patch description)

> @@ -533,7 +547,6 @@ static int __must_check __deliver_pfault_init(struct kvm_vcpu *vcpu)
>  	trace_kvm_s390_deliver_interrupt(vcpu->vcpu_id,
>  					 KVM_S390_INT_PFAULT_INIT,
>  					 0, ext.ext_params2);
> -

Nit: Unnecessary white space change.

Apart from that, the patch look ok to me.

 Thomas

Re: [RFCv2 05/37] s390/mm: provide memory management functions for protected KVM guests

[Janosch Frank]

[-- Attachment #1.1: Type: text/plain, Size: 13892 bytes --]

On 2/3/20 2:19 PM, Christian Borntraeger wrote:
> From: Claudio Imbrenda <imbrenda@linux.ibm.com>
> 
> This provides the basic ultravisor calls and page table handling to cope
> with secure guests.
> 
> Co-authored-by: Ulrich Weigand <Ulrich.Weigand@de.ibm.com>
> Signed-off-by: Claudio Imbrenda <imbrenda@linux.ibm.com>
> ---
>  arch/s390/include/asm/gmap.h        |   2 +
>  arch/s390/include/asm/mmu.h         |   2 +
>  arch/s390/include/asm/mmu_context.h |   1 +
>  arch/s390/include/asm/page.h        |   5 +
>  arch/s390/include/asm/pgtable.h     |  34 +++++-
>  arch/s390/include/asm/uv.h          |  59 ++++++++++
>  arch/s390/kernel/uv.c               | 170 ++++++++++++++++++++++++++++
>  7 files changed, 268 insertions(+), 5 deletions(-)
> 
> diff --git a/arch/s390/include/asm/gmap.h b/arch/s390/include/asm/gmap.h
> index 37f96b6f0e61..f2ab8b6d4b57 100644
> --- a/arch/s390/include/asm/gmap.h
> +++ b/arch/s390/include/asm/gmap.h
> @@ -9,6 +9,7 @@
>  #ifndef _ASM_S390_GMAP_H
>  #define _ASM_S390_GMAP_H
> 
> +#include <linux/radix-tree.h>
>  #include <linux/refcount.h>
> 
>  /* Generic bits for GMAP notification on DAT table entry changes. */
> @@ -61,6 +62,7 @@ struct gmap {
>  	spinlock_t shadow_lock;
>  	struct gmap *parent;
>  	unsigned long orig_asce;
> +	unsigned long se_handle;
>  	int edat_level;
>  	bool removed;
>  	bool initialized;
> diff --git a/arch/s390/include/asm/mmu.h b/arch/s390/include/asm/mmu.h
> index bcfb6371086f..984026cb3608 100644
> --- a/arch/s390/include/asm/mmu.h
> +++ b/arch/s390/include/asm/mmu.h
> @@ -16,6 +16,8 @@ typedef struct {
>  	unsigned long asce;
>  	unsigned long asce_limit;
>  	unsigned long vdso_base;
> +	/* The mmu context belongs to a secure guest. */
> +	atomic_t is_se;
>  	/*
>  	 * The following bitfields need a down_write on the mm
>  	 * semaphore when they are written to. As they are only
> diff --git a/arch/s390/include/asm/mmu_context.h b/arch/s390/include/asm/mmu_context.h
> index 8d04e6f3f796..0e5e67ecdaf8 100644
> --- a/arch/s390/include/asm/mmu_context.h
> +++ b/arch/s390/include/asm/mmu_context.h
> @@ -23,6 +23,7 @@ static inline int init_new_context(struct task_struct *tsk,
>  	INIT_LIST_HEAD(&mm->context.gmap_list);
>  	cpumask_clear(&mm->context.cpu_attach_mask);
>  	atomic_set(&mm->context.flush_count, 0);
> +	atomic_set(&mm->context.is_se, 0);
>  	mm->context.gmap_asce = 0;
>  	mm->context.flush_mm = 0;
>  	mm->context.compat_mm = test_thread_flag(TIF_31BIT);
> diff --git a/arch/s390/include/asm/page.h b/arch/s390/include/asm/page.h
> index a4d38092530a..eb209416c45b 100644
> --- a/arch/s390/include/asm/page.h
> +++ b/arch/s390/include/asm/page.h
> @@ -151,6 +151,11 @@ static inline int devmem_is_allowed(unsigned long pfn)
>  #define HAVE_ARCH_FREE_PAGE
>  #define HAVE_ARCH_ALLOC_PAGE
> 
> +#ifdef CONFIG_KVM_S390_PROTECTED_VIRTUALIZATION_HOST
> +int arch_make_page_accessible(struct page *page);
> +#define HAVE_ARCH_MAKE_PAGE_ACCESSIBLE
> +#endif
> +
>  #endif /* !__ASSEMBLY__ */
> 
>  #define __PAGE_OFFSET		0x0UL
> diff --git a/arch/s390/include/asm/pgtable.h b/arch/s390/include/asm/pgtable.h
> index 7b03037a8475..65b6bb47af0a 100644
> --- a/arch/s390/include/asm/pgtable.h
> +++ b/arch/s390/include/asm/pgtable.h
> @@ -19,6 +19,7 @@
>  #include <linux/atomic.h>
>  #include <asm/bug.h>
>  #include <asm/page.h>
> +#include <asm/uv.h>
> 
>  extern pgd_t swapper_pg_dir[];
>  extern void paging_init(void);
> @@ -520,6 +521,15 @@ static inline int mm_has_pgste(struct mm_struct *mm)
>  	return 0;
>  }
> 
> +static inline int mm_is_se(struct mm_struct *mm)
> +{
> +#ifdef CONFIG_KVM_S390_PROTECTED_VIRTUALIZATION_HOST
> +	if (unlikely(atomic_read(&mm->context.is_se)))
> +		return 1;
> +#endif
> +	return 0;
> +}
> +
>  static inline int mm_alloc_pgste(struct mm_struct *mm)
>  {
>  #ifdef CONFIG_PGSTE
> @@ -1059,7 +1069,12 @@ static inline int ptep_clear_flush_young(struct vm_area_struct *vma,
>  static inline pte_t ptep_get_and_clear(struct mm_struct *mm,
>  				       unsigned long addr, pte_t *ptep)
>  {
> -	return ptep_xchg_lazy(mm, addr, ptep, __pte(_PAGE_INVALID));
> +	pte_t res;
> +
> +	res = ptep_xchg_lazy(mm, addr, ptep, __pte(_PAGE_INVALID));
> +	if (mm_is_se(mm) && pte_present(res))
> +		uv_convert_from_secure(pte_val(res) & PAGE_MASK);
> +	return res;
>  }
> 
>  #define __HAVE_ARCH_PTEP_MODIFY_PROT_TRANSACTION
> @@ -1071,7 +1086,12 @@ void ptep_modify_prot_commit(struct vm_area_struct *, unsigned long,
>  static inline pte_t ptep_clear_flush(struct vm_area_struct *vma,
>  				     unsigned long addr, pte_t *ptep)
>  {
> -	return ptep_xchg_direct(vma->vm_mm, addr, ptep, __pte(_PAGE_INVALID));
> +	pte_t res;
> +
> +	res = ptep_xchg_direct(vma->vm_mm, addr, ptep, __pte(_PAGE_INVALID));
> +	if (mm_is_se(vma->vm_mm) && pte_present(res))
> +		uv_convert_from_secure(pte_val(res) & PAGE_MASK);
> +	return res;
>  }
> 
>  /*
> @@ -1086,12 +1106,16 @@ static inline pte_t ptep_get_and_clear_full(struct mm_struct *mm,
>  					    unsigned long addr,
>  					    pte_t *ptep, int full)
>  {
> +	pte_t res;
>  	if (full) {
> -		pte_t pte = *ptep;
> +		res = *ptep;
>  		*ptep = __pte(_PAGE_INVALID);
> -		return pte;
> +	} else {
> +		res = ptep_xchg_lazy(mm, addr, ptep, __pte(_PAGE_INVALID));
>  	}
> -	return ptep_xchg_lazy(mm, addr, ptep, __pte(_PAGE_INVALID));
> +	if (mm_is_se(mm) && pte_present(res))
> +		uv_convert_from_secure(pte_val(res) & PAGE_MASK);
> +	return res;
>  }
> 
>  #define __HAVE_ARCH_PTEP_SET_WRPROTECT
> diff --git a/arch/s390/include/asm/uv.h b/arch/s390/include/asm/uv.h
> index cdf2fd71d7ab..4eaea95f5c64 100644
> --- a/arch/s390/include/asm/uv.h
> +++ b/arch/s390/include/asm/uv.h
> @@ -15,6 +15,7 @@
>  #include <linux/errno.h>
>  #include <linux/bug.h>
>  #include <asm/page.h>
> +#include <asm/gmap.h>
> 
>  #define UVC_RC_EXECUTED		0x0001
>  #define UVC_RC_INV_CMD		0x0002
> @@ -24,6 +25,10 @@
> 
>  #define UVC_CMD_QUI			0x0001
>  #define UVC_CMD_INIT_UV			0x000f
> +#define UVC_CMD_CONV_TO_SEC_STOR	0x0200
> +#define UVC_CMD_CONV_FROM_SEC_STOR	0x0201
> +#define UVC_CMD_PIN_PAGE_SHARED		0x0341
> +#define UVC_CMD_UNPIN_PAGE_SHARED	0x0342
>  #define UVC_CMD_SET_SHARED_ACCESS	0x1000
>  #define UVC_CMD_REMOVE_SHARED_ACCESS	0x1001
> 
> @@ -31,8 +36,12 @@
>  enum uv_cmds_inst {
>  	BIT_UVC_CMD_QUI = 0,
>  	BIT_UVC_CMD_INIT_UV = 1,
> +	BIT_UVC_CMD_CONV_TO_SEC_STOR = 6,
> +	BIT_UVC_CMD_CONV_FROM_SEC_STOR = 7,
>  	BIT_UVC_CMD_SET_SHARED_ACCESS = 8,
>  	BIT_UVC_CMD_REMOVE_SHARED_ACCESS = 9,
> +	BIT_UVC_CMD_PIN_PAGE_SHARED = 21,
> +	BIT_UVC_CMD_UNPIN_PAGE_SHARED = 22,
>  };
> 
>  struct uv_cb_header {
> @@ -70,6 +79,19 @@ struct uv_cb_init {
> 
>  } __packed __aligned(8);
> 
> +struct uv_cb_cts {
> +	struct uv_cb_header header;
> +	u64 reserved08[2];
> +	u64 guest_handle;
> +	u64 gaddr;
> +} __packed __aligned(8);
> +
> +struct uv_cb_cfs {
> +	struct uv_cb_header header;
> +	u64 reserved08[2];
> +	u64 paddr;
> +} __packed __aligned(8);
> +
>  struct uv_cb_share {
>  	struct uv_cb_header header;
>  	u64 reserved08[3];
> @@ -170,12 +192,49 @@ static inline int is_prot_virt_host(void)
>  	return prot_virt_host;
>  }
> 
> +int uv_make_secure(struct gmap *gmap, unsigned long gaddr, void *uvcb, int pins);
> +int uv_convert_from_secure(unsigned long paddr);
> +
> +static inline int uv_convert_to_secure_pinned(struct gmap *gmap,
> +					      unsigned long gaddr,
> +					      int pins)
> +{
> +	struct uv_cb_cts uvcb = {
> +		.header.cmd = UVC_CMD_CONV_TO_SEC_STOR,
> +		.header.len = sizeof(uvcb),
> +		.guest_handle = gmap->se_handle,
> +		.gaddr = gaddr,
> +	};
> +
> +	return uv_make_secure(gmap, gaddr, &uvcb, pins);
> +}
> +
> +static inline int uv_convert_to_secure(struct gmap *gmap, unsigned long gaddr)
> +{
> +	return uv_convert_to_secure_pinned(gmap, gaddr, 0);
> +}
> +
>  void setup_uv(void);
>  void adjust_to_uv_max(unsigned long *vmax);
>  #else
>  #define is_prot_virt_host() 0
>  static inline void setup_uv(void) {}
>  static inline void adjust_to_uv_max(unsigned long *vmax) {}
> +
> +static inline int uv_make_secure(struct gmap *gmap, unsigned long gaddr, void *uvcb, int pins)
> +{
> +	return 0;
> +}
> +
> +static inline int uv_convert_from_secure(unsigned long paddr)
> +{
> +	return 0;
> +}
> +
> +static inline int uv_convert_to_secure(struct gmap *gmap, unsigned long gaddr)
> +{
> +	return 0;
> +}
>  #endif
> 
>  #if defined(CONFIG_PROTECTED_VIRTUALIZATION_GUEST) ||                          \
> diff --git a/arch/s390/kernel/uv.c b/arch/s390/kernel/uv.c
> index f7778493e829..136c60a8e3ca 100644
> --- a/arch/s390/kernel/uv.c
> +++ b/arch/s390/kernel/uv.c
> @@ -9,6 +9,8 @@
>  #include <linux/sizes.h>
>  #include <linux/bitmap.h>
>  #include <linux/memblock.h>
> +#include <linux/pagemap.h>
> +#include <linux/swap.h>
>  #include <asm/facility.h>
>  #include <asm/sections.h>
>  #include <asm/uv.h>
> @@ -98,4 +100,172 @@ void adjust_to_uv_max(unsigned long *vmax)
>  	if (prot_virt_host && *vmax > uv_info.max_sec_stor_addr)
>  		*vmax = uv_info.max_sec_stor_addr;
>  }
> +
> +static int __uv_pin_shared(unsigned long paddr)
> +{
> +	struct uv_cb_cfs uvcb = {
> +		.header.cmd	= UVC_CMD_PIN_PAGE_SHARED,
> +		.header.len	= sizeof(uvcb),
> +		.paddr		= paddr,
> +	};

We completely loose .header.rc and rrc if something goes wrong and hence
we'll have no way finding out what went wrong after the fact.

We should either make sure to warn_on_once() or come up with a way of
logging that to somewhere useful.

> +
> +	if (uv_call(0, (u64)&uvcb))
> +		return -EINVAL;
> +	return 0;
> +}
> +
> +/*
> + * Requests the Ultravisor to encrypt a guest page and make it
> + * accessible to the host for paging (export).
> + *
> + * @paddr: Absolute host address of page to be exported
> + */
> +int uv_convert_from_secure(unsigned long paddr)
> +{
> +	struct uv_cb_cfs uvcb = {
> +		.header.cmd = UVC_CMD_CONV_FROM_SEC_STOR,
> +		.header.len = sizeof(uvcb),
> +		.paddr = paddr
> +	};
> +
> +	uv_call(0, (u64)&uvcb);
> +
> +	if (uvcb.header.rc == 1 || uvcb.header.rc == 0x107)

Magic constant is magic
We either need a comment, or a constant with a fitting name. That also
goes for the 0x10a.

> +		return 0;
> +	return -EINVAL;
> +}
> +
> +static int expected_page_refs(struct page *page)
> +{
> +	int res;
> +
> +	res = page_mapcount(page);
> +	if (PageSwapCache(page))
> +		res++;
> +	else if (page_mapping(page)) {
> +		res++;
> +		if (page_has_private(page))
> +			res++;
> +	}
> +	return res;
> +}
> +
> +struct conv_params {
> +	struct uv_cb_header *uvcb;
> +	struct page *page;
> +	int extra_pins;
> +};
> +
> +static int make_secure_pte(pte_t *ptep, unsigned long addr, void *data)
> +{
> +	struct conv_params *params = data;
> +	pte_t entry = READ_ONCE(*ptep);
> +	struct page *page;
> +	int expected, rc = 0;
> +
> +	if (!pte_present(entry))
> +		return -ENXIO;
> +	if (pte_val(entry) & (_PAGE_INVALID | _PAGE_PROTECT))
> +		return -ENXIO;
> +
> +	page = pte_page(entry);
> +	if (page != params->page)
> +		return -ENXIO;
> +
> +	if (PageWriteback(page))
> +		return -EAGAIN;
> +	expected = expected_page_refs(page) + params->extra_pins;
> +	if (!page_ref_freeze(page, expected))
> +		return -EBUSY;
> +	set_bit(PG_arch_1, &page->flags);
> +	rc = uv_call(0, (u64)params->uvcb);
> +	page_ref_unfreeze(page, expected);
> +	if (rc)
> +		rc = (params->uvcb->rc == 0x10a) ? -ENXIO : -EINVAL;
> +	return rc;
> +}
> +
> +/*
> + * Requests the Ultravisor to make a page accessible to a guest.
> + * If it's brought in the first time, it will be cleared. If
> + * it has been exported before, it will be decrypted and integrity
> + * checked.
> + *
> + * @gmap: Guest mapping
> + * @gaddr: Guest 2 absolute address to be imported
> + */
> +int uv_make_secure(struct gmap *gmap, unsigned long gaddr, void *uvcb, int pins)
> +{
> +	struct conv_params params = { .uvcb = uvcb, .extra_pins = pins };
> +	struct vm_area_struct *vma;
> +	unsigned long uaddr;
> +	int rc, local_drain = 0;
> +
> +again:
> +	rc = -EFAULT;
> +	down_read(&gmap->mm->mmap_sem);
> +
> +	uaddr = __gmap_translate(gmap, gaddr);
> +	if (IS_ERR_VALUE(uaddr))
> +		goto out;
> +	vma = find_vma(gmap->mm, uaddr);
> +	if (!vma)
> +		goto out;
> +	if (is_vm_hugetlb_page(vma))
> +		goto out;
> +
> +	rc = -ENXIO;
> +	params.page = follow_page(vma, uaddr, FOLL_WRITE | FOLL_NOWAIT);
> +	if (IS_ERR_OR_NULL(params.page))
> +		goto out;
> +
> +	lock_page(params.page);
> +	rc = apply_to_page_range(gmap->mm, uaddr, PAGE_SIZE, make_secure_pte, &params);
> +	unlock_page(params.page);
> +out:
> +	up_read(&gmap->mm->mmap_sem);
> +
> +	if (rc == -EBUSY) {
> +		if (local_drain) {
> +			lru_add_drain_all();
> +			return -EAGAIN;
> +		}
> +		lru_add_drain();
> +		local_drain = 1;
> +		goto again;
> +	} else if (rc == -ENXIO) {
> +		if (gmap_fault(gmap, gaddr, FAULT_FLAG_WRITE))
> +			return -EFAULT;
> +		return -EAGAIN;
> +	}
> +	return rc;
> +}
> +EXPORT_SYMBOL_GPL(uv_make_secure);
> +
> +/**
> + * To be called with the page locked or with an extra reference!
> + */
> +int arch_make_page_accessible(struct page *page)
> +{
> +	int rc = 0;
> +
> +	if (!test_bit(PG_arch_1, &page->flags))
> +		return 0;
> +
> +	rc = __uv_pin_shared(page_to_phys(page));
> +	if (!rc) {
> +		clear_bit(PG_arch_1, &page->flags);
> +		return 0;
> +	}
> +
> +	rc = uv_convert_from_secure(page_to_phys(page));
> +	if (!rc) {
> +		clear_bit(PG_arch_1, &page->flags);
> +		return 0;
> +	}
> +
> +	return rc;
> +}
> +EXPORT_SYMBOL_GPL(arch_make_page_accessible);
> +
>  #endif
> 



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [RFCv2 08/37] KVM: s390: protvirt: Add initial lifecycle handling

[Janosch Frank]

[-- Attachment #1.1: Type: text/plain, Size: 3433 bytes --]

On 2/4/20 1:34 PM, Christian Borntraeger wrote:
> 
> 
> On 04.02.20 13:13, David Hildenbrand wrote:
>> [...]
>>
>>> +#ifdef CONFIG_KVM_S390_PROTECTED_VIRTUALIZATION_HOST
>>> +static int kvm_s390_handle_pv(struct kvm *kvm, struct kvm_pv_cmd *cmd)
>>> +{
>>> +	int r = 0;
>>> +	void __user *argp = (void __user *)cmd->data;
>>> +
>>> +	switch (cmd->cmd) {
>>> +	case KVM_PV_VM_CREATE: {
>>> +		r = -EINVAL;
>>> +		if (kvm_s390_pv_is_protected(kvm))
>>> +			break;
>>> +
>>> +		r = kvm_s390_pv_alloc_vm(kvm);
>>> +		if (r)
>>> +			break;
>>> +
>>> +		mutex_lock(&kvm->lock);
>>> +		kvm_s390_vcpu_block_all(kvm);
>>> +		/* FMT 4 SIE needs esca */
>>> +		r = sca_switch_to_extended(kvm);
>>> +		if (!r)
>>> +			r = kvm_s390_pv_create_vm(kvm);
>>> +		kvm_s390_vcpu_unblock_all(kvm);
>>> +		mutex_unlock(&kvm->lock);
>>> +		break;
>>> +	}
>>
>> I think KVM_PV_VM_ENABLE/KVM_PV_VM_DISABLE would be a better fit. You're
>> not creating/deleting VMs, aren't you? All you're doing is allocating
>> some data and performing some kind of a mode switch.
> 
> I kind of like the idea. Need to talk to Janosch about this. 
>> [...]
>>
>>>  	VM_EVENT(kvm, 3, "create cpu %d at 0x%pK, sie block at 0x%pK", id, vcpu,
>>>  		 vcpu->arch.sie_block);
>>>  	trace_kvm_s390_create_vcpu(id, vcpu, vcpu->arch.sie_block);
>>> @@ -4353,6 +4502,37 @@ long kvm_arch_vcpu_async_ioctl(struct file *filp,
>>>  	return -ENOIOCTLCMD;
>>>  }
>>>  
>>> +#ifdef CONFIG_KVM_S390_PROTECTED_VIRTUALIZATION_HOST
>>> +static int kvm_s390_handle_pv_vcpu(struct kvm_vcpu *vcpu,
>>> +				   struct kvm_pv_cmd *cmd)
>>> +{
>>> +	int r = 0;
>>> +
>>> +	if (!kvm_s390_pv_is_protected(vcpu->kvm))
>>> +		return -EINVAL;
>>> +
>>> +	switch (cmd->cmd) {
>>> +	case KVM_PV_VCPU_CREATE: {
>>> +		if (kvm_s390_pv_handle_cpu(vcpu))
>>> +			return -EINVAL;
>>> +
>>> +		r = kvm_s390_pv_create_cpu(vcpu);
>>> +		break;
>>> +	}
>>> +	case KVM_PV_VCPU_DESTROY: {
>>> +		if (!kvm_s390_pv_handle_cpu(vcpu))
>>> +			return -EINVAL;
>>> +
>>> +		r = kvm_s390_pv_destroy_cpu(vcpu);
>>> +		break;
>>> +	}
>>> +	default:
>>> +		r = -ENOTTY;
>>> +	}
>>> +	return r;
>>> +}
>>
>> I asked this already and didn't get an answer (lost in the flood of
>> comments :) )
>>
>> Can't we simply convert all VCPUs via KVM_PV_VM_CREATE and destoy them
>> via KVM_PV_VM_DESTROY? Then you can easily handle hotplug as well in the
>> kernel when a new VCPU is created and PV is active - oh and I see you
>> are already doing that in kvm_arch_vcpu_create(). So that screams for
>> doing either this a) completely triggered by user space or b) completely
>> in the kernel. I prefer the latter. One interface less required.
>>
>> I would assume that no VCPU is allowed to be running inside KVM while
> 
> right.
> 
>> performing the PV switch, which would make this even easier.
> 
> Same as above. I like the idea. Will need to talk to Janosch. 

My initial code had a global VM switch and I quickly ran into the
problems of the complexity that such a solution poses.

The benefits of splitting were:
	* Less time spend in kernel for one single ioctl
	* Error handling can be split up into VM and VCPU (my biggest problem
initially)
	* Error reporting is more granular (i.e. ioctl returns error for VCPU
or VM)
	* It's in line with KVM's concept of VCPUs and VMs where we create the
VM and then each VCPU




[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [RFCv2 11/37] KVM: s390/mm: Make pages accessible before destroying the guest

[Cornelia Huck]

On Mon,  3 Feb 2020 08:19:31 -0500
Christian Borntraeger <borntraeger@de.ibm.com> wrote:

> Before we destroy the secure configuration, we better make all
> pages accessible again. This also happens during reboot, where we reboot
> into a non-secure guest that then can go again into a secure mode. As

s/a secure/secure/

> this "new" secure guest will have a new ID we cannot reuse the old page
> state.
> 
> Signed-off-by: Christian Borntraeger <borntraeger@de.ibm.com>
> ---
>  arch/s390/include/asm/pgtable.h |  1 +
>  arch/s390/kvm/pv.c              |  2 ++
>  arch/s390/mm/gmap.c             | 35 +++++++++++++++++++++++++++++++++
>  3 files changed, 38 insertions(+)

(...)

> +void s390_reset_acc(struct mm_struct *mm)
> +{
> +	/*
> +	 * we might be called during
> +	 * reset:                            we walk the pages and clear
> +	 * close of all kvm file descriptor: we walk the pages and clear

s/descriptor/descriptors/

> +	 * exit of process on fd closure:    vma already gone, do nothing
> +	 */
> +	if (!mmget_not_zero(mm))
> +		return;
> +	down_read(&mm->mmap_sem);
> +	walk_page_range(mm, 0, TASK_SIZE, &reset_acc_walk_ops, NULL);
> +	up_read(&mm->mmap_sem);
> +	mmput(mm);
> +}
> +EXPORT_SYMBOL_GPL(s390_reset_acc);

Reviewed-by: Cornelia Huck <cohuck@redhat.com>

Re: [RFCv2 14/37] KVM: s390: protvirt: Add interruption injection controls

[Cornelia Huck]

On Wed, 5 Feb 2020 09:54:23 +0100
Michael Mueller <mimu@linux.ibm.com> wrote:

> On 05.02.20 07:59, Thomas Huth wrote:
> > On 03/02/2020 14.19, Christian Borntraeger wrote:  
> >> From: Michael Mueller <mimu@linux.ibm.com>
> >>
> >> Define the interruption injection codes and the related fields in the
> >> sie control block for PVM interruption injection.  
> > 
> > You seem to only add the details for external interrupts and I/O
> > interrupts here? Maybe mention this in the description ... otherwise it
> > is confusing when you read patch 17 later ... or maybe merge this patch
> > here with patch 17 ?  
> 
> In that case I will merge patch 17 into this one.

Merging patch 17 sounds good to me, as it gets us all of the new
interrupt controls in one go.

> 
> >   
> >> Signed-off-by: Michael Mueller <mimu@linux.ibm.com>
> >> ---
> >>   arch/s390/include/asm/kvm_host.h | 25 +++++++++++++++++++++----
> >>   1 file changed, 21 insertions(+), 4 deletions(-)
> >>
> >> diff --git a/arch/s390/include/asm/kvm_host.h b/arch/s390/include/asm/kvm_host.h
> >> index 58845b315be0..a45d10d87a8a 100644
> >> --- a/arch/s390/include/asm/kvm_host.h
> >> +++ b/arch/s390/include/asm/kvm_host.h
> >> @@ -222,7 +222,15 @@ struct kvm_s390_sie_block {
> >>   	__u8	icptcode;		/* 0x0050 */
> >>   	__u8	icptstatus;		/* 0x0051 */
> >>   	__u16	ihcpu;			/* 0x0052 */
> >> -	__u8	reserved54[2];		/* 0x0054 */
> >> +	__u8	reserved54;		/* 0x0054 */
> >> +#define IICTL_CODE_NONE		 0x00
> >> +#define IICTL_CODE_MCHK		 0x01
> >> +#define IICTL_CODE_EXT		 0x02
> >> +#define IICTL_CODE_IO		 0x03
> >> +#define IICTL_CODE_RESTART	 0x04
> >> +#define IICTL_CODE_SPECIFICATION 0x10
> >> +#define IICTL_CODE_OPERAND	 0x11
> >> +	__u8	iictl;			/* 0x0055 */
> >>   	__u16	ipa;			/* 0x0056 */
> >>   	__u32	ipb;			/* 0x0058 */
> >>   	__u32	scaoh;			/* 0x005c */
> >> @@ -259,7 +267,8 @@ struct kvm_s390_sie_block {
> >>   #define HPID_KVM	0x4
> >>   #define HPID_VSIE	0x5
> >>   	__u8	hpid;			/* 0x00b8 */
> >> -	__u8	reservedb9[11];		/* 0x00b9 */
> >> +	__u8	reservedb9[7];		/* 0x00b9 */
> >> +	__u32	eiparams;		/* 0x00c0 */
> >>   	__u16	extcpuaddr;		/* 0x00c4 */
> >>   	__u16	eic;			/* 0x00c6 */
> >>   	__u32	reservedc8;		/* 0x00c8 */
> >> @@ -275,8 +284,16 @@ struct kvm_s390_sie_block {
> >>   	__u8	oai;			/* 0x00e2 */
> >>   	__u8	armid;			/* 0x00e3 */
> >>   	__u8	reservede4[4];		/* 0x00e4 */
> >> -	__u64	tecmc;			/* 0x00e8 */
> >> -	__u8	reservedf0[12];		/* 0x00f0 */
> >> +	union {
> >> +		__u64	tecmc;		/* 0x00e8 */
> >> +		struct {
> >> +			__u16	subchannel_id;	/* 0x00e8 */
> >> +			__u16	subchannel_nr;	/* 0x00ea */
> >> +			__u32	io_int_parm;	/* 0x00ec */
> >> +			__u32	io_int_word;	/* 0x00f0 */
> >> +		};
> >> +	} __packed;
> >> +	__u8	reservedf4[8];		/* 0x00f4 */  
> > 
> > Maybe add a comment to the new struct for which injection type it is
> > good for ... otherwise this might get hard to understand in the future
> > (especially if more stuff gets added like in patch 17).  
> 
> Up to know we don't have comments in the the SIE control block struct at 
> all as this is part of the documentation, also for protected virtualization.

FWIW, I find the sie control block structure already hard to understand
right now. Some documentation would be useful, if possible.

> 
> Any other opinions on that are welcome.
> 
> > 
> > Anyway,
> > Reviewed-by: Thomas Huth <thuth@redhat.com>  
> 
> thanks
> 
> >   
> 
> Michael
> 

Re: [RFCv2 15/37] KVM: s390: protvirt: Implement interruption injection

[Christian Borntraeger]

On 05.02.20 10:51, Thomas Huth wrote:
> On 03/02/2020 14.19, Christian Borntraeger wrote:
>> From: Michael Mueller <mimu@linux.ibm.com>
>>
>> The patch implements interruption injection for the following
>> list of interruption types:
>>
>>   - I/O
>>     __deliver_io (III)
>>
>>   - External
>>     __deliver_cpu_timer (IEI)
>>     __deliver_ckc (IEI)
>>     __deliver_emergency_signal (IEI)
>>     __deliver_external_call (IEI)
>>
>>   - cpu restart
>>     __deliver_restart (IRI)
>>
>> The service interrupt is handled in a followup patch.
>>
>> Signed-off-by: Michael Mueller <mimu@linux.ibm.com>
>> Signed-off-by: Christian Borntraeger <borntraeger@de.ibm.com>
>> [fixes]
>> ---
>>  arch/s390/include/asm/kvm_host.h |  8 +++
>>  arch/s390/kvm/interrupt.c        | 93 ++++++++++++++++++++++----------
>>  2 files changed, 74 insertions(+), 27 deletions(-)
>>
>> diff --git a/arch/s390/include/asm/kvm_host.h b/arch/s390/include/asm/kvm_host.h
>> index a45d10d87a8a..989cea7a5591 100644
>> --- a/arch/s390/include/asm/kvm_host.h
>> +++ b/arch/s390/include/asm/kvm_host.h
>> @@ -563,6 +563,14 @@ enum irq_types {
>>  #define IRQ_PEND_MCHK_MASK ((1UL << IRQ_PEND_MCHK_REP) | \
>>  			    (1UL << IRQ_PEND_MCHK_EX))
>>  
>> +#define IRQ_PEND_MCHK_REP_MASK (1UL << IRQ_PEND_MCHK_REP)
>> +
>> +#define IRQ_PEND_EXT_II_MASK ((1UL << IRQ_PEND_EXT_CPU_TIMER)  | \
>> +			      (1UL << IRQ_PEND_EXT_CLOCK_COMP) | \
>> +			      (1UL << IRQ_PEND_EXT_EMERGENCY)  | \
>> +			      (1UL << IRQ_PEND_EXT_EXTERNAL)   | \
>> +			      (1UL << IRQ_PEND_EXT_SERVICE))
>> +
>>  struct kvm_s390_interrupt_info {
>>  	struct list_head list;
>>  	u64	type;
>> diff --git a/arch/s390/kvm/interrupt.c b/arch/s390/kvm/interrupt.c
>> index c06c89d370a7..ecdec6960a60 100644
>> --- a/arch/s390/kvm/interrupt.c
>> +++ b/arch/s390/kvm/interrupt.c
>> @@ -387,6 +387,12 @@ static unsigned long deliverable_irqs(struct kvm_vcpu *vcpu)
>>  		__clear_bit(IRQ_PEND_EXT_SERVICE, &active_mask);
>>  	if (psw_mchk_disabled(vcpu))
>>  		active_mask &= ~IRQ_PEND_MCHK_MASK;
>> +	/* PV guest cpus can have a single interruption injected at a time. */
>> +	if (kvm_s390_pv_is_protected(vcpu->kvm) &&
>> +	    vcpu->arch.sie_block->iictl != IICTL_CODE_NONE)
>> +		active_mask &= ~(IRQ_PEND_EXT_II_MASK |
>> +				 IRQ_PEND_IO_MASK |
>> +				 IRQ_PEND_MCHK_REP_MASK);
> 
> I don't quite understand why there is a difference between
> IRQ_PEND_MCHK_REP and IRQ_PEND_MCHK_EX here? Why not simply use
> IRQ_PEND_MCHK_MASK here? Could you elaborate? (and maybe add a sentence
> to the patch description)

I added that part. 
My idea was that an exigent machine check would be kind of fatal that it can override
the previous interrupt. Now we do not implement the override (kill the previous interrupt)
so I agree, maybe lets use IRQ_PEND_MCHK_MASK

>> @@ -533,7 +547,6 @@ static int __must_check __deliver_pfault_init(struct kvm_vcpu *vcpu)
>>  	trace_kvm_s390_deliver_interrupt(vcpu->vcpu_id,
>>  					 KVM_S390_INT_PFAULT_INIT,
>>  					 0, ext.ext_params2);
>> -

Ack.

Re: [RFCv2 20/37] KVM: s390: protvirt: Add new gprs location handling

[Thomas Huth]

On 03/02/2020 14.19, Christian Borntraeger wrote:
> From: Janosch Frank <frankja@linux.ibm.com>
> 
> Guest registers for protected guests are stored at offset 0x380.
> 
> Signed-off-by: Janosch Frank <frankja@linux.ibm.com>
> ---
>  arch/s390/include/asm/kvm_host.h |  4 +++-
>  arch/s390/kvm/kvm-s390.c         | 11 +++++++++++
>  2 files changed, 14 insertions(+), 1 deletion(-)
> 
> diff --git a/arch/s390/include/asm/kvm_host.h b/arch/s390/include/asm/kvm_host.h
> index f5ca53574406..125511ec6eb0 100644
> --- a/arch/s390/include/asm/kvm_host.h
> +++ b/arch/s390/include/asm/kvm_host.h
> @@ -343,7 +343,9 @@ struct kvm_s390_itdb {
>  struct sie_page {
>  	struct kvm_s390_sie_block sie_block;
>  	struct mcck_volatile_info mcck_info;	/* 0x0200 */
> -	__u8 reserved218[1000];		/* 0x0218 */
> +	__u8 reserved218[360];		/* 0x0218 */
> +	__u64 pv_grregs[16];		/* 0x380 */

s/0x380/0x0380/ to align with the other comments

> +	__u8 reserved400[512];

Maybe add a /* 0x0400 */ comment ... though it's obvious from the name
already.

>  	struct kvm_s390_itdb itdb;	/* 0x0600 */
>  	__u8 reserved700[2304];		/* 0x0700 */
>  };
> diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
> index 39bf39a10cf2..1945180b857a 100644
> --- a/arch/s390/kvm/kvm-s390.c
> +++ b/arch/s390/kvm/kvm-s390.c
> @@ -3996,6 +3996,7 @@ static int vcpu_post_run(struct kvm_vcpu *vcpu, int exit_reason)
>  static int __vcpu_run(struct kvm_vcpu *vcpu)
>  {
>  	int rc, exit_reason;
> +	struct sie_page *sie_page = (struct sie_page *)vcpu->arch.sie_block;
>  
>  	/*
>  	 * We try to hold kvm->srcu during most of vcpu_run (except when run-
> @@ -4017,8 +4018,18 @@ static int __vcpu_run(struct kvm_vcpu *vcpu)
>  		guest_enter_irqoff();
>  		__disable_cpu_timer_accounting(vcpu);
>  		local_irq_enable();
> +		if (kvm_s390_pv_is_protected(vcpu->kvm)) {
> +			memcpy(sie_page->pv_grregs,
> +			       vcpu->run->s.regs.gprs,
> +			       sizeof(sie_page->pv_grregs));
> +		}
>  		exit_reason = sie64a(vcpu->arch.sie_block,
>  				     vcpu->run->s.regs.gprs);
> +		if (kvm_s390_pv_is_protected(vcpu->kvm)) {
> +			memcpy(vcpu->run->s.regs.gprs,
> +			       sie_page->pv_grregs,
> +			       sizeof(sie_page->pv_grregs));
> +		}
>  		local_irq_disable();
>  		__enable_cpu_timer_accounting(vcpu);
>  		guest_exit_irqoff();
> 

Reviewed-by: Thomas Huth <thuth@redhat.com>

Re: [RFCv2 15/37] KVM: s390: protvirt: Implement interruption injection

[Cornelia Huck]

On Wed, 5 Feb 2020 11:48:42 +0100
Christian Borntraeger <borntraeger@de.ibm.com> wrote:

> On 05.02.20 10:51, Thomas Huth wrote:
> > On 03/02/2020 14.19, Christian Borntraeger wrote:  
> >> From: Michael Mueller <mimu@linux.ibm.com>
> >>
> >> The patch implements interruption injection for the following
> >> list of interruption types:
> >>
> >>   - I/O
> >>     __deliver_io (III)
> >>
> >>   - External
> >>     __deliver_cpu_timer (IEI)
> >>     __deliver_ckc (IEI)
> >>     __deliver_emergency_signal (IEI)
> >>     __deliver_external_call (IEI)
> >>
> >>   - cpu restart
> >>     __deliver_restart (IRI)
> >>
> >> The service interrupt is handled in a followup patch.
> >>
> >> Signed-off-by: Michael Mueller <mimu@linux.ibm.com>
> >> Signed-off-by: Christian Borntraeger <borntraeger@de.ibm.com>
> >> [fixes]
> >> ---
> >>  arch/s390/include/asm/kvm_host.h |  8 +++
> >>  arch/s390/kvm/interrupt.c        | 93 ++++++++++++++++++++++----------
> >>  2 files changed, 74 insertions(+), 27 deletions(-)
> >>
> >> diff --git a/arch/s390/include/asm/kvm_host.h b/arch/s390/include/asm/kvm_host.h
> >> index a45d10d87a8a..989cea7a5591 100644
> >> --- a/arch/s390/include/asm/kvm_host.h
> >> +++ b/arch/s390/include/asm/kvm_host.h
> >> @@ -563,6 +563,14 @@ enum irq_types {
> >>  #define IRQ_PEND_MCHK_MASK ((1UL << IRQ_PEND_MCHK_REP) | \
> >>  			    (1UL << IRQ_PEND_MCHK_EX))
> >>  
> >> +#define IRQ_PEND_MCHK_REP_MASK (1UL << IRQ_PEND_MCHK_REP)
> >> +
> >> +#define IRQ_PEND_EXT_II_MASK ((1UL << IRQ_PEND_EXT_CPU_TIMER)  | \
> >> +			      (1UL << IRQ_PEND_EXT_CLOCK_COMP) | \
> >> +			      (1UL << IRQ_PEND_EXT_EMERGENCY)  | \
> >> +			      (1UL << IRQ_PEND_EXT_EXTERNAL)   | \
> >> +			      (1UL << IRQ_PEND_EXT_SERVICE))
> >> +
> >>  struct kvm_s390_interrupt_info {
> >>  	struct list_head list;
> >>  	u64	type;
> >> diff --git a/arch/s390/kvm/interrupt.c b/arch/s390/kvm/interrupt.c
> >> index c06c89d370a7..ecdec6960a60 100644
> >> --- a/arch/s390/kvm/interrupt.c
> >> +++ b/arch/s390/kvm/interrupt.c
> >> @@ -387,6 +387,12 @@ static unsigned long deliverable_irqs(struct kvm_vcpu *vcpu)
> >>  		__clear_bit(IRQ_PEND_EXT_SERVICE, &active_mask);
> >>  	if (psw_mchk_disabled(vcpu))
> >>  		active_mask &= ~IRQ_PEND_MCHK_MASK;
> >> +	/* PV guest cpus can have a single interruption injected at a time. */
> >> +	if (kvm_s390_pv_is_protected(vcpu->kvm) &&
> >> +	    vcpu->arch.sie_block->iictl != IICTL_CODE_NONE)
> >> +		active_mask &= ~(IRQ_PEND_EXT_II_MASK |
> >> +				 IRQ_PEND_IO_MASK |
> >> +				 IRQ_PEND_MCHK_REP_MASK);  
> > 
> > I don't quite understand why there is a difference between
> > IRQ_PEND_MCHK_REP and IRQ_PEND_MCHK_EX here? Why not simply use
> > IRQ_PEND_MCHK_MASK here? Could you elaborate? (and maybe add a sentence
> > to the patch description)  
> 
> I added that part. 
> My idea was that an exigent machine check would be kind of fatal that it can override
> the previous interrupt. Now we do not implement the override (kill the previous interrupt)
> so I agree, maybe lets use IRQ_PEND_MCHK_MASK

This makes me wonder about interrupt priorities in general. Under which
circumstances can we have an interrupt with a lower priority already
injected (but not delivered) in the injection area when a
higher-priority interrupt comes along? I'm a bit confused here.

Re: [RFCv2 15/37] KVM: s390: protvirt: Implement interruption injection

[Cornelia Huck]

On Mon,  3 Feb 2020 08:19:35 -0500
Christian Borntraeger <borntraeger@de.ibm.com> wrote:

> From: Michael Mueller <mimu@linux.ibm.com>
> 
> The patch implements interruption injection for the following
> list of interruption types:
> 
>   - I/O
>     __deliver_io (III)
> 
>   - External
>     __deliver_cpu_timer (IEI)
>     __deliver_ckc (IEI)
>     __deliver_emergency_signal (IEI)
>     __deliver_external_call (IEI)
> 
>   - cpu restart
>     __deliver_restart (IRI)

Hm... what do 'III', 'IEI', and 'IRI' stand for?

> 
> The service interrupt is handled in a followup patch.
> 
> Signed-off-by: Michael Mueller <mimu@linux.ibm.com>
> Signed-off-by: Christian Borntraeger <borntraeger@de.ibm.com>
> [fixes]
> ---
>  arch/s390/include/asm/kvm_host.h |  8 +++
>  arch/s390/kvm/interrupt.c        | 93 ++++++++++++++++++++++----------
>  2 files changed, 74 insertions(+), 27 deletions(-)
> 
> diff --git a/arch/s390/include/asm/kvm_host.h b/arch/s390/include/asm/kvm_host.h
> index a45d10d87a8a..989cea7a5591 100644
> --- a/arch/s390/include/asm/kvm_host.h
> +++ b/arch/s390/include/asm/kvm_host.h
> @@ -563,6 +563,14 @@ enum irq_types {
>  #define IRQ_PEND_MCHK_MASK ((1UL << IRQ_PEND_MCHK_REP) | \
>  			    (1UL << IRQ_PEND_MCHK_EX))
>  
> +#define IRQ_PEND_MCHK_REP_MASK (1UL << IRQ_PEND_MCHK_REP)
> +
> +#define IRQ_PEND_EXT_II_MASK ((1UL << IRQ_PEND_EXT_CPU_TIMER)  | \

What does 'II' stand for? Interrupt Injection?

> +			      (1UL << IRQ_PEND_EXT_CLOCK_COMP) | \
> +			      (1UL << IRQ_PEND_EXT_EMERGENCY)  | \
> +			      (1UL << IRQ_PEND_EXT_EXTERNAL)   | \
> +			      (1UL << IRQ_PEND_EXT_SERVICE))
> +
>  struct kvm_s390_interrupt_info {
>  	struct list_head list;
>  	u64	type;

(...)

> @@ -1834,7 +1872,8 @@ static void __floating_irq_kick(struct kvm *kvm, u64 type)
>  		break;
>  	case KVM_S390_INT_IO_MIN...KVM_S390_INT_IO_MAX:
>  		if (!(type & KVM_S390_INT_IO_AI_MASK &&
> -		      kvm->arch.gisa_int.origin))
> +		      kvm->arch.gisa_int.origin) ||
> +		      kvm_s390_pv_handle_cpu(dst_vcpu))
>  			kvm_s390_set_cpuflags(dst_vcpu, CPUSTAT_IO_INT);
>  		break;
>  	default:

Looking at this... can you also talk about protected virt vs. exitless
interrupts?

Re: [RFCv2 00/37] KVM: s390: Add support for protected VMs

[David Hildenbrand]

On 03.02.20 14:19, Christian Borntraeger wrote:
> Protected VMs (PVM) are KVM VMs, where KVM can't access the VM's state
> like guest memory and guest registers anymore. Instead the PVMs are
> mostly managed by a new entity called Ultravisor (UV), which provides
> an API, so KVM and the PV can request management actions.
> 
> PVMs are encrypted at rest and protected from hypervisor access while
> running. They switch from a normal operation into protected mode, so
> we can still use the standard boot process to load a encrypted blob
> and then move it into protected mode.
> 
> Rebooting is only possible by passing through the unprotected/normal
> mode and switching to protected again.
> 
> All patches are in the protvirtv2 branch of the korg s390 kvm git
> (on top of Janoschs reset rework).
> 
> Claudio presented the technology at his presentation at KVM Forum
> 2019.
> 
> This contains a "pretty small" common code memory management change that
> will allow paging, guest backing with files etc almost just like normal
> VMs. Please note that the memory management part will still see some
> changes to deal with a corner case for the adapter interrupt indicator
> pages. So please focus on the non-mm parts (which hopefully has
> everthing addressed in the next version). Claudio will work with Andrea
> regarding this.
> 
> Christian Borntraeger (3):
>   KVM: s390/mm: Make pages accessible before destroying the guest
>   KVM: s390: protvirt: Add SCLP interrupt handling
>   KVM: s390: protvirt: do not inject interrupts after start
> 
> Claudio Imbrenda (3):
>   mm:gup/writeback: add callbacks for inaccessible pages
>   s390/mm: provide memory management functions for protected KVM guests
>   KVM: s390/mm: handle guest unpin events
> 
> Janosch Frank (24):
>   DOCUMENTATION: protvirt: Protected virtual machine introduction
>   KVM: s390: add new variants of UV CALL
>   KVM: s390: protvirt: Add initial lifecycle handling
>   KVM: s390: protvirt: Add KVM api documentation
>   KVM: s390: protvirt: Secure memory is not mergeable
>   KVM: s390: protvirt: Handle SE notification interceptions
>   KVM: s390: protvirt: Instruction emulation
>   KVM: s390: protvirt: Handle spec exception loops
>   KVM: s390: protvirt: Add new gprs location handling
>   KVM: S390: protvirt: Introduce instruction data area bounce buffer
>   KVM: s390: protvirt: handle secure guest prefix pages
>   KVM: s390: protvirt: Write sthyi data to instruction data area
>   KVM: s390: protvirt: STSI handling
>   KVM: s390: protvirt: disallow one_reg
>   KVM: s390: protvirt: Only sync fmt4 registers
>   KVM: s390: protvirt: Add program exception injection
>   DOCUMENTATION: protvirt: Diag 308 IPL
>   KVM: s390: protvirt: Add diag 308 subcode 8 - 10 handling
>   KVM: s390: protvirt: UV calls diag308 0, 1
>   KVM: s390: protvirt: Report CPU state to Ultravisor
>   KVM: s390: protvirt: Support cmd 5 operation state
>   KVM: s390: protvirt: Add UV debug trace
>   KVM: s390: protvirt: Mask PSW interrupt bits for interception 104 and
>     112
>   KVM: s390: protvirt: Add UV cpu reset calls
> 
> Michael Mueller (4):
>   KVM: s390: protvirt: Add interruption injection controls
>   KVM: s390: protvirt: Implement interruption injection
>   KVM: s390: protvirt: Add machine-check interruption injection controls
>   KVM: s390: protvirt: Implement machine-check interruption injection
> 
> Vasily Gorbik (3):
>   s390/protvirt: introduce host side setup
>   s390/protvirt: add ultravisor initialization
>   s390: add (non)secure page access exceptions handlers
> 
>  .../admin-guide/kernel-parameters.txt         |   5 +
>  Documentation/virt/kvm/api.txt                |  62 +++
>  Documentation/virt/kvm/s390-pv-boot.rst       |  64 +++
>  Documentation/virt/kvm/s390-pv.rst            | 103 ++++
>  MAINTAINERS                                   |   1 +
>  arch/s390/boot/Makefile                       |   2 +-
>  arch/s390/boot/uv.c                           |  20 +-
>  arch/s390/include/asm/gmap.h                  |   3 +
>  arch/s390/include/asm/kvm_host.h              | 112 +++-
>  arch/s390/include/asm/mmu.h                   |   2 +
>  arch/s390/include/asm/mmu_context.h           |   1 +
>  arch/s390/include/asm/page.h                  |   5 +
>  arch/s390/include/asm/pgtable.h               |  35 +-
>  arch/s390/include/asm/uv.h                    | 265 +++++++++-
>  arch/s390/kernel/Makefile                     |   1 +
>  arch/s390/kernel/pgm_check.S                  |   4 +-
>  arch/s390/kernel/setup.c                      |   7 +-
>  arch/s390/kernel/uv.c                         | 271 ++++++++++
>  arch/s390/kvm/Kconfig                         |  19 +
>  arch/s390/kvm/Makefile                        |   2 +-
>  arch/s390/kvm/diag.c                          |   7 +
>  arch/s390/kvm/intercept.c                     | 107 +++-
>  arch/s390/kvm/interrupt.c                     | 211 ++++++--
>  arch/s390/kvm/kvm-s390.c                      | 486 ++++++++++++++++--
>  arch/s390/kvm/kvm-s390.h                      |  56 ++
>  arch/s390/kvm/priv.c                          |   9 +-
>  arch/s390/kvm/pv.c                            | 293 +++++++++++
>  arch/s390/mm/fault.c                          |  87 ++++
>  arch/s390/mm/gmap.c                           |  63 ++-
>  include/linux/gfp.h                           |   6 +
>  include/uapi/linux/kvm.h                      |  47 +-
>  mm/gup.c                                      |   2 +
>  mm/page-writeback.c                           |   1 +
>  33 files changed, 2217 insertions(+), 142 deletions(-)
>  create mode 100644 Documentation/virt/kvm/s390-pv-boot.rst
>  create mode 100644 Documentation/virt/kvm/s390-pv.rst
>  create mode 100644 arch/s390/kernel/uv.c
>  create mode 100644 arch/s390/kvm/pv.c
> 

Due to the huge amount of review feedback (which makestime-consuming to
review if one doesn't want to comment the same thing again), I suggest
sending a new RFC rather soon-ish (e.g., on a weekly basis) - if nobody
objects. Would at least make my life easier :)

-- 
Thanks,

David / dhildenb

Re: [RFCv2 00/37] KVM: s390: Add support for protected VMs

[Christian Borntraeger]

On 05.02.20 12:34, David Hildenbrand wrote:

> Due to the huge amount of review feedback (which makestime-consuming to
> review if one doesn't want to comment the same thing again), I suggest
> sending a new RFC rather soon-ish (e.g., on a weekly basis) - if nobody
> objects. Would at least make my life easier :)

I can send one this evening?

Re: [RFCv2 00/37] KVM: s390: Add support for protected VMs

[Cornelia Huck]

On Wed, 5 Feb 2020 12:38:07 +0100
Christian Borntraeger <borntraeger@de.ibm.com> wrote:

> On 05.02.20 12:34, David Hildenbrand wrote:
> 
> > Due to the huge amount of review feedback (which makestime-consuming to
> > review if one doesn't want to comment the same thing again), I suggest
> > sending a new RFC rather soon-ish (e.g., on a weekly basis) - if nobody
> > objects. Would at least make my life easier :)  
> 
> I can send one this evening?
> 

Perhaps not that fast; I have not yet made my way through all patches
yet... let's wait until the end of the week?

Re: [RFCv2 21/37] KVM: S390: protvirt: Introduce instruction data area bounce buffer

[David Hildenbrand]

>  #ifdef CONFIG_KVM_S390_PROTECTED_VIRTUALIZATION_HOST
> +static long kvm_s390_guest_sida_op(struct kvm_vcpu *vcpu,
> +				   struct kvm_s390_mem_op *mop)
> +{
> +	int r = 0;
> +	void __user *uaddr = (void __user *)mop->buf;

Reverse christmas tree :)

> +
> +	if (mop->flags || !mop->size)
> +		return -EINVAL;
> +
> +	if (mop->size > sida_size(vcpu->arch.sie_block))
> +		return -E2BIG;

Should be caught by the check below as well (or is this an implicit
overflow check? - see below).

> +
> +	if (mop->size + mop->offset > sida_size(vcpu->arch.sie_block))
> +		return -E2BIG;
> +

Do we have to care about overflows? (at least the offset is 32-bit,
didn't check the size :))


> +	switch (mop->op) {
> +	case KVM_S390_MEMOP_SIDA_READ:
> +		r = 0;
> +		if (copy_to_user(uaddr, (void *)sida_origin(vcpu->arch.sie_block) +
> +				 mop->offset, mop->size))
> +			r = -EFAULT;
> +
> +		break;
> +	case KVM_S390_MEMOP_SIDA_WRITE:
> +		r = 0;
> +		if (copy_from_user((void *)vcpu->arch.sie_block->sidad +
> +				   mop->offset, uaddr, mop->size))
> +			r = -EFAULT;
> +		break;
> +	}
> +	return r;
> +}
> +
>  static int kvm_s390_handle_pv_vcpu(struct kvm_vcpu *vcpu,
>  				   struct kvm_pv_cmd *cmd)
>  {
> @@ -4708,6 +4743,20 @@ long kvm_arch_vcpu_ioctl(struct file *filp,
>  		r = kvm_s390_handle_pv_vcpu(vcpu, &args);
>  		break;
>  	}
> +	case KVM_S390_SIDA_OP: {
> +		struct kvm_s390_mem_op mem_op;
> +
> +		if (!kvm_s390_pv_is_protected(vcpu->kvm)) {
> +			r = -EINVAL;
> +			break;
> +		}

Could we race against a VM_DESTROY? Should we protect somehow?
[...]

> -/* for KVM_S390_MEM_OP */
> +/* for KVM_S390_MEM_OP and KVM_S390_SIDA_OP */
>  struct kvm_s390_mem_op {
>  	/* in */
>  	__u64 gaddr;		/* the guest address */
> @@ -475,11 +475,17 @@ struct kvm_s390_mem_op {
>  	__u32 op;		/* type of operation */
>  	__u64 buf;		/* buffer in userspace */
>  	__u8 ar;		/* the access register number */
> -	__u8 reserved[31];	/* should be set to 0 */
> +	__u8 reserved21[3];	/* should be set to 0 */
> +	__u32 offset;		/* offset into the sida */

maybe "side_offset"? or define a union, overlying the ar (because that
obviously doesn't apply to this memop). So eventually different layout
for different memop.

-- 
Thanks,

David / dhildenb

Re: [RFCv2 15/37] KVM: s390: protvirt: Implement interruption injection

[Michael Mueller]

On 05.02.20 12:31, Cornelia Huck wrote:
> On Mon,  3 Feb 2020 08:19:35 -0500
> Christian Borntraeger <borntraeger@de.ibm.com> wrote:
> 
>> From: Michael Mueller <mimu@linux.ibm.com>
>>
>> The patch implements interruption injection for the following
>> list of interruption types:
>>
>>    - I/O
>>      __deliver_io (III)
>>
>>    - External
>>      __deliver_cpu_timer (IEI)
>>      __deliver_ckc (IEI)
>>      __deliver_emergency_signal (IEI)
>>      __deliver_external_call (IEI)
>>
>>    - cpu restart
>>      __deliver_restart (IRI)
> 
> Hm... what do 'III', 'IEI', and 'IRI' stand for?

that's the kind of interruption injection being used:

inject io interruption
inject external interruption
inject restart interruption

> 
>>
>> The service interrupt is handled in a followup patch.
>>
>> Signed-off-by: Michael Mueller <mimu@linux.ibm.com>
>> Signed-off-by: Christian Borntraeger <borntraeger@de.ibm.com>
>> [fixes]
>> ---
>>   arch/s390/include/asm/kvm_host.h |  8 +++
>>   arch/s390/kvm/interrupt.c        | 93 ++++++++++++++++++++++----------
>>   2 files changed, 74 insertions(+), 27 deletions(-)
>>
>> diff --git a/arch/s390/include/asm/kvm_host.h b/arch/s390/include/asm/kvm_host.h
>> index a45d10d87a8a..989cea7a5591 100644
>> --- a/arch/s390/include/asm/kvm_host.h
>> +++ b/arch/s390/include/asm/kvm_host.h
>> @@ -563,6 +563,14 @@ enum irq_types {
>>   #define IRQ_PEND_MCHK_MASK ((1UL << IRQ_PEND_MCHK_REP) | \
>>   			    (1UL << IRQ_PEND_MCHK_EX))
>>   
>> +#define IRQ_PEND_MCHK_REP_MASK (1UL << IRQ_PEND_MCHK_REP)
>> +
>> +#define IRQ_PEND_EXT_II_MASK ((1UL << IRQ_PEND_EXT_CPU_TIMER)  | \
> 
> What does 'II' stand for? Interrupt Injection?

interruption injection

> 
>> +			      (1UL << IRQ_PEND_EXT_CLOCK_COMP) | \
>> +			      (1UL << IRQ_PEND_EXT_EMERGENCY)  | \
>> +			      (1UL << IRQ_PEND_EXT_EXTERNAL)   | \
>> +			      (1UL << IRQ_PEND_EXT_SERVICE))
>> +
>>   struct kvm_s390_interrupt_info {
>>   	struct list_head list;
>>   	u64	type;
> 
> (...)
> 
>> @@ -1834,7 +1872,8 @@ static void __floating_irq_kick(struct kvm *kvm, u64 type)
>>   		break;
>>   	case KVM_S390_INT_IO_MIN...KVM_S390_INT_IO_MAX:
>>   		if (!(type & KVM_S390_INT_IO_AI_MASK &&
>> -		      kvm->arch.gisa_int.origin))
>> +		      kvm->arch.gisa_int.origin) ||
>> +		      kvm_s390_pv_handle_cpu(dst_vcpu))
>>   			kvm_s390_set_cpuflags(dst_vcpu, CPUSTAT_IO_INT);
>>   		break;
>>   	default:
> 
> Looking at this... can you also talk about protected virt vs. exitless
> interrupts?
> 

Michael

Re: [RFCv2 22/37] KVM: s390: protvirt: handle secure guest prefix pages

[David Hildenbrand]

On 03.02.20 14:19, Christian Borntraeger wrote:
> From: Janosch Frank <frankja@linux.ibm.com>
> 
> The SPX instruction is handled by the ulravisor. We do get a
> notification intercept, though. Let us update our internal view.
> 
> In addition to that, when the guest prefix page is not secure, an
> intercept 112 (0x70) is indicated.  To avoid this for the most common
> cases, we can make the guest prefix page protected whenever we pin it.
> We have to deal with 112 nevertheless, e.g. when some host code triggers
> an export (e.g. qemu dump guest memory). We can simply re-run the
> pinning logic by doing a no-op prefix change.
> 
> Signed-off-by: Janosch Frank <frankja@linux.ibm.com>
> ---
>  arch/s390/include/asm/kvm_host.h |  1 +
>  arch/s390/kvm/intercept.c        | 15 +++++++++++++++
>  arch/s390/kvm/kvm-s390.c         | 14 ++++++++++++++
>  3 files changed, 30 insertions(+)
> 
> diff --git a/arch/s390/include/asm/kvm_host.h b/arch/s390/include/asm/kvm_host.h
> index 48f382680755..686b00ced55b 100644
> --- a/arch/s390/include/asm/kvm_host.h
> +++ b/arch/s390/include/asm/kvm_host.h
> @@ -225,6 +225,7 @@ struct kvm_s390_sie_block {
>  #define ICPT_PV_INT_EN	0x64
>  #define ICPT_PV_INSTR	0x68
>  #define ICPT_PV_NOTIF	0x6c
> +#define ICPT_PV_PREF	0x70
>  	__u8	icptcode;		/* 0x0050 */
>  	__u8	icptstatus;		/* 0x0051 */
>  	__u16	ihcpu;			/* 0x0052 */
> diff --git a/arch/s390/kvm/intercept.c b/arch/s390/kvm/intercept.c
> index d63f9cf10360..ceba0abb1900 100644
> --- a/arch/s390/kvm/intercept.c
> +++ b/arch/s390/kvm/intercept.c
> @@ -451,6 +451,15 @@ static int handle_operexc(struct kvm_vcpu *vcpu)
>  	return kvm_s390_inject_program_int(vcpu, PGM_OPERATION);
>  }
>  
> +static int handle_pv_spx(struct kvm_vcpu *vcpu)
> +{
> +	u32 pref = *(u32 *)vcpu->arch.sie_block->sidad;
> +
> +	kvm_s390_set_prefix(vcpu, pref);
> +	trace_kvm_s390_handle_prefix(vcpu, 1, pref);
> +	return 0;
> +}
> +
>  static int handle_pv_sclp(struct kvm_vcpu *vcpu)
>  {
>  	struct kvm_s390_float_interrupt *fi = &vcpu->kvm->arch.float_int;
> @@ -475,6 +484,8 @@ static int handle_pv_sclp(struct kvm_vcpu *vcpu)
>  
>  static int handle_pv_not(struct kvm_vcpu *vcpu)
>  {
> +	if (vcpu->arch.sie_block->ipa == 0xb210)
> +		return handle_pv_spx(vcpu);
>  	if (vcpu->arch.sie_block->ipa == 0xb220)
>  		return handle_pv_sclp(vcpu);
>  
> @@ -533,6 +544,10 @@ int kvm_handle_sie_intercept(struct kvm_vcpu *vcpu)
>  	case ICPT_PV_NOTIF:
>  		rc = handle_pv_not(vcpu);
>  		break;
> +	case ICPT_PV_PREF:
> +		rc = 0;
> +		kvm_s390_set_prefix(vcpu, kvm_s390_get_prefix(vcpu));

/me confused

This is the "request to map prefix" case, right?

I'd *really* prefer to have a comment and a manual

/* request to convert and pin the prefix pages again */
kvm_make_request(KVM_REQ_MMU_RELOAD, vcpu)

A TLB flush is IMHO not necessary, as the prefix did not change.

> +		break;
>  	default:
>  		return -EOPNOTSUPP;
>  	}
> diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
> index 76303b0f1226..6e74c7afae3a 100644
> --- a/arch/s390/kvm/kvm-s390.c
> +++ b/arch/s390/kvm/kvm-s390.c
> @@ -3675,6 +3675,20 @@ static int kvm_s390_handle_requests(struct kvm_vcpu *vcpu)
>  		rc = gmap_mprotect_notify(vcpu->arch.gmap,
>  					  kvm_s390_get_prefix(vcpu),
>  					  PAGE_SIZE * 2, PROT_WRITE);
> +		if (!rc && kvm_s390_pv_is_protected(vcpu->kvm)) {
> +			do {
> +				rc = uv_convert_to_secure(
> +						vcpu->arch.gmap,
> +						kvm_s390_get_prefix(vcpu));
> +			} while (rc == -EAGAIN);
> +			WARN_ONCE(rc, "Error while importing first prefix page. rc %d", rc);
> +			do {
> +				rc = uv_convert_to_secure(
> +						vcpu->arch.gmap,
> +						kvm_s390_get_prefix(vcpu) + PAGE_SIZE);
> +			} while (rc == -EAGAIN);
> +			WARN_ONCE(rc, "Error while importing second prefix page. rc %d", rc);

Maybe factor that out into a separate function (e.g., for a single page
and call that twice).

> +		}
>  		if (rc) {
>  			kvm_make_request(KVM_REQ_MMU_RELOAD, vcpu);
>  			return rc;
> 


-- 
Thanks,

David / dhildenb

Re: [RFCv2 22/37] KVM: s390: protvirt: handle secure guest prefix pages

[David Hildenbrand]

On 03.02.20 14:19, Christian Borntraeger wrote:
> From: Janosch Frank <frankja@linux.ibm.com>
> 
> The SPX instruction is handled by the ulravisor. We do get a

s/ulravisor/ultravisor/

> notification intercept, though. Let us update our internal view.
> 
> In addition to that, when the guest prefix page is not secure, an
> intercept 112 (0x70) is indicated.  To avoid this for the most common
> cases, we can make the guest prefix page protected whenever we pin it.
> We have to deal with 112 nevertheless, e.g. when some host code triggers
> an export (e.g. qemu dump guest memory). We can simply re-run the
> pinning logic by doing a no-op prefix change.
> 
> Signed-off-by: Janosch Frank <frankja@linux.ibm.com>


-- 
Thanks,

David / dhildenb

Re: [RFCv2 21/37] KVM: S390: protvirt: Introduce instruction data area bounce buffer

[Thomas Huth]

On 03/02/2020 14.19, Christian Borntraeger wrote:
> From: Janosch Frank <frankja@linux.ibm.com>
> 
> Now that we can't access guest memory anymore, we have a dedicated
> sattelite block that's a bounce buffer for instruction data.
> 
> We re-use the memop interface to copy the instruction data to / from
> userspace. This lets us re-use a lot of QEMU code which used that
> interface to make logical guest memory accesses which are not possible
> anymore in protected mode anyway.
> 
> Signed-off-by: Janosch Frank <frankja@linux.ibm.com>
> ---
[...]
> diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
> index eab741bc12c3..20969ce12096 100644
> --- a/include/uapi/linux/kvm.h
> +++ b/include/uapi/linux/kvm.h
> @@ -466,7 +466,7 @@ struct kvm_translation {
>  	__u8  pad[5];
>  };
>  
> -/* for KVM_S390_MEM_OP */
> +/* for KVM_S390_MEM_OP and KVM_S390_SIDA_OP */
>  struct kvm_s390_mem_op {
>  	/* in */
>  	__u64 gaddr;		/* the guest address */
> @@ -475,11 +475,17 @@ struct kvm_s390_mem_op {
>  	__u32 op;		/* type of operation */
>  	__u64 buf;		/* buffer in userspace */
>  	__u8 ar;		/* the access register number */
> -	__u8 reserved[31];	/* should be set to 0 */
> +	__u8 reserved21[3];	/* should be set to 0 */
> +	__u32 offset;		/* offset into the sida */
> +	__u8 reserved28[24];	/* should be set to 0 */
>  };
> +
> +
>  /* types for kvm_s390_mem_op->op */
>  #define KVM_S390_MEMOP_LOGICAL_READ	0
>  #define KVM_S390_MEMOP_LOGICAL_WRITE	1
> +#define KVM_S390_MEMOP_SIDA_READ	2
> +#define KVM_S390_MEMOP_SIDA_WRITE	3
>  /* flags for kvm_s390_mem_op->flags */
>  #define KVM_S390_MEMOP_F_CHECK_ONLY		(1ULL << 0)
>  #define KVM_S390_MEMOP_F_INJECT_EXCEPTION	(1ULL << 1)
> @@ -1510,6 +1516,7 @@ struct kvm_pv_cmd {
>  /* Available with KVM_CAP_S390_PROTECTED */
>  #define KVM_S390_PV_COMMAND		_IOW(KVMIO, 0xc5, struct kvm_pv_cmd)
>  #define KVM_S390_PV_COMMAND_VCPU	_IOW(KVMIO, 0xc6, struct kvm_pv_cmd)
> +#define KVM_S390_SIDA_OP		_IOW(KVMIO, 0xc7, struct kvm_s390_mem_op)
>  
>  /* Secure Encrypted Virtualization command */
>  enum sev_cmd_id {
> 

Uh, why the mix of a new ioctl with the existing mem_op stuff? Could you
please either properly integrate this into the MEM_OP ioctl (and e.g.
use gaddr as offset for the new SIDA_READ and SIDA_WRITE subcodes), or
completely separate it for a new ioctl, i.e. introduce a new struct for
the new ioctl instead of recycling the struct kvm_s390_mem_op here?
(and in case you ask me, I'd slightly prefer to integrate everything
into MEM_OP instead of introducing a new ioctl here).

In any case, please also update Documentation/virt/kvm/api.txt with the
new SIDA functions.

 Thanks,
  Thomas

Re: [RFCv2 24/37] KVM: s390: protvirt: Write sthyi data to instruction data area

[Thomas Huth]

On 03/02/2020 14.19, Christian Borntraeger wrote:
> From: Janosch Frank <frankja@linux.ibm.com>
> 
> STHYI data has to go through the bounce buffer.
> 
> Signed-off-by: Janosch Frank <frankja@linux.ibm.com>
> ---
>  arch/s390/kvm/intercept.c | 15 ++++++++++-----
>  1 file changed, 10 insertions(+), 5 deletions(-)
> 
> diff --git a/arch/s390/kvm/intercept.c b/arch/s390/kvm/intercept.c
> index 4b1effa44e41..06d1fa83ef4c 100644
> --- a/arch/s390/kvm/intercept.c
> +++ b/arch/s390/kvm/intercept.c
> @@ -392,7 +392,7 @@ int handle_sthyi(struct kvm_vcpu *vcpu)
>  		goto out;
>  	}
>  
> -	if (addr & ~PAGE_MASK)
> +	if (!kvm_s390_pv_is_protected(vcpu->kvm) && (addr & ~PAGE_MASK))
>  		return kvm_s390_inject_program_int(vcpu, PGM_SPECIFICATION);
>  
>  	sctns = (void *)get_zeroed_page(GFP_KERNEL);
> @@ -403,10 +403,15 @@ int handle_sthyi(struct kvm_vcpu *vcpu)
>  
>  out:
>  	if (!cc) {
> -		r = write_guest(vcpu, addr, reg2, sctns, PAGE_SIZE);
> -		if (r) {
> -			free_page((unsigned long)sctns);
> -			return kvm_s390_inject_prog_cond(vcpu, r);
> +		if (kvm_s390_pv_is_protected(vcpu->kvm)) {
> +			memcpy((void *)(vcpu->arch.sie_block->sidad & PAGE_MASK), sctns,

I think it would look nicer with sida_origin() here.

> +			       PAGE_SIZE);
> +		} else {
> +			r = write_guest(vcpu, addr, reg2, sctns, PAGE_SIZE);
> +			if (r) {
> +				free_page((unsigned long)sctns);
> +				return kvm_s390_inject_prog_cond(vcpu, r);
> +			}
>  		}
>  	}
>  
> 

Reviewed-by: Thomas Huth <thuth@redhat.com>

Re: [RFCv2 15/37] KVM: s390: protvirt: Implement interruption injection

[Cornelia Huck]

On Wed, 5 Feb 2020 12:46:39 +0100
Michael Mueller <mimu@linux.ibm.com> wrote:

> On 05.02.20 12:31, Cornelia Huck wrote:
> > On Mon,  3 Feb 2020 08:19:35 -0500
> > Christian Borntraeger <borntraeger@de.ibm.com> wrote:
> >   
> >> From: Michael Mueller <mimu@linux.ibm.com>
> >>
> >> The patch implements interruption injection for the following
> >> list of interruption types:
> >>
> >>    - I/O
> >>      __deliver_io (III)
> >>
> >>    - External
> >>      __deliver_cpu_timer (IEI)
> >>      __deliver_ckc (IEI)
> >>      __deliver_emergency_signal (IEI)
> >>      __deliver_external_call (IEI)
> >>
> >>    - cpu restart
> >>      __deliver_restart (IRI)  
> > 
> > Hm... what do 'III', 'IEI', and 'IRI' stand for?  
> 
> that's the kind of interruption injection being used:
> 
> inject io interruption
> inject external interruption
> inject restart interruption

So, maybe make this:

- I/O (uses inject io interruption)
  __ deliver_io

- External (uses inject external interruption)
(and so on)

I find using the acronyms without explanation very confusing.

> 
> >   
> >>
> >> The service interrupt is handled in a followup patch.
> >>
> >> Signed-off-by: Michael Mueller <mimu@linux.ibm.com>
> >> Signed-off-by: Christian Borntraeger <borntraeger@de.ibm.com>
> >> [fixes]
> >> ---
> >>   arch/s390/include/asm/kvm_host.h |  8 +++
> >>   arch/s390/kvm/interrupt.c        | 93 ++++++++++++++++++++++----------
> >>   2 files changed, 74 insertions(+), 27 deletions(-)

Re: [RFCv2 25/37] KVM: s390: protvirt: STSI handling

[Thomas Huth]

On 03/02/2020 14.19, Christian Borntraeger wrote:
> From: Janosch Frank <frankja@linux.ibm.com>
> 
> Save response to sidad and disable address checking for protected
> guests.
> 
> Signed-off-by: Janosch Frank <frankja@linux.ibm.com>
> ---
>  arch/s390/kvm/priv.c | 9 +++++++--
>  1 file changed, 7 insertions(+), 2 deletions(-)
> 
> diff --git a/arch/s390/kvm/priv.c b/arch/s390/kvm/priv.c
> index ed52ffa8d5d4..06c7e7a10825 100644
> --- a/arch/s390/kvm/priv.c
> +++ b/arch/s390/kvm/priv.c
> @@ -872,7 +872,7 @@ static int handle_stsi(struct kvm_vcpu *vcpu)
>  
>  	operand2 = kvm_s390_get_base_disp_s(vcpu, &ar);
>  
> -	if (operand2 & 0xfff)
> +	if (!kvm_s390_pv_is_protected(vcpu->kvm) && (operand2 & 0xfff))
>  		return kvm_s390_inject_program_int(vcpu, PGM_SPECIFICATION);
>  
>  	switch (fc) {
> @@ -893,8 +893,13 @@ static int handle_stsi(struct kvm_vcpu *vcpu)
>  		handle_stsi_3_2_2(vcpu, (void *) mem);
>  		break;
>  	}
> +	if (kvm_s390_pv_is_protected(vcpu->kvm)) {
> +		memcpy((void *)vcpu->arch.sie_block->sidad, (void *)mem,

That should use sida_origin() or "...->sidad & PAGE_MASK".

> +		       PAGE_SIZE);
> +		rc = 0;
> +	} else
> +		rc = write_guest(vcpu, operand2, ar, (void *)mem, PAGE_SIZE);

Coding style: If one branch of the if-statement uses curly braces,
please add them to the other branch as well.

> -	rc = write_guest(vcpu, operand2, ar, (void *)mem, PAGE_SIZE);
>  	if (rc) {
>  		rc = kvm_s390_inject_prog_cond(vcpu, rc);
>  		goto out;
> 

 Thomas

Re: [RFCv2 26/37] KVM: s390: protvirt: disallow one_reg

[Thomas Huth]

On 03/02/2020 14.19, Christian Borntraeger wrote:
> From: Janosch Frank <frankja@linux.ibm.com>
> 
> A lot of the registers are controlled by the Ultravisor and never
> visible to KVM. Some fields in the sie control block are overlayed,
> like gbea. As no userspace uses the ONE_REG interface on s390 it is safe
> to disable this for protected guests.
> 
> Signed-off-by: Janosch Frank <frankja@linux.ibm.com>
> ---
>  arch/s390/kvm/kvm-s390.c | 3 +++
>  1 file changed, 3 insertions(+)
> 
> diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
> index 6e74c7afae3a..b9692d722c1e 100644
> --- a/arch/s390/kvm/kvm-s390.c
> +++ b/arch/s390/kvm/kvm-s390.c
> @@ -4641,6 +4641,9 @@ long kvm_arch_vcpu_ioctl(struct file *filp,
>  	case KVM_SET_ONE_REG:
>  	case KVM_GET_ONE_REG: {
>  		struct kvm_one_reg reg;
> +		r = -EINVAL;
> +		if (kvm_s390_pv_is_protected(vcpu->kvm))
> +			break;
>  		r = -EFAULT;
>  		if (copy_from_user(&reg, argp, sizeof(reg)))
>  			break;

Reviewed-by: Thomas Huth <thuth@redhat.com>

PS:
Not sure, but maybe it would be also be good to add a sentence to
Documentation/virt/kvm/api.txt ?

Re: [RFCv2 21/37] KVM: S390: protvirt: Introduce instruction data area bounce buffer

[Janosch Frank]

[-- Attachment #1.1: Type: text/plain, Size: 3037 bytes --]

On 2/5/20 1:02 PM, Thomas Huth wrote:
> On 03/02/2020 14.19, Christian Borntraeger wrote:
>> From: Janosch Frank <frankja@linux.ibm.com>
>>
>> Now that we can't access guest memory anymore, we have a dedicated
>> sattelite block that's a bounce buffer for instruction data.
>>
>> We re-use the memop interface to copy the instruction data to / from
>> userspace. This lets us re-use a lot of QEMU code which used that
>> interface to make logical guest memory accesses which are not possible
>> anymore in protected mode anyway.
>>
>> Signed-off-by: Janosch Frank <frankja@linux.ibm.com>
>> ---
> [...]
>> diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
>> index eab741bc12c3..20969ce12096 100644
>> --- a/include/uapi/linux/kvm.h
>> +++ b/include/uapi/linux/kvm.h
>> @@ -466,7 +466,7 @@ struct kvm_translation {
>>  	__u8  pad[5];
>>  };
>>  
>> -/* for KVM_S390_MEM_OP */
>> +/* for KVM_S390_MEM_OP and KVM_S390_SIDA_OP */
>>  struct kvm_s390_mem_op {
>>  	/* in */
>>  	__u64 gaddr;		/* the guest address */
>> @@ -475,11 +475,17 @@ struct kvm_s390_mem_op {
>>  	__u32 op;		/* type of operation */
>>  	__u64 buf;		/* buffer in userspace */
>>  	__u8 ar;		/* the access register number */
>> -	__u8 reserved[31];	/* should be set to 0 */
>> +	__u8 reserved21[3];	/* should be set to 0 */
>> +	__u32 offset;		/* offset into the sida */
>> +	__u8 reserved28[24];	/* should be set to 0 */
>>  };
>> +
>> +
>>  /* types for kvm_s390_mem_op->op */
>>  #define KVM_S390_MEMOP_LOGICAL_READ	0
>>  #define KVM_S390_MEMOP_LOGICAL_WRITE	1
>> +#define KVM_S390_MEMOP_SIDA_READ	2
>> +#define KVM_S390_MEMOP_SIDA_WRITE	3
>>  /* flags for kvm_s390_mem_op->flags */
>>  #define KVM_S390_MEMOP_F_CHECK_ONLY		(1ULL << 0)
>>  #define KVM_S390_MEMOP_F_INJECT_EXCEPTION	(1ULL << 1)
>> @@ -1510,6 +1516,7 @@ struct kvm_pv_cmd {
>>  /* Available with KVM_CAP_S390_PROTECTED */
>>  #define KVM_S390_PV_COMMAND		_IOW(KVMIO, 0xc5, struct kvm_pv_cmd)
>>  #define KVM_S390_PV_COMMAND_VCPU	_IOW(KVMIO, 0xc6, struct kvm_pv_cmd)
>> +#define KVM_S390_SIDA_OP		_IOW(KVMIO, 0xc7, struct kvm_s390_mem_op)
>>  
>>  /* Secure Encrypted Virtualization command */
>>  enum sev_cmd_id {
>>
> 
> Uh, why the mix of a new ioctl with the existing mem_op stuff? Could you
> please either properly integrate this into the MEM_OP ioctl (and e.g.
> use gaddr as offset for the new SIDA_READ and SIDA_WRITE subcodes), or
> completely separate it for a new ioctl, i.e. introduce a new struct for
> the new ioctl instead of recycling the struct kvm_s390_mem_op here?
> (and in case you ask me, I'd slightly prefer to integrate everything
> into MEM_OP instead of introducing a new ioctl here).

*cough* David and Christian didn't like the memop solution and it took
me a long time to get this to work properly in QEMU...

I wouldn't mind a dedicated struct.

> 
> In any case, please also update Documentation/virt/kvm/api.txt with the
> new SIDA functions.
> 
>  Thanks,
>   Thomas
> 



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [RFCv2 16/37] KVM: s390: protvirt: Add SCLP interrupt handling

[Cornelia Huck]

On Mon,  3 Feb 2020 08:19:36 -0500
Christian Borntraeger <borntraeger@de.ibm.com> wrote:

> The sclp interrupt is kind of special. The ultravisor polices that we
> do not inject and sclp interrupt with payload if no sccb is outstanding.
> On the other hand we have "asynchronous" event interrupts, e.g. for
> console input.
> We separate both variants into sclp interrupt and sclp event interrupt.
> The sclp interrupt is masked until a previous servc instruction has
> finished (sie exit 108).
> 
> Signed-off-by: Christian Borntraeger <borntraeger@de.ibm.com>
> ---
>  arch/s390/include/asm/kvm_host.h |  6 ++-
>  arch/s390/kvm/intercept.c        | 25 +++++++++
>  arch/s390/kvm/interrupt.c        | 92 ++++++++++++++++++++++++++------
>  arch/s390/kvm/kvm-s390.c         |  2 +
>  4 files changed, 108 insertions(+), 17 deletions(-)
> 

> diff --git a/arch/s390/kvm/intercept.c b/arch/s390/kvm/intercept.c
> index 4b3fbbde1674..c22214967214 100644
> --- a/arch/s390/kvm/intercept.c
> +++ b/arch/s390/kvm/intercept.c
> @@ -444,8 +444,33 @@ static int handle_operexc(struct kvm_vcpu *vcpu)
>  	return kvm_s390_inject_program_int(vcpu, PGM_OPERATION);
>  }
>  
> +static int handle_pv_sclp(struct kvm_vcpu *vcpu)
> +{
> +	struct kvm_s390_float_interrupt *fi = &vcpu->kvm->arch.float_int;
> +
> +	spin_lock(&fi->lock);
> +	/*
> +	 * 2 cases:
> +	 * a: an sccb answering interrupt was already pending or in flight.
> +	 *    As the sccb value is not used we can simply set some more bits
> +	 *    and make sure that we deliver something

The sccb value is not used because the firmware handles this?

> +	 * b: an error sccb interrupt needs to be injected so we also inject
> +	 *    something and let firmware do the right thing.
> +	 * This makes sure, that both errors and real sccb returns will only
> +	 * be delivered when we are unmasked.

What makes this sure? That we only unmask here?

> +	 */
> +	fi->srv_signal.ext_params |= 0x43000;
> +	set_bit(IRQ_PEND_EXT_SERVICE, &fi->pending_irqs);
> +	clear_bit(IRQ_PEND_EXT_SERVICE, &fi->masked_irqs);
> +	spin_unlock(&fi->lock);
> +	return 0;
> +}
> +
>  static int handle_pv_not(struct kvm_vcpu *vcpu)
>  {
> +	if (vcpu->arch.sie_block->ipa == 0xb220)
> +		return handle_pv_sclp(vcpu);
> +
>  	return handle_instruction(vcpu);
>  }
>  

Re: [RFCv2 15/37] KVM: s390: protvirt: Implement interruption injection

[Michael Mueller]

On 05.02.20 13:11, Cornelia Huck wrote:
> On Wed, 5 Feb 2020 12:46:39 +0100
> Michael Mueller <mimu@linux.ibm.com> wrote:
> 
>> On 05.02.20 12:31, Cornelia Huck wrote:
>>> On Mon,  3 Feb 2020 08:19:35 -0500
>>> Christian Borntraeger <borntraeger@de.ibm.com> wrote:
>>>    
>>>> From: Michael Mueller <mimu@linux.ibm.com>
>>>>
>>>> The patch implements interruption injection for the following
>>>> list of interruption types:
>>>>
>>>>     - I/O
>>>>       __deliver_io (III)
>>>>
>>>>     - External
>>>>       __deliver_cpu_timer (IEI)
>>>>       __deliver_ckc (IEI)
>>>>       __deliver_emergency_signal (IEI)
>>>>       __deliver_external_call (IEI)
>>>>
>>>>     - cpu restart
>>>>       __deliver_restart (IRI)
>>>
>>> Hm... what do 'III', 'IEI', and 'IRI' stand for?
>>
>> that's the kind of interruption injection being used:
>>
>> inject io interruption
>> inject external interruption
>> inject restart interruption
> 
> So, maybe make this:
> 
> - I/O (uses inject io interruption)
>    __ deliver_io
> 
> - External (uses inject external interruption)
> (and so on)
> 
> I find using the acronyms without explanation very confusing.

Make a guess from where they are coming...

Christian, would you please update the description accordingly.


   - I/O (uses inject io interruption)
     __deliver_io

   - External (uses inject external interruption)
     __deliver_cpu_timer
     __deliver_ckc
     __deliver_emergency_signal
     __deliver_external_call

   - cpu restart (uses inject restart interruption)
     __deliver_restart



thanks

Michael

> 
>>
>>>    
>>>>
>>>> The service interrupt is handled in a followup patch.
>>>>
>>>> Signed-off-by: Michael Mueller <mimu@linux.ibm.com>
>>>> Signed-off-by: Christian Borntraeger <borntraeger@de.ibm.com>
>>>> [fixes]
>>>> ---
>>>>    arch/s390/include/asm/kvm_host.h |  8 +++
>>>>    arch/s390/kvm/interrupt.c        | 93 ++++++++++++++++++++++----------
>>>>    2 files changed, 74 insertions(+), 27 deletions(-)
> 

Re: [RFCv2 18/37] KVM: s390: protvirt: Implement machine-check interruption injection

[Cornelia Huck]

On Mon,  3 Feb 2020 08:19:38 -0500
Christian Borntraeger <borntraeger@de.ibm.com> wrote:

> From: Michael Mueller <mimu@linux.ibm.com>
> 
> Similar to external interrupts, the hypervisor can inject machine
> checks by providing the right data in the interrupt injection controls.

Maybe we should either merge this with the patch introducing the other
interrupt injections, or split that up into io/external/restart. It
seems slightly odd to single out machine checks here.

> 
> Signed-off-by: Michael Mueller <mimu@linux.ibm.com>
> ---
>  arch/s390/kvm/interrupt.c | 8 ++++++++
>  1 file changed, 8 insertions(+)
> 
> diff --git a/arch/s390/kvm/interrupt.c b/arch/s390/kvm/interrupt.c
> index c707725e618b..a98f1dfde8de 100644
> --- a/arch/s390/kvm/interrupt.c
> +++ b/arch/s390/kvm/interrupt.c
> @@ -571,6 +571,14 @@ static int __write_machine_check(struct kvm_vcpu *vcpu,
>  	union mci mci;
>  	int rc;
>  
> +	if (kvm_s390_pv_is_protected(vcpu->kvm)) {
> +		vcpu->arch.sie_block->iictl = IICTL_CODE_MCHK;
> +		vcpu->arch.sie_block->mcic = mchk->mcic;
> +		vcpu->arch.sie_block->faddr = mchk->failing_storage_address;
> +		vcpu->arch.sie_block->edc = mchk->ext_damage_code;

Maybe add a comment that we don't need with other machine-check related data?

> +		return 0;
> +	}
> +
>  	mci.val = mchk->mcic;
>  	/* take care of lazy register loading */
>  	save_fpu_regs();

Anyway,
Reviewed-by: Cornelia Huck <cohuck@redhat.com>

Re: [RFCv2 19/37] KVM: s390: protvirt: Handle spec exception loops

[Cornelia Huck]

On Mon,  3 Feb 2020 08:19:39 -0500
Christian Borntraeger <borntraeger@de.ibm.com> wrote:

> From: Janosch Frank <frankja@linux.ibm.com>
> 
> SIE intercept code 8 is used only on exception loops for protected
> guests. That means we need stop the guest when we see it.

s/need stop/need to stop/

...which is a task done by userspace?

> 
> Signed-off-by: Janosch Frank <frankja@linux.ibm.com>
> Reviewed-by: Thomas Huth <thuth@redhat.com>
> ---
>  arch/s390/kvm/intercept.c | 7 +++++++
>  1 file changed, 7 insertions(+)
> 
> diff --git a/arch/s390/kvm/intercept.c b/arch/s390/kvm/intercept.c
> index c22214967214..d63f9cf10360 100644
> --- a/arch/s390/kvm/intercept.c
> +++ b/arch/s390/kvm/intercept.c
> @@ -231,6 +231,13 @@ static int handle_prog(struct kvm_vcpu *vcpu)
>  
>  	vcpu->stat.exit_program_interruption++;
>  
> +	/*
> +	 * Intercept 8 indicates a loop of specification exceptions
> +	 * for protected guests

s/guests/guests./

> +	 */
> +	if (kvm_s390_pv_is_protected(vcpu->kvm))
> +		return -EOPNOTSUPP;
> +
>  	if (guestdbg_enabled(vcpu) && per_event(vcpu)) {
>  		rc = kvm_s390_handle_per_event(vcpu);
>  		if (rc)

Reviewed-by: Cornelia Huck <cohuck@redhat.com>

Re: [RFCv2 20/37] KVM: s390: protvirt: Add new gprs location handling

[Cornelia Huck]

On Mon,  3 Feb 2020 08:19:40 -0500
Christian Borntraeger <borntraeger@de.ibm.com> wrote:

> From: Janosch Frank <frankja@linux.ibm.com>
> 
> Guest registers for protected guests are stored at offset 0x380.
> 
> Signed-off-by: Janosch Frank <frankja@linux.ibm.com>
> ---
>  arch/s390/include/asm/kvm_host.h |  4 +++-
>  arch/s390/kvm/kvm-s390.c         | 11 +++++++++++
>  2 files changed, 14 insertions(+), 1 deletion(-)
> 

With the things pointed out by Thomas fixed:

Reviewed-by: Cornelia Huck <cohuck@redhat.com>

Re: [RFCv2 24/37] KVM: s390: protvirt: Write sthyi data to instruction data area

[Cornelia Huck]

On Mon,  3 Feb 2020 08:19:44 -0500
Christian Borntraeger <borntraeger@de.ibm.com> wrote:

> From: Janosch Frank <frankja@linux.ibm.com>
> 
> STHYI data has to go through the bounce buffer.
> 
> Signed-off-by: Janosch Frank <frankja@linux.ibm.com>
> ---
>  arch/s390/kvm/intercept.c | 15 ++++++++++-----
>  1 file changed, 10 insertions(+), 5 deletions(-)

Reviewed-by: Cornelia Huck <cohuck@redhat.com>

Re: [RFCv2 26/37] KVM: s390: protvirt: disallow one_reg

[Cornelia Huck]

On Mon,  3 Feb 2020 08:19:46 -0500
Christian Borntraeger <borntraeger@de.ibm.com> wrote:

> From: Janosch Frank <frankja@linux.ibm.com>
> 
> A lot of the registers are controlled by the Ultravisor and never
> visible to KVM. Some fields in the sie control block are overlayed,
> like gbea. As no userspace uses the ONE_REG interface on s390 it is safe
> to disable this for protected guests.

Hm, QEMU seems to fall back to ONE_REG if sync regs are not available,
doesn't it? So maybe it would be better to word this as

"As no known userspace uses the ONE_REG interface on s390 if sync regs
are available, no functionality is lost if it is disabled for protected
guests."

?

> 
> Signed-off-by: Janosch Frank <frankja@linux.ibm.com>
> ---
>  arch/s390/kvm/kvm-s390.c | 3 +++
>  1 file changed, 3 insertions(+)
> 
> diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
> index 6e74c7afae3a..b9692d722c1e 100644
> --- a/arch/s390/kvm/kvm-s390.c
> +++ b/arch/s390/kvm/kvm-s390.c
> @@ -4641,6 +4641,9 @@ long kvm_arch_vcpu_ioctl(struct file *filp,
>  	case KVM_SET_ONE_REG:
>  	case KVM_GET_ONE_REG: {
>  		struct kvm_one_reg reg;
> +		r = -EINVAL;
> +		if (kvm_s390_pv_is_protected(vcpu->kvm))
> +			break;
>  		r = -EFAULT;
>  		if (copy_from_user(&reg, argp, sizeof(reg)))
>  			break;

Reviewed-by: Cornelia Huck <cohuck@redhat.com>

Re: [RFCv2 15/37] KVM: s390: protvirt: Implement interruption injection

[Michael Mueller]

On 05.02.20 12:31, Cornelia Huck wrote:
> On Mon,  3 Feb 2020 08:19:35 -0500
> Christian Borntraeger <borntraeger@de.ibm.com> wrote:
> 
>> From: Michael Mueller <mimu@linux.ibm.com>
>>
>> The patch implements interruption injection for the following
>> list of interruption types:
>>
>>    - I/O
>>      __deliver_io (III)
>>
>>    - External
>>      __deliver_cpu_timer (IEI)
>>      __deliver_ckc (IEI)
>>      __deliver_emergency_signal (IEI)
>>      __deliver_external_call (IEI)
>>
>>    - cpu restart
>>      __deliver_restart (IRI)
> 
> Hm... what do 'III', 'IEI', and 'IRI' stand for?
> 
>>
>> The service interrupt is handled in a followup patch.
>>
>> Signed-off-by: Michael Mueller <mimu@linux.ibm.com>
>> Signed-off-by: Christian Borntraeger <borntraeger@de.ibm.com>
>> [fixes]
>> ---
>>   arch/s390/include/asm/kvm_host.h |  8 +++
>>   arch/s390/kvm/interrupt.c        | 93 ++++++++++++++++++++++----------
>>   2 files changed, 74 insertions(+), 27 deletions(-)
>>
>> diff --git a/arch/s390/include/asm/kvm_host.h b/arch/s390/include/asm/kvm_host.h
>> index a45d10d87a8a..989cea7a5591 100644
>> --- a/arch/s390/include/asm/kvm_host.h
>> +++ b/arch/s390/include/asm/kvm_host.h
>> @@ -563,6 +563,14 @@ enum irq_types {
>>   #define IRQ_PEND_MCHK_MASK ((1UL << IRQ_PEND_MCHK_REP) | \
>>   			    (1UL << IRQ_PEND_MCHK_EX))
>>   
>> +#define IRQ_PEND_MCHK_REP_MASK (1UL << IRQ_PEND_MCHK_REP)
>> +
>> +#define IRQ_PEND_EXT_II_MASK ((1UL << IRQ_PEND_EXT_CPU_TIMER)  | \
> 
> What does 'II' stand for? Interrupt Injection?
> 
>> +			      (1UL << IRQ_PEND_EXT_CLOCK_COMP) | \
>> +			      (1UL << IRQ_PEND_EXT_EMERGENCY)  | \
>> +			      (1UL << IRQ_PEND_EXT_EXTERNAL)   | \
>> +			      (1UL << IRQ_PEND_EXT_SERVICE))
>> +
>>   struct kvm_s390_interrupt_info {
>>   	struct list_head list;
>>   	u64	type;
> 
> (...)
> 
>> @@ -1834,7 +1872,8 @@ static void __floating_irq_kick(struct kvm *kvm, u64 type)
>>   		break;
>>   	case KVM_S390_INT_IO_MIN...KVM_S390_INT_IO_MAX:
>>   		if (!(type & KVM_S390_INT_IO_AI_MASK &&
>> -		      kvm->arch.gisa_int.origin))
>> +		      kvm->arch.gisa_int.origin) ||
>> +		      kvm_s390_pv_handle_cpu(dst_vcpu))
>>   			kvm_s390_set_cpuflags(dst_vcpu, CPUSTAT_IO_INT);
>>   		break;
>>   	default:
> 
> Looking at this... can you also talk about protected virt vs. exitless
> interrupts?
> 

Adapter interruptions are placed into the GISA but currently
not proceessed in SIE context when running in PV context.
Thus they get delivered by KVM explicitly by means of the
io interruption injection code.

...
00 01580915540:999281 4 - 04 00000000eb351b8a  inject: I/O (AI/gisa) isc 3
00 01580915540:999287 4 - 02 000000008345868d 
00[0706c00180000000-00000000f4d2c544]: deliver: I/O (AI/gisa) isc 3
00 01580915542:999283 4 - 04 00000000eb351b8a  inject: I/O (AI/gisa) isc 3
00 01580915542:999288 4 - 02 000000008345868d 
00[0706c00180000000-00000000f4d2c544]: deliver: I/O (AI/gisa) isc 3
00 01580915543:996622 4 - 04 00000000eb351b8a  inject: I/O (AI/gisa) isc 3
00 01580915543:996629 4 - 02 000000008345868d 
00[0706c00180000000-00000000f4d2c544]: deliver: I/O (AI/gisa) isc 3
00 01580915544:039280 4 - 04 00000000eb351b8a  inject: I/O (AI/gisa) isc 3
00 01580915544:039290 4 - 02 000000008345868d 
00[0706c00180000000-00000000f4d2c544]: deliver: I/O (AI/gisa) isc 3
00 01580915544:039313 4 - 04 00000000eb351b8a  inject: I/O (AI/gisa) isc 3
00 01580915544:039319 4 - 02 000000008345868d 
00[0706c00180000000-00000000f4d2c544]: deliver: I/O (AI/gisa) isc 3
...


Michael

Re: [RFCv2 27/37] KVM: s390: protvirt: Only sync fmt4 registers

[Cornelia Huck]

On Mon,  3 Feb 2020 08:19:47 -0500
Christian Borntraeger <borntraeger@de.ibm.com> wrote:

> From: Janosch Frank <frankja@linux.ibm.com>
> 
> A lot of the registers are controlled by the Ultravisor and never
> visible to KVM. Also some registers are overlayed, like gbea is with
> sidad, which might leak data to userspace.
> 
> Hence we sync a minimal set of registers for both SIE formats and then
> check and sync format 2 registers if necessary.
> 
> Also we disable set/get one reg for the same reason. It's an old
> interface anyway.
> 
> Signed-off-by: Janosch Frank <frankja@linux.ibm.com>
> Signed-off-by: Christian Borntraeger <borntraeger@de.ibm.com>
> [Fixes and patch splitting]
> ---
>  arch/s390/kvm/kvm-s390.c | 116 ++++++++++++++++++++++++---------------
>  1 file changed, 72 insertions(+), 44 deletions(-)
> 
> diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
> index b9692d722c1e..00a0ce4a3d35 100644
> --- a/arch/s390/kvm/kvm-s390.c
> +++ b/arch/s390/kvm/kvm-s390.c
> @@ -3444,9 +3444,11 @@ static void kvm_arch_vcpu_ioctl_initial_reset(struct kvm_vcpu *vcpu)
>  	vcpu->arch.sie_block->gcr[0] = CR0_INITIAL_MASK;
>  	vcpu->arch.sie_block->gcr[14] = CR14_INITIAL_MASK;
>  	vcpu->run->s.regs.fpc = 0;
> -	vcpu->arch.sie_block->gbea = 1;
> -	vcpu->arch.sie_block->pp = 0;
> -	vcpu->arch.sie_block->fpf &= ~FPF_BPBC;
> +	if (!kvm_s390_pv_handle_cpu(vcpu)) {
> +		vcpu->arch.sie_block->gbea = 1;
> +		vcpu->arch.sie_block->pp = 0;
> +		vcpu->arch.sie_block->fpf &= ~FPF_BPBC;

What about e.g. gcr[]? Is it something that just does not matter, while
these conflict somehow?

> +	}
>  }
>  
>  static void kvm_arch_vcpu_ioctl_clear_reset(struct kvm_vcpu *vcpu)
> @@ -4057,25 +4059,16 @@ static int __vcpu_run(struct kvm_vcpu *vcpu)
>  	return rc;
>  }
>  
> -static void sync_regs(struct kvm_vcpu *vcpu, struct kvm_run *kvm_run)
> +static void sync_regs_fmt2(struct kvm_vcpu *vcpu, struct kvm_run *kvm_run)
>  {
>  	struct runtime_instr_cb *riccb;
>  	struct gs_cb *gscb;
>  
> -	riccb = (struct runtime_instr_cb *) &kvm_run->s.regs.riccb;
> -	gscb = (struct gs_cb *) &kvm_run->s.regs.gscb;
>  	vcpu->arch.sie_block->gpsw.mask = kvm_run->psw_mask;
>  	vcpu->arch.sie_block->gpsw.addr = kvm_run->psw_addr;
> -	if (kvm_run->kvm_dirty_regs & KVM_SYNC_PREFIX)
> -		kvm_s390_set_prefix(vcpu, kvm_run->s.regs.prefix);
> -	if (kvm_run->kvm_dirty_regs & KVM_SYNC_CRS) {
> -		memcpy(&vcpu->arch.sie_block->gcr, &kvm_run->s.regs.crs, 128);
> -		/* some control register changes require a tlb flush */
> -		kvm_make_request(KVM_REQ_TLB_FLUSH, vcpu);
> -	}
> +	riccb = (struct runtime_instr_cb *) &kvm_run->s.regs.riccb;
> +	gscb = (struct gs_cb *) &kvm_run->s.regs.gscb;
>  	if (kvm_run->kvm_dirty_regs & KVM_SYNC_ARCH0) {
> -		kvm_s390_set_cpu_timer(vcpu, kvm_run->s.regs.cputm);
> -		vcpu->arch.sie_block->ckc = kvm_run->s.regs.ckc;
>  		vcpu->arch.sie_block->todpr = kvm_run->s.regs.todpr;
>  		vcpu->arch.sie_block->pp = kvm_run->s.regs.pp;
>  		vcpu->arch.sie_block->gbea = kvm_run->s.regs.gbea;
> @@ -4116,6 +4109,47 @@ static void sync_regs(struct kvm_vcpu *vcpu, struct kvm_run *kvm_run)
>  		vcpu->arch.sie_block->fpf &= ~FPF_BPBC;
>  		vcpu->arch.sie_block->fpf |= kvm_run->s.regs.bpbc ? FPF_BPBC : 0;
>  	}
> +	if (MACHINE_HAS_GS) {
> +		preempt_disable();
> +		__ctl_set_bit(2, 4);
> +		if (current->thread.gs_cb) {
> +			vcpu->arch.host_gscb = current->thread.gs_cb;
> +			save_gs_cb(vcpu->arch.host_gscb);
> +		}
> +		if (vcpu->arch.gs_enabled) {
> +			current->thread.gs_cb = (struct gs_cb *)
> +						&vcpu->run->s.regs.gscb;
> +			restore_gs_cb(current->thread.gs_cb);
> +		}
> +		preempt_enable();
> +	}
> +	/* SIE will load etoken directly from SDNX and therefore kvm_run */
> +}
> +
> +static void sync_regs(struct kvm_vcpu *vcpu, struct kvm_run *kvm_run)
> +{
> +	/*
> +	 * at several places we have to modify our internal view to not do
> +	 * things that are disallowed by the ultravisor. For example we must

But we are still free to do them for non-protected guests, right?

> +	 * not inject interrupts after specific exits (e.g. 112). We do this

Spell out what 112 is? Emergency call? ;)

> +	 * by turning off the MIE bits of our PSW copy. To avoid getting

What is MIE? The bits controlling machine check, I/O, external
interrupts?

> +	 * validity intercepts, we do only accept the condition code from
> +	 * userspace.
> +	 */
> +	vcpu->arch.sie_block->gpsw.mask &= ~PSW_MASK_CC;
> +	vcpu->arch.sie_block->gpsw.mask |= kvm_run->psw_mask & PSW_MASK_CC;
> +
> +	if (kvm_run->kvm_dirty_regs & KVM_SYNC_PREFIX)
> +		kvm_s390_set_prefix(vcpu, kvm_run->s.regs.prefix);
> +	if (kvm_run->kvm_dirty_regs & KVM_SYNC_CRS) {
> +		memcpy(&vcpu->arch.sie_block->gcr, &kvm_run->s.regs.crs, 128);
> +		/* some control register changes require a tlb flush */
> +		kvm_make_request(KVM_REQ_TLB_FLUSH, vcpu);
> +	}
> +	if (kvm_run->kvm_dirty_regs & KVM_SYNC_ARCH0) {
> +		kvm_s390_set_cpu_timer(vcpu, kvm_run->s.regs.cputm);
> +		vcpu->arch.sie_block->ckc = kvm_run->s.regs.ckc;
> +	}
>  	save_access_regs(vcpu->arch.host_acrs);
>  	restore_access_regs(vcpu->run->s.regs.acrs);
>  	/* save host (userspace) fprs/vrs */

Diff reordering makes this a bit hard to review, but it seems
reasonable at a glance.

Re: [RFCv2 28/37] KVM: s390: protvirt: Add program exception injection

[Cornelia Huck]

On Mon,  3 Feb 2020 08:19:48 -0500
Christian Borntraeger <borntraeger@de.ibm.com> wrote:

> From: Janosch Frank <frankja@linux.ibm.com>
> 
> Only two program exceptions can be injected for a protected guest:
> specification and operand

s/operand/operand./

> 
> Both have a code in offset 248 of the state description, as the lowcore

"For both, a code needs to be specified at offset 248 (iictl) of the
state description," ?

> is not accessible by KVM for such guests.

s/by/to/

> 
> Signed-off-by: Janosch Frank <frankja@linux.ibm.com>
> ---
>  arch/s390/kvm/interrupt.c | 18 ++++++++++++++++++
>  1 file changed, 18 insertions(+)

Reviewed-by: Cornelia Huck <cohuck@redhat.com>

Re: [RFCv2 21/37] KVM: S390: protvirt: Introduce instruction data area bounce buffer

[Thomas Huth]

[-- Attachment #1.1: Type: text/plain, Size: 3222 bytes --]

On 05/02/2020 13.16, Janosch Frank wrote:
> On 2/5/20 1:02 PM, Thomas Huth wrote:
>> On 03/02/2020 14.19, Christian Borntraeger wrote:
>>> From: Janosch Frank <frankja@linux.ibm.com>
>>>
>>> Now that we can't access guest memory anymore, we have a dedicated
>>> sattelite block that's a bounce buffer for instruction data.
>>>
>>> We re-use the memop interface to copy the instruction data to / from
>>> userspace. This lets us re-use a lot of QEMU code which used that
>>> interface to make logical guest memory accesses which are not possible
>>> anymore in protected mode anyway.
>>>
>>> Signed-off-by: Janosch Frank <frankja@linux.ibm.com>
>>> ---
>> [...]
>>> diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
>>> index eab741bc12c3..20969ce12096 100644
>>> --- a/include/uapi/linux/kvm.h
>>> +++ b/include/uapi/linux/kvm.h
>>> @@ -466,7 +466,7 @@ struct kvm_translation {
>>>  	__u8  pad[5];
>>>  };
>>>  
>>> -/* for KVM_S390_MEM_OP */
>>> +/* for KVM_S390_MEM_OP and KVM_S390_SIDA_OP */
>>>  struct kvm_s390_mem_op {
>>>  	/* in */
>>>  	__u64 gaddr;		/* the guest address */
>>> @@ -475,11 +475,17 @@ struct kvm_s390_mem_op {
>>>  	__u32 op;		/* type of operation */
>>>  	__u64 buf;		/* buffer in userspace */
>>>  	__u8 ar;		/* the access register number */
>>> -	__u8 reserved[31];	/* should be set to 0 */
>>> +	__u8 reserved21[3];	/* should be set to 0 */
>>> +	__u32 offset;		/* offset into the sida */
>>> +	__u8 reserved28[24];	/* should be set to 0 */
>>>  };
>>> +
>>> +
>>>  /* types for kvm_s390_mem_op->op */
>>>  #define KVM_S390_MEMOP_LOGICAL_READ	0
>>>  #define KVM_S390_MEMOP_LOGICAL_WRITE	1
>>> +#define KVM_S390_MEMOP_SIDA_READ	2
>>> +#define KVM_S390_MEMOP_SIDA_WRITE	3
>>>  /* flags for kvm_s390_mem_op->flags */
>>>  #define KVM_S390_MEMOP_F_CHECK_ONLY		(1ULL << 0)
>>>  #define KVM_S390_MEMOP_F_INJECT_EXCEPTION	(1ULL << 1)
>>> @@ -1510,6 +1516,7 @@ struct kvm_pv_cmd {
>>>  /* Available with KVM_CAP_S390_PROTECTED */
>>>  #define KVM_S390_PV_COMMAND		_IOW(KVMIO, 0xc5, struct kvm_pv_cmd)
>>>  #define KVM_S390_PV_COMMAND_VCPU	_IOW(KVMIO, 0xc6, struct kvm_pv_cmd)
>>> +#define KVM_S390_SIDA_OP		_IOW(KVMIO, 0xc7, struct kvm_s390_mem_op)
>>>  
>>>  /* Secure Encrypted Virtualization command */
>>>  enum sev_cmd_id {
>>>
>>
>> Uh, why the mix of a new ioctl with the existing mem_op stuff? Could you
>> please either properly integrate this into the MEM_OP ioctl (and e.g.
>> use gaddr as offset for the new SIDA_READ and SIDA_WRITE subcodes), or
>> completely separate it for a new ioctl, i.e. introduce a new struct for
>> the new ioctl instead of recycling the struct kvm_s390_mem_op here?
>> (and in case you ask me, I'd slightly prefer to integrate everything
>> into MEM_OP instead of introducing a new ioctl here).
> 
> *cough* David and Christian didn't like the memop solution and it took
> me a long time to get this to work properly in QEMU...

I also don't like to re-use MEMOP_LOGICAL_READ and MEMOP_LOGICAL_WRITE
for the SIDA like you've had it in RFC v1 ... but what's wrong with
using KVM_S390_MEMOP_SIDA_READ and KVM_S390_MEMOP_SIDA_WRITE with the
MEM_OP ioctl directly?

 Thomas


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [RFCv2 15/37] KVM: s390: protvirt: Implement interruption injection

[Christian Borntraeger]

On 05.02.20 12:25, Cornelia Huck wrote:
> On Wed, 5 Feb 2020 11:48:42 +0100
> Christian Borntraeger <borntraeger@de.ibm.com> wrote:
> 
>> On 05.02.20 10:51, Thomas Huth wrote:
>>> On 03/02/2020 14.19, Christian Borntraeger wrote:  
>>>> From: Michael Mueller <mimu@linux.ibm.com>
>>>>
>>>> The patch implements interruption injection for the following
>>>> list of interruption types:
>>>>
>>>>   - I/O
>>>>     __deliver_io (III)
>>>>
>>>>   - External
>>>>     __deliver_cpu_timer (IEI)
>>>>     __deliver_ckc (IEI)
>>>>     __deliver_emergency_signal (IEI)
>>>>     __deliver_external_call (IEI)
>>>>
>>>>   - cpu restart
>>>>     __deliver_restart (IRI)
>>>>
>>>> The service interrupt is handled in a followup patch.
>>>>
>>>> Signed-off-by: Michael Mueller <mimu@linux.ibm.com>
>>>> Signed-off-by: Christian Borntraeger <borntraeger@de.ibm.com>
>>>> [fixes]
>>>> ---
>>>>  arch/s390/include/asm/kvm_host.h |  8 +++
>>>>  arch/s390/kvm/interrupt.c        | 93 ++++++++++++++++++++++----------
>>>>  2 files changed, 74 insertions(+), 27 deletions(-)
>>>>
>>>> diff --git a/arch/s390/include/asm/kvm_host.h b/arch/s390/include/asm/kvm_host.h
>>>> index a45d10d87a8a..989cea7a5591 100644
>>>> --- a/arch/s390/include/asm/kvm_host.h
>>>> +++ b/arch/s390/include/asm/kvm_host.h
>>>> @@ -563,6 +563,14 @@ enum irq_types {
>>>>  #define IRQ_PEND_MCHK_MASK ((1UL << IRQ_PEND_MCHK_REP) | \
>>>>  			    (1UL << IRQ_PEND_MCHK_EX))
>>>>  
>>>> +#define IRQ_PEND_MCHK_REP_MASK (1UL << IRQ_PEND_MCHK_REP)
>>>> +
>>>> +#define IRQ_PEND_EXT_II_MASK ((1UL << IRQ_PEND_EXT_CPU_TIMER)  | \
>>>> +			      (1UL << IRQ_PEND_EXT_CLOCK_COMP) | \
>>>> +			      (1UL << IRQ_PEND_EXT_EMERGENCY)  | \
>>>> +			      (1UL << IRQ_PEND_EXT_EXTERNAL)   | \
>>>> +			      (1UL << IRQ_PEND_EXT_SERVICE))
>>>> +
>>>>  struct kvm_s390_interrupt_info {
>>>>  	struct list_head list;
>>>>  	u64	type;
>>>> diff --git a/arch/s390/kvm/interrupt.c b/arch/s390/kvm/interrupt.c
>>>> index c06c89d370a7..ecdec6960a60 100644
>>>> --- a/arch/s390/kvm/interrupt.c
>>>> +++ b/arch/s390/kvm/interrupt.c
>>>> @@ -387,6 +387,12 @@ static unsigned long deliverable_irqs(struct kvm_vcpu *vcpu)
>>>>  		__clear_bit(IRQ_PEND_EXT_SERVICE, &active_mask);
>>>>  	if (psw_mchk_disabled(vcpu))
>>>>  		active_mask &= ~IRQ_PEND_MCHK_MASK;
>>>> +	/* PV guest cpus can have a single interruption injected at a time. */
>>>> +	if (kvm_s390_pv_is_protected(vcpu->kvm) &&
>>>> +	    vcpu->arch.sie_block->iictl != IICTL_CODE_NONE)
>>>> +		active_mask &= ~(IRQ_PEND_EXT_II_MASK |
>>>> +				 IRQ_PEND_IO_MASK |
>>>> +				 IRQ_PEND_MCHK_REP_MASK);  
>>>
>>> I don't quite understand why there is a difference between
>>> IRQ_PEND_MCHK_REP and IRQ_PEND_MCHK_EX here? Why not simply use
>>> IRQ_PEND_MCHK_MASK here? Could you elaborate? (and maybe add a sentence
>>> to the patch description)  
>>
>> I added that part. 
>> My idea was that an exigent machine check would be kind of fatal that it can override
>> the previous interrupt. Now we do not implement the override (kill the previous interrupt)
>> so I agree, maybe lets use IRQ_PEND_MCHK_MASK
> 
> This makes me wonder about interrupt priorities in general. Under which
> circumstances can we have an interrupt with a lower priority already
> injected (but not delivered) in the injection area when a
> higher-priority interrupt comes along? I'm a bit confused here.

This is not an issue. If the interrupt is already in the injection area, then you can consider
that as delivered. And even highest priority interrupt cannot overtake an already delivered one.

Re: [RFCv2 15/37] KVM: s390: protvirt: Implement interruption injection

[Christian Borntraeger]

On 05.02.20 10:51, Thomas Huth wrote:
> On 03/02/2020 14.19, Christian Borntraeger wrote:
>> From: Michael Mueller <mimu@linux.ibm.com>
>>
>> The patch implements interruption injection for the following
>> list of interruption types:
>>
>>   - I/O
>>     __deliver_io (III)
>>
>>   - External
>>     __deliver_cpu_timer (IEI)
>>     __deliver_ckc (IEI)
>>     __deliver_emergency_signal (IEI)
>>     __deliver_external_call (IEI)
>>
>>   - cpu restart
>>     __deliver_restart (IRI)
>>
>> The service interrupt is handled in a followup patch.
>>
>> Signed-off-by: Michael Mueller <mimu@linux.ibm.com>
>> Signed-off-by: Christian Borntraeger <borntraeger@de.ibm.com>
>> [fixes]
>> ---
>>  arch/s390/include/asm/kvm_host.h |  8 +++
>>  arch/s390/kvm/interrupt.c        | 93 ++++++++++++++++++++++----------
>>  2 files changed, 74 insertions(+), 27 deletions(-)
>>
>> diff --git a/arch/s390/include/asm/kvm_host.h b/arch/s390/include/asm/kvm_host.h
>> index a45d10d87a8a..989cea7a5591 100644
>> --- a/arch/s390/include/asm/kvm_host.h
>> +++ b/arch/s390/include/asm/kvm_host.h
>> @@ -563,6 +563,14 @@ enum irq_types {
>>  #define IRQ_PEND_MCHK_MASK ((1UL << IRQ_PEND_MCHK_REP) | \
>>  			    (1UL << IRQ_PEND_MCHK_EX))
>>  
>> +#define IRQ_PEND_MCHK_REP_MASK (1UL << IRQ_PEND_MCHK_REP)
>> +
>> +#define IRQ_PEND_EXT_II_MASK ((1UL << IRQ_PEND_EXT_CPU_TIMER)  | \
>> +			      (1UL << IRQ_PEND_EXT_CLOCK_COMP) | \
>> +			      (1UL << IRQ_PEND_EXT_EMERGENCY)  | \
>> +			      (1UL << IRQ_PEND_EXT_EXTERNAL)   | \
>> +			      (1UL << IRQ_PEND_EXT_SERVICE))
>> +
>>  struct kvm_s390_interrupt_info {
>>  	struct list_head list;
>>  	u64	type;
>> diff --git a/arch/s390/kvm/interrupt.c b/arch/s390/kvm/interrupt.c
>> index c06c89d370a7..ecdec6960a60 100644
>> --- a/arch/s390/kvm/interrupt.c
>> +++ b/arch/s390/kvm/interrupt.c
>> @@ -387,6 +387,12 @@ static unsigned long deliverable_irqs(struct kvm_vcpu *vcpu)
>>  		__clear_bit(IRQ_PEND_EXT_SERVICE, &active_mask);
>>  	if (psw_mchk_disabled(vcpu))
>>  		active_mask &= ~IRQ_PEND_MCHK_MASK;
>> +	/* PV guest cpus can have a single interruption injected at a time. */
>> +	if (kvm_s390_pv_is_protected(vcpu->kvm) &&
>> +	    vcpu->arch.sie_block->iictl != IICTL_CODE_NONE)
>> +		active_mask &= ~(IRQ_PEND_EXT_II_MASK |
>> +				 IRQ_PEND_IO_MASK |
>> +				 IRQ_PEND_MCHK_REP_MASK);
> 
> I don't quite understand why there is a difference between
> IRQ_PEND_MCHK_REP and IRQ_PEND_MCHK_EX here? Why not simply use
> IRQ_PEND_MCHK_MASK here? Could you elaborate? (and maybe add a sentence
> to the patch description)

I changed to IRQ_PEND_MCHK_MASK by just considering the existing interrupt as delivered.

> 
>> @@ -533,7 +547,6 @@ static int __must_check __deliver_pfault_init(struct kvm_vcpu *vcpu)
>>  	trace_kvm_s390_deliver_interrupt(vcpu->vcpu_id,
>>  					 KVM_S390_INT_PFAULT_INIT,
>>  					 0, ext.ext_params2);
>> -
> 
> Nit: Unnecessary white space change.

ack

Re: [RFCv2 28/37] KVM: s390: protvirt: Add program exception injection

[Thomas Huth]

On 03/02/2020 14.19, Christian Borntraeger wrote:
> From: Janosch Frank <frankja@linux.ibm.com>
> 
> Only two program exceptions can be injected for a protected guest:
> specification and operand
> 
> Both have a code in offset 248 of the state description, as the lowcore
> is not accessible by KVM for such guests.

What do you mean with offset 248 here? You only touch the iictl field
here, and looking at patch 14/37, iictl is at offset 0x55 ...?

 Thomas

Re: [RFCv2 15/37] KVM: s390: protvirt: Implement interruption injection

[Christian Borntraeger]

On 05.02.20 13:26, Michael Mueller wrote:
> 
> 
> On 05.02.20 13:11, Cornelia Huck wrote:
>> On Wed, 5 Feb 2020 12:46:39 +0100
>> Michael Mueller <mimu@linux.ibm.com> wrote:
>>
>>> On 05.02.20 12:31, Cornelia Huck wrote:
>>>> On Mon,  3 Feb 2020 08:19:35 -0500
>>>> Christian Borntraeger <borntraeger@de.ibm.com> wrote:
>>>>   
>>>>> From: Michael Mueller <mimu@linux.ibm.com>
>>>>>
>>>>> The patch implements interruption injection for the following
>>>>> list of interruption types:
>>>>>
>>>>>     - I/O
>>>>>       __deliver_io (III)
>>>>>
>>>>>     - External
>>>>>       __deliver_cpu_timer (IEI)
>>>>>       __deliver_ckc (IEI)
>>>>>       __deliver_emergency_signal (IEI)
>>>>>       __deliver_external_call (IEI)
>>>>>
>>>>>     - cpu restart
>>>>>       __deliver_restart (IRI)
>>>>
>>>> Hm... what do 'III', 'IEI', and 'IRI' stand for?
>>>
>>> that's the kind of interruption injection being used:
>>>
>>> inject io interruption
>>> inject external interruption
>>> inject restart interruption
>>
>> So, maybe make this:
>>
>> - I/O (uses inject io interruption)
>>    __ deliver_io
>>
>> - External (uses inject external interruption)
>> (and so on)
>>
>> I find using the acronyms without explanation very confusing.
> 
> Make a guess from where they are coming...
> 
> Christian, would you please update the description accordingly.
> 
> 
>   - I/O (uses inject io interruption)
>     __deliver_io
> 
>   - External (uses inject external interruption)
>     __deliver_cpu_timer
>     __deliver_ckc
>     __deliver_emergency_signal
>     __deliver_external_call
> 
>   - cpu restart (uses inject restart interruption)
>     __deliver_restart

Will use that and also add.
Please note that posted interrupts (GISA) are not used for protected guests as of today.

> 
> thanks
> 
> Michael
> 
>>
>>>
>>>>   
>>>>>
>>>>> The service interrupt is handled in a followup patch.
>>>>>
>>>>> Signed-off-by: Michael Mueller <mimu@linux.ibm.com>
>>>>> Signed-off-by: Christian Borntraeger <borntraeger@de.ibm.com>
>>>>> [fixes]
>>>>> ---
>>>>>    arch/s390/include/asm/kvm_host.h |  8 +++
>>>>>    arch/s390/kvm/interrupt.c        | 93 ++++++++++++++++++++++----------
>>>>>    2 files changed, 74 insertions(+), 27 deletions(-)
>>
> 

Re: [RFCv2 16/37] KVM: s390: protvirt: Add SCLP interrupt handling

[Christian Borntraeger]

On 05.02.20 13:22, Cornelia Huck wrote:
> On Mon,  3 Feb 2020 08:19:36 -0500
> Christian Borntraeger <borntraeger@de.ibm.com> wrote:
> 
>> The sclp interrupt is kind of special. The ultravisor polices that we
>> do not inject and sclp interrupt with payload if no sccb is outstanding.
>> On the other hand we have "asynchronous" event interrupts, e.g. for
>> console input.
>> We separate both variants into sclp interrupt and sclp event interrupt.
>> The sclp interrupt is masked until a previous servc instruction has
>> finished (sie exit 108).
>>
>> Signed-off-by: Christian Borntraeger <borntraeger@de.ibm.com>
>> ---
>>  arch/s390/include/asm/kvm_host.h |  6 ++-
>>  arch/s390/kvm/intercept.c        | 25 +++++++++
>>  arch/s390/kvm/interrupt.c        | 92 ++++++++++++++++++++++++++------
>>  arch/s390/kvm/kvm-s390.c         |  2 +
>>  4 files changed, 108 insertions(+), 17 deletions(-)
>>
> 
>> diff --git a/arch/s390/kvm/intercept.c b/arch/s390/kvm/intercept.c
>> index 4b3fbbde1674..c22214967214 100644
>> --- a/arch/s390/kvm/intercept.c
>> +++ b/arch/s390/kvm/intercept.c
>> @@ -444,8 +444,33 @@ static int handle_operexc(struct kvm_vcpu *vcpu)
>>  	return kvm_s390_inject_program_int(vcpu, PGM_OPERATION);
>>  }
>>  
>> +static int handle_pv_sclp(struct kvm_vcpu *vcpu)
>> +{
>> +	struct kvm_s390_float_interrupt *fi = &vcpu->kvm->arch.float_int;
>> +
>> +	spin_lock(&fi->lock);
>> +	/*
>> +	 * 2 cases:
>> +	 * a: an sccb answering interrupt was already pending or in flight.
>> +	 *    As the sccb value is not used we can simply set some more bits
>> +	 *    and make sure that we deliver something
> 
> The sccb value is not used because the firmware handles this?

Yes, the firmware will use its copy of the sccb value when we have something !=0
> 
>> +	 * b: an error sccb interrupt needs to be injected so we also inject
>> +	 *    something and let firmware do the right thing.
>> +	 * This makes sure, that both errors and real sccb returns will only
>> +	 * be delivered when we are unmasked.
> 
> What makes this sure? That we only unmask here? 
Yes, we unmask here at a notification intercept (instruction is complete)..

I will use:

        /*
         * 2 cases:
         * a: an sccb answering interrupt was already pending or in flight.
         *    As the sccb value is not known we can simply set some value to
         *    trigger delivery of a saved SCCB. UV will then use its saved
         *    copy of the SCCB value.
         * b: an error SCCB interrupt needs to be injected so we also inject
         *    a fake SCCB address. Firmware will use the proper one.
         * This makes sure, that both errors and real sccb returns will only
         * be delivered after a notification intercept (instruction has
         * finished) but not after others.
         */

Re: [RFCv2 18/37] KVM: s390: protvirt: Implement machine-check interruption injection

[Christian Borntraeger]

On 05.02.20 14:47, Cornelia Huck wrote:
[..]
>> --- a/arch/s390/kvm/interrupt.c
>> +++ b/arch/s390/kvm/interrupt.c
>> @@ -571,6 +571,14 @@ static int __write_machine_check(struct kvm_vcpu *vcpu,
>>  	union mci mci;
>>  	int rc;
>>  
>> +	if (kvm_s390_pv_is_protected(vcpu->kvm)) {
>> +		vcpu->arch.sie_block->iictl = IICTL_CODE_MCHK;
>> +		vcpu->arch.sie_block->mcic = mchk->mcic;
>> +		vcpu->arch.sie_block->faddr = mchk->failing_storage_address;
>> +		vcpu->arch.sie_block->edc = mchk->ext_damage_code;
> 
> Maybe add a comment that we don't need with other machine-check related data?

Not sure I get this point. Can you make a proposal?

> 
>> +		return 0;
>> +	}
>> +
>>  	mci.val = mchk->mcic;
>>  	/* take care of lazy register loading */
>>  	save_fpu_regs();
> 
> Anyway,
> Reviewed-by: Cornelia Huck <cohuck@redhat.com>
> 

Re: [RFCv2 30/37] KVM: s390: protvirt: Add diag 308 subcode 8 - 10 handling

[Thomas Huth]

On 03/02/2020 14.19, Christian Borntraeger wrote:
> From: Janosch Frank <frankja@linux.ibm.com>
> 
> If the host initialized the Ultravisor, we can set stfle bit 161
> (protected virtual IPL enhancements facility), which indicates, that
> the IPL subcodes 8, 9 and are valid. These subcodes are used by a
> normal guest to set/retrieve a IPIB of type 5 and transition into
> protected mode.
> 
> Once in protected mode, the Ultravisor will conceal the facility
> bit. Therefore each boot into protected mode has to go through
> non-protected. There is no secure re-ipl with subcode 10 without a
> previous subcode 3.
> 
> In protected mode, there is no subcode 4 available, as the VM has no
> more access to its memory from non-protected mode. I.e. each IPL
> clears.
> 
> Signed-off-by: Janosch Frank <frankja@linux.ibm.com>
> ---
>  arch/s390/kvm/diag.c     | 6 ++++++
>  arch/s390/kvm/kvm-s390.c | 5 +++++
>  2 files changed, 11 insertions(+)
> 
> diff --git a/arch/s390/kvm/diag.c b/arch/s390/kvm/diag.c
> index 3fb54ec2cf3e..b951dbdcb6a0 100644
> --- a/arch/s390/kvm/diag.c
> +++ b/arch/s390/kvm/diag.c
> @@ -197,6 +197,12 @@ static int __diag_ipl_functions(struct kvm_vcpu *vcpu)
>  	case 4:
>  		vcpu->run->s390_reset_flags = 0;
>  		break;
> +	case 8:
> +	case 9:
> +	case 10:
> +		if (!test_kvm_facility(vcpu->kvm, 161))
> +			return kvm_s390_inject_program_int(vcpu, PGM_SPECIFICATION);
> +		/* fall through */
>  	default:
>  		return -EOPNOTSUPP;

If we don't handle these diags in the kernel anyway, why do you care
about injecting the program interrupt from kernel space? Couldn't that
be done by userspace instead? I.e. could the whole hunk be dropped?

 Thomas

Re: [RFCv2 19/37] KVM: s390: protvirt: Handle spec exception loops

[Christian Borntraeger]

On 05.02.20 14:50, Cornelia Huck wrote:
> On Mon,  3 Feb 2020 08:19:39 -0500
> Christian Borntraeger <borntraeger@de.ibm.com> wrote:
> 
>> From: Janosch Frank <frankja@linux.ibm.com>
>>
>> SIE intercept code 8 is used only on exception loops for protected
>> guests. That means we need stop the guest when we see it.
> 
> s/need stop/need to stop/

ack
> 
> ...which is a task done by userspace?

ack
>>
>> Signed-off-by: Janosch Frank <frankja@linux.ibm.com>
>> Reviewed-by: Thomas Huth <thuth@redhat.com>
>> ---
>>  arch/s390/kvm/intercept.c | 7 +++++++
>>  1 file changed, 7 insertions(+)
>>
>> diff --git a/arch/s390/kvm/intercept.c b/arch/s390/kvm/intercept.c
>> index c22214967214..d63f9cf10360 100644
>> --- a/arch/s390/kvm/intercept.c
>> +++ b/arch/s390/kvm/intercept.c
>> @@ -231,6 +231,13 @@ static int handle_prog(struct kvm_vcpu *vcpu)
>>  
>>  	vcpu->stat.exit_program_interruption++;
>>  
>> +	/*
>> +	 * Intercept 8 indicates a loop of specification exceptions
>> +	 * for protected guests
> 
> s/guests/guests./

ack.
> 
>> +	 */
>> +	if (kvm_s390_pv_is_protected(vcpu->kvm))
>> +		return -EOPNOTSUPP;
>> +
>>  	if (guestdbg_enabled(vcpu) && per_event(vcpu)) {
>>  		rc = kvm_s390_handle_per_event(vcpu);
>>  		if (rc)
> 
> Reviewed-by: Cornelia Huck <cohuck@redhat.com>
> 

Re: [RFCv2 37/37] KVM: s390: protvirt: Add UV cpu reset calls

[Thomas Huth]

On 03/02/2020 14.19, Christian Borntraeger wrote:
> From: Janosch Frank <frankja@linux.ibm.com>
> 
> For protected VMs, the VCPU resets are done by the Ultravisor, as KVM
> has no access to the VCPU registers.
> 
> As the Ultravisor will only accept a call for the reset that is
> needed, we need to fence the UV calls when chaining resets.
> 
> Signed-off-by: Janosch Frank <frankja@linux.ibm.com>
> ---
>  arch/s390/kvm/kvm-s390.c | 20 ++++++++++++++++++++
>  1 file changed, 20 insertions(+)
> 
> diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
> index 3e4716b3fc02..f7a3f84be064 100644
> --- a/arch/s390/kvm/kvm-s390.c
> +++ b/arch/s390/kvm/kvm-s390.c
> @@ -4699,6 +4699,7 @@ long kvm_arch_vcpu_ioctl(struct file *filp,
>  	void __user *argp = (void __user *)arg;
>  	int idx;
>  	long r;
> +	u32 ret;

I'd maybe name it "uvret" instead to avoid that it gets mixed up easily
with the "r" variable.

Anyway,
Reviewed-by: Thomas Huth <thuth@redhat.com>

Re: [RFCv2 11/37] KVM: s390/mm: Make pages accessible before destroying the guest

[Christian Borntraeger]

On 05.02.20 11:35, Cornelia Huck wrote:
> On Mon,  3 Feb 2020 08:19:31 -0500
> Christian Borntraeger <borntraeger@de.ibm.com> wrote:
> 
>> Before we destroy the secure configuration, we better make all
>> pages accessible again. This also happens during reboot, where we reboot
>> into a non-secure guest that then can go again into a secure mode. As
> 
> s/a secure/secure/

ack
> 
>> this "new" secure guest will have a new ID we cannot reuse the old page
>> state.
>>
>> Signed-off-by: Christian Borntraeger <borntraeger@de.ibm.com>
>> ---
>>  arch/s390/include/asm/pgtable.h |  1 +
>>  arch/s390/kvm/pv.c              |  2 ++
>>  arch/s390/mm/gmap.c             | 35 +++++++++++++++++++++++++++++++++
>>  3 files changed, 38 insertions(+)
> 
> (...)
> 
>> +void s390_reset_acc(struct mm_struct *mm)
>> +{
>> +	/*
>> +	 * we might be called during
>> +	 * reset:                            we walk the pages and clear
>> +	 * close of all kvm file descriptor: we walk the pages and clear
> 
> s/descriptor/descriptors/

ack

> 
>> +	 * exit of process on fd closure:    vma already gone, do nothing
>> +	 */
>> +	if (!mmget_not_zero(mm))
>> +		return;
>> +	down_read(&mm->mmap_sem);
>> +	walk_page_range(mm, 0, TASK_SIZE, &reset_acc_walk_ops, NULL);
>> +	up_read(&mm->mmap_sem);
>> +	mmput(mm);
>> +}
>> +EXPORT_SYMBOL_GPL(s390_reset_acc);
> 
> Reviewed-by: Cornelia Huck <cohuck@redhat.com>
> 

Re: [RFCv2 20/37] KVM: s390: protvirt: Add new gprs location handling

[Christian Borntraeger]

On 05.02.20 12:18, Thomas Huth wrote:
> On 03/02/2020 14.19, Christian Borntraeger wrote:
>> From: Janosch Frank <frankja@linux.ibm.com>
>>
>> Guest registers for protected guests are stored at offset 0x380.
>>
>> Signed-off-by: Janosch Frank <frankja@linux.ibm.com>
>> ---
>>  arch/s390/include/asm/kvm_host.h |  4 +++-
>>  arch/s390/kvm/kvm-s390.c         | 11 +++++++++++
>>  2 files changed, 14 insertions(+), 1 deletion(-)
>>
>> diff --git a/arch/s390/include/asm/kvm_host.h b/arch/s390/include/asm/kvm_host.h
>> index f5ca53574406..125511ec6eb0 100644
>> --- a/arch/s390/include/asm/kvm_host.h
>> +++ b/arch/s390/include/asm/kvm_host.h
>> @@ -343,7 +343,9 @@ struct kvm_s390_itdb {
>>  struct sie_page {
>>  	struct kvm_s390_sie_block sie_block;
>>  	struct mcck_volatile_info mcck_info;	/* 0x0200 */
>> -	__u8 reserved218[1000];		/* 0x0218 */
>> +	__u8 reserved218[360];		/* 0x0218 */
>> +	__u64 pv_grregs[16];		/* 0x380 */
> 
> s/0x380/0x0380/ to align with the other comments

Ack. 
> 
>> +	__u8 reserved400[512];
> 
> Maybe add a /* 0x0400 */ comment ... though it's obvious from the name
> already.

ack
> 
>>  	struct kvm_s390_itdb itdb;	/* 0x0600 */
>>  	__u8 reserved700[2304];		/* 0x0700 */
>>  };
>> diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
>> index 39bf39a10cf2..1945180b857a 100644
>> --- a/arch/s390/kvm/kvm-s390.c
>> +++ b/arch/s390/kvm/kvm-s390.c
>> @@ -3996,6 +3996,7 @@ static int vcpu_post_run(struct kvm_vcpu *vcpu, int exit_reason)
>>  static int __vcpu_run(struct kvm_vcpu *vcpu)
>>  {
>>  	int rc, exit_reason;
>> +	struct sie_page *sie_page = (struct sie_page *)vcpu->arch.sie_block;
>>  
>>  	/*
>>  	 * We try to hold kvm->srcu during most of vcpu_run (except when run-
>> @@ -4017,8 +4018,18 @@ static int __vcpu_run(struct kvm_vcpu *vcpu)
>>  		guest_enter_irqoff();
>>  		__disable_cpu_timer_accounting(vcpu);
>>  		local_irq_enable();
>> +		if (kvm_s390_pv_is_protected(vcpu->kvm)) {
>> +			memcpy(sie_page->pv_grregs,
>> +			       vcpu->run->s.regs.gprs,
>> +			       sizeof(sie_page->pv_grregs));
>> +		}
>>  		exit_reason = sie64a(vcpu->arch.sie_block,
>>  				     vcpu->run->s.regs.gprs);
>> +		if (kvm_s390_pv_is_protected(vcpu->kvm)) {
>> +			memcpy(vcpu->run->s.regs.gprs,
>> +			       sie_page->pv_grregs,
>> +			       sizeof(sie_page->pv_grregs));
>> +		}
>>  		local_irq_disable();
>>  		__enable_cpu_timer_accounting(vcpu);
>>  		guest_exit_irqoff();
>>
> 
> Reviewed-by: Thomas Huth <thuth@redhat.com>
> 

Re: [RFCv2 24/37] KVM: s390: protvirt: Write sthyi data to instruction data area

[Christian Borntraeger]

On 05.02.20 13:09, Thomas Huth wrote:
> On 03/02/2020 14.19, Christian Borntraeger wrote:
>> From: Janosch Frank <frankja@linux.ibm.com>
>>
>> STHYI data has to go through the bounce buffer.
>>
>> Signed-off-by: Janosch Frank <frankja@linux.ibm.com>
>> ---
>>  arch/s390/kvm/intercept.c | 15 ++++++++++-----
>>  1 file changed, 10 insertions(+), 5 deletions(-)
>>
>> diff --git a/arch/s390/kvm/intercept.c b/arch/s390/kvm/intercept.c
>> index 4b1effa44e41..06d1fa83ef4c 100644
>> --- a/arch/s390/kvm/intercept.c
>> +++ b/arch/s390/kvm/intercept.c
>> @@ -392,7 +392,7 @@ int handle_sthyi(struct kvm_vcpu *vcpu)
>>  		goto out;
>>  	}
>>  
>> -	if (addr & ~PAGE_MASK)
>> +	if (!kvm_s390_pv_is_protected(vcpu->kvm) && (addr & ~PAGE_MASK))
>>  		return kvm_s390_inject_program_int(vcpu, PGM_SPECIFICATION);
>>  
>>  	sctns = (void *)get_zeroed_page(GFP_KERNEL);
>> @@ -403,10 +403,15 @@ int handle_sthyi(struct kvm_vcpu *vcpu)
>>  
>>  out:
>>  	if (!cc) {
>> -		r = write_guest(vcpu, addr, reg2, sctns, PAGE_SIZE);
>> -		if (r) {
>> -			free_page((unsigned long)sctns);
>> -			return kvm_s390_inject_prog_cond(vcpu, r);
>> +		if (kvm_s390_pv_is_protected(vcpu->kvm)) {
>> +			memcpy((void *)(vcpu->arch.sie_block->sidad & PAGE_MASK), sctns,
> 
> I think it would look nicer with sida_origin() here.

ack.

> 
>> +			       PAGE_SIZE);
>> +		} else {
>> +			r = write_guest(vcpu, addr, reg2, sctns, PAGE_SIZE);
>> +			if (r) {
>> +				free_page((unsigned long)sctns);
>> +				return kvm_s390_inject_prog_cond(vcpu, r);
>> +			}
>>  		}
>>  	}
>>  
>>
> 
> Reviewed-by: Thomas Huth <thuth@redhat.com>
> 

Re: [RFCv2 25/37] KVM: s390: protvirt: STSI handling

[Christian Borntraeger]

On 05.02.20 13:13, Thomas Huth wrote:
> On 03/02/2020 14.19, Christian Borntraeger wrote:
>> From: Janosch Frank <frankja@linux.ibm.com>
>>
>> Save response to sidad and disable address checking for protected
>> guests.
>>
>> Signed-off-by: Janosch Frank <frankja@linux.ibm.com>
>> ---
>>  arch/s390/kvm/priv.c | 9 +++++++--
>>  1 file changed, 7 insertions(+), 2 deletions(-)
>>
>> diff --git a/arch/s390/kvm/priv.c b/arch/s390/kvm/priv.c
>> index ed52ffa8d5d4..06c7e7a10825 100644
>> --- a/arch/s390/kvm/priv.c
>> +++ b/arch/s390/kvm/priv.c
>> @@ -872,7 +872,7 @@ static int handle_stsi(struct kvm_vcpu *vcpu)
>>  
>>  	operand2 = kvm_s390_get_base_disp_s(vcpu, &ar);
>>  
>> -	if (operand2 & 0xfff)
>> +	if (!kvm_s390_pv_is_protected(vcpu->kvm) && (operand2 & 0xfff))
>>  		return kvm_s390_inject_program_int(vcpu, PGM_SPECIFICATION);
>>  
>>  	switch (fc) {
>> @@ -893,8 +893,13 @@ static int handle_stsi(struct kvm_vcpu *vcpu)
>>  		handle_stsi_3_2_2(vcpu, (void *) mem);
>>  		break;
>>  	}
>> +	if (kvm_s390_pv_is_protected(vcpu->kvm)) {
>> +		memcpy((void *)vcpu->arch.sie_block->sidad, (void *)mem,
> 
> That should use sida_origin() or "...->sidad & PAGE_MASK".

ack.
> 
>> +		       PAGE_SIZE);
>> +		rc = 0;
>> +	} else
>> +		rc = write_guest(vcpu, operand2, ar, (void *)mem, PAGE_SIZE);
> 
> Coding style: If one branch of the if-statement uses curly braces,
> please add them to the other branch as well.
> 

ack

>> -	rc = write_guest(vcpu, operand2, ar, (void *)mem, PAGE_SIZE);
>>  	if (rc) {
>>  		rc = kvm_s390_inject_prog_cond(vcpu, rc);
>>  		goto out;
>>
> 
>  Thomas
> 

Re: [RFCv2 26/37] KVM: s390: protvirt: disallow one_reg

[Christian Borntraeger]

On 05.02.20 13:16, Thomas Huth wrote:
> On 03/02/2020 14.19, Christian Borntraeger wrote:
>> From: Janosch Frank <frankja@linux.ibm.com>
>>
>> A lot of the registers are controlled by the Ultravisor and never
>> visible to KVM. Some fields in the sie control block are overlayed,
>> like gbea. As no userspace uses the ONE_REG interface on s390 it is safe
>> to disable this for protected guests.
>>
>> Signed-off-by: Janosch Frank <frankja@linux.ibm.com>
>> ---
>>  arch/s390/kvm/kvm-s390.c | 3 +++
>>  1 file changed, 3 insertions(+)
>>
>> diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
>> index 6e74c7afae3a..b9692d722c1e 100644
>> --- a/arch/s390/kvm/kvm-s390.c
>> +++ b/arch/s390/kvm/kvm-s390.c
>> @@ -4641,6 +4641,9 @@ long kvm_arch_vcpu_ioctl(struct file *filp,
>>  	case KVM_SET_ONE_REG:
>>  	case KVM_GET_ONE_REG: {
>>  		struct kvm_one_reg reg;
>> +		r = -EINVAL;
>> +		if (kvm_s390_pv_is_protected(vcpu->kvm))
>> +			break;
>>  		r = -EFAULT;
>>  		if (copy_from_user(&reg, argp, sizeof(reg)))
>>  			break;
> 
> Reviewed-by: Thomas Huth <thuth@redhat.com>
> 
> PS:
> Not sure, but maybe it would be also be good to add a sentence to
> Documentation/virt/kvm/api.txt ?


Ack:

diff --git a/Documentation/virt/kvm/api.txt b/Documentation/virt/kvm/api.txt
index 4874d42286ca..8239b3665736 100644
--- a/Documentation/virt/kvm/api.txt
+++ b/Documentation/virt/kvm/api.txt
@@ -1918,7 +1918,8 @@ Parameters: struct kvm_one_reg (in)
 Returns: 0 on success, negative value on failure
 Errors:
   ENOENT:   no such register
-  EINVAL:   invalid register ID, or no such register
+  EINVAL:   invalid register ID, or no such register, ONE_REG forbidden
+            for protected guests (s390).
   EPERM:    (arm64) register access not allowed before vcpu finalization
 (These error codes are indicative only: do not rely on a specific error
 code being returned in a specific situation.)
@@ -2311,7 +2312,8 @@ Parameters: struct kvm_one_reg (in and out)
 Returns: 0 on success, negative value on failure
 Errors include:
   ENOENT:   no such register
-  EINVAL:   invalid register ID, or no such register
+  EINVAL:   invalid register ID, or no such register,ONE_REG forbidden
+            for protected guests (s390)
   EPERM:    (arm64) register access not allowed before vcpu finalization
 (These error codes are indicative only: do not rely on a specific error
 code being returned in a specific situation.)

Re: [RFCv2 28/37] KVM: s390: protvirt: Add program exception injection

[Christian Borntraeger]

On 05.02.20 17:38, Cornelia Huck wrote:
> On Mon,  3 Feb 2020 08:19:48 -0500
> Christian Borntraeger <borntraeger@de.ibm.com> wrote:
> 
>> From: Janosch Frank <frankja@linux.ibm.com>
>>
>> Only two program exceptions can be injected for a protected guest:
>> specification and operand
> 
> s/operand/operand./
> 
ack
>>
>> Both have a code in offset 248 of the state description, as the lowcore
> 
> "For both, a code needs to be specified at offset 248 (iictl) of the
> state description," ?

ack.

> 
>> is not accessible by KVM for such guests.
> 
> s/by/to/

ack
 
>>
>> Signed-off-by: Janosch Frank <frankja@linux.ibm.com>
>> ---
>>  arch/s390/kvm/interrupt.c | 18 ++++++++++++++++++
>>  1 file changed, 18 insertions(+)
> 
> Reviewed-by: Cornelia Huck <cohuck@redhat.com>
> 

Re: [RFCv2 28/37] KVM: s390: protvirt: Add program exception injection

[Christian Borntraeger]

On 05.02.20 18:59, Thomas Huth wrote:
> On 03/02/2020 14.19, Christian Borntraeger wrote:
>> From: Janosch Frank <frankja@linux.ibm.com>
>>
>> Only two program exceptions can be injected for a protected guest:
>> specification and operand
>>
>> Both have a code in offset 248 of the state description, as the lowcore
>> is not accessible by KVM for such guests.
> 
> What do you mean with offset 248 here? You only touch the iictl field
> here, and looking at patch 14/37, iictl is at offset 0x55 ...?

Will fixup the patch description to just say iictl.

Re: [RFCv2 22/37] KVM: s390: protvirt: handle secure guest prefix pages

[Christian Borntraeger]

On 05.02.20 12:51, David Hildenbrand wrote:
> On 03.02.20 14:19, Christian Borntraeger wrote:
>> From: Janosch Frank <frankja@linux.ibm.com>
>>
>> The SPX instruction is handled by the ulravisor. We do get a
>> notification intercept, though. Let us update our internal view.
>>
>> In addition to that, when the guest prefix page is not secure, an
>> intercept 112 (0x70) is indicated.  To avoid this for the most common
>> cases, we can make the guest prefix page protected whenever we pin it.
>> We have to deal with 112 nevertheless, e.g. when some host code triggers
>> an export (e.g. qemu dump guest memory). We can simply re-run the
>> pinning logic by doing a no-op prefix change.
>>
>> Signed-off-by: Janosch Frank <frankja@linux.ibm.com>
>> ---
>>  arch/s390/include/asm/kvm_host.h |  1 +
>>  arch/s390/kvm/intercept.c        | 15 +++++++++++++++
>>  arch/s390/kvm/kvm-s390.c         | 14 ++++++++++++++
>>  3 files changed, 30 insertions(+)
>>
>> diff --git a/arch/s390/include/asm/kvm_host.h b/arch/s390/include/asm/kvm_host.h
>> index 48f382680755..686b00ced55b 100644
>> --- a/arch/s390/include/asm/kvm_host.h
>> +++ b/arch/s390/include/asm/kvm_host.h
>> @@ -225,6 +225,7 @@ struct kvm_s390_sie_block {
>>  #define ICPT_PV_INT_EN	0x64
>>  #define ICPT_PV_INSTR	0x68
>>  #define ICPT_PV_NOTIF	0x6c
>> +#define ICPT_PV_PREF	0x70
>>  	__u8	icptcode;		/* 0x0050 */
>>  	__u8	icptstatus;		/* 0x0051 */
>>  	__u16	ihcpu;			/* 0x0052 */
>> diff --git a/arch/s390/kvm/intercept.c b/arch/s390/kvm/intercept.c
>> index d63f9cf10360..ceba0abb1900 100644
>> --- a/arch/s390/kvm/intercept.c
>> +++ b/arch/s390/kvm/intercept.c
>> @@ -451,6 +451,15 @@ static int handle_operexc(struct kvm_vcpu *vcpu)
>>  	return kvm_s390_inject_program_int(vcpu, PGM_OPERATION);
>>  }
>>  
>> +static int handle_pv_spx(struct kvm_vcpu *vcpu)
>> +{
>> +	u32 pref = *(u32 *)vcpu->arch.sie_block->sidad;
>> +
>> +	kvm_s390_set_prefix(vcpu, pref);
>> +	trace_kvm_s390_handle_prefix(vcpu, 1, pref);
>> +	return 0;
>> +}
>> +
>>  static int handle_pv_sclp(struct kvm_vcpu *vcpu)
>>  {
>>  	struct kvm_s390_float_interrupt *fi = &vcpu->kvm->arch.float_int;
>> @@ -475,6 +484,8 @@ static int handle_pv_sclp(struct kvm_vcpu *vcpu)
>>  
>>  static int handle_pv_not(struct kvm_vcpu *vcpu)
>>  {
>> +	if (vcpu->arch.sie_block->ipa == 0xb210)
>> +		return handle_pv_spx(vcpu);
>>  	if (vcpu->arch.sie_block->ipa == 0xb220)
>>  		return handle_pv_sclp(vcpu);
>>  
>> @@ -533,6 +544,10 @@ int kvm_handle_sie_intercept(struct kvm_vcpu *vcpu)
>>  	case ICPT_PV_NOTIF:
>>  		rc = handle_pv_not(vcpu);
>>  		break;
>> +	case ICPT_PV_PREF:
>> +		rc = 0;
>> +		kvm_s390_set_prefix(vcpu, kvm_s390_get_prefix(vcpu));
> 
> /me confused
> 
> This is the "request to map prefix" case, right?

right. 
> 
> I'd *really* prefer to have a comment and a manual
> 
> /* request to convert and pin the prefix pages again */
> kvm_make_request(KVM_REQ_MMU_RELOAD, vcpu)

I have no objection, this should also work.
Fixed.


> 
> A TLB flush is IMHO not necessary, as the prefix did not change.
> 
>> +		break;
>>  	default:
>>  		return -EOPNOTSUPP;
>>  	}
>> diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
>> index 76303b0f1226..6e74c7afae3a 100644
>> --- a/arch/s390/kvm/kvm-s390.c
>> +++ b/arch/s390/kvm/kvm-s390.c
>> @@ -3675,6 +3675,20 @@ static int kvm_s390_handle_requests(struct kvm_vcpu *vcpu)
>>  		rc = gmap_mprotect_notify(vcpu->arch.gmap,
>>  					  kvm_s390_get_prefix(vcpu),
>>  					  PAGE_SIZE * 2, PROT_WRITE);
>> +		if (!rc && kvm_s390_pv_is_protected(vcpu->kvm)) {
>> +			do {
>> +				rc = uv_convert_to_secure(
>> +						vcpu->arch.gmap,
>> +						kvm_s390_get_prefix(vcpu));
>> +			} while (rc == -EAGAIN);
>> +			WARN_ONCE(rc, "Error while importing first prefix page. rc %d", rc);
>> +			do {
>> +				rc = uv_convert_to_secure(
>> +						vcpu->arch.gmap,
>> +						kvm_s390_get_prefix(vcpu) + PAGE_SIZE);
>> +			} while (rc == -EAGAIN);
>> +			WARN_ONCE(rc, "Error while importing second prefix page. rc %d", rc);
> 
> Maybe factor that out into a separate function (e.g., for a single page
> and call that twice).

I will wait until the memory management work is complete (we are almost there).

Re: [RFCv2 22/37] KVM: s390: protvirt: handle secure guest prefix pages

[Christian Borntraeger]

On 05.02.20 12:52, David Hildenbrand wrote:
> On 03.02.20 14:19, Christian Borntraeger wrote:
>> From: Janosch Frank <frankja@linux.ibm.com>
>>
>> The SPX instruction is handled by the ulravisor. We do get a
> 
> s/ulravisor/ultravisor/

ack

> 
>> notification intercept, though. Let us update our internal view.
>>
>> In addition to that, when the guest prefix page is not secure, an
>> intercept 112 (0x70) is indicated.  To avoid this for the most common
>> cases, we can make the guest prefix page protected whenever we pin it.
>> We have to deal with 112 nevertheless, e.g. when some host code triggers
>> an export (e.g. qemu dump guest memory). We can simply re-run the
>> pinning logic by doing a no-op prefix change.
>>
>> Signed-off-by: Janosch Frank <frankja@linux.ibm.com>
> 
> 

Re: [RFCv2 37/37] KVM: s390: protvirt: Add UV cpu reset calls

[Christian Borntraeger]

On 05.02.20 19:27, Thomas Huth wrote:
[...]

>> index 3e4716b3fc02..f7a3f84be064 100644
>> --- a/arch/s390/kvm/kvm-s390.c
>> +++ b/arch/s390/kvm/kvm-s390.c
>> @@ -4699,6 +4699,7 @@ long kvm_arch_vcpu_ioctl(struct file *filp,
>>  	void __user *argp = (void __user *)arg;
>>  	int idx;
>>  	long r;
>> +	u32 ret;
> 
> I'd maybe name it "uvret" instead to avoid that it gets mixed up easily
> with the "r" variable.

can do.

> 
> Anyway,
> Reviewed-by: Thomas Huth <thuth@redhat.com>
> 

Re: [RFCv2 15/37] KVM: s390: protvirt: Implement interruption injection

[Cornelia Huck]

On Wed, 5 Feb 2020 18:49:45 +0100
Christian Borntraeger <borntraeger@de.ibm.com> wrote:

> On 05.02.20 12:25, Cornelia Huck wrote:
> > On Wed, 5 Feb 2020 11:48:42 +0100
> > Christian Borntraeger <borntraeger@de.ibm.com> wrote:
> >   
> >> On 05.02.20 10:51, Thomas Huth wrote:  
> >>> On 03/02/2020 14.19, Christian Borntraeger wrote:    
> >>>> From: Michael Mueller <mimu@linux.ibm.com>
> >>>>
> >>>> The patch implements interruption injection for the following
> >>>> list of interruption types:
> >>>>
> >>>>   - I/O
> >>>>     __deliver_io (III)
> >>>>
> >>>>   - External
> >>>>     __deliver_cpu_timer (IEI)
> >>>>     __deliver_ckc (IEI)
> >>>>     __deliver_emergency_signal (IEI)
> >>>>     __deliver_external_call (IEI)
> >>>>
> >>>>   - cpu restart
> >>>>     __deliver_restart (IRI)
> >>>>
> >>>> The service interrupt is handled in a followup patch.
> >>>>
> >>>> Signed-off-by: Michael Mueller <mimu@linux.ibm.com>
> >>>> Signed-off-by: Christian Borntraeger <borntraeger@de.ibm.com>
> >>>> [fixes]
> >>>> ---
> >>>>  arch/s390/include/asm/kvm_host.h |  8 +++
> >>>>  arch/s390/kvm/interrupt.c        | 93 ++++++++++++++++++++++----------
> >>>>  2 files changed, 74 insertions(+), 27 deletions(-)
> >>>>
> >>>> diff --git a/arch/s390/include/asm/kvm_host.h b/arch/s390/include/asm/kvm_host.h
> >>>> index a45d10d87a8a..989cea7a5591 100644
> >>>> --- a/arch/s390/include/asm/kvm_host.h
> >>>> +++ b/arch/s390/include/asm/kvm_host.h
> >>>> @@ -563,6 +563,14 @@ enum irq_types {
> >>>>  #define IRQ_PEND_MCHK_MASK ((1UL << IRQ_PEND_MCHK_REP) | \
> >>>>  			    (1UL << IRQ_PEND_MCHK_EX))
> >>>>  
> >>>> +#define IRQ_PEND_MCHK_REP_MASK (1UL << IRQ_PEND_MCHK_REP)
> >>>> +
> >>>> +#define IRQ_PEND_EXT_II_MASK ((1UL << IRQ_PEND_EXT_CPU_TIMER)  | \
> >>>> +			      (1UL << IRQ_PEND_EXT_CLOCK_COMP) | \
> >>>> +			      (1UL << IRQ_PEND_EXT_EMERGENCY)  | \
> >>>> +			      (1UL << IRQ_PEND_EXT_EXTERNAL)   | \
> >>>> +			      (1UL << IRQ_PEND_EXT_SERVICE))
> >>>> +
> >>>>  struct kvm_s390_interrupt_info {
> >>>>  	struct list_head list;
> >>>>  	u64	type;
> >>>> diff --git a/arch/s390/kvm/interrupt.c b/arch/s390/kvm/interrupt.c
> >>>> index c06c89d370a7..ecdec6960a60 100644
> >>>> --- a/arch/s390/kvm/interrupt.c
> >>>> +++ b/arch/s390/kvm/interrupt.c
> >>>> @@ -387,6 +387,12 @@ static unsigned long deliverable_irqs(struct kvm_vcpu *vcpu)
> >>>>  		__clear_bit(IRQ_PEND_EXT_SERVICE, &active_mask);
> >>>>  	if (psw_mchk_disabled(vcpu))
> >>>>  		active_mask &= ~IRQ_PEND_MCHK_MASK;
> >>>> +	/* PV guest cpus can have a single interruption injected at a time. */
> >>>> +	if (kvm_s390_pv_is_protected(vcpu->kvm) &&
> >>>> +	    vcpu->arch.sie_block->iictl != IICTL_CODE_NONE)
> >>>> +		active_mask &= ~(IRQ_PEND_EXT_II_MASK |
> >>>> +				 IRQ_PEND_IO_MASK |
> >>>> +				 IRQ_PEND_MCHK_REP_MASK);    
> >>>
> >>> I don't quite understand why there is a difference between
> >>> IRQ_PEND_MCHK_REP and IRQ_PEND_MCHK_EX here? Why not simply use
> >>> IRQ_PEND_MCHK_MASK here? Could you elaborate? (and maybe add a sentence
> >>> to the patch description)    
> >>
> >> I added that part. 
> >> My idea was that an exigent machine check would be kind of fatal that it can override
> >> the previous interrupt. Now we do not implement the override (kill the previous interrupt)
> >> so I agree, maybe lets use IRQ_PEND_MCHK_MASK  
> > 
> > This makes me wonder about interrupt priorities in general. Under which
> > circumstances can we have an interrupt with a lower priority already
> > injected (but not delivered) in the injection area when a
> > higher-priority interrupt comes along? I'm a bit confused here.  
> 
> This is not an issue. If the interrupt is already in the injection area, then you can consider
> that as delivered. And even highest priority interrupt cannot overtake an already delivered one.
> 

Ah, ok. That 'override' above made me think it was possible.

Re: [RFCv2 15/37] KVM: s390: protvirt: Implement interruption injection

[Cornelia Huck]

On Wed, 5 Feb 2020 19:00:47 +0100
Christian Borntraeger <borntraeger@de.ibm.com> wrote:

> On 05.02.20 13:26, Michael Mueller wrote:
> > 
> > 
> > On 05.02.20 13:11, Cornelia Huck wrote:  
> >> On Wed, 5 Feb 2020 12:46:39 +0100
> >> Michael Mueller <mimu@linux.ibm.com> wrote:
> >>  
> >>> On 05.02.20 12:31, Cornelia Huck wrote:  
> >>>> On Mon,  3 Feb 2020 08:19:35 -0500
> >>>> Christian Borntraeger <borntraeger@de.ibm.com> wrote:
> >>>>     
> >>>>> From: Michael Mueller <mimu@linux.ibm.com>
> >>>>>
> >>>>> The patch implements interruption injection for the following
> >>>>> list of interruption types:
> >>>>>
> >>>>>     - I/O
> >>>>>       __deliver_io (III)
> >>>>>
> >>>>>     - External
> >>>>>       __deliver_cpu_timer (IEI)
> >>>>>       __deliver_ckc (IEI)
> >>>>>       __deliver_emergency_signal (IEI)
> >>>>>       __deliver_external_call (IEI)
> >>>>>
> >>>>>     - cpu restart
> >>>>>       __deliver_restart (IRI)  
> >>>>
> >>>> Hm... what do 'III', 'IEI', and 'IRI' stand for?  
> >>>
> >>> that's the kind of interruption injection being used:
> >>>
> >>> inject io interruption
> >>> inject external interruption
> >>> inject restart interruption  
> >>
> >> So, maybe make this:
> >>
> >> - I/O (uses inject io interruption)
> >>    __ deliver_io
> >>
> >> - External (uses inject external interruption)
> >> (and so on)
> >>
> >> I find using the acronyms without explanation very confusing.  
> > 
> > Make a guess from where they are coming...
> > 
> > Christian, would you please update the description accordingly.
> > 
> > 
> >   - I/O (uses inject io interruption)
> >     __deliver_io
> > 
> >   - External (uses inject external interruption)
> >     __deliver_cpu_timer
> >     __deliver_ckc
> >     __deliver_emergency_signal
> >     __deliver_external_call
> > 
> >   - cpu restart (uses inject restart interruption)
> >     __deliver_restart  
> 
> Will use that and also add.
> Please note that posted interrupts (GISA) are not used for protected guests as of today.

Thanks, that makes it more clear.

> 
> > 
> > thanks
> > 
> > Michael
> >   
> >>  
> >>>  
> >>>>     
> >>>>>
> >>>>> The service interrupt is handled in a followup patch.
> >>>>>
> >>>>> Signed-off-by: Michael Mueller <mimu@linux.ibm.com>
> >>>>> Signed-off-by: Christian Borntraeger <borntraeger@de.ibm.com>
> >>>>> [fixes]
> >>>>> ---
> >>>>>    arch/s390/include/asm/kvm_host.h |  8 +++
> >>>>>    arch/s390/kvm/interrupt.c        | 93 ++++++++++++++++++++++----------
> >>>>>    2 files changed, 74 insertions(+), 27 deletions(-)  
> >>  
> >   
> 

Re: [RFCv2 18/37] KVM: s390: protvirt: Implement machine-check interruption injection

[Cornelia Huck]

On Wed, 5 Feb 2020 19:18:44 +0100
Christian Borntraeger <borntraeger@de.ibm.com> wrote:

> On 05.02.20 14:47, Cornelia Huck wrote:
> [..]
> >> --- a/arch/s390/kvm/interrupt.c
> >> +++ b/arch/s390/kvm/interrupt.c
> >> @@ -571,6 +571,14 @@ static int __write_machine_check(struct kvm_vcpu *vcpu,
> >>  	union mci mci;
> >>  	int rc;
> >>  
> >> +	if (kvm_s390_pv_is_protected(vcpu->kvm)) {
> >> +		vcpu->arch.sie_block->iictl = IICTL_CODE_MCHK;
> >> +		vcpu->arch.sie_block->mcic = mchk->mcic;
> >> +		vcpu->arch.sie_block->faddr = mchk->failing_storage_address;
> >> +		vcpu->arch.sie_block->edc = mchk->ext_damage_code;  
> > 
> > Maybe add a comment that we don't need with other machine-check related data?  
> 
> Not sure I get this point. Can you make a proposal?

/*
 * All other possible payload for a machine check will
 * not be handled by the hypervisor, as it does not have
 * the needed information for protected guests.
 */

Something like that?

> 
> >   
> >> +		return 0;
> >> +	}
> >> +
> >>  	mci.val = mchk->mcic;
> >>  	/* take care of lazy register loading */
> >>  	save_fpu_regs();  
> > 
> > Anyway,
> > Reviewed-by: Cornelia Huck <cohuck@redhat.com>
> >   
> 

Re: [RFCv2 21/37] KVM: S390: protvirt: Introduce instruction data area bounce buffer

[Christian Borntraeger]

On 05.02.20 12:43, David Hildenbrand wrote:
>>  #ifdef CONFIG_KVM_S390_PROTECTED_VIRTUALIZATION_HOST
>> +static long kvm_s390_guest_sida_op(struct kvm_vcpu *vcpu,
>> +				   struct kvm_s390_mem_op *mop)
>> +{
>> +	int r = 0;
>> +	void __user *uaddr = (void __user *)mop->buf;
> 
> Reverse christmas tree :)

ack.
> 
>> +
>> +	if (mop->flags || !mop->size)
>> +		return -EINVAL;
>> +
>> +	if (mop->size > sida_size(vcpu->arch.sie_block))
>> +		return -E2BIG;
> 
> Should be caught by the check below as well (or is this an implicit
> overflow check? - see below).
> 
>> +
>> +	if (mop->size + mop->offset > sida_size(vcpu->arch.sie_block))
>> +		return -E2BIG;
>> +
> 
> Do we have to care about overflows? (at least the offset is 32-bit,
> didn't check the size :))

size and offset are both unsigned. So offset 0xfffffff0 + size 0x100 would not
be covered by both checks. 

Hmm, should we add this as well?

@@ -4538,6 +4538,9 @@ static long kvm_s390_guest_sida_op(struct kvm_vcpu *vcpu,
        if (mop->size > sida_size(vcpu->arch.sie_block))
                return -E2BIG;
 
+       if (mop->offset > sida_size(vcpu->arch.sie_block))
+               return -E2BIG;
+
        if (mop->size + mop->offset > sida_size(vcpu->arch.sie_block))
                return -E2BIG;
 


> 
> 
>> +	switch (mop->op) {
>> +	case KVM_S390_MEMOP_SIDA_READ:
>> +		r = 0;
>> +		if (copy_to_user(uaddr, (void *)sida_origin(vcpu->arch.sie_block) +
>> +				 mop->offset, mop->size))
>> +			r = -EFAULT;
>> +
>> +		break;
>> +	case KVM_S390_MEMOP_SIDA_WRITE:
>> +		r = 0;
>> +		if (copy_from_user((void *)vcpu->arch.sie_block->sidad +
>> +				   mop->offset, uaddr, mop->size))
>> +			r = -EFAULT;
>> +		break;
>> +	}
>> +	return r;
>> +}
>> +
>>  static int kvm_s390_handle_pv_vcpu(struct kvm_vcpu *vcpu,
>>  				   struct kvm_pv_cmd *cmd)
>>  {
>> @@ -4708,6 +4743,20 @@ long kvm_arch_vcpu_ioctl(struct file *filp,
>>  		r = kvm_s390_handle_pv_vcpu(vcpu, &args);
>>  		break;
>>  	}
>> +	case KVM_S390_SIDA_OP: {
>> +		struct kvm_s390_mem_op mem_op;
>> +
>> +		if (!kvm_s390_pv_is_protected(vcpu->kvm)) {
>> +			r = -EINVAL;
>> +			break;
>> +		}
> 
> Could we race against a VM_DESTROY? Should we protect somehow?

As far as I can tell SIDA_OP and the VCPU_DESTROY are both per cpu ioctls
and thus protected by the vcpu->mutex. (please double check).

> [...]
> 
>> -/* for KVM_S390_MEM_OP */
>> +/* for KVM_S390_MEM_OP and KVM_S390_SIDA_OP */
>>  struct kvm_s390_mem_op {
>>  	/* in */
>>  	__u64 gaddr;		/* the guest address */
>> @@ -475,11 +475,17 @@ struct kvm_s390_mem_op {
>>  	__u32 op;		/* type of operation */
>>  	__u64 buf;		/* buffer in userspace */
>>  	__u8 ar;		/* the access register number */
>> -	__u8 reserved[31];	/* should be set to 0 */
>> +	__u8 reserved21[3];	/* should be set to 0 */
>> +	__u32 offset;		/* offset into the sida */
> 
> maybe "side_offset"? or define a union, overlying the ar (because that
> obviously doesn't apply to this memop). So eventually different layout
> for different memop.

Will use sida_offset for now, but I have to look into Thomas proposal.

Re: [RFCv2 18/37] KVM: s390: protvirt: Implement machine-check interruption injection

[Christian Borntraeger]

On 06.02.20 09:25, Cornelia Huck wrote:
> On Wed, 5 Feb 2020 19:18:44 +0100
> Christian Borntraeger <borntraeger@de.ibm.com> wrote:
> 
>> On 05.02.20 14:47, Cornelia Huck wrote:
>> [..]
>>>> --- a/arch/s390/kvm/interrupt.c
>>>> +++ b/arch/s390/kvm/interrupt.c
>>>> @@ -571,6 +571,14 @@ static int __write_machine_check(struct kvm_vcpu *vcpu,
>>>>  	union mci mci;
>>>>  	int rc;
>>>>  
>>>> +	if (kvm_s390_pv_is_protected(vcpu->kvm)) {
>>>> +		vcpu->arch.sie_block->iictl = IICTL_CODE_MCHK;
>>>> +		vcpu->arch.sie_block->mcic = mchk->mcic;
>>>> +		vcpu->arch.sie_block->faddr = mchk->failing_storage_address;
>>>> +		vcpu->arch.sie_block->edc = mchk->ext_damage_code;  
>>>
>>> Maybe add a comment that we don't need with other machine-check related data?  
>>
>> Not sure I get this point. Can you make a proposal?
> 
> /*
>  * All other possible payload for a machine check will
>  * not be handled by the hypervisor, as it does not have
>  * the needed information for protected guests.
>  */
> 
> Something like that?

Ah, you mean the registers and so on for the checkout?
I will add 
        /*
         * All other possible payload for a machine check (e.g. the register
         * contents in the save area) will be handled by the ultravisor, as 
         * the hypervisor does not not have the needed information for
         * protected guests.
         */

Re: [RFCv2 21/37] KVM: S390: protvirt: Introduce instruction data area bounce buffer

[Christian Borntraeger]

On 05.02.20 18:00, Thomas Huth wrote:

>>>
>>> Uh, why the mix of a new ioctl with the existing mem_op stuff? Could you
>>> please either properly integrate this into the MEM_OP ioctl (and e.g.
>>> use gaddr as offset for the new SIDA_READ and SIDA_WRITE subcodes), or
>>> completely separate it for a new ioctl, i.e. introduce a new struct for
>>> the new ioctl instead of recycling the struct kvm_s390_mem_op here?
>>> (and in case you ask me, I'd slightly prefer to integrate everything
>>> into MEM_OP instead of introducing a new ioctl here).
>>
>> *cough* David and Christian didn't like the memop solution and it took
>> me a long time to get this to work properly in QEMU...
> 
> I also don't like to re-use MEMOP_LOGICAL_READ and MEMOP_LOGICAL_WRITE
> for the SIDA like you've had it in RFC v1 ... but what's wrong with
> using KVM_S390_MEMOP_SIDA_READ and KVM_S390_MEMOP_SIDA_WRITE with the
> MEM_OP ioctl directly?
> 
>  Thomas
> 

In essence something like the following?

@@ -4583,6 +4618,9 @@ static long kvm_s390_guest_mem_op(struct kvm_vcpu *vcpu,
                }
                r = write_guest(vcpu, mop->gaddr, mop->ar, tmpbuf, mop->size);
                break;
+       case KVM_S390_MEMOP_SIDA_READ:
+       case KVM_S390_MEMOP_SIDA_WRITE:
+               kvm_s390_guest_sida_op(vcpu, mop);
        default:
                r = -EINVAL;
        }


diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
index ea2b4d66e0c3..6e029753c955 100644
--- a/include/uapi/linux/kvm.h
+++ b/include/uapi/linux/kvm.h
@@ -1519,7 +1519,6 @@ struct kvm_pv_cmd {
 /* Available with KVM_CAP_S390_PROTECTED */
 #define KVM_S390_PV_COMMAND            _IOW(KVMIO, 0xc5, struct kvm_pv_cmd)
 #define KVM_S390_PV_COMMAND_VCPU       _IOW(KVMIO, 0xc6, struct kvm_pv_cmd)
-#define KVM_S390_SIDA_OP               _IOW(KVMIO, 0xc7, struct kvm_s390_mem_op)
 
 /* Secure Encrypted Virtualization command */
 enum sev_cmd_id {

Re: [RFCv2 18/37] KVM: s390: protvirt: Implement machine-check interruption injection

[Cornelia Huck]

On Thu, 6 Feb 2020 10:01:02 +0100
Christian Borntraeger <borntraeger@de.ibm.com> wrote:

> On 06.02.20 09:25, Cornelia Huck wrote:
> > On Wed, 5 Feb 2020 19:18:44 +0100
> > Christian Borntraeger <borntraeger@de.ibm.com> wrote:
> >   
> >> On 05.02.20 14:47, Cornelia Huck wrote:
> >> [..]  
> >>>> --- a/arch/s390/kvm/interrupt.c
> >>>> +++ b/arch/s390/kvm/interrupt.c
> >>>> @@ -571,6 +571,14 @@ static int __write_machine_check(struct kvm_vcpu *vcpu,
> >>>>  	union mci mci;
> >>>>  	int rc;
> >>>>  
> >>>> +	if (kvm_s390_pv_is_protected(vcpu->kvm)) {
> >>>> +		vcpu->arch.sie_block->iictl = IICTL_CODE_MCHK;
> >>>> +		vcpu->arch.sie_block->mcic = mchk->mcic;
> >>>> +		vcpu->arch.sie_block->faddr = mchk->failing_storage_address;
> >>>> +		vcpu->arch.sie_block->edc = mchk->ext_damage_code;    
> >>>
> >>> Maybe add a comment that we don't need with other machine-check related data?    
> >>
> >> Not sure I get this point. Can you make a proposal?  
> > 
> > /*
> >  * All other possible payload for a machine check will
> >  * not be handled by the hypervisor, as it does not have
> >  * the needed information for protected guests.
> >  */
> > 
> > Something like that?  
> 
> Ah, you mean the registers and so on for the checkout?
> I will add 
>         /*
>          * All other possible payload for a machine check (e.g. the register
>          * contents in the save area) will be handled by the ultravisor, as 
>          * the hypervisor does not not have the needed information for
>          * protected guests.
>          */
> 

Sounds good!

Re: [RFCv2 21/37] KVM: S390: protvirt: Introduce instruction data area bounce buffer

[Thomas Huth]

On 06/02/2020 10.07, Christian Borntraeger wrote:
> On 05.02.20 18:00, Thomas Huth wrote:
> 
>>>>
>>>> Uh, why the mix of a new ioctl with the existing mem_op stuff? Could you
>>>> please either properly integrate this into the MEM_OP ioctl (and e.g.
>>>> use gaddr as offset for the new SIDA_READ and SIDA_WRITE subcodes), or
>>>> completely separate it for a new ioctl, i.e. introduce a new struct for
>>>> the new ioctl instead of recycling the struct kvm_s390_mem_op here?
>>>> (and in case you ask me, I'd slightly prefer to integrate everything
>>>> into MEM_OP instead of introducing a new ioctl here).
>>>
>>> *cough* David and Christian didn't like the memop solution and it took
>>> me a long time to get this to work properly in QEMU...
>>
>> I also don't like to re-use MEMOP_LOGICAL_READ and MEMOP_LOGICAL_WRITE
>> for the SIDA like you've had it in RFC v1 ... but what's wrong with
>> using KVM_S390_MEMOP_SIDA_READ and KVM_S390_MEMOP_SIDA_WRITE with the
>> MEM_OP ioctl directly?
>>
>>  Thomas
>>
> 
> In essence something like the following?
> 
> @@ -4583,6 +4618,9 @@ static long kvm_s390_guest_mem_op(struct kvm_vcpu *vcpu,
>                 }
>                 r = write_guest(vcpu, mop->gaddr, mop->ar, tmpbuf, mop->size);
>                 break;
> +       case KVM_S390_MEMOP_SIDA_READ:
> +       case KVM_S390_MEMOP_SIDA_WRITE:
> +               kvm_s390_guest_sida_op(vcpu, mop);
>         default:
>                 r = -EINVAL;
>         }
> 
> 
> diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
> index ea2b4d66e0c3..6e029753c955 100644
> --- a/include/uapi/linux/kvm.h
> +++ b/include/uapi/linux/kvm.h
> @@ -1519,7 +1519,6 @@ struct kvm_pv_cmd {
>  /* Available with KVM_CAP_S390_PROTECTED */
>  #define KVM_S390_PV_COMMAND            _IOW(KVMIO, 0xc5, struct kvm_pv_cmd)
>  #define KVM_S390_PV_COMMAND_VCPU       _IOW(KVMIO, 0xc6, struct kvm_pv_cmd)
> -#define KVM_S390_SIDA_OP               _IOW(KVMIO, 0xc7, struct kvm_s390_mem_op)
>  
>  /* Secure Encrypted Virtualization command */
>  enum sev_cmd_id {

Right!

But maybe you should also fence the other subcodes in case of PV:

diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
index d9e6bf3d54f0..f99e7d7af6ea 100644
--- a/arch/s390/kvm/kvm-s390.c
+++ b/arch/s390/kvm/kvm-s390.c
@@ -4274,6 +4274,10 @@ static long kvm_s390_guest_mem_op(struct kvm_vcpu
*vcpu,

        switch (mop->op) {
        case KVM_S390_MEMOP_LOGICAL_READ:
+               if (kvm_s390_pv_is_protected(vcpu->kvm))
+                       r = -EINVAL;
+                       break;
+               }
                if (mop->flags & KVM_S390_MEMOP_F_CHECK_ONLY) {
                        r = check_gva_range(vcpu, mop->gaddr, mop->ar,
                                            mop->size, GACC_FETCH);
@@ -4286,6 +4290,10 @@ static long kvm_s390_guest_mem_op(struct kvm_vcpu
*vcpu,
                }
                break;
        case KVM_S390_MEMOP_LOGICAL_WRITE:
+               if (kvm_s390_pv_is_protected(vcpu->kvm))
+                       r = -EINVAL;
+                       break;
+               }
                if (mop->flags & KVM_S390_MEMOP_F_CHECK_ONLY) {
                        r = check_gva_range(vcpu, mop->gaddr, mop->ar,
                                            mop->size, GACC_STORE);

... not sure whether it's maybe easier in the end to move everything to
a new ioctl with a new struct instead ... whatever you prefer.

But I guess there should be a check like the above in
kvm_s390_guest_mem_op() anyway to avoid that userspace can write to
protected pages with this MEM_OP ioctl.

 Thomas

Re: [RFCv2 21/37] KVM: S390: protvirt: Introduce instruction data area bounce buffer

[Christian Borntraeger]

On 06.02.20 10:07, Christian Borntraeger wrote:
> On 05.02.20 18:00, Thomas Huth wrote:
> 
>>>>
>>>> Uh, why the mix of a new ioctl with the existing mem_op stuff? Could you
>>>> please either properly integrate this into the MEM_OP ioctl (and e.g.
>>>> use gaddr as offset for the new SIDA_READ and SIDA_WRITE subcodes), or
>>>> completely separate it for a new ioctl, i.e. introduce a new struct for
>>>> the new ioctl instead of recycling the struct kvm_s390_mem_op here?
>>>> (and in case you ask me, I'd slightly prefer to integrate everything
>>>> into MEM_OP instead of introducing a new ioctl here).
>>>
>>> *cough* David and Christian didn't like the memop solution and it took
>>> me a long time to get this to work properly in QEMU...
>>
>> I also don't like to re-use MEMOP_LOGICAL_READ and MEMOP_LOGICAL_WRITE
>> for the SIDA like you've had it in RFC v1 ... but what's wrong with
>> using KVM_S390_MEMOP_SIDA_READ and KVM_S390_MEMOP_SIDA_WRITE with the
>> MEM_OP ioctl directly?
>>
>>  Thomas
>>
> 
> In essence something like the following?
> 
> @@ -4583,6 +4618,9 @@ static long kvm_s390_guest_mem_op(struct kvm_vcpu *vcpu,
>                 }
>                 r = write_guest(vcpu, mop->gaddr, mop->ar, tmpbuf, mop->size);
>                 break;
> +       case KVM_S390_MEMOP_SIDA_READ:
> +       case KVM_S390_MEMOP_SIDA_WRITE:
> +               kvm_s390_guest_sida_op(vcpu, mop);

a break; here

>         default:
>                 r = -EINVAL;
>         }
> 
> 
> diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
> index ea2b4d66e0c3..6e029753c955 100644
> --- a/include/uapi/linux/kvm.h
> +++ b/include/uapi/linux/kvm.h
> @@ -1519,7 +1519,6 @@ struct kvm_pv_cmd {
>  /* Available with KVM_CAP_S390_PROTECTED */
>  #define KVM_S390_PV_COMMAND            _IOW(KVMIO, 0xc5, struct kvm_pv_cmd)
>  #define KVM_S390_PV_COMMAND_VCPU       _IOW(KVMIO, 0xc6, struct kvm_pv_cmd)
> -#define KVM_S390_SIDA_OP               _IOW(KVMIO, 0xc7, struct kvm_s390_mem_op)
>  
>  /* Secure Encrypted Virtualization command */
>  enum sev_cmd_id {
> 


Janosch, I just checked your qemu tree
the change would be just the following when we go down that path. (and it makes perfect sense)



diff --git a/target/s390x/kvm.c b/target/s390x/kvm.c
index 42d6ef5fad..29bcdf839f 100644
--- a/target/s390x/kvm.c
+++ b/target/s390x/kvm.c
@@ -862,7 +862,7 @@ int kvm_s390_mem_op_pv(S390CPU *cpu, uint64_t offset, void *hostbuf,
         return -ENOSYS;
     }
 
-    ret = kvm_vcpu_ioctl(CPU(cpu), KVM_S390_SIDA_OP, &mem_op);
+    ret = kvm_vcpu_ioctl(CPU(cpu), KVM_S390_MEM_OP, &mem_op);
     if (ret < 0) {
         warn_report("KVM_S390_MEM_OP failed: %s", strerror(-ret));
     }

Re: [RFCv2 21/37] KVM: S390: protvirt: Introduce instruction data area bounce buffer

[Christian Borntraeger]

On 06.02.20 10:18, Thomas Huth wrote:
> On 06/02/2020 10.07, Christian Borntraeger wrote:
>> On 05.02.20 18:00, Thomas Huth wrote:
>>
>>>>>
>>>>> Uh, why the mix of a new ioctl with the existing mem_op stuff? Could you
>>>>> please either properly integrate this into the MEM_OP ioctl (and e.g.
>>>>> use gaddr as offset for the new SIDA_READ and SIDA_WRITE subcodes), or
>>>>> completely separate it for a new ioctl, i.e. introduce a new struct for
>>>>> the new ioctl instead of recycling the struct kvm_s390_mem_op here?
>>>>> (and in case you ask me, I'd slightly prefer to integrate everything
>>>>> into MEM_OP instead of introducing a new ioctl here).
>>>>
>>>> *cough* David and Christian didn't like the memop solution and it took
>>>> me a long time to get this to work properly in QEMU...
>>>
>>> I also don't like to re-use MEMOP_LOGICAL_READ and MEMOP_LOGICAL_WRITE
>>> for the SIDA like you've had it in RFC v1 ... but what's wrong with
>>> using KVM_S390_MEMOP_SIDA_READ and KVM_S390_MEMOP_SIDA_WRITE with the
>>> MEM_OP ioctl directly?
>>>
>>>  Thomas
>>>
>>
>> In essence something like the following?
>>
>> @@ -4583,6 +4618,9 @@ static long kvm_s390_guest_mem_op(struct kvm_vcpu *vcpu,
>>                 }
>>                 r = write_guest(vcpu, mop->gaddr, mop->ar, tmpbuf, mop->size);
>>                 break;
>> +       case KVM_S390_MEMOP_SIDA_READ:
>> +       case KVM_S390_MEMOP_SIDA_WRITE:
>> +               kvm_s390_guest_sida_op(vcpu, mop);
>>         default:
>>                 r = -EINVAL;
>>         }
>>
>>
>> diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
>> index ea2b4d66e0c3..6e029753c955 100644
>> --- a/include/uapi/linux/kvm.h
>> +++ b/include/uapi/linux/kvm.h
>> @@ -1519,7 +1519,6 @@ struct kvm_pv_cmd {
>>  /* Available with KVM_CAP_S390_PROTECTED */
>>  #define KVM_S390_PV_COMMAND            _IOW(KVMIO, 0xc5, struct kvm_pv_cmd)
>>  #define KVM_S390_PV_COMMAND_VCPU       _IOW(KVMIO, 0xc6, struct kvm_pv_cmd)
>> -#define KVM_S390_SIDA_OP               _IOW(KVMIO, 0xc7, struct kvm_s390_mem_op)
>>  
>>  /* Secure Encrypted Virtualization command */
>>  enum sev_cmd_id {
> 
> Right!
> 
> But maybe you should also fence the other subcodes in case of PV:
> 
> diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
> index d9e6bf3d54f0..f99e7d7af6ea 100644
> --- a/arch/s390/kvm/kvm-s390.c
> +++ b/arch/s390/kvm/kvm-s390.c
> @@ -4274,6 +4274,10 @@ static long kvm_s390_guest_mem_op(struct kvm_vcpu
> *vcpu,
> 
>         switch (mop->op) {
>         case KVM_S390_MEMOP_LOGICAL_READ:
> +               if (kvm_s390_pv_is_protected(vcpu->kvm))
> +                       r = -EINVAL;
> +                       break;
> +               }

Yes, I will add those.

Re: [RFCv2 21/37] KVM: S390: protvirt: Introduce instruction data area bounce buffer

[David Hildenbrand]

On 06.02.20 10:07, Christian Borntraeger wrote:
> On 05.02.20 18:00, Thomas Huth wrote:
> 
>>>>
>>>> Uh, why the mix of a new ioctl with the existing mem_op stuff? Could you
>>>> please either properly integrate this into the MEM_OP ioctl (and e.g.
>>>> use gaddr as offset for the new SIDA_READ and SIDA_WRITE subcodes), or
>>>> completely separate it for a new ioctl, i.e. introduce a new struct for
>>>> the new ioctl instead of recycling the struct kvm_s390_mem_op here?
>>>> (and in case you ask me, I'd slightly prefer to integrate everything
>>>> into MEM_OP instead of introducing a new ioctl here).
>>>
>>> *cough* David and Christian didn't like the memop solution and it took
>>> me a long time to get this to work properly in QEMU...
>>
>> I also don't like to re-use MEMOP_LOGICAL_READ and MEMOP_LOGICAL_WRITE
>> for the SIDA like you've had it in RFC v1 ... but what's wrong with
>> using KVM_S390_MEMOP_SIDA_READ and KVM_S390_MEMOP_SIDA_WRITE with the
>> MEM_OP ioctl directly?
>>
>>  Thomas
>>
> 
> In essence something like the following?
> 
> @@ -4583,6 +4618,9 @@ static long kvm_s390_guest_mem_op(struct kvm_vcpu *vcpu,
>                 }
>                 r = write_guest(vcpu, mop->gaddr, mop->ar, tmpbuf, mop->size);
>                 break;
> +       case KVM_S390_MEMOP_SIDA_READ:
> +       case KVM_S390_MEMOP_SIDA_WRITE:
> +               kvm_s390_guest_sida_op(vcpu, mop);
>         default:

FWIW, I'd prefer that. It's a special type of memory access.


-- 
Thanks,

David / dhildenb

Re: [RFCv2 21/37] KVM: S390: protvirt: Introduce instruction data area bounce buffer

[Janosch Frank]

[-- Attachment #1.1: Type: text/plain, Size: 3019 bytes --]

On 2/6/20 10:20 AM, Christian Borntraeger wrote:
> 
> 
> On 06.02.20 10:07, Christian Borntraeger wrote:
>> On 05.02.20 18:00, Thomas Huth wrote:
>>
>>>>>
>>>>> Uh, why the mix of a new ioctl with the existing mem_op stuff? Could you
>>>>> please either properly integrate this into the MEM_OP ioctl (and e.g.
>>>>> use gaddr as offset for the new SIDA_READ and SIDA_WRITE subcodes), or
>>>>> completely separate it for a new ioctl, i.e. introduce a new struct for
>>>>> the new ioctl instead of recycling the struct kvm_s390_mem_op here?
>>>>> (and in case you ask me, I'd slightly prefer to integrate everything
>>>>> into MEM_OP instead of introducing a new ioctl here).
>>>>
>>>> *cough* David and Christian didn't like the memop solution and it took
>>>> me a long time to get this to work properly in QEMU...
>>>
>>> I also don't like to re-use MEMOP_LOGICAL_READ and MEMOP_LOGICAL_WRITE
>>> for the SIDA like you've had it in RFC v1 ... but what's wrong with
>>> using KVM_S390_MEMOP_SIDA_READ and KVM_S390_MEMOP_SIDA_WRITE with the
>>> MEM_OP ioctl directly?
>>>
>>>  Thomas
>>>
>>
>> In essence something like the following?
>>
>> @@ -4583,6 +4618,9 @@ static long kvm_s390_guest_mem_op(struct kvm_vcpu *vcpu,
>>                 }
>>                 r = write_guest(vcpu, mop->gaddr, mop->ar, tmpbuf, mop->size);
>>                 break;
>> +       case KVM_S390_MEMOP_SIDA_READ:
>> +       case KVM_S390_MEMOP_SIDA_WRITE:
>> +               kvm_s390_guest_sida_op(vcpu, mop);
> 
> a break; here
> 
>>         default:
>>                 r = -EINVAL;
>>         }
>>
>>
>> diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
>> index ea2b4d66e0c3..6e029753c955 100644
>> --- a/include/uapi/linux/kvm.h
>> +++ b/include/uapi/linux/kvm.h
>> @@ -1519,7 +1519,6 @@ struct kvm_pv_cmd {
>>  /* Available with KVM_CAP_S390_PROTECTED */
>>  #define KVM_S390_PV_COMMAND            _IOW(KVMIO, 0xc5, struct kvm_pv_cmd)
>>  #define KVM_S390_PV_COMMAND_VCPU       _IOW(KVMIO, 0xc6, struct kvm_pv_cmd)
>> -#define KVM_S390_SIDA_OP               _IOW(KVMIO, 0xc7, struct kvm_s390_mem_op)
>>  
>>  /* Secure Encrypted Virtualization command */
>>  enum sev_cmd_id {
>>
> 
> 
> Janosch, I just checked your qemu tree
> the change would be just the following when we go down that path. (and it makes perfect sense)

Well, it's one less new ioctl and I was worried about the amount of new
ioctls anyway...

> 
> 
> 
> diff --git a/target/s390x/kvm.c b/target/s390x/kvm.c
> index 42d6ef5fad..29bcdf839f 100644
> --- a/target/s390x/kvm.c
> +++ b/target/s390x/kvm.c
> @@ -862,7 +862,7 @@ int kvm_s390_mem_op_pv(S390CPU *cpu, uint64_t offset, void *hostbuf,
>          return -ENOSYS;
>      }
>  
> -    ret = kvm_vcpu_ioctl(CPU(cpu), KVM_S390_SIDA_OP, &mem_op);
> +    ret = kvm_vcpu_ioctl(CPU(cpu), KVM_S390_MEM_OP, &mem_op);
>      if (ret < 0) {
>          warn_report("KVM_S390_MEM_OP failed: %s", strerror(-ret));
>      }
> 



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[RFCv2.1] KVM: S390: protvirt: Introduce instruction data area bounce buffer

[Christian Borntraeger]

From: Janosch Frank <frankja@linux.ibm.com>

Now that we can't access guest memory anymore, we have a dedicated
sattelite block that's a bounce buffer for instruction data.

We re-use the memop interface to copy the instruction data to / from
userspace. This lets us re-use a lot of QEMU code which used that
interface to make logical guest memory accesses which are not possible
anymore in protected mode anyway.

Signed-off-by: Janosch Frank <frankja@linux.ibm.com>
---
 arch/s390/include/asm/kvm_host.h | 11 ++++++-
 arch/s390/kvm/kvm-s390.c         | 49 ++++++++++++++++++++++++++++++++
 arch/s390/kvm/pv.c               |  9 ++++++
 include/uapi/linux/kvm.h         | 10 +++++--
 4 files changed, 76 insertions(+), 3 deletions(-)

diff --git a/arch/s390/include/asm/kvm_host.h b/arch/s390/include/asm/kvm_host.h
index 9d7b248dcadc..2fe8d3c81951 100644
--- a/arch/s390/include/asm/kvm_host.h
+++ b/arch/s390/include/asm/kvm_host.h
@@ -127,6 +127,12 @@ struct mcck_volatile_info {
 #define CR14_INITIAL_MASK (CR14_UNUSED_32 | CR14_UNUSED_33 | \
 			   CR14_EXTERNAL_DAMAGE_SUBMASK)
 
+#define SIDAD_SIZE_MASK		0xff
+#define sida_origin(sie_block) \
+	(sie_block->sidad & PAGE_MASK)
+#define sida_size(sie_block) \
+	(((sie_block->sidad & SIDAD_SIZE_MASK) + 1) * PAGE_SIZE)
+
 #define CPUSTAT_STOPPED    0x80000000
 #define CPUSTAT_WAIT       0x10000000
 #define CPUSTAT_ECALL_PEND 0x08000000
@@ -315,7 +321,10 @@ struct kvm_s390_sie_block {
 #define CRYCB_FORMAT2 0x00000003
 	__u32	crycbd;			/* 0x00fc */
 	__u64	gcr[16];		/* 0x0100 */
-	__u64	gbea;			/* 0x0180 */
+	union {
+		__u64	gbea;			/* 0x0180 */
+		__u64	sidad;
+	};
 	__u8    reserved188[8];		/* 0x0188 */
 	__u64   sdnxo;			/* 0x0190 */
 	__u8    reserved198[8];		/* 0x0198 */
diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
index 6f90d16cad92..56488f9ed190 100644
--- a/arch/s390/kvm/kvm-s390.c
+++ b/arch/s390/kvm/kvm-s390.c
@@ -4435,6 +4435,41 @@ static int kvm_vcpu_ioctl_enable_cap(struct kvm_vcpu *vcpu,
 	return r;
 }
 
+static long kvm_s390_guest_sida_op(struct kvm_vcpu *vcpu,
+				   struct kvm_s390_mem_op *mop)
+{
+	void __user *uaddr = (void __user *)mop->buf;
+	int r = 0;
+
+	if (mop->flags || !mop->size)
+		return -EINVAL;
+
+	if (mop->size > sida_size(vcpu->arch.sie_block))
+		return -E2BIG;
+
+	if (mop->sida_offset > sida_size(vcpu->arch.sie_block))
+		return -E2BIG;
+
+	if (mop->size + mop->sida_offset > sida_size(vcpu->arch.sie_block))
+		return -E2BIG;
+
+	switch (mop->op) {
+	case KVM_S390_MEMOP_SIDA_READ:
+		r = 0;
+		if (copy_to_user(uaddr, (void *)(sida_origin(vcpu->arch.sie_block) +
+				 mop->sida_offset), mop->size))
+			r = -EFAULT;
+
+		break;
+	case KVM_S390_MEMOP_SIDA_WRITE:
+		r = 0;
+		if (copy_from_user((void *)(sida_origin(vcpu->arch.sie_block) +
+				   mop->sida_offset), uaddr, mop->size))
+			r = -EFAULT;
+		break;
+	}
+	return r;
+}
 static long kvm_s390_guest_mem_op(struct kvm_vcpu *vcpu,
 				  struct kvm_s390_mem_op *mop)
 {
@@ -4444,6 +4479,8 @@ static long kvm_s390_guest_mem_op(struct kvm_vcpu *vcpu,
 	const u64 supported_flags = KVM_S390_MEMOP_F_INJECT_EXCEPTION
 				    | KVM_S390_MEMOP_F_CHECK_ONLY;
 
+
+	BUILD_BUG_ON(sizeof(*mop) != 64);
 	if (mop->flags & ~supported_flags || mop->ar >= NUM_ACRS || !mop->size)
 		return -EINVAL;
 
@@ -4460,6 +4497,10 @@ static long kvm_s390_guest_mem_op(struct kvm_vcpu *vcpu,
 
 	switch (mop->op) {
 	case KVM_S390_MEMOP_LOGICAL_READ:
+		if (kvm_s390_pv_is_protected(vcpu->kvm)) {
+			r = -EINVAL;
+			break;
+		}
 		if (mop->flags & KVM_S390_MEMOP_F_CHECK_ONLY) {
 			r = check_gva_range(vcpu, mop->gaddr, mop->ar,
 					    mop->size, GACC_FETCH);
@@ -4472,6 +4513,10 @@ static long kvm_s390_guest_mem_op(struct kvm_vcpu *vcpu,
 		}
 		break;
 	case KVM_S390_MEMOP_LOGICAL_WRITE:
+		if (kvm_s390_pv_is_protected(vcpu->kvm)) {
+			r = -EINVAL;
+			break;
+		}
 		if (mop->flags & KVM_S390_MEMOP_F_CHECK_ONLY) {
 			r = check_gva_range(vcpu, mop->gaddr, mop->ar,
 					    mop->size, GACC_STORE);
@@ -4483,6 +4528,10 @@ static long kvm_s390_guest_mem_op(struct kvm_vcpu *vcpu,
 		}
 		r = write_guest(vcpu, mop->gaddr, mop->ar, tmpbuf, mop->size);
 		break;
+	case KVM_S390_MEMOP_SIDA_READ:
+	case KVM_S390_MEMOP_SIDA_WRITE:
+		r = kvm_s390_guest_sida_op(vcpu, mop);
+		break;
 	default:
 		r = -EINVAL;
 	}
diff --git a/arch/s390/kvm/pv.c b/arch/s390/kvm/pv.c
index 05e5ca8eab4f..d6345a0e348c 100644
--- a/arch/s390/kvm/pv.c
+++ b/arch/s390/kvm/pv.c
@@ -93,6 +93,7 @@ int kvm_s390_pv_destroy_cpu(struct kvm_vcpu *vcpu)
 
 	free_pages(vcpu->arch.pv.stor_base,
 		   get_order(uv_info.guest_cpu_stor_len));
+	free_page(sida_origin(vcpu->arch.sie_block));
 	vcpu->arch.sie_block->pv_handle_cpu = 0;
 	vcpu->arch.sie_block->pv_handle_config = 0;
 	memset(&vcpu->arch.pv, 0, sizeof(vcpu->arch.pv));
@@ -122,6 +123,14 @@ int kvm_s390_pv_create_cpu(struct kvm_vcpu *vcpu)
 	uvcb.state_origin = (u64)vcpu->arch.sie_block;
 	uvcb.stor_origin = (u64)vcpu->arch.pv.stor_base;
 
+	/* Alloc Secure Instruction Data Area Designation */
+	vcpu->arch.sie_block->sidad = __get_free_page(GFP_KERNEL | __GFP_ZERO);
+	if (!vcpu->arch.sie_block->sidad) {
+		free_pages(vcpu->arch.pv.stor_base,
+			   get_order(uv_info.guest_cpu_stor_len));
+		return -ENOMEM;
+	}
+
 	rc = uv_call(0, (u64)&uvcb);
 	VCPU_EVENT(vcpu, 3, "PROTVIRT CREATE VCPU: cpu %d handle %llx rc %x rrc %x",
 		   vcpu->vcpu_id, uvcb.cpu_handle, uvcb.header.rc,
diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
index eab741bc12c3..a772771baf9f 100644
--- a/include/uapi/linux/kvm.h
+++ b/include/uapi/linux/kvm.h
@@ -466,7 +466,7 @@ struct kvm_translation {
 	__u8  pad[5];
 };
 
-/* for KVM_S390_MEM_OP */
+/* for KVM_S390_MEM_OP and KVM_S390_SIDA_OP */
 struct kvm_s390_mem_op {
 	/* in */
 	__u64 gaddr;		/* the guest address */
@@ -475,11 +475,17 @@ struct kvm_s390_mem_op {
 	__u32 op;		/* type of operation */
 	__u64 buf;		/* buffer in userspace */
 	__u8 ar;		/* the access register number */
-	__u8 reserved[31];	/* should be set to 0 */
+	__u8 reserved21[3];	/* should be set to 0 */
+	__u32 sida_offset;	/* offset into the sida */
+	__u8 reserved28[24];	/* should be set to 0 */
 };
+
+
 /* types for kvm_s390_mem_op->op */
 #define KVM_S390_MEMOP_LOGICAL_READ	0
 #define KVM_S390_MEMOP_LOGICAL_WRITE	1
+#define KVM_S390_MEMOP_SIDA_READ	2
+#define KVM_S390_MEMOP_SIDA_WRITE	3
 /* flags for kvm_s390_mem_op->flags */
 #define KVM_S390_MEMOP_F_CHECK_ONLY		(1ULL << 0)
 #define KVM_S390_MEMOP_F_INJECT_EXCEPTION	(1ULL << 1)
-- 
2.24.0

Re: [RFCv2 34/37] KVM: s390: protvirt: Add UV debug trace

[Cornelia Huck]

On Mon,  3 Feb 2020 08:19:54 -0500
Christian Borntraeger <borntraeger@de.ibm.com> wrote:

> From: Janosch Frank <frankja@linux.ibm.com>
> 
> Let's have some debug traces which stay around for longer than the
> guest.
> 
> Signed-off-by: Janosch Frank <frankja@linux.ibm.com>
> ---
>  arch/s390/kvm/kvm-s390.c | 10 +++++++++-
>  arch/s390/kvm/kvm-s390.h |  9 +++++++++
>  arch/s390/kvm/pv.c       | 21 +++++++++++++++++++--
>  3 files changed, 37 insertions(+), 3 deletions(-)
> 
> diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
> index 2beb93f0572f..d4dc156e2c3e 100644
> --- a/arch/s390/kvm/kvm-s390.c
> +++ b/arch/s390/kvm/kvm-s390.c
> @@ -221,6 +221,7 @@ static struct kvm_s390_vm_cpu_subfunc kvm_s390_available_subfunc;
>  static struct gmap_notifier gmap_notifier;
>  static struct gmap_notifier vsie_gmap_notifier;
>  debug_info_t *kvm_s390_dbf;
> +debug_info_t *kvm_s390_dbf_uv;
>  
>  /* Section: not file related */
>  int kvm_arch_hardware_enable(void)
> @@ -462,7 +463,13 @@ int kvm_arch_init(void *opaque)
>  	if (!kvm_s390_dbf)
>  		return -ENOMEM;
>  
> -	if (debug_register_view(kvm_s390_dbf, &debug_sprintf_view))
> +	kvm_s390_dbf_uv = debug_register("kvm-uv", 32, 1, 7 * sizeof(long));
> +	if (!kvm_s390_dbf_uv)
> +		return -ENOMEM;

Doesn't that leak kvm_s390_dbf?

> +
> +

One blank line should be enough.

> +	if (debug_register_view(kvm_s390_dbf, &debug_sprintf_view) ||
> +	    debug_register_view(kvm_s390_dbf_uv, &debug_sprintf_view))
>  		goto out;
>  
>  	kvm_s390_cpu_feat_init();
> @@ -489,6 +496,7 @@ void kvm_arch_exit(void)
>  {
>  	kvm_s390_gib_destroy();
>  	debug_unregister(kvm_s390_dbf);
> +	debug_unregister(kvm_s390_dbf_uv);
>  }
>  
>  /* Section: device related */

(...)

> @@ -252,7 +269,7 @@ int kvm_s390_pv_unpack(struct kvm *kvm, unsigned long addr, unsigned long size,
>  		addr += PAGE_SIZE;
>  		tw[1] += PAGE_SIZE;
>  	}
> -	VM_EVENT(kvm, 3, "PROTVIRT VM UNPACK: finished rc %x", rc);
> +	VM_EVENT(kvm, 3, "PROTVIRT VM UNPACK: finished with rc %x", rc);

Can you merge this into the patch that introduces this log entry?

Also, do you want to add logging into the new dbf here as well?

>  	return rc;
>  }
>  

You often seem to log in pairs (into the per-vm dbf and into the new uv
dbf). Would it make sense to introduce a new helper for that, or is
that overkill?

Re: [RFCv2.1] KVM: S390: protvirt: Introduce instruction data area bounce buffer

[David Hildenbrand]

>  #define CPUSTAT_STOPPED    0x80000000
>  #define CPUSTAT_WAIT       0x10000000
>  #define CPUSTAT_ECALL_PEND 0x08000000
> @@ -315,7 +321,10 @@ struct kvm_s390_sie_block {
>  #define CRYCB_FORMAT2 0x00000003
>  	__u32	crycbd;			/* 0x00fc */
>  	__u64	gcr[16];		/* 0x0100 */
> -	__u64	gbea;			/* 0x0180 */
> +	union {
> +		__u64	gbea;			/* 0x0180 */
> +		__u64	sidad;
> +	};
>  	__u8    reserved188[8];		/* 0x0188 */
>  	__u64   sdnxo;			/* 0x0190 */
>  	__u8    reserved198[8];		/* 0x0198 */
> diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
> index 6f90d16cad92..56488f9ed190 100644
> --- a/arch/s390/kvm/kvm-s390.c
> +++ b/arch/s390/kvm/kvm-s390.c
> @@ -4435,6 +4435,41 @@ static int kvm_vcpu_ioctl_enable_cap(struct kvm_vcpu *vcpu,
>  	return r;
>  }
>  
> +static long kvm_s390_guest_sida_op(struct kvm_vcpu *vcpu,
> +				   struct kvm_s390_mem_op *mop)
> +{
> +	void __user *uaddr = (void __user *)mop->buf;
> +	int r = 0;
> +
> +	if (mop->flags || !mop->size)
> +		return -EINVAL;
> +
> +	if (mop->size > sida_size(vcpu->arch.sie_block))
> +		return -E2BIG;
> +
> +	if (mop->sida_offset > sida_size(vcpu->arch.sie_block))
> +		return -E2BIG;
> +
> +	if (mop->size + mop->sida_offset > sida_size(vcpu->arch.sie_block))
> +		return -E2BIG;
> +

if (mop->size + mop->sida_offset > mop->size)
	return -EINVAL;
if (mop->size + mop->sida_offset > sida_size(vcpu->arch.sie_block))
	return -E2BIG;

Should be sufficient (instead of 3 checks).

> +	switch (mop->op) {
> +	case KVM_S390_MEMOP_SIDA_READ:
> +		r = 0;
> +		if (copy_to_user(uaddr, (void *)(sida_origin(vcpu->arch.sie_block) +
> +				 mop->sida_offset), mop->size))
> +			r = -EFAULT;
> +
> +		break;
> +	case KVM_S390_MEMOP_SIDA_WRITE:
> +		r = 0;
> +		if (copy_from_user((void *)(sida_origin(vcpu->arch.sie_block) +
> +				   mop->sida_offset), uaddr, mop->size))
> +			r = -EFAULT;
> +		break;
> +	}
> +	return r;
> +}
>  static long kvm_s390_guest_mem_op(struct kvm_vcpu *vcpu,
>  				  struct kvm_s390_mem_op *mop)
>  {
> @@ -4444,6 +4479,8 @@ static long kvm_s390_guest_mem_op(struct kvm_vcpu *vcpu,
>  	const u64 supported_flags = KVM_S390_MEMOP_F_INJECT_EXCEPTION
>  				    | KVM_S390_MEMOP_F_CHECK_ONLY;
>  
> +
> +	BUILD_BUG_ON(sizeof(*mop) != 64);
>  	if (mop->flags & ~supported_flags || mop->ar >= NUM_ACRS || !mop->size)
>  		return -EINVAL;
>  
> @@ -4460,6 +4497,10 @@ static long kvm_s390_guest_mem_op(struct kvm_vcpu *vcpu,
>  
>  	switch (mop->op) {
>  	case KVM_S390_MEMOP_LOGICAL_READ:
> +		if (kvm_s390_pv_is_protected(vcpu->kvm)) {
> +			r = -EINVAL;
> +			break;
> +		}

Race with PV_VM_DESTROY (freeing sidad)

>  		if (mop->flags & KVM_S390_MEMOP_F_CHECK_ONLY) {
>  			r = check_gva_range(vcpu, mop->gaddr, mop->ar,
>  					    mop->size, GACC_FETCH);
> @@ -4472,6 +4513,10 @@ static long kvm_s390_guest_mem_op(struct kvm_vcpu *vcpu,
>  		}
>  		break;
>  	case KVM_S390_MEMOP_LOGICAL_WRITE:
> +		if (kvm_s390_pv_is_protected(vcpu->kvm)) {
> +			r = -EINVAL;
> +			break;
> +		}

Race with PV_VM_DESTROY (freeing sidad)


-- 
Thanks,

David / dhildenb

Re: [RFCv2.1] KVM: S390: protvirt: Introduce instruction data area bounce buffer

[Christian Borntraeger]

On 06.02.20 10:54, David Hildenbrand wrote:
>>  #define CPUSTAT_STOPPED    0x80000000
>>  #define CPUSTAT_WAIT       0x10000000
>>  #define CPUSTAT_ECALL_PEND 0x08000000
>> @@ -315,7 +321,10 @@ struct kvm_s390_sie_block {
>>  #define CRYCB_FORMAT2 0x00000003
>>  	__u32	crycbd;			/* 0x00fc */
>>  	__u64	gcr[16];		/* 0x0100 */
>> -	__u64	gbea;			/* 0x0180 */
>> +	union {
>> +		__u64	gbea;			/* 0x0180 */
>> +		__u64	sidad;
>> +	};
>>  	__u8    reserved188[8];		/* 0x0188 */
>>  	__u64   sdnxo;			/* 0x0190 */
>>  	__u8    reserved198[8];		/* 0x0198 */
>> diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
>> index 6f90d16cad92..56488f9ed190 100644
>> --- a/arch/s390/kvm/kvm-s390.c
>> +++ b/arch/s390/kvm/kvm-s390.c
>> @@ -4435,6 +4435,41 @@ static int kvm_vcpu_ioctl_enable_cap(struct kvm_vcpu *vcpu,
>>  	return r;
>>  }
>>  
>> +static long kvm_s390_guest_sida_op(struct kvm_vcpu *vcpu,
>> +				   struct kvm_s390_mem_op *mop)
>> +{
>> +	void __user *uaddr = (void __user *)mop->buf;
>> +	int r = 0;
>> +
>> +	if (mop->flags || !mop->size)
>> +		return -EINVAL;
>> +
>> +	if (mop->size > sida_size(vcpu->arch.sie_block))
>> +		return -E2BIG;
>> +
>> +	if (mop->sida_offset > sida_size(vcpu->arch.sie_block))
>> +		return -E2BIG;
>> +
>> +	if (mop->size + mop->sida_offset > sida_size(vcpu->arch.sie_block))
>> +		return -E2BIG;
>> +
> 
> if (mop->size + mop->sida_offset > mop->size)
> 	return -EINVAL;
> if (mop->size + mop->sida_offset > sida_size(vcpu->arch.sie_block))
> 	return -E2BIG;
> 
> Should be sufficient (instead of 3 checks).

Will have a look.

[..]
>>  
>>  	switch (mop->op) {
>>  	case KVM_S390_MEMOP_LOGICAL_READ:
>> +		if (kvm_s390_pv_is_protected(vcpu->kvm)) {
>> +			r = -EINVAL;
>> +			break;
>> +		}
> 
> Race with PV_VM_DESTROY (freeing sidad)

As I said in my other mail, the sida is freed in PV_CPU_DESTROY and this ioctl is locked against MEMOP.

Re: [RFCv2.1] KVM: S390: protvirt: Introduce instruction data area bounce buffer

[David Hildenbrand]

>>>  
>>>  	switch (mop->op) {
>>>  	case KVM_S390_MEMOP_LOGICAL_READ:
>>> +		if (kvm_s390_pv_is_protected(vcpu->kvm)) {
>>> +			r = -EINVAL;
>>> +			break;
>>> +		}
>>
>> Race with PV_VM_DESTROY (freeing sidad)
> 
> As I said in my other mail, the sida is freed in PV_CPU_DESTROY and this ioctl is locked against MEMOP.
> 

Missed that, thanks!

-- 
Thanks,

David / dhildenb

Re: [RFCv2 35/37] KVM: s390: protvirt: Mask PSW interrupt bits for interception 104 and 112

[Cornelia Huck]

On Mon,  3 Feb 2020 08:19:55 -0500
Christian Borntraeger <borntraeger@de.ibm.com> wrote:

> From: Janosch Frank <frankja@linux.ibm.com>
> 
> We're not allowed to inject interrupts on those intercept codes. As our

Spell out what these codes actually are?

> PSW is just a copy of the real one that will be replaced on the next
> exit, we can mask out the interrupt bits in the PSW to make sure that we
> do not inject anything.
> 
> Signed-off-by: Janosch Frank <frankja@linux.ibm.com>
> ---
>  arch/s390/kvm/kvm-s390.c | 6 ++++++
>  1 file changed, 6 insertions(+)
> 
> diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
> index d4dc156e2c3e..137ae5dc9101 100644
> --- a/arch/s390/kvm/kvm-s390.c
> +++ b/arch/s390/kvm/kvm-s390.c
> @@ -4050,6 +4050,7 @@ static int vcpu_post_run(struct kvm_vcpu *vcpu, int exit_reason)
>  	return vcpu_post_run_fault_in_sie(vcpu);
>  }
>  
> +#define PSW_INT_MASK (PSW_MASK_EXT | PSW_MASK_IO | PSW_MASK_MCHECK)
>  static int __vcpu_run(struct kvm_vcpu *vcpu)
>  {
>  	int rc, exit_reason;
> @@ -4082,10 +4083,15 @@ static int __vcpu_run(struct kvm_vcpu *vcpu)
>  		}
>  		exit_reason = sie64a(vcpu->arch.sie_block,
>  				     vcpu->run->s.regs.gprs);
> +		/* This will likely be moved into a new function. */

Hm?

>  		if (kvm_s390_pv_is_protected(vcpu->kvm)) {
>  			memcpy(vcpu->run->s.regs.gprs,
>  			       sie_page->pv_grregs,
>  			       sizeof(sie_page->pv_grregs));
> +			if (vcpu->arch.sie_block->icptcode == ICPT_PV_INSTR ||
> +			    vcpu->arch.sie_block->icptcode == ICPT_PV_PREF) {
> +				vcpu->arch.sie_block->gpsw.mask &= ~PSW_INT_MASK;
> +			}
>  		}
>  		local_irq_disable();
>  		__enable_cpu_timer_accounting(vcpu);

Re: [RFCv2.1] KVM: S390: protvirt: Introduce instruction data area bounce buffer

[Christian Borntraeger]

On 06.02.20 11:08, David Hildenbrand wrote:
> 
>>>>  
>>>>  	switch (mop->op) {
>>>>  	case KVM_S390_MEMOP_LOGICAL_READ:
>>>> +		if (kvm_s390_pv_is_protected(vcpu->kvm)) {
>>>> +			r = -EINVAL;
>>>> +			break;
>>>> +		}
>>>
>>> Race with PV_VM_DESTROY (freeing sidad)
>>
>> As I said in my other mail, the sida is freed in PV_CPU_DESTROY and this ioctl is locked against MEMOP.
>>
> 
> Missed that, thanks!

will add a comment. (and also use your checks as we only need 2 )

Re: [RFCv2 36/37] KVM: s390: protvirt: do not inject interrupts after start

[Cornelia Huck]

On Mon,  3 Feb 2020 08:19:56 -0500
Christian Borntraeger <borntraeger@de.ibm.com> wrote:

> As PSW restart is handled by the ultravisor (and we only get a start
> notification) we must re-check the PSW after a start before injecting
> interrupts.
> 
> Signed-off-by: Christian Borntraeger <borntraeger@de.ibm.com>
> ---
>  arch/s390/kvm/kvm-s390.c | 6 ++++++
>  1 file changed, 6 insertions(+)
> 
> diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
> index 137ae5dc9101..3e4716b3fc02 100644
> --- a/arch/s390/kvm/kvm-s390.c
> +++ b/arch/s390/kvm/kvm-s390.c
> @@ -4451,6 +4451,12 @@ void kvm_s390_vcpu_start(struct kvm_vcpu *vcpu)
>  	/* Let's tell the UV that we want to start again */
>  	kvm_s390_pv_set_cpu_state(vcpu, PV_CPU_STATE_OPR);
>  	kvm_s390_clear_cpuflags(vcpu, CPUSTAT_STOPPED);
> +	/*
> +	 * The real PSW might have changed due to an interpreted RESTART.

"a RESTART interpreted by the ultravisor" ?

> +	 * We block all interrupts and let the next sie exit refresh our view.
> +	 */
> +	if (kvm_s390_pv_is_protected(vcpu->kvm))
> +		vcpu->arch.sie_block->gpsw.mask &= ~PSW_INT_MASK;
>  	/*
>  	 * Another VCPU might have used IBS while we were offline.
>  	 * Let's play safe and flush the VCPU at startup.

Reviewed-by: Cornelia Huck <cohuck@redhat.com>

Re: [RFCv2.1] KVM: S390: protvirt: Introduce instruction data area bounce buffer

[Thomas Huth]

On 06/02/2020 10.39, Christian Borntraeger wrote:
> From: Janosch Frank <frankja@linux.ibm.com>
> 
> Now that we can't access guest memory anymore, we have a dedicated
> sattelite block that's a bounce buffer for instruction data.

s/sattelite/satellite/

> We re-use the memop interface to copy the instruction data to / from
> userspace. This lets us re-use a lot of QEMU code which used that
> interface to make logical guest memory accesses which are not possible
> anymore in protected mode anyway.
> 
> Signed-off-by: Janosch Frank <frankja@linux.ibm.com>
> ---
>  arch/s390/include/asm/kvm_host.h | 11 ++++++-
>  arch/s390/kvm/kvm-s390.c         | 49 ++++++++++++++++++++++++++++++++
>  arch/s390/kvm/pv.c               |  9 ++++++
>  include/uapi/linux/kvm.h         | 10 +++++--
>  4 files changed, 76 insertions(+), 3 deletions(-)
> 
> diff --git a/arch/s390/include/asm/kvm_host.h b/arch/s390/include/asm/kvm_host.h
> index 9d7b248dcadc..2fe8d3c81951 100644
> --- a/arch/s390/include/asm/kvm_host.h
> +++ b/arch/s390/include/asm/kvm_host.h
> @@ -127,6 +127,12 @@ struct mcck_volatile_info {
>  #define CR14_INITIAL_MASK (CR14_UNUSED_32 | CR14_UNUSED_33 | \
>  			   CR14_EXTERNAL_DAMAGE_SUBMASK)
>  
> +#define SIDAD_SIZE_MASK		0xff
> +#define sida_origin(sie_block) \
> +	(sie_block->sidad & PAGE_MASK)
> +#define sida_size(sie_block) \
> +	(((sie_block->sidad & SIDAD_SIZE_MASK) + 1) * PAGE_SIZE)
> +
>  #define CPUSTAT_STOPPED    0x80000000
>  #define CPUSTAT_WAIT       0x10000000
>  #define CPUSTAT_ECALL_PEND 0x08000000
> @@ -315,7 +321,10 @@ struct kvm_s390_sie_block {
>  #define CRYCB_FORMAT2 0x00000003
>  	__u32	crycbd;			/* 0x00fc */
>  	__u64	gcr[16];		/* 0x0100 */
> -	__u64	gbea;			/* 0x0180 */
> +	union {
> +		__u64	gbea;			/* 0x0180 */

Maybe adjust the spaces before the comment.

> +		__u64	sidad;
> +	};
>  	__u8    reserved188[8];		/* 0x0188 */
>  	__u64   sdnxo;			/* 0x0190 */
>  	__u8    reserved198[8];		/* 0x0198 */
> diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
> index 6f90d16cad92..56488f9ed190 100644
> --- a/arch/s390/kvm/kvm-s390.c
> +++ b/arch/s390/kvm/kvm-s390.c
> @@ -4435,6 +4435,41 @@ static int kvm_vcpu_ioctl_enable_cap(struct kvm_vcpu *vcpu,
>  	return r;
>  }
>  
> +static long kvm_s390_guest_sida_op(struct kvm_vcpu *vcpu,
> +				   struct kvm_s390_mem_op *mop)
> +{
> +	void __user *uaddr = (void __user *)mop->buf;
> +	int r = 0;
> +
> +	if (mop->flags || !mop->size)
> +		return -EINVAL;
> +
> +	if (mop->size > sida_size(vcpu->arch.sie_block))
> +		return -E2BIG;
> +
> +	if (mop->sida_offset > sida_size(vcpu->arch.sie_block))
> +		return -E2BIG;
> +
> +	if (mop->size + mop->sida_offset > sida_size(vcpu->arch.sie_block))
> +		return -E2BIG;
> +
> +	switch (mop->op) {
> +	case KVM_S390_MEMOP_SIDA_READ:
> +		r = 0;

r is alread pre-initialized with 0 where it is declared, so you could
remove the above line.

> +		if (copy_to_user(uaddr, (void *)(sida_origin(vcpu->arch.sie_block) +
> +				 mop->sida_offset), mop->size))
> +			r = -EFAULT;
> +
> +		break;
> +	case KVM_S390_MEMOP_SIDA_WRITE:
> +		r = 0;

dito.

> +		if (copy_from_user((void *)(sida_origin(vcpu->arch.sie_block) +
> +				   mop->sida_offset), uaddr, mop->size))
> +			r = -EFAULT;
> +		break;
> +	}
> +	return r;
> +}
>  static long kvm_s390_guest_mem_op(struct kvm_vcpu *vcpu,
>  				  struct kvm_s390_mem_op *mop)
>  {
> @@ -4444,6 +4479,8 @@ static long kvm_s390_guest_mem_op(struct kvm_vcpu *vcpu,
>  	const u64 supported_flags = KVM_S390_MEMOP_F_INJECT_EXCEPTION
>  				    | KVM_S390_MEMOP_F_CHECK_ONLY;
>  
> +
> +	BUILD_BUG_ON(sizeof(*mop) != 64);
>  	if (mop->flags & ~supported_flags || mop->ar >= NUM_ACRS || !mop->size)
>  		return -EINVAL;
>  
> @@ -4460,6 +4497,10 @@ static long kvm_s390_guest_mem_op(struct kvm_vcpu *vcpu,
>  
>  	switch (mop->op) {
>  	case KVM_S390_MEMOP_LOGICAL_READ:
> +		if (kvm_s390_pv_is_protected(vcpu->kvm)) {
> +			r = -EINVAL;
> +			break;
> +		}
>  		if (mop->flags & KVM_S390_MEMOP_F_CHECK_ONLY) {
>  			r = check_gva_range(vcpu, mop->gaddr, mop->ar,
>  					    mop->size, GACC_FETCH);
> @@ -4472,6 +4513,10 @@ static long kvm_s390_guest_mem_op(struct kvm_vcpu *vcpu,
>  		}
>  		break;
>  	case KVM_S390_MEMOP_LOGICAL_WRITE:
> +		if (kvm_s390_pv_is_protected(vcpu->kvm)) {
> +			r = -EINVAL;
> +			break;
> +		}
>  		if (mop->flags & KVM_S390_MEMOP_F_CHECK_ONLY) {
>  			r = check_gva_range(vcpu, mop->gaddr, mop->ar,
>  					    mop->size, GACC_STORE);
> @@ -4483,6 +4528,10 @@ static long kvm_s390_guest_mem_op(struct kvm_vcpu *vcpu,
>  		}
>  		r = write_guest(vcpu, mop->gaddr, mop->ar, tmpbuf, mop->size);
>  		break;
> +	case KVM_S390_MEMOP_SIDA_READ:
> +	case KVM_S390_MEMOP_SIDA_WRITE:
> +		r = kvm_s390_guest_sida_op(vcpu, mop);
> +		break;
>  	default:
>  		r = -EINVAL;
>  	}
> diff --git a/arch/s390/kvm/pv.c b/arch/s390/kvm/pv.c
> index 05e5ca8eab4f..d6345a0e348c 100644
> --- a/arch/s390/kvm/pv.c
> +++ b/arch/s390/kvm/pv.c
> @@ -93,6 +93,7 @@ int kvm_s390_pv_destroy_cpu(struct kvm_vcpu *vcpu)
>  
>  	free_pages(vcpu->arch.pv.stor_base,
>  		   get_order(uv_info.guest_cpu_stor_len));
> +	free_page(sida_origin(vcpu->arch.sie_block));
>  	vcpu->arch.sie_block->pv_handle_cpu = 0;
>  	vcpu->arch.sie_block->pv_handle_config = 0;
>  	memset(&vcpu->arch.pv, 0, sizeof(vcpu->arch.pv));
> @@ -122,6 +123,14 @@ int kvm_s390_pv_create_cpu(struct kvm_vcpu *vcpu)
>  	uvcb.state_origin = (u64)vcpu->arch.sie_block;
>  	uvcb.stor_origin = (u64)vcpu->arch.pv.stor_base;
>  
> +	/* Alloc Secure Instruction Data Area Designation */
> +	vcpu->arch.sie_block->sidad = __get_free_page(GFP_KERNEL | __GFP_ZERO);
> +	if (!vcpu->arch.sie_block->sidad) {
> +		free_pages(vcpu->arch.pv.stor_base,
> +			   get_order(uv_info.guest_cpu_stor_len));
> +		return -ENOMEM;
> +	}
> +
>  	rc = uv_call(0, (u64)&uvcb);
>  	VCPU_EVENT(vcpu, 3, "PROTVIRT CREATE VCPU: cpu %d handle %llx rc %x rrc %x",
>  		   vcpu->vcpu_id, uvcb.cpu_handle, uvcb.header.rc,
> diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
> index eab741bc12c3..a772771baf9f 100644
> --- a/include/uapi/linux/kvm.h
> +++ b/include/uapi/linux/kvm.h
> @@ -466,7 +466,7 @@ struct kvm_translation {
>  	__u8  pad[5];
>  };
>  
> -/* for KVM_S390_MEM_OP */
> +/* for KVM_S390_MEM_OP and KVM_S390_SIDA_OP */

Remove this change now, please.

>  struct kvm_s390_mem_op {
>  	/* in */
>  	__u64 gaddr;		/* the guest address */
> @@ -475,11 +475,17 @@ struct kvm_s390_mem_op {
>  	__u32 op;		/* type of operation */
>  	__u64 buf;		/* buffer in userspace */
>  	__u8 ar;		/* the access register number */
> -	__u8 reserved[31];	/* should be set to 0 */
> +	__u8 reserved21[3];	/* should be set to 0 */
> +	__u32 sida_offset;	/* offset into the sida */
> +	__u8 reserved28[24];	/* should be set to 0 */
>  };
> +
> +
>  /* types for kvm_s390_mem_op->op */
>  #define KVM_S390_MEMOP_LOGICAL_READ	0
>  #define KVM_S390_MEMOP_LOGICAL_WRITE	1
> +#define KVM_S390_MEMOP_SIDA_READ	2
> +#define KVM_S390_MEMOP_SIDA_WRITE	3
>  /* flags for kvm_s390_mem_op->flags */
>  #define KVM_S390_MEMOP_F_CHECK_ONLY		(1ULL << 0)
>  #define KVM_S390_MEMOP_F_INJECT_EXCEPTION	(1ULL << 1)

With the nits fixed:
Reviewed-by: Thomas Huth <thuth@redhat.com>

Re: [RFCv2 37/37] KVM: s390: protvirt: Add UV cpu reset calls

[Cornelia Huck]

On Mon,  3 Feb 2020 08:19:57 -0500
Christian Borntraeger <borntraeger@de.ibm.com> wrote:

> From: Janosch Frank <frankja@linux.ibm.com>
> 
> For protected VMs, the VCPU resets are done by the Ultravisor, as KVM
> has no access to the VCPU registers.
> 
> As the Ultravisor will only accept a call for the reset that is
> needed, we need to fence the UV calls when chaining resets.

Stale note, chaining resets is gone?

"Note that the ultravisor will only accept a call for the exact reset
that has been requested." ?

> 
> Signed-off-by: Janosch Frank <frankja@linux.ibm.com>
> ---
>  arch/s390/kvm/kvm-s390.c | 20 ++++++++++++++++++++
>  1 file changed, 20 insertions(+)
> 
> diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
> index 3e4716b3fc02..f7a3f84be064 100644
> --- a/arch/s390/kvm/kvm-s390.c
> +++ b/arch/s390/kvm/kvm-s390.c
> @@ -4699,6 +4699,7 @@ long kvm_arch_vcpu_ioctl(struct file *filp,
>  	void __user *argp = (void __user *)arg;
>  	int idx;
>  	long r;
> +	u32 ret;
>  
>  	vcpu_load(vcpu);
>  
> @@ -4720,14 +4721,33 @@ long kvm_arch_vcpu_ioctl(struct file *filp,
>  	case KVM_S390_CLEAR_RESET:
>  		r = 0;
>  		kvm_arch_vcpu_ioctl_clear_reset(vcpu);
> +		if (kvm_s390_pv_handle_cpu(vcpu)) {
> +			r = uv_cmd_nodata(kvm_s390_pv_handle_cpu(vcpu),
> +					  UVC_CMD_CPU_RESET_CLEAR, &ret);
> +			VCPU_EVENT(vcpu, 3, "PROTVIRT RESET CLEAR VCPU: cpu %d rc %x rrc %x",
> +				   vcpu->vcpu_id, ret >> 16, ret & 0x0000ffff);

I think VCPU_EVENT() already traces the vcpu_id, doesn't it?

> +		}
>  		break;
>  	case KVM_S390_INITIAL_RESET:
>  		r = 0;
>  		kvm_arch_vcpu_ioctl_initial_reset(vcpu);
> +		if (kvm_s390_pv_handle_cpu(vcpu)) {
> +			r = uv_cmd_nodata(kvm_s390_pv_handle_cpu(vcpu),
> +					  UVC_CMD_CPU_RESET_INITIAL,
> +					  &ret);
> +			VCPU_EVENT(vcpu, 3, "PROTVIRT RESET INITIAL VCPU: cpu %d rc %x rrc %x",
> +				   vcpu->vcpu_id, ret >> 16, ret & 0x0000ffff);
> +		}
>  		break;
>  	case KVM_S390_NORMAL_RESET:
>  		r = 0;
>  		kvm_arch_vcpu_ioctl_normal_reset(vcpu);
> +		if (kvm_s390_pv_handle_cpu(vcpu)) {
> +			r = uv_cmd_nodata(kvm_s390_pv_handle_cpu(vcpu),
> +					  UVC_CMD_CPU_RESET, &ret);
> +			VCPU_EVENT(vcpu, 3, "PROTVIRT RESET NORMAL VCPU: cpu %d rc %x rrc %x",
> +				   vcpu->vcpu_id, ret >> 16, ret & 0x0000ffff);
> +		}
>  		break;
>  	case KVM_SET_ONE_REG:
>  	case KVM_GET_ONE_REG: {

Otherwise, looks good.

Re: [RFCv2 30/37] KVM: s390: protvirt: Add diag 308 subcode 8 - 10 handling

[Christian Borntraeger]

On 05.02.20 19:21, Thomas Huth wrote:
> On 03/02/2020 14.19, Christian Borntraeger wrote:
>> From: Janosch Frank <frankja@linux.ibm.com>
>>
>> If the host initialized the Ultravisor, we can set stfle bit 161
>> (protected virtual IPL enhancements facility), which indicates, that
>> the IPL subcodes 8, 9 and are valid. These subcodes are used by a
>> normal guest to set/retrieve a IPIB of type 5 and transition into
>> protected mode.
>>
>> Once in protected mode, the Ultravisor will conceal the facility
>> bit. Therefore each boot into protected mode has to go through
>> non-protected. There is no secure re-ipl with subcode 10 without a
>> previous subcode 3.
>>
>> In protected mode, there is no subcode 4 available, as the VM has no
>> more access to its memory from non-protected mode. I.e. each IPL
>> clears.
>>
>> Signed-off-by: Janosch Frank <frankja@linux.ibm.com>
>> ---
>>  arch/s390/kvm/diag.c     | 6 ++++++
>>  arch/s390/kvm/kvm-s390.c | 5 +++++
>>  2 files changed, 11 insertions(+)
>>
>> diff --git a/arch/s390/kvm/diag.c b/arch/s390/kvm/diag.c
>> index 3fb54ec2cf3e..b951dbdcb6a0 100644
>> --- a/arch/s390/kvm/diag.c
>> +++ b/arch/s390/kvm/diag.c
>> @@ -197,6 +197,12 @@ static int __diag_ipl_functions(struct kvm_vcpu *vcpu)
>>  	case 4:
>>  		vcpu->run->s390_reset_flags = 0;
>>  		break;
>> +	case 8:
>> +	case 9:
>> +	case 10:
>> +		if (!test_kvm_facility(vcpu->kvm, 161))
>> +			return kvm_s390_inject_program_int(vcpu, PGM_SPECIFICATION);
>> +		/* fall through */
>>  	default:
>>  		return -EOPNOTSUPP;
> 
> If we don't handle these diags in the kernel anyway, why do you care
> about injecting the program interrupt from kernel space? Couldn't that
> be done by userspace instead? I.e. could the whole hunk be dropped?

Agreed. While this is a fast path we can certainly let QEMU handle the
error cases. Will remove and add a comment to the patch description that
userspace will handle the error cases.

Re: [RFCv2.1] KVM: S390: protvirt: Introduce instruction data area bounce buffer

[Christian Borntraeger]

On 06.02.20 11:32, Thomas Huth wrote:
> On 06/02/2020 10.39, Christian Borntraeger wrote:
>> From: Janosch Frank <frankja@linux.ibm.com>
>>
>> Now that we can't access guest memory anymore, we have a dedicated
>> sattelite block that's a bounce buffer for instruction data.
> 
> s/sattelite/satellite/
> 
>> We re-use the memop interface to copy the instruction data to / from
>> userspace. This lets us re-use a lot of QEMU code which used that
>> interface to make logical guest memory accesses which are not possible
>> anymore in protected mode anyway.
>>
>> Signed-off-by: Janosch Frank <frankja@linux.ibm.com>
>> ---
>>  arch/s390/include/asm/kvm_host.h | 11 ++++++-
>>  arch/s390/kvm/kvm-s390.c         | 49 ++++++++++++++++++++++++++++++++
>>  arch/s390/kvm/pv.c               |  9 ++++++
>>  include/uapi/linux/kvm.h         | 10 +++++--
>>  4 files changed, 76 insertions(+), 3 deletions(-)
>>
>> diff --git a/arch/s390/include/asm/kvm_host.h b/arch/s390/include/asm/kvm_host.h
>> index 9d7b248dcadc..2fe8d3c81951 100644
>> --- a/arch/s390/include/asm/kvm_host.h
>> +++ b/arch/s390/include/asm/kvm_host.h
>> @@ -127,6 +127,12 @@ struct mcck_volatile_info {
>>  #define CR14_INITIAL_MASK (CR14_UNUSED_32 | CR14_UNUSED_33 | \
>>  			   CR14_EXTERNAL_DAMAGE_SUBMASK)
>>  
>> +#define SIDAD_SIZE_MASK		0xff
>> +#define sida_origin(sie_block) \
>> +	(sie_block->sidad & PAGE_MASK)
>> +#define sida_size(sie_block) \
>> +	(((sie_block->sidad & SIDAD_SIZE_MASK) + 1) * PAGE_SIZE)
>> +
>>  #define CPUSTAT_STOPPED    0x80000000
>>  #define CPUSTAT_WAIT       0x10000000
>>  #define CPUSTAT_ECALL_PEND 0x08000000
>> @@ -315,7 +321,10 @@ struct kvm_s390_sie_block {
>>  #define CRYCB_FORMAT2 0x00000003
>>  	__u32	crycbd;			/* 0x00fc */
>>  	__u64	gcr[16];		/* 0x0100 */
>> -	__u64	gbea;			/* 0x0180 */
>> +	union {
>> +		__u64	gbea;			/* 0x0180 */
> 
> Maybe adjust the spaces before the comment.
> 
>> +		__u64	sidad;
[...]


ack

>> +	switch (mop->op) {
>> +	case KVM_S390_MEMOP_SIDA_READ:
>> +		r = 0;
> 
> r is alread pre-initialized with 0 where it is declared, so you could
> remove the above line.

ack

> 
>> +		if (copy_to_user(uaddr, (void *)(sida_origin(vcpu->arch.sie_block) +
>> +				 mop->sida_offset), mop->size))
>> +			r = -EFAULT;
>> +
>> +		break;
>> +	case KVM_S390_MEMOP_SIDA_WRITE:
>> +		r = 0;
> 
> dito.
> 

ack
[...]

>>  };
>>  
>> -/* for KVM_S390_MEM_OP */
>> +/* for KVM_S390_MEM_OP and KVM_S390_SIDA_OP */
> 
> Remove this change now, please.

ack

> 
>>  struct kvm_s390_mem_op {
>>  	/* in */
>>  	__u64 gaddr;		/* the guest address */
>> @@ -475,11 +475,17 @@ struct kvm_s390_mem_op {
>>  	__u32 op;		/* type of operation */
>>  	__u64 buf;		/* buffer in userspace */
>>  	__u8 ar;		/* the access register number */
>> -	__u8 reserved[31];	/* should be set to 0 */
>> +	__u8 reserved21[3];	/* should be set to 0 */
>> +	__u32 sida_offset;	/* offset into the sida */
>> +	__u8 reserved28[24];	/* should be set to 0 */
>>  };
>> +
>> +
>>  /* types for kvm_s390_mem_op->op */
>>  #define KVM_S390_MEMOP_LOGICAL_READ	0
>>  #define KVM_S390_MEMOP_LOGICAL_WRITE	1
>> +#define KVM_S390_MEMOP_SIDA_READ	2
>> +#define KVM_S390_MEMOP_SIDA_WRITE	3
>>  /* flags for kvm_s390_mem_op->flags */
>>  #define KVM_S390_MEMOP_F_CHECK_ONLY		(1ULL << 0)
>>  #define KVM_S390_MEMOP_F_INJECT_EXCEPTION	(1ULL << 1)
> 
> With the nits fixed:
> Reviewed-by: Thomas Huth <thuth@redhat.com>

thanks

Re: [RFCv2 35/37] KVM: s390: protvirt: Mask PSW interrupt bits for interception 104 and 112

[Christian Borntraeger]

On 06.02.20 11:10, Cornelia Huck wrote:
> On Mon,  3 Feb 2020 08:19:55 -0500
> Christian Borntraeger <borntraeger@de.ibm.com> wrote:
> 
>> From: Janosch Frank <frankja@linux.ibm.com>
>>
>> We're not allowed to inject interrupts on those intercept codes. As our
> 
> Spell out what these codes actually are?

We're not allowed to inject interrupts on intercepts that leave the guest state
in an "in-beetween" state where the next SIE entry will do a continuation.
Namely secure instruction interception and secure prefix interception...

>> @@ -4082,10 +4083,15 @@ static int __vcpu_run(struct kvm_vcpu *vcpu)
>>  		}
>>  		exit_reason = sie64a(vcpu->arch.sie_block,
>>  				     vcpu->run->s.regs.gprs);
>> +		/* This will likely be moved into a new function. */
> 
> Hm?

Unless you have a good naming for such function, will remove this comment.
I tried that but I the end result was not better or worse.

> 
>>  		if (kvm_s390_pv_is_protected(vcpu->kvm)) {
>>  			memcpy(vcpu->run->s.regs.gprs,
>>  			       sie_page->pv_grregs,
>>  			       sizeof(sie_page->pv_grregs));
>> +			if (vcpu->arch.sie_block->icptcode == ICPT_PV_INSTR ||
>> +			    vcpu->arch.sie_block->icptcode == ICPT_PV_PREF) {
>> +				vcpu->arch.sie_block->gpsw.mask &= ~PSW_INT_MASK;
>> +			}
>>  		}
>>  		local_irq_disable();
>>  		__enable_cpu_timer_accounting(vcpu);
> 

Re: [RFCv2 32/37] KVM: s390: protvirt: Report CPU state to Ultravisor

[Thomas Huth]

On 03/02/2020 14.19, Christian Borntraeger wrote:
> From: Janosch Frank <frankja@linux.ibm.com>
> 
> VCPU states have to be reported to the ultravisor for SIGP
> interpretation.
> 
> Signed-off-by: Janosch Frank <frankja@linux.ibm.com>
> ---
>  arch/s390/kvm/kvm-s390.c | 5 ++++-
>  1 file changed, 4 insertions(+), 1 deletion(-)
> 
> diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
> index 3ab5091ded6c..da8aeb24362a 100644
> --- a/arch/s390/kvm/kvm-s390.c
> +++ b/arch/s390/kvm/kvm-s390.c
> @@ -4434,7 +4434,8 @@ void kvm_s390_vcpu_start(struct kvm_vcpu *vcpu)
>  		 */
>  		__disable_ibs_on_all_vcpus(vcpu->kvm);
>  	}
> -
> +	/* Let's tell the UV that we want to start again */
> +	kvm_s390_pv_set_cpu_state(vcpu, PV_CPU_STATE_OPR);
>  	kvm_s390_clear_cpuflags(vcpu, CPUSTAT_STOPPED);
>  	/*
>  	 * Another VCPU might have used IBS while we were offline.
> @@ -4462,6 +4463,8 @@ void kvm_s390_vcpu_stop(struct kvm_vcpu *vcpu)
>  	kvm_s390_clear_stop_irq(vcpu);
>  
>  	kvm_s390_set_cpuflags(vcpu, CPUSTAT_STOPPED);
> +	/* Let's tell the UV that we successfully stopped the vcpu */
> +	kvm_s390_pv_set_cpu_state(vcpu, PV_CPU_STATE_STP);
>  	__disable_ibs_on_vcpu(vcpu);
>  
>  	for (i = 0; i < online_vcpus; i++) {

As already mentioned, I'd move the kvm_s390_pv_set_cpu_state() from the
previous patch to this one here.

Anyway,
Reviewed-by: Thomas Huth <thuth@redhat.com>

Re: [RFCv2 32/37] KVM: s390: protvirt: Report CPU state to Ultravisor

[Christian Borntraeger]

On 06.02.20 12:12, Thomas Huth wrote:
> On 03/02/2020 14.19, Christian Borntraeger wrote:
>> From: Janosch Frank <frankja@linux.ibm.com>
>>
>> VCPU states have to be reported to the ultravisor for SIGP
>> interpretation.
>>
>> Signed-off-by: Janosch Frank <frankja@linux.ibm.com>
>> ---
>>  arch/s390/kvm/kvm-s390.c | 5 ++++-
>>  1 file changed, 4 insertions(+), 1 deletion(-)
>>
>> diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
>> index 3ab5091ded6c..da8aeb24362a 100644
>> --- a/arch/s390/kvm/kvm-s390.c
>> +++ b/arch/s390/kvm/kvm-s390.c
>> @@ -4434,7 +4434,8 @@ void kvm_s390_vcpu_start(struct kvm_vcpu *vcpu)
>>  		 */
>>  		__disable_ibs_on_all_vcpus(vcpu->kvm);
>>  	}
>> -
>> +	/* Let's tell the UV that we want to start again */
>> +	kvm_s390_pv_set_cpu_state(vcpu, PV_CPU_STATE_OPR);
>>  	kvm_s390_clear_cpuflags(vcpu, CPUSTAT_STOPPED);
>>  	/*
>>  	 * Another VCPU might have used IBS while we were offline.
>> @@ -4462,6 +4463,8 @@ void kvm_s390_vcpu_stop(struct kvm_vcpu *vcpu)
>>  	kvm_s390_clear_stop_irq(vcpu);
>>  
>>  	kvm_s390_set_cpuflags(vcpu, CPUSTAT_STOPPED);
>> +	/* Let's tell the UV that we successfully stopped the vcpu */
>> +	kvm_s390_pv_set_cpu_state(vcpu, PV_CPU_STATE_STP);
>>  	__disable_ibs_on_vcpu(vcpu);
>>  
>>  	for (i = 0; i < online_vcpus; i++) {
> 
> As already mentioned, I'd move the kvm_s390_pv_set_cpu_state() from the
> previous patch to this one here.

ack.
> 
> Anyway,
> Reviewed-by: Thomas Huth <thuth@redhat.com>
> 

Re: [RFCv2 33/37] KVM: s390: protvirt: Support cmd 5 operation state

[Thomas Huth]

On 03/02/2020 14.19, Christian Borntraeger wrote:
> From: Janosch Frank <frankja@linux.ibm.com>
> 
> Code 5 for the set cpu state UV call tells the UV to load a PSW from
> the SE header (first IPL) or from guest location 0x0 (diag 308 subcode
> 0/1). Also it sets the cpu into operating state afterwards, so we can
> start it.
> 
> Signed-off-by: Janosch Frank <frankja@linux.ibm.com>
> ---
>  arch/s390/include/asm/uv.h | 1 +
>  arch/s390/kvm/kvm-s390.c   | 7 +++++++
>  include/uapi/linux/kvm.h   | 1 +
>  3 files changed, 9 insertions(+)

Reviewed-by: Thomas Huth <thuth@redhat.com>

Re: [RFCv2 35/37] KVM: s390: protvirt: Mask PSW interrupt bits for interception 104 and 112

[Thomas Huth]

On 03/02/2020 14.19, Christian Borntraeger wrote:
> From: Janosch Frank <frankja@linux.ibm.com>
> 
> We're not allowed to inject interrupts on those intercept codes. As our
> PSW is just a copy of the real one that will be replaced on the next
> exit, we can mask out the interrupt bits in the PSW to make sure that we
> do not inject anything.
> 
> Signed-off-by: Janosch Frank <frankja@linux.ibm.com>
> ---
>  arch/s390/kvm/kvm-s390.c | 6 ++++++
>  1 file changed, 6 insertions(+)
> 
> diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
> index d4dc156e2c3e..137ae5dc9101 100644
> --- a/arch/s390/kvm/kvm-s390.c
> +++ b/arch/s390/kvm/kvm-s390.c
> @@ -4050,6 +4050,7 @@ static int vcpu_post_run(struct kvm_vcpu *vcpu, int exit_reason)
>  	return vcpu_post_run_fault_in_sie(vcpu);
>  }
>  
> +#define PSW_INT_MASK (PSW_MASK_EXT | PSW_MASK_IO | PSW_MASK_MCHECK)
>  static int __vcpu_run(struct kvm_vcpu *vcpu)
>  {
>  	int rc, exit_reason;
> @@ -4082,10 +4083,15 @@ static int __vcpu_run(struct kvm_vcpu *vcpu)
>  		}
>  		exit_reason = sie64a(vcpu->arch.sie_block,
>  				     vcpu->run->s.regs.gprs);
> +		/* This will likely be moved into a new function. */

As already mentioned by Cornelia: Please remove the above comment.

>  		if (kvm_s390_pv_is_protected(vcpu->kvm)) {
>  			memcpy(vcpu->run->s.regs.gprs,
>  			       sie_page->pv_grregs,
>  			       sizeof(sie_page->pv_grregs));

... but I'd like to suggest to add a comment before the following
if-statement with some information from the patch description. Otherwise
this will later be one of the code spots where you scratch your head
while looking at it and wonder why this masking of bits is done here.

> +			if (vcpu->arch.sie_block->icptcode == ICPT_PV_INSTR ||
> +			    vcpu->arch.sie_block->icptcode == ICPT_PV_PREF) {
> +				vcpu->arch.sie_block->gpsw.mask &= ~PSW_INT_MASK;
> +			}
>  		}
>  		local_irq_disable();
>  		__enable_cpu_timer_accounting(vcpu);
> 

 Thomas

Re: [RFCv2 36/37] KVM: s390: protvirt: do not inject interrupts after start

[Thomas Huth]

On 03/02/2020 14.19, Christian Borntraeger wrote:
> As PSW restart is handled by the ultravisor (and we only get a start
> notification) we must re-check the PSW after a start before injecting
> interrupts.
> 
> Signed-off-by: Christian Borntraeger <borntraeger@de.ibm.com>
> ---
>  arch/s390/kvm/kvm-s390.c | 6 ++++++
>  1 file changed, 6 insertions(+)
> 
> diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
> index 137ae5dc9101..3e4716b3fc02 100644
> --- a/arch/s390/kvm/kvm-s390.c
> +++ b/arch/s390/kvm/kvm-s390.c
> @@ -4451,6 +4451,12 @@ void kvm_s390_vcpu_start(struct kvm_vcpu *vcpu)
>  	/* Let's tell the UV that we want to start again */
>  	kvm_s390_pv_set_cpu_state(vcpu, PV_CPU_STATE_OPR);
>  	kvm_s390_clear_cpuflags(vcpu, CPUSTAT_STOPPED);
> +	/*
> +	 * The real PSW might have changed due to an interpreted RESTART.
> +	 * We block all interrupts and let the next sie exit refresh our view.
> +	 */
> +	if (kvm_s390_pv_is_protected(vcpu->kvm))
> +		vcpu->arch.sie_block->gpsw.mask &= ~PSW_INT_MASK;
>  	/*
>  	 * Another VCPU might have used IBS while we were offline.
>  	 * Let's play safe and flush the VCPU at startup.
> 

Reviewed-by: Thomas Huth <thuth@redhat.com>

Re: [RFCv2.1] KVM: S390: protvirt: Introduce instruction data area bounce buffer

[Christian Borntraeger]

On 06.02.20 10:54, David Hildenbrand wrote:
> if (mop->size + mop->sida_offset > mop->size)
                                   <

with that it is sufficient

> 	return -EINVAL;
> if (mop->size + mop->sida_offset > sida_size(vcpu->arch.sie_block))
> 	return -E2BIG;
> 
> Should be sufficient (instead of 3 checks).

Re: [RFCv2 36/37] KVM: s390: protvirt: do not inject interrupts after start

[Christian Borntraeger]

On 06.02.20 11:13, Cornelia Huck wrote:
> On Mon,  3 Feb 2020 08:19:56 -0500
> Christian Borntraeger <borntraeger@de.ibm.com> wrote:
> 
>> As PSW restart is handled by the ultravisor (and we only get a start
>> notification) we must re-check the PSW after a start before injecting
>> interrupts.
>>
>> Signed-off-by: Christian Borntraeger <borntraeger@de.ibm.com>
>> ---
>>  arch/s390/kvm/kvm-s390.c | 6 ++++++
>>  1 file changed, 6 insertions(+)
>>
>> diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
>> index 137ae5dc9101..3e4716b3fc02 100644
>> --- a/arch/s390/kvm/kvm-s390.c
>> +++ b/arch/s390/kvm/kvm-s390.c
>> @@ -4451,6 +4451,12 @@ void kvm_s390_vcpu_start(struct kvm_vcpu *vcpu)
>>  	/* Let's tell the UV that we want to start again */
>>  	kvm_s390_pv_set_cpu_state(vcpu, PV_CPU_STATE_OPR);
>>  	kvm_s390_clear_cpuflags(vcpu, CPUSTAT_STOPPED);
>> +	/*
>> +	 * The real PSW might have changed due to an interpreted RESTART.
> 
> "a RESTART interpreted by the ultravisor" ?

ack

> 
>> +	 * We block all interrupts and let the next sie exit refresh our view.
>> +	 */
>> +	if (kvm_s390_pv_is_protected(vcpu->kvm))
>> +		vcpu->arch.sie_block->gpsw.mask &= ~PSW_INT_MASK;
>>  	/*
>>  	 * Another VCPU might have used IBS while we were offline.
>>  	 * Let's play safe and flush the VCPU at startup.
> 
> Reviewed-by: Cornelia Huck <cohuck@redhat.com>
> 

Re: [RFCv2 37/37] KVM: s390: protvirt: Add UV cpu reset calls

[Christian Borntraeger]

On 06.02.20 11:45, Cornelia Huck wrote:
> On Mon,  3 Feb 2020 08:19:57 -0500
> Christian Borntraeger <borntraeger@de.ibm.com> wrote:
> 
>> From: Janosch Frank <frankja@linux.ibm.com>
>>
>> For protected VMs, the VCPU resets are done by the Ultravisor, as KVM
>> has no access to the VCPU registers.
>>
>> As the Ultravisor will only accept a call for the reset that is
>> needed, we need to fence the UV calls when chaining resets.
> 
> Stale note, chaining resets is gone?
> 
> "Note that the ultravisor will only accept a call for the exact reset
> that has been requested." ?

ack

> 
>>
>> Signed-off-by: Janosch Frank <frankja@linux.ibm.com>
>> ---
>>  arch/s390/kvm/kvm-s390.c | 20 ++++++++++++++++++++
>>  1 file changed, 20 insertions(+)
>>
>> diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
>> index 3e4716b3fc02..f7a3f84be064 100644
>> --- a/arch/s390/kvm/kvm-s390.c
>> +++ b/arch/s390/kvm/kvm-s390.c
>> @@ -4699,6 +4699,7 @@ long kvm_arch_vcpu_ioctl(struct file *filp,
>>  	void __user *argp = (void __user *)arg;
>>  	int idx;
>>  	long r;
>> +	u32 ret;
>>  
>>  	vcpu_load(vcpu);
>>  
>> @@ -4720,14 +4721,33 @@ long kvm_arch_vcpu_ioctl(struct file *filp,
>>  	case KVM_S390_CLEAR_RESET:
>>  		r = 0;
>>  		kvm_arch_vcpu_ioctl_clear_reset(vcpu);
>> +		if (kvm_s390_pv_handle_cpu(vcpu)) {
>> +			r = uv_cmd_nodata(kvm_s390_pv_handle_cpu(vcpu),
>> +					  UVC_CMD_CPU_RESET_CLEAR, &ret);
>> +			VCPU_EVENT(vcpu, 3, "PROTVIRT RESET CLEAR VCPU: cpu %d rc %x rrc %x",
>> +				   vcpu->vcpu_id, ret >> 16, ret & 0x0000ffff);
> 
> I think VCPU_EVENT() already traces the vcpu_id, doesn't it?

Yes, it does. will fix.


> 
>> +		}
>>  		break;
>>  	case KVM_S390_INITIAL_RESET:
>>  		r = 0;
>>  		kvm_arch_vcpu_ioctl_initial_reset(vcpu);
>> +		if (kvm_s390_pv_handle_cpu(vcpu)) {
>> +			r = uv_cmd_nodata(kvm_s390_pv_handle_cpu(vcpu),
>> +					  UVC_CMD_CPU_RESET_INITIAL,
>> +					  &ret);
>> +			VCPU_EVENT(vcpu, 3, "PROTVIRT RESET INITIAL VCPU: cpu %d rc %x rrc %x",
>> +				   vcpu->vcpu_id, ret >> 16, ret & 0x0000ffff);
>> +		}
>>  		break;
>>  	case KVM_S390_NORMAL_RESET:
>>  		r = 0;
>>  		kvm_arch_vcpu_ioctl_normal_reset(vcpu);
>> +		if (kvm_s390_pv_handle_cpu(vcpu)) {
>> +			r = uv_cmd_nodata(kvm_s390_pv_handle_cpu(vcpu),
>> +					  UVC_CMD_CPU_RESET, &ret);
>> +			VCPU_EVENT(vcpu, 3, "PROTVIRT RESET NORMAL VCPU: cpu %d rc %x rrc %x",
>> +				   vcpu->vcpu_id, ret >> 16, ret & 0x0000ffff);
>> +		}
>>  		break;
>>  	case KVM_SET_ONE_REG:
>>  	case KVM_GET_ONE_REG: {
> 
> Otherwise, looks good.
> 

Re: [RFCv2 32/37] KVM: s390: protvirt: Report CPU state to Ultravisor

[Cornelia Huck]

On Mon,  3 Feb 2020 08:19:52 -0500
Christian Borntraeger <borntraeger@de.ibm.com> wrote:

> From: Janosch Frank <frankja@linux.ibm.com>
> 
> VCPU states have to be reported to the ultravisor for SIGP
> interpretation.
> 
> Signed-off-by: Janosch Frank <frankja@linux.ibm.com>
> ---
>  arch/s390/kvm/kvm-s390.c | 5 ++++-
>  1 file changed, 4 insertions(+), 1 deletion(-)

Reviewed-by: Cornelia Huck <cohuck@redhat.com>

Re: [RFCv2 37/37] KVM: s390: protvirt: Add UV cpu reset calls

[David Hildenbrand]

On 03.02.20 14:19, Christian Borntraeger wrote:
> From: Janosch Frank <frankja@linux.ibm.com>
> 
> For protected VMs, the VCPU resets are done by the Ultravisor, as KVM
> has no access to the VCPU registers.
> 
> As the Ultravisor will only accept a call for the reset that is
> needed, we need to fence the UV calls when chaining resets.
> 
> Signed-off-by: Janosch Frank <frankja@linux.ibm.com>
> ---
>  arch/s390/kvm/kvm-s390.c | 20 ++++++++++++++++++++
>  1 file changed, 20 insertions(+)
> 
> diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
> index 3e4716b3fc02..f7a3f84be064 100644
> --- a/arch/s390/kvm/kvm-s390.c
> +++ b/arch/s390/kvm/kvm-s390.c
> @@ -4699,6 +4699,7 @@ long kvm_arch_vcpu_ioctl(struct file *filp,
>  	void __user *argp = (void __user *)arg;
>  	int idx;
>  	long r;
> +	u32 ret;
>  
>  	vcpu_load(vcpu);
>  
> @@ -4720,14 +4721,33 @@ long kvm_arch_vcpu_ioctl(struct file *filp,
>  	case KVM_S390_CLEAR_RESET:
>  		r = 0;
>  		kvm_arch_vcpu_ioctl_clear_reset(vcpu);
> +		if (kvm_s390_pv_handle_cpu(vcpu)) {
> +			r = uv_cmd_nodata(kvm_s390_pv_handle_cpu(vcpu),
> +					  UVC_CMD_CPU_RESET_CLEAR, &ret);
> +			VCPU_EVENT(vcpu, 3, "PROTVIRT RESET CLEAR VCPU: cpu %d rc %x rrc %x",
> +				   vcpu->vcpu_id, ret >> 16, ret & 0x0000ffff);
> +		}
>  		break;
>  	case KVM_S390_INITIAL_RESET:
>  		r = 0;
>  		kvm_arch_vcpu_ioctl_initial_reset(vcpu);
> +		if (kvm_s390_pv_handle_cpu(vcpu)) {
> +			r = uv_cmd_nodata(kvm_s390_pv_handle_cpu(vcpu),
> +					  UVC_CMD_CPU_RESET_INITIAL,
> +					  &ret);
> +			VCPU_EVENT(vcpu, 3, "PROTVIRT RESET INITIAL VCPU: cpu %d rc %x rrc %x",
> +				   vcpu->vcpu_id, ret >> 16, ret & 0x0000ffff);
> +		}
>  		break;
>  	case KVM_S390_NORMAL_RESET:
>  		r = 0;
>  		kvm_arch_vcpu_ioctl_normal_reset(vcpu);
> +		if (kvm_s390_pv_handle_cpu(vcpu)) {
> +			r = uv_cmd_nodata(kvm_s390_pv_handle_cpu(vcpu),
> +					  UVC_CMD_CPU_RESET, &ret);
> +			VCPU_EVENT(vcpu, 3, "PROTVIRT RESET NORMAL VCPU: cpu %d rc %x rrc %x",
> +				   vcpu->vcpu_id, ret >> 16, ret & 0x0000ffff);
> +		}
>  		break;
>  	case KVM_SET_ONE_REG:
>  	case KVM_GET_ONE_REG: {
> 

Any reason why to not put that into the actual kvm_arch_vcpu_ioctl_*
functions?

Anyhow,

Reviewed-by: David Hildenbrand <david@redhat.com>

-- 
Thanks,

David / dhildenb

Re: [RFCv2 37/37] KVM: s390: protvirt: Add UV cpu reset calls

[Thomas Huth]

On 06/02/2020 15.39, David Hildenbrand wrote:
> On 03.02.20 14:19, Christian Borntraeger wrote:
>> From: Janosch Frank <frankja@linux.ibm.com>
>>
>> For protected VMs, the VCPU resets are done by the Ultravisor, as KVM
>> has no access to the VCPU registers.
>>
>> As the Ultravisor will only accept a call for the reset that is
>> needed, we need to fence the UV calls when chaining resets.
>>
>> Signed-off-by: Janosch Frank <frankja@linux.ibm.com>
>> ---
>>  arch/s390/kvm/kvm-s390.c | 20 ++++++++++++++++++++
>>  1 file changed, 20 insertions(+)
>>
>> diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
>> index 3e4716b3fc02..f7a3f84be064 100644
>> --- a/arch/s390/kvm/kvm-s390.c
>> +++ b/arch/s390/kvm/kvm-s390.c
>> @@ -4699,6 +4699,7 @@ long kvm_arch_vcpu_ioctl(struct file *filp,
>>  	void __user *argp = (void __user *)arg;
>>  	int idx;
>>  	long r;
>> +	u32 ret;
>>  
>>  	vcpu_load(vcpu);
>>  
>> @@ -4720,14 +4721,33 @@ long kvm_arch_vcpu_ioctl(struct file *filp,
>>  	case KVM_S390_CLEAR_RESET:
>>  		r = 0;
>>  		kvm_arch_vcpu_ioctl_clear_reset(vcpu);
>> +		if (kvm_s390_pv_handle_cpu(vcpu)) {
>> +			r = uv_cmd_nodata(kvm_s390_pv_handle_cpu(vcpu),
>> +					  UVC_CMD_CPU_RESET_CLEAR, &ret);
>> +			VCPU_EVENT(vcpu, 3, "PROTVIRT RESET CLEAR VCPU: cpu %d rc %x rrc %x",
>> +				   vcpu->vcpu_id, ret >> 16, ret & 0x0000ffff);
>> +		}
>>  		break;
>>  	case KVM_S390_INITIAL_RESET:
>>  		r = 0;
>>  		kvm_arch_vcpu_ioctl_initial_reset(vcpu);
>> +		if (kvm_s390_pv_handle_cpu(vcpu)) {
>> +			r = uv_cmd_nodata(kvm_s390_pv_handle_cpu(vcpu),
>> +					  UVC_CMD_CPU_RESET_INITIAL,
>> +					  &ret);
>> +			VCPU_EVENT(vcpu, 3, "PROTVIRT RESET INITIAL VCPU: cpu %d rc %x rrc %x",
>> +				   vcpu->vcpu_id, ret >> 16, ret & 0x0000ffff);
>> +		}
>>  		break;
>>  	case KVM_S390_NORMAL_RESET:
>>  		r = 0;
>>  		kvm_arch_vcpu_ioctl_normal_reset(vcpu);
>> +		if (kvm_s390_pv_handle_cpu(vcpu)) {
>> +			r = uv_cmd_nodata(kvm_s390_pv_handle_cpu(vcpu),
>> +					  UVC_CMD_CPU_RESET, &ret);
>> +			VCPU_EVENT(vcpu, 3, "PROTVIRT RESET NORMAL VCPU: cpu %d rc %x rrc %x",
>> +				   vcpu->vcpu_id, ret >> 16, ret & 0x0000ffff);
>> +		}
>>  		break;
>>  	case KVM_SET_ONE_REG:
>>  	case KVM_GET_ONE_REG: {
>>
> 
> Any reason why to not put that into the actual kvm_arch_vcpu_ioctl_*
> functions?

Because they are chained and you must not chain the UV calls.

 Thomas

Re: [RFCv2 33/37] KVM: s390: protvirt: Support cmd 5 operation state

[Cornelia Huck]

On Mon,  3 Feb 2020 08:19:53 -0500
Christian Borntraeger <borntraeger@de.ibm.com> wrote:

> From: Janosch Frank <frankja@linux.ibm.com>
> 
> Code 5 for the set cpu state UV call tells the UV to load a PSW from
> the SE header (first IPL) or from guest location 0x0 (diag 308 subcode
> 0/1). Also it sets the cpu into operating state afterwards, so we can
> start it.
> 
> Signed-off-by: Janosch Frank <frankja@linux.ibm.com>
> ---
>  arch/s390/include/asm/uv.h | 1 +
>  arch/s390/kvm/kvm-s390.c   | 7 +++++++
>  include/uapi/linux/kvm.h   | 1 +

Reviewed-by: Cornelia Huck <cohuck@redhat.com>

Re: [RFCv2 34/37] KVM: s390: protvirt: Add UV debug trace

[Christian Borntraeger]

On 06.02.20 10:41, Cornelia Huck wrote:
> On Mon,  3 Feb 2020 08:19:54 -0500
> Christian Borntraeger <borntraeger@de.ibm.com> wrote:
> 
>> From: Janosch Frank <frankja@linux.ibm.com>
>>
>> Let's have some debug traces which stay around for longer than the
>> guest.
>>
>> Signed-off-by: Janosch Frank <frankja@linux.ibm.com>
>> ---
>>  arch/s390/kvm/kvm-s390.c | 10 +++++++++-
>>  arch/s390/kvm/kvm-s390.h |  9 +++++++++
>>  arch/s390/kvm/pv.c       | 21 +++++++++++++++++++--
>>  3 files changed, 37 insertions(+), 3 deletions(-)
>>
>> diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
>> index 2beb93f0572f..d4dc156e2c3e 100644
>> --- a/arch/s390/kvm/kvm-s390.c
>> +++ b/arch/s390/kvm/kvm-s390.c
>> @@ -221,6 +221,7 @@ static struct kvm_s390_vm_cpu_subfunc kvm_s390_available_subfunc;
>>  static struct gmap_notifier gmap_notifier;
>>  static struct gmap_notifier vsie_gmap_notifier;
>>  debug_info_t *kvm_s390_dbf;
>> +debug_info_t *kvm_s390_dbf_uv;
>>  
>>  /* Section: not file related */
>>  int kvm_arch_hardware_enable(void)
>> @@ -462,7 +463,13 @@ int kvm_arch_init(void *opaque)
>>  	if (!kvm_s390_dbf)
>>  		return -ENOMEM;
>>  
>> -	if (debug_register_view(kvm_s390_dbf, &debug_sprintf_view))
>> +	kvm_s390_dbf_uv = debug_register("kvm-uv", 32, 1, 7 * sizeof(long));
>> +	if (!kvm_s390_dbf_uv)
>> +		return -ENOMEM;
> 
> Doesn't that leak kvm_s390_dbf?

Yes, it does.  I can simply goto out. The kvm_arch_exit will release all non-zero
dbfs.

> 
>> +
>> +
> 
> One blank line should be enough.

ack.

> 
>> +	if (debug_register_view(kvm_s390_dbf, &debug_sprintf_view) ||
>> +	    debug_register_view(kvm_s390_dbf_uv, &debug_sprintf_view))
>>  		goto out;
>>  
>>  	kvm_s390_cpu_feat_init();
>> @@ -489,6 +496,7 @@ void kvm_arch_exit(void)
>>  {
>>  	kvm_s390_gib_destroy();
>>  	debug_unregister(kvm_s390_dbf);
>> +	debug_unregister(kvm_s390_dbf_uv);
>>  }
>>  
>>  /* Section: device related */
> 
> (...)
> 
>> @@ -252,7 +269,7 @@ int kvm_s390_pv_unpack(struct kvm *kvm, unsigned long addr, unsigned long size,
>>  		addr += PAGE_SIZE;
>>  		tw[1] += PAGE_SIZE;
>>  	}
>> -	VM_EVENT(kvm, 3, "PROTVIRT VM UNPACK: finished rc %x", rc);
>> +	VM_EVENT(kvm, 3, "PROTVIRT VM UNPACK: finished with rc %x", rc);
> 
> Can you merge this into the patch that introduces this log entry?

ack.
> 
> Also, do you want to add logging into the new dbf here as well?

ack
> 
>>  	return rc;
>>  }
>>  
> 
> You often seem to log in pairs (into the per-vm dbf and into the new uv
> dbf). Would it make sense to introduce a new helper for that, or is
> that overkill?

I guess the logging will see several changes over time anyway. Let us start
with an initial variant. 

Re: [RFCv2 34/37] KVM: s390: protvirt: Add UV debug trace

[Cornelia Huck]

On Fri, 7 Feb 2020 09:05:27 +0100
Christian Borntraeger <borntraeger@de.ibm.com> wrote:

> On 06.02.20 10:41, Cornelia Huck wrote:
> > On Mon,  3 Feb 2020 08:19:54 -0500
> > Christian Borntraeger <borntraeger@de.ibm.com> wrote:
> >   
> >> From: Janosch Frank <frankja@linux.ibm.com>
> >>
> >> Let's have some debug traces which stay around for longer than the
> >> guest.
> >>
> >> Signed-off-by: Janosch Frank <frankja@linux.ibm.com>
> >> ---
> >>  arch/s390/kvm/kvm-s390.c | 10 +++++++++-
> >>  arch/s390/kvm/kvm-s390.h |  9 +++++++++
> >>  arch/s390/kvm/pv.c       | 21 +++++++++++++++++++--
> >>  3 files changed, 37 insertions(+), 3 deletions(-)

> > You often seem to log in pairs (into the per-vm dbf and into the new uv
> > dbf). Would it make sense to introduce a new helper for that, or is
> > that overkill?  
> 
> I guess the logging will see several changes over time anyway. Let us start
> with an initial variant. 

Fair enough.

Re: [RFCv2 34/37] KVM: s390: protvirt: Add UV debug trace

[Christian Borntraeger]

On 06.02.20 10:41, Cornelia Huck wrote:

> 
> You often seem to log in pairs (into the per-vm dbf and into the new uv
> dbf). Would it make sense to introduce a new helper for that, or is
> that overkill?

I think I had now a good idea.

I will let KVM_UV_EVENT always log into both logs (the per KVM and the global one).
If it is important enough for the global one it really should also be in the 
per kvm one. So I will split out all messages into the separate patches and
move this infrastructure at the beginning of the patch series.
Then we only need one line of code for each log. 

Re: [RFCv2 34/37] KVM: s390: protvirt: Add UV debug trace

[Cornelia Huck]

On Fri, 14 Feb 2020 09:32:26 +0100
Christian Borntraeger <borntraeger@de.ibm.com> wrote:

> On 06.02.20 10:41, Cornelia Huck wrote:
> 
> > 
> > You often seem to log in pairs (into the per-vm dbf and into the new uv
> > dbf). Would it make sense to introduce a new helper for that, or is
> > that overkill?  
> 
> I think I had now a good idea.
> 
> I will let KVM_UV_EVENT always log into both logs (the per KVM and the global one).
> If it is important enough for the global one it really should also be in the 
> per kvm one. So I will split out all messages into the separate patches and
> move this infrastructure at the beginning of the patch series.
> Then we only need one line of code for each log. 
> 

I think that makes a lot of sense.