* [kvm-unit-tests GIT PULL 01/11] s390x: Add SPDX and header comments for s390x/* and lib/s390x/*
2021-08-16 13:20 [kvm-unit-tests GIT PULL 00/11] s390x update 2021-16-08 Janosch Frank
@ 2021-08-16 13:20 ` Janosch Frank
2021-08-16 13:20 ` [kvm-unit-tests GIT PULL 02/11] s390x: Add SPDX and header comments for the snippets folder Janosch Frank
` (10 subsequent siblings)
11 siblings, 0 replies; 13+ messages in thread
From: Janosch Frank @ 2021-08-16 13:20 UTC (permalink / raw)
To: pbonzini
Cc: kvm, frankja, david, borntraeger, cohuck, linux-s390, imbrenda, thuth
Seems like I missed adding them.
Signed-off-by: Janosch Frank <frankja@linux.ibm.com>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Reviewed-by: Claudio Imbrenda <imbrenda@linux.ibm.com>
---
lib/s390x/uv.c | 9 +++++++++
s390x/mvpg-sie.c | 9 +++++++++
s390x/sie.c | 10 ++++++++++
3 files changed, 28 insertions(+)
diff --git a/lib/s390x/uv.c b/lib/s390x/uv.c
index 0d8c141c..fd9de944 100644
--- a/lib/s390x/uv.c
+++ b/lib/s390x/uv.c
@@ -1,3 +1,12 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Ultravisor related functionality
+ *
+ * Copyright 2020 IBM Corp.
+ *
+ * Authors:
+ * Janosch Frank <frankja@linux.ibm.com>
+ */
#include <libcflat.h>
#include <bitops.h>
#include <alloc.h>
diff --git a/s390x/mvpg-sie.c b/s390x/mvpg-sie.c
index 9bcd15a2..5e70f591 100644
--- a/s390x/mvpg-sie.c
+++ b/s390x/mvpg-sie.c
@@ -1,3 +1,12 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Tests mvpg SIE partial execution intercepts.
+ *
+ * Copyright 2021 IBM Corp.
+ *
+ * Authors:
+ * Janosch Frank <frankja@linux.ibm.com>
+ */
#include <libcflat.h>
#include <asm/asm-offsets.h>
#include <asm-generic/barrier.h>
diff --git a/s390x/sie.c b/s390x/sie.c
index cfc746f3..134d3c4f 100644
--- a/s390x/sie.c
+++ b/s390x/sie.c
@@ -1,3 +1,13 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Tests SIE diagnose intercepts.
+ * Mainly used as a template for SIE tests.
+ *
+ * Copyright 2021 IBM Corp.
+ *
+ * Authors:
+ * Janosch Frank <frankja@linux.ibm.com>
+ */
#include <libcflat.h>
#include <asm/asm-offsets.h>
#include <asm/arch_def.h>
--
2.31.1
^ permalink raw reply related [flat|nested] 13+ messages in thread* [kvm-unit-tests GIT PULL 02/11] s390x: Add SPDX and header comments for the snippets folder
2021-08-16 13:20 [kvm-unit-tests GIT PULL 00/11] s390x update 2021-16-08 Janosch Frank
2021-08-16 13:20 ` [kvm-unit-tests GIT PULL 01/11] s390x: Add SPDX and header comments for s390x/* and lib/s390x/* Janosch Frank
@ 2021-08-16 13:20 ` Janosch Frank
2021-08-16 13:20 ` [kvm-unit-tests GIT PULL 03/11] s390x: Fix my mail address in the headers Janosch Frank
` (9 subsequent siblings)
11 siblings, 0 replies; 13+ messages in thread
From: Janosch Frank @ 2021-08-16 13:20 UTC (permalink / raw)
To: pbonzini
Cc: kvm, frankja, david, borntraeger, cohuck, linux-s390, imbrenda, thuth
Seems like I missed adding them.
Signed-off-by: Janosch Frank <frankja@linux.ibm.com>
Reviewed-by: Claudio Imbrenda <imbrenda@linux.ibm.com>
Reviewed-by: Thomas Huth <thuth@redhat.com>
---
s390x/snippets/c/cstart.S | 9 +++++++++
s390x/snippets/c/mvpg-snippet.c | 9 +++++++++
2 files changed, 18 insertions(+)
diff --git a/s390x/snippets/c/cstart.S b/s390x/snippets/c/cstart.S
index 242568d6..a1754808 100644
--- a/s390x/snippets/c/cstart.S
+++ b/s390x/snippets/c/cstart.S
@@ -1,3 +1,12 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Start assembly for snippets
+ *
+ * Copyright (c) 2021 IBM Corp.
+ *
+ * Authors:
+ * Janosch Frank <frankja@linux.ibm.com>
+ */
#include <asm/sigp.h>
.section .init
diff --git a/s390x/snippets/c/mvpg-snippet.c b/s390x/snippets/c/mvpg-snippet.c
index c1eb5d77..e55caab4 100644
--- a/s390x/snippets/c/mvpg-snippet.c
+++ b/s390x/snippets/c/mvpg-snippet.c
@@ -1,3 +1,12 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Snippet used by the mvpg-sie.c test to check SIE PEI intercepts.
+ *
+ * Copyright (c) 2021 IBM Corp
+ *
+ * Authors:
+ * Janosch Frank <frankja@linux.ibm.com>
+ */
#include <libcflat.h>
static inline void force_exit(void)
--
2.31.1
^ permalink raw reply related [flat|nested] 13+ messages in thread* [kvm-unit-tests GIT PULL 03/11] s390x: Fix my mail address in the headers
2021-08-16 13:20 [kvm-unit-tests GIT PULL 00/11] s390x update 2021-16-08 Janosch Frank
2021-08-16 13:20 ` [kvm-unit-tests GIT PULL 01/11] s390x: Add SPDX and header comments for s390x/* and lib/s390x/* Janosch Frank
2021-08-16 13:20 ` [kvm-unit-tests GIT PULL 02/11] s390x: Add SPDX and header comments for the snippets folder Janosch Frank
@ 2021-08-16 13:20 ` Janosch Frank
2021-08-16 13:20 ` [kvm-unit-tests GIT PULL 04/11] s390x: sie: Add sie lib validity handling Janosch Frank
` (8 subsequent siblings)
11 siblings, 0 replies; 13+ messages in thread
From: Janosch Frank @ 2021-08-16 13:20 UTC (permalink / raw)
To: pbonzini
Cc: kvm, frankja, david, borntraeger, cohuck, linux-s390, imbrenda, thuth
I used the wrong one once and then copied it over...
Signed-off-by: Janosch Frank <frankja@linux.ibm.com>
Reviewed-by: Claudio Imbrenda <imbrenda@linux.ibm.com>
Reviewed-by: Thomas Huth <thuth@redhat.com>
---
lib/s390x/asm/mem.h | 2 +-
lib/s390x/mmu.h | 2 +-
lib/s390x/stack.c | 2 +-
s390x/gs.c | 2 +-
s390x/iep.c | 2 +-
s390x/vector.c | 2 +-
6 files changed, 6 insertions(+), 6 deletions(-)
diff --git a/lib/s390x/asm/mem.h b/lib/s390x/asm/mem.h
index 1963cef7..40b22b63 100644
--- a/lib/s390x/asm/mem.h
+++ b/lib/s390x/asm/mem.h
@@ -3,7 +3,7 @@
* Physical memory management related functions and definitions.
*
* Copyright IBM Corp. 2018
- * Author(s): Janosch Frank <frankja@de.ibm.com>
+ * Author(s): Janosch Frank <frankja@linux.ibm.com>
*/
#ifndef _ASMS390X_MEM_H_
#define _ASMS390X_MEM_H_
diff --git a/lib/s390x/mmu.h b/lib/s390x/mmu.h
index ab35d782..15f88e4f 100644
--- a/lib/s390x/mmu.h
+++ b/lib/s390x/mmu.h
@@ -5,7 +5,7 @@
* Copyright (c) 2018 IBM Corp
*
* Authors:
- * Janosch Frank <frankja@de.ibm.com>
+ * Janosch Frank <frankja@linux.ibm.com>
*/
#ifndef _S390X_MMU_H_
#define _S390X_MMU_H_
diff --git a/lib/s390x/stack.c b/lib/s390x/stack.c
index 4cf80dae..e714e07c 100644
--- a/lib/s390x/stack.c
+++ b/lib/s390x/stack.c
@@ -8,7 +8,7 @@
* Authors:
* Thomas Huth <thuth@redhat.com>
* David Hildenbrand <david@redhat.com>
- * Janosch Frank <frankja@de.ibm.com>
+ * Janosch Frank <frankja@linux.ibm.com>
*/
#include <libcflat.h>
#include <stack.h>
diff --git a/s390x/gs.c b/s390x/gs.c
index a017a97d..7567bb78 100644
--- a/s390x/gs.c
+++ b/s390x/gs.c
@@ -6,7 +6,7 @@
*
* Authors:
* Martin Schwidefsky <schwidefsky@de.ibm.com>
- * Janosch Frank <frankja@de.ibm.com>
+ * Janosch Frank <frankja@linux.ibm.com>
*/
#include <libcflat.h>
#include <asm/page.h>
diff --git a/s390x/iep.c b/s390x/iep.c
index 906c77b3..8d5e044b 100644
--- a/s390x/iep.c
+++ b/s390x/iep.c
@@ -5,7 +5,7 @@
* Copyright (c) 2018 IBM Corp
*
* Authors:
- * Janosch Frank <frankja@de.ibm.com>
+ * Janosch Frank <frankja@linux.ibm.com>
*/
#include <libcflat.h>
#include <vmalloc.h>
diff --git a/s390x/vector.c b/s390x/vector.c
index fdb0eee2..c8c14e33 100644
--- a/s390x/vector.c
+++ b/s390x/vector.c
@@ -5,7 +5,7 @@
* Copyright 2018 IBM Corp.
*
* Authors:
- * Janosch Frank <frankja@de.ibm.com>
+ * Janosch Frank <frankja@linux.ibm.com>
*/
#include <libcflat.h>
#include <asm/page.h>
--
2.31.1
^ permalink raw reply related [flat|nested] 13+ messages in thread* [kvm-unit-tests GIT PULL 04/11] s390x: sie: Add sie lib validity handling
2021-08-16 13:20 [kvm-unit-tests GIT PULL 00/11] s390x update 2021-16-08 Janosch Frank
` (2 preceding siblings ...)
2021-08-16 13:20 ` [kvm-unit-tests GIT PULL 03/11] s390x: Fix my mail address in the headers Janosch Frank
@ 2021-08-16 13:20 ` Janosch Frank
2021-08-16 13:20 ` [kvm-unit-tests GIT PULL 05/11] s390x: lib: Introduce HPAGE_* constants Janosch Frank
` (7 subsequent siblings)
11 siblings, 0 replies; 13+ messages in thread
From: Janosch Frank @ 2021-08-16 13:20 UTC (permalink / raw)
To: pbonzini
Cc: kvm, frankja, david, borntraeger, cohuck, linux-s390, imbrenda, thuth
Let's start off the SIE lib with validity handling code since that has
the least amount of dependencies to other files.
Signed-off-by: Janosch Frank <frankja@linux.ibm.com>
Reviewed-by: Claudio Imbrenda <imbrenda@linux.ibm.com>
---
lib/s390x/sie.c | 40 ++++++++++++++++++++++++++++++++++++++++
lib/s390x/sie.h | 3 +++
s390x/Makefile | 1 +
s390x/mvpg-sie.c | 2 +-
s390x/sie.c | 7 +------
5 files changed, 46 insertions(+), 7 deletions(-)
create mode 100644 lib/s390x/sie.c
diff --git a/lib/s390x/sie.c b/lib/s390x/sie.c
new file mode 100644
index 00000000..15ba407c
--- /dev/null
+++ b/lib/s390x/sie.c
@@ -0,0 +1,40 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Virtualization library that speeds up managing guests.
+ *
+ * Copyright (c) 2021 IBM Corp
+ *
+ * Authors:
+ * Janosch Frank <frankja@linux.ibm.com>
+ */
+
+#include <asm/barrier.h>
+#include <libcflat.h>
+#include <sie.h>
+
+static bool validity_expected;
+static uint16_t vir; /* Validity interception reason */
+
+void sie_expect_validity(void)
+{
+ validity_expected = true;
+ vir = 0;
+}
+
+void sie_check_validity(uint16_t vir_exp)
+{
+ report(vir_exp == vir, "VALIDITY: %x", vir);
+ vir = 0;
+}
+
+void sie_handle_validity(struct vm *vm)
+{
+ if (vm->sblk->icptcode != ICPT_VALIDITY)
+ return;
+
+ vir = vm->sblk->ipb >> 16;
+
+ if (!validity_expected)
+ report_abort("VALIDITY: %x", vir);
+ validity_expected = false;
+}
diff --git a/lib/s390x/sie.h b/lib/s390x/sie.h
index 6ba858a2..7ff98d2d 100644
--- a/lib/s390x/sie.h
+++ b/lib/s390x/sie.h
@@ -197,5 +197,8 @@ struct vm {
extern void sie_entry(void);
extern void sie_exit(void);
extern void sie64a(struct kvm_s390_sie_block *sblk, struct vm_save_area *save_area);
+void sie_expect_validity(void);
+void sie_check_validity(uint16_t vir_exp);
+void sie_handle_validity(struct vm *vm);
#endif /* _S390X_SIE_H_ */
diff --git a/s390x/Makefile b/s390x/Makefile
index 6565561b..ef8041a6 100644
--- a/s390x/Makefile
+++ b/s390x/Makefile
@@ -71,6 +71,7 @@ cflatobjs += lib/s390x/css_dump.o
cflatobjs += lib/s390x/css_lib.o
cflatobjs += lib/s390x/malloc_io.o
cflatobjs += lib/s390x/uv.o
+cflatobjs += lib/s390x/sie.o
OBJDIRS += lib/s390x
diff --git a/s390x/mvpg-sie.c b/s390x/mvpg-sie.c
index 5e70f591..2ac91eec 100644
--- a/s390x/mvpg-sie.c
+++ b/s390x/mvpg-sie.c
@@ -39,7 +39,7 @@ static void sie(struct vm *vm)
while (vm->sblk->icptcode == 0) {
sie64a(vm->sblk, &vm->save_area);
- assert(vm->sblk->icptcode != ICPT_VALIDITY);
+ sie_handle_validity(vm);
}
vm->save_area.guest.grs[14] = vm->sblk->gg14;
vm->save_area.guest.grs[15] = vm->sblk->gg15;
diff --git a/s390x/sie.c b/s390x/sie.c
index 134d3c4f..5c798a9e 100644
--- a/s390x/sie.c
+++ b/s390x/sie.c
@@ -24,17 +24,12 @@ static u8 *guest;
static u8 *guest_instr;
static struct vm vm;
-static void handle_validity(struct vm *vm)
-{
- report(0, "VALIDITY: %x", vm->sblk->ipb >> 16);
-}
static void sie(struct vm *vm)
{
while (vm->sblk->icptcode == 0) {
sie64a(vm->sblk, &vm->save_area);
- if (vm->sblk->icptcode == ICPT_VALIDITY)
- handle_validity(vm);
+ sie_handle_validity(vm);
}
vm->save_area.guest.grs[14] = vm->sblk->gg14;
vm->save_area.guest.grs[15] = vm->sblk->gg15;
--
2.31.1
^ permalink raw reply related [flat|nested] 13+ messages in thread* [kvm-unit-tests GIT PULL 05/11] s390x: lib: Introduce HPAGE_* constants
2021-08-16 13:20 [kvm-unit-tests GIT PULL 00/11] s390x update 2021-16-08 Janosch Frank
` (3 preceding siblings ...)
2021-08-16 13:20 ` [kvm-unit-tests GIT PULL 04/11] s390x: sie: Add sie lib validity handling Janosch Frank
@ 2021-08-16 13:20 ` Janosch Frank
2021-08-16 13:20 ` [kvm-unit-tests GIT PULL 06/11] s390x: lib: sie: Add struct vm (de)initialization functions Janosch Frank
` (6 subsequent siblings)
11 siblings, 0 replies; 13+ messages in thread
From: Janosch Frank @ 2021-08-16 13:20 UTC (permalink / raw)
To: pbonzini
Cc: kvm, frankja, david, borntraeger, cohuck, linux-s390, imbrenda, thuth
They come in handy when working with 1MB blocks/addresses.
Signed-off-by: Janosch Frank <frankja@linux.ibm.com>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Reviewed-by: Claudio Imbrenda <imbrenda@linux.ibm.com>
---
lib/s390x/asm/page.h | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/lib/s390x/asm/page.h b/lib/s390x/asm/page.h
index f130f936..2f4afd06 100644
--- a/lib/s390x/asm/page.h
+++ b/lib/s390x/asm/page.h
@@ -35,4 +35,8 @@ typedef struct { pteval_t pte; } pte_t;
#define __pmd(x) ((pmd_t) { (x) } )
#define __pte(x) ((pte_t) { (x) } )
+#define HPAGE_SHIFT 20
+#define HPAGE_SIZE (_AC(1,UL) << HPAGE_SHIFT)
+#define HPAGE_MASK (~(HPAGE_SIZE-1))
+
#endif
--
2.31.1
^ permalink raw reply related [flat|nested] 13+ messages in thread* [kvm-unit-tests GIT PULL 06/11] s390x: lib: sie: Add struct vm (de)initialization functions
2021-08-16 13:20 [kvm-unit-tests GIT PULL 00/11] s390x update 2021-16-08 Janosch Frank
` (4 preceding siblings ...)
2021-08-16 13:20 ` [kvm-unit-tests GIT PULL 05/11] s390x: lib: Introduce HPAGE_* constants Janosch Frank
@ 2021-08-16 13:20 ` Janosch Frank
2021-08-16 13:20 ` [kvm-unit-tests GIT PULL 07/11] lib: s390x: sie: Move sie function into library Janosch Frank
` (5 subsequent siblings)
11 siblings, 0 replies; 13+ messages in thread
From: Janosch Frank @ 2021-08-16 13:20 UTC (permalink / raw)
To: pbonzini
Cc: kvm, frankja, david, borntraeger, cohuck, linux-s390, imbrenda, thuth
Before I start copying the same code over and over lets move this into
the library.
Signed-off-by: Janosch Frank <frankja@linux.ibm.com>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Reviewed-by: Claudio Imbrenda <imbrenda@linux.ibm.com>
---
lib/s390x/sie.c | 30 ++++++++++++++++++++++++++++++
lib/s390x/sie.h | 3 +++
s390x/mvpg-sie.c | 18 ++----------------
s390x/sie.c | 19 +++----------------
4 files changed, 38 insertions(+), 32 deletions(-)
diff --git a/lib/s390x/sie.c b/lib/s390x/sie.c
index 15ba407c..1a93e518 100644
--- a/lib/s390x/sie.c
+++ b/lib/s390x/sie.c
@@ -11,6 +11,9 @@
#include <asm/barrier.h>
#include <libcflat.h>
#include <sie.h>
+#include <asm/page.h>
+#include <libcflat.h>
+#include <alloc_page.h>
static bool validity_expected;
static uint16_t vir; /* Validity interception reason */
@@ -38,3 +41,30 @@ void sie_handle_validity(struct vm *vm)
report_abort("VALIDITY: %x", vir);
validity_expected = false;
}
+
+/* Initializes the struct vm members like the SIE control block. */
+void sie_guest_create(struct vm *vm, uint64_t guest_mem, uint64_t guest_mem_len)
+{
+ vm->sblk = alloc_page();
+ memset(vm->sblk, 0, PAGE_SIZE);
+ vm->sblk->cpuflags = CPUSTAT_ZARCH | CPUSTAT_RUNNING;
+ vm->sblk->ihcpu = 0xffff;
+ vm->sblk->prefix = 0;
+
+ /* Guest memory chunks are always 1MB */
+ assert(!(guest_mem_len & ~HPAGE_MASK));
+ /* Currently MSO/MSL is the easiest option */
+ vm->sblk->mso = (uint64_t)guest_mem;
+ vm->sblk->msl = (uint64_t)guest_mem + ((guest_mem_len - 1) & HPAGE_MASK);
+
+ /* CRYCB needs to be in the first 2GB */
+ vm->crycb = alloc_pages_flags(0, AREA_DMA31);
+ vm->sblk->crycbd = (uint32_t)(uintptr_t)vm->crycb;
+}
+
+/* Frees the memory that was gathered on initialization */
+void sie_guest_destroy(struct vm *vm)
+{
+ free_page(vm->crycb);
+ free_page(vm->sblk);
+}
diff --git a/lib/s390x/sie.h b/lib/s390x/sie.h
index 7ff98d2d..946bd164 100644
--- a/lib/s390x/sie.h
+++ b/lib/s390x/sie.h
@@ -190,6 +190,7 @@ struct vm_save_area {
struct vm {
struct kvm_s390_sie_block *sblk;
struct vm_save_area save_area;
+ uint8_t *crycb; /* Crypto Control Block */
/* Ptr to first guest page */
uint8_t *guest_mem;
};
@@ -200,5 +201,7 @@ extern void sie64a(struct kvm_s390_sie_block *sblk, struct vm_save_area *save_ar
void sie_expect_validity(void);
void sie_check_validity(uint16_t vir_exp);
void sie_handle_validity(struct vm *vm);
+void sie_guest_create(struct vm *vm, uint64_t guest_mem, uint64_t guest_mem_len);
+void sie_guest_destroy(struct vm *vm);
#endif /* _S390X_SIE_H_ */
diff --git a/s390x/mvpg-sie.c b/s390x/mvpg-sie.c
index 2ac91eec..71ae4f88 100644
--- a/s390x/mvpg-sie.c
+++ b/s390x/mvpg-sie.c
@@ -110,22 +110,7 @@ static void setup_guest(void)
/* The first two pages are the lowcore */
guest_instr = guest + PAGE_SIZE * 2;
- vm.sblk = alloc_page();
-
- vm.sblk->cpuflags = CPUSTAT_ZARCH | CPUSTAT_RUNNING;
- vm.sblk->prefix = 0;
- /*
- * Pageable guest with the same ASCE as the test programm, but
- * the guest memory 0x0 is offset to start at the allocated
- * guest pages and end after 1MB.
- *
- * It's not pretty but faster and easier than managing guest ASCEs.
- */
- vm.sblk->mso = (u64)guest;
- vm.sblk->msl = (u64)guest;
- vm.sblk->ihcpu = 0xffff;
-
- vm.sblk->crycbd = (uint64_t)alloc_page();
+ sie_guest_create(&vm, (uint64_t)guest, HPAGE_SIZE);
vm.sblk->gpsw.addr = PAGE_SIZE * 4;
vm.sblk->gpsw.mask = 0x0000000180000000ULL;
@@ -150,6 +135,7 @@ int main(void)
setup_guest();
test_mvpg();
test_mvpg_pei();
+ sie_guest_destroy(&vm);
done:
report_prefix_pop();
diff --git a/s390x/sie.c b/s390x/sie.c
index 5c798a9e..9cb9b055 100644
--- a/s390x/sie.c
+++ b/s390x/sie.c
@@ -84,22 +84,7 @@ static void setup_guest(void)
/* The first two pages are the lowcore */
guest_instr = guest + PAGE_SIZE * 2;
- vm.sblk = alloc_page();
-
- vm.sblk->cpuflags = CPUSTAT_ZARCH | CPUSTAT_RUNNING;
- vm.sblk->prefix = 0;
- /*
- * Pageable guest with the same ASCE as the test programm, but
- * the guest memory 0x0 is offset to start at the allocated
- * guest pages and end after 1MB.
- *
- * It's not pretty but faster and easier than managing guest ASCEs.
- */
- vm.sblk->mso = (u64)guest;
- vm.sblk->msl = (u64)guest;
- vm.sblk->ihcpu = 0xffff;
-
- vm.sblk->crycbd = (uint64_t)alloc_page();
+ sie_guest_create(&vm, (uint64_t)guest, HPAGE_SIZE);
}
int main(void)
@@ -112,6 +97,8 @@ int main(void)
setup_guest();
test_diags();
+ sie_guest_destroy(&vm);
+
done:
report_prefix_pop();
return report_summary();
--
2.31.1
^ permalink raw reply related [flat|nested] 13+ messages in thread* [kvm-unit-tests GIT PULL 07/11] lib: s390x: sie: Move sie function into library
2021-08-16 13:20 [kvm-unit-tests GIT PULL 00/11] s390x update 2021-16-08 Janosch Frank
` (5 preceding siblings ...)
2021-08-16 13:20 ` [kvm-unit-tests GIT PULL 06/11] s390x: lib: sie: Add struct vm (de)initialization functions Janosch Frank
@ 2021-08-16 13:20 ` Janosch Frank
2021-08-16 13:20 ` [kvm-unit-tests GIT PULL 08/11] s390x: lib: Simplify stsi_get_fc and move it to library Janosch Frank
` (4 subsequent siblings)
11 siblings, 0 replies; 13+ messages in thread
From: Janosch Frank @ 2021-08-16 13:20 UTC (permalink / raw)
To: pbonzini
Cc: kvm, frankja, david, borntraeger, cohuck, linux-s390, imbrenda, thuth
Time to deduplicate more code.
Signed-off-by: Janosch Frank <frankja@linux.ibm.com>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Reviewed-by: Claudio Imbrenda <imbrenda@linux.ibm.com>
---
lib/s390x/sie.c | 15 ++++++++++++++-
lib/s390x/sie.h | 1 +
s390x/mvpg-sie.c | 13 -------------
s390x/sie.c | 17 -----------------
4 files changed, 15 insertions(+), 31 deletions(-)
diff --git a/lib/s390x/sie.c b/lib/s390x/sie.c
index 1a93e518..b965b314 100644
--- a/lib/s390x/sie.c
+++ b/lib/s390x/sie.c
@@ -1,6 +1,6 @@
/* SPDX-License-Identifier: GPL-2.0-only */
/*
- * Virtualization library that speeds up managing guests.
+ * Library for managing various aspects of guests
*
* Copyright (c) 2021 IBM Corp
*
@@ -42,6 +42,19 @@ void sie_handle_validity(struct vm *vm)
validity_expected = false;
}
+void sie(struct vm *vm)
+{
+ /* Reset icptcode so we don't trip over it below */
+ vm->sblk->icptcode = 0;
+
+ while (vm->sblk->icptcode == 0) {
+ sie64a(vm->sblk, &vm->save_area);
+ sie_handle_validity(vm);
+ }
+ vm->save_area.guest.grs[14] = vm->sblk->gg14;
+ vm->save_area.guest.grs[15] = vm->sblk->gg15;
+}
+
/* Initializes the struct vm members like the SIE control block. */
void sie_guest_create(struct vm *vm, uint64_t guest_mem, uint64_t guest_mem_len)
{
diff --git a/lib/s390x/sie.h b/lib/s390x/sie.h
index 946bd164..ca514ef3 100644
--- a/lib/s390x/sie.h
+++ b/lib/s390x/sie.h
@@ -198,6 +198,7 @@ struct vm {
extern void sie_entry(void);
extern void sie_exit(void);
extern void sie64a(struct kvm_s390_sie_block *sblk, struct vm_save_area *save_area);
+void sie(struct vm *vm);
void sie_expect_validity(void);
void sie_check_validity(uint16_t vir_exp);
void sie_handle_validity(struct vm *vm);
diff --git a/s390x/mvpg-sie.c b/s390x/mvpg-sie.c
index 71ae4f88..70d2fcfa 100644
--- a/s390x/mvpg-sie.c
+++ b/s390x/mvpg-sie.c
@@ -32,19 +32,6 @@ extern const char _binary_s390x_snippets_c_mvpg_snippet_gbin_start[];
extern const char _binary_s390x_snippets_c_mvpg_snippet_gbin_end[];
int binary_size;
-static void sie(struct vm *vm)
-{
- /* Reset icptcode so we don't trip over it below */
- vm->sblk->icptcode = 0;
-
- while (vm->sblk->icptcode == 0) {
- sie64a(vm->sblk, &vm->save_area);
- sie_handle_validity(vm);
- }
- vm->save_area.guest.grs[14] = vm->sblk->gg14;
- vm->save_area.guest.grs[15] = vm->sblk->gg15;
-}
-
static void test_mvpg_pei(void)
{
uint64_t **pei_dst = (uint64_t **)((uintptr_t) vm.sblk + 0xc0);
diff --git a/s390x/sie.c b/s390x/sie.c
index 9cb9b055..ed2c3263 100644
--- a/s390x/sie.c
+++ b/s390x/sie.c
@@ -24,22 +24,6 @@ static u8 *guest;
static u8 *guest_instr;
static struct vm vm;
-
-static void sie(struct vm *vm)
-{
- while (vm->sblk->icptcode == 0) {
- sie64a(vm->sblk, &vm->save_area);
- sie_handle_validity(vm);
- }
- vm->save_area.guest.grs[14] = vm->sblk->gg14;
- vm->save_area.guest.grs[15] = vm->sblk->gg15;
-}
-
-static void sblk_cleanup(struct vm *vm)
-{
- vm->sblk->icptcode = 0;
-}
-
static void test_diag(u32 instr)
{
vm.sblk->gpsw.addr = PAGE_SIZE * 2;
@@ -51,7 +35,6 @@ static void test_diag(u32 instr)
report(vm.sblk->icptcode == ICPT_INST &&
vm.sblk->ipa == instr >> 16 && vm.sblk->ipb == instr << 16,
"Intercept data");
- sblk_cleanup(&vm);
}
static struct {
--
2.31.1
^ permalink raw reply related [flat|nested] 13+ messages in thread* [kvm-unit-tests GIT PULL 08/11] s390x: lib: Simplify stsi_get_fc and move it to library
2021-08-16 13:20 [kvm-unit-tests GIT PULL 00/11] s390x update 2021-16-08 Janosch Frank
` (6 preceding siblings ...)
2021-08-16 13:20 ` [kvm-unit-tests GIT PULL 07/11] lib: s390x: sie: Move sie function into library Janosch Frank
@ 2021-08-16 13:20 ` Janosch Frank
2021-08-16 13:20 ` [kvm-unit-tests GIT PULL 09/11] lib: s390x: Add 0x3d, 0x3e and 0x3f PGM constants Janosch Frank
` (3 subsequent siblings)
11 siblings, 0 replies; 13+ messages in thread
From: Janosch Frank @ 2021-08-16 13:20 UTC (permalink / raw)
To: pbonzini
Cc: kvm, frankja, david, borntraeger, cohuck, linux-s390, imbrenda, thuth
From: Pierre Morel <pmorel@linux.ibm.com>
stsi_get_fc is now needed in multiple tests.
As it does not need to store information but only returns
the machine level, suppress the address parameter.
Signed-off-by: Pierre Morel <pmorel@linux.ibm.com>
Reviewed-by: Claudio Imbrenda <imbrenda@linux.ibm.com>
Reviewed-by: Janosch Frank <frankja@linux.ibm.com>
Link: https://lore.kernel.org/kvm/1628612544-25130-3-git-send-email-pmorel@linux.ibm.com/
Signed-off-by: Janosch Frank <frankja@linux.ibm.com>
---
lib/s390x/asm/arch_def.h | 16 ++++++++++++++++
s390x/stsi.c | 20 ++------------------
2 files changed, 18 insertions(+), 18 deletions(-)
diff --git a/lib/s390x/asm/arch_def.h b/lib/s390x/asm/arch_def.h
index 15cf7d48..2f70d840 100644
--- a/lib/s390x/asm/arch_def.h
+++ b/lib/s390x/asm/arch_def.h
@@ -328,6 +328,22 @@ static inline int stsi(void *addr, int fc, int sel1, int sel2)
return cc;
}
+static inline unsigned long stsi_get_fc(void)
+{
+ register unsigned long r0 asm("0") = 0;
+ register unsigned long r1 asm("1") = 0;
+ int cc;
+
+ asm volatile("stsi 0\n"
+ "ipm %[cc]\n"
+ "srl %[cc],28\n"
+ : "+d" (r0), [cc] "=d" (cc)
+ : "d" (r1)
+ : "cc", "memory");
+ assert(!cc);
+ return r0 >> 28;
+}
+
static inline int servc(uint32_t command, unsigned long sccb)
{
int cc;
diff --git a/s390x/stsi.c b/s390x/stsi.c
index 87d48047..391f8849 100644
--- a/s390x/stsi.c
+++ b/s390x/stsi.c
@@ -71,28 +71,12 @@ static void test_priv(void)
report_prefix_pop();
}
-static inline unsigned long stsi_get_fc(void *addr)
-{
- register unsigned long r0 asm("0") = 0;
- register unsigned long r1 asm("1") = 0;
- int cc;
-
- asm volatile("stsi 0(%[addr])\n"
- "ipm %[cc]\n"
- "srl %[cc],28\n"
- : "+d" (r0), [cc] "=d" (cc)
- : "d" (r1), [addr] "a" (addr)
- : "cc", "memory");
- assert(!cc);
- return r0 >> 28;
-}
-
static void test_fc(void)
{
report(stsi(pagebuf, 7, 0, 0) == 3, "invalid fc");
report(stsi(pagebuf, 1, 0, 1) == 3, "invalid selector 1");
report(stsi(pagebuf, 1, 1, 0) == 3, "invalid selector 2");
- report(stsi_get_fc(pagebuf) >= 2, "query fc >= 2");
+ report(stsi_get_fc() >= 2, "query fc >= 2");
}
static void test_3_2_2(void)
@@ -112,7 +96,7 @@ static void test_3_2_2(void)
report_prefix_push("3.2.2");
/* Is the function code available at all? */
- if (stsi_get_fc(pagebuf) < 3) {
+ if (stsi_get_fc() < 3) {
report_skip("Running under lpar, no level 3 to test.");
goto out;
}
--
2.31.1
^ permalink raw reply related [flat|nested] 13+ messages in thread* [kvm-unit-tests GIT PULL 09/11] lib: s390x: Add 0x3d, 0x3e and 0x3f PGM constants
2021-08-16 13:20 [kvm-unit-tests GIT PULL 00/11] s390x update 2021-16-08 Janosch Frank
` (7 preceding siblings ...)
2021-08-16 13:20 ` [kvm-unit-tests GIT PULL 08/11] s390x: lib: Simplify stsi_get_fc and move it to library Janosch Frank
@ 2021-08-16 13:20 ` Janosch Frank
2021-08-16 13:20 ` [kvm-unit-tests GIT PULL 10/11] lib: s390x: uv: Add rc 0x100 query error handling Janosch Frank
` (2 subsequent siblings)
11 siblings, 0 replies; 13+ messages in thread
From: Janosch Frank @ 2021-08-16 13:20 UTC (permalink / raw)
To: pbonzini
Cc: kvm, frankja, david, borntraeger, cohuck, linux-s390, imbrenda, thuth
For UV and format 4 SIE tests we need to handle the following PGM exceptions:
0x3d Secure Storage Access (non-secure CPU accesses secure storage)
0x3e Non-Secure Storage Access (secure CPU accesses non-secure storage)
0x3f Mapping of secure guest is wrong
Signed-off-by: Janosch Frank <frankja@linux.ibm.com>
Reviewed-by: Claudio Imbrenda <imbrenda@linux.ibm.com>
---
lib/s390x/asm/arch_def.h | 3 +++
lib/s390x/interrupt.c | 3 +++
2 files changed, 6 insertions(+)
diff --git a/lib/s390x/asm/arch_def.h b/lib/s390x/asm/arch_def.h
index 2f70d840..d9f51efb 100644
--- a/lib/s390x/asm/arch_def.h
+++ b/lib/s390x/asm/arch_def.h
@@ -177,6 +177,9 @@ _Static_assert(sizeof(struct lowcore) == 0x1900, "Lowcore size");
#define PGM_INT_CODE_REGION_FIRST_TRANS 0x39
#define PGM_INT_CODE_REGION_SECOND_TRANS 0x3a
#define PGM_INT_CODE_REGION_THIRD_TRANS 0x3b
+#define PGM_INT_CODE_SECURE_STOR_ACCESS 0x3d
+#define PGM_INT_CODE_NON_SECURE_STOR_ACCESS 0x3e
+#define PGM_INT_CODE_SECURE_STOR_VIOLATION 0x3f
#define PGM_INT_CODE_MONITOR_EVENT 0x40
#define PGM_INT_CODE_PER 0x80
#define PGM_INT_CODE_CRYPTO_OPERATION 0x119
diff --git a/lib/s390x/interrupt.c b/lib/s390x/interrupt.c
index 785b7355..01ded49d 100644
--- a/lib/s390x/interrupt.c
+++ b/lib/s390x/interrupt.c
@@ -115,6 +115,9 @@ static void fixup_pgm_int(struct stack_frame_int *stack)
case PGM_INT_CODE_REGION_THIRD_TRANS:
case PGM_INT_CODE_PER:
case PGM_INT_CODE_CRYPTO_OPERATION:
+ case PGM_INT_CODE_SECURE_STOR_ACCESS:
+ case PGM_INT_CODE_NON_SECURE_STOR_ACCESS:
+ case PGM_INT_CODE_SECURE_STOR_VIOLATION:
/* The interrupt was nullified, the old PSW points at the
* responsible instruction. Forward the PSW so we don't loop.
*/
--
2.31.1
^ permalink raw reply related [flat|nested] 13+ messages in thread* [kvm-unit-tests GIT PULL 10/11] lib: s390x: uv: Add rc 0x100 query error handling
2021-08-16 13:20 [kvm-unit-tests GIT PULL 00/11] s390x update 2021-16-08 Janosch Frank
` (8 preceding siblings ...)
2021-08-16 13:20 ` [kvm-unit-tests GIT PULL 09/11] lib: s390x: Add 0x3d, 0x3e and 0x3f PGM constants Janosch Frank
@ 2021-08-16 13:20 ` Janosch Frank
2021-08-16 13:20 ` [kvm-unit-tests GIT PULL 11/11] lib: s390x: Add PSW_MASK_64 Janosch Frank
2021-08-16 16:25 ` [kvm-unit-tests GIT PULL 00/11] s390x update 2021-16-08 Paolo Bonzini
11 siblings, 0 replies; 13+ messages in thread
From: Janosch Frank @ 2021-08-16 13:20 UTC (permalink / raw)
To: pbonzini
Cc: kvm, frankja, david, borntraeger, cohuck, linux-s390, imbrenda, thuth
Let's not get bitten by an extension of the query struct and handle
the rc 0x100 error properly which does indicate that the UV has more
data for us.
Signed-off-by: Janosch Frank <frankja@linux.ibm.com>
Reviewed-by: Claudio Imbrenda <imbrenda@linux.ibm.com>
---
lib/s390x/uv.c | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/lib/s390x/uv.c b/lib/s390x/uv.c
index fd9de944..c5c69c47 100644
--- a/lib/s390x/uv.c
+++ b/lib/s390x/uv.c
@@ -49,6 +49,8 @@ int uv_setup(void)
if (!test_facility(158))
return 0;
- assert(!uv_call(0, (u64)&uvcb_qui));
+ uv_call(0, (u64)&uvcb_qui);
+
+ assert(uvcb_qui.header.rc == 1 || uvcb_qui.header.rc == 0x100);
return 1;
}
--
2.31.1
^ permalink raw reply related [flat|nested] 13+ messages in thread* [kvm-unit-tests GIT PULL 11/11] lib: s390x: Add PSW_MASK_64
2021-08-16 13:20 [kvm-unit-tests GIT PULL 00/11] s390x update 2021-16-08 Janosch Frank
` (9 preceding siblings ...)
2021-08-16 13:20 ` [kvm-unit-tests GIT PULL 10/11] lib: s390x: uv: Add rc 0x100 query error handling Janosch Frank
@ 2021-08-16 13:20 ` Janosch Frank
2021-08-16 16:25 ` [kvm-unit-tests GIT PULL 00/11] s390x update 2021-16-08 Paolo Bonzini
11 siblings, 0 replies; 13+ messages in thread
From: Janosch Frank @ 2021-08-16 13:20 UTC (permalink / raw)
To: pbonzini
Cc: kvm, frankja, david, borntraeger, cohuck, linux-s390, imbrenda, thuth
Let's replace the magic 0x0000000180000000ULL numeric constants with
PSW_MASK_64 as it's used more often since the introduction of smp and
sie.
Signed-off-by: Janosch Frank <frankja@linux.ibm.com>
Reviewed-by: Claudio Imbrenda <imbrenda@linux.ibm.com>
---
lib/s390x/asm/arch_def.h | 3 +++
lib/s390x/smp.c | 2 +-
s390x/mvpg-sie.c | 2 +-
s390x/sie.c | 2 +-
s390x/skrf.c | 6 +++---
5 files changed, 9 insertions(+), 6 deletions(-)
diff --git a/lib/s390x/asm/arch_def.h b/lib/s390x/asm/arch_def.h
index d9f51efb..302ef1ff 100644
--- a/lib/s390x/asm/arch_def.h
+++ b/lib/s390x/asm/arch_def.h
@@ -46,6 +46,9 @@ struct psw {
#define PSW_MASK_DAT 0x0400000000000000UL
#define PSW_MASK_WAIT 0x0002000000000000UL
#define PSW_MASK_PSTATE 0x0001000000000000UL
+#define PSW_MASK_EA 0x0000000100000000UL
+#define PSW_MASK_BA 0x0000000080000000UL
+#define PSW_MASK_64 PSW_MASK_BA | PSW_MASK_EA;
#define CR0_EXTM_SCLP 0x0000000000000200UL
#define CR0_EXTM_EXTC 0x0000000000002000UL
diff --git a/lib/s390x/smp.c b/lib/s390x/smp.c
index ee68d676..228fe667 100644
--- a/lib/s390x/smp.c
+++ b/lib/s390x/smp.c
@@ -202,7 +202,7 @@ int smp_cpu_setup(uint16_t addr, struct psw psw)
cpu->lowcore->sw_int_psw.addr = psw.addr;
cpu->lowcore->sw_int_grs[14] = psw.addr;
cpu->lowcore->sw_int_grs[15] = (uint64_t)cpu->stack + (PAGE_SIZE * 4);
- lc->restart_new_psw.mask = 0x0000000180000000UL;
+ lc->restart_new_psw.mask = PSW_MASK_64;
lc->restart_new_psw.addr = (uint64_t)smp_cpu_setup_state;
lc->sw_int_crs[0] = 0x0000000000040000UL;
diff --git a/s390x/mvpg-sie.c b/s390x/mvpg-sie.c
index 70d2fcfa..ccc273b4 100644
--- a/s390x/mvpg-sie.c
+++ b/s390x/mvpg-sie.c
@@ -100,7 +100,7 @@ static void setup_guest(void)
sie_guest_create(&vm, (uint64_t)guest, HPAGE_SIZE);
vm.sblk->gpsw.addr = PAGE_SIZE * 4;
- vm.sblk->gpsw.mask = 0x0000000180000000ULL;
+ vm.sblk->gpsw.mask = PSW_MASK_64;
vm.sblk->ictl = ICTL_OPEREXC | ICTL_PINT;
/* Enable MVPG interpretation as we want to test KVM and not ourselves */
vm.sblk->eca = ECA_MVPGI;
diff --git a/s390x/sie.c b/s390x/sie.c
index ed2c3263..87575b29 100644
--- a/s390x/sie.c
+++ b/s390x/sie.c
@@ -27,7 +27,7 @@ static struct vm vm;
static void test_diag(u32 instr)
{
vm.sblk->gpsw.addr = PAGE_SIZE * 2;
- vm.sblk->gpsw.mask = 0x0000000180000000ULL;
+ vm.sblk->gpsw.mask = PSW_MASK_64;
memset(guest_instr, 0, PAGE_SIZE);
memcpy(guest_instr, &instr, 4);
diff --git a/s390x/skrf.c b/s390x/skrf.c
index 94e906a6..9488c32b 100644
--- a/s390x/skrf.c
+++ b/s390x/skrf.c
@@ -125,15 +125,15 @@ static void ecall_cleanup(void)
{
struct lowcore *lc = (void *)0x0;
- lc->ext_new_psw.mask = 0x0000000180000000UL;
lc->sw_int_crs[0] = 0x0000000000040000;
+ lc->ext_new_psw.mask = PSW_MASK_64;
/*
* PGM old contains the ext new PSW, we need to clean it up,
* so we don't get a special operation exception on the lpswe
* of pgm old.
*/
- lc->pgm_old_psw.mask = 0x0000000180000000UL;
+ lc->pgm_old_psw.mask = PSW_MASK_64;
check_pgm_int_code(PGM_INT_CODE_SPECIFICATION);
set_flag(1);
@@ -148,7 +148,7 @@ static void ecall_setup(void)
register_pgm_cleanup_func(ecall_cleanup);
expect_pgm_int();
/* Put a skey into the ext new psw */
- lc->ext_new_psw.mask = 0x00F0000180000000UL;
+ lc->ext_new_psw.mask = 0x00F0000000000000UL | PSW_MASK_64;
/* Open up ext masks */
ctl_set_bit(0, CTL0_EXTERNAL_CALL);
mask = extract_psw_mask();
--
2.31.1
^ permalink raw reply related [flat|nested] 13+ messages in thread* Re: [kvm-unit-tests GIT PULL 00/11] s390x update 2021-16-08
2021-08-16 13:20 [kvm-unit-tests GIT PULL 00/11] s390x update 2021-16-08 Janosch Frank
` (10 preceding siblings ...)
2021-08-16 13:20 ` [kvm-unit-tests GIT PULL 11/11] lib: s390x: Add PSW_MASK_64 Janosch Frank
@ 2021-08-16 16:25 ` Paolo Bonzini
11 siblings, 0 replies; 13+ messages in thread
From: Paolo Bonzini @ 2021-08-16 16:25 UTC (permalink / raw)
To: Janosch Frank
Cc: kvm, david, borntraeger, cohuck, linux-s390, imbrenda, thuth
On 16/08/21 15:20, Janosch Frank wrote:
> Dear Paolo,
>
> please merge or pull the following changes:
>
> - SPDX cleanup
> - SIE lib extensions
> - Fixes/cleanup in the lib
>
> MERGE:
> https://gitlab.com/kvm-unit-tests/kvm-unit-tests/-/merge_requests/15
>
> PIPELINE:
> https://gitlab.com/frankja/kvm-unit-tests/-/pipelines/353890035
>
> PULL:
> The following changes since commit c90c646d7381c99ac7d9d7812bd8535214458978:
>
> access: treat NX as reserved if EFER.NXE=0 (2021-08-13 07:29:28 -0400)
>
> are available in the Git repository at:
>
> https://gitlab.com/frankja/kvm-unit-tests.git s390x-pull-2021-16-08
>
> for you to fetch changes up to 454da83a513761a0cd2bfda08f335735f345ef87:
>
> lib: s390x: Add PSW_MASK_64 (2021-08-16 11:28:02 +0000)
>
> Janosch Frank (10):
> s390x: Add SPDX and header comments for s390x/* and lib/s390x/*
> s390x: Add SPDX and header comments for the snippets folder
> s390x: Fix my mail address in the headers
> s390x: sie: Add sie lib validity handling
> s390x: lib: Introduce HPAGE_* constants
> s390x: lib: sie: Add struct vm (de)initialization functions
> lib: s390x: sie: Move sie function into library
> lib: s390x: Add 0x3d, 0x3e and 0x3f PGM constants
> lib: s390x: uv: Add rc 0x100 query error handling
> lib: s390x: Add PSW_MASK_64
>
> Pierre Morel (1):
> s390x: lib: Simplify stsi_get_fc and move it to library
>
> lib/s390x/asm/arch_def.h | 22 +++++++++
> lib/s390x/asm/mem.h | 2 +-
> lib/s390x/asm/page.h | 4 ++
> lib/s390x/interrupt.c | 3 ++
> lib/s390x/mmu.h | 2 +-
> lib/s390x/sie.c | 83 +++++++++++++++++++++++++++++++++
> lib/s390x/sie.h | 7 +++
> lib/s390x/smp.c | 2 +-
> lib/s390x/stack.c | 2 +-
> lib/s390x/uv.c | 13 +++++-
> s390x/Makefile | 1 +
> s390x/gs.c | 2 +-
> s390x/iep.c | 2 +-
> s390x/mvpg-sie.c | 42 +++++------------
> s390x/sie.c | 53 ++++++---------------
> s390x/skrf.c | 6 +--
> s390x/snippets/c/cstart.S | 9 ++++
> s390x/snippets/c/mvpg-snippet.c | 9 ++++
> s390x/stsi.c | 20 +-------
> s390x/vector.c | 2 +-
> 20 files changed, 188 insertions(+), 98 deletions(-)
> create mode 100644 lib/s390x/sie.c
>
Merged, thanks!
Paolo
^ permalink raw reply [flat|nested] 13+ messages in thread