* [PATCH v14 00/11] s390x: CPU Topology
@ 2023-01-05 14:53 Pierre Morel
2023-01-05 14:53 ` [PATCH v14 01/11] s390x/cpu topology: adding s390 specificities to CPU topology Pierre Morel
` (10 more replies)
0 siblings, 11 replies; 80+ messages in thread
From: Pierre Morel @ 2023-01-05 14:53 UTC (permalink / raw)
To: qemu-s390x
Cc: qemu-devel, borntraeger, pasic, richard.henderson, david, thuth,
cohuck, mst, pbonzini, kvm, ehabkost, marcel.apfelbaum, eblake,
armbru, seiden, nrb, scgl, frankja, berrange, clg
Hi,
This new series adds three features:
1) the implementation of drawers and books in the topology geometry
and adds polarization and dedication as topology modifiers.
2) The Maximum Nested topology level (MNEST) can be read with SCLP
read info.
3) Dynamic modification of the CPU topology using the QEMU monitor.
Implementation discussions
==========================
CPU models
----------
Since the facility 11, S390_FEAT_CONFIGURATION_TOPOLOGY is already
in the CPU model for old QEMU we could not activate it as usual from
KVM but needed a KVM capability: KVM_CAP_S390_CPU_TOPOLOGY.
Checking and enabling this capability enables facility 11,
S390_FEAT_CONFIGURATION_TOPOLOGY.
It is the responsibility of the admin to ensure the same CPU
model for source and target host in a migration.
Migration
---------
When the target guest is started, the Multi-processor Topology Change
Report (MTCR) bit is set during the creation of the vCPU by KVM.
We do not need to migrate its state, in the worst case, the target
guest will see the MTCR and actualize its view of the topology
without necessity, but this will be done only one time.
Reset
-----
Reseting the topology is done during subsystem reset, the
polarization is reset to horizontal polarization.
Topology attributes
-------------------
The topology attributes are carried by the CPU object and defined
on object creation.
In the case the new attributes, socket, book, drawer, dedicated,
polarity are not provided QEMU provides defaults values.
- Geometry defaults
The geometry default are based on the core-id of the core to
fill the geometry in a monotone way starting with drawer 0,
book 0, and filling socket 0 with the number of cores per socket,
then filling socket 1, socket 2 ... etc until the book is complete
and all books until the first drawer is complete before starting with
the next drawer.
This allows to keep existing start scripts and Libvirt existing
interface until it is extended.
- Modifiers defaults
Default polarization is horizontal
Default dedication is not dedicated.
Dynamic topology modification
-----------------------------
QAPI interface is extended with:
- a command: 'change-topology'
- a query: 'query-topology'
- a event: 'POLARITY_CHANGE'
The admin may use query-topology to verify the topology provided
to the guest and change-topology to modify it.
The event POLARITY_CHANGE is sent when the guest uses the PTF
instruction to request a polarization change. In that case, the
admin is supposed to modify the CPU provisioning accordingly.
Testing
=======
To use the QEMU patches, you will need Linux V6-rc1 or newer,
or use the following Linux mainline patches:
f5ecfee94493 2022-07-20 KVM: s390: resetting the Topology-Change-Report
24fe0195bc19 2022-07-20 KVM: s390: guest support for topology function
0130337ec45b 2022-07-20 KVM: s390: Cleanup ipte lock access and SIIF fac..
Currently this code is for KVM only, I have no idea if it is interesting
to provide a TCG patch. If ever it will be done in another series.
Documentation
=============
To have a better understanding of the S390x CPU Topology and its
implementation in QEMU you can have a look at the documentation in the
last patch of this series.
The admin will want to match the host and the guest topology, taking
into account that the guest does not recognize multithreading.
Consequently, two vCPU assigned to threads of the same real CPU should
preferably be assigned to the same socket of the guest machine.
Regards,
Pierre
Pierre Morel (11):
s390x/cpu topology: adding s390 specificities to CPU topology
s390x/cpu topology: add topology entries on CPU hotplug
target/s390x/cpu topology: handle STSI(15) and build the SYSIB
s390x/sclp: reporting the maximum nested topology entries
s390x/cpu topology: resetting the Topology-Change-Report
s390x/cpu topology: interception of PTF instruction
target/s390x/cpu topology: activating CPU topology
qapi/s390/cpu topology: change-topology monitor command
qapi/s390/cpu topology: monitor query topology information
qapi/s390/cpu topology: POLARITY_CHANGE qapi event
docs/s390x/cpu topology: document s390x cpu topology
docs/system/s390x/cpu-topology.rst | 292 ++++++++++++++
docs/system/target-s390x.rst | 1 +
qapi/machine-target.json | 116 ++++++
qapi/machine.json | 14 +-
include/hw/boards.h | 10 +-
include/hw/s390x/cpu-topology.h | 77 ++++
include/hw/s390x/s390-virtio-ccw.h | 6 +
include/hw/s390x/sclp.h | 4 +-
include/monitor/hmp.h | 2 +
target/s390x/cpu.h | 86 ++++
target/s390x/kvm/kvm_s390x.h | 1 +
hw/core/machine-smp.c | 48 ++-
hw/core/machine.c | 4 +
hw/s390x/cpu-topology.c | 604 +++++++++++++++++++++++++++++
hw/s390x/s390-virtio-ccw.c | 13 +
hw/s390x/sclp.c | 4 +
softmmu/vl.c | 6 +
target/s390x/cpu-sysemu.c | 29 ++
target/s390x/cpu.c | 10 +
target/s390x/cpu_models.c | 1 +
target/s390x/kvm/cpu_topology.c | 136 +++++++
target/s390x/kvm/kvm.c | 46 ++-
hmp-commands-info.hx | 16 +
hmp-commands.hx | 16 +
hw/s390x/meson.build | 1 +
qemu-options.hx | 6 +-
target/s390x/kvm/meson.build | 3 +-
27 files changed, 1537 insertions(+), 15 deletions(-)
create mode 100644 docs/system/s390x/cpu-topology.rst
create mode 100644 include/hw/s390x/cpu-topology.h
create mode 100644 hw/s390x/cpu-topology.c
create mode 100644 target/s390x/kvm/cpu_topology.c
--
2.31.1
Since v13:
- Suppress the topology device to simplify the code
(Cedric)
- moved reset of MTCR from device reset into subsystem
reset and removed previous reviewed-by from Nico and
Janis
- No need for Migration
- No need for machine dependencies
(Christian, Thomas)
- Adding all features, drawer/book and dynamic
(Cedric)
- since v12
- suppress new CPU flag "disable-topology" just use ctop
- no use of special fields in CCW machine or in CPU
- modifications in documentation
- insert documentation in tree
(Cedric)
- moved cpu-topology.c from target/s390 to target/s390/kvm
to compile smoothly (without topology) for TCG
(Cedric)
- since v11
- new CPU flag "disable-topology"
I would have take "topology" if I was able to have
it false on default.
(Christian, Thomas)
- Build the topology during the interception of the
STSI instruction.
(Cedric)
- return CC3 in case the calculated SYSIB length is
greater than 4096.
(Janis)
- minor corections on documentation
- since v10
- change machine attribute "topology-disable" to "topology"
(Cedric)
- Add preliminary patch for machine properties
(Cedric)
- Use next machine as 7.2
(Cedric / Connie)
- Remove unecessary mutex
(Thomas)
- use ENOTSUP return value for kvm_s390_topology_set_mtcr()
(Cedric)
- Add explanation on container and cpu TLEs
(Thomas)
- use again cpu and socket count in topology structure
(Cedric)
- Suppress the S390TopoTLE structure and integrate
the TLE masks to the socket structure.
(-)
- the STSI instruction now finds the topology from the machine
(Cedric)
- since v9
- remove books and drawers
- remove thread denying and replace with a merge
of cores * threads to specify the CPUs available
to the guest
- add a class option to avoid topology on older
machines
(Cedric)
- Allocate a SYSIB buffer of the maximal length to
avoid overflow.
(Nico, Janis)
- suppress redundancy of smp parameters in topology
and use directly the machine smp structure
- Early check for topology support
(Cedric)
- since v8
- Linux patches are now mainline
- simplification of the implementation
(Janis)
- Migration, new machine definition
(Thomas)
- Documentation
- since v7
- Coherence with the Linux patch series changes for MTCR get
(Pierre)
- check return values during new CPU creation
(Thomas)
- Improving codding style and argument usages
(Thomas)
- since v6
- Changes on smp args in qemu-options
(Daniel)
- changed comments in machine.jason
(Daniel)
- Added reset
(Janosch)
- since v5
- rebasing on newer QEMU version
- reworked most lines above 80 characters.
- since v4
- Added drawer and books to topology
- Added numa topology
- Added documentation
- since v3
- Added migration
(Thomas)
- Separated STSI instruction from KVM to prepare TCG
(Thomas)
- Take care of endianess to prepare TCG
(Thomas)
- Added comments on STSI CPU container and PFT instruction
(Thomas)
- Moved enabling the instructions as the last patch
(Thomas)
^ permalink raw reply [flat|nested] 80+ messages in thread
* [PATCH v14 01/11] s390x/cpu topology: adding s390 specificities to CPU topology
2023-01-05 14:53 [PATCH v14 00/11] s390x: CPU Topology Pierre Morel
@ 2023-01-05 14:53 ` Pierre Morel
2023-01-10 11:37 ` Thomas Huth
2023-01-13 16:58 ` Nina Schoetterl-Glausch
2023-01-05 14:53 ` [PATCH v14 02/11] s390x/cpu topology: add topology entries on CPU hotplug Pierre Morel
` (9 subsequent siblings)
10 siblings, 2 replies; 80+ messages in thread
From: Pierre Morel @ 2023-01-05 14:53 UTC (permalink / raw)
To: qemu-s390x
Cc: qemu-devel, borntraeger, pasic, richard.henderson, david, thuth,
cohuck, mst, pbonzini, kvm, ehabkost, marcel.apfelbaum, eblake,
armbru, seiden, nrb, scgl, frankja, berrange, clg
S390 adds two new SMP levels, drawers and books to the CPU
topology.
The S390 CPU have specific toplogy features like dedication
and polarity to give to the guest indications on the host
vCPUs scheduling and help the guest take the best decisions
on the scheduling of threads on the vCPUs.
Let us provide the SMP properties with books and drawers levels
and S390 CPU with dedication and polarity,
Signed-off-by: Pierre Morel <pmorel@linux.ibm.com>
---
qapi/machine.json | 14 ++++++++--
include/hw/boards.h | 10 ++++++-
include/hw/s390x/cpu-topology.h | 23 ++++++++++++++++
target/s390x/cpu.h | 6 +++++
hw/core/machine-smp.c | 48 ++++++++++++++++++++++++++++-----
hw/core/machine.c | 4 +++
hw/s390x/s390-virtio-ccw.c | 2 ++
softmmu/vl.c | 6 +++++
target/s390x/cpu.c | 10 +++++++
qemu-options.hx | 6 +++--
10 files changed, 117 insertions(+), 12 deletions(-)
create mode 100644 include/hw/s390x/cpu-topology.h
diff --git a/qapi/machine.json b/qapi/machine.json
index b9228a5e46..ff8f2b0e84 100644
--- a/qapi/machine.json
+++ b/qapi/machine.json
@@ -900,13 +900,15 @@
# a CPU is being hotplugged.
#
# @node-id: NUMA node ID the CPU belongs to
-# @socket-id: socket number within node/board the CPU belongs to
+# @drawer-id: drawer number within node/board the CPU belongs to
+# @book-id: book number within drawer/node/board the CPU belongs to
+# @socket-id: socket number within book/node/board the CPU belongs to
# @die-id: die number within socket the CPU belongs to (since 4.1)
# @cluster-id: cluster number within die the CPU belongs to (since 7.1)
# @core-id: core number within cluster the CPU belongs to
# @thread-id: thread number within core the CPU belongs to
#
-# Note: currently there are 6 properties that could be present
+# Note: currently there are 8 properties that could be present
# but management should be prepared to pass through other
# properties with device_add command to allow for future
# interface extension. This also requires the filed names to be kept in
@@ -916,6 +918,8 @@
##
{ 'struct': 'CpuInstanceProperties',
'data': { '*node-id': 'int',
+ '*drawer-id': 'int',
+ '*book-id': 'int',
'*socket-id': 'int',
'*die-id': 'int',
'*cluster-id': 'int',
@@ -1465,6 +1469,10 @@
#
# @cpus: number of virtual CPUs in the virtual machine
#
+# @drawers: number of drawers in the CPU topology
+#
+# @books: number of books in the CPU topology
+#
# @sockets: number of sockets in the CPU topology
#
# @dies: number of dies per socket in the CPU topology
@@ -1481,6 +1489,8 @@
##
{ 'struct': 'SMPConfiguration', 'data': {
'*cpus': 'int',
+ '*drawers': 'int',
+ '*books': 'int',
'*sockets': 'int',
'*dies': 'int',
'*clusters': 'int',
diff --git a/include/hw/boards.h b/include/hw/boards.h
index d18d6d0073..239fc017b0 100644
--- a/include/hw/boards.h
+++ b/include/hw/boards.h
@@ -130,11 +130,15 @@ typedef struct {
* @prefer_sockets - whether sockets are preferred over cores in smp parsing
* @dies_supported - whether dies are supported by the machine
* @clusters_supported - whether clusters are supported by the machine
+ * @books_supported - whether books are supported by the machine
+ * @drawers_supported - whether drawers are supported by the machine
*/
typedef struct {
bool prefer_sockets;
bool dies_supported;
bool clusters_supported;
+ bool books_supported;
+ bool drawers_supported;
} SMPCompatProps;
/**
@@ -299,7 +303,9 @@ typedef struct DeviceMemoryState {
/**
* CpuTopology:
* @cpus: the number of present logical processors on the machine
- * @sockets: the number of sockets on the machine
+ * @drawers: the number of drawers on the machine
+ * @books: the number of books in one drawer
+ * @sockets: the number of sockets in one book
* @dies: the number of dies in one socket
* @clusters: the number of clusters in one die
* @cores: the number of cores in one cluster
@@ -308,6 +314,8 @@ typedef struct DeviceMemoryState {
*/
typedef struct CpuTopology {
unsigned int cpus;
+ unsigned int drawers;
+ unsigned int books;
unsigned int sockets;
unsigned int dies;
unsigned int clusters;
diff --git a/include/hw/s390x/cpu-topology.h b/include/hw/s390x/cpu-topology.h
new file mode 100644
index 0000000000..d945b57fc3
--- /dev/null
+++ b/include/hw/s390x/cpu-topology.h
@@ -0,0 +1,23 @@
+/*
+ * CPU Topology
+ *
+ * Copyright IBM Corp. 2022
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or (at
+ * your option) any later version. See the COPYING file in the top-level
+ * directory.
+ */
+#ifndef HW_S390X_CPU_TOPOLOGY_H
+#define HW_S390X_CPU_TOPOLOGY_H
+
+#define S390_TOPOLOGY_CPU_IFL 0x03
+
+#define S390_TOPOLOGY_POLARITY_HORIZONTAL 0x00
+#define S390_TOPOLOGY_POLARITY_VERTICAL_LOW 0x01
+#define S390_TOPOLOGY_POLARITY_VERTICAL_MEDIUM 0x02
+#define S390_TOPOLOGY_POLARITY_VERTICAL_HIGH 0x03
+
+#define S390_TOPOLOGY_SHARED 0x00
+#define S390_TOPOLOGY_DEDICATED 0x01
+
+#endif
diff --git a/target/s390x/cpu.h b/target/s390x/cpu.h
index 7d6d01325b..39ea63a416 100644
--- a/target/s390x/cpu.h
+++ b/target/s390x/cpu.h
@@ -131,6 +131,12 @@ struct CPUArchState {
#if !defined(CONFIG_USER_ONLY)
uint32_t core_id; /* PoP "CPU address", same as cpu_index */
+ int32_t socket_id;
+ int32_t book_id;
+ int32_t drawer_id;
+ int32_t dedicated;
+ int32_t polarity;
+ int32_t cpu_type;
uint64_t cpuid;
#endif
diff --git a/hw/core/machine-smp.c b/hw/core/machine-smp.c
index b39ed21e65..edabfdb67e 100644
--- a/hw/core/machine-smp.c
+++ b/hw/core/machine-smp.c
@@ -31,6 +31,14 @@ static char *cpu_hierarchy_to_string(MachineState *ms)
MachineClass *mc = MACHINE_GET_CLASS(ms);
GString *s = g_string_new(NULL);
+ if (mc->smp_props.drawers_supported) {
+ g_string_append_printf(s, " * drawers (%u)", ms->smp.drawers);
+ }
+
+ if (mc->smp_props.books_supported) {
+ g_string_append_printf(s, " * books (%u)", ms->smp.books);
+ }
+
g_string_append_printf(s, "sockets (%u)", ms->smp.sockets);
if (mc->smp_props.dies_supported) {
@@ -73,6 +81,8 @@ void machine_parse_smp_config(MachineState *ms,
{
MachineClass *mc = MACHINE_GET_CLASS(ms);
unsigned cpus = config->has_cpus ? config->cpus : 0;
+ unsigned drawers = config->has_drawers ? config->drawers : 0;
+ unsigned books = config->has_books ? config->books : 0;
unsigned sockets = config->has_sockets ? config->sockets : 0;
unsigned dies = config->has_dies ? config->dies : 0;
unsigned clusters = config->has_clusters ? config->clusters : 0;
@@ -85,6 +95,8 @@ void machine_parse_smp_config(MachineState *ms,
* explicit configuration like "cpus=0" is not allowed.
*/
if ((config->has_cpus && config->cpus == 0) ||
+ (config->has_drawers && config->drawers == 0) ||
+ (config->has_books && config->books == 0) ||
(config->has_sockets && config->sockets == 0) ||
(config->has_dies && config->dies == 0) ||
(config->has_clusters && config->clusters == 0) ||
@@ -111,6 +123,19 @@ void machine_parse_smp_config(MachineState *ms,
dies = dies > 0 ? dies : 1;
clusters = clusters > 0 ? clusters : 1;
+ if (!mc->smp_props.books_supported && books > 1) {
+ error_setg(errp, "books not supported by this machine's CPU topology");
+ return;
+ }
+ books = books > 0 ? books : 1;
+
+ if (!mc->smp_props.drawers_supported && drawers > 1) {
+ error_setg(errp,
+ "drawers not supported by this machine's CPU topology");
+ return;
+ }
+ drawers = drawers > 0 ? drawers : 1;
+
/* compute missing values based on the provided ones */
if (cpus == 0 && maxcpus == 0) {
sockets = sockets > 0 ? sockets : 1;
@@ -124,33 +149,41 @@ void machine_parse_smp_config(MachineState *ms,
if (sockets == 0) {
cores = cores > 0 ? cores : 1;
threads = threads > 0 ? threads : 1;
- sockets = maxcpus / (dies * clusters * cores * threads);
+ sockets = maxcpus /
+ (drawers * books * dies * clusters * cores * threads);
} else if (cores == 0) {
threads = threads > 0 ? threads : 1;
- cores = maxcpus / (sockets * dies * clusters * threads);
+ cores = maxcpus /
+ (drawers * books * sockets * dies * clusters * threads);
}
} else {
/* prefer cores over sockets since 6.2 */
if (cores == 0) {
sockets = sockets > 0 ? sockets : 1;
threads = threads > 0 ? threads : 1;
- cores = maxcpus / (sockets * dies * clusters * threads);
+ cores = maxcpus /
+ (drawers * books * sockets * dies * clusters * threads);
} else if (sockets == 0) {
threads = threads > 0 ? threads : 1;
- sockets = maxcpus / (dies * clusters * cores * threads);
+ sockets = maxcpus /
+ (drawers * books * dies * clusters * cores * threads);
}
}
/* try to calculate omitted threads at last */
if (threads == 0) {
- threads = maxcpus / (sockets * dies * clusters * cores);
+ threads = maxcpus /
+ (drawers * books * sockets * dies * clusters * cores);
}
}
- maxcpus = maxcpus > 0 ? maxcpus : sockets * dies * clusters * cores * threads;
+ maxcpus = maxcpus > 0 ? maxcpus : drawers * books * sockets * dies *
+ clusters * cores * threads;
cpus = cpus > 0 ? cpus : maxcpus;
ms->smp.cpus = cpus;
+ ms->smp.drawers = drawers;
+ ms->smp.books = books;
ms->smp.sockets = sockets;
ms->smp.dies = dies;
ms->smp.clusters = clusters;
@@ -159,7 +192,8 @@ void machine_parse_smp_config(MachineState *ms,
ms->smp.max_cpus = maxcpus;
/* sanity-check of the computed topology */
- if (sockets * dies * clusters * cores * threads != maxcpus) {
+ if (drawers * books * sockets * dies * clusters * cores * threads !=
+ maxcpus) {
g_autofree char *topo_msg = cpu_hierarchy_to_string(ms);
error_setg(errp, "Invalid CPU topology: "
"product of the hierarchy must match maxcpus: "
diff --git a/hw/core/machine.c b/hw/core/machine.c
index f589b92909..3c781fed07 100644
--- a/hw/core/machine.c
+++ b/hw/core/machine.c
@@ -827,6 +827,8 @@ static void machine_get_smp(Object *obj, Visitor *v, const char *name,
MachineState *ms = MACHINE(obj);
SMPConfiguration *config = &(SMPConfiguration){
.has_cpus = true, .cpus = ms->smp.cpus,
+ .has_drawers = true, .drawers = ms->smp.drawers,
+ .has_books = true, .books = ms->smp.books,
.has_sockets = true, .sockets = ms->smp.sockets,
.has_dies = true, .dies = ms->smp.dies,
.has_clusters = true, .clusters = ms->smp.clusters,
@@ -1092,6 +1094,8 @@ static void machine_initfn(Object *obj)
/* default to mc->default_cpus */
ms->smp.cpus = mc->default_cpus;
ms->smp.max_cpus = mc->default_cpus;
+ ms->smp.drawers = 1;
+ ms->smp.books = 1;
ms->smp.sockets = 1;
ms->smp.dies = 1;
ms->smp.clusters = 1;
diff --git a/hw/s390x/s390-virtio-ccw.c b/hw/s390x/s390-virtio-ccw.c
index f22f61b8b6..f3cc845d3b 100644
--- a/hw/s390x/s390-virtio-ccw.c
+++ b/hw/s390x/s390-virtio-ccw.c
@@ -733,6 +733,8 @@ static void ccw_machine_class_init(ObjectClass *oc, void *data)
mc->no_sdcard = 1;
mc->max_cpus = S390_MAX_CPUS;
mc->has_hotpluggable_cpus = true;
+ mc->smp_props.books_supported = true;
+ mc->smp_props.drawers_supported = true;
assert(!mc->get_hotplug_handler);
mc->get_hotplug_handler = s390_get_hotplug_handler;
mc->cpu_index_to_instance_props = s390_cpu_index_to_props;
diff --git a/softmmu/vl.c b/softmmu/vl.c
index 798e1dc933..e894ab32b0 100644
--- a/softmmu/vl.c
+++ b/softmmu/vl.c
@@ -724,6 +724,12 @@ static QemuOptsList qemu_smp_opts = {
{
.name = "cpus",
.type = QEMU_OPT_NUMBER,
+ }, {
+ .name = "drawers",
+ .type = QEMU_OPT_NUMBER,
+ }, {
+ .name = "books",
+ .type = QEMU_OPT_NUMBER,
}, {
.name = "sockets",
.type = QEMU_OPT_NUMBER,
diff --git a/target/s390x/cpu.c b/target/s390x/cpu.c
index 96562c516d..c9f980dbb5 100644
--- a/target/s390x/cpu.c
+++ b/target/s390x/cpu.c
@@ -35,6 +35,7 @@
#include "fpu/softfloat-helpers.h"
#include "disas/capstone.h"
#include "sysemu/tcg.h"
+#include "hw/s390x/cpu-topology.h"
#define CR0_RESET 0xE0UL
#define CR14_RESET 0xC2000000UL;
@@ -257,6 +258,15 @@ static gchar *s390_gdb_arch_name(CPUState *cs)
static Property s390x_cpu_properties[] = {
#if !defined(CONFIG_USER_ONLY)
DEFINE_PROP_UINT32("core-id", S390CPU, env.core_id, 0),
+ DEFINE_PROP_INT32("socket-id", S390CPU, env.socket_id, -1),
+ DEFINE_PROP_INT32("book-id", S390CPU, env.book_id, -1),
+ DEFINE_PROP_INT32("drawer-id", S390CPU, env.drawer_id, -1),
+ DEFINE_PROP_INT32("dedicated", S390CPU, env.dedicated,
+ S390_TOPOLOGY_SHARED),
+ DEFINE_PROP_INT32("polarity", S390CPU, env.polarity,
+ S390_TOPOLOGY_POLARITY_HORIZONTAL),
+ DEFINE_PROP_INT32("cpu-type", S390CPU, env.cpu_type,
+ S390_TOPOLOGY_CPU_IFL),
#endif
DEFINE_PROP_END_OF_LIST()
};
diff --git a/qemu-options.hx b/qemu-options.hx
index 7f99d15b23..8dc9a4c052 100644
--- a/qemu-options.hx
+++ b/qemu-options.hx
@@ -250,11 +250,13 @@ SRST
ERST
DEF("smp", HAS_ARG, QEMU_OPTION_smp,
- "-smp [[cpus=]n][,maxcpus=maxcpus][,sockets=sockets][,dies=dies][,clusters=clusters][,cores=cores][,threads=threads]\n"
+ "-smp [[cpus=]n][,maxcpus=maxcpus][,drawers=drawers][,books=books][,sockets=sockets][,dies=dies][,clusters=clusters][,cores=cores][,threads=threads]\n"
" set the number of initial CPUs to 'n' [default=1]\n"
" maxcpus= maximum number of total CPUs, including\n"
" offline CPUs for hotplug, etc\n"
- " sockets= number of sockets on the machine board\n"
+ " drawers= number of drawers on the machine board\n"
+ " books= number of books in one drawer\n"
+ " sockets= number of sockets in one book\n"
" dies= number of dies in one socket\n"
" clusters= number of clusters in one die\n"
" cores= number of cores in one cluster\n"
--
2.31.1
^ permalink raw reply related [flat|nested] 80+ messages in thread
* [PATCH v14 02/11] s390x/cpu topology: add topology entries on CPU hotplug
2023-01-05 14:53 [PATCH v14 00/11] s390x: CPU Topology Pierre Morel
2023-01-05 14:53 ` [PATCH v14 01/11] s390x/cpu topology: adding s390 specificities to CPU topology Pierre Morel
@ 2023-01-05 14:53 ` Pierre Morel
2023-01-10 13:00 ` Thomas Huth
2023-01-13 18:15 ` Nina Schoetterl-Glausch
2023-01-05 14:53 ` [PATCH v14 03/11] target/s390x/cpu topology: handle STSI(15) and build the SYSIB Pierre Morel
` (8 subsequent siblings)
10 siblings, 2 replies; 80+ messages in thread
From: Pierre Morel @ 2023-01-05 14:53 UTC (permalink / raw)
To: qemu-s390x
Cc: qemu-devel, borntraeger, pasic, richard.henderson, david, thuth,
cohuck, mst, pbonzini, kvm, ehabkost, marcel.apfelbaum, eblake,
armbru, seiden, nrb, scgl, frankja, berrange, clg
The topology information are attributes of the CPU and are
specified during the CPU device creation.
On hot plug, we gather the topology information on the core,
creates a list of topology entries, each entry contains a single
core mask of each core with identical topology and finaly we
orders the list in topological order.
The topological order is, from higher to lower priority:
- physical topology
- drawer
- book
- socket
- core origin, offset in 64bit increment from core 0.
- modifier attributes
- CPU type
- polarization entitlement
- dedication
The possibility to insert a CPU in a mask is dependent on the
number of cores allowed in a socket, a book or a drawer, the
checking is done during the hot plug of the CPU to have an
immediate answer.
If the complete topology is not specified, the core is added
in the physical topology based on its core ID and it gets
defaults values for the modifier attributes.
This way, starting QEMU without specifying the topology can
still get some adventage of the CPU topology.
Signed-off-by: Pierre Morel <pmorel@linux.ibm.com>
---
include/hw/s390x/cpu-topology.h | 48 ++++++
hw/s390x/cpu-topology.c | 293 ++++++++++++++++++++++++++++++++
hw/s390x/s390-virtio-ccw.c | 10 ++
hw/s390x/meson.build | 1 +
4 files changed, 352 insertions(+)
create mode 100644 hw/s390x/cpu-topology.c
diff --git a/include/hw/s390x/cpu-topology.h b/include/hw/s390x/cpu-topology.h
index d945b57fc3..b3fd752d8d 100644
--- a/include/hw/s390x/cpu-topology.h
+++ b/include/hw/s390x/cpu-topology.h
@@ -10,7 +10,11 @@
#ifndef HW_S390X_CPU_TOPOLOGY_H
#define HW_S390X_CPU_TOPOLOGY_H
+#include "qemu/queue.h"
+#include "hw/boards.h"
+
#define S390_TOPOLOGY_CPU_IFL 0x03
+#define S390_TOPOLOGY_MAX_ORIGIN ((63 + S390_MAX_CPUS) / 64)
#define S390_TOPOLOGY_POLARITY_HORIZONTAL 0x00
#define S390_TOPOLOGY_POLARITY_VERTICAL_LOW 0x01
@@ -20,4 +24,48 @@
#define S390_TOPOLOGY_SHARED 0x00
#define S390_TOPOLOGY_DEDICATED 0x01
+typedef union s390_topology_id {
+ uint64_t id;
+ struct {
+ uint64_t level_6:8; /* byte 0 BE */
+ uint64_t level_5:8; /* byte 1 BE */
+ uint64_t drawer:8; /* byte 2 BE */
+ uint64_t book:8; /* byte 3 BE */
+ uint64_t socket:8; /* byte 4 BE */
+ uint64_t rsrv:5;
+ uint64_t d:1;
+ uint64_t p:2; /* byte 5 BE */
+ uint64_t type:8; /* byte 6 BE */
+ uint64_t origin:2;
+ uint64_t core:6; /* byte 7 BE */
+ };
+} s390_topology_id;
+#define TOPO_CPU_MASK 0x000000000000003fUL
+
+typedef struct S390TopologyEntry {
+ s390_topology_id id;
+ QTAILQ_ENTRY(S390TopologyEntry) next;
+ uint64_t mask;
+} S390TopologyEntry;
+
+typedef struct S390Topology {
+ QTAILQ_HEAD(, S390TopologyEntry) list;
+ uint8_t *sockets;
+ CpuTopology *smp;
+} S390Topology;
+
+#ifdef CONFIG_KVM
+bool s390_has_topology(void);
+void s390_topology_set_cpu(MachineState *ms, S390CPU *cpu, Error **errp);
+#else
+static inline bool s390_has_topology(void)
+{
+ return false;
+}
+static inline void s390_topology_set_cpu(MachineState *ms,
+ S390CPU *cpu,
+ Error **errp) {}
+#endif
+extern S390Topology s390_topology;
+
#endif
diff --git a/hw/s390x/cpu-topology.c b/hw/s390x/cpu-topology.c
new file mode 100644
index 0000000000..438055c612
--- /dev/null
+++ b/hw/s390x/cpu-topology.c
@@ -0,0 +1,293 @@
+/*
+ * CPU Topology
+ *
+ * Copyright IBM Corp. 2022
+ * Author(s): Pierre Morel <pmorel@linux.ibm.com>
+
+ * This work is licensed under the terms of the GNU GPL, version 2 or (at
+ * your option) any later version. See the COPYING file in the top-level
+ * directory.
+ */
+
+#include "qemu/osdep.h"
+#include "qapi/error.h"
+#include "qemu/error-report.h"
+#include "hw/qdev-properties.h"
+#include "hw/boards.h"
+#include "qemu/typedefs.h"
+#include "target/s390x/cpu.h"
+#include "hw/s390x/s390-virtio-ccw.h"
+#include "hw/s390x/cpu-topology.h"
+
+/*
+ * s390_topology is used to keep the topology information.
+ * .list: queue the topology entries inside which
+ * we keep the information on the CPU topology.
+ *
+ * .smp: keeps track of the machine topology.
+ *
+ * .socket: tracks information on the count of cores per socket.
+ *
+ */
+S390Topology s390_topology = {
+ .list = QTAILQ_HEAD_INITIALIZER(s390_topology.list),
+ .sockets = NULL, /* will be initialized after the cpu model is realized */
+};
+
+/**
+ * s390_socket_nb:
+ * @id: s390_topology_id
+ *
+ * Returns the socket number used inside the socket array.
+ */
+static int s390_socket_nb(s390_topology_id id)
+{
+ return (id.socket + 1) * (id.book + 1) * (id.drawer + 1);
+}
+
+/**
+ * s390_has_topology:
+ *
+ * Return value: if the topology is supported by the machine.
+ */
+bool s390_has_topology(void)
+{
+ return false;
+}
+
+/**
+ * s390_topology_init:
+ * @ms: the machine state where the machine topology is defined
+ *
+ * Keep track of the machine topology.
+ * Allocate an array to keep the count of cores per socket.
+ * The index of the array starts at socket 0 from book 0 and
+ * drawer 0 up to the maximum allowed by the machine topology.
+ */
+static void s390_topology_init(MachineState *ms)
+{
+ CpuTopology *smp = &ms->smp;
+
+ s390_topology.smp = smp;
+ if (!s390_topology.sockets) {
+ s390_topology.sockets = g_new0(uint8_t, smp->sockets *
+ smp->books * smp->drawers);
+ }
+}
+
+/**
+ * s390_topology_from_cpu:
+ * @cpu: The S390CPU
+ *
+ * Initialize the topology id from the CPU environment.
+ */
+static s390_topology_id s390_topology_from_cpu(S390CPU *cpu)
+{
+ s390_topology_id topology_id;
+
+ topology_id.core = cpu->env.core_id;
+ topology_id.type = cpu->env.cpu_type;
+ topology_id.p = cpu->env.polarity;
+ topology_id.d = cpu->env.dedicated;
+ topology_id.socket = cpu->env.socket_id;
+ topology_id.book = cpu->env.book_id;
+ topology_id.drawer = cpu->env.drawer_id;
+
+ return topology_id;
+}
+
+/**
+ * s390_topology_set_entry:
+ * @entry: Topology entry to setup
+ * @id: topology id to use for the setup
+ *
+ * Set the core bit inside the topology mask and
+ * increments the number of cores for the socket.
+ */
+static void s390_topology_set_entry(S390TopologyEntry *entry,
+ s390_topology_id id)
+{
+ set_bit(63 - id.core, &entry->mask);
+ s390_topology.sockets[s390_socket_nb(id)]++;
+}
+
+/**
+ * s390_topology_new_entry:
+ * @id: s390_topology_id to add
+ *
+ * Allocate a new entry and initialize it.
+ *
+ * returns the newly allocated entry.
+ */
+static S390TopologyEntry *s390_topology_new_entry(s390_topology_id id)
+{
+ S390TopologyEntry *entry;
+
+ entry = g_malloc0(sizeof(S390TopologyEntry));
+ entry->id.id = id.id & ~TOPO_CPU_MASK;
+ s390_topology_set_entry(entry, id);
+
+ return entry;
+}
+
+/**
+ * s390_topology_insert:
+ *
+ * @id: s390_topology_id to insert.
+ *
+ * Parse the topology list to find if the entry already
+ * exist and add the core in it.
+ * If it does not exist, allocate a new entry and insert
+ * it in the queue from lower id to greater id.
+ */
+static void s390_topology_insert(s390_topology_id id)
+{
+ S390TopologyEntry *entry;
+ S390TopologyEntry *tmp = NULL;
+ uint64_t new_id;
+
+ new_id = id.id & ~TOPO_CPU_MASK;
+
+ /* First CPU to add to an entry */
+ if (QTAILQ_EMPTY(&s390_topology.list)) {
+ entry = s390_topology_new_entry(id);
+ QTAILQ_INSERT_HEAD(&s390_topology.list, entry, next);
+ return;
+ }
+
+ QTAILQ_FOREACH(tmp, &s390_topology.list, next) {
+ if (new_id == tmp->id.id) {
+ s390_topology_set_entry(tmp, id);
+ return;
+ } else if (new_id < tmp->id.id) {
+ entry = s390_topology_new_entry(id);
+ QTAILQ_INSERT_BEFORE(tmp, entry, next);
+ return;
+ }
+ }
+
+ entry = s390_topology_new_entry(id);
+ QTAILQ_INSERT_TAIL(&s390_topology.list, entry, next);
+}
+
+/**
+ * s390_topology_check:
+ * @errp: Error pointer
+ * id: s390_topology_id to be verified
+ *
+ * The function checks if the topology id fits inside the
+ * system topology.
+ */
+static void s390_topology_check(Error **errp, s390_topology_id id)
+{
+ CpuTopology *smp = s390_topology.smp;
+
+ if (id.socket > smp->sockets) {
+ error_setg(errp, "Unavailable socket: %d", id.socket);
+ return;
+ }
+ if (id.book > smp->books) {
+ error_setg(errp, "Unavailable book: %d", id.book);
+ return;
+ }
+ if (id.drawer > smp->drawers) {
+ error_setg(errp, "Unavailable drawer: %d", id.drawer);
+ return;
+ }
+ if (id.type != S390_TOPOLOGY_CPU_IFL) {
+ error_setg(errp, "Unknown cpu type: %d", id.type);
+ return;
+ }
+ /* Polarity and dedication can never be wrong */
+}
+
+/**
+ * s390_topology_cpu_default:
+ * @errp: Error pointer
+ * @cpu: pointer to a S390CPU
+ *
+ * Setup the default topology for unset attributes.
+ *
+ * The function accept only all all default values or all set values
+ * for the geometry topology.
+ *
+ * The function calculates the (drawer_id, book_id, socket_id)
+ * topology by filling the cores starting from the first socket
+ * (0, 0, 0) up to the last (smp->drawers, smp->books, smp->sockets).
+ *
+ */
+static void s390_topology_cpu_default(Error **errp, S390CPU *cpu)
+{
+ CpuTopology *smp = s390_topology.smp;
+ CPUS390XState *env = &cpu->env;
+
+ /* All geometry topology attributes must be set or all unset */
+ if ((env->socket_id < 0 || env->book_id < 0 || env->drawer_id < 0) &&
+ (env->socket_id >= 0 || env->book_id >= 0 || env->drawer_id >= 0)) {
+ error_setg(errp,
+ "Please define all or none of the topology geometry attributes");
+ return;
+ }
+
+ /* Check if one of the geometry topology is unset */
+ if (env->socket_id < 0) {
+ /* Calculate default geometry topology attributes */
+ env->socket_id = (env->core_id / smp->cores) % smp->sockets;
+ env->book_id = (env->core_id / (smp->sockets * smp->cores)) %
+ smp->books;
+ env->drawer_id = (env->core_id /
+ (smp->books * smp->sockets * smp->cores)) %
+ smp->drawers;
+ }
+}
+
+/**
+ * s390_topology_set_cpu:
+ * @ms: MachineState used to initialize the topology structure on
+ * first call.
+ * @cpu: the new S390CPU to insert in the topology structure
+ * @errp: the error pointer
+ *
+ * Called from CPU Hotplug to check and setup the CPU attributes
+ * before to insert the CPU in the topology.
+ */
+void s390_topology_set_cpu(MachineState *ms, S390CPU *cpu, Error **errp)
+{
+ Error *local_error = NULL;
+ s390_topology_id id;
+
+ /*
+ * We do not want to initialize the topology if the cpu model
+ * does not support topology consequently, we have to wait for
+ * the first CPU to be realized, which realizes the CPU model
+ * to initialize the topology structures.
+ *
+ * s390_topology_set_cpu() is called from the cpu hotplug.
+ */
+ if (!s390_topology.sockets) {
+ s390_topology_init(ms);
+ }
+
+ s390_topology_cpu_default(&local_error, cpu);
+ if (local_error) {
+ error_propagate(errp, local_error);
+ return;
+ }
+
+ id = s390_topology_from_cpu(cpu);
+
+ /* Check for space on the socket */
+ if (s390_topology.sockets[s390_socket_nb(id)] >=
+ s390_topology.smp->sockets) {
+ error_setg(&local_error, "No more space on socket");
+ return;
+ }
+
+ s390_topology_check(&local_error, id);
+ if (local_error) {
+ error_propagate(errp, local_error);
+ return;
+ }
+
+ s390_topology_insert(id);
+}
diff --git a/hw/s390x/s390-virtio-ccw.c b/hw/s390x/s390-virtio-ccw.c
index f3cc845d3b..c98b93a15f 100644
--- a/hw/s390x/s390-virtio-ccw.c
+++ b/hw/s390x/s390-virtio-ccw.c
@@ -44,6 +44,7 @@
#include "hw/s390x/pv.h"
#include "migration/blocker.h"
#include "qapi/visitor.h"
+#include "hw/s390x/cpu-topology.h"
static Error *pv_mig_blocker;
@@ -310,10 +311,19 @@ static void s390_cpu_plug(HotplugHandler *hotplug_dev,
{
MachineState *ms = MACHINE(hotplug_dev);
S390CPU *cpu = S390_CPU(dev);
+ Error *local_err = NULL;
g_assert(!ms->possible_cpus->cpus[cpu->env.core_id].cpu);
ms->possible_cpus->cpus[cpu->env.core_id].cpu = OBJECT(dev);
+ if (s390_has_topology()) {
+ s390_topology_set_cpu(ms, cpu, &local_err);
+ if (local_err) {
+ error_propagate(errp, local_err);
+ return;
+ }
+ }
+
if (dev->hotplugged) {
raise_irq_cpu_hotplug();
}
diff --git a/hw/s390x/meson.build b/hw/s390x/meson.build
index f291016fee..58dfbdff4f 100644
--- a/hw/s390x/meson.build
+++ b/hw/s390x/meson.build
@@ -24,6 +24,7 @@ s390x_ss.add(when: 'CONFIG_KVM', if_true: files(
's390-stattrib-kvm.c',
'pv.c',
's390-pci-kvm.c',
+ 'cpu-topology.c',
))
s390x_ss.add(when: 'CONFIG_TCG', if_true: files(
'tod-tcg.c',
--
2.31.1
^ permalink raw reply related [flat|nested] 80+ messages in thread
* [PATCH v14 03/11] target/s390x/cpu topology: handle STSI(15) and build the SYSIB
2023-01-05 14:53 [PATCH v14 00/11] s390x: CPU Topology Pierre Morel
2023-01-05 14:53 ` [PATCH v14 01/11] s390x/cpu topology: adding s390 specificities to CPU topology Pierre Morel
2023-01-05 14:53 ` [PATCH v14 02/11] s390x/cpu topology: add topology entries on CPU hotplug Pierre Morel
@ 2023-01-05 14:53 ` Pierre Morel
2023-01-10 14:29 ` Thomas Huth
2023-01-16 13:11 ` Nina Schoetterl-Glausch
2023-01-05 14:53 ` [PATCH v14 04/11] s390x/sclp: reporting the maximum nested topology entries Pierre Morel
` (7 subsequent siblings)
10 siblings, 2 replies; 80+ messages in thread
From: Pierre Morel @ 2023-01-05 14:53 UTC (permalink / raw)
To: qemu-s390x
Cc: qemu-devel, borntraeger, pasic, richard.henderson, david, thuth,
cohuck, mst, pbonzini, kvm, ehabkost, marcel.apfelbaum, eblake,
armbru, seiden, nrb, scgl, frankja, berrange, clg
On interception of STSI(15.1.x) the System Information Block
(SYSIB) is built from the list of pre-ordered topology entries.
Signed-off-by: Pierre Morel <pmorel@linux.ibm.com>
---
include/hw/s390x/cpu-topology.h | 3 +
include/hw/s390x/sclp.h | 1 +
target/s390x/cpu.h | 78 ++++++++++++++++++
target/s390x/kvm/cpu_topology.c | 136 ++++++++++++++++++++++++++++++++
target/s390x/kvm/kvm.c | 5 +-
target/s390x/kvm/meson.build | 3 +-
6 files changed, 224 insertions(+), 2 deletions(-)
create mode 100644 target/s390x/kvm/cpu_topology.c
diff --git a/include/hw/s390x/cpu-topology.h b/include/hw/s390x/cpu-topology.h
index b3fd752d8d..9571aa70e5 100644
--- a/include/hw/s390x/cpu-topology.h
+++ b/include/hw/s390x/cpu-topology.h
@@ -41,6 +41,9 @@ typedef union s390_topology_id {
};
} s390_topology_id;
#define TOPO_CPU_MASK 0x000000000000003fUL
+#define TOPO_SOCKET_MASK 0x0000ffffff000000UL
+#define TOPO_BOOK_MASK 0x0000ffff00000000UL
+#define TOPO_DRAWER_MASK 0x0000ff0000000000UL
typedef struct S390TopologyEntry {
s390_topology_id id;
diff --git a/include/hw/s390x/sclp.h b/include/hw/s390x/sclp.h
index d3ade40a5a..712fd68123 100644
--- a/include/hw/s390x/sclp.h
+++ b/include/hw/s390x/sclp.h
@@ -112,6 +112,7 @@ typedef struct CPUEntry {
} QEMU_PACKED CPUEntry;
#define SCLP_READ_SCP_INFO_FIXED_CPU_OFFSET 128
+#define SCLP_READ_SCP_INFO_MNEST 2
typedef struct ReadInfo {
SCCBHeader h;
uint16_t rnmax;
diff --git a/target/s390x/cpu.h b/target/s390x/cpu.h
index 39ea63a416..78988048dd 100644
--- a/target/s390x/cpu.h
+++ b/target/s390x/cpu.h
@@ -561,6 +561,25 @@ typedef struct SysIB_322 {
} SysIB_322;
QEMU_BUILD_BUG_ON(sizeof(SysIB_322) != 4096);
+#define S390_TOPOLOGY_MAG 6
+#define S390_TOPOLOGY_MAG6 0
+#define S390_TOPOLOGY_MAG5 1
+#define S390_TOPOLOGY_MAG4 2
+#define S390_TOPOLOGY_MAG3 3
+#define S390_TOPOLOGY_MAG2 4
+#define S390_TOPOLOGY_MAG1 5
+/* Configuration topology */
+typedef struct SysIB_151x {
+ uint8_t reserved0[2];
+ uint16_t length;
+ uint8_t mag[S390_TOPOLOGY_MAG];
+ uint8_t reserved1;
+ uint8_t mnest;
+ uint32_t reserved2;
+ char tle[];
+} QEMU_PACKED QEMU_ALIGNED(8) SysIB_151x;
+QEMU_BUILD_BUG_ON(sizeof(SysIB_151x) != 16);
+
typedef union SysIB {
SysIB_111 sysib_111;
SysIB_121 sysib_121;
@@ -568,9 +587,68 @@ typedef union SysIB {
SysIB_221 sysib_221;
SysIB_222 sysib_222;
SysIB_322 sysib_322;
+ SysIB_151x sysib_151x;
} SysIB;
QEMU_BUILD_BUG_ON(sizeof(SysIB) != 4096);
+/*
+ * CPU Topology List provided by STSI with fc=15 provides a list
+ * of two different Topology List Entries (TLE) types to specify
+ * the topology hierarchy.
+ *
+ * - Container Topology List Entry
+ * Defines a container to contain other Topology List Entries
+ * of any type, nested containers or CPU.
+ * - CPU Topology List Entry
+ * Specifies the CPUs position, type, entitlement and polarization
+ * of the CPUs contained in the last Container TLE.
+ *
+ * There can be theoretically up to five levels of containers, QEMU
+ * uses only one level, the socket level.
+ *
+ * A container of with a nesting level (NL) greater than 1 can only
+ * contain another container of nesting level NL-1.
+ *
+ * A container of nesting level 1 (socket), contains as many CPU TLE
+ * as needed to describe the position and qualities of all CPUs inside
+ * the container.
+ * The qualities of a CPU are polarization, entitlement and type.
+ *
+ * The CPU TLE defines the position of the CPUs of identical qualities
+ * using a 64bits mask which first bit has its offset defined by
+ * the CPU address orgin field of the CPU TLE like in:
+ * CPU address = origin * 64 + bit position within the mask
+ *
+ */
+/* Container type Topology List Entry */
+/* Container type Topology List Entry */
+typedef struct SysIBTl_container {
+ uint8_t nl;
+ uint8_t reserved[6];
+ uint8_t id;
+} QEMU_PACKED QEMU_ALIGNED(8) SysIBTl_container;
+QEMU_BUILD_BUG_ON(sizeof(SysIBTl_container) != 8);
+
+/* CPU type Topology List Entry */
+typedef struct SysIBTl_cpu {
+ uint8_t nl;
+ uint8_t reserved0[3];
+ uint8_t reserved1:5;
+ uint8_t dedicated:1;
+ uint8_t polarity:2;
+ uint8_t type;
+ uint16_t origin;
+ uint64_t mask;
+} QEMU_PACKED QEMU_ALIGNED(8) SysIBTl_cpu;
+QEMU_BUILD_BUG_ON(sizeof(SysIBTl_cpu) != 16);
+
+/* Max size of a SYSIB structure is when all CPU are alone in a container */
+#define S390_TOPOLOGY_SYSIB_SIZE (sizeof(SysIB_151x) + \
+ S390_MAX_CPUS * (sizeof(SysIBTl_container) + \
+ sizeof(SysIBTl_cpu)))
+
+void insert_stsi_15_1_x(S390CPU *cpu, int sel2, __u64 addr, uint8_t ar);
+
/* MMU defines */
#define ASCE_ORIGIN (~0xfffULL) /* segment table origin */
#define ASCE_SUBSPACE 0x200 /* subspace group control */
diff --git a/target/s390x/kvm/cpu_topology.c b/target/s390x/kvm/cpu_topology.c
new file mode 100644
index 0000000000..3831a3264c
--- /dev/null
+++ b/target/s390x/kvm/cpu_topology.c
@@ -0,0 +1,136 @@
+/*
+ * QEMU S390x CPU Topology
+ *
+ * Copyright IBM Corp. 2022
+ * Author(s): Pierre Morel <pmorel@linux.ibm.com>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or (at
+ * your option) any later version. See the COPYING file in the top-level
+ * directory.
+ */
+#include "qemu/osdep.h"
+#include "cpu.h"
+#include "hw/s390x/pv.h"
+#include "hw/sysbus.h"
+#include "hw/s390x/sclp.h"
+#include "hw/s390x/cpu-topology.h"
+
+static char *fill_container(char *p, int level, int id)
+{
+ SysIBTl_container *tle = (SysIBTl_container *)p;
+
+ tle->nl = level;
+ tle->id = id;
+ return p + sizeof(*tle);
+}
+
+static char *fill_tle_cpu(char *p, S390TopologyEntry *entry)
+{
+ SysIBTl_cpu *tle = (SysIBTl_cpu *)p;
+ s390_topology_id topology_id = entry->id;
+
+ tle->nl = 0;
+ tle->dedicated = topology_id.d;
+ tle->polarity = topology_id.p;
+ tle->type = topology_id.type;
+ tle->origin = topology_id.origin;
+ tle->mask = cpu_to_be64(entry->mask);
+ return p + sizeof(*tle);
+}
+
+static char *s390_top_set_level(char *p, int level)
+{
+ S390TopologyEntry *entry;
+ uint64_t last_socket = -1UL;
+ uint64_t last_book = -1UL;
+ uint64_t last_drawer = -1UL;
+ int drawer_cnt = 0;
+ int book_cnt = 0;
+ int socket_cnt = 0;
+
+ QTAILQ_FOREACH(entry, &s390_topology.list, next) {
+
+ if (level > 3 && (last_drawer != entry->id.drawer)) {
+ book_cnt = 0;
+ socket_cnt = 0;
+ p = fill_container(p, 3, drawer_cnt++);
+ last_drawer = entry->id.id & TOPO_DRAWER_MASK;
+ p = fill_container(p, 2, book_cnt++);
+ last_book = entry->id.id & TOPO_BOOK_MASK;
+ p = fill_container(p, 1, socket_cnt++);
+ last_socket = entry->id.id & TOPO_SOCKET_MASK;
+ p = fill_tle_cpu(p, entry);
+ } else if (level > 2 && (last_book !=
+ (entry->id.id & TOPO_BOOK_MASK))) {
+ socket_cnt = 0;
+ p = fill_container(p, 2, book_cnt++);
+ last_book = entry->id.id & TOPO_BOOK_MASK;
+ p = fill_container(p, 1, socket_cnt++);
+ last_socket = entry->id.id & TOPO_SOCKET_MASK;
+ p = fill_tle_cpu(p, entry);
+ } else if (last_socket != (entry->id.id & TOPO_SOCKET_MASK)) {
+ p = fill_container(p, 1, socket_cnt++);
+ last_socket = entry->id.id & TOPO_SOCKET_MASK;
+ p = fill_tle_cpu(p, entry);
+ } else {
+ p = fill_tle_cpu(p, entry);
+ }
+ }
+
+ return p;
+}
+
+static int setup_stsi(S390CPU *cpu, SysIB_151x *sysib, int level)
+{
+ char *p = sysib->tle;
+
+ sysib->mnest = level;
+ switch (level) {
+ case 4:
+ sysib->mag[S390_TOPOLOGY_MAG4] = current_machine->smp.drawers;
+ sysib->mag[S390_TOPOLOGY_MAG3] = current_machine->smp.books;
+ sysib->mag[S390_TOPOLOGY_MAG2] = current_machine->smp.sockets;
+ sysib->mag[S390_TOPOLOGY_MAG1] = current_machine->smp.cores;
+ break;
+ case 3:
+ sysib->mag[S390_TOPOLOGY_MAG3] = current_machine->smp.drawers *
+ current_machine->smp.books;
+ sysib->mag[S390_TOPOLOGY_MAG2] = current_machine->smp.sockets;
+ sysib->mag[S390_TOPOLOGY_MAG1] = current_machine->smp.cores;
+ break;
+ case 2:
+ sysib->mag[S390_TOPOLOGY_MAG2] = current_machine->smp.drawers *
+ current_machine->smp.books *
+ current_machine->smp.sockets;
+ sysib->mag[S390_TOPOLOGY_MAG1] = current_machine->smp.cores;
+ break;
+ }
+ p = s390_top_set_level(p, level);
+
+ return p - (char *)sysib;
+}
+
+void insert_stsi_15_1_x(S390CPU *cpu, int sel2, __u64 addr, uint8_t ar)
+{
+ union {
+ char place_holder[S390_TOPOLOGY_SYSIB_SIZE];
+ SysIB_151x sysib;
+ } buffer QEMU_ALIGNED(8) = {};
+ int len;
+
+ if (!s390_has_topology() || sel2 < 2 || sel2 > SCLP_READ_SCP_INFO_MNEST) {
+ setcc(cpu, 3);
+ return;
+ }
+
+ len = setup_stsi(cpu, &buffer.sysib, sel2);
+
+ if (len > 4096) {
+ setcc(cpu, 3);
+ return;
+ }
+
+ buffer.sysib.length = cpu_to_be16(len);
+ s390_cpu_virt_mem_write(cpu, addr, ar, &buffer.sysib, len);
+ setcc(cpu, 0);
+}
diff --git a/target/s390x/kvm/kvm.c b/target/s390x/kvm/kvm.c
index 3ac7ec9acf..5ea358cbb0 100644
--- a/target/s390x/kvm/kvm.c
+++ b/target/s390x/kvm/kvm.c
@@ -1919,9 +1919,12 @@ static int handle_stsi(S390CPU *cpu)
if (run->s390_stsi.sel1 != 2 || run->s390_stsi.sel2 != 2) {
return 0;
}
- /* Only sysib 3.2.2 needs post-handling for now. */
insert_stsi_3_2_2(cpu, run->s390_stsi.addr, run->s390_stsi.ar);
return 0;
+ case 15:
+ insert_stsi_15_1_x(cpu, run->s390_stsi.sel2, run->s390_stsi.addr,
+ run->s390_stsi.ar);
+ return 0;
default:
return 0;
}
diff --git a/target/s390x/kvm/meson.build b/target/s390x/kvm/meson.build
index aef52b6686..5daa5c6033 100644
--- a/target/s390x/kvm/meson.build
+++ b/target/s390x/kvm/meson.build
@@ -1,6 +1,7 @@
s390x_ss.add(when: 'CONFIG_KVM', if_true: files(
- 'kvm.c'
+ 'kvm.c',
+ 'cpu_topology.c'
), if_false: files(
'stubs.c'
))
--
2.31.1
^ permalink raw reply related [flat|nested] 80+ messages in thread
* [PATCH v14 04/11] s390x/sclp: reporting the maximum nested topology entries
2023-01-05 14:53 [PATCH v14 00/11] s390x: CPU Topology Pierre Morel
` (2 preceding siblings ...)
2023-01-05 14:53 ` [PATCH v14 03/11] target/s390x/cpu topology: handle STSI(15) and build the SYSIB Pierre Morel
@ 2023-01-05 14:53 ` Pierre Morel
2023-01-11 8:57 ` Thomas Huth
2023-01-11 17:52 ` Nina Schoetterl-Glausch
2023-01-05 14:53 ` [PATCH v14 05/11] s390x/cpu topology: resetting the Topology-Change-Report Pierre Morel
` (6 subsequent siblings)
10 siblings, 2 replies; 80+ messages in thread
From: Pierre Morel @ 2023-01-05 14:53 UTC (permalink / raw)
To: qemu-s390x
Cc: qemu-devel, borntraeger, pasic, richard.henderson, david, thuth,
cohuck, mst, pbonzini, kvm, ehabkost, marcel.apfelbaum, eblake,
armbru, seiden, nrb, scgl, frankja, berrange, clg
The maximum nested topology entries is used by the guest to know
how many nested topology are available on the machine.
Currently, SCLP READ SCP INFO reports MNEST = 0, which is the
equivalent of reporting the default value of 2.
Let's use the default SCLP value of 2 and increase this value in the
future patches implementing higher levels.
Signed-off-by: Pierre Morel <pmorel@linux.ibm.com>
---
include/hw/s390x/sclp.h | 5 +++--
hw/s390x/sclp.c | 4 ++++
2 files changed, 7 insertions(+), 2 deletions(-)
diff --git a/include/hw/s390x/sclp.h b/include/hw/s390x/sclp.h
index 712fd68123..4ce852473c 100644
--- a/include/hw/s390x/sclp.h
+++ b/include/hw/s390x/sclp.h
@@ -112,12 +112,13 @@ typedef struct CPUEntry {
} QEMU_PACKED CPUEntry;
#define SCLP_READ_SCP_INFO_FIXED_CPU_OFFSET 128
-#define SCLP_READ_SCP_INFO_MNEST 2
+#define SCLP_READ_SCP_INFO_MNEST 4
typedef struct ReadInfo {
SCCBHeader h;
uint16_t rnmax;
uint8_t rnsize;
- uint8_t _reserved1[16 - 11]; /* 11-15 */
+ uint8_t _reserved1[15 - 11]; /* 11-14 */
+ uint8_t stsi_parm; /* 15-16 */
uint16_t entries_cpu; /* 16-17 */
uint16_t offset_cpu; /* 18-19 */
uint8_t _reserved2[24 - 20]; /* 20-23 */
diff --git a/hw/s390x/sclp.c b/hw/s390x/sclp.c
index eff74479f4..07e3cb4cac 100644
--- a/hw/s390x/sclp.c
+++ b/hw/s390x/sclp.c
@@ -20,6 +20,7 @@
#include "hw/s390x/event-facility.h"
#include "hw/s390x/s390-pci-bus.h"
#include "hw/s390x/ipl.h"
+#include "hw/s390x/cpu-topology.h"
static inline SCLPDevice *get_sclp_device(void)
{
@@ -125,6 +126,9 @@ static void read_SCP_info(SCLPDevice *sclp, SCCB *sccb)
/* CPU information */
prepare_cpu_entries(machine, entries_start, &cpu_count);
+ if (s390_has_topology()) {
+ read_info->stsi_parm = SCLP_READ_SCP_INFO_MNEST;
+ }
read_info->entries_cpu = cpu_to_be16(cpu_count);
read_info->offset_cpu = cpu_to_be16(offset_cpu);
read_info->highest_cpu = cpu_to_be16(machine->smp.max_cpus - 1);
--
2.31.1
^ permalink raw reply related [flat|nested] 80+ messages in thread
* [PATCH v14 05/11] s390x/cpu topology: resetting the Topology-Change-Report
2023-01-05 14:53 [PATCH v14 00/11] s390x: CPU Topology Pierre Morel
` (3 preceding siblings ...)
2023-01-05 14:53 ` [PATCH v14 04/11] s390x/sclp: reporting the maximum nested topology entries Pierre Morel
@ 2023-01-05 14:53 ` Pierre Morel
2023-01-11 9:00 ` Thomas Huth
2023-01-05 14:53 ` [PATCH v14 06/11] s390x/cpu topology: interception of PTF instruction Pierre Morel
` (5 subsequent siblings)
10 siblings, 1 reply; 80+ messages in thread
From: Pierre Morel @ 2023-01-05 14:53 UTC (permalink / raw)
To: qemu-s390x
Cc: qemu-devel, borntraeger, pasic, richard.henderson, david, thuth,
cohuck, mst, pbonzini, kvm, ehabkost, marcel.apfelbaum, eblake,
armbru, seiden, nrb, scgl, frankja, berrange, clg
During a subsystem reset the Topology-Change-Report is cleared
by the machine.
Let's ask KVM to clear the Modified Topology Change Report (MTCR)
bit of the SCA in the case of a subsystem reset.
Signed-off-by: Pierre Morel <pmorel@linux.ibm.com>
---
target/s390x/cpu.h | 1 +
target/s390x/kvm/kvm_s390x.h | 1 +
hw/s390x/s390-virtio-ccw.c | 1 +
target/s390x/cpu-sysemu.c | 13 +++++++++++++
target/s390x/kvm/kvm.c | 17 +++++++++++++++++
5 files changed, 33 insertions(+)
diff --git a/target/s390x/cpu.h b/target/s390x/cpu.h
index 78988048dd..01ade07009 100644
--- a/target/s390x/cpu.h
+++ b/target/s390x/cpu.h
@@ -863,6 +863,7 @@ void s390_enable_css_support(S390CPU *cpu);
void s390_do_cpu_set_diag318(CPUState *cs, run_on_cpu_data arg);
int s390_assign_subch_ioeventfd(EventNotifier *notifier, uint32_t sch_id,
int vq, bool assign);
+void s390_cpu_topology_reset(void);
#ifndef CONFIG_USER_ONLY
unsigned int s390_cpu_set_state(uint8_t cpu_state, S390CPU *cpu);
#else
diff --git a/target/s390x/kvm/kvm_s390x.h b/target/s390x/kvm/kvm_s390x.h
index f9785564d0..649dae5948 100644
--- a/target/s390x/kvm/kvm_s390x.h
+++ b/target/s390x/kvm/kvm_s390x.h
@@ -47,5 +47,6 @@ void kvm_s390_crypto_reset(void);
void kvm_s390_restart_interrupt(S390CPU *cpu);
void kvm_s390_stop_interrupt(S390CPU *cpu);
void kvm_s390_set_diag318(CPUState *cs, uint64_t diag318_info);
+int kvm_s390_topology_set_mtcr(uint64_t attr);
#endif /* KVM_S390X_H */
diff --git a/hw/s390x/s390-virtio-ccw.c b/hw/s390x/s390-virtio-ccw.c
index c98b93a15f..14798ca305 100644
--- a/hw/s390x/s390-virtio-ccw.c
+++ b/hw/s390x/s390-virtio-ccw.c
@@ -122,6 +122,7 @@ static void subsystem_reset(void)
device_cold_reset(dev);
}
}
+ s390_cpu_topology_reset();
}
static int virtio_ccw_hcall_notify(const uint64_t *args)
diff --git a/target/s390x/cpu-sysemu.c b/target/s390x/cpu-sysemu.c
index 948e4bd3e0..e27864c5f5 100644
--- a/target/s390x/cpu-sysemu.c
+++ b/target/s390x/cpu-sysemu.c
@@ -306,3 +306,16 @@ void s390_do_cpu_set_diag318(CPUState *cs, run_on_cpu_data arg)
kvm_s390_set_diag318(cs, arg.host_ulong);
}
}
+
+void s390_cpu_topology_reset(void)
+{
+ int ret;
+
+ if (kvm_enabled()) {
+ ret = kvm_s390_topology_set_mtcr(0);
+ if (ret) {
+ error_report("Failed to set Modified Topology Change Report: %s",
+ strerror(-ret));
+ }
+ }
+}
diff --git a/target/s390x/kvm/kvm.c b/target/s390x/kvm/kvm.c
index 5ea358cbb0..bc953151ce 100644
--- a/target/s390x/kvm/kvm.c
+++ b/target/s390x/kvm/kvm.c
@@ -2592,6 +2592,23 @@ int kvm_s390_get_zpci_op(void)
return cap_zpci_op;
}
+int kvm_s390_topology_set_mtcr(uint64_t attr)
+{
+ struct kvm_device_attr attribute = {
+ .group = KVM_S390_VM_CPU_TOPOLOGY,
+ .attr = attr,
+ };
+
+ if (!s390_has_feat(S390_FEAT_CONFIGURATION_TOPOLOGY)) {
+ return 0;
+ }
+ if (!kvm_vm_check_attr(kvm_state, KVM_S390_VM_CPU_TOPOLOGY, attr)) {
+ return -ENOTSUP;
+ }
+
+ return kvm_vm_ioctl(kvm_state, KVM_SET_DEVICE_ATTR, &attribute);
+}
+
void kvm_arch_accel_class_init(ObjectClass *oc)
{
}
--
2.31.1
^ permalink raw reply related [flat|nested] 80+ messages in thread
* [PATCH v14 06/11] s390x/cpu topology: interception of PTF instruction
2023-01-05 14:53 [PATCH v14 00/11] s390x: CPU Topology Pierre Morel
` (4 preceding siblings ...)
2023-01-05 14:53 ` [PATCH v14 05/11] s390x/cpu topology: resetting the Topology-Change-Report Pierre Morel
@ 2023-01-05 14:53 ` Pierre Morel
2023-01-16 18:24 ` Nina Schoetterl-Glausch
2023-01-05 14:53 ` [PATCH v14 07/11] target/s390x/cpu topology: activating CPU topology Pierre Morel
` (4 subsequent siblings)
10 siblings, 1 reply; 80+ messages in thread
From: Pierre Morel @ 2023-01-05 14:53 UTC (permalink / raw)
To: qemu-s390x
Cc: qemu-devel, borntraeger, pasic, richard.henderson, david, thuth,
cohuck, mst, pbonzini, kvm, ehabkost, marcel.apfelbaum, eblake,
armbru, seiden, nrb, scgl, frankja, berrange, clg
When the host supports the CPU topology facility, the PTF
instruction with function code 2 is interpreted by the SIE,
provided that the userland hypervizor activates the interpretation
by using the KVM_CAP_S390_CPU_TOPOLOGY KVM extension.
The PTF instructions with function code 0 and 1 are intercepted
and must be emulated by the userland hypervizor.
During RESET all CPU of the configuration are placed in
horizontal polarity.
Signed-off-by: Pierre Morel <pmorel@linux.ibm.com>
---
include/hw/s390x/cpu-topology.h | 3 +
include/hw/s390x/s390-virtio-ccw.h | 6 ++
target/s390x/cpu.h | 1 +
hw/s390x/cpu-topology.c | 92 ++++++++++++++++++++++++++++++
target/s390x/cpu-sysemu.c | 16 ++++++
target/s390x/kvm/kvm.c | 11 ++++
6 files changed, 129 insertions(+)
diff --git a/include/hw/s390x/cpu-topology.h b/include/hw/s390x/cpu-topology.h
index 9571aa70e5..33e23d78b9 100644
--- a/include/hw/s390x/cpu-topology.h
+++ b/include/hw/s390x/cpu-topology.h
@@ -55,11 +55,13 @@ typedef struct S390Topology {
QTAILQ_HEAD(, S390TopologyEntry) list;
uint8_t *sockets;
CpuTopology *smp;
+ int polarity;
} S390Topology;
#ifdef CONFIG_KVM
bool s390_has_topology(void);
void s390_topology_set_cpu(MachineState *ms, S390CPU *cpu, Error **errp);
+void s390_topology_set_polarity(int polarity);
#else
static inline bool s390_has_topology(void)
{
@@ -68,6 +70,7 @@ static inline bool s390_has_topology(void)
static inline void s390_topology_set_cpu(MachineState *ms,
S390CPU *cpu,
Error **errp) {}
+static inline void s390_topology_set_polarity(int polarity) {}
#endif
extern S390Topology s390_topology;
diff --git a/include/hw/s390x/s390-virtio-ccw.h b/include/hw/s390x/s390-virtio-ccw.h
index 9bba21a916..c1d46e78af 100644
--- a/include/hw/s390x/s390-virtio-ccw.h
+++ b/include/hw/s390x/s390-virtio-ccw.h
@@ -30,6 +30,12 @@ struct S390CcwMachineState {
uint8_t loadparm[8];
};
+#define S390_PTF_REASON_NONE (0x00 << 8)
+#define S390_PTF_REASON_DONE (0x01 << 8)
+#define S390_PTF_REASON_BUSY (0x02 << 8)
+#define S390_TOPO_FC_MASK 0xffUL
+void s390_handle_ptf(S390CPU *cpu, uint8_t r1, uintptr_t ra);
+
struct S390CcwMachineClass {
/*< private >*/
MachineClass parent_class;
diff --git a/target/s390x/cpu.h b/target/s390x/cpu.h
index 01ade07009..5da4041576 100644
--- a/target/s390x/cpu.h
+++ b/target/s390x/cpu.h
@@ -864,6 +864,7 @@ void s390_do_cpu_set_diag318(CPUState *cs, run_on_cpu_data arg);
int s390_assign_subch_ioeventfd(EventNotifier *notifier, uint32_t sch_id,
int vq, bool assign);
void s390_cpu_topology_reset(void);
+void s390_cpu_topology_set(void);
#ifndef CONFIG_USER_ONLY
unsigned int s390_cpu_set_state(uint8_t cpu_state, S390CPU *cpu);
#else
diff --git a/hw/s390x/cpu-topology.c b/hw/s390x/cpu-topology.c
index 438055c612..e6b4692581 100644
--- a/hw/s390x/cpu-topology.c
+++ b/hw/s390x/cpu-topology.c
@@ -97,6 +97,98 @@ static s390_topology_id s390_topology_from_cpu(S390CPU *cpu)
}
/**
+ * s390_topology_set_polarity
+ * @polarity: horizontal or vertical
+ *
+ * Changes the polarity of all the CPU in the configuration.
+ *
+ * If the dedicated CPU modifier attribute is set a vertical
+ * polarization is always high (Architecture).
+ * Otherwise we decide to set it as medium.
+ *
+ * Once done, advertise a topology change.
+ */
+void s390_topology_set_polarity(int polarity)
+{
+ S390TopologyEntry *entry;
+
+ QTAILQ_FOREACH(entry, &s390_topology.list, next) {
+ if (polarity == S390_TOPOLOGY_POLARITY_HORIZONTAL) {
+ entry->id.p = polarity;
+ } else {
+ if (entry->id.d) {
+ entry->id.p = S390_TOPOLOGY_POLARITY_VERTICAL_HIGH;
+ } else {
+ entry->id.p = S390_TOPOLOGY_POLARITY_VERTICAL_MEDIUM;
+ }
+ }
+ }
+ s390_cpu_topology_set();
+}
+
+/*
+ * s390_handle_ptf:
+ *
+ * @register 1: contains the function code
+ *
+ * Function codes 0 and 1 handle the CPU polarization.
+ * We assume an horizontal topology, the only one supported currently
+ * by Linux, consequently we answer to function code 0, requesting
+ * horizontal polarization that it is already the current polarization
+ * and reject vertical polarization request without further explanation.
+ *
+ * Function code 2 is handling topology changes and is interpreted
+ * by the SIE.
+ */
+void s390_handle_ptf(S390CPU *cpu, uint8_t r1, uintptr_t ra)
+{
+ CPUS390XState *env = &cpu->env;
+ uint64_t reg = env->regs[r1];
+ uint8_t fc = reg & S390_TOPO_FC_MASK;
+
+ if (!s390_has_feat(S390_FEAT_CONFIGURATION_TOPOLOGY)) {
+ s390_program_interrupt(env, PGM_OPERATION, ra);
+ return;
+ }
+
+ if (env->psw.mask & PSW_MASK_PSTATE) {
+ s390_program_interrupt(env, PGM_PRIVILEGED, ra);
+ return;
+ }
+
+ if (reg & ~S390_TOPO_FC_MASK) {
+ s390_program_interrupt(env, PGM_SPECIFICATION, ra);
+ return;
+ }
+
+ switch (fc) {
+ case 0: /* Horizontal polarization is already set */
+ if (s390_topology.polarity == S390_TOPOLOGY_POLARITY_HORIZONTAL) {
+ env->regs[r1] |= S390_PTF_REASON_DONE;
+ setcc(cpu, 2);
+ } else {
+ s390_topology_set_polarity(S390_TOPOLOGY_POLARITY_HORIZONTAL);
+ s390_topology.polarity = S390_TOPOLOGY_POLARITY_HORIZONTAL;
+ setcc(cpu, 0);
+ }
+ break;
+ case 1: /* Vertical polarization is not supported */
+ if (s390_topology.polarity != S390_TOPOLOGY_POLARITY_HORIZONTAL) {
+ env->regs[r1] |= S390_PTF_REASON_DONE;
+ setcc(cpu, 2);
+ } else {
+ s390_topology_set_polarity(S390_TOPOLOGY_POLARITY_VERTICAL_LOW);
+ s390_topology.polarity = S390_TOPOLOGY_POLARITY_VERTICAL_LOW;
+ setcc(cpu, 0);
+ }
+ break;
+ default:
+ /* Note that fc == 2 is interpreted by the SIE */
+ s390_program_interrupt(env, PGM_SPECIFICATION, ra);
+ }
+}
+
+ /**
* s390_topology_set_entry:
* @entry: Topology entry to setup
* @id: topology id to use for the setup
diff --git a/target/s390x/cpu-sysemu.c b/target/s390x/cpu-sysemu.c
index e27864c5f5..8643320947 100644
--- a/target/s390x/cpu-sysemu.c
+++ b/target/s390x/cpu-sysemu.c
@@ -37,6 +37,7 @@
#include "sysemu/sysemu.h"
#include "sysemu/tcg.h"
#include "hw/core/sysemu-cpu-ops.h"
+#include "hw/s390x/cpu-topology.h"
/* S390CPUClass::load_normal() */
static void s390_cpu_load_normal(CPUState *s)
@@ -311,6 +312,8 @@ void s390_cpu_topology_reset(void)
{
int ret;
+ s390_topology_set_polarity(S390_TOPOLOGY_POLARITY_HORIZONTAL);
+
if (kvm_enabled()) {
ret = kvm_s390_topology_set_mtcr(0);
if (ret) {
@@ -319,3 +322,16 @@ void s390_cpu_topology_reset(void)
}
}
}
+
+void s390_cpu_topology_set(void)
+{
+ int ret;
+
+ if (kvm_enabled()) {
+ ret = kvm_s390_topology_set_mtcr(1);
+ if (ret) {
+ error_report("Failed to set Modified Topology Change Report: %s",
+ strerror(-ret));
+ }
+ }
+}
diff --git a/target/s390x/kvm/kvm.c b/target/s390x/kvm/kvm.c
index bc953151ce..fb63be41b7 100644
--- a/target/s390x/kvm/kvm.c
+++ b/target/s390x/kvm/kvm.c
@@ -96,6 +96,7 @@
#define PRIV_B9_EQBS 0x9c
#define PRIV_B9_CLP 0xa0
+#define PRIV_B9_PTF 0xa2
#define PRIV_B9_PCISTG 0xd0
#define PRIV_B9_PCILG 0xd2
#define PRIV_B9_RPCIT 0xd3
@@ -1464,6 +1465,13 @@ static int kvm_mpcifc_service_call(S390CPU *cpu, struct kvm_run *run)
}
}
+static void kvm_handle_ptf(S390CPU *cpu, struct kvm_run *run)
+{
+ uint8_t r1 = (run->s390_sieic.ipb >> 20) & 0x0f;
+
+ s390_handle_ptf(cpu, r1, RA_IGNORED);
+}
+
static int handle_b9(S390CPU *cpu, struct kvm_run *run, uint8_t ipa1)
{
int r = 0;
@@ -1481,6 +1489,9 @@ static int handle_b9(S390CPU *cpu, struct kvm_run *run, uint8_t ipa1)
case PRIV_B9_RPCIT:
r = kvm_rpcit_service_call(cpu, run);
break;
+ case PRIV_B9_PTF:
+ kvm_handle_ptf(cpu, run);
+ break;
case PRIV_B9_EQBS:
/* just inject exception */
r = -1;
--
2.31.1
^ permalink raw reply related [flat|nested] 80+ messages in thread
* [PATCH v14 07/11] target/s390x/cpu topology: activating CPU topology
2023-01-05 14:53 [PATCH v14 00/11] s390x: CPU Topology Pierre Morel
` (5 preceding siblings ...)
2023-01-05 14:53 ` [PATCH v14 06/11] s390x/cpu topology: interception of PTF instruction Pierre Morel
@ 2023-01-05 14:53 ` Pierre Morel
2023-01-11 10:04 ` Thomas Huth
2023-01-05 14:53 ` Pierre Morel
` (3 subsequent siblings)
10 siblings, 1 reply; 80+ messages in thread
From: Pierre Morel @ 2023-01-05 14:53 UTC (permalink / raw)
To: qemu-s390x
Cc: qemu-devel, borntraeger, pasic, richard.henderson, david, thuth,
cohuck, mst, pbonzini, kvm, ehabkost, marcel.apfelbaum, eblake,
armbru, seiden, nrb, scgl, frankja, berrange, clg
The KVM capability, KVM_CAP_S390_CPU_TOPOLOGY is used to
activate the S390_FEAT_CONFIGURATION_TOPOLOGY feature and
the topology facility for the guest in the case the topology
is available in QEMU and in KVM.
The feature is disabled by default and fenced for SE
(secure execution).
Signed-off-by: Pierre Morel <pmorel@linux.ibm.com>
---
hw/s390x/cpu-topology.c | 2 +-
target/s390x/cpu_models.c | 1 +
target/s390x/kvm/kvm.c | 13 +++++++++++++
3 files changed, 15 insertions(+), 1 deletion(-)
diff --git a/hw/s390x/cpu-topology.c b/hw/s390x/cpu-topology.c
index e6b4692581..b69955a1cd 100644
--- a/hw/s390x/cpu-topology.c
+++ b/hw/s390x/cpu-topology.c
@@ -52,7 +52,7 @@ static int s390_socket_nb(s390_topology_id id)
*/
bool s390_has_topology(void)
{
- return false;
+ return s390_has_feat(S390_FEAT_CONFIGURATION_TOPOLOGY);
}
/**
diff --git a/target/s390x/cpu_models.c b/target/s390x/cpu_models.c
index c3a4f80633..3f05e05fd3 100644
--- a/target/s390x/cpu_models.c
+++ b/target/s390x/cpu_models.c
@@ -253,6 +253,7 @@ bool s390_has_feat(S390Feat feat)
case S390_FEAT_SIE_CMMA:
case S390_FEAT_SIE_PFMFI:
case S390_FEAT_SIE_IBS:
+ case S390_FEAT_CONFIGURATION_TOPOLOGY:
return false;
break;
default:
diff --git a/target/s390x/kvm/kvm.c b/target/s390x/kvm/kvm.c
index fb63be41b7..4e2a2ff516 100644
--- a/target/s390x/kvm/kvm.c
+++ b/target/s390x/kvm/kvm.c
@@ -2470,6 +2470,19 @@ void kvm_s390_get_host_cpu_model(S390CPUModel *model, Error **errp)
set_bit(S390_FEAT_UNPACK, model->features);
}
+ /*
+ * If we have support for CPU Topology prevent overrule
+ * S390_FEAT_CONFIGURATION_TOPOLOGY with S390_FEAT_DISABLE_CPU_TOPOLOGY
+ * implemented in KVM, activate the CPU TOPOLOGY feature.
+ */
+ if (kvm_check_extension(kvm_state, KVM_CAP_S390_CPU_TOPOLOGY)) {
+ if (kvm_vm_enable_cap(kvm_state, KVM_CAP_S390_CPU_TOPOLOGY, 0) < 0) {
+ error_setg(errp, "KVM: Error enabling KVM_CAP_S390_CPU_TOPOLOGY");
+ return;
+ }
+ set_bit(S390_FEAT_CONFIGURATION_TOPOLOGY, model->features);
+ }
+
/* We emulate a zPCI bus and AEN, therefore we don't need HW support */
set_bit(S390_FEAT_ZPCI, model->features);
set_bit(S390_FEAT_ADAPTER_EVENT_NOTIFICATION, model->features);
--
2.31.1
^ permalink raw reply related [flat|nested] 80+ messages in thread
* [PATCH v14 08/11] qapi/s390/cpu topology: change-topology monitor command
2023-01-05 14:53 [PATCH v14 00/11] s390x: CPU Topology Pierre Morel
@ 2023-01-05 14:53 ` Pierre Morel
2023-01-05 14:53 ` [PATCH v14 02/11] s390x/cpu topology: add topology entries on CPU hotplug Pierre Morel
` (9 subsequent siblings)
10 siblings, 0 replies; 80+ messages in thread
From: Pierre Morel @ 2023-01-05 14:53 UTC (permalink / raw)
To: qemu-s390x
Cc: qemu-devel, borntraeger, pasic, richard.henderson, david, thuth,
cohuck, mst, pbonzini, kvm, ehabkost, marcel.apfelbaum, eblake,
armbru, seiden, nrb, scgl, frankja, berrange, clg
The modification of the CPU attributes are done through a monitor
commands.
It allows to move the core inside the topology tree to optimise
the cache usage in the case the host's hypervizor previously
moved the CPU.
The same command allows to modifiy the CPU attributes modifiers
like polarization entitlement and the dedicated attribute to notify
the guest if the host admin modified scheduling or dedication of a vCPU.
With this knowledge the guest has the possibility to optimize the
usage of the vCPUs.
Signed-off-by: Pierre Morel <pmorel@linux.ibm.com>
---
qapi/machine-target.json | 29 ++++++++
include/monitor/hmp.h | 1 +
hw/s390x/cpu-topology.c | 141 +++++++++++++++++++++++++++++++++++++++
hmp-commands.hx | 16 +++++
4 files changed, 187 insertions(+)
diff --git a/qapi/machine-target.json b/qapi/machine-target.json
index 2e267fa458..75b0aa254d 100644
--- a/qapi/machine-target.json
+++ b/qapi/machine-target.json
@@ -342,3 +342,32 @@
'TARGET_S390X',
'TARGET_MIPS',
'TARGET_LOONGARCH64' ] } }
+
+##
+# @change-topology:
+#
+# @core: the vCPU ID to be moved
+# @socket: the destination socket where to move the vCPU
+# @book: the destination book where to move the vCPU
+# @drawer: the destination drawer where to move the vCPU
+# @polarity: optional polarity, default is last polarity set by the guest
+# @dedicated: optional, if the vCPU is dedicated to a real CPU
+#
+# Modifies the topology by moving the CPU inside the topology
+# tree or by changing a modifier attribute of a CPU.
+#
+# Returns: Nothing on success, the reason on failure.
+#
+# Since: <next qemu stable release, eg. 1.0>
+##
+{ 'command': 'change-topology',
+ 'data': {
+ 'core': 'int',
+ 'socket': 'int',
+ 'book': 'int',
+ 'drawer': 'int',
+ '*polarity': 'int',
+ '*dedicated': 'bool'
+ },
+ 'if': { 'all': [ 'TARGET_S390X', 'CONFIG_KVM' ] }
+}
diff --git a/include/monitor/hmp.h b/include/monitor/hmp.h
index 27f86399f7..15c36bf549 100644
--- a/include/monitor/hmp.h
+++ b/include/monitor/hmp.h
@@ -144,5 +144,6 @@ void hmp_human_readable_text_helper(Monitor *mon,
HumanReadableText *(*qmp_handler)(Error **));
void hmp_info_stats(Monitor *mon, const QDict *qdict);
void hmp_pcie_aer_inject_error(Monitor *mon, const QDict *qdict);
+void hmp_change_topology(Monitor *mon, const QDict *qdict);
#endif
diff --git a/hw/s390x/cpu-topology.c b/hw/s390x/cpu-topology.c
index b69955a1cd..0faffe657e 100644
--- a/hw/s390x/cpu-topology.c
+++ b/hw/s390x/cpu-topology.c
@@ -18,6 +18,10 @@
#include "target/s390x/cpu.h"
#include "hw/s390x/s390-virtio-ccw.h"
#include "hw/s390x/cpu-topology.h"
+#include "qapi/qapi-commands-machine-target.h"
+#include "qapi/qmp/qdict.h"
+#include "monitor/hmp.h"
+#include "monitor/monitor.h"
/*
* s390_topology is used to keep the topology information.
@@ -203,6 +207,21 @@ static void s390_topology_set_entry(S390TopologyEntry *entry,
s390_topology.sockets[s390_socket_nb(id)]++;
}
+/**
+ * s390_topology_clear_entry:
+ * @entry: Topology entry to setup
+ * @id: topology id to use for the setup
+ *
+ * Clear the core bit inside the topology mask and
+ * decrements the number of cores for the socket.
+ */
+static void s390_topology_clear_entry(S390TopologyEntry *entry,
+ s390_topology_id id)
+{
+ clear_bit(63 - id.core, &entry->mask);
+ s390_topology.sockets[s390_socket_nb(id)]--;
+}
+
/**
* s390_topology_new_entry:
* @id: s390_topology_id to add
@@ -383,3 +402,125 @@ void s390_topology_set_cpu(MachineState *ms, S390CPU *cpu, Error **errp)
s390_topology_insert(id);
}
+
+/*
+ * qmp and hmp implementations
+ */
+
+static S390TopologyEntry *s390_topology_core_to_entry(int core)
+{
+ S390TopologyEntry *entry;
+
+ QTAILQ_FOREACH(entry, &s390_topology.list, next) {
+ if (entry->mask & (1UL << (63 - core))) {
+ return entry;
+ }
+ }
+ return NULL;
+}
+
+static void s390_change_topology(Error **errp, int64_t core, int64_t socket,
+ int64_t book, int64_t drawer,
+ int64_t polarity, bool dedicated)
+{
+ S390TopologyEntry *entry;
+ s390_topology_id new_id;
+ s390_topology_id old_id;
+ Error *local_error = NULL;
+
+ /* Get the old entry */
+ entry = s390_topology_core_to_entry(core);
+ if (!entry) {
+ error_setg(errp, "No core %ld", core);
+ return;
+ }
+
+ /* Compute old topology id */
+ old_id = entry->id;
+ old_id.core = core;
+
+ /* Compute new topology id */
+ new_id = entry->id;
+ new_id.core = core;
+ new_id.socket = socket;
+ new_id.book = book;
+ new_id.drawer = drawer;
+ new_id.p = polarity;
+ new_id.d = dedicated;
+ new_id.type = S390_TOPOLOGY_CPU_IFL;
+
+ /* Same topology entry, nothing to do */
+ if (entry->id.id == new_id.id) {
+ return;
+ }
+
+ /* Check for space on the socket if ids are different */
+ if ((s390_socket_nb(old_id) != s390_socket_nb(new_id)) &&
+ (s390_topology.sockets[s390_socket_nb(new_id)] >=
+ s390_topology.smp->sockets)) {
+ error_setg(errp, "No more space on this socket");
+ return;
+ }
+
+ /* Verify the new topology */
+ s390_topology_check(&local_error, new_id);
+ if (local_error) {
+ error_propagate(errp, local_error);
+ return;
+ }
+
+ /* Clear the old topology */
+ s390_topology_clear_entry(entry, old_id);
+
+ /* Insert the new topology */
+ s390_topology_insert(new_id);
+
+ /* Remove unused entry */
+ if (!entry->mask) {
+ QTAILQ_REMOVE(&s390_topology.list, entry, next);
+ g_free(entry);
+ }
+
+ /* Advertise the topology change */
+ s390_cpu_topology_set();
+}
+
+void qmp_change_topology(int64_t core, int64_t socket,
+ int64_t book, int64_t drawer,
+ bool has_polarity, int64_t polarity,
+ bool has_dedicated, bool dedicated,
+ Error **errp)
+{
+ Error *local_err = NULL;
+
+ if (!s390_has_topology()) {
+ error_setg(&local_err, "This machine doesn't support topology");
+ return;
+ }
+ s390_change_topology(&local_err, core, socket, book, drawer,
+ polarity, dedicated);
+ if (local_err) {
+ error_propagate(errp, local_err);
+ }
+}
+
+void hmp_change_topology(Monitor *mon, const QDict *qdict)
+{
+ const int64_t core = qdict_get_int(qdict, "core");
+ const int64_t socket = qdict_get_int(qdict, "socket");
+ const int64_t book = qdict_get_int(qdict, "book");
+ const int64_t drawer = qdict_get_int(qdict, "drawer");
+ bool has_polarity = qdict_haskey(qdict, "polarity");
+ const int64_t polarity = qdict_get_try_int(qdict, "polarity", 0);
+ bool has_dedicated = qdict_haskey(qdict, "dedicated");
+ const bool dedicated = qdict_get_try_bool(qdict, "dedicated", false);
+ Error *local_err = NULL;
+
+ qmp_change_topology(core, socket, book, drawer,
+ has_polarity, polarity,
+ has_dedicated, dedicated,
+ &local_err);
+ if (hmp_handle_error(mon, local_err)) {
+ return;
+ }
+}
diff --git a/hmp-commands.hx b/hmp-commands.hx
index 673e39a697..a617cfed0d 100644
--- a/hmp-commands.hx
+++ b/hmp-commands.hx
@@ -1815,3 +1815,19 @@ SRST
Dump the FDT in dtb format to *filename*.
ERST
#endif
+
+#if defined(TARGET_S390X) && defined(CONFIG_KVM)
+ {
+ .name = "change-topology",
+ .args_type = "core:l,socket:l,book:l,drawer:l,polarity:l?,dedicated:b?",
+ .params = "core socket book drawer [polarity] [dedicated]",
+ .help = "Move CPU 'core' to 'socket/book/drawer' "
+ "optionaly modifies polarity and dedication",
+ .cmd = hmp_change_topology,
+ },
+
+SRST
+``change-topology`` *core* *socket* *book* *drawer* *polarity* *dedicated*
+ Moves the CPU *core* to *socket* *book* *drawer* with *polarity* *dedicated*.
+ERST
+#endif
--
2.31.1
^ permalink raw reply related [flat|nested] 80+ messages in thread
* [PATCH v14 08/11] qapi/s390/cpu topology: change-topology monitor command
@ 2023-01-05 14:53 ` Pierre Morel
0 siblings, 0 replies; 80+ messages in thread
From: Pierre Morel @ 2023-01-05 14:53 UTC (permalink / raw)
To: qemu-s390x
Cc: qemu-devel, borntraeger, pasic, richard.henderson, david, thuth,
cohuck, mst, pbonzini, kvm, ehabkost, marcel.apfelbaum, eblake,
armbru, seiden, nrb, scgl, frankja, berrange, clg
The modification of the CPU attributes are done through a monitor
commands.
It allows to move the core inside the topology tree to optimise
the cache usage in the case the host's hypervizor previously
moved the CPU.
The same command allows to modifiy the CPU attributes modifiers
like polarization entitlement and the dedicated attribute to notify
the guest if the host admin modified scheduling or dedication of a vCPU.
With this knowledge the guest has the possibility to optimize the
usage of the vCPUs.
Signed-off-by: Pierre Morel <pmorel@linux.ibm.com>
---
qapi/machine-target.json | 29 ++++++++
include/monitor/hmp.h | 1 +
hw/s390x/cpu-topology.c | 141 +++++++++++++++++++++++++++++++++++++++
hmp-commands.hx | 16 +++++
4 files changed, 187 insertions(+)
diff --git a/qapi/machine-target.json b/qapi/machine-target.json
index 2e267fa458..75b0aa254d 100644
--- a/qapi/machine-target.json
+++ b/qapi/machine-target.json
@@ -342,3 +342,32 @@
'TARGET_S390X',
'TARGET_MIPS',
'TARGET_LOONGARCH64' ] } }
+
+##
+# @change-topology:
+#
+# @core: the vCPU ID to be moved
+# @socket: the destination socket where to move the vCPU
+# @book: the destination book where to move the vCPU
+# @drawer: the destination drawer where to move the vCPU
+# @polarity: optional polarity, default is last polarity set by the guest
+# @dedicated: optional, if the vCPU is dedicated to a real CPU
+#
+# Modifies the topology by moving the CPU inside the topology
+# tree or by changing a modifier attribute of a CPU.
+#
+# Returns: Nothing on success, the reason on failure.
+#
+# Since: <next qemu stable release, eg. 1.0>
+##
+{ 'command': 'change-topology',
+ 'data': {
+ 'core': 'int',
+ 'socket': 'int',
+ 'book': 'int',
+ 'drawer': 'int',
+ '*polarity': 'int',
+ '*dedicated': 'bool'
+ },
+ 'if': { 'all': [ 'TARGET_S390X', 'CONFIG_KVM' ] }
+}
diff --git a/include/monitor/hmp.h b/include/monitor/hmp.h
index 27f86399f7..15c36bf549 100644
--- a/include/monitor/hmp.h
+++ b/include/monitor/hmp.h
@@ -144,5 +144,6 @@ void hmp_human_readable_text_helper(Monitor *mon,
HumanReadableText *(*qmp_handler)(Error **));
void hmp_info_stats(Monitor *mon, const QDict *qdict);
void hmp_pcie_aer_inject_error(Monitor *mon, const QDict *qdict);
+void hmp_change_topology(Monitor *mon, const QDict *qdict);
#endif
diff --git a/hw/s390x/cpu-topology.c b/hw/s390x/cpu-topology.c
index b69955a1cd..0faffe657e 100644
--- a/hw/s390x/cpu-topology.c
+++ b/hw/s390x/cpu-topology.c
@@ -18,6 +18,10 @@
#include "target/s390x/cpu.h"
#include "hw/s390x/s390-virtio-ccw.h"
#include "hw/s390x/cpu-topology.h"
+#include "qapi/qapi-commands-machine-target.h"
+#include "qapi/qmp/qdict.h"
+#include "monitor/hmp.h"
+#include "monitor/monitor.h"
/*
* s390_topology is used to keep the topology information.
@@ -203,6 +207,21 @@ static void s390_topology_set_entry(S390TopologyEntry *entry,
s390_topology.sockets[s390_socket_nb(id)]++;
}
+/**
+ * s390_topology_clear_entry:
+ * @entry: Topology entry to setup
+ * @id: topology id to use for the setup
+ *
+ * Clear the core bit inside the topology mask and
+ * decrements the number of cores for the socket.
+ */
+static void s390_topology_clear_entry(S390TopologyEntry *entry,
+ s390_topology_id id)
+{
+ clear_bit(63 - id.core, &entry->mask);
+ s390_topology.sockets[s390_socket_nb(id)]--;
+}
+
/**
* s390_topology_new_entry:
* @id: s390_topology_id to add
@@ -383,3 +402,125 @@ void s390_topology_set_cpu(MachineState *ms, S390CPU *cpu, Error **errp)
s390_topology_insert(id);
}
+
+/*
+ * qmp and hmp implementations
+ */
+
+static S390TopologyEntry *s390_topology_core_to_entry(int core)
+{
+ S390TopologyEntry *entry;
+
+ QTAILQ_FOREACH(entry, &s390_topology.list, next) {
+ if (entry->mask & (1UL << (63 - core))) {
+ return entry;
+ }
+ }
+ return NULL;
+}
+
+static void s390_change_topology(Error **errp, int64_t core, int64_t socket,
+ int64_t book, int64_t drawer,
+ int64_t polarity, bool dedicated)
+{
+ S390TopologyEntry *entry;
+ s390_topology_id new_id;
+ s390_topology_id old_id;
+ Error *local_error = NULL;
+
+ /* Get the old entry */
+ entry = s390_topology_core_to_entry(core);
+ if (!entry) {
+ error_setg(errp, "No core %ld", core);
+ return;
+ }
+
+ /* Compute old topology id */
+ old_id = entry->id;
+ old_id.core = core;
+
+ /* Compute new topology id */
+ new_id = entry->id;
+ new_id.core = core;
+ new_id.socket = socket;
+ new_id.book = book;
+ new_id.drawer = drawer;
+ new_id.p = polarity;
+ new_id.d = dedicated;
+ new_id.type = S390_TOPOLOGY_CPU_IFL;
+
+ /* Same topology entry, nothing to do */
+ if (entry->id.id == new_id.id) {
+ return;
+ }
+
+ /* Check for space on the socket if ids are different */
+ if ((s390_socket_nb(old_id) != s390_socket_nb(new_id)) &&
+ (s390_topology.sockets[s390_socket_nb(new_id)] >=
+ s390_topology.smp->sockets)) {
+ error_setg(errp, "No more space on this socket");
+ return;
+ }
+
+ /* Verify the new topology */
+ s390_topology_check(&local_error, new_id);
+ if (local_error) {
+ error_propagate(errp, local_error);
+ return;
+ }
+
+ /* Clear the old topology */
+ s390_topology_clear_entry(entry, old_id);
+
+ /* Insert the new topology */
+ s390_topology_insert(new_id);
+
+ /* Remove unused entry */
+ if (!entry->mask) {
+ QTAILQ_REMOVE(&s390_topology.list, entry, next);
+ g_free(entry);
+ }
+
+ /* Advertise the topology change */
+ s390_cpu_topology_set();
+}
+
+void qmp_change_topology(int64_t core, int64_t socket,
+ int64_t book, int64_t drawer,
+ bool has_polarity, int64_t polarity,
+ bool has_dedicated, bool dedicated,
+ Error **errp)
+{
+ Error *local_err = NULL;
+
+ if (!s390_has_topology()) {
+ error_setg(&local_err, "This machine doesn't support topology");
+ return;
+ }
+ s390_change_topology(&local_err, core, socket, book, drawer,
+ polarity, dedicated);
+ if (local_err) {
+ error_propagate(errp, local_err);
+ }
+}
+
+void hmp_change_topology(Monitor *mon, const QDict *qdict)
+{
+ const int64_t core = qdict_get_int(qdict, "core");
+ const int64_t socket = qdict_get_int(qdict, "socket");
+ const int64_t book = qdict_get_int(qdict, "book");
+ const int64_t drawer = qdict_get_int(qdict, "drawer");
+ bool has_polarity = qdict_haskey(qdict, "polarity");
+ const int64_t polarity = qdict_get_try_int(qdict, "polarity", 0);
+ bool has_dedicated = qdict_haskey(qdict, "dedicated");
+ const bool dedicated = qdict_get_try_bool(qdict, "dedicated", false);
+ Error *local_err = NULL;
+
+ qmp_change_topology(core, socket, book, drawer,
+ has_polarity, polarity,
+ has_dedicated, dedicated,
+ &local_err);
+ if (hmp_handle_error(mon, local_err)) {
+ return;
+ }
+}
diff --git a/hmp-commands.hx b/hmp-commands.hx
index 673e39a697..a617cfed0d 100644
--- a/hmp-commands.hx
+++ b/hmp-commands.hx
@@ -1815,3 +1815,19 @@ SRST
Dump the FDT in dtb format to *filename*.
ERST
#endif
+
+#if defined(TARGET_S390X) && defined(CONFIG_KVM)
+ {
+ .name = "change-topology",
+ .args_type = "core:l,socket:l,book:l,drawer:l,polarity:l?,dedicated:b?",
+ .params = "core socket book drawer [polarity] [dedicated]",
+ .help = "Move CPU 'core' to 'socket/book/drawer' "
+ "optionaly modifies polarity and dedication",
+ .cmd = hmp_change_topology,
+ },
+
+SRST
+``change-topology`` *core* *socket* *book* *drawer* *polarity* *dedicated*
+ Moves the CPU *core* to *socket* *book* *drawer* with *polarity* *dedicated*.
+ERST
+#endif
--
2.31.1
^ permalink raw reply related [flat|nested] 80+ messages in thread
* [PATCH v14 09/11] qapi/s390/cpu topology: monitor query topology information
2023-01-05 14:53 [PATCH v14 00/11] s390x: CPU Topology Pierre Morel
` (7 preceding siblings ...)
2023-01-05 14:53 ` Pierre Morel
@ 2023-01-05 14:53 ` Pierre Morel
2023-01-12 11:48 ` Thomas Huth
2023-01-12 12:10 ` Daniel P. Berrangé
2023-01-05 14:53 ` [PATCH v14 10/11] qapi/s390/cpu topology: POLARITY_CHANGE qapi event Pierre Morel
2023-01-05 14:53 ` [PATCH v14 11/11] docs/s390x/cpu topology: document s390x cpu topology Pierre Morel
10 siblings, 2 replies; 80+ messages in thread
From: Pierre Morel @ 2023-01-05 14:53 UTC (permalink / raw)
To: qemu-s390x
Cc: qemu-devel, borntraeger, pasic, richard.henderson, david, thuth,
cohuck, mst, pbonzini, kvm, ehabkost, marcel.apfelbaum, eblake,
armbru, seiden, nrb, scgl, frankja, berrange, clg
Reporting the current topology informations to the admin through
the QEMU monitor.
Signed-off-by: Pierre Morel <pmorel@linux.ibm.com>
---
qapi/machine-target.json | 66 ++++++++++++++++++++++++++++++++++
include/monitor/hmp.h | 1 +
hw/s390x/cpu-topology.c | 76 ++++++++++++++++++++++++++++++++++++++++
hmp-commands-info.hx | 16 +++++++++
4 files changed, 159 insertions(+)
diff --git a/qapi/machine-target.json b/qapi/machine-target.json
index 75b0aa254d..927618a78f 100644
--- a/qapi/machine-target.json
+++ b/qapi/machine-target.json
@@ -371,3 +371,69 @@
},
'if': { 'all': [ 'TARGET_S390X', 'CONFIG_KVM' ] }
}
+
+##
+# @S390CpuTopology:
+#
+# CPU Topology information
+#
+# @drawer: the destination drawer where to move the vCPU
+#
+# @book: the destination book where to move the vCPU
+#
+# @socket: the destination socket where to move the vCPU
+#
+# @polarity: optional polarity, default is last polarity set by the guest
+#
+# @dedicated: optional, if the vCPU is dedicated to a real CPU
+#
+# @origin: offset of the first bit of the core mask
+#
+# @mask: mask of the cores sharing the same topology
+#
+# Since: 8.0
+##
+{ 'struct': 'S390CpuTopology',
+ 'data': {
+ 'drawer': 'int',
+ 'book': 'int',
+ 'socket': 'int',
+ 'polarity': 'int',
+ 'dedicated': 'bool',
+ 'origin': 'int',
+ 'mask': 'str'
+ },
+ 'if': { 'all': [ 'TARGET_S390X', 'CONFIG_KVM' ] }
+}
+
+##
+# @query-topology:
+#
+# Return information about CPU Topology
+#
+# Returns a @CpuTopology instance describing the CPU Toplogy
+# being currently used by QEMU.
+#
+# Since: 8.0
+#
+# Example:
+#
+# -> { "execute": "cpu-topology" }
+# <- {"return": [
+# {
+# "drawer": 0,
+# "book": 0,
+# "socket": 0,
+# "polarity": 0,
+# "dedicated": true,
+# "origin": 0,
+# "mask": 0xc000000000000000,
+# },
+# ]
+# }
+#
+##
+{ 'command': 'query-topology',
+ 'returns': ['S390CpuTopology'],
+ 'if': { 'all': [ 'TARGET_S390X', 'CONFIG_KVM' ] }
+}
diff --git a/include/monitor/hmp.h b/include/monitor/hmp.h
index 15c36bf549..0b3c758231 100644
--- a/include/monitor/hmp.h
+++ b/include/monitor/hmp.h
@@ -145,5 +145,6 @@ void hmp_human_readable_text_helper(Monitor *mon,
void hmp_info_stats(Monitor *mon, const QDict *qdict);
void hmp_pcie_aer_inject_error(Monitor *mon, const QDict *qdict);
void hmp_change_topology(Monitor *mon, const QDict *qdict);
+void hmp_query_topology(Monitor *mon, const QDict *qdict);
#endif
diff --git a/hw/s390x/cpu-topology.c b/hw/s390x/cpu-topology.c
index 0faffe657e..c3748654ff 100644
--- a/hw/s390x/cpu-topology.c
+++ b/hw/s390x/cpu-topology.c
@@ -524,3 +524,79 @@ void hmp_change_topology(Monitor *mon, const QDict *qdict)
return;
}
}
+
+static S390CpuTopologyList *s390_cpu_topology_list(void)
+{
+ S390CpuTopologyList *head = NULL;
+ S390TopologyEntry *entry;
+
+ QTAILQ_FOREACH_REVERSE(entry, &s390_topology.list, next) {
+ S390CpuTopology *item = g_new0(typeof(*item), 1);
+
+ item->drawer = entry->id.drawer;
+ item->book = entry->id.book;
+ item->socket = entry->id.socket;
+ item->polarity = entry->id.p;
+ if (entry->id.d) {
+ item->dedicated = true;
+ }
+ item->origin = entry->id.origin;
+ item->mask = g_strdup_printf("0x%016lx", entry->mask);
+
+ QAPI_LIST_PREPEND(head, item);
+ }
+ return head;
+}
+
+S390CpuTopologyList *qmp_query_topology(Error **errp)
+{
+ if (!s390_has_topology()) {
+ error_setg(errp, "This machine doesn't support topology");
+ return NULL;
+ }
+
+ return s390_cpu_topology_list();
+}
+
+void hmp_query_topology(Monitor *mon, const QDict *qdict)
+{
+ Error *err = NULL;
+ S390CpuTopologyList *l = qmp_query_topology(&err);
+
+ if (hmp_handle_error(mon, err)) {
+ return;
+ }
+
+ monitor_printf(mon, "CPU Topology:\n");
+ while (l) {
+ uint64_t d = -1UL;
+ uint64_t b = -1UL;
+ uint64_t s = -1UL;
+ uint64_t p = -1UL;
+ uint64_t dd = -1UL;
+
+ if (d != l->value->drawer) {
+ monitor_printf(mon, " drawer : \"%" PRIu64 "\"\n",
+ l->value->drawer);
+ }
+ if (b != l->value->book) {
+ monitor_printf(mon, " book : \"%" PRIu64 "\"\n",
+ l->value->book);
+ }
+ if (s != l->value->socket) {
+ monitor_printf(mon, " socket : \"%" PRIu64 "\"\n",
+ l->value->socket);
+ }
+ if (p != l->value->polarity) {
+ monitor_printf(mon, " polarity : \"%" PRIu64 "\"\n",
+ l->value->polarity);
+ }
+ if (dd != l->value->dedicated) {
+ monitor_printf(mon, " dedicated: \"%d\"\n", l->value->dedicated);
+ }
+ monitor_printf(mon, " mask : \"%s\"\n", l->value->mask);
+
+
+ l = l->next;
+ }
+}
diff --git a/hmp-commands-info.hx b/hmp-commands-info.hx
index 754b1e8408..5730a47f71 100644
--- a/hmp-commands-info.hx
+++ b/hmp-commands-info.hx
@@ -993,3 +993,19 @@ SRST
``info virtio-queue-element`` *path* *queue* [*index*]
Display element of a given virtio queue
ERST
+
+#if defined(TARGET_S390X) && defined(CONFIG_KVM)
+ {
+ .name = "query-topology",
+ .args_type = "",
+ .params = "",
+ .help = "Show information about CPU topology",
+ .cmd = hmp_query_topology,
+ .flags = "p",
+ },
+
+SRST
+ ``info query-topology``
+ Show information about CPU topology
+ERST
+#endif
--
2.31.1
^ permalink raw reply related [flat|nested] 80+ messages in thread
* [PATCH v14 10/11] qapi/s390/cpu topology: POLARITY_CHANGE qapi event
2023-01-05 14:53 [PATCH v14 00/11] s390x: CPU Topology Pierre Morel
` (8 preceding siblings ...)
2023-01-05 14:53 ` [PATCH v14 09/11] qapi/s390/cpu topology: monitor query topology information Pierre Morel
@ 2023-01-05 14:53 ` Pierre Morel
2023-01-12 11:52 ` Thomas Huth
2023-01-05 14:53 ` [PATCH v14 11/11] docs/s390x/cpu topology: document s390x cpu topology Pierre Morel
10 siblings, 1 reply; 80+ messages in thread
From: Pierre Morel @ 2023-01-05 14:53 UTC (permalink / raw)
To: qemu-s390x
Cc: qemu-devel, borntraeger, pasic, richard.henderson, david, thuth,
cohuck, mst, pbonzini, kvm, ehabkost, marcel.apfelbaum, eblake,
armbru, seiden, nrb, scgl, frankja, berrange, clg
When the guest asks to change the polarity this change
is forwarded to the admin using QAPI.
The admin is supposed to take according decisions concerning
CPU provisioning.
Signed-off-by: Pierre Morel <pmorel@linux.ibm.com>
---
qapi/machine-target.json | 21 +++++++++++++++++++++
hw/s390x/cpu-topology.c | 2 ++
2 files changed, 23 insertions(+)
diff --git a/qapi/machine-target.json b/qapi/machine-target.json
index 927618a78f..10235cfb45 100644
--- a/qapi/machine-target.json
+++ b/qapi/machine-target.json
@@ -437,3 +437,24 @@
'returns': ['S390CpuTopology'],
'if': { 'all': [ 'TARGET_S390X', 'CONFIG_KVM' ] }
}
+
+##
+# @POLARITY_CHANGE:
+#
+# Emitted when the guest asks to change the polarity.
+#
+# @polarity: polarity specified by the guest
+#
+# Since: 8.0
+#
+# Example:
+#
+# <- { "event": "POLARITY_CHANGE",
+# "data": { "polarity": 0 },
+# "timestamp": { "seconds": 1401385907, "microseconds": 422329 } }
+#
+##
+{ 'event': 'POLARITY_CHANGE',
+ 'data': { 'polarity': 'int' },
+ 'if': { 'all': [ 'TARGET_S390X', 'CONFIG_KVM'] }
+}
diff --git a/hw/s390x/cpu-topology.c b/hw/s390x/cpu-topology.c
index c3748654ff..45621387d5 100644
--- a/hw/s390x/cpu-topology.c
+++ b/hw/s390x/cpu-topology.c
@@ -19,6 +19,7 @@
#include "hw/s390x/s390-virtio-ccw.h"
#include "hw/s390x/cpu-topology.h"
#include "qapi/qapi-commands-machine-target.h"
+#include "qapi/qapi-events-machine-target.h"
#include "qapi/qmp/qdict.h"
#include "monitor/hmp.h"
#include "monitor/monitor.h"
@@ -128,6 +129,7 @@ void s390_topology_set_polarity(int polarity)
}
}
s390_cpu_topology_set();
+ qapi_event_send_polarity_change(polarity);
}
/*
--
2.31.1
^ permalink raw reply related [flat|nested] 80+ messages in thread
* [PATCH v14 11/11] docs/s390x/cpu topology: document s390x cpu topology
2023-01-05 14:53 [PATCH v14 00/11] s390x: CPU Topology Pierre Morel
` (9 preceding siblings ...)
2023-01-05 14:53 ` [PATCH v14 10/11] qapi/s390/cpu topology: POLARITY_CHANGE qapi event Pierre Morel
@ 2023-01-05 14:53 ` Pierre Morel
2023-01-12 11:46 ` Thomas Huth
2023-01-12 11:58 ` Daniel P. Berrangé
10 siblings, 2 replies; 80+ messages in thread
From: Pierre Morel @ 2023-01-05 14:53 UTC (permalink / raw)
To: qemu-s390x
Cc: qemu-devel, borntraeger, pasic, richard.henderson, david, thuth,
cohuck, mst, pbonzini, kvm, ehabkost, marcel.apfelbaum, eblake,
armbru, seiden, nrb, scgl, frankja, berrange, clg
Add some basic examples for the definition of cpu topology
in s390x.
Signed-off-by: Pierre Morel <pmorel@linux.ibm.com>
---
docs/system/s390x/cpu-topology.rst | 292 +++++++++++++++++++++++++++++
docs/system/target-s390x.rst | 1 +
2 files changed, 293 insertions(+)
create mode 100644 docs/system/s390x/cpu-topology.rst
diff --git a/docs/system/s390x/cpu-topology.rst b/docs/system/s390x/cpu-topology.rst
new file mode 100644
index 0000000000..0020b70b50
--- /dev/null
+++ b/docs/system/s390x/cpu-topology.rst
@@ -0,0 +1,292 @@
+CPU Topology on s390x
+=====================
+
+CPU Topology on S390x provides up to 5 levels of topology containers:
+nodes, drawers, books, sockets and CPUs.
+While the higher level containers, Containers Topology List Entries,
+(Containers TLE) define a tree hierarchy, the lowest level of topology
+definition, the CPU Topology List Entry (CPU TLE), provides the placement
+of the CPUs inside the parent container.
+
+Currently QEMU CPU topology uses a single level of container: the sockets.
+
+For backward compatibility, threads can be declared on the ``-smp`` command
+line. They will be seen as CPUs by the guest as long as multithreading
+is not really supported by QEMU for S390.
+
+Beside the topological tree, S390x provides 3 CPU attributes:
+- CPU type
+- polarity entitlement
+- dedication
+
+Prerequisites
+-------------
+
+To use CPU Topology a Linux QEMU/KVM machine providing the CPU Topology facility
+(STFLE bit 11) is required.
+
+However, since this facility has been enabled by default in an early version
+of QEMU, we use a capability, ``KVM_CAP_S390_CPU_TOPOLOGY``, to notify KVM
+QEMU use of the CPU Topology.
+
+Enabling CPU topology
+---------------------
+
+Currently, CPU topology is only enabled in the host model.
+
+Enabling CPU topology in a CPU model is done by setting the CPU flag
+``ctop`` to ``on`` like in:
+
+.. code-block:: bash
+
+ -cpu gen16b,ctop=on
+
+Having the topology disabled by default allows migration between
+old and new QEMU without adding new flags.
+
+Default topology usage
+----------------------
+
+The CPU Topology, can be specified on the QEMU command line
+with the ``-smp`` or the ``-device`` QEMU command arguments
+without using any new attributes.
+In this case, the topology will be calculated by simply adding
+to the topology the cores based on the core-id starting with
+core-0 at position 0 of socket-0, book-0, drawer-0 with default
+modifier attributes: horizontal polarity and no dedication.
+
+In the following machine we define 8 sockets with 4 cores each.
+Note that S390 QEMU machines do not implement multithreading.
+
+.. code-block:: bash
+
+ $ qemu-system-s390x -m 2G \
+ -cpu gen16b,ctop=on \
+ -smp cpus=5,sockets=8,cores=4,maxcpus=32 \
+ -device host-s390x-cpu,core-id=14 \
+
+New CPUs can be plugged using the device_add hmp command like in:
+
+.. code-block:: bash
+
+ (qemu) device_add gen16b-s390x-cpu,core-id=9
+
+The core-id defines the placement of the core in the topology by
+starting with core 0 in socket 0 up to maxcpus.
+
+In the example above:
+
+* There are 5 CPUs provided to the guest with the ``-smp`` command line
+ They will take the core-ids 0,1,2,3,4
+ As we have 4 cores in a socket, we have 4 CPUs provided
+ to the guest in socket 0, with core-ids 0,1,2,3.
+ The last cpu, with core-id 4, will be on socket 1.
+
+* the core with ID 14 provided by the ``-device`` command line will
+ be placed in socket 3, with core-id 14
+
+* the core with ID 9 provided by the ``device_add`` qmp command will
+ be placed in socket 2, with core-id 9
+
+Note that the core ID is machine wide and the CPU TLE masks provided
+by the STSI instruction will be written in a big endian mask:
+
+* in socket 0: 0xf000000000000000 (core id 0,1,2,3)
+* in socket 1: 0x0800000000000000 (core id 4)
+* in socket 2: 0x0040000000000000 (core id 9)
+* in socket 3: 0x0002000000000000 (core id 14)
+
+Defining the topology on command line
+-------------------------------------
+
+The topology can be defined entirely during the CPU definition,
+with the exception of CPU 0 which must be defined with the -smp
+argument.
+
+For example, here we set the position of the cores 1,2,3 on
+drawer 1, book 1, socket 2 and cores 0,9 and 14 on drawer 0,
+book 0, socket 0 with all horizontal polarity and not dedicated.
+The core 4, will be set on its default position on socket 1
+(since we have 4 core per socket) and we define it with dedication and
+vertical high entitlement.
+
+.. code-block:: bash
+
+ $ qemu-system-s390x -m 2G \
+ -cpu gen16b,ctop=on \
+ -smp cpus=1,sockets=8,cores=4,maxcpus=32 \
+ \
+ -device gen16b-s390x-cpu,drawer-id=1,book-id=1,socket-id=2,core-id=1 \
+ -device gen16b-s390x-cpu,drawer-id=1,book-id=1,socket-id=2,core-id=2 \
+ -device gen16b-s390x-cpu,drawer-id=1,book-id=1,socket-id=2,core-id=3 \
+ \
+ -device gen16b-s390x-cpu,drawer-id=0,book-id=0,socket-id=0,core-id=9 \
+ -device gen16b-s390x-cpu,drawer-id=0,book-id=0,socket-id=0,core-id=14 \
+ \
+ -device gen16b-s390x-cpu,core-id=4,dedicated=on,polarity=3 \
+
+Polarity and dedication
+-----------------------
+
+Polarity can be of two types: horizontal or vertical.
+
+The horizontal polarization specifies that all guest's vCPUs get
+almost the same amount of provisioning of real CPU by the host.
+
+The vertical polarization specifies that guest's vCPU can get
+different real CPU provisions:
+
+- a vCPU with Vertical high entitlement specifies that this
+ vCPU gets 100% of the real CPU provisioning.
+
+- a vCPU with Vertical medium entitlement specifies that this
+ vCPU shares the real CPU with other vCPU.
+
+- a vCPU with Vertical low entitlement specifies that this
+ vCPU only get real CPU provisioning when no other vCPU need it.
+
+In the case a vCPU with vertical high entitlement does not use
+the real CPU, the unused "slack" can be dispatched to other vCPU
+with medium or low entitlement.
+
+The host indicates to the guest how the real CPU resources are
+provided to the vCPUs through the SYSIB with two polarity bits
+inside the CPU TLE.
+
+Bits d - Polarization
+0 0 Horizontal
+0 1 Vertical low entitlement
+1 0 Vertical medium entitlement
+1 1 Vertical high entitlement
+
+A subsystem reset puts all vCPU of the configuration into the
+horizontal polarization.
+
+The admin specifies the dedicated bit when the vCPU is dedicated
+to a single real CPU.
+
+As for the Linux admin, the dedicated bit is an indication on the
+affinity of a vCPU for a real CPU while the entitlement indicates the
+sharing or exclusivity of use.
+
+QAPI interface for topology
+---------------------------
+
+Let's start QEMU with the following command:
+
+.. code-block:: bash
+
+ sudo /usr/local/bin/qemu-system-s390x \
+ -enable-kvm \
+ -cpu z14,ctop=on \
+ -smp 1,drawers=3,books=3,sockets=2,cores=2,maxcpus=36 \
+ \
+ -device z14-s390x-cpu,core-id=19,polarity=3 \
+ -device z14-s390x-cpu,core-id=11,polarity=1 \
+ -device z14-s390x-cpu,core-id=12,polarity=3 \
+ ...
+
+and see the result when using of the QAPI interface.
+
+query-topology
++++++++++++++++
+
+The command cpu-topology allows the admin to query the topology
+tree and modifier for all configured vCPU.
+
+.. code-block:: QMP
+
+ -> { "execute": "query-topology" }
+ {"return":
+ [
+ {
+ "origin": 0,
+ "dedicated": false,
+ "book": 0,
+ "socket": 0,
+ "drawer": 0,
+ "polarity": 0,
+ "mask": "0x8000000000000000"
+ },
+ {
+ "origin": 0,
+ "dedicated": false,
+ "book": 2,
+ "socket": 1,
+ "drawer": 0,
+ "polarity": 1,
+ "mask": "0x0010000000000000"
+ },
+ {
+ "origin": 0,
+ "dedicated": false,
+ "book": 0,
+ "socket": 0,
+ "drawer": 1,
+ "polarity": 3,
+ "mask": "0x0008000000000000"
+ },
+ {
+ "origin": 0,
+ "dedicated": false,
+ "book": 1,
+ "socket": 1,
+ "drawer": 1,
+ "polarity": 3,
+ "mask": "0x0000100000000000"
+ }
+ ]
+ }
+
+change-topology
++++++++++++++++
+
+The command change-topology allows the admin to modify the topology
+tree or the topology modifiers of a vCPU in the configuration.
+
+.. code-block:: QMP
+
+ -> { "execute": "change-topology",
+ "arguments": {
+ "core": 11,
+ "socket": 0,
+ "book": 0,
+ "drawer": 0,
+ "polarity": 0,
+ "dedicated": false
+ }
+ }
+ <- {"return": {}}
+
+
+event POLARITY_CHANGE
++++++++++++++++++++++
+
+When a guest is requesting a modification of the polarity,
+QEMU sends a POLARITY_CHANGE event.
+
+When requesting the change, the guest only specifies horizontal or
+vertical polarity.
+The dedication and fine grain vertical entitlement depends on admin
+to set according to its response to this event.
+
+Note that a vertical polarized dedicated vCPU can only have a high
+entitlement, this gives 6 possibilities for a vCPU polarity:
+
+- Horizontal
+- Horizontal dedicated
+- Vertical low
+- Vertical medium
+- Vertical high
+- Vertical high dedicated
+
+Example of the event received when the guest issues PTF(0) to request
+an horizontal polarity:
+
+.. code-block:: QMP
+
+ <- { "event": "POLARITY_CHANGE",
+ "data": { "polarity": 0 },
+ "timestamp": { "seconds": 1401385907, "microseconds": 422329 } }
+
+
diff --git a/docs/system/target-s390x.rst b/docs/system/target-s390x.rst
index c636f64113..ff0ffe04f3 100644
--- a/docs/system/target-s390x.rst
+++ b/docs/system/target-s390x.rst
@@ -33,3 +33,4 @@ Architectural features
.. toctree::
s390x/bootdevices
s390x/protvirt
+ s390x/cpu-topology
--
2.31.1
^ permalink raw reply related [flat|nested] 80+ messages in thread
* Re: [PATCH v14 01/11] s390x/cpu topology: adding s390 specificities to CPU topology
2023-01-05 14:53 ` [PATCH v14 01/11] s390x/cpu topology: adding s390 specificities to CPU topology Pierre Morel
@ 2023-01-10 11:37 ` Thomas Huth
2023-01-16 16:32 ` Pierre Morel
2023-01-13 16:58 ` Nina Schoetterl-Glausch
1 sibling, 1 reply; 80+ messages in thread
From: Thomas Huth @ 2023-01-10 11:37 UTC (permalink / raw)
To: Pierre Morel, qemu-s390x
Cc: qemu-devel, borntraeger, pasic, richard.henderson, david, cohuck,
mst, pbonzini, kvm, ehabkost, marcel.apfelbaum, eblake, armbru,
seiden, nrb, scgl, frankja, berrange, clg
On 05/01/2023 15.53, Pierre Morel wrote:
> S390 adds two new SMP levels, drawers and books to the CPU
> topology.
> The S390 CPU have specific toplogy features like dedication
> and polarity to give to the guest indications on the host
> vCPUs scheduling and help the guest take the best decisions
> on the scheduling of threads on the vCPUs.
>
> Let us provide the SMP properties with books and drawers levels
> and S390 CPU with dedication and polarity,
>
> Signed-off-by: Pierre Morel <pmorel@linux.ibm.com>
> ---
...
> diff --git a/qapi/machine.json b/qapi/machine.json
> index b9228a5e46..ff8f2b0e84 100644
> --- a/qapi/machine.json
> +++ b/qapi/machine.json
> @@ -900,13 +900,15 @@
> # a CPU is being hotplugged.
> #
> # @node-id: NUMA node ID the CPU belongs to
> -# @socket-id: socket number within node/board the CPU belongs to
> +# @drawer-id: drawer number within node/board the CPU belongs to
> +# @book-id: book number within drawer/node/board the CPU belongs to
> +# @socket-id: socket number within book/node/board the CPU belongs to
I think the new entries need a "(since 8.0)" comment (similar to die-id and
cluster-id below).
Other question: Do we have "node-id"s on s390x? If not, is that similar to
books or drawers, i.e. just another word? If so, we should maybe rather
re-use "nodes" instead of introducing a new name for the same thing?
> # @die-id: die number within socket the CPU belongs to (since 4.1)
> # @cluster-id: cluster number within die the CPU belongs to (since 7.1)
> # @core-id: core number within cluster the CPU belongs to
> # @thread-id: thread number within core the CPU belongs to
> #
> -# Note: currently there are 6 properties that could be present
> +# Note: currently there are 8 properties that could be present
> # but management should be prepared to pass through other
> # properties with device_add command to allow for future
> # interface extension. This also requires the filed names to be kept in
> @@ -916,6 +918,8 @@
> ##
> { 'struct': 'CpuInstanceProperties',
> 'data': { '*node-id': 'int',
> + '*drawer-id': 'int',
> + '*book-id': 'int',
> '*socket-id': 'int',
> '*die-id': 'int',
> '*cluster-id': 'int',
> @@ -1465,6 +1469,10 @@
> #
> # @cpus: number of virtual CPUs in the virtual machine
> #
> +# @drawers: number of drawers in the CPU topology
> +#
> +# @books: number of books in the CPU topology
> +#
These also need a "(since 8.0)" comment at the end.
> # @sockets: number of sockets in the CPU topology
> #
> # @dies: number of dies per socket in the CPU topology
> @@ -1481,6 +1489,8 @@
> ##
> { 'struct': 'SMPConfiguration', 'data': {
> '*cpus': 'int',
> + '*drawers': 'int',
> + '*books': 'int',
> '*sockets': 'int',
> '*dies': 'int',
> '*clusters': 'int',
...
> diff --git a/qemu-options.hx b/qemu-options.hx
> index 7f99d15b23..8dc9a4c052 100644
> --- a/qemu-options.hx
> +++ b/qemu-options.hx
> @@ -250,11 +250,13 @@ SRST
> ERST
>
> DEF("smp", HAS_ARG, QEMU_OPTION_smp,
> - "-smp [[cpus=]n][,maxcpus=maxcpus][,sockets=sockets][,dies=dies][,clusters=clusters][,cores=cores][,threads=threads]\n"
> + "-smp [[cpus=]n][,maxcpus=maxcpus][,drawers=drawers][,books=books][,sockets=sockets][,dies=dies][,clusters=clusters][,cores=cores][,threads=threads]\n"
This line now got too long. Please add a newline inbetween.
> " set the number of initial CPUs to 'n' [default=1]\n"
> " maxcpus= maximum number of total CPUs, including\n"
> " offline CPUs for hotplug, etc\n"
> - " sockets= number of sockets on the machine board\n"
> + " drawers= number of drawers on the machine board\n"
> + " books= number of books in one drawer\n"
> + " sockets= number of sockets in one book\n"
> " dies= number of dies in one socket\n"
> " clusters= number of clusters in one die\n"
> " cores= number of cores in one cluster\n"
Thomas
^ permalink raw reply [flat|nested] 80+ messages in thread
* Re: [PATCH v14 02/11] s390x/cpu topology: add topology entries on CPU hotplug
2023-01-05 14:53 ` [PATCH v14 02/11] s390x/cpu topology: add topology entries on CPU hotplug Pierre Morel
@ 2023-01-10 13:00 ` Thomas Huth
2023-01-11 9:23 ` Nina Schoetterl-Glausch
2023-01-16 18:24 ` Pierre Morel
2023-01-13 18:15 ` Nina Schoetterl-Glausch
1 sibling, 2 replies; 80+ messages in thread
From: Thomas Huth @ 2023-01-10 13:00 UTC (permalink / raw)
To: Pierre Morel, qemu-s390x
Cc: qemu-devel, borntraeger, pasic, richard.henderson, david, cohuck,
mst, pbonzini, kvm, ehabkost, marcel.apfelbaum, eblake, armbru,
seiden, nrb, scgl, frankja, berrange, clg
On 05/01/2023 15.53, Pierre Morel wrote:
> The topology information are attributes of the CPU and are
> specified during the CPU device creation.
>
> On hot plug, we gather the topology information on the core,
> creates a list of topology entries, each entry contains a single
> core mask of each core with identical topology and finaly we
> orders the list in topological order.
> The topological order is, from higher to lower priority:
> - physical topology
> - drawer
> - book
> - socket
> - core origin, offset in 64bit increment from core 0.
> - modifier attributes
> - CPU type
> - polarization entitlement
> - dedication
>
> The possibility to insert a CPU in a mask is dependent on the
> number of cores allowed in a socket, a book or a drawer, the
> checking is done during the hot plug of the CPU to have an
> immediate answer.
>
> If the complete topology is not specified, the core is added
> in the physical topology based on its core ID and it gets
> defaults values for the modifier attributes.
>
> This way, starting QEMU without specifying the topology can
> still get some adventage of the CPU topology.
s/adventage/advantage/
> Signed-off-by: Pierre Morel <pmorel@linux.ibm.com>
> ---
> include/hw/s390x/cpu-topology.h | 48 ++++++
> hw/s390x/cpu-topology.c | 293 ++++++++++++++++++++++++++++++++
> hw/s390x/s390-virtio-ccw.c | 10 ++
> hw/s390x/meson.build | 1 +
> 4 files changed, 352 insertions(+)
> create mode 100644 hw/s390x/cpu-topology.c
>
> diff --git a/include/hw/s390x/cpu-topology.h b/include/hw/s390x/cpu-topology.h
> index d945b57fc3..b3fd752d8d 100644
> --- a/include/hw/s390x/cpu-topology.h
> +++ b/include/hw/s390x/cpu-topology.h
> @@ -10,7 +10,11 @@
> #ifndef HW_S390X_CPU_TOPOLOGY_H
> #define HW_S390X_CPU_TOPOLOGY_H
>
> +#include "qemu/queue.h"
> +#include "hw/boards.h"
> +
> #define S390_TOPOLOGY_CPU_IFL 0x03
> +#define S390_TOPOLOGY_MAX_ORIGIN ((63 + S390_MAX_CPUS) / 64)
>
> #define S390_TOPOLOGY_POLARITY_HORIZONTAL 0x00
> #define S390_TOPOLOGY_POLARITY_VERTICAL_LOW 0x01
> @@ -20,4 +24,48 @@
> #define S390_TOPOLOGY_SHARED 0x00
> #define S390_TOPOLOGY_DEDICATED 0x01
>
> +typedef union s390_topology_id {
> + uint64_t id;
> + struct {
> + uint64_t level_6:8; /* byte 0 BE */
> + uint64_t level_5:8; /* byte 1 BE */
> + uint64_t drawer:8; /* byte 2 BE */
> + uint64_t book:8; /* byte 3 BE */
> + uint64_t socket:8; /* byte 4 BE */
> + uint64_t rsrv:5;
> + uint64_t d:1;
> + uint64_t p:2; /* byte 5 BE */
> + uint64_t type:8; /* byte 6 BE */
> + uint64_t origin:2;
> + uint64_t core:6; /* byte 7 BE */
> + };
> +} s390_topology_id;
Bitmasks are OK for code that will definitely only ever work with KVM ...
but this will certainly fail completely if we ever try to get it running
with TCG later. Do we care? ... if so, you should certainly avoid a bitfield
here. Especially since most of the fields are 8-bit anyway and could easily
be represented by a "uint8_t" variable. Otherwise, just ignore my comment.
> +#define TOPO_CPU_MASK 0x000000000000003fUL
> +
> +typedef struct S390TopologyEntry {
> + s390_topology_id id;
> + QTAILQ_ENTRY(S390TopologyEntry) next;
> + uint64_t mask;
> +} S390TopologyEntry;
> +
> +typedef struct S390Topology {
> + QTAILQ_HEAD(, S390TopologyEntry) list;
> + uint8_t *sockets;
So this "uint8_t" basically is a hidden limit of a maximum of 256 sockets
that can be used for per book? Do we check that limit somewhere? (I looked
for it, but I didn't spot such a check)
> + CpuTopology *smp;
> +} S390Topology;
> +
> +#ifdef CONFIG_KVM
> +bool s390_has_topology(void);
> +void s390_topology_set_cpu(MachineState *ms, S390CPU *cpu, Error **errp);
> +#else
> +static inline bool s390_has_topology(void)
> +{
> + return false;
> +}
> +static inline void s390_topology_set_cpu(MachineState *ms,
> + S390CPU *cpu,
> + Error **errp) {}
> +#endif
> +extern S390Topology s390_topology;
> +
> #endif
> diff --git a/hw/s390x/cpu-topology.c b/hw/s390x/cpu-topology.c
> new file mode 100644
> index 0000000000..438055c612
> --- /dev/null
> +++ b/hw/s390x/cpu-topology.c
> @@ -0,0 +1,293 @@
> +/*
> + * CPU Topology
> + *
> + * Copyright IBM Corp. 2022
Want to update to 2023 now?
> + * Author(s): Pierre Morel <pmorel@linux.ibm.com>
> +
> + * This work is licensed under the terms of the GNU GPL, version 2 or (at
> + * your option) any later version. See the COPYING file in the top-level
> + * directory.
> + */
> +
> +#include "qemu/osdep.h"
> +#include "qapi/error.h"
> +#include "qemu/error-report.h"
> +#include "hw/qdev-properties.h"
> +#include "hw/boards.h"
> +#include "qemu/typedefs.h"
> +#include "target/s390x/cpu.h"
> +#include "hw/s390x/s390-virtio-ccw.h"
> +#include "hw/s390x/cpu-topology.h"
> +
> +/*
> + * s390_topology is used to keep the topology information.
> + * .list: queue the topology entries inside which
> + * we keep the information on the CPU topology.
> + *
> + * .smp: keeps track of the machine topology.
> + *
> + * .socket: tracks information on the count of cores per socket.
> + *
> + */
> +S390Topology s390_topology = {
> + .list = QTAILQ_HEAD_INITIALIZER(s390_topology.list),
> + .sockets = NULL, /* will be initialized after the cpu model is realized */
> +};
> +
> +/**
> + * s390_socket_nb:
> + * @id: s390_topology_id
> + *
> + * Returns the socket number used inside the socket array.
> + */
> +static int s390_socket_nb(s390_topology_id id)
> +{
> + return (id.socket + 1) * (id.book + 1) * (id.drawer + 1); > +}
I think there might be an off-by-one error in here - you likely need a "- 1"
at the very end.
For example, assume that we have one socket, one book and one drawer, so
id.socket, id.book and id.drawer would all be 0. The function then returns 1 ...
> +static void s390_topology_init(MachineState *ms)
> +{
> + CpuTopology *smp = &ms->smp;
> +
> + s390_topology.smp = smp;
> + if (!s390_topology.sockets) {
> + s390_topology.sockets = g_new0(uint8_t, smp->sockets *
> + smp->books * smp->drawers);
... but here you only allocated one byte. So you later access
s390_topology.sockets[s390_socket_nb(id)], i.e. s390_topology.sockets[1]
which is out of bounds.
> + }
> +}
Thomas
^ permalink raw reply [flat|nested] 80+ messages in thread
* Re: [PATCH v14 03/11] target/s390x/cpu topology: handle STSI(15) and build the SYSIB
2023-01-05 14:53 ` [PATCH v14 03/11] target/s390x/cpu topology: handle STSI(15) and build the SYSIB Pierre Morel
@ 2023-01-10 14:29 ` Thomas Huth
2023-01-11 9:16 ` Thomas Huth
` (2 more replies)
2023-01-16 13:11 ` Nina Schoetterl-Glausch
1 sibling, 3 replies; 80+ messages in thread
From: Thomas Huth @ 2023-01-10 14:29 UTC (permalink / raw)
To: Pierre Morel, qemu-s390x, frankja
Cc: qemu-devel, borntraeger, pasic, richard.henderson, david, cohuck,
mst, pbonzini, kvm, ehabkost, marcel.apfelbaum, eblake, armbru,
seiden, nrb, scgl, berrange, clg
On 05/01/2023 15.53, Pierre Morel wrote:
> On interception of STSI(15.1.x) the System Information Block
> (SYSIB) is built from the list of pre-ordered topology entries.
>
> Signed-off-by: Pierre Morel <pmorel@linux.ibm.com>
> ---
...
> diff --git a/target/s390x/cpu.h b/target/s390x/cpu.h
> index 39ea63a416..78988048dd 100644
> --- a/target/s390x/cpu.h
> +++ b/target/s390x/cpu.h
> @@ -561,6 +561,25 @@ typedef struct SysIB_322 {
> } SysIB_322;
> QEMU_BUILD_BUG_ON(sizeof(SysIB_322) != 4096);
>
> +#define S390_TOPOLOGY_MAG 6
> +#define S390_TOPOLOGY_MAG6 0
> +#define S390_TOPOLOGY_MAG5 1
> +#define S390_TOPOLOGY_MAG4 2
> +#define S390_TOPOLOGY_MAG3 3
> +#define S390_TOPOLOGY_MAG2 4
> +#define S390_TOPOLOGY_MAG1 5
> +/* Configuration topology */
> +typedef struct SysIB_151x {
> + uint8_t reserved0[2];
> + uint16_t length;
> + uint8_t mag[S390_TOPOLOGY_MAG];
> + uint8_t reserved1;
> + uint8_t mnest;
> + uint32_t reserved2;
> + char tle[];
> +} QEMU_PACKED QEMU_ALIGNED(8) SysIB_151x;
> +QEMU_BUILD_BUG_ON(sizeof(SysIB_151x) != 16);
> +
> typedef union SysIB {
> SysIB_111 sysib_111;
> SysIB_121 sysib_121;
> @@ -568,9 +587,68 @@ typedef union SysIB {
> SysIB_221 sysib_221;
> SysIB_222 sysib_222;
> SysIB_322 sysib_322;
> + SysIB_151x sysib_151x;
> } SysIB;
> QEMU_BUILD_BUG_ON(sizeof(SysIB) != 4096);
>
> +/*
> + * CPU Topology List provided by STSI with fc=15 provides a list
> + * of two different Topology List Entries (TLE) types to specify
> + * the topology hierarchy.
> + *
> + * - Container Topology List Entry
> + * Defines a container to contain other Topology List Entries
> + * of any type, nested containers or CPU.
> + * - CPU Topology List Entry
> + * Specifies the CPUs position, type, entitlement and polarization
> + * of the CPUs contained in the last Container TLE.
> + *
> + * There can be theoretically up to five levels of containers, QEMU
> + * uses only one level, the socket level.
I guess that sentence needs an update again, now that you've re-added the
books and drawers?
> + * A container of with a nesting level (NL) greater than 1 can only
> + * contain another container of nesting level NL-1.
> + *
> + * A container of nesting level 1 (socket), contains as many CPU TLE
> + * as needed to describe the position and qualities of all CPUs inside
> + * the container.
> + * The qualities of a CPU are polarization, entitlement and type.
> + *
> + * The CPU TLE defines the position of the CPUs of identical qualities
> + * using a 64bits mask which first bit has its offset defined by
> + * the CPU address orgin field of the CPU TLE like in:
> + * CPU address = origin * 64 + bit position within the mask
> + *
> + */
> +/* Container type Topology List Entry */
> +/* Container type Topology List Entry */
Duplicated comment.
> +typedef struct SysIBTl_container {
> + uint8_t nl;
> + uint8_t reserved[6];
> + uint8_t id;
> +} QEMU_PACKED QEMU_ALIGNED(8) SysIBTl_container;
> +QEMU_BUILD_BUG_ON(sizeof(SysIBTl_container) != 8);
> +
> +/* CPU type Topology List Entry */
> +typedef struct SysIBTl_cpu {
> + uint8_t nl;
> + uint8_t reserved0[3];
> + uint8_t reserved1:5;
> + uint8_t dedicated:1;
> + uint8_t polarity:2;
Hmmm, yet another bitfield...
> + uint8_t type;
> + uint16_t origin;
> + uint64_t mask;
> +} QEMU_PACKED QEMU_ALIGNED(8) SysIBTl_cpu;
> +QEMU_BUILD_BUG_ON(sizeof(SysIBTl_cpu) != 16);
> +
> +/* Max size of a SYSIB structure is when all CPU are alone in a container */
> +#define S390_TOPOLOGY_SYSIB_SIZE (sizeof(SysIB_151x) + \
> + S390_MAX_CPUS * (sizeof(SysIBTl_container) + \
> + sizeof(SysIBTl_cpu)))
> +
> +void insert_stsi_15_1_x(S390CPU *cpu, int sel2, __u64 addr, uint8_t ar);
> +
> /* MMU defines */
> #define ASCE_ORIGIN (~0xfffULL) /* segment table origin */
> #define ASCE_SUBSPACE 0x200 /* subspace group control */
> diff --git a/target/s390x/kvm/cpu_topology.c b/target/s390x/kvm/cpu_topology.c
> new file mode 100644
> index 0000000000..3831a3264c
> --- /dev/null
> +++ b/target/s390x/kvm/cpu_topology.c
> @@ -0,0 +1,136 @@
> +/*
> + * QEMU S390x CPU Topology
> + *
> + * Copyright IBM Corp. 2022
Happy new year?
> + * Author(s): Pierre Morel <pmorel@linux.ibm.com>
> + *
> + * This work is licensed under the terms of the GNU GPL, version 2 or (at
> + * your option) any later version. See the COPYING file in the top-level
> + * directory.
> + */
> +#include "qemu/osdep.h"
> +#include "cpu.h"
> +#include "hw/s390x/pv.h"
> +#include "hw/sysbus.h"
> +#include "hw/s390x/sclp.h"
> +#include "hw/s390x/cpu-topology.h"
> +
> +static char *fill_container(char *p, int level, int id)
> +{
> + SysIBTl_container *tle = (SysIBTl_container *)p;
> +
> + tle->nl = level;
> + tle->id = id;
> + return p + sizeof(*tle);
> +}
> +
> +static char *fill_tle_cpu(char *p, S390TopologyEntry *entry)
> +{
> + SysIBTl_cpu *tle = (SysIBTl_cpu *)p;
> + s390_topology_id topology_id = entry->id;
What about the reserved fields? Should they get set to 0 ?
> + tle->nl = 0;
> + tle->dedicated = topology_id.d;
> + tle->polarity = topology_id.p;
> + tle->type = topology_id.type;
> + tle->origin = topology_id.origin;
> + tle->mask = cpu_to_be64(entry->mask);
So here you're already taking care of swapping the endianess in case we ever
run this code with TCG, too ... so I think it would be great to also
eliminate the bitfield in SysIBTl_cpu to be really on the safe side.
> + return p + sizeof(*tle);
> +}
...
> +void insert_stsi_15_1_x(S390CPU *cpu, int sel2, __u64 addr, uint8_t ar)
> +{
> + union {
> + char place_holder[S390_TOPOLOGY_SYSIB_SIZE];
> + SysIB_151x sysib;
> + } buffer QEMU_ALIGNED(8) = {};
> + int len;
> +
> + if (!s390_has_topology() || sel2 < 2 || sel2 > SCLP_READ_SCP_INFO_MNEST) {
> + setcc(cpu, 3);
> + return;
> + }
> +
> + len = setup_stsi(cpu, &buffer.sysib, sel2);
> +
> + if (len > 4096) {
Maybe use TARGET_PAGE_SIZE instead of 4096 ?
> + setcc(cpu, 3);
> + return;
> + }
> +
> + buffer.sysib.length = cpu_to_be16(len);
> + s390_cpu_virt_mem_write(cpu, addr, ar, &buffer.sysib, len);
Is this supposed to work with protected guests, too? If so, I think you
likely need to use s390_cpu_pv_mem_write() for protected guests?
Thomas
^ permalink raw reply [flat|nested] 80+ messages in thread
* Re: [PATCH v14 04/11] s390x/sclp: reporting the maximum nested topology entries
2023-01-05 14:53 ` [PATCH v14 04/11] s390x/sclp: reporting the maximum nested topology entries Pierre Morel
@ 2023-01-11 8:57 ` Thomas Huth
2023-01-17 17:36 ` Pierre Morel
2023-01-11 17:52 ` Nina Schoetterl-Glausch
1 sibling, 1 reply; 80+ messages in thread
From: Thomas Huth @ 2023-01-11 8:57 UTC (permalink / raw)
To: Pierre Morel, qemu-s390x
Cc: qemu-devel, borntraeger, pasic, richard.henderson, david, cohuck,
mst, pbonzini, kvm, ehabkost, marcel.apfelbaum, eblake, armbru,
seiden, nrb, scgl, frankja, berrange, clg
On 05/01/2023 15.53, Pierre Morel wrote:
> The maximum nested topology entries is used by the guest to know
> how many nested topology are available on the machine.
>
> Currently, SCLP READ SCP INFO reports MNEST = 0, which is the
> equivalent of reporting the default value of 2.
> Let's use the default SCLP value of 2 and increase this value in the
> future patches implementing higher levels.
I'm confused ... so does a SCLP value of 2 mean a MNEST level of 4 ?
> Signed-off-by: Pierre Morel <pmorel@linux.ibm.com>
> ---
> include/hw/s390x/sclp.h | 5 +++--
> hw/s390x/sclp.c | 4 ++++
> 2 files changed, 7 insertions(+), 2 deletions(-)
>
> diff --git a/include/hw/s390x/sclp.h b/include/hw/s390x/sclp.h
> index 712fd68123..4ce852473c 100644
> --- a/include/hw/s390x/sclp.h
> +++ b/include/hw/s390x/sclp.h
> @@ -112,12 +112,13 @@ typedef struct CPUEntry {
> } QEMU_PACKED CPUEntry;
>
> #define SCLP_READ_SCP_INFO_FIXED_CPU_OFFSET 128
> -#define SCLP_READ_SCP_INFO_MNEST 2
> +#define SCLP_READ_SCP_INFO_MNEST 4
... since you update it to 4 here.
> typedef struct ReadInfo {
> SCCBHeader h;
> uint16_t rnmax;
> uint8_t rnsize;
> - uint8_t _reserved1[16 - 11]; /* 11-15 */
> + uint8_t _reserved1[15 - 11]; /* 11-14 */
> + uint8_t stsi_parm; /* 15-16 */
> uint16_t entries_cpu; /* 16-17 */
> uint16_t offset_cpu; /* 18-19 */
> uint8_t _reserved2[24 - 20]; /* 20-23 */
> diff --git a/hw/s390x/sclp.c b/hw/s390x/sclp.c
> index eff74479f4..07e3cb4cac 100644
> --- a/hw/s390x/sclp.c
> +++ b/hw/s390x/sclp.c
> @@ -20,6 +20,7 @@
> #include "hw/s390x/event-facility.h"
> #include "hw/s390x/s390-pci-bus.h"
> #include "hw/s390x/ipl.h"
> +#include "hw/s390x/cpu-topology.h"
>
> static inline SCLPDevice *get_sclp_device(void)
> {
> @@ -125,6 +126,9 @@ static void read_SCP_info(SCLPDevice *sclp, SCCB *sccb)
>
> /* CPU information */
> prepare_cpu_entries(machine, entries_start, &cpu_count);
> + if (s390_has_topology()) {
> + read_info->stsi_parm = SCLP_READ_SCP_INFO_MNEST;
This seems to be in contradiction to what you've said in the commit
description - you set it to 4 and not to 2.
Thomas
> + }
> read_info->entries_cpu = cpu_to_be16(cpu_count);
> read_info->offset_cpu = cpu_to_be16(offset_cpu);
> read_info->highest_cpu = cpu_to_be16(machine->smp.max_cpus - 1);
^ permalink raw reply [flat|nested] 80+ messages in thread
* Re: [PATCH v14 05/11] s390x/cpu topology: resetting the Topology-Change-Report
2023-01-05 14:53 ` [PATCH v14 05/11] s390x/cpu topology: resetting the Topology-Change-Report Pierre Morel
@ 2023-01-11 9:00 ` Thomas Huth
2023-01-17 17:57 ` Pierre Morel
0 siblings, 1 reply; 80+ messages in thread
From: Thomas Huth @ 2023-01-11 9:00 UTC (permalink / raw)
To: Pierre Morel, qemu-s390x
Cc: qemu-devel, borntraeger, pasic, richard.henderson, david, cohuck,
mst, pbonzini, kvm, ehabkost, marcel.apfelbaum, eblake, armbru,
seiden, nrb, scgl, frankja, berrange, clg
On 05/01/2023 15.53, Pierre Morel wrote:
> During a subsystem reset the Topology-Change-Report is cleared
> by the machine.
> Let's ask KVM to clear the Modified Topology Change Report (MTCR)
> bit of the SCA in the case of a subsystem reset.
>
> Signed-off-by: Pierre Morel <pmorel@linux.ibm.com>
> ---
...
> diff --git a/target/s390x/kvm/kvm_s390x.h b/target/s390x/kvm/kvm_s390x.h
> index f9785564d0..649dae5948 100644
> --- a/target/s390x/kvm/kvm_s390x.h
> +++ b/target/s390x/kvm/kvm_s390x.h
> @@ -47,5 +47,6 @@ void kvm_s390_crypto_reset(void);
> void kvm_s390_restart_interrupt(S390CPU *cpu);
> void kvm_s390_stop_interrupt(S390CPU *cpu);
> void kvm_s390_set_diag318(CPUState *cs, uint64_t diag318_info);
> +int kvm_s390_topology_set_mtcr(uint64_t attr);
>
> #endif /* KVM_S390X_H */
> diff --git a/hw/s390x/s390-virtio-ccw.c b/hw/s390x/s390-virtio-ccw.c
> index c98b93a15f..14798ca305 100644
> --- a/hw/s390x/s390-virtio-ccw.c
> +++ b/hw/s390x/s390-virtio-ccw.c
> @@ -122,6 +122,7 @@ static void subsystem_reset(void)
> device_cold_reset(dev);
> }
> }
> + s390_cpu_topology_reset();
> }
Would it make sense to add a "if (s390_has_topology())" check around the new
line?
Anyway:
Reviewed-by: Thomas Huth <thuth@redhat.com>
^ permalink raw reply [flat|nested] 80+ messages in thread
* Re: [PATCH v14 03/11] target/s390x/cpu topology: handle STSI(15) and build the SYSIB
2023-01-10 14:29 ` Thomas Huth
@ 2023-01-11 9:16 ` Thomas Huth
2023-01-11 17:14 ` Nina Schoetterl-Glausch
2023-01-17 16:56 ` Pierre Morel
2 siblings, 0 replies; 80+ messages in thread
From: Thomas Huth @ 2023-01-11 9:16 UTC (permalink / raw)
To: Pierre Morel, qemu-s390x, frankja
Cc: qemu-devel, borntraeger, pasic, richard.henderson, david, cohuck,
mst, pbonzini, kvm, ehabkost, marcel.apfelbaum, eblake, armbru,
seiden, nrb, scgl, berrange, clg
On 10/01/2023 15.29, Thomas Huth wrote:
> On 05/01/2023 15.53, Pierre Morel wrote:
>> On interception of STSI(15.1.x) the System Information Block
>> (SYSIB) is built from the list of pre-ordered topology entries.
>>
>> Signed-off-by: Pierre Morel <pmorel@linux.ibm.com>
>> ---
...
>> +void insert_stsi_15_1_x(S390CPU *cpu, int sel2, __u64 addr, uint8_t ar)
>> +{
>> + union {
>> + char place_holder[S390_TOPOLOGY_SYSIB_SIZE];
>> + SysIB_151x sysib;
>> + } buffer QEMU_ALIGNED(8) = {};
>> + int len;
>> +
>> + if (!s390_has_topology() || sel2 < 2 || sel2 >
>> SCLP_READ_SCP_INFO_MNEST) {
>> + setcc(cpu, 3);
>> + return;
>> + }
>> +
>> + len = setup_stsi(cpu, &buffer.sysib, sel2);
>> +
>> + if (len > 4096) {
>
> Maybe use TARGET_PAGE_SIZE instead of 4096 ?
>
>> + setcc(cpu, 3);
>> + return;
>> + }
>> +
>> + buffer.sysib.length = cpu_to_be16(len);
>> + s390_cpu_virt_mem_write(cpu, addr, ar, &buffer.sysib, len);
>
> Is this supposed to work with protected guests, too? If so, I think you
> likely need to use s390_cpu_pv_mem_write() for protected guests?
I now saw in a later patch that the topology feature gets disabled for
protected guests - so never mind, please ignore my question here.
Thomas
^ permalink raw reply [flat|nested] 80+ messages in thread
* Re: [PATCH v14 02/11] s390x/cpu topology: add topology entries on CPU hotplug
2023-01-10 13:00 ` Thomas Huth
@ 2023-01-11 9:23 ` Nina Schoetterl-Glausch
2023-01-16 18:24 ` Pierre Morel
1 sibling, 0 replies; 80+ messages in thread
From: Nina Schoetterl-Glausch @ 2023-01-11 9:23 UTC (permalink / raw)
To: Thomas Huth, Pierre Morel, qemu-s390x
Cc: qemu-devel, borntraeger, pasic, richard.henderson, david, cohuck,
mst, pbonzini, kvm, ehabkost, marcel.apfelbaum, eblake, armbru,
seiden, nrb, frankja, berrange, clg
On Tue, 2023-01-10 at 14:00 +0100, Thomas Huth wrote:
> On 05/01/2023 15.53, Pierre Morel wrote:
> > The topology information are attributes of the CPU and are
> > specified during the CPU device creation.
> >
> > On hot plug, we gather the topology information on the core,
> > creates a list of topology entries, each entry contains a single
> > core mask of each core with identical topology and finaly we
> > orders the list in topological order.
> > The topological order is, from higher to lower priority:
> > - physical topology
> > - drawer
> > - book
> > - socket
> > - core origin, offset in 64bit increment from core 0.
> > - modifier attributes
> > - CPU type
> > - polarization entitlement
> > - dedication
> >
> > The possibility to insert a CPU in a mask is dependent on the
> > number of cores allowed in a socket, a book or a drawer, the
> > checking is done during the hot plug of the CPU to have an
> > immediate answer.
> >
> > If the complete topology is not specified, the core is added
> > in the physical topology based on its core ID and it gets
> > defaults values for the modifier attributes.
> >
> > This way, starting QEMU without specifying the topology can
> > still get some adventage of the CPU topology.
>
> s/adventage/advantage/
>
> > Signed-off-by: Pierre Morel <pmorel@linux.ibm.com>
> > ---
> > include/hw/s390x/cpu-topology.h | 48 ++++++
> > hw/s390x/cpu-topology.c | 293 ++++++++++++++++++++++++++++++++
> > hw/s390x/s390-virtio-ccw.c | 10 ++
> > hw/s390x/meson.build | 1 +
> > 4 files changed, 352 insertions(+)
> > create mode 100644 hw/s390x/cpu-topology.c
> >
> > diff --git a/include/hw/s390x/cpu-topology.h b/include/hw/s390x/cpu-topology.h
> > index d945b57fc3..b3fd752d8d 100644
> > --- a/include/hw/s390x/cpu-topology.h
> > +++ b/include/hw/s390x/cpu-topology.h
> >
[...]
> > +typedef struct S390Topology {
> > + QTAILQ_HEAD(, S390TopologyEntry) list;
> > + uint8_t *sockets;
>
> So this "uint8_t" basically is a hidden limit of a maximum of 256 sockets
> that can be used for per book? Do we check that limit somewhere? (I looked
> for it, but I didn't spot such a check)
S390_MAX_CPUS < 256. Might be a good idea to have a build time assert for that.
And one cannot have more sockets that maxcpus.
>
> > + CpuTopology *smp;
> > +} S390Topology;
> > +
> > +#ifdef CONFIG_KVM
> > +bool s390_has_topology(void);
> > +void s390_topology_set_cpu(MachineState *ms, S390CPU *cpu, Error **errp);
> > +#else
> > +static inline bool s390_has_topology(void)
> > +{
> > + return false;
> > +}
> > +static inline void s390_topology_set_cpu(MachineState *ms,
> > + S390CPU *cpu,
> > + Error **errp) {}
> > +#endif
> > +extern S390Topology s390_topology;
> > +
> > #endif
> > diff --git a/hw/s390x/cpu-topology.c b/hw/s390x/cpu-topology.c
> > new file mode 100644
> > index 0000000000..438055c612
> > --- /dev/null
> > +++ b/hw/s390x/cpu-topology.c
> > @@ -0,0 +1,293 @@
> > +/*
> > + * CPU Topology
> > + *
> > + * Copyright IBM Corp. 2022
>
> Want to update to 2023 now?
It's the year of first publication, and I'd guess this is a derivative work of what
was published to the mailing list last year.
>
^ permalink raw reply [flat|nested] 80+ messages in thread
* Re: [PATCH v14 07/11] target/s390x/cpu topology: activating CPU topology
2023-01-05 14:53 ` [PATCH v14 07/11] target/s390x/cpu topology: activating CPU topology Pierre Morel
@ 2023-01-11 10:04 ` Thomas Huth
2023-01-18 10:01 ` Pierre Morel
0 siblings, 1 reply; 80+ messages in thread
From: Thomas Huth @ 2023-01-11 10:04 UTC (permalink / raw)
To: Pierre Morel, qemu-s390x
Cc: qemu-devel, borntraeger, pasic, richard.henderson, david, cohuck,
mst, pbonzini, kvm, ehabkost, marcel.apfelbaum, eblake, armbru,
seiden, nrb, scgl, frankja, berrange, clg
On 05/01/2023 15.53, Pierre Morel wrote:
> The KVM capability, KVM_CAP_S390_CPU_TOPOLOGY is used to
Remove the "," in above line?
> activate the S390_FEAT_CONFIGURATION_TOPOLOGY feature and
> the topology facility for the guest in the case the topology
I'd like to suggest to add "in the host CPU model" after "facility".
> is available in QEMU and in KVM.
>
> The feature is disabled by default and fenced for SE
> (secure execution).
>
> Signed-off-by: Pierre Morel <pmorel@linux.ibm.com>
> ---
> hw/s390x/cpu-topology.c | 2 +-
> target/s390x/cpu_models.c | 1 +
> target/s390x/kvm/kvm.c | 13 +++++++++++++
> 3 files changed, 15 insertions(+), 1 deletion(-)
>
> diff --git a/hw/s390x/cpu-topology.c b/hw/s390x/cpu-topology.c
> index e6b4692581..b69955a1cd 100644
> --- a/hw/s390x/cpu-topology.c
> +++ b/hw/s390x/cpu-topology.c
> @@ -52,7 +52,7 @@ static int s390_socket_nb(s390_topology_id id)
> */
> bool s390_has_topology(void)
> {
> - return false;
> + return s390_has_feat(S390_FEAT_CONFIGURATION_TOPOLOGY);
> }
>
> /**
> diff --git a/target/s390x/cpu_models.c b/target/s390x/cpu_models.c
> index c3a4f80633..3f05e05fd3 100644
> --- a/target/s390x/cpu_models.c
> +++ b/target/s390x/cpu_models.c
> @@ -253,6 +253,7 @@ bool s390_has_feat(S390Feat feat)
> case S390_FEAT_SIE_CMMA:
> case S390_FEAT_SIE_PFMFI:
> case S390_FEAT_SIE_IBS:
> + case S390_FEAT_CONFIGURATION_TOPOLOGY:
> return false;
> break;
> default:
> diff --git a/target/s390x/kvm/kvm.c b/target/s390x/kvm/kvm.c
> index fb63be41b7..4e2a2ff516 100644
> --- a/target/s390x/kvm/kvm.c
> +++ b/target/s390x/kvm/kvm.c
> @@ -2470,6 +2470,19 @@ void kvm_s390_get_host_cpu_model(S390CPUModel *model, Error **errp)
> set_bit(S390_FEAT_UNPACK, model->features);
> }
>
> + /*
> + * If we have support for CPU Topology prevent overrule
> + * S390_FEAT_CONFIGURATION_TOPOLOGY with S390_FEAT_DISABLE_CPU_TOPOLOGY
That S390_FEAT_DISABLE_CPU_TOPOLOGY looks like a leftover from v12 ?
Apart from that, patch looks fine to me now.
Thomas
> + * implemented in KVM, activate the CPU TOPOLOGY feature.
> + */
> + if (kvm_check_extension(kvm_state, KVM_CAP_S390_CPU_TOPOLOGY)) {
> + if (kvm_vm_enable_cap(kvm_state, KVM_CAP_S390_CPU_TOPOLOGY, 0) < 0) {
> + error_setg(errp, "KVM: Error enabling KVM_CAP_S390_CPU_TOPOLOGY");
> + return;
> + }
> + set_bit(S390_FEAT_CONFIGURATION_TOPOLOGY, model->features);
> + }
> +
> /* We emulate a zPCI bus and AEN, therefore we don't need HW support */
> set_bit(S390_FEAT_ZPCI, model->features);
> set_bit(S390_FEAT_ADAPTER_EVENT_NOTIFICATION, model->features);
^ permalink raw reply [flat|nested] 80+ messages in thread
* Re: [PATCH v14 08/11] qapi/s390/cpu topology: change-topology monitor command
2023-01-05 14:53 ` Pierre Morel
(?)
@ 2023-01-11 10:09 ` Thomas Huth
2023-01-12 8:00 ` Thomas Huth
2023-01-18 14:23 ` Pierre Morel
-1 siblings, 2 replies; 80+ messages in thread
From: Thomas Huth @ 2023-01-11 10:09 UTC (permalink / raw)
To: Pierre Morel, qemu-s390x
Cc: qemu-devel, borntraeger, pasic, richard.henderson, david, cohuck,
mst, pbonzini, kvm, ehabkost, marcel.apfelbaum, eblake, armbru,
seiden, nrb, scgl, frankja, berrange, clg
On 05/01/2023 15.53, Pierre Morel wrote:
> The modification of the CPU attributes are done through a monitor
> commands.
s/commands/command/
> It allows to move the core inside the topology tree to optimise
> the cache usage in the case the host's hypervizor previously
s/hypervizor/hypervisor/
> moved the CPU.
>
> The same command allows to modifiy the CPU attributes modifiers
s/modifiy/modify/
> like polarization entitlement and the dedicated attribute to notify
> the guest if the host admin modified scheduling or dedication of a vCPU.
>
> With this knowledge the guest has the possibility to optimize the
> usage of the vCPUs.
Hmm, who is supposed to call this QMP command in the future? Will there be a
new daemon monitoring the CPU changes in the host? Or will there be a
libvirt addition for this? ... Seems like I still miss the big picture here...
Thomas
^ permalink raw reply [flat|nested] 80+ messages in thread
* Re: [PATCH v14 03/11] target/s390x/cpu topology: handle STSI(15) and build the SYSIB
2023-01-10 14:29 ` Thomas Huth
2023-01-11 9:16 ` Thomas Huth
@ 2023-01-11 17:14 ` Nina Schoetterl-Glausch
2023-01-17 16:58 ` Pierre Morel
2023-01-17 16:56 ` Pierre Morel
2 siblings, 1 reply; 80+ messages in thread
From: Nina Schoetterl-Glausch @ 2023-01-11 17:14 UTC (permalink / raw)
To: Thomas Huth, Pierre Morel, qemu-s390x, frankja
Cc: qemu-devel, borntraeger, pasic, richard.henderson, david, cohuck,
mst, pbonzini, kvm, ehabkost, marcel.apfelbaum, eblake, armbru,
seiden, nrb, berrange, clg
On Tue, 2023-01-10 at 15:29 +0100, Thomas Huth wrote:
> On 05/01/2023 15.53, Pierre Morel wrote:
> > On interception of STSI(15.1.x) the System Information Block
> > (SYSIB) is built from the list of pre-ordered topology entries.
> >
> > Signed-off-by: Pierre Morel <pmorel@linux.ibm.com>
> > ---
>
[...]
> > +void insert_stsi_15_1_x(S390CPU *cpu, int sel2, __u64 addr, uint8_t ar)
> > +{
> > + union {
> > + char place_holder[S390_TOPOLOGY_SYSIB_SIZE];
> > + SysIB_151x sysib;
> > + } buffer QEMU_ALIGNED(8) = {};
> > + int len;
> > +
> > + if (!s390_has_topology() || sel2 < 2 || sel2 > SCLP_READ_SCP_INFO_MNEST) {
> > + setcc(cpu, 3);
> > + return;
> > + }
> > +
> > + len = setup_stsi(cpu, &buffer.sysib, sel2);
> > +
> > + if (len > 4096) {
>
> Maybe use TARGET_PAGE_SIZE instead of 4096 ?
sizeof(SysIB) would be preferable IMO.
>
> > + setcc(cpu, 3);
> > + return;
> > + }
^ permalink raw reply [flat|nested] 80+ messages in thread
* Re: [PATCH v14 04/11] s390x/sclp: reporting the maximum nested topology entries
2023-01-05 14:53 ` [PATCH v14 04/11] s390x/sclp: reporting the maximum nested topology entries Pierre Morel
2023-01-11 8:57 ` Thomas Huth
@ 2023-01-11 17:52 ` Nina Schoetterl-Glausch
2023-01-17 17:44 ` Pierre Morel
1 sibling, 1 reply; 80+ messages in thread
From: Nina Schoetterl-Glausch @ 2023-01-11 17:52 UTC (permalink / raw)
To: Pierre Morel, qemu-s390x
Cc: qemu-devel, borntraeger, pasic, richard.henderson, david, thuth,
cohuck, mst, pbonzini, kvm, ehabkost, marcel.apfelbaum, eblake,
armbru, seiden, nrb, frankja, berrange, clg
On Thu, 2023-01-05 at 15:53 +0100, Pierre Morel wrote:
> The maximum nested topology entries is used by the guest to know
> how many nested topology are available on the machine.
>
> Currently, SCLP READ SCP INFO reports MNEST = 0, which is the
> equivalent of reporting the default value of 2.
> Let's use the default SCLP value of 2 and increase this value in the
> future patches implementing higher levels.
>
> Signed-off-by: Pierre Morel <pmorel@linux.ibm.com>
Reviewed-by: Nina Schoetterl-Glausch <nsg@linux.ibm.com>
if you address the issues Thomas found with the commit description
and the nits below.
> ---
> include/hw/s390x/sclp.h | 5 +++--
> hw/s390x/sclp.c | 4 ++++
> 2 files changed, 7 insertions(+), 2 deletions(-)
>
> diff --git a/include/hw/s390x/sclp.h b/include/hw/s390x/sclp.h
> index 712fd68123..4ce852473c 100644
> --- a/include/hw/s390x/sclp.h
> +++ b/include/hw/s390x/sclp.h
> @@ -112,12 +112,13 @@ typedef struct CPUEntry {
> } QEMU_PACKED CPUEntry;
>
> #define SCLP_READ_SCP_INFO_FIXED_CPU_OFFSET 128
> -#define SCLP_READ_SCP_INFO_MNEST 2
> +#define SCLP_READ_SCP_INFO_MNEST 4
> typedef struct ReadInfo {
> SCCBHeader h;
> uint16_t rnmax;
> uint8_t rnsize;
> - uint8_t _reserved1[16 - 11]; /* 11-15 */
> + uint8_t _reserved1[15 - 11]; /* 11-14 */
> + uint8_t stsi_parm; /* 15-16 */
The numbering here is the same as the one in the arch doc, instead
of the maybe more usual one where the right number is exclusive.
So 15-16 looks like a two byte field, so just do 15 or just drop it.
> uint16_t entries_cpu; /* 16-17 */
> uint16_t offset_cpu; /* 18-19 */
> uint8_t _reserved2[24 - 20]; /* 20-23 */
> diff --git a/hw/s390x/sclp.c b/hw/s390x/sclp.c
> index eff74479f4..07e3cb4cac 100644
> --- a/hw/s390x/sclp.c
> +++ b/hw/s390x/sclp.c
> @@ -20,6 +20,7 @@
> #include "hw/s390x/event-facility.h"
> #include "hw/s390x/s390-pci-bus.h"
> #include "hw/s390x/ipl.h"
> +#include "hw/s390x/cpu-topology.h"
>
> static inline SCLPDevice *get_sclp_device(void)
> {
> @@ -125,6 +126,9 @@ static void read_SCP_info(SCLPDevice *sclp, SCCB *sccb)
>
> /* CPU information */
> prepare_cpu_entries(machine, entries_start, &cpu_count);
> + if (s390_has_topology()) {
> + read_info->stsi_parm = SCLP_READ_SCP_INFO_MNEST;
> + }
Maybe move that up a bit, not sure if it belongs under the CPU information section.
I'd leave prepare_cpu_entries and read_info->entries_cpu = adjacent in any case.
> read_info->entries_cpu = cpu_to_be16(cpu_count);
> read_info->offset_cpu = cpu_to_be16(offset_cpu);
> read_info->highest_cpu = cpu_to_be16(machine->smp.max_cpus - 1);
^ permalink raw reply [flat|nested] 80+ messages in thread
* Re: [PATCH v14 08/11] qapi/s390/cpu topology: change-topology monitor command
2023-01-11 10:09 ` Thomas Huth
@ 2023-01-12 8:00 ` Thomas Huth
2023-01-18 14:23 ` Pierre Morel
1 sibling, 0 replies; 80+ messages in thread
From: Thomas Huth @ 2023-01-12 8:00 UTC (permalink / raw)
To: Pierre Morel, qemu-s390x
Cc: qemu-devel, borntraeger, pasic, richard.henderson, david, cohuck,
mst, pbonzini, kvm, ehabkost, marcel.apfelbaum, eblake, armbru,
seiden, nrb, scgl, frankja, berrange, clg
On 11/01/2023 11.09, Thomas Huth wrote:
> On 05/01/2023 15.53, Pierre Morel wrote:
>> The modification of the CPU attributes are done through a monitor
>> commands.
>
> s/commands/command/
>
>> It allows to move the core inside the topology tree to optimise
>> the cache usage in the case the host's hypervizor previously
>
> s/hypervizor/hypervisor/
>
>> moved the CPU.
>>
>> The same command allows to modifiy the CPU attributes modifiers
>
> s/modifiy/modify/
>
>> like polarization entitlement and the dedicated attribute to notify
>> the guest if the host admin modified scheduling or dedication of a vCPU.
>>
>> With this knowledge the guest has the possibility to optimize the
>> usage of the vCPUs.
>
> Hmm, who is supposed to call this QMP command in the future? Will there be a
> new daemon monitoring the CPU changes in the host? Or will there be a
> libvirt addition for this? ... Seems like I still miss the big picture here...
Or if this is just about to provide an API for future experiments, I'd
rather suggest to introduce the new commands with a "x-" prefix to mark them
as experimental (so we would also not need to go through the deprecation
process in case they don't work out as expected in the future).
Thomas
^ permalink raw reply [flat|nested] 80+ messages in thread
* Re: [PATCH v14 11/11] docs/s390x/cpu topology: document s390x cpu topology
2023-01-05 14:53 ` [PATCH v14 11/11] docs/s390x/cpu topology: document s390x cpu topology Pierre Morel
@ 2023-01-12 11:46 ` Thomas Huth
2023-01-19 14:48 ` Pierre Morel
2023-01-12 11:58 ` Daniel P. Berrangé
1 sibling, 1 reply; 80+ messages in thread
From: Thomas Huth @ 2023-01-12 11:46 UTC (permalink / raw)
To: Pierre Morel, qemu-s390x
Cc: qemu-devel, borntraeger, pasic, richard.henderson, david, cohuck,
mst, pbonzini, kvm, ehabkost, marcel.apfelbaum, eblake, armbru,
seiden, nrb, scgl, frankja, berrange, clg
On 05/01/2023 15.53, Pierre Morel wrote:
> Add some basic examples for the definition of cpu topology
> in s390x.
>
> Signed-off-by: Pierre Morel <pmorel@linux.ibm.com>
> ---
> docs/system/s390x/cpu-topology.rst | 292 +++++++++++++++++++++++++++++
> docs/system/target-s390x.rst | 1 +
> 2 files changed, 293 insertions(+)
> create mode 100644 docs/system/s390x/cpu-topology.rst
>
> diff --git a/docs/system/s390x/cpu-topology.rst b/docs/system/s390x/cpu-topology.rst
> new file mode 100644
> index 0000000000..0020b70b50
> --- /dev/null
> +++ b/docs/system/s390x/cpu-topology.rst
> @@ -0,0 +1,292 @@
> +CPU Topology on s390x
> +=====================
> +
> +CPU Topology on S390x provides up to 5 levels of topology containers:
You sometimes write "Topology" with a capital T, sometimes lower case ...
I'd suggest to write it lower case consistently everywhere.
> +nodes, drawers, books, sockets and CPUs.
Hmm, so here you mention that "nodes" are usable on s390x, too? ... in
another spot below, you don't mention these anymore...
> +While the higher level containers, Containers Topology List Entries,
> +(Containers TLE) define a tree hierarchy, the lowest level of topology
> +definition, the CPU Topology List Entry (CPU TLE), provides the placement
> +of the CPUs inside the parent container.
> +
> +Currently QEMU CPU topology uses a single level of container: the sockets.
> +
> +For backward compatibility, threads can be declared on the ``-smp`` command
> +line. They will be seen as CPUs by the guest as long as multithreading
> +is not really supported by QEMU for S390.
Maybe mention that threads are not allowed with machine types >= 7.2 anymore?
> +Beside the topological tree, S390x provides 3 CPU attributes:
> +- CPU type
> +- polarity entitlement
> +- dedication
> +
> +Prerequisites
> +-------------
> +
> +To use CPU Topology a Linux QEMU/KVM machine providing the CPU Topology facility
> +(STFLE bit 11) is required.
> +
> +However, since this facility has been enabled by default in an early version
> +of QEMU, we use a capability, ``KVM_CAP_S390_CPU_TOPOLOGY``, to notify KVM
> +QEMU use of the CPU Topology.
Has it? I thought bit 11 was not enabled by default in the past?
> +Enabling CPU topology
> +---------------------
> +
> +Currently, CPU topology is only enabled in the host model.
add a "by default if support is available in the host kernel" at the end of
the sentence?
> +Enabling CPU topology in a CPU model is done by setting the CPU flag
> +``ctop`` to ``on`` like in:
> +
> +.. code-block:: bash
> +
> + -cpu gen16b,ctop=on
> +
> +Having the topology disabled by default allows migration between
> +old and new QEMU without adding new flags.
> +
> +Default topology usage
> +----------------------
> +
> +The CPU Topology, can be specified on the QEMU command line
> +with the ``-smp`` or the ``-device`` QEMU command arguments
> +without using any new attributes.
> +In this case, the topology will be calculated by simply adding
> +to the topology the cores based on the core-id starting with
> +core-0 at position 0 of socket-0, book-0, drawer-0 with default
... here you don't mention "nodes" anymore (which you still mentioned at the
beginning of the doc).
> +modifier attributes: horizontal polarity and no dedication.
> +
> +In the following machine we define 8 sockets with 4 cores each.
> +Note that S390 QEMU machines do not implement multithreading.
I'd use s390x instead of S390 to avoid confusion with 31-bit machines.
> +.. code-block:: bash
> +
> + $ qemu-system-s390x -m 2G \
> + -cpu gen16b,ctop=on \
> + -smp cpus=5,sockets=8,cores=4,maxcpus=32 \
> + -device host-s390x-cpu,core-id=14 \
> +
> +New CPUs can be plugged using the device_add hmp command like in:
> +
> +.. code-block:: bash
> +
> + (qemu) device_add gen16b-s390x-cpu,core-id=9
> +
> +The core-id defines the placement of the core in the topology by
> +starting with core 0 in socket 0 up to maxcpus.
> +
> +In the example above:
> +
> +* There are 5 CPUs provided to the guest with the ``-smp`` command line
> + They will take the core-ids 0,1,2,3,4
> + As we have 4 cores in a socket, we have 4 CPUs provided
> + to the guest in socket 0, with core-ids 0,1,2,3.
> + The last cpu, with core-id 4, will be on socket 1.
> +
> +* the core with ID 14 provided by the ``-device`` command line will
> + be placed in socket 3, with core-id 14
> +
> +* the core with ID 9 provided by the ``device_add`` qmp command will
> + be placed in socket 2, with core-id 9
> +
> +Note that the core ID is machine wide and the CPU TLE masks provided
> +by the STSI instruction will be written in a big endian mask:
> +
> +* in socket 0: 0xf000000000000000 (core id 0,1,2,3)
> +* in socket 1: 0x0800000000000000 (core id 4)
> +* in socket 2: 0x0040000000000000 (core id 9)
> +* in socket 3: 0x0002000000000000 (core id 14)
Hmm, who's supposed to be the audience of this documentation? Users?
Developers? For a doc in docs/system/ I'd expect this to be a documentation
for users, so this seems to be way too much of implementation detail here
already. If this is supposed to be a doc for developers instead, the file
should likely rather go into doc/devel/ instead. Or maybe you want both? ...
then you should split the information in here in two files, I think, one in
docs/system/ and one in docs/devel/ .
> +Defining the topology on command line
> +-------------------------------------
> +
> +The topology can be defined entirely during the CPU definition,
> +with the exception of CPU 0 which must be defined with the -smp
> +argument.
> +
> +For example, here we set the position of the cores 1,2,3 on
> +drawer 1, book 1, socket 2 and cores 0,9 and 14 on drawer 0,
> +book 0, socket 0 with all horizontal polarity and not dedicated.
> +The core 4, will be set on its default position on socket 1
> +(since we have 4 core per socket) and we define it with dedication and
> +vertical high entitlement.
> +
> +.. code-block:: bash
> +
> + $ qemu-system-s390x -m 2G \
> + -cpu gen16b,ctop=on \
> + -smp cpus=1,sockets=8,cores=4,maxcpus=32 \
> + \
> + -device gen16b-s390x-cpu,drawer-id=1,book-id=1,socket-id=2,core-id=1 \
> + -device gen16b-s390x-cpu,drawer-id=1,book-id=1,socket-id=2,core-id=2 \
> + -device gen16b-s390x-cpu,drawer-id=1,book-id=1,socket-id=2,core-id=3 \
> + \
> + -device gen16b-s390x-cpu,drawer-id=0,book-id=0,socket-id=0,core-id=9 \
> + -device gen16b-s390x-cpu,drawer-id=0,book-id=0,socket-id=0,core-id=14 \
> + \
> + -device gen16b-s390x-cpu,core-id=4,dedicated=on,polarity=3 \
> +
> +Polarity and dedication
> +-----------------------
Since you are using the terms "polarity" and "dedication" in the previous
paragraphs already, it might make sense to move this section here earlier in
the document to teach the users about this first, before using the terms in
the other paragraphs?
> +Polarity can be of two types: horizontal or vertical.
> +
> +The horizontal polarization specifies that all guest's vCPUs get
> +almost the same amount of provisioning of real CPU by the host.
> +
> +The vertical polarization specifies that guest's vCPU can get
> +different real CPU provisions:
Please remove one space between "different" and "real".
> +- a vCPU with Vertical high entitlement specifies that this
> + vCPU gets 100% of the real CPU provisioning.
> +
> +- a vCPU with Vertical medium entitlement specifies that this
> + vCPU shares the real CPU with other vCPU.
"with *one* other vCPU" or rather "with other vCPU*s*" ?
> +
> +- a vCPU with Vertical low entitlement specifies that this
> + vCPU only get real CPU provisioning when no other vCPU need it.
> +
> +In the case a vCPU with vertical high entitlement does not use
> +the real CPU, the unused "slack" can be dispatched to other vCPU
> +with medium or low entitlement.
> +
> +The host indicates to the guest how the real CPU resources are
> +provided to the vCPUs through the SYSIB with two polarity bits
> +inside the CPU TLE.
> +
> +Bits d - Polarization
> +0 0 Horizontal
> +0 1 Vertical low entitlement
> +1 0 Vertical medium entitlement
> +1 1 Vertical high entitlement
That SYSIB stuff looks like details for developers again ... I think you
should either add more explanations here (I assume the average user does not
know the term SYSIB), move it to a separate developers file or drop it.
> +A subsystem reset puts all vCPU of the configuration into the
> +horizontal polarization.
> +
> +The admin specifies the dedicated bit when the vCPU is dedicated
> +to a single real CPU.
> +
> +As for the Linux admin, the dedicated bit is an indication on the
> +affinity of a vCPU for a real CPU while the entitlement indicates the
> +sharing or exclusivity of use.
> +
> +QAPI interface for topology
> +---------------------------
A "grep -r QAPI docs/system/" shows hardly any entries there. I think QAPI
documentation should go into docs/devel instead.
> +Let's start QEMU with the following command:
> +
> +.. code-block:: bash
> +
> + sudo /usr/local/bin/qemu-system-s390x \
> + -enable-kvm \
> + -cpu z14,ctop=on \
> + -smp 1,drawers=3,books=3,sockets=2,cores=2,maxcpus=36 \
> + \
> + -device z14-s390x-cpu,core-id=19,polarity=3 \
> + -device z14-s390x-cpu,core-id=11,polarity=1 \
> + -device z14-s390x-cpu,core-id=12,polarity=3 \
> + ...
> +
> +and see the result when using of the QAPI interface.
> +
> +query-topology
> ++++++++++++++++
> +
> +The command cpu-topology allows the admin to query the topology
Not sure if the average admin runs QMP directly ... maybe rather talk about
the "upper layers like libvirt" here or something similar.
> +tree and modifier for all configured vCPU.
> +
> +.. code-block:: QMP
> +
> + -> { "execute": "query-topology" }
> + {"return":
> + [
> + {
> + "origin": 0,
> + "dedicated": false,
> + "book": 0,
> + "socket": 0,
> + "drawer": 0,
> + "polarity": 0,
> + "mask": "0x8000000000000000"
> + },
> + {
> + "origin": 0,
> + "dedicated": false,
> + "book": 2,
> + "socket": 1,
> + "drawer": 0,
> + "polarity": 1,
> + "mask": "0x0010000000000000"
> + },
> + {
> + "origin": 0,
> + "dedicated": false,
> + "book": 0,
> + "socket": 0,
> + "drawer": 1,
> + "polarity": 3,
> + "mask": "0x0008000000000000"
> + },
> + {
> + "origin": 0,
> + "dedicated": false,
> + "book": 1,
> + "socket": 1,
> + "drawer": 1,
> + "polarity": 3,
> + "mask": "0x0000100000000000"
> + }
> + ]
> + }
> +
> +change-topology
> ++++++++++++++++
> +
> +The command change-topology allows the admin to modify the topology
> +tree or the topology modifiers of a vCPU in the configuration.
> +
> +.. code-block:: QMP
> +
> + -> { "execute": "change-topology",
> + "arguments": {
> + "core": 11,
> + "socket": 0,
> + "book": 0,
> + "drawer": 0,
> + "polarity": 0,
> + "dedicated": false
> + }
> + }
> + <- {"return": {}}
> +
> +
> +event POLARITY_CHANGE
> ++++++++++++++++++++++
> +
> +When a guest is requesting a modification of the polarity,
> +QEMU sends a POLARITY_CHANGE event.
> +
> +When requesting the change, the guest only specifies horizontal or
> +vertical polarity.
> +The dedication and fine grain vertical entitlement depends on admin
> +to set according to its response to this event.
> +
> +Note that a vertical polarized dedicated vCPU can only have a high
> +entitlement, this gives 6 possibilities for a vCPU polarity:
> +
> +- Horizontal
> +- Horizontal dedicated
> +- Vertical low
> +- Vertical medium
> +- Vertical high
> +- Vertical high dedicated
> +
> +Example of the event received when the guest issues PTF(0) to request
Please mention that PTF is a CPU instruction (and provide the full name).
Thomas
^ permalink raw reply [flat|nested] 80+ messages in thread
* Re: [PATCH v14 09/11] qapi/s390/cpu topology: monitor query topology information
2023-01-05 14:53 ` [PATCH v14 09/11] qapi/s390/cpu topology: monitor query topology information Pierre Morel
@ 2023-01-12 11:48 ` Thomas Huth
2023-01-18 15:59 ` Pierre Morel
2023-01-12 12:10 ` Daniel P. Berrangé
1 sibling, 1 reply; 80+ messages in thread
From: Thomas Huth @ 2023-01-12 11:48 UTC (permalink / raw)
To: Pierre Morel, qemu-s390x
Cc: qemu-devel, borntraeger, pasic, richard.henderson, david, cohuck,
mst, pbonzini, kvm, ehabkost, marcel.apfelbaum, eblake, armbru,
seiden, nrb, scgl, frankja, berrange, clg
On 05/01/2023 15.53, Pierre Morel wrote:
> Reporting the current topology informations to the admin through
> the QEMU monitor.
>
> Signed-off-by: Pierre Morel <pmorel@linux.ibm.com>
> ---
...
> diff --git a/hmp-commands-info.hx b/hmp-commands-info.hx
> index 754b1e8408..5730a47f71 100644
> --- a/hmp-commands-info.hx
> +++ b/hmp-commands-info.hx
> @@ -993,3 +993,19 @@ SRST
> ``info virtio-queue-element`` *path* *queue* [*index*]
> Display element of a given virtio queue
> ERST
> +
> +#if defined(TARGET_S390X) && defined(CONFIG_KVM)
> + {
> + .name = "query-topology",
> + .args_type = "",
> + .params = "",
> + .help = "Show information about CPU topology",
> + .cmd = hmp_query_topology,
> + .flags = "p",
> + },
> +
> +SRST
> + ``info query-topology``
"info query-topology" sounds weird ... I'd maybe rather call it only "info
topology" or "info cpu-topology" here.
Thomas
> + Show information about CPU topology
> +ERST
> +#endif
^ permalink raw reply [flat|nested] 80+ messages in thread
* Re: [PATCH v14 10/11] qapi/s390/cpu topology: POLARITY_CHANGE qapi event
2023-01-05 14:53 ` [PATCH v14 10/11] qapi/s390/cpu topology: POLARITY_CHANGE qapi event Pierre Morel
@ 2023-01-12 11:52 ` Thomas Huth
2023-01-18 17:09 ` Pierre Morel
0 siblings, 1 reply; 80+ messages in thread
From: Thomas Huth @ 2023-01-12 11:52 UTC (permalink / raw)
To: Pierre Morel, qemu-s390x
Cc: qemu-devel, borntraeger, pasic, richard.henderson, david, cohuck,
mst, pbonzini, kvm, ehabkost, marcel.apfelbaum, eblake, armbru,
seiden, nrb, scgl, frankja, berrange, clg
On 05/01/2023 15.53, Pierre Morel wrote:
> When the guest asks to change the polarity this change
> is forwarded to the admin using QAPI.
> The admin is supposed to take according decisions concerning
> CPU provisioning.
I somehow doubt that an average admin will monitor QEMU for such events ...
so this rather should be handled by upper layers like libvirt one day?
> Signed-off-by: Pierre Morel <pmorel@linux.ibm.com>
> ---
> qapi/machine-target.json | 21 +++++++++++++++++++++
> hw/s390x/cpu-topology.c | 2 ++
> 2 files changed, 23 insertions(+)
>
> diff --git a/qapi/machine-target.json b/qapi/machine-target.json
> index 927618a78f..10235cfb45 100644
> --- a/qapi/machine-target.json
> +++ b/qapi/machine-target.json
> @@ -437,3 +437,24 @@
> 'returns': ['S390CpuTopology'],
> 'if': { 'all': [ 'TARGET_S390X', 'CONFIG_KVM' ] }
> }
> +
> +##
> +# @POLARITY_CHANGE:
I'd maybe rather call it CPU_POLARITY_CHANGE ... in case "polarity" is one
day also used for some other devices.
> +#
> +# Emitted when the guest asks to change the polarity.
> +#
> +# @polarity: polarity specified by the guest
Please elaborate: Where does the value come from (the PTF instruction)?
Which values are possible?
Thomas
> +#
> +# Since: 8.0
> +#
> +# Example:
> +#
> +# <- { "event": "POLARITY_CHANGE",
> +# "data": { "polarity": 0 },
> +# "timestamp": { "seconds": 1401385907, "microseconds": 422329 } }
> +#
> +##
> +{ 'event': 'POLARITY_CHANGE',
> + 'data': { 'polarity': 'int' },
> + 'if': { 'all': [ 'TARGET_S390X', 'CONFIG_KVM'] }
> +}
> diff --git a/hw/s390x/cpu-topology.c b/hw/s390x/cpu-topology.c
> index c3748654ff..45621387d5 100644
> --- a/hw/s390x/cpu-topology.c
> +++ b/hw/s390x/cpu-topology.c
> @@ -19,6 +19,7 @@
> #include "hw/s390x/s390-virtio-ccw.h"
> #include "hw/s390x/cpu-topology.h"
> #include "qapi/qapi-commands-machine-target.h"
> +#include "qapi/qapi-events-machine-target.h"
> #include "qapi/qmp/qdict.h"
> #include "monitor/hmp.h"
> #include "monitor/monitor.h"
> @@ -128,6 +129,7 @@ void s390_topology_set_polarity(int polarity)
> }
> }
> s390_cpu_topology_set();
> + qapi_event_send_polarity_change(polarity);
> }
>
> /*
^ permalink raw reply [flat|nested] 80+ messages in thread
* Re: [PATCH v14 11/11] docs/s390x/cpu topology: document s390x cpu topology
2023-01-05 14:53 ` [PATCH v14 11/11] docs/s390x/cpu topology: document s390x cpu topology Pierre Morel
2023-01-12 11:46 ` Thomas Huth
@ 2023-01-12 11:58 ` Daniel P. Berrangé
2023-01-18 17:10 ` Pierre Morel
1 sibling, 1 reply; 80+ messages in thread
From: Daniel P. Berrangé @ 2023-01-12 11:58 UTC (permalink / raw)
To: Pierre Morel
Cc: qemu-s390x, qemu-devel, borntraeger, pasic, richard.henderson,
david, thuth, cohuck, mst, pbonzini, kvm, ehabkost,
marcel.apfelbaum, eblake, armbru, seiden, nrb, scgl, frankja,
clg
On Thu, Jan 05, 2023 at 03:53:13PM +0100, Pierre Morel wrote:
> Add some basic examples for the definition of cpu topology
> in s390x.
>
> Signed-off-by: Pierre Morel <pmorel@linux.ibm.com>
> ---
> docs/system/s390x/cpu-topology.rst | 292 +++++++++++++++++++++++++++++
> docs/system/target-s390x.rst | 1 +
> 2 files changed, 293 insertions(+)
> create mode 100644 docs/system/s390x/cpu-topology.rst
>
> diff --git a/docs/system/s390x/cpu-topology.rst b/docs/system/s390x/cpu-topology.rst
> new file mode 100644
> index 0000000000..0020b70b50
> --- /dev/null
> +++ b/docs/system/s390x/cpu-topology.rst
> @@ -0,0 +1,292 @@
> +CPU Topology on s390x
> +=====================
> +
> +CPU Topology on S390x provides up to 5 levels of topology containers:
> +nodes, drawers, books, sockets and CPUs.
The last level should be 'cores' not CPUs for QEMU terminology.
> +While the higher level containers, Containers Topology List Entries,
> +(Containers TLE) define a tree hierarchy, the lowest level of topology
> +definition, the CPU Topology List Entry (CPU TLE), provides the placement
> +of the CPUs inside the parent container.
With regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
^ permalink raw reply [flat|nested] 80+ messages in thread
* Re: [PATCH v14 08/11] qapi/s390/cpu topology: change-topology monitor command
2023-01-05 14:53 ` Pierre Morel
(?)
(?)
@ 2023-01-12 12:03 ` Daniel P. Berrangé
2023-01-18 13:17 ` Pierre Morel
-1 siblings, 1 reply; 80+ messages in thread
From: Daniel P. Berrangé @ 2023-01-12 12:03 UTC (permalink / raw)
To: Pierre Morel
Cc: qemu-s390x, qemu-devel, borntraeger, pasic, richard.henderson,
david, thuth, cohuck, mst, pbonzini, kvm, ehabkost,
marcel.apfelbaum, eblake, armbru, seiden, nrb, scgl, frankja,
clg
On Thu, Jan 05, 2023 at 03:53:10PM +0100, Pierre Morel wrote:
> The modification of the CPU attributes are done through a monitor
> commands.
>
> It allows to move the core inside the topology tree to optimise
> the cache usage in the case the host's hypervizor previously
> moved the CPU.
>
> The same command allows to modifiy the CPU attributes modifiers
> like polarization entitlement and the dedicated attribute to notify
> the guest if the host admin modified scheduling or dedication of a vCPU.
>
> With this knowledge the guest has the possibility to optimize the
> usage of the vCPUs.
>
> Signed-off-by: Pierre Morel <pmorel@linux.ibm.com>
> ---
> qapi/machine-target.json | 29 ++++++++
> include/monitor/hmp.h | 1 +
> hw/s390x/cpu-topology.c | 141 +++++++++++++++++++++++++++++++++++++++
> hmp-commands.hx | 16 +++++
> 4 files changed, 187 insertions(+)
>
> diff --git a/qapi/machine-target.json b/qapi/machine-target.json
> index 2e267fa458..75b0aa254d 100644
> --- a/qapi/machine-target.json
> +++ b/qapi/machine-target.json
> @@ -342,3 +342,32 @@
> 'TARGET_S390X',
> 'TARGET_MIPS',
> 'TARGET_LOONGARCH64' ] } }
> +
> +##
> +# @change-topology:
> +#
> +# @core: the vCPU ID to be moved
> +# @socket: the destination socket where to move the vCPU
> +# @book: the destination book where to move the vCPU
> +# @drawer: the destination drawer where to move the vCPU
This movement can be done while the guest OS is running ?
What happens to guest OS apps ? Every I know will read
topology once and assume it never changes at runtime.
What's the use case for wanting to re-arrange topology in
this manner ? It feels like its going to be a recipe for
hard to diagnose problems, as much code in libvirt and apps
above will assuming the vCPU IDs are assigned sequentially
starting from node=0,book=0,drawer=0,socket=0,core=0,
incrementing core, then incrementing socket, then
incrementing drawer, etc.
> +# @polarity: optional polarity, default is last polarity set by the guest
> +# @dedicated: optional, if the vCPU is dedicated to a real CPU
> +#
> +# Modifies the topology by moving the CPU inside the topology
> +# tree or by changing a modifier attribute of a CPU.
> +#
> +# Returns: Nothing on success, the reason on failure.
> +#
> +# Since: <next qemu stable release, eg. 1.0>
> +##
> +{ 'command': 'change-topology',
'set-cpu-topology'
> + 'data': {
> + 'core': 'int',
> + 'socket': 'int',
> + 'book': 'int',
> + 'drawer': 'int',
> + '*polarity': 'int',
> + '*dedicated': 'bool'
> + },
> + 'if': { 'all': [ 'TARGET_S390X', 'CONFIG_KVM' ] }
> +}
With regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
^ permalink raw reply [flat|nested] 80+ messages in thread
* Re: [PATCH v14 09/11] qapi/s390/cpu topology: monitor query topology information
2023-01-05 14:53 ` [PATCH v14 09/11] qapi/s390/cpu topology: monitor query topology information Pierre Morel
2023-01-12 11:48 ` Thomas Huth
@ 2023-01-12 12:10 ` Daniel P. Berrangé
2023-01-12 17:27 ` Nina Schoetterl-Glausch
2023-01-18 15:58 ` Pierre Morel
1 sibling, 2 replies; 80+ messages in thread
From: Daniel P. Berrangé @ 2023-01-12 12:10 UTC (permalink / raw)
To: Pierre Morel
Cc: qemu-s390x, qemu-devel, borntraeger, pasic, richard.henderson,
david, thuth, cohuck, mst, pbonzini, kvm, ehabkost,
marcel.apfelbaum, eblake, armbru, seiden, nrb, scgl, frankja,
clg
On Thu, Jan 05, 2023 at 03:53:11PM +0100, Pierre Morel wrote:
> Reporting the current topology informations to the admin through
> the QEMU monitor.
>
> Signed-off-by: Pierre Morel <pmorel@linux.ibm.com>
> ---
> qapi/machine-target.json | 66 ++++++++++++++++++++++++++++++++++
> include/monitor/hmp.h | 1 +
> hw/s390x/cpu-topology.c | 76 ++++++++++++++++++++++++++++++++++++++++
> hmp-commands-info.hx | 16 +++++++++
> 4 files changed, 159 insertions(+)
>
> diff --git a/qapi/machine-target.json b/qapi/machine-target.json
> index 75b0aa254d..927618a78f 100644
> --- a/qapi/machine-target.json
> +++ b/qapi/machine-target.json
> @@ -371,3 +371,69 @@
> },
> 'if': { 'all': [ 'TARGET_S390X', 'CONFIG_KVM' ] }
> }
> +
> +##
> +# @S390CpuTopology:
> +#
> +# CPU Topology information
> +#
> +# @drawer: the destination drawer where to move the vCPU
> +#
> +# @book: the destination book where to move the vCPU
> +#
> +# @socket: the destination socket where to move the vCPU
> +#
> +# @polarity: optional polarity, default is last polarity set by the guest
> +#
> +# @dedicated: optional, if the vCPU is dedicated to a real CPU
> +#
> +# @origin: offset of the first bit of the core mask
> +#
> +# @mask: mask of the cores sharing the same topology
> +#
> +# Since: 8.0
> +##
> +{ 'struct': 'S390CpuTopology',
> + 'data': {
> + 'drawer': 'int',
> + 'book': 'int',
> + 'socket': 'int',
> + 'polarity': 'int',
> + 'dedicated': 'bool',
> + 'origin': 'int',
> + 'mask': 'str'
> + },
> + 'if': { 'all': [ 'TARGET_S390X', 'CONFIG_KVM' ] }
> +}
> +
> +##
> +# @query-topology:
> +#
> +# Return information about CPU Topology
> +#
> +# Returns a @CpuTopology instance describing the CPU Toplogy
> +# being currently used by QEMU.
> +#
> +# Since: 8.0
> +#
> +# Example:
> +#
> +# -> { "execute": "cpu-topology" }
> +# <- {"return": [
> +# {
> +# "drawer": 0,
> +# "book": 0,
> +# "socket": 0,
> +# "polarity": 0,
> +# "dedicated": true,
> +# "origin": 0,
> +# "mask": 0xc000000000000000,
> +# },
> +# ]
> +# }
> +#
> +##
> +{ 'command': 'query-topology',
> + 'returns': ['S390CpuTopology'],
> + 'if': { 'all': [ 'TARGET_S390X', 'CONFIG_KVM' ] }
> +}
IIUC, you're using @mask as a way to compress the array returned
from query-topology, so that it doesn't have any repeated elements
with the same data. I guess I can understand that desire when the
core count can get very large, this can have a large saving.
The downside of using @mask, is that now you require the caller
to parse the string to turn it into a bitmask and expand the
data. Generally this is considered a bit of an anti-pattern in
QAPI design - we don't want callers to have to further parse
the data to extract information, we want to directly consumable
from the parsed JSON doc.
We already have 'query-cpus-fast' wich returns one entry for
each CPU. In fact why do we need to add query-topology at all.
Can't we just add book-id / drawer-id / polarity / dedicated
to the query-cpus-fast result ?
With regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
^ permalink raw reply [flat|nested] 80+ messages in thread
* Re: [PATCH v14 09/11] qapi/s390/cpu topology: monitor query topology information
2023-01-12 12:10 ` Daniel P. Berrangé
@ 2023-01-12 17:27 ` Nina Schoetterl-Glausch
2023-01-12 17:30 ` Daniel P. Berrangé
2023-01-18 15:58 ` Pierre Morel
1 sibling, 1 reply; 80+ messages in thread
From: Nina Schoetterl-Glausch @ 2023-01-12 17:27 UTC (permalink / raw)
To: Daniel P. Berrangé, Pierre Morel
Cc: qemu-s390x, qemu-devel, borntraeger, pasic, richard.henderson,
david, thuth, cohuck, mst, pbonzini, kvm, ehabkost,
marcel.apfelbaum, eblake, armbru, seiden, nrb, frankja, clg
On Thu, 2023-01-12 at 12:10 +0000, Daniel P. Berrangé wrote
[...]
>
> We already have 'query-cpus-fast' wich returns one entry for
> each CPU. In fact why do we need to add query-topology at all.
> Can't we just add book-id / drawer-id / polarity / dedicated
> to the query-cpus-fast result ?
Is there an existing command for setting cpu properties, also?
>
> With regards,
> Daniel
^ permalink raw reply [flat|nested] 80+ messages in thread
* Re: [PATCH v14 09/11] qapi/s390/cpu topology: monitor query topology information
2023-01-12 17:27 ` Nina Schoetterl-Glausch
@ 2023-01-12 17:30 ` Daniel P. Berrangé
0 siblings, 0 replies; 80+ messages in thread
From: Daniel P. Berrangé @ 2023-01-12 17:30 UTC (permalink / raw)
To: Nina Schoetterl-Glausch
Cc: Pierre Morel, qemu-s390x, qemu-devel, borntraeger, pasic,
richard.henderson, david, thuth, cohuck, mst, pbonzini, kvm,
ehabkost, marcel.apfelbaum, eblake, armbru, seiden, nrb, frankja,
clg
On Thu, Jan 12, 2023 at 06:27:04PM +0100, Nina Schoetterl-Glausch wrote:
> On Thu, 2023-01-12 at 12:10 +0000, Daniel P. Berrangé wrote
>
> [...]
> >
> > We already have 'query-cpus-fast' wich returns one entry for
> > each CPU. In fact why do we need to add query-topology at all.
> > Can't we just add book-id / drawer-id / polarity / dedicated
> > to the query-cpus-fast result ?
>
> Is there an existing command for setting cpu properties, also?
No, there's nothing applicable that I recall for runtime property
changes on CPUs.
With regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
^ permalink raw reply [flat|nested] 80+ messages in thread
* Re: [PATCH v14 01/11] s390x/cpu topology: adding s390 specificities to CPU topology
2023-01-05 14:53 ` [PATCH v14 01/11] s390x/cpu topology: adding s390 specificities to CPU topology Pierre Morel
2023-01-10 11:37 ` Thomas Huth
@ 2023-01-13 16:58 ` Nina Schoetterl-Glausch
2023-01-16 17:28 ` Pierre Morel
1 sibling, 1 reply; 80+ messages in thread
From: Nina Schoetterl-Glausch @ 2023-01-13 16:58 UTC (permalink / raw)
To: Pierre Morel, qemu-s390x
Cc: qemu-devel, borntraeger, pasic, richard.henderson, david, thuth,
cohuck, mst, pbonzini, kvm, ehabkost, marcel.apfelbaum, eblake,
armbru, seiden, nrb, frankja, berrange, clg
On Thu, 2023-01-05 at 15:53 +0100, Pierre Morel wrote:
> S390 adds two new SMP levels, drawers and books to the CPU
> topology.
> The S390 CPU have specific toplogy features like dedication
> and polarity to give to the guest indications on the host
> vCPUs scheduling and help the guest take the best decisions
> on the scheduling of threads on the vCPUs.
>
> Let us provide the SMP properties with books and drawers levels
> and S390 CPU with dedication and polarity,
>
> Signed-off-by: Pierre Morel <pmorel@linux.ibm.com>
> ---
> qapi/machine.json | 14 ++++++++--
> include/hw/boards.h | 10 ++++++-
> include/hw/s390x/cpu-topology.h | 23 ++++++++++++++++
> target/s390x/cpu.h | 6 +++++
> hw/core/machine-smp.c | 48 ++++++++++++++++++++++++++++-----
> hw/core/machine.c | 4 +++
> hw/s390x/s390-virtio-ccw.c | 2 ++
> softmmu/vl.c | 6 +++++
> target/s390x/cpu.c | 10 +++++++
> qemu-options.hx | 6 +++--
> 10 files changed, 117 insertions(+), 12 deletions(-)
> create mode 100644 include/hw/s390x/cpu-topology.h
>
[...]
> diff --git a/target/s390x/cpu.h b/target/s390x/cpu.h
> index 7d6d01325b..39ea63a416 100644
> --- a/target/s390x/cpu.h
> +++ b/target/s390x/cpu.h
> @@ -131,6 +131,12 @@ struct CPUArchState {
>
> #if !defined(CONFIG_USER_ONLY)
> uint32_t core_id; /* PoP "CPU address", same as cpu_index */
> + int32_t socket_id;
> + int32_t book_id;
> + int32_t drawer_id;
> + int32_t dedicated;
> + int32_t polarity;
If I understood the architecture correctly, the polarity is a property of the configuration,
not the cpus. So this should be vertical_entitlement, and there should be a machine (?) property
specifying if the polarity is horizontal or vertical.
> + int32_t cpu_type;
> uint64_t cpuid;
> #endif
>
[...]
^ permalink raw reply [flat|nested] 80+ messages in thread
* Re: [PATCH v14 02/11] s390x/cpu topology: add topology entries on CPU hotplug
2023-01-05 14:53 ` [PATCH v14 02/11] s390x/cpu topology: add topology entries on CPU hotplug Pierre Morel
2023-01-10 13:00 ` Thomas Huth
@ 2023-01-13 18:15 ` Nina Schoetterl-Glausch
2023-01-17 13:55 ` Pierre Morel
1 sibling, 1 reply; 80+ messages in thread
From: Nina Schoetterl-Glausch @ 2023-01-13 18:15 UTC (permalink / raw)
To: Pierre Morel, qemu-s390x
Cc: qemu-devel, borntraeger, pasic, richard.henderson, david, thuth,
cohuck, mst, pbonzini, kvm, ehabkost, marcel.apfelbaum, eblake,
armbru, seiden, nrb, frankja, berrange, clg
On Thu, 2023-01-05 at 15:53 +0100, Pierre Morel wrote:
> The topology information are attributes of the CPU and are
> specified during the CPU device creation.
>
> On hot plug, we gather the topology information on the core,
> creates a list of topology entries, each entry contains a single
> core mask of each core with identical topology and finaly we
s/finaly/finally/
> orders the list in topological order.
s/orders/order/
> The topological order is, from higher to lower priority:
> - physical topology
> - drawer
> - book
> - socket
> - core origin, offset in 64bit increment from core 0.
> - modifier attributes
> - CPU type
> - polarization entitlement
> - dedication
>
> The possibility to insert a CPU in a mask is dependent on the
> number of cores allowed in a socket, a book or a drawer, the
> checking is done during the hot plug of the CPU to have an
> immediate answer.
>
> If the complete topology is not specified, the core is added
> in the physical topology based on its core ID and it gets
> defaults values for the modifier attributes.
>
> This way, starting QEMU without specifying the topology can
> still get some adventage of the CPU topology.
>
> Signed-off-by: Pierre Morel <pmorel@linux.ibm.com>
> ---
> include/hw/s390x/cpu-topology.h | 48 ++++++
> hw/s390x/cpu-topology.c | 293 ++++++++++++++++++++++++++++++++
> hw/s390x/s390-virtio-ccw.c | 10 ++
> hw/s390x/meson.build | 1 +
> 4 files changed, 352 insertions(+)
> create mode 100644 hw/s390x/cpu-topology.c
>
> diff --git a/include/hw/s390x/cpu-topology.h b/include/hw/s390x/cpu-topology.h
> index d945b57fc3..b3fd752d8d 100644
> --- a/include/hw/s390x/cpu-topology.h
> +++ b/include/hw/s390x/cpu-topology.h
> @@ -10,7 +10,11 @@
> #ifndef HW_S390X_CPU_TOPOLOGY_H
> #define HW_S390X_CPU_TOPOLOGY_H
>
> +#include "qemu/queue.h"
> +#include "hw/boards.h"
> +
> #define S390_TOPOLOGY_CPU_IFL 0x03
> +#define S390_TOPOLOGY_MAX_ORIGIN ((63 + S390_MAX_CPUS) / 64)
>
> #define S390_TOPOLOGY_POLARITY_HORIZONTAL 0x00
> #define S390_TOPOLOGY_POLARITY_VERTICAL_LOW 0x01
> @@ -20,4 +24,48 @@
> #define S390_TOPOLOGY_SHARED 0x00
> #define S390_TOPOLOGY_DEDICATED 0x01
>
> +typedef union s390_topology_id {
> + uint64_t id;
> + struct {
> + uint64_t level_6:8; /* byte 0 BE */
> + uint64_t level_5:8; /* byte 1 BE */
> + uint64_t drawer:8; /* byte 2 BE */
> + uint64_t book:8; /* byte 3 BE */
> + uint64_t socket:8; /* byte 4 BE */
> + uint64_t rsrv:5;
> + uint64_t d:1;
> + uint64_t p:2; /* byte 5 BE */
> + uint64_t type:8; /* byte 6 BE */
> + uint64_t origin:2;
This is two bits because it's the core divided by 64, and we have 248 cores at most?
Where is this set?
> + uint64_t core:6; /* byte 7 BE */
> + };
> +} s390_topology_id;
This struct seems to do double duty, 1. it represents a cpu and 2. a topology entry.
You also use it for sorting.
I would suggest to just use a cpu object when referring to a specific cpu and
put the relevant fields directly into the topology entry.
You get rid of the bit field that way.
You'd then need a comparison function for a cpu object and a topology entry.
As long as that isn't the only type pair that shouldn't be too ugly.
> +#define TOPO_CPU_MASK 0x000000000000003fUL
> +
> +typedef struct S390TopologyEntry {
> + s390_topology_id id;
> + QTAILQ_ENTRY(S390TopologyEntry) next;
> + uint64_t mask;
> +} S390TopologyEntry;
> +
> +typedef struct S390Topology {
> + QTAILQ_HEAD(, S390TopologyEntry) list;
> + uint8_t *sockets;
> + CpuTopology *smp;
> +} S390Topology;
> +
> +#ifdef CONFIG_KVM
> +bool s390_has_topology(void);
> +void s390_topology_set_cpu(MachineState *ms, S390CPU *cpu, Error **errp);
> +#else
> +static inline bool s390_has_topology(void)
> +{
> + return false;
> +}
> +static inline void s390_topology_set_cpu(MachineState *ms,
> + S390CPU *cpu,
> + Error **errp) {}
> +#endif
> +extern S390Topology s390_topology;
> +
> #endif
> diff --git a/hw/s390x/cpu-topology.c b/hw/s390x/cpu-topology.c
> new file mode 100644
> index 0000000000..438055c612
> --- /dev/null
> +++ b/hw/s390x/cpu-topology.c
> @@ -0,0 +1,293 @@
> +/*
> + * CPU Topology
> + *
> + * Copyright IBM Corp. 2022
> + * Author(s): Pierre Morel <pmorel@linux.ibm.com>
> +
> + * This work is licensed under the terms of the GNU GPL, version 2 or (at
> + * your option) any later version. See the COPYING file in the top-level
> + * directory.
> + */
> +
> +#include "qemu/osdep.h"
> +#include "qapi/error.h"
> +#include "qemu/error-report.h"
> +#include "hw/qdev-properties.h"
> +#include "hw/boards.h"
> +#include "qemu/typedefs.h"
> +#include "target/s390x/cpu.h"
> +#include "hw/s390x/s390-virtio-ccw.h"
> +#include "hw/s390x/cpu-topology.h"
> +
> +/*
> + * s390_topology is used to keep the topology information.
> + * .list: queue the topology entries inside which
> + * we keep the information on the CPU topology.
> + *
> + * .smp: keeps track of the machine topology.
> + *
> + * .socket: tracks information on the count of cores per socket.
> + *
> + */
> +S390Topology s390_topology = {
> + .list = QTAILQ_HEAD_INITIALIZER(s390_topology.list),
> + .sockets = NULL, /* will be initialized after the cpu model is realized */
I guess you should do the same for .smp then also.
> +};
> +
> +/**
> + * s390_socket_nb:
> + * @id: s390_topology_id
> + *
> + * Returns the socket number used inside the socket array.
> + */
> +static int s390_socket_nb(s390_topology_id id)
> +{
> + return (id.socket + 1) * (id.book + 1) * (id.drawer + 1);
This calculation doesn't make a whole lot of sense to me.
It's symmetric with regards to the variables, so (s=0 b=1 d=1)
will have the same result as (s=1 b=0 d=1).
You want the "global" socket number right?
So that would be (drawer * books_per_drawer + book) * sockets_per_book + socket.
> +}
> +
> +/**
> + * s390_has_topology:
> + *
> + * Return value: if the topology is supported by the machine.
> + */
> +bool s390_has_topology(void)
> +{
> + return false;
> +}
> +
> +/**
> + * s390_topology_init:
> + * @ms: the machine state where the machine topology is defined
> + *
> + * Keep track of the machine topology.
> + * Allocate an array to keep the count of cores per socket.
> + * The index of the array starts at socket 0 from book 0 and
> + * drawer 0 up to the maximum allowed by the machine topology.
> + */
> +static void s390_topology_init(MachineState *ms)
> +{
> + CpuTopology *smp = &ms->smp;
> +
> + s390_topology.smp = smp;
> + if (!s390_topology.sockets) {
Is this function being called multiple times, or why the if?
Use an assert instead?
> + s390_topology.sockets = g_new0(uint8_t, smp->sockets *
> + smp->books * smp->drawers);
> + }
> +}
> +
> +/**
> + * s390_topology_from_cpu:
> + * @cpu: The S390CPU
> + *
> + * Initialize the topology id from the CPU environment.
> + */
> +static s390_topology_id s390_topology_from_cpu(S390CPU *cpu)
> +{
> + s390_topology_id topology_id;
> +
> + topology_id.core = cpu->env.core_id;
> + topology_id.type = cpu->env.cpu_type;
> + topology_id.p = cpu->env.polarity;
> + topology_id.d = cpu->env.dedicated;
> + topology_id.socket = cpu->env.socket_id;
> + topology_id.book = cpu->env.book_id;
> + topology_id.drawer = cpu->env.drawer_id;
> +
> + return topology_id;
> +}
> +
> +/**
> + * s390_topology_set_entry:
> + * @entry: Topology entry to setup
> + * @id: topology id to use for the setup
> + *
> + * Set the core bit inside the topology mask and
> + * increments the number of cores for the socket.
> + */
> +static void s390_topology_set_entry(S390TopologyEntry *entry,
Not sure if I like the name, what it does is to add a cpu to the entry.
> + s390_topology_id id)
> +{
> + set_bit(63 - id.core, &entry->mask);
You need to subtract the origin first or that might be negative.
> + s390_topology.sockets[s390_socket_nb(id)]++;
> +}
> +
> +/**
> + * s390_topology_new_entry:
> + * @id: s390_topology_id to add
> + *
> + * Allocate a new entry and initialize it.
> + *
> + * returns the newly allocated entry.
> + */
> +static S390TopologyEntry *s390_topology_new_entry(s390_topology_id id)
> +{
> + S390TopologyEntry *entry;
> +
> + entry = g_malloc0(sizeof(S390TopologyEntry));
> + entry->id.id = id.id & ~TOPO_CPU_MASK;
> + s390_topology_set_entry(entry, id);
> +
> + return entry;
> +}
> +
> +/**
> + * s390_topology_insert:
> + *
> + * @id: s390_topology_id to insert.
> + *
> + * Parse the topology list to find if the entry already
> + * exist and add the core in it.
> + * If it does not exist, allocate a new entry and insert
> + * it in the queue from lower id to greater id.
> + */
> +static void s390_topology_insert(s390_topology_id id)
> +{
> + S390TopologyEntry *entry;
> + S390TopologyEntry *tmp = NULL;
> + uint64_t new_id;
> +
> + new_id = id.id & ~TOPO_CPU_MASK;
> +
> + /* First CPU to add to an entry */
> + if (QTAILQ_EMPTY(&s390_topology.list)) {
> + entry = s390_topology_new_entry(id);
> + QTAILQ_INSERT_HEAD(&s390_topology.list, entry, next);
> + return;
> + }
> +
> + QTAILQ_FOREACH(tmp, &s390_topology.list, next) {
> + if (new_id == tmp->id.id) {
> + s390_topology_set_entry(tmp, id);
> + return;
> + } else if (new_id < tmp->id.id) {
> + entry = s390_topology_new_entry(id);
> + QTAILQ_INSERT_BEFORE(tmp, entry, next);
> + return;
> + }
> + }
> +
> + entry = s390_topology_new_entry(id);
> + QTAILQ_INSERT_TAIL(&s390_topology.list, entry, next);
Consider adding a sentinel entry "at infinity", then that whole code
would simplify.
> +}
> +
> +/**
> + * s390_topology_check:
> + * @errp: Error pointer
> + * id: s390_topology_id to be verified
> + *
> + * The function checks if the topology id fits inside the
> + * system topology.
> + */
> +static void s390_topology_check(Error **errp, s390_topology_id id)
> +{
> + CpuTopology *smp = s390_topology.smp;
> +
> + if (id.socket > smp->sockets) {
> + error_setg(errp, "Unavailable socket: %d", id.socket);
> + return;
> + }
> + if (id.book > smp->books) {
> + error_setg(errp, "Unavailable book: %d", id.book);
> + return;
> + }
> + if (id.drawer > smp->drawers) {
> + error_setg(errp, "Unavailable drawer: %d", id.drawer);
> + return;
> + }
> + if (id.type != S390_TOPOLOGY_CPU_IFL) {
> + error_setg(errp, "Unknown cpu type: %d", id.type);
> + return;
> + }
> + /* Polarity and dedication can never be wrong */
> +}
> +
> +/**
> + * s390_topology_cpu_default:
> + * @errp: Error pointer
> + * @cpu: pointer to a S390CPU
> + *
> + * Setup the default topology for unset attributes.
> + *
> + * The function accept only all all default values or all set values
> + * for the geometry topology.
> + *
> + * The function calculates the (drawer_id, book_id, socket_id)
> + * topology by filling the cores starting from the first socket
> + * (0, 0, 0) up to the last (smp->drawers, smp->books, smp->sockets).
> + *
> + */
> +static void s390_topology_cpu_default(Error **errp, S390CPU *cpu)
> +{
> + CpuTopology *smp = s390_topology.smp;
> + CPUS390XState *env = &cpu->env;
> +
> + /* All geometry topology attributes must be set or all unset */
> + if ((env->socket_id < 0 || env->book_id < 0 || env->drawer_id < 0) &&
> + (env->socket_id >= 0 || env->book_id >= 0 || env->drawer_id >= 0)) {
> + error_setg(errp,
> + "Please define all or none of the topology geometry attributes");
> + return;
> + }
> +
> + /* Check if one of the geometry topology is unset */
> + if (env->socket_id < 0) {
> + /* Calculate default geometry topology attributes */
> + env->socket_id = (env->core_id / smp->cores) % smp->sockets;
> + env->book_id = (env->core_id / (smp->sockets * smp->cores)) %
> + smp->books;
> + env->drawer_id = (env->core_id /
> + (smp->books * smp->sockets * smp->cores)) %
> + smp->drawers;
> + }
> +}
> +
> +/**
> + * s390_topology_set_cpu:
> + * @ms: MachineState used to initialize the topology structure on
> + * first call.
> + * @cpu: the new S390CPU to insert in the topology structure
> + * @errp: the error pointer
> + *
> + * Called from CPU Hotplug to check and setup the CPU attributes
> + * before to insert the CPU in the topology.
> + */
> +void s390_topology_set_cpu(MachineState *ms, S390CPU *cpu, Error **errp)
> +{
> + Error *local_error = NULL;
Can't you just use ERRP_GUARD ?
> + s390_topology_id id;
> +
> + /*
> + * We do not want to initialize the topology if the cpu model
> + * does not support topology consequently, we have to wait for
", consequently," I think. Could you do the initialization some where else,
after you know what the cpu model is? Not that I object to doing it this way.
> + * the first CPU to be realized, which realizes the CPU model
> + * to initialize the topology structures.
> + *
> + * s390_topology_set_cpu() is called from the cpu hotplug.
> + */
> + if (!s390_topology.sockets) {
> + s390_topology_init(ms);
> + }
> +
> + s390_topology_cpu_default(&local_error, cpu);
> + if (local_error) {
> + error_propagate(errp, local_error);
> + return;
> + }
> +
> + id = s390_topology_from_cpu(cpu);
> +
> + /* Check for space on the socket */
> + if (s390_topology.sockets[s390_socket_nb(id)] >=
> + s390_topology.smp->sockets) {
> + error_setg(&local_error, "No more space on socket");
> + return;
> + }
> +
> + s390_topology_check(&local_error, id);
> + if (local_error) {
> + error_propagate(errp, local_error);
> + return;
> + }
> +
> + s390_topology_insert(id);
> +}
> diff --git a/hw/s390x/s390-virtio-ccw.c b/hw/s390x/s390-virtio-ccw.c
> index f3cc845d3b..c98b93a15f 100644
> --- a/hw/s390x/s390-virtio-ccw.c
> +++ b/hw/s390x/s390-virtio-ccw.c
> @@ -44,6 +44,7 @@
> #include "hw/s390x/pv.h"
> #include "migration/blocker.h"
> #include "qapi/visitor.h"
> +#include "hw/s390x/cpu-topology.h"
>
> static Error *pv_mig_blocker;
>
> @@ -310,10 +311,19 @@ static void s390_cpu_plug(HotplugHandler *hotplug_dev,
> {
> MachineState *ms = MACHINE(hotplug_dev);
> S390CPU *cpu = S390_CPU(dev);
> + Error *local_err = NULL;
>
> g_assert(!ms->possible_cpus->cpus[cpu->env.core_id].cpu);
> ms->possible_cpus->cpus[cpu->env.core_id].cpu = OBJECT(dev);
>
> + if (s390_has_topology()) {
> + s390_topology_set_cpu(ms, cpu, &local_err);
> + if (local_err) {
> + error_propagate(errp, local_err);
> + return;
> + }
> + }
> +
> if (dev->hotplugged) {
> raise_irq_cpu_hotplug();
> }
> diff --git a/hw/s390x/meson.build b/hw/s390x/meson.build
> index f291016fee..58dfbdff4f 100644
> --- a/hw/s390x/meson.build
> +++ b/hw/s390x/meson.build
> @@ -24,6 +24,7 @@ s390x_ss.add(when: 'CONFIG_KVM', if_true: files(
> 's390-stattrib-kvm.c',
> 'pv.c',
> 's390-pci-kvm.c',
> + 'cpu-topology.c',
> ))
> s390x_ss.add(when: 'CONFIG_TCG', if_true: files(
> 'tod-tcg.c',
^ permalink raw reply [flat|nested] 80+ messages in thread
* Re: [PATCH v14 03/11] target/s390x/cpu topology: handle STSI(15) and build the SYSIB
2023-01-05 14:53 ` [PATCH v14 03/11] target/s390x/cpu topology: handle STSI(15) and build the SYSIB Pierre Morel
2023-01-10 14:29 ` Thomas Huth
@ 2023-01-16 13:11 ` Nina Schoetterl-Glausch
2023-01-16 15:39 ` Pierre Morel
1 sibling, 1 reply; 80+ messages in thread
From: Nina Schoetterl-Glausch @ 2023-01-16 13:11 UTC (permalink / raw)
To: Pierre Morel, qemu-s390x
Cc: qemu-devel, borntraeger, pasic, richard.henderson, david, thuth,
cohuck, mst, pbonzini, kvm, ehabkost, marcel.apfelbaum, eblake,
armbru, seiden, nrb, frankja, berrange, clg
On Thu, 2023-01-05 at 15:53 +0100, Pierre Morel wrote:
> On interception of STSI(15.1.x) the System Information Block
> (SYSIB) is built from the list of pre-ordered topology entries.
>
> Signed-off-by: Pierre Morel <pmorel@linux.ibm.com>
> ---
> include/hw/s390x/cpu-topology.h | 3 +
> include/hw/s390x/sclp.h | 1 +
> target/s390x/cpu.h | 78 ++++++++++++++++++
> target/s390x/kvm/cpu_topology.c | 136 ++++++++++++++++++++++++++++++++
> target/s390x/kvm/kvm.c | 5 +-
> target/s390x/kvm/meson.build | 3 +-
> 6 files changed, 224 insertions(+), 2 deletions(-)
> create mode 100644 target/s390x/kvm/cpu_topology.c
>
> diff --git a/include/hw/s390x/cpu-topology.h b/include/hw/s390x/cpu-topology.h
> index b3fd752d8d..9571aa70e5 100644
> --- a/include/hw/s390x/cpu-topology.h
> +++ b/include/hw/s390x/cpu-topology.h
> @@ -41,6 +41,9 @@ typedef union s390_topology_id {
> };
> } s390_topology_id;
> #define TOPO_CPU_MASK 0x000000000000003fUL
> +#define TOPO_SOCKET_MASK 0x0000ffffff000000UL
> +#define TOPO_BOOK_MASK 0x0000ffff00000000UL
> +#define TOPO_DRAWER_MASK 0x0000ff0000000000UL
>
> typedef struct S390TopologyEntry {
> s390_topology_id id;
> diff --git a/include/hw/s390x/sclp.h b/include/hw/s390x/sclp.h
> index d3ade40a5a..712fd68123 100644
> --- a/include/hw/s390x/sclp.h
> +++ b/include/hw/s390x/sclp.h
> @@ -112,6 +112,7 @@ typedef struct CPUEntry {
> } QEMU_PACKED CPUEntry;
>
> #define SCLP_READ_SCP_INFO_FIXED_CPU_OFFSET 128
> +#define SCLP_READ_SCP_INFO_MNEST 2
> typedef struct ReadInfo {
> SCCBHeader h;
> uint16_t rnmax;
> diff --git a/target/s390x/cpu.h b/target/s390x/cpu.h
> index 39ea63a416..78988048dd 100644
> --- a/target/s390x/cpu.h
> +++ b/target/s390x/cpu.h
> @@ -561,6 +561,25 @@ typedef struct SysIB_322 {
> } SysIB_322;
> QEMU_BUILD_BUG_ON(sizeof(SysIB_322) != 4096);
>
> +#define S390_TOPOLOGY_MAG 6
> +#define S390_TOPOLOGY_MAG6 0
> +#define S390_TOPOLOGY_MAG5 1
> +#define S390_TOPOLOGY_MAG4 2
> +#define S390_TOPOLOGY_MAG3 3
> +#define S390_TOPOLOGY_MAG2 4
> +#define S390_TOPOLOGY_MAG1 5
> +/* Configuration topology */
> +typedef struct SysIB_151x {
> + uint8_t reserved0[2];
> + uint16_t length;
> + uint8_t mag[S390_TOPOLOGY_MAG];
> + uint8_t reserved1;
> + uint8_t mnest;
> + uint32_t reserved2;
> + char tle[];
> +} QEMU_PACKED QEMU_ALIGNED(8) SysIB_151x;
> +QEMU_BUILD_BUG_ON(sizeof(SysIB_151x) != 16);
> +
> typedef union SysIB {
> SysIB_111 sysib_111;
> SysIB_121 sysib_121;
> @@ -568,9 +587,68 @@ typedef union SysIB {
> SysIB_221 sysib_221;
> SysIB_222 sysib_222;
> SysIB_322 sysib_322;
> + SysIB_151x sysib_151x;
> } SysIB;
> QEMU_BUILD_BUG_ON(sizeof(SysIB) != 4096);
>
> +/*
> + * CPU Topology List provided by STSI with fc=15 provides a list
> + * of two different Topology List Entries (TLE) types to specify
> + * the topology hierarchy.
> + *
> + * - Container Topology List Entry
> + * Defines a container to contain other Topology List Entries
> + * of any type, nested containers or CPU.
> + * - CPU Topology List Entry
> + * Specifies the CPUs position, type, entitlement and polarization
> + * of the CPUs contained in the last Container TLE.
> + *
> + * There can be theoretically up to five levels of containers, QEMU
> + * uses only one level, the socket level.
> + *
> + * A container of with a nesting level (NL) greater than 1 can only
> + * contain another container of nesting level NL-1.
> + *
> + * A container of nesting level 1 (socket), contains as many CPU TLE
> + * as needed to describe the position and qualities of all CPUs inside
> + * the container.
> + * The qualities of a CPU are polarization, entitlement and type.
> + *
> + * The CPU TLE defines the position of the CPUs of identical qualities
> + * using a 64bits mask which first bit has its offset defined by
> + * the CPU address orgin field of the CPU TLE like in:
> + * CPU address = origin * 64 + bit position within the mask
> + *
> + */
> +/* Container type Topology List Entry */
> +/* Container type Topology List Entry */
> +typedef struct SysIBTl_container {
> + uint8_t nl;
> + uint8_t reserved[6];
> + uint8_t id;
> +} QEMU_PACKED QEMU_ALIGNED(8) SysIBTl_container;
> +QEMU_BUILD_BUG_ON(sizeof(SysIBTl_container) != 8);
> +
> +/* CPU type Topology List Entry */
> +typedef struct SysIBTl_cpu {
> + uint8_t nl;
> + uint8_t reserved0[3];
> + uint8_t reserved1:5;
> + uint8_t dedicated:1;
> + uint8_t polarity:2;
> + uint8_t type;
> + uint16_t origin;
> + uint64_t mask;
> +} QEMU_PACKED QEMU_ALIGNED(8) SysIBTl_cpu;
> +QEMU_BUILD_BUG_ON(sizeof(SysIBTl_cpu) != 16);
> +
> +/* Max size of a SYSIB structure is when all CPU are alone in a container */
> +#define S390_TOPOLOGY_SYSIB_SIZE (sizeof(SysIB_151x) + \
> + S390_MAX_CPUS * (sizeof(SysIBTl_container) + \
> + sizeof(SysIBTl_cpu)))
I don't think this is accurate anymore, if you have drawers and books.
In that case you could have 3 containers per 1 cpu.
You could also use the maxcpus number at runtime instead of S390_MAX_CPUS.
I also think you could do sizeof(SysIB) + sizeof(SysIBTl_cpu) if you check
if the sysib overflows 4k while building it.
> +
> +void insert_stsi_15_1_x(S390CPU *cpu, int sel2, __u64 addr, uint8_t ar);
> +
> /* MMU defines */
> #define ASCE_ORIGIN (~0xfffULL) /* segment table origin */
> #define ASCE_SUBSPACE 0x200 /* subspace group control */
> diff --git a/target/s390x/kvm/cpu_topology.c b/target/s390x/kvm/cpu_topology.c
> new file mode 100644
> index 0000000000..3831a3264c
> --- /dev/null
> +++ b/target/s390x/kvm/cpu_topology.c
> @@ -0,0 +1,136 @@
> +/*
> + * QEMU S390x CPU Topology
> + *
> + * Copyright IBM Corp. 2022
> + * Author(s): Pierre Morel <pmorel@linux.ibm.com>
> + *
> + * This work is licensed under the terms of the GNU GPL, version 2 or (at
> + * your option) any later version. See the COPYING file in the top-level
> + * directory.
> + */
> +#include "qemu/osdep.h"
> +#include "cpu.h"
> +#include "hw/s390x/pv.h"
> +#include "hw/sysbus.h"
> +#include "hw/s390x/sclp.h"
> +#include "hw/s390x/cpu-topology.h"
> +
> +static char *fill_container(char *p, int level, int id)
> +{
> + SysIBTl_container *tle = (SysIBTl_container *)p;
> +
> + tle->nl = level;
> + tle->id = id;
> + return p + sizeof(*tle);
> +}
> +
> +static char *fill_tle_cpu(char *p, S390TopologyEntry *entry)
> +{
> + SysIBTl_cpu *tle = (SysIBTl_cpu *)p;
> + s390_topology_id topology_id = entry->id;
> +
> + tle->nl = 0;
> + tle->dedicated = topology_id.d;
> + tle->polarity = topology_id.p;
> + tle->type = topology_id.type;
> + tle->origin = topology_id.origin;
You need to multiply that value by 64, no?
And convert it to BE.
> + tle->mask = cpu_to_be64(entry->mask);
> + return p + sizeof(*tle);
> +}
> +
> +static char *s390_top_set_level(char *p, int level)
> +{
> + S390TopologyEntry *entry;
> + uint64_t last_socket = -1UL;
> + uint64_t last_book = -1UL;
> + uint64_t last_drawer = -1UL;
-1UL looks funny to me, but there is nothing wrong with it.
But I don't see a reason not to use int and initialize it with -1.
> + int drawer_cnt = 0;
> + int book_cnt = 0;
> + int socket_cnt = 0;
> +
> + QTAILQ_FOREACH(entry, &s390_topology.list, next) {
> +
> + if (level > 3 && (last_drawer != entry->id.drawer)) {
> + book_cnt = 0;
> + socket_cnt = 0;
> + p = fill_container(p, 3, drawer_cnt++);
> + last_drawer = entry->id.id & TOPO_DRAWER_MASK;
> + p = fill_container(p, 2, book_cnt++);
> + last_book = entry->id.id & TOPO_BOOK_MASK;
> + p = fill_container(p, 1, socket_cnt++);
> + last_socket = entry->id.id & TOPO_SOCKET_MASK;
> + p = fill_tle_cpu(p, entry);
> + } else if (level > 2 && (last_book !=
> + (entry->id.id & TOPO_BOOK_MASK))) {
> + socket_cnt = 0;
> + p = fill_container(p, 2, book_cnt++);
> + last_book = entry->id.id & TOPO_BOOK_MASK;
> + p = fill_container(p, 1, socket_cnt++);
> + last_socket = entry->id.id & TOPO_SOCKET_MASK;
> + p = fill_tle_cpu(p, entry);
> + } else if (last_socket != (entry->id.id & TOPO_SOCKET_MASK)) {
> + p = fill_container(p, 1, socket_cnt++);
> + last_socket = entry->id.id & TOPO_SOCKET_MASK;
> + p = fill_tle_cpu(p, entry);
> + } else {
> + p = fill_tle_cpu(p, entry);
> + }
> + }
> +
> + return p;
> +}
I think you can do this a bit more readable and reduce redundancy.
Pseudo code:
foreach entry:
bool drawer_change = last_drawer != current_drawer
bool book_change = drawer_change || last_book != current_book
bool socket_change = book_change || last_socket != current_socket
if (level > 3 && drawer_change)
reset book id
fill drawer container
drawer id++
if (level > 2 && book_change)
reset socket id
fill book container
book id++
if (socket_change)
fill socket container
socket id++
fill cpu entry
update last_drawer, _book, _socket
You can also check after after every fill if the buffer has been overflowed,
that is if the function wrote more than sizeof(SysIB) - sizeof(SysIB_151x) bytes.
Or you check it once at the end if you increase the size of the buffer a bit.
Then you don't need to allocate the absolute maximum.
I think you could also use global ids for the containers.
So directly use the drawer id from the entry,
use (drawer id * smp.books) + book id, and so on.
If you update last_* after setting *_changed you don't need to maintain ids,
you can just use last_*.
[...]
^ permalink raw reply [flat|nested] 80+ messages in thread
* Re: [PATCH v14 03/11] target/s390x/cpu topology: handle STSI(15) and build the SYSIB
2023-01-16 13:11 ` Nina Schoetterl-Glausch
@ 2023-01-16 15:39 ` Pierre Morel
0 siblings, 0 replies; 80+ messages in thread
From: Pierre Morel @ 2023-01-16 15:39 UTC (permalink / raw)
To: Nina Schoetterl-Glausch, qemu-s390x
Cc: qemu-devel, borntraeger, pasic, richard.henderson, david, thuth,
cohuck, mst, pbonzini, kvm, ehabkost, marcel.apfelbaum, eblake,
armbru, seiden, nrb, frankja, berrange, clg
On 1/16/23 14:11, Nina Schoetterl-Glausch wrote:
> On Thu, 2023-01-05 at 15:53 +0100, Pierre Morel wrote:
>> On interception of STSI(15.1.x) the System Information Block
>> (SYSIB) is built from the list of pre-ordered topology entries.
>>
>> Signed-off-by: Pierre Morel <pmorel@linux.ibm.com>
>> ---
>> include/hw/s390x/cpu-topology.h | 3 +
>> include/hw/s390x/sclp.h | 1 +
>> target/s390x/cpu.h | 78 ++++++++++++++++++
>> target/s390x/kvm/cpu_topology.c | 136 ++++++++++++++++++++++++++++++++
>> target/s390x/kvm/kvm.c | 5 +-
>> target/s390x/kvm/meson.build | 3 +-
>> 6 files changed, 224 insertions(+), 2 deletions(-)
>> create mode 100644 target/s390x/kvm/cpu_topology.c
>>
>> diff --git a/include/hw/s390x/cpu-topology.h b/include/hw/s390x/cpu-topology.h
>> index b3fd752d8d..9571aa70e5 100644
>> --- a/include/hw/s390x/cpu-topology.h
>> +++ b/include/hw/s390x/cpu-topology.h
>> @@ -41,6 +41,9 @@ typedef union s390_topology_id {
>> };
>> } s390_topology_id;
>> #define TOPO_CPU_MASK 0x000000000000003fUL
>> +#define TOPO_SOCKET_MASK 0x0000ffffff000000UL
>> +#define TOPO_BOOK_MASK 0x0000ffff00000000UL
>> +#define TOPO_DRAWER_MASK 0x0000ff0000000000UL
>>
>> typedef struct S390TopologyEntry {
>> s390_topology_id id;
>> diff --git a/include/hw/s390x/sclp.h b/include/hw/s390x/sclp.h
>> index d3ade40a5a..712fd68123 100644
>> --- a/include/hw/s390x/sclp.h
>> +++ b/include/hw/s390x/sclp.h
>> @@ -112,6 +112,7 @@ typedef struct CPUEntry {
>> } QEMU_PACKED CPUEntry;
>>
>> #define SCLP_READ_SCP_INFO_FIXED_CPU_OFFSET 128
>> +#define SCLP_READ_SCP_INFO_MNEST 2
>> typedef struct ReadInfo {
>> SCCBHeader h;
>> uint16_t rnmax;
>> diff --git a/target/s390x/cpu.h b/target/s390x/cpu.h
>> index 39ea63a416..78988048dd 100644
>> --- a/target/s390x/cpu.h
>> +++ b/target/s390x/cpu.h
>> @@ -561,6 +561,25 @@ typedef struct SysIB_322 {
>> } SysIB_322;
>> QEMU_BUILD_BUG_ON(sizeof(SysIB_322) != 4096);
>>
>> +#define S390_TOPOLOGY_MAG 6
>> +#define S390_TOPOLOGY_MAG6 0
>> +#define S390_TOPOLOGY_MAG5 1
>> +#define S390_TOPOLOGY_MAG4 2
>> +#define S390_TOPOLOGY_MAG3 3
>> +#define S390_TOPOLOGY_MAG2 4
>> +#define S390_TOPOLOGY_MAG1 5
>> +/* Configuration topology */
>> +typedef struct SysIB_151x {
>> + uint8_t reserved0[2];
>> + uint16_t length;
>> + uint8_t mag[S390_TOPOLOGY_MAG];
>> + uint8_t reserved1;
>> + uint8_t mnest;
>> + uint32_t reserved2;
>> + char tle[];
>> +} QEMU_PACKED QEMU_ALIGNED(8) SysIB_151x;
>> +QEMU_BUILD_BUG_ON(sizeof(SysIB_151x) != 16);
>> +
>> typedef union SysIB {
>> SysIB_111 sysib_111;
>> SysIB_121 sysib_121;
>> @@ -568,9 +587,68 @@ typedef union SysIB {
>> SysIB_221 sysib_221;
>> SysIB_222 sysib_222;
>> SysIB_322 sysib_322;
>> + SysIB_151x sysib_151x;
>> } SysIB;
>> QEMU_BUILD_BUG_ON(sizeof(SysIB) != 4096);
>>
>> +/*
>> + * CPU Topology List provided by STSI with fc=15 provides a list
>> + * of two different Topology List Entries (TLE) types to specify
>> + * the topology hierarchy.
>> + *
>> + * - Container Topology List Entry
>> + * Defines a container to contain other Topology List Entries
>> + * of any type, nested containers or CPU.
>> + * - CPU Topology List Entry
>> + * Specifies the CPUs position, type, entitlement and polarization
>> + * of the CPUs contained in the last Container TLE.
>> + *
>> + * There can be theoretically up to five levels of containers, QEMU
>> + * uses only one level, the socket level.
>> + *
>> + * A container of with a nesting level (NL) greater than 1 can only
>> + * contain another container of nesting level NL-1.
>> + *
>> + * A container of nesting level 1 (socket), contains as many CPU TLE
>> + * as needed to describe the position and qualities of all CPUs inside
>> + * the container.
>> + * The qualities of a CPU are polarization, entitlement and type.
>> + *
>> + * The CPU TLE defines the position of the CPUs of identical qualities
>> + * using a 64bits mask which first bit has its offset defined by
>> + * the CPU address orgin field of the CPU TLE like in:
>> + * CPU address = origin * 64 + bit position within the mask
>> + *
>> + */
>> +/* Container type Topology List Entry */
>> +/* Container type Topology List Entry */
>> +typedef struct SysIBTl_container {
>> + uint8_t nl;
>> + uint8_t reserved[6];
>> + uint8_t id;
>> +} QEMU_PACKED QEMU_ALIGNED(8) SysIBTl_container;
>> +QEMU_BUILD_BUG_ON(sizeof(SysIBTl_container) != 8);
>> +
>> +/* CPU type Topology List Entry */
>> +typedef struct SysIBTl_cpu {
>> + uint8_t nl;
>> + uint8_t reserved0[3];
>> + uint8_t reserved1:5;
>> + uint8_t dedicated:1;
>> + uint8_t polarity:2;
>> + uint8_t type;
>> + uint16_t origin;
>> + uint64_t mask;
>> +} QEMU_PACKED QEMU_ALIGNED(8) SysIBTl_cpu;
>> +QEMU_BUILD_BUG_ON(sizeof(SysIBTl_cpu) != 16);
>> +
>> +/* Max size of a SYSIB structure is when all CPU are alone in a container */
>> +#define S390_TOPOLOGY_SYSIB_SIZE (sizeof(SysIB_151x) + \
>> + S390_MAX_CPUS * (sizeof(SysIBTl_container) + \
>> + sizeof(SysIBTl_cpu)))
>
> I don't think this is accurate anymore, if you have drawers and books.
> In that case you could have 3 containers per 1 cpu.
> You could also use the maxcpus number at runtime instead of S390_MAX_CPUS.
> I also think you could do sizeof(SysIB) + sizeof(SysIBTl_cpu) if you check
> if the sysib overflows 4k while building it.
Right.
And I think your other proposal here under is better
>
>> +
>> +void insert_stsi_15_1_x(S390CPU *cpu, int sel2, __u64 addr, uint8_t ar);
>> +
>> /* MMU defines */
>> #define ASCE_ORIGIN (~0xfffULL) /* segment table origin */
>> #define ASCE_SUBSPACE 0x200 /* subspace group control */
>> diff --git a/target/s390x/kvm/cpu_topology.c b/target/s390x/kvm/cpu_topology.c
>> new file mode 100644
>> index 0000000000..3831a3264c
>> --- /dev/null
>> +++ b/target/s390x/kvm/cpu_topology.c
>> @@ -0,0 +1,136 @@
>> +/*
>> + * QEMU S390x CPU Topology
>> + *
>> + * Copyright IBM Corp. 2022
>> + * Author(s): Pierre Morel <pmorel@linux.ibm.com>
>> + *
>> + * This work is licensed under the terms of the GNU GPL, version 2 or (at
>> + * your option) any later version. See the COPYING file in the top-level
>> + * directory.
>> + */
>> +#include "qemu/osdep.h"
>> +#include "cpu.h"
>> +#include "hw/s390x/pv.h"
>> +#include "hw/sysbus.h"
>> +#include "hw/s390x/sclp.h"
>> +#include "hw/s390x/cpu-topology.h"
>> +
>> +static char *fill_container(char *p, int level, int id)
>> +{
>> + SysIBTl_container *tle = (SysIBTl_container *)p;
>> +
>> + tle->nl = level;
>> + tle->id = id;
>> + return p + sizeof(*tle);
>> +}
>> +
>> +static char *fill_tle_cpu(char *p, S390TopologyEntry *entry)
>> +{
>> + SysIBTl_cpu *tle = (SysIBTl_cpu *)p;
>> + s390_topology_id topology_id = entry->id;
>> +
>> + tle->nl = 0;
>> + tle->dedicated = topology_id.d;
>> + tle->polarity = topology_id.p;
>> + tle->type = topology_id.type;
>> + tle->origin = topology_id.origin;
>
> You need to multiply that value by 64, no?
> And convert it to BE.
Yes right, I already had this error, I must have lost it in a rebase.
>
>> + tle->mask = cpu_to_be64(entry->mask);
>> + return p + sizeof(*tle);
>> +}
>> +
>> +static char *s390_top_set_level(char *p, int level)
>> +{
>> + S390TopologyEntry *entry;
>> + uint64_t last_socket = -1UL;
>> + uint64_t last_book = -1UL;
>> + uint64_t last_drawer = -1UL;
>
> -1UL looks funny to me, but there is nothing wrong with it.
> But I don't see a reason not to use int and initialize it with -1.
>
>> + int drawer_cnt = 0;
>> + int book_cnt = 0;
>> + int socket_cnt = 0;
>> +
>> + QTAILQ_FOREACH(entry, &s390_topology.list, next) {
>> +
>> + if (level > 3 && (last_drawer != entry->id.drawer)) {
>> + book_cnt = 0;
>> + socket_cnt = 0;
>> + p = fill_container(p, 3, drawer_cnt++);
>> + last_drawer = entry->id.id & TOPO_DRAWER_MASK;
>> + p = fill_container(p, 2, book_cnt++);
>> + last_book = entry->id.id & TOPO_BOOK_MASK;
>> + p = fill_container(p, 1, socket_cnt++);
>> + last_socket = entry->id.id & TOPO_SOCKET_MASK;
>> + p = fill_tle_cpu(p, entry);
>> + } else if (level > 2 && (last_book !=
>> + (entry->id.id & TOPO_BOOK_MASK))) {
>> + socket_cnt = 0;
>> + p = fill_container(p, 2, book_cnt++);
>> + last_book = entry->id.id & TOPO_BOOK_MASK;
>> + p = fill_container(p, 1, socket_cnt++);
>> + last_socket = entry->id.id & TOPO_SOCKET_MASK;
>> + p = fill_tle_cpu(p, entry);
>> + } else if (last_socket != (entry->id.id & TOPO_SOCKET_MASK)) {
>> + p = fill_container(p, 1, socket_cnt++);
>> + last_socket = entry->id.id & TOPO_SOCKET_MASK;
>> + p = fill_tle_cpu(p, entry);
>> + } else {
>> + p = fill_tle_cpu(p, entry);
>> + }
>> + }
>> +
>> + return p;
>> +}
>
> I think you can do this a bit more readable and reduce redundancy.
> Pseudo code:
>
> foreach entry:
> bool drawer_change = last_drawer != current_drawer
> bool book_change = drawer_change || last_book != current_book
> bool socket_change = book_change || last_socket != current_socket
>
> if (level > 3 && drawer_change)
> reset book id
> fill drawer container
> drawer id++
> if (level > 2 && book_change)
> reset socket id
> fill book container
> book id++
> if (socket_change)
> fill socket container
> socket id++
> fill cpu entry
>
> update last_drawer, _book, _socket
>
> You can also check after after every fill if the buffer has been overflowed,
> that is if the function wrote more than sizeof(SysIB) - sizeof(SysIB_151x) bytes.
> Or you check it once at the end if you increase the size of the buffer a bit.
> Then you don't need to allocate the absolute maximum.
>
> I think you could also use global ids for the containers.
> So directly use the drawer id from the entry,
> use (drawer id * smp.books) + book id, and so on.
> If you update last_* after setting *_changed you don't need to maintain ids,
> you can just use last_*.
OK, seems better to me
Thanks
Regards,
Pierre
>
>
> [...]
--
Pierre Morel
IBM Lab Boeblingen
^ permalink raw reply [flat|nested] 80+ messages in thread
* Re: [PATCH v14 01/11] s390x/cpu topology: adding s390 specificities to CPU topology
2023-01-10 11:37 ` Thomas Huth
@ 2023-01-16 16:32 ` Pierre Morel
2023-01-17 7:25 ` Thomas Huth
0 siblings, 1 reply; 80+ messages in thread
From: Pierre Morel @ 2023-01-16 16:32 UTC (permalink / raw)
To: Thomas Huth, qemu-s390x
Cc: qemu-devel, borntraeger, pasic, richard.henderson, david, cohuck,
mst, pbonzini, kvm, ehabkost, marcel.apfelbaum, eblake, armbru,
seiden, nrb, scgl, frankja, berrange, clg
On 1/10/23 12:37, Thomas Huth wrote:
> On 05/01/2023 15.53, Pierre Morel wrote:
>> S390 adds two new SMP levels, drawers and books to the CPU
>> topology.
>> The S390 CPU have specific toplogy features like dedication
>> and polarity to give to the guest indications on the host
>> vCPUs scheduling and help the guest take the best decisions
>> on the scheduling of threads on the vCPUs.
>>
>> Let us provide the SMP properties with books and drawers levels
>> and S390 CPU with dedication and polarity,
>>
>> Signed-off-by: Pierre Morel <pmorel@linux.ibm.com>
>> ---
> ...
>> diff --git a/qapi/machine.json b/qapi/machine.json
>> index b9228a5e46..ff8f2b0e84 100644
>> --- a/qapi/machine.json
>> +++ b/qapi/machine.json
>> @@ -900,13 +900,15 @@
>> # a CPU is being hotplugged.
>> #
>> # @node-id: NUMA node ID the CPU belongs to
>> -# @socket-id: socket number within node/board the CPU belongs to
>> +# @drawer-id: drawer number within node/board the CPU belongs to
>> +# @book-id: book number within drawer/node/board the CPU belongs to
>> +# @socket-id: socket number within book/node/board the CPU belongs to
>
> I think the new entries need a "(since 8.0)" comment (similar to die-id
> and cluster-id below).
right
>
> Other question: Do we have "node-id"s on s390x? If not, is that similar
> to books or drawers, i.e. just another word? If so, we should maybe
> rather re-use "nodes" instead of introducing a new name for the same thing?
We have theoretically nodes-id on s390x, it is the level 5 of the
topology, above drawers.
Currently it is not used in s390x topology, the maximum level returned
to a LPAR host is 4.
I suppose that it adds a possibility to link several s390x with a fast
network.
>
>> # @die-id: die number within socket the CPU belongs to (since 4.1)
>> # @cluster-id: cluster number within die the CPU belongs to (since 7.1)
>> # @core-id: core number within cluster the CPU belongs to
>> # @thread-id: thread number within core the CPU belongs to
>> #
>> -# Note: currently there are 6 properties that could be present
>> +# Note: currently there are 8 properties that could be present
>> # but management should be prepared to pass through other
>> # properties with device_add command to allow for future
>> # interface extension. This also requires the filed names to
>> be kept in
>> @@ -916,6 +918,8 @@
>> ##
>> { 'struct': 'CpuInstanceProperties',
>> 'data': { '*node-id': 'int',
>> + '*drawer-id': 'int',
>> + '*book-id': 'int',
>> '*socket-id': 'int',
>> '*die-id': 'int',
>> '*cluster-id': 'int',
>> @@ -1465,6 +1469,10 @@
>> #
>> # @cpus: number of virtual CPUs in the virtual machine
>> #
>> +# @drawers: number of drawers in the CPU topology
>> +#
>> +# @books: number of books in the CPU topology
>> +#
>
> These also need a "(since 8.0)" comment at the end.
right again, I will add this.
>
>> # @sockets: number of sockets in the CPU topology
>> #
>> # @dies: number of dies per socket in the CPU topology
>> @@ -1481,6 +1489,8 @@
>> ##
>> { 'struct': 'SMPConfiguration', 'data': {
>> '*cpus': 'int',
>> + '*drawers': 'int',
>> + '*books': 'int',
>> '*sockets': 'int',
>> '*dies': 'int',
>> '*clusters': 'int',
> ...
>> diff --git a/qemu-options.hx b/qemu-options.hx
>> index 7f99d15b23..8dc9a4c052 100644
>> --- a/qemu-options.hx
>> +++ b/qemu-options.hx
>> @@ -250,11 +250,13 @@ SRST
>> ERST
>> DEF("smp", HAS_ARG, QEMU_OPTION_smp,
>> - "-smp
>> [[cpus=]n][,maxcpus=maxcpus][,sockets=sockets][,dies=dies][,clusters=clusters][,cores=cores][,threads=threads]\n"
>> + "-smp
>> [[cpus=]n][,maxcpus=maxcpus][,drawers=drawers][,books=books][,sockets=sockets][,dies=dies][,clusters=clusters][,cores=cores][,threads=threads]\n"
>
> This line now got too long. Please add a newline inbetween.
OK
Thanks.
Regards,
Pierre
--
Pierre Morel
IBM Lab Boeblingen
^ permalink raw reply [flat|nested] 80+ messages in thread
* Re: [PATCH v14 01/11] s390x/cpu topology: adding s390 specificities to CPU topology
2023-01-13 16:58 ` Nina Schoetterl-Glausch
@ 2023-01-16 17:28 ` Pierre Morel
2023-01-16 20:34 ` Nina Schoetterl-Glausch
2023-01-17 7:22 ` Thomas Huth
0 siblings, 2 replies; 80+ messages in thread
From: Pierre Morel @ 2023-01-16 17:28 UTC (permalink / raw)
To: Nina Schoetterl-Glausch, qemu-s390x
Cc: qemu-devel, borntraeger, pasic, richard.henderson, david, thuth,
cohuck, mst, pbonzini, kvm, ehabkost, marcel.apfelbaum, eblake,
armbru, seiden, nrb, frankja, berrange, clg
On 1/13/23 17:58, Nina Schoetterl-Glausch wrote:
> On Thu, 2023-01-05 at 15:53 +0100, Pierre Morel wrote:
>> S390 adds two new SMP levels, drawers and books to the CPU
>> topology.
>> The S390 CPU have specific toplogy features like dedication
>> and polarity to give to the guest indications on the host
>> vCPUs scheduling and help the guest take the best decisions
>> on the scheduling of threads on the vCPUs.
>>
>> Let us provide the SMP properties with books and drawers levels
>> and S390 CPU with dedication and polarity,
>>
>> Signed-off-by: Pierre Morel <pmorel@linux.ibm.com>
>> ---
>> qapi/machine.json | 14 ++++++++--
>> include/hw/boards.h | 10 ++++++-
>> include/hw/s390x/cpu-topology.h | 23 ++++++++++++++++
>> target/s390x/cpu.h | 6 +++++
>> hw/core/machine-smp.c | 48 ++++++++++++++++++++++++++++-----
>> hw/core/machine.c | 4 +++
>> hw/s390x/s390-virtio-ccw.c | 2 ++
>> softmmu/vl.c | 6 +++++
>> target/s390x/cpu.c | 10 +++++++
>> qemu-options.hx | 6 +++--
>> 10 files changed, 117 insertions(+), 12 deletions(-)
>> create mode 100644 include/hw/s390x/cpu-topology.h
>>
> [...]
>
>> diff --git a/target/s390x/cpu.h b/target/s390x/cpu.h
>> index 7d6d01325b..39ea63a416 100644
>> --- a/target/s390x/cpu.h
>> +++ b/target/s390x/cpu.h
>> @@ -131,6 +131,12 @@ struct CPUArchState {
>>
>> #if !defined(CONFIG_USER_ONLY)
>> uint32_t core_id; /* PoP "CPU address", same as cpu_index */
>> + int32_t socket_id;
>> + int32_t book_id;
>> + int32_t drawer_id;
>> + int32_t dedicated;
>> + int32_t polarity;
>
> If I understood the architecture correctly, the polarity is a property of the configuration,
> not the cpus. So this should be vertical_entitlement, and there should be a machine (?) property
> specifying if the polarity is horizontal or vertical.
You are right, considering PTF only, the documentation says PTF([01])
does the following:
"... a process is initiated to place all CPUs in the configuration into
the polarization specified by the function code, ..."
So on one side the polarization property is explicitly set on the CPU,
and on the other side all CPU are supposed to be in the same
polarization state.
So yes we can make the horizontal/vertical a machine property.
However, we do not need to set this tunable as the documentation says
that the machine always start with horizontal polarization.
On the other hand the documentation mixes a lot vertical with different
entitlement and horizontal polarization, for TLE order and slacks so I
prefer to keep the complete description of the polarization as CPU
properties in case we miss something.
PTF([01]) are no performance bottle neck and the number of CPU is likely
to be small, even a maximum of 248 is possible KVM warns above 16 CPU so
the loop for setting all CPU inside PTF interception is not very
problematic I think.
Doing like you say should simplify PTF interception (no loop) but
complicates (some more if/else) TLE handling and QMP information display
on CPU.
So I will have a look at the implications and answer again on this.
Thanks,
Regards,
Pierre
--
Pierre Morel
IBM Lab Boeblingen
^ permalink raw reply [flat|nested] 80+ messages in thread
* Re: [PATCH v14 02/11] s390x/cpu topology: add topology entries on CPU hotplug
2023-01-10 13:00 ` Thomas Huth
2023-01-11 9:23 ` Nina Schoetterl-Glausch
@ 2023-01-16 18:24 ` Pierre Morel
1 sibling, 0 replies; 80+ messages in thread
From: Pierre Morel @ 2023-01-16 18:24 UTC (permalink / raw)
To: Thomas Huth, qemu-s390x
Cc: qemu-devel, borntraeger, pasic, richard.henderson, david, cohuck,
mst, pbonzini, kvm, ehabkost, marcel.apfelbaum, eblake, armbru,
seiden, nrb, scgl, frankja, berrange, clg
On 1/10/23 14:00, Thomas Huth wrote:
> On 05/01/2023 15.53, Pierre Morel wrote:
>> The topology information are attributes of the CPU and are
>> specified during the CPU device creation.
>>
>> On hot plug, we gather the topology information on the core,
>> creates a list of topology entries, each entry contains a single
>> core mask of each core with identical topology and finaly we
>> orders the list in topological order.
>> The topological order is, from higher to lower priority:
>> - physical topology
>> - drawer
>> - book
>> - socket
>> - core origin, offset in 64bit increment from core 0.
>> - modifier attributes
>> - CPU type
>> - polarization entitlement
>> - dedication
>>
>> The possibility to insert a CPU in a mask is dependent on the
>> number of cores allowed in a socket, a book or a drawer, the
>> checking is done during the hot plug of the CPU to have an
>> immediate answer.
>>
>> If the complete topology is not specified, the core is added
>> in the physical topology based on its core ID and it gets
>> defaults values for the modifier attributes.
>>
>> This way, starting QEMU without specifying the topology can
>> still get some adventage of the CPU topology.
>
> s/adventage/advantage/
>
>> Signed-off-by: Pierre Morel <pmorel@linux.ibm.com>
>> ---
>> include/hw/s390x/cpu-topology.h | 48 ++++++
>> hw/s390x/cpu-topology.c | 293 ++++++++++++++++++++++++++++++++
>> hw/s390x/s390-virtio-ccw.c | 10 ++
>> hw/s390x/meson.build | 1 +
>> 4 files changed, 352 insertions(+)
>> create mode 100644 hw/s390x/cpu-topology.c
>>
>> diff --git a/include/hw/s390x/cpu-topology.h
>> b/include/hw/s390x/cpu-topology.h
>> index d945b57fc3..b3fd752d8d 100644
>> --- a/include/hw/s390x/cpu-topology.h
>> +++ b/include/hw/s390x/cpu-topology.h
>> @@ -10,7 +10,11 @@
>> #ifndef HW_S390X_CPU_TOPOLOGY_H
>> #define HW_S390X_CPU_TOPOLOGY_H
>> +#include "qemu/queue.h"
>> +#include "hw/boards.h"
>> +
>> #define S390_TOPOLOGY_CPU_IFL 0x03
>> +#define S390_TOPOLOGY_MAX_ORIGIN ((63 + S390_MAX_CPUS) / 64)
>> #define S390_TOPOLOGY_POLARITY_HORIZONTAL 0x00
>> #define S390_TOPOLOGY_POLARITY_VERTICAL_LOW 0x01
>> @@ -20,4 +24,48 @@
>> #define S390_TOPOLOGY_SHARED 0x00
>> #define S390_TOPOLOGY_DEDICATED 0x01
>> +typedef union s390_topology_id {
>> + uint64_t id;
>> + struct {
>> + uint64_t level_6:8; /* byte 0 BE */
>> + uint64_t level_5:8; /* byte 1 BE */
>> + uint64_t drawer:8; /* byte 2 BE */
>> + uint64_t book:8; /* byte 3 BE */
>> + uint64_t socket:8; /* byte 4 BE */
>> + uint64_t rsrv:5;
>> + uint64_t d:1;
>> + uint64_t p:2; /* byte 5 BE */
>> + uint64_t type:8; /* byte 6 BE */
>> + uint64_t origin:2;
>> + uint64_t core:6; /* byte 7 BE */
>> + };
>> +} s390_topology_id;
>
> Bitmasks are OK for code that will definitely only ever work with KVM
> ... but this will certainly fail completely if we ever try to get it
> running with TCG later. Do we care? ... if so, you should certainly
> avoid a bitfield here. Especially since most of the fields are 8-bit
> anyway and could easily be represented by a "uint8_t" variable.
> Otherwise, just ignore my comment.
The goal of using a bit mask here is not to use it with KVM but to have
an easy way to order the TLE using the natural order of the placement of
the fields in the uint64_t
However, if I remove the two unused levels 5 and 6 I can use uint8_t for
all the entries.
I doubt we use the levels 5 and 6 in a short future.
So I switch on 1 uint8_t for each entry.
...
>> diff --git a/hw/s390x/cpu-topology.c b/hw/s390x/cpu-topology.c
>> new file mode 100644
>> index 0000000000..438055c612
>> --- /dev/null
>> +++ b/hw/s390x/cpu-topology.c
>> @@ -0,0 +1,293 @@
>> +/*
>> + * CPU Topology
>> + *
>> + * Copyright IBM Corp. 2022
>
> Want to update to 2023 now?
>
>> + * Author(s): Pierre Morel <pmorel@linux.ibm.com>
>> +
>> + * This work is licensed under the terms of the GNU GPL, version 2 or
>> (at
>> + * your option) any later version. See the COPYING file in the top-level
>> + * directory.
>> + */
>> +
>> +#include "qemu/osdep.h"
>> +#include "qapi/error.h"
>> +#include "qemu/error-report.h"
>> +#include "hw/qdev-properties.h"
>> +#include "hw/boards.h"
>> +#include "qemu/typedefs.h"
>> +#include "target/s390x/cpu.h"
>> +#include "hw/s390x/s390-virtio-ccw.h"
>> +#include "hw/s390x/cpu-topology.h"
>> +
>> +/*
>> + * s390_topology is used to keep the topology information.
>> + * .list: queue the topology entries inside which
>> + * we keep the information on the CPU topology.
>> + *
>> + * .smp: keeps track of the machine topology.
>> + *
>> + * .socket: tracks information on the count of cores per socket.
>> + *
>> + */
>> +S390Topology s390_topology = {
>> + .list = QTAILQ_HEAD_INITIALIZER(s390_topology.list),
>> + .sockets = NULL, /* will be initialized after the cpu model is
>> realized */
>> +};
>> +
>> +/**
>> + * s390_socket_nb:
>> + * @id: s390_topology_id
>> + *
>> + * Returns the socket number used inside the socket array.
>> + */
>> +static int s390_socket_nb(s390_topology_id id)
>> +{
>> + return (id.socket + 1) * (id.book + 1) * (id.drawer + 1); > +}
> I think there might be an off-by-one error in here - you likely need a
> "- 1" at the very end.
>
> For example, assume that we have one socket, one book and one drawer, so
> id.socket, id.book and id.drawer would all be 0. The function then
> returns 1 ...
hum, I fear it is even more false than that but thanks for pointing this
error.
/o\
return (id.drawer * s390_topology.smp.books + id.book) *
s390_topology.smp.sockets + id.socket;
>
>> +static void s390_topology_init(MachineState *ms)
>> +{
>> + CpuTopology *smp = &ms->smp;
>> +
>> + s390_topology.smp = smp;
>> + if (!s390_topology.sockets) {
>> + s390_topology.sockets = g_new0(uint8_t, smp->sockets *
>> + smp->books * smp->drawers);
>
> ... but here you only allocated one byte. So you later access
> s390_topology.sockets[s390_socket_nb(id)], i.e. s390_topology.sockets[1]
> which is out of bounds.
Yes, thanks.
Regards,
Pierre
--
Pierre Morel
IBM Lab Boeblingen
^ permalink raw reply [flat|nested] 80+ messages in thread
* Re: [PATCH v14 06/11] s390x/cpu topology: interception of PTF instruction
2023-01-05 14:53 ` [PATCH v14 06/11] s390x/cpu topology: interception of PTF instruction Pierre Morel
@ 2023-01-16 18:24 ` Nina Schoetterl-Glausch
2023-01-18 9:54 ` Pierre Morel
2023-01-20 14:32 ` Pierre Morel
0 siblings, 2 replies; 80+ messages in thread
From: Nina Schoetterl-Glausch @ 2023-01-16 18:24 UTC (permalink / raw)
To: Pierre Morel, qemu-s390x
Cc: qemu-devel, borntraeger, pasic, richard.henderson, david, thuth,
cohuck, mst, pbonzini, kvm, ehabkost, marcel.apfelbaum, eblake,
armbru, seiden, nrb, frankja, berrange, clg
On Thu, 2023-01-05 at 15:53 +0100, Pierre Morel wrote:
> When the host supports the CPU topology facility, the PTF
> instruction with function code 2 is interpreted by the SIE,
> provided that the userland hypervizor activates the interpretation
> by using the KVM_CAP_S390_CPU_TOPOLOGY KVM extension.
>
> The PTF instructions with function code 0 and 1 are intercepted
> and must be emulated by the userland hypervizor.
>
> During RESET all CPU of the configuration are placed in
> horizontal polarity.
>
> Signed-off-by: Pierre Morel <pmorel@linux.ibm.com>
> ---
> include/hw/s390x/cpu-topology.h | 3 +
> include/hw/s390x/s390-virtio-ccw.h | 6 ++
> target/s390x/cpu.h | 1 +
> hw/s390x/cpu-topology.c | 92 ++++++++++++++++++++++++++++++
> target/s390x/cpu-sysemu.c | 16 ++++++
> target/s390x/kvm/kvm.c | 11 ++++
> 6 files changed, 129 insertions(+)
>
> diff --git a/include/hw/s390x/cpu-topology.h b/include/hw/s390x/cpu-topology.h
> index 9571aa70e5..33e23d78b9 100644
> --- a/include/hw/s390x/cpu-topology.h
> +++ b/include/hw/s390x/cpu-topology.h
> @@ -55,11 +55,13 @@ typedef struct S390Topology {
> QTAILQ_HEAD(, S390TopologyEntry) list;
> uint8_t *sockets;
> CpuTopology *smp;
> + int polarity;
> } S390Topology;
>
> #ifdef CONFIG_KVM
> bool s390_has_topology(void);
> void s390_topology_set_cpu(MachineState *ms, S390CPU *cpu, Error **errp);
> +void s390_topology_set_polarity(int polarity);
> #else
> static inline bool s390_has_topology(void)
> {
> @@ -68,6 +70,7 @@ static inline bool s390_has_topology(void)
> static inline void s390_topology_set_cpu(MachineState *ms,
> S390CPU *cpu,
> Error **errp) {}
> +static inline void s390_topology_set_polarity(int polarity) {}
> #endif
> extern S390Topology s390_topology;
>
> diff --git a/include/hw/s390x/s390-virtio-ccw.h b/include/hw/s390x/s390-virtio-ccw.h
> index 9bba21a916..c1d46e78af 100644
> --- a/include/hw/s390x/s390-virtio-ccw.h
> +++ b/include/hw/s390x/s390-virtio-ccw.h
> @@ -30,6 +30,12 @@ struct S390CcwMachineState {
> uint8_t loadparm[8];
> };
>
> +#define S390_PTF_REASON_NONE (0x00 << 8)
> +#define S390_PTF_REASON_DONE (0x01 << 8)
> +#define S390_PTF_REASON_BUSY (0x02 << 8)
> +#define S390_TOPO_FC_MASK 0xffUL
> +void s390_handle_ptf(S390CPU *cpu, uint8_t r1, uintptr_t ra);
> +
> struct S390CcwMachineClass {
> /*< private >*/
> MachineClass parent_class;
> diff --git a/target/s390x/cpu.h b/target/s390x/cpu.h
> index 01ade07009..5da4041576 100644
> --- a/target/s390x/cpu.h
> +++ b/target/s390x/cpu.h
> @@ -864,6 +864,7 @@ void s390_do_cpu_set_diag318(CPUState *cs, run_on_cpu_data arg);
> int s390_assign_subch_ioeventfd(EventNotifier *notifier, uint32_t sch_id,
> int vq, bool assign);
> void s390_cpu_topology_reset(void);
> +void s390_cpu_topology_set(void);
I don't like this name much, it's nondescript.
s390_cpu_topology_set_modified ?
> #ifndef CONFIG_USER_ONLY
> unsigned int s390_cpu_set_state(uint8_t cpu_state, S390CPU *cpu);
> #else
> diff --git a/hw/s390x/cpu-topology.c b/hw/s390x/cpu-topology.c
> index 438055c612..e6b4692581 100644
> --- a/hw/s390x/cpu-topology.c
> +++ b/hw/s390x/cpu-topology.c
> @@ -97,6 +97,98 @@ static s390_topology_id s390_topology_from_cpu(S390CPU *cpu)
> }
>
> /**
> + * s390_topology_set_polarity
> + * @polarity: horizontal or vertical
> + *
> + * Changes the polarity of all the CPU in the configuration.
> + *
> + * If the dedicated CPU modifier attribute is set a vertical
> + * polarization is always high (Architecture).
> + * Otherwise we decide to set it as medium.
> + *
> + * Once done, advertise a topology change.
> + */
> +void s390_topology_set_polarity(int polarity)
I don't like that this function ignores what kind of vertical polarization is passed,
it's confusing.
That seems like a further reason to split horizontal/vertical from the entitlement.
> +{
> + S390TopologyEntry *entry;
I also expected this function to set s390_topology.polarization, but it doesn't.
> +
> + QTAILQ_FOREACH(entry, &s390_topology.list, next) {
> + if (polarity == S390_TOPOLOGY_POLARITY_HORIZONTAL) {
> + entry->id.p = polarity;
> + } else {
> + if (entry->id.d) {
> + entry->id.p = S390_TOPOLOGY_POLARITY_VERTICAL_HIGH;
> + } else {
> + entry->id.p = S390_TOPOLOGY_POLARITY_VERTICAL_MEDIUM;
> + }
> + }
> + }
> + s390_cpu_topology_set();
> +}
> +
> +/*
> + * s390_handle_ptf:
> + *
> + * @register 1: contains the function code
> + *
> + * Function codes 0 and 1 handle the CPU polarization.
> + * We assume an horizontal topology, the only one supported currently
> + * by Linux, consequently we answer to function code 0, requesting
> + * horizontal polarization that it is already the current polarization
> + * and reject vertical polarization request without further explanation.
This comment is outdated, right? Same for those in the function body.
> + *
> + * Function code 2 is handling topology changes and is interpreted
> + * by the SIE.
> + */
> +void s390_handle_ptf(S390CPU *cpu, uint8_t r1, uintptr_t ra)
> +{
> + CPUS390XState *env = &cpu->env;
> + uint64_t reg = env->regs[r1];
> + uint8_t fc = reg & S390_TOPO_FC_MASK;
> +
> + if (!s390_has_feat(S390_FEAT_CONFIGURATION_TOPOLOGY)) {
> + s390_program_interrupt(env, PGM_OPERATION, ra);
> + return;
> + }
> +
> + if (env->psw.mask & PSW_MASK_PSTATE) {
> + s390_program_interrupt(env, PGM_PRIVILEGED, ra);
> + return;
> + }
> +
> + if (reg & ~S390_TOPO_FC_MASK) {
> + s390_program_interrupt(env, PGM_SPECIFICATION, ra);
> + return;
> + }
> +
> + switch (fc) {
> + case 0: /* Horizontal polarization is already set */
> + if (s390_topology.polarity == S390_TOPOLOGY_POLARITY_HORIZONTAL) {
> + env->regs[r1] |= S390_PTF_REASON_DONE;
> + setcc(cpu, 2);
> + } else {
> + s390_topology_set_polarity(S390_TOPOLOGY_POLARITY_HORIZONTAL);
> + s390_topology.polarity = S390_TOPOLOGY_POLARITY_HORIZONTAL;
> + setcc(cpu, 0);
> + }
> + break;
> + case 1: /* Vertical polarization is not supported */
> + if (s390_topology.polarity != S390_TOPOLOGY_POLARITY_HORIZONTAL) {
> + env->regs[r1] |= S390_PTF_REASON_DONE;
> + setcc(cpu, 2);
> + } else {
> + s390_topology_set_polarity(S390_TOPOLOGY_POLARITY_VERTICAL_LOW);
This is why I said it's confusing, nothing gets set to LOW.
> + s390_topology.polarity = S390_TOPOLOGY_POLARITY_VERTICAL_LOW;
Why LOW here?
> + setcc(cpu, 0);
> + }
> + break;
> + default:
> + /* Note that fc == 2 is interpreted by the SIE */
> + s390_program_interrupt(env, PGM_SPECIFICATION, ra);
> + }
You can simplify this by doing:
int new_polarity;
switch (fc) {
case 0:
new_polarity = S390_TOPOLOGY_POLARITY_HORIZONTAL;
break;
case 1:
new_polarity = S390_TOPOLOGY_POLARITY_VERTICAL_?;
break;
default:
/* Note that fc == 2 is interpreted by the SIE */
s390_program_interrupt(env, PGM_SPECIFICATION, ra);
return;
}
if same polarity:
rc done, rejected
else
set polarity, initiated
Might be a good idea to turn the polarity values into an enum.
> +}
[...]
^ permalink raw reply [flat|nested] 80+ messages in thread
* Re: [PATCH v14 01/11] s390x/cpu topology: adding s390 specificities to CPU topology
2023-01-16 17:28 ` Pierre Morel
@ 2023-01-16 20:34 ` Nina Schoetterl-Glausch
2023-01-17 9:49 ` Pierre Morel
2023-01-17 7:22 ` Thomas Huth
1 sibling, 1 reply; 80+ messages in thread
From: Nina Schoetterl-Glausch @ 2023-01-16 20:34 UTC (permalink / raw)
To: Pierre Morel, qemu-s390x
Cc: qemu-devel, borntraeger, pasic, richard.henderson, david, thuth,
cohuck, mst, pbonzini, kvm, ehabkost, marcel.apfelbaum, eblake,
armbru, seiden, nrb, frankja, berrange, clg
On Mon, 2023-01-16 at 18:28 +0100, Pierre Morel wrote:
>
> On 1/13/23 17:58, Nina Schoetterl-Glausch wrote:
> > On Thu, 2023-01-05 at 15:53 +0100, Pierre Morel wrote:
> > > S390 adds two new SMP levels, drawers and books to the CPU
> > > topology.
> > > The S390 CPU have specific toplogy features like dedication
> > > and polarity to give to the guest indications on the host
> > > vCPUs scheduling and help the guest take the best decisions
> > > on the scheduling of threads on the vCPUs.
> > >
> > > Let us provide the SMP properties with books and drawers levels
> > > and S390 CPU with dedication and polarity,
> > >
> > > Signed-off-by: Pierre Morel <pmorel@linux.ibm.com>
> > > ---
> > > qapi/machine.json | 14 ++++++++--
> > > include/hw/boards.h | 10 ++++++-
> > > include/hw/s390x/cpu-topology.h | 23 ++++++++++++++++
> > > target/s390x/cpu.h | 6 +++++
> > > hw/core/machine-smp.c | 48 ++++++++++++++++++++++++++++-----
> > > hw/core/machine.c | 4 +++
> > > hw/s390x/s390-virtio-ccw.c | 2 ++
> > > softmmu/vl.c | 6 +++++
> > > target/s390x/cpu.c | 10 +++++++
> > > qemu-options.hx | 6 +++--
> > > 10 files changed, 117 insertions(+), 12 deletions(-)
> > > create mode 100644 include/hw/s390x/cpu-topology.h
> > >
> > [...]
> >
> > > diff --git a/target/s390x/cpu.h b/target/s390x/cpu.h
> > > index 7d6d01325b..39ea63a416 100644
> > > --- a/target/s390x/cpu.h
> > > +++ b/target/s390x/cpu.h
> > > @@ -131,6 +131,12 @@ struct CPUArchState {
> > >
> > > #if !defined(CONFIG_USER_ONLY)
> > > uint32_t core_id; /* PoP "CPU address", same as cpu_index */
> > > + int32_t socket_id;
> > > + int32_t book_id;
> > > + int32_t drawer_id;
> > > + int32_t dedicated;
> > > + int32_t polarity;
> >
> > If I understood the architecture correctly, the polarity is a property of the configuration,
> > not the cpus. So this should be vertical_entitlement, and there should be a machine (?) property
> > specifying if the polarity is horizontal or vertical.
>
> You are right, considering PTF only, the documentation says PTF([01])
> does the following:
>
> "... a process is initiated to place all CPUs in the configuration into
> the polarization specified by the function code, ..."
>
> So on one side the polarization property is explicitly set on the CPU,
> and on the other side all CPU are supposed to be in the same
> polarization state.
I'm worried about STSI showing both horizontal and vertical CPUs at the same time.
I don't know if this is allowed.
If it is not, you need a way to switch between those atomically, which is harder
if every CPU has this property.
>
> So yes we can make the horizontal/vertical a machine property.
> However, we do not need to set this tunable as the documentation says
> that the machine always start with horizontal polarization.
>
> On the other hand the documentation mixes a lot vertical with different
> entitlement and horizontal polarization, for TLE order and slacks so I
> prefer to keep the complete description of the polarization as CPU
> properties in case we miss something.
>
> PTF([01]) are no performance bottle neck and the number of CPU is likely
> to be small, even a maximum of 248 is possible KVM warns above 16 CPU so
> the loop for setting all CPU inside PTF interception is not very
> problematic I think.
Yeah, I'm not worried about that.
>
> Doing like you say should simplify PTF interception (no loop) but
> complicates (some more if/else) TLE handling and QMP information display
> on CPU.
> So I will have a look at the implications and answer again on this.
>
> Thanks,
>
> Regards,
> Pierre
>
^ permalink raw reply [flat|nested] 80+ messages in thread
* Re: [PATCH v14 08/11] qapi/s390/cpu topology: change-topology monitor command
2023-01-05 14:53 ` Pierre Morel
` (2 preceding siblings ...)
(?)
@ 2023-01-16 21:09 ` Nina Schoetterl-Glausch
2023-01-17 7:30 ` Thomas Huth
2023-01-18 14:06 ` Pierre Morel
-1 siblings, 2 replies; 80+ messages in thread
From: Nina Schoetterl-Glausch @ 2023-01-16 21:09 UTC (permalink / raw)
To: Pierre Morel, qemu-s390x
Cc: qemu-devel, borntraeger, pasic, richard.henderson, david, thuth,
cohuck, mst, pbonzini, kvm, ehabkost, marcel.apfelbaum, eblake,
armbru, seiden, nrb, frankja, berrange, clg
On Thu, 2023-01-05 at 15:53 +0100, Pierre Morel wrote:
> The modification of the CPU attributes are done through a monitor
> commands.
>
> It allows to move the core inside the topology tree to optimise
> the cache usage in the case the host's hypervizor previously
> moved the CPU.
>
> The same command allows to modifiy the CPU attributes modifiers
> like polarization entitlement and the dedicated attribute to notify
> the guest if the host admin modified scheduling or dedication of a vCPU.
>
> With this knowledge the guest has the possibility to optimize the
> usage of the vCPUs.
>
> Signed-off-by: Pierre Morel <pmorel@linux.ibm.com>
> ---
> qapi/machine-target.json | 29 ++++++++
> include/monitor/hmp.h | 1 +
> hw/s390x/cpu-topology.c | 141 +++++++++++++++++++++++++++++++++++++++
> hmp-commands.hx | 16 +++++
> 4 files changed, 187 insertions(+)
>
> diff --git a/qapi/machine-target.json b/qapi/machine-target.json
> index 2e267fa458..75b0aa254d 100644
> --- a/qapi/machine-target.json
> +++ b/qapi/machine-target.json
> @@ -342,3 +342,32 @@
> 'TARGET_S390X',
> 'TARGET_MIPS',
> 'TARGET_LOONGARCH64' ] } }
> +
> +##
> +# @change-topology:
> +#
> +# @core: the vCPU ID to be moved
> +# @socket: the destination socket where to move the vCPU
> +# @book: the destination book where to move the vCPU
> +# @drawer: the destination drawer where to move the vCPU
> +# @polarity: optional polarity, default is last polarity set by the guest
> +# @dedicated: optional, if the vCPU is dedicated to a real CPU
> +#
> +# Modifies the topology by moving the CPU inside the topology
> +# tree or by changing a modifier attribute of a CPU.
> +#
> +# Returns: Nothing on success, the reason on failure.
> +#
> +# Since: <next qemu stable release, eg. 1.0>
> +##
> +{ 'command': 'change-topology',
> + 'data': {
> + 'core': 'int',
> + 'socket': 'int',
> + 'book': 'int',
> + 'drawer': 'int',
> + '*polarity': 'int',
> + '*dedicated': 'bool'
> + },
> + 'if': { 'all': [ 'TARGET_S390X', 'CONFIG_KVM' ] }
> +}
> diff --git a/include/monitor/hmp.h b/include/monitor/hmp.h
> index 27f86399f7..15c36bf549 100644
> --- a/include/monitor/hmp.h
> +++ b/include/monitor/hmp.h
> @@ -144,5 +144,6 @@ void hmp_human_readable_text_helper(Monitor *mon,
> HumanReadableText *(*qmp_handler)(Error **));
> void hmp_info_stats(Monitor *mon, const QDict *qdict);
> void hmp_pcie_aer_inject_error(Monitor *mon, const QDict *qdict);
> +void hmp_change_topology(Monitor *mon, const QDict *qdict);
>
> #endif
> diff --git a/hw/s390x/cpu-topology.c b/hw/s390x/cpu-topology.c
> index b69955a1cd..0faffe657e 100644
> --- a/hw/s390x/cpu-topology.c
> +++ b/hw/s390x/cpu-topology.c
> @@ -18,6 +18,10 @@
> #include "target/s390x/cpu.h"
> #include "hw/s390x/s390-virtio-ccw.h"
> #include "hw/s390x/cpu-topology.h"
> +#include "qapi/qapi-commands-machine-target.h"
> +#include "qapi/qmp/qdict.h"
> +#include "monitor/hmp.h"
> +#include "monitor/monitor.h"
>
> /*
> * s390_topology is used to keep the topology information.
> @@ -203,6 +207,21 @@ static void s390_topology_set_entry(S390TopologyEntry *entry,
> s390_topology.sockets[s390_socket_nb(id)]++;
> }
>
> +/**
> + * s390_topology_clear_entry:
> + * @entry: Topology entry to setup
> + * @id: topology id to use for the setup
> + *
> + * Clear the core bit inside the topology mask and
> + * decrements the number of cores for the socket.
> + */
> +static void s390_topology_clear_entry(S390TopologyEntry *entry,
> + s390_topology_id id)
> +{
> + clear_bit(63 - id.core, &entry->mask);
This doesn't take the origin into account.
> + s390_topology.sockets[s390_socket_nb(id)]--;
I suppose this function cannot run concurrently, so the same CPU doesn't get removed twice.
> +}
> +
> /**
> * s390_topology_new_entry:
> * @id: s390_topology_id to add
> @@ -383,3 +402,125 @@ void s390_topology_set_cpu(MachineState *ms, S390CPU *cpu, Error **errp)
>
> s390_topology_insert(id);
> }
> +
> +/*
> + * qmp and hmp implementations
> + */
> +
> +static S390TopologyEntry *s390_topology_core_to_entry(int core)
> +{
> + S390TopologyEntry *entry;
> +
> + QTAILQ_FOREACH(entry, &s390_topology.list, next) {
> + if (entry->mask & (1UL << (63 - core))) {
origin here also.
> + return entry;
> + }
> + }
> + return NULL;
This should not return NULL unless the core id is invalid.
Might be better to validate that somewhere else.
> +}
> +
> +static void s390_change_topology(Error **errp, int64_t core, int64_t socket,
> + int64_t book, int64_t drawer,
> + int64_t polarity, bool dedicated)
> +{
> + S390TopologyEntry *entry;
> + s390_topology_id new_id;
> + s390_topology_id old_id;
> + Error *local_error = NULL;
I think you could use ERRP_GUARD here also.
> +
> + /* Get the old entry */
> + entry = s390_topology_core_to_entry(core);
> + if (!entry) {
> + error_setg(errp, "No core %ld", core);
> + return;
> + }
> +
> + /* Compute old topology id */
> + old_id = entry->id;
> + old_id.core = core;
> +
> + /* Compute new topology id */
> + new_id = entry->id;
> + new_id.core = core;
> + new_id.socket = socket;
> + new_id.book = book;
> + new_id.drawer = drawer;
> + new_id.p = polarity;
> + new_id.d = dedicated;
> + new_id.type = S390_TOPOLOGY_CPU_IFL;
> +
> + /* Same topology entry, nothing to do */
> + if (entry->id.id == new_id.id) {
> + return;
> + }
> +
> + /* Check for space on the socket if ids are different */
> + if ((s390_socket_nb(old_id) != s390_socket_nb(new_id)) &&
> + (s390_topology.sockets[s390_socket_nb(new_id)] >=
> + s390_topology.smp->sockets)) {
> + error_setg(errp, "No more space on this socket");
> + return;
> + }
> +
> + /* Verify the new topology */
> + s390_topology_check(&local_error, new_id);
> + if (local_error) {
> + error_propagate(errp, local_error);
> + return;
> + }
> +
> + /* Clear the old topology */
> + s390_topology_clear_entry(entry, old_id);
> +
> + /* Insert the new topology */
> + s390_topology_insert(new_id);
> +
> + /* Remove unused entry */
> + if (!entry->mask) {
> + QTAILQ_REMOVE(&s390_topology.list, entry, next);
> + g_free(entry);
> + }
> +
> + /* Advertise the topology change */
> + s390_cpu_topology_set();
> +}
> +
> +void qmp_change_topology(int64_t core, int64_t socket,
> + int64_t book, int64_t drawer,
> + bool has_polarity, int64_t polarity,
> + bool has_dedicated, bool dedicated,
> + Error **errp)
> +{
> + Error *local_err = NULL;
> +
> + if (!s390_has_topology()) {
> + error_setg(&local_err, "This machine doesn't support topology");
> + return;
> + }
Do you really want to ignore has_polarity and has_dedicated here?
What happens in this case?
> + s390_change_topology(&local_err, core, socket, book, drawer,
> + polarity, dedicated);
> + if (local_err) {
> + error_propagate(errp, local_err);
> + }
> +}
> +
> +void hmp_change_topology(Monitor *mon, const QDict *qdict)
> +{
> + const int64_t core = qdict_get_int(qdict, "core");
> + const int64_t socket = qdict_get_int(qdict, "socket");
> + const int64_t book = qdict_get_int(qdict, "book");
> + const int64_t drawer = qdict_get_int(qdict, "drawer");
> + bool has_polarity = qdict_haskey(qdict, "polarity");
> + const int64_t polarity = qdict_get_try_int(qdict, "polarity", 0);
> + bool has_dedicated = qdict_haskey(qdict, "dedicated");
> + const bool dedicated = qdict_get_try_bool(qdict, "dedicated", false);
> + Error *local_err = NULL;
> +
> + qmp_change_topology(core, socket, book, drawer,
> + has_polarity, polarity,
> + has_dedicated, dedicated,
> + &local_err);
> + if (hmp_handle_error(mon, local_err)) {
> + return;
> + }
> +}
> diff --git a/hmp-commands.hx b/hmp-commands.hx
> index 673e39a697..a617cfed0d 100644
> --- a/hmp-commands.hx
> +++ b/hmp-commands.hx
> @@ -1815,3 +1815,19 @@ SRST
> Dump the FDT in dtb format to *filename*.
> ERST
> #endif
> +
> +#if defined(TARGET_S390X) && defined(CONFIG_KVM)
> + {
> + .name = "change-topology",
> + .args_type = "core:l,socket:l,book:l,drawer:l,polarity:l?,dedicated:b?",
> + .params = "core socket book drawer [polarity] [dedicated]",
> + .help = "Move CPU 'core' to 'socket/book/drawer' "
> + "optionaly modifies polarity and dedication",
> + .cmd = hmp_change_topology,
> + },
> +
> +SRST
> +``change-topology`` *core* *socket* *book* *drawer* *polarity* *dedicated*
> + Moves the CPU *core* to *socket* *book* *drawer* with *polarity* *dedicated*.
> +ERST
> +#endif
^ permalink raw reply [flat|nested] 80+ messages in thread
* Re: [PATCH v14 01/11] s390x/cpu topology: adding s390 specificities to CPU topology
2023-01-16 17:28 ` Pierre Morel
2023-01-16 20:34 ` Nina Schoetterl-Glausch
@ 2023-01-17 7:22 ` Thomas Huth
1 sibling, 0 replies; 80+ messages in thread
From: Thomas Huth @ 2023-01-17 7:22 UTC (permalink / raw)
To: Pierre Morel, Nina Schoetterl-Glausch, qemu-s390x
Cc: qemu-devel, borntraeger, pasic, richard.henderson, david, cohuck,
mst, pbonzini, kvm, ehabkost, marcel.apfelbaum, eblake, armbru,
seiden, nrb, frankja, berrange, clg
On 16/01/2023 18.28, Pierre Morel wrote:
>
>
> On 1/13/23 17:58, Nina Schoetterl-Glausch wrote:
>> On Thu, 2023-01-05 at 15:53 +0100, Pierre Morel wrote:
>>> S390 adds two new SMP levels, drawers and books to the CPU
>>> topology.
>>> The S390 CPU have specific toplogy features like dedication
>>> and polarity to give to the guest indications on the host
>>> vCPUs scheduling and help the guest take the best decisions
>>> on the scheduling of threads on the vCPUs.
>>>
>>> Let us provide the SMP properties with books and drawers levels
>>> and S390 CPU with dedication and polarity,
>>>
>>> Signed-off-by: Pierre Morel <pmorel@linux.ibm.com>
>>> ---
...
> PTF([01]) are no performance bottle neck and the number of CPU is likely to
> be small, even a maximum of 248 is possible KVM warns above 16 CPU so the
> loop for setting all CPU inside PTF interception is not very problematic I
> think.
KVM warns if you try to use more than the number of physical CPUs that you
have, not at hard-coded 16 CPUs. So if you've got an LPAR with 248 CPUs,
it's perfectly fine to use also 248 CPUs for your guest.
Thomas
^ permalink raw reply [flat|nested] 80+ messages in thread
* Re: [PATCH v14 01/11] s390x/cpu topology: adding s390 specificities to CPU topology
2023-01-16 16:32 ` Pierre Morel
@ 2023-01-17 7:25 ` Thomas Huth
0 siblings, 0 replies; 80+ messages in thread
From: Thomas Huth @ 2023-01-17 7:25 UTC (permalink / raw)
To: Pierre Morel, qemu-s390x, borntraeger
Cc: qemu-devel, pasic, richard.henderson, david, cohuck, mst,
pbonzini, kvm, ehabkost, marcel.apfelbaum, eblake, armbru,
seiden, nrb, scgl, frankja, berrange, clg
On 16/01/2023 17.32, Pierre Morel wrote:
>
> On 1/10/23 12:37, Thomas Huth wrote:
...
>> Other question: Do we have "node-id"s on s390x? If not, is that similar to
>> books or drawers, i.e. just another word? If so, we should maybe rather
>> re-use "nodes" instead of introducing a new name for the same thing?
>
> We have theoretically nodes-id on s390x, it is the level 5 of the topology,
> above drawers.
> Currently it is not used in s390x topology, the maximum level returned to a
> LPAR host is 4.
> I suppose that it adds a possibility to link several s390x with a fast network.
Ok, thanks. So if nodes are indeed a concept on top of the other layers, and
we do not really support these on s390x yet, should we maybe forbid them on
s390x like we recently did with the "threads" in the recent machine types?
... to avoid that users try to use them with the wrong expectations?
Thomas
^ permalink raw reply [flat|nested] 80+ messages in thread
* Re: [PATCH v14 08/11] qapi/s390/cpu topology: change-topology monitor command
2023-01-16 21:09 ` Nina Schoetterl-Glausch
@ 2023-01-17 7:30 ` Thomas Huth
2023-01-17 13:31 ` Nina Schoetterl-Glausch
2023-01-18 14:06 ` Pierre Morel
1 sibling, 1 reply; 80+ messages in thread
From: Thomas Huth @ 2023-01-17 7:30 UTC (permalink / raw)
To: Nina Schoetterl-Glausch, Pierre Morel, qemu-s390x
Cc: qemu-devel, borntraeger, pasic, richard.henderson, david, cohuck,
mst, pbonzini, kvm, ehabkost, marcel.apfelbaum, eblake, armbru,
seiden, nrb, frankja, berrange, clg
On 16/01/2023 22.09, Nina Schoetterl-Glausch wrote:
> On Thu, 2023-01-05 at 15:53 +0100, Pierre Morel wrote:
>> The modification of the CPU attributes are done through a monitor
>> commands.
>>
>> It allows to move the core inside the topology tree to optimise
>> the cache usage in the case the host's hypervizor previously
>> moved the CPU.
>>
>> The same command allows to modifiy the CPU attributes modifiers
>> like polarization entitlement and the dedicated attribute to notify
>> the guest if the host admin modified scheduling or dedication of a vCPU.
>>
>> With this knowledge the guest has the possibility to optimize the
>> usage of the vCPUs.
>>
>> Signed-off-by: Pierre Morel <pmorel@linux.ibm.com>
>> ---
...
>> diff --git a/hw/s390x/cpu-topology.c b/hw/s390x/cpu-topology.c
>> index b69955a1cd..0faffe657e 100644
>> --- a/hw/s390x/cpu-topology.c
>> +++ b/hw/s390x/cpu-topology.c
>> @@ -18,6 +18,10 @@
>> #include "target/s390x/cpu.h"
>> #include "hw/s390x/s390-virtio-ccw.h"
>> #include "hw/s390x/cpu-topology.h"
>> +#include "qapi/qapi-commands-machine-target.h"
>> +#include "qapi/qmp/qdict.h"
>> +#include "monitor/hmp.h"
>> +#include "monitor/monitor.h"
>>
>> /*
>> * s390_topology is used to keep the topology information.
>> @@ -203,6 +207,21 @@ static void s390_topology_set_entry(S390TopologyEntry *entry,
>> s390_topology.sockets[s390_socket_nb(id)]++;
>> }
>>
>> +/**
>> + * s390_topology_clear_entry:
>> + * @entry: Topology entry to setup
>> + * @id: topology id to use for the setup
>> + *
>> + * Clear the core bit inside the topology mask and
>> + * decrements the number of cores for the socket.
>> + */
>> +static void s390_topology_clear_entry(S390TopologyEntry *entry,
>> + s390_topology_id id)
>> +{
>> + clear_bit(63 - id.core, &entry->mask);
>
> This doesn't take the origin into account.
>
>> + s390_topology.sockets[s390_socket_nb(id)]--;
>
> I suppose this function cannot run concurrently, so the same CPU doesn't get removed twice.
QEMU has the so-called BQL - the Big Qemu Lock. Instructions handlers are
normally called with the lock taken, see qemu_mutex_lock_iothread() in
target/s390x/kvm/kvm.c.
(if you feel unsure, you can add a "assert(qemu_mutex_iothread_locked())"
statement, but I think it is not necessary here)
Thomas
^ permalink raw reply [flat|nested] 80+ messages in thread
* Re: [PATCH v14 01/11] s390x/cpu topology: adding s390 specificities to CPU topology
2023-01-16 20:34 ` Nina Schoetterl-Glausch
@ 2023-01-17 9:49 ` Pierre Morel
0 siblings, 0 replies; 80+ messages in thread
From: Pierre Morel @ 2023-01-17 9:49 UTC (permalink / raw)
To: Nina Schoetterl-Glausch, qemu-s390x
Cc: qemu-devel, borntraeger, pasic, richard.henderson, david, thuth,
cohuck, mst, pbonzini, kvm, ehabkost, marcel.apfelbaum, eblake,
armbru, seiden, nrb, frankja, berrange, clg
On 1/16/23 21:34, Nina Schoetterl-Glausch wrote:
> On Mon, 2023-01-16 at 18:28 +0100, Pierre Morel wrote:
>>
>> On 1/13/23 17:58, Nina Schoetterl-Glausch wrote:
>>> On Thu, 2023-01-05 at 15:53 +0100, Pierre Morel wrote:
>>>> S390 adds two new SMP levels, drawers and books to the CPU
>>>> topology.
>>>> The S390 CPU have specific toplogy features like dedication
>>>> and polarity to give to the guest indications on the host
>>>> vCPUs scheduling and help the guest take the best decisions
>>>> on the scheduling of threads on the vCPUs.
>>>>
>>>> Let us provide the SMP properties with books and drawers levels
>>>> and S390 CPU with dedication and polarity,
>>>>
>>>> Signed-off-by: Pierre Morel <pmorel@linux.ibm.com>
>>>> ---
>>>> qapi/machine.json | 14 ++++++++--
>>>> include/hw/boards.h | 10 ++++++-
>>>> include/hw/s390x/cpu-topology.h | 23 ++++++++++++++++
>>>> target/s390x/cpu.h | 6 +++++
>>>> hw/core/machine-smp.c | 48 ++++++++++++++++++++++++++++-----
>>>> hw/core/machine.c | 4 +++
>>>> hw/s390x/s390-virtio-ccw.c | 2 ++
>>>> softmmu/vl.c | 6 +++++
>>>> target/s390x/cpu.c | 10 +++++++
>>>> qemu-options.hx | 6 +++--
>>>> 10 files changed, 117 insertions(+), 12 deletions(-)
>>>> create mode 100644 include/hw/s390x/cpu-topology.h
>>>>
>>> [...]
>>>
>>>> diff --git a/target/s390x/cpu.h b/target/s390x/cpu.h
>>>> index 7d6d01325b..39ea63a416 100644
>>>> --- a/target/s390x/cpu.h
>>>> +++ b/target/s390x/cpu.h
>>>> @@ -131,6 +131,12 @@ struct CPUArchState {
>>>>
>>>> #if !defined(CONFIG_USER_ONLY)
>>>> uint32_t core_id; /* PoP "CPU address", same as cpu_index */
>>>> + int32_t socket_id;
>>>> + int32_t book_id;
>>>> + int32_t drawer_id;
>>>> + int32_t dedicated;
>>>> + int32_t polarity;
>>>
>>> If I understood the architecture correctly, the polarity is a property of the configuration,
>>> not the cpus. So this should be vertical_entitlement, and there should be a machine (?) property
>>> specifying if the polarity is horizontal or vertical.
>>
>> You are right, considering PTF only, the documentation says PTF([01])
>> does the following:
>>
>> "... a process is initiated to place all CPUs in the configuration into
>> the polarization specified by the function code, ..."
>>
>> So on one side the polarization property is explicitly set on the CPU,
>> and on the other side all CPU are supposed to be in the same
>> polarization state.
>
> I'm worried about STSI showing both horizontal and vertical CPUs at the same time.
> I don't know if this is allowed.
> If it is not, you need a way to switch between those atomically, which is harder
> if every CPU has this property.
The documentation explicitly provides the order in which to show the TLE
when both vertical and horizontal TLE are display inside the SYSIB.
So it seems expected by the architecture.
I understand that it is because the documentation says that the
instruction finished before all the CPU did change polarity.
But I think you are right on the point that:
- It is only transitional
- it does not make sense for my point of view to have both at the same time.
In QEMU, as these are two interceptions we will always finish the PTF
interception before we start the STSI interception, so we will never see
this happen.
Question are:
1) Should we do better than the architecture?
2) If yes, are we sure it is better?
Regards,
Pierre
--
Pierre Morel
IBM Lab Boeblingen
^ permalink raw reply [flat|nested] 80+ messages in thread
* Re: [PATCH v14 08/11] qapi/s390/cpu topology: change-topology monitor command
2023-01-17 7:30 ` Thomas Huth
@ 2023-01-17 13:31 ` Nina Schoetterl-Glausch
2023-01-18 10:53 ` Thomas Huth
0 siblings, 1 reply; 80+ messages in thread
From: Nina Schoetterl-Glausch @ 2023-01-17 13:31 UTC (permalink / raw)
To: Thomas Huth, Pierre Morel, qemu-s390x
Cc: qemu-devel, borntraeger, pasic, richard.henderson, david, cohuck,
mst, pbonzini, kvm, ehabkost, marcel.apfelbaum, eblake, armbru,
seiden, nrb, frankja, berrange, clg
On Tue, 2023-01-17 at 08:30 +0100, Thomas Huth wrote:
> On 16/01/2023 22.09, Nina Schoetterl-Glausch wrote:
> > On Thu, 2023-01-05 at 15:53 +0100, Pierre Morel wrote:
> > > The modification of the CPU attributes are done through a monitor
> > > commands.
> > >
> > > It allows to move the core inside the topology tree to optimise
> > > the cache usage in the case the host's hypervizor previously
> > > moved the CPU.
> > >
> > > The same command allows to modifiy the CPU attributes modifiers
> > > like polarization entitlement and the dedicated attribute to notify
> > > the guest if the host admin modified scheduling or dedication of a vCPU.
> > >
> > > With this knowledge the guest has the possibility to optimize the
> > > usage of the vCPUs.
> > >
> > > Signed-off-by: Pierre Morel <pmorel@linux.ibm.com>
> > > ---
> ...
> > > diff --git a/hw/s390x/cpu-topology.c b/hw/s390x/cpu-topology.c
> > > index b69955a1cd..0faffe657e 100644
> > > --- a/hw/s390x/cpu-topology.c
> > > +++ b/hw/s390x/cpu-topology.c
> > > @@ -18,6 +18,10 @@
> > > #include "target/s390x/cpu.h"
> > > #include "hw/s390x/s390-virtio-ccw.h"
> > > #include "hw/s390x/cpu-topology.h"
> > > +#include "qapi/qapi-commands-machine-target.h"
> > > +#include "qapi/qmp/qdict.h"
> > > +#include "monitor/hmp.h"
> > > +#include "monitor/monitor.h"
> > >
> > > /*
> > > * s390_topology is used to keep the topology information.
> > > @@ -203,6 +207,21 @@ static void s390_topology_set_entry(S390TopologyEntry *entry,
> > > s390_topology.sockets[s390_socket_nb(id)]++;
> > > }
> > >
> > > +/**
> > > + * s390_topology_clear_entry:
> > > + * @entry: Topology entry to setup
> > > + * @id: topology id to use for the setup
> > > + *
> > > + * Clear the core bit inside the topology mask and
> > > + * decrements the number of cores for the socket.
> > > + */
> > > +static void s390_topology_clear_entry(S390TopologyEntry *entry,
> > > + s390_topology_id id)
> > > +{
> > > + clear_bit(63 - id.core, &entry->mask);
> >
> > This doesn't take the origin into account.
> >
> > > + s390_topology.sockets[s390_socket_nb(id)]--;
> >
> > I suppose this function cannot run concurrently, so the same CPU doesn't get removed twice.
>
> QEMU has the so-called BQL - the Big Qemu Lock. Instructions handlers are
> normally called with the lock taken, see qemu_mutex_lock_iothread() in
> target/s390x/kvm/kvm.c.
That is good to know, but is that the relevant lock here?
We don't want to concurrent qmp commands. I looked at the code and it's pretty complicated.
>
> (if you feel unsure, you can add a "assert(qemu_mutex_iothread_locked())"
> statement, but I think it is not necessary here)
>
> Thomas
>
^ permalink raw reply [flat|nested] 80+ messages in thread
* Re: [PATCH v14 02/11] s390x/cpu topology: add topology entries on CPU hotplug
2023-01-13 18:15 ` Nina Schoetterl-Glausch
@ 2023-01-17 13:55 ` Pierre Morel
2023-01-17 16:48 ` Nina Schoetterl-Glausch
0 siblings, 1 reply; 80+ messages in thread
From: Pierre Morel @ 2023-01-17 13:55 UTC (permalink / raw)
To: Nina Schoetterl-Glausch, qemu-s390x
Cc: qemu-devel, borntraeger, pasic, richard.henderson, david, thuth,
cohuck, mst, pbonzini, kvm, ehabkost, marcel.apfelbaum, eblake,
armbru, seiden, nrb, frankja, berrange, clg
On 1/13/23 19:15, Nina Schoetterl-Glausch wrote:
> On Thu, 2023-01-05 at 15:53 +0100, Pierre Morel wrote:
>> The topology information are attributes of the CPU and are
>> specified during the CPU device creation.
>>
>> On hot plug, we gather the topology information on the core,
>> creates a list of topology entries, each entry contains a single
>> core mask of each core with identical topology and finaly we
> s/finaly/finally/
thx
>> orders the list in topological order.
> s/orders/order/
thx
>> The topological order is, from higher to lower priority:
>> - physical topology
>> - drawer
>> - book
>> - socket
>> - core origin, offset in 64bit increment from core 0.
>> - modifier attributes
>> - CPU type
>> - polarization entitlement
>> - dedication
>>
>> The possibility to insert a CPU in a mask is dependent on the
>> number of cores allowed in a socket, a book or a drawer, the
>> checking is done during the hot plug of the CPU to have an
>> immediate answer.
>>
>> If the complete topology is not specified, the core is added
>> in the physical topology based on its core ID and it gets
>> defaults values for the modifier attributes.
>>
>> This way, starting QEMU without specifying the topology can
>> still get some adventage of the CPU topology.
>>
>> Signed-off-by: Pierre Morel <pmorel@linux.ibm.com>
>> ---
>> include/hw/s390x/cpu-topology.h | 48 ++++++
>> hw/s390x/cpu-topology.c | 293 ++++++++++++++++++++++++++++++++
>> hw/s390x/s390-virtio-ccw.c | 10 ++
>> hw/s390x/meson.build | 1 +
>> 4 files changed, 352 insertions(+)
>> create mode 100644 hw/s390x/cpu-topology.c
>>
>> diff --git a/include/hw/s390x/cpu-topology.h b/include/hw/s390x/cpu-topology.h
>> index d945b57fc3..b3fd752d8d 100644
>> --- a/include/hw/s390x/cpu-topology.h
>> +++ b/include/hw/s390x/cpu-topology.h
>> @@ -10,7 +10,11 @@
>> #ifndef HW_S390X_CPU_TOPOLOGY_H
>> #define HW_S390X_CPU_TOPOLOGY_H
>>
>> +#include "qemu/queue.h"
>> +#include "hw/boards.h"
>> +
>> #define S390_TOPOLOGY_CPU_IFL 0x03
>> +#define S390_TOPOLOGY_MAX_ORIGIN ((63 + S390_MAX_CPUS) / 64)
>>
>> #define S390_TOPOLOGY_POLARITY_HORIZONTAL 0x00
>> #define S390_TOPOLOGY_POLARITY_VERTICAL_LOW 0x01
>> @@ -20,4 +24,48 @@
>> #define S390_TOPOLOGY_SHARED 0x00
>> #define S390_TOPOLOGY_DEDICATED 0x01
>>
>> +typedef union s390_topology_id {
>> + uint64_t id;
>> + struct {
>> + uint64_t level_6:8; /* byte 0 BE */
>> + uint64_t level_5:8; /* byte 1 BE */
>> + uint64_t drawer:8; /* byte 2 BE */
>> + uint64_t book:8; /* byte 3 BE */
>> + uint64_t socket:8; /* byte 4 BE */
>> + uint64_t rsrv:5;
>> + uint64_t d:1;
>> + uint64_t p:2; /* byte 5 BE */
>> + uint64_t type:8; /* byte 6 BE */
>> + uint64_t origin:2;
>
> This is two bits because it's the core divided by 64, and we have 248 cores at most?
> Where is this set?
right, must be set so does the core offset in the mask.
>
>> + uint64_t core:6; /* byte 7 BE */
>> + };
>> +} s390_topology_id;
>
> This struct seems to do double duty, 1. it represents a cpu and 2. a topology entry.
> You also use it for sorting.
> I would suggest to just use a cpu object when referring to a specific cpu and
> put the relevant fields directly into the topology entry.
Yes, I can remove the core:6.
After Thomas comment I will change all the bit field for uint8_t.
I think we should not use the real topology entry here if we want to use
TGE in the future.
> You get rid of the bit field that way.
> You'd then need a comparison function for a cpu object and a topology entry.
> As long as that isn't the only type pair that shouldn't be too ugly.
>
>> +#define TOPO_CPU_MASK 0x000000000000003fUL
>> +
>> +typedef struct S390TopologyEntry {
>> + s390_topology_id id;
>> + QTAILQ_ENTRY(S390TopologyEntry) next;
>> + uint64_t mask;
>> +} S390TopologyEntry;
>> +
>> +typedef struct S390Topology {
>> + QTAILQ_HEAD(, S390TopologyEntry) list;
>> + uint8_t *sockets;
>> + CpuTopology *smp;
>> +} S390Topology;
>> +
>> +#ifdef CONFIG_KVM
>> +bool s390_has_topology(void);
>> +void s390_topology_set_cpu(MachineState *ms, S390CPU *cpu, Error **errp);
>> +#else
>> +static inline bool s390_has_topology(void)
>> +{
>> + return false;
>> +}
>> +static inline void s390_topology_set_cpu(MachineState *ms,
>> + S390CPU *cpu,
>> + Error **errp) {}
>> +#endif
>> +extern S390Topology s390_topology;
>> +
>> #endif
>> diff --git a/hw/s390x/cpu-topology.c b/hw/s390x/cpu-topology.c
>> new file mode 100644
>> index 0000000000..438055c612
>> --- /dev/null
>> +++ b/hw/s390x/cpu-topology.c
>> @@ -0,0 +1,293 @@
>> +/*
>> + * CPU Topology
>> + *
>> + * Copyright IBM Corp. 2022
>> + * Author(s): Pierre Morel <pmorel@linux.ibm.com>
>> +
>> + * This work is licensed under the terms of the GNU GPL, version 2 or (at
>> + * your option) any later version. See the COPYING file in the top-level
>> + * directory.
>> + */
>> +
>> +#include "qemu/osdep.h"
>> +#include "qapi/error.h"
>> +#include "qemu/error-report.h"
>> +#include "hw/qdev-properties.h"
>> +#include "hw/boards.h"
>> +#include "qemu/typedefs.h"
>> +#include "target/s390x/cpu.h"
>> +#include "hw/s390x/s390-virtio-ccw.h"
>> +#include "hw/s390x/cpu-topology.h"
>> +
>> +/*
>> + * s390_topology is used to keep the topology information.
>> + * .list: queue the topology entries inside which
>> + * we keep the information on the CPU topology.
>> + *
>> + * .smp: keeps track of the machine topology.
>> + *
>> + * .socket: tracks information on the count of cores per socket.
>> + *
>> + */
>> +S390Topology s390_topology = {
>> + .list = QTAILQ_HEAD_INITIALIZER(s390_topology.list),
>> + .sockets = NULL, /* will be initialized after the cpu model is realized */
>
> I guess you should do the same for .smp then also.
OK
>
>> +};
>> +
>> +/**
>> + * s390_socket_nb:
>> + * @id: s390_topology_id
>> + *
>> + * Returns the socket number used inside the socket array.
>> + */
>> +static int s390_socket_nb(s390_topology_id id)
>> +{
>> + return (id.socket + 1) * (id.book + 1) * (id.drawer + 1);
>
> This calculation doesn't make a whole lot of sense to me.
> It's symmetric with regards to the variables, so (s=0 b=1 d=1)
> will have the same result as (s=1 b=0 d=1).
> You want the "global" socket number right?
> So that would be (drawer * books_per_drawer + book) * sockets_per_book + socket.
yes, already changed
>
>> +}
>> +
>> +/**
>> + * s390_has_topology:
>> + *
>> + * Return value: if the topology is supported by the machine.
>> + */
>> +bool s390_has_topology(void)
>> +{
>> + return false;
>> +}
>> +
>> +/**
>> + * s390_topology_init:
>> + * @ms: the machine state where the machine topology is defined
>> + *
>> + * Keep track of the machine topology.
>> + * Allocate an array to keep the count of cores per socket.
>> + * The index of the array starts at socket 0 from book 0 and
>> + * drawer 0 up to the maximum allowed by the machine topology.
>> + */
>> +static void s390_topology_init(MachineState *ms)
>> +{
>> + CpuTopology *smp = &ms->smp;
>> +
>> + s390_topology.smp = smp;
>> + if (!s390_topology.sockets) {
>
> Is this function being called multiple times, or why the if?
> Use an assert instead?
I forgot to remove this test.
The caller already check this.
>
>> + s390_topology.sockets = g_new0(uint8_t, smp->sockets *
>> + smp->books * smp->drawers);
>> + }
>> +}
>> +
>> +/**
>> + * s390_topology_from_cpu:
>> + * @cpu: The S390CPU
>> + *
>> + * Initialize the topology id from the CPU environment.
>> + */
>> +static s390_topology_id s390_topology_from_cpu(S390CPU *cpu)
>> +{
>> + s390_topology_id topology_id;
>> +
>> + topology_id.core = cpu->env.core_id;
>> + topology_id.type = cpu->env.cpu_type;
>> + topology_id.p = cpu->env.polarity;
>> + topology_id.d = cpu->env.dedicated;
>> + topology_id.socket = cpu->env.socket_id;
>> + topology_id.book = cpu->env.book_id;
>> + topology_id.drawer = cpu->env.drawer_id;
>> +
>> + return topology_id;
>> +}
>> +
>> +/**
>> + * s390_topology_set_entry:
>> + * @entry: Topology entry to setup
>> + * @id: topology id to use for the setup
>> + *
>> + * Set the core bit inside the topology mask and
>> + * increments the number of cores for the socket.
>> + */
>> +static void s390_topology_set_entry(S390TopologyEntry *entry,
>
> Not sure if I like the name, what it does is to add a cpu to the entry.
s390_topology_add_cpu_to_entry() ?
>
>> + s390_topology_id id)
>> +{
>> + set_bit(63 - id.core, &entry->mask);
>
> You need to subtract the origin first or that might be negative.
yes, origin is not handled correctly
>
>> + s390_topology.sockets[s390_socket_nb(id)]++;
>> +}
>> +
>> +/**
>> + * s390_topology_new_entry:
>> + * @id: s390_topology_id to add
>> + *
>> + * Allocate a new entry and initialize it.
>> + *
>> + * returns the newly allocated entry.
>> + */
>> +static S390TopologyEntry *s390_topology_new_entry(s390_topology_id id)
>> +{
>> + S390TopologyEntry *entry;
>> +
>> + entry = g_malloc0(sizeof(S390TopologyEntry));
>> + entry->id.id = id.id & ~TOPO_CPU_MASK;
>> + s390_topology_set_entry(entry, id);
>> +
>> + return entry;
>> +}
>> +
>> +/**
>> + * s390_topology_insert:
>> + *
>> + * @id: s390_topology_id to insert.
>> + *
>> + * Parse the topology list to find if the entry already
>> + * exist and add the core in it.
>> + * If it does not exist, allocate a new entry and insert
>> + * it in the queue from lower id to greater id.
>> + */
>> +static void s390_topology_insert(s390_topology_id id)
>> +{
>> + S390TopologyEntry *entry;
>> + S390TopologyEntry *tmp = NULL;
>> + uint64_t new_id;
>> +
>> + new_id = id.id & ~TOPO_CPU_MASK;
>> +
>> + /* First CPU to add to an entry */
>> + if (QTAILQ_EMPTY(&s390_topology.list)) {
>> + entry = s390_topology_new_entry(id);
>> + QTAILQ_INSERT_HEAD(&s390_topology.list, entry, next);
>> + return;
>> + }
>> +
>> + QTAILQ_FOREACH(tmp, &s390_topology.list, next) {
>> + if (new_id == tmp->id.id) {
>> + s390_topology_set_entry(tmp, id);
>> + return;
>> + } else if (new_id < tmp->id.id) {
>> + entry = s390_topology_new_entry(id);
>> + QTAILQ_INSERT_BEFORE(tmp, entry, next);
>> + return;
>> + }
>> + }
>> +
>> + entry = s390_topology_new_entry(id);
>> + QTAILQ_INSERT_TAIL(&s390_topology.list, entry, next);
>
> Consider adding a sentinel entry "at infinity", then that whole code
> would simplify.
looks good, thanks.
>
>> +}
>> +
>> +/**
>> + * s390_topology_check:
>> + * @errp: Error pointer
>> + * id: s390_topology_id to be verified
>> + *
>> + * The function checks if the topology id fits inside the
>> + * system topology.
>> + */
>> +static void s390_topology_check(Error **errp, s390_topology_id id)
>> +{
>> + CpuTopology *smp = s390_topology.smp;
>> +
>> + if (id.socket > smp->sockets) {
>> + error_setg(errp, "Unavailable socket: %d", id.socket);
>> + return;
>> + }
>> + if (id.book > smp->books) {
>> + error_setg(errp, "Unavailable book: %d", id.book);
>> + return;
>> + }
>> + if (id.drawer > smp->drawers) {
>> + error_setg(errp, "Unavailable drawer: %d", id.drawer);
>> + return;
>> + }
>> + if (id.type != S390_TOPOLOGY_CPU_IFL) {
>> + error_setg(errp, "Unknown cpu type: %d", id.type);
>> + return;
>> + }
>> + /* Polarity and dedication can never be wrong */
>> +}
>> +
>> +/**
>> + * s390_topology_cpu_default:
>> + * @errp: Error pointer
>> + * @cpu: pointer to a S390CPU
>> + *
>> + * Setup the default topology for unset attributes.
>> + *
>> + * The function accept only all all default values or all set values
>> + * for the geometry topology.
>> + *
>> + * The function calculates the (drawer_id, book_id, socket_id)
>> + * topology by filling the cores starting from the first socket
>> + * (0, 0, 0) up to the last (smp->drawers, smp->books, smp->sockets).
>> + *
>> + */
>> +static void s390_topology_cpu_default(Error **errp, S390CPU *cpu)
>> +{
>> + CpuTopology *smp = s390_topology.smp;
>> + CPUS390XState *env = &cpu->env;
>> +
>> + /* All geometry topology attributes must be set or all unset */
>> + if ((env->socket_id < 0 || env->book_id < 0 || env->drawer_id < 0) &&
>> + (env->socket_id >= 0 || env->book_id >= 0 || env->drawer_id >= 0)) {
>> + error_setg(errp,
>> + "Please define all or none of the topology geometry attributes");
>> + return;
>> + }
>> +
>> + /* Check if one of the geometry topology is unset */
>> + if (env->socket_id < 0) {
>> + /* Calculate default geometry topology attributes */
>> + env->socket_id = (env->core_id / smp->cores) % smp->sockets;
>> + env->book_id = (env->core_id / (smp->sockets * smp->cores)) %
>> + smp->books;
>> + env->drawer_id = (env->core_id /
>> + (smp->books * smp->sockets * smp->cores)) %
>> + smp->drawers;
>> + }
>> +}
>> +
>> +/**
>> + * s390_topology_set_cpu:
>> + * @ms: MachineState used to initialize the topology structure on
>> + * first call.
>> + * @cpu: the new S390CPU to insert in the topology structure
>> + * @errp: the error pointer
>> + *
>> + * Called from CPU Hotplug to check and setup the CPU attributes
>> + * before to insert the CPU in the topology.
>> + */
>> +void s390_topology_set_cpu(MachineState *ms, S390CPU *cpu, Error **errp)
>> +{
>> + Error *local_error = NULL;
>
> Can't you just use ERRP_GUARD ?
I do not think it is necessary and I find it obfuscating.
So, should I?
>
>> + s390_topology_id id;
>> +
>> + /*
>> + * We do not want to initialize the topology if the cpu model
>> + * does not support topology consequently, we have to wait for
>
> ", consequently," I think. Could you do the initialization some where else,
> after you know what the cpu model is? Not that I object to doing it this way.
>
I did not find a better place, it must be done after the CPU model is
initialize and before the first CPU is created.
The cpu model is initialized during the early creation of the first cpu.
Any idea?
Thanks.
Regards,
Pierre
--
Pierre Morel
IBM Lab Boeblingen
^ permalink raw reply [flat|nested] 80+ messages in thread
* Re: [PATCH v14 02/11] s390x/cpu topology: add topology entries on CPU hotplug
2023-01-17 13:55 ` Pierre Morel
@ 2023-01-17 16:48 ` Nina Schoetterl-Glausch
2023-01-19 13:34 ` Pierre Morel
0 siblings, 1 reply; 80+ messages in thread
From: Nina Schoetterl-Glausch @ 2023-01-17 16:48 UTC (permalink / raw)
To: Pierre Morel, qemu-s390x
Cc: qemu-devel, borntraeger, pasic, richard.henderson, david, thuth,
cohuck, mst, pbonzini, kvm, ehabkost, marcel.apfelbaum, eblake,
armbru, seiden, nrb, frankja, berrange, clg
On Tue, 2023-01-17 at 14:55 +0100, Pierre Morel wrote:
>
> On 1/13/23 19:15, Nina Schoetterl-Glausch wrote:
> >
[...]
> > > +/**
> > > + * s390_topology_set_entry:
> > > + * @entry: Topology entry to setup
> > > + * @id: topology id to use for the setup
> > > + *
> > > + * Set the core bit inside the topology mask and
> > > + * increments the number of cores for the socket.
> > > + */
> > > +static void s390_topology_set_entry(S390TopologyEntry *entry,
> >
> > Not sure if I like the name, what it does is to add a cpu to the entry.
>
> s390_topology_add_cpu_to_entry() ?
Yeah, that's better.
[...]
>
> > > +/**
> > > + * s390_topology_set_cpu:
> > > + * @ms: MachineState used to initialize the topology structure on
> > > + * first call.
> > > + * @cpu: the new S390CPU to insert in the topology structure
> > > + * @errp: the error pointer
> > > + *
> > > + * Called from CPU Hotplug to check and setup the CPU attributes
> > > + * before to insert the CPU in the topology.
> > > + */
> > > +void s390_topology_set_cpu(MachineState *ms, S390CPU *cpu, Error **errp)
> > > +{
> > > + Error *local_error = NULL;
> >
> > Can't you just use ERRP_GUARD ?
>
> I do not think it is necessary and I find it obfuscating.
> So, should I?
/*
* Propagate error object (if any) from @local_err to @dst_errp.
[...]
* Please use ERRP_GUARD() instead when possible.
* Please don't error_propagate(&error_fatal, ...), use
* error_report_err() and exit(), because that's more obvious.
*/
void error_propagate(Error **dst_errp, Error *local_err);
So I'd say yes.
>
> >
> > > + s390_topology_id id;
> > > +
> > > + /*
> > > + * We do not want to initialize the topology if the cpu model
> > > + * does not support topology consequently, we have to wait for
> >
> > ", consequently," I think. Could you do the initialization some where else,
> > after you know what the cpu model is? Not that I object to doing it this way.
> >
>
> I did not find a better place, it must be done after the CPU model is
> initialize and before the first CPU is created.
> The cpu model is initialized during the early creation of the first cpu.
>
> Any idea?
>
> Thanks.
>
> Regards,
> Pierre
>
^ permalink raw reply [flat|nested] 80+ messages in thread
* Re: [PATCH v14 03/11] target/s390x/cpu topology: handle STSI(15) and build the SYSIB
2023-01-10 14:29 ` Thomas Huth
2023-01-11 9:16 ` Thomas Huth
2023-01-11 17:14 ` Nina Schoetterl-Glausch
@ 2023-01-17 16:56 ` Pierre Morel
2023-01-18 10:26 ` Thomas Huth
2 siblings, 1 reply; 80+ messages in thread
From: Pierre Morel @ 2023-01-17 16:56 UTC (permalink / raw)
To: Thomas Huth, qemu-s390x, frankja
Cc: qemu-devel, borntraeger, pasic, richard.henderson, david, cohuck,
mst, pbonzini, kvm, ehabkost, marcel.apfelbaum, eblake, armbru,
seiden, nrb, scgl, berrange, clg
On 1/10/23 15:29, Thomas Huth wrote:
> On 05/01/2023 15.53, Pierre Morel wrote:
>> On interception of STSI(15.1.x) the System Information Block
>> (SYSIB) is built from the list of pre-ordered topology entries.
>>
>> Signed-off-by: Pierre Morel <pmorel@linux.ibm.com>
>> ---
> ...
>> diff --git a/target/s390x/cpu.h b/target/s390x/cpu.h
>> index 39ea63a416..78988048dd 100644
>> --- a/target/s390x/cpu.h
>> +++ b/target/s390x/cpu.h
>> @@ -561,6 +561,25 @@ typedef struct SysIB_322 {
>> } SysIB_322;
>> QEMU_BUILD_BUG_ON(sizeof(SysIB_322) != 4096);
>> +#define S390_TOPOLOGY_MAG 6
>> +#define S390_TOPOLOGY_MAG6 0
>> +#define S390_TOPOLOGY_MAG5 1
>> +#define S390_TOPOLOGY_MAG4 2
>> +#define S390_TOPOLOGY_MAG3 3
>> +#define S390_TOPOLOGY_MAG2 4
>> +#define S390_TOPOLOGY_MAG1 5
>> +/* Configuration topology */
>> +typedef struct SysIB_151x {
>> + uint8_t reserved0[2];
>> + uint16_t length;
>> + uint8_t mag[S390_TOPOLOGY_MAG];
>> + uint8_t reserved1;
>> + uint8_t mnest;
>> + uint32_t reserved2;
>> + char tle[];
>> +} QEMU_PACKED QEMU_ALIGNED(8) SysIB_151x;
>> +QEMU_BUILD_BUG_ON(sizeof(SysIB_151x) != 16);
>> +
>> typedef union SysIB {
>> SysIB_111 sysib_111;
>> SysIB_121 sysib_121;
>> @@ -568,9 +587,68 @@ typedef union SysIB {
>> SysIB_221 sysib_221;
>> SysIB_222 sysib_222;
>> SysIB_322 sysib_322;
>> + SysIB_151x sysib_151x;
>> } SysIB;
>> QEMU_BUILD_BUG_ON(sizeof(SysIB) != 4096);
>> +/*
>> + * CPU Topology List provided by STSI with fc=15 provides a list
>> + * of two different Topology List Entries (TLE) types to specify
>> + * the topology hierarchy.
>> + *
>> + * - Container Topology List Entry
>> + * Defines a container to contain other Topology List Entries
>> + * of any type, nested containers or CPU.
>> + * - CPU Topology List Entry
>> + * Specifies the CPUs position, type, entitlement and polarization
>> + * of the CPUs contained in the last Container TLE.
>> + *
>> + * There can be theoretically up to five levels of containers, QEMU
>> + * uses only one level, the socket level.
>
> I guess that sentence needs an update again, now that you've re-added
> the books and drawers?
yes
>
>> + * A container of with a nesting level (NL) greater than 1 can only
>> + * contain another container of nesting level NL-1.
>> + *
>> + * A container of nesting level 1 (socket), contains as many CPU TLE
>> + * as needed to describe the position and qualities of all CPUs inside
>> + * the container.
>> + * The qualities of a CPU are polarization, entitlement and type.
>> + *
>> + * The CPU TLE defines the position of the CPUs of identical qualities
>> + * using a 64bits mask which first bit has its offset defined by
>> + * the CPU address orgin field of the CPU TLE like in:
>> + * CPU address = origin * 64 + bit position within the mask
>> + *
>> + */
>> +/* Container type Topology List Entry */
>> +/* Container type Topology List Entry */
>
> Duplicated comment.
OK
>
>> +typedef struct SysIBTl_container {
>> + uint8_t nl;
>> + uint8_t reserved[6];
>> + uint8_t id;
>> +} QEMU_PACKED QEMU_ALIGNED(8) SysIBTl_container;
>> +QEMU_BUILD_BUG_ON(sizeof(SysIBTl_container) != 8);
>> +
>> +/* CPU type Topology List Entry */
>> +typedef struct SysIBTl_cpu {
>> + uint8_t nl;
>> + uint8_t reserved0[3];
>> + uint8_t reserved1:5;
>> + uint8_t dedicated:1;
>> + uint8_t polarity:2;
>
> Hmmm, yet another bitfield...
Yes, this is the firmware interface.
If it makes problem I can use masks and logic arithmetic
>
>> + uint8_t type;
>> + uint16_t origin;
>> + uint64_t mask;
>> +} QEMU_PACKED QEMU_ALIGNED(8) SysIBTl_cpu;
>> +QEMU_BUILD_BUG_ON(sizeof(SysIBTl_cpu) != 16);
>> +
>> +/* Max size of a SYSIB structure is when all CPU are alone in a
>> container */
>> +#define S390_TOPOLOGY_SYSIB_SIZE (sizeof(SysIB_151x)
>> + \
>> + S390_MAX_CPUS *
>> (sizeof(SysIBTl_container) + \
>> + sizeof(SysIBTl_cpu)))
>> +
>> +void insert_stsi_15_1_x(S390CPU *cpu, int sel2, __u64 addr, uint8_t ar);
>> +
>> /* MMU defines */
>> #define ASCE_ORIGIN (~0xfffULL) /* segment table
>> origin */
>> #define ASCE_SUBSPACE 0x200 /* subspace group
>> control */
>> diff --git a/target/s390x/kvm/cpu_topology.c
>> b/target/s390x/kvm/cpu_topology.c
>> new file mode 100644
>> index 0000000000..3831a3264c
>> --- /dev/null
>> +++ b/target/s390x/kvm/cpu_topology.c
>> @@ -0,0 +1,136 @@
>> +/*
>> + * QEMU S390x CPU Topology
>> + *
>> + * Copyright IBM Corp. 2022
>
> Happy new year?
So after Nina's comment what do I do?
let it be 22 because I started last year or update because what is
important is when it comes into mainline?
>
>> + * Author(s): Pierre Morel <pmorel@linux.ibm.com>
>> + *
>> + * This work is licensed under the terms of the GNU GPL, version 2 or
>> (at
>> + * your option) any later version. See the COPYING file in the top-level
>> + * directory.
>> + */
>> +#include "qemu/osdep.h"
>> +#include "cpu.h"
>> +#include "hw/s390x/pv.h"
>> +#include "hw/sysbus.h"
>> +#include "hw/s390x/sclp.h"
>> +#include "hw/s390x/cpu-topology.h"
>> +
>> +static char *fill_container(char *p, int level, int id)
>> +{
>> + SysIBTl_container *tle = (SysIBTl_container *)p;
>> +
>> + tle->nl = level;
>> + tle->id = id;
>> + return p + sizeof(*tle);
>> +}
>> +
>> +static char *fill_tle_cpu(char *p, S390TopologyEntry *entry)
>> +{
>> + SysIBTl_cpu *tle = (SysIBTl_cpu *)p;
>> + s390_topology_id topology_id = entry->id;
>
> What about the reserved fields? Should they get set to 0 ?
Good question.
I forgot this after changing the allocation.
It must be 0.
I will do that during the allocation.
>
>> + tle->nl = 0;
>> + tle->dedicated = topology_id.d;
>> + tle->polarity = topology_id.p;
>> + tle->type = topology_id.type;
>> + tle->origin = topology_id.origin;
>> + tle->mask = cpu_to_be64(entry->mask);
>
> So here you're already taking care of swapping the endianess in case we
> ever run this code with TCG, too ... so I think it would be great to
> also eliminate the bitfield in SysIBTl_cpu to be really on the safe side.
OK.
>
>> + return p + sizeof(*tle);
>> +}
> ...
>> +void insert_stsi_15_1_x(S390CPU *cpu, int sel2, __u64 addr, uint8_t ar)
>> +{
>> + union {
>> + char place_holder[S390_TOPOLOGY_SYSIB_SIZE];
>> + SysIB_151x sysib;
>> + } buffer QEMU_ALIGNED(8) = {};
>> + int len;
>> +
>> + if (!s390_has_topology() || sel2 < 2 || sel2 >
>> SCLP_READ_SCP_INFO_MNEST) {
>> + setcc(cpu, 3);
>> + return;
>> + }
>> +
>> + len = setup_stsi(cpu, &buffer.sysib, sel2);
>> +
>> + if (len > 4096) {
>
> Maybe use TARGET_PAGE_SIZE instead of 4096 ?
I am not sure about this, even TARGET_PAGE_SIZE will probably never
change, this is the maximal SYSIB size, probably related to page size but...
What about defining it in the cpu.h as
#define SYSIB_MAX_SIZE TARGET_PAGE_SIZE
?
>
>> + setcc(cpu, 3);
>> + return;
>> + }
>> +
>> + buffer.sysib.length = cpu_to_be16(len);
>> + s390_cpu_virt_mem_write(cpu, addr, ar, &buffer.sysib, len);
>
> Is this supposed to work with protected guests, too? If so, I think you
> likely need to use s390_cpu_pv_mem_write() for protected guests?
it is not supposed to work with protected guests.
Thanks.
Regards,
Pierre
--
Pierre Morel
IBM Lab Boeblingen
^ permalink raw reply [flat|nested] 80+ messages in thread
* Re: [PATCH v14 03/11] target/s390x/cpu topology: handle STSI(15) and build the SYSIB
2023-01-11 17:14 ` Nina Schoetterl-Glausch
@ 2023-01-17 16:58 ` Pierre Morel
0 siblings, 0 replies; 80+ messages in thread
From: Pierre Morel @ 2023-01-17 16:58 UTC (permalink / raw)
To: Nina Schoetterl-Glausch, Thomas Huth, qemu-s390x, frankja
Cc: qemu-devel, borntraeger, pasic, richard.henderson, david, cohuck,
mst, pbonzini, kvm, ehabkost, marcel.apfelbaum, eblake, armbru,
seiden, nrb, berrange, clg
On 1/11/23 18:14, Nina Schoetterl-Glausch wrote:
> On Tue, 2023-01-10 at 15:29 +0100, Thomas Huth wrote:
>> On 05/01/2023 15.53, Pierre Morel wrote:
>>> On interception of STSI(15.1.x) the System Information Block
>>> (SYSIB) is built from the list of pre-ordered topology entries.
>>>
>>> Signed-off-by: Pierre Morel <pmorel@linux.ibm.com>
>>> ---
>>
> [...]
>
>>> +void insert_stsi_15_1_x(S390CPU *cpu, int sel2, __u64 addr, uint8_t ar)
>>> +{
>>> + union {
>>> + char place_holder[S390_TOPOLOGY_SYSIB_SIZE];
>>> + SysIB_151x sysib;
>>> + } buffer QEMU_ALIGNED(8) = {};
>>> + int len;
>>> +
>>> + if (!s390_has_topology() || sel2 < 2 || sel2 > SCLP_READ_SCP_INFO_MNEST) {
>>> + setcc(cpu, 3);
>>> + return;
>>> + }
>>> +
>>> + len = setup_stsi(cpu, &buffer.sysib, sel2);
>>> +
>>> + if (len > 4096) {
>>
>> Maybe use TARGET_PAGE_SIZE instead of 4096 ?
>
> sizeof(SysIB) would be preferable IMO.
Yes better
Thanks,
Regards,
Pierre
--
Pierre Morel
IBM Lab Boeblingen
^ permalink raw reply [flat|nested] 80+ messages in thread
* Re: [PATCH v14 04/11] s390x/sclp: reporting the maximum nested topology entries
2023-01-11 8:57 ` Thomas Huth
@ 2023-01-17 17:36 ` Pierre Morel
2023-01-17 19:58 ` Nina Schoetterl-Glausch
0 siblings, 1 reply; 80+ messages in thread
From: Pierre Morel @ 2023-01-17 17:36 UTC (permalink / raw)
To: Thomas Huth, qemu-s390x
Cc: qemu-devel, borntraeger, pasic, richard.henderson, david, cohuck,
mst, pbonzini, kvm, ehabkost, marcel.apfelbaum, eblake, armbru,
seiden, nrb, scgl, frankja, berrange, clg
On 1/11/23 09:57, Thomas Huth wrote:
> On 05/01/2023 15.53, Pierre Morel wrote:
>> The maximum nested topology entries is used by the guest to know
>> how many nested topology are available on the machine.
>>
>> Currently, SCLP READ SCP INFO reports MNEST = 0, which is the
>> equivalent of reporting the default value of 2.
>> Let's use the default SCLP value of 2 and increase this value in the
>> future patches implementing higher levels.
>
> I'm confused ... so does a SCLP value of 2 mean a MNEST level of 4 ?
Sorry, I forgot to change this.
MNEST = 0 means no MNEST support and only socket is supported so it is
like MNEST = 2.
MNEST != 0 set the maximum nested level and correct values may be 2,3 or 4.
But this setting to 4 should already have been done in previous patch
where we introduced the books and drawers.
I change the commit message with:
---
s390x/sclp: reporting the maximum nested topology entries
The maximum nested topology entries is used by the guest to know
how many nested topology are available on the machine.
Let's return this information to the guest.
---
>
>> Signed-off-by: Pierre Morel <pmorel@linux.ibm.com>
>> ---
>> include/hw/s390x/sclp.h | 5 +++--
>> hw/s390x/sclp.c | 4 ++++
>> 2 files changed, 7 insertions(+), 2 deletions(-)
>>
>> diff --git a/include/hw/s390x/sclp.h b/include/hw/s390x/sclp.h
>> index 712fd68123..4ce852473c 100644
>> --- a/include/hw/s390x/sclp.h
>> +++ b/include/hw/s390x/sclp.h
>> @@ -112,12 +112,13 @@ typedef struct CPUEntry {
>> } QEMU_PACKED CPUEntry;
>> #define SCLP_READ_SCP_INFO_FIXED_CPU_OFFSET 128
>> -#define SCLP_READ_SCP_INFO_MNEST 2
>> +#define SCLP_READ_SCP_INFO_MNEST 4
>
> ... since you update it to 4 here.
Yes, in fact this should be set in the previous patch already to 4.
So I will do that.
>
>> typedef struct ReadInfo {
>> SCCBHeader h;
>> uint16_t rnmax;
>> uint8_t rnsize;
>> - uint8_t _reserved1[16 - 11]; /* 11-15 */
>> + uint8_t _reserved1[15 - 11]; /* 11-14 */
>> + uint8_t stsi_parm; /* 15-16 */
>> uint16_t entries_cpu; /* 16-17 */
>> uint16_t offset_cpu; /* 18-19 */
>> uint8_t _reserved2[24 - 20]; /* 20-23 */
>> diff --git a/hw/s390x/sclp.c b/hw/s390x/sclp.c
>> index eff74479f4..07e3cb4cac 100644
>> --- a/hw/s390x/sclp.c
>> +++ b/hw/s390x/sclp.c
>> @@ -20,6 +20,7 @@
>> #include "hw/s390x/event-facility.h"
>> #include "hw/s390x/s390-pci-bus.h"
>> #include "hw/s390x/ipl.h"
>> +#include "hw/s390x/cpu-topology.h"
>> static inline SCLPDevice *get_sclp_device(void)
>> {
>> @@ -125,6 +126,9 @@ static void read_SCP_info(SCLPDevice *sclp, SCCB
>> *sccb)
>> /* CPU information */
>> prepare_cpu_entries(machine, entries_start, &cpu_count);
>> + if (s390_has_topology()) {
>> + read_info->stsi_parm = SCLP_READ_SCP_INFO_MNEST;
>
> This seems to be in contradiction to what you've said in the commit
> description - you set it to 4 and not to 2.
Yes, I change the commit message.
Thanks.
Regards,
Pierre
--
Pierre Morel
IBM Lab Boeblingen
^ permalink raw reply [flat|nested] 80+ messages in thread
* Re: [PATCH v14 04/11] s390x/sclp: reporting the maximum nested topology entries
2023-01-11 17:52 ` Nina Schoetterl-Glausch
@ 2023-01-17 17:44 ` Pierre Morel
0 siblings, 0 replies; 80+ messages in thread
From: Pierre Morel @ 2023-01-17 17:44 UTC (permalink / raw)
To: Nina Schoetterl-Glausch, qemu-s390x
Cc: qemu-devel, borntraeger, pasic, richard.henderson, david, thuth,
cohuck, mst, pbonzini, kvm, ehabkost, marcel.apfelbaum, eblake,
armbru, seiden, nrb, frankja, berrange, clg
On 1/11/23 18:52, Nina Schoetterl-Glausch wrote:
> On Thu, 2023-01-05 at 15:53 +0100, Pierre Morel wrote:
>> The maximum nested topology entries is used by the guest to know
>> how many nested topology are available on the machine.
>>
>> Currently, SCLP READ SCP INFO reports MNEST = 0, which is the
>> equivalent of reporting the default value of 2.
>> Let's use the default SCLP value of 2 and increase this value in the
>> future patches implementing higher levels.
>>
>> Signed-off-by: Pierre Morel <pmorel@linux.ibm.com>
>
> Reviewed-by: Nina Schoetterl-Glausch <nsg@linux.ibm.com>
> if you address the issues Thomas found with the commit description
> and the nits below.
Thanks.
>
>> ---
>> include/hw/s390x/sclp.h | 5 +++--
>> hw/s390x/sclp.c | 4 ++++
>> 2 files changed, 7 insertions(+), 2 deletions(-)
>>
>> diff --git a/include/hw/s390x/sclp.h b/include/hw/s390x/sclp.h
>> index 712fd68123..4ce852473c 100644
>> --- a/include/hw/s390x/sclp.h
>> +++ b/include/hw/s390x/sclp.h
>> @@ -112,12 +112,13 @@ typedef struct CPUEntry {
>> } QEMU_PACKED CPUEntry;
>>
>> #define SCLP_READ_SCP_INFO_FIXED_CPU_OFFSET 128
>> -#define SCLP_READ_SCP_INFO_MNEST 2
>> +#define SCLP_READ_SCP_INFO_MNEST 4
>> typedef struct ReadInfo {
>> SCCBHeader h;
>> uint16_t rnmax;
>> uint8_t rnsize;
>> - uint8_t _reserved1[16 - 11]; /* 11-15 */
>> + uint8_t _reserved1[15 - 11]; /* 11-14 */
>> + uint8_t stsi_parm; /* 15-16 */
>
> The numbering here is the same as the one in the arch doc, instead
> of the maybe more usual one where the right number is exclusive.
> So 15-16 looks like a two byte field, so just do 15 or just drop it.
Yes right, thanks.
>
>> uint16_t entries_cpu; /* 16-17 */
>> uint16_t offset_cpu; /* 18-19 */
>> uint8_t _reserved2[24 - 20]; /* 20-23 */
>> diff --git a/hw/s390x/sclp.c b/hw/s390x/sclp.c
>> index eff74479f4..07e3cb4cac 100644
>> --- a/hw/s390x/sclp.c
>> +++ b/hw/s390x/sclp.c
>> @@ -20,6 +20,7 @@
>> #include "hw/s390x/event-facility.h"
>> #include "hw/s390x/s390-pci-bus.h"
>> #include "hw/s390x/ipl.h"
>> +#include "hw/s390x/cpu-topology.h"
>>
>> static inline SCLPDevice *get_sclp_device(void)
>> {
>> @@ -125,6 +126,9 @@ static void read_SCP_info(SCLPDevice *sclp, SCCB *sccb)
>>
>> /* CPU information */
>> prepare_cpu_entries(machine, entries_start, &cpu_count);
>> + if (s390_has_topology()) {
>> + read_info->stsi_parm = SCLP_READ_SCP_INFO_MNEST;
>> + }
>
> Maybe move that up a bit, not sure if it belongs under the CPU information section.
> I'd leave prepare_cpu_entries and read_info->entries_cpu = adjacent in any case.
>
Yes, I do that.
Thanks,
Regards,
Pierre
--
Pierre Morel
IBM Lab Boeblingen
^ permalink raw reply [flat|nested] 80+ messages in thread
* Re: [PATCH v14 05/11] s390x/cpu topology: resetting the Topology-Change-Report
2023-01-11 9:00 ` Thomas Huth
@ 2023-01-17 17:57 ` Pierre Morel
0 siblings, 0 replies; 80+ messages in thread
From: Pierre Morel @ 2023-01-17 17:57 UTC (permalink / raw)
To: Thomas Huth, qemu-s390x
Cc: qemu-devel, borntraeger, pasic, richard.henderson, david, cohuck,
mst, pbonzini, kvm, ehabkost, marcel.apfelbaum, eblake, armbru,
seiden, nrb, scgl, frankja, berrange, clg
On 1/11/23 10:00, Thomas Huth wrote:
> On 05/01/2023 15.53, Pierre Morel wrote:
>> During a subsystem reset the Topology-Change-Report is cleared
>> by the machine.
>> Let's ask KVM to clear the Modified Topology Change Report (MTCR)
>> bit of the SCA in the case of a subsystem reset.
>>
>> Signed-off-by: Pierre Morel <pmorel@linux.ibm.com>
>> ---
> ...
>> diff --git a/target/s390x/kvm/kvm_s390x.h b/target/s390x/kvm/kvm_s390x.h
>> index f9785564d0..649dae5948 100644
>> --- a/target/s390x/kvm/kvm_s390x.h
>> +++ b/target/s390x/kvm/kvm_s390x.h
>> @@ -47,5 +47,6 @@ void kvm_s390_crypto_reset(void);
>> void kvm_s390_restart_interrupt(S390CPU *cpu);
>> void kvm_s390_stop_interrupt(S390CPU *cpu);
>> void kvm_s390_set_diag318(CPUState *cs, uint64_t diag318_info);
>> +int kvm_s390_topology_set_mtcr(uint64_t attr);
>> #endif /* KVM_S390X_H */
>> diff --git a/hw/s390x/s390-virtio-ccw.c b/hw/s390x/s390-virtio-ccw.c
>> index c98b93a15f..14798ca305 100644
>> --- a/hw/s390x/s390-virtio-ccw.c
>> +++ b/hw/s390x/s390-virtio-ccw.c
>> @@ -122,6 +122,7 @@ static void subsystem_reset(void)
>> device_cold_reset(dev);
>> }
>> }
>> + s390_cpu_topology_reset();
>> }
>
> Would it make sense to add a "if (s390_has_topology())" check around the
> new line?
Yes I think you are right, otherwise we may get a warning if the kernel
does not support resetting the topology.
>
> Anyway:
> Reviewed-by: Thomas Huth <thuth@redhat.com>
>
>
Thanks,
Regards,
Pierre
--
Pierre Morel
IBM Lab Boeblingen
^ permalink raw reply [flat|nested] 80+ messages in thread
* Re: [PATCH v14 04/11] s390x/sclp: reporting the maximum nested topology entries
2023-01-17 17:36 ` Pierre Morel
@ 2023-01-17 19:58 ` Nina Schoetterl-Glausch
2023-01-19 13:08 ` Pierre Morel
0 siblings, 1 reply; 80+ messages in thread
From: Nina Schoetterl-Glausch @ 2023-01-17 19:58 UTC (permalink / raw)
To: Pierre Morel, Thomas Huth, qemu-s390x
Cc: qemu-devel, borntraeger, pasic, richard.henderson, david, cohuck,
mst, pbonzini, kvm, ehabkost, marcel.apfelbaum, eblake, armbru,
seiden, nrb, frankja, berrange, clg
On Tue, 2023-01-17 at 18:36 +0100, Pierre Morel wrote:
>
> On 1/11/23 09:57, Thomas Huth wrote:
> > On 05/01/2023 15.53, Pierre Morel wrote:
> > > The maximum nested topology entries is used by the guest to know
> > > how many nested topology are available on the machine.
> > >
> > > Currently, SCLP READ SCP INFO reports MNEST = 0, which is the
> > > equivalent of reporting the default value of 2.
> > > Let's use the default SCLP value of 2 and increase this value in the
> > > future patches implementing higher levels.
> >
> > I'm confused ... so does a SCLP value of 2 mean a MNEST level of 4 ?
>
> Sorry, I forgot to change this.
> MNEST = 0 means no MNEST support and only socket is supported so it is
> like MNEST = 2.
> MNEST != 0 set the maximum nested level and correct values may be 2,3 or 4.
> But this setting to 4 should already have been done in previous patch
> where we introduced the books and drawers.
I think setting it to 4 here is fine/preferable, since 2 is the default unless
you tell the guest that more are available, which you do in this patch.
It's only the commit description that is confusing.
>
> I change the commit message with:
> ---
> s390x/sclp: reporting the maximum nested topology entries
>
> The maximum nested topology entries is used by the guest to know
> how many nested topology are available on the machine.
>
> Let's return this information to the guest.
> ---
>
> >
> > > Signed-off-by: Pierre Morel <pmorel@linux.ibm.com>
> > > ---
> > > include/hw/s390x/sclp.h | 5 +++--
> > > hw/s390x/sclp.c | 4 ++++
> > > 2 files changed, 7 insertions(+), 2 deletions(-)
> > >
> > > diff --git a/include/hw/s390x/sclp.h b/include/hw/s390x/sclp.h
> > > index 712fd68123..4ce852473c 100644
> > > --- a/include/hw/s390x/sclp.h
> > > +++ b/include/hw/s390x/sclp.h
> > > @@ -112,12 +112,13 @@ typedef struct CPUEntry {
> > > } QEMU_PACKED CPUEntry;
> > > #define SCLP_READ_SCP_INFO_FIXED_CPU_OFFSET 128
> > > -#define SCLP_READ_SCP_INFO_MNEST 2
> > > +#define SCLP_READ_SCP_INFO_MNEST 4
> >
> > ... since you update it to 4 here.
>
> Yes, in fact this should be set in the previous patch already to 4.
> So I will do that.
>
> >
> > > typedef struct ReadInfo {
> > > SCCBHeader h;
> > > uint16_t rnmax;
> > > uint8_t rnsize;
> > > - uint8_t _reserved1[16 - 11]; /* 11-15 */
> > > + uint8_t _reserved1[15 - 11]; /* 11-14 */
> > > + uint8_t stsi_parm; /* 15-16 */
> > > uint16_t entries_cpu; /* 16-17 */
> > > uint16_t offset_cpu; /* 18-19 */
> > > uint8_t _reserved2[24 - 20]; /* 20-23 */
> > > diff --git a/hw/s390x/sclp.c b/hw/s390x/sclp.c
> > > index eff74479f4..07e3cb4cac 100644
> > > --- a/hw/s390x/sclp.c
> > > +++ b/hw/s390x/sclp.c
> > > @@ -20,6 +20,7 @@
> > > #include "hw/s390x/event-facility.h"
> > > #include "hw/s390x/s390-pci-bus.h"
> > > #include "hw/s390x/ipl.h"
> > > +#include "hw/s390x/cpu-topology.h"
> > > static inline SCLPDevice *get_sclp_device(void)
> > > {
> > > @@ -125,6 +126,9 @@ static void read_SCP_info(SCLPDevice *sclp, SCCB
> > > *sccb)
> > > /* CPU information */
> > > prepare_cpu_entries(machine, entries_start, &cpu_count);
> > > + if (s390_has_topology()) {
> > > + read_info->stsi_parm = SCLP_READ_SCP_INFO_MNEST;
> >
> > This seems to be in contradiction to what you've said in the commit
> > description - you set it to 4 and not to 2.
>
> Yes, I change the commit message.
>
> Thanks.
>
> Regards,
> Pierre
>
^ permalink raw reply [flat|nested] 80+ messages in thread
* Re: [PATCH v14 06/11] s390x/cpu topology: interception of PTF instruction
2023-01-16 18:24 ` Nina Schoetterl-Glausch
@ 2023-01-18 9:54 ` Pierre Morel
2023-01-20 14:32 ` Pierre Morel
1 sibling, 0 replies; 80+ messages in thread
From: Pierre Morel @ 2023-01-18 9:54 UTC (permalink / raw)
To: Nina Schoetterl-Glausch, qemu-s390x
Cc: qemu-devel, borntraeger, pasic, richard.henderson, david, thuth,
cohuck, mst, pbonzini, kvm, ehabkost, marcel.apfelbaum, eblake,
armbru, seiden, nrb, frankja, berrange, clg
On 1/16/23 19:24, Nina Schoetterl-Glausch wrote:
> On Thu, 2023-01-05 at 15:53 +0100, Pierre Morel wrote:
>> When the host supports the CPU topology facility, the PTF
>> instruction with function code 2 is interpreted by the SIE,
>> provided that the userland hypervizor activates the interpretation
>> by using the KVM_CAP_S390_CPU_TOPOLOGY KVM extension.
>>
>> The PTF instructions with function code 0 and 1 are intercepted
>> and must be emulated by the userland hypervizor.
>>
>> During RESET all CPU of the configuration are placed in
>> horizontal polarity.
>>
>> Signed-off-by: Pierre Morel <pmorel@linux.ibm.com>
>> ---
>> include/hw/s390x/cpu-topology.h | 3 +
>> include/hw/s390x/s390-virtio-ccw.h | 6 ++
>> target/s390x/cpu.h | 1 +
>> hw/s390x/cpu-topology.c | 92 ++++++++++++++++++++++++++++++
>> target/s390x/cpu-sysemu.c | 16 ++++++
>> target/s390x/kvm/kvm.c | 11 ++++
>> 6 files changed, 129 insertions(+)
>>
>> diff --git a/include/hw/s390x/cpu-topology.h b/include/hw/s390x/cpu-topology.h
>> index 9571aa70e5..33e23d78b9 100644
>> --- a/include/hw/s390x/cpu-topology.h
>> +++ b/include/hw/s390x/cpu-topology.h
>> @@ -55,11 +55,13 @@ typedef struct S390Topology {
>> QTAILQ_HEAD(, S390TopologyEntry) list;
>> uint8_t *sockets;
>> CpuTopology *smp;
>> + int polarity;
>> } S390Topology;
>>
>> #ifdef CONFIG_KVM
>> bool s390_has_topology(void);
>> void s390_topology_set_cpu(MachineState *ms, S390CPU *cpu, Error **errp);
>> +void s390_topology_set_polarity(int polarity);
>> #else
>> static inline bool s390_has_topology(void)
>> {
>> @@ -68,6 +70,7 @@ static inline bool s390_has_topology(void)
>> static inline void s390_topology_set_cpu(MachineState *ms,
>> S390CPU *cpu,
>> Error **errp) {}
>> +static inline void s390_topology_set_polarity(int polarity) {}
>> #endif
>> extern S390Topology s390_topology;
>>
>> diff --git a/include/hw/s390x/s390-virtio-ccw.h b/include/hw/s390x/s390-virtio-ccw.h
>> index 9bba21a916..c1d46e78af 100644
>> --- a/include/hw/s390x/s390-virtio-ccw.h
>> +++ b/include/hw/s390x/s390-virtio-ccw.h
>> @@ -30,6 +30,12 @@ struct S390CcwMachineState {
>> uint8_t loadparm[8];
>> };
>>
>> +#define S390_PTF_REASON_NONE (0x00 << 8)
>> +#define S390_PTF_REASON_DONE (0x01 << 8)
>> +#define S390_PTF_REASON_BUSY (0x02 << 8)
>> +#define S390_TOPO_FC_MASK 0xffUL
>> +void s390_handle_ptf(S390CPU *cpu, uint8_t r1, uintptr_t ra);
>> +
>> struct S390CcwMachineClass {
>> /*< private >*/
>> MachineClass parent_class;
>> diff --git a/target/s390x/cpu.h b/target/s390x/cpu.h
>> index 01ade07009..5da4041576 100644
>> --- a/target/s390x/cpu.h
>> +++ b/target/s390x/cpu.h
>> @@ -864,6 +864,7 @@ void s390_do_cpu_set_diag318(CPUState *cs, run_on_cpu_data arg);
>> int s390_assign_subch_ioeventfd(EventNotifier *notifier, uint32_t sch_id,
>> int vq, bool assign);
>> void s390_cpu_topology_reset(void);
>> +void s390_cpu_topology_set(void);
>
> I don't like this name much, it's nondescript.
> s390_cpu_topology_set_modified ?
yes, better.
>
>> #ifndef CONFIG_USER_ONLY
>> unsigned int s390_cpu_set_state(uint8_t cpu_state, S390CPU *cpu);
>> #else
>> diff --git a/hw/s390x/cpu-topology.c b/hw/s390x/cpu-topology.c
>> index 438055c612..e6b4692581 100644
>> --- a/hw/s390x/cpu-topology.c
>> +++ b/hw/s390x/cpu-topology.c
>> @@ -97,6 +97,98 @@ static s390_topology_id s390_topology_from_cpu(S390CPU *cpu)
>> }
>>
>> /**
>> + * s390_topology_set_polarity
>> + * @polarity: horizontal or vertical
>> + *
>> + * Changes the polarity of all the CPU in the configuration.
>> + *
>> + * If the dedicated CPU modifier attribute is set a vertical
>> + * polarization is always high (Architecture).
>> + * Otherwise we decide to set it as medium.
>> + *
>> + * Once done, advertise a topology change.
>> + */
>> +void s390_topology_set_polarity(int polarity)
>
> I don't like that this function ignores what kind of vertical polarization is passed,
> it's confusing.
> That seems like a further reason to split horizontal/vertical from the entitlement.
OK, you are right.
I remove this function and put the s390_cpu_topology_set() inside the
handle_ptf()
>
>> +{
>> + S390TopologyEntry *entry;
>
> I also expected this function to set s390_topology.polarization, but it doesn't.
>> +
>> + QTAILQ_FOREACH(entry, &s390_topology.list, next) {
>> + if (polarity == S390_TOPOLOGY_POLARITY_HORIZONTAL) {
>> + entry->id.p = polarity;
>> + } else {
>> + if (entry->id.d) {
>> + entry->id.p = S390_TOPOLOGY_POLARITY_VERTICAL_HIGH;
>> + } else {
>> + entry->id.p = S390_TOPOLOGY_POLARITY_VERTICAL_MEDIUM;
>> + }
>> + }
>> + }
>> + s390_cpu_topology_set();
>> +}
>> +
>> +/*
>> + * s390_handle_ptf:
>> + *
>> + * @register 1: contains the function code
>> + *
>> + * Function codes 0 and 1 handle the CPU polarization.
>> + * We assume an horizontal topology, the only one supported currently
>> + * by Linux, consequently we answer to function code 0, requesting
>> + * horizontal polarization that it is already the current polarization
>> + * and reject vertical polarization request without further explanation.
>
> This comment is outdated, right? Same for those in the function body.
>
>> + *
>> + * Function code 2 is handling topology changes and is interpreted
>> + * by the SIE.
>> + */
>> +void s390_handle_ptf(S390CPU *cpu, uint8_t r1, uintptr_t ra)
>> +{
>> + CPUS390XState *env = &cpu->env;
>> + uint64_t reg = env->regs[r1];
>> + uint8_t fc = reg & S390_TOPO_FC_MASK;
>> +
>> + if (!s390_has_feat(S390_FEAT_CONFIGURATION_TOPOLOGY)) {
>> + s390_program_interrupt(env, PGM_OPERATION, ra);
>> + return;
>> + }
>> +
>> + if (env->psw.mask & PSW_MASK_PSTATE) {
>> + s390_program_interrupt(env, PGM_PRIVILEGED, ra);
>> + return;
>> + }
>> +
>> + if (reg & ~S390_TOPO_FC_MASK) {
>> + s390_program_interrupt(env, PGM_SPECIFICATION, ra);
>> + return;
>> + }
>> +
>> + switch (fc) {
>> + case 0: /* Horizontal polarization is already set */
>> + if (s390_topology.polarity == S390_TOPOLOGY_POLARITY_HORIZONTAL) {
>> + env->regs[r1] |= S390_PTF_REASON_DONE;
>> + setcc(cpu, 2);
>> + } else {
>> + s390_topology_set_polarity(S390_TOPOLOGY_POLARITY_HORIZONTAL);
>> + s390_topology.polarity = S390_TOPOLOGY_POLARITY_HORIZONTAL;
>> + setcc(cpu, 0);
>> + }
>> + break;
>> + case 1: /* Vertical polarization is not supported */
>> + if (s390_topology.polarity != S390_TOPOLOGY_POLARITY_HORIZONTAL) {
>> + env->regs[r1] |= S390_PTF_REASON_DONE;
>> + setcc(cpu, 2);
>> + } else {
>> + s390_topology_set_polarity(S390_TOPOLOGY_POLARITY_VERTICAL_LOW);
>
> This is why I said it's confusing, nothing gets set to LOW.
>
>> + s390_topology.polarity = S390_TOPOLOGY_POLARITY_VERTICAL_LOW;
>
> Why LOW here?
I wanted something not being S390_TOPOLOGY_POLARITY_HORIZONTAL and did
not want to define a S390_TOPOLOGY_POLARITY_VERTICAL.
OK I define S390_TOPOLOGY_POLARITY_HORIZONTAL=0 and ..._VERTICAL=1
>
>> + setcc(cpu, 0);
>> + }
>> + break;
>> + default:
>> + /* Note that fc == 2 is interpreted by the SIE */
>> + s390_program_interrupt(env, PGM_SPECIFICATION, ra);
>> + }
>
> You can simplify this by doing:
>
> int new_polarity;
> switch (fc) {
> case 0:
> new_polarity = S390_TOPOLOGY_POLARITY_HORIZONTAL;
> break;
> case 1:
> new_polarity = S390_TOPOLOGY_POLARITY_VERTICAL_?;
> break;
> default:
> /* Note that fc == 2 is interpreted by the SIE */
> s390_program_interrupt(env, PGM_SPECIFICATION, ra);
> return;
> }
>
> if same polarity:
> rc done, rejected
> else
> set polarity, initiated
>
> Might be a good idea to turn the polarity values into an enum.
>
>> +}
> [...]
>
Even I never really understood the added value of an enum I can do this.
Thanks,
regards,
Pierre
--
Pierre Morel
IBM Lab Boeblingen
^ permalink raw reply [flat|nested] 80+ messages in thread
* Re: [PATCH v14 07/11] target/s390x/cpu topology: activating CPU topology
2023-01-11 10:04 ` Thomas Huth
@ 2023-01-18 10:01 ` Pierre Morel
0 siblings, 0 replies; 80+ messages in thread
From: Pierre Morel @ 2023-01-18 10:01 UTC (permalink / raw)
To: Thomas Huth, qemu-s390x
Cc: qemu-devel, borntraeger, pasic, richard.henderson, david, cohuck,
mst, pbonzini, kvm, ehabkost, marcel.apfelbaum, eblake, armbru,
seiden, nrb, scgl, frankja, berrange, clg
On 1/11/23 11:04, Thomas Huth wrote:
> On 05/01/2023 15.53, Pierre Morel wrote:
>> The KVM capability, KVM_CAP_S390_CPU_TOPOLOGY is used to
>
> Remove the "," in above line?
OK
>
>> activate the S390_FEAT_CONFIGURATION_TOPOLOGY feature and
>> the topology facility for the guest in the case the topology
>
> I'd like to suggest to add "in the host CPU model" after "facility".
Yes, thanks.
>
>> is available in QEMU and in KVM.
>>
>> The feature is disabled by default and fenced for SE
>> (secure execution).
>>
>> Signed-off-by: Pierre Morel <pmorel@linux.ibm.com>
>> ---
>> hw/s390x/cpu-topology.c | 2 +-
>> target/s390x/cpu_models.c | 1 +
>> target/s390x/kvm/kvm.c | 13 +++++++++++++
>> 3 files changed, 15 insertions(+), 1 deletion(-)
>>
>> diff --git a/hw/s390x/cpu-topology.c b/hw/s390x/cpu-topology.c
>> index e6b4692581..b69955a1cd 100644
>> --- a/hw/s390x/cpu-topology.c
>> +++ b/hw/s390x/cpu-topology.c
>> @@ -52,7 +52,7 @@ static int s390_socket_nb(s390_topology_id id)
>> */
>> bool s390_has_topology(void)
>> {
>> - return false;
>> + return s390_has_feat(S390_FEAT_CONFIGURATION_TOPOLOGY);
>> }
>> /**
>> diff --git a/target/s390x/cpu_models.c b/target/s390x/cpu_models.c
>> index c3a4f80633..3f05e05fd3 100644
>> --- a/target/s390x/cpu_models.c
>> +++ b/target/s390x/cpu_models.c
>> @@ -253,6 +253,7 @@ bool s390_has_feat(S390Feat feat)
>> case S390_FEAT_SIE_CMMA:
>> case S390_FEAT_SIE_PFMFI:
>> case S390_FEAT_SIE_IBS:
>> + case S390_FEAT_CONFIGURATION_TOPOLOGY:
>> return false;
>> break;
>> default:
>> diff --git a/target/s390x/kvm/kvm.c b/target/s390x/kvm/kvm.c
>> index fb63be41b7..4e2a2ff516 100644
>> --- a/target/s390x/kvm/kvm.c
>> +++ b/target/s390x/kvm/kvm.c
>> @@ -2470,6 +2470,19 @@ void kvm_s390_get_host_cpu_model(S390CPUModel
>> *model, Error **errp)
>> set_bit(S390_FEAT_UNPACK, model->features);
>> }
>> + /*
>> + * If we have support for CPU Topology prevent overrule
>> + * S390_FEAT_CONFIGURATION_TOPOLOGY with
>> S390_FEAT_DISABLE_CPU_TOPOLOGY
>
> That S390_FEAT_DISABLE_CPU_TOPOLOGY looks like a leftover from v12 ?
Right, sorry, I remove it and change the comment for:
/*
* If we have kernel support for CPU Topology indicate the
* configuration-topology facility.
*/
>
> Apart from that, patch looks fine to me now.
>
> Thomas
>
>
>> + * implemented in KVM, activate the CPU TOPOLOGY feature.
>> + */
>> + if (kvm_check_extension(kvm_state, KVM_CAP_S390_CPU_TOPOLOGY)) {
>> + if (kvm_vm_enable_cap(kvm_state, KVM_CAP_S390_CPU_TOPOLOGY,
>> 0) < 0) {
>> + error_setg(errp, "KVM: Error enabling
>> KVM_CAP_S390_CPU_TOPOLOGY");
>> + return;
>> + }
>> + set_bit(S390_FEAT_CONFIGURATION_TOPOLOGY, model->features);
>> + }
>> +
>> /* We emulate a zPCI bus and AEN, therefore we don't need HW
>> support */
>> set_bit(S390_FEAT_ZPCI, model->features);
>> set_bit(S390_FEAT_ADAPTER_EVENT_NOTIFICATION, model->features);
>
--
Pierre Morel
IBM Lab Boeblingen
^ permalink raw reply [flat|nested] 80+ messages in thread
* Re: [PATCH v14 03/11] target/s390x/cpu topology: handle STSI(15) and build the SYSIB
2023-01-17 16:56 ` Pierre Morel
@ 2023-01-18 10:26 ` Thomas Huth
2023-01-18 11:54 ` Nina Schoetterl-Glausch
0 siblings, 1 reply; 80+ messages in thread
From: Thomas Huth @ 2023-01-18 10:26 UTC (permalink / raw)
To: Pierre Morel, qemu-s390x, frankja
Cc: qemu-devel, borntraeger, pasic, richard.henderson, david, cohuck,
mst, pbonzini, kvm, ehabkost, marcel.apfelbaum, eblake, armbru,
seiden, nrb, scgl, berrange, clg
On 17/01/2023 17.56, Pierre Morel wrote:
>
>
> On 1/10/23 15:29, Thomas Huth wrote:
>> On 05/01/2023 15.53, Pierre Morel wrote:
>>> On interception of STSI(15.1.x) the System Information Block
>>> (SYSIB) is built from the list of pre-ordered topology entries.
>>>
>>> Signed-off-by: Pierre Morel <pmorel@linux.ibm.com>
>>> ---
...
>>> +typedef struct SysIBTl_container {
>>> + uint8_t nl;
>>> + uint8_t reserved[6];
>>> + uint8_t id;
>>> +} QEMU_PACKED QEMU_ALIGNED(8) SysIBTl_container;
>>> +QEMU_BUILD_BUG_ON(sizeof(SysIBTl_container) != 8);
>>> +
>>> +/* CPU type Topology List Entry */
>>> +typedef struct SysIBTl_cpu {
>>> + uint8_t nl;
>>> + uint8_t reserved0[3];
>>> + uint8_t reserved1:5;
>>> + uint8_t dedicated:1;
>>> + uint8_t polarity:2;
>>
>> Hmmm, yet another bitfield...
>
> Yes, this is the firmware interface.
> If it makes problem I can use masks and logic arithmetic
It depends ... if we are sure that this will ever only be used with KVM on
real s390x hardware, then bitfields are OK. If we think that this is
something that could be implemented in TCG, too, I'd scratch the bitfields
and use logic arithmetic instead...
I'm not too experienced with this CPU topology stuff, but it sounds like it
could be implemented in TCG without too much efforts one day, too, so I'd
rather go with the logic arithmetic immediately instead if it is not too
annoying for you right now.
>>> diff --git a/target/s390x/kvm/cpu_topology.c
>>> b/target/s390x/kvm/cpu_topology.c
>>> new file mode 100644
>>> index 0000000000..3831a3264c
>>> --- /dev/null
>>> +++ b/target/s390x/kvm/cpu_topology.c
>>> @@ -0,0 +1,136 @@
>>> +/*
>>> + * QEMU S390x CPU Topology
>>> + *
>>> + * Copyright IBM Corp. 2022
>>
>> Happy new year?
>
> So after Nina's comment what do I do?
> let it be 22 because I started last year or update because what is important
> is when it comes into mainline?
Honestly, I don't have a really good clue either... But keeping 2022 is
certainly fine for me, too.
Thomas
^ permalink raw reply [flat|nested] 80+ messages in thread
* Re: [PATCH v14 08/11] qapi/s390/cpu topology: change-topology monitor command
2023-01-17 13:31 ` Nina Schoetterl-Glausch
@ 2023-01-18 10:53 ` Thomas Huth
2023-01-18 14:09 ` Pierre Morel
2023-01-18 15:17 ` Kevin Wolf
0 siblings, 2 replies; 80+ messages in thread
From: Thomas Huth @ 2023-01-18 10:53 UTC (permalink / raw)
To: Nina Schoetterl-Glausch, Pierre Morel, qemu-s390x
Cc: qemu-devel, borntraeger, pasic, richard.henderson, david, cohuck,
mst, pbonzini, kvm, ehabkost, marcel.apfelbaum, eblake, armbru,
seiden, nrb, frankja, berrange, clg, Dr. David Alan Gilbert,
Stefan Hajnoczi, Eric Blake, Kevin Wolf, hreitz
On 17/01/2023 14.31, Nina Schoetterl-Glausch wrote:
> On Tue, 2023-01-17 at 08:30 +0100, Thomas Huth wrote:
>> On 16/01/2023 22.09, Nina Schoetterl-Glausch wrote:
>>> On Thu, 2023-01-05 at 15:53 +0100, Pierre Morel wrote:
>>>> The modification of the CPU attributes are done through a monitor
>>>> commands.
>>>>
>>>> It allows to move the core inside the topology tree to optimise
>>>> the cache usage in the case the host's hypervizor previously
>>>> moved the CPU.
>>>>
>>>> The same command allows to modifiy the CPU attributes modifiers
>>>> like polarization entitlement and the dedicated attribute to notify
>>>> the guest if the host admin modified scheduling or dedication of a vCPU.
>>>>
>>>> With this knowledge the guest has the possibility to optimize the
>>>> usage of the vCPUs.
>>>>
>>>> Signed-off-by: Pierre Morel <pmorel@linux.ibm.com>
>>>> ---
...
>>>> + s390_topology.sockets[s390_socket_nb(id)]--;
>>>
>>> I suppose this function cannot run concurrently, so the same CPU doesn't get removed twice.
>>
>> QEMU has the so-called BQL - the Big Qemu Lock. Instructions handlers are
>> normally called with the lock taken, see qemu_mutex_lock_iothread() in
>> target/s390x/kvm/kvm.c.
>
> That is good to know, but is that the relevant lock here?
> We don't want to concurrent qmp commands. I looked at the code and it's pretty complicated.
Not sure, but I believe that QMP commands are executed from the main
iothread, so I think this should be safe? ... CC:-ing some more people who
might know the correct answer.
Thomas
^ permalink raw reply [flat|nested] 80+ messages in thread
* Re: [PATCH v14 03/11] target/s390x/cpu topology: handle STSI(15) and build the SYSIB
2023-01-18 10:26 ` Thomas Huth
@ 2023-01-18 11:54 ` Nina Schoetterl-Glausch
2023-01-19 13:12 ` Pierre Morel
0 siblings, 1 reply; 80+ messages in thread
From: Nina Schoetterl-Glausch @ 2023-01-18 11:54 UTC (permalink / raw)
To: Thomas Huth, Pierre Morel, qemu-s390x, frankja
Cc: qemu-devel, borntraeger, pasic, richard.henderson, david, cohuck,
mst, pbonzini, kvm, ehabkost, marcel.apfelbaum, eblake, armbru,
seiden, nrb, berrange, clg
On Wed, 2023-01-18 at 11:26 +0100, Thomas Huth wrote:
> On 17/01/2023 17.56, Pierre Morel wrote:
> >
> >
> > On 1/10/23 15:29, Thomas Huth wrote:
> > > On 05/01/2023 15.53, Pierre Morel wrote:
> > > > On interception of STSI(15.1.x) the System Information Block
> > > > (SYSIB) is built from the list of pre-ordered topology entries.
> > > >
> > > > Signed-off-by: Pierre Morel <pmorel@linux.ibm.com>
> > > > ---
> ...
> > > > +typedef struct SysIBTl_container {
> > > > + uint8_t nl;
> > > > + uint8_t reserved[6];
> > > > + uint8_t id;
> > > > +} QEMU_PACKED QEMU_ALIGNED(8) SysIBTl_container;
> > > > +QEMU_BUILD_BUG_ON(sizeof(SysIBTl_container) != 8);
> > > > +
> > > > +/* CPU type Topology List Entry */
> > > > +typedef struct SysIBTl_cpu {
> > > > + uint8_t nl;
> > > > + uint8_t reserved0[3];
> > > > + uint8_t reserved1:5;
> > > > + uint8_t dedicated:1;
> > > > + uint8_t polarity:2;
> > >
> > > Hmmm, yet another bitfield...
> >
> > Yes, this is the firmware interface.
> > If it makes problem I can use masks and logic arithmetic
>
> It depends ... if we are sure that this will ever only be used with KVM on
> real s390x hardware, then bitfields are OK. If we think that this is
> something that could be implemented in TCG, too, I'd scratch the bitfields
> and use logic arithmetic instead...
Is there something like linux' bitfield.h in qemu?
In this case it's only two fields, and not too complicated, but I imagine it could
get quite ugly to do it manually in other cases.
>
> I'm not too experienced with this CPU topology stuff, but it sounds like it
> could be implemented in TCG without too much efforts one day, too, so I'd
> rather go with the logic arithmetic immediately instead if it is not too
> annoying for you right now.
>
> > > > diff --git a/target/s390x/kvm/cpu_topology.c
> > > > b/target/s390x/kvm/cpu_topology.c
> > > > new file mode 100644
> > > > index 0000000000..3831a3264c
> > > > --- /dev/null
> > > > +++ b/target/s390x/kvm/cpu_topology.c
> > > > @@ -0,0 +1,136 @@
> > > > +/*
> > > > + * QEMU S390x CPU Topology
> > > > + *
> > > > + * Copyright IBM Corp. 2022
> > >
> > > Happy new year?
> >
> > So after Nina's comment what do I do?
> > let it be 22 because I started last year or update because what is important
> > is when it comes into mainline?
>
> Honestly, I don't have a really good clue either... But keeping 2022 is
> certainly fine for me, too.
>
> Thomas
>
^ permalink raw reply [flat|nested] 80+ messages in thread
* Re: [PATCH v14 08/11] qapi/s390/cpu topology: change-topology monitor command
2023-01-12 12:03 ` Daniel P. Berrangé
@ 2023-01-18 13:17 ` Pierre Morel
0 siblings, 0 replies; 80+ messages in thread
From: Pierre Morel @ 2023-01-18 13:17 UTC (permalink / raw)
To: Daniel P. Berrangé
Cc: qemu-s390x, qemu-devel, borntraeger, pasic, richard.henderson,
david, thuth, cohuck, mst, pbonzini, kvm, ehabkost,
marcel.apfelbaum, eblake, armbru, seiden, nrb, scgl, frankja,
clg
On 1/12/23 13:03, Daniel P. Berrangé wrote:
> On Thu, Jan 05, 2023 at 03:53:10PM +0100, Pierre Morel wrote:
>> The modification of the CPU attributes are done through a monitor
>> commands.
>>
>> It allows to move the core inside the topology tree to optimise
>> the cache usage in the case the host's hypervizor previously
>> moved the CPU.
>>
>> The same command allows to modifiy the CPU attributes modifiers
>> like polarization entitlement and the dedicated attribute to notify
>> the guest if the host admin modified scheduling or dedication of a vCPU.
>>
>> With this knowledge the guest has the possibility to optimize the
>> usage of the vCPUs.
>>
>> Signed-off-by: Pierre Morel <pmorel@linux.ibm.com>
>> ---
>> qapi/machine-target.json | 29 ++++++++
>> include/monitor/hmp.h | 1 +
>> hw/s390x/cpu-topology.c | 141 +++++++++++++++++++++++++++++++++++++++
>> hmp-commands.hx | 16 +++++
>> 4 files changed, 187 insertions(+)
>>
>> diff --git a/qapi/machine-target.json b/qapi/machine-target.json
>> index 2e267fa458..75b0aa254d 100644
>> --- a/qapi/machine-target.json
>> +++ b/qapi/machine-target.json
>> @@ -342,3 +342,32 @@
>> 'TARGET_S390X',
>> 'TARGET_MIPS',
>> 'TARGET_LOONGARCH64' ] } }
>> +
>> +##
>> +# @change-topology:
>> +#
>> +# @core: the vCPU ID to be moved
>> +# @socket: the destination socket where to move the vCPU
>> +# @book: the destination book where to move the vCPU
>> +# @drawer: the destination drawer where to move the vCPU
>
> This movement can be done while the guest OS is running ?
> What happens to guest OS apps ? Every I know will read
> topology once and assume it never changes at runtime.
Yes this can change while the guest is running.
The S390 Logical PARtition, where the Linux runs is already a first
level of virtualization and the lpar CPU are already virtual CPU which
can be moved from one real CPU to another, the guest is at a second
level of virtualization.
On the LPAR host an admin can check the topology.
A lpar CPU can be moved to another real CPU because of multiple reasons:
maintenance, failure, other decision from the first level hypervisor
that I do not know, may be scheduling balancing.
There is a mechanism for the OS in which is running in LPAR to set a
flag for the guest on a topology change.
The guest use a specific instruction to get this flag.
This instruction PTF(2) is interpreted by the firmware and does not
appear in this patch series but in Linux patch series.
So we have, real CPU <-> lpar CPU <-> vCPU
>
> What's the use case for wanting to re-arrange topology in
> this manner ? It feels like its going to be a recipe for
> hard to diagnose problems, as much code in libvirt and apps
> above will assuming the vCPU IDs are assigned sequentially
> starting from node=0,book=0,drawer=0,socket=0,core=0,
> incrementing core, then incrementing socket, then
> incrementing drawer, etc.
The goal to rearrange the vCPU is to give the guest the knowledge of the
topology so it can takes benefit of it.
If a lpar CPU moved to another real CPU in another drawer we must move
the guest vCPU to another drawer so the guest OS can take the best
scheduling decisions.
Per default, if nothing is specified on the creation of a vCPU, the
creation is done exactly like you said, starting from (0,0,0,0) and
incrementing.
There are two possibility to set a vCPU at its place:
1) on creation by specifying the drawer,book,socket for a specific core-id
2) with this QAPI command to move the CPU while it is running.
Note that the core-id and the CPU address do not change when moving the
CPU so that there is no problem with scheduling, all we do is to provide
the topology up to the guest when it asks.
The period of checking by the Linux kernel if there is a change and if
there is a need to ask the topology is one minute.
The migration of CPU is not supposed to happen very often, (not every day).
>
>> +# @polarity: optional polarity, default is last polarity set by the guest
>> +# @dedicated: optional, if the vCPU is dedicated to a real CPU
>> +#
>> +# Modifies the topology by moving the CPU inside the topology
>> +# tree or by changing a modifier attribute of a CPU.
>> +#
>> +# Returns: Nothing on success, the reason on failure.
>> +#
>> +# Since: <next qemu stable release, eg. 1.0>
>> +##
>> +{ 'command': 'change-topology',
>
> 'set-cpu-topology'
OK, yes looks better.
>
>> + 'data': {
>> + 'core': 'int',
>> + 'socket': 'int',
>> + 'book': 'int',
>> + 'drawer': 'int',
>> + '*polarity': 'int',
>> + '*dedicated': 'bool'
>> + },
>> + 'if': { 'all': [ 'TARGET_S390X', 'CONFIG_KVM' ] }
>> +}
>
>
> With regards,
> Daniel
Thanks,
Regards,
Pierre
--
Pierre Morel
IBM Lab Boeblingen
^ permalink raw reply [flat|nested] 80+ messages in thread
* Re: [PATCH v14 08/11] qapi/s390/cpu topology: change-topology monitor command
2023-01-16 21:09 ` Nina Schoetterl-Glausch
2023-01-17 7:30 ` Thomas Huth
@ 2023-01-18 14:06 ` Pierre Morel
1 sibling, 0 replies; 80+ messages in thread
From: Pierre Morel @ 2023-01-18 14:06 UTC (permalink / raw)
To: Nina Schoetterl-Glausch, qemu-s390x
Cc: qemu-devel, borntraeger, pasic, richard.henderson, david, thuth,
cohuck, mst, pbonzini, kvm, ehabkost, marcel.apfelbaum, eblake,
armbru, seiden, nrb, frankja, berrange, clg
On 1/16/23 22:09, Nina Schoetterl-Glausch wrote:
> On Thu, 2023-01-05 at 15:53 +0100, Pierre Morel wrote:
>> The modification of the CPU attributes are done through a monitor
>> commands.
>>
>> It allows to move the core inside the topology tree to optimise
>> the cache usage in the case the host's hypervizor previously
>> moved the CPU.
>>
>> The same command allows to modifiy the CPU attributes modifiers
>> like polarization entitlement and the dedicated attribute to notify
>> the guest if the host admin modified scheduling or dedication of a vCPU.
>>
>> With this knowledge the guest has the possibility to optimize the
>> usage of the vCPUs.
>>
>> Signed-off-by: Pierre Morel <pmorel@linux.ibm.com>
>> ---
>> qapi/machine-target.json | 29 ++++++++
>> include/monitor/hmp.h | 1 +
>> hw/s390x/cpu-topology.c | 141 +++++++++++++++++++++++++++++++++++++++
>> hmp-commands.hx | 16 +++++
>> 4 files changed, 187 insertions(+)
>>
>> diff --git a/qapi/machine-target.json b/qapi/machine-target.json
>> index 2e267fa458..75b0aa254d 100644
>> --- a/qapi/machine-target.json
>> +++ b/qapi/machine-target.json
>> @@ -342,3 +342,32 @@
>> 'TARGET_S390X',
>> 'TARGET_MIPS',
>> 'TARGET_LOONGARCH64' ] } }
>> +
>> +##
>> +# @change-topology:
>> +#
>> +# @core: the vCPU ID to be moved
>> +# @socket: the destination socket where to move the vCPU
>> +# @book: the destination book where to move the vCPU
>> +# @drawer: the destination drawer where to move the vCPU
>> +# @polarity: optional polarity, default is last polarity set by the guest
>> +# @dedicated: optional, if the vCPU is dedicated to a real CPU
>> +#
>> +# Modifies the topology by moving the CPU inside the topology
>> +# tree or by changing a modifier attribute of a CPU.
>> +#
>> +# Returns: Nothing on success, the reason on failure.
>> +#
>> +# Since: <next qemu stable release, eg. 1.0>
>> +##
>> +{ 'command': 'change-topology',
>> + 'data': {
>> + 'core': 'int',
>> + 'socket': 'int',
>> + 'book': 'int',
>> + 'drawer': 'int',
>> + '*polarity': 'int',
>> + '*dedicated': 'bool'
>> + },
>> + 'if': { 'all': [ 'TARGET_S390X', 'CONFIG_KVM' ] }
>> +}
>> diff --git a/include/monitor/hmp.h b/include/monitor/hmp.h
>> index 27f86399f7..15c36bf549 100644
>> --- a/include/monitor/hmp.h
>> +++ b/include/monitor/hmp.h
>> @@ -144,5 +144,6 @@ void hmp_human_readable_text_helper(Monitor *mon,
>> HumanReadableText *(*qmp_handler)(Error **));
>> void hmp_info_stats(Monitor *mon, const QDict *qdict);
>> void hmp_pcie_aer_inject_error(Monitor *mon, const QDict *qdict);
>> +void hmp_change_topology(Monitor *mon, const QDict *qdict);
>>
>> #endif
>> diff --git a/hw/s390x/cpu-topology.c b/hw/s390x/cpu-topology.c
>> index b69955a1cd..0faffe657e 100644
>> --- a/hw/s390x/cpu-topology.c
>> +++ b/hw/s390x/cpu-topology.c
>> @@ -18,6 +18,10 @@
>> #include "target/s390x/cpu.h"
>> #include "hw/s390x/s390-virtio-ccw.h"
>> #include "hw/s390x/cpu-topology.h"
>> +#include "qapi/qapi-commands-machine-target.h"
>> +#include "qapi/qmp/qdict.h"
>> +#include "monitor/hmp.h"
>> +#include "monitor/monitor.h"
>>
>> /*
>> * s390_topology is used to keep the topology information.
>> @@ -203,6 +207,21 @@ static void s390_topology_set_entry(S390TopologyEntry *entry,
>> s390_topology.sockets[s390_socket_nb(id)]++;
>> }
>>
>> +/**
>> + * s390_topology_clear_entry:
>> + * @entry: Topology entry to setup
>> + * @id: topology id to use for the setup
>> + *
>> + * Clear the core bit inside the topology mask and
>> + * decrements the number of cores for the socket.
>> + */
>> +static void s390_topology_clear_entry(S390TopologyEntry *entry,
>> + s390_topology_id id)
>> +{
>> + clear_bit(63 - id.core, &entry->mask);
>
> This doesn't take the origin into account.
Yes, this error has been done everywhere, I modify this.
>
>> + s390_topology.sockets[s390_socket_nb(id)]--;
>
> I suppose this function cannot run concurrently, so the same CPU doesn't get removed twice.
This is only called on a change of topology and the qapi commands are
serialized by the BQL.
>
>> +}
>> +
>> /**
>> * s390_topology_new_entry:
>> * @id: s390_topology_id to add
>> @@ -383,3 +402,125 @@ void s390_topology_set_cpu(MachineState *ms, S390CPU *cpu, Error **errp)
>>
>> s390_topology_insert(id);
>> }
>> +
>> +/*
>> + * qmp and hmp implementations
>> + */
>> +
>> +static S390TopologyEntry *s390_topology_core_to_entry(int core)
>> +{
>> + S390TopologyEntry *entry;
>> +
>> + QTAILQ_FOREACH(entry, &s390_topology.list, next) {
>> + if (entry->mask & (1UL << (63 - core))) {
>
> origin here also.
yes
>
>> + return entry;
>> + }
>> + }
>> + return NULL;
>
> This should not return NULL unless the core id is invalid.
yes
> Might be better to validate that somewhere else.
How?
I can use CPU_FOREACH() but it would be quite the same and I will need
to rerun a loop for setting the topology anyway.
>
>> +}
>> +
>> +static void s390_change_topology(Error **errp, int64_t core, int64_t socket,
>> + int64_t book, int64_t drawer,
>> + int64_t polarity, bool dedicated)
>> +{
>> + S390TopologyEntry *entry;
>> + s390_topology_id new_id;
>> + s390_topology_id old_id;
>> + Error *local_error = NULL;
>
> I think you could use ERRP_GUARD here also.
You are right, I could.
Should I ?
I do not need it and find it obfuscating but if you insist I can do it.
>> +
>> + /* Get the old entry */
>> + entry = s390_topology_core_to_entry(core);
>> + if (!entry) {
>> + error_setg(errp, "No core %ld", core);
>> + return;
>> + }
>> +
>> + /* Compute old topology id */
>> + old_id = entry->id;
>> + old_id.core = core;
>> +
>> + /* Compute new topology id */
>> + new_id = entry->id;
>> + new_id.core = core;
>> + new_id.socket = socket;
>> + new_id.book = book;
>> + new_id.drawer = drawer;
>> + new_id.p = polarity;
>> + new_id.d = dedicated;
>> + new_id.type = S390_TOPOLOGY_CPU_IFL;
>> +
>> + /* Same topology entry, nothing to do */
>> + if (entry->id.id == new_id.id) {
>> + return;
>> + }
>> +
>> + /* Check for space on the socket if ids are different */
>> + if ((s390_socket_nb(old_id) != s390_socket_nb(new_id)) &&
>> + (s390_topology.sockets[s390_socket_nb(new_id)] >=
>> + s390_topology.smp->sockets)) {
>> + error_setg(errp, "No more space on this socket");
>> + return;
>> + }
>> +
>> + /* Verify the new topology */
>> + s390_topology_check(&local_error, new_id);
>> + if (local_error) {
>> + error_propagate(errp, local_error);
>> + return;
>> + }
>> +
>> + /* Clear the old topology */
>> + s390_topology_clear_entry(entry, old_id);
>> +
>> + /* Insert the new topology */
>> + s390_topology_insert(new_id);
>> +
>> + /* Remove unused entry */
>> + if (!entry->mask) {
>> + QTAILQ_REMOVE(&s390_topology.list, entry, next);
>> + g_free(entry);
>> + }
>> +
>> + /* Advertise the topology change */
>> + s390_cpu_topology_set();
>> +}
>> +
>> +void qmp_change_topology(int64_t core, int64_t socket,
>> + int64_t book, int64_t drawer,
>> + bool has_polarity, int64_t polarity,
>> + bool has_dedicated, bool dedicated,
>> + Error **errp)
>> +{
>> + Error *local_err = NULL;
>> +
>> + if (!s390_has_topology()) {
>> + error_setg(&local_err, "This machine doesn't support topology");
>> + return;
>> + }
>
> Do you really want to ignore has_polarity and has_dedicated here?
> What happens in this case?
no, seems I forgot to check it.
Thanks,
Regards,
Pierre
--
Pierre Morel
IBM Lab Boeblingen
^ permalink raw reply [flat|nested] 80+ messages in thread
* Re: [PATCH v14 08/11] qapi/s390/cpu topology: change-topology monitor command
2023-01-18 10:53 ` Thomas Huth
@ 2023-01-18 14:09 ` Pierre Morel
2023-01-18 15:17 ` Kevin Wolf
1 sibling, 0 replies; 80+ messages in thread
From: Pierre Morel @ 2023-01-18 14:09 UTC (permalink / raw)
To: Thomas Huth, Nina Schoetterl-Glausch, qemu-s390x
Cc: qemu-devel, borntraeger, pasic, richard.henderson, david, cohuck,
mst, pbonzini, kvm, ehabkost, marcel.apfelbaum, eblake, armbru,
seiden, nrb, frankja, berrange, clg, Dr. David Alan Gilbert,
Stefan Hajnoczi, Kevin Wolf, hreitz
On 1/18/23 11:53, Thomas Huth wrote:
> On 17/01/2023 14.31, Nina Schoetterl-Glausch wrote:
>> On Tue, 2023-01-17 at 08:30 +0100, Thomas Huth wrote:
>>> On 16/01/2023 22.09, Nina Schoetterl-Glausch wrote:
>>>> On Thu, 2023-01-05 at 15:53 +0100, Pierre Morel wrote:
>>>>> The modification of the CPU attributes are done through a monitor
>>>>> commands.
>>>>>
>>>>> It allows to move the core inside the topology tree to optimise
>>>>> the cache usage in the case the host's hypervizor previously
>>>>> moved the CPU.
>>>>>
>>>>> The same command allows to modifiy the CPU attributes modifiers
>>>>> like polarization entitlement and the dedicated attribute to notify
>>>>> the guest if the host admin modified scheduling or dedication of a
>>>>> vCPU.
>>>>>
>>>>> With this knowledge the guest has the possibility to optimize the
>>>>> usage of the vCPUs.
>>>>>
>>>>> Signed-off-by: Pierre Morel <pmorel@linux.ibm.com>
>>>>> ---
> ...
>>>>> + s390_topology.sockets[s390_socket_nb(id)]--;
>>>>
>>>> I suppose this function cannot run concurrently, so the same CPU
>>>> doesn't get removed twice.
>>>
>>> QEMU has the so-called BQL - the Big Qemu Lock. Instructions handlers
>>> are
>>> normally called with the lock taken, see qemu_mutex_lock_iothread() in
>>> target/s390x/kvm/kvm.c.
>>
>> That is good to know, but is that the relevant lock here?
>> We don't want to concurrent qmp commands. I looked at the code and
>> it's pretty complicated.
>
> Not sure, but I believe that QMP commands are executed from the main
> iothread, so I think this should be safe? ... CC:-ing some more people
> who might know the correct answer.
>
> Thomas
>
That is what I understand from the doc too.
See:
docs/devel/qapi-code-gen.rst
One must specify allow-oob:true to avoid the BQL, default is false.
Regards,
Pierre
--
Pierre Morel
IBM Lab Boeblingen
^ permalink raw reply [flat|nested] 80+ messages in thread
* Re: [PATCH v14 08/11] qapi/s390/cpu topology: change-topology monitor command
2023-01-11 10:09 ` Thomas Huth
2023-01-12 8:00 ` Thomas Huth
@ 2023-01-18 14:23 ` Pierre Morel
1 sibling, 0 replies; 80+ messages in thread
From: Pierre Morel @ 2023-01-18 14:23 UTC (permalink / raw)
To: Thomas Huth, qemu-s390x
Cc: qemu-devel, borntraeger, pasic, richard.henderson, david, cohuck,
mst, pbonzini, kvm, ehabkost, marcel.apfelbaum, eblake, armbru,
seiden, nrb, scgl, frankja, berrange, clg
On 1/11/23 11:09, Thomas Huth wrote:
> On 05/01/2023 15.53, Pierre Morel wrote:
>> The modification of the CPU attributes are done through a monitor
>> commands.
>
> s/commands/command/
thx
>
>> It allows to move the core inside the topology tree to optimise
>> the cache usage in the case the host's hypervizor previously
>
> s/hypervizor/hypervisor/
yes, thx
>
>> moved the CPU.
>>
>> The same command allows to modifiy the CPU attributes modifiers
>
> s/modifiy/modify/
thx
>
>> like polarization entitlement and the dedicated attribute to notify
>> the guest if the host admin modified scheduling or dedication of a vCPU.
>>
>> With this knowledge the guest has the possibility to optimize the
>> usage of the vCPUs.
>
> Hmm, who is supposed to call this QMP command in the future? Will there
> be a new daemon monitoring the CPU changes in the host? Or will there be
> a libvirt addition for this? ... Seems like I still miss the big picture
> here...
>
> Thomas
>
The first idea is to provide a daemon that could get the information on
real CPU from the host sysfs and to specify the vCPU topology according
to the real CPU.
There could be a libvirt command for this too.
The big picture is to provide the guest OS with the real topology so
that the guest OS can make decisions on the scheduling.
I think that a daemon is perfect I can not imagine anything else than
the alternative:
1) Do not specify anything and let things go more or less random as
today by setting the cores in socket,book,drawer in an incremental way.
2) specifying the exact topology
So I do not see the point to let the user or even libvirt specify a
random topology if it is specified it must be exact.
Regards,
Pierre
--
Pierre Morel
IBM Lab Boeblingen
^ permalink raw reply [flat|nested] 80+ messages in thread
* Re: [PATCH v14 08/11] qapi/s390/cpu topology: change-topology monitor command
2023-01-18 10:53 ` Thomas Huth
2023-01-18 14:09 ` Pierre Morel
@ 2023-01-18 15:17 ` Kevin Wolf
2023-01-18 15:48 ` Pierre Morel
1 sibling, 1 reply; 80+ messages in thread
From: Kevin Wolf @ 2023-01-18 15:17 UTC (permalink / raw)
To: Thomas Huth
Cc: Nina Schoetterl-Glausch, Pierre Morel, qemu-s390x, qemu-devel,
borntraeger, pasic, richard.henderson, david, cohuck, mst,
pbonzini, kvm, ehabkost, marcel.apfelbaum, eblake, armbru,
seiden, nrb, frankja, berrange, clg, Dr. David Alan Gilbert,
Stefan Hajnoczi, hreitz
Am 18.01.2023 um 11:53 hat Thomas Huth geschrieben:
> On 17/01/2023 14.31, Nina Schoetterl-Glausch wrote:
> > On Tue, 2023-01-17 at 08:30 +0100, Thomas Huth wrote:
> > > On 16/01/2023 22.09, Nina Schoetterl-Glausch wrote:
> > > > On Thu, 2023-01-05 at 15:53 +0100, Pierre Morel wrote:
> > > > > The modification of the CPU attributes are done through a monitor
> > > > > commands.
> > > > >
> > > > > It allows to move the core inside the topology tree to optimise
> > > > > the cache usage in the case the host's hypervizor previously
> > > > > moved the CPU.
> > > > >
> > > > > The same command allows to modifiy the CPU attributes modifiers
> > > > > like polarization entitlement and the dedicated attribute to notify
> > > > > the guest if the host admin modified scheduling or dedication of a vCPU.
> > > > >
> > > > > With this knowledge the guest has the possibility to optimize the
> > > > > usage of the vCPUs.
> > > > >
> > > > > Signed-off-by: Pierre Morel <pmorel@linux.ibm.com>
> > > > > ---
> ...
> > > > > + s390_topology.sockets[s390_socket_nb(id)]--;
> > > >
> > > > I suppose this function cannot run concurrently, so the same CPU doesn't get removed twice.
> > >
> > > QEMU has the so-called BQL - the Big Qemu Lock. Instructions handlers are
> > > normally called with the lock taken, see qemu_mutex_lock_iothread() in
> > > target/s390x/kvm/kvm.c.
> >
> > That is good to know, but is that the relevant lock here?
> > We don't want to concurrent qmp commands. I looked at the code and it's pretty complicated.
>
> Not sure, but I believe that QMP commands are executed from the main
> iothread, so I think this should be safe? ... CC:-ing some more people who
> might know the correct answer.
In general yes, QMP commands are processed one after another in the main
thread while holding the BQL. And I think this is the relevant case for
you.
The exception is out-of-band commands, which I think run in the monitor
thread while some other (normal) command could be processed. OOB
commands are quite limited in what they are allowed to do, though, and
there aren't many of them. They are mainly meant to fix situations where
something (including other QMP commands) got stuck.
Kevin
^ permalink raw reply [flat|nested] 80+ messages in thread
* Re: [PATCH v14 08/11] qapi/s390/cpu topology: change-topology monitor command
2023-01-18 15:17 ` Kevin Wolf
@ 2023-01-18 15:48 ` Pierre Morel
0 siblings, 0 replies; 80+ messages in thread
From: Pierre Morel @ 2023-01-18 15:48 UTC (permalink / raw)
To: Kevin Wolf, Thomas Huth
Cc: Nina Schoetterl-Glausch, qemu-s390x, qemu-devel, borntraeger,
pasic, richard.henderson, david, cohuck, mst, pbonzini, kvm,
ehabkost, marcel.apfelbaum, eblake, armbru, seiden, nrb, frankja,
berrange, clg, Dr. David Alan Gilbert, Stefan Hajnoczi, hreitz
On 1/18/23 16:17, Kevin Wolf wrote:
> Am 18.01.2023 um 11:53 hat Thomas Huth geschrieben:
>> On 17/01/2023 14.31, Nina Schoetterl-Glausch wrote:
>>> On Tue, 2023-01-17 at 08:30 +0100, Thomas Huth wrote:
>>>> On 16/01/2023 22.09, Nina Schoetterl-Glausch wrote:
>>>>> On Thu, 2023-01-05 at 15:53 +0100, Pierre Morel wrote:
>>>>>> The modification of the CPU attributes are done through a monitor
>>>>>> commands.
>>>>>>
>>>>>> It allows to move the core inside the topology tree to optimise
>>>>>> the cache usage in the case the host's hypervizor previously
>>>>>> moved the CPU.
>>>>>>
>>>>>> The same command allows to modifiy the CPU attributes modifiers
>>>>>> like polarization entitlement and the dedicated attribute to notify
>>>>>> the guest if the host admin modified scheduling or dedication of a vCPU.
>>>>>>
>>>>>> With this knowledge the guest has the possibility to optimize the
>>>>>> usage of the vCPUs.
>>>>>>
>>>>>> Signed-off-by: Pierre Morel <pmorel@linux.ibm.com>
>>>>>> ---
>> ...
>>>>>> + s390_topology.sockets[s390_socket_nb(id)]--;
>>>>>
>>>>> I suppose this function cannot run concurrently, so the same CPU doesn't get removed twice.
>>>>
>>>> QEMU has the so-called BQL - the Big Qemu Lock. Instructions handlers are
>>>> normally called with the lock taken, see qemu_mutex_lock_iothread() in
>>>> target/s390x/kvm/kvm.c.
>>>
>>> That is good to know, but is that the relevant lock here?
>>> We don't want to concurrent qmp commands. I looked at the code and it's pretty complicated.
>>
>> Not sure, but I believe that QMP commands are executed from the main
>> iothread, so I think this should be safe? ... CC:-ing some more people who
>> might know the correct answer.
>
> In general yes, QMP commands are processed one after another in the main
> thread while holding the BQL. And I think this is the relevant case for
> you.
>
> The exception is out-of-band commands, which I think run in the monitor
> thread while some other (normal) command could be processed. OOB
> commands are quite limited in what they are allowed to do, though, and
> there aren't many of them. They are mainly meant to fix situations where
> something (including other QMP commands) got stuck.
>
> Kevin
>
Thanks Kevin,
regards,
Pierre
--
Pierre Morel
IBM Lab Boeblingen
^ permalink raw reply [flat|nested] 80+ messages in thread
* Re: [PATCH v14 09/11] qapi/s390/cpu topology: monitor query topology information
2023-01-12 12:10 ` Daniel P. Berrangé
2023-01-12 17:27 ` Nina Schoetterl-Glausch
@ 2023-01-18 15:58 ` Pierre Morel
2023-01-18 16:08 ` Daniel P. Berrangé
1 sibling, 1 reply; 80+ messages in thread
From: Pierre Morel @ 2023-01-18 15:58 UTC (permalink / raw)
To: Daniel P. Berrangé
Cc: qemu-s390x, qemu-devel, borntraeger, pasic, richard.henderson,
david, thuth, cohuck, mst, pbonzini, kvm, ehabkost,
marcel.apfelbaum, eblake, armbru, seiden, nrb, scgl, frankja,
clg
On 1/12/23 13:10, Daniel P. Berrangé wrote:
> On Thu, Jan 05, 2023 at 03:53:11PM +0100, Pierre Morel wrote:
>> Reporting the current topology informations to the admin through
>> the QEMU monitor.
>>
>> Signed-off-by: Pierre Morel <pmorel@linux.ibm.com>
>> ---
>> qapi/machine-target.json | 66 ++++++++++++++++++++++++++++++++++
>> include/monitor/hmp.h | 1 +
>> hw/s390x/cpu-topology.c | 76 ++++++++++++++++++++++++++++++++++++++++
>> hmp-commands-info.hx | 16 +++++++++
>> 4 files changed, 159 insertions(+)
>>
>> diff --git a/qapi/machine-target.json b/qapi/machine-target.json
>> index 75b0aa254d..927618a78f 100644
>> --- a/qapi/machine-target.json
>> +++ b/qapi/machine-target.json
>> @@ -371,3 +371,69 @@
>> },
>> 'if': { 'all': [ 'TARGET_S390X', 'CONFIG_KVM' ] }
>> }
>> +
>> +##
>> +# @S390CpuTopology:
>> +#
>> +# CPU Topology information
>> +#
>> +# @drawer: the destination drawer where to move the vCPU
>> +#
>> +# @book: the destination book where to move the vCPU
>> +#
>> +# @socket: the destination socket where to move the vCPU
>> +#
>> +# @polarity: optional polarity, default is last polarity set by the guest
>> +#
>> +# @dedicated: optional, if the vCPU is dedicated to a real CPU
>> +#
>> +# @origin: offset of the first bit of the core mask
>> +#
>> +# @mask: mask of the cores sharing the same topology
>> +#
>> +# Since: 8.0
>> +##
>> +{ 'struct': 'S390CpuTopology',
>> + 'data': {
>> + 'drawer': 'int',
>> + 'book': 'int',
>> + 'socket': 'int',
>> + 'polarity': 'int',
>> + 'dedicated': 'bool',
>> + 'origin': 'int',
>> + 'mask': 'str'
>> + },
>> + 'if': { 'all': [ 'TARGET_S390X', 'CONFIG_KVM' ] }
>> +}
>> +
>> +##
>> +# @query-topology:
>> +#
>> +# Return information about CPU Topology
>> +#
>> +# Returns a @CpuTopology instance describing the CPU Toplogy
>> +# being currently used by QEMU.
>> +#
>> +# Since: 8.0
>> +#
>> +# Example:
>> +#
>> +# -> { "execute": "cpu-topology" }
>> +# <- {"return": [
>> +# {
>> +# "drawer": 0,
>> +# "book": 0,
>> +# "socket": 0,
>> +# "polarity": 0,
>> +# "dedicated": true,
>> +# "origin": 0,
>> +# "mask": 0xc000000000000000,
>> +# },
>> +# ]
>> +# }
>> +#
>> +##
>> +{ 'command': 'query-topology',
>> + 'returns': ['S390CpuTopology'],
>> + 'if': { 'all': [ 'TARGET_S390X', 'CONFIG_KVM' ] }
>> +}
>
> IIUC, you're using @mask as a way to compress the array returned
> from query-topology, so that it doesn't have any repeated elements
> with the same data. I guess I can understand that desire when the
> core count can get very large, this can have a large saving.
>
> The downside of using @mask, is that now you require the caller
> to parse the string to turn it into a bitmask and expand the
> data. Generally this is considered a bit of an anti-pattern in
> QAPI design - we don't want callers to have to further parse
> the data to extract information, we want to directly consumable
> from the parsed JSON doc.
Not exactly, the mask is computed by the firmware to provide it to the
guest and is already available when querying the topology.
But I understand that for the QAPI user the mask is not the right
solution, standard coma separated values like (1,3,5,7-11) would be much
easier to read.
>
> We already have 'query-cpus-fast' wich returns one entry for
> each CPU. In fact why do we need to add query-topology at all.
> Can't we just add book-id / drawer-id / polarity / dedicated
> to the query-cpus-fast result ?
Yes we can, I think we should, however when there are a lot of CPU it
will be complicated to find the CPU sharing the same socket and the same
attributes.
I think having both would be interesting.
What do you think?
regards,
Pierre
>
> With regards,
> Daniel
--
Pierre Morel
IBM Lab Boeblingen
^ permalink raw reply [flat|nested] 80+ messages in thread
* Re: [PATCH v14 09/11] qapi/s390/cpu topology: monitor query topology information
2023-01-12 11:48 ` Thomas Huth
@ 2023-01-18 15:59 ` Pierre Morel
0 siblings, 0 replies; 80+ messages in thread
From: Pierre Morel @ 2023-01-18 15:59 UTC (permalink / raw)
To: Thomas Huth, qemu-s390x
Cc: qemu-devel, borntraeger, pasic, richard.henderson, david, cohuck,
mst, pbonzini, kvm, ehabkost, marcel.apfelbaum, eblake, armbru,
seiden, nrb, scgl, frankja, berrange, clg
On 1/12/23 12:48, Thomas Huth wrote:
> On 05/01/2023 15.53, Pierre Morel wrote:
>> Reporting the current topology informations to the admin through
>> the QEMU monitor.
>>
>> Signed-off-by: Pierre Morel <pmorel@linux.ibm.com>
>> ---
> ...
>> diff --git a/hmp-commands-info.hx b/hmp-commands-info.hx
>> index 754b1e8408..5730a47f71 100644
>> --- a/hmp-commands-info.hx
>> +++ b/hmp-commands-info.hx
>> @@ -993,3 +993,19 @@ SRST
>> ``info virtio-queue-element`` *path* *queue* [*index*]
>> Display element of a given virtio queue
>> ERST
>> +
>> +#if defined(TARGET_S390X) && defined(CONFIG_KVM)
>> + {
>> + .name = "query-topology",
>> + .args_type = "",
>> + .params = "",
>> + .help = "Show information about CPU topology",
>> + .cmd = hmp_query_topology,
>> + .flags = "p",
>> + },
>> +
>> +SRST
>> + ``info query-topology``
>
> "info query-topology" sounds weird ... I'd maybe rather call it only
> "info topology" or "info cpu-topology" here.
info cpu-topology looks good for me.
thanks.
Regards,
Pierre
>
> Thomas
>
>
>> + Show information about CPU topology
>> +ERST
>> +#endif
>
--
Pierre Morel
IBM Lab Boeblingen
^ permalink raw reply [flat|nested] 80+ messages in thread
* Re: [PATCH v14 09/11] qapi/s390/cpu topology: monitor query topology information
2023-01-18 15:58 ` Pierre Morel
@ 2023-01-18 16:08 ` Daniel P. Berrangé
2023-01-18 16:57 ` Pierre Morel
0 siblings, 1 reply; 80+ messages in thread
From: Daniel P. Berrangé @ 2023-01-18 16:08 UTC (permalink / raw)
To: Pierre Morel
Cc: qemu-s390x, qemu-devel, borntraeger, pasic, richard.henderson,
david, thuth, cohuck, mst, pbonzini, kvm, ehabkost,
marcel.apfelbaum, eblake, armbru, seiden, nrb, scgl, frankja,
clg
On Wed, Jan 18, 2023 at 04:58:05PM +0100, Pierre Morel wrote:
>
>
> On 1/12/23 13:10, Daniel P. Berrangé wrote:
> > On Thu, Jan 05, 2023 at 03:53:11PM +0100, Pierre Morel wrote:
> > > Reporting the current topology informations to the admin through
> > > the QEMU monitor.
> > >
> > > Signed-off-by: Pierre Morel <pmorel@linux.ibm.com>
> > > ---
> > > qapi/machine-target.json | 66 ++++++++++++++++++++++++++++++++++
> > > include/monitor/hmp.h | 1 +
> > > hw/s390x/cpu-topology.c | 76 ++++++++++++++++++++++++++++++++++++++++
> > > hmp-commands-info.hx | 16 +++++++++
> > > 4 files changed, 159 insertions(+)
> > >
> > > diff --git a/qapi/machine-target.json b/qapi/machine-target.json
> > > index 75b0aa254d..927618a78f 100644
> > > --- a/qapi/machine-target.json
> > > +++ b/qapi/machine-target.json
> > > @@ -371,3 +371,69 @@
> > > },
> > > 'if': { 'all': [ 'TARGET_S390X', 'CONFIG_KVM' ] }
> > > }
> > > +
> > > +##
> > > +# @S390CpuTopology:
> > > +#
> > > +# CPU Topology information
> > > +#
> > > +# @drawer: the destination drawer where to move the vCPU
> > > +#
> > > +# @book: the destination book where to move the vCPU
> > > +#
> > > +# @socket: the destination socket where to move the vCPU
> > > +#
> > > +# @polarity: optional polarity, default is last polarity set by the guest
> > > +#
> > > +# @dedicated: optional, if the vCPU is dedicated to a real CPU
> > > +#
> > > +# @origin: offset of the first bit of the core mask
> > > +#
> > > +# @mask: mask of the cores sharing the same topology
> > > +#
> > > +# Since: 8.0
> > > +##
> > > +{ 'struct': 'S390CpuTopology',
> > > + 'data': {
> > > + 'drawer': 'int',
> > > + 'book': 'int',
> > > + 'socket': 'int',
> > > + 'polarity': 'int',
> > > + 'dedicated': 'bool',
> > > + 'origin': 'int',
> > > + 'mask': 'str'
> > > + },
> > > + 'if': { 'all': [ 'TARGET_S390X', 'CONFIG_KVM' ] }
> > > +}
> > > +
> > > +##
> > > +# @query-topology:
> > > +#
> > > +# Return information about CPU Topology
> > > +#
> > > +# Returns a @CpuTopology instance describing the CPU Toplogy
> > > +# being currently used by QEMU.
> > > +#
> > > +# Since: 8.0
> > > +#
> > > +# Example:
> > > +#
> > > +# -> { "execute": "cpu-topology" }
> > > +# <- {"return": [
> > > +# {
> > > +# "drawer": 0,
> > > +# "book": 0,
> > > +# "socket": 0,
> > > +# "polarity": 0,
> > > +# "dedicated": true,
> > > +# "origin": 0,
> > > +# "mask": 0xc000000000000000,
> > > +# },
> > > +# ]
> > > +# }
> > > +#
> > > +##
> > > +{ 'command': 'query-topology',
> > > + 'returns': ['S390CpuTopology'],
> > > + 'if': { 'all': [ 'TARGET_S390X', 'CONFIG_KVM' ] }
> > > +}
> >
> > IIUC, you're using @mask as a way to compress the array returned
> > from query-topology, so that it doesn't have any repeated elements
> > with the same data. I guess I can understand that desire when the
> > core count can get very large, this can have a large saving.
> >
> > The downside of using @mask, is that now you require the caller
> > to parse the string to turn it into a bitmask and expand the
> > data. Generally this is considered a bit of an anti-pattern in
> > QAPI design - we don't want callers to have to further parse
> > the data to extract information, we want to directly consumable
> > from the parsed JSON doc.
>
> Not exactly, the mask is computed by the firmware to provide it to the guest
> and is already available when querying the topology.
> But I understand that for the QAPI user the mask is not the right solution,
> standard coma separated values like (1,3,5,7-11) would be much easier to
> read.
That is still inventing a second level data format for an attribute
that needs to be parsed, and its arguably more complex.
> > We already have 'query-cpus-fast' wich returns one entry for
> > each CPU. In fact why do we need to add query-topology at all.
> > Can't we just add book-id / drawer-id / polarity / dedicated
> > to the query-cpus-fast result ?
>
> Yes we can, I think we should, however when there are a lot of CPU it will
> be complicated to find the CPU sharing the same socket and the same
> attributes.
It shouldn't be that hard to populate a hash table, using the set of
socket + attributes you want as the hash key.
> I think having both would be interesting.
IMHO this is undesirable if we can make query-cpus-fast report
sufficient information, as it gives a maint burden to QEMU and
is confusing to consumers to QEMU to have multiple commands with
largely overlapping functionality.
With regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
^ permalink raw reply [flat|nested] 80+ messages in thread
* Re: [PATCH v14 09/11] qapi/s390/cpu topology: monitor query topology information
2023-01-18 16:08 ` Daniel P. Berrangé
@ 2023-01-18 16:57 ` Pierre Morel
0 siblings, 0 replies; 80+ messages in thread
From: Pierre Morel @ 2023-01-18 16:57 UTC (permalink / raw)
To: Daniel P. Berrangé
Cc: qemu-s390x, qemu-devel, borntraeger, pasic, richard.henderson,
david, thuth, cohuck, mst, pbonzini, kvm, ehabkost,
marcel.apfelbaum, eblake, armbru, seiden, nrb, scgl, frankja,
clg
On 1/18/23 17:08, Daniel P. Berrangé wrote:
> On Wed, Jan 18, 2023 at 04:58:05PM +0100, Pierre Morel wrote:
>>
>>
>> On 1/12/23 13:10, Daniel P. Berrangé wrote:
>>> On Thu, Jan 05, 2023 at 03:53:11PM +0100, Pierre Morel wrote:
>>>> Reporting the current topology informations to the admin through
>>>> the QEMU monitor.
>>>>
>>>> Signed-off-by: Pierre Morel <pmorel@linux.ibm.com>
>>>> ---
>>>> qapi/machine-target.json | 66 ++++++++++++++++++++++++++++++++++
>>>> include/monitor/hmp.h | 1 +
>>>> hw/s390x/cpu-topology.c | 76 ++++++++++++++++++++++++++++++++++++++++
>>>> hmp-commands-info.hx | 16 +++++++++
>>>> 4 files changed, 159 insertions(+)
>>>>
>>>> diff --git a/qapi/machine-target.json b/qapi/machine-target.json
>>>> index 75b0aa254d..927618a78f 100644
>>>> --- a/qapi/machine-target.json
>>>> +++ b/qapi/machine-target.json
>>>> @@ -371,3 +371,69 @@
>>>> },
>>>> 'if': { 'all': [ 'TARGET_S390X', 'CONFIG_KVM' ] }
>>>> }
>>>> +
>>>> +##
>>>> +# @S390CpuTopology:
>>>> +#
>>>> +# CPU Topology information
>>>> +#
>>>> +# @drawer: the destination drawer where to move the vCPU
>>>> +#
>>>> +# @book: the destination book where to move the vCPU
>>>> +#
>>>> +# @socket: the destination socket where to move the vCPU
>>>> +#
>>>> +# @polarity: optional polarity, default is last polarity set by the guest
>>>> +#
>>>> +# @dedicated: optional, if the vCPU is dedicated to a real CPU
>>>> +#
>>>> +# @origin: offset of the first bit of the core mask
>>>> +#
>>>> +# @mask: mask of the cores sharing the same topology
>>>> +#
>>>> +# Since: 8.0
>>>> +##
>>>> +{ 'struct': 'S390CpuTopology',
>>>> + 'data': {
>>>> + 'drawer': 'int',
>>>> + 'book': 'int',
>>>> + 'socket': 'int',
>>>> + 'polarity': 'int',
>>>> + 'dedicated': 'bool',
>>>> + 'origin': 'int',
>>>> + 'mask': 'str'
>>>> + },
>>>> + 'if': { 'all': [ 'TARGET_S390X', 'CONFIG_KVM' ] }
>>>> +}
>>>> +
>>>> +##
>>>> +# @query-topology:
>>>> +#
>>>> +# Return information about CPU Topology
>>>> +#
>>>> +# Returns a @CpuTopology instance describing the CPU Toplogy
>>>> +# being currently used by QEMU.
>>>> +#
>>>> +# Since: 8.0
>>>> +#
>>>> +# Example:
>>>> +#
>>>> +# -> { "execute": "cpu-topology" }
>>>> +# <- {"return": [
>>>> +# {
>>>> +# "drawer": 0,
>>>> +# "book": 0,
>>>> +# "socket": 0,
>>>> +# "polarity": 0,
>>>> +# "dedicated": true,
>>>> +# "origin": 0,
>>>> +# "mask": 0xc000000000000000,
>>>> +# },
>>>> +# ]
>>>> +# }
>>>> +#
>>>> +##
>>>> +{ 'command': 'query-topology',
>>>> + 'returns': ['S390CpuTopology'],
>>>> + 'if': { 'all': [ 'TARGET_S390X', 'CONFIG_KVM' ] }
>>>> +}
>>>
>>> IIUC, you're using @mask as a way to compress the array returned
>>> from query-topology, so that it doesn't have any repeated elements
>>> with the same data. I guess I can understand that desire when the
>>> core count can get very large, this can have a large saving.
>>>
>>> The downside of using @mask, is that now you require the caller
>>> to parse the string to turn it into a bitmask and expand the
>>> data. Generally this is considered a bit of an anti-pattern in
>>> QAPI design - we don't want callers to have to further parse
>>> the data to extract information, we want to directly consumable
>>> from the parsed JSON doc.
>>
>> Not exactly, the mask is computed by the firmware to provide it to the guest
>> and is already available when querying the topology.
>> But I understand that for the QAPI user the mask is not the right solution,
>> standard coma separated values like (1,3,5,7-11) would be much easier to
>> read.
>
> That is still inventing a second level data format for an attribute
> that needs to be parsed, and its arguably more complex.
OK, I think I am too focused on my vision of the topology.
I add the new attributes to the query-cpus-fast
>
>>> We already have 'query-cpus-fast' wich returns one entry for
>>> each CPU. In fact why do we need to add query-topology at all.
>>> Can't we just add book-id / drawer-id / polarity / dedicated
>>> to the query-cpus-fast result ?
>>
>> Yes we can, I think we should, however when there are a lot of CPU it will
>> be complicated to find the CPU sharing the same socket and the same
>> attributes.
>
> It shouldn't be that hard to populate a hash table, using the set of
> socket + attributes you want as the hash key.
It is not a problem.
>
>> I think having both would be interesting.
>
> IMHO this is undesirable if we can make query-cpus-fast report
> sufficient information, as it gives a maint burden to QEMU and
> is confusing to consumers to QEMU to have multiple commands with
> largely overlapping functionality.
right.
Thanks Daniel.
Regards,
Pierre
>
>
> With regards,
> Daniel
--
Pierre Morel
IBM Lab Boeblingen
^ permalink raw reply [flat|nested] 80+ messages in thread
* Re: [PATCH v14 10/11] qapi/s390/cpu topology: POLARITY_CHANGE qapi event
2023-01-12 11:52 ` Thomas Huth
@ 2023-01-18 17:09 ` Pierre Morel
2023-01-20 11:56 ` Thomas Huth
0 siblings, 1 reply; 80+ messages in thread
From: Pierre Morel @ 2023-01-18 17:09 UTC (permalink / raw)
To: Thomas Huth, qemu-s390x
Cc: qemu-devel, borntraeger, pasic, richard.henderson, david, cohuck,
mst, pbonzini, kvm, ehabkost, marcel.apfelbaum, eblake, armbru,
seiden, nrb, scgl, frankja, berrange, clg
On 1/12/23 12:52, Thomas Huth wrote:
> On 05/01/2023 15.53, Pierre Morel wrote:
>> When the guest asks to change the polarity this change
>> is forwarded to the admin using QAPI.
>> The admin is supposed to take according decisions concerning
>> CPU provisioning.
>
> I somehow doubt that an average admin will monitor QEMU for such events
> ... so this rather should be handled by upper layers like libvirt one day?
Yes.
>
>> Signed-off-by: Pierre Morel <pmorel@linux.ibm.com>
>> ---
>> qapi/machine-target.json | 21 +++++++++++++++++++++
>> hw/s390x/cpu-topology.c | 2 ++
>> 2 files changed, 23 insertions(+)
>>
>> diff --git a/qapi/machine-target.json b/qapi/machine-target.json
>> index 927618a78f..10235cfb45 100644
>> --- a/qapi/machine-target.json
>> +++ b/qapi/machine-target.json
>> @@ -437,3 +437,24 @@
>> 'returns': ['S390CpuTopology'],
>> 'if': { 'all': [ 'TARGET_S390X', 'CONFIG_KVM' ] }
>> }
>> +
>> +##
>> +# @POLARITY_CHANGE:
>
> I'd maybe rather call it CPU_POLARITY_CHANGE ... in case "polarity" is
> one day also used for some other devices.
OK, right.
>
>> +#
>> +# Emitted when the guest asks to change the polarity.
>> +#
>> +# @polarity: polarity specified by the guest
>
> Please elaborate: Where does the value come from (the PTF instruction)?
> Which values are possible?
Yes what about:
# @polarity: the guest can specify with the PTF instruction a horizontal
# or a vertical polarity.
# On horizontal polarity the host is expected to provision
# the vCPU equally.
# On vertical polarity the host can provision each vCPU
# differently
# The guest can get information on the provisioning with
# the STSI(15) instruction.
Regards,
Pierre
--
Pierre Morel
IBM Lab Boeblingen
^ permalink raw reply [flat|nested] 80+ messages in thread
* Re: [PATCH v14 11/11] docs/s390x/cpu topology: document s390x cpu topology
2023-01-12 11:58 ` Daniel P. Berrangé
@ 2023-01-18 17:10 ` Pierre Morel
0 siblings, 0 replies; 80+ messages in thread
From: Pierre Morel @ 2023-01-18 17:10 UTC (permalink / raw)
To: Daniel P. Berrangé
Cc: qemu-s390x, qemu-devel, borntraeger, pasic, richard.henderson,
david, thuth, cohuck, mst, pbonzini, kvm, ehabkost,
marcel.apfelbaum, eblake, armbru, seiden, nrb, scgl, frankja,
clg
On 1/12/23 12:58, Daniel P. Berrangé wrote:
> On Thu, Jan 05, 2023 at 03:53:13PM +0100, Pierre Morel wrote:
>> Add some basic examples for the definition of cpu topology
>> in s390x.
>>
>> Signed-off-by: Pierre Morel <pmorel@linux.ibm.com>
>> ---
>> docs/system/s390x/cpu-topology.rst | 292 +++++++++++++++++++++++++++++
>> docs/system/target-s390x.rst | 1 +
>> 2 files changed, 293 insertions(+)
>> create mode 100644 docs/system/s390x/cpu-topology.rst
>>
>> diff --git a/docs/system/s390x/cpu-topology.rst b/docs/system/s390x/cpu-topology.rst
>> new file mode 100644
>> index 0000000000..0020b70b50
>> --- /dev/null
>> +++ b/docs/system/s390x/cpu-topology.rst
>> @@ -0,0 +1,292 @@
>> +CPU Topology on s390x
>> +=====================
>> +
>> +CPU Topology on S390x provides up to 5 levels of topology containers:
>> +nodes, drawers, books, sockets and CPUs.
>
> The last level should be 'cores' not CPUs for QEMU terminology.
Yes, thanks.
Regards,
Pierre
--
Pierre Morel
IBM Lab Boeblingen
^ permalink raw reply [flat|nested] 80+ messages in thread
* Re: [PATCH v14 04/11] s390x/sclp: reporting the maximum nested topology entries
2023-01-17 19:58 ` Nina Schoetterl-Glausch
@ 2023-01-19 13:08 ` Pierre Morel
0 siblings, 0 replies; 80+ messages in thread
From: Pierre Morel @ 2023-01-19 13:08 UTC (permalink / raw)
To: Nina Schoetterl-Glausch, Thomas Huth, qemu-s390x
Cc: qemu-devel, borntraeger, pasic, richard.henderson, david, cohuck,
mst, pbonzini, kvm, ehabkost, marcel.apfelbaum, eblake, armbru,
seiden, nrb, frankja, berrange, clg
On 1/17/23 20:58, Nina Schoetterl-Glausch wrote:
> On Tue, 2023-01-17 at 18:36 +0100, Pierre Morel wrote:
>>
>> On 1/11/23 09:57, Thomas Huth wrote:
>>> On 05/01/2023 15.53, Pierre Morel wrote:
>>>> The maximum nested topology entries is used by the guest to know
>>>> how many nested topology are available on the machine.
>>>>
>>>> Currently, SCLP READ SCP INFO reports MNEST = 0, which is the
>>>> equivalent of reporting the default value of 2.
>>>> Let's use the default SCLP value of 2 and increase this value in the
>>>> future patches implementing higher levels.
>>>
>>> I'm confused ... so does a SCLP value of 2 mean a MNEST level of 4 ?
>>
>> Sorry, I forgot to change this.
>> MNEST = 0 means no MNEST support and only socket is supported so it is
>> like MNEST = 2.
>> MNEST != 0 set the maximum nested level and correct values may be 2,3 or 4.
>> But this setting to 4 should already have been done in previous patch
>> where we introduced the books and drawers.
>
> I think setting it to 4 here is fine/preferable, since 2 is the default unless
> you tell the guest that more are available, which you do in this patch.
> It's only the commit description that is confusing.
Yes.
Only the commit message is confusing, it is to be set on 4 in this patch.
I change the commit message to:
---
s390x/sclp: reporting the maximum nested topology entries
The maximum nested topology entries is used by the guest to
know how many nested topology are available on the machine.
Let change the MNEST value from 2 to 4 in the SCLP READ INFO
structure now that we support books and drawers.
---
is it OK ?
Regards,
Pierre
>
>>
>> I change the commit message with:
>> ---
>> s390x/sclp: reporting the maximum nested topology entries
>>
>> The maximum nested topology entries is used by the guest to know
>> how many nested topology are available on the machine.
>>
>> Let's return this information to the guest.
>> ---
>>
>>>
>>>> Signed-off-by: Pierre Morel <pmorel@linux.ibm.com>
>>>> ---
>>>> include/hw/s390x/sclp.h | 5 +++--
>>>> hw/s390x/sclp.c | 4 ++++
>>>> 2 files changed, 7 insertions(+), 2 deletions(-)
>>>>
>>>> diff --git a/include/hw/s390x/sclp.h b/include/hw/s390x/sclp.h
>>>> index 712fd68123..4ce852473c 100644
>>>> --- a/include/hw/s390x/sclp.h
>>>> +++ b/include/hw/s390x/sclp.h
>>>> @@ -112,12 +112,13 @@ typedef struct CPUEntry {
>>>> } QEMU_PACKED CPUEntry;
>>>> #define SCLP_READ_SCP_INFO_FIXED_CPU_OFFSET 128
>>>> -#define SCLP_READ_SCP_INFO_MNEST 2
>>>> +#define SCLP_READ_SCP_INFO_MNEST 4
>>>
>>> ... since you update it to 4 here.
>>
>> Yes, in fact this should be set in the previous patch already to 4.
>> So I will do that.
>>
>>>
>>>> typedef struct ReadInfo {
>>>> SCCBHeader h;
>>>> uint16_t rnmax;
>>>> uint8_t rnsize;
>>>> - uint8_t _reserved1[16 - 11]; /* 11-15 */
>>>> + uint8_t _reserved1[15 - 11]; /* 11-14 */
>>>> + uint8_t stsi_parm; /* 15-16 */
>>>> uint16_t entries_cpu; /* 16-17 */
>>>> uint16_t offset_cpu; /* 18-19 */
>>>> uint8_t _reserved2[24 - 20]; /* 20-23 */
>>>> diff --git a/hw/s390x/sclp.c b/hw/s390x/sclp.c
>>>> index eff74479f4..07e3cb4cac 100644
>>>> --- a/hw/s390x/sclp.c
>>>> +++ b/hw/s390x/sclp.c
>>>> @@ -20,6 +20,7 @@
>>>> #include "hw/s390x/event-facility.h"
>>>> #include "hw/s390x/s390-pci-bus.h"
>>>> #include "hw/s390x/ipl.h"
>>>> +#include "hw/s390x/cpu-topology.h"
>>>> static inline SCLPDevice *get_sclp_device(void)
>>>> {
>>>> @@ -125,6 +126,9 @@ static void read_SCP_info(SCLPDevice *sclp, SCCB
>>>> *sccb)
>>>> /* CPU information */
>>>> prepare_cpu_entries(machine, entries_start, &cpu_count);
>>>> + if (s390_has_topology()) {
>>>> + read_info->stsi_parm = SCLP_READ_SCP_INFO_MNEST;
>>>
>>> This seems to be in contradiction to what you've said in the commit
>>> description - you set it to 4 and not to 2.
>>
>> Yes, I change the commit message.
>>
>> Thanks.
>>
>> Regards,
>> Pierre
>>
>
--
Pierre Morel
IBM Lab Boeblingen
^ permalink raw reply [flat|nested] 80+ messages in thread
* Re: [PATCH v14 03/11] target/s390x/cpu topology: handle STSI(15) and build the SYSIB
2023-01-18 11:54 ` Nina Schoetterl-Glausch
@ 2023-01-19 13:12 ` Pierre Morel
0 siblings, 0 replies; 80+ messages in thread
From: Pierre Morel @ 2023-01-19 13:12 UTC (permalink / raw)
To: Nina Schoetterl-Glausch, Thomas Huth, qemu-s390x, frankja
Cc: qemu-devel, borntraeger, pasic, richard.henderson, david, cohuck,
mst, pbonzini, kvm, ehabkost, marcel.apfelbaum, eblake, armbru,
seiden, nrb, berrange, clg
On 1/18/23 12:54, Nina Schoetterl-Glausch wrote:
> On Wed, 2023-01-18 at 11:26 +0100, Thomas Huth wrote:
>> On 17/01/2023 17.56, Pierre Morel wrote:
>>>
>>>
>>> On 1/10/23 15:29, Thomas Huth wrote:
>>>> On 05/01/2023 15.53, Pierre Morel wrote:
>>>>> On interception of STSI(15.1.x) the System Information Block
>>>>> (SYSIB) is built from the list of pre-ordered topology entries.
>>>>>
>>>>> Signed-off-by: Pierre Morel <pmorel@linux.ibm.com>
>>>>> ---
>> ...
>>>>> +typedef struct SysIBTl_container {
>>>>> + uint8_t nl;
>>>>> + uint8_t reserved[6];
>>>>> + uint8_t id;
>>>>> +} QEMU_PACKED QEMU_ALIGNED(8) SysIBTl_container;
>>>>> +QEMU_BUILD_BUG_ON(sizeof(SysIBTl_container) != 8);
>>>>> +
>>>>> +/* CPU type Topology List Entry */
>>>>> +typedef struct SysIBTl_cpu {
>>>>> + uint8_t nl;
>>>>> + uint8_t reserved0[3];
>>>>> + uint8_t reserved1:5;
>>>>> + uint8_t dedicated:1;
>>>>> + uint8_t polarity:2;
>>>>
>>>> Hmmm, yet another bitfield...
>>>
>>> Yes, this is the firmware interface.
>>> If it makes problem I can use masks and logic arithmetic
>>
>> It depends ... if we are sure that this will ever only be used with KVM on
>> real s390x hardware, then bitfields are OK. If we think that this is
>> something that could be implemented in TCG, too, I'd scratch the bitfields
>> and use logic arithmetic instead...
>
> Is there something like linux' bitfield.h in qemu?
> In this case it's only two fields, and not too complicated, but I imagine it could
> get quite ugly to do it manually in other cases.
>>
>> I'm not too experienced with this CPU topology stuff, but it sounds like it
>> could be implemented in TCG without too much efforts one day, too, so I'd
>> rather go with the logic arithmetic immediately instead if it is not too
>> annoying for you right now.
No problem, I changed all fields to uint8_t and it fits in a uint64_t so
I have only a single bit and a single mask to define.
>>
>>>>> diff --git a/target/s390x/kvm/cpu_topology.c
>>>>> b/target/s390x/kvm/cpu_topology.c
>>>>> new file mode 100644
>>>>> index 0000000000..3831a3264c
>>>>> --- /dev/null
>>>>> +++ b/target/s390x/kvm/cpu_topology.c
>>>>> @@ -0,0 +1,136 @@
>>>>> +/*
>>>>> + * QEMU S390x CPU Topology
>>>>> + *
>>>>> + * Copyright IBM Corp. 2022
>>>>
>>>> Happy new year?
>>>
>>> So after Nina's comment what do I do?
>>> let it be 22 because I started last year or update because what is important
>>> is when it comes into mainline?
>>
>> Honestly, I don't have a really good clue either... But keeping 2022 is
>> certainly fine for me, too.
OK then I keep 2022
Regards,
Pierre
--
Pierre Morel
IBM Lab Boeblingen
^ permalink raw reply [flat|nested] 80+ messages in thread
* Re: [PATCH v14 02/11] s390x/cpu topology: add topology entries on CPU hotplug
2023-01-17 16:48 ` Nina Schoetterl-Glausch
@ 2023-01-19 13:34 ` Pierre Morel
0 siblings, 0 replies; 80+ messages in thread
From: Pierre Morel @ 2023-01-19 13:34 UTC (permalink / raw)
To: Nina Schoetterl-Glausch, qemu-s390x
Cc: qemu-devel, borntraeger, pasic, richard.henderson, david, thuth,
cohuck, mst, pbonzini, kvm, ehabkost, marcel.apfelbaum, eblake,
armbru, seiden, nrb, frankja, berrange, clg
On 1/17/23 17:48, Nina Schoetterl-Glausch wrote:
> On Tue, 2023-01-17 at 14:55 +0100, Pierre Morel wrote:
>>
>> On 1/13/23 19:15, Nina Schoetterl-Glausch wrote:
>>>
> [...]
>
>>>> +/**
>>>> + * s390_topology_set_entry:
>>>> + * @entry: Topology entry to setup
>>>> + * @id: topology id to use for the setup
>>>> + *
>>>> + * Set the core bit inside the topology mask and
>>>> + * increments the number of cores for the socket.
>>>> + */
>>>> +static void s390_topology_set_entry(S390TopologyEntry *entry,
>>>
>>> Not sure if I like the name, what it does is to add a cpu to the entry.
>>
>> s390_topology_add_cpu_to_entry() ?
>
> Yeah, that's better.
>
> [...]
>>
>>>> +/**
>>>> + * s390_topology_set_cpu:
>>>> + * @ms: MachineState used to initialize the topology structure on
>>>> + * first call.
>>>> + * @cpu: the new S390CPU to insert in the topology structure
>>>> + * @errp: the error pointer
>>>> + *
>>>> + * Called from CPU Hotplug to check and setup the CPU attributes
>>>> + * before to insert the CPU in the topology.
>>>> + */
>>>> +void s390_topology_set_cpu(MachineState *ms, S390CPU *cpu, Error **errp)
>>>> +{
>>>> + Error *local_error = NULL;
>>>
>>> Can't you just use ERRP_GUARD ?
>>
>> I do not think it is necessary and I find it obfuscating.
>> So, should I?
>
> /*
> * Propagate error object (if any) from @local_err to @dst_errp.
> [...]
> * Please use ERRP_GUARD() instead when possible.
> * Please don't error_propagate(&error_fatal, ...), use
> * error_report_err() and exit(), because that's more obvious.
> */
> void error_propagate(Error **dst_errp, Error *local_err);
>
> So I'd say yes.
OK, you are right it is better.
Regards,
Pierre
--
Pierre Morel
IBM Lab Boeblingen
^ permalink raw reply [flat|nested] 80+ messages in thread
* Re: [PATCH v14 11/11] docs/s390x/cpu topology: document s390x cpu topology
2023-01-12 11:46 ` Thomas Huth
@ 2023-01-19 14:48 ` Pierre Morel
0 siblings, 0 replies; 80+ messages in thread
From: Pierre Morel @ 2023-01-19 14:48 UTC (permalink / raw)
To: Thomas Huth, qemu-s390x
Cc: qemu-devel, borntraeger, pasic, richard.henderson, david, cohuck,
mst, pbonzini, kvm, ehabkost, marcel.apfelbaum, eblake, armbru,
seiden, nrb, scgl, frankja, berrange, clg
On 1/12/23 12:46, Thomas Huth wrote:
> On 05/01/2023 15.53, Pierre Morel wrote:
>> Add some basic examples for the definition of cpu topology
>> in s390x.
>>
>> Signed-off-by: Pierre Morel <pmorel@linux.ibm.com>
>> ---
>> docs/system/s390x/cpu-topology.rst | 292 +++++++++++++++++++++++++++++
>> docs/system/target-s390x.rst | 1 +
>> 2 files changed, 293 insertions(+)
>> create mode 100644 docs/system/s390x/cpu-topology.rst
>>
>> diff --git a/docs/system/s390x/cpu-topology.rst
>> b/docs/system/s390x/cpu-topology.rst
>> new file mode 100644
>> index 0000000000..0020b70b50
>> --- /dev/null
>> +++ b/docs/system/s390x/cpu-topology.rst
>> @@ -0,0 +1,292 @@
>> +CPU Topology on s390x
>> +=====================
>> +
>> +CPU Topology on S390x provides up to 5 levels of topology containers:
>
> You sometimes write "Topology" with a capital T, sometimes lower case
> ... I'd suggest to write it lower case consistently everywhere.
OK
>
>> +nodes, drawers, books, sockets and CPUs.
>
> Hmm, so here you mention that "nodes" are usable on s390x, too? ... in
> another spot below, you don't mention these anymore...
No, no nodes needed here, I remove that.
>
>> +While the higher level containers, Containers Topology List Entries,
>> +(Containers TLE) define a tree hierarchy, the lowest level of topology
>> +definition, the CPU Topology List Entry (CPU TLE), provides the
>> placement
>> +of the CPUs inside the parent container.
>> +
>> +Currently QEMU CPU topology uses a single level of container: the
>> sockets.
>> +
>> +For backward compatibility, threads can be declared on the ``-smp``
>> command
>> +line. They will be seen as CPUs by the guest as long as multithreading
>> +is not really supported by QEMU for S390.
>
> Maybe mention that threads are not allowed with machine types >= 7.2
> anymore?
yes
>
>> +Beside the topological tree, S390x provides 3 CPU attributes:
>> +- CPU type
>> +- polarity entitlement
>> +- dedication
>> +
>> +Prerequisites
>> +-------------
>> +
>> +To use CPU Topology a Linux QEMU/KVM machine providing the CPU
>> Topology facility
>> +(STFLE bit 11) is required.
>> +
>> +However, since this facility has been enabled by default in an early
>> version
>> +of QEMU, we use a capability, ``KVM_CAP_S390_CPU_TOPOLOGY``, to
>> notify KVM
>> +QEMU use of the CPU Topology.
>
> Has it? I thought bit 11 was not enabled by default in the past?
bit 11 enabled by default in QEMU, not in KVM.
However no code has been provided to support the STSI(15) and the PTF
instruction which are enabled by facility 11.
So if we had enabled facility 11 in KVM without precaution a guest
seeing facility 11 will use the PTF instruction and get a program interrupt.
Therefore we need a KVM capability to enable bit 11 in KVM
>
>> +Enabling CPU topology
>> +---------------------
>> +
>> +Currently, CPU topology is only enabled in the host model.
>
> add a "by default if support is available in the host kernel" at the end
> of the sentence?
yes, thx
>
>> +Enabling CPU topology in a CPU model is done by setting the CPU flag
>> +``ctop`` to ``on`` like in:
>> +
>> +.. code-block:: bash
>> +
>> + -cpu gen16b,ctop=on
>> +
>> +Having the topology disabled by default allows migration between
>> +old and new QEMU without adding new flags.
>> +
>> +Default topology usage
>> +----------------------
>> +
>> +The CPU Topology, can be specified on the QEMU command line
>> +with the ``-smp`` or the ``-device`` QEMU command arguments
>> +without using any new attributes.
>> +In this case, the topology will be calculated by simply adding
>> +to the topology the cores based on the core-id starting with
>> +core-0 at position 0 of socket-0, book-0, drawer-0 with default
>
> ... here you don't mention "nodes" anymore (which you still mentioned at
> the beginning of the doc).
I removed it
>
>> +modifier attributes: horizontal polarity and no dedication.
>> +
>> +In the following machine we define 8 sockets with 4 cores each.
>> +Note that S390 QEMU machines do not implement multithreading.
>
> I'd use s390x instead of S390 to avoid confusion with 31-bit machines.
OK
>
>> +.. code-block:: bash
>> +
>> + $ qemu-system-s390x -m 2G \
>> + -cpu gen16b,ctop=on \
>> + -smp cpus=5,sockets=8,cores=4,maxcpus=32 \
>> + -device host-s390x-cpu,core-id=14 \
>> +
>> +New CPUs can be plugged using the device_add hmp command like in:
>> +
>> +.. code-block:: bash
>> +
>> + (qemu) device_add gen16b-s390x-cpu,core-id=9
>> +
>> +The core-id defines the placement of the core in the topology by
>> +starting with core 0 in socket 0 up to maxcpus.
>> +
>> +In the example above:
>> +
>> +* There are 5 CPUs provided to the guest with the ``-smp`` command line
>> + They will take the core-ids 0,1,2,3,4
>> + As we have 4 cores in a socket, we have 4 CPUs provided
>> + to the guest in socket 0, with core-ids 0,1,2,3.
>> + The last cpu, with core-id 4, will be on socket 1.
>> +
>> +* the core with ID 14 provided by the ``-device`` command line will
>> + be placed in socket 3, with core-id 14
>> +
>> +* the core with ID 9 provided by the ``device_add`` qmp command will
>> + be placed in socket 2, with core-id 9
>> +
>> +Note that the core ID is machine wide and the CPU TLE masks provided
>> +by the STSI instruction will be written in a big endian mask:
>> +
>> +* in socket 0: 0xf000000000000000 (core id 0,1,2,3)
>> +* in socket 1: 0x0800000000000000 (core id 4)
>> +* in socket 2: 0x0040000000000000 (core id 9)
>> +* in socket 3: 0x0002000000000000 (core id 14)
>
> Hmm, who's supposed to be the audience of this documentation? Users?
> Developers? For a doc in docs/system/ I'd expect this to be a
> documentation for users, so this seems to be way too much of
> implementation detail here already. If this is supposed to be a doc for
> developers instead, the file should likely rather go into doc/devel/
> instead. Or maybe you want both? ... then you should split the
> information in here in two files, I think, one in docs/system/ and one
> in docs/devel/ .
I am not sure in devel there is all doc on QAPI interface not on commands.
On the other hand the QAPI seems to have its own way to document the
commands.
So what I write here is more detailed than in the QAPI documentation.
May be I better write these details there and suppress them here,
just naming the command, info and event without details.
>
>> +Defining the topology on command line
>> +-------------------------------------
>> +
>> +The topology can be defined entirely during the CPU definition,
>> +with the exception of CPU 0 which must be defined with the -smp
>> +argument.
>> +
>> +For example, here we set the position of the cores 1,2,3 on
>> +drawer 1, book 1, socket 2 and cores 0,9 and 14 on drawer 0,
>> +book 0, socket 0 with all horizontal polarity and not dedicated.
>> +The core 4, will be set on its default position on socket 1
>> +(since we have 4 core per socket) and we define it with dedication and
>> +vertical high entitlement.
>> +
>> +.. code-block:: bash
>> +
>> + $ qemu-system-s390x -m 2G \
>> + -cpu gen16b,ctop=on \
>> + -smp cpus=1,sockets=8,cores=4,maxcpus=32 \
>> + \
>> + -device
>> gen16b-s390x-cpu,drawer-id=1,book-id=1,socket-id=2,core-id=1 \
>> + -device
>> gen16b-s390x-cpu,drawer-id=1,book-id=1,socket-id=2,core-id=2 \
>> + -device
>> gen16b-s390x-cpu,drawer-id=1,book-id=1,socket-id=2,core-id=3 \
>> + \
>> + -device
>> gen16b-s390x-cpu,drawer-id=0,book-id=0,socket-id=0,core-id=9 \
>> + -device
>> gen16b-s390x-cpu,drawer-id=0,book-id=0,socket-id=0,core-id=14 \
>> + \
>> + -device gen16b-s390x-cpu,core-id=4,dedicated=on,polarity=3 \
>> +
>> +Polarity and dedication
>> +-----------------------
>
> Since you are using the terms "polarity" and "dedication" in the
> previous paragraphs already, it might make sense to move this section
> here earlier in the document to teach the users about this first, before
> using the terms in the other paragraphs?
yes
>
>> +Polarity can be of two types: horizontal or vertical.
>> +
>> +The horizontal polarization specifies that all guest's vCPUs get
>> +almost the same amount of provisioning of real CPU by the host.
>> +
>> +The vertical polarization specifies that guest's vCPU can get
>> +different real CPU provisions:
>
> Please remove one space between "different" and "real".
OK
>
>> +- a vCPU with Vertical high entitlement specifies that this
>> + vCPU gets 100% of the real CPU provisioning.
>> +
>> +- a vCPU with Vertical medium entitlement specifies that this
>> + vCPU shares the real CPU with other vCPU.
>
> "with *one* other vCPU" or rather "with other vCPU*s*" ?
thx, vCPUs
>
>> +
>> +- a vCPU with Vertical low entitlement specifies that this
>> + vCPU only get real CPU provisioning when no other vCPU need it.
>> +
>> +In the case a vCPU with vertical high entitlement does not use
>> +the real CPU, the unused "slack" can be dispatched to other vCPU
>> +with medium or low entitlement.
>> +
>> +The host indicates to the guest how the real CPU resources are
>> +provided to the vCPUs through the SYSIB with two polarity bits
>> +inside the CPU TLE.
>> +
>> +Bits d - Polarization
>> +0 0 Horizontal
>> +0 1 Vertical low entitlement
>> +1 0 Vertical medium entitlement
>> +1 1 Vertical high entitlement
>
> That SYSIB stuff looks like details for developers again ... I think you
> should either add more explanations here (I assume the average user does
> not know the term SYSIB), move it to a separate developers file or drop it.
>
OK, I drop it
>> +A subsystem reset puts all vCPU of the configuration into the
>> +horizontal polarization.
>> +
>> +The admin specifies the dedicated bit when the vCPU is dedicated
>> +to a single real CPU.
>> +
>> +As for the Linux admin, the dedicated bit is an indication on the
>> +affinity of a vCPU for a real CPU while the entitlement indicates the
>> +sharing or exclusivity of use.
>> +
>> +QAPI interface for topology
>> +---------------------------
>
> A "grep -r QAPI docs/system/" shows hardly any entries there. I think
> QAPI documentation should go into docs/devel instead.
discussion above.
I enhance the QAPI internal doc or I move it into devel.
>
>> +Let's start QEMU with the following command:
>> +
>> +.. code-block:: bash
>> +
>> + sudo /usr/local/bin/qemu-system-s390x \
>> + -enable-kvm \
>> + -cpu z14,ctop=on \
>> + -smp 1,drawers=3,books=3,sockets=2,cores=2,maxcpus=36 \
>> + \
>> + -device z14-s390x-cpu,core-id=19,polarity=3 \
>> + -device z14-s390x-cpu,core-id=11,polarity=1 \
>> + -device z14-s390x-cpu,core-id=12,polarity=3 \
>> + ...
>> +
>> +and see the result when using of the QAPI interface.
>> +
>> +query-topology
>> ++++++++++++++++
>> +
>> +The command cpu-topology allows the admin to query the topology
>
> Not sure if the average admin runs QMP directly ... maybe rather talk
> about the "upper layers like libvirt" here or something similar.
>
>> +tree and modifier for all configured vCPU.
>> +
>> +.. code-block:: QMP
>> +
>> + -> { "execute": "query-topology" }
>> + {"return":
>> + [
>> + {
>> + "origin": 0,
>> + "dedicated": false,
>> + "book": 0,
>> + "socket": 0,
>> + "drawer": 0,
>> + "polarity": 0,
>> + "mask": "0x8000000000000000"
>> + },
>> + {
>> + "origin": 0,
>> + "dedicated": false,
>> + "book": 2,
>> + "socket": 1,
>> + "drawer": 0,
>> + "polarity": 1,
>> + "mask": "0x0010000000000000"
>> + },
>> + {
>> + "origin": 0,
>> + "dedicated": false,
>> + "book": 0,
>> + "socket": 0,
>> + "drawer": 1,
>> + "polarity": 3,
>> + "mask": "0x0008000000000000"
>> + },
>> + {
>> + "origin": 0,
>> + "dedicated": false,
>> + "book": 1,
>> + "socket": 1,
>> + "drawer": 1,
>> + "polarity": 3,
>> + "mask": "0x0000100000000000"
>> + }
>> + ]
>> + }
>> +
>> +change-topology
>> ++++++++++++++++
>> +
>> +The command change-topology allows the admin to modify the topology
>> +tree or the topology modifiers of a vCPU in the configuration.
>> +
>> +.. code-block:: QMP
>> +
>> + -> { "execute": "change-topology",
>> + "arguments": {
>> + "core": 11,
>> + "socket": 0,
>> + "book": 0,
>> + "drawer": 0,
>> + "polarity": 0,
>> + "dedicated": false
>> + }
>> + }
>> + <- {"return": {}}
>> +
>> +
>> +event POLARITY_CHANGE
>> ++++++++++++++++++++++
>> +
>> +When a guest is requesting a modification of the polarity,
>> +QEMU sends a POLARITY_CHANGE event.
>> +
>> +When requesting the change, the guest only specifies horizontal or
>> +vertical polarity.
>> +The dedication and fine grain vertical entitlement depends on admin
>> +to set according to its response to this event.
>> +
>> +Note that a vertical polarized dedicated vCPU can only have a high
>> +entitlement, this gives 6 possibilities for a vCPU polarity:
>> +
>> +- Horizontal
>> +- Horizontal dedicated
>> +- Vertical low
>> +- Vertical medium
>> +- Vertical high
>> +- Vertical high dedicated
>> +
>> +Example of the event received when the guest issues PTF(0) to request
>
> Please mention that PTF is a CPU instruction (and provide the full name).
Yes, thanks.
regards,
Pierre
>
> Thomas
>
--
Pierre Morel
IBM Lab Boeblingen
^ permalink raw reply [flat|nested] 80+ messages in thread
* Re: [PATCH v14 10/11] qapi/s390/cpu topology: POLARITY_CHANGE qapi event
2023-01-18 17:09 ` Pierre Morel
@ 2023-01-20 11:56 ` Thomas Huth
2023-01-20 14:22 ` Pierre Morel
0 siblings, 1 reply; 80+ messages in thread
From: Thomas Huth @ 2023-01-20 11:56 UTC (permalink / raw)
To: Pierre Morel, qemu-s390x
Cc: qemu-devel, borntraeger, pasic, richard.henderson, david, cohuck,
mst, pbonzini, kvm, ehabkost, marcel.apfelbaum, eblake, armbru,
seiden, nrb, scgl, frankja, berrange, clg
On 18/01/2023 18.09, Pierre Morel wrote:
>
> On 1/12/23 12:52, Thomas Huth wrote:
>> On 05/01/2023 15.53, Pierre Morel wrote:
...>>> +#
>>> +# Emitted when the guest asks to change the polarity.
>>> +#
>>> +# @polarity: polarity specified by the guest
>>
>> Please elaborate: Where does the value come from (the PTF instruction)?
>> Which values are possible?
>
> Yes what about:
>
> # @polarity: the guest can specify with the PTF instruction a horizontal
> # or a vertical polarity.
Maybe something like: "The guest can tell the host (via the PTF instruction)
whether a CPU should have horizontal or vertical polarity." ?
> # On horizontal polarity the host is expected to provision
> # the vCPU equally.
Maybe: "all vCPUs equally" ?
Or: "each vCPU equally" ?
> # On vertical polarity the host can provision each vCPU
> # differently
> # The guest can get information on the provisioning with
> # the STSI(15) instruction.
Thomas
^ permalink raw reply [flat|nested] 80+ messages in thread
* Re: [PATCH v14 10/11] qapi/s390/cpu topology: POLARITY_CHANGE qapi event
2023-01-20 11:56 ` Thomas Huth
@ 2023-01-20 14:22 ` Pierre Morel
0 siblings, 0 replies; 80+ messages in thread
From: Pierre Morel @ 2023-01-20 14:22 UTC (permalink / raw)
To: Thomas Huth, qemu-s390x
Cc: qemu-devel, borntraeger, pasic, richard.henderson, david, cohuck,
mst, pbonzini, kvm, ehabkost, marcel.apfelbaum, eblake, armbru,
seiden, nrb, scgl, frankja, berrange, clg
On 1/20/23 12:56, Thomas Huth wrote:
> On 18/01/2023 18.09, Pierre Morel wrote:
>>
>> On 1/12/23 12:52, Thomas Huth wrote:
>>> On 05/01/2023 15.53, Pierre Morel wrote:
> ...>>> +#
OK
>>>> +# Emitted when the guest asks to change the polarity.
>>>> +#
>>>> +# @polarity: polarity specified by the guest
>>>
>>> Please elaborate: Where does the value come from (the PTF
>>> instruction)? Which values are possible?
>>
>> Yes what about:
>>
>> # @polarity: the guest can specify with the PTF instruction a horizontal
>> # or a vertical polarity.
>
> Maybe something like: "The guest can tell the host (via the PTF
> instruction) whether a CPU should have horizontal or vertical polarity." ?
Yes thanks, much better.
>
>> # On horizontal polarity the host is expected to provision
>> # the vCPU equally.
>
> Maybe: "all vCPUs equally" ?
> Or: "each vCPU equally" ?
yes, thx.
>
>> # On vertical polarity the host can provision each vCPU
>> # differently
>> # The guest can get information on the provisioning with
>> # the STSI(15) instruction.
>
> Thomas
>
I make the changes.
Regards,
Pierre
--
Pierre Morel
IBM Lab Boeblingen
^ permalink raw reply [flat|nested] 80+ messages in thread
* Re: [PATCH v14 06/11] s390x/cpu topology: interception of PTF instruction
2023-01-16 18:24 ` Nina Schoetterl-Glausch
2023-01-18 9:54 ` Pierre Morel
@ 2023-01-20 14:32 ` Pierre Morel
1 sibling, 0 replies; 80+ messages in thread
From: Pierre Morel @ 2023-01-20 14:32 UTC (permalink / raw)
To: Nina Schoetterl-Glausch, qemu-s390x
Cc: qemu-devel, borntraeger, pasic, richard.henderson, david, thuth,
cohuck, mst, pbonzini, kvm, ehabkost, marcel.apfelbaum, eblake,
armbru, seiden, nrb, frankja, berrange, clg
On 1/16/23 19:24, Nina Schoetterl-Glausch wrote:
> On Thu, 2023-01-05 at 15:53 +0100, Pierre Morel wrote:
>> When the host supports the CPU topology facility, the PTF
>> instruction with function code 2 is interpreted by the SIE,
>> provided that the userland hypervizor activates the interpretation
>> by using the KVM_CAP_S390_CPU_TOPOLOGY KVM extension.
>>
>> The PTF instructions with function code 0 and 1 are intercepted
>> and must be emulated by the userland hypervizor.
>>
>> During RESET all CPU of the configuration are placed in
>> horizontal polarity.
>>
>> Signed-off-by: Pierre Morel <pmorel@linux.ibm.com>
>> ---
>> include/hw/s390x/cpu-topology.h | 3 +
>> include/hw/s390x/s390-virtio-ccw.h | 6 ++
>> target/s390x/cpu.h | 1 +
>> hw/s390x/cpu-topology.c | 92 ++++++++++++++++++++++++++++++
>> target/s390x/cpu-sysemu.c | 16 ++++++
>> target/s390x/kvm/kvm.c | 11 ++++
>> 6 files changed, 129 insertions(+)
>>
>> diff --git a/include/hw/s390x/cpu-topology.h b/include/hw/s390x/cpu-topology.h
>> index 9571aa70e5..33e23d78b9 100644
>> --- a/include/hw/s390x/cpu-topology.h
>> +++ b/include/hw/s390x/cpu-topology.h
>> @@ -55,11 +55,13 @@ typedef struct S390Topology {
>> QTAILQ_HEAD(, S390TopologyEntry) list;
>> uint8_t *sockets;
>> CpuTopology *smp;
>> + int polarity;
>> } S390Topology;
>>
>> #ifdef CONFIG_KVM
>> bool s390_has_topology(void);
>> void s390_topology_set_cpu(MachineState *ms, S390CPU *cpu, Error **errp);
>> +void s390_topology_set_polarity(int polarity);
>> #else
>> static inline bool s390_has_topology(void)
>> {
>> @@ -68,6 +70,7 @@ static inline bool s390_has_topology(void)
>> static inline void s390_topology_set_cpu(MachineState *ms,
>> S390CPU *cpu,
>> Error **errp) {}
>> +static inline void s390_topology_set_polarity(int polarity) {}
>> #endif
>> extern S390Topology s390_topology;
>>
>> diff --git a/include/hw/s390x/s390-virtio-ccw.h b/include/hw/s390x/s390-virtio-ccw.h
>> index 9bba21a916..c1d46e78af 100644
>> --- a/include/hw/s390x/s390-virtio-ccw.h
>> +++ b/include/hw/s390x/s390-virtio-ccw.h
>> @@ -30,6 +30,12 @@ struct S390CcwMachineState {
>> uint8_t loadparm[8];
>> };
>>
>> +#define S390_PTF_REASON_NONE (0x00 << 8)
>> +#define S390_PTF_REASON_DONE (0x01 << 8)
>> +#define S390_PTF_REASON_BUSY (0x02 << 8)
>> +#define S390_TOPO_FC_MASK 0xffUL
>> +void s390_handle_ptf(S390CPU *cpu, uint8_t r1, uintptr_t ra);
>> +
>> struct S390CcwMachineClass {
>> /*< private >*/
>> MachineClass parent_class;
>> diff --git a/target/s390x/cpu.h b/target/s390x/cpu.h
>> index 01ade07009..5da4041576 100644
>> --- a/target/s390x/cpu.h
>> +++ b/target/s390x/cpu.h
>> @@ -864,6 +864,7 @@ void s390_do_cpu_set_diag318(CPUState *cs, run_on_cpu_data arg);
>> int s390_assign_subch_ioeventfd(EventNotifier *notifier, uint32_t sch_id,
>> int vq, bool assign);
>> void s390_cpu_topology_reset(void);
>> +void s390_cpu_topology_set(void);
>
> I don't like this name much, it's nondescript.
> s390_cpu_topology_set_modified ?
>
>> #ifndef CONFIG_USER_ONLY
>> unsigned int s390_cpu_set_state(uint8_t cpu_state, S390CPU *cpu);
>> #else
>> diff --git a/hw/s390x/cpu-topology.c b/hw/s390x/cpu-topology.c
>> index 438055c612..e6b4692581 100644
>> --- a/hw/s390x/cpu-topology.c
>> +++ b/hw/s390x/cpu-topology.c
>> @@ -97,6 +97,98 @@ static s390_topology_id s390_topology_from_cpu(S390CPU *cpu)
>> }
>>
>> /**
>> + * s390_topology_set_polarity
>> + * @polarity: horizontal or vertical
>> + *
>> + * Changes the polarity of all the CPU in the configuration.
>> + *
>> + * If the dedicated CPU modifier attribute is set a vertical
>> + * polarization is always high (Architecture).
>> + * Otherwise we decide to set it as medium.
>> + *
>> + * Once done, advertise a topology change.
>> + */
>> +void s390_topology_set_polarity(int polarity)
>
> I don't like that this function ignores what kind of vertical polarization is passed,
> it's confusing.
> That seems like a further reason to split horizontal/vertical from the entitlement.
>
Yes.
I will also make the ordering of the TLE at the last moment instead of
doing it during hotplug or QAPI command.
This is something Cedric proposed some time ago and I think it is right,
it will be easier to do it once in STSI than on every change of topology,
hotplug, QAPI and polarization change with PTF.
Regards,
Pierre
--
Pierre Morel
IBM Lab Boeblingen
^ permalink raw reply [flat|nested] 80+ messages in thread
end of thread, other threads:[~2023-01-20 14:32 UTC | newest]
Thread overview: 80+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2023-01-05 14:53 [PATCH v14 00/11] s390x: CPU Topology Pierre Morel
2023-01-05 14:53 ` [PATCH v14 01/11] s390x/cpu topology: adding s390 specificities to CPU topology Pierre Morel
2023-01-10 11:37 ` Thomas Huth
2023-01-16 16:32 ` Pierre Morel
2023-01-17 7:25 ` Thomas Huth
2023-01-13 16:58 ` Nina Schoetterl-Glausch
2023-01-16 17:28 ` Pierre Morel
2023-01-16 20:34 ` Nina Schoetterl-Glausch
2023-01-17 9:49 ` Pierre Morel
2023-01-17 7:22 ` Thomas Huth
2023-01-05 14:53 ` [PATCH v14 02/11] s390x/cpu topology: add topology entries on CPU hotplug Pierre Morel
2023-01-10 13:00 ` Thomas Huth
2023-01-11 9:23 ` Nina Schoetterl-Glausch
2023-01-16 18:24 ` Pierre Morel
2023-01-13 18:15 ` Nina Schoetterl-Glausch
2023-01-17 13:55 ` Pierre Morel
2023-01-17 16:48 ` Nina Schoetterl-Glausch
2023-01-19 13:34 ` Pierre Morel
2023-01-05 14:53 ` [PATCH v14 03/11] target/s390x/cpu topology: handle STSI(15) and build the SYSIB Pierre Morel
2023-01-10 14:29 ` Thomas Huth
2023-01-11 9:16 ` Thomas Huth
2023-01-11 17:14 ` Nina Schoetterl-Glausch
2023-01-17 16:58 ` Pierre Morel
2023-01-17 16:56 ` Pierre Morel
2023-01-18 10:26 ` Thomas Huth
2023-01-18 11:54 ` Nina Schoetterl-Glausch
2023-01-19 13:12 ` Pierre Morel
2023-01-16 13:11 ` Nina Schoetterl-Glausch
2023-01-16 15:39 ` Pierre Morel
2023-01-05 14:53 ` [PATCH v14 04/11] s390x/sclp: reporting the maximum nested topology entries Pierre Morel
2023-01-11 8:57 ` Thomas Huth
2023-01-17 17:36 ` Pierre Morel
2023-01-17 19:58 ` Nina Schoetterl-Glausch
2023-01-19 13:08 ` Pierre Morel
2023-01-11 17:52 ` Nina Schoetterl-Glausch
2023-01-17 17:44 ` Pierre Morel
2023-01-05 14:53 ` [PATCH v14 05/11] s390x/cpu topology: resetting the Topology-Change-Report Pierre Morel
2023-01-11 9:00 ` Thomas Huth
2023-01-17 17:57 ` Pierre Morel
2023-01-05 14:53 ` [PATCH v14 06/11] s390x/cpu topology: interception of PTF instruction Pierre Morel
2023-01-16 18:24 ` Nina Schoetterl-Glausch
2023-01-18 9:54 ` Pierre Morel
2023-01-20 14:32 ` Pierre Morel
2023-01-05 14:53 ` [PATCH v14 07/11] target/s390x/cpu topology: activating CPU topology Pierre Morel
2023-01-11 10:04 ` Thomas Huth
2023-01-18 10:01 ` Pierre Morel
2023-01-05 14:53 ` [PATCH v14 08/11] qapi/s390/cpu topology: change-topology monitor command Pierre Morel
2023-01-05 14:53 ` Pierre Morel
2023-01-11 10:09 ` Thomas Huth
2023-01-12 8:00 ` Thomas Huth
2023-01-18 14:23 ` Pierre Morel
2023-01-12 12:03 ` Daniel P. Berrangé
2023-01-18 13:17 ` Pierre Morel
2023-01-16 21:09 ` Nina Schoetterl-Glausch
2023-01-17 7:30 ` Thomas Huth
2023-01-17 13:31 ` Nina Schoetterl-Glausch
2023-01-18 10:53 ` Thomas Huth
2023-01-18 14:09 ` Pierre Morel
2023-01-18 15:17 ` Kevin Wolf
2023-01-18 15:48 ` Pierre Morel
2023-01-18 14:06 ` Pierre Morel
2023-01-05 14:53 ` [PATCH v14 09/11] qapi/s390/cpu topology: monitor query topology information Pierre Morel
2023-01-12 11:48 ` Thomas Huth
2023-01-18 15:59 ` Pierre Morel
2023-01-12 12:10 ` Daniel P. Berrangé
2023-01-12 17:27 ` Nina Schoetterl-Glausch
2023-01-12 17:30 ` Daniel P. Berrangé
2023-01-18 15:58 ` Pierre Morel
2023-01-18 16:08 ` Daniel P. Berrangé
2023-01-18 16:57 ` Pierre Morel
2023-01-05 14:53 ` [PATCH v14 10/11] qapi/s390/cpu topology: POLARITY_CHANGE qapi event Pierre Morel
2023-01-12 11:52 ` Thomas Huth
2023-01-18 17:09 ` Pierre Morel
2023-01-20 11:56 ` Thomas Huth
2023-01-20 14:22 ` Pierre Morel
2023-01-05 14:53 ` [PATCH v14 11/11] docs/s390x/cpu topology: document s390x cpu topology Pierre Morel
2023-01-12 11:46 ` Thomas Huth
2023-01-19 14:48 ` Pierre Morel
2023-01-12 11:58 ` Daniel P. Berrangé
2023-01-18 17:10 ` Pierre Morel
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.