* [PATCH v4 0/5] s390x: CPU Topology
@ 2021-11-17 16:48 Pierre Morel
2021-11-17 16:48 ` [PATCH v4 1/5] linux-headers update Pierre Morel
` (5 more replies)
0 siblings, 6 replies; 7+ messages in thread
From: Pierre Morel @ 2021-11-17 16:48 UTC (permalink / raw)
To: qemu-s390x
Cc: thuth, david, cohuck, richard.henderson, qemu-devel, pasic, borntraeger
Hi,
This series is a first part of the implementation of CPU topology
for S390 greatly reduced from the first spin.
In particular, we reduced the scope to the S390x specificities, removing
all code touching to SMP or NUMA, with the goal to:
- facilitate review and acceptance
- let for later the SMP part currently actively discussed in mainline
- be able despite the reduction of code to handle CPU topology for S390
using the current S390 topology provided by QEMU with cores and sockets
only.
To use these patches, you will need the Linux series version 4.
You find it there:
https://lkml.org/lkml/2021/9/16/576
Currently this code is for KVM only, I have no idea if it is interesting
to provide a TCG patch. If ever it will be done in another series.
A short introduction
====================
CPU Topology is described in the S390 POP with essentially the description
of two instructions:
PTF Perform Topology function used to poll for topology change
and used to set the polarization but this part is not part of this item.
STSI Store System Information and the SYSIB 15.1.x providing the Topology
configuration.
S390 Topology is a 6 levels hierarchical topology with up to 5 level
of containers. The last topology level, specifying the CPU cores.
This patch series only uses the two lower levels sockets and cores.
To get the information on the topology, S390 provides the STSI
instruction, which stores a structures providing the list of the
containers used in the Machine topology: the SYSIB.
A selector within the STSI instruction allow to chose how many topology
levels will be provide in the SYSIB.
Using the Topology List Entries (TLE) provided inside the SYSIB we
the Linux kernel is able to compute the information about the cache
distance between two cores and can use this information to take
scheduling decisions.
Note:
-----
Z15 reports 3 levels of containers, drawers, book, sockets as
Container-TLEs above the core description inside CPU-TLEs.
The Topology can be seen at several places inside zLinux:
- sysfs: /sys/devices/system/cpu/cpuX/topology
- procfs: /proc/sysinfo and /proc/cpuinfo
- lscpu -e : gives toplogy information
The different Topology levels have names:
- Node - Drawer - Book - sockets or physical package - core
Threads:
Multithreading, is not part of the topology as described by the
SYSIB 15.1.x
The interest of the guest to know the CPU topology is obviously to be
able to optimise the load balancing and the migration of threads.
KVM will have the same interest concerning vCPUs scheduling and cache
optimisation.
The design
==========
1) To be ready for hotplug, I chose an Object oriented design
of the topology containers:
- A node is a bridge on the SYSBUS and defines a "node bus"
- A drawer is hotplug on the "node bus"
- A book on the "drawer bus"
- A socket on the "book bus"
- And the CPU Topology List Entry (CPU-TLE)sits on the socket bus.
These objects will be enhanced with the cache information when
NUMA is implemented.
This also allows for easy retrieval when building the different SYSIB
for Store Topology System Information (STSI)
2) Perform Topology Function (PTF) instruction is made available to the
guest with a new KVM capability and intercepted in QEMU, allowing the
guest to pool for topology changes.
Features and TBD list
=====================
- There is no direct match between IDs shown by:
- lscpu (unrelated numbered list),
- SYSIB 15.1.x (topology ID)
- The CPU number, left column of lscpu, is used to reference a CPU
by Linux tools
While the CPU address is used by QEMU for hotplug.
- Effect of -smp parsing on the topology with an example:
-smp 9,sockets=4,cores=4,maxcpus=16
We have 4 socket each holding 4 cores so that we have a maximum
of 16 CPU, 9 of them are active on boot. (Should be obvious)
# lscpu -e
CPU NODE DRAWER BOOK SOCKET CORE L1d:L1i:L2d:L2i ONLINE CONFIGURED POLARIZATION ADDRESS
0 0 0 0 0 0 0:0:0:0 yes yes horizontal 0
1 0 0 0 0 1 1:1:1:1 yes yes horizontal 1
2 0 0 0 0 2 2:2:2:2 yes yes horizontal 2
3 0 0 0 0 3 3:3:3:3 yes yes horizontal 3
4 0 0 0 1 4 4:4:4:4 yes yes horizontal 4
5 0 0 0 1 5 5:5:5:5 yes yes horizontal 5
6 0 0 0 1 6 6:6:6:6 yes yes horizontal 6
7 0 0 0 1 7 7:7:7:7 yes yes horizontal 7
8 0 0 0 2 8 8:8:8:8 yes yes horizontal 8
#
- To plug a new CPU inside the topology one can simply use the CPU
address like in:
(qemu) device_add host-s390x-cpu,core-id=12
# lscpu -e
CPU NODE DRAWER BOOK SOCKET CORE L1d:L1i:L2d:L2i ONLINE CONFIGURED POLARIZATION ADDRESS
0 0 0 0 0 0 0:0:0:0 yes yes horizontal 0
1 0 0 0 0 1 1:1:1:1 yes yes horizontal 1
2 0 0 0 0 2 2:2:2:2 yes yes horizontal 2
3 0 0 0 0 3 3:3:3:3 yes yes horizontal 3
4 0 0 0 1 4 4:4:4:4 yes yes horizontal 4
5 0 0 0 1 5 5:5:5:5 yes yes horizontal 5
6 0 0 0 1 6 6:6:6:6 yes yes horizontal 6
7 0 0 0 1 7 7:7:7:7 yes yes horizontal 7
8 0 0 0 2 8 8:8:8:8 yes yes horizontal 8
9 - - - - - ::: no yes horizontal 12
# chcpu -e 9
CPU 9 enabled
# lscpu -e
CPU NODE DRAWER BOOK SOCKET CORE L1d:L1i:L2d:L2i ONLINE CONFIGURED POLARIZATION ADDRESS
0 0 0 0 0 0 0:0:0:0 yes yes horizontal 0
1 0 0 0 0 1 1:1:1:1 yes yes horizontal 1
2 0 0 0 0 2 2:2:2:2 yes yes horizontal 2
3 0 0 0 0 3 3:3:3:3 yes yes horizontal 3
4 0 0 0 1 4 4:4:4:4 yes yes horizontal 4
5 0 0 0 1 5 5:5:5:5 yes yes horizontal 5
6 0 0 0 1 6 6:6:6:6 yes yes horizontal 6
7 0 0 0 1 7 7:7:7:7 yes yes horizontal 7
8 0 0 0 2 8 8:8:8:8 yes yes horizontal 8
9 0 0 0 3 9 9:9:9:9 yes yes horizontal 12
#
It is up to the admin level, Libvirt for example, to pin the righ CPU to the right
vCPU, but as we can see without NUMA, chosing separate sockets for CPUs is not easy
without hotplug because without information the code will assign the vCPU and fill
the sockets one after the other.
Note that this is also the default behavior on the LPAR.
Conclusion
==========
This patch, together with the associated KVM patch allows to provide CPU topology
information to the guest.
Currently, only dedicated vCPU and CPU are supported and a NUMA topology can only
be handled using CPU hotplug inside the guest.
Next extensions are to provide:
- adding books and drawers levels
- NUMA using the -numa QEMU parameter.
- Topology information change for shared CPU
Regards,
Pierre
Pierre Morel (5):
linux-headers update
s390x: topology: CPU topology objects and structures
s390x: topology: implementating Store Topology System Information
s390x: CPU topology: CPU topology migration
s390x: kvm: topology: interception of PTF instruction
hw/s390x/cpu-topology.c | 361 ++++++++++++++++++++++++++++
hw/s390x/meson.build | 1 +
hw/s390x/s390-virtio-ccw.c | 54 +++++
include/hw/s390x/cpu-topology.h | 74 ++++++
include/hw/s390x/s390-virtio-ccw.h | 6 +
linux-headers/linux/kvm.h | 1 +
target/s390x/cpu.h | 50 ++++
target/s390x/cpu_features_def.h.inc | 1 +
target/s390x/cpu_models.c | 2 +
target/s390x/cpu_topology.c | 113 +++++++++
target/s390x/gen-features.c | 3 +
target/s390x/kvm/kvm.c | 26 ++
target/s390x/machine.c | 48 ++++
target/s390x/meson.build | 1 +
14 files changed, 741 insertions(+)
create mode 100644 hw/s390x/cpu-topology.c
create mode 100644 include/hw/s390x/cpu-topology.h
create mode 100644 target/s390x/cpu_topology.c
--
2.27.0
Changelog:
- since v3
- Added migration
(Thomas)
- Separated STSI instruction from KVM to prepare TCG
(Thomas)
- Take care of endianess to prepare TCG
(Thomas)
- Added comments on STSI CPU container and PFT instruction
(Thomas)
- Moved enabling the instructions as the last patch
(Thomas)
^ permalink raw reply [flat|nested] 7+ messages in thread
* [PATCH v4 1/5] linux-headers update
2021-11-17 16:48 [PATCH v4 0/5] s390x: CPU Topology Pierre Morel
@ 2021-11-17 16:48 ` Pierre Morel
2021-11-17 16:48 ` [PATCH v4 2/5] s390x: topology: CPU topology objects and structures Pierre Morel
` (4 subsequent siblings)
5 siblings, 0 replies; 7+ messages in thread
From: Pierre Morel @ 2021-11-17 16:48 UTC (permalink / raw)
To: qemu-s390x
Cc: thuth, david, cohuck, richard.henderson, qemu-devel, pasic, borntraeger
Signed-off-by: Pierre Morel <pmorel@linux.ibm.com>
---
| 1 +
1 file changed, 1 insertion(+)
--git a/linux-headers/linux/kvm.h b/linux-headers/linux/kvm.h
index bcaf66cc4d..38e96ea6f7 100644
--- a/linux-headers/linux/kvm.h
+++ b/linux-headers/linux/kvm.h
@@ -1112,6 +1112,7 @@ struct kvm_ppc_resize_hpt {
#define KVM_CAP_BINARY_STATS_FD 203
#define KVM_CAP_EXIT_ON_EMULATION_FAILURE 204
#define KVM_CAP_ARM_MTE 205
+#define KVM_CAP_S390_CPU_TOPOLOGY 206
#ifdef KVM_CAP_IRQ_ROUTING
--
2.27.0
^ permalink raw reply related [flat|nested] 7+ messages in thread
* [PATCH v4 2/5] s390x: topology: CPU topology objects and structures
2021-11-17 16:48 [PATCH v4 0/5] s390x: CPU Topology Pierre Morel
2021-11-17 16:48 ` [PATCH v4 1/5] linux-headers update Pierre Morel
@ 2021-11-17 16:48 ` Pierre Morel
2021-11-17 16:48 ` [PATCH v4 3/5] s390x: topology: implementating Store Topology System Information Pierre Morel
` (3 subsequent siblings)
5 siblings, 0 replies; 7+ messages in thread
From: Pierre Morel @ 2021-11-17 16:48 UTC (permalink / raw)
To: qemu-s390x
Cc: thuth, david, cohuck, richard.henderson, qemu-devel, pasic, borntraeger
We use new objects to have a dynamic administration of the CPU topology.
The highest level object in this implementation is the s390 book and
in this first implementation of CPU topology for S390 we have a single
book.
The book is built as a SYSBUS bridge during the CPU initialization.
Every object under this single book will be build dynamically
immediately after a CPU has be realized if it is needed.
The CPU will fill the sockets once after the other, according to the
number of core per socket defined during the smp parsing.
Each CPU inside a socket will be represented by a bit in a 64bit
unsigned long. Set on plug and clear on unplug of a CPU.
Signed-off-by: Pierre Morel <pmorel@linux.ibm.com>
---
hw/s390x/cpu-topology.c | 361 ++++++++++++++++++++++++++++++++
hw/s390x/meson.build | 1 +
hw/s390x/s390-virtio-ccw.c | 4 +
include/hw/s390x/cpu-topology.h | 74 +++++++
target/s390x/cpu.h | 47 +++++
5 files changed, 487 insertions(+)
create mode 100644 hw/s390x/cpu-topology.c
create mode 100644 include/hw/s390x/cpu-topology.h
diff --git a/hw/s390x/cpu-topology.c b/hw/s390x/cpu-topology.c
new file mode 100644
index 0000000000..9120af843a
--- /dev/null
+++ b/hw/s390x/cpu-topology.c
@@ -0,0 +1,361 @@
+/*
+ * CPU Topology
+ *
+ * Copyright 2021 IBM Corp.
+ * Author(s): Pierre Morel <pmorel@linux.ibm.com>
+
+ * This work is licensed under the terms of the GNU GPL, version 2 or (at
+ * your option) any later version. See the COPYING file in the top-level
+ * directory.
+ */
+
+#include "qemu/osdep.h"
+#include "qapi/error.h"
+#include "qemu/error-report.h"
+#include "hw/sysbus.h"
+#include "hw/s390x/cpu-topology.h"
+#include "hw/qdev-properties.h"
+#include "hw/boards.h"
+#include "qemu/typedefs.h"
+#include "target/s390x/cpu.h"
+#include "hw/s390x/s390-virtio-ccw.h"
+
+static S390TopologyCores *s390_create_cores(S390TopologySocket *socket,
+ int origin)
+{
+ DeviceState *dev;
+ S390TopologyCores *cores;
+ const MachineState *ms = MACHINE(qdev_get_machine());
+
+ if (socket->bus->num_children >= ms->smp.cores) {
+ return NULL;
+ }
+
+ dev = qdev_new(TYPE_S390_TOPOLOGY_CORES);
+ qdev_realize_and_unref(dev, socket->bus, &error_fatal);
+
+ cores = S390_TOPOLOGY_CORES(dev);
+ cores->origin = origin;
+ socket->cnt += 1;
+
+ return cores;
+}
+
+static S390TopologySocket *s390_create_socket(S390TopologyBook *book, int id)
+{
+ DeviceState *dev;
+ S390TopologySocket *socket;
+ const MachineState *ms = MACHINE(qdev_get_machine());
+
+ if (book->bus->num_children >= ms->smp.sockets) {
+ return NULL;
+ }
+
+ dev = qdev_new(TYPE_S390_TOPOLOGY_SOCKET);
+ qdev_realize_and_unref(dev, book->bus, &error_fatal);
+
+ socket = S390_TOPOLOGY_SOCKET(dev);
+ socket->socket_id = id;
+ book->cnt++;
+
+ return socket;
+}
+
+/*
+ * s390_get_cores:
+ * @socket: the socket to search into
+ * @origin: the origin specified for the S390TopologyCores
+ *
+ * returns a pointer to a S390TopologyCores structure within a socket having
+ * the specified origin.
+ * First search if the socket is already containing the S390TopologyCores
+ * structure and if not create one with this origin.
+ */
+static S390TopologyCores *s390_get_cores(S390TopologySocket *socket, int origin)
+{
+ S390TopologyCores *cores;
+ BusChild *kid;
+
+ QTAILQ_FOREACH(kid, &socket->bus->children, sibling) {
+ cores = S390_TOPOLOGY_CORES(kid->child);
+ if (cores->origin == origin) {
+ return cores;
+ }
+ }
+ return s390_create_cores(socket, origin);
+}
+
+/*
+ * s390_get_socket:
+ * @book: The book to search into
+ * @socket_id: the identifier of the socket to search for
+ *
+ * returns a pointer to a S390TopologySocket structure within a book having
+ * the specified socket_id.
+ * First search if the book is already containing the S390TopologySocket
+ * structure and if not create one with this socket_id.
+ */
+static S390TopologySocket *s390_get_socket(S390TopologyBook *book,
+ int socket_id)
+{
+ S390TopologySocket *socket;
+ BusChild *kid;
+
+ QTAILQ_FOREACH(kid, &book->bus->children, sibling) {
+ socket = S390_TOPOLOGY_SOCKET(kid->child);
+ if (socket->socket_id == socket_id) {
+ return socket;
+ }
+ }
+ return s390_create_socket(book, socket_id);
+}
+
+/*
+ * s390_topology_new_cpu:
+ * @core_id: the core ID is machine wide
+ *
+ * We have a single book returned by s390_get_topology(),
+ * then we build the hierarchy on demand.
+ * Note that we do not destroy the hierarchy on error creating
+ * an entry in the topology, we just keep it empty.
+ * We do not need to worry about not finding a topology level
+ * entry this would have been caught during smp parsing.
+ */
+void s390_topology_new_cpu(int core_id)
+{
+ const MachineState *ms = MACHINE(qdev_get_machine());
+ S390TopologyBook *book;
+ S390TopologySocket *socket;
+ S390TopologyCores *cores;
+ int cores_per_socket, sock_idx;
+ int origin, bit;
+
+ book = s390_get_topology();
+
+ cores_per_socket = ms->smp.max_cpus / ms->smp.sockets;
+
+ sock_idx = (core_id / cores_per_socket);
+ socket = s390_get_socket(book, sock_idx);
+
+ /*
+ * At the core level, each CPU is represented by a bit in a 64bit
+ * unsigned long. Set on plug and clear on unplug of a CPU.
+ * The firmware assume that all CPU in the core description have the same
+ * type, polarization and are all dedicated or shared.
+ * In the case a socket contains CPU with different type, polarization
+ * or dedication then they will be defined in different CPU containers.
+ * Currently we assume all CPU are identical and the only reason to have
+ * several S390TopologyCores inside a socket is to have more than 64 CPUs
+ * in that case the origin field, representing the offset of the first CPU
+ * in the CPU container allows to represent up to the maximal number of
+ * CPU inside several CPU containers inside the socket container.
+ */
+ origin = 64 * (core_id / 64);
+
+ cores = s390_get_cores(socket, origin);
+
+ bit = 63 - (core_id - origin);
+ set_bit(bit, &cores->mask);
+ cores->origin = origin;
+}
+
+/*
+ * Setting the first topology: 1 book, 1 socket
+ * This is enough for 64 cores if the topology is flat (single socket)
+ */
+void s390_topology_setup(MachineState *ms)
+{
+ DeviceState *dev;
+
+ /* Create BOOK bridge device */
+ dev = qdev_new(TYPE_S390_TOPOLOGY_BOOK);
+ object_property_add_child(qdev_get_machine(),
+ TYPE_S390_TOPOLOGY_BOOK, OBJECT(dev));
+ sysbus_realize_and_unref(SYS_BUS_DEVICE(dev), &error_fatal);
+}
+
+S390TopologyBook *s390_get_topology(void)
+{
+ static S390TopologyBook *book;
+
+ if (!book) {
+ book = S390_TOPOLOGY_BOOK(
+ object_resolve_path(TYPE_S390_TOPOLOGY_BOOK, NULL));
+ assert(book != NULL);
+ }
+
+ return book;
+}
+
+/* --- CORES Definitions --- */
+
+static Property s390_topology_cores_properties[] = {
+ DEFINE_PROP_BOOL("dedicated", S390TopologyCores, dedicated, false),
+ DEFINE_PROP_UINT8("polarity", S390TopologyCores, polarity,
+ S390_TOPOLOGY_POLARITY_H),
+ DEFINE_PROP_UINT8("cputype", S390TopologyCores, cputype,
+ S390_TOPOLOGY_CPU_TYPE),
+ DEFINE_PROP_UINT16("origin", S390TopologyCores, origin, 0),
+ DEFINE_PROP_UINT64("mask", S390TopologyCores, mask, 0),
+ DEFINE_PROP_UINT8("id", S390TopologyCores, id, 0),
+ DEFINE_PROP_END_OF_LIST(),
+};
+
+static void cpu_cores_class_init(ObjectClass *oc, void *data)
+{
+ DeviceClass *dc = DEVICE_CLASS(oc);
+ HotplugHandlerClass *hc = HOTPLUG_HANDLER_CLASS(oc);
+
+ device_class_set_props(dc, s390_topology_cores_properties);
+ hc->unplug = qdev_simple_device_unplug_cb;
+ dc->bus_type = TYPE_S390_TOPOLOGY_SOCKET_BUS;
+ dc->desc = "topology cpu entry";
+}
+
+static const TypeInfo cpu_cores_info = {
+ .name = TYPE_S390_TOPOLOGY_CORES,
+ .parent = TYPE_DEVICE,
+ .instance_size = sizeof(S390TopologyCores),
+ .class_init = cpu_cores_class_init,
+ .interfaces = (InterfaceInfo[]) {
+ { TYPE_HOTPLUG_HANDLER },
+ { }
+ }
+};
+
+/* --- SOCKETS Definitions --- */
+static Property s390_topology_socket_properties[] = {
+ DEFINE_PROP_UINT8("socket_id", S390TopologySocket, socket_id, 0),
+ DEFINE_PROP_END_OF_LIST(),
+};
+
+static char *socket_bus_get_dev_path(DeviceState *dev)
+{
+ S390TopologySocket *socket = S390_TOPOLOGY_SOCKET(dev);
+ DeviceState *book = dev->parent_bus->parent;
+ char *id = qdev_get_dev_path(book);
+ char *ret;
+
+ if (id) {
+ ret = g_strdup_printf("%s:%02d", id, socket->socket_id);
+ g_free(id);
+ } else {
+ ret = g_strdup_printf("_:%02d", socket->socket_id);
+ }
+
+ return ret;
+}
+
+static void socket_bus_class_init(ObjectClass *oc, void *data)
+{
+ BusClass *k = BUS_CLASS(oc);
+
+ k->get_dev_path = socket_bus_get_dev_path;
+ k->max_dev = S390_MAX_SOCKETS;
+}
+
+static const TypeInfo socket_bus_info = {
+ .name = TYPE_S390_TOPOLOGY_SOCKET_BUS,
+ .parent = TYPE_BUS,
+ .instance_size = 0,
+ .class_init = socket_bus_class_init,
+};
+
+static void s390_socket_device_realize(DeviceState *dev, Error **errp)
+{
+ S390TopologySocket *socket = S390_TOPOLOGY_SOCKET(dev);
+ BusState *bus;
+
+ bus = qbus_new(TYPE_S390_TOPOLOGY_SOCKET_BUS, dev,
+ TYPE_S390_TOPOLOGY_SOCKET_BUS);
+ qbus_set_hotplug_handler(bus, OBJECT(dev));
+ socket->bus = bus;
+}
+
+static void socket_class_init(ObjectClass *oc, void *data)
+{
+ DeviceClass *dc = DEVICE_CLASS(oc);
+ HotplugHandlerClass *hc = HOTPLUG_HANDLER_CLASS(oc);
+
+ hc->unplug = qdev_simple_device_unplug_cb;
+ set_bit(DEVICE_CATEGORY_BRIDGE, dc->categories);
+ dc->bus_type = TYPE_S390_TOPOLOGY_BOOK_BUS;
+ dc->realize = s390_socket_device_realize;
+ device_class_set_props(dc, s390_topology_socket_properties);
+ dc->desc = "topology socket";
+}
+
+static const TypeInfo socket_info = {
+ .name = TYPE_S390_TOPOLOGY_SOCKET,
+ .parent = TYPE_DEVICE,
+ .instance_size = sizeof(S390TopologySocket),
+ .class_init = socket_class_init,
+ .interfaces = (InterfaceInfo[]) {
+ { TYPE_HOTPLUG_HANDLER },
+ { }
+ }
+};
+
+static char *book_bus_get_dev_path(DeviceState *dev)
+{
+ return g_strdup_printf("00");
+}
+
+static void book_bus_class_init(ObjectClass *oc, void *data)
+{
+ BusClass *k = BUS_CLASS(oc);
+
+ k->get_dev_path = book_bus_get_dev_path;
+ k->max_dev = S390_MAX_BOOKS;
+}
+
+static const TypeInfo book_bus_info = {
+ .name = TYPE_S390_TOPOLOGY_BOOK_BUS,
+ .parent = TYPE_BUS,
+ .instance_size = 0,
+ .class_init = book_bus_class_init,
+};
+
+static void s390_book_device_realize(DeviceState *dev, Error **errp)
+{
+ S390TopologyBook *book = S390_TOPOLOGY_BOOK(dev);
+ BusState *bus;
+
+ bus = qbus_new(TYPE_S390_TOPOLOGY_BOOK_BUS, dev,
+ TYPE_S390_TOPOLOGY_BOOK_BUS);
+ qbus_set_hotplug_handler(bus, OBJECT(dev));
+ book->bus = bus;
+}
+
+static void book_class_init(ObjectClass *oc, void *data)
+{
+ DeviceClass *dc = DEVICE_CLASS(oc);
+ HotplugHandlerClass *hc = HOTPLUG_HANDLER_CLASS(oc);
+
+ hc->unplug = qdev_simple_device_unplug_cb;
+ set_bit(DEVICE_CATEGORY_BRIDGE, dc->categories);
+ dc->realize = s390_book_device_realize;
+ dc->desc = "topology book";
+}
+
+static const TypeInfo book_info = {
+ .name = TYPE_S390_TOPOLOGY_BOOK,
+ .parent = TYPE_SYS_BUS_DEVICE,
+ .instance_size = sizeof(S390TopologyBook),
+ .class_init = book_class_init,
+ .interfaces = (InterfaceInfo[]) {
+ { TYPE_HOTPLUG_HANDLER },
+ { }
+ }
+};
+
+static void topology_register(void)
+{
+ type_register_static(&cpu_cores_info);
+ type_register_static(&socket_bus_info);
+ type_register_static(&socket_info);
+ type_register_static(&book_bus_info);
+ type_register_static(&book_info);
+}
+
+type_init(topology_register);
diff --git a/hw/s390x/meson.build b/hw/s390x/meson.build
index 28484256ec..74678861cf 100644
--- a/hw/s390x/meson.build
+++ b/hw/s390x/meson.build
@@ -2,6 +2,7 @@ s390x_ss = ss.source_set()
s390x_ss.add(files(
'ap-bridge.c',
'ap-device.c',
+ 'cpu-topology.c',
'ccw-device.c',
'css-bridge.c',
'css.c',
diff --git a/hw/s390x/s390-virtio-ccw.c b/hw/s390x/s390-virtio-ccw.c
index 653587ea62..8b624c2e0c 100644
--- a/hw/s390x/s390-virtio-ccw.c
+++ b/hw/s390x/s390-virtio-ccw.c
@@ -42,6 +42,7 @@
#include "sysemu/sysemu.h"
#include "hw/s390x/pv.h"
#include "migration/blocker.h"
+#include "hw/s390x/cpu-topology.h"
static Error *pv_mig_blocker;
@@ -88,6 +89,7 @@ static void s390_init_cpus(MachineState *machine)
/* initialize possible_cpus */
mc->possible_cpu_arch_ids(machine);
+ s390_topology_setup(machine);
for (i = 0; i < machine->smp.cpus; i++) {
s390x_new_cpu(machine->cpu_type, i, &error_fatal);
}
@@ -305,6 +307,8 @@ static void s390_cpu_plug(HotplugHandler *hotplug_dev,
g_assert(!ms->possible_cpus->cpus[cpu->env.core_id].cpu);
ms->possible_cpus->cpus[cpu->env.core_id].cpu = OBJECT(dev);
+ s390_topology_new_cpu(cpu->env.core_id);
+
if (dev->hotplugged) {
raise_irq_cpu_hotplug();
}
diff --git a/include/hw/s390x/cpu-topology.h b/include/hw/s390x/cpu-topology.h
new file mode 100644
index 0000000000..e6e013a8b8
--- /dev/null
+++ b/include/hw/s390x/cpu-topology.h
@@ -0,0 +1,74 @@
+/*
+ * CPU Topology
+ *
+ * Copyright 2021 IBM Corp.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or (at
+ * your option) any later version. See the COPYING file in the top-level
+ * directory.
+ */
+#ifndef HW_S390X_CPU_TOPOLOGY_H
+#define HW_S390X_CPU_TOPOLOGY_H
+
+#include "hw/qdev-core.h"
+#include "qom/object.h"
+
+#define S390_TOPOLOGY_CPU_TYPE 0x03
+
+#define S390_TOPOLOGY_POLARITY_H 0x00
+#define S390_TOPOLOGY_POLARITY_VL 0x01
+#define S390_TOPOLOGY_POLARITY_VM 0x02
+#define S390_TOPOLOGY_POLARITY_VH 0x03
+
+#define TYPE_S390_TOPOLOGY_CORES "topology cores"
+ /*
+ * Each CPU inside a socket will be represented by a bit in a 64bit
+ * unsigned long. Set on plug and clear on unplug of a CPU.
+ * All CPU inside a mask share the same dedicated, polarity and
+ * cputype values.
+ * The origin is the offset of the first CPU in a mask.
+ */
+struct S390TopologyCores {
+ DeviceState parent_obj;
+ uint8_t id;
+ bool dedicated;
+ uint8_t polarity;
+ uint8_t cputype;
+ uint16_t origin;
+ uint64_t mask;
+ int cnt;
+};
+typedef struct S390TopologyCores S390TopologyCores;
+OBJECT_DECLARE_SIMPLE_TYPE(S390TopologyCores, S390_TOPOLOGY_CORES)
+
+#define TYPE_S390_TOPOLOGY_SOCKET "topology socket"
+#define TYPE_S390_TOPOLOGY_SOCKET_BUS "socket-bus"
+struct S390TopologySocket {
+ DeviceState parent_obj;
+ BusState *bus;
+ uint8_t socket_id;
+ int cnt;
+};
+typedef struct S390TopologySocket S390TopologySocket;
+OBJECT_DECLARE_SIMPLE_TYPE(S390TopologySocket, S390_TOPOLOGY_SOCKET)
+#define S390_MAX_SOCKETS 4
+
+#define TYPE_S390_TOPOLOGY_BOOK "topology book"
+#define TYPE_S390_TOPOLOGY_BOOK_BUS "book-bus"
+struct S390TopologyBook {
+ SysBusDevice parent_obj;
+ BusState *bus;
+ uint8_t book_id;
+ int cnt;
+};
+typedef struct S390TopologyBook S390TopologyBook;
+OBJECT_DECLARE_SIMPLE_TYPE(S390TopologyBook, S390_TOPOLOGY_BOOK)
+#define S390_MAX_BOOKS 1
+
+S390TopologyBook *s390_init_topology(void);
+
+S390TopologyBook *s390_get_topology(void);
+void s390_topology_setup(MachineState *ms);
+void s390_topology_new_cpu(int core_id);
+
+#endif
diff --git a/target/s390x/cpu.h b/target/s390x/cpu.h
index 3153d053e9..9dfe183b89 100644
--- a/target/s390x/cpu.h
+++ b/target/s390x/cpu.h
@@ -564,6 +564,53 @@ typedef union SysIB {
} SysIB;
QEMU_BUILD_BUG_ON(sizeof(SysIB) != 4096);
+/* CPU type Topology List Entry */
+typedef struct SysIBTl_cpu {
+ uint8_t nl;
+ uint8_t reserved0[3];
+ uint8_t reserved1:5;
+ uint8_t dedicated:1;
+ uint8_t polarity:2;
+ uint8_t type;
+ uint16_t origin;
+ uint64_t mask;
+} SysIBTl_cpu;
+QEMU_BUILD_BUG_ON(sizeof(SysIBTl_cpu) != 16);
+
+/* Container type Topology List Entry */
+typedef struct SysIBTl_container {
+ uint8_t nl;
+ uint8_t reserved[6];
+ uint8_t id;
+} QEMU_PACKED SysIBTl_container;
+QEMU_BUILD_BUG_ON(sizeof(SysIBTl_container) != 8);
+
+/* Generic Topology List Entry */
+typedef union SysIBTl_entry {
+ uint8_t nl;
+ SysIBTl_container container;
+ SysIBTl_cpu cpu;
+} SysIBTl_entry;
+
+#define TOPOLOGY_NR_MAG 6
+#define TOPOLOGY_NR_MAG6 0
+#define TOPOLOGY_NR_MAG5 1
+#define TOPOLOGY_NR_MAG4 2
+#define TOPOLOGY_NR_MAG3 3
+#define TOPOLOGY_NR_MAG2 4
+#define TOPOLOGY_NR_MAG1 5
+/* Configuration topology */
+typedef struct SysIB_151x {
+ uint8_t res0[2];
+ uint16_t length;
+ uint8_t mag[TOPOLOGY_NR_MAG];
+ uint8_t res1;
+ uint8_t mnest;
+ uint32_t res2;
+ SysIBTl_entry tle[0];
+} SysIB_151x;
+QEMU_BUILD_BUG_ON(sizeof(SysIB_151x) != 16);
+
/* MMU defines */
#define ASCE_ORIGIN (~0xfffULL) /* segment table origin */
#define ASCE_SUBSPACE 0x200 /* subspace group control */
--
2.27.0
^ permalink raw reply related [flat|nested] 7+ messages in thread
* [PATCH v4 3/5] s390x: topology: implementating Store Topology System Information
2021-11-17 16:48 [PATCH v4 0/5] s390x: CPU Topology Pierre Morel
2021-11-17 16:48 ` [PATCH v4 1/5] linux-headers update Pierre Morel
2021-11-17 16:48 ` [PATCH v4 2/5] s390x: topology: CPU topology objects and structures Pierre Morel
@ 2021-11-17 16:48 ` Pierre Morel
2021-11-17 16:48 ` [PATCH v4 4/5] s390x: CPU topology: CPU topology migration Pierre Morel
` (2 subsequent siblings)
5 siblings, 0 replies; 7+ messages in thread
From: Pierre Morel @ 2021-11-17 16:48 UTC (permalink / raw)
To: qemu-s390x
Cc: thuth, david, cohuck, richard.henderson, qemu-devel, pasic, borntraeger
The handling of STSI is enhanced with the interception of the
function code 15 for storing CPU topology.
Using the objects built during the pluging of CPU, we build the
SYSIB 15_1_x structures.
With this patch the maximum MNEST level is 2, this is also
the only level allowed and only SYSIB 15_1_2 will be built.
Signed-off-by: Pierre Morel <pmorel@linux.ibm.com>
---
target/s390x/cpu.h | 1 +
target/s390x/cpu_topology.c | 113 ++++++++++++++++++++++++++++++++++++
target/s390x/kvm/kvm.c | 5 ++
target/s390x/meson.build | 1 +
4 files changed, 120 insertions(+)
create mode 100644 target/s390x/cpu_topology.c
diff --git a/target/s390x/cpu.h b/target/s390x/cpu.h
index 9dfe183b89..de05cc48cc 100644
--- a/target/s390x/cpu.h
+++ b/target/s390x/cpu.h
@@ -897,4 +897,5 @@ typedef S390CPU ArchCPU;
#include "exec/cpu-all.h"
+void insert_stsi_15_1_x(S390CPU *cpu, int sel2, __u64 addr, uint8_t ar);
#endif
diff --git a/target/s390x/cpu_topology.c b/target/s390x/cpu_topology.c
new file mode 100644
index 0000000000..e74fbdabd2
--- /dev/null
+++ b/target/s390x/cpu_topology.c
@@ -0,0 +1,113 @@
+/*
+ * QEMU S390x CPU Topology
+ *
+ * Copyright IBM Corp. 2021
+ * Author(s): Pierre Morel <pmorel@linux.ibm.com>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or (at
+ * your option) any later version. See the COPYING file in the top-level
+ * directory.
+ */
+
+#include "qemu/osdep.h"
+#include "cpu.h"
+#include "hw/s390x/pv.h"
+#include "hw/sysbus.h"
+#include "hw/s390x/cpu-topology.h"
+
+static int stsi_15_container(void *p, int nl, int id)
+{
+ SysIBTl_container *tle = (SysIBTl_container *)p;
+
+ tle->nl = nl;
+ tle->id = id;
+
+ return sizeof(*tle);
+}
+
+static int stsi_15_cpus(void *p, S390TopologyCores *cd)
+{
+ SysIBTl_cpu *tle = (SysIBTl_cpu *)p;
+
+ tle->nl = 0;
+ tle->dedicated = cd->dedicated;
+ tle->polarity = cd->polarity;
+ tle->type = cd->cputype;
+ tle->origin = be16_to_cpu(cd->origin);
+ tle->mask = be64_to_cpu(cd->mask);
+
+ return sizeof(*tle);
+}
+
+static int set_socket(const MachineState *ms, void *p,
+ S390TopologySocket *socket)
+{
+ BusChild *kid;
+ int l, len = 0;
+
+ len += stsi_15_container(p, 1, socket->socket_id);
+ p += len;
+
+ QTAILQ_FOREACH_REVERSE(kid, &socket->bus->children, sibling) {
+ l = stsi_15_cpus(p, S390_TOPOLOGY_CORES(kid->child));
+ p += l;
+ len += l;
+ }
+ return len;
+}
+
+static void insert_stsi_15_1_2(const MachineState *ms, void *p)
+{
+ S390TopologyBook *book;
+ SysIB_151x *sysib;
+ BusChild *kid;
+ int level = 2;
+ int len, l;
+
+ sysib = (SysIB_151x *)p;
+ sysib->mnest = level;
+ sysib->mag[TOPOLOGY_NR_MAG2] = ms->smp.sockets;
+ sysib->mag[TOPOLOGY_NR_MAG1] = ms->smp.cores;
+
+ book = s390_get_topology();
+ len = sizeof(SysIB_151x);
+ p += len;
+
+ QTAILQ_FOREACH_REVERSE(kid, &book->bus->children, sibling) {
+ l = set_socket(ms, p, S390_TOPOLOGY_SOCKET(kid->child));
+ p += l;
+ len += l;
+ }
+
+ sysib->length = be16_to_cpu(len);
+}
+
+void insert_stsi_15_1_x(S390CPU *cpu, int sel2, __u64 addr, uint8_t ar)
+{
+ const MachineState *machine = MACHINE(qdev_get_machine());
+ void *p;
+ int ret, cc;
+
+ /*
+ * Until the SCLP STSI Facility reporting the MNEST value is used,
+ * a sel2 value of 2 is the only value allowed in STSI 15.1.x.
+ */
+ if (sel2 != 2) {
+ setcc(cpu, 3);
+ return;
+ }
+
+ p = g_malloc0(TARGET_PAGE_SIZE);
+
+ insert_stsi_15_1_2(machine, p);
+
+ if (s390_is_pv()) {
+ ret = s390_cpu_pv_mem_write(cpu, 0, p, TARGET_PAGE_SIZE);
+ } else {
+ ret = s390_cpu_virt_mem_write(cpu, addr, ar, p, TARGET_PAGE_SIZE);
+ }
+ cc = ret ? 3 : 0;
+ setcc(cpu, cc);
+ g_free(p);
+}
+
diff --git a/target/s390x/kvm/kvm.c b/target/s390x/kvm/kvm.c
index 5b1fdb55c4..c17e92fc9c 100644
--- a/target/s390x/kvm/kvm.c
+++ b/target/s390x/kvm/kvm.c
@@ -52,6 +52,7 @@
#include "hw/s390x/s390-virtio-ccw.h"
#include "hw/s390x/s390-virtio-hcall.h"
#include "hw/s390x/pv.h"
+#include "hw/s390x/cpu-topology.h"
#ifndef DEBUG_KVM
#define DEBUG_KVM 0
@@ -1906,6 +1907,10 @@ static int handle_stsi(S390CPU *cpu)
/* Only sysib 3.2.2 needs post-handling for now. */
insert_stsi_3_2_2(cpu, run->s390_stsi.addr, run->s390_stsi.ar);
return 0;
+ case 15:
+ insert_stsi_15_1_x(cpu, run->s390_stsi.sel2, run->s390_stsi.addr,
+ run->s390_stsi.ar);
+ return 0;
default:
return 0;
}
diff --git a/target/s390x/meson.build b/target/s390x/meson.build
index 84c1402a6a..890ccfa789 100644
--- a/target/s390x/meson.build
+++ b/target/s390x/meson.build
@@ -29,6 +29,7 @@ s390x_softmmu_ss.add(files(
'sigp.c',
'cpu-sysemu.c',
'cpu_models_sysemu.c',
+ 'cpu_topology.c',
))
s390x_user_ss = ss.source_set()
--
2.27.0
^ permalink raw reply related [flat|nested] 7+ messages in thread
* [PATCH v4 4/5] s390x: CPU topology: CPU topology migration
2021-11-17 16:48 [PATCH v4 0/5] s390x: CPU Topology Pierre Morel
` (2 preceding siblings ...)
2021-11-17 16:48 ` [PATCH v4 3/5] s390x: topology: implementating Store Topology System Information Pierre Morel
@ 2021-11-17 16:48 ` Pierre Morel
2021-11-17 16:48 ` [PATCH v4 5/5] s390x: kvm: topology: interception of PTF instruction Pierre Morel
2021-12-09 15:09 ` [PATCH v4 0/5] s390x: CPU Topology Pierre Morel
5 siblings, 0 replies; 7+ messages in thread
From: Pierre Morel @ 2021-11-17 16:48 UTC (permalink / raw)
To: qemu-s390x
Cc: thuth, david, cohuck, richard.henderson, qemu-devel, pasic, borntraeger
Both source and target must have the same configuration
regarding the activation of Perform Topology Function
and Store Topology System Information.
Signed-off-by: Pierre Morel <pmorel@linux.ibm.com>
---
target/s390x/cpu.h | 2 ++
target/s390x/cpu_features_def.h.inc | 1 +
target/s390x/cpu_models.c | 2 ++
target/s390x/gen-features.c | 3 ++
target/s390x/kvm/kvm.c | 6 ++++
target/s390x/machine.c | 48 +++++++++++++++++++++++++++++
6 files changed, 62 insertions(+)
diff --git a/target/s390x/cpu.h b/target/s390x/cpu.h
index de05cc48cc..e3ca549774 100644
--- a/target/s390x/cpu.h
+++ b/target/s390x/cpu.h
@@ -150,6 +150,8 @@ struct CPUS390XState {
/* currently processed sigp order */
uint8_t sigp_order;
+ /* Using Perform CPU Topology Function*/
+ bool using_ptf;
};
static inline uint64_t *get_freg(CPUS390XState *cs, int nr)
diff --git a/target/s390x/cpu_features_def.h.inc b/target/s390x/cpu_features_def.h.inc
index e86662bb3b..27e8904978 100644
--- a/target/s390x/cpu_features_def.h.inc
+++ b/target/s390x/cpu_features_def.h.inc
@@ -146,6 +146,7 @@ DEF_FEAT(SIE_CEI, "cei", SCLP_CPU, 43, "SIE: Conditional-external-interception f
DEF_FEAT(DAT_ENH_2, "dateh2", MISC, 0, "DAT-enhancement facility 2")
DEF_FEAT(CMM, "cmm", MISC, 0, "Collaborative-memory-management facility")
DEF_FEAT(AP, "ap", MISC, 0, "AP instructions installed")
+DEF_FEAT(CPU_TOPOLOGY, "cpu_topology", MISC, 0, "CPU Topology available")
/* Features exposed via the PLO instruction. */
DEF_FEAT(PLO_CL, "plo-cl", PLO, 0, "PLO Compare and load (32 bit in general registers)")
diff --git a/target/s390x/cpu_models.c b/target/s390x/cpu_models.c
index 11e06cc51f..1f9ea6479d 100644
--- a/target/s390x/cpu_models.c
+++ b/target/s390x/cpu_models.c
@@ -238,6 +238,7 @@ bool s390_has_feat(S390Feat feat)
if (s390_is_pv()) {
switch (feat) {
+ case S390_FEAT_CPU_TOPOLOGY:
case S390_FEAT_DIAG_318:
case S390_FEAT_HPMA2:
case S390_FEAT_SIE_F2:
@@ -467,6 +468,7 @@ static void check_consistency(const S390CPUModel *model)
{ S390_FEAT_DIAG_318, S390_FEAT_EXTENDED_LENGTH_SCCB },
{ S390_FEAT_NNPA, S390_FEAT_VECTOR },
{ S390_FEAT_RDP, S390_FEAT_LOCAL_TLB_CLEARING },
+ { S390_FEAT_CPU_TOPOLOGY, S390_FEAT_CONFIGURATION_TOPOLOGY },
};
int i;
diff --git a/target/s390x/gen-features.c b/target/s390x/gen-features.c
index 7cb1a6ec10..377d8aad90 100644
--- a/target/s390x/gen-features.c
+++ b/target/s390x/gen-features.c
@@ -488,6 +488,7 @@ static uint16_t full_GEN9_GA3[] = {
static uint16_t full_GEN10_GA1[] = {
S390_FEAT_EDAT,
S390_FEAT_CONFIGURATION_TOPOLOGY,
+ S390_FEAT_CPU_TOPOLOGY,
S390_FEAT_GROUP_MSA_EXT_2,
S390_FEAT_ESOP,
S390_FEAT_SIE_PFMFI,
@@ -604,6 +605,8 @@ static uint16_t default_GEN9_GA1[] = {
static uint16_t default_GEN10_GA1[] = {
S390_FEAT_EDAT,
S390_FEAT_GROUP_MSA_EXT_2,
+ S390_FEAT_CONFIGURATION_TOPOLOGY,
+ S390_FEAT_CPU_TOPOLOGY,
};
#define default_GEN10_GA2 EmptyFeat
diff --git a/target/s390x/kvm/kvm.c b/target/s390x/kvm/kvm.c
index c17e92fc9c..6ffc697b51 100644
--- a/target/s390x/kvm/kvm.c
+++ b/target/s390x/kvm/kvm.c
@@ -612,6 +612,7 @@ int kvm_arch_put_registers(CPUState *cs, int level)
} else {
/* prefix is only supported via sync regs */
}
+
return 0;
}
@@ -2427,6 +2428,10 @@ void kvm_s390_get_host_cpu_model(S390CPUModel *model, Error **errp)
clear_bit(S390_FEAT_CMM_NT, model->features);
}
+ if (kvm_check_extension(kvm_state, KVM_CAP_S390_CPU_TOPOLOGY)) {
+ set_bit(S390_FEAT_CPU_TOPOLOGY, model->features);
+ }
+
/* bpb needs kernel support for migration, VSIE and reset */
if (!kvm_check_extension(kvm_state, KVM_CAP_S390_BPB)) {
clear_bit(S390_FEAT_BPB, model->features);
@@ -2535,6 +2540,7 @@ void kvm_s390_apply_cpu_model(const S390CPUModel *model, Error **errp)
error_setg(errp, "KVM: Error configuring CPU subfunctions: %d", rc);
return;
}
+
/* enable CMM via CMMA */
if (test_bit(S390_FEAT_CMM, model->features)) {
kvm_s390_enable_cmma();
diff --git a/target/s390x/machine.c b/target/s390x/machine.c
index 37a076858c..6036e8e856 100644
--- a/target/s390x/machine.c
+++ b/target/s390x/machine.c
@@ -250,6 +250,53 @@ const VMStateDescription vmstate_diag318 = {
}
};
+static int cpu_topology_presave(void *opaque)
+{
+ S390CPU *cpu = opaque;
+
+ cpu->env.using_ptf = s390_has_feat(S390_FEAT_CPU_TOPOLOGY);
+ return 0;
+}
+
+static int cpu_topology_postload(void *opaque, int version_id)
+{
+ S390CPU *cpu = opaque;
+
+ if ((cpu->env.using_ptf == s390_has_feat(S390_FEAT_CPU_TOPOLOGY)) &&
+ (cpu->env.using_ptf == s390_has_feat(S390_FEAT_CONFIGURATION_TOPOLOGY))) {
+ return 0;
+ }
+ if (cpu->env.using_ptf) {
+ error_report("Target needs CPU Topology enabled");
+ } else {
+ if (s390_has_feat(S390_FEAT_CONFIGURATION_TOPOLOGY)) {
+ error_report("Target is not allowed to enable configuration Topology");
+ }
+ if (s390_has_feat(S390_FEAT_CPU_TOPOLOGY)) {
+ error_report("Target is not allowed to enable CPU Topology");
+ }
+ }
+ return -EINVAL;
+}
+
+static bool cpu_topology_needed(void *opaque)
+{
+ return 1;
+}
+
+const VMStateDescription vmstate_cpu_topology = {
+ .name = "cpu/cpu_topology",
+ .version_id = 1,
+ .pre_save = cpu_topology_presave,
+ .post_load = cpu_topology_postload,
+ .minimum_version_id = 1,
+ .needed = cpu_topology_needed,
+ .fields = (VMStateField[]) {
+ VMSTATE_BOOL(env.using_ptf, S390CPU),
+ VMSTATE_END_OF_LIST()
+ }
+};
+
const VMStateDescription vmstate_s390_cpu = {
.name = "cpu",
.post_load = cpu_post_load,
@@ -287,6 +334,7 @@ const VMStateDescription vmstate_s390_cpu = {
&vmstate_bpbc,
&vmstate_etoken,
&vmstate_diag318,
+ &vmstate_cpu_topology,
NULL
},
};
--
2.27.0
^ permalink raw reply related [flat|nested] 7+ messages in thread
* [PATCH v4 5/5] s390x: kvm: topology: interception of PTF instruction
2021-11-17 16:48 [PATCH v4 0/5] s390x: CPU Topology Pierre Morel
` (3 preceding siblings ...)
2021-11-17 16:48 ` [PATCH v4 4/5] s390x: CPU topology: CPU topology migration Pierre Morel
@ 2021-11-17 16:48 ` Pierre Morel
2021-12-09 15:09 ` [PATCH v4 0/5] s390x: CPU Topology Pierre Morel
5 siblings, 0 replies; 7+ messages in thread
From: Pierre Morel @ 2021-11-17 16:48 UTC (permalink / raw)
To: qemu-s390x
Cc: thuth, david, cohuck, richard.henderson, qemu-devel, pasic, borntraeger
When the host supports the CPU topology facility, the PTF
instruction with function code 2 is interpreted by the SIE,
provided that the userland hypervizor activates the interpretation
by using the KVM_CAP_S390_CPU_TOPOLOGY KVM extension.
The PTF instructions with function code 0 and 1 are intercepted
and must be emulated by the userland hypervizor.
Signed-off-by: Pierre Morel <pmorel@linux.ibm.com>
---
hw/s390x/s390-virtio-ccw.c | 50 ++++++++++++++++++++++++++++++
include/hw/s390x/s390-virtio-ccw.h | 6 ++++
target/s390x/kvm/kvm.c | 15 +++++++++
3 files changed, 71 insertions(+)
diff --git a/hw/s390x/s390-virtio-ccw.c b/hw/s390x/s390-virtio-ccw.c
index 8b624c2e0c..6218352e68 100644
--- a/hw/s390x/s390-virtio-ccw.c
+++ b/hw/s390x/s390-virtio-ccw.c
@@ -408,6 +408,56 @@ static void s390_pv_prepare_reset(S390CcwMachineState *ms)
s390_pv_prep_reset();
}
+/*
+ * s390_handle_ptf:
+ *
+ * @register 1: contains the function code
+ *
+ * Function codes 0 and 1 handle the CPU polarization.
+ * We assume an horizontal topology, the only one supported currently
+ * by Linux, consequently we answer to function code 0, requesting
+ * horizontal polarization that it is already the current polarization
+ * and reject vertical polarization request without further explanation.
+ *
+ * Function code 2 is handling topology changes and is interpreted
+ * by the SIE.
+ */
+int s390_handle_ptf(S390CPU *cpu, uint8_t r1, uintptr_t ra)
+{
+ CPUS390XState *env = &cpu->env;
+ uint64_t reg = env->regs[r1];
+ uint8_t fc = reg & S390_TOPO_FC_MASK;
+
+ if (!s390_has_feat(S390_FEAT_CONFIGURATION_TOPOLOGY)) {
+ s390_program_interrupt(env, PGM_OPERATION, ra);
+ return 0;
+ }
+
+ if (env->psw.mask & PSW_MASK_PSTATE) {
+ s390_program_interrupt(env, PGM_PRIVILEGED, ra);
+ return 0;
+ }
+
+ if (reg & ~S390_TOPO_FC_MASK) {
+ s390_program_interrupt(env, PGM_SPECIFICATION, ra);
+ return 0;
+ }
+
+ switch (fc) {
+ case 0: /* Horizontal polarization is already set */
+ env->regs[r1] |= S390_PTF_REASON_DONE;
+ return 2;
+ case 1: /* Vertical polarization is not supported */
+ env->regs[r1] |= S390_PTF_REASON_NONE;
+ return 2;
+ default:
+ /* Note that fc == 2 is interpreted by the SIE */
+ s390_program_interrupt(env, PGM_SPECIFICATION, ra);
+ }
+
+ return 0;
+}
+
static void s390_machine_reset(MachineState *machine)
{
S390CcwMachineState *ms = S390_CCW_MACHINE(machine);
diff --git a/include/hw/s390x/s390-virtio-ccw.h b/include/hw/s390x/s390-virtio-ccw.h
index 3331990e02..ac4b4a92e7 100644
--- a/include/hw/s390x/s390-virtio-ccw.h
+++ b/include/hw/s390x/s390-virtio-ccw.h
@@ -30,6 +30,12 @@ struct S390CcwMachineState {
uint8_t loadparm[8];
};
+#define S390_PTF_REASON_NONE (0x00 << 8)
+#define S390_PTF_REASON_DONE (0x01 << 8)
+#define S390_PTF_REASON_BUSY (0x02 << 8)
+#define S390_TOPO_FC_MASK 0xffUL
+int s390_handle_ptf(S390CPU *cpu, uint8_t r1, uintptr_t ra);
+
struct S390CcwMachineClass {
/*< private >*/
MachineClass parent_class;
diff --git a/target/s390x/kvm/kvm.c b/target/s390x/kvm/kvm.c
index 6ffc697b51..42eda0faee 100644
--- a/target/s390x/kvm/kvm.c
+++ b/target/s390x/kvm/kvm.c
@@ -98,6 +98,7 @@
#define PRIV_B9_EQBS 0x9c
#define PRIV_B9_CLP 0xa0
+#define PRIV_B9_PTF 0xa2
#define PRIV_B9_PCISTG 0xd0
#define PRIV_B9_PCILG 0xd2
#define PRIV_B9_RPCIT 0xd3
@@ -363,6 +364,7 @@ int kvm_arch_init(MachineState *ms, KVMState *s)
kvm_vm_enable_cap(s, KVM_CAP_S390_USER_SIGP, 0);
kvm_vm_enable_cap(s, KVM_CAP_S390_VECTOR_REGISTERS, 0);
kvm_vm_enable_cap(s, KVM_CAP_S390_USER_STSI, 0);
+ kvm_vm_enable_cap(s, KVM_CAP_S390_CPU_TOPOLOGY, 0);
if (ri_allowed()) {
if (kvm_vm_enable_cap(s, KVM_CAP_S390_RI, 0) == 0) {
cap_ri = 1;
@@ -1454,6 +1456,16 @@ static int kvm_mpcifc_service_call(S390CPU *cpu, struct kvm_run *run)
}
}
+static int kvm_handle_ptf(S390CPU *cpu, struct kvm_run *run)
+{
+ uint8_t r1 = (run->s390_sieic.ipb >> 20) & 0x0f;
+ int ret;
+
+ ret = s390_handle_ptf(cpu, r1, RA_IGNORED);
+ setcc(cpu, ret);
+ return 0;
+}
+
static int handle_b9(S390CPU *cpu, struct kvm_run *run, uint8_t ipa1)
{
int r = 0;
@@ -1471,6 +1483,9 @@ static int handle_b9(S390CPU *cpu, struct kvm_run *run, uint8_t ipa1)
case PRIV_B9_RPCIT:
r = kvm_rpcit_service_call(cpu, run);
break;
+ case PRIV_B9_PTF:
+ r = kvm_handle_ptf(cpu, run);
+ break;
case PRIV_B9_EQBS:
/* just inject exception */
r = -1;
--
2.27.0
^ permalink raw reply related [flat|nested] 7+ messages in thread
* Re: [PATCH v4 0/5] s390x: CPU Topology
2021-11-17 16:48 [PATCH v4 0/5] s390x: CPU Topology Pierre Morel
` (4 preceding siblings ...)
2021-11-17 16:48 ` [PATCH v4 5/5] s390x: kvm: topology: interception of PTF instruction Pierre Morel
@ 2021-12-09 15:09 ` Pierre Morel
5 siblings, 0 replies; 7+ messages in thread
From: Pierre Morel @ 2021-12-09 15:09 UTC (permalink / raw)
To: qemu-s390x
Cc: thuth, david, cohuck, richard.henderson, qemu-devel, pasic, borntraeger
Hi,
This series is updated by a v5 series with documentation and numa
extensions.
Some changes have been made in some of the patches contained in this
series too.
Regards,
Pierre
On 11/17/21 17:48, Pierre Morel wrote:
> Hi,
>
> This series is a first part of the implementation of CPU topology
> for S390 greatly reduced from the first spin.
>
> In particular, we reduced the scope to the S390x specificities, removing
> all code touching to SMP or NUMA, with the goal to:
> - facilitate review and acceptance
> - let for later the SMP part currently actively discussed in mainline
> - be able despite the reduction of code to handle CPU topology for S390
> using the current S390 topology provided by QEMU with cores and sockets
> only.
>
> To use these patches, you will need the Linux series version 4.
> You find it there:
> https://lkml.org/lkml/2021/9/16/576
>
> Currently this code is for KVM only, I have no idea if it is interesting
> to provide a TCG patch. If ever it will be done in another series.
>
> A short introduction
> ====================
>
> CPU Topology is described in the S390 POP with essentially the description
> of two instructions:
>
> PTF Perform Topology function used to poll for topology change
> and used to set the polarization but this part is not part of this item.
>
> STSI Store System Information and the SYSIB 15.1.x providing the Topology
> configuration.
>
> S390 Topology is a 6 levels hierarchical topology with up to 5 level
> of containers. The last topology level, specifying the CPU cores.
>
> This patch series only uses the two lower levels sockets and cores.
>
> To get the information on the topology, S390 provides the STSI
> instruction, which stores a structures providing the list of the
> containers used in the Machine topology: the SYSIB.
> A selector within the STSI instruction allow to chose how many topology
> levels will be provide in the SYSIB.
>
> Using the Topology List Entries (TLE) provided inside the SYSIB we
> the Linux kernel is able to compute the information about the cache
> distance between two cores and can use this information to take
> scheduling decisions.
>
> Note:
> -----
> Z15 reports 3 levels of containers, drawers, book, sockets as
> Container-TLEs above the core description inside CPU-TLEs.
>
> The Topology can be seen at several places inside zLinux:
> - sysfs: /sys/devices/system/cpu/cpuX/topology
> - procfs: /proc/sysinfo and /proc/cpuinfo
> - lscpu -e : gives toplogy information
>
> The different Topology levels have names:
> - Node - Drawer - Book - sockets or physical package - core
>
> Threads:
> Multithreading, is not part of the topology as described by the
> SYSIB 15.1.x
>
> The interest of the guest to know the CPU topology is obviously to be
> able to optimise the load balancing and the migration of threads.
> KVM will have the same interest concerning vCPUs scheduling and cache
> optimisation.
>
>
> The design
> ==========
>
> 1) To be ready for hotplug, I chose an Object oriented design
> of the topology containers:
> - A node is a bridge on the SYSBUS and defines a "node bus"
> - A drawer is hotplug on the "node bus"
> - A book on the "drawer bus"
> - A socket on the "book bus"
> - And the CPU Topology List Entry (CPU-TLE)sits on the socket bus.
> These objects will be enhanced with the cache information when
> NUMA is implemented.
>
> This also allows for easy retrieval when building the different SYSIB
> for Store Topology System Information (STSI)
>
> 2) Perform Topology Function (PTF) instruction is made available to the
> guest with a new KVM capability and intercepted in QEMU, allowing the
> guest to pool for topology changes.
>
>
> Features and TBD list
> =====================
>
> - There is no direct match between IDs shown by:
> - lscpu (unrelated numbered list),
> - SYSIB 15.1.x (topology ID)
>
> - The CPU number, left column of lscpu, is used to reference a CPU
> by Linux tools
> While the CPU address is used by QEMU for hotplug.
>
> - Effect of -smp parsing on the topology with an example:
> -smp 9,sockets=4,cores=4,maxcpus=16
>
> We have 4 socket each holding 4 cores so that we have a maximum
> of 16 CPU, 9 of them are active on boot. (Should be obvious)
>
> # lscpu -e
> CPU NODE DRAWER BOOK SOCKET CORE L1d:L1i:L2d:L2i ONLINE CONFIGURED POLARIZATION ADDRESS
> 0 0 0 0 0 0 0:0:0:0 yes yes horizontal 0
> 1 0 0 0 0 1 1:1:1:1 yes yes horizontal 1
> 2 0 0 0 0 2 2:2:2:2 yes yes horizontal 2
> 3 0 0 0 0 3 3:3:3:3 yes yes horizontal 3
> 4 0 0 0 1 4 4:4:4:4 yes yes horizontal 4
> 5 0 0 0 1 5 5:5:5:5 yes yes horizontal 5
> 6 0 0 0 1 6 6:6:6:6 yes yes horizontal 6
> 7 0 0 0 1 7 7:7:7:7 yes yes horizontal 7
> 8 0 0 0 2 8 8:8:8:8 yes yes horizontal 8
> #
>
>
> - To plug a new CPU inside the topology one can simply use the CPU
> address like in:
>
> (qemu) device_add host-s390x-cpu,core-id=12
> # lscpu -e
> CPU NODE DRAWER BOOK SOCKET CORE L1d:L1i:L2d:L2i ONLINE CONFIGURED POLARIZATION ADDRESS
> 0 0 0 0 0 0 0:0:0:0 yes yes horizontal 0
> 1 0 0 0 0 1 1:1:1:1 yes yes horizontal 1
> 2 0 0 0 0 2 2:2:2:2 yes yes horizontal 2
> 3 0 0 0 0 3 3:3:3:3 yes yes horizontal 3
> 4 0 0 0 1 4 4:4:4:4 yes yes horizontal 4
> 5 0 0 0 1 5 5:5:5:5 yes yes horizontal 5
> 6 0 0 0 1 6 6:6:6:6 yes yes horizontal 6
> 7 0 0 0 1 7 7:7:7:7 yes yes horizontal 7
> 8 0 0 0 2 8 8:8:8:8 yes yes horizontal 8
> 9 - - - - - ::: no yes horizontal 12
> # chcpu -e 9
> CPU 9 enabled
> # lscpu -e
> CPU NODE DRAWER BOOK SOCKET CORE L1d:L1i:L2d:L2i ONLINE CONFIGURED POLARIZATION ADDRESS
> 0 0 0 0 0 0 0:0:0:0 yes yes horizontal 0
> 1 0 0 0 0 1 1:1:1:1 yes yes horizontal 1
> 2 0 0 0 0 2 2:2:2:2 yes yes horizontal 2
> 3 0 0 0 0 3 3:3:3:3 yes yes horizontal 3
> 4 0 0 0 1 4 4:4:4:4 yes yes horizontal 4
> 5 0 0 0 1 5 5:5:5:5 yes yes horizontal 5
> 6 0 0 0 1 6 6:6:6:6 yes yes horizontal 6
> 7 0 0 0 1 7 7:7:7:7 yes yes horizontal 7
> 8 0 0 0 2 8 8:8:8:8 yes yes horizontal 8
> 9 0 0 0 3 9 9:9:9:9 yes yes horizontal 12
> #
>
> It is up to the admin level, Libvirt for example, to pin the righ CPU to the right
> vCPU, but as we can see without NUMA, chosing separate sockets for CPUs is not easy
> without hotplug because without information the code will assign the vCPU and fill
> the sockets one after the other.
> Note that this is also the default behavior on the LPAR.
>
> Conclusion
> ==========
>
> This patch, together with the associated KVM patch allows to provide CPU topology
> information to the guest.
> Currently, only dedicated vCPU and CPU are supported and a NUMA topology can only
> be handled using CPU hotplug inside the guest.
>
> Next extensions are to provide:
> - adding books and drawers levels
> - NUMA using the -numa QEMU parameter.
> - Topology information change for shared CPU
>
> Regards,
> Pierre
>
> Pierre Morel (5):
> linux-headers update
> s390x: topology: CPU topology objects and structures
> s390x: topology: implementating Store Topology System Information
> s390x: CPU topology: CPU topology migration
> s390x: kvm: topology: interception of PTF instruction
>
> hw/s390x/cpu-topology.c | 361 ++++++++++++++++++++++++++++
> hw/s390x/meson.build | 1 +
> hw/s390x/s390-virtio-ccw.c | 54 +++++
> include/hw/s390x/cpu-topology.h | 74 ++++++
> include/hw/s390x/s390-virtio-ccw.h | 6 +
> linux-headers/linux/kvm.h | 1 +
> target/s390x/cpu.h | 50 ++++
> target/s390x/cpu_features_def.h.inc | 1 +
> target/s390x/cpu_models.c | 2 +
> target/s390x/cpu_topology.c | 113 +++++++++
> target/s390x/gen-features.c | 3 +
> target/s390x/kvm/kvm.c | 26 ++
> target/s390x/machine.c | 48 ++++
> target/s390x/meson.build | 1 +
> 14 files changed, 741 insertions(+)
> create mode 100644 hw/s390x/cpu-topology.c
> create mode 100644 include/hw/s390x/cpu-topology.h
> create mode 100644 target/s390x/cpu_topology.c
>
--
Pierre Morel
IBM Lab Boeblingen
^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2021-12-09 15:10 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-11-17 16:48 [PATCH v4 0/5] s390x: CPU Topology Pierre Morel
2021-11-17 16:48 ` [PATCH v4 1/5] linux-headers update Pierre Morel
2021-11-17 16:48 ` [PATCH v4 2/5] s390x: topology: CPU topology objects and structures Pierre Morel
2021-11-17 16:48 ` [PATCH v4 3/5] s390x: topology: implementating Store Topology System Information Pierre Morel
2021-11-17 16:48 ` [PATCH v4 4/5] s390x: CPU topology: CPU topology migration Pierre Morel
2021-11-17 16:48 ` [PATCH v4 5/5] s390x: kvm: topology: interception of PTF instruction Pierre Morel
2021-12-09 15:09 ` [PATCH v4 0/5] s390x: CPU Topology Pierre Morel
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.