* [PATCH v7 00/13] APIC ID fixes for AMD EPYC CPU model
@ 2020-03-11 22:52 Babu Moger
2020-03-11 22:52 ` [PATCH v7 01/13] hw/i386: Introduce X86CPUTopoInfo to contain topology info Babu Moger
` (13 more replies)
0 siblings, 14 replies; 38+ messages in thread
From: Babu Moger @ 2020-03-11 22:52 UTC (permalink / raw)
To: ehabkost, marcel.apfelbaum, pbonzini, rth, mst, imammedo; +Cc: qemu-devel
This series fixes APIC ID encoding problem reported on AMD EPYC cpu models.
https://bugzilla.redhat.com/show_bug.cgi?id=1728166
Currently, the APIC ID is decoded based on the sequence
sockets->dies->cores->threads. This works for most standard AMD and other
vendors' configurations, but this decoding sequence does not follow that of
AMD's APIC ID enumeration strictly. In some cases this can cause CPU topology
inconsistency. When booting a guest VM, the kernel tries to validate the
topology, and finds it inconsistent with the enumeration of EPYC cpu models.
To fix the problem we need to build the topology as per the Processor
Programming Reference (PPR) for AMD Family 17h Model 01h, Revision B1
Processors. The documentation is available from the bugzilla Link below.
Link: https://bugzilla.kernel.org/show_bug.cgi?id=206537
Here is the text from the PPR.
Operating systems are expected to use Core::X86::Cpuid::SizeId[ApicIdSize], the
number of least significant bits in the Initial APIC ID that indicate core ID
within a processor, in constructing per-core CPUID masks.
Core::X86::Cpuid::SizeId[ApicIdSize] determines the maximum number of cores
(MNC) that the processor could theoretically support, not the actual number of
cores that are actually implemented or enabled on the processor, as indicated
by Core::X86::Cpuid::SizeId[NC].
Each Core::X86::Apic::ApicId[ApicId] register is preset as follows:
• ApicId[6] = Socket ID.
• ApicId[5:4] = Node ID.
• ApicId[3] = Logical CCX L3 complex ID
• ApicId[2:0]= (SMT) ? {LogicalCoreID[1:0],ThreadId} : {1'b0,LogicalCoreID[1:0]}
v7:
Generated the patches on top of git://github.com/ehabkost/qemu.git (x86-next).
Changes from v6.
1. Added new function x86_set_epyc_topo_handlers to override the apic id
encoding handlers.
2. Separated the code to set use_epyc_apic_id_encoding and added as a new patch
as it looked more logical.
3. Fixed minor typos.
v6:
https://lore.kernel.org/qemu-devel/158389385028.22020.7608244627303132902.stgit@naples-babu.amd.com/
Generated the patches on top of git://github.com/ehabkost/qemu.git (x86-next).
Changes from v5.
1. Eduardo has already queued couple of patches, submitting the rest here.
2. Major change is how the EPYC mode apic id encoding handlers are loaded.
Added a boolean variable use_epyc_apic_id_encoding in X86CPUDefinition.
The variable is will be used to tell if we need to use EPYC mode encoding.
3. Eduardo reported bysectability problem with x86 unit test code.
Quashed the patches in 1 and 2 to resolve it. Problem was change in calling
conventions of topology related functions.
4. Also set the use_epyc_apic_id_encoding for EPYC-Rome. This model is
added recently to the cpu table.
v5:
https://lore.kernel.org/qemu-devel/158326531474.40452.11433722850425537745.stgit@naples-babu.amd.com/
Generated the patches on top of git://github.com/ehabkost/qemu.git (x86-next).
Changes from v4.
1. Re-arranged the patches 2 and 4 as suggested by Igor.
2. Kept the apicid handler functions inside X86MachineState as discussed.
These handlers are loaded from X86CPUDefinitions.
3. Removed unnecessary X86CPUstate initialization from x86_cpu_new. Suggested
by Igor.
4. And other minor changes related to patch format.
v4:
https://lore.kernel.org/qemu-devel/158161767653.48948.10578064482878399556.stgit@naples-babu.amd.com/
Changes from v3.
1. Moved the arch_id calculation inside the function x86_cpus_init. With this change,
we dont need to change common numa code.(suggested by Igor)
2. Introduced the model specific handlers inside X86CPUDefinitions.
These handlers are loaded into X86MachineState during the init.
3. Removed llc_id from x86CPU.
4. Removed init_apicid_fn hanlder from MachineClass. Kept all the code changes
inside the x86.
5. Added new handler function apicid_pkg_offset for pkg_offset calculation.
6. And some Other minor changes.
v3:
https://lore.kernel.org/qemu-devel/157541968844.46157.17994918142533791313.stgit@naples-babu.amd.com/
1. Consolidated the topology information in structure X86CPUTopoInfo.
2. Changed the ccx_id to llc_id as commented by upstream.
3. Generalized the apic id decoding. It is mostly similar to current apic id
except that it adds new field llc_id when numa configured. Removes all the
hardcoded values.
4. Removed the earlier parse_numa split. And moved the numa node initialization
inside the numa_complete_configuration. This is bit cleaner as commented by
Eduardo.
5. Added new function init_apicid_fn inside machine_class structure. This
will be used to update the apic id handler specific to cpu model.
6. Updated the cpuid unit tests.
7. TODO : Need to figure out how to dynamically update the handlers using cpu models.
I might some guidance on that.
v2:
https://lore.kernel.org/qemu-devel/156779689013.21957.1631551572950676212.stgit@localhost.localdomain/
1. Introduced the new property epyc to enable new epyc mode.
2. Separated the epyc mode and non epyc mode function.
3. Introduced function pointers in PCMachineState to handle the
differences.
4. Mildly tested different combinations to make things are working as expected.
5. TODO : Setting the epyc feature bit needs to be worked out. This feature is
supported only on AMD EPYC models. I may need some guidance on that.
v1:
https://lore.kernel.org/qemu-devel/20190731232032.51786-1-babu.moger@amd.com/
---
Babu Moger (13):
hw/i386: Introduce X86CPUTopoInfo to contain topology info
hw/i386: Consolidate topology functions
machine: Add SMP Sockets in CpuTopology
hw/i386: Remove unnecessary initialization in x86_cpu_new
hw/i386: Update structures to save the number of nodes per package
hw/i386: Rename apicid_from_topo_ids to x86_apicid_from_topo_ids
hw/386: Add EPYC mode topology decoding functions
target/i386: Cleanup and use the EPYC mode topology functions
hw/i386: Introduce apicid functions inside X86MachineState
i386: Introduce use_epyc_apic_id_encoding in X86CPUDefinition
hw/i386: Move arch_id decode inside x86_cpus_init
target/i386: Enable new apic id encoding for EPYC based cpus models
i386: Fix pkg_id offset for EPYC cpu models
hw/core/machine.c | 1
hw/i386/pc.c | 15 ++-
hw/i386/x86.c | 73 ++++++++++++----
include/hw/boards.h | 2
include/hw/i386/topology.h | 195 ++++++++++++++++++++++++++++++------------
include/hw/i386/x86.h | 12 +++
softmmu/vl.c | 1
target/i386/cpu.c | 203 ++++++++++++++------------------------------
target/i386/cpu.h | 3 +
tests/test-x86-cpuid.c | 116 +++++++++++++++----------
10 files changed, 358 insertions(+), 263 deletions(-)
--
Signature
^ permalink raw reply [flat|nested] 38+ messages in thread
* [PATCH v7 01/13] hw/i386: Introduce X86CPUTopoInfo to contain topology info
2020-03-11 22:52 [PATCH v7 00/13] APIC ID fixes for AMD EPYC CPU model Babu Moger
@ 2020-03-11 22:52 ` Babu Moger
2020-03-12 11:11 ` Igor Mammedov
2020-03-11 22:52 ` [PATCH v7 02/13] hw/i386: Consolidate topology functions Babu Moger
` (12 subsequent siblings)
13 siblings, 1 reply; 38+ messages in thread
From: Babu Moger @ 2020-03-11 22:52 UTC (permalink / raw)
To: ehabkost, marcel.apfelbaum, pbonzini, rth, mst, imammedo; +Cc: qemu-devel
This is an effort to re-arrange few data structure for better readability.
1. Add X86CPUTopoInfo which will have all the topology informations
required to build the cpu topology. There is no functional changes.
2. Introduce init_topo_info to initialize X86CPUTopoInfo members from
X86MachineState.
3. Update x86 unit tests for new calling convention with parameter X86CPUTopoInfo
There is no functional changes.
Signed-off-by: Babu Moger <babu.moger@amd.com>
---
hw/i386/pc.c | 12 ++++++------
hw/i386/x86.c | 32 ++++++++++++++++++++++++--------
include/hw/i386/topology.h | 38 ++++++++++++++++++++++++--------------
include/hw/i386/x86.h | 3 +++
tests/test-x86-cpuid.c | 43 ++++++++++++++++++++++++-------------------
5 files changed, 81 insertions(+), 47 deletions(-)
diff --git a/hw/i386/pc.c b/hw/i386/pc.c
index f52e84b2ba..662abb549d 100644
--- a/hw/i386/pc.c
+++ b/hw/i386/pc.c
@@ -1513,6 +1513,7 @@ static void pc_cpu_pre_plug(HotplugHandler *hotplug_dev,
X86MachineState *x86ms = X86_MACHINE(pcms);
unsigned int smp_cores = ms->smp.cores;
unsigned int smp_threads = ms->smp.threads;
+ X86CPUTopoInfo topo_info;
if(!object_dynamic_cast(OBJECT(cpu), ms->cpu_type)) {
error_setg(errp, "Invalid CPU type, expected cpu type: '%s'",
@@ -1520,6 +1521,8 @@ static void pc_cpu_pre_plug(HotplugHandler *hotplug_dev,
return;
}
+ init_topo_info(&topo_info, x86ms);
+
env->nr_dies = x86ms->smp_dies;
/*
@@ -1575,16 +1578,14 @@ static void pc_cpu_pre_plug(HotplugHandler *hotplug_dev,
topo_ids.die_id = cpu->die_id;
topo_ids.core_id = cpu->core_id;
topo_ids.smt_id = cpu->thread_id;
- cpu->apic_id = apicid_from_topo_ids(x86ms->smp_dies, smp_cores,
- smp_threads, &topo_ids);
+ cpu->apic_id = apicid_from_topo_ids(&topo_info, &topo_ids);
}
cpu_slot = pc_find_cpu_slot(MACHINE(pcms), cpu->apic_id, &idx);
if (!cpu_slot) {
MachineState *ms = MACHINE(pcms);
- x86_topo_ids_from_apicid(cpu->apic_id, x86ms->smp_dies,
- smp_cores, smp_threads, &topo_ids);
+ x86_topo_ids_from_apicid(cpu->apic_id, &topo_info, &topo_ids);
error_setg(errp,
"Invalid CPU [socket: %u, die: %u, core: %u, thread: %u] with"
" APIC ID %" PRIu32 ", valid index range 0:%d",
@@ -1605,8 +1606,7 @@ static void pc_cpu_pre_plug(HotplugHandler *hotplug_dev,
/* TODO: move socket_id/core_id/thread_id checks into x86_cpu_realizefn()
* once -smp refactoring is complete and there will be CPU private
* CPUState::nr_cores and CPUState::nr_threads fields instead of globals */
- x86_topo_ids_from_apicid(cpu->apic_id, x86ms->smp_dies,
- smp_cores, smp_threads, &topo_ids);
+ x86_topo_ids_from_apicid(cpu->apic_id, &topo_info, &topo_ids);
if (cpu->socket_id != -1 && cpu->socket_id != topo_ids.pkg_id) {
error_setg(errp, "property socket-id: %u doesn't match set apic-id:"
" 0x%x (socket-id: %u)", cpu->socket_id, cpu->apic_id,
diff --git a/hw/i386/x86.c b/hw/i386/x86.c
index 322fb6abbc..dbbff46a4b 100644
--- a/hw/i386/x86.c
+++ b/hw/i386/x86.c
@@ -57,6 +57,16 @@
/* Physical Address of PVH entry point read from kernel ELF NOTE */
static size_t pvh_start_addr;
+inline void init_topo_info(X86CPUTopoInfo *topo_info,
+ const X86MachineState *x86ms)
+{
+ MachineState *ms = MACHINE(x86ms);
+
+ topo_info->dies_per_pkg = x86ms->smp_dies;
+ topo_info->cores_per_die = ms->smp.cores;
+ topo_info->threads_per_core = ms->smp.threads;
+}
+
/*
* Calculates initial APIC ID for a specific CPU index
*
@@ -68,13 +78,14 @@ static size_t pvh_start_addr;
uint32_t x86_cpu_apic_id_from_index(X86MachineState *x86ms,
unsigned int cpu_index)
{
- MachineState *ms = MACHINE(x86ms);
X86MachineClass *x86mc = X86_MACHINE_GET_CLASS(x86ms);
+ X86CPUTopoInfo topo_info;
uint32_t correct_id;
static bool warned;
- correct_id = x86_apicid_from_cpu_idx(x86ms->smp_dies, ms->smp.cores,
- ms->smp.threads, cpu_index);
+ init_topo_info(&topo_info, x86ms);
+
+ correct_id = x86_apicid_from_cpu_idx(&topo_info, cpu_index);
if (x86mc->compat_apic_id_mode) {
if (cpu_index != correct_id && !warned && !qtest_enabled()) {
error_report("APIC IDs set in compatibility mode, "
@@ -145,19 +156,22 @@ int64_t x86_get_default_cpu_node_id(const MachineState *ms, int idx)
{
X86CPUTopoIDs topo_ids;
X86MachineState *x86ms = X86_MACHINE(ms);
+ X86CPUTopoInfo topo_info;
+
+ init_topo_info(&topo_info, x86ms);
assert(idx < ms->possible_cpus->len);
x86_topo_ids_from_apicid(ms->possible_cpus->cpus[idx].arch_id,
- x86ms->smp_dies, ms->smp.cores,
- ms->smp.threads, &topo_ids);
+ &topo_info, &topo_ids);
return topo_ids.pkg_id % ms->numa_state->num_nodes;
}
const CPUArchIdList *x86_possible_cpu_arch_ids(MachineState *ms)
{
X86MachineState *x86ms = X86_MACHINE(ms);
- int i;
unsigned int max_cpus = ms->smp.max_cpus;
+ X86CPUTopoInfo topo_info;
+ int i;
if (ms->possible_cpus) {
/*
@@ -171,6 +185,9 @@ const CPUArchIdList *x86_possible_cpu_arch_ids(MachineState *ms)
ms->possible_cpus = g_malloc0(sizeof(CPUArchIdList) +
sizeof(CPUArchId) * max_cpus);
ms->possible_cpus->len = max_cpus;
+
+ init_topo_info(&topo_info, x86ms);
+
for (i = 0; i < ms->possible_cpus->len; i++) {
X86CPUTopoIDs topo_ids;
@@ -179,8 +196,7 @@ const CPUArchIdList *x86_possible_cpu_arch_ids(MachineState *ms)
ms->possible_cpus->cpus[i].arch_id =
x86_cpu_apic_id_from_index(x86ms, i);
x86_topo_ids_from_apicid(ms->possible_cpus->cpus[i].arch_id,
- x86ms->smp_dies, ms->smp.cores,
- ms->smp.threads, &topo_ids);
+ &topo_info, &topo_ids);
ms->possible_cpus->cpus[i].props.has_socket_id = true;
ms->possible_cpus->cpus[i].props.socket_id = topo_ids.pkg_id;
if (x86ms->smp_dies > 1) {
diff --git a/include/hw/i386/topology.h b/include/hw/i386/topology.h
index 52def68610..7ea507f376 100644
--- a/include/hw/i386/topology.h
+++ b/include/hw/i386/topology.h
@@ -52,6 +52,12 @@ typedef struct X86CPUTopoIDs {
unsigned smt_id;
} X86CPUTopoIDs;
+typedef struct X86CPUTopoInfo {
+ unsigned dies_per_pkg;
+ unsigned cores_per_die;
+ unsigned threads_per_core;
+} X86CPUTopoInfo;
+
/* Return the bit width needed for 'count' IDs
*/
static unsigned apicid_bitwidth_for_count(unsigned count)
@@ -119,11 +125,13 @@ static inline unsigned apicid_pkg_offset(unsigned nr_dies,
*
* The caller must make sure core_id < nr_cores and smt_id < nr_threads.
*/
-static inline apic_id_t apicid_from_topo_ids(unsigned nr_dies,
- unsigned nr_cores,
- unsigned nr_threads,
+static inline apic_id_t apicid_from_topo_ids(X86CPUTopoInfo *topo_info,
const X86CPUTopoIDs *topo_ids)
{
+ unsigned nr_dies = topo_info->dies_per_pkg;
+ unsigned nr_cores = topo_info->cores_per_die;
+ unsigned nr_threads = topo_info->threads_per_core;
+
return (topo_ids->pkg_id <<
apicid_pkg_offset(nr_dies, nr_cores, nr_threads)) |
(topo_ids->die_id <<
@@ -136,12 +144,14 @@ static inline apic_id_t apicid_from_topo_ids(unsigned nr_dies,
/* Calculate thread/core/package IDs for a specific topology,
* based on (contiguous) CPU index
*/
-static inline void x86_topo_ids_from_idx(unsigned nr_dies,
- unsigned nr_cores,
- unsigned nr_threads,
+static inline void x86_topo_ids_from_idx(X86CPUTopoInfo *topo_info,
unsigned cpu_index,
X86CPUTopoIDs *topo_ids)
{
+ unsigned nr_dies = topo_info->dies_per_pkg;
+ unsigned nr_cores = topo_info->cores_per_die;
+ unsigned nr_threads = topo_info->threads_per_core;
+
topo_ids->pkg_id = cpu_index / (nr_dies * nr_cores * nr_threads);
topo_ids->die_id = cpu_index / (nr_cores * nr_threads) % nr_dies;
topo_ids->core_id = cpu_index / nr_threads % nr_cores;
@@ -152,11 +162,13 @@ static inline void x86_topo_ids_from_idx(unsigned nr_dies,
* based on APIC ID
*/
static inline void x86_topo_ids_from_apicid(apic_id_t apicid,
- unsigned nr_dies,
- unsigned nr_cores,
- unsigned nr_threads,
+ X86CPUTopoInfo *topo_info,
X86CPUTopoIDs *topo_ids)
{
+ unsigned nr_dies = topo_info->dies_per_pkg;
+ unsigned nr_cores = topo_info->cores_per_die;
+ unsigned nr_threads = topo_info->threads_per_core;
+
topo_ids->smt_id = apicid &
~(0xFFFFFFFFUL << apicid_smt_width(nr_dies, nr_cores, nr_threads));
topo_ids->core_id =
@@ -173,14 +185,12 @@ static inline void x86_topo_ids_from_apicid(apic_id_t apicid,
*
* 'cpu_index' is a sequential, contiguous ID for the CPU.
*/
-static inline apic_id_t x86_apicid_from_cpu_idx(unsigned nr_dies,
- unsigned nr_cores,
- unsigned nr_threads,
+static inline apic_id_t x86_apicid_from_cpu_idx(X86CPUTopoInfo *topo_info,
unsigned cpu_index)
{
X86CPUTopoIDs topo_ids;
- x86_topo_ids_from_idx(nr_dies, nr_cores, nr_threads, cpu_index, &topo_ids);
- return apicid_from_topo_ids(nr_dies, nr_cores, nr_threads, &topo_ids);
+ x86_topo_ids_from_idx(topo_info, cpu_index, &topo_ids);
+ return apicid_from_topo_ids(topo_info, &topo_ids);
}
#endif /* HW_I386_TOPOLOGY_H */
diff --git a/include/hw/i386/x86.h b/include/hw/i386/x86.h
index 41fe37b8a3..22babcb3bb 100644
--- a/include/hw/i386/x86.h
+++ b/include/hw/i386/x86.h
@@ -21,6 +21,7 @@
#include "exec/hwaddr.h"
#include "qemu/notify.h"
+#include "hw/i386/topology.h"
#include "hw/boards.h"
#include "hw/nmi.h"
#include "hw/isa/isa.h"
@@ -82,6 +83,8 @@ typedef struct {
#define X86_MACHINE_CLASS(class) \
OBJECT_CLASS_CHECK(X86MachineClass, class, TYPE_X86_MACHINE)
+void init_topo_info(X86CPUTopoInfo *topo_info, const X86MachineState *x86ms);
+
uint32_t x86_cpu_apic_id_from_index(X86MachineState *pcms,
unsigned int cpu_index);
diff --git a/tests/test-x86-cpuid.c b/tests/test-x86-cpuid.c
index 1942287f33..66b953113b 100644
--- a/tests/test-x86-cpuid.c
+++ b/tests/test-x86-cpuid.c
@@ -28,15 +28,18 @@
static void test_topo_bits(void)
{
+ X86CPUTopoInfo topo_info = {0};
+
/* simple tests for 1 thread per core, 1 core per die, 1 die per package */
g_assert_cmpuint(apicid_smt_width(1, 1, 1), ==, 0);
g_assert_cmpuint(apicid_core_width(1, 1, 1), ==, 0);
g_assert_cmpuint(apicid_die_width(1, 1, 1), ==, 0);
- g_assert_cmpuint(x86_apicid_from_cpu_idx(1, 1, 1, 0), ==, 0);
- g_assert_cmpuint(x86_apicid_from_cpu_idx(1, 1, 1, 1), ==, 1);
- g_assert_cmpuint(x86_apicid_from_cpu_idx(1, 1, 1, 2), ==, 2);
- g_assert_cmpuint(x86_apicid_from_cpu_idx(1, 1, 1, 3), ==, 3);
+ topo_info = (X86CPUTopoInfo) {1, 1, 1};
+ g_assert_cmpuint(x86_apicid_from_cpu_idx(&topo_info, 0), ==, 0);
+ g_assert_cmpuint(x86_apicid_from_cpu_idx(&topo_info, 1), ==, 1);
+ g_assert_cmpuint(x86_apicid_from_cpu_idx(&topo_info, 2), ==, 2);
+ g_assert_cmpuint(x86_apicid_from_cpu_idx(&topo_info, 3), ==, 3);
/* Test field width calculation for multiple values
@@ -71,36 +74,38 @@ static void test_topo_bits(void)
g_assert_cmpuint(apicid_die_offset(1, 6, 3), ==, 5);
g_assert_cmpuint(apicid_pkg_offset(1, 6, 3), ==, 5);
- g_assert_cmpuint(x86_apicid_from_cpu_idx(1, 6, 3, 0), ==, 0);
- g_assert_cmpuint(x86_apicid_from_cpu_idx(1, 6, 3, 1), ==, 1);
- g_assert_cmpuint(x86_apicid_from_cpu_idx(1, 6, 3, 2), ==, 2);
+ topo_info = (X86CPUTopoInfo) {1, 6, 3};
+ g_assert_cmpuint(x86_apicid_from_cpu_idx(&topo_info, 0), ==, 0);
+ g_assert_cmpuint(x86_apicid_from_cpu_idx(&topo_info, 1), ==, 1);
+ g_assert_cmpuint(x86_apicid_from_cpu_idx(&topo_info, 2), ==, 2);
- g_assert_cmpuint(x86_apicid_from_cpu_idx(1, 6, 3, 1 * 3 + 0), ==,
+ topo_info = (X86CPUTopoInfo) {1, 6, 3};
+ g_assert_cmpuint(x86_apicid_from_cpu_idx(&topo_info, 1 * 3 + 0), ==,
(1 << 2) | 0);
- g_assert_cmpuint(x86_apicid_from_cpu_idx(1, 6, 3, 1 * 3 + 1), ==,
+ g_assert_cmpuint(x86_apicid_from_cpu_idx(&topo_info, 1 * 3 + 1), ==,
(1 << 2) | 1);
- g_assert_cmpuint(x86_apicid_from_cpu_idx(1, 6, 3, 1 * 3 + 2), ==,
+ g_assert_cmpuint(x86_apicid_from_cpu_idx(&topo_info, 1 * 3 + 2), ==,
(1 << 2) | 2);
- g_assert_cmpuint(x86_apicid_from_cpu_idx(1, 6, 3, 2 * 3 + 0), ==,
+ g_assert_cmpuint(x86_apicid_from_cpu_idx(&topo_info, 2 * 3 + 0), ==,
(2 << 2) | 0);
- g_assert_cmpuint(x86_apicid_from_cpu_idx(1, 6, 3, 2 * 3 + 1), ==,
+ g_assert_cmpuint(x86_apicid_from_cpu_idx(&topo_info, 2 * 3 + 1), ==,
(2 << 2) | 1);
- g_assert_cmpuint(x86_apicid_from_cpu_idx(1, 6, 3, 2 * 3 + 2), ==,
+ g_assert_cmpuint(x86_apicid_from_cpu_idx(&topo_info, 2 * 3 + 2), ==,
(2 << 2) | 2);
- g_assert_cmpuint(x86_apicid_from_cpu_idx(1, 6, 3, 5 * 3 + 0), ==,
+ g_assert_cmpuint(x86_apicid_from_cpu_idx(&topo_info, 5 * 3 + 0), ==,
(5 << 2) | 0);
- g_assert_cmpuint(x86_apicid_from_cpu_idx(1, 6, 3, 5 * 3 + 1), ==,
+ g_assert_cmpuint(x86_apicid_from_cpu_idx(&topo_info, 5 * 3 + 1), ==,
(5 << 2) | 1);
- g_assert_cmpuint(x86_apicid_from_cpu_idx(1, 6, 3, 5 * 3 + 2), ==,
+ g_assert_cmpuint(x86_apicid_from_cpu_idx(&topo_info, 5 * 3 + 2), ==,
(5 << 2) | 2);
- g_assert_cmpuint(x86_apicid_from_cpu_idx(1, 6, 3,
+ g_assert_cmpuint(x86_apicid_from_cpu_idx(&topo_info,
1 * 6 * 3 + 0 * 3 + 0), ==, (1 << 5));
- g_assert_cmpuint(x86_apicid_from_cpu_idx(1, 6, 3,
+ g_assert_cmpuint(x86_apicid_from_cpu_idx(&topo_info,
1 * 6 * 3 + 1 * 3 + 1), ==, (1 << 5) | (1 << 2) | 1);
- g_assert_cmpuint(x86_apicid_from_cpu_idx(1, 6, 3,
+ g_assert_cmpuint(x86_apicid_from_cpu_idx(&topo_info,
3 * 6 * 3 + 5 * 3 + 2), ==, (3 << 5) | (5 << 2) | 2);
}
^ permalink raw reply related [flat|nested] 38+ messages in thread
* [PATCH v7 02/13] hw/i386: Consolidate topology functions
2020-03-11 22:52 [PATCH v7 00/13] APIC ID fixes for AMD EPYC CPU model Babu Moger
2020-03-11 22:52 ` [PATCH v7 01/13] hw/i386: Introduce X86CPUTopoInfo to contain topology info Babu Moger
@ 2020-03-11 22:52 ` Babu Moger
2020-03-11 22:53 ` [PATCH v7 03/13] machine: Add SMP Sockets in CpuTopology Babu Moger
` (11 subsequent siblings)
13 siblings, 0 replies; 38+ messages in thread
From: Babu Moger @ 2020-03-11 22:52 UTC (permalink / raw)
To: ehabkost, marcel.apfelbaum, pbonzini, rth, mst, imammedo; +Cc: qemu-devel
Now that we have all the parameters in X86CPUTopoInfo, we can just
pass the structure to calculate the offsets and width.
Signed-off-by: Babu Moger <babu.moger@amd.com>
Reviewed-by: Igor Mammedov <imammedo@redhat.com>
Acked-by: Michael S. Tsirkin <mst@redhat.com>
---
include/hw/i386/topology.h | 68 +++++++++++++------------------------------
target/i386/cpu.c | 23 +++++++--------
tests/test-x86-cpuid.c | 69 +++++++++++++++++++++++++++-----------------
3 files changed, 75 insertions(+), 85 deletions(-)
diff --git a/include/hw/i386/topology.h b/include/hw/i386/topology.h
index 7ea507f376..ba52d49079 100644
--- a/include/hw/i386/topology.h
+++ b/include/hw/i386/topology.h
@@ -69,56 +69,42 @@ static unsigned apicid_bitwidth_for_count(unsigned count)
/* Bit width of the SMT_ID (thread ID) field on the APIC ID
*/
-static inline unsigned apicid_smt_width(unsigned nr_dies,
- unsigned nr_cores,
- unsigned nr_threads)
+static inline unsigned apicid_smt_width(X86CPUTopoInfo *topo_info)
{
- return apicid_bitwidth_for_count(nr_threads);
+ return apicid_bitwidth_for_count(topo_info->threads_per_core);
}
/* Bit width of the Core_ID field
*/
-static inline unsigned apicid_core_width(unsigned nr_dies,
- unsigned nr_cores,
- unsigned nr_threads)
+static inline unsigned apicid_core_width(X86CPUTopoInfo *topo_info)
{
- return apicid_bitwidth_for_count(nr_cores);
+ return apicid_bitwidth_for_count(topo_info->cores_per_die);
}
/* Bit width of the Die_ID field */
-static inline unsigned apicid_die_width(unsigned nr_dies,
- unsigned nr_cores,
- unsigned nr_threads)
+static inline unsigned apicid_die_width(X86CPUTopoInfo *topo_info)
{
- return apicid_bitwidth_for_count(nr_dies);
+ return apicid_bitwidth_for_count(topo_info->dies_per_pkg);
}
/* Bit offset of the Core_ID field
*/
-static inline unsigned apicid_core_offset(unsigned nr_dies,
- unsigned nr_cores,
- unsigned nr_threads)
+static inline unsigned apicid_core_offset(X86CPUTopoInfo *topo_info)
{
- return apicid_smt_width(nr_dies, nr_cores, nr_threads);
+ return apicid_smt_width(topo_info);
}
/* Bit offset of the Die_ID field */
-static inline unsigned apicid_die_offset(unsigned nr_dies,
- unsigned nr_cores,
- unsigned nr_threads)
+static inline unsigned apicid_die_offset(X86CPUTopoInfo *topo_info)
{
- return apicid_core_offset(nr_dies, nr_cores, nr_threads) +
- apicid_core_width(nr_dies, nr_cores, nr_threads);
+ return apicid_core_offset(topo_info) + apicid_core_width(topo_info);
}
/* Bit offset of the Pkg_ID (socket ID) field
*/
-static inline unsigned apicid_pkg_offset(unsigned nr_dies,
- unsigned nr_cores,
- unsigned nr_threads)
+static inline unsigned apicid_pkg_offset(X86CPUTopoInfo *topo_info)
{
- return apicid_die_offset(nr_dies, nr_cores, nr_threads) +
- apicid_die_width(nr_dies, nr_cores, nr_threads);
+ return apicid_die_offset(topo_info) + apicid_die_width(topo_info);
}
/* Make APIC ID for the CPU based on Pkg_ID, Core_ID, SMT_ID
@@ -128,16 +114,9 @@ static inline unsigned apicid_pkg_offset(unsigned nr_dies,
static inline apic_id_t apicid_from_topo_ids(X86CPUTopoInfo *topo_info,
const X86CPUTopoIDs *topo_ids)
{
- unsigned nr_dies = topo_info->dies_per_pkg;
- unsigned nr_cores = topo_info->cores_per_die;
- unsigned nr_threads = topo_info->threads_per_core;
-
- return (topo_ids->pkg_id <<
- apicid_pkg_offset(nr_dies, nr_cores, nr_threads)) |
- (topo_ids->die_id <<
- apicid_die_offset(nr_dies, nr_cores, nr_threads)) |
- (topo_ids->core_id <<
- apicid_core_offset(nr_dies, nr_cores, nr_threads)) |
+ return (topo_ids->pkg_id << apicid_pkg_offset(topo_info)) |
+ (topo_ids->die_id << apicid_die_offset(topo_info)) |
+ (topo_ids->core_id << apicid_core_offset(topo_info)) |
topo_ids->smt_id;
}
@@ -165,20 +144,15 @@ static inline void x86_topo_ids_from_apicid(apic_id_t apicid,
X86CPUTopoInfo *topo_info,
X86CPUTopoIDs *topo_ids)
{
- unsigned nr_dies = topo_info->dies_per_pkg;
- unsigned nr_cores = topo_info->cores_per_die;
- unsigned nr_threads = topo_info->threads_per_core;
-
topo_ids->smt_id = apicid &
- ~(0xFFFFFFFFUL << apicid_smt_width(nr_dies, nr_cores, nr_threads));
+ ~(0xFFFFFFFFUL << apicid_smt_width(topo_info));
topo_ids->core_id =
- (apicid >> apicid_core_offset(nr_dies, nr_cores, nr_threads)) &
- ~(0xFFFFFFFFUL << apicid_core_width(nr_dies, nr_cores, nr_threads));
+ (apicid >> apicid_core_offset(topo_info)) &
+ ~(0xFFFFFFFFUL << apicid_core_width(topo_info));
topo_ids->die_id =
- (apicid >> apicid_die_offset(nr_dies, nr_cores, nr_threads)) &
- ~(0xFFFFFFFFUL << apicid_die_width(nr_dies, nr_cores, nr_threads));
- topo_ids->pkg_id =
- apicid >> apicid_pkg_offset(nr_dies, nr_cores, nr_threads);
+ (apicid >> apicid_die_offset(topo_info)) &
+ ~(0xFFFFFFFFUL << apicid_die_width(topo_info));
+ topo_ids->pkg_id = apicid >> apicid_pkg_offset(topo_info);
}
/* Make APIC ID for the CPU 'cpu_index'
diff --git a/target/i386/cpu.c b/target/i386/cpu.c
index 350b51b346..6df3127fd7 100644
--- a/target/i386/cpu.c
+++ b/target/i386/cpu.c
@@ -5497,6 +5497,11 @@ void cpu_x86_cpuid(CPUX86State *env, uint32_t index, uint32_t count,
uint32_t die_offset;
uint32_t limit;
uint32_t signature[3];
+ X86CPUTopoInfo topo_info;
+
+ topo_info.dies_per_pkg = env->nr_dies;
+ topo_info.cores_per_die = cs->nr_cores;
+ topo_info.threads_per_core = cs->nr_threads;
/* Calculate & apply limits for different index ranges */
if (index >= 0xC0000000) {
@@ -5583,8 +5588,7 @@ void cpu_x86_cpuid(CPUX86State *env, uint32_t index, uint32_t count,
eax, ebx, ecx, edx);
break;
case 3: /* L3 cache info */
- die_offset = apicid_die_offset(env->nr_dies,
- cs->nr_cores, cs->nr_threads);
+ die_offset = apicid_die_offset(&topo_info);
if (cpu->enable_l3_cache) {
encode_cache_cpuid4(env->cache_info_cpuid4.l3_cache,
(1 << die_offset), cs->nr_cores,
@@ -5675,14 +5679,12 @@ void cpu_x86_cpuid(CPUX86State *env, uint32_t index, uint32_t count,
switch (count) {
case 0:
- *eax = apicid_core_offset(env->nr_dies,
- cs->nr_cores, cs->nr_threads);
+ *eax = apicid_core_offset(&topo_info);
*ebx = cs->nr_threads;
*ecx |= CPUID_TOPOLOGY_LEVEL_SMT;
break;
case 1:
- *eax = apicid_pkg_offset(env->nr_dies,
- cs->nr_cores, cs->nr_threads);
+ *eax = apicid_pkg_offset(&topo_info);
*ebx = cs->nr_cores * cs->nr_threads;
*ecx |= CPUID_TOPOLOGY_LEVEL_CORE;
break;
@@ -5706,20 +5708,17 @@ void cpu_x86_cpuid(CPUX86State *env, uint32_t index, uint32_t count,
*edx = cpu->apic_id;
switch (count) {
case 0:
- *eax = apicid_core_offset(env->nr_dies, cs->nr_cores,
- cs->nr_threads);
+ *eax = apicid_core_offset(&topo_info);
*ebx = cs->nr_threads;
*ecx |= CPUID_TOPOLOGY_LEVEL_SMT;
break;
case 1:
- *eax = apicid_die_offset(env->nr_dies, cs->nr_cores,
- cs->nr_threads);
+ *eax = apicid_die_offset(&topo_info);
*ebx = cs->nr_cores * cs->nr_threads;
*ecx |= CPUID_TOPOLOGY_LEVEL_CORE;
break;
case 2:
- *eax = apicid_pkg_offset(env->nr_dies, cs->nr_cores,
- cs->nr_threads);
+ *eax = apicid_pkg_offset(&topo_info);
*ebx = env->nr_dies * cs->nr_cores * cs->nr_threads;
*ecx |= CPUID_TOPOLOGY_LEVEL_DIE;
break;
diff --git a/tests/test-x86-cpuid.c b/tests/test-x86-cpuid.c
index 66b953113b..bfabc0403a 100644
--- a/tests/test-x86-cpuid.c
+++ b/tests/test-x86-cpuid.c
@@ -31,9 +31,10 @@ static void test_topo_bits(void)
X86CPUTopoInfo topo_info = {0};
/* simple tests for 1 thread per core, 1 core per die, 1 die per package */
- g_assert_cmpuint(apicid_smt_width(1, 1, 1), ==, 0);
- g_assert_cmpuint(apicid_core_width(1, 1, 1), ==, 0);
- g_assert_cmpuint(apicid_die_width(1, 1, 1), ==, 0);
+ topo_info = (X86CPUTopoInfo) {1, 1, 1};
+ g_assert_cmpuint(apicid_smt_width(&topo_info), ==, 0);
+ g_assert_cmpuint(apicid_core_width(&topo_info), ==, 0);
+ g_assert_cmpuint(apicid_die_width(&topo_info), ==, 0);
topo_info = (X86CPUTopoInfo) {1, 1, 1};
g_assert_cmpuint(x86_apicid_from_cpu_idx(&topo_info, 0), ==, 0);
@@ -44,35 +45,51 @@ static void test_topo_bits(void)
/* Test field width calculation for multiple values
*/
- g_assert_cmpuint(apicid_smt_width(1, 1, 2), ==, 1);
- g_assert_cmpuint(apicid_smt_width(1, 1, 3), ==, 2);
- g_assert_cmpuint(apicid_smt_width(1, 1, 4), ==, 2);
-
- g_assert_cmpuint(apicid_smt_width(1, 1, 14), ==, 4);
- g_assert_cmpuint(apicid_smt_width(1, 1, 15), ==, 4);
- g_assert_cmpuint(apicid_smt_width(1, 1, 16), ==, 4);
- g_assert_cmpuint(apicid_smt_width(1, 1, 17), ==, 5);
-
-
- g_assert_cmpuint(apicid_core_width(1, 30, 2), ==, 5);
- g_assert_cmpuint(apicid_core_width(1, 31, 2), ==, 5);
- g_assert_cmpuint(apicid_core_width(1, 32, 2), ==, 5);
- g_assert_cmpuint(apicid_core_width(1, 33, 2), ==, 6);
-
- g_assert_cmpuint(apicid_die_width(1, 30, 2), ==, 0);
- g_assert_cmpuint(apicid_die_width(2, 30, 2), ==, 1);
- g_assert_cmpuint(apicid_die_width(3, 30, 2), ==, 2);
- g_assert_cmpuint(apicid_die_width(4, 30, 2), ==, 2);
+ topo_info = (X86CPUTopoInfo) {1, 1, 2};
+ g_assert_cmpuint(apicid_smt_width(&topo_info), ==, 1);
+ topo_info = (X86CPUTopoInfo) {1, 1, 3};
+ g_assert_cmpuint(apicid_smt_width(&topo_info), ==, 2);
+ topo_info = (X86CPUTopoInfo) {1, 1, 4};
+ g_assert_cmpuint(apicid_smt_width(&topo_info), ==, 2);
+
+ topo_info = (X86CPUTopoInfo) {1, 1, 14};
+ g_assert_cmpuint(apicid_smt_width(&topo_info), ==, 4);
+ topo_info = (X86CPUTopoInfo) {1, 1, 15};
+ g_assert_cmpuint(apicid_smt_width(&topo_info), ==, 4);
+ topo_info = (X86CPUTopoInfo) {1, 1, 16};
+ g_assert_cmpuint(apicid_smt_width(&topo_info), ==, 4);
+ topo_info = (X86CPUTopoInfo) {1, 1, 17};
+ g_assert_cmpuint(apicid_smt_width(&topo_info), ==, 5);
+
+
+ topo_info = (X86CPUTopoInfo) {1, 30, 2};
+ g_assert_cmpuint(apicid_core_width(&topo_info), ==, 5);
+ topo_info = (X86CPUTopoInfo) {1, 31, 2};
+ g_assert_cmpuint(apicid_core_width(&topo_info), ==, 5);
+ topo_info = (X86CPUTopoInfo) {1, 32, 2};
+ g_assert_cmpuint(apicid_core_width(&topo_info), ==, 5);
+ topo_info = (X86CPUTopoInfo) {1, 33, 2};
+ g_assert_cmpuint(apicid_core_width(&topo_info), ==, 6);
+
+ topo_info = (X86CPUTopoInfo) {1, 30, 2};
+ g_assert_cmpuint(apicid_die_width(&topo_info), ==, 0);
+ topo_info = (X86CPUTopoInfo) {2, 30, 2};
+ g_assert_cmpuint(apicid_die_width(&topo_info), ==, 1);
+ topo_info = (X86CPUTopoInfo) {3, 30, 2};
+ g_assert_cmpuint(apicid_die_width(&topo_info), ==, 2);
+ topo_info = (X86CPUTopoInfo) {4, 30, 2};
+ g_assert_cmpuint(apicid_die_width(&topo_info), ==, 2);
/* build a weird topology and see if IDs are calculated correctly
*/
/* This will use 2 bits for thread ID and 3 bits for core ID
*/
- g_assert_cmpuint(apicid_smt_width(1, 6, 3), ==, 2);
- g_assert_cmpuint(apicid_core_offset(1, 6, 3), ==, 2);
- g_assert_cmpuint(apicid_die_offset(1, 6, 3), ==, 5);
- g_assert_cmpuint(apicid_pkg_offset(1, 6, 3), ==, 5);
+ topo_info = (X86CPUTopoInfo) {1, 6, 3};
+ g_assert_cmpuint(apicid_smt_width(&topo_info), ==, 2);
+ g_assert_cmpuint(apicid_core_offset(&topo_info), ==, 2);
+ g_assert_cmpuint(apicid_die_offset(&topo_info), ==, 5);
+ g_assert_cmpuint(apicid_pkg_offset(&topo_info), ==, 5);
topo_info = (X86CPUTopoInfo) {1, 6, 3};
g_assert_cmpuint(x86_apicid_from_cpu_idx(&topo_info, 0), ==, 0);
^ permalink raw reply related [flat|nested] 38+ messages in thread
* [PATCH v7 03/13] machine: Add SMP Sockets in CpuTopology
2020-03-11 22:52 [PATCH v7 00/13] APIC ID fixes for AMD EPYC CPU model Babu Moger
2020-03-11 22:52 ` [PATCH v7 01/13] hw/i386: Introduce X86CPUTopoInfo to contain topology info Babu Moger
2020-03-11 22:52 ` [PATCH v7 02/13] hw/i386: Consolidate topology functions Babu Moger
@ 2020-03-11 22:53 ` Babu Moger
2020-03-11 22:53 ` [PATCH v7 04/13] hw/i386: Remove unnecessary initialization in x86_cpu_new Babu Moger
` (10 subsequent siblings)
13 siblings, 0 replies; 38+ messages in thread
From: Babu Moger @ 2020-03-11 22:53 UTC (permalink / raw)
To: ehabkost, marcel.apfelbaum, pbonzini, rth, mst, imammedo; +Cc: qemu-devel
Store the smp sockets in CpuTopology. The socket information required to
build the apic id in EPYC mode. Right now socket information is not passed
to down when decoding the apic id. Add the socket information here.
Signed-off-by: Babu Moger <babu.moger@amd.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: Igor Mammedov <imammedo@redhat.com>
Acked-by: Michael S. Tsirkin <mst@redhat.com>
---
hw/core/machine.c | 1 +
hw/i386/pc.c | 1 +
include/hw/boards.h | 2 ++
softmmu/vl.c | 1 +
4 files changed, 5 insertions(+)
diff --git a/hw/core/machine.c b/hw/core/machine.c
index 4778bc6b08..b958cd1b99 100644
--- a/hw/core/machine.c
+++ b/hw/core/machine.c
@@ -757,6 +757,7 @@ static void smp_parse(MachineState *ms, QemuOpts *opts)
ms->smp.cpus = cpus;
ms->smp.cores = cores;
ms->smp.threads = threads;
+ ms->smp.sockets = sockets;
}
if (ms->smp.cpus > 1) {
diff --git a/hw/i386/pc.c b/hw/i386/pc.c
index 662abb549d..05e7f1090f 100644
--- a/hw/i386/pc.c
+++ b/hw/i386/pc.c
@@ -781,6 +781,7 @@ void pc_smp_parse(MachineState *ms, QemuOpts *opts)
ms->smp.cpus = cpus;
ms->smp.cores = cores;
ms->smp.threads = threads;
+ ms->smp.sockets = sockets;
x86ms->smp_dies = dies;
}
diff --git a/include/hw/boards.h b/include/hw/boards.h
index 9bc42dfb22..d01056286a 100644
--- a/include/hw/boards.h
+++ b/include/hw/boards.h
@@ -236,12 +236,14 @@ typedef struct DeviceMemoryState {
* @cpus: the number of present logical processors on the machine
* @cores: the number of cores in one package
* @threads: the number of threads in one core
+ * @sockets: the number of sockets on the machine
* @max_cpus: the maximum number of logical processors on the machine
*/
typedef struct CpuTopology {
unsigned int cpus;
unsigned int cores;
unsigned int threads;
+ unsigned int sockets;
unsigned int max_cpus;
} CpuTopology;
diff --git a/softmmu/vl.c b/softmmu/vl.c
index ff2685dff8..dadb798ac7 100644
--- a/softmmu/vl.c
+++ b/softmmu/vl.c
@@ -3935,6 +3935,7 @@ void qemu_init(int argc, char **argv, char **envp)
current_machine->smp.max_cpus = machine_class->default_cpus;
current_machine->smp.cores = 1;
current_machine->smp.threads = 1;
+ current_machine->smp.sockets = 1;
machine_class->smp_parse(current_machine,
qemu_opts_find(qemu_find_opts("smp-opts"), NULL));
^ permalink raw reply related [flat|nested] 38+ messages in thread
* [PATCH v7 04/13] hw/i386: Remove unnecessary initialization in x86_cpu_new
2020-03-11 22:52 [PATCH v7 00/13] APIC ID fixes for AMD EPYC CPU model Babu Moger
` (2 preceding siblings ...)
2020-03-11 22:53 ` [PATCH v7 03/13] machine: Add SMP Sockets in CpuTopology Babu Moger
@ 2020-03-11 22:53 ` Babu Moger
2020-03-11 22:53 ` [PATCH v7 05/13] hw/i386: Update structures to save the number of nodes per package Babu Moger
` (9 subsequent siblings)
13 siblings, 0 replies; 38+ messages in thread
From: Babu Moger @ 2020-03-11 22:53 UTC (permalink / raw)
To: ehabkost, marcel.apfelbaum, pbonzini, rth, mst, imammedo; +Cc: qemu-devel
The function pc_cpu_pre_plug takes care of initialization of CPUX86State.
So, remove the initialization here.
Suggested-by: Igor Mammedov <imammedo@redhat.com>
Signed-off-by: Babu Moger <babu.moger@amd.com>
Reviewed-by: Igor Mammedov <imammedo@redhat.com>
Acked-by: Michael S. Tsirkin <mst@redhat.com>
---
hw/i386/x86.c | 4 ----
1 file changed, 4 deletions(-)
diff --git a/hw/i386/x86.c b/hw/i386/x86.c
index dbbff46a4b..84d1944a34 100644
--- a/hw/i386/x86.c
+++ b/hw/i386/x86.c
@@ -103,13 +103,9 @@ void x86_cpu_new(X86MachineState *x86ms, int64_t apic_id, Error **errp)
{
Object *cpu = NULL;
Error *local_err = NULL;
- CPUX86State *env = NULL;
cpu = object_new(MACHINE(x86ms)->cpu_type);
- env = &X86_CPU(cpu)->env;
- env->nr_dies = x86ms->smp_dies;
-
object_property_set_uint(cpu, apic_id, "apic-id", &local_err);
object_property_set_bool(cpu, true, "realized", &local_err);
^ permalink raw reply related [flat|nested] 38+ messages in thread
* [PATCH v7 05/13] hw/i386: Update structures to save the number of nodes per package
2020-03-11 22:52 [PATCH v7 00/13] APIC ID fixes for AMD EPYC CPU model Babu Moger
` (3 preceding siblings ...)
2020-03-11 22:53 ` [PATCH v7 04/13] hw/i386: Remove unnecessary initialization in x86_cpu_new Babu Moger
@ 2020-03-11 22:53 ` Babu Moger
2020-03-11 22:53 ` [PATCH v7 06/13] hw/i386: Rename apicid_from_topo_ids to x86_apicid_from_topo_ids Babu Moger
` (8 subsequent siblings)
13 siblings, 0 replies; 38+ messages in thread
From: Babu Moger @ 2020-03-11 22:53 UTC (permalink / raw)
To: ehabkost, marcel.apfelbaum, pbonzini, rth, mst, imammedo; +Cc: qemu-devel
Update structures X86CPUTopoIDs and CPUX86State to hold the number of
nodes per package. This is required to build EPYC mode topology.
Signed-off-by: Babu Moger <babu.moger@amd.com>
Reviewed-by: Igor Mammedov <imammedo@redhat.com>
Acked-by: Michael S. Tsirkin <mst@redhat.com>
---
hw/i386/pc.c | 1 +
hw/i386/x86.c | 1 +
include/hw/i386/topology.h | 1 +
target/i386/cpu.c | 1 +
target/i386/cpu.h | 1 +
tests/test-x86-cpuid.c | 40 ++++++++++++++++++++--------------------
6 files changed, 25 insertions(+), 20 deletions(-)
diff --git a/hw/i386/pc.c b/hw/i386/pc.c
index 05e7f1090f..ee89fcd1c3 100644
--- a/hw/i386/pc.c
+++ b/hw/i386/pc.c
@@ -1525,6 +1525,7 @@ static void pc_cpu_pre_plug(HotplugHandler *hotplug_dev,
init_topo_info(&topo_info, x86ms);
env->nr_dies = x86ms->smp_dies;
+ env->nr_nodes = topo_info.nodes_per_pkg;
/*
* If APIC ID is not set,
diff --git a/hw/i386/x86.c b/hw/i386/x86.c
index 84d1944a34..42834d2319 100644
--- a/hw/i386/x86.c
+++ b/hw/i386/x86.c
@@ -62,6 +62,7 @@ inline void init_topo_info(X86CPUTopoInfo *topo_info,
{
MachineState *ms = MACHINE(x86ms);
+ topo_info->nodes_per_pkg = ms->numa_state->num_nodes / ms->smp.sockets;
topo_info->dies_per_pkg = x86ms->smp_dies;
topo_info->cores_per_die = ms->smp.cores;
topo_info->threads_per_core = ms->smp.threads;
diff --git a/include/hw/i386/topology.h b/include/hw/i386/topology.h
index ba52d49079..04f01e2a09 100644
--- a/include/hw/i386/topology.h
+++ b/include/hw/i386/topology.h
@@ -53,6 +53,7 @@ typedef struct X86CPUTopoIDs {
} X86CPUTopoIDs;
typedef struct X86CPUTopoInfo {
+ unsigned nodes_per_pkg;
unsigned dies_per_pkg;
unsigned cores_per_die;
unsigned threads_per_core;
diff --git a/target/i386/cpu.c b/target/i386/cpu.c
index 6df3127fd7..2e5be37b21 100644
--- a/target/i386/cpu.c
+++ b/target/i386/cpu.c
@@ -6957,6 +6957,7 @@ static void x86_cpu_initfn(Object *obj)
FeatureWord w;
env->nr_dies = 1;
+ env->nr_nodes = 1;
cpu_set_cpustate_pointers(cpu);
object_property_add(obj, "family", "int",
diff --git a/target/i386/cpu.h b/target/i386/cpu.h
index 68b186d258..7e9e963d78 100644
--- a/target/i386/cpu.h
+++ b/target/i386/cpu.h
@@ -1609,6 +1609,7 @@ typedef struct CPUX86State {
TPRAccess tpr_access_type;
unsigned nr_dies;
+ unsigned nr_nodes;
} CPUX86State;
struct kvm_msrs;
diff --git a/tests/test-x86-cpuid.c b/tests/test-x86-cpuid.c
index bfabc0403a..049030a50e 100644
--- a/tests/test-x86-cpuid.c
+++ b/tests/test-x86-cpuid.c
@@ -31,12 +31,12 @@ static void test_topo_bits(void)
X86CPUTopoInfo topo_info = {0};
/* simple tests for 1 thread per core, 1 core per die, 1 die per package */
- topo_info = (X86CPUTopoInfo) {1, 1, 1};
+ topo_info = (X86CPUTopoInfo) {0, 1, 1, 1};
g_assert_cmpuint(apicid_smt_width(&topo_info), ==, 0);
g_assert_cmpuint(apicid_core_width(&topo_info), ==, 0);
g_assert_cmpuint(apicid_die_width(&topo_info), ==, 0);
- topo_info = (X86CPUTopoInfo) {1, 1, 1};
+ topo_info = (X86CPUTopoInfo) {0, 1, 1, 1};
g_assert_cmpuint(x86_apicid_from_cpu_idx(&topo_info, 0), ==, 0);
g_assert_cmpuint(x86_apicid_from_cpu_idx(&topo_info, 1), ==, 1);
g_assert_cmpuint(x86_apicid_from_cpu_idx(&topo_info, 2), ==, 2);
@@ -45,39 +45,39 @@ static void test_topo_bits(void)
/* Test field width calculation for multiple values
*/
- topo_info = (X86CPUTopoInfo) {1, 1, 2};
+ topo_info = (X86CPUTopoInfo) {0, 1, 1, 2};
g_assert_cmpuint(apicid_smt_width(&topo_info), ==, 1);
- topo_info = (X86CPUTopoInfo) {1, 1, 3};
+ topo_info = (X86CPUTopoInfo) {0, 1, 1, 3};
g_assert_cmpuint(apicid_smt_width(&topo_info), ==, 2);
- topo_info = (X86CPUTopoInfo) {1, 1, 4};
+ topo_info = (X86CPUTopoInfo) {0, 1, 1, 4};
g_assert_cmpuint(apicid_smt_width(&topo_info), ==, 2);
- topo_info = (X86CPUTopoInfo) {1, 1, 14};
+ topo_info = (X86CPUTopoInfo) {0, 1, 1, 14};
g_assert_cmpuint(apicid_smt_width(&topo_info), ==, 4);
- topo_info = (X86CPUTopoInfo) {1, 1, 15};
+ topo_info = (X86CPUTopoInfo) {0, 1, 1, 15};
g_assert_cmpuint(apicid_smt_width(&topo_info), ==, 4);
- topo_info = (X86CPUTopoInfo) {1, 1, 16};
+ topo_info = (X86CPUTopoInfo) {0, 1, 1, 16};
g_assert_cmpuint(apicid_smt_width(&topo_info), ==, 4);
- topo_info = (X86CPUTopoInfo) {1, 1, 17};
+ topo_info = (X86CPUTopoInfo) {0, 1, 1, 17};
g_assert_cmpuint(apicid_smt_width(&topo_info), ==, 5);
- topo_info = (X86CPUTopoInfo) {1, 30, 2};
+ topo_info = (X86CPUTopoInfo) {0, 1, 30, 2};
g_assert_cmpuint(apicid_core_width(&topo_info), ==, 5);
- topo_info = (X86CPUTopoInfo) {1, 31, 2};
+ topo_info = (X86CPUTopoInfo) {0, 1, 31, 2};
g_assert_cmpuint(apicid_core_width(&topo_info), ==, 5);
- topo_info = (X86CPUTopoInfo) {1, 32, 2};
+ topo_info = (X86CPUTopoInfo) {0, 1, 32, 2};
g_assert_cmpuint(apicid_core_width(&topo_info), ==, 5);
- topo_info = (X86CPUTopoInfo) {1, 33, 2};
+ topo_info = (X86CPUTopoInfo) {0, 1, 33, 2};
g_assert_cmpuint(apicid_core_width(&topo_info), ==, 6);
- topo_info = (X86CPUTopoInfo) {1, 30, 2};
+ topo_info = (X86CPUTopoInfo) {0, 1, 30, 2};
g_assert_cmpuint(apicid_die_width(&topo_info), ==, 0);
- topo_info = (X86CPUTopoInfo) {2, 30, 2};
+ topo_info = (X86CPUTopoInfo) {0, 2, 30, 2};
g_assert_cmpuint(apicid_die_width(&topo_info), ==, 1);
- topo_info = (X86CPUTopoInfo) {3, 30, 2};
+ topo_info = (X86CPUTopoInfo) {0, 3, 30, 2};
g_assert_cmpuint(apicid_die_width(&topo_info), ==, 2);
- topo_info = (X86CPUTopoInfo) {4, 30, 2};
+ topo_info = (X86CPUTopoInfo) {0, 4, 30, 2};
g_assert_cmpuint(apicid_die_width(&topo_info), ==, 2);
/* build a weird topology and see if IDs are calculated correctly
@@ -85,18 +85,18 @@ static void test_topo_bits(void)
/* This will use 2 bits for thread ID and 3 bits for core ID
*/
- topo_info = (X86CPUTopoInfo) {1, 6, 3};
+ topo_info = (X86CPUTopoInfo) {0, 1, 6, 3};
g_assert_cmpuint(apicid_smt_width(&topo_info), ==, 2);
g_assert_cmpuint(apicid_core_offset(&topo_info), ==, 2);
g_assert_cmpuint(apicid_die_offset(&topo_info), ==, 5);
g_assert_cmpuint(apicid_pkg_offset(&topo_info), ==, 5);
- topo_info = (X86CPUTopoInfo) {1, 6, 3};
+ topo_info = (X86CPUTopoInfo) {0, 1, 6, 3};
g_assert_cmpuint(x86_apicid_from_cpu_idx(&topo_info, 0), ==, 0);
g_assert_cmpuint(x86_apicid_from_cpu_idx(&topo_info, 1), ==, 1);
g_assert_cmpuint(x86_apicid_from_cpu_idx(&topo_info, 2), ==, 2);
- topo_info = (X86CPUTopoInfo) {1, 6, 3};
+ topo_info = (X86CPUTopoInfo) {0, 1, 6, 3};
g_assert_cmpuint(x86_apicid_from_cpu_idx(&topo_info, 1 * 3 + 0), ==,
(1 << 2) | 0);
g_assert_cmpuint(x86_apicid_from_cpu_idx(&topo_info, 1 * 3 + 1), ==,
^ permalink raw reply related [flat|nested] 38+ messages in thread
* [PATCH v7 06/13] hw/i386: Rename apicid_from_topo_ids to x86_apicid_from_topo_ids
2020-03-11 22:52 [PATCH v7 00/13] APIC ID fixes for AMD EPYC CPU model Babu Moger
` (4 preceding siblings ...)
2020-03-11 22:53 ` [PATCH v7 05/13] hw/i386: Update structures to save the number of nodes per package Babu Moger
@ 2020-03-11 22:53 ` Babu Moger
2020-03-11 22:53 ` [PATCH v7 07/13] hw/386: Add EPYC mode topology decoding functions Babu Moger
` (7 subsequent siblings)
13 siblings, 0 replies; 38+ messages in thread
From: Babu Moger @ 2020-03-11 22:53 UTC (permalink / raw)
To: ehabkost, marcel.apfelbaum, pbonzini, rth, mst, imammedo; +Cc: qemu-devel
For consistency rename apicid_from_topo_ids to x86_apicid_from_topo_ids.
No functional change.
Signed-off-by: Babu Moger <babu.moger@amd.com>
Reviewed-by: Igor Mammedov <imammedo@redhat.com>
Acked-by: Michael S. Tsirkin <mst@redhat.com>
---
hw/i386/pc.c | 2 +-
include/hw/i386/topology.h | 6 +++---
2 files changed, 4 insertions(+), 4 deletions(-)
diff --git a/hw/i386/pc.c b/hw/i386/pc.c
index ee89fcd1c3..98ee763f68 100644
--- a/hw/i386/pc.c
+++ b/hw/i386/pc.c
@@ -1580,7 +1580,7 @@ static void pc_cpu_pre_plug(HotplugHandler *hotplug_dev,
topo_ids.die_id = cpu->die_id;
topo_ids.core_id = cpu->core_id;
topo_ids.smt_id = cpu->thread_id;
- cpu->apic_id = apicid_from_topo_ids(&topo_info, &topo_ids);
+ cpu->apic_id = x86_apicid_from_topo_ids(&topo_info, &topo_ids);
}
cpu_slot = pc_find_cpu_slot(MACHINE(pcms), cpu->apic_id, &idx);
diff --git a/include/hw/i386/topology.h b/include/hw/i386/topology.h
index 04f01e2a09..b9593b9905 100644
--- a/include/hw/i386/topology.h
+++ b/include/hw/i386/topology.h
@@ -112,8 +112,8 @@ static inline unsigned apicid_pkg_offset(X86CPUTopoInfo *topo_info)
*
* The caller must make sure core_id < nr_cores and smt_id < nr_threads.
*/
-static inline apic_id_t apicid_from_topo_ids(X86CPUTopoInfo *topo_info,
- const X86CPUTopoIDs *topo_ids)
+static inline apic_id_t x86_apicid_from_topo_ids(X86CPUTopoInfo *topo_info,
+ const X86CPUTopoIDs *topo_ids)
{
return (topo_ids->pkg_id << apicid_pkg_offset(topo_info)) |
(topo_ids->die_id << apicid_die_offset(topo_info)) |
@@ -165,7 +165,7 @@ static inline apic_id_t x86_apicid_from_cpu_idx(X86CPUTopoInfo *topo_info,
{
X86CPUTopoIDs topo_ids;
x86_topo_ids_from_idx(topo_info, cpu_index, &topo_ids);
- return apicid_from_topo_ids(topo_info, &topo_ids);
+ return x86_apicid_from_topo_ids(topo_info, &topo_ids);
}
#endif /* HW_I386_TOPOLOGY_H */
^ permalink raw reply related [flat|nested] 38+ messages in thread
* [PATCH v7 07/13] hw/386: Add EPYC mode topology decoding functions
2020-03-11 22:52 [PATCH v7 00/13] APIC ID fixes for AMD EPYC CPU model Babu Moger
` (5 preceding siblings ...)
2020-03-11 22:53 ` [PATCH v7 06/13] hw/i386: Rename apicid_from_topo_ids to x86_apicid_from_topo_ids Babu Moger
@ 2020-03-11 22:53 ` Babu Moger
2020-03-12 12:39 ` Igor Mammedov
2020-06-02 17:18 ` Eduardo Habkost
2020-03-11 22:53 ` [PATCH v7 08/13] target/i386: Cleanup and use the EPYC mode topology functions Babu Moger
` (6 subsequent siblings)
13 siblings, 2 replies; 38+ messages in thread
From: Babu Moger @ 2020-03-11 22:53 UTC (permalink / raw)
To: ehabkost, marcel.apfelbaum, pbonzini, rth, mst, imammedo; +Cc: qemu-devel
These functions add support for building EPYC mode topology given the smp
details like numa nodes, cores, threads and sockets.
The new apic id decoding is mostly similar to current apic id decoding
except that it adds a new field node_id when numa configured. Removes all
the hardcoded values. Subsequent patches will use these functions to build
the topology.
Following functions are added.
apicid_llc_width_epyc
apicid_llc_offset_epyc
apicid_pkg_offset_epyc
apicid_from_topo_ids_epyc
x86_topo_ids_from_idx_epyc
x86_topo_ids_from_apicid_epyc
x86_apicid_from_cpu_idx_epyc
The topology details are available in Processor Programming Reference (PPR)
for AMD Family 17h Model 01h, Revision B1 Processors. The revision guides are
available from the bugzilla Link below.
Link: https://bugzilla.kernel.org/show_bug.cgi?id=206537
Signed-off-by: Babu Moger <babu.moger@amd.com>
Acked-by: Igor Mammedov <imammedo@redhat.com>
Acked-by: Michael S. Tsirkin <mst@redhat.com>
---
include/hw/i386/topology.h | 100 ++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 100 insertions(+)
diff --git a/include/hw/i386/topology.h b/include/hw/i386/topology.h
index b9593b9905..07239f95f4 100644
--- a/include/hw/i386/topology.h
+++ b/include/hw/i386/topology.h
@@ -47,6 +47,7 @@ typedef uint32_t apic_id_t;
typedef struct X86CPUTopoIDs {
unsigned pkg_id;
+ unsigned node_id;
unsigned die_id;
unsigned core_id;
unsigned smt_id;
@@ -88,6 +89,11 @@ static inline unsigned apicid_die_width(X86CPUTopoInfo *topo_info)
return apicid_bitwidth_for_count(topo_info->dies_per_pkg);
}
+/* Bit width of the node_id field per socket */
+static inline unsigned apicid_node_width_epyc(X86CPUTopoInfo *topo_info)
+{
+ return apicid_bitwidth_for_count(MAX(topo_info->nodes_per_pkg, 1));
+}
/* Bit offset of the Core_ID field
*/
static inline unsigned apicid_core_offset(X86CPUTopoInfo *topo_info)
@@ -108,6 +114,100 @@ static inline unsigned apicid_pkg_offset(X86CPUTopoInfo *topo_info)
return apicid_die_offset(topo_info) + apicid_die_width(topo_info);
}
+#define NODE_ID_OFFSET 3 /* Minimum node_id offset if numa configured */
+
+/*
+ * Bit offset of the node_id field
+ *
+ * Make sure nodes_per_pkg > 0 if numa configured else zero.
+ */
+static inline unsigned apicid_node_offset_epyc(X86CPUTopoInfo *topo_info)
+{
+ unsigned offset = apicid_die_offset(topo_info) +
+ apicid_die_width(topo_info);
+
+ if (topo_info->nodes_per_pkg) {
+ return MAX(NODE_ID_OFFSET, offset);
+ } else {
+ return offset;
+ }
+}
+
+/* Bit offset of the Pkg_ID (socket ID) field */
+static inline unsigned apicid_pkg_offset_epyc(X86CPUTopoInfo *topo_info)
+{
+ return apicid_node_offset_epyc(topo_info) +
+ apicid_node_width_epyc(topo_info);
+}
+
+/*
+ * Make APIC ID for the CPU based on Pkg_ID, Core_ID, SMT_ID
+ *
+ * The caller must make sure core_id < nr_cores and smt_id < nr_threads.
+ */
+static inline apic_id_t
+x86_apicid_from_topo_ids_epyc(X86CPUTopoInfo *topo_info,
+ const X86CPUTopoIDs *topo_ids)
+{
+ return (topo_ids->pkg_id << apicid_pkg_offset_epyc(topo_info)) |
+ (topo_ids->node_id << apicid_node_offset_epyc(topo_info)) |
+ (topo_ids->die_id << apicid_die_offset(topo_info)) |
+ (topo_ids->core_id << apicid_core_offset(topo_info)) |
+ topo_ids->smt_id;
+}
+
+static inline void x86_topo_ids_from_idx_epyc(X86CPUTopoInfo *topo_info,
+ unsigned cpu_index,
+ X86CPUTopoIDs *topo_ids)
+{
+ unsigned nr_nodes = MAX(topo_info->nodes_per_pkg, 1);
+ unsigned nr_dies = topo_info->dies_per_pkg;
+ unsigned nr_cores = topo_info->cores_per_die;
+ unsigned nr_threads = topo_info->threads_per_core;
+ unsigned cores_per_node = DIV_ROUND_UP((nr_dies * nr_cores * nr_threads),
+ nr_nodes);
+
+ topo_ids->pkg_id = cpu_index / (nr_dies * nr_cores * nr_threads);
+ topo_ids->node_id = (cpu_index / cores_per_node) % nr_nodes;
+ topo_ids->die_id = cpu_index / (nr_cores * nr_threads) % nr_dies;
+ topo_ids->core_id = cpu_index / nr_threads % nr_cores;
+ topo_ids->smt_id = cpu_index % nr_threads;
+}
+
+/*
+ * Calculate thread/core/package IDs for a specific topology,
+ * based on APIC ID
+ */
+static inline void x86_topo_ids_from_apicid_epyc(apic_id_t apicid,
+ X86CPUTopoInfo *topo_info,
+ X86CPUTopoIDs *topo_ids)
+{
+ topo_ids->smt_id = apicid &
+ ~(0xFFFFFFFFUL << apicid_smt_width(topo_info));
+ topo_ids->core_id =
+ (apicid >> apicid_core_offset(topo_info)) &
+ ~(0xFFFFFFFFUL << apicid_core_width(topo_info));
+ topo_ids->die_id =
+ (apicid >> apicid_die_offset(topo_info)) &
+ ~(0xFFFFFFFFUL << apicid_die_width(topo_info));
+ topo_ids->node_id =
+ (apicid >> apicid_node_offset_epyc(topo_info)) &
+ ~(0xFFFFFFFFUL << apicid_node_width_epyc(topo_info));
+ topo_ids->pkg_id = apicid >> apicid_pkg_offset_epyc(topo_info);
+}
+
+/*
+ * Make APIC ID for the CPU 'cpu_index'
+ *
+ * 'cpu_index' is a sequential, contiguous ID for the CPU.
+ */
+static inline apic_id_t x86_apicid_from_cpu_idx_epyc(X86CPUTopoInfo *topo_info,
+ unsigned cpu_index)
+{
+ X86CPUTopoIDs topo_ids;
+ x86_topo_ids_from_idx_epyc(topo_info, cpu_index, &topo_ids);
+ return x86_apicid_from_topo_ids_epyc(topo_info, &topo_ids);
+}
/* Make APIC ID for the CPU based on Pkg_ID, Core_ID, SMT_ID
*
* The caller must make sure core_id < nr_cores and smt_id < nr_threads.
^ permalink raw reply related [flat|nested] 38+ messages in thread
* [PATCH v7 08/13] target/i386: Cleanup and use the EPYC mode topology functions
2020-03-11 22:52 [PATCH v7 00/13] APIC ID fixes for AMD EPYC CPU model Babu Moger
` (6 preceding siblings ...)
2020-03-11 22:53 ` [PATCH v7 07/13] hw/386: Add EPYC mode topology decoding functions Babu Moger
@ 2020-03-11 22:53 ` Babu Moger
2020-03-11 22:53 ` [PATCH v7 09/13] hw/i386: Introduce apicid functions inside X86MachineState Babu Moger
` (5 subsequent siblings)
13 siblings, 0 replies; 38+ messages in thread
From: Babu Moger @ 2020-03-11 22:53 UTC (permalink / raw)
To: ehabkost, marcel.apfelbaum, pbonzini, rth, mst, imammedo; +Cc: qemu-devel
Use the new functions from topology.h and delete the unused code. Given the
sockets, nodes, cores and threads, the new functions generate apic id for EPYC
mode. Removes all the hardcoded values.
Signed-off-by: Babu Moger <babu.moger@amd.com>
Acked-by: Michael S. Tsirkin <mst@redhat.com>
Acked-by: Igor Mammedov <imammedo@redhat.com>
---
target/i386/cpu.c | 161 +++++++++++------------------------------------------
1 file changed, 34 insertions(+), 127 deletions(-)
diff --git a/target/i386/cpu.c b/target/i386/cpu.c
index 2e5be37b21..7361a53166 100644
--- a/target/i386/cpu.c
+++ b/target/i386/cpu.c
@@ -338,68 +338,15 @@ static void encode_cache_cpuid80000006(CPUCacheInfo *l2,
}
}
-/*
- * Definitions used for building CPUID Leaf 0x8000001D and 0x8000001E
- * Please refer to the AMD64 Architecture Programmer’s Manual Volume 3.
- * Define the constants to build the cpu topology. Right now, TOPOEXT
- * feature is enabled only on EPYC. So, these constants are based on
- * EPYC supported configurations. We may need to handle the cases if
- * these values change in future.
- */
-/* Maximum core complexes in a node */
-#define MAX_CCX 2
-/* Maximum cores in a core complex */
-#define MAX_CORES_IN_CCX 4
-/* Maximum cores in a node */
-#define MAX_CORES_IN_NODE 8
-/* Maximum nodes in a socket */
-#define MAX_NODES_PER_SOCKET 4
-
-/*
- * Figure out the number of nodes required to build this config.
- * Max cores in a node is 8
- */
-static int nodes_in_socket(int nr_cores)
-{
- int nodes;
-
- nodes = DIV_ROUND_UP(nr_cores, MAX_CORES_IN_NODE);
-
- /* Hardware does not support config with 3 nodes, return 4 in that case */
- return (nodes == 3) ? 4 : nodes;
-}
-
-/*
- * Decide the number of cores in a core complex with the given nr_cores using
- * following set constants MAX_CCX, MAX_CORES_IN_CCX, MAX_CORES_IN_NODE and
- * MAX_NODES_PER_SOCKET. Maintain symmetry as much as possible
- * L3 cache is shared across all cores in a core complex. So, this will also
- * tell us how many cores are sharing the L3 cache.
- */
-static int cores_in_core_complex(int nr_cores)
-{
- int nodes;
-
- /* Check if we can fit all the cores in one core complex */
- if (nr_cores <= MAX_CORES_IN_CCX) {
- return nr_cores;
- }
- /* Get the number of nodes required to build this config */
- nodes = nodes_in_socket(nr_cores);
-
- /*
- * Divide the cores accros all the core complexes
- * Return rounded up value
- */
- return DIV_ROUND_UP(nr_cores, nodes * MAX_CCX);
-}
-
/* Encode cache info for CPUID[8000001D] */
-static void encode_cache_cpuid8000001d(CPUCacheInfo *cache, CPUState *cs,
- uint32_t *eax, uint32_t *ebx,
- uint32_t *ecx, uint32_t *edx)
+static void encode_cache_cpuid8000001d(CPUCacheInfo *cache,
+ X86CPUTopoInfo *topo_info,
+ uint32_t *eax, uint32_t *ebx,
+ uint32_t *ecx, uint32_t *edx)
{
uint32_t l3_cores;
+ unsigned nodes = MAX(topo_info->nodes_per_pkg, 1);
+
assert(cache->size == cache->line_size * cache->associativity *
cache->partitions * cache->sets);
@@ -408,10 +355,13 @@ static void encode_cache_cpuid8000001d(CPUCacheInfo *cache, CPUState *cs,
/* L3 is shared among multiple cores */
if (cache->level == 3) {
- l3_cores = cores_in_core_complex(cs->nr_cores);
- *eax |= ((l3_cores * cs->nr_threads) - 1) << 14;
+ l3_cores = DIV_ROUND_UP((topo_info->dies_per_pkg *
+ topo_info->cores_per_die *
+ topo_info->threads_per_core),
+ nodes);
+ *eax |= (l3_cores - 1) << 14;
} else {
- *eax |= ((cs->nr_threads - 1) << 14);
+ *eax |= ((topo_info->threads_per_core - 1) << 14);
}
assert(cache->line_size > 0);
@@ -431,55 +381,17 @@ static void encode_cache_cpuid8000001d(CPUCacheInfo *cache, CPUState *cs,
(cache->complex_indexing ? CACHE_COMPLEX_IDX : 0);
}
-/* Data structure to hold the configuration info for a given core index */
-struct core_topology {
- /* core complex id of the current core index */
- int ccx_id;
- /*
- * Adjusted core index for this core in the topology
- * This can be 0,1,2,3 with max 4 cores in a core complex
- */
- int core_id;
- /* Node id for this core index */
- int node_id;
- /* Number of nodes in this config */
- int num_nodes;
-};
-
-/*
- * Build the configuration closely match the EPYC hardware. Using the EPYC
- * hardware configuration values (MAX_CCX, MAX_CORES_IN_CCX, MAX_CORES_IN_NODE)
- * right now. This could change in future.
- * nr_cores : Total number of cores in the config
- * core_id : Core index of the current CPU
- * topo : Data structure to hold all the config info for this core index
- */
-static void build_core_topology(int nr_cores, int core_id,
- struct core_topology *topo)
-{
- int nodes, cores_in_ccx;
-
- /* First get the number of nodes required */
- nodes = nodes_in_socket(nr_cores);
-
- cores_in_ccx = cores_in_core_complex(nr_cores);
-
- topo->node_id = core_id / (cores_in_ccx * MAX_CCX);
- topo->ccx_id = (core_id % (cores_in_ccx * MAX_CCX)) / cores_in_ccx;
- topo->core_id = core_id % cores_in_ccx;
- topo->num_nodes = nodes;
-}
-
/* Encode cache info for CPUID[8000001E] */
-static void encode_topo_cpuid8000001e(CPUState *cs, X86CPU *cpu,
+static void encode_topo_cpuid8000001e(X86CPUTopoInfo *topo_info, X86CPU *cpu,
uint32_t *eax, uint32_t *ebx,
uint32_t *ecx, uint32_t *edx)
{
- struct core_topology topo = {0};
- unsigned long nodes;
+ X86CPUTopoIDs topo_ids = {0};
+ unsigned long nodes = MAX(topo_info->nodes_per_pkg, 1);
int shift;
- build_core_topology(cs->nr_cores, cpu->core_id, &topo);
+ x86_topo_ids_from_apicid_epyc(cpu->apic_id, topo_info, &topo_ids);
+
*eax = cpu->apic_id;
/*
* CPUID_Fn8000001E_EBX
@@ -496,12 +408,8 @@ static void encode_topo_cpuid8000001e(CPUState *cs, X86CPU *cpu,
* 3 Core complex id
* 1:0 Core id
*/
- if (cs->nr_threads - 1) {
- *ebx = ((cs->nr_threads - 1) << 8) | (topo.node_id << 3) |
- (topo.ccx_id << 2) | topo.core_id;
- } else {
- *ebx = (topo.node_id << 4) | (topo.ccx_id << 3) | topo.core_id;
- }
+ *ebx = ((topo_info->threads_per_core - 1) << 8) | (topo_ids.node_id << 3) |
+ (topo_ids.core_id);
/*
* CPUID_Fn8000001E_ECX
* 31:11 Reserved
@@ -510,9 +418,8 @@ static void encode_topo_cpuid8000001e(CPUState *cs, X86CPU *cpu,
* 2 Socket id
* 1:0 Node id
*/
- if (topo.num_nodes <= 4) {
- *ecx = ((topo.num_nodes - 1) << 8) | (cpu->socket_id << 2) |
- topo.node_id;
+ if (nodes <= 4) {
+ *ecx = ((nodes - 1) << 8) | (topo_ids.pkg_id << 2) | topo_ids.node_id;
} else {
/*
* Node id fix up. Actual hardware supports up to 4 nodes. But with
@@ -527,10 +434,10 @@ static void encode_topo_cpuid8000001e(CPUState *cs, X86CPU *cpu,
* number of nodes. find_last_bit returns last set bit(0 based). Left
* shift(+1) the socket id to represent all the nodes.
*/
- nodes = topo.num_nodes - 1;
+ nodes -= 1;
shift = find_last_bit(&nodes, 8);
- *ecx = ((topo.num_nodes - 1) << 8) | (cpu->socket_id << (shift + 1)) |
- topo.node_id;
+ *ecx = (nodes << 8) | (topo_ids.pkg_id << (shift + 1)) |
+ topo_ids.node_id;
}
*edx = 0;
}
@@ -5499,6 +5406,7 @@ void cpu_x86_cpuid(CPUX86State *env, uint32_t index, uint32_t count,
uint32_t signature[3];
X86CPUTopoInfo topo_info;
+ topo_info.nodes_per_pkg = env->nr_nodes;
topo_info.dies_per_pkg = env->nr_dies;
topo_info.cores_per_die = cs->nr_cores;
topo_info.threads_per_core = cs->nr_threads;
@@ -5918,20 +5826,20 @@ void cpu_x86_cpuid(CPUX86State *env, uint32_t index, uint32_t count,
}
switch (count) {
case 0: /* L1 dcache info */
- encode_cache_cpuid8000001d(env->cache_info_amd.l1d_cache, cs,
- eax, ebx, ecx, edx);
+ encode_cache_cpuid8000001d(env->cache_info_amd.l1d_cache,
+ &topo_info, eax, ebx, ecx, edx);
break;
case 1: /* L1 icache info */
- encode_cache_cpuid8000001d(env->cache_info_amd.l1i_cache, cs,
- eax, ebx, ecx, edx);
+ encode_cache_cpuid8000001d(env->cache_info_amd.l1i_cache,
+ &topo_info, eax, ebx, ecx, edx);
break;
case 2: /* L2 cache info */
- encode_cache_cpuid8000001d(env->cache_info_amd.l2_cache, cs,
- eax, ebx, ecx, edx);
+ encode_cache_cpuid8000001d(env->cache_info_amd.l2_cache,
+ &topo_info, eax, ebx, ecx, edx);
break;
case 3: /* L3 cache info */
- encode_cache_cpuid8000001d(env->cache_info_amd.l3_cache, cs,
- eax, ebx, ecx, edx);
+ encode_cache_cpuid8000001d(env->cache_info_amd.l3_cache,
+ &topo_info, eax, ebx, ecx, edx);
break;
default: /* end of info */
*eax = *ebx = *ecx = *edx = 0;
@@ -5940,8 +5848,7 @@ void cpu_x86_cpuid(CPUX86State *env, uint32_t index, uint32_t count,
break;
case 0x8000001E:
assert(cpu->core_id <= 255);
- encode_topo_cpuid8000001e(cs, cpu,
- eax, ebx, ecx, edx);
+ encode_topo_cpuid8000001e(&topo_info, cpu, eax, ebx, ecx, edx);
break;
case 0xC0000000:
*eax = env->cpuid_xlevel2;
^ permalink raw reply related [flat|nested] 38+ messages in thread
* [PATCH v7 09/13] hw/i386: Introduce apicid functions inside X86MachineState
2020-03-11 22:52 [PATCH v7 00/13] APIC ID fixes for AMD EPYC CPU model Babu Moger
` (7 preceding siblings ...)
2020-03-11 22:53 ` [PATCH v7 08/13] target/i386: Cleanup and use the EPYC mode topology functions Babu Moger
@ 2020-03-11 22:53 ` Babu Moger
2020-03-11 22:53 ` [PATCH v7 10/13] i386: Introduce use_epyc_apic_id_encoding in X86CPUDefinition Babu Moger
` (4 subsequent siblings)
13 siblings, 0 replies; 38+ messages in thread
From: Babu Moger @ 2020-03-11 22:53 UTC (permalink / raw)
To: ehabkost, marcel.apfelbaum, pbonzini, rth, mst, imammedo; +Cc: qemu-devel
Introduce model specific apicid functions inside X86MachineState.
These functions will be loaded from X86CPUDefinition.
Signed-off-by: Babu Moger <babu.moger@amd.com>
Reviewed-by: Igor Mammedov <imammedo@redhat.com>
Acked-by: Michael S. Tsirkin <mst@redhat.com>
---
hw/i386/x86.c | 5 +++++
include/hw/i386/x86.h | 9 +++++++++
2 files changed, 14 insertions(+)
diff --git a/hw/i386/x86.c b/hw/i386/x86.c
index 42834d2319..0a81ab5151 100644
--- a/hw/i386/x86.c
+++ b/hw/i386/x86.c
@@ -911,6 +911,11 @@ static void x86_machine_initfn(Object *obj)
x86ms->smm = ON_OFF_AUTO_AUTO;
x86ms->max_ram_below_4g = 0; /* use default */
x86ms->smp_dies = 1;
+
+ x86ms->apicid_from_cpu_idx = x86_apicid_from_cpu_idx;
+ x86ms->topo_ids_from_apicid = x86_topo_ids_from_apicid;
+ x86ms->apicid_from_topo_ids = x86_apicid_from_topo_ids;
+ x86ms->apicid_pkg_offset = apicid_pkg_offset;
}
static void x86_machine_class_init(ObjectClass *oc, void *data)
diff --git a/include/hw/i386/x86.h b/include/hw/i386/x86.h
index 22babcb3bb..2643b57629 100644
--- a/include/hw/i386/x86.h
+++ b/include/hw/i386/x86.h
@@ -65,6 +65,15 @@ typedef struct {
OnOffAuto smm;
+ /* Apic id specific handlers */
+ uint32_t (*apicid_from_cpu_idx)(X86CPUTopoInfo *topo_info,
+ unsigned cpu_index);
+ void (*topo_ids_from_apicid)(apic_id_t apicid, X86CPUTopoInfo *topo_info,
+ X86CPUTopoIDs *topo_ids);
+ apic_id_t (*apicid_from_topo_ids)(X86CPUTopoInfo *topo_info,
+ const X86CPUTopoIDs *topo_ids);
+ uint32_t (*apicid_pkg_offset)(X86CPUTopoInfo *topo_info);
+
/*
* Address space used by IOAPIC device. All IOAPIC interrupts
* will be translated to MSI messages in the address space.
^ permalink raw reply related [flat|nested] 38+ messages in thread
* [PATCH v7 10/13] i386: Introduce use_epyc_apic_id_encoding in X86CPUDefinition
2020-03-11 22:52 [PATCH v7 00/13] APIC ID fixes for AMD EPYC CPU model Babu Moger
` (8 preceding siblings ...)
2020-03-11 22:53 ` [PATCH v7 09/13] hw/i386: Introduce apicid functions inside X86MachineState Babu Moger
@ 2020-03-11 22:53 ` Babu Moger
2020-03-12 12:26 ` Igor Mammedov
2020-03-11 22:54 ` [PATCH v7 11/13] hw/i386: Move arch_id decode inside x86_cpus_init Babu Moger
` (3 subsequent siblings)
13 siblings, 1 reply; 38+ messages in thread
From: Babu Moger @ 2020-03-11 22:53 UTC (permalink / raw)
To: ehabkost, marcel.apfelbaum, pbonzini, rth, mst, imammedo; +Cc: qemu-devel
Add a boolean variable use_epyc_apic_id_encoding in X86CPUDefinition.
This will be set if this cpu model needs to use new EPYC based
apic id encoding.
Override the handlers with EPYC based handlers if use_epyc_apic_id_encoding
is set. This will be done in x86_cpus_init.
Signed-off-by: Babu Moger <babu.moger@amd.com>
---
target/i386/cpu.c | 16 ++++++++++++++++
target/i386/cpu.h | 1 +
2 files changed, 17 insertions(+)
diff --git a/target/i386/cpu.c b/target/i386/cpu.c
index 7361a53166..1e4400df7a 100644
--- a/target/i386/cpu.c
+++ b/target/i386/cpu.c
@@ -1614,6 +1614,10 @@ typedef struct X86CPUDefinition {
FeatureWordArray features;
const char *model_id;
CPUCaches *cache_info;
+
+ /* Use AMD EPYC encoding for apic id */
+ bool use_epyc_apic_id_encoding;
+
/*
* Definitions for alternative versions of CPU model.
* List is terminated by item with version == 0.
@@ -1655,6 +1659,18 @@ static const X86CPUVersionDefinition *x86_cpu_def_get_versions(X86CPUDefinition
return def->versions ?: default_version_list;
}
+bool cpu_x86_use_epyc_apic_id_encoding(const char *cpu_type)
+{
+ X86CPUClass *xcc = X86_CPU_CLASS(object_class_by_name(cpu_type));
+
+ assert(xcc);
+ if (xcc->model && xcc->model->cpudef) {
+ return xcc->model->cpudef->use_epyc_apic_id_encoding;
+ } else {
+ return false;
+ }
+}
+
static CPUCaches epyc_cache_info = {
.l1d_cache = &(CPUCacheInfo) {
.type = DATA_CACHE,
diff --git a/target/i386/cpu.h b/target/i386/cpu.h
index 7e9e963d78..6e522fcd34 100644
--- a/target/i386/cpu.h
+++ b/target/i386/cpu.h
@@ -1897,6 +1897,7 @@ void cpu_clear_apic_feature(CPUX86State *env);
void host_cpuid(uint32_t function, uint32_t count,
uint32_t *eax, uint32_t *ebx, uint32_t *ecx, uint32_t *edx);
void host_vendor_fms(char *vendor, int *family, int *model, int *stepping);
+bool cpu_x86_use_epyc_apic_id_encoding(const char *cpu_type);
/* helper.c */
bool x86_cpu_tlb_fill(CPUState *cs, vaddr address, int size,
^ permalink raw reply related [flat|nested] 38+ messages in thread
* [PATCH v7 11/13] hw/i386: Move arch_id decode inside x86_cpus_init
2020-03-11 22:52 [PATCH v7 00/13] APIC ID fixes for AMD EPYC CPU model Babu Moger
` (9 preceding siblings ...)
2020-03-11 22:53 ` [PATCH v7 10/13] i386: Introduce use_epyc_apic_id_encoding in X86CPUDefinition Babu Moger
@ 2020-03-11 22:54 ` Babu Moger
2020-03-12 12:31 ` Igor Mammedov
2020-03-11 22:54 ` [PATCH v7 12/13] target/i386: Enable new apic id encoding for EPYC based cpus models Babu Moger
` (2 subsequent siblings)
13 siblings, 1 reply; 38+ messages in thread
From: Babu Moger @ 2020-03-11 22:54 UTC (permalink / raw)
To: ehabkost, marcel.apfelbaum, pbonzini, rth, mst, imammedo; +Cc: qemu-devel
Apicid calculation depends on knowing the total number of numa nodes
for EPYC cpu models. Right now, we are calculating the arch_id while
parsing the numa(parse_numa). At this time, it is not known how many
total numa nodes are configured in the system.
Move the arch_id calculation inside x86_cpus_init. At this time, smp
parse is already completed and numa node information is available.
Override the handlers if use_epyc_apic_id_encoding is enabled in
cpu model definition.
Also replace the calling convention to use handlers from
X86MachineState.
Signed-off-by: Babu Moger <babu.moger@amd.com>
---
hw/i386/pc.c | 6 +++---
hw/i386/x86.c | 37 ++++++++++++++++++++++++++++++-------
2 files changed, 33 insertions(+), 10 deletions(-)
diff --git a/hw/i386/pc.c b/hw/i386/pc.c
index 98ee763f68..2d7d611184 100644
--- a/hw/i386/pc.c
+++ b/hw/i386/pc.c
@@ -1580,14 +1580,14 @@ static void pc_cpu_pre_plug(HotplugHandler *hotplug_dev,
topo_ids.die_id = cpu->die_id;
topo_ids.core_id = cpu->core_id;
topo_ids.smt_id = cpu->thread_id;
- cpu->apic_id = x86_apicid_from_topo_ids(&topo_info, &topo_ids);
+ cpu->apic_id = x86ms->apicid_from_topo_ids(&topo_info, &topo_ids);
}
cpu_slot = pc_find_cpu_slot(MACHINE(pcms), cpu->apic_id, &idx);
if (!cpu_slot) {
MachineState *ms = MACHINE(pcms);
- x86_topo_ids_from_apicid(cpu->apic_id, &topo_info, &topo_ids);
+ x86ms->topo_ids_from_apicid(cpu->apic_id, &topo_info, &topo_ids);
error_setg(errp,
"Invalid CPU [socket: %u, die: %u, core: %u, thread: %u] with"
" APIC ID %" PRIu32 ", valid index range 0:%d",
@@ -1608,7 +1608,7 @@ static void pc_cpu_pre_plug(HotplugHandler *hotplug_dev,
/* TODO: move socket_id/core_id/thread_id checks into x86_cpu_realizefn()
* once -smp refactoring is complete and there will be CPU private
* CPUState::nr_cores and CPUState::nr_threads fields instead of globals */
- x86_topo_ids_from_apicid(cpu->apic_id, &topo_info, &topo_ids);
+ x86ms->topo_ids_from_apicid(cpu->apic_id, &topo_info, &topo_ids);
if (cpu->socket_id != -1 && cpu->socket_id != topo_ids.pkg_id) {
error_setg(errp, "property socket-id: %u doesn't match set apic-id:"
" 0x%x (socket-id: %u)", cpu->socket_id, cpu->apic_id,
diff --git a/hw/i386/x86.c b/hw/i386/x86.c
index 0a81ab5151..023dce1dbd 100644
--- a/hw/i386/x86.c
+++ b/hw/i386/x86.c
@@ -68,6 +68,22 @@ inline void init_topo_info(X86CPUTopoInfo *topo_info,
topo_info->threads_per_core = ms->smp.threads;
}
+/*
+ * Set up with the new EPYC topology handlers
+ *
+ * AMD uses different apic id encoding for EPYC based cpus. Override
+ * the default topo handlers with EPYC encoding handlers.
+ */
+static void x86_set_epyc_topo_handlers(MachineState *machine)
+{
+ X86MachineState *x86ms = X86_MACHINE(machine);
+
+ x86ms->apicid_from_cpu_idx = x86_apicid_from_cpu_idx_epyc;
+ x86ms->topo_ids_from_apicid = x86_topo_ids_from_apicid_epyc;
+ x86ms->apicid_from_topo_ids = x86_apicid_from_topo_ids_epyc;
+ x86ms->apicid_pkg_offset = apicid_pkg_offset_epyc;
+}
+
/*
* Calculates initial APIC ID for a specific CPU index
*
@@ -86,7 +102,7 @@ uint32_t x86_cpu_apic_id_from_index(X86MachineState *x86ms,
init_topo_info(&topo_info, x86ms);
- correct_id = x86_apicid_from_cpu_idx(&topo_info, cpu_index);
+ correct_id = x86ms->apicid_from_cpu_idx(&topo_info, cpu_index);
if (x86mc->compat_apic_id_mode) {
if (cpu_index != correct_id && !warned && !qtest_enabled()) {
error_report("APIC IDs set in compatibility mode, "
@@ -121,6 +137,11 @@ void x86_cpus_init(X86MachineState *x86ms, int default_cpu_version)
MachineState *ms = MACHINE(x86ms);
MachineClass *mc = MACHINE_GET_CLASS(x86ms);
+ /* Check for apicid encoding */
+ if (cpu_x86_use_epyc_apic_id_encoding(ms->cpu_type)) {
+ x86_set_epyc_topo_handlers(ms);
+ }
+
x86_cpu_set_default_version(default_cpu_version);
/*
@@ -134,6 +155,12 @@ void x86_cpus_init(X86MachineState *x86ms, int default_cpu_version)
x86ms->apic_id_limit = x86_cpu_apic_id_from_index(x86ms,
ms->smp.max_cpus - 1) + 1;
possible_cpus = mc->possible_cpu_arch_ids(ms);
+
+ for (i = 0; i < ms->smp.cpus; i++) {
+ ms->possible_cpus->cpus[i].arch_id =
+ x86_cpu_apic_id_from_index(x86ms, i);
+ }
+
for (i = 0; i < ms->smp.cpus; i++) {
x86_cpu_new(x86ms, possible_cpus->cpus[i].arch_id, &error_fatal);
}
@@ -158,8 +185,7 @@ int64_t x86_get_default_cpu_node_id(const MachineState *ms, int idx)
init_topo_info(&topo_info, x86ms);
assert(idx < ms->possible_cpus->len);
- x86_topo_ids_from_apicid(ms->possible_cpus->cpus[idx].arch_id,
- &topo_info, &topo_ids);
+ x86_topo_ids_from_idx(&topo_info, idx, &topo_ids);
return topo_ids.pkg_id % ms->numa_state->num_nodes;
}
@@ -190,10 +216,7 @@ const CPUArchIdList *x86_possible_cpu_arch_ids(MachineState *ms)
ms->possible_cpus->cpus[i].type = ms->cpu_type;
ms->possible_cpus->cpus[i].vcpus_count = 1;
- ms->possible_cpus->cpus[i].arch_id =
- x86_cpu_apic_id_from_index(x86ms, i);
- x86_topo_ids_from_apicid(ms->possible_cpus->cpus[i].arch_id,
- &topo_info, &topo_ids);
+ x86_topo_ids_from_idx(&topo_info, i, &topo_ids);
ms->possible_cpus->cpus[i].props.has_socket_id = true;
ms->possible_cpus->cpus[i].props.socket_id = topo_ids.pkg_id;
if (x86ms->smp_dies > 1) {
^ permalink raw reply related [flat|nested] 38+ messages in thread
* [PATCH v7 12/13] target/i386: Enable new apic id encoding for EPYC based cpus models
2020-03-11 22:52 [PATCH v7 00/13] APIC ID fixes for AMD EPYC CPU model Babu Moger
` (10 preceding siblings ...)
2020-03-11 22:54 ` [PATCH v7 11/13] hw/i386: Move arch_id decode inside x86_cpus_init Babu Moger
@ 2020-03-11 22:54 ` Babu Moger
2020-03-11 22:54 ` [PATCH v7 13/13] i386: Fix pkg_id offset for EPYC cpu models Babu Moger
2020-03-12 16:28 ` [PATCH v7 00/13] APIC ID fixes for AMD EPYC CPU model Babu Moger
13 siblings, 0 replies; 38+ messages in thread
From: Babu Moger @ 2020-03-11 22:54 UTC (permalink / raw)
To: ehabkost, marcel.apfelbaum, pbonzini, rth, mst, imammedo; +Cc: qemu-devel
The APIC ID is decoded based on the sequence sockets->dies->cores->threads.
This works fine for most standard AMD and other vendors' configurations,
but this decoding sequence does not follow that of AMD's APIC ID enumeration
strictly. In some cases this can cause CPU topology inconsistency.
When booting a guest VM, the kernel tries to validate the topology, and finds
it inconsistent with the enumeration of EPYC cpu models. The more details are
in the bug https://bugzilla.redhat.com/show_bug.cgi?id=1728166.
To fix the problem we need to build the topology as per the Processor
Programming Reference (PPR) for AMD Family 17h Model 01h, Revision B1
Processors. The documentation is available from the bugzilla Link below.
Link: https://bugzilla.kernel.org/show_bug.cgi?id=206537
It is also available at
https://www.amd.com/system/files/TechDocs/55570-B1_PUB.zip
Here is the text from the PPR.
Operating systems are expected to use Core::X86::Cpuid::SizeId[ApicIdSize], the
number of least significant bits in the Initial APIC ID that indicate core ID
within a processor, in constructing per-core CPUID masks.
Core::X86::Cpuid::SizeId[ApicIdSize] determines the maximum number of cores
(MNC) that the processor could theoretically support, not the actual number of
cores that are actually implemented or enabled on the processor, as indicated
by Core::X86::Cpuid::SizeId[NC].
Each Core::X86::Apic::ApicId[ApicId] register is preset as follows:
• ApicId[6] = Socket ID.
• ApicId[5:4] = Node ID.
• ApicId[3] = Logical CCX L3 complex ID
• ApicId[2:0]= (SMT) ? {LogicalCoreID[1:0],ThreadId} : {1'b0,LogicalCoreID[1:0]}
The new apic id encoding is enabled for EPYC and EPYC-Rome models.
Signed-off-by: Babu Moger <babu.moger@amd.com>
Acked-by: Michael S. Tsirkin <mst@redhat.com>
Acked-by: Igor Mammedov <imammedo@redhat.com>
---
target/i386/cpu.c | 2 ++
1 file changed, 2 insertions(+)
diff --git a/target/i386/cpu.c b/target/i386/cpu.c
index 1e4400df7a..9ffaf5e2a0 100644
--- a/target/i386/cpu.c
+++ b/target/i386/cpu.c
@@ -3925,6 +3925,7 @@ static X86CPUDefinition builtin_x86_defs[] = {
.xlevel = 0x8000001E,
.model_id = "AMD EPYC Processor",
.cache_info = &epyc_cache_info,
+ .use_epyc_apic_id_encoding = 1,
.versions = (X86CPUVersionDefinition[]) {
{ .version = 1 },
{
@@ -4052,6 +4053,7 @@ static X86CPUDefinition builtin_x86_defs[] = {
.xlevel = 0x8000001E,
.model_id = "AMD EPYC-Rome Processor",
.cache_info = &epyc_rome_cache_info,
+ .use_epyc_apic_id_encoding = 1,
},
};
^ permalink raw reply related [flat|nested] 38+ messages in thread
* [PATCH v7 13/13] i386: Fix pkg_id offset for EPYC cpu models
2020-03-11 22:52 [PATCH v7 00/13] APIC ID fixes for AMD EPYC CPU model Babu Moger
` (11 preceding siblings ...)
2020-03-11 22:54 ` [PATCH v7 12/13] target/i386: Enable new apic id encoding for EPYC based cpus models Babu Moger
@ 2020-03-11 22:54 ` Babu Moger
2020-03-12 16:28 ` [PATCH v7 00/13] APIC ID fixes for AMD EPYC CPU model Babu Moger
13 siblings, 0 replies; 38+ messages in thread
From: Babu Moger @ 2020-03-11 22:54 UTC (permalink / raw)
To: ehabkost, marcel.apfelbaum, pbonzini, rth, mst, imammedo; +Cc: qemu-devel
If the system is numa configured the pkg_offset needs
to be adjusted for EPYC cpu models. Fix it calling the
model specific handler.
Signed-off-by: Babu Moger <babu.moger@amd.com>
Reviewed-by: Igor Mammedov <imammedo@redhat.com>
Acked-by: Michael S. Tsirkin <mst@redhat.com>
---
hw/i386/pc.c | 1 +
target/i386/cpu.c | 4 ++--
target/i386/cpu.h | 1 +
3 files changed, 4 insertions(+), 2 deletions(-)
diff --git a/hw/i386/pc.c b/hw/i386/pc.c
index 2d7d611184..ab6da19bab 100644
--- a/hw/i386/pc.c
+++ b/hw/i386/pc.c
@@ -1526,6 +1526,7 @@ static void pc_cpu_pre_plug(HotplugHandler *hotplug_dev,
env->nr_dies = x86ms->smp_dies;
env->nr_nodes = topo_info.nodes_per_pkg;
+ env->pkg_offset = x86ms->apicid_pkg_offset(&topo_info);
/*
* If APIC ID is not set,
diff --git a/target/i386/cpu.c b/target/i386/cpu.c
index 9ffaf5e2a0..c58ac38d29 100644
--- a/target/i386/cpu.c
+++ b/target/i386/cpu.c
@@ -5610,7 +5610,7 @@ void cpu_x86_cpuid(CPUX86State *env, uint32_t index, uint32_t count,
*ecx |= CPUID_TOPOLOGY_LEVEL_SMT;
break;
case 1:
- *eax = apicid_pkg_offset(&topo_info);
+ *eax = env->pkg_offset;
*ebx = cs->nr_cores * cs->nr_threads;
*ecx |= CPUID_TOPOLOGY_LEVEL_CORE;
break;
@@ -5644,7 +5644,7 @@ void cpu_x86_cpuid(CPUX86State *env, uint32_t index, uint32_t count,
*ecx |= CPUID_TOPOLOGY_LEVEL_CORE;
break;
case 2:
- *eax = apicid_pkg_offset(&topo_info);
+ *eax = env->pkg_offset;
*ebx = env->nr_dies * cs->nr_cores * cs->nr_threads;
*ecx |= CPUID_TOPOLOGY_LEVEL_DIE;
break;
diff --git a/target/i386/cpu.h b/target/i386/cpu.h
index 6e522fcd34..92872d2b7a 100644
--- a/target/i386/cpu.h
+++ b/target/i386/cpu.h
@@ -1610,6 +1610,7 @@ typedef struct CPUX86State {
unsigned nr_dies;
unsigned nr_nodes;
+ unsigned pkg_offset;
} CPUX86State;
struct kvm_msrs;
^ permalink raw reply related [flat|nested] 38+ messages in thread
* Re: [PATCH v7 01/13] hw/i386: Introduce X86CPUTopoInfo to contain topology info
2020-03-11 22:52 ` [PATCH v7 01/13] hw/i386: Introduce X86CPUTopoInfo to contain topology info Babu Moger
@ 2020-03-12 11:11 ` Igor Mammedov
0 siblings, 0 replies; 38+ messages in thread
From: Igor Mammedov @ 2020-03-12 11:11 UTC (permalink / raw)
To: Babu Moger; +Cc: ehabkost, mst, qemu-devel, pbonzini, rth
On Wed, 11 Mar 2020 17:52:52 -0500
Babu Moger <babu.moger@amd.com> wrote:
> This is an effort to re-arrange few data structure for better readability.
>
> 1. Add X86CPUTopoInfo which will have all the topology informations
> required to build the cpu topology. There is no functional changes.
>
> 2. Introduce init_topo_info to initialize X86CPUTopoInfo members from
> X86MachineState.
>
> 3. Update x86 unit tests for new calling convention with parameter X86CPUTopoInfo
>
> There is no functional changes.
>
> Signed-off-by: Babu Moger <babu.moger@amd.com>
Reviewed-by: Igor Mammedov <imammedo@redhat.com>
> ---
> hw/i386/pc.c | 12 ++++++------
> hw/i386/x86.c | 32 ++++++++++++++++++++++++--------
> include/hw/i386/topology.h | 38 ++++++++++++++++++++++++--------------
> include/hw/i386/x86.h | 3 +++
> tests/test-x86-cpuid.c | 43 ++++++++++++++++++++++++-------------------
> 5 files changed, 81 insertions(+), 47 deletions(-)
>
> diff --git a/hw/i386/pc.c b/hw/i386/pc.c
> index f52e84b2ba..662abb549d 100644
> --- a/hw/i386/pc.c
> +++ b/hw/i386/pc.c
> @@ -1513,6 +1513,7 @@ static void pc_cpu_pre_plug(HotplugHandler *hotplug_dev,
> X86MachineState *x86ms = X86_MACHINE(pcms);
> unsigned int smp_cores = ms->smp.cores;
> unsigned int smp_threads = ms->smp.threads;
> + X86CPUTopoInfo topo_info;
>
> if(!object_dynamic_cast(OBJECT(cpu), ms->cpu_type)) {
> error_setg(errp, "Invalid CPU type, expected cpu type: '%s'",
> @@ -1520,6 +1521,8 @@ static void pc_cpu_pre_plug(HotplugHandler *hotplug_dev,
> return;
> }
>
> + init_topo_info(&topo_info, x86ms);
> +
> env->nr_dies = x86ms->smp_dies;
>
> /*
> @@ -1575,16 +1578,14 @@ static void pc_cpu_pre_plug(HotplugHandler *hotplug_dev,
> topo_ids.die_id = cpu->die_id;
> topo_ids.core_id = cpu->core_id;
> topo_ids.smt_id = cpu->thread_id;
> - cpu->apic_id = apicid_from_topo_ids(x86ms->smp_dies, smp_cores,
> - smp_threads, &topo_ids);
> + cpu->apic_id = apicid_from_topo_ids(&topo_info, &topo_ids);
> }
>
> cpu_slot = pc_find_cpu_slot(MACHINE(pcms), cpu->apic_id, &idx);
> if (!cpu_slot) {
> MachineState *ms = MACHINE(pcms);
>
> - x86_topo_ids_from_apicid(cpu->apic_id, x86ms->smp_dies,
> - smp_cores, smp_threads, &topo_ids);
> + x86_topo_ids_from_apicid(cpu->apic_id, &topo_info, &topo_ids);
> error_setg(errp,
> "Invalid CPU [socket: %u, die: %u, core: %u, thread: %u] with"
> " APIC ID %" PRIu32 ", valid index range 0:%d",
> @@ -1605,8 +1606,7 @@ static void pc_cpu_pre_plug(HotplugHandler *hotplug_dev,
> /* TODO: move socket_id/core_id/thread_id checks into x86_cpu_realizefn()
> * once -smp refactoring is complete and there will be CPU private
> * CPUState::nr_cores and CPUState::nr_threads fields instead of globals */
> - x86_topo_ids_from_apicid(cpu->apic_id, x86ms->smp_dies,
> - smp_cores, smp_threads, &topo_ids);
> + x86_topo_ids_from_apicid(cpu->apic_id, &topo_info, &topo_ids);
> if (cpu->socket_id != -1 && cpu->socket_id != topo_ids.pkg_id) {
> error_setg(errp, "property socket-id: %u doesn't match set apic-id:"
> " 0x%x (socket-id: %u)", cpu->socket_id, cpu->apic_id,
> diff --git a/hw/i386/x86.c b/hw/i386/x86.c
> index 322fb6abbc..dbbff46a4b 100644
> --- a/hw/i386/x86.c
> +++ b/hw/i386/x86.c
> @@ -57,6 +57,16 @@
> /* Physical Address of PVH entry point read from kernel ELF NOTE */
> static size_t pvh_start_addr;
>
> +inline void init_topo_info(X86CPUTopoInfo *topo_info,
> + const X86MachineState *x86ms)
> +{
> + MachineState *ms = MACHINE(x86ms);
> +
> + topo_info->dies_per_pkg = x86ms->smp_dies;
> + topo_info->cores_per_die = ms->smp.cores;
> + topo_info->threads_per_core = ms->smp.threads;
> +}
> +
> /*
> * Calculates initial APIC ID for a specific CPU index
> *
> @@ -68,13 +78,14 @@ static size_t pvh_start_addr;
> uint32_t x86_cpu_apic_id_from_index(X86MachineState *x86ms,
> unsigned int cpu_index)
> {
> - MachineState *ms = MACHINE(x86ms);
> X86MachineClass *x86mc = X86_MACHINE_GET_CLASS(x86ms);
> + X86CPUTopoInfo topo_info;
> uint32_t correct_id;
> static bool warned;
>
> - correct_id = x86_apicid_from_cpu_idx(x86ms->smp_dies, ms->smp.cores,
> - ms->smp.threads, cpu_index);
> + init_topo_info(&topo_info, x86ms);
> +
> + correct_id = x86_apicid_from_cpu_idx(&topo_info, cpu_index);
> if (x86mc->compat_apic_id_mode) {
> if (cpu_index != correct_id && !warned && !qtest_enabled()) {
> error_report("APIC IDs set in compatibility mode, "
> @@ -145,19 +156,22 @@ int64_t x86_get_default_cpu_node_id(const MachineState *ms, int idx)
> {
> X86CPUTopoIDs topo_ids;
> X86MachineState *x86ms = X86_MACHINE(ms);
> + X86CPUTopoInfo topo_info;
> +
> + init_topo_info(&topo_info, x86ms);
>
> assert(idx < ms->possible_cpus->len);
> x86_topo_ids_from_apicid(ms->possible_cpus->cpus[idx].arch_id,
> - x86ms->smp_dies, ms->smp.cores,
> - ms->smp.threads, &topo_ids);
> + &topo_info, &topo_ids);
> return topo_ids.pkg_id % ms->numa_state->num_nodes;
> }
>
> const CPUArchIdList *x86_possible_cpu_arch_ids(MachineState *ms)
> {
> X86MachineState *x86ms = X86_MACHINE(ms);
> - int i;
> unsigned int max_cpus = ms->smp.max_cpus;
> + X86CPUTopoInfo topo_info;
> + int i;
>
> if (ms->possible_cpus) {
> /*
> @@ -171,6 +185,9 @@ const CPUArchIdList *x86_possible_cpu_arch_ids(MachineState *ms)
> ms->possible_cpus = g_malloc0(sizeof(CPUArchIdList) +
> sizeof(CPUArchId) * max_cpus);
> ms->possible_cpus->len = max_cpus;
> +
> + init_topo_info(&topo_info, x86ms);
> +
> for (i = 0; i < ms->possible_cpus->len; i++) {
> X86CPUTopoIDs topo_ids;
>
> @@ -179,8 +196,7 @@ const CPUArchIdList *x86_possible_cpu_arch_ids(MachineState *ms)
> ms->possible_cpus->cpus[i].arch_id =
> x86_cpu_apic_id_from_index(x86ms, i);
> x86_topo_ids_from_apicid(ms->possible_cpus->cpus[i].arch_id,
> - x86ms->smp_dies, ms->smp.cores,
> - ms->smp.threads, &topo_ids);
> + &topo_info, &topo_ids);
> ms->possible_cpus->cpus[i].props.has_socket_id = true;
> ms->possible_cpus->cpus[i].props.socket_id = topo_ids.pkg_id;
> if (x86ms->smp_dies > 1) {
> diff --git a/include/hw/i386/topology.h b/include/hw/i386/topology.h
> index 52def68610..7ea507f376 100644
> --- a/include/hw/i386/topology.h
> +++ b/include/hw/i386/topology.h
> @@ -52,6 +52,12 @@ typedef struct X86CPUTopoIDs {
> unsigned smt_id;
> } X86CPUTopoIDs;
>
> +typedef struct X86CPUTopoInfo {
> + unsigned dies_per_pkg;
> + unsigned cores_per_die;
> + unsigned threads_per_core;
> +} X86CPUTopoInfo;
> +
> /* Return the bit width needed for 'count' IDs
> */
> static unsigned apicid_bitwidth_for_count(unsigned count)
> @@ -119,11 +125,13 @@ static inline unsigned apicid_pkg_offset(unsigned nr_dies,
> *
> * The caller must make sure core_id < nr_cores and smt_id < nr_threads.
> */
> -static inline apic_id_t apicid_from_topo_ids(unsigned nr_dies,
> - unsigned nr_cores,
> - unsigned nr_threads,
> +static inline apic_id_t apicid_from_topo_ids(X86CPUTopoInfo *topo_info,
> const X86CPUTopoIDs *topo_ids)
> {
> + unsigned nr_dies = topo_info->dies_per_pkg;
> + unsigned nr_cores = topo_info->cores_per_die;
> + unsigned nr_threads = topo_info->threads_per_core;
> +
> return (topo_ids->pkg_id <<
> apicid_pkg_offset(nr_dies, nr_cores, nr_threads)) |
> (topo_ids->die_id <<
> @@ -136,12 +144,14 @@ static inline apic_id_t apicid_from_topo_ids(unsigned nr_dies,
> /* Calculate thread/core/package IDs for a specific topology,
> * based on (contiguous) CPU index
> */
> -static inline void x86_topo_ids_from_idx(unsigned nr_dies,
> - unsigned nr_cores,
> - unsigned nr_threads,
> +static inline void x86_topo_ids_from_idx(X86CPUTopoInfo *topo_info,
> unsigned cpu_index,
> X86CPUTopoIDs *topo_ids)
> {
> + unsigned nr_dies = topo_info->dies_per_pkg;
> + unsigned nr_cores = topo_info->cores_per_die;
> + unsigned nr_threads = topo_info->threads_per_core;
> +
> topo_ids->pkg_id = cpu_index / (nr_dies * nr_cores * nr_threads);
> topo_ids->die_id = cpu_index / (nr_cores * nr_threads) % nr_dies;
> topo_ids->core_id = cpu_index / nr_threads % nr_cores;
> @@ -152,11 +162,13 @@ static inline void x86_topo_ids_from_idx(unsigned nr_dies,
> * based on APIC ID
> */
> static inline void x86_topo_ids_from_apicid(apic_id_t apicid,
> - unsigned nr_dies,
> - unsigned nr_cores,
> - unsigned nr_threads,
> + X86CPUTopoInfo *topo_info,
> X86CPUTopoIDs *topo_ids)
> {
> + unsigned nr_dies = topo_info->dies_per_pkg;
> + unsigned nr_cores = topo_info->cores_per_die;
> + unsigned nr_threads = topo_info->threads_per_core;
> +
> topo_ids->smt_id = apicid &
> ~(0xFFFFFFFFUL << apicid_smt_width(nr_dies, nr_cores, nr_threads));
> topo_ids->core_id =
> @@ -173,14 +185,12 @@ static inline void x86_topo_ids_from_apicid(apic_id_t apicid,
> *
> * 'cpu_index' is a sequential, contiguous ID for the CPU.
> */
> -static inline apic_id_t x86_apicid_from_cpu_idx(unsigned nr_dies,
> - unsigned nr_cores,
> - unsigned nr_threads,
> +static inline apic_id_t x86_apicid_from_cpu_idx(X86CPUTopoInfo *topo_info,
> unsigned cpu_index)
> {
> X86CPUTopoIDs topo_ids;
> - x86_topo_ids_from_idx(nr_dies, nr_cores, nr_threads, cpu_index, &topo_ids);
> - return apicid_from_topo_ids(nr_dies, nr_cores, nr_threads, &topo_ids);
> + x86_topo_ids_from_idx(topo_info, cpu_index, &topo_ids);
> + return apicid_from_topo_ids(topo_info, &topo_ids);
> }
>
> #endif /* HW_I386_TOPOLOGY_H */
> diff --git a/include/hw/i386/x86.h b/include/hw/i386/x86.h
> index 41fe37b8a3..22babcb3bb 100644
> --- a/include/hw/i386/x86.h
> +++ b/include/hw/i386/x86.h
> @@ -21,6 +21,7 @@
> #include "exec/hwaddr.h"
> #include "qemu/notify.h"
>
> +#include "hw/i386/topology.h"
> #include "hw/boards.h"
> #include "hw/nmi.h"
> #include "hw/isa/isa.h"
> @@ -82,6 +83,8 @@ typedef struct {
> #define X86_MACHINE_CLASS(class) \
> OBJECT_CLASS_CHECK(X86MachineClass, class, TYPE_X86_MACHINE)
>
> +void init_topo_info(X86CPUTopoInfo *topo_info, const X86MachineState *x86ms);
> +
> uint32_t x86_cpu_apic_id_from_index(X86MachineState *pcms,
> unsigned int cpu_index);
>
> diff --git a/tests/test-x86-cpuid.c b/tests/test-x86-cpuid.c
> index 1942287f33..66b953113b 100644
> --- a/tests/test-x86-cpuid.c
> +++ b/tests/test-x86-cpuid.c
> @@ -28,15 +28,18 @@
>
> static void test_topo_bits(void)
> {
> + X86CPUTopoInfo topo_info = {0};
> +
> /* simple tests for 1 thread per core, 1 core per die, 1 die per package */
> g_assert_cmpuint(apicid_smt_width(1, 1, 1), ==, 0);
> g_assert_cmpuint(apicid_core_width(1, 1, 1), ==, 0);
> g_assert_cmpuint(apicid_die_width(1, 1, 1), ==, 0);
>
> - g_assert_cmpuint(x86_apicid_from_cpu_idx(1, 1, 1, 0), ==, 0);
> - g_assert_cmpuint(x86_apicid_from_cpu_idx(1, 1, 1, 1), ==, 1);
> - g_assert_cmpuint(x86_apicid_from_cpu_idx(1, 1, 1, 2), ==, 2);
> - g_assert_cmpuint(x86_apicid_from_cpu_idx(1, 1, 1, 3), ==, 3);
> + topo_info = (X86CPUTopoInfo) {1, 1, 1};
> + g_assert_cmpuint(x86_apicid_from_cpu_idx(&topo_info, 0), ==, 0);
> + g_assert_cmpuint(x86_apicid_from_cpu_idx(&topo_info, 1), ==, 1);
> + g_assert_cmpuint(x86_apicid_from_cpu_idx(&topo_info, 2), ==, 2);
> + g_assert_cmpuint(x86_apicid_from_cpu_idx(&topo_info, 3), ==, 3);
>
>
> /* Test field width calculation for multiple values
> @@ -71,36 +74,38 @@ static void test_topo_bits(void)
> g_assert_cmpuint(apicid_die_offset(1, 6, 3), ==, 5);
> g_assert_cmpuint(apicid_pkg_offset(1, 6, 3), ==, 5);
>
> - g_assert_cmpuint(x86_apicid_from_cpu_idx(1, 6, 3, 0), ==, 0);
> - g_assert_cmpuint(x86_apicid_from_cpu_idx(1, 6, 3, 1), ==, 1);
> - g_assert_cmpuint(x86_apicid_from_cpu_idx(1, 6, 3, 2), ==, 2);
> + topo_info = (X86CPUTopoInfo) {1, 6, 3};
> + g_assert_cmpuint(x86_apicid_from_cpu_idx(&topo_info, 0), ==, 0);
> + g_assert_cmpuint(x86_apicid_from_cpu_idx(&topo_info, 1), ==, 1);
> + g_assert_cmpuint(x86_apicid_from_cpu_idx(&topo_info, 2), ==, 2);
>
> - g_assert_cmpuint(x86_apicid_from_cpu_idx(1, 6, 3, 1 * 3 + 0), ==,
> + topo_info = (X86CPUTopoInfo) {1, 6, 3};
> + g_assert_cmpuint(x86_apicid_from_cpu_idx(&topo_info, 1 * 3 + 0), ==,
> (1 << 2) | 0);
> - g_assert_cmpuint(x86_apicid_from_cpu_idx(1, 6, 3, 1 * 3 + 1), ==,
> + g_assert_cmpuint(x86_apicid_from_cpu_idx(&topo_info, 1 * 3 + 1), ==,
> (1 << 2) | 1);
> - g_assert_cmpuint(x86_apicid_from_cpu_idx(1, 6, 3, 1 * 3 + 2), ==,
> + g_assert_cmpuint(x86_apicid_from_cpu_idx(&topo_info, 1 * 3 + 2), ==,
> (1 << 2) | 2);
>
> - g_assert_cmpuint(x86_apicid_from_cpu_idx(1, 6, 3, 2 * 3 + 0), ==,
> + g_assert_cmpuint(x86_apicid_from_cpu_idx(&topo_info, 2 * 3 + 0), ==,
> (2 << 2) | 0);
> - g_assert_cmpuint(x86_apicid_from_cpu_idx(1, 6, 3, 2 * 3 + 1), ==,
> + g_assert_cmpuint(x86_apicid_from_cpu_idx(&topo_info, 2 * 3 + 1), ==,
> (2 << 2) | 1);
> - g_assert_cmpuint(x86_apicid_from_cpu_idx(1, 6, 3, 2 * 3 + 2), ==,
> + g_assert_cmpuint(x86_apicid_from_cpu_idx(&topo_info, 2 * 3 + 2), ==,
> (2 << 2) | 2);
>
> - g_assert_cmpuint(x86_apicid_from_cpu_idx(1, 6, 3, 5 * 3 + 0), ==,
> + g_assert_cmpuint(x86_apicid_from_cpu_idx(&topo_info, 5 * 3 + 0), ==,
> (5 << 2) | 0);
> - g_assert_cmpuint(x86_apicid_from_cpu_idx(1, 6, 3, 5 * 3 + 1), ==,
> + g_assert_cmpuint(x86_apicid_from_cpu_idx(&topo_info, 5 * 3 + 1), ==,
> (5 << 2) | 1);
> - g_assert_cmpuint(x86_apicid_from_cpu_idx(1, 6, 3, 5 * 3 + 2), ==,
> + g_assert_cmpuint(x86_apicid_from_cpu_idx(&topo_info, 5 * 3 + 2), ==,
> (5 << 2) | 2);
>
> - g_assert_cmpuint(x86_apicid_from_cpu_idx(1, 6, 3,
> + g_assert_cmpuint(x86_apicid_from_cpu_idx(&topo_info,
> 1 * 6 * 3 + 0 * 3 + 0), ==, (1 << 5));
> - g_assert_cmpuint(x86_apicid_from_cpu_idx(1, 6, 3,
> + g_assert_cmpuint(x86_apicid_from_cpu_idx(&topo_info,
> 1 * 6 * 3 + 1 * 3 + 1), ==, (1 << 5) | (1 << 2) | 1);
> - g_assert_cmpuint(x86_apicid_from_cpu_idx(1, 6, 3,
> + g_assert_cmpuint(x86_apicid_from_cpu_idx(&topo_info,
> 3 * 6 * 3 + 5 * 3 + 2), ==, (3 << 5) | (5 << 2) | 2);
> }
>
>
^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH v7 10/13] i386: Introduce use_epyc_apic_id_encoding in X86CPUDefinition
2020-03-11 22:53 ` [PATCH v7 10/13] i386: Introduce use_epyc_apic_id_encoding in X86CPUDefinition Babu Moger
@ 2020-03-12 12:26 ` Igor Mammedov
0 siblings, 0 replies; 38+ messages in thread
From: Igor Mammedov @ 2020-03-12 12:26 UTC (permalink / raw)
To: Babu Moger; +Cc: ehabkost, mst, qemu-devel, pbonzini, rth
On Wed, 11 Mar 2020 17:53:55 -0500
Babu Moger <babu.moger@amd.com> wrote:
> Add a boolean variable use_epyc_apic_id_encoding in X86CPUDefinition.
> This will be set if this cpu model needs to use new EPYC based
> apic id encoding.
>
> Override the handlers with EPYC based handlers if use_epyc_apic_id_encoding
> is set. This will be done in x86_cpus_init.
>
> Signed-off-by: Babu Moger <babu.moger@amd.com>
Acked-by: Igor Mammedov <imammedo@redhat.com>
> ---
> target/i386/cpu.c | 16 ++++++++++++++++
> target/i386/cpu.h | 1 +
> 2 files changed, 17 insertions(+)
>
> diff --git a/target/i386/cpu.c b/target/i386/cpu.c
> index 7361a53166..1e4400df7a 100644
> --- a/target/i386/cpu.c
> +++ b/target/i386/cpu.c
> @@ -1614,6 +1614,10 @@ typedef struct X86CPUDefinition {
> FeatureWordArray features;
> const char *model_id;
> CPUCaches *cache_info;
> +
> + /* Use AMD EPYC encoding for apic id */
> + bool use_epyc_apic_id_encoding;
> +
> /*
> * Definitions for alternative versions of CPU model.
> * List is terminated by item with version == 0.
> @@ -1655,6 +1659,18 @@ static const X86CPUVersionDefinition *x86_cpu_def_get_versions(X86CPUDefinition
> return def->versions ?: default_version_list;
> }
>
> +bool cpu_x86_use_epyc_apic_id_encoding(const char *cpu_type)
> +{
> + X86CPUClass *xcc = X86_CPU_CLASS(object_class_by_name(cpu_type));
> +
> + assert(xcc);
> + if (xcc->model && xcc->model->cpudef) {
> + return xcc->model->cpudef->use_epyc_apic_id_encoding;
> + } else {
> + return false;
> + }
> +}
> +
> static CPUCaches epyc_cache_info = {
> .l1d_cache = &(CPUCacheInfo) {
> .type = DATA_CACHE,
> diff --git a/target/i386/cpu.h b/target/i386/cpu.h
> index 7e9e963d78..6e522fcd34 100644
> --- a/target/i386/cpu.h
> +++ b/target/i386/cpu.h
> @@ -1897,6 +1897,7 @@ void cpu_clear_apic_feature(CPUX86State *env);
> void host_cpuid(uint32_t function, uint32_t count,
> uint32_t *eax, uint32_t *ebx, uint32_t *ecx, uint32_t *edx);
> void host_vendor_fms(char *vendor, int *family, int *model, int *stepping);
> +bool cpu_x86_use_epyc_apic_id_encoding(const char *cpu_type);
>
> /* helper.c */
> bool x86_cpu_tlb_fill(CPUState *cs, vaddr address, int size,
>
>
^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH v7 11/13] hw/i386: Move arch_id decode inside x86_cpus_init
2020-03-11 22:54 ` [PATCH v7 11/13] hw/i386: Move arch_id decode inside x86_cpus_init Babu Moger
@ 2020-03-12 12:31 ` Igor Mammedov
0 siblings, 0 replies; 38+ messages in thread
From: Igor Mammedov @ 2020-03-12 12:31 UTC (permalink / raw)
To: Babu Moger; +Cc: ehabkost, mst, qemu-devel, pbonzini, rth
On Wed, 11 Mar 2020 17:54:02 -0500
Babu Moger <babu.moger@amd.com> wrote:
> Apicid calculation depends on knowing the total number of numa nodes
> for EPYC cpu models. Right now, we are calculating the arch_id while
> parsing the numa(parse_numa). At this time, it is not known how many
> total numa nodes are configured in the system.
>
> Move the arch_id calculation inside x86_cpus_init. At this time, smp
> parse is already completed and numa node information is available.
>
> Override the handlers if use_epyc_apic_id_encoding is enabled in
> cpu model definition.
>
> Also replace the calling convention to use handlers from
> X86MachineState.
>
> Signed-off-by: Babu Moger <babu.moger@amd.com>
Reviewed-by: Igor Mammedov <imammedo@redhat.com>
> ---
> hw/i386/pc.c | 6 +++---
> hw/i386/x86.c | 37 ++++++++++++++++++++++++++++++-------
> 2 files changed, 33 insertions(+), 10 deletions(-)
>
> diff --git a/hw/i386/pc.c b/hw/i386/pc.c
> index 98ee763f68..2d7d611184 100644
> --- a/hw/i386/pc.c
> +++ b/hw/i386/pc.c
> @@ -1580,14 +1580,14 @@ static void pc_cpu_pre_plug(HotplugHandler *hotplug_dev,
> topo_ids.die_id = cpu->die_id;
> topo_ids.core_id = cpu->core_id;
> topo_ids.smt_id = cpu->thread_id;
> - cpu->apic_id = x86_apicid_from_topo_ids(&topo_info, &topo_ids);
> + cpu->apic_id = x86ms->apicid_from_topo_ids(&topo_info, &topo_ids);
> }
>
> cpu_slot = pc_find_cpu_slot(MACHINE(pcms), cpu->apic_id, &idx);
> if (!cpu_slot) {
> MachineState *ms = MACHINE(pcms);
>
> - x86_topo_ids_from_apicid(cpu->apic_id, &topo_info, &topo_ids);
> + x86ms->topo_ids_from_apicid(cpu->apic_id, &topo_info, &topo_ids);
> error_setg(errp,
> "Invalid CPU [socket: %u, die: %u, core: %u, thread: %u] with"
> " APIC ID %" PRIu32 ", valid index range 0:%d",
> @@ -1608,7 +1608,7 @@ static void pc_cpu_pre_plug(HotplugHandler *hotplug_dev,
> /* TODO: move socket_id/core_id/thread_id checks into x86_cpu_realizefn()
> * once -smp refactoring is complete and there will be CPU private
> * CPUState::nr_cores and CPUState::nr_threads fields instead of globals */
> - x86_topo_ids_from_apicid(cpu->apic_id, &topo_info, &topo_ids);
> + x86ms->topo_ids_from_apicid(cpu->apic_id, &topo_info, &topo_ids);
> if (cpu->socket_id != -1 && cpu->socket_id != topo_ids.pkg_id) {
> error_setg(errp, "property socket-id: %u doesn't match set apic-id:"
> " 0x%x (socket-id: %u)", cpu->socket_id, cpu->apic_id,
> diff --git a/hw/i386/x86.c b/hw/i386/x86.c
> index 0a81ab5151..023dce1dbd 100644
> --- a/hw/i386/x86.c
> +++ b/hw/i386/x86.c
> @@ -68,6 +68,22 @@ inline void init_topo_info(X86CPUTopoInfo *topo_info,
> topo_info->threads_per_core = ms->smp.threads;
> }
>
> +/*
> + * Set up with the new EPYC topology handlers
> + *
> + * AMD uses different apic id encoding for EPYC based cpus. Override
> + * the default topo handlers with EPYC encoding handlers.
> + */
> +static void x86_set_epyc_topo_handlers(MachineState *machine)
> +{
> + X86MachineState *x86ms = X86_MACHINE(machine);
> +
> + x86ms->apicid_from_cpu_idx = x86_apicid_from_cpu_idx_epyc;
> + x86ms->topo_ids_from_apicid = x86_topo_ids_from_apicid_epyc;
> + x86ms->apicid_from_topo_ids = x86_apicid_from_topo_ids_epyc;
> + x86ms->apicid_pkg_offset = apicid_pkg_offset_epyc;
> +}
> +
> /*
> * Calculates initial APIC ID for a specific CPU index
> *
> @@ -86,7 +102,7 @@ uint32_t x86_cpu_apic_id_from_index(X86MachineState *x86ms,
>
> init_topo_info(&topo_info, x86ms);
>
> - correct_id = x86_apicid_from_cpu_idx(&topo_info, cpu_index);
> + correct_id = x86ms->apicid_from_cpu_idx(&topo_info, cpu_index);
> if (x86mc->compat_apic_id_mode) {
> if (cpu_index != correct_id && !warned && !qtest_enabled()) {
> error_report("APIC IDs set in compatibility mode, "
> @@ -121,6 +137,11 @@ void x86_cpus_init(X86MachineState *x86ms, int default_cpu_version)
> MachineState *ms = MACHINE(x86ms);
> MachineClass *mc = MACHINE_GET_CLASS(x86ms);
>
> + /* Check for apicid encoding */
> + if (cpu_x86_use_epyc_apic_id_encoding(ms->cpu_type)) {
> + x86_set_epyc_topo_handlers(ms);
> + }
> +
> x86_cpu_set_default_version(default_cpu_version);
>
> /*
> @@ -134,6 +155,12 @@ void x86_cpus_init(X86MachineState *x86ms, int default_cpu_version)
> x86ms->apic_id_limit = x86_cpu_apic_id_from_index(x86ms,
> ms->smp.max_cpus - 1) + 1;
> possible_cpus = mc->possible_cpu_arch_ids(ms);
> +
> + for (i = 0; i < ms->smp.cpus; i++) {
> + ms->possible_cpus->cpus[i].arch_id =
> + x86_cpu_apic_id_from_index(x86ms, i);
> + }
> +
> for (i = 0; i < ms->smp.cpus; i++) {
> x86_cpu_new(x86ms, possible_cpus->cpus[i].arch_id, &error_fatal);
> }
> @@ -158,8 +185,7 @@ int64_t x86_get_default_cpu_node_id(const MachineState *ms, int idx)
> init_topo_info(&topo_info, x86ms);
>
> assert(idx < ms->possible_cpus->len);
> - x86_topo_ids_from_apicid(ms->possible_cpus->cpus[idx].arch_id,
> - &topo_info, &topo_ids);
> + x86_topo_ids_from_idx(&topo_info, idx, &topo_ids);
> return topo_ids.pkg_id % ms->numa_state->num_nodes;
> }
>
> @@ -190,10 +216,7 @@ const CPUArchIdList *x86_possible_cpu_arch_ids(MachineState *ms)
>
> ms->possible_cpus->cpus[i].type = ms->cpu_type;
> ms->possible_cpus->cpus[i].vcpus_count = 1;
> - ms->possible_cpus->cpus[i].arch_id =
> - x86_cpu_apic_id_from_index(x86ms, i);
> - x86_topo_ids_from_apicid(ms->possible_cpus->cpus[i].arch_id,
> - &topo_info, &topo_ids);
> + x86_topo_ids_from_idx(&topo_info, i, &topo_ids);
> ms->possible_cpus->cpus[i].props.has_socket_id = true;
> ms->possible_cpus->cpus[i].props.socket_id = topo_ids.pkg_id;
> if (x86ms->smp_dies > 1) {
>
^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH v7 07/13] hw/386: Add EPYC mode topology decoding functions
2020-03-11 22:53 ` [PATCH v7 07/13] hw/386: Add EPYC mode topology decoding functions Babu Moger
@ 2020-03-12 12:39 ` Igor Mammedov
2020-03-12 13:44 ` Babu Moger
2020-06-02 17:18 ` Eduardo Habkost
1 sibling, 1 reply; 38+ messages in thread
From: Igor Mammedov @ 2020-03-12 12:39 UTC (permalink / raw)
To: Babu Moger; +Cc: ehabkost, mst, qemu-devel, pbonzini, rth
On Wed, 11 Mar 2020 17:53:34 -0500
Babu Moger <babu.moger@amd.com> wrote:
> These functions add support for building EPYC mode topology given the smp
> details like numa nodes, cores, threads and sockets.
>
> The new apic id decoding is mostly similar to current apic id decoding
> except that it adds a new field node_id when numa configured. Removes all
> the hardcoded values. Subsequent patches will use these functions to build
> the topology.
>
> Following functions are added.
[...]
> x86_topo_ids_from_idx_epyc
you forgot to remove unused anymore function.
No need to respin whole series for it though, you can post as reply to
this patch v8 or do it as a patch on top.
[...]
>
> The topology details are available in Processor Programming Reference (PPR)
> for AMD Family 17h Model 01h, Revision B1 Processors. The revision guides are
> available from the bugzilla Link below.
> Link: https://bugzilla.kernel.org/show_bug.cgi?id=206537
>
> Signed-off-by: Babu Moger <babu.moger@amd.com>
> Acked-by: Igor Mammedov <imammedo@redhat.com>
> Acked-by: Michael S. Tsirkin <mst@redhat.com>
[...]
> +static inline void x86_topo_ids_from_idx_epyc(X86CPUTopoInfo *topo_info,
> + unsigned cpu_index,
> + X86CPUTopoIDs *topo_ids)
> +{
> + unsigned nr_nodes = MAX(topo_info->nodes_per_pkg, 1);
> + unsigned nr_dies = topo_info->dies_per_pkg;
> + unsigned nr_cores = topo_info->cores_per_die;
> + unsigned nr_threads = topo_info->threads_per_core;
> + unsigned cores_per_node = DIV_ROUND_UP((nr_dies * nr_cores * nr_threads),
> + nr_nodes);
> +
> + topo_ids->pkg_id = cpu_index / (nr_dies * nr_cores * nr_threads);
> + topo_ids->node_id = (cpu_index / cores_per_node) % nr_nodes;
> + topo_ids->die_id = cpu_index / (nr_cores * nr_threads) % nr_dies;
> + topo_ids->core_id = cpu_index / nr_threads % nr_cores;
> + topo_ids->smt_id = cpu_index % nr_threads;
> +}
> +
> +/*
[...]
^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH v7 07/13] hw/386: Add EPYC mode topology decoding functions
2020-03-12 12:39 ` Igor Mammedov
@ 2020-03-12 13:44 ` Babu Moger
0 siblings, 0 replies; 38+ messages in thread
From: Babu Moger @ 2020-03-12 13:44 UTC (permalink / raw)
To: Igor Mammedov; +Cc: ehabkost, mst, qemu-devel, pbonzini, rth
On 3/12/20 7:39 AM, Igor Mammedov wrote:
> On Wed, 11 Mar 2020 17:53:34 -0500
> Babu Moger <babu.moger@amd.com> wrote:
>
>> These functions add support for building EPYC mode topology given the smp
>> details like numa nodes, cores, threads and sockets.
>>
>> The new apic id decoding is mostly similar to current apic id decoding
>> except that it adds a new field node_id when numa configured. Removes all
>> the hardcoded values. Subsequent patches will use these functions to build
>> the topology.
>>
>> Following functions are added.
> [...]
>> x86_topo_ids_from_idx_epyc
> you forgot to remove unused anymore function.
> No need to respin whole series for it though, you can post as reply to
> this patch v8 or do it as a patch on top.
Igor, The function x86_topo_ids_from_idx_epyc(or x86_topo_ids_from_idx) is
still there. We are using it internally now. It is used by
x86_apicid_from_cpu_idx_epyc(or x86_apicid_from_cpu_idx). We removed it as
callback function. So, we are good here. Thanks
>
>
>
>
> [...]
>>
>> The topology details are available in Processor Programming Reference (PPR)
>> for AMD Family 17h Model 01h, Revision B1 Processors. The revision guides are
>> available from the bugzilla Link below.
>> Link: https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugzilla.kernel.org%2Fshow_bug.cgi%3Fid%3D206537&data=02%7C01%7Cbabu.moger%40amd.com%7C3d1032fb1cc94a5a197308d7c68268c5%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637196135715465573&sdata=13zSN7AqPGKHFG%2FePmkWTVbwM0qzktrnolEidnNzyhs%3D&reserved=0
>>
>> Signed-off-by: Babu Moger <babu.moger@amd.com>
>> Acked-by: Igor Mammedov <imammedo@redhat.com>
>> Acked-by: Michael S. Tsirkin <mst@redhat.com>
> [...]
>> +static inline void x86_topo_ids_from_idx_epyc(X86CPUTopoInfo *topo_info,
>> + unsigned cpu_index,
>> + X86CPUTopoIDs *topo_ids)
>> +{
>> + unsigned nr_nodes = MAX(topo_info->nodes_per_pkg, 1);
>> + unsigned nr_dies = topo_info->dies_per_pkg;
>> + unsigned nr_cores = topo_info->cores_per_die;
>> + unsigned nr_threads = topo_info->threads_per_core;
>> + unsigned cores_per_node = DIV_ROUND_UP((nr_dies * nr_cores * nr_threads),
>> + nr_nodes);
>> +
>> + topo_ids->pkg_id = cpu_index / (nr_dies * nr_cores * nr_threads);
>> + topo_ids->node_id = (cpu_index / cores_per_node) % nr_nodes;
>> + topo_ids->die_id = cpu_index / (nr_cores * nr_threads) % nr_dies;
>> + topo_ids->core_id = cpu_index / nr_threads % nr_cores;
>> + topo_ids->smt_id = cpu_index % nr_threads;
>> +}
>> +
>> +/*
> [...]
>
^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH v7 00/13] APIC ID fixes for AMD EPYC CPU model
2020-03-11 22:52 [PATCH v7 00/13] APIC ID fixes for AMD EPYC CPU model Babu Moger
` (12 preceding siblings ...)
2020-03-11 22:54 ` [PATCH v7 13/13] i386: Fix pkg_id offset for EPYC cpu models Babu Moger
@ 2020-03-12 16:28 ` Babu Moger
2020-03-17 23:22 ` Eduardo Habkost
13 siblings, 1 reply; 38+ messages in thread
From: Babu Moger @ 2020-03-12 16:28 UTC (permalink / raw)
To: ehabkost, marcel.apfelbaum, pbonzini, rth, mst, imammedo; +Cc: qemu-devel
Eduardo, Can you please queue the series if there are no concerns.
Thanks
On 3/11/20 5:52 PM, Babu Moger wrote:
> This series fixes APIC ID encoding problem reported on AMD EPYC cpu models.
> https://bugzilla.redhat.com/show_bug.cgi?id=1728166
>
> Currently, the APIC ID is decoded based on the sequence
> sockets->dies->cores->threads. This works for most standard AMD and other
> vendors' configurations, but this decoding sequence does not follow that of
> AMD's APIC ID enumeration strictly. In some cases this can cause CPU topology
> inconsistency. When booting a guest VM, the kernel tries to validate the
> topology, and finds it inconsistent with the enumeration of EPYC cpu models.
>
> To fix the problem we need to build the topology as per the Processor
> Programming Reference (PPR) for AMD Family 17h Model 01h, Revision B1
> Processors. The documentation is available from the bugzilla Link below.
>
> Link: https://bugzilla.kernel.org/show_bug.cgi?id=206537
>
> Here is the text from the PPR.
> Operating systems are expected to use Core::X86::Cpuid::SizeId[ApicIdSize], the
> number of least significant bits in the Initial APIC ID that indicate core ID
> within a processor, in constructing per-core CPUID masks.
> Core::X86::Cpuid::SizeId[ApicIdSize] determines the maximum number of cores
> (MNC) that the processor could theoretically support, not the actual number of
> cores that are actually implemented or enabled on the processor, as indicated
> by Core::X86::Cpuid::SizeId[NC].
> Each Core::X86::Apic::ApicId[ApicId] register is preset as follows:
> • ApicId[6] = Socket ID.
> • ApicId[5:4] = Node ID.
> • ApicId[3] = Logical CCX L3 complex ID
> • ApicId[2:0]= (SMT) ? {LogicalCoreID[1:0],ThreadId} : {1'b0,LogicalCoreID[1:0]}
>
> v7:
> Generated the patches on top of git://github.com/ehabkost/qemu.git (x86-next).
> Changes from v6.
> 1. Added new function x86_set_epyc_topo_handlers to override the apic id
> encoding handlers.
> 2. Separated the code to set use_epyc_apic_id_encoding and added as a new patch
> as it looked more logical.
> 3. Fixed minor typos.
>
> v6:
> https://lore.kernel.org/qemu-devel/158389385028.22020.7608244627303132902.stgit@naples-babu.amd.com/
> Generated the patches on top of git://github.com/ehabkost/qemu.git (x86-next).
> Changes from v5.
> 1. Eduardo has already queued couple of patches, submitting the rest here.
> 2. Major change is how the EPYC mode apic id encoding handlers are loaded.
> Added a boolean variable use_epyc_apic_id_encoding in X86CPUDefinition.
> The variable is will be used to tell if we need to use EPYC mode encoding.
> 3. Eduardo reported bysectability problem with x86 unit test code.
> Quashed the patches in 1 and 2 to resolve it. Problem was change in calling
> conventions of topology related functions.
> 4. Also set the use_epyc_apic_id_encoding for EPYC-Rome. This model is
> added recently to the cpu table.
>
> v5:
> https://lore.kernel.org/qemu-devel/158326531474.40452.11433722850425537745.stgit@naples-babu.amd.com/
> Generated the patches on top of git://github.com/ehabkost/qemu.git (x86-next).
> Changes from v4.
> 1. Re-arranged the patches 2 and 4 as suggested by Igor.
> 2. Kept the apicid handler functions inside X86MachineState as discussed.
> These handlers are loaded from X86CPUDefinitions.
> 3. Removed unnecessary X86CPUstate initialization from x86_cpu_new. Suggested
> by Igor.
> 4. And other minor changes related to patch format.
>
> v4:
> https://lore.kernel.org/qemu-devel/158161767653.48948.10578064482878399556.stgit@naples-babu.amd.com/
> Changes from v3.
> 1. Moved the arch_id calculation inside the function x86_cpus_init. With this change,
> we dont need to change common numa code.(suggested by Igor)
> 2. Introduced the model specific handlers inside X86CPUDefinitions.
> These handlers are loaded into X86MachineState during the init.
> 3. Removed llc_id from x86CPU.
> 4. Removed init_apicid_fn hanlder from MachineClass. Kept all the code changes
> inside the x86.
> 5. Added new handler function apicid_pkg_offset for pkg_offset calculation.
> 6. And some Other minor changes.
>
> v3:
> https://lore.kernel.org/qemu-devel/157541968844.46157.17994918142533791313.stgit@naples-babu.amd.com/
> 1. Consolidated the topology information in structure X86CPUTopoInfo.
> 2. Changed the ccx_id to llc_id as commented by upstream.
> 3. Generalized the apic id decoding. It is mostly similar to current apic id
> except that it adds new field llc_id when numa configured. Removes all the
> hardcoded values.
> 4. Removed the earlier parse_numa split. And moved the numa node initialization
> inside the numa_complete_configuration. This is bit cleaner as commented by
> Eduardo.
> 5. Added new function init_apicid_fn inside machine_class structure. This
> will be used to update the apic id handler specific to cpu model.
> 6. Updated the cpuid unit tests.
> 7. TODO : Need to figure out how to dynamically update the handlers using cpu models.
> I might some guidance on that.
>
> v2:
> https://lore.kernel.org/qemu-devel/156779689013.21957.1631551572950676212.stgit@localhost.localdomain/
> 1. Introduced the new property epyc to enable new epyc mode.
> 2. Separated the epyc mode and non epyc mode function.
> 3. Introduced function pointers in PCMachineState to handle the
> differences.
> 4. Mildly tested different combinations to make things are working as expected.
> 5. TODO : Setting the epyc feature bit needs to be worked out. This feature is
> supported only on AMD EPYC models. I may need some guidance on that.
>
> v1:
> https://lore.kernel.org/qemu-devel/20190731232032.51786-1-babu.moger@amd.com/
> ---
>
> Babu Moger (13):
> hw/i386: Introduce X86CPUTopoInfo to contain topology info
> hw/i386: Consolidate topology functions
> machine: Add SMP Sockets in CpuTopology
> hw/i386: Remove unnecessary initialization in x86_cpu_new
> hw/i386: Update structures to save the number of nodes per package
> hw/i386: Rename apicid_from_topo_ids to x86_apicid_from_topo_ids
> hw/386: Add EPYC mode topology decoding functions
> target/i386: Cleanup and use the EPYC mode topology functions
> hw/i386: Introduce apicid functions inside X86MachineState
> i386: Introduce use_epyc_apic_id_encoding in X86CPUDefinition
> hw/i386: Move arch_id decode inside x86_cpus_init
> target/i386: Enable new apic id encoding for EPYC based cpus models
> i386: Fix pkg_id offset for EPYC cpu models
>
>
> hw/core/machine.c | 1
> hw/i386/pc.c | 15 ++-
> hw/i386/x86.c | 73 ++++++++++++----
> include/hw/boards.h | 2
> include/hw/i386/topology.h | 195 ++++++++++++++++++++++++++++++------------
> include/hw/i386/x86.h | 12 +++
> softmmu/vl.c | 1
> target/i386/cpu.c | 203 ++++++++++++++------------------------------
> target/i386/cpu.h | 3 +
> tests/test-x86-cpuid.c | 116 +++++++++++++++----------
> 10 files changed, 358 insertions(+), 263 deletions(-)
>
> --
> Signature
>
^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH v7 00/13] APIC ID fixes for AMD EPYC CPU model
2020-03-12 16:28 ` [PATCH v7 00/13] APIC ID fixes for AMD EPYC CPU model Babu Moger
@ 2020-03-17 23:22 ` Eduardo Habkost
2020-03-17 23:31 ` Moger, Babu
2020-03-17 23:46 ` Eduardo Habkost
0 siblings, 2 replies; 38+ messages in thread
From: Eduardo Habkost @ 2020-03-17 23:22 UTC (permalink / raw)
To: Babu Moger; +Cc: mst, qemu-devel, imammedo, pbonzini, rth
On Thu, Mar 12, 2020 at 11:28:47AM -0500, Babu Moger wrote:
> Eduardo, Can you please queue the series if there are no concerns.
> Thanks
I had queued it for today's pull request, but it looks like it
breaks "make check". See https://travis-ci.org/github/ehabkost/qemu/jobs/663529282
PASS 4 bios-tables-test /x86_64/acpi/piix4/ipmi
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: -accel kvm: failed to initialize kvm: No such file or directory
qemu-system-x86_64: falling back to tcg
qemu-system-x86_64: Invalid CPU [socket: 0, die: 0, core: 1, thread: 0] with APIC ID 1, valid index range 0:5
Broken pipe
/home/travis/build/ehabkost/qemu/tests/qtest/libqtest.c:166: kill_qemu() tried to terminate QEMU process but encountered exit status 1 (expected 0)
Aborted (core dumped)
ERROR - too few tests run (expected 17, got 4)
/home/travis/build/ehabkost/qemu/tests/Makefile.include:633: recipe for target 'check-qtest-x86_64' failed
make: *** [check-qtest-x86_64] Error 1
>
> On 3/11/20 5:52 PM, Babu Moger wrote:
> > This series fixes APIC ID encoding problem reported on AMD EPYC cpu models.
> > https://bugzilla.redhat.com/show_bug.cgi?id=1728166
> >
> > Currently, the APIC ID is decoded based on the sequence
> > sockets->dies->cores->threads. This works for most standard AMD and other
> > vendors' configurations, but this decoding sequence does not follow that of
> > AMD's APIC ID enumeration strictly. In some cases this can cause CPU topology
> > inconsistency. When booting a guest VM, the kernel tries to validate the
> > topology, and finds it inconsistent with the enumeration of EPYC cpu models.
> >
> > To fix the problem we need to build the topology as per the Processor
> > Programming Reference (PPR) for AMD Family 17h Model 01h, Revision B1
> > Processors. The documentation is available from the bugzilla Link below.
> >
> > Link: https://bugzilla.kernel.org/show_bug.cgi?id=206537
> >
> > Here is the text from the PPR.
> > Operating systems are expected to use Core::X86::Cpuid::SizeId[ApicIdSize], the
> > number of least significant bits in the Initial APIC ID that indicate core ID
> > within a processor, in constructing per-core CPUID masks.
> > Core::X86::Cpuid::SizeId[ApicIdSize] determines the maximum number of cores
> > (MNC) that the processor could theoretically support, not the actual number of
> > cores that are actually implemented or enabled on the processor, as indicated
> > by Core::X86::Cpuid::SizeId[NC].
> > Each Core::X86::Apic::ApicId[ApicId] register is preset as follows:
> > • ApicId[6] = Socket ID.
> > • ApicId[5:4] = Node ID.
> > • ApicId[3] = Logical CCX L3 complex ID
> > • ApicId[2:0]= (SMT) ? {LogicalCoreID[1:0],ThreadId} : {1'b0,LogicalCoreID[1:0]}
> >
> > v7:
> > Generated the patches on top of git://github.com/ehabkost/qemu.git (x86-next).
> > Changes from v6.
> > 1. Added new function x86_set_epyc_topo_handlers to override the apic id
> > encoding handlers.
> > 2. Separated the code to set use_epyc_apic_id_encoding and added as a new patch
> > as it looked more logical.
> > 3. Fixed minor typos.
> >
> > v6:
> > https://lore.kernel.org/qemu-devel/158389385028.22020.7608244627303132902.stgit@naples-babu.amd.com/
> > Generated the patches on top of git://github.com/ehabkost/qemu.git (x86-next).
> > Changes from v5.
> > 1. Eduardo has already queued couple of patches, submitting the rest here.
> > 2. Major change is how the EPYC mode apic id encoding handlers are loaded.
> > Added a boolean variable use_epyc_apic_id_encoding in X86CPUDefinition.
> > The variable is will be used to tell if we need to use EPYC mode encoding.
> > 3. Eduardo reported bysectability problem with x86 unit test code.
> > Quashed the patches in 1 and 2 to resolve it. Problem was change in calling
> > conventions of topology related functions.
> > 4. Also set the use_epyc_apic_id_encoding for EPYC-Rome. This model is
> > added recently to the cpu table.
> >
> > v5:
> > https://lore.kernel.org/qemu-devel/158326531474.40452.11433722850425537745.stgit@naples-babu.amd.com/
> > Generated the patches on top of git://github.com/ehabkost/qemu.git (x86-next).
> > Changes from v4.
> > 1. Re-arranged the patches 2 and 4 as suggested by Igor.
> > 2. Kept the apicid handler functions inside X86MachineState as discussed.
> > These handlers are loaded from X86CPUDefinitions.
> > 3. Removed unnecessary X86CPUstate initialization from x86_cpu_new. Suggested
> > by Igor.
> > 4. And other minor changes related to patch format.
> >
> > v4:
> > https://lore.kernel.org/qemu-devel/158161767653.48948.10578064482878399556.stgit@naples-babu.amd.com/
> > Changes from v3.
> > 1. Moved the arch_id calculation inside the function x86_cpus_init. With this change,
> > we dont need to change common numa code.(suggested by Igor)
> > 2. Introduced the model specific handlers inside X86CPUDefinitions.
> > These handlers are loaded into X86MachineState during the init.
> > 3. Removed llc_id from x86CPU.
> > 4. Removed init_apicid_fn hanlder from MachineClass. Kept all the code changes
> > inside the x86.
> > 5. Added new handler function apicid_pkg_offset for pkg_offset calculation.
> > 6. And some Other minor changes.
> >
> > v3:
> > https://lore.kernel.org/qemu-devel/157541968844.46157.17994918142533791313.stgit@naples-babu.amd.com/
> > 1. Consolidated the topology information in structure X86CPUTopoInfo.
> > 2. Changed the ccx_id to llc_id as commented by upstream.
> > 3. Generalized the apic id decoding. It is mostly similar to current apic id
> > except that it adds new field llc_id when numa configured. Removes all the
> > hardcoded values.
> > 4. Removed the earlier parse_numa split. And moved the numa node initialization
> > inside the numa_complete_configuration. This is bit cleaner as commented by
> > Eduardo.
> > 5. Added new function init_apicid_fn inside machine_class structure. This
> > will be used to update the apic id handler specific to cpu model.
> > 6. Updated the cpuid unit tests.
> > 7. TODO : Need to figure out how to dynamically update the handlers using cpu models.
> > I might some guidance on that.
> >
> > v2:
> > https://lore.kernel.org/qemu-devel/156779689013.21957.1631551572950676212.stgit@localhost.localdomain/
> > 1. Introduced the new property epyc to enable new epyc mode.
> > 2. Separated the epyc mode and non epyc mode function.
> > 3. Introduced function pointers in PCMachineState to handle the
> > differences.
> > 4. Mildly tested different combinations to make things are working as expected.
> > 5. TODO : Setting the epyc feature bit needs to be worked out. This feature is
> > supported only on AMD EPYC models. I may need some guidance on that.
> >
> > v1:
> > https://lore.kernel.org/qemu-devel/20190731232032.51786-1-babu.moger@amd.com/
> > ---
> >
> > Babu Moger (13):
> > hw/i386: Introduce X86CPUTopoInfo to contain topology info
> > hw/i386: Consolidate topology functions
> > machine: Add SMP Sockets in CpuTopology
> > hw/i386: Remove unnecessary initialization in x86_cpu_new
> > hw/i386: Update structures to save the number of nodes per package
> > hw/i386: Rename apicid_from_topo_ids to x86_apicid_from_topo_ids
> > hw/386: Add EPYC mode topology decoding functions
> > target/i386: Cleanup and use the EPYC mode topology functions
> > hw/i386: Introduce apicid functions inside X86MachineState
> > i386: Introduce use_epyc_apic_id_encoding in X86CPUDefinition
> > hw/i386: Move arch_id decode inside x86_cpus_init
> > target/i386: Enable new apic id encoding for EPYC based cpus models
> > i386: Fix pkg_id offset for EPYC cpu models
> >
> >
> > hw/core/machine.c | 1
> > hw/i386/pc.c | 15 ++-
> > hw/i386/x86.c | 73 ++++++++++++----
> > include/hw/boards.h | 2
> > include/hw/i386/topology.h | 195 ++++++++++++++++++++++++++++++------------
> > include/hw/i386/x86.h | 12 +++
> > softmmu/vl.c | 1
> > target/i386/cpu.c | 203 ++++++++++++++------------------------------
> > target/i386/cpu.h | 3 +
> > tests/test-x86-cpuid.c | 116 +++++++++++++++----------
> > 10 files changed, 358 insertions(+), 263 deletions(-)
> >
> > --
> > Signature
> >
>
--
Eduardo
^ permalink raw reply [flat|nested] 38+ messages in thread
* RE: [PATCH v7 00/13] APIC ID fixes for AMD EPYC CPU model
2020-03-17 23:22 ` Eduardo Habkost
@ 2020-03-17 23:31 ` Moger, Babu
2020-03-17 23:46 ` Eduardo Habkost
1 sibling, 0 replies; 38+ messages in thread
From: Moger, Babu @ 2020-03-17 23:31 UTC (permalink / raw)
To: Eduardo Habkost; +Cc: mst, qemu-devel, imammedo, pbonzini, rth
[AMD Official Use Only - Internal Distribution Only]
Ok. I am looking at it.
> -----Original Message-----
> From: Eduardo Habkost <ehabkost@redhat.com>
> Sent: Tuesday, March 17, 2020 6:22 PM
> To: Moger, Babu <Babu.Moger@amd.com>
> Cc: marcel.apfelbaum@gmail.com; pbonzini@redhat.com; rth@twiddle.net;
> mst@redhat.com; imammedo@redhat.com; qemu-devel@nongnu.org
> Subject: Re: [PATCH v7 00/13] APIC ID fixes for AMD EPYC CPU model
>
> On Thu, Mar 12, 2020 at 11:28:47AM -0500, Babu Moger wrote:
> > Eduardo, Can you please queue the series if there are no concerns.
> > Thanks
>
> I had queued it for today's pull request, but it looks like it
> breaks "make check". See
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftravis-
> ci.org%2Fgithub%2Fehabkost%2Fqemu%2Fjobs%2F663529282&data=02%7
> C01%7Cbabu.moger%40amd.com%7Ccf8b7161fda34176840208d7caca06af%7
> C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637200841360244448&
> amp;sdata=qtBc13zinZ%2BGMT1a%2BniRJk6moGNzjgPWOJU42mL%2FZnM%3D
> &reserved=0
>
> PASS 4 bios-tables-test /x86_64/acpi/piix4/ipmi
> Could not access KVM kernel module: No such file or directory
> qemu-system-x86_64: -accel kvm: failed to initialize kvm: No such file or
> directory
> qemu-system-x86_64: falling back to tcg
> qemu-system-x86_64: Invalid CPU [socket: 0, die: 0, core: 1, thread: 0] with
> APIC ID 1, valid index range 0:5
> Broken pipe
> /home/travis/build/ehabkost/qemu/tests/qtest/libqtest.c:166: kill_qemu() tried
> to terminate QEMU process but encountered exit status 1 (expected 0)
> Aborted (core dumped)
> ERROR - too few tests run (expected 17, got 4)
> /home/travis/build/ehabkost/qemu/tests/Makefile.include:633: recipe for
> target 'check-qtest-x86_64' failed
> make: *** [check-qtest-x86_64] Error 1
>
>
> >
> > On 3/11/20 5:52 PM, Babu Moger wrote:
> > > This series fixes APIC ID encoding problem reported on AMD EPYC cpu
> models.
> > >
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugzilla.
> redhat.com%2Fshow_bug.cgi%3Fid%3D1728166&data=02%7C01%7Cbabu.
> moger%40amd.com%7Ccf8b7161fda34176840208d7caca06af%7C3dd8961fe48
> 84e608e11a82d994e183d%7C0%7C0%7C637200841360244448&sdata=uH
> 3lQsaE99WjC1kJJbnF%2FLvi1HM3rUwesp67pci5UgE%3D&reserved=0
> > >
> > > Currently, the APIC ID is decoded based on the sequence
> > > sockets->dies->cores->threads. This works for most standard AMD and other
> > > vendors' configurations, but this decoding sequence does not follow that of
> > > AMD's APIC ID enumeration strictly. In some cases this can cause CPU
> topology
> > > inconsistency. When booting a guest VM, the kernel tries to validate the
> > > topology, and finds it inconsistent with the enumeration of EPYC cpu models.
> > >
> > > To fix the problem we need to build the topology as per the Processor
> > > Programming Reference (PPR) for AMD Family 17h Model 01h, Revision B1
> > > Processors. The documentation is available from the bugzilla Link below.
> > >
> > > Link:
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugzilla.
> kernel.org%2Fshow_bug.cgi%3Fid%3D206537&data=02%7C01%7Cbabu.m
> oger%40amd.com%7Ccf8b7161fda34176840208d7caca06af%7C3dd8961fe4884
> e608e11a82d994e183d%7C0%7C0%7C637200841360244448&sdata=PECu
> xeA9RJ1Wgb5X2zjFqZHFlMQFxbU6n9PkuOvqtvA%3D&reserved=0
> > >
> > > Here is the text from the PPR.
> > > Operating systems are expected to use Core::X86::Cpuid::SizeId[ApicIdSize],
> the
> > > number of least significant bits in the Initial APIC ID that indicate core ID
> > > within a processor, in constructing per-core CPUID masks.
> > > Core::X86::Cpuid::SizeId[ApicIdSize] determines the maximum number of
> cores
> > > (MNC) that the processor could theoretically support, not the actual number
> of
> > > cores that are actually implemented or enabled on the processor, as
> indicated
> > > by Core::X86::Cpuid::SizeId[NC].
> > > Each Core::X86::Apic::ApicId[ApicId] register is preset as follows:
> > > • ApicId[6] = Socket ID.
> > > • ApicId[5:4] = Node ID.
> > > • ApicId[3] = Logical CCX L3 complex ID
> > > • ApicId[2:0]= (SMT) ? {LogicalCoreID[1:0],ThreadId} :
> {1'b0,LogicalCoreID[1:0]}
> > >
> > > v7:
> > > Generated the patches on top of git://github.com/ehabkost/qemu.git (x86-
> next).
> > > Changes from v6.
> > > 1. Added new function x86_set_epyc_topo_handlers to override the apic id
> > > encoding handlers.
> > > 2. Separated the code to set use_epyc_apic_id_encoding and added as a
> new patch
> > > as it looked more logical.
> > > 3. Fixed minor typos.
> > >
> > > v6:
> > >
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flore.ker
> nel.org%2Fqemu-
> devel%2F158389385028.22020.7608244627303132902.stgit%40naples-
> babu.amd.com%2F&data=02%7C01%7Cbabu.moger%40amd.com%7Ccf8b
> 7161fda34176840208d7caca06af%7C3dd8961fe4884e608e11a82d994e183d%7
> C0%7C0%7C637200841360244448&sdata=NmsbgY6609%2Bi2V7K310dihj%
> 2BfAQgoq1F%2Bgs%2BtMkpw1E%3D&reserved=0
> > > Generated the patches on top of git://github.com/ehabkost/qemu.git (x86-
> next).
> > > Changes from v5.
> > > 1. Eduardo has already queued couple of patches, submitting the rest here.
> > > 2. Major change is how the EPYC mode apic id encoding handlers are
> loaded.
> > > Added a boolean variable use_epyc_apic_id_encoding in
> X86CPUDefinition.
> > > The variable is will be used to tell if we need to use EPYC mode encoding.
> > > 3. Eduardo reported bysectability problem with x86 unit test code.
> > > Quashed the patches in 1 and 2 to resolve it. Problem was change in calling
> > > conventions of topology related functions.
> > > 4. Also set the use_epyc_apic_id_encoding for EPYC-Rome. This model is
> > > added recently to the cpu table.
> > >
> > > v5:
> > >
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flore.ker
> nel.org%2Fqemu-
> devel%2F158326531474.40452.11433722850425537745.stgit%40naples-
> babu.amd.com%2F&data=02%7C01%7Cbabu.moger%40amd.com%7Ccf8b
> 7161fda34176840208d7caca06af%7C3dd8961fe4884e608e11a82d994e183d%7
> C0%7C0%7C637200841360244448&sdata=7d6Kq7XTOz7CLJ9Gi%2BmBasrp
> NSuYFsCwtwi1rSczfZE%3D&reserved=0
> > > Generated the patches on top of git://github.com/ehabkost/qemu.git (x86-
> next).
> > > Changes from v4.
> > > 1. Re-arranged the patches 2 and 4 as suggested by Igor.
> > > 2. Kept the apicid handler functions inside X86MachineState as discussed.
> > > These handlers are loaded from X86CPUDefinitions.
> > > 3. Removed unnecessary X86CPUstate initialization from x86_cpu_new.
> Suggested
> > > by Igor.
> > > 4. And other minor changes related to patch format.
> > >
> > > v4:
> > >
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flore.ker
> nel.org%2Fqemu-
> devel%2F158161767653.48948.10578064482878399556.stgit%40naples-
> babu.amd.com%2F&data=02%7C01%7Cbabu.moger%40amd.com%7Ccf8b
> 7161fda34176840208d7caca06af%7C3dd8961fe4884e608e11a82d994e183d%7
> C0%7C0%7C637200841360244448&sdata=XSbNF9qgLYXqqNq7AgPXD4eKf
> 0a1el302yZ9lGS6BRg%3D&reserved=0
> > > Changes from v3.
> > > 1. Moved the arch_id calculation inside the function x86_cpus_init. With this
> change,
> > > we dont need to change common numa code.(suggested by Igor)
> > > 2. Introduced the model specific handlers inside X86CPUDefinitions.
> > > These handlers are loaded into X86MachineState during the init.
> > > 3. Removed llc_id from x86CPU.
> > > 4. Removed init_apicid_fn hanlder from MachineClass. Kept all the code
> changes
> > > inside the x86.
> > > 5. Added new handler function apicid_pkg_offset for pkg_offset calculation.
> > > 6. And some Other minor changes.
> > >
> > > v3:
> > >
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flore.ker
> nel.org%2Fqemu-
> devel%2F157541968844.46157.17994918142533791313.stgit%40naples-
> babu.amd.com%2F&data=02%7C01%7Cbabu.moger%40amd.com%7Ccf8b
> 7161fda34176840208d7caca06af%7C3dd8961fe4884e608e11a82d994e183d%7
> C0%7C0%7C637200841360244448&sdata=0pxCGXWkrov89FCmMPVGKhXc
> T1ixJu1JLhSxpdiRYTY%3D&reserved=0
> > > 1. Consolidated the topology information in structure X86CPUTopoInfo.
> > > 2. Changed the ccx_id to llc_id as commented by upstream.
> > > 3. Generalized the apic id decoding. It is mostly similar to current apic id
> > > except that it adds new field llc_id when numa configured. Removes all
> the
> > > hardcoded values.
> > > 4. Removed the earlier parse_numa split. And moved the numa node
> initialization
> > > inside the numa_complete_configuration. This is bit cleaner as
> commented by
> > > Eduardo.
> > > 5. Added new function init_apicid_fn inside machine_class structure. This
> > > will be used to update the apic id handler specific to cpu model.
> > > 6. Updated the cpuid unit tests.
> > > 7. TODO : Need to figure out how to dynamically update the handlers using
> cpu models.
> > > I might some guidance on that.
> > >
> > > v2:
> > >
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flore.ker
> nel.org%2Fqemu-
> devel%2F156779689013.21957.1631551572950676212.stgit%40localhost.locald
> omain%2F&data=02%7C01%7Cbabu.moger%40amd.com%7Ccf8b7161fda
> 34176840208d7caca06af%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0
> %7C637200841360244448&sdata=TF4F0j8lXu8uzEsTC8swVwMKx6HRzq77S
> yRm7cEG2Zs%3D&reserved=0
> > > 1. Introduced the new property epyc to enable new epyc mode.
> > > 2. Separated the epyc mode and non epyc mode function.
> > > 3. Introduced function pointers in PCMachineState to handle the
> > > differences.
> > > 4. Mildly tested different combinations to make things are working as
> expected.
> > > 5. TODO : Setting the epyc feature bit needs to be worked out. This feature
> is
> > > supported only on AMD EPYC models. I may need some guidance on that.
> > >
> > > v1:
> > >
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flore.ker
> nel.org%2Fqemu-devel%2F20190731232032.51786-1-
> babu.moger%40amd.com%2F&data=02%7C01%7Cbabu.moger%40amd.co
> m%7Ccf8b7161fda34176840208d7caca06af%7C3dd8961fe4884e608e11a82d99
> 4e183d%7C0%7C0%7C637200841360254447&sdata=U8yPTkTqonZDEuE74
> hw4tpNNvVEKdrMTM9sr1ECR6mw%3D&reserved=0
> > > ---
> > >
> > > Babu Moger (13):
> > > hw/i386: Introduce X86CPUTopoInfo to contain topology info
> > > hw/i386: Consolidate topology functions
> > > machine: Add SMP Sockets in CpuTopology
> > > hw/i386: Remove unnecessary initialization in x86_cpu_new
> > > hw/i386: Update structures to save the number of nodes per package
> > > hw/i386: Rename apicid_from_topo_ids to x86_apicid_from_topo_ids
> > > hw/386: Add EPYC mode topology decoding functions
> > > target/i386: Cleanup and use the EPYC mode topology functions
> > > hw/i386: Introduce apicid functions inside X86MachineState
> > > i386: Introduce use_epyc_apic_id_encoding in X86CPUDefinition
> > > hw/i386: Move arch_id decode inside x86_cpus_init
> > > target/i386: Enable new apic id encoding for EPYC based cpus models
> > > i386: Fix pkg_id offset for EPYC cpu models
> > >
> > >
> > > hw/core/machine.c | 1
> > > hw/i386/pc.c | 15 ++-
> > > hw/i386/x86.c | 73 ++++++++++++----
> > > include/hw/boards.h | 2
> > > include/hw/i386/topology.h | 195 ++++++++++++++++++++++++++++++-----
> -------
> > > include/hw/i386/x86.h | 12 +++
> > > softmmu/vl.c | 1
> > > target/i386/cpu.c | 203 ++++++++++++++------------------------------
> > > target/i386/cpu.h | 3 +
> > > tests/test-x86-cpuid.c | 116 +++++++++++++++----------
> > > 10 files changed, 358 insertions(+), 263 deletions(-)
> > >
> > > --
> > > Signature
> > >
> >
>
> --
> Eduardo
^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH v7 00/13] APIC ID fixes for AMD EPYC CPU model
2020-03-17 23:22 ` Eduardo Habkost
2020-03-17 23:31 ` Moger, Babu
@ 2020-03-17 23:46 ` Eduardo Habkost
2020-03-18 2:43 ` Moger, Babu
1 sibling, 1 reply; 38+ messages in thread
From: Eduardo Habkost @ 2020-03-17 23:46 UTC (permalink / raw)
To: Babu Moger; +Cc: mst, qemu-devel, imammedo, pbonzini, rth
On Tue, Mar 17, 2020 at 07:22:06PM -0400, Eduardo Habkost wrote:
> On Thu, Mar 12, 2020 at 11:28:47AM -0500, Babu Moger wrote:
> > Eduardo, Can you please queue the series if there are no concerns.
> > Thanks
>
> I had queued it for today's pull request, but it looks like it
> breaks "make check". See https://travis-ci.org/github/ehabkost/qemu/jobs/663529282
>
> PASS 4 bios-tables-test /x86_64/acpi/piix4/ipmi
> Could not access KVM kernel module: No such file or directory
> qemu-system-x86_64: -accel kvm: failed to initialize kvm: No such file or directory
> qemu-system-x86_64: falling back to tcg
> qemu-system-x86_64: Invalid CPU [socket: 0, die: 0, core: 1, thread: 0] with APIC ID 1, valid index range 0:5
> Broken pipe
> /home/travis/build/ehabkost/qemu/tests/qtest/libqtest.c:166: kill_qemu() tried to terminate QEMU process but encountered exit status 1 (expected 0)
> Aborted (core dumped)
> ERROR - too few tests run (expected 17, got 4)
> /home/travis/build/ehabkost/qemu/tests/Makefile.include:633: recipe for target 'check-qtest-x86_64' failed
> make: *** [check-qtest-x86_64] Error 1
Failure is at the /x86_64/acpi/piix4/cpuhp test case:
$ QTEST_QEMU_BINARY=x86_64-softmmu/qemu-system-x86_64 QTEST_QEMU_IMG=qemu-img tests/qtest/bios-tables-test -m=quick --verbose --debug-log
[...]
{*LOG(start):{/x86_64/acpi/piix4/cpuhp}:LOG*}
# starting QEMU: exec x86_64-softmmu/qemu-system-x86_64 -qtest unix:/tmp/qtest-2052313.sock -qtest-log /dev/null -chardev socket,path=/tmp/qtest-2052313.qmp,id=char0 -mon chardev=char0,mode=control -display none -machine pc,kernel-irqchip=off -accel kvm -accel tcg -net none -display none -smp 2,cores=3,sockets=2,maxcpus=6 -object memory-backend-ram,id=ram0,size=64M -object memory-backend-ram,id=ram1,size=64M -numa node,memdev=ram0 -numa node,memdev=ram1 -numa dist,src=0,dst=1,val=21 -drive id=hd0,if=none,file=tests/acpi-test-disk-PVjFru,format=raw -device ide-hd,drive=hd0 -accel qtest
{*LOG(message):{starting QEMU: exec x86_64-softmmu/qemu-system-x86_64 -qtest unix:/tmp/qtest-2052313.sock -qtest-log /dev/null -chardev socket,path=/tmp/qtest-2052313.qmp,id=char0 -mon chardev=char0,mode=control -display none -machine pc,kernel-irqchip=off -accel kvm -accel tcg -net none -display none -smp 2,cores=3,sockets=2,maxcpus=6 -object memory-backend-ram,id=ram0,size=64M -object memory-backend-ram,id=ram1,size=64M -numa node,memdev=ram0 -numa node,memdev=ram1 -numa dist,src=0,dst=1,val=21 -drive id=hd0,if=none,file=tests/acpi-test-disk-PVjFru,format=raw -device ide-hd,drive=hd0 -accel qtest}:LOG*}
qemu-system-x86_64: Invalid CPU [socket: 0, die: 0, core: 1, thread: 0] with APIC ID 1, valid index range 0:5
Broken pipe
>
>
> >
> > On 3/11/20 5:52 PM, Babu Moger wrote:
> > > This series fixes APIC ID encoding problem reported on AMD EPYC cpu models.
> > > https://bugzilla.redhat.com/show_bug.cgi?id=1728166
> > >
> > > Currently, the APIC ID is decoded based on the sequence
> > > sockets->dies->cores->threads. This works for most standard AMD and other
> > > vendors' configurations, but this decoding sequence does not follow that of
> > > AMD's APIC ID enumeration strictly. In some cases this can cause CPU topology
> > > inconsistency. When booting a guest VM, the kernel tries to validate the
> > > topology, and finds it inconsistent with the enumeration of EPYC cpu models.
> > >
> > > To fix the problem we need to build the topology as per the Processor
> > > Programming Reference (PPR) for AMD Family 17h Model 01h, Revision B1
> > > Processors. The documentation is available from the bugzilla Link below.
> > >
> > > Link: https://bugzilla.kernel.org/show_bug.cgi?id=206537
> > >
> > > Here is the text from the PPR.
> > > Operating systems are expected to use Core::X86::Cpuid::SizeId[ApicIdSize], the
> > > number of least significant bits in the Initial APIC ID that indicate core ID
> > > within a processor, in constructing per-core CPUID masks.
> > > Core::X86::Cpuid::SizeId[ApicIdSize] determines the maximum number of cores
> > > (MNC) that the processor could theoretically support, not the actual number of
> > > cores that are actually implemented or enabled on the processor, as indicated
> > > by Core::X86::Cpuid::SizeId[NC].
> > > Each Core::X86::Apic::ApicId[ApicId] register is preset as follows:
> > > • ApicId[6] = Socket ID.
> > > • ApicId[5:4] = Node ID.
> > > • ApicId[3] = Logical CCX L3 complex ID
> > > • ApicId[2:0]= (SMT) ? {LogicalCoreID[1:0],ThreadId} : {1'b0,LogicalCoreID[1:0]}
> > >
> > > v7:
> > > Generated the patches on top of git://github.com/ehabkost/qemu.git (x86-next).
> > > Changes from v6.
> > > 1. Added new function x86_set_epyc_topo_handlers to override the apic id
> > > encoding handlers.
> > > 2. Separated the code to set use_epyc_apic_id_encoding and added as a new patch
> > > as it looked more logical.
> > > 3. Fixed minor typos.
> > >
> > > v6:
> > > https://lore.kernel.org/qemu-devel/158389385028.22020.7608244627303132902.stgit@naples-babu.amd.com/
> > > Generated the patches on top of git://github.com/ehabkost/qemu.git (x86-next).
> > > Changes from v5.
> > > 1. Eduardo has already queued couple of patches, submitting the rest here.
> > > 2. Major change is how the EPYC mode apic id encoding handlers are loaded.
> > > Added a boolean variable use_epyc_apic_id_encoding in X86CPUDefinition.
> > > The variable is will be used to tell if we need to use EPYC mode encoding.
> > > 3. Eduardo reported bysectability problem with x86 unit test code.
> > > Quashed the patches in 1 and 2 to resolve it. Problem was change in calling
> > > conventions of topology related functions.
> > > 4. Also set the use_epyc_apic_id_encoding for EPYC-Rome. This model is
> > > added recently to the cpu table.
> > >
> > > v5:
> > > https://lore.kernel.org/qemu-devel/158326531474.40452.11433722850425537745.stgit@naples-babu.amd.com/
> > > Generated the patches on top of git://github.com/ehabkost/qemu.git (x86-next).
> > > Changes from v4.
> > > 1. Re-arranged the patches 2 and 4 as suggested by Igor.
> > > 2. Kept the apicid handler functions inside X86MachineState as discussed.
> > > These handlers are loaded from X86CPUDefinitions.
> > > 3. Removed unnecessary X86CPUstate initialization from x86_cpu_new. Suggested
> > > by Igor.
> > > 4. And other minor changes related to patch format.
> > >
> > > v4:
> > > https://lore.kernel.org/qemu-devel/158161767653.48948.10578064482878399556.stgit@naples-babu.amd.com/
> > > Changes from v3.
> > > 1. Moved the arch_id calculation inside the function x86_cpus_init. With this change,
> > > we dont need to change common numa code.(suggested by Igor)
> > > 2. Introduced the model specific handlers inside X86CPUDefinitions.
> > > These handlers are loaded into X86MachineState during the init.
> > > 3. Removed llc_id from x86CPU.
> > > 4. Removed init_apicid_fn hanlder from MachineClass. Kept all the code changes
> > > inside the x86.
> > > 5. Added new handler function apicid_pkg_offset for pkg_offset calculation.
> > > 6. And some Other minor changes.
> > >
> > > v3:
> > > https://lore.kernel.org/qemu-devel/157541968844.46157.17994918142533791313.stgit@naples-babu.amd.com/
> > > 1. Consolidated the topology information in structure X86CPUTopoInfo.
> > > 2. Changed the ccx_id to llc_id as commented by upstream.
> > > 3. Generalized the apic id decoding. It is mostly similar to current apic id
> > > except that it adds new field llc_id when numa configured. Removes all the
> > > hardcoded values.
> > > 4. Removed the earlier parse_numa split. And moved the numa node initialization
> > > inside the numa_complete_configuration. This is bit cleaner as commented by
> > > Eduardo.
> > > 5. Added new function init_apicid_fn inside machine_class structure. This
> > > will be used to update the apic id handler specific to cpu model.
> > > 6. Updated the cpuid unit tests.
> > > 7. TODO : Need to figure out how to dynamically update the handlers using cpu models.
> > > I might some guidance on that.
> > >
> > > v2:
> > > https://lore.kernel.org/qemu-devel/156779689013.21957.1631551572950676212.stgit@localhost.localdomain/
> > > 1. Introduced the new property epyc to enable new epyc mode.
> > > 2. Separated the epyc mode and non epyc mode function.
> > > 3. Introduced function pointers in PCMachineState to handle the
> > > differences.
> > > 4. Mildly tested different combinations to make things are working as expected.
> > > 5. TODO : Setting the epyc feature bit needs to be worked out. This feature is
> > > supported only on AMD EPYC models. I may need some guidance on that.
> > >
> > > v1:
> > > https://lore.kernel.org/qemu-devel/20190731232032.51786-1-babu.moger@amd.com/
> > > ---
> > >
> > > Babu Moger (13):
> > > hw/i386: Introduce X86CPUTopoInfo to contain topology info
> > > hw/i386: Consolidate topology functions
> > > machine: Add SMP Sockets in CpuTopology
> > > hw/i386: Remove unnecessary initialization in x86_cpu_new
> > > hw/i386: Update structures to save the number of nodes per package
> > > hw/i386: Rename apicid_from_topo_ids to x86_apicid_from_topo_ids
> > > hw/386: Add EPYC mode topology decoding functions
> > > target/i386: Cleanup and use the EPYC mode topology functions
> > > hw/i386: Introduce apicid functions inside X86MachineState
> > > i386: Introduce use_epyc_apic_id_encoding in X86CPUDefinition
> > > hw/i386: Move arch_id decode inside x86_cpus_init
> > > target/i386: Enable new apic id encoding for EPYC based cpus models
> > > i386: Fix pkg_id offset for EPYC cpu models
> > >
> > >
> > > hw/core/machine.c | 1
> > > hw/i386/pc.c | 15 ++-
> > > hw/i386/x86.c | 73 ++++++++++++----
> > > include/hw/boards.h | 2
> > > include/hw/i386/topology.h | 195 ++++++++++++++++++++++++++++++------------
> > > include/hw/i386/x86.h | 12 +++
> > > softmmu/vl.c | 1
> > > target/i386/cpu.c | 203 ++++++++++++++------------------------------
> > > target/i386/cpu.h | 3 +
> > > tests/test-x86-cpuid.c | 116 +++++++++++++++----------
> > > 10 files changed, 358 insertions(+), 263 deletions(-)
> > >
> > > --
> > > Signature
> > >
> >
>
> --
> Eduardo
--
Eduardo
^ permalink raw reply [flat|nested] 38+ messages in thread
* RE: [PATCH v7 00/13] APIC ID fixes for AMD EPYC CPU model
2020-03-17 23:46 ` Eduardo Habkost
@ 2020-03-18 2:43 ` Moger, Babu
2020-03-18 10:47 ` Igor Mammedov
0 siblings, 1 reply; 38+ messages in thread
From: Moger, Babu @ 2020-03-18 2:43 UTC (permalink / raw)
To: Eduardo Habkost; +Cc: mst, qemu-devel, imammedo, pbonzini, rth
[AMD Official Use Only - Internal Distribution Only]
> -----Original Message-----
> From: Eduardo Habkost <ehabkost@redhat.com>
> Sent: Tuesday, March 17, 2020 6:46 PM
> To: Moger, Babu <Babu.Moger@amd.com>
> Cc: marcel.apfelbaum@gmail.com; pbonzini@redhat.com; rth@twiddle.net;
> mst@redhat.com; imammedo@redhat.com; qemu-devel@nongnu.org
> Subject: Re: [PATCH v7 00/13] APIC ID fixes for AMD EPYC CPU model
>
> On Tue, Mar 17, 2020 at 07:22:06PM -0400, Eduardo Habkost wrote:
> > On Thu, Mar 12, 2020 at 11:28:47AM -0500, Babu Moger wrote:
> > > Eduardo, Can you please queue the series if there are no concerns.
> > > Thanks
> >
> > I had queued it for today's pull request, but it looks like it
> > breaks "make check". See
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftravis-
> ci.org%2Fgithub%2Fehabkost%2Fqemu%2Fjobs%2F663529282&data=02%7
> C01%7Cbabu.moger%40amd.com%7C43bba959c4d34e3be5fd08d7cacd634d%7
> C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637200855817408351&
> amp;sdata=cfjMVDKMgByvtUIqqGtcjNWGAf3PKFKxDLaS1eVME3U%3D&re
> served=0
> >
> > PASS 4 bios-tables-test /x86_64/acpi/piix4/ipmi
> > Could not access KVM kernel module: No such file or directory
> > qemu-system-x86_64: -accel kvm: failed to initialize kvm: No such file or
> directory
> > qemu-system-x86_64: falling back to tcg
> > qemu-system-x86_64: Invalid CPU [socket: 0, die: 0, core: 1, thread: 0] with
> APIC ID 1, valid index range 0:5
> > Broken pipe
> > /home/travis/build/ehabkost/qemu/tests/qtest/libqtest.c:166: kill_qemu()
> tried to terminate QEMU process but encountered exit status 1 (expected 0)
> > Aborted (core dumped)
> > ERROR - too few tests run (expected 17, got 4)
> > /home/travis/build/ehabkost/qemu/tests/Makefile.include:633: recipe for
> target 'check-qtest-x86_64' failed
> > make: *** [check-qtest-x86_64] Error 1
>
> Failure is at the /x86_64/acpi/piix4/cpuhp test case:
>
> $ QTEST_QEMU_BINARY=x86_64-softmmu/qemu-system-x86_64
> QTEST_QEMU_IMG=qemu-img tests/qtest/bios-tables-test -m=quick --verbose -
> -debug-log
> [...]
> {*LOG(start):{/x86_64/acpi/piix4/cpuhp}:LOG*}
> # starting QEMU: exec x86_64-softmmu/qemu-system-x86_64 -qtest
> unix:/tmp/qtest-2052313.sock -qtest-log /dev/null -chardev
> socket,path=/tmp/qtest-2052313.qmp,id=char0 -mon
> chardev=char0,mode=control -display none -machine pc,kernel-irqchip=off -
> accel kvm -accel tcg -net none -display none -smp
> 2,cores=3,sockets=2,maxcpus=6 -object memory-backend-
> ram,id=ram0,size=64M -object memory-backend-ram,id=ram1,size=64M -numa
> node,memdev=ram0 -numa node,memdev=ram1 -numa dist,src=0,dst=1,val=21
> -drive id=hd0,if=none,file=tests/acpi-test-disk-PVjFru,format=raw -device ide-
> hd,drive=hd0 -accel qtest
> {*LOG(message):{starting QEMU: exec x86_64-softmmu/qemu-system-x86_64
> -qtest unix:/tmp/qtest-2052313.sock -qtest-log /dev/null -chardev
> socket,path=/tmp/qtest-2052313.qmp,id=char0 -mon
> chardev=char0,mode=control -display none -machine pc,kernel-irqchip=off -
> accel kvm -accel tcg -net none -display none -smp
> 2,cores=3,sockets=2,maxcpus=6 -object memory-backend-
> ram,id=ram0,size=64M -object memory-backend-ram,id=ram1,size=64M -numa
> node,memdev=ram0 -numa node,memdev=ram1 -numa dist,src=0,dst=1,val=21
> -drive id=hd0,if=none,file=tests/acpi-test-disk-PVjFru,format=raw -device ide-
> hd,drive=hd0 -accel qtest}:LOG*}
> qemu-system-x86_64: Invalid CPU [socket: 0, die: 0, core: 1, thread: 0] with
> APIC ID 1, valid index range 0:5
> Broken pipe
The ms->smp.cpus Is not initialized to max cpus in this case. Looks like smp_parse did not run in this path.
For that reason the apicid is not initialized for all the cpus. Following patch fixes the problem.
I will test all the combinations and send the patch tomorrow. Let me know which tree I should use the to
generate the patch. It appears some patches are already pulled. I can send top of
git://github.com/ehabkost/qemu.git (x86-next).
diff --git a/hw/i386/x86.c b/hw/i386/x86.c
index 023dce1dbd..1eeb7b9732 100644
--- a/hw/i386/x86.c
+++ b/hw/i386/x86.c
@@ -156,7 +156,7 @@ void x86_cpus_init(X86MachineState *x86ms, int default_cpu_version)
ms->smp.max_cpus - 1) + 1;
possible_cpus = mc->possible_cpu_arch_ids(ms);
- for (i = 0; i < ms->smp.cpus; i++) {
+ for (i = 0; i < ms->possible_cpus->len; i++) {
ms->possible_cpus->cpus[i].arch_id =
x86_cpu_apic_id_from_index(x86ms, i);
}
>
>
> >
> >
> > >
> > > On 3/11/20 5:52 PM, Babu Moger wrote:
> > > > This series fixes APIC ID encoding problem reported on AMD EPYC cpu
> models.
> > > >
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugzilla.
> redhat.com%2Fshow_bug.cgi%3Fid%3D1728166&data=02%7C01%7Cbabu.
> moger%40amd.com%7C43bba959c4d34e3be5fd08d7cacd634d%7C3dd8961fe4
> 884e608e11a82d994e183d%7C0%7C0%7C637200855817408351&sdata=m
> E%2FiWq9sB2Jp9GtQesFZtU2lGT4MU6IVgm7HxhyfO9w%3D&reserved=0
> > > >
> > > > Currently, the APIC ID is decoded based on the sequence
> > > > sockets->dies->cores->threads. This works for most standard AMD and
> other
> > > > vendors' configurations, but this decoding sequence does not follow that
> of
> > > > AMD's APIC ID enumeration strictly. In some cases this can cause CPU
> topology
> > > > inconsistency. When booting a guest VM, the kernel tries to validate the
> > > > topology, and finds it inconsistent with the enumeration of EPYC cpu
> models.
> > > >
> > > > To fix the problem we need to build the topology as per the Processor
> > > > Programming Reference (PPR) for AMD Family 17h Model 01h, Revision B1
> > > > Processors. The documentation is available from the bugzilla Link below.
> > > >
> > > > Link:
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugzilla.
> kernel.org%2Fshow_bug.cgi%3Fid%3D206537&data=02%7C01%7Cbabu.m
> oger%40amd.com%7C43bba959c4d34e3be5fd08d7cacd634d%7C3dd8961fe488
> 4e608e11a82d994e183d%7C0%7C0%7C637200855817408351&sdata=BH1
> L3fcVzZdjo2zU3TclzJzZKJq%2BxpT3P%2FJwZXvs6Pc%3D&reserved=0
> > > >
> > > > Here is the text from the PPR.
> > > > Operating systems are expected to use
> Core::X86::Cpuid::SizeId[ApicIdSize], the
> > > > number of least significant bits in the Initial APIC ID that indicate core ID
> > > > within a processor, in constructing per-core CPUID masks.
> > > > Core::X86::Cpuid::SizeId[ApicIdSize] determines the maximum number of
> cores
> > > > (MNC) that the processor could theoretically support, not the actual
> number of
> > > > cores that are actually implemented or enabled on the processor, as
> indicated
> > > > by Core::X86::Cpuid::SizeId[NC].
> > > > Each Core::X86::Apic::ApicId[ApicId] register is preset as follows:
> > > > • ApicId[6] = Socket ID.
> > > > • ApicId[5:4] = Node ID.
> > > > • ApicId[3] = Logical CCX L3 complex ID
> > > > • ApicId[2:0]= (SMT) ? {LogicalCoreID[1:0],ThreadId} :
> {1'b0,LogicalCoreID[1:0]}
> > > >
> > > > v7:
> > > > Generated the patches on top of git://github.com/ehabkost/qemu.git
> (x86-next).
> > > > Changes from v6.
> > > > 1. Added new function x86_set_epyc_topo_handlers to override the apic
> id
> > > > encoding handlers.
> > > > 2. Separated the code to set use_epyc_apic_id_encoding and added as a
> new patch
> > > > as it looked more logical.
> > > > 3. Fixed minor typos.
> > > >
> > > > v6:
> > > >
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flore.ker
> nel.org%2Fqemu-
> devel%2F158389385028.22020.7608244627303132902.stgit%40naples-
> babu.amd.com%2F&data=02%7C01%7Cbabu.moger%40amd.com%7C43bb
> a959c4d34e3be5fd08d7cacd634d%7C3dd8961fe4884e608e11a82d994e183d%7
> C0%7C0%7C637200855817408351&sdata=7BZjkRROVX9M5nW1RmQYnITY
> fndrgR1jcHSWQGLSYco%3D&reserved=0
> > > > Generated the patches on top of git://github.com/ehabkost/qemu.git
> (x86-next).
> > > > Changes from v5.
> > > > 1. Eduardo has already queued couple of patches, submitting the rest
> here.
> > > > 2. Major change is how the EPYC mode apic id encoding handlers are
> loaded.
> > > > Added a boolean variable use_epyc_apic_id_encoding in
> X86CPUDefinition.
> > > > The variable is will be used to tell if we need to use EPYC mode encoding.
> > > > 3. Eduardo reported bysectability problem with x86 unit test code.
> > > > Quashed the patches in 1 and 2 to resolve it. Problem was change in
> calling
> > > > conventions of topology related functions.
> > > > 4. Also set the use_epyc_apic_id_encoding for EPYC-Rome. This model is
> > > > added recently to the cpu table.
> > > >
> > > > v5:
> > > >
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flore.ker
> nel.org%2Fqemu-
> devel%2F158326531474.40452.11433722850425537745.stgit%40naples-
> babu.amd.com%2F&data=02%7C01%7Cbabu.moger%40amd.com%7C43bb
> a959c4d34e3be5fd08d7cacd634d%7C3dd8961fe4884e608e11a82d994e183d%7
> C0%7C0%7C637200855817413332&sdata=rVqY3p6vUGeEu%2FbHTfE%2FfI
> gTtp0vuxzrE1egl5%2FYsGQ%3D&reserved=0
> > > > Generated the patches on top of git://github.com/ehabkost/qemu.git
> (x86-next).
> > > > Changes from v4.
> > > > 1. Re-arranged the patches 2 and 4 as suggested by Igor.
> > > > 2. Kept the apicid handler functions inside X86MachineState as discussed.
> > > > These handlers are loaded from X86CPUDefinitions.
> > > > 3. Removed unnecessary X86CPUstate initialization from x86_cpu_new.
> Suggested
> > > > by Igor.
> > > > 4. And other minor changes related to patch format.
> > > >
> > > > v4:
> > > >
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flore.ker
> nel.org%2Fqemu-
> devel%2F158161767653.48948.10578064482878399556.stgit%40naples-
> babu.amd.com%2F&data=02%7C01%7Cbabu.moger%40amd.com%7C43bb
> a959c4d34e3be5fd08d7cacd634d%7C3dd8961fe4884e608e11a82d994e183d%7
> C0%7C0%7C637200855817413332&sdata=pyd8T6rE%2BbR3FZf2c4cdtLr%2
> Fbxz%2FgW%2FWrap14mMt7To%3D&reserved=0
> > > > Changes from v3.
> > > > 1. Moved the arch_id calculation inside the function x86_cpus_init. With
> this change,
> > > > we dont need to change common numa code.(suggested by Igor)
> > > > 2. Introduced the model specific handlers inside X86CPUDefinitions.
> > > > These handlers are loaded into X86MachineState during the init.
> > > > 3. Removed llc_id from x86CPU.
> > > > 4. Removed init_apicid_fn hanlder from MachineClass. Kept all the code
> changes
> > > > inside the x86.
> > > > 5. Added new handler function apicid_pkg_offset for pkg_offset
> calculation.
> > > > 6. And some Other minor changes.
> > > >
> > > > v3:
> > > >
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flore.ker
> nel.org%2Fqemu-
> devel%2F157541968844.46157.17994918142533791313.stgit%40naples-
> babu.amd.com%2F&data=02%7C01%7Cbabu.moger%40amd.com%7C43bb
> a959c4d34e3be5fd08d7cacd634d%7C3dd8961fe4884e608e11a82d994e183d%7
> C0%7C0%7C637200855817413332&sdata=OM5sjNorayyzETuwa4FBPSLMb
> XtpbeXEG0AxRotIcXA%3D&reserved=0
> > > > 1. Consolidated the topology information in structure X86CPUTopoInfo.
> > > > 2. Changed the ccx_id to llc_id as commented by upstream.
> > > > 3. Generalized the apic id decoding. It is mostly similar to current apic id
> > > > except that it adds new field llc_id when numa configured. Removes all
> the
> > > > hardcoded values.
> > > > 4. Removed the earlier parse_numa split. And moved the numa node
> initialization
> > > > inside the numa_complete_configuration. This is bit cleaner as
> commented by
> > > > Eduardo.
> > > > 5. Added new function init_apicid_fn inside machine_class structure. This
> > > > will be used to update the apic id handler specific to cpu model.
> > > > 6. Updated the cpuid unit tests.
> > > > 7. TODO : Need to figure out how to dynamically update the handlers
> using cpu models.
> > > > I might some guidance on that.
> > > >
> > > > v2:
> > > >
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flore.ker
> nel.org%2Fqemu-
> devel%2F156779689013.21957.1631551572950676212.stgit%40localhost.locald
> omain%2F&data=02%7C01%7Cbabu.moger%40amd.com%7C43bba959c4d
> 34e3be5fd08d7cacd634d%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0
> %7C637200855817413332&sdata=HhZyaoTcB93BX2wFBQ46QXkLZywhIeq
> Rt9FkiqUmwaI%3D&reserved=0
> > > > 1. Introduced the new property epyc to enable new epyc mode.
> > > > 2. Separated the epyc mode and non epyc mode function.
> > > > 3. Introduced function pointers in PCMachineState to handle the
> > > > differences.
> > > > 4. Mildly tested different combinations to make things are working as
> expected.
> > > > 5. TODO : Setting the epyc feature bit needs to be worked out. This
> feature is
> > > > supported only on AMD EPYC models. I may need some guidance on
> that.
> > > >
> > > > v1:
> > > >
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flore.ker
> nel.org%2Fqemu-devel%2F20190731232032.51786-1-
> babu.moger%40amd.com%2F&data=02%7C01%7Cbabu.moger%40amd.co
> m%7C43bba959c4d34e3be5fd08d7cacd634d%7C3dd8961fe4884e608e11a82d9
> 94e183d%7C0%7C0%7C637200855817413332&sdata=BbMBhRE5C5lfdtc%
> 2FepDHzz2aOwhVbX7uDfUu737LNjA%3D&reserved=0
> > > > ---
> > > >
> > > > Babu Moger (13):
> > > > hw/i386: Introduce X86CPUTopoInfo to contain topology info
> > > > hw/i386: Consolidate topology functions
> > > > machine: Add SMP Sockets in CpuTopology
> > > > hw/i386: Remove unnecessary initialization in x86_cpu_new
> > > > hw/i386: Update structures to save the number of nodes per package
> > > > hw/i386: Rename apicid_from_topo_ids to x86_apicid_from_topo_ids
> > > > hw/386: Add EPYC mode topology decoding functions
> > > > target/i386: Cleanup and use the EPYC mode topology functions
> > > > hw/i386: Introduce apicid functions inside X86MachineState
> > > > i386: Introduce use_epyc_apic_id_encoding in X86CPUDefinition
> > > > hw/i386: Move arch_id decode inside x86_cpus_init
> > > > target/i386: Enable new apic id encoding for EPYC based cpus models
> > > > i386: Fix pkg_id offset for EPYC cpu models
> > > >
> > > >
> > > > hw/core/machine.c | 1
> > > > hw/i386/pc.c | 15 ++-
> > > > hw/i386/x86.c | 73 ++++++++++++----
> > > > include/hw/boards.h | 2
> > > > include/hw/i386/topology.h | 195 ++++++++++++++++++++++++++++++---
> ---------
> > > > include/hw/i386/x86.h | 12 +++
> > > > softmmu/vl.c | 1
> > > > target/i386/cpu.c | 203 ++++++++++++++------------------------------
> > > > target/i386/cpu.h | 3 +
> > > > tests/test-x86-cpuid.c | 116 +++++++++++++++----------
> > > > 10 files changed, 358 insertions(+), 263 deletions(-)
> > > >
> > > > --
> > > > Signature
> > > >
> > >
> >
> > --
> > Eduardo
>
> --
> Eduardo
^ permalink raw reply related [flat|nested] 38+ messages in thread
* Re: [PATCH v7 00/13] APIC ID fixes for AMD EPYC CPU model
2020-03-18 2:43 ` Moger, Babu
@ 2020-03-18 10:47 ` Igor Mammedov
2020-03-23 15:35 ` Moger, Babu
0 siblings, 1 reply; 38+ messages in thread
From: Igor Mammedov @ 2020-03-18 10:47 UTC (permalink / raw)
To: Moger, Babu; +Cc: Eduardo Habkost, mst, qemu-devel, pbonzini, rth
On Wed, 18 Mar 2020 02:43:57 +0000
"Moger, Babu" <Babu.Moger@amd.com> wrote:
> [AMD Official Use Only - Internal Distribution Only]
>
>
>
> > -----Original Message-----
> > From: Eduardo Habkost <ehabkost@redhat.com>
> > Sent: Tuesday, March 17, 2020 6:46 PM
> > To: Moger, Babu <Babu.Moger@amd.com>
> > Cc: marcel.apfelbaum@gmail.com; pbonzini@redhat.com; rth@twiddle.net;
> > mst@redhat.com; imammedo@redhat.com; qemu-devel@nongnu.org
> > Subject: Re: [PATCH v7 00/13] APIC ID fixes for AMD EPYC CPU model
> >
> > On Tue, Mar 17, 2020 at 07:22:06PM -0400, Eduardo Habkost wrote:
> > > On Thu, Mar 12, 2020 at 11:28:47AM -0500, Babu Moger wrote:
> > > > Eduardo, Can you please queue the series if there are no concerns.
> > > > Thanks
> > >
> > > I had queued it for today's pull request, but it looks like it
> > > breaks "make check". See
> > https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftravis-
> > ci.org%2Fgithub%2Fehabkost%2Fqemu%2Fjobs%2F663529282&data=02%7
> > C01%7Cbabu.moger%40amd.com%7C43bba959c4d34e3be5fd08d7cacd634d%7
> > C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637200855817408351&
> > amp;sdata=cfjMVDKMgByvtUIqqGtcjNWGAf3PKFKxDLaS1eVME3U%3D&re
> > served=0
> > >
> > > PASS 4 bios-tables-test /x86_64/acpi/piix4/ipmi
> > > Could not access KVM kernel module: No such file or directory
> > > qemu-system-x86_64: -accel kvm: failed to initialize kvm: No such file or
> > directory
> > > qemu-system-x86_64: falling back to tcg
> > > qemu-system-x86_64: Invalid CPU [socket: 0, die: 0, core: 1, thread: 0] with
> > APIC ID 1, valid index range 0:5
> > > Broken pipe
> > > /home/travis/build/ehabkost/qemu/tests/qtest/libqtest.c:166: kill_qemu()
> > tried to terminate QEMU process but encountered exit status 1 (expected 0)
> > > Aborted (core dumped)
> > > ERROR - too few tests run (expected 17, got 4)
> > > /home/travis/build/ehabkost/qemu/tests/Makefile.include:633: recipe for
> > target 'check-qtest-x86_64' failed
> > > make: *** [check-qtest-x86_64] Error 1
> >
> > Failure is at the /x86_64/acpi/piix4/cpuhp test case:
> >
> > $ QTEST_QEMU_BINARY=x86_64-softmmu/qemu-system-x86_64
> > QTEST_QEMU_IMG=qemu-img tests/qtest/bios-tables-test -m=quick --verbose -
> > -debug-log
> > [...]
> > {*LOG(start):{/x86_64/acpi/piix4/cpuhp}:LOG*}
> > # starting QEMU: exec x86_64-softmmu/qemu-system-x86_64 -qtest
> > unix:/tmp/qtest-2052313.sock -qtest-log /dev/null -chardev
> > socket,path=/tmp/qtest-2052313.qmp,id=char0 -mon
> > chardev=char0,mode=control -display none -machine pc,kernel-irqchip=off -
> > accel kvm -accel tcg -net none -display none -smp
> > 2,cores=3,sockets=2,maxcpus=6 -object memory-backend-
> > ram,id=ram0,size=64M -object memory-backend-ram,id=ram1,size=64M -numa
> > node,memdev=ram0 -numa node,memdev=ram1 -numa dist,src=0,dst=1,val=21
> > -drive id=hd0,if=none,file=tests/acpi-test-disk-PVjFru,format=raw -device ide-
> > hd,drive=hd0 -accel qtest
> > {*LOG(message):{starting QEMU: exec x86_64-softmmu/qemu-system-x86_64
> > -qtest unix:/tmp/qtest-2052313.sock -qtest-log /dev/null -chardev
> > socket,path=/tmp/qtest-2052313.qmp,id=char0 -mon
> > chardev=char0,mode=control -display none -machine pc,kernel-irqchip=off -
> > accel kvm -accel tcg -net none -display none -smp
> > 2,cores=3,sockets=2,maxcpus=6 -object memory-backend-
> > ram,id=ram0,size=64M -object memory-backend-ram,id=ram1,size=64M -numa
> > node,memdev=ram0 -numa node,memdev=ram1 -numa dist,src=0,dst=1,val=21
> > -drive id=hd0,if=none,file=tests/acpi-test-disk-PVjFru,format=raw -device ide-
> > hd,drive=hd0 -accel qtest}:LOG*}
> > qemu-system-x86_64: Invalid CPU [socket: 0, die: 0, core: 1, thread: 0] with
> > APIC ID 1, valid index range 0:5
> > Broken pipe
>
> The ms->smp.cpus Is not initialized to max cpus in this case. Looks like smp_parse did not run in this path.
> For that reason the apicid is not initialized for all the cpus. Following patch fixes the problem.
> I will test all the combinations and send the patch tomorrow. Let me know which tree I should use the to
> generate the patch. It appears some patches are already pulled. I can send top of
> git://github.com/ehabkost/qemu.git (x86-next).
>
> diff --git a/hw/i386/x86.c b/hw/i386/x86.c
> index 023dce1dbd..1eeb7b9732 100644
> --- a/hw/i386/x86.c
> +++ b/hw/i386/x86.c
> @@ -156,7 +156,7 @@ void x86_cpus_init(X86MachineState *x86ms, int default_cpu_version)
> ms->smp.max_cpus - 1) + 1;
> possible_cpus = mc->possible_cpu_arch_ids(ms);
>
> - for (i = 0; i < ms->smp.cpus; i++) {
> + for (i = 0; i < ms->possible_cpus->len; i++) {
> ms->possible_cpus->cpus[i].arch_id =
> x86_cpu_apic_id_from_index(x86ms, i);
> }
indeed, it should use possible_cpus->len instead of initial cpus number
> >
> >
> > >
> > >
> > > >
> > > > On 3/11/20 5:52 PM, Babu Moger wrote:
> > > > > This series fixes APIC ID encoding problem reported on AMD EPYC cpu
> > models.
> > > > >
> > https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugzilla.
> > redhat.com%2Fshow_bug.cgi%3Fid%3D1728166&data=02%7C01%7Cbabu.
> > moger%40amd.com%7C43bba959c4d34e3be5fd08d7cacd634d%7C3dd8961fe4
> > 884e608e11a82d994e183d%7C0%7C0%7C637200855817408351&sdata=m
> > E%2FiWq9sB2Jp9GtQesFZtU2lGT4MU6IVgm7HxhyfO9w%3D&reserved=0
> > > > >
> > > > > Currently, the APIC ID is decoded based on the sequence
> > > > > sockets->dies->cores->threads. This works for most standard AMD and
> > other
> > > > > vendors' configurations, but this decoding sequence does not follow that
> > of
> > > > > AMD's APIC ID enumeration strictly. In some cases this can cause CPU
> > topology
> > > > > inconsistency. When booting a guest VM, the kernel tries to validate the
> > > > > topology, and finds it inconsistent with the enumeration of EPYC cpu
> > models.
> > > > >
> > > > > To fix the problem we need to build the topology as per the Processor
> > > > > Programming Reference (PPR) for AMD Family 17h Model 01h, Revision B1
> > > > > Processors. The documentation is available from the bugzilla Link below.
> > > > >
> > > > > Link:
> > https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugzilla.
> > kernel.org%2Fshow_bug.cgi%3Fid%3D206537&data=02%7C01%7Cbabu.m
> > oger%40amd.com%7C43bba959c4d34e3be5fd08d7cacd634d%7C3dd8961fe488
> > 4e608e11a82d994e183d%7C0%7C0%7C637200855817408351&sdata=BH1
> > L3fcVzZdjo2zU3TclzJzZKJq%2BxpT3P%2FJwZXvs6Pc%3D&reserved=0
> > > > >
> > > > > Here is the text from the PPR.
> > > > > Operating systems are expected to use
> > Core::X86::Cpuid::SizeId[ApicIdSize], the
> > > > > number of least significant bits in the Initial APIC ID that indicate core ID
> > > > > within a processor, in constructing per-core CPUID masks.
> > > > > Core::X86::Cpuid::SizeId[ApicIdSize] determines the maximum number of
> > cores
> > > > > (MNC) that the processor could theoretically support, not the actual
> > number of
> > > > > cores that are actually implemented or enabled on the processor, as
> > indicated
> > > > > by Core::X86::Cpuid::SizeId[NC].
> > > > > Each Core::X86::Apic::ApicId[ApicId] register is preset as follows:
> > > > > • ApicId[6] = Socket ID.
> > > > > • ApicId[5:4] = Node ID.
> > > > > • ApicId[3] = Logical CCX L3 complex ID
> > > > > • ApicId[2:0]= (SMT) ? {LogicalCoreID[1:0],ThreadId} :
> > {1'b0,LogicalCoreID[1:0]}
> > > > >
> > > > > v7:
> > > > > Generated the patches on top of git://github.com/ehabkost/qemu.git
> > (x86-next).
> > > > > Changes from v6.
> > > > > 1. Added new function x86_set_epyc_topo_handlers to override the apic
> > id
> > > > > encoding handlers.
> > > > > 2. Separated the code to set use_epyc_apic_id_encoding and added as a
> > new patch
> > > > > as it looked more logical.
> > > > > 3. Fixed minor typos.
> > > > >
> > > > > v6:
> > > > >
> > https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flore.ker
> > nel.org%2Fqemu-
> > devel%2F158389385028.22020.7608244627303132902.stgit%40naples-
> > babu.amd.com%2F&data=02%7C01%7Cbabu.moger%40amd.com%7C43bb
> > a959c4d34e3be5fd08d7cacd634d%7C3dd8961fe4884e608e11a82d994e183d%7
> > C0%7C0%7C637200855817408351&sdata=7BZjkRROVX9M5nW1RmQYnITY
> > fndrgR1jcHSWQGLSYco%3D&reserved=0
> > > > > Generated the patches on top of git://github.com/ehabkost/qemu.git
> > (x86-next).
> > > > > Changes from v5.
> > > > > 1. Eduardo has already queued couple of patches, submitting the rest
> > here.
> > > > > 2. Major change is how the EPYC mode apic id encoding handlers are
> > loaded.
> > > > > Added a boolean variable use_epyc_apic_id_encoding in
> > X86CPUDefinition.
> > > > > The variable is will be used to tell if we need to use EPYC mode encoding.
> > > > > 3. Eduardo reported bysectability problem with x86 unit test code.
> > > > > Quashed the patches in 1 and 2 to resolve it. Problem was change in
> > calling
> > > > > conventions of topology related functions.
> > > > > 4. Also set the use_epyc_apic_id_encoding for EPYC-Rome. This model is
> > > > > added recently to the cpu table.
> > > > >
> > > > > v5:
> > > > >
> > https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flore.ker
> > nel.org%2Fqemu-
> > devel%2F158326531474.40452.11433722850425537745.stgit%40naples-
> > babu.amd.com%2F&data=02%7C01%7Cbabu.moger%40amd.com%7C43bb
> > a959c4d34e3be5fd08d7cacd634d%7C3dd8961fe4884e608e11a82d994e183d%7
> > C0%7C0%7C637200855817413332&sdata=rVqY3p6vUGeEu%2FbHTfE%2FfI
> > gTtp0vuxzrE1egl5%2FYsGQ%3D&reserved=0
> > > > > Generated the patches on top of git://github.com/ehabkost/qemu.git
> > (x86-next).
> > > > > Changes from v4.
> > > > > 1. Re-arranged the patches 2 and 4 as suggested by Igor.
> > > > > 2. Kept the apicid handler functions inside X86MachineState as discussed.
> > > > > These handlers are loaded from X86CPUDefinitions.
> > > > > 3. Removed unnecessary X86CPUstate initialization from x86_cpu_new.
> > Suggested
> > > > > by Igor.
> > > > > 4. And other minor changes related to patch format.
> > > > >
> > > > > v4:
> > > > >
> > https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flore.ker
> > nel.org%2Fqemu-
> > devel%2F158161767653.48948.10578064482878399556.stgit%40naples-
> > babu.amd.com%2F&data=02%7C01%7Cbabu.moger%40amd.com%7C43bb
> > a959c4d34e3be5fd08d7cacd634d%7C3dd8961fe4884e608e11a82d994e183d%7
> > C0%7C0%7C637200855817413332&sdata=pyd8T6rE%2BbR3FZf2c4cdtLr%2
> > Fbxz%2FgW%2FWrap14mMt7To%3D&reserved=0
> > > > > Changes from v3.
> > > > > 1. Moved the arch_id calculation inside the function x86_cpus_init. With
> > this change,
> > > > > we dont need to change common numa code.(suggested by Igor)
> > > > > 2. Introduced the model specific handlers inside X86CPUDefinitions.
> > > > > These handlers are loaded into X86MachineState during the init.
> > > > > 3. Removed llc_id from x86CPU.
> > > > > 4. Removed init_apicid_fn hanlder from MachineClass. Kept all the code
> > changes
> > > > > inside the x86.
> > > > > 5. Added new handler function apicid_pkg_offset for pkg_offset
> > calculation.
> > > > > 6. And some Other minor changes.
> > > > >
> > > > > v3:
> > > > >
> > https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flore.ker
> > nel.org%2Fqemu-
> > devel%2F157541968844.46157.17994918142533791313.stgit%40naples-
> > babu.amd.com%2F&data=02%7C01%7Cbabu.moger%40amd.com%7C43bb
> > a959c4d34e3be5fd08d7cacd634d%7C3dd8961fe4884e608e11a82d994e183d%7
> > C0%7C0%7C637200855817413332&sdata=OM5sjNorayyzETuwa4FBPSLMb
> > XtpbeXEG0AxRotIcXA%3D&reserved=0
> > > > > 1. Consolidated the topology information in structure X86CPUTopoInfo.
> > > > > 2. Changed the ccx_id to llc_id as commented by upstream.
> > > > > 3. Generalized the apic id decoding. It is mostly similar to current apic id
> > > > > except that it adds new field llc_id when numa configured. Removes all
> > the
> > > > > hardcoded values.
> > > > > 4. Removed the earlier parse_numa split. And moved the numa node
> > initialization
> > > > > inside the numa_complete_configuration. This is bit cleaner as
> > commented by
> > > > > Eduardo.
> > > > > 5. Added new function init_apicid_fn inside machine_class structure. This
> > > > > will be used to update the apic id handler specific to cpu model.
> > > > > 6. Updated the cpuid unit tests.
> > > > > 7. TODO : Need to figure out how to dynamically update the handlers
> > using cpu models.
> > > > > I might some guidance on that.
> > > > >
> > > > > v2:
> > > > >
> > https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flore.ker
> > nel.org%2Fqemu-
> > devel%2F156779689013.21957.1631551572950676212.stgit%40localhost.locald
> > omain%2F&data=02%7C01%7Cbabu.moger%40amd.com%7C43bba959c4d
> > 34e3be5fd08d7cacd634d%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0
> > %7C637200855817413332&sdata=HhZyaoTcB93BX2wFBQ46QXkLZywhIeq
> > Rt9FkiqUmwaI%3D&reserved=0
> > > > > 1. Introduced the new property epyc to enable new epyc mode.
> > > > > 2. Separated the epyc mode and non epyc mode function.
> > > > > 3. Introduced function pointers in PCMachineState to handle the
> > > > > differences.
> > > > > 4. Mildly tested different combinations to make things are working as
> > expected.
> > > > > 5. TODO : Setting the epyc feature bit needs to be worked out. This
> > feature is
> > > > > supported only on AMD EPYC models. I may need some guidance on
> > that.
> > > > >
> > > > > v1:
> > > > >
> > https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flore.ker
> > nel.org%2Fqemu-devel%2F20190731232032.51786-1-
> > babu.moger%40amd.com%2F&data=02%7C01%7Cbabu.moger%40amd.co
> > m%7C43bba959c4d34e3be5fd08d7cacd634d%7C3dd8961fe4884e608e11a82d9
> > 94e183d%7C0%7C0%7C637200855817413332&sdata=BbMBhRE5C5lfdtc%
> > 2FepDHzz2aOwhVbX7uDfUu737LNjA%3D&reserved=0
> > > > > ---
> > > > >
> > > > > Babu Moger (13):
> > > > > hw/i386: Introduce X86CPUTopoInfo to contain topology info
> > > > > hw/i386: Consolidate topology functions
> > > > > machine: Add SMP Sockets in CpuTopology
> > > > > hw/i386: Remove unnecessary initialization in x86_cpu_new
> > > > > hw/i386: Update structures to save the number of nodes per package
> > > > > hw/i386: Rename apicid_from_topo_ids to x86_apicid_from_topo_ids
> > > > > hw/386: Add EPYC mode topology decoding functions
> > > > > target/i386: Cleanup and use the EPYC mode topology functions
> > > > > hw/i386: Introduce apicid functions inside X86MachineState
> > > > > i386: Introduce use_epyc_apic_id_encoding in X86CPUDefinition
> > > > > hw/i386: Move arch_id decode inside x86_cpus_init
> > > > > target/i386: Enable new apic id encoding for EPYC based cpus models
> > > > > i386: Fix pkg_id offset for EPYC cpu models
> > > > >
> > > > >
> > > > > hw/core/machine.c | 1
> > > > > hw/i386/pc.c | 15 ++-
> > > > > hw/i386/x86.c | 73 ++++++++++++----
> > > > > include/hw/boards.h | 2
> > > > > include/hw/i386/topology.h | 195 ++++++++++++++++++++++++++++++---
> > ---------
> > > > > include/hw/i386/x86.h | 12 +++
> > > > > softmmu/vl.c | 1
> > > > > target/i386/cpu.c | 203 ++++++++++++++------------------------------
> > > > > target/i386/cpu.h | 3 +
> > > > > tests/test-x86-cpuid.c | 116 +++++++++++++++----------
> > > > > 10 files changed, 358 insertions(+), 263 deletions(-)
> > > > >
> > > > > --
> > > > > Signature
> > > > >
> > > >
> > >
> > > --
> > > Eduardo
> >
> > --
> > Eduardo
^ permalink raw reply [flat|nested] 38+ messages in thread
* RE: [PATCH v7 00/13] APIC ID fixes for AMD EPYC CPU model
2020-03-18 10:47 ` Igor Mammedov
@ 2020-03-23 15:35 ` Moger, Babu
0 siblings, 0 replies; 38+ messages in thread
From: Moger, Babu @ 2020-03-23 15:35 UTC (permalink / raw)
To: Igor Mammedov; +Cc: Eduardo Habkost, mst, qemu-devel, pbonzini, rth
[AMD Official Use Only - Internal Distribution Only]
> -----Original Message-----
> From: Igor Mammedov <imammedo@redhat.com>
> Sent: Wednesday, March 18, 2020 5:47 AM
> To: Moger, Babu <Babu.Moger@amd.com>
> Cc: Eduardo Habkost <ehabkost@redhat.com>; marcel.apfelbaum@gmail.com;
> pbonzini@redhat.com; rth@twiddle.net; mst@redhat.com; qemu-
> devel@nongnu.org
> Subject: Re: [PATCH v7 00/13] APIC ID fixes for AMD EPYC CPU model
>
> On Wed, 18 Mar 2020 02:43:57 +0000
> "Moger, Babu" <Babu.Moger@amd.com> wrote:
>
> > [AMD Official Use Only - Internal Distribution Only]
> >
> >
> >
> > > -----Original Message-----
> > > From: Eduardo Habkost <ehabkost@redhat.com>
> > > Sent: Tuesday, March 17, 2020 6:46 PM
> > > To: Moger, Babu <Babu.Moger@amd.com>
> > > Cc: marcel.apfelbaum@gmail.com; pbonzini@redhat.com; rth@twiddle.net;
> > > mst@redhat.com; imammedo@redhat.com; qemu-devel@nongnu.org
> > > Subject: Re: [PATCH v7 00/13] APIC ID fixes for AMD EPYC CPU model
> > >
> > > On Tue, Mar 17, 2020 at 07:22:06PM -0400, Eduardo Habkost wrote:
> > > > On Thu, Mar 12, 2020 at 11:28:47AM -0500, Babu Moger wrote:
> > > > > Eduardo, Can you please queue the series if there are no concerns.
> > > > > Thanks
> > > >
> > > > I had queued it for today's pull request, but it looks like it
> > > > breaks "make check". See
> > >
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftravis-
> > >
> ci.org%2Fgithub%2Fehabkost%2Fqemu%2Fjobs%2F663529282&data=02%7
> > >
> C01%7Cbabu.moger%40amd.com%7C43bba959c4d34e3be5fd08d7cacd634d%7
> > >
> C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637200855817408351&
> > >
> amp;sdata=cfjMVDKMgByvtUIqqGtcjNWGAf3PKFKxDLaS1eVME3U%3D&re
> > > served=0
> > > >
> > > > PASS 4 bios-tables-test /x86_64/acpi/piix4/ipmi
> > > > Could not access KVM kernel module: No such file or directory
> > > > qemu-system-x86_64: -accel kvm: failed to initialize kvm: No such file or
> > > directory
> > > > qemu-system-x86_64: falling back to tcg
> > > > qemu-system-x86_64: Invalid CPU [socket: 0, die: 0, core: 1, thread: 0]
> with
> > > APIC ID 1, valid index range 0:5
> > > > Broken pipe
> > > > /home/travis/build/ehabkost/qemu/tests/qtest/libqtest.c:166: kill_qemu()
> > > tried to terminate QEMU process but encountered exit status 1 (expected 0)
> > > > Aborted (core dumped)
> > > > ERROR - too few tests run (expected 17, got 4)
> > > > /home/travis/build/ehabkost/qemu/tests/Makefile.include:633: recipe
> for
> > > target 'check-qtest-x86_64' failed
> > > > make: *** [check-qtest-x86_64] Error 1
> > >
> > > Failure is at the /x86_64/acpi/piix4/cpuhp test case:
> > >
> > > $ QTEST_QEMU_BINARY=x86_64-softmmu/qemu-system-x86_64
> > > QTEST_QEMU_IMG=qemu-img tests/qtest/bios-tables-test -m=quick --
> verbose -
> > > -debug-log
> > > [...]
> > > {*LOG(start):{/x86_64/acpi/piix4/cpuhp}:LOG*}
> > > # starting QEMU: exec x86_64-softmmu/qemu-system-x86_64 -qtest
> > > unix:/tmp/qtest-2052313.sock -qtest-log /dev/null -chardev
> > > socket,path=/tmp/qtest-2052313.qmp,id=char0 -mon
> > > chardev=char0,mode=control -display none -machine pc,kernel-irqchip=off -
> > > accel kvm -accel tcg -net none -display none -smp
> > > 2,cores=3,sockets=2,maxcpus=6 -object memory-backend-
> > > ram,id=ram0,size=64M -object memory-backend-ram,id=ram1,size=64M -
> numa
> > > node,memdev=ram0 -numa node,memdev=ram1 -numa
> dist,src=0,dst=1,val=21
> > > -drive id=hd0,if=none,file=tests/acpi-test-disk-PVjFru,format=raw -device
> ide-
> > > hd,drive=hd0 -accel qtest
> > > {*LOG(message):{starting QEMU: exec x86_64-softmmu/qemu-system-
> x86_64
> > > -qtest unix:/tmp/qtest-2052313.sock -qtest-log /dev/null -chardev
> > > socket,path=/tmp/qtest-2052313.qmp,id=char0 -mon
> > > chardev=char0,mode=control -display none -machine pc,kernel-irqchip=off -
> > > accel kvm -accel tcg -net none -display none -smp
> > > 2,cores=3,sockets=2,maxcpus=6 -object memory-backend-
> > > ram,id=ram0,size=64M -object memory-backend-ram,id=ram1,size=64M -
> numa
> > > node,memdev=ram0 -numa node,memdev=ram1 -numa
> dist,src=0,dst=1,val=21
> > > -drive id=hd0,if=none,file=tests/acpi-test-disk-PVjFru,format=raw -device
> ide-
> > > hd,drive=hd0 -accel qtest}:LOG*}
> > > qemu-system-x86_64: Invalid CPU [socket: 0, die: 0, core: 1, thread: 0] with
> > > APIC ID 1, valid index range 0:5
> > > Broken pipe
> >
> > The ms->smp.cpus Is not initialized to max cpus in this case. Looks like
> smp_parse did not run in this path.
> > For that reason the apicid is not initialized for all the cpus. Following patch
> fixes the problem.
> > I will test all the combinations and send the patch tomorrow. Let me know
> which tree I should use the to
> > generate the patch. It appears some patches are already pulled. I can send top
> of
> > git://github.com/ehabkost/qemu.git (x86-next).
> >
> > diff --git a/hw/i386/x86.c b/hw/i386/x86.c
> > index 023dce1dbd..1eeb7b9732 100644
> > --- a/hw/i386/x86.c
> > +++ b/hw/i386/x86.c
> > @@ -156,7 +156,7 @@ void x86_cpus_init(X86MachineState *x86ms, int
> default_cpu_version)
> > ms->smp.max_cpus - 1) + 1;
> > possible_cpus = mc->possible_cpu_arch_ids(ms);
> >
> > - for (i = 0; i < ms->smp.cpus; i++) {
> > + for (i = 0; i < ms->possible_cpus->len; i++) {
> > ms->possible_cpus->cpus[i].arch_id =
> > x86_cpu_apic_id_from_index(x86ms, i);
> > }
>
> indeed, it should use possible_cpus->len instead of initial cpus number
Just to update everyone.. This fix has been already picked up by Eduardo. Thanks
>
> > >
> > >
> > > >
> > > >
> > > > >
> > > > > On 3/11/20 5:52 PM, Babu Moger wrote:
> > > > > > This series fixes APIC ID encoding problem reported on AMD EPYC cpu
> > > models.
> > > > > >
> > >
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugzilla.
> > >
> redhat.com%2Fshow_bug.cgi%3Fid%3D1728166&data=02%7C01%7Cbabu.
> > >
> moger%40amd.com%7C43bba959c4d34e3be5fd08d7cacd634d%7C3dd8961fe4
> > >
> 884e608e11a82d994e183d%7C0%7C0%7C637200855817408351&sdata=m
> > >
> E%2FiWq9sB2Jp9GtQesFZtU2lGT4MU6IVgm7HxhyfO9w%3D&reserved=0
> > > > > >
> > > > > > Currently, the APIC ID is decoded based on the sequence
> > > > > > sockets->dies->cores->threads. This works for most standard AMD and
> > > other
> > > > > > vendors' configurations, but this decoding sequence does not follow
> that
> > > of
> > > > > > AMD's APIC ID enumeration strictly. In some cases this can cause CPU
> > > topology
> > > > > > inconsistency. When booting a guest VM, the kernel tries to validate
> the
> > > > > > topology, and finds it inconsistent with the enumeration of EPYC cpu
> > > models.
> > > > > >
> > > > > > To fix the problem we need to build the topology as per the Processor
> > > > > > Programming Reference (PPR) for AMD Family 17h Model 01h, Revision
> B1
> > > > > > Processors. The documentation is available from the bugzilla Link
> below.
> > > > > >
> > > > > > Link:
> > >
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugzilla.
> > >
> kernel.org%2Fshow_bug.cgi%3Fid%3D206537&data=02%7C01%7Cbabu.m
> > >
> oger%40amd.com%7C43bba959c4d34e3be5fd08d7cacd634d%7C3dd8961fe488
> > >
> 4e608e11a82d994e183d%7C0%7C0%7C637200855817408351&sdata=BH1
> > > L3fcVzZdjo2zU3TclzJzZKJq%2BxpT3P%2FJwZXvs6Pc%3D&reserved=0
> > > > > >
> > > > > > Here is the text from the PPR.
> > > > > > Operating systems are expected to use
> > > Core::X86::Cpuid::SizeId[ApicIdSize], the
> > > > > > number of least significant bits in the Initial APIC ID that indicate core
> ID
> > > > > > within a processor, in constructing per-core CPUID masks.
> > > > > > Core::X86::Cpuid::SizeId[ApicIdSize] determines the maximum number
> of
> > > cores
> > > > > > (MNC) that the processor could theoretically support, not the actual
> > > number of
> > > > > > cores that are actually implemented or enabled on the processor, as
> > > indicated
> > > > > > by Core::X86::Cpuid::SizeId[NC].
> > > > > > Each Core::X86::Apic::ApicId[ApicId] register is preset as follows:
> > > > > > • ApicId[6] = Socket ID.
> > > > > > • ApicId[5:4] = Node ID.
> > > > > > • ApicId[3] = Logical CCX L3 complex ID
> > > > > > • ApicId[2:0]= (SMT) ? {LogicalCoreID[1:0],ThreadId} :
> > > {1'b0,LogicalCoreID[1:0]}
> > > > > >
> > > > > > v7:
> > > > > > Generated the patches on top of git://github.com/ehabkost/qemu.git
> > > (x86-next).
> > > > > > Changes from v6.
> > > > > > 1. Added new function x86_set_epyc_topo_handlers to override the
> apic
> > > id
> > > > > > encoding handlers.
> > > > > > 2. Separated the code to set use_epyc_apic_id_encoding and added as
> a
> > > new patch
> > > > > > as it looked more logical.
> > > > > > 3. Fixed minor typos.
> > > > > >
> > > > > > v6:
> > > > > >
> > >
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flore.ker
> > > nel.org%2Fqemu-
> > > devel%2F158389385028.22020.7608244627303132902.stgit%40naples-
> > >
> babu.amd.com%2F&data=02%7C01%7Cbabu.moger%40amd.com%7C43bb
> > >
> a959c4d34e3be5fd08d7cacd634d%7C3dd8961fe4884e608e11a82d994e183d%7
> > >
> C0%7C0%7C637200855817408351&sdata=7BZjkRROVX9M5nW1RmQYnITY
> > > fndrgR1jcHSWQGLSYco%3D&reserved=0
> > > > > > Generated the patches on top of git://github.com/ehabkost/qemu.git
> > > (x86-next).
> > > > > > Changes from v5.
> > > > > > 1. Eduardo has already queued couple of patches, submitting the rest
> > > here.
> > > > > > 2. Major change is how the EPYC mode apic id encoding handlers are
> > > loaded.
> > > > > > Added a boolean variable use_epyc_apic_id_encoding in
> > > X86CPUDefinition.
> > > > > > The variable is will be used to tell if we need to use EPYC mode
> encoding.
> > > > > > 3. Eduardo reported bysectability problem with x86 unit test code.
> > > > > > Quashed the patches in 1 and 2 to resolve it. Problem was change in
> > > calling
> > > > > > conventions of topology related functions.
> > > > > > 4. Also set the use_epyc_apic_id_encoding for EPYC-Rome. This model
> is
> > > > > > added recently to the cpu table.
> > > > > >
> > > > > > v5:
> > > > > >
> > >
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flore.ker
> > > nel.org%2Fqemu-
> > > devel%2F158326531474.40452.11433722850425537745.stgit%40naples-
> > >
> babu.amd.com%2F&data=02%7C01%7Cbabu.moger%40amd.com%7C43bb
> > >
> a959c4d34e3be5fd08d7cacd634d%7C3dd8961fe4884e608e11a82d994e183d%7
> > >
> C0%7C0%7C637200855817413332&sdata=rVqY3p6vUGeEu%2FbHTfE%2FfI
> > > gTtp0vuxzrE1egl5%2FYsGQ%3D&reserved=0
> > > > > > Generated the patches on top of git://github.com/ehabkost/qemu.git
> > > (x86-next).
> > > > > > Changes from v4.
> > > > > > 1. Re-arranged the patches 2 and 4 as suggested by Igor.
> > > > > > 2. Kept the apicid handler functions inside X86MachineState as
> discussed.
> > > > > > These handlers are loaded from X86CPUDefinitions.
> > > > > > 3. Removed unnecessary X86CPUstate initialization from
> x86_cpu_new.
> > > Suggested
> > > > > > by Igor.
> > > > > > 4. And other minor changes related to patch format.
> > > > > >
> > > > > > v4:
> > > > > >
> > >
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flore.ker
> > > nel.org%2Fqemu-
> > > devel%2F158161767653.48948.10578064482878399556.stgit%40naples-
> > >
> babu.amd.com%2F&data=02%7C01%7Cbabu.moger%40amd.com%7C43bb
> > >
> a959c4d34e3be5fd08d7cacd634d%7C3dd8961fe4884e608e11a82d994e183d%7
> > >
> C0%7C0%7C637200855817413332&sdata=pyd8T6rE%2BbR3FZf2c4cdtLr%2
> > > Fbxz%2FgW%2FWrap14mMt7To%3D&reserved=0
> > > > > > Changes from v3.
> > > > > > 1. Moved the arch_id calculation inside the function x86_cpus_init.
> With
> > > this change,
> > > > > > we dont need to change common numa code.(suggested by Igor)
> > > > > > 2. Introduced the model specific handlers inside X86CPUDefinitions.
> > > > > > These handlers are loaded into X86MachineState during the init.
> > > > > > 3. Removed llc_id from x86CPU.
> > > > > > 4. Removed init_apicid_fn hanlder from MachineClass. Kept all the
> code
> > > changes
> > > > > > inside the x86.
> > > > > > 5. Added new handler function apicid_pkg_offset for pkg_offset
> > > calculation.
> > > > > > 6. And some Other minor changes.
> > > > > >
> > > > > > v3:
> > > > > >
> > >
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flore.ker
> > > nel.org%2Fqemu-
> > > devel%2F157541968844.46157.17994918142533791313.stgit%40naples-
> > >
> babu.amd.com%2F&data=02%7C01%7Cbabu.moger%40amd.com%7C43bb
> > >
> a959c4d34e3be5fd08d7cacd634d%7C3dd8961fe4884e608e11a82d994e183d%7
> > >
> C0%7C0%7C637200855817413332&sdata=OM5sjNorayyzETuwa4FBPSLMb
> > > XtpbeXEG0AxRotIcXA%3D&reserved=0
> > > > > > 1. Consolidated the topology information in structure
> X86CPUTopoInfo.
> > > > > > 2. Changed the ccx_id to llc_id as commented by upstream.
> > > > > > 3. Generalized the apic id decoding. It is mostly similar to current apic
> id
> > > > > > except that it adds new field llc_id when numa configured. Removes
> all
> > > the
> > > > > > hardcoded values.
> > > > > > 4. Removed the earlier parse_numa split. And moved the numa node
> > > initialization
> > > > > > inside the numa_complete_configuration. This is bit cleaner as
> > > commented by
> > > > > > Eduardo.
> > > > > > 5. Added new function init_apicid_fn inside machine_class structure.
> This
> > > > > > will be used to update the apic id handler specific to cpu model.
> > > > > > 6. Updated the cpuid unit tests.
> > > > > > 7. TODO : Need to figure out how to dynamically update the handlers
> > > using cpu models.
> > > > > > I might some guidance on that.
> > > > > >
> > > > > > v2:
> > > > > >
> > >
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flore.ker
> > > nel.org%2Fqemu-
> > >
> devel%2F156779689013.21957.1631551572950676212.stgit%40localhost.locald
> > >
> omain%2F&data=02%7C01%7Cbabu.moger%40amd.com%7C43bba959c4d
> > >
> 34e3be5fd08d7cacd634d%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0
> > >
> %7C637200855817413332&sdata=HhZyaoTcB93BX2wFBQ46QXkLZywhIeq
> > > Rt9FkiqUmwaI%3D&reserved=0
> > > > > > 1. Introduced the new property epyc to enable new epyc mode.
> > > > > > 2. Separated the epyc mode and non epyc mode function.
> > > > > > 3. Introduced function pointers in PCMachineState to handle the
> > > > > > differences.
> > > > > > 4. Mildly tested different combinations to make things are working as
> > > expected.
> > > > > > 5. TODO : Setting the epyc feature bit needs to be worked out. This
> > > feature is
> > > > > > supported only on AMD EPYC models. I may need some guidance on
> > > that.
> > > > > >
> > > > > > v1:
> > > > > >
> > >
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flore.ker
> > > nel.org%2Fqemu-devel%2F20190731232032.51786-1-
> > >
> babu.moger%40amd.com%2F&data=02%7C01%7Cbabu.moger%40amd.co
> > >
> m%7C43bba959c4d34e3be5fd08d7cacd634d%7C3dd8961fe4884e608e11a82d9
> > >
> 94e183d%7C0%7C0%7C637200855817413332&sdata=BbMBhRE5C5lfdtc%
> > > 2FepDHzz2aOwhVbX7uDfUu737LNjA%3D&reserved=0
> > > > > > ---
> > > > > >
> > > > > > Babu Moger (13):
> > > > > > hw/i386: Introduce X86CPUTopoInfo to contain topology info
> > > > > > hw/i386: Consolidate topology functions
> > > > > > machine: Add SMP Sockets in CpuTopology
> > > > > > hw/i386: Remove unnecessary initialization in x86_cpu_new
> > > > > > hw/i386: Update structures to save the number of nodes per
> package
> > > > > > hw/i386: Rename apicid_from_topo_ids to
> x86_apicid_from_topo_ids
> > > > > > hw/386: Add EPYC mode topology decoding functions
> > > > > > target/i386: Cleanup and use the EPYC mode topology functions
> > > > > > hw/i386: Introduce apicid functions inside X86MachineState
> > > > > > i386: Introduce use_epyc_apic_id_encoding in X86CPUDefinition
> > > > > > hw/i386: Move arch_id decode inside x86_cpus_init
> > > > > > target/i386: Enable new apic id encoding for EPYC based cpus
> models
> > > > > > i386: Fix pkg_id offset for EPYC cpu models
> > > > > >
> > > > > >
> > > > > > hw/core/machine.c | 1
> > > > > > hw/i386/pc.c | 15 ++-
> > > > > > hw/i386/x86.c | 73 ++++++++++++----
> > > > > > include/hw/boards.h | 2
> > > > > > include/hw/i386/topology.h | 195
> ++++++++++++++++++++++++++++++---
> > > ---------
> > > > > > include/hw/i386/x86.h | 12 +++
> > > > > > softmmu/vl.c | 1
> > > > > > target/i386/cpu.c | 203 ++++++++++++++-----------------------------
> -
> > > > > > target/i386/cpu.h | 3 +
> > > > > > tests/test-x86-cpuid.c | 116 +++++++++++++++----------
> > > > > > 10 files changed, 358 insertions(+), 263 deletions(-)
> > > > > >
> > > > > > --
> > > > > > Signature
> > > > > >
> > > > >
> > > >
> > > > --
> > > > Eduardo
> > >
> > > --
> > > Eduardo
^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH v7 07/13] hw/386: Add EPYC mode topology decoding functions
2020-03-11 22:53 ` [PATCH v7 07/13] hw/386: Add EPYC mode topology decoding functions Babu Moger
2020-03-12 12:39 ` Igor Mammedov
@ 2020-06-02 17:18 ` Eduardo Habkost
2020-06-02 17:27 ` Babu Moger
2020-06-02 21:59 ` Babu Moger
1 sibling, 2 replies; 38+ messages in thread
From: Eduardo Habkost @ 2020-06-02 17:18 UTC (permalink / raw)
To: Babu Moger; +Cc: mst, qemu-devel, imammedo, pbonzini, rth
Hi,
It looks like this series breaks -device and CPU hotplug:
On Wed, Mar 11, 2020 at 05:53:34PM -0500, Babu Moger wrote:
> These functions add support for building EPYC mode topology given the smp
> details like numa nodes, cores, threads and sockets.
>
> The new apic id decoding is mostly similar to current apic id decoding
> except that it adds a new field node_id when numa configured. Removes all
> the hardcoded values. Subsequent patches will use these functions to build
> the topology.
>
> Following functions are added.
> apicid_llc_width_epyc
> apicid_llc_offset_epyc
> apicid_pkg_offset_epyc
> apicid_from_topo_ids_epyc
> x86_topo_ids_from_idx_epyc
> x86_topo_ids_from_apicid_epyc
> x86_apicid_from_cpu_idx_epyc
>
> The topology details are available in Processor Programming Reference (PPR)
> for AMD Family 17h Model 01h, Revision B1 Processors. The revision guides are
> available from the bugzilla Link below.
> Link: https://bugzilla.kernel.org/show_bug.cgi?id=206537
>
> Signed-off-by: Babu Moger <babu.moger@amd.com>
> Acked-by: Igor Mammedov <imammedo@redhat.com>
> Acked-by: Michael S. Tsirkin <mst@redhat.com>
> ---
[...]
> typedef struct X86CPUTopoIDs {
> unsigned pkg_id;
> + unsigned node_id;
You have added a new field here.
> unsigned die_id;
> unsigned core_id;
> unsigned smt_id;
[...]
> +static inline apic_id_t
> +x86_apicid_from_topo_ids_epyc(X86CPUTopoInfo *topo_info,
> + const X86CPUTopoIDs *topo_ids)
> +{
> + return (topo_ids->pkg_id << apicid_pkg_offset_epyc(topo_info)) |
> + (topo_ids->node_id << apicid_node_offset_epyc(topo_info)) |
You are using the new field here.
> + (topo_ids->die_id << apicid_die_offset(topo_info)) |
> + (topo_ids->core_id << apicid_core_offset(topo_info)) |
> + topo_ids->smt_id;
> +}
But you are not initializing node_id in one caller of apicid_from_topo_ids():
static void pc_cpu_pre_plug(HotplugHandler *hotplug_dev,
DeviceState *dev, Error **errp)
{
[...]
X86CPUTopoIDs topo_ids;
[...]
if (cpu->apic_id == UNASSIGNED_APIC_ID) {
[...]
topo_ids.pkg_id = cpu->socket_id;
topo_ids.die_id = cpu->die_id;
topo_ids.core_id = cpu->core_id;
topo_ids.smt_id = cpu->thread_id;
cpu->apic_id = x86ms->apicid_from_topo_ids(&topo_info, &topo_ids);
}
[...]
}
Result: -device is broken when using -cpu EPYC:
$ qemu-system-x86_64 -machine q35,accel=kvm -smp 1,maxcpus=2,cores=1,threads=1,sockets=2 -cpu EPYC -device EPYC-x86_64-cpu,core-id=0,socket-id=1,thread-id=0
qemu-system-x86_64: -device EPYC-x86_64-cpu,core-id=0,socket-id=1,thread-id=0: Invalid CPU [socket: 21855, die: 0, core: 0, thread: 0] with APIC ID 21855, valid index range 0:1
This happens because APIC ID is calculated using uninitialized
memory.
--
Eduardo
^ permalink raw reply [flat|nested] 38+ messages in thread
* RE: [PATCH v7 07/13] hw/386: Add EPYC mode topology decoding functions
2020-06-02 17:18 ` Eduardo Habkost
@ 2020-06-02 17:27 ` Babu Moger
2020-06-02 21:59 ` Babu Moger
1 sibling, 0 replies; 38+ messages in thread
From: Babu Moger @ 2020-06-02 17:27 UTC (permalink / raw)
To: Eduardo Habkost; +Cc: mst, qemu-devel, imammedo, pbonzini, rth
Oh. Ok.. Will look at it. thanks
> -----Original Message-----
> From: Eduardo Habkost <ehabkost@redhat.com>
> Sent: Tuesday, June 2, 2020 12:19 PM
> To: Moger, Babu <Babu.Moger@amd.com>
> Cc: marcel.apfelbaum@gmail.com; pbonzini@redhat.com; rth@twiddle.net;
> mst@redhat.com; imammedo@redhat.com; qemu-devel@nongnu.org
> Subject: Re: [PATCH v7 07/13] hw/386: Add EPYC mode topology decoding
> functions
>
> Hi,
>
> It looks like this series breaks -device and CPU hotplug:
>
> On Wed, Mar 11, 2020 at 05:53:34PM -0500, Babu Moger wrote:
> > These functions add support for building EPYC mode topology given the smp
> > details like numa nodes, cores, threads and sockets.
> >
> > The new apic id decoding is mostly similar to current apic id decoding
> > except that it adds a new field node_id when numa configured. Removes all
> > the hardcoded values. Subsequent patches will use these functions to build
> > the topology.
> >
> > Following functions are added.
> > apicid_llc_width_epyc
> > apicid_llc_offset_epyc
> > apicid_pkg_offset_epyc
> > apicid_from_topo_ids_epyc
> > x86_topo_ids_from_idx_epyc
> > x86_topo_ids_from_apicid_epyc
> > x86_apicid_from_cpu_idx_epyc
> >
> > The topology details are available in Processor Programming Reference (PPR)
> > for AMD Family 17h Model 01h, Revision B1 Processors. The revision guides
> are
> > available from the bugzilla Link below.
> > Link:
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugzilla.
> kernel.org%2Fshow_bug.cgi%3Fid%3D206537&data=02%7C01%7Cbabu.m
> oger%40amd.com%7C3487f40d37df4d59097d08d807190248%7C3dd8961fe488
> 4e608e11a82d994e183d%7C0%7C0%7C637267151289763739&sdata=wE0
> ukXIVh0l5eNQWsv6VDE9UZEVJmisofaW192gcZAI%3D&reserved=0
> >
> > Signed-off-by: Babu Moger <babu.moger@amd.com>
> > Acked-by: Igor Mammedov <imammedo@redhat.com>
> > Acked-by: Michael S. Tsirkin <mst@redhat.com>
> > ---
> [...]
> > typedef struct X86CPUTopoIDs {
> > unsigned pkg_id;
> > + unsigned node_id;
>
> You have added a new field here.
>
> > unsigned die_id;
> > unsigned core_id;
> > unsigned smt_id;
> [...]
> > +static inline apic_id_t
> > +x86_apicid_from_topo_ids_epyc(X86CPUTopoInfo *topo_info,
> > + const X86CPUTopoIDs *topo_ids)
> > +{
> > + return (topo_ids->pkg_id << apicid_pkg_offset_epyc(topo_info)) |
> > + (topo_ids->node_id << apicid_node_offset_epyc(topo_info)) |
>
> You are using the new field here.
>
> > + (topo_ids->die_id << apicid_die_offset(topo_info)) |
> > + (topo_ids->core_id << apicid_core_offset(topo_info)) |
> > + topo_ids->smt_id;
> > +}
>
> But you are not initializing node_id in one caller of apicid_from_topo_ids():
>
> static void pc_cpu_pre_plug(HotplugHandler *hotplug_dev,
> DeviceState *dev, Error **errp)
> {
> [...]
> X86CPUTopoIDs topo_ids;
> [...]
> if (cpu->apic_id == UNASSIGNED_APIC_ID) {
> [...]
> topo_ids.pkg_id = cpu->socket_id;
> topo_ids.die_id = cpu->die_id;
> topo_ids.core_id = cpu->core_id;
> topo_ids.smt_id = cpu->thread_id;
> cpu->apic_id = x86ms->apicid_from_topo_ids(&topo_info, &topo_ids);
> }
> [...]
> }
>
> Result: -device is broken when using -cpu EPYC:
>
> $ qemu-system-x86_64 -machine q35,accel=kvm -smp
> 1,maxcpus=2,cores=1,threads=1,sockets=2 -cpu EPYC -device EPYC-x86_64-
> cpu,core-id=0,socket-id=1,thread-id=0
> qemu-system-x86_64: -device EPYC-x86_64-cpu,core-id=0,socket-id=1,thread-
> id=0: Invalid CPU [socket: 21855, die: 0, core: 0, thread: 0] with APIC ID 21855,
> valid index range 0:1
>
> This happens because APIC ID is calculated using uninitialized
> memory.
>
> --
> Eduardo
^ permalink raw reply [flat|nested] 38+ messages in thread
* RE: [PATCH v7 07/13] hw/386: Add EPYC mode topology decoding functions
2020-06-02 17:18 ` Eduardo Habkost
2020-06-02 17:27 ` Babu Moger
@ 2020-06-02 21:59 ` Babu Moger
2020-06-02 23:01 ` Eduardo Habkost
1 sibling, 1 reply; 38+ messages in thread
From: Babu Moger @ 2020-06-02 21:59 UTC (permalink / raw)
To: Eduardo Habkost; +Cc: mst, qemu-devel, imammedo, pbonzini, rth
> -----Original Message-----
> From: Eduardo Habkost <ehabkost@redhat.com>
> Sent: Tuesday, June 2, 2020 12:19 PM
> To: Moger, Babu <Babu.Moger@amd.com>
> Cc: marcel.apfelbaum@gmail.com; pbonzini@redhat.com; rth@twiddle.net;
> mst@redhat.com; imammedo@redhat.com; qemu-devel@nongnu.org
> Subject: Re: [PATCH v7 07/13] hw/386: Add EPYC mode topology decoding
> functions
>
> Hi,
>
> It looks like this series breaks -device and CPU hotplug:
>
> On Wed, Mar 11, 2020 at 05:53:34PM -0500, Babu Moger wrote:
> > These functions add support for building EPYC mode topology given the smp
> > details like numa nodes, cores, threads and sockets.
> >
> > The new apic id decoding is mostly similar to current apic id decoding
> > except that it adds a new field node_id when numa configured. Removes all
> > the hardcoded values. Subsequent patches will use these functions to build
> > the topology.
> >
> > Following functions are added.
> > apicid_llc_width_epyc
> > apicid_llc_offset_epyc
> > apicid_pkg_offset_epyc
> > apicid_from_topo_ids_epyc
> > x86_topo_ids_from_idx_epyc
> > x86_topo_ids_from_apicid_epyc
> > x86_apicid_from_cpu_idx_epyc
> >
> > The topology details are available in Processor Programming Reference (PPR)
> > for AMD Family 17h Model 01h, Revision B1 Processors. The revision guides
> are
> > available from the bugzilla Link below.
> > Link:
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugzilla.
> kernel.org%2Fshow_bug.cgi%3Fid%3D206537&data=02%7C01%7Cbabu.m
> oger%40amd.com%7C3487f40d37df4d59097d08d807190248%7C3dd8961fe488
> 4e608e11a82d994e183d%7C0%7C0%7C637267151289763739&sdata=wE0
> ukXIVh0l5eNQWsv6VDE9UZEVJmisofaW192gcZAI%3D&reserved=0
> >
> > Signed-off-by: Babu Moger <babu.moger@amd.com>
> > Acked-by: Igor Mammedov <imammedo@redhat.com>
> > Acked-by: Michael S. Tsirkin <mst@redhat.com>
> > ---
> [...]
> > typedef struct X86CPUTopoIDs {
> > unsigned pkg_id;
> > + unsigned node_id;
>
> You have added a new field here.
>
> > unsigned die_id;
> > unsigned core_id;
> > unsigned smt_id;
> [...]
> > +static inline apic_id_t
> > +x86_apicid_from_topo_ids_epyc(X86CPUTopoInfo *topo_info,
> > + const X86CPUTopoIDs *topo_ids)
> > +{
> > + return (topo_ids->pkg_id << apicid_pkg_offset_epyc(topo_info)) |
> > + (topo_ids->node_id << apicid_node_offset_epyc(topo_info)) |
>
> You are using the new field here.
>
> > + (topo_ids->die_id << apicid_die_offset(topo_info)) |
> > + (topo_ids->core_id << apicid_core_offset(topo_info)) |
> > + topo_ids->smt_id;
> > +}
>
> But you are not initializing node_id in one caller of apicid_from_topo_ids():
>
> static void pc_cpu_pre_plug(HotplugHandler *hotplug_dev,
> DeviceState *dev, Error **errp)
> {
> [...]
> X86CPUTopoIDs topo_ids;
> [...]
> if (cpu->apic_id == UNASSIGNED_APIC_ID) {
> [...]
> topo_ids.pkg_id = cpu->socket_id;
> topo_ids.die_id = cpu->die_id;
> topo_ids.core_id = cpu->core_id;
> topo_ids.smt_id = cpu->thread_id;
> cpu->apic_id = x86ms->apicid_from_topo_ids(&topo_info, &topo_ids);
> }
> [...]
> }
>
> Result: -device is broken when using -cpu EPYC:
>
> $ qemu-system-x86_64 -machine q35,accel=kvm -smp
> 1,maxcpus=2,cores=1,threads=1,sockets=2 -cpu EPYC -device EPYC-x86_64-
> cpu,core-id=0,socket-id=1,thread-id=0
> qemu-system-x86_64: -device EPYC-x86_64-cpu,core-id=0,socket-id=1,thread-
> id=0: Invalid CPU [socket: 21855, die: 0, core: 0, thread: 0] with APIC ID 21855,
> valid index range 0:1
>
> This happens because APIC ID is calculated using uninitialized
> memory.
This patch should initialize the node_id. But I am not sure how to
reproduce the bug. Can you please send me the full command line to
reproduce the problem. Also test different options.
diff --git a/hw/i386/pc.c b/hw/i386/pc.c
index 2128f3d6fe..047b4b9391 100644
--- a/hw/i386/pc.c
+++ b/hw/i386/pc.c
@@ -1539,6 +1539,9 @@ static void pc_cpu_pre_plug(HotplugHandler *hotplug_dev,
if (cpu->apic_id == UNASSIGNED_APIC_ID) {
int max_socket = (ms->smp.max_cpus - 1) /
smp_threads / smp_cores / x86ms->smp_dies;
+ unsigned nr_nodes = MAX(topo_info.nodes_per_pkg, 1);
+ unsigned cores_per_node = DIV_ROUND_UP((x86ms->smp_dies * smp_cores *
+ smp_threads), nr_nodes);
/*
* die-id was optional in QEMU 4.0 and older, so keep it optional
@@ -1585,6 +1588,7 @@ static void pc_cpu_pre_plug(HotplugHandler *hotplug_dev,
topo_ids.die_id = cpu->die_id;
topo_ids.core_id = cpu->core_id;
topo_ids.smt_id = cpu->thread_id;
+ topo_ids.node_id = (cpu->core_id / cores_per_node) % nr_nodes;
cpu->apic_id = x86ms->apicid_from_topo_ids(&topo_info, &topo_ids);
}
^ permalink raw reply related [flat|nested] 38+ messages in thread
* Re: [PATCH v7 07/13] hw/386: Add EPYC mode topology decoding functions
2020-06-02 21:59 ` Babu Moger
@ 2020-06-02 23:01 ` Eduardo Habkost
2020-06-03 0:07 ` Babu Moger
2020-06-03 15:22 ` Babu Moger
0 siblings, 2 replies; 38+ messages in thread
From: Eduardo Habkost @ 2020-06-02 23:01 UTC (permalink / raw)
To: Babu Moger; +Cc: mst, qemu-devel, imammedo, pbonzini, rth
On Tue, Jun 02, 2020 at 04:59:19PM -0500, Babu Moger wrote:
>
>
> > -----Original Message-----
> > From: Eduardo Habkost <ehabkost@redhat.com>
> > Sent: Tuesday, June 2, 2020 12:19 PM
> > To: Moger, Babu <Babu.Moger@amd.com>
> > Cc: marcel.apfelbaum@gmail.com; pbonzini@redhat.com; rth@twiddle.net;
> > mst@redhat.com; imammedo@redhat.com; qemu-devel@nongnu.org
> > Subject: Re: [PATCH v7 07/13] hw/386: Add EPYC mode topology decoding
> > functions
> >
> > Hi,
> >
> > It looks like this series breaks -device and CPU hotplug:
> >
> > On Wed, Mar 11, 2020 at 05:53:34PM -0500, Babu Moger wrote:
> > > These functions add support for building EPYC mode topology given the smp
> > > details like numa nodes, cores, threads and sockets.
> > >
> > > The new apic id decoding is mostly similar to current apic id decoding
> > > except that it adds a new field node_id when numa configured. Removes all
> > > the hardcoded values. Subsequent patches will use these functions to build
> > > the topology.
> > >
> > > Following functions are added.
> > > apicid_llc_width_epyc
> > > apicid_llc_offset_epyc
> > > apicid_pkg_offset_epyc
> > > apicid_from_topo_ids_epyc
> > > x86_topo_ids_from_idx_epyc
> > > x86_topo_ids_from_apicid_epyc
> > > x86_apicid_from_cpu_idx_epyc
> > >
> > > The topology details are available in Processor Programming Reference (PPR)
> > > for AMD Family 17h Model 01h, Revision B1 Processors. The revision guides
> > are
> > > available from the bugzilla Link below.
> > > Link:
> > https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugzilla.
> > kernel.org%2Fshow_bug.cgi%3Fid%3D206537&data=02%7C01%7Cbabu.m
> > oger%40amd.com%7C3487f40d37df4d59097d08d807190248%7C3dd8961fe488
> > 4e608e11a82d994e183d%7C0%7C0%7C637267151289763739&sdata=wE0
> > ukXIVh0l5eNQWsv6VDE9UZEVJmisofaW192gcZAI%3D&reserved=0
> > >
> > > Signed-off-by: Babu Moger <babu.moger@amd.com>
> > > Acked-by: Igor Mammedov <imammedo@redhat.com>
> > > Acked-by: Michael S. Tsirkin <mst@redhat.com>
> > > ---
> > [...]
> > > typedef struct X86CPUTopoIDs {
> > > unsigned pkg_id;
> > > + unsigned node_id;
> >
> > You have added a new field here.
> >
> > > unsigned die_id;
> > > unsigned core_id;
> > > unsigned smt_id;
> > [...]
> > > +static inline apic_id_t
> > > +x86_apicid_from_topo_ids_epyc(X86CPUTopoInfo *topo_info,
> > > + const X86CPUTopoIDs *topo_ids)
> > > +{
> > > + return (topo_ids->pkg_id << apicid_pkg_offset_epyc(topo_info)) |
> > > + (topo_ids->node_id << apicid_node_offset_epyc(topo_info)) |
> >
> > You are using the new field here.
> >
> > > + (topo_ids->die_id << apicid_die_offset(topo_info)) |
> > > + (topo_ids->core_id << apicid_core_offset(topo_info)) |
> > > + topo_ids->smt_id;
> > > +}
> >
> > But you are not initializing node_id in one caller of apicid_from_topo_ids():
> >
> > static void pc_cpu_pre_plug(HotplugHandler *hotplug_dev,
> > DeviceState *dev, Error **errp)
> > {
> > [...]
> > X86CPUTopoIDs topo_ids;
> > [...]
> > if (cpu->apic_id == UNASSIGNED_APIC_ID) {
> > [...]
> > topo_ids.pkg_id = cpu->socket_id;
> > topo_ids.die_id = cpu->die_id;
> > topo_ids.core_id = cpu->core_id;
> > topo_ids.smt_id = cpu->thread_id;
> > cpu->apic_id = x86ms->apicid_from_topo_ids(&topo_info, &topo_ids);
> > }
> > [...]
> > }
> >
> > Result: -device is broken when using -cpu EPYC:
> >
> > $ qemu-system-x86_64 -machine q35,accel=kvm -smp
> > 1,maxcpus=2,cores=1,threads=1,sockets=2 -cpu EPYC -device EPYC-x86_64-
> > cpu,core-id=0,socket-id=1,thread-id=0
[1]
> > qemu-system-x86_64: -device EPYC-x86_64-cpu,core-id=0,socket-id=1,thread-
> > id=0: Invalid CPU [socket: 21855, die: 0, core: 0, thread: 0] with APIC ID 21855,
> > valid index range 0:1
> >
> > This happens because APIC ID is calculated using uninitialized
> > memory.
> This patch should initialize the node_id. But I am not sure how to
> reproduce the bug. Can you please send me the full command line to
> reproduce the problem. Also test different options.
The full command line is above[1].
>
> diff --git a/hw/i386/pc.c b/hw/i386/pc.c
> index 2128f3d6fe..047b4b9391 100644
> --- a/hw/i386/pc.c
> +++ b/hw/i386/pc.c
> @@ -1539,6 +1539,9 @@ static void pc_cpu_pre_plug(HotplugHandler *hotplug_dev,
> if (cpu->apic_id == UNASSIGNED_APIC_ID) {
> int max_socket = (ms->smp.max_cpus - 1) /
> smp_threads / smp_cores / x86ms->smp_dies;
So, here's the input you are using to calculate topo_ids.node_id:
> + unsigned nr_nodes = MAX(topo_info.nodes_per_pkg, 1);
When is topo_info.nodes_per_pkg allowed to be 0?
> + unsigned cores_per_node = DIV_ROUND_UP((x86ms->smp_dies * smp_cores *
> + smp_threads), nr_nodes);
x86ms->smp_dies should be available at topo_info.dies_per_pkg,
smp_cores should available at topo_info.cores_per_die,
smp_threads should be available at topo_info.threads_per_core,
nr_nodes should be available at topo_info.nodes_per_pkg.
>
> /*
> * die-id was optional in QEMU 4.0 and older, so keep it optional
> @@ -1585,6 +1588,7 @@ static void pc_cpu_pre_plug(HotplugHandler *hotplug_dev,
> topo_ids.die_id = cpu->die_id;
> topo_ids.core_id = cpu->core_id;
> topo_ids.smt_id = cpu->thread_id;
> + topo_ids.node_id = (cpu->core_id / cores_per_node) % nr_nodes;
apicid_from_topo_ids() have access to topo_info and topo_ids, If
all the information you need to calculate node_id is already
available inside topo_info + topo_ids, we could be calculating it
inside apicid_from_topo_ids(). Why don't we do it?
Also, is topo_ids.core_id really allowed to be larger than
cores_per_node when calling apicid_from_topo_ids()?
> cpu->apic_id = x86ms->apicid_from_topo_ids(&topo_info, &topo_ids);
> }
>
--
Eduardo
^ permalink raw reply [flat|nested] 38+ messages in thread
* RE: [PATCH v7 07/13] hw/386: Add EPYC mode topology decoding functions
2020-06-02 23:01 ` Eduardo Habkost
@ 2020-06-03 0:07 ` Babu Moger
2020-06-03 15:22 ` Babu Moger
1 sibling, 0 replies; 38+ messages in thread
From: Babu Moger @ 2020-06-03 0:07 UTC (permalink / raw)
To: Eduardo Habkost; +Cc: mst, qemu-devel, imammedo, pbonzini, rth
> -----Original Message-----
> From: Eduardo Habkost <ehabkost@redhat.com>
> Sent: Tuesday, June 2, 2020 6:01 PM
> To: Moger, Babu <Babu.Moger@amd.com>
> Cc: marcel.apfelbaum@gmail.com; pbonzini@redhat.com; rth@twiddle.net;
> mst@redhat.com; imammedo@redhat.com; qemu-devel@nongnu.org
> Subject: Re: [PATCH v7 07/13] hw/386: Add EPYC mode topology decoding
> functions
>
> On Tue, Jun 02, 2020 at 04:59:19PM -0500, Babu Moger wrote:
> >
> >
> > > -----Original Message-----
> > > From: Eduardo Habkost <ehabkost@redhat.com>
> > > Sent: Tuesday, June 2, 2020 12:19 PM
> > > To: Moger, Babu <Babu.Moger@amd.com>
> > > Cc: marcel.apfelbaum@gmail.com; pbonzini@redhat.com; rth@twiddle.net;
> > > mst@redhat.com; imammedo@redhat.com; qemu-devel@nongnu.org
> > > Subject: Re: [PATCH v7 07/13] hw/386: Add EPYC mode topology decoding
> > > functions
> > >
> > > Hi,
> > >
> > > It looks like this series breaks -device and CPU hotplug:
> > >
> > > On Wed, Mar 11, 2020 at 05:53:34PM -0500, Babu Moger wrote:
> > > > These functions add support for building EPYC mode topology given the
> smp
> > > > details like numa nodes, cores, threads and sockets.
> > > >
> > > > The new apic id decoding is mostly similar to current apic id decoding
> > > > except that it adds a new field node_id when numa configured. Removes
> all
> > > > the hardcoded values. Subsequent patches will use these functions to build
> > > > the topology.
> > > >
> > > > Following functions are added.
> > > > apicid_llc_width_epyc
> > > > apicid_llc_offset_epyc
> > > > apicid_pkg_offset_epyc
> > > > apicid_from_topo_ids_epyc
> > > > x86_topo_ids_from_idx_epyc
> > > > x86_topo_ids_from_apicid_epyc
> > > > x86_apicid_from_cpu_idx_epyc
> > > >
> > > > The topology details are available in Processor Programming Reference
> (PPR)
> > > > for AMD Family 17h Model 01h, Revision B1 Processors. The revision
> guides
> > > are
> > > > available from the bugzilla Link below.
> > > > Link:
> > >
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugzilla.
> > >
> kernel.org%2Fshow_bug.cgi%3Fid%3D206537&data=02%7C01%7Cbabu.m
> > >
> oger%40amd.com%7C3487f40d37df4d59097d08d807190248%7C3dd8961fe488
> > >
> 4e608e11a82d994e183d%7C0%7C0%7C637267151289763739&sdata=wE0
> > > ukXIVh0l5eNQWsv6VDE9UZEVJmisofaW192gcZAI%3D&reserved=0
> > > >
> > > > Signed-off-by: Babu Moger <babu.moger@amd.com>
> > > > Acked-by: Igor Mammedov <imammedo@redhat.com>
> > > > Acked-by: Michael S. Tsirkin <mst@redhat.com>
> > > > ---
> > > [...]
> > > > typedef struct X86CPUTopoIDs {
> > > > unsigned pkg_id;
> > > > + unsigned node_id;
> > >
> > > You have added a new field here.
> > >
> > > > unsigned die_id;
> > > > unsigned core_id;
> > > > unsigned smt_id;
> > > [...]
> > > > +static inline apic_id_t
> > > > +x86_apicid_from_topo_ids_epyc(X86CPUTopoInfo *topo_info,
> > > > + const X86CPUTopoIDs *topo_ids)
> > > > +{
> > > > + return (topo_ids->pkg_id << apicid_pkg_offset_epyc(topo_info)) |
> > > > + (topo_ids->node_id << apicid_node_offset_epyc(topo_info)) |
> > >
> > > You are using the new field here.
> > >
> > > > + (topo_ids->die_id << apicid_die_offset(topo_info)) |
> > > > + (topo_ids->core_id << apicid_core_offset(topo_info)) |
> > > > + topo_ids->smt_id;
> > > > +}
> > >
> > > But you are not initializing node_id in one caller of apicid_from_topo_ids():
> > >
> > > static void pc_cpu_pre_plug(HotplugHandler *hotplug_dev,
> > > DeviceState *dev, Error **errp)
> > > {
> > > [...]
> > > X86CPUTopoIDs topo_ids;
> > > [...]
> > > if (cpu->apic_id == UNASSIGNED_APIC_ID) {
> > > [...]
> > > topo_ids.pkg_id = cpu->socket_id;
> > > topo_ids.die_id = cpu->die_id;
> > > topo_ids.core_id = cpu->core_id;
> > > topo_ids.smt_id = cpu->thread_id;
> > > cpu->apic_id = x86ms->apicid_from_topo_ids(&topo_info, &topo_ids);
> > > }
> > > [...]
> > > }
> > >
> > > Result: -device is broken when using -cpu EPYC:
> > >
> > > $ qemu-system-x86_64 -machine q35,accel=kvm -smp
> > > 1,maxcpus=2,cores=1,threads=1,sockets=2 -cpu EPYC -device EPYC-x86_64-
> > > cpu,core-id=0,socket-id=1,thread-id=0
>
> [1]
>
> > > qemu-system-x86_64: -device EPYC-x86_64-cpu,core-id=0,socket-
> id=1,thread-
> > > id=0: Invalid CPU [socket: 21855, die: 0, core: 0, thread: 0] with APIC ID
> 21855,
> > > valid index range 0:1
> > >
> > > This happens because APIC ID is calculated using uninitialized
> > > memory.
> > This patch should initialize the node_id. But I am not sure how to
> > reproduce the bug. Can you please send me the full command line to
> > reproduce the problem. Also test different options.
>
> The full command line is above[1].
>
>
> >
> > diff --git a/hw/i386/pc.c b/hw/i386/pc.c
> > index 2128f3d6fe..047b4b9391 100644
> > --- a/hw/i386/pc.c
> > +++ b/hw/i386/pc.c
> > @@ -1539,6 +1539,9 @@ static void pc_cpu_pre_plug(HotplugHandler
> *hotplug_dev,
> > if (cpu->apic_id == UNASSIGNED_APIC_ID) {
> > int max_socket = (ms->smp.max_cpus - 1) /
> > smp_threads / smp_cores / x86ms->smp_dies;
>
> So, here's the input you are using to calculate topo_ids.node_id:
>
> > + unsigned nr_nodes = MAX(topo_info.nodes_per_pkg, 1);
>
> When is topo_info.nodes_per_pkg allowed to be 0?
>
> > + unsigned cores_per_node = DIV_ROUND_UP((x86ms->smp_dies *
> smp_cores *
> > + smp_threads), nr_nodes);
>
> x86ms->smp_dies should be available at topo_info.dies_per_pkg,
> smp_cores should available at topo_info.cores_per_die,
> smp_threads should be available at topo_info.threads_per_core,
> nr_nodes should be available at topo_info.nodes_per_pkg.
>
> >
> > /*
> > * die-id was optional in QEMU 4.0 and older, so keep it optional
> > @@ -1585,6 +1588,7 @@ static void pc_cpu_pre_plug(HotplugHandler
> *hotplug_dev,
> > topo_ids.die_id = cpu->die_id;
> > topo_ids.core_id = cpu->core_id;
> > topo_ids.smt_id = cpu->thread_id;
> > + topo_ids.node_id = (cpu->core_id / cores_per_node) % nr_nodes;
>
> apicid_from_topo_ids() have access to topo_info and topo_ids, If
> all the information you need to calculate node_id is already
> available inside topo_info + topo_ids, we could be calculating it
> inside apicid_from_topo_ids(). Why don't we do it?
Sure. I will try it. Will post revised patch tomorrow.
>
> Also, is topo_ids.core_id really allowed to be larger than
> cores_per_node when calling apicid_from_topo_ids()?
>
> > cpu->apic_id = x86ms->apicid_from_topo_ids(&topo_info, &topo_ids);
> > }
> >
>
> --
> Eduardo
^ permalink raw reply [flat|nested] 38+ messages in thread
* RE: [PATCH v7 07/13] hw/386: Add EPYC mode topology decoding functions
2020-06-02 23:01 ` Eduardo Habkost
2020-06-03 0:07 ` Babu Moger
@ 2020-06-03 15:22 ` Babu Moger
2020-06-03 15:31 ` Eduardo Habkost
1 sibling, 1 reply; 38+ messages in thread
From: Babu Moger @ 2020-06-03 15:22 UTC (permalink / raw)
To: Eduardo Habkost; +Cc: mst, qemu-devel, imammedo, pbonzini, rth
> -----Original Message-----
> From: Eduardo Habkost <ehabkost@redhat.com>
> Sent: Tuesday, June 2, 2020 6:01 PM
> To: Moger, Babu <Babu.Moger@amd.com>
> Cc: marcel.apfelbaum@gmail.com; pbonzini@redhat.com; rth@twiddle.net;
> mst@redhat.com; imammedo@redhat.com; qemu-devel@nongnu.org
> Subject: Re: [PATCH v7 07/13] hw/386: Add EPYC mode topology decoding
> functions
>
> On Tue, Jun 02, 2020 at 04:59:19PM -0500, Babu Moger wrote:
> >
> >
> > > -----Original Message-----
> > > From: Eduardo Habkost <ehabkost@redhat.com>
> > > Sent: Tuesday, June 2, 2020 12:19 PM
> > > To: Moger, Babu <Babu.Moger@amd.com>
> > > Cc: marcel.apfelbaum@gmail.com; pbonzini@redhat.com; rth@twiddle.net;
> > > mst@redhat.com; imammedo@redhat.com; qemu-devel@nongnu.org
> > > Subject: Re: [PATCH v7 07/13] hw/386: Add EPYC mode topology decoding
> > > functions
> > >
> > > Hi,
> > >
> > > It looks like this series breaks -device and CPU hotplug:
> > >
> > > On Wed, Mar 11, 2020 at 05:53:34PM -0500, Babu Moger wrote:
> > > > These functions add support for building EPYC mode topology given the
> smp
> > > > details like numa nodes, cores, threads and sockets.
> > > >
> > > > The new apic id decoding is mostly similar to current apic id decoding
> > > > except that it adds a new field node_id when numa configured. Removes
> all
> > > > the hardcoded values. Subsequent patches will use these functions to build
> > > > the topology.
> > > >
> > > > Following functions are added.
> > > > apicid_llc_width_epyc
> > > > apicid_llc_offset_epyc
> > > > apicid_pkg_offset_epyc
> > > > apicid_from_topo_ids_epyc
> > > > x86_topo_ids_from_idx_epyc
> > > > x86_topo_ids_from_apicid_epyc
> > > > x86_apicid_from_cpu_idx_epyc
> > > >
> > > > The topology details are available in Processor Programming Reference
> (PPR)
> > > > for AMD Family 17h Model 01h, Revision B1 Processors. The revision
> guides
> > > are
> > > > available from the bugzilla Link below.
> > > > Link:
> > >
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugzilla.
> > >
> kernel.org%2Fshow_bug.cgi%3Fid%3D206537&data=02%7C01%7Cbabu.m
> > >
> oger%40amd.com%7C3487f40d37df4d59097d08d807190248%7C3dd8961fe488
> > >
> 4e608e11a82d994e183d%7C0%7C0%7C637267151289763739&sdata=wE0
> > > ukXIVh0l5eNQWsv6VDE9UZEVJmisofaW192gcZAI%3D&reserved=0
> > > >
> > > > Signed-off-by: Babu Moger <babu.moger@amd.com>
> > > > Acked-by: Igor Mammedov <imammedo@redhat.com>
> > > > Acked-by: Michael S. Tsirkin <mst@redhat.com>
> > > > ---
> > > [...]
> > > > typedef struct X86CPUTopoIDs {
> > > > unsigned pkg_id;
> > > > + unsigned node_id;
> > >
> > > You have added a new field here.
> > >
> > > > unsigned die_id;
> > > > unsigned core_id;
> > > > unsigned smt_id;
> > > [...]
> > > > +static inline apic_id_t
> > > > +x86_apicid_from_topo_ids_epyc(X86CPUTopoInfo *topo_info,
> > > > + const X86CPUTopoIDs *topo_ids)
> > > > +{
> > > > + return (topo_ids->pkg_id << apicid_pkg_offset_epyc(topo_info)) |
> > > > + (topo_ids->node_id << apicid_node_offset_epyc(topo_info)) |
> > >
> > > You are using the new field here.
> > >
> > > > + (topo_ids->die_id << apicid_die_offset(topo_info)) |
> > > > + (topo_ids->core_id << apicid_core_offset(topo_info)) |
> > > > + topo_ids->smt_id;
> > > > +}
> > >
> > > But you are not initializing node_id in one caller of apicid_from_topo_ids():
> > >
> > > static void pc_cpu_pre_plug(HotplugHandler *hotplug_dev,
> > > DeviceState *dev, Error **errp)
> > > {
> > > [...]
> > > X86CPUTopoIDs topo_ids;
> > > [...]
> > > if (cpu->apic_id == UNASSIGNED_APIC_ID) {
> > > [...]
> > > topo_ids.pkg_id = cpu->socket_id;
> > > topo_ids.die_id = cpu->die_id;
> > > topo_ids.core_id = cpu->core_id;
> > > topo_ids.smt_id = cpu->thread_id;
> > > cpu->apic_id = x86ms->apicid_from_topo_ids(&topo_info, &topo_ids);
> > > }
> > > [...]
> > > }
> > >
> > > Result: -device is broken when using -cpu EPYC:
> > >
> > > $ qemu-system-x86_64 -machine q35,accel=kvm -smp
> > > 1,maxcpus=2,cores=1,threads=1,sockets=2 -cpu EPYC -device EPYC-x86_64-
> > > cpu,core-id=0,socket-id=1,thread-id=0
>
> [1]
>
> > > qemu-system-x86_64: -device EPYC-x86_64-cpu,core-id=0,socket-
> id=1,thread-
> > > id=0: Invalid CPU [socket: 21855, die: 0, core: 0, thread: 0] with APIC ID
> 21855,
> > > valid index range 0:1
> > >
> > > This happens because APIC ID is calculated using uninitialized
> > > memory.
> > This patch should initialize the node_id. But I am not sure how to
> > reproduce the bug. Can you please send me the full command line to
> > reproduce the problem. Also test different options.
>
> The full command line is above[1].
I just picked up the latest tree from
git clone https://git.qemu.org/git/qemu.git qemu
Not seeing the problem.
./x86_64-softmmu/qemu-system-x86_64 -machine q35,accel=kvm -smp
1,maxcpus=2,cores=1,threads=1,sockets=2 -cpu EPYC -device
EPYC-x86_64-cpu,core-id=0,socket-id=1,thread-id=0
VNC server running on ::1:5900
It appears to run ok.
>
>
> >
> > diff --git a/hw/i386/pc.c b/hw/i386/pc.c
> > index 2128f3d6fe..047b4b9391 100644
> > --- a/hw/i386/pc.c
> > +++ b/hw/i386/pc.c
> > @@ -1539,6 +1539,9 @@ static void pc_cpu_pre_plug(HotplugHandler
> *hotplug_dev,
> > if (cpu->apic_id == UNASSIGNED_APIC_ID) {
> > int max_socket = (ms->smp.max_cpus - 1) /
> > smp_threads / smp_cores / x86ms->smp_dies;
>
> So, here's the input you are using to calculate topo_ids.node_id:
>
> > + unsigned nr_nodes = MAX(topo_info.nodes_per_pkg, 1);
>
> When is topo_info.nodes_per_pkg allowed to be 0?
It will be zero if numa is not configured. Node_id will be zero for all
the cores.
>
> > + unsigned cores_per_node = DIV_ROUND_UP((x86ms->smp_dies *
> smp_cores *
> > + smp_threads), nr_nodes);
>
> x86ms->smp_dies should be available at topo_info.dies_per_pkg,
> smp_cores should available at topo_info.cores_per_die,
> smp_threads should be available at topo_info.threads_per_core,
> nr_nodes should be available at topo_info.nodes_per_pkg.
>
> >
> > /*
> > * die-id was optional in QEMU 4.0 and older, so keep it optional
> > @@ -1585,6 +1588,7 @@ static void pc_cpu_pre_plug(HotplugHandler
> *hotplug_dev,
> > topo_ids.die_id = cpu->die_id;
> > topo_ids.core_id = cpu->core_id;
> > topo_ids.smt_id = cpu->thread_id;
> > + topo_ids.node_id = (cpu->core_id / cores_per_node) % nr_nodes;
>
> apicid_from_topo_ids() have access to topo_info and topo_ids, If
> all the information you need to calculate node_id is already
> available inside topo_info + topo_ids, we could be calculating it
> inside apicid_from_topo_ids(). Why don't we do it?
>
> Also, is topo_ids.core_id really allowed to be larger than
> cores_per_node when calling apicid_from_topo_ids()?
Yes. It is. If we have two numa nodes and 8 cores. Then cores_per_node is
4. Nr_nodes =2. Yes. Core_id can be larger than cores_per_node.
>
> > cpu->apic_id = x86ms->apicid_from_topo_ids(&topo_info, &topo_ids);
> > }
> >
>
> --
> Eduardo
^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH v7 07/13] hw/386: Add EPYC mode topology decoding functions
2020-06-03 15:22 ` Babu Moger
@ 2020-06-03 15:31 ` Eduardo Habkost
2020-06-03 15:38 ` Babu Moger
0 siblings, 1 reply; 38+ messages in thread
From: Eduardo Habkost @ 2020-06-03 15:31 UTC (permalink / raw)
To: Babu Moger; +Cc: mst, qemu-devel, imammedo, pbonzini, rth
On Wed, Jun 03, 2020 at 10:22:10AM -0500, Babu Moger wrote:
>
>
> > -----Original Message-----
> > From: Eduardo Habkost <ehabkost@redhat.com>
> > Sent: Tuesday, June 2, 2020 6:01 PM
> > To: Moger, Babu <Babu.Moger@amd.com>
> > Cc: marcel.apfelbaum@gmail.com; pbonzini@redhat.com; rth@twiddle.net;
> > mst@redhat.com; imammedo@redhat.com; qemu-devel@nongnu.org
> > Subject: Re: [PATCH v7 07/13] hw/386: Add EPYC mode topology decoding
> > functions
> >
> > On Tue, Jun 02, 2020 at 04:59:19PM -0500, Babu Moger wrote:
> > >
> > >
> > > > -----Original Message-----
> > > > From: Eduardo Habkost <ehabkost@redhat.com>
> > > > Sent: Tuesday, June 2, 2020 12:19 PM
> > > > To: Moger, Babu <Babu.Moger@amd.com>
> > > > Cc: marcel.apfelbaum@gmail.com; pbonzini@redhat.com; rth@twiddle.net;
> > > > mst@redhat.com; imammedo@redhat.com; qemu-devel@nongnu.org
> > > > Subject: Re: [PATCH v7 07/13] hw/386: Add EPYC mode topology decoding
> > > > functions
> > > >
> > > > Hi,
> > > >
> > > > It looks like this series breaks -device and CPU hotplug:
> > > >
> > > > On Wed, Mar 11, 2020 at 05:53:34PM -0500, Babu Moger wrote:
> > > > > These functions add support for building EPYC mode topology given the
> > smp
> > > > > details like numa nodes, cores, threads and sockets.
> > > > >
> > > > > The new apic id decoding is mostly similar to current apic id decoding
> > > > > except that it adds a new field node_id when numa configured. Removes
> > all
> > > > > the hardcoded values. Subsequent patches will use these functions to build
> > > > > the topology.
> > > > >
> > > > > Following functions are added.
> > > > > apicid_llc_width_epyc
> > > > > apicid_llc_offset_epyc
> > > > > apicid_pkg_offset_epyc
> > > > > apicid_from_topo_ids_epyc
> > > > > x86_topo_ids_from_idx_epyc
> > > > > x86_topo_ids_from_apicid_epyc
> > > > > x86_apicid_from_cpu_idx_epyc
> > > > >
> > > > > The topology details are available in Processor Programming Reference
> > (PPR)
> > > > > for AMD Family 17h Model 01h, Revision B1 Processors. The revision
> > guides
> > > > are
> > > > > available from the bugzilla Link below.
> > > > > Link:
> > > >
> > https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugzilla.
> > > >
> > kernel.org%2Fshow_bug.cgi%3Fid%3D206537&data=02%7C01%7Cbabu.m
> > > >
> > oger%40amd.com%7C3487f40d37df4d59097d08d807190248%7C3dd8961fe488
> > > >
> > 4e608e11a82d994e183d%7C0%7C0%7C637267151289763739&sdata=wE0
> > > > ukXIVh0l5eNQWsv6VDE9UZEVJmisofaW192gcZAI%3D&reserved=0
> > > > >
> > > > > Signed-off-by: Babu Moger <babu.moger@amd.com>
> > > > > Acked-by: Igor Mammedov <imammedo@redhat.com>
> > > > > Acked-by: Michael S. Tsirkin <mst@redhat.com>
> > > > > ---
> > > > [...]
> > > > > typedef struct X86CPUTopoIDs {
> > > > > unsigned pkg_id;
> > > > > + unsigned node_id;
> > > >
> > > > You have added a new field here.
> > > >
> > > > > unsigned die_id;
> > > > > unsigned core_id;
> > > > > unsigned smt_id;
> > > > [...]
> > > > > +static inline apic_id_t
> > > > > +x86_apicid_from_topo_ids_epyc(X86CPUTopoInfo *topo_info,
> > > > > + const X86CPUTopoIDs *topo_ids)
> > > > > +{
> > > > > + return (topo_ids->pkg_id << apicid_pkg_offset_epyc(topo_info)) |
> > > > > + (topo_ids->node_id << apicid_node_offset_epyc(topo_info)) |
> > > >
> > > > You are using the new field here.
> > > >
> > > > > + (topo_ids->die_id << apicid_die_offset(topo_info)) |
> > > > > + (topo_ids->core_id << apicid_core_offset(topo_info)) |
> > > > > + topo_ids->smt_id;
> > > > > +}
> > > >
> > > > But you are not initializing node_id in one caller of apicid_from_topo_ids():
> > > >
> > > > static void pc_cpu_pre_plug(HotplugHandler *hotplug_dev,
> > > > DeviceState *dev, Error **errp)
> > > > {
> > > > [...]
> > > > X86CPUTopoIDs topo_ids;
> > > > [...]
> > > > if (cpu->apic_id == UNASSIGNED_APIC_ID) {
> > > > [...]
> > > > topo_ids.pkg_id = cpu->socket_id;
> > > > topo_ids.die_id = cpu->die_id;
> > > > topo_ids.core_id = cpu->core_id;
> > > > topo_ids.smt_id = cpu->thread_id;
> > > > cpu->apic_id = x86ms->apicid_from_topo_ids(&topo_info, &topo_ids);
> > > > }
> > > > [...]
> > > > }
> > > >
> > > > Result: -device is broken when using -cpu EPYC:
> > > >
> > > > $ qemu-system-x86_64 -machine q35,accel=kvm -smp
> > > > 1,maxcpus=2,cores=1,threads=1,sockets=2 -cpu EPYC -device EPYC-x86_64-
> > > > cpu,core-id=0,socket-id=1,thread-id=0
> >
> > [1]
> >
> > > > qemu-system-x86_64: -device EPYC-x86_64-cpu,core-id=0,socket-
> > id=1,thread-
> > > > id=0: Invalid CPU [socket: 21855, die: 0, core: 0, thread: 0] with APIC ID
> > 21855,
> > > > valid index range 0:1
> > > >
> > > > This happens because APIC ID is calculated using uninitialized
> > > > memory.
> > > This patch should initialize the node_id. But I am not sure how to
> > > reproduce the bug. Can you please send me the full command line to
> > > reproduce the problem. Also test different options.
> >
> > The full command line is above[1].
>
> I just picked up the latest tree from
> git clone https://git.qemu.org/git/qemu.git qemu
> Not seeing the problem.
>
> ./x86_64-softmmu/qemu-system-x86_64 -machine q35,accel=kvm -smp
> 1,maxcpus=2,cores=1,threads=1,sockets=2 -cpu EPYC -device
> EPYC-x86_64-cpu,core-id=0,socket-id=1,thread-id=0
> VNC server running on ::1:5900
>
> It appears to run ok.
>
> >
> >
> > >
> > > diff --git a/hw/i386/pc.c b/hw/i386/pc.c
> > > index 2128f3d6fe..047b4b9391 100644
> > > --- a/hw/i386/pc.c
> > > +++ b/hw/i386/pc.c
> > > @@ -1539,6 +1539,9 @@ static void pc_cpu_pre_plug(HotplugHandler
> > *hotplug_dev,
> > > if (cpu->apic_id == UNASSIGNED_APIC_ID) {
> > > int max_socket = (ms->smp.max_cpus - 1) /
> > > smp_threads / smp_cores / x86ms->smp_dies;
> >
> > So, here's the input you are using to calculate topo_ids.node_id:
> >
> > > + unsigned nr_nodes = MAX(topo_info.nodes_per_pkg, 1);
> >
> > When is topo_info.nodes_per_pkg allowed to be 0?
>
> It will be zero if numa is not configured. Node_id will be zero for all
> the cores.
>
> >
> > > + unsigned cores_per_node = DIV_ROUND_UP((x86ms->smp_dies *
> > smp_cores *
> > > + smp_threads), nr_nodes);
> >
> > x86ms->smp_dies should be available at topo_info.dies_per_pkg,
> > smp_cores should available at topo_info.cores_per_die,
> > smp_threads should be available at topo_info.threads_per_core,
> > nr_nodes should be available at topo_info.nodes_per_pkg.
> >
> > >
> > > /*
> > > * die-id was optional in QEMU 4.0 and older, so keep it optional
> > > @@ -1585,6 +1588,7 @@ static void pc_cpu_pre_plug(HotplugHandler
> > *hotplug_dev,
> > > topo_ids.die_id = cpu->die_id;
> > > topo_ids.core_id = cpu->core_id;
> > > topo_ids.smt_id = cpu->thread_id;
> > > + topo_ids.node_id = (cpu->core_id / cores_per_node) % nr_nodes;
> >
> > apicid_from_topo_ids() have access to topo_info and topo_ids, If
> > all the information you need to calculate node_id is already
> > available inside topo_info + topo_ids, we could be calculating it
> > inside apicid_from_topo_ids(). Why don't we do it?
> >
> > Also, is topo_ids.core_id really allowed to be larger than
> > cores_per_node when calling apicid_from_topo_ids()?
>
> Yes. It is. If we have two numa nodes and 8 cores. Then cores_per_node is
> 4. Nr_nodes =2. Yes. Core_id can be larger than cores_per_node.
I assumed core_id identified the core inside a die, and the range
would be [0, cores_per_die). This seems to be what
apicid_from_topo_ids_epyc() expects.
We need clearer documentation on the semantics of each *_id
field, to avoid confusion.
>
> >
> > > cpu->apic_id =
> > > x86ms->apicid_from_topo_ids(&topo_info,
> > > &topo_ids); }
> > >
> >
> > -- Eduardo
>
--
Eduardo
^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH v7 07/13] hw/386: Add EPYC mode topology decoding functions
2020-06-03 15:31 ` Eduardo Habkost
@ 2020-06-03 15:38 ` Babu Moger
2020-06-03 15:45 ` Eduardo Habkost
0 siblings, 1 reply; 38+ messages in thread
From: Babu Moger @ 2020-06-03 15:38 UTC (permalink / raw)
To: Eduardo Habkost; +Cc: mst, qemu-devel, imammedo, pbonzini, rth
On 6/3/20 10:31 AM, Eduardo Habkost wrote:
> On Wed, Jun 03, 2020 at 10:22:10AM -0500, Babu Moger wrote:
>>
>>
>>> -----Original Message-----
>>> From: Eduardo Habkost <ehabkost@redhat.com>
>>> Sent: Tuesday, June 2, 2020 6:01 PM
>>> To: Moger, Babu <Babu.Moger@amd.com>
>>> Cc: marcel.apfelbaum@gmail.com; pbonzini@redhat.com; rth@twiddle.net;
>>> mst@redhat.com; imammedo@redhat.com; qemu-devel@nongnu.org
>>> Subject: Re: [PATCH v7 07/13] hw/386: Add EPYC mode topology decoding
>>> functions
>>>
>>> On Tue, Jun 02, 2020 at 04:59:19PM -0500, Babu Moger wrote:
>>>>
>>>>
>>>>> -----Original Message-----
>>>>> From: Eduardo Habkost <ehabkost@redhat.com>
>>>>> Sent: Tuesday, June 2, 2020 12:19 PM
>>>>> To: Moger, Babu <Babu.Moger@amd.com>
>>>>> Cc: marcel.apfelbaum@gmail.com; pbonzini@redhat.com; rth@twiddle.net;
>>>>> mst@redhat.com; imammedo@redhat.com; qemu-devel@nongnu.org
>>>>> Subject: Re: [PATCH v7 07/13] hw/386: Add EPYC mode topology decoding
>>>>> functions
>>>>>
>>>>> Hi,
>>>>>
>>>>> It looks like this series breaks -device and CPU hotplug:
>>>>>
>>>>> On Wed, Mar 11, 2020 at 05:53:34PM -0500, Babu Moger wrote:
>>>>>> These functions add support for building EPYC mode topology given the
>>> smp
>>>>>> details like numa nodes, cores, threads and sockets.
>>>>>>
>>>>>> The new apic id decoding is mostly similar to current apic id decoding
>>>>>> except that it adds a new field node_id when numa configured. Removes
>>> all
>>>>>> the hardcoded values. Subsequent patches will use these functions to build
>>>>>> the topology.
>>>>>>
>>>>>> Following functions are added.
>>>>>> apicid_llc_width_epyc
>>>>>> apicid_llc_offset_epyc
>>>>>> apicid_pkg_offset_epyc
>>>>>> apicid_from_topo_ids_epyc
>>>>>> x86_topo_ids_from_idx_epyc
>>>>>> x86_topo_ids_from_apicid_epyc
>>>>>> x86_apicid_from_cpu_idx_epyc
>>>>>>
>>>>>> The topology details are available in Processor Programming Reference
>>> (PPR)
>>>>>> for AMD Family 17h Model 01h, Revision B1 Processors. The revision
>>> guides
>>>>> are
>>>>>> available from the bugzilla Link below.
>>>>>> Link:
>>>>>
>>> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugzilla.
>>>>>
>>> kernel.org%2Fshow_bug.cgi%3Fid%3D206537&data=02%7C01%7Cbabu.m
>>>>>
>>> oger%40amd.com%7C3487f40d37df4d59097d08d807190248%7C3dd8961fe488
>>>>>
>>> 4e608e11a82d994e183d%7C0%7C0%7C637267151289763739&sdata=wE0
>>>>> ukXIVh0l5eNQWsv6VDE9UZEVJmisofaW192gcZAI%3D&reserved=0
>>>>>>
>>>>>> Signed-off-by: Babu Moger <babu.moger@amd.com>
>>>>>> Acked-by: Igor Mammedov <imammedo@redhat.com>
>>>>>> Acked-by: Michael S. Tsirkin <mst@redhat.com>
>>>>>> ---
>>>>> [...]
>>>>>> typedef struct X86CPUTopoIDs {
>>>>>> unsigned pkg_id;
>>>>>> + unsigned node_id;
>>>>>
>>>>> You have added a new field here.
>>>>>
>>>>>> unsigned die_id;
>>>>>> unsigned core_id;
>>>>>> unsigned smt_id;
>>>>> [...]
>>>>>> +static inline apic_id_t
>>>>>> +x86_apicid_from_topo_ids_epyc(X86CPUTopoInfo *topo_info,
>>>>>> + const X86CPUTopoIDs *topo_ids)
>>>>>> +{
>>>>>> + return (topo_ids->pkg_id << apicid_pkg_offset_epyc(topo_info)) |
>>>>>> + (topo_ids->node_id << apicid_node_offset_epyc(topo_info)) |
>>>>>
>>>>> You are using the new field here.
>>>>>
>>>>>> + (topo_ids->die_id << apicid_die_offset(topo_info)) |
>>>>>> + (topo_ids->core_id << apicid_core_offset(topo_info)) |
>>>>>> + topo_ids->smt_id;
>>>>>> +}
>>>>>
>>>>> But you are not initializing node_id in one caller of apicid_from_topo_ids():
>>>>>
>>>>> static void pc_cpu_pre_plug(HotplugHandler *hotplug_dev,
>>>>> DeviceState *dev, Error **errp)
>>>>> {
>>>>> [...]
>>>>> X86CPUTopoIDs topo_ids;
>>>>> [...]
>>>>> if (cpu->apic_id == UNASSIGNED_APIC_ID) {
>>>>> [...]
>>>>> topo_ids.pkg_id = cpu->socket_id;
>>>>> topo_ids.die_id = cpu->die_id;
>>>>> topo_ids.core_id = cpu->core_id;
>>>>> topo_ids.smt_id = cpu->thread_id;
>>>>> cpu->apic_id = x86ms->apicid_from_topo_ids(&topo_info, &topo_ids);
>>>>> }
>>>>> [...]
>>>>> }
>>>>>
>>>>> Result: -device is broken when using -cpu EPYC:
>>>>>
>>>>> $ qemu-system-x86_64 -machine q35,accel=kvm -smp
>>>>> 1,maxcpus=2,cores=1,threads=1,sockets=2 -cpu EPYC -device EPYC-x86_64-
>>>>> cpu,core-id=0,socket-id=1,thread-id=0
>>>
>>> [1]
>>>
>>>>> qemu-system-x86_64: -device EPYC-x86_64-cpu,core-id=0,socket-
>>> id=1,thread-
>>>>> id=0: Invalid CPU [socket: 21855, die: 0, core: 0, thread: 0] with APIC ID
>>> 21855,
>>>>> valid index range 0:1
>>>>>
>>>>> This happens because APIC ID is calculated using uninitialized
>>>>> memory.
>>>> This patch should initialize the node_id. But I am not sure how to
>>>> reproduce the bug. Can you please send me the full command line to
>>>> reproduce the problem. Also test different options.
>>>
>>> The full command line is above[1].
>>
>> I just picked up the latest tree from
>> git clone https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit.qemu.org%2Fgit%2Fqemu.git&data=02%7C01%7Cbabu.moger%40amd.com%7C7752c36b7ad24df4039d08d807d334fa%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637267951007572953&sdata=69rG4w79FnLs9bhNmroxsRckzmy%2BGuH1dMjyP8PPRUo%3D&reserved=0 qemu
>> Not seeing the problem.
>>
>> ./x86_64-softmmu/qemu-system-x86_64 -machine q35,accel=kvm -smp
>> 1,maxcpus=2,cores=1,threads=1,sockets=2 -cpu EPYC -device
>> EPYC-x86_64-cpu,core-id=0,socket-id=1,thread-id=0
>> VNC server running on ::1:5900
>>
>> It appears to run ok.
[2]
>>
>>>
>>>
>>>>
>>>> diff --git a/hw/i386/pc.c b/hw/i386/pc.c
>>>> index 2128f3d6fe..047b4b9391 100644
>>>> --- a/hw/i386/pc.c
>>>> +++ b/hw/i386/pc.c
>>>> @@ -1539,6 +1539,9 @@ static void pc_cpu_pre_plug(HotplugHandler
>>> *hotplug_dev,
>>>> if (cpu->apic_id == UNASSIGNED_APIC_ID) {
>>>> int max_socket = (ms->smp.max_cpus - 1) /
>>>> smp_threads / smp_cores / x86ms->smp_dies;
>>>
>>> So, here's the input you are using to calculate topo_ids.node_id:
>>>
>>>> + unsigned nr_nodes = MAX(topo_info.nodes_per_pkg, 1);
>>>
>>> When is topo_info.nodes_per_pkg allowed to be 0?
>>
>> It will be zero if numa is not configured. Node_id will be zero for all
>> the cores.
>>
>>>
>>>> + unsigned cores_per_node = DIV_ROUND_UP((x86ms->smp_dies *
>>> smp_cores *
>>>> + smp_threads), nr_nodes);
>>>
>>> x86ms->smp_dies should be available at topo_info.dies_per_pkg,
>>> smp_cores should available at topo_info.cores_per_die,
>>> smp_threads should be available at topo_info.threads_per_core,
>>> nr_nodes should be available at topo_info.nodes_per_pkg.
>>>
>>>>
>>>> /*
>>>> * die-id was optional in QEMU 4.0 and older, so keep it optional
>>>> @@ -1585,6 +1588,7 @@ static void pc_cpu_pre_plug(HotplugHandler
>>> *hotplug_dev,
>>>> topo_ids.die_id = cpu->die_id;
>>>> topo_ids.core_id = cpu->core_id;
>>>> topo_ids.smt_id = cpu->thread_id;
>>>> + topo_ids.node_id = (cpu->core_id / cores_per_node) % nr_nodes;
>>>
>>> apicid_from_topo_ids() have access to topo_info and topo_ids, If
>>> all the information you need to calculate node_id is already
>>> available inside topo_info + topo_ids, we could be calculating it
>>> inside apicid_from_topo_ids(). Why don't we do it?
>>>
>>> Also, is topo_ids.core_id really allowed to be larger than
>>> cores_per_node when calling apicid_from_topo_ids()?
>>
>> Yes. It is. If we have two numa nodes and 8 cores. Then cores_per_node is
>> 4. Nr_nodes =2. Yes. Core_id can be larger than cores_per_node.
>
> I assumed core_id identified the core inside a die, and the range
> would be [0, cores_per_die). This seems to be what
> apicid_from_topo_ids_epyc() expects.
>
> We need clearer documentation on the semantics of each *_id
> field, to avoid confusion.
Ok. Sure. I will add that. Can you please clarify on how to repro the
issue. Marked the question at [2].
>
>>
>>>
>>>> cpu->apic_id =
>>>> x86ms->apicid_from_topo_ids(&topo_info,
>>>> &topo_ids); }
>>>>
>>>
>>> -- Eduardo
>>
>
^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH v7 07/13] hw/386: Add EPYC mode topology decoding functions
2020-06-03 15:38 ` Babu Moger
@ 2020-06-03 15:45 ` Eduardo Habkost
2020-06-03 15:49 ` Babu Moger
2020-06-03 21:49 ` Babu Moger
0 siblings, 2 replies; 38+ messages in thread
From: Eduardo Habkost @ 2020-06-03 15:45 UTC (permalink / raw)
To: Babu Moger; +Cc: mst, qemu-devel, imammedo, pbonzini, rth
On Wed, Jun 03, 2020 at 10:38:46AM -0500, Babu Moger wrote:
>
>
> On 6/3/20 10:31 AM, Eduardo Habkost wrote:
> > On Wed, Jun 03, 2020 at 10:22:10AM -0500, Babu Moger wrote:
> >>
> >>
> >>> -----Original Message-----
> >>> From: Eduardo Habkost <ehabkost@redhat.com>
> >>> Sent: Tuesday, June 2, 2020 6:01 PM
> >>> To: Moger, Babu <Babu.Moger@amd.com>
> >>> Cc: marcel.apfelbaum@gmail.com; pbonzini@redhat.com; rth@twiddle.net;
> >>> mst@redhat.com; imammedo@redhat.com; qemu-devel@nongnu.org
> >>> Subject: Re: [PATCH v7 07/13] hw/386: Add EPYC mode topology decoding
> >>> functions
> >>>
> >>> On Tue, Jun 02, 2020 at 04:59:19PM -0500, Babu Moger wrote:
> >>>>
> >>>>
> >>>>> -----Original Message-----
> >>>>> From: Eduardo Habkost <ehabkost@redhat.com>
> >>>>> Sent: Tuesday, June 2, 2020 12:19 PM
> >>>>> To: Moger, Babu <Babu.Moger@amd.com>
> >>>>> Cc: marcel.apfelbaum@gmail.com; pbonzini@redhat.com; rth@twiddle.net;
> >>>>> mst@redhat.com; imammedo@redhat.com; qemu-devel@nongnu.org
> >>>>> Subject: Re: [PATCH v7 07/13] hw/386: Add EPYC mode topology decoding
> >>>>> functions
> >>>>>
> >>>>> Hi,
> >>>>>
> >>>>> It looks like this series breaks -device and CPU hotplug:
> >>>>>
> >>>>> On Wed, Mar 11, 2020 at 05:53:34PM -0500, Babu Moger wrote:
> >>>>>> These functions add support for building EPYC mode topology given the
> >>> smp
> >>>>>> details like numa nodes, cores, threads and sockets.
> >>>>>>
> >>>>>> The new apic id decoding is mostly similar to current apic id decoding
> >>>>>> except that it adds a new field node_id when numa configured. Removes
> >>> all
> >>>>>> the hardcoded values. Subsequent patches will use these functions to build
> >>>>>> the topology.
> >>>>>>
> >>>>>> Following functions are added.
> >>>>>> apicid_llc_width_epyc
> >>>>>> apicid_llc_offset_epyc
> >>>>>> apicid_pkg_offset_epyc
> >>>>>> apicid_from_topo_ids_epyc
> >>>>>> x86_topo_ids_from_idx_epyc
> >>>>>> x86_topo_ids_from_apicid_epyc
> >>>>>> x86_apicid_from_cpu_idx_epyc
> >>>>>>
> >>>>>> The topology details are available in Processor Programming Reference
> >>> (PPR)
> >>>>>> for AMD Family 17h Model 01h, Revision B1 Processors. The revision
> >>> guides
> >>>>> are
> >>>>>> available from the bugzilla Link below.
> >>>>>> Link:
> >>>>>
> >>> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugzilla.
> >>>>>
> >>> kernel.org%2Fshow_bug.cgi%3Fid%3D206537&data=02%7C01%7Cbabu.m
> >>>>>
> >>> oger%40amd.com%7C3487f40d37df4d59097d08d807190248%7C3dd8961fe488
> >>>>>
> >>> 4e608e11a82d994e183d%7C0%7C0%7C637267151289763739&sdata=wE0
> >>>>> ukXIVh0l5eNQWsv6VDE9UZEVJmisofaW192gcZAI%3D&reserved=0
> >>>>>>
> >>>>>> Signed-off-by: Babu Moger <babu.moger@amd.com>
> >>>>>> Acked-by: Igor Mammedov <imammedo@redhat.com>
> >>>>>> Acked-by: Michael S. Tsirkin <mst@redhat.com>
> >>>>>> ---
> >>>>> [...]
> >>>>>> typedef struct X86CPUTopoIDs {
> >>>>>> unsigned pkg_id;
> >>>>>> + unsigned node_id;
> >>>>>
> >>>>> You have added a new field here.
> >>>>>
> >>>>>> unsigned die_id;
> >>>>>> unsigned core_id;
> >>>>>> unsigned smt_id;
> >>>>> [...]
> >>>>>> +static inline apic_id_t
> >>>>>> +x86_apicid_from_topo_ids_epyc(X86CPUTopoInfo *topo_info,
> >>>>>> + const X86CPUTopoIDs *topo_ids)
> >>>>>> +{
> >>>>>> + return (topo_ids->pkg_id << apicid_pkg_offset_epyc(topo_info)) |
> >>>>>> + (topo_ids->node_id << apicid_node_offset_epyc(topo_info)) |
> >>>>>
> >>>>> You are using the new field here.
> >>>>>
> >>>>>> + (topo_ids->die_id << apicid_die_offset(topo_info)) |
> >>>>>> + (topo_ids->core_id << apicid_core_offset(topo_info)) |
> >>>>>> + topo_ids->smt_id;
> >>>>>> +}
> >>>>>
> >>>>> But you are not initializing node_id in one caller of apicid_from_topo_ids():
> >>>>>
> >>>>> static void pc_cpu_pre_plug(HotplugHandler *hotplug_dev,
> >>>>> DeviceState *dev, Error **errp)
> >>>>> {
> >>>>> [...]
> >>>>> X86CPUTopoIDs topo_ids;
> >>>>> [...]
> >>>>> if (cpu->apic_id == UNASSIGNED_APIC_ID) {
> >>>>> [...]
> >>>>> topo_ids.pkg_id = cpu->socket_id;
> >>>>> topo_ids.die_id = cpu->die_id;
> >>>>> topo_ids.core_id = cpu->core_id;
> >>>>> topo_ids.smt_id = cpu->thread_id;
> >>>>> cpu->apic_id = x86ms->apicid_from_topo_ids(&topo_info, &topo_ids);
> >>>>> }
> >>>>> [...]
> >>>>> }
> >>>>>
> >>>>> Result: -device is broken when using -cpu EPYC:
> >>>>>
> >>>>> $ qemu-system-x86_64 -machine q35,accel=kvm -smp
> >>>>> 1,maxcpus=2,cores=1,threads=1,sockets=2 -cpu EPYC -device EPYC-x86_64-
> >>>>> cpu,core-id=0,socket-id=1,thread-id=0
> >>>
> >>> [1]
> >>>
> >>>>> qemu-system-x86_64: -device EPYC-x86_64-cpu,core-id=0,socket-
> >>> id=1,thread-
> >>>>> id=0: Invalid CPU [socket: 21855, die: 0, core: 0, thread: 0] with APIC ID
> >>> 21855,
> >>>>> valid index range 0:1
> >>>>>
> >>>>> This happens because APIC ID is calculated using uninitialized
> >>>>> memory.
> >>>> This patch should initialize the node_id. But I am not sure how to
> >>>> reproduce the bug. Can you please send me the full command line to
> >>>> reproduce the problem. Also test different options.
> >>>
> >>> The full command line is above[1].
> >>
> >> I just picked up the latest tree from
> >> git clone https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit.qemu.org%2Fgit%2Fqemu.git&data=02%7C01%7Cbabu.moger%40amd.com%7C7752c36b7ad24df4039d08d807d334fa%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637267951007572953&sdata=69rG4w79FnLs9bhNmroxsRckzmy%2BGuH1dMjyP8PPRUo%3D&reserved=0 qemu
> >> Not seeing the problem.
> >>
> >> ./x86_64-softmmu/qemu-system-x86_64 -machine q35,accel=kvm -smp
> >> 1,maxcpus=2,cores=1,threads=1,sockets=2 -cpu EPYC -device
> >> EPYC-x86_64-cpu,core-id=0,socket-id=1,thread-id=0
> >> VNC server running on ::1:5900
> >>
> >> It appears to run ok.
>
> [2]
>
> >>
> >>>
> >>>
> >>>>
> >>>> diff --git a/hw/i386/pc.c b/hw/i386/pc.c
> >>>> index 2128f3d6fe..047b4b9391 100644
> >>>> --- a/hw/i386/pc.c
> >>>> +++ b/hw/i386/pc.c
> >>>> @@ -1539,6 +1539,9 @@ static void pc_cpu_pre_plug(HotplugHandler
> >>> *hotplug_dev,
> >>>> if (cpu->apic_id == UNASSIGNED_APIC_ID) {
> >>>> int max_socket = (ms->smp.max_cpus - 1) /
> >>>> smp_threads / smp_cores / x86ms->smp_dies;
> >>>
> >>> So, here's the input you are using to calculate topo_ids.node_id:
> >>>
> >>>> + unsigned nr_nodes = MAX(topo_info.nodes_per_pkg, 1);
> >>>
> >>> When is topo_info.nodes_per_pkg allowed to be 0?
> >>
> >> It will be zero if numa is not configured. Node_id will be zero for all
> >> the cores.
> >>
> >>>
> >>>> + unsigned cores_per_node = DIV_ROUND_UP((x86ms->smp_dies *
> >>> smp_cores *
> >>>> + smp_threads), nr_nodes);
> >>>
> >>> x86ms->smp_dies should be available at topo_info.dies_per_pkg,
> >>> smp_cores should available at topo_info.cores_per_die,
> >>> smp_threads should be available at topo_info.threads_per_core,
> >>> nr_nodes should be available at topo_info.nodes_per_pkg.
> >>>
> >>>>
> >>>> /*
> >>>> * die-id was optional in QEMU 4.0 and older, so keep it optional
> >>>> @@ -1585,6 +1588,7 @@ static void pc_cpu_pre_plug(HotplugHandler
> >>> *hotplug_dev,
> >>>> topo_ids.die_id = cpu->die_id;
> >>>> topo_ids.core_id = cpu->core_id;
> >>>> topo_ids.smt_id = cpu->thread_id;
> >>>> + topo_ids.node_id = (cpu->core_id / cores_per_node) % nr_nodes;
> >>>
> >>> apicid_from_topo_ids() have access to topo_info and topo_ids, If
> >>> all the information you need to calculate node_id is already
> >>> available inside topo_info + topo_ids, we could be calculating it
> >>> inside apicid_from_topo_ids(). Why don't we do it?
> >>>
> >>> Also, is topo_ids.core_id really allowed to be larger than
> >>> cores_per_node when calling apicid_from_topo_ids()?
> >>
> >> Yes. It is. If we have two numa nodes and 8 cores. Then cores_per_node is
> >> 4. Nr_nodes =2. Yes. Core_id can be larger than cores_per_node.
> >
> > I assumed core_id identified the core inside a die, and the range
> > would be [0, cores_per_die). This seems to be what
> > apicid_from_topo_ids_epyc() expects.
> >
> > We need clearer documentation on the semantics of each *_id
> > field, to avoid confusion.
>
> Ok. Sure. I will add that. Can you please clarify on how to repro the
> issue. Marked the question at [2].
As it is usage of uninitialized memory, behavior is undefined and
not guaranteed to crash. My QEMU binary was built with
--enable-debug, maybe it makes it easier to reproduce.
--
Eduardo
^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH v7 07/13] hw/386: Add EPYC mode topology decoding functions
2020-06-03 15:45 ` Eduardo Habkost
@ 2020-06-03 15:49 ` Babu Moger
2020-06-03 21:49 ` Babu Moger
1 sibling, 0 replies; 38+ messages in thread
From: Babu Moger @ 2020-06-03 15:49 UTC (permalink / raw)
To: Eduardo Habkost; +Cc: mst, qemu-devel, imammedo, pbonzini, rth
On 6/3/20 10:45 AM, Eduardo Habkost wrote:
> On Wed, Jun 03, 2020 at 10:38:46AM -0500, Babu Moger wrote:
>>
>>
>> On 6/3/20 10:31 AM, Eduardo Habkost wrote:
>>> On Wed, Jun 03, 2020 at 10:22:10AM -0500, Babu Moger wrote:
>>>>
>>>>
>>>>> -----Original Message-----
>>>>> From: Eduardo Habkost <ehabkost@redhat.com>
>>>>> Sent: Tuesday, June 2, 2020 6:01 PM
>>>>> To: Moger, Babu <Babu.Moger@amd.com>
>>>>> Cc: marcel.apfelbaum@gmail.com; pbonzini@redhat.com; rth@twiddle.net;
>>>>> mst@redhat.com; imammedo@redhat.com; qemu-devel@nongnu.org
>>>>> Subject: Re: [PATCH v7 07/13] hw/386: Add EPYC mode topology decoding
>>>>> functions
>>>>>
>>>>> On Tue, Jun 02, 2020 at 04:59:19PM -0500, Babu Moger wrote:
>>>>>>
>>>>>>
>>>>>>> -----Original Message-----
>>>>>>> From: Eduardo Habkost <ehabkost@redhat.com>
>>>>>>> Sent: Tuesday, June 2, 2020 12:19 PM
>>>>>>> To: Moger, Babu <Babu.Moger@amd.com>
>>>>>>> Cc: marcel.apfelbaum@gmail.com; pbonzini@redhat.com; rth@twiddle.net;
>>>>>>> mst@redhat.com; imammedo@redhat.com; qemu-devel@nongnu.org
>>>>>>> Subject: Re: [PATCH v7 07/13] hw/386: Add EPYC mode topology decoding
>>>>>>> functions
>>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> It looks like this series breaks -device and CPU hotplug:
>>>>>>>
>>>>>>> On Wed, Mar 11, 2020 at 05:53:34PM -0500, Babu Moger wrote:
>>>>>>>> These functions add support for building EPYC mode topology given the
>>>>> smp
>>>>>>>> details like numa nodes, cores, threads and sockets.
>>>>>>>>
>>>>>>>> The new apic id decoding is mostly similar to current apic id decoding
>>>>>>>> except that it adds a new field node_id when numa configured. Removes
>>>>> all
>>>>>>>> the hardcoded values. Subsequent patches will use these functions to build
>>>>>>>> the topology.
>>>>>>>>
>>>>>>>> Following functions are added.
>>>>>>>> apicid_llc_width_epyc
>>>>>>>> apicid_llc_offset_epyc
>>>>>>>> apicid_pkg_offset_epyc
>>>>>>>> apicid_from_topo_ids_epyc
>>>>>>>> x86_topo_ids_from_idx_epyc
>>>>>>>> x86_topo_ids_from_apicid_epyc
>>>>>>>> x86_apicid_from_cpu_idx_epyc
>>>>>>>>
>>>>>>>> The topology details are available in Processor Programming Reference
>>>>> (PPR)
>>>>>>>> for AMD Family 17h Model 01h, Revision B1 Processors. The revision
>>>>> guides
>>>>>>> are
>>>>>>>> available from the bugzilla Link below.
>>>>>>>> Link:
>>>>>>>
>>>>> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugzilla.
>>>>>>>
>>>>> kernel.org%2Fshow_bug.cgi%3Fid%3D206537&data=02%7C01%7Cbabu.m
>>>>>>>
>>>>> oger%40amd.com%7C3487f40d37df4d59097d08d807190248%7C3dd8961fe488
>>>>>>>
>>>>> 4e608e11a82d994e183d%7C0%7C0%7C637267151289763739&sdata=wE0
>>>>>>> ukXIVh0l5eNQWsv6VDE9UZEVJmisofaW192gcZAI%3D&reserved=0
>>>>>>>>
>>>>>>>> Signed-off-by: Babu Moger <babu.moger@amd.com>
>>>>>>>> Acked-by: Igor Mammedov <imammedo@redhat.com>
>>>>>>>> Acked-by: Michael S. Tsirkin <mst@redhat.com>
>>>>>>>> ---
>>>>>>> [...]
>>>>>>>> typedef struct X86CPUTopoIDs {
>>>>>>>> unsigned pkg_id;
>>>>>>>> + unsigned node_id;
>>>>>>>
>>>>>>> You have added a new field here.
>>>>>>>
>>>>>>>> unsigned die_id;
>>>>>>>> unsigned core_id;
>>>>>>>> unsigned smt_id;
>>>>>>> [...]
>>>>>>>> +static inline apic_id_t
>>>>>>>> +x86_apicid_from_topo_ids_epyc(X86CPUTopoInfo *topo_info,
>>>>>>>> + const X86CPUTopoIDs *topo_ids)
>>>>>>>> +{
>>>>>>>> + return (topo_ids->pkg_id << apicid_pkg_offset_epyc(topo_info)) |
>>>>>>>> + (topo_ids->node_id << apicid_node_offset_epyc(topo_info)) |
>>>>>>>
>>>>>>> You are using the new field here.
>>>>>>>
>>>>>>>> + (topo_ids->die_id << apicid_die_offset(topo_info)) |
>>>>>>>> + (topo_ids->core_id << apicid_core_offset(topo_info)) |
>>>>>>>> + topo_ids->smt_id;
>>>>>>>> +}
>>>>>>>
>>>>>>> But you are not initializing node_id in one caller of apicid_from_topo_ids():
>>>>>>>
>>>>>>> static void pc_cpu_pre_plug(HotplugHandler *hotplug_dev,
>>>>>>> DeviceState *dev, Error **errp)
>>>>>>> {
>>>>>>> [...]
>>>>>>> X86CPUTopoIDs topo_ids;
>>>>>>> [...]
>>>>>>> if (cpu->apic_id == UNASSIGNED_APIC_ID) {
>>>>>>> [...]
>>>>>>> topo_ids.pkg_id = cpu->socket_id;
>>>>>>> topo_ids.die_id = cpu->die_id;
>>>>>>> topo_ids.core_id = cpu->core_id;
>>>>>>> topo_ids.smt_id = cpu->thread_id;
>>>>>>> cpu->apic_id = x86ms->apicid_from_topo_ids(&topo_info, &topo_ids);
>>>>>>> }
>>>>>>> [...]
>>>>>>> }
>>>>>>>
>>>>>>> Result: -device is broken when using -cpu EPYC:
>>>>>>>
>>>>>>> $ qemu-system-x86_64 -machine q35,accel=kvm -smp
>>>>>>> 1,maxcpus=2,cores=1,threads=1,sockets=2 -cpu EPYC -device EPYC-x86_64-
>>>>>>> cpu,core-id=0,socket-id=1,thread-id=0
>>>>>
>>>>> [1]
>>>>>
>>>>>>> qemu-system-x86_64: -device EPYC-x86_64-cpu,core-id=0,socket-
>>>>> id=1,thread-
>>>>>>> id=0: Invalid CPU [socket: 21855, die: 0, core: 0, thread: 0] with APIC ID
>>>>> 21855,
>>>>>>> valid index range 0:1
>>>>>>>
>>>>>>> This happens because APIC ID is calculated using uninitialized
>>>>>>> memory.
>>>>>> This patch should initialize the node_id. But I am not sure how to
>>>>>> reproduce the bug. Can you please send me the full command line to
>>>>>> reproduce the problem. Also test different options.
>>>>>
>>>>> The full command line is above[1].
>>>>
>>>> I just picked up the latest tree from
>>>> git clone https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit.qemu.org%2Fgit%2Fqemu.git&data=02%7C01%7Cbabu.moger%40amd.com%7C4b409b77ff2c4c4ed90608d807d53570%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637267959601716206&sdata=ih6bo9Wp0RbJgRpryzSa2D0v6kr3Zfww%2B1uB%2FNyngk8%3D&reserved=0 qemu
>>>> Not seeing the problem.
>>>>
>>>> ./x86_64-softmmu/qemu-system-x86_64 -machine q35,accel=kvm -smp
>>>> 1,maxcpus=2,cores=1,threads=1,sockets=2 -cpu EPYC -device
>>>> EPYC-x86_64-cpu,core-id=0,socket-id=1,thread-id=0
>>>> VNC server running on ::1:5900
>>>>
>>>> It appears to run ok.
>>
>> [2]
>>
>>>>
>>>>>
>>>>>
>>>>>>
>>>>>> diff --git a/hw/i386/pc.c b/hw/i386/pc.c
>>>>>> index 2128f3d6fe..047b4b9391 100644
>>>>>> --- a/hw/i386/pc.c
>>>>>> +++ b/hw/i386/pc.c
>>>>>> @@ -1539,6 +1539,9 @@ static void pc_cpu_pre_plug(HotplugHandler
>>>>> *hotplug_dev,
>>>>>> if (cpu->apic_id == UNASSIGNED_APIC_ID) {
>>>>>> int max_socket = (ms->smp.max_cpus - 1) /
>>>>>> smp_threads / smp_cores / x86ms->smp_dies;
>>>>>
>>>>> So, here's the input you are using to calculate topo_ids.node_id:
>>>>>
>>>>>> + unsigned nr_nodes = MAX(topo_info.nodes_per_pkg, 1);
>>>>>
>>>>> When is topo_info.nodes_per_pkg allowed to be 0?
>>>>
>>>> It will be zero if numa is not configured. Node_id will be zero for all
>>>> the cores.
>>>>
>>>>>
>>>>>> + unsigned cores_per_node = DIV_ROUND_UP((x86ms->smp_dies *
>>>>> smp_cores *
>>>>>> + smp_threads), nr_nodes);
>>>>>
>>>>> x86ms->smp_dies should be available at topo_info.dies_per_pkg,
>>>>> smp_cores should available at topo_info.cores_per_die,
>>>>> smp_threads should be available at topo_info.threads_per_core,
>>>>> nr_nodes should be available at topo_info.nodes_per_pkg.
>>>>>
>>>>>>
>>>>>> /*
>>>>>> * die-id was optional in QEMU 4.0 and older, so keep it optional
>>>>>> @@ -1585,6 +1588,7 @@ static void pc_cpu_pre_plug(HotplugHandler
>>>>> *hotplug_dev,
>>>>>> topo_ids.die_id = cpu->die_id;
>>>>>> topo_ids.core_id = cpu->core_id;
>>>>>> topo_ids.smt_id = cpu->thread_id;
>>>>>> + topo_ids.node_id = (cpu->core_id / cores_per_node) % nr_nodes;
>>>>>
>>>>> apicid_from_topo_ids() have access to topo_info and topo_ids, If
>>>>> all the information you need to calculate node_id is already
>>>>> available inside topo_info + topo_ids, we could be calculating it
>>>>> inside apicid_from_topo_ids(). Why don't we do it?
>>>>>
>>>>> Also, is topo_ids.core_id really allowed to be larger than
>>>>> cores_per_node when calling apicid_from_topo_ids()?
>>>>
>>>> Yes. It is. If we have two numa nodes and 8 cores. Then cores_per_node is
>>>> 4. Nr_nodes =2. Yes. Core_id can be larger than cores_per_node.
>>>
>>> I assumed core_id identified the core inside a die, and the range
>>> would be [0, cores_per_die). This seems to be what
>>> apicid_from_topo_ids_epyc() expects.
>>>
>>> We need clearer documentation on the semantics of each *_id
>>> field, to avoid confusion.
>>
>> Ok. Sure. I will add that. Can you please clarify on how to repro the
>> issue. Marked the question at [2].
>
> As it is usage of uninitialized memory, behavior is undefined and
> not guaranteed to crash. My QEMU binary was built with
> --enable-debug, maybe it makes it easier to reproduce.
Ok. That is fine. Will try..Thanks
^ permalink raw reply [flat|nested] 38+ messages in thread
* RE: [PATCH v7 07/13] hw/386: Add EPYC mode topology decoding functions
2020-06-03 15:45 ` Eduardo Habkost
2020-06-03 15:49 ` Babu Moger
@ 2020-06-03 21:49 ` Babu Moger
2020-06-15 11:44 ` Dr. David Alan Gilbert
1 sibling, 1 reply; 38+ messages in thread
From: Babu Moger @ 2020-06-03 21:49 UTC (permalink / raw)
To: Eduardo Habkost; +Cc: mst, qemu-devel, imammedo, pbonzini, rth
> -----Original Message-----
> From: Eduardo Habkost <ehabkost@redhat.com>
> Sent: Wednesday, June 3, 2020 10:46 AM
> To: Moger, Babu <Babu.Moger@amd.com>
> Cc: marcel.apfelbaum@gmail.com; pbonzini@redhat.com; rth@twiddle.net;
> mst@redhat.com; imammedo@redhat.com; qemu-devel@nongnu.org
> Subject: Re: [PATCH v7 07/13] hw/386: Add EPYC mode topology decoding
> functions
>
> On Wed, Jun 03, 2020 at 10:38:46AM -0500, Babu Moger wrote:
> >
> >
> > On 6/3/20 10:31 AM, Eduardo Habkost wrote:
> > > On Wed, Jun 03, 2020 at 10:22:10AM -0500, Babu Moger wrote:
> > >>
> > >>
> > >>> -----Original Message-----
> > >>> From: Eduardo Habkost <ehabkost@redhat.com>
> > >>> Sent: Tuesday, June 2, 2020 6:01 PM
> > >>> To: Moger, Babu <Babu.Moger@amd.com>
> > >>> Cc: marcel.apfelbaum@gmail.com; pbonzini@redhat.com;
> rth@twiddle.net;
> > >>> mst@redhat.com; imammedo@redhat.com; qemu-devel@nongnu.org
> > >>> Subject: Re: [PATCH v7 07/13] hw/386: Add EPYC mode topology decoding
> > >>> functions
> > >>>
> > >>> On Tue, Jun 02, 2020 at 04:59:19PM -0500, Babu Moger wrote:
> > >>>>
> > >>>>
> > >>>>> -----Original Message-----
> > >>>>> From: Eduardo Habkost <ehabkost@redhat.com>
> > >>>>> Sent: Tuesday, June 2, 2020 12:19 PM
> > >>>>> To: Moger, Babu <Babu.Moger@amd.com>
> > >>>>> Cc: marcel.apfelbaum@gmail.com; pbonzini@redhat.com;
> rth@twiddle.net;
> > >>>>> mst@redhat.com; imammedo@redhat.com; qemu-devel@nongnu.org
> > >>>>> Subject: Re: [PATCH v7 07/13] hw/386: Add EPYC mode topology
> decoding
> > >>>>> functions
> > >>>>>
> > >>>>> Hi,
> > >>>>>
> > >>>>> It looks like this series breaks -device and CPU hotplug:
> > >>>>>
> > >>>>> On Wed, Mar 11, 2020 at 05:53:34PM -0500, Babu Moger wrote:
> > >>>>>> These functions add support for building EPYC mode topology given
> the
> > >>> smp
> > >>>>>> details like numa nodes, cores, threads and sockets.
> > >>>>>>
> > >>>>>> The new apic id decoding is mostly similar to current apic id decoding
> > >>>>>> except that it adds a new field node_id when numa configured.
> Removes
> > >>> all
> > >>>>>> the hardcoded values. Subsequent patches will use these functions to
> build
> > >>>>>> the topology.
> > >>>>>>
> > >>>>>> Following functions are added.
> > >>>>>> apicid_llc_width_epyc
> > >>>>>> apicid_llc_offset_epyc
> > >>>>>> apicid_pkg_offset_epyc
> > >>>>>> apicid_from_topo_ids_epyc
> > >>>>>> x86_topo_ids_from_idx_epyc
> > >>>>>> x86_topo_ids_from_apicid_epyc
> > >>>>>> x86_apicid_from_cpu_idx_epyc
> > >>>>>>
> > >>>>>> The topology details are available in Processor Programming
> Reference
> > >>> (PPR)
> > >>>>>> for AMD Family 17h Model 01h, Revision B1 Processors. The revision
> > >>> guides
> > >>>>> are
> > >>>>>> available from the bugzilla Link below.
> > >>>>>> Link:
> > >>>>>
> > >>>
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugzilla.
> > >>>>>
> > >>>
> kernel.org%2Fshow_bug.cgi%3Fid%3D206537&data=02%7C01%7Cbabu.m
> > >>>>>
> > >>>
> oger%40amd.com%7C3487f40d37df4d59097d08d807190248%7C3dd8961fe488
> > >>>>>
> > >>>
> 4e608e11a82d994e183d%7C0%7C0%7C637267151289763739&sdata=wE0
> > >>>>> ukXIVh0l5eNQWsv6VDE9UZEVJmisofaW192gcZAI%3D&reserved=0
> > >>>>>>
> > >>>>>> Signed-off-by: Babu Moger <babu.moger@amd.com>
> > >>>>>> Acked-by: Igor Mammedov <imammedo@redhat.com>
> > >>>>>> Acked-by: Michael S. Tsirkin <mst@redhat.com>
> > >>>>>> ---
> > >>>>> [...]
> > >>>>>> typedef struct X86CPUTopoIDs {
> > >>>>>> unsigned pkg_id;
> > >>>>>> + unsigned node_id;
> > >>>>>
> > >>>>> You have added a new field here.
> > >>>>>
> > >>>>>> unsigned die_id;
> > >>>>>> unsigned core_id;
> > >>>>>> unsigned smt_id;
> > >>>>> [...]
> > >>>>>> +static inline apic_id_t
> > >>>>>> +x86_apicid_from_topo_ids_epyc(X86CPUTopoInfo *topo_info,
> > >>>>>> + const X86CPUTopoIDs *topo_ids)
> > >>>>>> +{
> > >>>>>> + return (topo_ids->pkg_id << apicid_pkg_offset_epyc(topo_info)) |
> > >>>>>> + (topo_ids->node_id << apicid_node_offset_epyc(topo_info)) |
> > >>>>>
> > >>>>> You are using the new field here.
> > >>>>>
> > >>>>>> + (topo_ids->die_id << apicid_die_offset(topo_info)) |
> > >>>>>> + (topo_ids->core_id << apicid_core_offset(topo_info)) |
> > >>>>>> + topo_ids->smt_id;
> > >>>>>> +}
> > >>>>>
> > >>>>> But you are not initializing node_id in one caller of
> apicid_from_topo_ids():
> > >>>>>
> > >>>>> static void pc_cpu_pre_plug(HotplugHandler *hotplug_dev,
> > >>>>> DeviceState *dev, Error **errp)
> > >>>>> {
> > >>>>> [...]
> > >>>>> X86CPUTopoIDs topo_ids;
> > >>>>> [...]
> > >>>>> if (cpu->apic_id == UNASSIGNED_APIC_ID) {
> > >>>>> [...]
> > >>>>> topo_ids.pkg_id = cpu->socket_id;
> > >>>>> topo_ids.die_id = cpu->die_id;
> > >>>>> topo_ids.core_id = cpu->core_id;
> > >>>>> topo_ids.smt_id = cpu->thread_id;
> > >>>>> cpu->apic_id = x86ms->apicid_from_topo_ids(&topo_info,
> &topo_ids);
> > >>>>> }
> > >>>>> [...]
> > >>>>> }
> > >>>>>
> > >>>>> Result: -device is broken when using -cpu EPYC:
> > >>>>>
> > >>>>> $ qemu-system-x86_64 -machine q35,accel=kvm -smp
> > >>>>> 1,maxcpus=2,cores=1,threads=1,sockets=2 -cpu EPYC -device EPYC-
> x86_64-
> > >>>>> cpu,core-id=0,socket-id=1,thread-id=0
> > >>>
> > >>> [1]
> > >>>
> > >>>>> qemu-system-x86_64: -device EPYC-x86_64-cpu,core-id=0,socket-
> > >>> id=1,thread-
> > >>>>> id=0: Invalid CPU [socket: 21855, die: 0, core: 0, thread: 0] with APIC ID
> > >>> 21855,
> > >>>>> valid index range 0:1
> > >>>>>
> > >>>>> This happens because APIC ID is calculated using uninitialized
> > >>>>> memory.
> > >>>> This patch should initialize the node_id. But I am not sure how to
> > >>>> reproduce the bug. Can you please send me the full command line to
> > >>>> reproduce the problem. Also test different options.
> > >>>
> > >>> The full command line is above[1].
> > >>
> > >> I just picked up the latest tree from
> > >> git clone
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit.qem
> u.org%2Fgit%2Fqemu.git&data=02%7C01%7Cbabu.moger%40amd.com%7
> C4b409b77ff2c4c4ed90608d807d53570%7C3dd8961fe4884e608e11a82d994e1
> 83d%7C0%7C0%7C637267959601716206&sdata=ih6bo9Wp0RbJgRpryzSa2
> D0v6kr3Zfww%2B1uB%2FNyngk8%3D&reserved=0 qemu
> > >> Not seeing the problem.
> > >>
> > >> ./x86_64-softmmu/qemu-system-x86_64 -machine q35,accel=kvm -smp
> > >> 1,maxcpus=2,cores=1,threads=1,sockets=2 -cpu EPYC -device
> > >> EPYC-x86_64-cpu,core-id=0,socket-id=1,thread-id=0
> > >> VNC server running on ::1:5900
> > >>
> > >> It appears to run ok.
> >
> > [2]
> >
> > >>
> > >>>
> > >>>
> > >>>>
> > >>>> diff --git a/hw/i386/pc.c b/hw/i386/pc.c
> > >>>> index 2128f3d6fe..047b4b9391 100644
> > >>>> --- a/hw/i386/pc.c
> > >>>> +++ b/hw/i386/pc.c
> > >>>> @@ -1539,6 +1539,9 @@ static void pc_cpu_pre_plug(HotplugHandler
> > >>> *hotplug_dev,
> > >>>> if (cpu->apic_id == UNASSIGNED_APIC_ID) {
> > >>>> int max_socket = (ms->smp.max_cpus - 1) /
> > >>>> smp_threads / smp_cores / x86ms->smp_dies;
> > >>>
> > >>> So, here's the input you are using to calculate topo_ids.node_id:
> > >>>
> > >>>> + unsigned nr_nodes = MAX(topo_info.nodes_per_pkg, 1);
> > >>>
> > >>> When is topo_info.nodes_per_pkg allowed to be 0?
> > >>
> > >> It will be zero if numa is not configured. Node_id will be zero for all
> > >> the cores.
> > >>
> > >>>
> > >>>> + unsigned cores_per_node = DIV_ROUND_UP((x86ms->smp_dies *
> > >>> smp_cores *
> > >>>> + smp_threads), nr_nodes);
> > >>>
> > >>> x86ms->smp_dies should be available at topo_info.dies_per_pkg,
> > >>> smp_cores should available at topo_info.cores_per_die,
> > >>> smp_threads should be available at topo_info.threads_per_core,
> > >>> nr_nodes should be available at topo_info.nodes_per_pkg.
> > >>>
> > >>>>
> > >>>> /*
> > >>>> * die-id was optional in QEMU 4.0 and older, so keep it optional
> > >>>> @@ -1585,6 +1588,7 @@ static void pc_cpu_pre_plug(HotplugHandler
> > >>> *hotplug_dev,
> > >>>> topo_ids.die_id = cpu->die_id;
> > >>>> topo_ids.core_id = cpu->core_id;
> > >>>> topo_ids.smt_id = cpu->thread_id;
> > >>>> + topo_ids.node_id = (cpu->core_id / cores_per_node) % nr_nodes;
> > >>>
> > >>> apicid_from_topo_ids() have access to topo_info and topo_ids, If
> > >>> all the information you need to calculate node_id is already
> > >>> available inside topo_info + topo_ids, we could be calculating it
> > >>> inside apicid_from_topo_ids(). Why don't we do it?
> > >>>
> > >>> Also, is topo_ids.core_id really allowed to be larger than
> > >>> cores_per_node when calling apicid_from_topo_ids()?
> > >>
> > >> Yes. It is. If we have two numa nodes and 8 cores. Then cores_per_node is
> > >> 4. Nr_nodes =2. Yes. Core_id can be larger than cores_per_node.
> > >
> > > I assumed core_id identified the core inside a die, and the range
> > > would be [0, cores_per_die). This seems to be what
> > > apicid_from_topo_ids_epyc() expects.
> > >
> > > We need clearer documentation on the semantics of each *_id
> > > field, to avoid confusion.
> >
> > Ok. Sure. I will add that. Can you please clarify on how to repro the
> > issue. Marked the question at [2].
>
> As it is usage of uninitialized memory, behavior is undefined and
> not guaranteed to crash. My QEMU binary was built with
> --enable-debug, maybe it makes it easier to reproduce.
This patch works fine. Tested few combination. Will send the formal
version if patch looks good.
diff --git a/include/hw/i386/topology.h b/include/hw/i386/topology.h
index 07239f95f4..cb28f0a57c 100644
--- a/include/hw/i386/topology.h
+++ b/include/hw/i386/topology.h
@@ -140,6 +140,17 @@ static inline unsigned
apicid_pkg_offset_epyc(X86CPUTopoInfo *topo_info)
apicid_node_width_epyc(topo_info);
}
+static inline unsigned x86_node_id_for_epyc(X86CPUTopoInfo *topo_info,
+ const X86CPUTopoIDs *topo_ids)
+{
+ unsigned nr_nodes = MAX(topo_info->nodes_per_pkg, 1);
+ unsigned cores_per_node = DIV_ROUND_UP((topo_info->dies_per_pkg *
+ topo_info->cores_per_die *
+ topo_info->threads_per_core),
+ nr_nodes);
+
+ return ((topo_ids->core_id / cores_per_node) % nr_nodes);
+}
/*
* Make APIC ID for the CPU based on Pkg_ID, Core_ID, SMT_ID
*
@@ -149,8 +160,11 @@ static inline apic_id_t
x86_apicid_from_topo_ids_epyc(X86CPUTopoInfo *topo_info,
const X86CPUTopoIDs *topo_ids)
{
+ /* node_id can be uninitialized. Initialize here */
+ unsigned node_id = x86_node_id_for_epyc(topo_info, topo_ids);
+
return (topo_ids->pkg_id << apicid_pkg_offset_epyc(topo_info)) |
- (topo_ids->node_id << apicid_node_offset_epyc(topo_info)) |
+ (node_id << apicid_node_offset_epyc(topo_info)) |
(topo_ids->die_id << apicid_die_offset(topo_info)) |
(topo_ids->core_id << apicid_core_offset(topo_info)) |
topo_ids->smt_id;
^ permalink raw reply related [flat|nested] 38+ messages in thread
* Re: [PATCH v7 07/13] hw/386: Add EPYC mode topology decoding functions
2020-06-03 21:49 ` Babu Moger
@ 2020-06-15 11:44 ` Dr. David Alan Gilbert
0 siblings, 0 replies; 38+ messages in thread
From: Dr. David Alan Gilbert @ 2020-06-15 11:44 UTC (permalink / raw)
To: Babu Moger; +Cc: Eduardo Habkost, mst, qemu-devel, pbonzini, imammedo, rth
* Babu Moger (babu.moger@amd.com) wrote:
>
>
> > -----Original Message-----
> > From: Eduardo Habkost <ehabkost@redhat.com>
> > Sent: Wednesday, June 3, 2020 10:46 AM
> > To: Moger, Babu <Babu.Moger@amd.com>
> > Cc: marcel.apfelbaum@gmail.com; pbonzini@redhat.com; rth@twiddle.net;
> > mst@redhat.com; imammedo@redhat.com; qemu-devel@nongnu.org
> > Subject: Re: [PATCH v7 07/13] hw/386: Add EPYC mode topology decoding
> > functions
> >
> > On Wed, Jun 03, 2020 at 10:38:46AM -0500, Babu Moger wrote:
> > >
> > >
> > > On 6/3/20 10:31 AM, Eduardo Habkost wrote:
> > > > On Wed, Jun 03, 2020 at 10:22:10AM -0500, Babu Moger wrote:
> > > >>
> > > >>
> > > >>> -----Original Message-----
> > > >>> From: Eduardo Habkost <ehabkost@redhat.com>
> > > >>> Sent: Tuesday, June 2, 2020 6:01 PM
> > > >>> To: Moger, Babu <Babu.Moger@amd.com>
> > > >>> Cc: marcel.apfelbaum@gmail.com; pbonzini@redhat.com;
> > rth@twiddle.net;
> > > >>> mst@redhat.com; imammedo@redhat.com; qemu-devel@nongnu.org
> > > >>> Subject: Re: [PATCH v7 07/13] hw/386: Add EPYC mode topology decoding
> > > >>> functions
> > > >>>
> > > >>> On Tue, Jun 02, 2020 at 04:59:19PM -0500, Babu Moger wrote:
> > > >>>>
> > > >>>>
> > > >>>>> -----Original Message-----
> > > >>>>> From: Eduardo Habkost <ehabkost@redhat.com>
> > > >>>>> Sent: Tuesday, June 2, 2020 12:19 PM
> > > >>>>> To: Moger, Babu <Babu.Moger@amd.com>
> > > >>>>> Cc: marcel.apfelbaum@gmail.com; pbonzini@redhat.com;
> > rth@twiddle.net;
> > > >>>>> mst@redhat.com; imammedo@redhat.com; qemu-devel@nongnu.org
> > > >>>>> Subject: Re: [PATCH v7 07/13] hw/386: Add EPYC mode topology
> > decoding
> > > >>>>> functions
> > > >>>>>
> > > >>>>> Hi,
> > > >>>>>
> > > >>>>> It looks like this series breaks -device and CPU hotplug:
> > > >>>>>
> > > >>>>> On Wed, Mar 11, 2020 at 05:53:34PM -0500, Babu Moger wrote:
> > > >>>>>> These functions add support for building EPYC mode topology given
> > the
> > > >>> smp
> > > >>>>>> details like numa nodes, cores, threads and sockets.
> > > >>>>>>
> > > >>>>>> The new apic id decoding is mostly similar to current apic id decoding
> > > >>>>>> except that it adds a new field node_id when numa configured.
> > Removes
> > > >>> all
> > > >>>>>> the hardcoded values. Subsequent patches will use these functions to
> > build
> > > >>>>>> the topology.
> > > >>>>>>
> > > >>>>>> Following functions are added.
> > > >>>>>> apicid_llc_width_epyc
> > > >>>>>> apicid_llc_offset_epyc
> > > >>>>>> apicid_pkg_offset_epyc
> > > >>>>>> apicid_from_topo_ids_epyc
> > > >>>>>> x86_topo_ids_from_idx_epyc
> > > >>>>>> x86_topo_ids_from_apicid_epyc
> > > >>>>>> x86_apicid_from_cpu_idx_epyc
> > > >>>>>>
> > > >>>>>> The topology details are available in Processor Programming
> > Reference
> > > >>> (PPR)
> > > >>>>>> for AMD Family 17h Model 01h, Revision B1 Processors. The revision
> > > >>> guides
> > > >>>>> are
> > > >>>>>> available from the bugzilla Link below.
> > > >>>>>> Link:
> > > >>>>>
> > > >>>
> > https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugzilla.
> > > >>>>>
> > > >>>
> > kernel.org%2Fshow_bug.cgi%3Fid%3D206537&data=02%7C01%7Cbabu.m
> > > >>>>>
> > > >>>
> > oger%40amd.com%7C3487f40d37df4d59097d08d807190248%7C3dd8961fe488
> > > >>>>>
> > > >>>
> > 4e608e11a82d994e183d%7C0%7C0%7C637267151289763739&sdata=wE0
> > > >>>>> ukXIVh0l5eNQWsv6VDE9UZEVJmisofaW192gcZAI%3D&reserved=0
> > > >>>>>>
> > > >>>>>> Signed-off-by: Babu Moger <babu.moger@amd.com>
> > > >>>>>> Acked-by: Igor Mammedov <imammedo@redhat.com>
> > > >>>>>> Acked-by: Michael S. Tsirkin <mst@redhat.com>
> > > >>>>>> ---
> > > >>>>> [...]
> > > >>>>>> typedef struct X86CPUTopoIDs {
> > > >>>>>> unsigned pkg_id;
> > > >>>>>> + unsigned node_id;
> > > >>>>>
> > > >>>>> You have added a new field here.
> > > >>>>>
> > > >>>>>> unsigned die_id;
> > > >>>>>> unsigned core_id;
> > > >>>>>> unsigned smt_id;
> > > >>>>> [...]
> > > >>>>>> +static inline apic_id_t
> > > >>>>>> +x86_apicid_from_topo_ids_epyc(X86CPUTopoInfo *topo_info,
> > > >>>>>> + const X86CPUTopoIDs *topo_ids)
> > > >>>>>> +{
> > > >>>>>> + return (topo_ids->pkg_id << apicid_pkg_offset_epyc(topo_info)) |
> > > >>>>>> + (topo_ids->node_id << apicid_node_offset_epyc(topo_info)) |
> > > >>>>>
> > > >>>>> You are using the new field here.
> > > >>>>>
> > > >>>>>> + (topo_ids->die_id << apicid_die_offset(topo_info)) |
> > > >>>>>> + (topo_ids->core_id << apicid_core_offset(topo_info)) |
> > > >>>>>> + topo_ids->smt_id;
> > > >>>>>> +}
> > > >>>>>
> > > >>>>> But you are not initializing node_id in one caller of
> > apicid_from_topo_ids():
> > > >>>>>
> > > >>>>> static void pc_cpu_pre_plug(HotplugHandler *hotplug_dev,
> > > >>>>> DeviceState *dev, Error **errp)
> > > >>>>> {
> > > >>>>> [...]
> > > >>>>> X86CPUTopoIDs topo_ids;
> > > >>>>> [...]
> > > >>>>> if (cpu->apic_id == UNASSIGNED_APIC_ID) {
> > > >>>>> [...]
> > > >>>>> topo_ids.pkg_id = cpu->socket_id;
> > > >>>>> topo_ids.die_id = cpu->die_id;
> > > >>>>> topo_ids.core_id = cpu->core_id;
> > > >>>>> topo_ids.smt_id = cpu->thread_id;
> > > >>>>> cpu->apic_id = x86ms->apicid_from_topo_ids(&topo_info,
> > &topo_ids);
> > > >>>>> }
> > > >>>>> [...]
> > > >>>>> }
> > > >>>>>
> > > >>>>> Result: -device is broken when using -cpu EPYC:
> > > >>>>>
> > > >>>>> $ qemu-system-x86_64 -machine q35,accel=kvm -smp
> > > >>>>> 1,maxcpus=2,cores=1,threads=1,sockets=2 -cpu EPYC -device EPYC-
> > x86_64-
> > > >>>>> cpu,core-id=0,socket-id=1,thread-id=0
> > > >>>
> > > >>> [1]
> > > >>>
> > > >>>>> qemu-system-x86_64: -device EPYC-x86_64-cpu,core-id=0,socket-
> > > >>> id=1,thread-
> > > >>>>> id=0: Invalid CPU [socket: 21855, die: 0, core: 0, thread: 0] with APIC ID
> > > >>> 21855,
> > > >>>>> valid index range 0:1
> > > >>>>>
> > > >>>>> This happens because APIC ID is calculated using uninitialized
> > > >>>>> memory.
> > > >>>> This patch should initialize the node_id. But I am not sure how to
> > > >>>> reproduce the bug. Can you please send me the full command line to
> > > >>>> reproduce the problem. Also test different options.
> > > >>>
> > > >>> The full command line is above[1].
> > > >>
> > > >> I just picked up the latest tree from
> > > >> git clone
> > https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit.qem
> > u.org%2Fgit%2Fqemu.git&data=02%7C01%7Cbabu.moger%40amd.com%7
> > C4b409b77ff2c4c4ed90608d807d53570%7C3dd8961fe4884e608e11a82d994e1
> > 83d%7C0%7C0%7C637267959601716206&sdata=ih6bo9Wp0RbJgRpryzSa2
> > D0v6kr3Zfww%2B1uB%2FNyngk8%3D&reserved=0 qemu
> > > >> Not seeing the problem.
> > > >>
> > > >> ./x86_64-softmmu/qemu-system-x86_64 -machine q35,accel=kvm -smp
> > > >> 1,maxcpus=2,cores=1,threads=1,sockets=2 -cpu EPYC -device
> > > >> EPYC-x86_64-cpu,core-id=0,socket-id=1,thread-id=0
> > > >> VNC server running on ::1:5900
> > > >>
> > > >> It appears to run ok.
> > >
> > > [2]
> > >
> > > >>
> > > >>>
> > > >>>
> > > >>>>
> > > >>>> diff --git a/hw/i386/pc.c b/hw/i386/pc.c
> > > >>>> index 2128f3d6fe..047b4b9391 100644
> > > >>>> --- a/hw/i386/pc.c
> > > >>>> +++ b/hw/i386/pc.c
> > > >>>> @@ -1539,6 +1539,9 @@ static void pc_cpu_pre_plug(HotplugHandler
> > > >>> *hotplug_dev,
> > > >>>> if (cpu->apic_id == UNASSIGNED_APIC_ID) {
> > > >>>> int max_socket = (ms->smp.max_cpus - 1) /
> > > >>>> smp_threads / smp_cores / x86ms->smp_dies;
> > > >>>
> > > >>> So, here's the input you are using to calculate topo_ids.node_id:
> > > >>>
> > > >>>> + unsigned nr_nodes = MAX(topo_info.nodes_per_pkg, 1);
> > > >>>
> > > >>> When is topo_info.nodes_per_pkg allowed to be 0?
> > > >>
> > > >> It will be zero if numa is not configured. Node_id will be zero for all
> > > >> the cores.
> > > >>
> > > >>>
> > > >>>> + unsigned cores_per_node = DIV_ROUND_UP((x86ms->smp_dies *
> > > >>> smp_cores *
> > > >>>> + smp_threads), nr_nodes);
> > > >>>
> > > >>> x86ms->smp_dies should be available at topo_info.dies_per_pkg,
> > > >>> smp_cores should available at topo_info.cores_per_die,
> > > >>> smp_threads should be available at topo_info.threads_per_core,
> > > >>> nr_nodes should be available at topo_info.nodes_per_pkg.
> > > >>>
> > > >>>>
> > > >>>> /*
> > > >>>> * die-id was optional in QEMU 4.0 and older, so keep it optional
> > > >>>> @@ -1585,6 +1588,7 @@ static void pc_cpu_pre_plug(HotplugHandler
> > > >>> *hotplug_dev,
> > > >>>> topo_ids.die_id = cpu->die_id;
> > > >>>> topo_ids.core_id = cpu->core_id;
> > > >>>> topo_ids.smt_id = cpu->thread_id;
> > > >>>> + topo_ids.node_id = (cpu->core_id / cores_per_node) % nr_nodes;
> > > >>>
> > > >>> apicid_from_topo_ids() have access to topo_info and topo_ids, If
> > > >>> all the information you need to calculate node_id is already
> > > >>> available inside topo_info + topo_ids, we could be calculating it
> > > >>> inside apicid_from_topo_ids(). Why don't we do it?
> > > >>>
> > > >>> Also, is topo_ids.core_id really allowed to be larger than
> > > >>> cores_per_node when calling apicid_from_topo_ids()?
> > > >>
> > > >> Yes. It is. If we have two numa nodes and 8 cores. Then cores_per_node is
> > > >> 4. Nr_nodes =2. Yes. Core_id can be larger than cores_per_node.
> > > >
> > > > I assumed core_id identified the core inside a die, and the range
> > > > would be [0, cores_per_die). This seems to be what
> > > > apicid_from_topo_ids_epyc() expects.
> > > >
> > > > We need clearer documentation on the semantics of each *_id
> > > > field, to avoid confusion.
> > >
> > > Ok. Sure. I will add that. Can you please clarify on how to repro the
> > > issue. Marked the question at [2].
> >
> > As it is usage of uninitialized memory, behavior is undefined and
> > not guaranteed to crash. My QEMU binary was built with
> > --enable-debug, maybe it makes it easier to reproduce.
>
> This patch works fine. Tested few combination. Will send the formal
> version if patch looks good.
>
> diff --git a/include/hw/i386/topology.h b/include/hw/i386/topology.h
> index 07239f95f4..cb28f0a57c 100644
> --- a/include/hw/i386/topology.h
> +++ b/include/hw/i386/topology.h
> @@ -140,6 +140,17 @@ static inline unsigned
> apicid_pkg_offset_epyc(X86CPUTopoInfo *topo_info)
> apicid_node_width_epyc(topo_info);
> }
>
> +static inline unsigned x86_node_id_for_epyc(X86CPUTopoInfo *topo_info,
> + const X86CPUTopoIDs *topo_ids)
> +{
> + unsigned nr_nodes = MAX(topo_info->nodes_per_pkg, 1);
> + unsigned cores_per_node = DIV_ROUND_UP((topo_info->dies_per_pkg *
> + topo_info->cores_per_die *
> + topo_info->threads_per_core),
> + nr_nodes);
> +
> + return ((topo_ids->core_id / cores_per_node) % nr_nodes);
> +}
> /*
> * Make APIC ID for the CPU based on Pkg_ID, Core_ID, SMT_ID
> *
> @@ -149,8 +160,11 @@ static inline apic_id_t
> x86_apicid_from_topo_ids_epyc(X86CPUTopoInfo *topo_info,
> const X86CPUTopoIDs *topo_ids)
> {
> + /* node_id can be uninitialized. Initialize here */
> + unsigned node_id = x86_node_id_for_epyc(topo_info, topo_ids);
> +
> return (topo_ids->pkg_id << apicid_pkg_offset_epyc(topo_info)) |
> - (topo_ids->node_id << apicid_node_offset_epyc(topo_info)) |
> + (node_id << apicid_node_offset_epyc(topo_info)) |
> (topo_ids->die_id << apicid_die_offset(topo_info)) |
> (topo_ids->core_id << apicid_core_offset(topo_info)) |
> topo_ids->smt_id;
>
What's the status of this? It seems to fix a hotplug case our QE hit
(RH bz 1828750)
Dave
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
^ permalink raw reply [flat|nested] 38+ messages in thread
end of thread, other threads:[~2020-06-15 11:45 UTC | newest]
Thread overview: 38+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-03-11 22:52 [PATCH v7 00/13] APIC ID fixes for AMD EPYC CPU model Babu Moger
2020-03-11 22:52 ` [PATCH v7 01/13] hw/i386: Introduce X86CPUTopoInfo to contain topology info Babu Moger
2020-03-12 11:11 ` Igor Mammedov
2020-03-11 22:52 ` [PATCH v7 02/13] hw/i386: Consolidate topology functions Babu Moger
2020-03-11 22:53 ` [PATCH v7 03/13] machine: Add SMP Sockets in CpuTopology Babu Moger
2020-03-11 22:53 ` [PATCH v7 04/13] hw/i386: Remove unnecessary initialization in x86_cpu_new Babu Moger
2020-03-11 22:53 ` [PATCH v7 05/13] hw/i386: Update structures to save the number of nodes per package Babu Moger
2020-03-11 22:53 ` [PATCH v7 06/13] hw/i386: Rename apicid_from_topo_ids to x86_apicid_from_topo_ids Babu Moger
2020-03-11 22:53 ` [PATCH v7 07/13] hw/386: Add EPYC mode topology decoding functions Babu Moger
2020-03-12 12:39 ` Igor Mammedov
2020-03-12 13:44 ` Babu Moger
2020-06-02 17:18 ` Eduardo Habkost
2020-06-02 17:27 ` Babu Moger
2020-06-02 21:59 ` Babu Moger
2020-06-02 23:01 ` Eduardo Habkost
2020-06-03 0:07 ` Babu Moger
2020-06-03 15:22 ` Babu Moger
2020-06-03 15:31 ` Eduardo Habkost
2020-06-03 15:38 ` Babu Moger
2020-06-03 15:45 ` Eduardo Habkost
2020-06-03 15:49 ` Babu Moger
2020-06-03 21:49 ` Babu Moger
2020-06-15 11:44 ` Dr. David Alan Gilbert
2020-03-11 22:53 ` [PATCH v7 08/13] target/i386: Cleanup and use the EPYC mode topology functions Babu Moger
2020-03-11 22:53 ` [PATCH v7 09/13] hw/i386: Introduce apicid functions inside X86MachineState Babu Moger
2020-03-11 22:53 ` [PATCH v7 10/13] i386: Introduce use_epyc_apic_id_encoding in X86CPUDefinition Babu Moger
2020-03-12 12:26 ` Igor Mammedov
2020-03-11 22:54 ` [PATCH v7 11/13] hw/i386: Move arch_id decode inside x86_cpus_init Babu Moger
2020-03-12 12:31 ` Igor Mammedov
2020-03-11 22:54 ` [PATCH v7 12/13] target/i386: Enable new apic id encoding for EPYC based cpus models Babu Moger
2020-03-11 22:54 ` [PATCH v7 13/13] i386: Fix pkg_id offset for EPYC cpu models Babu Moger
2020-03-12 16:28 ` [PATCH v7 00/13] APIC ID fixes for AMD EPYC CPU model Babu Moger
2020-03-17 23:22 ` Eduardo Habkost
2020-03-17 23:31 ` Moger, Babu
2020-03-17 23:46 ` Eduardo Habkost
2020-03-18 2:43 ` Moger, Babu
2020-03-18 10:47 ` Igor Mammedov
2020-03-23 15:35 ` Moger, Babu
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.