[patch 00/53] x86/topology: The final installment

[patch 00/53] x86/topology: The final installment

[Thomas Gleixner]

Hi!

This is the (for now) last part of reworking topology enumeration and
management. It's based on the APIC and CPUID rework series which can be
found here:

      https://lore.kernel.org/lkml/20230802101635.459108805@linutronix.de

With these preparatory changes in place, it's now possible to address the
real issues of the current topology code:

  - Wrong core count on hybrid systems

  - Heuristics based size information for packages and dies which
    are failing to work correctly with certain command line parameters.

  - Full evaluation fail for a theoretical hybrid system which boots
    from an E-core

  - The complete insanity of manipulating global data from firmware parsers
    or the XEN/PV fake SMP enumeration. The latter is really a piece of art.

This series addresses this by

  - Mopping up some more historical technical debt

  - Consolidating all topology relevant functionality into one place

  - Providing separate interfaces for boot time and ACPI hotplug operations

  - A sane ordering of command line options and restrictions

  - A sensible way to handle the BSP problem in kdump kernels instead of
    the unreliable command line option.

  - Confinement of topology relevant variables by replacing the XEN/PV SMP
    enumeration fake with something halfways sensible.

  - Evaluation of sizes by analysing the topology via the CPUID provided
    APIC ID segmentation and the actual APIC IDs which are registered at
    boot time.

  - Removal of heuristics and broken size calculations

The idea behind this is the following:

The APIC IDs describe the system topology in multiple domain levels. The
CPUID topology parser provides the information which part of the APIC ID is
associated to the individual levels (Intel terminology):

   [ROOT][PACKAGE][DIE][TILE][MODULE][CORE][THREAD]

The root space contains the package (socket) IDs. Not enumerated levels
consume 0 bits space, but conceptually they are always represented. If
e.g. only CORE and THREAD levels are enumerated then the DIE, MODULE and
TILE have the same physical ID as the PACKAGE.

If SMT is not supported, then the THREAD domain is still used. It then
has the same physical ID as the CORE domain and is the only child of
the core domain.

This allows an unified view on the system independent of the enumerated
domain levels without requiring any conditionals in the code.

AMD does only expose 4 domain levels with obviously different terminology,
but that can be easily mapped into the Intel variant with a trivial lookup
table added to the CPUID parser.

The resulting topology information of an ADL hybrid system with 8 P-Cores
and 8 E-Cores looks like this:

 CPU topo: Max. logical packages:   1
 CPU topo: Max. logical dies:       1
 CPU topo: Max. dies per package:   1
 CPU topo: Max. threads per core:   2
 CPU topo: Num. cores per package:    16
 CPU topo: Num. threads per package:  24
 CPU topo: Allowing 24 present CPUs plus 0 hotplug CPUs
 CPU topo: Thread    :    24
 CPU topo: Core      :    16
 CPU topo: Module    :     1
 CPU topo: Tile      :     1
 CPU topo: Die       :     1
 CPU topo: Package   :     1

This is happening on the boot CPU before any of the APs is started and
provides correct size information right from the start.

Even the XEN/PV trainwreck makes use of this now. On Dom0 it utilizes the
MADT and on DomU it provides fake APIC IDs, which combined with the
provided CPUID information make it at least look halfways realistic instead
of claiming to have one CPU per package as the current upstream code does.

This is solely addressing the core topology issues, but there is a plan for
further consolidation of other topology related information into one single
source of information instead of having a gazillion of localized special
parsers and representations all over the place. There are quite some other
things which can be simplified on top of this, like updating the various
cpumasks during CPU bringup, but that's all left for later.

So another 53 patches later, the resulting diffstat is:

   64 files changed, 830 insertions(+), 955 deletions(-)

and the combo diffstat of all three series combined:

  115 files changed, 2414 insertions(+), 3035 deletions(-)

The current series applies on top of 

   git://git.kernel.org/pub/scm/linux/kernel/git/tglx/devel.git topo-cpuid-v3

and is available from git here:

   git://git.kernel.org/pub/scm/linux/kernel/git/tglx/devel.git topo-full-v1

Thanks,

	tglx
---
 Documentation/admin-guide/kdump/kdump.rst                      |    7 
 Documentation/admin-guide/kernel-parameters.txt                |    9 
 Documentation/arch/x86/topology.rst                            |   24 
 arch/x86/events/intel/cstate.c                                 |    2 
 arch/x86/events/intel/uncore.c                                 |    2 
 arch/x86/events/intel/uncore_nhmex.c                           |    4 
 arch/x86/events/intel/uncore_snb.c                             |    8 
 arch/x86/events/intel/uncore_snbep.c                           |   18 
 arch/x86/events/rapl.c                                         |    2 
 arch/x86/include/asm/apic.h                                    |   17 
 arch/x86/include/asm/io_apic.h                                 |    1 
 arch/x86/include/asm/mpspec.h                                  |   66 -
 arch/x86/include/asm/perf_event_p4.h                           |   11 
 arch/x86/include/asm/processor.h                               |    2 
 arch/x86/include/asm/prom.h                                    |    4 
 arch/x86/include/asm/smp.h                                     |    6 
 arch/x86/include/asm/topology.h                                |   60 -
 arch/x86/include/asm/x86_init.h                                |   10 
 arch/x86/kernel/acpi/boot.c                                    |   59 -
 arch/x86/kernel/apic/apic.c                                    |  207 ----
 arch/x86/kernel/apic/apic_common.c                             |   15 
 arch/x86/kernel/apic/apic_flat_64.c                            |    9 
 arch/x86/kernel/apic/apic_noop.c                               |    2 
 arch/x86/kernel/apic/apic_numachip.c                           |   12 
 arch/x86/kernel/apic/bigsmp_32.c                               |   14 
 arch/x86/kernel/apic/io_apic.c                                 |   85 -
 arch/x86/kernel/apic/local.h                                   |    4 
 arch/x86/kernel/apic/probe_32.c                                |    3 
 arch/x86/kernel/apic/x2apic_cluster.c                          |    3 
 arch/x86/kernel/apic/x2apic_phys.c                             |    6 
 arch/x86/kernel/apic/x2apic_uv_x.c                             |    6 
 arch/x86/kernel/cpu/Makefile                                   |   12 
 arch/x86/kernel/cpu/cacheinfo.c                                |    2 
 arch/x86/kernel/cpu/common.c                                   |   42 
 arch/x86/kernel/cpu/debugfs.c                                  |   15 
 arch/x86/kernel/cpu/mce/inject.c                               |    3 
 arch/x86/kernel/cpu/microcode/intel.c                          |    4 
 arch/x86/kernel/cpu/topology.c                                 |  497 ++++++++++
 arch/x86/kernel/cpu/topology.h                                 |    5 
 arch/x86/kernel/cpu/topology_common.c                          |   57 -
 arch/x86/kernel/devicetree.c                                   |    4 
 arch/x86/kernel/jailhouse.c                                    |   30 
 arch/x86/kernel/mpparse.c                                      |   35 
 arch/x86/kernel/process.c                                      |    2 
 arch/x86/kernel/setup.c                                        |   27 
 arch/x86/kernel/smpboot.c                                      |  229 ----
 arch/x86/kernel/x86_init.c                                     |    5 
 arch/x86/mm/amdtopology.c                                      |    7 
 arch/x86/platform/ce4100/ce4100.c                              |   14 
 arch/x86/platform/intel-mid/intel-mid.c                        |    5 
 arch/x86/xen/apic.c                                            |   21 
 arch/x86/xen/enlighten_hvm.c                                   |    2 
 arch/x86/xen/enlighten_pv.c                                    |    3 
 arch/x86/xen/smp.c                                             |    2 
 arch/x86/xen/smp.h                                             |    2 
 arch/x86/xen/smp_pv.c                                          |   66 -
 drivers/gpu/drm/amd/pm/swsmu/smu11/vangogh_ppt.c               |    2 
 drivers/hwmon/coretemp.c                                       |    2 
 drivers/hwmon/fam15h_power.c                                   |    2 
 drivers/platform/x86/intel/uncore-frequency/uncore-frequency.c |    2 
 drivers/powercap/intel_rapl_common.c                           |    2 
 drivers/thermal/intel/intel_hfi.c                              |    2 
 drivers/thermal/intel/intel_powerclamp.c                       |    2 
 drivers/thermal/intel/x86_pkg_temp_thermal.c                   |    2 
 64 files changed, 830 insertions(+), 955 deletions(-)

[patch 01/53] x86/cpu/topology: Cure off by one in fake_topology()

[Thomas Gleixner]

The shift count for a single core CPU is obviously 0 and not 1.

Not that it matters much, but keep it correct.

Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
---
 arch/x86/kernel/cpu/topology_common.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

--- a/arch/x86/kernel/cpu/topology_common.c
+++ b/arch/x86/kernel/cpu/topology_common.c
@@ -75,7 +75,7 @@ static bool fake_topology(struct topo_sc
 	 * which has useless CPUID information.
 	 */
 	topology_set_dom(tscan, TOPO_SMT_DOMAIN, 0, 1);
-	topology_set_dom(tscan, TOPO_CORE_DOMAIN, 1, 1);
+	topology_set_dom(tscan, TOPO_CORE_DOMAIN, 0, 1);
 
 	return tscan->c->cpuid_level < 1 || xen_pv_domain();
 }

[patch 02/53] x86/cpu/topology: Make the APIC mismatch warnings complete

[Thomas Gleixner]

Detect all possible combinations of mismatch right in the CPUID evaluation
code.

Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
---
 arch/x86/kernel/cpu/common.c          |   15 ++-------------
 arch/x86/kernel/cpu/topology_common.c |   10 ++++++++++
 2 files changed, 12 insertions(+), 13 deletions(-)

--- a/arch/x86/kernel/cpu/common.c
+++ b/arch/x86/kernel/cpu/common.c
@@ -1676,22 +1676,11 @@ static void generic_identify(struct cpui
 #endif
 }
 
-/*
- * Validate that ACPI/mptables have the same information about the
- * effective APIC id and update the package map.
- */
-static void validate_apic_and_package_id(struct cpuinfo_x86 *c)
+static void update_package_map(struct cpuinfo_x86 *c)
 {
 #ifdef CONFIG_SMP
 	unsigned int cpu = smp_processor_id();
-	u32 apicid;
 
-	apicid = apic->cpu_present_to_apicid(cpu);
-
-	if (apicid != c->topo.apicid) {
-		pr_err(FW_BUG "CPU%u: APIC id mismatch. Firmware: %x APIC: %x\n",
-		       cpu, apicid, c->topo.initial_apicid);
-	}
 	BUG_ON(topology_update_package_map(c->topo.pkg_id, cpu));
 	BUG_ON(topology_update_die_map(c->topo.die_id, cpu));
 #else
@@ -1876,7 +1865,7 @@ void identify_secondary_cpu(struct cpuin
 #ifdef CONFIG_X86_32
 	enable_sep_cpu();
 #endif
-	validate_apic_and_package_id(c);
+	update_package_map(c);
 	x86_spec_ctrl_setup_ap();
 	update_srbds_msr();
 
--- a/arch/x86/kernel/cpu/topology_common.c
+++ b/arch/x86/kernel/cpu/topology_common.c
@@ -176,6 +176,16 @@ void cpu_parse_topology(struct cpuinfo_x
 
 	parse_topology(&tscan, false);
 
+	if (c->topo.initial_apicid != c->topo.apicid) {
+		pr_err(FW_BUG "CPU%4u: APIC ID mismatch. CPUID: 0x%04x APIC: 0x%04x\n",
+		       cpu, c->topo.initial_apicid, c->topo.apicid);
+	}
+
+	if (c->topo.apicid != cpuid_to_apicid[cpu]) {
+		pr_err(FW_BUG "CPU%4u: APIC ID mismatch. Firmware: 0x%04x APIC: 0x%04x\n",
+		       cpu, cpuid_to_apicid[cpu], c->topo.apicid);
+	}
+
 	for (dom = TOPO_SMT_DOMAIN; dom < TOPO_MAX_DOMAIN; dom++) {
 		if (tscan.dom_shifts[dom] == x86_topo_system.dom_shifts[dom])
 			continue;

[patch 03/53] x86/platform/ce4100: Dont override x86_init.mpparse.setup_ioapic_ids

[Thomas Gleixner]

There is no point to do that. The ATOMs have an XAPIC for which this
function is a pointless exercise.

Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Cc: Andy Shevchenko <andy@infradead.org>
---
 arch/x86/include/asm/io_apic.h    |    1 -
 arch/x86/kernel/apic/io_apic.c    |    2 +-
 arch/x86/platform/ce4100/ce4100.c |    1 -
 3 files changed, 1 insertion(+), 3 deletions(-)

--- a/arch/x86/include/asm/io_apic.h
+++ b/arch/x86/include/asm/io_apic.h
@@ -140,7 +140,6 @@ extern void mask_ioapic_entries(void);
 extern int restore_ioapic_entries(void);
 
 extern void setup_ioapic_ids_from_mpc(void);
-extern void setup_ioapic_ids_from_mpc_nocheck(void);
 
 extern int mp_find_ioapic(u32 gsi);
 extern int mp_find_ioapic_pin(int ioapic, u32 gsi);
--- a/arch/x86/kernel/apic/io_apic.c
+++ b/arch/x86/kernel/apic/io_apic.c
@@ -1458,7 +1458,7 @@ void restore_boot_irq_mode(void)
  *
  * by Matt Domsch <Matt_Domsch@dell.com>  Tue Dec 21 12:25:05 CST 1999
  */
-void __init setup_ioapic_ids_from_mpc_nocheck(void)
+static void __init setup_ioapic_ids_from_mpc_nocheck(void)
 {
 	union IO_APIC_reg_00 reg_00;
 	physid_mask_t phys_id_present_map;
--- a/arch/x86/platform/ce4100/ce4100.c
+++ b/arch/x86/platform/ce4100/ce4100.c
@@ -139,7 +139,6 @@ void __init x86_ce4100_early_setup(void)
 	x86_init.resources.probe_roms = x86_init_noop;
 	x86_init.mpparse.get_smp_config = x86_init_uint_noop;
 	x86_init.mpparse.find_smp_config = x86_init_noop;
-	x86_init.mpparse.setup_ioapic_ids = setup_ioapic_ids_from_mpc_nocheck;
 	x86_init.pci.init = ce4100_pci_init;
 	x86_init.pci.init_irq = sdv_pci_init;
 

[patch 04/53] x86/ioapic: Replace some more set bit nonsense

[Thomas Gleixner]

Yet another set_bit() operation wrapped in oring a mask.

Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
---
 arch/x86/include/asm/mpspec.h  |    3 ---
 arch/x86/kernel/apic/io_apic.c |    6 ++----
 2 files changed, 2 insertions(+), 7 deletions(-)

--- a/arch/x86/include/asm/mpspec.h
+++ b/arch/x86/include/asm/mpspec.h
@@ -86,9 +86,6 @@ typedef struct physid_mask physid_mask_t
 #define physid_set(physid, map)			set_bit(physid, (map).mask)
 #define physid_isset(physid, map)		test_bit(physid, (map).mask)
 
-#define physids_or(dst, src1, src2)					\
-	bitmap_or((dst).mask, (src1).mask, (src2).mask, MAX_LOCAL_APIC)
-
 #define physids_clear(map)					\
 	bitmap_zero((map).mask, MAX_LOCAL_APIC)
 
--- a/arch/x86/kernel/apic/io_apic.c
+++ b/arch/x86/kernel/apic/io_apic.c
@@ -2494,9 +2494,8 @@ unsigned int arch_dynirq_lower_bound(uns
 #ifdef CONFIG_X86_32
 static int io_apic_get_unique_id(int ioapic, int apic_id)
 {
-	union IO_APIC_reg_00 reg_00;
 	static physid_mask_t apic_id_map = PHYSID_MASK_NONE;
-	physid_mask_t tmp;
+	union IO_APIC_reg_00 reg_00;
 	unsigned long flags;
 	int i = 0;
 
@@ -2542,8 +2541,7 @@ static int io_apic_get_unique_id(int ioa
 		apic_id = i;
 	}
 
-	physid_set_mask_of_physid(apic_id, &tmp);
-	physids_or(apic_id_map, apic_id_map, tmp);
+	physid_set(apic_id, apic_id_map);
 
 	if (reg_00.bits.ID != apic_id) {
 		reg_00.bits.ID = apic_id;

[patch 05/53] x86/apic: Get rid of get_physical_broadcast()

[Thomas Gleixner]

There is no point for this function. The only case where this is used when
there is no XAPIC available, which means the broadcast address is 0xF.

Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
---
 arch/x86/kernel/apic/apic.c    |   10 ---------
 arch/x86/kernel/apic/io_apic.c |   42 ++++++++++++++++++-----------------------
 2 files changed, 19 insertions(+), 33 deletions(-)

--- a/arch/x86/kernel/apic/apic.c
+++ b/arch/x86/kernel/apic/apic.c
@@ -259,16 +259,6 @@ u64 native_apic_icr_read(void)
 	return icr1 | ((u64)icr2 << 32);
 }
 
-#ifdef CONFIG_X86_32
-/**
- * get_physical_broadcast - Get number of physical broadcast IDs
- */
-int get_physical_broadcast(void)
-{
-	return modern_apic() ? 0xff : 0xf;
-}
-#endif
-
 /**
  * lapic_get_maxlvt - get the maximum number of local vector table entries
  */
--- a/arch/x86/kernel/apic/io_apic.c
+++ b/arch/x86/kernel/apic/io_apic.c
@@ -1460,12 +1460,12 @@ void restore_boot_irq_mode(void)
  */
 static void __init setup_ioapic_ids_from_mpc_nocheck(void)
 {
-	union IO_APIC_reg_00 reg_00;
 	physid_mask_t phys_id_present_map;
-	int ioapic_idx;
-	int i;
+	const u32 broadcast_id = 0xF;
+	union IO_APIC_reg_00 reg_00;
 	unsigned char old_id;
 	unsigned long flags;
+	int ioapic_idx, i;
 
 	/*
 	 * This is broken; anything with a real cpu count has to
@@ -1484,11 +1484,10 @@ static void __init setup_ioapic_ids_from
 
 		old_id = mpc_ioapic_id(ioapic_idx);
 
-		if (mpc_ioapic_id(ioapic_idx) >= get_physical_broadcast()) {
-			printk(KERN_ERR "BIOS bug, IO-APIC#%d ID is %d in the MPC table!...\n",
-				ioapic_idx, mpc_ioapic_id(ioapic_idx));
-			printk(KERN_ERR "... fixing up to %d. (tell your hw vendor)\n",
-				reg_00.bits.ID);
+		if (mpc_ioapic_id(ioapic_idx) >= broadcast_id) {
+			pr_err("BIOS bug, IO-APIC#%d ID is %d in the MPC table!...\n",
+			       ioapic_idx, mpc_ioapic_id(ioapic_idx));
+			pr_err("... fixing up to %d. (tell your hw vendor)\n", reg_00.bits.ID);
 			ioapics[ioapic_idx].mp_config.apicid = reg_00.bits.ID;
 		}
 
@@ -1499,15 +1498,14 @@ static void __init setup_ioapic_ids_from
 		 */
 		if (apic->check_apicid_used(&phys_id_present_map,
 					    mpc_ioapic_id(ioapic_idx))) {
-			printk(KERN_ERR "BIOS bug, IO-APIC#%d ID %d is already used!...\n",
+			pr_err("BIOS bug, IO-APIC#%d ID %d is already used!...\n",
 				ioapic_idx, mpc_ioapic_id(ioapic_idx));
-			for (i = 0; i < get_physical_broadcast(); i++)
+			for (i = 0; i < broadcast_id; i++)
 				if (!physid_isset(i, phys_id_present_map))
 					break;
-			if (i >= get_physical_broadcast())
+			if (i >= broadcast_id)
 				panic("Max APIC ID exceeded!\n");
-			printk(KERN_ERR "... fixing up to %d. (tell your hw vendor)\n",
-				i);
+			pr_err("... fixing up to %d. (tell your hw vendor)\n", i);
 			physid_set(i, phys_id_present_map);
 			ioapics[ioapic_idx].mp_config.apicid = i;
 		} else {
@@ -2495,6 +2493,7 @@ unsigned int arch_dynirq_lower_bound(uns
 static int io_apic_get_unique_id(int ioapic, int apic_id)
 {
 	static physid_mask_t apic_id_map = PHYSID_MASK_NONE;
+	const u32 broadcast_id = 0xF;
 	union IO_APIC_reg_00 reg_00;
 	unsigned long flags;
 	int i = 0;
@@ -2515,9 +2514,9 @@ static int io_apic_get_unique_id(int ioa
 	reg_00.raw = io_apic_read(ioapic, 0);
 	raw_spin_unlock_irqrestore(&ioapic_lock, flags);
 
-	if (apic_id >= get_physical_broadcast()) {
-		printk(KERN_WARNING "IOAPIC[%d]: Invalid apic_id %d, trying "
-			"%d\n", ioapic, apic_id, reg_00.bits.ID);
+	if (apic_id >= broadcast_id) {
+		pr_warn("IOAPIC[%d]: Invalid apic_id %d, trying %d\n",
+			ioapic, apic_id, reg_00.bits.ID);
 		apic_id = reg_00.bits.ID;
 	}
 
@@ -2527,17 +2526,15 @@ static int io_apic_get_unique_id(int ioa
 	 */
 	if (apic->check_apicid_used(&apic_id_map, apic_id)) {
 
-		for (i = 0; i < get_physical_broadcast(); i++) {
+		for (i = 0; i < broadcast_id; i++) {
 			if (!apic->check_apicid_used(&apic_id_map, i))
 				break;
 		}
 
-		if (i == get_physical_broadcast())
+		if (i == broadcast_id)
 			panic("Max apic_id exceeded!\n");
 
-		printk(KERN_WARNING "IOAPIC[%d]: apic_id %d already used, "
-			"trying %d\n", ioapic, apic_id, i);
-
+		pr_warn("IOAPIC[%d]: apic_id %d already used, trying %d\n", ioapic, apic_id, i);
 		apic_id = i;
 	}
 
@@ -2567,8 +2564,7 @@ static int io_apic_get_unique_id(int ioa
 
 static u8 io_apic_unique_id(int idx, u8 id)
 {
-	if ((boot_cpu_data.x86_vendor == X86_VENDOR_INTEL) &&
-	    !APIC_XAPIC(boot_cpu_apic_version))
+	if ((boot_cpu_data.x86_vendor == X86_VENDOR_INTEL) && !APIC_XAPIC(boot_cpu_apic_version))
 		return io_apic_get_unique_id(idx, id);
 	else
 		return id;

[patch 06/53] x86/ioapic: Make io_apic_get_unique_id() simpler

[Thomas Gleixner]

No need to go through APIC callbacks. It's already established that this is
an ancient APIC. So just copy the present mask and use the direct physid*
functions all over the place.

Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
---
 arch/x86/kernel/apic/io_apic.c |   22 +++++-----------------
 1 file changed, 5 insertions(+), 17 deletions(-)

--- a/arch/x86/kernel/apic/io_apic.c
+++ b/arch/x86/kernel/apic/io_apic.c
@@ -2498,17 +2498,9 @@ static int io_apic_get_unique_id(int ioa
 	unsigned long flags;
 	int i = 0;
 
-	/*
-	 * The P4 platform supports up to 256 APIC IDs on two separate APIC
-	 * buses (one for LAPICs, one for IOAPICs), where predecessors only
-	 * supports up to 16 on one shared APIC bus.
-	 *
-	 * TBD: Expand LAPIC/IOAPIC support on P4-class systems to take full
-	 *      advantage of new APIC bus architecture.
-	 */
-
+	/* Initialize the ID map */
 	if (physids_empty(apic_id_map))
-		apic->ioapic_phys_id_map(&phys_cpu_present_map, &apic_id_map);
+		apic_id_map = phys_cpu_present_map;
 
 	raw_spin_lock_irqsave(&ioapic_lock, flags);
 	reg_00.raw = io_apic_read(ioapic, 0);
@@ -2520,14 +2512,10 @@ static int io_apic_get_unique_id(int ioa
 		apic_id = reg_00.bits.ID;
 	}
 
-	/*
-	 * Every APIC in a system must have a unique ID or we get lots of nice
-	 * 'stuck on smp_invalidate_needed IPI wait' messages.
-	 */
-	if (apic->check_apicid_used(&apic_id_map, apic_id)) {
-
+	/* Every APIC in a system must have a unique ID */
+	if (physid_isset(apic_id, apic_id_map)) {
 		for (i = 0; i < broadcast_id; i++) {
-			if (!apic->check_apicid_used(&apic_id_map, i))
+			if (!physid_isset(i, apic_id_map))
 				break;
 		}
 

[patch 07/53] x86/ioapic: Simplify setup_ioapic_ids_from_mpc_nocheck()

[Thomas Gleixner]

No need to go through APIC callbacks. It's already established that this is
an ancient APIC. So just copy the present mask and use the direct physid*
functions all over the place.

Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
---
 arch/x86/kernel/apic/io_apic.c |    5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)

--- a/arch/x86/kernel/apic/io_apic.c
+++ b/arch/x86/kernel/apic/io_apic.c
@@ -1471,7 +1471,7 @@ static void __init setup_ioapic_ids_from
 	 * This is broken; anything with a real cpu count has to
 	 * circumvent this idiocy regardless.
 	 */
-	apic->ioapic_phys_id_map(&phys_cpu_present_map, &phys_id_present_map);
+	phys_id_present_map = phys_cpu_present_map;
 
 	/*
 	 * Set the IOAPIC ID to the value stored in the MPC table.
@@ -1496,8 +1496,7 @@ static void __init setup_ioapic_ids_from
 		 * system must have a unique ID or we get lots of nice
 		 * 'stuck on smp_invalidate_needed IPI wait' messages.
 		 */
-		if (apic->check_apicid_used(&phys_id_present_map,
-					    mpc_ioapic_id(ioapic_idx))) {
+		if (physid_isset(mpc_ioapic_id(ioapic_idx), phys_id_present_map)) {
 			pr_err("BIOS bug, IO-APIC#%d ID %d is already used!...\n",
 				ioapic_idx, mpc_ioapic_id(ioapic_idx));
 			for (i = 0; i < broadcast_id; i++)

[patch 08/53] x86/apic: Remove check_apicid_used() and ioapic_phys_id_map()

[Thomas Gleixner]

No more users.

Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
---
 arch/x86/include/asm/apic.h           |    3 ---
 arch/x86/include/asm/mpspec.h         |    6 ------
 arch/x86/kernel/apic/apic_noop.c      |    2 --
 arch/x86/kernel/apic/bigsmp_32.c      |   13 -------------
 arch/x86/kernel/apic/probe_32.c       |    2 --
 arch/x86/kernel/apic/x2apic_cluster.c |    2 --
 6 files changed, 28 deletions(-)

--- a/arch/x86/include/asm/apic.h
+++ b/arch/x86/include/asm/apic.h
@@ -292,9 +292,7 @@ struct apic {
 	int	(*acpi_madt_oem_check)(char *oem_id, char *oem_table_id);
 	bool	(*apic_id_registered)(void);
 
-	bool	(*check_apicid_used)(physid_mask_t *map, u32 apicid);
 	void	(*init_apic_ldr)(void);
-	void	(*ioapic_phys_id_map)(physid_mask_t *phys_map, physid_mask_t *retmap);
 	u32	(*cpu_present_to_apicid)(int mps_cpu);
 
 	u32	(*get_apic_id)(u32 id);
@@ -527,7 +525,6 @@ extern int default_apic_id_valid(u32 api
 extern u32 apic_default_calc_apicid(unsigned int cpu);
 extern u32 apic_flat_calc_apicid(unsigned int cpu);
 
-extern void default_ioapic_phys_id_map(physid_mask_t *phys_map, physid_mask_t *retmap);
 extern u32 default_cpu_present_to_apicid(int mps_cpu);
 
 #else /* CONFIG_X86_LOCAL_APIC */
--- a/arch/x86/include/asm/mpspec.h
+++ b/arch/x86/include/asm/mpspec.h
@@ -92,12 +92,6 @@ typedef struct physid_mask physid_mask_t
 #define physids_empty(map)					\
 	bitmap_empty((map).mask, MAX_LOCAL_APIC)
 
-static inline void physids_promote(unsigned long physids, physid_mask_t *map)
-{
-	physids_clear(*map);
-	map->mask[0] = physids;
-}
-
 static inline void physid_set_mask_of_physid(int physid, physid_mask_t *map)
 {
 	physids_clear(*map);
--- a/arch/x86/kernel/apic/apic_noop.c
+++ b/arch/x86/kernel/apic/apic_noop.c
@@ -51,8 +51,6 @@ struct apic apic_noop __ro_after_init =
 
 	.disable_esr			= 0,
 
-	.check_apicid_used		= default_check_apicid_used,
-	.ioapic_phys_id_map		= default_ioapic_phys_id_map,
 	.cpu_present_to_apicid		= default_cpu_present_to_apicid,
 
 	.max_apic_id			= 0xFE,
--- a/arch/x86/kernel/apic/bigsmp_32.c
+++ b/arch/x86/kernel/apic/bigsmp_32.c
@@ -18,17 +18,6 @@ static u32 bigsmp_get_apic_id(u32 x)
 	return (x >> 24) & 0xFF;
 }
 
-static bool bigsmp_check_apicid_used(physid_mask_t *map, u32 apicid)
-{
-	return false;
-}
-
-static void bigsmp_ioapic_phys_id_map(physid_mask_t *phys_map, physid_mask_t *retmap)
-{
-	/* For clustered we don't have a good way to do this yet - hack */
-	physids_promote(0xFFL, retmap);
-}
-
 static void bigsmp_send_IPI_allbutself(int vector)
 {
 	default_send_IPI_mask_allbutself_phys(cpu_online_mask, vector);
@@ -80,8 +69,6 @@ static struct apic apic_bigsmp __ro_afte
 
 	.disable_esr			= 1,
 
-	.check_apicid_used		= bigsmp_check_apicid_used,
-	.ioapic_phys_id_map		= bigsmp_ioapic_phys_id_map,
 	.cpu_present_to_apicid		= default_cpu_present_to_apicid,
 
 	.max_apic_id			= 0xFE,
--- a/arch/x86/kernel/apic/probe_32.c
+++ b/arch/x86/kernel/apic/probe_32.c
@@ -45,9 +45,7 @@ static struct apic apic_default __ro_aft
 
 	.disable_esr			= 0,
 
-	.check_apicid_used		= default_check_apicid_used,
 	.init_apic_ldr			= default_init_apic_ldr,
-	.ioapic_phys_id_map		= default_ioapic_phys_id_map,
 	.cpu_present_to_apicid		= default_cpu_present_to_apicid,
 
 	.max_apic_id			= 0xFE,
--- a/arch/x86/kernel/apic/x2apic_cluster.c
+++ b/arch/x86/kernel/apic/x2apic_cluster.c
@@ -232,9 +232,7 @@ static struct apic apic_x2apic_cluster _
 
 	.disable_esr			= 0,
 
-	.check_apicid_used		= NULL,
 	.init_apic_ldr			= init_x2apic_ldr,
-	.ioapic_phys_id_map		= NULL,
 	.cpu_present_to_apicid		= default_cpu_present_to_apicid,
 
 	.max_apic_id			= UINT_MAX,

[patch 09/53] x86/mpparse: Rename default_find_smp_config()

[Thomas Gleixner]

MPTABLE is not longer the default SMP configuration mechanism.
Rename it to mpparse_find_mptable() because that's what it does.

Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
---
 arch/x86/include/asm/mpspec.h           |   13 ++++---------
 arch/x86/include/asm/x86_init.h         |    4 ++--
 arch/x86/kernel/mpparse.c               |    2 +-
 arch/x86/kernel/setup.c                 |    6 ++----
 arch/x86/kernel/x86_init.c              |    2 +-
 arch/x86/platform/ce4100/ce4100.c       |    2 +-
 arch/x86/platform/intel-mid/intel-mid.c |    2 +-
 arch/x86/xen/smp_pv.c                   |    2 +-
 8 files changed, 13 insertions(+), 20 deletions(-)

--- a/arch/x86/include/asm/mpspec.h
+++ b/arch/x86/include/asm/mpspec.h
@@ -56,21 +56,16 @@ static inline void early_get_smp_config(
 	x86_init.mpparse.get_smp_config(1);
 }
 
-static inline void find_smp_config(void)
-{
-	x86_init.mpparse.find_smp_config();
-}
-
 #ifdef CONFIG_X86_MPPARSE
 extern void e820__memblock_alloc_reserved_mpc_new(void);
 extern int enable_update_mptable;
-extern void default_find_smp_config(void);
+extern void mpparse_find_mptable(void);
 extern void default_get_smp_config(unsigned int early);
 #else
 static inline void e820__memblock_alloc_reserved_mpc_new(void) { }
-#define enable_update_mptable 0
-#define default_find_smp_config x86_init_noop
-#define default_get_smp_config x86_init_uint_noop
+#define enable_update_mptable	(0)
+#define mpparse_find_mptable	x86_init_noop
+#define default_get_smp_config	x86_init_uint_noop
 #endif
 
 int generic_processor_info(int apicid);
--- a/arch/x86/include/asm/x86_init.h
+++ b/arch/x86/include/asm/x86_init.h
@@ -15,12 +15,12 @@ struct irq_domain;
 /**
  * struct x86_init_mpparse - platform specific mpparse ops
  * @setup_ioapic_ids:		platform specific ioapic id override
- * @find_smp_config:		find the smp configuration
+ * @find_mptable:		Find MPTABLE early to reserve the memory region
  * @get_smp_config:		get the smp configuration
  */
 struct x86_init_mpparse {
 	void (*setup_ioapic_ids)(void);
-	void (*find_smp_config)(void);
+	void (*find_mptable)(void);
 	void (*get_smp_config)(unsigned int early);
 };
 
--- a/arch/x86/kernel/mpparse.c
+++ b/arch/x86/kernel/mpparse.c
@@ -590,7 +590,7 @@ static int __init smp_scan_config(unsign
 	return ret;
 }
 
-void __init default_find_smp_config(void)
+void __init mpparse_find_mptable(void)
 {
 	unsigned int address;
 
--- a/arch/x86/kernel/setup.c
+++ b/arch/x86/kernel/setup.c
@@ -1101,10 +1101,8 @@ void __init setup_arch(char **cmdline_p)
 	high_memory = (void *)__va(max_pfn * PAGE_SIZE - 1) + 1;
 #endif
 
-	/*
-	 * Find and reserve possible boot-time SMP configuration:
-	 */
-	find_smp_config();
+	/* Find and reserve MPTABLE area */
+	x86_init.mpparse.find_mptable();
 
 	early_alloc_pgt_buf();
 
--- a/arch/x86/kernel/x86_init.c
+++ b/arch/x86/kernel/x86_init.c
@@ -70,7 +70,7 @@ struct x86_init_ops x86_init __initdata
 
 	.mpparse = {
 		.setup_ioapic_ids	= x86_init_noop,
-		.find_smp_config	= default_find_smp_config,
+		.find_mptable		= mpparse_find_mptable,
 		.get_smp_config		= default_get_smp_config,
 	},
 
--- a/arch/x86/platform/ce4100/ce4100.c
+++ b/arch/x86/platform/ce4100/ce4100.c
@@ -138,7 +138,7 @@ void __init x86_ce4100_early_setup(void)
 	x86_init.oem.arch_setup = sdv_arch_setup;
 	x86_init.resources.probe_roms = x86_init_noop;
 	x86_init.mpparse.get_smp_config = x86_init_uint_noop;
-	x86_init.mpparse.find_smp_config = x86_init_noop;
+	x86_init.mpparse.find_mptable = x86_init_noop;
 	x86_init.pci.init = ce4100_pci_init;
 	x86_init.pci.init_irq = sdv_pci_init;
 
--- a/arch/x86/platform/intel-mid/intel-mid.c
+++ b/arch/x86/platform/intel-mid/intel-mid.c
@@ -118,7 +118,7 @@ void __init x86_intel_mid_early_setup(vo
 	machine_ops.emergency_restart  = intel_mid_reboot;
 
 	/* Avoid searching for BIOS MP tables */
-	x86_init.mpparse.find_smp_config = x86_init_noop;
+	x86_init.mpparse.find_mptable = x86_init_noop;
 	x86_init.mpparse.get_smp_config = x86_init_uint_noop;
 	set_bit(MP_BUS_ISA, mp_bus_not_pci);
 }
--- a/arch/x86/xen/smp_pv.c
+++ b/arch/x86/xen/smp_pv.c
@@ -455,6 +455,6 @@ void __init xen_smp_init(void)
 	smp_ops = xen_smp_ops;
 
 	/* Avoid searching for BIOS MP tables */
-	x86_init.mpparse.find_smp_config = x86_init_noop;
+	x86_init.mpparse.find_mptable = x86_init_noop;
 	x86_init.mpparse.get_smp_config = _get_smp_config;
 }

[patch 10/53] x86/mpparse: Provide separate early/late callbacks

[Thomas Gleixner]

The early argument of x86_init::mpparse::get_smp_config() is more than
confusing. Provide two callbacks, one for each purpose.

Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
---
 arch/x86/include/asm/x86_init.h |    4 ++++
 1 file changed, 4 insertions(+)

--- a/arch/x86/include/asm/x86_init.h
+++ b/arch/x86/include/asm/x86_init.h
@@ -17,11 +17,15 @@ struct irq_domain;
  * @setup_ioapic_ids:		platform specific ioapic id override
  * @find_mptable:		Find MPTABLE early to reserve the memory region
  * @get_smp_config:		get the smp configuration
+ * @early_parse_smp_cfg:	Parse the SMP configuration data early before initmem_init()
+ * @parse_smp_cfg:		Parse the SMP configuration data
  */
 struct x86_init_mpparse {
 	void (*setup_ioapic_ids)(void);
 	void (*find_mptable)(void);
 	void (*get_smp_config)(unsigned int early);
+	void (*early_parse_smp_cfg)(void);
+	void (*parse_smp_cfg)(void);
 };
 
 /**

[patch 11/53] x86/mpparse: Prepare for callback separation

[Thomas Gleixner]

In preparation of splitting the get_smp_config() callback, rename
default_get_smp_config() to mpparse_get_smp_config() and provide an early
and late wrapper.

Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
---
 arch/x86/include/asm/mpspec.h |   12 ++++++++----
 arch/x86/kernel/mpparse.c     |   12 +++++++++++-
 arch/x86/kernel/x86_init.c    |    2 +-
 3 files changed, 20 insertions(+), 6 deletions(-)

--- a/arch/x86/include/asm/mpspec.h
+++ b/arch/x86/include/asm/mpspec.h
@@ -60,12 +60,16 @@ static inline void early_get_smp_config(
 extern void e820__memblock_alloc_reserved_mpc_new(void);
 extern int enable_update_mptable;
 extern void mpparse_find_mptable(void);
-extern void default_get_smp_config(unsigned int early);
+extern void mpparse_parse_early_smp_config(void);
+extern void mpparse_parse_smp_config(void);
+extern void mpparse_get_smp_config(unsigned int early);
 #else
 static inline void e820__memblock_alloc_reserved_mpc_new(void) { }
-#define enable_update_mptable	(0)
-#define mpparse_find_mptable	x86_init_noop
-#define default_get_smp_config	x86_init_uint_noop
+#define enable_update_mptable		(0)
+#define mpparse_find_mptable		x86_init_noop
+#define mpparse_parse_early_smp_config	x86_init_noop
+#define mpparse_parse_smp_config	x86_init_noop
+#define mpparse_get_smp_config		x86_init_uint_noop
 #endif
 
 int generic_processor_info(int apicid);
--- a/arch/x86/kernel/mpparse.c
+++ b/arch/x86/kernel/mpparse.c
@@ -476,7 +476,7 @@ static int __init check_physptr(struct m
 /*
  * Scan the memory blocks for an SMP configuration block.
  */
-void __init default_get_smp_config(unsigned int early)
+void __init mpparse_get_smp_config(unsigned int early)
 {
 	struct mpf_intel *mpf;
 
@@ -541,6 +541,16 @@ void __init default_get_smp_config(unsig
 	early_memunmap(mpf, sizeof(*mpf));
 }
 
+void __init mpparse_parse_early_smp_config(void)
+{
+	mpparse_get_smp_config(true);
+}
+
+void __init mpparse_parse_smp_config(void)
+{
+	mpparse_get_smp_config(false);
+}
+
 static void __init smp_reserve_memory(struct mpf_intel *mpf)
 {
 	memblock_reserve(mpf->physptr, get_mpc_size(mpf->physptr));
--- a/arch/x86/kernel/x86_init.c
+++ b/arch/x86/kernel/x86_init.c
@@ -71,7 +71,7 @@ struct x86_init_ops x86_init __initdata
 	.mpparse = {
 		.setup_ioapic_ids	= x86_init_noop,
 		.find_mptable		= mpparse_find_mptable,
-		.get_smp_config		= default_get_smp_config,
+		.get_smp_config		= mpparse_get_smp_config,
 	},
 
 	.irqs = {

[patch 12/53] x86/dtb: Rename x86_dtb_init()

[Thomas Gleixner]

x86_dtb_init() is a misnomer and it really should be used as a SMP
configuration parser which is selected by the platform via
x86_init::mpparse:parse_smp_config().

Rename it to x86_dtb_parse_smp_config() in preparation for that.

Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
---
 arch/x86/include/asm/prom.h  |    4 ++--
 arch/x86/kernel/devicetree.c |    2 +-
 arch/x86/kernel/setup.c      |    2 +-
 3 files changed, 4 insertions(+), 4 deletions(-)

--- a/arch/x86/include/asm/prom.h
+++ b/arch/x86/include/asm/prom.h
@@ -23,11 +23,11 @@ extern int of_ioapic;
 extern u64 initial_dtb;
 extern void add_dtb(u64 data);
 void x86_of_pci_init(void);
-void x86_dtb_init(void);
+void x86_dtb_parse_smp_config(void);
 #else
 static inline void add_dtb(u64 data) { }
 static inline void x86_of_pci_init(void) { }
-static inline void x86_dtb_init(void) { }
+static inline void x86_dtb_parse_smp_config(void) { }
 #define of_ioapic 0
 #endif
 
--- a/arch/x86/kernel/devicetree.c
+++ b/arch/x86/kernel/devicetree.c
@@ -304,7 +304,7 @@ static void __init x86_flattree_get_conf
 static inline void x86_flattree_get_config(void) { }
 #endif
 
-void __init x86_dtb_init(void)
+void __init x86_dtb_parse_smp_config(void)
 {
 	x86_flattree_get_config();
 
--- a/arch/x86/kernel/setup.c
+++ b/arch/x86/kernel/setup.c
@@ -1260,7 +1260,7 @@ void __init setup_arch(char **cmdline_p)
 	 * Read APIC and some other early information from ACPI tables.
 	 */
 	acpi_boot_init();
-	x86_dtb_init();
+	x86_dtb_parse_smp_config();
 
 	/*
 	 * get boot-time SMP configuration:

[patch 13/53] x86/platform/ce4100: Prepare for separate mpparse callbacks

[Thomas Gleixner]

Select x86_dtb_parse_smp_config() as SMP configuration parser in
preparation of splitting up the get_smp_config() callback.

Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
---
 arch/x86/platform/ce4100/ce4100.c |   14 ++++++++------
 1 file changed, 8 insertions(+), 6 deletions(-)

--- a/arch/x86/platform/ce4100/ce4100.c
+++ b/arch/x86/platform/ce4100/ce4100.c
@@ -135,12 +135,14 @@ static void sdv_pci_init(void)
  */
 void __init x86_ce4100_early_setup(void)
 {
-	x86_init.oem.arch_setup = sdv_arch_setup;
-	x86_init.resources.probe_roms = x86_init_noop;
-	x86_init.mpparse.get_smp_config = x86_init_uint_noop;
-	x86_init.mpparse.find_mptable = x86_init_noop;
-	x86_init.pci.init = ce4100_pci_init;
-	x86_init.pci.init_irq = sdv_pci_init;
+	x86_init.oem.arch_setup			= sdv_arch_setup;
+	x86_init.resources.probe_roms		= x86_init_noop;
+	x86_init.mpparse.find_mptable		= x86_init_noop;
+	x86_init.mpparse.early_parse_smp_cfg	= x86_init_noop;
+	x86_init.mpparse.parse_smp_cfg		= x86_dtb_parse_smp_config;
+	x86_init.mpparse.get_smp_config		= x86_init_uint_noop;
+	x86_init.pci.init			= ce4100_pci_init;
+	x86_init.pci.init_irq			= sdv_pci_init;
 
 	/*
 	 * By default, the reboot method is ACPI which is supported by the

[patch 14/53] x86/platform/intel-mid: Prepare for separate mpparse callbacks

[Thomas Gleixner]

Initialize the split SMP configuration callbacks with NOOPs as MID is
strictly ACPI only.

Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
---
 arch/x86/platform/intel-mid/intel-mid.c |    6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

--- a/arch/x86/platform/intel-mid/intel-mid.c
+++ b/arch/x86/platform/intel-mid/intel-mid.c
@@ -118,7 +118,9 @@ void __init x86_intel_mid_early_setup(vo
 	machine_ops.emergency_restart  = intel_mid_reboot;
 
 	/* Avoid searching for BIOS MP tables */
-	x86_init.mpparse.find_mptable = x86_init_noop;
-	x86_init.mpparse.get_smp_config = x86_init_uint_noop;
+	x86_init.mpparse.find_mptable		= x86_init_noop;
+	x86_init.mpparse.early_parse_smp_cfg	= x86_init_noop;
+	x86_init.mpparse.parse_smp_cfg		= x86_init_noop;
+	x86_init.mpparse.get_smp_config		= x86_init_uint_noop;
 	set_bit(MP_BUS_ISA, mp_bus_not_pci);
 }

[patch 15/53] x86/jailhouse: Prepare for separate mpparse callbacks

[Thomas Gleixner]

Provide a wrapper around the existing function and fill the new callbacks
in.

No functional change as the new callbacks are not yet operational.

Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
---
 arch/x86/kernel/jailhouse.c |   34 +++++++++++++++++++++-------------
 1 file changed, 21 insertions(+), 13 deletions(-)

--- a/arch/x86/kernel/jailhouse.c
+++ b/arch/x86/kernel/jailhouse.c
@@ -118,6 +118,11 @@ static void __init jailhouse_get_smp_con
 	}
 }
 
+static void __init jailhouse_parse_smp_config(void)
+{
+	jailhouse_get_smp_config(false);
+}
+
 static void jailhouse_no_restart(void)
 {
 	pr_notice("Jailhouse: Restart not supported, halting\n");
@@ -201,21 +206,24 @@ static void __init jailhouse_init_platfo
 	struct setup_data header;
 	void *mapping;
 
-	x86_init.irqs.pre_vector_init	= x86_init_noop;
-	x86_init.timers.timer_init	= jailhouse_timer_init;
-	x86_init.mpparse.get_smp_config	= jailhouse_get_smp_config;
-	x86_init.pci.arch_init		= jailhouse_pci_arch_init;
-
-	x86_platform.calibrate_cpu	= jailhouse_get_tsc;
-	x86_platform.calibrate_tsc	= jailhouse_get_tsc;
-	x86_platform.get_wallclock	= jailhouse_get_wallclock;
-	x86_platform.legacy.rtc		= 0;
-	x86_platform.legacy.warm_reset	= 0;
-	x86_platform.legacy.i8042	= X86_LEGACY_I8042_PLATFORM_ABSENT;
+	x86_init.irqs.pre_vector_init		= x86_init_noop;
+	x86_init.timers.timer_init		= jailhouse_timer_init;
+	x86_init.mpparse.find_mptable		= x86_init_noop;
+	x86_init.mpparse.early_parse_smp_cfg	= x86_init_noop;
+	x86_init.mpparse.parse_smp_cfg		= jailhouse_parse_smp_config;
+	x86_init.mpparse.get_smp_config		= jailhouse_get_smp_config;
+	x86_init.pci.arch_init			= jailhouse_pci_arch_init;
+
+	x86_platform.calibrate_cpu		= jailhouse_get_tsc;
+	x86_platform.calibrate_tsc		= jailhouse_get_tsc;
+	x86_platform.get_wallclock		= jailhouse_get_wallclock;
+	x86_platform.legacy.rtc			= 0;
+	x86_platform.legacy.warm_reset		= 0;
+	x86_platform.legacy.i8042		= X86_LEGACY_I8042_PLATFORM_ABSENT;
 
-	legacy_pic			= &null_legacy_pic;
+	legacy_pic				= &null_legacy_pic;
 
-	machine_ops.emergency_restart	= jailhouse_no_restart;
+	machine_ops.emergency_restart		= jailhouse_no_restart;
 
 	while (pa_data) {
 		mapping = early_memremap(pa_data, sizeof(header));

[patch 16/53] x86/xen/smp_pv: Prepare for separate mpparse callbacks

[Thomas Gleixner]

Provide a wrapper around the existing function and fill the new callbacks
in.

No functional change as the new callbacks are not yet operational.

Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
---
 arch/x86/xen/smp_pv.c |   11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)

--- a/arch/x86/xen/smp_pv.c
+++ b/arch/x86/xen/smp_pv.c
@@ -185,6 +185,11 @@ static void __init _get_smp_config(unsig
 	smp_found_config = 1;
 }
 
+static void __init xen_pv_smp_config(void)
+{
+	_get_smp_config(false);
+}
+
 static void __init xen_pv_smp_prepare_boot_cpu(void)
 {
 	BUG_ON(smp_processor_id() != 0);
@@ -455,6 +460,8 @@ void __init xen_smp_init(void)
 	smp_ops = xen_smp_ops;
 
 	/* Avoid searching for BIOS MP tables */
-	x86_init.mpparse.find_mptable = x86_init_noop;
-	x86_init.mpparse.get_smp_config = _get_smp_config;
+	x86_init.mpparse.find_mptable		= x86_init_noop;
+	x86_init.mpparse.early_parse_smp_cfg	= x86_init_noop;
+	x86_init.mpparse.parse_smp_cfg		= xen_pv_smp_config;
+	x86_init.mpparse.get_smp_config		= _get_smp_config;
 }

[patch 17/53] x86/mpparse: Switch to new init callbacks

[Thomas Gleixner]

Now that all platforms have the new split SMP configuration callbacks set
up, flip the switch and remove the old callback pointer and mop up the
platform code.

Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
---
 arch/x86/include/asm/mpspec.h           |    9 +--------
 arch/x86/include/asm/x86_init.h         |    2 --
 arch/x86/kernel/jailhouse.c             |    8 +-------
 arch/x86/kernel/mpparse.c               |    2 +-
 arch/x86/kernel/setup.c                 |   10 +++-------
 arch/x86/kernel/x86_init.c              |    3 ++-
 arch/x86/platform/ce4100/ce4100.c       |    1 -
 arch/x86/platform/intel-mid/intel-mid.c |    1 -
 arch/x86/xen/smp_pv.c                   |   11 +----------
 9 files changed, 9 insertions(+), 38 deletions(-)

--- a/arch/x86/include/asm/mpspec.h
+++ b/arch/x86/include/asm/mpspec.h
@@ -46,14 +46,9 @@ extern int smp_found_config;
 # define smp_found_config 0
 #endif
 
-static inline void get_smp_config(void)
-{
-	x86_init.mpparse.get_smp_config(0);
-}
-
 static inline void early_get_smp_config(void)
 {
-	x86_init.mpparse.get_smp_config(1);
+	x86_init.mpparse.early_parse_smp_cfg();
 }
 
 #ifdef CONFIG_X86_MPPARSE
@@ -62,14 +57,12 @@ extern int enable_update_mptable;
 extern void mpparse_find_mptable(void);
 extern void mpparse_parse_early_smp_config(void);
 extern void mpparse_parse_smp_config(void);
-extern void mpparse_get_smp_config(unsigned int early);
 #else
 static inline void e820__memblock_alloc_reserved_mpc_new(void) { }
 #define enable_update_mptable		(0)
 #define mpparse_find_mptable		x86_init_noop
 #define mpparse_parse_early_smp_config	x86_init_noop
 #define mpparse_parse_smp_config	x86_init_noop
-#define mpparse_get_smp_config		x86_init_uint_noop
 #endif
 
 int generic_processor_info(int apicid);
--- a/arch/x86/include/asm/x86_init.h
+++ b/arch/x86/include/asm/x86_init.h
@@ -16,14 +16,12 @@ struct irq_domain;
  * struct x86_init_mpparse - platform specific mpparse ops
  * @setup_ioapic_ids:		platform specific ioapic id override
  * @find_mptable:		Find MPTABLE early to reserve the memory region
- * @get_smp_config:		get the smp configuration
  * @early_parse_smp_cfg:	Parse the SMP configuration data early before initmem_init()
  * @parse_smp_cfg:		Parse the SMP configuration data
  */
 struct x86_init_mpparse {
 	void (*setup_ioapic_ids)(void);
 	void (*find_mptable)(void);
-	void (*get_smp_config)(unsigned int early);
 	void (*early_parse_smp_cfg)(void);
 	void (*parse_smp_cfg)(void);
 };
--- a/arch/x86/kernel/jailhouse.c
+++ b/arch/x86/kernel/jailhouse.c
@@ -89,7 +89,7 @@ static void __init jailhouse_x2apic_init
 #endif
 }
 
-static void __init jailhouse_get_smp_config(unsigned int early)
+static void __init jailhouse_parse_smp_config(void)
 {
 	struct ioapic_domain_cfg ioapic_cfg = {
 		.type = IOAPIC_DOMAIN_STRICT,
@@ -118,11 +118,6 @@ static void __init jailhouse_get_smp_con
 	}
 }
 
-static void __init jailhouse_parse_smp_config(void)
-{
-	jailhouse_get_smp_config(false);
-}
-
 static void jailhouse_no_restart(void)
 {
 	pr_notice("Jailhouse: Restart not supported, halting\n");
@@ -211,7 +206,6 @@ static void __init jailhouse_init_platfo
 	x86_init.mpparse.find_mptable		= x86_init_noop;
 	x86_init.mpparse.early_parse_smp_cfg	= x86_init_noop;
 	x86_init.mpparse.parse_smp_cfg		= jailhouse_parse_smp_config;
-	x86_init.mpparse.get_smp_config		= jailhouse_get_smp_config;
 	x86_init.pci.arch_init			= jailhouse_pci_arch_init;
 
 	x86_platform.calibrate_cpu		= jailhouse_get_tsc;
--- a/arch/x86/kernel/mpparse.c
+++ b/arch/x86/kernel/mpparse.c
@@ -476,7 +476,7 @@ static int __init check_physptr(struct m
 /*
  * Scan the memory blocks for an SMP configuration block.
  */
-void __init mpparse_get_smp_config(unsigned int early)
+static __init void mpparse_get_smp_config(unsigned int early)
 {
 	struct mpf_intel *mpf;
 
--- a/arch/x86/kernel/setup.c
+++ b/arch/x86/kernel/setup.c
@@ -1257,15 +1257,11 @@ void __init setup_arch(char **cmdline_p)
 	early_quirks();
 
 	/*
-	 * Read APIC and some other early information from ACPI tables.
+	 * Parse SMP configuration. Try ACPI first and then the platform
+	 * specific parser.
 	 */
 	acpi_boot_init();
-	x86_dtb_parse_smp_config();
-
-	/*
-	 * get boot-time SMP configuration:
-	 */
-	get_smp_config();
+	x86_init.mpparse.parse_smp_cfg();
 
 	/*
 	 * Systems w/o ACPI and mptables might not have it mapped the local
--- a/arch/x86/kernel/x86_init.c
+++ b/arch/x86/kernel/x86_init.c
@@ -71,7 +71,8 @@ struct x86_init_ops x86_init __initdata
 	.mpparse = {
 		.setup_ioapic_ids	= x86_init_noop,
 		.find_mptable		= mpparse_find_mptable,
-		.get_smp_config		= mpparse_get_smp_config,
+		.early_parse_smp_cfg	= mpparse_parse_early_smp_config,
+		.parse_smp_cfg		= mpparse_parse_smp_config,
 	},
 
 	.irqs = {
--- a/arch/x86/platform/ce4100/ce4100.c
+++ b/arch/x86/platform/ce4100/ce4100.c
@@ -140,7 +140,6 @@ void __init x86_ce4100_early_setup(void)
 	x86_init.mpparse.find_mptable		= x86_init_noop;
 	x86_init.mpparse.early_parse_smp_cfg	= x86_init_noop;
 	x86_init.mpparse.parse_smp_cfg		= x86_dtb_parse_smp_config;
-	x86_init.mpparse.get_smp_config		= x86_init_uint_noop;
 	x86_init.pci.init			= ce4100_pci_init;
 	x86_init.pci.init_irq			= sdv_pci_init;
 
--- a/arch/x86/platform/intel-mid/intel-mid.c
+++ b/arch/x86/platform/intel-mid/intel-mid.c
@@ -121,6 +121,5 @@ void __init x86_intel_mid_early_setup(vo
 	x86_init.mpparse.find_mptable		= x86_init_noop;
 	x86_init.mpparse.early_parse_smp_cfg	= x86_init_noop;
 	x86_init.mpparse.parse_smp_cfg		= x86_init_noop;
-	x86_init.mpparse.get_smp_config		= x86_init_uint_noop;
 	set_bit(MP_BUS_ISA, mp_bus_not_pci);
 }
--- a/arch/x86/xen/smp_pv.c
+++ b/arch/x86/xen/smp_pv.c
@@ -148,14 +148,11 @@ int xen_smp_intr_init_pv(unsigned int cp
 	return rc;
 }
 
-static void __init _get_smp_config(unsigned int early)
+static void __init xen_pv_smp_config(void)
 {
 	int i, rc;
 	unsigned int subtract = 0;
 
-	if (early)
-		return;
-
 	num_processors = 0;
 	disabled_cpus = 0;
 	for (i = 0; i < nr_cpu_ids; i++) {
@@ -185,11 +182,6 @@ static void __init _get_smp_config(unsig
 	smp_found_config = 1;
 }
 
-static void __init xen_pv_smp_config(void)
-{
-	_get_smp_config(false);
-}
-
 static void __init xen_pv_smp_prepare_boot_cpu(void)
 {
 	BUG_ON(smp_processor_id() != 0);
@@ -463,5 +455,4 @@ void __init xen_smp_init(void)
 	x86_init.mpparse.find_mptable		= x86_init_noop;
 	x86_init.mpparse.early_parse_smp_cfg	= x86_init_noop;
 	x86_init.mpparse.parse_smp_cfg		= xen_pv_smp_config;
-	x86_init.mpparse.get_smp_config		= _get_smp_config;
 }

[patch 18/53] x86/mm/numa: Move early mptable evaluation into common code

[Thomas Gleixner]

There is no reason to have the early mptable evaluation conditionally
invoked only from the AMD numa topology code.

Make it explicit and invoke it from setup_arch() right after the
corresponding ACPI init call. Remove the pointless wrapper and invoke
x86_init::mpparse::early_parse_smp_config() directly.

Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
---
 arch/x86/include/asm/mpspec.h |    5 -----
 arch/x86/kernel/setup.c       |    2 ++
 arch/x86/mm/amdtopology.c     |    7 -------
 3 files changed, 2 insertions(+), 12 deletions(-)

--- a/arch/x86/include/asm/mpspec.h
+++ b/arch/x86/include/asm/mpspec.h
@@ -46,11 +46,6 @@ extern int smp_found_config;
 # define smp_found_config 0
 #endif
 
-static inline void early_get_smp_config(void)
-{
-	x86_init.mpparse.early_parse_smp_cfg();
-}
-
 #ifdef CONFIG_X86_MPPARSE
 extern void e820__memblock_alloc_reserved_mpc_new(void);
 extern int enable_update_mptable;
--- a/arch/x86/kernel/setup.c
+++ b/arch/x86/kernel/setup.c
@@ -1217,7 +1217,9 @@ void __init setup_arch(char **cmdline_p)
 
 	early_platform_quirks();
 
+	/* Some platforms need the APIC registered for NUMA configuration */
 	early_acpi_boot_init();
+	x86_init.mpparse.early_parse_smp_cfg();
 
 	initmem_init();
 	dma_contiguous_reserve(max_pfn_mapped << PAGE_SHIFT);
--- a/arch/x86/mm/amdtopology.c
+++ b/arch/x86/mm/amdtopology.c
@@ -161,13 +161,6 @@ int __init amd_numa_init(void)
 	 */
 	cores = topology_get_domain_size(TOPO_CORE_DOMAIN);
 
-	/*
-	 * Scan MPTABLE to map the local APIC and ensure that the boot CPU
-	 * APIC ID is valid. This is required because on pre ACPI/SRAT
-	 * systems IO-APICs are mapped before the boot CPU.
-	 */
-	early_get_smp_config();
-
 	apicid = boot_cpu_physical_apicid;
 	if (apicid > 0)
 		pr_info("BSP APIC ID: %02x\n", apicid);

[patch 19/53] x86/mpparse: Remove the physid_t bitmap wrapper

[Thomas Gleixner]

physid_t is a wrapper around bitmap. Just remove the onion layer and use
bitmap functionality directly.

Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
---
 arch/x86/include/asm/mpspec.h      |   32 +++++++++-----------------------
 arch/x86/kernel/apic/apic.c        |   11 +++++------
 arch/x86/kernel/apic/apic_common.c |   12 +-----------
 arch/x86/kernel/apic/io_apic.c     |   24 ++++++++++++------------
 arch/x86/kernel/apic/local.h       |    1 -
 arch/x86/kernel/smpboot.c          |    8 +++-----
 6 files changed, 30 insertions(+), 58 deletions(-)

--- a/arch/x86/include/asm/mpspec.h
+++ b/arch/x86/include/asm/mpspec.h
@@ -2,6 +2,7 @@
 #ifndef _ASM_X86_MPSPEC_H
 #define _ASM_X86_MPSPEC_H
 
+#include <linux/types.h>
 
 #include <asm/mpspec_def.h>
 #include <asm/x86_init.h>
@@ -62,32 +63,17 @@ static inline void e820__memblock_alloc_
 
 int generic_processor_info(int apicid);
 
-#define PHYSID_ARRAY_SIZE	BITS_TO_LONGS(MAX_LOCAL_APIC)
+extern DECLARE_BITMAP(phys_cpu_present_map, MAX_LOCAL_APIC);
 
-struct physid_mask {
-	unsigned long mask[PHYSID_ARRAY_SIZE];
-};
-
-typedef struct physid_mask physid_mask_t;
-
-#define physid_set(physid, map)			set_bit(physid, (map).mask)
-#define physid_isset(physid, map)		test_bit(physid, (map).mask)
-
-#define physids_clear(map)					\
-	bitmap_zero((map).mask, MAX_LOCAL_APIC)
-
-#define physids_empty(map)					\
-	bitmap_empty((map).mask, MAX_LOCAL_APIC)
-
-static inline void physid_set_mask_of_physid(int physid, physid_mask_t *map)
+static inline void reset_phys_cpu_present_map(u32 apicid)
 {
-	physids_clear(*map);
-	physid_set(physid, *map);
+	bitmap_zero(phys_cpu_present_map, MAX_LOCAL_APIC);
+	set_bit(apicid, phys_cpu_present_map);
 }
 
-#define PHYSID_MASK_ALL		{ {[0 ... PHYSID_ARRAY_SIZE-1] = ~0UL} }
-#define PHYSID_MASK_NONE	{ {[0 ... PHYSID_ARRAY_SIZE-1] = 0UL} }
-
-extern physid_mask_t phys_cpu_present_map;
+static inline void copy_phys_cpu_present_map(unsigned long *dst)
+{
+	bitmap_copy(dst, phys_cpu_present_map, MAX_LOCAL_APIC);
+}
 
 #endif /* _ASM_X86_MPSPEC_H */
--- a/arch/x86/kernel/apic/apic.c
+++ b/arch/x86/kernel/apic/apic.c
@@ -22,6 +22,7 @@
 #include <linux/clockchips.h>
 #include <linux/interrupt.h>
 #include <linux/memblock.h>
+#include <linux/bitmap.h>
 #include <linux/ftrace.h>
 #include <linux/ioport.h>
 #include <linux/export.h>
@@ -77,10 +78,8 @@ EXPORT_SYMBOL_GPL(boot_cpu_physical_apic
 
 u8 boot_cpu_apic_version __ro_after_init;
 
-/*
- * Bitmask of physically existing CPUs:
- */
-physid_mask_t phys_cpu_present_map;
+/* Bitmap of physically present CPUs. */
+DECLARE_BITMAP(phys_cpu_present_map, MAX_LOCAL_APIC);
 
 /*
  * Processor to be disabled specified by kernel parameter
@@ -2387,7 +2386,7 @@ static void cpu_update_apic(int cpu, u32
 	early_per_cpu(x86_cpu_to_apicid, cpu) = apicid;
 #endif
 	set_cpu_possible(cpu, true);
-	physid_set(apicid, phys_cpu_present_map);
+	set_bit(apicid, phys_cpu_present_map);
 	set_cpu_present(cpu, true);
 	num_processors++;
 
@@ -2489,7 +2488,7 @@ static void __init apic_bsp_up_setup(voi
 #ifdef CONFIG_X86_64
 	apic_write(APIC_ID, apic->set_apic_id(boot_cpu_physical_apicid));
 #endif
-	physid_set_mask_of_physid(boot_cpu_physical_apicid, &phys_cpu_present_map);
+	reset_phys_cpu_present_map(boot_cpu_physical_apicid);
 }
 
 /**
--- a/arch/x86/kernel/apic/apic_common.c
+++ b/arch/x86/kernel/apic/apic_common.c
@@ -18,16 +18,6 @@ u32 apic_flat_calc_apicid(unsigned int c
 	return 1U << cpu;
 }
 
-bool default_check_apicid_used(physid_mask_t *map, u32 apicid)
-{
-	return physid_isset(apicid, *map);
-}
-
-void default_ioapic_phys_id_map(physid_mask_t *phys_map, physid_mask_t *retmap)
-{
-	*retmap = *phys_map;
-}
-
 u32 default_cpu_present_to_apicid(int mps_cpu)
 {
 	if (mps_cpu < nr_cpu_ids && cpu_present(mps_cpu))
@@ -39,7 +29,7 @@ EXPORT_SYMBOL_GPL(default_cpu_present_to
 
 bool default_apic_id_registered(void)
 {
-	return physid_isset(read_apic_id(), phys_cpu_present_map);
+	return test_bit(read_apic_id(), phys_cpu_present_map);
 }
 
 /*
--- a/arch/x86/kernel/apic/io_apic.c
+++ b/arch/x86/kernel/apic/io_apic.c
@@ -1460,7 +1460,7 @@ void restore_boot_irq_mode(void)
  */
 static void __init setup_ioapic_ids_from_mpc_nocheck(void)
 {
-	physid_mask_t phys_id_present_map;
+	DECLARE_BITMAP(phys_id_present_map, MAX_LOCAL_APIC);
 	const u32 broadcast_id = 0xF;
 	union IO_APIC_reg_00 reg_00;
 	unsigned char old_id;
@@ -1471,7 +1471,7 @@ static void __init setup_ioapic_ids_from
 	 * This is broken; anything with a real cpu count has to
 	 * circumvent this idiocy regardless.
 	 */
-	phys_id_present_map = phys_cpu_present_map;
+	copy_phys_cpu_present_map(phys_id_present_map);
 
 	/*
 	 * Set the IOAPIC ID to the value stored in the MPC table.
@@ -1496,21 +1496,21 @@ static void __init setup_ioapic_ids_from
 		 * system must have a unique ID or we get lots of nice
 		 * 'stuck on smp_invalidate_needed IPI wait' messages.
 		 */
-		if (physid_isset(mpc_ioapic_id(ioapic_idx), phys_id_present_map)) {
+		if (test_bit(mpc_ioapic_id(ioapic_idx), phys_id_present_map)) {
 			pr_err("BIOS bug, IO-APIC#%d ID %d is already used!...\n",
 				ioapic_idx, mpc_ioapic_id(ioapic_idx));
 			for (i = 0; i < broadcast_id; i++)
-				if (!physid_isset(i, phys_id_present_map))
+				if (!test_bit(i, phys_id_present_map))
 					break;
 			if (i >= broadcast_id)
 				panic("Max APIC ID exceeded!\n");
 			pr_err("... fixing up to %d. (tell your hw vendor)\n", i);
-			physid_set(i, phys_id_present_map);
+			set_bit(i, phys_id_present_map);
 			ioapics[ioapic_idx].mp_config.apicid = i;
 		} else {
 			apic_printk(APIC_VERBOSE, "Setting %d in the phys_id_present_map\n",
 				    mpc_ioapic_id(ioapic_idx));
-			physid_set(mpc_ioapic_id(ioapic_idx), phys_id_present_map);
+			set_bit(mpc_ioapic_id(ioapic_idx), phys_id_present_map);
 		}
 
 		/*
@@ -2491,15 +2491,15 @@ unsigned int arch_dynirq_lower_bound(uns
 #ifdef CONFIG_X86_32
 static int io_apic_get_unique_id(int ioapic, int apic_id)
 {
-	static physid_mask_t apic_id_map = PHYSID_MASK_NONE;
+	static DECLARE_BITMAP(apic_id_map, MAX_LOCAL_APIC);
 	const u32 broadcast_id = 0xF;
 	union IO_APIC_reg_00 reg_00;
 	unsigned long flags;
 	int i = 0;
 
 	/* Initialize the ID map */
-	if (physids_empty(apic_id_map))
-		apic_id_map = phys_cpu_present_map;
+	if (bitmap_empty(apic_id_map, MAX_LOCAL_APIC))
+		copy_phys_cpu_present_map(apic_id_map);
 
 	raw_spin_lock_irqsave(&ioapic_lock, flags);
 	reg_00.raw = io_apic_read(ioapic, 0);
@@ -2512,9 +2512,9 @@ static int io_apic_get_unique_id(int ioa
 	}
 
 	/* Every APIC in a system must have a unique ID */
-	if (physid_isset(apic_id, apic_id_map)) {
+	if (test_bit(apic_id, apic_id_map)) {
 		for (i = 0; i < broadcast_id; i++) {
-			if (!physid_isset(i, apic_id_map))
+			if (!test_bit(i, apic_id_map))
 				break;
 		}
 
@@ -2525,7 +2525,7 @@ static int io_apic_get_unique_id(int ioa
 		apic_id = i;
 	}
 
-	physid_set(apic_id, apic_id_map);
+	set_bit(apic_id, apic_id_map);
 
 	if (reg_00.bits.ID != apic_id) {
 		reg_00.bits.ID = apic_id;
--- a/arch/x86/kernel/apic/local.h
+++ b/arch/x86/kernel/apic/local.h
@@ -63,7 +63,6 @@ void default_send_IPI_all(int vector);
 void default_send_IPI_self(int vector);
 
 bool default_apic_id_registered(void);
-bool default_check_apicid_used(physid_mask_t *map, u32 apicid);
 
 #ifdef CONFIG_X86_32
 void default_send_IPI_mask_sequence_logical(const struct cpumask *mask, int vector);
--- a/arch/x86/kernel/smpboot.c
+++ b/arch/x86/kernel/smpboot.c
@@ -1081,7 +1081,7 @@ int native_kick_ap(unsigned int cpu, str
 
 	pr_debug("++++++++++++++++++++=_---CPU UP  %u\n", cpu);
 
-	if (apicid == BAD_APICID || !physid_isset(apicid, phys_cpu_present_map) ||
+	if (apicid == BAD_APICID || !test_bit(apicid, phys_cpu_present_map) ||
 	    !apic_id_valid(apicid)) {
 		pr_err("%s: bad cpu %d\n", __func__, cpu);
 		return -EINVAL;
@@ -1156,10 +1156,8 @@ static __init void disable_smp(void)
 	init_cpu_present(cpumask_of(0));
 	init_cpu_possible(cpumask_of(0));
 
-	if (smp_found_config)
-		physid_set_mask_of_physid(boot_cpu_physical_apicid, &phys_cpu_present_map);
-	else
-		physid_set_mask_of_physid(0, &phys_cpu_present_map);
+	reset_phys_cpu_present_map(smp_found_config ? boot_cpu_physical_apicid : 0);
+
 	cpumask_set_cpu(0, topology_sibling_cpumask(0));
 	cpumask_set_cpu(0, topology_core_cpumask(0));
 	cpumask_set_cpu(0, topology_die_cpumask(0));

[patch 20/53] x86/apic: Remove the pointless writeback of boot_cpu_physical_apicid

[Thomas Gleixner]

There is absolutely no point to write the APIC ID which was read from the
local APIC earlier, back into the local APIC for the 64-bit UP case.

Remove that along with the apic callback which is solely there for this
pointless exercise.

Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
---
 arch/x86/include/asm/apic.h           |    1 -
 arch/x86/kernel/apic/apic.c           |    3 ---
 arch/x86/kernel/apic/apic_flat_64.c   |    7 -------
 arch/x86/kernel/apic/apic_numachip.c  |   12 ------------
 arch/x86/kernel/apic/bigsmp_32.c      |    1 -
 arch/x86/kernel/apic/local.h          |    1 -
 arch/x86/kernel/apic/x2apic_cluster.c |    1 -
 arch/x86/kernel/apic/x2apic_phys.c    |    6 ------
 arch/x86/kernel/apic/x2apic_uv_x.c    |    6 ------
 arch/x86/xen/apic.c                   |    7 -------
 10 files changed, 45 deletions(-)

--- a/arch/x86/include/asm/apic.h
+++ b/arch/x86/include/asm/apic.h
@@ -296,7 +296,6 @@ struct apic {
 	u32	(*cpu_present_to_apicid)(int mps_cpu);
 
 	u32	(*get_apic_id)(u32 id);
-	u32	(*set_apic_id)(u32 apicid);
 
 	/* wakeup_secondary_cpu */
 	int	(*wakeup_secondary_cpu)(u32 apicid, unsigned long start_eip);
--- a/arch/x86/kernel/apic/apic.c
+++ b/arch/x86/kernel/apic/apic.c
@@ -2474,9 +2474,6 @@ EXPORT_SYMBOL_GPL(x86_msi_msg_get_destid
 
 static void __init apic_bsp_up_setup(void)
 {
-#ifdef CONFIG_X86_64
-	apic_write(APIC_ID, apic->set_apic_id(boot_cpu_physical_apicid));
-#endif
 	reset_phys_cpu_present_map(boot_cpu_physical_apicid);
 }
 
--- a/arch/x86/kernel/apic/apic_flat_64.c
+++ b/arch/x86/kernel/apic/apic_flat_64.c
@@ -61,11 +61,6 @@ static u32 flat_get_apic_id(u32 x)
 	return (x >> 24) & 0xFF;
 }
 
-static u32 set_apic_id(u32 id)
-{
-	return (id & 0xFF) << 24;
-}
-
 static int flat_probe(void)
 {
 	return 1;
@@ -87,7 +82,6 @@ static struct apic apic_flat __ro_after_
 
 	.max_apic_id			= 0xFE,
 	.get_apic_id			= flat_get_apic_id,
-	.set_apic_id			= set_apic_id,
 
 	.calc_dest_apicid		= apic_flat_calc_apicid,
 
@@ -156,7 +150,6 @@ static struct apic apic_physflat __ro_af
 
 	.max_apic_id			= 0xFE,
 	.get_apic_id			= flat_get_apic_id,
-	.set_apic_id			= set_apic_id,
 
 	.calc_dest_apicid		= apic_default_calc_apicid,
 
--- a/arch/x86/kernel/apic/apic_numachip.c
+++ b/arch/x86/kernel/apic/apic_numachip.c
@@ -38,11 +38,6 @@ static u32 numachip1_get_apic_id(u32 x)
 	return id;
 }
 
-static u32 numachip1_set_apic_id(u32 id)
-{
-	return (id & 0xff) << 24;
-}
-
 static u32 numachip2_get_apic_id(u32 x)
 {
 	u64 mcfg;
@@ -51,11 +46,6 @@ static u32 numachip2_get_apic_id(u32 x)
 	return ((mcfg >> (28 - 8)) & 0xfff00) | (x >> 24);
 }
 
-static u32 numachip2_set_apic_id(u32 id)
-{
-	return id << 24;
-}
-
 static void numachip1_apic_icr_write(int apicid, unsigned int val)
 {
 	write_lcsr(CSR_G3_EXT_IRQ_GEN, (apicid << 16) | val);
@@ -226,7 +216,6 @@ static const struct apic apic_numachip1
 
 	.max_apic_id			= UINT_MAX,
 	.get_apic_id			= numachip1_get_apic_id,
-	.set_apic_id			= numachip1_set_apic_id,
 
 	.calc_dest_apicid		= apic_default_calc_apicid,
 
@@ -262,7 +251,6 @@ static const struct apic apic_numachip2
 
 	.max_apic_id			= UINT_MAX,
 	.get_apic_id			= numachip2_get_apic_id,
-	.set_apic_id			= numachip2_set_apic_id,
 
 	.calc_dest_apicid		= apic_default_calc_apicid,
 
--- a/arch/x86/kernel/apic/bigsmp_32.c
+++ b/arch/x86/kernel/apic/bigsmp_32.c
@@ -73,7 +73,6 @@ static struct apic apic_bigsmp __ro_afte
 
 	.max_apic_id			= 0xFE,
 	.get_apic_id			= bigsmp_get_apic_id,
-	.set_apic_id			= NULL,
 
 	.calc_dest_apicid		= apic_default_calc_apicid,
 
--- a/arch/x86/kernel/apic/local.h
+++ b/arch/x86/kernel/apic/local.h
@@ -16,7 +16,6 @@
 /* X2APIC */
 void __x2apic_send_IPI_dest(unsigned int apicid, int vector, unsigned int dest);
 u32 x2apic_get_apic_id(u32 id);
-u32 x2apic_set_apic_id(u32 id);
 
 void x2apic_send_IPI_all(int vector);
 void x2apic_send_IPI_allbutself(int vector);
--- a/arch/x86/kernel/apic/x2apic_cluster.c
+++ b/arch/x86/kernel/apic/x2apic_cluster.c
@@ -238,7 +238,6 @@ static struct apic apic_x2apic_cluster _
 	.max_apic_id			= UINT_MAX,
 	.x2apic_set_max_apicid		= true,
 	.get_apic_id			= x2apic_get_apic_id,
-	.set_apic_id			= x2apic_set_apic_id,
 
 	.calc_dest_apicid		= x2apic_calc_apicid,
 
--- a/arch/x86/kernel/apic/x2apic_phys.c
+++ b/arch/x86/kernel/apic/x2apic_phys.c
@@ -129,11 +129,6 @@ u32 x2apic_get_apic_id(u32 id)
 	return id;
 }
 
-u32 x2apic_set_apic_id(u32 id)
-{
-	return id;
-}
-
 static struct apic apic_x2apic_phys __ro_after_init = {
 
 	.name				= "physical x2apic",
@@ -150,7 +145,6 @@ static struct apic apic_x2apic_phys __ro
 	.max_apic_id			= UINT_MAX,
 	.x2apic_set_max_apicid		= true,
 	.get_apic_id			= x2apic_get_apic_id,
-	.set_apic_id			= x2apic_set_apic_id,
 
 	.calc_dest_apicid		= apic_default_calc_apicid,
 
--- a/arch/x86/kernel/apic/x2apic_uv_x.c
+++ b/arch/x86/kernel/apic/x2apic_uv_x.c
@@ -746,11 +746,6 @@ static void uv_send_IPI_all(int vector)
 	uv_send_IPI_mask(cpu_online_mask, vector);
 }
 
-static u32 set_apic_id(u32 id)
-{
-	return id;
-}
-
 static int uv_probe(void)
 {
 	return apic == &apic_x2apic_uv_x;
@@ -771,7 +766,6 @@ static struct apic apic_x2apic_uv_x __ro
 
 	.max_apic_id			= UINT_MAX,
 	.get_apic_id			= x2apic_get_apic_id,
-	.set_apic_id			= set_apic_id,
 
 	.calc_dest_apicid		= apic_default_calc_apicid,
 
--- a/arch/x86/xen/apic.c
+++ b/arch/x86/xen/apic.c
@@ -33,12 +33,6 @@ static unsigned int xen_io_apic_read(uns
 	return 0xfd;
 }
 
-static u32 xen_set_apic_id(u32 x)
-{
-	WARN_ON(1);
-	return x;
-}
-
 static u32 xen_get_apic_id(u32 x)
 {
 	return ((x)>>24) & 0xFFu;
@@ -131,7 +125,6 @@ static struct apic xen_pv_apic __ro_afte
 
 	.max_apic_id			= UINT_MAX,
 	.get_apic_id			= xen_get_apic_id,
-	.set_apic_id			= xen_set_apic_id,
 
 	.calc_dest_apicid		= apic_flat_calc_apicid,
 

[patch 21/53] x86/apic: Remove yet another dubious callback

[Thomas Gleixner]

Paranoia is not wrong, but having an APIC callback which is in most
implementations a complete NOOP and in one actually looking whether the
APICID of an upcoming CPU has been registered. The same APICID which was
used to bring the CPU out of wait for startup.

That's paranoia for the paranoia sake. Remove the voodoo.

Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
---
 arch/x86/include/asm/apic.h         |    1 -
 arch/x86/kernel/apic/apic.c         |    3 ---
 arch/x86/kernel/apic/apic_common.c  |    5 -----
 arch/x86/kernel/apic/apic_flat_64.c |    2 --
 arch/x86/kernel/apic/local.h        |    2 --
 arch/x86/kernel/apic/probe_32.c     |    1 -
 6 files changed, 14 deletions(-)

--- a/arch/x86/include/asm/apic.h
+++ b/arch/x86/include/asm/apic.h
@@ -290,7 +290,6 @@ struct apic {
 	/* Probe, setup and smpboot functions */
 	int	(*probe)(void);
 	int	(*acpi_madt_oem_check)(char *oem_id, char *oem_table_id);
-	bool	(*apic_id_registered)(void);
 
 	void	(*init_apic_ldr)(void);
 	u32	(*cpu_present_to_apicid)(int mps_cpu);
--- a/arch/x86/kernel/apic/apic.c
+++ b/arch/x86/kernel/apic/apic.c
@@ -1538,9 +1538,6 @@ static void setup_local_APIC(void)
 		apic_write(APIC_ESR, 0);
 	}
 #endif
-	/* Validate that the APIC is registered if required */
-	BUG_ON(apic->apic_id_registered && !apic->apic_id_registered());
-
 	/*
 	 * Intel recommends to set DFR, LDR and TPR before enabling
 	 * an APIC.  See e.g. "AP-388 82489DX User's Manual" (Intel
--- a/arch/x86/kernel/apic/apic_common.c
+++ b/arch/x86/kernel/apic/apic_common.c
@@ -27,11 +27,6 @@ u32 default_cpu_present_to_apicid(int mp
 }
 EXPORT_SYMBOL_GPL(default_cpu_present_to_apicid);
 
-bool default_apic_id_registered(void)
-{
-	return test_bit(read_apic_id(), phys_cpu_present_map);
-}
-
 /*
  * Set up the logical destination ID when the APIC operates in logical
  * destination mode.
--- a/arch/x86/kernel/apic/apic_flat_64.c
+++ b/arch/x86/kernel/apic/apic_flat_64.c
@@ -70,7 +70,6 @@ static struct apic apic_flat __ro_after_
 	.name				= "flat",
 	.probe				= flat_probe,
 	.acpi_madt_oem_check		= flat_acpi_madt_oem_check,
-	.apic_id_registered		= default_apic_id_registered,
 
 	.delivery_mode			= APIC_DELIVERY_MODE_FIXED,
 	.dest_mode_logical		= true,
@@ -139,7 +138,6 @@ static struct apic apic_physflat __ro_af
 	.name				= "physical flat",
 	.probe				= physflat_probe,
 	.acpi_madt_oem_check		= physflat_acpi_madt_oem_check,
-	.apic_id_registered		= default_apic_id_registered,
 
 	.delivery_mode			= APIC_DELIVERY_MODE_FIXED,
 	.dest_mode_logical		= false,
--- a/arch/x86/kernel/apic/local.h
+++ b/arch/x86/kernel/apic/local.h
@@ -61,8 +61,6 @@ void default_send_IPI_allbutself(int vec
 void default_send_IPI_all(int vector);
 void default_send_IPI_self(int vector);
 
-bool default_apic_id_registered(void);
-
 #ifdef CONFIG_X86_32
 void default_send_IPI_mask_sequence_logical(const struct cpumask *mask, int vector);
 void default_send_IPI_mask_allbutself_logical(const struct cpumask *mask, int vector);
--- a/arch/x86/kernel/apic/probe_32.c
+++ b/arch/x86/kernel/apic/probe_32.c
@@ -38,7 +38,6 @@ static struct apic apic_default __ro_aft
 
 	.name				= "default",
 	.probe				= probe_default,
-	.apic_id_registered		= default_apic_id_registered,
 
 	.delivery_mode			= APIC_DELIVERY_MODE_FIXED,
 	.dest_mode_logical		= true,

[patch 22/53] x86/apic: Use a proper define for invalid ACPI CPU ID

[Thomas Gleixner]

The ACPI ID for CPUs is preset with U32_MAX which is completely non
obvious. Use a proper define for it.

Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
---
 arch/x86/include/asm/apic.h  |    2 ++
 arch/x86/kernel/apic/apic.c  |    2 +-
 arch/x86/xen/enlighten_hvm.c |    2 +-
 3 files changed, 4 insertions(+), 2 deletions(-)

--- a/arch/x86/include/asm/apic.h
+++ b/arch/x86/include/asm/apic.h
@@ -46,6 +46,8 @@ extern void x86_32_probe_apic(void);
 static inline void x86_32_probe_apic(void) { }
 #endif
 
+#define CPU_ACPIID_INVALID	U32_MAX
+
 #ifdef CONFIG_X86_LOCAL_APIC
 
 extern int apic_verbosity;
--- a/arch/x86/kernel/apic/apic.c
+++ b/arch/x86/kernel/apic/apic.c
@@ -111,7 +111,7 @@ static inline bool apic_accessible(void)
  * Map cpu index to physical APIC ID
  */
 DEFINE_EARLY_PER_CPU_READ_MOSTLY(u32, x86_cpu_to_apicid, BAD_APICID);
-DEFINE_EARLY_PER_CPU_READ_MOSTLY(u32, x86_cpu_to_acpiid, U32_MAX);
+DEFINE_EARLY_PER_CPU_READ_MOSTLY(u32, x86_cpu_to_acpiid, CPU_ACPIID_INVALID);
 EXPORT_EARLY_PER_CPU_SYMBOL(x86_cpu_to_apicid);
 EXPORT_EARLY_PER_CPU_SYMBOL(x86_cpu_to_acpiid);
 
--- a/arch/x86/xen/enlighten_hvm.c
+++ b/arch/x86/xen/enlighten_hvm.c
@@ -168,7 +168,7 @@ static int xen_cpu_up_prepare_hvm(unsign
 	 */
 	xen_uninit_lock_cpu(cpu);
 
-	if (cpu_acpi_id(cpu) != U32_MAX)
+	if (cpu_acpi_id(cpu) != CPU_ACPIID_INVALID)
 		per_cpu(xen_vcpu_id, cpu) = cpu_acpi_id(cpu);
 	else
 		per_cpu(xen_vcpu_id, cpu) = cpu;

[patch 23/53] x86/cpu/topology: Move registration out of APIC code

[Thomas Gleixner]

The APIC/CPU registration sits in the middle of the APIC code. In fact this
is a topology evaluation function and has nothing to do with the inner
workings of the local APIC.

Move it out into a file which reflects what this is about.

Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
---
 arch/x86/include/asm/apic.h    |    2 
 arch/x86/kernel/apic/apic.c    |  185 -----------------------------------------
 arch/x86/kernel/cpu/Makefile   |   12 +-
 arch/x86/kernel/cpu/topology.c |  184 ++++++++++++++++++++++++++++++++++++++++
 4 files changed, 195 insertions(+), 188 deletions(-)

--- a/arch/x86/include/asm/apic.h
+++ b/arch/x86/include/asm/apic.h
@@ -171,6 +171,8 @@ extern bool apic_needs_pit(void);
 
 extern void apic_send_IPI_allbutself(unsigned int vector);
 
+extern void topology_register_boot_apic(u32 apic_id);
+
 #else /* !CONFIG_X86_LOCAL_APIC */
 static inline void lapic_shutdown(void) { }
 #define local_apic_timer_c2_ok		1
--- a/arch/x86/kernel/apic/apic.c
+++ b/arch/x86/kernel/apic/apic.c
@@ -68,26 +68,12 @@
 
 #include "local.h"
 
-unsigned int num_processors;
-
-unsigned disabled_cpus;
-
 /* Processor that is doing the boot up */
 u32 boot_cpu_physical_apicid __ro_after_init = BAD_APICID;
 EXPORT_SYMBOL_GPL(boot_cpu_physical_apicid);
 
 u8 boot_cpu_apic_version __ro_after_init;
 
-/* Bitmap of physically present CPUs. */
-DECLARE_BITMAP(phys_cpu_present_map, MAX_LOCAL_APIC);
-
-/*
- * Processor to be disabled specified by kernel parameter
- * disable_cpu_apicid=<int>, mostly used for the kdump 2nd kernel to
- * avoid undefined behaviour caused by sending INIT from AP to BSP.
- */
-static u32 disabled_cpu_apicid __ro_after_init = BAD_APICID;
-
 /*
  * This variable controls which CPUs receive external NMIs.  By default,
  * external NMIs are delivered only to the BSP.
@@ -107,14 +93,6 @@ static inline bool apic_accessible(void)
 	return x2apic_mode || apic_mmio_base;
 }
 
-/*
- * Map cpu index to physical APIC ID
- */
-DEFINE_EARLY_PER_CPU_READ_MOSTLY(u32, x86_cpu_to_apicid, BAD_APICID);
-DEFINE_EARLY_PER_CPU_READ_MOSTLY(u32, x86_cpu_to_acpiid, CPU_ACPIID_INVALID);
-EXPORT_EARLY_PER_CPU_SYMBOL(x86_cpu_to_apicid);
-EXPORT_EARLY_PER_CPU_SYMBOL(x86_cpu_to_acpiid);
-
 #ifdef CONFIG_X86_32
 /* Local APIC was disabled by the BIOS and enabled by the kernel */
 static int enabled_via_apicbase __ro_after_init;
@@ -1676,8 +1654,6 @@ void apic_ap_setup(void)
 	end_local_APIC_setup();
 }
 
-static __init void cpu_set_boot_apic(void);
-
 static __init void apic_read_boot_cpu_id(bool x2apic)
 {
 	/*
@@ -1692,7 +1668,8 @@ static __init void apic_read_boot_cpu_id
 		boot_cpu_physical_apicid = read_apic_id();
 		boot_cpu_apic_version = GET_APIC_VERSION(apic_read(APIC_LVR));
 	}
-	cpu_set_boot_apic();
+	topology_register_boot_apic(boot_cpu_physical_apicid);
+	x86_32_probe_bigsmp_early();
 }
 
 #ifdef CONFIG_X86_X2APIC
@@ -2291,155 +2268,6 @@ void disconnect_bsp_APIC(int virt_wire_s
 	apic_write(APIC_LVT1, value);
 }
 
-/*
- * The number of allocated logical CPU IDs. Since logical CPU IDs are allocated
- * contiguously, it equals to current allocated max logical CPU ID plus 1.
- * All allocated CPU IDs should be in the [0, nr_logical_cpuids) range,
- * so the maximum of nr_logical_cpuids is nr_cpu_ids.
- *
- * NOTE: Reserve 0 for BSP.
- */
-static int nr_logical_cpuids = 1;
-
-/*
- * Used to store mapping between logical CPU IDs and APIC IDs.
- */
-u32 cpuid_to_apicid[] = { [0 ... NR_CPUS - 1] = BAD_APICID, };
-
-bool arch_match_cpu_phys_id(int cpu, u64 phys_id)
-{
-	return phys_id == (u64)cpuid_to_apicid[cpu];
-}
-
-#ifdef CONFIG_SMP
-static void cpu_mark_primary_thread(unsigned int cpu, unsigned int apicid)
-{
-	/* Isolate the SMT bit(s) in the APICID and check for 0 */
-	u32 mask = (1U << (fls(smp_num_siblings) - 1)) - 1;
-
-	if (smp_num_siblings == 1 || !(apicid & mask))
-		cpumask_set_cpu(cpu, &__cpu_primary_thread_mask);
-}
-
-/*
- * Due to the utter mess of CPUID evaluation smp_num_siblings is not valid
- * during early boot. Initialize the primary thread mask before SMP
- * bringup.
- */
-static int __init smp_init_primary_thread_mask(void)
-{
-	unsigned int cpu;
-
-	/*
-	 * XEN/PV provides either none or useless topology information.
-	 * Pretend that all vCPUs are primary threads.
-	 */
-	if (xen_pv_domain()) {
-		cpumask_copy(&__cpu_primary_thread_mask, cpu_possible_mask);
-		return 0;
-	}
-
-	for (cpu = 0; cpu < nr_logical_cpuids; cpu++)
-		cpu_mark_primary_thread(cpu, cpuid_to_apicid[cpu]);
-	return 0;
-}
-early_initcall(smp_init_primary_thread_mask);
-#else
-static inline void cpu_mark_primary_thread(unsigned int cpu, unsigned int apicid) { }
-#endif
-
-/*
- * Should use this API to allocate logical CPU IDs to keep nr_logical_cpuids
- * and cpuid_to_apicid[] synchronized.
- */
-static int allocate_logical_cpuid(int apicid)
-{
-	int i;
-
-	/*
-	 * cpuid <-> apicid mapping is persistent, so when a cpu is up,
-	 * check if the kernel has allocated a cpuid for it.
-	 */
-	for (i = 0; i < nr_logical_cpuids; i++) {
-		if (cpuid_to_apicid[i] == apicid)
-			return i;
-	}
-
-	/* Allocate a new cpuid. */
-	if (nr_logical_cpuids >= nr_cpu_ids) {
-		WARN_ONCE(1, "APIC: NR_CPUS/possible_cpus limit of %u reached. "
-			     "Processor %d/0x%x and the rest are ignored.\n",
-			     nr_cpu_ids, nr_logical_cpuids, apicid);
-		return -EINVAL;
-	}
-
-	cpuid_to_apicid[nr_logical_cpuids] = apicid;
-	return nr_logical_cpuids++;
-}
-
-static void cpu_update_apic(int cpu, u32 apicid)
-{
-#if defined(CONFIG_SMP) || defined(CONFIG_X86_64)
-	early_per_cpu(x86_cpu_to_apicid, cpu) = apicid;
-#endif
-	set_cpu_possible(cpu, true);
-	set_bit(apicid, phys_cpu_present_map);
-	set_cpu_present(cpu, true);
-	num_processors++;
-
-	if (system_state != SYSTEM_BOOTING)
-		cpu_mark_primary_thread(cpu, apicid);
-}
-
-static __init void cpu_set_boot_apic(void)
-{
-	cpuid_to_apicid[0] = boot_cpu_physical_apicid;
-	cpu_update_apic(0, boot_cpu_physical_apicid);
-	x86_32_probe_bigsmp_early();
-}
-
-int generic_processor_info(int apicid)
-{
-	int cpu, max = nr_cpu_ids;
-
-	/* The boot CPU must be set before MADT/MPTABLE parsing happens */
-	if (cpuid_to_apicid[0] == BAD_APICID)
-		panic("Boot CPU APIC not registered yet\n");
-
-	if (apicid == boot_cpu_physical_apicid)
-		return 0;
-
-	if (disabled_cpu_apicid == apicid) {
-		int thiscpu = num_processors + disabled_cpus;
-
-		pr_warn("APIC: Disabling requested cpu. Processor %d/0x%x ignored.\n",
-			thiscpu, apicid);
-
-		disabled_cpus++;
-		return -ENODEV;
-	}
-
-	if (num_processors >= nr_cpu_ids) {
-		int thiscpu = max + disabled_cpus;
-
-		pr_warn("APIC: NR_CPUS/possible_cpus limit of %i reached. "
-			"Processor %d/0x%x ignored.\n", max, thiscpu, apicid);
-
-		disabled_cpus++;
-		return -EINVAL;
-	}
-
-	cpu = allocate_logical_cpuid(apicid);
-	if (cpu < 0) {
-		disabled_cpus++;
-		return -EINVAL;
-	}
-
-	cpu_update_apic(cpu, apicid);
-	return cpu;
-}
-
-
 void __irq_msi_compose_msg(struct irq_cfg *cfg, struct msi_msg *msg,
 			   bool dmar)
 {
@@ -2828,15 +2656,6 @@ static int __init lapic_insert_resource(
  */
 late_initcall(lapic_insert_resource);
 
-static int __init apic_set_disabled_cpu_apicid(char *arg)
-{
-	if (!arg || !get_option(&arg, &disabled_cpu_apicid))
-		return -EINVAL;
-
-	return 0;
-}
-early_param("disable_cpu_apicid", apic_set_disabled_cpu_apicid);
-
 static int __init apic_set_extnmi(char *arg)
 {
 	if (!arg)
--- a/arch/x86/kernel/cpu/Makefile
+++ b/arch/x86/kernel/cpu/Makefile
@@ -26,14 +26,16 @@ obj-y			+= bugs.o
 obj-y			+= aperfmperf.o
 obj-y			+= cpuid-deps.o
 obj-y			+= umwait.o
+obj-y 			+= capflags.o powerflags.o
 
-obj-$(CONFIG_PROC_FS)	+= proc.o
-obj-y += capflags.o powerflags.o
+obj-$(CONFIG_X86_LOCAL_APIC)		+= topology.o
 
-obj-$(CONFIG_IA32_FEAT_CTL) += feat_ctl.o
+obj-$(CONFIG_PROC_FS)			+= proc.o
+
+obj-$(CONFIG_IA32_FEAT_CTL)		+= feat_ctl.o
 ifdef CONFIG_CPU_SUP_INTEL
-obj-y			+= intel.o intel_pconfig.o tsx.o
-obj-$(CONFIG_PM)	+= intel_epb.o
+obj-y					+= intel.o intel_pconfig.o tsx.o
+obj-$(CONFIG_PM)			+= intel_epb.o
 endif
 obj-$(CONFIG_CPU_SUP_AMD)		+= amd.o
 obj-$(CONFIG_CPU_SUP_HYGON)		+= hygon.o
--- /dev/null
+++ b/arch/x86/kernel/cpu/topology.c
@@ -0,0 +1,184 @@
+// SPDX-License-Identifier: GPL-2.0-only
+
+#include <linux/cpu.h>
+
+#include <xen/xen.h>
+
+#include <asm/apic.h>
+#include <asm/mpspec.h>
+#include <asm/smp.h>
+
+/*
+ * Map cpu index to physical APIC ID
+ */
+DEFINE_EARLY_PER_CPU_READ_MOSTLY(u32, x86_cpu_to_apicid, BAD_APICID);
+DEFINE_EARLY_PER_CPU_READ_MOSTLY(u32, x86_cpu_to_acpiid, CPU_ACPIID_INVALID);
+EXPORT_EARLY_PER_CPU_SYMBOL(x86_cpu_to_apicid);
+EXPORT_EARLY_PER_CPU_SYMBOL(x86_cpu_to_acpiid);
+
+/* Bitmap of physically present CPUs. */
+DECLARE_BITMAP(phys_cpu_present_map, MAX_LOCAL_APIC) __read_mostly;
+
+/* Used for CPU number allocation and parallel CPU bringup */
+u32 cpuid_to_apicid[] __read_mostly = { [0 ... NR_CPUS - 1] = BAD_APICID, };
+
+/*
+ * Processor to be disabled specified by kernel parameter
+ * disable_cpu_apicid=<int>, mostly used for the kdump 2nd kernel to
+ * avoid undefined behaviour caused by sending INIT from AP to BSP.
+ */
+static u32 disabled_cpu_apicid __ro_after_init = BAD_APICID;
+
+unsigned int num_processors;
+unsigned disabled_cpus;
+
+/*
+ * The number of allocated logical CPU IDs. Since logical CPU IDs are allocated
+ * contiguously, it equals to current allocated max logical CPU ID plus 1.
+ * All allocated CPU IDs should be in the [0, nr_logical_cpuids) range,
+ * so the maximum of nr_logical_cpuids is nr_cpu_ids.
+ *
+ * NOTE: Reserve 0 for BSP.
+ */
+static int nr_logical_cpuids = 1;
+
+bool arch_match_cpu_phys_id(int cpu, u64 phys_id)
+{
+	return phys_id == (u64)cpuid_to_apicid[cpu];
+}
+
+#ifdef CONFIG_SMP
+static void cpu_mark_primary_thread(unsigned int cpu, unsigned int apicid)
+{
+	/* Isolate the SMT bit(s) in the APICID and check for 0 */
+	u32 mask = (1U << (fls(smp_num_siblings) - 1)) - 1;
+
+	if (smp_num_siblings == 1 || !(apicid & mask))
+		cpumask_set_cpu(cpu, &__cpu_primary_thread_mask);
+}
+
+/*
+ * Due to the utter mess of CPUID evaluation smp_num_siblings is not valid
+ * during early boot. Initialize the primary thread mask before SMP
+ * bringup.
+ */
+static int __init smp_init_primary_thread_mask(void)
+{
+	unsigned int cpu;
+
+	/*
+	 * XEN/PV provides either none or useless topology information.
+	 * Pretend that all vCPUs are primary threads.
+	 */
+	if (xen_pv_domain()) {
+		cpumask_copy(&__cpu_primary_thread_mask, cpu_possible_mask);
+		return 0;
+	}
+
+	for (cpu = 0; cpu < nr_logical_cpuids; cpu++)
+		cpu_mark_primary_thread(cpu, cpuid_to_apicid[cpu]);
+	return 0;
+}
+early_initcall(smp_init_primary_thread_mask);
+#else
+static inline void cpu_mark_primary_thread(unsigned int cpu, unsigned int apicid) { }
+#endif
+
+/*
+ * Should use this API to allocate logical CPU IDs to keep nr_logical_cpuids
+ * and cpuid_to_apicid[] synchronized.
+ */
+static int allocate_logical_cpuid(int apicid)
+{
+	int i;
+
+	/*
+	 * cpuid <-> apicid mapping is persistent, so when a cpu is up,
+	 * check if the kernel has allocated a cpuid for it.
+	 */
+	for (i = 0; i < nr_logical_cpuids; i++) {
+		if (cpuid_to_apicid[i] == apicid)
+			return i;
+	}
+
+	/* Allocate a new cpuid. */
+	if (nr_logical_cpuids >= nr_cpu_ids) {
+		WARN_ONCE(1, "APIC: NR_CPUS/possible_cpus limit of %u reached. "
+			     "Processor %d/0x%x and the rest are ignored.\n",
+			     nr_cpu_ids, nr_logical_cpuids, apicid);
+		return -EINVAL;
+	}
+
+	cpuid_to_apicid[nr_logical_cpuids] = apicid;
+	return nr_logical_cpuids++;
+}
+
+static void cpu_update_apic(int cpu, u32 apicid)
+{
+#if defined(CONFIG_SMP) || defined(CONFIG_X86_64)
+	early_per_cpu(x86_cpu_to_apicid, cpu) = apicid;
+#endif
+	set_cpu_possible(cpu, true);
+	set_bit(apicid, phys_cpu_present_map);
+	set_cpu_present(cpu, true);
+	num_processors++;
+
+	if (system_state != SYSTEM_BOOTING)
+		cpu_mark_primary_thread(cpu, apicid);
+}
+
+void __init topology_register_boot_apic(u32 apic_id)
+{
+	cpuid_to_apicid[0] = apic_id;
+	cpu_update_apic(0, apic_id);
+}
+
+int generic_processor_info(int apicid)
+{
+	int cpu, max = nr_cpu_ids;
+
+	/* The boot CPU must be set before MADT/MPTABLE parsing happens */
+	if (cpuid_to_apicid[0] == BAD_APICID)
+		panic("Boot CPU APIC not registered yet\n");
+
+	if (apicid == boot_cpu_physical_apicid)
+		return 0;
+
+	if (disabled_cpu_apicid == apicid) {
+		int thiscpu = num_processors + disabled_cpus;
+
+		pr_warn("APIC: Disabling requested cpu. Processor %d/0x%x ignored.\n",
+			thiscpu, apicid);
+
+		disabled_cpus++;
+		return -ENODEV;
+	}
+
+	if (num_processors >= nr_cpu_ids) {
+		int thiscpu = max + disabled_cpus;
+
+		pr_warn("APIC: NR_CPUS/possible_cpus limit of %i reached. "
+			"Processor %d/0x%x ignored.\n", max, thiscpu, apicid);
+
+		disabled_cpus++;
+		return -EINVAL;
+	}
+
+	cpu = allocate_logical_cpuid(apicid);
+	if (cpu < 0) {
+		disabled_cpus++;
+		return -EINVAL;
+	}
+
+	cpu_update_apic(cpu, apicid);
+	return cpu;
+}
+
+static int __init apic_set_disabled_cpu_apicid(char *arg)
+{
+	if (!arg || !get_option(&arg, &disabled_cpu_apicid))
+		return -EINVAL;
+
+	return 0;
+}
+early_param("disable_cpu_apicid", apic_set_disabled_cpu_apicid);

[patch 24/53] x86/cpu/topology: Provide separate APIC registration functions

[Thomas Gleixner]

generic_processor_info() aside of being a complete misnomer is used for
both early boot registration and ACPI CPU hotplug.

While it's arguable that this can share some code, it results in code which
is hard to understand and kept around post init for no real reason.

Also the call sites do lots of manual fiddling in topology related
variables instead of having proper interfaces for the purpose which handle
the topology internals correctly.

Provide topology_register_apic(), topology_hotplug_apic() and
topology_hot_unplug_apic() which have the extra magic of the call sites
incorporated and for now are wrappers around generic_processor_info().

Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
---
 arch/x86/include/asm/apic.h         |    3 
 arch/x86/kernel/cpu/topology.c |  113 ++++++++++++++++++++++++++++++------
 2 files changed, 98 insertions(+), 18 deletions(-)

--- a/arch/x86/include/asm/apic.h
+++ b/arch/x86/include/asm/apic.h
@@ -171,7 +171,10 @@ extern bool apic_needs_pit(void);
 
 extern void apic_send_IPI_allbutself(unsigned int vector);
 
+extern void topology_register_apic(u32 apic_id, u32 acpi_id, bool present);
 extern void topology_register_boot_apic(u32 apic_id);
+extern int topology_hotplug_apic(u32 apic_id, u32 acpi_id);
+extern void topology_hotunplug_apic(unsigned int cpu);
 
 #else /* !CONFIG_X86_LOCAL_APIC */
 static inline void lapic_shutdown(void) { }
--- a/arch/x86/kernel/cpu/topology.c
+++ b/arch/x86/kernel/cpu/topology.c
@@ -79,32 +79,38 @@ early_initcall(smp_init_primary_thread_m
 static inline void cpu_mark_primary_thread(unsigned int cpu, unsigned int apicid) { }
 #endif
 
-/*
- * Should use this API to allocate logical CPU IDs to keep nr_logical_cpuids
- * and cpuid_to_apicid[] synchronized.
- */
-static int allocate_logical_cpuid(int apicid)
+static int topo_lookup_cpuid(u32 apic_id)
 {
 	int i;
 
-	/*
-	 * cpuid <-> apicid mapping is persistent, so when a cpu is up,
-	 * check if the kernel has allocated a cpuid for it.
-	 */
+	/* CPU# to APICID mapping is persistent once it is established */
 	for (i = 0; i < nr_logical_cpuids; i++) {
-		if (cpuid_to_apicid[i] == apicid)
+		if (cpuid_to_apicid[i] == apic_id)
 			return i;
 	}
+	return -ENODEV;
+}
+
+/*
+ * Should use this API to allocate logical CPU IDs to keep nr_logical_cpuids
+ * and cpuid_to_apicid[] synchronized.
+ */
+static int allocate_logical_cpuid(u32 apic_id)
+{
+	int cpu = topo_lookup_cpuid(apic_id);
+
+	if (cpu >= 0)
+		return cpu;
 
 	/* Allocate a new cpuid. */
 	if (nr_logical_cpuids >= nr_cpu_ids) {
 		WARN_ONCE(1, "APIC: NR_CPUS/possible_cpus limit of %u reached. "
 			     "Processor %d/0x%x and the rest are ignored.\n",
-			     nr_cpu_ids, nr_logical_cpuids, apicid);
+			     nr_cpu_ids, nr_logical_cpuids, apic_id);
 		return -EINVAL;
 	}
 
-	cpuid_to_apicid[nr_logical_cpuids] = apicid;
+	cpuid_to_apicid[nr_logical_cpuids] = apic_id;
 	return nr_logical_cpuids++;
 }
 
@@ -122,12 +128,6 @@ static void cpu_update_apic(int cpu, u32
 		cpu_mark_primary_thread(cpu, apicid);
 }
 
-void __init topology_register_boot_apic(u32 apic_id)
-{
-	cpuid_to_apicid[0] = apic_id;
-	cpu_update_apic(0, apic_id);
-}
-
 int generic_processor_info(int apicid)
 {
 	int cpu, max = nr_cpu_ids;
@@ -169,6 +169,83 @@ int generic_processor_info(int apicid)
 	return cpu;
 }
 
+/**
+ * topology_register_apic - Register an APIC in early topology maps
+ * @apic_id:	The APIC ID to set up
+ * @acpi_id:	The ACPI ID associated to the APIC
+ * @present:	True if the corresponding CPU is present
+ */
+void __init topology_register_apic(u32 apic_id, u32 acpi_id, bool present)
+{
+	int cpu;
+
+	if (apic_id >= MAX_LOCAL_APIC) {
+		pr_err_once("APIC ID %x exceeds kernel limit of: %x\n", apic_id, MAX_LOCAL_APIC - 1);
+		return;
+	}
+
+	if (!present) {
+		disabled_cpus++;
+		return;
+	}
+
+	cpu = generic_processor_info(apic_id);
+	if (cpu >= 0)
+		early_per_cpu(x86_cpu_to_acpiid, cpu) = acpi_id;
+}
+
+/**
+ * topology_register_boot_apic - Register the boot CPU APIC
+ * @apic_id:	The APIC ID to set up
+ *
+ * Separate so CPU #0 can be assigned
+ */
+void __init topology_register_boot_apic(u32 apic_id)
+{
+	cpuid_to_apicid[0] = apic_id;
+	cpu_update_apic(0, apic_id);
+}
+
+#ifdef CONFIG_ACPI_HOTPLUG_CPU
+/**
+ * topology_hotplug_apic - Handle a physical hotplugged APIC after boot
+ * @apic_id:	The APIC ID to set up
+ * @acpi_id:	The ACPI ID associated to the APIC
+ */
+int topology_hotplug_apic(u32 apic_id, u32 acpi_id)
+{
+	int cpu;
+
+	if (apic_id >= MAX_LOCAL_APIC)
+		return -EINVAL;
+
+	cpu = topo_lookup_cpuid(apic_id);
+	if (cpu < 0) {
+		cpu = generic_processor_info(apic_id);
+		if (cpu >= 0)
+			per_cpu(x86_cpu_to_acpiid, cpu) = acpi_id;
+	}
+	return cpu;
+}
+
+/**
+ * topology_hotunplug_apic - Remove a physical hotplugged APIC after boot
+ * @cpu:	The CPU number for which the APIC ID is removed
+ */
+void topology_hotunplug_apic(unsigned int cpu)
+{
+	u32 apic_id = cpuid_to_apicid[cpu];
+
+	if (apic_id == BAD_APICID)
+		return;
+
+	per_cpu(x86_cpu_to_apicid, cpu) = BAD_APICID;
+	clear_bit(apic_id, phys_cpu_present_map);
+	set_cpu_present(cpu, false);
+	num_processors--;
+}
+#endif
+
 static int __init apic_set_disabled_cpu_apicid(char *arg)
 {
 	if (!arg || !get_option(&arg, &disabled_cpu_apicid))

[patch 25/53] x86/acpi: Use new APIC registration functions

[Thomas Gleixner]

Use the new topology registration functions and make the early boot code
path __init. No functional change intended.

Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
---
 arch/x86/kernel/acpi/boot.c |   44 +++++++-------------------------------------
 1 file changed, 7 insertions(+), 37 deletions(-)

--- a/arch/x86/kernel/acpi/boot.c
+++ b/arch/x86/kernel/acpi/boot.c
@@ -159,33 +159,9 @@ static int __init acpi_parse_madt(struct
 	return 0;
 }
 
-/**
- * acpi_register_lapic - register a local apic and generates a logic cpu number
- * @id: local apic id to register
- * @acpiid: ACPI id to register
- * @enabled: this cpu is enabled or not
- *
- * Returns the logic cpu number which maps to the local apic
- */
-static int acpi_register_lapic(int id, u32 acpiid, u8 enabled)
+static __init void acpi_register_lapic(u32 apic_id, u32 acpi_id, bool present)
 {
-	int cpu;
-
-	if (id >= MAX_LOCAL_APIC) {
-		pr_info("skipped apicid that is too big\n");
-		return -EINVAL;
-	}
-
-	if (!enabled) {
-		++disabled_cpus;
-		return -EINVAL;
-	}
-
-	cpu = generic_processor_info(id);
-	if (cpu >= 0)
-		early_per_cpu(x86_cpu_to_acpiid, cpu) = acpiid;
-
-	return cpu;
+	topology_register_apic(apic_id, acpi_id, present);
 }
 
 static bool __init acpi_is_processor_usable(u32 lapic_flags)
@@ -827,12 +803,10 @@ static int acpi_map_cpu2node(acpi_handle
 	return 0;
 }
 
-int acpi_map_cpu(acpi_handle handle, phys_cpuid_t physid, u32 acpi_id,
-		 int *pcpu)
+int acpi_map_cpu(acpi_handle handle, phys_cpuid_t physid, u32 acpi_id, int *pcpu)
 {
-	int cpu;
+	int cpu = topology_hotplug_apic(physid, acpi_id);
 
-	cpu = acpi_register_lapic(physid, acpi_id, ACPI_MADT_ENABLED);
 	if (cpu < 0) {
 		pr_info("Unable to map lapic to logical cpu number\n");
 		return cpu;
@@ -851,15 +825,11 @@ int acpi_unmap_cpu(int cpu)
 #ifdef CONFIG_ACPI_NUMA
 	set_apicid_to_node(per_cpu(x86_cpu_to_apicid, cpu), NUMA_NO_NODE);
 #endif
-
-	per_cpu(x86_cpu_to_apicid, cpu) = BAD_APICID;
-	set_cpu_present(cpu, false);
-	num_processors--;
-
-	return (0);
+	topology_hotunplug_apic(cpu);
+	return 0;
 }
 EXPORT_SYMBOL(acpi_unmap_cpu);
-#endif				/* CONFIG_ACPI_HOTPLUG_CPU */
+#endif	/* CONFIG_ACPI_HOTPLUG_CPU */
 
 int acpi_register_ioapic(acpi_handle handle, u64 phys_addr, u32 gsi_base)
 {

[patch 26/53] x86/jailhouse: Use new APIC registration function

[Thomas Gleixner]

No functional change intended.

Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
---
 arch/x86/kernel/jailhouse.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

--- a/arch/x86/kernel/jailhouse.c
+++ b/arch/x86/kernel/jailhouse.c
@@ -102,7 +102,7 @@ static void __init jailhouse_parse_smp_c
 	register_lapic_address(0xfee00000);
 
 	for (cpu = 0; cpu < setup_data.v1.num_cpus; cpu++)
-		generic_processor_info(setup_data.v1.cpu_ids[cpu]);
+		topology_register_apic(setup_data.v1.cpu_ids[cpu], CPU_ACPIID_INVALID, true);
 
 	smp_found_config = 1;
 

[patch 27/53] x86/of: Use new APIC registration functions

[Thomas Gleixner]

No functional change intended.

Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
---
 arch/x86/kernel/devicetree.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

--- a/arch/x86/kernel/devicetree.c
+++ b/arch/x86/kernel/devicetree.c
@@ -136,7 +136,7 @@ static void __init dtb_cpu_setup(void)
 			pr_warn("%pOF: missing local APIC ID\n", dn);
 			continue;
 		}
-		generic_processor_info(apic_id);
+		topology_register_apic(apic_id, CPU_ACPIID_INVALID, true);
 	}
 }
 

[patch 28/53] x86/mpparse: Use new APIC registration function

[Thomas Gleixner]

Aside of switching over to the new interface, record the number of
registered CPUs locally, which allows to make num_processors and
disabled_cpus confined to the topology code.

No functional change intended.

Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
---
 arch/x86/include/asm/mpspec.h       |    2 --
 arch/x86/kernel/cpu/topology.c |    2 +-
 arch/x86/kernel/mpparse.c           |   21 ++++++++++-----------
 3 files changed, 11 insertions(+), 14 deletions(-)

--- a/arch/x86/include/asm/mpspec.h
+++ b/arch/x86/include/asm/mpspec.h
@@ -61,8 +61,6 @@ static inline void e820__memblock_alloc_
 #define mpparse_parse_smp_config	x86_init_noop
 #endif
 
-int generic_processor_info(int apicid);
-
 extern DECLARE_BITMAP(phys_cpu_present_map, MAX_LOCAL_APIC);
 
 static inline void reset_phys_cpu_present_map(u32 apicid)
--- a/arch/x86/kernel/cpu/topology.c
+++ b/arch/x86/kernel/cpu/topology.c
@@ -128,7 +128,7 @@ static void cpu_update_apic(int cpu, u32
 		cpu_mark_primary_thread(cpu, apicid);
 }
 
-int generic_processor_info(int apicid)
+static int generic_processor_info(int apicid)
 {
 	int cpu, max = nr_cpu_ids;
 
--- a/arch/x86/kernel/mpparse.c
+++ b/arch/x86/kernel/mpparse.c
@@ -36,6 +36,8 @@
  * Checksum an MP configuration block.
  */
 
+static unsigned int num_procs __initdata;
+
 static int __init mpf_checksum(unsigned char *mp, int len)
 {
 	int sum = 0;
@@ -48,21 +50,18 @@ static int __init mpf_checksum(unsigned
 
 static void __init MP_processor_info(struct mpc_cpu *m)
 {
-	int apicid;
 	char *bootup_cpu = "";
 
-	if (!(m->cpuflag & CPU_ENABLED)) {
-		disabled_cpus++;
-		return;
-	}
+	topology_register_apic(m->apicid, CPU_ACPIID_INVALID, m->cpuflag & CPU_ENABLED);
 
-	apicid = m->apicid;
+	if (!(m->cpuflag & CPU_ENABLED))
+		return;
 
 	if (m->cpuflag & CPU_BOOTPROCESSOR)
 		bootup_cpu = " (Bootup-CPU)";
 
 	pr_info("Processor #%d%s\n", m->apicid, bootup_cpu);
-	generic_processor_info(apicid);
+	num_procs++;
 }
 
 #ifdef CONFIG_X86_IO_APIC
@@ -239,9 +238,9 @@ static int __init smp_read_mpc(struct mp
 		}
 	}
 
-	if (!num_processors)
+	if (!num_procs && !acpi_lapic)
 		pr_err("MPTABLE: no processors registered!\n");
-	return num_processors;
+	return num_procs || acpi_lapic;
 }
 
 #ifdef CONFIG_X86_IO_APIC
@@ -532,8 +531,8 @@ static __init void mpparse_get_smp_confi
 	} else
 		BUG();
 
-	if (!early)
-		pr_info("Processors: %d\n", num_processors);
+	if (!early && !acpi_lapic)
+		pr_info("Processors: %d\n", num_procs);
 	/*
 	 * Only use the first configuration found.
 	 */

[patch 29/53] x86/acpi: Dont invoke topology_register_apic() for XEN PV

[Thomas Gleixner]

The MADT table for XEN/PV dom0 is not really useful and registering the
APICs is momentarily a pointless exercise because XENPV does not use an
APIC at all.

It overrides the x86_init.mpparse.parse_smp_config() callback, resets
num_processors and counts how many of them are provided by the hypervisor.

This is in the way of cleaning up the APIC registration. Prevent MADT
registration for XEN/PV temporarily until the rework is completed and
XEN/PV can use the MADT again.

Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
---
 arch/x86/kernel/acpi/boot.c |    8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

--- a/arch/x86/kernel/acpi/boot.c
+++ b/arch/x86/kernel/acpi/boot.c
@@ -23,6 +23,8 @@
 #include <linux/serial_core.h>
 #include <linux/pgtable.h>
 
+#include <xen/xen.h>
+
 #include <asm/e820/api.h>
 #include <asm/irqdomain.h>
 #include <asm/pci_x86.h>
@@ -161,7 +163,8 @@ static int __init acpi_parse_madt(struct
 
 static __init void acpi_register_lapic(u32 apic_id, u32 acpi_id, bool present)
 {
-	topology_register_apic(apic_id, acpi_id, present);
+	if (!xen_pv_domain())
+		topology_register_apic(apic_id, acpi_id, present);
 }
 
 static bool __init acpi_is_processor_usable(u32 lapic_flags)
@@ -1070,7 +1073,8 @@ static int __init early_acpi_parse_madt_
 		return count;
 	}
 
-	register_lapic_address(acpi_lapic_addr);
+	if (!xen_pv_domain())
+		register_lapic_address(acpi_lapic_addr);
 
 	return count;
 }

[patch 30/53] x86/xen/smp_pv: Register fake APICs

[Thomas Gleixner]

XENPV does not use the APIC. It's just piggy packing on the infrastructure
and fiddles with global variables as it sees fit.

These global variables are going away, so let XENPV register pseudo APIC
IDs to keep the accounting correct and keep up the illusion that XEN/PV is
something sane.

Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
---
 arch/x86/xen/smp_pv.c |   35 +++++++++--------------------------
 1 file changed, 9 insertions(+), 26 deletions(-)

--- a/arch/x86/xen/smp_pv.c
+++ b/arch/x86/xen/smp_pv.c
@@ -29,6 +29,7 @@
 #include <asm/idtentry.h>
 #include <asm/desc.h>
 #include <asm/cpu.h>
+#include <asm/apic.h>
 #include <asm/io_apic.h>
 
 #include <xen/interface/xen.h>
@@ -150,34 +151,16 @@ int xen_smp_intr_init_pv(unsigned int cp
 
 static void __init xen_pv_smp_config(void)
 {
-	int i, rc;
-	unsigned int subtract = 0;
+	u32 apicid = 0;
+	int i;
 
-	num_processors = 0;
-	disabled_cpus = 0;
-	for (i = 0; i < nr_cpu_ids; i++) {
-		rc = HYPERVISOR_vcpu_op(VCPUOP_is_up, i, NULL);
-		if (rc >= 0) {
-			num_processors++;
-			set_cpu_possible(i, true);
-		} else {
-			set_cpu_possible(i, false);
-			set_cpu_present(i, false);
-			subtract++;
-		}
+	topology_register_boot_apic(apicid++);
+
+	for (i = 1; i < nr_cpu_ids; i++) {
+		if (HYPERVISOR_vcpu_op(VCPUOP_is_up, i, NULL) < 0)
+			break;
+		topology_register_apic(apicid++, CPU_ACPIID_INVALID, true);
 	}
-#ifdef CONFIG_HOTPLUG_CPU
-	/* This is akin to using 'nr_cpus' on the Linux command line.
-	 * Which is OK as when we use 'dom0_max_vcpus=X' we can only
-	 * have up to X, while nr_cpu_ids is greater than X. This
-	 * normally is not a problem, except when CPU hotplugging
-	 * is involved and then there might be more than X CPUs
-	 * in the guest - which will not work as there is no
-	 * hypercall to expand the max number of VCPUs an already
-	 * running guest has. So cap it up to X. */
-	if (subtract)
-		set_nr_cpu_ids(nr_cpu_ids - subtract);
-#endif
 	/* Pretend to be a proper enumerated system */
 	smp_found_config = 1;
 }

[patch 31/53] x86/cpu/topology: Confine topology information

[Thomas Gleixner]

Now that all external fiddling with num_processors and disabled_cpus is
gone, move the last user prefill_possible_map() into the topology code too
and remove the global visibility of these variables.

Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
---
 arch/x86/include/asm/smp.h          |    3 -
 arch/x86/kernel/apic/apic.c         |    1 
 arch/x86/kernel/cpu/topology.c |   76 +++++++++++++++++++++++++++++++++++-
 arch/x86/kernel/smpboot.c           |   72 ----------------------------------
 4 files changed, 74 insertions(+), 78 deletions(-)

--- a/arch/x86/include/asm/smp.h
+++ b/arch/x86/include/asm/smp.h
@@ -9,7 +9,6 @@
 #include <asm/thread_info.h>
 
 extern int smp_num_siblings;
-extern unsigned int num_processors;
 
 DECLARE_PER_CPU_READ_MOSTLY(cpumask_var_t, cpu_sibling_map);
 DECLARE_PER_CPU_READ_MOSTLY(cpumask_var_t, cpu_core_map);
@@ -178,8 +177,6 @@ static inline struct cpumask *cpu_llc_sh
 }
 #endif /* CONFIG_SMP */
 
-extern unsigned disabled_cpus;
-
 #ifdef CONFIG_DEBUG_NMI_SELFTEST
 extern void nmi_selftest(void);
 #else
--- a/arch/x86/kernel/apic/apic.c
+++ b/arch/x86/kernel/apic/apic.c
@@ -2054,7 +2054,6 @@ void __init init_apic_mappings(void)
 			pr_info("APIC: disable apic facility\n");
 			apic_disable();
 		}
-		num_processors = 1;
 	}
 }
 
--- a/arch/x86/kernel/cpu/topology.c
+++ b/arch/x86/kernel/cpu/topology.c
@@ -29,8 +29,8 @@ u32 cpuid_to_apicid[] __read_mostly = {
  */
 static u32 disabled_cpu_apicid __ro_after_init = BAD_APICID;
 
-unsigned int num_processors;
-unsigned disabled_cpus;
+static unsigned int num_processors;
+static unsigned int disabled_cpus;
 
 /*
  * The number of allocated logical CPU IDs. Since logical CPU IDs are allocated
@@ -169,6 +169,71 @@ static int generic_processor_info(int ap
 	return cpu;
 }
 
+static int __initdata setup_possible_cpus = -1;
+
+/*
+ * cpu_possible_mask should be static, it cannot change as cpu's
+ * are onlined, or offlined. The reason is per-cpu data-structures
+ * are allocated by some modules at init time, and don't expect to
+ * do this dynamically on cpu arrival/departure.
+ * cpu_present_mask on the other hand can change dynamically.
+ * In case when cpu_hotplug is not compiled, then we resort to current
+ * behaviour, which is cpu_possible == cpu_present.
+ * - Ashok Raj
+ *
+ * Three ways to find out the number of additional hotplug CPUs:
+ * - If the BIOS specified disabled CPUs in ACPI/mptables use that.
+ * - The user can overwrite it with possible_cpus=NUM
+ * - Otherwise don't reserve additional CPUs.
+ * We do this because additional CPUs waste a lot of memory.
+ * -AK
+ */
+__init void prefill_possible_map(void)
+{
+	int i, possible;
+
+	i = setup_max_cpus ?: 1;
+	if (setup_possible_cpus == -1) {
+		possible = num_processors;
+#ifdef CONFIG_HOTPLUG_CPU
+		if (setup_max_cpus)
+			possible += disabled_cpus;
+#else
+		if (possible > i)
+			possible = i;
+#endif
+	} else
+		possible = setup_possible_cpus;
+
+	total_cpus = max_t(int, possible, num_processors + disabled_cpus);
+
+	/* nr_cpu_ids could be reduced via nr_cpus= */
+	if (possible > nr_cpu_ids) {
+		pr_warn("%d Processors exceeds NR_CPUS limit of %u\n",
+			possible, nr_cpu_ids);
+		possible = nr_cpu_ids;
+	}
+
+#ifdef CONFIG_HOTPLUG_CPU
+	if (!setup_max_cpus)
+#endif
+	if (possible > i) {
+		pr_warn("%d Processors exceeds max_cpus limit of %u\n",
+			possible, setup_max_cpus);
+		possible = i;
+	}
+
+	set_nr_cpu_ids(possible);
+
+	pr_info("Allowing %d CPUs, %d hotplug CPUs\n",
+		possible, max_t(int, possible - num_processors, 0));
+
+	reset_cpu_possible_mask();
+
+	for (i = 0; i < possible; i++)
+		set_cpu_possible(i, true);
+}
+
 /**
  * topology_register_apic - Register an APIC in early topology maps
  * @apic_id:	The APIC ID to set up
@@ -246,6 +311,13 @@ void topology_hotunplug_apic(unsigned in
 }
 #endif
 
+static int __init _setup_possible_cpus(char *str)
+{
+	get_option(&str, &setup_possible_cpus);
+	return 0;
+}
+early_param("possible_cpus", _setup_possible_cpus);
+
 static int __init apic_set_disabled_cpu_apicid(char *arg)
 {
 	if (!arg || !get_option(&arg, &disabled_cpu_apicid))
--- a/arch/x86/kernel/smpboot.c
+++ b/arch/x86/kernel/smpboot.c
@@ -1327,78 +1327,6 @@ void __init native_smp_cpus_done(unsigne
 	cache_aps_init();
 }
 
-static int __initdata setup_possible_cpus = -1;
-static int __init _setup_possible_cpus(char *str)
-{
-	get_option(&str, &setup_possible_cpus);
-	return 0;
-}
-early_param("possible_cpus", _setup_possible_cpus);
-
-
-/*
- * cpu_possible_mask should be static, it cannot change as cpu's
- * are onlined, or offlined. The reason is per-cpu data-structures
- * are allocated by some modules at init time, and don't expect to
- * do this dynamically on cpu arrival/departure.
- * cpu_present_mask on the other hand can change dynamically.
- * In case when cpu_hotplug is not compiled, then we resort to current
- * behaviour, which is cpu_possible == cpu_present.
- * - Ashok Raj
- *
- * Three ways to find out the number of additional hotplug CPUs:
- * - If the BIOS specified disabled CPUs in ACPI/mptables use that.
- * - The user can overwrite it with possible_cpus=NUM
- * - Otherwise don't reserve additional CPUs.
- * We do this because additional CPUs waste a lot of memory.
- * -AK
- */
-__init void prefill_possible_map(void)
-{
-	int i, possible;
-
-	i = setup_max_cpus ?: 1;
-	if (setup_possible_cpus == -1) {
-		possible = num_processors;
-#ifdef CONFIG_HOTPLUG_CPU
-		if (setup_max_cpus)
-			possible += disabled_cpus;
-#else
-		if (possible > i)
-			possible = i;
-#endif
-	} else
-		possible = setup_possible_cpus;
-
-	total_cpus = max_t(int, possible, num_processors + disabled_cpus);
-
-	/* nr_cpu_ids could be reduced via nr_cpus= */
-	if (possible > nr_cpu_ids) {
-		pr_warn("%d Processors exceeds NR_CPUS limit of %u\n",
-			possible, nr_cpu_ids);
-		possible = nr_cpu_ids;
-	}
-
-#ifdef CONFIG_HOTPLUG_CPU
-	if (!setup_max_cpus)
-#endif
-	if (possible > i) {
-		pr_warn("%d Processors exceeds max_cpus limit of %u\n",
-			possible, setup_max_cpus);
-		possible = i;
-	}
-
-	set_nr_cpu_ids(possible);
-
-	pr_info("Allowing %d CPUs, %d hotplug CPUs\n",
-		possible, max_t(int, possible - num_processors, 0));
-
-	reset_cpu_possible_mask();
-
-	for (i = 0; i < possible; i++)
-		set_cpu_possible(i, true);
-}
-
 /* correctly size the local cpu masks */
 void __init setup_cpu_local_masks(void)
 {

[patch 32/53] x86/cpu/topology: Simplify APIC registration

[Thomas Gleixner]

Having the same check whether the number of assigned CPUs has reached the
nr_cpu_ids limit twice in the same code path is pointless. Repeating the
information that CPUs are ignored over and over is also pointless noise.

Remove the redundant check and reduce the noise by using a pr_warn_once().

Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
---
 arch/x86/kernel/cpu/topology.c |   23 +++--------------------
 1 file changed, 3 insertions(+), 20 deletions(-)

--- a/arch/x86/kernel/cpu/topology.c
+++ b/arch/x86/kernel/cpu/topology.c
@@ -102,14 +102,6 @@ static int allocate_logical_cpuid(u32 ap
 	if (cpu >= 0)
 		return cpu;
 
-	/* Allocate a new cpuid. */
-	if (nr_logical_cpuids >= nr_cpu_ids) {
-		WARN_ONCE(1, "APIC: NR_CPUS/possible_cpus limit of %u reached. "
-			     "Processor %d/0x%x and the rest are ignored.\n",
-			     nr_cpu_ids, nr_logical_cpuids, apic_id);
-		return -EINVAL;
-	}
-
 	cpuid_to_apicid[nr_logical_cpuids] = apic_id;
 	return nr_logical_cpuids++;
 }
@@ -130,7 +122,7 @@ static void cpu_update_apic(int cpu, u32
 
 static int generic_processor_info(int apicid)
 {
-	int cpu, max = nr_cpu_ids;
+	int cpu;
 
 	/* The boot CPU must be set before MADT/MPTABLE parsing happens */
 	if (cpuid_to_apicid[0] == BAD_APICID)
@@ -150,21 +142,12 @@ static int generic_processor_info(int ap
 	}
 
 	if (num_processors >= nr_cpu_ids) {
-		int thiscpu = max + disabled_cpus;
-
-		pr_warn("APIC: NR_CPUS/possible_cpus limit of %i reached. "
-			"Processor %d/0x%x ignored.\n", max, thiscpu, apicid);
-
+		pr_warn_once("APIC: CPU limit of %d reached. Ignoring further CPUs\n", nr_cpu_ids);
 		disabled_cpus++;
-		return -EINVAL;
+		return -ENOSPC;
 	}
 
 	cpu = allocate_logical_cpuid(apicid);
-	if (cpu < 0) {
-		disabled_cpus++;
-		return -EINVAL;
-	}
-
 	cpu_update_apic(cpu, apicid);
 	return cpu;
 }

[patch 33/53] x86/cpu/topology: Use a data structure for topology info

[Thomas Gleixner]

Put the processor accounting into a data structure, which will gain more
topology related information in the next steps, and sanitize the accounting.

Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
---
 arch/x86/kernel/cpu/topology.c |   59 ++++++++++++++++++++---------------------
 1 file changed, 29 insertions(+), 30 deletions(-)

--- a/arch/x86/kernel/cpu/topology.c
+++ b/arch/x86/kernel/cpu/topology.c
@@ -23,25 +23,24 @@ DECLARE_BITMAP(phys_cpu_present_map, MAX
 u32 cpuid_to_apicid[] __read_mostly = { [0 ... NR_CPUS - 1] = BAD_APICID, };
 
 /*
+ * Keep track of assigned, disabled and rejected CPUs. Present assigned
+ * with 1 as CPU #0 is reserved for the boot CPU.
+ */
+struct {
+	unsigned int		nr_assigned_cpus;
+	unsigned int		nr_disabled_cpus;
+	unsigned int		nr_rejected_cpus;
+} topo_info __read_mostly = {
+	.nr_assigned_cpus	= 1,
+};
+
+/*
  * Processor to be disabled specified by kernel parameter
  * disable_cpu_apicid=<int>, mostly used for the kdump 2nd kernel to
  * avoid undefined behaviour caused by sending INIT from AP to BSP.
  */
 static u32 disabled_cpu_apicid __ro_after_init = BAD_APICID;
 
-static unsigned int num_processors;
-static unsigned int disabled_cpus;
-
-/*
- * The number of allocated logical CPU IDs. Since logical CPU IDs are allocated
- * contiguously, it equals to current allocated max logical CPU ID plus 1.
- * All allocated CPU IDs should be in the [0, nr_logical_cpuids) range,
- * so the maximum of nr_logical_cpuids is nr_cpu_ids.
- *
- * NOTE: Reserve 0 for BSP.
- */
-static int nr_logical_cpuids = 1;
-
 bool arch_match_cpu_phys_id(int cpu, u64 phys_id)
 {
 	return phys_id == (u64)cpuid_to_apicid[cpu];
@@ -75,7 +74,7 @@ static int __init smp_init_primary_threa
 		return 0;
 	}
 
-	for (cpu = 0; cpu < nr_logical_cpuids; cpu++)
+	for (cpu = 0; cpu < topo_info.nr_assigned_cpus; cpu++)
 		cpu_mark_primary_thread(cpu, cpuid_to_apicid[cpu]);
 	return 0;
 }
@@ -89,7 +88,7 @@ static int topo_lookup_cpuid(u32 apic_id
 	int i;
 
 	/* CPU# to APICID mapping is persistent once it is established */
-	for (i = 0; i < nr_logical_cpuids; i++) {
+	for (i = 0; i < topo_info.nr_assigned_cpus; i++) {
 		if (cpuid_to_apicid[i] == apic_id)
 			return i;
 	}
@@ -107,22 +106,21 @@ static int allocate_logical_cpuid(u32 ap
 	if (cpu >= 0)
 		return cpu;
 
-	cpuid_to_apicid[nr_logical_cpuids] = apic_id;
-	return nr_logical_cpuids++;
+	return topo_info.nr_assigned_cpus++;
 }
 
-static void cpu_update_apic(int cpu, u32 apicid)
+static void cpu_update_apic(unsigned int cpu, u32 apic_id)
 {
 #if defined(CONFIG_SMP) || defined(CONFIG_X86_64)
-	early_per_cpu(x86_cpu_to_apicid, cpu) = apicid;
+	early_per_cpu(x86_cpu_to_apicid, cpu) = apic_id;
 #endif
+	cpuid_to_apicid[cpu] = apic_id;
 	set_cpu_possible(cpu, true);
-	set_bit(apicid, phys_cpu_present_map);
+	set_bit(apic_id, phys_cpu_present_map);
 	set_cpu_present(cpu, true);
-	num_processors++;
 
 	if (system_state != SYSTEM_BOOTING)
-		cpu_mark_primary_thread(cpu, apicid);
+		cpu_mark_primary_thread(cpu, apic_id);
 }
 
 static int generic_processor_info(int apicid)
@@ -137,18 +135,18 @@ static int generic_processor_info(int ap
 		return 0;
 
 	if (disabled_cpu_apicid == apicid) {
-		int thiscpu = num_processors + disabled_cpus;
+		int thiscpu = topo_info.nr_assigned_cpus + topo_info.nr_disabled_cpus;
 
 		pr_warn("APIC: Disabling requested cpu. Processor %d/0x%x ignored.\n",
 			thiscpu, apicid);
 
-		disabled_cpus++;
+		topo_info.nr_rejected_cpus++;
 		return -ENODEV;
 	}
 
-	if (num_processors >= nr_cpu_ids) {
+	if (topo_info.nr_assigned_cpus >= nr_cpu_ids) {
 		pr_warn_once("APIC: CPU limit of %d reached. Ignoring further CPUs\n", nr_cpu_ids);
-		disabled_cpus++;
+		topo_info.nr_rejected_cpus++;
 		return -ENOSPC;
 	}
 
@@ -178,14 +176,16 @@ static int __initdata setup_possible_cpu
  */
 __init void prefill_possible_map(void)
 {
+	unsigned int num_processors = topo_info.nr_assigned_cpus;
+	unsigned int disabled_cpus = topo_info.nr_disabled_cpus;
 	int i, possible;
 
 	i = setup_max_cpus ?: 1;
 	if (setup_possible_cpus == -1) {
-		possible = num_processors;
+		possible = topo_info.nr_assigned_cpus;
 #ifdef CONFIG_HOTPLUG_CPU
 		if (setup_max_cpus)
-			possible += disabled_cpus;
+			possible += num_processors;
 #else
 		if (possible > i)
 			possible = i;
@@ -238,7 +238,7 @@ void __init topology_register_apic(u32 a
 	}
 
 	if (!present) {
-		disabled_cpus++;
+		topo_info.nr_disabled_cpus++;
 		return;
 	}
 
@@ -295,7 +295,6 @@ void topology_hotunplug_apic(unsigned in
 	per_cpu(x86_cpu_to_apicid, cpu) = BAD_APICID;
 	clear_bit(apic_id, phys_cpu_present_map);
 	set_cpu_present(cpu, false);
-	num_processors--;
 }
 #endif
 

[patch 34/53] x86/smpboot: Make error message actually useful

[Thomas Gleixner]

"smpboot: native_kick_ap: bad cpu 33" is absolutely useless information.

Replace it with something meaningful which allows to decode the failure
condition.

Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
---
 arch/x86/kernel/smpboot.c |   10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)

--- a/arch/x86/kernel/smpboot.c
+++ b/arch/x86/kernel/smpboot.c
@@ -1081,9 +1081,13 @@ int native_kick_ap(unsigned int cpu, str
 
 	pr_debug("++++++++++++++++++++=_---CPU UP  %u\n", cpu);
 
-	if (apicid == BAD_APICID || !test_bit(apicid, phys_cpu_present_map) ||
-	    !apic_id_valid(apicid)) {
-		pr_err("%s: bad cpu %d\n", __func__, cpu);
+	if (apicid == BAD_APICID || !apic_id_valid(apicid)) {
+		pr_err("CPU %u has invalid APIC ID %x. Aborting bringup\n", cpu, apicid);
+		return -EINVAL;
+	}
+
+	if (!test_bit(apicid, phys_cpu_present_map)) {
+		pr_err("CPU %u APIC ID %x is not present. Aborting bringup\n", cpu, apicid);
 		return -EINVAL;
 	}
 

[patch 35/53] x86/cpu/topology: Sanitize the APIC admission logic

[Thomas Gleixner]

Move the actually required content of generic_processor_id() into the call
sites and use common helper functions for them. This separates the early
boot registration and the ACPI hotplug mechanism completely which allows
further cleanups and improvements.

Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
---
 arch/x86/kernel/cpu/topology.c |  160 +++++++++++++++++-------------------
 1 file changed, 78 insertions(+), 82 deletions(-)

--- a/arch/x86/kernel/cpu/topology.c
+++ b/arch/x86/kernel/cpu/topology.c
@@ -30,8 +30,10 @@ struct {
 	unsigned int		nr_assigned_cpus;
 	unsigned int		nr_disabled_cpus;
 	unsigned int		nr_rejected_cpus;
+	u32			boot_cpu_apic_id;
 } topo_info __read_mostly = {
 	.nr_assigned_cpus	= 1,
+	.boot_cpu_apic_id	= BAD_APICID,
 };
 
 /*
@@ -78,78 +80,6 @@ early_initcall(smp_init_primary_thread_m
 static inline void cpu_mark_primary_thread(unsigned int cpu, unsigned int apicid) { }
 #endif
 
-static int topo_lookup_cpuid(u32 apic_id)
-{
-	int i;
-
-	/* CPU# to APICID mapping is persistent once it is established */
-	for (i = 0; i < topo_info.nr_assigned_cpus; i++) {
-		if (cpuid_to_apicid[i] == apic_id)
-			return i;
-	}
-	return -ENODEV;
-}
-
-/*
- * Should use this API to allocate logical CPU IDs to keep nr_logical_cpuids
- * and cpuid_to_apicid[] synchronized.
- */
-static int allocate_logical_cpuid(u32 apic_id)
-{
-	int cpu = topo_lookup_cpuid(apic_id);
-
-	if (cpu >= 0)
-		return cpu;
-
-	return topo_info.nr_assigned_cpus++;
-}
-
-static void cpu_update_apic(unsigned int cpu, u32 apic_id)
-{
-#if defined(CONFIG_SMP) || defined(CONFIG_X86_64)
-	early_per_cpu(x86_cpu_to_apicid, cpu) = apic_id;
-#endif
-	cpuid_to_apicid[cpu] = apic_id;
-	set_cpu_possible(cpu, true);
-	set_bit(apic_id, phys_cpu_present_map);
-	set_cpu_present(cpu, true);
-
-	if (system_state != SYSTEM_BOOTING)
-		cpu_mark_primary_thread(cpu, apic_id);
-}
-
-static int generic_processor_info(int apicid)
-{
-	int cpu;
-
-	/* The boot CPU must be set before MADT/MPTABLE parsing happens */
-	if (cpuid_to_apicid[0] == BAD_APICID)
-		panic("Boot CPU APIC not registered yet\n");
-
-	if (apicid == boot_cpu_physical_apicid)
-		return 0;
-
-	if (disabled_cpu_apicid == apicid) {
-		int thiscpu = topo_info.nr_assigned_cpus + topo_info.nr_disabled_cpus;
-
-		pr_warn("APIC: Disabling requested cpu. Processor %d/0x%x ignored.\n",
-			thiscpu, apicid);
-
-		topo_info.nr_rejected_cpus++;
-		return -ENODEV;
-	}
-
-	if (topo_info.nr_assigned_cpus >= nr_cpu_ids) {
-		pr_warn_once("APIC: CPU limit of %d reached. Ignoring further CPUs\n", nr_cpu_ids);
-		topo_info.nr_rejected_cpus++;
-		return -ENOSPC;
-	}
-
-	cpu = allocate_logical_cpuid(apicid);
-	cpu_update_apic(cpu, apicid);
-	return cpu;
-}
-
 static int __initdata setup_possible_cpus = -1;
 
 /*
@@ -217,6 +147,43 @@ static int __initdata setup_possible_cpu
 		set_cpu_possible(i, true);
 }
 
+static int topo_lookup_cpuid(u32 apic_id)
+{
+	int i;
+
+	/* CPU# to APICID mapping is persistent once it is established */
+	for (i = 0; i < topo_info.nr_assigned_cpus; i++) {
+		if (cpuid_to_apicid[i] == apic_id)
+			return i;
+	}
+	return -ENODEV;
+}
+
+static int topo_assign_cpunr(u32 apic_id)
+{
+	int cpu = topo_lookup_cpuid(apic_id);
+
+	if (cpu >= 0)
+		return cpu;
+
+	return topo_info.nr_assigned_cpus++;
+}
+
+static void topo_set_cpuids(unsigned int cpu, u32 apic_id, u32 acpi_id)
+{
+#if defined(CONFIG_SMP) || defined(CONFIG_X86_64)
+	early_per_cpu(x86_cpu_to_apicid, cpu) = apic_id;
+	early_per_cpu(x86_cpu_to_acpiid, cpu) = acpi_id;
+#endif
+	cpuid_to_apicid[cpu] = apic_id;
+
+	set_cpu_possible(cpu, true);
+	set_cpu_present(cpu, true);
+
+	if (system_state != SYSTEM_BOOTING)
+		cpu_mark_primary_thread(cpu, apic_id);
+}
+
 /**
  * topology_register_apic - Register an APIC in early topology maps
  * @apic_id:	The APIC ID to set up
@@ -229,17 +196,41 @@ void __init topology_register_apic(u32 a
 
 	if (apic_id >= MAX_LOCAL_APIC) {
 		pr_err_once("APIC ID %x exceeds kernel limit of: %x\n", apic_id, MAX_LOCAL_APIC - 1);
+		topo_info.nr_rejected_cpus++;
 		return;
 	}
 
-	if (!present) {
-		topo_info.nr_disabled_cpus++;
+	/* CPU numbers exhausted? */
+	if (topo_info.nr_assigned_cpus >= nr_cpu_ids) {
+		pr_warn_once("CPU limit of %d reached. Ignoring further CPUs\n", nr_cpu_ids);
+		topo_info.nr_rejected_cpus++;
 		return;
 	}
 
-	cpu = generic_processor_info(apic_id);
-	if (cpu >= 0)
-		early_per_cpu(x86_cpu_to_acpiid, cpu) = acpi_id;
+	if (disabled_cpu_apicid == apic_id) {
+		pr_info("Disabling CPU as requested via 'disable_cpu_apicid=0x%x'.\n", apic_id);
+		topo_info.nr_rejected_cpus++;
+		return;
+	}
+
+	if (present) {
+		/*
+		 * Prevent double registration, which is valid in case of
+		 * the boot CPU APIC because that is registered before the
+		 * enumeration of the APICs via firmware parsers or VM
+		 * guest mechanisms.
+		 */
+		if (test_and_set_bit(apic_id, phys_cpu_present_map))
+			return;
+
+		if (apic_id == topo_info.boot_cpu_apic_id)
+			cpu = 0;
+		else
+			cpu = topo_assign_cpunr(apic_id);
+		topo_set_cpuids(cpu, apic_id, acpi_id);
+	} else {
+		topo_info.nr_disabled_cpus++;
+	}
 }
 
 /**
@@ -250,8 +241,10 @@ void __init topology_register_apic(u32 a
  */
 void __init topology_register_boot_apic(u32 apic_id)
 {
-	cpuid_to_apicid[0] = apic_id;
-	cpu_update_apic(0, apic_id);
+	WARN_ON_ONCE(topo_info.boot_cpu_apic_id != BAD_APICID);
+
+	topo_info.boot_cpu_apic_id = apic_id;
+	topology_register_apic(apic_id, CPU_ACPIID_INVALID, true);
 }
 
 #ifdef CONFIG_ACPI_HOTPLUG_CPU
@@ -269,10 +262,13 @@ int topology_hotplug_apic(u32 apic_id, u
 
 	cpu = topo_lookup_cpuid(apic_id);
 	if (cpu < 0) {
-		cpu = generic_processor_info(apic_id);
-		if (cpu >= 0)
-			per_cpu(x86_cpu_to_acpiid, cpu) = acpi_id;
+		if (topo_info.nr_assigned_cpus >= nr_cpu_ids)
+			return -ENOSPC;
+
+		cpu = topo_assign_cpunr(apic_id);
 	}
+	set_bit(apic_id, phys_cpu_present_map);
+	topo_set_cpuids(cpu, apic_id, acpi_id);
 	return cpu;
 }
 

[patch 36/53] x86/cpu/topology: Rework possible CPU management

[Thomas Gleixner]

Managing possible CPUs is an unreadable and uncomprehensible maze. Aside of
that it's backwards because it applies command line limits after
registering all APICs.

Rewrite it so that it:

  - Applies the command line limits upfront so that only the allowed amount
    of APIC IDs can be registered.

  - Applies eventual late restrictions in an understandable way

  - Uses simple min_t() calculations which are trivial to follow.

  - Provides a separate function for resetting to UP mode late in the
    bringup process.

Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
---
 arch/x86/include/asm/apic.h     |    5 +
 arch/x86/include/asm/topology.h |    4 
 arch/x86/kernel/cpu/topology.c  |  176 ++++++++++++++++++++++++----------------
 arch/x86/kernel/setup.c         |    9 --
 arch/x86/kernel/smpboot.c       |    6 -
 5 files changed, 120 insertions(+), 80 deletions(-)

--- a/arch/x86/include/asm/apic.h
+++ b/arch/x86/include/asm/apic.h
@@ -175,6 +175,9 @@ extern void topology_register_apic(u32 a
 extern void topology_register_boot_apic(u32 apic_id);
 extern int topology_hotplug_apic(u32 apic_id, u32 acpi_id);
 extern void topology_hotunplug_apic(unsigned int cpu);
+extern void topology_apply_cmdline_limits_early(void);
+extern void topology_init_possible_cpus(void);
+extern void topology_reset_possible_cpus_up(void);
 
 #else /* !CONFIG_X86_LOCAL_APIC */
 static inline void lapic_shutdown(void) { }
@@ -190,6 +193,8 @@ static inline void apic_intr_mode_init(v
 static inline void lapic_assign_system_vectors(void) { }
 static inline void lapic_assign_legacy_vector(unsigned int i, bool r) { }
 static inline bool apic_needs_pit(void) { return true; }
+static inline void topology_apply_cmdline_limits_early(void) { }
+static inline void topology_init_possible_cpus(void) { }
 #endif /* !CONFIG_X86_LOCAL_APIC */
 
 #ifdef CONFIG_X86_X2APIC
--- a/arch/x86/include/asm/topology.h
+++ b/arch/x86/include/asm/topology.h
@@ -190,6 +190,9 @@ static inline bool topology_is_primary_t
 {
 	return cpumask_test_cpu(cpu, cpu_primary_thread_mask);
 }
+
+void topology_apply_cmdline_limits_early(void);
+
 #else /* CONFIG_SMP */
 #define topology_max_packages()			(1)
 static inline int
@@ -202,6 +205,7 @@ static inline int topology_max_smt_threa
 static inline bool topology_is_primary_thread(unsigned int cpu) { return true; }
 static inline bool topology_smt_supported(void) { return false; }
 static inline unsigned int topology_amd_nodes_per_pkg(void) { return 0; };
+static inline void topology_apply_cmdline_limits_early(void) { }
 #endif /* !CONFIG_SMP */
 
 static inline void arch_fix_phys_package_id(int num, u32 slot)
--- a/arch/x86/kernel/cpu/topology.c
+++ b/arch/x86/kernel/cpu/topology.c
@@ -5,6 +5,7 @@
 #include <xen/xen.h>
 
 #include <asm/apic.h>
+#include <asm/io_apic.h>
 #include <asm/mpspec.h>
 #include <asm/smp.h>
 
@@ -85,73 +86,6 @@ early_initcall(smp_init_primary_thread_m
 static inline void cpu_mark_primary_thread(unsigned int cpu, unsigned int apicid) { }
 #endif
 
-static int __initdata setup_possible_cpus = -1;
-
-/*
- * cpu_possible_mask should be static, it cannot change as cpu's
- * are onlined, or offlined. The reason is per-cpu data-structures
- * are allocated by some modules at init time, and don't expect to
- * do this dynamically on cpu arrival/departure.
- * cpu_present_mask on the other hand can change dynamically.
- * In case when cpu_hotplug is not compiled, then we resort to current
- * behaviour, which is cpu_possible == cpu_present.
- * - Ashok Raj
- *
- * Three ways to find out the number of additional hotplug CPUs:
- * - If the BIOS specified disabled CPUs in ACPI/mptables use that.
- * - The user can overwrite it with possible_cpus=NUM
- * - Otherwise don't reserve additional CPUs.
- * We do this because additional CPUs waste a lot of memory.
- * -AK
- */
-__init void prefill_possible_map(void)
-{
-	unsigned int num_processors = topo_info.nr_assigned_cpus;
-	unsigned int disabled_cpus = topo_info.nr_disabled_cpus;
-	int i, possible;
-
-	i = setup_max_cpus ?: 1;
-	if (setup_possible_cpus == -1) {
-		possible = topo_info.nr_assigned_cpus;
-#ifdef CONFIG_HOTPLUG_CPU
-		if (setup_max_cpus)
-			possible += num_processors;
-#else
-		if (possible > i)
-			possible = i;
-#endif
-	} else
-		possible = setup_possible_cpus;
-
-	total_cpus = max_t(int, possible, num_processors + disabled_cpus);
-
-	/* nr_cpu_ids could be reduced via nr_cpus= */
-	if (possible > nr_cpu_ids) {
-		pr_warn("%d Processors exceeds NR_CPUS limit of %u\n",
-			possible, nr_cpu_ids);
-		possible = nr_cpu_ids;
-	}
-
-#ifdef CONFIG_HOTPLUG_CPU
-	if (!setup_max_cpus)
-#endif
-	if (possible > i) {
-		pr_warn("%d Processors exceeds max_cpus limit of %u\n",
-			possible, setup_max_cpus);
-		possible = i;
-	}
-
-	set_nr_cpu_ids(possible);
-
-	pr_info("Allowing %d CPUs, %d hotplug CPUs\n",
-		possible, max_t(int, possible - num_processors, 0));
-
-	reset_cpu_possible_mask();
-
-	for (i = 0; i < possible; i++)
-		set_cpu_possible(i, true);
-}
-
 static int topo_lookup_cpuid(u32 apic_id)
 {
 	int i;
@@ -294,12 +228,114 @@ void topology_hotunplug_apic(unsigned in
 }
 #endif
 
-static int __init _setup_possible_cpus(char *str)
+#ifdef CONFIG_SMP
+static unsigned int max_possible_cpus __initdata = NR_CPUS;
+
+/**
+ * topology_apply_cmdline_limits_early - Apply topology command line limits early
+ *
+ * Ensure that command line limits are in effect before firmware parsing
+ * takes place.
+ */
+void __init topology_apply_cmdline_limits_early(void)
+{
+	unsigned int possible = nr_cpu_ids;
+
+	/* 'maxcpus=0' 'nosmp' 'nolapic' 'disableapic' 'noapic' */
+	if (!setup_max_cpus || ioapic_is_disabled || apic_is_disabled)
+		possible = 1;
+
+	/* 'possible_cpus=N' */
+	possible = min_t(unsigned int, max_possible_cpus, possible);
+
+	if (possible < nr_cpu_ids) {
+		pr_info("Limiting to %u possible CPUs\n", possible);
+		set_nr_cpu_ids(possible);
+	}
+}
+
+static __init bool restrict_to_up(void)
+{
+	if (!smp_found_config || ioapic_is_disabled)
+		return true;
+	/*
+	 * XEN PV is special as it does not advertise the local APIC
+	 * properly, but provides a fake topology for it so that the
+	 * infrastructure works. So don't apply the restrictions vs. APIC
+	 * here.
+	 */
+	if (xen_pv_domain())
+		return false;
+
+	return apic_is_disabled;
+}
+
+void __init topology_init_possible_cpus(void)
+{
+	unsigned int assigned = topo_info.nr_assigned_cpus;
+	unsigned int disabled = topo_info.nr_disabled_cpus;
+	unsigned int total = assigned + disabled;
+	unsigned int cpu, allowed = 1;
+
+	if (!restrict_to_up()) {
+		if (WARN_ON_ONCE(assigned > nr_cpu_ids)) {
+			disabled += assigned - nr_cpu_ids;
+			assigned = nr_cpu_ids;
+		}
+		allowed = min_t(unsigned int, total, nr_cpu_ids);
+	}
+
+	if (total > allowed)
+		pr_warn("%u possible CPUs exceed the limit of %u\n", total, allowed);
+
+	assigned = min_t(unsigned int, allowed, assigned);
+	disabled = allowed - assigned;
+
+	topo_info.nr_assigned_cpus = assigned;
+	topo_info.nr_disabled_cpus = disabled;
+
+	total_cpus = allowed;
+	set_nr_cpu_ids(allowed);
+
+	pr_info("Allowing %u present CPUs plus %u hotplug CPUs\n", assigned, disabled);
+	if (topo_info.nr_rejected_cpus)
+		pr_info("Rejected CPUs %u\n", topo_info.nr_rejected_cpus);
+
+	init_cpu_present(cpumask_of(0));
+	init_cpu_possible(cpumask_of(0));
+
+	for (cpu = 0; cpu < allowed; cpu++) {
+		u32 apicid = cpuid_to_apicid[cpu];
+
+		set_cpu_possible(cpu, true);
+
+		if (apicid == BAD_APICID)
+			continue;
+
+		set_cpu_present(cpu, test_bit(apicid, phys_cpu_present_map));
+	}
+}
+
+/*
+ * Late SMP disable after sizing CPU masks when APIC/IOAPIC setup failed.
+ */
+void __init topology_reset_possible_cpus_up(void)
 {
-	get_option(&str, &setup_possible_cpus);
+	init_cpu_present(cpumask_of(0));
+	init_cpu_possible(cpumask_of(0));
+
+	bitmap_zero(phys_cpu_present_map, MAX_LOCAL_APIC);
+	if (topo_info.boot_cpu_apic_id != BAD_APICID)
+		set_bit(topo_info.boot_cpu_apic_id, phys_cpu_present_map);
+}
+
+static int __init setup_possible_cpus(char *str)
+{
+	get_option(&str, &max_possible_cpus);
 	return 0;
 }
-early_param("possible_cpus", _setup_possible_cpus);
+early_param("possible_cpus", setup_possible_cpus);
+#endif
 
 static int __init apic_set_disabled_cpu_apicid(char *arg)
 {
--- a/arch/x86/kernel/setup.c
+++ b/arch/x86/kernel/setup.c
@@ -1258,6 +1258,8 @@ void __init setup_arch(char **cmdline_p)
 
 	early_quirks();
 
+	topology_apply_cmdline_limits_early();
+
 	/*
 	 * Parse SMP configuration. Try ACPI first and then the platform
 	 * specific parser.
@@ -1265,13 +1267,10 @@ void __init setup_arch(char **cmdline_p)
 	acpi_boot_init();
 	x86_init.mpparse.parse_smp_cfg();
 
-	/*
-	 * Systems w/o ACPI and mptables might not have it mapped the local
-	 * APIC yet, but prefill_possible_map() might need to access it.
-	 */
+	/* Last opportunity to detect and map the local APIC */
 	init_apic_mappings();
 
-	prefill_possible_map();
+	topology_init_possible_cpus();
 
 	init_cpu_to_node();
 	init_gi_nodes();
--- a/arch/x86/kernel/smpboot.c
+++ b/arch/x86/kernel/smpboot.c
@@ -1156,11 +1156,7 @@ static __init void disable_smp(void)
 	pr_info("SMP disabled\n");
 
 	disable_ioapic_support();
-
-	init_cpu_present(cpumask_of(0));
-	init_cpu_possible(cpumask_of(0));
-
-	reset_phys_cpu_present_map(smp_found_config ? boot_cpu_physical_apicid : 0);
+	topology_reset_possible_cpus_up();
 
 	cpumask_set_cpu(0, topology_sibling_cpumask(0));
 	cpumask_set_cpu(0, topology_core_cpumask(0));

[patch 37/53] x86/cpu: Detect real BSP on crash kernels

[Thomas Gleixner]

When a kdump kernel is started from a crashing CPU then there is no
guarantee that this CPU is the real boot CPU (BSP). If the kdump kernel
tries to online the BSP then the INIT sequence will reset the machine.

There is a command line option to prevent this, but in case of nested kdump
kernels this is wrong.

But that command line option is not required at all because the real BSP
has the lowest local APIC ID in the system. That was not always true, but
support for the only known system which was different (Voyager) got removed
long ago.

Detect whether the boot CPU APIC ID is the lowest APIC ID in the system.
If the lowest registered APIC ID is not the boot CPU APIC ID, then remove
it from the present bitmap and let the possible map initialization ignore
it.

Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
---
 Documentation/admin-guide/kdump/kdump.rst       |    7 --
 Documentation/admin-guide/kernel-parameters.txt |    9 ---
 arch/x86/kernel/cpu/topology.c                  |   59 +++++++++++++++---------
 3 files changed, 39 insertions(+), 36 deletions(-)

--- a/Documentation/admin-guide/kdump/kdump.rst
+++ b/Documentation/admin-guide/kdump/kdump.rst
@@ -191,9 +191,7 @@ Dump-capture kernel config options (Arch
    CPU is enough for kdump kernel to dump vmcore on most of systems.
 
    However, you can also specify nr_cpus=X to enable multiple processors
-   in kdump kernel. In this case, "disable_cpu_apicid=" is needed to
-   tell kdump kernel which cpu is 1st kernel's BSP. Please refer to
-   admin-guide/kernel-parameters.txt for more details.
+   in kdump kernel.
 
    With CONFIG_SMP=n, the above things are not related.
 
@@ -485,8 +483,7 @@ loading dump-capture kernel.
   to use multi-thread programs with it, such as parallel dump feature of
   makedumpfile. Otherwise, the multi-thread program may have a great
   performance degradation. To enable multi-cpu support, you should bring up an
-  SMP dump-capture kernel and specify maxcpus/nr_cpus, disable_cpu_apicid=[X]
-  options while loading it.
+  SMP dump-capture kernel and specify maxcpus/nr_cpus options while loading it.
 
 * For s390x there are two kdump modes: If a ELF header is specified with
   the elfcorehdr= kernel parameter, it is used by the kdump kernel as it
--- a/Documentation/admin-guide/kernel-parameters.txt
+++ b/Documentation/admin-guide/kernel-parameters.txt
@@ -1071,15 +1071,6 @@
 			Disable TLBIE instruction. Currently does not work
 			with KVM, with HASH MMU, or with coherent accelerators.
 
-	disable_cpu_apicid= [X86,APIC,SMP]
-			Format: <int>
-			The number of initial APIC ID for the
-			corresponding CPU to be disabled at boot,
-			mostly used for the kdump 2nd kernel to
-			disable BSP to wake up multiple CPUs without
-			causing system reset or hang due to sending
-			INIT from AP to BSP.
-
 	disable_ddw	[PPC/PSERIES]
 			Disable Dynamic DMA Window support. Use this
 			to workaround buggy firmware.
--- a/arch/x86/kernel/cpu/topology.c
+++ b/arch/x86/kernel/cpu/topology.c
@@ -32,18 +32,13 @@ struct {
 	unsigned int		nr_disabled_cpus;
 	unsigned int		nr_rejected_cpus;
 	u32			boot_cpu_apic_id;
+	u32			real_bsp_apic_id;
 } topo_info __read_mostly = {
 	.nr_assigned_cpus	= 1,
 	.boot_cpu_apic_id	= BAD_APICID,
+	.real_bsp_apic_id	= BAD_APICID,
 };
 
-/*
- * Processor to be disabled specified by kernel parameter
- * disable_cpu_apicid=<int>, mostly used for the kdump 2nd kernel to
- * avoid undefined behaviour caused by sending INIT from AP to BSP.
- */
-static u32 disabled_cpu_apicid __ro_after_init = BAD_APICID;
-
 bool arch_match_cpu_phys_id(int cpu, u64 phys_id)
 {
 	return phys_id == (u64)cpuid_to_apicid[cpu];
@@ -146,12 +141,6 @@ void __init topology_register_apic(u32 a
 		return;
 	}
 
-	if (disabled_cpu_apicid == apic_id) {
-		pr_info("Disabling CPU as requested via 'disable_cpu_apicid=0x%x'.\n", apic_id);
-		topo_info.nr_rejected_cpus++;
-		return;
-	}
-
 	if (present) {
 		/*
 		 * Prevent double registration, which is valid in case of
@@ -270,6 +259,30 @@ static __init bool restrict_to_up(void)
 	return apic_is_disabled;
 }
 
+static __init void check_for_kdump_kernel(void)
+{
+	u32 bsp_apicid;
+
+	/*
+	 * There is no real good way to detect whether this a kdump()
+	 * kernel, but except on the Voyager SMP monstrosity which is not
+	 * longer supported, the real BSP has always the lowest numbered
+	 * APIC ID. If a crash happened on an AP, which then ends up as
+	 * boot CPU in the kdump() kernel, then sending INIT to the real
+	 * BSP would reset the whole system.
+	 */
+	bsp_apicid = find_first_bit(phys_cpu_present_map, MAX_LOCAL_APIC);
+	if (bsp_apicid == topo_info.boot_cpu_apic_id)
+		return;
+
+	pr_warn("Boot CPU APIC ID not the lowest APIC ID: %x > %x\n",
+		topo_info.boot_cpu_apic_id, bsp_apicid);
+	pr_warn("Crash kernel detected. Disabling real BSP to prevent machine INIT\n");
+
+	topo_info.real_bsp_apic_id = bsp_apicid;
+	clear_bit(bsp_apicid, phys_cpu_present_map);
+}
+
 void __init topology_init_possible_cpus(void)
 {
 	unsigned int assigned = topo_info.nr_assigned_cpus;
@@ -278,6 +291,9 @@ void __init topology_init_possible_cpus(
 	unsigned int cpu, allowed = 1;
 
 	if (!restrict_to_up()) {
+		if (total > 1)
+			check_for_kdump_kernel();
+
 		if (WARN_ON_ONCE(assigned > nr_cpu_ids)) {
 			disabled += assigned - nr_cpu_ids;
 			assigned = nr_cpu_ids;
@@ -308,6 +324,14 @@ void __init topology_init_possible_cpus(
 	for (cpu = 0; cpu < allowed; cpu++) {
 		u32 apicid = cpuid_to_apicid[cpu];
 
+		/*
+		 * In case of a kdump() kernel, don't mark the real BSP in
+		 * the present and possible maps. Sending INIT to it resets
+		 * the machine.
+		 */
+		if (apicid != BAD_APICID && apicid == topo_info.real_bsp_apic_id)
+			continue;
+
 		set_cpu_possible(cpu, true);
 
 		if (apicid == BAD_APICID)
@@ -337,12 +361,3 @@ static int __init setup_possible_cpus(ch
 }
 early_param("possible_cpus", setup_possible_cpus);
 #endif
-
-static int __init apic_set_disabled_cpu_apicid(char *arg)
-{
-	if (!arg || !get_option(&arg, &disabled_cpu_apicid))
-		return -EINVAL;
-
-	return 0;
-}
-early_param("disable_cpu_apicid", apic_set_disabled_cpu_apicid);

[patch 38/53] x86/topology: Add a mechanism to track topology via APIC IDs

[Thomas Gleixner]

Topology on X86 is determined by the registered APIC IDs and the
segmentation information retrieved from CPUID. Depending on the granularity
of the provided CPUID information the most fine grained scheme looks like
this according to Intel terminology:

   [PKG][DIE][TILE][MODULE][CORE][THREAD]

Not enumerated domain levels consume 0 bits in the APIC ID. This allows to
provide a consistent view at the topology and determine other information
precisely like the number of cores in a package on hybrid systems, where
the existing assumption that number or cores == number of threads / threads
per core does not hold.

Provide per domain level bitmaps which record the APIC ID split into the
domain levels to make later evaluation of domain level specific information
simple. This allows to calculate e.g. the logical IDs without any further
extra logic.

Contrary to the existing registration mechanism this records disabled CPUs,
which are subject to later hotplug as well. That's useful for boot time
sizing of package or die dependent allocations without using heuristics.

Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
---
 arch/x86/kernel/cpu/debugfs.c  |    8 ----
 arch/x86/kernel/cpu/topology.c |   68 +++++++++++++++++++++++++++++++++++++++--
 arch/x86/kernel/cpu/topology.h |    2 +
 3 files changed, 67 insertions(+), 11 deletions(-)

--- a/arch/x86/kernel/cpu/debugfs.c
+++ b/arch/x86/kernel/cpu/debugfs.c
@@ -48,14 +48,6 @@ static const struct file_operations dfs_
 
 static int dom_debug_show(struct seq_file *m, void *p)
 {
-	static const char *domain_names[TOPO_ROOT_DOMAIN] = {
-		[TOPO_SMT_DOMAIN]	= "Thread",
-		[TOPO_CORE_DOMAIN]	= "Core",
-		[TOPO_MODULE_DOMAIN]	= "Module",
-		[TOPO_TILE_DOMAIN]	= "Tile",
-		[TOPO_DIE_DOMAIN]	= "Die",
-		[TOPO_PKG_DOMAIN]	= "Package",
-	};
 	unsigned int dom, nthreads = 1;
 
 	for (dom = 0; dom < TOPO_ROOT_DOMAIN; dom++) {
--- a/arch/x86/kernel/cpu/topology.c
+++ b/arch/x86/kernel/cpu/topology.c
@@ -1,5 +1,27 @@
 // SPDX-License-Identifier: GPL-2.0-only
-
+/*
+ * CPU/APIC topology
+ *
+ * The APIC IDs describe the system topology in multiple domain levels.
+ * The CPUID topology parser provides the information which part of the
+ * APIC ID is associated to the individual levels:
+ *
+ * [ROOT][PACKAGE][DIE][TILE][MODULE][CORE][THREAD]
+ *
+ * The root space contains the package (socket) IDs.
+ *
+ * Not enumerated levels consume 0 bits space, but conceptually they are
+ * always represented. If e.g. only CORE and THREAD levels are enumerated
+ * then the DIE, MODULE and TILE have the same physical ID as the PACKAGE.
+ *
+ * If SMT is not supported, then the THREAD domain is still used. It then
+ * has the same physical ID as the CORE domain and is the only child of
+ * the core domain.
+ *
+ * This allows a unified view on the system independent of the enumerated
+ * domain levels without requiring any conditionals in the code.
+ */
+#define pr_fmt(fmt) "CPU topo: " fmt
 #include <linux/cpu.h>
 
 #include <xen/xen.h>
@@ -9,6 +31,8 @@
 #include <asm/mpspec.h>
 #include <asm/smp.h>
 
+#include "cpu.h"
+
 /*
  * Map cpu index to physical APIC ID
  */
@@ -23,6 +47,9 @@ DECLARE_BITMAP(phys_cpu_present_map, MAX
 /* Used for CPU number allocation and parallel CPU bringup */
 u32 cpuid_to_apicid[] __read_mostly = { [0 ... NR_CPUS - 1] = BAD_APICID, };
 
+/* Bitmaps to mark registered APICs at each topology domain */
+static struct { DECLARE_BITMAP(map, MAX_LOCAL_APIC); } apic_maps[TOPO_MAX_DOMAIN] __ro_after_init;
+
 /*
  * Keep track of assigned, disabled and rejected CPUs. Present assigned
  * with 1 as CPU #0 is reserved for the boot CPU.
@@ -39,6 +66,18 @@ struct {
 	.real_bsp_apic_id	= BAD_APICID,
 };
 
+const char *domain_names[TOPO_MAX_DOMAIN] = {
+	[TOPO_SMT_DOMAIN]	= "Thread",
+	[TOPO_CORE_DOMAIN]	= "Core",
+	[TOPO_MODULE_DOMAIN]	= "Module",
+	[TOPO_TILE_DOMAIN]	= "Tile",
+	[TOPO_DIE_DOMAIN]	= "Die",
+	[TOPO_PKG_DOMAIN]	= "Package",
+	[TOPO_ROOT_DOMAIN]	= "Root",
+};
+
+#define domain_weight(_dom)	bitmap_weight(apic_maps[_dom].map, MAX_LOCAL_APIC)
+
 bool arch_match_cpu_phys_id(int cpu, u64 phys_id)
 {
 	return phys_id == (u64)cpuid_to_apicid[cpu];
@@ -81,6 +120,17 @@ early_initcall(smp_init_primary_thread_m
 static inline void cpu_mark_primary_thread(unsigned int cpu, unsigned int apicid) { }
 #endif
 
+/*
+ * Convert the APIC ID to a domain level ID by masking out the low bits
+ * including the domain level @dom.
+ */
+static inline u32 topo_apicid(u32 apicid, enum x86_topology_domains dom)
+{
+	if (dom == TOPO_SMT_DOMAIN)
+		return apicid;
+	return apicid & (UINT_MAX << x86_topo_system.dom_shifts[dom - 1]);
+}
+
 static int topo_lookup_cpuid(u32 apic_id)
 {
 	int i;
@@ -126,7 +176,7 @@ static void topo_set_cpuids(unsigned int
  */
 void __init topology_register_apic(u32 apic_id, u32 acpi_id, bool present)
 {
-	int cpu;
+	int cpu, dom;
 
 	if (apic_id >= MAX_LOCAL_APIC) {
 		pr_err_once("APIC ID %x exceeds kernel limit of: %x\n", apic_id, MAX_LOCAL_APIC - 1);
@@ -159,6 +209,10 @@ void __init topology_register_apic(u32 a
 	} else {
 		topo_info.nr_disabled_cpus++;
 	}
+
+	/* Register present and possible CPUs in the domain maps */
+	for (dom = TOPO_SMT_DOMAIN; dom < TOPO_ROOT_DOMAIN; dom++)
+		set_bit(topo_apicid(apic_id, dom), apic_maps[dom].map);
 }
 
 /**
@@ -281,6 +335,11 @@ static __init void check_for_kdump_kerne
 
 	topo_info.real_bsp_apic_id = bsp_apicid;
 	clear_bit(bsp_apicid, phys_cpu_present_map);
+	/*
+	 * Remove it from the SMT level, but no propagation as that would
+	 * corrupt the data set.
+	 */
+	clear_bit(bsp_apicid, apic_maps[TOPO_SMT_DOMAIN].map);
 }
 
 void __init topology_init_possible_cpus(void)
@@ -288,7 +347,7 @@ void __init topology_init_possible_cpus(
 	unsigned int assigned = topo_info.nr_assigned_cpus;
 	unsigned int disabled = topo_info.nr_disabled_cpus;
 	unsigned int total = assigned + disabled;
-	unsigned int cpu, allowed = 1;
+	unsigned int cpu, dom, allowed = 1;
 
 	if (!restrict_to_up()) {
 		if (total > 1)
@@ -318,6 +377,9 @@ void __init topology_init_possible_cpus(
 	if (topo_info.nr_rejected_cpus)
 		pr_info("Rejected CPUs %u\n", topo_info.nr_rejected_cpus);
 
+	for (dom = TOPO_SMT_DOMAIN; dom < TOPO_ROOT_DOMAIN; dom++)
+		pr_info("%-10s: %5u\n", domain_names[dom], domain_weight(dom));
+
 	init_cpu_present(cpumask_of(0));
 	init_cpu_possible(cpumask_of(0));
 
--- a/arch/x86/kernel/cpu/topology.h
+++ b/arch/x86/kernel/cpu/topology.h
@@ -48,4 +48,6 @@ static inline void topology_update_dom(s
 	tscan->dom_ncpus[dom] = ncpus;
 }
 
+extern const char *domain_names[TOPO_MAX_DOMAIN];
+
 #endif /* ARCH_X86_TOPOLOGY_H */

[patch 39/53] x86/cpu/topology: Reject unknown APIC IDs on ACPI hotplug

[Thomas Gleixner]

The topology bitmaps track all possible APIC IDs which have been registered
during enumeration. As sizing and further topology information is going to
be derived from these bitmaps, reject attempts to hotplug an APIC ID which
was not registered during enumeration.

Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
---
 arch/x86/kernel/cpu/topology.c |    4 ++++
 1 file changed, 4 insertions(+)

--- a/arch/x86/kernel/cpu/topology.c
+++ b/arch/x86/kernel/cpu/topology.c
@@ -237,6 +237,10 @@ int topology_hotplug_apic(u32 apic_id, u
 	if (apic_id >= MAX_LOCAL_APIC)
 		return -EINVAL;
 
+	/* Reject if the APIC ID was not registered during enumeration. */
+	if (!test_bit(apic_id, apic_maps[TOPO_SMT_DOMAIN].map))
+		return -ENODEV;
+
 	cpu = topo_lookup_cpuid(apic_id);
 	if (cpu < 0) {
 		if (topo_info.nr_assigned_cpus >= nr_cpu_ids)

[patch 40/53] x86/cpu/topology: Assign hotpluggable CPUIDs during init

[Thomas Gleixner]

There is no point in assigning the CPU numbers during ACPI physical
hotplug. The number of possible hotplug CPUs is known when the possible map
is initialized, so the CPU numbers can be associated to the registered
non-present APIC IDs right there.

This allows to put more code into the __init section and makes the related
data __ro_after_init.

Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
---
 arch/x86/kernel/cpu/topology.c |   28 +++++++++++++++++-----------
 1 file changed, 17 insertions(+), 11 deletions(-)

--- a/arch/x86/kernel/cpu/topology.c
+++ b/arch/x86/kernel/cpu/topology.c
@@ -45,7 +45,7 @@ EXPORT_EARLY_PER_CPU_SYMBOL(x86_cpu_to_a
 DECLARE_BITMAP(phys_cpu_present_map, MAX_LOCAL_APIC) __read_mostly;
 
 /* Used for CPU number allocation and parallel CPU bringup */
-u32 cpuid_to_apicid[] __read_mostly = { [0 ... NR_CPUS - 1] = BAD_APICID, };
+u32 cpuid_to_apicid[] __ro_after_init = { [0 ... NR_CPUS - 1] = BAD_APICID, };
 
 /* Bitmaps to mark registered APICs at each topology domain */
 static struct { DECLARE_BITMAP(map, MAX_LOCAL_APIC); } apic_maps[TOPO_MAX_DOMAIN] __ro_after_init;
@@ -60,7 +60,7 @@ struct {
 	unsigned int		nr_rejected_cpus;
 	u32			boot_cpu_apic_id;
 	u32			real_bsp_apic_id;
-} topo_info __read_mostly = {
+} topo_info __ro_after_init = {
 	.nr_assigned_cpus	= 1,
 	.boot_cpu_apic_id	= BAD_APICID,
 	.real_bsp_apic_id	= BAD_APICID,
@@ -143,7 +143,7 @@ static int topo_lookup_cpuid(u32 apic_id
 	return -ENODEV;
 }
 
-static int topo_assign_cpunr(u32 apic_id)
+static __init int topo_assign_cpunr(u32 apic_id)
 {
 	int cpu = topo_lookup_cpuid(apic_id);
 
@@ -159,8 +159,6 @@ static void topo_set_cpuids(unsigned int
 	early_per_cpu(x86_cpu_to_apicid, cpu) = apic_id;
 	early_per_cpu(x86_cpu_to_acpiid, cpu) = acpi_id;
 #endif
-	cpuid_to_apicid[cpu] = apic_id;
-
 	set_cpu_possible(cpu, true);
 	set_cpu_present(cpu, true);
 
@@ -205,6 +203,8 @@ void __init topology_register_apic(u32 a
 			cpu = 0;
 		else
 			cpu = topo_assign_cpunr(apic_id);
+
+		cpuid_to_apicid[cpu] = apic_id;
 		topo_set_cpuids(cpu, apic_id, acpi_id);
 	} else {
 		topo_info.nr_disabled_cpus++;
@@ -247,12 +247,9 @@ int topology_hotplug_apic(u32 apic_id, u
 		return -ENODEV;
 
 	cpu = topo_lookup_cpuid(apic_id);
-	if (cpu < 0) {
-		if (topo_info.nr_assigned_cpus >= nr_cpu_ids)
-			return -ENOSPC;
+	if (cpu < 0)
+		return -ENOSPC;
 
-		cpu = topo_assign_cpunr(apic_id);
-	}
 	set_bit(apic_id, phys_cpu_present_map);
 	topo_set_cpuids(cpu, apic_id, acpi_id);
 	return cpu;
@@ -352,6 +349,7 @@ void __init topology_init_possible_cpus(
 	unsigned int disabled = topo_info.nr_disabled_cpus;
 	unsigned int total = assigned + disabled;
 	unsigned int cpu, dom, allowed = 1;
+	u32 apicid;
 
 	if (!restrict_to_up()) {
 		if (total > 1)
@@ -386,8 +384,16 @@ void __init topology_init_possible_cpus(
 	init_cpu_present(cpumask_of(0));
 	init_cpu_possible(cpumask_of(0));
 
+	for (apicid = 0; disabled; disabled--, apicid++) {
+		apicid = find_next_andnot_bit(apic_maps[TOPO_SMT_DOMAIN].map, phys_cpu_present_map,
+					      MAX_LOCAL_APIC, apicid);
+		if (apicid >= MAX_LOCAL_APIC)
+			break;
+		cpuid_to_apicid[topo_info.nr_assigned_cpus++] = apicid;
+	}
+
 	for (cpu = 0; cpu < allowed; cpu++) {
-		u32 apicid = cpuid_to_apicid[cpu];
+		apicid = cpuid_to_apicid[cpu];
 
 		/*
 		 * In case of a kdump() kernel, don't mark the real BSP in

[patch 41/53] x86/xen/smp_pv: Count number of vCPUs early

[Thomas Gleixner]

XEN/PV has a completely broken vCPU enumeration scheme, which just works by
chance and provides zero topology information. Each vCPU ends up being a
single core package.

Dom0 provides MADT which can be used for topology information, but that
table is the unmodified host table, which means that there can be more CPUs
registered than the number of vCPUs XEN provides for the dom0 guest.

DomU does not have ACPI and both rely on counting the possible vCPUs via an
hypercall.

To prepare for using CPUID topology information either via MADT or via fake
APIC IDs count the number of possible CPUs during early boot and adjust
nr_cpu_ids() accordingly.

Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
---
 arch/x86/xen/enlighten_pv.c |    3 +++
 arch/x86/xen/smp.h          |    2 ++
 arch/x86/xen/smp_pv.c       |   14 ++++++++++++++
 3 files changed, 19 insertions(+)

--- a/arch/x86/xen/enlighten_pv.c
+++ b/arch/x86/xen/enlighten_pv.c
@@ -189,6 +189,9 @@ static void __init xen_pv_init_platform(
 		xen_set_mtrr_data();
 	else
 		mtrr_overwrite_state(NULL, 0, MTRR_TYPE_WRBACK);
+
+	/* Adjust nr_cpu_ids before "enumeration" happens */
+	xen_smp_count_cpus();
 }
 
 static void __init xen_pv_guest_late_init(void)
--- a/arch/x86/xen/smp.h
+++ b/arch/x86/xen/smp.h
@@ -19,6 +19,7 @@ extern void xen_smp_intr_free(unsigned i
 int xen_smp_intr_init_pv(unsigned int cpu);
 void xen_smp_intr_free_pv(unsigned int cpu);
 
+void xen_smp_count_cpus(void);
 void xen_smp_cpus_done(unsigned int max_cpus);
 
 void xen_smp_send_reschedule(int cpu);
@@ -44,6 +45,7 @@ static inline int xen_smp_intr_init_pv(u
 	return 0;
 }
 static inline void xen_smp_intr_free_pv(unsigned int cpu) {}
+static inline void xen_smp_count_cpus(void) { }
 #endif /* CONFIG_SMP */
 
 #endif
--- a/arch/x86/xen/smp_pv.c
+++ b/arch/x86/xen/smp_pv.c
@@ -411,6 +411,20 @@ static irqreturn_t xen_irq_work_interrup
 	return IRQ_HANDLED;
 }
 
+void __init xen_smp_count_cpus(void)
+{
+	unsigned int cpus;
+
+	for (cpus = 0; cpus < nr_cpu_ids; cpus++) {
+		if (HYPERVISOR_vcpu_op(VCPUOP_is_up, cpus, NULL) < 0)
+			break;
+	}
+
+	pr_info("Xen PV: Detected %u vCPUS\n", cpus);
+	if (cpus < nr_cpu_ids)
+		set_nr_cpu_ids(cpus);
+}
+
 static const struct smp_ops xen_smp_ops __initconst = {
 	.smp_prepare_boot_cpu = xen_pv_smp_prepare_boot_cpu,
 	.smp_prepare_cpus = xen_pv_smp_prepare_cpus,

[patch 42/53] x86/cpu/topology: Let XEN/PV use topology from CPUID/MADT

[Thomas Gleixner]

It turns out that XEN/PV Dom0 has halfways usable CPUID/MADT enumeration
except that it cannot deal with CPUs which are enumerated as disabled in
MADT.

DomU has no MADT and provides at least rudimentary topology information in
CPUID leaves 1 and 4.

For both it's important that there are not more possible Linux CPUs than
vCPUs provided by the hypervisor.

As this is ensured by counting the vCPUs before enumeration happens:

  - lift the restrictions in the CPUID evaluation and the MADT parser

  - Utilize MADT registration for Dom0

  - Keep the fake APIC ID registration for DomU

  - Fix the XEN APIC fake so the readout of the local APIC ID works for
    Dom0 via the hypercall and for DomU by returning the registered
    fake APIC IDs.

With that the XEN/PV fake approximates usefulness.

Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
---
 arch/x86/kernel/acpi/boot.c           |   25 ++++++++-----------------
 arch/x86/kernel/cpu/topology_common.c |    2 +-
 arch/x86/xen/apic.c                   |   14 +++++++-------
 arch/x86/xen/smp_pv.c                 |   13 ++++++++-----
 4 files changed, 24 insertions(+), 30 deletions(-)
--- a/arch/x86/kernel/acpi/boot.c
+++ b/arch/x86/kernel/acpi/boot.c
@@ -23,8 +23,6 @@
 #include <linux/serial_core.h>
 #include <linux/pgtable.h>
 
-#include <xen/xen.h>
-
 #include <asm/e820/api.h>
 #include <asm/irqdomain.h>
 #include <asm/pci_x86.h>
@@ -161,12 +159,6 @@ static int __init acpi_parse_madt(struct
 	return 0;
 }
 
-static __init void acpi_register_lapic(u32 apic_id, u32 acpi_id, bool present)
-{
-	if (!xen_pv_domain())
-		topology_register_apic(apic_id, acpi_id, present);
-}
-
 static bool __init acpi_is_processor_usable(u32 lapic_flags)
 {
 	if (lapic_flags & ACPI_MADT_ENABLED)
@@ -220,7 +212,7 @@ acpi_parse_x2apic(union acpi_subtable_he
 		return 0;
 	}
 
-	acpi_register_lapic(apic_id, processor->uid, enabled);
+	topology_register_apic(apic_id, processor->uid, enabled);
 #else
 	pr_warn("x2apic entry ignored\n");
 #endif
@@ -255,9 +247,9 @@ acpi_parse_lapic(union acpi_subtable_hea
 	 * to not preallocating memory for all NR_CPUS
 	 * when we use CPU hotplug.
 	 */
-	acpi_register_lapic(processor->id,	/* APIC ID */
-			    processor->processor_id, /* ACPI ID */
-			    processor->lapic_flags & ACPI_MADT_ENABLED);
+	topology_register_apic(processor->id,	/* APIC ID */
+			       processor->processor_id, /* ACPI ID */
+			       processor->lapic_flags & ACPI_MADT_ENABLED);
 
 	return 0;
 }
@@ -274,9 +266,9 @@ acpi_parse_sapic(union acpi_subtable_hea
 
 	acpi_table_print_madt_entry(&header->common);
 
-	acpi_register_lapic((processor->id << 8) | processor->eid,/* APIC ID */
-			    processor->processor_id, /* ACPI ID */
-			    processor->lapic_flags & ACPI_MADT_ENABLED);
+	topology_register_apic((processor->id << 8) | processor->eid,/* APIC ID */
+			       processor->processor_id, /* ACPI ID */
+			       processor->lapic_flags & ACPI_MADT_ENABLED);
 
 	return 0;
 }
@@ -1073,8 +1065,7 @@ static int __init early_acpi_parse_madt_
 		return count;
 	}
 
-	if (!xen_pv_domain())
-		register_lapic_address(acpi_lapic_addr);
+	register_lapic_address(acpi_lapic_addr);
 
 	return count;
 }
--- a/arch/x86/kernel/cpu/topology_common.c
+++ b/arch/x86/kernel/cpu/topology_common.c
@@ -77,7 +77,7 @@ static bool fake_topology(struct topo_sc
 	topology_set_dom(tscan, TOPO_SMT_DOMAIN, 0, 1);
 	topology_set_dom(tscan, TOPO_CORE_DOMAIN, 0, 1);
 
-	return tscan->c->cpuid_level < 1 || xen_pv_domain();
+	return tscan->c->cpuid_level < 1;
 }
 
 static void parse_topology(struct topo_scan *tscan, bool early)
--- a/arch/x86/xen/apic.c
+++ b/arch/x86/xen/apic.c
@@ -43,20 +43,20 @@ static u32 xen_apic_read(u32 reg)
 	struct xen_platform_op op = {
 		.cmd = XENPF_get_cpuinfo,
 		.interface_version = XENPF_INTERFACE_VERSION,
-		.u.pcpu_info.xen_cpuid = 0,
 	};
-	int ret;
-
-	/* Shouldn't need this as APIC is turned off for PV, and we only
-	 * get called on the bootup processor. But just in case. */
-	if (!xen_initial_domain() || smp_processor_id())
-		return 0;
+	int ret, cpu;
 
 	if (reg == APIC_LVR)
 		return 0x14;
 	if (reg != APIC_ID)
 		return 0;
 
+	cpu = smp_processor_id();
+	if (!xen_initial_domain())
+		return cpu ? cpuid_to_apicid[cpu] << 24 : 0;
+
+	op.u.pcpu_info.xen_cpuid = cpu;
+
 	ret = HYPERVISOR_platform_op(&op);
 	if (ret)
 		op.u.pcpu_info.apic_id = BAD_APICID;
--- a/arch/x86/xen/smp_pv.c
+++ b/arch/x86/xen/smp_pv.c
@@ -156,11 +156,9 @@ static void __init xen_pv_smp_config(voi
 
 	topology_register_boot_apic(apicid++);
 
-	for (i = 1; i < nr_cpu_ids; i++) {
-		if (HYPERVISOR_vcpu_op(VCPUOP_is_up, i, NULL) < 0)
-			break;
+	for (i = 1; i < nr_cpu_ids; i++)
 		topology_register_apic(apicid++, CPU_ACPIID_INVALID, true);
-	}
+
 	/* Pretend to be a proper enumerated system */
 	smp_found_config = 1;
 }
@@ -451,5 +449,10 @@ void __init xen_smp_init(void)
 	/* Avoid searching for BIOS MP tables */
 	x86_init.mpparse.find_mptable		= x86_init_noop;
 	x86_init.mpparse.early_parse_smp_cfg	= x86_init_noop;
-	x86_init.mpparse.parse_smp_cfg		= xen_pv_smp_config;
+
+	/* XEN/PV Dom0 has halfways sane topology information via CPUID/MADT */
+	if (xen_initial_domain())
+		x86_init.mpparse.parse_smp_cfg	= x86_init_noop;
+	else
+		x86_init.mpparse.parse_smp_cfg	= xen_pv_smp_config;
 }

[patch 43/53] x86/cpu/topology: Use topology bitmaps for sizing

[Thomas Gleixner]

Now that all possible APIC IDs are tracked in the topology bitmaps, its
trivial to retrieve the real information from there.

This gets rid of the guesstimates for the maximal packages and dies per
package as the actual numbers can be determined before a single AP has been
brought up.

The number of SMT threads can now be determined correctly from the bitmaps
in all situations. Up to now a system which has SMT disabled in the BIOS
will still claim that it is SMT capable, because the lowest APIC ID bit is
reserved for that and CPUID leaf 0xb/0x1f still enumerates the SMT domain
accordingly. By calculating the bitmap weights of the SMT and the CORE
domain and setting them into relation the SMT disabled in BIOS situation
reports correctly that the system is not SMT capable.

It also handles the situation correctly when a hybrid systems boot CPU does
not have SMT as it takes the SMT capability of the APs fully into account.

Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
---
 arch/x86/include/asm/smp.h            |    3 +--
 arch/x86/include/asm/topology.h       |   23 ++++++++++++-----------
 arch/x86/kernel/cpu/common.c          |    9 ++++++---
 arch/x86/kernel/cpu/debugfs.c         |    2 +-
 arch/x86/kernel/cpu/topology.c        |   16 +++++++++++++++-
 arch/x86/kernel/cpu/topology_common.c |   24 ------------------------
 arch/x86/kernel/smpboot.c             |   16 ----------------
 arch/x86/xen/smp.c                    |    2 --
 8 files changed, 35 insertions(+), 60 deletions(-)

--- a/arch/x86/include/asm/smp.h
+++ b/arch/x86/include/asm/smp.h
@@ -8,7 +8,7 @@
 #include <asm/current.h>
 #include <asm/thread_info.h>
 
-extern int smp_num_siblings;
+extern unsigned int smp_num_siblings;
 
 DECLARE_PER_CPU_READ_MOSTLY(cpumask_var_t, cpu_sibling_map);
 DECLARE_PER_CPU_READ_MOSTLY(cpumask_var_t, cpu_core_map);
@@ -109,7 +109,6 @@ void cpu_disable_common(void);
 void native_smp_prepare_boot_cpu(void);
 void smp_prepare_cpus_common(void);
 void native_smp_prepare_cpus(unsigned int max_cpus);
-void calculate_max_logical_packages(void);
 void native_smp_cpus_done(unsigned int max_cpus);
 int common_cpu_up(unsigned int cpunum, struct task_struct *tidle);
 int native_kick_ap(unsigned int cpu, struct task_struct *tidle);
--- a/arch/x86/include/asm/topology.h
+++ b/arch/x86/include/asm/topology.h
@@ -143,7 +143,18 @@ extern const struct cpumask *cpu_cluster
 
 #define topology_amd_node_id(cpu)		(cpu_data(cpu).topo.amd_node_id)
 
-extern unsigned int __max_die_per_package;
+extern unsigned int __max_dies_per_package;
+extern unsigned int __max_logical_packages;
+
+static inline unsigned int topology_max_packages(void)
+{
+	return __max_logical_packages;
+}
+
+static inline unsigned int topology_max_die_per_package(void)
+{
+	return __max_dies_per_package;
+}
 
 #ifdef CONFIG_SMP
 #define topology_cluster_id(cpu)		(cpu_data(cpu).topo.l2c_id)
@@ -152,14 +163,6 @@ extern unsigned int __max_die_per_packag
 #define topology_core_cpumask(cpu)		(per_cpu(cpu_core_map, cpu))
 #define topology_sibling_cpumask(cpu)		(per_cpu(cpu_sibling_map, cpu))
 
-extern unsigned int __max_logical_packages;
-#define topology_max_packages()			(__max_logical_packages)
-
-static inline int topology_max_die_per_package(void)
-{
-	return __max_die_per_package;
-}
-
 extern int __max_smt_threads;
 
 static inline int topology_max_smt_threads(void)
@@ -194,13 +197,11 @@ static inline bool topology_is_primary_t
 void topology_apply_cmdline_limits_early(void);
 
 #else /* CONFIG_SMP */
-#define topology_max_packages()			(1)
 static inline int
 topology_update_package_map(unsigned int apicid, unsigned int cpu) { return 0; }
 static inline int
 topology_update_die_map(unsigned int dieid, unsigned int cpu) { return 0; }
 static inline int topology_phys_to_logical_pkg(unsigned int pkg) { return 0; }
-static inline int topology_max_die_per_package(void) { return 1; }
 static inline int topology_max_smt_threads(void) { return 1; }
 static inline bool topology_is_primary_thread(unsigned int cpu) { return true; }
 static inline bool topology_smt_supported(void) { return false; }
--- a/arch/x86/kernel/cpu/common.c
+++ b/arch/x86/kernel/cpu/common.c
@@ -72,11 +72,14 @@
 u32 elf_hwcap2 __read_mostly;
 
 /* Number of siblings per CPU package */
-int smp_num_siblings = 1;
+unsigned int smp_num_siblings __ro_after_init = 1;
 EXPORT_SYMBOL(smp_num_siblings);
 
-unsigned int __max_die_per_package __read_mostly = 1;
-EXPORT_SYMBOL(__max_die_per_package);
+unsigned int __max_dies_per_package __ro_after_init = 1;
+EXPORT_SYMBOL(__max_dies_per_package);
+
+unsigned int __max_logical_packages __ro_after_init = 1;
+EXPORT_SYMBOL(__max_logical_packages);
 
 static struct ppin_info {
 	int	feature;
--- a/arch/x86/kernel/cpu/debugfs.c
+++ b/arch/x86/kernel/cpu/debugfs.c
@@ -29,7 +29,7 @@ static int cpu_debug_show(struct seq_fil
 	seq_printf(m, "amd_node_id:         %u\n", c->topo.amd_node_id);
 	seq_printf(m, "amd_nodes_per_pkg:   %u\n", topology_amd_nodes_per_pkg());
 	seq_printf(m, "max_cores:           %u\n", c->x86_max_cores);
-	seq_printf(m, "max_die_per_pkg:     %u\n", __max_die_per_package);
+	seq_printf(m, "max_dies_per_pkg:    %u\n", __max_dies_per_package);
 	seq_printf(m, "smp_num_siblings:    %u\n", smp_num_siblings);
 	return 0;
 }
--- a/arch/x86/kernel/cpu/topology.c
+++ b/arch/x86/kernel/cpu/topology.c
@@ -347,8 +347,8 @@ void __init topology_init_possible_cpus(
 {
 	unsigned int assigned = topo_info.nr_assigned_cpus;
 	unsigned int disabled = topo_info.nr_disabled_cpus;
+	unsigned int cnta, cntb, cpu, dom, allowed = 1;
 	unsigned int total = assigned + disabled;
-	unsigned int cpu, dom, allowed = 1;
 	u32 apicid;
 
 	if (!restrict_to_up()) {
@@ -374,6 +374,20 @@ void __init topology_init_possible_cpus(
 	total_cpus = allowed;
 	set_nr_cpu_ids(allowed);
 
+	cnta = domain_weight(TOPO_PKG_DOMAIN);
+	cntb = domain_weight(TOPO_DIE_DOMAIN);
+	__max_logical_packages = cnta;
+	__max_dies_per_package = 1U << (get_count_order(cntb) - get_count_order(cnta));
+
+	pr_info("Max. logical packages: %3u\n", cnta);
+	pr_info("Max. logical dies:     %3u\n", cntb);
+	pr_info("Max. dies per package: %3u\n", __max_dies_per_package);
+
+	cnta = domain_weight(TOPO_CORE_DOMAIN);
+	cntb = domain_weight(TOPO_SMT_DOMAIN);
+	smp_num_siblings = 1U << (get_count_order(cntb) - get_count_order(cnta));
+	pr_info("Max. threads per core: %3u\n", smp_num_siblings);
+
 	pr_info("Allowing %u present CPUs plus %u hotplug CPUs\n", assigned, disabled);
 	if (topo_info.nr_rejected_cpus)
 		pr_info("Rejected CPUs %u\n", topo_info.nr_rejected_cpus);
--- a/arch/x86/kernel/cpu/topology_common.c
+++ b/arch/x86/kernel/cpu/topology_common.c
@@ -193,16 +193,6 @@ void cpu_parse_topology(struct cpuinfo_x
 		       tscan.dom_shifts[dom], x86_topo_system.dom_shifts[dom]);
 	}
 
-	/* Bug compatible with the existing parsers */
-	if (tscan.dom_ncpus[TOPO_SMT_DOMAIN] > smp_num_siblings) {
-		if (system_state == SYSTEM_BOOTING) {
-			pr_warn_once("CPU%d: SMT detected and enabled late\n", cpu);
-			smp_num_siblings = tscan.dom_ncpus[TOPO_SMT_DOMAIN];
-		} else {
-			pr_warn_once("CPU%d: SMT detected after init. Too late!\n", cpu);
-		}
-	}
-
 	topo_set_ids(&tscan);
 	topo_set_max_cores(&tscan);
 }
@@ -229,20 +219,6 @@ void __init cpu_init_topology(struct cpu
 	topo_set_max_cores(&tscan);
 
 	/*
-	 * Bug compatible with the existing code. If the boot CPU does not
-	 * have SMT this ends up with one sibling. This needs way deeper
-	 * changes further down the road to get it right during early boot.
-	 */
-	smp_num_siblings = tscan.dom_ncpus[TOPO_SMT_DOMAIN];
-
-	/*
-	 * Neither it's clear whether there are as many dies as the APIC
-	 * space indicating die level is. But assume that the actual number
-	 * of CPUs gives a proper indication for now to stay bug compatible.
-	 */
-	__max_die_per_package = tscan.dom_ncpus[TOPO_DIE_DOMAIN] /
-		tscan.dom_ncpus[TOPO_DIE_DOMAIN - 1];
-	/*
 	 * AMD systems have Nodes per package which cannot be mapped to
 	 * APIC ID (yet).
 	 */
--- a/arch/x86/kernel/smpboot.c
+++ b/arch/x86/kernel/smpboot.c
@@ -138,8 +138,6 @@ static DEFINE_PER_CPU_READ_MOSTLY(struct
 	.phys_die_id	= U32_MAX,
 };
 
-unsigned int __max_logical_packages __read_mostly;
-EXPORT_SYMBOL(__max_logical_packages);
 static unsigned int logical_packages __read_mostly;
 static unsigned int logical_die __read_mostly;
 
@@ -1303,24 +1301,10 @@ void __init native_smp_prepare_boot_cpu(
 	native_pv_lock_init();
 }
 
-void __init calculate_max_logical_packages(void)
-{
-	int ncpus;
-
-	/*
-	 * Today neither Intel nor AMD support heterogeneous systems so
-	 * extrapolate the boot cpu's data to all packages.
-	 */
-	ncpus = cpu_data(0).booted_cores * topology_max_smt_threads();
-	__max_logical_packages = DIV_ROUND_UP(total_cpus, ncpus);
-	pr_info("Max logical packages: %u\n", __max_logical_packages);
-}
-
 void __init native_smp_cpus_done(unsigned int max_cpus)
 {
 	pr_debug("Boot done\n");
 
-	calculate_max_logical_packages();
 	build_sched_topology();
 	nmi_selftest();
 	impress_friends();
--- a/arch/x86/xen/smp.c
+++ b/arch/x86/xen/smp.c
@@ -123,8 +123,6 @@ void __init xen_smp_cpus_done(unsigned i
 {
 	if (xen_hvm_domain())
 		native_smp_cpus_done(max_cpus);
-	else
-		calculate_max_logical_packages();
 }
 
 void xen_smp_send_reschedule(int cpu)

[patch 44/53] x86/cpu/topology: Mop up primary thread mask handling

[Thomas Gleixner]

The early initcall to initialize the primary thread mask is not longer
required because topology_init_possible_cpus() can mark primary threads
correctly when initializing the possible and present map as the number of
SMT threads is already determined correctly.

The XENPV workaround is not longer required because XENPV now registers
fake APIC IDs which will just work like any other enumeration.

Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
---
 arch/x86/kernel/cpu/topology.c |   29 ++---------------------------
 1 file changed, 2 insertions(+), 27 deletions(-)

--- a/arch/x86/kernel/cpu/topology.c
+++ b/arch/x86/kernel/cpu/topology.c
@@ -92,30 +92,6 @@ static void cpu_mark_primary_thread(unsi
 	if (smp_num_siblings == 1 || !(apicid & mask))
 		cpumask_set_cpu(cpu, &__cpu_primary_thread_mask);
 }
-
-/*
- * Due to the utter mess of CPUID evaluation smp_num_siblings is not valid
- * during early boot. Initialize the primary thread mask before SMP
- * bringup.
- */
-static int __init smp_init_primary_thread_mask(void)
-{
-	unsigned int cpu;
-
-	/*
-	 * XEN/PV provides either none or useless topology information.
-	 * Pretend that all vCPUs are primary threads.
-	 */
-	if (xen_pv_domain()) {
-		cpumask_copy(&__cpu_primary_thread_mask, cpu_possible_mask);
-		return 0;
-	}
-
-	for (cpu = 0; cpu < topo_info.nr_assigned_cpus; cpu++)
-		cpu_mark_primary_thread(cpu, cpuid_to_apicid[cpu]);
-	return 0;
-}
-early_initcall(smp_init_primary_thread_mask);
 #else
 static inline void cpu_mark_primary_thread(unsigned int cpu, unsigned int apicid) { }
 #endif
@@ -161,9 +137,6 @@ static void topo_set_cpuids(unsigned int
 #endif
 	set_cpu_possible(cpu, true);
 	set_cpu_present(cpu, true);
-
-	if (system_state != SYSTEM_BOOTING)
-		cpu_mark_primary_thread(cpu, apic_id);
 }
 
 /**
@@ -252,6 +225,7 @@ int topology_hotplug_apic(u32 apic_id, u
 
 	set_bit(apic_id, phys_cpu_present_map);
 	topo_set_cpuids(cpu, apic_id, acpi_id);
+	cpu_mark_primary_thread(cpu, apic_id);
 	return cpu;
 }
 
@@ -422,6 +396,7 @@ void __init topology_init_possible_cpus(
 		if (apicid == BAD_APICID)
 			continue;
 
+		cpu_mark_primary_thread(cpu, apicid);
 		set_cpu_present(cpu, test_bit(apicid, phys_cpu_present_map));
 	}
 }

[patch 45/53] x86/cpu/topology: Simplify cpu_mark_primary_thread()

[Thomas Gleixner]

No point in creating a mask via fls(). smp_num_siblings is guaranteed to be
a power of 2. So just using (smp_num_siblings - 1) has the same effect.

Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
---
 arch/x86/kernel/cpu/topology.c |    5 +----
 1 file changed, 1 insertion(+), 4 deletions(-)

--- a/arch/x86/kernel/cpu/topology.c
+++ b/arch/x86/kernel/cpu/topology.c
@@ -86,10 +86,7 @@ bool arch_match_cpu_phys_id(int cpu, u64
 #ifdef CONFIG_SMP
 static void cpu_mark_primary_thread(unsigned int cpu, unsigned int apicid)
 {
-	/* Isolate the SMT bit(s) in the APICID and check for 0 */
-	u32 mask = (1U << (fls(smp_num_siblings) - 1)) - 1;
-
-	if (smp_num_siblings == 1 || !(apicid & mask))
+	if (!(apicid & (smp_num_siblings - 1)))
 		cpumask_set_cpu(cpu, &__cpu_primary_thread_mask);
 }
 #else

[patch 46/53] x86/cpu/topology: Provide logical pkg/die mapping

[Thomas Gleixner]

With the topology bitmaps in place the logical package and die IDs can
trivially be retrieved by determining the bitmap weight of the relevant
topology domain level up to and including the physical ID in question.

Provide a function to that effect.

Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
---
 arch/x86/include/asm/topology.h |    2 ++
 arch/x86/kernel/cpu/topology.c  |   28 ++++++++++++++++++++++++++++
 2 files changed, 30 insertions(+)

--- a/arch/x86/include/asm/topology.h
+++ b/arch/x86/include/asm/topology.h
@@ -156,6 +156,8 @@ static inline unsigned int topology_max_
 	return __max_dies_per_package;
 }
 
+int topology_get_logical_id(u32 apicid, enum x86_topology_domains at_level);
+
 #ifdef CONFIG_SMP
 #define topology_cluster_id(cpu)		(cpu_data(cpu).topo.l2c_id)
 #define topology_die_cpumask(cpu)		(per_cpu(cpu_die_map, cpu))
--- a/arch/x86/kernel/cpu/topology.c
+++ b/arch/x86/kernel/cpu/topology.c
@@ -199,6 +199,34 @@ void __init topology_register_boot_apic(
 	topology_register_apic(apic_id, CPU_ACPIID_INVALID, true);
 }
 
+/**
+ * topology_get_logical_id - Retrieve the logical ID at a given topology domain level
+ * @apicid:		The APIC ID for which to lookup the logical ID
+ * @at_level:		The topology domain level to use
+ *
+ * @apicid must be a full APIC ID, not the normalized variant. It's valid to have
+ * all bits below the domain level specified by @at_level to be clear. So both
+ * real APIC IDs and backshifted normalized APIC IDs work correctly.
+ *
+ * Returns:
+ *  - >= 0:	The requested logical ID
+ *  - -ERANGE:	@apicid is out of range
+ *  - -ENODEV:	@apicid is not registered
+ */
+int topology_get_logical_id(u32 apicid, enum x86_topology_domains at_level)
+{
+	/* Remove the bits below @at_level to get the proper level ID of @apicid */
+	unsigned int lvlid = topo_apicid(apicid, at_level);
+
+	if (lvlid >= MAX_LOCAL_APIC)
+		return -ERANGE;
+	if (!test_bit(lvlid, apic_maps[at_level].map))
+		return -ENODEV;
+	/* Get the number of set bits before @lvlid. */
+	return bitmap_weight(apic_maps[at_level].map, lvlid);
+}
+EXPORT_SYMBOL_GPL(topology_get_logical_id);
+
 #ifdef CONFIG_ACPI_HOTPLUG_CPU
 /**
  * topology_hotplug_apic - Handle a physical hotplugged APIC after boot

[patch 47/53] x86/cpu/topology: Use topology logical mapping mechanism

[Thomas Gleixner]

Replace the logical package and die management functionality and retrieve
the logical IDs from the topology bitmaps.

Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
---
 arch/x86/include/asm/topology.h       |   13 +--
 arch/x86/kernel/cpu/common.c          |   13 ---
 arch/x86/kernel/cpu/topology_common.c |    4 +
 arch/x86/kernel/smpboot.c             |  111 ----------------------------------
 4 files changed, 11 insertions(+), 130 deletions(-)

--- a/arch/x86/include/asm/topology.h
+++ b/arch/x86/include/asm/topology.h
@@ -158,6 +158,12 @@ static inline unsigned int topology_max_
 
 int topology_get_logical_id(u32 apicid, enum x86_topology_domains at_level);
 
+static inline int topology_phys_to_logical_pkg(unsigned int pkg)
+{
+	return topology_get_logical_id(pkg << x86_topo_system.dom_shifts[TOPO_PKG_DOMAIN],
+				       TOPO_PKG_DOMAIN);
+}
+
 #ifdef CONFIG_SMP
 #define topology_cluster_id(cpu)		(cpu_data(cpu).topo.l2c_id)
 #define topology_die_cpumask(cpu)		(per_cpu(cpu_die_map, cpu))
@@ -172,9 +178,6 @@ static inline int topology_max_smt_threa
 	return __max_smt_threads;
 }
 
-int topology_update_package_map(unsigned int apicid, unsigned int cpu);
-int topology_update_die_map(unsigned int dieid, unsigned int cpu);
-int topology_phys_to_logical_pkg(unsigned int pkg);
 bool topology_smt_supported(void);
 
 extern unsigned int __amd_nodes_per_pkg;
@@ -199,10 +202,6 @@ static inline bool topology_is_primary_t
 void topology_apply_cmdline_limits_early(void);
 
 #else /* CONFIG_SMP */
-static inline int
-topology_update_package_map(unsigned int apicid, unsigned int cpu) { return 0; }
-static inline int
-topology_update_die_map(unsigned int dieid, unsigned int cpu) { return 0; }
 static inline int topology_phys_to_logical_pkg(unsigned int pkg) { return 0; }
 static inline int topology_max_smt_threads(void) { return 1; }
 static inline bool topology_is_primary_thread(unsigned int cpu) { return true; }
--- a/arch/x86/kernel/cpu/common.c
+++ b/arch/x86/kernel/cpu/common.c
@@ -1679,18 +1679,6 @@ static void generic_identify(struct cpui
 #endif
 }
 
-static void update_package_map(struct cpuinfo_x86 *c)
-{
-#ifdef CONFIG_SMP
-	unsigned int cpu = smp_processor_id();
-
-	BUG_ON(topology_update_package_map(c->topo.pkg_id, cpu));
-	BUG_ON(topology_update_die_map(c->topo.die_id, cpu));
-#else
-	c->topo.logical_pkg_id = 0;
-#endif
-}
-
 /*
  * This does the hard work of actually picking apart the CPU stuff...
  */
@@ -1868,7 +1856,6 @@ void identify_secondary_cpu(struct cpuin
 #ifdef CONFIG_X86_32
 	enable_sep_cpu();
 #endif
-	update_package_map(c);
 	x86_spec_ctrl_setup_ap();
 	update_srbds_msr();
 
--- a/arch/x86/kernel/cpu/topology_common.c
+++ b/arch/x86/kernel/cpu/topology_common.c
@@ -10,6 +10,7 @@
 #include "cpu.h"
 
 struct x86_topology_system x86_topo_system __ro_after_init;
+EXPORT_SYMBOL_GPL(x86_topo_system);
 
 unsigned int __amd_nodes_per_pkg __ro_after_init;
 EXPORT_SYMBOL_GPL(__amd_nodes_per_pkg);
@@ -147,6 +148,9 @@ static void topo_set_ids(struct topo_sca
 	c->topo.pkg_id = topo_shift_apicid(apicid, TOPO_PKG_DOMAIN);
 	c->topo.die_id = topo_shift_apicid(apicid, TOPO_DIE_DOMAIN);
 
+	c->topo.logical_pkg_id = topology_get_logical_id(apicid, TOPO_PKG_DOMAIN);
+	c->topo.logical_die_id = topology_get_logical_id(apicid, TOPO_DIE_DOMAIN);
+
 	/* Relative core ID */
 	c->topo.core_id = topo_relative_domain_id(apicid, TOPO_CORE_DOMAIN);
 
--- a/arch/x86/kernel/smpboot.c
+++ b/arch/x86/kernel/smpboot.c
@@ -124,23 +124,6 @@ struct mwait_cpu_dead {
  */
 static DEFINE_PER_CPU_ALIGNED(struct mwait_cpu_dead, mwait_cpu_dead);
 
-/* Logical package management. */
-struct logical_maps {
-	u32	phys_pkg_id;
-	u32	phys_die_id;
-	u32	logical_pkg_id;
-	u32	logical_die_id;
-};
-
-/* Temporary workaround until the full topology mechanics is in place */
-static DEFINE_PER_CPU_READ_MOSTLY(struct logical_maps, logical_maps) = {
-	.phys_pkg_id	= U32_MAX,
-	.phys_die_id	= U32_MAX,
-};
-
-static unsigned int logical_packages __read_mostly;
-static unsigned int logical_die __read_mostly;
-
 /* Maximum number of SMT threads on any online core */
 int __read_mostly __max_smt_threads = 1;
 
@@ -345,103 +328,11 @@ bool topology_smt_supported(void)
 	return smp_num_siblings > 1;
 }
 
-/**
- * topology_phys_to_logical_pkg - Map a physical package id to a logical
- * @phys_pkg:	The physical package id to map
- *
- * Returns logical package id or -1 if not found
- */
-int topology_phys_to_logical_pkg(unsigned int phys_pkg)
-{
-	int cpu;
-
-	for_each_possible_cpu(cpu) {
-		if (per_cpu(logical_maps.phys_pkg_id, cpu) == phys_pkg)
-			return per_cpu(logical_maps.logical_pkg_id, cpu);
-	}
-	return -1;
-}
-EXPORT_SYMBOL(topology_phys_to_logical_pkg);
-
-/**
- * topology_phys_to_logical_die - Map a physical die id to logical
- * @die_id:	The physical die id to map
- * @cur_cpu:	The CPU for which the mapping is done
- *
- * Returns logical die id or -1 if not found
- */
-static int topology_phys_to_logical_die(unsigned int die_id, unsigned int cur_cpu)
-{
-	int cpu, proc_id = cpu_data(cur_cpu).topo.pkg_id;
-
-	for_each_possible_cpu(cpu) {
-		if (per_cpu(logical_maps.phys_pkg_id, cpu) == proc_id &&
-		    per_cpu(logical_maps.phys_die_id, cpu) == die_id)
-			return per_cpu(logical_maps.logical_die_id, cpu);
-	}
-	return -1;
-}
-
-/**
- * topology_update_package_map - Update the physical to logical package map
- * @pkg:	The physical package id as retrieved via CPUID
- * @cpu:	The cpu for which this is updated
- */
-int topology_update_package_map(unsigned int pkg, unsigned int cpu)
-{
-	int new;
-
-	/* Already available somewhere? */
-	new = topology_phys_to_logical_pkg(pkg);
-	if (new >= 0)
-		goto found;
-
-	new = logical_packages++;
-	if (new != pkg) {
-		pr_info("CPU %u Converting physical %u to logical package %u\n",
-			cpu, pkg, new);
-	}
-found:
-	per_cpu(logical_maps.phys_pkg_id, cpu) = pkg;
-	per_cpu(logical_maps.logical_pkg_id, cpu) = new;
-	cpu_data(cpu).topo.logical_pkg_id = new;
-	return 0;
-}
-/**
- * topology_update_die_map - Update the physical to logical die map
- * @die:	The die id as retrieved via CPUID
- * @cpu:	The cpu for which this is updated
- */
-int topology_update_die_map(unsigned int die, unsigned int cpu)
-{
-	int new;
-
-	/* Already available somewhere? */
-	new = topology_phys_to_logical_die(die, cpu);
-	if (new >= 0)
-		goto found;
-
-	new = logical_die++;
-	if (new != die) {
-		pr_info("CPU %u Converting physical %u to logical die %u\n",
-			cpu, die, new);
-	}
-found:
-	per_cpu(logical_maps.phys_die_id, cpu) = die;
-	per_cpu(logical_maps.logical_die_id, cpu) = new;
-	cpu_data(cpu).topo.logical_die_id = new;
-	return 0;
-}
-
 void __init smp_store_boot_cpu_info(void)
 {
-	int id = 0; /* CPU 0 */
-	struct cpuinfo_x86 *c = &cpu_data(id);
+	struct cpuinfo_x86 *c = &cpu_data(0);
 
 	*c = boot_cpu_data;
-	c->cpu_index = id;
-	topology_update_package_map(c->topo.pkg_id, id);
-	topology_update_die_map(c->topo.die_id, id);
 	c->initialized = true;
 }
 

[patch 48/53] x86/cpu/topology: Retrieve cores per package from topology bitmaps

[Thomas Gleixner]

Similar to other sizing information the number of cores per package can be
established from the topology bitmap.

Provide a function for retrieving that information and replace the buggy
hack in the CPUID evaluation with it.

Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
---
 arch/x86/kernel/cpu/topology.c        |   43 ++++++++++++++++++++++++++++++++++
 arch/x86/kernel/cpu/topology.h        |    3 ++
 arch/x86/kernel/cpu/topology_common.c |   18 ++------------
 3 files changed, 49 insertions(+), 15 deletions(-)

--- a/arch/x86/kernel/cpu/topology.c
+++ b/arch/x86/kernel/cpu/topology.c
@@ -227,6 +227,49 @@ int topology_get_logical_id(u32 apicid,
 }
 EXPORT_SYMBOL_GPL(topology_get_logical_id);
 
+/**
+ * topology_unit_count - Retrieve the count of specified units at a given topology domain level
+ * @apicid:		The APIC ID which specifies the search range
+ * @which_units:	The domain level specifying the units to count
+ * @at_level:		The domain level at which @which_units have to be counted
+ *
+ * This returns the number of possible units according to the enumerated
+ * information.
+ *
+ * E.g. topology_count_units(apicid, TOPO_CORE_DOMAIN, TOPO_PKG_DOMAIN)
+ * counts the number of possible cores in the package to which @apicid
+ * belongs.
+ *
+ * @at_level must obviously be greater than @which_level to produce useful
+ * results.  If @at_level is equal to @which_units the result is
+ * unsurprisingly 1. If @at_level is less than @which_units the results
+ * is by definition undefined and the function returns 0.
+ */
+unsigned int topology_unit_count(u32 apicid, enum x86_topology_domains which_units,
+				 enum x86_topology_domains at_level)
+{
+	/* Remove the bits below @at_level to get the proper level ID of @apicid */
+	unsigned int lvlid = topo_apicid(apicid, at_level);
+	unsigned int id, end, cnt = 0;
+
+	if (lvlid >= MAX_LOCAL_APIC)
+		return 0;
+	if (!test_bit(lvlid, apic_maps[at_level].map))
+		return 0;
+	if (which_units > at_level)
+		return 0;
+	if (which_units == at_level)
+		return 1;
+
+	/* Calculate the exclusive end */
+	end = lvlid + (1U << x86_topo_system.dom_shifts[at_level]);
+	/* Unfortunately there is no bitmap_weight_range() */
+	for (id = find_next_bit(apic_maps[which_units].map, end, lvlid);
+	     id < end; id = find_next_bit(apic_maps[which_units].map, end, ++id))
+		cnt++;
+	return cnt;
+}
+
 #ifdef CONFIG_ACPI_HOTPLUG_CPU
 /**
  * topology_hotplug_apic - Handle a physical hotplugged APIC after boot
--- a/arch/x86/kernel/cpu/topology.h
+++ b/arch/x86/kernel/cpu/topology.h
@@ -50,4 +50,7 @@ static inline void topology_update_dom(s
 
 extern const char *domain_names[TOPO_MAX_DOMAIN];
 
+unsigned int topology_unit_count(u32 apicid, enum x86_topology_domains which_units,
+				 enum x86_topology_domains at_level);
+
 #endif /* ARCH_X86_TOPOLOGY_H */
--- a/arch/x86/kernel/cpu/topology_common.c
+++ b/arch/x86/kernel/cpu/topology_common.c
@@ -154,25 +154,15 @@ static void topo_set_ids(struct topo_sca
 	/* Relative core ID */
 	c->topo.core_id = topo_relative_domain_id(apicid, TOPO_CORE_DOMAIN);
 
+	/* Maximum number of cores on this package */
+	c->x86_max_cores = topology_unit_count(apicid, TOPO_CORE_DOMAIN, TOPO_PKG_DOMAIN);
+
 	c->topo.amd_node_id = tscan->amd_node_id;
 
 	if (c->x86_vendor == X86_VENDOR_AMD)
 		cpu_topology_fixup_amd(tscan);
 }
 
-static void topo_set_max_cores(struct topo_scan *tscan)
-{
-	/*
-	 * Bug compatible for now. This is broken on hybrid systems:
-	 * 8 cores SMT + 8 cores w/o SMT
-	 * tscan.dom_ncpus[TOPO_CORE_DOMAIN] = 24; 24 / 2 = 12 !!
-	 *
-	 * Cannot be fixed without further topology enumeration changes.
-	 */
-	tscan->c->x86_max_cores = tscan->dom_ncpus[TOPO_CORE_DOMAIN] >>
-		x86_topo_system.dom_shifts[TOPO_SMT_DOMAIN];
-}
-
 void cpu_parse_topology(struct cpuinfo_x86 *c)
 {
 	unsigned int dom, cpu = smp_processor_id();
@@ -198,7 +188,6 @@ void cpu_parse_topology(struct cpuinfo_x
 	}
 
 	topo_set_ids(&tscan);
-	topo_set_max_cores(&tscan);
 }
 
 void __init cpu_init_topology(struct cpuinfo_x86 *c)
@@ -220,7 +209,6 @@ void __init cpu_init_topology(struct cpu
 	}
 
 	topo_set_ids(&tscan);
-	topo_set_max_cores(&tscan);
 
 	/*
 	 * AMD systems have Nodes per package which cannot be mapped to

[patch 49/53] x86: Use topology functions instead of smp_num_siblings where applicable

[Thomas Gleixner]

Accessors exist for a reason.

Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
---
 arch/x86/include/asm/perf_event_p4.h |   11 ++---------
 arch/x86/kernel/process.c            |    2 +-
 arch/x86/kernel/smpboot.c            |    2 +-
 3 files changed, 4 insertions(+), 11 deletions(-)

--- a/arch/x86/include/asm/perf_event_p4.h
+++ b/arch/x86/include/asm/perf_event_p4.h
@@ -180,19 +180,12 @@ static inline u64 p4_clear_ht_bit(u64 co
 
 static inline int p4_ht_active(void)
 {
-#ifdef CONFIG_SMP
-	return smp_num_siblings > 1;
-#endif
-	return 0;
+	return topology_smt_supported();
 }
 
 static inline int p4_ht_thread(int cpu)
 {
-#ifdef CONFIG_SMP
-	if (smp_num_siblings == 2)
-		return cpu != cpumask_first(this_cpu_cpumask_var_ptr(cpu_sibling_map));
-#endif
-	return 0;
+	return !topology_is_primary_thread(cpu);
 }
 
 static inline int p4_should_swap_ts(u64 config, int cpu)
--- a/arch/x86/kernel/process.c
+++ b/arch/x86/kernel/process.c
@@ -924,7 +924,7 @@ static __cpuidle void mwait_idle(void)
 void select_idle_routine(const struct cpuinfo_x86 *c)
 {
 #ifdef CONFIG_SMP
-	if (boot_option_idle_override == IDLE_POLL && smp_num_siblings > 1)
+	if (boot_option_idle_override == IDLE_POLL && topology_smt_supported())
 		pr_warn_once("WARNING: polling idle and HT enabled, performance may degrade\n");
 #endif
 	if (x86_idle_set() || boot_option_idle_override == IDLE_POLL)
--- a/arch/x86/kernel/smpboot.c
+++ b/arch/x86/kernel/smpboot.c
@@ -573,7 +573,7 @@ static void __init build_sched_topology(
 
 void set_cpu_sibling_map(int cpu)
 {
-	bool has_smt = smp_num_siblings > 1;
+	bool has_smt = topology_smt_supported();
 	bool has_mp = has_smt || boot_cpu_data.x86_max_cores > 1;
 	struct cpuinfo_x86 *c = &cpu_data(cpu);
 	struct cpuinfo_x86 *o;

[patch 50/53] x86/cpu/topology: Rename smp_num_siblings

[Thomas Gleixner]

It's really a non-intuitive name. Rename it to __max_threads_per_core which
is obvious.

Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
---
 arch/x86/include/asm/smp.h       |    2 --
 arch/x86/include/asm/topology.h  |    6 +++++-
 arch/x86/kernel/cpu/common.c     |    4 ++--
 arch/x86/kernel/cpu/debugfs.c    |    2 +-
 arch/x86/kernel/cpu/mce/inject.c |    2 +-
 arch/x86/kernel/cpu/topology.c   |    6 +++---
 arch/x86/kernel/smpboot.c        |    8 --------
 7 files changed, 12 insertions(+), 18 deletions(-)

--- a/arch/x86/include/asm/smp.h
+++ b/arch/x86/include/asm/smp.h
@@ -8,8 +8,6 @@
 #include <asm/current.h>
 #include <asm/thread_info.h>
 
-extern unsigned int smp_num_siblings;
-
 DECLARE_PER_CPU_READ_MOSTLY(cpumask_var_t, cpu_sibling_map);
 DECLARE_PER_CPU_READ_MOSTLY(cpumask_var_t, cpu_core_map);
 DECLARE_PER_CPU_READ_MOSTLY(cpumask_var_t, cpu_die_map);
--- a/arch/x86/include/asm/topology.h
+++ b/arch/x86/include/asm/topology.h
@@ -145,6 +145,7 @@ extern const struct cpumask *cpu_cluster
 
 extern unsigned int __max_dies_per_package;
 extern unsigned int __max_logical_packages;
+extern unsigned int __max_threads_per_core;
 
 static inline unsigned int topology_max_packages(void)
 {
@@ -178,7 +179,10 @@ static inline int topology_max_smt_threa
 	return __max_smt_threads;
 }
 
-bool topology_smt_supported(void);
+static inline bool topology_smt_supported(void)
+{
+	return __max_threads_per_core > 1;
+}
 
 extern unsigned int __amd_nodes_per_pkg;
 
--- a/arch/x86/kernel/cpu/common.c
+++ b/arch/x86/kernel/cpu/common.c
@@ -72,8 +72,8 @@
 u32 elf_hwcap2 __read_mostly;
 
 /* Number of siblings per CPU package */
-unsigned int smp_num_siblings __ro_after_init = 1;
-EXPORT_SYMBOL(smp_num_siblings);
+unsigned int __max_threads_per_core __ro_after_init = 1;
+EXPORT_SYMBOL(__max_threads_per_core);
 
 unsigned int __max_dies_per_package __ro_after_init = 1;
 EXPORT_SYMBOL(__max_dies_per_package);
--- a/arch/x86/kernel/cpu/debugfs.c
+++ b/arch/x86/kernel/cpu/debugfs.c
@@ -30,7 +30,7 @@ static int cpu_debug_show(struct seq_fil
 	seq_printf(m, "amd_nodes_per_pkg:   %u\n", topology_amd_nodes_per_pkg());
 	seq_printf(m, "max_cores:           %u\n", c->x86_max_cores);
 	seq_printf(m, "max_dies_per_pkg:    %u\n", __max_dies_per_package);
-	seq_printf(m, "smp_num_siblings:    %u\n", smp_num_siblings);
+	seq_printf(m, "max_threads_per_core:%u\n", __max_threads_per_core);
 	return 0;
 }
 
--- a/arch/x86/kernel/cpu/mce/inject.c
+++ b/arch/x86/kernel/cpu/mce/inject.c
@@ -433,7 +433,7 @@ static u32 get_nbc_for_node(int node_id)
 	struct cpuinfo_x86 *c = &boot_cpu_data;
 	u32 cores_per_node;
 
-	cores_per_node = (c->x86_max_cores * smp_num_siblings) / topology_amd_nodes_per_pkg();
+	cores_per_node = (c->x86_max_cores * __max_threads_per_core) / topology_amd_nodes_per_pkg();
 	return cores_per_node * node_id;
 }
 
--- a/arch/x86/kernel/cpu/topology.c
+++ b/arch/x86/kernel/cpu/topology.c
@@ -86,7 +86,7 @@ bool arch_match_cpu_phys_id(int cpu, u64
 #ifdef CONFIG_SMP
 static void cpu_mark_primary_thread(unsigned int cpu, unsigned int apicid)
 {
-	if (!(apicid & (smp_num_siblings - 1)))
+	if (!(apicid & (__max_threads_per_core - 1)))
 		cpumask_set_cpu(cpu, &__cpu_primary_thread_mask);
 }
 #else
@@ -427,8 +427,8 @@ void __init topology_init_possible_cpus(
 
 	cnta = domain_weight(TOPO_CORE_DOMAIN);
 	cntb = domain_weight(TOPO_SMT_DOMAIN);
-	smp_num_siblings = 1U << (get_count_order(cntb) - get_count_order(cnta));
-	pr_info("Max. threads per core: %3u\n", smp_num_siblings);
+	__max_threads_per_core = 1U << (get_count_order(cntb) - get_count_order(cnta));
+	pr_info("Max. threads per core: %3u\n", __max_threads_per_core);
 
 	pr_info("Allowing %u present CPUs plus %u hotplug CPUs\n", assigned, disabled);
 	if (topo_info.nr_rejected_cpus)
--- a/arch/x86/kernel/smpboot.c
+++ b/arch/x86/kernel/smpboot.c
@@ -320,14 +320,6 @@ static void notrace start_secondary(void
 	cpu_startup_entry(CPUHP_AP_ONLINE_IDLE);
 }
 
-/**
- * topology_smt_supported - Check whether SMT is supported by the CPUs
- */
-bool topology_smt_supported(void)
-{
-	return smp_num_siblings > 1;
-}
-
 void __init smp_store_boot_cpu_info(void)
 {
 	struct cpuinfo_x86 *c = &cpu_data(0);

[patch 51/53] x86/cpu/topology: Rename topology_max_die_per_package()

[Thomas Gleixner]

The plural of die is dies.

Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
---
 arch/x86/events/intel/cstate.c                                 |    2 +-
 arch/x86/events/intel/uncore.c                                 |    2 +-
 arch/x86/events/intel/uncore_snbep.c                           |    2 +-
 arch/x86/events/rapl.c                                         |    2 +-
 arch/x86/include/asm/topology.h                                |    2 +-
 drivers/hwmon/coretemp.c                                       |    2 +-
 drivers/platform/x86/intel/uncore-frequency/uncore-frequency.c |    2 +-
 drivers/powercap/intel_rapl_common.c                           |    2 +-
 drivers/thermal/intel/intel_hfi.c                              |    2 +-
 drivers/thermal/intel/intel_powerclamp.c                       |    2 +-
 drivers/thermal/intel/x86_pkg_temp_thermal.c                   |    2 +-
 11 files changed, 11 insertions(+), 11 deletions(-)

--- a/arch/x86/events/intel/cstate.c
+++ b/arch/x86/events/intel/cstate.c
@@ -751,7 +751,7 @@ static int __init cstate_init(void)
 	}
 
 	if (has_cstate_pkg) {
-		if (topology_max_die_per_package() > 1) {
+		if (topology_max_dies_per_package() > 1) {
 			err = perf_pmu_register(&cstate_pkg_pmu,
 						"cstate_die", -1);
 		} else {
--- a/arch/x86/events/intel/uncore.c
+++ b/arch/x86/events/intel/uncore.c
@@ -1881,7 +1881,7 @@ static int __init intel_uncore_init(void
 		return -ENODEV;
 
 	__uncore_max_dies =
-		topology_max_packages() * topology_max_die_per_package();
+		topology_max_packages() * topology_max_dies_per_package();
 
 	id = x86_match_cpu(intel_uncore_match);
 	if (!id) {
--- a/arch/x86/events/intel/uncore_snbep.c
+++ b/arch/x86/events/intel/uncore_snbep.c
@@ -1441,7 +1441,7 @@ static int snbep_pci2phy_map_init(int de
 			 */
 			for (i = 0; i < 8; i++) {
 				if (nodeid == GIDNIDMAP(config, i)) {
-					if (topology_max_die_per_package() > 1)
+					if (topology_max_dies_per_package() > 1)
 						die_id = i;
 					else
 						die_id = topology_phys_to_logical_pkg(i);
--- a/arch/x86/events/rapl.c
+++ b/arch/x86/events/rapl.c
@@ -678,7 +678,7 @@ static const struct attribute_group *rap
 
 static int __init init_rapl_pmus(void)
 {
-	int maxdie = topology_max_packages() * topology_max_die_per_package();
+	int maxdie = topology_max_packages() * topology_max_dies_per_package();
 	size_t size;
 
 	size = sizeof(*rapl_pmus) + maxdie * sizeof(struct rapl_pmu *);
--- a/arch/x86/include/asm/topology.h
+++ b/arch/x86/include/asm/topology.h
@@ -152,7 +152,7 @@ static inline unsigned int topology_max_
 	return __max_logical_packages;
 }
 
-static inline unsigned int topology_max_die_per_package(void)
+static inline unsigned int topology_max_dies_per_package(void)
 {
 	return __max_dies_per_package;
 }
--- a/drivers/hwmon/coretemp.c
+++ b/drivers/hwmon/coretemp.c
@@ -780,7 +780,7 @@ static int __init coretemp_init(void)
 	if (!x86_match_cpu(coretemp_ids))
 		return -ENODEV;
 
-	max_zones = topology_max_packages() * topology_max_die_per_package();
+	max_zones = topology_max_packages() * topology_max_dies_per_package();
 	zone_devices = kcalloc(max_zones, sizeof(struct platform_device *),
 			      GFP_KERNEL);
 	if (!zone_devices)
--- a/drivers/platform/x86/intel/uncore-frequency/uncore-frequency.c
+++ b/drivers/platform/x86/intel/uncore-frequency/uncore-frequency.c
@@ -229,7 +229,7 @@ static int __init intel_uncore_init(void
 		return -ENODEV;
 
 	uncore_max_entries = topology_max_packages() *
-					topology_max_die_per_package();
+					topology_max_dies_per_package();
 	uncore_instances = kcalloc(uncore_max_entries,
 				   sizeof(*uncore_instances), GFP_KERNEL);
 	if (!uncore_instances)
--- a/drivers/powercap/intel_rapl_common.c
+++ b/drivers/powercap/intel_rapl_common.c
@@ -1566,7 +1566,7 @@ struct rapl_package *rapl_add_package(in
 	if (id_is_cpu) {
 		rp->id = topology_logical_die_id(id);
 		rp->lead_cpu = id;
-		if (topology_max_die_per_package() > 1)
+		if (topology_max_dies_per_package() > 1)
 			snprintf(rp->name, PACKAGE_DOMAIN_NAME_LENGTH, "package-%d-die-%d",
 				 topology_physical_package_id(id), topology_die_id(id));
 		else
--- a/drivers/thermal/intel/intel_hfi.c
+++ b/drivers/thermal/intel/intel_hfi.c
@@ -542,7 +542,7 @@ void __init intel_hfi_init(void)
 
 	/* There is one HFI instance per die/package. */
 	max_hfi_instances = topology_max_packages() *
-			    topology_max_die_per_package();
+			    topology_max_dies_per_package();
 
 	/*
 	 * This allocation may fail. CPU hotplug callbacks must check
--- a/drivers/thermal/intel/intel_powerclamp.c
+++ b/drivers/thermal/intel/intel_powerclamp.c
@@ -616,7 +616,7 @@ static int powerclamp_idle_injection_reg
 	poll_pkg_cstate_enable = false;
 	if (cpumask_equal(cpu_present_mask, idle_injection_cpu_mask)) {
 		ii_dev = idle_inject_register_full(idle_injection_cpu_mask, idle_inject_update);
-		if (topology_max_packages() == 1 && topology_max_die_per_package() == 1)
+		if (topology_max_packages() == 1 && topology_max_dies_per_package() == 1)
 			poll_pkg_cstate_enable = true;
 	} else {
 		ii_dev = idle_inject_register(idle_injection_cpu_mask);
--- a/drivers/thermal/intel/x86_pkg_temp_thermal.c
+++ b/drivers/thermal/intel/x86_pkg_temp_thermal.c
@@ -494,7 +494,7 @@ static int __init pkg_temp_thermal_init(
 	if (!x86_match_cpu(pkg_temp_thermal_ids))
 		return -ENODEV;
 
-	max_id = topology_max_packages() * topology_max_die_per_package();
+	max_id = topology_max_packages() * topology_max_dies_per_package();
 	zones = kcalloc(max_id, sizeof(struct zone_device *),
 			   GFP_KERNEL);
 	if (!zones)

[patch 52/53] x86/cpu/topology: Provide __num_[cores|threads]_per_package

[Thomas Gleixner]

Expose properly accounted information and accessors so the fiddling with
other topology variables can be replaced.

Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
---
 arch/x86/include/asm/topology.h |   12 ++++++++++++
 arch/x86/kernel/cpu/common.c    |    6 ++++++
 arch/x86/kernel/cpu/topology.c  |    8 +++++++-
 3 files changed, 25 insertions(+), 1 deletion(-)

--- a/arch/x86/include/asm/topology.h
+++ b/arch/x86/include/asm/topology.h
@@ -146,6 +146,8 @@ extern const struct cpumask *cpu_cluster
 extern unsigned int __max_dies_per_package;
 extern unsigned int __max_logical_packages;
 extern unsigned int __max_threads_per_core;
+extern unsigned int __num_threads_per_package;
+extern unsigned int __num_cores_per_package;
 
 static inline unsigned int topology_max_packages(void)
 {
@@ -157,6 +159,16 @@ static inline unsigned int topology_max_
 	return __max_dies_per_package;
 }
 
+static inline unsigned int topology_num_cores_per_package(void)
+{
+	return __num_cores_per_package;
+}
+
+static inline unsigned int topology_num_threads_per_package(void)
+{
+	return __num_threads_per_package;
+}
+
 int topology_get_logical_id(u32 apicid, enum x86_topology_domains at_level);
 
 static inline int topology_phys_to_logical_pkg(unsigned int pkg)
--- a/arch/x86/kernel/cpu/common.c
+++ b/arch/x86/kernel/cpu/common.c
@@ -81,6 +81,12 @@ EXPORT_SYMBOL(__max_dies_per_package);
 unsigned int __max_logical_packages __ro_after_init = 1;
 EXPORT_SYMBOL(__max_logical_packages);
 
+unsigned int __num_cores_per_package __ro_after_init = 1;
+EXPORT_SYMBOL(__num_cores_per_package);
+
+unsigned int __num_threads_per_package __ro_after_init = 1;
+EXPORT_SYMBOL(__num_threads_per_package);
+
 static struct ppin_info {
 	int	feature;
 	int	msr_ppin_ctl;
--- a/arch/x86/kernel/cpu/topology.c
+++ b/arch/x86/kernel/cpu/topology.c
@@ -391,7 +391,7 @@ void __init topology_init_possible_cpus(
 	unsigned int disabled = topo_info.nr_disabled_cpus;
 	unsigned int cnta, cntb, cpu, dom, allowed = 1;
 	unsigned int total = assigned + disabled;
-	u32 apicid;
+	u32 apicid, firstid;
 
 	if (!restrict_to_up()) {
 		if (total > 1)
@@ -430,6 +430,12 @@ void __init topology_init_possible_cpus(
 	__max_threads_per_core = 1U << (get_count_order(cntb) - get_count_order(cnta));
 	pr_info("Max. threads per core: %3u\n", __max_threads_per_core);
 
+	firstid = find_first_bit(apic_maps[TOPO_SMT_DOMAIN].map, MAX_LOCAL_APIC);
+	__num_cores_per_package = topology_unit_count(firstid, TOPO_CORE_DOMAIN, TOPO_PKG_DOMAIN);
+	pr_info("Num. cores per package:   %3u\n", __num_cores_per_package);
+	__num_threads_per_package = topology_unit_count(firstid, TOPO_SMT_DOMAIN, TOPO_PKG_DOMAIN);
+	pr_info("Num. threads per package: %3u\n", __num_threads_per_package);
+
 	pr_info("Allowing %u present CPUs plus %u hotplug CPUs\n", assigned, disabled);
 	if (topo_info.nr_rejected_cpus)
 		pr_info("Rejected CPUs %u\n", topo_info.nr_rejected_cpus);

[patch 53/53] x86/cpu/topology: Get rid of cpuinfo::x86_max_cores

[Thomas Gleixner]

Now that __num_cores_per_package and __num_threads_per_package are
available, cpuinfo::x86_max_cores and the related math all over the place
can be replaced with the ready to consume data.

Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
---
 Documentation/arch/x86/topology.rst              |   24 ++++++++---------------
 arch/x86/events/intel/uncore_nhmex.c             |    4 +--
 arch/x86/events/intel/uncore_snb.c               |    8 +++----
 arch/x86/events/intel/uncore_snbep.c             |   16 +++++++--------
 arch/x86/include/asm/processor.h                 |    2 -
 arch/x86/kernel/cpu/cacheinfo.c                  |    2 -
 arch/x86/kernel/cpu/common.c                     |    1 
 arch/x86/kernel/cpu/debugfs.c                    |    3 +-
 arch/x86/kernel/cpu/mce/inject.c                 |    3 --
 arch/x86/kernel/cpu/microcode/intel.c            |    4 +--
 arch/x86/kernel/cpu/topology_common.c            |    3 --
 arch/x86/kernel/smpboot.c                        |    2 -
 drivers/gpu/drm/amd/pm/swsmu/smu11/vangogh_ppt.c |    2 -
 drivers/hwmon/fam15h_power.c                     |    2 -
 14 files changed, 32 insertions(+), 44 deletions(-)

--- a/Documentation/arch/x86/topology.rst
+++ b/Documentation/arch/x86/topology.rst
@@ -47,17 +47,21 @@ AMD nomenclature for package is 'Node'.
 
 Package-related topology information in the kernel:
 
-  - cpuinfo_x86.x86_max_cores:
+  - topology_num_threads_per_package()
 
-    The number of cores in a package. This information is retrieved via CPUID.
+    The number of threads in a package.
 
-  - cpuinfo_x86.x86_max_dies:
+  - topology_num_cores_per_package()
 
-    The number of dies in a package. This information is retrieved via CPUID.
+    The number of cores in a package.
+
+  - topology_max_dies_per_package()
+
+    The maximum number of dies in a package.
 
   - cpuinfo_x86.topo_die_id:
 
-    The physical ID of the die. This information is retrieved via CPUID.
+    The physical ID of the die.
 
   - cpuinfo_x86.topo.pkg_id:
 
@@ -96,16 +100,6 @@ are SMT- or CMT-type threads.
 AMDs nomenclature for a CMT core is "Compute Unit". The kernel always uses
 "core".
 
-Core-related topology information in the kernel:
-
-  - smp_num_siblings:
-
-    The number of threads in a core. The number of threads in a package can be
-    calculated by::
-
-	threads_per_package = cpuinfo_x86.x86_max_cores * smp_num_siblings
-
-
 Threads
 =======
 A thread is a single scheduling unit. It's the equivalent to a logical Linux
--- a/arch/x86/events/intel/uncore_nhmex.c
+++ b/arch/x86/events/intel/uncore_nhmex.c
@@ -1221,8 +1221,8 @@ void nhmex_uncore_cpu_init(void)
 		uncore_nhmex = true;
 	else
 		nhmex_uncore_mbox.event_descs = wsmex_uncore_mbox_events;
-	if (nhmex_uncore_cbox.num_boxes > boot_cpu_data.x86_max_cores)
-		nhmex_uncore_cbox.num_boxes = boot_cpu_data.x86_max_cores;
+	if (nhmex_uncore_cbox.num_boxes > topology_num_cores_per_package())
+		nhmex_uncore_cbox.num_boxes = topology_num_cores_per_package();
 	uncore_msr_uncores = nhmex_msr_uncores;
 }
 /* end of Nehalem-EX uncore support */
--- a/arch/x86/events/intel/uncore_snb.c
+++ b/arch/x86/events/intel/uncore_snb.c
@@ -364,8 +364,8 @@ static struct intel_uncore_type *snb_msr
 void snb_uncore_cpu_init(void)
 {
 	uncore_msr_uncores = snb_msr_uncores;
-	if (snb_uncore_cbox.num_boxes > boot_cpu_data.x86_max_cores)
-		snb_uncore_cbox.num_boxes = boot_cpu_data.x86_max_cores;
+	if (snb_uncore_cbox.num_boxes > topology_num_cores_per_package())
+		snb_uncore_cbox.num_boxes = topology_num_cores_per_package();
 }
 
 static void skl_uncore_msr_init_box(struct intel_uncore_box *box)
@@ -428,8 +428,8 @@ static struct intel_uncore_type *skl_msr
 void skl_uncore_cpu_init(void)
 {
 	uncore_msr_uncores = skl_msr_uncores;
-	if (skl_uncore_cbox.num_boxes > boot_cpu_data.x86_max_cores)
-		skl_uncore_cbox.num_boxes = boot_cpu_data.x86_max_cores;
+	if (skl_uncore_cbox.num_boxes > topology_num_cores_per_package())
+		skl_uncore_cbox.num_boxes = topology_num_cores_per_package();
 	snb_uncore_arb.ops = &skl_uncore_msr_ops;
 }
 
--- a/arch/x86/events/intel/uncore_snbep.c
+++ b/arch/x86/events/intel/uncore_snbep.c
@@ -1172,8 +1172,8 @@ static struct intel_uncore_type *snbep_m
 
 void snbep_uncore_cpu_init(void)
 {
-	if (snbep_uncore_cbox.num_boxes > boot_cpu_data.x86_max_cores)
-		snbep_uncore_cbox.num_boxes = boot_cpu_data.x86_max_cores;
+	if (snbep_uncore_cbox.num_boxes > topology_num_cores_per_package())
+		snbep_uncore_cbox.num_boxes = topology_num_cores_per_package();
 	uncore_msr_uncores = snbep_msr_uncores;
 }
 
@@ -1837,8 +1837,8 @@ static struct intel_uncore_type *ivbep_m
 
 void ivbep_uncore_cpu_init(void)
 {
-	if (ivbep_uncore_cbox.num_boxes > boot_cpu_data.x86_max_cores)
-		ivbep_uncore_cbox.num_boxes = boot_cpu_data.x86_max_cores;
+	if (ivbep_uncore_cbox.num_boxes > topology_num_cores_per_package())
+		ivbep_uncore_cbox.num_boxes = topology_num_cores_per_package();
 	uncore_msr_uncores = ivbep_msr_uncores;
 }
 
@@ -2909,8 +2909,8 @@ static bool hswep_has_limit_sbox(unsigne
 
 void hswep_uncore_cpu_init(void)
 {
-	if (hswep_uncore_cbox.num_boxes > boot_cpu_data.x86_max_cores)
-		hswep_uncore_cbox.num_boxes = boot_cpu_data.x86_max_cores;
+	if (hswep_uncore_cbox.num_boxes > topology_num_cores_per_package())
+		hswep_uncore_cbox.num_boxes = topology_num_cores_per_package();
 
 	/* Detect 6-8 core systems with only two SBOXes */
 	if (hswep_has_limit_sbox(HSWEP_PCU_DID))
@@ -3272,8 +3272,8 @@ static struct event_constraint bdx_uncor
 
 void bdx_uncore_cpu_init(void)
 {
-	if (bdx_uncore_cbox.num_boxes > boot_cpu_data.x86_max_cores)
-		bdx_uncore_cbox.num_boxes = boot_cpu_data.x86_max_cores;
+	if (bdx_uncore_cbox.num_boxes > topology_num_cores_per_package())
+		bdx_uncore_cbox.num_boxes = topology_num_cores_per_package();
 	uncore_msr_uncores = bdx_msr_uncores;
 
 	/* Detect systems with no SBOXes */
--- a/arch/x86/include/asm/processor.h
+++ b/arch/x86/include/asm/processor.h
@@ -148,8 +148,6 @@ struct cpuinfo_x86 {
 	unsigned long		loops_per_jiffy;
 	/* protected processor identification number */
 	u64			ppin;
-	/* cpuid returned max cores value: */
-	u16			x86_max_cores;
 	u16			x86_clflush_size;
 	/* number of cores as seen by the OS: */
 	u16			booted_cores;
--- a/arch/x86/kernel/cpu/cacheinfo.c
+++ b/arch/x86/kernel/cpu/cacheinfo.c
@@ -301,7 +301,7 @@ amd_cpuid4(int leaf, union _cpuid4_leaf_
 	eax->split.type = types[leaf];
 	eax->split.level = levels[leaf];
 	eax->split.num_threads_sharing = 0;
-	eax->split.num_cores_on_die = __this_cpu_read(cpu_info.x86_max_cores) - 1;
+	eax->split.num_cores_on_die = topology_num_cores_per_package();
 
 
 	if (assoc == 0xffff)
--- a/arch/x86/kernel/cpu/common.c
+++ b/arch/x86/kernel/cpu/common.c
@@ -1698,7 +1698,6 @@ static void identify_cpu(struct cpuinfo_
 	c->x86_model = c->x86_stepping = 0;	/* So far unknown... */
 	c->x86_vendor_id[0] = '\0'; /* Unset */
 	c->x86_model_id[0] = '\0';  /* Unset */
-	c->x86_max_cores = 1;
 #ifdef CONFIG_X86_64
 	c->x86_clflush_size = 64;
 	c->x86_phys_bits = 36;
--- a/arch/x86/kernel/cpu/debugfs.c
+++ b/arch/x86/kernel/cpu/debugfs.c
@@ -28,7 +28,8 @@ static int cpu_debug_show(struct seq_fil
 	seq_printf(m, "l2c_id:              %u\n", c->topo.l2c_id);
 	seq_printf(m, "amd_node_id:         %u\n", c->topo.amd_node_id);
 	seq_printf(m, "amd_nodes_per_pkg:   %u\n", topology_amd_nodes_per_pkg());
-	seq_printf(m, "max_cores:           %u\n", c->x86_max_cores);
+	seq_printf(m, "num_threads:         %u\n", __num_threads_per_package);
+	seq_printf(m, "num_cores:           %u\n", __num_cores_per_package);
 	seq_printf(m, "max_dies_per_pkg:    %u\n", __max_dies_per_package);
 	seq_printf(m, "max_threads_per_core:%u\n", __max_threads_per_core);
 	return 0;
--- a/arch/x86/kernel/cpu/mce/inject.c
+++ b/arch/x86/kernel/cpu/mce/inject.c
@@ -430,10 +430,9 @@ static void trigger_thr_int(void *info)
 
 static u32 get_nbc_for_node(int node_id)
 {
-	struct cpuinfo_x86 *c = &boot_cpu_data;
 	u32 cores_per_node;
 
-	cores_per_node = (c->x86_max_cores * __max_threads_per_core) / topology_amd_nodes_per_pkg();
+	cores_per_node = topology_num_threads_per_package() / topology_amd_nodes_per_pkg();
 	return cores_per_node * node_id;
 }
 
--- a/arch/x86/kernel/cpu/microcode/intel.c
+++ b/arch/x86/kernel/cpu/microcode/intel.c
@@ -43,7 +43,7 @@ static const char ucode_path[] = "kernel
 static struct microcode_intel *intel_ucode_patch;
 
 /* last level cache size per core */
-static int llc_size_per_core;
+static unsigned long llc_size_per_core;
 
 /*
  * Returns 1 if update has been found, 0 otherwise.
@@ -777,7 +777,7 @@ static int __init calc_llc_size_per_core
 {
 	u64 llc_size = c->x86_cache_size * 1024ULL;
 
-	do_div(llc_size, c->x86_max_cores);
+	do_div(llc_size, topology_num_cores_per_package());
 
 	return (int)llc_size;
 }
--- a/arch/x86/kernel/cpu/topology_common.c
+++ b/arch/x86/kernel/cpu/topology_common.c
@@ -154,9 +154,6 @@ static void topo_set_ids(struct topo_sca
 	/* Relative core ID */
 	c->topo.core_id = topo_relative_domain_id(apicid, TOPO_CORE_DOMAIN);
 
-	/* Maximum number of cores on this package */
-	c->x86_max_cores = topology_unit_count(apicid, TOPO_CORE_DOMAIN, TOPO_PKG_DOMAIN);
-
 	c->topo.amd_node_id = tscan->amd_node_id;
 
 	if (c->x86_vendor == X86_VENDOR_AMD)
--- a/arch/x86/kernel/smpboot.c
+++ b/arch/x86/kernel/smpboot.c
@@ -566,7 +566,7 @@ static void __init build_sched_topology(
 void set_cpu_sibling_map(int cpu)
 {
 	bool has_smt = topology_smt_supported();
-	bool has_mp = has_smt || boot_cpu_data.x86_max_cores > 1;
+	bool has_mp = has_smt || topology_num_cores_per_package() > 1;
 	struct cpuinfo_x86 *c = &cpu_data(cpu);
 	struct cpuinfo_x86 *o;
 	int i, threads;
--- a/drivers/gpu/drm/amd/pm/swsmu/smu11/vangogh_ppt.c
+++ b/drivers/gpu/drm/amd/pm/swsmu/smu11/vangogh_ppt.c
@@ -472,7 +472,7 @@ static int vangogh_init_smc_tables(struc
 
 #ifdef CONFIG_X86
 	/* AMD x86 APU only */
-	smu->cpu_core_num = boot_cpu_data.x86_max_cores;
+	smu->cpu_core_num = topology_num_cores_per_package();
 #else
 	smu->cpu_core_num = 4;
 #endif
--- a/drivers/hwmon/fam15h_power.c
+++ b/drivers/hwmon/fam15h_power.c
@@ -209,7 +209,7 @@ static ssize_t power1_average_show(struc
 	 * With the new x86 topology modelling, x86_max_cores is the
 	 * compute unit number.
 	 */
-	cu_num = boot_cpu_data.x86_max_cores;
+	cu_num = topology_num_cores_per_package();
 
 	ret = read_registers(data);
 	if (ret)

Re: [patch 02/53] x86/cpu/topology: Make the APIC mismatch warnings complete

[Arjan van de Ven]

On 8/7/2023 6:52 AM, T
>   
> --- a/arch/x86/kernel/cpu/topology_common.c
> +++ b/arch/x86/kernel/cpu/topology_common.c
> @@ -176,6 +176,16 @@ void cpu_parse_topology(struct cpuinfo_x
>   
>   	parse_topology(&tscan, false);
>   
> +	if (c->topo.initial_apicid != c->topo.apicid) {
> +		pr_err(FW_BUG "CPU%4u: APIC ID mismatch. CPUID: 0x%04x APIC: 0x%04x\n",
> +		       cpu, c->topo.initial_apicid, c->topo.apicid);
> +	}
> +
> +	if (c->topo.apicid != cpuid_to_apicid[cpu]) {
> +		pr_err(FW_BUG "CPU%4u: APIC ID mismatch. Firmware: 0x%04x APIC: 0x%04x\n",
> +		       cpu, cpuid_to_apicid[cpu], c->topo.apicid);
> +	}
> +

while these messages are basically the same as current ones they are short one key thing for the user
... which one of the two will be used. Yes one can look up in the source code where the message comes from
and reverse engineer that... or we can just add this to these pr_err() messages


like

pr_err(FW_BUG "CPU%4u: APIC ID mismatch. CPUID: 0x%04x APIC: 0x%04x. APIC value will be used.\n",
		       cpu, c->topo.initial_apicid, c->topo.apicid);


>   	for (dom = TOPO_SMT_DOMAIN; dom < TOPO_MAX_DOMAIN; dom++) {
>   		if (tscan.dom_shifts[dom] == x86_topo_system.dom_shifts[dom])
>   			continue;
> 

Re: [patch 02/53] x86/cpu/topology: Make the APIC mismatch warnings complete

[Thomas Gleixner]

On Mon, Aug 07 2023 at 07:28, Arjan van de Ven wrote:

> On 8/7/2023 6:52 AM, T
>>   
>> --- a/arch/x86/kernel/cpu/topology_common.c
>> +++ b/arch/x86/kernel/cpu/topology_common.c
>> @@ -176,6 +176,16 @@ void cpu_parse_topology(struct cpuinfo_x
>>   
>>   	parse_topology(&tscan, false);
>>   
>> +	if (c->topo.initial_apicid != c->topo.apicid) {
>> +		pr_err(FW_BUG "CPU%4u: APIC ID mismatch. CPUID: 0x%04x APIC: 0x%04x\n",
>> +		       cpu, c->topo.initial_apicid, c->topo.apicid);
>> +	}
>> +
>> +	if (c->topo.apicid != cpuid_to_apicid[cpu]) {
>> +		pr_err(FW_BUG "CPU%4u: APIC ID mismatch. Firmware: 0x%04x APIC: 0x%04x\n",
>> +		       cpu, cpuid_to_apicid[cpu], c->topo.apicid);
>> +	}
>> +
>
> while these messages are basically the same as current ones they are short one key thing for the user
> ... which one of the two will be used. Yes one can look up in the source code where the message comes from
> and reverse engineer that... or we can just add this to these pr_err() messages
>
>
> like
>
> pr_err(FW_BUG "CPU%4u: APIC ID mismatch. CPUID: 0x%04x APIC: 0x%04x. APIC value will be used.\n",
> 		       cpu, c->topo.initial_apicid, c->topo.apicid);

Good point.

Re: [patch 03/53] x86/platform/ce4100: Dont override x86_init.mpparse.setup_ioapic_ids

[Andy Shevchenko]

On Mon, Aug 7, 2023 at 4:52 PM Thomas Gleixner <tglx@linutronix.de> wrote:
>
> There is no point to do that. The ATOMs have an XAPIC for which this
> function is a pointless exercise.

CE4x00 are DT based platforms. I never looked at them deeply, so I
don't know how this affects DT parsing on x86.

I have one system at hand, but it has a signed BIOS and I have had no
time to actually break through it and be able to boot a newer kernel
to test...

-- 
With Best Regards,
Andy Shevchenko

Re: [patch 05/53] x86/apic: Get rid of get_physical_broadcast()

[Andy Shevchenko]

On Mon, Aug 7, 2023 at 4:52 PM Thomas Gleixner <tglx@linutronix.de> wrote:
>
> There is no point for this function. The only case where this is used when

used is when

> there is no XAPIC available, which means the broadcast address is 0xF.

...

> +               if (mpc_ioapic_id(ioapic_idx) >= broadcast_id) {
> +                       pr_err("BIOS bug, IO-APIC#%d ID is %d in the MPC table!...\n",

Wouldn't it be better to switch to FW_BUG instead of the "BIOS bug,  " thingy?
Same for other similar cases.

> +                              ioapic_idx, mpc_ioapic_id(ioapic_idx));
> +                       pr_err("... fixing up to %d. (tell your hw vendor)\n", reg_00.bits.ID);
>                         ioapics[ioapic_idx].mp_config.apicid = reg_00.bits.ID;
>                 }

...

> -       if ((boot_cpu_data.x86_vendor == X86_VENDOR_INTEL) &&
> -           !APIC_XAPIC(boot_cpu_apic_version))
> +       if ((boot_cpu_data.x86_vendor == X86_VENDOR_INTEL) && !APIC_XAPIC(boot_cpu_apic_version))
>                 return io_apic_get_unique_id(idx, id);

>         else

You can also kill the redundant 'else'.

>                 return id;

-- 
With Best Regards,
Andy Shevchenko

Re: [patch 25/53] x86/acpi: Use new APIC registration functions

[Peter Zijlstra]

On Mon, Aug 07, 2023 at 03:53:12PM +0200, Thomas Gleixner wrote:
> +static __init void acpi_register_lapic(u32 apic_id, u32 acpi_id, bool present)

just stating on the record what I said during review last week: whoever
caused these APIC and ACPI names to go together was a sadist.

Same 4 letters but permuted differently :-/ this is so very hard to keep
straight.

Re: [patch 25/53] x86/acpi: Use new APIC registration functions

[Andrew Cooper]

On 07/08/2023 4:27 pm, Peter Zijlstra wrote:
> On Mon, Aug 07, 2023 at 03:53:12PM +0200, Thomas Gleixner wrote:
>> +static __init void acpi_register_lapic(u32 apic_id, u32 acpi_id, bool present)
> just stating on the record what I said during review last week: whoever
> caused these APIC and ACPI names to go together was a sadist.

My feedback was surprisingly similar.

I still look at that and think it ought to fail to compile.

~Andrew

Re: [patch 25/53] x86/acpi: Use new APIC registration functions

[Thomas Gleixner]

On Mon, Aug 07 2023 at 17:27, Peter Zijlstra wrote:

> On Mon, Aug 07, 2023 at 03:53:12PM +0200, Thomas Gleixner wrote:
>> +static __init void acpi_register_lapic(u32 apic_id, u32 acpi_id, bool present)
>
> just stating on the record what I said during review last week: whoever
> caused these APIC and ACPI names to go together was a sadist.
>
> Same 4 letters but permuted differently :-/ this is so very hard to keep
> straight.

Good thing that the kernel has no use for Atmospheric Pressure Chemical
Ionization otherwise we'd have APCI right next to it.

Re: [patch 09/53] x86/mpparse: Rename default_find_smp_config()

[Andy Shevchenko]

On Mon, Aug 7, 2023 at 4:52 PM Thomas Gleixner <tglx@linutronix.de> wrote:
>
> MPTABLE is not longer the default SMP configuration mechanism.

not --> no

> Rename it to mpparse_find_mptable() because that's what it does.

...

> -#define enable_update_mptable 0

> +#define enable_update_mptable  (0)

Hmm.. What's the point of this change?

-- 
With Best Regards,
Andy Shevchenko

Re: [patch 14/53] x86/platform/intel-mid: Prepare for separate mpparse callbacks

[Andy Shevchenko]

On Mon, Aug 7, 2023 at 4:52 PM Thomas Gleixner <tglx@linutronix.de> wrote:
>
> Initialize the split SMP configuration callbacks with NOOPs as MID is
> strictly ACPI only.

Not really, but we decided to support only ACPI configurations in the kernel.

Acked-by: Andy Shevchenko <andy.shevchenko@gmail.com>

-- 
With Best Regards,
Andy Shevchenko

Re: [patch 09/53] x86/mpparse: Rename default_find_smp_config()

[Thomas Gleixner]

On Mon, Aug 07 2023 at 19:03, Andy Shevchenko wrote:
> On Mon, Aug 7, 2023 at 4:52 PM Thomas Gleixner <tglx@linutronix.de> wrote:
>>
>> MPTABLE is not longer the default SMP configuration mechanism.
>
> not --> no
>
>> Rename it to mpparse_find_mptable() because that's what it does.
>
> ...
>
>> -#define enable_update_mptable 0
>
>> +#define enable_update_mptable  (0)
>
> Hmm.. What's the point of this change?

Good question. Definitely not intentional.

Re: [patch 00/53] x86/topology: The final installment

[Juergen Gross]

[-- Attachment #1.1.1: Type: text/plain, Size: 5486 bytes --]

On 07.08.23 15:52, Thomas Gleixner wrote:
> Hi!
> 
> This is the (for now) last part of reworking topology enumeration and
> management. It's based on the APIC and CPUID rework series which can be
> found here:
> 
>        https://lore.kernel.org/lkml/20230802101635.459108805@linutronix.de
> 
> With these preparatory changes in place, it's now possible to address the
> real issues of the current topology code:
> 
>    - Wrong core count on hybrid systems
> 
>    - Heuristics based size information for packages and dies which
>      are failing to work correctly with certain command line parameters.
> 
>    - Full evaluation fail for a theoretical hybrid system which boots
>      from an E-core
> 
>    - The complete insanity of manipulating global data from firmware parsers
>      or the XEN/PV fake SMP enumeration. The latter is really a piece of art.
> 
> This series addresses this by
> 
>    - Mopping up some more historical technical debt
> 
>    - Consolidating all topology relevant functionality into one place
> 
>    - Providing separate interfaces for boot time and ACPI hotplug operations
> 
>    - A sane ordering of command line options and restrictions
> 
>    - A sensible way to handle the BSP problem in kdump kernels instead of
>      the unreliable command line option.
> 
>    - Confinement of topology relevant variables by replacing the XEN/PV SMP
>      enumeration fake with something halfways sensible.
> 
>    - Evaluation of sizes by analysing the topology via the CPUID provided
>      APIC ID segmentation and the actual APIC IDs which are registered at
>      boot time.
> 
>    - Removal of heuristics and broken size calculations
> 
> The idea behind this is the following:
> 
> The APIC IDs describe the system topology in multiple domain levels. The
> CPUID topology parser provides the information which part of the APIC ID is
> associated to the individual levels (Intel terminology):
> 
>     [ROOT][PACKAGE][DIE][TILE][MODULE][CORE][THREAD]
> 
> The root space contains the package (socket) IDs. Not enumerated levels
> consume 0 bits space, but conceptually they are always represented. If
> e.g. only CORE and THREAD levels are enumerated then the DIE, MODULE and
> TILE have the same physical ID as the PACKAGE.
> 
> If SMT is not supported, then the THREAD domain is still used. It then
> has the same physical ID as the CORE domain and is the only child of
> the core domain.
> 
> This allows an unified view on the system independent of the enumerated
> domain levels without requiring any conditionals in the code.
> 
> AMD does only expose 4 domain levels with obviously different terminology,
> but that can be easily mapped into the Intel variant with a trivial lookup
> table added to the CPUID parser.
> 
> The resulting topology information of an ADL hybrid system with 8 P-Cores
> and 8 E-Cores looks like this:
> 
>   CPU topo: Max. logical packages:   1
>   CPU topo: Max. logical dies:       1
>   CPU topo: Max. dies per package:   1
>   CPU topo: Max. threads per core:   2
>   CPU topo: Num. cores per package:    16
>   CPU topo: Num. threads per package:  24
>   CPU topo: Allowing 24 present CPUs plus 0 hotplug CPUs
>   CPU topo: Thread    :    24
>   CPU topo: Core      :    16
>   CPU topo: Module    :     1
>   CPU topo: Tile      :     1
>   CPU topo: Die       :     1
>   CPU topo: Package   :     1
> 
> This is happening on the boot CPU before any of the APs is started and
> provides correct size information right from the start.
> 
> Even the XEN/PV trainwreck makes use of this now. On Dom0 it utilizes the
> MADT and on DomU it provides fake APIC IDs, which combined with the
> provided CPUID information make it at least look halfways realistic instead
> of claiming to have one CPU per package as the current upstream code does.
> 
> This is solely addressing the core topology issues, but there is a plan for
> further consolidation of other topology related information into one single
> source of information instead of having a gazillion of localized special
> parsers and representations all over the place. There are quite some other
> things which can be simplified on top of this, like updating the various
> cpumasks during CPU bringup, but that's all left for later.
> 
> So another 53 patches later, the resulting diffstat is:
> 
>     64 files changed, 830 insertions(+), 955 deletions(-)
> 
> and the combo diffstat of all three series combined:
> 
>    115 files changed, 2414 insertions(+), 3035 deletions(-)
> 
> The current series applies on top of
> 
>     git://git.kernel.org/pub/scm/linux/kernel/git/tglx/devel.git topo-cpuid-v3
> 
> and is available from git here:
> 
>     git://git.kernel.org/pub/scm/linux/kernel/git/tglx/devel.git topo-full-v1

Tested on an Intel system with Xen:

- PV dom0 is working fine. I couldn't test physical cpu hotplug, but removing
   and then re-adding vcpus to dom0 worked.

- PV domU is working fine, too. A test with starting using 2 vcpus initially
   and onlining another 2 vcpus later was doing fine.

So for Xen PV you can add my:

Tested-by: Juergen Gross <jgross@suse.com>

One other thing to mention: with this series the reported topology via "lscpu"
and "cat /proc/cpuinfo" inside a PV guest/dom0 is looking sane for the first
time. :-)

Thanks for this significant improvement!


Juergen

[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 3149 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 495 bytes --]

Re: [patch 00/53] x86/topology: The final installment

[Andrew Cooper]

On 08/08/2023 8:40 am, Juergen Gross wrote:
> Tested on an Intel system with Xen:
>
> - PV dom0 is working fine. I couldn't test physical cpu hotplug, but
> removing
>   and then re-adding vcpus to dom0 worked.

It turns out that physical CPU hotplug with XenPV is broken in at least
two ways.

It's dom0 (not Xen) that gets the hot-unplug event, after which the Xen
code in Linux succumbs to a preempt-check failure while trying to
offline the vCPU that aliases the pCPU wanting to go offline.

~Andrew

Re: [patch 19/53] x86/mpparse: Remove the physid_t bitmap wrapper

[Andy Shevchenko]

On Mon, Aug 7, 2023 at 4:53 PM Thomas Gleixner <tglx@linutronix.de> wrote:
>
> physid_t is a wrapper around bitmap. Just remove the onion layer and use
> bitmap functionality directly.

...

>  #include <linux/clockchips.h>
>  #include <linux/interrupt.h>
>  #include <linux/memblock.h>

> +#include <linux/bitmap.h>

Hmm... What is the ordering here? Reversed xmas tree?
Otherwise wouldn't it be better to put this before clockchips.h?

>  #include <linux/ftrace.h>
>  #include <linux/ioport.h>
>  #include <linux/export.h>

-- 
With Best Regards,
Andy Shevchenko

Re: [patch 00/53] x86/topology: The final installment

[Sohil Mehta]

On 8/7/2023 6:52 AM, Thomas Gleixner wrote:
> 
> The current series applies on top of 
> 
>    git://git.kernel.org/pub/scm/linux/kernel/git/tglx/devel.git topo-cpuid-v3
> 
> and is available from git here:
> 
>    git://git.kernel.org/pub/scm/linux/kernel/git/tglx/devel.git topo-full-v1
> 

I tested this on a 2S Ivy bridge system with 10 cores per socket
resulting in a total of 20 cores/40 threads. The specifications are
listed at:
https://www.intel.com/content/www/us/en/products/sku/75279/intel-xeon-processor-e52690-v2-25m-cache-3-00-ghz/specifications.html

However, I am a bit confused with the dmesg results.

Dmesg output
------------
CPU topo: Max. logical packages:   4
CPU topo: Max. logical dies:       4
CPU topo: Max. dies per package:   1
CPU topo: Max. threads per core:   2
CPU topo: Num. cores per package:    16
CPU topo: Num. threads per package:  32
CPU topo: Allowing 40 present CPUs plus 120 hotplug CPUs
CPU topo: Thread    :   120
CPU topo: Core      :    60
CPU topo: Module    :     4
CPU topo: Tile      :     4
CPU topo: Die       :     4
CPU topo: Package   :     4

smpboot: x86: Booting SMP configuration:
.... node  #0, CPUs:          #1   #2   #3   #4   #5   #6   #7   #8   #9
.... node  #1, CPUs:    #10  #11  #12  #13  #14  #15  #16  #17  #18  #19
.... node  #0, CPUs:    #20  #21  #22  #23  #24  #25  #26  #27  #28  #29
.... node  #1, CPUs:    #30  #31  #32  #33  #34  #35  #36  #37  #38  #39
smp: Brought up 2 nodes, 40 CPUs
smpboot: Total of 40 processors activated (239426.00 BogoMIPS)

Debugfs
-------
# cat /sys/kernel/debug/x86/topo/cpus/39
online:              1
initial_apicid:      39
apicid:              39
pkg_id:              1
die_id:              1
cu_id:               255
core_id:             12
logical_pkg_id:      1
logical_die_id:      1
llc_id:              32
l2c_id:              56
amd_node_id:         0
amd_nodes_per_pkg:   0
num_threads:         32
num_cores:           16
max_dies_per_pkg:    1
max_threads_per_core:2

lscpu output
------------
Architecture:            x86_64
  CPU op-mode(s):        32-bit, 64-bit
  Address sizes:         46 bits physical, 48 bits virtual
  Byte Order:            Little Endian
CPU(s):                  40
  On-line CPU(s) list:   0-39
Vendor ID:               GenuineIntel
  Model name:            Intel(R) Xeon(R) CPU E5-2690 v2 @ 3.00GHz
    CPU family:          6
    Model:               62
    Thread(s) per core:  2
    Core(s) per socket:  10
    Socket(s):           2
    Stepping:            4
    CPU max MHz:         3600.0000
    CPU min MHz:         1200.0000
    BogoMIPS:            5985.65

Virtualization features:
  Virtualization:        VT-x
Caches (sum of all):
  L1d:                   640 KiB (20 instances)
  L1i:                   640 KiB (20 instances)
  L2:                    5 MiB (20 instances)
  L3:                    50 MiB (2 instances)
NUMA:
  NUMA node(s):          2
  NUMA node0 CPU(s):     0-9,20-29
  NUMA node1 CPU(s):     10-19,30-39


Questions
---------
1) Before this series, the Max logical packages used to be logged as 8
in dmesg. But now it shows up as 4. Is that expected?
To me, it seems that to get to 160 potential CPUs with 10cores/20threads
per package, the Max logical packages should be 8.

2) The "Num. cores per package" is listed as 16 in dmesg but shows as 10
in lscpu. The lscpu one seems correct to me. Why does dmesg show this as
16? I don't believe it refers to Max cores per package either?

3) Should the domain name and weight print be a little more descriptive?
	pr_info("%-10s: %5u\n", domain_names[dom], domain_weight(dom));
Thread    :   120
Core      :    60
Module    :     4
Tile      :     4
Die       :     4
Package   :     4

This seems a little hard to decipher for the end user without some
context. Can you please help explain what does Thread: 120 refer to wrt
this system?

Please let me know if you need any additional info.

Maybe some of these things are not specific to this series. I apologize
for not raising these questions earlier. The new prints in dmesg caught
my eye and I started to wonder what it exactly means.

Thanks,
Sohil

Re: [patch 00/53] x86/topology: The final installment

[Thomas Gleixner]

On Tue, Aug 08 2023 at 12:20, Andrew Cooper wrote:
> On 08/08/2023 8:40 am, Juergen Gross wrote:
>> Tested on an Intel system with Xen:
>>
>> - PV dom0 is working fine. I couldn't test physical cpu hotplug, but
>> removing
>>   and then re-adding vcpus to dom0 worked.
>
> It turns out that physical CPU hotplug with XenPV is broken in at least
> two ways.
>
> It's dom0 (not Xen) that gets the hot-unplug event, after which the Xen
> code in Linux succumbs to a preempt-check failure while trying to
> offline the vCPU that aliases the pCPU wanting to go offline.

That should be gone by now :)

Re: [patch 00/53] x86/topology: The final installment

[Thomas Gleixner]

On Tue, Aug 08 2023 at 11:29, Sohil Mehta wrote:
> On 8/7/2023 6:52 AM, Thomas Gleixner wrote:
> However, I am a bit confused with the dmesg results.
>
> Dmesg output
> ------------
> CPU topo: Max. logical packages:   4
> CPU topo: Max. logical dies:       4
> CPU topo: Max. dies per package:   1
> CPU topo: Max. threads per core:   2
> CPU topo: Num. cores per package:    16
> CPU topo: Num. threads per package:  32

That's indeed weird. Can you please provide:

     - the output of 'cpuid -r'
     - the output of /sys/kernel/debug/x86/topo/domains
     - the APIC IDs of all CPUs (see below patch)

> Questions
> ---------
> 1) Before this series, the Max logical packages used to be logged as 8
> in dmesg. But now it shows up as 4. Is that expected?
> To me, it seems that to get to 160 potential CPUs with 10cores/20threads
> per package, the Max logical packages should be 8.

As the number of cores per package is not consistent, this is not a
surprise. 160/32 = 5, but yes something is fishy there.

Oh. Is this perhaps one of those machines where the APICs are enumerated
twice. Can you apply the patch below?

The ACPI part is a modified variant of:

    https://lore.kernel.org/r/23ccb5aad770fb52014dca9fb1e4353b5c3cede9.camel@intel.com

Please apply the topology hunks first without the ACPI changes and then
try the ACPI change on top.

Thanks,

        tglx
---
 arch/x86/kernel/acpi/boot.c    |   31 ++++++++++++++++---------------
 arch/x86/kernel/cpu/topology.c |    2 ++
 2 files changed, 18 insertions(+), 15 deletions(-)

--- a/arch/x86/kernel/acpi/boot.c
+++ b/arch/x86/kernel/acpi/boot.c
@@ -171,6 +171,8 @@ static bool __init acpi_is_processor_usa
 	return false;
 }
 
+static bool has_lapic_cpus;
+
 static int __init
 acpi_parse_x2apic(union acpi_subtable_headers *header, const unsigned long end)
 {
@@ -241,6 +243,14 @@ acpi_parse_lapic(union acpi_subtable_hea
 		return 0;
 
 	/*
+	 * According to https://uefi.org/specs/ACPI/6.5/05_ACPI_Software_Programming_Model.html#processor-local-x2apic-structure
+	 * when MADT provides both valid LAPIC and x2APIC entries, the APIC ID
+	 * in x2APIC must be equal or greater than 0xff.
+	 */
+	if (has_lapic_cpus && apic_id < 0xff)
+		return 0;
+
+	/*
 	 * We need to register disabled CPU as well to permit
 	 * counting disabled CPUs. This allows us to size
 	 * cpus_possible_map more accurately, to permit
@@ -1084,21 +1094,12 @@ static int __init acpi_parse_madt_lapic_
 				      acpi_parse_sapic, MAX_LOCAL_APIC);
 
 	if (!count) {
-		memset(madt_proc, 0, sizeof(madt_proc));
-		madt_proc[0].id = ACPI_MADT_TYPE_LOCAL_APIC;
-		madt_proc[0].handler = acpi_parse_lapic;
-		madt_proc[1].id = ACPI_MADT_TYPE_LOCAL_X2APIC;
-		madt_proc[1].handler = acpi_parse_x2apic;
-		ret = acpi_table_parse_entries_array(ACPI_SIG_MADT,
-				sizeof(struct acpi_table_madt),
-				madt_proc, ARRAY_SIZE(madt_proc), MAX_LOCAL_APIC);
-		if (ret < 0) {
-			pr_err("Error parsing LAPIC/X2APIC entries\n");
-			return ret;
-		}
-
-		count = madt_proc[0].count;
-		x2count = madt_proc[1].count;
+		count = acpi_table_parse_madt(ACPI_MADT_TYPE_LOCAL_LAPIC,
+					acpi_parse_lapic, MAX_LOCAL_APIC);
+		if (count)
+			has_lapic_cpus = true;
+		x2count = acpi_table_parse_madt(ACPI_MADT_TYPE_LOCAL_X2APIC,
+					acpi_parse_x2apic, MAX_LOCAL_APIC);
 	}
 	if (!count && !x2count) {
 		pr_err("No LAPIC entries present\n");
--- a/arch/x86/kernel/cpu/topology.c
+++ b/arch/x86/kernel/cpu/topology.c
@@ -159,6 +159,8 @@ void __init topology_register_apic(u32 a
 		return;
 	}
 
+	pr_info("Register %03x %d\n", apic_id, present);
+
 	if (present) {
 		/*
 		 * Prevent double registration, which is valid in case of

Re: [patch 00/53] x86/topology: The final installment

[Sohil Mehta]

[-- Attachment #1: Type: text/plain, Size: 2833 bytes --]

On 8/8/2023 12:10 PM, Thomas Gleixner wrote:
> That's indeed weird. Can you please provide:
> 
>      - the output of 'cpuid -r'
>      - the output of /sys/kernel/debug/x86/topo/domains
>      - the APIC IDs of all CPUs (see below patch)
> 

Domains
-------
domain: Thread     shift: 1 dom_size:     2 max_threads:     2
domain: Core       shift: 5 dom_size:    16 max_threads:    32
domain: Module     shift: 5 dom_size:     1 max_threads:    32
domain: Tile       shift: 5 dom_size:     1 max_threads:    32
domain: Die        shift: 5 dom_size:     1 max_threads:    32
domain: Package    shift: 5 dom_size:     1 max_threads:    32

Attached the cpuid output and the apic id list.

Also, I see a warning message that only seems to show up with the final
installment series applied. I attached the complete dmesg as well (just
in case):

unchecked MSR access error: WRMSR to 0xe44 (tried to write
0x0000000000000003) at rIP: 0xffffffff8d2a6698 (native_write_msr+0x8/0x30)
Call Trace:
 <TASK>
 ? show_stack_regs+0x27/0x30
 ? ex_handler_msr+0x10f/0x180
 ? search_extable+0x2b/0x40
 ? fixup_exception+0x315/0x380
 ? exc_general_protection+0x139/0x460
 ? idr_alloc_cyclic+0x59/0xc0
 ? asm_exc_general_protection+0x2b/0x30
 ? native_write_msr+0x8/0x30
 ? ivbep_uncore_msr_init_box+0x47/0x60
 uncore_box_ref.part.0+0xa6/0xe0
 uncore_event_cpu_online+0x6e/0x1c0
 ? __pfx_uncore_event_cpu_online+0x10/0x10
 cpuhp_invoke_callback+0x165/0x4b0
 ? try_to_wake_up+0x284/0x6b0
 cpuhp_thread_fun+0xc4/0x1e0
 ? __pfx_smpboot_thread_fn+0x10/0x10
 smpboot_thread_fn+0xe7/0x1e0
 kthread+0xfb/0x130
 ? __pfx_kthread+0x10/0x10
 ret_from_fork+0x40/0x60
 ? __pfx_kthread+0x10/0x10
 ret_from_fork_asm+0x1b/0x30
 </TASK>


> 
> Please apply the topology hunks first without the ACPI changes and then
> try the ACPI change on top.
> 

There are compile issues with the ACPI hunk.

> arch/x86/kernel/acpi/boot.c: In function ‘acpi_parse_lapic’:
> arch/x86/kernel/acpi/boot.c:250:31: error: ‘apic_id’ undeclared (first use in this function)
>   250 |         if (has_lapic_cpus && apic_id < 0xff)
>       |                               ^~~~~~~
> arch/x86/kernel/acpi/boot.c:250:31: note: each undeclared identifier is reported only once for each function it appears in
> arch/x86/kernel/acpi/boot.c: In function ‘acpi_parse_madt_lapic_entries’:
> arch/x86/kernel/acpi/boot.c:1097:47: error: ‘ACPI_MADT_TYPE_LOCAL_LAPIC’ undeclared (first use in this function); did you mean ‘ACPI_MADT_TYPE_LOCAL_SAPIC’?
>  1097 |                 count = acpi_table_parse_madt(ACPI_MADT_TYPE_LOCAL_LAPIC,
>       |                                               ^~~~~~~~~~~~~~~~~~~~~~~~~~
>       |                                               ACPI_MADT_TYPE_LOCAL_SAPIC


Did I miss applying something? I didn't try to understand and fix them.

Sohil


[-- Attachment #2: cpuid.txt --]
[-- Type: text/plain, Size: 109110 bytes --]

CPU 0:
   0x00000000 0x00: eax=0x0000000d ebx=0x756e6547 ecx=0x6c65746e edx=0x49656e69
   0x00000001 0x00: eax=0x000306e4 ebx=0x00200800 ecx=0x7fbee3ff edx=0xbfebfbff
   0x00000002 0x00: eax=0x76036301 ebx=0x00f0b2ff ecx=0x00000000 edx=0x00ca0000
   0x00000003 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000004 0x00: eax=0x3c004121 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000
   0x00000004 0x01: eax=0x3c004122 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000
   0x00000004 0x02: eax=0x3c004143 ebx=0x01c0003f ecx=0x000001ff edx=0x00000000
   0x00000004 0x03: eax=0x3c07c163 ebx=0x04c0003f ecx=0x00004fff edx=0x00000006
   0x00000004 0x04: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000005 0x00: eax=0x00000040 ebx=0x00000040 ecx=0x00000003 edx=0x00001120
   0x00000006 0x00: eax=0x00000077 ebx=0x00000002 ecx=0x00000009 edx=0x00000000
   0x00000007 0x00: eax=0x00000000 ebx=0x00000281 ecx=0x00000000 edx=0x9c000400
   0x00000008 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000009 0x00: eax=0x00000001 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000a 0x00: eax=0x07300403 ebx=0x00000000 ecx=0x00000000 edx=0x00000603
   0x0000000b 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000000
   0x0000000b 0x01: eax=0x00000005 ebx=0x00000014 ecx=0x00000201 edx=0x00000000
   0x0000000b 0x02: eax=0x00000000 ebx=0x00000000 ecx=0x00000002 edx=0x00000000
   0x0000000c 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000d 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0x0000000d 0x01: eax=0x00000001 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000d 0x02: eax=0x00000100 ebx=0x00000240 ecx=0x00000000 edx=0x00000000
   0x20000000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0x80000000 0x00: eax=0x80000008 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000001 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000001 edx=0x2c100800
   0x80000002 0x00: eax=0x20202020 ebx=0x6e492020 ecx=0x286c6574 edx=0x58202952
   0x80000003 0x00: eax=0x286e6f65 ebx=0x43202952 ecx=0x45205550 edx=0x36322d35
   0x80000004 0x00: eax=0x76203039 ebx=0x20402032 ecx=0x30302e33 edx=0x007a4847
   0x80000005 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000006 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x01006040 edx=0x00000000
   0x80000007 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000100
   0x80000008 0x00: eax=0x0000302e ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80860000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0xc0000000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
CPU 1:
   0x00000000 0x00: eax=0x0000000d ebx=0x756e6547 ecx=0x6c65746e edx=0x49656e69
   0x00000001 0x00: eax=0x000306e4 ebx=0x02200800 ecx=0x7fbee3ff edx=0xbfebfbff
   0x00000002 0x00: eax=0x76036301 ebx=0x00f0b2ff ecx=0x00000000 edx=0x00ca0000
   0x00000003 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000004 0x00: eax=0x3c004121 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000
   0x00000004 0x01: eax=0x3c004122 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000
   0x00000004 0x02: eax=0x3c004143 ebx=0x01c0003f ecx=0x000001ff edx=0x00000000
   0x00000004 0x03: eax=0x3c07c163 ebx=0x04c0003f ecx=0x00004fff edx=0x00000006
   0x00000004 0x04: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000005 0x00: eax=0x00000040 ebx=0x00000040 ecx=0x00000003 edx=0x00001120
   0x00000006 0x00: eax=0x00000077 ebx=0x00000002 ecx=0x00000009 edx=0x00000000
   0x00000007 0x00: eax=0x00000000 ebx=0x00000281 ecx=0x00000000 edx=0x9c000400
   0x00000008 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000009 0x00: eax=0x00000001 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000a 0x00: eax=0x07300403 ebx=0x00000000 ecx=0x00000000 edx=0x00000603
   0x0000000b 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000002
   0x0000000b 0x01: eax=0x00000005 ebx=0x00000014 ecx=0x00000201 edx=0x00000002
   0x0000000b 0x02: eax=0x00000000 ebx=0x00000000 ecx=0x00000002 edx=0x00000002
   0x0000000c 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000d 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0x0000000d 0x01: eax=0x00000001 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000d 0x02: eax=0x00000100 ebx=0x00000240 ecx=0x00000000 edx=0x00000000
   0x20000000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0x80000000 0x00: eax=0x80000008 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000001 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000001 edx=0x2c100800
   0x80000002 0x00: eax=0x20202020 ebx=0x6e492020 ecx=0x286c6574 edx=0x58202952
   0x80000003 0x00: eax=0x286e6f65 ebx=0x43202952 ecx=0x45205550 edx=0x36322d35
   0x80000004 0x00: eax=0x76203039 ebx=0x20402032 ecx=0x30302e33 edx=0x007a4847
   0x80000005 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000006 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x01006040 edx=0x00000000
   0x80000007 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000100
   0x80000008 0x00: eax=0x0000302e ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80860000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0xc0000000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
CPU 2:
   0x00000000 0x00: eax=0x0000000d ebx=0x756e6547 ecx=0x6c65746e edx=0x49656e69
   0x00000001 0x00: eax=0x000306e4 ebx=0x04200800 ecx=0x7fbee3ff edx=0xbfebfbff
   0x00000002 0x00: eax=0x76036301 ebx=0x00f0b2ff ecx=0x00000000 edx=0x00ca0000
   0x00000003 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000004 0x00: eax=0x3c004121 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000
   0x00000004 0x01: eax=0x3c004122 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000
   0x00000004 0x02: eax=0x3c004143 ebx=0x01c0003f ecx=0x000001ff edx=0x00000000
   0x00000004 0x03: eax=0x3c07c163 ebx=0x04c0003f ecx=0x00004fff edx=0x00000006
   0x00000004 0x04: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000005 0x00: eax=0x00000040 ebx=0x00000040 ecx=0x00000003 edx=0x00001120
   0x00000006 0x00: eax=0x00000077 ebx=0x00000002 ecx=0x00000009 edx=0x00000000
   0x00000007 0x00: eax=0x00000000 ebx=0x00000281 ecx=0x00000000 edx=0x9c000400
   0x00000008 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000009 0x00: eax=0x00000001 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000a 0x00: eax=0x07300403 ebx=0x00000000 ecx=0x00000000 edx=0x00000603
   0x0000000b 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000004
   0x0000000b 0x01: eax=0x00000005 ebx=0x00000014 ecx=0x00000201 edx=0x00000004
   0x0000000b 0x02: eax=0x00000000 ebx=0x00000000 ecx=0x00000002 edx=0x00000004
   0x0000000c 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000d 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0x0000000d 0x01: eax=0x00000001 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000d 0x02: eax=0x00000100 ebx=0x00000240 ecx=0x00000000 edx=0x00000000
   0x20000000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0x80000000 0x00: eax=0x80000008 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000001 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000001 edx=0x2c100800
   0x80000002 0x00: eax=0x20202020 ebx=0x6e492020 ecx=0x286c6574 edx=0x58202952
   0x80000003 0x00: eax=0x286e6f65 ebx=0x43202952 ecx=0x45205550 edx=0x36322d35
   0x80000004 0x00: eax=0x76203039 ebx=0x20402032 ecx=0x30302e33 edx=0x007a4847
   0x80000005 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000006 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x01006040 edx=0x00000000
   0x80000007 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000100
   0x80000008 0x00: eax=0x0000302e ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80860000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0xc0000000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
CPU 3:
   0x00000000 0x00: eax=0x0000000d ebx=0x756e6547 ecx=0x6c65746e edx=0x49656e69
   0x00000001 0x00: eax=0x000306e4 ebx=0x06200800 ecx=0x7fbee3ff edx=0xbfebfbff
   0x00000002 0x00: eax=0x76036301 ebx=0x00f0b2ff ecx=0x00000000 edx=0x00ca0000
   0x00000003 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000004 0x00: eax=0x3c004121 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000
   0x00000004 0x01: eax=0x3c004122 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000
   0x00000004 0x02: eax=0x3c004143 ebx=0x01c0003f ecx=0x000001ff edx=0x00000000
   0x00000004 0x03: eax=0x3c07c163 ebx=0x04c0003f ecx=0x00004fff edx=0x00000006
   0x00000004 0x04: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000005 0x00: eax=0x00000040 ebx=0x00000040 ecx=0x00000003 edx=0x00001120
   0x00000006 0x00: eax=0x00000077 ebx=0x00000002 ecx=0x00000009 edx=0x00000000
   0x00000007 0x00: eax=0x00000000 ebx=0x00000281 ecx=0x00000000 edx=0x9c000400
   0x00000008 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000009 0x00: eax=0x00000001 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000a 0x00: eax=0x07300403 ebx=0x00000000 ecx=0x00000000 edx=0x00000603
   0x0000000b 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000006
   0x0000000b 0x01: eax=0x00000005 ebx=0x00000014 ecx=0x00000201 edx=0x00000006
   0x0000000b 0x02: eax=0x00000000 ebx=0x00000000 ecx=0x00000002 edx=0x00000006
   0x0000000c 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000d 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0x0000000d 0x01: eax=0x00000001 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000d 0x02: eax=0x00000100 ebx=0x00000240 ecx=0x00000000 edx=0x00000000
   0x20000000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0x80000000 0x00: eax=0x80000008 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000001 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000001 edx=0x2c100800
   0x80000002 0x00: eax=0x20202020 ebx=0x6e492020 ecx=0x286c6574 edx=0x58202952
   0x80000003 0x00: eax=0x286e6f65 ebx=0x43202952 ecx=0x45205550 edx=0x36322d35
   0x80000004 0x00: eax=0x76203039 ebx=0x20402032 ecx=0x30302e33 edx=0x007a4847
   0x80000005 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000006 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x01006040 edx=0x00000000
   0x80000007 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000100
   0x80000008 0x00: eax=0x0000302e ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80860000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0xc0000000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
CPU 4:
   0x00000000 0x00: eax=0x0000000d ebx=0x756e6547 ecx=0x6c65746e edx=0x49656e69
   0x00000001 0x00: eax=0x000306e4 ebx=0x08200800 ecx=0x7fbee3ff edx=0xbfebfbff
   0x00000002 0x00: eax=0x76036301 ebx=0x00f0b2ff ecx=0x00000000 edx=0x00ca0000
   0x00000003 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000004 0x00: eax=0x3c004121 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000
   0x00000004 0x01: eax=0x3c004122 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000
   0x00000004 0x02: eax=0x3c004143 ebx=0x01c0003f ecx=0x000001ff edx=0x00000000
   0x00000004 0x03: eax=0x3c07c163 ebx=0x04c0003f ecx=0x00004fff edx=0x00000006
   0x00000004 0x04: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000005 0x00: eax=0x00000040 ebx=0x00000040 ecx=0x00000003 edx=0x00001120
   0x00000006 0x00: eax=0x00000077 ebx=0x00000002 ecx=0x00000009 edx=0x00000000
   0x00000007 0x00: eax=0x00000000 ebx=0x00000281 ecx=0x00000000 edx=0x9c000400
   0x00000008 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000009 0x00: eax=0x00000001 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000a 0x00: eax=0x07300403 ebx=0x00000000 ecx=0x00000000 edx=0x00000603
   0x0000000b 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000008
   0x0000000b 0x01: eax=0x00000005 ebx=0x00000014 ecx=0x00000201 edx=0x00000008
   0x0000000b 0x02: eax=0x00000000 ebx=0x00000000 ecx=0x00000002 edx=0x00000008
   0x0000000c 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000d 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0x0000000d 0x01: eax=0x00000001 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000d 0x02: eax=0x00000100 ebx=0x00000240 ecx=0x00000000 edx=0x00000000
   0x20000000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0x80000000 0x00: eax=0x80000008 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000001 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000001 edx=0x2c100800
   0x80000002 0x00: eax=0x20202020 ebx=0x6e492020 ecx=0x286c6574 edx=0x58202952
   0x80000003 0x00: eax=0x286e6f65 ebx=0x43202952 ecx=0x45205550 edx=0x36322d35
   0x80000004 0x00: eax=0x76203039 ebx=0x20402032 ecx=0x30302e33 edx=0x007a4847
   0x80000005 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000006 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x01006040 edx=0x00000000
   0x80000007 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000100
   0x80000008 0x00: eax=0x0000302e ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80860000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0xc0000000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
CPU 5:
   0x00000000 0x00: eax=0x0000000d ebx=0x756e6547 ecx=0x6c65746e edx=0x49656e69
   0x00000001 0x00: eax=0x000306e4 ebx=0x10200800 ecx=0x7fbee3ff edx=0xbfebfbff
   0x00000002 0x00: eax=0x76036301 ebx=0x00f0b2ff ecx=0x00000000 edx=0x00ca0000
   0x00000003 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000004 0x00: eax=0x3c004121 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000
   0x00000004 0x01: eax=0x3c004122 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000
   0x00000004 0x02: eax=0x3c004143 ebx=0x01c0003f ecx=0x000001ff edx=0x00000000
   0x00000004 0x03: eax=0x3c07c163 ebx=0x04c0003f ecx=0x00004fff edx=0x00000006
   0x00000004 0x04: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000005 0x00: eax=0x00000040 ebx=0x00000040 ecx=0x00000003 edx=0x00001120
   0x00000006 0x00: eax=0x00000077 ebx=0x00000002 ecx=0x00000009 edx=0x00000000
   0x00000007 0x00: eax=0x00000000 ebx=0x00000281 ecx=0x00000000 edx=0x9c000400
   0x00000008 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000009 0x00: eax=0x00000001 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000a 0x00: eax=0x07300403 ebx=0x00000000 ecx=0x00000000 edx=0x00000603
   0x0000000b 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000010
   0x0000000b 0x01: eax=0x00000005 ebx=0x00000014 ecx=0x00000201 edx=0x00000010
   0x0000000b 0x02: eax=0x00000000 ebx=0x00000000 ecx=0x00000002 edx=0x00000010
   0x0000000c 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000d 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0x0000000d 0x01: eax=0x00000001 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000d 0x02: eax=0x00000100 ebx=0x00000240 ecx=0x00000000 edx=0x00000000
   0x20000000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0x80000000 0x00: eax=0x80000008 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000001 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000001 edx=0x2c100800
   0x80000002 0x00: eax=0x20202020 ebx=0x6e492020 ecx=0x286c6574 edx=0x58202952
   0x80000003 0x00: eax=0x286e6f65 ebx=0x43202952 ecx=0x45205550 edx=0x36322d35
   0x80000004 0x00: eax=0x76203039 ebx=0x20402032 ecx=0x30302e33 edx=0x007a4847
   0x80000005 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000006 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x01006040 edx=0x00000000
   0x80000007 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000100
   0x80000008 0x00: eax=0x0000302e ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80860000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0xc0000000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
CPU 6:
   0x00000000 0x00: eax=0x0000000d ebx=0x756e6547 ecx=0x6c65746e edx=0x49656e69
   0x00000001 0x00: eax=0x000306e4 ebx=0x12200800 ecx=0x7fbee3ff edx=0xbfebfbff
   0x00000002 0x00: eax=0x76036301 ebx=0x00f0b2ff ecx=0x00000000 edx=0x00ca0000
   0x00000003 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000004 0x00: eax=0x3c004121 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000
   0x00000004 0x01: eax=0x3c004122 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000
   0x00000004 0x02: eax=0x3c004143 ebx=0x01c0003f ecx=0x000001ff edx=0x00000000
   0x00000004 0x03: eax=0x3c07c163 ebx=0x04c0003f ecx=0x00004fff edx=0x00000006
   0x00000004 0x04: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000005 0x00: eax=0x00000040 ebx=0x00000040 ecx=0x00000003 edx=0x00001120
   0x00000006 0x00: eax=0x00000077 ebx=0x00000002 ecx=0x00000009 edx=0x00000000
   0x00000007 0x00: eax=0x00000000 ebx=0x00000281 ecx=0x00000000 edx=0x9c000400
   0x00000008 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000009 0x00: eax=0x00000001 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000a 0x00: eax=0x07300403 ebx=0x00000000 ecx=0x00000000 edx=0x00000603
   0x0000000b 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000012
   0x0000000b 0x01: eax=0x00000005 ebx=0x00000014 ecx=0x00000201 edx=0x00000012
   0x0000000b 0x02: eax=0x00000000 ebx=0x00000000 ecx=0x00000002 edx=0x00000012
   0x0000000c 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000d 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0x0000000d 0x01: eax=0x00000001 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000d 0x02: eax=0x00000100 ebx=0x00000240 ecx=0x00000000 edx=0x00000000
   0x20000000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0x80000000 0x00: eax=0x80000008 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000001 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000001 edx=0x2c100800
   0x80000002 0x00: eax=0x20202020 ebx=0x6e492020 ecx=0x286c6574 edx=0x58202952
   0x80000003 0x00: eax=0x286e6f65 ebx=0x43202952 ecx=0x45205550 edx=0x36322d35
   0x80000004 0x00: eax=0x76203039 ebx=0x20402032 ecx=0x30302e33 edx=0x007a4847
   0x80000005 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000006 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x01006040 edx=0x00000000
   0x80000007 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000100
   0x80000008 0x00: eax=0x0000302e ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80860000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0xc0000000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
CPU 7:
   0x00000000 0x00: eax=0x0000000d ebx=0x756e6547 ecx=0x6c65746e edx=0x49656e69
   0x00000001 0x00: eax=0x000306e4 ebx=0x14200800 ecx=0x7fbee3ff edx=0xbfebfbff
   0x00000002 0x00: eax=0x76036301 ebx=0x00f0b2ff ecx=0x00000000 edx=0x00ca0000
   0x00000003 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000004 0x00: eax=0x3c004121 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000
   0x00000004 0x01: eax=0x3c004122 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000
   0x00000004 0x02: eax=0x3c004143 ebx=0x01c0003f ecx=0x000001ff edx=0x00000000
   0x00000004 0x03: eax=0x3c07c163 ebx=0x04c0003f ecx=0x00004fff edx=0x00000006
   0x00000004 0x04: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000005 0x00: eax=0x00000040 ebx=0x00000040 ecx=0x00000003 edx=0x00001120
   0x00000006 0x00: eax=0x00000077 ebx=0x00000002 ecx=0x00000009 edx=0x00000000
   0x00000007 0x00: eax=0x00000000 ebx=0x00000281 ecx=0x00000000 edx=0x9c000400
   0x00000008 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000009 0x00: eax=0x00000001 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000a 0x00: eax=0x07300403 ebx=0x00000000 ecx=0x00000000 edx=0x00000603
   0x0000000b 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000014
   0x0000000b 0x01: eax=0x00000005 ebx=0x00000014 ecx=0x00000201 edx=0x00000014
   0x0000000b 0x02: eax=0x00000000 ebx=0x00000000 ecx=0x00000002 edx=0x00000014
   0x0000000c 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000d 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0x0000000d 0x01: eax=0x00000001 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000d 0x02: eax=0x00000100 ebx=0x00000240 ecx=0x00000000 edx=0x00000000
   0x20000000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0x80000000 0x00: eax=0x80000008 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000001 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000001 edx=0x2c100800
   0x80000002 0x00: eax=0x20202020 ebx=0x6e492020 ecx=0x286c6574 edx=0x58202952
   0x80000003 0x00: eax=0x286e6f65 ebx=0x43202952 ecx=0x45205550 edx=0x36322d35
   0x80000004 0x00: eax=0x76203039 ebx=0x20402032 ecx=0x30302e33 edx=0x007a4847
   0x80000005 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000006 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x01006040 edx=0x00000000
   0x80000007 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000100
   0x80000008 0x00: eax=0x0000302e ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80860000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0xc0000000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
CPU 8:
   0x00000000 0x00: eax=0x0000000d ebx=0x756e6547 ecx=0x6c65746e edx=0x49656e69
   0x00000001 0x00: eax=0x000306e4 ebx=0x16200800 ecx=0x7fbee3ff edx=0xbfebfbff
   0x00000002 0x00: eax=0x76036301 ebx=0x00f0b2ff ecx=0x00000000 edx=0x00ca0000
   0x00000003 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000004 0x00: eax=0x3c004121 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000
   0x00000004 0x01: eax=0x3c004122 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000
   0x00000004 0x02: eax=0x3c004143 ebx=0x01c0003f ecx=0x000001ff edx=0x00000000
   0x00000004 0x03: eax=0x3c07c163 ebx=0x04c0003f ecx=0x00004fff edx=0x00000006
   0x00000004 0x04: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000005 0x00: eax=0x00000040 ebx=0x00000040 ecx=0x00000003 edx=0x00001120
   0x00000006 0x00: eax=0x00000077 ebx=0x00000002 ecx=0x00000009 edx=0x00000000
   0x00000007 0x00: eax=0x00000000 ebx=0x00000281 ecx=0x00000000 edx=0x9c000400
   0x00000008 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000009 0x00: eax=0x00000001 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000a 0x00: eax=0x07300403 ebx=0x00000000 ecx=0x00000000 edx=0x00000603
   0x0000000b 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000016
   0x0000000b 0x01: eax=0x00000005 ebx=0x00000014 ecx=0x00000201 edx=0x00000016
   0x0000000b 0x02: eax=0x00000000 ebx=0x00000000 ecx=0x00000002 edx=0x00000016
   0x0000000c 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000d 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0x0000000d 0x01: eax=0x00000001 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000d 0x02: eax=0x00000100 ebx=0x00000240 ecx=0x00000000 edx=0x00000000
   0x20000000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0x80000000 0x00: eax=0x80000008 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000001 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000001 edx=0x2c100800
   0x80000002 0x00: eax=0x20202020 ebx=0x6e492020 ecx=0x286c6574 edx=0x58202952
   0x80000003 0x00: eax=0x286e6f65 ebx=0x43202952 ecx=0x45205550 edx=0x36322d35
   0x80000004 0x00: eax=0x76203039 ebx=0x20402032 ecx=0x30302e33 edx=0x007a4847
   0x80000005 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000006 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x01006040 edx=0x00000000
   0x80000007 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000100
   0x80000008 0x00: eax=0x0000302e ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80860000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0xc0000000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
CPU 9:
   0x00000000 0x00: eax=0x0000000d ebx=0x756e6547 ecx=0x6c65746e edx=0x49656e69
   0x00000001 0x00: eax=0x000306e4 ebx=0x18200800 ecx=0x7fbee3ff edx=0xbfebfbff
   0x00000002 0x00: eax=0x76036301 ebx=0x00f0b2ff ecx=0x00000000 edx=0x00ca0000
   0x00000003 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000004 0x00: eax=0x3c004121 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000
   0x00000004 0x01: eax=0x3c004122 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000
   0x00000004 0x02: eax=0x3c004143 ebx=0x01c0003f ecx=0x000001ff edx=0x00000000
   0x00000004 0x03: eax=0x3c07c163 ebx=0x04c0003f ecx=0x00004fff edx=0x00000006
   0x00000004 0x04: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000005 0x00: eax=0x00000040 ebx=0x00000040 ecx=0x00000003 edx=0x00001120
   0x00000006 0x00: eax=0x00000077 ebx=0x00000002 ecx=0x00000009 edx=0x00000000
   0x00000007 0x00: eax=0x00000000 ebx=0x00000281 ecx=0x00000000 edx=0x9c000400
   0x00000008 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000009 0x00: eax=0x00000001 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000a 0x00: eax=0x07300403 ebx=0x00000000 ecx=0x00000000 edx=0x00000603
   0x0000000b 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000018
   0x0000000b 0x01: eax=0x00000005 ebx=0x00000014 ecx=0x00000201 edx=0x00000018
   0x0000000b 0x02: eax=0x00000000 ebx=0x00000000 ecx=0x00000002 edx=0x00000018
   0x0000000c 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000d 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0x0000000d 0x01: eax=0x00000001 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000d 0x02: eax=0x00000100 ebx=0x00000240 ecx=0x00000000 edx=0x00000000
   0x20000000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0x80000000 0x00: eax=0x80000008 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000001 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000001 edx=0x2c100800
   0x80000002 0x00: eax=0x20202020 ebx=0x6e492020 ecx=0x286c6574 edx=0x58202952
   0x80000003 0x00: eax=0x286e6f65 ebx=0x43202952 ecx=0x45205550 edx=0x36322d35
   0x80000004 0x00: eax=0x76203039 ebx=0x20402032 ecx=0x30302e33 edx=0x007a4847
   0x80000005 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000006 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x01006040 edx=0x00000000
   0x80000007 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000100
   0x80000008 0x00: eax=0x0000302e ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80860000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0xc0000000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
CPU 10:
   0x00000000 0x00: eax=0x0000000d ebx=0x756e6547 ecx=0x6c65746e edx=0x49656e69
   0x00000001 0x00: eax=0x000306e4 ebx=0x20200800 ecx=0x7fbee3ff edx=0xbfebfbff
   0x00000002 0x00: eax=0x76036301 ebx=0x00f0b2ff ecx=0x00000000 edx=0x00ca0000
   0x00000003 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000004 0x00: eax=0x3c004121 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000
   0x00000004 0x01: eax=0x3c004122 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000
   0x00000004 0x02: eax=0x3c004143 ebx=0x01c0003f ecx=0x000001ff edx=0x00000000
   0x00000004 0x03: eax=0x3c07c163 ebx=0x04c0003f ecx=0x00004fff edx=0x00000006
   0x00000004 0x04: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000005 0x00: eax=0x00000040 ebx=0x00000040 ecx=0x00000003 edx=0x00001120
   0x00000006 0x00: eax=0x00000077 ebx=0x00000002 ecx=0x00000009 edx=0x00000000
   0x00000007 0x00: eax=0x00000000 ebx=0x00000281 ecx=0x00000000 edx=0x9c000400
   0x00000008 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000009 0x00: eax=0x00000001 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000a 0x00: eax=0x07300403 ebx=0x00000000 ecx=0x00000000 edx=0x00000603
   0x0000000b 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000020
   0x0000000b 0x01: eax=0x00000005 ebx=0x00000014 ecx=0x00000201 edx=0x00000020
   0x0000000b 0x02: eax=0x00000000 ebx=0x00000000 ecx=0x00000002 edx=0x00000020
   0x0000000c 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000d 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0x0000000d 0x01: eax=0x00000001 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000d 0x02: eax=0x00000100 ebx=0x00000240 ecx=0x00000000 edx=0x00000000
   0x20000000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0x80000000 0x00: eax=0x80000008 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000001 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000001 edx=0x2c100800
   0x80000002 0x00: eax=0x20202020 ebx=0x6e492020 ecx=0x286c6574 edx=0x58202952
   0x80000003 0x00: eax=0x286e6f65 ebx=0x43202952 ecx=0x45205550 edx=0x36322d35
   0x80000004 0x00: eax=0x76203039 ebx=0x20402032 ecx=0x30302e33 edx=0x007a4847
   0x80000005 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000006 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x01006040 edx=0x00000000
   0x80000007 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000100
   0x80000008 0x00: eax=0x0000302e ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80860000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0xc0000000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
CPU 11:
   0x00000000 0x00: eax=0x0000000d ebx=0x756e6547 ecx=0x6c65746e edx=0x49656e69
   0x00000001 0x00: eax=0x000306e4 ebx=0x22200800 ecx=0x7fbee3ff edx=0xbfebfbff
   0x00000002 0x00: eax=0x76036301 ebx=0x00f0b2ff ecx=0x00000000 edx=0x00ca0000
   0x00000003 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000004 0x00: eax=0x3c004121 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000
   0x00000004 0x01: eax=0x3c004122 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000
   0x00000004 0x02: eax=0x3c004143 ebx=0x01c0003f ecx=0x000001ff edx=0x00000000
   0x00000004 0x03: eax=0x3c07c163 ebx=0x04c0003f ecx=0x00004fff edx=0x00000006
   0x00000004 0x04: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000005 0x00: eax=0x00000040 ebx=0x00000040 ecx=0x00000003 edx=0x00001120
   0x00000006 0x00: eax=0x00000077 ebx=0x00000002 ecx=0x00000009 edx=0x00000000
   0x00000007 0x00: eax=0x00000000 ebx=0x00000281 ecx=0x00000000 edx=0x9c000400
   0x00000008 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000009 0x00: eax=0x00000001 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000a 0x00: eax=0x07300403 ebx=0x00000000 ecx=0x00000000 edx=0x00000603
   0x0000000b 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000022
   0x0000000b 0x01: eax=0x00000005 ebx=0x00000014 ecx=0x00000201 edx=0x00000022
   0x0000000b 0x02: eax=0x00000000 ebx=0x00000000 ecx=0x00000002 edx=0x00000022
   0x0000000c 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000d 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0x0000000d 0x01: eax=0x00000001 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000d 0x02: eax=0x00000100 ebx=0x00000240 ecx=0x00000000 edx=0x00000000
   0x20000000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0x80000000 0x00: eax=0x80000008 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000001 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000001 edx=0x2c100800
   0x80000002 0x00: eax=0x20202020 ebx=0x6e492020 ecx=0x286c6574 edx=0x58202952
   0x80000003 0x00: eax=0x286e6f65 ebx=0x43202952 ecx=0x45205550 edx=0x36322d35
   0x80000004 0x00: eax=0x76203039 ebx=0x20402032 ecx=0x30302e33 edx=0x007a4847
   0x80000005 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000006 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x01006040 edx=0x00000000
   0x80000007 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000100
   0x80000008 0x00: eax=0x0000302e ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80860000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0xc0000000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
CPU 12:
   0x00000000 0x00: eax=0x0000000d ebx=0x756e6547 ecx=0x6c65746e edx=0x49656e69
   0x00000001 0x00: eax=0x000306e4 ebx=0x24200800 ecx=0x7fbee3ff edx=0xbfebfbff
   0x00000002 0x00: eax=0x76036301 ebx=0x00f0b2ff ecx=0x00000000 edx=0x00ca0000
   0x00000003 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000004 0x00: eax=0x3c004121 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000
   0x00000004 0x01: eax=0x3c004122 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000
   0x00000004 0x02: eax=0x3c004143 ebx=0x01c0003f ecx=0x000001ff edx=0x00000000
   0x00000004 0x03: eax=0x3c07c163 ebx=0x04c0003f ecx=0x00004fff edx=0x00000006
   0x00000004 0x04: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000005 0x00: eax=0x00000040 ebx=0x00000040 ecx=0x00000003 edx=0x00001120
   0x00000006 0x00: eax=0x00000077 ebx=0x00000002 ecx=0x00000009 edx=0x00000000
   0x00000007 0x00: eax=0x00000000 ebx=0x00000281 ecx=0x00000000 edx=0x9c000400
   0x00000008 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000009 0x00: eax=0x00000001 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000a 0x00: eax=0x07300403 ebx=0x00000000 ecx=0x00000000 edx=0x00000603
   0x0000000b 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000024
   0x0000000b 0x01: eax=0x00000005 ebx=0x00000014 ecx=0x00000201 edx=0x00000024
   0x0000000b 0x02: eax=0x00000000 ebx=0x00000000 ecx=0x00000002 edx=0x00000024
   0x0000000c 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000d 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0x0000000d 0x01: eax=0x00000001 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000d 0x02: eax=0x00000100 ebx=0x00000240 ecx=0x00000000 edx=0x00000000
   0x20000000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0x80000000 0x00: eax=0x80000008 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000001 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000001 edx=0x2c100800
   0x80000002 0x00: eax=0x20202020 ebx=0x6e492020 ecx=0x286c6574 edx=0x58202952
   0x80000003 0x00: eax=0x286e6f65 ebx=0x43202952 ecx=0x45205550 edx=0x36322d35
   0x80000004 0x00: eax=0x76203039 ebx=0x20402032 ecx=0x30302e33 edx=0x007a4847
   0x80000005 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000006 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x01006040 edx=0x00000000
   0x80000007 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000100
   0x80000008 0x00: eax=0x0000302e ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80860000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0xc0000000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
CPU 13:
   0x00000000 0x00: eax=0x0000000d ebx=0x756e6547 ecx=0x6c65746e edx=0x49656e69
   0x00000001 0x00: eax=0x000306e4 ebx=0x26200800 ecx=0x7fbee3ff edx=0xbfebfbff
   0x00000002 0x00: eax=0x76036301 ebx=0x00f0b2ff ecx=0x00000000 edx=0x00ca0000
   0x00000003 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000004 0x00: eax=0x3c004121 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000
   0x00000004 0x01: eax=0x3c004122 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000
   0x00000004 0x02: eax=0x3c004143 ebx=0x01c0003f ecx=0x000001ff edx=0x00000000
   0x00000004 0x03: eax=0x3c07c163 ebx=0x04c0003f ecx=0x00004fff edx=0x00000006
   0x00000004 0x04: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000005 0x00: eax=0x00000040 ebx=0x00000040 ecx=0x00000003 edx=0x00001120
   0x00000006 0x00: eax=0x00000077 ebx=0x00000002 ecx=0x00000009 edx=0x00000000
   0x00000007 0x00: eax=0x00000000 ebx=0x00000281 ecx=0x00000000 edx=0x9c000400
   0x00000008 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000009 0x00: eax=0x00000001 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000a 0x00: eax=0x07300403 ebx=0x00000000 ecx=0x00000000 edx=0x00000603
   0x0000000b 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000026
   0x0000000b 0x01: eax=0x00000005 ebx=0x00000014 ecx=0x00000201 edx=0x00000026
   0x0000000b 0x02: eax=0x00000000 ebx=0x00000000 ecx=0x00000002 edx=0x00000026
   0x0000000c 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000d 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0x0000000d 0x01: eax=0x00000001 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000d 0x02: eax=0x00000100 ebx=0x00000240 ecx=0x00000000 edx=0x00000000
   0x20000000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0x80000000 0x00: eax=0x80000008 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000001 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000001 edx=0x2c100800
   0x80000002 0x00: eax=0x20202020 ebx=0x6e492020 ecx=0x286c6574 edx=0x58202952
   0x80000003 0x00: eax=0x286e6f65 ebx=0x43202952 ecx=0x45205550 edx=0x36322d35
   0x80000004 0x00: eax=0x76203039 ebx=0x20402032 ecx=0x30302e33 edx=0x007a4847
   0x80000005 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000006 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x01006040 edx=0x00000000
   0x80000007 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000100
   0x80000008 0x00: eax=0x0000302e ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80860000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0xc0000000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
CPU 14:
   0x00000000 0x00: eax=0x0000000d ebx=0x756e6547 ecx=0x6c65746e edx=0x49656e69
   0x00000001 0x00: eax=0x000306e4 ebx=0x28200800 ecx=0x7fbee3ff edx=0xbfebfbff
   0x00000002 0x00: eax=0x76036301 ebx=0x00f0b2ff ecx=0x00000000 edx=0x00ca0000
   0x00000003 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000004 0x00: eax=0x3c004121 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000
   0x00000004 0x01: eax=0x3c004122 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000
   0x00000004 0x02: eax=0x3c004143 ebx=0x01c0003f ecx=0x000001ff edx=0x00000000
   0x00000004 0x03: eax=0x3c07c163 ebx=0x04c0003f ecx=0x00004fff edx=0x00000006
   0x00000004 0x04: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000005 0x00: eax=0x00000040 ebx=0x00000040 ecx=0x00000003 edx=0x00001120
   0x00000006 0x00: eax=0x00000077 ebx=0x00000002 ecx=0x00000009 edx=0x00000000
   0x00000007 0x00: eax=0x00000000 ebx=0x00000281 ecx=0x00000000 edx=0x9c000400
   0x00000008 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000009 0x00: eax=0x00000001 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000a 0x00: eax=0x07300403 ebx=0x00000000 ecx=0x00000000 edx=0x00000603
   0x0000000b 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000028
   0x0000000b 0x01: eax=0x00000005 ebx=0x00000014 ecx=0x00000201 edx=0x00000028
   0x0000000b 0x02: eax=0x00000000 ebx=0x00000000 ecx=0x00000002 edx=0x00000028
   0x0000000c 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000d 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0x0000000d 0x01: eax=0x00000001 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000d 0x02: eax=0x00000100 ebx=0x00000240 ecx=0x00000000 edx=0x00000000
   0x20000000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0x80000000 0x00: eax=0x80000008 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000001 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000001 edx=0x2c100800
   0x80000002 0x00: eax=0x20202020 ebx=0x6e492020 ecx=0x286c6574 edx=0x58202952
   0x80000003 0x00: eax=0x286e6f65 ebx=0x43202952 ecx=0x45205550 edx=0x36322d35
   0x80000004 0x00: eax=0x76203039 ebx=0x20402032 ecx=0x30302e33 edx=0x007a4847
   0x80000005 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000006 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x01006040 edx=0x00000000
   0x80000007 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000100
   0x80000008 0x00: eax=0x0000302e ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80860000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0xc0000000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
CPU 15:
   0x00000000 0x00: eax=0x0000000d ebx=0x756e6547 ecx=0x6c65746e edx=0x49656e69
   0x00000001 0x00: eax=0x000306e4 ebx=0x30200800 ecx=0x7fbee3ff edx=0xbfebfbff
   0x00000002 0x00: eax=0x76036301 ebx=0x00f0b2ff ecx=0x00000000 edx=0x00ca0000
   0x00000003 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000004 0x00: eax=0x3c004121 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000
   0x00000004 0x01: eax=0x3c004122 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000
   0x00000004 0x02: eax=0x3c004143 ebx=0x01c0003f ecx=0x000001ff edx=0x00000000
   0x00000004 0x03: eax=0x3c07c163 ebx=0x04c0003f ecx=0x00004fff edx=0x00000006
   0x00000004 0x04: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000005 0x00: eax=0x00000040 ebx=0x00000040 ecx=0x00000003 edx=0x00001120
   0x00000006 0x00: eax=0x00000077 ebx=0x00000002 ecx=0x00000009 edx=0x00000000
   0x00000007 0x00: eax=0x00000000 ebx=0x00000281 ecx=0x00000000 edx=0x9c000400
   0x00000008 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000009 0x00: eax=0x00000001 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000a 0x00: eax=0x07300403 ebx=0x00000000 ecx=0x00000000 edx=0x00000603
   0x0000000b 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000030
   0x0000000b 0x01: eax=0x00000005 ebx=0x00000014 ecx=0x00000201 edx=0x00000030
   0x0000000b 0x02: eax=0x00000000 ebx=0x00000000 ecx=0x00000002 edx=0x00000030
   0x0000000c 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000d 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0x0000000d 0x01: eax=0x00000001 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000d 0x02: eax=0x00000100 ebx=0x00000240 ecx=0x00000000 edx=0x00000000
   0x20000000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0x80000000 0x00: eax=0x80000008 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000001 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000001 edx=0x2c100800
   0x80000002 0x00: eax=0x20202020 ebx=0x6e492020 ecx=0x286c6574 edx=0x58202952
   0x80000003 0x00: eax=0x286e6f65 ebx=0x43202952 ecx=0x45205550 edx=0x36322d35
   0x80000004 0x00: eax=0x76203039 ebx=0x20402032 ecx=0x30302e33 edx=0x007a4847
   0x80000005 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000006 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x01006040 edx=0x00000000
   0x80000007 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000100
   0x80000008 0x00: eax=0x0000302e ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80860000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0xc0000000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
CPU 16:
   0x00000000 0x00: eax=0x0000000d ebx=0x756e6547 ecx=0x6c65746e edx=0x49656e69
   0x00000001 0x00: eax=0x000306e4 ebx=0x32200800 ecx=0x7fbee3ff edx=0xbfebfbff
   0x00000002 0x00: eax=0x76036301 ebx=0x00f0b2ff ecx=0x00000000 edx=0x00ca0000
   0x00000003 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000004 0x00: eax=0x3c004121 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000
   0x00000004 0x01: eax=0x3c004122 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000
   0x00000004 0x02: eax=0x3c004143 ebx=0x01c0003f ecx=0x000001ff edx=0x00000000
   0x00000004 0x03: eax=0x3c07c163 ebx=0x04c0003f ecx=0x00004fff edx=0x00000006
   0x00000004 0x04: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000005 0x00: eax=0x00000040 ebx=0x00000040 ecx=0x00000003 edx=0x00001120
   0x00000006 0x00: eax=0x00000077 ebx=0x00000002 ecx=0x00000009 edx=0x00000000
   0x00000007 0x00: eax=0x00000000 ebx=0x00000281 ecx=0x00000000 edx=0x9c000400
   0x00000008 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000009 0x00: eax=0x00000001 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000a 0x00: eax=0x07300403 ebx=0x00000000 ecx=0x00000000 edx=0x00000603
   0x0000000b 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000032
   0x0000000b 0x01: eax=0x00000005 ebx=0x00000014 ecx=0x00000201 edx=0x00000032
   0x0000000b 0x02: eax=0x00000000 ebx=0x00000000 ecx=0x00000002 edx=0x00000032
   0x0000000c 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000d 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0x0000000d 0x01: eax=0x00000001 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000d 0x02: eax=0x00000100 ebx=0x00000240 ecx=0x00000000 edx=0x00000000
   0x20000000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0x80000000 0x00: eax=0x80000008 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000001 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000001 edx=0x2c100800
   0x80000002 0x00: eax=0x20202020 ebx=0x6e492020 ecx=0x286c6574 edx=0x58202952
   0x80000003 0x00: eax=0x286e6f65 ebx=0x43202952 ecx=0x45205550 edx=0x36322d35
   0x80000004 0x00: eax=0x76203039 ebx=0x20402032 ecx=0x30302e33 edx=0x007a4847
   0x80000005 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000006 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x01006040 edx=0x00000000
   0x80000007 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000100
   0x80000008 0x00: eax=0x0000302e ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80860000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0xc0000000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
CPU 17:
   0x00000000 0x00: eax=0x0000000d ebx=0x756e6547 ecx=0x6c65746e edx=0x49656e69
   0x00000001 0x00: eax=0x000306e4 ebx=0x34200800 ecx=0x7fbee3ff edx=0xbfebfbff
   0x00000002 0x00: eax=0x76036301 ebx=0x00f0b2ff ecx=0x00000000 edx=0x00ca0000
   0x00000003 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000004 0x00: eax=0x3c004121 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000
   0x00000004 0x01: eax=0x3c004122 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000
   0x00000004 0x02: eax=0x3c004143 ebx=0x01c0003f ecx=0x000001ff edx=0x00000000
   0x00000004 0x03: eax=0x3c07c163 ebx=0x04c0003f ecx=0x00004fff edx=0x00000006
   0x00000004 0x04: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000005 0x00: eax=0x00000040 ebx=0x00000040 ecx=0x00000003 edx=0x00001120
   0x00000006 0x00: eax=0x00000077 ebx=0x00000002 ecx=0x00000009 edx=0x00000000
   0x00000007 0x00: eax=0x00000000 ebx=0x00000281 ecx=0x00000000 edx=0x9c000400
   0x00000008 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000009 0x00: eax=0x00000001 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000a 0x00: eax=0x07300403 ebx=0x00000000 ecx=0x00000000 edx=0x00000603
   0x0000000b 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000034
   0x0000000b 0x01: eax=0x00000005 ebx=0x00000014 ecx=0x00000201 edx=0x00000034
   0x0000000b 0x02: eax=0x00000000 ebx=0x00000000 ecx=0x00000002 edx=0x00000034
   0x0000000c 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000d 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0x0000000d 0x01: eax=0x00000001 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000d 0x02: eax=0x00000100 ebx=0x00000240 ecx=0x00000000 edx=0x00000000
   0x20000000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0x80000000 0x00: eax=0x80000008 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000001 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000001 edx=0x2c100800
   0x80000002 0x00: eax=0x20202020 ebx=0x6e492020 ecx=0x286c6574 edx=0x58202952
   0x80000003 0x00: eax=0x286e6f65 ebx=0x43202952 ecx=0x45205550 edx=0x36322d35
   0x80000004 0x00: eax=0x76203039 ebx=0x20402032 ecx=0x30302e33 edx=0x007a4847
   0x80000005 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000006 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x01006040 edx=0x00000000
   0x80000007 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000100
   0x80000008 0x00: eax=0x0000302e ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80860000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0xc0000000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
CPU 18:
   0x00000000 0x00: eax=0x0000000d ebx=0x756e6547 ecx=0x6c65746e edx=0x49656e69
   0x00000001 0x00: eax=0x000306e4 ebx=0x36200800 ecx=0x7fbee3ff edx=0xbfebfbff
   0x00000002 0x00: eax=0x76036301 ebx=0x00f0b2ff ecx=0x00000000 edx=0x00ca0000
   0x00000003 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000004 0x00: eax=0x3c004121 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000
   0x00000004 0x01: eax=0x3c004122 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000
   0x00000004 0x02: eax=0x3c004143 ebx=0x01c0003f ecx=0x000001ff edx=0x00000000
   0x00000004 0x03: eax=0x3c07c163 ebx=0x04c0003f ecx=0x00004fff edx=0x00000006
   0x00000004 0x04: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000005 0x00: eax=0x00000040 ebx=0x00000040 ecx=0x00000003 edx=0x00001120
   0x00000006 0x00: eax=0x00000077 ebx=0x00000002 ecx=0x00000009 edx=0x00000000
   0x00000007 0x00: eax=0x00000000 ebx=0x00000281 ecx=0x00000000 edx=0x9c000400
   0x00000008 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000009 0x00: eax=0x00000001 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000a 0x00: eax=0x07300403 ebx=0x00000000 ecx=0x00000000 edx=0x00000603
   0x0000000b 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000036
   0x0000000b 0x01: eax=0x00000005 ebx=0x00000014 ecx=0x00000201 edx=0x00000036
   0x0000000b 0x02: eax=0x00000000 ebx=0x00000000 ecx=0x00000002 edx=0x00000036
   0x0000000c 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000d 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0x0000000d 0x01: eax=0x00000001 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000d 0x02: eax=0x00000100 ebx=0x00000240 ecx=0x00000000 edx=0x00000000
   0x20000000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0x80000000 0x00: eax=0x80000008 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000001 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000001 edx=0x2c100800
   0x80000002 0x00: eax=0x20202020 ebx=0x6e492020 ecx=0x286c6574 edx=0x58202952
   0x80000003 0x00: eax=0x286e6f65 ebx=0x43202952 ecx=0x45205550 edx=0x36322d35
   0x80000004 0x00: eax=0x76203039 ebx=0x20402032 ecx=0x30302e33 edx=0x007a4847
   0x80000005 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000006 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x01006040 edx=0x00000000
   0x80000007 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000100
   0x80000008 0x00: eax=0x0000302e ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80860000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0xc0000000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
CPU 19:
   0x00000000 0x00: eax=0x0000000d ebx=0x756e6547 ecx=0x6c65746e edx=0x49656e69
   0x00000001 0x00: eax=0x000306e4 ebx=0x38200800 ecx=0x7fbee3ff edx=0xbfebfbff
   0x00000002 0x00: eax=0x76036301 ebx=0x00f0b2ff ecx=0x00000000 edx=0x00ca0000
   0x00000003 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000004 0x00: eax=0x3c004121 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000
   0x00000004 0x01: eax=0x3c004122 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000
   0x00000004 0x02: eax=0x3c004143 ebx=0x01c0003f ecx=0x000001ff edx=0x00000000
   0x00000004 0x03: eax=0x3c07c163 ebx=0x04c0003f ecx=0x00004fff edx=0x00000006
   0x00000004 0x04: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000005 0x00: eax=0x00000040 ebx=0x00000040 ecx=0x00000003 edx=0x00001120
   0x00000006 0x00: eax=0x00000077 ebx=0x00000002 ecx=0x00000009 edx=0x00000000
   0x00000007 0x00: eax=0x00000000 ebx=0x00000281 ecx=0x00000000 edx=0x9c000400
   0x00000008 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000009 0x00: eax=0x00000001 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000a 0x00: eax=0x07300403 ebx=0x00000000 ecx=0x00000000 edx=0x00000603
   0x0000000b 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000038
   0x0000000b 0x01: eax=0x00000005 ebx=0x00000014 ecx=0x00000201 edx=0x00000038
   0x0000000b 0x02: eax=0x00000000 ebx=0x00000000 ecx=0x00000002 edx=0x00000038
   0x0000000c 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000d 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0x0000000d 0x01: eax=0x00000001 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000d 0x02: eax=0x00000100 ebx=0x00000240 ecx=0x00000000 edx=0x00000000
   0x20000000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0x80000000 0x00: eax=0x80000008 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000001 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000001 edx=0x2c100800
   0x80000002 0x00: eax=0x20202020 ebx=0x6e492020 ecx=0x286c6574 edx=0x58202952
   0x80000003 0x00: eax=0x286e6f65 ebx=0x43202952 ecx=0x45205550 edx=0x36322d35
   0x80000004 0x00: eax=0x76203039 ebx=0x20402032 ecx=0x30302e33 edx=0x007a4847
   0x80000005 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000006 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x01006040 edx=0x00000000
   0x80000007 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000100
   0x80000008 0x00: eax=0x0000302e ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80860000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0xc0000000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
CPU 20:
   0x00000000 0x00: eax=0x0000000d ebx=0x756e6547 ecx=0x6c65746e edx=0x49656e69
   0x00000001 0x00: eax=0x000306e4 ebx=0x01200800 ecx=0x7fbee3ff edx=0xbfebfbff
   0x00000002 0x00: eax=0x76036301 ebx=0x00f0b2ff ecx=0x00000000 edx=0x00ca0000
   0x00000003 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000004 0x00: eax=0x3c004121 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000
   0x00000004 0x01: eax=0x3c004122 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000
   0x00000004 0x02: eax=0x3c004143 ebx=0x01c0003f ecx=0x000001ff edx=0x00000000
   0x00000004 0x03: eax=0x3c07c163 ebx=0x04c0003f ecx=0x00004fff edx=0x00000006
   0x00000004 0x04: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000005 0x00: eax=0x00000040 ebx=0x00000040 ecx=0x00000003 edx=0x00001120
   0x00000006 0x00: eax=0x00000077 ebx=0x00000002 ecx=0x00000009 edx=0x00000000
   0x00000007 0x00: eax=0x00000000 ebx=0x00000281 ecx=0x00000000 edx=0x9c000400
   0x00000008 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000009 0x00: eax=0x00000001 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000a 0x00: eax=0x07300403 ebx=0x00000000 ecx=0x00000000 edx=0x00000603
   0x0000000b 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000001
   0x0000000b 0x01: eax=0x00000005 ebx=0x00000014 ecx=0x00000201 edx=0x00000001
   0x0000000b 0x02: eax=0x00000000 ebx=0x00000000 ecx=0x00000002 edx=0x00000001
   0x0000000c 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000d 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0x0000000d 0x01: eax=0x00000001 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000d 0x02: eax=0x00000100 ebx=0x00000240 ecx=0x00000000 edx=0x00000000
   0x20000000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0x80000000 0x00: eax=0x80000008 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000001 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000001 edx=0x2c100800
   0x80000002 0x00: eax=0x20202020 ebx=0x6e492020 ecx=0x286c6574 edx=0x58202952
   0x80000003 0x00: eax=0x286e6f65 ebx=0x43202952 ecx=0x45205550 edx=0x36322d35
   0x80000004 0x00: eax=0x76203039 ebx=0x20402032 ecx=0x30302e33 edx=0x007a4847
   0x80000005 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000006 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x01006040 edx=0x00000000
   0x80000007 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000100
   0x80000008 0x00: eax=0x0000302e ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80860000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0xc0000000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
CPU 21:
   0x00000000 0x00: eax=0x0000000d ebx=0x756e6547 ecx=0x6c65746e edx=0x49656e69
   0x00000001 0x00: eax=0x000306e4 ebx=0x03200800 ecx=0x7fbee3ff edx=0xbfebfbff
   0x00000002 0x00: eax=0x76036301 ebx=0x00f0b2ff ecx=0x00000000 edx=0x00ca0000
   0x00000003 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000004 0x00: eax=0x3c004121 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000
   0x00000004 0x01: eax=0x3c004122 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000
   0x00000004 0x02: eax=0x3c004143 ebx=0x01c0003f ecx=0x000001ff edx=0x00000000
   0x00000004 0x03: eax=0x3c07c163 ebx=0x04c0003f ecx=0x00004fff edx=0x00000006
   0x00000004 0x04: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000005 0x00: eax=0x00000040 ebx=0x00000040 ecx=0x00000003 edx=0x00001120
   0x00000006 0x00: eax=0x00000077 ebx=0x00000002 ecx=0x00000009 edx=0x00000000
   0x00000007 0x00: eax=0x00000000 ebx=0x00000281 ecx=0x00000000 edx=0x9c000400
   0x00000008 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000009 0x00: eax=0x00000001 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000a 0x00: eax=0x07300403 ebx=0x00000000 ecx=0x00000000 edx=0x00000603
   0x0000000b 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000003
   0x0000000b 0x01: eax=0x00000005 ebx=0x00000014 ecx=0x00000201 edx=0x00000003
   0x0000000b 0x02: eax=0x00000000 ebx=0x00000000 ecx=0x00000002 edx=0x00000003
   0x0000000c 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000d 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0x0000000d 0x01: eax=0x00000001 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000d 0x02: eax=0x00000100 ebx=0x00000240 ecx=0x00000000 edx=0x00000000
   0x20000000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0x80000000 0x00: eax=0x80000008 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000001 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000001 edx=0x2c100800
   0x80000002 0x00: eax=0x20202020 ebx=0x6e492020 ecx=0x286c6574 edx=0x58202952
   0x80000003 0x00: eax=0x286e6f65 ebx=0x43202952 ecx=0x45205550 edx=0x36322d35
   0x80000004 0x00: eax=0x76203039 ebx=0x20402032 ecx=0x30302e33 edx=0x007a4847
   0x80000005 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000006 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x01006040 edx=0x00000000
   0x80000007 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000100
   0x80000008 0x00: eax=0x0000302e ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80860000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0xc0000000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
CPU 22:
   0x00000000 0x00: eax=0x0000000d ebx=0x756e6547 ecx=0x6c65746e edx=0x49656e69
   0x00000001 0x00: eax=0x000306e4 ebx=0x05200800 ecx=0x7fbee3ff edx=0xbfebfbff
   0x00000002 0x00: eax=0x76036301 ebx=0x00f0b2ff ecx=0x00000000 edx=0x00ca0000
   0x00000003 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000004 0x00: eax=0x3c004121 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000
   0x00000004 0x01: eax=0x3c004122 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000
   0x00000004 0x02: eax=0x3c004143 ebx=0x01c0003f ecx=0x000001ff edx=0x00000000
   0x00000004 0x03: eax=0x3c07c163 ebx=0x04c0003f ecx=0x00004fff edx=0x00000006
   0x00000004 0x04: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000005 0x00: eax=0x00000040 ebx=0x00000040 ecx=0x00000003 edx=0x00001120
   0x00000006 0x00: eax=0x00000077 ebx=0x00000002 ecx=0x00000009 edx=0x00000000
   0x00000007 0x00: eax=0x00000000 ebx=0x00000281 ecx=0x00000000 edx=0x9c000400
   0x00000008 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000009 0x00: eax=0x00000001 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000a 0x00: eax=0x07300403 ebx=0x00000000 ecx=0x00000000 edx=0x00000603
   0x0000000b 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000005
   0x0000000b 0x01: eax=0x00000005 ebx=0x00000014 ecx=0x00000201 edx=0x00000005
   0x0000000b 0x02: eax=0x00000000 ebx=0x00000000 ecx=0x00000002 edx=0x00000005
   0x0000000c 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000d 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0x0000000d 0x01: eax=0x00000001 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000d 0x02: eax=0x00000100 ebx=0x00000240 ecx=0x00000000 edx=0x00000000
   0x20000000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0x80000000 0x00: eax=0x80000008 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000001 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000001 edx=0x2c100800
   0x80000002 0x00: eax=0x20202020 ebx=0x6e492020 ecx=0x286c6574 edx=0x58202952
   0x80000003 0x00: eax=0x286e6f65 ebx=0x43202952 ecx=0x45205550 edx=0x36322d35
   0x80000004 0x00: eax=0x76203039 ebx=0x20402032 ecx=0x30302e33 edx=0x007a4847
   0x80000005 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000006 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x01006040 edx=0x00000000
   0x80000007 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000100
   0x80000008 0x00: eax=0x0000302e ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80860000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0xc0000000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
CPU 23:
   0x00000000 0x00: eax=0x0000000d ebx=0x756e6547 ecx=0x6c65746e edx=0x49656e69
   0x00000001 0x00: eax=0x000306e4 ebx=0x07200800 ecx=0x7fbee3ff edx=0xbfebfbff
   0x00000002 0x00: eax=0x76036301 ebx=0x00f0b2ff ecx=0x00000000 edx=0x00ca0000
   0x00000003 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000004 0x00: eax=0x3c004121 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000
   0x00000004 0x01: eax=0x3c004122 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000
   0x00000004 0x02: eax=0x3c004143 ebx=0x01c0003f ecx=0x000001ff edx=0x00000000
   0x00000004 0x03: eax=0x3c07c163 ebx=0x04c0003f ecx=0x00004fff edx=0x00000006
   0x00000004 0x04: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000005 0x00: eax=0x00000040 ebx=0x00000040 ecx=0x00000003 edx=0x00001120
   0x00000006 0x00: eax=0x00000077 ebx=0x00000002 ecx=0x00000009 edx=0x00000000
   0x00000007 0x00: eax=0x00000000 ebx=0x00000281 ecx=0x00000000 edx=0x9c000400
   0x00000008 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000009 0x00: eax=0x00000001 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000a 0x00: eax=0x07300403 ebx=0x00000000 ecx=0x00000000 edx=0x00000603
   0x0000000b 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000007
   0x0000000b 0x01: eax=0x00000005 ebx=0x00000014 ecx=0x00000201 edx=0x00000007
   0x0000000b 0x02: eax=0x00000000 ebx=0x00000000 ecx=0x00000002 edx=0x00000007
   0x0000000c 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000d 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0x0000000d 0x01: eax=0x00000001 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000d 0x02: eax=0x00000100 ebx=0x00000240 ecx=0x00000000 edx=0x00000000
   0x20000000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0x80000000 0x00: eax=0x80000008 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000001 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000001 edx=0x2c100800
   0x80000002 0x00: eax=0x20202020 ebx=0x6e492020 ecx=0x286c6574 edx=0x58202952
   0x80000003 0x00: eax=0x286e6f65 ebx=0x43202952 ecx=0x45205550 edx=0x36322d35
   0x80000004 0x00: eax=0x76203039 ebx=0x20402032 ecx=0x30302e33 edx=0x007a4847
   0x80000005 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000006 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x01006040 edx=0x00000000
   0x80000007 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000100
   0x80000008 0x00: eax=0x0000302e ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80860000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0xc0000000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
CPU 24:
   0x00000000 0x00: eax=0x0000000d ebx=0x756e6547 ecx=0x6c65746e edx=0x49656e69
   0x00000001 0x00: eax=0x000306e4 ebx=0x09200800 ecx=0x7fbee3ff edx=0xbfebfbff
   0x00000002 0x00: eax=0x76036301 ebx=0x00f0b2ff ecx=0x00000000 edx=0x00ca0000
   0x00000003 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000004 0x00: eax=0x3c004121 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000
   0x00000004 0x01: eax=0x3c004122 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000
   0x00000004 0x02: eax=0x3c004143 ebx=0x01c0003f ecx=0x000001ff edx=0x00000000
   0x00000004 0x03: eax=0x3c07c163 ebx=0x04c0003f ecx=0x00004fff edx=0x00000006
   0x00000004 0x04: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000005 0x00: eax=0x00000040 ebx=0x00000040 ecx=0x00000003 edx=0x00001120
   0x00000006 0x00: eax=0x00000077 ebx=0x00000002 ecx=0x00000009 edx=0x00000000
   0x00000007 0x00: eax=0x00000000 ebx=0x00000281 ecx=0x00000000 edx=0x9c000400
   0x00000008 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000009 0x00: eax=0x00000001 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000a 0x00: eax=0x07300403 ebx=0x00000000 ecx=0x00000000 edx=0x00000603
   0x0000000b 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000009
   0x0000000b 0x01: eax=0x00000005 ebx=0x00000014 ecx=0x00000201 edx=0x00000009
   0x0000000b 0x02: eax=0x00000000 ebx=0x00000000 ecx=0x00000002 edx=0x00000009
   0x0000000c 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000d 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0x0000000d 0x01: eax=0x00000001 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000d 0x02: eax=0x00000100 ebx=0x00000240 ecx=0x00000000 edx=0x00000000
   0x20000000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0x80000000 0x00: eax=0x80000008 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000001 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000001 edx=0x2c100800
   0x80000002 0x00: eax=0x20202020 ebx=0x6e492020 ecx=0x286c6574 edx=0x58202952
   0x80000003 0x00: eax=0x286e6f65 ebx=0x43202952 ecx=0x45205550 edx=0x36322d35
   0x80000004 0x00: eax=0x76203039 ebx=0x20402032 ecx=0x30302e33 edx=0x007a4847
   0x80000005 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000006 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x01006040 edx=0x00000000
   0x80000007 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000100
   0x80000008 0x00: eax=0x0000302e ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80860000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0xc0000000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
CPU 25:
   0x00000000 0x00: eax=0x0000000d ebx=0x756e6547 ecx=0x6c65746e edx=0x49656e69
   0x00000001 0x00: eax=0x000306e4 ebx=0x11200800 ecx=0x7fbee3ff edx=0xbfebfbff
   0x00000002 0x00: eax=0x76036301 ebx=0x00f0b2ff ecx=0x00000000 edx=0x00ca0000
   0x00000003 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000004 0x00: eax=0x3c004121 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000
   0x00000004 0x01: eax=0x3c004122 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000
   0x00000004 0x02: eax=0x3c004143 ebx=0x01c0003f ecx=0x000001ff edx=0x00000000
   0x00000004 0x03: eax=0x3c07c163 ebx=0x04c0003f ecx=0x00004fff edx=0x00000006
   0x00000004 0x04: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000005 0x00: eax=0x00000040 ebx=0x00000040 ecx=0x00000003 edx=0x00001120
   0x00000006 0x00: eax=0x00000077 ebx=0x00000002 ecx=0x00000009 edx=0x00000000
   0x00000007 0x00: eax=0x00000000 ebx=0x00000281 ecx=0x00000000 edx=0x9c000400
   0x00000008 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000009 0x00: eax=0x00000001 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000a 0x00: eax=0x07300403 ebx=0x00000000 ecx=0x00000000 edx=0x00000603
   0x0000000b 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000011
   0x0000000b 0x01: eax=0x00000005 ebx=0x00000014 ecx=0x00000201 edx=0x00000011
   0x0000000b 0x02: eax=0x00000000 ebx=0x00000000 ecx=0x00000002 edx=0x00000011
   0x0000000c 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000d 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0x0000000d 0x01: eax=0x00000001 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000d 0x02: eax=0x00000100 ebx=0x00000240 ecx=0x00000000 edx=0x00000000
   0x20000000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0x80000000 0x00: eax=0x80000008 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000001 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000001 edx=0x2c100800
   0x80000002 0x00: eax=0x20202020 ebx=0x6e492020 ecx=0x286c6574 edx=0x58202952
   0x80000003 0x00: eax=0x286e6f65 ebx=0x43202952 ecx=0x45205550 edx=0x36322d35
   0x80000004 0x00: eax=0x76203039 ebx=0x20402032 ecx=0x30302e33 edx=0x007a4847
   0x80000005 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000006 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x01006040 edx=0x00000000
   0x80000007 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000100
   0x80000008 0x00: eax=0x0000302e ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80860000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0xc0000000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
CPU 26:
   0x00000000 0x00: eax=0x0000000d ebx=0x756e6547 ecx=0x6c65746e edx=0x49656e69
   0x00000001 0x00: eax=0x000306e4 ebx=0x13200800 ecx=0x7fbee3ff edx=0xbfebfbff
   0x00000002 0x00: eax=0x76036301 ebx=0x00f0b2ff ecx=0x00000000 edx=0x00ca0000
   0x00000003 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000004 0x00: eax=0x3c004121 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000
   0x00000004 0x01: eax=0x3c004122 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000
   0x00000004 0x02: eax=0x3c004143 ebx=0x01c0003f ecx=0x000001ff edx=0x00000000
   0x00000004 0x03: eax=0x3c07c163 ebx=0x04c0003f ecx=0x00004fff edx=0x00000006
   0x00000004 0x04: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000005 0x00: eax=0x00000040 ebx=0x00000040 ecx=0x00000003 edx=0x00001120
   0x00000006 0x00: eax=0x00000077 ebx=0x00000002 ecx=0x00000009 edx=0x00000000
   0x00000007 0x00: eax=0x00000000 ebx=0x00000281 ecx=0x00000000 edx=0x9c000400
   0x00000008 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000009 0x00: eax=0x00000001 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000a 0x00: eax=0x07300403 ebx=0x00000000 ecx=0x00000000 edx=0x00000603
   0x0000000b 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000013
   0x0000000b 0x01: eax=0x00000005 ebx=0x00000014 ecx=0x00000201 edx=0x00000013
   0x0000000b 0x02: eax=0x00000000 ebx=0x00000000 ecx=0x00000002 edx=0x00000013
   0x0000000c 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000d 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0x0000000d 0x01: eax=0x00000001 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000d 0x02: eax=0x00000100 ebx=0x00000240 ecx=0x00000000 edx=0x00000000
   0x20000000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0x80000000 0x00: eax=0x80000008 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000001 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000001 edx=0x2c100800
   0x80000002 0x00: eax=0x20202020 ebx=0x6e492020 ecx=0x286c6574 edx=0x58202952
   0x80000003 0x00: eax=0x286e6f65 ebx=0x43202952 ecx=0x45205550 edx=0x36322d35
   0x80000004 0x00: eax=0x76203039 ebx=0x20402032 ecx=0x30302e33 edx=0x007a4847
   0x80000005 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000006 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x01006040 edx=0x00000000
   0x80000007 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000100
   0x80000008 0x00: eax=0x0000302e ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80860000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0xc0000000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
CPU 27:
   0x00000000 0x00: eax=0x0000000d ebx=0x756e6547 ecx=0x6c65746e edx=0x49656e69
   0x00000001 0x00: eax=0x000306e4 ebx=0x15200800 ecx=0x7fbee3ff edx=0xbfebfbff
   0x00000002 0x00: eax=0x76036301 ebx=0x00f0b2ff ecx=0x00000000 edx=0x00ca0000
   0x00000003 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000004 0x00: eax=0x3c004121 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000
   0x00000004 0x01: eax=0x3c004122 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000
   0x00000004 0x02: eax=0x3c004143 ebx=0x01c0003f ecx=0x000001ff edx=0x00000000
   0x00000004 0x03: eax=0x3c07c163 ebx=0x04c0003f ecx=0x00004fff edx=0x00000006
   0x00000004 0x04: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000005 0x00: eax=0x00000040 ebx=0x00000040 ecx=0x00000003 edx=0x00001120
   0x00000006 0x00: eax=0x00000077 ebx=0x00000002 ecx=0x00000009 edx=0x00000000
   0x00000007 0x00: eax=0x00000000 ebx=0x00000281 ecx=0x00000000 edx=0x9c000400
   0x00000008 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000009 0x00: eax=0x00000001 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000a 0x00: eax=0x07300403 ebx=0x00000000 ecx=0x00000000 edx=0x00000603
   0x0000000b 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000015
   0x0000000b 0x01: eax=0x00000005 ebx=0x00000014 ecx=0x00000201 edx=0x00000015
   0x0000000b 0x02: eax=0x00000000 ebx=0x00000000 ecx=0x00000002 edx=0x00000015
   0x0000000c 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000d 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0x0000000d 0x01: eax=0x00000001 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000d 0x02: eax=0x00000100 ebx=0x00000240 ecx=0x00000000 edx=0x00000000
   0x20000000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0x80000000 0x00: eax=0x80000008 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000001 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000001 edx=0x2c100800
   0x80000002 0x00: eax=0x20202020 ebx=0x6e492020 ecx=0x286c6574 edx=0x58202952
   0x80000003 0x00: eax=0x286e6f65 ebx=0x43202952 ecx=0x45205550 edx=0x36322d35
   0x80000004 0x00: eax=0x76203039 ebx=0x20402032 ecx=0x30302e33 edx=0x007a4847
   0x80000005 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000006 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x01006040 edx=0x00000000
   0x80000007 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000100
   0x80000008 0x00: eax=0x0000302e ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80860000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0xc0000000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
CPU 28:
   0x00000000 0x00: eax=0x0000000d ebx=0x756e6547 ecx=0x6c65746e edx=0x49656e69
   0x00000001 0x00: eax=0x000306e4 ebx=0x17200800 ecx=0x7fbee3ff edx=0xbfebfbff
   0x00000002 0x00: eax=0x76036301 ebx=0x00f0b2ff ecx=0x00000000 edx=0x00ca0000
   0x00000003 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000004 0x00: eax=0x3c004121 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000
   0x00000004 0x01: eax=0x3c004122 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000
   0x00000004 0x02: eax=0x3c004143 ebx=0x01c0003f ecx=0x000001ff edx=0x00000000
   0x00000004 0x03: eax=0x3c07c163 ebx=0x04c0003f ecx=0x00004fff edx=0x00000006
   0x00000004 0x04: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000005 0x00: eax=0x00000040 ebx=0x00000040 ecx=0x00000003 edx=0x00001120
   0x00000006 0x00: eax=0x00000077 ebx=0x00000002 ecx=0x00000009 edx=0x00000000
   0x00000007 0x00: eax=0x00000000 ebx=0x00000281 ecx=0x00000000 edx=0x9c000400
   0x00000008 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000009 0x00: eax=0x00000001 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000a 0x00: eax=0x07300403 ebx=0x00000000 ecx=0x00000000 edx=0x00000603
   0x0000000b 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000017
   0x0000000b 0x01: eax=0x00000005 ebx=0x00000014 ecx=0x00000201 edx=0x00000017
   0x0000000b 0x02: eax=0x00000000 ebx=0x00000000 ecx=0x00000002 edx=0x00000017
   0x0000000c 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000d 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0x0000000d 0x01: eax=0x00000001 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000d 0x02: eax=0x00000100 ebx=0x00000240 ecx=0x00000000 edx=0x00000000
   0x20000000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0x80000000 0x00: eax=0x80000008 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000001 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000001 edx=0x2c100800
   0x80000002 0x00: eax=0x20202020 ebx=0x6e492020 ecx=0x286c6574 edx=0x58202952
   0x80000003 0x00: eax=0x286e6f65 ebx=0x43202952 ecx=0x45205550 edx=0x36322d35
   0x80000004 0x00: eax=0x76203039 ebx=0x20402032 ecx=0x30302e33 edx=0x007a4847
   0x80000005 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000006 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x01006040 edx=0x00000000
   0x80000007 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000100
   0x80000008 0x00: eax=0x0000302e ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80860000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0xc0000000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
CPU 29:
   0x00000000 0x00: eax=0x0000000d ebx=0x756e6547 ecx=0x6c65746e edx=0x49656e69
   0x00000001 0x00: eax=0x000306e4 ebx=0x19200800 ecx=0x7fbee3ff edx=0xbfebfbff
   0x00000002 0x00: eax=0x76036301 ebx=0x00f0b2ff ecx=0x00000000 edx=0x00ca0000
   0x00000003 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000004 0x00: eax=0x3c004121 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000
   0x00000004 0x01: eax=0x3c004122 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000
   0x00000004 0x02: eax=0x3c004143 ebx=0x01c0003f ecx=0x000001ff edx=0x00000000
   0x00000004 0x03: eax=0x3c07c163 ebx=0x04c0003f ecx=0x00004fff edx=0x00000006
   0x00000004 0x04: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000005 0x00: eax=0x00000040 ebx=0x00000040 ecx=0x00000003 edx=0x00001120
   0x00000006 0x00: eax=0x00000077 ebx=0x00000002 ecx=0x00000009 edx=0x00000000
   0x00000007 0x00: eax=0x00000000 ebx=0x00000281 ecx=0x00000000 edx=0x9c000400
   0x00000008 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000009 0x00: eax=0x00000001 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000a 0x00: eax=0x07300403 ebx=0x00000000 ecx=0x00000000 edx=0x00000603
   0x0000000b 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000019
   0x0000000b 0x01: eax=0x00000005 ebx=0x00000014 ecx=0x00000201 edx=0x00000019
   0x0000000b 0x02: eax=0x00000000 ebx=0x00000000 ecx=0x00000002 edx=0x00000019
   0x0000000c 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000d 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0x0000000d 0x01: eax=0x00000001 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000d 0x02: eax=0x00000100 ebx=0x00000240 ecx=0x00000000 edx=0x00000000
   0x20000000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0x80000000 0x00: eax=0x80000008 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000001 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000001 edx=0x2c100800
   0x80000002 0x00: eax=0x20202020 ebx=0x6e492020 ecx=0x286c6574 edx=0x58202952
   0x80000003 0x00: eax=0x286e6f65 ebx=0x43202952 ecx=0x45205550 edx=0x36322d35
   0x80000004 0x00: eax=0x76203039 ebx=0x20402032 ecx=0x30302e33 edx=0x007a4847
   0x80000005 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000006 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x01006040 edx=0x00000000
   0x80000007 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000100
   0x80000008 0x00: eax=0x0000302e ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80860000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0xc0000000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
CPU 30:
   0x00000000 0x00: eax=0x0000000d ebx=0x756e6547 ecx=0x6c65746e edx=0x49656e69
   0x00000001 0x00: eax=0x000306e4 ebx=0x21200800 ecx=0x7fbee3ff edx=0xbfebfbff
   0x00000002 0x00: eax=0x76036301 ebx=0x00f0b2ff ecx=0x00000000 edx=0x00ca0000
   0x00000003 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000004 0x00: eax=0x3c004121 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000
   0x00000004 0x01: eax=0x3c004122 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000
   0x00000004 0x02: eax=0x3c004143 ebx=0x01c0003f ecx=0x000001ff edx=0x00000000
   0x00000004 0x03: eax=0x3c07c163 ebx=0x04c0003f ecx=0x00004fff edx=0x00000006
   0x00000004 0x04: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000005 0x00: eax=0x00000040 ebx=0x00000040 ecx=0x00000003 edx=0x00001120
   0x00000006 0x00: eax=0x00000077 ebx=0x00000002 ecx=0x00000009 edx=0x00000000
   0x00000007 0x00: eax=0x00000000 ebx=0x00000281 ecx=0x00000000 edx=0x9c000400
   0x00000008 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000009 0x00: eax=0x00000001 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000a 0x00: eax=0x07300403 ebx=0x00000000 ecx=0x00000000 edx=0x00000603
   0x0000000b 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000021
   0x0000000b 0x01: eax=0x00000005 ebx=0x00000014 ecx=0x00000201 edx=0x00000021
   0x0000000b 0x02: eax=0x00000000 ebx=0x00000000 ecx=0x00000002 edx=0x00000021
   0x0000000c 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000d 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0x0000000d 0x01: eax=0x00000001 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000d 0x02: eax=0x00000100 ebx=0x00000240 ecx=0x00000000 edx=0x00000000
   0x20000000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0x80000000 0x00: eax=0x80000008 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000001 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000001 edx=0x2c100800
   0x80000002 0x00: eax=0x20202020 ebx=0x6e492020 ecx=0x286c6574 edx=0x58202952
   0x80000003 0x00: eax=0x286e6f65 ebx=0x43202952 ecx=0x45205550 edx=0x36322d35
   0x80000004 0x00: eax=0x76203039 ebx=0x20402032 ecx=0x30302e33 edx=0x007a4847
   0x80000005 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000006 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x01006040 edx=0x00000000
   0x80000007 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000100
   0x80000008 0x00: eax=0x0000302e ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80860000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0xc0000000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
CPU 31:
   0x00000000 0x00: eax=0x0000000d ebx=0x756e6547 ecx=0x6c65746e edx=0x49656e69
   0x00000001 0x00: eax=0x000306e4 ebx=0x23200800 ecx=0x7fbee3ff edx=0xbfebfbff
   0x00000002 0x00: eax=0x76036301 ebx=0x00f0b2ff ecx=0x00000000 edx=0x00ca0000
   0x00000003 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000004 0x00: eax=0x3c004121 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000
   0x00000004 0x01: eax=0x3c004122 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000
   0x00000004 0x02: eax=0x3c004143 ebx=0x01c0003f ecx=0x000001ff edx=0x00000000
   0x00000004 0x03: eax=0x3c07c163 ebx=0x04c0003f ecx=0x00004fff edx=0x00000006
   0x00000004 0x04: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000005 0x00: eax=0x00000040 ebx=0x00000040 ecx=0x00000003 edx=0x00001120
   0x00000006 0x00: eax=0x00000077 ebx=0x00000002 ecx=0x00000009 edx=0x00000000
   0x00000007 0x00: eax=0x00000000 ebx=0x00000281 ecx=0x00000000 edx=0x9c000400
   0x00000008 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000009 0x00: eax=0x00000001 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000a 0x00: eax=0x07300403 ebx=0x00000000 ecx=0x00000000 edx=0x00000603
   0x0000000b 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000023
   0x0000000b 0x01: eax=0x00000005 ebx=0x00000014 ecx=0x00000201 edx=0x00000023
   0x0000000b 0x02: eax=0x00000000 ebx=0x00000000 ecx=0x00000002 edx=0x00000023
   0x0000000c 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000d 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0x0000000d 0x01: eax=0x00000001 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000d 0x02: eax=0x00000100 ebx=0x00000240 ecx=0x00000000 edx=0x00000000
   0x20000000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0x80000000 0x00: eax=0x80000008 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000001 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000001 edx=0x2c100800
   0x80000002 0x00: eax=0x20202020 ebx=0x6e492020 ecx=0x286c6574 edx=0x58202952
   0x80000003 0x00: eax=0x286e6f65 ebx=0x43202952 ecx=0x45205550 edx=0x36322d35
   0x80000004 0x00: eax=0x76203039 ebx=0x20402032 ecx=0x30302e33 edx=0x007a4847
   0x80000005 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000006 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x01006040 edx=0x00000000
   0x80000007 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000100
   0x80000008 0x00: eax=0x0000302e ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80860000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0xc0000000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
CPU 32:
   0x00000000 0x00: eax=0x0000000d ebx=0x756e6547 ecx=0x6c65746e edx=0x49656e69
   0x00000001 0x00: eax=0x000306e4 ebx=0x25200800 ecx=0x7fbee3ff edx=0xbfebfbff
   0x00000002 0x00: eax=0x76036301 ebx=0x00f0b2ff ecx=0x00000000 edx=0x00ca0000
   0x00000003 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000004 0x00: eax=0x3c004121 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000
   0x00000004 0x01: eax=0x3c004122 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000
   0x00000004 0x02: eax=0x3c004143 ebx=0x01c0003f ecx=0x000001ff edx=0x00000000
   0x00000004 0x03: eax=0x3c07c163 ebx=0x04c0003f ecx=0x00004fff edx=0x00000006
   0x00000004 0x04: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000005 0x00: eax=0x00000040 ebx=0x00000040 ecx=0x00000003 edx=0x00001120
   0x00000006 0x00: eax=0x00000077 ebx=0x00000002 ecx=0x00000009 edx=0x00000000
   0x00000007 0x00: eax=0x00000000 ebx=0x00000281 ecx=0x00000000 edx=0x9c000400
   0x00000008 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000009 0x00: eax=0x00000001 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000a 0x00: eax=0x07300403 ebx=0x00000000 ecx=0x00000000 edx=0x00000603
   0x0000000b 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000025
   0x0000000b 0x01: eax=0x00000005 ebx=0x00000014 ecx=0x00000201 edx=0x00000025
   0x0000000b 0x02: eax=0x00000000 ebx=0x00000000 ecx=0x00000002 edx=0x00000025
   0x0000000c 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000d 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0x0000000d 0x01: eax=0x00000001 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000d 0x02: eax=0x00000100 ebx=0x00000240 ecx=0x00000000 edx=0x00000000
   0x20000000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0x80000000 0x00: eax=0x80000008 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000001 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000001 edx=0x2c100800
   0x80000002 0x00: eax=0x20202020 ebx=0x6e492020 ecx=0x286c6574 edx=0x58202952
   0x80000003 0x00: eax=0x286e6f65 ebx=0x43202952 ecx=0x45205550 edx=0x36322d35
   0x80000004 0x00: eax=0x76203039 ebx=0x20402032 ecx=0x30302e33 edx=0x007a4847
   0x80000005 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000006 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x01006040 edx=0x00000000
   0x80000007 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000100
   0x80000008 0x00: eax=0x0000302e ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80860000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0xc0000000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
CPU 33:
   0x00000000 0x00: eax=0x0000000d ebx=0x756e6547 ecx=0x6c65746e edx=0x49656e69
   0x00000001 0x00: eax=0x000306e4 ebx=0x27200800 ecx=0x7fbee3ff edx=0xbfebfbff
   0x00000002 0x00: eax=0x76036301 ebx=0x00f0b2ff ecx=0x00000000 edx=0x00ca0000
   0x00000003 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000004 0x00: eax=0x3c004121 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000
   0x00000004 0x01: eax=0x3c004122 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000
   0x00000004 0x02: eax=0x3c004143 ebx=0x01c0003f ecx=0x000001ff edx=0x00000000
   0x00000004 0x03: eax=0x3c07c163 ebx=0x04c0003f ecx=0x00004fff edx=0x00000006
   0x00000004 0x04: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000005 0x00: eax=0x00000040 ebx=0x00000040 ecx=0x00000003 edx=0x00001120
   0x00000006 0x00: eax=0x00000077 ebx=0x00000002 ecx=0x00000009 edx=0x00000000
   0x00000007 0x00: eax=0x00000000 ebx=0x00000281 ecx=0x00000000 edx=0x9c000400
   0x00000008 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000009 0x00: eax=0x00000001 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000a 0x00: eax=0x07300403 ebx=0x00000000 ecx=0x00000000 edx=0x00000603
   0x0000000b 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000027
   0x0000000b 0x01: eax=0x00000005 ebx=0x00000014 ecx=0x00000201 edx=0x00000027
   0x0000000b 0x02: eax=0x00000000 ebx=0x00000000 ecx=0x00000002 edx=0x00000027
   0x0000000c 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000d 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0x0000000d 0x01: eax=0x00000001 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000d 0x02: eax=0x00000100 ebx=0x00000240 ecx=0x00000000 edx=0x00000000
   0x20000000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0x80000000 0x00: eax=0x80000008 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000001 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000001 edx=0x2c100800
   0x80000002 0x00: eax=0x20202020 ebx=0x6e492020 ecx=0x286c6574 edx=0x58202952
   0x80000003 0x00: eax=0x286e6f65 ebx=0x43202952 ecx=0x45205550 edx=0x36322d35
   0x80000004 0x00: eax=0x76203039 ebx=0x20402032 ecx=0x30302e33 edx=0x007a4847
   0x80000005 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000006 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x01006040 edx=0x00000000
   0x80000007 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000100
   0x80000008 0x00: eax=0x0000302e ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80860000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0xc0000000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
CPU 34:
   0x00000000 0x00: eax=0x0000000d ebx=0x756e6547 ecx=0x6c65746e edx=0x49656e69
   0x00000001 0x00: eax=0x000306e4 ebx=0x29200800 ecx=0x7fbee3ff edx=0xbfebfbff
   0x00000002 0x00: eax=0x76036301 ebx=0x00f0b2ff ecx=0x00000000 edx=0x00ca0000
   0x00000003 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000004 0x00: eax=0x3c004121 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000
   0x00000004 0x01: eax=0x3c004122 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000
   0x00000004 0x02: eax=0x3c004143 ebx=0x01c0003f ecx=0x000001ff edx=0x00000000
   0x00000004 0x03: eax=0x3c07c163 ebx=0x04c0003f ecx=0x00004fff edx=0x00000006
   0x00000004 0x04: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000005 0x00: eax=0x00000040 ebx=0x00000040 ecx=0x00000003 edx=0x00001120
   0x00000006 0x00: eax=0x00000077 ebx=0x00000002 ecx=0x00000009 edx=0x00000000
   0x00000007 0x00: eax=0x00000000 ebx=0x00000281 ecx=0x00000000 edx=0x9c000400
   0x00000008 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000009 0x00: eax=0x00000001 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000a 0x00: eax=0x07300403 ebx=0x00000000 ecx=0x00000000 edx=0x00000603
   0x0000000b 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000029
   0x0000000b 0x01: eax=0x00000005 ebx=0x00000014 ecx=0x00000201 edx=0x00000029
   0x0000000b 0x02: eax=0x00000000 ebx=0x00000000 ecx=0x00000002 edx=0x00000029
   0x0000000c 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000d 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0x0000000d 0x01: eax=0x00000001 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000d 0x02: eax=0x00000100 ebx=0x00000240 ecx=0x00000000 edx=0x00000000
   0x20000000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0x80000000 0x00: eax=0x80000008 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000001 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000001 edx=0x2c100800
   0x80000002 0x00: eax=0x20202020 ebx=0x6e492020 ecx=0x286c6574 edx=0x58202952
   0x80000003 0x00: eax=0x286e6f65 ebx=0x43202952 ecx=0x45205550 edx=0x36322d35
   0x80000004 0x00: eax=0x76203039 ebx=0x20402032 ecx=0x30302e33 edx=0x007a4847
   0x80000005 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000006 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x01006040 edx=0x00000000
   0x80000007 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000100
   0x80000008 0x00: eax=0x0000302e ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80860000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0xc0000000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
CPU 35:
   0x00000000 0x00: eax=0x0000000d ebx=0x756e6547 ecx=0x6c65746e edx=0x49656e69
   0x00000001 0x00: eax=0x000306e4 ebx=0x31200800 ecx=0x7fbee3ff edx=0xbfebfbff
   0x00000002 0x00: eax=0x76036301 ebx=0x00f0b2ff ecx=0x00000000 edx=0x00ca0000
   0x00000003 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000004 0x00: eax=0x3c004121 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000
   0x00000004 0x01: eax=0x3c004122 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000
   0x00000004 0x02: eax=0x3c004143 ebx=0x01c0003f ecx=0x000001ff edx=0x00000000
   0x00000004 0x03: eax=0x3c07c163 ebx=0x04c0003f ecx=0x00004fff edx=0x00000006
   0x00000004 0x04: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000005 0x00: eax=0x00000040 ebx=0x00000040 ecx=0x00000003 edx=0x00001120
   0x00000006 0x00: eax=0x00000077 ebx=0x00000002 ecx=0x00000009 edx=0x00000000
   0x00000007 0x00: eax=0x00000000 ebx=0x00000281 ecx=0x00000000 edx=0x9c000400
   0x00000008 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000009 0x00: eax=0x00000001 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000a 0x00: eax=0x07300403 ebx=0x00000000 ecx=0x00000000 edx=0x00000603
   0x0000000b 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000031
   0x0000000b 0x01: eax=0x00000005 ebx=0x00000014 ecx=0x00000201 edx=0x00000031
   0x0000000b 0x02: eax=0x00000000 ebx=0x00000000 ecx=0x00000002 edx=0x00000031
   0x0000000c 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000d 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0x0000000d 0x01: eax=0x00000001 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000d 0x02: eax=0x00000100 ebx=0x00000240 ecx=0x00000000 edx=0x00000000
   0x20000000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0x80000000 0x00: eax=0x80000008 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000001 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000001 edx=0x2c100800
   0x80000002 0x00: eax=0x20202020 ebx=0x6e492020 ecx=0x286c6574 edx=0x58202952
   0x80000003 0x00: eax=0x286e6f65 ebx=0x43202952 ecx=0x45205550 edx=0x36322d35
   0x80000004 0x00: eax=0x76203039 ebx=0x20402032 ecx=0x30302e33 edx=0x007a4847
   0x80000005 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000006 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x01006040 edx=0x00000000
   0x80000007 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000100
   0x80000008 0x00: eax=0x0000302e ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80860000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0xc0000000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
CPU 36:
   0x00000000 0x00: eax=0x0000000d ebx=0x756e6547 ecx=0x6c65746e edx=0x49656e69
   0x00000001 0x00: eax=0x000306e4 ebx=0x33200800 ecx=0x7fbee3ff edx=0xbfebfbff
   0x00000002 0x00: eax=0x76036301 ebx=0x00f0b2ff ecx=0x00000000 edx=0x00ca0000
   0x00000003 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000004 0x00: eax=0x3c004121 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000
   0x00000004 0x01: eax=0x3c004122 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000
   0x00000004 0x02: eax=0x3c004143 ebx=0x01c0003f ecx=0x000001ff edx=0x00000000
   0x00000004 0x03: eax=0x3c07c163 ebx=0x04c0003f ecx=0x00004fff edx=0x00000006
   0x00000004 0x04: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000005 0x00: eax=0x00000040 ebx=0x00000040 ecx=0x00000003 edx=0x00001120
   0x00000006 0x00: eax=0x00000077 ebx=0x00000002 ecx=0x00000009 edx=0x00000000
   0x00000007 0x00: eax=0x00000000 ebx=0x00000281 ecx=0x00000000 edx=0x9c000400
   0x00000008 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000009 0x00: eax=0x00000001 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000a 0x00: eax=0x07300403 ebx=0x00000000 ecx=0x00000000 edx=0x00000603
   0x0000000b 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000033
   0x0000000b 0x01: eax=0x00000005 ebx=0x00000014 ecx=0x00000201 edx=0x00000033
   0x0000000b 0x02: eax=0x00000000 ebx=0x00000000 ecx=0x00000002 edx=0x00000033
   0x0000000c 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000d 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0x0000000d 0x01: eax=0x00000001 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000d 0x02: eax=0x00000100 ebx=0x00000240 ecx=0x00000000 edx=0x00000000
   0x20000000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0x80000000 0x00: eax=0x80000008 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000001 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000001 edx=0x2c100800
   0x80000002 0x00: eax=0x20202020 ebx=0x6e492020 ecx=0x286c6574 edx=0x58202952
   0x80000003 0x00: eax=0x286e6f65 ebx=0x43202952 ecx=0x45205550 edx=0x36322d35
   0x80000004 0x00: eax=0x76203039 ebx=0x20402032 ecx=0x30302e33 edx=0x007a4847
   0x80000005 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000006 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x01006040 edx=0x00000000
   0x80000007 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000100
   0x80000008 0x00: eax=0x0000302e ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80860000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0xc0000000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
CPU 37:
   0x00000000 0x00: eax=0x0000000d ebx=0x756e6547 ecx=0x6c65746e edx=0x49656e69
   0x00000001 0x00: eax=0x000306e4 ebx=0x35200800 ecx=0x7fbee3ff edx=0xbfebfbff
   0x00000002 0x00: eax=0x76036301 ebx=0x00f0b2ff ecx=0x00000000 edx=0x00ca0000
   0x00000003 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000004 0x00: eax=0x3c004121 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000
   0x00000004 0x01: eax=0x3c004122 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000
   0x00000004 0x02: eax=0x3c004143 ebx=0x01c0003f ecx=0x000001ff edx=0x00000000
   0x00000004 0x03: eax=0x3c07c163 ebx=0x04c0003f ecx=0x00004fff edx=0x00000006
   0x00000004 0x04: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000005 0x00: eax=0x00000040 ebx=0x00000040 ecx=0x00000003 edx=0x00001120
   0x00000006 0x00: eax=0x00000077 ebx=0x00000002 ecx=0x00000009 edx=0x00000000
   0x00000007 0x00: eax=0x00000000 ebx=0x00000281 ecx=0x00000000 edx=0x9c000400
   0x00000008 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000009 0x00: eax=0x00000001 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000a 0x00: eax=0x07300403 ebx=0x00000000 ecx=0x00000000 edx=0x00000603
   0x0000000b 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000035
   0x0000000b 0x01: eax=0x00000005 ebx=0x00000014 ecx=0x00000201 edx=0x00000035
   0x0000000b 0x02: eax=0x00000000 ebx=0x00000000 ecx=0x00000002 edx=0x00000035
   0x0000000c 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000d 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0x0000000d 0x01: eax=0x00000001 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000d 0x02: eax=0x00000100 ebx=0x00000240 ecx=0x00000000 edx=0x00000000
   0x20000000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0x80000000 0x00: eax=0x80000008 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000001 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000001 edx=0x2c100800
   0x80000002 0x00: eax=0x20202020 ebx=0x6e492020 ecx=0x286c6574 edx=0x58202952
   0x80000003 0x00: eax=0x286e6f65 ebx=0x43202952 ecx=0x45205550 edx=0x36322d35
   0x80000004 0x00: eax=0x76203039 ebx=0x20402032 ecx=0x30302e33 edx=0x007a4847
   0x80000005 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000006 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x01006040 edx=0x00000000
   0x80000007 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000100
   0x80000008 0x00: eax=0x0000302e ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80860000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0xc0000000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
CPU 38:
   0x00000000 0x00: eax=0x0000000d ebx=0x756e6547 ecx=0x6c65746e edx=0x49656e69
   0x00000001 0x00: eax=0x000306e4 ebx=0x37200800 ecx=0x7fbee3ff edx=0xbfebfbff
   0x00000002 0x00: eax=0x76036301 ebx=0x00f0b2ff ecx=0x00000000 edx=0x00ca0000
   0x00000003 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000004 0x00: eax=0x3c004121 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000
   0x00000004 0x01: eax=0x3c004122 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000
   0x00000004 0x02: eax=0x3c004143 ebx=0x01c0003f ecx=0x000001ff edx=0x00000000
   0x00000004 0x03: eax=0x3c07c163 ebx=0x04c0003f ecx=0x00004fff edx=0x00000006
   0x00000004 0x04: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000005 0x00: eax=0x00000040 ebx=0x00000040 ecx=0x00000003 edx=0x00001120
   0x00000006 0x00: eax=0x00000077 ebx=0x00000002 ecx=0x00000009 edx=0x00000000
   0x00000007 0x00: eax=0x00000000 ebx=0x00000281 ecx=0x00000000 edx=0x9c000400
   0x00000008 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000009 0x00: eax=0x00000001 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000a 0x00: eax=0x07300403 ebx=0x00000000 ecx=0x00000000 edx=0x00000603
   0x0000000b 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000037
   0x0000000b 0x01: eax=0x00000005 ebx=0x00000014 ecx=0x00000201 edx=0x00000037
   0x0000000b 0x02: eax=0x00000000 ebx=0x00000000 ecx=0x00000002 edx=0x00000037
   0x0000000c 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000d 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0x0000000d 0x01: eax=0x00000001 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000d 0x02: eax=0x00000100 ebx=0x00000240 ecx=0x00000000 edx=0x00000000
   0x20000000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0x80000000 0x00: eax=0x80000008 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000001 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000001 edx=0x2c100800
   0x80000002 0x00: eax=0x20202020 ebx=0x6e492020 ecx=0x286c6574 edx=0x58202952
   0x80000003 0x00: eax=0x286e6f65 ebx=0x43202952 ecx=0x45205550 edx=0x36322d35
   0x80000004 0x00: eax=0x76203039 ebx=0x20402032 ecx=0x30302e33 edx=0x007a4847
   0x80000005 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000006 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x01006040 edx=0x00000000
   0x80000007 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000100
   0x80000008 0x00: eax=0x0000302e ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80860000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0xc0000000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
CPU 39:
   0x00000000 0x00: eax=0x0000000d ebx=0x756e6547 ecx=0x6c65746e edx=0x49656e69
   0x00000001 0x00: eax=0x000306e4 ebx=0x39200800 ecx=0x7fbee3ff edx=0xbfebfbff
   0x00000002 0x00: eax=0x76036301 ebx=0x00f0b2ff ecx=0x00000000 edx=0x00ca0000
   0x00000003 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000004 0x00: eax=0x3c004121 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000
   0x00000004 0x01: eax=0x3c004122 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000
   0x00000004 0x02: eax=0x3c004143 ebx=0x01c0003f ecx=0x000001ff edx=0x00000000
   0x00000004 0x03: eax=0x3c07c163 ebx=0x04c0003f ecx=0x00004fff edx=0x00000006
   0x00000004 0x04: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000005 0x00: eax=0x00000040 ebx=0x00000040 ecx=0x00000003 edx=0x00001120
   0x00000006 0x00: eax=0x00000077 ebx=0x00000002 ecx=0x00000009 edx=0x00000000
   0x00000007 0x00: eax=0x00000000 ebx=0x00000281 ecx=0x00000000 edx=0x9c000400
   0x00000008 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000009 0x00: eax=0x00000001 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000a 0x00: eax=0x07300403 ebx=0x00000000 ecx=0x00000000 edx=0x00000603
   0x0000000b 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000039
   0x0000000b 0x01: eax=0x00000005 ebx=0x00000014 ecx=0x00000201 edx=0x00000039
   0x0000000b 0x02: eax=0x00000000 ebx=0x00000000 ecx=0x00000002 edx=0x00000039
   0x0000000c 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000d 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0x0000000d 0x01: eax=0x00000001 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000d 0x02: eax=0x00000100 ebx=0x00000240 ecx=0x00000000 edx=0x00000000
   0x20000000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0x80000000 0x00: eax=0x80000008 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000001 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000001 edx=0x2c100800
   0x80000002 0x00: eax=0x20202020 ebx=0x6e492020 ecx=0x286c6574 edx=0x58202952
   0x80000003 0x00: eax=0x286e6f65 ebx=0x43202952 ecx=0x45205550 edx=0x36322d35
   0x80000004 0x00: eax=0x76203039 ebx=0x20402032 ecx=0x30302e33 edx=0x007a4847
   0x80000005 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000006 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x01006040 edx=0x00000000
   0x80000007 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000100
   0x80000008 0x00: eax=0x0000302e ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80860000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0xc0000000 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000

[-- Attachment #3: apic-ids.txt --]
[-- Type: text/plain, Size: 7088 bytes --]

[    0.011712] CPU topo: Register 000 1
[    0.187210] CPU topo: Register 000 1
[    0.187211] CPU topo: Register 002 1
[    0.187212] CPU topo: Register 004 1
[    0.187213] CPU topo: Register 006 1
[    0.187214] CPU topo: Register 008 1
[    0.187215] CPU topo: Register 010 1
[    0.187216] CPU topo: Register 012 1
[    0.187217] CPU topo: Register 014 1
[    0.187218] CPU topo: Register 016 1
[    0.187219] CPU topo: Register 018 1
[    0.187219] CPU topo: Register 020 1
[    0.187220] CPU topo: Register 022 1
[    0.187221] CPU topo: Register 024 1
[    0.187222] CPU topo: Register 026 1
[    0.187223] CPU topo: Register 028 1
[    0.187223] CPU topo: Register 030 1
[    0.187224] CPU topo: Register 032 1
[    0.187225] CPU topo: Register 034 1
[    0.187226] CPU topo: Register 036 1
[    0.187227] CPU topo: Register 038 1
[    0.187228] CPU topo: Register 001 1
[    0.187228] CPU topo: Register 003 1
[    0.187229] CPU topo: Register 005 1
[    0.187230] CPU topo: Register 007 1
[    0.187230] CPU topo: Register 009 1
[    0.187231] CPU topo: Register 011 1
[    0.187232] CPU topo: Register 013 1
[    0.187233] CPU topo: Register 015 1
[    0.187233] CPU topo: Register 017 1
[    0.187234] CPU topo: Register 019 1
[    0.187235] CPU topo: Register 021 1
[    0.187235] CPU topo: Register 023 1
[    0.187236] CPU topo: Register 025 1
[    0.187237] CPU topo: Register 027 1
[    0.187238] CPU topo: Register 029 1
[    0.187238] CPU topo: Register 031 1
[    0.187239] CPU topo: Register 033 1
[    0.187240] CPU topo: Register 035 1
[    0.187241] CPU topo: Register 037 1
[    0.187241] CPU topo: Register 039 1
[    0.187244] CPU topo: Register 000 0
[    0.187244] CPU topo: Register 001 0
[    0.187245] CPU topo: Register 002 0
[    0.187246] CPU topo: Register 003 0
[    0.187246] CPU topo: Register 004 0
[    0.187247] CPU topo: Register 005 0
[    0.187248] CPU topo: Register 006 0
[    0.187249] CPU topo: Register 007 0
[    0.187249] CPU topo: Register 008 0
[    0.187250] CPU topo: Register 009 0
[    0.187251] CPU topo: Register 00a 0
[    0.187252] CPU topo: Register 00b 0
[    0.187252] CPU topo: Register 00c 0
[    0.187253] CPU topo: Register 00d 0
[    0.187254] CPU topo: Register 00e 0
[    0.187254] CPU topo: Register 00f 0
[    0.187255] CPU topo: Register 010 0
[    0.187256] CPU topo: Register 011 0
[    0.187257] CPU topo: Register 012 0
[    0.187257] CPU topo: Register 013 0
[    0.187258] CPU topo: Register 014 0
[    0.187259] CPU topo: Register 015 0
[    0.187260] CPU topo: Register 016 0
[    0.187261] CPU topo: Register 017 0
[    0.187262] CPU topo: Register 018 0
[    0.187262] CPU topo: Register 019 0
[    0.187263] CPU topo: Register 01a 0
[    0.187264] CPU topo: Register 01b 0
[    0.187265] CPU topo: Register 01c 0
[    0.187265] CPU topo: Register 01d 0
[    0.187266] CPU topo: Register 01e 0
[    0.187267] CPU topo: Register 01f 0
[    0.187268] CPU topo: Register 020 0
[    0.187268] CPU topo: Register 021 0
[    0.187269] CPU topo: Register 022 0
[    0.187270] CPU topo: Register 023 0
[    0.187271] CPU topo: Register 024 0
[    0.187272] CPU topo: Register 025 0
[    0.187272] CPU topo: Register 026 0
[    0.187273] CPU topo: Register 027 0
[    0.187274] CPU topo: Register 028 0
[    0.187275] CPU topo: Register 029 0
[    0.187275] CPU topo: Register 02a 0
[    0.187276] CPU topo: Register 02b 0
[    0.187277] CPU topo: Register 02c 0
[    0.187278] CPU topo: Register 02d 0
[    0.187279] CPU topo: Register 02e 0
[    0.187279] CPU topo: Register 02f 0
[    0.187280] CPU topo: Register 030 0
[    0.187281] CPU topo: Register 031 0
[    0.187282] CPU topo: Register 032 0
[    0.187282] CPU topo: Register 033 0
[    0.187283] CPU topo: Register 034 0
[    0.187284] CPU topo: Register 035 0
[    0.187285] CPU topo: Register 036 0
[    0.187285] CPU topo: Register 037 0
[    0.187286] CPU topo: Register 038 0
[    0.187287] CPU topo: Register 039 0
[    0.187287] CPU topo: Register 03a 0
[    0.187288] CPU topo: Register 03b 0
[    0.187289] CPU topo: Register 03c 0
[    0.187290] CPU topo: Register 03d 0
[    0.187290] CPU topo: Register 03e 0
[    0.187291] CPU topo: Register 03f 0
[    0.187292] CPU topo: Register 040 0
[    0.187293] CPU topo: Register 041 0
[    0.187294] CPU topo: Register 042 0
[    0.187294] CPU topo: Register 043 0
[    0.187295] CPU topo: Register 044 0
[    0.187296] CPU topo: Register 045 0
[    0.187297] CPU topo: Register 046 0
[    0.187298] CPU topo: Register 047 0
[    0.187299] CPU topo: Register 048 0
[    0.187300] CPU topo: Register 049 0
[    0.187300] CPU topo: Register 04a 0
[    0.187301] CPU topo: Register 04b 0
[    0.187302] CPU topo: Register 04c 0
[    0.187303] CPU topo: Register 04d 0
[    0.187303] CPU topo: Register 04e 0
[    0.187304] CPU topo: Register 04f 0
[    0.187305] CPU topo: Register 050 0
[    0.187306] CPU topo: Register 051 0
[    0.187306] CPU topo: Register 052 0
[    0.187307] CPU topo: Register 053 0
[    0.187308] CPU topo: Register 054 0
[    0.187309] CPU topo: Register 055 0
[    0.187310] CPU topo: Register 056 0
[    0.187310] CPU topo: Register 057 0
[    0.187311] CPU topo: Register 058 0
[    0.187312] CPU topo: Register 059 0
[    0.187313] CPU topo: Register 05a 0
[    0.187313] CPU topo: Register 05b 0
[    0.187314] CPU topo: Register 05c 0
[    0.187315] CPU topo: Register 05d 0
[    0.187316] CPU topo: Register 05e 0
[    0.187316] CPU topo: Register 05f 0
[    0.187317] CPU topo: Register 060 0
[    0.187318] CPU topo: Register 061 0
[    0.187319] CPU topo: Register 062 0
[    0.187319] CPU topo: Register 063 0
[    0.187320] CPU topo: Register 064 0
[    0.187321] CPU topo: Register 065 0
[    0.187322] CPU topo: Register 066 0
[    0.187322] CPU topo: Register 067 0
[    0.187323] CPU topo: Register 068 0
[    0.187324] CPU topo: Register 069 0
[    0.187325] CPU topo: Register 06a 0
[    0.187325] CPU topo: Register 06b 0
[    0.187326] CPU topo: Register 06c 0
[    0.187327] CPU topo: Register 06d 0
[    0.187327] CPU topo: Register 06e 0
[    0.187328] CPU topo: Register 06f 0
[    0.187329] CPU topo: Register 070 0
[    0.187330] CPU topo: Register 071 0
[    0.187331] CPU topo: Register 072 0
[    0.187332] CPU topo: Register 073 0
[    0.187332] CPU topo: Register 074 0
[    0.187333] CPU topo: Register 075 0
[    0.187334] CPU topo: Register 076 0
[    0.187335] CPU topo: Register 077 0
[    0.187412] CPU topo: Max. logical packages:   4
[    0.187413] CPU topo: Max. logical dies:       4
[    0.187413] CPU topo: Max. dies per package:   1
[    0.187418] CPU topo: Max. threads per core:   2
[    0.187420] CPU topo: Num. cores per package:    16
[    0.187421] CPU topo: Num. threads per package:  32
[    0.187421] CPU topo: Allowing 40 present CPUs plus 120 hotplug CPUs
[    0.187425] CPU topo: Thread    :   120
[    0.187428] CPU topo: Core      :    60
[    0.187431] CPU topo: Module    :     4
[    0.187434] CPU topo: Tile      :     4
[    0.187437] CPU topo: Die       :     4
[    0.187440] CPU topo: Package   :     4

[-- Attachment #4: dmesg.txt --]
[-- Type: text/plain, Size: 103780 bytes --]

[    0.000000] microcode: updated early: 0x424 -> 0x42e, date = 2019-03-14
[    0.000000] Linux version 6.5.0-rc4+ (sohil@sohilbuildbox) (gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0, GNU ld (GNU Binutils for Ubuntu) 2.38) #15 SMP PREEMPT_DYNAMIC Tue Aug  8 12:40:30 PDT 2023
[    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-6.5.0-rc4+ root=UUID=b0358c51-91a1-4785-90fe-f46828ae9f39 ro
[    0.000000] KERNEL supported cpus:
[    0.000000]   Intel GenuineIntel
[    0.000000]   AMD AuthenticAMD
[    0.000000]   Hygon HygonGenuine
[    0.000000]   Centaur CentaurHauls
[    0.000000]   zhaoxin   Shanghai  
[    0.000000] BIOS-provided physical RAM map:
[    0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009ffff] usable
[    0.000000] BIOS-e820: [mem 0x00000000000a0000-0x00000000000fffff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bad47fff] usable
[    0.000000] BIOS-e820: [mem 0x00000000bad48000-0x00000000baf8efff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000baf8f000-0x00000000bafc4fff] usable
[    0.000000] BIOS-e820: [mem 0x00000000bafc5000-0x00000000bafd9fff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000bafda000-0x00000000bb3d3fff] usable
[    0.000000] BIOS-e820: [mem 0x00000000bb3d4000-0x00000000bdd2efff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000bdd2f000-0x00000000bddccfff] ACPI NVS
[    0.000000] BIOS-e820: [mem 0x00000000bddcd000-0x00000000bdea0fff] ACPI data
[    0.000000] BIOS-e820: [mem 0x00000000bdea1000-0x00000000bdf2efff] ACPI NVS
[    0.000000] BIOS-e820: [mem 0x00000000bdf2f000-0x00000000bdfabfff] ACPI data
[    0.000000] BIOS-e820: [mem 0x00000000bdfac000-0x00000000bdffffff] usable
[    0.000000] BIOS-e820: [mem 0x00000000be000000-0x00000000cfffffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000fec00000-0x00000000fec00fff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000fed19000-0x00000000fed19fff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed1ffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000fee00000-0x00000000fee00fff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000ffa20000-0x00000000ffffffff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000103fffffff] usable
[    0.000000] NX (Execute Disable) protection: active
[    0.000000] APIC: Static calls initialized
[    0.000000] e820: update [mem 0xb6b86018-0xb6b96057] usable ==> usable
[    0.000000] e820: update [mem 0xb6b86018-0xb6b96057] usable ==> usable
[    0.000000] e820: update [mem 0xb6944018-0xb695e857] usable ==> usable
[    0.000000] e820: update [mem 0xb6944018-0xb695e857] usable ==> usable
[    0.000000] e820: update [mem 0xb6932018-0xb6943057] usable ==> usable
[    0.000000] e820: update [mem 0xb6932018-0xb6943057] usable ==> usable
[    0.000000] e820: update [mem 0xb6572018-0xb6583057] usable ==> usable
[    0.000000] e820: update [mem 0xb6572018-0xb6583057] usable ==> usable
[    0.000000] e820: update [mem 0xb6cfa018-0xb6d02057] usable ==> usable
[    0.000000] e820: update [mem 0xb6cfa018-0xb6d02057] usable ==> usable
[    0.000000] extended physical RAM map:
[    0.000000] reserve setup_data: [mem 0x0000000000000000-0x000000000009ffff] usable
[    0.000000] reserve setup_data: [mem 0x00000000000a0000-0x00000000000fffff] reserved
[    0.000000] reserve setup_data: [mem 0x0000000000100000-0x00000000b6572017] usable
[    0.000000] reserve setup_data: [mem 0x00000000b6572018-0x00000000b6583057] usable
[    0.000000] reserve setup_data: [mem 0x00000000b6583058-0x00000000b6932017] usable
[    0.000000] reserve setup_data: [mem 0x00000000b6932018-0x00000000b6943057] usable
[    0.000000] reserve setup_data: [mem 0x00000000b6943058-0x00000000b6944017] usable
[    0.000000] reserve setup_data: [mem 0x00000000b6944018-0x00000000b695e857] usable
[    0.000000] reserve setup_data: [mem 0x00000000b695e858-0x00000000b6b86017] usable
[    0.000000] reserve setup_data: [mem 0x00000000b6b86018-0x00000000b6b96057] usable
[    0.000000] reserve setup_data: [mem 0x00000000b6b96058-0x00000000b6cfa017] usable
[    0.000000] reserve setup_data: [mem 0x00000000b6cfa018-0x00000000b6d02057] usable
[    0.000000] reserve setup_data: [mem 0x00000000b6d02058-0x00000000bad47fff] usable
[    0.000000] reserve setup_data: [mem 0x00000000bad48000-0x00000000baf8efff] reserved
[    0.000000] reserve setup_data: [mem 0x00000000baf8f000-0x00000000bafc4fff] usable
[    0.000000] reserve setup_data: [mem 0x00000000bafc5000-0x00000000bafd9fff] reserved
[    0.000000] reserve setup_data: [mem 0x00000000bafda000-0x00000000bb3d3fff] usable
[    0.000000] reserve setup_data: [mem 0x00000000bb3d4000-0x00000000bdd2efff] reserved
[    0.000000] reserve setup_data: [mem 0x00000000bdd2f000-0x00000000bddccfff] ACPI NVS
[    0.000000] reserve setup_data: [mem 0x00000000bddcd000-0x00000000bdea0fff] ACPI data
[    0.000000] reserve setup_data: [mem 0x00000000bdea1000-0x00000000bdf2efff] ACPI NVS
[    0.000000] reserve setup_data: [mem 0x00000000bdf2f000-0x00000000bdfabfff] ACPI data
[    0.000000] reserve setup_data: [mem 0x00000000bdfac000-0x00000000bdffffff] usable
[    0.000000] reserve setup_data: [mem 0x00000000be000000-0x00000000cfffffff] reserved
[    0.000000] reserve setup_data: [mem 0x00000000fec00000-0x00000000fec00fff] reserved
[    0.000000] reserve setup_data: [mem 0x00000000fed19000-0x00000000fed19fff] reserved
[    0.000000] reserve setup_data: [mem 0x00000000fed1c000-0x00000000fed1ffff] reserved
[    0.000000] reserve setup_data: [mem 0x00000000fee00000-0x00000000fee00fff] reserved
[    0.000000] reserve setup_data: [mem 0x00000000ffa20000-0x00000000ffffffff] reserved
[    0.000000] reserve setup_data: [mem 0x0000000100000000-0x000000103fffffff] usable
[    0.000000] efi: EFI v2.3.1 by American Megatrends
                                            BIOS ID: SE5C600.86B.02.03.0003.201404191333
[    0.000000] efi: ACPI 2.0=0xbdfabf98 SMBIOS=0xf0440 
[    0.000000] efi: Remove mem89: MMIO range=[0xc0000000-0xcfffffff] (256MB) from e820 map
[    0.000000] e820: remove [mem 0xc0000000-0xcfffffff] reserved
[    0.000000] efi: Not removing mem90: MMIO range=[0xfec00000-0xfec00fff] (4KB) from e820 map
[    0.000000] efi: Not removing mem91: MMIO range=[0xfed19000-0xfed19fff] (4KB) from e820 map
[    0.000000] efi: Not removing mem92: MMIO range=[0xfed1c000-0xfed1ffff] (16KB) from e820 map
[    0.000000] efi: Not removing mem93: MMIO range=[0xfee00000-0xfee00fff] (4KB) from e820 map
[    0.000000] efi: Remove mem94: MMIO range=[0xffa20000-0xffffffff] (5MB) from e820 map
[    0.000000] e820: remove [mem 0xffa20000-0xffffffff] reserved
[    0.000000] SMBIOS 2.6 present.
[    0.000000] DMI: Intel Corporation W2600CR/W2600CR, BIOS SE5C600.86B.02.03.0003.041920141333 04/19/2014
[    0.000000] tsc: Fast TSC calibration using PIT
[    0.000000] tsc: Detected 2992.545 MHz processor
[    0.001254] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved
[    0.001257] e820: remove [mem 0x000a0000-0x000fffff] usable
[    0.001270] last_pfn = 0x1040000 max_arch_pfn = 0x400000000
[    0.001279] MTRR map: 7 entries (4 fixed + 3 variable; max 24), built from 10 variable MTRRs
[    0.001281] x86/PAT: Configuration [0-7]: WB  WC  UC- UC  WB  WP  UC- WT  
[    0.001736] last_pfn = 0xbe000 max_arch_pfn = 0x400000000
[    0.011188] found SMP MP-table at [mem 0x000fd3e0-0x000fd3ef]
[    0.011214] Using GB pages for direct mapping
[    0.011572] Secure boot disabled
[    0.011573] RAMDISK: [mem 0xb41f5000-0xb5919fff]
[    0.011579] ACPI: Early table checksum verification disabled
[    0.011584] ACPI: RSDP 0x00000000BDFABF98 000024 (v02 INTEL )
[    0.011588] ACPI: XSDT 0x00000000BDFA9D98 0000C4 (v01 INTEL  W2600CR  06222004 INTL 20090903)
[    0.011595] ACPI: FACP 0x00000000BDFA9918 0000F4 (v04 INTEL  W2600CR  06222004 INTL 20090903)
[    0.011600] ACPI BIOS Warning (bug): Invalid length for FADT/Pm1aControlBlock: 32, using default 16 (20230331/tbfadt-669)
[    0.011604] ACPI: DSDT 0x00000000BDF8F018 0183D9 (v02 INTEL  W2600CR  00000003 INTL 20100331)
[    0.011608] ACPI: FACS 0x00000000BDFA9F40 000040
[    0.011611] ACPI: APIC 0x00000000BDF8B018 000BAA (v03 INTEL  W2600CR  06222004 INTL 20090903)
[    0.011615] ACPI: SPMI 0x00000000BDFABA98 000041 (v05 INTEL  W2600CR  06222004 INTL 20090903)
[    0.011618] ACPI: FPDT 0x00000000BDFABA18 000044 (v01 INTEL  W2600CR  00000000      00000000)
[    0.011622] ACPI: MCFG 0x00000000BDFABF18 00003C (v01 INTEL  W2600CR  06222004 INTL 20090903)
[    0.011625] ACPI: WDDT 0x00000000BDFABE98 000040 (v01 INTEL  W2600CR  06222004 INTL 20090903)
[    0.011628] ACPI: SRAT 0x00000000BDF8CA98 000328 (v03 INTEL  W2600CR  06222004 INTL 20090903)
[    0.011632] ACPI: SLIT 0x00000000BDFABE18 000030 (v01 INTEL  W2600CR  06222004 INTL 20090903)
[    0.011635] ACPI: MSCT 0x00000000BDFAAD18 000090 (v01 INTEL  W2600CR  06222004 INTL 20090903)
[    0.011639] ACPI: HPET 0x00000000BDFABD98 000038 (v01 INTEL  W2600CR  06222004 INTL 20090903)
[    0.011642] ACPI: SSDT 0x00000000BDFABC98 00002B (v02 INTEL  W2600CR  00001000 INTL 20100331)
[    0.011646] ACPI: SSDT 0x00000000BDDCD018 0D30C8 (v02 INTEL  W2600CR  00004000 INTL 20100331)
[    0.011649] ACPI: BGRT 0x00000000BDFABD18 000038 (v00 INTEL  W2600CR  06222004 AMI  00010013)
[    0.011653] ACPI: HEST 0x00000000BDF8EF18 0000A8 (v01 INTEL  W2600CR  00000001 INTL 00000001)
[    0.011656] ACPI: BERT 0x00000000BDFABB98 000030 (v01 INTEL  W2600CR  00000001 INTL 00000001)
[    0.011659] ACPI: ERST 0x00000000BDF8EC98 000230 (v01 INTEL  W2600CR  00000001 INTL 00000001)
[    0.011663] ACPI: EINJ 0x00000000BDFA9798 000130 (v01 INTEL  W2600CR  00000001 INTL 00000001)
[    0.011666] ACPI: SSDT 0x00000000BDF86018 001729 (v02 INTEL  W2600CR  00000002 INTL 20100331)
[    0.011670] ACPI: SSDT 0x00000000BDFABC18 000045 (v02 INTEL  W2600CR  00000001 INTL 20100331)
[    0.011673] ACPI: SSDT 0x00000000BDF8CE18 000181 (v02 INTEL  W2600CR  00000003 INTL 20100331)
[    0.011676] ACPI: Reserving FACP table memory at [mem 0xbdfa9918-0xbdfa9a0b]
[    0.011677] ACPI: Reserving DSDT table memory at [mem 0xbdf8f018-0xbdfa73f0]
[    0.011678] ACPI: Reserving FACS table memory at [mem 0xbdfa9f40-0xbdfa9f7f]
[    0.011679] ACPI: Reserving APIC table memory at [mem 0xbdf8b018-0xbdf8bbc1]
[    0.011680] ACPI: Reserving SPMI table memory at [mem 0xbdfaba98-0xbdfabad8]
[    0.011681] ACPI: Reserving FPDT table memory at [mem 0xbdfaba18-0xbdfaba5b]
[    0.011682] ACPI: Reserving MCFG table memory at [mem 0xbdfabf18-0xbdfabf53]
[    0.011683] ACPI: Reserving WDDT table memory at [mem 0xbdfabe98-0xbdfabed7]
[    0.011684] ACPI: Reserving SRAT table memory at [mem 0xbdf8ca98-0xbdf8cdbf]
[    0.011685] ACPI: Reserving SLIT table memory at [mem 0xbdfabe18-0xbdfabe47]
[    0.011686] ACPI: Reserving MSCT table memory at [mem 0xbdfaad18-0xbdfaada7]
[    0.011687] ACPI: Reserving HPET table memory at [mem 0xbdfabd98-0xbdfabdcf]
[    0.011688] ACPI: Reserving SSDT table memory at [mem 0xbdfabc98-0xbdfabcc2]
[    0.011689] ACPI: Reserving SSDT table memory at [mem 0xbddcd018-0xbdea00df]
[    0.011689] ACPI: Reserving BGRT table memory at [mem 0xbdfabd18-0xbdfabd4f]
[    0.011690] ACPI: Reserving HEST table memory at [mem 0xbdf8ef18-0xbdf8efbf]
[    0.011691] ACPI: Reserving BERT table memory at [mem 0xbdfabb98-0xbdfabbc7]
[    0.011692] ACPI: Reserving ERST table memory at [mem 0xbdf8ec98-0xbdf8eec7]
[    0.011693] ACPI: Reserving EINJ table memory at [mem 0xbdfa9798-0xbdfa98c7]
[    0.011694] ACPI: Reserving SSDT table memory at [mem 0xbdf86018-0xbdf87740]
[    0.011695] ACPI: Reserving SSDT table memory at [mem 0xbdfabc18-0xbdfabc5c]
[    0.011696] ACPI: Reserving SSDT table memory at [mem 0xbdf8ce18-0xbdf8cf98]
[    0.011712] CPU topo: Register 000 1
[    0.011745] SRAT: PXM 0 -> APIC 0x00 -> Node 0
[    0.011746] SRAT: PXM 0 -> APIC 0x01 -> Node 0
[    0.011747] SRAT: PXM 0 -> APIC 0x02 -> Node 0
[    0.011748] SRAT: PXM 0 -> APIC 0x03 -> Node 0
[    0.011748] SRAT: PXM 0 -> APIC 0x04 -> Node 0
[    0.011749] SRAT: PXM 0 -> APIC 0x05 -> Node 0
[    0.011750] SRAT: PXM 0 -> APIC 0x06 -> Node 0
[    0.011751] SRAT: PXM 0 -> APIC 0x07 -> Node 0
[    0.011751] SRAT: PXM 0 -> APIC 0x08 -> Node 0
[    0.011752] SRAT: PXM 0 -> APIC 0x09 -> Node 0
[    0.011753] SRAT: PXM 0 -> APIC 0x10 -> Node 0
[    0.011754] SRAT: PXM 0 -> APIC 0x11 -> Node 0
[    0.011754] SRAT: PXM 0 -> APIC 0x12 -> Node 0
[    0.011755] SRAT: PXM 0 -> APIC 0x13 -> Node 0
[    0.011756] SRAT: PXM 0 -> APIC 0x14 -> Node 0
[    0.011757] SRAT: PXM 0 -> APIC 0x15 -> Node 0
[    0.011757] SRAT: PXM 0 -> APIC 0x16 -> Node 0
[    0.011758] SRAT: PXM 0 -> APIC 0x17 -> Node 0
[    0.011759] SRAT: PXM 0 -> APIC 0x18 -> Node 0
[    0.011760] SRAT: PXM 0 -> APIC 0x19 -> Node 0
[    0.011761] SRAT: PXM 1 -> APIC 0x20 -> Node 1
[    0.011761] SRAT: PXM 1 -> APIC 0x21 -> Node 1
[    0.011762] SRAT: PXM 1 -> APIC 0x22 -> Node 1
[    0.011763] SRAT: PXM 1 -> APIC 0x23 -> Node 1
[    0.011764] SRAT: PXM 1 -> APIC 0x24 -> Node 1
[    0.011764] SRAT: PXM 1 -> APIC 0x25 -> Node 1
[    0.011765] SRAT: PXM 1 -> APIC 0x26 -> Node 1
[    0.011766] SRAT: PXM 1 -> APIC 0x27 -> Node 1
[    0.011767] SRAT: PXM 1 -> APIC 0x28 -> Node 1
[    0.011767] SRAT: PXM 1 -> APIC 0x29 -> Node 1
[    0.011768] SRAT: PXM 1 -> APIC 0x30 -> Node 1
[    0.011769] SRAT: PXM 1 -> APIC 0x31 -> Node 1
[    0.011770] SRAT: PXM 1 -> APIC 0x32 -> Node 1
[    0.011770] SRAT: PXM 1 -> APIC 0x33 -> Node 1
[    0.011771] SRAT: PXM 1 -> APIC 0x34 -> Node 1
[    0.011772] SRAT: PXM 1 -> APIC 0x35 -> Node 1
[    0.011773] SRAT: PXM 1 -> APIC 0x36 -> Node 1
[    0.011773] SRAT: PXM 1 -> APIC 0x37 -> Node 1
[    0.011774] SRAT: PXM 1 -> APIC 0x38 -> Node 1
[    0.011775] SRAT: PXM 1 -> APIC 0x39 -> Node 1
[    0.011777] ACPI: SRAT: Node 0 PXM 0 [mem 0x00000000-0xbfffffff]
[    0.011779] ACPI: SRAT: Node 0 PXM 0 [mem 0x100000000-0x83fffffff]
[    0.011780] ACPI: SRAT: Node 1 PXM 1 [mem 0x840000000-0x103fffffff]
[    0.011786] NUMA: Initialized distance table, cnt=2
[    0.011789] NUMA: Node 0 [mem 0x00000000-0xbfffffff] + [mem 0x100000000-0x83fffffff] -> [mem 0x00000000-0x83fffffff]
[    0.011801] NODE_DATA(0) allocated [mem 0x83ffd5000-0x83fffffff]
[    0.011824] NODE_DATA(1) allocated [mem 0x103ffd4000-0x103fffefff]
[    0.012082] Zone ranges:
[    0.012083]   DMA      [mem 0x0000000000001000-0x0000000000ffffff]
[    0.012085]   DMA32    [mem 0x0000000001000000-0x00000000ffffffff]
[    0.012087]   Normal   [mem 0x0000000100000000-0x000000103fffffff]
[    0.012088]   Device   empty
[    0.012089] Movable zone start for each node
[    0.012092] Early memory node ranges
[    0.012092]   node   0: [mem 0x0000000000001000-0x000000000009ffff]
[    0.012094]   node   0: [mem 0x0000000000100000-0x00000000bad47fff]
[    0.012095]   node   0: [mem 0x00000000baf8f000-0x00000000bafc4fff]
[    0.012096]   node   0: [mem 0x00000000bafda000-0x00000000bb3d3fff]
[    0.012097]   node   0: [mem 0x00000000bdfac000-0x00000000bdffffff]
[    0.012098]   node   0: [mem 0x0000000100000000-0x000000083fffffff]
[    0.012102]   node   1: [mem 0x0000000840000000-0x000000103fffffff]
[    0.012107] Initmem setup node 0 [mem 0x0000000000001000-0x000000083fffffff]
[    0.012110] Initmem setup node 1 [mem 0x0000000840000000-0x000000103fffffff]
[    0.012114] On node 0, zone DMA: 1 pages in unavailable ranges
[    0.012145] On node 0, zone DMA: 96 pages in unavailable ranges
[    0.018270] On node 0, zone DMA32: 583 pages in unavailable ranges
[    0.018285] On node 0, zone DMA32: 21 pages in unavailable ranges
[    0.018402] On node 0, zone DMA32: 11224 pages in unavailable ranges
[    0.080584] On node 0, zone Normal: 8192 pages in unavailable ranges
[    0.187197] ACPI: PM-Timer IO Port: 0x408
[    0.187210] CPU topo: Register 000 1
[    0.187211] CPU topo: Register 002 1
[    0.187212] CPU topo: Register 004 1
[    0.187213] CPU topo: Register 006 1
[    0.187214] CPU topo: Register 008 1
[    0.187215] CPU topo: Register 010 1
[    0.187216] CPU topo: Register 012 1
[    0.187217] CPU topo: Register 014 1
[    0.187218] CPU topo: Register 016 1
[    0.187219] CPU topo: Register 018 1
[    0.187219] CPU topo: Register 020 1
[    0.187220] CPU topo: Register 022 1
[    0.187221] CPU topo: Register 024 1
[    0.187222] CPU topo: Register 026 1
[    0.187223] CPU topo: Register 028 1
[    0.187223] CPU topo: Register 030 1
[    0.187224] CPU topo: Register 032 1
[    0.187225] CPU topo: Register 034 1
[    0.187226] CPU topo: Register 036 1
[    0.187227] CPU topo: Register 038 1
[    0.187228] CPU topo: Register 001 1
[    0.187228] CPU topo: Register 003 1
[    0.187229] CPU topo: Register 005 1
[    0.187230] CPU topo: Register 007 1
[    0.187230] CPU topo: Register 009 1
[    0.187231] CPU topo: Register 011 1
[    0.187232] CPU topo: Register 013 1
[    0.187233] CPU topo: Register 015 1
[    0.187233] CPU topo: Register 017 1
[    0.187234] CPU topo: Register 019 1
[    0.187235] CPU topo: Register 021 1
[    0.187235] CPU topo: Register 023 1
[    0.187236] CPU topo: Register 025 1
[    0.187237] CPU topo: Register 027 1
[    0.187238] CPU topo: Register 029 1
[    0.187238] CPU topo: Register 031 1
[    0.187239] CPU topo: Register 033 1
[    0.187240] CPU topo: Register 035 1
[    0.187241] CPU topo: Register 037 1
[    0.187241] CPU topo: Register 039 1
[    0.187244] CPU topo: Register 000 0
[    0.187244] CPU topo: Register 001 0
[    0.187245] CPU topo: Register 002 0
[    0.187246] CPU topo: Register 003 0
[    0.187246] CPU topo: Register 004 0
[    0.187247] CPU topo: Register 005 0
[    0.187248] CPU topo: Register 006 0
[    0.187249] CPU topo: Register 007 0
[    0.187249] CPU topo: Register 008 0
[    0.187250] CPU topo: Register 009 0
[    0.187251] CPU topo: Register 00a 0
[    0.187252] CPU topo: Register 00b 0
[    0.187252] CPU topo: Register 00c 0
[    0.187253] CPU topo: Register 00d 0
[    0.187254] CPU topo: Register 00e 0
[    0.187254] CPU topo: Register 00f 0
[    0.187255] CPU topo: Register 010 0
[    0.187256] CPU topo: Register 011 0
[    0.187257] CPU topo: Register 012 0
[    0.187257] CPU topo: Register 013 0
[    0.187258] CPU topo: Register 014 0
[    0.187259] CPU topo: Register 015 0
[    0.187260] CPU topo: Register 016 0
[    0.187261] CPU topo: Register 017 0
[    0.187262] CPU topo: Register 018 0
[    0.187262] CPU topo: Register 019 0
[    0.187263] CPU topo: Register 01a 0
[    0.187264] CPU topo: Register 01b 0
[    0.187265] CPU topo: Register 01c 0
[    0.187265] CPU topo: Register 01d 0
[    0.187266] CPU topo: Register 01e 0
[    0.187267] CPU topo: Register 01f 0
[    0.187268] CPU topo: Register 020 0
[    0.187268] CPU topo: Register 021 0
[    0.187269] CPU topo: Register 022 0
[    0.187270] CPU topo: Register 023 0
[    0.187271] CPU topo: Register 024 0
[    0.187272] CPU topo: Register 025 0
[    0.187272] CPU topo: Register 026 0
[    0.187273] CPU topo: Register 027 0
[    0.187274] CPU topo: Register 028 0
[    0.187275] CPU topo: Register 029 0
[    0.187275] CPU topo: Register 02a 0
[    0.187276] CPU topo: Register 02b 0
[    0.187277] CPU topo: Register 02c 0
[    0.187278] CPU topo: Register 02d 0
[    0.187279] CPU topo: Register 02e 0
[    0.187279] CPU topo: Register 02f 0
[    0.187280] CPU topo: Register 030 0
[    0.187281] CPU topo: Register 031 0
[    0.187282] CPU topo: Register 032 0
[    0.187282] CPU topo: Register 033 0
[    0.187283] CPU topo: Register 034 0
[    0.187284] CPU topo: Register 035 0
[    0.187285] CPU topo: Register 036 0
[    0.187285] CPU topo: Register 037 0
[    0.187286] CPU topo: Register 038 0
[    0.187287] CPU topo: Register 039 0
[    0.187287] CPU topo: Register 03a 0
[    0.187288] CPU topo: Register 03b 0
[    0.187289] CPU topo: Register 03c 0
[    0.187290] CPU topo: Register 03d 0
[    0.187290] CPU topo: Register 03e 0
[    0.187291] CPU topo: Register 03f 0
[    0.187292] CPU topo: Register 040 0
[    0.187293] CPU topo: Register 041 0
[    0.187294] CPU topo: Register 042 0
[    0.187294] CPU topo: Register 043 0
[    0.187295] CPU topo: Register 044 0
[    0.187296] CPU topo: Register 045 0
[    0.187297] CPU topo: Register 046 0
[    0.187298] CPU topo: Register 047 0
[    0.187299] CPU topo: Register 048 0
[    0.187300] CPU topo: Register 049 0
[    0.187300] CPU topo: Register 04a 0
[    0.187301] CPU topo: Register 04b 0
[    0.187302] CPU topo: Register 04c 0
[    0.187303] CPU topo: Register 04d 0
[    0.187303] CPU topo: Register 04e 0
[    0.187304] CPU topo: Register 04f 0
[    0.187305] CPU topo: Register 050 0
[    0.187306] CPU topo: Register 051 0
[    0.187306] CPU topo: Register 052 0
[    0.187307] CPU topo: Register 053 0
[    0.187308] CPU topo: Register 054 0
[    0.187309] CPU topo: Register 055 0
[    0.187310] CPU topo: Register 056 0
[    0.187310] CPU topo: Register 057 0
[    0.187311] CPU topo: Register 058 0
[    0.187312] CPU topo: Register 059 0
[    0.187313] CPU topo: Register 05a 0
[    0.187313] CPU topo: Register 05b 0
[    0.187314] CPU topo: Register 05c 0
[    0.187315] CPU topo: Register 05d 0
[    0.187316] CPU topo: Register 05e 0
[    0.187316] CPU topo: Register 05f 0
[    0.187317] CPU topo: Register 060 0
[    0.187318] CPU topo: Register 061 0
[    0.187319] CPU topo: Register 062 0
[    0.187319] CPU topo: Register 063 0
[    0.187320] CPU topo: Register 064 0
[    0.187321] CPU topo: Register 065 0
[    0.187322] CPU topo: Register 066 0
[    0.187322] CPU topo: Register 067 0
[    0.187323] CPU topo: Register 068 0
[    0.187324] CPU topo: Register 069 0
[    0.187325] CPU topo: Register 06a 0
[    0.187325] CPU topo: Register 06b 0
[    0.187326] CPU topo: Register 06c 0
[    0.187327] CPU topo: Register 06d 0
[    0.187327] CPU topo: Register 06e 0
[    0.187328] CPU topo: Register 06f 0
[    0.187329] CPU topo: Register 070 0
[    0.187330] CPU topo: Register 071 0
[    0.187331] CPU topo: Register 072 0
[    0.187332] CPU topo: Register 073 0
[    0.187332] CPU topo: Register 074 0
[    0.187333] CPU topo: Register 075 0
[    0.187334] CPU topo: Register 076 0
[    0.187335] CPU topo: Register 077 0
[    0.187340] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])
[    0.187352] IOAPIC[0]: apic_id 0, version 32, address 0xfec00000, GSI 0-23
[    0.187357] IOAPIC[1]: apic_id 1, version 32, address 0xfec3f000, GSI 24-47
[    0.187362] IOAPIC[2]: apic_id 2, version 32, address 0xfec7f000, GSI 48-71
[    0.187365] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
[    0.187367] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
[    0.187374] ACPI: Using ACPI (MADT) for SMP configuration information
[    0.187375] ACPI: HPET id: 0x8086a701 base: 0xfed00000
[    0.187388] e820: update [mem 0xb6d5f000-0xb6e40fff] usable ==> reserved
[    0.187406] TSC deadline timer available
[    0.187412] CPU topo: Max. logical packages:   4
[    0.187413] CPU topo: Max. logical dies:       4
[    0.187413] CPU topo: Max. dies per package:   1
[    0.187418] CPU topo: Max. threads per core:   2
[    0.187420] CPU topo: Num. cores per package:    16
[    0.187421] CPU topo: Num. threads per package:  32
[    0.187421] CPU topo: Allowing 40 present CPUs plus 120 hotplug CPUs
[    0.187425] CPU topo: Thread    :   120
[    0.187428] CPU topo: Core      :    60
[    0.187431] CPU topo: Module    :     4
[    0.187434] CPU topo: Tile      :     4
[    0.187437] CPU topo: Die       :     4
[    0.187440] CPU topo: Package   :     4
[    0.187467] PM: hibernation: Registered nosave memory: [mem 0x00000000-0x00000fff]
[    0.187469] PM: hibernation: Registered nosave memory: [mem 0x000a0000-0x000fffff]
[    0.187471] PM: hibernation: Registered nosave memory: [mem 0xb6572000-0xb6572fff]
[    0.187473] PM: hibernation: Registered nosave memory: [mem 0xb6583000-0xb6583fff]
[    0.187475] PM: hibernation: Registered nosave memory: [mem 0xb6932000-0xb6932fff]
[    0.187477] PM: hibernation: Registered nosave memory: [mem 0xb6943000-0xb6943fff]
[    0.187478] PM: hibernation: Registered nosave memory: [mem 0xb6944000-0xb6944fff]
[    0.187480] PM: hibernation: Registered nosave memory: [mem 0xb695e000-0xb695efff]
[    0.187482] PM: hibernation: Registered nosave memory: [mem 0xb6b86000-0xb6b86fff]
[    0.187484] PM: hibernation: Registered nosave memory: [mem 0xb6b96000-0xb6b96fff]
[    0.187486] PM: hibernation: Registered nosave memory: [mem 0xb6cfa000-0xb6cfafff]
[    0.187487] PM: hibernation: Registered nosave memory: [mem 0xb6d02000-0xb6d02fff]
[    0.187489] PM: hibernation: Registered nosave memory: [mem 0xb6d5f000-0xb6e40fff]
[    0.187491] PM: hibernation: Registered nosave memory: [mem 0xbad48000-0xbaf8efff]
[    0.187493] PM: hibernation: Registered nosave memory: [mem 0xbafc5000-0xbafd9fff]
[    0.187495] PM: hibernation: Registered nosave memory: [mem 0xbb3d4000-0xbdd2efff]
[    0.187496] PM: hibernation: Registered nosave memory: [mem 0xbdd2f000-0xbddccfff]
[    0.187497] PM: hibernation: Registered nosave memory: [mem 0xbddcd000-0xbdea0fff]
[    0.187498] PM: hibernation: Registered nosave memory: [mem 0xbdea1000-0xbdf2efff]
[    0.187498] PM: hibernation: Registered nosave memory: [mem 0xbdf2f000-0xbdfabfff]
[    0.187501] PM: hibernation: Registered nosave memory: [mem 0xbe000000-0xbfffffff]
[    0.187501] PM: hibernation: Registered nosave memory: [mem 0xc0000000-0xfebfffff]
[    0.187502] PM: hibernation: Registered nosave memory: [mem 0xfec00000-0xfec00fff]
[    0.187503] PM: hibernation: Registered nosave memory: [mem 0xfec01000-0xfed18fff]
[    0.187504] PM: hibernation: Registered nosave memory: [mem 0xfed19000-0xfed19fff]
[    0.187504] PM: hibernation: Registered nosave memory: [mem 0xfed1a000-0xfed1bfff]
[    0.187505] PM: hibernation: Registered nosave memory: [mem 0xfed1c000-0xfed1ffff]
[    0.187506] PM: hibernation: Registered nosave memory: [mem 0xfed20000-0xfedfffff]
[    0.187507] PM: hibernation: Registered nosave memory: [mem 0xfee00000-0xfee00fff]
[    0.187508] PM: hibernation: Registered nosave memory: [mem 0xfee01000-0xffffffff]
[    0.187509] [mem 0xc0000000-0xfebfffff] available for PCI devices
[    0.187511] Booting paravirtualized kernel on bare hardware
[    0.187514] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645519600211568 ns
[    0.187523] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:160 nr_cpu_ids:160 nr_node_ids:2
[    0.200389] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u262144
[    0.200406] pcpu-alloc: s221184 r8192 d28672 u262144 alloc=1*2097152
[    0.200409] pcpu-alloc: [0] 000 001 002 003 004 005 006 007 
[    0.200414] pcpu-alloc: [0] 008 009 020 021 022 023 024 025 
[    0.200420] pcpu-alloc: [0] 026 027 028 029 040 042 044 046 
[    0.200426] pcpu-alloc: [0] 048 050 052 054 056 058 060 062 
[    0.200431] pcpu-alloc: [0] 064 066 068 070 072 074 076 078 
[    0.200437] pcpu-alloc: [0] 080 082 084 086 088 090 092 094 
[    0.200442] pcpu-alloc: [0] 096 098 100 102 104 106 108 110 
[    0.200448] pcpu-alloc: [0] 112 114 116 118 120 122 124 126 
[    0.200453] pcpu-alloc: [0] 128 130 132 134 136 138 140 142 
[    0.200459] pcpu-alloc: [0] 144 146 148 150 152 154 156 158 
[    0.200464] pcpu-alloc: [1] 010 011 012 013 014 015 016 017 
[    0.200470] pcpu-alloc: [1] 018 019 030 031 032 033 034 035 
[    0.200475] pcpu-alloc: [1] 036 037 038 039 041 043 045 047 
[    0.200480] pcpu-alloc: [1] 049 051 053 055 057 059 061 063 
[    0.200486] pcpu-alloc: [1] 065 067 069 071 073 075 077 079 
[    0.200491] pcpu-alloc: [1] 081 083 085 087 089 091 093 095 
[    0.200497] pcpu-alloc: [1] 097 099 101 103 105 107 109 111 
[    0.200502] pcpu-alloc: [1] 113 115 117 119 121 123 125 127 
[    0.200507] pcpu-alloc: [1] 129 131 133 135 137 139 141 143 
[    0.200513] pcpu-alloc: [1] 145 147 149 151 153 155 157 159 
[    0.200559] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-6.5.0-rc4+ root=UUID=b0358c51-91a1-4785-90fe-f46828ae9f39 ro
[    0.200605] Unknown kernel command line parameters "BOOT_IMAGE=/boot/vmlinuz-6.5.0-rc4+", will be passed to user space.
[    0.200627] random: crng init done
[    0.200628] printk: log_buf_len individual max cpu contribution: 4096 bytes
[    0.200630] printk: log_buf_len total cpu_extra contributions: 651264 bytes
[    0.200630] printk: log_buf_len min size: 262144 bytes
[    0.201755] printk: log_buf_len: 1048576 bytes
[    0.201756] printk: early log buf free: 236624(90%)
[    0.205619] Fallback order for Node 0: 0 1 
[    0.205623] Fallback order for Node 1: 1 0 
[    0.205691] Built 2 zonelists, mobility grouping on.  Total pages: 16495108
[    0.205693] Policy zone: Normal
[    0.205698] mem auto-init: stack:off, heap alloc:on, heap free:off
[    0.205704] software IO TLB: area num 256.
[    0.432626] Memory: 65706720K/67028396K available (18432K kernel code, 4089K rwdata, 6880K rodata, 4576K init, 17800K bss, 1321416K reserved, 0K cma-reserved)
[    0.433781] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=160, Nodes=2
[    0.433850] Kernel/User page tables isolation: enabled
[    0.434063] ftrace: allocating 49600 entries in 194 pages
[    0.443370] ftrace: allocated 194 pages with 3 groups
[    0.444630] Dynamic Preempt: voluntary
[    0.445292] rcu: Preemptible hierarchical RCU implementation.
[    0.445293] rcu: 	RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=160.
[    0.445295] 	Trampoline variant of Tasks RCU enabled.
[    0.445296] 	Rude variant of Tasks RCU enabled.
[    0.445296] 	Tracing variant of Tasks RCU enabled.
[    0.445297] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.
[    0.445299] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=160
[    0.448566] NR_IRQS: 524544, nr_irqs: 2520, preallocated irqs: 16
[    0.448935] rcu: srcu_init: Setting srcu_struct sizes to big.
[    0.449528] Console: colour dummy device 80x25
[    0.449531] printk: console [tty0] enabled
[    0.450671] mempolicy: Enabling automatic NUMA balancing. Configure with numa_balancing= or the kernel.numa_balancing sysctl
[    0.450681] ACPI: Core revision 20230331
[    0.451253] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 133484882848 ns
[    0.451270] APIC: Switch to symmetric I/O mode setup
[    0.451396] x2apic: IRQ remapping doesn't support X2APIC mode
[    0.451476] APIC: Switched APIC routing to: physical flat
[    0.452085] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
[    0.471296] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x2b22c2cc76b, max_idle_ns: 440795213268 ns
[    0.471304] Calibrating delay loop (skipped), value calculated using timer frequency.. 5985.09 BogoMIPS (lpj=11970180)
[    0.471333] CPU0: Thermal monitoring enabled (TM1)
[    0.471405] process: using mwait in idle threads
[    0.471410] Last level iTLB entries: 4KB 512, 2MB 8, 4MB 8
[    0.471413] Last level dTLB entries: 4KB 512, 2MB 0, 4MB 0, 1GB 4
[    0.471418] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization
[    0.471423] Spectre V2 : Mitigation: Retpolines
[    0.471425] Spectre V2 : Spectre v2 / SpectreRSB mitigation: Filling RSB on context switch
[    0.471428] Spectre V2 : Spectre v2 / SpectreRSB : Filling RSB on VMEXIT
[    0.471431] Spectre V2 : Enabling Restricted Speculation for firmware calls
[    0.471435] Spectre V2 : mitigation: Enabling conditional Indirect Branch Prediction Barrier
[    0.471439] Spectre V2 : User space: Mitigation: STIBP via prctl
[    0.471442] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl
[    0.471448] MDS: Mitigation: Clear CPU buffers
[    0.471450] MMIO Stale Data: Unknown: No mitigations
[    0.471457] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'
[    0.471461] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'
[    0.471463] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'
[    0.471467] x86/fpu: xstate_offset[2]:  576, xstate_sizes[2]:  256
[    0.471470] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format.
[    0.491566] Freeing SMP alternatives memory: 40K
[    0.491571] pid_max: default: 163840 minimum: 1280
[    0.499421] LSM: initializing lsm=lockdown,capability,landlock,yama,integrity
[    0.499489] landlock: Up and running.
[    0.499491] Yama: becoming mindful.
[    0.519530] Dentry cache hash table entries: 8388608 (order: 14, 67108864 bytes, vmalloc hugepage)
[    0.529470] Inode-cache hash table entries: 4194304 (order: 13, 33554432 bytes, vmalloc hugepage)
[    0.529912] Mount-cache hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc)
[    0.530236] Mountpoint-cache hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc)
[    0.531559] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2690 v2 @ 3.00GHz (family: 0x6, model: 0x3e, stepping: 0x4)
[    0.531954] RCU Tasks: Setting shift to 8 and lim to 1 rcu_task_cb_adjust=1.
[    0.532027] RCU Tasks Rude: Setting shift to 8 and lim to 1 rcu_task_cb_adjust=1.
[    0.532083] RCU Tasks Trace: Setting shift to 8 and lim to 1 rcu_task_cb_adjust=1.
[    0.532108] Performance Events: PEBS fmt1+, IvyBridge events, 16-deep LBR, full-width counters, Intel PMU driver.
[    0.532135] ... version:                3
[    0.532137] ... bit width:              48
[    0.532139] ... generic registers:      4
[    0.532141] ... value mask:             0000ffffffffffff
[    0.532144] ... max period:             00007fffffffffff
[    0.532146] ... fixed-purpose events:   3
[    0.532148] ... event mask:             000000070000000f
[    0.532369] signal: max sigframe size: 1776
[    0.532392] Estimated ratio of average max frequency by base frequency (times 1024): 1126
[    0.532447] rcu: Hierarchical SRCU implementation.
[    0.532451] rcu: 	Max phase no-delay instances is 1000.
[    0.546201] NMI watchdog: Enabled. Permanently consumes one hw-PMU counter.
[    0.548576] smp: Bringing up secondary CPUs ...
[    0.548775] smpboot: x86: Booting SMP configuration:
[    0.548778] .... node  #0, CPUs:          #1   #2   #3   #4   #5   #6   #7   #8   #9
[    0.552113] .... node  #1, CPUs:    #10  #11  #12  #13  #14  #15  #16  #17  #18  #19
[    0.593307] .... node  #0, CPUs:    #20  #21  #22  #23  #24  #25  #26  #27  #28  #29
[    0.599211] .... node  #1, CPUs:    #30  #31  #32  #33  #34  #35  #36  #37  #38  #39
[    0.603697] MDS CPU bug present and SMT on, data leak possible. See https://www.kernel.org/doc/html/latest/admin-guide/hw-vuln/mds.html for more details.
[    0.641308] smp: Brought up 2 nodes, 40 CPUs
[    0.641308] smpboot: Total of 40 processors activated (239403.60 BogoMIPS)
[    0.647646] devtmpfs: initialized
[    0.647646] x86/mm: Memory block size: 2048MB
[    0.650495] ACPI: PM: Registering ACPI NVS region [mem 0xbdd2f000-0xbddccfff] (647168 bytes)
[    0.650495] ACPI: PM: Registering ACPI NVS region [mem 0xbdea1000-0xbdf2efff] (581632 bytes)
[    0.651385] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
[    0.652651] futex hash table entries: 65536 (order: 10, 4194304 bytes, vmalloc hugepage)
[    0.653227] pinctrl core: initialized pinctrl subsystem
[    0.653348] PM: RTC time: 19:42:19, date: 2023-08-08
[    0.655468] NET: Registered PF_NETLINK/PF_ROUTE protocol family
[    0.656538] DMA: preallocated 4096 KiB GFP_KERNEL pool for atomic allocations
[    0.657457] DMA: preallocated 4096 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations
[    0.658370] DMA: preallocated 4096 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations
[    0.658384] audit: initializing netlink subsys (disabled)
[    0.658398] audit: type=2000 audit(1691523739.204:1): state=initialized audit_enabled=0 res=1
[    0.658398] thermal_sys: Registered thermal governor 'fair_share'
[    0.658398] thermal_sys: Registered thermal governor 'bang_bang'
[    0.658398] thermal_sys: Registered thermal governor 'step_wise'
[    0.658398] thermal_sys: Registered thermal governor 'user_space'
[    0.658398] thermal_sys: Registered thermal governor 'power_allocator'
[    0.658398] EISA bus registered
[    0.658398] cpuidle: using governor ladder
[    0.658398] cpuidle: using governor menu
[    0.658398] ACPI FADT declares the system doesn't support PCIe ASPM, so disable it
[    0.658398] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
[    0.658398] PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem 0xc0000000-0xcfffffff] (base 0xc0000000)
[    0.658398] PCI: not using MMCONFIG
[    0.658398] PCI: Using configuration type 1 for base access
[    0.658398] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on
[    0.660102] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.
[    0.667451] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages
[    0.667451] HugeTLB: 16380 KiB vmemmap can be freed for a 1.00 GiB page
[    0.667451] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages
[    0.667451] HugeTLB: 28 KiB vmemmap can be freed for a 2.00 MiB page
[    0.667548] fbcon: Taking over console
[    0.667652] ACPI: Added _OSI(Module Device)
[    0.667656] ACPI: Added _OSI(Processor Device)
[    0.667658] ACPI: Added _OSI(3.0 _SCP Extensions)
[    0.667661] ACPI: Added _OSI(Processor Aggregator Device)
[    1.036860] ACPI: 6 ACPI AML tables successfully acquired and loaded
[    1.084891] ACPI: [Firmware Bug]: BIOS _OSI(Linux) query ignored
[    1.133167] ACPI: Interpreter enabled
[    1.133225] ACPI: PM: (supports S0 S1 S3 S5)
[    1.133233] ACPI: Using IOAPIC for interrupt routing
[    1.133305] PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem 0xc0000000-0xcfffffff] (base 0xc0000000)
[    1.135188] PCI: MMCONFIG at [mem 0xc0000000-0xcfffffff] reserved as ACPI motherboard resource
[    1.135361] HEST: Table parsing has been initialized.
[    1.135701] GHES: APEI firmware first mode is enabled by APEI bit and WHEA _OSC.
[    1.135726] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug
[    1.135735] PCI: Using E820 reservations for host bridge windows
[    1.137508] ACPI: Enabled 9 GPEs in block 00 to 3F
[    1.218908] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-7e])
[    1.218934] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI EDR HPX-Type3]
[    1.219631] acpi PNP0A08:00: _OSC: platform does not support [SHPCHotplug AER LTR DPC]
[    1.220336] acpi PNP0A08:00: _OSC: OS now controls [PCIeHotplug PME PCIeCapability]
[    1.220348] acpi PNP0A08:00: FADT indicates ASPM is unsupported, using BIOS configuration
[    1.220798] acpi PNP0A08:00: host bridge window expanded to [io  0x0000-0xbfff]; [io  0x0000-0xbfff window] ignored
[    1.220817] acpi PNP0A08:00: ignoring host bridge window [mem 0x000d0000-0x000d3fff window] (conflicts with Adapter ROM [mem 0x000c8000-0x000d19ff])
[    1.221745] PCI host bridge to bus 0000:00
[    1.221757] pci_bus 0000:00: root bus resource [io  0x0000-0xbfff]
[    1.221767] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000cffff window]
[    1.221777] pci_bus 0000:00: root bus resource [mem 0x000d4000-0x000fffff window]
[    1.221785] pci_bus 0000:00: root bus resource [mem 0xd0000000-0xebffffff window]
[    1.221794] pci_bus 0000:00: root bus resource [mem 0x380000000000-0x38007fffffff window]
[    1.221803] pci_bus 0000:00: root bus resource [bus 00-7e]
[    1.221859] pci 0000:00:00.0: [8086:0e00] type 00 class 0x060000
[    1.221979] pci 0000:00:00.0: PME# supported from D0 D3hot D3cold
[    1.222168] pci 0000:00:01.0: [8086:0e02] type 01 class 0x060400
[    1.222306] pci 0000:00:01.0: PME# supported from D0 D3hot D3cold
[    1.222598] pci 0000:00:02.0: [8086:0e04] type 01 class 0x060400
[    1.222657] pci 0000:00:02.0: enabling Extended Tags
[    1.222747] pci 0000:00:02.0: PME# supported from D0 D3hot D3cold
[    1.223019] pci 0000:00:03.0: [8086:0e08] type 01 class 0x060400
[    1.223077] pci 0000:00:03.0: enabling Extended Tags
[    1.223169] pci 0000:00:03.0: PME# supported from D0 D3hot D3cold
[    1.223433] pci 0000:00:04.0: [8086:0e20] type 00 class 0x088000
[    1.223464] pci 0000:00:04.0: reg 0x10: [mem 0x38007ffa0000-0x38007ffa3fff 64bit]
[    1.223669] pci 0000:00:04.1: [8086:0e21] type 00 class 0x088000
[    1.223696] pci 0000:00:04.1: reg 0x10: [mem 0x38007ff90000-0x38007ff93fff 64bit]
[    1.223906] pci 0000:00:04.2: [8086:0e22] type 00 class 0x088000
[    1.223933] pci 0000:00:04.2: reg 0x10: [mem 0x38007ff80000-0x38007ff83fff 64bit]
[    1.224126] pci 0000:00:04.3: [8086:0e23] type 00 class 0x088000
[    1.224154] pci 0000:00:04.3: reg 0x10: [mem 0x38007ff70000-0x38007ff73fff 64bit]
[    1.224343] pci 0000:00:04.4: [8086:0e24] type 00 class 0x088000
[    1.224371] pci 0000:00:04.4: reg 0x10: [mem 0x38007ff60000-0x38007ff63fff 64bit]
[    1.224557] pci 0000:00:04.5: [8086:0e25] type 00 class 0x088000
[    1.224585] pci 0000:00:04.5: reg 0x10: [mem 0x38007ff50000-0x38007ff53fff 64bit]
[    1.224775] pci 0000:00:04.6: [8086:0e26] type 00 class 0x088000
[    1.224803] pci 0000:00:04.6: reg 0x10: [mem 0x38007ff40000-0x38007ff43fff 64bit]
[    1.224990] pci 0000:00:04.7: [8086:0e27] type 00 class 0x088000
[    1.225018] pci 0000:00:04.7: reg 0x10: [mem 0x38007ff30000-0x38007ff33fff 64bit]
[    1.225204] pci 0000:00:05.0: [8086:0e28] type 00 class 0x088000
[    1.225399] pci 0000:00:05.1: [8086:0e29] type 00 class 0x088000
[    1.225650] pci 0000:00:05.2: [8086:0e2a] type 00 class 0x088000
[    1.225840] pci 0000:00:05.4: [8086:0e2c] type 00 class 0x080020
[    1.225863] pci 0000:00:05.4: reg 0x10: [mem 0xd0d60000-0xd0d60fff]
[    1.226085] pci 0000:00:11.0: [8086:1d3e] type 01 class 0x060400
[    1.226242] pci 0000:00:11.0: PME# supported from D0 D3hot D3cold
[    1.226469] pci 0000:00:16.0: [8086:1d3a] type 00 class 0x078000
[    1.226501] pci 0000:00:16.0: reg 0x10: [mem 0xd0d50000-0xd0d5000f 64bit]
[    1.226598] pci 0000:00:16.0: PME# supported from D0 D3hot D3cold
[    1.226763] pci 0000:00:16.1: [8086:1d3b] type 00 class 0x078000
[    1.226795] pci 0000:00:16.1: reg 0x10: [mem 0xd0d40000-0xd0d4000f 64bit]
[    1.226892] pci 0000:00:16.1: PME# supported from D0 D3hot D3cold
[    1.227086] pci 0000:00:1a.0: [8086:1d2d] type 00 class 0x0c0320
[    1.227116] pci 0000:00:1a.0: reg 0x10: [mem 0xd0d20000-0xd0d203ff]
[    1.227237] pci 0000:00:1a.0: PME# supported from D0 D3hot D3cold
[    1.227430] pci 0000:00:1b.0: [8086:1d20] type 00 class 0x040300
[    1.227461] pci 0000:00:1b.0: reg 0x10: [mem 0x38007ff20000-0x38007ff23fff 64bit]
[    1.227566] pci 0000:00:1b.0: PME# supported from D0 D3hot D3cold
[    1.227764] pci 0000:00:1c.0: [8086:1d10] type 01 class 0x060400
[    1.227901] pci 0000:00:1c.0: PME# supported from D0 D3hot D3cold
[    1.228121] pci 0000:00:1c.5: [8086:1d1a] type 01 class 0x060400
[    1.228255] pci 0000:00:1c.5: PME# supported from D0 D3hot D3cold
[    1.228455] pci 0000:00:1c.6: [8086:1d1c] type 01 class 0x060400
[    1.228589] pci 0000:00:1c.6: PME# supported from D0 D3hot D3cold
[    1.228782] pci 0000:00:1c.7: [8086:1d1e] type 01 class 0x060400
[    1.228916] pci 0000:00:1c.7: PME# supported from D0 D3hot D3cold
[    1.229110] pci 0000:00:1d.0: [8086:1d26] type 00 class 0x0c0320
[    1.229141] pci 0000:00:1d.0: reg 0x10: [mem 0xd0d10000-0xd0d103ff]
[    1.229262] pci 0000:00:1d.0: PME# supported from D0 D3hot D3cold
[    1.229458] pci 0000:00:1e.0: [8086:244e] type 01 class 0x060401
[    1.229701] pci 0000:00:1f.0: [8086:1d41] type 00 class 0x060100
[    1.230034] pci 0000:00:1f.2: [8086:1d02] type 00 class 0x010601
[    1.230061] pci 0000:00:1f.2: reg 0x10: [io  0x3070-0x3077]
[    1.230077] pci 0000:00:1f.2: reg 0x14: [io  0x3060-0x3063]
[    1.230094] pci 0000:00:1f.2: reg 0x18: [io  0x3050-0x3057]
[    1.230109] pci 0000:00:1f.2: reg 0x1c: [io  0x3040-0x3043]
[    1.230124] pci 0000:00:1f.2: reg 0x20: [io  0x3020-0x303f]
[    1.230138] pci 0000:00:1f.2: reg 0x24: [mem 0xd0d00000-0xd0d007ff]
[    1.230206] pci 0000:00:1f.2: PME# supported from D3hot
[    1.230368] pci 0000:00:1f.3: [8086:1d22] type 00 class 0x0c0500
[    1.230400] pci 0000:00:1f.3: reg 0x10: [mem 0x38007ff10000-0x38007ff100ff 64bit]
[    1.230435] pci 0000:00:1f.3: reg 0x20: [io  0x3000-0x301f]
[    1.230697] pci 0000:00:01.0: PCI bridge to [bus 01]
[    1.230913] pci 0000:00:02.0: PCI bridge to [bus 02]
[    1.231122] pci 0000:00:03.0: PCI bridge to [bus 03]
[    1.231251] pci 0000:04:00.0: [8086:1d6b] type 00 class 0x010700
[    1.231288] pci 0000:04:00.0: reg 0x10: [mem 0x38007fc00000-0x38007fc03fff 64bit pref]
[    1.231322] pci 0000:04:00.0: reg 0x18: [mem 0x38007f800000-0x38007fbfffff 64bit pref]
[    1.231343] pci 0000:04:00.0: reg 0x20: [io  0x2000-0x20ff]
[    1.231382] pci 0000:04:00.0: enabling Extended Tags
[    1.231519] pci 0000:04:00.0: reg 0x164: [mem 0x38007fc10000-0x38007fc13fff 64bit pref]
[    1.231531] pci 0000:04:00.0: VF(n) BAR0 space: [mem 0x38007fc10000-0x38007fc8bfff 64bit pref] (contains BAR0 for 31 VFs)
[    1.231864] pci 0000:04:00.3: [8086:1d70] type 00 class 0x0c0500
[    1.231894] pci 0000:04:00.3: reg 0x10: [mem 0xd0c00000-0xd0c00fff]
[    1.231937] pci 0000:04:00.3: reg 0x20: [io  0x2100-0x211f]
[    1.232044] pci 0000:04:00.3: PME# supported from D0 D3hot D3cold
[    1.232253] pci 0000:00:11.0: PCI bridge to [bus 04]
[    1.232267] pci 0000:00:11.0:   bridge window [io  0x2000-0x2fff]
[    1.232277] pci 0000:00:11.0:   bridge window [mem 0xd0c00000-0xd0cfffff]
[    1.232291] pci 0000:00:11.0:   bridge window [mem 0x38007f800000-0x38007fcfffff 64bit pref]
[    1.232451] pci 0000:05:00.0: [8086:1521] type 00 class 0x020000
[    1.232488] pci 0000:05:00.0: reg 0x10: [mem 0xd0b20000-0xd0b3ffff]
[    1.232525] pci 0000:05:00.0: reg 0x18: [io  0x1020-0x103f]
[    1.232547] pci 0000:05:00.0: reg 0x1c: [mem 0xd0b50000-0xd0b53fff]
[    1.232751] pci 0000:05:00.0: PME# supported from D0 D3hot D3cold
[    1.232807] pci 0000:05:00.0: reg 0x184: [mem 0xd0bc0000-0xd0bc3fff]
[    1.232816] pci 0000:05:00.0: VF(n) BAR0 space: [mem 0xd0bc0000-0xd0bdffff] (contains BAR0 for 8 VFs)
[    1.232864] pci 0000:05:00.0: reg 0x190: [mem 0xd0ba0000-0xd0ba3fff]
[    1.232872] pci 0000:05:00.0: VF(n) BAR3 space: [mem 0xd0ba0000-0xd0bbffff] (contains BAR3 for 8 VFs)
[    1.233298] pci 0000:05:00.1: [8086:1521] type 00 class 0x020000
[    1.233336] pci 0000:05:00.1: reg 0x10: [mem 0xd0b00000-0xd0b1ffff]
[    1.233372] pci 0000:05:00.1: reg 0x18: [io  0x1000-0x101f]
[    1.233395] pci 0000:05:00.1: reg 0x1c: [mem 0xd0b40000-0xd0b43fff]
[    1.233591] pci 0000:05:00.1: PME# supported from D0 D3hot D3cold
[    1.233647] pci 0000:05:00.1: reg 0x184: [mem 0xd0b80000-0xd0b83fff]
[    1.233656] pci 0000:05:00.1: VF(n) BAR0 space: [mem 0xd0b80000-0xd0b9ffff] (contains BAR0 for 8 VFs)
[    1.233702] pci 0000:05:00.1: reg 0x190: [mem 0xd0b60000-0xd0b63fff]
[    1.233710] pci 0000:05:00.1: VF(n) BAR3 space: [mem 0xd0b60000-0xd0b7ffff] (contains BAR3 for 8 VFs)
[    1.234050] pci 0000:00:1c.0: PCI bridge to [bus 05-06]
[    1.234063] pci 0000:00:1c.0:   bridge window [io  0x1000-0x1fff]
[    1.234073] pci 0000:00:1c.0:   bridge window [mem 0xd0b00000-0xd0bfffff]
[    1.234222] pci 0000:07:00.0: [104c:823e] type 01 class 0x060400
[    1.234432] pci 0000:07:00.0: supports D1 D2
[    1.234662] pci 0000:00:1c.5: PCI bridge to [bus 07-08]
[    1.234678] pci 0000:00:1c.5:   bridge window [mem 0xd0a00000-0xd0afffff]
[    1.234762] pci_bus 0000:08: extended config space not accessible
[    1.234815] pci 0000:08:00.0: [104c:823f] type 00 class 0x0c0010
[    1.234858] pci 0000:08:00.0: reg 0x10: [mem 0xd0a10000-0xd0a107ff]
[    1.234884] pci 0000:08:00.0: reg 0x14: [mem 0xd0a00000-0xd0a03fff]
[    1.235052] pci 0000:08:00.0: supports D1 D2
[    1.235060] pci 0000:08:00.0: PME# supported from D0 D1 D2 D3hot
[    1.235237] pci 0000:07:00.0: PCI bridge to [bus 08]
[    1.235260] pci 0000:07:00.0:   bridge window [mem 0xd0a00000-0xd0afffff]
[    1.235424] pci 0000:09:00.0: [104c:8241] type 00 class 0x0c0330
[    1.235466] pci 0000:09:00.0: reg 0x10: [mem 0xd0910000-0xd091ffff 64bit]
[    1.235494] pci 0000:09:00.0: reg 0x18: [mem 0xd0900000-0xd0901fff 64bit]
[    1.235641] pci 0000:09:00.0: supports D1 D2
[    1.235649] pci 0000:09:00.0: PME# supported from D0 D1 D2 D3hot D3cold
[    1.235901] pci 0000:00:1c.6: PCI bridge to [bus 09]
[    1.235918] pci 0000:00:1c.6:   bridge window [mem 0xd0900000-0xd09fffff]
[    1.236048] pci 0000:0a:00.0: [102b:0522] type 00 class 0x030000
[    1.236091] pci 0000:0a:00.0: reg 0x10: [mem 0xea000000-0xeaffffff pref]
[    1.236119] pci 0000:0a:00.0: reg 0x14: [mem 0xd0810000-0xd0813fff]
[    1.236144] pci 0000:0a:00.0: reg 0x18: [mem 0xd0000000-0xd07fffff]
[    1.236221] pci 0000:0a:00.0: reg 0x30: [mem 0xd0800000-0xd080ffff pref]
[    1.236259] pci 0000:0a:00.0: BAR 0: assigned to efifb
[    1.236276] pci 0000:0a:00.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]
[    1.236606] pci 0000:00:1c.7: PCI bridge to [bus 0a]
[    1.236621] pci 0000:00:1c.7:   bridge window [mem 0xd0000000-0xd08fffff]
[    1.236634] pci 0000:00:1c.7:   bridge window [mem 0xea000000-0xeaffffff 64bit pref]
[    1.236681] pci_bus 0000:0b: extended config space not accessible
[    1.236767] pci 0000:00:1e.0: PCI bridge to [bus 0b] (subtractive decode)
[    1.236786] pci 0000:00:1e.0:   bridge window [io  0x0000-0xbfff] (subtractive decode)
[    1.236796] pci 0000:00:1e.0:   bridge window [mem 0x000a0000-0x000cffff window] (subtractive decode)
[    1.236806] pci 0000:00:1e.0:   bridge window [mem 0x000d4000-0x000fffff window] (subtractive decode)
[    1.236815] pci 0000:00:1e.0:   bridge window [mem 0xd0000000-0xebffffff window] (subtractive decode)
[    1.236825] pci 0000:00:1e.0:   bridge window [mem 0x380000000000-0x38007fffffff window] (subtractive decode)
[    1.236903] pci_bus 0000:00: on NUMA node 0
[    1.238831] ACPI: PCI: Interrupt link LNKA configured for IRQ 11
[    1.238983] ACPI: PCI: Interrupt link LNKB configured for IRQ 10
[    1.239138] ACPI: PCI: Interrupt link LNKC configured for IRQ 5
[    1.239283] ACPI: PCI: Interrupt link LNKD configured for IRQ 4
[    1.239439] ACPI: PCI: Interrupt link LNKE configured for IRQ 10
[    1.239585] ACPI: PCI: Interrupt link LNKF configured for IRQ 5
[    1.239728] ACPI: PCI: Interrupt link LNKG configured for IRQ 11
[    1.239871] ACPI: PCI: Interrupt link LNKH configured for IRQ 4
[    1.240358] ACPI: PCI Root Bridge [PCI1] (domain 0000 [bus 80-fe])
[    1.240377] acpi PNP0A08:01: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI EDR HPX-Type3]
[    1.240638] acpi PNP0A08:01: _OSC: platform does not support [SHPCHotplug AER LTR DPC]
[    1.241094] acpi PNP0A08:01: _OSC: OS now controls [PCIeHotplug PME PCIeCapability]
[    1.241106] acpi PNP0A08:01: FADT indicates ASPM is unsupported, using BIOS configuration
[    1.241867] PCI host bridge to bus 0000:80
[    1.241879] pci_bus 0000:80: root bus resource [io  0x03b0-0x03df window]
[    1.241888] pci_bus 0000:80: root bus resource [io  0xc000-0xffff window]
[    1.241896] pci_bus 0000:80: root bus resource [mem 0x000a0000-0x000bffff window]
[    1.241905] pci_bus 0000:80: root bus resource [mem 0xec000000-0xfbffffff window]
[    1.241914] pci_bus 0000:80: root bus resource [mem 0x380080000000-0x3800ffffffff window]
[    1.241923] pci_bus 0000:80: root bus resource [bus 80-fe]
[    1.241974] pci 0000:80:00.0: [8086:0e01] type 01 class 0x060400
[    1.242117] pci 0000:80:00.0: PME# supported from D0 D3hot D3cold
[    1.242395] pci 0000:80:01.0: [8086:0e02] type 01 class 0x060400
[    1.242538] pci 0000:80:01.0: PME# supported from D0 D3hot D3cold
[    1.242796] pci 0000:80:02.0: [8086:0e04] type 01 class 0x060400
[    1.242857] pci 0000:80:02.0: enabling Extended Tags
[    1.242951] pci 0000:80:02.0: PME# supported from D0 D3hot D3cold
[    1.243219] pci 0000:80:03.0: [8086:0e08] type 01 class 0x060400
[    1.243280] pci 0000:80:03.0: enabling Extended Tags
[    1.243380] pci 0000:80:03.0: PME# supported from D0 D3hot D3cold
[    1.243641] pci 0000:80:04.0: [8086:0e20] type 00 class 0x088000
[    1.243672] pci 0000:80:04.0: reg 0x10: [mem 0x3800fff70000-0x3800fff73fff 64bit]
[    1.243881] pci 0000:80:04.1: [8086:0e21] type 00 class 0x088000
[    1.243910] pci 0000:80:04.1: reg 0x10: [mem 0x3800fff60000-0x3800fff63fff 64bit]
[    1.244107] pci 0000:80:04.2: [8086:0e22] type 00 class 0x088000
[    1.244137] pci 0000:80:04.2: reg 0x10: [mem 0x3800fff50000-0x3800fff53fff 64bit]
[    1.244329] pci 0000:80:04.3: [8086:0e23] type 00 class 0x088000
[    1.244358] pci 0000:80:04.3: reg 0x10: [mem 0x3800fff40000-0x3800fff43fff 64bit]
[    1.244563] pci 0000:80:04.4: [8086:0e24] type 00 class 0x088000
[    1.244591] pci 0000:80:04.4: reg 0x10: [mem 0x3800fff30000-0x3800fff33fff 64bit]
[    1.244784] pci 0000:80:04.5: [8086:0e25] type 00 class 0x088000
[    1.244813] pci 0000:80:04.5: reg 0x10: [mem 0x3800fff20000-0x3800fff23fff 64bit]
[    1.245003] pci 0000:80:04.6: [8086:0e26] type 00 class 0x088000
[    1.245032] pci 0000:80:04.6: reg 0x10: [mem 0x3800fff10000-0x3800fff13fff 64bit]
[    1.245223] pci 0000:80:04.7: [8086:0e27] type 00 class 0x088000
[    1.245252] pci 0000:80:04.7: reg 0x10: [mem 0x3800fff00000-0x3800fff03fff 64bit]
[    1.245445] pci 0000:80:05.0: [8086:0e28] type 00 class 0x088000
[    1.245641] pci 0000:80:05.1: [8086:0e29] type 00 class 0x088000
[    1.245881] pci 0000:80:05.2: [8086:0e2a] type 00 class 0x088000
[    1.246073] pci 0000:80:05.4: [8086:0e2c] type 00 class 0x080020
[    1.246095] pci 0000:80:05.4: reg 0x10: [mem 0xec000000-0xec000fff]
[    1.246501] pci 0000:80:00.0: PCI bridge to [bus 81]
[    1.246600] pci 0000:80:01.0: PCI bridge to [bus 82]
[    1.246812] pci 0000:80:02.0: PCI bridge to [bus 83]
[    1.247020] pci 0000:80:03.0: PCI bridge to [bus 84]
[    1.247082] pci_bus 0000:80: on NUMA node 1
[    1.268049] ACPI: PCI Root Bridge [UCR0] (domain 0000 [bus 7f])
[    1.268069] acpi PNP0A03:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI EDR HPX-Type3]
[    1.268127] acpi PNP0A03:00: _OSC: platform does not support [PCIeHotplug SHPCHotplug PME AER]
[    1.268215] acpi PNP0A03:00: _OSC: OS now controls [PCIeCapability LTR DPC]
[    1.268224] acpi PNP0A03:00: FADT indicates ASPM is unsupported, using BIOS configuration
[    1.268399] PCI host bridge to bus 0000:7f
[    1.268409] pci_bus 0000:7f: Unknown NUMA node; performance will be reduced
[    1.268418] pci_bus 0000:7f: root bus resource [bus 7f]
[    1.268448] pci 0000:7f:08.0: [8086:0e80] type 00 class 0x088000
[    1.268583] pci 0000:7f:09.0: [8086:0e90] type 00 class 0x088000
[    1.268706] pci 0000:7f:0a.0: [8086:0ec0] type 00 class 0x088000
[    1.268817] pci 0000:7f:0a.1: [8086:0ec1] type 00 class 0x088000
[    1.268925] pci 0000:7f:0a.2: [8086:0ec2] type 00 class 0x088000
[    1.269037] pci 0000:7f:0a.3: [8086:0ec3] type 00 class 0x088000
[    1.269147] pci 0000:7f:0b.0: [8086:0e1e] type 00 class 0x088000
[    1.269253] pci 0000:7f:0b.3: [8086:0e1f] type 00 class 0x088000
[    1.269376] pci 0000:7f:0c.0: [8086:0ee0] type 00 class 0x088000
[    1.269481] pci 0000:7f:0c.1: [8086:0ee2] type 00 class 0x088000
[    1.269590] pci 0000:7f:0c.2: [8086:0ee4] type 00 class 0x088000
[    1.269695] pci 0000:7f:0c.3: [8086:0ee6] type 00 class 0x088000
[    1.269800] pci 0000:7f:0c.4: [8086:0ee8] type 00 class 0x088000
[    1.269920] pci 0000:7f:0d.0: [8086:0ee1] type 00 class 0x088000
[    1.270025] pci 0000:7f:0d.1: [8086:0ee3] type 00 class 0x088000
[    1.270140] pci 0000:7f:0d.2: [8086:0ee5] type 00 class 0x088000
[    1.270260] pci 0000:7f:0d.3: [8086:0ee7] type 00 class 0x088000
[    1.270366] pci 0000:7f:0d.4: [8086:0ee9] type 00 class 0x088000
[    1.270476] pci 0000:7f:0e.0: [8086:0ea0] type 00 class 0x088000
[    1.270595] pci 0000:7f:0e.1: [8086:0e30] type 00 class 0x110100
[    1.270722] pci 0000:7f:0f.0: [8086:0ea8] type 00 class 0x088000
[    1.270867] pci 0000:7f:0f.1: [8086:0e71] type 00 class 0x088000
[    1.271006] pci 0000:7f:0f.2: [8086:0eaa] type 00 class 0x088000
[    1.271148] pci 0000:7f:0f.3: [8086:0eab] type 00 class 0x088000
[    1.271302] pci 0000:7f:0f.4: [8086:0eac] type 00 class 0x088000
[    1.271456] pci 0000:7f:0f.5: [8086:0ead] type 00 class 0x088000
[    1.271599] pci 0000:7f:10.0: [8086:0eb0] type 00 class 0x088000
[    1.271739] pci 0000:7f:10.1: [8086:0eb1] type 00 class 0x088000
[    1.271881] pci 0000:7f:10.2: [8086:0eb2] type 00 class 0x088000
[    1.272027] pci 0000:7f:10.3: [8086:0eb3] type 00 class 0x088000
[    1.272167] pci 0000:7f:10.4: [8086:0eb4] type 00 class 0x088000
[    1.272311] pci 0000:7f:10.5: [8086:0eb5] type 00 class 0x088000
[    1.272464] pci 0000:7f:10.6: [8086:0eb6] type 00 class 0x088000
[    1.272605] pci 0000:7f:10.7: [8086:0eb7] type 00 class 0x088000
[    1.272745] pci 0000:7f:13.0: [8086:0e1d] type 00 class 0x088000
[    1.272854] pci 0000:7f:13.1: [8086:0e34] type 00 class 0x110100
[    1.272964] pci 0000:7f:13.4: [8086:0e81] type 00 class 0x088000
[    1.273073] pci 0000:7f:13.5: [8086:0e36] type 00 class 0x110100
[    1.273182] pci 0000:7f:13.6: [8086:0e37] type 00 class 0x110100
[    1.273297] pci 0000:7f:16.0: [8086:0ec8] type 00 class 0x088000
[    1.273419] pci 0000:7f:16.1: [8086:0ec9] type 00 class 0x088000
[    1.273526] pci 0000:7f:16.2: [8086:0eca] type 00 class 0x088000
[    1.273834] ACPI: PCI Root Bridge [UCR1] (domain 0000 [bus ff])
[    1.273852] acpi PNP0A03:01: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI EDR HPX-Type3]
[    1.273911] acpi PNP0A03:01: _OSC: platform does not support [PCIeHotplug SHPCHotplug PME AER]
[    1.274000] acpi PNP0A03:01: _OSC: OS now controls [PCIeCapability LTR DPC]
[    1.274010] acpi PNP0A03:01: FADT indicates ASPM is unsupported, using BIOS configuration
[    1.274181] PCI host bridge to bus 0000:ff
[    1.274190] pci_bus 0000:ff: Unknown NUMA node; performance will be reduced
[    1.274198] pci_bus 0000:ff: root bus resource [bus ff]
[    1.274226] pci 0000:ff:08.0: [8086:0e80] type 00 class 0x088000
[    1.274351] pci 0000:ff:09.0: [8086:0e90] type 00 class 0x088000
[    1.274470] pci 0000:ff:0a.0: [8086:0ec0] type 00 class 0x088000
[    1.274584] pci 0000:ff:0a.1: [8086:0ec1] type 00 class 0x088000
[    1.274694] pci 0000:ff:0a.2: [8086:0ec2] type 00 class 0x088000
[    1.274806] pci 0000:ff:0a.3: [8086:0ec3] type 00 class 0x088000
[    1.274941] pci 0000:ff:0b.0: [8086:0e1e] type 00 class 0x088000
[    1.275058] pci 0000:ff:0b.3: [8086:0e1f] type 00 class 0x088000
[    1.275174] pci 0000:ff:0c.0: [8086:0ee0] type 00 class 0x088000
[    1.275293] pci 0000:ff:0c.1: [8086:0ee2] type 00 class 0x088000
[    1.275412] pci 0000:ff:0c.2: [8086:0ee4] type 00 class 0x088000
[    1.275527] pci 0000:ff:0c.3: [8086:0ee6] type 00 class 0x088000
[    1.275643] pci 0000:ff:0c.4: [8086:0ee8] type 00 class 0x088000
[    1.275769] pci 0000:ff:0d.0: [8086:0ee1] type 00 class 0x088000
[    1.275897] pci 0000:ff:0d.1: [8086:0ee3] type 00 class 0x088000
[    1.276008] pci 0000:ff:0d.2: [8086:0ee5] type 00 class 0x088000
[    1.276125] pci 0000:ff:0d.3: [8086:0ee7] type 00 class 0x088000
[    1.276240] pci 0000:ff:0d.4: [8086:0ee9] type 00 class 0x088000
[    1.276354] pci 0000:ff:0e.0: [8086:0ea0] type 00 class 0x088000
[    1.276473] pci 0000:ff:0e.1: [8086:0e30] type 00 class 0x110100
[    1.276606] pci 0000:ff:0f.0: [8086:0ea8] type 00 class 0x088000
[    1.276762] pci 0000:ff:0f.1: [8086:0e71] type 00 class 0x088000
[    1.276927] pci 0000:ff:0f.2: [8086:0eaa] type 00 class 0x088000
[    1.277074] pci 0000:ff:0f.3: [8086:0eab] type 00 class 0x088000
[    1.277231] pci 0000:ff:0f.4: [8086:0eac] type 00 class 0x088000
[    1.277379] pci 0000:ff:0f.5: [8086:0ead] type 00 class 0x088000
[    1.277530] pci 0000:ff:10.0: [8086:0eb0] type 00 class 0x088000
[    1.277684] pci 0000:ff:10.1: [8086:0eb1] type 00 class 0x088000
[    1.277833] pci 0000:ff:10.2: [8086:0eb2] type 00 class 0x088000
[    1.277981] pci 0000:ff:10.3: [8086:0eb3] type 00 class 0x088000
[    1.278151] pci 0000:ff:10.4: [8086:0eb4] type 00 class 0x088000
[    1.278300] pci 0000:ff:10.5: [8086:0eb5] type 00 class 0x088000
[    1.278448] pci 0000:ff:10.6: [8086:0eb6] type 00 class 0x088000
[    1.278601] pci 0000:ff:10.7: [8086:0eb7] type 00 class 0x088000
[    1.278747] pci 0000:ff:13.0: [8086:0e1d] type 00 class 0x088000
[    1.278861] pci 0000:ff:13.1: [8086:0e34] type 00 class 0x110100
[    1.278984] pci 0000:ff:13.4: [8086:0e81] type 00 class 0x088000
[    1.279101] pci 0000:ff:13.5: [8086:0e36] type 00 class 0x110100
[    1.279231] pci 0000:ff:13.6: [8086:0e37] type 00 class 0x110100
[    1.279356] pci 0000:ff:16.0: [8086:0ec8] type 00 class 0x088000
[    1.279472] pci 0000:ff:16.1: [8086:0ec9] type 00 class 0x088000
[    1.279590] pci 0000:ff:16.2: [8086:0eca] type 00 class 0x088000
[    1.280615] iommu: Default domain type: Translated
[    1.280615] iommu: DMA domain TLB invalidation policy: lazy mode
[    1.280615] SCSI subsystem initialized
[    1.280615] libata version 3.00 loaded.
[    1.280615] ACPI: bus type USB registered
[    1.280615] usbcore: registered new interface driver usbfs
[    1.280615] usbcore: registered new interface driver hub
[    1.280615] usbcore: registered new device driver usb
[    1.280615] pps_core: LinuxPPS API ver. 1 registered
[    1.280615] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
[    1.280615] PTP clock support registered
[    1.280615] EDAC MC: Ver: 3.0.0
[    1.285697] efivars: Registered efivars operations
[    1.285697] NetLabel: Initializing
[    1.285697] NetLabel:  domain hash size = 128
[    1.285697] NetLabel:  protocols = UNLABELED CIPSOv4 CALIPSO
[    1.285697] NetLabel:  unlabeled traffic allowed by default
[    1.285697] PCI: Using ACPI for IRQ routing
[    1.293058] PCI: pci_cache_line_size set to 64 bytes
[    1.293391] e820: reserve RAM buffer [mem 0xb6572018-0xb7ffffff]
[    1.293398] e820: reserve RAM buffer [mem 0xb6932018-0xb7ffffff]
[    1.293402] e820: reserve RAM buffer [mem 0xb6944018-0xb7ffffff]
[    1.293405] e820: reserve RAM buffer [mem 0xb6b86018-0xb7ffffff]
[    1.293409] e820: reserve RAM buffer [mem 0xb6cfa018-0xb7ffffff]
[    1.293412] e820: reserve RAM buffer [mem 0xb6d5f000-0xb7ffffff]
[    1.293415] e820: reserve RAM buffer [mem 0xbad48000-0xbbffffff]
[    1.293418] e820: reserve RAM buffer [mem 0xbafc5000-0xbbffffff]
[    1.293421] e820: reserve RAM buffer [mem 0xbb3d4000-0xbbffffff]
[    1.293425] e820: reserve RAM buffer [mem 0xbe000000-0xbfffffff]
[    1.293606] pci 0000:0a:00.0: vgaarb: setting as boot VGA device
[    1.293618] pci 0000:0a:00.0: vgaarb: bridge control possible
[    1.293624] pci 0000:0a:00.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none
[    1.293669] vgaarb: loaded
[    1.296396] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0, 0, 0, 0, 0, 0
[    1.296421] hpet0: 8 comparators, 64-bit 14.318180 MHz counter
[    1.298705] clocksource: Switched to clocksource tsc-early
[    1.298760] VFS: Disk quotas dquot_6.6.0
[    1.298760] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[    1.298760] pnp: PnP ACPI init
[    1.298760] system 00:00: [io  0x0680-0x069f] has been reserved
[    1.298760] system 00:00: [io  0xffff] has been reserved
[    1.298760] system 00:00: [io  0xffff] has been reserved
[    1.298760] system 00:00: [io  0xffff] has been reserved
[    1.298760] system 00:00: [io  0x0400-0x0453] has been reserved
[    1.298760] system 00:00: [io  0x0458-0x047f] has been reserved
[    1.298760] system 00:00: [io  0x0500-0x057f] has been reserved
[    1.298760] system 00:00: [io  0x0600-0x061f] has been reserved
[    1.298760] system 00:00: [io  0x0ca2-0x0ca5] could not be reserved
[    1.298760] system 00:00: [io  0x0cf9] could not be reserved
[    1.298760] system 00:02: [io  0x0454-0x0457] has been reserved
[    1.298760] pnp 00:03: [dma 0 disabled]
[    1.299060] system 00:04: [mem 0xfed1c000-0xfed1ffff] has been reserved
[    1.299075] system 00:04: [mem 0xebfff000-0xebffffff] has been reserved
[    1.299083] system 00:04: [mem 0xc0000000-0xcfffffff] has been reserved
[    1.299090] system 00:04: [mem 0xfed20000-0xfed3ffff] has been reserved
[    1.299098] system 00:04: [mem 0xfed45000-0xfed8ffff] has been reserved
[    1.299106] system 00:04: [mem 0xff000000-0xffffffff] has been reserved
[    1.299114] system 00:04: [mem 0xfee00000-0xfeefffff] could not be reserved
[    1.299122] system 00:04: [mem 0xfec00000-0xfecfffff] could not be reserved
[    1.299130] system 00:04: [mem 0xd0d70000-0xd0d70fff] has been reserved
[    1.299891] system 00:05: [mem 0x00000000-0x0009cfff] could not be reserved
[    1.299934] pnp: PnP ACPI: found 6 devices
[    1.308478] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns
[    1.308792] NET: Registered PF_INET protocol family
[    1.309833] IP idents hash table entries: 262144 (order: 9, 2097152 bytes, vmalloc)
[    1.319070] tcp_listen_portaddr_hash hash table entries: 32768 (order: 7, 524288 bytes, vmalloc)
[    1.319369] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, vmalloc)
[    1.321205] TCP established hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc hugepage)
[    1.323069] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc)
[    1.323658] TCP: Hash tables configured (established 524288 bind 65536)
[    1.325326] MPTCP token hash table entries: 65536 (order: 8, 1572864 bytes, vmalloc)
[    1.326060] UDP hash table entries: 32768 (order: 8, 1048576 bytes, vmalloc)
[    1.326774] UDP-Lite hash table entries: 32768 (order: 8, 1048576 bytes, vmalloc)
[    1.327503] NET: Registered PF_UNIX/PF_LOCAL protocol family
[    1.327533] NET: Registered PF_XDP protocol family
[    1.327627] pci 0000:00:01.0: PCI bridge to [bus 01]
[    1.327651] pci 0000:00:02.0: PCI bridge to [bus 02]
[    1.327668] pci 0000:00:03.0: PCI bridge to [bus 03]
[    1.327688] pci 0000:00:11.0: PCI bridge to [bus 04]
[    1.327696] pci 0000:00:11.0:   bridge window [io  0x2000-0x2fff]
[    1.327709] pci 0000:00:11.0:   bridge window [mem 0xd0c00000-0xd0cfffff]
[    1.327720] pci 0000:00:11.0:   bridge window [mem 0x38007f800000-0x38007fcfffff 64bit pref]
[    1.327738] pci 0000:00:1c.0: PCI bridge to [bus 05-06]
[    1.327746] pci 0000:00:1c.0:   bridge window [io  0x1000-0x1fff]
[    1.327756] pci 0000:00:1c.0:   bridge window [mem 0xd0b00000-0xd0bfffff]
[    1.327773] pci 0000:07:00.0: PCI bridge to [bus 08]
[    1.327786] pci 0000:07:00.0:   bridge window [mem 0xd0a00000-0xd0afffff]
[    1.327808] pci 0000:00:1c.5: PCI bridge to [bus 07-08]
[    1.327818] pci 0000:00:1c.5:   bridge window [mem 0xd0a00000-0xd0afffff]
[    1.327835] pci 0000:00:1c.6: PCI bridge to [bus 09]
[    1.327844] pci 0000:00:1c.6:   bridge window [mem 0xd0900000-0xd09fffff]
[    1.327862] pci 0000:00:1c.7: PCI bridge to [bus 0a]
[    1.327873] pci 0000:00:1c.7:   bridge window [mem 0xd0000000-0xd08fffff]
[    1.327882] pci 0000:00:1c.7:   bridge window [mem 0xea000000-0xeaffffff 64bit pref]
[    1.327898] pci 0000:00:1e.0: PCI bridge to [bus 0b]
[    1.327917] pci_bus 0000:00: resource 4 [io  0x0000-0xbfff]
[    1.327924] pci_bus 0000:00: resource 5 [mem 0x000a0000-0x000cffff window]
[    1.327932] pci_bus 0000:00: resource 6 [mem 0x000d4000-0x000fffff window]
[    1.327940] pci_bus 0000:00: resource 7 [mem 0xd0000000-0xebffffff window]
[    1.327947] pci_bus 0000:00: resource 8 [mem 0x380000000000-0x38007fffffff window]
[    1.327956] pci_bus 0000:04: resource 0 [io  0x2000-0x2fff]
[    1.327963] pci_bus 0000:04: resource 1 [mem 0xd0c00000-0xd0cfffff]
[    1.327970] pci_bus 0000:04: resource 2 [mem 0x38007f800000-0x38007fcfffff 64bit pref]
[    1.327979] pci_bus 0000:05: resource 0 [io  0x1000-0x1fff]
[    1.327985] pci_bus 0000:05: resource 1 [mem 0xd0b00000-0xd0bfffff]
[    1.327993] pci_bus 0000:07: resource 1 [mem 0xd0a00000-0xd0afffff]
[    1.328001] pci_bus 0000:08: resource 1 [mem 0xd0a00000-0xd0afffff]
[    1.328008] pci_bus 0000:09: resource 1 [mem 0xd0900000-0xd09fffff]
[    1.328016] pci_bus 0000:0a: resource 1 [mem 0xd0000000-0xd08fffff]
[    1.328023] pci_bus 0000:0a: resource 2 [mem 0xea000000-0xeaffffff 64bit pref]
[    1.328032] pci_bus 0000:0b: resource 4 [io  0x0000-0xbfff]
[    1.328039] pci_bus 0000:0b: resource 5 [mem 0x000a0000-0x000cffff window]
[    1.328046] pci_bus 0000:0b: resource 6 [mem 0x000d4000-0x000fffff window]
[    1.328053] pci_bus 0000:0b: resource 7 [mem 0xd0000000-0xebffffff window]
[    1.328061] pci_bus 0000:0b: resource 8 [mem 0x380000000000-0x38007fffffff window]
[    1.328370] pci 0000:80:00.0: PCI bridge to [bus 81]
[    1.328391] pci 0000:80:01.0: PCI bridge to [bus 82]
[    1.328409] pci 0000:80:02.0: PCI bridge to [bus 83]
[    1.328425] pci 0000:80:03.0: PCI bridge to [bus 84]
[    1.328444] pci_bus 0000:80: resource 4 [io  0x03b0-0x03df window]
[    1.328451] pci_bus 0000:80: resource 5 [io  0xc000-0xffff window]
[    1.328460] pci_bus 0000:80: resource 6 [mem 0x000a0000-0x000bffff window]
[    1.328467] pci_bus 0000:80: resource 7 [mem 0xec000000-0xfbffffff window]
[    1.328474] pci_bus 0000:80: resource 8 [mem 0x380080000000-0x3800ffffffff window]
[    1.328658] pci 0000:00:05.0: disabled boot interrupts on device [8086:0e28]
[    1.329940] pci 0000:09:00.0: enabling device (0140 -> 0142)
[    1.330153] pci 0000:80:05.0: disabled boot interrupts on device [8086:0e28]
[    1.330465] PCI: CLS 64 bytes, default 64
[    1.330482] PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
[    1.330489] software IO TLB: mapped [mem 0x00000000b01f5000-0x00000000b41f5000] (64MB)
[    1.330594] Unpacking initramfs...
[    1.332700] unchecked MSR access error: WRMSR to 0xe44 (tried to write 0x0000000000000003) at rIP: 0xffffffff8d2a6698 (native_write_msr+0x8/0x30)
[    1.332718] Call Trace:
[    1.332723]  <TASK>
[    1.332727]  ? show_stack_regs+0x27/0x30
[    1.332737]  ? ex_handler_msr+0x10f/0x180
[    1.332744]  ? search_extable+0x2b/0x40
[    1.332754]  ? fixup_exception+0x315/0x380
[    1.332760]  ? exc_general_protection+0x139/0x460
[    1.332766]  ? idr_alloc_cyclic+0x59/0xc0
[    1.332773]  ? asm_exc_general_protection+0x2b/0x30
[    1.332783]  ? native_write_msr+0x8/0x30
[    1.332789]  ? ivbep_uncore_msr_init_box+0x47/0x60
[    1.332796]  uncore_box_ref.part.0+0xa6/0xe0
[    1.332803]  uncore_event_cpu_online+0x6e/0x1c0
[    1.332810]  ? __pfx_uncore_event_cpu_online+0x10/0x10
[    1.332817]  cpuhp_invoke_callback+0x165/0x4b0
[    1.332824]  ? try_to_wake_up+0x284/0x6b0
[    1.332832]  cpuhp_thread_fun+0xc4/0x1e0
[    1.332838]  ? __pfx_smpboot_thread_fn+0x10/0x10
[    1.332846]  smpboot_thread_fn+0xe7/0x1e0
[    1.332853]  kthread+0xfb/0x130
[    1.332859]  ? __pfx_kthread+0x10/0x10
[    1.332865]  ret_from_fork+0x40/0x60
[    1.332871]  ? __pfx_kthread+0x10/0x10
[    1.332876]  ret_from_fork_asm+0x1b/0x30
[    1.332886]  </TASK>
[    1.338362] Initialise system trusted keyrings
[    1.338550] workingset: timestamp_bits=36 max_order=24 bucket_order=0
[    1.338609] zbud: loaded
[    1.339788] squashfs: version 4.0 (2009/01/31) Phillip Lougher
[    1.340079] fuse: init (API version 7.38)
[    1.360915] Key type asymmetric registered
[    1.360925] Asymmetric key parser 'x509' registered
[    1.361009] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 243)
[    1.361245] io scheduler mq-deadline registered
[    1.362511] pcieport 0000:00:01.0: PME: Signaling with IRQ 25
[    1.362892] pcieport 0000:00:02.0: PME: Signaling with IRQ 26
[    1.363267] pcieport 0000:00:03.0: PME: Signaling with IRQ 27
[    1.363677] pcieport 0000:00:11.0: PME: Signaling with IRQ 28
[    1.364035] pcieport 0000:00:1c.0: PME: Signaling with IRQ 29
[    1.364358] pcieport 0000:00:1c.5: PME: Signaling with IRQ 30
[    1.364517] pcieport 0000:00:1c.6: PME: Signaling with IRQ 31
[    1.364815] pcieport 0000:00:1c.7: PME: Signaling with IRQ 32
[    1.365439] pcieport 0000:80:00.0: PME: Signaling with IRQ 34
[    1.365740] pcieport 0000:80:01.0: PME: Signaling with IRQ 35
[    1.366001] pcieport 0000:80:02.0: PME: Signaling with IRQ 36
[    1.366264] pcieport 0000:80:03.0: PME: Signaling with IRQ 37
[    1.366710] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4
[    1.366925] efifb: probing for efifb
[    1.366960] efifb: framebuffer at 0xea000000, using 1876k, total 1875k
[    1.366969] efifb: mode is 800x600x32, linelength=3200, pages=1
[    1.366976] efifb: scrolling: redraw
[    1.366980] efifb: Truecolor: size=8:8:8:8, shift=24:16:8:0
[    1.367175] Console: switching to colour frame buffer device 100x37
[    1.380507] fb0: EFI VGA frame buffer device
[    1.380859] Monitor-Mwait will be used to enter C-1 state
[    1.380871] Monitor-Mwait will be used to enter C-2 state
[    1.380877] ACPI: \_SB_.SCK0.CP00: Found 2 idle states
[    1.383201] input: Sleep Button as /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0C0E:00/input/input0
[    1.383644] ACPI: button: Sleep Button [SLPB]
[    1.383852] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input1
[    1.384188] ACPI: button: Power Button [PWRF]
[    1.434284] ERST: Error Record Serialization Table (ERST) support is initialized.
[    1.434602] pstore: Registered erst as persistent store backend
[    1.435020] Serial: 8250/16550 driver, 32 ports, IRQ sharing enabled
[    1.455901] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A
[    1.459706] Linux agpgart interface v0.103
[    1.474519] loop: module loaded
[    1.475026] Freeing initrd memory: 23700K
[    1.480741] tun: Universal TUN/TAP device driver, 1.6
[    1.492030] PPP generic driver version 2.4.2
[    1.497977] VFIO - User Level meta-driver version: 0.3
[    1.503954] i8042: PNP: No PS/2 controller found.
[    1.504227] ehci-pci 0000:00:1a.0: EHCI Host Controller
[    1.509733] mousedev: PS/2 mouse device common for all mice
[    1.515527] ehci-pci 0000:00:1a.0: new USB bus registered, assigned bus number 1
[    1.521332] rtc_cmos 00:01: RTC can wake from S4
[    1.527422] ehci-pci 0000:00:1a.0: debug port 2
[    1.534753] rtc_cmos 00:01: registered as rtc0
[    1.544038] ehci-pci 0000:00:1a.0: irq 22, io mem 0xd0d20000
[    1.546649] rtc_cmos 00:01: setting system clock to 2023-08-08T19:42:20 UTC (1691523740)
[    1.560110] rtc_cmos 00:01: alarms up to one month, y3k, 242 bytes nvram, hpet irqs
[    1.566385] ehci-pci 0000:00:1a.0: USB 2.0 started, EHCI 1.00
[    1.567278] i2c_dev: i2c /dev entries driver
[    1.574605] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 6.05
[    1.589714] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    1.597438] usb usb1: Product: EHCI Host Controller
[    1.605065] usb usb1: Manufacturer: Linux 6.5.0-rc4+ ehci_hcd
[    1.612719] usb usb1: SerialNumber: 0000:00:1a.0
[    1.620467] hub 1-0:1.0: USB hub found
[    1.622541] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.
[    1.627995] hub 1-0:1.0: 2 ports detected
[    1.643678] device-mapper: uevent: version 1.0.3
[    1.651585] ehci-pci 0000:00:1d.0: EHCI Host Controller
[    1.659340] device-mapper: ioctl: 4.48.0-ioctl (2023-03-01) initialised: dm-devel@redhat.com
[    1.666866] ehci-pci 0000:00:1d.0: new USB bus registered, assigned bus number 2
[    1.674643] platform eisa.0: Probing EISA bus 0
[    1.682464] ehci-pci 0000:00:1d.0: debug port 2
[    1.690057] platform eisa.0: EISA: Cannot allocate resource for mainboard
[    1.701661] ehci-pci 0000:00:1d.0: irq 20, io mem 0xd0d10000
[    1.705210] platform eisa.0: Cannot allocate resource for EISA slot 1
[    1.720380] platform eisa.0: Cannot allocate resource for EISA slot 2
[    1.726388] ehci-pci 0000:00:1d.0: USB 2.0 started, EHCI 1.00
[    1.727763] platform eisa.0: Cannot allocate resource for EISA slot 3
[    1.735229] usb usb2: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 6.05
[    1.742720] platform eisa.0: Cannot allocate resource for EISA slot 4
[    1.750463] usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    1.758265] platform eisa.0: Cannot allocate resource for EISA slot 5
[    1.766182] usb usb2: Product: EHCI Host Controller
[    1.774155] platform eisa.0: Cannot allocate resource for EISA slot 6
[    1.782093] usb usb2: Manufacturer: Linux 6.5.0-rc4+ ehci_hcd
[    1.782094] usb usb2: SerialNumber: 0000:00:1d.0
[    1.790129] platform eisa.0: Cannot allocate resource for EISA slot 7
[    1.798241] hub 2-0:1.0: USB hub found
[    1.805919] platform eisa.0: Cannot allocate resource for EISA slot 8
[    1.813900] hub 2-0:1.0: 2 ports detected
[    1.821835] platform eisa.0: EISA: Detected 0 cards
[    1.845842] intel_pstate: Intel P-state driver initializing
[    1.864192] ledtrig-cpu: registered to indicate activity on CPUs
[    1.872427] drop_monitor: Initializing network drop monitor service
[    1.880514] NET: Registered PF_INET6 protocol family
[    1.900670] Segment Routing with IPv6
[    1.906647] usb 1-1: new high-speed USB device number 2 using ehci-pci
[    1.908446] In-situ OAM (IOAM) with IPv6
[    1.924037] NET: Registered PF_PACKET protocol family
[    1.932006] Key type dns_resolver registered
[    1.945487] microcode: Microcode Update Driver: v2.2.
[    1.945643] IPI shorthand broadcast: enabled
[    1.962151] sched_clock: Marking stable (1958399532, 2311085)->(2060814870, -100104253)
[    1.970188] registered taskstats version 1
[    1.982235] Loading compiled-in X.509 certificates
[    1.990763] Loaded X.509 cert 'Build time autogenerated kernel key: d2d3ceb6490e4146b306479590527fedf74296ce'
[    2.008388] Key type .fscrypt registered
[    2.016100] Key type fscrypt-provisioning registered
[    2.023941] pstore: Using crash dump compression: deflate
[    2.036135] Key type encrypted registered
[    2.043684] ima: No TPM chip found, activating TPM-bypass!
[    2.051441] Loading compiled-in module X.509 certificates
[    2.059807] Loaded X.509 cert 'Build time autogenerated kernel key: d2d3ceb6490e4146b306479590527fedf74296ce'
[    2.071257] usb 1-1: New USB device found, idVendor=8087, idProduct=0024, bcdDevice= 0.00
[    2.075432] ima: Allocated hash algorithm: sha1
[    2.083405] usb 1-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
[    2.090329] usb 2-1: new high-speed USB device number 2 using ehci-pci
[    2.091269] ima: No architecture policies found
[    2.101076] hub 1-1:1.0: USB hub found
[    2.107193] evm: Initialising EVM extended attributes:
[    2.115272] hub 1-1:1.0: 6 ports detected
[    2.122767] evm: security.selinux
[    2.146018] evm: security.SMACK64
[    2.153503] evm: security.SMACK64EXEC
[    2.160828] evm: security.SMACK64TRANSMUTE
[    2.168079] evm: security.SMACK64MMAP
[    2.175176] evm: security.apparmor (disabled)
[    2.182202] evm: security.ima
[    2.189052] evm: security.capability
[    2.195774] evm: HMAC attrs: 0x1
[    2.202818] PM:   Magic number: 3:314:748
[    2.209404] PM:   hash matches drivers/base/power/main.c:1266
[    2.216019] alarmtimer alarmtimer.0.auto: hash matches
[    2.222769] acpi LNXCPU:73: hash matches
[    2.229655] RAS: Correctable Errors collector initialized.
[    2.236323] clk: Disabling unused clocks
[    2.244719] Freeing unused decrypted memory: 2036K
[    2.252648] Freeing unused kernel image (initmem) memory: 4576K
[    2.259103] usb 2-1: New USB device found, idVendor=8087, idProduct=0024, bcdDevice= 0.00
[    2.265745] usb 2-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
[    2.272693] hub 2-1:1.0: USB hub found
[    2.279173] hub 2-1:1.0: 8 ports detected
[    2.286334] Write protecting the kernel read-only data: 26624k
[    2.293376] Freeing unused kernel image (rodata/data gap) memory: 1312K
[    2.338330] tsc: Refined TSC clocksource calibration: 2992.788 MHz
[    2.344804] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x2b23a839720, max_idle_ns: 440795205310 ns
[    2.352607] x86/mm: Checked W+X mappings: passed, no W+X pages found.
[    2.364593] x86/mm: Checking user space page tables
[    2.371045] clocksource: Switched to clocksource tsc
[    2.421726] x86/mm: Checked W+X mappings: passed, no W+X pages found.
[    2.428006] Run /init as init process
[    2.434129]   with arguments:
[    2.434131]     /init
[    2.434133]   with environment:
[    2.434134]     HOME=/
[    2.434135]     TERM=linux
[    2.434136]     BOOT_IMAGE=/boot/vmlinuz-6.5.0-rc4+
[    2.434327] usb 1-1.2: new full-speed USB device number 3 using ehci-pci
[    2.563261] usb 1-1.2: New USB device found, idVendor=14dd, idProduct=1005, bcdDevice= 0.00
[    2.569747] usb 1-1.2: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[    2.576244] usb 1-1.2: Product: D2CIM-VUSB
[    2.582688] usb 1-1.2: Manufacturer: Raritan
[    2.589198] usb 1-1.2: SerialNumber: EFFB212D0A6E63D
[    2.595834] usb 2-1.2: new high-speed USB device number 3 using ehci-pci
[    2.623774] dca service started, version 1.12.1
[    2.631850] i801_smbus 0000:00:1f.3: enabling device (0141 -> 0143)
[    2.639146] i801_smbus 0000:00:1f.3: SMBus using PCI interrupt
[    2.646244] i801_smbus 0000:04:00.3: enabling device (0000 -> 0003)
[    2.653498] i801_smbus 0000:04:00.3: Enabling SMBus device
[    2.660725] i801_smbus 0000:04:00.3: SMBus using PCI interrupt
[    2.668214] xhci_hcd 0000:09:00.0: xHCI Host Controller
[    2.675372] xhci_hcd 0000:09:00.0: new USB bus registered, assigned bus number 3
[    2.683545] i2c i2c-0: 8/16 memory slots populated (from DMI)
[    2.683893] hid: raw HID events driver (C) Jiri Kosina
[    2.690746] i2c i2c-0: Systems with more than 4 memory slots not supported yet, not instantiating SPD
[    2.691190] xhci_hcd 0000:09:00.0: hcc params 0x0270f06d hci version 0x96 quirks 0x0000000004000000
[    2.728298] ahci 0000:00:1f.2: version 3.0
[    2.729831] xhci_hcd 0000:09:00.0: xHCI Host Controller
[    2.737554] xhci_hcd 0000:09:00.0: new USB bus registered, assigned bus number 4
[    2.745357] ahci 0000:00:1f.2: AHCI 0001.0300 32 slots 6 ports 6 Gbps 0x3f impl SATA mode
[    2.753357] ahci 0000:00:1f.2: flags: 64bit ncq sntf pm led clo pio slum part ems apst 
[    2.761393] xhci_hcd 0000:09:00.0: Host supports USB 3.0 SuperSpeed
[    2.769474] usb usb3: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 6.05
[    2.777601] usb usb3: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    2.785582] usb usb3: Product: xHCI Host Controller
[    2.793349] usb usb3: Manufacturer: Linux 6.5.0-rc4+ xhci-hcd
[    2.801125] usb usb3: SerialNumber: 0000:09:00.0
[    2.809006] usb 2-1.2: New USB device found, idVendor=0424, idProduct=2514, bcdDevice= b.b3
[    2.816920] usb 2-1.2: New USB device strings: Mfr=0, Product=0, SerialNumber=0
[    2.825094] hub 2-1.2:1.0: USB hub found
[    2.831038] usbcore: registered new interface driver usbhid
[    2.832842] hub 3-0:1.0: USB hub found
[    2.840630] usbhid: USB HID core driver
[    2.855998] hub 2-1.2:1.0: 4 ports detected
[    2.863490] hub 3-0:1.0: 2 ports detected
[    2.871010] firewire_ohci 0000:08:00.0: added OHCI v1.10 device as card 0, 8 IR + 8 IT contexts, quirks 0x2
[    2.886572] isci: Intel(R) C600 SAS Controller Driver - version 1.2.0
[    2.886638] igb: Intel(R) Gigabit Ethernet Network Driver
[    2.886978] scsi host0: ahci
[    2.887273] scsi host1: ahci
[    2.887432] scsi host2: ahci
[    2.887606] scsi host3: ahci
[    2.887681] usb usb4: We don't know the algorithms for LPM for this host, disabling LPM.
[    2.887763] usb usb4: New USB device found, idVendor=1d6b, idProduct=0003, bcdDevice= 6.05
[    2.887766] usb usb4: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    2.887767] usb usb4: Product: xHCI Host Controller
[    2.887768] usb usb4: Manufacturer: Linux 6.5.0-rc4+ xhci-hcd
[    2.887770] usb usb4: SerialNumber: 0000:09:00.0
[    2.887845] scsi host4: ahci
[    2.888010] scsi host5: ahci
[    2.888079] ata1: SATA max UDMA/133 abar m2048@0xd0d00000 port 0xd0d00100 irq 38
[    2.888089] ata2: SATA max UDMA/133 abar m2048@0xd0d00000 port 0xd0d00180 irq 38
[    2.888098] ata3: SATA max UDMA/133 abar m2048@0xd0d00000 port 0xd0d00200 irq 38
[    2.888108] ata4: SATA max UDMA/133 abar m2048@0xd0d00000 port 0xd0d00280 irq 38
[    2.888117] ata5: SATA max UDMA/133 abar m2048@0xd0d00000 port 0xd0d00300 irq 38
[    2.888127] ata6: SATA max UDMA/133 abar m2048@0xd0d00000 port 0xd0d00380 irq 38
[    2.888248] hub 4-0:1.0: USB hub found
[    2.888277] hub 4-0:1.0: 2 ports detected
[    2.894458] isci 0000:04:00.0: driver configured for rev: 6 silicon
[    2.902269] igb: Copyright (c) 2007-2014 Intel Corporation.
[    2.924396] isci 0000:04:00.0: Unable to obtain EFI var data for OEM parms
[    2.932547] input: Raritan D2CIM-VUSB Keyboard as /devices/pci0000:00/0000:00:1a.0/usb1/1-1/1-1.2/1-1.2:1.0/0003:14DD:1005.0001/input/input2
[    2.941853] resource: resource sanity check: requesting [mem 0x00000000000c8000-0x00000000000d19ff], which spans more than PCI Bus 0000:00 [mem 0x000a0000-0x000cffff window]
[    3.006569] input: Raritan D2CIM-VUSB Mouse as /devices/pci0000:00/0000:00:1a.0/usb1/1-1/1-1.2/1-1.2:1.0/0003:14DD:1005.0001/input/input3
[    3.006951] caller pci_map_biosrom+0x2a/0x30 mapping multiple BARs
[    3.013872] hid-generic 0003:14DD:1005.0001: input,hidraw0: USB HID v1.11 Keyboard [Raritan D2CIM-VUSB] on usb-0000:00:1a.0-1.2/input0
[    3.020208] usb 2-1.4: new full-speed USB device number 4 using ehci-pci
[    3.137957] isci 0000:04:00.0: OEM parameter table found in OROM
[    3.145077] isci 0000:04:00.0: OEM SAS parameters (version: 1.1) loaded (platform)
[    3.152482] isci 0000:04:00.0: SCU controller 0: phy 3-0 cables: {short, short, short, short}
[    3.163085] scsi host6: isci
[    3.196717] igb 0000:05:00.0: added PHC on eth0
[    3.200940] ata5: SATA link down (SStatus 0 SControl 300)
[    3.203972] igb 0000:05:00.0: Intel(R) Gigabit Ethernet Network Connection
[    3.211547] ata1: SATA link down (SStatus 0 SControl 300)
[    3.219028] igb 0000:05:00.0: eth0: (PCIe:5.0Gb/s:Width x4) 00:1e:67:b0:d7:cf
[    3.219155] igb 0000:05:00.0: eth0: PBA No: 100000-000
[    3.226796] ata3: SATA link down (SStatus 0 SControl 300)
[    3.234267] igb 0000:05:00.0: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s)
[    3.257272] ata4: SATA link down (SStatus 0 SControl 300)
[    3.265188] ata6: SATA link down (SStatus 0 SControl 300)
[    3.272995] ata2: SATA link down (SStatus 0 SControl 300)
[    3.298744] igb 0000:05:00.1: added PHC on eth1
[    3.306364] igb 0000:05:00.1: Intel(R) Gigabit Ethernet Network Connection
[    3.314218] igb 0000:05:00.1: eth1: (PCIe:5.0Gb/s:Width x4) 00:1e:67:b0:d7:d0
[    3.322322] igb 0000:05:00.1: eth1: PBA No: 100000-000
[    3.330400] igb 0000:05:00.1: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s)
[    3.350307] usb 2-1.4: New USB device found, idVendor=046b, idProduct=ff10, bcdDevice= 1.00
[    3.358977] usb 2-1.4: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[    3.367536] usb 2-1.4: Product: Virtual Keyboard and Mouse
[    3.376166] usb 2-1.4: Manufacturer: American Megatrends Inc.
[    3.384755] usb 2-1.4: SerialNumber: serial
[    3.394444] firewire_core 0000:08:00.0: created device fw0: GUID 00902700029f41bc, S800
[    3.395054] input: American Megatrends Inc. Virtual Keyboard and Mouse as /devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.4/2-1.4:1.0/0003:046B:FF10.0002/input/input4
[    3.420481] hid-generic 0003:046B:FF10.0002: input,hidraw1: USB HID v1.10 Keyboard [American Megatrends Inc. Virtual Keyboard and Mouse] on usb-0000:00:1d.0-1.4/input0
[    3.438745] input: American Megatrends Inc. Virtual Keyboard and Mouse as /devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.4/2-1.4:1.1/0003:046B:FF10.0003/input/input5
[    3.456077] hid-generic 0003:046B:FF10.0003: input,hidraw2: USB HID v1.10 Mouse [American Megatrends Inc. Virtual Keyboard and Mouse] on usb-0000:00:1d.0-1.4/input1
[    4.194501] sas: phy-6:0 added to port-6:0, phy_mask:0x1 (5001e67b0d7cf000)
[    4.194767] sas: DOING DISCOVERY on port 0, pid:282
[    4.194858] sas: Enter sas_scsi_recover_host busy: 0 failed: 0
[    4.203488] sas: ata7: end_device-6:0: dev error handler
[    4.362617] ata7.00: ATA-9: INTEL SSDSC2BB480G4, D2010370, max UDMA/133
[    4.371265] ata7.00: 937703088 sectors, multi 1: LBA48 NCQ (depth 32)
[    4.380105] ata7.00: configured for UDMA/133
[    4.388625] sas: --- Exit sas_scsi_recover_host: busy: 0 failed: 0 tries: 1
[    4.399509] scsi 6:0:0:0: Direct-Access     ATA      INTEL SSDSC2BB48 0370 PQ: 0 ANSI: 5
[    4.408351] sas: DONE DISCOVERY on port 0, pid:282, result:0
[    4.408528] sd 6:0:0:0: Attached scsi generic sg0 type 0
[    4.408857] ata7.00: Enabling discard_zeroes_data
[    4.425830] sd 6:0:0:0: [sda] 937703088 512-byte logical blocks: (480 GB/447 GiB)
[    4.434567] sd 6:0:0:0: [sda] 4096-byte physical blocks
[    4.443181] sd 6:0:0:0: [sda] Write Protect is off
[    4.451612] sd 6:0:0:0: [sda] Mode Sense: 00 3a 00 00
[    4.451634] sd 6:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[    4.468713] sd 6:0:0:0: [sda] Preferred minimum I/O size 4096 bytes
[    4.478305] ata7.00: Enabling discard_zeroes_data
[    4.488320]  sda: sda1 sda2 sda3
[    4.497103] sd 6:0:0:0: [sda] Attached SCSI disk
[    4.512433] igb 0000:05:00.0 eno0: renamed from eth0
[    4.550539] igb 0000:05:00.1 enp5s0f1: renamed from eth1
[    4.757701] EXT4-fs (sda2): mounted filesystem b0358c51-91a1-4785-90fe-f46828ae9f39 ro with ordered data mode. Quota mode: none.
[    4.944429] systemd[1]: Inserted module 'autofs4'
[    5.183080] systemd[1]: systemd 249.11-0ubuntu3.9 running in system mode (+PAM +AUDIT +SELINUX +APPARMOR +IMA +SMACK +SECCOMP +GCRYPT +GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 -PWQUALITY -P11KIT -QRENCODE +BZIP2 +LZ4 +XZ +ZLIB +ZSTD -XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified)
[    5.215783] systemd[1]: Detected architecture x86-64.
[    5.249345] systemd[1]: Hostname set to <sohilbuildbox>.
[    5.300934] memfd_create() without MFD_EXEC nor MFD_NOEXEC_SEAL, pid=1 'systemd'
[    5.350702] block sda: the capability attribute has been deprecated.
[    5.531935] systemd[1]: Queued start job for default target Graphical Interface.
[    5.559315] systemd[1]: Created slice Slice /system/modprobe.
[    5.576491] systemd[1]: Created slice Slice /system/systemd-fsck.
[    5.592926] systemd[1]: Created slice User and Session Slice.
[    5.609025] systemd[1]: Started Forward Password Requests to Wall Directory Watch.
[    5.625846] systemd[1]: Set up automount Arbitrary Executable File Formats File System Automount Point.
[    5.651994] systemd[1]: Reached target Remote File Systems.
[    5.669928] systemd[1]: Reached target Slice Units.
[    5.687624] systemd[1]: Reached target Mounting snaps.
[    5.705192] systemd[1]: Reached target Local Verity Protected Volumes.
[    5.722926] systemd[1]: Listening on Syslog Socket.
[    5.740194] systemd[1]: Listening on fsck to fsckd communication Socket.
[    5.758045] systemd[1]: Listening on initctl Compatibility Named Pipe.
[    5.776329] systemd[1]: Listening on Journal Audit Socket.
[    5.794071] systemd[1]: Listening on Journal Socket (/dev/log).
[    5.811604] systemd[1]: Listening on Journal Socket.
[    5.829439] systemd[1]: Listening on udev Control Socket.
[    5.847365] systemd[1]: Listening on udev Kernel Socket.
[    5.878510] systemd[1]: Mounting Huge Pages File System...
[    5.896978] systemd[1]: Mounting POSIX Message Queue File System...
[    5.916182] systemd[1]: Mounting Kernel Debug File System...
[    5.934611] systemd[1]: Mounting Kernel Trace File System...
[    5.953771] systemd[1]: Starting Journal Service...
[    5.971971] systemd[1]: Starting Set the console keyboard layout...
[    5.991110] systemd[1]: Starting Create List of Static Device Nodes...
[    6.009157] systemd[1]: Starting Load Kernel Module chromeos_pstore...
[    6.027010] systemd[1]: Starting Load Kernel Module configfs...
[    6.045002] systemd[1]: Starting Load Kernel Module drm...
[    6.062840] systemd[1]: Starting Load Kernel Module efi_pstore...
[    6.078901] pstore: backend 'erst' already in use: ignoring 'efi_pstore'
[    6.089584] systemd[1]: Starting Load Kernel Module fuse...
[    6.107282] systemd[1]: Starting Load Kernel Module pstore_blk...
[    6.124829] systemd[1]: Starting Load Kernel Module pstore_zone...
[    6.142369] systemd[1]: Starting Load Kernel Module ramoops...
[    6.159142] systemd[1]: Condition check resulted in Load AppArmor profiles managed internally by snapd being skipped.
[    6.176199] systemd[1]: Condition check resulted in File System Check on Root Device being skipped.
[    6.195468] systemd[1]: Starting Load Kernel Modules...
[    6.208372] ACPI: bus type drm_connector registered
[    6.222646] systemd[1]: Starting Remount Root and Kernel File Systems...
[    6.232038] EXT4-fs (sda2): re-mounted b0358c51-91a1-4785-90fe-f46828ae9f39 r/w. Quota mode: none.
[    6.233340] lp: driver loaded but no devices found
[    6.239261] ppdev: user-space parallel port driver
[    6.276482] systemd[1]: Starting Coldplug All udev Devices...
[    6.296373] systemd[1]: Started Journal Service.
[    6.489629] systemd-journald[536]: Received client request to flush runtime journal.
[    6.606512] loop0: detected capacity change from 0 to 8
[    6.612407] loop1: detected capacity change from 0 to 113944
[    6.621836] loop2: detected capacity change from 0 to 114000
[    6.627725] loop3: detected capacity change from 0 to 129936
[    6.633502] loop4: detected capacity change from 0 to 129944
[    6.640035] loop5: detected capacity change from 0 to 151256
[    6.645830] loop6: detected capacity change from 0 to 151296
[    6.653533] loop7: detected capacity change from 0 to 485808
[    6.659582] loop8: detected capacity change from 0 to 485688
[    6.665259] loop9: detected capacity change from 0 to 716168
[    6.671489] loop10: detected capacity change from 0 to 716176
[    6.677316] loop11: detected capacity change from 0 to 994336
[    6.684021] loop12: detected capacity change from 0 to 994336
[    6.688615] loop13: detected capacity change from 0 to 166424
[    6.693858] loop14: detected capacity change from 0 to 187776
[    6.700092] loop15: detected capacity change from 0 to 94064
[    6.706493] loop16: detected capacity change from 0 to 25240
[    6.711939] loop17: detected capacity change from 0 to 109032
[    6.717565] loop18: detected capacity change from 0 to 109072
[    6.722906] loop19: detected capacity change from 0 to 904
[    6.951336] acpi PNP0C14:00: duplicate WMI GUID 0E7AF9F2-44A1-4C6F-A4B0-A7678480DA61 (first instance was on PNP0C14:00)
[    6.951342] acpi PNP0C14:00: duplicate WMI GUID 0E7AF9F2-44A1-4C6F-A4B0-A7678480DA61 (first instance was on PNP0C14:00)
[    6.976848] ioatdma: Intel(R) QuickData Technology Driver 5.00
[    6.976920] ioatdma 0000:00:04.0: enabling device (0000 -> 0002)
[    6.992782] igb 0000:05:00.0: DCA enabled
[    6.992804] igb 0000:05:00.1: DCA enabled
[    6.995131] ioatdma 0000:00:04.1: enabling device (0000 -> 0002)
[    7.001380] IPMI message handler: version 39.2
[    7.004226] ioatdma 0000:00:04.2: enabling device (0000 -> 0002)
[    7.013847] ioatdma 0000:00:04.3: enabling device (0000 -> 0002)
[    7.019132] ipmi device interface
[    7.023762] ioatdma 0000:00:04.4: enabling device (0000 -> 0002)
[    7.032160] ioatdma 0000:00:04.5: enabling device (0000 -> 0002)
[    7.038078] RAPL PMU: API unit is 2^-32 Joules, 3 fixed counters, 163840 ms ovfl timer
[    7.038085] RAPL PMU: hw unit of domain pp0-core 2^-16 Joules
[    7.038087] RAPL PMU: hw unit of domain package 2^-16 Joules
[    7.038088] RAPL PMU: hw unit of domain dram 2^-16 Joules
[    7.041338] mei_me 0000:00:16.0: Device doesn't have valid ME Interface
[    7.041375] ioatdma 0000:00:04.6: enabling device (0000 -> 0002)
[    7.049682] ioatdma 0000:00:04.7: enabling device (0000 -> 0002)
[    7.057415] ioatdma 0000:80:04.0: enabling device (0000 -> 0002)
[    7.059026] cryptd: max_cpu_qlen set to 1000
[    7.068771] ipmi_si: IPMI System Interface driver
[    7.068797] ipmi_si dmi-ipmi-si.0: ipmi_platform: probing via SMBIOS
[    7.068799] ipmi_platform: ipmi_si: SMBIOS: io 0xca2 regsize 1 spacing 1 irq 0
[    7.068801] ipmi_si: Adding SMBIOS-specified kcs state machine
[    7.068946] ipmi_si: Trying SMBIOS-specified kcs state machine at i/o address 0xca2, slave address 0x20, irq 0
[    7.071193] ioatdma 0000:80:04.1: enabling device (0000 -> 0002)
[    7.077032] AVX version of gcm_enc/dec engaged.
[    7.077141] AES CTR mode by8 optimization enabled
[    7.080321] ioatdma 0000:80:04.2: enabling device (0000 -> 0002)
[    7.088890] ioatdma 0000:80:04.3: enabling device (0000 -> 0002)
[    7.096869] ioatdma 0000:80:04.4: enabling device (0000 -> 0002)
[    7.105302] ioatdma 0000:80:04.5: enabling device (0000 -> 0002)
[    7.113436] ioatdma 0000:80:04.6: enabling device (0000 -> 0002)
[    7.121291] ioatdma 0000:80:04.7: enabling device (0000 -> 0002)
[    7.144528] Console: switching to colour dummy device 80x25
[    7.144563] mgag200 0000:0a:00.0: vgaarb: deactivate vga console
[    7.154116] [drm] Initialized mgag200 1.0.0 20110418 for 0000:0a:00.0 on minor 0
[    7.290299] Adding 67028988k swap on /dev/sda3.  Priority:-2 extents:1 across:67028988k SSFS
[    7.298061] EDID block 0 (tag 0x00) checksum is invalid, remainder is 125
[    7.298067] 	[00] BAD  00 ff ff ff ff ff ff 00 ff ff ff ff ff ff ff ff
[    7.298069] 	[00] BAD  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
[    7.298069] 	[00] BAD  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
[    7.298070] 	[00] BAD  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
[    7.298071] 	[00] BAD  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
[    7.298072] 	[00] BAD  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
[    7.298073] 	[00] BAD  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
[    7.298074] 	[00] BAD  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
[    7.298103] snd_hda_intel 0000:00:1b.0: enabling device (0140 -> 0142)
[    7.299092] fbcon: mgag200drmfb (fb0) is primary device
[    7.355710] snd_hda_codec_realtek hdaudioC0D0: autoconfig for ALC889: line_outs=4 (0x14/0x15/0x16/0x17/0x0) type:line
[    7.355716] snd_hda_codec_realtek hdaudioC0D0:    speaker_outs=0 (0x0/0x0/0x0/0x0/0x0)
[    7.355718] snd_hda_codec_realtek hdaudioC0D0:    hp_outs=1 (0x1b/0x0/0x0/0x0/0x0)
[    7.355719] snd_hda_codec_realtek hdaudioC0D0:    mono: mono_out=0x0
[    7.355721] snd_hda_codec_realtek hdaudioC0D0:    dig-out=0x1e/0x0
[    7.355722] snd_hda_codec_realtek hdaudioC0D0:    inputs:
[    7.355723] snd_hda_codec_realtek hdaudioC0D0:      Front Mic=0x19
[    7.355725] snd_hda_codec_realtek hdaudioC0D0:      Rear Mic=0x18
[    7.355726] snd_hda_codec_realtek hdaudioC0D0:      Line=0x1a
[    7.365073] Console: switching to colour frame buffer device 128x48
[    7.383992] ipmi_si dmi-ipmi-si.0: IPMI message handler: Found new BMC (man_id: 0x000157, prod_id: 0x0056, dev_id: 0x21)
[    7.410623] mgag200 0000:0a:00.0: [drm] fb0: mgag200drmfb frame buffer device
[    7.414325] input: HDA Intel PCH Front Mic as /devices/pci0000:00/0000:00:1b.0/sound/card0/input6
[    7.414436] input: HDA Intel PCH Rear Mic as /devices/pci0000:00/0000:00:1b.0/sound/card0/input7
[    7.414499] input: HDA Intel PCH Line as /devices/pci0000:00/0000:00:1b.0/sound/card0/input8
[    7.414556] input: HDA Intel PCH Line Out Front as /devices/pci0000:00/0000:00:1b.0/sound/card0/input9
[    7.414617] input: HDA Intel PCH Line Out Surround as /devices/pci0000:00/0000:00:1b.0/sound/card0/input10
[    7.414675] input: HDA Intel PCH Line Out CLFE as /devices/pci0000:00/0000:00:1b.0/sound/card0/input11
[    7.414740] input: HDA Intel PCH Line Out Side as /devices/pci0000:00/0000:00:1b.0/sound/card0/input12
[    7.414788] input: HDA Intel PCH Front Headphone as /devices/pci0000:00/0000:00:1b.0/sound/card0/input13
[    7.462893] ipmi_si dmi-ipmi-si.0: IPMI kcs interface initialized
[    7.589254] EDAC sbridge: Seeking for: PCI ID 8086:0ea0
[    7.589282] EDAC sbridge: Seeking for: PCI ID 8086:0ea0
[    7.589291] EDAC sbridge: Seeking for: PCI ID 8086:0ea0
[    7.589295] EDAC sbridge: Seeking for: PCI ID 8086:0e60
[    7.589301] EDAC sbridge: Seeking for: PCI ID 8086:0ea8
[    7.589306] EDAC sbridge: Seeking for: PCI ID 8086:0ea8
[    7.589310] EDAC sbridge: Seeking for: PCI ID 8086:0ea8
[    7.589311] EDAC sbridge: Seeking for: PCI ID 8086:0e71
[    7.589316] EDAC sbridge: Seeking for: PCI ID 8086:0e71
[    7.589320] EDAC sbridge: Seeking for: PCI ID 8086:0e71
[    7.589322] EDAC sbridge: Seeking for: PCI ID 8086:0eaa
[    7.589327] EDAC sbridge: Seeking for: PCI ID 8086:0eaa
[    7.589331] EDAC sbridge: Seeking for: PCI ID 8086:0eaa
[    7.589332] EDAC sbridge: Seeking for: PCI ID 8086:0eab
[    7.589337] EDAC sbridge: Seeking for: PCI ID 8086:0eab
[    7.589341] EDAC sbridge: Seeking for: PCI ID 8086:0eab
[    7.589342] EDAC sbridge: Seeking for: PCI ID 8086:0eac
[    7.589348] EDAC sbridge: Seeking for: PCI ID 8086:0eac
[    7.589351] EDAC sbridge: Seeking for: PCI ID 8086:0eac
[    7.589353] EDAC sbridge: Seeking for: PCI ID 8086:0ead
[    7.589358] EDAC sbridge: Seeking for: PCI ID 8086:0ead
[    7.589362] EDAC sbridge: Seeking for: PCI ID 8086:0ead
[    7.589363] EDAC sbridge: Seeking for: PCI ID 8086:0e68
[    7.589369] EDAC sbridge: Seeking for: PCI ID 8086:0e79
[    7.589374] EDAC sbridge: Seeking for: PCI ID 8086:0e6a
[    7.589380] EDAC sbridge: Seeking for: PCI ID 8086:0e6b
[    7.589386] EDAC sbridge: Seeking for: PCI ID 8086:0e6c
[    7.589391] EDAC sbridge: Seeking for: PCI ID 8086:0e6d
[    7.589397] EDAC sbridge: Seeking for: PCI ID 8086:0eb8
[    7.589403] EDAC sbridge: Seeking for: PCI ID 8086:0ebc
[    7.589408] EDAC sbridge: Seeking for: PCI ID 8086:0ec8
[    7.589414] EDAC sbridge: Seeking for: PCI ID 8086:0ec8
[    7.589417] EDAC sbridge: Seeking for: PCI ID 8086:0ec8
[    7.589418] EDAC sbridge: Seeking for: PCI ID 8086:0ec9
[    7.589423] EDAC sbridge: Seeking for: PCI ID 8086:0ec9
[    7.589427] EDAC sbridge: Seeking for: PCI ID 8086:0ec9
[    7.589428] EDAC sbridge: Seeking for: PCI ID 8086:0eca
[    7.589433] EDAC sbridge: Seeking for: PCI ID 8086:0eca
[    7.589436] EDAC sbridge: Seeking for: PCI ID 8086:0eca
[    7.589525] EDAC MC0: Giving out device to module sb_edac controller Ivy Bridge SrcID#0_Ha#0: DEV 0000:7f:0e.0 (INTERRUPT)
[    7.589586] EDAC MC1: Giving out device to module sb_edac controller Ivy Bridge SrcID#1_Ha#0: DEV 0000:ff:0e.0 (INTERRUPT)
[    7.589588] EDAC sbridge:  Ver: 1.1.2 
[    7.603706] intel_rapl_common: Found RAPL domain package
[    7.603710] intel_rapl_common: Found RAPL domain core
[    7.603714] intel_rapl_common: Found RAPL domain dram
[    7.603721] intel_rapl_common: package-0:core:long_term locked by BIOS
[    7.603948] intel_rapl_common: Found RAPL domain package
[    7.603952] intel_rapl_common: Found RAPL domain core
[    7.603955] intel_rapl_common: Found RAPL domain dram
[    7.603961] intel_rapl_common: package-1:core:long_term locked by BIOS
[    8.903600] loop20: detected capacity change from 0 to 8
[    9.301733] EDID block 0 (tag 0x00) checksum is invalid, remainder is 125
[    9.449499] EDID block 0 (tag 0x00) checksum is invalid, remainder is 125
[    9.872698] igb 0000:05:00.0 eno0: igb: eno0 NIC Link is Up 100 Mbps Full Duplex, Flow Control: RX
[   10.788020] EDID block 0 (tag 0x00) checksum is invalid, remainder is 125
[   12.352885] rfkill: input handler disabled

Re: [patch 00/53] x86/topology: The final installment

[Thomas Gleixner]

On Tue, Aug 08 2023 at 13:30, Sohil Mehta wrote:
> There are compile issues with the ACPI hunk.
>
>> arch/x86/kernel/acpi/boot.c: In function ‘acpi_parse_lapic’:
>> arch/x86/kernel/acpi/boot.c:250:31: error: ‘apic_id’ undeclared (first use in this function)
>>   250 |         if (has_lapic_cpus && apic_id < 0xff)
>>       |                               ^~~~~~~
>> arch/x86/kernel/acpi/boot.c:250:31: note: each undeclared identifier is reported only once for each function it appears in
>> arch/x86/kernel/acpi/boot.c: In function ‘acpi_parse_madt_lapic_entries’:
>> arch/x86/kernel/acpi/boot.c:1097:47: error: ‘ACPI_MADT_TYPE_LOCAL_LAPIC’ undeclared (first use in this function); did you mean ‘ACPI_MADT_TYPE_LOCAL_SAPIC’?
>>  1097 |                 count = acpi_table_parse_madt(ACPI_MADT_TYPE_LOCAL_LAPIC,
>>       |                                               ^~~~~~~~~~~~~~~~~~~~~~~~~~
>>       |                                               ACPI_MADT_TYPE_LOCAL_SAPIC
>

Duh. Yes. I just picked the thing from the list and hacked it
up. Compilable variant below.

Thanks,

        tglx
---
 arch/x86/kernel/acpi/boot.c    |   35 +++++++++++++++++------------------
 arch/x86/kernel/cpu/topology.c |    2 ++
 2 files changed, 19 insertions(+), 18 deletions(-)

--- a/arch/x86/kernel/acpi/boot.c
+++ b/arch/x86/kernel/acpi/boot.c
@@ -171,6 +171,8 @@ static bool __init acpi_is_processor_usa
 	return false;
 }
 
+static bool has_lapic_cpus;
+
 static int __init
 acpi_parse_x2apic(union acpi_subtable_headers *header, const unsigned long end)
 {
@@ -241,6 +243,14 @@ acpi_parse_lapic(union acpi_subtable_hea
 		return 0;
 
 	/*
+	 * According to https://uefi.org/specs/ACPI/6.5/05_ACPI_Software_Programming_Model.html#processor-local-x2apic-structure
+	 * when MADT provides both valid LAPIC and x2APIC entries, the APIC ID
+	 * in x2APIC must be equal or greater than 0xff.
+	 */
+	if (has_lapic_cpus && processor->id < 0xff)
+		return 0;
+
+	/*
 	 * We need to register disabled CPU as well to permit
 	 * counting disabled CPUs. This allows us to size
 	 * cpus_possible_map more accurately, to permit
@@ -1072,10 +1082,8 @@ static int __init early_acpi_parse_madt_
 
 static int __init acpi_parse_madt_lapic_entries(void)
 {
-	int count;
 	int x2count = 0;
-	int ret;
-	struct acpi_subtable_proc madt_proc[2];
+	int count;
 
 	if (!boot_cpu_has(X86_FEATURE_APIC))
 		return -ENODEV;
@@ -1084,21 +1092,12 @@ static int __init acpi_parse_madt_lapic_
 				      acpi_parse_sapic, MAX_LOCAL_APIC);
 
 	if (!count) {
-		memset(madt_proc, 0, sizeof(madt_proc));
-		madt_proc[0].id = ACPI_MADT_TYPE_LOCAL_APIC;
-		madt_proc[0].handler = acpi_parse_lapic;
-		madt_proc[1].id = ACPI_MADT_TYPE_LOCAL_X2APIC;
-		madt_proc[1].handler = acpi_parse_x2apic;
-		ret = acpi_table_parse_entries_array(ACPI_SIG_MADT,
-				sizeof(struct acpi_table_madt),
-				madt_proc, ARRAY_SIZE(madt_proc), MAX_LOCAL_APIC);
-		if (ret < 0) {
-			pr_err("Error parsing LAPIC/X2APIC entries\n");
-			return ret;
-		}
-
-		count = madt_proc[0].count;
-		x2count = madt_proc[1].count;
+		count = acpi_table_parse_madt(ACPI_MADT_TYPE_LOCAL_APIC,
+					      acpi_parse_lapic, MAX_LOCAL_APIC);
+		if (count > 0)
+			has_lapic_cpus = true;
+		x2count = acpi_table_parse_madt(ACPI_MADT_TYPE_LOCAL_X2APIC,
+						acpi_parse_x2apic, MAX_LOCAL_APIC);
 	}
 	if (!count && !x2count) {
 		pr_err("No LAPIC entries present\n");

Re: [patch 00/53] x86/topology: The final installment

[Thomas Gleixner]

On Tue, Aug 08 2023 at 13:30, Sohil Mehta wrote:
> On 8/8/2023 12:10 PM, Thomas Gleixner wrote:

> domain: Thread     shift: 1 dom_size:     2 max_threads:     2
> domain: Core       shift: 5 dom_size:    16 max_threads:    32
> domain: Module     shift: 5 dom_size:     1 max_threads:    32
> domain: Tile       shift: 5 dom_size:     1 max_threads:    32
> domain: Die        shift: 5 dom_size:     1 max_threads:    32
> domain: Package    shift: 5 dom_size:     1 max_threads:    32
>
> CPU 0:
>    0x0000000b 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000000
>    0x0000000b 0x01: eax=0x00000005 ebx=0x00000014 ecx=0x00000201 edx=0x00000000

Ok. So this is consistent.

> Also, I see a warning message that only seems to show up with the final
> installment series applied. I attached the complete dmesg as well (just
> in case):
>
> unchecked MSR access error: WRMSR to 0xe44 (tried to write
> 0x0000000000000003) at rIP: 0xffffffff8d2a6698 (native_write_msr+0x8/0x30)
>  uncore_box_ref.part.0+0xa6/0xe0
>  uncore_event_cpu_online+0x6e/0x1c0
>  ? __pfx_uncore_event_cpu_online+0x10/0x10
>  cpuhp_invoke_callback+0x165/0x4b0

That's probably a consequence of the inconsistency.

> [    0.187210] CPU topo: Register 000 1
> [    0.187211] CPU topo: Register 002 1
> [    0.187212] CPU topo: Register 004 1
> [    0.187213] CPU topo: Register 006 1
> [    0.187214] CPU topo: Register 008 1
> [    0.187215] CPU topo: Register 010 1
> [    0.187216] CPU topo: Register 012 1
> [    0.187217] CPU topo: Register 014 1
> [    0.187218] CPU topo: Register 016 1
> [    0.187219] CPU topo: Register 018 1

The first package (primary threads)

> [    0.187219] CPU topo: Register 020 1
> [    0.187220] CPU topo: Register 022 1
> [    0.187221] CPU topo: Register 024 1
> [    0.187222] CPU topo: Register 026 1
> [    0.187223] CPU topo: Register 028 1
> [    0.187223] CPU topo: Register 030 1
> [    0.187224] CPU topo: Register 032 1
> [    0.187225] CPU topo: Register 034 1
> [    0.187226] CPU topo: Register 036 1
> [    0.187227] CPU topo: Register 038 1

The second package (primary threads)

> [    0.187228] CPU topo: Register 001 1
> [    0.187228] CPU topo: Register 003 1
> [    0.187229] CPU topo: Register 005 1
> [    0.187230] CPU topo: Register 007 1
> [    0.187230] CPU topo: Register 009 1
> [    0.187231] CPU topo: Register 011 1
> [    0.187232] CPU topo: Register 013 1
> [    0.187233] CPU topo: Register 015 1
> [    0.187233] CPU topo: Register 017 1
> [    0.187234] CPU topo: Register 019 1

The second package (secondary threads)

> [    0.187235] CPU topo: Register 021 1
> [    0.187235] CPU topo: Register 023 1
> [    0.187236] CPU topo: Register 025 1
> [    0.187237] CPU topo: Register 027 1
> [    0.187238] CPU topo: Register 029 1
> [    0.187238] CPU topo: Register 031 1
> [    0.187239] CPU topo: Register 033 1
> [    0.187240] CPU topo: Register 035 1
> [    0.187241] CPU topo: Register 037 1
> [    0.187241] CPU topo: Register 039 1

The second package (secondary threads)

> [    0.187244] CPU topo: Register 000 0
> [    0.187244] CPU topo: Register 001 0

... PKG 0

> [    0.187266] CPU topo: Register 01e 0
> [    0.187267] CPU topo: Register 01f 0

Ah. that's indeed the issue which the ACPI patch addresses. So that
table claims that the packages are truly filled up to capacity, i.e. 32
threads. The old code did not notice because they are all marked
non-present, but with the new approach these are rightfully accounted as
pluggable and show up in the bitmaps accordingly. Sigh...

> [    0.187268] CPU topo: Register 020 0
... PKG 1
> [    0.187291] CPU topo: Register 03f 0

> [    0.187292] CPU topo: Register 040 0
... PKG 2
> [    0.187304] CPU topo: Register 05f 0

> [    0.187305] CPU topo: Register 060 0
... PKG 3
> [    0.187335] CPU topo: Register 077 0

This one is funny as it stops at 0x77, i.e 8 CPUs short of the full
range.

So this:

> [    0.187412] CPU topo: Max. logical packages:   4

_IS_ correct according to the above.

I bet that the ACPI patch cures it.

Thanks,

        tglx

Re: [patch 00/53] x86/topology: The final installment

[Peter Zijlstra]

On Tue, Aug 08, 2023 at 10:41:51PM +0200, Thomas Gleixner wrote:

> --- a/arch/x86/kernel/acpi/boot.c
> +++ b/arch/x86/kernel/acpi/boot.c
> @@ -171,6 +171,8 @@ static bool __init acpi_is_processor_usa
>  	return false;
>  }
>  
> +static bool has_lapic_cpus;
> +
>  static int __init
>  acpi_parse_x2apic(union acpi_subtable_headers *header, const unsigned long end)
>  {
> @@ -241,6 +243,14 @@ acpi_parse_lapic(union acpi_subtable_hea
>  		return 0;
>  
>  	/*
> +	 * According to https://uefi.org/specs/ACPI/6.5/05_ACPI_Software_Programming_Model.html#processor-local-x2apic-structure
> +	 * when MADT provides both valid LAPIC and x2APIC entries, the APIC ID
> +	 * in x2APIC must be equal or greater than 0xff.
> +	 */
> +	if (has_lapic_cpus && processor->id < 0xff)
> +		return 0;
> +
> +	/*
>  	 * We need to register disabled CPU as well to permit
>  	 * counting disabled CPUs. This allows us to size
>  	 * cpus_possible_map more accurately, to permit

It works better if you move this hunk into acpi_parse_x2apic() instead.
Then I can indeed confirm it works as advertised -- also having one of
them afflicted ivb-ep machines.

Tested-by: Peter Zijlstra (Intel) <peterz@infradead.org>

---
diff --git a/arch/x86/kernel/acpi/boot.c b/arch/x86/kernel/acpi/boot.c
index 088323ed6179..f6cff99d6087 100644
--- a/arch/x86/kernel/acpi/boot.c
+++ b/arch/x86/kernel/acpi/boot.c
@@ -171,6 +171,8 @@ static bool __init acpi_is_processor_usable(u32 lapic_flags)
 	return false;
 }
 
+static bool has_lapic_cpus;
+
 static int __init
 acpi_parse_x2apic(union acpi_subtable_headers *header, const unsigned long end)
 {
@@ -199,6 +201,14 @@ acpi_parse_x2apic(union acpi_subtable_headers *header, const unsigned long end)
 	if (!acpi_is_processor_usable(processor->lapic_flags))
 		return 0;
 
+	/*
+	 * According to https://uefi.org/specs/ACPI/6.5/05_ACPI_Software_Programming_Model.html#processor-local-x2apic-structure
+	 * when MADT provides both valid LAPIC and x2APIC entries, the APIC ID
+	 * in x2APIC must be equal or greater than 0xff.
+	 */
+	if (has_lapic_cpus && apic_id < 0xff)
+		return 0;
+
 	/*
 	 * We need to register disabled CPU as well to permit
 	 * counting disabled CPUs. This allows us to size
@@ -1072,10 +1083,8 @@ static int __init early_acpi_parse_madt_lapic_addr_ovr(void)
 
 static int __init acpi_parse_madt_lapic_entries(void)
 {
-	int count;
 	int x2count = 0;
-	int ret;
-	struct acpi_subtable_proc madt_proc[2];
+	int count;
 
 	if (!boot_cpu_has(X86_FEATURE_APIC))
 		return -ENODEV;
@@ -1084,21 +1093,12 @@ static int __init acpi_parse_madt_lapic_entries(void)
 				      acpi_parse_sapic, MAX_LOCAL_APIC);
 
 	if (!count) {
-		memset(madt_proc, 0, sizeof(madt_proc));
-		madt_proc[0].id = ACPI_MADT_TYPE_LOCAL_APIC;
-		madt_proc[0].handler = acpi_parse_lapic;
-		madt_proc[1].id = ACPI_MADT_TYPE_LOCAL_X2APIC;
-		madt_proc[1].handler = acpi_parse_x2apic;
-		ret = acpi_table_parse_entries_array(ACPI_SIG_MADT,
-				sizeof(struct acpi_table_madt),
-				madt_proc, ARRAY_SIZE(madt_proc), MAX_LOCAL_APIC);
-		if (ret < 0) {
-			pr_err("Error parsing LAPIC/X2APIC entries\n");
-			return ret;
-		}
-
-		count = madt_proc[0].count;
-		x2count = madt_proc[1].count;
+		count = acpi_table_parse_madt(ACPI_MADT_TYPE_LOCAL_APIC,
+					      acpi_parse_lapic, MAX_LOCAL_APIC);
+		if (count > 0)
+			has_lapic_cpus = true;
+		x2count = acpi_table_parse_madt(ACPI_MADT_TYPE_LOCAL_X2APIC,
+						acpi_parse_x2apic, MAX_LOCAL_APIC);
 	}
 	if (!count && !x2count) {
 		pr_err("No LAPIC entries present\n");

Re: [patch 00/53] x86/topology: The final installment

[Sohil Mehta]

On 8/8/2023 3:10 PM, Peter Zijlstra wrote:
> It works better if you move this hunk into acpi_parse_x2apic() instead.
> Then I can indeed confirm it works as advertised -- also having one of
> them afflicted ivb-ep machines.
> 

I had a disappointed email typed up and was about to send it when I saw
this.

The inconsistency and warning on my system resolves with this. I lost
120 imaginary hotpluggable cpus but other than that everything seems fine :)

CPU topo: Max. logical packages:   2
CPU topo: Max. logical dies:       2
CPU topo: Max. dies per package:   1
CPU topo: Max. threads per core:   2
CPU topo: Num. cores per package:    10
CPU topo: Num. threads per package:  20
CPU topo: Allowing 40 present CPUs plus 0 hotplug CPUs
CPU topo: Thread    :    40
CPU topo: Core      :    20
CPU topo: Module    :     2
CPU topo: Tile      :     2
CPU topo: Die       :     2
CPU topo: Package   :     2

domain: Thread     shift: 1 dom_size:     2 max_threads:     2
domain: Core       shift: 5 dom_size:    16 max_threads:    32
domain: Module     shift: 5 dom_size:     1 max_threads:    32
domain: Tile       shift: 5 dom_size:     1 max_threads:    32
domain: Die        shift: 5 dom_size:     1 max_threads:    32
domain: Package    shift: 5 dom_size:     1 max_threads:    32

/sys/kernel/debug/x86/topo/cpus/39
online:              1
initial_apicid:      39
apicid:              39
pkg_id:              1
die_id:              1
cu_id:               255
core_id:             12
logical_pkg_id:      1
logical_die_id:      1
llc_id:              32
l2c_id:              56
amd_node_id:         0
amd_nodes_per_pkg:   0
num_threads:         20
num_cores:           10
max_dies_per_pkg:    1
max_threads_per_core:2

Sohil

Re: [patch 00/53] x86/topology: The final installment

[Thomas Gleixner]

On Tue, Aug 08 2023 at 15:58, Sohil Mehta wrote:

> On 8/8/2023 3:10 PM, Peter Zijlstra wrote:
>> It works better if you move this hunk into acpi_parse_x2apic() instead.
>> Then I can indeed confirm it works as advertised -- also having one of
>> them afflicted ivb-ep machines.
>> 
>
> I had a disappointed email typed up

Rightfully though as I'm clearly too tired and too grumpy to think
straight.
 
> and was about to send it when I saw this.

:)

> The inconsistency and warning on my system resolves with this. I lost
> 120 imaginary hotpluggable cpus but other than that everything seems
> fine :)

Sorry about that loss. :)

> CPU topo: Max. logical packages:   2
> CPU topo: Max. logical dies:       2
> CPU topo: Max. dies per package:   1
> CPU topo: Max. threads per core:   2
> CPU topo: Num. cores per package:    10
> CPU topo: Num. threads per package:  20
> CPU topo: Allowing 40 present CPUs plus 0 hotplug CPUs
> CPU topo: Thread    :    40
> CPU topo: Core      :    20
> CPU topo: Module    :     2
> CPU topo: Tile      :     2
> CPU topo: Die       :     2
> CPU topo: Package   :     2
>
> domain: Thread     shift: 1 dom_size:     2 max_threads:     2
> domain: Core       shift: 5 dom_size:    16 max_threads:    32
> domain: Module     shift: 5 dom_size:     1 max_threads:    32
> domain: Tile       shift: 5 dom_size:     1 max_threads:    32
> domain: Die        shift: 5 dom_size:     1 max_threads:    32
> domain: Package    shift: 5 dom_size:     1 max_threads:    32
>
> /sys/kernel/debug/x86/topo/cpus/39
> online:              1
> initial_apicid:      39
> apicid:              39
> pkg_id:              1
> die_id:              1
> cu_id:               255
> core_id:             12
> logical_pkg_id:      1
> logical_die_id:      1
> llc_id:              32
> l2c_id:              56
> amd_node_id:         0
> amd_nodes_per_pkg:   0
> num_threads:         20
> num_cores:           10
> max_dies_per_pkg:    1
> max_threads_per_core:2

That makes much more sense now.

Zhang, can you please follow up on:

  https://lore.kernel.org/all/613df280116378115585d0c483f7e186cffaeb58.camel@intel.com/

or otherwise I just polish up PeterZ's variant of it tomorrow.

Thanks,

        tglx

Re: [patch 00/53] x86/topology: The final installment

[Qiuxu Zhuo]

Hi Thomas,

> From: Thomas Gleixner <tglx@linutronix.de>
> ...
> Subject: [patch 00/53] x86/topology: The final installment
> ...
> 
> The current series applies on top of 
> 
>    git://git.kernel.org/pub/scm/linux/kernel/git/tglx/devel.git topo-cpuid-v3
> 
> and is available from git here:
> 
>    git://git.kernel.org/pub/scm/linux/kernel/git/tglx/devel.git topo-full-v1

Test Machine
------------
I tested the 'topo-full-v1' branch on a Sapphire Rapids server with 2 sockets,
each containing 48 cores, resulting in a total of 192 threads.


Test Results
------------
The following test results (same result either w/ or w/o [1]) show that
this branch worked well on this server. Based on the test results,
  
  Tested-by: Qiuxu Zhuo <qiuxu.zhuo@intel.com>

[1] https://lore.kernel.org/all/20230808221039.GH212435@hirez.programming.kicks-ass.net/


Logs (hyper-thread enabled)
---------------------------
1.1 dmesg | grep "CPU topo" :

    CPU topo: Max. logical packages:   2
    CPU topo: Max. logical dies:       2
    CPU topo: Max. dies per package:   1
    CPU topo: Max. threads per core:   2
    CPU topo: Num. cores per package:    48
    CPU topo: Num. threads per package:  96
    CPU topo: Allowing 192 present CPUs plus 0 hotplug CPUs
    CPU topo: Thread    :   192
    CPU topo: Core      :    96
    CPU topo: Module    :     2
    CPU topo: Tile      :     2
    CPU topo: Die       :     2
    CPU topo: Package   :     2

1.2 cat /sys/kernel/debug/x86/topo/domains :

    domain: Thread     shift: 1 dom_size:     2 max_threads:     2
    domain: Core       shift: 7 dom_size:    64 max_threads:   128
    domain: Module     shift: 7 dom_size:     1 max_threads:   128
    domain: Tile       shift: 7 dom_size:     1 max_threads:   128
    domain: Die        shift: 7 dom_size:     1 max_threads:   128
    domain: Package    shift: 7 dom_size:     1 max_threads:   128

1.3 <1st socket, 1st core, primary thread>
    cat /sys/kernel/debug/x86/topo/cpus/0
    online:              1
    initial_apicid:      0
    apicid:              0
    pkg_id:              0
    die_id:              0
    cu_id:               255
    core_id:             0
    logical_pkg_id:      0
    logical_die_id:      0
    llc_id:              0
    l2c_id:              0
    amd_node_id:         0
    amd_nodes_per_pkg:   0
    num_threads:         96
    num_cores:           48
    max_dies_per_pkg:    1
    max_threads_per_core:2

    <1st socket, last core, primary thread>
    cat /sys/kernel/debug/x86/topo/cpus/47
    online:              1
    initial_apicid:      5e
    apicid:              5e
    pkg_id:              0
    die_id:              0
    cu_id:               255
    core_id:             47
    logical_pkg_id:      0
    logical_die_id:      0
    llc_id:              0
    l2c_id:              94
    amd_node_id:         0
    amd_nodes_per_pkg:   0
    num_threads:         96
    num_cores:           48
    max_dies_per_pkg:    1
    max_threads_per_core:2
    
    <2nd socket, 1st core, primary thread>
    cat /sys/kernel/debug/x86/topo/cpus/48
    online:              1
    initial_apicid:      80
    apicid:              80
    pkg_id:              1
    die_id:              1
    cu_id:               255
    core_id:             0
    logical_pkg_id:      1
    logical_die_id:      1
    llc_id:              128
    l2c_id:              128
    amd_node_id:         0
    amd_nodes_per_pkg:   0
    num_threads:         96
    num_cores:           48
    max_dies_per_pkg:    1
    max_threads_per_core:2
    
    <2nd socket, last core, primary thread>
    cat /sys/kernel/debug/x86/topo/cpus/95
    online:              1
    initial_apicid:      de
    apicid:              de
    pkg_id:              1
    die_id:              1
    cu_id:               255
    core_id:             47
    logical_pkg_id:      1
    logical_die_id:      1
    llc_id:              128
    l2c_id:              222
    amd_node_id:         0
    amd_nodes_per_pkg:   0
    num_threads:         96
    num_cores:           48
    max_dies_per_pkg:    1
    max_threads_per_core:2

    <1st socket, 1st core, secondary thread>
    cat /sys/kernel/debug/x86/topo/cpus/96
    online:              1
    initial_apicid:      1
    apicid:              1
    pkg_id:              0
    die_id:              0
    cu_id:               255
    core_id:             0
    logical_pkg_id:      0
    logical_die_id:      0
    llc_id:              0
    l2c_id:              0
    amd_node_id:         0
    amd_nodes_per_pkg:   0
    num_threads:         96
    num_cores:           48
    max_dies_per_pkg:    1
    max_threads_per_core:2
    
    <1st socket, last core, secondary thread>
    cat /sys/kernel/debug/x86/topo/cpus/143
    online:              1
    initial_apicid:      5f
    apicid:              5f
    pkg_id:              0
    die_id:              0
    cu_id:               255
    core_id:             47
    logical_pkg_id:      0
    logical_die_id:      0
    llc_id:              0
    l2c_id:              94
    amd_node_id:         0
    amd_nodes_per_pkg:   0
    num_threads:         96
    num_cores:           48
    max_dies_per_pkg:    1
    max_threads_per_core:2

    <2nd socket, 1st core, secondary thread>
    cat /sys/kernel/debug/x86/topo/cpus/144
    online:              1
    initial_apicid:      81
    apicid:              81
    pkg_id:              1
    die_id:              1
    cu_id:               255
    core_id:             0
    logical_pkg_id:      1
    logical_die_id:      1
    llc_id:              128
    l2c_id:              128
    amd_node_id:         0
    amd_nodes_per_pkg:   0
    num_threads:         96
    num_cores:           48
    max_dies_per_pkg:    1
    max_threads_per_core:2
    
    <2nd socket, last core, secondary thread>
    cat /sys/kernel/debug/x86/topo/cpus/191
    online:              1
    initial_apicid:      df
    apicid:              df
    pkg_id:              1
    die_id:              1
    cu_id:               255
    core_id:             47
    logical_pkg_id:      1
    logical_die_id:      1
    llc_id:              128
    l2c_id:              222
    amd_node_id:         0
    amd_nodes_per_pkg:   0
    num_threads:         96
    num_cores:           48
    max_dies_per_pkg:    1
    max_threads_per_core:2

Logs (hyper-thread disabled)
----------------------------
2.1 dmesg | grep "CPU topo" :

    CPU topo: Max. logical packages:   2
    CPU topo: Max. logical dies:       2
    CPU topo: Max. dies per package:   1
    CPU topo: Max. threads per core:   1
    CPU topo: Num. cores per package:    48
    CPU topo: Num. threads per package:  48
    CPU topo: Allowing 96 present CPUs plus 0 hotplug CPUs
    CPU topo: Thread    :    96
    CPU topo: Core      :    96
    CPU topo: Module    :     2
    CPU topo: Tile      :     2
    CPU topo: Die       :     2
    CPU topo: Package   :     2

2.2 cat /sys/kernel/debug/x86/topo/domains :

    domain: Thread     shift: 1 dom_size:     2 max_threads:     2
    domain: Core       shift: 7 dom_size:    64 max_threads:   128
    domain: Module     shift: 7 dom_size:     1 max_threads:   128
    domain: Tile       shift: 7 dom_size:     1 max_threads:   128
    domain: Die        shift: 7 dom_size:     1 max_threads:   128
    domain: Package    shift: 7 dom_size:     1 max_threads:   128

2.3 <1st socket, 1st core>
    cat /sys/kernel/debug/x86/topo/cpus/0
    online:              1
    initial_apicid:      0
    apicid:              0
    pkg_id:              0
    die_id:              0
    cu_id:               255
    core_id:             0
    logical_pkg_id:      0
    logical_die_id:      0
    llc_id:              0
    l2c_id:              0
    amd_node_id:         0
    amd_nodes_per_pkg:   0
    num_threads:         48
    num_cores:           48
    max_dies_per_pkg:    1
    max_threads_per_core:1

    <1st socket, last core>
    cat /sys/kernel/debug/x86/topo/cpus/47
    online:              1
    initial_apicid:      5e
    apicid:              5e
    pkg_id:              0
    die_id:              0
    cu_id:               255
    core_id:             47
    logical_pkg_id:      0
    logical_die_id:      0
    llc_id:              0
    l2c_id:              94
    amd_node_id:         0
    amd_nodes_per_pkg:   0
    num_threads:         48
    num_cores:           48
    max_dies_per_pkg:    1
    max_threads_per_core:1

    <2nd socket, 1st core>
    cat /sys/kernel/debug/x86/topo/cpus/48
    online:              1
    initial_apicid:      80
    apicid:              80
    pkg_id:              1
    die_id:              1
    cu_id:               255
    core_id:             0
    logical_pkg_id:      1
    logical_die_id:      1
    llc_id:              128
    l2c_id:              128
    amd_node_id:         0
    amd_nodes_per_pkg:   0
    num_threads:         48
    num_cores:           48
    max_dies_per_pkg:    1
    max_threads_per_core:1

    <2nd socket, last core>
    cat /sys/kernel/debug/x86/topo/cpus/95
    online:              1
    initial_apicid:      de
    apicid:              de
    pkg_id:              1
    die_id:              1
    cu_id:               255
    core_id:             47
    logical_pkg_id:      1
    logical_die_id:      1
    llc_id:              128
    l2c_id:              222
    amd_node_id:         0
    amd_nodes_per_pkg:   0
    num_threads:         48
    num_cores:           48
    max_dies_per_pkg:    1
    max_threads_per_core:1

-Qiuxu

Re: [patch 00/53] x86/topology: The final installment

[Qiuxu Zhuo]

Hi Sohil,

> From: Sohil Mehta <sohil.mehta@intel.com>
> ...
> Subject: Re: [patch 00/53] x86/topology: The final installment
> ...
> Debugfs
> -------
> # cat /sys/kernel/debug/x86/topo/cpus/39
> online:              1
> initial_apicid:      39
> apicid:              39
> ...

Did you convert the output formats of 'initial_apicid' and 'apicid'
from hexadecimal to decimal in your testing? It was really coincidental
that if the output values were '39' in hexadecimal for the CPU '39'
in decimal :-).

I noticed they were represented in hexadecimal format in
arch/x86/kernel/cpu/debugfs.c:

    seq_printf(m, "initial_apicid:      %x\n", c->topo.initial_apicid);
    seq_printf(m, "apicid:              %x\n", c->topo.apicid);

Thanks!
-Qiuxu

Re: [patch 00/53] x86/topology: The final installment

[Sohil Mehta]

On 8/8/2023 4:20 PM, Thomas Gleixner wrote:
> That makes much more sense now.
> 
> Zhang, can you please follow up on:
> 
>   https://lore.kernel.org/all/613df280116378115585d0c483f7e186cffaeb58.camel@intel.com/
> 
> or otherwise I just polish up PeterZ's variant of it tomorrow.
> 
> Thanks,
> 
>         tglx

For the full series (with PeterZ's patch applied),

Tested-by: Sohil Mehta <sohil.mehta@intel.com>

Zhang, Thomas, please let me know if there is a different variant of the
patch that needs to be tested.

Sohil

Re: [patch 00/53] x86/topology: The final installment

[Sohil Mehta]

Hi Qiuxu,

On 8/9/2023 9:50 AM, Qiuxu Zhuo wrote:
>> Debugfs
>> -------
>> # cat /sys/kernel/debug/x86/topo/cpus/39
>> online:              1
>> initial_apicid:      39
>> apicid:              39
>> ...
> 
> Did you convert the output formats of 'initial_apicid' and 'apicid'
> from hexadecimal to decimal in your testing? It was really coincidental
> that if the output values were '39' in hexadecimal for the CPU '39'
> in decimal :-).
> 

I didn't convert the output formats. That is purely coincidental or some
trick by firmware developers to avoid alphabets in the hexadecimal APIC IDs.

I have now modified the print as shown below which generates the
following output:

> pr_info("Register 0x%03x as CPU %d is_present %d\n", apic_id, cpu, present);


CPU topo: Register 0x000 as CPU 0 is_present 1
CPU topo: Register 0x002 as CPU 1 is_present 1
CPU topo: Register 0x004 as CPU 2 is_present 1
CPU topo: Register 0x006 as CPU 3 is_present 1
CPU topo: Register 0x008 as CPU 4 is_present 1
CPU topo: Register 0x010 as CPU 5 is_present 1
CPU topo: Register 0x012 as CPU 6 is_present 1
CPU topo: Register 0x014 as CPU 7 is_present 1
CPU topo: Register 0x016 as CPU 8 is_present 1
CPU topo: Register 0x018 as CPU 9 is_present 1
CPU topo: Register 0x020 as CPU 10 is_present 1
CPU topo: Register 0x022 as CPU 11 is_present 1
CPU topo: Register 0x024 as CPU 12 is_present 1
CPU topo: Register 0x026 as CPU 13 is_present 1
CPU topo: Register 0x028 as CPU 14 is_present 1
CPU topo: Register 0x030 as CPU 15 is_present 1
CPU topo: Register 0x032 as CPU 16 is_present 1
CPU topo: Register 0x034 as CPU 17 is_present 1
CPU topo: Register 0x036 as CPU 18 is_present 1
CPU topo: Register 0x038 as CPU 19 is_present 1
CPU topo: Register 0x001 as CPU 20 is_present 1
CPU topo: Register 0x003 as CPU 21 is_present 1
CPU topo: Register 0x005 as CPU 22 is_present 1
CPU topo: Register 0x007 as CPU 23 is_present 1
CPU topo: Register 0x009 as CPU 24 is_present 1
CPU topo: Register 0x011 as CPU 25 is_present 1
CPU topo: Register 0x013 as CPU 26 is_present 1
CPU topo: Register 0x015 as CPU 27 is_present 1
CPU topo: Register 0x017 as CPU 28 is_present 1
CPU topo: Register 0x019 as CPU 29 is_present 1
CPU topo: Register 0x021 as CPU 30 is_present 1
CPU topo: Register 0x023 as CPU 31 is_present 1
CPU topo: Register 0x025 as CPU 32 is_present 1
CPU topo: Register 0x027 as CPU 33 is_present 1
CPU topo: Register 0x029 as CPU 34 is_present 1
CPU topo: Register 0x031 as CPU 35 is_present 1
CPU topo: Register 0x033 as CPU 36 is_present 1
CPU topo: Register 0x035 as CPU 37 is_present 1
CPU topo: Register 0x037 as CPU 38 is_present 1
CPU topo: Register 0x039 as CPU 39 is_present 1

Sohil

RE: [patch 00/53] x86/topology: The final installment

[Zhuo, Qiuxu]

> From: Mehta, Sohil <sohil.mehta@intel.com>
>...
>I didn't convert the output formats. That is purely coincidental or some
>trick by firmware developers to avoid alphabets in the hexadecimal APIC IDs.

From your new log below, looks like this is indeed the case.

>I have now modified the print as shown below which generates the
>following output:
>
>> pr_info("Register 0x%03x as CPU %d is_present %d\n", apic_id, cpu, present);
>
>
>CPU topo: Register 0x000 as CPU 0 is_present 1
>CPU topo: Register 0x002 as CPU 1 is_present 1
>CPU topo: Register 0x004 as CPU 2 is_present 1
>CPU topo: Register 0x006 as CPU 3 is_present 1
>CPU topo: Register 0x008 as CPU 4 is_present 1
>CPU topo: Register 0x010 as CPU 5 is_present 1
>CPU topo: Register 0x012 as CPU 6 is_present 1
>CPU topo: Register 0x014 as CPU 7 is_present 1
>CPU topo: Register 0x016 as CPU 8 is_present 1
>CPU topo: Register 0x018 as CPU 9 is_present 1
>CPU topo: Register 0x020 as CPU 10 is_present 1
>CPU topo: Register 0x022 as CPU 11 is_present 1
>CPU topo: Register 0x024 as CPU 12 is_present 1
>CPU topo: Register 0x026 as CPU 13 is_present 1
>CPU topo: Register 0x028 as CPU 14 is_present 1
>CPU topo: Register 0x030 as CPU 15 is_present 1
>CPU topo: Register 0x032 as CPU 16 is_present 1
>CPU topo: Register 0x034 as CPU 17 is_present 1
>CPU topo: Register 0x036 as CPU 18 is_present 1
>CPU topo: Register 0x038 as CPU 19 is_present 1
>CPU topo: Register 0x001 as CPU 20 is_present 1
>CPU topo: Register 0x003 as CPU 21 is_present 1
>CPU topo: Register 0x005 as CPU 22 is_present 1
>CPU topo: Register 0x007 as CPU 23 is_present 1
>CPU topo: Register 0x009 as CPU 24 is_present 1
>CPU topo: Register 0x011 as CPU 25 is_present 1
>CPU topo: Register 0x013 as CPU 26 is_present 1
>CPU topo: Register 0x015 as CPU 27 is_present 1
>CPU topo: Register 0x017 as CPU 28 is_present 1
>CPU topo: Register 0x019 as CPU 29 is_present 1
>CPU topo: Register 0x021 as CPU 30 is_present 1
>CPU topo: Register 0x023 as CPU 31 is_present 1
>CPU topo: Register 0x025 as CPU 32 is_present 1
>CPU topo: Register 0x027 as CPU 33 is_present 1
>CPU topo: Register 0x029 as CPU 34 is_present 1
>CPU topo: Register 0x031 as CPU 35 is_present 1
>CPU topo: Register 0x033 as CPU 36 is_present 1
>CPU topo: Register 0x035 as CPU 37 is_present 1
>CPU topo: Register 0x037 as CPU 38 is_present 1
>CPU topo: Register 0x039 as CPU 39 is_present 1

Thank you for sharing the output from your machine.
Interesting :-).

-Qiuxu

Re: [patch 00/53] x86/topology: The final installment

[Zhang, Rui]

Hi, Thomas,

> > CPU topo: Max. logical packages:   2
> > CPU topo: Max. logical dies:       2
> > CPU topo: Max. dies per package:   1
> > CPU topo: Max. threads per core:   2
> > CPU topo: Num. cores per package:    10
> > CPU topo: Num. threads per package:  20
> > CPU topo: Allowing 40 present CPUs plus 0 hotplug CPUs
> > CPU topo: Thread    :    40
> > CPU topo: Core      :    20
> > CPU topo: Module    :     2
> > CPU topo: Tile      :     2
> > CPU topo: Die       :     2
> > CPU topo: Package   :     2
> > 
> > domain: Thread     shift: 1 dom_size:     2 max_threads:     2
> > domain: Core       shift: 5 dom_size:    16 max_threads:    32
> > domain: Module     shift: 5 dom_size:     1 max_threads:    32
> > domain: Tile       shift: 5 dom_size:     1 max_threads:    32
> > domain: Die        shift: 5 dom_size:     1 max_threads:    32
> > domain: Package    shift: 5 dom_size:     1 max_threads:    32
> > 
> > /sys/kernel/debug/x86/topo/cpus/39
> > online:              1
> > initial_apicid:      39
> > apicid:              39
> > pkg_id:              1
> > die_id:              1
> > cu_id:               255
> > core_id:             12
> > logical_pkg_id:      1
> > logical_die_id:      1
> > llc_id:              32
> > l2c_id:              56
> > amd_node_id:         0
> > amd_nodes_per_pkg:   0
> > num_threads:         20
> > num_cores:           10
> > max_dies_per_pkg:    1
> > max_threads_per_core:2
> 
> That makes much more sense now.
> 
> Zhang, can you please follow up on:
> 
>  
> https://lore.kernel.org/all/613df280116378115585d0c483f7e186cffaeb58.camel@intel.com/
> 
> or otherwise I just polish up PeterZ's variant of it tomorrow.
> 
Sorry for the late response. I was in the reviewing of the previous
patch series and missed this one.

IMO, PeterZ' patch already follows the suggestion at
https://lore.kernel.org/all/87pm4bp54z.ffs@tglx/
And https://lore.kernel.org/all/87jzukqjvf.ffs@tglx/ is not needed
because there are no duplicate APIC IDs from LAPIC and x2APIC anymore
with the above patch.

So I think we can go with PeterZ' patch.

thanks,
rui

Re: [patch 24/53] x86/cpu/topology: Provide separate APIC registration functions

[Zhang, Rui]

On Mon, 2023-08-07 at 15:53 +0200, Thomas Gleixner wrote:
> generic_processor_info() aside of being a complete misnomer is used
> for
> both early boot registration and ACPI CPU hotplug.
> 
> While it's arguable that this can share some code, it results in code
> which
> is hard to understand and kept around post init for no real reason.
> 
> Also the call sites do lots of manual fiddling in topology related
> variables instead of having proper interfaces for the purpose which
> handle
> the topology internals correctly.
> 
> Provide topology_register_apic(), topology_hotplug_apic() and
> topology_hot_unplug_apic()

s/topology_hot_unplug_apic/topology_hotunplug_apic

thanks,
rui

>  which have the extra magic of the call sites
> incorporated and for now are wrappers around
> generic_processor_info().
> 
> Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
> ---
>  arch/x86/include/asm/apic.h         |    3 
>  arch/x86/kernel/cpu/topology.c |  113
> ++++++++++++++++++++++++++++++------
>  2 files changed, 98 insertions(+), 18 deletions(-)
> 
> --- a/arch/x86/include/asm/apic.h
> +++ b/arch/x86/include/asm/apic.h
> @@ -171,7 +171,10 @@ extern bool apic_needs_pit(void);
>  
>  extern void apic_send_IPI_allbutself(unsigned int vector);
>  
> +extern void topology_register_apic(u32 apic_id, u32 acpi_id, bool
> present);
>  extern void topology_register_boot_apic(u32 apic_id);
> +extern int topology_hotplug_apic(u32 apic_id, u32 acpi_id);
> +extern void topology_hotunplug_apic(unsigned int cpu);
>  
>  #else /* !CONFIG_X86_LOCAL_APIC */
>  static inline void lapic_shutdown(void) { }
> --- a/arch/x86/kernel/cpu/topology.c
> +++ b/arch/x86/kernel/cpu/topology.c
> @@ -79,32 +79,38 @@ early_initcall(smp_init_primary_thread_m
>  static inline void cpu_mark_primary_thread(unsigned int cpu,
> unsigned int apicid) { }
>  #endif
>  
> -/*
> - * Should use this API to allocate logical CPU IDs to keep
> nr_logical_cpuids
> - * and cpuid_to_apicid[] synchronized.
> - */
> -static int allocate_logical_cpuid(int apicid)
> +static int topo_lookup_cpuid(u32 apic_id)
>  {
>         int i;
>  
> -       /*
> -        * cpuid <-> apicid mapping is persistent, so when a cpu is
> up,
> -        * check if the kernel has allocated a cpuid for it.
> -        */
> +       /* CPU# to APICID mapping is persistent once it is
> established */
>         for (i = 0; i < nr_logical_cpuids; i++) {
> -               if (cpuid_to_apicid[i] == apicid)
> +               if (cpuid_to_apicid[i] == apic_id)
>                         return i;
>         }
> +       return -ENODEV;
> +}
> +
> +/*
> + * Should use this API to allocate logical CPU IDs to keep
> nr_logical_cpuids
> + * and cpuid_to_apicid[] synchronized.
> + */
> +static int allocate_logical_cpuid(u32 apic_id)
> +{
> +       int cpu = topo_lookup_cpuid(apic_id);
> +
> +       if (cpu >= 0)
> +               return cpu;
>  
>         /* Allocate a new cpuid. */
>         if (nr_logical_cpuids >= nr_cpu_ids) {
>                 WARN_ONCE(1, "APIC: NR_CPUS/possible_cpus limit of %u
> reached. "
>                              "Processor %d/0x%x and the rest are
> ignored.\n",
> -                            nr_cpu_ids, nr_logical_cpuids, apicid);
> +                            nr_cpu_ids, nr_logical_cpuids, apic_id);
>                 return -EINVAL;
>         }
>  
> -       cpuid_to_apicid[nr_logical_cpuids] = apicid;
> +       cpuid_to_apicid[nr_logical_cpuids] = apic_id;
>         return nr_logical_cpuids++;
>  }
>  
> @@ -122,12 +128,6 @@ static void cpu_update_apic(int cpu, u32
>                 cpu_mark_primary_thread(cpu, apicid);
>  }
>  
> -void __init topology_register_boot_apic(u32 apic_id)
> -{
> -       cpuid_to_apicid[0] = apic_id;
> -       cpu_update_apic(0, apic_id);
> -}
> -
>  int generic_processor_info(int apicid)
>  {
>         int cpu, max = nr_cpu_ids;
> @@ -169,6 +169,83 @@ int generic_processor_info(int apicid)
>         return cpu;
>  }
>  
> +/**
> + * topology_register_apic - Register an APIC in early topology maps
> + * @apic_id:   The APIC ID to set up
> + * @acpi_id:   The ACPI ID associated to the APIC
> + * @present:   True if the corresponding CPU is present
> + */
> +void __init topology_register_apic(u32 apic_id, u32 acpi_id, bool
> present)
> +{
> +       int cpu;
> +
> +       if (apic_id >= MAX_LOCAL_APIC) {
> +               pr_err_once("APIC ID %x exceeds kernel limit of:
> %x\n", apic_id, MAX_LOCAL_APIC - 1);
> +               return;
> +       }
> +
> +       if (!present) {
> +               disabled_cpus++;
> +               return;
> +       }
> +
> +       cpu = generic_processor_info(apic_id);
> +       if (cpu >= 0)
> +               early_per_cpu(x86_cpu_to_acpiid, cpu) = acpi_id;
> +}
> +
> +/**
> + * topology_register_boot_apic - Register the boot CPU APIC
> + * @apic_id:   The APIC ID to set up
> + *
> + * Separate so CPU #0 can be assigned
> + */
> +void __init topology_register_boot_apic(u32 apic_id)
> +{
> +       cpuid_to_apicid[0] = apic_id;
> +       cpu_update_apic(0, apic_id);
> +}
> +
> +#ifdef CONFIG_ACPI_HOTPLUG_CPU
> +/**
> + * topology_hotplug_apic - Handle a physical hotplugged APIC after
> boot
> + * @apic_id:   The APIC ID to set up
> + * @acpi_id:   The ACPI ID associated to the APIC
> + */
> +int topology_hotplug_apic(u32 apic_id, u32 acpi_id)
> +{
> +       int cpu;
> +
> +       if (apic_id >= MAX_LOCAL_APIC)
> +               return -EINVAL;
> +
> +       cpu = topo_lookup_cpuid(apic_id);
> +       if (cpu < 0) {
> +               cpu = generic_processor_info(apic_id);
> +               if (cpu >= 0)
> +                       per_cpu(x86_cpu_to_acpiid, cpu) = acpi_id;
> +       }
> +       return cpu;
> +}
> +
> +/**
> + * topology_hotunplug_apic - Remove a physical hotplugged APIC after
> boot
> + * @cpu:       The CPU number for which the APIC ID is removed
> + */
> +void topology_hotunplug_apic(unsigned int cpu)
> +{
> +       u32 apic_id = cpuid_to_apicid[cpu];
> +
> +       if (apic_id == BAD_APICID)
> +               return;
> +
> +       per_cpu(x86_cpu_to_apicid, cpu) = BAD_APICID;
> +       clear_bit(apic_id, phys_cpu_present_map);
> +       set_cpu_present(cpu, false);
> +       num_processors--;
> +}
> +#endif
> +
>  static int __init apic_set_disabled_cpu_apicid(char *arg)
>  {
>         if (!arg || !get_option(&arg, &disabled_cpu_apicid))
> 

Re: [patch 53/53] x86/cpu/topology: Get rid of cpuinfo::x86_max_cores

[Zhang, Rui]

> --- a/arch/x86/kernel/smpboot.c
> +++ b/arch/x86/kernel/smpboot.c
> @@ -566,7 +566,7 @@ static void __init build_sched_topology(
>  void set_cpu_sibling_map(int cpu)
>  {
>         bool has_smt = topology_smt_supported();
> -       bool has_mp = has_smt || boot_cpu_data.x86_max_cores > 1;
> +       bool has_mp = has_smt || topology_num_cores_per_package() >
> 1;

or use
	bool has_mp = topology_num_threads_per_package() > 1;
?

thanks,
rui

RE: [patch 00/53] x86/topology: The final installment

[Michael Kelley (LINUX)]

From: Thomas Gleixner <tglx@linutronix.de> Sent: Monday, August 7, 2023 6:53 AM
> 
> Hi!
> 
> This is the (for now) last part of reworking topology enumeration and
> management. It's based on the APIC and CPUID rework series which can be
> found here:
> 
> https://lore.kernel.org/lkml/20230802101635.459108805@linutronix.de/
> 
> With these preparatory changes in place, it's now possible to address the
> real issues of the current topology code:
> 
>   - Wrong core count on hybrid systems
> 
>   - Heuristics based size information for packages and dies which
>     are failing to work correctly with certain command line parameters.
> 
>   - Full evaluation fail for a theoretical hybrid system which boots
>     from an E-core
> 
>   - The complete insanity of manipulating global data from firmware parsers
>     or the XEN/PV fake SMP enumeration. The latter is really a piece of art.
> 
> This series addresses this by
> 
>   - Mopping up some more historical technical debt
> 
>   - Consolidating all topology relevant functionality into one place
> 
>   - Providing separate interfaces for boot time and ACPI hotplug operations
> 
>   - A sane ordering of command line options and restrictions
> 
>   - A sensible way to handle the BSP problem in kdump kernels instead of
>     the unreliable command line option.
> 
>   - Confinement of topology relevant variables by replacing the XEN/PV SMP
>     enumeration fake with something halfways sensible.
> 
>   - Evaluation of sizes by analysing the topology via the CPUID provided
>     APIC ID segmentation and the actual APIC IDs which are registered at
>     boot time.
> 
>   - Removal of heuristics and broken size calculations
> 
> The idea behind this is the following:
> 
> The APIC IDs describe the system topology in multiple domain levels. The
> CPUID topology parser provides the information which part of the APIC ID is
> associated to the individual levels (Intel terminology):
> 
>    [ROOT][PACKAGE][DIE][TILE][MODULE][CORE][THREAD]
> 
> The root space contains the package (socket) IDs. Not enumerated levels
> consume 0 bits space, but conceptually they are always represented. If
> e.g. only CORE and THREAD levels are enumerated then the DIE, MODULE and
> TILE have the same physical ID as the PACKAGE.
> 
> If SMT is not supported, then the THREAD domain is still used. It then
> has the same physical ID as the CORE domain and is the only child of
> the core domain.
> 
> This allows an unified view on the system independent of the enumerated
> domain levels without requiring any conditionals in the code.
> 
> AMD does only expose 4 domain levels with obviously different terminology,
> but that can be easily mapped into the Intel variant with a trivial lookup
> table added to the CPUID parser.
> 
> The resulting topology information of an ADL hybrid system with 8 P-Cores
> and 8 E-Cores looks like this:
> 
>  CPU topo: Max. logical packages:   1
>  CPU topo: Max. logical dies:       1
>  CPU topo: Max. dies per package:   1
>  CPU topo: Max. threads per core:   2
>  CPU topo: Num. cores per package:    16
>  CPU topo: Num. threads per package:  24
>  CPU topo: Allowing 24 present CPUs plus 0 hotplug CPUs
>  CPU topo: Thread    :    24
>  CPU topo: Core      :    16
>  CPU topo: Module    :     1
>  CPU topo: Tile      :     1
>  CPU topo: Die       :     1
>  CPU topo: Package   :     1
> 
> This is happening on the boot CPU before any of the APs is started and
> provides correct size information right from the start.
> 
> Even the XEN/PV trainwreck makes use of this now. On Dom0 it utilizes the
> MADT and on DomU it provides fake APIC IDs, which combined with the
> provided CPUID information make it at least look halfways realistic instead
> of claiming to have one CPU per package as the current upstream code does.
> 
> This is solely addressing the core topology issues, but there is a plan for
> further consolidation of other topology related information into one single
> source of information instead of having a gazillion of localized special
> parsers and representations all over the place. There are quite some other
> things which can be simplified on top of this, like updating the various
> cpumasks during CPU bringup, but that's all left for later.
> 
> So another 53 patches later, the resulting diffstat is:
> 
>    64 files changed, 830 insertions(+), 955 deletions(-)
> 
> and the combo diffstat of all three series combined:
> 
>   115 files changed, 2414 insertions(+), 3035 deletions(-)
> 
> The current series applies on top of
> 
>    git://git.kernel.org/pub/scm/linux/kernel/git/tglx/devel.git topo-cpuid-v3
> 
> and is available from git here:
> 
>    git://git.kernel.org/pub/scm/linux/kernel/git/tglx/devel.git topo-full-v1
> 
> Thanks,
> 
> 	tglx

Tested the full series on Hyper-V VMs on Intel and AMD Zen processors.
Tested with hyper-threading enabled and disabled, and with a variety of
NUMA and L3 cache configurations.  All looks good, modulo the known
issue with Hyper-V providing incorrect APIC IDs in some NUMA configs,
but this patch series did not make that problem any worse.

Tested-by: Michael Kelley <mikelley@microsoft.com>

Re: [patch 36/53] x86/cpu/topology: Rework possible CPU management

[Zhang, Rui]

Hi, Thomas,

I think the declaration for prefill_possible_map() in
arch/x86/include/asm/cpu.h can also be removed in this patch.

And sorry for switching among all these threads, I need to read them
all together and narrow down to different patches when I find something
suspicious.

thanks,
rui

On Mon, 2023-08-07 at 15:53 +0200, Thomas Gleixner wrote:
> Managing possible CPUs is an unreadable and uncomprehensible maze.
> Aside of
> that it's backwards because it applies command line limits after
> registering all APICs.
> 
> Rewrite it so that it:
> 
>   - Applies the command line limits upfront so that only the allowed
> amount
>     of APIC IDs can be registered.
> 
>   - Applies eventual late restrictions in an understandable way
> 
>   - Uses simple min_t() calculations which are trivial to follow.
> 
>   - Provides a separate function for resetting to UP mode late in the
>     bringup process.
> 
> Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
> ---
>  arch/x86/include/asm/apic.h     |    5 +
>  arch/x86/include/asm/topology.h |    4 
>  arch/x86/kernel/cpu/topology.c  |  176 ++++++++++++++++++++++++-----
> -----------
>  arch/x86/kernel/setup.c         |    9 --
>  arch/x86/kernel/smpboot.c       |    6 -
>  5 files changed, 120 insertions(+), 80 deletions(-)
> 
> --- a/arch/x86/include/asm/apic.h
> +++ b/arch/x86/include/asm/apic.h
> @@ -175,6 +175,9 @@ extern void topology_register_apic(u32 a
>  extern void topology_register_boot_apic(u32 apic_id);
>  extern int topology_hotplug_apic(u32 apic_id, u32 acpi_id);
>  extern void topology_hotunplug_apic(unsigned int cpu);
> +extern void topology_apply_cmdline_limits_early(void);
> +extern void topology_init_possible_cpus(void);
> +extern void topology_reset_possible_cpus_up(void);
>  
>  #else /* !CONFIG_X86_LOCAL_APIC */
>  static inline void lapic_shutdown(void) { }
> @@ -190,6 +193,8 @@ static inline void apic_intr_mode_init(v
>  static inline void lapic_assign_system_vectors(void) { }
>  static inline void lapic_assign_legacy_vector(unsigned int i, bool
> r) { }
>  static inline bool apic_needs_pit(void) { return true; }
> +static inline void topology_apply_cmdline_limits_early(void) { }
> +static inline void topology_init_possible_cpus(void) { }
>  #endif /* !CONFIG_X86_LOCAL_APIC */
>  
>  #ifdef CONFIG_X86_X2APIC
> --- a/arch/x86/include/asm/topology.h
> +++ b/arch/x86/include/asm/topology.h
> @@ -190,6 +190,9 @@ static inline bool topology_is_primary_t
>  {
>         return cpumask_test_cpu(cpu, cpu_primary_thread_mask);
>  }
> +
> +void topology_apply_cmdline_limits_early(void);
> +
>  #else /* CONFIG_SMP */
>  #define topology_max_packages()                        (1)
>  static inline int
> @@ -202,6 +205,7 @@ static inline int topology_max_smt_threa
>  static inline bool topology_is_primary_thread(unsigned int cpu) {
> return true; }
>  static inline bool topology_smt_supported(void) { return false; }
>  static inline unsigned int topology_amd_nodes_per_pkg(void) { return
> 0; };
> +static inline void topology_apply_cmdline_limits_early(void) { }
>  #endif /* !CONFIG_SMP */
>  
>  static inline void arch_fix_phys_package_id(int num, u32 slot)
> --- a/arch/x86/kernel/cpu/topology.c
> +++ b/arch/x86/kernel/cpu/topology.c
> @@ -5,6 +5,7 @@
>  #include <xen/xen.h>
>  
>  #include <asm/apic.h>
> +#include <asm/io_apic.h>
>  #include <asm/mpspec.h>
>  #include <asm/smp.h>
>  
> @@ -85,73 +86,6 @@ early_initcall(smp_init_primary_thread_m
>  static inline void cpu_mark_primary_thread(unsigned int cpu,
> unsigned int apicid) { }
>  #endif
>  
> -static int __initdata setup_possible_cpus = -1;
> -
> -/*
> - * cpu_possible_mask should be static, it cannot change as cpu's
> - * are onlined, or offlined. The reason is per-cpu data-structures
> - * are allocated by some modules at init time, and don't expect to
> - * do this dynamically on cpu arrival/departure.
> - * cpu_present_mask on the other hand can change dynamically.
> - * In case when cpu_hotplug is not compiled, then we resort to
> current
> - * behaviour, which is cpu_possible == cpu_present.
> - * - Ashok Raj
> - *
> - * Three ways to find out the number of additional hotplug CPUs:
> - * - If the BIOS specified disabled CPUs in ACPI/mptables use that.
> - * - The user can overwrite it with possible_cpus=NUM
> - * - Otherwise don't reserve additional CPUs.
> - * We do this because additional CPUs waste a lot of memory.
> - * -AK
> - */
> -__init void prefill_possible_map(void)
> -{
> -       unsigned int num_processors = topo_info.nr_assigned_cpus;
> -       unsigned int disabled_cpus = topo_info.nr_disabled_cpus;
> -       int i, possible;
> -
> -       i = setup_max_cpus ?: 1;
> -       if (setup_possible_cpus == -1) {
> -               possible = topo_info.nr_assigned_cpus;
> -#ifdef CONFIG_HOTPLUG_CPU
> -               if (setup_max_cpus)
> -                       possible += num_processors;
> -#else
> -               if (possible > i)
> -                       possible = i;
> -#endif
> -       } else
> -               possible = setup_possible_cpus;
> -
> -       total_cpus = max_t(int, possible, num_processors +
> disabled_cpus);
> -
> -       /* nr_cpu_ids could be reduced via nr_cpus= */
> -       if (possible > nr_cpu_ids) {
> -               pr_warn("%d Processors exceeds NR_CPUS limit of
> %u\n",
> -                       possible, nr_cpu_ids);
> -               possible = nr_cpu_ids;
> -       }
> -
> -#ifdef CONFIG_HOTPLUG_CPU
> -       if (!setup_max_cpus)
> -#endif
> -       if (possible > i) {
> -               pr_warn("%d Processors exceeds max_cpus limit of
> %u\n",
> -                       possible, setup_max_cpus);
> -               possible = i;
> -       }
> -
> -       set_nr_cpu_ids(possible);
> -
> -       pr_info("Allowing %d CPUs, %d hotplug CPUs\n",
> -               possible, max_t(int, possible - num_processors, 0));
> -
> -       reset_cpu_possible_mask();
> -
> -       for (i = 0; i < possible; i++)
> -               set_cpu_possible(i, true);
> -}
> -
>  static int topo_lookup_cpuid(u32 apic_id)
>  {
>         int i;
> @@ -294,12 +228,114 @@ void topology_hotunplug_apic(unsigned in
>  }
>  #endif
>  
> -static int __init _setup_possible_cpus(char *str)
> +#ifdef CONFIG_SMP
> +static unsigned int max_possible_cpus __initdata = NR_CPUS;
> +
> +/**
> + * topology_apply_cmdline_limits_early - Apply topology command line
> limits early
> + *
> + * Ensure that command line limits are in effect before firmware
> parsing
> + * takes place.
> + */
> +void __init topology_apply_cmdline_limits_early(void)
> +{
> +       unsigned int possible = nr_cpu_ids;
> +
> +       /* 'maxcpus=0' 'nosmp' 'nolapic' 'disableapic' 'noapic' */
> +       if (!setup_max_cpus || ioapic_is_disabled ||
> apic_is_disabled)
> +               possible = 1;
> +
> +       /* 'possible_cpus=N' */
> +       possible = min_t(unsigned int, max_possible_cpus, possible);
> +
> +       if (possible < nr_cpu_ids) {
> +               pr_info("Limiting to %u possible CPUs\n", possible);
> +               set_nr_cpu_ids(possible);
> +       }
> +}
> +
> +static __init bool restrict_to_up(void)
> +{
> +       if (!smp_found_config || ioapic_is_disabled)
> +               return true;
> +       /*
> +        * XEN PV is special as it does not advertise the local APIC
> +        * properly, but provides a fake topology for it so that the
> +        * infrastructure works. So don't apply the restrictions vs.
> APIC
> +        * here.
> +        */
> +       if (xen_pv_domain())
> +               return false;
> +
> +       return apic_is_disabled;
> +}
> +
> +void __init topology_init_possible_cpus(void)
> +{
> +       unsigned int assigned = topo_info.nr_assigned_cpus;
> +       unsigned int disabled = topo_info.nr_disabled_cpus;
> +       unsigned int total = assigned + disabled;
> +       unsigned int cpu, allowed = 1;
> +
> +       if (!restrict_to_up()) {
> +               if (WARN_ON_ONCE(assigned > nr_cpu_ids)) {
> +                       disabled += assigned - nr_cpu_ids;
> +                       assigned = nr_cpu_ids;
> +               }
> +               allowed = min_t(unsigned int, total, nr_cpu_ids);
> +       }
> +
> +       if (total > allowed)
> +               pr_warn("%u possible CPUs exceed the limit of %u\n",
> total, allowed);
> +
> +       assigned = min_t(unsigned int, allowed, assigned);
> +       disabled = allowed - assigned;
> +
> +       topo_info.nr_assigned_cpus = assigned;
> +       topo_info.nr_disabled_cpus = disabled;
> +
> +       total_cpus = allowed;
> +       set_nr_cpu_ids(allowed);
> +
> +       pr_info("Allowing %u present CPUs plus %u hotplug CPUs\n",
> assigned, disabled);
> +       if (topo_info.nr_rejected_cpus)
> +               pr_info("Rejected CPUs %u\n",
> topo_info.nr_rejected_cpus);
> +
> +       init_cpu_present(cpumask_of(0));
> +       init_cpu_possible(cpumask_of(0));
> +
> +       for (cpu = 0; cpu < allowed; cpu++) {
> +               u32 apicid = cpuid_to_apicid[cpu];
> +
> +               set_cpu_possible(cpu, true);
> +
> +               if (apicid == BAD_APICID)
> +                       continue;
> +
> +               set_cpu_present(cpu, test_bit(apicid,
> phys_cpu_present_map));
> +       }
> +}
> +
> +/*
> + * Late SMP disable after sizing CPU masks when APIC/IOAPIC setup
> failed.
> + */
> +void __init topology_reset_possible_cpus_up(void)
>  {
> -       get_option(&str, &setup_possible_cpus);
> +       init_cpu_present(cpumask_of(0));
> +       init_cpu_possible(cpumask_of(0));
> +
> +       bitmap_zero(phys_cpu_present_map, MAX_LOCAL_APIC);
> +       if (topo_info.boot_cpu_apic_id != BAD_APICID)
> +               set_bit(topo_info.boot_cpu_apic_id,
> phys_cpu_present_map);
> +}
> +
> +static int __init setup_possible_cpus(char *str)
> +{
> +       get_option(&str, &max_possible_cpus);
>         return 0;
>  }
> -early_param("possible_cpus", _setup_possible_cpus);
> +early_param("possible_cpus", setup_possible_cpus);
> +#endif
>  
>  static int __init apic_set_disabled_cpu_apicid(char *arg)
>  {
> --- a/arch/x86/kernel/setup.c
> +++ b/arch/x86/kernel/setup.c
> @@ -1258,6 +1258,8 @@ void __init setup_arch(char **cmdline_p)
>  
>         early_quirks();
>  
> +       topology_apply_cmdline_limits_early();
> +
>         /*
>          * Parse SMP configuration. Try ACPI first and then the
> platform
>          * specific parser.
> @@ -1265,13 +1267,10 @@ void __init setup_arch(char **cmdline_p)
>         acpi_boot_init();
>         x86_init.mpparse.parse_smp_cfg();
>  
> -       /*
> -        * Systems w/o ACPI and mptables might not have it mapped the
> local
> -        * APIC yet, but prefill_possible_map() might need to access
> it.
> -        */
> +       /* Last opportunity to detect and map the local APIC */
>         init_apic_mappings();
>  
> -       prefill_possible_map();
> +       topology_init_possible_cpus();
>  
>         init_cpu_to_node();
>         init_gi_nodes();
> --- a/arch/x86/kernel/smpboot.c
> +++ b/arch/x86/kernel/smpboot.c
> @@ -1156,11 +1156,7 @@ static __init void disable_smp(void)
>         pr_info("SMP disabled\n");
>  
>         disable_ioapic_support();
> -
> -       init_cpu_present(cpumask_of(0));
> -       init_cpu_possible(cpumask_of(0));
> -
> -       reset_phys_cpu_present_map(smp_found_config ?
> boot_cpu_physical_apicid : 0);
> +       topology_reset_possible_cpus_up();
>  
>         cpumask_set_cpu(0, topology_sibling_cpumask(0));
>         cpumask_set_cpu(0, topology_core_cpumask(0));
> 

Re: [patch 53/53] x86/cpu/topology: Get rid of cpuinfo::x86_max_cores

[Zhang, Rui]

> --- a/arch/x86/kernel/smpboot.c
> +++ b/arch/x86/kernel/smpboot.c
> @@ -566,7 +566,7 @@ static void __init build_sched_topology(
>  void set_cpu_sibling_map(int cpu)
>  {
>         bool has_smt = topology_smt_supported();
> -       bool has_mp = has_smt || boot_cpu_data.x86_max_cores > 1;
> +       bool has_mp = has_smt || topology_num_cores_per_package() >
> 1;

This patch series, especially the change here, indeed fixes a bug
observed on a platform with below cpuid.1f output,

# taskset -c 0 cpuid -1 -l 0x1f -s 0
CPU:
   V2 extended topology (0x1f):
      x2APIC ID of logical processor = 0x0 (0)
      --- level 0 ---
      level number                          = 0x0 (0)
      level type                            = thread (1)
      bit width of level & previous levels  = 0x1 (1)
      number of logical processors at level = 0x1 (1)
# taskset -c 0 cpuid -1 -l 0x1f -s 1
CPU:
      --- level 1 ---
      level number                          = 0x1 (1)
      level type                            = core (2)
      bit width of level & previous levels  = 0x3 (3)
      number of logical processors at level = 0x1 (1)
# taskset -c 0 cpuid -1 -l 0x1f -s 2
CPU:
      --- level 2 ---
      level number                          = 0x2 (2)
      level type                            = module (3)
      bit width of level & previous levels  = 0x7 (7)
      number of logical processors at level = 0x18 (24)

The bug is that, on this platform, without this patch series,
1. has_smt is false because there is indeed no HT
2. has_mp is false because
	1. core_level_siblings from CPUID.1F Core level EBX is 1
	2. current code does not parse Module type and use
core_level_siblings as the number_of_cpus_in_a_package, so
	boot_cpu_data.x86_max_cores =
		core_level_siblings / smp_num_siblings = 1

As a result, set_cpu_sibling_map() assumes this is a UP system and does
not update any sibling maps, the topology is totally broken then.

thanks,
rui

Re: [patch 37/53] x86/cpu: Detect real BSP on crash kernels

[Zhang, Rui]

> +static __init void check_for_kdump_kernel(void)
> +{
> +       u32 bsp_apicid;
> +
> +       /*
> +        * There is no real good way to detect whether this a kdump()
> +        * kernel, but except on the Voyager SMP monstrosity which is
> not
> +        * longer supported, the real BSP has always the lowest
> numbered
> +        * APIC ID. If a crash happened on an AP, which then ends up
> as
> +        * boot CPU in the kdump() kernel, then sending INIT to the
> real
> +        * BSP would reset the whole system.
> +        */


Hi, Thomas,

Unfortunately this causes a regression on Intel Meteorlake platform,
where the BSP APIC ID is NOT the lowest numbered APIC ID (instead,
CPU12, the first Ecore CPU, has APIC ID 0).

And this causes the system fails to enumerate CPU12 (I didn't do
bisect. I suspect this patch breaks it by reading the code).

log with 6.7-rc vanilla kernel,
[    0.335133] smp: Bringing up secondary CPUs ...
[    0.335133] smpboot: x86: Booting SMP configuration:
[    0.335133] .... node  #0, CPUs:        #1  #3  #6  #8 #10 #12 #13
#14 #15 #16 #17 #18 #19 #20 #21
[    0.010435] core: cpu_atom PMU driver: PEBS-via-PT 
[    0.010435] ... version:                5
[    0.010435] ... bit width:              48
[    0.010435] ... generic registers:      8
[    0.010435] ... value mask:             0000ffffffffffff
[    0.010435] ... max period:             00007fffffffffff
[    0.010435] ... fixed-purpose events:   3
[    0.010435] ... event mask:             00000007000000ff
[    0.339203]   #2  #4  #5  #7  #9 #11
[    0.343208] smp: Brought up 1 node, 22 CPUs

log with 6.5-rc4 kernel + your patch series,
[    2.208960] smpboot: x86: Booting SMP configuration:
[    2.209869] .... node  #0, CPUs:        #1  #3  #6  #8 #10 #13 #14
#15 #16 #17 #18 #19 #20 #21
[    1.796167] core: cpu_atom PMU driver: PEBS-via-PT
[    1.796167] ... version:                5
[    1.796167] ... bit width:              48
[    1.796167] ... generic registers:      8
[    1.796167] ... value mask:             0000ffffffffffff
[    1.796167] ... max period:             00007fffffffffff
[    1.796167] ... fixed-purpose events:   3
[    1.796167] ... event mask:             00000007000000ff
[    2.260958]   #2  #4  #5  #7  #9 #11
[    2.263906] smp: Brought up 1 node, 21 CPUs


thanks,
rui

# cpuid -l 0x1f -s 0 | grep x2APIC
      x2APIC ID of logical processor = 0x20 (32)
      x2APIC ID of logical processor = 0x10 (16)
      x2APIC ID of logical processor = 0x11 (17)
      x2APIC ID of logical processor = 0x18 (24)
      x2APIC ID of logical processor = 0x19 (25)
      x2APIC ID of logical processor = 0x21 (33)
      x2APIC ID of logical processor = 0x28 (40)
      x2APIC ID of logical processor = 0x29 (41)
      x2APIC ID of logical processor = 0x30 (48)
      x2APIC ID of logical processor = 0x31 (49)
      x2APIC ID of logical processor = 0x38 (56)
      x2APIC ID of logical processor = 0x39 (57)
      x2APIC ID of logical processor = 0x0 (0)
      x2APIC ID of logical processor = 0x2 (2)
      x2APIC ID of logical processor = 0x4 (4)
      x2APIC ID of logical processor = 0x6 (6)
      x2APIC ID of logical processor = 0x8 (8)
      x2APIC ID of logical processor = 0xa (10)
      x2APIC ID of logical processor = 0xc (12)
      x2APIC ID of logical processor = 0xe (14)
      x2APIC ID of logical processor = 0x40 (64)
      x2APIC ID of logical processor = 0x42 (66)

Re: [patch 37/53] x86/cpu: Detect real BSP on crash kernels

[Thomas Gleixner]

On Mon, Jan 08 2024 at 14:11, Zhang, Rui wrote:
>> +static __init void check_for_kdump_kernel(void)
>> +{
>> +       u32 bsp_apicid;
>> +
>> +       /*
>> +        * There is no real good way to detect whether this a kdump()
>> +        * kernel, but except on the Voyager SMP monstrosity which is
>> not
>> +        * longer supported, the real BSP has always the lowest
>> numbered
>> +        * APIC ID. If a crash happened on an AP, which then ends up
>> as
>> +        * boot CPU in the kdump() kernel, then sending INIT to the
>> real
>> +        * BSP would reset the whole system.
>> +        */
>
>
> Hi, Thomas,
>
> Unfortunately this causes a regression on Intel Meteorlake platform,
> where the BSP APIC ID is NOT the lowest numbered APIC ID (instead,
> CPU12, the first Ecore CPU, has APIC ID 0).

Bah. Let me think about that.

Re: [patch 37/53] x86/cpu: Detect real BSP on crash kernels

[Thomas Gleixner]

On Mon, Jan 08 2024 at 15:54, Thomas Gleixner wrote:

> On Mon, Jan 08 2024 at 14:11, Zhang, Rui wrote:
>>> +static __init void check_for_kdump_kernel(void)
>>> +{
>>> +       u32 bsp_apicid;
>>> +
>>> +       /*
>>> +        * There is no real good way to detect whether this a kdump()
>>> +        * kernel, but except on the Voyager SMP monstrosity which is
>>> not
>>> +        * longer supported, the real BSP has always the lowest
>>> numbered
>>> +        * APIC ID. If a crash happened on an AP, which then ends up
>>> as
>>> +        * boot CPU in the kdump() kernel, then sending INIT to the
>>> real
>>> +        * BSP would reset the whole system.
>>> +        */
>>
>>
>> Hi, Thomas,
>>
>> Unfortunately this causes a regression on Intel Meteorlake platform,
>> where the BSP APIC ID is NOT the lowest numbered APIC ID (instead,
>> CPU12, the first Ecore CPU, has APIC ID 0).
>
> Bah. Let me think about that.

In which order are the APICs/CPUs enumerated by ACPI?

Re: [patch 37/53] x86/cpu: Detect real BSP on crash kernels

[Zhang, Rui]

On Mon, 2024-01-08 at 17:13 +0100, Thomas Gleixner wrote:
> On Mon, Jan 08 2024 at 15:54, Thomas Gleixner wrote:
> 
> > On Mon, Jan 08 2024 at 14:11, Zhang, Rui wrote:
> > > > +static __init void check_for_kdump_kernel(void)
> > > > +{
> > > > +       u32 bsp_apicid;
> > > > +
> > > > +       /*
> > > > +        * There is no real good way to detect whether this a
> > > > kdump()
> > > > +        * kernel, but except on the Voyager SMP monstrosity
> > > > which is
> > > > not
> > > > +        * longer supported, the real BSP has always the lowest
> > > > numbered
> > > > +        * APIC ID. If a crash happened on an AP, which then
> > > > ends up
> > > > as
> > > > +        * boot CPU in the kdump() kernel, then sending INIT to
> > > > the
> > > > real
> > > > +        * BSP would reset the whole system.
> > > > +        */
> > > 
> > > 
> > > Hi, Thomas,
> > > 
> > > Unfortunately this causes a regression on Intel Meteorlake
> > > platform,
> > > where the BSP APIC ID is NOT the lowest numbered APIC ID
> > > (instead,
> > > CPU12, the first Ecore CPU, has APIC ID 0).
> > 
> > Bah. Let me think about that.
> 
> In which order are the APICs/CPUs enumerated by ACPI?


This is the order in MADT,
$ cat apic.dsl  | grep x2Apic
[030h 0048   4]          Processor x2Apic ID : 00000010
[040h 0064   4]          Processor x2Apic ID : 00000011
[050h 0080   4]          Processor x2Apic ID : 00000018
[060h 0096   4]          Processor x2Apic ID : 00000019
[070h 0112   4]          Processor x2Apic ID : 00000020
[080h 0128   4]          Processor x2Apic ID : 00000021
[090h 0144   4]          Processor x2Apic ID : 00000028
[0A0h 0160   4]          Processor x2Apic ID : 00000029
[0B0h 0176   4]          Processor x2Apic ID : 00000030
[0C0h 0192   4]          Processor x2Apic ID : 00000031
[0D0h 0208   4]          Processor x2Apic ID : 00000038
[0E0h 0224   4]          Processor x2Apic ID : 00000039
[0F0h 0240   4]          Processor x2Apic ID : 00000000
[100h 0256   4]          Processor x2Apic ID : 00000002
[110h 0272   4]          Processor x2Apic ID : 00000004
[120h 0288   4]          Processor x2Apic ID : 00000006
[130h 0304   4]          Processor x2Apic ID : 00000008
[140h 0320   4]          Processor x2Apic ID : 0000000A
[150h 0336   4]          Processor x2Apic ID : 0000000C
[160h 0352   4]          Processor x2Apic ID : 0000000E
[170h 0368   4]          Processor x2Apic ID : 00000040
[180h 0384   4]          Processor x2Apic ID : 00000042

and this is the order in Linux (from CPU0 to CPUN)
      x2APIC ID of logical processor = 0x20 (32)
      x2APIC ID of logical processor = 0x10 (16)
      x2APIC ID of logical processor = 0x11 (17)
      x2APIC ID of logical processor = 0x18 (24)
      x2APIC ID of logical processor = 0x19 (25)
      x2APIC ID of logical processor = 0x21 (33)
      x2APIC ID of logical processor = 0x28 (40)
      x2APIC ID of logical processor = 0x29 (41)
      x2APIC ID of logical processor = 0x30 (48)
      x2APIC ID of logical processor = 0x31 (49)
      x2APIC ID of logical processor = 0x38 (56)
      x2APIC ID of logical processor = 0x39 (57)
      x2APIC ID of logical processor = 0x0 (0)
      x2APIC ID of logical processor = 0x2 (2)
      x2APIC ID of logical processor = 0x4 (4)
      x2APIC ID of logical processor = 0x6 (6)
      x2APIC ID of logical processor = 0x8 (8)
      x2APIC ID of logical processor = 0xa (10)
      x2APIC ID of logical processor = 0xc (12)
      x2APIC ID of logical processor = 0xe (14)
      x2APIC ID of logical processor = 0x40 (64)
      x2APIC ID of logical processor = 0x42 (66)

thanks,
rui

Re: [patch 37/53] x86/cpu: Detect real BSP on crash kernels

[Thomas Gleixner]

On Tue, Jan 09 2024 at 01:54, Zhang, Rui wrote:
> On Mon, 2024-01-08 at 17:13 +0100, Thomas Gleixner wrote:
>> > > Unfortunately this causes a regression on Intel Meteorlake
>> > > platform,
>> > > where the BSP APIC ID is NOT the lowest numbered APIC ID
>> > > (instead,
>> > > CPU12, the first Ecore CPU, has APIC ID 0).
>> > 
>> > Bah. Let me think about that.
>> 
>> In which order are the APICs/CPUs enumerated by ACPI?
>
>
> This is the order in MADT,
> $ cat apic.dsl  | grep x2Apic
> [030h 0048   4]          Processor x2Apic ID : 00000010
> [040h 0064   4]          Processor x2Apic ID : 00000011
> [050h 0080   4]          Processor x2Apic ID : 00000018
> [060h 0096   4]          Processor x2Apic ID : 00000019
> [070h 0112   4]          Processor x2Apic ID : 00000020
> [080h 0128   4]          Processor x2Apic ID : 00000021
> [090h 0144   4]          Processor x2Apic ID : 00000028
> [0A0h 0160   4]          Processor x2Apic ID : 00000029
> [0B0h 0176   4]          Processor x2Apic ID : 00000030
> [0C0h 0192   4]          Processor x2Apic ID : 00000031
> [0D0h 0208   4]          Processor x2Apic ID : 00000038
> [0E0h 0224   4]          Processor x2Apic ID : 00000039
> [0F0h 0240   4]          Processor x2Apic ID : 00000000
> [100h 0256   4]          Processor x2Apic ID : 00000002
> [110h 0272   4]          Processor x2Apic ID : 00000004
> [120h 0288   4]          Processor x2Apic ID : 00000006
> [130h 0304   4]          Processor x2Apic ID : 00000008
> [140h 0320   4]          Processor x2Apic ID : 0000000A
> [150h 0336   4]          Processor x2Apic ID : 0000000C
> [160h 0352   4]          Processor x2Apic ID : 0000000E
> [170h 0368   4]          Processor x2Apic ID : 00000040
> [180h 0384   4]          Processor x2Apic ID : 00000042
>
> and this is the order in Linux (from CPU0 to CPUN)
>       x2APIC ID of logical processor = 0x20 (32)
>       x2APIC ID of logical processor = 0x10 (16)
>       x2APIC ID of logical processor = 0x11 (17)
>       x2APIC ID of logical processor = 0x18 (24)
>       x2APIC ID of logical processor = 0x19 (25)
>       x2APIC ID of logical processor = 0x21 (33)
>       x2APIC ID of logical processor = 0x28 (40)
>       x2APIC ID of logical processor = 0x29 (41)
>       x2APIC ID of logical processor = 0x30 (48)
>       x2APIC ID of logical processor = 0x31 (49)
>       x2APIC ID of logical processor = 0x38 (56)
>       x2APIC ID of logical processor = 0x39 (57)
>       x2APIC ID of logical processor = 0x0 (0)
>       x2APIC ID of logical processor = 0x2 (2)
>       x2APIC ID of logical processor = 0x4 (4)
>       x2APIC ID of logical processor = 0x6 (6)
>       x2APIC ID of logical processor = 0x8 (8)
>       x2APIC ID of logical processor = 0xa (10)
>       x2APIC ID of logical processor = 0xc (12)
>       x2APIC ID of logical processor = 0xe (14)
>       x2APIC ID of logical processor = 0x40 (64)
>       x2APIC ID of logical processor = 0x42 (66)

What a mess...

Re: [patch 37/53] x86/cpu: Detect real BSP on crash kernels

[Thomas Gleixner]

On Wed, Jan 10 2024 at 15:19, Thomas Gleixner wrote:
>> This is the order in MADT,
>> $ cat apic.dsl  | grep x2Apic
>> [030h 0048   4]          Processor x2Apic ID : 00000010
>> [040h 0064   4]          Processor x2Apic ID : 00000011
...
>> and this is the order in Linux (from CPU0 to CPUN)
>>       x2APIC ID of logical processor = 0x20 (32)
>>       x2APIC ID of logical processor = 0x10 (16)
>
> What a mess...

And clearly not according to the spec

  "The second is that platform firmware should list the boot processor
   as the first processor entry in the MADT."

Oh well. There are reasons why this is written the way it is.

Re: [patch 37/53] x86/cpu: Detect real BSP on crash kernels

[Zhang, Rui]

On Wed, 2024-01-10 at 16:14 +0100, Thomas Gleixner wrote:
> On Wed, Jan 10 2024 at 15:19, Thomas Gleixner wrote:
> > > This is the order in MADT,
> > > $ cat apic.dsl  | grep x2Apic
> > > [030h 0048   4]          Processor x2Apic ID : 00000010
> > > [040h 0064   4]          Processor x2Apic ID : 00000011
> ...
> > > and this is the order in Linux (from CPU0 to CPUN)
> > >       x2APIC ID of logical processor = 0x20 (32)
> > >       x2APIC ID of logical processor = 0x10 (16)
> > 
> > What a mess...
> 
> And clearly not according to the spec
> 
>   "The second is that platform firmware should list the boot
> processor
>    as the first processor entry in the MADT."
> 
> Oh well. There are reasons why this is written the way it is.

Let me sync internally to see why it is designed in this way.

thanks,
rui

Re: [patch 37/53] x86/cpu: Detect real BSP on crash kernels

[Zhang, Rui]

Add Len.

On Wed, 2024-01-10 at 16:14 +0100, Thomas Gleixner wrote:
> On Wed, Jan 10 2024 at 15:19, Thomas Gleixner wrote:
> > > This is the order in MADT,
> > > $ cat apic.dsl  | grep x2Apic
> > > [030h 0048   4]          Processor x2Apic ID : 00000010
> > > [040h 0064   4]          Processor x2Apic ID : 00000011
> ...
> > > and this is the order in Linux (from CPU0 to CPUN)
> > >       x2APIC ID of logical processor = 0x20 (32)
> > >       x2APIC ID of logical processor = 0x10 (16)
> > 
> > What a mess...
> 
> And clearly not according to the spec
> 
>   "The second is that platform firmware should list the boot
> processor
>    as the first processor entry in the MADT."
> 
> Oh well. There are reasons why this is written the way it is.

This is indeed a violation of the ACPI spec and we should modify the
order in MADT. But this doesn't bring any actual effect as Linux
already handles this, right?

For the BSP APIC ID 0x20, I didn't find out a specific reason why we
have to do it in that way, but it is still legal. We may need to figure
out another way to distinguish the kdump kernel.

thanks,
rui

Re: [patch 37/53] x86/cpu: Detect real BSP on crash kernels

[Thomas Gleixner]

On Fri, Jan 12 2024 at 09:14, Zhang, Rui wrote:
> On Wed, 2024-01-10 at 16:14 +0100, Thomas Gleixner wrote:
>> On Wed, Jan 10 2024 at 15:19, Thomas Gleixner wrote:
>> > > This is the order in MADT,
>> > > $ cat apic.dsl  | grep x2Apic
>> > > [030h 0048   4]          Processor x2Apic ID : 00000010
>> > > [040h 0064   4]          Processor x2Apic ID : 00000011
>> ...
>> > > and this is the order in Linux (from CPU0 to CPUN)
>> > >       x2APIC ID of logical processor = 0x20 (32)
>> > >       x2APIC ID of logical processor = 0x10 (16)
>> > 
>> > What a mess...
>> 
>> And clearly not according to the spec
>> 
>>   "The second is that platform firmware should list the boot
>> processor
>>    as the first processor entry in the MADT."
>> 
>> Oh well. There are reasons why this is written the way it is.
>
> This is indeed a violation of the ACPI spec and we should modify the
> order in MADT. But this doesn't bring any actual effect as Linux
> already handles this, right?

It brings the effect that we can detect when we are not booting (kexec
case) on the actual boot CPU because then the first enumerated APIC ID
is not the same as the boot CPU APIC ID. No?

> For the BSP APIC ID 0x20, I didn't find out a specific reason why we
> have to do it in that way, but it is still legal.

Linux does not really care in which order the APICs are enumerated.

> We may need to figure out another way to distinguish the kdump kernel.

Having the first enumerated APIC in the MADT as the actual boot CPU is a
sensible and functional way. Everything else including the silly kexec
boot parameter is error prone.

I agree that MADT is error prone too given the fact that not even Intel
can get it right....

Thanks,

        tglx

Re: [patch 37/53] x86/cpu: Detect real BSP on crash kernels

[Zhang, Rui]

On Fri, 2024-01-12 at 16:39 +0100, Thomas Gleixner wrote:
> On Fri, Jan 12 2024 at 09:14, Zhang, Rui wrote:
> > On Wed, 2024-01-10 at 16:14 +0100, Thomas Gleixner wrote:
> > > On Wed, Jan 10 2024 at 15:19, Thomas Gleixner wrote:
> > > > > This is the order in MADT,
> > > > > $ cat apic.dsl  | grep x2Apic
> > > > > [030h 0048   4]          Processor x2Apic ID : 00000010
> > > > > [040h 0064   4]          Processor x2Apic ID : 00000011
> > > ...
> > > > > and this is the order in Linux (from CPU0 to CPUN)
> > > > >       x2APIC ID of logical processor = 0x20 (32)
> > > > >       x2APIC ID of logical processor = 0x10 (16)
> > > > 
> > > > What a mess...
> > > 
> > > And clearly not according to the spec
> > > 
> > >   "The second is that platform firmware should list the boot
> > > processor
> > >    as the first processor entry in the MADT."
> > > 
> > > Oh well. There are reasons why this is written the way it is.
> > 
> > This is indeed a violation of the ACPI spec and we should modify
> > the
> > order in MADT. But this doesn't bring any actual effect as Linux
> > already handles this, right?
> 
> It brings the effect that we can detect when we are not booting
> (kexec
> case) on the actual boot CPU because then the first enumerated APIC
> ID
> is not the same as the boot CPU APIC ID. No?

Right.
I was thinking in the way this patch series does, which just compares
the boot CPU APIC ID and the lowest numbered APIC ID.

> 
> > For the BSP APIC ID 0x20, I didn't find out a specific reason why
> > we
> > have to do it in that way, but it is still legal.
> 
> Linux does not really care in which order the APICs are enumerated.
> 
> > We may need to figure out another way to distinguish the kdump
> > kernel.
> 
> Having the first enumerated APIC in the MADT as the actual boot CPU
> is a
> sensible and functional way. Everything else including the silly
> kexec
> boot parameter is error prone.
> 
> I agree that MADT is error prone too given the fact that not even
> Intel
> can get it right....

For this MTL, I can raise an internal ticket to get it right.

Are there quite some platforms with BSP not listed as the first entry
in MADT?
if so, we still have to live with the kexec boot parameter? :)

thanks,
rui

Re: [patch 37/53] x86/cpu: Detect real BSP on crash kernels

[Thomas Gleixner]

On Sat, Jan 13 2024 at 07:35, Zhang, Rui wrote:
> On Fri, 2024-01-12 at 16:39 +0100, Thomas Gleixner wrote:
>> It brings the effect that we can detect when we are not booting
>> (kexec
>> case) on the actual boot CPU because then the first enumerated APIC
>> ID
>> is not the same as the boot CPU APIC ID. No?
>
> Right.
> I was thinking in the way this patch series does, which just compares
> the boot CPU APIC ID and the lowest numbered APIC ID.

Yes, that's obviously not working when the lowest APIC ID is not the
actual boot CPU.

>> I agree that MADT is error prone too given the fact that not even
>> Intel
>> can get it right....
>
> For this MTL, I can raise an internal ticket to get it right.
>
> Are there quite some platforms with BSP not listed as the first entry
> in MADT?
> if so, we still have to live with the kexec boot parameter? :)

I haven't found one yet, but you might look through the P/E systems to
get a picture.