[PATCH 00/22] x86: Enhance MTRR functionality to support multiple CPUs

[PATCH 00/22] x86: Enhance MTRR functionality to support multiple CPUs

[Simon Glass]

At present MTRRs are mirrored to the secondary CPUs only once, as those
CPUs are started up. But U-Boot may add more MTRRs later, e.g. if it
decides that a video console must be set up.

This series enhances the x86 multi-processor support to allow MTRRs to
be updated at any time. It also updates the 'mtrr' command to support
setting the MTRRs on CPUs other than the boot CPU.


Simon Glass (22):
  x86: mp_init: Switch to livetree
  x86: Move MP code into mp_init
  x86: mp_init: Avoid declarations in header files
  x86: mp_init: Switch parameter names in start_aps()
  x86: mp_init: Drop the num_cpus static variable
  x86: mtrr: Fix 'ensable' typo
  x86: mp_init: Set up the CPU numbers at the start
  x86: mp_init: Adjust bsp_init() to return more information
  x86: cpu: Remove unnecessary #ifdefs
  x86: mp: Support APs waiting for instructions
  global_data: Add a generic global_data flag for SMP state
  x86: Set the SMP flag when MP init is complete
  x86: mp: Allow running functions on multiple CPUs
  x86: mp: Park CPUs before running the OS
  x86: mp: Add iterators for CPUs
  x86: mtrr: Use MP calls to list the MTRRs
  x86: mtrr: Update MTRRs on all CPUs
  x86: mtrr: Add support for writing to MTRRs on any CPU
  x86: mtrr: Update the command to use the new mtrr calls
  x86: mtrr: Restructure so command execution is in one place
  x86: mtrr: Update 'mtrr' to allow setting MTRRs on any CPU
  x86: mtrr: Enhance 'mtrr' command to list MTRRs on any CPU

 arch/x86/cpu/cpu.c                |  63 ++---
 arch/x86/cpu/i386/cpu.c           |  26 +--
 arch/x86/cpu/mp_init.c            | 377 +++++++++++++++++++++++++-----
 arch/x86/cpu/mtrr.c               | 149 ++++++++++++
 arch/x86/include/asm/mp.h         | 118 ++++++++--
 arch/x86/include/asm/mtrr.h       |  51 ++++
 cmd/x86/mtrr.c                    | 148 ++++++++----
 include/asm-generic/global_data.h |   1 +
 include/dm/uclass.h               |   2 +-
 9 files changed, 751 insertions(+), 184 deletions(-)

-- 
2.27.0.rc0.183.gde8f92d652-goog

[PATCH 01/22] x86: mp_init: Switch to livetree

[Simon Glass]

Update this code to use livetree calls instead of flat-tree.

Signed-off-by: Simon Glass <sjg@chromium.org>
---

 arch/x86/cpu/mp_init.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/arch/x86/cpu/mp_init.c b/arch/x86/cpu/mp_init.c
index 7fde4ff7e16..c25d17c6474 100644
--- a/arch/x86/cpu/mp_init.c
+++ b/arch/x86/cpu/mp_init.c
@@ -507,8 +507,7 @@ int mp_init_cpu(struct udevice *cpu, void *unused)
 	 * seq num in the uclass_resolve_seq() during device_probe(). To avoid
 	 * this, set req_seq to the reg number in the device tree in advance.
 	 */
-	cpu->req_seq = fdtdec_get_int(gd->fdt_blob, dev_of_offset(cpu), "reg",
-				      -1);
+	cpu->req_seq = dev_read_u32_default(cpu, "reg", -1);
 	plat->ucode_version = microcode_read_rev();
 	plat->device_id = gd->arch.x86_device;
 
-- 
2.27.0.rc0.183.gde8f92d652-goog

[PATCH 01/22] x86: mp_init: Switch to livetree

[Wolfgang Wallner]

Hi Simon,

-----"Simon Glass" <sjg@chromium.org> schrieb: -----
> Betreff: [PATCH 01/22] x86: mp_init: Switch to livetree
> 
> Update this code to use livetree calls instead of flat-tree.
> 
> Signed-off-by: Simon Glass <sjg@chromium.org>
> ---
> 
>  arch/x86/cpu/mp_init.c | 3 +--
>  1 file changed, 1 insertion(+), 2 deletions(-)
> 
> diff --git a/arch/x86/cpu/mp_init.c b/arch/x86/cpu/mp_init.c
> index 7fde4ff7e16..c25d17c6474 100644
> --- a/arch/x86/cpu/mp_init.c
> +++ b/arch/x86/cpu/mp_init.c
> @@ -507,8 +507,7 @@ int mp_init_cpu(struct udevice *cpu, void *unused)
>  	 * seq num in the uclass_resolve_seq() during device_probe(). To avoid
>  	 * this, set req_seq to the reg number in the device tree in advance.
>  	 */
> -	cpu->req_seq = fdtdec_get_int(gd->fdt_blob, dev_of_offset(cpu), "reg",
> -				      -1);
> +	cpu->req_seq = dev_read_u32_default(cpu, "reg", -1);
>  	plat->ucode_version = microcode_read_rev();
>  	plat->device_id = gd->arch.x86_device;
>  
> -- 
> 2.27.0.rc0.183.gde8f92d652-goog
> 

Reviewed-by: Wolfgang Wallner <wolfgang.wallner@br-automation.com>

[PATCH 02/22] x86: Move MP code into mp_init

[Simon Glass]

At present the 'flight plan' for CPUs is passed into mp_init. But it is
always the same. Move it into the mp_init file so everything is in one
place. Also drop the SMI function since it does nothing. If we implement
SMIs, more refactoring will be needed anyway.

Signed-off-by: Simon Glass <sjg@chromium.org>
---

 arch/x86/cpu/i386/cpu.c   | 24 +++++-------------------
 arch/x86/cpu/mp_init.c    | 22 ++++++++++------------
 arch/x86/include/asm/mp.h | 17 +----------------
 3 files changed, 16 insertions(+), 47 deletions(-)

diff --git a/arch/x86/cpu/i386/cpu.c b/arch/x86/cpu/i386/cpu.c
index d27324cb4e2..9809ac51117 100644
--- a/arch/x86/cpu/i386/cpu.c
+++ b/arch/x86/cpu/i386/cpu.c
@@ -21,6 +21,7 @@
 #include <common.h>
 #include <cpu_func.h>
 #include <init.h>
+#include <log.h>
 #include <malloc.h>
 #include <spl.h>
 #include <asm/control_regs.h>
@@ -626,29 +627,14 @@ int cpu_jump_to_64bit_uboot(ulong target)
 }
 
 #ifdef CONFIG_SMP
-static int enable_smis(struct udevice *cpu, void *unused)
-{
-	return 0;
-}
-
-static struct mp_flight_record mp_steps[] = {
-	MP_FR_BLOCK_APS(mp_init_cpu, NULL, mp_init_cpu, NULL),
-	/* Wait for APs to finish initialization before proceeding */
-	MP_FR_BLOCK_APS(NULL, NULL, enable_smis, NULL),
-};
-
 int x86_mp_init(void)
 {
-	struct mp_params mp_params;
-
-	mp_params.parallel_microcode_load = 0,
-	mp_params.flight_plan = &mp_steps[0];
-	mp_params.num_records = ARRAY_SIZE(mp_steps);
-	mp_params.microcode_pointer = 0;
+	int ret;
 
-	if (mp_init(&mp_params)) {
+	ret = mp_init();
+	if (ret) {
 		printf("Warning: MP init failure\n");
-		return -EIO;
+		return log_ret(ret);
 	}
 
 	return 0;
diff --git a/arch/x86/cpu/mp_init.c b/arch/x86/cpu/mp_init.c
index c25d17c6474..831fd7035d1 100644
--- a/arch/x86/cpu/mp_init.c
+++ b/arch/x86/cpu/mp_init.c
@@ -41,6 +41,9 @@ struct saved_msr {
 	uint32_t hi;
 } __packed;
 
+static struct mp_flight_record mp_steps[] = {
+	MP_FR_BLOCK_APS(mp_init_cpu, NULL, mp_init_cpu, NULL),
+};
 
 struct mp_flight_plan {
 	int num_records;
@@ -372,7 +375,7 @@ static int start_aps(int ap_count, atomic_t *num_aps)
 	return 0;
 }
 
-static int bsp_do_flight_plan(struct udevice *cpu, struct mp_params *mp_params)
+static int bsp_do_flight_plan(struct udevice *cpu, struct mp_flight_plan *plan)
 {
 	int i;
 	int ret = 0;
@@ -380,8 +383,8 @@ static int bsp_do_flight_plan(struct udevice *cpu, struct mp_params *mp_params)
 	const int step_us = 100;
 	int num_aps = num_cpus - 1;
 
-	for (i = 0; i < mp_params->num_records; i++) {
-		struct mp_flight_record *rec = &mp_params->flight_plan[i];
+	for (i = 0; i < plan->num_records; i++) {
+		struct mp_flight_record *rec = &plan->records[i];
 
 		/* Wait for APs if the record is not released */
 		if (atomic_read(&rec->barrier) == 0) {
@@ -420,7 +423,7 @@ static int init_bsp(struct udevice **devp)
 	return 0;
 }
 
-int mp_init(struct mp_params *p)
+int mp_init(void)
 {
 	int num_aps;
 	atomic_t *ap_count;
@@ -445,11 +448,6 @@ int mp_init(struct mp_params *p)
 		return ret;
 	}
 
-	if (p == NULL || p->flight_plan == NULL || p->num_records < 1) {
-		printf("Invalid MP parameters\n");
-		return -EINVAL;
-	}
-
 	num_cpus = cpu_get_count(cpu);
 	if (num_cpus < 0) {
 		debug("Cannot get number of CPUs: err=%d\n", num_cpus);
@@ -464,8 +462,8 @@ int mp_init(struct mp_params *p)
 		debug("Warning: Device tree does not describe all CPUs. Extra ones will not be started correctly\n");
 
 	/* Copy needed parameters so that APs have a reference to the plan */
-	mp_info.num_records = p->num_records;
-	mp_info.records = p->flight_plan;
+	mp_info.num_records = ARRAY_SIZE(mp_steps);
+	mp_info.records = mp_steps;
 
 	/* Load the SIPI vector */
 	ret = load_sipi_vector(&ap_count, num_cpus);
@@ -489,7 +487,7 @@ int mp_init(struct mp_params *p)
 	}
 
 	/* Walk the flight plan for the BSP */
-	ret = bsp_do_flight_plan(cpu, p);
+	ret = bsp_do_flight_plan(cpu, &mp_info);
 	if (ret) {
 		debug("CPU init failed: err=%d\n", ret);
 		return ret;
diff --git a/arch/x86/include/asm/mp.h b/arch/x86/include/asm/mp.h
index 9dddf88b5a1..db02904ecb5 100644
--- a/arch/x86/include/asm/mp.h
+++ b/arch/x86/include/asm/mp.h
@@ -51,21 +51,6 @@ struct mp_flight_record {
 #define MP_FR_NOBLOCK_APS(ap_func, ap_arg, bsp_func, bsp_arg) \
 	MP_FLIGHT_RECORD(1, ap_func, ap_arg, bsp_func, bsp_arg)
 
-/*
- * The mp_params structure provides the arguments to the mp subsystem
- * for bringing up APs.
- *
- * At present this is overkill for U-Boot, but it may make it easier to add
- * SMM support.
- */
-struct mp_params {
-	int parallel_microcode_load;
-	const void *microcode_pointer;
-	/* Flight plan  for APs and BSP */
-	struct mp_flight_record *flight_plan;
-	int num_records;
-};
-
 /*
  * mp_init() will set up the SIPI vector and bring up the APs according to
  * mp_params. Each flight record will be executed according to the plan. Note
@@ -85,7 +70,7 @@ struct mp_params {
  *
  * mp_init() returns < 0 on error, 0 on success.
  */
-int mp_init(struct mp_params *params);
+int mp_init(void);
 
 /* Probes the CPU device */
 int mp_init_cpu(struct udevice *cpu, void *unused);
-- 
2.27.0.rc0.183.gde8f92d652-goog

[PATCH 02/22] x86: Move MP code into mp_init

[Wolfgang Wallner]

Hi Simon,

-----"Simon Glass" <sjg@chromium.org> schrieb: -----
> Betreff: [PATCH 02/22] x86: Move MP code into mp_init
> 
> At present the 'flight plan' for CPUs is passed into mp_init. But it is
> always the same. Move it into the mp_init file so everything is in one
> place. Also drop the SMI function since it does nothing. If we implement
> SMIs, more refactoring will be needed anyway.
> 
> Signed-off-by: Simon Glass <sjg@chromium.org>
> ---
> 
>  arch/x86/cpu/i386/cpu.c   | 24 +++++-------------------
>  arch/x86/cpu/mp_init.c    | 22 ++++++++++------------
>  arch/x86/include/asm/mp.h | 17 +----------------
>  3 files changed, 16 insertions(+), 47 deletions(-)
>

Reviewed-by: Wolfgang Wallner <wolfgang.wallner@br-automation.com>

[PATCH 03/22] x86: mp_init: Avoid declarations in header files

[Simon Glass]

The functions used by the flight plan are declared in the header file but
are not used in any other file.

Move the flight plan steps down to just above where it is used so that we
can make these function static.

Signed-off-by: Simon Glass <sjg@chromium.org>
---

 arch/x86/cpu/mp_init.c    | 40 +++++++++++++++++++--------------------
 arch/x86/include/asm/mp.h |  3 ---
 2 files changed, 20 insertions(+), 23 deletions(-)

diff --git a/arch/x86/cpu/mp_init.c b/arch/x86/cpu/mp_init.c
index 831fd7035d1..e77d7f2cd6c 100644
--- a/arch/x86/cpu/mp_init.c
+++ b/arch/x86/cpu/mp_init.c
@@ -41,10 +41,6 @@ struct saved_msr {
 	uint32_t hi;
 } __packed;
 
-static struct mp_flight_record mp_steps[] = {
-	MP_FR_BLOCK_APS(mp_init_cpu, NULL, mp_init_cpu, NULL),
-};
-
 struct mp_flight_plan {
 	int num_records;
 	struct mp_flight_record *records;
@@ -423,6 +419,26 @@ static int init_bsp(struct udevice **devp)
 	return 0;
 }
 
+static int mp_init_cpu(struct udevice *cpu, void *unused)
+{
+	struct cpu_platdata *plat = dev_get_parent_platdata(cpu);
+
+	/*
+	 * Multiple APs are brought up simultaneously and they may get the same
+	 * seq num in the uclass_resolve_seq() during device_probe(). To avoid
+	 * this, set req_seq to the reg number in the device tree in advance.
+	 */
+	cpu->req_seq = dev_read_u32_default(cpu, "reg", -1);
+	plat->ucode_version = microcode_read_rev();
+	plat->device_id = gd->arch.x86_device;
+
+	return device_probe(cpu);
+}
+
+static struct mp_flight_record mp_steps[] = {
+	MP_FR_BLOCK_APS(mp_init_cpu, NULL, mp_init_cpu, NULL),
+};
+
 int mp_init(void)
 {
 	int num_aps;
@@ -495,19 +511,3 @@ int mp_init(void)
 
 	return 0;
 }
-
-int mp_init_cpu(struct udevice *cpu, void *unused)
-{
-	struct cpu_platdata *plat = dev_get_parent_platdata(cpu);
-
-	/*
-	 * Multiple APs are brought up simultaneously and they may get the same
-	 * seq num in the uclass_resolve_seq() during device_probe(). To avoid
-	 * this, set req_seq to the reg number in the device tree in advance.
-	 */
-	cpu->req_seq = dev_read_u32_default(cpu, "reg", -1);
-	plat->ucode_version = microcode_read_rev();
-	plat->device_id = gd->arch.x86_device;
-
-	return device_probe(cpu);
-}
diff --git a/arch/x86/include/asm/mp.h b/arch/x86/include/asm/mp.h
index db02904ecb5..94af819ad9a 100644
--- a/arch/x86/include/asm/mp.h
+++ b/arch/x86/include/asm/mp.h
@@ -72,9 +72,6 @@ struct mp_flight_record {
  */
 int mp_init(void);
 
-/* Probes the CPU device */
-int mp_init_cpu(struct udevice *cpu, void *unused);
-
 /* Set up additional CPUs */
 int x86_mp_init(void);
 
-- 
2.27.0.rc0.183.gde8f92d652-goog

[PATCH 03/22] x86: mp_init: Avoid declarations in header files

[Wolfgang Wallner]

Hi Simon,

-----"Simon Glass" <sjg@chromium.org> schrieb: -----
> Betreff: [PATCH 03/22] x86: mp_init: Avoid declarations in header files
> 
> The functions used by the flight plan are declared in the header file but
> are not used in any other file.
> 
> Move the flight plan steps down to just above where it is used so that we
> can make these function static.
> 
> Signed-off-by: Simon Glass <sjg@chromium.org>
> ---
> 
>  arch/x86/cpu/mp_init.c    | 40 +++++++++++++++++++--------------------
>  arch/x86/include/asm/mp.h |  3 ---
>  2 files changed, 20 insertions(+), 23 deletions(-)
> 

Reviewed-by: Wolfgang Wallner <wolfgang.wallner@br-automation.com>

[PATCH 04/22] x86: mp_init: Switch parameter names in start_aps()

[Simon Glass]

These parameters are named differently from elsewhere in this file. Switch
them to avoid confusion.

Signed-off-by: Simon Glass <sjg@chromium.org>
---

 arch/x86/cpu/mp_init.c | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

diff --git a/arch/x86/cpu/mp_init.c b/arch/x86/cpu/mp_init.c
index e77d7f2cd6c..4b678cde313 100644
--- a/arch/x86/cpu/mp_init.c
+++ b/arch/x86/cpu/mp_init.c
@@ -308,13 +308,13 @@ static int apic_wait_timeout(int total_delay, const char *msg)
 	return 0;
 }
 
-static int start_aps(int ap_count, atomic_t *num_aps)
+static int start_aps(int num_aps, atomic_t *ap_count)
 {
 	int sipi_vector;
 	/* Max location is 4KiB below 1MiB */
 	const int max_vector_loc = ((1 << 20) - (1 << 12)) >> 12;
 
-	if (ap_count == 0)
+	if (num_aps == 0)
 		return 0;
 
 	/* The vector is sent as a 4k aligned address in one byte */
@@ -326,7 +326,7 @@ static int start_aps(int ap_count, atomic_t *num_aps)
 		return -ENOSPC;
 	}
 
-	debug("Attempting to start %d APs\n", ap_count);
+	debug("Attempting to start %d APs\n", num_aps);
 
 	if (apic_wait_timeout(1000, "ICR not to be busy"))
 		return -ETIMEDOUT;
@@ -349,7 +349,7 @@ static int start_aps(int ap_count, atomic_t *num_aps)
 		return -ETIMEDOUT;
 
 	/* Wait for CPUs to check in up to 200 us */
-	wait_for_aps(num_aps, ap_count, 200, 15);
+	wait_for_aps(ap_count, num_aps, 200, 15);
 
 	/* Send 2nd SIPI */
 	if (apic_wait_timeout(1000, "ICR not to be busy"))
@@ -362,9 +362,9 @@ static int start_aps(int ap_count, atomic_t *num_aps)
 		return -ETIMEDOUT;
 
 	/* Wait for CPUs to check in */
-	if (wait_for_aps(num_aps, ap_count, 10000, 50)) {
+	if (wait_for_aps(ap_count, num_aps, 10000, 50)) {
 		debug("Not all APs checked in: %d/%d\n",
-		      atomic_read(num_aps), ap_count);
+		      atomic_read(ap_count), num_aps);
 		return -EIO;
 	}
 
-- 
2.27.0.rc0.183.gde8f92d652-goog

[PATCH 04/22] x86: mp_init: Switch parameter names in start_aps()

[Wolfgang Wallner]

Hi Simon,

-----"Simon Glass" <sjg@chromium.org> schrieb: -----
> Betreff: [PATCH 04/22] x86: mp_init: Switch parameter names in start_aps()
> 
> These parameters are named differently from elsewhere in this file. Switch
> them to avoid confusion.
> 
> Signed-off-by: Simon Glass <sjg@chromium.org>
> ---
> 
>  arch/x86/cpu/mp_init.c | 12 ++++++------
>  1 file changed, 6 insertions(+), 6 deletions(-)
> 
> diff --git a/arch/x86/cpu/mp_init.c b/arch/x86/cpu/mp_init.c
> index e77d7f2cd6c..4b678cde313 100644
> --- a/arch/x86/cpu/mp_init.c
> +++ b/arch/x86/cpu/mp_init.c
> @@ -308,13 +308,13 @@ static int apic_wait_timeout(int total_delay, const char *msg)
>  	return 0;
>  }
>  
> -static int start_aps(int ap_count, atomic_t *num_aps)
> +static int start_aps(int num_aps, atomic_t *ap_count)

A description of this function and its parameters would be really helpful.
I got confused trying to understand what it did before the patch, what it
does now, and what the original intent was ...

>  {
>  	int sipi_vector;
>  	/* Max location is 4KiB below 1MiB */
>  	const int max_vector_loc = ((1 << 20) - (1 << 12)) >> 12;
>  
> -	if (ap_count == 0)
> +	if (num_aps == 0)
>  		return 0;
>  
>  	/* The vector is sent as a 4k aligned address in one byte */
> @@ -326,7 +326,7 @@ static int start_aps(int ap_count, atomic_t *num_aps)
>  		return -ENOSPC;
>  	}
>  
> -	debug("Attempting to start %d APs\n", ap_count);
> +	debug("Attempting to start %d APs\n", num_aps);
>  
>  	if (apic_wait_timeout(1000, "ICR not to be busy"))
>  		return -ETIMEDOUT;
> @@ -349,7 +349,7 @@ static int start_aps(int ap_count, atomic_t *num_aps)
>  		return -ETIMEDOUT;
>  
>  	/* Wait for CPUs to check in up to 200 us */
> -	wait_for_aps(num_aps, ap_count, 200, 15);
> +	wait_for_aps(ap_count, num_aps, 200, 15);
>  
>  	/* Send 2nd SIPI */
>  	if (apic_wait_timeout(1000, "ICR not to be busy"))
> @@ -362,9 +362,9 @@ static int start_aps(int ap_count, atomic_t *num_aps)
>  		return -ETIMEDOUT;
>  
>  	/* Wait for CPUs to check in */
> -	if (wait_for_aps(num_aps, ap_count, 10000, 50)) {
> +	if (wait_for_aps(ap_count, num_aps, 10000, 50)) {
>  		debug("Not all APs checked in: %d/%d\n",
> -		      atomic_read(num_aps), ap_count);
> +		      atomic_read(ap_count), num_aps);
>  		return -EIO;
>  	}
>  
> -- 
> 2.27.0.rc0.183.gde8f92d652-goog

regards, Wolfgang

[PATCH 05/22] x86: mp_init: Drop the num_cpus static variable

[Simon Glass]

This does not need to be global across all functions in this file. Pass a
parameter instead.

Signed-off-by: Simon Glass <sjg@chromium.org>
---

 arch/x86/cpu/mp_init.c | 22 +++++++++++++++-------
 1 file changed, 15 insertions(+), 7 deletions(-)

diff --git a/arch/x86/cpu/mp_init.c b/arch/x86/cpu/mp_init.c
index 4b678cde313..bb39fd30d18 100644
--- a/arch/x86/cpu/mp_init.c
+++ b/arch/x86/cpu/mp_init.c
@@ -31,9 +31,6 @@
 
 DECLARE_GLOBAL_DATA_PTR;
 
-/* Total CPUs include BSP */
-static int num_cpus;
-
 /* This also needs to match the sipi.S assembly code for saved MSR encoding */
 struct saved_msr {
 	uint32_t index;
@@ -371,13 +368,23 @@ static int start_aps(int num_aps, atomic_t *ap_count)
 	return 0;
 }
 
-static int bsp_do_flight_plan(struct udevice *cpu, struct mp_flight_plan *plan)
+/**
+ * bsp_do_flight_plan() - Do the flight plan on all CPUs
+ *
+ * This runs the flight plan on the main CPU used to boot U-Boot
+ *
+ * @cpu: Device for the main CPU
+ * @plan: Flight plan to run
+ * @num_aps: Number of APs (CPUs other than the BSP)
+ * @returns 0 on success, -ETIMEDOUT if an AP failed to come up
+ */
+static int bsp_do_flight_plan(struct udevice *cpu, struct mp_flight_plan *plan,
+			      int num_aps)
 {
 	int i;
 	int ret = 0;
 	const int timeout_us = 100000;
 	const int step_us = 100;
-	int num_aps = num_cpus - 1;
 
 	for (i = 0; i < plan->num_records; i++) {
 		struct mp_flight_record *rec = &plan->records[i];
@@ -397,6 +404,7 @@ static int bsp_do_flight_plan(struct udevice *cpu, struct mp_flight_plan *plan)
 
 		release_barrier(&rec->barrier);
 	}
+
 	return ret;
 }
 
@@ -441,7 +449,7 @@ static struct mp_flight_record mp_steps[] = {
 
 int mp_init(void)
 {
-	int num_aps;
+	int num_aps, num_cpus;
 	atomic_t *ap_count;
 	struct udevice *cpu;
 	int ret;
@@ -503,7 +511,7 @@ int mp_init(void)
 	}
 
 	/* Walk the flight plan for the BSP */
-	ret = bsp_do_flight_plan(cpu, &mp_info);
+	ret = bsp_do_flight_plan(cpu, &mp_info, num_aps);
 	if (ret) {
 		debug("CPU init failed: err=%d\n", ret);
 		return ret;
-- 
2.27.0.rc0.183.gde8f92d652-goog

[PATCH 05/22] x86: mp_init: Drop the num_cpus static variable

[Wolfgang Wallner]

Hi Simon,

-----"Simon Glass" <sjg@chromium.org> schrieb: -----
>Betreff: [PATCH 05/22] x86: mp_init: Drop the num_cpus static variable
>
>This does not need to be global across all functions in this file. Pass a
>parameter instead.
>
>Signed-off-by: Simon Glass <sjg@chromium.org>
>---
>
> arch/x86/cpu/mp_init.c | 22 +++++++++++++++-------
> 1 file changed, 15 insertions(+), 7 deletions(-)
>

Reviewed-by: Wolfgang Wallner <wolfgang.wallner@br-automation.com>

[PATCH 06/22] x86: mtrr: Fix 'ensable' typo

[Simon Glass]

Fix a typo in the command help.

Signed-off-by: Simon Glass <sjg@chromium.org>
---

 cmd/x86/mtrr.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/cmd/x86/mtrr.c b/cmd/x86/mtrr.c
index 084d7315f43..5d25c5802af 100644
--- a/cmd/x86/mtrr.c
+++ b/cmd/x86/mtrr.c
@@ -135,5 +135,5 @@ U_BOOT_CMD(
 	"set <reg> <type> <start> <size>   - set a register\n"
 	"\t<type> is Uncacheable, Combine, Through, Protect, Back\n"
 	"disable <reg>      - disable a register\n"
-	"ensable <reg>      - enable a register"
+	"enable <reg>       - enable a register"
 );
-- 
2.27.0.rc0.183.gde8f92d652-goog

[PATCH 06/22] x86: mtrr: Fix 'ensable' typo

[Wolfgang Wallner]

Hi Simon,

-----"Simon Glass" <sjg@chromium.org> schrieb: -----
> Betreff: [PATCH 06/22] x86: mtrr: Fix 'ensable' typo
> 
> Fix a typo in the command help.
> 
> Signed-off-by: Simon Glass <sjg@chromium.org>
> ---
> 
>  cmd/x86/mtrr.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/cmd/x86/mtrr.c b/cmd/x86/mtrr.c
> index 084d7315f43..5d25c5802af 100644
> --- a/cmd/x86/mtrr.c
> +++ b/cmd/x86/mtrr.c
> @@ -135,5 +135,5 @@ U_BOOT_CMD(
>  	"set <reg> <type> <start> <size>   - set a register\n"
>  	"\t<type> is Uncacheable, Combine, Through, Protect, Back\n"
>  	"disable <reg>      - disable a register\n"
> -	"ensable <reg>      - enable a register"
> +	"enable <reg>       - enable a register"
>  );
> -- 
> 2.27.0.rc0.183.gde8f92d652-goog

Reviewed-by: Wolfgang Wallner <wolfgang.wallner@br-automation.com>

[PATCH 07/22] x86: mp_init: Set up the CPU numbers at the start

[Simon Glass]

At present each CPU is given a number when it starts itself up. While this
saves a tiny amount of time by doing the device-tree read in parallel, it
is confusing that the numbering happens on the fly.

Move this code into mp_init() and do it at the start.

Signed-off-by: Simon Glass <sjg@chromium.org>
---

 arch/x86/cpu/mp_init.c | 21 +++++++++------------
 1 file changed, 9 insertions(+), 12 deletions(-)

diff --git a/arch/x86/cpu/mp_init.c b/arch/x86/cpu/mp_init.c
index bb39fd30d18..8b4c72bbcf2 100644
--- a/arch/x86/cpu/mp_init.c
+++ b/arch/x86/cpu/mp_init.c
@@ -431,12 +431,6 @@ static int mp_init_cpu(struct udevice *cpu, void *unused)
 {
 	struct cpu_platdata *plat = dev_get_parent_platdata(cpu);
 
-	/*
-	 * Multiple APs are brought up simultaneously and they may get the same
-	 * seq num in the uclass_resolve_seq() during device_probe(). To avoid
-	 * this, set req_seq to the reg number in the device tree in advance.
-	 */
-	cpu->req_seq = dev_read_u32_default(cpu, "reg", -1);
 	plat->ucode_version = microcode_read_rev();
 	plat->device_id = gd->arch.x86_device;
 
@@ -452,13 +446,8 @@ int mp_init(void)
 	int num_aps, num_cpus;
 	atomic_t *ap_count;
 	struct udevice *cpu;
-	int ret;
-
-	/* This will cause the CPUs devices to be bound */
 	struct uclass *uc;
-	ret = uclass_get(UCLASS_CPU, &uc);
-	if (ret)
-		return ret;
+	int ret;
 
 	if (IS_ENABLED(CONFIG_QFW)) {
 		ret = qemu_cpu_fixup();
@@ -466,6 +455,14 @@ int mp_init(void)
 			return ret;
 	}
 
+	/*
+	 * Multiple APs are brought up simultaneously and they may get the same
+	 * seq num in the uclass_resolve_seq() during device_probe(). To avoid
+	 * this, set req_seq to the reg number in the device tree in advance.
+	 */
+	uclass_id_foreach_dev(UCLASS_CPU, cpu, uc)
+		cpu->req_seq = dev_read_u32_default(cpu, "reg", -1);
+
 	ret = init_bsp(&cpu);
 	if (ret) {
 		debug("Cannot init boot CPU: err=%d\n", ret);
-- 
2.27.0.rc0.183.gde8f92d652-goog

[PATCH 07/22] x86: mp_init: Set up the CPU numbers at the start

[Wolfgang Wallner]

Hi Simon,

-----"Simon Glass" <sjg@chromium.org> schrieb: -----
> Betreff: [PATCH 07/22] x86: mp_init: Set up the CPU numbers at the start
> 
> At present each CPU is given a number when it starts itself up. While this
> saves a tiny amount of time by doing the device-tree read in parallel, it
> is confusing that the numbering happens on the fly.
> 
> Move this code into mp_init() and do it at the start.
> 
> Signed-off-by: Simon Glass <sjg@chromium.org>
> ---
> 
>  arch/x86/cpu/mp_init.c | 21 +++++++++------------
>  1 file changed, 9 insertions(+), 12 deletions(-)
> 

Reviewed-by: Wolfgang Wallner <wolfgang.wallner@br-automation.com>

[PATCH 08/22] x86: mp_init: Adjust bsp_init() to return more information

[Simon Glass]

This function is misnamed since it does not actually init the BSP. Also
it is convenient to adjust it to return a little more information.

Rename and update the function, to allow it to return the BSP CPU device
and number, as well as the total number of CPUs.

Signed-off-by: Simon Glass <sjg@chromium.org>
---

 arch/x86/cpu/mp_init.c | 35 ++++++++++++++++++++++-------------
 include/dm/uclass.h    |  2 +-
 2 files changed, 23 insertions(+), 14 deletions(-)

diff --git a/arch/x86/cpu/mp_init.c b/arch/x86/cpu/mp_init.c
index 8b4c72bbcf2..ccb68b8b89b 100644
--- a/arch/x86/cpu/mp_init.c
+++ b/arch/x86/cpu/mp_init.c
@@ -408,9 +408,17 @@ static int bsp_do_flight_plan(struct udevice *cpu, struct mp_flight_plan *plan,
 	return ret;
 }
 
-static int init_bsp(struct udevice **devp)
+/**
+ * get_bsp() - Get information about the bootstrap processor
+ *
+ * @devp: If non-NULL, returns CPU device corresponding to the BSP
+ * @cpu_countp: If non-NULL, returns the total number of CPUs
+ * @return CPU number of the BSP
+ */
+static int get_bsp(struct udevice **devp, int *cpu_countp)
 {
 	char processor_name[CPU_MAX_NAME_LEN];
+	struct udevice *dev;
 	int apic_id;
 	int ret;
 
@@ -418,13 +426,20 @@ static int init_bsp(struct udevice **devp)
 	debug("CPU: %s\n", processor_name);
 
 	apic_id = lapicid();
-	ret = find_cpu_by_apic_id(apic_id, devp);
-	if (ret) {
+	ret = find_cpu_by_apic_id(apic_id, &dev);
+	if (ret < 0) {
 		printf("Cannot find boot CPU, APIC ID %d\n", apic_id);
 		return ret;
 	}
+	ret = cpu_get_count(dev);
+	if (ret < 0)
+		return log_msg_ret("count", ret);
+	if (devp)
+		*devp = dev;
+	if (cpu_countp)
+		*cpu_countp = ret;
 
-	return 0;
+	return dev->req_seq;
 }
 
 static int mp_init_cpu(struct udevice *cpu, void *unused)
@@ -463,24 +478,18 @@ int mp_init(void)
 	uclass_id_foreach_dev(UCLASS_CPU, cpu, uc)
 		cpu->req_seq = dev_read_u32_default(cpu, "reg", -1);
 
-	ret = init_bsp(&cpu);
-	if (ret) {
+	ret = get_bsp(&cpu, &num_cpus);
+	if (ret < 0) {
 		debug("Cannot init boot CPU: err=%d\n", ret);
 		return ret;
 	}
 
-	num_cpus = cpu_get_count(cpu);
-	if (num_cpus < 0) {
-		debug("Cannot get number of CPUs: err=%d\n", num_cpus);
-		return num_cpus;
-	}
-
 	if (num_cpus < 2)
 		debug("Warning: Only 1 CPU is detected\n");
 
 	ret = check_cpu_devices(num_cpus);
 	if (ret)
-		debug("Warning: Device tree does not describe all CPUs. Extra ones will not be started correctly\n");
+		log_warning("Warning: Device tree does not describe all CPUs. Extra ones will not be started correctly\n");
 
 	/* Copy needed parameters so that APs have a reference to the plan */
 	mp_info.num_records = ARRAY_SIZE(mp_steps);
diff --git a/include/dm/uclass.h b/include/dm/uclass.h
index 70fca79b449..67ff7466c86 100644
--- a/include/dm/uclass.h
+++ b/include/dm/uclass.h
@@ -390,7 +390,7 @@ int uclass_resolve_seq(struct udevice *dev);
  * @id: enum uclass_id ID to use
  * @pos: struct udevice * to hold the current device. Set to NULL when there
  * are no more devices.
- * @uc: temporary uclass variable (struct udevice *)
+ * @uc: temporary uclass variable (struct uclass *)
  */
 #define uclass_id_foreach_dev(id, pos, uc) \
 	if (!uclass_get(id, &uc)) \
-- 
2.27.0.rc0.183.gde8f92d652-goog

[PATCH 08/22] x86: mp_init: Adjust bsp_init() to return more information

[Wolfgang Wallner]

Hi Simon,

-----"Simon Glass" <sjg@chromium.org> schrieb: -----
> Betreff: [PATCH 08/22] x86: mp_init: Adjust bsp_init() to return more information
> 
> This function is misnamed since it does not actually init the BSP. Also
> it is convenient to adjust it to return a little more information.
> 
> Rename and update the function, to allow it to return the BSP CPU device
> and number, as well as the total number of CPUs.
> 
> Signed-off-by: Simon Glass <sjg@chromium.org>
> ---
> 
>  arch/x86/cpu/mp_init.c | 35 ++++++++++++++++++++++-------------
>  include/dm/uclass.h    |  2 +-
>  2 files changed, 23 insertions(+), 14 deletions(-)
> 
> diff --git a/arch/x86/cpu/mp_init.c b/arch/x86/cpu/mp_init.c
> index 8b4c72bbcf2..ccb68b8b89b 100644
> --- a/arch/x86/cpu/mp_init.c
> +++ b/arch/x86/cpu/mp_init.c
> @@ -408,9 +408,17 @@ static int bsp_do_flight_plan(struct udevice *cpu, struct mp_flight_plan *plan,
>  	return ret;
>  }
>  
> -static int init_bsp(struct udevice **devp)
> +/**
> + * get_bsp() - Get information about the bootstrap processor
> + *
> + * @devp: If non-NULL, returns CPU device corresponding to the BSP
> + * @cpu_countp: If non-NULL, returns the total number of CPUs
> + * @return CPU number of the BSP

Nit: As it could also return an error, I would add ", -ve on error"

> + */
> +static int get_bsp(struct udevice **devp, int *cpu_countp)
>  {
>  	char processor_name[CPU_MAX_NAME_LEN];
> +	struct udevice *dev;
>  	int apic_id;
>  	int ret;
>  
> @@ -418,13 +426,20 @@ static int init_bsp(struct udevice **devp)
>  	debug("CPU: %s\n", processor_name);
>  
>  	apic_id = lapicid();
> -	ret = find_cpu_by_apic_id(apic_id, devp);
> -	if (ret) {
> +	ret = find_cpu_by_apic_id(apic_id, &dev);
> +	if (ret < 0) {
>  		printf("Cannot find boot CPU, APIC ID %d\n", apic_id);
>  		return ret;
>  	}
> +	ret = cpu_get_count(dev);
> +	if (ret < 0)
> +		return log_msg_ret("count", ret);
> +	if (devp)
> +		*devp = dev;
> +	if (cpu_countp)
> +		*cpu_countp = ret;
>  
> -	return 0;
> +	return dev->req_seq;
>  }
>  
>  static int mp_init_cpu(struct udevice *cpu, void *unused)
> @@ -463,24 +478,18 @@ int mp_init(void)
>  	uclass_id_foreach_dev(UCLASS_CPU, cpu, uc)
>  		cpu->req_seq = dev_read_u32_default(cpu, "reg", -1);
>  
> -	ret = init_bsp(&cpu);
> -	if (ret) {
> +	ret = get_bsp(&cpu, &num_cpus);
> +	if (ret < 0) {
>  		debug("Cannot init boot CPU: err=%d\n", ret);
>  		return ret;
>  	}
>  
> -	num_cpus = cpu_get_count(cpu);
> -	if (num_cpus < 0) {
> -		debug("Cannot get number of CPUs: err=%d\n", num_cpus);
> -		return num_cpus;
> -	}
> -
>  	if (num_cpus < 2)
>  		debug("Warning: Only 1 CPU is detected\n");
>  
>  	ret = check_cpu_devices(num_cpus);
>  	if (ret)
> -		debug("Warning: Device tree does not describe all CPUs. Extra ones will not be started correctly\n");
> +		log_warning("Warning: Device tree does not describe all CPUs. Extra ones will not be started correctly\n");
>  
>  	/* Copy needed parameters so that APs have a reference to the plan */
>  	mp_info.num_records = ARRAY_SIZE(mp_steps);
> diff --git a/include/dm/uclass.h b/include/dm/uclass.h
> index 70fca79b449..67ff7466c86 100644
> --- a/include/dm/uclass.h
> +++ b/include/dm/uclass.h
> @@ -390,7 +390,7 @@ int uclass_resolve_seq(struct udevice *dev);
>   * @id: enum uclass_id ID to use
>   * @pos: struct udevice * to hold the current device. Set to NULL when there
>   * are no more devices.
> - * @uc: temporary uclass variable (struct udevice *)
> + * @uc: temporary uclass variable (struct uclass *)
>   */
>  #define uclass_id_foreach_dev(id, pos, uc) \
>  	if (!uclass_get(id, &uc)) \
> -- 
> 2.27.0.rc0.183.gde8f92d652-goog

Reviewed-by: Wolfgang Wallner <wolfgang.wallner@br-automation.com>

[PATCH 09/22] x86: cpu: Remove unnecessary #ifdefs

[Simon Glass]

Drop some #ifdefs that are not needed or can be converted to compile-time
checks.

Signed-off-by: Simon Glass <sjg@chromium.org>
---

 arch/x86/cpu/cpu.c      | 58 ++++++++++++++++++++---------------------
 arch/x86/cpu/i386/cpu.c |  2 --
 2 files changed, 28 insertions(+), 32 deletions(-)

diff --git a/arch/x86/cpu/cpu.c b/arch/x86/cpu/cpu.c
index 23a4d633d2d..d0720fb7fb5 100644
--- a/arch/x86/cpu/cpu.c
+++ b/arch/x86/cpu/cpu.c
@@ -66,10 +66,8 @@ static const char *const x86_vendor_name[] = {
 
 int __weak x86_cleanup_before_linux(void)
 {
-#ifdef CONFIG_BOOTSTAGE_STASH
 	bootstage_stash((void *)CONFIG_BOOTSTAGE_STASH_ADDR,
 			CONFIG_BOOTSTAGE_STASH_SIZE);
-#endif
 
 	return 0;
 }
@@ -200,18 +198,19 @@ int last_stage_init(void)
 
 	write_tables();
 
-#ifdef CONFIG_GENERATE_ACPI_TABLE
-	fadt = acpi_find_fadt();
+	if (IS_ENABLED(CONFIG_GENERATE_ACPI_TABLE)) {
+		fadt = acpi_find_fadt();
 
-	/* Don't touch ACPI hardware on HW reduced platforms */
-	if (fadt && !(fadt->flags & ACPI_FADT_HW_REDUCED_ACPI)) {
-		/*
-		 * Other than waiting for OSPM to request us to switch to ACPI
-		 * mode, do it by ourselves, since SMI will not be triggered.
-		 */
-		enter_acpi_mode(fadt->pm1a_cnt_blk);
+		/* Don't touch ACPI hardware on HW reduced platforms */
+		if (fadt && !(fadt->flags & ACPI_FADT_HW_REDUCED_ACPI)) {
+			/*
+			 * Other than waiting for OSPM to request us to switch
+			 * to ACPI * mode, do it by ourselves, since SMI will
+			 * not be triggered.
+			 */
+			enter_acpi_mode(fadt->pm1a_cnt_blk);
+		}
 	}
-#endif
 
 	return 0;
 }
@@ -219,19 +218,20 @@ int last_stage_init(void)
 
 static int x86_init_cpus(void)
 {
-#ifdef CONFIG_SMP
-	debug("Init additional CPUs\n");
-	x86_mp_init();
-#else
-	struct udevice *dev;
+	if (IS_ENABLED(CONFIG_SMP)) {
+		debug("Init additional CPUs\n");
+		x86_mp_init();
+	} else {
+		struct udevice *dev;
 
-	/*
-	 * This causes the cpu-x86 driver to be probed.
-	 * We don't check return value here as we want to allow boards
-	 * which have not been converted to use cpu uclass driver to boot.
-	 */
-	uclass_first_device(UCLASS_CPU, &dev);
-#endif
+		/*
+		 * This causes the cpu-x86 driver to be probed.
+		 * We don't check return value here as we want to allow boards
+		 * which have not been converted to use cpu uclass driver to
+		 * boot.
+		 */
+		uclass_first_device(UCLASS_CPU, &dev);
+	}
 
 	return 0;
 }
@@ -269,13 +269,11 @@ int cpu_init_r(void)
 #ifndef CONFIG_EFI_STUB
 int reserve_arch(void)
 {
-#ifdef CONFIG_ENABLE_MRC_CACHE
-	mrccache_reserve();
-#endif
+	if (IS_ENABLED(CONFIG_ENABLE_MRC_CACHE))
+		mrccache_reserve();
 
-#ifdef CONFIG_SEABIOS
-	high_table_reserve();
-#endif
+	if (IS_ENABLED(CONFIG_SEABIOS))
+		high_table_reserve();
 
 	if (IS_ENABLED(CONFIG_HAVE_ACPI_RESUME)) {
 		acpi_s3_reserve();
diff --git a/arch/x86/cpu/i386/cpu.c b/arch/x86/cpu/i386/cpu.c
index 9809ac51117..fca3f79b697 100644
--- a/arch/x86/cpu/i386/cpu.c
+++ b/arch/x86/cpu/i386/cpu.c
@@ -626,7 +626,6 @@ int cpu_jump_to_64bit_uboot(ulong target)
 	return -EFAULT;
 }
 
-#ifdef CONFIG_SMP
 int x86_mp_init(void)
 {
 	int ret;
@@ -639,4 +638,3 @@ int x86_mp_init(void)
 
 	return 0;
 }
-#endif
-- 
2.27.0.rc0.183.gde8f92d652-goog

[PATCH 09/22] x86: cpu: Remove unnecessary #ifdefs

[Wolfgang Wallner]

Hi Simon,

-----"Simon Glass" <sjg@chromium.org> schrieb: -----
> Betreff: [PATCH 09/22] x86: cpu: Remove unnecessary #ifdefs
> 
> Drop some #ifdefs that are not needed or can be converted to compile-time
> checks.
> 
> Signed-off-by: Simon Glass <sjg@chromium.org>
> ---
> 
>  arch/x86/cpu/cpu.c      | 58 ++++++++++++++++++++---------------------
>  arch/x86/cpu/i386/cpu.c |  2 --
>  2 files changed, 28 insertions(+), 32 deletions(-)
> 

Reviewed-by: Wolfgang Wallner <wolfgang.wallner@br-automation.com>

[PATCH 10/22] x86: mp: Support APs waiting for instructions

[Simon Glass]

At present the APs (non-boot CPUs) are inited once and then parked ready
for the OS to use them. However in some cases we want to send new requests
through, such as to change MTRRs and keep them consistent across CPUs.

Change the last state of the flight plan to go into a wait loop, accepting
instructions from the main CPU.

Signed-off-by: Simon Glass <sjg@chromium.org>
---

 arch/x86/cpu/mp_init.c    | 99 ++++++++++++++++++++++++++++++++++++---
 arch/x86/include/asm/mp.h | 11 +++++
 2 files changed, 104 insertions(+), 6 deletions(-)

diff --git a/arch/x86/cpu/mp_init.c b/arch/x86/cpu/mp_init.c
index ccb68b8b89b..c424f283807 100644
--- a/arch/x86/cpu/mp_init.c
+++ b/arch/x86/cpu/mp_init.c
@@ -43,14 +43,36 @@ struct mp_flight_plan {
 	struct mp_flight_record *records;
 };
 
-static struct mp_flight_plan mp_info;
-
 struct cpu_map {
 	struct udevice *dev;
 	int apic_id;
 	int err_code;
 };
 
+struct mp_callback {
+	/**
+	 * func() - Function to call on the AP
+	 *
+	 * @arg: Argument to pass
+	 */
+	void (*func)(void *arg);
+	void *arg;
+	int logical_cpu_number;
+};
+
+static struct mp_flight_plan mp_info;
+
+/*
+ * ap_callbacks - Callback mailbox array
+ *
+ * Array of callback, one entry for each available CPU, indexed by the CPU
+ * number, which is dev->req_seq. The entry for the main CPU is never used.
+ * When this is NULL, there is no pending work for the CPU to run. When
+ * non-NULL it points to the mp_callback structure. This is shared between all
+ * CPUs, so should only be written by the main CPU.
+ */
+static struct mp_callback **ap_callbacks;
+
 static inline void barrier_wait(atomic_t *b)
 {
 	while (atomic_read(b) == 0)
@@ -147,11 +169,9 @@ static void ap_init(unsigned int cpu_index)
 	debug("AP: slot %d apic_id %x, dev %s\n", cpu_index, apic_id,
 	      dev ? dev->name : "(apic_id not found)");
 
-	/* Walk the flight plan */
+	/* Walk the flight plan, which never returns */
 	ap_do_flight_plan(dev);
-
-	/* Park the AP */
-	debug("parking\n");
+	debug("Unexpected return\n");
 done:
 	stop_this_cpu();
 }
@@ -442,6 +462,68 @@ static int get_bsp(struct udevice **devp, int *cpu_countp)
 	return dev->req_seq;
 }
 
+static struct mp_callback *read_callback(struct mp_callback **slot)
+{
+	struct mp_callback *ret;
+
+	asm volatile ("mov	%1, %0\n"
+		: "=r" (ret)
+		: "m" (*slot)
+		: "memory"
+	);
+	return ret;
+}
+
+static void store_callback(struct mp_callback **slot, struct mp_callback *val)
+{
+	asm volatile ("mov	%1, %0\n"
+		: "=m" (*slot)
+		: "r" (val)
+		: "memory"
+	);
+}
+
+/**
+ * ap_wait_for_instruction() - Wait for and process requests from the main CPU
+ *
+ * This is called by APs (here, everything other than the main boot CPU) to
+ * await instructions. They arrive in the form of a function call and argument,
+ * which is then called. This uses a simple mailbox with atomic read/set
+ *
+ * @cpu: CPU that is waiting
+ * @unused: Optional argument provided by struct mp_flight_record, not used here
+ * @return Does not return
+ */
+static int ap_wait_for_instruction(struct udevice *cpu, void *unused)
+{
+	struct mp_callback lcb;
+	struct mp_callback **per_cpu_slot;
+
+	per_cpu_slot = &ap_callbacks[cpu->req_seq];
+
+	while (1) {
+		struct mp_callback *cb = read_callback(per_cpu_slot);
+
+		if (!cb) {
+			asm ("pause");
+			continue;
+		}
+
+		/* Copy to local variable before using the value */
+		memcpy(&lcb, cb, sizeof(lcb));
+		mfence();
+		if (lcb.logical_cpu_number == MP_SELECT_ALL ||
+		    lcb.logical_cpu_number == MP_SELECT_APS ||
+		    cpu->req_seq == lcb.logical_cpu_number)
+			lcb.func(lcb.arg);
+
+		/* Indicate we are finished */
+		store_callback(per_cpu_slot, NULL);
+	}
+
+	return 0;
+}
+
 static int mp_init_cpu(struct udevice *cpu, void *unused)
 {
 	struct cpu_platdata *plat = dev_get_parent_platdata(cpu);
@@ -454,6 +536,7 @@ static int mp_init_cpu(struct udevice *cpu, void *unused)
 
 static struct mp_flight_record mp_steps[] = {
 	MP_FR_BLOCK_APS(mp_init_cpu, NULL, mp_init_cpu, NULL),
+	MP_FR_BLOCK_APS(ap_wait_for_instruction, NULL, NULL, NULL),
 };
 
 int mp_init(void)
@@ -491,6 +574,10 @@ int mp_init(void)
 	if (ret)
 		log_warning("Warning: Device tree does not describe all CPUs. Extra ones will not be started correctly\n");
 
+	ap_callbacks = calloc(num_cpus, sizeof(struct mp_callback *));
+	if (!ap_callbacks)
+		return -ENOMEM;
+
 	/* Copy needed parameters so that APs have a reference to the plan */
 	mp_info.num_records = ARRAY_SIZE(mp_steps);
 	mp_info.records = mp_steps;
diff --git a/arch/x86/include/asm/mp.h b/arch/x86/include/asm/mp.h
index 94af819ad9a..41b1575f4be 100644
--- a/arch/x86/include/asm/mp.h
+++ b/arch/x86/include/asm/mp.h
@@ -11,6 +11,17 @@
 #include <asm/atomic.h>
 #include <asm/cache.h>
 
+enum {
+	/* Indicates that the function should run on all CPUs */
+	MP_SELECT_ALL	= -1,
+
+	/* Run on boot CPUs */
+	MP_SELECT_BSP	= -2,
+
+	/* Run on non-boot CPUs */
+	MP_SELECT_APS	= -3,
+};
+
 typedef int (*mp_callback_t)(struct udevice *cpu, void *arg);
 
 /*
-- 
2.27.0.rc0.183.gde8f92d652-goog

[PATCH 10/22] x86: mp: Support APs waiting for instructions

[Wolfgang Wallner]

Hi Simon,


-----"Simon Glass" <sjg@chromium.org> schrieb: -----
> Betreff: [PATCH 10/22] x86: mp: Support APs waiting for instructions
> 
> At present the APs (non-boot CPUs) are inited once and then parked ready
> for the OS to use them. However in some cases we want to send new requests
> through, such as to change MTRRs and keep them consistent across CPUs.
> 
> Change the last state of the flight plan to go into a wait loop, accepting
> instructions from the main CPU.
> 
> Signed-off-by: Simon Glass <sjg@chromium.org>
> ---
> 
>  arch/x86/cpu/mp_init.c    | 99 ++++++++++++++++++++++++++++++++++++---
>  arch/x86/include/asm/mp.h | 11 +++++
>  2 files changed, 104 insertions(+), 6 deletions(-)
> 
> diff --git a/arch/x86/cpu/mp_init.c b/arch/x86/cpu/mp_init.c
> index ccb68b8b89b..c424f283807 100644
> --- a/arch/x86/cpu/mp_init.c
> +++ b/arch/x86/cpu/mp_init.c
> @@ -43,14 +43,36 @@ struct mp_flight_plan {
>  	struct mp_flight_record *records;
>  };
>  
> -static struct mp_flight_plan mp_info;
> -
>  struct cpu_map {
>  	struct udevice *dev;
>  	int apic_id;
>  	int err_code;
>  };
>  
> +struct mp_callback {
> +	/**
> +	 * func() - Function to call on the AP
> +	 *
> +	 * @arg: Argument to pass
> +	 */
> +	void (*func)(void *arg);
> +	void *arg;
> +	int logical_cpu_number;
> +};
> +
> +static struct mp_flight_plan mp_info;
> +
> +/*
> + * ap_callbacks - Callback mailbox array
> + *
> + * Array of callback, one entry for each available CPU, indexed by the CPU
> + * number, which is dev->req_seq. The entry for the main CPU is never used.
> + * When this is NULL, there is no pending work for the CPU to run. When
> + * non-NULL it points to the mp_callback structure. This is shared between all
> + * CPUs, so should only be written by the main CPU.
> + */
> +static struct mp_callback **ap_callbacks;
> +
>  static inline void barrier_wait(atomic_t *b)
>  {
>  	while (atomic_read(b) == 0)
> @@ -147,11 +169,9 @@ static void ap_init(unsigned int cpu_index)
>  	debug("AP: slot %d apic_id %x, dev %s\n", cpu_index, apic_id,
>  	      dev ? dev->name : "(apic_id not found)");
>  
> -	/* Walk the flight plan */
> +	/* Walk the flight plan, which never returns */
>  	ap_do_flight_plan(dev);
> -
> -	/* Park the AP */
> -	debug("parking\n");
> +	debug("Unexpected return\n");
>  done:
>  	stop_this_cpu();
>  }
> @@ -442,6 +462,68 @@ static int get_bsp(struct udevice **devp, int *cpu_countp)
>  	return dev->req_seq;
>  }
>  
> +static struct mp_callback *read_callback(struct mp_callback **slot)
> +{
> +	struct mp_callback *ret;
> +
> +	asm volatile ("mov	%1, %0\n"
> +		: "=r" (ret)
> +		: "m" (*slot)
> +		: "memory"
> +	);
> +	return ret;
> +}
> +
> +static void store_callback(struct mp_callback **slot, struct mp_callback *val)
> +{
> +	asm volatile ("mov	%1, %0\n"
> +		: "=m" (*slot)
> +		: "r" (val)
> +		: "memory"
> +	);
> +}

Descriptions for the two functions above would make them easier to understand.

> +
> +/**
> + * ap_wait_for_instruction() - Wait for and process requests from the main CPU
> + *
> + * This is called by APs (here, everything other than the main boot CPU) to
> + * await instructions. They arrive in the form of a function call and argument,
> + * which is then called. This uses a simple mailbox with atomic read/set
> + *
> + * @cpu: CPU that is waiting
> + * @unused: Optional argument provided by struct mp_flight_record, not used here
> + * @return Does not return
> + */
> +static int ap_wait_for_instruction(struct udevice *cpu, void *unused)
> +{
> +	struct mp_callback lcb;
> +	struct mp_callback **per_cpu_slot;
> +
> +	per_cpu_slot = &ap_callbacks[cpu->req_seq];
> +
> +	while (1) {
> +		struct mp_callback *cb = read_callback(per_cpu_slot);
> +
> +		if (!cb) {
> +			asm ("pause");
> +			continue;
> +		}
> +
> +		/* Copy to local variable before using the value */
> +		memcpy(&lcb, cb, sizeof(lcb));
> +		mfence();
> +		if (lcb.logical_cpu_number == MP_SELECT_ALL ||
> +		    lcb.logical_cpu_number == MP_SELECT_APS ||
> +		    cpu->req_seq == lcb.logical_cpu_number)
> +			lcb.func(lcb.arg);
> +
> +		/* Indicate we are finished */
> +		store_callback(per_cpu_slot, NULL);
> +	}
> +
> +	return 0;
> +}
> +
>  static int mp_init_cpu(struct udevice *cpu, void *unused)
>  {
>  	struct cpu_platdata *plat = dev_get_parent_platdata(cpu);
> @@ -454,6 +536,7 @@ static int mp_init_cpu(struct udevice *cpu, void *unused)
>  
>  static struct mp_flight_record mp_steps[] = {
>  	MP_FR_BLOCK_APS(mp_init_cpu, NULL, mp_init_cpu, NULL),
> +	MP_FR_BLOCK_APS(ap_wait_for_instruction, NULL, NULL, NULL),
>  };
>  
>  int mp_init(void)
> @@ -491,6 +574,10 @@ int mp_init(void)
>  	if (ret)
>  		log_warning("Warning: Device tree does not describe all CPUs. Extra ones will not be started correctly\n");
>  
> +	ap_callbacks = calloc(num_cpus, sizeof(struct mp_callback *));
> +	if (!ap_callbacks)
> +		return -ENOMEM;
> +
>  	/* Copy needed parameters so that APs have a reference to the plan */
>  	mp_info.num_records = ARRAY_SIZE(mp_steps);
>  	mp_info.records = mp_steps;
> diff --git a/arch/x86/include/asm/mp.h b/arch/x86/include/asm/mp.h
> index 94af819ad9a..41b1575f4be 100644
> --- a/arch/x86/include/asm/mp.h
> +++ b/arch/x86/include/asm/mp.h
> @@ -11,6 +11,17 @@
>  #include <asm/atomic.h>
>  #include <asm/cache.h>
>  
> +enum {
> +	/* Indicates that the function should run on all CPUs */
> +	MP_SELECT_ALL	= -1,
> +
> +	/* Run on boot CPUs */
> +	MP_SELECT_BSP	= -2,
> +
> +	/* Run on non-boot CPUs */
> +	MP_SELECT_APS	= -3,
> +};
> +
>  typedef int (*mp_callback_t)(struct udevice *cpu, void *arg);
>  
>  /*
> -- 
> 2.27.0.rc0.183.gde8f92d652-goog

regards, Wolfgang

[PATCH 11/22] global_data: Add a generic global_data flag for SMP state

[Simon Glass]

Allow keeping track of whether all CPUs have been enabled yet. This allows
us to know whether other CPUs need to be considered when updating
CPU-specific settings such as MTRRs on x86.

Signed-off-by: Simon Glass <sjg@chromium.org>
---

 include/asm-generic/global_data.h | 1 +
 1 file changed, 1 insertion(+)

diff --git a/include/asm-generic/global_data.h b/include/asm-generic/global_data.h
index 8c78792cc98..345f365d794 100644
--- a/include/asm-generic/global_data.h
+++ b/include/asm-generic/global_data.h
@@ -167,5 +167,6 @@ typedef struct global_data {
 #define GD_FLG_LOG_READY	0x08000 /* Log system is ready for use	   */
 #define GD_FLG_WDT_READY	0x10000 /* Watchdog is ready for use	   */
 #define GD_FLG_SKIP_LL_INIT	0x20000	/* Don't perform low-level init	   */
+#define GD_FLG_SMP_INIT		0x40000	/* SMP init is complete		   */
 
 #endif /* __ASM_GENERIC_GBL_DATA_H */
-- 
2.27.0.rc0.183.gde8f92d652-goog

[PATCH 11/22] global_data: Add a generic global_data flag for SMP state

[Wolfgang Wallner]

Hi Simon,

-----"Simon Glass" <sjg@chromium.org> schrieb: -----
> Betreff: [PATCH 11/22] global_data: Add a generic global_data flag for SMP state
> 
> Allow keeping track of whether all CPUs have been enabled yet. This allows
> us to know whether other CPUs need to be considered when updating
> CPU-specific settings such as MTRRs on x86.
> 
> Signed-off-by: Simon Glass <sjg@chromium.org>
> ---
> 
>  include/asm-generic/global_data.h | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/include/asm-generic/global_data.h b/include/asm-generic/global_data.h
> index 8c78792cc98..345f365d794 100644
> --- a/include/asm-generic/global_data.h
> +++ b/include/asm-generic/global_data.h
> @@ -167,5 +167,6 @@ typedef struct global_data {
>  #define GD_FLG_LOG_READY	0x08000 /* Log system is ready for use	   */
>  #define GD_FLG_WDT_READY	0x10000 /* Watchdog is ready for use	   */
>  #define GD_FLG_SKIP_LL_INIT	0x20000	/* Don't perform low-level init	   */
> +#define GD_FLG_SMP_INIT		0x40000	/* SMP init is complete		   */
>  
>  #endif /* __ASM_GENERIC_GBL_DATA_H */
> -- 
> 2.27.0.rc0.183.gde8f92d652-goog

Reviewed-by: Wolfgang Wallner <wolfgang.wallner@br-automation.com>

[PATCH 12/22] x86: Set the SMP flag when MP init is complete

[Simon Glass]

Set this flag so we can track when it is safe to use CPUs other than the
main one.

Signed-off-by: Simon Glass <sjg@chromium.org>
---

 arch/x86/cpu/mp_init.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/arch/x86/cpu/mp_init.c b/arch/x86/cpu/mp_init.c
index c424f283807..139e9749e74 100644
--- a/arch/x86/cpu/mp_init.c
+++ b/arch/x86/cpu/mp_init.c
@@ -609,6 +609,7 @@ int mp_init(void)
 		debug("CPU init failed: err=%d\n", ret);
 		return ret;
 	}
+	gd->flags |= GD_FLG_SMP_INIT;
 
 	return 0;
 }
-- 
2.27.0.rc0.183.gde8f92d652-goog

[PATCH 12/22] x86: Set the SMP flag when MP init is complete

[Wolfgang Wallner]

Hi Simon,

-----"Simon Glass" <sjg@chromium.org> schrieb: -----
> Betreff: [PATCH 12/22] x86: Set the SMP flag when MP init is complete
> 
> Set this flag so we can track when it is safe to use CPUs other than the
> main one.
> 
> Signed-off-by: Simon Glass <sjg@chromium.org>
> ---
> 
>  arch/x86/cpu/mp_init.c | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/arch/x86/cpu/mp_init.c b/arch/x86/cpu/mp_init.c
> index c424f283807..139e9749e74 100644
> --- a/arch/x86/cpu/mp_init.c
> +++ b/arch/x86/cpu/mp_init.c
> @@ -609,6 +609,7 @@ int mp_init(void)
>  		debug("CPU init failed: err=%d\n", ret);
>  		return ret;
>  	}
> +	gd->flags |= GD_FLG_SMP_INIT;
>  
>  	return 0;
>  }
> -- 
> 2.27.0.rc0.183.gde8f92d652-goog

Reviewed-by: Wolfgang Wallner <wolfgang.wallner@br-automation.com>

[PATCH 13/22] x86: mp: Allow running functions on multiple CPUs

[Simon Glass]

Add a way to run a function on a selection of CPUs. This supports either
a single CPU, all CPUs, just the main CPU or just the 'APs', in Intel
terminology.

It works by writing into a mailbox and then waiting for the CPUs to notice
it, take action and indicate they are done.

When SMP is not yet enabled, this just calls the function on the main CPU.

Signed-off-by: Simon Glass <sjg@chromium.org>
---

 arch/x86/cpu/mp_init.c    | 82 ++++++++++++++++++++++++++++++++++++---
 arch/x86/include/asm/mp.h | 30 ++++++++++++++
 2 files changed, 106 insertions(+), 6 deletions(-)

diff --git a/arch/x86/cpu/mp_init.c b/arch/x86/cpu/mp_init.c
index 139e9749e74..9f97dc7a9ba 100644
--- a/arch/x86/cpu/mp_init.c
+++ b/arch/x86/cpu/mp_init.c
@@ -50,12 +50,7 @@ struct cpu_map {
 };
 
 struct mp_callback {
-	/**
-	 * func() - Function to call on the AP
-	 *
-	 * @arg: Argument to pass
-	 */
-	void (*func)(void *arg);
+	mp_run_func func;
 	void *arg;
 	int logical_cpu_number;
 };
@@ -483,6 +478,50 @@ static void store_callback(struct mp_callback **slot, struct mp_callback *val)
 	);
 }
 
+static int run_ap_work(struct mp_callback *callback, struct udevice *bsp,
+		       int num_cpus, uint expire_ms)
+{
+	int cur_cpu = bsp->req_seq;
+	int num_aps = num_cpus - 1; /* number of non-BSPs to get this message */
+	int cpus_accepted;
+	ulong start;
+	int i;
+
+	/* Signal to all the APs to run the func. */
+	for (i = 0; i < num_cpus; i++) {
+		if (cur_cpu != i)
+			store_callback(&ap_callbacks[i], callback);
+	}
+	mfence();
+
+	/* Wait for all the APs to signal back that call has been accepted. */
+	start = get_timer(0);
+
+	do {
+		mdelay(1);
+		cpus_accepted = 0;
+
+		for (i = 0; i < num_cpus; i++) {
+			if (cur_cpu == i)
+				continue;
+			if (!read_callback(&ap_callbacks[i]))
+				cpus_accepted++;
+		}
+
+		if (expire_ms && get_timer(start) >= expire_ms) {
+			log(UCLASS_CPU, LOGL_CRIT,
+			    "AP call expired; %d/%d CPUs accepted\n",
+			    cpus_accepted, num_aps);
+			return -ETIMEDOUT;
+		}
+	} while (cpus_accepted != num_aps);
+
+	/* Make sure we can see any data written by the APs */
+	mfence();
+
+	return 0;
+}
+
 /**
  * ap_wait_for_instruction() - Wait for and process requests from the main CPU
  *
@@ -539,6 +578,37 @@ static struct mp_flight_record mp_steps[] = {
 	MP_FR_BLOCK_APS(ap_wait_for_instruction, NULL, NULL, NULL),
 };
 
+int mp_run_on_cpus(int cpu_select, mp_run_func func, void *arg)
+{
+	struct mp_callback lcb = {
+		.func = func,
+		.arg = arg,
+		.logical_cpu_number = cpu_select,
+	};
+	struct udevice *dev;
+	int num_cpus;
+	int ret;
+
+	if (!(gd->flags & GD_FLG_SMP_INIT))
+		return -ENXIO;
+
+	ret = get_bsp(&dev, &num_cpus);
+	if (ret < 0)
+		return log_msg_ret("bsp", ret);
+	if (cpu_select == MP_SELECT_ALL || cpu_select == MP_SELECT_BSP ||
+	    cpu_select == ret) {
+		/* Run on BSP first */
+		func(arg);
+	}
+
+	/* Allow up to 1 second for all APs to finish */
+	ret = run_ap_work(&lcb, dev, num_cpus, 1000 /* ms */);
+	if (ret)
+		return log_msg_ret("aps", ret);
+
+	return 0;
+}
+
 int mp_init(void)
 {
 	int num_aps, num_cpus;
diff --git a/arch/x86/include/asm/mp.h b/arch/x86/include/asm/mp.h
index 41b1575f4be..0272b3c0b6a 100644
--- a/arch/x86/include/asm/mp.h
+++ b/arch/x86/include/asm/mp.h
@@ -86,4 +86,34 @@ int mp_init(void);
 /* Set up additional CPUs */
 int x86_mp_init(void);
 
+/**
+ * mp_run_func() - Function to call on the AP
+ *
+ * @arg: Argument to pass
+ */
+typedef void (*mp_run_func)(void *arg);
+
+#if defined(CONFIG_SMP) && !CONFIG_IS_ENABLED(X86_64)
+/**
+ * mp_run_on_cpus() - Run a function on one or all CPUs
+ *
+ * This does not return until all CPUs have completed the work
+ *
+ * @cpu_select: CPU to run on, or MP_SELECT_ALL for all, or MP_SELECT_BSP for
+ *	BSP
+ * @func: Function to run
+ * @arg: Argument to pass to the function
+ * @return 0 on success, -ve on error
+ */
+int mp_run_on_cpus(int cpu_select, mp_run_func func, void *arg);
+#else
+static inline int mp_run_on_cpus(int cpu_select, mp_run_func func, void *arg)
+{
+	/* There is only one CPU, so just call the function here */
+	func(arg);
+
+	return 0;
+}
+#endif
+
 #endif /* _X86_MP_H_ */
-- 
2.27.0.rc0.183.gde8f92d652-goog

[PATCH 13/22] x86: mp: Allow running functions on multiple CPUs

[Wolfgang Wallner]

Hi Simon,

-----"Simon Glass" <sjg@chromium.org> schrieb: -----
> Betreff: [PATCH 13/22] x86: mp: Allow running functions on multiple CPUs
> 
> Add a way to run a function on a selection of CPUs. This supports either
> a single CPU, all CPUs, just the main CPU or just the 'APs', in Intel
> terminology.
> 
> It works by writing into a mailbox and then waiting for the CPUs to notice
> it, take action and indicate they are done.
> 
> When SMP is not yet enabled, this just calls the function on the main CPU.
> 
> Signed-off-by: Simon Glass <sjg@chromium.org>
> ---
> 
>  arch/x86/cpu/mp_init.c    | 82 ++++++++++++++++++++++++++++++++++++---
>  arch/x86/include/asm/mp.h | 30 ++++++++++++++
>  2 files changed, 106 insertions(+), 6 deletions(-)
> 
> diff --git a/arch/x86/cpu/mp_init.c b/arch/x86/cpu/mp_init.c
> index 139e9749e74..9f97dc7a9ba 100644
> --- a/arch/x86/cpu/mp_init.c
> +++ b/arch/x86/cpu/mp_init.c
> @@ -50,12 +50,7 @@ struct cpu_map {
>  };
>  
>  struct mp_callback {
> -	/**
> -	 * func() - Function to call on the AP
> -	 *
> -	 * @arg: Argument to pass
> -	 */
> -	void (*func)(void *arg);
> +	mp_run_func func;
>  	void *arg;
>  	int logical_cpu_number;
>  };
> @@ -483,6 +478,50 @@ static void store_callback(struct mp_callback **slot, struct mp_callback *val)
>  	);
>  }

Could you please add a function description for run_ap_work()?

> +static int run_ap_work(struct mp_callback *callback, struct udevice *bsp,
> +		       int num_cpus, uint expire_ms)
> +{
> +	int cur_cpu = bsp->req_seq;
> +	int num_aps = num_cpus - 1; /* number of non-BSPs to get this message */
> +	int cpus_accepted;
> +	ulong start;
> +	int i;
> +
> +	/* Signal to all the APs to run the func. */
> +	for (i = 0; i < num_cpus; i++) {
> +		if (cur_cpu != i)
> +			store_callback(&ap_callbacks[i], callback);
> +	}
> +	mfence();
> +
> +	/* Wait for all the APs to signal back that call has been accepted. */
> +	start = get_timer(0);
> +
> +	do {
> +		mdelay(1);
> +		cpus_accepted = 0;
> +
> +		for (i = 0; i < num_cpus; i++) {
> +			if (cur_cpu == i)
> +				continue;
> +			if (!read_callback(&ap_callbacks[i]))
> +				cpus_accepted++;
> +		}
> +
> +		if (expire_ms && get_timer(start) >= expire_ms) {
> +			log(UCLASS_CPU, LOGL_CRIT,
> +			    "AP call expired; %d/%d CPUs accepted\n",
> +			    cpus_accepted, num_aps);
> +			return -ETIMEDOUT;
> +		}
> +	} while (cpus_accepted != num_aps);
> +
> +	/* Make sure we can see any data written by the APs */
> +	mfence();
> +
> +	return 0;
> +}
> +
>  /**
>   * ap_wait_for_instruction() - Wait for and process requests from the main CPU
>   *
> @@ -539,6 +578,37 @@ static struct mp_flight_record mp_steps[] = {
>  	MP_FR_BLOCK_APS(ap_wait_for_instruction, NULL, NULL, NULL),
>  };
>  
> +int mp_run_on_cpus(int cpu_select, mp_run_func func, void *arg)
> +{
> +	struct mp_callback lcb = {
> +		.func = func,
> +		.arg = arg,
> +		.logical_cpu_number = cpu_select,
> +	};
> +	struct udevice *dev;
> +	int num_cpus;
> +	int ret;
> +
> +	if (!(gd->flags & GD_FLG_SMP_INIT))
> +		return -ENXIO;
> +
> +	ret = get_bsp(&dev, &num_cpus);
> +	if (ret < 0)
> +		return log_msg_ret("bsp", ret);
> +	if (cpu_select == MP_SELECT_ALL || cpu_select == MP_SELECT_BSP ||
> +	    cpu_select == ret) {
> +		/* Run on BSP first */
> +		func(arg);
> +	}
> +
> +	/* Allow up to 1 second for all APs to finish */
> +	ret = run_ap_work(&lcb, dev, num_cpus, 1000 /* ms */);
> +	if (ret)
> +		return log_msg_ret("aps", ret);
> +
> +	return 0;
> +}
> +
>  int mp_init(void)
>  {
>  	int num_aps, num_cpus;
> diff --git a/arch/x86/include/asm/mp.h b/arch/x86/include/asm/mp.h
> index 41b1575f4be..0272b3c0b6a 100644
> --- a/arch/x86/include/asm/mp.h
> +++ b/arch/x86/include/asm/mp.h
> @@ -86,4 +86,34 @@ int mp_init(void);
>  /* Set up additional CPUs */
>  int x86_mp_init(void);
>  
> +/**
> + * mp_run_func() - Function to call on the AP
> + *
> + * @arg: Argument to pass
> + */
> +typedef void (*mp_run_func)(void *arg);
> +
> +#if defined(CONFIG_SMP) && !CONFIG_IS_ENABLED(X86_64)
> +/**
> + * mp_run_on_cpus() - Run a function on one or all CPUs
> + *
> + * This does not return until all CPUs have completed the work
> + *
> + * @cpu_select: CPU to run on, or MP_SELECT_ALL for all, or MP_SELECT_BSP for
> + *	BSP
> + * @func: Function to run
> + * @arg: Argument to pass to the function
> + * @return 0 on success, -ve on error
> + */
> +int mp_run_on_cpus(int cpu_select, mp_run_func func, void *arg);
> +#else
> +static inline int mp_run_on_cpus(int cpu_select, mp_run_func func, void *arg)
> +{
> +	/* There is only one CPU, so just call the function here */
> +	func(arg);
> +
> +	return 0;
> +}
> +#endif
> +
>  #endif /* _X86_MP_H_ */
> -- 
> 2.27.0.rc0.183.gde8f92d652-goog

regards, Wolfgang

[PATCH 14/22] x86: mp: Park CPUs before running the OS

[Simon Glass]

With the new MP features the CPUs are no-longer parked when the OS is run.
Fix this by calling a special function to park them, just before the OS is
started.

Signed-off-by: Simon Glass <sjg@chromium.org>
---

 arch/x86/cpu/cpu.c        |  5 +++++
 arch/x86/cpu/mp_init.c    | 18 ++++++++++++++++++
 arch/x86/include/asm/mp.h | 17 +++++++++++++++++
 3 files changed, 40 insertions(+)

diff --git a/arch/x86/cpu/cpu.c b/arch/x86/cpu/cpu.c
index d0720fb7fb5..baa7dae172e 100644
--- a/arch/x86/cpu/cpu.c
+++ b/arch/x86/cpu/cpu.c
@@ -66,6 +66,11 @@ static const char *const x86_vendor_name[] = {
 
 int __weak x86_cleanup_before_linux(void)
 {
+	int ret;
+
+	ret = mp_park_aps();
+	if (ret)
+		return log_msg_ret("park", ret);
 	bootstage_stash((void *)CONFIG_BOOTSTAGE_STASH_ADDR,
 			CONFIG_BOOTSTAGE_STASH_SIZE);
 
diff --git a/arch/x86/cpu/mp_init.c b/arch/x86/cpu/mp_init.c
index 9f97dc7a9ba..a16be28647a 100644
--- a/arch/x86/cpu/mp_init.c
+++ b/arch/x86/cpu/mp_init.c
@@ -609,6 +609,24 @@ int mp_run_on_cpus(int cpu_select, mp_run_func func, void *arg)
 	return 0;
 }
 
+static void park_this_cpu(void *unused)
+{
+	stop_this_cpu();
+}
+
+int mp_park_aps(void)
+{
+	unsigned long start;
+	int ret;
+
+	start = get_timer(0);
+	ret = mp_run_on_cpus(MP_SELECT_APS, park_this_cpu, NULL);
+	if (ret)
+		return ret;
+
+	return get_timer(start);
+}
+
 int mp_init(void)
 {
 	int num_aps, num_cpus;
diff --git a/arch/x86/include/asm/mp.h b/arch/x86/include/asm/mp.h
index 0272b3c0b6a..38961ca44b3 100644
--- a/arch/x86/include/asm/mp.h
+++ b/arch/x86/include/asm/mp.h
@@ -106,6 +106,15 @@ typedef void (*mp_run_func)(void *arg);
  * @return 0 on success, -ve on error
  */
 int mp_run_on_cpus(int cpu_select, mp_run_func func, void *arg);
+
+/**
+ * mp_park_aps() - Park the APs ready for the OS
+ *
+ * This halts all CPUs except the main one, ready for the OS to use them
+ *
+ * @return 0 on success, -ve on error
+ */
+int mp_park_aps(void);
 #else
 static inline int mp_run_on_cpus(int cpu_select, mp_run_func func, void *arg)
 {
@@ -114,6 +123,14 @@ static inline int mp_run_on_cpus(int cpu_select, mp_run_func func, void *arg)
 
 	return 0;
 }
+
+static inline int mp_park_aps(void)
+{
+	/* No APs to park */
+
+	return 0;
+}
+
 #endif
 
 #endif /* _X86_MP_H_ */
-- 
2.27.0.rc0.183.gde8f92d652-goog

[PATCH 14/22] x86: mp: Park CPUs before running the OS

[Wolfgang Wallner]

Hi Simon,

-----"Simon Glass" <sjg@chromium.org> schrieb: -----
> Betreff: [PATCH 14/22] x86: mp: Park CPUs before running the OS
> 
> With the new MP features the CPUs are no-longer parked when the OS is run.
> Fix this by calling a special function to park them, just before the OS is
> started.
> 
> Signed-off-by: Simon Glass <sjg@chromium.org>
> ---
> 
>  arch/x86/cpu/cpu.c        |  5 +++++
>  arch/x86/cpu/mp_init.c    | 18 ++++++++++++++++++
>  arch/x86/include/asm/mp.h | 17 +++++++++++++++++
>  3 files changed, 40 insertions(+)
> 
 
Reviewed-by: Wolfgang Wallner <wolfgang.wallner@br-automation.com>

[PATCH 15/22] x86: mp: Add iterators for CPUs

[Simon Glass]

It is convenient to iterate through the CPUs performing work on each one
and processing the result. Add a few iterator functions which handle this.
These can be used by any client code. It can call mp_run_on_cpus() on
each CPU that is returned, handling them one at a time.

Signed-off-by: Simon Glass <sjg@chromium.org>
---

 arch/x86/cpu/mp_init.c    | 62 +++++++++++++++++++++++++++++++++++++++
 arch/x86/include/asm/mp.h | 40 +++++++++++++++++++++++++
 2 files changed, 102 insertions(+)

diff --git a/arch/x86/cpu/mp_init.c b/arch/x86/cpu/mp_init.c
index a16be28647a..ef33a380171 100644
--- a/arch/x86/cpu/mp_init.c
+++ b/arch/x86/cpu/mp_init.c
@@ -627,6 +627,68 @@ int mp_park_aps(void)
 	return get_timer(start);
 }
 
+int mp_first_cpu(int cpu_select)
+{
+	struct udevice *dev;
+	int num_cpus;
+	int ret;
+
+	/*
+	 * This assumes that CPUs are numbered from 0. This function tries to
+	 * avoid assuming the CPU 0 is the boot CPU
+	 */
+	if (cpu_select == MP_SELECT_ALL)
+		return 0;   /* start with the first one */
+
+	ret = get_bsp(&dev, &num_cpus);
+	if (ret < 0)
+		return log_msg_ret("bsp", ret);
+
+	/* Return boot CPU if requested */
+	if (cpu_select == MP_SELECT_BSP)
+		return ret;
+
+	/* Return something other than the boot CPU, if APs requested */
+	if (cpu_select == MP_SELECT_APS && num_cpus > 1)
+		return ret == 0 ? 1 : 0;
+
+	/* Try to check for an invalid value */
+	if (cpu_select < 0 || cpu_select >= num_cpus)
+		return -EINVAL;
+
+	return cpu_select;  /* return the only selected one */
+}
+
+int mp_next_cpu(int cpu_select, int prev_cpu)
+{
+	struct udevice *dev;
+	int num_cpus;
+	int ret;
+	int bsp;
+
+	/* If we selected the BSP or a particular single CPU, we are done */
+	if (cpu_select == MP_SELECT_BSP || cpu_select >= 0)
+		return -EFBIG;
+
+	/* Must be doing MP_SELECT_ALL or MP_SELECT_APS; return the next CPU */
+	ret = get_bsp(&dev, &num_cpus);
+	if (ret < 0)
+		return log_msg_ret("bsp", ret);
+	bsp = ret;
+
+	/* Move to the next CPU */
+	assert(prev_cpu >= 0);
+	ret = prev_cpu + 1;
+
+	/* Skip the BSP if needed */
+	if (cpu_select == MP_SELECT_APS && ret == bsp)
+		ret++;
+	if (ret >= num_cpus)
+		return -EFBIG;
+
+	return ret;
+}
+
 int mp_init(void)
 {
 	int num_aps, num_cpus;
diff --git a/arch/x86/include/asm/mp.h b/arch/x86/include/asm/mp.h
index 38961ca44b3..9f4223ae8c3 100644
--- a/arch/x86/include/asm/mp.h
+++ b/arch/x86/include/asm/mp.h
@@ -115,6 +115,31 @@ int mp_run_on_cpus(int cpu_select, mp_run_func func, void *arg);
  * @return 0 on success, -ve on error
  */
 int mp_park_aps(void);
+
+/**
+ * mp_first_cpu() - Get the first CPU to process, from a selection
+ *
+ * This is used to iterate through selected CPUs. Call this function first, then
+ * call mp_next_cpu() repeatedly until it returns -EFBIG.
+ *
+ * @cpu_select: Selected CPUs (either a CPU number or MP_SELECT_...)
+ * @return next CPU number to run on (e.g. 0)
+ */
+int mp_first_cpu(int cpu_select);
+
+/**
+ * mp_next_cpu() - Get the next CPU to process, from a selection
+ *
+ * This is used to iterate through selected CPUs. After first calling
+ * mp_first_cpu() once, call this function repeatedly until it returns -EFBIG.
+ *
+ * The value of @cpu_select must be the same for all calls.
+ *
+ * @cpu_select: Selected CPUs (either a CPU number or MP_SELECT_...)
+ * @prev_cpu: Previous value returned by mp_first_cpu()/mp_next_cpu()
+ * @return next CPU number to run on (e.g. 0)
+ */
+int mp_next_cpu(int cpu_select, int prev_cpu);
 #else
 static inline int mp_run_on_cpus(int cpu_select, mp_run_func func, void *arg)
 {
@@ -131,6 +156,21 @@ static inline int mp_park_aps(void)
 	return 0;
 }
 
+static inline int mp_first_cpu(int cpu_select)
+{
+	/* We cannot run on any APs, nor a selected CPU */
+	return cpu_select == MP_SELECT_APS ? -EFBIG : MP_SELECT_BSP;
+}
+
+static inline int mp_next_cpu(int cpu_select, int prev_cpu)
+{
+	/*
+	 * When MP is not enabled, there is only one CPU and we did it in
+	 * mp_first_cpu()
+	 */
+	return -EFBIG;
+}
+
 #endif
 
 #endif /* _X86_MP_H_ */
-- 
2.27.0.rc0.183.gde8f92d652-goog

[PATCH 15/22] x86: mp: Add iterators for CPUs

[Wolfgang Wallner]

Hi Simon,

-----"Simon Glass" <sjg@chromium.org> schrieb: -----
> Betreff: [PATCH 15/22] x86: mp: Add iterators for CPUs
> 
> It is convenient to iterate through the CPUs performing work on each one
> and processing the result. Add a few iterator functions which handle this.
> These can be used by any client code. It can call mp_run_on_cpus() on
> each CPU that is returned, handling them one at a time.
> 
> Signed-off-by: Simon Glass <sjg@chromium.org>
> ---
> 
>  arch/x86/cpu/mp_init.c    | 62 +++++++++++++++++++++++++++++++++++++++
>  arch/x86/include/asm/mp.h | 40 +++++++++++++++++++++++++
>  2 files changed, 102 insertions(+)

Reviewed-by: Wolfgang Wallner <wolfgang.wallner@br-automation.com>

[PATCH 16/22] x86: mtrr: Use MP calls to list the MTRRs

[Simon Glass]

Update the mtrr command to use mp_run_on_cpus() to obtain its information.
Since the selected CPU is the boot CPU this does not change the result,
but it sets the stage for supporting other CPUs.

Signed-off-by: Simon Glass <sjg@chromium.org>
---

 arch/x86/cpu/mtrr.c         | 11 +++++++++++
 arch/x86/include/asm/mtrr.h | 30 ++++++++++++++++++++++++++++++
 cmd/x86/mtrr.c              | 25 +++++++++++++++++++++----
 3 files changed, 62 insertions(+), 4 deletions(-)

diff --git a/arch/x86/cpu/mtrr.c b/arch/x86/cpu/mtrr.c
index 7ec0733337d..11f3ef08172 100644
--- a/arch/x86/cpu/mtrr.c
+++ b/arch/x86/cpu/mtrr.c
@@ -21,6 +21,7 @@
 #include <log.h>
 #include <asm/cache.h>
 #include <asm/io.h>
+#include <asm/mp.h>
 #include <asm/msr.h>
 #include <asm/mtrr.h>
 
@@ -63,6 +64,16 @@ static void set_var_mtrr(uint reg, uint type, uint64_t start, uint64_t size)
 	wrmsrl(MTRR_PHYS_MASK_MSR(reg), mask | MTRR_PHYS_MASK_VALID);
 }
 
+void mtrr_save_all(struct mtrr_info *info)
+{
+	int i;
+
+	for (i = 0; i < MTRR_COUNT; i++) {
+		info->mtrr[i].base = native_read_msr(MTRR_PHYS_BASE_MSR(i));
+		info->mtrr[i].mask = native_read_msr(MTRR_PHYS_MASK_MSR(i));
+	}
+}
+
 int mtrr_commit(bool do_caches)
 {
 	struct mtrr_request *req = gd->arch.mtrr_req;
diff --git a/arch/x86/include/asm/mtrr.h b/arch/x86/include/asm/mtrr.h
index 212a699c1b2..476d6f8a9cf 100644
--- a/arch/x86/include/asm/mtrr.h
+++ b/arch/x86/include/asm/mtrr.h
@@ -70,6 +70,26 @@ struct mtrr_state {
 	bool enable_cache;
 };
 
+/**
+ * struct mtrr - Information about a single MTRR
+ *
+ * @base: Base address and MTRR_BASE_TYPE_MASK
+ * @mask: Mask and MTRR_PHYS_MASK_VALID
+ */
+struct mtrr {
+	u64 base;
+	u64 mask;
+};
+
+/**
+ * struct mtrr_info - Information about all MTRRs
+ *
+ * @mtrr: Information about each mtrr
+ */
+struct mtrr_info {
+	struct mtrr mtrr[MTRR_COUNT];
+};
+
 /**
  * mtrr_open() - Prepare to adjust MTRRs
  *
@@ -129,6 +149,16 @@ int mtrr_commit(bool do_caches);
  */
 int mtrr_set_next_var(uint type, uint64_t base, uint64_t size);
 
+/**
+ * mtrr_save_all() - Save all the MTRRs
+ *
+ * This writes all MTRRs from the boot CPU into a struct so they can be loaded
+ * onto other CPUs
+ *
+ * @info: Place to put the MTRR info
+ */
+void mtrr_save_all(struct mtrr_info *info);
+
 #endif
 
 #if ((CONFIG_XIP_ROM_SIZE & (CONFIG_XIP_ROM_SIZE - 1)) != 0)
diff --git a/cmd/x86/mtrr.c b/cmd/x86/mtrr.c
index 5d25c5802af..01197044452 100644
--- a/cmd/x86/mtrr.c
+++ b/cmd/x86/mtrr.c
@@ -5,7 +5,9 @@
 
 #include <common.h>
 #include <command.h>
+#include <log.h>
 #include <asm/msr.h>
+#include <asm/mp.h>
 #include <asm/mtrr.h>
 
 static const char *const mtrr_type_name[MTRR_TYPE_COUNT] = {
@@ -18,19 +20,32 @@ static const char *const mtrr_type_name[MTRR_TYPE_COUNT] = {
 	"Back",
 };
 
-static int do_mtrr_list(void)
+static void save_mtrrs(void *arg)
 {
+	struct mtrr_info *info = arg;
+
+	mtrr_save_all(info);
+}
+
+static int do_mtrr_list(int cpu_select)
+{
+	struct mtrr_info info;
+	int ret;
 	int i;
 
 	printf("Reg Valid Write-type   %-16s %-16s %-16s\n", "Base   ||",
 	       "Mask   ||", "Size   ||");
+	memset(&info, '\0', sizeof(info));
+	ret = mp_run_on_cpus(cpu_select, save_mtrrs, &info);
+	if (ret)
+		return log_msg_ret("run", ret);
 	for (i = 0; i < MTRR_COUNT; i++) {
 		const char *type = "Invalid";
 		uint64_t base, mask, size;
 		bool valid;
 
-		base = native_read_msr(MTRR_PHYS_BASE_MSR(i));
-		mask = native_read_msr(MTRR_PHYS_MASK_MSR(i));
+		base = info.mtrr[i].base;
+		mask = info.mtrr[i].mask;
 		size = ~mask & ((1ULL << CONFIG_CPU_ADDR_BITS) - 1);
 		size |= (1 << 12) - 1;
 		size += 1;
@@ -102,11 +117,13 @@ static int do_mtrr(struct cmd_tbl *cmdtp, int flag, int argc,
 		   char *const argv[])
 {
 	const char *cmd;
+	int cpu_select;
 	uint reg;
 
+	cpu_select = MP_SELECT_BSP;
 	cmd = argv[1];
 	if (argc < 2 || *cmd == 'l')
-		return do_mtrr_list();
+		return do_mtrr_list(cpu_select);
 	argc -= 2;
 	argv += 2;
 	if (argc <= 0)
-- 
2.27.0.rc0.183.gde8f92d652-goog

[PATCH 16/22] x86: mtrr: Use MP calls to list the MTRRs

[Wolfgang Wallner]

Hi Simon,

-----"Simon Glass" <sjg@chromium.org> schrieb: -----
> Betreff: [PATCH 16/22] x86: mtrr: Use MP calls to list the MTRRs
> 
> Update the mtrr command to use mp_run_on_cpus() to obtain its information.
> Since the selected CPU is the boot CPU this does not change the result,
> but it sets the stage for supporting other CPUs.
> 
> Signed-off-by: Simon Glass <sjg@chromium.org>
> ---
> 
>  arch/x86/cpu/mtrr.c         | 11 +++++++++++
>  arch/x86/include/asm/mtrr.h | 30 ++++++++++++++++++++++++++++++
>  cmd/x86/mtrr.c              | 25 +++++++++++++++++++++----
>  3 files changed, 62 insertions(+), 4 deletions(-)
> 
> diff --git a/arch/x86/cpu/mtrr.c b/arch/x86/cpu/mtrr.c
> index 7ec0733337d..11f3ef08172 100644
> --- a/arch/x86/cpu/mtrr.c
> +++ b/arch/x86/cpu/mtrr.c
> @@ -21,6 +21,7 @@
>  #include <log.h>
>  #include <asm/cache.h>
>  #include <asm/io.h>
> +#include <asm/mp.h>
>  #include <asm/msr.h>
>  #include <asm/mtrr.h>
>  
> @@ -63,6 +64,16 @@ static void set_var_mtrr(uint reg, uint type, uint64_t start, uint64_t size)
>  	wrmsrl(MTRR_PHYS_MASK_MSR(reg), mask | MTRR_PHYS_MASK_VALID);
>  }
>  
> +void mtrr_save_all(struct mtrr_info *info)

Nit: This is just a personal view, but to me, "save" sounds like writing to
an MTRR. But this function reads them. How about calling it "mtrr_read_all"?

> +{
> +	int i;
> +
> +	for (i = 0; i < MTRR_COUNT; i++) {
> +		info->mtrr[i].base = native_read_msr(MTRR_PHYS_BASE_MSR(i));
> +		info->mtrr[i].mask = native_read_msr(MTRR_PHYS_MASK_MSR(i));
> +	}
> +}
> +
>  int mtrr_commit(bool do_caches)
>  {
>  	struct mtrr_request *req = gd->arch.mtrr_req;
> diff --git a/arch/x86/include/asm/mtrr.h b/arch/x86/include/asm/mtrr.h
> index 212a699c1b2..476d6f8a9cf 100644
> --- a/arch/x86/include/asm/mtrr.h
> +++ b/arch/x86/include/asm/mtrr.h
> @@ -70,6 +70,26 @@ struct mtrr_state {
>  	bool enable_cache;
>  };
>  
> +/**
> + * struct mtrr - Information about a single MTRR
> + *
> + * @base: Base address and MTRR_BASE_TYPE_MASK
> + * @mask: Mask and MTRR_PHYS_MASK_VALID
> + */
> +struct mtrr {
> +	u64 base;
> +	u64 mask;
> +};
> +
> +/**
> + * struct mtrr_info - Information about all MTRRs
> + *
> + * @mtrr: Information about each mtrr
> + */
> +struct mtrr_info {
> +	struct mtrr mtrr[MTRR_COUNT];
> +};
> +
>  /**
>   * mtrr_open() - Prepare to adjust MTRRs
>   *
> @@ -129,6 +149,16 @@ int mtrr_commit(bool do_caches);
>   */
>  int mtrr_set_next_var(uint type, uint64_t base, uint64_t size);
>  
> +/**
> + * mtrr_save_all() - Save all the MTRRs
> + *
> + * This writes all MTRRs from the boot CPU into a struct so they can be loaded
> + * onto other CPUs
> + *
> + * @info: Place to put the MTRR info
> + */
> +void mtrr_save_all(struct mtrr_info *info);
> +
>  #endif
>  
>  #if ((CONFIG_XIP_ROM_SIZE & (CONFIG_XIP_ROM_SIZE - 1)) != 0)
> diff --git a/cmd/x86/mtrr.c b/cmd/x86/mtrr.c
> index 5d25c5802af..01197044452 100644
> --- a/cmd/x86/mtrr.c
> +++ b/cmd/x86/mtrr.c
> @@ -5,7 +5,9 @@
>  
>  #include <common.h>
>  #include <command.h>
> +#include <log.h>
>  #include <asm/msr.h>
> +#include <asm/mp.h>
>  #include <asm/mtrr.h>
>  
>  static const char *const mtrr_type_name[MTRR_TYPE_COUNT] = {
> @@ -18,19 +20,32 @@ static const char *const mtrr_type_name[MTRR_TYPE_COUNT] = {
>  	"Back",
>  };
>  
> -static int do_mtrr_list(void)
> +static void save_mtrrs(void *arg)
>  {
> +	struct mtrr_info *info = arg;
> +
> +	mtrr_save_all(info);
> +}
> +
> +static int do_mtrr_list(int cpu_select)
> +{
> +	struct mtrr_info info;
> +	int ret;
>  	int i;
>  
>  	printf("Reg Valid Write-type   %-16s %-16s %-16s\n", "Base   ||",
>  	       "Mask   ||", "Size   ||");
> +	memset(&info, '\0', sizeof(info));
> +	ret = mp_run_on_cpus(cpu_select, save_mtrrs, &info);
> +	if (ret)
> +		return log_msg_ret("run", ret);
>  	for (i = 0; i < MTRR_COUNT; i++) {
>  		const char *type = "Invalid";
>  		uint64_t base, mask, size;
>  		bool valid;
>  
> -		base = native_read_msr(MTRR_PHYS_BASE_MSR(i));
> -		mask = native_read_msr(MTRR_PHYS_MASK_MSR(i));
> +		base = info.mtrr[i].base;
> +		mask = info.mtrr[i].mask;
>  		size = ~mask & ((1ULL << CONFIG_CPU_ADDR_BITS) - 1);
>  		size |= (1 << 12) - 1;
>  		size += 1;
> @@ -102,11 +117,13 @@ static int do_mtrr(struct cmd_tbl *cmdtp, int flag, int argc,
>  		   char *const argv[])
>  {
>  	const char *cmd;
> +	int cpu_select;
>  	uint reg;
>  
> +	cpu_select = MP_SELECT_BSP;
>  	cmd = argv[1];
>  	if (argc < 2 || *cmd == 'l')
> -		return do_mtrr_list();
> +		return do_mtrr_list(cpu_select);
>  	argc -= 2;
>  	argv += 2;
>  	if (argc <= 0)
> -- 
> 2.27.0.rc0.183.gde8f92d652-goog

Reviewed-by: Wolfgang Wallner <wolfgang.wallner@br-automation.com>

[PATCH 17/22] x86: mtrr: Update MTRRs on all CPUs

[Simon Glass]

When the boot CPU MTRRs are updated, perform the same update on all other
CPUs so they are kept in sync.

This avoids kernel warnings about mismatched MTRRs.

Signed-off-by: Simon Glass <sjg@chromium.org>
---

 arch/x86/cpu/mtrr.c | 57 +++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 57 insertions(+)

diff --git a/arch/x86/cpu/mtrr.c b/arch/x86/cpu/mtrr.c
index 11f3ef08172..a48c9d8232e 100644
--- a/arch/x86/cpu/mtrr.c
+++ b/arch/x86/cpu/mtrr.c
@@ -74,10 +74,61 @@ void mtrr_save_all(struct mtrr_info *info)
 	}
 }
 
+void mtrr_load_all(struct mtrr_info *info)
+{
+	struct mtrr_state state;
+	int i;
+
+	for (i = 0; i < MTRR_COUNT; i++) {
+		mtrr_open(&state, true);
+		wrmsrl(MTRR_PHYS_BASE_MSR(i), info->mtrr[i].base);
+		wrmsrl(MTRR_PHYS_MASK_MSR(i), info->mtrr[i].mask);
+		mtrr_close(&state, true);
+	}
+}
+
+static void save_mtrrs(void *arg)
+{
+	struct mtrr_info *info = arg;
+
+	mtrr_save_all(info);
+}
+
+static void load_mtrrs(void *arg)
+{
+	struct mtrr_info *info = arg;
+
+	mtrr_load_all(info);
+}
+
+/**
+ * mtrr_copy_to_aps() - Copy the MTRRs from the boot CPU to other CPUs
+ *
+ * @return 0 on success, -ve on failure
+ */
+static int mtrr_copy_to_aps(void)
+{
+	struct mtrr_info info;
+	int ret;
+
+	ret = mp_run_on_cpus(MP_SELECT_BSP, save_mtrrs, &info);
+	if (ret == -ENXIO)
+		return 0;
+	else if (ret)
+		return log_msg_ret("bsp", ret);
+
+	ret = mp_run_on_cpus(MP_SELECT_APS, load_mtrrs, &info);
+	if (ret)
+		return log_msg_ret("bsp", ret);
+
+	return 0;
+}
+
 int mtrr_commit(bool do_caches)
 {
 	struct mtrr_request *req = gd->arch.mtrr_req;
 	struct mtrr_state state;
+	int ret;
 	int i;
 
 	debug("%s: enabled=%d, count=%d\n", __func__, gd->arch.has_mtrr,
@@ -99,6 +150,12 @@ int mtrr_commit(bool do_caches)
 	mtrr_close(&state, do_caches);
 	debug("mtrr done\n");
 
+	if (gd->flags & GD_FLG_RELOC) {
+		ret = mtrr_copy_to_aps();
+		if (ret)
+			return log_msg_ret("copy", ret);
+	}
+
 	return 0;
 }
 
-- 
2.27.0.rc0.183.gde8f92d652-goog

[PATCH 17/22] x86: mtrr: Update MTRRs on all CPUs

[Wolfgang Wallner]

Hi Simon,

-----"Simon Glass" <sjg@chromium.org> schrieb: -----
> Betreff: [PATCH 17/22] x86: mtrr: Update MTRRs on all CPUs
> 
> When the boot CPU MTRRs are updated, perform the same update on all other
> CPUs so they are kept in sync.
> 
> This avoids kernel warnings about mismatched MTRRs.
> 
> Signed-off-by: Simon Glass <sjg@chromium.org>
> ---
> 
>  arch/x86/cpu/mtrr.c | 57 +++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 57 insertions(+)
> 
> diff --git a/arch/x86/cpu/mtrr.c b/arch/x86/cpu/mtrr.c
> index 11f3ef08172..a48c9d8232e 100644
> --- a/arch/x86/cpu/mtrr.c
> +++ b/arch/x86/cpu/mtrr.c
> @@ -74,10 +74,61 @@ void mtrr_save_all(struct mtrr_info *info)
>  	}
>  }
>  
> +void mtrr_load_all(struct mtrr_info *info)

Nit: Similar request as in the previous patch. How about naming this
function "mtrr_write_all" ?

> +{
> +	struct mtrr_state state;
> +	int i;
> +
> +	for (i = 0; i < MTRR_COUNT; i++) {
> +		mtrr_open(&state, true);
> +		wrmsrl(MTRR_PHYS_BASE_MSR(i), info->mtrr[i].base);
> +		wrmsrl(MTRR_PHYS_MASK_MSR(i), info->mtrr[i].mask);
> +		mtrr_close(&state, true);
> +	}
> +}
> +
> +static void save_mtrrs(void *arg)
> +{
> +	struct mtrr_info *info = arg;
> +
> +	mtrr_save_all(info);
> +}
> +
> +static void load_mtrrs(void *arg)
> +{
> +	struct mtrr_info *info = arg;
> +
> +	mtrr_load_all(info);
> +}
> +
> +/**
> + * mtrr_copy_to_aps() - Copy the MTRRs from the boot CPU to other CPUs
> + *
> + * @return 0 on success, -ve on failure
> + */
> +static int mtrr_copy_to_aps(void)
> +{
> +	struct mtrr_info info;
> +	int ret;
> +
> +	ret = mp_run_on_cpus(MP_SELECT_BSP, save_mtrrs, &info);
> +	if (ret == -ENXIO)
> +		return 0;
> +	else if (ret)
> +		return log_msg_ret("bsp", ret);
> +
> +	ret = mp_run_on_cpus(MP_SELECT_APS, load_mtrrs, &info);
> +	if (ret)
> +		return log_msg_ret("bsp", ret);
> +
> +	return 0;
> +}
> +
>  int mtrr_commit(bool do_caches)
>  {
>  	struct mtrr_request *req = gd->arch.mtrr_req;
>  	struct mtrr_state state;
> +	int ret;
>  	int i;
>  
>  	debug("%s: enabled=%d, count=%d\n", __func__, gd->arch.has_mtrr,
> @@ -99,6 +150,12 @@ int mtrr_commit(bool do_caches)
>  	mtrr_close(&state, do_caches);
>  	debug("mtrr done\n");
>  
> +	if (gd->flags & GD_FLG_RELOC) {
> +		ret = mtrr_copy_to_aps();
> +		if (ret)
> +			return log_msg_ret("copy", ret);
> +	}
> +
>  	return 0;
>  }
>  
> -- 
> 2.27.0.rc0.183.gde8f92d652-goog

Reviewed-by: Wolfgang Wallner <wolfgang.wallner@br-automation.com>

[PATCH 18/22] x86: mtrr: Add support for writing to MTRRs on any CPU

[Simon Glass]

To enable support for the 'mtrr' command, add a way to perform MTRR
operations on selected CPUs.

This works by setting up a little 'operation' structure and sending it
around the CPUs for action.

Signed-off-by: Simon Glass <sjg@chromium.org>
---

 arch/x86/cpu/mtrr.c         | 81 +++++++++++++++++++++++++++++++++++++
 arch/x86/include/asm/mtrr.h | 21 ++++++++++
 2 files changed, 102 insertions(+)

diff --git a/arch/x86/cpu/mtrr.c b/arch/x86/cpu/mtrr.c
index a48c9d8232e..25f317d4298 100644
--- a/arch/x86/cpu/mtrr.c
+++ b/arch/x86/cpu/mtrr.c
@@ -221,3 +221,84 @@ int mtrr_set_next_var(uint type, uint64_t start, uint64_t size)
 
 	return 0;
 }
+
+/** enum mtrr_opcode - supported operations for mtrr_do_oper() */
+enum mtrr_opcode {
+	MTRR_OP_SET,
+	MTRR_OP_SET_VALID,
+};
+
+/**
+ * struct mtrr_oper - An MTRR operation to perform on a CPU
+ *
+ * @opcode: Indicates operation to perform
+ * @reg: MTRR reg number to select (0-7, -1 = all)
+ * @valid: Valid value to write for MTRR_OP_SET_VALID
+ * @base: Base value to write for MTRR_OP_SET
+ * @mask: Mask value to write for MTRR_OP_SET
+ */
+struct mtrr_oper {
+	enum mtrr_opcode opcode;
+	int reg;
+	bool valid;
+	u64 base;
+	u64 mask;
+};
+
+static void mtrr_do_oper(void *arg)
+{
+	struct mtrr_oper *oper = arg;
+	u64 mask;
+
+	switch (oper->opcode) {
+	case MTRR_OP_SET_VALID:
+		mask = native_read_msr(MTRR_PHYS_MASK_MSR(oper->reg));
+		if (oper->valid)
+			mask |= MTRR_PHYS_MASK_VALID;
+		else
+			mask &= ~MTRR_PHYS_MASK_VALID;
+		wrmsrl(MTRR_PHYS_MASK_MSR(oper->reg), mask);
+		break;
+	case MTRR_OP_SET:
+		wrmsrl(MTRR_PHYS_BASE_MSR(oper->reg), oper->base);
+		wrmsrl(MTRR_PHYS_MASK_MSR(oper->reg), oper->mask);
+		break;
+	}
+}
+
+static int mtrr_start_op(int cpu_select, struct mtrr_oper *oper)
+{
+	struct mtrr_state state;
+	int ret;
+
+	mtrr_open(&state, true);
+	ret = mp_run_on_cpus(cpu_select, mtrr_do_oper, oper);
+	mtrr_close(&state, true);
+	if (ret)
+		return log_msg_ret("run", ret);
+
+	return 0;
+}
+
+int mtrr_set_valid(int cpu_select, int reg, bool valid)
+{
+	struct mtrr_oper oper;
+
+	oper.opcode = MTRR_OP_SET_VALID;
+	oper.reg = reg;
+	oper.valid = valid;
+
+	return mtrr_start_op(cpu_select, &oper);
+}
+
+int mtrr_set(int cpu_select, int reg, u64 base, u64 mask)
+{
+	struct mtrr_oper oper;
+
+	oper.opcode = MTRR_OP_SET;
+	oper.reg = reg;
+	oper.base = base;
+	oper.mask = mask;
+
+	return mtrr_start_op(cpu_select, &oper);
+}
diff --git a/arch/x86/include/asm/mtrr.h b/arch/x86/include/asm/mtrr.h
index 476d6f8a9cf..6c50a67e1fe 100644
--- a/arch/x86/include/asm/mtrr.h
+++ b/arch/x86/include/asm/mtrr.h
@@ -159,6 +159,27 @@ int mtrr_set_next_var(uint type, uint64_t base, uint64_t size);
  */
 void mtrr_save_all(struct mtrr_info *info);
 
+/**
+ * mtrr_set_valid() - Set the valid flag for a selected MTRR and CPU(s)
+ *
+ * @cpu_select: Selected CPUs (either a CPU number or MP_SELECT_...)
+ * @reg: MTRR register to write (0-7)
+ * @valid: Valid flag to write
+ * @return 0 on success, -ve on error
+ */
+int mtrr_set_valid(int cpu_select, int reg, bool valid);
+
+/**
+ * mtrr_set() - Set the valid flag for a selected MTRR and CPU(s)
+ *
+ * @cpu_select: Selected CPUs (either a CPU number or MP_SELECT_...)
+ * @reg: MTRR register to write (0-7)
+ * @base: Base address and MTRR_BASE_TYPE_MASK
+ * @mask: Mask and MTRR_PHYS_MASK_VALID
+ * @return 0 on success, -ve on error
+ */
+int mtrr_set(int cpu_select, int reg, u64 base, u64 mask);
+
 #endif
 
 #if ((CONFIG_XIP_ROM_SIZE & (CONFIG_XIP_ROM_SIZE - 1)) != 0)
-- 
2.27.0.rc0.183.gde8f92d652-goog

[PATCH 18/22] x86: mtrr: Add support for writing to MTRRs on any CPU

[Wolfgang Wallner]

Hi Simon,

-----"Simon Glass" <sjg@chromium.org> schrieb: -----
> Betreff: [PATCH 18/22] x86: mtrr: Add support for writing to MTRRs on any CPU
> 
> To enable support for the 'mtrr' command, add a way to perform MTRR
> operations on selected CPUs.
> 
> This works by setting up a little 'operation' structure and sending it
> around the CPUs for action.
> 
> Signed-off-by: Simon Glass <sjg@chromium.org>
> ---
> 
>  arch/x86/cpu/mtrr.c         | 81 +++++++++++++++++++++++++++++++++++++
>  arch/x86/include/asm/mtrr.h | 21 ++++++++++
>  2 files changed, 102 insertions(+)
> 
> diff --git a/arch/x86/cpu/mtrr.c b/arch/x86/cpu/mtrr.c
> index a48c9d8232e..25f317d4298 100644
> --- a/arch/x86/cpu/mtrr.c
> +++ b/arch/x86/cpu/mtrr.c
> @@ -221,3 +221,84 @@ int mtrr_set_next_var(uint type, uint64_t start, uint64_t size)
>  
>  	return 0;
>  }
> +
> +/** enum mtrr_opcode - supported operations for mtrr_do_oper() */
> +enum mtrr_opcode {
> +	MTRR_OP_SET,
> +	MTRR_OP_SET_VALID,
> +};
> +
> +/**
> + * struct mtrr_oper - An MTRR operation to perform on a CPU
> + *
> + * @opcode: Indicates operation to perform
> + * @reg: MTRR reg number to select (0-7, -1 = all)
> + * @valid: Valid value to write for MTRR_OP_SET_VALID
> + * @base: Base value to write for MTRR_OP_SET
> + * @mask: Mask value to write for MTRR_OP_SET
> + */
> +struct mtrr_oper {
> +	enum mtrr_opcode opcode;
> +	int reg;
> +	bool valid;
> +	u64 base;
> +	u64 mask;
> +};
> +
> +static void mtrr_do_oper(void *arg)
> +{
> +	struct mtrr_oper *oper = arg;
> +	u64 mask;
> +
> +	switch (oper->opcode) {
> +	case MTRR_OP_SET_VALID:
> +		mask = native_read_msr(MTRR_PHYS_MASK_MSR(oper->reg));
> +		if (oper->valid)
> +			mask |= MTRR_PHYS_MASK_VALID;
> +		else
> +			mask &= ~MTRR_PHYS_MASK_VALID;
> +		wrmsrl(MTRR_PHYS_MASK_MSR(oper->reg), mask);
> +		break;
> +	case MTRR_OP_SET:
> +		wrmsrl(MTRR_PHYS_BASE_MSR(oper->reg), oper->base);
> +		wrmsrl(MTRR_PHYS_MASK_MSR(oper->reg), oper->mask);
> +		break;
> +	}
> +}
> +
> +static int mtrr_start_op(int cpu_select, struct mtrr_oper *oper)
> +{
> +	struct mtrr_state state;
> +	int ret;
> +
> +	mtrr_open(&state, true);
> +	ret = mp_run_on_cpus(cpu_select, mtrr_do_oper, oper);
> +	mtrr_close(&state, true);
> +	if (ret)
> +		return log_msg_ret("run", ret);
> +
> +	return 0;
> +}
> +
> +int mtrr_set_valid(int cpu_select, int reg, bool valid)

With the introduction of this function in this patch there are now two
conflicting implementations of mtrr_set_valid(), and this commit does not
compile.

The conflicting function is removed in the following patch. Could these
two patches be merged so we don't break bisectability?

> +{
> +	struct mtrr_oper oper;
> +
> +	oper.opcode = MTRR_OP_SET_VALID;
> +	oper.reg = reg;
> +	oper.valid = valid;
> +
> +	return mtrr_start_op(cpu_select, &oper);
> +}
> +
> +int mtrr_set(int cpu_select, int reg, u64 base, u64 mask)
> +{
> +	struct mtrr_oper oper;
> +
> +	oper.opcode = MTRR_OP_SET;
> +	oper.reg = reg;
> +	oper.base = base;
> +	oper.mask = mask;
> +
> +	return mtrr_start_op(cpu_select, &oper);
> +}
> diff --git a/arch/x86/include/asm/mtrr.h b/arch/x86/include/asm/mtrr.h
> index 476d6f8a9cf..6c50a67e1fe 100644
> --- a/arch/x86/include/asm/mtrr.h
> +++ b/arch/x86/include/asm/mtrr.h
> @@ -159,6 +159,27 @@ int mtrr_set_next_var(uint type, uint64_t base, uint64_t size);
>   */
>  void mtrr_save_all(struct mtrr_info *info);
>  
> +/**
> + * mtrr_set_valid() - Set the valid flag for a selected MTRR and CPU(s)
> + *
> + * @cpu_select: Selected CPUs (either a CPU number or MP_SELECT_...)
> + * @reg: MTRR register to write (0-7)
> + * @valid: Valid flag to write
> + * @return 0 on success, -ve on error
> + */
> +int mtrr_set_valid(int cpu_select, int reg, bool valid);
> +
> +/**
> + * mtrr_set() - Set the valid flag for a selected MTRR and CPU(s)
> + *
> + * @cpu_select: Selected CPUs (either a CPU number or MP_SELECT_...)
> + * @reg: MTRR register to write (0-7)
> + * @base: Base address and MTRR_BASE_TYPE_MASK
> + * @mask: Mask and MTRR_PHYS_MASK_VALID
> + * @return 0 on success, -ve on error
> + */
> +int mtrr_set(int cpu_select, int reg, u64 base, u64 mask);
> +
>  #endif
>  
>  #if ((CONFIG_XIP_ROM_SIZE & (CONFIG_XIP_ROM_SIZE - 1)) != 0)
> -- 
> 2.27.0.rc0.183.gde8f92d652-goog

regards, Wolfgang

[PATCH 19/22] x86: mtrr: Update the command to use the new mtrr calls

[Simon Glass]

Use the multi-CPU calls to set the MTRR values. This still supports only
the boot CPU for now.

Signed-off-by: Simon Glass <sjg@chromium.org>
---

 cmd/x86/mtrr.c | 34 ++++++++--------------------------
 1 file changed, 8 insertions(+), 26 deletions(-)

diff --git a/cmd/x86/mtrr.c b/cmd/x86/mtrr.c
index 01197044452..4e48a16cf43 100644
--- a/cmd/x86/mtrr.c
+++ b/cmd/x86/mtrr.c
@@ -59,14 +59,14 @@ static int do_mtrr_list(int cpu_select)
 	return 0;
 }
 
-static int do_mtrr_set(uint reg, int argc, char *const argv[])
+static int do_mtrr_set(int cpu_select, uint reg, int argc, char *const argv[])
 {
 	const char *typename = argv[0];
-	struct mtrr_state state;
 	uint32_t start, size;
 	uint64_t base, mask;
 	int i, type = -1;
 	bool valid;
+	int ret;
 
 	if (argc < 3)
 		return CMD_RET_USAGE;
@@ -88,27 +88,9 @@ static int do_mtrr_set(uint reg, int argc, char *const argv[])
 	if (valid)
 		mask |= MTRR_PHYS_MASK_VALID;
 
-	mtrr_open(&state, true);
-	wrmsrl(MTRR_PHYS_BASE_MSR(reg), base);
-	wrmsrl(MTRR_PHYS_MASK_MSR(reg), mask);
-	mtrr_close(&state, true);
-
-	return 0;
-}
-
-static int mtrr_set_valid(int reg, bool valid)
-{
-	struct mtrr_state state;
-	uint64_t mask;
-
-	mtrr_open(&state, true);
-	mask = native_read_msr(MTRR_PHYS_MASK_MSR(reg));
-	if (valid)
-		mask |= MTRR_PHYS_MASK_VALID;
-	else
-		mask &= ~MTRR_PHYS_MASK_VALID;
-	wrmsrl(MTRR_PHYS_MASK_MSR(reg), mask);
-	mtrr_close(&state, true);
+	ret = mtrr_set(cpu_select, reg, base, mask);
+	if (ret)
+		return CMD_RET_FAILURE;
 
 	return 0;
 }
@@ -134,11 +116,11 @@ static int do_mtrr(struct cmd_tbl *cmdtp, int flag, int argc,
 		return CMD_RET_USAGE;
 	}
 	if (*cmd == 'e')
-		return mtrr_set_valid(reg, true);
+		return mtrr_set_valid(cpu_select, reg, true);
 	else if (*cmd == 'd')
-		return mtrr_set_valid(reg, false);
+		return mtrr_set_valid(cpu_select, reg, false);
 	else if (*cmd == 's')
-		return do_mtrr_set(reg, argc - 1, argv + 1);
+		return do_mtrr_set(cpu_select, reg, argc - 1, argv + 1);
 	else
 		return CMD_RET_USAGE;
 
-- 
2.27.0.rc0.183.gde8f92d652-goog

[PATCH 19/22] x86: mtrr: Update the command to use the new mtrr calls

[Wolfgang Wallner]

Hi Simon,

-----"Simon Glass" <sjg@chromium.org> schrieb: -----
> Betreff: [PATCH 19/22] x86: mtrr: Update the command to use the new mtrr calls
> 
> Use the multi-CPU calls to set the MTRR values. This still supports only
> the boot CPU for now.
> 
> Signed-off-by: Simon Glass <sjg@chromium.org>
> ---
> 
>  cmd/x86/mtrr.c | 34 ++++++++--------------------------
>  1 file changed, 8 insertions(+), 26 deletions(-)

Reviewed-by: Wolfgang Wallner <wolfgang.wallner@br-automation.com>

[PATCH 20/22] x86: mtrr: Restructure so command execution is in one place

[Simon Glass]

At present do_mtrr() does the 'list' subcommand at the top and the rest
below. Update it to do them all in the same place so we can (in a later
patch) add parsing of the CPU number for all subcommands.

Signed-off-by: Simon Glass <sjg@chromium.org>
---

 cmd/x86/mtrr.c | 55 +++++++++++++++++++++++++++++++++-----------------
 1 file changed, 36 insertions(+), 19 deletions(-)

diff --git a/cmd/x86/mtrr.c b/cmd/x86/mtrr.c
index 4e48a16cf43..fea7a437db8 100644
--- a/cmd/x86/mtrr.c
+++ b/cmd/x86/mtrr.c
@@ -98,31 +98,48 @@ static int do_mtrr_set(int cpu_select, uint reg, int argc, char *const argv[])
 static int do_mtrr(struct cmd_tbl *cmdtp, int flag, int argc,
 		   char *const argv[])
 {
-	const char *cmd;
+	int cmd;
 	int cpu_select;
 	uint reg;
+	int ret;
 
 	cpu_select = MP_SELECT_BSP;
-	cmd = argv[1];
-	if (argc < 2 || *cmd == 'l')
+	argc--;
+	argv++;
+	cmd = argv[0] ? *argv[0] : 0;
+	if (argc < 1 || !cmd) {
+		cmd = 'l';
+		reg = 0;
+	} else {
+		if (argc < 2)
+			return CMD_RET_USAGE;
+		reg = simple_strtoul(argv[1], NULL, 16);
+		if (reg >= MTRR_COUNT) {
+			printf("Invalid register number\n");
+			return CMD_RET_USAGE;
+		}
+	}
+	if (cmd == 'l') {
 		return do_mtrr_list(cpu_select);
-	argc -= 2;
-	argv += 2;
-	if (argc <= 0)
-		return CMD_RET_USAGE;
-	reg = simple_strtoul(argv[0], NULL, 16);
-	if (reg >= MTRR_COUNT) {
-		printf("Invalid register number\n");
-		return CMD_RET_USAGE;
+	} else {
+		switch (cmd) {
+		case 'e':
+			ret = mtrr_set_valid(cpu_select, reg, true);
+			break;
+		case 'd':
+			ret = mtrr_set_valid(cpu_select, reg, false);
+			break;
+		case 's':
+			ret = do_mtrr_set(cpu_select, reg, argc - 2, argv + 2);
+			break;
+		default:
+			return CMD_RET_USAGE;
+		}
+		if (ret) {
+			printf("Operation failed (err=%d)\n", ret);
+			return CMD_RET_FAILURE;
+		}
 	}
-	if (*cmd == 'e')
-		return mtrr_set_valid(cpu_select, reg, true);
-	else if (*cmd == 'd')
-		return mtrr_set_valid(cpu_select, reg, false);
-	else if (*cmd == 's')
-		return do_mtrr_set(cpu_select, reg, argc - 1, argv + 1);
-	else
-		return CMD_RET_USAGE;
 
 	return 0;
 }
-- 
2.27.0.rc0.183.gde8f92d652-goog

[PATCH 20/22] x86: mtrr: Restructure so command execution is in one place

[Wolfgang Wallner]

Hi Simon,

-----"Simon Glass" <sjg@chromium.org> schrieb: -----
> Betreff: [PATCH 20/22] x86: mtrr: Restructure so command execution is in one place
> 
> At present do_mtrr() does the 'list' subcommand at the top and the rest
> below. Update it to do them all in the same place so we can (in a later
> patch) add parsing of the CPU number for all subcommands.
> 
> Signed-off-by: Simon Glass <sjg@chromium.org>
> ---
> 
>  cmd/x86/mtrr.c | 55 +++++++++++++++++++++++++++++++++-----------------
>  1 file changed, 36 insertions(+), 19 deletions(-)

Reviewed-by: Wolfgang Wallner <wolfgang.wallner@br-automation.com>

[PATCH 21/22] x86: mtrr: Update 'mtrr' to allow setting MTRRs on any CPU

[Simon Glass]

Add a -c option to mtrr to allow any CPU to be updated with this command.

Signed-off-by: Simon Glass <sjg@chromium.org>
---

 cmd/x86/mtrr.c | 18 ++++++++++++++++--
 1 file changed, 16 insertions(+), 2 deletions(-)

diff --git a/cmd/x86/mtrr.c b/cmd/x86/mtrr.c
index fea7a437db8..8365a7978ff 100644
--- a/cmd/x86/mtrr.c
+++ b/cmd/x86/mtrr.c
@@ -104,6 +104,17 @@ static int do_mtrr(struct cmd_tbl *cmdtp, int flag, int argc,
 	int ret;
 
 	cpu_select = MP_SELECT_BSP;
+	if (argc >= 3 && !strcmp("-c", argv[1])) {
+		const char *cpustr;
+
+		cpustr = argv[2];
+		if (*cpustr == 'a')
+			cpu_select = MP_SELECT_ALL;
+		else
+			cpu_select = simple_strtol(cpustr, NULL, 16);
+		argc -= 2;
+		argv += 2;
+	}
 	argc--;
 	argv++;
 	cmd = argv[0] ? *argv[0] : 0;
@@ -145,11 +156,14 @@ static int do_mtrr(struct cmd_tbl *cmdtp, int flag, int argc,
 }
 
 U_BOOT_CMD(
-	mtrr,	6,	1,	do_mtrr,
+	mtrr,	8,	1,	do_mtrr,
 	"Use x86 memory type range registers (32-bit only)",
 	"[list]        - list current registers\n"
 	"set <reg> <type> <start> <size>   - set a register\n"
 	"\t<type> is Uncacheable, Combine, Through, Protect, Back\n"
 	"disable <reg>      - disable a register\n"
-	"enable <reg>       - enable a register"
+	"enable <reg>       - enable a register\n"
+	"\n"
+	"Precede command with '-c <n>|all' to access a particular CPU, e.g.\n"
+	"   mtrr -c all list; mtrr -c 2e list"
 );
-- 
2.27.0.rc0.183.gde8f92d652-goog

[PATCH 21/22] x86: mtrr: Update 'mtrr' to allow setting MTRRs on any CPU

[Wolfgang Wallner]

Hi Simon,

-----"Simon Glass" <sjg@chromium.org> schrieb: -----
> Betreff: [PATCH 21/22] x86: mtrr: Update 'mtrr' to allow setting MTRRs on any CPU
> 
> Add a -c option to mtrr to allow any CPU to be updated with this command.
> 
> Signed-off-by: Simon Glass <sjg@chromium.org>
> ---
> 
>  cmd/x86/mtrr.c | 18 ++++++++++++++++--
>  1 file changed, 16 insertions(+), 2 deletions(-)

Reviewed-by: Wolfgang Wallner <wolfgang.wallner@br-automation.com>

[PATCH 22/22] x86: mtrr: Enhance 'mtrr' command to list MTRRs on any CPU

[Simon Glass]

Update this command so it can list the MTRRs on a selected CPU. If
'-c all' is used, then all CPUs are listed.

Signed-off-by: Simon Glass <sjg@chromium.org>
---

 cmd/x86/mtrr.c | 22 +++++++++++++++++++++-
 1 file changed, 21 insertions(+), 1 deletion(-)

diff --git a/cmd/x86/mtrr.c b/cmd/x86/mtrr.c
index 8365a7978ff..66ef48ff9f3 100644
--- a/cmd/x86/mtrr.c
+++ b/cmd/x86/mtrr.c
@@ -131,7 +131,27 @@ static int do_mtrr(struct cmd_tbl *cmdtp, int flag, int argc,
 		}
 	}
 	if (cmd == 'l') {
-		return do_mtrr_list(cpu_select);
+		bool first;
+		int i;
+
+		i = mp_first_cpu(cpu_select);
+		if (i < 0) {
+			printf("Invalid CPU (err=%d)\n", i);
+			return CMD_RET_FAILURE;
+		}
+		first = true;
+		for (; i >= 0; i = mp_next_cpu(cpu_select, i)) {
+			if (!first)
+				printf("\n");
+			printf("CPU %d:\n", i);
+			ret = do_mtrr_list(i);
+			if (ret) {
+				printf("Failed to read CPU %d (err=%d)\n", i,
+				       ret);
+				return CMD_RET_FAILURE;
+			}
+			first = false;
+		}
 	} else {
 		switch (cmd) {
 		case 'e':
-- 
2.27.0.rc0.183.gde8f92d652-goog

[PATCH 22/22] x86: mtrr: Enhance 'mtrr' command to list MTRRs on any CPU

[Wolfgang Wallner]

Hi Simon,

-----"Simon Glass" <sjg@chromium.org> schrieb: -----
> Betreff: [PATCH 22/22] x86: mtrr: Enhance 'mtrr' command to list MTRRs on any CPU
> 
> Update this command so it can list the MTRRs on a selected CPU. If
> '-c all' is used, then all CPUs are listed.
> 
> Signed-off-by: Simon Glass <sjg@chromium.org>
> ---
> 
>  cmd/x86/mtrr.c | 22 +++++++++++++++++++++-
>  1 file changed, 21 insertions(+), 1 deletion(-)

Reviewed-by: Wolfgang Wallner <wolfgang.wallner@br-automation.com>

[PATCH 00/22] x86: Enhance MTRR functionality to support multiple CPUs

[Andy Shevchenko]

On Thu, May 21, 2020 at 08:23:04PM -0600, Simon Glass wrote:
> At present MTRRs are mirrored to the secondary CPUs only once, as those
> CPUs are started up. But U-Boot may add more MTRRs later, e.g. if it
> decides that a video console must be set up.
> 
> This series enhances the x86 multi-processor support to allow MTRRs to
> be updated at any time. It also updates the 'mtrr' command to support
> setting the MTRRs on CPUs other than the boot CPU.

Hmm... Why do you need MTRR if CPU supports PAT?

> 
> 
> Simon Glass (22):
>   x86: mp_init: Switch to livetree
>   x86: Move MP code into mp_init
>   x86: mp_init: Avoid declarations in header files
>   x86: mp_init: Switch parameter names in start_aps()
>   x86: mp_init: Drop the num_cpus static variable
>   x86: mtrr: Fix 'ensable' typo
>   x86: mp_init: Set up the CPU numbers at the start
>   x86: mp_init: Adjust bsp_init() to return more information
>   x86: cpu: Remove unnecessary #ifdefs
>   x86: mp: Support APs waiting for instructions
>   global_data: Add a generic global_data flag for SMP state
>   x86: Set the SMP flag when MP init is complete
>   x86: mp: Allow running functions on multiple CPUs
>   x86: mp: Park CPUs before running the OS
>   x86: mp: Add iterators for CPUs
>   x86: mtrr: Use MP calls to list the MTRRs
>   x86: mtrr: Update MTRRs on all CPUs
>   x86: mtrr: Add support for writing to MTRRs on any CPU
>   x86: mtrr: Update the command to use the new mtrr calls
>   x86: mtrr: Restructure so command execution is in one place
>   x86: mtrr: Update 'mtrr' to allow setting MTRRs on any CPU
>   x86: mtrr: Enhance 'mtrr' command to list MTRRs on any CPU
> 
>  arch/x86/cpu/cpu.c                |  63 ++---
>  arch/x86/cpu/i386/cpu.c           |  26 +--
>  arch/x86/cpu/mp_init.c            | 377 +++++++++++++++++++++++++-----
>  arch/x86/cpu/mtrr.c               | 149 ++++++++++++
>  arch/x86/include/asm/mp.h         | 118 ++++++++--
>  arch/x86/include/asm/mtrr.h       |  51 ++++
>  cmd/x86/mtrr.c                    | 148 ++++++++----
>  include/asm-generic/global_data.h |   1 +
>  include/dm/uclass.h               |   2 +-
>  9 files changed, 751 insertions(+), 184 deletions(-)
> 
> -- 
> 2.27.0.rc0.183.gde8f92d652-goog
> 

-- 
With Best Regards,
Andy Shevchenko

[PATCH 00/22] x86: Enhance MTRR functionality to support multiple CPUs

[Simon Glass]

Hi Andy,

On Fri, 22 May 2020 at 04:55, Andy Shevchenko
<andriy.shevchenko@linux.intel.com> wrote:
>
> On Thu, May 21, 2020 at 08:23:04PM -0600, Simon Glass wrote:
> > At present MTRRs are mirrored to the secondary CPUs only once, as those
> > CPUs are started up. But U-Boot may add more MTRRs later, e.g. if it
> > decides that a video console must be set up.
> >
> > This series enhances the x86 multi-processor support to allow MTRRs to
> > be updated at any time. It also updates the 'mtrr' command to support
> > setting the MTRRs on CPUs other than the boot CPU.
>
> Hmm... Why do you need MTRR if CPU supports PAT?

I suspect U-Boot could move to PAT one day but the needs are small in firmware.

Regards,
Simon