* [PATCH] arm64: kexec: add support for kexec with spin-table
@ 2021-07-14 17:41 ` Henry Willard
0 siblings, 0 replies; 14+ messages in thread
From: Henry Willard @ 2021-07-14 17:41 UTC (permalink / raw)
To: catalin.marinas, will, mark.rutland, tabba, keescook, ardb,
samitolvanen, joe, nixiaoming, linux-arm-kernel, linux-kernel
With one special exception kexec is not supported on systems
that use spin-table as the cpu enablement method instead of PSCI.
The spin-table implementation lacks cpu_die() and several other
methods needed by the hotplug framework used by kexec on Arm64.
Some embedded systems may not have a need for the Arm Trusted
Firmware, or they may lack it during early bring-up. Some of
these may have a more primitive version of u-boot that uses a
special device from which to load the kernel. Kexec can be
especially useful for testing new kernels in such an environment.
What is needed to support kexec is some place for cpu_die to park
the secondary CPUs outside the kernel while the primary copies
the new kernel into place and starts it. One possibility is to
use the control-code-page where arm64_relocate_new_kernel_size()
executes, but that requires a complicated and racy dance to get
the secondary CPUs from the control-code-page to the new
kernel after it has been copied.
The spin-table mechanism is setup before the Linux kernel
is entered with details provided in the device tree. The
"release-address" DT variable provides the address of a word the
secondary CPUs are polling. The boot CPU will store the real address
of secondary_holding_pen() at that address, and the secondary CPUs
will branch to that address. secondary_holding_pen() is another
loop where the secondary CPUs wait to be called up by the boot CPU.
This patch uses that mechanism to implement cpu_die(). In modern
versions of u-boot that implement spin-table, the address of the
loop in protected memory can be derived from the "release-address"
value. The patch validates the existence of the loop before
proceeding. smp_spin_table_cpu_die() uses cpu_soft_restart() to
branch to the loop with the MMU and caching turned off where the
CPU waits until released by the new kernel. After that kexec
reboot proceeds normally.
The special exception is the kdump capture kernel, which gets
started even if the secondaries can't be stopped.
Signed-off-by: Henry Willard <henry.willard@oracle.com>
---
arch/arm64/kernel/smp_spin_table.c | 111 +++++++++++++++++++++++++++++++++++++
1 file changed, 111 insertions(+)
diff --git a/arch/arm64/kernel/smp_spin_table.c b/arch/arm64/kernel/smp_spin_table.c
index 7e1624ecab3c..35c7fa764476 100644
--- a/arch/arm64/kernel/smp_spin_table.c
+++ b/arch/arm64/kernel/smp_spin_table.c
@@ -13,16 +13,27 @@
#include <linux/mm.h>
#include <asm/cacheflush.h>
+#include <asm/daifflags.h>
#include <asm/cpu_ops.h>
#include <asm/cputype.h>
#include <asm/io.h>
#include <asm/smp_plat.h>
+#include <asm/mmu_context.h>
+#include <asm/kexec.h>
+
+#include "cpu-reset.h"
extern void secondary_holding_pen(void);
volatile unsigned long __section(".mmuoff.data.read")
secondary_holding_pen_release = INVALID_HWID;
static phys_addr_t cpu_release_addr[NR_CPUS];
+static unsigned int spin_table_loop[4] = {
+ 0xd503205f, /* wfe */
+ 0x58000060, /* ldr x0, spin_table_cpu_release_addr */
+ 0xb4ffffc0, /* cbnz x0, 0b */
+ 0xd61f0000 /* br x0 */
+};
/*
* Write secondary_holding_pen_release in a way that is guaranteed to be
@@ -119,9 +130,109 @@ static int smp_spin_table_cpu_boot(unsigned int cpu)
return 0;
}
+
+/*
+ * There is a four instruction loop set aside in protected
+ * memory by u-boot where secondary CPUs wait for the kernel to
+ * start.
+ *
+ * 0: wfe
+ * ldr x0, spin_table_cpu_release_addr
+ * cbz x0, 0b
+ * br x0
+ * spin_table_cpu_release_addr:
+ * .quad 0
+ *
+ * The address of spin_table_cpu_release_addr is passed in the
+ * "release-address" property in the device table.
+ * smp_spin_table_cpu_prepare() stores the real address of
+ * secondary_holding_pen() where the secondary CPUs loop
+ * until they are released one at a time by smp_spin_table_cpu_boot().
+ * We reuse the spin-table loop by clearing spin_table_cpu_release_addr,
+ * and branching to the beginning of the loop via cpu_soft_restart(),
+ * which turns off the MMU and caching.
+ */
+static void smp_spin_table_cpu_die(unsigned int cpu)
+{
+ __le64 __iomem *release_addr;
+ unsigned int *spin_table_inst;
+ unsigned long spin_table_start;
+
+ if (!cpu_release_addr[cpu])
+ goto spin;
+
+ spin_table_start = (cpu_release_addr[cpu] - sizeof(spin_table_loop));
+
+ /*
+ * The cpu-release-addr may or may not be inside the linear mapping.
+ * As ioremap_cache will either give us a new mapping or reuse the
+ * existing linear mapping, we can use it to cover both cases. In
+ * either case the memory will be MT_NORMAL.
+ */
+ release_addr = ioremap_cache(spin_table_start,
+ sizeof(*release_addr) +
+ sizeof(spin_table_loop));
+
+ if (!release_addr)
+ goto spin;
+
+ spin_table_inst = (unsigned int *)release_addr;
+ if (spin_table_inst[0] != spin_table_loop[0] ||
+ spin_table_inst[1] != spin_table_loop[1] ||
+ spin_table_inst[2] != spin_table_loop[2] ||
+ spin_table_inst[3] != spin_table_loop[3])
+ goto spin;
+
+ /*
+ * Clear the release address, so that we can use it again
+ */
+ writeq_relaxed(0, release_addr + 2);
+ dcache_clean_inval_poc((__force unsigned long)(release_addr + 2),
+ (__force unsigned long)(release_addr + 2) +
+ sizeof(*release_addr));
+
+ iounmap(release_addr);
+
+ local_daif_mask();
+ cpu_soft_restart(spin_table_start, 0, 0, 0);
+
+ BUG(); /* Should never get here */
+
+spin:
+ cpu_park_loop();
+
+}
+
+static int smp_spin_table_cpu_kill(unsigned int cpu)
+{
+ unsigned long start, end;
+
+ start = jiffies;
+ end = start + msecs_to_jiffies(100);
+
+ do {
+ if (!cpu_online(cpu)) {
+ pr_info("CPU%d killed\n", cpu);
+ return 0;
+ }
+ } while (time_before(jiffies, end));
+ pr_warn("CPU%d may not have shut down cleanly\n", cpu);
+ return -ETIMEDOUT;
+
+}
+
+/* Nothing to do here */
+static int smp_spin_table_cpu_disable(unsigned int cpu)
+{
+ return 0;
+}
+
const struct cpu_operations smp_spin_table_ops = {
.name = "spin-table",
.cpu_init = smp_spin_table_cpu_init,
.cpu_prepare = smp_spin_table_cpu_prepare,
.cpu_boot = smp_spin_table_cpu_boot,
+ .cpu_die = smp_spin_table_cpu_die,
+ .cpu_kill = smp_spin_table_cpu_kill,
+ .cpu_disable = smp_spin_table_cpu_disable,
};
--
1.8.3.1
^ permalink raw reply related [flat|nested] 14+ messages in thread
* [PATCH] arm64: kexec: add support for kexec with spin-table
@ 2021-07-14 17:41 ` Henry Willard
0 siblings, 0 replies; 14+ messages in thread
From: Henry Willard @ 2021-07-14 17:41 UTC (permalink / raw)
To: catalin.marinas, will, mark.rutland, tabba, keescook, ardb,
samitolvanen, joe, nixiaoming, linux-arm-kernel, linux-kernel
With one special exception kexec is not supported on systems
that use spin-table as the cpu enablement method instead of PSCI.
The spin-table implementation lacks cpu_die() and several other
methods needed by the hotplug framework used by kexec on Arm64.
Some embedded systems may not have a need for the Arm Trusted
Firmware, or they may lack it during early bring-up. Some of
these may have a more primitive version of u-boot that uses a
special device from which to load the kernel. Kexec can be
especially useful for testing new kernels in such an environment.
What is needed to support kexec is some place for cpu_die to park
the secondary CPUs outside the kernel while the primary copies
the new kernel into place and starts it. One possibility is to
use the control-code-page where arm64_relocate_new_kernel_size()
executes, but that requires a complicated and racy dance to get
the secondary CPUs from the control-code-page to the new
kernel after it has been copied.
The spin-table mechanism is setup before the Linux kernel
is entered with details provided in the device tree. The
"release-address" DT variable provides the address of a word the
secondary CPUs are polling. The boot CPU will store the real address
of secondary_holding_pen() at that address, and the secondary CPUs
will branch to that address. secondary_holding_pen() is another
loop where the secondary CPUs wait to be called up by the boot CPU.
This patch uses that mechanism to implement cpu_die(). In modern
versions of u-boot that implement spin-table, the address of the
loop in protected memory can be derived from the "release-address"
value. The patch validates the existence of the loop before
proceeding. smp_spin_table_cpu_die() uses cpu_soft_restart() to
branch to the loop with the MMU and caching turned off where the
CPU waits until released by the new kernel. After that kexec
reboot proceeds normally.
The special exception is the kdump capture kernel, which gets
started even if the secondaries can't be stopped.
Signed-off-by: Henry Willard <henry.willard@oracle.com>
---
arch/arm64/kernel/smp_spin_table.c | 111 +++++++++++++++++++++++++++++++++++++
1 file changed, 111 insertions(+)
diff --git a/arch/arm64/kernel/smp_spin_table.c b/arch/arm64/kernel/smp_spin_table.c
index 7e1624ecab3c..35c7fa764476 100644
--- a/arch/arm64/kernel/smp_spin_table.c
+++ b/arch/arm64/kernel/smp_spin_table.c
@@ -13,16 +13,27 @@
#include <linux/mm.h>
#include <asm/cacheflush.h>
+#include <asm/daifflags.h>
#include <asm/cpu_ops.h>
#include <asm/cputype.h>
#include <asm/io.h>
#include <asm/smp_plat.h>
+#include <asm/mmu_context.h>
+#include <asm/kexec.h>
+
+#include "cpu-reset.h"
extern void secondary_holding_pen(void);
volatile unsigned long __section(".mmuoff.data.read")
secondary_holding_pen_release = INVALID_HWID;
static phys_addr_t cpu_release_addr[NR_CPUS];
+static unsigned int spin_table_loop[4] = {
+ 0xd503205f, /* wfe */
+ 0x58000060, /* ldr x0, spin_table_cpu_release_addr */
+ 0xb4ffffc0, /* cbnz x0, 0b */
+ 0xd61f0000 /* br x0 */
+};
/*
* Write secondary_holding_pen_release in a way that is guaranteed to be
@@ -119,9 +130,109 @@ static int smp_spin_table_cpu_boot(unsigned int cpu)
return 0;
}
+
+/*
+ * There is a four instruction loop set aside in protected
+ * memory by u-boot where secondary CPUs wait for the kernel to
+ * start.
+ *
+ * 0: wfe
+ * ldr x0, spin_table_cpu_release_addr
+ * cbz x0, 0b
+ * br x0
+ * spin_table_cpu_release_addr:
+ * .quad 0
+ *
+ * The address of spin_table_cpu_release_addr is passed in the
+ * "release-address" property in the device table.
+ * smp_spin_table_cpu_prepare() stores the real address of
+ * secondary_holding_pen() where the secondary CPUs loop
+ * until they are released one at a time by smp_spin_table_cpu_boot().
+ * We reuse the spin-table loop by clearing spin_table_cpu_release_addr,
+ * and branching to the beginning of the loop via cpu_soft_restart(),
+ * which turns off the MMU and caching.
+ */
+static void smp_spin_table_cpu_die(unsigned int cpu)
+{
+ __le64 __iomem *release_addr;
+ unsigned int *spin_table_inst;
+ unsigned long spin_table_start;
+
+ if (!cpu_release_addr[cpu])
+ goto spin;
+
+ spin_table_start = (cpu_release_addr[cpu] - sizeof(spin_table_loop));
+
+ /*
+ * The cpu-release-addr may or may not be inside the linear mapping.
+ * As ioremap_cache will either give us a new mapping or reuse the
+ * existing linear mapping, we can use it to cover both cases. In
+ * either case the memory will be MT_NORMAL.
+ */
+ release_addr = ioremap_cache(spin_table_start,
+ sizeof(*release_addr) +
+ sizeof(spin_table_loop));
+
+ if (!release_addr)
+ goto spin;
+
+ spin_table_inst = (unsigned int *)release_addr;
+ if (spin_table_inst[0] != spin_table_loop[0] ||
+ spin_table_inst[1] != spin_table_loop[1] ||
+ spin_table_inst[2] != spin_table_loop[2] ||
+ spin_table_inst[3] != spin_table_loop[3])
+ goto spin;
+
+ /*
+ * Clear the release address, so that we can use it again
+ */
+ writeq_relaxed(0, release_addr + 2);
+ dcache_clean_inval_poc((__force unsigned long)(release_addr + 2),
+ (__force unsigned long)(release_addr + 2) +
+ sizeof(*release_addr));
+
+ iounmap(release_addr);
+
+ local_daif_mask();
+ cpu_soft_restart(spin_table_start, 0, 0, 0);
+
+ BUG(); /* Should never get here */
+
+spin:
+ cpu_park_loop();
+
+}
+
+static int smp_spin_table_cpu_kill(unsigned int cpu)
+{
+ unsigned long start, end;
+
+ start = jiffies;
+ end = start + msecs_to_jiffies(100);
+
+ do {
+ if (!cpu_online(cpu)) {
+ pr_info("CPU%d killed\n", cpu);
+ return 0;
+ }
+ } while (time_before(jiffies, end));
+ pr_warn("CPU%d may not have shut down cleanly\n", cpu);
+ return -ETIMEDOUT;
+
+}
+
+/* Nothing to do here */
+static int smp_spin_table_cpu_disable(unsigned int cpu)
+{
+ return 0;
+}
+
const struct cpu_operations smp_spin_table_ops = {
.name = "spin-table",
.cpu_init = smp_spin_table_cpu_init,
.cpu_prepare = smp_spin_table_cpu_prepare,
.cpu_boot = smp_spin_table_cpu_boot,
+ .cpu_die = smp_spin_table_cpu_die,
+ .cpu_kill = smp_spin_table_cpu_kill,
+ .cpu_disable = smp_spin_table_cpu_disable,
};
--
1.8.3.1
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 14+ messages in thread
* Re: [PATCH] arm64: kexec: add support for kexec with spin-table
2021-07-14 17:41 ` Henry Willard
@ 2021-07-14 18:47 ` Mark Rutland
-1 siblings, 0 replies; 14+ messages in thread
From: Mark Rutland @ 2021-07-14 18:47 UTC (permalink / raw)
To: Henry Willard
Cc: catalin.marinas, will, tabba, keescook, ardb, samitolvanen, joe,
nixiaoming, linux-arm-kernel, linux-kernel
Hi Henry,
On Wed, Jul 14, 2021 at 10:41:13AM -0700, Henry Willard wrote:
> With one special exception kexec is not supported on systems
> that use spin-table as the cpu enablement method instead of PSCI.
> The spin-table implementation lacks cpu_die() and several other
> methods needed by the hotplug framework used by kexec on Arm64.
>
> Some embedded systems may not have a need for the Arm Trusted
> Firmware, or they may lack it during early bring-up. Some of
> these may have a more primitive version of u-boot that uses a
> special device from which to load the kernel. Kexec can be
> especially useful for testing new kernels in such an environment.
>
> What is needed to support kexec is some place for cpu_die to park
> the secondary CPUs outside the kernel while the primary copies
> the new kernel into place and starts it. One possibility is to
> use the control-code-page where arm64_relocate_new_kernel_size()
> executes, but that requires a complicated and racy dance to get
> the secondary CPUs from the control-code-page to the new
> kernel after it has been copied.
>
> The spin-table mechanism is setup before the Linux kernel
> is entered with details provided in the device tree. The
> "release-address" DT variable provides the address of a word the
> secondary CPUs are polling. The boot CPU will store the real address
> of secondary_holding_pen() at that address, and the secondary CPUs
> will branch to that address. secondary_holding_pen() is another
> loop where the secondary CPUs wait to be called up by the boot CPU.
>
> This patch uses that mechanism to implement cpu_die(). In modern
> versions of u-boot that implement spin-table, the address of the
> loop in protected memory can be derived from the "release-address"
> value. The patch validates the existence of the loop before
> proceeding. smp_spin_table_cpu_die() uses cpu_soft_restart() to
> branch to the loop with the MMU and caching turned off where the
> CPU waits until released by the new kernel. After that kexec
> reboot proceeds normally.
This isn't true for all spin-table implementations; for example this is
not safe with the boot-wrapper.
While, I'm not necessarily opposed to providing a mechanism to return a
CPU back to the spin-table, the presence of that mechanism needs to be
explicitly defined in the device tree (e.g. with a "cpu-return-addr"
property or similar), and we need to thoroughly document the contract
(e.g. what state the CPU is in when it is returned). We've generally
steered clear of this since it is much more complicated than it may
initially seem, and there is immense scope for error.
If we do choose to extend spin-table in this way, we'll also need to
enforce that each cpu has a unique cpu-release-address, or this is
unsound to begin with (since e.g. the kernel can't return CPUs that it
doesn't know are stuck in the holding pen). We will also need a
mechanism to reliably identify when the CPU has been successfully
returned.
I would very much like to avoid this if possible. U-Boot does have a
PSCI implementation that some platforms use; is it not possible to use
this?
If this is for early bringup, and you're using the first kernel as a
bootloader, I'd suggest that you boot that with "nosmp", such that the
first kernel doesn't touch the secondary CPUs at all.
> The special exception is the kdump capture kernel, which gets
> started even if the secondaries can't be stopped.
>
> Signed-off-by: Henry Willard <henry.willard@oracle.com>
> ---
> arch/arm64/kernel/smp_spin_table.c | 111 +++++++++++++++++++++++++++++++++++++
> 1 file changed, 111 insertions(+)
>
> diff --git a/arch/arm64/kernel/smp_spin_table.c b/arch/arm64/kernel/smp_spin_table.c
> index 7e1624ecab3c..35c7fa764476 100644
> --- a/arch/arm64/kernel/smp_spin_table.c
> +++ b/arch/arm64/kernel/smp_spin_table.c
> @@ -13,16 +13,27 @@
> #include <linux/mm.h>
>
> #include <asm/cacheflush.h>
> +#include <asm/daifflags.h>
> #include <asm/cpu_ops.h>
> #include <asm/cputype.h>
> #include <asm/io.h>
> #include <asm/smp_plat.h>
> +#include <asm/mmu_context.h>
> +#include <asm/kexec.h>
> +
> +#include "cpu-reset.h"
>
> extern void secondary_holding_pen(void);
> volatile unsigned long __section(".mmuoff.data.read")
> secondary_holding_pen_release = INVALID_HWID;
>
> static phys_addr_t cpu_release_addr[NR_CPUS];
> +static unsigned int spin_table_loop[4] = {
> + 0xd503205f, /* wfe */
> + 0x58000060, /* ldr x0, spin_table_cpu_release_addr */
> + 0xb4ffffc0, /* cbnz x0, 0b */
> + 0xd61f0000 /* br x0 */
> +};
>
> /*
> * Write secondary_holding_pen_release in a way that is guaranteed to be
> @@ -119,9 +130,109 @@ static int smp_spin_table_cpu_boot(unsigned int cpu)
> return 0;
> }
>
> +
> +/*
> + * There is a four instruction loop set aside in protected
> + * memory by u-boot where secondary CPUs wait for the kernel to
> + * start.
> + *
> + * 0: wfe
> + * ldr x0, spin_table_cpu_release_addr
> + * cbz x0, 0b
> + * br x0
> + * spin_table_cpu_release_addr:
> + * .quad 0
> + *
> + * The address of spin_table_cpu_release_addr is passed in the
> + * "release-address" property in the device table.
> + * smp_spin_table_cpu_prepare() stores the real address of
> + * secondary_holding_pen() where the secondary CPUs loop
> + * until they are released one at a time by smp_spin_table_cpu_boot().
> + * We reuse the spin-table loop by clearing spin_table_cpu_release_addr,
> + * and branching to the beginning of the loop via cpu_soft_restart(),
> + * which turns off the MMU and caching.
> + */
> +static void smp_spin_table_cpu_die(unsigned int cpu)
> +{
> + __le64 __iomem *release_addr;
> + unsigned int *spin_table_inst;
> + unsigned long spin_table_start;
> +
> + if (!cpu_release_addr[cpu])
> + goto spin;
> +
> + spin_table_start = (cpu_release_addr[cpu] - sizeof(spin_table_loop));
> +
> + /*
> + * The cpu-release-addr may or may not be inside the linear mapping.
> + * As ioremap_cache will either give us a new mapping or reuse the
> + * existing linear mapping, we can use it to cover both cases. In
> + * either case the memory will be MT_NORMAL.
> + */
> + release_addr = ioremap_cache(spin_table_start,
> + sizeof(*release_addr) +
> + sizeof(spin_table_loop));
> +
> + if (!release_addr)
> + goto spin;
> +
> + spin_table_inst = (unsigned int *)release_addr;
> + if (spin_table_inst[0] != spin_table_loop[0] ||
> + spin_table_inst[1] != spin_table_loop[1] ||
> + spin_table_inst[2] != spin_table_loop[2] ||
> + spin_table_inst[3] != spin_table_loop[3])
> + goto spin;
Please don't hard-code a specific sequence for this; if we *really* need
this, we should be given a cpu-return-addr explicitly, and we should
simply trust it.
> +
> + /*
> + * Clear the release address, so that we can use it again
> + */
> + writeq_relaxed(0, release_addr + 2);
> + dcache_clean_inval_poc((__force unsigned long)(release_addr + 2),
> + (__force unsigned long)(release_addr + 2) +
> + sizeof(*release_addr));
What is the `+ 2` for?
> +
> + iounmap(release_addr);
> +
> + local_daif_mask();
> + cpu_soft_restart(spin_table_start, 0, 0, 0);
> +
> + BUG(); /* Should never get here */
> +
> +spin:
> + cpu_park_loop();
> +
> +}
> +
> +static int smp_spin_table_cpu_kill(unsigned int cpu)
> +{
> + unsigned long start, end;
> +
> + start = jiffies;
> + end = start + msecs_to_jiffies(100);
> +
> + do {
> + if (!cpu_online(cpu)) {
> + pr_info("CPU%d killed\n", cpu);
> + return 0;
> + }
> + } while (time_before(jiffies, end));
> + pr_warn("CPU%d may not have shut down cleanly\n", cpu);
> + return -ETIMEDOUT;
> +
> +}
If we're going to extend this, we must add a mechanism to reliably
identify when the CPU has been returned successfully. We can't rely on
cpu_online(), becuase there's a window between the CPU marking itself as
offline and actually exiting the kernel.
> +
> +/* Nothing to do here */
> +static int smp_spin_table_cpu_disable(unsigned int cpu)
> +{
> + return 0;
> +}
For implementations where we cannot return the CPU, cpu_disable() *must*
fail.
Thanks,
Mark.
> +
> const struct cpu_operations smp_spin_table_ops = {
> .name = "spin-table",
> .cpu_init = smp_spin_table_cpu_init,
> .cpu_prepare = smp_spin_table_cpu_prepare,
> .cpu_boot = smp_spin_table_cpu_boot,
> + .cpu_die = smp_spin_table_cpu_die,
> + .cpu_kill = smp_spin_table_cpu_kill,
> + .cpu_disable = smp_spin_table_cpu_disable,
> };
> --
> 1.8.3.1
>
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH] arm64: kexec: add support for kexec with spin-table
@ 2021-07-14 18:47 ` Mark Rutland
0 siblings, 0 replies; 14+ messages in thread
From: Mark Rutland @ 2021-07-14 18:47 UTC (permalink / raw)
To: Henry Willard
Cc: catalin.marinas, will, tabba, keescook, ardb, samitolvanen, joe,
nixiaoming, linux-arm-kernel, linux-kernel
Hi Henry,
On Wed, Jul 14, 2021 at 10:41:13AM -0700, Henry Willard wrote:
> With one special exception kexec is not supported on systems
> that use spin-table as the cpu enablement method instead of PSCI.
> The spin-table implementation lacks cpu_die() and several other
> methods needed by the hotplug framework used by kexec on Arm64.
>
> Some embedded systems may not have a need for the Arm Trusted
> Firmware, or they may lack it during early bring-up. Some of
> these may have a more primitive version of u-boot that uses a
> special device from which to load the kernel. Kexec can be
> especially useful for testing new kernels in such an environment.
>
> What is needed to support kexec is some place for cpu_die to park
> the secondary CPUs outside the kernel while the primary copies
> the new kernel into place and starts it. One possibility is to
> use the control-code-page where arm64_relocate_new_kernel_size()
> executes, but that requires a complicated and racy dance to get
> the secondary CPUs from the control-code-page to the new
> kernel after it has been copied.
>
> The spin-table mechanism is setup before the Linux kernel
> is entered with details provided in the device tree. The
> "release-address" DT variable provides the address of a word the
> secondary CPUs are polling. The boot CPU will store the real address
> of secondary_holding_pen() at that address, and the secondary CPUs
> will branch to that address. secondary_holding_pen() is another
> loop where the secondary CPUs wait to be called up by the boot CPU.
>
> This patch uses that mechanism to implement cpu_die(). In modern
> versions of u-boot that implement spin-table, the address of the
> loop in protected memory can be derived from the "release-address"
> value. The patch validates the existence of the loop before
> proceeding. smp_spin_table_cpu_die() uses cpu_soft_restart() to
> branch to the loop with the MMU and caching turned off where the
> CPU waits until released by the new kernel. After that kexec
> reboot proceeds normally.
This isn't true for all spin-table implementations; for example this is
not safe with the boot-wrapper.
While, I'm not necessarily opposed to providing a mechanism to return a
CPU back to the spin-table, the presence of that mechanism needs to be
explicitly defined in the device tree (e.g. with a "cpu-return-addr"
property or similar), and we need to thoroughly document the contract
(e.g. what state the CPU is in when it is returned). We've generally
steered clear of this since it is much more complicated than it may
initially seem, and there is immense scope for error.
If we do choose to extend spin-table in this way, we'll also need to
enforce that each cpu has a unique cpu-release-address, or this is
unsound to begin with (since e.g. the kernel can't return CPUs that it
doesn't know are stuck in the holding pen). We will also need a
mechanism to reliably identify when the CPU has been successfully
returned.
I would very much like to avoid this if possible. U-Boot does have a
PSCI implementation that some platforms use; is it not possible to use
this?
If this is for early bringup, and you're using the first kernel as a
bootloader, I'd suggest that you boot that with "nosmp", such that the
first kernel doesn't touch the secondary CPUs at all.
> The special exception is the kdump capture kernel, which gets
> started even if the secondaries can't be stopped.
>
> Signed-off-by: Henry Willard <henry.willard@oracle.com>
> ---
> arch/arm64/kernel/smp_spin_table.c | 111 +++++++++++++++++++++++++++++++++++++
> 1 file changed, 111 insertions(+)
>
> diff --git a/arch/arm64/kernel/smp_spin_table.c b/arch/arm64/kernel/smp_spin_table.c
> index 7e1624ecab3c..35c7fa764476 100644
> --- a/arch/arm64/kernel/smp_spin_table.c
> +++ b/arch/arm64/kernel/smp_spin_table.c
> @@ -13,16 +13,27 @@
> #include <linux/mm.h>
>
> #include <asm/cacheflush.h>
> +#include <asm/daifflags.h>
> #include <asm/cpu_ops.h>
> #include <asm/cputype.h>
> #include <asm/io.h>
> #include <asm/smp_plat.h>
> +#include <asm/mmu_context.h>
> +#include <asm/kexec.h>
> +
> +#include "cpu-reset.h"
>
> extern void secondary_holding_pen(void);
> volatile unsigned long __section(".mmuoff.data.read")
> secondary_holding_pen_release = INVALID_HWID;
>
> static phys_addr_t cpu_release_addr[NR_CPUS];
> +static unsigned int spin_table_loop[4] = {
> + 0xd503205f, /* wfe */
> + 0x58000060, /* ldr x0, spin_table_cpu_release_addr */
> + 0xb4ffffc0, /* cbnz x0, 0b */
> + 0xd61f0000 /* br x0 */
> +};
>
> /*
> * Write secondary_holding_pen_release in a way that is guaranteed to be
> @@ -119,9 +130,109 @@ static int smp_spin_table_cpu_boot(unsigned int cpu)
> return 0;
> }
>
> +
> +/*
> + * There is a four instruction loop set aside in protected
> + * memory by u-boot where secondary CPUs wait for the kernel to
> + * start.
> + *
> + * 0: wfe
> + * ldr x0, spin_table_cpu_release_addr
> + * cbz x0, 0b
> + * br x0
> + * spin_table_cpu_release_addr:
> + * .quad 0
> + *
> + * The address of spin_table_cpu_release_addr is passed in the
> + * "release-address" property in the device table.
> + * smp_spin_table_cpu_prepare() stores the real address of
> + * secondary_holding_pen() where the secondary CPUs loop
> + * until they are released one at a time by smp_spin_table_cpu_boot().
> + * We reuse the spin-table loop by clearing spin_table_cpu_release_addr,
> + * and branching to the beginning of the loop via cpu_soft_restart(),
> + * which turns off the MMU and caching.
> + */
> +static void smp_spin_table_cpu_die(unsigned int cpu)
> +{
> + __le64 __iomem *release_addr;
> + unsigned int *spin_table_inst;
> + unsigned long spin_table_start;
> +
> + if (!cpu_release_addr[cpu])
> + goto spin;
> +
> + spin_table_start = (cpu_release_addr[cpu] - sizeof(spin_table_loop));
> +
> + /*
> + * The cpu-release-addr may or may not be inside the linear mapping.
> + * As ioremap_cache will either give us a new mapping or reuse the
> + * existing linear mapping, we can use it to cover both cases. In
> + * either case the memory will be MT_NORMAL.
> + */
> + release_addr = ioremap_cache(spin_table_start,
> + sizeof(*release_addr) +
> + sizeof(spin_table_loop));
> +
> + if (!release_addr)
> + goto spin;
> +
> + spin_table_inst = (unsigned int *)release_addr;
> + if (spin_table_inst[0] != spin_table_loop[0] ||
> + spin_table_inst[1] != spin_table_loop[1] ||
> + spin_table_inst[2] != spin_table_loop[2] ||
> + spin_table_inst[3] != spin_table_loop[3])
> + goto spin;
Please don't hard-code a specific sequence for this; if we *really* need
this, we should be given a cpu-return-addr explicitly, and we should
simply trust it.
> +
> + /*
> + * Clear the release address, so that we can use it again
> + */
> + writeq_relaxed(0, release_addr + 2);
> + dcache_clean_inval_poc((__force unsigned long)(release_addr + 2),
> + (__force unsigned long)(release_addr + 2) +
> + sizeof(*release_addr));
What is the `+ 2` for?
> +
> + iounmap(release_addr);
> +
> + local_daif_mask();
> + cpu_soft_restart(spin_table_start, 0, 0, 0);
> +
> + BUG(); /* Should never get here */
> +
> +spin:
> + cpu_park_loop();
> +
> +}
> +
> +static int smp_spin_table_cpu_kill(unsigned int cpu)
> +{
> + unsigned long start, end;
> +
> + start = jiffies;
> + end = start + msecs_to_jiffies(100);
> +
> + do {
> + if (!cpu_online(cpu)) {
> + pr_info("CPU%d killed\n", cpu);
> + return 0;
> + }
> + } while (time_before(jiffies, end));
> + pr_warn("CPU%d may not have shut down cleanly\n", cpu);
> + return -ETIMEDOUT;
> +
> +}
If we're going to extend this, we must add a mechanism to reliably
identify when the CPU has been returned successfully. We can't rely on
cpu_online(), becuase there's a window between the CPU marking itself as
offline and actually exiting the kernel.
> +
> +/* Nothing to do here */
> +static int smp_spin_table_cpu_disable(unsigned int cpu)
> +{
> + return 0;
> +}
For implementations where we cannot return the CPU, cpu_disable() *must*
fail.
Thanks,
Mark.
> +
> const struct cpu_operations smp_spin_table_ops = {
> .name = "spin-table",
> .cpu_init = smp_spin_table_cpu_init,
> .cpu_prepare = smp_spin_table_cpu_prepare,
> .cpu_boot = smp_spin_table_cpu_boot,
> + .cpu_die = smp_spin_table_cpu_die,
> + .cpu_kill = smp_spin_table_cpu_kill,
> + .cpu_disable = smp_spin_table_cpu_disable,
> };
> --
> 1.8.3.1
>
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH] arm64: kexec: add support for kexec with spin-table
2021-07-14 17:41 ` Henry Willard
@ 2021-07-14 22:24 ` kernel test robot
-1 siblings, 0 replies; 14+ messages in thread
From: kernel test robot @ 2021-07-14 22:24 UTC (permalink / raw)
To: Henry Willard, catalin.marinas, will, mark.rutland, tabba,
keescook, ardb, samitolvanen, joe, nixiaoming, linux-arm-kernel
Cc: clang-built-linux, kbuild-all
[-- Attachment #1: Type: text/plain, Size: 2927 bytes --]
Hi Henry,
Thank you for the patch! Yet something to improve:
[auto build test ERROR on arm64/for-next/core]
[also build test ERROR on v5.14-rc1 next-20210714]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]
url: https://github.com/0day-ci/linux/commits/Henry-Willard/arm64-kexec-add-support-for-kexec-with-spin-table/20210715-014204
base: https://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git for-next/core
config: arm64-randconfig-r025-20210714 (attached as .config)
compiler: clang version 13.0.0 (https://github.com/llvm/llvm-project 8d69635ed9ecf36fd0ca85906bfde17949671cbe)
reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
chmod +x ~/bin/make.cross
# install arm64 cross compiling tool for clang build
# apt-get install binutils-aarch64-linux-gnu
# https://github.com/0day-ci/linux/commit/3dd4112e1b67732182a5e12891867db4e139980c
git remote add linux-review https://github.com/0day-ci/linux
git fetch --no-tags linux-review Henry-Willard/arm64-kexec-add-support-for-kexec-with-spin-table/20210715-014204
git checkout 3dd4112e1b67732182a5e12891867db4e139980c
# save the attached .config to linux build tree
mkdir build_dir
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=clang make.cross O=build_dir ARCH=arm64 SHELL=/bin/bash arch/arm64/kernel/
If you fix the issue, kindly add following tag as appropriate
Reported-by: kernel test robot <lkp@intel.com>
All errors (new ones prefixed by >>):
>> arch/arm64/kernel/smp_spin_table.c:235:3: error: field designator 'cpu_die' does not refer to any field in type 'const struct cpu_operations'
.cpu_die = smp_spin_table_cpu_die,
^
>> arch/arm64/kernel/smp_spin_table.c:236:3: error: field designator 'cpu_kill' does not refer to any field in type 'const struct cpu_operations'
.cpu_kill = smp_spin_table_cpu_kill,
^
>> arch/arm64/kernel/smp_spin_table.c:237:3: error: field designator 'cpu_disable' does not refer to any field in type 'const struct cpu_operations'
.cpu_disable = smp_spin_table_cpu_disable,
^
3 errors generated.
vim +235 arch/arm64/kernel/smp_spin_table.c
229
230 const struct cpu_operations smp_spin_table_ops = {
231 .name = "spin-table",
232 .cpu_init = smp_spin_table_cpu_init,
233 .cpu_prepare = smp_spin_table_cpu_prepare,
234 .cpu_boot = smp_spin_table_cpu_boot,
> 235 .cpu_die = smp_spin_table_cpu_die,
> 236 .cpu_kill = smp_spin_table_cpu_kill,
> 237 .cpu_disable = smp_spin_table_cpu_disable,
---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all@lists.01.org
[-- Attachment #2: .config.gz --]
[-- Type: application/gzip, Size: 29318 bytes --]
[-- Attachment #3: Type: text/plain, Size: 176 bytes --]
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH] arm64: kexec: add support for kexec with spin-table
@ 2021-07-14 22:24 ` kernel test robot
0 siblings, 0 replies; 14+ messages in thread
From: kernel test robot @ 2021-07-14 22:24 UTC (permalink / raw)
To: kbuild-all
[-- Attachment #1: Type: text/plain, Size: 2988 bytes --]
Hi Henry,
Thank you for the patch! Yet something to improve:
[auto build test ERROR on arm64/for-next/core]
[also build test ERROR on v5.14-rc1 next-20210714]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]
url: https://github.com/0day-ci/linux/commits/Henry-Willard/arm64-kexec-add-support-for-kexec-with-spin-table/20210715-014204
base: https://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git for-next/core
config: arm64-randconfig-r025-20210714 (attached as .config)
compiler: clang version 13.0.0 (https://github.com/llvm/llvm-project 8d69635ed9ecf36fd0ca85906bfde17949671cbe)
reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
chmod +x ~/bin/make.cross
# install arm64 cross compiling tool for clang build
# apt-get install binutils-aarch64-linux-gnu
# https://github.com/0day-ci/linux/commit/3dd4112e1b67732182a5e12891867db4e139980c
git remote add linux-review https://github.com/0day-ci/linux
git fetch --no-tags linux-review Henry-Willard/arm64-kexec-add-support-for-kexec-with-spin-table/20210715-014204
git checkout 3dd4112e1b67732182a5e12891867db4e139980c
# save the attached .config to linux build tree
mkdir build_dir
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=clang make.cross O=build_dir ARCH=arm64 SHELL=/bin/bash arch/arm64/kernel/
If you fix the issue, kindly add following tag as appropriate
Reported-by: kernel test robot <lkp@intel.com>
All errors (new ones prefixed by >>):
>> arch/arm64/kernel/smp_spin_table.c:235:3: error: field designator 'cpu_die' does not refer to any field in type 'const struct cpu_operations'
.cpu_die = smp_spin_table_cpu_die,
^
>> arch/arm64/kernel/smp_spin_table.c:236:3: error: field designator 'cpu_kill' does not refer to any field in type 'const struct cpu_operations'
.cpu_kill = smp_spin_table_cpu_kill,
^
>> arch/arm64/kernel/smp_spin_table.c:237:3: error: field designator 'cpu_disable' does not refer to any field in type 'const struct cpu_operations'
.cpu_disable = smp_spin_table_cpu_disable,
^
3 errors generated.
vim +235 arch/arm64/kernel/smp_spin_table.c
229
230 const struct cpu_operations smp_spin_table_ops = {
231 .name = "spin-table",
232 .cpu_init = smp_spin_table_cpu_init,
233 .cpu_prepare = smp_spin_table_cpu_prepare,
234 .cpu_boot = smp_spin_table_cpu_boot,
> 235 .cpu_die = smp_spin_table_cpu_die,
> 236 .cpu_kill = smp_spin_table_cpu_kill,
> 237 .cpu_disable = smp_spin_table_cpu_disable,
---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org
[-- Attachment #2: config.gz --]
[-- Type: application/gzip, Size: 29318 bytes --]
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH] arm64: kexec: add support for kexec with spin-table
2021-07-14 17:41 ` Henry Willard
@ 2021-07-14 22:29 ` kernel test robot
-1 siblings, 0 replies; 14+ messages in thread
From: kernel test robot @ 2021-07-14 22:29 UTC (permalink / raw)
To: Henry Willard, catalin.marinas, will, mark.rutland, tabba,
keescook, ardb, samitolvanen, joe, nixiaoming, linux-arm-kernel
Cc: kbuild-all
[-- Attachment #1: Type: text/plain, Size: 4551 bytes --]
Hi Henry,
Thank you for the patch! Perhaps something to improve:
[auto build test WARNING on arm64/for-next/core]
[also build test WARNING on v5.14-rc1 next-20210714]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]
url: https://github.com/0day-ci/linux/commits/Henry-Willard/arm64-kexec-add-support-for-kexec-with-spin-table/20210715-014204
base: https://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git for-next/core
config: arm64-randconfig-s031-20210714 (attached as .config)
compiler: aarch64-linux-gcc (GCC) 9.3.0
reproduce:
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
chmod +x ~/bin/make.cross
# apt-get install sparse
# sparse version: v0.6.3-341-g8af24329-dirty
# https://github.com/0day-ci/linux/commit/3dd4112e1b67732182a5e12891867db4e139980c
git remote add linux-review https://github.com/0day-ci/linux
git fetch --no-tags linux-review Henry-Willard/arm64-kexec-add-support-for-kexec-with-spin-table/20210715-014204
git checkout 3dd4112e1b67732182a5e12891867db4e139980c
# save the attached .config to linux build tree
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-9.3.0 make.cross C=1 CF='-fdiagnostic-prefix -D__CHECK_ENDIAN__' O=build_dir ARCH=arm64 SHELL=/bin/bash arch/arm64/kernel/
If you fix the issue, kindly add following tag as appropriate
Reported-by: kernel test robot <lkp@intel.com>
sparse warnings: (new ones prefixed by >>)
>> arch/arm64/kernel/smp_spin_table.c:179:28: sparse: sparse: cast removes address space '__iomem' of expression
vim +/__iomem +179 arch/arm64/kernel/smp_spin_table.c
132
133
134 /*
135 * There is a four instruction loop set aside in protected
136 * memory by u-boot where secondary CPUs wait for the kernel to
137 * start.
138 *
139 * 0: wfe
140 * ldr x0, spin_table_cpu_release_addr
141 * cbz x0, 0b
142 * br x0
143 * spin_table_cpu_release_addr:
144 * .quad 0
145 *
146 * The address of spin_table_cpu_release_addr is passed in the
147 * "release-address" property in the device table.
148 * smp_spin_table_cpu_prepare() stores the real address of
149 * secondary_holding_pen() where the secondary CPUs loop
150 * until they are released one at a time by smp_spin_table_cpu_boot().
151 * We reuse the spin-table loop by clearing spin_table_cpu_release_addr,
152 * and branching to the beginning of the loop via cpu_soft_restart(),
153 * which turns off the MMU and caching.
154 */
155 static void smp_spin_table_cpu_die(unsigned int cpu)
156 {
157 __le64 __iomem *release_addr;
158 unsigned int *spin_table_inst;
159 unsigned long spin_table_start;
160
161 if (!cpu_release_addr[cpu])
162 goto spin;
163
164 spin_table_start = (cpu_release_addr[cpu] - sizeof(spin_table_loop));
165
166 /*
167 * The cpu-release-addr may or may not be inside the linear mapping.
168 * As ioremap_cache will either give us a new mapping or reuse the
169 * existing linear mapping, we can use it to cover both cases. In
170 * either case the memory will be MT_NORMAL.
171 */
172 release_addr = ioremap_cache(spin_table_start,
173 sizeof(*release_addr) +
174 sizeof(spin_table_loop));
175
176 if (!release_addr)
177 goto spin;
178
> 179 spin_table_inst = (unsigned int *)release_addr;
180 if (spin_table_inst[0] != spin_table_loop[0] ||
181 spin_table_inst[1] != spin_table_loop[1] ||
182 spin_table_inst[2] != spin_table_loop[2] ||
183 spin_table_inst[3] != spin_table_loop[3])
184 goto spin;
185
186 /*
187 * Clear the release address, so that we can use it again
188 */
189 writeq_relaxed(0, release_addr + 2);
190 dcache_clean_inval_poc((__force unsigned long)(release_addr + 2),
191 (__force unsigned long)(release_addr + 2) +
192 sizeof(*release_addr));
193
194 iounmap(release_addr);
195
196 local_daif_mask();
197 cpu_soft_restart(spin_table_start, 0, 0, 0);
198
199 BUG(); /* Should never get here */
200
201 spin:
202 cpu_park_loop();
203
---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all@lists.01.org
[-- Attachment #2: .config.gz --]
[-- Type: application/gzip, Size: 32250 bytes --]
[-- Attachment #3: Type: text/plain, Size: 176 bytes --]
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH] arm64: kexec: add support for kexec with spin-table
@ 2021-07-14 22:29 ` kernel test robot
0 siblings, 0 replies; 14+ messages in thread
From: kernel test robot @ 2021-07-14 22:29 UTC (permalink / raw)
To: kbuild-all
[-- Attachment #1: Type: text/plain, Size: 4664 bytes --]
Hi Henry,
Thank you for the patch! Perhaps something to improve:
[auto build test WARNING on arm64/for-next/core]
[also build test WARNING on v5.14-rc1 next-20210714]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]
url: https://github.com/0day-ci/linux/commits/Henry-Willard/arm64-kexec-add-support-for-kexec-with-spin-table/20210715-014204
base: https://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git for-next/core
config: arm64-randconfig-s031-20210714 (attached as .config)
compiler: aarch64-linux-gcc (GCC) 9.3.0
reproduce:
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
chmod +x ~/bin/make.cross
# apt-get install sparse
# sparse version: v0.6.3-341-g8af24329-dirty
# https://github.com/0day-ci/linux/commit/3dd4112e1b67732182a5e12891867db4e139980c
git remote add linux-review https://github.com/0day-ci/linux
git fetch --no-tags linux-review Henry-Willard/arm64-kexec-add-support-for-kexec-with-spin-table/20210715-014204
git checkout 3dd4112e1b67732182a5e12891867db4e139980c
# save the attached .config to linux build tree
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-9.3.0 make.cross C=1 CF='-fdiagnostic-prefix -D__CHECK_ENDIAN__' O=build_dir ARCH=arm64 SHELL=/bin/bash arch/arm64/kernel/
If you fix the issue, kindly add following tag as appropriate
Reported-by: kernel test robot <lkp@intel.com>
sparse warnings: (new ones prefixed by >>)
>> arch/arm64/kernel/smp_spin_table.c:179:28: sparse: sparse: cast removes address space '__iomem' of expression
vim +/__iomem +179 arch/arm64/kernel/smp_spin_table.c
132
133
134 /*
135 * There is a four instruction loop set aside in protected
136 * memory by u-boot where secondary CPUs wait for the kernel to
137 * start.
138 *
139 * 0: wfe
140 * ldr x0, spin_table_cpu_release_addr
141 * cbz x0, 0b
142 * br x0
143 * spin_table_cpu_release_addr:
144 * .quad 0
145 *
146 * The address of spin_table_cpu_release_addr is passed in the
147 * "release-address" property in the device table.
148 * smp_spin_table_cpu_prepare() stores the real address of
149 * secondary_holding_pen() where the secondary CPUs loop
150 * until they are released one at a time by smp_spin_table_cpu_boot().
151 * We reuse the spin-table loop by clearing spin_table_cpu_release_addr,
152 * and branching to the beginning of the loop via cpu_soft_restart(),
153 * which turns off the MMU and caching.
154 */
155 static void smp_spin_table_cpu_die(unsigned int cpu)
156 {
157 __le64 __iomem *release_addr;
158 unsigned int *spin_table_inst;
159 unsigned long spin_table_start;
160
161 if (!cpu_release_addr[cpu])
162 goto spin;
163
164 spin_table_start = (cpu_release_addr[cpu] - sizeof(spin_table_loop));
165
166 /*
167 * The cpu-release-addr may or may not be inside the linear mapping.
168 * As ioremap_cache will either give us a new mapping or reuse the
169 * existing linear mapping, we can use it to cover both cases. In
170 * either case the memory will be MT_NORMAL.
171 */
172 release_addr = ioremap_cache(spin_table_start,
173 sizeof(*release_addr) +
174 sizeof(spin_table_loop));
175
176 if (!release_addr)
177 goto spin;
178
> 179 spin_table_inst = (unsigned int *)release_addr;
180 if (spin_table_inst[0] != spin_table_loop[0] ||
181 spin_table_inst[1] != spin_table_loop[1] ||
182 spin_table_inst[2] != spin_table_loop[2] ||
183 spin_table_inst[3] != spin_table_loop[3])
184 goto spin;
185
186 /*
187 * Clear the release address, so that we can use it again
188 */
189 writeq_relaxed(0, release_addr + 2);
190 dcache_clean_inval_poc((__force unsigned long)(release_addr + 2),
191 (__force unsigned long)(release_addr + 2) +
192 sizeof(*release_addr));
193
194 iounmap(release_addr);
195
196 local_daif_mask();
197 cpu_soft_restart(spin_table_start, 0, 0, 0);
198
199 BUG(); /* Should never get here */
200
201 spin:
202 cpu_park_loop();
203
---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org
[-- Attachment #2: config.gz --]
[-- Type: application/gzip, Size: 32250 bytes --]
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH] arm64: kexec: add support for kexec with spin-table
2021-07-14 18:47 ` Mark Rutland
@ 2021-07-15 0:08 ` Henry Willard
-1 siblings, 0 replies; 14+ messages in thread
From: Henry Willard @ 2021-07-15 0:08 UTC (permalink / raw)
To: Mark Rutland
Cc: catalin.marinas, will, tabba, keescook, ardb, samitolvanen, joe,
nixiaoming, linux-arm-kernel, linux-kernel
Hi, Mark,
Thanks for reviewing this. I am not in a position to go into too much detail about the particular device, but the u-boot we are using is the u-boot we have to use, at least for now. We would have preferred to have PSCI, but that option is not available. Modifying u-boot is not an option.
It is possible to do this without relying on the spin-table loop. I implemented such a version using the kexec code control page before I got my hands on the device actually using spin-table. That implementaiton needed changes in a lot of places, because the secondary CPUs had to leave the code control page before the boot CPU enters the new kernel. Reusing the spin-table loop simplified things quite a bit.
This has been useful to us, so we thought we would pass it along to see if it is useful to anyone else in the same situation.
> On Jul 14, 2021, at 11:47 AM, Mark Rutland <mark.rutland@arm.com> wrote:
>
> Hi Henry,
>
> On Wed, Jul 14, 2021 at 10:41:13AM -0700, Henry Willard wrote:
>> With one special exception kexec is not supported on systems
>> that use spin-table as the cpu enablement method instead of PSCI.
>> The spin-table implementation lacks cpu_die() and several other
>> methods needed by the hotplug framework used by kexec on Arm64.
>>
>> Some embedded systems may not have a need for the Arm Trusted
>> Firmware, or they may lack it during early bring-up. Some of
>> these may have a more primitive version of u-boot that uses a
>> special device from which to load the kernel. Kexec can be
>> especially useful for testing new kernels in such an environment.
>>
>> What is needed to support kexec is some place for cpu_die to park
>> the secondary CPUs outside the kernel while the primary copies
>> the new kernel into place and starts it. One possibility is to
>> use the control-code-page where arm64_relocate_new_kernel_size()
>> executes, but that requires a complicated and racy dance to get
>> the secondary CPUs from the control-code-page to the new
>> kernel after it has been copied.
>>
>> The spin-table mechanism is setup before the Linux kernel
>> is entered with details provided in the device tree. The
>> "release-address" DT variable provides the address of a word the
>> secondary CPUs are polling. The boot CPU will store the real address
>> of secondary_holding_pen() at that address, and the secondary CPUs
>> will branch to that address. secondary_holding_pen() is another
>> loop where the secondary CPUs wait to be called up by the boot CPU.
>>
>> This patch uses that mechanism to implement cpu_die(). In modern
>> versions of u-boot that implement spin-table, the address of the
>> loop in protected memory can be derived from the "release-address"
>> value. The patch validates the existence of the loop before
>> proceeding. smp_spin_table_cpu_die() uses cpu_soft_restart() to
>> branch to the loop with the MMU and caching turned off where the
>> CPU waits until released by the new kernel. After that kexec
>> reboot proceeds normally.
>
> This isn't true for all spin-table implementations; for example this is
> not safe with the boot-wrapper.
>
> While, I'm not necessarily opposed to providing a mechanism to return a
> CPU back to the spin-table, the presence of that mechanism needs to be
> explicitly defined in the device tree (e.g. with a "cpu-return-addr"
> property or similar), and we need to thoroughly document the contract
> (e.g. what state the CPU is in when it is returned). We've generally
> steered clear of this since it is much more complicated than it may
> initially seem, and there is immense scope for error.
>
> If we do choose to extend spin-table in this way, we'll also need to
> enforce that each cpu has a unique cpu-release-address, or this is
> unsound to begin with (since e.g. the kernel can't return CPUs that it
> doesn't know are stuck in the holding pen). We will also need a
> mechanism to reliably identify when the CPU has been successfully
> returned.
>
> I would very much like to avoid this if possible. U-Boot does have a
> PSCI implementation that some platforms use; is it not possible to use
> this?
Unfortunately, no. If we had that we would never have bothered with this.
>
> If this is for early bringup, and you're using the first kernel as a
> bootloader, I'd suggest that you boot that with "nosmp", such that the
> first kernel doesn't touch the secondary CPUs at all.
The particular case that spawned this is past that. There are a number of reasons why we need to be able to kexec a new kernel. Being able to bypass the kernel installation process, which is a little more complicated than normal, to test a new kernels is an added benefit.
>
>> The special exception is the kdump capture kernel, which gets
>> started even if the secondaries can't be stopped.
>>
>> Signed-off-by: Henry Willard <henry.willard@oracle.com>
>> ---
>> arch/arm64/kernel/smp_spin_table.c | 111 +++++++++++++++++++++++++++++++++++++
>> 1 file changed, 111 insertions(+)
>>
>> diff --git a/arch/arm64/kernel/smp_spin_table.c b/arch/arm64/kernel/smp_spin_table.c
>> index 7e1624ecab3c..35c7fa764476 100644
>> --- a/arch/arm64/kernel/smp_spin_table.c
>> +++ b/arch/arm64/kernel/smp_spin_table.c
>> @@ -13,16 +13,27 @@
>> #include <linux/mm.h>
>>
>> #include <asm/cacheflush.h>
>> +#include <asm/daifflags.h>
>> #include <asm/cpu_ops.h>
>> #include <asm/cputype.h>
>> #include <asm/io.h>
>> #include <asm/smp_plat.h>
>> +#include <asm/mmu_context.h>
>> +#include <asm/kexec.h>
>> +
>> +#include "cpu-reset.h"
>>
>> extern void secondary_holding_pen(void);
>> volatile unsigned long __section(".mmuoff.data.read")
>> secondary_holding_pen_release = INVALID_HWID;
>>
>> static phys_addr_t cpu_release_addr[NR_CPUS];
>> +static unsigned int spin_table_loop[4] = {
>> + 0xd503205f, /* wfe */
>> + 0x58000060, /* ldr x0, spin_table_cpu_release_addr */
>> + 0xb4ffffc0, /* cbnz x0, 0b */
>> + 0xd61f0000 /* br x0 */
>> +};
>>
>> /*
>> * Write secondary_holding_pen_release in a way that is guaranteed to be
>> @@ -119,9 +130,109 @@ static int smp_spin_table_cpu_boot(unsigned int cpu)
>> return 0;
>> }
>>
>> +
>> +/*
>> + * There is a four instruction loop set aside in protected
>> + * memory by u-boot where secondary CPUs wait for the kernel to
>> + * start.
>> + *
>> + * 0: wfe
>> + * ldr x0, spin_table_cpu_release_addr
>> + * cbz x0, 0b
>> + * br x0
>> + * spin_table_cpu_release_addr:
>> + * .quad 0
>> + *
>> + * The address of spin_table_cpu_release_addr is passed in the
>> + * "release-address" property in the device table.
>> + * smp_spin_table_cpu_prepare() stores the real address of
>> + * secondary_holding_pen() where the secondary CPUs loop
>> + * until they are released one at a time by smp_spin_table_cpu_boot().
>> + * We reuse the spin-table loop by clearing spin_table_cpu_release_addr,
>> + * and branching to the beginning of the loop via cpu_soft_restart(),
>> + * which turns off the MMU and caching.
>> + */
>> +static void smp_spin_table_cpu_die(unsigned int cpu)
>> +{
>> + __le64 __iomem *release_addr;
>> + unsigned int *spin_table_inst;
>> + unsigned long spin_table_start;
>> +
>> + if (!cpu_release_addr[cpu])
>> + goto spin;
>> +
>> + spin_table_start = (cpu_release_addr[cpu] - sizeof(spin_table_loop));
>> +
>> + /*
>> + * The cpu-release-addr may or may not be inside the linear mapping.
>> + * As ioremap_cache will either give us a new mapping or reuse the
>> + * existing linear mapping, we can use it to cover both cases. In
>> + * either case the memory will be MT_NORMAL.
>> + */
>> + release_addr = ioremap_cache(spin_table_start,
>> + sizeof(*release_addr) +
>> + sizeof(spin_table_loop));
>> +
>> + if (!release_addr)
>> + goto spin;
>> +
>> + spin_table_inst = (unsigned int *)release_addr;
>> + if (spin_table_inst[0] != spin_table_loop[0] ||
>> + spin_table_inst[1] != spin_table_loop[1] ||
>> + spin_table_inst[2] != spin_table_loop[2] ||
>> + spin_table_inst[3] != spin_table_loop[3])
>> + goto spin;
>
> Please don't hard-code a specific sequence for this; if we *really* need
> this, we should be given a cpu-return-addr explicitly, and we should
> simply trust it.
That would require changes to u-boot. The purpose is to detect if we get a new version of u-boot with a different loop. Seems remote since this particular loop has been this way for quite some time, and it works well.
>
>> +
>> + /*
>> + * Clear the release address, so that we can use it again
>> + */
>> + writeq_relaxed(0, release_addr + 2);
>> + dcache_clean_inval_poc((__force unsigned long)(release_addr + 2),
>> + (__force unsigned long)(release_addr + 2) +
>> + sizeof(*release_addr));
>
> What is the `+ 2` for?
Yeah, I could have been clearer. The spin_table_cpu_release_addr variable sits at +0x10 past the spin-table loop.
>
>> +
>> + iounmap(release_addr);
>> +
>> + local_daif_mask();
>> + cpu_soft_restart(spin_table_start, 0, 0, 0);
>> +
>> + BUG(); /* Should never get here */
>> +
>> +spin:
>> + cpu_park_loop();
>> +
>> +}
>> +
>> +static int smp_spin_table_cpu_kill(unsigned int cpu)
>> +{
>> + unsigned long start, end;
>> +
>> + start = jiffies;
>> + end = start + msecs_to_jiffies(100);
>> +
>> + do {
>> + if (!cpu_online(cpu)) {
>> + pr_info("CPU%d killed\n", cpu);
>> + return 0;
>> + }
>> + } while (time_before(jiffies, end));
>> + pr_warn("CPU%d may not have shut down cleanly\n", cpu);
>> + return -ETIMEDOUT;
>> +
>> +}
>
> If we're going to extend this, we must add a mechanism to reliably
> identify when the CPU has been returned successfully. We can't rely on
> cpu_online(), becuase there's a window between the CPU marking itself as
> offline and actually exiting the kernel.
>
>> +
>> +/* Nothing to do here */
>> +static int smp_spin_table_cpu_disable(unsigned int cpu)
>> +{
>> + return 0;
>> +}
>
> For implementations where we cannot return the CPU, cpu_disable() *must*
> fail.
>
> Thanks,
> Mark.
Thanks for taking the time to review this.
Henry
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH] arm64: kexec: add support for kexec with spin-table
@ 2021-07-15 0:08 ` Henry Willard
0 siblings, 0 replies; 14+ messages in thread
From: Henry Willard @ 2021-07-15 0:08 UTC (permalink / raw)
To: Mark Rutland
Cc: catalin.marinas, will, tabba, keescook, ardb, samitolvanen, joe,
nixiaoming, linux-arm-kernel, linux-kernel
Hi, Mark,
Thanks for reviewing this. I am not in a position to go into too much detail about the particular device, but the u-boot we are using is the u-boot we have to use, at least for now. We would have preferred to have PSCI, but that option is not available. Modifying u-boot is not an option.
It is possible to do this without relying on the spin-table loop. I implemented such a version using the kexec code control page before I got my hands on the device actually using spin-table. That implementaiton needed changes in a lot of places, because the secondary CPUs had to leave the code control page before the boot CPU enters the new kernel. Reusing the spin-table loop simplified things quite a bit.
This has been useful to us, so we thought we would pass it along to see if it is useful to anyone else in the same situation.
> On Jul 14, 2021, at 11:47 AM, Mark Rutland <mark.rutland@arm.com> wrote:
>
> Hi Henry,
>
> On Wed, Jul 14, 2021 at 10:41:13AM -0700, Henry Willard wrote:
>> With one special exception kexec is not supported on systems
>> that use spin-table as the cpu enablement method instead of PSCI.
>> The spin-table implementation lacks cpu_die() and several other
>> methods needed by the hotplug framework used by kexec on Arm64.
>>
>> Some embedded systems may not have a need for the Arm Trusted
>> Firmware, or they may lack it during early bring-up. Some of
>> these may have a more primitive version of u-boot that uses a
>> special device from which to load the kernel. Kexec can be
>> especially useful for testing new kernels in such an environment.
>>
>> What is needed to support kexec is some place for cpu_die to park
>> the secondary CPUs outside the kernel while the primary copies
>> the new kernel into place and starts it. One possibility is to
>> use the control-code-page where arm64_relocate_new_kernel_size()
>> executes, but that requires a complicated and racy dance to get
>> the secondary CPUs from the control-code-page to the new
>> kernel after it has been copied.
>>
>> The spin-table mechanism is setup before the Linux kernel
>> is entered with details provided in the device tree. The
>> "release-address" DT variable provides the address of a word the
>> secondary CPUs are polling. The boot CPU will store the real address
>> of secondary_holding_pen() at that address, and the secondary CPUs
>> will branch to that address. secondary_holding_pen() is another
>> loop where the secondary CPUs wait to be called up by the boot CPU.
>>
>> This patch uses that mechanism to implement cpu_die(). In modern
>> versions of u-boot that implement spin-table, the address of the
>> loop in protected memory can be derived from the "release-address"
>> value. The patch validates the existence of the loop before
>> proceeding. smp_spin_table_cpu_die() uses cpu_soft_restart() to
>> branch to the loop with the MMU and caching turned off where the
>> CPU waits until released by the new kernel. After that kexec
>> reboot proceeds normally.
>
> This isn't true for all spin-table implementations; for example this is
> not safe with the boot-wrapper.
>
> While, I'm not necessarily opposed to providing a mechanism to return a
> CPU back to the spin-table, the presence of that mechanism needs to be
> explicitly defined in the device tree (e.g. with a "cpu-return-addr"
> property or similar), and we need to thoroughly document the contract
> (e.g. what state the CPU is in when it is returned). We've generally
> steered clear of this since it is much more complicated than it may
> initially seem, and there is immense scope for error.
>
> If we do choose to extend spin-table in this way, we'll also need to
> enforce that each cpu has a unique cpu-release-address, or this is
> unsound to begin with (since e.g. the kernel can't return CPUs that it
> doesn't know are stuck in the holding pen). We will also need a
> mechanism to reliably identify when the CPU has been successfully
> returned.
>
> I would very much like to avoid this if possible. U-Boot does have a
> PSCI implementation that some platforms use; is it not possible to use
> this?
Unfortunately, no. If we had that we would never have bothered with this.
>
> If this is for early bringup, and you're using the first kernel as a
> bootloader, I'd suggest that you boot that with "nosmp", such that the
> first kernel doesn't touch the secondary CPUs at all.
The particular case that spawned this is past that. There are a number of reasons why we need to be able to kexec a new kernel. Being able to bypass the kernel installation process, which is a little more complicated than normal, to test a new kernels is an added benefit.
>
>> The special exception is the kdump capture kernel, which gets
>> started even if the secondaries can't be stopped.
>>
>> Signed-off-by: Henry Willard <henry.willard@oracle.com>
>> ---
>> arch/arm64/kernel/smp_spin_table.c | 111 +++++++++++++++++++++++++++++++++++++
>> 1 file changed, 111 insertions(+)
>>
>> diff --git a/arch/arm64/kernel/smp_spin_table.c b/arch/arm64/kernel/smp_spin_table.c
>> index 7e1624ecab3c..35c7fa764476 100644
>> --- a/arch/arm64/kernel/smp_spin_table.c
>> +++ b/arch/arm64/kernel/smp_spin_table.c
>> @@ -13,16 +13,27 @@
>> #include <linux/mm.h>
>>
>> #include <asm/cacheflush.h>
>> +#include <asm/daifflags.h>
>> #include <asm/cpu_ops.h>
>> #include <asm/cputype.h>
>> #include <asm/io.h>
>> #include <asm/smp_plat.h>
>> +#include <asm/mmu_context.h>
>> +#include <asm/kexec.h>
>> +
>> +#include "cpu-reset.h"
>>
>> extern void secondary_holding_pen(void);
>> volatile unsigned long __section(".mmuoff.data.read")
>> secondary_holding_pen_release = INVALID_HWID;
>>
>> static phys_addr_t cpu_release_addr[NR_CPUS];
>> +static unsigned int spin_table_loop[4] = {
>> + 0xd503205f, /* wfe */
>> + 0x58000060, /* ldr x0, spin_table_cpu_release_addr */
>> + 0xb4ffffc0, /* cbnz x0, 0b */
>> + 0xd61f0000 /* br x0 */
>> +};
>>
>> /*
>> * Write secondary_holding_pen_release in a way that is guaranteed to be
>> @@ -119,9 +130,109 @@ static int smp_spin_table_cpu_boot(unsigned int cpu)
>> return 0;
>> }
>>
>> +
>> +/*
>> + * There is a four instruction loop set aside in protected
>> + * memory by u-boot where secondary CPUs wait for the kernel to
>> + * start.
>> + *
>> + * 0: wfe
>> + * ldr x0, spin_table_cpu_release_addr
>> + * cbz x0, 0b
>> + * br x0
>> + * spin_table_cpu_release_addr:
>> + * .quad 0
>> + *
>> + * The address of spin_table_cpu_release_addr is passed in the
>> + * "release-address" property in the device table.
>> + * smp_spin_table_cpu_prepare() stores the real address of
>> + * secondary_holding_pen() where the secondary CPUs loop
>> + * until they are released one at a time by smp_spin_table_cpu_boot().
>> + * We reuse the spin-table loop by clearing spin_table_cpu_release_addr,
>> + * and branching to the beginning of the loop via cpu_soft_restart(),
>> + * which turns off the MMU and caching.
>> + */
>> +static void smp_spin_table_cpu_die(unsigned int cpu)
>> +{
>> + __le64 __iomem *release_addr;
>> + unsigned int *spin_table_inst;
>> + unsigned long spin_table_start;
>> +
>> + if (!cpu_release_addr[cpu])
>> + goto spin;
>> +
>> + spin_table_start = (cpu_release_addr[cpu] - sizeof(spin_table_loop));
>> +
>> + /*
>> + * The cpu-release-addr may or may not be inside the linear mapping.
>> + * As ioremap_cache will either give us a new mapping or reuse the
>> + * existing linear mapping, we can use it to cover both cases. In
>> + * either case the memory will be MT_NORMAL.
>> + */
>> + release_addr = ioremap_cache(spin_table_start,
>> + sizeof(*release_addr) +
>> + sizeof(spin_table_loop));
>> +
>> + if (!release_addr)
>> + goto spin;
>> +
>> + spin_table_inst = (unsigned int *)release_addr;
>> + if (spin_table_inst[0] != spin_table_loop[0] ||
>> + spin_table_inst[1] != spin_table_loop[1] ||
>> + spin_table_inst[2] != spin_table_loop[2] ||
>> + spin_table_inst[3] != spin_table_loop[3])
>> + goto spin;
>
> Please don't hard-code a specific sequence for this; if we *really* need
> this, we should be given a cpu-return-addr explicitly, and we should
> simply trust it.
That would require changes to u-boot. The purpose is to detect if we get a new version of u-boot with a different loop. Seems remote since this particular loop has been this way for quite some time, and it works well.
>
>> +
>> + /*
>> + * Clear the release address, so that we can use it again
>> + */
>> + writeq_relaxed(0, release_addr + 2);
>> + dcache_clean_inval_poc((__force unsigned long)(release_addr + 2),
>> + (__force unsigned long)(release_addr + 2) +
>> + sizeof(*release_addr));
>
> What is the `+ 2` for?
Yeah, I could have been clearer. The spin_table_cpu_release_addr variable sits at +0x10 past the spin-table loop.
>
>> +
>> + iounmap(release_addr);
>> +
>> + local_daif_mask();
>> + cpu_soft_restart(spin_table_start, 0, 0, 0);
>> +
>> + BUG(); /* Should never get here */
>> +
>> +spin:
>> + cpu_park_loop();
>> +
>> +}
>> +
>> +static int smp_spin_table_cpu_kill(unsigned int cpu)
>> +{
>> + unsigned long start, end;
>> +
>> + start = jiffies;
>> + end = start + msecs_to_jiffies(100);
>> +
>> + do {
>> + if (!cpu_online(cpu)) {
>> + pr_info("CPU%d killed\n", cpu);
>> + return 0;
>> + }
>> + } while (time_before(jiffies, end));
>> + pr_warn("CPU%d may not have shut down cleanly\n", cpu);
>> + return -ETIMEDOUT;
>> +
>> +}
>
> If we're going to extend this, we must add a mechanism to reliably
> identify when the CPU has been returned successfully. We can't rely on
> cpu_online(), becuase there's a window between the CPU marking itself as
> offline and actually exiting the kernel.
>
>> +
>> +/* Nothing to do here */
>> +static int smp_spin_table_cpu_disable(unsigned int cpu)
>> +{
>> + return 0;
>> +}
>
> For implementations where we cannot return the CPU, cpu_disable() *must*
> fail.
>
> Thanks,
> Mark.
Thanks for taking the time to review this.
Henry
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH] arm64: kexec: add support for kexec with spin-table
2021-07-14 17:41 ` Henry Willard
@ 2021-07-15 6:54 ` kernel test robot
-1 siblings, 0 replies; 14+ messages in thread
From: kernel test robot @ 2021-07-15 6:54 UTC (permalink / raw)
To: Henry Willard, catalin.marinas, will, mark.rutland, tabba,
keescook, ardb, samitolvanen, joe, nixiaoming, linux-arm-kernel
Cc: kbuild-all
[-- Attachment #1: Type: text/plain, Size: 3413 bytes --]
Hi Henry,
Thank you for the patch! Yet something to improve:
[auto build test ERROR on arm64/for-next/core]
[also build test ERROR on v5.14-rc1 next-20210715]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]
url: https://github.com/0day-ci/linux/commits/Henry-Willard/arm64-kexec-add-support-for-kexec-with-spin-table/20210715-014204
base: https://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git for-next/core
config: arm64-randconfig-r025-20210715 (attached as .config)
compiler: aarch64-linux-gcc (GCC) 10.3.0
reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
chmod +x ~/bin/make.cross
# https://github.com/0day-ci/linux/commit/3dd4112e1b67732182a5e12891867db4e139980c
git remote add linux-review https://github.com/0day-ci/linux
git fetch --no-tags linux-review Henry-Willard/arm64-kexec-add-support-for-kexec-with-spin-table/20210715-014204
git checkout 3dd4112e1b67732182a5e12891867db4e139980c
# save the attached .config to linux build tree
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-10.3.0 make.cross ARCH=arm64
If you fix the issue, kindly add following tag as appropriate
Reported-by: kernel test robot <lkp@intel.com>
All errors (new ones prefixed by >>):
arch/arm64/kernel/smp_spin_table.c:235:3: error: 'const struct cpu_operations' has no member named 'cpu_die'
235 | .cpu_die = smp_spin_table_cpu_die,
| ^~~~~~~
arch/arm64/kernel/smp_spin_table.c:235:13: error: initialization of 'void (*)(void)' from incompatible pointer type 'void (*)(unsigned int)' [-Werror=incompatible-pointer-types]
235 | .cpu_die = smp_spin_table_cpu_die,
| ^~~~~~~~~~~~~~~~~~~~~~
arch/arm64/kernel/smp_spin_table.c:235:13: note: (near initialization for 'smp_spin_table_ops.cpu_postboot')
arch/arm64/kernel/smp_spin_table.c:236:3: error: 'const struct cpu_operations' has no member named 'cpu_kill'
236 | .cpu_kill = smp_spin_table_cpu_kill,
| ^~~~~~~~
arch/arm64/kernel/smp_spin_table.c:237:3: error: 'const struct cpu_operations' has no member named 'cpu_disable'
237 | .cpu_disable = smp_spin_table_cpu_disable,
| ^~~~~~~~~~~
>> arch/arm64/kernel/smp_spin_table.c:237:17: error: initialization of 'int (*)(long unsigned int)' from incompatible pointer type 'int (*)(unsigned int)' [-Werror=incompatible-pointer-types]
237 | .cpu_disable = smp_spin_table_cpu_disable,
| ^~~~~~~~~~~~~~~~~~~~~~~~~~
arch/arm64/kernel/smp_spin_table.c:237:17: note: (near initialization for 'smp_spin_table_ops.cpu_suspend')
cc1: some warnings being treated as errors
vim +237 arch/arm64/kernel/smp_spin_table.c
229
230 const struct cpu_operations smp_spin_table_ops = {
231 .name = "spin-table",
232 .cpu_init = smp_spin_table_cpu_init,
233 .cpu_prepare = smp_spin_table_cpu_prepare,
234 .cpu_boot = smp_spin_table_cpu_boot,
> 235 .cpu_die = smp_spin_table_cpu_die,
236 .cpu_kill = smp_spin_table_cpu_kill,
> 237 .cpu_disable = smp_spin_table_cpu_disable,
---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all@lists.01.org
[-- Attachment #2: .config.gz --]
[-- Type: application/gzip, Size: 37355 bytes --]
[-- Attachment #3: Type: text/plain, Size: 176 bytes --]
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH] arm64: kexec: add support for kexec with spin-table
@ 2021-07-15 6:54 ` kernel test robot
0 siblings, 0 replies; 14+ messages in thread
From: kernel test robot @ 2021-07-15 6:54 UTC (permalink / raw)
To: kbuild-all
[-- Attachment #1: Type: text/plain, Size: 3479 bytes --]
Hi Henry,
Thank you for the patch! Yet something to improve:
[auto build test ERROR on arm64/for-next/core]
[also build test ERROR on v5.14-rc1 next-20210715]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]
url: https://github.com/0day-ci/linux/commits/Henry-Willard/arm64-kexec-add-support-for-kexec-with-spin-table/20210715-014204
base: https://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git for-next/core
config: arm64-randconfig-r025-20210715 (attached as .config)
compiler: aarch64-linux-gcc (GCC) 10.3.0
reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
chmod +x ~/bin/make.cross
# https://github.com/0day-ci/linux/commit/3dd4112e1b67732182a5e12891867db4e139980c
git remote add linux-review https://github.com/0day-ci/linux
git fetch --no-tags linux-review Henry-Willard/arm64-kexec-add-support-for-kexec-with-spin-table/20210715-014204
git checkout 3dd4112e1b67732182a5e12891867db4e139980c
# save the attached .config to linux build tree
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-10.3.0 make.cross ARCH=arm64
If you fix the issue, kindly add following tag as appropriate
Reported-by: kernel test robot <lkp@intel.com>
All errors (new ones prefixed by >>):
arch/arm64/kernel/smp_spin_table.c:235:3: error: 'const struct cpu_operations' has no member named 'cpu_die'
235 | .cpu_die = smp_spin_table_cpu_die,
| ^~~~~~~
arch/arm64/kernel/smp_spin_table.c:235:13: error: initialization of 'void (*)(void)' from incompatible pointer type 'void (*)(unsigned int)' [-Werror=incompatible-pointer-types]
235 | .cpu_die = smp_spin_table_cpu_die,
| ^~~~~~~~~~~~~~~~~~~~~~
arch/arm64/kernel/smp_spin_table.c:235:13: note: (near initialization for 'smp_spin_table_ops.cpu_postboot')
arch/arm64/kernel/smp_spin_table.c:236:3: error: 'const struct cpu_operations' has no member named 'cpu_kill'
236 | .cpu_kill = smp_spin_table_cpu_kill,
| ^~~~~~~~
arch/arm64/kernel/smp_spin_table.c:237:3: error: 'const struct cpu_operations' has no member named 'cpu_disable'
237 | .cpu_disable = smp_spin_table_cpu_disable,
| ^~~~~~~~~~~
>> arch/arm64/kernel/smp_spin_table.c:237:17: error: initialization of 'int (*)(long unsigned int)' from incompatible pointer type 'int (*)(unsigned int)' [-Werror=incompatible-pointer-types]
237 | .cpu_disable = smp_spin_table_cpu_disable,
| ^~~~~~~~~~~~~~~~~~~~~~~~~~
arch/arm64/kernel/smp_spin_table.c:237:17: note: (near initialization for 'smp_spin_table_ops.cpu_suspend')
cc1: some warnings being treated as errors
vim +237 arch/arm64/kernel/smp_spin_table.c
229
230 const struct cpu_operations smp_spin_table_ops = {
231 .name = "spin-table",
232 .cpu_init = smp_spin_table_cpu_init,
233 .cpu_prepare = smp_spin_table_cpu_prepare,
234 .cpu_boot = smp_spin_table_cpu_boot,
> 235 .cpu_die = smp_spin_table_cpu_die,
236 .cpu_kill = smp_spin_table_cpu_kill,
> 237 .cpu_disable = smp_spin_table_cpu_disable,
---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org
[-- Attachment #2: config.gz --]
[-- Type: application/gzip, Size: 37355 bytes --]
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH] arm64: kexec: add support for kexec with spin-table
2021-07-14 17:41 ` Henry Willard
@ 2021-07-15 11:48 ` kernel test robot
-1 siblings, 0 replies; 14+ messages in thread
From: kernel test robot @ 2021-07-15 11:48 UTC (permalink / raw)
To: Henry Willard, catalin.marinas, will, mark.rutland, tabba,
keescook, ardb, samitolvanen, joe, nixiaoming, linux-arm-kernel
Cc: clang-built-linux, kbuild-all
[-- Attachment #1: Type: text/plain, Size: 2395 bytes --]
Hi Henry,
Thank you for the patch! Yet something to improve:
[auto build test ERROR on arm64/for-next/core]
[also build test ERROR on v5.14-rc1 next-20210715]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]
url: https://github.com/0day-ci/linux/commits/Henry-Willard/arm64-kexec-add-support-for-kexec-with-spin-table/20210715-014204
base: https://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git for-next/core
config: arm64-randconfig-r011-20210714 (attached as .config)
compiler: clang version 13.0.0 (https://github.com/llvm/llvm-project 8d69635ed9ecf36fd0ca85906bfde17949671cbe)
reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
chmod +x ~/bin/make.cross
# install arm64 cross compiling tool for clang build
# apt-get install binutils-aarch64-linux-gnu
# https://github.com/0day-ci/linux/commit/3dd4112e1b67732182a5e12891867db4e139980c
git remote add linux-review https://github.com/0day-ci/linux
git fetch --no-tags linux-review Henry-Willard/arm64-kexec-add-support-for-kexec-with-spin-table/20210715-014204
git checkout 3dd4112e1b67732182a5e12891867db4e139980c
# save the attached .config to linux build tree
mkdir build_dir
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=clang make.cross O=build_dir ARCH=arm64 SHELL=/bin/bash
If you fix the issue, kindly add following tag as appropriate
Reported-by: kernel test robot <lkp@intel.com>
All errors (new ones prefixed by >>):
aarch64-linux-gnu-ld: arch/arm64/kernel/smp_spin_table.o: in function `cpu_soft_restart':
>> smp_spin_table.c:(.text+0x524): undefined reference to `__cpu_soft_restart'
aarch64-linux-gnu-ld: arch/arm64/kernel/smp_spin_table.o: relocation R_AARCH64_ADR_PREL_PG_HI21 against symbol `__cpu_soft_restart' which may bind externally can not be used when making a shared object; recompile with -fPIC
smp_spin_table.c:(.text+0x524): dangerous relocation: unsupported relocation
>> aarch64-linux-gnu-ld: smp_spin_table.c:(.text+0x528): undefined reference to `__cpu_soft_restart'
---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all@lists.01.org
[-- Attachment #2: .config.gz --]
[-- Type: application/gzip, Size: 32528 bytes --]
[-- Attachment #3: Type: text/plain, Size: 176 bytes --]
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH] arm64: kexec: add support for kexec with spin-table
@ 2021-07-15 11:48 ` kernel test robot
0 siblings, 0 replies; 14+ messages in thread
From: kernel test robot @ 2021-07-15 11:48 UTC (permalink / raw)
To: kbuild-all
[-- Attachment #1: Type: text/plain, Size: 2438 bytes --]
Hi Henry,
Thank you for the patch! Yet something to improve:
[auto build test ERROR on arm64/for-next/core]
[also build test ERROR on v5.14-rc1 next-20210715]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]
url: https://github.com/0day-ci/linux/commits/Henry-Willard/arm64-kexec-add-support-for-kexec-with-spin-table/20210715-014204
base: https://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git for-next/core
config: arm64-randconfig-r011-20210714 (attached as .config)
compiler: clang version 13.0.0 (https://github.com/llvm/llvm-project 8d69635ed9ecf36fd0ca85906bfde17949671cbe)
reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
chmod +x ~/bin/make.cross
# install arm64 cross compiling tool for clang build
# apt-get install binutils-aarch64-linux-gnu
# https://github.com/0day-ci/linux/commit/3dd4112e1b67732182a5e12891867db4e139980c
git remote add linux-review https://github.com/0day-ci/linux
git fetch --no-tags linux-review Henry-Willard/arm64-kexec-add-support-for-kexec-with-spin-table/20210715-014204
git checkout 3dd4112e1b67732182a5e12891867db4e139980c
# save the attached .config to linux build tree
mkdir build_dir
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=clang make.cross O=build_dir ARCH=arm64 SHELL=/bin/bash
If you fix the issue, kindly add following tag as appropriate
Reported-by: kernel test robot <lkp@intel.com>
All errors (new ones prefixed by >>):
aarch64-linux-gnu-ld: arch/arm64/kernel/smp_spin_table.o: in function `cpu_soft_restart':
>> smp_spin_table.c:(.text+0x524): undefined reference to `__cpu_soft_restart'
aarch64-linux-gnu-ld: arch/arm64/kernel/smp_spin_table.o: relocation R_AARCH64_ADR_PREL_PG_HI21 against symbol `__cpu_soft_restart' which may bind externally can not be used when making a shared object; recompile with -fPIC
smp_spin_table.c:(.text+0x524): dangerous relocation: unsupported relocation
>> aarch64-linux-gnu-ld: smp_spin_table.c:(.text+0x528): undefined reference to `__cpu_soft_restart'
---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org
[-- Attachment #2: config.gz --]
[-- Type: application/gzip, Size: 32528 bytes --]
^ permalink raw reply [flat|nested] 14+ messages in thread
end of thread, other threads:[~2021-07-15 11:51 UTC | newest]
Thread overview: 14+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-07-14 17:41 [PATCH] arm64: kexec: add support for kexec with spin-table Henry Willard
2021-07-14 17:41 ` Henry Willard
2021-07-14 18:47 ` Mark Rutland
2021-07-14 18:47 ` Mark Rutland
2021-07-15 0:08 ` Henry Willard
2021-07-15 0:08 ` Henry Willard
2021-07-14 22:24 ` kernel test robot
2021-07-14 22:24 ` kernel test robot
2021-07-14 22:29 ` kernel test robot
2021-07-14 22:29 ` kernel test robot
2021-07-15 6:54 ` kernel test robot
2021-07-15 6:54 ` kernel test robot
2021-07-15 11:48 ` kernel test robot
2021-07-15 11:48 ` kernel test robot
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.