* [U-Boot] [PATCH 0/7] SMP support for RISC-V @ 2019-02-11 22:13 Lukas Auer 2019-02-11 22:13 ` [U-Boot] [PATCH 1/7] riscv: add infrastructure for calling functions on other harts Lukas Auer ` (7 more replies) 0 siblings, 8 replies; 45+ messages in thread From: Lukas Auer @ 2019-02-11 22:13 UTC (permalink / raw) To: u-boot This patch series adds SMP support for RISC-V to U-Boot. It allows U-Boot to run on multi-hart systems and will boot images passed to bootm on all harts. The bootm command is currently the only one that will boot images on all harts, bootefi is not yet supported. The patches have been successfully tested on both QEMU (machine and supervisor mode) and the HiFive Unleashed board [1] (supervisor mode), using BBL and OpenSBI. Mainline QEMU requires two patches [2, 3] to run in this configuration. I will send a follow-up patch to enable SMP support on the HiFive Unleashed board. [1]: https://patchwork.ozlabs.org/project/uboot/list/?series=91125 [2]: https://patchwork.ozlabs.org/patch/1039493/ [3]: https://patchwork.ozlabs.org/patch/1039082/ Lukas Auer (7): riscv: add infrastructure for calling functions on other harts riscv: import the supervisor binary interface header file riscv: implement IPI platform functions using SBI riscv: delay initialization of caches and debug UART riscv: add support for multi-hart systems riscv: boot images passed to bootm on all harts riscv: qemu: enable SMP arch/riscv/Kconfig | 36 +++++++++ arch/riscv/cpu/start.S | 116 +++++++++++++++++++++++++-- arch/riscv/include/asm/csr.h | 1 + arch/riscv/include/asm/global_data.h | 5 ++ arch/riscv/include/asm/sbi.h | 94 ++++++++++++++++++++++ arch/riscv/include/asm/smp.h | 53 ++++++++++++ arch/riscv/lib/Makefile | 2 + arch/riscv/lib/bootm.c | 13 ++- arch/riscv/lib/sbi_ipi.c | 25 ++++++ arch/riscv/lib/smp.c | 110 +++++++++++++++++++++++++ board/emulation/qemu-riscv/Kconfig | 1 + 11 files changed, 447 insertions(+), 9 deletions(-) create mode 100644 arch/riscv/include/asm/sbi.h create mode 100644 arch/riscv/include/asm/smp.h create mode 100644 arch/riscv/lib/sbi_ipi.c create mode 100644 arch/riscv/lib/smp.c -- 2.20.1 ^ permalink raw reply [flat|nested] 45+ messages in thread
* [U-Boot] [PATCH 1/7] riscv: add infrastructure for calling functions on other harts 2019-02-11 22:13 [U-Boot] [PATCH 0/7] SMP support for RISC-V Lukas Auer @ 2019-02-11 22:13 ` Lukas Auer 2019-02-12 1:44 ` Anup Patel ` (2 more replies) 2019-02-11 22:13 ` [U-Boot] [PATCH 2/7] riscv: import the supervisor binary interface header file Lukas Auer ` (6 subsequent siblings) 7 siblings, 3 replies; 45+ messages in thread From: Lukas Auer @ 2019-02-11 22:13 UTC (permalink / raw) To: u-boot Harts on RISC-V boot independently and U-Boot is responsible for managing them. Functions are called on other harts with smp_call_function(), which sends inter-processor interrupts (IPIs) to all other harts. Functions are specified with their address and two function arguments (argument 2 and 3). The first function argument is always the hart ID of the hart calling the function. On the other harts, the IPI interrupt handler handle_ipi() must be called on software interrupts to handle the request and call the specified function. Functions are stored in the ipi_data data structure. Every hart has its own data structure in global data. While this is not required at the moment (all harts are expected to boot Linux), this does allow future expansion, where other harts may be used for monitoring or other tasks. Signed-off-by: Lukas Auer <lukas.auer@aisec.fraunhofer.de> --- arch/riscv/Kconfig | 19 +++++ arch/riscv/include/asm/global_data.h | 5 ++ arch/riscv/include/asm/smp.h | 53 +++++++++++++ arch/riscv/lib/Makefile | 1 + arch/riscv/lib/smp.c | 110 +++++++++++++++++++++++++++ 5 files changed, 188 insertions(+) create mode 100644 arch/riscv/include/asm/smp.h create mode 100644 arch/riscv/lib/smp.c diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index c45e4d73a8..c0842178dd 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -116,4 +116,23 @@ config RISCV_RDTIME config SYS_MALLOC_F_LEN default 0x1000 +config SMP + bool "Symmetric Multi-Processing" + help + This enables support for systems with more than one CPU. If + you say N here, U-Boot will run on single and multiprocessor + machines, but will use only one CPU of a multiprocessor + machine. If you say Y here, U-Boot will run on many, but not + all, single processor machines. + +config NR_CPUS + int "Maximum number of CPUs (2-32)" + range 2 32 + depends on SMP + default "8" + help + On multiprocessor machines, U-Boot sets up a stack for each CPU. + Stack memory is pre-allocated. U-Boot must therefore know the + maximum number of CPUs that may be present. + endmenu diff --git a/arch/riscv/include/asm/global_data.h b/arch/riscv/include/asm/global_data.h index a3a342c6e1..23a5f35af5 100644 --- a/arch/riscv/include/asm/global_data.h +++ b/arch/riscv/include/asm/global_data.h @@ -10,12 +10,17 @@ #ifndef __ASM_GBL_DATA_H #define __ASM_GBL_DATA_H +#include <asm/smp.h> + /* Architecture-specific global data */ struct arch_global_data { long boot_hart; /* boot hart id */ #ifdef CONFIG_SIFIVE_CLINT void __iomem *clint; /* clint base address */ #endif +#ifdef CONFIG_SMP + struct ipi_data ipi[CONFIG_NR_CPUS]; +#endif }; #include <asm-generic/global_data.h> diff --git a/arch/riscv/include/asm/smp.h b/arch/riscv/include/asm/smp.h new file mode 100644 index 0000000000..bc863fdbaf --- /dev/null +++ b/arch/riscv/include/asm/smp.h @@ -0,0 +1,53 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * Copyright (C) 2019 Fraunhofer AISEC, + * Lukas Auer <lukas.auer@aisec.fraunhofer.de> + */ + +#ifndef _ASM_RISCV_SMP_H +#define _ASM_RISCV_SMP_H + +/** + * struct ipi_data - Inter-processor interrupt (IPI) data structure + * + * IPIs are used for SMP support to communicate to other harts what function to + * call. Functions are in the form + * void (*addr)(ulong hart, ulong arg0, ulong arg1). + * + * The function address and the two arguments, arg0 and arg1, are stored in the + * IPI data structure. The hart ID is inserted by the hart handling the IPI and + * calling the function. + * + * @addr: Address of function + * @arg0: First argument of function + * @arg1: Second argument of function + */ +struct ipi_data { + ulong addr; + ulong arg0; + ulong arg1; +}; + +/** + * handle_ipi() - interrupt handler for software interrupts + * + * The IPI interrupt handler must be called to handle software interrupts. It + * calls the function specified in the hart's IPI data structure. + * + * @hart: Hart ID of the current hart + */ +void handle_ipi(ulong hart); + +/** + * smp_call_function() - Call a function on all other harts + * + * Send IPIs with the specified function call to all harts. + * + * @addr: Address of function + * @arg0: First argument of function + * @arg1: Second argument of function + * @return 0 if OK, -ve on error + */ +int smp_call_function(ulong addr, ulong arg0, ulong arg1); + +#endif diff --git a/arch/riscv/lib/Makefile b/arch/riscv/lib/Makefile index edfa61690c..19370f9749 100644 --- a/arch/riscv/lib/Makefile +++ b/arch/riscv/lib/Makefile @@ -14,6 +14,7 @@ obj-$(CONFIG_SIFIVE_CLINT) += sifive_clint.o obj-y += interrupts.o obj-y += reset.o obj-y += setjmp.o +obj-$(CONFIG_SMP) += smp.o # For building EFI apps CFLAGS_$(EFI_CRT0) := $(CFLAGS_EFI) diff --git a/arch/riscv/lib/smp.c b/arch/riscv/lib/smp.c new file mode 100644 index 0000000000..1266a2a0ef --- /dev/null +++ b/arch/riscv/lib/smp.c @@ -0,0 +1,110 @@ +// SPDX-License-Identifier: GPL-2.0+ +/* + * Copyright (C) 2019 Fraunhofer AISEC, + * Lukas Auer <lukas.auer@aisec.fraunhofer.de> + */ + +#include <common.h> +#include <dm.h> +#include <asm/barrier.h> +#include <asm/smp.h> + +DECLARE_GLOBAL_DATA_PTR; + +/** + * riscv_send_ipi() - Send inter-processor interrupt (IPI) + * + * Platform code must provide this function. + * + * @hart: Hart ID of receiving hart + * @return 0 if OK, -ve on error + */ +extern int riscv_send_ipi(int hart); + +/** + * riscv_clear_ipi() - Clear inter-processor interrupt (IPI) + * + * Platform code must provide this function. + * + * @hart: Hart ID of hart to be cleared + * @return 0 if OK, -ve on error + */ +extern int riscv_clear_ipi(int hart); + +static int send_ipi_many(struct ipi_data *ipi) +{ + ofnode node, cpus; + u32 reg; + int ret; + + cpus = ofnode_path("/cpus"); + if (!ofnode_valid(cpus)) { + pr_err("Can't find cpus node!\n"); + return -EINVAL; + } + + ofnode_for_each_subnode(node, cpus) { + if (!ofnode_is_available(node)) + continue; + + /* read hart ID of CPU */ + ret = ofnode_read_u32(node, "reg", ®); + if (ret) + continue; + + /* skip if it is the hart we are running on */ + if (reg == gd->arch.boot_hart) + continue; + + if (reg >= CONFIG_NR_CPUS) { + pr_err("Hart ID %d is out of range, increase CONFIG_NR_CPUS\n", + reg); + continue; + } + + gd->arch.ipi[reg].addr = ipi->addr; + gd->arch.ipi[reg].arg0 = ipi->arg0; + gd->arch.ipi[reg].arg1 = ipi->arg1; + mb(); + + ret = riscv_send_ipi(reg); + if (ret) + return ret; + } + + return 0; +} + +void handle_ipi(ulong hart) +{ + int ret; + void (*smp_function)(ulong hart, ulong arg0, ulong arg1); + + if (hart >= CONFIG_NR_CPUS) + return; + + ret = riscv_clear_ipi(hart); + if (ret) { + pr_err("Cannot clear IPI\n"); + return; + } + + smp_function = (void (*)(ulong, ulong, ulong))gd->arch.ipi[hart].addr; + invalidate_icache_all(); + + smp_function(hart, gd->arch.ipi[hart].arg0, gd->arch.ipi[hart].arg1); +} + +int smp_call_function(ulong addr, ulong arg0, ulong arg1) +{ + int ret = 0; + struct ipi_data ipi; + + ipi.addr = addr; + ipi.arg0 = arg0; + ipi.arg1 = arg1; + + ret = send_ipi_many(&ipi); + + return ret; +} -- 2.20.1 ^ permalink raw reply related [flat|nested] 45+ messages in thread
* [U-Boot] [PATCH 1/7] riscv: add infrastructure for calling functions on other harts 2019-02-11 22:13 ` [U-Boot] [PATCH 1/7] riscv: add infrastructure for calling functions on other harts Lukas Auer @ 2019-02-12 1:44 ` Anup Patel 2019-02-17 21:55 ` Auer, Lukas 2019-02-12 3:03 ` Bin Meng 2019-02-18 4:58 ` Anup Patel 2 siblings, 1 reply; 45+ messages in thread From: Anup Patel @ 2019-02-12 1:44 UTC (permalink / raw) To: u-boot > -----Original Message----- > From: Lukas Auer [mailto:lukas.auer at aisec.fraunhofer.de] > Sent: Tuesday, February 12, 2019 3:44 AM > To: u-boot at lists.denx.de > Cc: Atish Patra <Atish.Patra@wdc.com>; Anup Patel > <Anup.Patel@wdc.com>; Bin Meng <bmeng.cn@gmail.com>; Andreas > Schwab <schwab@suse.de>; Palmer Dabbelt <palmer@sifive.com>; > Alexander Graf <agraf@suse.de>; Lukas Auer > <lukas.auer@aisec.fraunhofer.de>; Anup Patel <anup@brainfault.org>; Rick > Chen <rick@andestech.com> > Subject: [PATCH 1/7] riscv: add infrastructure for calling functions on other > harts > > Harts on RISC-V boot independently and U-Boot is responsible for managing > them. Functions are called on other harts with smp_call_function(), which > sends inter-processor interrupts (IPIs) to all other harts. Functions are > specified with their address and two function arguments (argument 2 and 3). > The first function argument is always the hart ID of the hart calling the > function. On the other harts, the IPI interrupt handler handle_ipi() must be > called on software interrupts to handle the request and call the specified > function. > > Functions are stored in the ipi_data data structure. Every hart has its own > data structure in global data. While this is not required at the moment (all > harts are expected to boot Linux), this does allow future expansion, where > other harts may be used for monitoring or other tasks. > > Signed-off-by: Lukas Auer <lukas.auer@aisec.fraunhofer.de> > --- > > arch/riscv/Kconfig | 19 +++++ > arch/riscv/include/asm/global_data.h | 5 ++ > arch/riscv/include/asm/smp.h | 53 +++++++++++++ > arch/riscv/lib/Makefile | 1 + > arch/riscv/lib/smp.c | 110 +++++++++++++++++++++++++++ > 5 files changed, 188 insertions(+) > create mode 100644 arch/riscv/include/asm/smp.h create mode 100644 > arch/riscv/lib/smp.c > > diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index > c45e4d73a8..c0842178dd 100644 > --- a/arch/riscv/Kconfig > +++ b/arch/riscv/Kconfig > @@ -116,4 +116,23 @@ config RISCV_RDTIME config SYS_MALLOC_F_LEN > default 0x1000 > > +config SMP > + bool "Symmetric Multi-Processing" > + help > + This enables support for systems with more than one CPU. If > + you say N here, U-Boot will run on single and multiprocessor > + machines, but will use only one CPU of a multiprocessor > + machine. If you say Y here, U-Boot will run on many, but not > + all, single processor machines. > + > +config NR_CPUS > + int "Maximum number of CPUs (2-32)" > + range 2 32 > + depends on SMP > + default "8" > + help > + On multiprocessor machines, U-Boot sets up a stack for each CPU. > + Stack memory is pre-allocated. U-Boot must therefore know the > + maximum number of CPUs that may be present. > + > endmenu > diff --git a/arch/riscv/include/asm/global_data.h > b/arch/riscv/include/asm/global_data.h > index a3a342c6e1..23a5f35af5 100644 > --- a/arch/riscv/include/asm/global_data.h > +++ b/arch/riscv/include/asm/global_data.h > @@ -10,12 +10,17 @@ > #ifndef __ASM_GBL_DATA_H > #define __ASM_GBL_DATA_H > > +#include <asm/smp.h> > + > /* Architecture-specific global data */ struct arch_global_data { > long boot_hart; /* boot hart id */ > #ifdef CONFIG_SIFIVE_CLINT > void __iomem *clint; /* clint base address */ > #endif > +#ifdef CONFIG_SMP > + struct ipi_data ipi[CONFIG_NR_CPUS]; > +#endif > }; > > #include <asm-generic/global_data.h> > diff --git a/arch/riscv/include/asm/smp.h b/arch/riscv/include/asm/smp.h > new file mode 100644 index 0000000000..bc863fdbaf > --- /dev/null > +++ b/arch/riscv/include/asm/smp.h > @@ -0,0 +1,53 @@ > +/* SPDX-License-Identifier: GPL-2.0 */ > +/* > + * Copyright (C) 2019 Fraunhofer AISEC, > + * Lukas Auer <lukas.auer@aisec.fraunhofer.de> */ > + > +#ifndef _ASM_RISCV_SMP_H > +#define _ASM_RISCV_SMP_H > + > +/** > + * struct ipi_data - Inter-processor interrupt (IPI) data structure > + * > + * IPIs are used for SMP support to communicate to other harts what > +function to > + * call. Functions are in the form > + * void (*addr)(ulong hart, ulong arg0, ulong arg1). > + * > + * The function address and the two arguments, arg0 and arg1, are > +stored in the > + * IPI data structure. The hart ID is inserted by the hart handling the > +IPI and > + * calling the function. > + * > + * @addr: Address of function > + * @arg0: First argument of function > + * @arg1: Second argument of function > + */ > +struct ipi_data { > + ulong addr; > + ulong arg0; > + ulong arg1; > +}; > + > +/** > + * handle_ipi() - interrupt handler for software interrupts > + * > + * The IPI interrupt handler must be called to handle software > +interrupts. It > + * calls the function specified in the hart's IPI data structure. > + * > + * @hart: Hart ID of the current hart > + */ > +void handle_ipi(ulong hart); > + > +/** > + * smp_call_function() - Call a function on all other harts > + * > + * Send IPIs with the specified function call to all harts. > + * > + * @addr: Address of function > + * @arg0: First argument of function > + * @arg1: Second argument of function > + * @return 0 if OK, -ve on error > + */ > +int smp_call_function(ulong addr, ulong arg0, ulong arg1); > + > +#endif > diff --git a/arch/riscv/lib/Makefile b/arch/riscv/lib/Makefile index > edfa61690c..19370f9749 100644 > --- a/arch/riscv/lib/Makefile > +++ b/arch/riscv/lib/Makefile > @@ -14,6 +14,7 @@ obj-$(CONFIG_SIFIVE_CLINT) += sifive_clint.o > obj-y += interrupts.o > obj-y += reset.o > obj-y += setjmp.o > +obj-$(CONFIG_SMP) += smp.o > > # For building EFI apps > CFLAGS_$(EFI_CRT0) := $(CFLAGS_EFI) > diff --git a/arch/riscv/lib/smp.c b/arch/riscv/lib/smp.c new file mode 100644 > index 0000000000..1266a2a0ef > --- /dev/null > +++ b/arch/riscv/lib/smp.c > @@ -0,0 +1,110 @@ > +// SPDX-License-Identifier: GPL-2.0+ > +/* > + * Copyright (C) 2019 Fraunhofer AISEC, > + * Lukas Auer <lukas.auer@aisec.fraunhofer.de> */ > + > +#include <common.h> > +#include <dm.h> > +#include <asm/barrier.h> > +#include <asm/smp.h> > + > +DECLARE_GLOBAL_DATA_PTR; > + > +/** > + * riscv_send_ipi() - Send inter-processor interrupt (IPI) > + * > + * Platform code must provide this function. > + * > + * @hart: Hart ID of receiving hart > + * @return 0 if OK, -ve on error > + */ > +extern int riscv_send_ipi(int hart); > + > +/** > + * riscv_clear_ipi() - Clear inter-processor interrupt (IPI) > + * > + * Platform code must provide this function. > + * > + * @hart: Hart ID of hart to be cleared > + * @return 0 if OK, -ve on error > + */ > +extern int riscv_clear_ipi(int hart); > + > +static int send_ipi_many(struct ipi_data *ipi) { > + ofnode node, cpus; > + u32 reg; > + int ret; > + > + cpus = ofnode_path("/cpus"); > + if (!ofnode_valid(cpus)) { > + pr_err("Can't find cpus node!\n"); > + return -EINVAL; > + } > + > + ofnode_for_each_subnode(node, cpus) { > + if (!ofnode_is_available(node)) > + continue; It is not correct to assume that whatever CPUs are marked available will come online. It is possible that certain available CPUs failed to come online due HW failure. Better approach would be keep an atomic bitmask of HARTs that have entered U-Boot. All HARTs that enter U-Boot will update the atomic HART available bitmask. We send IPI only to HARTs that are available as-per atomic HART bitmask. > + > + /* read hart ID of CPU */ > + ret = ofnode_read_u32(node, "reg", ®); > + if (ret) > + continue; > + > + /* skip if it is the hart we are running on */ > + if (reg == gd->arch.boot_hart) > + continue; > + > + if (reg >= CONFIG_NR_CPUS) { > + pr_err("Hart ID %d is out of range, increase > CONFIG_NR_CPUS\n", > + reg); > + continue; > + } > + > + gd->arch.ipi[reg].addr = ipi->addr; > + gd->arch.ipi[reg].arg0 = ipi->arg0; > + gd->arch.ipi[reg].arg1 = ipi->arg1; > + mb(); > + > + ret = riscv_send_ipi(reg); > + if (ret) > + return ret; > + } > + > + return 0; > +} > + > +void handle_ipi(ulong hart) > +{ > + int ret; > + void (*smp_function)(ulong hart, ulong arg0, ulong arg1); > + > + if (hart >= CONFIG_NR_CPUS) > + return; > + > + ret = riscv_clear_ipi(hart); > + if (ret) { > + pr_err("Cannot clear IPI\n"); > + return; > + } > + > + smp_function = (void (*)(ulong, ulong, ulong))gd- > >arch.ipi[hart].addr; > + invalidate_icache_all(); > + > + smp_function(hart, gd->arch.ipi[hart].arg0, gd->arch.ipi[hart].arg1); > +} > + > +int smp_call_function(ulong addr, ulong arg0, ulong arg1) { > + int ret = 0; > + struct ipi_data ipi; > + > + ipi.addr = addr; > + ipi.arg0 = arg0; > + ipi.arg1 = arg1; > + > + ret = send_ipi_many(&ipi); > + > + return ret; > +} > -- > 2.20.1 Regards, Anup ^ permalink raw reply [flat|nested] 45+ messages in thread
* [U-Boot] [PATCH 1/7] riscv: add infrastructure for calling functions on other harts 2019-02-12 1:44 ` Anup Patel @ 2019-02-17 21:55 ` Auer, Lukas [not found] ` <752D002CFF5D0F4FA35C0100F1D73F3FA4087D3C@ATCPCS16.andestech.com> 0 siblings, 1 reply; 45+ messages in thread From: Auer, Lukas @ 2019-02-17 21:55 UTC (permalink / raw) To: u-boot On Tue, 2019-02-12 at 01:44 +0000, Anup Patel wrote: > > -----Original Message----- > > From: Lukas Auer [mailto:lukas.auer at aisec.fraunhofer.de] > > Sent: Tuesday, February 12, 2019 3:44 AM > > To: u-boot at lists.denx.de > > Cc: Atish Patra <Atish.Patra@wdc.com>; Anup Patel > > <Anup.Patel@wdc.com>; Bin Meng <bmeng.cn@gmail.com>; Andreas > > Schwab <schwab@suse.de>; Palmer Dabbelt <palmer@sifive.com>; > > Alexander Graf <agraf@suse.de>; Lukas Auer > > <lukas.auer@aisec.fraunhofer.de>; Anup Patel <anup@brainfault.org>; > > Rick > > Chen <rick@andestech.com> > > Subject: [PATCH 1/7] riscv: add infrastructure for calling > > functions on other > > harts > > > > Harts on RISC-V boot independently and U-Boot is responsible for > > managing > > them. Functions are called on other harts with smp_call_function(), > > which > > sends inter-processor interrupts (IPIs) to all other harts. > > Functions are > > specified with their address and two function arguments (argument 2 > > and 3). > > The first function argument is always the hart ID of the hart > > calling the > > function. On the other harts, the IPI interrupt handler > > handle_ipi() must be > > called on software interrupts to handle the request and call the > > specified > > function. > > > > Functions are stored in the ipi_data data structure. Every hart has > > its own > > data structure in global data. While this is not required at the > > moment (all > > harts are expected to boot Linux), this does allow future > > expansion, where > > other harts may be used for monitoring or other tasks. > > > > Signed-off-by: Lukas Auer <lukas.auer@aisec.fraunhofer.de> > > --- > > > > arch/riscv/Kconfig | 19 +++++ > > arch/riscv/include/asm/global_data.h | 5 ++ > > arch/riscv/include/asm/smp.h | 53 +++++++++++++ > > arch/riscv/lib/Makefile | 1 + > > arch/riscv/lib/smp.c | 110 > > +++++++++++++++++++++++++++ > > 5 files changed, 188 insertions(+) > > create mode 100644 arch/riscv/include/asm/smp.h create mode > > 100644 > > arch/riscv/lib/smp.c > > > > diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index > > c45e4d73a8..c0842178dd 100644 > > --- a/arch/riscv/Kconfig > > +++ b/arch/riscv/Kconfig > > @@ -116,4 +116,23 @@ config RISCV_RDTIME config SYS_MALLOC_F_LEN > > default 0x1000 > > > > +config SMP > > + bool "Symmetric Multi-Processing" > > + help > > + This enables support for systems with more than one CPU. If > > + you say N here, U-Boot will run on single and multiprocessor > > + machines, but will use only one CPU of a multiprocessor > > + machine. If you say Y here, U-Boot will run on many, but not > > + all, single processor machines. > > + > > +config NR_CPUS > > + int "Maximum number of CPUs (2-32)" > > + range 2 32 > > + depends on SMP > > + default "8" > > + help > > + On multiprocessor machines, U-Boot sets up a stack for each > > CPU. > > + Stack memory is pre-allocated. U-Boot must therefore know the > > + maximum number of CPUs that may be present. > > + > > endmenu > > diff --git a/arch/riscv/include/asm/global_data.h > > b/arch/riscv/include/asm/global_data.h > > index a3a342c6e1..23a5f35af5 100644 > > --- a/arch/riscv/include/asm/global_data.h > > +++ b/arch/riscv/include/asm/global_data.h > > @@ -10,12 +10,17 @@ > > #ifndef __ASM_GBL_DATA_H > > #define __ASM_GBL_DATA_H > > > > +#include <asm/smp.h> > > + > > /* Architecture-specific global data */ struct arch_global_data { > > long boot_hart; /* boot hart id */ > > #ifdef CONFIG_SIFIVE_CLINT > > void __iomem *clint; /* clint base address */ > > #endif > > +#ifdef CONFIG_SMP > > + struct ipi_data ipi[CONFIG_NR_CPUS]; > > +#endif > > }; > > > > #include <asm-generic/global_data.h> > > diff --git a/arch/riscv/include/asm/smp.h > > b/arch/riscv/include/asm/smp.h > > new file mode 100644 index 0000000000..bc863fdbaf > > --- /dev/null > > +++ b/arch/riscv/include/asm/smp.h > > @@ -0,0 +1,53 @@ > > +/* SPDX-License-Identifier: GPL-2.0 */ > > +/* > > + * Copyright (C) 2019 Fraunhofer AISEC, > > + * Lukas Auer <lukas.auer@aisec.fraunhofer.de> */ > > + > > +#ifndef _ASM_RISCV_SMP_H > > +#define _ASM_RISCV_SMP_H > > + > > +/** > > + * struct ipi_data - Inter-processor interrupt (IPI) data > > structure > > + * > > + * IPIs are used for SMP support to communicate to other harts > > what > > +function to > > + * call. Functions are in the form > > + * void (*addr)(ulong hart, ulong arg0, ulong arg1). > > + * > > + * The function address and the two arguments, arg0 and arg1, are > > +stored in the > > + * IPI data structure. The hart ID is inserted by the hart > > handling the > > +IPI and > > + * calling the function. > > + * > > + * @addr: Address of function > > + * @arg0: First argument of function > > + * @arg1: Second argument of function > > + */ > > +struct ipi_data { > > + ulong addr; > > + ulong arg0; > > + ulong arg1; > > +}; > > + > > +/** > > + * handle_ipi() - interrupt handler for software interrupts > > + * > > + * The IPI interrupt handler must be called to handle software > > +interrupts. It > > + * calls the function specified in the hart's IPI data structure. > > + * > > + * @hart: Hart ID of the current hart > > + */ > > +void handle_ipi(ulong hart); > > + > > +/** > > + * smp_call_function() - Call a function on all other harts > > + * > > + * Send IPIs with the specified function call to all harts. > > + * > > + * @addr: Address of function > > + * @arg0: First argument of function > > + * @arg1: Second argument of function > > + * @return 0 if OK, -ve on error > > + */ > > +int smp_call_function(ulong addr, ulong arg0, ulong arg1); > > + > > +#endif > > diff --git a/arch/riscv/lib/Makefile b/arch/riscv/lib/Makefile > > index > > edfa61690c..19370f9749 100644 > > --- a/arch/riscv/lib/Makefile > > +++ b/arch/riscv/lib/Makefile > > @@ -14,6 +14,7 @@ obj-$(CONFIG_SIFIVE_CLINT) += sifive_clint.o > > obj-y += interrupts.o > > obj-y += reset.o > > obj-y += setjmp.o > > +obj-$(CONFIG_SMP) += smp.o > > > > # For building EFI apps > > CFLAGS_$(EFI_CRT0) := $(CFLAGS_EFI) > > diff --git a/arch/riscv/lib/smp.c b/arch/riscv/lib/smp.c new file > > mode 100644 > > index 0000000000..1266a2a0ef > > --- /dev/null > > +++ b/arch/riscv/lib/smp.c > > @@ -0,0 +1,110 @@ > > +// SPDX-License-Identifier: GPL-2.0+ > > +/* > > + * Copyright (C) 2019 Fraunhofer AISEC, > > + * Lukas Auer <lukas.auer@aisec.fraunhofer.de> */ > > + > > +#include <common.h> > > +#include <dm.h> > > +#include <asm/barrier.h> > > +#include <asm/smp.h> > > + > > +DECLARE_GLOBAL_DATA_PTR; > > + > > +/** > > + * riscv_send_ipi() - Send inter-processor interrupt (IPI) > > + * > > + * Platform code must provide this function. > > + * > > + * @hart: Hart ID of receiving hart > > + * @return 0 if OK, -ve on error > > + */ > > +extern int riscv_send_ipi(int hart); > > + > > +/** > > + * riscv_clear_ipi() - Clear inter-processor interrupt (IPI) > > + * > > + * Platform code must provide this function. > > + * > > + * @hart: Hart ID of hart to be cleared > > + * @return 0 if OK, -ve on error > > + */ > > +extern int riscv_clear_ipi(int hart); > > + > > +static int send_ipi_many(struct ipi_data *ipi) { > > + ofnode node, cpus; > > + u32 reg; > > + int ret; > > + > > + cpus = ofnode_path("/cpus"); > > + if (!ofnode_valid(cpus)) { > > + pr_err("Can't find cpus node!\n"); > > + return -EINVAL; > > + } > > + > > + ofnode_for_each_subnode(node, cpus) { > > + if (!ofnode_is_available(node)) > > + continue; > > It is not correct to assume that whatever CPUs are marked > available will come online. It is possible that certain available > CPUs failed to come online due HW failure. > This was intended so that we don't send IPIs to harts, which have been explicitly marked as disabled. > Better approach would be keep an atomic bitmask of HARTs > that have entered U-Boot. All HARTs that enter U-Boot will > update the atomic HART available bitmask. We send IPI only > to HARTs that are available as-per atomic HART bitmask. > I'm not sure if this is required in U-Boot, since we are not relying on all harts to boot for U-Boot to function. We only try to boot all harts listed as available in the device tree. Is there a situation, in which we would want to avoid sending an IPI to an unavailable hart? Thanks, Lukas > > + > > + /* read hart ID of CPU */ > > + ret = ofnode_read_u32(node, "reg", ®); > > + if (ret) > > + continue; > > + > > + /* skip if it is the hart we are running on */ > > + if (reg == gd->arch.boot_hart) > > + continue; > > + > > + if (reg >= CONFIG_NR_CPUS) { > > + pr_err("Hart ID %d is out of range, increase > > CONFIG_NR_CPUS\n", > > + reg); > > + continue; > > + } > > + > > + gd->arch.ipi[reg].addr = ipi->addr; > > + gd->arch.ipi[reg].arg0 = ipi->arg0; > > + gd->arch.ipi[reg].arg1 = ipi->arg1; > > + mb(); > > + > > + ret = riscv_send_ipi(reg); > > + if (ret) > > + return ret; > > + } > > + > > + return 0; > > +} > > + > > +void handle_ipi(ulong hart) > > +{ > > + int ret; > > + void (*smp_function)(ulong hart, ulong arg0, ulong arg1); > > + > > + if (hart >= CONFIG_NR_CPUS) > > + return; > > + > > + ret = riscv_clear_ipi(hart); > > + if (ret) { > > + pr_err("Cannot clear IPI\n"); > > + return; > > + } > > + > > + smp_function = (void (*)(ulong, ulong, ulong))gd- > > > arch.ipi[hart].addr; > > + invalidate_icache_all(); > > + > > + smp_function(hart, gd->arch.ipi[hart].arg0, gd- > > >arch.ipi[hart].arg1); > > +} > > + > > +int smp_call_function(ulong addr, ulong arg0, ulong arg1) { > > + int ret = 0; > > + struct ipi_data ipi; > > + > > + ipi.addr = addr; > > + ipi.arg0 = arg0; > > + ipi.arg1 = arg1; > > + > > + ret = send_ipi_many(&ipi); > > + > > + return ret; > > +} > > -- > > 2.20.1 > > Regards, > Anup ^ permalink raw reply [flat|nested] 45+ messages in thread
[parent not found: <752D002CFF5D0F4FA35C0100F1D73F3FA4087D3C@ATCPCS16.andestech.com>]
* [U-Boot] [PATCH 1/7] riscv: add infrastructure for calling functions on other harts [not found] ` <752D002CFF5D0F4FA35C0100F1D73F3FA4087D3C@ATCPCS16.andestech.com> @ 2019-02-18 3:24 ` Rick Chen 2019-02-18 3:40 ` Anup Patel 0 siblings, 1 reply; 45+ messages in thread From: Rick Chen @ 2019-02-18 3:24 UTC (permalink / raw) To: u-boot <rick@andestech.com> 於 2019年2月18日 週一 上午11:00寫道: > > > > > -----Original Message----- > > From: Auer, Lukas [mailto:lukas.auer at aisec.fraunhofer.de] > > Sent: Monday, February 18, 2019 5:55 AM > > To: u-boot at lists.denx.de; Anup.Patel at wdc.com > > Cc: anup at brainfault.org; bmeng.cn at gmail.com; schwab at suse.de; Rick Jian-Zhi > > Chen(陳建志); palmer at sifive.com; Atish.Patra at wdc.com; agraf at suse.de > > Subject: Re: [PATCH 1/7] riscv: add infrastructure for calling functions on other > > harts > > > > On Tue, 2019-02-12 at 01:44 +0000, Anup Patel wrote: > > > > -----Original Message----- > > > > From: Lukas Auer [mailto:lukas.auer at aisec.fraunhofer.de] > > > > Sent: Tuesday, February 12, 2019 3:44 AM > > > > To: u-boot at lists.denx.de > > > > Cc: Atish Patra <Atish.Patra@wdc.com>; Anup Patel > > > > <Anup.Patel@wdc.com>; Bin Meng <bmeng.cn@gmail.com>; Andreas > > Schwab > > > > <schwab@suse.de>; Palmer Dabbelt <palmer@sifive.com>; Alexander Graf > > > > <agraf@suse.de>; Lukas Auer <lukas.auer@aisec.fraunhofer.de>; Anup > > > > Patel <anup@brainfault.org>; Rick Chen <rick@andestech.com> > > > > Subject: [PATCH 1/7] riscv: add infrastructure for calling functions > > > > on other harts > > > > > > > > Harts on RISC-V boot independently and U-Boot is responsible for > > > > managing them. Functions are called on other harts with > > > > smp_call_function(), which sends inter-processor interrupts (IPIs) > > > > to all other harts. > > > > Functions are > > > > specified with their address and two function arguments (argument 2 > > > > and 3). > > > > The first function argument is always the hart ID of the hart > > > > calling the function. On the other harts, the IPI interrupt handler > > > > handle_ipi() must be > > > > called on software interrupts to handle the request and call the > > > > specified function. > > > > > > > > Functions are stored in the ipi_data data structure. Every hart has > > > > its own data structure in global data. While this is not required at > > > > the moment (all harts are expected to boot Linux), this does allow > > > > future expansion, where other harts may be used for monitoring or > > > > other tasks. > > > > > > > > Signed-off-by: Lukas Auer <lukas.auer@aisec.fraunhofer.de> > > > > --- > > > > > > > > arch/riscv/Kconfig | 19 +++++ > > > > arch/riscv/include/asm/global_data.h | 5 ++ > > > > arch/riscv/include/asm/smp.h | 53 +++++++++++++ > > > > arch/riscv/lib/Makefile | 1 + > > > > arch/riscv/lib/smp.c | 110 > > > > +++++++++++++++++++++++++++ > > > > 5 files changed, 188 insertions(+) > > > > create mode 100644 arch/riscv/include/asm/smp.h create mode > > > > 100644 > > > > arch/riscv/lib/smp.c > > > > > > > > diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index > > > > c45e4d73a8..c0842178dd 100644 > > > > --- a/arch/riscv/Kconfig > > > > +++ b/arch/riscv/Kconfig > > > > @@ -116,4 +116,23 @@ config RISCV_RDTIME config SYS_MALLOC_F_LEN > > > > default 0x1000 > > > > > > > > +config SMP > > > > + bool "Symmetric Multi-Processing" > > > > + help > > > > + This enables support for systems with more than one CPU. If > > > > + you say N here, U-Boot will run on single and multiprocessor > > > > + machines, but will use only one CPU of a multiprocessor > > > > + machine. If you say Y here, U-Boot will run on many, but not > > > > + all, single processor machines. > > > > + > > > > +config NR_CPUS > > > > + int "Maximum number of CPUs (2-32)" > > > > + range 2 32 > > > > + depends on SMP > > > > + default "8" > > > > + help > > > > + On multiprocessor machines, U-Boot sets up a stack for each > > > > CPU. > > > > + Stack memory is pre-allocated. U-Boot must therefore know the > > > > + maximum number of CPUs that may be present. > > > > + > > > > endmenu > > > > diff --git a/arch/riscv/include/asm/global_data.h > > > > b/arch/riscv/include/asm/global_data.h > > > > index a3a342c6e1..23a5f35af5 100644 > > > > --- a/arch/riscv/include/asm/global_data.h > > > > +++ b/arch/riscv/include/asm/global_data.h > > > > @@ -10,12 +10,17 @@ > > > > #ifndef __ASM_GBL_DATA_H > > > > #define __ASM_GBL_DATA_H > > > > > > > > +#include <asm/smp.h> > > > > + > > > > /* Architecture-specific global data */ struct arch_global_data { > > > > long boot_hart; /* boot hart id */ > > > > #ifdef CONFIG_SIFIVE_CLINT > > > > void __iomem *clint; /* clint base address */ > > > > #endif > > > > +#ifdef CONFIG_SMP > > > > + struct ipi_data ipi[CONFIG_NR_CPUS]; #endif > > > > }; > > > > > > > > #include <asm-generic/global_data.h> > > > > diff --git a/arch/riscv/include/asm/smp.h > > > > b/arch/riscv/include/asm/smp.h > > > > new file mode 100644 index 0000000000..bc863fdbaf > > > > --- /dev/null > > > > +++ b/arch/riscv/include/asm/smp.h > > > > @@ -0,0 +1,53 @@ > > > > +/* SPDX-License-Identifier: GPL-2.0 */ > > > > +/* > > > > + * Copyright (C) 2019 Fraunhofer AISEC, > > > > + * Lukas Auer <lukas.auer@aisec.fraunhofer.de> */ > > > > + > > > > +#ifndef _ASM_RISCV_SMP_H > > > > +#define _ASM_RISCV_SMP_H > > > > + > > > > +/** > > > > + * struct ipi_data - Inter-processor interrupt (IPI) data > > > > structure > > > > + * > > > > + * IPIs are used for SMP support to communicate to other harts > > > > what > > > > +function to > > > > + * call. Functions are in the form > > > > + * void (*addr)(ulong hart, ulong arg0, ulong arg1). > > > > + * > > > > + * The function address and the two arguments, arg0 and arg1, are > > > > +stored in the > > > > + * IPI data structure. The hart ID is inserted by the hart > > > > handling the > > > > +IPI and > > > > + * calling the function. > > > > + * > > > > + * @addr: Address of function > > > > + * @arg0: First argument of function > > > > + * @arg1: Second argument of function > > > > + */ > > > > +struct ipi_data { > > > > + ulong addr; > > > > + ulong arg0; > > > > + ulong arg1; > > > > +}; > > > > + > > > > +/** > > > > + * handle_ipi() - interrupt handler for software interrupts > > > > + * > > > > + * The IPI interrupt handler must be called to handle software > > > > +interrupts. It > > > > + * calls the function specified in the hart's IPI data structure. > > > > + * > > > > + * @hart: Hart ID of the current hart > > > > + */ > > > > +void handle_ipi(ulong hart); > > > > + > > > > +/** > > > > + * smp_call_function() - Call a function on all other harts > > > > + * > > > > + * Send IPIs with the specified function call to all harts. > > > > + * > > > > + * @addr: Address of function > > > > + * @arg0: First argument of function > > > > + * @arg1: Second argument of function > > > > + * @return 0 if OK, -ve on error > > > > + */ > > > > +int smp_call_function(ulong addr, ulong arg0, ulong arg1); > > > > + > > > > +#endif > > > > diff --git a/arch/riscv/lib/Makefile b/arch/riscv/lib/Makefile > > > > index > > > > edfa61690c..19370f9749 100644 > > > > --- a/arch/riscv/lib/Makefile > > > > +++ b/arch/riscv/lib/Makefile > > > > @@ -14,6 +14,7 @@ obj-$(CONFIG_SIFIVE_CLINT) += sifive_clint.o > > > > obj-y += interrupts.o > > > > obj-y += reset.o > > > > obj-y += setjmp.o > > > > +obj-$(CONFIG_SMP) += smp.o > > > > > > > > # For building EFI apps > > > > CFLAGS_$(EFI_CRT0) := $(CFLAGS_EFI) > > > > diff --git a/arch/riscv/lib/smp.c b/arch/riscv/lib/smp.c new file > > > > mode 100644 > > > > index 0000000000..1266a2a0ef > > > > --- /dev/null > > > > +++ b/arch/riscv/lib/smp.c > > > > @@ -0,0 +1,110 @@ > > > > +// SPDX-License-Identifier: GPL-2.0+ > > > > +/* > > > > + * Copyright (C) 2019 Fraunhofer AISEC, > > > > + * Lukas Auer <lukas.auer@aisec.fraunhofer.de> */ > > > > + > > > > +#include <common.h> > > > > +#include <dm.h> > > > > +#include <asm/barrier.h> > > > > +#include <asm/smp.h> > > > > + > > > > +DECLARE_GLOBAL_DATA_PTR; > > > > + > > > > +/** > > > > + * riscv_send_ipi() - Send inter-processor interrupt (IPI) > > > > + * > > > > + * Platform code must provide this function. > > > > + * > > > > + * @hart: Hart ID of receiving hart > > > > + * @return 0 if OK, -ve on error > > > > + */ > > > > +extern int riscv_send_ipi(int hart); > > > > + > > > > +/** > > > > + * riscv_clear_ipi() - Clear inter-processor interrupt (IPI) > > > > + * > > > > + * Platform code must provide this function. > > > > + * > > > > + * @hart: Hart ID of hart to be cleared > > > > + * @return 0 if OK, -ve on error > > > > + */ > > > > +extern int riscv_clear_ipi(int hart); > > > > + > > > > +static int send_ipi_many(struct ipi_data *ipi) { > > > > + ofnode node, cpus; > > > > + u32 reg; > > > > + int ret; > > > > + > > > > + cpus = ofnode_path("/cpus"); > > > > + if (!ofnode_valid(cpus)) { > > > > + pr_err("Can't find cpus node!\n"); > > > > + return -EINVAL; > > > > + } > > > > + > > > > + ofnode_for_each_subnode(node, cpus) { > > > > + if (!ofnode_is_available(node)) > > > > + continue; > > > > > > It is not correct to assume that whatever CPUs are marked > > > available will come online. It is possible that certain available > > > CPUs failed to come online due HW failure. > > > > > > > This was intended so that we don't send IPIs to harts, which have been > > explicitly marked as disabled. > > > > > Better approach would be keep an atomic bitmask of HARTs > > > that have entered U-Boot. All HARTs that enter U-Boot will > > > update the atomic HART available bitmask. We send IPI only > > > to HARTs that are available as-per atomic HART bitmask. > > > > > > > I'm not sure if this is required in U-Boot, since we are not relying on > > all harts to boot for U-Boot to function. We only try to boot all harts > > listed as available in the device tree. > > It may also need to get information from each cpu nodes. The cpu nodes is important, they must be match with the bitmask, or it maybe go wrong. So the bitmask way seem not be necessary. BR Rick > > Is there a situation, in which we would want to avoid sending an IPI to > > an unavailable hart? > > > > Thanks, > > Lukas > > > > > > + > > > > + /* read hart ID of CPU */ > > > > + ret = ofnode_read_u32(node, "reg", ®); > > > > + if (ret) > > > > + continue; > > > > + > > > > + /* skip if it is the hart we are running on */ > > > > + if (reg == gd->arch.boot_hart) > > > > + continue; > > > > + > > > > + if (reg >= CONFIG_NR_CPUS) { > > > > + pr_err("Hart ID %d is out of range, increase > > > > CONFIG_NR_CPUS\n", > > > > + reg); > > > > + continue; > > > > + } > > > > + > > > > + gd->arch.ipi[reg].addr = ipi->addr; > > > > + gd->arch.ipi[reg].arg0 = ipi->arg0; > > > > + gd->arch.ipi[reg].arg1 = ipi->arg1; > > > > + mb(); > > > > + > > > > + ret = riscv_send_ipi(reg); > > > > + if (ret) > > > > + return ret; > > > > + } > > > > + > > > > + return 0; > > > > +} > > > > + > > > > +void handle_ipi(ulong hart) > > > > +{ > > > > + int ret; > > > > + void (*smp_function)(ulong hart, ulong arg0, ulong arg1); > > > > + > > > > + if (hart >= CONFIG_NR_CPUS) > > > > + return; > > > > + > > > > + ret = riscv_clear_ipi(hart); > > > > + if (ret) { > > > > + pr_err("Cannot clear IPI\n"); > > > > + return; > > > > + } > > > > + > > > > + smp_function = (void (*)(ulong, ulong, ulong))gd- > > > > > arch.ipi[hart].addr; > > > > + invalidate_icache_all(); > > > > + > > > > + smp_function(hart, gd->arch.ipi[hart].arg0, gd- > > > > >arch.ipi[hart].arg1); > > > > +} > > > > + > > > > +int smp_call_function(ulong addr, ulong arg0, ulong arg1) { > > > > + int ret = 0; > > > > + struct ipi_data ipi; > > > > + > > > > + ipi.addr = addr; > > > > + ipi.arg0 = arg0; > > > > + ipi.arg1 = arg1; > > > > + > > > > + ret = send_ipi_many(&ipi); > > > > + > > > > + return ret; > > > > +} > > > > -- > > > > 2.20.1 > > > > > > Regards, > > > Anup ^ permalink raw reply [flat|nested] 45+ messages in thread
* [U-Boot] [PATCH 1/7] riscv: add infrastructure for calling functions on other harts 2019-02-18 3:24 ` Rick Chen @ 2019-02-18 3:40 ` Anup Patel 2019-02-18 9:53 ` Auer, Lukas 0 siblings, 1 reply; 45+ messages in thread From: Anup Patel @ 2019-02-18 3:40 UTC (permalink / raw) To: u-boot On Mon, Feb 18, 2019 at 8:53 AM Rick Chen <rickchen36@gmail.com> wrote: > > <rick@andestech.com> 於 2019年2月18日 週一 上午11:00寫道: > > > > > > > > > -----Original Message----- > > > From: Auer, Lukas [mailto:lukas.auer at aisec.fraunhofer.de] > > > Sent: Monday, February 18, 2019 5:55 AM > > > To: u-boot at lists.denx.de; Anup.Patel at wdc.com > > > Cc: anup at brainfault.org; bmeng.cn at gmail.com; schwab at suse.de; Rick Jian-Zhi > > > Chen(陳建志); palmer at sifive.com; Atish.Patra at wdc.com; agraf at suse.de > > > Subject: Re: [PATCH 1/7] riscv: add infrastructure for calling functions on other > > > harts > > > > > > On Tue, 2019-02-12 at 01:44 +0000, Anup Patel wrote: > > > > > -----Original Message----- > > > > > From: Lukas Auer [mailto:lukas.auer at aisec.fraunhofer.de] > > > > > Sent: Tuesday, February 12, 2019 3:44 AM > > > > > To: u-boot at lists.denx.de > > > > > Cc: Atish Patra <Atish.Patra@wdc.com>; Anup Patel > > > > > <Anup.Patel@wdc.com>; Bin Meng <bmeng.cn@gmail.com>; Andreas > > > Schwab > > > > > <schwab@suse.de>; Palmer Dabbelt <palmer@sifive.com>; Alexander Graf > > > > > <agraf@suse.de>; Lukas Auer <lukas.auer@aisec.fraunhofer.de>; Anup > > > > > Patel <anup@brainfault.org>; Rick Chen <rick@andestech.com> > > > > > Subject: [PATCH 1/7] riscv: add infrastructure for calling functions > > > > > on other harts > > > > > > > > > > Harts on RISC-V boot independently and U-Boot is responsible for > > > > > managing them. Functions are called on other harts with > > > > > smp_call_function(), which sends inter-processor interrupts (IPIs) > > > > > to all other harts. > > > > > Functions are > > > > > specified with their address and two function arguments (argument 2 > > > > > and 3). > > > > > The first function argument is always the hart ID of the hart > > > > > calling the function. On the other harts, the IPI interrupt handler > > > > > handle_ipi() must be > > > > > called on software interrupts to handle the request and call the > > > > > specified function. > > > > > > > > > > Functions are stored in the ipi_data data structure. Every hart has > > > > > its own data structure in global data. While this is not required at > > > > > the moment (all harts are expected to boot Linux), this does allow > > > > > future expansion, where other harts may be used for monitoring or > > > > > other tasks. > > > > > > > > > > Signed-off-by: Lukas Auer <lukas.auer@aisec.fraunhofer.de> > > > > > --- > > > > > > > > > > arch/riscv/Kconfig | 19 +++++ > > > > > arch/riscv/include/asm/global_data.h | 5 ++ > > > > > arch/riscv/include/asm/smp.h | 53 +++++++++++++ > > > > > arch/riscv/lib/Makefile | 1 + > > > > > arch/riscv/lib/smp.c | 110 > > > > > +++++++++++++++++++++++++++ > > > > > 5 files changed, 188 insertions(+) > > > > > create mode 100644 arch/riscv/include/asm/smp.h create mode > > > > > 100644 > > > > > arch/riscv/lib/smp.c > > > > > > > > > > diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index > > > > > c45e4d73a8..c0842178dd 100644 > > > > > --- a/arch/riscv/Kconfig > > > > > +++ b/arch/riscv/Kconfig > > > > > @@ -116,4 +116,23 @@ config RISCV_RDTIME config SYS_MALLOC_F_LEN > > > > > default 0x1000 > > > > > > > > > > +config SMP > > > > > + bool "Symmetric Multi-Processing" > > > > > + help > > > > > + This enables support for systems with more than one CPU. If > > > > > + you say N here, U-Boot will run on single and multiprocessor > > > > > + machines, but will use only one CPU of a multiprocessor > > > > > + machine. If you say Y here, U-Boot will run on many, but not > > > > > + all, single processor machines. > > > > > + > > > > > +config NR_CPUS > > > > > + int "Maximum number of CPUs (2-32)" > > > > > + range 2 32 > > > > > + depends on SMP > > > > > + default "8" > > > > > + help > > > > > + On multiprocessor machines, U-Boot sets up a stack for each > > > > > CPU. > > > > > + Stack memory is pre-allocated. U-Boot must therefore know the > > > > > + maximum number of CPUs that may be present. > > > > > + > > > > > endmenu > > > > > diff --git a/arch/riscv/include/asm/global_data.h > > > > > b/arch/riscv/include/asm/global_data.h > > > > > index a3a342c6e1..23a5f35af5 100644 > > > > > --- a/arch/riscv/include/asm/global_data.h > > > > > +++ b/arch/riscv/include/asm/global_data.h > > > > > @@ -10,12 +10,17 @@ > > > > > #ifndef __ASM_GBL_DATA_H > > > > > #define __ASM_GBL_DATA_H > > > > > > > > > > +#include <asm/smp.h> > > > > > + > > > > > /* Architecture-specific global data */ struct arch_global_data { > > > > > long boot_hart; /* boot hart id */ > > > > > #ifdef CONFIG_SIFIVE_CLINT > > > > > void __iomem *clint; /* clint base address */ > > > > > #endif > > > > > +#ifdef CONFIG_SMP > > > > > + struct ipi_data ipi[CONFIG_NR_CPUS]; #endif > > > > > }; > > > > > > > > > > #include <asm-generic/global_data.h> > > > > > diff --git a/arch/riscv/include/asm/smp.h > > > > > b/arch/riscv/include/asm/smp.h > > > > > new file mode 100644 index 0000000000..bc863fdbaf > > > > > --- /dev/null > > > > > +++ b/arch/riscv/include/asm/smp.h > > > > > @@ -0,0 +1,53 @@ > > > > > +/* SPDX-License-Identifier: GPL-2.0 */ > > > > > +/* > > > > > + * Copyright (C) 2019 Fraunhofer AISEC, > > > > > + * Lukas Auer <lukas.auer@aisec.fraunhofer.de> */ > > > > > + > > > > > +#ifndef _ASM_RISCV_SMP_H > > > > > +#define _ASM_RISCV_SMP_H > > > > > + > > > > > +/** > > > > > + * struct ipi_data - Inter-processor interrupt (IPI) data > > > > > structure > > > > > + * > > > > > + * IPIs are used for SMP support to communicate to other harts > > > > > what > > > > > +function to > > > > > + * call. Functions are in the form > > > > > + * void (*addr)(ulong hart, ulong arg0, ulong arg1). > > > > > + * > > > > > + * The function address and the two arguments, arg0 and arg1, are > > > > > +stored in the > > > > > + * IPI data structure. The hart ID is inserted by the hart > > > > > handling the > > > > > +IPI and > > > > > + * calling the function. > > > > > + * > > > > > + * @addr: Address of function > > > > > + * @arg0: First argument of function > > > > > + * @arg1: Second argument of function > > > > > + */ > > > > > +struct ipi_data { > > > > > + ulong addr; > > > > > + ulong arg0; > > > > > + ulong arg1; > > > > > +}; > > > > > + > > > > > +/** > > > > > + * handle_ipi() - interrupt handler for software interrupts > > > > > + * > > > > > + * The IPI interrupt handler must be called to handle software > > > > > +interrupts. It > > > > > + * calls the function specified in the hart's IPI data structure. > > > > > + * > > > > > + * @hart: Hart ID of the current hart > > > > > + */ > > > > > +void handle_ipi(ulong hart); > > > > > + > > > > > +/** > > > > > + * smp_call_function() - Call a function on all other harts > > > > > + * > > > > > + * Send IPIs with the specified function call to all harts. > > > > > + * > > > > > + * @addr: Address of function > > > > > + * @arg0: First argument of function > > > > > + * @arg1: Second argument of function > > > > > + * @return 0 if OK, -ve on error > > > > > + */ > > > > > +int smp_call_function(ulong addr, ulong arg0, ulong arg1); > > > > > + > > > > > +#endif > > > > > diff --git a/arch/riscv/lib/Makefile b/arch/riscv/lib/Makefile > > > > > index > > > > > edfa61690c..19370f9749 100644 > > > > > --- a/arch/riscv/lib/Makefile > > > > > +++ b/arch/riscv/lib/Makefile > > > > > @@ -14,6 +14,7 @@ obj-$(CONFIG_SIFIVE_CLINT) += sifive_clint.o > > > > > obj-y += interrupts.o > > > > > obj-y += reset.o > > > > > obj-y += setjmp.o > > > > > +obj-$(CONFIG_SMP) += smp.o > > > > > > > > > > # For building EFI apps > > > > > CFLAGS_$(EFI_CRT0) := $(CFLAGS_EFI) > > > > > diff --git a/arch/riscv/lib/smp.c b/arch/riscv/lib/smp.c new file > > > > > mode 100644 > > > > > index 0000000000..1266a2a0ef > > > > > --- /dev/null > > > > > +++ b/arch/riscv/lib/smp.c > > > > > @@ -0,0 +1,110 @@ > > > > > +// SPDX-License-Identifier: GPL-2.0+ > > > > > +/* > > > > > + * Copyright (C) 2019 Fraunhofer AISEC, > > > > > + * Lukas Auer <lukas.auer@aisec.fraunhofer.de> */ > > > > > + > > > > > +#include <common.h> > > > > > +#include <dm.h> > > > > > +#include <asm/barrier.h> > > > > > +#include <asm/smp.h> > > > > > + > > > > > +DECLARE_GLOBAL_DATA_PTR; > > > > > + > > > > > +/** > > > > > + * riscv_send_ipi() - Send inter-processor interrupt (IPI) > > > > > + * > > > > > + * Platform code must provide this function. > > > > > + * > > > > > + * @hart: Hart ID of receiving hart > > > > > + * @return 0 if OK, -ve on error > > > > > + */ > > > > > +extern int riscv_send_ipi(int hart); > > > > > + > > > > > +/** > > > > > + * riscv_clear_ipi() - Clear inter-processor interrupt (IPI) > > > > > + * > > > > > + * Platform code must provide this function. > > > > > + * > > > > > + * @hart: Hart ID of hart to be cleared > > > > > + * @return 0 if OK, -ve on error > > > > > + */ > > > > > +extern int riscv_clear_ipi(int hart); > > > > > + > > > > > +static int send_ipi_many(struct ipi_data *ipi) { > > > > > + ofnode node, cpus; > > > > > + u32 reg; > > > > > + int ret; > > > > > + > > > > > + cpus = ofnode_path("/cpus"); > > > > > + if (!ofnode_valid(cpus)) { > > > > > + pr_err("Can't find cpus node!\n"); > > > > > + return -EINVAL; > > > > > + } > > > > > + > > > > > + ofnode_for_each_subnode(node, cpus) { > > > > > + if (!ofnode_is_available(node)) > > > > > + continue; > > > > > > > > It is not correct to assume that whatever CPUs are marked > > > > available will come online. It is possible that certain available > > > > CPUs failed to come online due HW failure. > > > > > > > > > > This was intended so that we don't send IPIs to harts, which have been > > > explicitly marked as disabled. > > > > > > > Better approach would be keep an atomic bitmask of HARTs > > > > that have entered U-Boot. All HARTs that enter U-Boot will > > > > update the atomic HART available bitmask. We send IPI only > > > > to HARTs that are available as-per atomic HART bitmask. > > > > > > > > > > I'm not sure if this is required in U-Boot, since we are not relying on > > > all harts to boot for U-Boot to function. We only try to boot all harts > > > listed as available in the device tree. > > > > > It may also need to get information from each cpu nodes. > The cpu nodes is important, they must be match with the bitmask, or it > maybe go wrong. > So the bitmask way seem not be necessary. Makes sense. How about considering both HART DT nodes availability and HART online bitmask when sending IPI ? My motivation behind HART online bitmask in U-Boot is because: 1. HART DT nodes marked available in DTB might actually be in unknown state in case of HW failure. It is commonly seen that as silicon ages or due to continuous heating of silicon new HW failures start showing up over-time. This means even HARTs can fail to come online over-time. 2. We are moving towards SBI v0.2 and SBI PM extensions. This means RISC-V SoCs that implement SBI PM extensions will only bring-up one HART at boot-time. Rest of the HARTs, will have to powerd-up using SBI calls. This means that even if HART DT node is marked available, the HART might be still be offline until someone from S-mode does SBI call to power-up HART. Of course, we will be handling in above cases in OpenSBI too but it is better to have "HART online bitmask" in U-Boot as well so that it works fine if some buggy proprietary SBI runtime firmware is not handling above cases. Regards, Anup ^ permalink raw reply [flat|nested] 45+ messages in thread
* [U-Boot] [PATCH 1/7] riscv: add infrastructure for calling functions on other harts 2019-02-18 3:40 ` Anup Patel @ 2019-02-18 9:53 ` Auer, Lukas 0 siblings, 0 replies; 45+ messages in thread From: Auer, Lukas @ 2019-02-18 9:53 UTC (permalink / raw) To: u-boot On Mon, 2019-02-18 at 09:10 +0530, Anup Patel wrote: > On Mon, Feb 18, 2019 at 8:53 AM Rick Chen <rickchen36@gmail.com> > wrote: > > <rick@andestech.com> 於 2019年2月18日 週一 上午11:00寫道: > > > > > > > > > > -----Original Message----- > > > > From: Auer, Lukas [mailto:lukas.auer at aisec.fraunhofer.de] > > > > Sent: Monday, February 18, 2019 5:55 AM > > > > To: u-boot at lists.denx.de; Anup.Patel at wdc.com > > > > Cc: anup at brainfault.org; bmeng.cn at gmail.com; schwab at suse.de; > > > > Rick Jian-Zhi > > > > Chen(陳建志); palmer at sifive.com; Atish.Patra at wdc.com; > > > > agraf at suse.de > > > > Subject: Re: [PATCH 1/7] riscv: add infrastructure for calling > > > > functions on other > > > > harts > > > > > > > > On Tue, 2019-02-12 at 01:44 +0000, Anup Patel wrote: > > > > > > -----Original Message----- > > > > > > From: Lukas Auer [mailto:lukas.auer at aisec.fraunhofer.de] > > > > > > Sent: Tuesday, February 12, 2019 3:44 AM > > > > > > To: u-boot at lists.denx.de > > > > > > Cc: Atish Patra <Atish.Patra@wdc.com>; Anup Patel > > > > > > <Anup.Patel@wdc.com>; Bin Meng <bmeng.cn@gmail.com>; > > > > > > Andreas > > > > Schwab > > > > > > <schwab@suse.de>; Palmer Dabbelt <palmer@sifive.com>; > > > > > > Alexander Graf > > > > > > <agraf@suse.de>; Lukas Auer <lukas.auer@aisec.fraunhofer.de > > > > > > >; Anup > > > > > > Patel <anup@brainfault.org>; Rick Chen <rick@andestech.com> > > > > > > Subject: [PATCH 1/7] riscv: add infrastructure for calling > > > > > > functions > > > > > > on other harts > > > > > > > > > > > > Harts on RISC-V boot independently and U-Boot is > > > > > > responsible for > > > > > > managing them. Functions are called on other harts with > > > > > > smp_call_function(), which sends inter-processor interrupts > > > > > > (IPIs) > > > > > > to all other harts. > > > > > > Functions are > > > > > > specified with their address and two function arguments > > > > > > (argument 2 > > > > > > and 3). > > > > > > The first function argument is always the hart ID of the > > > > > > hart > > > > > > calling the function. On the other harts, the IPI interrupt > > > > > > handler > > > > > > handle_ipi() must be > > > > > > called on software interrupts to handle the request and > > > > > > call the > > > > > > specified function. > > > > > > > > > > > > Functions are stored in the ipi_data data structure. Every > > > > > > hart has > > > > > > its own data structure in global data. While this is not > > > > > > required at > > > > > > the moment (all harts are expected to boot Linux), this > > > > > > does allow > > > > > > future expansion, where other harts may be used for > > > > > > monitoring or > > > > > > other tasks. > > > > > > > > > > > > Signed-off-by: Lukas Auer <lukas.auer@aisec.fraunhofer.de> > > > > > > --- > > > > > > > > > > > > arch/riscv/Kconfig | 19 +++++ > > > > > > arch/riscv/include/asm/global_data.h | 5 ++ > > > > > > arch/riscv/include/asm/smp.h | 53 +++++++++++++ > > > > > > arch/riscv/lib/Makefile | 1 + > > > > > > arch/riscv/lib/smp.c | 110 > > > > > > +++++++++++++++++++++++++++ > > > > > > 5 files changed, 188 insertions(+) > > > > > > create mode 100644 arch/riscv/include/asm/smp.h create > > > > > > mode > > > > > > 100644 > > > > > > arch/riscv/lib/smp.c > > > > > > > > > > > > diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index > > > > > > c45e4d73a8..c0842178dd 100644 > > > > > > --- a/arch/riscv/Kconfig > > > > > > +++ b/arch/riscv/Kconfig > > > > > > @@ -116,4 +116,23 @@ config RISCV_RDTIME config > > > > > > SYS_MALLOC_F_LEN > > > > > > default 0x1000 > > > > > > > > > > > > +config SMP > > > > > > + bool "Symmetric Multi-Processing" > > > > > > + help > > > > > > + This enables support for systems with more than one > > > > > > CPU. If > > > > > > + you say N here, U-Boot will run on single and > > > > > > multiprocessor > > > > > > + machines, but will use only one CPU of a multiprocessor > > > > > > + machine. If you say Y here, U-Boot will run on many, > > > > > > but not > > > > > > + all, single processor machines. > > > > > > + > > > > > > +config NR_CPUS > > > > > > + int "Maximum number of CPUs (2-32)" > > > > > > + range 2 32 > > > > > > + depends on SMP > > > > > > + default "8" > > > > > > + help > > > > > > + On multiprocessor machines, U-Boot sets up a stack for > > > > > > each > > > > > > CPU. > > > > > > + Stack memory is pre-allocated. U-Boot must therefore > > > > > > know the > > > > > > + maximum number of CPUs that may be present. > > > > > > + > > > > > > endmenu > > > > > > diff --git a/arch/riscv/include/asm/global_data.h > > > > > > b/arch/riscv/include/asm/global_data.h > > > > > > index a3a342c6e1..23a5f35af5 100644 > > > > > > --- a/arch/riscv/include/asm/global_data.h > > > > > > +++ b/arch/riscv/include/asm/global_data.h > > > > > > @@ -10,12 +10,17 @@ > > > > > > #ifndef __ASM_GBL_DATA_H > > > > > > #define __ASM_GBL_DATA_H > > > > > > > > > > > > +#include <asm/smp.h> > > > > > > + > > > > > > /* Architecture-specific global data */ struct > > > > > > arch_global_data { > > > > > > long boot_hart; /* boot hart id */ > > > > > > #ifdef CONFIG_SIFIVE_CLINT > > > > > > void __iomem *clint; /* clint base address */ > > > > > > #endif > > > > > > +#ifdef CONFIG_SMP > > > > > > + struct ipi_data ipi[CONFIG_NR_CPUS]; #endif > > > > > > }; > > > > > > > > > > > > #include <asm-generic/global_data.h> > > > > > > diff --git a/arch/riscv/include/asm/smp.h > > > > > > b/arch/riscv/include/asm/smp.h > > > > > > new file mode 100644 index 0000000000..bc863fdbaf > > > > > > --- /dev/null > > > > > > +++ b/arch/riscv/include/asm/smp.h > > > > > > @@ -0,0 +1,53 @@ > > > > > > +/* SPDX-License-Identifier: GPL-2.0 */ > > > > > > +/* > > > > > > + * Copyright (C) 2019 Fraunhofer AISEC, > > > > > > + * Lukas Auer <lukas.auer@aisec.fraunhofer.de> */ > > > > > > + > > > > > > +#ifndef _ASM_RISCV_SMP_H > > > > > > +#define _ASM_RISCV_SMP_H > > > > > > + > > > > > > +/** > > > > > > + * struct ipi_data - Inter-processor interrupt (IPI) data > > > > > > structure > > > > > > + * > > > > > > + * IPIs are used for SMP support to communicate to other > > > > > > harts > > > > > > what > > > > > > +function to > > > > > > + * call. Functions are in the form > > > > > > + * void (*addr)(ulong hart, ulong arg0, ulong arg1). > > > > > > + * > > > > > > + * The function address and the two arguments, arg0 and > > > > > > arg1, are > > > > > > +stored in the > > > > > > + * IPI data structure. The hart ID is inserted by the hart > > > > > > handling the > > > > > > +IPI and > > > > > > + * calling the function. > > > > > > + * > > > > > > + * @addr: Address of function > > > > > > + * @arg0: First argument of function > > > > > > + * @arg1: Second argument of function > > > > > > + */ > > > > > > +struct ipi_data { > > > > > > + ulong addr; > > > > > > + ulong arg0; > > > > > > + ulong arg1; > > > > > > +}; > > > > > > + > > > > > > +/** > > > > > > + * handle_ipi() - interrupt handler for software > > > > > > interrupts > > > > > > + * > > > > > > + * The IPI interrupt handler must be called to handle > > > > > > software > > > > > > +interrupts. It > > > > > > + * calls the function specified in the hart's IPI data > > > > > > structure. > > > > > > + * > > > > > > + * @hart: Hart ID of the current hart > > > > > > + */ > > > > > > +void handle_ipi(ulong hart); > > > > > > + > > > > > > +/** > > > > > > + * smp_call_function() - Call a function on all other > > > > > > harts > > > > > > + * > > > > > > + * Send IPIs with the specified function call to all > > > > > > harts. > > > > > > + * > > > > > > + * @addr: Address of function > > > > > > + * @arg0: First argument of function > > > > > > + * @arg1: Second argument of function > > > > > > + * @return 0 if OK, -ve on error > > > > > > + */ > > > > > > +int smp_call_function(ulong addr, ulong arg0, ulong arg1); > > > > > > + > > > > > > +#endif > > > > > > diff --git a/arch/riscv/lib/Makefile > > > > > > b/arch/riscv/lib/Makefile > > > > > > index > > > > > > edfa61690c..19370f9749 100644 > > > > > > --- a/arch/riscv/lib/Makefile > > > > > > +++ b/arch/riscv/lib/Makefile > > > > > > @@ -14,6 +14,7 @@ obj-$(CONFIG_SIFIVE_CLINT) += > > > > > > sifive_clint.o > > > > > > obj-y += interrupts.o > > > > > > obj-y += reset.o > > > > > > obj-y += setjmp.o > > > > > > +obj-$(CONFIG_SMP) += smp.o > > > > > > > > > > > > # For building EFI apps > > > > > > CFLAGS_$(EFI_CRT0) := $(CFLAGS_EFI) > > > > > > diff --git a/arch/riscv/lib/smp.c b/arch/riscv/lib/smp.c > > > > > > new file > > > > > > mode 100644 > > > > > > index 0000000000..1266a2a0ef > > > > > > --- /dev/null > > > > > > +++ b/arch/riscv/lib/smp.c > > > > > > @@ -0,0 +1,110 @@ > > > > > > +// SPDX-License-Identifier: GPL-2.0+ > > > > > > +/* > > > > > > + * Copyright (C) 2019 Fraunhofer AISEC, > > > > > > + * Lukas Auer <lukas.auer@aisec.fraunhofer.de> */ > > > > > > + > > > > > > +#include <common.h> > > > > > > +#include <dm.h> > > > > > > +#include <asm/barrier.h> > > > > > > +#include <asm/smp.h> > > > > > > + > > > > > > +DECLARE_GLOBAL_DATA_PTR; > > > > > > + > > > > > > +/** > > > > > > + * riscv_send_ipi() - Send inter-processor interrupt (IPI) > > > > > > + * > > > > > > + * Platform code must provide this function. > > > > > > + * > > > > > > + * @hart: Hart ID of receiving hart > > > > > > + * @return 0 if OK, -ve on error > > > > > > + */ > > > > > > +extern int riscv_send_ipi(int hart); > > > > > > + > > > > > > +/** > > > > > > + * riscv_clear_ipi() - Clear inter-processor interrupt > > > > > > (IPI) > > > > > > + * > > > > > > + * Platform code must provide this function. > > > > > > + * > > > > > > + * @hart: Hart ID of hart to be cleared > > > > > > + * @return 0 if OK, -ve on error > > > > > > + */ > > > > > > +extern int riscv_clear_ipi(int hart); > > > > > > + > > > > > > +static int send_ipi_many(struct ipi_data *ipi) { > > > > > > + ofnode node, cpus; > > > > > > + u32 reg; > > > > > > + int ret; > > > > > > + > > > > > > + cpus = ofnode_path("/cpus"); > > > > > > + if (!ofnode_valid(cpus)) { > > > > > > + pr_err("Can't find cpus node!\n"); > > > > > > + return -EINVAL; > > > > > > + } > > > > > > + > > > > > > + ofnode_for_each_subnode(node, cpus) { > > > > > > + if (!ofnode_is_available(node)) > > > > > > + continue; > > > > > > > > > > It is not correct to assume that whatever CPUs are marked > > > > > available will come online. It is possible that certain > > > > > available > > > > > CPUs failed to come online due HW failure. > > > > > > > > > > > > > This was intended so that we don't send IPIs to harts, which > > > > have been > > > > explicitly marked as disabled. > > > > > > > > > Better approach would be keep an atomic bitmask of HARTs > > > > > that have entered U-Boot. All HARTs that enter U-Boot will > > > > > update the atomic HART available bitmask. We send IPI only > > > > > to HARTs that are available as-per atomic HART bitmask. > > > > > > > > > > > > > I'm not sure if this is required in U-Boot, since we are not > > > > relying on > > > > all harts to boot for U-Boot to function. We only try to boot > > > > all harts > > > > listed as available in the device tree. > > > > > > > > It may also need to get information from each cpu nodes. > > The cpu nodes is important, they must be match with the bitmask, or > > it > > maybe go wrong. > > So the bitmask way seem not be necessary. > > Makes sense. How about considering both HART DT nodes availability > and HART online bitmask when sending IPI ? > > My motivation behind HART online bitmask in U-Boot is because: > > 1. HART DT nodes marked available in DTB might actually be in unknown > state in case of HW failure. It is commonly seen that as silicon ages > or > due to continuous heating of silicon new HW failures start showing up > over-time. This means even HARTs can fail to come online over-time. > > 2. We are moving towards SBI v0.2 and SBI PM extensions. This means > RISC-V SoCs that implement SBI PM extensions will only bring-up one > HART at boot-time. Rest of the HARTs, will have to powerd-up using > SBI calls. This means that even if HART DT node is marked available, > the HART might be still be offline until someone from S-mode does > SBI call to power-up HART. > > Of course, we will be handling in above cases in OpenSBI too but it > is better to have "HART online bitmask" in U-Boot as well so that it > works fine if some buggy proprietary SBI runtime firmware is not > handling above cases. > Thanks for explaining! I agree, especially for case 2 we will need to handle offline harts. I'll integrate a hart online bitmask in the next version. Thanks, Lukas ^ permalink raw reply [flat|nested] 45+ messages in thread
* [U-Boot] [PATCH 1/7] riscv: add infrastructure for calling functions on other harts 2019-02-11 22:13 ` [U-Boot] [PATCH 1/7] riscv: add infrastructure for calling functions on other harts Lukas Auer 2019-02-12 1:44 ` Anup Patel @ 2019-02-12 3:03 ` Bin Meng 2019-02-17 22:00 ` Auer, Lukas 2019-02-18 4:58 ` Anup Patel 2 siblings, 1 reply; 45+ messages in thread From: Bin Meng @ 2019-02-12 3:03 UTC (permalink / raw) To: u-boot Hi Lukas, On Tue, Feb 12, 2019 at 6:14 AM Lukas Auer <lukas.auer@aisec.fraunhofer.de> wrote: > > Harts on RISC-V boot independently and U-Boot is responsible for > managing them. Functions are called on other harts with > smp_call_function(), which sends inter-processor interrupts (IPIs) to > all other harts. Functions are specified with their address and two > function arguments (argument 2 and 3). The first function argument is > always the hart ID of the hart calling the function. On the other harts, > the IPI interrupt handler handle_ipi() must be called on software > interrupts to handle the request and call the specified function. > > Functions are stored in the ipi_data data structure. Every hart has its > own data structure in global data. While this is not required at the > moment (all harts are expected to boot Linux), this does allow future > expansion, where other harts may be used for monitoring or other tasks. > > Signed-off-by: Lukas Auer <lukas.auer@aisec.fraunhofer.de> > --- > > arch/riscv/Kconfig | 19 +++++ > arch/riscv/include/asm/global_data.h | 5 ++ > arch/riscv/include/asm/smp.h | 53 +++++++++++++ > arch/riscv/lib/Makefile | 1 + > arch/riscv/lib/smp.c | 110 +++++++++++++++++++++++++++ > 5 files changed, 188 insertions(+) > create mode 100644 arch/riscv/include/asm/smp.h > create mode 100644 arch/riscv/lib/smp.c > Looks pretty clean to me. Thanks! Some nits below. > diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig > index c45e4d73a8..c0842178dd 100644 > --- a/arch/riscv/Kconfig > +++ b/arch/riscv/Kconfig > @@ -116,4 +116,23 @@ config RISCV_RDTIME > config SYS_MALLOC_F_LEN > default 0x1000 > > +config SMP > + bool "Symmetric Multi-Processing" > + help > + This enables support for systems with more than one CPU. If > + you say N here, U-Boot will run on single and multiprocessor > + machines, but will use only one CPU of a multiprocessor > + machine. If you say Y here, U-Boot will run on many, but not > + all, single processor machines. U-Boot will run on both single and multiprocessor machines? > + > +config NR_CPUS > + int "Maximum number of CPUs (2-32)" > + range 2 32 > + depends on SMP > + default "8" no quotation mark? > + help > + On multiprocessor machines, U-Boot sets up a stack for each CPU. > + Stack memory is pre-allocated. U-Boot must therefore know the > + maximum number of CPUs that may be present. > + > endmenu > diff --git a/arch/riscv/include/asm/global_data.h b/arch/riscv/include/asm/global_data.h > index a3a342c6e1..23a5f35af5 100644 > --- a/arch/riscv/include/asm/global_data.h > +++ b/arch/riscv/include/asm/global_data.h > @@ -10,12 +10,17 @@ > #ifndef __ASM_GBL_DATA_H > #define __ASM_GBL_DATA_H > > +#include <asm/smp.h> > + > /* Architecture-specific global data */ > struct arch_global_data { > long boot_hart; /* boot hart id */ > #ifdef CONFIG_SIFIVE_CLINT > void __iomem *clint; /* clint base address */ > #endif > +#ifdef CONFIG_SMP > + struct ipi_data ipi[CONFIG_NR_CPUS]; > +#endif > }; > > #include <asm-generic/global_data.h> > diff --git a/arch/riscv/include/asm/smp.h b/arch/riscv/include/asm/smp.h > new file mode 100644 > index 0000000000..bc863fdbaf > --- /dev/null > +++ b/arch/riscv/include/asm/smp.h > @@ -0,0 +1,53 @@ > +/* SPDX-License-Identifier: GPL-2.0 */ > +/* > + * Copyright (C) 2019 Fraunhofer AISEC, > + * Lukas Auer <lukas.auer@aisec.fraunhofer.de> > + */ > + > +#ifndef _ASM_RISCV_SMP_H > +#define _ASM_RISCV_SMP_H > + > +/** > + * struct ipi_data - Inter-processor interrupt (IPI) data structure > + * > + * IPIs are used for SMP support to communicate to other harts what function to > + * call. Functions are in the form > + * void (*addr)(ulong hart, ulong arg0, ulong arg1). > + * > + * The function address and the two arguments, arg0 and arg1, are stored in the > + * IPI data structure. The hart ID is inserted by the hart handling the IPI and > + * calling the function. > + * > + * @addr: Address of function > + * @arg0: First argument of function > + * @arg1: Second argument of function > + */ > +struct ipi_data { > + ulong addr; > + ulong arg0; > + ulong arg1; > +}; > + > +/** > + * handle_ipi() - interrupt handler for software interrupts > + * > + * The IPI interrupt handler must be called to handle software interrupts. It > + * calls the function specified in the hart's IPI data structure. > + * > + * @hart: Hart ID of the current hart > + */ > +void handle_ipi(ulong hart); > + > +/** > + * smp_call_function() - Call a function on all other harts > + * > + * Send IPIs with the specified function call to all harts. > + * > + * @addr: Address of function > + * @arg0: First argument of function > + * @arg1: Second argument of function > + * @return 0 if OK, -ve on error > + */ > +int smp_call_function(ulong addr, ulong arg0, ulong arg1); > + > +#endif > diff --git a/arch/riscv/lib/Makefile b/arch/riscv/lib/Makefile > index edfa61690c..19370f9749 100644 > --- a/arch/riscv/lib/Makefile > +++ b/arch/riscv/lib/Makefile > @@ -14,6 +14,7 @@ obj-$(CONFIG_SIFIVE_CLINT) += sifive_clint.o > obj-y += interrupts.o > obj-y += reset.o > obj-y += setjmp.o > +obj-$(CONFIG_SMP) += smp.o > > # For building EFI apps > CFLAGS_$(EFI_CRT0) := $(CFLAGS_EFI) > diff --git a/arch/riscv/lib/smp.c b/arch/riscv/lib/smp.c > new file mode 100644 > index 0000000000..1266a2a0ef > --- /dev/null > +++ b/arch/riscv/lib/smp.c > @@ -0,0 +1,110 @@ > +// SPDX-License-Identifier: GPL-2.0+ > +/* > + * Copyright (C) 2019 Fraunhofer AISEC, > + * Lukas Auer <lukas.auer@aisec.fraunhofer.de> > + */ > + > +#include <common.h> > +#include <dm.h> > +#include <asm/barrier.h> > +#include <asm/smp.h> > + > +DECLARE_GLOBAL_DATA_PTR; > + > +/** > + * riscv_send_ipi() - Send inter-processor interrupt (IPI) > + * > + * Platform code must provide this function. > + * > + * @hart: Hart ID of receiving hart > + * @return 0 if OK, -ve on error > + */ > +extern int riscv_send_ipi(int hart); > + > +/** > + * riscv_clear_ipi() - Clear inter-processor interrupt (IPI) > + * > + * Platform code must provide this function. > + * > + * @hart: Hart ID of hart to be cleared > + * @return 0 if OK, -ve on error > + */ > +extern int riscv_clear_ipi(int hart); > + > +static int send_ipi_many(struct ipi_data *ipi) > +{ > + ofnode node, cpus; > + u32 reg; > + int ret; > + > + cpus = ofnode_path("/cpus"); > + if (!ofnode_valid(cpus)) { > + pr_err("Can't find cpus node!\n"); > + return -EINVAL; > + } > + > + ofnode_for_each_subnode(node, cpus) { > + if (!ofnode_is_available(node)) > + continue; > + > + /* read hart ID of CPU */ > + ret = ofnode_read_u32(node, "reg", ®); > + if (ret) > + continue; > + > + /* skip if it is the hart we are running on */ > + if (reg == gd->arch.boot_hart) > + continue; > + > + if (reg >= CONFIG_NR_CPUS) { > + pr_err("Hart ID %d is out of range, increase CONFIG_NR_CPUS\n", > + reg); > + continue; > + } > + > + gd->arch.ipi[reg].addr = ipi->addr; > + gd->arch.ipi[reg].arg0 = ipi->arg0; > + gd->arch.ipi[reg].arg1 = ipi->arg1; > + mb(); > + > + ret = riscv_send_ipi(reg); > + if (ret) > + return ret; > + } > + > + return 0; > +} > + > +void handle_ipi(ulong hart) > +{ > + int ret; > + void (*smp_function)(ulong hart, ulong arg0, ulong arg1); > + > + if (hart >= CONFIG_NR_CPUS) > + return; > + > + ret = riscv_clear_ipi(hart); > + if (ret) { > + pr_err("Cannot clear IPI\n"); > + return; > + } > + > + smp_function = (void (*)(ulong, ulong, ulong))gd->arch.ipi[hart].addr; > + invalidate_icache_all(); > + > + smp_function(hart, gd->arch.ipi[hart].arg0, gd->arch.ipi[hart].arg1); > +} > + > +int smp_call_function(ulong addr, ulong arg0, ulong arg1) > +{ > + int ret = 0; > + struct ipi_data ipi; > + > + ipi.addr = addr; > + ipi.arg0 = arg0; > + ipi.arg1 = arg1; > + > + ret = send_ipi_many(&ipi); > + > + return ret; > +} > -- Regards, Bin ^ permalink raw reply [flat|nested] 45+ messages in thread
* [U-Boot] [PATCH 1/7] riscv: add infrastructure for calling functions on other harts 2019-02-12 3:03 ` Bin Meng @ 2019-02-17 22:00 ` Auer, Lukas 0 siblings, 0 replies; 45+ messages in thread From: Auer, Lukas @ 2019-02-17 22:00 UTC (permalink / raw) To: u-boot Hi Bin, On Tue, 2019-02-12 at 11:03 +0800, Bin Meng wrote: > Hi Lukas, > > On Tue, Feb 12, 2019 at 6:14 AM Lukas Auer > <lukas.auer@aisec.fraunhofer.de> wrote: > > Harts on RISC-V boot independently and U-Boot is responsible for > > managing them. Functions are called on other harts with > > smp_call_function(), which sends inter-processor interrupts (IPIs) > > to > > all other harts. Functions are specified with their address and two > > function arguments (argument 2 and 3). The first function argument > > is > > always the hart ID of the hart calling the function. On the other > > harts, > > the IPI interrupt handler handle_ipi() must be called on software > > interrupts to handle the request and call the specified function. > > > > Functions are stored in the ipi_data data structure. Every hart has > > its > > own data structure in global data. While this is not required at > > the > > moment (all harts are expected to boot Linux), this does allow > > future > > expansion, where other harts may be used for monitoring or other > > tasks. > > > > Signed-off-by: Lukas Auer <lukas.auer@aisec.fraunhofer.de> > > --- > > > > arch/riscv/Kconfig | 19 +++++ > > arch/riscv/include/asm/global_data.h | 5 ++ > > arch/riscv/include/asm/smp.h | 53 +++++++++++++ > > arch/riscv/lib/Makefile | 1 + > > arch/riscv/lib/smp.c | 110 > > +++++++++++++++++++++++++++ > > 5 files changed, 188 insertions(+) > > create mode 100644 arch/riscv/include/asm/smp.h > > create mode 100644 arch/riscv/lib/smp.c > > > > Looks pretty clean to me. Thanks! Some nits below. > Thank you, and thanks for your review! > > diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig > > index c45e4d73a8..c0842178dd 100644 > > --- a/arch/riscv/Kconfig > > +++ b/arch/riscv/Kconfig > > @@ -116,4 +116,23 @@ config RISCV_RDTIME > > config SYS_MALLOC_F_LEN > > default 0x1000 > > > > +config SMP > > + bool "Symmetric Multi-Processing" > > + help > > + This enables support for systems with more than one CPU. > > If > > + you say N here, U-Boot will run on single and > > multiprocessor > > + machines, but will use only one CPU of a multiprocessor > > + machine. If you say Y here, U-Boot will run on many, but > > not > > + all, single processor machines. > > U-Boot will run on both single and multiprocessor machines? > I simply adapted the help text used in the Linux kernel here. Should I rephrase it? > > + > > +config NR_CPUS > > + int "Maximum number of CPUs (2-32)" > > + range 2 32 > > + depends on SMP > > + default "8" > > no quotation mark? > Yes, I will remove them. Thanks, Lukas > > + help > > + On multiprocessor machines, U-Boot sets up a stack for > > each CPU. > > + Stack memory is pre-allocated. U-Boot must therefore know > > the > > + maximum number of CPUs that may be present. > > + > > endmenu > > diff --git a/arch/riscv/include/asm/global_data.h > > b/arch/riscv/include/asm/global_data.h > > index a3a342c6e1..23a5f35af5 100644 > > --- a/arch/riscv/include/asm/global_data.h > > +++ b/arch/riscv/include/asm/global_data.h > > @@ -10,12 +10,17 @@ > > #ifndef __ASM_GBL_DATA_H > > #define __ASM_GBL_DATA_H > > > > +#include <asm/smp.h> > > + > > /* Architecture-specific global data */ > > struct arch_global_data { > > long boot_hart; /* boot hart id */ > > #ifdef CONFIG_SIFIVE_CLINT > > void __iomem *clint; /* clint base address */ > > #endif > > +#ifdef CONFIG_SMP > > + struct ipi_data ipi[CONFIG_NR_CPUS]; > > +#endif > > }; > > > > #include <asm-generic/global_data.h> > > diff --git a/arch/riscv/include/asm/smp.h > > b/arch/riscv/include/asm/smp.h > > new file mode 100644 > > index 0000000000..bc863fdbaf > > --- /dev/null > > +++ b/arch/riscv/include/asm/smp.h > > @@ -0,0 +1,53 @@ > > +/* SPDX-License-Identifier: GPL-2.0 */ > > +/* > > + * Copyright (C) 2019 Fraunhofer AISEC, > > + * Lukas Auer <lukas.auer@aisec.fraunhofer.de> > > + */ > > + > > +#ifndef _ASM_RISCV_SMP_H > > +#define _ASM_RISCV_SMP_H > > + > > +/** > > + * struct ipi_data - Inter-processor interrupt (IPI) data > > structure > > + * > > + * IPIs are used for SMP support to communicate to other harts > > what function to > > + * call. Functions are in the form > > + * void (*addr)(ulong hart, ulong arg0, ulong arg1). > > + * > > + * The function address and the two arguments, arg0 and arg1, are > > stored in the > > + * IPI data structure. The hart ID is inserted by the hart > > handling the IPI and > > + * calling the function. > > + * > > + * @addr: Address of function > > + * @arg0: First argument of function > > + * @arg1: Second argument of function > > + */ > > +struct ipi_data { > > + ulong addr; > > + ulong arg0; > > + ulong arg1; > > +}; > > + > > +/** > > + * handle_ipi() - interrupt handler for software interrupts > > + * > > + * The IPI interrupt handler must be called to handle software > > interrupts. It > > + * calls the function specified in the hart's IPI data structure. > > + * > > + * @hart: Hart ID of the current hart > > + */ > > +void handle_ipi(ulong hart); > > + > > +/** > > + * smp_call_function() - Call a function on all other harts > > + * > > + * Send IPIs with the specified function call to all harts. > > + * > > + * @addr: Address of function > > + * @arg0: First argument of function > > + * @arg1: Second argument of function > > + * @return 0 if OK, -ve on error > > + */ > > +int smp_call_function(ulong addr, ulong arg0, ulong arg1); > > + > > +#endif > > diff --git a/arch/riscv/lib/Makefile b/arch/riscv/lib/Makefile > > index edfa61690c..19370f9749 100644 > > --- a/arch/riscv/lib/Makefile > > +++ b/arch/riscv/lib/Makefile > > @@ -14,6 +14,7 @@ obj-$(CONFIG_SIFIVE_CLINT) += sifive_clint.o > > obj-y += interrupts.o > > obj-y += reset.o > > obj-y += setjmp.o > > +obj-$(CONFIG_SMP) += smp.o > > > > # For building EFI apps > > CFLAGS_$(EFI_CRT0) := $(CFLAGS_EFI) > > diff --git a/arch/riscv/lib/smp.c b/arch/riscv/lib/smp.c > > new file mode 100644 > > index 0000000000..1266a2a0ef > > --- /dev/null > > +++ b/arch/riscv/lib/smp.c > > @@ -0,0 +1,110 @@ > > +// SPDX-License-Identifier: GPL-2.0+ > > +/* > > + * Copyright (C) 2019 Fraunhofer AISEC, > > + * Lukas Auer <lukas.auer@aisec.fraunhofer.de> > > + */ > > + > > +#include <common.h> > > +#include <dm.h> > > +#include <asm/barrier.h> > > +#include <asm/smp.h> > > + > > +DECLARE_GLOBAL_DATA_PTR; > > + > > +/** > > + * riscv_send_ipi() - Send inter-processor interrupt (IPI) > > + * > > + * Platform code must provide this function. > > + * > > + * @hart: Hart ID of receiving hart > > + * @return 0 if OK, -ve on error > > + */ > > +extern int riscv_send_ipi(int hart); > > + > > +/** > > + * riscv_clear_ipi() - Clear inter-processor interrupt (IPI) > > + * > > + * Platform code must provide this function. > > + * > > + * @hart: Hart ID of hart to be cleared > > + * @return 0 if OK, -ve on error > > + */ > > +extern int riscv_clear_ipi(int hart); > > + > > +static int send_ipi_many(struct ipi_data *ipi) > > +{ > > + ofnode node, cpus; > > + u32 reg; > > + int ret; > > + > > + cpus = ofnode_path("/cpus"); > > + if (!ofnode_valid(cpus)) { > > + pr_err("Can't find cpus node!\n"); > > + return -EINVAL; > > + } > > + > > + ofnode_for_each_subnode(node, cpus) { > > + if (!ofnode_is_available(node)) > > + continue; > > + > > + /* read hart ID of CPU */ > > + ret = ofnode_read_u32(node, "reg", ®); > > + if (ret) > > + continue; > > + > > + /* skip if it is the hart we are running on */ > > + if (reg == gd->arch.boot_hart) > > + continue; > > + > > + if (reg >= CONFIG_NR_CPUS) { > > + pr_err("Hart ID %d is out of range, > > increase CONFIG_NR_CPUS\n", > > + reg); > > + continue; > > + } > > + > > + gd->arch.ipi[reg].addr = ipi->addr; > > + gd->arch.ipi[reg].arg0 = ipi->arg0; > > + gd->arch.ipi[reg].arg1 = ipi->arg1; > > + mb(); > > + > > + ret = riscv_send_ipi(reg); > > + if (ret) > > + return ret; > > + } > > + > > + return 0; > > +} > > + > > +void handle_ipi(ulong hart) > > +{ > > + int ret; > > + void (*smp_function)(ulong hart, ulong arg0, ulong arg1); > > + > > + if (hart >= CONFIG_NR_CPUS) > > + return; > > + > > + ret = riscv_clear_ipi(hart); > > + if (ret) { > > + pr_err("Cannot clear IPI\n"); > > + return; > > + } > > + > > + smp_function = (void (*)(ulong, ulong, ulong))gd- > > >arch.ipi[hart].addr; > > + invalidate_icache_all(); > > + > > + smp_function(hart, gd->arch.ipi[hart].arg0, gd- > > >arch.ipi[hart].arg1); > > +} > > + > > +int smp_call_function(ulong addr, ulong arg0, ulong arg1) > > +{ > > + int ret = 0; > > + struct ipi_data ipi; > > + > > + ipi.addr = addr; > > + ipi.arg0 = arg0; > > + ipi.arg1 = arg1; > > + > > + ret = send_ipi_many(&ipi); > > + > > + return ret; > > +} > > -- > > Regards, > Bin ^ permalink raw reply [flat|nested] 45+ messages in thread
* [U-Boot] [PATCH 1/7] riscv: add infrastructure for calling functions on other harts 2019-02-11 22:13 ` [U-Boot] [PATCH 1/7] riscv: add infrastructure for calling functions on other harts Lukas Auer 2019-02-12 1:44 ` Anup Patel 2019-02-12 3:03 ` Bin Meng @ 2019-02-18 4:58 ` Anup Patel 2019-02-18 10:01 ` Auer, Lukas 2 siblings, 1 reply; 45+ messages in thread From: Anup Patel @ 2019-02-18 4:58 UTC (permalink / raw) To: u-boot On Tue, Feb 12, 2019 at 3:44 AM Lukas Auer <lukas.auer@aisec.fraunhofer.de> wrote: > > Harts on RISC-V boot independently and U-Boot is responsible for > managing them. Functions are called on other harts with > smp_call_function(), which sends inter-processor interrupts (IPIs) to > all other harts. Functions are specified with their address and two > function arguments (argument 2 and 3). The first function argument is > always the hart ID of the hart calling the function. On the other harts, > the IPI interrupt handler handle_ipi() must be called on software > interrupts to handle the request and call the specified function. > > Functions are stored in the ipi_data data structure. Every hart has its > own data structure in global data. While this is not required at the > moment (all harts are expected to boot Linux), this does allow future > expansion, where other harts may be used for monitoring or other tasks. > > Signed-off-by: Lukas Auer <lukas.auer@aisec.fraunhofer.de> > --- > > arch/riscv/Kconfig | 19 +++++ > arch/riscv/include/asm/global_data.h | 5 ++ > arch/riscv/include/asm/smp.h | 53 +++++++++++++ > arch/riscv/lib/Makefile | 1 + > arch/riscv/lib/smp.c | 110 +++++++++++++++++++++++++++ > 5 files changed, 188 insertions(+) > create mode 100644 arch/riscv/include/asm/smp.h > create mode 100644 arch/riscv/lib/smp.c > > diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig > index c45e4d73a8..c0842178dd 100644 > --- a/arch/riscv/Kconfig > +++ b/arch/riscv/Kconfig > @@ -116,4 +116,23 @@ config RISCV_RDTIME > config SYS_MALLOC_F_LEN > default 0x1000 > > +config SMP > + bool "Symmetric Multi-Processing" > + help > + This enables support for systems with more than one CPU. If > + you say N here, U-Boot will run on single and multiprocessor > + machines, but will use only one CPU of a multiprocessor > + machine. If you say Y here, U-Boot will run on many, but not > + all, single processor machines. > + > +config NR_CPUS > + int "Maximum number of CPUs (2-32)" > + range 2 32 > + depends on SMP > + default "8" > + help > + On multiprocessor machines, U-Boot sets up a stack for each CPU. > + Stack memory is pre-allocated. U-Boot must therefore know the > + maximum number of CPUs that may be present. > + > endmenu > diff --git a/arch/riscv/include/asm/global_data.h b/arch/riscv/include/asm/global_data.h > index a3a342c6e1..23a5f35af5 100644 > --- a/arch/riscv/include/asm/global_data.h > +++ b/arch/riscv/include/asm/global_data.h > @@ -10,12 +10,17 @@ > #ifndef __ASM_GBL_DATA_H > #define __ASM_GBL_DATA_H > > +#include <asm/smp.h> > + > /* Architecture-specific global data */ > struct arch_global_data { > long boot_hart; /* boot hart id */ > #ifdef CONFIG_SIFIVE_CLINT > void __iomem *clint; /* clint base address */ > #endif > +#ifdef CONFIG_SMP > + struct ipi_data ipi[CONFIG_NR_CPUS]; The data passed by "main HART" via global data to other HARTs is not visible reliably and randomly few HARTs fail to enter Linux kernel on my board. I am suspecting per-HART "ipi" data in global data not being cache-line aligned as the cause of behavior but there could be other issue too. I have a hack which works reliable for me. As-per this hack, we add "mdelay(10)" just before calling riscv_send_ipi() in send_ipi_many(). This hack helped me conclude that there is some sync issue in per-HART "ipi" data. The above issue is not seen on QEMU so we are fine there. I would suggest to make per-HART "ipi" data cache-line aligned (just like Linux kernel). Regards, Anup ^ permalink raw reply [flat|nested] 45+ messages in thread
* [U-Boot] [PATCH 1/7] riscv: add infrastructure for calling functions on other harts 2019-02-18 4:58 ` Anup Patel @ 2019-02-18 10:01 ` Auer, Lukas 2019-02-18 11:41 ` Auer, Lukas 0 siblings, 1 reply; 45+ messages in thread From: Auer, Lukas @ 2019-02-18 10:01 UTC (permalink / raw) To: u-boot On Mon, 2019-02-18 at 10:28 +0530, Anup Patel wrote: > On Tue, Feb 12, 2019 at 3:44 AM Lukas Auer > <lukas.auer@aisec.fraunhofer.de> wrote: > > Harts on RISC-V boot independently and U-Boot is responsible for > > managing them. Functions are called on other harts with > > smp_call_function(), which sends inter-processor interrupts (IPIs) > > to > > all other harts. Functions are specified with their address and two > > function arguments (argument 2 and 3). The first function argument > > is > > always the hart ID of the hart calling the function. On the other > > harts, > > the IPI interrupt handler handle_ipi() must be called on software > > interrupts to handle the request and call the specified function. > > > > Functions are stored in the ipi_data data structure. Every hart has > > its > > own data structure in global data. While this is not required at > > the > > moment (all harts are expected to boot Linux), this does allow > > future > > expansion, where other harts may be used for monitoring or other > > tasks. > > > > Signed-off-by: Lukas Auer <lukas.auer@aisec.fraunhofer.de> > > --- > > > > arch/riscv/Kconfig | 19 +++++ > > arch/riscv/include/asm/global_data.h | 5 ++ > > arch/riscv/include/asm/smp.h | 53 +++++++++++++ > > arch/riscv/lib/Makefile | 1 + > > arch/riscv/lib/smp.c | 110 > > +++++++++++++++++++++++++++ > > 5 files changed, 188 insertions(+) > > create mode 100644 arch/riscv/include/asm/smp.h > > create mode 100644 arch/riscv/lib/smp.c > > > > diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig > > index c45e4d73a8..c0842178dd 100644 > > --- a/arch/riscv/Kconfig > > +++ b/arch/riscv/Kconfig > > @@ -116,4 +116,23 @@ config RISCV_RDTIME > > config SYS_MALLOC_F_LEN > > default 0x1000 > > > > +config SMP > > + bool "Symmetric Multi-Processing" > > + help > > + This enables support for systems with more than one CPU. > > If > > + you say N here, U-Boot will run on single and > > multiprocessor > > + machines, but will use only one CPU of a multiprocessor > > + machine. If you say Y here, U-Boot will run on many, but > > not > > + all, single processor machines. > > + > > +config NR_CPUS > > + int "Maximum number of CPUs (2-32)" > > + range 2 32 > > + depends on SMP > > + default "8" > > + help > > + On multiprocessor machines, U-Boot sets up a stack for > > each CPU. > > + Stack memory is pre-allocated. U-Boot must therefore know > > the > > + maximum number of CPUs that may be present. > > + > > endmenu > > diff --git a/arch/riscv/include/asm/global_data.h > > b/arch/riscv/include/asm/global_data.h > > index a3a342c6e1..23a5f35af5 100644 > > --- a/arch/riscv/include/asm/global_data.h > > +++ b/arch/riscv/include/asm/global_data.h > > @@ -10,12 +10,17 @@ > > #ifndef __ASM_GBL_DATA_H > > #define __ASM_GBL_DATA_H > > > > +#include <asm/smp.h> > > + > > /* Architecture-specific global data */ > > struct arch_global_data { > > long boot_hart; /* boot hart id */ > > #ifdef CONFIG_SIFIVE_CLINT > > void __iomem *clint; /* clint base address */ > > #endif > > +#ifdef CONFIG_SMP > > + struct ipi_data ipi[CONFIG_NR_CPUS]; > > The data passed by "main HART" via global data to other HARTs > is not visible reliably and randomly few HARTs fail to enter Linux > kernel on my board. I am suspecting per-HART "ipi" data in global > data not being cache-line aligned as the cause of behavior but there > could be other issue too. > > I have a hack which works reliable for me. As-per this hack, we add > "mdelay(10)" just before calling riscv_send_ipi() in send_ipi_many(). > This hack helped me conclude that there is some sync issue in per- > HART > "ipi" data. > > The above issue is not seen on QEMU so we are fine there. > > I would suggest to make per-HART "ipi" data cache-line aligned > (just like Linux kernel). > Interesting, this is definitely a memory coherency issue, probably inadequate fences. I am not sure if making the ipi data structure cache-line aligned will reliably fix it. I'll try to replicate the issue and get it fixed. Thanks for reporting it! Thanks, Lukas ^ permalink raw reply [flat|nested] 45+ messages in thread
* [U-Boot] [PATCH 1/7] riscv: add infrastructure for calling functions on other harts 2019-02-18 10:01 ` Auer, Lukas @ 2019-02-18 11:41 ` Auer, Lukas 2019-02-18 13:16 ` Anup Patel 2019-02-19 8:16 ` Anup Patel 0 siblings, 2 replies; 45+ messages in thread From: Auer, Lukas @ 2019-02-18 11:41 UTC (permalink / raw) To: u-boot On Mon, 2019-02-18 at 10:01 +0000, Auer, Lukas wrote: > On Mon, 2019-02-18 at 10:28 +0530, Anup Patel wrote: > > On Tue, Feb 12, 2019 at 3:44 AM Lukas Auer > > <lukas.auer@aisec.fraunhofer.de> wrote: > > > Harts on RISC-V boot independently and U-Boot is responsible for > > > managing them. Functions are called on other harts with > > > smp_call_function(), which sends inter-processor interrupts > > > (IPIs) > > > to > > > all other harts. Functions are specified with their address and > > > two > > > function arguments (argument 2 and 3). The first function > > > argument > > > is > > > always the hart ID of the hart calling the function. On the other > > > harts, > > > the IPI interrupt handler handle_ipi() must be called on software > > > interrupts to handle the request and call the specified function. > > > > > > Functions are stored in the ipi_data data structure. Every hart > > > has > > > its > > > own data structure in global data. While this is not required at > > > the > > > moment (all harts are expected to boot Linux), this does allow > > > future > > > expansion, where other harts may be used for monitoring or other > > > tasks. > > > > > > Signed-off-by: Lukas Auer <lukas.auer@aisec.fraunhofer.de> > > > --- > > > > > > arch/riscv/Kconfig | 19 +++++ > > > arch/riscv/include/asm/global_data.h | 5 ++ > > > arch/riscv/include/asm/smp.h | 53 +++++++++++++ > > > arch/riscv/lib/Makefile | 1 + > > > arch/riscv/lib/smp.c | 110 > > > +++++++++++++++++++++++++++ > > > 5 files changed, 188 insertions(+) > > > create mode 100644 arch/riscv/include/asm/smp.h > > > create mode 100644 arch/riscv/lib/smp.c > > > > > > diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig > > > index c45e4d73a8..c0842178dd 100644 > > > --- a/arch/riscv/Kconfig > > > +++ b/arch/riscv/Kconfig > > > @@ -116,4 +116,23 @@ config RISCV_RDTIME > > > config SYS_MALLOC_F_LEN > > > default 0x1000 > > > > > > +config SMP > > > + bool "Symmetric Multi-Processing" > > > + help > > > + This enables support for systems with more than one > > > CPU. > > > If > > > + you say N here, U-Boot will run on single and > > > multiprocessor > > > + machines, but will use only one CPU of a multiprocessor > > > + machine. If you say Y here, U-Boot will run on many, > > > but > > > not > > > + all, single processor machines. > > > + > > > +config NR_CPUS > > > + int "Maximum number of CPUs (2-32)" > > > + range 2 32 > > > + depends on SMP > > > + default "8" > > > + help > > > + On multiprocessor machines, U-Boot sets up a stack for > > > each CPU. > > > + Stack memory is pre-allocated. U-Boot must therefore > > > know > > > the > > > + maximum number of CPUs that may be present. > > > + > > > endmenu > > > diff --git a/arch/riscv/include/asm/global_data.h > > > b/arch/riscv/include/asm/global_data.h > > > index a3a342c6e1..23a5f35af5 100644 > > > --- a/arch/riscv/include/asm/global_data.h > > > +++ b/arch/riscv/include/asm/global_data.h > > > @@ -10,12 +10,17 @@ > > > #ifndef __ASM_GBL_DATA_H > > > #define __ASM_GBL_DATA_H > > > > > > +#include <asm/smp.h> > > > + > > > /* Architecture-specific global data */ > > > struct arch_global_data { > > > long boot_hart; /* boot hart id */ > > > #ifdef CONFIG_SIFIVE_CLINT > > > void __iomem *clint; /* clint base address */ > > > #endif > > > +#ifdef CONFIG_SMP > > > + struct ipi_data ipi[CONFIG_NR_CPUS]; > > > > The data passed by "main HART" via global data to other HARTs > > is not visible reliably and randomly few HARTs fail to enter Linux > > kernel on my board. I am suspecting per-HART "ipi" data in global > > data not being cache-line aligned as the cause of behavior but > > there > > could be other issue too. > > > > I have a hack which works reliable for me. As-per this hack, we add > > "mdelay(10)" just before calling riscv_send_ipi() in > > send_ipi_many(). > > This hack helped me conclude that there is some sync issue in per- > > HART > > "ipi" data. > > > > The above issue is not seen on QEMU so we are fine there. > > > > I would suggest to make per-HART "ipi" data cache-line aligned > > (just like Linux kernel). > > > > Interesting, this is definitely a memory coherency issue, probably > inadequate fences. I am not sure if making the ipi data structure > cache-line aligned will reliably fix it. I'll try to replicate the > issue and get it fixed. Thanks for reporting it! > Not sure if it is connected, but I have noticed a regression with the current OpenSBI. Testing U-Boot with the SMP patches applied on QEMU with 4 harts, hart 4 will not receive software interrupts. I have bisected the issue to the following commit. 918c1354b75c74b62f67c4e929551d643f035443 is the first bad commit commit 918c1354b75c74b62f67c4e929551d643f035443 Author: Nick Kossifidis <mickflemm@gmail.com> Date: Sun Feb 17 09:00:20 2019 +0200 lib: Improve delivery of SBI_IPI_EVENT_HALT When sbi_ipi_send_many gets called with the current hartid included on pmask (or when pmask is NULL), and we send a HALT event, since the for loop works sequentially over all hartids on the mask, we may send a HALT event to the current hart before the loop finishes. So we will halt the current hart before it can deliver a HALT IPI to the rest and some harts will remain active. Make sure we send an IPI to the current hart after we've finished with everybody else. Signed-off-by: Nick Kossifidis <mick@ics.forth.gr> I don't have time to look into it right now, but I will do so later. Thanks, Lukas ^ permalink raw reply [flat|nested] 45+ messages in thread
* [U-Boot] [PATCH 1/7] riscv: add infrastructure for calling functions on other harts 2019-02-18 11:41 ` Auer, Lukas @ 2019-02-18 13:16 ` Anup Patel 2019-02-18 16:04 ` Auer, Lukas 2019-02-19 8:16 ` Anup Patel 1 sibling, 1 reply; 45+ messages in thread From: Anup Patel @ 2019-02-18 13:16 UTC (permalink / raw) To: u-boot On Mon, Feb 18, 2019 at 5:11 PM Auer, Lukas <lukas.auer@aisec.fraunhofer.de> wrote: > > On Mon, 2019-02-18 at 10:01 +0000, Auer, Lukas wrote: > > On Mon, 2019-02-18 at 10:28 +0530, Anup Patel wrote: > > > On Tue, Feb 12, 2019 at 3:44 AM Lukas Auer > > > <lukas.auer@aisec.fraunhofer.de> wrote: > > > > Harts on RISC-V boot independently and U-Boot is responsible for > > > > managing them. Functions are called on other harts with > > > > smp_call_function(), which sends inter-processor interrupts > > > > (IPIs) > > > > to > > > > all other harts. Functions are specified with their address and > > > > two > > > > function arguments (argument 2 and 3). The first function > > > > argument > > > > is > > > > always the hart ID of the hart calling the function. On the other > > > > harts, > > > > the IPI interrupt handler handle_ipi() must be called on software > > > > interrupts to handle the request and call the specified function. > > > > > > > > Functions are stored in the ipi_data data structure. Every hart > > > > has > > > > its > > > > own data structure in global data. While this is not required at > > > > the > > > > moment (all harts are expected to boot Linux), this does allow > > > > future > > > > expansion, where other harts may be used for monitoring or other > > > > tasks. > > > > > > > > Signed-off-by: Lukas Auer <lukas.auer@aisec.fraunhofer.de> > > > > --- > > > > > > > > arch/riscv/Kconfig | 19 +++++ > > > > arch/riscv/include/asm/global_data.h | 5 ++ > > > > arch/riscv/include/asm/smp.h | 53 +++++++++++++ > > > > arch/riscv/lib/Makefile | 1 + > > > > arch/riscv/lib/smp.c | 110 > > > > +++++++++++++++++++++++++++ > > > > 5 files changed, 188 insertions(+) > > > > create mode 100644 arch/riscv/include/asm/smp.h > > > > create mode 100644 arch/riscv/lib/smp.c > > > > > > > > diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig > > > > index c45e4d73a8..c0842178dd 100644 > > > > --- a/arch/riscv/Kconfig > > > > +++ b/arch/riscv/Kconfig > > > > @@ -116,4 +116,23 @@ config RISCV_RDTIME > > > > config SYS_MALLOC_F_LEN > > > > default 0x1000 > > > > > > > > +config SMP > > > > + bool "Symmetric Multi-Processing" > > > > + help > > > > + This enables support for systems with more than one > > > > CPU. > > > > If > > > > + you say N here, U-Boot will run on single and > > > > multiprocessor > > > > + machines, but will use only one CPU of a multiprocessor > > > > + machine. If you say Y here, U-Boot will run on many, > > > > but > > > > not > > > > + all, single processor machines. > > > > + > > > > +config NR_CPUS > > > > + int "Maximum number of CPUs (2-32)" > > > > + range 2 32 > > > > + depends on SMP > > > > + default "8" > > > > + help > > > > + On multiprocessor machines, U-Boot sets up a stack for > > > > each CPU. > > > > + Stack memory is pre-allocated. U-Boot must therefore > > > > know > > > > the > > > > + maximum number of CPUs that may be present. > > > > + > > > > endmenu > > > > diff --git a/arch/riscv/include/asm/global_data.h > > > > b/arch/riscv/include/asm/global_data.h > > > > index a3a342c6e1..23a5f35af5 100644 > > > > --- a/arch/riscv/include/asm/global_data.h > > > > +++ b/arch/riscv/include/asm/global_data.h > > > > @@ -10,12 +10,17 @@ > > > > #ifndef __ASM_GBL_DATA_H > > > > #define __ASM_GBL_DATA_H > > > > > > > > +#include <asm/smp.h> > > > > + > > > > /* Architecture-specific global data */ > > > > struct arch_global_data { > > > > long boot_hart; /* boot hart id */ > > > > #ifdef CONFIG_SIFIVE_CLINT > > > > void __iomem *clint; /* clint base address */ > > > > #endif > > > > +#ifdef CONFIG_SMP > > > > + struct ipi_data ipi[CONFIG_NR_CPUS]; > > > > > > The data passed by "main HART" via global data to other HARTs > > > is not visible reliably and randomly few HARTs fail to enter Linux > > > kernel on my board. I am suspecting per-HART "ipi" data in global > > > data not being cache-line aligned as the cause of behavior but > > > there > > > could be other issue too. > > > > > > I have a hack which works reliable for me. As-per this hack, we add > > > "mdelay(10)" just before calling riscv_send_ipi() in > > > send_ipi_many(). > > > This hack helped me conclude that there is some sync issue in per- > > > HART > > > "ipi" data. > > > > > > The above issue is not seen on QEMU so we are fine there. > > > > > > I would suggest to make per-HART "ipi" data cache-line aligned > > > (just like Linux kernel). > > > > > > > Interesting, this is definitely a memory coherency issue, probably > > inadequate fences. I am not sure if making the ipi data structure > > cache-line aligned will reliably fix it. I'll try to replicate the > > issue and get it fixed. Thanks for reporting it! > > > > Not sure if it is connected, but I have noticed a regression with the > current OpenSBI. Testing U-Boot with the SMP patches applied on QEMU > with 4 harts, hart 4 will not receive software interrupts. I have > bisected the issue to the following commit. > > 918c1354b75c74b62f67c4e929551d643f035443 is the first bad commit > commit 918c1354b75c74b62f67c4e929551d643f035443 > Author: Nick Kossifidis <mickflemm@gmail.com> > Date: Sun Feb 17 09:00:20 2019 +0200 > > lib: Improve delivery of SBI_IPI_EVENT_HALT > > When sbi_ipi_send_many gets called with the current hartid > included on pmask (or when pmask is NULL), and we send > a HALT event, since the for loop works sequentially over > all hartids on the mask, we may send a HALT event to the > current hart before the loop finishes. So we will halt > the current hart before it can deliver a HALT IPI to the > rest and some harts will remain active. > > Make sure we send an IPI to the current hart after we've > finished with everybody else. > > Signed-off-by: Nick Kossifidis <mick@ics.forth.gr> > No issue. Already fixed it with following commit: lib: Fix mask shift in sbi_ipi_send_many() The mask shift in for-loop of sbi_ipi_send_many() is broken with commit 918c135 ("lib: Improve delivery of SBI_IPI_EVENT_HALT") This patch fix it. Signed-off-by: Anup Patel <anup.patel@wdc.com> Please pull latest OpenSBI repo. Thanks, Anup ^ permalink raw reply [flat|nested] 45+ messages in thread
* [U-Boot] [PATCH 1/7] riscv: add infrastructure for calling functions on other harts 2019-02-18 13:16 ` Anup Patel @ 2019-02-18 16:04 ` Auer, Lukas 0 siblings, 0 replies; 45+ messages in thread From: Auer, Lukas @ 2019-02-18 16:04 UTC (permalink / raw) To: u-boot On Mon, 2019-02-18 at 18:46 +0530, Anup Patel wrote: > On Mon, Feb 18, 2019 at 5:11 PM Auer, Lukas > <lukas.auer@aisec.fraunhofer.de> wrote: > > On Mon, 2019-02-18 at 10:01 +0000, Auer, Lukas wrote: > > > On Mon, 2019-02-18 at 10:28 +0530, Anup Patel wrote: > > > > On Tue, Feb 12, 2019 at 3:44 AM Lukas Auer > > > > <lukas.auer@aisec.fraunhofer.de> wrote: > > > > > Harts on RISC-V boot independently and U-Boot is responsible > > > > > for > > > > > managing them. Functions are called on other harts with > > > > > smp_call_function(), which sends inter-processor interrupts > > > > > (IPIs) > > > > > to > > > > > all other harts. Functions are specified with their address > > > > > and > > > > > two > > > > > function arguments (argument 2 and 3). The first function > > > > > argument > > > > > is > > > > > always the hart ID of the hart calling the function. On the > > > > > other > > > > > harts, > > > > > the IPI interrupt handler handle_ipi() must be called on > > > > > software > > > > > interrupts to handle the request and call the specified > > > > > function. > > > > > > > > > > Functions are stored in the ipi_data data structure. Every > > > > > hart > > > > > has > > > > > its > > > > > own data structure in global data. While this is not required > > > > > at > > > > > the > > > > > moment (all harts are expected to boot Linux), this does > > > > > allow > > > > > future > > > > > expansion, where other harts may be used for monitoring or > > > > > other > > > > > tasks. > > > > > > > > > > Signed-off-by: Lukas Auer <lukas.auer@aisec.fraunhofer.de> > > > > > --- > > > > > > > > > > arch/riscv/Kconfig | 19 +++++ > > > > > arch/riscv/include/asm/global_data.h | 5 ++ > > > > > arch/riscv/include/asm/smp.h | 53 +++++++++++++ > > > > > arch/riscv/lib/Makefile | 1 + > > > > > arch/riscv/lib/smp.c | 110 > > > > > +++++++++++++++++++++++++++ > > > > > 5 files changed, 188 insertions(+) > > > > > create mode 100644 arch/riscv/include/asm/smp.h > > > > > create mode 100644 arch/riscv/lib/smp.c > > > > > > > > > > diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig > > > > > index c45e4d73a8..c0842178dd 100644 > > > > > --- a/arch/riscv/Kconfig > > > > > +++ b/arch/riscv/Kconfig > > > > > @@ -116,4 +116,23 @@ config RISCV_RDTIME > > > > > config SYS_MALLOC_F_LEN > > > > > default 0x1000 > > > > > > > > > > +config SMP > > > > > + bool "Symmetric Multi-Processing" > > > > > + help > > > > > + This enables support for systems with more than one > > > > > CPU. > > > > > If > > > > > + you say N here, U-Boot will run on single and > > > > > multiprocessor > > > > > + machines, but will use only one CPU of a > > > > > multiprocessor > > > > > + machine. If you say Y here, U-Boot will run on > > > > > many, > > > > > but > > > > > not > > > > > + all, single processor machines. > > > > > + > > > > > +config NR_CPUS > > > > > + int "Maximum number of CPUs (2-32)" > > > > > + range 2 32 > > > > > + depends on SMP > > > > > + default "8" > > > > > + help > > > > > + On multiprocessor machines, U-Boot sets up a stack > > > > > for > > > > > each CPU. > > > > > + Stack memory is pre-allocated. U-Boot must > > > > > therefore > > > > > know > > > > > the > > > > > + maximum number of CPUs that may be present. > > > > > + > > > > > endmenu > > > > > diff --git a/arch/riscv/include/asm/global_data.h > > > > > b/arch/riscv/include/asm/global_data.h > > > > > index a3a342c6e1..23a5f35af5 100644 > > > > > --- a/arch/riscv/include/asm/global_data.h > > > > > +++ b/arch/riscv/include/asm/global_data.h > > > > > @@ -10,12 +10,17 @@ > > > > > #ifndef __ASM_GBL_DATA_H > > > > > #define __ASM_GBL_DATA_H > > > > > > > > > > +#include <asm/smp.h> > > > > > + > > > > > /* Architecture-specific global data */ > > > > > struct arch_global_data { > > > > > long boot_hart; /* boot hart id */ > > > > > #ifdef CONFIG_SIFIVE_CLINT > > > > > void __iomem *clint; /* clint base address */ > > > > > #endif > > > > > +#ifdef CONFIG_SMP > > > > > + struct ipi_data ipi[CONFIG_NR_CPUS]; > > > > > > > > The data passed by "main HART" via global data to other HARTs > > > > is not visible reliably and randomly few HARTs fail to enter > > > > Linux > > > > kernel on my board. I am suspecting per-HART "ipi" data in > > > > global > > > > data not being cache-line aligned as the cause of behavior but > > > > there > > > > could be other issue too. > > > > > > > > I have a hack which works reliable for me. As-per this hack, we > > > > add > > > > "mdelay(10)" just before calling riscv_send_ipi() in > > > > send_ipi_many(). > > > > This hack helped me conclude that there is some sync issue in > > > > per- > > > > HART > > > > "ipi" data. > > > > > > > > The above issue is not seen on QEMU so we are fine there. > > > > > > > > I would suggest to make per-HART "ipi" data cache-line aligned > > > > (just like Linux kernel). > > > > > > > > > > Interesting, this is definitely a memory coherency issue, > > > probably > > > inadequate fences. I am not sure if making the ipi data structure > > > cache-line aligned will reliably fix it. I'll try to replicate > > > the > > > issue and get it fixed. Thanks for reporting it! > > > > > > > Not sure if it is connected, but I have noticed a regression with > > the > > current OpenSBI. Testing U-Boot with the SMP patches applied on > > QEMU > > with 4 harts, hart 4 will not receive software interrupts. I have > > bisected the issue to the following commit. > > > > 918c1354b75c74b62f67c4e929551d643f035443 is the first bad commit > > commit 918c1354b75c74b62f67c4e929551d643f035443 > > Author: Nick Kossifidis <mickflemm@gmail.com> > > Date: Sun Feb 17 09:00:20 2019 +0200 > > > > lib: Improve delivery of SBI_IPI_EVENT_HALT > > > > When sbi_ipi_send_many gets called with the current hartid > > included on pmask (or when pmask is NULL), and we send > > a HALT event, since the for loop works sequentially over > > all hartids on the mask, we may send a HALT event to the > > current hart before the loop finishes. So we will halt > > the current hart before it can deliver a HALT IPI to the > > rest and some harts will remain active. > > > > Make sure we send an IPI to the current hart after we've > > finished with everybody else. > > > > Signed-off-by: Nick Kossifidis <mick@ics.forth.gr> > > > > No issue. Already fixed it with following commit: > > lib: Fix mask shift in sbi_ipi_send_many() > > The mask shift in for-loop of sbi_ipi_send_many() is > broken with commit 918c135 > ("lib: Improve delivery of SBI_IPI_EVENT_HALT") > > This patch fix it. > > Signed-off-by: Anup Patel <anup.patel@wdc.com> > > > Please pull latest OpenSBI repo. > Thanks, that fixed it. Lukas ^ permalink raw reply [flat|nested] 45+ messages in thread
* [U-Boot] [PATCH 1/7] riscv: add infrastructure for calling functions on other harts 2019-02-18 11:41 ` Auer, Lukas 2019-02-18 13:16 ` Anup Patel @ 2019-02-19 8:16 ` Anup Patel 2019-02-25 11:13 ` Auer, Lukas 1 sibling, 1 reply; 45+ messages in thread From: Anup Patel @ 2019-02-19 8:16 UTC (permalink / raw) To: u-boot On Mon, Feb 18, 2019 at 5:11 PM Auer, Lukas <lukas.auer@aisec.fraunhofer.de> wrote: > > On Mon, 2019-02-18 at 10:01 +0000, Auer, Lukas wrote: > > On Mon, 2019-02-18 at 10:28 +0530, Anup Patel wrote: > > > On Tue, Feb 12, 2019 at 3:44 AM Lukas Auer > > > <lukas.auer@aisec.fraunhofer.de> wrote: > > > > Harts on RISC-V boot independently and U-Boot is responsible for > > > > managing them. Functions are called on other harts with > > > > smp_call_function(), which sends inter-processor interrupts > > > > (IPIs) > > > > to > > > > all other harts. Functions are specified with their address and > > > > two > > > > function arguments (argument 2 and 3). The first function > > > > argument > > > > is > > > > always the hart ID of the hart calling the function. On the other > > > > harts, > > > > the IPI interrupt handler handle_ipi() must be called on software > > > > interrupts to handle the request and call the specified function. > > > > > > > > Functions are stored in the ipi_data data structure. Every hart > > > > has > > > > its > > > > own data structure in global data. While this is not required at > > > > the > > > > moment (all harts are expected to boot Linux), this does allow > > > > future > > > > expansion, where other harts may be used for monitoring or other > > > > tasks. > > > > > > > > Signed-off-by: Lukas Auer <lukas.auer@aisec.fraunhofer.de> > > > > --- > > > > > > > > arch/riscv/Kconfig | 19 +++++ > > > > arch/riscv/include/asm/global_data.h | 5 ++ > > > > arch/riscv/include/asm/smp.h | 53 +++++++++++++ > > > > arch/riscv/lib/Makefile | 1 + > > > > arch/riscv/lib/smp.c | 110 > > > > +++++++++++++++++++++++++++ > > > > 5 files changed, 188 insertions(+) > > > > create mode 100644 arch/riscv/include/asm/smp.h > > > > create mode 100644 arch/riscv/lib/smp.c > > > > > > > > diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig > > > > index c45e4d73a8..c0842178dd 100644 > > > > --- a/arch/riscv/Kconfig > > > > +++ b/arch/riscv/Kconfig > > > > @@ -116,4 +116,23 @@ config RISCV_RDTIME > > > > config SYS_MALLOC_F_LEN > > > > default 0x1000 > > > > > > > > +config SMP > > > > + bool "Symmetric Multi-Processing" > > > > + help > > > > + This enables support for systems with more than one > > > > CPU. > > > > If > > > > + you say N here, U-Boot will run on single and > > > > multiprocessor > > > > + machines, but will use only one CPU of a multiprocessor > > > > + machine. If you say Y here, U-Boot will run on many, > > > > but > > > > not > > > > + all, single processor machines. > > > > + > > > > +config NR_CPUS > > > > + int "Maximum number of CPUs (2-32)" > > > > + range 2 32 > > > > + depends on SMP > > > > + default "8" > > > > + help > > > > + On multiprocessor machines, U-Boot sets up a stack for > > > > each CPU. > > > > + Stack memory is pre-allocated. U-Boot must therefore > > > > know > > > > the > > > > + maximum number of CPUs that may be present. > > > > + > > > > endmenu > > > > diff --git a/arch/riscv/include/asm/global_data.h > > > > b/arch/riscv/include/asm/global_data.h > > > > index a3a342c6e1..23a5f35af5 100644 > > > > --- a/arch/riscv/include/asm/global_data.h > > > > +++ b/arch/riscv/include/asm/global_data.h > > > > @@ -10,12 +10,17 @@ > > > > #ifndef __ASM_GBL_DATA_H > > > > #define __ASM_GBL_DATA_H > > > > > > > > +#include <asm/smp.h> > > > > + > > > > /* Architecture-specific global data */ > > > > struct arch_global_data { > > > > long boot_hart; /* boot hart id */ > > > > #ifdef CONFIG_SIFIVE_CLINT > > > > void __iomem *clint; /* clint base address */ > > > > #endif > > > > +#ifdef CONFIG_SMP > > > > + struct ipi_data ipi[CONFIG_NR_CPUS]; > > > > > > The data passed by "main HART" via global data to other HARTs > > > is not visible reliably and randomly few HARTs fail to enter Linux > > > kernel on my board. I am suspecting per-HART "ipi" data in global > > > data not being cache-line aligned as the cause of behavior but > > > there > > > could be other issue too. > > > > > > I have a hack which works reliable for me. As-per this hack, we add > > > "mdelay(10)" just before calling riscv_send_ipi() in > > > send_ipi_many(). > > > This hack helped me conclude that there is some sync issue in per- > > > HART > > > "ipi" data. > > > > > > The above issue is not seen on QEMU so we are fine there. > > > > > > I would suggest to make per-HART "ipi" data cache-line aligned > > > (just like Linux kernel). > > > > > > > Interesting, this is definitely a memory coherency issue, probably > > inadequate fences. I am not sure if making the ipi data structure > > cache-line aligned will reliably fix it. I'll try to replicate the > > issue and get it fixed. Thanks for reporting it! > > > > Not sure if it is connected, but I have noticed a regression with the > current OpenSBI. Testing U-Boot with the SMP patches applied on QEMU > with 4 harts, hart 4 will not receive software interrupts. I have > bisected the issue to the following commit. > It's not related to OpenSBI IPI issue we fixed recently. I was seeing this issue before OpenSBI IPI breakage. I tried again with latest OpenSBI and it worked for me 8 out-of 10 times. Strangely, Atish and you don't see this issue. Maybe it's just my board. In any case, it will be good to have: 1. Few more fences (taking reference from Linux code) 2. Ensure per-HART "ipi" data is cache line aligned Maybe with above things in-place, it will become unlikely for me to see this issue. You can even address this issue as separate patches later since it's only me seeing it occasionally. Regards, Anup ^ permalink raw reply [flat|nested] 45+ messages in thread
* [U-Boot] [PATCH 1/7] riscv: add infrastructure for calling functions on other harts 2019-02-19 8:16 ` Anup Patel @ 2019-02-25 11:13 ` Auer, Lukas 0 siblings, 0 replies; 45+ messages in thread From: Auer, Lukas @ 2019-02-25 11:13 UTC (permalink / raw) To: u-boot On Tue, 2019-02-19 at 13:46 +0530, Anup Patel wrote: > On Mon, Feb 18, 2019 at 5:11 PM Auer, Lukas > <lukas.auer@aisec.fraunhofer.de> wrote: > > On Mon, 2019-02-18 at 10:01 +0000, Auer, Lukas wrote: > > > On Mon, 2019-02-18 at 10:28 +0530, Anup Patel wrote: > > > > On Tue, Feb 12, 2019 at 3:44 AM Lukas Auer > > > > <lukas.auer@aisec.fraunhofer.de> wrote: > > > > > Harts on RISC-V boot independently and U-Boot is responsible > > > > > for > > > > > managing them. Functions are called on other harts with > > > > > smp_call_function(), which sends inter-processor interrupts > > > > > (IPIs) > > > > > to > > > > > all other harts. Functions are specified with their address > > > > > and > > > > > two > > > > > function arguments (argument 2 and 3). The first function > > > > > argument > > > > > is > > > > > always the hart ID of the hart calling the function. On the > > > > > other > > > > > harts, > > > > > the IPI interrupt handler handle_ipi() must be called on > > > > > software > > > > > interrupts to handle the request and call the specified > > > > > function. > > > > > > > > > > Functions are stored in the ipi_data data structure. Every > > > > > hart > > > > > has > > > > > its > > > > > own data structure in global data. While this is not required > > > > > at > > > > > the > > > > > moment (all harts are expected to boot Linux), this does > > > > > allow > > > > > future > > > > > expansion, where other harts may be used for monitoring or > > > > > other > > > > > tasks. > > > > > > > > > > Signed-off-by: Lukas Auer <lukas.auer@aisec.fraunhofer.de> > > > > > --- > > > > > > > > > > arch/riscv/Kconfig | 19 +++++ > > > > > arch/riscv/include/asm/global_data.h | 5 ++ > > > > > arch/riscv/include/asm/smp.h | 53 +++++++++++++ > > > > > arch/riscv/lib/Makefile | 1 + > > > > > arch/riscv/lib/smp.c | 110 > > > > > +++++++++++++++++++++++++++ > > > > > 5 files changed, 188 insertions(+) > > > > > create mode 100644 arch/riscv/include/asm/smp.h > > > > > create mode 100644 arch/riscv/lib/smp.c > > > > > > > > > > diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig > > > > > index c45e4d73a8..c0842178dd 100644 > > > > > --- a/arch/riscv/Kconfig > > > > > +++ b/arch/riscv/Kconfig > > > > > @@ -116,4 +116,23 @@ config RISCV_RDTIME > > > > > config SYS_MALLOC_F_LEN > > > > > default 0x1000 > > > > > > > > > > +config SMP > > > > > + bool "Symmetric Multi-Processing" > > > > > + help > > > > > + This enables support for systems with more than one > > > > > CPU. > > > > > If > > > > > + you say N here, U-Boot will run on single and > > > > > multiprocessor > > > > > + machines, but will use only one CPU of a > > > > > multiprocessor > > > > > + machine. If you say Y here, U-Boot will run on > > > > > many, > > > > > but > > > > > not > > > > > + all, single processor machines. > > > > > + > > > > > +config NR_CPUS > > > > > + int "Maximum number of CPUs (2-32)" > > > > > + range 2 32 > > > > > + depends on SMP > > > > > + default "8" > > > > > + help > > > > > + On multiprocessor machines, U-Boot sets up a stack > > > > > for > > > > > each CPU. > > > > > + Stack memory is pre-allocated. U-Boot must > > > > > therefore > > > > > know > > > > > the > > > > > + maximum number of CPUs that may be present. > > > > > + > > > > > endmenu > > > > > diff --git a/arch/riscv/include/asm/global_data.h > > > > > b/arch/riscv/include/asm/global_data.h > > > > > index a3a342c6e1..23a5f35af5 100644 > > > > > --- a/arch/riscv/include/asm/global_data.h > > > > > +++ b/arch/riscv/include/asm/global_data.h > > > > > @@ -10,12 +10,17 @@ > > > > > #ifndef __ASM_GBL_DATA_H > > > > > #define __ASM_GBL_DATA_H > > > > > > > > > > +#include <asm/smp.h> > > > > > + > > > > > /* Architecture-specific global data */ > > > > > struct arch_global_data { > > > > > long boot_hart; /* boot hart id */ > > > > > #ifdef CONFIG_SIFIVE_CLINT > > > > > void __iomem *clint; /* clint base address */ > > > > > #endif > > > > > +#ifdef CONFIG_SMP > > > > > + struct ipi_data ipi[CONFIG_NR_CPUS]; > > > > > > > > The data passed by "main HART" via global data to other HARTs > > > > is not visible reliably and randomly few HARTs fail to enter > > > > Linux > > > > kernel on my board. I am suspecting per-HART "ipi" data in > > > > global > > > > data not being cache-line aligned as the cause of behavior but > > > > there > > > > could be other issue too. > > > > > > > > I have a hack which works reliable for me. As-per this hack, we > > > > add > > > > "mdelay(10)" just before calling riscv_send_ipi() in > > > > send_ipi_many(). > > > > This hack helped me conclude that there is some sync issue in > > > > per- > > > > HART > > > > "ipi" data. > > > > > > > > The above issue is not seen on QEMU so we are fine there. > > > > > > > > I would suggest to make per-HART "ipi" data cache-line aligned > > > > (just like Linux kernel). > > > > > > > > > > Interesting, this is definitely a memory coherency issue, > > > probably > > > inadequate fences. I am not sure if making the ipi data structure > > > cache-line aligned will reliably fix it. I'll try to replicate > > > the > > > issue and get it fixed. Thanks for reporting it! > > > > > > > Not sure if it is connected, but I have noticed a regression with > > the > > current OpenSBI. Testing U-Boot with the SMP patches applied on > > QEMU > > with 4 harts, hart 4 will not receive software interrupts. I have > > bisected the issue to the following commit. > > > > It's not related to OpenSBI IPI issue we fixed recently. I was seeing > this issue before OpenSBI IPI breakage. > > I tried again with latest OpenSBI and it worked for me 8 out-of 10 > times. > > Strangely, Atish and you don't see this issue. Maybe it's just my > board. > > In any case, it will be good to have: > 1. Few more fences (taking reference from Linux code) > 2. Ensure per-HART "ipi" data is cache line aligned > > Maybe with above things in-place, it will become unlikely for > me to see this issue. > > You can even address this issue as separate patches later > since it's only me seeing it occasionally. > I was unable to reproduce this issue. I did however see strange behavior when using the reset button to reset the board [1]. Did you use the reset button to reset the board? In any case, I will move the memory barrier to the handle_ipi() function. It should not be needed in send_ipi_many, since the IO access function already include suitable memory barriers. We do not have those on the handle_ipi side, so this could actually be the cause of the issue you are seeing. Thanks, Lukas [1]: https://github.com/riscv/opensbi/issues/65 ^ permalink raw reply [flat|nested] 45+ messages in thread
* [U-Boot] [PATCH 2/7] riscv: import the supervisor binary interface header file 2019-02-11 22:13 [U-Boot] [PATCH 0/7] SMP support for RISC-V Lukas Auer 2019-02-11 22:13 ` [U-Boot] [PATCH 1/7] riscv: add infrastructure for calling functions on other harts Lukas Auer @ 2019-02-11 22:13 ` Lukas Auer 2019-02-12 2:32 ` Anup Patel 2019-02-12 3:03 ` Bin Meng 2019-02-11 22:13 ` [U-Boot] [PATCH 3/7] riscv: implement IPI platform functions using SBI Lukas Auer ` (5 subsequent siblings) 7 siblings, 2 replies; 45+ messages in thread From: Lukas Auer @ 2019-02-11 22:13 UTC (permalink / raw) To: u-boot Import the supervisor binary interface (SBI) header file from Linux (arch/riscv/include/asm/sbi.h). The last change to it was in commit 6d60b6ee0c97 ("RISC-V: Device, timer, IRQs, and the SBI"). Signed-off-by: Lukas Auer <lukas.auer@aisec.fraunhofer.de> --- arch/riscv/include/asm/sbi.h | 94 ++++++++++++++++++++++++++++++++++++ 1 file changed, 94 insertions(+) create mode 100644 arch/riscv/include/asm/sbi.h diff --git a/arch/riscv/include/asm/sbi.h b/arch/riscv/include/asm/sbi.h new file mode 100644 index 0000000000..ced57defdd --- /dev/null +++ b/arch/riscv/include/asm/sbi.h @@ -0,0 +1,94 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * Copyright (C) 2015 Regents of the University of California + * + * Taken from Linux arch/riscv/include/asm/sbi.h + */ + +#ifndef _ASM_RISCV_SBI_H +#define _ASM_RISCV_SBI_H + +#include <linux/types.h> + +#define SBI_SET_TIMER 0 +#define SBI_CONSOLE_PUTCHAR 1 +#define SBI_CONSOLE_GETCHAR 2 +#define SBI_CLEAR_IPI 3 +#define SBI_SEND_IPI 4 +#define SBI_REMOTE_FENCE_I 5 +#define SBI_REMOTE_SFENCE_VMA 6 +#define SBI_REMOTE_SFENCE_VMA_ASID 7 +#define SBI_SHUTDOWN 8 + +#define SBI_CALL(which, arg0, arg1, arg2) ({ \ + register uintptr_t a0 asm ("a0") = (uintptr_t)(arg0); \ + register uintptr_t a1 asm ("a1") = (uintptr_t)(arg1); \ + register uintptr_t a2 asm ("a2") = (uintptr_t)(arg2); \ + register uintptr_t a7 asm ("a7") = (uintptr_t)(which); \ + asm volatile ("ecall" \ + : "+r" (a0) \ + : "r" (a1), "r" (a2), "r" (a7) \ + : "memory"); \ + a0; \ +}) + +/* Lazy implementations until SBI is finalized */ +#define SBI_CALL_0(which) SBI_CALL(which, 0, 0, 0) +#define SBI_CALL_1(which, arg0) SBI_CALL(which, arg0, 0, 0) +#define SBI_CALL_2(which, arg0, arg1) SBI_CALL(which, arg0, arg1, 0) + +static inline void sbi_console_putchar(int ch) +{ + SBI_CALL_1(SBI_CONSOLE_PUTCHAR, ch); +} + +static inline int sbi_console_getchar(void) +{ + return SBI_CALL_0(SBI_CONSOLE_GETCHAR); +} + +static inline void sbi_set_timer(uint64_t stime_value) +{ +#if __riscv_xlen == 32 + SBI_CALL_2(SBI_SET_TIMER, stime_value, stime_value >> 32); +#else + SBI_CALL_1(SBI_SET_TIMER, stime_value); +#endif +} + +static inline void sbi_shutdown(void) +{ + SBI_CALL_0(SBI_SHUTDOWN); +} + +static inline void sbi_clear_ipi(void) +{ + SBI_CALL_0(SBI_CLEAR_IPI); +} + +static inline void sbi_send_ipi(const unsigned long *hart_mask) +{ + SBI_CALL_1(SBI_SEND_IPI, hart_mask); +} + +static inline void sbi_remote_fence_i(const unsigned long *hart_mask) +{ + SBI_CALL_1(SBI_REMOTE_FENCE_I, hart_mask); +} + +static inline void sbi_remote_sfence_vma(const unsigned long *hart_mask, + unsigned long start, + unsigned long size) +{ + SBI_CALL_1(SBI_REMOTE_SFENCE_VMA, hart_mask); +} + +static inline void sbi_remote_sfence_vma_asid(const unsigned long *hart_mask, + unsigned long start, + unsigned long size, + unsigned long asid) +{ + SBI_CALL_1(SBI_REMOTE_SFENCE_VMA_ASID, hart_mask); +} + +#endif -- 2.20.1 ^ permalink raw reply related [flat|nested] 45+ messages in thread
* [U-Boot] [PATCH 2/7] riscv: import the supervisor binary interface header file 2019-02-11 22:13 ` [U-Boot] [PATCH 2/7] riscv: import the supervisor binary interface header file Lukas Auer @ 2019-02-12 2:32 ` Anup Patel 2019-02-12 3:03 ` Bin Meng 1 sibling, 0 replies; 45+ messages in thread From: Anup Patel @ 2019-02-12 2:32 UTC (permalink / raw) To: u-boot > -----Original Message----- > From: Lukas Auer [mailto:lukas.auer at aisec.fraunhofer.de] > Sent: Tuesday, February 12, 2019 3:44 AM > To: u-boot at lists.denx.de > Cc: Atish Patra <Atish.Patra@wdc.com>; Anup Patel > <Anup.Patel@wdc.com>; Bin Meng <bmeng.cn@gmail.com>; Andreas > Schwab <schwab@suse.de>; Palmer Dabbelt <palmer@sifive.com>; > Alexander Graf <agraf@suse.de>; Lukas Auer > <lukas.auer@aisec.fraunhofer.de>; Rick Chen <rick@andestech.com> > Subject: [PATCH 2/7] riscv: import the supervisor binary interface header file > > Import the supervisor binary interface (SBI) header file from Linux > (arch/riscv/include/asm/sbi.h). The last change to it was in commit > 6d60b6ee0c97 ("RISC-V: Device, timer, IRQs, and the SBI"). > > Signed-off-by: Lukas Auer <lukas.auer@aisec.fraunhofer.de> > --- > > arch/riscv/include/asm/sbi.h | 94 > ++++++++++++++++++++++++++++++++++++ > 1 file changed, 94 insertions(+) > create mode 100644 arch/riscv/include/asm/sbi.h > > diff --git a/arch/riscv/include/asm/sbi.h b/arch/riscv/include/asm/sbi.h new > file mode 100644 index 0000000000..ced57defdd > --- /dev/null > +++ b/arch/riscv/include/asm/sbi.h > @@ -0,0 +1,94 @@ > +/* SPDX-License-Identifier: GPL-2.0 */ > +/* > + * Copyright (C) 2015 Regents of the University of California > + * > + * Taken from Linux arch/riscv/include/asm/sbi.h */ > + > +#ifndef _ASM_RISCV_SBI_H > +#define _ASM_RISCV_SBI_H > + > +#include <linux/types.h> > + > +#define SBI_SET_TIMER 0 > +#define SBI_CONSOLE_PUTCHAR 1 > +#define SBI_CONSOLE_GETCHAR 2 > +#define SBI_CLEAR_IPI 3 > +#define SBI_SEND_IPI 4 > +#define SBI_REMOTE_FENCE_I 5 > +#define SBI_REMOTE_SFENCE_VMA 6 > +#define SBI_REMOTE_SFENCE_VMA_ASID 7 > +#define SBI_SHUTDOWN 8 > + > +#define SBI_CALL(which, arg0, arg1, arg2) ({ \ > + register uintptr_t a0 asm ("a0") = (uintptr_t)(arg0); \ > + register uintptr_t a1 asm ("a1") = (uintptr_t)(arg1); \ > + register uintptr_t a2 asm ("a2") = (uintptr_t)(arg2); \ > + register uintptr_t a7 asm ("a7") = (uintptr_t)(which); \ > + asm volatile ("ecall" \ > + : "+r" (a0) \ > + : "r" (a1), "r" (a2), "r" (a7) \ > + : "memory"); \ > + a0; \ > +}) > + > +/* Lazy implementations until SBI is finalized */ #define > +SBI_CALL_0(which) SBI_CALL(which, 0, 0, 0) #define SBI_CALL_1(which, > +arg0) SBI_CALL(which, arg0, 0, 0) #define SBI_CALL_2(which, arg0, arg1) > +SBI_CALL(which, arg0, arg1, 0) > + > +static inline void sbi_console_putchar(int ch) { > + SBI_CALL_1(SBI_CONSOLE_PUTCHAR, ch); > +} > + > +static inline int sbi_console_getchar(void) { > + return SBI_CALL_0(SBI_CONSOLE_GETCHAR); } > + > +static inline void sbi_set_timer(uint64_t stime_value) { #if > +__riscv_xlen == 32 > + SBI_CALL_2(SBI_SET_TIMER, stime_value, stime_value >> 32); #else > + SBI_CALL_1(SBI_SET_TIMER, stime_value); #endif } > + > +static inline void sbi_shutdown(void) > +{ > + SBI_CALL_0(SBI_SHUTDOWN); > +} > + > +static inline void sbi_clear_ipi(void) > +{ > + SBI_CALL_0(SBI_CLEAR_IPI); > +} > + > +static inline void sbi_send_ipi(const unsigned long *hart_mask) { > + SBI_CALL_1(SBI_SEND_IPI, hart_mask); > +} > + > +static inline void sbi_remote_fence_i(const unsigned long *hart_mask) { > + SBI_CALL_1(SBI_REMOTE_FENCE_I, hart_mask); } > + > +static inline void sbi_remote_sfence_vma(const unsigned long *hart_mask, > + unsigned long start, > + unsigned long size) > +{ > + SBI_CALL_1(SBI_REMOTE_SFENCE_VMA, hart_mask); } > + > +static inline void sbi_remote_sfence_vma_asid(const unsigned long > *hart_mask, > + unsigned long start, > + unsigned long size, > + unsigned long asid) > +{ > + SBI_CALL_1(SBI_REMOTE_SFENCE_VMA_ASID, hart_mask); } > + > +#endif > -- > 2.20.1 Looks good to me. Reviewed-by: Anup Patel <anup.patel@wdc.com> Regards, Anup ^ permalink raw reply [flat|nested] 45+ messages in thread
* [U-Boot] [PATCH 2/7] riscv: import the supervisor binary interface header file 2019-02-11 22:13 ` [U-Boot] [PATCH 2/7] riscv: import the supervisor binary interface header file Lukas Auer 2019-02-12 2:32 ` Anup Patel @ 2019-02-12 3:03 ` Bin Meng 1 sibling, 0 replies; 45+ messages in thread From: Bin Meng @ 2019-02-12 3:03 UTC (permalink / raw) To: u-boot On Tue, Feb 12, 2019 at 6:14 AM Lukas Auer <lukas.auer@aisec.fraunhofer.de> wrote: > > Import the supervisor binary interface (SBI) header file from Linux > (arch/riscv/include/asm/sbi.h). The last change to it was in commit > 6d60b6ee0c97 ("RISC-V: Device, timer, IRQs, and the SBI"). > > Signed-off-by: Lukas Auer <lukas.auer@aisec.fraunhofer.de> > --- > > arch/riscv/include/asm/sbi.h | 94 ++++++++++++++++++++++++++++++++++++ > 1 file changed, 94 insertions(+) > create mode 100644 arch/riscv/include/asm/sbi.h > Reviewed-by: Bin Meng <bmeng.cn@gmail.com> ^ permalink raw reply [flat|nested] 45+ messages in thread
* [U-Boot] [PATCH 3/7] riscv: implement IPI platform functions using SBI 2019-02-11 22:13 [U-Boot] [PATCH 0/7] SMP support for RISC-V Lukas Auer 2019-02-11 22:13 ` [U-Boot] [PATCH 1/7] riscv: add infrastructure for calling functions on other harts Lukas Auer 2019-02-11 22:13 ` [U-Boot] [PATCH 2/7] riscv: import the supervisor binary interface header file Lukas Auer @ 2019-02-11 22:13 ` Lukas Auer 2019-02-12 2:32 ` Anup Patel 2019-02-12 3:03 ` Bin Meng 2019-02-11 22:13 ` [U-Boot] [PATCH 4/7] riscv: delay initialization of caches and debug UART Lukas Auer ` (4 subsequent siblings) 7 siblings, 2 replies; 45+ messages in thread From: Lukas Auer @ 2019-02-11 22:13 UTC (permalink / raw) To: u-boot The supervisor binary interface (SBI) provides the necessary functions to implement the platform IPI functions riscv_send_ipi() and riscv_clear_ipi(). Use it to implement them. This adds support for inter-processor interrupts (IPIs) on RISC-V CPUs running in supervisor mode. Support for machine mode is already available for CPUs that include the SiFive CLINT. Signed-off-by: Lukas Auer <lukas.auer@aisec.fraunhofer.de> --- arch/riscv/Kconfig | 5 +++++ arch/riscv/lib/Makefile | 1 + arch/riscv/lib/sbi_ipi.c | 25 +++++++++++++++++++++++++ 3 files changed, 31 insertions(+) create mode 100644 arch/riscv/lib/sbi_ipi.c diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index c0842178dd..3a51339c4d 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -135,4 +135,9 @@ config NR_CPUS Stack memory is pre-allocated. U-Boot must therefore know the maximum number of CPUs that may be present. +config SBI_IPI + bool + default y if RISCV_SMODE + depends on SMP + endmenu diff --git a/arch/riscv/lib/Makefile b/arch/riscv/lib/Makefile index 19370f9749..35dbf643e4 100644 --- a/arch/riscv/lib/Makefile +++ b/arch/riscv/lib/Makefile @@ -13,6 +13,7 @@ obj-$(CONFIG_RISCV_RDTIME) += rdtime.o obj-$(CONFIG_SIFIVE_CLINT) += sifive_clint.o obj-y += interrupts.o obj-y += reset.o +obj-$(CONFIG_SBI_IPI) += sbi_ipi.o obj-y += setjmp.o obj-$(CONFIG_SMP) += smp.o diff --git a/arch/riscv/lib/sbi_ipi.c b/arch/riscv/lib/sbi_ipi.c new file mode 100644 index 0000000000..170346da68 --- /dev/null +++ b/arch/riscv/lib/sbi_ipi.c @@ -0,0 +1,25 @@ +// SPDX-License-Identifier: GPL-2.0+ +/* + * Copyright (C) 2019 Fraunhofer AISEC, + * Lukas Auer <lukas.auer@aisec.fraunhofer.de> + */ + +#include <common.h> +#include <asm/sbi.h> + +int riscv_send_ipi(int hart) +{ + ulong mask; + + mask = 1UL << hart; + sbi_send_ipi(&mask); + + return 0; +} + +int riscv_clear_ipi(int hart) +{ + sbi_clear_ipi(); + + return 0; +} -- 2.20.1 ^ permalink raw reply related [flat|nested] 45+ messages in thread
* [U-Boot] [PATCH 3/7] riscv: implement IPI platform functions using SBI 2019-02-11 22:13 ` [U-Boot] [PATCH 3/7] riscv: implement IPI platform functions using SBI Lukas Auer @ 2019-02-12 2:32 ` Anup Patel 2019-02-12 3:03 ` Bin Meng 1 sibling, 0 replies; 45+ messages in thread From: Anup Patel @ 2019-02-12 2:32 UTC (permalink / raw) To: u-boot > -----Original Message----- > From: Lukas Auer [mailto:lukas.auer at aisec.fraunhofer.de] > Sent: Tuesday, February 12, 2019 3:44 AM > To: u-boot at lists.denx.de > Cc: Atish Patra <Atish.Patra@wdc.com>; Anup Patel > <Anup.Patel@wdc.com>; Bin Meng <bmeng.cn@gmail.com>; Andreas > Schwab <schwab@suse.de>; Palmer Dabbelt <palmer@sifive.com>; > Alexander Graf <agraf@suse.de>; Lukas Auer > <lukas.auer@aisec.fraunhofer.de>; Anup Patel <anup@brainfault.org>; Rick > Chen <rick@andestech.com> > Subject: [PATCH 3/7] riscv: implement IPI platform functions using SBI > > The supervisor binary interface (SBI) provides the necessary functions to > implement the platform IPI functions riscv_send_ipi() and riscv_clear_ipi(). > Use it to implement them. > > This adds support for inter-processor interrupts (IPIs) on RISC-V CPUs > running in supervisor mode. Support for machine mode is already available > for CPUs that include the SiFive CLINT. > > Signed-off-by: Lukas Auer <lukas.auer@aisec.fraunhofer.de> > --- > > arch/riscv/Kconfig | 5 +++++ > arch/riscv/lib/Makefile | 1 + > arch/riscv/lib/sbi_ipi.c | 25 +++++++++++++++++++++++++ > 3 files changed, 31 insertions(+) > create mode 100644 arch/riscv/lib/sbi_ipi.c > > diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index > c0842178dd..3a51339c4d 100644 > --- a/arch/riscv/Kconfig > +++ b/arch/riscv/Kconfig > @@ -135,4 +135,9 @@ config NR_CPUS > Stack memory is pre-allocated. U-Boot must therefore know the > maximum number of CPUs that may be present. > > +config SBI_IPI > + bool > + default y if RISCV_SMODE > + depends on SMP > + > endmenu > diff --git a/arch/riscv/lib/Makefile b/arch/riscv/lib/Makefile index > 19370f9749..35dbf643e4 100644 > --- a/arch/riscv/lib/Makefile > +++ b/arch/riscv/lib/Makefile > @@ -13,6 +13,7 @@ obj-$(CONFIG_RISCV_RDTIME) += rdtime.o > obj-$(CONFIG_SIFIVE_CLINT) += sifive_clint.o > obj-y += interrupts.o > obj-y += reset.o > +obj-$(CONFIG_SBI_IPI) += sbi_ipi.o > obj-y += setjmp.o > obj-$(CONFIG_SMP) += smp.o > > diff --git a/arch/riscv/lib/sbi_ipi.c b/arch/riscv/lib/sbi_ipi.c new file mode > 100644 index 0000000000..170346da68 > --- /dev/null > +++ b/arch/riscv/lib/sbi_ipi.c > @@ -0,0 +1,25 @@ > +// SPDX-License-Identifier: GPL-2.0+ > +/* > + * Copyright (C) 2019 Fraunhofer AISEC, > + * Lukas Auer <lukas.auer@aisec.fraunhofer.de> */ > + > +#include <common.h> > +#include <asm/sbi.h> > + > +int riscv_send_ipi(int hart) > +{ > + ulong mask; > + > + mask = 1UL << hart; > + sbi_send_ipi(&mask); > + > + return 0; > +} > + > +int riscv_clear_ipi(int hart) > +{ > + sbi_clear_ipi(); > + > + return 0; > +} > -- > 2.20.1 Looks good to me. Reviewed-by: Anup Patel <anup.patel@wdc.com> Regards, Anup ^ permalink raw reply [flat|nested] 45+ messages in thread
* [U-Boot] [PATCH 3/7] riscv: implement IPI platform functions using SBI 2019-02-11 22:13 ` [U-Boot] [PATCH 3/7] riscv: implement IPI platform functions using SBI Lukas Auer 2019-02-12 2:32 ` Anup Patel @ 2019-02-12 3:03 ` Bin Meng 1 sibling, 0 replies; 45+ messages in thread From: Bin Meng @ 2019-02-12 3:03 UTC (permalink / raw) To: u-boot On Tue, Feb 12, 2019 at 6:14 AM Lukas Auer <lukas.auer@aisec.fraunhofer.de> wrote: > > The supervisor binary interface (SBI) provides the necessary functions > to implement the platform IPI functions riscv_send_ipi() and > riscv_clear_ipi(). Use it to implement them. > > This adds support for inter-processor interrupts (IPIs) on RISC-V CPUs > running in supervisor mode. Support for machine mode is already > available for CPUs that include the SiFive CLINT. > > Signed-off-by: Lukas Auer <lukas.auer@aisec.fraunhofer.de> > --- > > arch/riscv/Kconfig | 5 +++++ > arch/riscv/lib/Makefile | 1 + > arch/riscv/lib/sbi_ipi.c | 25 +++++++++++++++++++++++++ > 3 files changed, 31 insertions(+) > create mode 100644 arch/riscv/lib/sbi_ipi.c > Reviewed-by: Bin Meng <bmeng.cn@gmail.com> ^ permalink raw reply [flat|nested] 45+ messages in thread
* [U-Boot] [PATCH 4/7] riscv: delay initialization of caches and debug UART 2019-02-11 22:13 [U-Boot] [PATCH 0/7] SMP support for RISC-V Lukas Auer ` (2 preceding siblings ...) 2019-02-11 22:13 ` [U-Boot] [PATCH 3/7] riscv: implement IPI platform functions using SBI Lukas Auer @ 2019-02-11 22:13 ` Lukas Auer 2019-02-12 2:32 ` Anup Patel 2019-02-12 3:03 ` Bin Meng 2019-02-11 22:13 ` [U-Boot] [PATCH 5/7] riscv: add support for multi-hart systems Lukas Auer ` (3 subsequent siblings) 7 siblings, 2 replies; 45+ messages in thread From: Lukas Auer @ 2019-02-11 22:13 UTC (permalink / raw) To: u-boot Move the initialization of the caches and the debug UART until after board_init_f_init_reserve. This is in preparation for SMP support, where code prior to this point will be executed by all harts. This ensures that initialization will only be performed once for the main hart running U-Boot. Signed-off-by: Lukas Auer <lukas.auer@aisec.fraunhofer.de> --- arch/riscv/cpu/start.S | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/arch/riscv/cpu/start.S b/arch/riscv/cpu/start.S index 81ea52b170..a30f6f7194 100644 --- a/arch/riscv/cpu/start.S +++ b/arch/riscv/cpu/start.S @@ -45,10 +45,6 @@ _start: /* mask all interrupts */ csrw MODE_PREFIX(ie), zero - /* Enable cache */ - jal icache_enable - jal dcache_enable - /* * Set stackpointer in internal/ex RAM to call board_init_f */ @@ -57,10 +53,6 @@ call_board_init_f: li t1, CONFIG_SYS_INIT_SP_ADDR and sp, t1, t0 /* force 16 byte alignment */ -#ifdef CONFIG_DEBUG_UART - jal debug_uart_init -#endif - call_board_init_f_0: mv a0, sp jal board_init_f_alloc_reserve @@ -74,6 +66,14 @@ call_board_init_f_0: /* save the boot hart id to global_data */ SREG s0, GD_BOOT_HART(gp) + /* Enable cache */ + jal icache_enable + jal dcache_enable + +#ifdef CONFIG_DEBUG_UART + jal debug_uart_init +#endif + mv a0, zero /* a0 <-- boot_flags = 0 */ la t5, board_init_f jr t5 /* jump to board_init_f() */ -- 2.20.1 ^ permalink raw reply related [flat|nested] 45+ messages in thread
* [U-Boot] [PATCH 4/7] riscv: delay initialization of caches and debug UART 2019-02-11 22:13 ` [U-Boot] [PATCH 4/7] riscv: delay initialization of caches and debug UART Lukas Auer @ 2019-02-12 2:32 ` Anup Patel 2019-02-12 3:03 ` Bin Meng 1 sibling, 0 replies; 45+ messages in thread From: Anup Patel @ 2019-02-12 2:32 UTC (permalink / raw) To: u-boot > -----Original Message----- > From: Lukas Auer [mailto:lukas.auer at aisec.fraunhofer.de] > Sent: Tuesday, February 12, 2019 3:44 AM > To: u-boot at lists.denx.de > Cc: Atish Patra <Atish.Patra@wdc.com>; Anup Patel > <Anup.Patel@wdc.com>; Bin Meng <bmeng.cn@gmail.com>; Andreas > Schwab <schwab@suse.de>; Palmer Dabbelt <palmer@sifive.com>; > Alexander Graf <agraf@suse.de>; Lukas Auer > <lukas.auer@aisec.fraunhofer.de>; Rick Chen <rick@andestech.com>; Anup > Patel <anup@brainfault.org> > Subject: [PATCH 4/7] riscv: delay initialization of caches and debug UART > > Move the initialization of the caches and the debug UART until after > board_init_f_init_reserve. This is in preparation for SMP support, where > code prior to this point will be executed by all harts. This ensures that > initialization will only be performed once for the main hart running U-Boot. > > Signed-off-by: Lukas Auer <lukas.auer@aisec.fraunhofer.de> > --- > > arch/riscv/cpu/start.S | 16 ++++++++-------- > 1 file changed, 8 insertions(+), 8 deletions(-) > > diff --git a/arch/riscv/cpu/start.S b/arch/riscv/cpu/start.S index > 81ea52b170..a30f6f7194 100644 > --- a/arch/riscv/cpu/start.S > +++ b/arch/riscv/cpu/start.S > @@ -45,10 +45,6 @@ _start: > /* mask all interrupts */ > csrw MODE_PREFIX(ie), zero > > - /* Enable cache */ > - jal icache_enable > - jal dcache_enable > - > /* > * Set stackpointer in internal/ex RAM to call board_init_f > */ > @@ -57,10 +53,6 @@ call_board_init_f: > li t1, CONFIG_SYS_INIT_SP_ADDR > and sp, t1, t0 /* force 16 byte alignment */ > > -#ifdef CONFIG_DEBUG_UART > - jal debug_uart_init > -#endif > - > call_board_init_f_0: > mv a0, sp > jal board_init_f_alloc_reserve > @@ -74,6 +66,14 @@ call_board_init_f_0: > /* save the boot hart id to global_data */ > SREG s0, GD_BOOT_HART(gp) > > + /* Enable cache */ > + jal icache_enable > + jal dcache_enable > + > +#ifdef CONFIG_DEBUG_UART > + jal debug_uart_init > +#endif > + > mv a0, zero /* a0 <-- boot_flags = 0 */ > la t5, board_init_f > jr t5 /* jump to board_init_f() */ > -- > 2.20.1 Looks good to me. Reviewed-by: Anup Patel <anup.patel@wdc.com> Regards, Anup ^ permalink raw reply [flat|nested] 45+ messages in thread
* [U-Boot] [PATCH 4/7] riscv: delay initialization of caches and debug UART 2019-02-11 22:13 ` [U-Boot] [PATCH 4/7] riscv: delay initialization of caches and debug UART Lukas Auer 2019-02-12 2:32 ` Anup Patel @ 2019-02-12 3:03 ` Bin Meng 1 sibling, 0 replies; 45+ messages in thread From: Bin Meng @ 2019-02-12 3:03 UTC (permalink / raw) To: u-boot On Tue, Feb 12, 2019 at 6:14 AM Lukas Auer <lukas.auer@aisec.fraunhofer.de> wrote: > > Move the initialization of the caches and the debug UART until after > board_init_f_init_reserve. This is in preparation for SMP support, where > code prior to this point will be executed by all harts. This ensures > that initialization will only be performed once for the main hart > running U-Boot. > > Signed-off-by: Lukas Auer <lukas.auer@aisec.fraunhofer.de> > --- > > arch/riscv/cpu/start.S | 16 ++++++++-------- > 1 file changed, 8 insertions(+), 8 deletions(-) > Reviewed-by: Bin Meng <bmeng.cn@gmail.com> ^ permalink raw reply [flat|nested] 45+ messages in thread
* [U-Boot] [PATCH 5/7] riscv: add support for multi-hart systems 2019-02-11 22:13 [U-Boot] [PATCH 0/7] SMP support for RISC-V Lukas Auer ` (3 preceding siblings ...) 2019-02-11 22:13 ` [U-Boot] [PATCH 4/7] riscv: delay initialization of caches and debug UART Lukas Auer @ 2019-02-11 22:13 ` Lukas Auer 2019-02-12 1:48 ` Anup Patel [not found] ` <752D002CFF5D0F4FA35C0100F1D73F3FA408502C@ATCPCS16.andestech.com> 2019-02-11 22:13 ` [U-Boot] [PATCH 6/7] riscv: boot images passed to bootm on all harts Lukas Auer ` (2 subsequent siblings) 7 siblings, 2 replies; 45+ messages in thread From: Lukas Auer @ 2019-02-11 22:13 UTC (permalink / raw) To: u-boot On RISC-V, all harts boot independently. To be able to run on a multi-hart system, U-Boot must be extended with the functionality to manage all harts in the system. A new config option, CONFIG_MAIN_HART, is used to select the hart U-Boot runs on. All other harts are halted. U-Boot can delegate functions to them using smp_call_function(). Every hart has a valid pointer to the global data structure and a 8KiB stack by default. The stack size is set with CONFIG_STACK_SIZE_SHIFT. Signed-off-by: Lukas Auer <lukas.auer@aisec.fraunhofer.de> --- arch/riscv/Kconfig | 12 +++++ arch/riscv/cpu/start.S | 102 ++++++++++++++++++++++++++++++++++- arch/riscv/include/asm/csr.h | 1 + 3 files changed, 114 insertions(+), 1 deletion(-) diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index 3a51339c4d..af8d0f8d67 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -140,4 +140,16 @@ config SBI_IPI default y if RISCV_SMODE depends on SMP +config MAIN_HART + int "Main hart in system" + default 0 + help + Some SoCs include harts of various sizes, some of which might not + be suitable for running U-Boot. CONFIG_MAIN_HART is used to select + the hart U-Boot runs on. + +config STACK_SIZE_SHIFT + int + default 13 + endmenu diff --git a/arch/riscv/cpu/start.S b/arch/riscv/cpu/start.S index a30f6f7194..ce7230df37 100644 --- a/arch/riscv/cpu/start.S +++ b/arch/riscv/cpu/start.S @@ -13,6 +13,7 @@ #include <config.h> #include <common.h> #include <elf.h> +#include <asm/csr.h> #include <asm/encoding.h> #include <generated/asm-offsets.h> @@ -45,6 +46,23 @@ _start: /* mask all interrupts */ csrw MODE_PREFIX(ie), zero +#ifdef CONFIG_SMP + /* check if hart is within range */ + /* s0: hart id */ + li t0, CONFIG_NR_CPUS + bge s0, t0, hart_out_of_bounds_loop +#endif + +#ifdef CONFIG_SMP + /* set xSIE bit to receive IPIs */ +#ifdef CONFIG_RISCV_MMODE + li t0, MIE_MSIE +#else + li t0, SIE_SSIE +#endif + csrs MODE_PREFIX(ie), t0 +#endif + /* * Set stackpointer in internal/ex RAM to call board_init_f */ @@ -56,7 +74,25 @@ call_board_init_f: call_board_init_f_0: mv a0, sp jal board_init_f_alloc_reserve + + /* + * Set global data pointer here for all harts, uninitialized at this + * point. + */ + mv gp, a0 + + /* setup stack */ +#ifdef CONFIG_SMP + /* s0: hart id */ + slli t0, s0, CONFIG_STACK_SIZE_SHIFT + sub sp, a0, t0 +#else mv sp, a0 +#endif + + /* Continue on main hart, others branch to secondary_hart_loop */ + li t0, CONFIG_MAIN_HART + bne s0, t0, secondary_hart_loop la t0, prior_stage_fdt_address SREG s1, 0(t0) @@ -95,7 +131,14 @@ relocate_code: *Set up the stack */ stack_setup: +#ifdef CONFIG_SMP + /* s0: hart id */ + slli t0, s0, CONFIG_STACK_SIZE_SHIFT + sub sp, s2, t0 +#else mv sp, s2 +#endif + la t0, _start sub t6, s4, t0 /* t6 <- relocation offset */ beq t0, s4, clear_bss /* skip relocation */ @@ -175,13 +218,30 @@ clear_bss: add t0, t0, t6 /* t0 <- rel __bss_start in RAM */ la t1, __bss_end /* t1 <- rel __bss_end in FLASH */ add t1, t1, t6 /* t1 <- rel __bss_end in RAM */ - beq t0, t1, call_board_init_r + beq t0, t1, relocate_secondary_harts clbss_l: SREG zero, 0(t0) /* clear loop... */ addi t0, t0, REGBYTES bne t0, t1, clbss_l +relocate_secondary_harts: +#ifdef CONFIG_SMP + /* send relocation IPI */ + la t0, secondary_hart_relocate + add a0, t0, t6 + + /* store relocation offset */ + mv s5, t6 + + mv a1, s2 + mv a2, s3 + jal smp_call_function + + /* restore relocation offset */ + mv t6, s5 +#endif + /* * We are done. Do not return, instead branch to second part of board * initialization, now running from RAM. @@ -202,3 +262,43 @@ call_board_init_r: * jump to it ... */ jr t4 /* jump to board_init_r() */ + +#ifdef CONFIG_SMP +hart_out_of_bounds_loop: + /* Harts in this loop are out of bounds, increase CONFIG_NR_CPUS. */ + wfi + j hart_out_of_bounds_loop +#endif + +#ifdef CONFIG_SMP +/* SMP relocation entry */ +secondary_hart_relocate: + /* a1: new sp */ + /* a2: new gd */ + /* s0: hart id */ + + /* setup stack */ + slli t0, s0, CONFIG_STACK_SIZE_SHIFT + sub sp, a1, t0 + + /* update global data pointer */ + mv gp, a2 +#endif + +secondary_hart_loop: + wfi + +#ifdef CONFIG_SMP + csrr t0, MODE_PREFIX(ip) +#ifdef CONFIG_RISCV_MMODE + andi t0, t0, MIE_MSIE +#else + andi t0, t0, SIE_SSIE +#endif + beqz t0, secondary_hart_loop + + mv a0, s0 + jal handle_ipi +#endif + + j secondary_hart_loop diff --git a/arch/riscv/include/asm/csr.h b/arch/riscv/include/asm/csr.h index 86136f542c..644e6baa15 100644 --- a/arch/riscv/include/asm/csr.h +++ b/arch/riscv/include/asm/csr.h @@ -46,6 +46,7 @@ #endif /* Interrupt Enable and Interrupt Pending flags */ +#define MIE_MSIE _AC(0x00000008, UL) /* Software Interrupt Enable */ #define SIE_SSIE _AC(0x00000002, UL) /* Software Interrupt Enable */ #define SIE_STIE _AC(0x00000020, UL) /* Timer Interrupt Enable */ -- 2.20.1 ^ permalink raw reply related [flat|nested] 45+ messages in thread
* [U-Boot] [PATCH 5/7] riscv: add support for multi-hart systems 2019-02-11 22:13 ` [U-Boot] [PATCH 5/7] riscv: add support for multi-hart systems Lukas Auer @ 2019-02-12 1:48 ` Anup Patel 2019-02-17 22:02 ` Auer, Lukas [not found] ` <752D002CFF5D0F4FA35C0100F1D73F3FA408502C@ATCPCS16.andestech.com> 1 sibling, 1 reply; 45+ messages in thread From: Anup Patel @ 2019-02-12 1:48 UTC (permalink / raw) To: u-boot > -----Original Message----- > From: Lukas Auer [mailto:lukas.auer at aisec.fraunhofer.de] > Sent: Tuesday, February 12, 2019 3:44 AM > To: u-boot at lists.denx.de > Cc: Atish Patra <Atish.Patra@wdc.com>; Anup Patel > <Anup.Patel@wdc.com>; Bin Meng <bmeng.cn@gmail.com>; Andreas > Schwab <schwab@suse.de>; Palmer Dabbelt <palmer@sifive.com>; > Alexander Graf <agraf@suse.de>; Lukas Auer > <lukas.auer@aisec.fraunhofer.de>; Anup Patel <anup@brainfault.org>; Rick > Chen <rick@andestech.com>; Baruch Siach <baruch@tkos.co.il>; Stefan > Roese <sr@denx.de> > Subject: [PATCH 5/7] riscv: add support for multi-hart systems > > On RISC-V, all harts boot independently. To be able to run on a multi-hart > system, U-Boot must be extended with the functionality to manage all harts > in the system. A new config option, CONFIG_MAIN_HART, is used to select > the hart U-Boot runs on. All other harts are halted. > U-Boot can delegate functions to them using smp_call_function(). > > Every hart has a valid pointer to the global data structure and a 8KiB stack by > default. The stack size is set with CONFIG_STACK_SIZE_SHIFT. > > Signed-off-by: Lukas Auer <lukas.auer@aisec.fraunhofer.de> > --- > > arch/riscv/Kconfig | 12 +++++ > arch/riscv/cpu/start.S | 102 ++++++++++++++++++++++++++++++++++- > arch/riscv/include/asm/csr.h | 1 + > 3 files changed, 114 insertions(+), 1 deletion(-) > > diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index > 3a51339c4d..af8d0f8d67 100644 > --- a/arch/riscv/Kconfig > +++ b/arch/riscv/Kconfig > @@ -140,4 +140,16 @@ config SBI_IPI > default y if RISCV_SMODE > depends on SMP > > +config MAIN_HART > + int "Main hart in system" > + default 0 > + help > + Some SoCs include harts of various sizes, some of which might not > + be suitable for running U-Boot. CONFIG_MAIN_HART is used to > select > + the hart U-Boot runs on. This config option can be avoided altogether if we have lottery based system to select "Main HART" in start.S. With the MAIN_HART config option in-place, every system will have to pick a "Main HART". What if the "Main HART" itself does not come online due to HW failure. Regards, Anup ^ permalink raw reply [flat|nested] 45+ messages in thread
* [U-Boot] [PATCH 5/7] riscv: add support for multi-hart systems 2019-02-12 1:48 ` Anup Patel @ 2019-02-17 22:02 ` Auer, Lukas 0 siblings, 0 replies; 45+ messages in thread From: Auer, Lukas @ 2019-02-17 22:02 UTC (permalink / raw) To: u-boot On Tue, 2019-02-12 at 01:48 +0000, Anup Patel wrote: > > -----Original Message----- > > From: Lukas Auer [mailto:lukas.auer at aisec.fraunhofer.de] > > Sent: Tuesday, February 12, 2019 3:44 AM > > To: u-boot at lists.denx.de > > Cc: Atish Patra <Atish.Patra@wdc.com>; Anup Patel > > <Anup.Patel@wdc.com>; Bin Meng <bmeng.cn@gmail.com>; Andreas > > Schwab <schwab@suse.de>; Palmer Dabbelt <palmer@sifive.com>; > > Alexander Graf <agraf@suse.de>; Lukas Auer > > <lukas.auer@aisec.fraunhofer.de>; Anup Patel <anup@brainfault.org>; > > Rick > > Chen <rick@andestech.com>; Baruch Siach <baruch@tkos.co.il>; Stefan > > Roese <sr@denx.de> > > Subject: [PATCH 5/7] riscv: add support for multi-hart systems > > > > On RISC-V, all harts boot independently. To be able to run on a > > multi-hart > > system, U-Boot must be extended with the functionality to manage > > all harts > > in the system. A new config option, CONFIG_MAIN_HART, is used to > > select > > the hart U-Boot runs on. All other harts are halted. > > U-Boot can delegate functions to them using smp_call_function(). > > > > Every hart has a valid pointer to the global data structure and a > > 8KiB stack by > > default. The stack size is set with CONFIG_STACK_SIZE_SHIFT. > > > > Signed-off-by: Lukas Auer <lukas.auer@aisec.fraunhofer.de> > > --- > > > > arch/riscv/Kconfig | 12 +++++ > > arch/riscv/cpu/start.S | 102 > > ++++++++++++++++++++++++++++++++++- > > arch/riscv/include/asm/csr.h | 1 + > > 3 files changed, 114 insertions(+), 1 deletion(-) > > > > diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index > > 3a51339c4d..af8d0f8d67 100644 > > --- a/arch/riscv/Kconfig > > +++ b/arch/riscv/Kconfig > > @@ -140,4 +140,16 @@ config SBI_IPI > > default y if RISCV_SMODE > > depends on SMP > > > > +config MAIN_HART > > + int "Main hart in system" > > + default 0 > > + help > > + Some SoCs include harts of various sizes, some of which might > > not > > + be suitable for running U-Boot. CONFIG_MAIN_HART is used to > > select > > + the hart U-Boot runs on. > > This config option can be avoided altogether if we have > lottery based system to select "Main HART" in start.S. > > With the MAIN_HART config option in-place, every system > will have to pick a "Main HART". What if the "Main HART" > itself does not come online due to HW failure. > Good point, I did not consider this. I will add a lottery-based system in the next version. Thanks, Lukas ^ permalink raw reply [flat|nested] 45+ messages in thread
[parent not found: <752D002CFF5D0F4FA35C0100F1D73F3FA408502C@ATCPCS16.andestech.com>]
* [U-Boot] [PATCH 5/7] riscv: add support for multi-hart systems [not found] ` <752D002CFF5D0F4FA35C0100F1D73F3FA408502C@ATCPCS16.andestech.com> @ 2019-02-15 6:51 ` Rick Chen 2019-02-17 22:06 ` Auer, Lukas 2019-03-07 9:30 ` Rick Chen 1 sibling, 1 reply; 45+ messages in thread From: Rick Chen @ 2019-02-15 6:51 UTC (permalink / raw) To: u-boot Hi Lukas > > From: Lukas Auer [mailto:lukas.auer at aisec.fraunhofer.de] > > Sent: Tuesday, February 12, 2019 6:14 AM > > To: u-boot at lists.denx.de > > Cc: Atish Patra; Anup Patel; Bin Meng; Andreas Schwab; Palmer Dabbelt; > > Alexander Graf; Lukas Auer; Anup Patel; Rick Jian-Zhi Chen(陳建志); Baruch Siach; > > Stefan Roese > > Subject: [PATCH 5/7] riscv: add support for multi-hart systems > > > > On RISC-V, all harts boot independently. To be able to run on a multi-hart system, > > U-Boot must be extended with the functionality to manage all harts in the > > system. A new config option, CONFIG_MAIN_HART, is used to select the hart > > U-Boot runs on. All other harts are halted. > > U-Boot can delegate functions to them using smp_call_function(). > > > > Every hart has a valid pointer to the global data structure and a 8KiB stack by > > default. The stack size is set with CONFIG_STACK_SIZE_SHIFT. > > > > Signed-off-by: Lukas Auer <lukas.auer@aisec.fraunhofer.de> > > --- > > > > arch/riscv/Kconfig | 12 +++++ > > arch/riscv/cpu/start.S | 102 ++++++++++++++++++++++++++++++++++- > > arch/riscv/include/asm/csr.h | 1 + > > 3 files changed, 114 insertions(+), 1 deletion(-) > > > > diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index 3a51339c4d..af8d0f8d67 > > 100644 > > --- a/arch/riscv/Kconfig > > +++ b/arch/riscv/Kconfig > > @@ -140,4 +140,16 @@ config SBI_IPI > > default y if RISCV_SMODE > > depends on SMP > > > > +config MAIN_HART > > + int "Main hart in system" > > + default 0 > > + help > > + Some SoCs include harts of various sizes, some of which might not > > + be suitable for running U-Boot. CONFIG_MAIN_HART is used to select > > + the hart U-Boot runs on. > > + > > +config STACK_SIZE_SHIFT > > + int > > + default 13 > > + > > endmenu > > diff --git a/arch/riscv/cpu/start.S b/arch/riscv/cpu/start.S index > > a30f6f7194..ce7230df37 100644 > > --- a/arch/riscv/cpu/start.S > > +++ b/arch/riscv/cpu/start.S > > @@ -13,6 +13,7 @@ > > #include <config.h> > > #include <common.h> > > #include <elf.h> > > +#include <asm/csr.h> > > #include <asm/encoding.h> > > #include <generated/asm-offsets.h> > > If u-boot boot from flash by itself in M-mode without any FSBL or gdb, in this case there will be no chance to assign a0. Can we add some code as below : _start: +#ifdef CONFIG_RISCV_MMODE + csrr a0, mhartid +#endif /* save hart id and dtb pointer */ mv s0, a0 mv s1, a1 How do you think about it ? Thanks Rick > > @@ -45,6 +46,23 @@ _start: > > /* mask all interrupts */ > > csrw MODE_PREFIX(ie), zero > > > > +#ifdef CONFIG_SMP > > + /* check if hart is within range */ > > + /* s0: hart id */ > > + li t0, CONFIG_NR_CPUS > > + bge s0, t0, hart_out_of_bounds_loop > > +#endif ^ permalink raw reply [flat|nested] 45+ messages in thread
* [U-Boot] [PATCH 5/7] riscv: add support for multi-hart systems 2019-02-15 6:51 ` Rick Chen @ 2019-02-17 22:06 ` Auer, Lukas 0 siblings, 0 replies; 45+ messages in thread From: Auer, Lukas @ 2019-02-17 22:06 UTC (permalink / raw) To: u-boot Hi Rick, On Fri, 2019-02-15 at 14:51 +0800, Rick Chen wrote: > Hi Lukas > > > > From: Lukas Auer [mailto:lukas.auer at aisec.fraunhofer.de] > > > Sent: Tuesday, February 12, 2019 6:14 AM > > > To: u-boot at lists.denx.de > > > Cc: Atish Patra; Anup Patel; Bin Meng; Andreas Schwab; Palmer > > > Dabbelt; > > > Alexander Graf; Lukas Auer; Anup Patel; Rick Jian-Zhi Chen(陳建志); > > > Baruch Siach; > > > Stefan Roese > > > Subject: [PATCH 5/7] riscv: add support for multi-hart systems > > > > > > On RISC-V, all harts boot independently. To be able to run on a > > > multi-hart system, > > > U-Boot must be extended with the functionality to manage all > > > harts in the > > > system. A new config option, CONFIG_MAIN_HART, is used to select > > > the hart > > > U-Boot runs on. All other harts are halted. > > > U-Boot can delegate functions to them using smp_call_function(). > > > > > > Every hart has a valid pointer to the global data structure and a > > > 8KiB stack by > > > default. The stack size is set with CONFIG_STACK_SIZE_SHIFT. > > > > > > Signed-off-by: Lukas Auer <lukas.auer@aisec.fraunhofer.de> > > > --- > > > > > > arch/riscv/Kconfig | 12 +++++ > > > arch/riscv/cpu/start.S | 102 > > > ++++++++++++++++++++++++++++++++++- > > > arch/riscv/include/asm/csr.h | 1 + > > > 3 files changed, 114 insertions(+), 1 deletion(-) > > > > > > diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index > > > 3a51339c4d..af8d0f8d67 > > > 100644 > > > --- a/arch/riscv/Kconfig > > > +++ b/arch/riscv/Kconfig > > > @@ -140,4 +140,16 @@ config SBI_IPI > > > default y if RISCV_SMODE > > > depends on SMP > > > > > > +config MAIN_HART > > > + int "Main hart in system" > > > + default 0 > > > + help > > > + Some SoCs include harts of various sizes, some of which > > > might not > > > + be suitable for running U-Boot. CONFIG_MAIN_HART is used > > > to select > > > + the hart U-Boot runs on. > > > + > > > +config STACK_SIZE_SHIFT > > > + int > > > + default 13 > > > + > > > endmenu > > > diff --git a/arch/riscv/cpu/start.S b/arch/riscv/cpu/start.S > > > index > > > a30f6f7194..ce7230df37 100644 > > > --- a/arch/riscv/cpu/start.S > > > +++ b/arch/riscv/cpu/start.S > > > @@ -13,6 +13,7 @@ > > > #include <config.h> > > > #include <common.h> > > > #include <elf.h> > > > +#include <asm/csr.h> > > > #include <asm/encoding.h> > > > #include <generated/asm-offsets.h> > > > > > If u-boot boot from flash by itself in M-mode without any FSBL or > gdb, > in this case there will be no chance to assign a0. > Can we add some code as below : > > _start: > +#ifdef CONFIG_RISCV_MMODE > + csrr a0, mhartid > +#endif > /* save hart id and dtb pointer */ > mv s0, a0 > mv s1, a1 > > How do you think about it ? > Good point, thanks! Even without SMP support this could cause issues if we pass an invalid hart ID to Linux. I will add a patch for this. Thanks, Lukas > Thanks > Rick > > > > > @@ -45,6 +46,23 @@ _start: > > > /* mask all interrupts */ > > > csrw MODE_PREFIX(ie), zero > > > > > > +#ifdef CONFIG_SMP > > > + /* check if hart is within range */ > > > + /* s0: hart id */ > > > + li t0, CONFIG_NR_CPUS > > > + bge s0, t0, hart_out_of_bounds_loop > > > +#endif ^ permalink raw reply [flat|nested] 45+ messages in thread
* [U-Boot] [PATCH 5/7] riscv: add support for multi-hart systems [not found] ` <752D002CFF5D0F4FA35C0100F1D73F3FA408502C@ATCPCS16.andestech.com> 2019-02-15 6:51 ` Rick Chen @ 2019-03-07 9:30 ` Rick Chen 2019-03-10 13:58 ` Auer, Lukas 1 sibling, 1 reply; 45+ messages in thread From: Rick Chen @ 2019-03-07 9:30 UTC (permalink / raw) To: u-boot Hi Lukas > > From: Lukas Auer [mailto:lukas.auer at aisec.fraunhofer.de] > > Sent: Tuesday, February 12, 2019 6:14 AM > > To: u-boot at lists.denx.de > > Cc: Atish Patra; Anup Patel; Bin Meng; Andreas Schwab; Palmer Dabbelt; > > Alexander Graf; Lukas Auer; Anup Patel; Rick Jian-Zhi Chen(陳建志); Baruch Siach; > > Stefan Roese > > Subject: [PATCH 5/7] riscv: add support for multi-hart systems > > > > On RISC-V, all harts boot independently. To be able to run on a multi-hart system, > > U-Boot must be extended with the functionality to manage all harts in the > > system. A new config option, CONFIG_MAIN_HART, is used to select the hart > > U-Boot runs on. All other harts are halted. > > U-Boot can delegate functions to them using smp_call_function(). > > > > Every hart has a valid pointer to the global data structure and a 8KiB stack by > > default. The stack size is set with CONFIG_STACK_SIZE_SHIFT. > > > > Signed-off-by: Lukas Auer <lukas.auer@aisec.fraunhofer.de> > > --- > > > > arch/riscv/Kconfig | 12 +++++ > > arch/riscv/cpu/start.S | 102 ++++++++++++++++++++++++++++++++++- > > arch/riscv/include/asm/csr.h | 1 + > > 3 files changed, 114 insertions(+), 1 deletion(-) > > > > diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index 3a51339c4d..af8d0f8d67 > > 100644 > > --- a/arch/riscv/Kconfig > > +++ b/arch/riscv/Kconfig > > @@ -140,4 +140,16 @@ config SBI_IPI > > default y if RISCV_SMODE > > depends on SMP > > > > +config MAIN_HART > > + int "Main hart in system" > > + default 0 > > + help > > + Some SoCs include harts of various sizes, some of which might not > > + be suitable for running U-Boot. CONFIG_MAIN_HART is used to select > > + the hart U-Boot runs on. > > + > > +config STACK_SIZE_SHIFT > > + int > > + default 13 > > + > > endmenu > > diff --git a/arch/riscv/cpu/start.S b/arch/riscv/cpu/start.S index > > a30f6f7194..ce7230df37 100644 > > --- a/arch/riscv/cpu/start.S > > +++ b/arch/riscv/cpu/start.S > > @@ -13,6 +13,7 @@ > > #include <config.h> > > #include <common.h> > > #include <elf.h> > > +#include <asm/csr.h> > > #include <asm/encoding.h> > > #include <generated/asm-offsets.h> > > > > @@ -45,6 +46,23 @@ _start: > > /* mask all interrupts */ > > csrw MODE_PREFIX(ie), zero > > > > +#ifdef CONFIG_SMP > > + /* check if hart is within range */ > > + /* s0: hart id */ > > + li t0, CONFIG_NR_CPUS > > + bge s0, t0, hart_out_of_bounds_loop > > +#endif > > + > > +#ifdef CONFIG_SMP > > + /* set xSIE bit to receive IPIs */ > > +#ifdef CONFIG_RISCV_MMODE > > + li t0, MIE_MSIE > > +#else > > + li t0, SIE_SSIE > > +#endif > > + csrs MODE_PREFIX(ie), t0 > > +#endif > > + > > /* > > * Set stackpointer in internal/ex RAM to call board_init_f > > */ > > @@ -56,7 +74,25 @@ call_board_init_f: > > call_board_init_f_0: > > mv a0, sp > > jal board_init_f_alloc_reserve > > + > > + /* > > + * Set global data pointer here for all harts, uninitialized at this > > + * point. > > + */ > > + mv gp, a0 > > + > > + /* setup stack */ > > +#ifdef CONFIG_SMP > > + /* s0: hart id */ > > + slli t0, s0, CONFIG_STACK_SIZE_SHIFT > > + sub sp, a0, t0 > > +#else > > mv sp, a0 > > +#endif > > + > > + /* Continue on main hart, others branch to secondary_hart_loop */ > > + li t0, CONFIG_MAIN_HART > > + bne s0, t0, secondary_hart_loop > > > > la t0, prior_stage_fdt_address > > SREG s1, 0(t0) > > @@ -95,7 +131,14 @@ relocate_code: > > *Set up the stack > > */ > > stack_setup: > > +#ifdef CONFIG_SMP > > + /* s0: hart id */ > > + slli t0, s0, CONFIG_STACK_SIZE_SHIFT > > + sub sp, s2, t0 > > +#else > > mv sp, s2 > > +#endif > > + > > la t0, _start > > sub t6, s4, t0 /* t6 <- relocation offset */ > > beq t0, s4, clear_bss /* skip relocation */ > > @@ -175,13 +218,30 @@ clear_bss: > > add t0, t0, t6 /* t0 <- rel __bss_start in RAM */ > > la t1, __bss_end /* t1 <- rel __bss_end in FLASH */ > > add t1, t1, t6 /* t1 <- rel __bss_end in RAM */ > > - beq t0, t1, call_board_init_r > > + beq t0, t1, relocate_secondary_harts > > > > clbss_l: > > SREG zero, 0(t0) /* clear loop... */ > > addi t0, t0, REGBYTES > > bne t0, t1, clbss_l > > > > +relocate_secondary_harts: > > +#ifdef CONFIG_SMP > > + /* send relocation IPI */ > > + la t0, secondary_hart_relocate > > + add a0, t0, t6 > > + > > + /* store relocation offset */ > > + mv s5, t6 > > + > > + mv a1, s2 > > + mv a2, s3 > > + jal smp_call_function > > + > > + /* restore relocation offset */ > > + mv t6, s5 > > +#endif > > + > > /* > > * We are done. Do not return, instead branch to second part of board > > * initialization, now running from RAM. > > @@ -202,3 +262,43 @@ call_board_init_r: > > * jump to it ... > > */ > > jr t4 /* jump to board_init_r() */ > > + > > +#ifdef CONFIG_SMP > > +hart_out_of_bounds_loop: > > + /* Harts in this loop are out of bounds, increase CONFIG_NR_CPUS. */ > > + wfi > > + j hart_out_of_bounds_loop > > +#endif > > + > > +#ifdef CONFIG_SMP > > +/* SMP relocation entry */ > > +secondary_hart_relocate: > > + /* a1: new sp */ > > + /* a2: new gd */ > > + /* s0: hart id */ > > + > > + /* setup stack */ > > + slli t0, s0, CONFIG_STACK_SIZE_SHIFT > > + sub sp, a1, t0 > > + > > + /* update global data pointer */ > > + mv gp, a2 > > +#endif > > + > > +secondary_hart_loop: > > + wfi > > + > > +#ifdef CONFIG_SMP > > + csrr t0, MODE_PREFIX(ip) > > +#ifdef CONFIG_RISCV_MMODE > > + andi t0, t0, MIE_MSIE > > +#else > > + andi t0, t0, SIE_SSIE > > +#endif > > + beqz t0, secondary_hart_loop > > + > > + mv a0, s0 > > + jal handle_ipi I found that s0 maybe corrupted after execute handle_ipi. Because smp_function will be treated as a return function by compiler, so compiler will generate codes to execute restore after smp_function(). But actually it is a no-return function. So there maybe no chance to execute restore. And s0 will be corrupted somehow. The usage of s0 in v2 flow seem the same as v1. So I reply mail in v1 patch. Thanks Rick > > +#endif > > + > > + j secondary_hart_loop > > diff --git a/arch/riscv/include/asm/csr.h b/arch/riscv/include/asm/csr.h index > > 86136f542c..644e6baa15 100644 > > --- a/arch/riscv/include/asm/csr.h > > +++ b/arch/riscv/include/asm/csr.h > > @@ -46,6 +46,7 @@ > > #endif > > > > /* Interrupt Enable and Interrupt Pending flags */ > > +#define MIE_MSIE _AC(0x00000008, UL) /* Software Interrupt Enable */ > > #define SIE_SSIE _AC(0x00000002, UL) /* Software Interrupt Enable */ > > #define SIE_STIE _AC(0x00000020, UL) /* Timer Interrupt Enable */ > > > > -- > > 2.20.1 ^ permalink raw reply [flat|nested] 45+ messages in thread
* [U-Boot] [PATCH 5/7] riscv: add support for multi-hart systems 2019-03-07 9:30 ` Rick Chen @ 2019-03-10 13:58 ` Auer, Lukas 2019-03-10 14:54 ` Anup Patel 0 siblings, 1 reply; 45+ messages in thread From: Auer, Lukas @ 2019-03-10 13:58 UTC (permalink / raw) To: u-boot Hi Rick, On Thu, 2019-03-07 at 17:30 +0800, Rick Chen wrote: > Hi Lukas > > > > From: Lukas Auer [mailto:lukas.auer at aisec.fraunhofer.de] > > > Sent: Tuesday, February 12, 2019 6:14 AM > > > To: u-boot at lists.denx.de > > > Cc: Atish Patra; Anup Patel; Bin Meng; Andreas Schwab; Palmer > > > Dabbelt; > > > Alexander Graf; Lukas Auer; Anup Patel; Rick Jian-Zhi Chen(陳建志); > > > Baruch Siach; > > > Stefan Roese > > > Subject: [PATCH 5/7] riscv: add support for multi-hart systems > > > > > > On RISC-V, all harts boot independently. To be able to run on a > > > multi-hart system, > > > U-Boot must be extended with the functionality to manage all > > > harts in the > > > system. A new config option, CONFIG_MAIN_HART, is used to select > > > the hart > > > U-Boot runs on. All other harts are halted. > > > U-Boot can delegate functions to them using smp_call_function(). > > > > > > Every hart has a valid pointer to the global data structure and a > > > 8KiB stack by > > > default. The stack size is set with CONFIG_STACK_SIZE_SHIFT. > > > > > > Signed-off-by: Lukas Auer <lukas.auer@aisec.fraunhofer.de> > > > --- > > > > > > arch/riscv/Kconfig | 12 +++++ > > > arch/riscv/cpu/start.S | 102 > > > ++++++++++++++++++++++++++++++++++- > > > arch/riscv/include/asm/csr.h | 1 + > > > 3 files changed, 114 insertions(+), 1 deletion(-) > > > > > > diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index > > > 3a51339c4d..af8d0f8d67 > > > 100644 > > > --- a/arch/riscv/Kconfig > > > +++ b/arch/riscv/Kconfig > > > @@ -140,4 +140,16 @@ config SBI_IPI > > > default y if RISCV_SMODE > > > depends on SMP > > > > > > +config MAIN_HART > > > + int "Main hart in system" > > > + default 0 > > > + help > > > + Some SoCs include harts of various sizes, some of which > > > might not > > > + be suitable for running U-Boot. CONFIG_MAIN_HART is used > > > to select > > > + the hart U-Boot runs on. > > > + > > > +config STACK_SIZE_SHIFT > > > + int > > > + default 13 > > > + > > > endmenu > > > diff --git a/arch/riscv/cpu/start.S b/arch/riscv/cpu/start.S > > > index > > > a30f6f7194..ce7230df37 100644 > > > --- a/arch/riscv/cpu/start.S > > > +++ b/arch/riscv/cpu/start.S > > > @@ -13,6 +13,7 @@ > > > #include <config.h> > > > #include <common.h> > > > #include <elf.h> > > > +#include <asm/csr.h> > > > #include <asm/encoding.h> > > > #include <generated/asm-offsets.h> > > > > > > @@ -45,6 +46,23 @@ _start: > > > /* mask all interrupts */ > > > csrw MODE_PREFIX(ie), zero > > > > > > +#ifdef CONFIG_SMP > > > + /* check if hart is within range */ > > > + /* s0: hart id */ > > > + li t0, CONFIG_NR_CPUS > > > + bge s0, t0, hart_out_of_bounds_loop > > > +#endif > > > + > > > +#ifdef CONFIG_SMP > > > + /* set xSIE bit to receive IPIs */ > > > +#ifdef CONFIG_RISCV_MMODE > > > + li t0, MIE_MSIE > > > +#else > > > + li t0, SIE_SSIE > > > +#endif > > > + csrs MODE_PREFIX(ie), t0 > > > +#endif > > > + > > > /* > > > * Set stackpointer in internal/ex RAM to call board_init_f > > > */ > > > @@ -56,7 +74,25 @@ call_board_init_f: > > > call_board_init_f_0: > > > mv a0, sp > > > jal board_init_f_alloc_reserve > > > + > > > + /* > > > + * Set global data pointer here for all harts, > > > uninitialized at this > > > + * point. > > > + */ > > > + mv gp, a0 > > > + > > > + /* setup stack */ > > > +#ifdef CONFIG_SMP > > > + /* s0: hart id */ > > > + slli t0, s0, CONFIG_STACK_SIZE_SHIFT > > > + sub sp, a0, t0 > > > +#else > > > mv sp, a0 > > > +#endif > > > + > > > + /* Continue on main hart, others branch to > > > secondary_hart_loop */ > > > + li t0, CONFIG_MAIN_HART > > > + bne s0, t0, secondary_hart_loop > > > > > > la t0, prior_stage_fdt_address > > > SREG s1, 0(t0) > > > @@ -95,7 +131,14 @@ relocate_code: > > > *Set up the stack > > > */ > > > stack_setup: > > > +#ifdef CONFIG_SMP > > > + /* s0: hart id */ > > > + slli t0, s0, CONFIG_STACK_SIZE_SHIFT > > > + sub sp, s2, t0 > > > +#else > > > mv sp, s2 > > > +#endif > > > + > > > la t0, _start > > > sub t6, s4, t0 /* t6 <- relocation offset > > > */ > > > beq t0, s4, clear_bss /* skip relocation */ > > > @@ -175,13 +218,30 @@ clear_bss: > > > add t0, t0, t6 /* t0 <- rel __bss_start in > > > RAM */ > > > la t1, __bss_end /* t1 <- rel __bss_end in > > > FLASH */ > > > add t1, t1, t6 /* t1 <- rel __bss_end in > > > RAM */ > > > - beq t0, t1, call_board_init_r > > > + beq t0, t1, relocate_secondary_harts > > > > > > clbss_l: > > > SREG zero, 0(t0) /* clear loop... */ > > > addi t0, t0, REGBYTES > > > bne t0, t1, clbss_l > > > > > > +relocate_secondary_harts: > > > +#ifdef CONFIG_SMP > > > + /* send relocation IPI */ > > > + la t0, secondary_hart_relocate > > > + add a0, t0, t6 > > > + > > > + /* store relocation offset */ > > > + mv s5, t6 > > > + > > > + mv a1, s2 > > > + mv a2, s3 > > > + jal smp_call_function > > > + > > > + /* restore relocation offset */ > > > + mv t6, s5 > > > +#endif > > > + > > > /* > > > * We are done. Do not return, instead branch to second part of > > > board > > > * initialization, now running from RAM. > > > @@ -202,3 +262,43 @@ call_board_init_r: > > > * jump to it ... > > > */ > > > jr t4 /* jump to board_init_r() > > > */ > > > + > > > +#ifdef CONFIG_SMP > > > +hart_out_of_bounds_loop: > > > + /* Harts in this loop are out of bounds, increase > > > CONFIG_NR_CPUS. */ > > > + wfi > > > + j hart_out_of_bounds_loop > > > +#endif > > > + > > > +#ifdef CONFIG_SMP > > > +/* SMP relocation entry */ > > > +secondary_hart_relocate: > > > + /* a1: new sp */ > > > + /* a2: new gd */ > > > + /* s0: hart id */ > > > + > > > + /* setup stack */ > > > + slli t0, s0, CONFIG_STACK_SIZE_SHIFT > > > + sub sp, a1, t0 > > > + > > > + /* update global data pointer */ > > > + mv gp, a2 > > > +#endif > > > + > > > +secondary_hart_loop: > > > + wfi > > > + > > > +#ifdef CONFIG_SMP > > > + csrr t0, MODE_PREFIX(ip) > > > +#ifdef CONFIG_RISCV_MMODE > > > + andi t0, t0, MIE_MSIE > > > +#else > > > + andi t0, t0, SIE_SSIE > > > +#endif > > > + beqz t0, secondary_hart_loop > > > + > > > + mv a0, s0 > > > + jal handle_ipi > > I found that s0 maybe corrupted after execute handle_ipi. > Because smp_function will be treated as a return function by > compiler, > so compiler will generate codes to execute restore after > smp_function(). > > But actually it is a no-return function. So there maybe no chance to > execute > restore. And s0 will be corrupted somehow. > > The usage of s0 in v2 flow seem the same as v1. > So I reply mail in v1 patch. > Thanks for reporting this issue! What compiler version are you using? I never saw this issue with GCC 8.2.0, because optimization removes the ret following the call to smp_function(), it is called using jr. The registers are therefore restored before jumping to smp_function(). This system is probably too fragile. A simple solution would be to store the hart ID somewhere else, for example register tp. What do you think? Thanks, Lukas ^ permalink raw reply [flat|nested] 45+ messages in thread
* [U-Boot] [PATCH 5/7] riscv: add support for multi-hart systems 2019-03-10 13:58 ` Auer, Lukas @ 2019-03-10 14:54 ` Anup Patel 2019-03-10 18:12 ` Auer, Lukas 0 siblings, 1 reply; 45+ messages in thread From: Anup Patel @ 2019-03-10 14:54 UTC (permalink / raw) To: u-boot On Sun, Mar 10, 2019 at 7:28 PM Auer, Lukas <lukas.auer@aisec.fraunhofer.de> wrote: > > Hi Rick, > > On Thu, 2019-03-07 at 17:30 +0800, Rick Chen wrote: > > Hi Lukas > > > > > > From: Lukas Auer [mailto:lukas.auer at aisec.fraunhofer.de] > > > > Sent: Tuesday, February 12, 2019 6:14 AM > > > > To: u-boot at lists.denx.de > > > > Cc: Atish Patra; Anup Patel; Bin Meng; Andreas Schwab; Palmer > > > > Dabbelt; > > > > Alexander Graf; Lukas Auer; Anup Patel; Rick Jian-Zhi Chen(陳建志); > > > > Baruch Siach; > > > > Stefan Roese > > > > Subject: [PATCH 5/7] riscv: add support for multi-hart systems > > > > > > > > On RISC-V, all harts boot independently. To be able to run on a > > > > multi-hart system, > > > > U-Boot must be extended with the functionality to manage all > > > > harts in the > > > > system. A new config option, CONFIG_MAIN_HART, is used to select > > > > the hart > > > > U-Boot runs on. All other harts are halted. > > > > U-Boot can delegate functions to them using smp_call_function(). > > > > > > > > Every hart has a valid pointer to the global data structure and a > > > > 8KiB stack by > > > > default. The stack size is set with CONFIG_STACK_SIZE_SHIFT. > > > > > > > > Signed-off-by: Lukas Auer <lukas.auer@aisec.fraunhofer.de> > > > > --- > > > > > > > > arch/riscv/Kconfig | 12 +++++ > > > > arch/riscv/cpu/start.S | 102 > > > > ++++++++++++++++++++++++++++++++++- > > > > arch/riscv/include/asm/csr.h | 1 + > > > > 3 files changed, 114 insertions(+), 1 deletion(-) > > > > > > > > diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index > > > > 3a51339c4d..af8d0f8d67 > > > > 100644 > > > > --- a/arch/riscv/Kconfig > > > > +++ b/arch/riscv/Kconfig > > > > @@ -140,4 +140,16 @@ config SBI_IPI > > > > default y if RISCV_SMODE > > > > depends on SMP > > > > > > > > +config MAIN_HART > > > > + int "Main hart in system" > > > > + default 0 > > > > + help > > > > + Some SoCs include harts of various sizes, some of which > > > > might not > > > > + be suitable for running U-Boot. CONFIG_MAIN_HART is used > > > > to select > > > > + the hart U-Boot runs on. > > > > + > > > > +config STACK_SIZE_SHIFT > > > > + int > > > > + default 13 > > > > + > > > > endmenu > > > > diff --git a/arch/riscv/cpu/start.S b/arch/riscv/cpu/start.S > > > > index > > > > a30f6f7194..ce7230df37 100644 > > > > --- a/arch/riscv/cpu/start.S > > > > +++ b/arch/riscv/cpu/start.S > > > > @@ -13,6 +13,7 @@ > > > > #include <config.h> > > > > #include <common.h> > > > > #include <elf.h> > > > > +#include <asm/csr.h> > > > > #include <asm/encoding.h> > > > > #include <generated/asm-offsets.h> > > > > > > > > @@ -45,6 +46,23 @@ _start: > > > > /* mask all interrupts */ > > > > csrw MODE_PREFIX(ie), zero > > > > > > > > +#ifdef CONFIG_SMP > > > > + /* check if hart is within range */ > > > > + /* s0: hart id */ > > > > + li t0, CONFIG_NR_CPUS > > > > + bge s0, t0, hart_out_of_bounds_loop > > > > +#endif > > > > + > > > > +#ifdef CONFIG_SMP > > > > + /* set xSIE bit to receive IPIs */ > > > > +#ifdef CONFIG_RISCV_MMODE > > > > + li t0, MIE_MSIE > > > > +#else > > > > + li t0, SIE_SSIE > > > > +#endif > > > > + csrs MODE_PREFIX(ie), t0 > > > > +#endif > > > > + > > > > /* > > > > * Set stackpointer in internal/ex RAM to call board_init_f > > > > */ > > > > @@ -56,7 +74,25 @@ call_board_init_f: > > > > call_board_init_f_0: > > > > mv a0, sp > > > > jal board_init_f_alloc_reserve > > > > + > > > > + /* > > > > + * Set global data pointer here for all harts, > > > > uninitialized at this > > > > + * point. > > > > + */ > > > > + mv gp, a0 > > > > + > > > > + /* setup stack */ > > > > +#ifdef CONFIG_SMP > > > > + /* s0: hart id */ > > > > + slli t0, s0, CONFIG_STACK_SIZE_SHIFT > > > > + sub sp, a0, t0 > > > > +#else > > > > mv sp, a0 > > > > +#endif > > > > + > > > > + /* Continue on main hart, others branch to > > > > secondary_hart_loop */ > > > > + li t0, CONFIG_MAIN_HART > > > > + bne s0, t0, secondary_hart_loop > > > > > > > > la t0, prior_stage_fdt_address > > > > SREG s1, 0(t0) > > > > @@ -95,7 +131,14 @@ relocate_code: > > > > *Set up the stack > > > > */ > > > > stack_setup: > > > > +#ifdef CONFIG_SMP > > > > + /* s0: hart id */ > > > > + slli t0, s0, CONFIG_STACK_SIZE_SHIFT > > > > + sub sp, s2, t0 > > > > +#else > > > > mv sp, s2 > > > > +#endif > > > > + > > > > la t0, _start > > > > sub t6, s4, t0 /* t6 <- relocation offset > > > > */ > > > > beq t0, s4, clear_bss /* skip relocation */ > > > > @@ -175,13 +218,30 @@ clear_bss: > > > > add t0, t0, t6 /* t0 <- rel __bss_start in > > > > RAM */ > > > > la t1, __bss_end /* t1 <- rel __bss_end in > > > > FLASH */ > > > > add t1, t1, t6 /* t1 <- rel __bss_end in > > > > RAM */ > > > > - beq t0, t1, call_board_init_r > > > > + beq t0, t1, relocate_secondary_harts > > > > > > > > clbss_l: > > > > SREG zero, 0(t0) /* clear loop... */ > > > > addi t0, t0, REGBYTES > > > > bne t0, t1, clbss_l > > > > > > > > +relocate_secondary_harts: > > > > +#ifdef CONFIG_SMP > > > > + /* send relocation IPI */ > > > > + la t0, secondary_hart_relocate > > > > + add a0, t0, t6 > > > > + > > > > + /* store relocation offset */ > > > > + mv s5, t6 > > > > + > > > > + mv a1, s2 > > > > + mv a2, s3 > > > > + jal smp_call_function > > > > + > > > > + /* restore relocation offset */ > > > > + mv t6, s5 > > > > +#endif > > > > + > > > > /* > > > > * We are done. Do not return, instead branch to second part of > > > > board > > > > * initialization, now running from RAM. > > > > @@ -202,3 +262,43 @@ call_board_init_r: > > > > * jump to it ... > > > > */ > > > > jr t4 /* jump to board_init_r() > > > > */ > > > > + > > > > +#ifdef CONFIG_SMP > > > > +hart_out_of_bounds_loop: > > > > + /* Harts in this loop are out of bounds, increase > > > > CONFIG_NR_CPUS. */ > > > > + wfi > > > > + j hart_out_of_bounds_loop > > > > +#endif > > > > + > > > > +#ifdef CONFIG_SMP > > > > +/* SMP relocation entry */ > > > > +secondary_hart_relocate: > > > > + /* a1: new sp */ > > > > + /* a2: new gd */ > > > > + /* s0: hart id */ > > > > + > > > > + /* setup stack */ > > > > + slli t0, s0, CONFIG_STACK_SIZE_SHIFT > > > > + sub sp, a1, t0 > > > > + > > > > + /* update global data pointer */ > > > > + mv gp, a2 > > > > +#endif > > > > + > > > > +secondary_hart_loop: > > > > + wfi > > > > + > > > > +#ifdef CONFIG_SMP > > > > + csrr t0, MODE_PREFIX(ip) > > > > +#ifdef CONFIG_RISCV_MMODE > > > > + andi t0, t0, MIE_MSIE > > > > +#else > > > > + andi t0, t0, SIE_SSIE > > > > +#endif > > > > + beqz t0, secondary_hart_loop > > > > + > > > > + mv a0, s0 > > > > + jal handle_ipi > > > > I found that s0 maybe corrupted after execute handle_ipi. > > Because smp_function will be treated as a return function by > > compiler, > > so compiler will generate codes to execute restore after > > smp_function(). > > > > But actually it is a no-return function. So there maybe no chance to > > execute > > restore. And s0 will be corrupted somehow. > > > > The usage of s0 in v2 flow seem the same as v1. > > So I reply mail in v1 patch. > > > > Thanks for reporting this issue! > What compiler version are you using? I never saw this issue with GCC > 8.2.0, because optimization removes the ret following the call to > smp_function(), it is called using jr. The registers are therefore > restored before jumping to smp_function(). > > This system is probably too fragile. A simple solution would be to > store the hart ID somewhere else, for example register tp. What do you > think? I think you can also use scratch/mscratch (if it is not used anywhere else). Regards, Anup ^ permalink raw reply [flat|nested] 45+ messages in thread
* [U-Boot] [PATCH 5/7] riscv: add support for multi-hart systems 2019-03-10 14:54 ` Anup Patel @ 2019-03-10 18:12 ` Auer, Lukas 2019-03-10 20:56 ` Anup Patel 2019-03-12 1:15 ` Rick Chen 0 siblings, 2 replies; 45+ messages in thread From: Auer, Lukas @ 2019-03-10 18:12 UTC (permalink / raw) To: u-boot On Sun, 2019-03-10 at 20:24 +0530, Anup Patel wrote: > On Sun, Mar 10, 2019 at 7:28 PM Auer, Lukas > <lukas.auer@aisec.fraunhofer.de> wrote: > > Hi Rick, > > > > On Thu, 2019-03-07 at 17:30 +0800, Rick Chen wrote: > > > Hi Lukas > > > > > > > > From: Lukas Auer [mailto:lukas.auer at aisec.fraunhofer.de] > > > > > Sent: Tuesday, February 12, 2019 6:14 AM > > > > > To: u-boot at lists.denx.de > > > > > Cc: Atish Patra; Anup Patel; Bin Meng; Andreas Schwab; Palmer > > > > > Dabbelt; > > > > > Alexander Graf; Lukas Auer; Anup Patel; Rick Jian-Zhi > > > > > Chen(陳建志); > > > > > Baruch Siach; > > > > > Stefan Roese > > > > > Subject: [PATCH 5/7] riscv: add support for multi-hart > > > > > systems > > > > > > > > > > On RISC-V, all harts boot independently. To be able to run on > > > > > a > > > > > multi-hart system, > > > > > U-Boot must be extended with the functionality to manage all > > > > > harts in the > > > > > system. A new config option, CONFIG_MAIN_HART, is used to > > > > > select > > > > > the hart > > > > > U-Boot runs on. All other harts are halted. > > > > > U-Boot can delegate functions to them using > > > > > smp_call_function(). > > > > > > > > > > Every hart has a valid pointer to the global data structure > > > > > and a > > > > > 8KiB stack by > > > > > default. The stack size is set with CONFIG_STACK_SIZE_SHIFT. > > > > > > > > > > Signed-off-by: Lukas Auer <lukas.auer@aisec.fraunhofer.de> > > > > > --- > > > > > > > > > > arch/riscv/Kconfig | 12 +++++ > > > > > arch/riscv/cpu/start.S | 102 > > > > > ++++++++++++++++++++++++++++++++++- > > > > > arch/riscv/include/asm/csr.h | 1 + > > > > > 3 files changed, 114 insertions(+), 1 deletion(-) > > > > > > > > > > diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index > > > > > 3a51339c4d..af8d0f8d67 > > > > > 100644 > > > > > --- a/arch/riscv/Kconfig > > > > > +++ b/arch/riscv/Kconfig > > > > > @@ -140,4 +140,16 @@ config SBI_IPI > > > > > default y if RISCV_SMODE > > > > > depends on SMP > > > > > > > > > > +config MAIN_HART > > > > > + int "Main hart in system" > > > > > + default 0 > > > > > + help > > > > > + Some SoCs include harts of various sizes, some of > > > > > which > > > > > might not > > > > > + be suitable for running U-Boot. CONFIG_MAIN_HART is > > > > > used > > > > > to select > > > > > + the hart U-Boot runs on. > > > > > + > > > > > +config STACK_SIZE_SHIFT > > > > > + int > > > > > + default 13 > > > > > + > > > > > endmenu > > > > > diff --git a/arch/riscv/cpu/start.S b/arch/riscv/cpu/start.S > > > > > index > > > > > a30f6f7194..ce7230df37 100644 > > > > > --- a/arch/riscv/cpu/start.S > > > > > +++ b/arch/riscv/cpu/start.S > > > > > @@ -13,6 +13,7 @@ > > > > > #include <config.h> > > > > > #include <common.h> > > > > > #include <elf.h> > > > > > +#include <asm/csr.h> > > > > > #include <asm/encoding.h> > > > > > #include <generated/asm-offsets.h> > > > > > > > > > > @@ -45,6 +46,23 @@ _start: > > > > > /* mask all interrupts */ > > > > > csrw MODE_PREFIX(ie), zero > > > > > > > > > > +#ifdef CONFIG_SMP > > > > > + /* check if hart is within range */ > > > > > + /* s0: hart id */ > > > > > + li t0, CONFIG_NR_CPUS > > > > > + bge s0, t0, hart_out_of_bounds_loop > > > > > +#endif > > > > > + > > > > > +#ifdef CONFIG_SMP > > > > > + /* set xSIE bit to receive IPIs */ > > > > > +#ifdef CONFIG_RISCV_MMODE > > > > > + li t0, MIE_MSIE > > > > > +#else > > > > > + li t0, SIE_SSIE > > > > > +#endif > > > > > + csrs MODE_PREFIX(ie), t0 > > > > > +#endif > > > > > + > > > > > /* > > > > > * Set stackpointer in internal/ex RAM to call board_init_f > > > > > */ > > > > > @@ -56,7 +74,25 @@ call_board_init_f: > > > > > call_board_init_f_0: > > > > > mv a0, sp > > > > > jal board_init_f_alloc_reserve > > > > > + > > > > > + /* > > > > > + * Set global data pointer here for all harts, > > > > > uninitialized at this > > > > > + * point. > > > > > + */ > > > > > + mv gp, a0 > > > > > + > > > > > + /* setup stack */ > > > > > +#ifdef CONFIG_SMP > > > > > + /* s0: hart id */ > > > > > + slli t0, s0, CONFIG_STACK_SIZE_SHIFT > > > > > + sub sp, a0, t0 > > > > > +#else > > > > > mv sp, a0 > > > > > +#endif > > > > > + > > > > > + /* Continue on main hart, others branch to > > > > > secondary_hart_loop */ > > > > > + li t0, CONFIG_MAIN_HART > > > > > + bne s0, t0, secondary_hart_loop > > > > > > > > > > la t0, prior_stage_fdt_address > > > > > SREG s1, 0(t0) > > > > > @@ -95,7 +131,14 @@ relocate_code: > > > > > *Set up the stack > > > > > */ > > > > > stack_setup: > > > > > +#ifdef CONFIG_SMP > > > > > + /* s0: hart id */ > > > > > + slli t0, s0, CONFIG_STACK_SIZE_SHIFT > > > > > + sub sp, s2, t0 > > > > > +#else > > > > > mv sp, s2 > > > > > +#endif > > > > > + > > > > > la t0, _start > > > > > sub t6, s4, t0 /* t6 <- relocation > > > > > offset > > > > > */ > > > > > beq t0, s4, clear_bss /* skip relocation */ > > > > > @@ -175,13 +218,30 @@ clear_bss: > > > > > add t0, t0, t6 /* t0 <- rel > > > > > __bss_start in > > > > > RAM */ > > > > > la t1, __bss_end /* t1 <- rel __bss_end > > > > > in > > > > > FLASH */ > > > > > add t1, t1, t6 /* t1 <- rel __bss_end > > > > > in > > > > > RAM */ > > > > > - beq t0, t1, call_board_init_r > > > > > + beq t0, t1, relocate_secondary_harts > > > > > > > > > > clbss_l: > > > > > SREG zero, 0(t0) /* clear loop... */ > > > > > addi t0, t0, REGBYTES > > > > > bne t0, t1, clbss_l > > > > > > > > > > +relocate_secondary_harts: > > > > > +#ifdef CONFIG_SMP > > > > > + /* send relocation IPI */ > > > > > + la t0, secondary_hart_relocate > > > > > + add a0, t0, t6 > > > > > + > > > > > + /* store relocation offset */ > > > > > + mv s5, t6 > > > > > + > > > > > + mv a1, s2 > > > > > + mv a2, s3 > > > > > + jal smp_call_function > > > > > + > > > > > + /* restore relocation offset */ > > > > > + mv t6, s5 > > > > > +#endif > > > > > + > > > > > /* > > > > > * We are done. Do not return, instead branch to second part > > > > > of > > > > > board > > > > > * initialization, now running from RAM. > > > > > @@ -202,3 +262,43 @@ call_board_init_r: > > > > > * jump to it ... > > > > > */ > > > > > jr t4 /* jump to > > > > > board_init_r() > > > > > */ > > > > > + > > > > > +#ifdef CONFIG_SMP > > > > > +hart_out_of_bounds_loop: > > > > > + /* Harts in this loop are out of bounds, increase > > > > > CONFIG_NR_CPUS. */ > > > > > + wfi > > > > > + j hart_out_of_bounds_loop > > > > > +#endif > > > > > + > > > > > +#ifdef CONFIG_SMP > > > > > +/* SMP relocation entry */ > > > > > +secondary_hart_relocate: > > > > > + /* a1: new sp */ > > > > > + /* a2: new gd */ > > > > > + /* s0: hart id */ > > > > > + > > > > > + /* setup stack */ > > > > > + slli t0, s0, CONFIG_STACK_SIZE_SHIFT > > > > > + sub sp, a1, t0 > > > > > + > > > > > + /* update global data pointer */ > > > > > + mv gp, a2 > > > > > +#endif > > > > > + > > > > > +secondary_hart_loop: > > > > > + wfi > > > > > + > > > > > +#ifdef CONFIG_SMP > > > > > + csrr t0, MODE_PREFIX(ip) > > > > > +#ifdef CONFIG_RISCV_MMODE > > > > > + andi t0, t0, MIE_MSIE > > > > > +#else > > > > > + andi t0, t0, SIE_SSIE > > > > > +#endif > > > > > + beqz t0, secondary_hart_loop > > > > > + > > > > > + mv a0, s0 > > > > > + jal handle_ipi > > > > > > I found that s0 maybe corrupted after execute handle_ipi. > > > Because smp_function will be treated as a return function by > > > compiler, > > > so compiler will generate codes to execute restore after > > > smp_function(). > > > > > > But actually it is a no-return function. So there maybe no chance > > > to > > > execute > > > restore. And s0 will be corrupted somehow. > > > > > > The usage of s0 in v2 flow seem the same as v1. > > > So I reply mail in v1 patch. > > > > > > > Thanks for reporting this issue! > > What compiler version are you using? I never saw this issue with > > GCC > > 8.2.0, because optimization removes the ret following the call to > > smp_function(), it is called using jr. The registers are therefore > > restored before jumping to smp_function(). > > > > This system is probably too fragile. A simple solution would be to > > store the hart ID somewhere else, for example register tp. What do > > you > > think? > > I think you can also use scratch/mscratch (if it is not used anywhere > else). > You are right, sscratch and mscratch are also not currently used in U-Boot. Is there an advantage to use the scratch CSRs instead of tp? Thanks, Lukas ^ permalink raw reply [flat|nested] 45+ messages in thread
* [U-Boot] [PATCH 5/7] riscv: add support for multi-hart systems 2019-03-10 18:12 ` Auer, Lukas @ 2019-03-10 20:56 ` Anup Patel 2019-03-12 1:15 ` Rick Chen 1 sibling, 0 replies; 45+ messages in thread From: Anup Patel @ 2019-03-10 20:56 UTC (permalink / raw) To: u-boot > -----Original Message----- > From: U-Boot <u-boot-bounces@lists.denx.de> On Behalf Of Auer, Lukas > Sent: Sunday, March 10, 2019 11:42 PM > To: anup at brainfault.org > Cc: rickchen36 at gmail.com; baruch at tkos.co.il; sr at denx.de; > cmchen at andestech.com; greentime at andestech.com; schwab at suse.de; > palmer at sifive.com; agraf at suse.de; u-boot at lists.denx.de; > kito at andestech.com > Subject: Re: [U-Boot] [PATCH 5/7] riscv: add support for multi-hart systems > > On Sun, 2019-03-10 at 20:24 +0530, Anup Patel wrote: > > On Sun, Mar 10, 2019 at 7:28 PM Auer, Lukas > > <lukas.auer@aisec.fraunhofer.de> wrote: > > > Hi Rick, > > > > > > On Thu, 2019-03-07 at 17:30 +0800, Rick Chen wrote: > > > > Hi Lukas > > > > > > > > > > From: Lukas Auer [mailto:lukas.auer at aisec.fraunhofer.de] > > > > > > Sent: Tuesday, February 12, 2019 6:14 AM > > > > > > To: u-boot at lists.denx.de > > > > > > Cc: Atish Patra; Anup Patel; Bin Meng; Andreas Schwab; Palmer > > > > > > Dabbelt; Alexander Graf; Lukas Auer; Anup Patel; Rick Jian-Zhi > > > > > > Chen(陳建志); > > > > > > Baruch Siach; > > > > > > Stefan Roese > > > > > > Subject: [PATCH 5/7] riscv: add support for multi-hart systems > > > > > > > > > > > > On RISC-V, all harts boot independently. To be able to run on > > > > > > a multi-hart system, U-Boot must be extended with the > > > > > > functionality to manage all harts in the system. A new config > > > > > > option, CONFIG_MAIN_HART, is used to select the hart U-Boot > > > > > > runs on. All other harts are halted. > > > > > > U-Boot can delegate functions to them using > > > > > > smp_call_function(). > > > > > > > > > > > > Every hart has a valid pointer to the global data structure > > > > > > and a 8KiB stack by default. The stack size is set with > > > > > > CONFIG_STACK_SIZE_SHIFT. > > > > > > > > > > > > Signed-off-by: Lukas Auer <lukas.auer@aisec.fraunhofer.de> > > > > > > --- > > > > > > > > > > > > arch/riscv/Kconfig | 12 +++++ > > > > > > arch/riscv/cpu/start.S | 102 > > > > > > ++++++++++++++++++++++++++++++++++- > > > > > > arch/riscv/include/asm/csr.h | 1 + > > > > > > 3 files changed, 114 insertions(+), 1 deletion(-) > > > > > > > > > > > > diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index > > > > > > 3a51339c4d..af8d0f8d67 > > > > > > 100644 > > > > > > --- a/arch/riscv/Kconfig > > > > > > +++ b/arch/riscv/Kconfig > > > > > > @@ -140,4 +140,16 @@ config SBI_IPI > > > > > > default y if RISCV_SMODE > > > > > > depends on SMP > > > > > > > > > > > > +config MAIN_HART > > > > > > + int "Main hart in system" > > > > > > + default 0 > > > > > > + help > > > > > > + Some SoCs include harts of various sizes, some of > > > > > > which > > > > > > might not > > > > > > + be suitable for running U-Boot. CONFIG_MAIN_HART is > > > > > > used > > > > > > to select > > > > > > + the hart U-Boot runs on. > > > > > > + > > > > > > +config STACK_SIZE_SHIFT > > > > > > + int > > > > > > + default 13 > > > > > > + > > > > > > endmenu > > > > > > diff --git a/arch/riscv/cpu/start.S b/arch/riscv/cpu/start.S > > > > > > index > > > > > > a30f6f7194..ce7230df37 100644 > > > > > > --- a/arch/riscv/cpu/start.S > > > > > > +++ b/arch/riscv/cpu/start.S > > > > > > @@ -13,6 +13,7 @@ > > > > > > #include <config.h> > > > > > > #include <common.h> > > > > > > #include <elf.h> > > > > > > +#include <asm/csr.h> > > > > > > #include <asm/encoding.h> > > > > > > #include <generated/asm-offsets.h> > > > > > > > > > > > > @@ -45,6 +46,23 @@ _start: > > > > > > /* mask all interrupts */ > > > > > > csrw MODE_PREFIX(ie), zero > > > > > > > > > > > > +#ifdef CONFIG_SMP > > > > > > + /* check if hart is within range */ > > > > > > + /* s0: hart id */ > > > > > > + li t0, CONFIG_NR_CPUS > > > > > > + bge s0, t0, hart_out_of_bounds_loop > > > > > > +#endif > > > > > > + > > > > > > +#ifdef CONFIG_SMP > > > > > > + /* set xSIE bit to receive IPIs */ #ifdef > > > > > > +CONFIG_RISCV_MMODE > > > > > > + li t0, MIE_MSIE > > > > > > +#else > > > > > > + li t0, SIE_SSIE > > > > > > +#endif > > > > > > + csrs MODE_PREFIX(ie), t0 > > > > > > +#endif > > > > > > + > > > > > > /* > > > > > > * Set stackpointer in internal/ex RAM to call board_init_f > > > > > > */ > > > > > > @@ -56,7 +74,25 @@ call_board_init_f: > > > > > > call_board_init_f_0: > > > > > > mv a0, sp > > > > > > jal board_init_f_alloc_reserve > > > > > > + > > > > > > + /* > > > > > > + * Set global data pointer here for all harts, > > > > > > uninitialized at this > > > > > > + * point. > > > > > > + */ > > > > > > + mv gp, a0 > > > > > > + > > > > > > + /* setup stack */ > > > > > > +#ifdef CONFIG_SMP > > > > > > + /* s0: hart id */ > > > > > > + slli t0, s0, CONFIG_STACK_SIZE_SHIFT > > > > > > + sub sp, a0, t0 > > > > > > +#else > > > > > > mv sp, a0 > > > > > > +#endif > > > > > > + > > > > > > + /* Continue on main hart, others branch to > > > > > > secondary_hart_loop */ > > > > > > + li t0, CONFIG_MAIN_HART > > > > > > + bne s0, t0, secondary_hart_loop > > > > > > > > > > > > la t0, prior_stage_fdt_address > > > > > > SREG s1, 0(t0) > > > > > > @@ -95,7 +131,14 @@ relocate_code: > > > > > > *Set up the stack > > > > > > */ > > > > > > stack_setup: > > > > > > +#ifdef CONFIG_SMP > > > > > > + /* s0: hart id */ > > > > > > + slli t0, s0, CONFIG_STACK_SIZE_SHIFT > > > > > > + sub sp, s2, t0 > > > > > > +#else > > > > > > mv sp, s2 > > > > > > +#endif > > > > > > + > > > > > > la t0, _start > > > > > > sub t6, s4, t0 /* t6 <- relocation > > > > > > offset > > > > > > */ > > > > > > beq t0, s4, clear_bss /* skip relocation */ > > > > > > @@ -175,13 +218,30 @@ clear_bss: > > > > > > add t0, t0, t6 /* t0 <- rel > > > > > > __bss_start in > > > > > > RAM */ > > > > > > la t1, __bss_end /* t1 <- rel __bss_end > > > > > > in > > > > > > FLASH */ > > > > > > add t1, t1, t6 /* t1 <- rel __bss_end > > > > > > in > > > > > > RAM */ > > > > > > - beq t0, t1, call_board_init_r > > > > > > + beq t0, t1, relocate_secondary_harts > > > > > > > > > > > > clbss_l: > > > > > > SREG zero, 0(t0) /* clear loop... */ > > > > > > addi t0, t0, REGBYTES > > > > > > bne t0, t1, clbss_l > > > > > > > > > > > > +relocate_secondary_harts: > > > > > > +#ifdef CONFIG_SMP > > > > > > + /* send relocation IPI */ > > > > > > + la t0, secondary_hart_relocate > > > > > > + add a0, t0, t6 > > > > > > + > > > > > > + /* store relocation offset */ > > > > > > + mv s5, t6 > > > > > > + > > > > > > + mv a1, s2 > > > > > > + mv a2, s3 > > > > > > + jal smp_call_function > > > > > > + > > > > > > + /* restore relocation offset */ > > > > > > + mv t6, s5 > > > > > > +#endif > > > > > > + > > > > > > /* > > > > > > * We are done. Do not return, instead branch to second part > > > > > > of board > > > > > > * initialization, now running from RAM. > > > > > > @@ -202,3 +262,43 @@ call_board_init_r: > > > > > > * jump to it ... > > > > > > */ > > > > > > jr t4 /* jump to > > > > > > board_init_r() > > > > > > */ > > > > > > + > > > > > > +#ifdef CONFIG_SMP > > > > > > +hart_out_of_bounds_loop: > > > > > > + /* Harts in this loop are out of bounds, increase > > > > > > CONFIG_NR_CPUS. */ > > > > > > + wfi > > > > > > + j hart_out_of_bounds_loop > > > > > > +#endif > > > > > > + > > > > > > +#ifdef CONFIG_SMP > > > > > > +/* SMP relocation entry */ > > > > > > +secondary_hart_relocate: > > > > > > + /* a1: new sp */ > > > > > > + /* a2: new gd */ > > > > > > + /* s0: hart id */ > > > > > > + > > > > > > + /* setup stack */ > > > > > > + slli t0, s0, CONFIG_STACK_SIZE_SHIFT > > > > > > + sub sp, a1, t0 > > > > > > + > > > > > > + /* update global data pointer */ > > > > > > + mv gp, a2 > > > > > > +#endif > > > > > > + > > > > > > +secondary_hart_loop: > > > > > > + wfi > > > > > > + > > > > > > +#ifdef CONFIG_SMP > > > > > > + csrr t0, MODE_PREFIX(ip) > > > > > > +#ifdef CONFIG_RISCV_MMODE > > > > > > + andi t0, t0, MIE_MSIE > > > > > > +#else > > > > > > + andi t0, t0, SIE_SSIE > > > > > > +#endif > > > > > > + beqz t0, secondary_hart_loop > > > > > > + > > > > > > + mv a0, s0 > > > > > > + jal handle_ipi > > > > > > > > I found that s0 maybe corrupted after execute handle_ipi. > > > > Because smp_function will be treated as a return function by > > > > compiler, so compiler will generate codes to execute restore after > > > > smp_function(). > > > > > > > > But actually it is a no-return function. So there maybe no chance > > > > to execute restore. And s0 will be corrupted somehow. > > > > > > > > The usage of s0 in v2 flow seem the same as v1. > > > > So I reply mail in v1 patch. > > > > > > > > > > Thanks for reporting this issue! > > > What compiler version are you using? I never saw this issue with GCC > > > 8.2.0, because optimization removes the ret following the call to > > > smp_function(), it is called using jr. The registers are therefore > > > restored before jumping to smp_function(). > > > > > > This system is probably too fragile. A simple solution would be to > > > store the hart ID somewhere else, for example register tp. What do > > > you think? > > > > I think you can also use scratch/mscratch (if it is not used anywhere > > else). > > > > You are right, sscratch and mscratch are also not currently used in U-Boot. Is > there an advantage to use the scratch CSRs instead of tp? No specific advantage as such. This was another possibility. I suggest you go ahead with TP because mscratch will be used by M-mode U-Boot when linked with OpenSBI as library. Regards, Anup ^ permalink raw reply [flat|nested] 45+ messages in thread
* [U-Boot] [PATCH 5/7] riscv: add support for multi-hart systems 2019-03-10 18:12 ` Auer, Lukas 2019-03-10 20:56 ` Anup Patel @ 2019-03-12 1:15 ` Rick Chen 2019-03-14 21:04 ` Auer, Lukas 1 sibling, 1 reply; 45+ messages in thread From: Rick Chen @ 2019-03-12 1:15 UTC (permalink / raw) To: u-boot Hi Lukas Auer, Lukas <lukas.auer@aisec.fraunhofer.de> 於 2019年3月11日 週一 上午2:12寫道: > > On Sun, 2019-03-10 at 20:24 +0530, Anup Patel wrote: > > On Sun, Mar 10, 2019 at 7:28 PM Auer, Lukas > > <lukas.auer@aisec.fraunhofer.de> wrote: > > > Hi Rick, > > > > > > On Thu, 2019-03-07 at 17:30 +0800, Rick Chen wrote: > > > > Hi Lukas > > > > > > > > > > From: Lukas Auer [mailto:lukas.auer at aisec.fraunhofer.de] > > > > > > Sent: Tuesday, February 12, 2019 6:14 AM > > > > > > To: u-boot at lists.denx.de > > > > > > Cc: Atish Patra; Anup Patel; Bin Meng; Andreas Schwab; Palmer > > > > > > Dabbelt; > > > > > > Alexander Graf; Lukas Auer; Anup Patel; Rick Jian-Zhi > > > > > > Chen(陳建志); > > > > > > Baruch Siach; > > > > > > Stefan Roese > > > > > > Subject: [PATCH 5/7] riscv: add support for multi-hart > > > > > > systems > > > > > > > > > > > > On RISC-V, all harts boot independently. To be able to run on > > > > > > a > > > > > > multi-hart system, > > > > > > U-Boot must be extended with the functionality to manage all > > > > > > harts in the > > > > > > system. A new config option, CONFIG_MAIN_HART, is used to > > > > > > select > > > > > > the hart > > > > > > U-Boot runs on. All other harts are halted. > > > > > > U-Boot can delegate functions to them using > > > > > > smp_call_function(). > > > > > > > > > > > > Every hart has a valid pointer to the global data structure > > > > > > and a > > > > > > 8KiB stack by > > > > > > default. The stack size is set with CONFIG_STACK_SIZE_SHIFT. > > > > > > > > > > > > Signed-off-by: Lukas Auer <lukas.auer@aisec.fraunhofer.de> > > > > > > --- > > > > > > > > > > > > arch/riscv/Kconfig | 12 +++++ > > > > > > arch/riscv/cpu/start.S | 102 > > > > > > ++++++++++++++++++++++++++++++++++- > > > > > > arch/riscv/include/asm/csr.h | 1 + > > > > > > 3 files changed, 114 insertions(+), 1 deletion(-) > > > > > > > > > > > > diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index > > > > > > 3a51339c4d..af8d0f8d67 > > > > > > 100644 > > > > > > --- a/arch/riscv/Kconfig > > > > > > +++ b/arch/riscv/Kconfig > > > > > > @@ -140,4 +140,16 @@ config SBI_IPI > > > > > > default y if RISCV_SMODE > > > > > > depends on SMP > > > > > > > > > > > > +config MAIN_HART > > > > > > + int "Main hart in system" > > > > > > + default 0 > > > > > > + help > > > > > > + Some SoCs include harts of various sizes, some of > > > > > > which > > > > > > might not > > > > > > + be suitable for running U-Boot. CONFIG_MAIN_HART is > > > > > > used > > > > > > to select > > > > > > + the hart U-Boot runs on. > > > > > > + > > > > > > +config STACK_SIZE_SHIFT > > > > > > + int > > > > > > + default 13 > > > > > > + > > > > > > endmenu > > > > > > diff --git a/arch/riscv/cpu/start.S b/arch/riscv/cpu/start.S > > > > > > index > > > > > > a30f6f7194..ce7230df37 100644 > > > > > > --- a/arch/riscv/cpu/start.S > > > > > > +++ b/arch/riscv/cpu/start.S > > > > > > @@ -13,6 +13,7 @@ > > > > > > #include <config.h> > > > > > > #include <common.h> > > > > > > #include <elf.h> > > > > > > +#include <asm/csr.h> > > > > > > #include <asm/encoding.h> > > > > > > #include <generated/asm-offsets.h> > > > > > > > > > > > > @@ -45,6 +46,23 @@ _start: > > > > > > /* mask all interrupts */ > > > > > > csrw MODE_PREFIX(ie), zero > > > > > > > > > > > > +#ifdef CONFIG_SMP > > > > > > + /* check if hart is within range */ > > > > > > + /* s0: hart id */ > > > > > > + li t0, CONFIG_NR_CPUS > > > > > > + bge s0, t0, hart_out_of_bounds_loop > > > > > > +#endif > > > > > > + > > > > > > +#ifdef CONFIG_SMP > > > > > > + /* set xSIE bit to receive IPIs */ > > > > > > +#ifdef CONFIG_RISCV_MMODE > > > > > > + li t0, MIE_MSIE > > > > > > +#else > > > > > > + li t0, SIE_SSIE > > > > > > +#endif > > > > > > + csrs MODE_PREFIX(ie), t0 > > > > > > +#endif > > > > > > + > > > > > > /* > > > > > > * Set stackpointer in internal/ex RAM to call board_init_f > > > > > > */ > > > > > > @@ -56,7 +74,25 @@ call_board_init_f: > > > > > > call_board_init_f_0: > > > > > > mv a0, sp > > > > > > jal board_init_f_alloc_reserve > > > > > > + > > > > > > + /* > > > > > > + * Set global data pointer here for all harts, > > > > > > uninitialized at this > > > > > > + * point. > > > > > > + */ > > > > > > + mv gp, a0 > > > > > > + > > > > > > + /* setup stack */ > > > > > > +#ifdef CONFIG_SMP > > > > > > + /* s0: hart id */ > > > > > > + slli t0, s0, CONFIG_STACK_SIZE_SHIFT > > > > > > + sub sp, a0, t0 > > > > > > +#else > > > > > > mv sp, a0 > > > > > > +#endif > > > > > > + > > > > > > + /* Continue on main hart, others branch to > > > > > > secondary_hart_loop */ > > > > > > + li t0, CONFIG_MAIN_HART > > > > > > + bne s0, t0, secondary_hart_loop > > > > > > > > > > > > la t0, prior_stage_fdt_address > > > > > > SREG s1, 0(t0) > > > > > > @@ -95,7 +131,14 @@ relocate_code: > > > > > > *Set up the stack > > > > > > */ > > > > > > stack_setup: > > > > > > +#ifdef CONFIG_SMP > > > > > > + /* s0: hart id */ > > > > > > + slli t0, s0, CONFIG_STACK_SIZE_SHIFT > > > > > > + sub sp, s2, t0 > > > > > > +#else > > > > > > mv sp, s2 > > > > > > +#endif > > > > > > + > > > > > > la t0, _start > > > > > > sub t6, s4, t0 /* t6 <- relocation > > > > > > offset > > > > > > */ > > > > > > beq t0, s4, clear_bss /* skip relocation */ > > > > > > @@ -175,13 +218,30 @@ clear_bss: > > > > > > add t0, t0, t6 /* t0 <- rel > > > > > > __bss_start in > > > > > > RAM */ > > > > > > la t1, __bss_end /* t1 <- rel __bss_end > > > > > > in > > > > > > FLASH */ > > > > > > add t1, t1, t6 /* t1 <- rel __bss_end > > > > > > in > > > > > > RAM */ > > > > > > - beq t0, t1, call_board_init_r > > > > > > + beq t0, t1, relocate_secondary_harts > > > > > > > > > > > > clbss_l: > > > > > > SREG zero, 0(t0) /* clear loop... */ > > > > > > addi t0, t0, REGBYTES > > > > > > bne t0, t1, clbss_l > > > > > > > > > > > > +relocate_secondary_harts: > > > > > > +#ifdef CONFIG_SMP > > > > > > + /* send relocation IPI */ > > > > > > + la t0, secondary_hart_relocate > > > > > > + add a0, t0, t6 > > > > > > + > > > > > > + /* store relocation offset */ > > > > > > + mv s5, t6 > > > > > > + > > > > > > + mv a1, s2 > > > > > > + mv a2, s3 > > > > > > + jal smp_call_function > > > > > > + > > > > > > + /* restore relocation offset */ > > > > > > + mv t6, s5 > > > > > > +#endif > > > > > > + > > > > > > /* > > > > > > * We are done. Do not return, instead branch to second part > > > > > > of > > > > > > board > > > > > > * initialization, now running from RAM. > > > > > > @@ -202,3 +262,43 @@ call_board_init_r: > > > > > > * jump to it ... > > > > > > */ > > > > > > jr t4 /* jump to > > > > > > board_init_r() > > > > > > */ > > > > > > + > > > > > > +#ifdef CONFIG_SMP > > > > > > +hart_out_of_bounds_loop: > > > > > > + /* Harts in this loop are out of bounds, increase > > > > > > CONFIG_NR_CPUS. */ > > > > > > + wfi > > > > > > + j hart_out_of_bounds_loop > > > > > > +#endif > > > > > > + > > > > > > +#ifdef CONFIG_SMP > > > > > > +/* SMP relocation entry */ > > > > > > +secondary_hart_relocate: > > > > > > + /* a1: new sp */ > > > > > > + /* a2: new gd */ > > > > > > + /* s0: hart id */ > > > > > > + > > > > > > + /* setup stack */ > > > > > > + slli t0, s0, CONFIG_STACK_SIZE_SHIFT > > > > > > + sub sp, a1, t0 > > > > > > + > > > > > > + /* update global data pointer */ > > > > > > + mv gp, a2 > > > > > > +#endif > > > > > > + > > > > > > +secondary_hart_loop: > > > > > > + wfi > > > > > > + > > > > > > +#ifdef CONFIG_SMP > > > > > > + csrr t0, MODE_PREFIX(ip) > > > > > > +#ifdef CONFIG_RISCV_MMODE > > > > > > + andi t0, t0, MIE_MSIE > > > > > > +#else > > > > > > + andi t0, t0, SIE_SSIE > > > > > > +#endif > > > > > > + beqz t0, secondary_hart_loop > > > > > > + > > > > > > + mv a0, s0 > > > > > > + jal handle_ipi > > > > > > > > I found that s0 maybe corrupted after execute handle_ipi. > > > > Because smp_function will be treated as a return function by > > > > compiler, > > > > so compiler will generate codes to execute restore after > > > > smp_function(). > > > > > > > > But actually it is a no-return function. So there maybe no chance > > > > to > > > > execute > > > > restore. And s0 will be corrupted somehow. > > > > > > > > The usage of s0 in v2 flow seem the same as v1. > > > > So I reply mail in v1 patch. > > > > > > > > > > Thanks for reporting this issue! > > > What compiler version are you using? I never saw this issue with > > > GCC > > > 8.2.0, because optimization removes the ret following the call to > > > smp_function(), it is called using jr. The registers are therefore > > > restored before jumping to smp_function(). > > > I found this issue with Andestech's toolchain (GCC version is 7.3.0) And it's code gen will be as below: 00000e94 <handle_ipi>: e94: 4785 c.li a5,1 e96: 04a7e663 bltu a5,a0,ee2 <handle_ipi+0x4e> e9a: 456362ef jal t0,372f0 <__riscv_save_0> e9e: 84aa c.mv s1,a0 ea0: 3539 c.jal cae <riscv_clear_ipi> ea2: cd19 c.beqz a0,ec0 <handle_ipi+0x2c> ea4: 0003d517 auipc a0,0x3d ea8: 7c450513 addi a0,a0,1988 # 3e668 <freqs.8638+0x430> eac: 17c320ef jal ra,33028 <printf> eb0: 0003d517 auipc a0,0x3d eb4: 7b850513 addi a0,a0,1976 # 3e668 <freqs.8638+0x430> eb8: 170320ef jal ra,33028 <printf> ebc: 4583606f j 37314 <__riscv_restore_0> ec0: 47b1 c.li a5,12 ec2: 02f48433 mul s0,s1,a5 s0 : 1 => 0xc ec6: 003407b3 add a5,s0,gp eca: 0bc7a903 lw s2,188(a5) # 800000bc <__bss_end+0x7ff8f718> ece: 3b21 c.jal be6 <invalidate_icache_all> ed0: 008187b3 add a5,gp,s0 ed4: 0c47a603 lw a2,196(a5) ed8: 0c07a583 lw a1,192(a5) edc: 8526 c.mv a0,s1 ede: 9902 c.jalr s2 s2=0x3ff8f194 , s0=0xcsou ee0: bff1 c.j ebc <handle_ipi+0x28> ee2: 8082 c.jr ra As I said last mail, compiler will treat smp_function() as a return call, so restore() will be executed after it. This behavior shall comply with the calling convention. Thanks for your understanding and tolerant. > > > This system is probably too fragile. A simple solution would be to > > > store the hart ID somewhere else, for example register tp. What do > > > you > > > think? > > > > I think you can also use scratch/mscratch (if it is not used anywhere > > else). > > > > You are right, sscratch and mscratch are also not currently used in > U-Boot. Is there an advantage to use the scratch CSRs instead of tp? > I will prefer to use tp. xscratch shall consider mode distinguish additionally. Rick > Thanks, > Lukas ^ permalink raw reply [flat|nested] 45+ messages in thread
* [U-Boot] [PATCH 5/7] riscv: add support for multi-hart systems 2019-03-12 1:15 ` Rick Chen @ 2019-03-14 21:04 ` Auer, Lukas 0 siblings, 0 replies; 45+ messages in thread From: Auer, Lukas @ 2019-03-14 21:04 UTC (permalink / raw) To: u-boot Hi Rick, On Tue, 2019-03-12 at 09:15 +0800, Rick Chen wrote: > Hi Lukas > > Auer, Lukas <lukas.auer@aisec.fraunhofer.de> 於 2019年3月11日 週一 > 上午2:12寫道: > > On Sun, 2019-03-10 at 20:24 +0530, Anup Patel wrote: > > > On Sun, Mar 10, 2019 at 7:28 PM Auer, Lukas > > > <lukas.auer@aisec.fraunhofer.de> wrote: > > > > Hi Rick, > > > > > > > > On Thu, 2019-03-07 at 17:30 +0800, Rick Chen wrote: > > > > > Hi Lukas > > > > > > > > > > > > From: Lukas Auer [mailto:lukas.auer at aisec.fraunhofer.de] > > > > > > > Sent: Tuesday, February 12, 2019 6:14 AM > > > > > > > To: u-boot at lists.denx.de > > > > > > > Cc: Atish Patra; Anup Patel; Bin Meng; Andreas Schwab; > > > > > > > Palmer > > > > > > > Dabbelt; > > > > > > > Alexander Graf; Lukas Auer; Anup Patel; Rick Jian-Zhi > > > > > > > Chen(陳建志); > > > > > > > Baruch Siach; > > > > > > > Stefan Roese > > > > > > > Subject: [PATCH 5/7] riscv: add support for multi-hart > > > > > > > systems > > > > > > > > > > > > > > On RISC-V, all harts boot independently. To be able to > > > > > > > run on > > > > > > > a > > > > > > > multi-hart system, > > > > > > > U-Boot must be extended with the functionality to manage > > > > > > > all > > > > > > > harts in the > > > > > > > system. A new config option, CONFIG_MAIN_HART, is used to > > > > > > > select > > > > > > > the hart > > > > > > > U-Boot runs on. All other harts are halted. > > > > > > > U-Boot can delegate functions to them using > > > > > > > smp_call_function(). > > > > > > > > > > > > > > Every hart has a valid pointer to the global data > > > > > > > structure > > > > > > > and a > > > > > > > 8KiB stack by > > > > > > > default. The stack size is set with > > > > > > > CONFIG_STACK_SIZE_SHIFT. > > > > > > > > > > > > > > Signed-off-by: Lukas Auer <lukas.auer@aisec.fraunhofer.de > > > > > > > > > > > > > > > --- > > > > > > > > > > > > > > arch/riscv/Kconfig | 12 +++++ > > > > > > > arch/riscv/cpu/start.S | 102 > > > > > > > ++++++++++++++++++++++++++++++++++- > > > > > > > arch/riscv/include/asm/csr.h | 1 + > > > > > > > 3 files changed, 114 insertions(+), 1 deletion(-) > > > > > > > > > > > > > > diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig > > > > > > > index > > > > > > > 3a51339c4d..af8d0f8d67 > > > > > > > 100644 > > > > > > > --- a/arch/riscv/Kconfig > > > > > > > +++ b/arch/riscv/Kconfig > > > > > > > @@ -140,4 +140,16 @@ config SBI_IPI > > > > > > > default y if RISCV_SMODE > > > > > > > depends on SMP > > > > > > > > > > > > > > +config MAIN_HART > > > > > > > + int "Main hart in system" > > > > > > > + default 0 > > > > > > > + help > > > > > > > + Some SoCs include harts of various sizes, some of > > > > > > > which > > > > > > > might not > > > > > > > + be suitable for running U-Boot. CONFIG_MAIN_HART > > > > > > > is > > > > > > > used > > > > > > > to select > > > > > > > + the hart U-Boot runs on. > > > > > > > + > > > > > > > +config STACK_SIZE_SHIFT > > > > > > > + int > > > > > > > + default 13 > > > > > > > + > > > > > > > endmenu > > > > > > > diff --git a/arch/riscv/cpu/start.S > > > > > > > b/arch/riscv/cpu/start.S > > > > > > > index > > > > > > > a30f6f7194..ce7230df37 100644 > > > > > > > --- a/arch/riscv/cpu/start.S > > > > > > > +++ b/arch/riscv/cpu/start.S > > > > > > > @@ -13,6 +13,7 @@ > > > > > > > #include <config.h> > > > > > > > #include <common.h> > > > > > > > #include <elf.h> > > > > > > > +#include <asm/csr.h> > > > > > > > #include <asm/encoding.h> > > > > > > > #include <generated/asm-offsets.h> > > > > > > > > > > > > > > @@ -45,6 +46,23 @@ _start: > > > > > > > /* mask all interrupts */ > > > > > > > csrw MODE_PREFIX(ie), zero > > > > > > > > > > > > > > +#ifdef CONFIG_SMP > > > > > > > + /* check if hart is within range */ > > > > > > > + /* s0: hart id */ > > > > > > > + li t0, CONFIG_NR_CPUS > > > > > > > + bge s0, t0, hart_out_of_bounds_loop > > > > > > > +#endif > > > > > > > + > > > > > > > +#ifdef CONFIG_SMP > > > > > > > + /* set xSIE bit to receive IPIs */ > > > > > > > +#ifdef CONFIG_RISCV_MMODE > > > > > > > + li t0, MIE_MSIE > > > > > > > +#else > > > > > > > + li t0, SIE_SSIE > > > > > > > +#endif > > > > > > > + csrs MODE_PREFIX(ie), t0 > > > > > > > +#endif > > > > > > > + > > > > > > > /* > > > > > > > * Set stackpointer in internal/ex RAM to call > > > > > > > board_init_f > > > > > > > */ > > > > > > > @@ -56,7 +74,25 @@ call_board_init_f: > > > > > > > call_board_init_f_0: > > > > > > > mv a0, sp > > > > > > > jal board_init_f_alloc_reserve > > > > > > > + > > > > > > > + /* > > > > > > > + * Set global data pointer here for all harts, > > > > > > > uninitialized at this > > > > > > > + * point. > > > > > > > + */ > > > > > > > + mv gp, a0 > > > > > > > + > > > > > > > + /* setup stack */ > > > > > > > +#ifdef CONFIG_SMP > > > > > > > + /* s0: hart id */ > > > > > > > + slli t0, s0, CONFIG_STACK_SIZE_SHIFT > > > > > > > + sub sp, a0, t0 > > > > > > > +#else > > > > > > > mv sp, a0 > > > > > > > +#endif > > > > > > > + > > > > > > > + /* Continue on main hart, others branch to > > > > > > > secondary_hart_loop */ > > > > > > > + li t0, CONFIG_MAIN_HART > > > > > > > + bne s0, t0, secondary_hart_loop > > > > > > > > > > > > > > la t0, prior_stage_fdt_address > > > > > > > SREG s1, 0(t0) > > > > > > > @@ -95,7 +131,14 @@ relocate_code: > > > > > > > *Set up the stack > > > > > > > */ > > > > > > > stack_setup: > > > > > > > +#ifdef CONFIG_SMP > > > > > > > + /* s0: hart id */ > > > > > > > + slli t0, s0, CONFIG_STACK_SIZE_SHIFT > > > > > > > + sub sp, s2, t0 > > > > > > > +#else > > > > > > > mv sp, s2 > > > > > > > +#endif > > > > > > > + > > > > > > > la t0, _start > > > > > > > sub t6, s4, t0 /* t6 <- relocation > > > > > > > offset > > > > > > > */ > > > > > > > beq t0, s4, clear_bss /* skip relocation > > > > > > > */ > > > > > > > @@ -175,13 +218,30 @@ clear_bss: > > > > > > > add t0, t0, t6 /* t0 <- rel > > > > > > > __bss_start in > > > > > > > RAM */ > > > > > > > la t1, __bss_end /* t1 <- rel > > > > > > > __bss_end > > > > > > > in > > > > > > > FLASH */ > > > > > > > add t1, t1, t6 /* t1 <- rel > > > > > > > __bss_end > > > > > > > in > > > > > > > RAM */ > > > > > > > - beq t0, t1, call_board_init_r > > > > > > > + beq t0, t1, relocate_secondary_harts > > > > > > > > > > > > > > clbss_l: > > > > > > > SREG zero, 0(t0) /* clear loop... */ > > > > > > > addi t0, t0, REGBYTES > > > > > > > bne t0, t1, clbss_l > > > > > > > > > > > > > > +relocate_secondary_harts: > > > > > > > +#ifdef CONFIG_SMP > > > > > > > + /* send relocation IPI */ > > > > > > > + la t0, secondary_hart_relocate > > > > > > > + add a0, t0, t6 > > > > > > > + > > > > > > > + /* store relocation offset */ > > > > > > > + mv s5, t6 > > > > > > > + > > > > > > > + mv a1, s2 > > > > > > > + mv a2, s3 > > > > > > > + jal smp_call_function > > > > > > > + > > > > > > > + /* restore relocation offset */ > > > > > > > + mv t6, s5 > > > > > > > +#endif > > > > > > > + > > > > > > > /* > > > > > > > * We are done. Do not return, instead branch to second > > > > > > > part > > > > > > > of > > > > > > > board > > > > > > > * initialization, now running from RAM. > > > > > > > @@ -202,3 +262,43 @@ call_board_init_r: > > > > > > > * jump to it ... > > > > > > > */ > > > > > > > jr t4 /* jump to > > > > > > > board_init_r() > > > > > > > */ > > > > > > > + > > > > > > > +#ifdef CONFIG_SMP > > > > > > > +hart_out_of_bounds_loop: > > > > > > > + /* Harts in this loop are out of bounds, increase > > > > > > > CONFIG_NR_CPUS. */ > > > > > > > + wfi > > > > > > > + j hart_out_of_bounds_loop > > > > > > > +#endif > > > > > > > + > > > > > > > +#ifdef CONFIG_SMP > > > > > > > +/* SMP relocation entry */ > > > > > > > +secondary_hart_relocate: > > > > > > > + /* a1: new sp */ > > > > > > > + /* a2: new gd */ > > > > > > > + /* s0: hart id */ > > > > > > > + > > > > > > > + /* setup stack */ > > > > > > > + slli t0, s0, CONFIG_STACK_SIZE_SHIFT > > > > > > > + sub sp, a1, t0 > > > > > > > + > > > > > > > + /* update global data pointer */ > > > > > > > + mv gp, a2 > > > > > > > +#endif > > > > > > > + > > > > > > > +secondary_hart_loop: > > > > > > > + wfi > > > > > > > + > > > > > > > +#ifdef CONFIG_SMP > > > > > > > + csrr t0, MODE_PREFIX(ip) > > > > > > > +#ifdef CONFIG_RISCV_MMODE > > > > > > > + andi t0, t0, MIE_MSIE > > > > > > > +#else > > > > > > > + andi t0, t0, SIE_SSIE > > > > > > > +#endif > > > > > > > + beqz t0, secondary_hart_loop > > > > > > > + > > > > > > > + mv a0, s0 > > > > > > > + jal handle_ipi > > > > > > > > > > I found that s0 maybe corrupted after execute handle_ipi. > > > > > Because smp_function will be treated as a return function by > > > > > compiler, > > > > > so compiler will generate codes to execute restore after > > > > > smp_function(). > > > > > > > > > > But actually it is a no-return function. So there maybe no > > > > > chance > > > > > to > > > > > execute > > > > > restore. And s0 will be corrupted somehow. > > > > > > > > > > The usage of s0 in v2 flow seem the same as v1. > > > > > So I reply mail in v1 patch. > > > > > > > > > > > > > Thanks for reporting this issue! > > > > What compiler version are you using? I never saw this issue > > > > with > > > > GCC > > > > 8.2.0, because optimization removes the ret following the call > > > > to > > > > smp_function(), it is called using jr. The registers are > > > > therefore > > > > restored before jumping to smp_function(). > > > > > > I found this issue with Andestech's toolchain (GCC version is 7.3.0) > And it's code gen will be as below: > > 00000e94 <handle_ipi>: > e94: 4785 c.li a5,1 > e96: 04a7e663 bltu a5,a0,ee2 > <handle_ipi+0x4e> > e9a: 456362ef jal t0,372f0 > <__riscv_save_0> > e9e: 84aa c.mv s1,a0 > ea0: 3539 c.jal cae <riscv_clear_ipi> > ea2: cd19 c.beqz a0,ec0 > <handle_ipi+0x2c> > ea4: 0003d517 auipc a0,0x3d > ea8: 7c450513 addi a0,a0,1988 # 3e668 > <freqs.8638+0x430> > eac: 17c320ef jal ra,33028 <printf> > eb0: 0003d517 auipc a0,0x3d > eb4: 7b850513 addi a0,a0,1976 # 3e668 > <freqs.8638+0x430> > eb8: 170320ef jal ra,33028 <printf> > ebc: 4583606f j 37314 > <__riscv_restore_0> > ec0: 47b1 c.li a5,12 > ec2: 02f48433 mul s0,s1,a5 s0 : 1 => > 0xc > ec6: 003407b3 add a5,s0,gp > eca: 0bc7a903 lw s2,188(a5) # 800000bc > <__bss_end+0x7ff8f718> > ece: 3b21 c.jal be6 > <invalidate_icache_all> > ed0: 008187b3 add a5,gp,s0 > ed4: 0c47a603 lw a2,196(a5) > ed8: 0c07a583 lw a1,192(a5) > edc: 8526 c.mv a0,s1 > ede: 9902 c.jalr s2 s2=0x3ff8f194 , > s0=0xcsou > ee0: bff1 c.j ebc <handle_ipi+0x28> > ee2: 8082 c.jr ra > > As I said last mail, compiler will treat smp_function() as a return > call, so restore() will be executed after it. > This behavior shall comply with the calling convention. > > Thanks for your understanding and tolerant. No problem, I was just asking out of interest. Thanks for providing the toolchain output above! > > > > > This system is probably too fragile. A simple solution would be > > > > to > > > > store the hart ID somewhere else, for example register tp. What > > > > do > > > > you > > > > think? > > > > > > I think you can also use scratch/mscratch (if it is not used > > > anywhere > > > else). > > > > > > > You are right, sscratch and mscratch are also not currently used in > > U-Boot. Is there an advantage to use the scratch CSRs instead of > > tp? > > > > I will prefer to use tp. > xscratch shall consider mode distinguish additionally. > Ok, I will update my series and use tp instead of s0. Thanks, Lukas ^ permalink raw reply [flat|nested] 45+ messages in thread
* [U-Boot] [PATCH 6/7] riscv: boot images passed to bootm on all harts 2019-02-11 22:13 [U-Boot] [PATCH 0/7] SMP support for RISC-V Lukas Auer ` (4 preceding siblings ...) 2019-02-11 22:13 ` [U-Boot] [PATCH 5/7] riscv: add support for multi-hart systems Lukas Auer @ 2019-02-11 22:13 ` Lukas Auer 2019-02-12 2:33 ` Anup Patel 2019-02-12 3:04 ` Bin Meng 2019-02-11 22:13 ` [U-Boot] [PATCH 7/7] riscv: qemu: enable SMP Lukas Auer 2019-02-11 22:16 ` [U-Boot] [PATCH 0/7] SMP support for RISC-V Philipp Tomsich 7 siblings, 2 replies; 45+ messages in thread From: Lukas Auer @ 2019-02-11 22:13 UTC (permalink / raw) To: u-boot Signed-off-by: Lukas Auer <lukas.auer@aisec.fraunhofer.de> --- arch/riscv/lib/bootm.c | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/arch/riscv/lib/bootm.c b/arch/riscv/lib/bootm.c index f36b8702ef..efbd3e23e7 100644 --- a/arch/riscv/lib/bootm.c +++ b/arch/riscv/lib/bootm.c @@ -13,6 +13,7 @@ #include <image.h> #include <asm/byteorder.h> #include <asm/csr.h> +#include <asm/smp.h> #include <dm/device.h> #include <dm/root.h> #include <u-boot/zlib.h> @@ -81,6 +82,9 @@ static void boot_jump_linux(bootm_headers_t *images, int flag) { void (*kernel)(ulong hart, void *dtb); int fake = (flag & BOOTM_STATE_OS_FAKE_GO); +#ifdef CONFIG_SMP + int ret; +#endif kernel = (void (*)(ulong, void *))images->ep; @@ -92,8 +96,15 @@ static void boot_jump_linux(bootm_headers_t *images, int flag) announce_and_cleanup(fake); if (!fake) { - if (IMAGE_ENABLE_OF_LIBFDT && images->ft_len) + if (IMAGE_ENABLE_OF_LIBFDT && images->ft_len) { +#ifdef CONFIG_SMP + ret = smp_call_function(images->ep, + (ulong)images->ft_addr, 0); + if (ret) + hang(); +#endif kernel(gd->arch.boot_hart, images->ft_addr); + } } } -- 2.20.1 ^ permalink raw reply related [flat|nested] 45+ messages in thread
* [U-Boot] [PATCH 6/7] riscv: boot images passed to bootm on all harts 2019-02-11 22:13 ` [U-Boot] [PATCH 6/7] riscv: boot images passed to bootm on all harts Lukas Auer @ 2019-02-12 2:33 ` Anup Patel 2019-02-12 3:04 ` Bin Meng 1 sibling, 0 replies; 45+ messages in thread From: Anup Patel @ 2019-02-12 2:33 UTC (permalink / raw) To: u-boot > -----Original Message----- > From: Lukas Auer [mailto:lukas.auer at aisec.fraunhofer.de] > Sent: Tuesday, February 12, 2019 3:44 AM > To: u-boot at lists.denx.de > Cc: Atish Patra <Atish.Patra@wdc.com>; Anup Patel > <Anup.Patel@wdc.com>; Bin Meng <bmeng.cn@gmail.com>; Andreas > Schwab <schwab@suse.de>; Palmer Dabbelt <palmer@sifive.com>; > Alexander Graf <agraf@suse.de>; Lukas Auer > <lukas.auer@aisec.fraunhofer.de>; Anup Patel <anup@brainfault.org>; Rick > Chen <rick@andestech.com>; Simon Glass <sjg@chromium.org> > Subject: [PATCH 6/7] riscv: boot images passed to bootm on all harts > > Signed-off-by: Lukas Auer <lukas.auer@aisec.fraunhofer.de> > --- > > arch/riscv/lib/bootm.c | 13 ++++++++++++- > 1 file changed, 12 insertions(+), 1 deletion(-) > > diff --git a/arch/riscv/lib/bootm.c b/arch/riscv/lib/bootm.c index > f36b8702ef..efbd3e23e7 100644 > --- a/arch/riscv/lib/bootm.c > +++ b/arch/riscv/lib/bootm.c > @@ -13,6 +13,7 @@ > #include <image.h> > #include <asm/byteorder.h> > #include <asm/csr.h> > +#include <asm/smp.h> > #include <dm/device.h> > #include <dm/root.h> > #include <u-boot/zlib.h> > @@ -81,6 +82,9 @@ static void boot_jump_linux(bootm_headers_t > *images, int flag) { > void (*kernel)(ulong hart, void *dtb); > int fake = (flag & BOOTM_STATE_OS_FAKE_GO); > +#ifdef CONFIG_SMP > + int ret; > +#endif > > kernel = (void (*)(ulong, void *))images->ep; > > @@ -92,8 +96,15 @@ static void boot_jump_linux(bootm_headers_t > *images, int flag) > announce_and_cleanup(fake); > > if (!fake) { > - if (IMAGE_ENABLE_OF_LIBFDT && images->ft_len) > + if (IMAGE_ENABLE_OF_LIBFDT && images->ft_len) { #ifdef > CONFIG_SMP > + ret = smp_call_function(images->ep, > + (ulong)images->ft_addr, 0); > + if (ret) > + hang(); > +#endif > kernel(gd->arch.boot_hart, images->ft_addr); > + } > } > } > > -- > 2.20.1 Looks good to me. Reviewed-by: Anup Patel <anup.patel@wdc.com> Regards, Anup ^ permalink raw reply [flat|nested] 45+ messages in thread
* [U-Boot] [PATCH 6/7] riscv: boot images passed to bootm on all harts 2019-02-11 22:13 ` [U-Boot] [PATCH 6/7] riscv: boot images passed to bootm on all harts Lukas Auer 2019-02-12 2:33 ` Anup Patel @ 2019-02-12 3:04 ` Bin Meng 1 sibling, 0 replies; 45+ messages in thread From: Bin Meng @ 2019-02-12 3:04 UTC (permalink / raw) To: u-boot On Tue, Feb 12, 2019 at 6:14 AM Lukas Auer <lukas.auer@aisec.fraunhofer.de> wrote: > > Signed-off-by: Lukas Auer <lukas.auer@aisec.fraunhofer.de> > --- > > arch/riscv/lib/bootm.c | 13 ++++++++++++- > 1 file changed, 12 insertions(+), 1 deletion(-) > Reviewed-by: Bin Meng <bmeng.cn@gmail.com> ^ permalink raw reply [flat|nested] 45+ messages in thread
* [U-Boot] [PATCH 7/7] riscv: qemu: enable SMP 2019-02-11 22:13 [U-Boot] [PATCH 0/7] SMP support for RISC-V Lukas Auer ` (5 preceding siblings ...) 2019-02-11 22:13 ` [U-Boot] [PATCH 6/7] riscv: boot images passed to bootm on all harts Lukas Auer @ 2019-02-11 22:13 ` Lukas Auer 2019-02-12 2:34 ` Anup Patel 2019-02-12 3:05 ` Bin Meng 2019-02-11 22:16 ` [U-Boot] [PATCH 0/7] SMP support for RISC-V Philipp Tomsich 7 siblings, 2 replies; 45+ messages in thread From: Lukas Auer @ 2019-02-11 22:13 UTC (permalink / raw) To: u-boot Signed-off-by: Lukas Auer <lukas.auer@aisec.fraunhofer.de> --- board/emulation/qemu-riscv/Kconfig | 1 + 1 file changed, 1 insertion(+) diff --git a/board/emulation/qemu-riscv/Kconfig b/board/emulation/qemu-riscv/Kconfig index 0d865acf10..b3300c64a8 100644 --- a/board/emulation/qemu-riscv/Kconfig +++ b/board/emulation/qemu-riscv/Kconfig @@ -34,5 +34,6 @@ config BOARD_SPECIFIC_OPTIONS # dummy imply BOARD_LATE_INIT imply OF_BOARD_SETUP imply SIFIVE_SERIAL + imply SMP endif -- 2.20.1 ^ permalink raw reply related [flat|nested] 45+ messages in thread
* [U-Boot] [PATCH 7/7] riscv: qemu: enable SMP 2019-02-11 22:13 ` [U-Boot] [PATCH 7/7] riscv: qemu: enable SMP Lukas Auer @ 2019-02-12 2:34 ` Anup Patel 2019-02-12 3:05 ` Bin Meng 1 sibling, 0 replies; 45+ messages in thread From: Anup Patel @ 2019-02-12 2:34 UTC (permalink / raw) To: u-boot > -----Original Message----- > From: Lukas Auer [mailto:lukas.auer at aisec.fraunhofer.de] > Sent: Tuesday, February 12, 2019 3:44 AM > To: u-boot at lists.denx.de > Cc: Atish Patra <Atish.Patra@wdc.com>; Anup Patel > <Anup.Patel@wdc.com>; Bin Meng <bmeng.cn@gmail.com>; Andreas > Schwab <schwab@suse.de>; Palmer Dabbelt <palmer@sifive.com>; > Alexander Graf <agraf@suse.de>; Lukas Auer > <lukas.auer@aisec.fraunhofer.de> > Subject: [PATCH 7/7] riscv: qemu: enable SMP > > Signed-off-by: Lukas Auer <lukas.auer@aisec.fraunhofer.de> > --- > > board/emulation/qemu-riscv/Kconfig | 1 + > 1 file changed, 1 insertion(+) > > diff --git a/board/emulation/qemu-riscv/Kconfig b/board/emulation/qemu- > riscv/Kconfig > index 0d865acf10..b3300c64a8 100644 > --- a/board/emulation/qemu-riscv/Kconfig > +++ b/board/emulation/qemu-riscv/Kconfig > @@ -34,5 +34,6 @@ config BOARD_SPECIFIC_OPTIONS # dummy > imply BOARD_LATE_INIT > imply OF_BOARD_SETUP > imply SIFIVE_SERIAL > + imply SMP > > endif > -- > 2.20.1 Looks good to me. Reviewed-by: Anup Patel <anup.patel@wdc.com> Regards, Anup ^ permalink raw reply [flat|nested] 45+ messages in thread
* [U-Boot] [PATCH 7/7] riscv: qemu: enable SMP 2019-02-11 22:13 ` [U-Boot] [PATCH 7/7] riscv: qemu: enable SMP Lukas Auer 2019-02-12 2:34 ` Anup Patel @ 2019-02-12 3:05 ` Bin Meng 1 sibling, 0 replies; 45+ messages in thread From: Bin Meng @ 2019-02-12 3:05 UTC (permalink / raw) To: u-boot On Tue, Feb 12, 2019 at 6:14 AM Lukas Auer <lukas.auer@aisec.fraunhofer.de> wrote: > > Signed-off-by: Lukas Auer <lukas.auer@aisec.fraunhofer.de> > --- > > board/emulation/qemu-riscv/Kconfig | 1 + > 1 file changed, 1 insertion(+) > Reviewed-by: Bin Meng <bmeng.cn@gmail.com> ^ permalink raw reply [flat|nested] 45+ messages in thread
* [U-Boot] [PATCH 0/7] SMP support for RISC-V 2019-02-11 22:13 [U-Boot] [PATCH 0/7] SMP support for RISC-V Lukas Auer ` (6 preceding siblings ...) 2019-02-11 22:13 ` [U-Boot] [PATCH 7/7] riscv: qemu: enable SMP Lukas Auer @ 2019-02-11 22:16 ` Philipp Tomsich 2019-02-11 22:44 ` Auer, Lukas 7 siblings, 1 reply; 45+ messages in thread From: Philipp Tomsich @ 2019-02-11 22:16 UTC (permalink / raw) To: u-boot On 11.02.2019, at 23:13, Lukas Auer <lukas.auer@aisec.fraunhofer.de> wrote: > > This patch series adds SMP support for RISC-V to U-Boot. It allows > U-Boot to run on multi-hart systems and will boot images passed to bootm > on all harts. The bootm command is currently the only one that will boot > images on all harts, bootefi is not yet supported. You might want to clarify somewhere that a ‘hart’ is RISC-V terminology for a hardware thread. > > The patches have been successfully tested on both QEMU (machine and > supervisor mode) and the HiFive Unleashed board [1] (supervisor mode), > using BBL and OpenSBI. > Mainline QEMU requires two patches [2, 3] to run in this configuration. > I will send a follow-up patch to enable SMP support on the HiFive > Unleashed board. > > [1]: https://patchwork.ozlabs.org/project/uboot/list/?series=91125 > [2]: https://patchwork.ozlabs.org/patch/1039493/ > [3]: https://patchwork.ozlabs.org/patch/1039082/ > > > Lukas Auer (7): > riscv: add infrastructure for calling functions on other harts > riscv: import the supervisor binary interface header file > riscv: implement IPI platform functions using SBI > riscv: delay initialization of caches and debug UART > riscv: add support for multi-hart systems > riscv: boot images passed to bootm on all harts > riscv: qemu: enable SMP > > arch/riscv/Kconfig | 36 +++++++++ > arch/riscv/cpu/start.S | 116 +++++++++++++++++++++++++-- > arch/riscv/include/asm/csr.h | 1 + > arch/riscv/include/asm/global_data.h | 5 ++ > arch/riscv/include/asm/sbi.h | 94 ++++++++++++++++++++++ > arch/riscv/include/asm/smp.h | 53 ++++++++++++ > arch/riscv/lib/Makefile | 2 + > arch/riscv/lib/bootm.c | 13 ++- > arch/riscv/lib/sbi_ipi.c | 25 ++++++ > arch/riscv/lib/smp.c | 110 +++++++++++++++++++++++++ > board/emulation/qemu-riscv/Kconfig | 1 + > 11 files changed, 447 insertions(+), 9 deletions(-) > create mode 100644 arch/riscv/include/asm/sbi.h > create mode 100644 arch/riscv/include/asm/smp.h > create mode 100644 arch/riscv/lib/sbi_ipi.c > create mode 100644 arch/riscv/lib/smp.c > > -- > 2.20.1 > > _______________________________________________ > U-Boot mailing list > U-Boot at lists.denx.de > https://lists.denx.de/listinfo/u-boot ^ permalink raw reply [flat|nested] 45+ messages in thread
* [U-Boot] [PATCH 0/7] SMP support for RISC-V 2019-02-11 22:16 ` [U-Boot] [PATCH 0/7] SMP support for RISC-V Philipp Tomsich @ 2019-02-11 22:44 ` Auer, Lukas 0 siblings, 0 replies; 45+ messages in thread From: Auer, Lukas @ 2019-02-11 22:44 UTC (permalink / raw) To: u-boot On Mon, 2019-02-11 at 23:16 +0100, Philipp Tomsich wrote: > On 11.02.2019, at 23:13, Lukas Auer <lukas.auer@aisec.fraunhofer.de> > wrote: > > This patch series adds SMP support for RISC-V to U-Boot. It allows > > U-Boot to run on multi-hart systems and will boot images passed to > > bootm > > on all harts. The bootm command is currently the only one that will > > boot > > images on all harts, bootefi is not yet supported. > > You might want to clarify somewhere that a ‘hart’ is RISC-V > terminology > for a hardware thread. > Good point, I will add a note in the next version. Thanks! Lukas > > The patches have been successfully tested on both QEMU (machine and > > supervisor mode) and the HiFive Unleashed board [1] (supervisor > > mode), > > using BBL and OpenSBI. > > Mainline QEMU requires two patches [2, 3] to run in this > > configuration. > > I will send a follow-up patch to enable SMP support on the HiFive > > Unleashed board. > > > > [1]: https://patchwork.ozlabs.org/project/uboot/list/?series=91125 > > [2]: https://patchwork.ozlabs.org/patch/1039493/ > > [3]: https://patchwork.ozlabs.org/patch/1039082/ > > > > > > Lukas Auer (7): > > riscv: add infrastructure for calling functions on other harts > > riscv: import the supervisor binary interface header file > > riscv: implement IPI platform functions using SBI > > riscv: delay initialization of caches and debug UART > > riscv: add support for multi-hart systems > > riscv: boot images passed to bootm on all harts > > riscv: qemu: enable SMP > > > > arch/riscv/Kconfig | 36 +++++++++ > > arch/riscv/cpu/start.S | 116 > > +++++++++++++++++++++++++-- > > arch/riscv/include/asm/csr.h | 1 + > > arch/riscv/include/asm/global_data.h | 5 ++ > > arch/riscv/include/asm/sbi.h | 94 ++++++++++++++++++++++ > > arch/riscv/include/asm/smp.h | 53 ++++++++++++ > > arch/riscv/lib/Makefile | 2 + > > arch/riscv/lib/bootm.c | 13 ++- > > arch/riscv/lib/sbi_ipi.c | 25 ++++++ > > arch/riscv/lib/smp.c | 110 > > +++++++++++++++++++++++++ > > board/emulation/qemu-riscv/Kconfig | 1 + > > 11 files changed, 447 insertions(+), 9 deletions(-) > > create mode 100644 arch/riscv/include/asm/sbi.h > > create mode 100644 arch/riscv/include/asm/smp.h > > create mode 100644 arch/riscv/lib/sbi_ipi.c > > create mode 100644 arch/riscv/lib/smp.c > > > > -- > > 2.20.1 > > > > _______________________________________________ > > U-Boot mailing list > > U-Boot at lists.denx.de > > https://lists.denx.de/listinfo/u-boot ^ permalink raw reply [flat|nested] 45+ messages in thread
end of thread, other threads:[~2019-03-14 21:04 UTC | newest] Thread overview: 45+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2019-02-11 22:13 [U-Boot] [PATCH 0/7] SMP support for RISC-V Lukas Auer 2019-02-11 22:13 ` [U-Boot] [PATCH 1/7] riscv: add infrastructure for calling functions on other harts Lukas Auer 2019-02-12 1:44 ` Anup Patel 2019-02-17 21:55 ` Auer, Lukas [not found] ` <752D002CFF5D0F4FA35C0100F1D73F3FA4087D3C@ATCPCS16.andestech.com> 2019-02-18 3:24 ` Rick Chen 2019-02-18 3:40 ` Anup Patel 2019-02-18 9:53 ` Auer, Lukas 2019-02-12 3:03 ` Bin Meng 2019-02-17 22:00 ` Auer, Lukas 2019-02-18 4:58 ` Anup Patel 2019-02-18 10:01 ` Auer, Lukas 2019-02-18 11:41 ` Auer, Lukas 2019-02-18 13:16 ` Anup Patel 2019-02-18 16:04 ` Auer, Lukas 2019-02-19 8:16 ` Anup Patel 2019-02-25 11:13 ` Auer, Lukas 2019-02-11 22:13 ` [U-Boot] [PATCH 2/7] riscv: import the supervisor binary interface header file Lukas Auer 2019-02-12 2:32 ` Anup Patel 2019-02-12 3:03 ` Bin Meng 2019-02-11 22:13 ` [U-Boot] [PATCH 3/7] riscv: implement IPI platform functions using SBI Lukas Auer 2019-02-12 2:32 ` Anup Patel 2019-02-12 3:03 ` Bin Meng 2019-02-11 22:13 ` [U-Boot] [PATCH 4/7] riscv: delay initialization of caches and debug UART Lukas Auer 2019-02-12 2:32 ` Anup Patel 2019-02-12 3:03 ` Bin Meng 2019-02-11 22:13 ` [U-Boot] [PATCH 5/7] riscv: add support for multi-hart systems Lukas Auer 2019-02-12 1:48 ` Anup Patel 2019-02-17 22:02 ` Auer, Lukas [not found] ` <752D002CFF5D0F4FA35C0100F1D73F3FA408502C@ATCPCS16.andestech.com> 2019-02-15 6:51 ` Rick Chen 2019-02-17 22:06 ` Auer, Lukas 2019-03-07 9:30 ` Rick Chen 2019-03-10 13:58 ` Auer, Lukas 2019-03-10 14:54 ` Anup Patel 2019-03-10 18:12 ` Auer, Lukas 2019-03-10 20:56 ` Anup Patel 2019-03-12 1:15 ` Rick Chen 2019-03-14 21:04 ` Auer, Lukas 2019-02-11 22:13 ` [U-Boot] [PATCH 6/7] riscv: boot images passed to bootm on all harts Lukas Auer 2019-02-12 2:33 ` Anup Patel 2019-02-12 3:04 ` Bin Meng 2019-02-11 22:13 ` [U-Boot] [PATCH 7/7] riscv: qemu: enable SMP Lukas Auer 2019-02-12 2:34 ` Anup Patel 2019-02-12 3:05 ` Bin Meng 2019-02-11 22:16 ` [U-Boot] [PATCH 0/7] SMP support for RISC-V Philipp Tomsich 2019-02-11 22:44 ` Auer, Lukas
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.