[U-Boot] [PATCH v2 0/9] SMP support for RISC-V

[U-Boot] [PATCH v2 0/9] SMP support for RISC-V

[Lukas Auer]

This patch series adds SMP support for RISC-V to U-Boot. It allows
U-Boot to run on multi-hart systems (hart is the RISC-V terminology for
hardware thread). Images passed to bootm will be started on all harts.
The bootm command is currently the only one that will boot images on all
harts, bootefi is not yet supported.

The patches have been successfully tested on both QEMU (machine and
supervisor mode) and the HiFive Unleashed board (supervisor mode), using
BBL and OpenSBI.
Mainline QEMU requires two patches [1, 2] to run in this configuration.

[1]: https://patchwork.ozlabs.org/patch/1039493/
[2]: https://patchwork.ozlabs.org/patch/1039082/

Changes in v2:
- Remove unneeded quotes from NR_CPUS Kconfig entry
- Move memory barrier from send_ipi_many() to handle_ipi()
- Add check in send_ipi_many so that IPIs are only sent to available
harts as indicated by the available_harts mask
- Implement hart lottery to pick main hart to run U-Boot
- Remove CONFIG_MAIN_HART as it is not required anymore
- Register available harts in the available_harts mask
- New patch to populate register a0 with the hart ID from the mhartid
CSR in machine-mode
- New patch to enable SMP on the SiFive FU540, which was previously sent
independently

Lukas Auer (9):
  riscv: add infrastructure for calling functions on other harts
  riscv: import the supervisor binary interface header file
  riscv: implement IPI platform functions using SBI
  riscv: delay initialization of caches and debug UART
  riscv: add support for multi-hart systems
  riscv: boot images passed to bootm on all harts
  riscv: do not rely on hart ID passed by previous boot stage
  riscv: fu540: enable SMP
  riscv: qemu: enable SMP

 arch/riscv/Kconfig                   |  28 +++++
 arch/riscv/cpu/cpu.c                 |   9 +-
 arch/riscv/cpu/start.S               | 152 +++++++++++++++++++++++++--
 arch/riscv/include/asm/csr.h         |   1 +
 arch/riscv/include/asm/global_data.h |   6 ++
 arch/riscv/include/asm/sbi.h         |  94 +++++++++++++++++
 arch/riscv/include/asm/smp.h         |  53 ++++++++++
 arch/riscv/lib/Makefile              |   2 +
 arch/riscv/lib/asm-offsets.c         |   1 +
 arch/riscv/lib/bootm.c               |  13 ++-
 arch/riscv/lib/sbi_ipi.c             |  25 +++++
 arch/riscv/lib/smp.c                 | 116 ++++++++++++++++++++
 board/emulation/qemu-riscv/Kconfig   |   1 +
 board/sifive/fu540/Kconfig           |   1 +
 14 files changed, 492 insertions(+), 10 deletions(-)
 create mode 100644 arch/riscv/include/asm/sbi.h
 create mode 100644 arch/riscv/include/asm/smp.h
 create mode 100644 arch/riscv/lib/sbi_ipi.c
 create mode 100644 arch/riscv/lib/smp.c

-- 
2.20.1

[U-Boot] [PATCH v2 1/9] riscv: add infrastructure for calling functions on other harts

[Lukas Auer]

Harts on RISC-V boot independently, U-Boot is responsible for managing
them. Functions are called on other harts with smp_call_function(),
which sends inter-processor interrupts (IPIs) to all other available
harts. Available harts are those marked as available in the device tree
and present in the available_harts mask stored in global data. The
available_harts mask is used to register all harts that have entered
U-Boot. Functions are specified with their address and two function
arguments (argument 2 and 3). The first function argument is always the
hart ID of the hart calling the function. On the other harts, the IPI
interrupt handler handle_ipi() must be called on software interrupts to
handle the request and call the specified function.

Functions are stored in the ipi_data data structure. Every hart has its
own data structure in global data. While this is not required at the
moment (all harts are expected to boot Linux), this does allow future
expansion, where other harts may be used for monitoring or other tasks.

Signed-off-by: Lukas Auer <lukas.auer@aisec.fraunhofer.de>
---

Changes in v2:
- Remove unneeded quotes from NR_CPUS Kconfig entry
- Move memory barrier from send_ipi_many() to handle_ipi()
- Add check in send_ipi_many so that IPIs are only sent to available
harts as indicated by the available_harts mask

 arch/riscv/Kconfig                   |  19 +++++
 arch/riscv/include/asm/global_data.h |   6 ++
 arch/riscv/include/asm/smp.h         |  53 ++++++++++++
 arch/riscv/lib/Makefile              |   1 +
 arch/riscv/lib/smp.c                 | 116 +++++++++++++++++++++++++++
 5 files changed, 195 insertions(+)
 create mode 100644 arch/riscv/include/asm/smp.h
 create mode 100644 arch/riscv/lib/smp.c

diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig
index 36512a8995..4d7a115569 100644
--- a/arch/riscv/Kconfig
+++ b/arch/riscv/Kconfig
@@ -120,4 +120,23 @@ config RISCV_RDTIME
 config SYS_MALLOC_F_LEN
 	default 0x1000
 
+config SMP
+	bool "Symmetric Multi-Processing"
+	help
+	  This enables support for systems with more than one CPU. If
+	  you say N here, U-Boot will run on single and multiprocessor
+	  machines, but will use only one CPU of a multiprocessor
+	  machine. If you say Y here, U-Boot will run on many, but not
+	  all, single processor machines.
+
+config NR_CPUS
+	int "Maximum number of CPUs (2-32)"
+	range 2 32
+	depends on SMP
+	default 8
+	help
+	  On multiprocessor machines, U-Boot sets up a stack for each CPU.
+	  Stack memory is pre-allocated. U-Boot must therefore know the
+	  maximum number of CPUs that may be present.
+
 endmenu
diff --git a/arch/riscv/include/asm/global_data.h b/arch/riscv/include/asm/global_data.h
index a3a342c6e1..80e3165e39 100644
--- a/arch/riscv/include/asm/global_data.h
+++ b/arch/riscv/include/asm/global_data.h
@@ -10,12 +10,18 @@
 #ifndef	__ASM_GBL_DATA_H
 #define __ASM_GBL_DATA_H
 
+#include <asm/smp.h>
+
 /* Architecture-specific global data */
 struct arch_global_data {
 	long boot_hart;		/* boot hart id */
 #ifdef CONFIG_SIFIVE_CLINT
 	void __iomem *clint;	/* clint base address */
 #endif
+#ifdef CONFIG_SMP
+	struct ipi_data ipi[CONFIG_NR_CPUS];
+#endif
+	ulong available_harts;
 };
 
 #include <asm-generic/global_data.h>
diff --git a/arch/riscv/include/asm/smp.h b/arch/riscv/include/asm/smp.h
new file mode 100644
index 0000000000..bc863fdbaf
--- /dev/null
+++ b/arch/riscv/include/asm/smp.h
@@ -0,0 +1,53 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (C) 2019 Fraunhofer AISEC,
+ * Lukas Auer <lukas.auer@aisec.fraunhofer.de>
+ */
+
+#ifndef _ASM_RISCV_SMP_H
+#define _ASM_RISCV_SMP_H
+
+/**
+ * struct ipi_data - Inter-processor interrupt (IPI) data structure
+ *
+ * IPIs are used for SMP support to communicate to other harts what function to
+ * call. Functions are in the form
+ * void (*addr)(ulong hart, ulong arg0, ulong arg1).
+ *
+ * The function address and the two arguments, arg0 and arg1, are stored in the
+ * IPI data structure. The hart ID is inserted by the hart handling the IPI and
+ * calling the function.
+ *
+ * @addr: Address of function
+ * @arg0: First argument of function
+ * @arg1: Second argument of function
+ */
+struct ipi_data {
+	ulong addr;
+	ulong arg0;
+	ulong arg1;
+};
+
+/**
+ * handle_ipi() - interrupt handler for software interrupts
+ *
+ * The IPI interrupt handler must be called to handle software interrupts. It
+ * calls the function specified in the hart's IPI data structure.
+ *
+ * @hart: Hart ID of the current hart
+ */
+void handle_ipi(ulong hart);
+
+/**
+ * smp_call_function() - Call a function on all other harts
+ *
+ * Send IPIs with the specified function call to all harts.
+ *
+ * @addr: Address of function
+ * @arg0: First argument of function
+ * @arg1: Second argument of function
+ * @return 0 if OK, -ve on error
+ */
+int smp_call_function(ulong addr, ulong arg0, ulong arg1);
+
+#endif
diff --git a/arch/riscv/lib/Makefile b/arch/riscv/lib/Makefile
index edfa61690c..19370f9749 100644
--- a/arch/riscv/lib/Makefile
+++ b/arch/riscv/lib/Makefile
@@ -14,6 +14,7 @@ obj-$(CONFIG_SIFIVE_CLINT) += sifive_clint.o
 obj-y	+= interrupts.o
 obj-y	+= reset.o
 obj-y   += setjmp.o
+obj-$(CONFIG_SMP) += smp.o
 
 # For building EFI apps
 CFLAGS_$(EFI_CRT0) := $(CFLAGS_EFI)
diff --git a/arch/riscv/lib/smp.c b/arch/riscv/lib/smp.c
new file mode 100644
index 0000000000..edef8a687d
--- /dev/null
+++ b/arch/riscv/lib/smp.c
@@ -0,0 +1,116 @@
+// SPDX-License-Identifier: GPL-2.0+
+/*
+ * Copyright (C) 2019 Fraunhofer AISEC,
+ * Lukas Auer <lukas.auer@aisec.fraunhofer.de>
+ */
+
+#include <common.h>
+#include <dm.h>
+#include <asm/barrier.h>
+#include <asm/smp.h>
+
+DECLARE_GLOBAL_DATA_PTR;
+
+/**
+ * riscv_send_ipi() - Send inter-processor interrupt (IPI)
+ *
+ * Platform code must provide this function.
+ *
+ * @hart: Hart ID of receiving hart
+ * @return 0 if OK, -ve on error
+ */
+extern int riscv_send_ipi(int hart);
+
+/**
+ * riscv_clear_ipi() - Clear inter-processor interrupt (IPI)
+ *
+ * Platform code must provide this function.
+ *
+ * @hart: Hart ID of hart to be cleared
+ * @return 0 if OK, -ve on error
+ */
+extern int riscv_clear_ipi(int hart);
+
+static int send_ipi_many(struct ipi_data *ipi)
+{
+	ofnode node, cpus;
+	u32 reg;
+	int ret;
+
+	cpus = ofnode_path("/cpus");
+	if (!ofnode_valid(cpus)) {
+		pr_err("Can't find cpus node!\n");
+		return -EINVAL;
+	}
+
+	ofnode_for_each_subnode(node, cpus) {
+		/* skip if hart is marked as not available in the device tree */
+		if (!ofnode_is_available(node))
+			continue;
+
+		/* read hart ID of CPU */
+		ret = ofnode_read_u32(node, "reg", &reg);
+		if (ret)
+			continue;
+
+		/* skip if it is the hart we are running on */
+		if (reg == gd->arch.boot_hart)
+			continue;
+
+		if (reg >= CONFIG_NR_CPUS) {
+			pr_err("Hart ID %d is out of range, increase CONFIG_NR_CPUS\n",
+			       reg);
+			continue;
+		}
+
+		/* skip if hart is not available */
+		if (!(gd->arch.available_harts & (1 << reg)))
+			continue;
+
+		gd->arch.ipi[reg].addr = ipi->addr;
+		gd->arch.ipi[reg].arg0 = ipi->arg0;
+		gd->arch.ipi[reg].arg1 = ipi->arg1;
+
+		ret = riscv_send_ipi(reg);
+		if (ret)
+			return ret;
+	}
+
+	return 0;
+}
+
+void handle_ipi(ulong hart)
+{
+	int ret;
+	void (*smp_function)(ulong hart, ulong arg0, ulong arg1);
+
+	if (hart >= CONFIG_NR_CPUS)
+		return;
+
+	ret = riscv_clear_ipi(hart);
+	if (ret) {
+		pr_err("Cannot clear IPI\n");
+		return;
+	}
+
+	__smp_mb();
+
+	smp_function = (void (*)(ulong, ulong, ulong))gd->arch.ipi[hart].addr;
+	invalidate_icache_all();
+
+	smp_function(hart, gd->arch.ipi[hart].arg0, gd->arch.ipi[hart].arg1);
+}
+
+int smp_call_function(ulong addr, ulong arg0, ulong arg1)
+{
+	int ret = 0;
+	struct ipi_data ipi;
+
+	ipi.addr = addr;
+	ipi.arg0 = arg0;
+	ipi.arg1 = arg1;
+
+	ret = send_ipi_many(&ipi);
+
+	return ret;
+}
-- 
2.20.1

[U-Boot] [PATCH v2 2/9] riscv: import the supervisor binary interface header file

[Lukas Auer]

Import the supervisor binary interface (SBI) header file from Linux
(arch/riscv/include/asm/sbi.h). The last change to it was in commit
6d60b6ee0c97 ("RISC-V: Device, timer, IRQs, and the SBI").

Signed-off-by: Lukas Auer <lukas.auer@aisec.fraunhofer.de>
Reviewed-by: Anup Patel <anup.patel@wdc.com>
Reviewed-by: Bin Meng <bmeng.cn@gmail.com>
---

Changes in v2: None

 arch/riscv/include/asm/sbi.h | 94 ++++++++++++++++++++++++++++++++++++
 1 file changed, 94 insertions(+)
 create mode 100644 arch/riscv/include/asm/sbi.h

diff --git a/arch/riscv/include/asm/sbi.h b/arch/riscv/include/asm/sbi.h
new file mode 100644
index 0000000000..ced57defdd
--- /dev/null
+++ b/arch/riscv/include/asm/sbi.h
@@ -0,0 +1,94 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (C) 2015 Regents of the University of California
+ *
+ * Taken from Linux arch/riscv/include/asm/sbi.h
+ */
+
+#ifndef _ASM_RISCV_SBI_H
+#define _ASM_RISCV_SBI_H
+
+#include <linux/types.h>
+
+#define SBI_SET_TIMER 0
+#define SBI_CONSOLE_PUTCHAR 1
+#define SBI_CONSOLE_GETCHAR 2
+#define SBI_CLEAR_IPI 3
+#define SBI_SEND_IPI 4
+#define SBI_REMOTE_FENCE_I 5
+#define SBI_REMOTE_SFENCE_VMA 6
+#define SBI_REMOTE_SFENCE_VMA_ASID 7
+#define SBI_SHUTDOWN 8
+
+#define SBI_CALL(which, arg0, arg1, arg2) ({			\
+	register uintptr_t a0 asm ("a0") = (uintptr_t)(arg0);	\
+	register uintptr_t a1 asm ("a1") = (uintptr_t)(arg1);	\
+	register uintptr_t a2 asm ("a2") = (uintptr_t)(arg2);	\
+	register uintptr_t a7 asm ("a7") = (uintptr_t)(which);	\
+	asm volatile ("ecall"					\
+		      : "+r" (a0)				\
+		      : "r" (a1), "r" (a2), "r" (a7)		\
+		      : "memory");				\
+	a0;							\
+})
+
+/* Lazy implementations until SBI is finalized */
+#define SBI_CALL_0(which) SBI_CALL(which, 0, 0, 0)
+#define SBI_CALL_1(which, arg0) SBI_CALL(which, arg0, 0, 0)
+#define SBI_CALL_2(which, arg0, arg1) SBI_CALL(which, arg0, arg1, 0)
+
+static inline void sbi_console_putchar(int ch)
+{
+	SBI_CALL_1(SBI_CONSOLE_PUTCHAR, ch);
+}
+
+static inline int sbi_console_getchar(void)
+{
+	return SBI_CALL_0(SBI_CONSOLE_GETCHAR);
+}
+
+static inline void sbi_set_timer(uint64_t stime_value)
+{
+#if __riscv_xlen == 32
+	SBI_CALL_2(SBI_SET_TIMER, stime_value, stime_value >> 32);
+#else
+	SBI_CALL_1(SBI_SET_TIMER, stime_value);
+#endif
+}
+
+static inline void sbi_shutdown(void)
+{
+	SBI_CALL_0(SBI_SHUTDOWN);
+}
+
+static inline void sbi_clear_ipi(void)
+{
+	SBI_CALL_0(SBI_CLEAR_IPI);
+}
+
+static inline void sbi_send_ipi(const unsigned long *hart_mask)
+{
+	SBI_CALL_1(SBI_SEND_IPI, hart_mask);
+}
+
+static inline void sbi_remote_fence_i(const unsigned long *hart_mask)
+{
+	SBI_CALL_1(SBI_REMOTE_FENCE_I, hart_mask);
+}
+
+static inline void sbi_remote_sfence_vma(const unsigned long *hart_mask,
+					 unsigned long start,
+					 unsigned long size)
+{
+	SBI_CALL_1(SBI_REMOTE_SFENCE_VMA, hart_mask);
+}
+
+static inline void sbi_remote_sfence_vma_asid(const unsigned long *hart_mask,
+					      unsigned long start,
+					      unsigned long size,
+					      unsigned long asid)
+{
+	SBI_CALL_1(SBI_REMOTE_SFENCE_VMA_ASID, hart_mask);
+}
+
+#endif
-- 
2.20.1

[U-Boot] [PATCH v2 3/9] riscv: implement IPI platform functions using SBI

[Lukas Auer]

The supervisor binary interface (SBI) provides the necessary functions
to implement the platform IPI functions riscv_send_ipi() and
riscv_clear_ipi(). Use it to implement them.

This adds support for inter-processor interrupts (IPIs) on RISC-V CPUs
running in supervisor mode. Support for machine mode is already
available for CPUs that include the SiFive CLINT.

Signed-off-by: Lukas Auer <lukas.auer@aisec.fraunhofer.de>
Reviewed-by: Anup Patel <anup.patel@wdc.com>
Reviewed-by: Bin Meng <bmeng.cn@gmail.com>
---

Changes in v2: None

 arch/riscv/Kconfig       |  5 +++++
 arch/riscv/lib/Makefile  |  1 +
 arch/riscv/lib/sbi_ipi.c | 25 +++++++++++++++++++++++++
 3 files changed, 31 insertions(+)
 create mode 100644 arch/riscv/lib/sbi_ipi.c

diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig
index 4d7a115569..9da609b33b 100644
--- a/arch/riscv/Kconfig
+++ b/arch/riscv/Kconfig
@@ -139,4 +139,9 @@ config NR_CPUS
 	  Stack memory is pre-allocated. U-Boot must therefore know the
 	  maximum number of CPUs that may be present.
 
+config SBI_IPI
+	bool
+	default y if RISCV_SMODE
+	depends on SMP
+
 endmenu
diff --git a/arch/riscv/lib/Makefile b/arch/riscv/lib/Makefile
index 19370f9749..35dbf643e4 100644
--- a/arch/riscv/lib/Makefile
+++ b/arch/riscv/lib/Makefile
@@ -13,6 +13,7 @@ obj-$(CONFIG_RISCV_RDTIME) += rdtime.o
 obj-$(CONFIG_SIFIVE_CLINT) += sifive_clint.o
 obj-y	+= interrupts.o
 obj-y	+= reset.o
+obj-$(CONFIG_SBI_IPI) += sbi_ipi.o
 obj-y   += setjmp.o
 obj-$(CONFIG_SMP) += smp.o
 
diff --git a/arch/riscv/lib/sbi_ipi.c b/arch/riscv/lib/sbi_ipi.c
new file mode 100644
index 0000000000..170346da68
--- /dev/null
+++ b/arch/riscv/lib/sbi_ipi.c
@@ -0,0 +1,25 @@
+// SPDX-License-Identifier: GPL-2.0+
+/*
+ * Copyright (C) 2019 Fraunhofer AISEC,
+ * Lukas Auer <lukas.auer@aisec.fraunhofer.de>
+ */
+
+#include <common.h>
+#include <asm/sbi.h>
+
+int riscv_send_ipi(int hart)
+{
+	ulong mask;
+
+	mask = 1UL << hart;
+	sbi_send_ipi(&mask);
+
+	return 0;
+}
+
+int riscv_clear_ipi(int hart)
+{
+	sbi_clear_ipi();
+
+	return 0;
+}
-- 
2.20.1

[U-Boot] [PATCH v2 4/9] riscv: delay initialization of caches and debug UART

[Lukas Auer]

Move the initialization of the caches and the debug UART until after
board_init_f_init_reserve. This is in preparation for SMP support, where
code prior to this point will be executed by all harts. This ensures
that initialization will only be performed once for the main hart
running U-Boot.

Signed-off-by: Lukas Auer <lukas.auer@aisec.fraunhofer.de>
Reviewed-by: Anup Patel <anup.patel@wdc.com>
Reviewed-by: Bin Meng <bmeng.cn@gmail.com>
---

Changes in v2: None

 arch/riscv/cpu/start.S | 16 ++++++++--------
 1 file changed, 8 insertions(+), 8 deletions(-)

diff --git a/arch/riscv/cpu/start.S b/arch/riscv/cpu/start.S
index 81ea52b170..a30f6f7194 100644
--- a/arch/riscv/cpu/start.S
+++ b/arch/riscv/cpu/start.S
@@ -45,10 +45,6 @@ _start:
 	/* mask all interrupts */
 	csrw	MODE_PREFIX(ie), zero
 
-	/* Enable cache */
-	jal	icache_enable
-	jal	dcache_enable
-
 /*
  * Set stackpointer in internal/ex RAM to call board_init_f
  */
@@ -57,10 +53,6 @@ call_board_init_f:
 	li	t1, CONFIG_SYS_INIT_SP_ADDR
 	and	sp, t1, t0		/* force 16 byte alignment */
 
-#ifdef CONFIG_DEBUG_UART
-	jal	debug_uart_init
-#endif
-
 call_board_init_f_0:
 	mv	a0, sp
 	jal	board_init_f_alloc_reserve
@@ -74,6 +66,14 @@ call_board_init_f_0:
 	/* save the boot hart id to global_data */
 	SREG	s0, GD_BOOT_HART(gp)
 
+	/* Enable cache */
+	jal	icache_enable
+	jal	dcache_enable
+
+#ifdef CONFIG_DEBUG_UART
+	jal	debug_uart_init
+#endif
+
 	mv	a0, zero		/* a0 <-- boot_flags = 0 */
 	la	t5, board_init_f
 	jr	t5			/* jump to board_init_f() */
-- 
2.20.1

[U-Boot] [PATCH v2 5/9] riscv: add support for multi-hart systems

[Lukas Auer]

On RISC-V, all harts boot independently. To be able to run on a
multi-hart system, U-Boot must be extended with the functionality to
manage all harts in the system. All harts entering U-Boot are registered
in the available_harts mask stored in global data. A hart lottery system
as used in the Linux kernel selects the hart U-Boot runs on. All other
harts are halted. U-Boot can delegate functions to them using
smp_call_function().

Every hart has a valid pointer to the global data structure and a 8KiB
stack by default. The stack size is set with CONFIG_STACK_SIZE_SHIFT.

Signed-off-by: Lukas Auer <lukas.auer@aisec.fraunhofer.de>
---

Changes in v2:
- Implement hart lottery to pick main hart to run U-Boot
- Remove CONFIG_MAIN_HART as it is not required anymore
- Register available harts in the available_harts mask

 arch/riscv/Kconfig           |   4 ++
 arch/riscv/cpu/cpu.c         |   9 ++-
 arch/riscv/cpu/start.S       | 134 ++++++++++++++++++++++++++++++++++-
 arch/riscv/include/asm/csr.h |   1 +
 arch/riscv/lib/asm-offsets.c |   1 +
 5 files changed, 147 insertions(+), 2 deletions(-)

diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig
index 9da609b33b..3a4470daf3 100644
--- a/arch/riscv/Kconfig
+++ b/arch/riscv/Kconfig
@@ -144,4 +144,8 @@ config SBI_IPI
 	default y if RISCV_SMODE
 	depends on SMP
 
+config STACK_SIZE_SHIFT
+	int
+	default 13
+
 endmenu
diff --git a/arch/riscv/cpu/cpu.c b/arch/riscv/cpu/cpu.c
index e662140427..c32de8a4c3 100644
--- a/arch/riscv/cpu/cpu.c
+++ b/arch/riscv/cpu/cpu.c
@@ -12,10 +12,17 @@
 #include <dm/uclass-internal.h>
 
 /*
- * prior_stage_fdt_address must be stored in the data section since it is used
+ * The variables here must be stored in the data section since they are used
  * before the bss section is available.
  */
 phys_addr_t prior_stage_fdt_address __attribute__((section(".data")));
+u32 hart_lottery __attribute__((section(".data"))) = 0;
+
+/*
+ * The main hart running U-Boot has acquired available_harts_lock until it has
+ * finished initialization of global data.
+ */
+u32 available_harts_lock = 1;
 
 static inline bool supports_extension(char ext)
 {
diff --git a/arch/riscv/cpu/start.S b/arch/riscv/cpu/start.S
index a30f6f7194..79b753847c 100644
--- a/arch/riscv/cpu/start.S
+++ b/arch/riscv/cpu/start.S
@@ -13,6 +13,7 @@
 #include <config.h>
 #include <common.h>
 #include <elf.h>
+#include <asm/csr.h>
 #include <asm/encoding.h>
 #include <generated/asm-offsets.h>
 
@@ -45,6 +46,23 @@ _start:
 	/* mask all interrupts */
 	csrw	MODE_PREFIX(ie), zero
 
+#ifdef CONFIG_SMP
+	/* check if hart is within range */
+	/* s0: hart id */
+	li	t0, CONFIG_NR_CPUS
+	bge	s0, t0, hart_out_of_bounds_loop
+#endif
+
+#ifdef CONFIG_SMP
+	/* set xSIE bit to receive IPIs */
+#ifdef CONFIG_RISCV_MMODE
+	li	t0, MIE_MSIE
+#else
+	li	t0, SIE_SSIE
+#endif
+	csrs	MODE_PREFIX(ie), t0
+#endif
+
 /*
  * Set stackpointer in internal/ex RAM to call board_init_f
  */
@@ -56,7 +74,30 @@ call_board_init_f:
 call_board_init_f_0:
 	mv	a0, sp
 	jal	board_init_f_alloc_reserve
+
+	/*
+	 * Set global data pointer here for all harts, uninitialized at this
+	 * point.
+	 */
+	mv	gp, a0
+
+	/* setup stack */
+#ifdef CONFIG_SMP
+	/* s0: hart id */
+	slli	t0, s0, CONFIG_STACK_SIZE_SHIFT
+	sub	sp, a0, t0
+#else
 	mv	sp, a0
+#endif
+
+	/*
+	 * Pick hart to initialize global data and run U-Boot. The other harts
+	 * wait for initialization to complete.
+	 */
+	la	t0, hart_lottery
+	li	s2, 1
+	amoswap.w s2, t1, 0(t0)
+	bnez	s2, wait_for_gd_init
 
 	la	t0, prior_stage_fdt_address
 	SREG	s1, 0(t0)
@@ -66,6 +107,33 @@ call_board_init_f_0:
 	/* save the boot hart id to global_data */
 	SREG	s0, GD_BOOT_HART(gp)
 
+	la	t0, available_harts_lock
+	fence	rw, w
+	amoswap.w zero, zero, 0(t0)
+
+wait_for_gd_init:
+	la	t0, available_harts_lock
+	li	t1, 1
+1:	amoswap.w t1, t1, 0(t0)
+	fence	r, rw
+	bnez	t1, 1b
+
+	/* register available harts in the available_harts mask */
+	li	t1, 1
+	sll	t1, t1, s0
+	LREG	t2, GD_AVAILABLE_HARTS(gp)
+	or	t2, t2, t1
+	SREG	t2, GD_AVAILABLE_HARTS(gp)
+
+	fence	rw, w
+	amoswap.w zero, zero, 0(t0)
+
+	/*
+	 * Continue on hart lottery winner, others branch to
+	 * secondary_hart_loop.
+	 */
+	bnez	s2, secondary_hart_loop
+
 	/* Enable cache */
 	jal	icache_enable
 	jal	dcache_enable
@@ -95,7 +163,14 @@ relocate_code:
  *Set up the stack
  */
 stack_setup:
+#ifdef CONFIG_SMP
+	/* s0: hart id */
+	slli	t0, s0, CONFIG_STACK_SIZE_SHIFT
+	sub	sp, s2, t0
+#else
 	mv	sp, s2
+#endif
+
 	la	t0, _start
 	sub	t6, s4, t0		/* t6 <- relocation offset */
 	beq	t0, s4, clear_bss	/* skip relocation */
@@ -175,13 +250,30 @@ clear_bss:
 	add	t0, t0, t6		/* t0 <- rel __bss_start in RAM */
 	la	t1, __bss_end		/* t1 <- rel __bss_end in FLASH */
 	add	t1, t1, t6		/* t1 <- rel __bss_end in RAM */
-	beq	t0, t1, call_board_init_r
+	beq	t0, t1, relocate_secondary_harts
 
 clbss_l:
 	SREG	zero, 0(t0)		/* clear loop... */
 	addi	t0, t0, REGBYTES
 	bne	t0, t1, clbss_l
 
+relocate_secondary_harts:
+#ifdef CONFIG_SMP
+	/* send relocation IPI */
+	la	t0, secondary_hart_relocate
+	add	a0, t0, t6
+
+	/* store relocation offset */
+	mv	s5, t6
+
+	mv	a1, s2
+	mv	a2, s3
+	jal	smp_call_function
+
+	/* restore relocation offset */
+	mv	t6, s5
+#endif
+
 /*
  * We are done. Do not return, instead branch to second part of board
  * initialization, now running from RAM.
@@ -202,3 +294,43 @@ call_board_init_r:
  * jump to it ...
  */
 	jr	t4			/* jump to board_init_r() */
+
+#ifdef CONFIG_SMP
+hart_out_of_bounds_loop:
+	/* Harts in this loop are out of bounds, increase CONFIG_NR_CPUS. */
+	wfi
+	j	hart_out_of_bounds_loop
+#endif
+
+#ifdef CONFIG_SMP
+/* SMP relocation entry */
+secondary_hart_relocate:
+	/* a1: new sp */
+	/* a2: new gd */
+	/* s0: hart id */
+
+	/* setup stack */
+	slli	t0, s0, CONFIG_STACK_SIZE_SHIFT
+	sub	sp, a1, t0
+
+	/* update global data pointer */
+	mv	gp, a2
+#endif
+
+secondary_hart_loop:
+	wfi
+
+#ifdef CONFIG_SMP
+	csrr	t0, MODE_PREFIX(ip)
+#ifdef CONFIG_RISCV_MMODE
+	andi	t0, t0, MIE_MSIE
+#else
+	andi	t0, t0, SIE_SSIE
+#endif
+	beqz	t0, secondary_hart_loop
+
+	mv	a0, s0
+	jal	handle_ipi
+#endif
+
+	j	secondary_hart_loop
diff --git a/arch/riscv/include/asm/csr.h b/arch/riscv/include/asm/csr.h
index 86136f542c..644e6baa15 100644
--- a/arch/riscv/include/asm/csr.h
+++ b/arch/riscv/include/asm/csr.h
@@ -46,6 +46,7 @@
 #endif
 
 /* Interrupt Enable and Interrupt Pending flags */
+#define MIE_MSIE	_AC(0x00000008, UL) /* Software Interrupt Enable */
 #define SIE_SSIE	_AC(0x00000002, UL) /* Software Interrupt Enable */
 #define SIE_STIE	_AC(0x00000020, UL) /* Timer Interrupt Enable */
 
diff --git a/arch/riscv/lib/asm-offsets.c b/arch/riscv/lib/asm-offsets.c
index e0b71f5691..f998402bd1 100644
--- a/arch/riscv/lib/asm-offsets.c
+++ b/arch/riscv/lib/asm-offsets.c
@@ -14,6 +14,7 @@
 int main(void)
 {
 	DEFINE(GD_BOOT_HART, offsetof(gd_t, arch.boot_hart));
+	DEFINE(GD_AVAILABLE_HARTS, offsetof(gd_t, arch.available_harts));
 
 	return 0;
 }
-- 
2.20.1

[U-Boot] [PATCH v2 6/9] riscv: boot images passed to bootm on all harts

[Lukas Auer]

Signed-off-by: Lukas Auer <lukas.auer@aisec.fraunhofer.de>
Reviewed-by: Anup Patel <anup.patel@wdc.com>
Reviewed-by: Bin Meng <bmeng.cn@gmail.com>
---

Changes in v2: None

 arch/riscv/lib/bootm.c | 13 ++++++++++++-
 1 file changed, 12 insertions(+), 1 deletion(-)

diff --git a/arch/riscv/lib/bootm.c b/arch/riscv/lib/bootm.c
index f36b8702ef..efbd3e23e7 100644
--- a/arch/riscv/lib/bootm.c
+++ b/arch/riscv/lib/bootm.c
@@ -13,6 +13,7 @@
 #include <image.h>
 #include <asm/byteorder.h>
 #include <asm/csr.h>
+#include <asm/smp.h>
 #include <dm/device.h>
 #include <dm/root.h>
 #include <u-boot/zlib.h>
@@ -81,6 +82,9 @@ static void boot_jump_linux(bootm_headers_t *images, int flag)
 {
 	void (*kernel)(ulong hart, void *dtb);
 	int fake = (flag & BOOTM_STATE_OS_FAKE_GO);
+#ifdef CONFIG_SMP
+	int ret;
+#endif
 
 	kernel = (void (*)(ulong, void *))images->ep;
 
@@ -92,8 +96,15 @@ static void boot_jump_linux(bootm_headers_t *images, int flag)
 	announce_and_cleanup(fake);
 
 	if (!fake) {
-		if (IMAGE_ENABLE_OF_LIBFDT && images->ft_len)
+		if (IMAGE_ENABLE_OF_LIBFDT && images->ft_len) {
+#ifdef CONFIG_SMP
+			ret = smp_call_function(images->ep,
+						(ulong)images->ft_addr, 0);
+			if (ret)
+				hang();
+#endif
 			kernel(gd->arch.boot_hart, images->ft_addr);
+		}
 	}
 }
 
-- 
2.20.1

[U-Boot] [PATCH v2 7/9] riscv: do not rely on hart ID passed by previous boot stage

[Lukas Auer]

RISC-V U-Boot expects the hart ID to be passed to it via register a0 by
the previous boot stage. Machine mode firmware such as BBL and OpenSBI
do this when starting their payload (U-Boot) in supervisor mode. If
U-Boot is running in machine mode, this task must be handled by the boot
ROM. Explicitly populate register a0 with the hart ID from the mhartid
CSR to avoid possible problems on RISC-V processors with a boot ROM that
does not handle this task.

Suggested-by: Rick Chen <rick@andestech.com>
Signed-off-by: Lukas Auer <lukas.auer@aisec.fraunhofer.de>
---

Changes in v2:
- New patch to populate register a0 with the hart ID from the mhartid
CSR in machine-mode

 arch/riscv/cpu/start.S | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/arch/riscv/cpu/start.S b/arch/riscv/cpu/start.S
index 79b753847c..d4daa6e0bf 100644
--- a/arch/riscv/cpu/start.S
+++ b/arch/riscv/cpu/start.S
@@ -36,6 +36,10 @@
 .section .text
 .globl _start
 _start:
+#ifdef CONFIG_RISCV_MMODE
+	csrr	a0, mhartid
+#endif
+
 	/* save hart id and dtb pointer */
 	mv	s0, a0
 	mv	s1, a1
-- 
2.20.1

[U-Boot] [PATCH v2 8/9] riscv: fu540: enable SMP

[Lukas Auer]

Signed-off-by: Lukas Auer <lukas.auer@aisec.fraunhofer.de>
Reviewed-by: Anup Patel <anup.patel@wdc.com>
---

Changes in v2:
- New patch to enable SMP on the SiFive FU540, which was previously sent
independently

 board/sifive/fu540/Kconfig | 1 +
 1 file changed, 1 insertion(+)

diff --git a/board/sifive/fu540/Kconfig b/board/sifive/fu540/Kconfig
index 6be3d88144..f46437901d 100644
--- a/board/sifive/fu540/Kconfig
+++ b/board/sifive/fu540/Kconfig
@@ -38,5 +38,6 @@ config BOARD_SPECIFIC_OPTIONS # dummy
 	imply PHY_LIB
 	imply PHY_MSCC
 	imply SIFIVE_SERIAL
+	imply SMP
 
 endif
-- 
2.20.1

[U-Boot] [PATCH v2 9/9] riscv: qemu: enable SMP

[Lukas Auer]

Signed-off-by: Lukas Auer <lukas.auer@aisec.fraunhofer.de>
Reviewed-by: Anup Patel <anup.patel@wdc.com>
Reviewed-by: Bin Meng <bmeng.cn@gmail.com>
---

Changes in v2: None

 board/emulation/qemu-riscv/Kconfig | 1 +
 1 file changed, 1 insertion(+)

diff --git a/board/emulation/qemu-riscv/Kconfig b/board/emulation/qemu-riscv/Kconfig
index 88d07d568e..cf057e7de6 100644
--- a/board/emulation/qemu-riscv/Kconfig
+++ b/board/emulation/qemu-riscv/Kconfig
@@ -34,5 +34,6 @@ config BOARD_SPECIFIC_OPTIONS # dummy
 	imply BOARD_LATE_INIT
 	imply OF_BOARD_SETUP
 	imply SIFIVE_SERIAL
+	imply SMP
 
 endif
-- 
2.20.1

[U-Boot] [PATCH v2 1/9] riscv: add infrastructure for calling functions on other harts

[Anup Patel]

> -----Original Message-----
> From: Lukas Auer <lukas.auer@aisec.fraunhofer.de>
> Sent: Wednesday, March 6, 2019 4:23 AM
> To: u-boot at lists.denx.de
> Cc: Atish Patra <Atish.Patra@wdc.com>; Anup Patel
> <Anup.Patel@wdc.com>; Bin Meng <bmeng.cn@gmail.com>; Andreas
> Schwab <schwab@suse.de>; Palmer Dabbelt <palmer@sifive.com>;
> Alexander Graf <agraf@suse.de>; Lukas Auer
> <lukas.auer@aisec.fraunhofer.de>; Anup Patel <anup@brainfault.org>; Rick
> Chen <rick@andestech.com>
> Subject: [PATCH v2 1/9] riscv: add infrastructure for calling functions on other
> harts
> 
> Harts on RISC-V boot independently, U-Boot is responsible for managing
> them. Functions are called on other harts with smp_call_function(), which
> sends inter-processor interrupts (IPIs) to all other available harts. Available
> harts are those marked as available in the device tree and present in the
> available_harts mask stored in global data. The available_harts mask is used
> to register all harts that have entered U-Boot. Functions are specified with
> their address and two function arguments (argument 2 and 3). The first
> function argument is always the hart ID of the hart calling the function. On
> the other harts, the IPI interrupt handler handle_ipi() must be called on
> software interrupts to handle the request and call the specified function.
> 
> Functions are stored in the ipi_data data structure. Every hart has its own
> data structure in global data. While this is not required at the moment (all
> harts are expected to boot Linux), this does allow future expansion, where
> other harts may be used for monitoring or other tasks.
> 
> Signed-off-by: Lukas Auer <lukas.auer@aisec.fraunhofer.de>
> ---
> 
> Changes in v2:
> - Remove unneeded quotes from NR_CPUS Kconfig entry
> - Move memory barrier from send_ipi_many() to handle_ipi()
> - Add check in send_ipi_many so that IPIs are only sent to available harts as
> indicated by the available_harts mask

Reviewed-by: Anup Patel <anup.patel@wdc.com>

Regards,
Anup

[U-Boot] [PATCH v2 5/9] riscv: add support for multi-hart systems

[Anup Patel]

> -----Original Message-----
> From: Lukas Auer <lukas.auer@aisec.fraunhofer.de>
> Sent: Wednesday, March 6, 2019 4:23 AM
> To: u-boot at lists.denx.de
> Cc: Atish Patra <Atish.Patra@wdc.com>; Anup Patel
> <Anup.Patel@wdc.com>; Bin Meng <bmeng.cn@gmail.com>; Andreas
> Schwab <schwab@suse.de>; Palmer Dabbelt <palmer@sifive.com>;
> Alexander Graf <agraf@suse.de>; Lukas Auer
> <lukas.auer@aisec.fraunhofer.de>; Anup Patel <anup@brainfault.org>; Rick
> Chen <rick@andestech.com>; Baruch Siach <baruch@tkos.co.il>; Stefan
> Roese <sr@denx.de>; Daniel Schwierzeck <daniel.schwierzeck@gmail.com>
> Subject: [PATCH v2 5/9] riscv: add support for multi-hart systems
> 
> On RISC-V, all harts boot independently. To be able to run on a multi-hart
> system, U-Boot must be extended with the functionality to manage all harts
> in the system. All harts entering U-Boot are registered in the available_harts
> mask stored in global data. A hart lottery system as used in the Linux kernel
> selects the hart U-Boot runs on. All other harts are halted. U-Boot can
> delegate functions to them using smp_call_function().
> 
> Every hart has a valid pointer to the global data structure and a 8KiB stack by
> default. The stack size is set with CONFIG_STACK_SIZE_SHIFT.
> 
> Signed-off-by: Lukas Auer <lukas.auer@aisec.fraunhofer.de>
> ---
> 
> Changes in v2:
> - Implement hart lottery to pick main hart to run U-Boot
> - Remove CONFIG_MAIN_HART as it is not required anymore
> - Register available harts in the available_harts mask
> 
>  arch/riscv/Kconfig           |   4 ++
>  arch/riscv/cpu/cpu.c         |   9 ++-
>  arch/riscv/cpu/start.S       | 134 ++++++++++++++++++++++++++++++++++-
>  arch/riscv/include/asm/csr.h |   1 +
>  arch/riscv/lib/asm-offsets.c |   1 +
>  5 files changed, 147 insertions(+), 2 deletions(-)

Reviewed-by: Anup Patel <anup.patel@wdc.com>

Regards,
Anup

[U-Boot] [PATCH v2 7/9] riscv: do not rely on hart ID passed by previous boot stage

[Anup Patel]

> -----Original Message-----
> From: Lukas Auer <lukas.auer@aisec.fraunhofer.de>
> Sent: Wednesday, March 6, 2019 4:23 AM
> To: u-boot at lists.denx.de
> Cc: Atish Patra <Atish.Patra@wdc.com>; Anup Patel
> <Anup.Patel@wdc.com>; Bin Meng <bmeng.cn@gmail.com>; Andreas
> Schwab <schwab@suse.de>; Palmer Dabbelt <palmer@sifive.com>;
> Alexander Graf <agraf@suse.de>; Lukas Auer
> <lukas.auer@aisec.fraunhofer.de>; Rick Chen <rick@andestech.com>; Anup
> Patel <anup@brainfault.org>
> Subject: [PATCH v2 7/9] riscv: do not rely on hart ID passed by previous boot
> stage
> 
> RISC-V U-Boot expects the hart ID to be passed to it via register a0 by the
> previous boot stage. Machine mode firmware such as BBL and OpenSBI do
> this when starting their payload (U-Boot) in supervisor mode. If U-Boot is
> running in machine mode, this task must be handled by the boot ROM.
> Explicitly populate register a0 with the hart ID from the mhartid CSR to avoid
> possible problems on RISC-V processors with a boot ROM that does not
> handle this task.
> 
> Suggested-by: Rick Chen <rick@andestech.com>
> Signed-off-by: Lukas Auer <lukas.auer@aisec.fraunhofer.de>
> ---
> 
> Changes in v2:
> - New patch to populate register a0 with the hart ID from the mhartid CSR in
> machine-mode
> 
>  arch/riscv/cpu/start.S | 4 ++++
>  1 file changed, 4 insertions(+)
> 
> diff --git a/arch/riscv/cpu/start.S b/arch/riscv/cpu/start.S index
> 79b753847c..d4daa6e0bf 100644
> --- a/arch/riscv/cpu/start.S
> +++ b/arch/riscv/cpu/start.S
> @@ -36,6 +36,10 @@
>  .section .text
>  .globl _start
>  _start:
> +#ifdef CONFIG_RISCV_MMODE
> +	csrr	a0, mhartid
> +#endif
> +
>  	/* save hart id and dtb pointer */
>  	mv	s0, a0
>  	mv	s1, a1
> --
> 2.20.1

Reviewed-by: Anup Patel <anup.patel@wdc.com>

Regards,
Anup

[U-Boot] [PATCH v2 0/9] SMP support for RISC-V

[Anup Patel]

> -----Original Message-----
> From: Lukas Auer <lukas.auer@aisec.fraunhofer.de>
> Sent: Wednesday, March 6, 2019 4:23 AM
> To: u-boot at lists.denx.de
> Cc: Atish Patra <Atish.Patra@wdc.com>; Anup Patel
> <Anup.Patel@wdc.com>; Bin Meng <bmeng.cn@gmail.com>; Andreas
> Schwab <schwab@suse.de>; Palmer Dabbelt <palmer@sifive.com>;
> Alexander Graf <agraf@suse.de>; Lukas Auer
> <lukas.auer@aisec.fraunhofer.de>; Anup Patel <Anup.Patel@wdc.com>;
> Anup Patel <anup@brainfault.org>; Rick Chen <rick@andestech.com>;
> Baruch Siach <baruch@tkos.co.il>; Atish Patra <Atish.Patra@wdc.com>;
> Stefan Roese <sr@denx.de>; Paul Walmsley <paul.walmsley@sifive.com>;
> Daniel Schwierzeck <daniel.schwierzeck@gmail.com>
> Subject: [PATCH v2 0/9] SMP support for RISC-V
> 
> This patch series adds SMP support for RISC-V to U-Boot. It allows U-Boot to
> run on multi-hart systems (hart is the RISC-V terminology for hardware
> thread). Images passed to bootm will be started on all harts.
> The bootm command is currently the only one that will boot images on all
> harts, bootefi is not yet supported.
> 
> The patches have been successfully tested on both QEMU (machine and
> supervisor mode) and the HiFive Unleashed board (supervisor mode), using
> BBL and OpenSBI.
> Mainline QEMU requires two patches [1, 2] to run in this configuration.
> 
> [1]: https://patchwork.ozlabs.org/patch/1039493/
> [2]: https://patchwork.ozlabs.org/patch/1039082/
> 
> Changes in v2:
> - Remove unneeded quotes from NR_CPUS Kconfig entry
> - Move memory barrier from send_ipi_many() to handle_ipi()
> - Add check in send_ipi_many so that IPIs are only sent to available harts as
> indicated by the available_harts mask
> - Implement hart lottery to pick main hart to run U-Boot
> - Remove CONFIG_MAIN_HART as it is not required anymore
> - Register available harts in the available_harts mask
> - New patch to populate register a0 with the hart ID from the mhartid CSR in
> machine-mode
> - New patch to enable SMP on the SiFive FU540, which was previously sent
> independently
> 
> Lukas Auer (9):
>   riscv: add infrastructure for calling functions on other harts
>   riscv: import the supervisor binary interface header file
>   riscv: implement IPI platform functions using SBI
>   riscv: delay initialization of caches and debug UART
>   riscv: add support for multi-hart systems
>   riscv: boot images passed to bootm on all harts
>   riscv: do not rely on hart ID passed by previous boot stage
>   riscv: fu540: enable SMP
>   riscv: qemu: enable SMP
> 
>  arch/riscv/Kconfig                   |  28 +++++
>  arch/riscv/cpu/cpu.c                 |   9 +-
>  arch/riscv/cpu/start.S               | 152 +++++++++++++++++++++++++--
>  arch/riscv/include/asm/csr.h         |   1 +
>  arch/riscv/include/asm/global_data.h |   6 ++
>  arch/riscv/include/asm/sbi.h         |  94 +++++++++++++++++
>  arch/riscv/include/asm/smp.h         |  53 ++++++++++
>  arch/riscv/lib/Makefile              |   2 +
>  arch/riscv/lib/asm-offsets.c         |   1 +
>  arch/riscv/lib/bootm.c               |  13 ++-
>  arch/riscv/lib/sbi_ipi.c             |  25 +++++
>  arch/riscv/lib/smp.c                 | 116 ++++++++++++++++++++
>  board/emulation/qemu-riscv/Kconfig   |   1 +
>  board/sifive/fu540/Kconfig           |   1 +
>  14 files changed, 492 insertions(+), 10 deletions(-)  create mode 100644
> arch/riscv/include/asm/sbi.h  create mode 100644
> arch/riscv/include/asm/smp.h  create mode 100644 arch/riscv/lib/sbi_ipi.c
> create mode 100644 arch/riscv/lib/smp.c
> 
> --
> 2.20.1

I tried this series on U-Boot-2019.04-rc3 and works fine on
SiFive Unleashed board.

Tested-by: Anup Patel <anup.patel@wdc.com>

I have also pushed these patches to riscv_sifive_fu540_smp_v4
branch of https://github.com/avpatel/u-boot.git

Regards,
Anup

[U-Boot] [PATCH v2 0/9] SMP support for RISC-V

[Auer, Lukas]

On Wed, 2019-03-06 at 04:00 +0000, Anup Patel wrote:
> > -----Original Message-----
> > From: Lukas Auer <lukas.auer@aisec.fraunhofer.de>
> > Sent: Wednesday, March 6, 2019 4:23 AM
> > To: u-boot at lists.denx.de
> > Cc: Atish Patra <Atish.Patra@wdc.com>; Anup Patel
> > <Anup.Patel@wdc.com>; Bin Meng <bmeng.cn@gmail.com>; Andreas
> > Schwab <schwab@suse.de>; Palmer Dabbelt <palmer@sifive.com>;
> > Alexander Graf <agraf@suse.de>; Lukas Auer
> > <lukas.auer@aisec.fraunhofer.de>; Anup Patel <Anup.Patel@wdc.com>;
> > Anup Patel <anup@brainfault.org>; Rick Chen <rick@andestech.com>;
> > Baruch Siach <baruch@tkos.co.il>; Atish Patra <Atish.Patra@wdc.com>
> > ;
> > Stefan Roese <sr@denx.de>; Paul Walmsley <paul.walmsley@sifive.com>
> > ;
> > Daniel Schwierzeck <daniel.schwierzeck@gmail.com>
> > Subject: [PATCH v2 0/9] SMP support for RISC-V
> > 
> > This patch series adds SMP support for RISC-V to U-Boot. It allows
> > U-Boot to
> > run on multi-hart systems (hart is the RISC-V terminology for
> > hardware
> > thread). Images passed to bootm will be started on all harts.
> > The bootm command is currently the only one that will boot images
> > on all
> > harts, bootefi is not yet supported.
> > 
> > The patches have been successfully tested on both QEMU (machine and
> > supervisor mode) and the HiFive Unleashed board (supervisor mode),
> > using
> > BBL and OpenSBI.
> > Mainline QEMU requires two patches [1, 2] to run in this
> > configuration.
> > 
> > [1]: https://patchwork.ozlabs.org/patch/1039493/
> > [2]: https://patchwork.ozlabs.org/patch/1039082/
> > 
> > Changes in v2:
> > - Remove unneeded quotes from NR_CPUS Kconfig entry
> > - Move memory barrier from send_ipi_many() to handle_ipi()
> > - Add check in send_ipi_many so that IPIs are only sent to
> > available harts as
> > indicated by the available_harts mask
> > - Implement hart lottery to pick main hart to run U-Boot
> > - Remove CONFIG_MAIN_HART as it is not required anymore
> > - Register available harts in the available_harts mask
> > - New patch to populate register a0 with the hart ID from the
> > mhartid CSR in
> > machine-mode
> > - New patch to enable SMP on the SiFive FU540, which was previously
> > sent
> > independently
> > 
> > Lukas Auer (9):
> >   riscv: add infrastructure for calling functions on other harts
> >   riscv: import the supervisor binary interface header file
> >   riscv: implement IPI platform functions using SBI
> >   riscv: delay initialization of caches and debug UART
> >   riscv: add support for multi-hart systems
> >   riscv: boot images passed to bootm on all harts
> >   riscv: do not rely on hart ID passed by previous boot stage
> >   riscv: fu540: enable SMP
> >   riscv: qemu: enable SMP
> > 
> >  arch/riscv/Kconfig                   |  28 +++++
> >  arch/riscv/cpu/cpu.c                 |   9 +-
> >  arch/riscv/cpu/start.S               | 152
> > +++++++++++++++++++++++++--
> >  arch/riscv/include/asm/csr.h         |   1 +
> >  arch/riscv/include/asm/global_data.h |   6 ++
> >  arch/riscv/include/asm/sbi.h         |  94 +++++++++++++++++
> >  arch/riscv/include/asm/smp.h         |  53 ++++++++++
> >  arch/riscv/lib/Makefile              |   2 +
> >  arch/riscv/lib/asm-offsets.c         |   1 +
> >  arch/riscv/lib/bootm.c               |  13 ++-
> >  arch/riscv/lib/sbi_ipi.c             |  25 +++++
> >  arch/riscv/lib/smp.c                 | 116 ++++++++++++++++++++
> >  board/emulation/qemu-riscv/Kconfig   |   1 +
> >  board/sifive/fu540/Kconfig           |   1 +
> >  14 files changed, 492 insertions(+), 10 deletions(-)  create mode
> > 100644
> > arch/riscv/include/asm/sbi.h  create mode 100644
> > arch/riscv/include/asm/smp.h  create mode 100644
> > arch/riscv/lib/sbi_ipi.c
> > create mode 100644 arch/riscv/lib/smp.c
> > 
> > --
> > 2.20.1
> 
> I tried this series on U-Boot-2019.04-rc3 and works fine on
> SiFive Unleashed board.
> 
> Tested-by: Anup Patel <anup.patel@wdc.com>
> 
> I have also pushed these patches to riscv_sifive_fu540_smp_v4
> branch of https://github.com/avpatel/u-boot.git
> 

Thank you for testing the series, Anup!
Did you observe anymore issues, where not all harts enter Linux?

Thanks,
Lukas

[U-Boot] [PATCH v2 0/9] SMP support for RISC-V

[Anup Patel]

> -----Original Message-----
> From: Auer, Lukas <lukas.auer@aisec.fraunhofer.de>
> Sent: Wednesday, March 6, 2019 2:52 PM
> To: u-boot at lists.denx.de; Anup Patel <Anup.Patel@wdc.com>
> Cc: paul.walmsley at sifive.com; agraf at suse.de; anup at brainfault.org;
> baruch at tkos.co.il; daniel.schwierzeck at gmail.com; bmeng.cn at gmail.com;
> rick at andestech.com; sr at denx.de; schwab at suse.de; palmer at sifive.com;
> Atish Patra <Atish.Patra@wdc.com>
> Subject: Re: [PATCH v2 0/9] SMP support for RISC-V
> 
> On Wed, 2019-03-06 at 04:00 +0000, Anup Patel wrote:
> > > -----Original Message-----
> > > From: Lukas Auer <lukas.auer@aisec.fraunhofer.de>
> > > Sent: Wednesday, March 6, 2019 4:23 AM
> > > To: u-boot at lists.denx.de
> > > Cc: Atish Patra <Atish.Patra@wdc.com>; Anup Patel
> > > <Anup.Patel@wdc.com>; Bin Meng <bmeng.cn@gmail.com>; Andreas
> Schwab
> > > <schwab@suse.de>; Palmer Dabbelt <palmer@sifive.com>; Alexander
> Graf
> > > <agraf@suse.de>; Lukas Auer <lukas.auer@aisec.fraunhofer.de>; Anup
> > > Patel <Anup.Patel@wdc.com>; Anup Patel <anup@brainfault.org>; Rick
> > > Chen <rick@andestech.com>; Baruch Siach <baruch@tkos.co.il>; Atish
> > > Patra <Atish.Patra@wdc.com> ; Stefan Roese <sr@denx.de>; Paul
> > > Walmsley <paul.walmsley@sifive.com> ; Daniel Schwierzeck
> > > <daniel.schwierzeck@gmail.com>
> > > Subject: [PATCH v2 0/9] SMP support for RISC-V
> > >
> > > This patch series adds SMP support for RISC-V to U-Boot. It allows
> > > U-Boot to run on multi-hart systems (hart is the RISC-V terminology
> > > for hardware thread). Images passed to bootm will be started on all
> > > harts.
> > > The bootm command is currently the only one that will boot images on
> > > all harts, bootefi is not yet supported.
> > >
> > > The patches have been successfully tested on both QEMU (machine and
> > > supervisor mode) and the HiFive Unleashed board (supervisor mode),
> > > using BBL and OpenSBI.
> > > Mainline QEMU requires two patches [1, 2] to run in this
> > > configuration.
> > >
> > > [1]: https://patchwork.ozlabs.org/patch/1039493/
> > > [2]: https://patchwork.ozlabs.org/patch/1039082/
> > >
> > > Changes in v2:
> > > - Remove unneeded quotes from NR_CPUS Kconfig entry
> > > - Move memory barrier from send_ipi_many() to handle_ipi()
> > > - Add check in send_ipi_many so that IPIs are only sent to available
> > > harts as indicated by the available_harts mask
> > > - Implement hart lottery to pick main hart to run U-Boot
> > > - Remove CONFIG_MAIN_HART as it is not required anymore
> > > - Register available harts in the available_harts mask
> > > - New patch to populate register a0 with the hart ID from the
> > > mhartid CSR in machine-mode
> > > - New patch to enable SMP on the SiFive FU540, which was previously
> > > sent independently
> > >
> > > Lukas Auer (9):
> > >   riscv: add infrastructure for calling functions on other harts
> > >   riscv: import the supervisor binary interface header file
> > >   riscv: implement IPI platform functions using SBI
> > >   riscv: delay initialization of caches and debug UART
> > >   riscv: add support for multi-hart systems
> > >   riscv: boot images passed to bootm on all harts
> > >   riscv: do not rely on hart ID passed by previous boot stage
> > >   riscv: fu540: enable SMP
> > >   riscv: qemu: enable SMP
> > >
> > >  arch/riscv/Kconfig                   |  28 +++++
> > >  arch/riscv/cpu/cpu.c                 |   9 +-
> > >  arch/riscv/cpu/start.S               | 152
> > > +++++++++++++++++++++++++--
> > >  arch/riscv/include/asm/csr.h         |   1 +
> > >  arch/riscv/include/asm/global_data.h |   6 ++
> > >  arch/riscv/include/asm/sbi.h         |  94 +++++++++++++++++
> > >  arch/riscv/include/asm/smp.h         |  53 ++++++++++
> > >  arch/riscv/lib/Makefile              |   2 +
> > >  arch/riscv/lib/asm-offsets.c         |   1 +
> > >  arch/riscv/lib/bootm.c               |  13 ++-
> > >  arch/riscv/lib/sbi_ipi.c             |  25 +++++
> > >  arch/riscv/lib/smp.c                 | 116 ++++++++++++++++++++
> > >  board/emulation/qemu-riscv/Kconfig   |   1 +
> > >  board/sifive/fu540/Kconfig           |   1 +
> > >  14 files changed, 492 insertions(+), 10 deletions(-)  create mode
> > > 100644
> > > arch/riscv/include/asm/sbi.h  create mode 100644
> > > arch/riscv/include/asm/smp.h  create mode 100644
> > > arch/riscv/lib/sbi_ipi.c create mode 100644 arch/riscv/lib/smp.c
> > >
> > > --
> > > 2.20.1
> >
> > I tried this series on U-Boot-2019.04-rc3 and works fine on SiFive
> > Unleashed board.
> >
> > Tested-by: Anup Patel <anup.patel@wdc.com>
> >
> > I have also pushed these patches to riscv_sifive_fu540_smp_v4 branch
> > of https://github.com/avpatel/u-boot.git
> >
> 
> Thank you for testing the series, Anup!
> Did you observe anymore issues, where not all harts enter Linux?

I tried using cold-boot (using power-on button) worked fine for me
10 times. I did not try more.

We are trying to make OpenSBI stable with warm-boot (using reset
button) as well but I have not tried that with U-Boot using reset button.

Regards,
Anup


> 
> Thanks,
> Lukas

[U-Boot] [PATCH v2 0/9] SMP support for RISC-V

[Andreas Schwab]

Apparently sometimes u-boot tries to boot the kernel on heart 0 (the E51
core), which will then fail to start userspace, since that cannot cope
with the missing fpu.

Andreas.

-- 
Andreas Schwab, SUSE Labs, schwab at suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

[U-Boot] [PATCH v2 0/9] SMP support for RISC-V

[Anup Patel]

> -----Original Message-----
> From: Andreas Schwab <schwab@suse.de>
> Sent: Wednesday, March 6, 2019 4:27 PM
> To: Anup Patel <Anup.Patel@wdc.com>
> Cc: Auer, Lukas <lukas.auer@aisec.fraunhofer.de>; u-boot at lists.denx.de;
> paul.walmsley at sifive.com; agraf at suse.de; anup at brainfault.org;
> baruch at tkos.co.il; daniel.schwierzeck at gmail.com; bmeng.cn at gmail.com;
> rick at andestech.com; sr at denx.de; palmer at sifive.com; Atish Patra
> <Atish.Patra@wdc.com>
> Subject: Re: [PATCH v2 0/9] SMP support for RISC-V
> 
> Apparently sometimes u-boot tries to boot the kernel on heart 0 (the E51
> core), which will then fail to start userspace, since that cannot cope with the
> missing fpu.

That's not possible because in this series we have "available_hart_mask"
to track HARTs that entered U-Boot.

Recently, Atish made some progress with OpenSBI warm-boot issues. I
will let him provide details about it.

Regards,
Anup

[U-Boot] [PATCH v2 0/9] SMP support for RISC-V

[Andreas Schwab]

On Mär 06 2019, Anup Patel <Anup.Patel@wdc.com> wrote:

>> -----Original Message-----
>> From: Andreas Schwab <schwab@suse.de>
>> Sent: Wednesday, March 6, 2019 4:27 PM
>> To: Anup Patel <Anup.Patel@wdc.com>
>> Cc: Auer, Lukas <lukas.auer@aisec.fraunhofer.de>; u-boot at lists.denx.de;
>> paul.walmsley at sifive.com; agraf at suse.de; anup at brainfault.org;
>> baruch at tkos.co.il; daniel.schwierzeck at gmail.com; bmeng.cn at gmail.com;
>> rick at andestech.com; sr at denx.de; palmer at sifive.com; Atish Patra
>> <Atish.Patra@wdc.com>
>> Subject: Re: [PATCH v2 0/9] SMP support for RISC-V
>> 
>> Apparently sometimes u-boot tries to boot the kernel on heart 0 (the E51
>> core), which will then fail to start userspace, since that cannot cope with the
>> missing fpu.
>
> That's not possible

Yes, it is.


OpenSBI v0.3 (Mar  6 2019 10:55:01)
   ____                    _____ ____ _____
  / __ \                  / ____|  _ \_   _|
 | |  | |_ __   ___ _ __ | (___ | |_) || |
 | |  | | '_ \ / _ \ '_ \ \___ \|  _ < | |
 | |__| | |_) |  __/ | | |____) | |_) || |_
  \____/| .__/ \___|_| |_|_____/|____/_____|
        | |
        |_|

Platform Name          : SiFive Freedom U540
Platform HART Features : RV64ACDFIMSU
Platform Max HARTs     : 5
Current Hart           : 2
Firmware Base          : 0x80000000
Firmware Size          : 88 KB
Runtime SBI Version    : 0.1

PMP0: 0x0000000080000000-0x000000008001ffff (A)
PMP1: 0x0000000000000000-0x0000007fffffffff (A,R,W,X)


U-Boot 2019.04-rc3-00010-g3ea5582c09 (Mar 06 2019 - 10:06:10 +0100)

CPU:   rv64imac
Model: sifive,hifive-unleashed-a00
DRAM:  8 GiB

Andreas.

-- 
Andreas Schwab, SUSE Labs, schwab@suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

[U-Boot] [PATCH v2 0/9] SMP support for RISC-V

[Anup Patel]

On Wed, Mar 6, 2019 at 5:17 PM Andreas Schwab <schwab@suse.de> wrote:
>
> On Mär 06 2019, Anup Patel <Anup.Patel@wdc.com> wrote:
>
> >> -----Original Message-----
> >> From: Andreas Schwab <schwab@suse.de>
> >> Sent: Wednesday, March 6, 2019 4:27 PM
> >> To: Anup Patel <Anup.Patel@wdc.com>
> >> Cc: Auer, Lukas <lukas.auer@aisec.fraunhofer.de>; u-boot at lists.denx.de;
> >> paul.walmsley at sifive.com; agraf at suse.de; anup at brainfault.org;
> >> baruch at tkos.co.il; daniel.schwierzeck at gmail.com; bmeng.cn at gmail.com;
> >> rick at andestech.com; sr at denx.de; palmer at sifive.com; Atish Patra
> >> <Atish.Patra@wdc.com>
> >> Subject: Re: [PATCH v2 0/9] SMP support for RISC-V
> >>
> >> Apparently sometimes u-boot tries to boot the kernel on heart 0 (the E51
> >> core), which will then fail to start userspace, since that cannot cope with the
> >> missing fpu.
> >
> > That's not possible
>
> Yes, it is.
>
>
> OpenSBI v0.3 (Mar  6 2019 10:55:01)
>    ____                    _____ ____ _____
>   / __ \                  / ____|  _ \_   _|
>  | |  | |_ __   ___ _ __ | (___ | |_) || |
>  | |  | | '_ \ / _ \ '_ \ \___ \|  _ < | |
>  | |__| | |_) |  __/ | | |____) | |_) || |_
>   \____/| .__/ \___|_| |_|_____/|____/_____|
>         | |
>         |_|
>
> Platform Name          : SiFive Freedom U540
> Platform HART Features : RV64ACDFIMSU
> Platform Max HARTs     : 5
> Current Hart           : 2
> Firmware Base          : 0x80000000
> Firmware Size          : 88 KB
> Runtime SBI Version    : 0.1
>
> PMP0: 0x0000000080000000-0x000000008001ffff (A)
> PMP1: 0x0000000000000000-0x0000007fffffffff (A,R,W,X)
>
>
> U-Boot 2019.04-rc3-00010-g3ea5582c09 (Mar 06 2019 - 10:06:10 +0100)
>
> CPU:   rv64imac
> Model: sifive,hifive-unleashed-a00
> DRAM:  8 GiB

How does this prove that U-Boot is booting on HART 0?

This seems to be warm reset issues for which fixes from Atish are
not yet merged.

Regards,
Anup

[U-Boot] [PATCH v2 0/9] SMP support for RISC-V

[Andreas Schwab]

On Mär 06 2019, Anup Patel <anup@brainfault.org> wrote:

> On Wed, Mar 6, 2019 at 5:17 PM Andreas Schwab <schwab@suse.de> wrote:
>>
>> On Mär 06 2019, Anup Patel <Anup.Patel@wdc.com> wrote:
>>
>> >> -----Original Message-----
>> >> From: Andreas Schwab <schwab@suse.de>
>> >> Sent: Wednesday, March 6, 2019 4:27 PM
>> >> To: Anup Patel <Anup.Patel@wdc.com>
>> >> Cc: Auer, Lukas <lukas.auer@aisec.fraunhofer.de>; u-boot at lists.denx.de;
>> >> paul.walmsley at sifive.com; agraf at suse.de; anup at brainfault.org;
>> >> baruch at tkos.co.il; daniel.schwierzeck at gmail.com; bmeng.cn at gmail.com;
>> >> rick at andestech.com; sr at denx.de; palmer at sifive.com; Atish Patra
>> >> <Atish.Patra@wdc.com>
>> >> Subject: Re: [PATCH v2 0/9] SMP support for RISC-V
>> >>
>> >> Apparently sometimes u-boot tries to boot the kernel on heart 0 (the E51
>> >> core), which will then fail to start userspace, since that cannot cope with the
>> >> missing fpu.
>> >
>> > That's not possible
>>
>> Yes, it is.
>>
>>
>> OpenSBI v0.3 (Mar  6 2019 10:55:01)
>>    ____                    _____ ____ _____
>>   / __ \                  / ____|  _ \_   _|
>>  | |  | |_ __   ___ _ __ | (___ | |_) || |
>>  | |  | | '_ \ / _ \ '_ \ \___ \|  _ < | |
>>  | |__| | |_) |  __/ | | |____) | |_) || |_
>>   \____/| .__/ \___|_| |_|_____/|____/_____|
>>         | |
>>         |_|
>>
>> Platform Name          : SiFive Freedom U540
>> Platform HART Features : RV64ACDFIMSU
>> Platform Max HARTs     : 5
>> Current Hart           : 2
>> Firmware Base          : 0x80000000
>> Firmware Size          : 88 KB
>> Runtime SBI Version    : 0.1
>>
>> PMP0: 0x0000000080000000-0x000000008001ffff (A)
>> PMP1: 0x0000000000000000-0x0000007fffffffff (A,R,W,X)
>>
>>
>> U-Boot 2019.04-rc3-00010-g3ea5582c09 (Mar 06 2019 - 10:06:10 +0100)
>>
>> CPU:   rv64imac
>> Model: sifive,hifive-unleashed-a00
>> DRAM:  8 GiB
>
> How does this prove that U-Boot is booting on HART 0?

See the CPU isa.

Andreas.

-- 
Andreas Schwab, SUSE Labs, schwab at suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

[U-Boot] [PATCH v2 0/9] SMP support for RISC-V

[Auer, Lukas]

On Wed, 2019-03-06 at 13:01 +0100, Andreas Schwab wrote:
> On Mär 06 2019, Anup Patel <anup@brainfault.org> wrote:
> 
> > On Wed, Mar 6, 2019 at 5:17 PM Andreas Schwab <schwab@suse.de>
> > wrote:
> > > On Mär 06 2019, Anup Patel <Anup.Patel@wdc.com> wrote:
> > > 
> > > > > -----Original Message-----
> > > > > From: Andreas Schwab <schwab@suse.de>
> > > > > Sent: Wednesday, March 6, 2019 4:27 PM
> > > > > To: Anup Patel <Anup.Patel@wdc.com>
> > > > > Cc: Auer, Lukas <lukas.auer@aisec.fraunhofer.de>; 
> > > > > u-boot at lists.denx.de;
> > > > > paul.walmsley at sifive.com; agraf at suse.de; anup at brainfault.org;
> > > > > baruch at tkos.co.il; daniel.schwierzeck at gmail.com; 
> > > > > bmeng.cn at gmail.com;
> > > > > rick at andestech.com; sr at denx.de; palmer at sifive.com; Atish
> > > > > Patra
> > > > > <Atish.Patra@wdc.com>
> > > > > Subject: Re: [PATCH v2 0/9] SMP support for RISC-V
> > > > > 
> > > > > Apparently sometimes u-boot tries to boot the kernel on heart
> > > > > 0 (the E51
> > > > > core), which will then fail to start userspace, since that
> > > > > cannot cope with the
> > > > > missing fpu.
> > > > 
> > > > That's not possible
> > > 
> > > Yes, it is.
> > > 
> > > 
> > > OpenSBI v0.3 (Mar  6 2019 10:55:01)
> > >    ____                    _____ ____ _____
> > >   / __ \                  / ____|  _ \_   _|
> > >  | |  | |_ __   ___ _ __ | (___ | |_) || |
> > >  | |  | | '_ \ / _ \ '_ \ \___ \|  _ < | |
> > >  | |__| | |_) |  __/ | | |____) | |_) || |_
> > >   \____/| .__/ \___|_| |_|_____/|____/_____|
> > >         | |
> > >         |_|
> > > 
> > > Platform Name          : SiFive Freedom U540
> > > Platform HART Features : RV64ACDFIMSU
> > > Platform Max HARTs     : 5
> > > Current Hart           : 2
> > > Firmware Base          : 0x80000000
> > > Firmware Size          : 88 KB
> > > Runtime SBI Version    : 0.1
> > > 
> > > PMP0: 0x0000000080000000-0x000000008001ffff (A)
> > > PMP1: 0x0000000000000000-0x0000007fffffffff (A,R,W,X)
> > > 
> > > 
> > > U-Boot 2019.04-rc3-00010-g3ea5582c09 (Mar 06 2019 - 10:06:10
> > > +0100)
> > > 
> > > CPU:   rv64imac
> > > Model: sifive,hifive-unleashed-a00
> > > DRAM:  8 GiB
> > 
> > How does this prove that U-Boot is booting on HART 0?
> 
> See the CPU isa.
> 

Interesting.. U-Boot assumes that it can run on any core it is started
on. In this case, OpenSBI must have booted its payload on hart 0.

Lukas

[U-Boot] [PATCH v2 0/9] SMP support for RISC-V

[Auer, Lukas]

On Wed, 2019-03-06 at 10:07 +0000, Anup Patel wrote:
> > -----Original Message-----
> > From: Auer, Lukas <lukas.auer@aisec.fraunhofer.de>
> > Sent: Wednesday, March 6, 2019 2:52 PM
> > To: u-boot at lists.denx.de; Anup Patel <Anup.Patel@wdc.com>
> > Cc: paul.walmsley at sifive.com; agraf at suse.de; anup at brainfault.org;
> > baruch at tkos.co.il; daniel.schwierzeck at gmail.com; bmeng.cn at gmail.com
> > ;
> > rick at andestech.com; sr at denx.de; schwab at suse.de; palmer at sifive.com;
> > Atish Patra <Atish.Patra@wdc.com>
> > Subject: Re: [PATCH v2 0/9] SMP support for RISC-V
> > 
> > On Wed, 2019-03-06 at 04:00 +0000, Anup Patel wrote:
> > > > -----Original Message-----
> > > > From: Lukas Auer <lukas.auer@aisec.fraunhofer.de>
> > > > Sent: Wednesday, March 6, 2019 4:23 AM
> > > > To: u-boot at lists.denx.de
> > > > Cc: Atish Patra <Atish.Patra@wdc.com>; Anup Patel
> > > > <Anup.Patel@wdc.com>; Bin Meng <bmeng.cn@gmail.com>; Andreas
> > Schwab
> > > > <schwab@suse.de>; Palmer Dabbelt <palmer@sifive.com>; Alexander
> > Graf
> > > > <agraf@suse.de>; Lukas Auer <lukas.auer@aisec.fraunhofer.de>;
> > > > Anup
> > > > Patel <Anup.Patel@wdc.com>; Anup Patel <anup@brainfault.org>;
> > > > Rick
> > > > Chen <rick@andestech.com>; Baruch Siach <baruch@tkos.co.il>;
> > > > Atish
> > > > Patra <Atish.Patra@wdc.com> ; Stefan Roese <sr@denx.de>; Paul
> > > > Walmsley <paul.walmsley@sifive.com> ; Daniel Schwierzeck
> > > > <daniel.schwierzeck@gmail.com>
> > > > Subject: [PATCH v2 0/9] SMP support for RISC-V
> > > > 
> > > > This patch series adds SMP support for RISC-V to U-Boot. It
> > > > allows
> > > > U-Boot to run on multi-hart systems (hart is the RISC-V
> > > > terminology
> > > > for hardware thread). Images passed to bootm will be started on
> > > > all
> > > > harts.
> > > > The bootm command is currently the only one that will boot
> > > > images on
> > > > all harts, bootefi is not yet supported.
> > > > 
> > > > The patches have been successfully tested on both QEMU (machine
> > > > and
> > > > supervisor mode) and the HiFive Unleashed board (supervisor
> > > > mode),
> > > > using BBL and OpenSBI.
> > > > Mainline QEMU requires two patches [1, 2] to run in this
> > > > configuration.
> > > > 
> > > > [1]: https://patchwork.ozlabs.org/patch/1039493/
> > > > [2]: https://patchwork.ozlabs.org/patch/1039082/
> > > > 
> > > > Changes in v2:
> > > > - Remove unneeded quotes from NR_CPUS Kconfig entry
> > > > - Move memory barrier from send_ipi_many() to handle_ipi()
> > > > - Add check in send_ipi_many so that IPIs are only sent to
> > > > available
> > > > harts as indicated by the available_harts mask
> > > > - Implement hart lottery to pick main hart to run U-Boot
> > > > - Remove CONFIG_MAIN_HART as it is not required anymore
> > > > - Register available harts in the available_harts mask
> > > > - New patch to populate register a0 with the hart ID from the
> > > > mhartid CSR in machine-mode
> > > > - New patch to enable SMP on the SiFive FU540, which was
> > > > previously
> > > > sent independently
> > > > 
> > > > Lukas Auer (9):
> > > >   riscv: add infrastructure for calling functions on other
> > > > harts
> > > >   riscv: import the supervisor binary interface header file
> > > >   riscv: implement IPI platform functions using SBI
> > > >   riscv: delay initialization of caches and debug UART
> > > >   riscv: add support for multi-hart systems
> > > >   riscv: boot images passed to bootm on all harts
> > > >   riscv: do not rely on hart ID passed by previous boot stage
> > > >   riscv: fu540: enable SMP
> > > >   riscv: qemu: enable SMP
> > > > 
> > > >  arch/riscv/Kconfig                   |  28 +++++
> > > >  arch/riscv/cpu/cpu.c                 |   9 +-
> > > >  arch/riscv/cpu/start.S               | 152
> > > > +++++++++++++++++++++++++--
> > > >  arch/riscv/include/asm/csr.h         |   1 +
> > > >  arch/riscv/include/asm/global_data.h |   6 ++
> > > >  arch/riscv/include/asm/sbi.h         |  94 +++++++++++++++++
> > > >  arch/riscv/include/asm/smp.h         |  53 ++++++++++
> > > >  arch/riscv/lib/Makefile              |   2 +
> > > >  arch/riscv/lib/asm-offsets.c         |   1 +
> > > >  arch/riscv/lib/bootm.c               |  13 ++-
> > > >  arch/riscv/lib/sbi_ipi.c             |  25 +++++
> > > >  arch/riscv/lib/smp.c                 | 116
> > > > ++++++++++++++++++++
> > > >  board/emulation/qemu-riscv/Kconfig   |   1 +
> > > >  board/sifive/fu540/Kconfig           |   1 +
> > > >  14 files changed, 492 insertions(+), 10 deletions(-)  create
> > > > mode
> > > > 100644
> > > > arch/riscv/include/asm/sbi.h  create mode 100644
> > > > arch/riscv/include/asm/smp.h  create mode 100644
> > > > arch/riscv/lib/sbi_ipi.c create mode 100644
> > > > arch/riscv/lib/smp.c
> > > > 
> > > > --
> > > > 2.20.1
> > > 
> > > I tried this series on U-Boot-2019.04-rc3 and works fine on
> > > SiFive
> > > Unleashed board.
> > > 
> > > Tested-by: Anup Patel <anup.patel@wdc.com>
> > > 
> > > I have also pushed these patches to riscv_sifive_fu540_smp_v4
> > > branch
> > > of https://github.com/avpatel/u-boot.git
> > > 
> > 
> > Thank you for testing the series, Anup!
> > Did you observe anymore issues, where not all harts enter Linux?
> 
> I tried using cold-boot (using power-on button) worked fine for me
> 10 times. I did not try more.
> 
> We are trying to make OpenSBI stable with warm-boot (using reset
> button) as well but I have not tried that with U-Boot using reset
> button.
> 

Great, so it seems the cold-boot issue is fixed now. Thanks for testing
this!

Lukas

[U-Boot] [PATCH v2 0/9] SMP support for RISC-V

[Anup Patel]

On Wed, Mar 6, 2019 at 5:45 PM Auer, Lukas
<lukas.auer@aisec.fraunhofer.de> wrote:
>
> On Wed, 2019-03-06 at 13:01 +0100, Andreas Schwab wrote:
> > On Mär 06 2019, Anup Patel <anup@brainfault.org> wrote:
> >
> > > On Wed, Mar 6, 2019 at 5:17 PM Andreas Schwab <schwab@suse.de>
> > > wrote:
> > > > On Mär 06 2019, Anup Patel <Anup.Patel@wdc.com> wrote:
> > > >
> > > > > > -----Original Message-----
> > > > > > From: Andreas Schwab <schwab@suse.de>
> > > > > > Sent: Wednesday, March 6, 2019 4:27 PM
> > > > > > To: Anup Patel <Anup.Patel@wdc.com>
> > > > > > Cc: Auer, Lukas <lukas.auer@aisec.fraunhofer.de>;
> > > > > > u-boot at lists.denx.de;
> > > > > > paul.walmsley at sifive.com; agraf at suse.de; anup at brainfault.org;
> > > > > > baruch at tkos.co.il; daniel.schwierzeck at gmail.com;
> > > > > > bmeng.cn at gmail.com;
> > > > > > rick at andestech.com; sr at denx.de; palmer at sifive.com; Atish
> > > > > > Patra
> > > > > > <Atish.Patra@wdc.com>
> > > > > > Subject: Re: [PATCH v2 0/9] SMP support for RISC-V
> > > > > >
> > > > > > Apparently sometimes u-boot tries to boot the kernel on heart
> > > > > > 0 (the E51
> > > > > > core), which will then fail to start userspace, since that
> > > > > > cannot cope with the
> > > > > > missing fpu.
> > > > >
> > > > > That's not possible
> > > >
> > > > Yes, it is.
> > > >
> > > >
> > > > OpenSBI v0.3 (Mar  6 2019 10:55:01)
> > > >    ____                    _____ ____ _____
> > > >   / __ \                  / ____|  _ \_   _|
> > > >  | |  | |_ __   ___ _ __ | (___ | |_) || |
> > > >  | |  | | '_ \ / _ \ '_ \ \___ \|  _ < | |
> > > >  | |__| | |_) |  __/ | | |____) | |_) || |_
> > > >   \____/| .__/ \___|_| |_|_____/|____/_____|
> > > >         | |
> > > >         |_|
> > > >
> > > > Platform Name          : SiFive Freedom U540
> > > > Platform HART Features : RV64ACDFIMSU
> > > > Platform Max HARTs     : 5
> > > > Current Hart           : 2
> > > > Firmware Base          : 0x80000000
> > > > Firmware Size          : 88 KB
> > > > Runtime SBI Version    : 0.1
> > > >
> > > > PMP0: 0x0000000080000000-0x000000008001ffff (A)
> > > > PMP1: 0x0000000000000000-0x0000007fffffffff (A,R,W,X)
> > > >
> > > >
> > > > U-Boot 2019.04-rc3-00010-g3ea5582c09 (Mar 06 2019 - 10:06:10
> > > > +0100)
> > > >
> > > > CPU:   rv64imac
> > > > Model: sifive,hifive-unleashed-a00
> > > > DRAM:  8 GiB
> > >
> > > How does this prove that U-Boot is booting on HART 0?
> >
> > See the CPU isa.
> >
>
> Interesting.. U-Boot assumes that it can run on any core it is started
> on. In this case, OpenSBI must have booted its payload on hart 0.

This is certainly not reproducible on cold-boot at my end but this
does mean OpenSBI has booted HART0 and let it jump to U-Boot.

Now OpenSBI (by default) on SiFive FU540 does not allow HART0
to go forward due to lack of S-mode. I think this is definitely the
warm-boot issue where OpenSBI sees corrupted memory contents.

Regards,
Anup

[U-Boot] [PATCH v2 0/9] SMP support for RISC-V

[Atish Patra]

On 3/6/19 4:32 AM, Anup Patel wrote:
> On Wed, Mar 6, 2019 at 5:45 PM Auer, Lukas
> <lukas.auer@aisec.fraunhofer.de> wrote:
>>
>> On Wed, 2019-03-06 at 13:01 +0100, Andreas Schwab wrote:
>>> On Mär 06 2019, Anup Patel <anup@brainfault.org> wrote:
>>>
>>>> On Wed, Mar 6, 2019 at 5:17 PM Andreas Schwab <schwab@suse.de>
>>>> wrote:
>>>>> On Mär 06 2019, Anup Patel <Anup.Patel@wdc.com> wrote:
>>>>>
>>>>>>> -----Original Message-----
>>>>>>> From: Andreas Schwab <schwab@suse.de>
>>>>>>> Sent: Wednesday, March 6, 2019 4:27 PM
>>>>>>> To: Anup Patel <Anup.Patel@wdc.com>
>>>>>>> Cc: Auer, Lukas <lukas.auer@aisec.fraunhofer.de>;
>>>>>>> u-boot at lists.denx.de;
>>>>>>> paul.walmsley at sifive.com; agraf at suse.de; anup at brainfault.org;
>>>>>>> baruch at tkos.co.il; daniel.schwierzeck at gmail.com;
>>>>>>> bmeng.cn at gmail.com;
>>>>>>> rick at andestech.com; sr at denx.de; palmer at sifive.com; Atish
>>>>>>> Patra
>>>>>>> <Atish.Patra@wdc.com>
>>>>>>> Subject: Re: [PATCH v2 0/9] SMP support for RISC-V
>>>>>>>
>>>>>>> Apparently sometimes u-boot tries to boot the kernel on heart
>>>>>>> 0 (the E51
>>>>>>> core), which will then fail to start userspace, since that
>>>>>>> cannot cope with the
>>>>>>> missing fpu.
>>>>>>
>>>>>> That's not possible
>>>>>
>>>>> Yes, it is.
>>>>>
>>>>>
>>>>> OpenSBI v0.3 (Mar  6 2019 10:55:01)
>>>>>     ____                    _____ ____ _____
>>>>>    / __ \                  / ____|  _ \_   _|
>>>>>   | |  | |_ __   ___ _ __ | (___ | |_) || |
>>>>>   | |  | | '_ \ / _ \ '_ \ \___ \|  _ < | |
>>>>>   | |__| | |_) |  __/ | | |____) | |_) || |_
>>>>>    \____/| .__/ \___|_| |_|_____/|____/_____|
>>>>>          | |
>>>>>          |_|
>>>>>
>>>>> Platform Name          : SiFive Freedom U540
>>>>> Platform HART Features : RV64ACDFIMSU
>>>>> Platform Max HARTs     : 5
>>>>> Current Hart           : 2
>>>>> Firmware Base          : 0x80000000
>>>>> Firmware Size          : 88 KB
>>>>> Runtime SBI Version    : 0.1
>>>>>
>>>>> PMP0: 0x0000000080000000-0x000000008001ffff (A)
>>>>> PMP1: 0x0000000000000000-0x0000007fffffffff (A,R,W,X)
>>>>>
>>>>>
>>>>> U-Boot 2019.04-rc3-00010-g3ea5582c09 (Mar 06 2019 - 10:06:10
>>>>> +0100)
>>>>>
>>>>> CPU:   rv64imac
>>>>> Model: sifive,hifive-unleashed-a00
>>>>> DRAM:  8 GiB
>>>>
>>>> How does this prove that U-Boot is booting on HART 0?
>>>
>>> See the CPU isa.
>>>
>>
>> Interesting.. U-Boot assumes that it can run on any core it is started
>> on. In this case, OpenSBI must have booted its payload on hart 0.
> 
> This is certainly not reproducible on cold-boot at my end but this
> does mean OpenSBI has booted HART0 and let it jump to U-Boot.
> 
> Now OpenSBI (by default) on SiFive FU540 does not allow HART0
> to go forward due to lack of S-mode. I think this is definitely the
> warm-boot issue where OpenSBI sees corrupted memory contents.
> 

I am able to test both warm-boot and cold-boot several times(>10) 
without any issue with following pending PR in openSBI.

https://github.com/riscv/opensbi/pull/84

@Andreas @Anup: Can you please apply the above PR on top of master and 
verify at your end as well?

All the harts booted in Linux every time as well.

Regards,
Atish
> Regards,
> Anup
> 

[U-Boot] [PATCH v2 1/9] riscv: add infrastructure for calling functions on other harts

[Atish Patra]

On 3/5/19 2:54 PM, Lukas Auer wrote:
> Harts on RISC-V boot independently, U-Boot is responsible for managing
> them. Functions are called on other harts with smp_call_function(),
> which sends inter-processor interrupts (IPIs) to all other available
> harts. Available harts are those marked as available in the device tree
> and present in the available_harts mask stored in global data. The
> available_harts mask is used to register all harts that have entered
> U-Boot. Functions are specified with their address and two function
> arguments (argument 2 and 3). The first function argument is always the
> hart ID of the hart calling the function. On the other harts, the IPI
> interrupt handler handle_ipi() must be called on software interrupts to
> handle the request and call the specified function.
> 
> Functions are stored in the ipi_data data structure. Every hart has its
> own data structure in global data. While this is not required at the
> moment (all harts are expected to boot Linux), this does allow future
> expansion, where other harts may be used for monitoring or other tasks.
> 
> Signed-off-by: Lukas Auer <lukas.auer@aisec.fraunhofer.de>
> ---
> 
> Changes in v2:
> - Remove unneeded quotes from NR_CPUS Kconfig entry
> - Move memory barrier from send_ipi_many() to handle_ipi()
> - Add check in send_ipi_many so that IPIs are only sent to available
> harts as indicated by the available_harts mask
> 
>   arch/riscv/Kconfig                   |  19 +++++
>   arch/riscv/include/asm/global_data.h |   6 ++
>   arch/riscv/include/asm/smp.h         |  53 ++++++++++++
>   arch/riscv/lib/Makefile              |   1 +
>   arch/riscv/lib/smp.c                 | 116 +++++++++++++++++++++++++++
>   5 files changed, 195 insertions(+)
>   create mode 100644 arch/riscv/include/asm/smp.h
>   create mode 100644 arch/riscv/lib/smp.c
> 
> diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig
> index 36512a8995..4d7a115569 100644
> --- a/arch/riscv/Kconfig
> +++ b/arch/riscv/Kconfig
> @@ -120,4 +120,23 @@ config RISCV_RDTIME
>   config SYS_MALLOC_F_LEN
>   	default 0x1000
>   
> +config SMP
> +	bool "Symmetric Multi-Processing"
> +	help
> +	  This enables support for systems with more than one CPU. If
> +	  you say N here, U-Boot will run on single and multiprocessor
> +	  machines, but will use only one CPU of a multiprocessor
> +	  machine. If you say Y here, U-Boot will run on many, but not
> +	  all, single processor machines.
> +
> +config NR_CPUS
> +	int "Maximum number of CPUs (2-32)"
> +	range 2 32
> +	depends on SMP
> +	default 8
> +	help
> +	  On multiprocessor machines, U-Boot sets up a stack for each CPU.
> +	  Stack memory is pre-allocated. U-Boot must therefore know the
> +	  maximum number of CPUs that may be present.
> +
>   endmenu
> diff --git a/arch/riscv/include/asm/global_data.h b/arch/riscv/include/asm/global_data.h
> index a3a342c6e1..80e3165e39 100644
> --- a/arch/riscv/include/asm/global_data.h
> +++ b/arch/riscv/include/asm/global_data.h
> @@ -10,12 +10,18 @@
>   #ifndef	__ASM_GBL_DATA_H
>   #define __ASM_GBL_DATA_H
>   
> +#include <asm/smp.h>
> +
>   /* Architecture-specific global data */
>   struct arch_global_data {
>   	long boot_hart;		/* boot hart id */
>   #ifdef CONFIG_SIFIVE_CLINT
>   	void __iomem *clint;	/* clint base address */
>   #endif
> +#ifdef CONFIG_SMP
> +	struct ipi_data ipi[CONFIG_NR_CPUS];
> +#endif
> +	ulong available_harts;
>   };
>   
>   #include <asm-generic/global_data.h>
> diff --git a/arch/riscv/include/asm/smp.h b/arch/riscv/include/asm/smp.h
> new file mode 100644
> index 0000000000..bc863fdbaf
> --- /dev/null
> +++ b/arch/riscv/include/asm/smp.h
> @@ -0,0 +1,53 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +/*
> + * Copyright (C) 2019 Fraunhofer AISEC,
> + * Lukas Auer <lukas.auer@aisec.fraunhofer.de>
> + */
> +
> +#ifndef _ASM_RISCV_SMP_H
> +#define _ASM_RISCV_SMP_H
> +
> +/**
> + * struct ipi_data - Inter-processor interrupt (IPI) data structure
> + *
> + * IPIs are used for SMP support to communicate to other harts what function to
> + * call. Functions are in the form
> + * void (*addr)(ulong hart, ulong arg0, ulong arg1).
> + *
> + * The function address and the two arguments, arg0 and arg1, are stored in the
> + * IPI data structure. The hart ID is inserted by the hart handling the IPI and
> + * calling the function.
> + *
> + * @addr: Address of function
> + * @arg0: First argument of function
> + * @arg1: Second argument of function
> + */
> +struct ipi_data {
> +	ulong addr;
> +	ulong arg0;
> +	ulong arg1;
> +};
> +
> +/**
> + * handle_ipi() - interrupt handler for software interrupts
> + *
> + * The IPI interrupt handler must be called to handle software interrupts. It
> + * calls the function specified in the hart's IPI data structure.
> + *
> + * @hart: Hart ID of the current hart
> + */
> +void handle_ipi(ulong hart);
> +
> +/**
> + * smp_call_function() - Call a function on all other harts
> + *
> + * Send IPIs with the specified function call to all harts.
> + *
> + * @addr: Address of function
> + * @arg0: First argument of function
> + * @arg1: Second argument of function
> + * @return 0 if OK, -ve on error
> + */
> +int smp_call_function(ulong addr, ulong arg0, ulong arg1);
> +
> +#endif
> diff --git a/arch/riscv/lib/Makefile b/arch/riscv/lib/Makefile
> index edfa61690c..19370f9749 100644
> --- a/arch/riscv/lib/Makefile
> +++ b/arch/riscv/lib/Makefile
> @@ -14,6 +14,7 @@ obj-$(CONFIG_SIFIVE_CLINT) += sifive_clint.o
>   obj-y	+= interrupts.o
>   obj-y	+= reset.o
>   obj-y   += setjmp.o
> +obj-$(CONFIG_SMP) += smp.o
>   
>   # For building EFI apps
>   CFLAGS_$(EFI_CRT0) := $(CFLAGS_EFI)
> diff --git a/arch/riscv/lib/smp.c b/arch/riscv/lib/smp.c
> new file mode 100644
> index 0000000000..edef8a687d
> --- /dev/null
> +++ b/arch/riscv/lib/smp.c
> @@ -0,0 +1,116 @@
> +// SPDX-License-Identifier: GPL-2.0+
> +/*
> + * Copyright (C) 2019 Fraunhofer AISEC,
> + * Lukas Auer <lukas.auer@aisec.fraunhofer.de>
> + */
> +
> +#include <common.h>
> +#include <dm.h>
> +#include <asm/barrier.h>
> +#include <asm/smp.h>
> +
> +DECLARE_GLOBAL_DATA_PTR;
> +
> +/**
> + * riscv_send_ipi() - Send inter-processor interrupt (IPI)
> + *
> + * Platform code must provide this function.
> + *
> + * @hart: Hart ID of receiving hart
> + * @return 0 if OK, -ve on error
> + */
> +extern int riscv_send_ipi(int hart);
> +
> +/**
> + * riscv_clear_ipi() - Clear inter-processor interrupt (IPI)
> + *
> + * Platform code must provide this function.
> + *
> + * @hart: Hart ID of hart to be cleared
> + * @return 0 if OK, -ve on error
> + */
> +extern int riscv_clear_ipi(int hart);
> +
> +static int send_ipi_many(struct ipi_data *ipi)
> +{
> +	ofnode node, cpus;
> +	u32 reg;
> +	int ret;
> +
> +	cpus = ofnode_path("/cpus");
> +	if (!ofnode_valid(cpus)) {
> +		pr_err("Can't find cpus node!\n");
> +		return -EINVAL;
> +	}
> +
> +	ofnode_for_each_subnode(node, cpus) {
> +		/* skip if hart is marked as not available in the device tree */
> +		if (!ofnode_is_available(node))
> +			continue;
> +
> +		/* read hart ID of CPU */
> +		ret = ofnode_read_u32(node, "reg", &reg);
> +		if (ret)
> +			continue;
> +
> +		/* skip if it is the hart we are running on */
> +		if (reg == gd->arch.boot_hart)
> +			continue;
> +
> +		if (reg >= CONFIG_NR_CPUS) {
> +			pr_err("Hart ID %d is out of range, increase CONFIG_NR_CPUS\n",
> +			       reg);
> +			continue
> +		}
> +
> +		/* skip if hart is not available */
> +		if (!(gd->arch.available_harts & (1 << reg)))
> +			continue;
> +
> +		gd->arch.ipi[reg].addr = ipi->addr;
> +		gd->arch.ipi[reg].arg0 = ipi->arg0;
> +		gd->arch.ipi[reg].arg1 = ipi->arg1;
> +
> +		ret = riscv_send_ipi(reg);
> +		if (ret)
> +			return ret;
> +	}
> +
> +	return 0;
> +}
> +
> +void handle_ipi(ulong hart)
> +{
> +	int ret;
> +	void (*smp_function)(ulong hart, ulong arg0, ulong arg1);
> +
> +	if (hart >= CONFIG_NR_CPUS)
> +		return;
> +

A warning here may help.

> +	ret = riscv_clear_ipi(hart);
> +	if (ret) {
> +		pr_err("Cannot clear IPI\n");
> +		return;
> +	}
> +

Currently, sbi_clear_ipi() doesn't return anything and riscv_clear_ipi() 
is always returning 0. I guess you kept the check here so that we don't 
have make changes in future. But IMHO, we should add it if SBI spec is 
changed so that sbi_clear_ipi can return errors.

Regards,
Atish
> +	__smp_mb();
> +
> +	smp_function = (void (*)(ulong, ulong, ulong))gd->arch.ipi[hart].addr;
> +	invalidate_icache_all();
> +
> +	smp_function(hart, gd->arch.ipi[hart].arg0, gd->arch.ipi[hart].arg1);
> +}
> +
> +int smp_call_function(ulong addr, ulong arg0, ulong arg1)
> +{
> +	int ret = 0;
> +	struct ipi_data ipi;
> +
> +	ipi.addr = addr;
> +	ipi.arg0 = arg0;
> +	ipi.arg1 = arg1;
> +
> +	ret = send_ipi_many(&ipi);
> +
> +	return ret;
> +}
> 

[U-Boot] [PATCH v2 2/9] riscv: import the supervisor binary interface header file

[Atish Patra]

On 3/5/19 2:54 PM, Lukas Auer wrote:
> Import the supervisor binary interface (SBI) header file from Linux
> (arch/riscv/include/asm/sbi.h). The last change to it was in commit
> 6d60b6ee0c97 ("RISC-V: Device, timer, IRQs, and the SBI").
> 
> Signed-off-by: Lukas Auer <lukas.auer@aisec.fraunhofer.de>
> Reviewed-by: Anup Patel <anup.patel@wdc.com>
> Reviewed-by: Bin Meng <bmeng.cn@gmail.com>
> ---
> 
> Changes in v2: None
> 
>   arch/riscv/include/asm/sbi.h | 94 ++++++++++++++++++++++++++++++++++++
>   1 file changed, 94 insertions(+)
>   create mode 100644 arch/riscv/include/asm/sbi.h
> 
> diff --git a/arch/riscv/include/asm/sbi.h b/arch/riscv/include/asm/sbi.h
> new file mode 100644
> index 0000000000..ced57defdd
> --- /dev/null
> +++ b/arch/riscv/include/asm/sbi.h
> @@ -0,0 +1,94 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +/*
> + * Copyright (C) 2015 Regents of the University of California
> + *
> + * Taken from Linux arch/riscv/include/asm/sbi.h
> + */
> +
> +#ifndef _ASM_RISCV_SBI_H
> +#define _ASM_RISCV_SBI_H
> +
> +#include <linux/types.h>
> +
> +#define SBI_SET_TIMER 0
> +#define SBI_CONSOLE_PUTCHAR 1
> +#define SBI_CONSOLE_GETCHAR 2
> +#define SBI_CLEAR_IPI 3
> +#define SBI_SEND_IPI 4
> +#define SBI_REMOTE_FENCE_I 5
> +#define SBI_REMOTE_SFENCE_VMA 6
> +#define SBI_REMOTE_SFENCE_VMA_ASID 7
> +#define SBI_SHUTDOWN 8
> +
> +#define SBI_CALL(which, arg0, arg1, arg2) ({			\
> +	register uintptr_t a0 asm ("a0") = (uintptr_t)(arg0);	\
> +	register uintptr_t a1 asm ("a1") = (uintptr_t)(arg1);	\
> +	register uintptr_t a2 asm ("a2") = (uintptr_t)(arg2);	\
> +	register uintptr_t a7 asm ("a7") = (uintptr_t)(which);	\
> +	asm volatile ("ecall"					\
> +		      : "+r" (a0)				\
> +		      : "r" (a1), "r" (a2), "r" (a7)		\
> +		      : "memory");				\
> +	a0;							\
> +})
> +
> +/* Lazy implementations until SBI is finalized */
> +#define SBI_CALL_0(which) SBI_CALL(which, 0, 0, 0)
> +#define SBI_CALL_1(which, arg0) SBI_CALL(which, arg0, 0, 0)
> +#define SBI_CALL_2(which, arg0, arg1) SBI_CALL(which, arg0, arg1, 0)
> +
> +static inline void sbi_console_putchar(int ch)
> +{
> +	SBI_CALL_1(SBI_CONSOLE_PUTCHAR, ch);
> +}
> +
> +static inline int sbi_console_getchar(void)
> +{
> +	return SBI_CALL_0(SBI_CONSOLE_GETCHAR);
> +}
> +
> +static inline void sbi_set_timer(uint64_t stime_value)
> +{
> +#if __riscv_xlen == 32
> +	SBI_CALL_2(SBI_SET_TIMER, stime_value, stime_value >> 32);
> +#else
> +	SBI_CALL_1(SBI_SET_TIMER, stime_value);
> +#endif
> +}
> +
> +static inline void sbi_shutdown(void)
> +{
> +	SBI_CALL_0(SBI_SHUTDOWN);
> +}
> +
> +static inline void sbi_clear_ipi(void)
> +{
> +	SBI_CALL_0(SBI_CLEAR_IPI);
> +}
> +
> +static inline void sbi_send_ipi(const unsigned long *hart_mask)
> +{
> +	SBI_CALL_1(SBI_SEND_IPI, hart_mask);
> +}
> +
> +static inline void sbi_remote_fence_i(const unsigned long *hart_mask)
> +{
> +	SBI_CALL_1(SBI_REMOTE_FENCE_I, hart_mask);
> +}
> +
> +static inline void sbi_remote_sfence_vma(const unsigned long *hart_mask,
> +					 unsigned long start,
> +					 unsigned long size)
> +{
> +	SBI_CALL_1(SBI_REMOTE_SFENCE_VMA, hart_mask);
> +}
> +
> +static inline void sbi_remote_sfence_vma_asid(const unsigned long *hart_mask,
> +					      unsigned long start,
> +					      unsigned long size,
> +					      unsigned long asid)
> +{
> +	SBI_CALL_1(SBI_REMOTE_SFENCE_VMA_ASID, hart_mask);
> +}
> +
> +#endif
> 

Reviewed-by: Atish Patra <atish.patra@wdc.com>

Regards,
Atish

[U-Boot] [PATCH v2 3/9] riscv: implement IPI platform functions using SBI

[Atish Patra]

On 3/5/19 2:54 PM, Lukas Auer wrote:
> The supervisor binary interface (SBI) provides the necessary functions
> to implement the platform IPI functions riscv_send_ipi() and
> riscv_clear_ipi(). Use it to implement them.
> 
> This adds support for inter-processor interrupts (IPIs) on RISC-V CPUs
> running in supervisor mode. Support for machine mode is already
> available for CPUs that include the SiFive CLINT.
> 
> Signed-off-by: Lukas Auer <lukas.auer@aisec.fraunhofer.de>
> Reviewed-by: Anup Patel <anup.patel@wdc.com>
> Reviewed-by: Bin Meng <bmeng.cn@gmail.com>
> ---
> 
> Changes in v2: None
> 
>   arch/riscv/Kconfig       |  5 +++++
>   arch/riscv/lib/Makefile  |  1 +
>   arch/riscv/lib/sbi_ipi.c | 25 +++++++++++++++++++++++++
>   3 files changed, 31 insertions(+)
>   create mode 100644 arch/riscv/lib/sbi_ipi.c
> 
> diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig
> index 4d7a115569..9da609b33b 100644
> --- a/arch/riscv/Kconfig
> +++ b/arch/riscv/Kconfig
> @@ -139,4 +139,9 @@ config NR_CPUS
>   	  Stack memory is pre-allocated. U-Boot must therefore know the
>   	  maximum number of CPUs that may be present.
>   
> +config SBI_IPI
> +	bool
> +	default y if RISCV_SMODE
> +	depends on SMP
> +
>   endmenu
> diff --git a/arch/riscv/lib/Makefile b/arch/riscv/lib/Makefile
> index 19370f9749..35dbf643e4 100644
> --- a/arch/riscv/lib/Makefile
> +++ b/arch/riscv/lib/Makefile
> @@ -13,6 +13,7 @@ obj-$(CONFIG_RISCV_RDTIME) += rdtime.o
>   obj-$(CONFIG_SIFIVE_CLINT) += sifive_clint.o
>   obj-y	+= interrupts.o
>   obj-y	+= reset.o
> +obj-$(CONFIG_SBI_IPI) += sbi_ipi.o
>   obj-y   += setjmp.o
>   obj-$(CONFIG_SMP) += smp.o
>   
> diff --git a/arch/riscv/lib/sbi_ipi.c b/arch/riscv/lib/sbi_ipi.c
> new file mode 100644
> index 0000000000..170346da68
> --- /dev/null
> +++ b/arch/riscv/lib/sbi_ipi.c
> @@ -0,0 +1,25 @@
> +// SPDX-License-Identifier: GPL-2.0+
> +/*
> + * Copyright (C) 2019 Fraunhofer AISEC,
> + * Lukas Auer <lukas.auer@aisec.fraunhofer.de>
> + */
> +
> +#include <common.h>
> +#include <asm/sbi.h>
> +
> +int riscv_send_ipi(int hart)
> +{
> +	ulong mask;
> +
> +	mask = 1UL << hart;
> +	sbi_send_ipi(&mask);
> +
> +	return 0;
> +}
> +
> +int riscv_clear_ipi(int hart)
> +{
> +	sbi_clear_ipi();
> +
> +	return 0;
> +}
> 

LGTM.

Reviewed-by: Atish Patra <atish.patra@wdc.com>

Regards,
Atish

[U-Boot] [PATCH v2 7/9] riscv: do not rely on hart ID passed by previous boot stage

[Atish Patra]

On 3/5/19 2:54 PM, Lukas Auer wrote:
> RISC-V U-Boot expects the hart ID to be passed to it via register a0 by
> the previous boot stage. Machine mode firmware such as BBL and OpenSBI
> do this when starting their payload (U-Boot) in supervisor mode. If
> U-Boot is running in machine mode, this task must be handled by the boot
> ROM. Explicitly populate register a0 with the hart ID from the mhartid
> CSR to avoid possible problems on RISC-V processors with a boot ROM that
> does not handle this task.
> 
> Suggested-by: Rick Chen <rick@andestech.com>
> Signed-off-by: Lukas Auer <lukas.auer@aisec.fraunhofer.de>
> ---
> 
> Changes in v2:
> - New patch to populate register a0 with the hart ID from the mhartid
> CSR in machine-mode
> 
>   arch/riscv/cpu/start.S | 4 ++++
>   1 file changed, 4 insertions(+)
> 
> diff --git a/arch/riscv/cpu/start.S b/arch/riscv/cpu/start.S
> index 79b753847c..d4daa6e0bf 100644
> --- a/arch/riscv/cpu/start.S
> +++ b/arch/riscv/cpu/start.S
> @@ -36,6 +36,10 @@
>   .section .text
>   .globl _start
>   _start:
> +#ifdef CONFIG_RISCV_MMODE
> +	csrr	a0, mhartid
> +#endif
> +
>   	/* save hart id and dtb pointer */
>   	mv	s0, a0
>   	mv	s1, a1
> 

Reviewed-by: Atish Patra <atish.patra@wdc.com>

Regards,
Atish

[U-Boot] [PATCH v2 0/9] SMP support for RISC-V

[Anup Patel]

> -----Original Message-----
> From: Atish Patra <atish.patra@wdc.com>
> Sent: Thursday, March 7, 2019 5:20 AM
> To: Anup Patel <anup@brainfault.org>; Auer, Lukas
> <lukas.auer@aisec.fraunhofer.de>
> Cc: schwab at suse.de; paul.walmsley at sifive.com; agraf at suse.de; u-
> boot at lists.denx.de; baruch at tkos.co.il; daniel.schwierzeck at gmail.com;
> bmeng.cn at gmail.com; rick at andestech.com; sr at denx.de;
> palmer at sifive.com; Anup Patel <Anup.Patel@wdc.com>
> Subject: Re: [PATCH v2 0/9] SMP support for RISC-V
> 
> On 3/6/19 4:32 AM, Anup Patel wrote:
> > On Wed, Mar 6, 2019 at 5:45 PM Auer, Lukas
> > <lukas.auer@aisec.fraunhofer.de> wrote:
> >>
> >> On Wed, 2019-03-06 at 13:01 +0100, Andreas Schwab wrote:
> >>> On Mär 06 2019, Anup Patel <anup@brainfault.org> wrote:
> >>>
> >>>> On Wed, Mar 6, 2019 at 5:17 PM Andreas Schwab <schwab@suse.de>
> >>>> wrote:
> >>>>> On Mär 06 2019, Anup Patel <Anup.Patel@wdc.com> wrote:
> >>>>>
> >>>>>>> -----Original Message-----
> >>>>>>> From: Andreas Schwab <schwab@suse.de>
> >>>>>>> Sent: Wednesday, March 6, 2019 4:27 PM
> >>>>>>> To: Anup Patel <Anup.Patel@wdc.com>
> >>>>>>> Cc: Auer, Lukas <lukas.auer@aisec.fraunhofer.de>;
> >>>>>>> u-boot at lists.denx.de; paul.walmsley at sifive.com; agraf at suse.de;
> >>>>>>> anup at brainfault.org; baruch at tkos.co.il;
> >>>>>>> daniel.schwierzeck at gmail.com; bmeng.cn at gmail.com;
> >>>>>>> rick at andestech.com; sr at denx.de; palmer at sifive.com; Atish Patra
> >>>>>>> <Atish.Patra@wdc.com>
> >>>>>>> Subject: Re: [PATCH v2 0/9] SMP support for RISC-V
> >>>>>>>
> >>>>>>> Apparently sometimes u-boot tries to boot the kernel on heart
> >>>>>>> 0 (the E51
> >>>>>>> core), which will then fail to start userspace, since that
> >>>>>>> cannot cope with the missing fpu.
> >>>>>>
> >>>>>> That's not possible
> >>>>>
> >>>>> Yes, it is.
> >>>>>
> >>>>>
> >>>>> OpenSBI v0.3 (Mar  6 2019 10:55:01)
> >>>>>     ____                    _____ ____ _____
> >>>>>    / __ \                  / ____|  _ \_   _|
> >>>>>   | |  | |_ __   ___ _ __ | (___ | |_) || |
> >>>>>   | |  | | '_ \ / _ \ '_ \ \___ \|  _ < | |
> >>>>>   | |__| | |_) |  __/ | | |____) | |_) || |_
> >>>>>    \____/| .__/ \___|_| |_|_____/|____/_____|
> >>>>>          | |
> >>>>>          |_|
> >>>>>
> >>>>> Platform Name          : SiFive Freedom U540
> >>>>> Platform HART Features : RV64ACDFIMSU
> >>>>> Platform Max HARTs     : 5
> >>>>> Current Hart           : 2
> >>>>> Firmware Base          : 0x80000000
> >>>>> Firmware Size          : 88 KB
> >>>>> Runtime SBI Version    : 0.1
> >>>>>
> >>>>> PMP0: 0x0000000080000000-0x000000008001ffff (A)
> >>>>> PMP1: 0x0000000000000000-0x0000007fffffffff (A,R,W,X)
> >>>>>
> >>>>>
> >>>>> U-Boot 2019.04-rc3-00010-g3ea5582c09 (Mar 06 2019 - 10:06:10
> >>>>> +0100)
> >>>>>
> >>>>> CPU:   rv64imac
> >>>>> Model: sifive,hifive-unleashed-a00
> >>>>> DRAM:  8 GiB
> >>>>
> >>>> How does this prove that U-Boot is booting on HART 0?
> >>>
> >>> See the CPU isa.
> >>>
> >>
> >> Interesting.. U-Boot assumes that it can run on any core it is
> >> started on. In this case, OpenSBI must have booted its payload on hart 0.
> >
> > This is certainly not reproducible on cold-boot at my end but this
> > does mean OpenSBI has booted HART0 and let it jump to U-Boot.
> >
> > Now OpenSBI (by default) on SiFive FU540 does not allow HART0 to go
> > forward due to lack of S-mode. I think this is definitely the
> > warm-boot issue where OpenSBI sees corrupted memory contents.
> >
> 
> I am able to test both warm-boot and cold-boot several times(>10) without
> any issue with following pending PR in openSBI.
> 
> https://github.com/riscv/opensbi/pull/84

Thanks Atish, your changes have been merged in OpenSBI.

> 
> @Andreas @Anup: Can you please apply the above PR on top of master and
> verify at your end as well?
> 
> All the harts booted in Linux every time as well.

I tried warm-boot 10 times from U-Boot prompt and I did not see any issue
at all.

I also tried warm-boot 10 times from Linux prompt and I did not see any
Issue at all.

In both above cases, it came back to U-Boot prompt after warm-boot.

Also, tried cold-boot couple of times. I works perfectly fine as well.

Like I mentioned, there is no functional issue with this series. The
warm-boot issues were fixed in OpenSBI.

@Andreas, please try at your end.

Regards,
Anup

[U-Boot] [PATCH v2 0/9] SMP support for RISC-V

[Andreas Schwab]

On Mär 07 2019, Anup Patel <Anup.Patel@wdc.com> wrote:

> Like I mentioned, there is no functional issue with this series. The
> warm-boot issues were fixed in OpenSBI.
>
> @Andreas, please try at your end.

As long as issue#65 isn't fixed opensbi is mostly a no-go for me.  At
least it gives me more reasons to press the reset button. :-)

Andreas.

-- 
Andreas Schwab, SUSE Labs, schwab at suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

[U-Boot] [PATCH v2 0/9] SMP support for RISC-V

[Anup Patel]

> -----Original Message-----
> From: Andreas Schwab <schwab@suse.de>
> Sent: Thursday, March 7, 2019 2:50 PM
> To: Anup Patel <Anup.Patel@wdc.com>
> Cc: Atish Patra <Atish.Patra@wdc.com>; Anup Patel <anup@brainfault.org>;
> Auer, Lukas <lukas.auer@aisec.fraunhofer.de>; paul.walmsley at sifive.com;
> agraf at suse.de; u-boot at lists.denx.de; baruch at tkos.co.il;
> daniel.schwierzeck at gmail.com; bmeng.cn at gmail.com;
> rick at andestech.com; sr at denx.de; palmer at sifive.com
> Subject: Re: [PATCH v2 0/9] SMP support for RISC-V
> 
> On Mär 07 2019, Anup Patel <Anup.Patel@wdc.com> wrote:
> 
> > Like I mentioned, there is no functional issue with this series. The
> > warm-boot issues were fixed in OpenSBI.
> >
> > @Andreas, please try at your end.
> 
> As long as issue#65 isn't fixed opensbi is mostly a no-go for me.  At least it
> gives me more reasons to press the reset button. :-)

The reset button works fine for me an Atish. I am sure it works fine for lot of
other folks too.

BTW, as-per discussion with SiFive folks the reset button on Unleashed
Board is not much tested and it can misbehave on certain boards. It is quite
possible that you might have a "flaky" board.

Regards,
Anup

[U-Boot] [PATCH v2 1/9] riscv: add infrastructure for calling functions on other harts

[Bin Meng]

On Wed, Mar 6, 2019 at 6:54 AM Lukas Auer
<lukas.auer@aisec.fraunhofer.de> wrote:
>
> Harts on RISC-V boot independently, U-Boot is responsible for managing
> them. Functions are called on other harts with smp_call_function(),
> which sends inter-processor interrupts (IPIs) to all other available
> harts. Available harts are those marked as available in the device tree
> and present in the available_harts mask stored in global data. The
> available_harts mask is used to register all harts that have entered
> U-Boot. Functions are specified with their address and two function
> arguments (argument 2 and 3). The first function argument is always the
> hart ID of the hart calling the function. On the other harts, the IPI
> interrupt handler handle_ipi() must be called on software interrupts to
> handle the request and call the specified function.
>
> Functions are stored in the ipi_data data structure. Every hart has its
> own data structure in global data. While this is not required at the
> moment (all harts are expected to boot Linux), this does allow future
> expansion, where other harts may be used for monitoring or other tasks.
>
> Signed-off-by: Lukas Auer <lukas.auer@aisec.fraunhofer.de>
> ---
>
> Changes in v2:
> - Remove unneeded quotes from NR_CPUS Kconfig entry
> - Move memory barrier from send_ipi_many() to handle_ipi()
> - Add check in send_ipi_many so that IPIs are only sent to available
> harts as indicated by the available_harts mask
>
>  arch/riscv/Kconfig                   |  19 +++++
>  arch/riscv/include/asm/global_data.h |   6 ++
>  arch/riscv/include/asm/smp.h         |  53 ++++++++++++
>  arch/riscv/lib/Makefile              |   1 +
>  arch/riscv/lib/smp.c                 | 116 +++++++++++++++++++++++++++
>  5 files changed, 195 insertions(+)
>  create mode 100644 arch/riscv/include/asm/smp.h
>  create mode 100644 arch/riscv/lib/smp.c
>

Reviewed-by: Bin Meng <bmeng.cn@gmail.com>
Tested-by: Bin Meng <bmeng.cn@gmail.com>

[U-Boot] [PATCH v2 3/9] riscv: implement IPI platform functions using SBI

[Bin Meng]

On Wed, Mar 6, 2019 at 6:54 AM Lukas Auer
<lukas.auer@aisec.fraunhofer.de> wrote:
>
> The supervisor binary interface (SBI) provides the necessary functions
> to implement the platform IPI functions riscv_send_ipi() and
> riscv_clear_ipi(). Use it to implement them.
>
> This adds support for inter-processor interrupts (IPIs) on RISC-V CPUs
> running in supervisor mode. Support for machine mode is already
> available for CPUs that include the SiFive CLINT.
>
> Signed-off-by: Lukas Auer <lukas.auer@aisec.fraunhofer.de>
> Reviewed-by: Anup Patel <anup.patel@wdc.com>
> Reviewed-by: Bin Meng <bmeng.cn@gmail.com>
> ---
>
> Changes in v2: None
>
>  arch/riscv/Kconfig       |  5 +++++
>  arch/riscv/lib/Makefile  |  1 +
>  arch/riscv/lib/sbi_ipi.c | 25 +++++++++++++++++++++++++
>  3 files changed, 31 insertions(+)
>  create mode 100644 arch/riscv/lib/sbi_ipi.c
>

Tested-by: Bin Meng <bmeng.cn@gmail.com>

[U-Boot] [PATCH v2 5/9] riscv: add support for multi-hart systems

[Bin Meng]

On Wed, Mar 6, 2019 at 6:54 AM Lukas Auer
<lukas.auer@aisec.fraunhofer.de> wrote:
>
> On RISC-V, all harts boot independently. To be able to run on a
> multi-hart system, U-Boot must be extended with the functionality to
> manage all harts in the system. All harts entering U-Boot are registered
> in the available_harts mask stored in global data. A hart lottery system
> as used in the Linux kernel selects the hart U-Boot runs on. All other
> harts are halted. U-Boot can delegate functions to them using
> smp_call_function().
>
> Every hart has a valid pointer to the global data structure and a 8KiB
> stack by default. The stack size is set with CONFIG_STACK_SIZE_SHIFT.
>
> Signed-off-by: Lukas Auer <lukas.auer@aisec.fraunhofer.de>
> ---
>
> Changes in v2:
> - Implement hart lottery to pick main hart to run U-Boot
> - Remove CONFIG_MAIN_HART as it is not required anymore
> - Register available harts in the available_harts mask
>
>  arch/riscv/Kconfig           |   4 ++
>  arch/riscv/cpu/cpu.c         |   9 ++-
>  arch/riscv/cpu/start.S       | 134 ++++++++++++++++++++++++++++++++++-
>  arch/riscv/include/asm/csr.h |   1 +
>  arch/riscv/lib/asm-offsets.c |   1 +
>  5 files changed, 147 insertions(+), 2 deletions(-)
>

Looks quite good!

Reviewed-by: Bin Meng <bmeng.cn@gmail.com>
Tested-by: Bin Meng <bmeng.cn@gmail.com>

[U-Boot] [PATCH v2 6/9] riscv: boot images passed to bootm on all harts

[Bin Meng]

On Wed, Mar 6, 2019 at 6:54 AM Lukas Auer
<lukas.auer@aisec.fraunhofer.de> wrote:
>
> Signed-off-by: Lukas Auer <lukas.auer@aisec.fraunhofer.de>
> Reviewed-by: Anup Patel <anup.patel@wdc.com>
> Reviewed-by: Bin Meng <bmeng.cn@gmail.com>
> ---
>
> Changes in v2: None
>
>  arch/riscv/lib/bootm.c | 13 ++++++++++++-
>  1 file changed, 12 insertions(+), 1 deletion(-)
>

Tested-by: Bin Meng <bmeng.cn@gmail.com>

[U-Boot] [PATCH v2 7/9] riscv: do not rely on hart ID passed by previous boot stage

[Bin Meng]

On Wed, Mar 6, 2019 at 6:54 AM Lukas Auer
<lukas.auer@aisec.fraunhofer.de> wrote:
>
> RISC-V U-Boot expects the hart ID to be passed to it via register a0 by
> the previous boot stage. Machine mode firmware such as BBL and OpenSBI
> do this when starting their payload (U-Boot) in supervisor mode. If
> U-Boot is running in machine mode, this task must be handled by the boot
> ROM. Explicitly populate register a0 with the hart ID from the mhartid
> CSR to avoid possible problems on RISC-V processors with a boot ROM that
> does not handle this task.
>
> Suggested-by: Rick Chen <rick@andestech.com>
> Signed-off-by: Lukas Auer <lukas.auer@aisec.fraunhofer.de>
> ---
>
> Changes in v2:
> - New patch to populate register a0 with the hart ID from the mhartid
> CSR in machine-mode
>
>  arch/riscv/cpu/start.S | 4 ++++
>  1 file changed, 4 insertions(+)
>

Reviewed-by: Bin Meng <bmeng.cn@gmail.com>
Tested-by: Bin Meng <bmeng.cn@gmail.com>

[U-Boot] [PATCH v2 8/9] riscv: fu540: enable SMP

[Bin Meng]

On Wed, Mar 6, 2019 at 6:54 AM Lukas Auer
<lukas.auer@aisec.fraunhofer.de> wrote:
>
> Signed-off-by: Lukas Auer <lukas.auer@aisec.fraunhofer.de>
> Reviewed-by: Anup Patel <anup.patel@wdc.com>
> ---
>
> Changes in v2:
> - New patch to enable SMP on the SiFive FU540, which was previously sent
> independently
>
>  board/sifive/fu540/Kconfig | 1 +
>  1 file changed, 1 insertion(+)
>

Reviewed-by: Bin Meng <bmeng.cn@gmail.com>

[U-Boot] [PATCH v2 9/9] riscv: qemu: enable SMP

[Bin Meng]

On Wed, Mar 6, 2019 at 6:54 AM Lukas Auer
<lukas.auer@aisec.fraunhofer.de> wrote:
>
> Signed-off-by: Lukas Auer <lukas.auer@aisec.fraunhofer.de>
> Reviewed-by: Anup Patel <anup.patel@wdc.com>
> Reviewed-by: Bin Meng <bmeng.cn@gmail.com>
> ---
>
> Changes in v2: None
>
>  board/emulation/qemu-riscv/Kconfig | 1 +
>  1 file changed, 1 insertion(+)
>

Tested-by: Bin Meng <bmeng.cn@gmail.com>

[U-Boot] [PATCH v2 5/9] riscv: add support for multi-hart systems

[Auer, Lukas]

On Sun, 2019-03-10 at 21:01 +0800, Bin Meng wrote:
> On Wed, Mar 6, 2019 at 6:54 AM Lukas Auer
> <lukas.auer@aisec.fraunhofer.de> wrote:
> > On RISC-V, all harts boot independently. To be able to run on a
> > multi-hart system, U-Boot must be extended with the functionality
> > to
> > manage all harts in the system. All harts entering U-Boot are
> > registered
> > in the available_harts mask stored in global data. A hart lottery
> > system
> > as used in the Linux kernel selects the hart U-Boot runs on. All
> > other
> > harts are halted. U-Boot can delegate functions to them using
> > smp_call_function().
> > 
> > Every hart has a valid pointer to the global data structure and a
> > 8KiB
> > stack by default. The stack size is set with
> > CONFIG_STACK_SIZE_SHIFT.
> > 
> > Signed-off-by: Lukas Auer <lukas.auer@aisec.fraunhofer.de>
> > ---
> > 
> > Changes in v2:
> > - Implement hart lottery to pick main hart to run U-Boot
> > - Remove CONFIG_MAIN_HART as it is not required anymore
> > - Register available harts in the available_harts mask
> > 
> >  arch/riscv/Kconfig           |   4 ++
> >  arch/riscv/cpu/cpu.c         |   9 ++-
> >  arch/riscv/cpu/start.S       | 134
> > ++++++++++++++++++++++++++++++++++-
> >  arch/riscv/include/asm/csr.h |   1 +
> >  arch/riscv/lib/asm-offsets.c |   1 +
> >  5 files changed, 147 insertions(+), 2 deletions(-)
> > 
> 
> Looks quite good!
> 

Thank you and thanks for testing and reviewing the series!

Lukas

[U-Boot] [PATCH v2 1/9] riscv: add infrastructure for calling functions on other harts

[Auer, Lukas]

On Wed, 2019-03-06 at 19:20 -0800, Atish Patra wrote:
> On 3/5/19 2:54 PM, Lukas Auer wrote:
> > Harts on RISC-V boot independently, U-Boot is responsible for
> > managing
> > them. Functions are called on other harts with smp_call_function(),
> > which sends inter-processor interrupts (IPIs) to all other
> > available
> > harts. Available harts are those marked as available in the device
> > tree
> > and present in the available_harts mask stored in global data. The
> > available_harts mask is used to register all harts that have
> > entered
> > U-Boot. Functions are specified with their address and two function
> > arguments (argument 2 and 3). The first function argument is always
> > the
> > hart ID of the hart calling the function. On the other harts, the
> > IPI
> > interrupt handler handle_ipi() must be called on software
> > interrupts to
> > handle the request and call the specified function.
> > 
> > Functions are stored in the ipi_data data structure. Every hart has
> > its
> > own data structure in global data. While this is not required at
> > the
> > moment (all harts are expected to boot Linux), this does allow
> > future
> > expansion, where other harts may be used for monitoring or other
> > tasks.
> > 
> > Signed-off-by: Lukas Auer <lukas.auer@aisec.fraunhofer.de>
> > ---
> > 
> > Changes in v2:
> > - Remove unneeded quotes from NR_CPUS Kconfig entry
> > - Move memory barrier from send_ipi_many() to handle_ipi()
> > - Add check in send_ipi_many so that IPIs are only sent to
> > available
> > harts as indicated by the available_harts mask
> > 
> >   arch/riscv/Kconfig                   |  19 +++++
> >   arch/riscv/include/asm/global_data.h |   6 ++
> >   arch/riscv/include/asm/smp.h         |  53 ++++++++++++
> >   arch/riscv/lib/Makefile              |   1 +
> >   arch/riscv/lib/smp.c                 | 116
> > +++++++++++++++++++++++++++
> >   5 files changed, 195 insertions(+)
> >   create mode 100644 arch/riscv/include/asm/smp.h
> >   create mode 100644 arch/riscv/lib/smp.c
> > 
> > diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig
> > index 36512a8995..4d7a115569 100644
> > --- a/arch/riscv/Kconfig
> > +++ b/arch/riscv/Kconfig
> > @@ -120,4 +120,23 @@ config RISCV_RDTIME
> >   config SYS_MALLOC_F_LEN
> >   	default 0x1000
> >   
> > +config SMP
> > +	bool "Symmetric Multi-Processing"
> > +	help
> > +	  This enables support for systems with more than one CPU. If
> > +	  you say N here, U-Boot will run on single and multiprocessor
> > +	  machines, but will use only one CPU of a multiprocessor
> > +	  machine. If you say Y here, U-Boot will run on many, but not
> > +	  all, single processor machines.
> > +
> > +config NR_CPUS
> > +	int "Maximum number of CPUs (2-32)"
> > +	range 2 32
> > +	depends on SMP
> > +	default 8
> > +	help
> > +	  On multiprocessor machines, U-Boot sets up a stack for each
> > CPU.
> > +	  Stack memory is pre-allocated. U-Boot must therefore know the
> > +	  maximum number of CPUs that may be present.
> > +
> >   endmenu
> > diff --git a/arch/riscv/include/asm/global_data.h
> > b/arch/riscv/include/asm/global_data.h
> > index a3a342c6e1..80e3165e39 100644
> > --- a/arch/riscv/include/asm/global_data.h
> > +++ b/arch/riscv/include/asm/global_data.h
> > @@ -10,12 +10,18 @@
> >   #ifndef	__ASM_GBL_DATA_H
> >   #define __ASM_GBL_DATA_H
> >   
> > +#include <asm/smp.h>
> > +
> >   /* Architecture-specific global data */
> >   struct arch_global_data {
> >   	long boot_hart;		/* boot hart id */
> >   #ifdef CONFIG_SIFIVE_CLINT
> >   	void __iomem *clint;	/* clint base address */
> >   #endif
> > +#ifdef CONFIG_SMP
> > +	struct ipi_data ipi[CONFIG_NR_CPUS];
> > +#endif
> > +	ulong available_harts;
> >   };
> >   
> >   #include <asm-generic/global_data.h>
> > diff --git a/arch/riscv/include/asm/smp.h
> > b/arch/riscv/include/asm/smp.h
> > new file mode 100644
> > index 0000000000..bc863fdbaf
> > --- /dev/null
> > +++ b/arch/riscv/include/asm/smp.h
> > @@ -0,0 +1,53 @@
> > +/* SPDX-License-Identifier: GPL-2.0 */
> > +/*
> > + * Copyright (C) 2019 Fraunhofer AISEC,
> > + * Lukas Auer <lukas.auer@aisec.fraunhofer.de>
> > + */
> > +
> > +#ifndef _ASM_RISCV_SMP_H
> > +#define _ASM_RISCV_SMP_H
> > +
> > +/**
> > + * struct ipi_data - Inter-processor interrupt (IPI) data
> > structure
> > + *
> > + * IPIs are used for SMP support to communicate to other harts
> > what function to
> > + * call. Functions are in the form
> > + * void (*addr)(ulong hart, ulong arg0, ulong arg1).
> > + *
> > + * The function address and the two arguments, arg0 and arg1, are
> > stored in the
> > + * IPI data structure. The hart ID is inserted by the hart
> > handling the IPI and
> > + * calling the function.
> > + *
> > + * @addr: Address of function
> > + * @arg0: First argument of function
> > + * @arg1: Second argument of function
> > + */
> > +struct ipi_data {
> > +	ulong addr;
> > +	ulong arg0;
> > +	ulong arg1;
> > +};
> > +
> > +/**
> > + * handle_ipi() - interrupt handler for software interrupts
> > + *
> > + * The IPI interrupt handler must be called to handle software
> > interrupts. It
> > + * calls the function specified in the hart's IPI data structure.
> > + *
> > + * @hart: Hart ID of the current hart
> > + */
> > +void handle_ipi(ulong hart);
> > +
> > +/**
> > + * smp_call_function() - Call a function on all other harts
> > + *
> > + * Send IPIs with the specified function call to all harts.
> > + *
> > + * @addr: Address of function
> > + * @arg0: First argument of function
> > + * @arg1: Second argument of function
> > + * @return 0 if OK, -ve on error
> > + */
> > +int smp_call_function(ulong addr, ulong arg0, ulong arg1);
> > +
> > +#endif
> > diff --git a/arch/riscv/lib/Makefile b/arch/riscv/lib/Makefile
> > index edfa61690c..19370f9749 100644
> > --- a/arch/riscv/lib/Makefile
> > +++ b/arch/riscv/lib/Makefile
> > @@ -14,6 +14,7 @@ obj-$(CONFIG_SIFIVE_CLINT) += sifive_clint.o
> >   obj-y	+= interrupts.o
> >   obj-y	+= reset.o
> >   obj-y   += setjmp.o
> > +obj-$(CONFIG_SMP) += smp.o
> >   
> >   # For building EFI apps
> >   CFLAGS_$(EFI_CRT0) := $(CFLAGS_EFI)
> > diff --git a/arch/riscv/lib/smp.c b/arch/riscv/lib/smp.c
> > new file mode 100644
> > index 0000000000..edef8a687d
> > --- /dev/null
> > +++ b/arch/riscv/lib/smp.c
> > @@ -0,0 +1,116 @@
> > +// SPDX-License-Identifier: GPL-2.0+
> > +/*
> > + * Copyright (C) 2019 Fraunhofer AISEC,
> > + * Lukas Auer <lukas.auer@aisec.fraunhofer.de>
> > + */
> > +
> > +#include <common.h>
> > +#include <dm.h>
> > +#include <asm/barrier.h>
> > +#include <asm/smp.h>
> > +
> > +DECLARE_GLOBAL_DATA_PTR;
> > +
> > +/**
> > + * riscv_send_ipi() - Send inter-processor interrupt (IPI)
> > + *
> > + * Platform code must provide this function.
> > + *
> > + * @hart: Hart ID of receiving hart
> > + * @return 0 if OK, -ve on error
> > + */
> > +extern int riscv_send_ipi(int hart);
> > +
> > +/**
> > + * riscv_clear_ipi() - Clear inter-processor interrupt (IPI)
> > + *
> > + * Platform code must provide this function.
> > + *
> > + * @hart: Hart ID of hart to be cleared
> > + * @return 0 if OK, -ve on error
> > + */
> > +extern int riscv_clear_ipi(int hart);
> > +
> > +static int send_ipi_many(struct ipi_data *ipi)
> > +{
> > +	ofnode node, cpus;
> > +	u32 reg;
> > +	int ret;
> > +
> > +	cpus = ofnode_path("/cpus");
> > +	if (!ofnode_valid(cpus)) {
> > +		pr_err("Can't find cpus node!\n");
> > +		return -EINVAL;
> > +	}
> > +
> > +	ofnode_for_each_subnode(node, cpus) {
> > +		/* skip if hart is marked as not available in the
> > device tree */
> > +		if (!ofnode_is_available(node))
> > +			continue;
> > +
> > +		/* read hart ID of CPU */
> > +		ret = ofnode_read_u32(node, "reg", &reg);
> > +		if (ret)
> > +			continue;
> > +
> > +		/* skip if it is the hart we are running on */
> > +		if (reg == gd->arch.boot_hart)
> > +			continue;
> > +
> > +		if (reg >= CONFIG_NR_CPUS) {
> > +			pr_err("Hart ID %d is out of range, increase
> > CONFIG_NR_CPUS\n",
> > +			       reg);
> > +			continue
> > +		}
> > +
> > +		/* skip if hart is not available */
> > +		if (!(gd->arch.available_harts & (1 << reg)))
> > +			continue;
> > +
> > +		gd->arch.ipi[reg].addr = ipi->addr;
> > +		gd->arch.ipi[reg].arg0 = ipi->arg0;
> > +		gd->arch.ipi[reg].arg1 = ipi->arg1;
> > +
> > +		ret = riscv_send_ipi(reg);
> > +		if (ret)
> > +			return ret;
> > +	}
> > +
> > +	return 0;
> > +}
> > +
> > +void handle_ipi(ulong hart)
> > +{
> > +	int ret;
> > +	void (*smp_function)(ulong hart, ulong arg0, ulong arg1);
> > +
> > +	if (hart >= CONFIG_NR_CPUS)
> > +		return;
> > +
> 
> A warning here may help.
> 

This is meant more as a sanity check. Harts with an ID greater than
CONFIG_NR_CPUS should never run handle_ipi().
Warnings are already printed by the hart sending the IPIs, in
send_ipi_many() called by smp_call_function().

> > +	ret = riscv_clear_ipi(hart);
> > +	if (ret) {
> > +		pr_err("Cannot clear IPI\n");
> > +		return;
> > +	}
> > +
> 
> Currently, sbi_clear_ipi() doesn't return anything and
> riscv_clear_ipi() 
> is always returning 0. I guess you kept the check here so that we
> don't 
> have make changes in future. But IMHO, we should add it if SBI spec
> is 
> changed so that sbi_clear_ipi can return errors.
> 

The check is required if U-Boot is running in machine mode. The IPI
implementation using the CLINT driver returns an error if it cannot get
the memory address of the CLINT device.

I just noticed that no error is printed if riscv_send_ipi() returns an
error in send_ipi_many(). I will add one in the next version.

Thanks,
Lukas

> Regards,
> Atish
> > +	__smp_mb();
> > +
> > +	smp_function = (void (*)(ulong, ulong, ulong))gd-
> > >arch.ipi[hart].addr;
> > +	invalidate_icache_all();
> > +
> > +	smp_function(hart, gd->arch.ipi[hart].arg0, gd-
> > >arch.ipi[hart].arg1);
> > +}
> > +
> > +int smp_call_function(ulong addr, ulong arg0, ulong arg1)
> > +{
> > +	int ret = 0;
> > +	struct ipi_data ipi;
> > +
> > +	ipi.addr = addr;
> > +	ipi.arg0 = arg0;
> > +	ipi.arg1 = arg1;
> > +
> > +	ret = send_ipi_many(&ipi);
> > +
> > +	return ret;
> > +}
> > 

[U-Boot] [PATCH v2 0/9] SMP support for RISC-V

[Andreas Schwab]

On Mär 08 2019, Anup Patel <Anup.Patel@wdc.com> wrote:

> The reset button works fine for me an Atish. I am sure it works fine for lot of
> other folks too.

There is no issue with the reset button, only with openSBI.

Andreas.

-- 
Andreas Schwab, SUSE Labs, schwab at suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

[U-Boot] [PATCH v2 0/9] SMP support for RISC-V

[Palmer Dabbelt]

On Thu, 07 Mar 2019 19:37:30 PST (-0800), Anup Patel wrote:
> 
> 
>> -----Original Message-----
>> From: Andreas Schwab <schwab@suse.de>
>> Sent: Thursday, March 7, 2019 2:50 PM
>> To: Anup Patel <Anup.Patel@wdc.com>
>> Cc: Atish Patra <Atish.Patra@wdc.com>; Anup Patel <anup@brainfault.org>;
>> Auer, Lukas <lukas.auer@aisec.fraunhofer.de>; paul.walmsley at sifive.com;
>> agraf at suse.de; u-boot at lists.denx.de; baruch at tkos.co.il;
>> daniel.schwierzeck at gmail.com; bmeng.cn at gmail.com;
>> rick at andestech.com; sr at denx.de; palmer at sifive.com
>> Subject: Re: [PATCH v2 0/9] SMP support for RISC-V
>> 
>> On Mär 07 2019, Anup Patel <Anup.Patel@wdc.com> wrote:
>> 
>> > Like I mentioned, there is no functional issue with this series. The
>> > warm-boot issues were fixed in OpenSBI.
>> >
>> > @Andreas, please try at your end.
>> 
>> As long as issue#65 isn't fixed opensbi is mostly a no-go for me.  At least it
>> gives me more reasons to press the reset button. :-)
> 
> The reset button works fine for me an Atish. I am sure it works fine for lot of
> other folks too.
> 
> BTW, as-per discussion with SiFive folks the reset button on Unleashed
> Board is not much tested and it can misbehave on certain boards. It is quite
> possible that you might have a "flaky" board.

I don't think the reset button differs between boards.  As far as I know, the 
issues are really just that it doesn't reset everything -- specifically some of 
the IP on the chip (clock, power, JTAG) isn't reset and nothing on the board 
(SD, ethernet, PCIe, etc) is reset.  This frequently results in flakiness when 
debugging drivers, but the cores and memory system should all be OK.

Is that issue 65 on github.com/opensbi?  If so it clearly says this isn't a 
reset button issue.

[U-Boot] [PATCH v2 0/9] SMP support for RISC-V

[Anup Patel]

> -----Original Message-----
> From: Palmer Dabbelt <palmer@sifive.com>
> Sent: Monday, March 11, 2019 5:26 PM
> To: Anup Patel <Anup.Patel@wdc.com>
> Cc: schwab at suse.de; Atish Patra <Atish.Patra@wdc.com>;
> anup at brainfault.org; lukas.auer at aisec.fraunhofer.de; Paul Walmsley
> <paul.walmsley@sifive.com>; agraf at suse.de; u-boot at lists.denx.de;
> baruch at tkos.co.il; daniel.schwierzeck at gmail.com; bmeng.cn at gmail.com;
> rick at andestech.com; sr at denx.de
> Subject: RE: [PATCH v2 0/9] SMP support for RISC-V
> 
> On Thu, 07 Mar 2019 19:37:30 PST (-0800), Anup Patel wrote:
> >
> >
> >> -----Original Message-----
> >> From: Andreas Schwab <schwab@suse.de>
> >> Sent: Thursday, March 7, 2019 2:50 PM
> >> To: Anup Patel <Anup.Patel@wdc.com>
> >> Cc: Atish Patra <Atish.Patra@wdc.com>; Anup Patel
> >> <anup@brainfault.org>; Auer, Lukas <lukas.auer@aisec.fraunhofer.de>;
> >> paul.walmsley at sifive.com; agraf at suse.de; u-boot at lists.denx.de;
> >> baruch at tkos.co.il; daniel.schwierzeck at gmail.com;
> bmeng.cn at gmail.com;
> >> rick at andestech.com; sr at denx.de; palmer at sifive.com
> >> Subject: Re: [PATCH v2 0/9] SMP support for RISC-V
> >>
> >> On Mär 07 2019, Anup Patel <Anup.Patel@wdc.com> wrote:
> >>
> >> > Like I mentioned, there is no functional issue with this series.
> >> > The warm-boot issues were fixed in OpenSBI.
> >> >
> >> > @Andreas, please try at your end.
> >>
> >> As long as issue#65 isn't fixed opensbi is mostly a no-go for me.  At
> >> least it gives me more reasons to press the reset button. :-)
> >
> > The reset button works fine for me an Atish. I am sure it works fine
> > for lot of other folks too.
> >
> > BTW, as-per discussion with SiFive folks the reset button on Unleashed
> > Board is not much tested and it can misbehave on certain boards. It is
> > quite possible that you might have a "flaky" board.
> 
> I don't think the reset button differs between boards.  As far as I know, the
> issues are really just that it doesn't reset everything -- specifically some of
> the IP on the chip (clock, power, JTAG) isn't reset and nothing on the board
> (SD, ethernet, PCIe, etc) is reset.  This frequently results in flakiness when
> debugging drivers, but the cores and memory system should all be OK.
> 
> Is that issue 65 on github.com/opensbi?  If so it clearly says this isn't a reset
> button issue.

The issue#65 on githuh.com/opensbi is not a clearly defined and it went in
various directions. We tried various things suggested by Andreas and we were
only able to replicate issue with reset-button press. This fixed now and
reset-button press works perfectly fine with OpenSBI.

Apart from reset-button thingy, we tried all other things reported by Andreas
but we were not able to reproduce issue at our end.

Regards,
Anup

[U-Boot] [PATCH v2 7/9] riscv: do not rely on hart ID passed by previous boot stage

[Rick Chen]

Hi Lukas

<rick@andestech.com> 於 2019年3月12日 週二 下午7:04寫道：
>
>
>
> > -----Original Message-----
> > From: Lukas Auer [mailto:lukas.auer at aisec.fraunhofer.de]
> > Sent: Wednesday, March 06, 2019 6:53 AM
> > To: u-boot at lists.denx.de
> > Cc: Atish Patra; Anup Patel; Bin Meng; Andreas Schwab; Palmer Dabbelt;
> > Alexander Graf; Lukas Auer; Rick Jian-Zhi Chen(陳建志); Anup Patel
> > Subject: [PATCH v2 7/9] riscv: do not rely on hart ID passed by previous boot
> > stage
> >
> > RISC-V U-Boot expects the hart ID to be passed to it via register a0 by the
> > previous boot stage. Machine mode firmware such as BBL and OpenSBI do this
> > when starting their payload (U-Boot) in supervisor mode. If U-Boot is running in
> > machine mode, this task must be handled by the boot ROM. Explicitly populate
> > register a0 with the hart ID from the mhartid CSR to avoid possible problems on
> > RISC-V processors with a boot ROM that does not handle this task.
> >
> > Suggested-by: Rick Chen <rick@andestech.com>
> > Signed-off-by: Lukas Auer <lukas.auer@aisec.fraunhofer.de>
> > ---
> >
> > Changes in v2:
> > - New patch to populate register a0 with the hart ID from the mhartid CSR in
> > machine-mode
> >
> >  arch/riscv/cpu/start.S | 4 ++++
> >  1 file changed, 4 insertions(+)
> >
> > diff --git a/arch/riscv/cpu/start.S b/arch/riscv/cpu/start.S index
> > 79b753847c..d4daa6e0bf 100644
> > --- a/arch/riscv/cpu/start.S
> > +++ b/arch/riscv/cpu/start.S
> > @@ -36,6 +36,10 @@
> >  .section .text
> >  .globl _start
> >  _start:
> > +#ifdef CONFIG_RISCV_MMODE
> > +     csrr    a0, mhartid
> > +#endif
> > +
> >       /* save hart id and dtb pointer */
> >       mv      s0, a0
> >       mv      s1, a1

Reviewed-by: Rick Chen <rick@andestech.com>
Tested-by: Rick Chen <rick@andestech.com>

Thanks
Rick