From: Anup Patel <anup@brainfault.org>
To: Atish Patra <atishp@atishpatra.org>
Cc: "linux-kernel@vger.kernel.org List"
<linux-kernel@vger.kernel.org>, Atish Patra <atishp@rivosinc.com>,
Alexandre Ghiti <alex@ghiti.fr>, Anup Patel <anup.patel@wdc.com>,
Greentime Hu <greentime.hu@sifive.com>,
Guo Ren <guoren@linux.alibaba.com>,
Heinrich Schuchardt <xypron.glpk@gmx.de>,
Ingo Molnar <mingo@kernel.org>,
Jisheng Zhang <jszhang@kernel.org>,
kvm-riscv@lists.infradead.org, KVM General <kvm@vger.kernel.org>,
linux-riscv <linux-riscv@lists.infradead.org>,
Marc Zyngier <maz@kernel.org>,
Nanyong Sun <sunnanyong@huawei.com>,
Nick Kossifidis <mick@ics.forth.gr>,
Palmer Dabbelt <palmer@dabbelt.com>,
Paul Walmsley <paul.walmsley@sifive.com>,
Pekka Enberg <penberg@kernel.org>,
Vincent Chen <vincent.chen@sifive.com>,
Vitaly Wool <vitaly.wool@konsulko.com>
Subject: Re: [RFC 1/6] RISC-V: Avoid using per cpu array for ordered booting
Date: Mon, 13 Dec 2021 18:18:59 +0530 [thread overview]
Message-ID: <CAAhSdy2YsrGSk4P41hneNkJJg6je9fMYV9-py6vim=ZEexigOQ@mail.gmail.com> (raw)
In-Reply-To: <20211204002038.113653-2-atishp@atishpatra.org>
On Sat, Dec 4, 2021 at 5:50 AM Atish Patra <atishp@atishpatra.org> wrote:
>
> From: Atish Patra <atishp@rivosinc.com>
>
> Currently both order booting and spinwait approach uses a per cpu
> array to update stack & task pointer. This approach will not work for the
> following cases.
> 1. If NR_CPUs are configured to be less than highest hart id.
> 2. A platform has sparse hartid.
>
> This issue can be fixed for ordered booting as the booting cpu brings up
> one cpu at a time using SBI HSM extension which has opaque parameter
> that is unused until now.
>
> Introduce a common secondary boot data structure that can store the stack
> and task pointer. Secondary harts will use this data while booting up
> to setup the sp & tp.
>
> Signed-off-by: Atish Patra <atishp@rivosinc.com>
> ---
> arch/riscv/include/asm/cpu_ops_sbi.h | 28 ++++++++++++++++++++++++++++
> arch/riscv/kernel/cpu_ops_sbi.c | 23 ++++++++++++++++++++---
> arch/riscv/kernel/head.S | 19 ++++++++++---------
> 3 files changed, 58 insertions(+), 12 deletions(-)
> create mode 100644 arch/riscv/include/asm/cpu_ops_sbi.h
>
> diff --git a/arch/riscv/include/asm/cpu_ops_sbi.h b/arch/riscv/include/asm/cpu_ops_sbi.h
> new file mode 100644
> index 000000000000..ccb9a6d30486
> --- /dev/null
> +++ b/arch/riscv/include/asm/cpu_ops_sbi.h
> @@ -0,0 +1,28 @@
> +/* SPDX-License-Identifier: GPL-2.0-only */
> +/*
> + * Copyright (c) 2021 by Rivos Inc.
> + */
> +#ifndef __ASM_CPU_OPS_SBI_H
> +#define __ASM_CPU_OPS_SBI_H
> +
> +#ifndef __ASSEMBLY__
> +#include <linux/init.h>
> +#include <linux/sched.h>
> +#include <linux/threads.h>
> +
> +/**
> + * struct sbi_hart_boot_data - Hart specific boot used during booting and
> + * cpu hotplug.
> + * @task_ptr: A pointer to the hart specific tp
> + * @stack_ptr: A pointer to the hart specific sp
> + */
> +struct sbi_hart_boot_data {
> + void *task_ptr;
> + void *stack_ptr;
> +};
> +#endif
> +
> +#define SBI_HART_BOOT_TASK_PTR_OFFSET (0x00)
> +#define SBI_HART_BOOT_STACK_PTR_OFFSET RISCV_SZPTR
Don't manually create these defines instead generate this
defines at compile time by adding entries in kernel/asm-offsets.c
> +
> +#endif /* ifndef __ASM_CPU_OPS_H */
> diff --git a/arch/riscv/kernel/cpu_ops_sbi.c b/arch/riscv/kernel/cpu_ops_sbi.c
> index 685fae72b7f5..2e7a9dd9c2a7 100644
> --- a/arch/riscv/kernel/cpu_ops_sbi.c
> +++ b/arch/riscv/kernel/cpu_ops_sbi.c
> @@ -7,13 +7,22 @@
>
> #include <linux/init.h>
> #include <linux/mm.h>
> +#include <linux/sched/task_stack.h>
> #include <asm/cpu_ops.h>
> +#include <asm/cpu_ops_sbi.h>
> #include <asm/sbi.h>
> #include <asm/smp.h>
>
> extern char secondary_start_sbi[];
> const struct cpu_operations cpu_ops_sbi;
>
> +/*
> + * Ordered booting via HSM brings one cpu at a time. However, cpu hotplug can
> + * be invoked from multiple threads in paralle. Define a per cpu data
> + * to handle that.
> + */
> +DEFINE_PER_CPU(struct sbi_hart_boot_data, boot_data);
> +
> static int sbi_hsm_hart_start(unsigned long hartid, unsigned long saddr,
> unsigned long priv)
> {
> @@ -58,9 +67,17 @@ static int sbi_cpu_start(unsigned int cpuid, struct task_struct *tidle)
> int rc;
> unsigned long boot_addr = __pa_symbol(secondary_start_sbi);
> int hartid = cpuid_to_hartid_map(cpuid);
> -
> - cpu_update_secondary_bootdata(cpuid, tidle);
> - rc = sbi_hsm_hart_start(hartid, boot_addr, 0);
> + unsigned long hsm_data;
> + struct sbi_hart_boot_data *bdata = &per_cpu(boot_data, cpuid);
> +
> + /* Make sure tidle is updated */
> + smp_mb();
> + bdata->task_ptr = tidle;
> + bdata->stack_ptr = task_stack_page(tidle) + THREAD_SIZE;
> + /* Make sure boot data is updated */
> + smp_mb();
> + hsm_data = __pa(bdata);
> + rc = sbi_hsm_hart_start(hartid, boot_addr, hsm_data);
>
> return rc;
> }
> diff --git a/arch/riscv/kernel/head.S b/arch/riscv/kernel/head.S
> index f52f01ecbeea..40d4c625513c 100644
> --- a/arch/riscv/kernel/head.S
> +++ b/arch/riscv/kernel/head.S
> @@ -11,6 +11,7 @@
> #include <asm/page.h>
> #include <asm/pgtable.h>
> #include <asm/csr.h>
> +#include <asm/cpu_ops_sbi.h>
> #include <asm/hwcap.h>
> #include <asm/image.h>
> #include "efi-header.S"
> @@ -167,15 +168,15 @@ secondary_start_sbi:
> la a3, .Lsecondary_park
> csrw CSR_TVEC, a3
>
> - slli a3, a0, LGREG
> - la a4, __cpu_up_stack_pointer
> - XIP_FIXUP_OFFSET a4
> - la a5, __cpu_up_task_pointer
> - XIP_FIXUP_OFFSET a5
> - add a4, a3, a4
> - add a5, a3, a5
> - REG_L sp, (a4)
> - REG_L tp, (a5)
> + /* a0 contains the hartid & a1 contains boot data */
> + li a2, SBI_HART_BOOT_TASK_PTR_OFFSET
> + XIP_FIXUP_OFFSET a2
> + add a2, a2, a1
> + REG_L tp, (a2)
> + li a3, SBI_HART_BOOT_STACK_PTR_OFFSET
> + XIP_FIXUP_OFFSET a3
> + add a3, a3, a1
> + REG_L sp, (a3)
>
> .global secondary_start_common
> secondary_start_common:
> --
> 2.33.1
>
Regards,
Anup
next prev parent reply other threads:[~2021-12-13 12:49 UTC|newest]
Thread overview: 19+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-12-04 0:20 [RFC 0/6] Sparse HART id support Atish Patra
2021-12-04 0:20 ` [RFC 1/6] RISC-V: Avoid using per cpu array for ordered booting Atish Patra
2021-12-13 12:48 ` Anup Patel [this message]
2021-12-13 21:05 ` Atish Patra
2021-12-04 0:20 ` [RFC 2/6] RISC-V: Do not print the SBI version during HSM extension boot print Atish Patra
2021-12-13 12:49 ` Anup Patel
2021-12-04 0:20 ` [RFC 3/6] RISC-V: Use __cpu_up_stack/task_pointer only for spinwait method Atish Patra
2021-12-13 12:50 ` Anup Patel
2021-12-13 12:59 ` Marc Zyngier
2021-12-13 21:12 ` Atish Patra
2021-12-04 0:20 ` [RFC 5/6] RISC-V: Move spinwait booting method to its own config Atish Patra
2021-12-04 0:40 ` Randy Dunlap
2021-12-13 13:01 ` Anup Patel
2021-12-13 21:08 ` Atish Patra
2021-12-04 0:20 ` [RFC 6/6] RISC-V: Do not use cpumask data structure for hartid bitmap Atish Patra
2021-12-06 15:28 ` [RFC 0/6] Sparse HART id support Rob Herring
2021-12-13 21:27 ` Atish Patra
2021-12-13 23:11 ` Rob Herring
2021-12-14 0:58 ` Atish Patra
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to='CAAhSdy2YsrGSk4P41hneNkJJg6je9fMYV9-py6vim=ZEexigOQ@mail.gmail.com' \
--to=anup@brainfault.org \
--cc=alex@ghiti.fr \
--cc=anup.patel@wdc.com \
--cc=atishp@atishpatra.org \
--cc=atishp@rivosinc.com \
--cc=greentime.hu@sifive.com \
--cc=guoren@linux.alibaba.com \
--cc=jszhang@kernel.org \
--cc=kvm-riscv@lists.infradead.org \
--cc=kvm@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-riscv@lists.infradead.org \
--cc=maz@kernel.org \
--cc=mick@ics.forth.gr \
--cc=mingo@kernel.org \
--cc=palmer@dabbelt.com \
--cc=paul.walmsley@sifive.com \
--cc=penberg@kernel.org \
--cc=sunnanyong@huawei.com \
--cc=vincent.chen@sifive.com \
--cc=vitaly.wool@konsulko.com \
--cc=xypron.glpk@gmx.de \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).