[U-Boot] Hart lottery and CONFIG_XIP

[U-Boot] Hart lottery and CONFIG_XIP

[Troy Benjegerdes]

> On Jun 2, 2019, at 9:22 PM, Rick Chen <rickchen36@gmail.com> wrote:
> 
> Hi Troy
> 
>>> From: Troy Benjegerdes [mailto:troy.benjegerdes at sifive.com]
>>> Sent: Saturday, June 01, 2019 12:24 AM
>>> To: Auer, Lukas
>>> Cc: Rick Jian-Zhi Chen(陳建志); bmeng.cn at gmail.com; palmer at sifive.com
>>> Subject: Re: Hart lottery and CONFIG_XIP
>>> 
>>> 
>>> 
>>>> On May 31, 2019, at 9:13 AM, Auer, Lukas <lukas.auer@aisec.fraunhofer.de>
>>> wrote:
>>>> 
>>>> Hi Troy,
>>>> 
>>>> On Fri, 2019-05-31 at 08:18 -0500, Troy Benjegerdes wrote:
>>>>> Wouldn't the following line in head.S fail when running from flash
>>>>> (although maybe not in a way that prevents booting)
>>>>> 
>>>>>       /* save the boot hart id to global_data */
>>>>>       SREG    tp, GD_BOOT_HART(gp)
>>>>> 
>>>>> Shouldn’t this be protected by CONFIG_XIP?
>>>> 
>>>> The boot hart ID is stored in global data, which is allocated from the
>>>> stack (in board_init_f_alloc_reserve() ). It is therefore writable and
>>>> won't cause any issues when running from flash.
>>> 
>>> Sorry about the confusion, I was reading it wrong earlier.
>>> 
>>> I’m hoping to have a couple of patches ready later today to change the
>>> CONFIG_XIP patch to ‘CONFIG_HART_LOTTERY’ since it is also useful to
>>> remove the potential indeterminism of which hart wins the lottery when doing
>>> board bringup and debugging.
> 
> I am OK with that.
> Actually my preliminary patch about
> [PATCH 0/4] AE350 support SMP boot from flash
> did as your wish.
> 
> You can refer it :
> [PATCH 1/4] riscv: hart_lottery and available harts feature can be seletable
> https://www.mail-archive.com/u-boot at lists.denx.de/msg323419.html
> 
> Rick

To follow up on https://www.mail-archive.com/u-boot at lists.denx.de/msg323526.html

It is confusing and misleading to mix CONFIG_XIP and CONFIG_HART_LOTTERY.

From an system testing and validation point of view, I would find it much better
(especially at very early boot stages, where U-boot might be the first non-ROM code
running) to have a deterministic process to determine what core runs U-boot. This 
necessarily seems as it will require a CONFIG_BOOT_HART option, as we are not
in a position to parse a device tree in head.S

[U-Boot] Hart lottery and CONFIG_XIP

[Bin Meng]

+Anup

Hi Troy,

On Mon, Jun 3, 2019 at 9:19 PM Troy Benjegerdes
<troy.benjegerdes@sifive.com> wrote:
>
>
>
> > On Jun 2, 2019, at 9:22 PM, Rick Chen <rickchen36@gmail.com> wrote:
> >
> > Hi Troy
> >
> >>> From: Troy Benjegerdes [mailto:troy.benjegerdes at sifive.com]
> >>> Sent: Saturday, June 01, 2019 12:24 AM
> >>> To: Auer, Lukas
> >>> Cc: Rick Jian-Zhi Chen(陳建志); bmeng.cn at gmail.com; palmer at sifive.com
> >>> Subject: Re: Hart lottery and CONFIG_XIP
> >>>
> >>>
> >>>
> >>>> On May 31, 2019, at 9:13 AM, Auer, Lukas <lukas.auer@aisec.fraunhofer.de>
> >>> wrote:
> >>>>
> >>>> Hi Troy,
> >>>>
> >>>> On Fri, 2019-05-31 at 08:18 -0500, Troy Benjegerdes wrote:
> >>>>> Wouldn't the following line in head.S fail when running from flash
> >>>>> (although maybe not in a way that prevents booting)
> >>>>>
> >>>>>       /* save the boot hart id to global_data */
> >>>>>       SREG    tp, GD_BOOT_HART(gp)
> >>>>>
> >>>>> Shouldn’t this be protected by CONFIG_XIP?
> >>>>
> >>>> The boot hart ID is stored in global data, which is allocated from the
> >>>> stack (in board_init_f_alloc_reserve() ). It is therefore writable and
> >>>> won't cause any issues when running from flash.
> >>>
> >>> Sorry about the confusion, I was reading it wrong earlier.
> >>>
> >>> I’m hoping to have a couple of patches ready later today to change the
> >>> CONFIG_XIP patch to ‘CONFIG_HART_LOTTERY’ since it is also useful to
> >>> remove the potential indeterminism of which hart wins the lottery when doing
> >>> board bringup and debugging.
> >
> > I am OK with that.
> > Actually my preliminary patch about
> > [PATCH 0/4] AE350 support SMP boot from flash
> > did as your wish.
> >
> > You can refer it :
> > [PATCH 1/4] riscv: hart_lottery and available harts feature can be seletable
> > https://www.mail-archive.com/u-boot at lists.denx.de/msg323419.html
> >
> > Rick
>
> To follow up on https://www.mail-archive.com/u-boot at lists.denx.de/msg323526.html
>
> It is confusing and misleading to mix CONFIG_XIP and CONFIG_HART_LOTTERY.
>

I am not sure what caused the confusion. CONFIG_XIP was added to
support U-Boot executing from ROM directly.

> From an system testing and validation point of view, I would find it much better
> (especially at very early boot stages, where U-boot might be the first non-ROM code
> running) to have a deterministic process to determine what core runs U-boot. This

I remember when SMP patches were submitted by Lukas for the first time
it was deterministic using some macros like CONFIG_BOOT_HART, however
per Anup's request, the hart lottery feature, similar to what Linux
has, was added.

> necessarily seems as it will require a CONFIG_BOOT_HART option, as we are not
> in a position to parse a device tree in head.S
>

Yes, this is understood. So the CONFIG_XIP was added to support such
case, for which the lottery feature relies on a memory location which
isn't writable yet.

Regards,
Bin

[U-Boot] Hart lottery and CONFIG_XIP

[Troy Benjegerdes]

> On Jun 3, 2019, at 9:19 AM, Bin Meng <bmeng.cn@gmail.com> wrote:
> 
> +Anup
> 
> Hi Troy,
> 
> On Mon, Jun 3, 2019 at 9:19 PM Troy Benjegerdes
> <troy.benjegerdes@sifive.com> wrote:
>> 
>> 
>> 
>>> On Jun 2, 2019, at 9:22 PM, Rick Chen <rickchen36@gmail.com> wrote:
>>> 
>>> Hi Troy
>>> 
>>>>> From: Troy Benjegerdes [mailto:troy.benjegerdes at sifive.com]
>>>>> Sent: Saturday, June 01, 2019 12:24 AM
>>>>> To: Auer, Lukas
>>>>> Cc: Rick Jian-Zhi Chen(陳建志); bmeng.cn at gmail.com; palmer at sifive.com
>>>>> Subject: Re: Hart lottery and CONFIG_XIP
>>>>> 
>>>>> 
>>>>> 
>>>>>> On May 31, 2019, at 9:13 AM, Auer, Lukas <lukas.auer@aisec.fraunhofer.de>
>>>>> wrote:
>>>>>> 
>>>>>> Hi Troy,
>>>>>> 
>>>>>> On Fri, 2019-05-31 at 08:18 -0500, Troy Benjegerdes wrote:
>>>>>>> Wouldn't the following line in head.S fail when running from flash
>>>>>>> (although maybe not in a way that prevents booting)
>>>>>>> 
>>>>>>>      /* save the boot hart id to global_data */
>>>>>>>      SREG    tp, GD_BOOT_HART(gp)
>>>>>>> 
>>>>>>> Shouldn’t this be protected by CONFIG_XIP?
>>>>>> 
>>>>>> The boot hart ID is stored in global data, which is allocated from the
>>>>>> stack (in board_init_f_alloc_reserve() ). It is therefore writable and
>>>>>> won't cause any issues when running from flash.
>>>>> 
>>>>> Sorry about the confusion, I was reading it wrong earlier.
>>>>> 
>>>>> I’m hoping to have a couple of patches ready later today to change the
>>>>> CONFIG_XIP patch to ‘CONFIG_HART_LOTTERY’ since it is also useful to
>>>>> remove the potential indeterminism of which hart wins the lottery when doing
>>>>> board bringup and debugging.
>>> 
>>> I am OK with that.
>>> Actually my preliminary patch about
>>> [PATCH 0/4] AE350 support SMP boot from flash
>>> did as your wish.
>>> 
>>> You can refer it :
>>> [PATCH 1/4] riscv: hart_lottery and available harts feature can be seletable
>>> https://www.mail-archive.com/u-boot at lists.denx.de/msg323419.html
>>> 
>>> Rick
>> 
>> To follow up on https://www.mail-archive.com/u-boot at lists.denx.de/msg323526.html
>> 
>> It is confusing and misleading to mix CONFIG_XIP and CONFIG_HART_LOTTERY.
>> 
> 
> I am not sure what caused the confusion. CONFIG_XIP was added to
> support U-Boot executing from ROM directly.
> 

The confusion is use cases where you don’t necessarily need CONFIG_XIP,
but you do want deterministic SMP booting, and the code is a lot more understandable
with ‘#ifdef CONFIG_HART_LOTTERY’, and a Kconfig option and/or documentation
warning that says CONFIG_HART_LOTTERY depends on !CONFIG_XIP

>> From an system testing and validation point of view, I would find it much better
>> (especially at very early boot stages, where U-boot might be the first non-ROM code
>> running) to have a deterministic process to determine what core runs U-boot. This
> 
> I remember when SMP patches were submitted by Lukas for the first time
> it was deterministic using some macros like CONFIG_BOOT_HART, however
> per Anup's request, the hart lottery feature, similar to what Linux
> has, was added.

I’d like to have CONFIG_BOOT_HART available as a config option as well, particularly
for system validation testing. Also along those lines, if we are going to use a lottery
to determine what CPU boots the system, how do we know afterwards which one
it was?

> 
>> necessarily seems as it will require a CONFIG_BOOT_HART option, as we are not
>> in a position to parse a device tree in head.S
>> 
> 
> Yes, this is understood. So the CONFIG_XIP was added to support such
> case, for which the lottery feature relies on a memory location which
> isn't writable yet.
> 
> Regards,
> Bin

[U-Boot] Hart lottery and CONFIG_XIP

[Bin Meng]

Hi Troy,

On Mon, Jun 3, 2019 at 10:53 PM Troy Benjegerdes
<troy.benjegerdes@sifive.com> wrote:
>
>
>
> > On Jun 3, 2019, at 9:19 AM, Bin Meng <bmeng.cn@gmail.com> wrote:
> >
> > +Anup
> >
> > Hi Troy,
> >
> > On Mon, Jun 3, 2019 at 9:19 PM Troy Benjegerdes
> > <troy.benjegerdes@sifive.com> wrote:
> >>
> >>
> >>
> >>> On Jun 2, 2019, at 9:22 PM, Rick Chen <rickchen36@gmail.com> wrote:
> >>>
> >>> Hi Troy
> >>>
> >>>>> From: Troy Benjegerdes [mailto:troy.benjegerdes at sifive.com]
> >>>>> Sent: Saturday, June 01, 2019 12:24 AM
> >>>>> To: Auer, Lukas
> >>>>> Cc: Rick Jian-Zhi Chen(陳建志); bmeng.cn at gmail.com; palmer at sifive.com
> >>>>> Subject: Re: Hart lottery and CONFIG_XIP
> >>>>>
> >>>>>
> >>>>>
> >>>>>> On May 31, 2019, at 9:13 AM, Auer, Lukas <lukas.auer@aisec.fraunhofer.de>
> >>>>> wrote:
> >>>>>>
> >>>>>> Hi Troy,
> >>>>>>
> >>>>>> On Fri, 2019-05-31 at 08:18 -0500, Troy Benjegerdes wrote:
> >>>>>>> Wouldn't the following line in head.S fail when running from flash
> >>>>>>> (although maybe not in a way that prevents booting)
> >>>>>>>
> >>>>>>>      /* save the boot hart id to global_data */
> >>>>>>>      SREG    tp, GD_BOOT_HART(gp)
> >>>>>>>
> >>>>>>> Shouldn’t this be protected by CONFIG_XIP?
> >>>>>>
> >>>>>> The boot hart ID is stored in global data, which is allocated from the
> >>>>>> stack (in board_init_f_alloc_reserve() ). It is therefore writable and
> >>>>>> won't cause any issues when running from flash.
> >>>>>
> >>>>> Sorry about the confusion, I was reading it wrong earlier.
> >>>>>
> >>>>> I’m hoping to have a couple of patches ready later today to change the
> >>>>> CONFIG_XIP patch to ‘CONFIG_HART_LOTTERY’ since it is also useful to
> >>>>> remove the potential indeterminism of which hart wins the lottery when doing
> >>>>> board bringup and debugging.
> >>>
> >>> I am OK with that.
> >>> Actually my preliminary patch about
> >>> [PATCH 0/4] AE350 support SMP boot from flash
> >>> did as your wish.
> >>>
> >>> You can refer it :
> >>> [PATCH 1/4] riscv: hart_lottery and available harts feature can be seletable
> >>> https://www.mail-archive.com/u-boot at lists.denx.de/msg323419.html
> >>>
> >>> Rick
> >>
> >> To follow up on https://www.mail-archive.com/u-boot at lists.denx.de/msg323526.html
> >>
> >> It is confusing and misleading to mix CONFIG_XIP and CONFIG_HART_LOTTERY.
> >>
> >
> > I am not sure what caused the confusion. CONFIG_XIP was added to
> > support U-Boot executing from ROM directly.
> >
>
> The confusion is use cases where you don’t necessarily need CONFIG_XIP,
> but you do want deterministic SMP booting, and the code is a lot more understandable
> with ‘#ifdef CONFIG_HART_LOTTERY’, and a Kconfig option and/or documentation
> warning that says CONFIG_HART_LOTTERY depends on !CONFIG_XIP
>

OK, as I pointed out, the deterministic SMP booting was originally
posted, but later was changed to the lottery mechanism per Anup's
review comments. I am personally fine with both.

> >> From an system testing and validation point of view, I would find it much better
> >> (especially at very early boot stages, where U-boot might be the first non-ROM code
> >> running) to have a deterministic process to determine what core runs U-boot. This
> >
> > I remember when SMP patches were submitted by Lukas for the first time
> > it was deterministic using some macros like CONFIG_BOOT_HART, however
> > per Anup's request, the hart lottery feature, similar to what Linux
> > has, was added.
>
> I’d like to have CONFIG_BOOT_HART available as a config option as well, particularly
> for system validation testing. Also along those lines, if we are going to use a lottery
> to determine what CPU boots the system, how do we know afterwards which one
> it was?
>

Yes, there is a way to know. Type 'dm tree' from U-Boot shell, then
look at which cpu node has a child node of "riscv_timer".

Regards,
Bin

[U-Boot] Hart lottery and CONFIG_XIP

[Rick Chen]

Hi Troy

Bin Meng <bmeng.cn@gmail.com> 於 2019年6月3日 週一 下午10:56寫道：
>
> Hi Troy,
>
> On Mon, Jun 3, 2019 at 10:53 PM Troy Benjegerdes
> <troy.benjegerdes@sifive.com> wrote:
> >
> >
> >
> > > On Jun 3, 2019, at 9:19 AM, Bin Meng <bmeng.cn@gmail.com> wrote:
> > >
> > > +Anup
> > >
> > > Hi Troy,
> > >
> > > On Mon, Jun 3, 2019 at 9:19 PM Troy Benjegerdes
> > > <troy.benjegerdes@sifive.com> wrote:
> > >>
> > >>
> > >>
> > >>> On Jun 2, 2019, at 9:22 PM, Rick Chen <rickchen36@gmail.com> wrote:
> > >>>
> > >>> Hi Troy
> > >>>
> > >>>>> From: Troy Benjegerdes [mailto:troy.benjegerdes at sifive.com]
> > >>>>> Sent: Saturday, June 01, 2019 12:24 AM
> > >>>>> To: Auer, Lukas
> > >>>>> Cc: Rick Jian-Zhi Chen(陳建志); bmeng.cn at gmail.com; palmer at sifive.com
> > >>>>> Subject: Re: Hart lottery and CONFIG_XIP
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>>> On May 31, 2019, at 9:13 AM, Auer, Lukas <lukas.auer@aisec.fraunhofer.de>
> > >>>>> wrote:
> > >>>>>>
> > >>>>>> Hi Troy,
> > >>>>>>
> > >>>>>> On Fri, 2019-05-31 at 08:18 -0500, Troy Benjegerdes wrote:
> > >>>>>>> Wouldn't the following line in head.S fail when running from flash
> > >>>>>>> (although maybe not in a way that prevents booting)
> > >>>>>>>
> > >>>>>>>      /* save the boot hart id to global_data */
> > >>>>>>>      SREG    tp, GD_BOOT_HART(gp)
> > >>>>>>>
> > >>>>>>> Shouldn’t this be protected by CONFIG_XIP?
> > >>>>>>
> > >>>>>> The boot hart ID is stored in global data, which is allocated from the
> > >>>>>> stack (in board_init_f_alloc_reserve() ). It is therefore writable and
> > >>>>>> won't cause any issues when running from flash.
> > >>>>>
> > >>>>> Sorry about the confusion, I was reading it wrong earlier.
> > >>>>>
> > >>>>> I’m hoping to have a couple of patches ready later today to change the
> > >>>>> CONFIG_XIP patch to ‘CONFIG_HART_LOTTERY’ since it is also useful to
> > >>>>> remove the potential indeterminism of which hart wins the lottery when doing
> > >>>>> board bringup and debugging.
> > >>>
> > >>> I am OK with that.
> > >>> Actually my preliminary patch about
> > >>> [PATCH 0/4] AE350 support SMP boot from flash
> > >>> did as your wish.
> > >>>
> > >>> You can refer it :
> > >>> [PATCH 1/4] riscv: hart_lottery and available harts feature can be seletable
> > >>> https://www.mail-archive.com/u-boot at lists.denx.de/msg323419.html
> > >>>
> > >>> Rick
> > >>
> > >> To follow up on https://www.mail-archive.com/u-boot at lists.denx.de/msg323526.html
> > >>
> > >> It is confusing and misleading to mix CONFIG_XIP and CONFIG_HART_LOTTERY.
> > >>
> > >
> > > I am not sure what caused the confusion. CONFIG_XIP was added to
> > > support U-Boot executing from ROM directly.
> > >
> >
> > The confusion is use cases where you don’t necessarily need CONFIG_XIP,
> > but you do want deterministic SMP booting, and the code is a lot more understandable
> > with ‘#ifdef CONFIG_HART_LOTTERY’, and a Kconfig option and/or documentation
> > warning that says CONFIG_HART_LOTTERY depends on !CONFIG_XIP
> >
>
> OK, as I pointed out, the deterministic SMP booting was originally
> posted, but later was changed to the lottery mechanism per Anup's
> review comments. I am personally fine with both.

Yes. I am agree with that.
But it shall also support booting form ROM and RAM as now.
>
> > >> From an system testing and validation point of view, I would find it much better
> > >> (especially at very early boot stages, where U-boot might be the first non-ROM code
> > >> running) to have a deterministic process to determine what core runs U-boot. This
> > >
> > > I remember when SMP patches were submitted by Lukas for the first time
> > > it was deterministic using some macros like CONFIG_BOOT_HART, however
> > > per Anup's request, the hart lottery feature, similar to what Linux
> > > has, was added.
> >
> > I’d like to have CONFIG_BOOT_HART available as a config option as well, particularly
> > for system validation testing. Also along those lines, if we are going to use a lottery
> > to determine what CPU boots the system, how do we know afterwards which one
> > it was?
> >
>
> Yes, there is a way to know. Type 'dm tree' from U-Boot shell, then
> look at which cpu node has a child node of "riscv_timer".

Or you can check gd->arch.boot_hart by gdb

(gdb) p/x gd->arch.boot_hart
$3 = 0x0

B.R
Rick

>
> Regards,
> Bin

[U-Boot] Hart lottery and CONFIG_XIP

[Auer, Lukas]

On Tue, 2019-06-04 at 09:42 +0800, Rick Chen wrote:
> Hi Troy
> 
> Bin Meng <bmeng.cn@gmail.com> 於 2019年6月3日 週一 下午10:56寫道：
> > Hi Troy,
> > 
> > On Mon, Jun 3, 2019 at 10:53 PM Troy Benjegerdes
> > <troy.benjegerdes@sifive.com> wrote:
> > > 
> > > 
> > > > On Jun 3, 2019, at 9:19 AM, Bin Meng <bmeng.cn@gmail.com> wrote:
> > > > 
> > > > +Anup
> > > > 
> > > > Hi Troy,
> > > > 
> > > > On Mon, Jun 3, 2019 at 9:19 PM Troy Benjegerdes
> > > > <troy.benjegerdes@sifive.com> wrote:
> > > > > 
> > > > > 
> > > > > > On Jun 2, 2019, at 9:22 PM, Rick Chen <rickchen36@gmail.com> wrote:
> > > > > > 
> > > > > > Hi Troy
> > > > > > 
> > > > > > > > From: Troy Benjegerdes [mailto:troy.benjegerdes at sifive.com]
> > > > > > > > Sent: Saturday, June 01, 2019 12:24 AM
> > > > > > > > To: Auer, Lukas
> > > > > > > > Cc: Rick Jian-Zhi Chen(陳建志); bmeng.cn at gmail.com; palmer at sifive.com
> > > > > > > > Subject: Re: Hart lottery and CONFIG_XIP
> > > > > > > > 
> > > > > > > > 
> > > > > > > > 
> > > > > > > > > On May 31, 2019, at 9:13 AM, Auer, Lukas <lukas.auer@aisec.fraunhofer.de>
> > > > > > > > wrote:
> > > > > > > > > Hi Troy,
> > > > > > > > > 
> > > > > > > > > On Fri, 2019-05-31 at 08:18 -0500, Troy Benjegerdes wrote:
> > > > > > > > > > Wouldn't the following line in head.S fail when running from flash
> > > > > > > > > > (although maybe not in a way that prevents booting)
> > > > > > > > > > 
> > > > > > > > > >      /* save the boot hart id to global_data */
> > > > > > > > > >      SREG    tp, GD_BOOT_HART(gp)
> > > > > > > > > > 
> > > > > > > > > > Shouldn’t this be protected by CONFIG_XIP?
> > > > > > > > > 
> > > > > > > > > The boot hart ID is stored in global data, which is allocated from the
> > > > > > > > > stack (in board_init_f_alloc_reserve() ). It is therefore writable and
> > > > > > > > > won't cause any issues when running from flash.
> > > > > > > > 
> > > > > > > > Sorry about the confusion, I was reading it wrong earlier.
> > > > > > > > 
> > > > > > > > I’m hoping to have a couple of patches ready later today to change the
> > > > > > > > CONFIG_XIP patch to ‘CONFIG_HART_LOTTERY’ since it is also useful to
> > > > > > > > remove the potential indeterminism of which hart wins the lottery when doing
> > > > > > > > board bringup and debugging.
> > > > > > 
> > > > > > I am OK with that.
> > > > > > Actually my preliminary patch about
> > > > > > [PATCH 0/4] AE350 support SMP boot from flash
> > > > > > did as your wish.
> > > > > > 
> > > > > > You can refer it :
> > > > > > [PATCH 1/4] riscv: hart_lottery and available harts feature can be seletable
> > > > > > https://www.mail-archive.com/u-boot at lists.denx.de/msg323419.html
> > > > > > 
> > > > > > Rick
> > > > > 
> > > > > To follow up on https://www.mail-archive.com/u-boot at lists.denx.de/msg323526.html
> > > > > 
> > > > > It is confusing and misleading to mix CONFIG_XIP and CONFIG_HART_LOTTERY.
> > > > > 
> > > > 
> > > > I am not sure what caused the confusion. CONFIG_XIP was added to
> > > > support U-Boot executing from ROM directly.
> > > > 
> > > 
> > > The confusion is use cases where you don’t necessarily need CONFIG_XIP,
> > > but you do want deterministic SMP booting, and the code is a lot more understandable
> > > with ‘#ifdef CONFIG_HART_LOTTERY’, and a Kconfig option and/or documentation
> > > warning that says CONFIG_HART_LOTTERY depends on !CONFIG_XIP
> > > 
> > 
> > OK, as I pointed out, the deterministic SMP booting was originally
> > posted, but later was changed to the lottery mechanism per Anup's
> > review comments. I am personally fine with both.
> 
> Yes. I am agree with that.
> But it shall also support booting form ROM and RAM as now.

Just to give a bit of context information. Anup's reasoning was that
with a fixed boot hart, we have to rely on it to work. If there is a
hardware failure in just the boot hart, the system won't be able to
boot anymore. With a hart lottery that is not a problem.

With that in mind, it makes sense to have to option of a fixed boot
hart, but I would not enable that option in any of the upstream
defconfigs. That way it can be manually enabled if required for the
application.

Thanks,
Lukas

> > > > > From an system testing and validation point of view, I would find it much better
> > > > > (especially at very early boot stages, where U-boot might be the first non-ROM code
> > > > > running) to have a deterministic process to determine what core runs U-boot. This
> > > > 
> > > > I remember when SMP patches were submitted by Lukas for the first time
> > > > it was deterministic using some macros like CONFIG_BOOT_HART, however
> > > > per Anup's request, the hart lottery feature, similar to what Linux
> > > > has, was added.
> > > 
> > > I’d like to have CONFIG_BOOT_HART available as a config option as well, particularly
> > > for system validation testing. Also along those lines, if we are going to use a lottery
> > > to determine what CPU boots the system, how do we know afterwards which one
> > > it was?
> > > 
> > 
> > Yes, there is a way to know. Type 'dm tree' from U-Boot shell, then
> > look at which cpu node has a child node of "riscv_timer".
> 
> Or you can check gd->arch.boot_hart by gdb
> 
> (gdb) p/x gd->arch.boot_hart
> $3 = 0x0
> 
> B.R
> Rick
> 
> > Regards,
> > Bin

[U-Boot] Hart lottery and CONFIG_XIP

[Anup Patel]

On Tue, Jun 4, 2019 at 3:00 PM Auer, Lukas
<lukas.auer@aisec.fraunhofer.de> wrote:
>
> On Tue, 2019-06-04 at 09:42 +0800, Rick Chen wrote:
> > Hi Troy
> >
> > Bin Meng <bmeng.cn@gmail.com> 於 2019年6月3日 週一 下午10:56寫道：
> > > Hi Troy,
> > >
> > > On Mon, Jun 3, 2019 at 10:53 PM Troy Benjegerdes
> > > <troy.benjegerdes@sifive.com> wrote:
> > > >
> > > >
> > > > > On Jun 3, 2019, at 9:19 AM, Bin Meng <bmeng.cn@gmail.com> wrote:
> > > > >
> > > > > +Anup
> > > > >
> > > > > Hi Troy,
> > > > >
> > > > > On Mon, Jun 3, 2019 at 9:19 PM Troy Benjegerdes
> > > > > <troy.benjegerdes@sifive.com> wrote:
> > > > > >
> > > > > >
> > > > > > > On Jun 2, 2019, at 9:22 PM, Rick Chen <rickchen36@gmail.com> wrote:
> > > > > > >
> > > > > > > Hi Troy
> > > > > > >
> > > > > > > > > From: Troy Benjegerdes [mailto:troy.benjegerdes at sifive.com]
> > > > > > > > > Sent: Saturday, June 01, 2019 12:24 AM
> > > > > > > > > To: Auer, Lukas
> > > > > > > > > Cc: Rick Jian-Zhi Chen(陳建志); bmeng.cn at gmail.com; palmer at sifive.com
> > > > > > > > > Subject: Re: Hart lottery and CONFIG_XIP
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > > On May 31, 2019, at 9:13 AM, Auer, Lukas <lukas.auer@aisec.fraunhofer.de>
> > > > > > > > > wrote:
> > > > > > > > > > Hi Troy,
> > > > > > > > > >
> > > > > > > > > > On Fri, 2019-05-31 at 08:18 -0500, Troy Benjegerdes wrote:
> > > > > > > > > > > Wouldn't the following line in head.S fail when running from flash
> > > > > > > > > > > (although maybe not in a way that prevents booting)
> > > > > > > > > > >
> > > > > > > > > > >      /* save the boot hart id to global_data */
> > > > > > > > > > >      SREG    tp, GD_BOOT_HART(gp)
> > > > > > > > > > >
> > > > > > > > > > > Shouldn’t this be protected by CONFIG_XIP?
> > > > > > > > > >
> > > > > > > > > > The boot hart ID is stored in global data, which is allocated from the
> > > > > > > > > > stack (in board_init_f_alloc_reserve() ). It is therefore writable and
> > > > > > > > > > won't cause any issues when running from flash.
> > > > > > > > >
> > > > > > > > > Sorry about the confusion, I was reading it wrong earlier.
> > > > > > > > >
> > > > > > > > > I’m hoping to have a couple of patches ready later today to change the
> > > > > > > > > CONFIG_XIP patch to ‘CONFIG_HART_LOTTERY’ since it is also useful to
> > > > > > > > > remove the potential indeterminism of which hart wins the lottery when doing
> > > > > > > > > board bringup and debugging.
> > > > > > >
> > > > > > > I am OK with that.
> > > > > > > Actually my preliminary patch about
> > > > > > > [PATCH 0/4] AE350 support SMP boot from flash
> > > > > > > did as your wish.
> > > > > > >
> > > > > > > You can refer it :
> > > > > > > [PATCH 1/4] riscv: hart_lottery and available harts feature can be seletable
> > > > > > > https://www.mail-archive.com/u-boot at lists.denx.de/msg323419.html
> > > > > > >
> > > > > > > Rick
> > > > > >
> > > > > > To follow up on https://www.mail-archive.com/u-boot at lists.denx.de/msg323526.html
> > > > > >
> > > > > > It is confusing and misleading to mix CONFIG_XIP and CONFIG_HART_LOTTERY.
> > > > > >
> > > > >
> > > > > I am not sure what caused the confusion. CONFIG_XIP was added to
> > > > > support U-Boot executing from ROM directly.
> > > > >
> > > >
> > > > The confusion is use cases where you don’t necessarily need CONFIG_XIP,
> > > > but you do want deterministic SMP booting, and the code is a lot more understandable
> > > > with ‘#ifdef CONFIG_HART_LOTTERY’, and a Kconfig option and/or documentation
> > > > warning that says CONFIG_HART_LOTTERY depends on !CONFIG_XIP
> > > >
> > >
> > > OK, as I pointed out, the deterministic SMP booting was originally
> > > posted, but later was changed to the lottery mechanism per Anup's
> > > review comments. I am personally fine with both.
> >
> > Yes. I am agree with that.
> > But it shall also support booting form ROM and RAM as now.
>
> Just to give a bit of context information. Anup's reasoning was that
> with a fixed boot hart, we have to rely on it to work. If there is a
> hardware failure in just the boot hart, the system won't be able to
> boot anymore. With a hart lottery that is not a problem.
>
> With that in mind, it makes sense to have to option of a fixed boot
> hart, but I would not enable that option in any of the upstream
> defconfigs. That way it can be manually enabled if required for the
> application.

Sorry for pitching in late.

Apart from hardware failure on fixed boot hart, another reason in
support of lottery based system is that it will work nicely with upcoming
SBI HART power managment where more HARTs/CPUs are kept in
power-off state by OpenSBI (RUNTIME firmware) and Linux will
power-on HARTs/CPUs as required at boot-time/run-time.

Regards,
Anup