OpenBSI and U-Boot

OpenBSI and U-Boot

[Heinrich Schuchardt]

Hello Anup,

I have looking at you OpenSBI code firmware/payloads/test_head.S. Here
like in U-Boot's common/spl/spl_opensbi.c you put all but one hart in to
an enless loop (hang).

When Linux boots via UEFI it will wake up the extra harts after
ExitBootServices(). So I assume we should define function hang() in
lib/hang.c as __efi_runtime to avoid seeing it overwritten by the EFI
payload.

@Ard:
Does Linux take care of the hanging harts and redirect them to its own
routine before SetVirtualAddressMap()? Otherwise anything could happen.

On the Kendryte K210 we don't have SPL. So we will not boot in the
sequence SPL->OpenSBI->U-Boot but OpenSBI->U-Boot. Does this imply that
we have to implement the hart lottery at the entry point of main U-Boot
in this case?

Best regards

Heinrich

OpenBSI and U-Boot

[Sean Anderson]

On 8/8/20 10:59 AM, Heinrich Schuchardt wrote:
> Hello Anup,
> 
> I have looking at you OpenSBI code firmware/payloads/test_head.S. Here

I think the real start is in firmware/fw_base.S. From there, secondary
harts loop first in _wait_relocate_copy_done, and then in
_wait_for_boot_hart, and then they execute the next stage via
_start_warm and sbi_init.

> like in U-Boot's common/spl/spl_opensbi.c you put all but one hart in to
> an enless loop (hang).

As far as I can tell, U-Boot has all harts execute the next stage when
SMP is enabled. smp_call_function has all harts execute that function.

> 
> When Linux boots via UEFI it will wake up the extra harts after
> ExitBootServices(). So I assume we should define function hang() in
> lib/hang.c as __efi_runtime to avoid seeing it overwritten by the EFI
> payload.
> 
> @Ard:
> Does Linux take care of the hanging harts and redirect them to its own
> routine before SetVirtualAddressMap()? Otherwise anything could happen.
> 
> On the Kendryte K210 we don't have SPL. So we will not boot in the
> sequence SPL->OpenSBI->U-Boot but OpenSBI->U-Boot. Does this imply that
> we have to implement the hart lottery at the entry point of main U-Boot
> in this case?

Isn't the hart lottery already implemented for U-Boot? E.g. around line
100 of arch/riscv/cpu/start.S.

On another note, does Linux support S-Mode NOMMU? I was under the
impression that NOMMU implied M-Mode (or the other way around).

--Sean

OpenBSI and U-Boot

[Heinrich Schuchardt]

On 8/8/20 5:32 PM, Sean Anderson wrote:
> On 8/8/20 10:59 AM, Heinrich Schuchardt wrote:
>> Hello Anup,
>>
>> I have looking at you OpenSBI code firmware/payloads/test_head.S. Here
>
> I think the real start is in firmware/fw_base.S. From there, secondary
> harts loop first in _wait_relocate_copy_done, and then in
> _wait_for_boot_hart, and then they execute the next stage via
> _start_warm and sbi_init.
>
>> like in U-Boot's common/spl/spl_opensbi.c you put all but one hart in to
>> an enless loop (hang).
>
> As far as I can tell, U-Boot has all harts execute the next stage when
> SMP is enabled. smp_call_function has all harts execute that function.

U-Boot can only run on one hart. Are the other harts trapped in
secondary_hart_loop()? How do we ensure that an UEFI payload does not
overwrite this memory location?

spl_secondary_hart_stack_gd_setup() can jump to hang() if the call to
secondary_hart_relocate() fails.

>
>>
>> When Linux boots via UEFI it will wake up the extra harts after
>> ExitBootServices(). So I assume we should define function hang() in
>> lib/hang.c as __efi_runtime to avoid seeing it overwritten by the EFI
>> payload.
>>
>> @Ard:
>> Does Linux take care of the hanging harts and redirect them to its own
>> routine before SetVirtualAddressMap()? Otherwise anything could happen.
>>
>> On the Kendryte K210 we don't have SPL. So we will not boot in the
>> sequence SPL->OpenSBI->U-Boot but OpenSBI->U-Boot. Does this imply that
>> we have to implement the hart lottery at the entry point of main U-Boot
>> in this case?
>
> Isn't the hart lottery already implemented for U-Boot? E.g. around line
> 100 of arch/riscv/cpu/start.S.

Thanks for the hint.

>
> On another note, does Linux support S-Mode NOMMU? I was under the
> impression that NOMMU implied M-Mode (or the other way around).

Have a look at

https://linuxplumbersconf.org/event/4/contributions/386/attachments/298/502/RISC-V-NOMMU-Linux-Plumbers-2019.pdf

Best regards

Heinrich

OpenBSI and U-Boot

[Sean Anderson]

On 8/8/20 12:17 PM, Heinrich Schuchardt wrote:
> On 8/8/20 5:32 PM, Sean Anderson wrote:
>> On 8/8/20 10:59 AM, Heinrich Schuchardt wrote:
>>> Hello Anup,
>>>
>>> I have looking at you OpenSBI code firmware/payloads/test_head.S. Here
>>
>> I think the real start is in firmware/fw_base.S. From there, secondary
>> harts loop first in _wait_relocate_copy_done, and then in
>> _wait_for_boot_hart, and then they execute the next stage via
>> _start_warm and sbi_init.
>>
>>> like in U-Boot's common/spl/spl_opensbi.c you put all but one hart in to
>>> an enless loop (hang).
>>
>> As far as I can tell, U-Boot has all harts execute the next stage when
>> SMP is enabled. smp_call_function has all harts execute that function.
> 
> U-Boot can only run on one hart. Are the other harts trapped in
> secondary_hart_loop()?

Yes. They also need handle_ipi, and by extension riscv_clear_ipi. This
latter function currently requires that gd_t be valid, and may require
other structures (e.g. a struct udevice) to be valid in the future.

> How do we ensure that an UEFI payload does not overwrite this memory location?

The most foolproof is probably to wait for all harts to start running
UEFI code before making any modifications to ram outside the binary. One
easy way to do this is to use amoadd instead of amoswap (e.g. a semaphor
and not a mutex) in the standard boot lottery code. Whichever hart gets
to it first then waits for the value of hart_lottery to reach the
expected number of harts.

> spl_secondary_hart_stack_gd_setup() can jump to hang() if the call to
> secondary_hart_relocate() fails.
> 
>>
>>>
>>> When Linux boots via UEFI it will wake up the extra harts after
>>> ExitBootServices(). So I assume we should define function hang() in
>>> lib/hang.c as __efi_runtime to avoid seeing it overwritten by the EFI
>>> payload.
>>>
>>> @Ard:
>>> Does Linux take care of the hanging harts and redirect them to its own
>>> routine before SetVirtualAddressMap()? Otherwise anything could happen.
>>>
>>> On the Kendryte K210 we don't have SPL. So we will not boot in the
>>> sequence SPL->OpenSBI->U-Boot but OpenSBI->U-Boot. Does this imply that
>>> we have to implement the hart lottery at the entry point of main U-Boot
>>> in this case?
>>
>> Isn't the hart lottery already implemented for U-Boot? E.g. around line
>> 100 of arch/riscv/cpu/start.S.
> 
> Thanks for the hint.
> 
>>
>> On another note, does Linux support S-Mode NOMMU? I was under the
>> impression that NOMMU implied M-Mode (or the other way around).
> 
> Have a look at
> 
> https://linuxplumbersconf.org/event/4/contributions/386/attachments/298/502/RISC-V-NOMMU-Linux-Plumbers-2019.pdf

Ok, so NOMMU implies M-Mode. Then use of OpenSBI should preclude Linux.

--Sean

OpenBSI and U-Boot

[Heinrich Schuchardt]

On 8/8/20 7:22 PM, Sean Anderson wrote:
> On 8/8/20 12:17 PM, Heinrich Schuchardt wrote:
>> On 8/8/20 5:32 PM, Sean Anderson wrote:
>>> On 8/8/20 10:59 AM, Heinrich Schuchardt wrote:
>>>> Hello Anup,
>>>>
>>>> I have looking at you OpenSBI code firmware/payloads/test_head.S. Here
>>>
>>> I think the real start is in firmware/fw_base.S. From there, secondary
>>> harts loop first in _wait_relocate_copy_done, and then in
>>> _wait_for_boot_hart, and then they execute the next stage via
>>> _start_warm and sbi_init.
>>>
>>>> like in U-Boot's common/spl/spl_opensbi.c you put all but one hart in to
>>>> an enless loop (hang).
>>>
>>> As far as I can tell, U-Boot has all harts execute the next stage when
>>> SMP is enabled. smp_call_function has all harts execute that function.
>>
>> U-Boot can only run on one hart. Are the other harts trapped in
>> secondary_hart_loop()?
>
> Yes. They also need handle_ipi, and by extension riscv_clear_ipi. This
> latter function currently requires that gd_t be valid, and may require
> other structures (e.g. a struct udevice) to be valid in the future.
>
>> How do we ensure that an UEFI payload does not overwrite this memory location?
>
> The most foolproof is probably to wait for all harts to start running
> UEFI code before making any modifications to ram outside the binary. One
> easy way to do this is to use amoadd instead of amoswap (e.g. a semaphor
> and not a mutex) in the standard boot lottery code. Whichever hart gets
> to it first then waits for the value of hart_lottery to reach the
> expected number of harts.

There is no such requirement in the UEFI specification.

The way to tell which memory should not be overwritten by the UEFI
payload is an entry in the memory map that the payload can read via
GetMemoryMap(). So we have to make reservations in the memory map, by
calling efi_add_memory_map() or by putting the code into the
__efi_runtime section.

Do I understand it correctly that the secondary harts stay in the
unrelocated secondary_hart_loop()? In this case __efi_runtime would not
be the right section, because that memory section is also relocated and
only the relocated code is reserved in the memory map.

We would also have to consider the location of secondary_hart_loop()
when defining the load address of any payload be it UEFI or not.

Best regards

Heinrich

>
>> spl_secondary_hart_stack_gd_setup() can jump to hang() if the call to
>> secondary_hart_relocate() fails.
>>
>>>
>>>>
>>>> When Linux boots via UEFI it will wake up the extra harts after
>>>> ExitBootServices(). So I assume we should define function hang() in
>>>> lib/hang.c as __efi_runtime to avoid seeing it overwritten by the EFI
>>>> payload.
>>>>
>>>> @Ard:
>>>> Does Linux take care of the hanging harts and redirect them to its own
>>>> routine before SetVirtualAddressMap()? Otherwise anything could happen.
>>>>
>>>> On the Kendryte K210 we don't have SPL. So we will not boot in the
>>>> sequence SPL->OpenSBI->U-Boot but OpenSBI->U-Boot. Does this imply that
>>>> we have to implement the hart lottery at the entry point of main U-Boot
>>>> in this case?
>>>
>>> Isn't the hart lottery already implemented for U-Boot? E.g. around line
>>> 100 of arch/riscv/cpu/start.S.
>>
>> Thanks for the hint.
>>
>>>
>>> On another note, does Linux support S-Mode NOMMU? I was under the
>>> impression that NOMMU implied M-Mode (or the other way around).
>>
>> Have a look at
>>
>> https://linuxplumbersconf.org/event/4/contributions/386/attachments/298/502/RISC-V-NOMMU-Linux-Plumbers-2019.pdf
>
> Ok, so NOMMU implies M-Mode. Then use of OpenSBI should preclude Linux.
>
> --Sean
>

OpenBSI and U-Boot

[Sean Anderson]

On 8/8/20 2:56 PM, Heinrich Schuchardt wrote:
> On 8/8/20 7:22 PM, Sean Anderson wrote:
>> On 8/8/20 12:17 PM, Heinrich Schuchardt wrote:
>>> On 8/8/20 5:32 PM, Sean Anderson wrote:
>>>> On 8/8/20 10:59 AM, Heinrich Schuchardt wrote:
>>>>> Hello Anup,
>>>>>
>>>>> I have looking at you OpenSBI code firmware/payloads/test_head.S. Here
>>>>
>>>> I think the real start is in firmware/fw_base.S. From there, secondary
>>>> harts loop first in _wait_relocate_copy_done, and then in
>>>> _wait_for_boot_hart, and then they execute the next stage via
>>>> _start_warm and sbi_init.
>>>>
>>>>> like in U-Boot's common/spl/spl_opensbi.c you put all but one hart in to
>>>>> an enless loop (hang).
>>>>
>>>> As far as I can tell, U-Boot has all harts execute the next stage when
>>>> SMP is enabled. smp_call_function has all harts execute that function.
>>>
>>> U-Boot can only run on one hart. Are the other harts trapped in
>>> secondary_hart_loop()?
>>
>> Yes. They also need handle_ipi, and by extension riscv_clear_ipi. This
>> latter function currently requires that gd_t be valid, and may require
>> other structures (e.g. a struct udevice) to be valid in the future.
>>
>>> How do we ensure that an UEFI payload does not overwrite this memory location?
>>
>> The most foolproof is probably to wait for all harts to start running
>> UEFI code before making any modifications to ram outside the binary. One
>> easy way to do this is to use amoadd instead of amoswap (e.g. a semaphor
>> and not a mutex) in the standard boot lottery code. Whichever hart gets
>> to it first then waits for the value of hart_lottery to reach the
>> expected number of harts.
> 
> There is no such requirement in the UEFI specification.

Hm, well perhaps there should be a shim (or patch to U-Boot) which
implements such a requirement. AFAIK (and please correct me if there is
another option) there is no way to communicate between harts except by
interrupt or shared memory. Interrupts may require substantial code to
handle properly (which could be difficult to trace the requirements of).
However, shared memory only requires one or two functions to be valid,
plus the memory location itself. I think that solution has been mostly
avoided by U-Boot since the rest of U-Boot is not designed for SMP, so
there is not much infrastructure for atomics, etc.
 
> The way to tell which memory should not be overwritten by the UEFI
> payload is an entry in the memory map that the payload can read via
> GetMemoryMap(). So we have to make reservations in the memory map, by
> calling efi_add_memory_map() or by putting the code into the
> __efi_runtime section.
> 
> Do I understand it correctly that the secondary harts stay in the
> unrelocated secondary_hart_loop()? In this case __efi_runtime would not
> be the right section, because that memory section is also relocated and
> only the relocated code is reserved in the memory map.

The other harts get relocated in relocate_secondary_harts.

> 
> We would also have to consider the location of secondary_hart_loop()
> when defining the load address of any payload be it UEFI or not.

Yes. I was looking at your patch to add load addresses [1], and I'm
concerned that 80400000 may be too high for the average kernel. Since
U-Boot relocates to just below 80600000, that leaves only 1.5M or so for
the kernel to be loaded into.

--Sean

[1] https://patchwork.ozlabs.org/project/uboot/patch/20200729154235.90766-4-xypron.glpk at gmx.de/

OpenBSI and U-Boot

[Heinrich Schuchardt]

Am 9. August 2020 03:48:19 MESZ schrieb Sean Anderson <seanga2@gmail.com>:
>On 8/8/20 2:56 PM, Heinrich Schuchardt wrote:
>> On 8/8/20 7:22 PM, Sean Anderson wrote:
>>> On 8/8/20 12:17 PM, Heinrich Schuchardt wrote:
>>>> On 8/8/20 5:32 PM, Sean Anderson wrote:
>>>>> On 8/8/20 10:59 AM, Heinrich Schuchardt wrote:
>>>>>> Hello Anup,
>>>>>>
>>>>>> I have looking at you OpenSBI code firmware/payloads/test_head.S.
>Here
>>>>>
>>>>> I think the real start is in firmware/fw_base.S. From there,
>secondary
>>>>> harts loop first in _wait_relocate_copy_done, and then in
>>>>> _wait_for_boot_hart, and then they execute the next stage via
>>>>> _start_warm and sbi_init.
>>>>>
>>>>>> like in U-Boot's common/spl/spl_opensbi.c you put all but one
>hart in to
>>>>>> an enless loop (hang).
>>>>>
>>>>> As far as I can tell, U-Boot has all harts execute the next stage
>when
>>>>> SMP is enabled. smp_call_function has all harts execute that
>function.
>>>>
>>>> U-Boot can only run on one hart. Are the other harts trapped in
>>>> secondary_hart_loop()?
>>>
>>> Yes. They also need handle_ipi, and by extension riscv_clear_ipi.
>This
>>> latter function currently requires that gd_t be valid, and may
>require
>>> other structures (e.g. a struct udevice) to be valid in the future.
>>>
>>>> How do we ensure that an UEFI payload does not overwrite this
>memory location?
>>>
>>> The most foolproof is probably to wait for all harts to start
>running
>>> UEFI code before making any modifications to ram outside the binary.
>One
>>> easy way to do this is to use amoadd instead of amoswap (e.g. a
>semaphor
>>> and not a mutex) in the standard boot lottery code. Whichever hart
>gets
>>> to it first then waits for the value of hart_lottery to reach the
>>> expected number of harts.
>> 
>> There is no such requirement in the UEFI specification.
>
>Hm, well perhaps there should be a shim (or patch to U-Boot) which
>implements such a requirement. AFAIK (and please correct me if there is
>another option) there is no way to communicate between harts except by
>interrupt or shared memory. Interrupts may require substantial code to
>handle properly (which could be difficult to trace the requirements
>of).
>However, shared memory only requires one or two functions to be valid,
>plus the memory location itself. I think that solution has been mostly
>avoided by U-Boot since the rest of U-Boot is not designed for SMP, so
>there is not much infrastructure for atomics, etc.
> 
>> The way to tell which memory should not be overwritten by the UEFI
>> payload is an entry in the memory map that the payload can read via
>> GetMemoryMap(). So we have to make reservations in the memory map, by
>> calling efi_add_memory_map() or by putting the code into the
>> __efi_runtime section.
>> 
>> Do I understand it correctly that the secondary harts stay in the
>> unrelocated secondary_hart_loop()? In this case __efi_runtime would
>not
>> be the right section, because that memory section is also relocated
>and
>> only the relocated code is reserved in the memory map.
>
>The other harts get relocated in relocate_secondary_harts.
>
>> 
>> We would also have to consider the location of secondary_hart_loop()
>> when defining the load address of any payload be it UEFI or not.
>
>Yes. I was looking at your patch to add load addresses [1], and I'm
>concerned that 80400000 may be too high for the average kernel. Since
>U-Boot relocates to just below 80600000, that leaves only 1.5M or so
>for
>the kernel to be loaded into.

The patch has:

kernel_addr_r=0x80040000

You somehow missed the position of "4".

>
>--Sean
>
>[1]
>https://patchwork.ozlabs.org/project/uboot/patch/20200729154235.90766-4-xypron.glpk at gmx.de/

OpenBSI and U-Boot

[Sean Anderson]

On 8/9/20 12:23 AM, Heinrich Schuchardt wrote:
> Am 9. August 2020 03:48:19 MESZ schrieb Sean Anderson <seanga2@gmail.com>:
>> On 8/8/20 2:56 PM, Heinrich Schuchardt wrote:
>>> On 8/8/20 7:22 PM, Sean Anderson wrote:
>>>> On 8/8/20 12:17 PM, Heinrich Schuchardt wrote:
>>>>> On 8/8/20 5:32 PM, Sean Anderson wrote:
>>>>>> On 8/8/20 10:59 AM, Heinrich Schuchardt wrote:
>>>>>>> Hello Anup,
>>>>>>>
>>>>>>> I have looking at you OpenSBI code firmware/payloads/test_head.S.
>> Here
>>>>>>
>>>>>> I think the real start is in firmware/fw_base.S. From there,
>> secondary
>>>>>> harts loop first in _wait_relocate_copy_done, and then in
>>>>>> _wait_for_boot_hart, and then they execute the next stage via
>>>>>> _start_warm and sbi_init.
>>>>>>
>>>>>>> like in U-Boot's common/spl/spl_opensbi.c you put all but one
>> hart in to
>>>>>>> an enless loop (hang).
>>>>>>
>>>>>> As far as I can tell, U-Boot has all harts execute the next stage
>> when
>>>>>> SMP is enabled. smp_call_function has all harts execute that
>> function.
>>>>>
>>>>> U-Boot can only run on one hart. Are the other harts trapped in
>>>>> secondary_hart_loop()?
>>>>
>>>> Yes. They also need handle_ipi, and by extension riscv_clear_ipi.
>> This
>>>> latter function currently requires that gd_t be valid, and may
>> require
>>>> other structures (e.g. a struct udevice) to be valid in the future.
>>>>
>>>>> How do we ensure that an UEFI payload does not overwrite this
>> memory location?
>>>>
>>>> The most foolproof is probably to wait for all harts to start
>> running
>>>> UEFI code before making any modifications to ram outside the binary.
>> One
>>>> easy way to do this is to use amoadd instead of amoswap (e.g. a
>> semaphor
>>>> and not a mutex) in the standard boot lottery code. Whichever hart
>> gets
>>>> to it first then waits for the value of hart_lottery to reach the
>>>> expected number of harts.
>>>
>>> There is no such requirement in the UEFI specification.
>>
>> Hm, well perhaps there should be a shim (or patch to U-Boot) which
>> implements such a requirement. AFAIK (and please correct me if there is
>> another option) there is no way to communicate between harts except by
>> interrupt or shared memory. Interrupts may require substantial code to
>> handle properly (which could be difficult to trace the requirements
>> of).
>> However, shared memory only requires one or two functions to be valid,
>> plus the memory location itself. I think that solution has been mostly
>> avoided by U-Boot since the rest of U-Boot is not designed for SMP, so
>> there is not much infrastructure for atomics, etc.
>>
>>> The way to tell which memory should not be overwritten by the UEFI
>>> payload is an entry in the memory map that the payload can read via
>>> GetMemoryMap(). So we have to make reservations in the memory map, by
>>> calling efi_add_memory_map() or by putting the code into the
>>> __efi_runtime section.
>>>
>>> Do I understand it correctly that the secondary harts stay in the
>>> unrelocated secondary_hart_loop()? In this case __efi_runtime would
>> not
>>> be the right section, because that memory section is also relocated
>> and
>>> only the relocated code is reserved in the memory map.
>>
>> The other harts get relocated in relocate_secondary_harts.
>>
>>>
>>> We would also have to consider the location of secondary_hart_loop()
>>> when defining the load address of any payload be it UEFI or not.
>>
>> Yes. I was looking at your patch to add load addresses [1], and I'm
>> concerned that 80400000 may be too high for the average kernel. Since
>> U-Boot relocates to just below 80600000, that leaves only 1.5M or so
>> for
>> the kernel to be loaded into.
> 
> The patch has:
> 
> kernel_addr_r=0x80040000
> 
> You somehow missed the position of "4".

Ah, so I did. That looks fine then.

--Sean

OpenBSI and U-Boot

[Heinrich Schuchardt]

On 8/9/20 3:48 AM, Sean Anderson wrote:
> On 8/8/20 2:56 PM, Heinrich Schuchardt wrote:
>> On 8/8/20 7:22 PM, Sean Anderson wrote:
>>> On 8/8/20 12:17 PM, Heinrich Schuchardt wrote:
>>>> On 8/8/20 5:32 PM, Sean Anderson wrote:
>>>>> On 8/8/20 10:59 AM, Heinrich Schuchardt wrote:
>>>>>> Hello Anup,
>>>>>>
>>>>>> I have looking at you OpenSBI code firmware/payloads/test_head.S. Here
>>>>>
>>>>> I think the real start is in firmware/fw_base.S. From there, secondary
>>>>> harts loop first in _wait_relocate_copy_done, and then in
>>>>> _wait_for_boot_hart, and then they execute the next stage via
>>>>> _start_warm and sbi_init.
>>>>>
>>>>>> like in U-Boot's common/spl/spl_opensbi.c you put all but one hart in to
>>>>>> an enless loop (hang).
>>>>>
>>>>> As far as I can tell, U-Boot has all harts execute the next stage when
>>>>> SMP is enabled. smp_call_function has all harts execute that function.
>>>>
>>>> U-Boot can only run on one hart. Are the other harts trapped in
>>>> secondary_hart_loop()?
>>>
>>> Yes. They also need handle_ipi, and by extension riscv_clear_ipi. This
>>> latter function currently requires that gd_t be valid, and may require
>>> other structures (e.g. a struct udevice) to be valid in the future.
>>>
>>>> How do we ensure that an UEFI payload does not overwrite this memory location?
>>>
>>> The most foolproof is probably to wait for all harts to start running
>>> UEFI code before making any modifications to ram outside the binary. One
>>> easy way to do this is to use amoadd instead of amoswap (e.g. a semaphor
>>> and not a mutex) in the standard boot lottery code. Whichever hart gets
>>> to it first then waits for the value of hart_lottery to reach the
>>> expected number of harts.
>>
>> There is no such requirement in the UEFI specification.
>
> Hm, well perhaps there should be a shim (or patch to U-Boot) which
> implements such a requirement. AFAIK (and please correct me if there is
> another option) there is no way to communicate between harts except by
> interrupt or shared memory. Interrupts may require substantial code to
> handle properly (which could be difficult to trace the requirements of).
> However, shared memory only requires one or two functions to be valid,
> plus the memory location itself. I think that solution has been mostly
> avoided by U-Boot since the rest of U-Boot is not designed for SMP, so
> there is not much infrastructure for atomics, etc.
>
>> The way to tell which memory should not be overwritten by the UEFI
>> payload is an entry in the memory map that the payload can read via
>> GetMemoryMap(). So we have to make reservations in the memory map, by
>> calling efi_add_memory_map() or by putting the code into the
>> __efi_runtime section.
>>
>> Do I understand it correctly that the secondary harts stay in the
>> unrelocated secondary_hart_loop()? In this case __efi_runtime would not
>> be the right section, because that memory section is also relocated and
>> only the relocated code is reserved in the memory map.
>
> The other harts get relocated in relocate_secondary_harts.

I added the following:

 secondary_hart_loop:
+
+       lui     a5,0x20040
+       addi    a5,a5,0x48c
+       slli    a5,a5,0x2
+       /* a5 = 80101230 */
+       auipc   a4, 0
+       sw      a4,0(a5)
+       lw      a4,8(a5)
+       addw    a4,a4,1
+       sw      a4,8(a5)
+

With the mm.w command I can see what is stored at 0x80101230 - 0x8010123f.

On the MaixDuino secondary_hart_loop() is executed 1 to 2 times.
Shouldn't it be running all the time?

relocate_secondary_harts() is called once.

hang() is never executed.

Any idea what the secondary hart is executing after
relocate_secondary_harts()?

Best regards

Heinrich

>
>>
>> We would also have to consider the location of secondary_hart_loop()
>> when defining the load address of any payload be it UEFI or not.
>
> Yes. I was looking at your patch to add load addresses [1], and I'm
> concerned that 80400000 may be too high for the average kernel. Since
> U-Boot relocates to just below 80600000, that leaves only 1.5M or so for
> the kernel to be loaded into.
>
> --Sean
>
> [1] https://patchwork.ozlabs.org/project/uboot/patch/20200729154235.90766-4-xypron.glpk at gmx.de/
>

OpenBSI and U-Boot

[Atish Patra]

On Sat, Aug 8, 2020 at 7:59 AM Heinrich Schuchardt <xypron.glpk@gmx.de> wrote:
>
> Hello Anup,
>
> I have looking at you OpenSBI code firmware/payloads/test_head.S. Here
> like in U-Boot's common/spl/spl_opensbi.c you put all but one hart in to
> an enless loop (hang).
>
> When Linux boots via UEFI it will wake up the extra harts after
> ExitBootServices(). So I assume we should define function hang() in
> lib/hang.c as __efi_runtime to avoid seeing it overwritten by the EFI
> payload.
>
Proper Linux will wake up non-boot harts through SBI HSM APIs.
Until then all other harts are held on OpenSBI in sbi_hsm_wait() [1].

[1] https://github.com/riscv/opensbi/blob/master/lib/sbi/sbi_hsm.c#L133

Here are the details on SBI HSM extension if you are interested.
https://fosdem.org/2020/schedule/event/riscv_bootflow/

> @Ard:
> Does Linux take care of the hanging harts and redirect them to its own
> routine before SetVirtualAddressMap()? Otherwise anything could happen.
>

Linux doesn't need to as SetVirtualAddressMap is part of efistub which
is  only executed in the booting hart.

> On the Kendryte K210 we don't have SPL. So we will not boot in the
> sequence SPL->OpenSBI->U-Boot but OpenSBI->U-Boot. Does this imply that
> we have to implement the hart lottery at the entry point of main U-Boot
> in this case?
>

Do you want OpenSBI->U-Boot or OpenSBI->U-Boot->Linux for kendryte ?
Is this related to testing U-Boot EFI on Kendryte right ?

> Best regards
>
> Heinrich



-- 
Regards,
Atish

OpenBSI and U-Boot

[Atish Patra]

On Sat, Aug 8, 2020 at 9:17 AM Heinrich Schuchardt <xypron.glpk@gmx.de> wrote:
>
> On 8/8/20 5:32 PM, Sean Anderson wrote:
> > On 8/8/20 10:59 AM, Heinrich Schuchardt wrote:
> >> Hello Anup,
> >>
> >> I have looking at you OpenSBI code firmware/payloads/test_head.S. Here
> >
> > I think the real start is in firmware/fw_base.S. From there, secondary
> > harts loop first in _wait_relocate_copy_done, and then in
> > _wait_for_boot_hart, and then they execute the next stage via
> > _start_warm and sbi_init.
> >
> >> like in U-Boot's common/spl/spl_opensbi.c you put all but one hart in to
> >> an enless loop (hang).
> >
> > As far as I can tell, U-Boot has all harts execute the next stage when
> > SMP is enabled. smp_call_function has all harts execute that function.
>
> U-Boot can only run on one hart. Are the other harts trapped in
> secondary_hart_loop()? How do we ensure that an UEFI payload does not
> overwrite this memory location?
>

If you are booting Linux, U-Boot runs in S-mode and SMP is disabled.
All memory regions containing OpenSBI code are reserved. Thus, U-Boot
won't touch that.
U-Boot also marks the run time services memory region as reserved as well.
Thus, Linux kernel won't touch any of those memory regions either.

> spl_secondary_hart_stack_gd_setup() can jump to hang() if the call to
> secondary_hart_relocate() fails.
>
> >
> >>
> >> When Linux boots via UEFI it will wake up the extra harts after
> >> ExitBootServices(). So I assume we should define function hang() in
> >> lib/hang.c as __efi_runtime to avoid seeing it overwritten by the EFI
> >> payload.
> >>
> >> @Ard:
> >> Does Linux take care of the hanging harts and redirect them to its own
> >> routine before SetVirtualAddressMap()? Otherwise anything could happen.
> >>
> >> On the Kendryte K210 we don't have SPL. So we will not boot in the
> >> sequence SPL->OpenSBI->U-Boot but OpenSBI->U-Boot. Does this imply that
> >> we have to implement the hart lottery at the entry point of main U-Boot
> >> in this case?
> >
> > Isn't the hart lottery already implemented for U-Boot? E.g. around line
> > 100 of arch/riscv/cpu/start.S.
>
> Thanks for the hint.
>
> >
> > On another note, does Linux support S-Mode NOMMU? I was under the
> > impression that NOMMU implied M-Mode (or the other way around).
>
> Have a look at
>
> https://linuxplumbersconf.org/event/4/contributions/386/attachments/298/502/RISC-V-NOMMU-Linux-Plumbers-2019.pdf
>
> Best regards
>
> Heinrich



-- 
Regards,
Atish

OpenBSI and U-Boot

[Heinrich Schuchardt]

Am 9. August 2020 22:08:23 MESZ schrieb Atish Patra <atishp@atishpatra.org>:
>On Sat, Aug 8, 2020 at 9:17 AM Heinrich Schuchardt <xypron.glpk@gmx.de>
>wrote:
>>
>> On 8/8/20 5:32 PM, Sean Anderson wrote:
>> > On 8/8/20 10:59 AM, Heinrich Schuchardt wrote:
>> >> Hello Anup,
>> >>
>> >> I have looking at you OpenSBI code firmware/payloads/test_head.S.
>Here
>> >
>> > I think the real start is in firmware/fw_base.S. From there,
>secondary
>> > harts loop first in _wait_relocate_copy_done, and then in
>> > _wait_for_boot_hart, and then they execute the next stage via
>> > _start_warm and sbi_init.
>> >
>> >> like in U-Boot's common/spl/spl_opensbi.c you put all but one hart
>in to
>> >> an enless loop (hang).
>> >
>> > As far as I can tell, U-Boot has all harts execute the next stage
>when
>> > SMP is enabled. smp_call_function has all harts execute that
>function.
>>
>> U-Boot can only run on one hart. Are the other harts trapped in
>> secondary_hart_loop()? How do we ensure that an UEFI payload does not
>> overwrite this memory location?
>>
>
>If you are booting Linux, U-Boot runs in S-mode and SMP is disabled.

Would that also hold true on the Kendryte K210? For all what can see the secondary hart enters U-Boot and is only restrained by WFI and otherwise kept in an endless loop.

I am wondering if that endless loop needs to be marked as reserved memory for Linux.

>All memory regions containing OpenSBI code are reserved. Thus, U-Boot
>won't touch that.
>U-Boot also marks the run time services memory region as reserved as
>well.
>Thus, Linux kernel won't touch any of those memory regions either.
>
>> spl_secondary_hart_stack_gd_setup() can jump to hang() if the call to
>> secondary_hart_relocate() fails.
>>
>> >
>> >>
>> >> When Linux boots via UEFI it will wake up the extra harts after
>> >> ExitBootServices(). So I assume we should define function hang()
>in
>> >> lib/hang.c as __efi_runtime to avoid seeing it overwritten by the
>EFI
>> >> payload.
>> >>
>> >> @Ard:
>> >> Does Linux take care of the hanging harts and redirect them to its
>own
>> >> routine before SetVirtualAddressMap()? Otherwise anything could
>happen.
>> >>
>> >> On the Kendryte K210 we don't have SPL. So we will not boot in the
>> >> sequence SPL->OpenSBI->U-Boot but OpenSBI->U-Boot. Does this imply
>that
>> >> we have to implement the hart lottery at the entry point of main
>U-Boot
>> >> in this case?
>> >
>> > Isn't the hart lottery already implemented for U-Boot? E.g. around
>line
>> > 100 of arch/riscv/cpu/start.S.
>>
>> Thanks for the hint.
>>
>> >
>> > On another note, does Linux support S-Mode NOMMU? I was under the
>> > impression that NOMMU implied M-Mode (or the other way around).
>>
>> Have a look at
>>
>>
>https://linuxplumbersconf.org/event/4/contributions/386/attachments/298/502/RISC-V-NOMMU-Linux-Plumbers-2019.pdf
>>
>> Best regards
>>
>> Heinrich

OpenBSI and U-Boot

[Atish Patra]

On Sun, Aug 9, 2020 at 1:28 PM Heinrich Schuchardt <xypron.glpk@gmx.de> wrote:
>
> Am 9. August 2020 22:08:23 MESZ schrieb Atish Patra <atishp@atishpatra.org>:
> >On Sat, Aug 8, 2020 at 9:17 AM Heinrich Schuchardt <xypron.glpk@gmx.de>
> >wrote:
> >>
> >> On 8/8/20 5:32 PM, Sean Anderson wrote:
> >> > On 8/8/20 10:59 AM, Heinrich Schuchardt wrote:
> >> >> Hello Anup,
> >> >>
> >> >> I have looking at you OpenSBI code firmware/payloads/test_head.S.
> >Here
> >> >
> >> > I think the real start is in firmware/fw_base.S. From there,
> >secondary
> >> > harts loop first in _wait_relocate_copy_done, and then in
> >> > _wait_for_boot_hart, and then they execute the next stage via
> >> > _start_warm and sbi_init.
> >> >
> >> >> like in U-Boot's common/spl/spl_opensbi.c you put all but one hart
> >in to
> >> >> an enless loop (hang).
> >> >
> >> > As far as I can tell, U-Boot has all harts execute the next stage
> >when
> >> > SMP is enabled. smp_call_function has all harts execute that
> >function.
> >>
> >> U-Boot can only run on one hart. Are the other harts trapped in
> >> secondary_hart_loop()? How do we ensure that an UEFI payload does not
> >> overwrite this memory location?
> >>
> >
> >If you are booting Linux, U-Boot runs in S-mode and SMP is disabled.
>
> Would that also hold true on the Kendryte K210? For all what can see the secondary hart enters U-Boot and is only restrained by WFI and otherwise kept in an endless loop.
>
> I am wondering if that endless loop needs to be marked as reserved memory for Linux.
>

Is U-boot running in S-mode or M-mode ?
Are you trying to boot Linux on a Kendryte board using OpenSBI
->U-Boot -> Linux? (all in M-mode ?)

> >All memory regions containing OpenSBI code are reserved. Thus, U-Boot
> >won't touch that.
> >U-Boot also marks the run time services memory region as reserved as
> >well.
> >Thus, Linux kernel won't touch any of those memory regions either.
> >
> >> spl_secondary_hart_stack_gd_setup() can jump to hang() if the call to
> >> secondary_hart_relocate() fails.
> >>
> >> >
> >> >>
> >> >> When Linux boots via UEFI it will wake up the extra harts after
> >> >> ExitBootServices(). So I assume we should define function hang()
> >in
> >> >> lib/hang.c as __efi_runtime to avoid seeing it overwritten by the
> >EFI
> >> >> payload.
> >> >>
> >> >> @Ard:
> >> >> Does Linux take care of the hanging harts and redirect them to its
> >own
> >> >> routine before SetVirtualAddressMap()? Otherwise anything could
> >happen.
> >> >>
> >> >> On the Kendryte K210 we don't have SPL. So we will not boot in the
> >> >> sequence SPL->OpenSBI->U-Boot but OpenSBI->U-Boot. Does this imply
> >that
> >> >> we have to implement the hart lottery at the entry point of main
> >U-Boot
> >> >> in this case?
> >> >
> >> > Isn't the hart lottery already implemented for U-Boot? E.g. around
> >line
> >> > 100 of arch/riscv/cpu/start.S.
> >>
> >> Thanks for the hint.
> >>
> >> >
> >> > On another note, does Linux support S-Mode NOMMU? I was under the
> >> > impression that NOMMU implied M-Mode (or the other way around).
> >>
> >> Have a look at
> >>
> >>
> >https://linuxplumbersconf.org/event/4/contributions/386/attachments/298/502/RISC-V-NOMMU-Linux-Plumbers-2019.pdf
> >>
> >> Best regards
> >>
> >> Heinrich
>


-- 
Regards,
Atish

OpenBSI and U-Boot

[Rick Chen]

Hi Heinrich

> Am 9. August 2020 22:08:23 MESZ schrieb Atish Patra <atishp@atishpatra.org>:
> >On Sat, Aug 8, 2020 at 9:17 AM Heinrich Schuchardt <xypron.glpk@gmx.de>
> >wrote:
> >>
> >> On 8/8/20 5:32 PM, Sean Anderson wrote:
> >> > On 8/8/20 10:59 AM, Heinrich Schuchardt wrote:
> >> >> Hello Anup,
> >> >>
> >> >> I have looking at you OpenSBI code firmware/payloads/test_head.S.
> >Here
> >> >
> >> > I think the real start is in firmware/fw_base.S. From there,
> >secondary
> >> > harts loop first in _wait_relocate_copy_done, and then in
> >> > _wait_for_boot_hart, and then they execute the next stage via
> >> > _start_warm and sbi_init.
> >> >
> >> >> like in U-Boot's common/spl/spl_opensbi.c you put all but one hart
> >in to
> >> >> an enless loop (hang).
> >> >
> >> > As far as I can tell, U-Boot has all harts execute the next stage
> >when
> >> > SMP is enabled. smp_call_function has all harts execute that
> >function.
> >>
> >> U-Boot can only run on one hart. Are the other harts trapped in
> >> secondary_hart_loop()? How do we ensure that an UEFI payload does not
> >> overwrite this memory location?
> >>
> >
> >If you are booting Linux, U-Boot runs in S-mode and SMP is disabled.
>
> Would that also hold true on the Kendryte K210? For all what can see the secondary hart enters U-Boot and is only restrained by WFI and otherwise kept in an endless loop.
>
> I am wondering if that endless loop needs to be marked as reserved memory for Linux.

Currently if U-Boot run in S-Mode, SMP is disable, there will exist
OpenSBI version compatible issue.
You shall use OpenSBI v0.7 with HSM, thus it will trap the other harts
in OpenSBI and only main hart will jump to kernel from U-Boot proper.

Thanks,
Rick

>
> >All memory regions containing OpenSBI code are reserved. Thus, U-Boot
> >won't touch that.
> >U-Boot also marks the run time services memory region as reserved as
> >well.
> >Thus, Linux kernel won't touch any of those memory regions either.
> >
> >> spl_secondary_hart_stack_gd_setup() can jump to hang() if the call to
> >> secondary_hart_relocate() fails.
> >>
> >> >
> >> >>
> >> >> When Linux boots via UEFI it will wake up the extra harts after
> >> >> ExitBootServices(). So I assume we should define function hang()
> >in
> >> >> lib/hang.c as __efi_runtime to avoid seeing it overwritten by the
> >EFI
> >> >> payload.
> >> >>
> >> >> @Ard:
> >> >> Does Linux take care of the hanging harts and redirect them to its
> >own
> >> >> routine before SetVirtualAddressMap()? Otherwise anything could
> >happen.
> >> >>
> >> >> On the Kendryte K210 we don't have SPL. So we will not boot in the
> >> >> sequence SPL->OpenSBI->U-Boot but OpenSBI->U-Boot. Does this imply
> >that
> >> >> we have to implement the hart lottery at the entry point of main
> >U-Boot
> >> >> in this case?
> >> >
> >> > Isn't the hart lottery already implemented for U-Boot? E.g. around
> >line
> >> > 100 of arch/riscv/cpu/start.S.
> >>
> >> Thanks for the hint.
> >>
> >> >
> >> > On another note, does Linux support S-Mode NOMMU? I was under the
> >> > impression that NOMMU implied M-Mode (or the other way around).
> >>
> >> Have a look at
> >>
> >>
> >https://linuxplumbersconf.org/event/4/contributions/386/attachments/298/502/RISC-V-NOMMU-Linux-Plumbers-2019.pdf
> >>
> >> Best regards
> >>
> >> Heinrich
>

OpenBSI and U-Boot

[Heinrich Schuchardt]

On 8/11/20 3:55 AM, Rick Chen wrote:
> Hi Heinrich
>
>> Am 9. August 2020 22:08:23 MESZ schrieb Atish Patra <atishp@atishpatra.org>:
>>> On Sat, Aug 8, 2020 at 9:17 AM Heinrich Schuchardt <xypron.glpk@gmx.de>
>>> wrote:
>>>>
>>>> On 8/8/20 5:32 PM, Sean Anderson wrote:
>>>>> On 8/8/20 10:59 AM, Heinrich Schuchardt wrote:
>>>>>> Hello Anup,
>>>>>>
>>>>>> I have looking at you OpenSBI code firmware/payloads/test_head.S.
>>> Here
>>>>>
>>>>> I think the real start is in firmware/fw_base.S. From there,
>>> secondary
>>>>> harts loop first in _wait_relocate_copy_done, and then in
>>>>> _wait_for_boot_hart, and then they execute the next stage via
>>>>> _start_warm and sbi_init.
>>>>>
>>>>>> like in U-Boot's common/spl/spl_opensbi.c you put all but one hart
>>> in to
>>>>>> an enless loop (hang).
>>>>>
>>>>> As far as I can tell, U-Boot has all harts execute the next stage
>>> when
>>>>> SMP is enabled. smp_call_function has all harts execute that
>>> function.
>>>>
>>>> U-Boot can only run on one hart. Are the other harts trapped in
>>>> secondary_hart_loop()? How do we ensure that an UEFI payload does not
>>>> overwrite this memory location?
>>>>
>>>
>>> If you are booting Linux, U-Boot runs in S-mode and SMP is disabled.
>>
>> Would that also hold true on the Kendryte K210? For all what can see the secondary hart enters U-Boot and is only restrained by WFI and otherwise kept in an endless loop.
>>
>> I am wondering if that endless loop needs to be marked as reserved memory for Linux.
>
> Currently if U-Boot run in S-Mode, SMP is disable, there will exist
> OpenSBI version compatible issue.
> You shall use OpenSBI v0.7 with HSM, thus it will trap the other harts
> in OpenSBI and only main hart will jump to kernel from U-Boot proper.
>
> Thanks,
> Rick
>

HSM is described here:
https://github.com/riscv/riscv-sbi-doc/blob/master/riscv-sbi.adoc#hart-state-management-extension-extension-id-0x48534d-hsm

Currently the Kendryte K210 has no SPL. So OpenSBI would be running
before U-Boot.

Will OpenSBI v0.7 by itself stop the other harts or do we need to call
sbi_hart_stop() in U-Boot? Unfortunately opensbi/README.md and
riscv-sbi.adoc both leave this open.

Best regards

Heinrich

>>
>>> All memory regions containing OpenSBI code are reserved. Thus, U-Boot
>>> won't touch that.
>>> U-Boot also marks the run time services memory region as reserved as
>>> well.
>>> Thus, Linux kernel won't touch any of those memory regions either.
>>>
>>>> spl_secondary_hart_stack_gd_setup() can jump to hang() if the call to
>>>> secondary_hart_relocate() fails.
>>>>
>>>>>
>>>>>>
>>>>>> When Linux boots via UEFI it will wake up the extra harts after
>>>>>> ExitBootServices(). So I assume we should define function hang()
>>> in
>>>>>> lib/hang.c as __efi_runtime to avoid seeing it overwritten by the
>>> EFI
>>>>>> payload.
>>>>>>
>>>>>> @Ard:
>>>>>> Does Linux take care of the hanging harts and redirect them to its
>>> own
>>>>>> routine before SetVirtualAddressMap()? Otherwise anything could
>>> happen.
>>>>>>
>>>>>> On the Kendryte K210 we don't have SPL. So we will not boot in the
>>>>>> sequence SPL->OpenSBI->U-Boot but OpenSBI->U-Boot. Does this imply
>>> that
>>>>>> we have to implement the hart lottery at the entry point of main
>>> U-Boot
>>>>>> in this case?
>>>>>
>>>>> Isn't the hart lottery already implemented for U-Boot? E.g. around
>>> line
>>>>> 100 of arch/riscv/cpu/start.S.
>>>>
>>>> Thanks for the hint.
>>>>
>>>>>
>>>>> On another note, does Linux support S-Mode NOMMU? I was under the
>>>>> impression that NOMMU implied M-Mode (or the other way around).
>>>>
>>>> Have a look at
>>>>
>>>>
>>> https://linuxplumbersconf.org/event/4/contributions/386/attachments/298/502/RISC-V-NOMMU-Linux-Plumbers-2019.pdf
>>>>
>>>> Best regards
>>>>
>>>> Heinrich
>>

OpenBSI and U-Boot

[Atish Patra]

On Mon, Aug 10, 2020 at 10:30 PM Heinrich Schuchardt <xypron.glpk@gmx.de> wrote:
>
> On 8/11/20 3:55 AM, Rick Chen wrote:
> > Hi Heinrich
> >
> >> Am 9. August 2020 22:08:23 MESZ schrieb Atish Patra <atishp@atishpatra.org>:
> >>> On Sat, Aug 8, 2020 at 9:17 AM Heinrich Schuchardt <xypron.glpk@gmx.de>
> >>> wrote:
> >>>>
> >>>> On 8/8/20 5:32 PM, Sean Anderson wrote:
> >>>>> On 8/8/20 10:59 AM, Heinrich Schuchardt wrote:
> >>>>>> Hello Anup,
> >>>>>>
> >>>>>> I have looking at you OpenSBI code firmware/payloads/test_head.S.
> >>> Here
> >>>>>
> >>>>> I think the real start is in firmware/fw_base.S. From there,
> >>> secondary
> >>>>> harts loop first in _wait_relocate_copy_done, and then in
> >>>>> _wait_for_boot_hart, and then they execute the next stage via
> >>>>> _start_warm and sbi_init.
> >>>>>
> >>>>>> like in U-Boot's common/spl/spl_opensbi.c you put all but one hart
> >>> in to
> >>>>>> an enless loop (hang).
> >>>>>
> >>>>> As far as I can tell, U-Boot has all harts execute the next stage
> >>> when
> >>>>> SMP is enabled. smp_call_function has all harts execute that
> >>> function.
> >>>>
> >>>> U-Boot can only run on one hart. Are the other harts trapped in
> >>>> secondary_hart_loop()? How do we ensure that an UEFI payload does not
> >>>> overwrite this memory location?
> >>>>
> >>>
> >>> If you are booting Linux, U-Boot runs in S-mode and SMP is disabled.
> >>
> >> Would that also hold true on the Kendryte K210? For all what can see the secondary hart enters U-Boot and is only restrained by WFI and otherwise kept in an endless loop.
> >>
> >> I am wondering if that endless loop needs to be marked as reserved memory for Linux.
> >
> > Currently if U-Boot run in S-Mode, SMP is disable, there will exist
> > OpenSBI version compatible issue.
> > You shall use OpenSBI v0.7 with HSM, thus it will trap the other harts
> > in OpenSBI and only main hart will jump to kernel from U-Boot proper.
> >
> > Thanks,
> > Rick
> >
>
> HSM is described here:
> https://github.com/riscv/riscv-sbi-doc/blob/master/riscv-sbi.adoc#hart-state-management-extension-extension-id-0x48534d-hsm
>
> Currently the Kendryte K210 has no SPL. So OpenSBI would be running
> before U-Boot.
>
> Will OpenSBI v0.7 by itself stop the other harts or do we need to call
> sbi_hart_stop() in U-Boot? Unfortunately opensbi/README.md and
> riscv-sbi.adoc both leave this open.
>

OpenSBI will put all non-booting harts into wfi in sbi_hsm_wait
https://github.com/riscv/opensbi/blob/master/lib/sbi/sbi_hsm.c#L119

The S-mode OS is required to call sbi_hsm_start for all non-booting harts
so that they enter S-mode.

Are you planning to add HSM extension just for Kendryte ?

> Best regards
>
> Heinrich
>
> >>
> >>> All memory regions containing OpenSBI code are reserved. Thus, U-Boot
> >>> won't touch that.
> >>> U-Boot also marks the run time services memory region as reserved as
> >>> well.
> >>> Thus, Linux kernel won't touch any of those memory regions either.
> >>>
> >>>> spl_secondary_hart_stack_gd_setup() can jump to hang() if the call to
> >>>> secondary_hart_relocate() fails.
> >>>>
> >>>>>
> >>>>>>
> >>>>>> When Linux boots via UEFI it will wake up the extra harts after
> >>>>>> ExitBootServices(). So I assume we should define function hang()
> >>> in
> >>>>>> lib/hang.c as __efi_runtime to avoid seeing it overwritten by the
> >>> EFI
> >>>>>> payload.
> >>>>>>
> >>>>>> @Ard:
> >>>>>> Does Linux take care of the hanging harts and redirect them to its
> >>> own
> >>>>>> routine before SetVirtualAddressMap()? Otherwise anything could
> >>> happen.
> >>>>>>
> >>>>>> On the Kendryte K210 we don't have SPL. So we will not boot in the
> >>>>>> sequence SPL->OpenSBI->U-Boot but OpenSBI->U-Boot. Does this imply
> >>> that
> >>>>>> we have to implement the hart lottery at the entry point of main
> >>> U-Boot
> >>>>>> in this case?
> >>>>>
> >>>>> Isn't the hart lottery already implemented for U-Boot? E.g. around
> >>> line
> >>>>> 100 of arch/riscv/cpu/start.S.
> >>>>
> >>>> Thanks for the hint.
> >>>>
> >>>>>
> >>>>> On another note, does Linux support S-Mode NOMMU? I was under the
> >>>>> impression that NOMMU implied M-Mode (or the other way around).
> >>>>
> >>>> Have a look at
> >>>>
> >>>>
> >>> https://linuxplumbersconf.org/event/4/contributions/386/attachments/298/502/RISC-V-NOMMU-Linux-Plumbers-2019.pdf
> >>>>
> >>>> Best regards
> >>>>
> >>>> Heinrich
> >>
>


-- 
Regards,
Atish

OpenBSI and U-Boot

[Rick Chen]

Hi Atish

> On Mon, Aug 10, 2020 at 10:30 PM Heinrich Schuchardt <xypron.glpk@gmx.de> wrote:
> >
> > On 8/11/20 3:55 AM, Rick Chen wrote:
> > > Hi Heinrich
> > >
> > >> Am 9. August 2020 22:08:23 MESZ schrieb Atish Patra <atishp@atishpatra.org>:
> > >>> On Sat, Aug 8, 2020 at 9:17 AM Heinrich Schuchardt <xypron.glpk@gmx.de>
> > >>> wrote:
> > >>>>
> > >>>> On 8/8/20 5:32 PM, Sean Anderson wrote:
> > >>>>> On 8/8/20 10:59 AM, Heinrich Schuchardt wrote:
> > >>>>>> Hello Anup,
> > >>>>>>
> > >>>>>> I have looking at you OpenSBI code firmware/payloads/test_head.S.
> > >>> Here
> > >>>>>
> > >>>>> I think the real start is in firmware/fw_base.S. From there,
> > >>> secondary
> > >>>>> harts loop first in _wait_relocate_copy_done, and then in
> > >>>>> _wait_for_boot_hart, and then they execute the next stage via
> > >>>>> _start_warm and sbi_init.
> > >>>>>
> > >>>>>> like in U-Boot's common/spl/spl_opensbi.c you put all but one hart
> > >>> in to
> > >>>>>> an enless loop (hang).
> > >>>>>
> > >>>>> As far as I can tell, U-Boot has all harts execute the next stage
> > >>> when
> > >>>>> SMP is enabled. smp_call_function has all harts execute that
> > >>> function.
> > >>>>
> > >>>> U-Boot can only run on one hart. Are the other harts trapped in
> > >>>> secondary_hart_loop()? How do we ensure that an UEFI payload does not
> > >>>> overwrite this memory location?
> > >>>>
> > >>>
> > >>> If you are booting Linux, U-Boot runs in S-mode and SMP is disabled.
> > >>
> > >> Would that also hold true on the Kendryte K210? For all what can see the secondary hart enters U-Boot and is only restrained by WFI and otherwise kept in an endless loop.
> > >>
> > >> I am wondering if that endless loop needs to be marked as reserved memory for Linux.
> > >
> > > Currently if U-Boot run in S-Mode, SMP is disable, there will exist
> > > OpenSBI version compatible issue.
> > > You shall use OpenSBI v0.7 with HSM, thus it will trap the other harts
> > > in OpenSBI and only main hart will jump to kernel from U-Boot proper.
> > >
> > > Thanks,
> > > Rick
> > >
> >
> > HSM is described here:
> > https://github.com/riscv/riscv-sbi-doc/blob/master/riscv-sbi.adoc#hart-state-management-extension-extension-id-0x48534d-hsm
> >
> > Currently the Kendryte K210 has no SPL. So OpenSBI would be running
> > before U-Boot.
> >
> > Will OpenSBI v0.7 by itself stop the other harts or do we need to call
> > sbi_hart_stop() in U-Boot? Unfortunately opensbi/README.md and
> > riscv-sbi.adoc both leave this open.
> >
>
> OpenSBI will put all non-booting harts into wfi in sbi_hsm_wait
> https://github.com/riscv/opensbi/blob/master/lib/sbi/sbi_hsm.c#L119
>
> The S-mode OS is required to call sbi_hsm_start for all non-booting harts
> so that they enter S-mode.
>
> Are you planning to add HSM extension just for Kendryte ?

I have finished this part of jobs internally but still need some time
to do code cleanup and prepare patchs.

Thanks,
Rick

>
> > Best regards
> >
> > Heinrich
> >
> > >>
> > >>> All memory regions containing OpenSBI code are reserved. Thus, U-Boot
> > >>> won't touch that.
> > >>> U-Boot also marks the run time services memory region as reserved as
> > >>> well.
> > >>> Thus, Linux kernel won't touch any of those memory regions either.
> > >>>
> > >>>> spl_secondary_hart_stack_gd_setup() can jump to hang() if the call to
> > >>>> secondary_hart_relocate() fails.
> > >>>>
> > >>>>>
> > >>>>>>
> > >>>>>> When Linux boots via UEFI it will wake up the extra harts after
> > >>>>>> ExitBootServices(). So I assume we should define function hang()
> > >>> in
> > >>>>>> lib/hang.c as __efi_runtime to avoid seeing it overwritten by the
> > >>> EFI
> > >>>>>> payload.
> > >>>>>>
> > >>>>>> @Ard:
> > >>>>>> Does Linux take care of the hanging harts and redirect them to its
> > >>> own
> > >>>>>> routine before SetVirtualAddressMap()? Otherwise anything could
> > >>> happen.
> > >>>>>>
> > >>>>>> On the Kendryte K210 we don't have SPL. So we will not boot in the
> > >>>>>> sequence SPL->OpenSBI->U-Boot but OpenSBI->U-Boot. Does this imply
> > >>> that
> > >>>>>> we have to implement the hart lottery at the entry point of main
> > >>> U-Boot
> > >>>>>> in this case?
> > >>>>>
> > >>>>> Isn't the hart lottery already implemented for U-Boot? E.g. around
> > >>> line
> > >>>>> 100 of arch/riscv/cpu/start.S.
> > >>>>
> > >>>> Thanks for the hint.
> > >>>>
> > >>>>>
> > >>>>> On another note, does Linux support S-Mode NOMMU? I was under the
> > >>>>> impression that NOMMU implied M-Mode (or the other way around).
> > >>>>
> > >>>> Have a look at
> > >>>>
> > >>>>
> > >>> https://linuxplumbersconf.org/event/4/contributions/386/attachments/298/502/RISC-V-NOMMU-Linux-Plumbers-2019.pdf
> > >>>>
> > >>>> Best regards
> > >>>>
> > >>>> Heinrich
> > >>
> >
>
>
> --
> Regards,
> Atish