Re: [Xen-devel] Starting to port xen on beagleboard-x15 (GSoC 2019 project)

Re: [Xen-devel] Starting to port xen on beagleboard-x15 (GSoC 2019 project)

[Andrii Anisov]

+xen-devel

Hello Julien,

> I am a bit confused. Linux is able to bring-up CPU in hyp mode with the current
> U-boot. Why would we need more changes for Xen?

TI's ROM code starts all CPUs in NS PL1, doesn't matter if it is boot or secondary core.
If you look at Linux code [1], you'll see, that after the secondary core bringup Linux does switch that core from NS PL1 to PL2 (hyp mode) using ROM code interface (smc 0).
At some moment XEN dropped all to-hyp-mode switching code. So I did switch secondaries to hyp mode in u-boot with the loop which replicates the ROM Code logic in terms of waiting for an event and checking aux registers. 

[1] https://elixir.bootlin.com/linux/v5.1/source/arch/arm/mach-omap2/omap-headsmp.S#L62

ANDRII ANISOV

Lead Systems Engineer

 

Office: +380 44 390 5457 x 66766   Cell: +380 50 573 8852   Email: andrii_anisov@epam.com

Kyiv, Ukraine (GMT+3)   epam.com

 

CONFIDENTIALITY CAUTION AND DISCLAIMER
This message is intended only for the use of the individual(s) or entity(ies) to which it is addressed and contains information that is legally privileged and confidential. If you are not the intended recipient, or the person responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. All unintended recipients are obliged to delete this message and destroy any printed copies.


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] Starting to port xen on beagleboard-x15 (GSoC 2019 project)

[Julien Grall]

On 18/06/2019 13:28, Andrii Anisov wrote:
> +xen-devel

Please don't cross-post e-mail. If you move the thread to xen-devel, then 
xen-users should be droppped.

> 
> Hello Julien,
> 
>> I am a bit confused. Linux is able to bring-up CPU in hyp mode with the current
>> U-boot. Why would we need more changes for Xen?
> 
> TI's ROM code starts all CPUs in NS PL1, doesn't matter if it is boot or secondary core.
> If you look at Linux code [1], you'll see, that after the secondary core bringup Linux does switch that core from NS PL1 to PL2 (hyp mode) using ROM code interface (smc 0).
> At some moment XEN dropped all to-hyp-mode switching code. So I did switch secondaries to hyp mode in u-boot with the loop which replicates the ROM Code logic in terms of waiting for an event and checking aux registers.

Switch to hyp-mode is fairly complex and depends on your processor. Hence why it 
was dropped in both Linux and Xen.

However, calling an SMC would be acceptable to me. Stefano, any opinion?

> 
> [1] https://elixir.bootlin.com/linux/v5.1/source/arch/arm/mach-omap2/omap-headsmp.S#L62

[...]

> 
> CONFIDENTIALITY CAUTION AND DISCLAIMER
> This message is intended only for the use of the individual(s) or entity(ies) to which it is addressed and contains information that is legally privileged and confidential. If you are not the intended recipient, or the person responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. All unintended recipients are obliged to delete this message and destroy any printed copies.

Please configure your e-mail client to drop the footer.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] Starting to port xen on beagleboard-x15 (GSoC 2019 project)

[Andrii Anisov]

On 18.06.19 15:54, Julien Grall wrote:
> Switch to hyp-mode is fairly complex and depends on your processor. Hence why it was dropped in both Linux and Xen.

Julien, I know it. We already discussed that few years ago.

> However, calling an SMC would be acceptable to me. Stefano, any opinion?

That patch (for u-boot) is 5+ years old. I do not work with TI's stuff for almost 3 years. So I'd not bother myself to play with the stuff again.
If the topic author(s) have enough time and inspiration, they are welcome to improve OMAP5-like SoCs support.

> Please configure your e-mail client to drop the footer.

I think it is impossible, it is a corporate piece of ...email.
I've missed manual removal of the footer, yet I do not use that email for discussions normally.

-- 
Sincerely,
Andrii Anisov.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] Starting to port xen on beagleboard-x15 (GSoC 2019 project)

[Andrii Anisov]

Sorry I've re-read your statement, and got another idea.

On 18.06.19 16:27, Andrii Anisov wrote:
> On 18.06.19 15:54, Julien Grall wrote:
>> Switch to hyp-mode is fairly complex and depends on your processor.

Switch to hyp-mode depends on the processor. So it may be complex. Or not.

>> Hence why it was dropped in both Linux and Xen.

As you can see, for OMAP5 Linux kernel did leave the switch to hyp mode. I guess the reasoning behind that also assumes a complexity of changes to a bootloader, like in the patch you noticed.

-- 
Sincerely,
Andrii Anisov.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] Starting to port xen on beagleboard-x15 (GSoC 2019 project)

[Julien Grall]

On 18/06/2019 14:27, Andrii Anisov wrote:
> 
> 
> On 18.06.19 15:54, Julien Grall wrote:
>> Switch to hyp-mode is fairly complex and depends on your processor. Hence why 
>> it was dropped in both Linux and Xen.
> 
> Julien, I know it. We already discussed that few years ago.
> 
>> However, calling an SMC would be acceptable to me. Stefano, any opinion?
> 
> That patch (for u-boot) is 5+ years old. I do not work with TI's stuff for 
> almost 3 years. So I'd not bother myself to play with the stuff again.
> If the topic author(s) have enough time and inspiration, they are welcome to 
> improve OMAP5-like SoCs support.
Denis (the author of the thread) is doing a GSOC to port Xen on the BeagleBoard 
X15. You ended up CCed because you can provide feedback how to proceed. Not 
because we wanted you to implement it...

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] Starting to port xen on beagleboard-x15 (GSoC 2019 project)

[Stefano Stabellini]

On Tue, 18 Jun 2019, Julien Grall wrote:
> On 18/06/2019 13:28, Andrii Anisov wrote:
> > +xen-devel
> 
> Please don't cross-post e-mail. If you move the thread to xen-devel, then
> xen-users should be droppped.
> 
> > 
> > Hello Julien,
> > 
> > > I am a bit confused. Linux is able to bring-up CPU in hyp mode with the
> > > current
> > > U-boot. Why would we need more changes for Xen?
> > 
> > TI's ROM code starts all CPUs in NS PL1, doesn't matter if it is boot or
> > secondary core.
> > If you look at Linux code [1], you'll see, that after the secondary core
> > bringup Linux does switch that core from NS PL1 to PL2 (hyp mode) using ROM
> > code interface (smc 0).
> > At some moment XEN dropped all to-hyp-mode switching code. So I did switch
> > secondaries to hyp mode in u-boot with the loop which replicates the ROM
> > Code logic in terms of waiting for an event and checking aux registers.
> 
> Switch to hyp-mode is fairly complex and depends on your processor. Hence why
> it was dropped in both Linux and Xen.
> 
> However, calling an SMC would be acceptable to me. Stefano, any opinion?

Yes, it would be trivial to maintain something like that.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] Starting to port xen on beagleboard-x15 (GSoC 2019 project)

[Andrii Anisov]

On 18.06.19 19:19, Julien Grall wrote:
> Denis (the author of the thread) is doing a GSOC to port Xen on the BeagleBoard X15. You ended up CCed because you can provide feedback how to proceed. Not because we wanted you to implement it...

OK then.

Denis,

Feel free to contact me in case you need clarifications about the stuff.

-- 
Sincerely,
Andrii Anisov.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] Starting to port xen on beagleboard-x15 (GSoC 2019 project)

[Denis Obrezkov]

Hi,
ср, 19 июн. 2019 г. в 14:01, Andrii Anisov <andrii.anisov@gmail.com>:
>
>
>
> On 18.06.19 19:19, Julien Grall wrote:
> > Denis (the author of the thread) is doing a GSOC to port Xen on the BeagleBoard X15. You ended up CCed because you can provide feedback how to proceed. Not because we wanted you to implement it...
>
> OK then.
>
> Denis,
>
> Feel free to contact me in case you need clarifications about the stuff.
thank you
>
> --
> Sincerely,
> Andrii Anisov.

So, right now I get a bit further and it seems that CPU1 was switched
to hyp mode:
https://github.com/embeddedden/xen/commit/2d76ae7aacb7c0ea7312eaddb91c3eb1e1963cc9

and the output:
https://pastebin.com/3JBw6S4K

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] Starting to port xen on beagleboard-x15 (GSoC 2019 project)

[Julien Grall]

On 19/06/2019 15:33, Denis Obrezkov wrote:
> Hi,

Hi Denis,

> ср, 19 июн. 2019 г. в 14:01, Andrii Anisov <andrii.anisov@gmail.com>:
>>
>>
>>
>> On 18.06.19 19:19, Julien Grall wrote:
>>> Denis (the author of the thread) is doing a GSOC to port Xen on the BeagleBoard X15. You ended up CCed because you can provide feedback how to proceed. Not because we wanted you to implement it...
>>
>> OK then.
>>
>> Denis,
>>
>> Feel free to contact me in case you need clarifications about the stuff.
> thank you
>>
>> --
>> Sincerely,
>> Andrii Anisov.

Don't forget to strip unnecessary bits of the e-mail you quote :).

> 
> So, right now I get a bit further and it seems that CPU1 was switched
> to hyp mode:
> https://github.com/embeddedden/xen/commit/2d76ae7aacb7c0ea7312eaddb91c3eb1e1963cc9

Nice to see some progress here! :)

Just to keep record of the discussion on IRC:

	- omap5_init_secondary is the entry point to Xen and SMC #0 is called right 
after. There are nothing in r2-r12 that we care about, so there are no need to 
save/restore them. On a side note, you could not use sp because we don't know 
the value stored in it. So you may rewrite Xen (or any other modules).
	
	- From the pastebin "(XEN) Generic Timer IRQ: phys=30 hyp=26 virt=27 Freq: 0 
KHz". It looks like the register CNTFREQ is not configured correctly on the CPU. 
Looking at Linux, they have code to setup CNTFRQ (set_cntfrq) for all the CPUs 
(see realtime_counter_init in arch/arm/mach-omap2/timer.c and 
omap4_secondary_init in arch/arm/mach-omap2/omap-smp.c).
	  In the case of Xen, I think we want to call set_cntfreq in omap5_init_time() 
for the boot CPU. For the secondary CPUs, we may need to introduce a callback in 
struct platform_desc to be called during secondary startup.

Lastly, please clean-up the code and send the patch on xen-devel. I will have a 
closer look at that time. Feel free to ping me on IRC if you have any doubt how 
to proceed.

> and the output:
> https://pastebin.com/3JBw6S4K

(XEN) *** LOADING DOMAIN 0 ***
(XEN) Missing kernel boot module?
(XEN)
(XEN) ****************************************
(XEN) Panic on CPU 0:
(XEN) Could not set up DOM0 guest OS
(XEN) ****************************************

You probably haven't set up the Dom0 kernel here. Check you u-boot runes for this.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] Starting to port xen on beagleboard-x15 (GSoC 2019 project)

[Andrii Anisov]

On 19.06.19 18:06, Julien Grall wrote:
> Lastly, please clean-up the code and send the patch on xen-devel. I will have a closer look at that time. Feel free to ping me on IRC if you have any doubt how to proceed.

About the code: I think omap5_init_secondary() must be moved to the platform code (omap5.c).

I've nothing to add about the log. Pastebin is banned in our network.

-- 
Sincerely,
Andrii Anisov.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] Starting to port xen on beagleboard-x15 (GSoC 2019 project)

[Julien Grall]

Hi,

On 19/06/2019 16:27, Andrii Anisov wrote:
> 
> 
> On 19.06.19 18:06, Julien Grall wrote:
>> Lastly, please clean-up the code and send the patch on xen-devel. I will have 
>> a closer look at that time. Feel free to ping me on IRC if you have any doubt 
>> how to proceed.
> 
> About the code: I think omap5_init_secondary() must be moved to the platform 
> code (omap5.c).

Well omap5.c is C code... You can't call C in the boot process at least until 
the MMU is on and we fully setup the processor.

Regarding the placement of the code, I am still split between two minds (either 
head.S or a new specific .S for omap). However, this could be discussed once the 
patch is submitted.

(Bear in mind this is likely the first time Denis works with the Xen community).

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] Starting to port xen on beagleboard-x15 (GSoC 2019 project)

[Andrii Anisov]

On 19.06.19 18:32, Julien Grall wrote:
> Regarding the placement of the code, I am still split between two minds (either head.S or a new specific .S for omap). However, this could be discussed once the patch is submitted.

IMHO, that is a pure platform specific code. FMPOV it should be an inline assembly in omap5.c, or omap5.S under platform.

> (Bear in mind this is likely the first time Denis works with the Xen community).
Yep:)

-- 
Sincerely,
Andrii Anisov.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] Starting to port xen on beagleboard-x15 (GSoC 2019 project)

[Julien Grall]

On 19/06/2019 17:10, Andrii Anisov wrote:
> 
> 
> On 19.06.19 18:32, Julien Grall wrote:
>> Regarding the placement of the code, I am still split between two minds 
>> (either head.S or a new specific .S for omap). However, this could be 
>> discussed once the patch is submitted.
> 
> IMHO, that is a pure platform specific code. FMPOV it should be an inline 
> assembly in omap5.c, or omap5.S under platform.

I would be happy to consider omap5.c if there are a nice way to do it. I can't 
find one so far. Please suggest...

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] Starting to port xen on beagleboard-x15 (GSoC 2019 project)

[Denis Obrezkov]

[-- Attachment #1.1.1: Type: text/plain, Size: 728 bytes --]

Hi,

On 6/19/19 5:32 PM, Julien Grall wrote:
> Hi,
> 
> On 19/06/2019 16:27, Andrii Anisov wrote:
>>
>>
>> On 19.06.19 18:06, Julien Grall wrote:
>>> Lastly, please clean-up the code and send the patch on xen-devel. I
>>> will have a closer look at that time. Feel free to ping me on IRC if
>>> you have any doubt how to proceed.
>>
>> About the code: I think omap5_init_secondary() must be moved to the
>> platform code (omap5.c).
> 
> Well omap5.c is C code... You can't call C in the boot process at least
> until the MMU is on and we fully setup the processor.
I don't understand this since init_secondary is installed in C code,
isn't it? (in omap5.c:omap5_smp_init())

-- 
Regards, Denis Obrezkov


[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] Starting to port xen on beagleboard-x15 (GSoC 2019 project)

[Denis Obrezkov]

[-- Attachment #1.1.1: Type: text/plain, Size: 2787 bytes --]

Hi,

On 6/19/19 5:06 PM, Julien Grall wrote:
> 
> 
> On 19/06/2019 15:33, Denis Obrezkov wrote:
>> Hi,
> 
> Hi Denis,
> 
>> ср, 19 июн. 2019 г. в 14:01, Andrii Anisov <andrii.anisov@gmail.com>:
>>>
>>>
>>>
>>> On 18.06.19 19:19, Julien Grall wrote:
>>>> Denis (the author of the thread) is doing a GSOC to port Xen on the
>>>> BeagleBoard X15. You ended up CCed because you can provide feedback
>>>> how to proceed. Not because we wanted you to implement it...
>>>
>>> OK then.
>>>
>>> Denis,
>>>
>>> Feel free to contact me in case you need clarifications about the stuff.
>> thank you
>>>
>>> -- 
>>> Sincerely,
>>> Andrii Anisov.
> 
> Don't forget to strip unnecessary bits of the e-mail you quote :).
> 
>>
>> So, right now I get a bit further and it seems that CPU1 was switched
>> to hyp mode:
>> https://github.com/embeddedden/xen/commit/2d76ae7aacb7c0ea7312eaddb91c3eb1e1963cc9
>>
> 
> Nice to see some progress here! :)
> 
> Just to keep record of the discussion on IRC:
> 
>     - omap5_init_secondary is the entry point to Xen and SMC #0 is
> called right after. There are nothing in r2-r12 that we care about, so
> there are no need to save/restore them. On a side note, you could not
> use sp because we don't know the value stored in it. So you may rewrite
> Xen (or any other modules).
>     
>     - From the pastebin "(XEN) Generic Timer IRQ: phys=30 hyp=26 virt=27
> Freq: 0 KHz". It looks like the register CNTFREQ is not configured
> correctly on the CPU. Looking at Linux, they have code to setup CNTFRQ
> (set_cntfrq) for all the CPUs (see realtime_counter_init in
> arch/arm/mach-omap2/timer.c and omap4_secondary_init in
> arch/arm/mach-omap2/omap-smp.c).
>       In the case of Xen, I think we want to call set_cntfreq in
> omap5_init_time() for the boot CPU. For the secondary CPUs, we may need
> to introduce a callback in struct platform_desc to be called during
> secondary startup.
> 
> Lastly, please clean-up the code and send the patch on xen-devel. I will
> have a closer look at that time. Feel free to ping me on IRC if you have
> any doubt how to proceed.
Ok, I will read xen code guideline and send the patch.
> 
>> and the output:
>> https://pastebin.com/3JBw6S4K
> 
> (XEN) *** LOADING DOMAIN 0 ***
> (XEN) Missing kernel boot module?
> (XEN)
> (XEN) ****************************************
> (XEN) Panic on CPU 0:
> (XEN) Could not set up DOM0 guest OS
> (XEN) ****************************************
> 
> You probably haven't set up the Dom0 kernel here. Check you u-boot runes
> for this.
So, I configured dtb and got:
https://pastebin.com/uDYiGsHL
So, it seems that timer interrupts don't work for now.

-- 
Regards, Denis Obrezkov


[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] Starting to port xen on beagleboard-x15 (GSoC 2019 project)

[Julien Grall]

Hi,

On 6/19/19 10:51 PM, Denis Obrezkov wrote:
> Hi,
> 
> On 6/19/19 5:32 PM, Julien Grall wrote:
>> Hi,
>>
>> On 19/06/2019 16:27, Andrii Anisov wrote:
>>>
>>>
>>> On 19.06.19 18:06, Julien Grall wrote:
>>>> Lastly, please clean-up the code and send the patch on xen-devel. I
>>>> will have a closer look at that time. Feel free to ping me on IRC if
>>>> you have any doubt how to proceed.
>>>
>>> About the code: I think omap5_init_secondary() must be moved to the
>>> platform code (omap5.c).
>>
>> Well omap5.c is C code... You can't call C in the boot process at least
>> until the MMU is on and we fully setup the processor.
> I don't understand this since init_secondary is installed in C code,
> isn't it? (in omap5.c:omap5_smp_init())

init_secondary is installed by the boot CPU. But init_secondary is
executed on the secondary CPU.

You can't execute C code that early for a few reasons:
    1) You don't know the state of the registers. So you need to set 
them up in a sane state.
    2) Xen is compiled position depend so far. This means some of the C 
code use hardcoded address to access data or jump to other function. The 
hardcoded address are virtual. In most of the cases, Xen will not be 
loaded at at virtual address == physical address. So you need to setup 
the MMU first.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] Starting to port xen on beagleboard-x15 (GSoC 2019 project)

[Julien Grall]

Hi,

On 6/19/19 11:06 PM, Denis Obrezkov wrote:
>> (XEN) *** LOADING DOMAIN 0 ***
>> (XEN) Missing kernel boot module?
>> (XEN)
>> (XEN) ****************************************
>> (XEN) Panic on CPU 0:
>> (XEN) Could not set up DOM0 guest OS
>> (XEN) ****************************************
>>
>> You probably haven't set up the Dom0 kernel here. Check you u-boot runes
>> for this.
> So, I configured dtb and got:
> https://pastebin.com/uDYiGsHL
> So, it seems that timer interrupts don't work for now.

Most likely this is because the frequency is incorrect. Have a look at 
set_cntfreq() (see my previous e-mail).

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel