ARM - Successful install on RockPro64

ARM - Successful install on RockPro64

[Richard Simpson]

Hello,

Just to report that I have successfully installed Xen on a Pine 
RockPro64 ARM SBC.

I am using Xen 4.13 booting directly from u-boot on an SD card and my 
dom0 distribution is Gentoo.

I haven't tried to create a domU yet and I am doing everything via the 
serial console so I can't say anything about the graphics.

My biggest hurdle (apart from understanding u-boot) was needing to apply 
the vgic-v3: fix GICD_ISACTIVER patch.

I will be happy to provide more details when I have got a bit further, 
but after two weeks of effort I was so delighted to finally be able to 
log into dom0 that I thought I'd better let somebody know.

Richard Simpson

Re: ARM - Successful install on RockPro64

[Bertrand Marquis]

Hi Richard,

+ Julien and Stefano

> On 15 Jun 2020, at 23:29, Richard Simpson <xen@huskydog.org.uk> wrote:
> 
> Hello,
> 
> Just to report that I have successfully installed Xen on a Pine RockPro64 ARM SBC.

Very nice :-)

> 
> I am using Xen 4.13 booting directly from u-boot on an SD card and my dom0 distribution is Gentoo.
> 
> I haven't tried to create a domU yet and I am doing everything via the serial console so I can't say anything about the graphics.
> 
> My biggest hurdle (apart from understanding u-boot) was needing to apply the vgic-v3: fix GICD_ISACTIVER patch.

What version of Linux are you running ?
I added Julien and Stefano so that we reactivate the discussion on this patch.

> 
> I will be happy to provide more details when I have got a bit further, but after two weeks of effort I was so delighted to finally be able to log into dom0 that I thought I'd better let somebody know.

Thanks a lot for the notice, and congrats again :-)

Cheers
Bertrand

Re: ARM - Successful install on RockPro64

[Richard Simpson]

Dear Bertrand,

Just to confirm that the Linux distro is Gentoo.  I know it's not very 
common these days, but I have used it for very many years and it is what 
I am used to.

Gentoo has a xen package but only for arm32 and then masked as unstable 
so I just grabbed 4.13 via git.  It also has a xen-tools package which 
is packaged for arm64 (although still masked as unstable) so I unmasked 
and used that.  There were a couple of dependency problems with the 
package but after I resolved those it seemed to work OK.

There is also a Gentoo u-boot tools package for arm64 but again I got 
u-boot via git.

I would be happy to try to report my success via the smoke test page 
(https://wiki.xenproject.org/wiki/Xen_ARM_Manual_Smoke_Test/Results) if 
I can figure out how.  Strangely, I can't see anything listed under 
"Test Results" from anyone else.  Perhaps it is a problem with my browser.

I also notice an instruction which reads "Test hypervisor 
functionalities: clone raisin on the platform and run ./raise test".  I 
can try to do this if it will help.  Do I just run "git clone <link from 
web page>" and then presumably the test prints out some results?

Happy to try a beta version of Xen if you decide to include the patch 
and I can also try some of the interrupt config options if you want.

Cheers,

     Richard

On 6/16/20 9:03 AM, Bertrand Marquis wrote:
> Hi Richard,
>
> + Julien and Stefano
>
>> On 15 Jun 2020, at 23:29, Richard Simpson <xen@huskydog.org.uk> wrote:
>>
>> Hello,
>>
>> Just to report that I have successfully installed Xen on a Pine RockPro64 ARM SBC.
> Very nice :-)
>
>> I am using Xen 4.13 booting directly from u-boot on an SD card and my dom0 distribution is Gentoo.
>>
>> I haven't tried to create a domU yet and I am doing everything via the serial console so I can't say anything about the graphics.
>>
>> My biggest hurdle (apart from understanding u-boot) was needing to apply the vgic-v3: fix GICD_ISACTIVER patch.
> What version of Linux are you running ?
> I added Julien and Stefano so that we reactivate the discussion on this patch.
>
>> I will be happy to provide more details when I have got a bit further, but after two weeks of effort I was so delighted to finally be able to log into dom0 that I thought I'd better let somebody know.
> Thanks a lot for the notice, and congrats again :-)
>
> Cheers
> Bertrand
>

Re: ARM - Successful install on RockPro64

[Julien Grall]

Hello,

On 16/06/2020 09:33, Richard Simpson wrote:
> I would be happy to try to report my success via the smoke test page 
> (https://wiki.xenproject.org/wiki/Xen_ARM_Manual_Smoke_Test/Results) if 
> I can figure out how.  Strangely, I can't see anything listed under 
> "Test Results" from anyone else.  Perhaps it is a problem with my browser.
This is not a browser problem :). In the past, we did attempt to list 
all the boards we know works on a given version of Xen. But this never 
really gain any momentum.

If there is any specific setup you need for your board (e.g. 
non-upstream kernel, a new U-boot...), then I would suggest to create a 
new page with steps to boot Xen the platform. You can have a look how we 
list the other boards in [1].

> 
> I also notice an instruction which reads "Test hypervisor 
> functionalities: clone raisin on the platform and run ./raise test".  I 
> can try to do this if it will help.  Do I just run "git clone <link from 
> web page>" and then presumably the test prints out some results?

It is just meant to be an easy way to test basic functionality of Xen 
(e.g booting a guest). You seem to have done it manually, so it should 
be sufficient.

> 
> Happy to try a beta version of Xen if you decide to include the patch 
> and I can also try some of the interrupt config options if you want.

The patch should already be included in xen 4.14-rc2. Would you mind to 
give a spin?

Best regards,

[1] https://wiki.xenproject.org/wiki/Xen_ARM_with_Virtualization_Extensions

-- 
Julien Grall

Re: ARM - Successful install on RockPro64

[Richard Simpson]

Hello Julien,

I have just tried 4.14-rc2 and it seems to work fine.

I think that the most useful page regarding the board is the one for the 
Ibox3399 since this refers to the RK3399 chip which the RockPro64 uses 
(shouldn't the page actually be called RK3399 to make it more generic).

Perhaps I can most usefully record what I did by updating that page and 
making sure that the instructions work correctly. If there is additional 
stuff relevant to the RockPro64 over and above the generic RK3399 info 
then I'll give some thought to how to best record it.  I will eventually 
be writing a fuller report on my progress on my blog at 
funfoodfreedom.huskydog.org.uk.

I now need to finish automating the boot process (still requires manual 
u-boot command) and figure out how to get the console log to work.  
Currently I can either see the xen and linux kernel boot messages OR see 
the dom0 console, but not both.  I then want to wipe the install and 
start again without all the blind alleys that I went down the first time 
(mostly a function of my ignorance). This should allow me to clearly 
document what is actually required from both a Xen and Gentoo perspective.

I'll also try to more of the Xen config options to see what works and 
what is actually needed.

On one more related note:  I suspect that Xen would run on the 
PineBookPro as well as I get the impression that it uses very similar 
hardware.  Of course that would rely on the GPU etc which I haven't 
tested at all as I am using the serial console.

Finally, when I joined this mailing list I asked for a daily digest.  
However I seem to be getting a new digest every hour or so.  Is this right?

Regards,

     Richard

On 6/16/20 11:26 AM, Julien Grall wrote:
> Hello,
>
> On 16/06/2020 09:33, Richard Simpson wrote:
>> I would be happy to try to report my success via the smoke test page 
>> (https://wiki.xenproject.org/wiki/Xen_ARM_Manual_Smoke_Test/Results) 
>> if I can figure out how.  Strangely, I can't see anything listed 
>> under "Test Results" from anyone else.  Perhaps it is a problem with 
>> my browser.
> This is not a browser problem :). In the past, we did attempt to list 
> all the boards we know works on a given version of Xen. But this never 
> really gain any momentum.
>
> If there is any specific setup you need for your board (e.g. 
> non-upstream kernel, a new U-boot...), then I would suggest to create 
> a new page with steps to boot Xen the platform. You can have a look 
> how we list the other boards in [1].
>
>>
>> I also notice an instruction which reads "Test hypervisor 
>> functionalities: clone raisin on the platform and run ./raise test".  
>> I can try to do this if it will help.  Do I just run "git clone <link 
>> from web page>" and then presumably the test prints out some results?
>
> It is just meant to be an easy way to test basic functionality of Xen 
> (e.g booting a guest). You seem to have done it manually, so it should 
> be sufficient.
>
>>
>> Happy to try a beta version of Xen if you decide to include the patch 
>> and I can also try some of the interrupt config options if you want.
>
> The patch should already be included in xen 4.14-rc2. Would you mind 
> to give a spin?
>
> Best regards,
>
> [1] 
> https://wiki.xenproject.org/wiki/Xen_ARM_with_Virtualization_Extensions
>

Re: ARM - Successful install on RockPro64

[Julien Grall]

On 17/06/2020 23:28, Richard Simpson wrote:
> Hello Julien,

Hello Richard,

Apologies for the late answer.

> I have just tried 4.14-rc2 and it seems to work fine.

Glad to hear that. Thank you for the testing!

> I think that the most useful page regarding the board is the one for the 
> Ibox3399 since this refers to the RK3399 chip which the RockPro64 uses 
> (shouldn't the page actually be called RK3399 to make it more generic).

I agree with the renaming here.

> Perhaps I can most usefully record what I did by updating that page and 
> making sure that the instructions work correctly. If there is additional 
> stuff relevant to the RockPro64 over and above the generic RK3399 info 
> then I'll give some thought to how to best record it.  I will eventually 
> be writing a fuller report on my progress on my blog at 
> funfoodfreedom.huskydog.org.uk.

Any additional content on the wiki will be greatly appreciated. By 
default new wiki account doesn't have write permission, but we can 
enable it for you if you provide us your username.

> 
> I now need to finish automating the boot process (still requires manual 
> u-boot command) and figure out how to get the console log to work. 

I wrote a small u-boot script in the past to try to automate the boot 
(see [2]).

I vaguely remember some quoting issue and missing 0x in front of values 
depending on the U-boot configuration you use. So you may have to tweak 
it a bit.

> Currently I can either see the xen and linux kernel boot messages OR see 
> the dom0 console, but not both.

Can you provide the kernel/xen command lines you use in the two cases?

As an aside, I know that on some setup Linux will try to disable the 
clock of the UART used by Xen. One of the symptoms is the UART is 
becoming completely unusable half way through Linux boot.

You may want to try to pass clk_ignored_unused to see if it helps.

> On one more related note:  I suspect that Xen would run on the 
> PineBookPro as well as I get the impression that it uses very similar 
> hardware.  Of course that would rely on the GPU etc which I haven't 
> tested at all as I am using the serial console.
I wouldn't expect any issue to use the GPU in dom0 at least if you don't 
have an IOMMU on the platform. The trouble may be more with the 
bootloader if it doesn't drop you in hypervisor mode.

> 
> Finally, when I joined this mailing list I asked for a daily digest. 
> However I seem to be getting a new digest every hour or so.  Is this right?

I haven't used the digest myself. I CC Ian Jackson who may be able to 
help you.

Cheers,

[2] https://xenbits.xen.org/people/julieng/load-xen-tftp.scr.txt

-- 
Julien Grall

Re: ARM - Successful install on RockPro64

[Richard Simpson]

Hello Julien,

I also have been busy and unable to reply.


On 6/24/20 1:42 PM, Julien Grall wrote:
>
>
> On 17/06/2020 23:28, Richard Simpson wrote:
>> Hello Julien,
>
> Hello Richard,
>
> Apologies for the late answer.
>
>> I have just tried 4.14-rc2 and it seems to work fine.
>
> Glad to hear that. Thank you for the testing!
>
>> I think that the most useful page regarding the board is the one for 
>> the Ibox3399 since this refers to the RK3399 chip which the RockPro64 
>> uses (shouldn't the page actually be called RK3399 to make it more 
>> generic).
>
> I agree with the renaming here.
>
>> Perhaps I can most usefully record what I did by updating that page 
>> and making sure that the instructions work correctly. If there is 
>> additional stuff relevant to the RockPro64 over and above the generic 
>> RK3399 info then I'll give some thought to how to best record it.  I 
>> will eventually be writing a fuller report on my progress on my blog 
>> at funfoodfreedom.huskydog.org.uk.
>
> Any additional content on the wiki will be greatly appreciated. By 
> default new wiki account doesn't have write permission, but we can 
> enable it for you if you provide us your username.
I now have editor access on the Wiki and have made a small test edit in 
the hardware section.  I will now try to make other changes as I learn 
more whilst trying very hard not to break anything!
>>
>> I now need to finish automating the boot process (still requires 
>> manual u-boot command) and figure out how to get the console log to 
>> work. 
>
> I wrote a small u-boot script in the past to try to automate the boot 
> (see [2]).
>
> I vaguely remember some quoting issue and missing 0x in front of 
> values depending on the U-boot configuration you use. So you may have 
> to tweak it a bit.
I now have a boot script that seems to automate the boot OK.  I am sure 
it could be better and I would like to add a boot menu but these are 
refinements I can do later.  I will see what I can learn from your example.
>
>> Currently I can either see the xen and linux kernel boot messages OR 
>> see the dom0 console, but not both.
>
> Can you provide the kernel/xen command lines you use in the two cases?
>
> As an aside, I know that on some setup Linux will try to disable the 
> clock of the UART used by Xen. One of the symptoms is the UART is 
> becoming completely unusable half way through Linux boot.
>
> You may want to try to pass clk_ignored_unused to see if it helps.
I still haven't fixed the boot message problem but I got very confused 
about what changes were having which effect.  Hopefully this weekend I 
will systematically try the various xen and Linux options and produce a 
table of what the effect is each time.  I'll also try your clk_ignored 
suggestion.  If I don't find a working combination then I'll post a 
question and I guess this should be on the users list rather than this one.
>
>> On one more related note:  I suspect that Xen would run on the 
>> PineBookPro as well as I get the impression that it uses very similar 
>> hardware.  Of course that would rely on the GPU etc which I haven't 
>> tested at all as I am using the serial console.
> I wouldn't expect any issue to use the GPU in dom0 at least if you 
> don't have an IOMMU on the platform. The trouble may be more with the 
> bootloader if it doesn't drop you in hypervisor mode.
>
>>
>> Finally, when I joined this mailing list I asked for a daily digest. 
>> However I seem to be getting a new digest every hour or so.  Is this 
>> right?
>
> I haven't used the digest myself. I CC Ian Jackson who may be able to 
> help you.
>
> Cheers,
>
> [2] https://xenbits.xen.org/people/julieng/load-xen-tftp.scr.txt
>

Re: ARM - Successful install on RockPro64

[Richard Simpson]

Hello Julien, yes the clk_ignored_unused option fixed the boot message 
problem.  I would never have worked that out for myself.

The relevant lines now read:

Xen: console=dtuart dtuart=serial2

Linux: console=hvc0 earlyprintk=xen clk_ignore_unused

inittab: h0:12345:respawn:/sbin/agetty 1500000 hvc0 vt100

I have added a note to the Xen Serial Console Wiki page which - I was 
shocked to observe - hadn't been updated since 2014.  I'll add it to my 
list of pages to refresh, noting that I only have Xen on this one ARM 
SBC (excluding my QubesOS laptop) so I can only check anything I write 
relative to that.

Regards,

     Richard

On 6/24/20 1:42 PM, Julien Grall wrote:
>
>
> On 17/06/2020 23:28, Richard Simpson wrote:
>> Hello Julien,
>
> Hello Richard,
>
> Apologies for the late answer.
>
>> I have just tried 4.14-rc2 and it seems to work fine.
>
> Glad to hear that. Thank you for the testing!
>
>> I think that the most useful page regarding the board is the one for 
>> the Ibox3399 since this refers to the RK3399 chip which the RockPro64 
>> uses (shouldn't the page actually be called RK3399 to make it more 
>> generic).
>
> I agree with the renaming here.
>
>> Perhaps I can most usefully record what I did by updating that page 
>> and making sure that the instructions work correctly. If there is 
>> additional stuff relevant to the RockPro64 over and above the generic 
>> RK3399 info then I'll give some thought to how to best record it.  I 
>> will eventually be writing a fuller report on my progress on my blog 
>> at funfoodfreedom.huskydog.org.uk.
>
> Any additional content on the wiki will be greatly appreciated. By 
> default new wiki account doesn't have write permission, but we can 
> enable it for you if you provide us your username.
>
>>
>> I now need to finish automating the boot process (still requires 
>> manual u-boot command) and figure out how to get the console log to 
>> work. 
>
> I wrote a small u-boot script in the past to try to automate the boot 
> (see [2]).
>
> I vaguely remember some quoting issue and missing 0x in front of 
> values depending on the U-boot configuration you use. So you may have 
> to tweak it a bit.
>
>> Currently I can either see the xen and linux kernel boot messages OR 
>> see the dom0 console, but not both.
>
> Can you provide the kernel/xen command lines you use in the two cases?
>
> As an aside, I know that on some setup Linux will try to disable the 
> clock of the UART used by Xen. One of the symptoms is the UART is 
> becoming completely unusable half way through Linux boot.
>
> You may want to try to pass clk_ignored_unused to see if it helps.
>
>> On one more related note:  I suspect that Xen would run on the 
>> PineBookPro as well as I get the impression that it uses very similar 
>> hardware.  Of course that would rely on the GPU etc which I haven't 
>> tested at all as I am using the serial console.
> I wouldn't expect any issue to use the GPU in dom0 at least if you 
> don't have an IOMMU on the platform. The trouble may be more with the 
> bootloader if it doesn't drop you in hypervisor mode.
>
>>
>> Finally, when I joined this mailing list I asked for a daily digest. 
>> However I seem to be getting a new digest every hour or so.  Is this 
>> right?
>
> I haven't used the digest myself. I CC Ian Jackson who may be able to 
> help you.
>
> Cheers,
>
> [2] https://xenbits.xen.org/people/julieng/load-xen-tftp.scr.txt
>

Re: ARM - Successful install on RockPro64

[Julien Grall]

Hi,

On 16/06/2020 09:03, Bertrand Marquis wrote:
> Hi Richard,
> 
> + Julien and Stefano
> 
>> On 15 Jun 2020, at 23:29, Richard Simpson <xen@huskydog.org.uk> wrote:
>>
>> Hello,
>>
>> Just to report that I have successfully installed Xen on a Pine RockPro64 ARM SBC.
> 
> Very nice :-)
> 
>>
>> I am using Xen 4.13 booting directly from u-boot on an SD card and my dom0 distribution is Gentoo.
>>
>> I haven't tried to create a domU yet and I am doing everything via the serial console so I can't say anything about the graphics.
>>
>> My biggest hurdle (apart from understanding u-boot) was needing to apply the vgic-v3: fix GICD_ISACTIVER patch.
> 
> What version of Linux are you running ?
> I added Julien and Stefano so that we reactivate the discussion on this patch.

A temporary patch to mark the register RAZ has been merged a couple of 
months ago. See 0796cb907f2c "xen/arm: vgic-v3: fix GICD_ISACTIVER 
range". This should probably be backported to Xen 4.13 and 4.12.

Although, I haven't seen any update for a proper fix yet.

Cheers,

-- 
Julien Grall