Re: RT Xen on ARM - R-Car series

Re: RT Xen on ARM - R-Car series

[Andrii Anisov]

Hello Jairo,

From: LOPEZ, FUENTES NACARINO Jairo Eduardo <jairo@ruri.waseda.jp>
Sent: Saturday, December 22, 2018 3:55 AM
To: Andrii Anisov; Iurii Artemenko
Subject: RT Xen on ARM - R-Car series

> My name is Jairo Lopez.

> I am currently working as a doctorate student at Waseda University in the area of distributed systems.

> We have previously used the R-Car M3 Starter Kit Pro and R-Car H3 Starter Kit Premier
> (https://www.renesas.com/us/en/solutions/automotive/adas/solution-kits/r-car-starter-kit.html) for some
> demos and were interested in using them to test Xen on these machines. Since we are looking at
> real-time distributed systems, your work on real-time for Xen greatly interests me.

Thank you for your interest in XEN and RT in XEN. I hope you'll have great progress in your work and enrich XEN hypervisor with your findings.

> I have attempted to use the Salvator-X guidelines (https://wiki.xenproject.org/wiki/Xen_ARM_with_Virtualization_Extensions/Salvator-X) on 
> the Xen Wiki to try to test Xen on the above mentioned machines

What SoC revisions do you have on your boards? Keep in mind this [1], although it should not be a problem for you yet.

> and although the build succeeds

I hope you built x3ulcb targets instead of salvator-x. BTW, the BSP mentioned in the wiki is pretty old. Me currently using 3.9. But it should not be a big problem on this stage.

> I am left with a wonderful empty screen.
And what do you see in a console? Any logs?

> It is currently possible to boot Xen on the R-Car M3 or R-Car H3?
Yes, sure. Our team constantly work with R-Car Gen3 based boards. Unfortunately, Renesas BSP still needs some adjustments to get XEN built and run with it.

> If my laboratory is keen on getting Xen working on these machines, how could we go on about doing this?
Carefully follow instructions. Provide detailed descriptions of your steps and results (build/boot logs) in case something went wrong on your site.
  
ps. You know, XEN is an open-source project, it's better to ask questions through the mailing list [2]. You might have more answers hints and opinions from the community. I've copied this email to the list for you.

[1] https://elinux.org/R-Car/Boards/Yocto-Gen3#Known_issues_and_limitations
[2] https://www.xenproject.org/help/mailing-list.html

ANDRII ANISOV

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: RT Xen on ARM - R-Car series

[LOPEZ, FUENTES NACARINO Jairo Eduardo]

[-- Attachment #1.1: Type: text/plain, Size: 9145 bytes --]

Andrii,

Thank you for your response.

I am sorry for not being able to make time to answer sooner.

Hello to everyone from the Xen Development mailing list.

As I told Andrii, I am interested in RT Xen, but before getting anywhere
near that, I need to actually get my board to use Xen.

As a total newcomer to these boards and knowing that Xen is a little
complex to get running, I am sure there are a lot of steps that I am either
doing wrong, or have made the wrong assumptions.

The SoC information of the R-Car H3 I have is below:

CPU: Renesas Electronics R8A7795 rev 2.0
Board: H3ULCB
I2C:   ready
DRAM:  3.9 GiB
MMC:   sh-sdhi: 0, sh-sdhi: 1
In:    serial
Out:   serial
Err:   serial
Net:   ravb
Hit any key to stop autoboot:  0
=> [    0.000158] NOTICE:  BL2: R-Car Gen3 Initial Program Loader(CA57)
Rev.1.0.16
[    0.005718] NOTICE:  BL2: PRR is R-Car H3 Ver2.0
[    0.010301] NOTICE:  BL2: Board is Starter Kit Rev1.0
[    0.015328] NOTICE:  BL2: Boot device is HyperFlash(80MHz)
[    0.020754] NOTICE:  BL2: LCM state is CM
[    0.024799] NOTICE:  BL2: AVS setting succeeded. DVFS_SetVID=0x53
[    0.030782] NOTICE:  BL2: DDR3200(rev.0.27)NOTICE:  [COLD_BOOT]NOTICE:
..0
[    0.057151] NOTICE:  BL2: DRAM Split is 4ch
[    0.061034] NOTICE:  BL2: QoS is default setting(rev.0.18)
[    0.066533] NOTICE:  BL2: Lossy Decomp areas
[    0.070711] NOTICE:       Entry 0: DCMPAREACRAx:0x80000540
DCMPAREACRBx:0x570
[    0.077796] NOTICE:       Entry 1: DCMPAREACRAx:0x40000000
DCMPAREACRBx:0x0
[    0.084708] NOTICE:       Entry 2: DCMPAREACRAx:0x20000000
DCMPAREACRBx:0x0
[    0.091623] NOTICE:  BL2: v1.3(release):0387358
[    0.096113] NOTICE:  BL2: Built : 04:35:44, Jun 22 2018
[    0.101300] NOTICE:  BL2: Normal boot
[    0.104941] NOTICE:  BL2: dst=0xe6320210 src=0x8180000 len=512(0x200)
[    0.111487] NOTICE:  BL2: dst=0x43f00000 src=0x8180400 len=6144(0x1800)
[    0.117949] NOTICE:  BL2: dst=0x44000000 src=0x81c0000 len=65536(0x10000)
[    0.125175] NOTICE:  BL2: dst=0x44100000 src=0x8200000
len=524288(0x80000)
[    0.135655] NOTICE:  BL2: dst=0x50000000 src=0x8640000
len=1048576(0x100000)
[    0.149770] NOTICE:  BL3-1 : Rev.1.0.16
[    0.154065] NOTICE:  BL31: v1.3(release):0387358
[    0.157189] NOTICE:  BL31: Built : 04:35:47, Jun 22 2018


U-Boot 2015.04 (Jun 22 2018 - 13:36:27)

CPU: Renesas Electronics R8A7795 rev 2.0
Board: H3ULCB
I2C:   ready
DRAM:  3.9 GiB
MMC:   sh-sdhi: 0, sh-sdhi: 1
In:    serial
Out:   serial
Err:   serial
Net:   ravb

I believe this is the SoC information. If there is any other method of
extracting the information, please let me know so I can transmit it.
I took a look at [1] and decided to start from scratch to attempt to get
the minimum workspace functioning.

As I mentioned before I attempted using the Salvator-X guideline for the
H3ULCB.

In previous attempts, I had to modify some recipes to get the compilation
working, but this time I would like to confirm with everyone the initial
steps before I take them.

I have not done anything other than what the Wiki (
https://www.renesas.com/us/en/solutions/automotive/adas/solution-kits/r-car-starter-kit.html)
says I should.
After adding all the layers, I start bitbake:

$ bitbake core-image-minimal

Build
Configuration:
BB_VERSION        =
"1.30.0"
BUILD_SYS         =
"x86_64-linux"
NATIVELSBSTRING   =
"universal"
TARGET_SYS        =
"aarch64-poky-linux"
MACHINE           =
"h3ulcb"
DISTRO            =
"poky"
DISTRO_VERSION    =
"2.1.2"
TUNE_FEATURES     = "aarch64
cortexa57-cortexa53"
TARGET_FPU        =
""
SOC_FAMILY        =
"rcar-gen3:r8a7796"
meta

meta-poky

meta-yocto-bsp    =
"tmp:cca8dd15c8096626052f6d8d25ff1e9a606104a3"
meta-rcar-gen3    =
"tmp:95cb48ba09bc7e55fd549817e3e26723409e68d5"
meta-linaro-toolchain

meta-optee        =
"tmp:2f51d38048599d9878f149d6d15539fb97603f8f"
meta-oe           =
"tmp:55c8a76da5dc099a7bc3838495c672140cedb78e"
meta-virtualization =
"morty:6249631f59ad6ee3dc93762de49fc4b443d99abc"
meta-selinux      =
"jethro:4c75d9cbcf1d75043c7c5ab315aa383d9b227510"
meta-networking

meta-python       =
"tmp:55c8a76da5dc099a7bc3838495c672140cedb78e"
meta-rcar-gen3-xen =
"master:60699c631d541aeeaebaeec9a087efed9385ee42"


NOTE: Preparing
RunQueue
NOTE: Executing SetScene
Tasks
NOTE: Executing RunQueue
Tasks
ERROR: xen-unstable+gitAUTOINC+9d357cbaf7-r0 do_package: QA Issue: xen:
Files/directories were installed but not shipped in any package:
  /usr/lib/libxenfsimage.so
  /usr/lib/libxenfsimage.so.4.12
  /usr/lib/libxenfsimage.so.4.12.0
  /usr/lib/xenfsimage/ext2fs-lib/fsimage.so
  /usr/lib/xenfsimage/ufs/fsimage.so
  /usr/lib/xenfsimage/fat/fsimage.so
  /usr/lib/xenfsimage/iso9660/fsimage.so
  /usr/lib/xenfsimage/reiserfs/fsimage.so
  /usr/lib/xenfsimage/zfs/fsimage.so
  /usr/lib/xen/bin/depriv-fd-checker
  /usr/sbin/xenmon
Please set FILES such that these items are packaged. Alternatively if they
are unneeded, avoid installing them or delete them within do_install.
xen: 11 installed and not shipped files. [installed-vs-shipped]
ERROR: xen-unstable+gitAUTOINC+9d357cbaf7-r0 do_package: Fatal QA errors
found, failing task.
ERROR: xen-unstable+gitAUTOINC+9d357cbaf7-r0 do_package: Function failed:
do_package
ERROR: Logfile of failure stored in:
/home/yocto/r-car/build/tmp/work/aarch64-poky-linux/xen/unstable+gitAUTOINC+9d357cbaf7-r0/temp/log.do_package.8954
ERROR: Task 329 (/home/yocto/r-car/meta-virtualization/recipes-extended/xen/
xen_git.bb, do_package) failed with exit code '1'
NOTE: Tasks Summary: Attempted 3642 tasks of which 3641 didn't need to be
rerun and 1 failed.
No currently running tasks (2517 of 3653)

Summary: 1 task failed:
  /home/yocto/r-car/meta-virtualization/recipes-extended/xen/xen_git.bb,
do_package
Summary: There were 3 WARNING messages shown.
Summary: There were 3 ERROR messages shown, returning a non-zero exit code.

I am aware I am using a very old BSP. If there is a slightly better version
with which to start with, I would greatly like everyone's opinion.

>From the log it would seem that the xen_git.bb in the meta-virtualization
layer is being called and thus the recipe is attempting to compile the
newest version of Xen.

So my second question would be, what version of Xen should I point towards
for the board I am using?

I am sorry for the completely newcomer questions, but I hope to someday
soon be able to help with Xen ARM development.

Best regards,

2018年12月22日(土) 21:21 Andrii Anisov <Andrii_Anisov@epam.com>:

> Hello Jairo,
>
> From: LOPEZ, FUENTES NACARINO Jairo Eduardo <jairo@ruri.waseda.jp>
> Sent: Saturday, December 22, 2018 3:55 AM
> To: Andrii Anisov; Iurii Artemenko
> Subject: RT Xen on ARM - R-Car series
>
> > My name is Jairo Lopez.
>
> > I am currently working as a doctorate student at Waseda University in
> the area of distributed systems.
>
> > We have previously used the R-Car M3 Starter Kit Pro and R-Car H3
> Starter Kit Premier
> > (
> https://www.renesas.com/us/en/solutions/automotive/adas/solution-kits/r-car-starter-kit.html)
> for some
> > demos and were interested in using them to test Xen on these machines.
> Since we are looking at
> > real-time distributed systems, your work on real-time for Xen greatly
> interests me.
>
> Thank you for your interest in XEN and RT in XEN. I hope you'll have great
> progress in your work and enrich XEN hypervisor with your findings.
>
> > I have attempted to use the Salvator-X guidelines (
> https://wiki.xenproject.org/wiki/Xen_ARM_with_Virtualization_Extensions/Salvator-X)
> on
> > the Xen Wiki to try to test Xen on the above mentioned machines
>
> What SoC revisions do you have on your boards? Keep in mind this [1],
> although it should not be a problem for you yet.
>
> > and although the build succeeds
>
> I hope you built x3ulcb targets instead of salvator-x. BTW, the BSP
> mentioned in the wiki is pretty old. Me currently using 3.9. But it should
> not be a big problem on this stage.
>
> > I am left with a wonderful empty screen.
> And what do you see in a console? Any logs?
>
> > It is currently possible to boot Xen on the R-Car M3 or R-Car H3?
> Yes, sure. Our team constantly work with R-Car Gen3 based boards.
> Unfortunately, Renesas BSP still needs some adjustments to get XEN built
> and run with it.
>
> > If my laboratory is keen on getting Xen working on these machines, how
> could we go on about doing this?
> Carefully follow instructions. Provide detailed descriptions of your steps
> and results (build/boot logs) in case something went wrong on your site.
>
> ps. You know, XEN is an open-source project, it's better to ask questions
> through the mailing list [2]. You might have more answers hints and
> opinions from the community. I've copied this email to the list for you.
>
> [1]
> https://elinux.org/R-Car/Boards/Yocto-Gen3#Known_issues_and_limitations
> [2] https://www.xenproject.org/help/mailing-list.html
>
> ANDRII ANISOV
>

[-- Attachment #1.2: Type: text/html, Size: 13646 bytes --]

[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: RT Xen on ARM - R-Car series

[Andrii Anisov]

Hello Jairo,

On 25.12.18 18:07, LOPEZ, FUENTES NACARINO Jairo Eduardo wrote:
> I believe this is the SoC information. If there is any other method of extracting the information, please let me know so I can transmit it.

That output gives a full set of required information. Actually, I worried you
might have an obsolete SoC revision. But it is not your case.

> I took a look at [1] and decided to start from scratch to attempt to get the minimum workspace functioning.

I mainly mentioned point 1 from those limitations. But you have H3 ES2.0 so
ready for the latest BSP as well.

> In previous attempts, I had to modify some recipes to get the compilation working, but this time I would like to confirm with everyone the initial steps before I take them.
> ERROR: xen-unstable+gitAUTOINC+9d357cbaf7-r0 do_package: QA Issue: xen: Files/directories were installed but not shipped in any package:
>    /usr/lib/libxenfsimage.so
>    /usr/lib/libxenfsimage.so.4.12
>    /usr/lib/libxenfsimage.so.4.12.0
>    /usr/lib/xenfsimage/ext2fs-lib/fsimage.so
>    /usr/lib/xenfsimage/ufs/fsimage.so
>    /usr/lib/xenfsimage/fat/fsimage.so
>    /usr/lib/xenfsimage/iso9660/fsimage.so
>    /usr/lib/xenfsimage/reiserfs/fsimage.so
>    /usr/lib/xenfsimage/zfs/fsimage.so
>    /usr/lib/xen/bin/depriv-fd-checker
>    /usr/sbin/xenmon
> Please set FILES such that these items are packaged. Alternatively if they are unneeded, avoid installing them or delete them within do_install.
> xen: 11 installed and not shipped files. [installed-vs-shipped]
> ERROR: xen-unstable+gitAUTOINC+9d357cbaf7-r0 do_package: Fatal QA errors found, failing task.
> ERROR: xen-unstable+gitAUTOINC+9d357cbaf7-r0 do_package: Function failed: do_package
> ERROR: Logfile of failure stored in: /home/yocto/r-car/build/tmp/work/aarch64-poky-linux/xen/unstable+gitAUTOINC+9d357cbaf7-r0/temp/log.do_package.8954
> ERROR: Task 329 (/home/yocto/r-car/meta-virtualization/recipes-extended/xen/xen_git.bb <http://xen_git.bb>, do_package) failed with exit code '1'
> NOTE: Tasks Summary: Attempted 3642 tasks of which 3641 didn't need to be rerun and 1 failed.
> No currently running tasks (2517 of 3653)
> 
> Summary: 1 task failed:
>    /home/yocto/r-car/meta-virtualization/recipes-extended/xen/xen_git.bb <http://xen_git.bb>, do_package
> Summary: There were 3 WARNING messages shown.
> Summary: There were 3 ERROR messages shown, returning a non-zero exit code.

It's a known issue. Let's say Yocto's specifics. XEN does evolve so its tools
set of libs and apps is being changed. But Yocto tracks all products of
compilation and eager to know what to do with each of those files.

> I am aware I am using a very old BSP. If there is a slightly better version with which to start with, I would greatly like everyone's opinion.

It is mainly not because of BSP, but the meta-virtualization layer version, it
describes how to build and install XEN. As you can see in the last patch to
xen_git.bbappend [1] it is adjusted for 4.10-rc1. I suppose you already did
required changes for the current 4.12-unstable version when saying:

> In previous attempts, I had to modify some recipes to get the compilation working

Good, you have the BSP with XEN built. Could you please reveal your changes to
let me know which XEN you actually built? In the previous email you said, you
tried running freshly built BSP with XEN, and it does not show anything to
display. But what about the console output for that case?

>  From the log it would seem that the xen_git.bb <http://xen_git.bb> in the meta-virtualization layer is being called and thus the recipe is attempting to compile the newest version of Xen.

Right you are. That is the idea. But Yocto's way of BSP compilation makes it
tending to break up. You can edit XEN recipe to build it from a specific
revision, e.g. 4.10.0-rc1. You should replace `${AUTOREV}` in this line [2]
with the correspondent commit-id `24fb44e971a62b345c7b6ca3c03b454a1e150abe` to
do so.
But I suppose it is not what you really need. Since 4.10.0-rc1 there are a
number of changes to scheduling. You might need have those bits up to date for
your work.

> So my second question would be, what version of Xen should I point towards for the board I am using?I guess it is better to use the latest and greatest for your work, so XEN 4.12 unstable should suit you.

Also I do understand that we have our meta-demo layer quite outdated both from
XEN and BSP sides. Renesas's 2.x BSPs are baked with Linux kernel 4.9.x, it is
really old. Even with LK 4.14, we are using from BSP 3.9, I faced an issue [3]
while playing with the latest and greatest XEN.

[1] https://github.com/xen-troops/meta-demo/commit/a4178158ca3ebb739c9bc71c517ec7b65f563218
[2] https://github.com/xen-troops/meta-demo/blob/master/meta-rcar-gen3-xen/recipes-extended/xen/xen_git.bbappend#L9
[3] https://lists.xenproject.org/archives/html/xen-devel/2018-12/msg01373.html

-- 
Sincerely,
Andrii Anisov.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: RT Xen on ARM - R-Car series

[LOPEZ, FUENTES NACARINO Jairo Eduardo]

[-- Attachment #1.1: Type: text/plain, Size: 10065 bytes --]

Hello Andrii,

Thanks again for responding and for clarifying some of the underlying
workings of Yocto.

2018年12月27日(木) 20:07 Andrii Anisov <andrii.anisov@gmail.com>:

> Hello Jairo,
>
> On 25.12.18 18:07, LOPEZ, FUENTES NACARINO Jairo Eduardo wrote:
> > I believe this is the SoC information. If there is any other method of
> extracting the information, please let me know so I can transmit it.
>
> That output gives a full set of required information. Actually, I worried
> you
> might have an obsolete SoC revision. But it is not your case.
>

I am actually very relieved that this is the case.


> > I took a look at [1] and decided to start from scratch to attempt to get
> the minimum workspace functioning.
>
> I mainly mentioned point 1 from those limitations. But you have H3 ES2.0 so
> ready for the latest BSP as well.
>
> > In previous attempts, I had to modify some recipes to get the
> compilation working, but this time I would like to confirm with everyone
> the initial steps before I take them.
> > ERROR: xen-unstable+gitAUTOINC+9d357cbaf7-r0 do_package: QA Issue: xen:
> Files/directories were installed but not shipped in any package:
> >    /usr/lib/libxenfsimage.so
> >    /usr/lib/libxenfsimage.so.4.12
> >    /usr/lib/libxenfsimage.so.4.12.0
> >    /usr/lib/xenfsimage/ext2fs-lib/fsimage.so
> >    /usr/lib/xenfsimage/ufs/fsimage.so
> >    /usr/lib/xenfsimage/fat/fsimage.so
> >    /usr/lib/xenfsimage/iso9660/fsimage.so
> >    /usr/lib/xenfsimage/reiserfs/fsimage.so
> >    /usr/lib/xenfsimage/zfs/fsimage.so
> >    /usr/lib/xen/bin/depriv-fd-checker
> >    /usr/sbin/xenmon
> > Please set FILES such that these items are packaged. Alternatively if
> they are unneeded, avoid installing them or delete them within do_install.
> > xen: 11 installed and not shipped files. [installed-vs-shipped]
> > ERROR: xen-unstable+gitAUTOINC+9d357cbaf7-r0 do_package: Fatal QA errors
> found, failing task.
> > ERROR: xen-unstable+gitAUTOINC+9d357cbaf7-r0 do_package: Function
> failed: do_package
> > ERROR: Logfile of failure stored in:
> /home/yocto/r-car/build/tmp/work/aarch64-poky-linux/xen/unstable+gitAUTOINC+9d357cbaf7-r0/temp/log.do_package.8954
> > ERROR: Task 329
> (/home/yocto/r-car/meta-virtualization/recipes-extended/xen/xen_git.bb <
> http://xen_git.bb>, do_package) failed with exit code '1'
> > NOTE: Tasks Summary: Attempted 3642 tasks of which 3641 didn't need to
> be rerun and 1 failed.
> > No currently running tasks (2517 of 3653)
> >
> > Summary: 1 task failed:
> >    /home/yocto/r-car/meta-virtualization/recipes-extended/xen/xen_git.bb
> <http://xen_git.bb>, do_package
> > Summary: There were 3 WARNING messages shown.
> > Summary: There were 3 ERROR messages shown, returning a non-zero exit
> code.
>
> It's a known issue. Let's say Yocto's specifics. XEN does evolve so its
> tools
> set of libs and apps is being changed. But Yocto tracks all products of
> compilation and eager to know what to do with each of those files.
>
> > I am aware I am using a very old BSP. If there is a slightly better
> version with which to start with, I would greatly like everyone's opinion.
>
> It is mainly not because of BSP, but the meta-virtualization layer
> version, it
> describes how to build and install XEN. As you can see in the last patch to
> xen_git.bbappend [1] it is adjusted for 4.10-rc1. I suppose you already did
> required changes for the current 4.12-unstable version when saying:
>
> > In previous attempts, I had to modify some recipes to get the
> compilation working
>
> Good, you have the BSP with XEN built. Could you please reveal your
> changes to
> let me know which XEN you actually built?



In order to get Xen to compile and to prepare for the boot, I modified the
meta-demo layer to include the Xen files that were not already included and
the hand modified dts file. I have attempted to attach these to the mail.

I am not sure if there is an easier way to get that information, but I
basically did a git show within the tmp/work/aarch64-poky-linux/xen folder
and got the hash 7f28661f6a7ce3d82f881b9afedfebca7f2cf116 which points to
the current head of the master branch of the xen.git repository, if I am
not mistaken.

In the previous email you said, you
> tried running freshly built BSP with XEN, and it does not show anything to
> display. But what about the console output for that case?
>
>
Via serial console I get the following output for the images created after
modifying the dts and applying the patch below:

U-Boot 2015.04 (Jun 22 2018 - 13:36:27)

CPU: Renesas Electronics R8A7795 rev 2.0
Board: H3ULCB
I2C:   ready
DRAM:  3.9 GiB
MMC:   sh-sdhi: 0, sh-sdhi: 1
In:    serial
Out:   serial
Err:   serial
Net:   ravb
Hit any key to stop autoboot:  0
=> setenv bootargs
=> setenv serverip 192.168.1.100
=> tftp 0x48080000 xen-h3ulcb.uImage
ravb Waiting for PHY auto negotiation to complete...... done
ravb: 1000Base/Full
Using ravb device
TFTP from server 192.168.1.100; our IP address is 192.168.1.40
Filename 'xen-h3ulcb.uImage'.
Load address: 0x48080000
Loading: #############################################################
         171.9 KiB/s
done
Bytes transferred = 886160 (d8590 hex)
=> tftp 0x48000000 Image-r8a7795-h3ulcb-xen-local.dtb
ravb:0 is connected to ravb.  Reconnecting to ravb
ravb Waiting for PHY auto negotiation to complete... done
ravb: 1000Base/Full
Using ravb device
TFTP from server 192.168.1.100; our IP address is 192.168.1.40
Filename 'Image-r8a7795-h3ulcb-xen-local.dtb'.
Load address: 0x48000000
Loading: #####
         11.7 KiB/s
done
Bytes transferred = 63810 (f942 hex)
=> tftp 0x7a000000 Image
ravb:0 is connected to ravb.  Reconnecting to ravb
ravb Waiting for PHY auto negotiation to complete........ done
ravb: 1000Base/Full
Using ravb device
TFTP from server 192.168.1.100; our IP address is 192.168.1.40
Filename 'Image'.
Load address: 0x7a000000
Loading: #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         ############################################
         2.1 MiB/s
done
Bytes transferred = 15911424 (f2ca00 hex)
=> bootm 0x48080000 - 0x48000000
## Booting kernel from Legacy Image at 48080000 ...
   Image Name:   XEN
   Image Type:   AArch64 Linux Kernel Image (uncompressed)
   Data Size:    886096 Bytes = 865.3 KiB
   Load Address: 78080000
   Entry Point:  78080000
   Verifying Checksum ... OK
## Flattened Device Tree blob at 48000000
   Booting using the fdt blob at 0x48000000
   Loading Kernel Image ... OK
   Using Device Tree in place at 0000000048000000, end 0000000048012941

Starting kernel ...


And that is it.
Please note that although I am getting the images and dtb from tftp, the
root filesystem is on the microSD card.

I am guessing I should have seen the console output for Xen. I also am not
sure how to test if dom0 is actually running at this stage.


> >  From the log it would seem that the xen_git.bb <http://xen_git.bb> in
> the meta-virtualization layer is being called and thus the recipe is
> attempting to compile the newest version of Xen.
>
> Right you are. That is the idea. But Yocto's way of BSP compilation makes
> it
> tending to break up. You can edit XEN recipe to build it from a specific
> revision, e.g. 4.10.0-rc1. You should replace `${AUTOREV}` in this line [2]
> with the correspondent commit-id
> `24fb44e971a62b345c7b6ca3c03b454a1e150abe` to
> do so.
> But I suppose it is not what you really need. Since 4.10.0-rc1 there are a
> number of changes to scheduling. You might need have those bits up to date
> for
> your work.
>

> > So my second question would be, what version of Xen should I point
> towards for the board I am using?I guess it is better to use the latest and
> greatest for your work, so XEN 4.12 unstable should suit you.
>
>
Yes, I have been tracking some of the scheduling changes being done to Xen
and I would really like to have a relatively newer version up and running.
However, just having something, anything, running at this stage would be
great.

Also I do understand that we have our meta-demo layer quite outdated both
> from
> XEN and BSP sides. Renesas's 2.x BSPs are baked with Linux kernel 4.9.x,
> it is
> really old. Even with LK 4.14, we are using from BSP 3.9, I faced an issue
> [3]
> while playing with the latest and greatest XEN.
>
> [1]
> https://github.com/xen-troops/meta-demo/commit/a4178158ca3ebb739c9bc71c517ec7b65f563218
> [2]
> https://github.com/xen-troops/meta-demo/blob/master/meta-rcar-gen3-xen/recipes-extended/xen/xen_git.bbappend#L9
> [3]
> https://lists.xenproject.org/archives/html/xen-devel/2018-12/msg01373.html
>
> --
> Sincerely,
> Andrii Anisov.
>

Thank you very much for all your support.

Best regards,

Jairo

[-- Attachment #1.2: Type: text/html, Size: 12953 bytes --]

[-- Attachment #2: r8a7795-h3ulcb-xen-local.dts --]
[-- Type: application/octet-stream, Size: 11108 bytes --]

/*
 * Device Tree Source for the H3ULCB (R-Car Starter Kit Premier) board
 *
 * Copyright (C) 2016-2017 Renesas Electronics Corp.
 * Copyright (C) 2016-2017 Cogent Embedded, Inc.
 *
 * This file is licensed under the terms of the GNU General Public License
 * version 2.  This program is licensed "as is" without any warranty of any
 * kind, whether express or implied.
 */

/dts-v1/;
#include "r8a7795.dtsi"
#include <dt-bindings/gpio/gpio.h>
#include <dt-bindings/input/input.h>

/ {
	model = "Renesas H3ULCB board based on r8a7795";
	compatible = "renesas,h3ulcb", "renesas,r8a7795";

	aliases {
		serial0 = &scif2;
		ethernet0 = &avb;
	};

        chosen {
		stdout-path = "serial0:115200n8";
		xen,xen-bootargs = "dom0_mem=752M console=dtuart dtuart=serial0 dom0_max_vcpus=4";
		xen,dom0-bootargs = "console=hvc0 clk_ignore_unused root=/dev/mmcblk1p2 rw rootwait ignore_loglevel cma=128M";

		#address-cells = <2>;
		#size-cells = <2>;
		modules {
			module@0 {
				compatible = "xen,linux-zimage", "xen,multiboot-module";
				reg = <0x0 0x7a000000 0x0 0x02000000>;
			};
		};
	};

	memory@48000000 {
		device_type = "memory";
		/* first 128MB is reserved for secure area. */
		reg = <0x0 0x48000000 0x0 0x38000000>;
	};

	memory@500000000 {
		device_type = "memory";
		reg = <0x5 0x00000000 0x0 0x40000000>;
	};

	memory@600000000 {
		device_type = "memory";
		reg = <0x6 0x00000000 0x0 0x40000000>;
	};

	memory@700000000 {
		device_type = "memory";
		reg = <0x7 0x00000000 0x0 0x40000000>;
	};

	reserved-memory {
		#address-cells = <2>;
		#size-cells = <2>;
		ranges;

		/* device specific region for Lossy Decompression */
		lossy_decompress: linux,lossy_decompress {
			no-map;
			reg = <0x00000000 0x54000000 0x0 0x03000000>;
		};

		/* global autoconfigured region for contiguous allocations */
		linux,cma {
			compatible = "shared-dma-pool";
			reusable;
			reg = <0x00000000 0x58000000 0x0 0x18000000>;
			linux,cma-default;
		};

		/* device specific region for contiguous allocations */
		mmp_reserved: linux,multimedia {
			compatible = "shared-dma-pool";
			reusable;
			reg = <0x00000000 0x70000000 0x0 0x10000000>;
		};
	};

	leds {
		compatible = "gpio-leds";

		led5 {
			gpios = <&gpio6 12 GPIO_ACTIVE_HIGH>;
		};
		led6 {
			gpios = <&gpio6 13 GPIO_ACTIVE_HIGH>;
		};
	};

	keyboard {
		compatible = "gpio-keys";

		key-1 {
			linux,code = <KEY_1>;
			label = "SW3";
			wakeup-source;
			debounce-interval = <20>;
			gpios = <&gpio6 11 GPIO_ACTIVE_LOW>;
		};
	};

	mmngr {
		compatible = "renesas,mmngr";
		memory-region = <&mmp_reserved>, <&lossy_decompress>;
	};

	mmngrbuf {
		compatible = "renesas,mmngrbuf";
	};

	avb-mch {
		compatible = "renesas,avb-mch-gen3";
		reg =	<0 0xec5a0100 0 0x100>;  /* ADG_AVB */
		reg-name = "adg_avb";

		clocks = <&cpg CPG_MOD 922>;
		clock-names = "adg";
	};

	x12_clk: x12 {
		compatible = "fixed-clock";
		#clock-cells = <0>;
		clock-frequency = <24576000>;
	};

	reg_1p8v: regulator0 {
		compatible = "regulator-fixed";
		regulator-name = "fixed-1.8V";
		regulator-min-microvolt = <1800000>;
		regulator-max-microvolt = <1800000>;
		regulator-boot-on;
		regulator-always-on;
	};

	reg_3p3v: regulator1 {
		compatible = "regulator-fixed";
		regulator-name = "fixed-3.3V";
		regulator-min-microvolt = <3300000>;
		regulator-max-microvolt = <3300000>;
		regulator-boot-on;
		regulator-always-on;
	};

	vcc_sdhi0: regulator-vcc-sdhi0 {
		compatible = "regulator-fixed";

		regulator-name = "SDHI0 Vcc";
		regulator-min-microvolt = <3300000>;
		regulator-max-microvolt = <3300000>;

		gpio = <&gpio5 2 GPIO_ACTIVE_HIGH>;
		enable-active-high;
	};

	vccq_sdhi0: regulator-vccq-sdhi0 {
		compatible = "regulator-gpio";

		regulator-name = "SDHI0 VccQ";
		regulator-min-microvolt = <1800000>;
		regulator-max-microvolt = <3300000>;

		gpios = <&gpio5 1 GPIO_ACTIVE_HIGH>;
		gpios-states = <1>;
		states = <3300000 1
			  1800000 0>;
	};

	audio_clkout: audio-clkout {
		/*
		 * This is same as <&rcar_sound 0>
		 * but needed to avoid cs2000/rcar_sound probe dead-lock
		 */
		compatible = "fixed-clock";
		#clock-cells = <0>;
		clock-frequency = <11289600>;
	};

	rsnd_ak4613: sound {
		compatible = "simple-audio-card";

		simple-audio-card,format = "left_j";
		simple-audio-card,bitclock-master = <&sndcpu>;
		simple-audio-card,frame-master = <&sndcpu>;

		sndcpu: simple-audio-card,cpu {
			sound-dai = <&rcar_sound>;
		};

		sndcodec: simple-audio-card,codec {
			sound-dai = <&ak4613>;
		};
	};

	vspm_if {
		compatible = "renesas,vspm_if";
	};

	hdmi0-out {
		compatible = "hdmi-connector";
		type = "a";

		port {
			hdmi0_con: endpoint {
				remote-endpoint = <&rcar_dw_hdmi0_out>;
			};
		};
	};
};

&du_dotclkin0 {
	clock-frequency = <148500000>;
};

&du_dotclkin1 {
	clock-frequency = <33000000>;
};

&du_dotclkin2 {
	clock-frequency = <33000000>;
};

&du_dotclkin3 {
	clock-frequency = <108000000>;
};

&a57_0 {
	cpu-supply = <&vdd_dvfs>;
};

&du {
	status = "okay";

	ports {
		port@1 {
			endpoint {
				remote-endpoint = <&rcar_dw_hdmi0_in>;
			};
		};
	};
};

&hdmi0 {
	status = "okay";

	ports {
		#address-cells = <1>;
		#size-cells = <0>;
		port@0 {
			reg = <0>;
			rcar_dw_hdmi0_in: endpoint {
				remote-endpoint = <&du_out_hdmi0>;
			};
		};
		port@1 {
			reg = <1>;
			rcar_dw_hdmi0_out: endpoint {
				remote-endpoint = <&hdmi0_con>;
			};
		};
	};
};

&extal_clk {
	clock-frequency = <16666666>;
};

&extalr_clk {
	clock-frequency = <32768>;
};

&pfc {
	pinctrl-0 = <&scif_clk_pins>;
	pinctrl-names = "default";

	scif2_pins: scif2 {
		groups = "scif2_data_a";
		function = "scif2";
	};

	scif_clk_pins: scif_clk {
		groups = "scif_clk_a";
		function = "scif_clk";
	};

	i2c2_pins: i2c2 {
		groups = "i2c2_a";
		function = "i2c2";
	};

	avb_pins: avb {
		groups = "avb_mdc";
		function = "avb";
	};

	sdhi0_pins: sd0 {
		groups = "sdhi0_data4", "sdhi0_ctrl";
		function = "sdhi0";
		power-source = <3300>;
	};

	sdhi0_pins_uhs: sd0_uhs {
		groups = "sdhi0_data4", "sdhi0_ctrl";
		function = "sdhi0";
		power-source = <1800>;
	};

	sdhi2_pins: sd2 {
		groups = "sdhi2_data8", "sdhi2_ctrl", "sdhi2_ds";
		function = "sdhi2";
		power-source = <3300>;
	};

	sdhi2_pins_uhs: sd2_uhs {
		groups = "sdhi2_data8", "sdhi2_ctrl", "sdhi2_ds";
		function = "sdhi2";
		power-source = <1800>;
	};

	msiof1_pins: spi2 {
		groups = "msiof1_clk_a", "msiof1_sync_a",
				"msiof1_rxd_a",  "msiof1_txd_a";
		function = "msiof1";
	};

	sound_pins: sound {
		groups = "ssi01239_ctrl", "ssi0_data", "ssi1_data_a";
		function = "ssi";
	};

	sound_clk_pins: sound-clk {
		groups = "audio_clk_a_a", "audio_clk_b_a", "audio_clk_c_a",
			 "audio_clkout_a", "audio_clkout3_a";
		function = "audio_clk";
	};

	usb1_pins: usb1 {
		groups = "usb1";
		function = "usb1";
	};
};

&scif2 {
	pinctrl-0 = <&scif2_pins>;
	pinctrl-names = "default";

	status = "okay";
};

&scif_clk {
	clock-frequency = <14745600>;
	status = "okay";
};

&i2c2 {
	pinctrl-0 = <&i2c2_pins>;
	pinctrl-names = "default";

	status = "okay";

	clock-frequency = <100000>;

	ak4613: codec@10 {
		compatible = "asahi-kasei,ak4613";
		#sound-dai-cells = <0>;
		reg = <0x10>;
		clocks = <&rcar_sound 3>;

		asahi-kasei,in1-single-end;
		asahi-kasei,in2-single-end;
		asahi-kasei,out1-single-end;
		asahi-kasei,out2-single-end;
		asahi-kasei,out3-single-end;
		asahi-kasei,out4-single-end;
		asahi-kasei,out5-single-end;
		asahi-kasei,out6-single-end;
	};

	cs2000: clk-multiplier@4f {
		#clock-cells = <0>;
		compatible = "cirrus,cs2000-cp";
		reg = <0x4f>;
		clocks = <&audio_clkout>, <&x12_clk>;
		clock-names = "clk_in", "ref_clk";

		assigned-clocks = <&cs2000>;
		assigned-clock-rates = <24576000>; /* 1/1 divide */
	};
};

&i2c4 {
	status = "okay";

	clock-frequency = <400000>;
};

&rcar_sound {
	pinctrl-0 = <&sound_pins &sound_clk_pins>;
	pinctrl-names = "default";

	/* Single DAI */
	#sound-dai-cells = <0>;

	/* audio_clkout0/1/2/3 */
	#clock-cells = <1>;
	clock-frequency = <12288000 11289600>;
	clkout-lr-synchronous;

	status = "okay";

	/* update <audio_clk_b> to <cs2000> */
	clocks = <&cpg CPG_MOD 1005>,
		 <&cpg CPG_MOD 1006>, <&cpg CPG_MOD 1007>,
		 <&cpg CPG_MOD 1008>, <&cpg CPG_MOD 1009>,
		 <&cpg CPG_MOD 1010>, <&cpg CPG_MOD 1011>,
		 <&cpg CPG_MOD 1012>, <&cpg CPG_MOD 1013>,
		 <&cpg CPG_MOD 1014>, <&cpg CPG_MOD 1015>,
		 <&cpg CPG_MOD 1022>, <&cpg CPG_MOD 1023>,
		 <&cpg CPG_MOD 1024>, <&cpg CPG_MOD 1025>,
		 <&cpg CPG_MOD 1026>, <&cpg CPG_MOD 1027>,
		 <&cpg CPG_MOD 1028>, <&cpg CPG_MOD 1029>,
		 <&cpg CPG_MOD 1030>, <&cpg CPG_MOD 1031>,
		 <&cpg CPG_MOD 1020>, <&cpg CPG_MOD 1021>,
		 <&cpg CPG_MOD 1020>, <&cpg CPG_MOD 1021>,
		 <&cpg CPG_MOD 1019>, <&cpg CPG_MOD 1018>,
		 <&audio_clk_a>, <&cs2000>,
		 <&audio_clk_c>,
		 <&cpg CPG_CORE R8A7795_CLK_S0D4>;

	rcar_sound,dai {
		dai0 {
			playback = <&ssi0 &src0 &dvc0>;
			capture  = <&ssi1 &src1 &dvc1>;
		};
	};
};

&sdhi0 {
	pinctrl-0 = <&sdhi0_pins>;
	pinctrl-1 = <&sdhi0_pins_uhs>;
	pinctrl-names = "default", "state_uhs";

	vmmc-supply = <&vcc_sdhi0>;
	vqmmc-supply = <&vccq_sdhi0>;
	cd-gpios = <&gpio3 12 GPIO_ACTIVE_LOW>;
	bus-width = <4>;
	sd-uhs-sdr50;
	sd-uhs-sdr104;
	status = "okay";
};

&sdhi2 {
	/* used for on-board 8bit eMMC */
	pinctrl-0 = <&sdhi2_pins>;
	pinctrl-1 = <&sdhi2_pins_uhs>;
	pinctrl-names = "default", "state_uhs";

	vmmc-supply = <&reg_3p3v>;
	vqmmc-supply = <&reg_1p8v>;
	mmc-hs200-1_8v;
	mmc-hs400-1_8v;
	bus-width = <8>;
	non-removable;
	status = "okay";
};

&ssi1 {
	shared-pin;
};

&wdt0 {
	timeout-sec = <60>;
	status = "okay";
};

&audio_clk_a {
	clock-frequency = <22579200>;
};

&i2c_dvfs {
	status = "okay";
	clock-frequency = <400000>;

	vdd_dvfs: regulator@30 {
		compatible = "rohm,bd9571mwv";
		reg = <0x30>;

		regulator-min-microvolt = <750000>;
		regulator-max-microvolt = <1030000>;
		regulator-boot-on;
		regulator-always-on;
	};
};

&avb {
	pinctrl-0 = <&avb_pins>;
	pinctrl-names = "default";
	renesas,no-ether-link;
	phy-handle = <&phy0>;
	status = "okay";
	phy-gpios = <&gpio2 11 GPIO_ACTIVE_LOW>;
	phy-reset-gpios = <&gpio2 10 GPIO_ACTIVE_LOW>;

	phy0: ethernet-phy@0 {
		rxc-skew-ps = <1500>;
		rxdv-skew-ps = <420>; /* default */
		rxd0-skew-ps = <420>; /* default */
		rxd1-skew-ps = <420>; /* default */
		rxd2-skew-ps = <420>; /* default */
		rxd3-skew-ps = <420>; /* default */
		txc-skew-ps = <900>; /* default */
		txen-skew-ps = <420>; /* default */
		txd0-skew-ps = <420>; /* default */
		txd1-skew-ps = <420>; /* default */
		txd2-skew-ps = <420>; /* default */
		txd3-skew-ps = <420>; /* default */
		reg = <0>;
		interrupt-parent = <&gpio2>;
		interrupts = <11 IRQ_TYPE_LEVEL_LOW>;
	};
};

&usb2_phy1 {
	pinctrl-0 = <&usb1_pins>;
	pinctrl-names = "default";

	status = "okay";
};

&ehci1 {
	status = "okay";
};

&ohci1 {
	status = "okay";
};

&msiof_ref_clk {
	clock-frequency = <66666666>;
};

&msiof1 {
	pinctrl-0 = <&msiof1_pins>;
	pinctrl-names = "default";
	status = "okay";

	spidev@0 {
		compatible = "renesas,sh-msiof";
		reg = <0>;
		spi-max-frequency = <66666666>;
		spi-cpha;
		spi-cpol;
	};
};

&vspbc {
	status = "okay";
};

&vspbd {
	status = "okay";
};

&vspi0 {
	status = "okay";
};

&vspi1 {
	status = "okay";
};

[-- Attachment #3: xen-20181228.patch --]
[-- Type: text/x-patch, Size: 3414 bytes --]

diff --git a/meta-rcar-gen3-xen/recipes-extended/xen/xen_git.bbappend b/meta-rcar-gen3-xen/recipes-extended/xen/xen_git.bbappend
index b48a710..9021cee 100644
--- a/meta-rcar-gen3-xen/recipes-extended/xen/xen_git.bbappend
+++ b/meta-rcar-gen3-xen/recipes-extended/xen/xen_git.bbappend
@@ -61,3 +61,99 @@ RDEPENDS_${PN}-base_remove = "\
     ${PN}-libvhd \
     "
 
+FILES_${PN}-staticdev += "\
+    ${libdir}/libxenforeignmemory.a \
+    ${libdir}/libxentoollog.a \
+    ${libdir}/libxengnttab.a \
+    ${libdir}/libxendevicemodel.a \
+    ${libdir}/libxenevtchn.a \
+    ${libdir}/libxencall.a \
+    ${libdir}/libxentoolcore.a \
+    "
+
+FILES_${PN}-libxencall-dev += " \
+    ${libdir}/libxencall.so \
+    ${datadir}/pkgconfig/xencall.pc \
+    "
+
+FILES_${PN}-libxenctrl-dev += " \
+    ${libdir}/libxenctrl.so \
+    ${datadir}/pkgconfig/xencontrol.pc \
+    "
+
+FILES_${PN}-libxendevicemodel += "${libdir}/libxendevicemodel.so.*"
+FILES_${PN}-libxendevicemodel-dev += " \
+    ${libdir}/libxendevicemodel.so \
+    ${datadir}/pkgconfig/xendevicemodel.pc \
+    "
+
+FILES_${PN}-libxenevtchn-dev += " \
+    ${libdir}/libxenevtchn.so \
+    ${datadir}/pkgconfig/xenevtchn.pc \
+    "
+
+FILES_${PN}-libxenforeignmemory-dev += "\
+    ${libdir}/libxenforeignmemory.so \
+    ${datadir}/pkgconfig/xenforeignmemory.pc \
+    "
+
+FILES_${PN}-libxengnttab-dev += " \
+    ${libdir}/libxengnttab.so \
+    ${datadir}/pkgconfig/xengnttab.pc \
+    "
+
+FILES_${PN}-libxenguest-dev += " \
+    ${libdir}/libxenguest.so \
+    ${datadir}/pkgconfig/xenguest.pc \
+    "
+
+FILES_${PN}-libxenstat-dev += " \
+    ${libdir}/libxenstat.so \
+    ${datadir}/pkgconfig/xenstat.pc \
+    "
+
+FILES_${PN}-libxenstore-dev += " \
+    ${libdir}/libxenstore.so \
+    ${datadir}/pkgconfig/xenstore.pc \
+    "
+
+FILES_${PN}-libxentoolcore += "${libdir}/libxentoolcore.so.*"
+FILES_${PN}-libxentoolcore-dev += " \
+    ${libdir}/libxentoolcore.so \
+    ${datadir}/pkgconfig/xentoolcore.pc \
+    "
+
+FILES_${PN}-libxentoollog-dev += " \
+    ${libdir}/libxentoollog.so \
+    ${datadir}/pkgconfig/xentoollog.pc \
+    "
+
+FILES_${PN}-libxenvchan-dev += " \
+    ${libdir}/libxenvchan.so \
+    ${datadir}/pkgconfig/xenvchan.pc \
+    "
+
+FILES_${PN}-misc += "\
+    ${sbindir}/xen-diag \
+    ${libdir}/xen/bin/depriv-fd-checker \
+    "
+
+FILES_${PN}-efi = "\
+    ${exec_prefix}/lib64/efi/xen* \
+    "
+
+FILES_${PN}-xenmon += "\
+    ${sbindir}/xenmon \
+    "
+
+FILES_${PN}-fsimage += "\
+    ${libdir}/xenfsimage/*/fsimage.so \
+    "
+
+FILES_${PN}-libfsimage += "\
+    ${libdir}/libxenfsimage.so.* \
+    "
+
+FILES_${PN}-libfsimage-dev += "\
+    ${libdir}/libxenfsimage.so \
+    "
diff --git a/meta-rcar-gen3-xen/recipes-kernel/linux/linux-renesas_4.9.bbappend b/meta-rcar-gen3-xen/recipes-kernel/linux/linux-renesas_4.9.bbappend
index 200b299..50ea38f 100644
--- a/meta-rcar-gen3-xen/recipes-kernel/linux/linux-renesas_4.9.bbappend
+++ b/meta-rcar-gen3-xen/recipes-kernel/linux/linux-renesas_4.9.bbappend
@@ -11,3 +11,8 @@ SRC_URI_append_salvator-x = " \
     file://r8a7795-salvator-x-xen.dts;subdir=git/arch/${ARCH}/boot/dts/renesas \
 "
 KERNEL_DEVICETREE_append_salvator-x = "renesas/r8a7795-salvator-x-xen.dtb"
+
+SRC_URI_append_h3ulcb = " \
+file://r8a7795-h3ulcb-xen-local.dts;subdir=git/arch/${ARCH}/boot/dts/renesas \
+"
+KERNEL_DEVICETREE_append_h3ulcb = "renesas/r8a7795-h3ulcb-xen-local.dtb"

[-- Attachment #4: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: RT Xen on ARM - R-Car series

[Andrii Anisov]

Hello Jairo,

This is just a quick reply.

On 28.12.18 17:22, LOPEZ, FUENTES NACARINO Jairo Eduardo wrote:
> Via serial console I get the following output for the images created after modifying the dts and applying the patch below:
> 
> U-Boot 2015.04 (Jun 22 2018 - 13:36:27)

I see, you did not update bootloaders on the board.
And that is the problem. Xen is started in NS EL1 instead of EL2, as required.
You would correspondent prints if built XEN with EARLY_PRINTK enabled. But it is quite tricky to do with yocto.
Please give me some time to find the instruction to update bootloaders on ulcb.

-- 
Sincerely,
Andrii Anisov.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: RT Xen on ARM - R-Car series

[Andrii Anisov]

On 28.12.18 10:28, Andrii Anisov wrote:
> Hello Jairo,
> 
> This is just a quick reply.
> 
> On 28.12.18 17:22, LOPEZ, FUENTES NACARINO Jairo Eduardo wrote:
>> Via serial console I get the following output for the images created after modifying the dts and applying the patch below:
>>
>> U-Boot 2015.04 (Jun 22 2018 - 13:36:27)
> 
> I see, you did not update bootloaders on the board.

Actually it is clearly stated on the wiki page [1]

[1] https://wiki.xenproject.org/wiki/Xen_ARM_with_Virtualization_Extensions/Salvator-X#BL2_.26_U-boot_update_procedure

-- 
Sincerely,
Andrii Anisov.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: RT Xen on ARM - R-Car series

[Andrii Anisov]

On 28.12.18 10:28, Andrii Anisov wrote:
> Please give me some time to find the instruction to update bootloaders on ulcb.

Here is the instruction for h3ulcb to flash the firmware for Yocto 2.19 [1].
Yet it is quite old.

[1] https://elinux.org/index.php?title=R-Car/Boards/H3SK&oldid=449556

-- 
Sincerely,
Andrii Anisov.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: RT Xen on ARM - R-Car series

[LOPEZ, FUENTES NACARINO Jairo Eduardo]

[-- Attachment #1.1: Type: text/plain, Size: 684 bytes --]

Andrii,

Sorry for not responding sooner.

First of all happy new year!

I seriously overlooked the firmware update. I will attempt that next week
and see if the boot is successful.

Thanks for the help.

Best regards,

Jairo


2018年12月28日(金) 17:39 Andrii Anisov <andrii.anisov@gmail.com>:

>
>
> On 28.12.18 10:28, Andrii Anisov wrote:
> > Please give me some time to find the instruction to update bootloaders
> on ulcb.
>
> Here is the instruction for h3ulcb to flash the firmware for Yocto 2.19
> [1].
> Yet it is quite old.
>
> [1] https://elinux.org/index.php?title=R-Car/Boards/H3SK&oldid=449556
>
> --
> Sincerely,
> Andrii Anisov.
>

[-- Attachment #1.2: Type: text/html, Size: 1254 bytes --]

[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: RT Xen on ARM - R-Car series

[LOPEZ, FUENTES NACARINO Jairo Eduardo]

[-- Attachment #1.1: Type: text/plain, Size: 5654 bytes --]

Andrii,

I have attempted to update the bootloader from the srec files created by
the bitbake command.

I believe I have been able to update it, but I am still obtaining the same
results. The serial output is shown below:

[    0.000161] NOTICE:  BL2: R-Car Gen3 Initial Program Loader(CA57)
Rev.1.0.17
[    0.005721] NOTICE:  BL2: PRR is R-Car H3 Ver2.0
[    0.010304] NOTICE:  BL2: Board is Starter Kit Rev1.0
[    0.015331] NOTICE:  BL2: Boot device is HyperFlash(80MHz)
[    0.020757] NOTICE:  BL2: LCM state is CM
[    0.024802] NOTICE:  BL2: AVS setting succeeded. DVFS_SetVID=0x53
[    0.030786] NOTICE:  BL2: DDR3200(rev.0.28)NOTICE:  [COLD_BOOT]NOTICE:
..0
[    0.059996] NOTICE:  BL2: DRAM Split is 4ch
[    0.063882] NOTICE:  BL2: QoS is default setting(rev.0.18)
[    0.069381] NOTICE:  BL2: Lossy Decomp areas
[    0.073559] NOTICE:       Entry 0: DCMPAREACRAx:0x80000540
DCMPAREACRBx:0x570
[    0.080643] NOTICE:       Entry 1: DCMPAREACRAx:0x40000000
DCMPAREACRBx:0x0
[    0.087555] NOTICE:       Entry 2: DCMPAREACRAx:0x20000000
DCMPAREACRBx:0x0
[    0.094470] NOTICE:  BL2: v1.3(release):b15fefa
[    0.098960] NOTICE:  BL2: Built : 12:21:32, Dec 28 2018
[    0.104147] NOTICE:  BL2: Normal boot
[    0.107788] NOTICE:  BL2: dst=0xe6322210 src=0x8180000 len=512(0x200)
[    0.114335] NOTICE:  BL2: dst=0x43f00000 src=0x8180400 len=6144(0x1800)
[    0.120797] NOTICE:  BL2: dst=0x44000000 src=0x81c0000 len=65536(0x10000)
[    0.128022] NOTICE:  BL2: dst=0x44100000 src=0x8200000
len=524288(0x80000)
[    0.138503] NOTICE:  BL2: dst=0x50000000 src=0x8640000
len=1048576(0x100000)


U-Boot 2015.04 (Dec 28 2018 - 02:46:24)

CPU: Renesas Electronics R8A7795 rev 2.0
Board: H3ULCB
I2C:   ready
DRAM:  3.9 GiB
MMC:   sh-sdhi: 0, sh-sdhi: 1
In:    serial
Out:   serial
Err:   serial
Net:   ravb
Hit any key to stop autoboot:  0
=> setenv bootargs
=> setenv serverip 192.168.1.100
=> tftp 0x48080000 xen-h3ulcb.uImage
ravb Waiting for PHY auto negotiation to complete......... TIMEOUT !
Half
Using ravb device
TFTP from server 192.168.1.100; our IP address is 192.168.1.40
Filename 'xen-h3ulcb.uImage'.
Load address: 0x48080000
Loading: *
ARP Retry count exceeded; starting again
ravb:0 is connected to ravb.  Reconnecting to ravb
ravb Waiting for PHY auto negotiation to complete......... TIMEOUT !
Half

ARP Retry count exceeded; starting again
ravb:0 is connected to ravb.  Reconnecting to ravb
ravb Waiting for PHY auto negotiation to complete... done
ravb: 1000Base/Full
Using ravb device
TFTP from server 192.168.1.100; our IP address is 192.168.1.40
Filename 'xen-h3ulcb.uImage'.
Load address: 0x48080000
Loading: ###########################################################
         25.4 MiB/s
done
Bytes transferred = 852368 (d0190 hex)
=> tftp 0x48000000 Image-r8a7795-h3ulcb-xen-local.dtb
ravb:0 is connected to ravb.  Reconnecting to ravb
ravb Waiting for PHY auto negotiation to complete..... done
ravb: 1000Base/Full
Using ravb device
TFTP from server 192.168.1.100; our IP address is 192.168.1.40
Filename 'Image-r8a7795-h3ulcb-xen-local.dtb'.
Load address: 0x48000000
Loading: #####
         11.7 KiB/s
done
Bytes transferred = 63778 (f922 hex)
=> tftp 0x7a000000 h3ulcb-image
ravb:0 is connected to ravb.  Reconnecting to ravb
ravb Waiting for PHY auto negotiation to complete... done
ravb: 1000Base/Full
Using ravb device
TFTP from server 192.168.1.100; our IP address is 192.168.1.40
Filename 'h3ulcb-image'.
Load address: 0x7a000000
Loading: #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         ########################################
         2.1 MiB/s
done
Bytes transferred = 15843840 (f1c200 hex)
=> bootm 0x48080000 - 0x48000000
## Booting kernel from Legacy Image at 48080000 ...
   Image Name:   XEN
   Image Type:   AArch64 Linux Kernel Image (uncompressed)
   Data Size:    852304 Bytes = 832.3 KiB
   Load Address: 78080000
   Entry Point:  78080000
   Verifying Checksum ... OK
## Flattened Device Tree blob at 48000000
   Booting using the fdt blob at 0x48000000
   Loading Kernel Image ... OK
   Using Device Tree in place at 0000000048000000, end 0000000048012921

Starting kernel ...

</end output>

I have noticed that the uboot date has not changed from 2015.04 although
the build date is from the last time I ran a bitbake command.

I do not know what version of uboot should be used without breaking the
current compiling versions.

Are there any other modifications that I can attempt?

Thank you very much for your time and patience.

Best regards,

Jairo

[-- Attachment #1.2: Type: text/html, Size: 6702 bytes --]

[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: RT Xen on ARM - R-Car series

[LOPEZ, FUENTES NACARINO Jairo Eduardo]

[-- Attachment #1.1: Type: text/plain, Size: 20537 bytes --]

Andrii,

Due to some events on my side, I had to move from the R-Car H3 I was using
to the R-Car M3 in my posession.

I followed your initial advice and updated my environment to a newer
version, opting to stick to the rocko version for the m3ulcb.

The Yocto build configuration is shown below:

Build Configuration:
BB_VERSION           = "1.36.0"
BUILD_SYS            = "x86_64-linux"
NATIVELSBSTRING      = "universal-4.9"
TARGET_SYS           = "aarch64-poky-linux"
MACHINE              = "m3ulcb"
DISTRO               = "poky"
DISTRO_VERSION       = "2.4.2"
TUNE_FEATURES        = "aarch64 cortexa57-cortexa53"
TARGET_FPU           = ""
SOC_FAMILY           = "rcar-gen3:r8a7796"
meta
meta-poky
meta-yocto-bsp       = "HEAD:342fbd6a3e57021c8e28b124b3adb241936f3d9d"
meta-rcar-gen3       = "rocko:8af0b7d6e445b532088a068dc012757001be3a1f"
meta-optee           = "rocko:75dfb67bbb14a70cd47afda9726e2e1c76731885"
meta-oe
meta-python
meta-networking      = "rocko:eae996301d9c097bcbeb8046f08041dc82bb62f8"
meta-selinux         = "rocko:d855c624f32c5e599bf27e06cb8f5b25b3aae12d"
meta-virtualization  = "rocko:b704c689b67639214b9568a3d62e82df27e9434f"
meta-rcar-gen3-xen   = "master:60699c631d541aeeaebaeec9a087efed9385ee42"

I had to modify the dtb to the version shown below:

// SPDX-License-Identifier:
GPL-2.0
/*

 * Device Tree Source for the M3ULCB (R-Car Starter Kit Pro)
board
 *

 * Copyright (C) 2016-2018 Renesas Electronics
Corp.
 * Copyright (C) 2016 Cogent Embedded,
Inc.
 */



/dts-v1/;

#include
"r8a7796.dtsi"

#include
"ulcb.dtsi"



/
{

        model = "Renesas M3ULCB board based on
r8a7796";
        compatible = "renesas,m3ulcb",
"renesas,r8a7796";


        cpus
{
                idle-states
{
                        /delete-node/
cpu-sleep-1;

};

};



        memory@48000000
{
                device_type =
"memory";
                /* first 128MB is reserved for secure area.
*/
                reg = <0x0 0x48000000 0x0
0x38000000>;
        };

        memory@600000000 {
                device_type = "memory";
                reg = <0x6 0x00000000 0x0 0x40000000>;
        };

        reserved-memory {
                #address-cells = <2>;
                #size-cells = <2>;
                ranges;

                /* device specific region for Lossy Decompression */
                lossy_decompress: linux,lossy_decompress@54000000 {
                        no-map;
                        reg = <0x00000000 0x54000000 0x0 0x03000000>;
                };

                /* global autoconfigured region for contiguous allocations
*/
                linux,cma@57000000 {
                        compatible = "shared-dma-pool";
                        reusable;
                        reg = <0x00000000 0x57000000 0x0 0x19000000>;
                        linux,cma-default;
                };

                /* device specific region for contiguous allocations */
                mmp_reserved: linux,multimedia@70000000 {
                        compatible = "shared-dma-pool";
                        reusable;
                        reg = <0x00000000 0x70000000 0x0 0x10000000>;
                };
        };

        chosen {
                stdout-path = "serial0:115200n8";
                xen,xen-bootargs = "dom0_mem=752M console=dtuart
dtuart=serial0 dom0_max_vcpus=4";
                xen,dom0-bootargs = "console=hvc0 clk_ignore_unused
root=/dev/mmcblk1p2 rw rootwait ignore_loglevel cma=128M";

                #address-cells = <2>;
                #size-cells = <2>;
                modules {
                        module@0 {
                                compatible = "xen,linux-zimage",
"xen,multiboot-module";
                                reg = <0x0 0x7a000000 0x0 0x02000000>;
                        };
                };
        };

        mmngr {
                compatible = "renesas,mmngr";
                memory-region = <&mmp_reserved>, <&lossy_decompress>;
        };

        mmngrbuf {
                compatible = "renesas,mmngrbuf";
        };

        vspm_if {
                compatible = "renesas,vspm_if";
        };

        versaclock5_out3: versaclk-3 {
                compatible = "fixed-clock";
                #clock-cells = <0>;
                /* Initial value of versaclock out3 */
                clock-frequency = <33000000>;
        };
};

&a53_0 {
        /delete-property/ cpu-idle-states;
};

&a53_1 {
        /delete-property/ cpu-idle-states;
};

&a53_2 {
        /delete-property/ cpu-idle-states;
};

&a53_3 {
        /delete-property/ cpu-idle-states;
};

&du {
        clocks = <&cpg CPG_MOD 724>,
                 <&cpg CPG_MOD 723>,
                 <&cpg CPG_MOD 722>,
                 <&versaclock5 1>,
                 <&versaclock5_out3>,
                 <&versaclock5 2>;
        clock-names = "du.0", "du.1", "du.2",
                      "dclkin.0", "dclkin.1", "dclkin.2";
};

&vspb {
        status = "okay";
};

&vspi0 {
        status = "okay";
};

With the new information, I finally got some output for Xen, but was met
with a completely different issue for which I can't find any information
for the error I am receiving.

I realize that I may have an issue with the version of the board I am
using, but a simple online search doesn't throw any red flags. The serial
console output is below:

[    0.000177] NOTICE:  BL2: R-Car M3 Initial Program Loader(CA57)
[    0.004613] NOTICE:  BL2: Initial Program Loader(Rev.2.0.0)
[    0.010147] NOTICE:  BL2: PRR is R-Car M3 Ver.1.0
[    0.014815] NOTICE:  BL2: Board is Starter Kit Rev.1.0
[    0.019928] NOTICE:  BL2: Boot device is HyperFlash(80MHz)
[    0.025354] NOTICE:  BL2: LCM state is CM
[    0.029395] NOTICE:  BL2: AVS setting succeeded. DVFS_SetVID=0x53
[    0.035383] NOTICE:  BL2: DDR3200(rev.0.34)
[    0.046779] NOTICE:  BL2: [COLD_BOOT]
[    0.085117] NOTICE:  BL2: DRAM Split is 2ch
[    0.087812] NOTICE:  BL2: QoS is default setting(rev.0.19)
[    0.093315] NOTICE:  BL2: Lossy Decomp areas
[    0.097489] NOTICE:       Entry 0: DCMPAREACRAx:0x80000540
DCMPAREACRBx:0x570
[    0.104573] NOTICE:       Entry 1: DCMPAREACRAx:0x40000000
DCMPAREACRBx:0x0
[    0.111485] NOTICE:       Entry 2: DCMPAREACRAx:0x20000000
DCMPAREACRBx:0x0
[    0.118400] NOTICE:  BL2: v1.5(release):556a11e
[    0.122890] NOTICE:  BL2: Built : 18:03:48, Jan 11 2019
[    0.128077] NOTICE:  BL2: Normal boot
[    0.131718] NOTICE:  BL2: dst=0xe6321100 src=0x8180000 len=512(0x200)
[    0.138106] NOTICE:  BL2: dst=0x43f00000 src=0x8180400 len=6144(0x1800)
[    0.144726] NOTICE:  BL2: dst=0x44000000 src=0x81c0000 len=65536(0x10000)
[    0.151952] NOTICE:  BL2: dst=0x44100000 src=0x8200000
len=1048576(0x100000)
[    0.166777] NOTICE:  BL2: dst=0x50000000 src=0x8640000
len=1048576(0x100000)
[    0.180752] NOTICE:  BL2: Booting BL31


U-Boot 2018.09 (Jan 12 2019 - 03:03:43 +0900)

CPU: Renesas Electronics R8A7796 rev 1.0
Model: Renesas M3ULCB board based on r8a7796
DRAM:  1.9 GiB
Bank #0: 0x048000000 - 0x07fffffff, 896 MiB
Bank #1: 0x600000000 - 0x63fffffff, 1 GiB

MMC:   sd@ee100000: 0, sd@ee140000: 1
Loading Environment from MMC... communication out of sync
communication out of sync
communication out of sync
communication out of sync
communication out of sync
communication out of sync
communication out of sync
communication out of sync
communication out of sync
communication out of sync
communication out of sync
communication out of sync
OK
In:    serial@e6e88000
Out:   serial@e6e88000
Err:   serial@e6e88000
Net:   eth0: ethernet@e6800000
Hit any key to stop autoboot:  0
ethernet@e6800000 Waiting for PHY auto negotiation to complete... done
Using ethernet@e6800000 device
TFTP from server 192.168.1.100; our IP address is 192.168.1.3
Filename 'Image'.
Load address: 0x48080000
Loading: #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         ############################################
         2.7 MiB/s
done
Bytes transferred = 15911424 (f2ca00 hex)
Using ethernet@e6800000 device
TFTP from server 192.168.1.100; our IP address is 192.168.1.3
Filename 'Image-r8a7796-m3ulcb.dtb'.
Load address: 0x48000000
Loading: *
TFTP error: 'File not found' (1)
Not retrying...
ERROR: Did not find a cmdline Flattened Device Tree
Could not find a valid device tree
=> setenv bootargs
=> tftp 0x48080000 xen-m3ulcb.uImage
Using ethernet@e6800000
device
TFTP from server 192.168.1.100; our IP address is
192.168.1.3
Filename
'xen-m3ulcb.uImage'.
Load address:
0x48080000
Loading:
###############################################################
         28.3
MiB/s
done

Bytes transferred = 918928 (e0590
hex)
=> tftp 0x48000000
Image-r8a7796-m3ulcb-xen-local.dtb
Using ethernet@e6800000
device
TFTP from server 192.168.1.100; our IP address is
192.168.1.3
Filename
'Image-r8a7796-m3ulcb-xen-local.dtb'.
Load address:
0x48000000
Loading:
#####
         21.8
MiB/s
done

Bytes transferred = 68545 (10bc1
hex)
=> tftp 0x7a000000
Image-m3ulcb.bin
Using ethernet@e6800000
device
TFTP from server 192.168.1.100; our IP address is 192.168.1.3
Filename 'Image-m3ulcb.bin'.
Load address: 0x7a000000
Loading: #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         ################################################################
         28.1 MiB/s
done
Bytes transferred = 19059200 (122d200 hex)
=> bootm 0x48080000 - 0x48000000
## Booting kernel from Legacy Image at 48080000 ...
   Image Name:   XEN
   Image Type:   AArch64 Linux Kernel Image (uncompressed)
   Data Size:    918864 Bytes = 897.3 KiB
   Load Address: 78080000
   Entry Point:  78080000
   Verifying Checksum ... OK
## Flattened Device Tree blob at 48000000
   Booting using the fdt blob at 0x48000000
   Loading Kernel Image ... OK
   Using Device Tree in place at 0000000048000000, end 0000000048013bc0

Starting kernel ...

 Xen 4.12-unstable
(XEN) Xen version 4.12-unstable (jelfn@waseda.jp) (aarch64-poky-linux-gc9
(XEN) Latest ChangeSet: Mon Dec 17 09:22:59 2018 +0000 git:a5b0eb3-dirty
(XEN) Processor: 411fd073: "ARM Limited", variant: 0x1, part 0xd07, rev 0x3
(XEN) 64-bit Execution:
(XEN)   Processor Features: 0000000000002222 0000000000000000
(XEN)     Exception Levels: EL3:64+32 EL2:64+32 EL1:64+32 EL0:64+32
(XEN)     Extensions: FloatingPoint AdvancedSIMD
(XEN)   Debug Features: 0000000010305106 0000000000000000
(XEN)   Auxiliary Features: 0000000000000000 0000000000000000
(XEN)   Memory Model Features: 0000000000001124 0000000000000000
(XEN)   ISA Features:  0000000000011120 0000000000000000
(XEN) 32-bit Execution:
(XEN)   Processor Features: 00000131:00011011
(XEN)     Instruction Sets: AArch32 A32 Thumb Thumb-2 Jazelle
(XEN)     Extensions: GenericTimer Security
(XEN)   Debug Features: 03010066
(XEN)   Auxiliary Features: 00000000
(XEN)   Memory Model Features: 10201105 40000000 01260000 02102211
(XEN)  ISA Features: 02101110 13112111 21232042 01112131 00011142 00011121
(XEN) Using SMC Calling Convention v1.1
(XEN) Using PSCI v1.1
(XEN) SMP: Allowing 6 CPUs
(XEN)
(XEN) ****************************************
(XEN) Panic on CPU 0:
(XEN) Timer: Unable to retrieve IRQ 0 from the device tree
(XEN) ****************************************
(XEN)
(XEN) Reboot in five seconds...

Is there anything I have overlooked in the compilation or configuration for
Xen?

Thank you very much for all the help you have offered up to this point. I
am at least happy to get some output, even if it is on the R-Car M3 board
in my possession.

Best regards,

Jairo


2019年1月9日(水) 2:04 LOPEZ, FUENTES NACARINO Jairo Eduardo <
jairo@ruri.waseda.jp>:

> Andrii,
>
> I have attempted to update the bootloader from the srec files created by
> the bitbake command.
>
> I believe I have been able to update it, but I am still obtaining the same
> results. The serial output is shown below:
>
> [    0.000161] NOTICE:  BL2: R-Car Gen3 Initial Program Loader(CA57)
> Rev.1.0.17
> [    0.005721] NOTICE:  BL2: PRR is R-Car H3 Ver2.0
> [    0.010304] NOTICE:  BL2: Board is Starter Kit Rev1.0
> [    0.015331] NOTICE:  BL2: Boot device is HyperFlash(80MHz)
> [    0.020757] NOTICE:  BL2: LCM state is CM
> [    0.024802] NOTICE:  BL2: AVS setting succeeded. DVFS_SetVID=0x53
> [    0.030786] NOTICE:  BL2: DDR3200(rev.0.28)NOTICE:  [COLD_BOOT]NOTICE:
> ..0
> [    0.059996] NOTICE:  BL2: DRAM Split is 4ch
> [    0.063882] NOTICE:  BL2: QoS is default setting(rev.0.18)
> [    0.069381] NOTICE:  BL2: Lossy Decomp areas
> [    0.073559] NOTICE:       Entry 0: DCMPAREACRAx:0x80000540
> DCMPAREACRBx:0x570
> [    0.080643] NOTICE:       Entry 1: DCMPAREACRAx:0x40000000
> DCMPAREACRBx:0x0
> [    0.087555] NOTICE:       Entry 2: DCMPAREACRAx:0x20000000
> DCMPAREACRBx:0x0
> [    0.094470] NOTICE:  BL2: v1.3(release):b15fefa
> [    0.098960] NOTICE:  BL2: Built : 12:21:32, Dec 28 2018
> [    0.104147] NOTICE:  BL2: Normal boot
> [    0.107788] NOTICE:  BL2: dst=0xe6322210 src=0x8180000 len=512(0x200)
> [    0.114335] NOTICE:  BL2: dst=0x43f00000 src=0x8180400 len=6144(0x1800)
> [    0.120797] NOTICE:  BL2: dst=0x44000000 src=0x81c0000
> len=65536(0x10000)
> [    0.128022] NOTICE:  BL2: dst=0x44100000 src=0x8200000
> len=524288(0x80000)
> [    0.138503] NOTICE:  BL2: dst=0x50000000 src=0x8640000
> len=1048576(0x100000)
>
>
> U-Boot 2015.04 (Dec 28 2018 - 02:46:24)
>
> CPU: Renesas Electronics R8A7795 rev 2.0
> Board: H3ULCB
> I2C:   ready
> DRAM:  3.9 GiB
> MMC:   sh-sdhi: 0, sh-sdhi: 1
> In:    serial
> Out:   serial
> Err:   serial
> Net:   ravb
> Hit any key to stop autoboot:  0
> => setenv bootargs
> => setenv serverip 192.168.1.100
> => tftp 0x48080000 xen-h3ulcb.uImage
> ravb Waiting for PHY auto negotiation to complete......... TIMEOUT !
> Half
> Using ravb device
> TFTP from server 192.168.1.100; our IP address is 192.168.1.40
> Filename 'xen-h3ulcb.uImage'.
> Load address: 0x48080000
> Loading: *
> ARP Retry count exceeded; starting again
> ravb:0 is connected to ravb.  Reconnecting to ravb
> ravb Waiting for PHY auto negotiation to complete......... TIMEOUT !
> Half
>
> ARP Retry count exceeded; starting again
> ravb:0 is connected to ravb.  Reconnecting to ravb
> ravb Waiting for PHY auto negotiation to complete... done
> ravb: 1000Base/Full
> Using ravb device
> TFTP from server 192.168.1.100; our IP address is 192.168.1.40
> Filename 'xen-h3ulcb.uImage'.
> Load address: 0x48080000
> Loading: ###########################################################
>          25.4 MiB/s
> done
> Bytes transferred = 852368 (d0190 hex)
> => tftp 0x48000000 Image-r8a7795-h3ulcb-xen-local.dtb
> ravb:0 is connected to ravb.  Reconnecting to ravb
> ravb Waiting for PHY auto negotiation to complete..... done
> ravb: 1000Base/Full
> Using ravb device
> TFTP from server 192.168.1.100; our IP address is 192.168.1.40
> Filename 'Image-r8a7795-h3ulcb-xen-local.dtb'.
> Load address: 0x48000000
> Loading: #####
>          11.7 KiB/s
> done
> Bytes transferred = 63778 (f922 hex)
> => tftp 0x7a000000 h3ulcb-image
> ravb:0 is connected to ravb.  Reconnecting to ravb
> ravb Waiting for PHY auto negotiation to complete... done
> ravb: 1000Base/Full
> Using ravb device
> TFTP from server 192.168.1.100; our IP address is 192.168.1.40
> Filename 'h3ulcb-image'.
> Load address: 0x7a000000
> Loading: #################################################################
>          #################################################################
>          #################################################################
>          #################################################################
>          #################################################################
>          #################################################################
>          #################################################################
>          #################################################################
>          #################################################################
>          #################################################################
>          #################################################################
>          #################################################################
>          #################################################################
>          #################################################################
>          #################################################################
>          #################################################################
>          ########################################
>          2.1 MiB/s
> done
> Bytes transferred = 15843840 (f1c200 hex)
> => bootm 0x48080000 - 0x48000000
> ## Booting kernel from Legacy Image at 48080000 ...
>    Image Name:   XEN
>    Image Type:   AArch64 Linux Kernel Image (uncompressed)
>    Data Size:    852304 Bytes = 832.3 KiB
>    Load Address: 78080000
>    Entry Point:  78080000
>    Verifying Checksum ... OK
> ## Flattened Device Tree blob at 48000000
>    Booting using the fdt blob at 0x48000000
>    Loading Kernel Image ... OK
>    Using Device Tree in place at 0000000048000000, end 0000000048012921
>
> Starting kernel ...
>
> </end output>
>
> I have noticed that the uboot date has not changed from 2015.04 although
> the build date is from the last time I ran a bitbake command.
>
> I do not know what version of uboot should be used without breaking the
> current compiling versions.
>
> Are there any other modifications that I can attempt?
>
> Thank you very much for your time and patience.
>
> Best regards,
>
> Jairo
>
>
>
>

[-- Attachment #1.2: Type: text/html, Size: 29798 bytes --]

[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: RT Xen on ARM - R-Car series

[Andrii Anisov]

Hello Jairo,

I was on vacations last week, and a regular CES rush the week before.
Please give me some time to sort out what else has happened during that period, and I'll get back to you.

-- 
Sincerely,
Andrii Anisov.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: RT Xen on ARM - R-Car series

[Andrii Anisov]

Jairo,

On 11.01.19 22:12, LOPEZ, FUENTES NACARINO Jairo Eduardo wrote:
> I followed your initial advice and updated my environment to a newer version, opting to stick to the rocko version for the m3ulcb.

I'm a bit confused are you using Renesas Yocto 3.9 or the latest 3.13?

> 
> The Yocto build configuration is shown below:
> 
> Build Configuration:
> BB_VERSION           = "1.36.0"
> BUILD_SYS            = "x86_64-linux"
> NATIVELSBSTRING      = "universal-4.9"
> TARGET_SYS           = "aarch64-poky-linux"
> MACHINE              = "m3ulcb"
> DISTRO               = "poky"
> DISTRO_VERSION       = "2.4.2"

Looks like 3.9, but why do you have u-boot 2018.09 below?

> TUNE_FEATURES        = "aarch64 cortexa57-cortexa53"
> TARGET_FPU           = ""
> SOC_FAMILY           = "rcar-gen3:r8a7796"
> meta
> meta-poky
> meta-yocto-bsp       = "HEAD:342fbd6a3e57021c8e28b124b3adb241936f3d9d"
> meta-rcar-gen3       = "rocko:8af0b7d6e445b532088a068dc012757001be3a1f"
> meta-optee           = "rocko:75dfb67bbb14a70cd47afda9726e2e1c76731885"
> meta-oe
> meta-python
> meta-networking      = "rocko:eae996301d9c097bcbeb8046f08041dc82bb62f8"
> meta-selinux         = "rocko:d855c624f32c5e599bf27e06cb8f5b25b3aae12d"
> meta-virtualization  = "rocko:b704c689b67639214b9568a3d62e82df27e9434f"
> meta-rcar-gen3-xen   = "master:60699c631d541aeeaebaeec9a087efed9385ee42"
> 
> I had to modify the dtb to the version shown below:
> 
> // SPDX-License-Identifier: GPL-2.0
> /*
>   * Device Tree Source for the M3ULCB (R-Car Starter Kit Pro) board
>   *
>   * Copyright (C) 2016-2018 Renesas Electronics Corp.
>   * Copyright (C) 2016 Cogent Embedded, Inc.
>   */
> 
> /dts-v1/;
> #include "r8a7796.dtsi"
> #include "ulcb.dtsi"
> 
> / {
>          model = "Renesas M3ULCB board based on r8a7796";
>          compatible = "renesas,m3ulcb", "renesas,r8a7796";
> 
>          cpus {
>                  idle-states {
>                          /delete-node/ cpu-sleep-1;
>                  };
>          };
> 
>          memory@48000000 {
>                  device_type = "memory";
>                  /* first 128MB is reserved for secure area. */
>                  reg = <0x0 0x48000000 0x0 0x38000000>;
>          };
> 
>          memory@600000000 {
>                  device_type = "memory";
>                  reg = <0x6 0x00000000 0x0 0x40000000>;
>          };
Merge memory nodes to be sure u-boot do not duplicate memory ranges.

> 
>          reserved-memory {
>                  #address-cells = <2>;
>                  #size-cells = <2>;
>                  ranges;
> 
>                  /* device specific region for Lossy Decompression */
>                  lossy_decompress: linux,lossy_decompress@54000000 {
>                          no-map;
>                          reg = <0x00000000 0x54000000 0x0 0x03000000>;
>                  };
> 
>                  /* global autoconfigured region for contiguous allocations */
>                  linux,cma@57000000 {
>                          compatible = "shared-dma-pool";
>                          reusable;
>                          reg = <0x00000000 0x57000000 0x0 0x19000000>;
>                          linux,cma-default;
>                  };
> 
>                  /* device specific region for contiguous allocations */
>                  mmp_reserved: linux,multimedia@70000000 {
>                          compatible = "shared-dma-pool";
>                          reusable;
>                          reg = <0x00000000 0x70000000 0x0 0x10000000>;
>                  };
>          };
> 
>          chosen {
>                  stdout-path = "serial0:115200n8";
>                  xen,xen-bootargs = "dom0_mem=752M console=dtuart dtuart=serial0 dom0_max_vcpus=4";
>                  xen,dom0-bootargs = "console=hvc0 clk_ignore_unused root=/dev/mmcblk1p2 rw rootwait ignore_loglevel cma=128M";
> 
>                  #address-cells = <2>;
>                  #size-cells = <2>;
>                  modules {
>                          module@0 {
>                                  compatible = "xen,linux-zimage", "xen,multiboot-module";
>                                  reg = <0x0 0x7a000000 0x0 0x02000000>;
>                          };
>                  };
>          };
> 
>          mmngr {
>                  compatible = "renesas,mmngr";
>                  memory-region = <&mmp_reserved>, <&lossy_decompress>;
>          };
> 
>          mmngrbuf {
>                  compatible = "renesas,mmngrbuf";
>          };
> 
>          vspm_if {
>                  compatible = "renesas,vspm_if";
>          };
> 
>          versaclock5_out3: versaclk-3 {
>                  compatible = "fixed-clock";
>                  #clock-cells = <0>;
>                  /* Initial value of versaclock out3 */
>                  clock-frequency = <33000000>;
>          };
> };
> 
> &a53_0 {
>          /delete-property/ cpu-idle-states;
> };
> 
> &a53_1 {
>          /delete-property/ cpu-idle-states;
> };
> 
> &a53_2 {
>          /delete-property/ cpu-idle-states;
> };
> 
> &a53_3 {
>          /delete-property/ cpu-idle-states;
> };
> 
> &du {
>          clocks = <&cpg CPG_MOD 724>,
>                   <&cpg CPG_MOD 723>,
>                   <&cpg CPG_MOD 722>,
>                   <&versaclock5 1>,
>                   <&versaclock5_out3>,
>                   <&versaclock5 2>;
>          clock-names = "du.0", "du.1", "du.2",
>                        "dclkin.0", "dclkin.1", "dclkin.2";
> };
> 
> &vspb {
>          status = "okay";
> };
> 
> &vspi0 {
>          status = "okay";
> };


> With the new information, I finally got some output for Xen, but was met with a completely different issue for which I can't find any information for the error I am receiving.
> 
> I realize that I may have an issue with the version of the board I am using, but a simple online search doesn't throw any red flags. The serial console output is below:
> 
> [    0.000177] NOTICE:  BL2: R-Car M3 Initial Program Loader(CA57)
> [    0.004613] NOTICE:  BL2: Initial Program Loader(Rev.2.0.0)
> [    0.010147] NOTICE:  BL2: PRR is R-Car M3 Ver.1.0

Your M3 revision is really old. We did not use something older than M3 1.1. But for the start it might be OK. You would not have IPMMU functional, though you do not need it now.

> [    0.014815] NOTICE:  BL2: Board is Starter Kit Rev.1.0
> [    0.019928] NOTICE:  BL2: Boot device is HyperFlash(80MHz)
> [    0.025354] NOTICE:  BL2: LCM state is CM
> [    0.029395] NOTICE:  BL2: AVS setting succeeded. DVFS_SetVID=0x53
> [    0.035383] NOTICE:  BL2: DDR3200(rev.0.34)
> [    0.046779] NOTICE:  BL2: [COLD_BOOT]
> [    0.085117] NOTICE:  BL2: DRAM Split is 2ch
> [    0.087812] NOTICE:  BL2: QoS is default setting(rev.0.19)
> [    0.093315] NOTICE:  BL2: Lossy Decomp areas
> [    0.097489] NOTICE:       Entry 0: DCMPAREACRAx:0x80000540 DCMPAREACRBx:0x570
> [    0.104573] NOTICE:       Entry 1: DCMPAREACRAx:0x40000000 DCMPAREACRBx:0x0
> [    0.111485] NOTICE:       Entry 2: DCMPAREACRAx:0x20000000 DCMPAREACRBx:0x0
> [    0.118400] NOTICE:  BL2: v1.5(release):556a11e
> [    0.122890] NOTICE:  BL2: Built : 18:03:48, Jan 11 2019
> [    0.128077] NOTICE:  BL2: Normal boot
> [    0.131718] NOTICE:  BL2: dst=0xe6321100 src=0x8180000 len=512(0x200)
> [    0.138106] NOTICE:  BL2: dst=0x43f00000 src=0x8180400 len=6144(0x1800)
> [    0.144726] NOTICE:  BL2: dst=0x44000000 src=0x81c0000 len=65536(0x10000)
> [    0.151952] NOTICE:  BL2: dst=0x44100000 src=0x8200000 len=1048576(0x100000)
> [    0.166777] NOTICE:  BL2: dst=0x50000000 src=0x8640000 len=1048576(0x100000)
> [    0.180752] NOTICE:  BL2: Booting BL31
> 
> 
> U-Boot 2018.09 (Jan 12 2019 - 03:03:43 +0900)
It is a u-boot from Renesas Yocto 3.13. I would not recommend you using it.
Seeing the problem below, it looks like u-boot messes with the device tree in a different way now.

> 
> CPU: Renesas Electronics R8A7796 rev 1.0
> Model: Renesas M3ULCB board based on r8a7796
> DRAM:  1.9 GiB
> Bank #0: 0x048000000 - 0x07fffffff, 896 MiB
> Bank #1: 0x600000000 - 0x63fffffff, 1 GiB
> 
> MMC:   sd@ee100000: 0, sd@ee140000: 1
> Loading Environment from MMC... communication out of sync
> communication out of sync
> communication out of sync
> communication out of sync
> communication out of sync
> communication out of sync
> communication out of sync
> communication out of sync
> communication out of sync
> communication out of sync
> communication out of sync
> communication out of sync
> OK
> In:    serial@e6e88000
> Out:   serial@e6e88000
> Err:   serial@e6e88000
> Net:   eth0: ethernet@e6800000
> Hit any key to stop autoboot:  0
> ethernet@e6800000 Waiting for PHY auto negotiation to complete... done
> Using ethernet@e6800000 device
> TFTP from server 192.168.1.100; our IP address is 192.168.1.3
> Filename 'Image'.
> Load address: 0x48080000
> Loading: #################################################################
>           #################################################################
>           #################################################################
>           #################################################################
>           #################################################################
>           #################################################################
>           #################################################################
>           #################################################################
>           #################################################################
>           #################################################################
>           #################################################################
>           #################################################################
>           #################################################################
>           #################################################################
>           #################################################################
>           #################################################################
>           ############################################
>           2.7 MiB/s
> done
> Bytes transferred = 15911424 (f2ca00 hex)
> Using ethernet@e6800000 device
> TFTP from server 192.168.1.100; our IP address is 192.168.1.3
> Filename 'Image-r8a7796-m3ulcb.dtb'.
> Load address: 0x48000000
> Loading: *
> TFTP error: 'File not found' (1)
> Not retrying...
> ERROR: Did not find a cmdline Flattened Device Tree
> Could not find a valid device tree
> => setenv bootargs
> => tftp 0x48080000 xen-m3ulcb.uImage
> Using ethernet@e6800000 device
> TFTP from server 192.168.1.100; our IP address is 192.168.1.3
> Filename 'xen-m3ulcb.uImage'.
> Load address: 0x48080000
> Loading: ###############################################################
>           28.3 MiB/s
> done
> Bytes transferred = 918928 (e0590 hex)
> => tftp 0x48000000 Image-r8a7796-m3ulcb-xen-local.dtb
> Using ethernet@e6800000 device
> TFTP from server 192.168.1.100; our IP address is 192.168.1.3
> Filename 'Image-r8a7796-m3ulcb-xen-local.dtb'.
> Load address: 0x48000000
> Loading: #####
>           21.8 MiB/s
> done
> Bytes transferred = 68545 (10bc1 hex)
> => tftp 0x7a000000 Image-m3ulcb.bin
> Using ethernet@e6800000 device
> TFTP from server 192.168.1.100; our IP address is 192.168.1.3
> Filename 'Image-m3ulcb.bin'.
> Load address: 0x7a000000
> Loading: #################################################################
>           #################################################################
>           #################################################################
>           #################################################################
>           #################################################################
>           #################################################################
>           #################################################################
>           #################################################################
>           #################################################################
>           #################################################################
>           #################################################################
>           #################################################################
>           #################################################################
>           #################################################################
>           #################################################################
>           #################################################################
>           #################################################################
>           #################################################################
>           #################################################################
>           ################################################################
>           28.1 MiB/s
> done
> Bytes transferred = 19059200 (122d200 hex)
> => bootm 0x48080000 - 0x48000000
> ## Booting kernel from Legacy Image at 48080000 ...
>     Image Name:   XEN
>     Image Type:   AArch64 Linux Kernel Image (uncompressed)
>     Data Size:    918864 Bytes = 897.3 KiB
>     Load Address: 78080000
>     Entry Point:  78080000
>     Verifying Checksum ... OK
> ## Flattened Device Tree blob at 48000000
>     Booting using the fdt blob at 0x48000000
>     Loading Kernel Image ... OK
>     Using Device Tree in place at 0000000048000000, end 0000000048013bc0
> 
> Starting kernel ...
> 
>   Xen 4.12-unstable
> (XEN) Xen version 4.12-unstable (jelfn@waseda.jp <mailto:jelfn@waseda.jp>) (aarch64-poky-linux-gc9
> (XEN) Latest ChangeSet: Mon Dec 17 09:22:59 2018 +0000 git:a5b0eb3-dirty
> (XEN) Processor: 411fd073: "ARM Limited", variant: 0x1, part 0xd07, rev 0x3
> (XEN) 64-bit Execution:
> (XEN)   Processor Features: 0000000000002222 0000000000000000
> (XEN)     Exception Levels: EL3:64+32 EL2:64+32 EL1:64+32 EL0:64+32
> (XEN)     Extensions: FloatingPoint AdvancedSIMD
> (XEN)   Debug Features: 0000000010305106 0000000000000000
> (XEN)   Auxiliary Features: 0000000000000000 0000000000000000
> (XEN)   Memory Model Features: 0000000000001124 0000000000000000
> (XEN)   ISA Features:  0000000000011120 0000000000000000
> (XEN) 32-bit Execution:
> (XEN)   Processor Features: 00000131:00011011
> (XEN)     Instruction Sets: AArch32 A32 Thumb Thumb-2 Jazelle
> (XEN)     Extensions: GenericTimer Security
> (XEN)   Debug Features: 03010066
> (XEN)   Auxiliary Features: 00000000
> (XEN)   Memory Model Features: 10201105 40000000 01260000 02102211
> (XEN)  ISA Features: 02101110 13112111 21232042 01112131 00011142 00011121
> (XEN) Using SMC Calling Convention v1.1
> (XEN) Using PSCI v1.1
> (XEN) SMP: Allowing 6 CPUs
> (XEN)
> (XEN) ****************************************
> (XEN) Panic on CPU 0:
> (XEN) Timer: Unable to retrieve IRQ 0 from the device tree

XEN failed to find timer node in the device tree. r8a7796.dtsi has timer node described, so it must be u-boot changed dtb.
We have scheduled switching to Renesas Yocto 3.13 soon, so will coupe with the issue.
But for now I would recommend you using 3.9 at most.

-- 
Sincerely,
Andrii Anisov.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: RT Xen on ARM - R-Car series

[LOPEZ, FUENTES NACARINO Jairo Eduardo]

[-- Attachment #1.1: Type: text/plain, Size: 29043 bytes --]

Andrii,

I went through the instructions again and found the instructions for what
you call Yocto 3.9 here [1].

I attempted following the initial Xen ARM instructions found here [2]

The biggest difference is in the commit revisions for meta-selinux and
meta-virtualization that I didn't know if going with jethro and morty
respectively was still necessary.I went for rocko. Since the system
compiled, I continued with this revision.

The full compile output is shown below:

Loading cache: 100% |############################################| Time:
0:00:00
Loaded 2895 entries from dependency cache.
Parsing recipes: 100% |##########################################| Time:
0:00:01
Parsing of 2088 .bb files complete (2082 cached, 6 parsed). 2901 targets,
87 skipped, 4 masked, 0 errors.
NOTE: Resolving any missing task queue dependencies
NOTE: Multiple providers are available for iasl-native (iasl-native,
acpica-native)
Consider defining a PREFERRED_PROVIDER entry to match iasl-native

Build Configuration:
BB_VERSION           = "1.36.0"
BUILD_SYS            = "x86_64-linux"
NATIVELSBSTRING      = "universal-4.9"
TARGET_SYS           = "aarch64-poky-linux"
MACHINE              = "m3ulcb"
DISTRO               = "poky"
DISTRO_VERSION       = "2.4.2"
TUNE_FEATURES        = "aarch64 cortexa57-cortexa53"
TARGET_FPU           = ""
SOC_FAMILY           = "rcar-gen3:r8a7796"
meta
meta-poky
meta-yocto-bsp       = "tmp:342fbd6a3e57021c8e28b124b3adb241936f3d9d"
meta-rcar-gen3       = "tmp:fd078b6ece537d986852cb827bd21e022a797b2f"
meta-optee           = "tmp:75dfb67bbb14a70cd47afda9726e2e1c76731885"
meta-oe
meta-python
meta-networking      = "tmp:dacfa2b1920e285531bec55cd2f08743390aaf57"
meta-selinux         = "rocko:d855c624f32c5e599bf27e06cb8f5b25b3aae12d"
meta-virtualization  = "rocko:b704c689b67639214b9568a3d62e82df27e9434f"
meta-rcar-gen3-xen   =
"lastest_xen:cc6e6e1c07197207769331b0532fc88a9165a855"

Initialising tasks: 100% |#######################################| Time:
0:00:08
NOTE: Executing SetScene Tasks
NOTE: Executing RunQueue Tasks
NOTE: Tasks Summary: Attempted 6243 tasks of which 5052 didn't need to be
rerun and all succeeded.

I followed the R-Car M3 firmware update instructions without issue from
[3]. I was confused with the ascii part, but when I realized that the
instructions were for use with the minicom program, things were relatively
smooth sailing.

After the firmware update, I attempted to boot the Linux created from
running bitbake and was able to boot the following without issue:


U-Boot 2015.04 (Jan 21 2019 - 20:33:02)

CPU: Renesas Electronics R8A7796 rev 1.0
Board: M3ULCB
I2C:   ready
DRAM:  1.9 GiB
Bank #0: 0x048000000 - 0x07fffffff, 896 MiB
Bank #1: 0x600000000 - 0x63fffffff, 1 GiB

... <redacted because it felt like unimportant data>

Poky (Yocto Project Reference Distro) 2.4.2 m3ulcb ttySC0

m3ulcb login: root
root@m3ulcb:~# uname -a
Linux m3ulcb 4.14.35-yocto-standard #2 SMP PREEMPT Mon Jan 21 23:44:14 JST
2019x

Which means that at least my Linux build is fine.
The U-Boot version is back to 2015.04 so I am guessing that this is the
version you were expected for a Yocto 3.9 build.

The final step to load Xen brings me back to the initial issue which is
that I obtain absolutely no output.

U-Boot 2015.04 (Jan 21 2019 - 20:33:02)

CPU: Renesas Electronics R8A7796 rev 1.0
Board: M3ULCB
I2C:   ready
DRAM:  1.9 GiB
Bank #0: 0x048000000 - 0x07fffffff, 896 MiB
Bank #1: 0x600000000 - 0x63fffffff, 1 GiB

MMC:   sh-sdhi: 0, sh-sdhi: 1
In:    serial
Out:   serial
Err:   serial
Net:   ravb
Hit any key to stop autoboot:  0
=> dhcp
ravb Waiting for PHY auto negotiation to complete........ done
ravb: 1000Base/Full
BOOTP broadcast 1
BOOTP broadcast 2
DHCP client bound to address 192.168.1.10 (256 ms)
*** Warning: no boot file name; using 'C0A8010A.img'
Using ravb device
TFTP from server 192.168.1.100; our IP address is 192.168.1.10
Filename 'C0A8010A.img'.
Load address: 0x48080000
Loading: *
TFTP error: 'File not found' (1)
Not retrying...
=> setenv bootargs
=> tftp 0x48080000 xen-m3ulcb.uImage
ravb:0 is connected to ravb.  Reconnecting to ravb
ravb Waiting for PHY auto negotiation to complete... done
ravb: 1000Base/Full
Using ravb device
TFTP from server 192.168.1.100; our IP address is 192.168.1.3
Filename 'xen-m3ulcb.uImage'.
Load address: 0x48080000
Loading: #############################################################
         171.9 KiB/s
done
Bytes transferred = 886160 (d8590 hex)
=> tftp 0x48000000 Image-r8a7796-m3ulcb-xen-local.dtb
ravb:0 is connected to ravb.  Reconnecting to ravb
ravb Waiting for PHY auto negotiation to complete... done
ravb: 1000Base/Full
Using ravb device
TFTP from server 192.168.1.100; our IP address is 192.168.1.3
Filename 'Image-r8a7796-m3ulcb-xen-local.dtb'.
Load address: 0x48000000
Loading: #####
         12.7 KiB/s
done
Bytes transferred = 68545 (10bc1 hex)
=> tftp 0x7a000000 Image-m3ulcb.bin
ravb:0 is connected to ravb.  Reconnecting to ravb
ravb Waiting for PHY auto negotiation to complete... done
ravb: 1000Base/Full
Using ravb device
TFTP from server 192.168.1.100; our IP address is 192.168.1.3
Filename 'Image-m3ulcb.bin'.
Load address: 0x7a000000
Loading: #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         ################################
         2.4 MiB/s
done
Bytes transferred = 18588160 (11ba200 hex)
=> bootm 0x48080000 - 0x48000000
## Booting kernel from Legacy Image at 48080000 ...
   Image Name:   XEN
   Image Type:   AArch64 Linux Kernel Image (uncompressed)
   Data Size:    886096 Bytes = 865.3 KiB
   Load Address: 78080000
   Entry Point:  78080000
   Verifying Checksum ... OK
## Flattened Device Tree blob at 48000000
   Booting using the fdt blob at 0x48000000
   Loading Kernel Image ... OK
   Using Device Tree in place at 0000000048000000, end 0000000048013bc0

Starting kernel ...

Obtaining no output is still a little unsettling. I am still a little
worried that my dtb configuration is wrong but since the only thing I
should need to modify is the chosen section, I place that below:

       chosen {
                stdout-path = "serial0:115200n8";
                xen,xen-bootargs = "dom0_mem=752M console=dtuart
dtuart=serial0 dom0_max_vcpus=4";
                xen,dom0-bootargs = "console=hvc0 clk_ignore_unused
root=/dev/mmcblk1p2 rw rootwait ignore_loglevel cma=128M";

                #address-cells = <2>;
                #size-cells = <2>;
                modules {
                        module@0 {
                                compatible = "xen,linux-zimage",
"xen,multiboot-module";
                                reg = <0x0 0x7a000000 0x0 0x02000000>;
                        };
                };
        };

Since I am using the sd-card to attempt to boot various versions, I have
chosen the second partition for Linux.

Last time there was a mention of modifying the internal build scripts for
Xen.

If there is any additional information I can obtain for you let me know.

For now I think I am finally in Yocto 3.9 running a 4.14 Linux kernel and
attempting to run and unstable Xen with the following modified
xen_git.bbappend from the Xen troops meta-demo repository [4].

FILESEXTRAPATHS_prepend := "${THISDIR}/${PN}:"

DEPENDS += "u-boot-mkimage-native"

XEN_REL="unstable"

XEN_BRANCH = "master"

SRCREV = "${AUTOREV}"

EXTRA_OEMAKE += " CONFIG_QEMU_XEN=n"

DEFAULT_PREFERENCE = "1"

do_deploy_append () {
    if [ -f ${D}/boot/xen ]; then
        uboot-mkimage -A arm64 -C none -T kernel -a 0x78080000 -e
0x78080000 -n "XEN" -d ${D}/boot/xen ${DEPLOYDIR}/xen-${MACHINE}.uImage
    fi
}

FILES_${PN}-xencommons_remove += " \
    ${systemd_unitdir}/system/xenstored.socket \
    ${systemd_unitdir}/system/xenstored_ro.socket \
    "

FILES_${PN}-xencommons += " \
    ${systemd_unitdir}/system/xendriverdomain.service \
    "

SYSTEMD_SERVICE_${PN}-xencommons_remove += " \
    xenstored.socket \
    xenstored_ro.socket \
    "

#FILES_${PN}-libxendevicemodel = "${libdir}/libxendevicemodel.so.*"
#FILES_${PN}-libxendevicemodel-dev = "${libdir}/libxendevicemodel.so"

FILES_${PN}-libxentoolcore = "${libdir}/libxentoolcore.so.*"
FILES_${PN}-libxentoolcore-dev = "${libdir}/libxentoolcore.so"

FILES_${PN}-pkgconfig = "\
    ${datadir}/pkgconfig \
    "

FILES_${PN}-xendiag += " \
    ${sbindir}/xen-diag \
    "

PACKAGES_append = "\
    ${PN}-libxentoolcore \
    ${PN}-libxentoolcore-dev \
    ${PN}-pkgconfig \
    ${PN}-xendiag \
    "

RDEPENDS_${PN}-base_remove = "\
    ${PN}-blktap \
    ${PN}-libblktapctl \
    ${PN}-libvhd \
    "

FILES_${PN}-staticdev += "\
    ${libdir}/libxenforeignmemory.a \
    ${libdir}/libxentoollog.a \
    ${libdir}/libxengnttab.a \
    ${libdir}/libxendevicemodel.a \
    ${libdir}/libxenevtchn.a \
    ${libdir}/libxencall.a \
    ${libdir}/libxentoolcore.a \
    "

FILES_${PN}-libxencall-dev += " \
    ${libdir}/libxencall.so \
    ${datadir}/pkgconfig/xencall.pc \
    "

FILES_${PN}-libxenctrl-dev += " \
    ${libdir}/libxenctrl.so \
    ${datadir}/pkgconfig/xencontrol.pc \
    "

FILES_${PN}-libxendevicemodel += "${libdir}/libxendevicemodel.so.*"
FILES_${PN}-libxendevicemodel-dev += " \
    ${libdir}/libxendevicemodel.so \
    ${datadir}/pkgconfig/xendevicemodel.pc \
    "

FILES_${PN}-libxenevtchn-dev += " \
    ${libdir}/libxenevtchn.so \
    ${datadir}/pkgconfig/xenevtchn.pc \
    "

FILES_${PN}-libxenforeignmemory-dev += "\
    ${libdir}/libxenforeignmemory.so \
    ${datadir}/pkgconfig/xenforeignmemory.pc \
    "

FILES_${PN}-libxengnttab-dev += " \
    ${libdir}/libxengnttab.so \
    ${datadir}/pkgconfig/xengnttab.pc \
    "

FILES_${PN}-libxenguest-dev += " \
    ${libdir}/libxenguest.so \
    ${datadir}/pkgconfig/xenguest.pc \
    "

FILES_${PN}-libxenstat-dev += " \
    ${libdir}/libxenstat.so \
    ${datadir}/pkgconfig/xenstat.pc \
    "

FILES_${PN}-libxenstore-dev += " \
    ${libdir}/libxenstore.so \
    ${datadir}/pkgconfig/xenstore.pc \
    "

FILES_${PN}-libxentoolcore += "${libdir}/libxentoolcore.so.*"
FILES_${PN}-libxentoolcore-dev += " \
    ${libdir}/libxentoolcore.so \
    ${datadir}/pkgconfig/xentoolcore.pc \
    "

FILES_${PN}-libxentoollog-dev += " \
    ${libdir}/libxentoollog.so \
    ${datadir}/pkgconfig/xentoollog.pc \
    "

FILES_${PN}-libxenvchan-dev += " \
    ${libdir}/libxenvchan.so \
    ${datadir}/pkgconfig/xenvchan.pc \
    "

FILES_${PN}-misc += "\
    ${sbindir}/xen-diag \
    ${libdir}/xen/bin/depriv-fd-checker \
    "

FILES_${PN}-efi = "\
    ${exec_prefix}/lib64/efi/xen* \
    "

FILES_${PN}-xenmon += "\
    ${sbindir}/xenmon \
    "

FILES_${PN}-fsimage += "\
    ${libdir}/xenfsimage/*/fsimage.so \
    "

FILES_${PN}-libfsimage += "\
    ${libdir}/libxenfsimage.so.* \
    "

FILES_${PN}-libfsimage-dev += "\
    ${libdir}/libxenfsimage.so \
    "

Best regards,

Jairo

[1]
https://elinux.org/R-Car/Boards/Yocto-Gen3#Building_the_BSP_for_Renesas_H3_Starter_Kit.2C_M3_Starter_Kit
[2]
https://wiki.xenproject.org/wiki/Xen_ARM_with_Virtualization_Extensions/Salvator-X
[3] https://elinux.org/R-Car/Boards/M3SK#Flashing_firmware
[4] https://github.com/xen-troops/meta-demo


2019年1月16日(水) 16:53 Andrii Anisov <andrii.anisov@gmail.com>:

> Jairo,
>
> On 11.01.19 22:12, LOPEZ, FUENTES NACARINO Jairo Eduardo wrote:
> > I followed your initial advice and updated my environment to a newer
> version, opting to stick to the rocko version for the m3ulcb.
>
> I'm a bit confused are you using Renesas Yocto 3.9 or the latest 3.13?
>
> >
> > The Yocto build configuration is shown below:
> >
> > Build Configuration:
> > BB_VERSION           = "1.36.0"
> > BUILD_SYS            = "x86_64-linux"
> > NATIVELSBSTRING      = "universal-4.9"
> > TARGET_SYS           = "aarch64-poky-linux"
> > MACHINE              = "m3ulcb"
> > DISTRO               = "poky"
> > DISTRO_VERSION       = "2.4.2"
>
> Looks like 3.9, but why do you have u-boot 2018.09 below?
>
> > TUNE_FEATURES        = "aarch64 cortexa57-cortexa53"
> > TARGET_FPU           = ""
> > SOC_FAMILY           = "rcar-gen3:r8a7796"
> > meta
> > meta-poky
> > meta-yocto-bsp       = "HEAD:342fbd6a3e57021c8e28b124b3adb241936f3d9d"
> > meta-rcar-gen3       = "rocko:8af0b7d6e445b532088a068dc012757001be3a1f"
> > meta-optee           = "rocko:75dfb67bbb14a70cd47afda9726e2e1c76731885"
> > meta-oe
> > meta-python
> > meta-networking      = "rocko:eae996301d9c097bcbeb8046f08041dc82bb62f8"
> > meta-selinux         = "rocko:d855c624f32c5e599bf27e06cb8f5b25b3aae12d"
> > meta-virtualization  = "rocko:b704c689b67639214b9568a3d62e82df27e9434f"
> > meta-rcar-gen3-xen   = "master:60699c631d541aeeaebaeec9a087efed9385ee42"
> >
> > I had to modify the dtb to the version shown below:
> >
> > // SPDX-License-Identifier: GPL-2.0
> > /*
> >   * Device Tree Source for the M3ULCB (R-Car Starter Kit Pro) board
> >   *
> >   * Copyright (C) 2016-2018 Renesas Electronics Corp.
> >   * Copyright (C) 2016 Cogent Embedded, Inc.
> >   */
> >
> > /dts-v1/;
> > #include "r8a7796.dtsi"
> > #include "ulcb.dtsi"
> >
> > / {
> >          model = "Renesas M3ULCB board based on r8a7796";
> >          compatible = "renesas,m3ulcb", "renesas,r8a7796";
> >
> >          cpus {
> >                  idle-states {
> >                          /delete-node/ cpu-sleep-1;
> >                  };
> >          };
> >
> >          memory@48000000 {
> >                  device_type = "memory";
> >                  /* first 128MB is reserved for secure area. */
> >                  reg = <0x0 0x48000000 0x0 0x38000000>;
> >          };
> >
> >          memory@600000000 {
> >                  device_type = "memory";
> >                  reg = <0x6 0x00000000 0x0 0x40000000>;
> >          };
> Merge memory nodes to be sure u-boot do not duplicate memory ranges.
>
> >
> >          reserved-memory {
> >                  #address-cells = <2>;
> >                  #size-cells = <2>;
> >                  ranges;
> >
> >                  /* device specific region for Lossy Decompression */
> >                  lossy_decompress: linux,lossy_decompress@54000000 {
> >                          no-map;
> >                          reg = <0x00000000 0x54000000 0x0 0x03000000>;
> >                  };
> >
> >                  /* global autoconfigured region for contiguous
> allocations */
> >                  linux,cma@57000000 {
> >                          compatible = "shared-dma-pool";
> >                          reusable;
> >                          reg = <0x00000000 0x57000000 0x0 0x19000000>;
> >                          linux,cma-default;
> >                  };
> >
> >                  /* device specific region for contiguous allocations */
> >                  mmp_reserved: linux,multimedia@70000000 {
> >                          compatible = "shared-dma-pool";
> >                          reusable;
> >                          reg = <0x00000000 0x70000000 0x0 0x10000000>;
> >                  };
> >          };
> >
> >          chosen {
> >                  stdout-path = "serial0:115200n8";
> >                  xen,xen-bootargs = "dom0_mem=752M console=dtuart
> dtuart=serial0 dom0_max_vcpus=4";
> >                  xen,dom0-bootargs = "console=hvc0 clk_ignore_unused
> root=/dev/mmcblk1p2 rw rootwait ignore_loglevel cma=128M";
> >
> >                  #address-cells = <2>;
> >                  #size-cells = <2>;
> >                  modules {
> >                          module@0 {
> >                                  compatible = "xen,linux-zimage",
> "xen,multiboot-module";
> >                                  reg = <0x0 0x7a000000 0x0 0x02000000>;
> >                          };
> >                  };
> >          };
> >
> >          mmngr {
> >                  compatible = "renesas,mmngr";
> >                  memory-region = <&mmp_reserved>, <&lossy_decompress>;
> >          };
> >
> >          mmngrbuf {
> >                  compatible = "renesas,mmngrbuf";
> >          };
> >
> >          vspm_if {
> >                  compatible = "renesas,vspm_if";
> >          };
> >
> >          versaclock5_out3: versaclk-3 {
> >                  compatible = "fixed-clock";
> >                  #clock-cells = <0>;
> >                  /* Initial value of versaclock out3 */
> >                  clock-frequency = <33000000>;
> >          };
> > };
> >
> > &a53_0 {
> >          /delete-property/ cpu-idle-states;
> > };
> >
> > &a53_1 {
> >          /delete-property/ cpu-idle-states;
> > };
> >
> > &a53_2 {
> >          /delete-property/ cpu-idle-states;
> > };
> >
> > &a53_3 {
> >          /delete-property/ cpu-idle-states;
> > };
> >
> > &du {
> >          clocks = <&cpg CPG_MOD 724>,
> >                   <&cpg CPG_MOD 723>,
> >                   <&cpg CPG_MOD 722>,
> >                   <&versaclock5 1>,
> >                   <&versaclock5_out3>,
> >                   <&versaclock5 2>;
> >          clock-names = "du.0", "du.1", "du.2",
> >                        "dclkin.0", "dclkin.1", "dclkin.2";
> > };
> >
> > &vspb {
> >          status = "okay";
> > };
> >
> > &vspi0 {
> >          status = "okay";
> > };
>
>
> > With the new information, I finally got some output for Xen, but was met
> with a completely different issue for which I can't find any information
> for the error I am receiving.
> >
> > I realize that I may have an issue with the version of the board I am
> using, but a simple online search doesn't throw any red flags. The serial
> console output is below:
> >
> > [    0.000177] NOTICE:  BL2: R-Car M3 Initial Program Loader(CA57)
> > [    0.004613] NOTICE:  BL2: Initial Program Loader(Rev.2.0.0)
> > [    0.010147] NOTICE:  BL2: PRR is R-Car M3 Ver.1.0
>
> Your M3 revision is really old. We did not use something older than M3
> 1.1. But for the start it might be OK. You would not have IPMMU functional,
> though you do not need it now.
>
> > [    0.014815] NOTICE:  BL2: Board is Starter Kit Rev.1.0
> > [    0.019928] NOTICE:  BL2: Boot device is HyperFlash(80MHz)
> > [    0.025354] NOTICE:  BL2: LCM state is CM
> > [    0.029395] NOTICE:  BL2: AVS setting succeeded. DVFS_SetVID=0x53
> > [    0.035383] NOTICE:  BL2: DDR3200(rev.0.34)
> > [    0.046779] NOTICE:  BL2: [COLD_BOOT]
> > [    0.085117] NOTICE:  BL2: DRAM Split is 2ch
> > [    0.087812] NOTICE:  BL2: QoS is default setting(rev.0.19)
> > [    0.093315] NOTICE:  BL2: Lossy Decomp areas
> > [    0.097489] NOTICE:       Entry 0: DCMPAREACRAx:0x80000540
> DCMPAREACRBx:0x570
> > [    0.104573] NOTICE:       Entry 1: DCMPAREACRAx:0x40000000
> DCMPAREACRBx:0x0
> > [    0.111485] NOTICE:       Entry 2: DCMPAREACRAx:0x20000000
> DCMPAREACRBx:0x0
> > [    0.118400] NOTICE:  BL2: v1.5(release):556a11e
> > [    0.122890] NOTICE:  BL2: Built : 18:03:48, Jan 11 2019
> > [    0.128077] NOTICE:  BL2: Normal boot
> > [    0.131718] NOTICE:  BL2: dst=0xe6321100 src=0x8180000 len=512(0x200)
> > [    0.138106] NOTICE:  BL2: dst=0x43f00000 src=0x8180400
> len=6144(0x1800)
> > [    0.144726] NOTICE:  BL2: dst=0x44000000 src=0x81c0000
> len=65536(0x10000)
> > [    0.151952] NOTICE:  BL2: dst=0x44100000 src=0x8200000
> len=1048576(0x100000)
> > [    0.166777] NOTICE:  BL2: dst=0x50000000 src=0x8640000
> len=1048576(0x100000)
> > [    0.180752] NOTICE:  BL2: Booting BL31
> >
> >
> > U-Boot 2018.09 (Jan 12 2019 - 03:03:43 +0900)
> It is a u-boot from Renesas Yocto 3.13. I would not recommend you using it.
> Seeing the problem below, it looks like u-boot messes with the device tree
> in a different way now.
>
> >
> > CPU: Renesas Electronics R8A7796 rev 1.0
> > Model: Renesas M3ULCB board based on r8a7796
> > DRAM:  1.9 GiB
> > Bank #0: 0x048000000 - 0x07fffffff, 896 MiB
> > Bank #1: 0x600000000 - 0x63fffffff, 1 GiB
> >
> > MMC:   sd@ee100000: 0, sd@ee140000: 1
> > Loading Environment from MMC... communication out of sync
> > communication out of sync
> > communication out of sync
> > communication out of sync
> > communication out of sync
> > communication out of sync
> > communication out of sync
> > communication out of sync
> > communication out of sync
> > communication out of sync
> > communication out of sync
> > communication out of sync
> > OK
> > In:    serial@e6e88000
> > Out:   serial@e6e88000
> > Err:   serial@e6e88000
> > Net:   eth0: ethernet@e6800000
> > Hit any key to stop autoboot:  0
> > ethernet@e6800000 Waiting for PHY auto negotiation to complete... done
> > Using ethernet@e6800000 device
> > TFTP from server 192.168.1.100; our IP address is 192.168.1.3
> > Filename 'Image'.
> > Load address: 0x48080000
> > Loading:
> #################################################################
> >
> #################################################################
> >
> #################################################################
> >
> #################################################################
> >
> #################################################################
> >
> #################################################################
> >
> #################################################################
> >
> #################################################################
> >
> #################################################################
> >
> #################################################################
> >
> #################################################################
> >
> #################################################################
> >
> #################################################################
> >
> #################################################################
> >
> #################################################################
> >
> #################################################################
> >           ############################################
> >           2.7 MiB/s
> > done
> > Bytes transferred = 15911424 (f2ca00 hex)
> > Using ethernet@e6800000 device
> > TFTP from server 192.168.1.100; our IP address is 192.168.1.3
> > Filename 'Image-r8a7796-m3ulcb.dtb'.
> > Load address: 0x48000000
> > Loading: *
> > TFTP error: 'File not found' (1)
> > Not retrying...
> > ERROR: Did not find a cmdline Flattened Device Tree
> > Could not find a valid device tree
> > => setenv bootargs
> > => tftp 0x48080000 xen-m3ulcb.uImage
> > Using ethernet@e6800000 device
> > TFTP from server 192.168.1.100; our IP address is 192.168.1.3
> > Filename 'xen-m3ulcb.uImage'.
> > Load address: 0x48080000
> > Loading: ###############################################################
> >           28.3 MiB/s
> > done
> > Bytes transferred = 918928 (e0590 hex)
> > => tftp 0x48000000 Image-r8a7796-m3ulcb-xen-local.dtb
> > Using ethernet@e6800000 device
> > TFTP from server 192.168.1.100; our IP address is 192.168.1.3
> > Filename 'Image-r8a7796-m3ulcb-xen-local.dtb'.
> > Load address: 0x48000000
> > Loading: #####
> >           21.8 MiB/s
> > done
> > Bytes transferred = 68545 (10bc1 hex)
> > => tftp 0x7a000000 Image-m3ulcb.bin
> > Using ethernet@e6800000 device
> > TFTP from server 192.168.1.100; our IP address is 192.168.1.3
> > Filename 'Image-m3ulcb.bin'.
> > Load address: 0x7a000000
> > Loading:
> #################################################################
> >
> #################################################################
> >
> #################################################################
> >
> #################################################################
> >
> #################################################################
> >
> #################################################################
> >
> #################################################################
> >
> #################################################################
> >
> #################################################################
> >
> #################################################################
> >
> #################################################################
> >
> #################################################################
> >
> #################################################################
> >
> #################################################################
> >
> #################################################################
> >
> #################################################################
> >
> #################################################################
> >
> #################################################################
> >
> #################################################################
> >
> ################################################################
> >           28.1 MiB/s
> > done
> > Bytes transferred = 19059200 (122d200 hex)
> > => bootm 0x48080000 - 0x48000000
> > ## Booting kernel from Legacy Image at 48080000 ...
> >     Image Name:   XEN
> >     Image Type:   AArch64 Linux Kernel Image (uncompressed)
> >     Data Size:    918864 Bytes = 897.3 KiB
> >     Load Address: 78080000
> >     Entry Point:  78080000
> >     Verifying Checksum ... OK
> > ## Flattened Device Tree blob at 48000000
> >     Booting using the fdt blob at 0x48000000
> >     Loading Kernel Image ... OK
> >     Using Device Tree in place at 0000000048000000, end 0000000048013bc0
> >
> > Starting kernel ...
> >
> >   Xen 4.12-unstable
> > (XEN) Xen version 4.12-unstable (jelfn@waseda.jp <mailto:jelfn@waseda.jp>)
> (aarch64-poky-linux-gc9
> > (XEN) Latest ChangeSet: Mon Dec 17 09:22:59 2018 +0000 git:a5b0eb3-dirty
> > (XEN) Processor: 411fd073: "ARM Limited", variant: 0x1, part 0xd07, rev
> 0x3
> > (XEN) 64-bit Execution:
> > (XEN)   Processor Features: 0000000000002222 0000000000000000
> > (XEN)     Exception Levels: EL3:64+32 EL2:64+32 EL1:64+32 EL0:64+32
> > (XEN)     Extensions: FloatingPoint AdvancedSIMD
> > (XEN)   Debug Features: 0000000010305106 0000000000000000
> > (XEN)   Auxiliary Features: 0000000000000000 0000000000000000
> > (XEN)   Memory Model Features: 0000000000001124 0000000000000000
> > (XEN)   ISA Features:  0000000000011120 0000000000000000
> > (XEN) 32-bit Execution:
> > (XEN)   Processor Features: 00000131:00011011
> > (XEN)     Instruction Sets: AArch32 A32 Thumb Thumb-2 Jazelle
> > (XEN)     Extensions: GenericTimer Security
> > (XEN)   Debug Features: 03010066
> > (XEN)   Auxiliary Features: 00000000
> > (XEN)   Memory Model Features: 10201105 40000000 01260000 02102211
> > (XEN)  ISA Features: 02101110 13112111 21232042 01112131 00011142
> 00011121
> > (XEN) Using SMC Calling Convention v1.1
> > (XEN) Using PSCI v1.1
> > (XEN) SMP: Allowing 6 CPUs
> > (XEN)
> > (XEN) ****************************************
> > (XEN) Panic on CPU 0:
> > (XEN) Timer: Unable to retrieve IRQ 0 from the device tree
>
> XEN failed to find timer node in the device tree. r8a7796.dtsi has timer
> node described, so it must be u-boot changed dtb.
> We have scheduled switching to Renesas Yocto 3.13 soon, so will coupe with
> the issue.
> But for now I would recommend you using 3.9 at most.
>
> --
> Sincerely,
> Andrii Anisov.
>

[-- Attachment #1.2: Type: text/html, Size: 37446 bytes --]

[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: RT Xen on ARM - R-Car series

[LOPEZ, FUENTES NACARINO Jairo Eduardo]

[-- Attachment #1.1: Type: text/plain, Size: 7553 bytes --]

Andrii,

Thank you for your previous mail.

I am sorry for taking so long to answer.

The rest of the mail is inline

2019年1月22日(火) 18:00 Andrii Anisov <andrii.anisov@gmail.com>:

>
>
> On 22.01.19 12:45, LOPEZ, FUENTES NACARINO Jairo Eduardo wrote:
> > Yes. Since you pointed out that the U-boot version was not what you
> expected, I downgraded to Yocto 3.9. I am able to boot Linux 4.14 but Xen,
> of any version continues to give me absolutely no output.
> Ok, let you try to start with earlyprintk. Follow next steps
>   - Go to <your Yocto dir>/build/tmp/work/aarch64-poky-linux/xen/<your
> version>/git/
>   - Run `make distclean`
>   - Go to <your Yocto dir>/build/tmp/work/aarch64-poky-linux/xen/<your
> version>/temp/
>   - Edit a file hidden behind a symbolic link `run.do_compile`: find
> `oe_runmake_call()` function, to the line `make -j8 STDVGA....` add
> `CONFIG_EARLY_PRINTK=rcar3`, correct `debug=y`.
>   - Run `./run.do_configure`
>   - Run ./run.do_compile
>   - Go to <your Yocto dir>/build/tmp/work/aarch64-poky-linux/xen/<your
> version>/git/xen/
>   - Run `mkimage -A arm64 -C none -T kernel -a 0x78080000 -e 0x78080000 -n
> "XEN" -d xen xen.uImage`
>   - Boot your board with just created xen.uImage.
>
>
I was able to compile the Xen image with earlyprintk without issue. It is
comforting to get some sort of feedback from the device, even if it is
failing.


> But I'm 99 percents sure you will see memory initialization error from
> XEN, so you should merge your memory nodes.
>
> >     Do not forget following:
> >
> >      >      >          memory@48000000 {
> >      >      >                  device_type = "memory";
> >      >      >                  /* first 128MB is reserved for secure
> area. */
> >      >      >                  reg = <0x0 0x48000000 0x0 0x38000000>;
> >      >      >          };
> >      >      >
> >      >      >          memory@600000000 {
> >      >      >                  device_type = "memory";
> >      >      >                  reg = <0x6 0x00000000 0x0 0x40000000>;
> >      >      >          };
> >      >     Merge memory nodes to be sure u-boot do not duplicate memory
> ranges.
> >
> >
> > Is this merge also required for Renesas' Yocto 3.9?
> Yes, we escalated the problem and the solution to Renesas, but 3.9 still
> needs memory nodes merging.
> They should have unset `CONFIG_ARCH_FIXUP_FDT` in a u-boot config. But
> Yocto 3.9 still have `CONFIG_ARCH_FIXUP_FDT=y` configured for u-boot.
>
>
I was also able to make a compilable DTB file by merging the memory ranges.
I used the following configuration:

 memory@48000000 {
                device_type = "memory";
                /* first 128MB is reserved for secure area. */
               reg = <0x0 0x48000000 0x0 0x38000000>,  <0x6 0x00000000 0x0
0x40000000>;
}



> >      > That my M3 is old sounds unfortunate.
> >      > What limitation for Xen would I have to live with, with a lack of
> a IPMMU?
> >     I'm not 100% sure, but as long as it works, there should not be
> other issues.
> >     You just will not be able to passthrough peripherals to guest
> domains (that is what IPMMU is needed for).
> >
> >
> > Ah, it is a shame that I won't be able to passthrough peripherals to
> guest domains. Fortunately at this point in time I only need to benchmark
> the CPUs so this should not affect me.
> It must be ok from the technical point of view.
>
> --
> Sincerely,
> Andrii Anisov.
>


When attempting to boot the Xen image created I did get print messages but
I get the following:

[    0.000191] NOTICE:  BL2: R-Car Gen3 Initial Program Loader(CA57)
Rev.1.0.21
[    0.005753] NOTICE:  BL2: PRR is R-Car M3
Ver.1.0
[    0.010422] NOTICE:  BL2: Board is Starter Kit
Rev.1.0
[    0.015534] NOTICE:  BL2: Boot device is
HyperFlash(80MHz)
[    0.020960] NOTICE:  BL2: LCM state is
CM
[    0.025002] NOTICE:  BL2: AVS setting succeeded. DVFS_SetVID=0x53
[    0.030989] NOTICE:  BL2: DDR3200(rev.0.33)NOTICE:  [COLD_BOOT]NOTICE:
..0
[    0.086118] NOTICE:  BL2: DRAM Split is 2ch
[    0.090003] NOTICE:  BL2: QoS is default setting(rev.0.19)
[    0.095507] NOTICE:  BL2: Lossy Decomp areas
[    0.099680] NOTICE:       Entry 0: DCMPAREACRAx:0x80000540
DCMPAREACRBx:0x570
[    0.106765] NOTICE:       Entry 1: DCMPAREACRAx:0x40000000
DCMPAREACRBx:0x0
[    0.113677] NOTICE:       Entry 2: DCMPAREACRAx:0x20000000
DCMPAREACRBx:0x0
[    0.120591] NOTICE:  BL2: v1.4(release):15dba6b
[    0.125081] NOTICE:  BL2: Built : 11:34:26, Jan 21 2019
[    0.130269] NOTICE:  BL2: Normal boot
[    0.133914] NOTICE:  BL2: dst=0xe6320d00 src=0x8180000 len=512(0x200)
[    0.140297] NOTICE:  BL2: dst=0x43f00000 src=0x8180400 len=6144(0x1800)
[    0.146918] NOTICE:  BL2: dst=0x44000000 src=0x81c0000 len=65536(0x10000)
[    0.154148] NOTICE:  BL2: dst=0x44100000 src=0x8200000
len=1048576(0x100000)
[    0.168971] NOTICE:  BL2: dst=0x50000000 src=0x8640000
len=1048576(0x100000)

</ redacted u-boot image load information>

U-Boot 2015.04 (Jan 21 2019 - 20:33:02)

CPU: Renesas Electronics R8A7796 rev 1.0
Board: M3ULCB
I2C:   ready
DRAM:  1.9 GiB
Bank #0: 0x048000000 - 0x07fffffff, 896 MiB
Bank #1: 0x600000000 - 0x63fffffff, 1 GiB

MMC:   sh-sdhi: 0, sh-sdhi: 1
In:    serial
Out:   serial
Err:   serial
Net:   ravb
Hit any key to stop autoboot:  0

=> bootm 0x48080000 - 0x48000000
## Booting kernel from Legacy Image at 48080000 ...
   Image Name:   XEN
   Image Type:   AArch64 Linux Kernel Image (uncompressed)
   Data Size:    886096 Bytes = 865.3 KiB
   Load Address: 78080000
   Entry Point:  78080000
   Verifying Checksum ... OK
## Flattened Device Tree blob at 48000000
   Booting using the fdt blob at 0x48000000
   Loading Kernel Image ... OK
   Using Device Tree in place at 0000000048000000, end 0000000048013bc0

Starting kernel ...

- UART enabled -
- CPU 00000000 booting -
- Current EL 00000004 -
- Xen must be entered in NS EL2 mode -
- Please update the bootloader -
- Boot failed -

I thought that using the U-boot image from Yocto 3.9 would automatically
let me avoid the issue with boot loader entering the wrong mode.

As I mentioned before I am using the rocko branches from
meta-virtualization and meta-selinux as shown below:

Build Configuration:
BB_VERSION           = "1.36.0"
BUILD_SYS            = "x86_64-linux"
NATIVELSBSTRING      = "universal-4.9"
TARGET_SYS           = "aarch64-poky-linux"
MACHINE              = "m3ulcb"
DISTRO               = "poky"
DISTRO_VERSION       = "2.4.2"
TUNE_FEATURES        = "aarch64 cortexa57-cortexa53"
TARGET_FPU           = ""
SOC_FAMILY           = "rcar-gen3:r8a7796"
meta
meta-poky
meta-yocto-bsp       = "tmp:342fbd6a3e57021c8e28b124b3adb241936f3d9d"
meta-rcar-gen3       = "tmp:fd078b6ece537d986852cb827bd21e022a797b2f"
meta-optee           = "tmp:75dfb67bbb14a70cd47afda9726e2e1c76731885"
meta-oe
meta-python
meta-networking      = "tmp:dacfa2b1920e285531bec55cd2f08743390aaf57"
meta-selinux         = "rocko:d855c624f32c5e599bf27e06cb8f5b25b3aae12d"
meta-virtualization  = "rocko:b704c689b67639214b9568a3d62e82df27e9434f"
meta-rcar-gen3-xen   =
"lastest_xen:cc6e6e1c07197207769331b0532fc88a9165a855"

I have no idea if that affects the u-boot creation, but I thought I might
as well mention that.

Once again thank you for all the help. If there is any other information I
can give you, please let me know.

Best regards,

Jairo

[-- Attachment #1.2: Type: text/html, Size: 10257 bytes --]

[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: RT Xen on ARM - R-Car series

[Andrii Anisov]

Hello Jairo,

On 28.01.19 19:20, LOPEZ, FUENTES NACARINO Jairo Eduardo wrote:

> I was able to compile the Xen image with earlyprintk without issue.

Cool.

> It is comforting to get some sort of feedback from the device, even if it is failing.

Indeed:)

> When attempting to boot the Xen image created I did get print messages but I get the following:
...

I see you have freshly built BL2:

> [    0.120591] NOTICE:  BL2: v1.4(release):15dba6b
> [    0.125081] NOTICE:  BL2: Built : 11:34:26, Jan 21 2019
...
> - UART enabled -
> - CPU 00000000 booting -
> - Current EL 00000004 -
> - Xen must be entered in NS EL2 mode -

But something went wrong, and your BL2 did not start u-boot in EL2. Consequently, hypervisor is not started in EL2 and can not do its work.
Please verify you have your ATF built with ` RCAR_BL33_EXECUTION_EL=BL33_EL2` as it is stated in the correspondent recipe [1].
You can search for it in <yocto dir>/build/tmp/work/<board name>-poky-linux/v1.4+renesas+<revision>/tmp/* .
And if it is compiled with the flag, then take care to reflash all bootloader images on your board.

[1] https://github.com/xen-troops/meta-demo/blob/master/meta-rcar-gen3-xen/recipes-bsp/arm-trusted-firmware/arm-trusted-firmware_git.bbappend

-- 
Sincerely,
Andrii Anisov.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: RT Xen on ARM - R-Car series

[LOPEZ, FUENTES NACARINO Jairo Eduardo]

[-- Attachment #1.1: Type: text/plain, Size: 8938 bytes --]

Andrii,

YEY!

I have finally got an error message!

The rest of the mail is inline.

2019年1月28日(月) 17:25 Andrii Anisov <andrii.anisov@gmail.com>:

> Hello Jairo,
>
> On 28.01.19 19:20, LOPEZ, FUENTES NACARINO Jairo Eduardo wrote:
>
> > I was able to compile the Xen image with earlyprintk without issue.
>
> Cool.
>
> > It is comforting to get some sort of feedback from the device, even if
> it is failing.
>
> Indeed:)
>
> > When attempting to boot the Xen image created I did get print messages
> but I get the following:
> ...
>
> I see you have freshly built BL2:
>
> > [    0.120591] NOTICE:  BL2: v1.4(release):15dba6b
> > [    0.125081] NOTICE:  BL2: Built : 11:34:26, Jan 21 2019
> ...
> > - UART enabled -
> > - CPU 00000000 booting -
> > - Current EL 00000004 -
> > - Xen must be entered in NS EL2 mode -
>
> But something went wrong, and your BL2 did not start u-boot in EL2.
> Consequently, hypervisor is not started in EL2 and can not do its work.
> Please verify you have your ATF built with `
> RCAR_BL33_EXECUTION_EL=BL33_EL2` as it is stated in the correspondent
> recipe [1].
> You can search for it in <yocto dir>/build/tmp/work/<board
> name>-poky-linux/v1.4+renesas+<revision>/tmp/* .
> And if it is compiled with the flag, then take care to reflash all
> bootloader images on your board.
>
> [1]
> https://github.com/xen-troops/meta-demo/blob/master/meta-rcar-gen3-xen/recipes-bsp/arm-trusted-firmware/arm-trusted-firmware_git.bbappend
>
> --
> Sincerely,
> Andrii Anisov.
>


As you mentioned, I checked the ATF build at the following path:

<yocto dir>/build/tmp/work/<board
name>-poky-linux/arm-trusted-firmware/v1.4+renesas+<revision>/temp

Much to my surprise I found no trace of thr RCAR_BL33_EXECUTION_EL=BL33_EL2.

It seems that at some point in time I removed the Xen troops meta-demo
layer which nullified the ATFW_OPT_append in that layer.

I was able to readjust the layers and check that the compilation included
the necessary ATF option.

After a successful u-boot firmware update, I was met with the following
information on the R-Car M3:

[    0.000191] NOTICE:  BL2: R-Car Gen3 Initial Program Loader(CA57)
Rev.1.0.21
[    0.005753] NOTICE:  BL2: PRR is R-Car M3 Ver.1.0
[    0.010422] NOTICE:  BL2: Board is Starter Kit Rev.1.0
[    0.015534] NOTICE:  BL2: Boot device is HyperFlash(80MHz)
[    0.020960] NOTICE:  BL2: LCM state is CM
[    0.025002] NOTICE:  BL2: AVS setting succeeded. DVFS_SetVID=0x53
[    0.030989] NOTICE:  BL2: DDR3200(rev.0.33)NOTICE:  [COLD_BOOT]NOTICE:
..0
[    0.086113] NOTICE:  BL2: DRAM Split is 2ch
[    0.089998] NOTICE:  BL2: QoS is default setting(rev.0.19)
[    0.095502] NOTICE:  BL2: Lossy Decomp areas
[    0.099675] NOTICE:       Entry 0: DCMPAREACRAx:0x80000540
DCMPAREACRBx:0x570
[    0.106760] NOTICE:       Entry 1: DCMPAREACRAx:0x40000000
DCMPAREACRBx:0x0
[    0.113672] NOTICE:       Entry 2: DCMPAREACRAx:0x20000000
DCMPAREACRBx:0x0
[    0.120586] NOTICE:  BL2: v1.4(release):15dba6b
[    0.125076] NOTICE:  BL2: Built : 14:48:12, Jan 29 2019
[    0.130264] NOTICE:  BL2: Normal boot
[    0.133909] NOTICE:  BL2: dst=0xe6320d00 src=0x8180000 len=512(0x200)
[    0.140292] NOTICE:  BL2: dst=0x43f00000 src=0x8180400 len=6144(0x1800)
[    0.146913] NOTICE:  BL2: dst=0x44000000 src=0x81c0000 len=65536(0x10000)
[    0.154144] NOTICE:  BL2: dst=0x44100000 src=0x8200000
len=1048576(0x100000)
[    0.168967] NOTICE:  BL2: dst=0x50000000 src=0x8640000
len=1048576(0x100000)


U-Boot 2015.04 (Jan 21 2019 - 20:33:02)

CPU: Renesas Electronics R8A7796 rev 1.0
Board: M3ULCB
I2C:   ready
DRAM:  1.9 GiB
Bank #0: 0x048000000 - 0x07fffffff, 896 MiB
Bank #1: 0x600000000 - 0x63fffffff, 1 GiB

MMC:   sh-sdhi: 0, sh-sdhi: 1
In:    serial
Out:   serial
Err:   serial
Net:   ravb

First, it is nice to know where to look for the build time for the BL2!

After loading earlyprintk enabled Xen and the 4.14 Linux kernel from Yocto
3.9, I was met with the following output:

Starting kernel ...

- UART enabled -
- CPU 00000000 booting -
- Current EL 00000008 -
- Xen starting at EL2 -
- Zero BSS -
- Setting up control registers -
- Turning on paging -
- Ready -
(XEN) Checking for initrd in /chosen
(XEN) RAM: 0000000048000000 - 000000007fffffff
(XEN) RAM: 0000000600000000 - 000000063fffffff
(XEN) RAM: 0000000600000000 - 000000063fffffff
(XEN)
(XEN) MODULE[0]: 0000000048000000 - 0000000048011000 Device Tree
(XEN) MODULE[1]: 000000007a000000 - 000000007c000000 Kernel
(XEN)  RESVD[0]: 0000000048000000 - 0000000048011000
(XEN)
(XEN)
(XEN) Command line: dom0_mem=752M console=dtuart dtuart=serial0
dom0_max_vcpus=4
(XEN) PFN compression on bits 19...20
(XEN) Xen BUG at page_alloc.c:274
(XEN) ----[ Xen-4.12.0-rc  arm64  debug=y   Not tainted ]----
(XEN) CPU:    0
(XEN) PC:     000000000028b310 page_alloc.c#bootmem_region_add+0x188/0x198
(XEN) LR:     000000000028b370
(XEN) SP:     00000000002d7d00
(XEN) CPSR:   800003c9 MODE:64-bit EL2h (Hypervisor, handler)
(XEN)      X0: 0000000000600000  X1: 0000000000000003  X2: 0000000000000004
(XEN)      X3: 0000000000640000  X4: 0000800008011000  X5: 0000000000000028
(XEN)      X6: 0000000000600000  X7: 0000000000000004  X8: 0000000000000001
(XEN)      X9: 000000000000000a X10: 00000000002d7afa X11: 0000000000000032
(XEN)     X12: 0000000000000002 X13: 000000000026e5d8 X14: 0000000000000020
(XEN)     X15: 0000000000400000 X16: 0000000000000000 X17: 0000000000000000
(XEN)     X18: 000000007fe5ae30 X19: 0000000000600000 X20: 0000000000640000
(XEN)     X21: 0000000640000000 X22: 0000000600000000 X23: 0000000640000000
(XEN)     X24: 000000000028b320 X25: 00000000002b83e0 X26: 0000000048000000
(XEN)     X27: 0000000640000000 X28: 0000000000000002  FP: 00000000002d7d00
(XEN)
(XEN)   VTCR_EL2: 80000000
(XEN)  VTTBR_EL2: 0000000000000000
(XEN)
(XEN)  SCTLR_EL2: 30cd183d
(XEN)    HCR_EL2: 000000000078663f
(XEN)  TTBR0_EL2: 0000000078194000
(XEN)
(XEN)    ESR_EL2: f2000001
(XEN)  HPFAR_EL2: 0000000000000000
(XEN)    FAR_EL2: 0000000000000000
(XEN)
(XEN) Xen stack trace from sp=00000000002d7d00:
(XEN)    00000000002d7d40 000000000028b370 0000000000000001 0000000600000000
(XEN)    0000000640000000 0000000600000000 0000000640000000 00000000002d7d90
(XEN)    00000000002d7d90 000000000029c81c 0000000000000001 0000000600000000
(XEN)    0000000640000000 0000000600000000 0000000640000000 0000000000000001
(XEN)    00000000002e0000 0000000600000000 00000000002d7de0 000000000029d9e8
(XEN)    0000000640000000 0000000600000000 0000000640000000 00000000002b83c0
(XEN)    ffffffffffffffff 00000000002b83d8 0000000048000000 0000000048011000
(XEN)    000000007fe5a940 0000000000200608 0000000078080000 0000000077e80000
(XEN)    0000000048000000 0000000000000000 0000000000400000 0000000000000000
(XEN)    0000000000000001 0000000000000000 0000000078080000 0000000000000400
(XEN)    0000000000000000 0000000000011000 0000000048000000 00000000b8000000
(XEN)    00000000002b83c0 000000000028b320 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000300000000 ffffffff00000000 0000040000000040
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN) Xen call trace:
(XEN)    [<000000000028b310>] page_alloc.c#bootmem_region_add+0x188/0x198
(PC)
(XEN)    [<000000000028b370>] init_boot_pages+0x50/0x17c (LR)
(XEN)    [<000000000028b370>] init_boot_pages+0x50/0x17c
(XEN)    [<000000000029c81c>] dt_unreserved_regions+0xc8/0xdc
(XEN)    [<000000000029d9e8>] start_xen+0xbec/0xc5c
(XEN)    [<0000000000200608>] arm64/head.o#paging+0x60/0x98
(XEN)
(XEN)
(XEN) ****************************************
(XEN) Panic on CPU 0:
(XEN) Xen BUG at page_alloc.c:274
(XEN) ****************************************
(XEN)
(XEN) Reboot in five seconds...

I am using a 4.12.0-rc Xen, specifically from commit
08b908ba63dee8bc313983c5e412852cbcbcda85 dated Jan 16, 16:16:56 2019.

Do you have any idea what could be the matter?

Thank you very much for all the support.

Best regards,

Jairo

[-- Attachment #1.2: Type: text/html, Size: 10645 bytes --]

[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: RT Xen on ARM - R-Car series

[Andrii Anisov]

Hello Jairo,

On 29.01.19 17:30, LOPEZ, FUENTES NACARINO Jairo Eduardo wrote:
> (XEN) Checking for initrd in /chosen
> (XEN) RAM: 0000000048000000 - 000000007fffffff
> (XEN) RAM: 0000000600000000 - 000000063fffffff
> (XEN) RAM: 0000000600000000 - 000000063fffffff
Duplication of RAM range `0000000600000000 - 000000063fffffff` gives me an idea that you still did not merge memory nodes in your device tree.

> (XEN) Xen call trace:
> (XEN)    [<000000000028b310>] page_alloc.c#bootmem_region_add+0x188/0x198 (PC)
And XEN can not handle that.

-- 
Sincerely,
Andrii Anisov.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: RT Xen on ARM - R-Car series

[LOPEZ, FUENTES NACARINO Jairo Eduardo]

[-- Attachment #1.1: Type: text/plain, Size: 15266 bytes --]

Hello Andrii,

2019年1月29日(火) 22:30 Andrii Anisov <andrii.anisov@gmail.com>:

> Hello Jairo,
>
> On 29.01.19 17:30, LOPEZ, FUENTES NACARINO Jairo Eduardo wrote:
> > (XEN) Checking for initrd in /chosen
> > (XEN) RAM: 0000000048000000 - 000000007fffffff
> > (XEN) RAM: 0000000600000000 - 000000063fffffff
> > (XEN) RAM: 0000000600000000 - 000000063fffffff
> Duplication of RAM range `0000000600000000 - 000000063fffffff` gives me an
> idea that you still did not merge memory nodes in your device tree.
>
> > (XEN) Xen call trace:
> > (XEN)    [<000000000028b310>]
> page_alloc.c#bootmem_region_add+0x188/0x198 (PC)
> And XEN can not handle that.
>
> --
> Sincerely,
> Andrii Anisov.
>

As you correctly pointed out, I was using an older version of a dtb file
which didn't have the memory modification.

I was able to check that the dtb file I was using was finally correct by
utilizing dtc on the compiled dtb.

dtc -I dtb -O dts -o test.dtb r8a7796-m3ulcb-xen-local.dtb

It seems that finally Xen was able to boot with the Dom0 Linux also
somewhat booting. At the end I get a p2m.c error repeatedly which loops the
machine. The complete output is below:

- UART enabled -
- CPU 00000000 booting -
- Current EL 00000008 -
- Xen starting at EL2 -
- Zero BSS -
- Setting up control registers -
- Turning on paging -
- Ready -
(XEN) Checking for initrd in /chosen
(XEN) RAM: 0000000048000000 - 000000007fffffff
(XEN) RAM: 0000000600000000 - 000000063fffffff
(XEN)
(XEN) MODULE[0]: 0000000048000000 - 0000000048011000 Device Tree
(XEN) MODULE[1]: 000000007a000000 - 000000007c000000 Kernel
(XEN)  RESVD[0]: 0000000048000000 - 0000000048011000
(XEN)
(XEN)
(XEN) Command line: dom0_mem=752M console=dtuart dtuart=serial0
dom0_max_vcpus=4
(XEN) PFN compression on bits 19...20
(XEN) Domain heap initialised
(XEN) Booting using Device Tree
(XEN) Platform: Generic System
(XEN) Looking for dtuart at "serial0", options ""
 Xen 4.12.0-rc
(XEN) Xen version 4.12.0-rc (jelfn@waseda.jp) (aarch64-poky-linux-gcc (GCC)
7.3.0) debug=y  Tue Jan 29 01:08:01 JST 2019
(XEN) Latest ChangeSet: Fri Jan 18 12:47:45 2019 +0000 git:1912f12-dirty
(XEN) Processor: 411fd073: "ARM Limited", variant: 0x1, part 0xd07, rev 0x3
(XEN) 64-bit Execution:
(XEN)   Processor Features: 0000000000002222 0000000000000000
(XEN)     Exception Levels: EL3:64+32 EL2:64+32 EL1:64+32 EL0:64+32
(XEN)     Extensions: FloatingPoint AdvancedSIMD
(XEN)   Debug Features: 0000000010305106 0000000000000000
(XEN)   Auxiliary Features: 0000000000000000 0000000000000000
(XEN)   Memory Model Features: 0000000000001124 0000000000000000
(XEN)   ISA Features:  0000000000011120 0000000000000000
(XEN) 32-bit Execution:
(XEN)   Processor Features: 00000131:00011011
(XEN)     Instruction Sets: AArch32 A32 Thumb Thumb-2 Jazelle
(XEN)     Extensions: GenericTimer Security
(XEN)   Debug Features: 03010066
(XEN)   Auxiliary Features: 00000000
(XEN)   Memory Model Features: 10201105 40000000 01260000 02102211
(XEN)  ISA Features: 02101110 13112111 21232042 01112131 00011142 00011121
(XEN) Using SMC Calling Convention v1.1
(XEN) Using PSCI v1.0
(XEN) SMP: Allowing 6 CPUs
(XEN) Generic Timer IRQ: phys=30 hyp=26 virt=27 Freq: 8333 KHz
(XEN) GICv2 initialization:
(XEN)         gic_dist_addr=00000000f1010000
(XEN)         gic_cpu_addr=00000000f1020000
(XEN)         gic_hyp_addr=00000000f1040000
(XEN)         gic_vcpu_addr=00000000f1060000
(XEN)         gic_maintenance_irq=25
(XEN) GICv2: Adjusting CPU interface base to 0xf102f000
(XEN) GICv2: 512 lines, 8 cpus, secure (IID 0200043b).
(XEN) Using scheduler: SMP Credit Scheduler rev2 (credit2)
(XEN) Initializing Credit2 scheduler
(XEN)  load_precision_shift: 18
(XEN)  load_window_shift: 30
(XEN)  underload_balance_tolerance: 0
(XEN)  overload_balance_tolerance: -3
(XEN)  runqueues arrangement: socket
(XEN)  cap enforcement granularity: 10ms
(XEN) load tracking window length 1073741824 ns
(XEN) Adding cpu 0 to runqueue 0
(XEN)  First cpu on runqueue, activating
(XEN) Allocated console ring of 64 KiB.
(XEN) Bringing up CPU1
- CPU 00000001 booting -
- Current EL 00000008 -
- Xen starting at EL2 -
- Setting up control registers -
- Turning on paging -
- Ready -
(XEN) Adding cpu 1 to runqueue 0
(XEN) CPU 1 booted.
(XEN) Bringing up CPU2
- CPU 00000100 booting -
- Current EL 00000008 -
- Xen starting at EL2 -
- Setting up control registers -
- Turning on paging -
- Ready -
(XEN) CPU2 MIDR (0x410fd034) does not match boot CPU MIDR (0x411fd073),
(XEN) disable cpu (see big.LITTLE.txt under docs/).
(XEN) CPU2 never came online
(XEN) Failed to bring up CPU 2 (error -5)
(XEN) Bringing up CPU3
- CPU 00000101 booting -
- Current EL 00000008 -
- Xen starting at EL2 -
- Setting up control registers -
- Turning on paging -
- Ready -
(XEN) CPU3 MIDR (0x410fd034) does not match boot CPU MIDR (0x411fd073),
(XEN) disable cpu (see big.LITTLE.txt under docs/).
(XEN) CPU3 never came online
(XEN) Failed to bring up CPU 3 (error -5)
(XEN) Bringing up CPU4
- CPU 00000102 booting -
- Current EL 00000008 -
- Xen starting at EL2 -
- Setting up control registers -
- Turning on paging -
- Ready -
(XEN) CPU4 MIDR (0x410fd034) does not match boot CPU MIDR (0x411fd073),
(XEN) disable cpu (see big.LITTLE.txt under docs/).
(XEN) CPU4 never came online
(XEN) Failed to bring up CPU 4 (error -5)
(XEN) Bringing up CPU5
- CPU 00000103 booting -
- Current EL 00000008 -
- Xen starting at EL2 -
- Setting up control registers -
- Turning on paging -
- Ready -
(XEN) CPU5 MIDR (0x410fd034) does not match boot CPU MIDR (0x411fd073),
(XEN) disable cpu (see big.LITTLE.txt under docs/).
(XEN) CPU5 never came online
(XEN) Failed to bring up CPU 5 (error -5)
(XEN) Brought up 2 CPUs
(XEN) P2M: 44-bit IPA with 44-bit PA and 8-bit VMID
(XEN) P2M: 4 levels with order-0 root, VTCR 0x80043594
(XEN) I/O virtualisation disabled
(XEN) build-id: 55fe145621420426dea5dd5aabe4a4e16006031d
(XEN) alternatives: Patching with alt table 00000000002abbb8 ->
00000000002ac1dc
(XEN) CPU0 will call ARM_SMCCC_ARCH_WORKAROUND_1 on exception entry
(XEN) CPU1 will call ARM_SMCCC_ARCH_WORKAROUND_1 on exception entry
(XEN) *** LOADING DOMAIN 0 ***
(XEN) Loading Domd0 kernel from boot module @ 000000007a000000
(XEN) Allocating 1:1 mappings totalling 752MB for dom0:
(XEN) BANK[0] 0x00000049000000-0x00000078000000 (752MB)
(XEN) Grant table range: 0x00000078080000-0x000000780c0000
(XEN) Allocating PPI 16 for event channel interrupt
(XEN) Loading zImage from 000000007a000000 to
0000000049080000-000000004b080000
(XEN) Loading dom0 DTB to 0x0000000051000000-0x000000005100fc9f
(XEN) Initial low memory virq threshold set at 0x4000 pages.
(XEN) Scrubbing Free RAM in background
(XEN) Std. Loglevel: All
(XEN) Guest Loglevel: All
(XEN) *** Serial input to DOM0 (type 'CTRL-a' three times to switch input)
(XEN) Freed 344kB init memory.
(XEN) d0v0: vGICD: unhandled word write 0xffffffff to ICACTIVER4
(XEN) d0v0: vGICD: unhandled word write 0xffffffff to ICACTIVER8
(XEN) d0v0: vGICD: unhandled word write 0xffffffff to ICACTIVER12
(XEN) d0v0: vGICD: unhandled word write 0xffffffff to ICACTIVER16
(XEN) d0v0: vGICD: unhandled word write 0xffffffff to ICACTIVER20
(XEN) d0v0: vGICD: unhandled word write 0xffffffff to ICACTIVER24
(XEN) d0v0: vGICD: unhandled word write 0xffffffff to ICACTIVER28
(XEN) d0v0: vGICD: unhandled word write 0xffffffff to ICACTIVER32
(XEN) d0v0: vGICD: unhandled word write 0xffffffff to ICACTIVER36
(XEN) d0v0: vGICD: unhandled word write 0xffffffff to ICACTIVER40
(XEN) d0v0: vGICD: unhandled word write 0xffffffff to ICACTIVER44
(XEN) d0v0: vGICD: unhandled word write 0xffffffff to ICACTIVER48
(XEN) d0v0: vGICD: unhandled word write 0xffffffff to ICACTIVER52
(XEN) d0v0: vGICD: unhandled word write 0xffffffff to ICACTIVER56
(XEN) d0v0: vGICD: unhandled word write 0xffffffff to ICACTIVER60
(XEN) d0v0: vGICD: unhandled word write 0xffffffff to ICACTIVER0
[    0.000000] Booting Linux on physical CPU 0x0

[    0.000000] Linux version 4.14.35-yocto-standard (oe-user@oe-host) (gcc
version 7.3.0 (GCC)) #2 SMP PREEMPT Mon Jan 21 23:44:14 JST 2019

[    0.000000] Boot CPU: AArch64 Processor [411fd073]

[    0.000000] Machine model: Renesas M3ULCB board based on r8a7796

[    0.000000] debug: ignoring loglevel setting.

[    0.000000] Xen 4.12 support found

[    0.000000] efi: Getting EFI parameters from FDT:

[    0.000000] efi: UEFI not found.

[    0.000000] Reserved memory: created CMA memory pool at
0x0000000057000000, size 400 MiB

[    0.000000] OF: reserved mem: initialized node linux,cma@57000000,
compatible id shared-dma-pool

[    0.000000] Reserved memory: created CMA memory pool at
0x0000000070000000, size 256 MiB

[    0.000000] OF: reserved mem: initialized node linux,multimedia@70000000,
compatible id shared-dma-pool

[    0.000000] NUMA: No NUMA configuration found

[    0.000000] NUMA: Faking a node at [mem
0x0000000000000000-0x0000000077ffffff]

[    0.000000] NUMA: NODE_DATA [mem 0x53f9a000-0x53f9baff]

[    0.000000] Zone ranges:

[    0.000000]   DMA      [mem 0x0000000049000000-0x0000000077ffffff]

[    0.000000]   Normal   empty

[    0.000000] Movable zone start for each node

[    0.000000] Early memory node ranges

[    0.000000]   node   0: [mem 0x0000000049000000-0x0000000053ffffff]

[    0.000000]   node   0: [mem 0x0000000057000000-0x0000000077ffffff]

[    0.000000] Initmem setup node 0 [mem
0x0000000049000000-0x0000000077ffffff]

[    0.000000] On node 0 totalpages: 180224

[    0.000000]   DMA zone: 2816 pages used for memmap

[    0.000000]   DMA zone: 0 pages reserved

[    0.000000]   DMA zone: 180224 pages, LIFO batch:31

[    0.000000] psci: probing for conduit method from DT.

[    0.000000] psci: PSCIv1.1 detected in firmware.

[    0.000000] psci: Using standard PSCI v0.2 function IDs

[    0.000000] psci: Trusted OS migration not required

[    0.000000] psci: SMC Calling Convention v1.1

[    0.000000] random: fast init done

[    0.000000] percpu: Embedded 23 pages/cpu @ffff800013f39000 s55448 r8192
d30568 u94208

[    0.000000] pcpu-alloc: s55448 r8192 d30568 u94208 alloc=23*4096

[    0.000000] pcpu-alloc: [0] 0 [0] 1 [0] 2 [0] 3

[    0.000000] Detected PIPT I-cache on CPU0

[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 177408

[    0.000000] Policy zone: DMA

[    0.000000] Kernel command line: console=hvc0 clk_ignore_unused
root=/dev/mmcblk1p2 rw rootwait ignore_loglevel cma=128M

[    0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes)

[    0.000000] Memory: 144356K/720896K available (10748K kernel code, 1192K
rwdata, 5004K rodata, 1152K init, 412K bss, 18446744073709456412K reserved,
671744K cma-reserved)

[    0.000000] Virtual kernel memory layout:

[    0.000000]     modules : 0xffff000000000000 - 0xffff000008000000   (
128 MB)

[    0.000000]     vmalloc : 0xffff000008000000 - 0xffff7dffbfff0000
(129022 GB)

[    0.000000]       .text : 0xffff000008080000 - 0xffff000008b00000   (
10752 KB)

[    0.000000]     .rodata : 0xffff000008b00000 - 0xffff000008ff0000   (
5056 KB)

[    0.000000]       .init : 0xffff000008ff0000 - 0xffff000009110000   (
1152 KB)

[    0.000000]       .data : 0xffff000009110000 - 0xffff00000923a200   (
1193 KB)

[    0.000000]        .bss : 0xffff00000923a200 - 0xffff0000092a14e0   (
413 KB)

[    0.000000]     fixed   : 0xffff7dfffe7f9000 - 0xffff7dfffec00000   (
4124 KB)

[    0.000000]     PCI I/O : 0xffff7dfffee00000 - 0xffff7dffffe00000   (
16 MB)

[    0.000000]     vmemmap : 0xffff7e0000000000 - 0xffff800000000000   (
2048 GB maximum)

[    0.000000]               0xffff7e0000240000 - 0xffff7e0000e00000   (
11 MB actual)

[    0.000000]     memory  : 0xffff800009000000 - 0xffff800038000000   (
752 MB)

[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1

[    0.000000] Preemptible hierarchical RCU implementation.

[    0.000000]     RCU restricting CPUs from NR_CPUS=64 to nr_cpu_ids=4.

[    0.000000]     Tasks RCU enabled.

[    0.000000] RCU: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4

[    0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0

[    0.000000] arch_timer: cp15 timer(s) running at 8.33MHz (virt).

[    0.000000] clocksource: arch_sys_counter: mask: 0xffffffffffffff
max_cycles: 0x1ec02923e, max_idle_ns: 440795202125 ns

[    0.000003] sched_clock: 56 bits at 8MHz, resolution 120ns, wraps every
2199023255496ns

[    0.000393] Console: colour dummy device 80x25

[    0.401164] console [hvc0] enabled

[    0.404660] Calibrating delay loop (skipped), value calculated using
timer frequency.. 16.66 BogoMIPS (lpj=33333)

[    0.414905] pid_max: default: 32768 minimum: 301

[    0.419609] Security Framework initialized

[    0.424151] Dentry cache hash table entries: 131072 (order: 8, 1048576
bytes)

[    0.431419] Inode-cache hash table entries: 65536 (order: 7, 524288
bytes)

[    0.438218] Mount-cache hash table entries: 2048 (order: 2, 16384 bytes)

[    0.444945] Mountpoint-cache hash table entries: 2048 (order: 2, 16384
bytes)

[    0.468219] ASID allocator initialised with 32768 entries

[    0.474295] xen:grant_table: Grant tables using version 1 layout

[    0.480256] Grant table initialized

[    0.483760] xen:events: Using FIFO-based ABI

[    0.488113] Xen: initializing cpu0

[    0.496120] Hierarchical SRCU implementation.

[    0.508876] Detected Renesas R-Car Gen3 r8a7796 ES1.0

[    0.514874] EFI services will not be available.

[    0.527365] smp: Bringing up secondary CPUs ...

(XEN) d0v1: vGICD: unhandled word write 0xffffffff to ICACTIVER0
(XEN) d0v2: vGICD: unhandled word write 0xffffffff to ICACTIVER0
(XEN) d0v3: vGICD: unhandled word write 0xffffffff to ICACTIVER0
[    0.560023] Detected PIPT I-cache on CPU1

[    0.565657] Xen: initializing cpu1

[    0.565681] CPU1: Booted secondary processor [411fd073]

[    0.592149] Detected PIPT I-cache on CPU2

[    0.597770] Xen: initializing cpu2

[    0.597789] CPU2: Booted secondary processor [411fd073]

[    0.624278] Detected PIPT I-cache on CPU3

[    0.629891] Xen: initializing cpu3

[    0.629909] CPU3: Booted secondary processor [411fd073]

[    0.629998] smp: Brought up 1 node, 4 CPUs

[    0.672375] SMP: Total of 4 processors activated.

[    0.677122] CPU features: detected feature: 32-bit EL0 Support

[    0.682996] CPU features: detected feature: Kernel page table isolation
(KPTI)

(XEN) p2m.c:1844: d0v1: Failed to walk page-table va 0xffff800013f59c57
(XEN) p2m.c:1844: d0v2: Failed to walk page-table va 0xffff800013f70c57
(XEN) p2m.c:1844: d0v1: Failed to walk page-table va 0xffff800013f59c48
(XEN) p2m.c:1844: d0v2: Failed to walk page-table va 0xffff800013f70c48

</ this continues infinitely>

Is this normal. Could the error have anything to do with me using the
dom0_max_vpcus=4 bootarg for Xen?

Thank you for all the help.

Best regards,

Jairo

[-- Attachment #1.2: Type: text/html, Size: 17220 bytes --]

[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: RT Xen on ARM - R-Car series

[Julien Grall]

Hi,

On 30/01/2019 20:23, LOPEZ, FUENTES NACARINO Jairo Eduardo wrote:

My e-mail client thinks the e-mail was sent from the future. Can you make sure 
your timezone is set correctly?

> [    0.682996] CPU features: detected feature: Kernel page table isolation (KPTI)
> 
> (XEN) p2m.c:1844: d0v1: Failed to walk page-table va 0xffff800013f59c57
> (XEN) p2m.c:1844: d0v2: Failed to walk page-table va 0xffff800013f70c57
> (XEN) p2m.c:1844: d0v1: Failed to walk page-table va 0xffff800013f59c48
> (XEN) p2m.c:1844: d0v2: Failed to walk page-table va 0xffff800013f70c48
> 
> </ this continues infinitely>
> 
> Is this normal. Could the error have anything to do with me using the 
> dom0_max_vpcus=4 bootarg for Xen?

No this is a unknown benign issue when using KPTI kernel on Xen (see [1]). I 
still haven't found time to fix it and would appreciate help to solve the issue.

Cheers,

[1] https://lists.xen.org/archives/html/xen-devel/2018-03/msg00223.html

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: RT Xen on ARM - R-Car series

[Andrii Anisov]

Julien,

On 30.01.19 14:12, Julien Grall wrote:
> Hi,
> 
> On 30/01/2019 20:23, LOPEZ, FUENTES NACARINO Jairo Eduardo wrote:
> 
> My e-mail client thinks the e-mail was sent from the future. Can you make sure your timezone is set correctly?
> 
>> [    0.682996] CPU features: detected feature: Kernel page table isolation (KPTI)
>>
>> (XEN) p2m.c:1844: d0v1: Failed to walk page-table va 0xffff800013f59c57
>> (XEN) p2m.c:1844: d0v2: Failed to walk page-table va 0xffff800013f70c57
>> (XEN) p2m.c:1844: d0v1: Failed to walk page-table va 0xffff800013f59c48
>> (XEN) p2m.c:1844: d0v2: Failed to walk page-table va 0xffff800013f70c48
>>
>> </ this continues infinitely>
>>
>> Is this normal. Could the error have anything to do with me using the dom0_max_vpcus=4 bootarg for Xen?
> 
> No this is a unknown 
I'm sure you did mean "a known".

> benign issue when using KPTI kernel on Xen (see [1]). I still haven't found time to fix it and would appreciate help to solve the issue.

I've looked through the thread, and it seems it is not even decided how should it be fixed. Should it be a new interface, of fix would fit the existing, and the thread died pretty long ago.
On my side I just commented out that printk, because it renders a debug build unusable.
And I'm pretty sure Jairo is not up to fixing such XEN issues.

> [1] https://lists.xen.org/archives/html/xen-devel/2018-03/msg00223.html

-- 
Sincerely,
Andrii Anisov.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: RT Xen on ARM - R-Car series

[Julien Grall]

On 31/01/2019 11:22, Andrii Anisov wrote:
> Julien,
> 
> On 30.01.19 14:12, Julien Grall wrote:
>> Hi,
>>
>> On 30/01/2019 20:23, LOPEZ, FUENTES NACARINO Jairo Eduardo wrote:
>>
>> My e-mail client thinks the e-mail was sent from the future. Can you make sure 
>> your timezone is set correctly?
>>
>>> [    0.682996] CPU features: detected feature: Kernel page table isolation 
>>> (KPTI)
>>>
>>> (XEN) p2m.c:1844: d0v1: Failed to walk page-table va 0xffff800013f59c57
>>> (XEN) p2m.c:1844: d0v2: Failed to walk page-table va 0xffff800013f70c57
>>> (XEN) p2m.c:1844: d0v1: Failed to walk page-table va 0xffff800013f59c48
>>> (XEN) p2m.c:1844: d0v2: Failed to walk page-table va 0xffff800013f70c48
>>>
>>> </ this continues infinitely>
>>>
>>> Is this normal. Could the error have anything to do with me using the 
>>> dom0_max_vpcus=4 bootarg for Xen?
>>
>> No this is a unknown 
> I'm sure you did mean "a known".
> 
>> benign issue when using KPTI kernel on Xen (see [1]). I still haven't found 
>> time to fix it and would appreciate help to solve the issue.
> 
> I've looked through the thread, and it seems it is not even decided how should 
> it be fixed. Should it be a new interface, of fix would fit the existing, and 
> the thread died pretty long ago.

Well, someone needs to drive the discussion and write the patch. This is in my 
long ever growing list of things to improve in Xen. But...

> On my side I just commented out that printk, because it renders a debug build 
> unusable.

... if it is unusable, why don't you try to tackle the problem? This is a good 
way to understand better the intrinsics of Xen. I will be more than happy to 
provide guidance and review the patches.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: RT Xen on ARM - R-Car series

[Andrii Anisov]

Hello Julien,

On 31.01.19 13:37, Julien Grall wrote:
>> On my side I just commented out that printk, because it renders a debug build unusable.
> 
> ... if it is unusable, why don't you try to tackle the problem?
Because of...

> This is in my long ever growing list of things to

... be done.

Some of things get solutions, some WAs.

-- 
Sincerely,
Andrii Anisov.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: RT Xen on ARM - R-Car series

[Julien Grall]

On 31/01/2019 12:00, Andrii Anisov wrote:
> Hello Julien,
> 
> On 31.01.19 13:37, Julien Grall wrote:
>>> On my side I just commented out that printk, because it renders a debug build 
>>> unusable.
>>
>> ... if it is unusable, why don't you try to tackle the problem?
> Because of...
> 
>> This is in my long ever growing list of things to
> 
> ... be done.
> 
> Some of things get solutions, some WAs.

I can't see a good workaround for this. At some point someone would have to pick 
it up rather than building a house of cards.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: RT Xen on ARM - R-Car series

[Stefano Stabellini]

On Thu, 31 Jan 2019, Julien Grall wrote:
> On 31/01/2019 12:00, Andrii Anisov wrote:
> > Hello Julien,
> > 
> > On 31.01.19 13:37, Julien Grall wrote:
> > > > On my side I just commented out that printk, because it renders a debug
> > > > build unusable.
> > > 
> > > ... if it is unusable, why don't you try to tackle the problem?
> > Because of...
> > 
> > > This is in my long ever growing list of things to
> > 
> > ... be done.
> > 
> > Some of things get solutions, some WAs.
> 
> I can't see a good workaround for this. At some point someone would have to
> pick it up rather than building a house of cards.

I ran into this problem as well not too long ago too. It is very
annoying and it is basically impossible to work-around, the only thing
possible would be to suppress the warning, but that doesn't even count
as a work-around :-)

The way forward is to modify the existing
VCPUOP_register_runstate_memory_area interface. I liked Julien's idea in
https://lists.xenproject.org/archives/html/xen-devel/2018-03/msg00227.html:
we don't necessarily need to change the parameters of the hypercalls
from a guest virtual address to a guest physical address. It should be
enough to convert the guest virtual address into a guest physical
address in Xen when VCPUOP_register_runstate_memory_area is called
(xen/common/domain.c:VCPUOP_register_runstate_memory_area), then store
the guest physical address or its mapping in v->runstate_guest (the type
of runstate_guest needs to change) and always use the guest physical
address for future updates on the runstate memory area.

It doesn't seem too difficult.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: RT Xen on ARM - R-Car series

[Julien Grall]

Hi Stefano,

On 1/31/19 9:56 PM, Stefano Stabellini wrote:
> On Thu, 31 Jan 2019, Julien Grall wrote:
>> On 31/01/2019 12:00, Andrii Anisov wrote:
>>> Hello Julien,
>>>
>>> On 31.01.19 13:37, Julien Grall wrote:
>>>>> On my side I just commented out that printk, because it renders a debug
>>>>> build unusable.
>>>>
>>>> ... if it is unusable, why don't you try to tackle the problem?
>>> Because of...
>>>
>>>> This is in my long ever growing list of things to
>>>
>>> ... be done.
>>>
>>> Some of things get solutions, some WAs.
>>
>> I can't see a good workaround for this. At some point someone would have to
>> pick it up rather than building a house of cards.
> 
> I ran into this problem as well not too long ago too. It is very
> annoying and it is basically impossible to work-around, the only thing
> possible would be to suppress the warning, but that doesn't even count
> as a work-around :-)

I am sure I will regret to have said that, but I will for fairness :).

If security is not a concern within the guest, then you can disable kpti 
(either via Kconfig or command line). All the errors should go away for 
Linux guest.

> 
> The way forward is to modify the existing
> VCPUOP_register_runstate_memory_area interface. I liked Julien's idea in
> https://lists.xenproject.org/archives/html/xen-devel/2018-03/msg00227.html:
> we don't necessarily need to change the parameters of the hypercalls
> from a guest virtual address to a guest physical address. It should be
> enough to convert the guest virtual address into a guest physical
> address in Xen when VCPUOP_register_runstate_memory_area is called
> (xen/common/domain.c:VCPUOP_register_runstate_memory_area), then store
> the guest physical address or its mapping in v->runstate_guest (the type
> of runstate_guest needs to change) and always use the guest physical
> address for future updates on the runstate memory area.

I would love to say it is that easy :). However, there are some research 
to do regarding how this is used by guests today. The hypercall is 
taking a virtual address, so technically it would be possible for a 
guest to pass a non page-aligned virtual address. So this would span 
onto two buffers (it seems to happen on older Linux).

Furthermore, because this is a virtual address, a guess would be free to 
modify the mapping at any time.

So if we want to use guest physical address in Xen, we need to know if 
it will not break any current guest. This would also probably needs to 
be documented in the interface.

With the current interface workaround, we are still playing with devil 
(see [2]). So it would be nice to get a new interface that does not use 
virtual address.

> 
> It doesn't seem too difficult.

Even with my comments above, I agree :).

Cheers,

[2] <9fa77816-a25c-c19b-cc26-e0d28cc2e160@citrix.com>



-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: RT Xen on ARM - R-Car series

[Andrii Anisov]

On 01.02.19 01:14, Julien Grall wrote:
> With the current interface workaround, we are still playing with devil (see [2]). So it would be nice to get a new interface that does not use virtual address.
I'm sorry for my ignorance, I know nearly nothing about runstate areas implementation, but why not to setup a shared page(s) for the purpose?

> [2] <9fa77816-a25c-c19b-cc26-e0d28cc2e160@citrix.com>
Could you please suggest how to follow that link?

-- 
Sincerely,
Andrii Anisov.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: RT Xen on ARM - R-Car series

[Julien Grall]

Hi,

On 2/1/19 10:02 AM, Andrii Anisov wrote:
> 
> 
> On 01.02.19 01:14, Julien Grall wrote:
>> With the current interface workaround, we are still playing with devil 
>> (see [2]). So it would be nice to get a new interface that does not 
>> use virtual address.
> I'm sorry for my ignorance, I know nearly nothing about runstate areas 
> implementation, but why not to setup a shared page(s) for the purpose?

This is actually a shared page, the page is allocated by the domain and 
shared with Xen. So what do you mean?

> 
>> [2] <9fa77816-a25c-c19b-cc26-e0d28cc2e160@citrix.com>
> Could you please suggest how to follow that link?

That's not a link but a Message-ID. You can either use your favorite 
client for looking the e-mail or use http://marc.info?i=<message-id>. 
For instance:

http://marc.info?i=<9fa77816-a25c-c19b-cc26-e0d28cc2e160@citrix.com>

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: RT Xen on ARM - R-Car series

[Andrii Anisov]

Hello,

On 01.02.19 12:12, Julien Grall wrote:
> This is actually a shared page, the page is allocated by the domain and shared with Xen. So what do you mean?

I'm curious if it can be allocated on hypervisor side.

> That's not a link but a Message-ID. You can either use your favorite client for looking the e-mail

It only works if you have it in the current mailbox, I guess.

> or use http://marc.info?i=<message-id>. For instance:
> http://marc.info?i=<9fa77816-a25c-c19b-cc26-e0d28cc2e160@citrix.com>

Good hint!

-- 
Sincerely,
Andrii Anisov.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: RT Xen on ARM - R-Car series

[Julien Grall]

On 2/1/19 10:35 AM, Andrii Anisov wrote:
> Hello,

Hi,

> 
> On 01.02.19 12:12, Julien Grall wrote:
>> This is actually a shared page, the page is allocated by the domain 
>> and shared with Xen. So what do you mean?
> 
> I'm curious if it can be allocated on hypervisor side.

There are very limited case where the hypervisor is allocating memory to 
share with the guest. In general, we want to account the memory to the 
guest, so the best way is to let the guest provide the buffer.

In that case this will not help because the hypercall is taking a 
virtual address (not a guest virtual frame, nor guest physical frame!). 
So the buffer can span across the page-boundary or even have the mapping 
changed.

We can possibly add a restriction on the mapping not changing (pending 
investigation). But I think requesting the address to be page-aligned is 
going to be hard because the runstate is not filling a page.

For instance, Linux is using a per-cpu variable for that. I am not 
entirely sure whether it promise the structure will never cross a 
page-boundary.

> 
>> That's not a link but a Message-ID. You can either use your favorite 
>> client for looking the e-mail
> 
> It only works if you have it in the current mailbox, I guess.


That's correct.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: RT Xen on ARM - R-Car series

[Roger Pau Monné]

On Thu, Jan 31, 2019 at 11:14:37PM +0000, Julien Grall wrote:
> Hi Stefano,
> 
> On 1/31/19 9:56 PM, Stefano Stabellini wrote:
> > On Thu, 31 Jan 2019, Julien Grall wrote:
> > > On 31/01/2019 12:00, Andrii Anisov wrote:
> > > > Hello Julien,
> > > > 
> > > > On 31.01.19 13:37, Julien Grall wrote:
> > > > > > On my side I just commented out that printk, because it renders a debug
> > > > > > build unusable.
> > > > > 
> > > > > ... if it is unusable, why don't you try to tackle the problem?
> > > > Because of...
> > > > 
> > > > > This is in my long ever growing list of things to
> > > > 
> > > > ... be done.
> > > > 
> > > > Some of things get solutions, some WAs.
> > > 
> > > I can't see a good workaround for this. At some point someone would have to
> > > pick it up rather than building a house of cards.
> > 
> > I ran into this problem as well not too long ago too. It is very
> > annoying and it is basically impossible to work-around, the only thing
> > possible would be to suppress the warning, but that doesn't even count
> > as a work-around :-)
> 
> I am sure I will regret to have said that, but I will for fairness :).
> 
> If security is not a concern within the guest, then you can disable kpti
> (either via Kconfig or command line). All the errors should go away for
> Linux guest.
> 
> > 
> > The way forward is to modify the existing
> > VCPUOP_register_runstate_memory_area interface. I liked Julien's idea in
> > https://lists.xenproject.org/archives/html/xen-devel/2018-03/msg00227.html:
> > we don't necessarily need to change the parameters of the hypercalls
> > from a guest virtual address to a guest physical address. It should be
> > enough to convert the guest virtual address into a guest physical
> > address in Xen when VCPUOP_register_runstate_memory_area is called
> > (xen/common/domain.c:VCPUOP_register_runstate_memory_area), then store
> > the guest physical address or its mapping in v->runstate_guest (the type
> > of runstate_guest needs to change) and always use the guest physical
> > address for future updates on the runstate memory area.
> 
> I would love to say it is that easy :). However, there are some research to
> do regarding how this is used by guests today. The hypercall is taking a
> virtual address, so technically it would be possible for a guest to pass a
> non page-aligned virtual address. So this would span onto two buffers (it
> seems to happen on older Linux).
> 
> Furthermore, because this is a virtual address, a guess would be free to
> modify the mapping at any time.
> 
> So if we want to use guest physical address in Xen, we need to know if it
> will not break any current guest. This would also probably needs to be
> documented in the interface.
> 
> With the current interface workaround, we are still playing with devil (see
> [2]). So it would be nice to get a new interface that does not use virtual
> address.

So, I've got a hacky patch to 'fix' this on x86, by taking a reference
to the mfn behind the virtual address provided when setting up the
hypercall and mapping it in Xen. This however doesn't work on ARM due
to the lack of paging_gva_to_gfn. I guess there's something similar to
translate a guest virtual address into a gfn or a mfn?

I've only tested this very slightly, so there might be dragons...

Anyway, this is what I have so far:

---8<---
commit 9b88c4aba641abd188144c4bad1d69345625678f
Author: Roger Pau Monne <roger.pau@citrix.com>
Date:   Fri Feb 1 12:13:47 2019 +0100

    keep runstate area mapped

diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
index 6dc633ed50..cfd60ed7a7 100644
--- a/xen/arch/arm/domain.c
+++ b/xen/arch/arm/domain.c
@@ -277,29 +277,23 @@ static void ctxt_switch_to(struct vcpu *n)
 /* Update per-VCPU guest runstate shared memory area (if registered). */
 static void update_runstate_area(struct vcpu *v)
 {
-    void __user *guest_handle = NULL;
-
-    if ( guest_handle_is_null(runstate_guest(v)) )
+    if ( !v->runstate_guest(v) )
         return;
 
     if ( VM_ASSIST(v->domain, runstate_update_flag) )
     {
-        guest_handle = &v->runstate_guest.p->state_entry_time + 1;
-        guest_handle--;
         v->runstate.state_entry_time |= XEN_RUNSTATE_UPDATE;
-        __raw_copy_to_guest(guest_handle,
-                            (void *)(&v->runstate.state_entry_time + 1) - 1, 1);
+        v->runstate_guest->state_entry_time |= XEN_RUNSTATE_UPDATE;
         smp_wmb();
     }
 
-    __copy_to_guest(runstate_guest(v), &v->runstate, 1);
+    memcpy(v->runstate_guest, &v->runstate, sizeof(v->runstate));
 
-    if ( guest_handle )
+    if ( VM_ASSIST(v->domain, runstate_update_flag) )
     {
         v->runstate.state_entry_time &= ~XEN_RUNSTATE_UPDATE;
         smp_wmb();
-        __raw_copy_to_guest(guest_handle,
-                            (void *)(&v->runstate.state_entry_time + 1) - 1, 1);
+        v->runstate_guest->state_entry_time &= ~XEN_RUNSTATE_UPDATE;
     }
 }
 
diff --git a/xen/arch/x86/domain.c b/xen/arch/x86/domain.c
index 32dc4253ff..b86d8d94c7 100644
--- a/xen/arch/x86/domain.c
+++ b/xen/arch/x86/domain.c
@@ -1604,26 +1604,22 @@ void paravirt_ctxt_switch_to(struct vcpu *v)
 }
 
 /* Update per-VCPU guest runstate shared memory area (if registered). */
-bool update_runstate_area(struct vcpu *v)
+void update_runstate_area(struct vcpu *v)
 {
-    bool rc;
     struct guest_memory_policy policy = { .nested_guest_mode = false };
-    void __user *guest_handle = NULL;
 
-    if ( guest_handle_is_null(runstate_guest(v)) )
-        return true;
+    if ( !v->runstate_guest )
+        return;
 
     update_guest_memory_policy(v, &policy);
 
     if ( VM_ASSIST(v->domain, runstate_update_flag) )
     {
-        guest_handle = has_32bit_shinfo(v->domain)
-            ? &v->runstate_guest.compat.p->state_entry_time + 1
-            : &v->runstate_guest.native.p->state_entry_time + 1;
-        guest_handle--;
         v->runstate.state_entry_time |= XEN_RUNSTATE_UPDATE;
-        __raw_copy_to_guest(guest_handle,
-                            (void *)(&v->runstate.state_entry_time + 1) - 1, 1);
+        if ( has_32bit_shinfo((v)->domain) )
+            v->compat_runstate_guest->state_entry_time |= XEN_RUNSTATE_UPDATE;
+        else
+            v->runstate_guest->state_entry_time |= XEN_RUNSTATE_UPDATE;
         smp_wmb();
     }
 
@@ -1632,31 +1628,22 @@ bool update_runstate_area(struct vcpu *v)
         struct compat_vcpu_runstate_info info;
 
         XLAT_vcpu_runstate_info(&info, &v->runstate);
-        __copy_to_guest(v->runstate_guest.compat, &info, 1);
-        rc = true;
+        memcpy(v->compat_runstate_guest, &info, sizeof(info));
     }
     else
-        rc = __copy_to_guest(runstate_guest(v), &v->runstate, 1) !=
-             sizeof(v->runstate);
+        memcpy(v->runstate_guest, &v->runstate, sizeof(v->runstate));
 
-    if ( guest_handle )
+    if ( VM_ASSIST(v->domain, runstate_update_flag) )
     {
         v->runstate.state_entry_time &= ~XEN_RUNSTATE_UPDATE;
         smp_wmb();
-        __raw_copy_to_guest(guest_handle,
-                            (void *)(&v->runstate.state_entry_time + 1) - 1, 1);
+        if ( has_32bit_shinfo((v)->domain) )
+            v->compat_runstate_guest->state_entry_time &= ~XEN_RUNSTATE_UPDATE;
+        else
+            v->runstate_guest->state_entry_time &= ~XEN_RUNSTATE_UPDATE;
     }
 
     update_guest_memory_policy(v, &policy);
-
-    return rc;
-}
-
-static void _update_runstate_area(struct vcpu *v)
-{
-    if ( !update_runstate_area(v) && is_pv_vcpu(v) &&
-         !(v->arch.flags & TF_kernel_mode) )
-        v->arch.pv.need_update_runstate_area = 1;
 }
 
 static inline bool need_full_gdt(const struct domain *d)
@@ -1779,7 +1766,7 @@ void context_switch(struct vcpu *prev, struct vcpu *next)
 
     if ( prev != next )
     {
-        _update_runstate_area(prev);
+        update_runstate_area(prev);
         vpmu_switch_from(prev);
         np2m_schedule(NP2M_SCHEDLE_OUT);
     }
@@ -1841,7 +1828,7 @@ void context_switch(struct vcpu *prev, struct vcpu *next)
 
     if ( prev != next )
     {
-        _update_runstate_area(next);
+        update_runstate_area(next);
 
         /* Must be done with interrupts enabled */
         vpmu_switch_to(next);
diff --git a/xen/arch/x86/hvm/dom0_build.c b/xen/arch/x86/hvm/dom0_build.c
index a571d15c13..3046bd4bc9 100644
--- a/xen/arch/x86/hvm/dom0_build.c
+++ b/xen/arch/x86/hvm/dom0_build.c
@@ -421,7 +421,7 @@ static int __init pvh_setup_p2m(struct domain *d)
     pvh_setup_e820(d, nr_pages);
     do {
         preempted = false;
-        paging_set_allocation(d, dom0_paging_pages(d, nr_pages),
+        paging_set_allocation(d, dom0_paging_pages(d, nr_pages) * 10,
                               &preempted);
         process_pending_softirqs();
     } while ( preempted );
diff --git a/xen/arch/x86/pv/domain.c b/xen/arch/x86/pv/domain.c
index 7e84b04082..027bcf4eb0 100644
--- a/xen/arch/x86/pv/domain.c
+++ b/xen/arch/x86/pv/domain.c
@@ -313,9 +313,6 @@ static void _toggle_guest_pt(struct vcpu *v)
     if ( !(v->arch.flags & TF_kernel_mode) )
         return;
 
-    if ( v->arch.pv.need_update_runstate_area && update_runstate_area(v) )
-        v->arch.pv.need_update_runstate_area = 0;
-
     if ( v->arch.pv.pending_system_time.version &&
          update_secondary_system_time(v, &v->arch.pv.pending_system_time) )
         v->arch.pv.pending_system_time.version = 0;
diff --git a/xen/arch/x86/x86_64/domain.c b/xen/arch/x86/x86_64/domain.c
index c46dccc25a..9734cd39ef 100644
--- a/xen/arch/x86/x86_64/domain.c
+++ b/xen/arch/x86/x86_64/domain.c
@@ -22,34 +22,21 @@ arch_compat_vcpu_op(
     {
     case VCPUOP_register_runstate_memory_area:
     {
-        struct compat_vcpu_register_runstate_memory_area area;
-        struct compat_vcpu_runstate_info info;
-
-        area.addr.p = 0;
+        union {
+            struct compat_vcpu_register_runstate_memory_area compat;
+            struct vcpu_register_runstate_memory_area native;
+        } area = { };
 
         rc = -EFAULT;
-        if ( copy_from_guest(&area.addr.h, arg, 1) )
+        if ( copy_from_guest(&area.compat.addr.v, arg, 1) )
             break;
 
-        if ( area.addr.h.c != area.addr.p ||
-             !compat_handle_okay(area.addr.h, 1) )
+        unmap_runstate_area(v);
+        rc = map_runstate_area(v, &area.native);
+        if ( rc )
             break;
 
-        rc = 0;
-        guest_from_compat_handle(v->runstate_guest.compat, area.addr.h);
-
-        if ( v == current )
-        {
-            XLAT_vcpu_runstate_info(&info, &v->runstate);
-        }
-        else
-        {
-            struct vcpu_runstate_info runstate;
-
-            vcpu_runstate_get(v, &runstate);
-            XLAT_vcpu_runstate_info(&info, &runstate);
-        }
-        __copy_to_guest(v->runstate_guest.compat, &info, 1);
+        update_runstate_area(v);
 
         break;
     }
diff --git a/xen/common/domain.c b/xen/common/domain.c
index c623daec56..4a4c4a414d 100644
--- a/xen/common/domain.c
+++ b/xen/common/domain.c
@@ -1177,7 +1177,7 @@ int domain_soft_reset(struct domain *d)
 
     for_each_vcpu ( d, v )
     {
-        set_xen_guest_handle(runstate_guest(v), NULL);
+        unmap_runstate_area(v);
         unmap_vcpu_info(v);
     }
 
@@ -1318,6 +1318,84 @@ void unmap_vcpu_info(struct vcpu *v)
     put_page_and_type(mfn_to_page(mfn));
 }
 
+int map_runstate_area(struct vcpu *v,
+                      struct vcpu_register_runstate_memory_area *area)
+{
+    unsigned long offset;
+    unsigned int i;
+    struct domain *d = v->domain;
+    size_t size =
+#ifdef CONFIG_COMPAT
+        has_32bit_shinfo((v)->domain) ? sizeof(*v->compat_runstate_guest) :
+#endif
+                                        sizeof(*v->runstate_guest);
+
+    if ( v->runstate_guest )
+    {
+        ASSERT_UNREACHABLE();
+        return -EBUSY;
+    }
+
+    offset = area->addr.p & ~PAGE_MASK;
+
+    for ( i = 0; i < ARRAY_SIZE(v->runstate_mfn); i++ )
+    {
+        p2m_type_t t;
+        uint32_t pfec = PFEC_page_present;
+        gfn_t gfn = _gfn(paging_gva_to_gfn(v, area->addr.p, &pfec));
+        struct page_info *pg;
+
+        if ( gfn_eq(gfn, INVALID_GFN) )
+            return -EFAULT;
+
+        v->runstate_mfn[i] = get_gfn(d, gfn_x(gfn), &t);
+        if ( t != p2m_ram_rw || mfn_eq(v->runstate_mfn[i], INVALID_MFN) )
+            return -EFAULT;
+
+        pg = mfn_to_page(v->runstate_mfn[i]);
+        if ( !pg || !get_page_type(pg, PGT_writable_page) )
+        {
+            put_gfn(d, gfn_x(gfn));
+            return -EFAULT;
+        }
+
+        put_gfn(d, gfn_x(gfn));
+
+        if ( PFN_DOWN(gfn_x(gfn) + size) == PFN_DOWN(gfn_x(gfn)) )
+            break;
+
+        area->addr.p += PAGE_SIZE - offset;
+    }
+
+    v->runstate_nr = i + 1;
+
+    v->runstate_guest = vmap(v->runstate_mfn, v->runstate_nr);
+    if ( !v->runstate_guest )
+    {
+        for ( i = 0; i < v->runstate_nr; i++)
+            put_page_and_type(mfn_to_page(v->runstate_mfn[i]));
+        return -EFAULT;
+    }
+    v->runstate_guest = (void *)v->runstate_guest + offset;
+
+    return 0;
+}
+
+void unmap_runstate_area(struct vcpu *v)
+{
+    unsigned int i;
+
+    if ( !v->runstate_guest )
+        return;
+
+    vunmap(v->runstate_guest);
+    for ( i = 0; i < v->runstate_nr; i++ )
+        put_page_and_type(mfn_to_page(v->runstate_mfn[i]));
+
+    v->runstate_guest = NULL;
+    v->runstate_nr = 0;
+}
+
 int default_initialise_vcpu(struct vcpu *v, XEN_GUEST_HANDLE_PARAM(void) arg)
 {
     struct vcpu_guest_context *ctxt;
@@ -1495,27 +1573,17 @@ long do_vcpu_op(int cmd, unsigned int vcpuid, XEN_GUEST_HANDLE_PARAM(void) arg)
     case VCPUOP_register_runstate_memory_area:
     {
         struct vcpu_register_runstate_memory_area area;
-        struct vcpu_runstate_info runstate;
 
         rc = -EFAULT;
         if ( copy_from_guest(&area, arg, 1) )
             break;
 
-        if ( !guest_handle_okay(area.addr.h, 1) )
+        unmap_runstate_area(v);
+        rc = map_runstate_area(v, &area);
+        if ( rc )
             break;
 
-        rc = 0;
-        runstate_guest(v) = area.addr.h;
-
-        if ( v == current )
-        {
-            __copy_to_guest(runstate_guest(v), &v->runstate, 1);
-        }
-        else
-        {
-            vcpu_runstate_get(v, &runstate);
-            __copy_to_guest(runstate_guest(v), &runstate, 1);
-        }
+        update_runstate_area(v);
 
         break;
     }
diff --git a/xen/include/asm-x86/domain.h b/xen/include/asm-x86/domain.h
index 277f99f633..70b3aeb9a8 100644
--- a/xen/include/asm-x86/domain.h
+++ b/xen/include/asm-x86/domain.h
@@ -554,7 +554,6 @@ struct pv_vcpu
     uint32_t dr7_emul;
 
     /* Deferred VA-based update state. */
-    bool_t need_update_runstate_area;
     struct vcpu_time_info pending_system_time;
 };
 
@@ -646,7 +645,7 @@ struct guest_memory_policy
 void update_guest_memory_policy(struct vcpu *v,
                                 struct guest_memory_policy *policy);
 
-bool update_runstate_area(struct vcpu *);
+void update_runstate_area(struct vcpu *);
 bool update_secondary_system_time(struct vcpu *,
                                   struct vcpu_time_info *);
 
diff --git a/xen/include/xen/domain.h b/xen/include/xen/domain.h
index d1bfc82f57..090a54df97 100644
--- a/xen/include/xen/domain.h
+++ b/xen/include/xen/domain.h
@@ -118,4 +118,9 @@ struct vnuma_info {
 
 void vnuma_destroy(struct vnuma_info *vnuma);
 
+struct vcpu_register_runstate_memory_area;
+int map_runstate_area(struct vcpu *v,
+                      struct vcpu_register_runstate_memory_area *area);
+void unmap_runstate_area(struct vcpu *v);
+
 #endif /* __XEN_DOMAIN_H__ */
diff --git a/xen/include/xen/sched.h b/xen/include/xen/sched.h
index 86f15b11e0..cb4bec9e0c 100644
--- a/xen/include/xen/sched.h
+++ b/xen/include/xen/sched.h
@@ -163,15 +163,15 @@ struct vcpu
     void            *sched_priv;    /* scheduler-specific data */
 
     struct vcpu_runstate_info runstate;
+    mfn_t runstate_mfn[2];
+    unsigned int runstate_nr;
 #ifndef CONFIG_COMPAT
-# define runstate_guest(v) ((v)->runstate_guest)
-    XEN_GUEST_HANDLE(vcpu_runstate_info_t) runstate_guest; /* guest address */
+    struct vcpu_runstate_info *runstate_guest;
 #else
-# define runstate_guest(v) ((v)->runstate_guest.native)
     union {
-        XEN_GUEST_HANDLE(vcpu_runstate_info_t) native;
-        XEN_GUEST_HANDLE(vcpu_runstate_info_compat_t) compat;
-    } runstate_guest; /* guest address */
+        struct vcpu_runstate_info *runstate_guest;
+        struct compat_vcpu_runstate_info *compat_runstate_guest;
+    };
 #endif
 
     /* last time when vCPU is scheduled out */


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: RT Xen on ARM - R-Car series

[Julien Grall]

Hi,

On 01/02/2019 16:53, Roger Pau Monné wrote:
> On Thu, Jan 31, 2019 at 11:14:37PM +0000, Julien Grall wrote:
>> On 1/31/19 9:56 PM, Stefano Stabellini wrote:
>>> On Thu, 31 Jan 2019, Julien Grall wrote:
>>>> On 31/01/2019 12:00, Andrii Anisov wrote:
>>>>> On 31.01.19 13:37, Julien Grall wrote:
> So, I've got a hacky patch to 'fix' this on x86, by taking a reference
> to the mfn behind the virtual address provided when setting up the
> hypercall and mapping it in Xen.

That was the idea I had in mind :). Hopefully, no guest is modifying the mapping 
(i.e the virtual address point to a different physical address) afterwards.

> This however doesn't work on ARM due
> to the lack of paging_gva_to_gfn. I guess there's something similar to
> translate a guest virtual address into a gfn or a mfn?

get_page_from_gva should to the job for you.
> +int map_runstate_area(struct vcpu *v,
> +                      struct vcpu_register_runstate_memory_area *area)
> +{
> +    unsigned long offset;
> +    unsigned int i;
> +    struct domain *d = v->domain;
> +    size_t size =
> +#ifdef CONFIG_COMPAT
> +        has_32bit_shinfo((v)->domain) ? sizeof(*v->compat_runstate_guest) :
> +#endif
> +                                        sizeof(*v->runstate_guest);
> +
> +    if ( v->runstate_guest )
> +    {
> +        ASSERT_UNREACHABLE();
> +        return -EBUSY;
> +    }
> +
> +    offset = area->addr.p & ~PAGE_MASK;
> +
> +    for ( i = 0; i < ARRAY_SIZE(v->runstate_mfn); i++ )
> +    {
> +        p2m_type_t t;
> +        uint32_t pfec = PFEC_page_present;
> +        gfn_t gfn = _gfn(paging_gva_to_gfn(v, area->addr.p, &pfec));
> +        struct page_info *pg;
> +
> +        if ( gfn_eq(gfn, INVALID_GFN) )
> +            return -EFAULT;
> +
> +        v->runstate_mfn[i] = get_gfn(d, gfn_x(gfn), &t);

get_gfn would need to be implemented on Arm.

> +        if ( t != p2m_ram_rw || mfn_eq(v->runstate_mfn[i], INVALID_MFN) )
> +            return -EFAULT;
> +
> +        pg = mfn_to_page(v->runstate_mfn[i]);
> +        if ( !pg || !get_page_type(pg, PGT_writable_page) )
> +        {
> +            put_gfn(d, gfn_x(gfn));
> +            return -EFAULT;
> +        }
> +
> +        put_gfn(d, gfn_x(gfn));
> +
> +        if ( PFN_DOWN(gfn_x(gfn) + size) == PFN_DOWN(gfn_x(gfn)) )

This looks wrong, you seem to mix address and frame. I think you might want:

if ( gfn_eq(gfn_add(gfn, PFN_DOWN(size)), gfn) )

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: RT Xen on ARM - R-Car series

[Andrii Anisov]

Hello,

On 01.02.19 19:40, Julien Grall wrote:
> On 01/02/2019 16:53, Roger Pau Monné wrote:
>> On Thu, Jan 31, 2019 at 11:14:37PM +0000, Julien Grall wrote:
>>> On 1/31/19 9:56 PM, Stefano Stabellini wrote:
>>>> On Thu, 31 Jan 2019, Julien Grall wrote:
>>>>> On 31/01/2019 12:00, Andrii Anisov wrote:
>>>>>> On 31.01.19 13:37, Julien Grall wrote:
>> So, I've got a hacky patch to 'fix' this on x86, by taking a reference
>> to the mfn behind the virtual address provided when setting up the
>> hypercall and mapping it in Xen.
> 
> That was the idea I had in mind :). 
Looks interesting.

> Hopefully, no guest is modifying the mapping (i.e the virtual address point to a different physical address) afterwards.
I guess, that mapping should not be moved around. Otherwise it would be broken even with the current implementation.
Moreover, having that buffer mapped to XEN will reduce context switch time as a side effect.

>> This however doesn't work on ARM due
>> to the lack of paging_gva_to_gfn. I guess there's something similar to
>> translate a guest virtual address into a gfn or a mfn?
> 
> get_page_from_gva should to the job for you.
>> +int map_runstate_area(struct vcpu *v,
>> +                      struct vcpu_register_runstate_memory_area *area)
>> +{
>> +    unsigned long offset;
>> +    unsigned int i;
>> +    struct domain *d = v->domain;
>> +    size_t size =
>> +#ifdef CONFIG_COMPAT
>> +        has_32bit_shinfo((v)->domain) ? sizeof(*v->compat_runstate_guest) :
>> +#endif
>> +                                        sizeof(*v->runstate_guest);
>> +
>> +    if ( v->runstate_guest )
>> +    {
>> +        ASSERT_UNREACHABLE();
>> +        return -EBUSY;
>> +    }
>> +
>> +    offset = area->addr.p & ~PAGE_MASK;
>> +
>> +    for ( i = 0; i < ARRAY_SIZE(v->runstate_mfn); i++ )
>> +    {
>> +        p2m_type_t t;
>> +        uint32_t pfec = PFEC_page_present;
>> +        gfn_t gfn = _gfn(paging_gva_to_gfn(v, area->addr.p, &pfec));
>> +        struct page_info *pg;
>> +
>> +        if ( gfn_eq(gfn, INVALID_GFN) )
>> +            return -EFAULT;
>> +
>> +        v->runstate_mfn[i] = get_gfn(d, gfn_x(gfn), &t);
> 
> get_gfn would need to be implemented on Arm.
I'm going to step into this, tomorrow I guess. I have to finish smth today.

-- 
Sincerely,
Andrii Anisov.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: RT Xen on ARM - R-Car series

[Julien Grall]

On 2/4/19 10:28 AM, Andrii Anisov wrote:
> Hello,

Hi Andrii,


> On 01.02.19 19:40, Julien Grall wrote:
>> On 01/02/2019 16:53, Roger Pau Monné wrote:
>>> On Thu, Jan 31, 2019 at 11:14:37PM +0000, Julien Grall wrote:
>>>> On 1/31/19 9:56 PM, Stefano Stabellini wrote:
>>>>> On Thu, 31 Jan 2019, Julien Grall wrote:
>>>>>> On 31/01/2019 12:00, Andrii Anisov wrote:
>>>>>>> On 31.01.19 13:37, Julien Grall wrote:
>>> So, I've got a hacky patch to 'fix' this on x86, by taking a reference
>>> to the mfn behind the virtual address provided when setting up the
>>> hypercall and mapping it in Xen.
>>
>> That was the idea I had in mind :). 
> Looks interesting.
> 
>> Hopefully, no guest is modifying the mapping (i.e the virtual address 
>> point to a different physical address) afterwards.
> I guess, that mapping should not be moved around. Otherwise it would be 
> broken even with the current implementation.

What I meant is the virtual address stays the same but the guest 
physical address may change. I don't see how this could be broken today, 
can you explain it?

> Moreover, having that buffer mapped to XEN will reduce context switch 
> time as a side effect.

I am still unsure whether we really want to keep that always mapped.

Each guest can support up to 128 vCPUs. So we would have 128 runstates 
mapped. Each runstate would take up to 2 pages. This means that each 
guest would require up to 1MB of vmap.

The VMAP in Xen is quite limited (1GB at most) and shared with device 
mapping (e.g ITS...).

On the other side, not mapping the pages contiguously is going to be a 
pain. So maybe the downside is worth it. It would be interesting to have 
the pros/cons of each solution written down in the series.

> 
>>> This however doesn't work on ARM due
>>> to the lack of paging_gva_to_gfn. I guess there's something similar to
>>> translate a guest virtual address into a gfn or a mfn?
>>
>> get_page_from_gva should to the job for you.
>>> +int map_runstate_area(struct vcpu *v,
>>> +                      struct vcpu_register_runstate_memory_area *area)
>>> +{
>>> +    unsigned long offset;
>>> +    unsigned int i;
>>> +    struct domain *d = v->domain;
>>> +    size_t size =
>>> +#ifdef CONFIG_COMPAT
>>> +        has_32bit_shinfo((v)->domain) ? 
>>> sizeof(*v->compat_runstate_guest) :
>>> +#endif
>>> +                                        sizeof(*v->runstate_guest);
>>> +
>>> +    if ( v->runstate_guest )
>>> +    {
>>> +        ASSERT_UNREACHABLE();
>>> +        return -EBUSY;
>>> +    }
>>> +
>>> +    offset = area->addr.p & ~PAGE_MASK;
>>> +
>>> +    for ( i = 0; i < ARRAY_SIZE(v->runstate_mfn); i++ )
>>> +    {
>>> +        p2m_type_t t;
>>> +        uint32_t pfec = PFEC_page_present;
>>> +        gfn_t gfn = _gfn(paging_gva_to_gfn(v, area->addr.p, &pfec));
>>> +        struct page_info *pg;
>>> +
>>> +        if ( gfn_eq(gfn, INVALID_GFN) )
>>> +            return -EFAULT;
>>> +
>>> +        v->runstate_mfn[i] = get_gfn(d, gfn_x(gfn), &t);
>>
>> get_gfn would need to be implemented on Arm.
> I'm going to step into this, tomorrow I guess. I have to finish smth today.

I thought more about it during the week-end. I would actually not 
implement get_gfn but implement a function similar to get_page_from_gva 
on x86. The reason behind this is the function on Arm is quite complex 
as it caters many different use case.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: RT Xen on ARM - R-Car series

[Andrii Anisov]

On 04.02.19 13:21, Julien Grall wrote:
> What I meant is the virtual address stays the same but the guest physical address may change. I don't see how this could be broken today, can you explain it?

I suppose guest's mapping change is not quite atomic from the hypervisor point of view, so domain could be caught in the middle.

> 
>> Moreover, having that buffer mapped to XEN will reduce context switch time as a side effect.
> 
> I am still unsure whether we really want to keep that always mapped.
> 
> Each guest can support up to 128 vCPUs. So we would have 128 runstates mapped. Each runstate would take up to 2 pages. This means that each guest would require up to 1MB of vmap.

Here buffer allocation on XEN side might benefit, even aligning/fitting the runstate into one page might work. But I understand it is undesirable and requires lot of changes

> I thought more about it during the week-end. I would actually not implement get_gfn but implement a function similar to get_page_from_gva on x86. The reason behind this is the function on Arm is quite complex as it caters many different use case.

I'll look into this. But I have to massage my yocto first.

-- 
Sincerely,
Andrii Anisov.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: RT Xen on ARM - R-Car series

[Julien Grall]

[-- Attachment #1.1: Type: text/plain, Size: 2042 bytes --]

Hi,

Sorry for the formatting.

On Mon, 4 Feb 2019, 15:52 Andrii Anisov, <andrii.anisov@gmail.com> wrote:

>
>
> On 04.02.19 13:21, Julien Grall wrote:
> > What I meant is the virtual address stays the same but the guest
> physical address may change. I don't see how this could be broken today,
> can you explain it?
>
> I suppose guest's mapping change is not quite atomic from the hypervisor
> point of view, so domain could be caught in the middle.
>

Well that's an issue with any hypercall using virtual address. You are not
even protected because of shatterring.

Thanksfully that does not seem to happen on Linux. Although, I have seen
error some times on osstest...

Virtual address has other issues if you can't access page-tables. So
ideally, we want to switch all hypercalls to use guest physical address.


> >
> >> Moreover, having that buffer mapped to XEN will reduce context switch
> time as a side effect.
> >
> > I am still unsure whether we really want to keep that always mapped.
> >
> > Each guest can support up to 128 vCPUs. So we would have 128 runstates
> mapped. Each runstate would take up to 2 pages. This means that each guest
> would require up to 1MB of vmap.
>
> Here buffer allocation on XEN side might benefit, even aligning/fitting
> the runstate into one page might work. But I understand it is undesirable
> and requires lot of changes
>

For a new interface,  the way to go is using guest physical address. Any
hypercall using virtual address should be killed.


> > I thought more about it during the week-end. I would actually not
> implement get_gfn but implement a function similar to get_page_from_gva on
> x86. The reason behind this is the function on Arm is quite complex as it
> caters many different use case.
>
> I'll look into this. But I have to massage my yocto first.
>

Thank you!



> --
> Sincerely,
> Andrii Anisov.
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xenproject.org
> https://lists.xenproject.org/mailman/listinfo/xen-devel

[-- Attachment #1.2: Type: text/html, Size: 3226 bytes --]

[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: RT Xen on ARM - R-Car series

[Roger Pau Monné]

On Fri, Feb 01, 2019 at 05:40:14PM +0000, Julien Grall wrote:
> Hi,
> 
> On 01/02/2019 16:53, Roger Pau Monné wrote:
> > On Thu, Jan 31, 2019 at 11:14:37PM +0000, Julien Grall wrote:
> > > On 1/31/19 9:56 PM, Stefano Stabellini wrote:
> > > > On Thu, 31 Jan 2019, Julien Grall wrote:
> > > > > On 31/01/2019 12:00, Andrii Anisov wrote:
> > > > > > On 31.01.19 13:37, Julien Grall wrote:
> > So, I've got a hacky patch to 'fix' this on x86, by taking a reference
> > to the mfn behind the virtual address provided when setting up the
> > hypercall and mapping it in Xen.
> 
> That was the idea I had in mind :). Hopefully, no guest is modifying the
> mapping (i.e the virtual address point to a different physical address)
> afterwards.
> 
> > This however doesn't work on ARM due
> > to the lack of paging_gva_to_gfn. I guess there's something similar to
> > translate a guest virtual address into a gfn or a mfn?
> 
> get_page_from_gva should to the job for you.
> > +int map_runstate_area(struct vcpu *v,
> > +                      struct vcpu_register_runstate_memory_area *area)
> > +{
> > +    unsigned long offset;
> > +    unsigned int i;
> > +    struct domain *d = v->domain;
> > +    size_t size =
> > +#ifdef CONFIG_COMPAT
> > +        has_32bit_shinfo((v)->domain) ? sizeof(*v->compat_runstate_guest) :
> > +#endif
> > +                                        sizeof(*v->runstate_guest);
> > +
> > +    if ( v->runstate_guest )
> > +    {
> > +        ASSERT_UNREACHABLE();
> > +        return -EBUSY;
> > +    }
> > +
> > +    offset = area->addr.p & ~PAGE_MASK;
> > +
> > +    for ( i = 0; i < ARRAY_SIZE(v->runstate_mfn); i++ )
> > +    {
> > +        p2m_type_t t;
> > +        uint32_t pfec = PFEC_page_present;
> > +        gfn_t gfn = _gfn(paging_gva_to_gfn(v, area->addr.p, &pfec));
> > +        struct page_info *pg;
> > +
> > +        if ( gfn_eq(gfn, INVALID_GFN) )
> > +            return -EFAULT;
> > +
> > +        v->runstate_mfn[i] = get_gfn(d, gfn_x(gfn), &t);
> 
> get_gfn would need to be implemented on Arm.
> 
> > +        if ( t != p2m_ram_rw || mfn_eq(v->runstate_mfn[i], INVALID_MFN) )
> > +            return -EFAULT;
> > +
> > +        pg = mfn_to_page(v->runstate_mfn[i]);
> > +        if ( !pg || !get_page_type(pg, PGT_writable_page) )
> > +        {
> > +            put_gfn(d, gfn_x(gfn));
> > +            return -EFAULT;
> > +        }
> > +
> > +        put_gfn(d, gfn_x(gfn));
> > +
> > +        if ( PFN_DOWN(gfn_x(gfn) + size) == PFN_DOWN(gfn_x(gfn)) )
> 
> This looks wrong, you seem to mix address and frame. I think you might want:
> 
> if ( gfn_eq(gfn_add(gfn, PFN_DOWN(size)), gfn) )

Thanks!

Here is an updated version which seems to build on ARM. I don't have
an easy way to test this, could you give it a spin?

I don't like adding CONFIG_X86/ARM in common code, so it might be
worth to either try to factor this out into arch specific code, or
even better, provide common functions to translate a guest virtual
address into a gfn, mfn or page.

---8<---
diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
index 6dc633ed50..6fb69a9e12 100644
--- a/xen/arch/arm/domain.c
+++ b/xen/arch/arm/domain.c
@@ -275,31 +275,25 @@ static void ctxt_switch_to(struct vcpu *n)
 }
 
 /* Update per-VCPU guest runstate shared memory area (if registered). */
-static void update_runstate_area(struct vcpu *v)
+void update_runstate_area(struct vcpu *v)
 {
-    void __user *guest_handle = NULL;
-
-    if ( guest_handle_is_null(runstate_guest(v)) )
+    if ( !v->runstate_guest )
         return;
 
     if ( VM_ASSIST(v->domain, runstate_update_flag) )
     {
-        guest_handle = &v->runstate_guest.p->state_entry_time + 1;
-        guest_handle--;
         v->runstate.state_entry_time |= XEN_RUNSTATE_UPDATE;
-        __raw_copy_to_guest(guest_handle,
-                            (void *)(&v->runstate.state_entry_time + 1) - 1, 1);
+        v->runstate_guest->state_entry_time |= XEN_RUNSTATE_UPDATE;
         smp_wmb();
     }
 
-    __copy_to_guest(runstate_guest(v), &v->runstate, 1);
+    memcpy(v->runstate_guest, &v->runstate, sizeof(v->runstate));
 
-    if ( guest_handle )
+    if ( VM_ASSIST(v->domain, runstate_update_flag) )
     {
         v->runstate.state_entry_time &= ~XEN_RUNSTATE_UPDATE;
         smp_wmb();
-        __raw_copy_to_guest(guest_handle,
-                            (void *)(&v->runstate.state_entry_time + 1) - 1, 1);
+        v->runstate_guest->state_entry_time &= ~XEN_RUNSTATE_UPDATE;
     }
 }
 
diff --git a/xen/arch/x86/domain.c b/xen/arch/x86/domain.c
index 32dc4253ff..b86d8d94c7 100644
--- a/xen/arch/x86/domain.c
+++ b/xen/arch/x86/domain.c
@@ -1604,26 +1604,22 @@ void paravirt_ctxt_switch_to(struct vcpu *v)
 }
 
 /* Update per-VCPU guest runstate shared memory area (if registered). */
-bool update_runstate_area(struct vcpu *v)
+void update_runstate_area(struct vcpu *v)
 {
-    bool rc;
     struct guest_memory_policy policy = { .nested_guest_mode = false };
-    void __user *guest_handle = NULL;
 
-    if ( guest_handle_is_null(runstate_guest(v)) )
-        return true;
+    if ( !v->runstate_guest )
+        return;
 
     update_guest_memory_policy(v, &policy);
 
     if ( VM_ASSIST(v->domain, runstate_update_flag) )
     {
-        guest_handle = has_32bit_shinfo(v->domain)
-            ? &v->runstate_guest.compat.p->state_entry_time + 1
-            : &v->runstate_guest.native.p->state_entry_time + 1;
-        guest_handle--;
         v->runstate.state_entry_time |= XEN_RUNSTATE_UPDATE;
-        __raw_copy_to_guest(guest_handle,
-                            (void *)(&v->runstate.state_entry_time + 1) - 1, 1);
+        if ( has_32bit_shinfo((v)->domain) )
+            v->compat_runstate_guest->state_entry_time |= XEN_RUNSTATE_UPDATE;
+        else
+            v->runstate_guest->state_entry_time |= XEN_RUNSTATE_UPDATE;
         smp_wmb();
     }
 
@@ -1632,31 +1628,22 @@ bool update_runstate_area(struct vcpu *v)
         struct compat_vcpu_runstate_info info;
 
         XLAT_vcpu_runstate_info(&info, &v->runstate);
-        __copy_to_guest(v->runstate_guest.compat, &info, 1);
-        rc = true;
+        memcpy(v->compat_runstate_guest, &info, sizeof(info));
     }
     else
-        rc = __copy_to_guest(runstate_guest(v), &v->runstate, 1) !=
-             sizeof(v->runstate);
+        memcpy(v->runstate_guest, &v->runstate, sizeof(v->runstate));
 
-    if ( guest_handle )
+    if ( VM_ASSIST(v->domain, runstate_update_flag) )
     {
         v->runstate.state_entry_time &= ~XEN_RUNSTATE_UPDATE;
         smp_wmb();
-        __raw_copy_to_guest(guest_handle,
-                            (void *)(&v->runstate.state_entry_time + 1) - 1, 1);
+        if ( has_32bit_shinfo((v)->domain) )
+            v->compat_runstate_guest->state_entry_time &= ~XEN_RUNSTATE_UPDATE;
+        else
+            v->runstate_guest->state_entry_time &= ~XEN_RUNSTATE_UPDATE;
     }
 
     update_guest_memory_policy(v, &policy);
-
-    return rc;
-}
-
-static void _update_runstate_area(struct vcpu *v)
-{
-    if ( !update_runstate_area(v) && is_pv_vcpu(v) &&
-         !(v->arch.flags & TF_kernel_mode) )
-        v->arch.pv.need_update_runstate_area = 1;
 }
 
 static inline bool need_full_gdt(const struct domain *d)
@@ -1779,7 +1766,7 @@ void context_switch(struct vcpu *prev, struct vcpu *next)
 
     if ( prev != next )
     {
-        _update_runstate_area(prev);
+        update_runstate_area(prev);
         vpmu_switch_from(prev);
         np2m_schedule(NP2M_SCHEDLE_OUT);
     }
@@ -1841,7 +1828,7 @@ void context_switch(struct vcpu *prev, struct vcpu *next)
 
     if ( prev != next )
     {
-        _update_runstate_area(next);
+        update_runstate_area(next);
 
         /* Must be done with interrupts enabled */
         vpmu_switch_to(next);
diff --git a/xen/arch/x86/hvm/dom0_build.c b/xen/arch/x86/hvm/dom0_build.c
index a571d15c13..3046bd4bc9 100644
--- a/xen/arch/x86/hvm/dom0_build.c
+++ b/xen/arch/x86/hvm/dom0_build.c
@@ -421,7 +421,7 @@ static int __init pvh_setup_p2m(struct domain *d)
     pvh_setup_e820(d, nr_pages);
     do {
         preempted = false;
-        paging_set_allocation(d, dom0_paging_pages(d, nr_pages),
+        paging_set_allocation(d, dom0_paging_pages(d, nr_pages) * 10,
                               &preempted);
         process_pending_softirqs();
     } while ( preempted );
diff --git a/xen/arch/x86/pv/domain.c b/xen/arch/x86/pv/domain.c
index 7e84b04082..027bcf4eb0 100644
--- a/xen/arch/x86/pv/domain.c
+++ b/xen/arch/x86/pv/domain.c
@@ -313,9 +313,6 @@ static void _toggle_guest_pt(struct vcpu *v)
     if ( !(v->arch.flags & TF_kernel_mode) )
         return;
 
-    if ( v->arch.pv.need_update_runstate_area && update_runstate_area(v) )
-        v->arch.pv.need_update_runstate_area = 0;
-
     if ( v->arch.pv.pending_system_time.version &&
          update_secondary_system_time(v, &v->arch.pv.pending_system_time) )
         v->arch.pv.pending_system_time.version = 0;
diff --git a/xen/arch/x86/x86_64/domain.c b/xen/arch/x86/x86_64/domain.c
index c46dccc25a..9734cd39ef 100644
--- a/xen/arch/x86/x86_64/domain.c
+++ b/xen/arch/x86/x86_64/domain.c
@@ -22,34 +22,21 @@ arch_compat_vcpu_op(
     {
     case VCPUOP_register_runstate_memory_area:
     {
-        struct compat_vcpu_register_runstate_memory_area area;
-        struct compat_vcpu_runstate_info info;
-
-        area.addr.p = 0;
+        union {
+            struct compat_vcpu_register_runstate_memory_area compat;
+            struct vcpu_register_runstate_memory_area native;
+        } area = { };
 
         rc = -EFAULT;
-        if ( copy_from_guest(&area.addr.h, arg, 1) )
+        if ( copy_from_guest(&area.compat.addr.v, arg, 1) )
             break;
 
-        if ( area.addr.h.c != area.addr.p ||
-             !compat_handle_okay(area.addr.h, 1) )
+        unmap_runstate_area(v);
+        rc = map_runstate_area(v, &area.native);
+        if ( rc )
             break;
 
-        rc = 0;
-        guest_from_compat_handle(v->runstate_guest.compat, area.addr.h);
-
-        if ( v == current )
-        {
-            XLAT_vcpu_runstate_info(&info, &v->runstate);
-        }
-        else
-        {
-            struct vcpu_runstate_info runstate;
-
-            vcpu_runstate_get(v, &runstate);
-            XLAT_vcpu_runstate_info(&info, &runstate);
-        }
-        __copy_to_guest(v->runstate_guest.compat, &info, 1);
+        update_runstate_area(v);
 
         break;
     }
diff --git a/xen/common/domain.c b/xen/common/domain.c
index c623daec56..3b362a38c8 100644
--- a/xen/common/domain.c
+++ b/xen/common/domain.c
@@ -15,6 +15,7 @@
 #include <xen/mm.h>
 #include <xen/event.h>
 #include <xen/vm_event.h>
+#include <xen/vmap.h>
 #include <xen/time.h>
 #include <xen/console.h>
 #include <xen/softirq.h>
@@ -1177,7 +1178,7 @@ int domain_soft_reset(struct domain *d)
 
     for_each_vcpu ( d, v )
     {
-        set_xen_guest_handle(runstate_guest(v), NULL);
+        unmap_runstate_area(v);
         unmap_vcpu_info(v);
     }
 
@@ -1318,6 +1319,96 @@ void unmap_vcpu_info(struct vcpu *v)
     put_page_and_type(mfn_to_page(mfn));
 }
 
+int map_runstate_area(struct vcpu *v,
+                      struct vcpu_register_runstate_memory_area *area)
+{
+    unsigned long offset;
+    unsigned int i;
+    size_t size =
+#ifdef CONFIG_COMPAT
+        has_32bit_shinfo((v)->domain) ? sizeof(*v->compat_runstate_guest) :
+#endif
+                                        sizeof(*v->runstate_guest);
+
+    if ( v->runstate_guest || v->runstate_nr )
+    {
+        ASSERT_UNREACHABLE();
+        return -EBUSY;
+    }
+
+    offset = area->addr.p & ~PAGE_MASK;
+
+    for ( i = 0; i < ARRAY_SIZE(v->runstate_mfn); i++ )
+    {
+        struct page_info *pg;
+#ifdef CONFIG_X86
+        p2m_type_t t;
+        struct domain *d = v->domain;
+        uint32_t pfec = PFEC_page_present;
+        gfn_t gfn = _gfn(paging_gva_to_gfn(v, area->addr.p, &pfec));
+
+        if ( gfn_eq(gfn, INVALID_GFN) )
+            goto release;
+
+        v->runstate_mfn[i] = get_gfn(d, gfn_x(gfn), &t);
+        if ( t != p2m_ram_rw || mfn_eq(v->runstate_mfn[i], INVALID_MFN) )
+            goto release;
+
+        pg = mfn_to_page(v->runstate_mfn[i]);
+        if ( !pg || !get_page_and_type(pg, d, PGT_writable_page) )
+        {
+            put_gfn(d, gfn_x(gfn));
+            goto release;
+        }
+        put_gfn(d, gfn_x(gfn));
+#elif defined(CONFIG_ARM)
+        pg = get_page_from_gva(v, area->addr.p, GV2M_WRITE);
+        if ( !pg || !get_page_type(pg, PGT_writable_page) )
+            goto release;
+
+        v->runstate_mfn[i] = page_to_mfn(pg);
+#else
+#error Unsopported arquitecture
+#endif
+
+        v->runstate_nr++;
+
+        if ( offset + size <= PAGE_SIZE )
+            break;
+
+        area->addr.p += PAGE_SIZE - offset;
+    }
+
+    v->runstate_guest = vmap(v->runstate_mfn, v->runstate_nr);
+    if ( !v->runstate_guest )
+        goto release;
+    v->runstate_guest = (void *)v->runstate_guest + offset;
+
+    return 0;
+
+ release:
+    for ( i = 0; i < v->runstate_nr; i++)
+        put_page_and_type(mfn_to_page(v->runstate_mfn[i]));
+    v->runstate_nr = 0;
+
+    return -EFAULT;
+}
+
+void unmap_runstate_area(struct vcpu *v)
+{
+    unsigned int i;
+
+    if ( !v->runstate_guest )
+        return;
+
+    vunmap(v->runstate_guest);
+    for ( i = 0; i < v->runstate_nr; i++ )
+        put_page_and_type(mfn_to_page(v->runstate_mfn[i]));
+
+    v->runstate_guest = NULL;
+    v->runstate_nr = 0;
+}
+
 int default_initialise_vcpu(struct vcpu *v, XEN_GUEST_HANDLE_PARAM(void) arg)
 {
     struct vcpu_guest_context *ctxt;
@@ -1495,27 +1586,17 @@ long do_vcpu_op(int cmd, unsigned int vcpuid, XEN_GUEST_HANDLE_PARAM(void) arg)
     case VCPUOP_register_runstate_memory_area:
     {
         struct vcpu_register_runstate_memory_area area;
-        struct vcpu_runstate_info runstate;
 
         rc = -EFAULT;
         if ( copy_from_guest(&area, arg, 1) )
             break;
 
-        if ( !guest_handle_okay(area.addr.h, 1) )
+        unmap_runstate_area(v);
+        rc = map_runstate_area(v, &area);
+        if ( rc )
             break;
 
-        rc = 0;
-        runstate_guest(v) = area.addr.h;
-
-        if ( v == current )
-        {
-            __copy_to_guest(runstate_guest(v), &v->runstate, 1);
-        }
-        else
-        {
-            vcpu_runstate_get(v, &runstate);
-            __copy_to_guest(runstate_guest(v), &runstate, 1);
-        }
+        update_runstate_area(v);
 
         break;
     }
diff --git a/xen/include/asm-x86/domain.h b/xen/include/asm-x86/domain.h
index 277f99f633..5cf22a32a2 100644
--- a/xen/include/asm-x86/domain.h
+++ b/xen/include/asm-x86/domain.h
@@ -554,7 +554,6 @@ struct pv_vcpu
     uint32_t dr7_emul;
 
     /* Deferred VA-based update state. */
-    bool_t need_update_runstate_area;
     struct vcpu_time_info pending_system_time;
 };
 
@@ -646,7 +645,6 @@ struct guest_memory_policy
 void update_guest_memory_policy(struct vcpu *v,
                                 struct guest_memory_policy *policy);
 
-bool update_runstate_area(struct vcpu *);
 bool update_secondary_system_time(struct vcpu *,
                                   struct vcpu_time_info *);
 
diff --git a/xen/include/xen/domain.h b/xen/include/xen/domain.h
index d1bfc82f57..88d3d96317 100644
--- a/xen/include/xen/domain.h
+++ b/xen/include/xen/domain.h
@@ -118,4 +118,10 @@ struct vnuma_info {
 
 void vnuma_destroy(struct vnuma_info *vnuma);
 
+struct vcpu_register_runstate_memory_area;
+int map_runstate_area(struct vcpu *v,
+                      struct vcpu_register_runstate_memory_area *area);
+void unmap_runstate_area(struct vcpu *v);
+void update_runstate_area(struct vcpu *);
+
 #endif /* __XEN_DOMAIN_H__ */
diff --git a/xen/include/xen/sched.h b/xen/include/xen/sched.h
index 86f15b11e0..cb4bec9e0c 100644
--- a/xen/include/xen/sched.h
+++ b/xen/include/xen/sched.h
@@ -163,15 +163,15 @@ struct vcpu
     void            *sched_priv;    /* scheduler-specific data */
 
     struct vcpu_runstate_info runstate;
+    mfn_t runstate_mfn[2];
+    unsigned int runstate_nr;
 #ifndef CONFIG_COMPAT
-# define runstate_guest(v) ((v)->runstate_guest)
-    XEN_GUEST_HANDLE(vcpu_runstate_info_t) runstate_guest; /* guest address */
+    struct vcpu_runstate_info *runstate_guest;
 #else
-# define runstate_guest(v) ((v)->runstate_guest.native)
     union {
-        XEN_GUEST_HANDLE(vcpu_runstate_info_t) native;
-        XEN_GUEST_HANDLE(vcpu_runstate_info_compat_t) compat;
-    } runstate_guest; /* guest address */
+        struct vcpu_runstate_info *runstate_guest;
+        struct compat_vcpu_runstate_info *compat_runstate_guest;
+    };
 #endif
 
     /* last time when vCPU is scheduled out */


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: RT Xen on ARM - R-Car series

[Julien Grall]

Hi,

On 2/4/19 12:53 PM, Roger Pau Monné wrote:
> On Fri, Feb 01, 2019 at 05:40:14PM +0000, Julien Grall wrote:
>> Hi,
>>
>> On 01/02/2019 16:53, Roger Pau Monné wrote:
>>> On Thu, Jan 31, 2019 at 11:14:37PM +0000, Julien Grall wrote:
>>>> On 1/31/19 9:56 PM, Stefano Stabellini wrote:
>>>>> On Thu, 31 Jan 2019, Julien Grall wrote:
>>>>>> On 31/01/2019 12:00, Andrii Anisov wrote:
>>>>>>> On 31.01.19 13:37, Julien Grall wrote:
>>> So, I've got a hacky patch to 'fix' this on x86, by taking a reference
>>> to the mfn behind the virtual address provided when setting up the
>>> hypercall and mapping it in Xen.
>>
>> That was the idea I had in mind :). Hopefully, no guest is modifying the
>> mapping (i.e the virtual address point to a different physical address)
>> afterwards.
>>
>>> This however doesn't work on ARM due
>>> to the lack of paging_gva_to_gfn. I guess there's something similar to
>>> translate a guest virtual address into a gfn or a mfn?
>>
>> get_page_from_gva should to the job for you.
>>> +int map_runstate_area(struct vcpu *v,
>>> +                      struct vcpu_register_runstate_memory_area *area)
>>> +{
>>> +    unsigned long offset;
>>> +    unsigned int i;
>>> +    struct domain *d = v->domain;
>>> +    size_t size =
>>> +#ifdef CONFIG_COMPAT
>>> +        has_32bit_shinfo((v)->domain) ? sizeof(*v->compat_runstate_guest) :
>>> +#endif
>>> +                                        sizeof(*v->runstate_guest);
>>> +
>>> +    if ( v->runstate_guest )
>>> +    {
>>> +        ASSERT_UNREACHABLE();
>>> +        return -EBUSY;
>>> +    }
>>> +
>>> +    offset = area->addr.p & ~PAGE_MASK;
>>> +
>>> +    for ( i = 0; i < ARRAY_SIZE(v->runstate_mfn); i++ )
>>> +    {
>>> +        p2m_type_t t;
>>> +        uint32_t pfec = PFEC_page_present;
>>> +        gfn_t gfn = _gfn(paging_gva_to_gfn(v, area->addr.p, &pfec));
>>> +        struct page_info *pg;
>>> +
>>> +        if ( gfn_eq(gfn, INVALID_GFN) )
>>> +            return -EFAULT;
>>> +
>>> +        v->runstate_mfn[i] = get_gfn(d, gfn_x(gfn), &t);
>>
>> get_gfn would need to be implemented on Arm.
>>
>>> +        if ( t != p2m_ram_rw || mfn_eq(v->runstate_mfn[i], INVALID_MFN) )
>>> +            return -EFAULT;
>>> +
>>> +        pg = mfn_to_page(v->runstate_mfn[i]);
>>> +        if ( !pg || !get_page_type(pg, PGT_writable_page) )
>>> +        {
>>> +            put_gfn(d, gfn_x(gfn));
>>> +            return -EFAULT;
>>> +        }
>>> +
>>> +        put_gfn(d, gfn_x(gfn));
>>> +
>>> +        if ( PFN_DOWN(gfn_x(gfn) + size) == PFN_DOWN(gfn_x(gfn)) )
>>
>> This looks wrong, you seem to mix address and frame. I think you might want:
>>
>> if ( gfn_eq(gfn_add(gfn, PFN_DOWN(size)), gfn) )
> 
> Thanks!
> 
> Here is an updated version which seems to build on ARM. I don't have
> an easy way to test this, could you give it a spin?

Thank you for writing a patch. I am back in France this week for family 
reason and will not have time properly give a spin this week. Stefano, 
Andrii, can you test it?

> 
> I don't like adding CONFIG_X86/ARM in common code, so it might be
> worth to either try to factor this out into arch specific code, or
> even better, provide common functions to translate a guest virtual
> address into a gfn, mfn or page.

I would prefer GVA to MFN/page because we already have a fairly 
complicate helper handling all the cases (e.g memaccess, 
break-before-make, ...) for translation and taking a reference on the 
page. Although we could potentially split the function in two if we want 
to cater any other translation.

One comment below.

> +#elif defined(CONFIG_ARM)
> +        pg = get_page_from_gva(v, area->addr.p, GV2M_WRITE);
> +        if ( !pg || !get_page_type(pg, PGT_writable_page) )

One reference is already taken by get_page_from_gva. So you should not 
need to take another here.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: RT Xen on ARM - R-Car series

[Andrii Anisov]

On 04.02.19 23:58, Julien Grall wrote:
> Thank you for writing a patch. I am back in France this week for family reason and will not have time properly give a spin this week. Stefano, Andrii, can you test it?
I'll do that today.
Guys, could you please suggest what are evidences that runstate function properly?
Is it enough just to build and run it, so that it should not crash?

-- 
Sincerely,
Andrii Anisov.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: RT Xen on ARM - R-Car series

[Roger Pau Monné]

On Tue, Feb 05, 2019 at 10:40:19AM +0200, Andrii Anisov wrote:
> 
> 
> On 04.02.19 23:58, Julien Grall wrote:
> > Thank you for writing a patch. I am back in France this week for family reason and will not have time properly give a spin this week. Stefano, Andrii, can you test it?
> I'll do that today.
> Guys, could you please suggest what are evidences that runstate function properly?

You should see a reasonable amount of stolen time. A simple test would
be to spin 2 VMs with 1 vCPU each and pin them to the same physical
CPU. Then run a CPU intensive workload on each of them, and the 'st'
time shown in `top` should be ~50% I think.

> Is it enough just to build and run it, so that it should not crash?

Not crashing would be good indeed :).

Thanks, Roger.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: RT Xen on ARM - R-Car series

[Andrii Anisov]

Hello Roger,

On 05.02.19 11:45, Roger Pau Monné wrote:
> You should see a reasonable amount of stolen time. A simple test would
> be to spin 2 VMs with 1 vCPU each and pin them to the same physical
> CPU. Then run a CPU intensive workload on each of them, and the 'st'
> time shown in `top` should be ~50% I think.

My system, built with yocto employs busybox top. It does not show stolen time. I guess I have to build another top.

-- 
Sincerely,
Andrii Anisov.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: RT Xen on ARM - R-Car series

[Roger Pau Monné]

On Tue, Feb 05, 2019 at 11:54:12AM +0200, Andrii Anisov wrote:
> Hello Roger,
> 
> On 05.02.19 11:45, Roger Pau Monné wrote:
> > You should see a reasonable amount of stolen time. A simple test would
> > be to spin 2 VMs with 1 vCPU each and pin them to the same physical
> > CPU. Then run a CPU intensive workload on each of them, and the 'st'
> > time shown in `top` should be ~50% I think.
> 
> My system, built with yocto employs busybox top. It does not show stolen time. I guess I have to build another top.

You can also get this information from /proc/stat in a raw form.

Roger.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: RT Xen on ARM - R-Car series

[Andrii Anisov]

Hello,

On 05.02.19 10:40, Andrii Anisov wrote:
> On 04.02.19 23:58, Julien Grall wrote:
>> Thank you for writing a patch. I am back in France this week for family reason and will not have time properly give a spin this week. Stefano, Andrii, can you test it?
> I'll do that today.

I actually did it today.
So that it is compiled and runs. I uncommented that print, and it does not appear.
I see steal time slightly bouncing around 50% in a following configuration:
  - 4PCPUs
  - 2 VMs of 4 VCPUs
  - both VMs running `for i in 1 2 3 4; do dd if=/dev/zero of=/dev/null & done` from their consoles

It seems that results are as expected.

I also checked the build without the patch. I expected it would demonstrate more inconsistency in numbers, but no, it also shows 50% steal time pretty stable.

-- 
Sincerely,
Andrii Anisov.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: RT Xen on ARM - R-Car series

[Stefano Stabellini]

On Wed, 6 Feb 2019, Andrii Anisov wrote:
> Hello,
> 
> On 05.02.19 10:40, Andrii Anisov wrote:
> > On 04.02.19 23:58, Julien Grall wrote:
> > > Thank you for writing a patch. I am back in France this week for family
> > > reason and will not have time properly give a spin this week. Stefano,
> > > Andrii, can you test it?
> > I'll do that today.
> 
> I actually did it today.
> So that it is compiled and runs. I uncommented that print, and it does not
> appear.
> I see steal time slightly bouncing around 50% in a following configuration:
>  - 4PCPUs
>  - 2 VMs of 4 VCPUs
>  - both VMs running `for i in 1 2 3 4; do dd if=/dev/zero of=/dev/null & done`
> from their consoles
> 
> It seems that results are as expected.
> 
> I also checked the build without the patch. I expected it would demonstrate
> more inconsistency in numbers, but no, it also shows 50% steal time pretty
> stable.

That's great. Could you or Roger take care of cleaning up the patch and
properly submitting it to the list? And also double check that it won't
break any guests (at least the ones we know about: Linux and Windows on
x86).

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: RT Xen on ARM - R-Car series

[Andrii Anisov]

Hello All,

On 06.02.19 23:03, Stefano Stabellini wrote:
> That's great. Could you or Roger take care of cleaning up the patch and
> properly submitting it to the list?

I can take it for cleaning up.

> And also double check that it won't
> break any guests (at least the ones we know about: Linux and Windows on
> x86).

I'm not sure I could properly check it for x86. For sure can't do that for windows guest.

-- 
Sincerely,
Andrii Anisov.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: RT Xen on ARM - R-Car series

[Roger Pau Monné]

On Thu, Feb 07, 2019 at 11:42:16AM +0200, Andrii Anisov wrote:
> Hello All,
> 
> On 06.02.19 23:03, Stefano Stabellini wrote:
> > That's great. Could you or Roger take care of cleaning up the patch and
> > properly submitting it to the list?
> 
> I can take it for cleaning up.
> 
> > And also double check that it won't
> > break any guests (at least the ones we know about: Linux and Windows on
> > x86).
> 
> I'm not sure I could properly check it for x86. For sure can't do that for windows guest.

I've been thinking about this with other Citrix folks, and I'm not
sure the proposed patch is a good solution. It's not possible for us
to know whether there's a kernel somewhere relying on changing the
virtual address of the runtime state area without issuing a new
hypercall.

If such kernel existed by making this change we would introduce random
memory corruption to that kernel, which would be very hard to track
and considered a regression.

I think the best way to move forward is to pick my patch and introduce
a new hypercall that instead of a virtual address takes a guest
physical address. Will you be OK with this Andrii?

Note that the Linux kernel would also need to be modified to make use
of this new hypercall, but that's likely close to a 1 line change.

Thanks, Roger.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: RT Xen on ARM - R-Car series

[Julien Grall]

[-- Attachment #1.1: Type: text/plain, Size: 1406 bytes --]

(sorry for the formatting)

On Thu, 7 Feb 2019, 11:37 Roger Pau Monné, <roger.pau@citrix.com> wrote:

> On Thu, Feb 07, 2019 at 11:42:16AM +0200, Andrii Anisov wrote:
> > Hello All,
> >
> > On 06.02.19 23:03, Stefano Stabellini wrote:
> > > That's great. Could you or Roger take care of cleaning up the patch and
> > > properly submitting it to the list?
> >
> > I can take it for cleaning up.
> >
> > > And also double check that it won't
> > > break any guests (at least the ones we know about: Linux and Windows on
> > > x86).
> >
> > I'm not sure I could properly check it for x86. For sure can't do that
> for windows guest.
>
> I've been thinking about this with other Citrix folks, and I'm not
> sure the proposed patch is a good solution. It's not possible for us
> to know whether there's a kernel somewhere relying on changing the
> virtual address of the runtime state area without issuing a new
> hypercall.
>
> If such kernel existed by making this change we would introduce random
> memory corruption to that kernel, which would be very hard to track
> and considered a regression.
>
> I think the best way to move forward is to pick my patch and introduce
> a new hypercall that instead of a virtual address takes a guest
> physical address. Will you be OK with this Andrii
>

In that case I would prefer if we don't keep the runstate mapped.

Cheers,

[-- Attachment #1.2: Type: text/html, Size: 1816 bytes --]

[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: RT Xen on ARM - R-Car series

[Andrii Anisov]

Hello Julien,

On 07.02.19 12:59, Julien Grall wrote:
> In that case I would prefer if we don't keep the runstate mapped.

Actually I'm going to see runstate update impact on the context switch time. For that I will extend TBM with runstate setup.
I really do not like a bunch of `copy_to_guest()` done on each context switch because of runstate.
If you are against runstate mapping, I'd like to fit runstate into a page, than access it directly from the hypervisor.

-- 
Sincerely,
Andrii Anisov.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: RT Xen on ARM - R-Car series

[Julien Grall]

[-- Attachment #1.1: Type: text/plain, Size: 1139 bytes --]

On Tue, 12 Feb 2019, 19:21 Andrii Anisov, <andrii.anisov@gmail.com> wrote:

> Hello Julien,
>

Hi,

Sorry for the formatting.


> On 07.02.19 12:59, Julien Grall wrote:
> > In that case I would prefer if we don't keep the runstate mapped.
>
> Actually I'm going to see runstate update impact on the context switch
> time. For that I will extend TBM with runstate setup.
> I really do not like a bunch of `copy_to_guest()` done on each context
> switch because of runstate.


Please provide more meaningful arguments other than "I don't like it". I
provided potential drawbacks on my previous e-mails that you haven't yet
addressed.

FWIW, Volodymyr had the same argument on OP-TEE and I requested to avoid
the global mapping. I haven't seen any concern for performance devredation
afterwards. Please feel free to come with numbers here.

If you are against runstate mapping, I'd like to fit runstate into a page,
> than access it directly from the hypervisor.


This is not really a policy in Xen. If the guest cares about it, it can
ensure that the runstate does not cross a page boundary.

Cheers,


> --
> Sincerely,
> Andrii Anisov.
>

[-- Attachment #1.2: Type: text/html, Size: 2085 bytes --]

[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: RT Xen on ARM - R-Car series

[Andrii Anisov]

Hello Julien,

On 12.02.19 21:21, Julien Grall wrote:
> Please provide more meaningful arguments other than "I don't like it". I provided potential drawbacks on my previous e-mails that you haven't yet addressed.

Well, currently, on each runstate update, `copy_to_guest()` which translates the buffer virtual address is called three times. On context switch, runstate update is done twice, when we change a vcpu.

> FWIW, Volodymyr had the same argument on OP-TEE and I requested to avoid the global mapping. I haven't seen any concern for performance devredation afterwards. Please feel free to come with numbers here.
The numbers here are controversial and really confusing to me.
I've measured the raw `update_runstate_area()` execution time. With runstate mapped - its execution time is less than my timer tick (120ns), with runstate not mapped - I've seen its execution time as 4 to 8 ticks (480-960ns).
But using TBM, I encountered that making runstate mapped with Roger's patch increases the IRQ latency from ~7000ns to ~7900ns. It is opposite to my expectations and to the raw decrease of `runstate_update_area()` execution time.
Also glmark2 benchmark shown me slight performance drop on getting runstate mapping.

-- 
Sincerely,
Andrii Anisov.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: RT Xen on ARM - R-Car series

[Julien Grall]

On Thu, Feb 14, 2019 at 3:18 PM Andrii Anisov <andrii.anisov@gmail.com> wrote:
>
> Hello Julien,

Hi,

Sorry for the formatting, I am using the gmail web-interface for once.


> On 12.02.19 21:21, Julien Grall wrote:
> > Please provide more meaningful arguments other than "I don't like it". I provided potential drawbacks on my previous e-mails that you haven't yet addressed.
>
> Well, currently, on each runstate update, `copy_to_guest()` which translates the buffer virtual address is called three times. On context switch, runstate update is done twice, when we change a vcpu.

I am not sure why you are speaking about the current implementation
when my point was about the new implementation.

I guess your point stick even if we decide to use guest physical
address. Although, I am still unconvinced of the benefits to keep it
mapped.

>
> > FWIW, Volodymyr had the same argument on OP-TEE and I requested to avoid the global mapping. I haven't seen any concern for performance devredation afterwards. Please feel free to come with numbers here.
> The numbers here are controversial and really confusing to me.
> I've measured the raw `update_runstate_area()` execution time. With runstate mapped - its execution time is less than my timer tick (120ns), with runstate not mapped - I've seen
its execution time as 4 to 8 ticks (480-960ns).

Please provide the code you use to measure it. How often do you call it?

> But using TBM, I encountered that making runstate mapped with Roger's patch increases the IRQ latency from ~7000ns to ~7900ns. It is opposite to my expectations and to the raw decrease of `runstate_update_area()` execution time.

Raw benchmarks should be taken with a grain of salt. The more if you
only benchmark a single function as the context switch may introduce
latency/cache eviction.

Although, I would have expected the numbers to be the same. What is
your configuration here? Do you have others guests running? How many
context switch do you have?

Also, what are the modifications you made in TBM to use runstate?

> Also glmark2 benchmark shown me slight performance drop on getting runstate mapping.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: RT Xen on ARM - R-Car series

[Andrii Anisov]

Hello Julien,

On 14.02.19 19:29, Julien Grall wrote:
> I am not sure why you are speaking about the current implementation
> when my point was about the new implementation.
> 
> I guess your point stick even if we decide to use guest physical
> address.
For sure. The type of guest address used by hypervisor to reach runstate area and having that area mapped are quite orthogonal questions. But, IMHO, tightly coupled and might be solved together.

> Although, I am still unconvinced of the benefits to keep it
> mapped.
My point was reducing context switch time, but those controversial numbers left me confused.

>> I've measured the raw `update_runstate_area()` execution time. With runstate mapped - its execution time is less than my timer tick (120ns), with runstate not mapped - I've seen
> its execution time as 4 to 8 ticks (480-960ns).
> 
> Please provide the code you use to measure it. How often do you call it?
The code to see that is as simple as following:

diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
index 0a2e997..d673d00 100644
--- a/xen/arch/arm/domain.c
+++ b/xen/arch/arm/domain.c
@@ -314,8 +314,16 @@ static void schedule_tail(struct vcpu *prev)
      context_saved(prev);
  
      if ( prev != current )
+    {
+        s_time_t t = 0;
+        if (current->domain->domain_id == 1)
+            t = NOW();
          update_runstate_area(current);
  
+        if (current->domain->domain_id == 1)
+            printk("cp = %"PRI_stime"\n", NOW()-t);
+    }
+
      /* Ensure that the vcpu has an up-to-date time base. */
      update_vcpu_system_time(current);
  }

>> But using TBM, I encountered that making runstate mapped with Roger's patch increases the IRQ latency from ~7000ns to ~7900ns. It is opposite to my expectations and to the raw decrease of `runstate_update_area()` execution time.
> 
> Raw benchmarks should be taken with a grain of salt. The more if you
> only benchmark a single function as the context switch may introduce
> latency/cache eviction.
> 
> Although, I would have expected the numbers to be the same. What is
> your configuration here? Do you have others guests running? How many
> context switch do you have?

The configuration is the same as here [1].


> Also, what are the modifications you made in TBM to use runstate?

I've just registered the runstate area with following [2].

[1] https://lists.xenproject.org/archives/html/xen-devel/2018-12/msg00881.html
[2] https://github.com/aanisov/tbm/commit/806e8a7e6e6c72aef10f1c8a579424252aca2635


-- 
Sincerely,
Andrii Anisov.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: RT Xen on ARM - R-Car series

[Julien Grall]

On Fri, Feb 15, 2019 at 12:30 PM Andrii Anisov <andrii.anisov@gmail.com> wrote:
>
> Hello Julien,

Hi,

> On 14.02.19 19:29, Julien Grall wrote:
> > I am not sure why you are speaking about the current implementation
> > when my point was about the new implementation.
> >
> > I guess your point stick even if we decide to use guest physical
> > address.
> For sure. The type of guest address used by hypervisor to reach runstate area and having that area mapped are quite orthogonal questions. But, IMHO, tightly coupled and might be solved together.

Not really... This is an implementation details that does not matter
on the OS side.

>
> > Although, I am still unconvinced of the benefits to keep it
> > mapped.
> My point was reducing context switch time, but those controversial numbers left me confused.

You missed my point.  I don't say performance is not important but you
have to take into account the drawbacks.
I am not entirely happy to keep the runstate always mapped if it uses
more memory in Xen (vmap is quite limited)
and does not make significant improvement in the context switch time.

>
> >> I've measured the raw `update_runstate_area()` execution time. With runstate mapped - its execution time is less than my timer tick (120ns), with runstate not mapped - I've seen
> > its execution time as 4 to 8 ticks (480-960ns).
> >
> > Please provide the code you use to measure it. How often do you call it?
> The code to see that is as simple as following:
>
> diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
> index 0a2e997..d673d00 100644
> --- a/xen/arch/arm/domain.c
> +++ b/xen/arch/arm/domain.c
> @@ -314,8 +314,16 @@ static void schedule_tail(struct vcpu *prev)
>       context_saved(prev);
>
>       if ( prev != current )
> +    {
> +        s_time_t t = 0;
> +        if (current->domain->domain_id == 1)
> +            t = NOW();

If you want accurate number, then the NOW() macro is not sufficient
enough. Read to CNTPCTL_EL0 can occur speculatively and out of order
relative to other instructions executed on the same PE. So the PE can
potentially execute CNTPCTL_EL0 before update_runstate_area(...).

You would want to add an isb() at least before NOW() and potentially
after (unless you have register dependency).
I have a patch for adding an isb()  in the NOW() macro. I will send it later on.

Note that I have no idea whether the isb()s matter on the processors
you use. But it would be best to have it to make sure the numbers are
accurate.

>           update_runstate_area(current);
>
> +        if (current->domain->domain_id == 1)
> +            printk("cp = %"PRI_stime"\n", NOW()-t);

That's only one number.  Did you do an average over multiple context
switch (say 1000) ?

> +    }
> +
>       /* Ensure that the vcpu has an up-to-date time base. */
>       update_vcpu_system_time(current);
>   }
>
> >> But using TBM, I encountered that making runstate mapped with Roger's patch increases the IRQ latency from ~7000ns to ~7900ns. It is opposite to my expectations and to the raw decrease of `runstate_update_area()` execution time.
> >
> > Raw benchmarks should be taken with a grain of salt. The more if you
> > only benchmark a single function as the context switch may introduce
> > latency/cache eviction.
> >
> > Although, I would have expected the numbers to be the same. What is
> > your configuration here? Do you have others guests running? How many
> > context switch do you have?
>
> The configuration is the same as here [1].

Well, your e-mail contains multiple configuration. From my
understanding, TTBM will run exclusively on one CPU, so you will look
at context switch between idle vCPU and TTBM.
Do you trap wfi/wfe?

Also, you didn't answer to my last question regarding the number of
context switch. How long do you leave the test run?

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: RT Xen on ARM - R-Car series

[Andrii Anisov]

On 15.02.19 19:13, Julien Grall wrote:
> Not really... This is an implementation details that does not matter
> on the OS side.
And on hypervisor side?

> If you want accurate number, then the NOW() macro is not sufficient
> enough. Read to CNTPCTL_EL0 can occur speculatively and out of order
> relative to other instructions executed on the same PE. So the PE can
> potentially execute CNTPCTL_EL0 before update_runstate_area(...).
> 
> You would want to add an isb() at least before NOW() and potentially
> after (unless you have register dependency).
> I have a patch for adding an isb()  in the NOW() macro. I will send it later on.Good hint.

> 
>>            update_runstate_area(current);
>>
>> +        if (current->domain->domain_id == 1)
>> +            printk("cp = %"PRI_stime"\n", NOW()-t);
> 
> That's only one number.  Did you do an average over multiple context
> switch (say 1000) ?
No, just got them on the console to see the numbers. They seems to not change significantly during a minute run.
That's why I did not answer about context switches number.

I can do it in a bit more complicated way. But I do not expect I would encounter differences.

-- 
Sincerely,
Andrii Anisov.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: RT Xen on ARM - R-Car series

[Julien Grall]

[-- Attachment #1.1: Type: text/plain, Size: 2090 bytes --]

On Fri, 15 Feb 2019, 18:41 Andrii Anisov, <andrii.anisov@gmail.com> wrote:

>
> On 15.02.19 19:13, Julien Grall wrote:
> > Not really... This is an implementation details that does not matter
> > on the OS side.
> And on hypervisor side?
>

My point is the interface should be the same whether we decide to keep the
runstate mapped or not. This is just an implementation details.


> > If you want accurate number, then the NOW() macro is not sufficient
> > enough. Read to CNTPCTL_EL0 can occur speculatively and out of order
> > relative to other instructions executed on the same PE. So the PE can
> > potentially execute CNTPCTL_EL0 before update_runstate_area(...).
> >
> > You would want to add an isb() at least before NOW() and potentially
> > after (unless you have register dependency).
> > I have a patch for adding an isb()  in the NOW() macro. I will send it
> later on.Good hint.
>
> >
> >>            update_runstate_area(current);
> >>
> >> +        if (current->domain->domain_id == 1)
> >> +            printk("cp = %"PRI_stime"\n", NOW()-t);
> >
> > That's only one number.  Did you do an average over multiple context
> > switch (say 1000) ?
> No, just got them on the console to see the numbers. They seems to not
> change significantly during a minute run.
>

This does not really tell me a rough number of context switch... The more,
I still don't have the full pictures on your setup as you haven't answered
to my question on the previous e-mail...

Can you please get the number of context switch and the average value?

That's why I did not answer about context switches number


Couldn't you have said that from the beginning? This is rather annoying to
have question ignored.
If you want meaningful feedback, then please start answering *all*
questions. I would have been happy if you said "I didn't do it".

I can do it in a bit more complicated way. But I do not expect I would
> encounter differences.
>

A grep and sed should help to confirm your finding. I want to be sure that
the numbers are correct and not just an impression.

Cheers,

-- 
Julien Grall

[-- Attachment #1.2: Type: text/html, Size: 3204 bytes --]

[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: RT Xen on ARM - R-Car series

[Andrii Anisov]

Hello Roger,

Sorry for a delayed response.

On 07.02.19 12:35, Roger Pau Monné wrote:
> I've been thinking about this with other Citrix folks, and I'm not
> sure the proposed patch is a good solution. It's not possible for us
> to know whether there's a kernel somewhere relying on changing the
> virtual address of the runtime state area without issuing a new
> hypercall.

Do you mean allocating another buffer by VM and mapping it to a virtual address known to XEN? Or remapping existing buffer to a different virtual address?

> If such kernel existed by making this change we would introduce random
> memory corruption to that kernel, which would be very hard to track
> and considered a regression.

I guess you actually mean that VM is trying to map another physical buffer to a vaddr known to XEN. As I said here [1], even current implementation looks problematic, because VM's changes in PT are not atomic from the hypervisor point of view.
I stated that for ARM, but x86 does not seem to differ here.

Actually VM trying to make changes behind a hypervisor's back is a really bad idea. Because the hypervisor *is* always behind the VM's back.

> I think the best way to move forward is to pick my patch and introduce
> a new hypercall that instead of a virtual address takes a guest
> physical address. Will you be OK with this Andrii?
It might work better for this. And introducing a new interface is a chance to get rid of a mixed legacy.

> Note that the Linux kernel would also need to be modified to make use
> of this new hypercall, but that's likely close to a 1 line change.I would not say it is a 1 line change.
I think about a page alignment for the runstate area, I'd like to have it directly accessed from XEN. I do not like `update_runstate_area()` with its all kind of copy_to_guest, done twice a context switch.

[1] https://lists.xenproject.org/archives/html/xen-devel/2019-02/msg00164.html

-- 
Sincerely,
Andrii Anisov.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: RT Xen on ARM - R-Car series

[Roger Pau Monné]

On Tue, Feb 12, 2019 at 08:21:32PM +0200, Andrii Anisov wrote:
> Hello Roger,
> 
> Sorry for a delayed response.
> 
> On 07.02.19 12:35, Roger Pau Monné wrote:
> > I've been thinking about this with other Citrix folks, and I'm not
> > sure the proposed patch is a good solution. It's not possible for us
> > to know whether there's a kernel somewhere relying on changing the
> > virtual address of the runtime state area without issuing a new
> > hypercall.
> 
> Do you mean allocating another buffer by VM and mapping it to a virtual address known to XEN? Or remapping existing buffer to a different virtual address?

I meant that with the current interface a user could change the
backing memory behind the virtual address passed in the runstate
register hypercall and expect Xen to write to the new physical memory
area without having to do anything else.

Attempting to do that with my proposed patch can result in hard to
debug guest memory corruption.

> > If such kernel existed by making this change we would introduce random
> > memory corruption to that kernel, which would be very hard to track
> > and considered a regression.
> 
> I guess you actually mean that VM is trying to map another physical buffer to a vaddr known to XEN. As I said here [1], even current implementation looks problematic, because VM's changes in PT are not atomic from the hypervisor point of view.
> I stated that for ARM, but x86 does not seem to differ here.

OSes use atomic operations to update a PTE, so I'm not sure how that
could be problematic. Xen will either get the new or the old address
from the PTE, but never a half-written value. Any OS wanting to do
this should be very careful about how it's page tables are updated.

> Actually VM trying to make changes behind a hypervisor's back is a really bad idea. Because the hypervisor *is* always behind the VM's back.
> 
> > I think the best way to move forward is to pick my patch and introduce
> > a new hypercall that instead of a virtual address takes a guest
> > physical address. Will you be OK with this Andrii?
> It might work better for this. And introducing a new interface is a chance to get rid of a mixed legacy.

Yes, I think introducing a new hypercall is the best way to move
forward.

> > Note that the Linux kernel would also need to be modified to make use
> > of this new hypercall, but that's likely close to a 1 line change.I would not say it is a 1 line change.
> I think about a page alignment for the runstate area, I'd like to have it directly accessed from XEN. I do not like `update_runstate_area()` with its all kind of copy_to_guest, done twice a context switch.

In order to simplify stuff the new interface could require runstate
areas to be page aligned, but I think the check can be relaxed to
simply require runstate areas to not cross a page boundary.

If you use a guest physical address for the interface you don't have
to use copy_to_guest because that expects a guest virtual addresses.

Thanks, Roger.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: RT Xen on ARM - R-Car series

[Andrii Anisov]

Hello Roger,

On 14.02.19 18:29, Roger Pau Monné wrote:
> I meant that with the current interface a user could change the
> backing memory behind the virtual address passed in the runstate
> register hypercall and expect Xen to write to the new physical memory
> area without having to do anything else.
> 
> Attempting to do that with my proposed patch can result in hard to
> debug guest memory corruption.

It's true.

> OSes use atomic operations to update a PTE, so I'm not sure how that
> could be problematic. Xen will either get the new or the old address
> from the PTE, but never a half-written value.

I did mean using the old address, which I suppose might result in the same issues as you mentioned above.

> In order to simplify stuff the new interface could require runstate
> areas to be page aligned, but I think the check can be relaxed to
> simply require runstate areas to not cross a page boundary.

My idea so far is to keep the same `struct vcpu_runstate_info` but harden it with `__attribute__((aligned(64)))` right in the interface file vcpu.h. Also add some guard asserts verifying that its actual size is less than 64 bytes.
Then, on the new initcall verify if it crosses the page boundary.

-- 
Sincerely,
Andrii Anisov.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: RT Xen on ARM - R-Car series

[Julien Grall]

Hi,

On Fri, Feb 15, 2019 at 4:23 PM Andrii Anisov <andrii.anisov@gmail.com> wrote:
> On 14.02.19 18:29, Roger Pau Monné wrote:
> > In order to simplify stuff the new interface could require runstate
> > areas to be page aligned, but I think the check can be relaxed to
> > simply require runstate areas to not cross a page boundary.
>
> My idea so far is to keep the same `struct vcpu_runstate_info` but harden it with `__attribute__((aligned(64)))` right in the interface file vcpu.h. Also add some guard asserts verifying that its actual size is less than 64 bytes.

Why? Is it because you want to be cache-aligned?  If so, requiring the
structure to be 64-bytes is not enough. You also want the address to
be 64-bytes aligned.
However, I am not sure why we should bother with that in the
interface.... The cacheline varies between architectures and even SoC.
If an OS cares about it, then it can aligned itself here.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: RT Xen on ARM - R-Car series

[Andrii Anisov]

Hello Julien,

On 15.02.19 18:31, Julien Grall wrote:
> Why? Is it because you want to be cache-aligned?  If so, requiring the
> structure to be 64-bytes is not enough.

I did not mean caches.

> You also want the address to
> be 64-bytes aligned.

I would keep it as a hint for static/dynamic allocations in VMs, hoping the address would be normally 64 bytes aligned.
I hope it might be stronger than, only commenting it should not cross a page boundary. E.g. like `struct vcpu_register_vcpu_info` is commented.

I've got this idea after looking at runstate definition as per-cpu in Linux [1]

> If an OS cares about it, then it can aligned itself here.
I suppose we can hint the OS by structure alignment in the interface header, and require it from OS verifying it on hypercall handling.

[1] https://elixir.bootlin.com/linux/v5.0-rc6/source/drivers/xen/time.c#L22

-- 
Sincerely,
Andrii Anisov.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: RT Xen on ARM - R-Car series

[Julien Grall]

[-- Attachment #1.1: Type: text/plain, Size: 1431 bytes --]

Sorry for the formatting.

On Fri, 15 Feb 2019, 18:30 Andrii Anisov, <andrii.anisov@gmail.com> wrote:

> Hello Julien,
>
> On 15.02.19 18:31, Julien Grall wrote:
> > Why? Is it because you want to be cache-aligned?  If so, requiring the
> > structure to be 64-bytes is not enough.
>
> I did not mean caches.
>

What is the reason then?

> You also want the address to
> > be 64-bytes aligned.
>
> I would keep it as a hint for static/dynamic allocations in VMs, hoping
> the address would be normally 64 bytes aligned.
> I hope it might be stronger than, only commenting it should not cross a
> page boundary. E.g. like `struct vcpu_register_vcpu_info` is commented.
>
> I've got this idea after looking at runstate definition as per-cpu in
> Linux [1]
>

It is not obvious why it would be 64-bytes alignment from the definition.
Can you please explain the rationale to impose that alignment?

I really appreciate you suggest ideas/patches but  it would be helpful if
you provide rationale at the same time. This would avoid a round of e-mails
just for asking the reasons and delay the interesting bits.


> > If an OS cares about it, then it can aligned itself here.
> I suppose we can hint the OS by structure alignment in the interface
> header, and require it from OS verifying it on hypercall handling

.


> [1]
> https://elixir.bootlin.com/linux/v5.0-rc6/source/drivers/xen/time.c#L22
>
> --
> Sincerely,
> Andrii Anisov.
>

[-- Attachment #1.2: Type: text/html, Size: 2554 bytes --]

[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: RT Xen on ARM - R-Car series

[Julien Grall]

Hi,

Sorry for the formatting, using the gmail web-interface.

On Tue, Feb 12, 2019 at 7:23 PM Andrii Anisov <andrii.anisov@gmail.com> wrote:
>
> Hello Roger,
>
> Sorry for a delayed response.
>
> On 07.02.19 12:35, Roger Pau Monné wrote:
> > I've been thinking about this with other Citrix folks, and I'm not
> > sure the proposed patch is a good solution. It's not possible for us
> > to know whether there's a kernel somewhere relying on changing the
> > virtual address of the runtime state area without issuing a new
> > hypercall.
>
> Do you mean allocating another buffer by VM and mapping it to a virtual address known to XEN? Or remapping existing buffer to a different virtual address?
>
> > If such kernel existed by making this change we would introduce random
> > memory corruption to that kernel, which would be very hard to track
> > and considered a regression.
>
> I guess you actually mean that VM is trying to map another physical buffer to a vaddr known to XEN. As I said here [1], even current implementation looks problematic, because VM's changes in PT are not atomic from the hypervisor point of view.

The Arm Arm requires the modification in the page-table to be atomic.
Otherwise you may face strange behavior as the entry may be use by the
page-table walker at the same time. This is the same in the hypervisor
case, you will either see the old value, a non-existent entry (because
of break-before-make), or the new value.

> I stated that for ARM, but x86 does not seem to differ here.
>
> Actually VM trying to make changes behind a hypervisor's back is a really bad idea. Because the hypervisor *is* always behind the VM's back.

You can't really control what a VM is doing with its page-tables. An
OS may decide to shatter/gather pages in smaller/bigger mapping. This
is actually done by Linux for memory used by userspace. We have some
workaround in the privcmd driver for preventing Linux to play with
buffer passed via hypercall. Although, I suspect this is not enough
given the weird error I see on osstest time to time.

Another issue with the virtual address is in the case of a blind
hypervisor. The OS may decide to hide its page-tables from the
hypervisor and therefore you would not be able to translate a virtual
address.

This is why virtual address should be avoided as much as possible.
IIRC, there was some discussion during Xen Summit 2017 in Budapest
about using guest physical address rather than guest virtual address.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: RT Xen on ARM - R-Car series

[Roger Pau Monné]

On Mon, Feb 04, 2019 at 09:58:43PM +0000, Julien Grall wrote:
> Hi,
> 
> On 2/4/19 12:53 PM, Roger Pau Monné wrote:
> > On Fri, Feb 01, 2019 at 05:40:14PM +0000, Julien Grall wrote:
> > > Hi,
> > > 
> > > On 01/02/2019 16:53, Roger Pau Monné wrote:
> > Thanks!
> > 
> > Here is an updated version which seems to build on ARM. I don't have
> > an easy way to test this, could you give it a spin?
> 
> Thank you for writing a patch. I am back in France this week for family
> reason and will not have time properly give a spin this week. Stefano,
> Andrii, can you test it?

No problem. I'm also happy if someone wants to pick this up, test it
and formally post it to xen-devel as a patch with whatever changes are
required.

> > 
> > I don't like adding CONFIG_X86/ARM in common code, so it might be
> > worth to either try to factor this out into arch specific code, or
> > even better, provide common functions to translate a guest virtual
> > address into a gfn, mfn or page.
> 
> I would prefer GVA to MFN/page because we already have a fairly complicate
> helper handling all the cases (e.g memaccess, break-before-make, ...) for
> translation and taking a reference on the page. Although we could
> potentially split the function in two if we want to cater any other
> translation.

I think providing a helper to translate a guest VA into a gfn might be
helpful in some cases, but I don't have a strong opinion. I would
however prefer to have some kind of common helper here that's
implemented for all arches.

> One comment below.
> 
> > +#elif defined(CONFIG_ARM)
> > +        pg = get_page_from_gva(v, area->addr.p, GV2M_WRITE);
> > +        if ( !pg || !get_page_type(pg, PGT_writable_page) )
> 
> One reference is already taken by get_page_from_gva. So you should not need
> to take another here.

This is taking a type reference, which is a noop on ARM, but matters
on x86. This needs to be done on ARM so that the cleanup path can be
shared between x86 and ARM, which uses put_page_and_type.

get_page_from_gva performs a get_page, but not a get_page_type (or a
get_page_and_type).

Thanks, Roger.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: RT Xen on ARM - R-Car series

[Andrii Anisov]

On 31.01.19 23:56, Stefano Stabellini wrote:
> I ran into this problem as well not too long ago too. It is very
> annoying and it is basically impossible to work-around, the only thing
> possible would be to suppress the warning, but that doesn't even count
> as a work-around :-)
Well, yet it is adopted for non-debug builds de facto :\

-- 
Sincerely,
Andrii Anisov.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: RT Xen on ARM - R-Car series

[Julien Grall]

On 2/1/19 10:07 AM, Andrii Anisov wrote:
> 
> 
> On 31.01.19 23:56, Stefano Stabellini wrote:
>> I ran into this problem as well not too long ago too. It is very
>> annoying and it is basically impossible to work-around, the only thing
>> possible would be to suppress the warning, but that doesn't even count
>> as a work-around :-)
> Well, yet it is adopted for non-debug builds de facto :\

The thing is the presence of the printk is not the real problem. It only 
tells you the mapping is inexistent. The problem is if in debug build 
you don't see at all this message (assuming you have kpti enabled). This 
would mean you would overwrite a random page.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: RT Xen on ARM - R-Car series

[Andrii Anisov]

On 01.02.19 12:16, Julien Grall wrote:
> The thing is the presence of the printk is not the real problem.
It is the problem when the intention is to play in a sandbox with different things.

> It only tells you the mapping is inexistent. The problem is if in debug build you don't see at all this message (assuming you have kpti enabled). This would mean you would overwrite a random page.
I already have that understanding ;) But curious if XEN 4.12 is going to be released with such the issue remains?

-- 
Sincerely,
Andrii Anisov.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: RT Xen on ARM - R-Car series

[Julien Grall]

Hi Andrii,

On 2/1/19 10:35 AM, Andrii Anisov wrote:
> 
> 
> On 01.02.19 12:16, Julien Grall wrote:
>> The thing is the presence of the printk is not the real problem.
> It is the problem when the intention is to play in a sandbox with 
> different things.

I don't consider polluting your log a real problem. It is just an 
annoyance that could be fixed by using "ack -v"

> 
>> It only tells you the mapping is inexistent. The problem is if in 
>> debug build you don't see at all this message (assuming you have kpti 
>> enabled). This would mean you would overwrite a random page.
> I already have that understanding ;) But curious if XEN 4.12 is going to 
> be released with such the issue remains?

I don't consider it as a critical issue because of the type of guest we 
currently support, so it is not in my queue for Xen 4.12 fixes.

Feel free to suggest it as a blocker if you think it is.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: RT Xen on ARM - R-Car series

[Stefano Stabellini]

+Juergen

On Fri, 1 Feb 2019, Julien Grall wrote:
> Hi Andrii,
> 
> On 2/1/19 10:35 AM, Andrii Anisov wrote:
> > 
> > 
> > On 01.02.19 12:16, Julien Grall wrote:
> > > The thing is the presence of the printk is not the real problem.
> > It is the problem when the intention is to play in a sandbox with different
> > things.
> 
> I don't consider polluting your log a real problem. It is just an annoyance
> that could be fixed by using "ack -v"
> 
> > 
> > > It only tells you the mapping is inexistent. The problem is if in debug
> > > build you don't see at all this message (assuming you have kpti enabled).
> > > This would mean you would overwrite a random page.
> > I already have that understanding ;) But curious if XEN 4.12 is going to be
> > released with such the issue remains?
> 
> I don't consider it as a critical issue because of the type of guest we
> currently support, so it is not in my queue for Xen 4.12 fixes.
> 
> Feel free to suggest it as a blocker if you think it is.
> 
> Cheers,
> 
> -- 
> Julien Grall
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: RT Xen on ARM - R-Car series

[Andrii Anisov]

Hello Julien,

On 01.02.19 12:51, Julien Grall wrote:
> I don't consider polluting your log a real problem.
Unless the system becomes an insane typewriter :)

> I don't consider it as a critical issue because of the type of guest we currently support, so it is not in my queue for Xen 4.12 fixes.
> 
> Feel free to suggest it as a blocker if you think it is.
Well, for us the situation is as following: Renesas requested XEN 4.12 for the next release of their virtualization package. And they employ KPTI enabled kernel in the BSP.
That reveals another critical issue for us, in addition to Set/Way issue and possible performance drops/irq latency raise due to specter mitigation measures.
I'm not sure if this is sufficient a justification to make it the release blocker, but we are up to this stuff.

-- 
Sincerely,
Andrii Anisov.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: RT Xen on ARM - R-Car series

[Julien Grall]

On 2/4/19 10:32 AM, Andrii Anisov wrote:
> Hello Julien,

Hi Andrii,

> On 01.02.19 12:51, Julien Grall wrote:
>> I don't consider polluting your log a real problem.
> Unless the system becomes an insane typewriter :)
> 
>> I don't consider it as a critical issue because of the type of guest 
>> we currently support, so it is not in my queue for Xen 4.12 fixes.
>>
>> Feel free to suggest it as a blocker if you think it is.
> Well, for us the situation is as following: Renesas requested XEN 4.12 
> for the next release of their virtualization package.

That's a good news! Let me try to address your concerns below one by one.

> And they employ 
> KPTI enabled kernel in the BSP.
KPTI is going to work on Xen. There are no known issue with Linux as the 
virtual address is not going to be re-used for other purpose in the 
virtual address space. The only inconvenience is the message in debug build.

Just in case, I am not saying it should not be fixed :).

> That reveals another critical issue for us, in addition to Set/Way issue 

 From the discussion on the another thread and with other people, this 
is not entirely the fault of Xen. This was a misuse of the instructions 
by the driver. While you may want to deal with this in your case, I 
would like to avoid promoting bad behavior when using Xen upstream.

> and possible performance drops/irq latency raise due to specter 
> mitigation measures.

Can you remind me the cores you are using?

Also, when you mean possible, does it mean you haven't looked the 
performance regression?

> I'm not sure if this is sufficient a justification to make it the 
> release blocker, but we are up to this stuff.

Which one?

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: RT Xen on ARM - R-Car series

[Andrii Anisov]

On 04.02.19 13:36, Julien Grall wrote:
> That's a good news! Let me try to address your concerns below one by one.
Lets do it:)

>> And they employ KPTI enabled kernel in the BSP.
> KPTI is going to work on Xen. There are no known issue with Linux as the virtual address is not going to be re-used for other purpose in the virtual address space. The only inconvenience is the message in debug build.
> Just in case, I am not saying it should not be fixed :).
We have a decision to eliminate it.

>> That reveals another critical issue for us, in addition to Set/Way issue 
> 
>  From the discussion on the another thread and with other people, this is not entirely the fault of Xen. This was a misuse of the instructions by the driver.

Yep, but it is still on our team to solve.

> While you may want to deal with this in your case, I would like to avoid promoting bad behavior when using Xen upstream.

Ah, ok. Will say that in different words.

> 
>> and possible performance drops/irq latency raise due to specter mitigation measures.
> 
> Can you remind me the cores you are using?

A57+A53.

I see the following in my log:

     (XEN) alternatives: Patching with alt table 00000000002c6608 -> 00000000002c6c80
     (XEN) CPU0 will call ARM_SMCCC_ARCH_WORKAROUND_1 on exception entry
     (XEN) CPU2 will call ARM_SMCCC_ARCH_WORKAROUND_1 on exception entry
     (XEN) CPU3 will call ARM_SMCCC_ARCH_WORKAROUND_1 on exception entry
     (XEN) CPU1 will call ARM_SMCCC_ARCH_WORKAROUND_1 on exception entry

> Also, when you mean possible, does it mean you haven't looked the performance regression?
We have a preliminary results about performance drop with xen4.12-unstable comparing to a our system with 4.10.
But those results are from Nov IIRC, and we have to recheck. After fixing Set/Way misuse in our proprietary driver ;)

>> I'm not sure if this is sufficient a justification to make it the release blocker, but we are up to this stuff.
> Which one?

KPTI issue, as you suggested.

-- 
Sincerely,
Andrii Anisov.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: RT Xen on ARM - R-Car series

[Julien Grall]

Hi Andrii,

On 2/4/19 3:19 PM, Andrii Anisov wrote:
> 
> 
> On 04.02.19 13:36, Julien Grall wrote:
>> That's a good news! Let me try to address your concerns below one by one.
> Lets do it:)
> 
>>> And they employ KPTI enabled kernel in the BSP.
>> KPTI is going to work on Xen. There are no known issue with Linux as 
>> the virtual address is not going to be re-used for other purpose in 
>> the virtual address space. The only inconvenience is the message in 
>> debug build.
>> Just in case, I am not saying it should not be fixed :).
> We have a decision to eliminate it.
> 
>>> That reveals another critical issue for us, in addition to Set/Way issue 
>>
>>  From the discussion on the another thread and with other people, this 
>> is not entirely the fault of Xen. This was a misuse of the 
>> instructions by the driver.
> 
> Yep, but it is still on our team to solve.
> 
>> While you may want to deal with this in your case, I would like to 
>> avoid promoting bad behavior when using Xen upstream.
> 
> Ah, ok. Will say that in different words.
> 
>>
>>> and possible performance drops/irq latency raise due to specter 
>>> mitigation measures.
>>
>> Can you remind me the cores you are using?
> 
> A57+A53.
> 
> I see the following in my log:
> 
>      (XEN) alternatives: Patching with alt table 00000000002c6608 -> 
> 00000000002c6c80
>      (XEN) CPU0 will call ARM_SMCCC_ARCH_WORKAROUND_1 on exception entry
>      (XEN) CPU2 will call ARM_SMCCC_ARCH_WORKAROUND_1 on exception entry
>      (XEN) CPU3 will call ARM_SMCCC_ARCH_WORKAROUND_1 on exception entry
>      (XEN) CPU1 will call ARM_SMCCC_ARCH_WORKAROUND_1 on exception entry

Cortex-A53 should not be affected by spectre v2, so I imagine they are 
only for A57?

It is going to be hard to disable the workarounds by default. But we can 
consider to provide host-wide or per-guest option to disable them on 
trusted environment.

>> Also, when you mean possible, does it mean you haven't looked the 
>> performance regression?
> We have a preliminary results about performance drop with 
> xen4.12-unstable comparing to a our system with 4.10.

A lot of patches have not been backported in Xen 4.10 (including 
Spectre/Meltdown) that will definitely fix hole but may have an impact 
on the performance. There were not backported because of performance 
reason but because of the complexity of the port and seemly lack of 
interest.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: RT Xen on ARM - R-Car series

[Andrii Anisov]

On 05.02.19 00:06, Julien Grall wrote:
>> A57+A53.
>>
>> I see the following in my log:
>>
>>      (XEN) alternatives: Patching with alt table 00000000002c6608 -> 00000000002c6c80
>>      (XEN) CPU0 will call ARM_SMCCC_ARCH_WORKAROUND_1 on exception entry
>>      (XEN) CPU2 will call ARM_SMCCC_ARCH_WORKAROUND_1 on exception entry
>>      (XEN) CPU3 will call ARM_SMCCC_ARCH_WORKAROUND_1 on exception entry
>>      (XEN) CPU1 will call ARM_SMCCC_ARCH_WORKAROUND_1 on exception entry
> 
> Cortex-A53 should not be affected by spectre v2, so I imagine they are only for A57?
Yes, the log says the workaround is applied to the big cores only. As it should be.

> It is going to be hard to disable the workarounds by default. But we can consider to provide host-wide or per-guest option to disable them on trusted environment.
We have to get numbers first than decide how to proceed.

>>> Also, when you mean possible, does it mean you haven't looked the performance regression?
>> We have a preliminary results about performance drop with xen4.12-unstable comparing to a our system with 4.10.
> 
> A lot of patches have not been backported in Xen 4.10 (including Spectre/Meltdown) that will definitely fix hole but may have an impact on the performance. There were not backported because of performance reason but because of the complexity of the port and seemly lack of interest.
I know that story. But customers are customers. And performance drop in the next SW version is always painful for them.
So we need a good explanation (which Spectre mitigation might be), or better to show up no performance drop :).

-- 
Sincerely,
Andrii Anisov.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: RT Xen on ARM - R-Car series

[Stefano Stabellini]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 2035 bytes --]

On Tue, 5 Feb 2019, Andrii Anisov wrote:
> On 05.02.19 00:06, Julien Grall wrote:
> > > A57+A53.
> > > 
> > > I see the following in my log:
> > > 
> > >      (XEN) alternatives: Patching with alt table 00000000002c6608 ->
> > > 00000000002c6c80
> > >      (XEN) CPU0 will call ARM_SMCCC_ARCH_WORKAROUND_1 on exception entry
> > >      (XEN) CPU2 will call ARM_SMCCC_ARCH_WORKAROUND_1 on exception entry
> > >      (XEN) CPU3 will call ARM_SMCCC_ARCH_WORKAROUND_1 on exception entry
> > >      (XEN) CPU1 will call ARM_SMCCC_ARCH_WORKAROUND_1 on exception entry
> > 
> > Cortex-A53 should not be affected by spectre v2, so I imagine they are only
> > for A57?
> Yes, the log says the workaround is applied to the big cores only. As it
> should be.
> 
> > It is going to be hard to disable the workarounds by default. But we can
> > consider to provide host-wide or per-guest option to disable them on trusted
> > environment.
> We have to get numbers first than decide how to proceed.
> 
> > > > Also, when you mean possible, does it mean you haven't looked the
> > > > performance regression?
> > > We have a preliminary results about performance drop with xen4.12-unstable
> > > comparing to a our system with 4.10.
> > 
> > A lot of patches have not been backported in Xen 4.10 (including
> > Spectre/Meltdown) that will definitely fix hole but may have an impact on
> > the performance. There were not backported because of performance reason but
> > because of the complexity of the port and seemly lack of interest.
> I know that story. But customers are customers. And performance drop in the
> next SW version is always painful for them.
> So we need a good explanation (which Spectre mitigation might be), or better
> to show up no performance drop :).

I think it is acceptable to intruduce a "I know what I am doing, just
disable the fix" option. There might be cases where the user doesn't
care for Spectre mitigations. I see that Linux is going in this
direction of offering more disabling options too.

[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: RT Xen on ARM - R-Car series

[Andrii Anisov]

On 05.02.19 21:18, Stefano Stabellini wrote:
> I think it is acceptable to intruduce a "I know what I am doing, just
> disable the fix" option. There might be cases where the user doesn't
> care for Spectre mitigations. I see that Linux is going in this
> direction of offering more disabling options too.
This might be an option.
But we have to have numbers first. Then think of required measures.

-- 
Sincerely,
Andrii Anisov.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: RT Xen on ARM - R-Car series

[Julien Grall]

(CC Xen-devel)

On 31/01/2019 05:50, LOPEZ, FUENTES NACARINO Jairo Eduardo wrote:
> Hello Julien,
> 

Hi,

> 2019年1月30日(水) 21:12、Julien Grall さん（julien.grall@arm.com 
> <mailto:julien.grall@arm.com>）のメッセージ:
> 
>     Hi,
> 
>     On 30/01/2019 20:23, LOPEZ, FUENTES NACARINO Jairo Eduardo wrote:
> 
>     My e-mail client thinks the e-mail was sent from the future. Can you make sure
>     your timezone is set correctly?
> 
> 
> Very sorry about the timezone issue. We did have an NTP problem. Hopefully 
> things should be ok now.
> 
> 
>      > [    0.682996] CPU features: detected feature: Kernel page table
>     isolation (KPTI)
>      >
>      > (XEN) p2m.c:1844: d0v1: Failed to walk page-table va 0xffff800013f59c57
>      > (XEN) p2m.c:1844: d0v2: Failed to walk page-table va 0xffff800013f70c57
>      > (XEN) p2m.c:1844: d0v1: Failed to walk page-table va 0xffff800013f59c48
>      > (XEN) p2m.c:1844: d0v2: Failed to walk page-table va 0xffff800013f70c48
>      >
>      > </ this continues infinitely>
>      >
>      > Is this normal. Could the error have anything to do with me using the
>      > dom0_max_vpcus=4 bootarg for Xen?
> 
>     No this is a unknown benign issue when using KPTI kernel on Xen (see [1]). I
>     still haven't found time to fix it and would appreciate help to solve the issue.
> 
> 
> I had a lot of issues getting the R-Car M3 running as a user and only got it 
> working with Andrii's instructions. I am personally not that familiar with the 
> internal functionality of KPTI or Xen.

Don't worry, it was more an open question as many people seems to hit the issues 
but no-one have found time so far.

Regarding running Xen on R-Car M3, would you mind to update/add the instructions 
on the wiki?

> 
> If you have any tests you would want to do on my current configuration I will 
> gladly give it a go.

Thank you for the offer. I will let you know when I managed to reach that in my 
list.

Best regards,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: RT Xen on ARM - R-Car series

[Andrii Anisov]

[-- Attachment #1.1: Type: text/plain, Size: 22405 bytes --]

Kudos to Oleksandr Tyshchenko, he stepped into the same, and found the
rootcause.

The problem is in a patch https://patchwork.kernel.org/patch/10084561/ .

I guess you can workaround it with adding interrupts and interrupt-parent
properties to timer node through your dtb. Something like following:


 	&timer {


		interrupt-parent = <&gic>;

		interrupts = <GIC_PPI 13 (GIC_CPU_MASK_SIMPLE(8) | IRQ_TYPE_LEVEL_LOW)>,

				     <GIC_PPI 14 (GIC_CPU_MASK_SIMPLE(8) | IRQ_TYPE_LEVEL_LOW)>,

				     <GIC_PPI 11 (GIC_CPU_MASK_SIMPLE(8) | IRQ_TYPE_LEVEL_LOW)>,

				     <GIC_PPI 10 (GIC_CPU_MASK_SIMPLE(8) | IRQ_TYPE_LEVEL_LOW)>;

 	};



For sure the proper fix is something different, i.e. teaching xen to cope
with interrupts-extended.


Sent from my Android.

пт, 11 січ. 2019, 12:59 LOPEZ, FUENTES NACARINO Jairo Eduardo користувач
jairo@ruri.waseda.jp пише:

> Andrii,
>
> Due to some events on my side, I had to move from the R-Car H3 I was using
> to the R-Car M3 in my posession.
>
> I followed your initial advice and updated my environment to a newer
> version, opting to stick to the rocko version for the m3ulcb.
>
> The Yocto build configuration is shown below:
>
> Build Configuration:
> BB_VERSION           = "1.36.0"
> BUILD_SYS            = "x86_64-linux"
> NATIVELSBSTRING      = "universal-4.9"
> TARGET_SYS           = "aarch64-poky-linux"
> MACHINE              = "m3ulcb"
> DISTRO               = "poky"
> DISTRO_VERSION       = "2.4.2"
> TUNE_FEATURES        = "aarch64 cortexa57-cortexa53"
> TARGET_FPU           = ""
> SOC_FAMILY           = "rcar-gen3:r8a7796"
> meta
> meta-poky
> meta-yocto-bsp       = "HEAD:342fbd6a3e57021c8e28b124b3adb241936f3d9d"
> meta-rcar-gen3       = "rocko:8af0b7d6e445b532088a068dc012757001be3a1f"
> meta-optee           = "rocko:75dfb67bbb14a70cd47afda9726e2e1c76731885"
> meta-oe
> meta-python
> meta-networking      = "rocko:eae996301d9c097bcbeb8046f08041dc82bb62f8"
> meta-selinux         = "rocko:d855c624f32c5e599bf27e06cb8f5b25b3aae12d"
> meta-virtualization  = "rocko:b704c689b67639214b9568a3d62e82df27e9434f"
> meta-rcar-gen3-xen   = "master:60699c631d541aeeaebaeec9a087efed9385ee42"
>
> I had to modify the dtb to the version shown below:
>
> // SPDX-License-Identifier:
> GPL-2.0
> /*
>
>  * Device Tree Source for the M3ULCB (R-Car Starter Kit Pro)
> board
>  *
>
>  * Copyright (C) 2016-2018 Renesas Electronics
> Corp.
>  * Copyright (C) 2016 Cogent Embedded,
> Inc.
>  */
>
>
>
> /dts-v1/;
>
> #include
> "r8a7796.dtsi"
>
> #include
> "ulcb.dtsi"
>
>
>
> /
> {
>
>         model = "Renesas M3ULCB board based on
> r8a7796";
>         compatible = "renesas,m3ulcb",
> "renesas,r8a7796";
>
>
>         cpus
> {
>                 idle-states
> {
>                         /delete-node/
> cpu-sleep-1;
>
> };
>
> };
>
>
>
>         memory@48000000
> {
>                 device_type =
> "memory";
>                 /* first 128MB is reserved for secure area.
> */
>                 reg = <0x0 0x48000000 0x0
> 0x38000000>;
>         };
>
>         memory@600000000 {
>                 device_type = "memory";
>                 reg = <0x6 0x00000000 0x0 0x40000000>;
>         };
>
>         reserved-memory {
>                 #address-cells = <2>;
>                 #size-cells = <2>;
>                 ranges;
>
>                 /* device specific region for Lossy Decompression */
>                 lossy_decompress: linux,lossy_decompress@54000000 {
>                         no-map;
>                         reg = <0x00000000 0x54000000 0x0 0x03000000>;
>                 };
>
>                 /* global autoconfigured region for contiguous allocations
> */
>                 linux,cma@57000000 {
>                         compatible = "shared-dma-pool";
>                         reusable;
>                         reg = <0x00000000 0x57000000 0x0 0x19000000>;
>                         linux,cma-default;
>                 };
>
>                 /* device specific region for contiguous allocations */
>                 mmp_reserved: linux,multimedia@70000000 {
>                         compatible = "shared-dma-pool";
>                         reusable;
>                         reg = <0x00000000 0x70000000 0x0 0x10000000>;
>                 };
>         };
>
>         chosen {
>                 stdout-path = "serial0:115200n8";
>                 xen,xen-bootargs = "dom0_mem=752M console=dtuart
> dtuart=serial0 dom0_max_vcpus=4";
>                 xen,dom0-bootargs = "console=hvc0 clk_ignore_unused
> root=/dev/mmcblk1p2 rw rootwait ignore_loglevel cma=128M";
>
>                 #address-cells = <2>;
>                 #size-cells = <2>;
>                 modules {
>                         module@0 {
>                                 compatible = "xen,linux-zimage",
> "xen,multiboot-module";
>                                 reg = <0x0 0x7a000000 0x0 0x02000000>;
>                         };
>                 };
>         };
>
>         mmngr {
>                 compatible = "renesas,mmngr";
>                 memory-region = <&mmp_reserved>, <&lossy_decompress>;
>         };
>
>         mmngrbuf {
>                 compatible = "renesas,mmngrbuf";
>         };
>
>         vspm_if {
>                 compatible = "renesas,vspm_if";
>         };
>
>         versaclock5_out3: versaclk-3 {
>                 compatible = "fixed-clock";
>                 #clock-cells = <0>;
>                 /* Initial value of versaclock out3 */
>                 clock-frequency = <33000000>;
>         };
> };
>
> &a53_0 {
>         /delete-property/ cpu-idle-states;
> };
>
> &a53_1 {
>         /delete-property/ cpu-idle-states;
> };
>
> &a53_2 {
>         /delete-property/ cpu-idle-states;
> };
>
> &a53_3 {
>         /delete-property/ cpu-idle-states;
> };
>
> &du {
>         clocks = <&cpg CPG_MOD 724>,
>                  <&cpg CPG_MOD 723>,
>                  <&cpg CPG_MOD 722>,
>                  <&versaclock5 1>,
>                  <&versaclock5_out3>,
>                  <&versaclock5 2>;
>         clock-names = "du.0", "du.1", "du.2",
>                       "dclkin.0", "dclkin.1", "dclkin.2";
> };
>
> &vspb {
>         status = "okay";
> };
>
> &vspi0 {
>         status = "okay";
> };
>
> With the new information, I finally got some output for Xen, but was met
> with a completely different issue for which I can't find any information
> for the error I am receiving.
>
> I realize that I may have an issue with the version of the board I am
> using, but a simple online search doesn't throw any red flags. The serial
> console output is below:
>
> [    0.000177] NOTICE:  BL2: R-Car M3 Initial Program Loader(CA57)
> [    0.004613] NOTICE:  BL2: Initial Program Loader(Rev.2.0.0)
> [    0.010147] NOTICE:  BL2: PRR is R-Car M3 Ver.1.0
> [    0.014815] NOTICE:  BL2: Board is Starter Kit Rev.1.0
> [    0.019928] NOTICE:  BL2: Boot device is HyperFlash(80MHz)
> [    0.025354] NOTICE:  BL2: LCM state is CM
> [    0.029395] NOTICE:  BL2: AVS setting succeeded. DVFS_SetVID=0x53
> [    0.035383] NOTICE:  BL2: DDR3200(rev.0.34)
> [    0.046779] NOTICE:  BL2: [COLD_BOOT]
> [    0.085117] NOTICE:  BL2: DRAM Split is 2ch
> [    0.087812] NOTICE:  BL2: QoS is default setting(rev.0.19)
> [    0.093315] NOTICE:  BL2: Lossy Decomp areas
> [    0.097489] NOTICE:       Entry 0: DCMPAREACRAx:0x80000540
> DCMPAREACRBx:0x570
> [    0.104573] NOTICE:       Entry 1: DCMPAREACRAx:0x40000000
> DCMPAREACRBx:0x0
> [    0.111485] NOTICE:       Entry 2: DCMPAREACRAx:0x20000000
> DCMPAREACRBx:0x0
> [    0.118400] NOTICE:  BL2: v1.5(release):556a11e
> [    0.122890] NOTICE:  BL2: Built : 18:03:48, Jan 11 2019
> [    0.128077] NOTICE:  BL2: Normal boot
> [    0.131718] NOTICE:  BL2: dst=0xe6321100 src=0x8180000 len=512(0x200)
> [    0.138106] NOTICE:  BL2: dst=0x43f00000 src=0x8180400 len=6144(0x1800)
> [    0.144726] NOTICE:  BL2: dst=0x44000000 src=0x81c0000
> len=65536(0x10000)
> [    0.151952] NOTICE:  BL2: dst=0x44100000 src=0x8200000
> len=1048576(0x100000)
> [    0.166777] NOTICE:  BL2: dst=0x50000000 src=0x8640000
> len=1048576(0x100000)
> [    0.180752] NOTICE:  BL2: Booting BL31
>
>
> U-Boot 2018.09 (Jan 12 2019 - 03:03:43 +0900)
>
> CPU: Renesas Electronics R8A7796 rev 1.0
> Model: Renesas M3ULCB board based on r8a7796
> DRAM:  1.9 GiB
> Bank #0: 0x048000000 - 0x07fffffff, 896 MiB
> Bank #1: 0x600000000 - 0x63fffffff, 1 GiB
>
> MMC:   sd@ee100000: 0, sd@ee140000: 1
> Loading Environment from MMC... communication out of sync
> communication out of sync
> communication out of sync
> communication out of sync
> communication out of sync
> communication out of sync
> communication out of sync
> communication out of sync
> communication out of sync
> communication out of sync
> communication out of sync
> communication out of sync
> OK
> In:    serial@e6e88000
> Out:   serial@e6e88000
> Err:   serial@e6e88000
> Net:   eth0: ethernet@e6800000
> Hit any key to stop autoboot:  0
> ethernet@e6800000 Waiting for PHY auto negotiation to complete... done
> Using ethernet@e6800000 device
> TFTP from server 192.168.1.100; our IP address is 192.168.1.3
> Filename 'Image'.
> Load address: 0x48080000
> Loading: #################################################################
>          #################################################################
>          #################################################################
>          #################################################################
>          #################################################################
>          #################################################################
>          #################################################################
>          #################################################################
>          #################################################################
>          #################################################################
>          #################################################################
>          #################################################################
>          #################################################################
>          #################################################################
>          #################################################################
>          #################################################################
>          ############################################
>          2.7 MiB/s
> done
> Bytes transferred = 15911424 (f2ca00 hex)
> Using ethernet@e6800000 device
> TFTP from server 192.168.1.100; our IP address is 192.168.1.3
> Filename 'Image-r8a7796-m3ulcb.dtb'.
> Load address: 0x48000000
> Loading: *
> TFTP error: 'File not found' (1)
> Not retrying...
> ERROR: Did not find a cmdline Flattened Device Tree
> Could not find a valid device tree
> => setenv bootargs
> => tftp 0x48080000 xen-m3ulcb.uImage
> Using ethernet@e6800000
> device
> TFTP from server 192.168.1.100; our IP address is
> 192.168.1.3
> Filename
> 'xen-m3ulcb.uImage'.
> Load address:
> 0x48080000
> Loading:
> ###############################################################
>          28.3
> MiB/s
> done
>
> Bytes transferred = 918928 (e0590
> hex)
> => tftp 0x48000000
> Image-r8a7796-m3ulcb-xen-local.dtb
> Using ethernet@e6800000
> device
> TFTP from server 192.168.1.100; our IP address is
> 192.168.1.3
> Filename
> 'Image-r8a7796-m3ulcb-xen-local.dtb'.
> Load address:
> 0x48000000
> Loading:
> #####
>          21.8
> MiB/s
> done
>
> Bytes transferred = 68545 (10bc1
> hex)
> => tftp 0x7a000000
> Image-m3ulcb.bin
> Using ethernet@e6800000
> device
> TFTP from server 192.168.1.100; our IP address is 192.168.1.3
> Filename 'Image-m3ulcb.bin'.
> Load address: 0x7a000000
> Loading: #################################################################
>          #################################################################
>          #################################################################
>          #################################################################
>          #################################################################
>          #################################################################
>          #################################################################
>          #################################################################
>          #################################################################
>          #################################################################
>          #################################################################
>          #################################################################
>          #################################################################
>          #################################################################
>          #################################################################
>          #################################################################
>          #################################################################
>          #################################################################
>          #################################################################
>          ################################################################
>          28.1 MiB/s
> done
> Bytes transferred = 19059200 (122d200 hex)
> => bootm 0x48080000 - 0x48000000
> ## Booting kernel from Legacy Image at 48080000 ...
>    Image Name:   XEN
>    Image Type:   AArch64 Linux Kernel Image (uncompressed)
>    Data Size:    918864 Bytes = 897.3 KiB
>    Load Address: 78080000
>    Entry Point:  78080000
>    Verifying Checksum ... OK
> ## Flattened Device Tree blob at 48000000
>    Booting using the fdt blob at 0x48000000
>    Loading Kernel Image ... OK
>    Using Device Tree in place at 0000000048000000, end 0000000048013bc0
>
> Starting kernel ...
>
>  Xen 4.12-unstable
> (XEN) Xen version 4.12-unstable (jelfn@waseda.jp) (aarch64-poky-linux-gc9
> (XEN) Latest ChangeSet: Mon Dec 17 09:22:59 2018 +0000 git:a5b0eb3-dirty
> (XEN) Processor: 411fd073: "ARM Limited", variant: 0x1, part 0xd07, rev 0x3
> (XEN) 64-bit Execution:
> (XEN)   Processor Features: 0000000000002222 0000000000000000
> (XEN)     Exception Levels: EL3:64+32 EL2:64+32 EL1:64+32 EL0:64+32
> (XEN)     Extensions: FloatingPoint AdvancedSIMD
> (XEN)   Debug Features: 0000000010305106 0000000000000000
> (XEN)   Auxiliary Features: 0000000000000000 0000000000000000
> (XEN)   Memory Model Features: 0000000000001124 0000000000000000
> (XEN)   ISA Features:  0000000000011120 0000000000000000
> (XEN) 32-bit Execution:
> (XEN)   Processor Features: 00000131:00011011
> (XEN)     Instruction Sets: AArch32 A32 Thumb Thumb-2 Jazelle
> (XEN)     Extensions: GenericTimer Security
> (XEN)   Debug Features: 03010066
> (XEN)   Auxiliary Features: 00000000
> (XEN)   Memory Model Features: 10201105 40000000 01260000 02102211
> (XEN)  ISA Features: 02101110 13112111 21232042 01112131 00011142 00011121
> (XEN) Using SMC Calling Convention v1.1
> (XEN) Using PSCI v1.1
> (XEN) SMP: Allowing 6 CPUs
> (XEN)
> (XEN) ****************************************
> (XEN) Panic on CPU 0:
> (XEN) Timer: Unable to retrieve IRQ 0 from the device tree
> (XEN) ****************************************
> (XEN)
> (XEN) Reboot in five seconds...
>
> Is there anything I have overlooked in the compilation or configuration
> for Xen?
>
> Thank you very much for all the help you have offered up to this point. I
> am at least happy to get some output, even if it is on the R-Car M3 board
> in my possession.
>
> Best regards,
>
> Jairo
>
>
> 2019年1月9日(水) 2:04 LOPEZ, FUENTES NACARINO Jairo Eduardo <
> jairo@ruri.waseda.jp>:
>
>> Andrii,
>>
>> I have attempted to update the bootloader from the srec files created by
>> the bitbake command.
>>
>> I believe I have been able to update it, but I am still obtaining the
>> same results. The serial output is shown below:
>>
>> [    0.000161] NOTICE:  BL2: R-Car Gen3 Initial Program Loader(CA57)
>> Rev.1.0.17
>> [    0.005721] NOTICE:  BL2: PRR is R-Car H3 Ver2.0
>> [    0.010304] NOTICE:  BL2: Board is Starter Kit Rev1.0
>> [    0.015331] NOTICE:  BL2: Boot device is HyperFlash(80MHz)
>> [    0.020757] NOTICE:  BL2: LCM state is CM
>> [    0.024802] NOTICE:  BL2: AVS setting succeeded. DVFS_SetVID=0x53
>> [    0.030786] NOTICE:  BL2: DDR3200(rev.0.28)NOTICE:
>> [COLD_BOOT]NOTICE:  ..0
>> [    0.059996] NOTICE:  BL2: DRAM Split is 4ch
>> [    0.063882] NOTICE:  BL2: QoS is default setting(rev.0.18)
>> [    0.069381] NOTICE:  BL2: Lossy Decomp areas
>> [    0.073559] NOTICE:       Entry 0: DCMPAREACRAx:0x80000540
>> DCMPAREACRBx:0x570
>> [    0.080643] NOTICE:       Entry 1: DCMPAREACRAx:0x40000000
>> DCMPAREACRBx:0x0
>> [    0.087555] NOTICE:       Entry 2: DCMPAREACRAx:0x20000000
>> DCMPAREACRBx:0x0
>> [    0.094470] NOTICE:  BL2: v1.3(release):b15fefa
>> [    0.098960] NOTICE:  BL2: Built : 12:21:32, Dec 28 2018
>> [    0.104147] NOTICE:  BL2: Normal boot
>> [    0.107788] NOTICE:  BL2: dst=0xe6322210 src=0x8180000 len=512(0x200)
>> [    0.114335] NOTICE:  BL2: dst=0x43f00000 src=0x8180400 len=6144(0x1800)
>> [    0.120797] NOTICE:  BL2: dst=0x44000000 src=0x81c0000
>> len=65536(0x10000)
>> [    0.128022] NOTICE:  BL2: dst=0x44100000 src=0x8200000
>> len=524288(0x80000)
>> [    0.138503] NOTICE:  BL2: dst=0x50000000 src=0x8640000
>> len=1048576(0x100000)
>>
>>
>> U-Boot 2015.04 (Dec 28 2018 - 02:46:24)
>>
>> CPU: Renesas Electronics R8A7795 rev 2.0
>> Board: H3ULCB
>> I2C:   ready
>> DRAM:  3.9 GiB
>> MMC:   sh-sdhi: 0, sh-sdhi: 1
>> In:    serial
>> Out:   serial
>> Err:   serial
>> Net:   ravb
>> Hit any key to stop autoboot:  0
>> => setenv bootargs
>> => setenv serverip 192.168.1.100
>> => tftp 0x48080000 xen-h3ulcb.uImage
>> ravb Waiting for PHY auto negotiation to complete......... TIMEOUT !
>> Half
>> Using ravb device
>> TFTP from server 192.168.1.100; our IP address is 192.168.1.40
>> Filename 'xen-h3ulcb.uImage'.
>> Load address: 0x48080000
>> Loading: *
>> ARP Retry count exceeded; starting again
>> ravb:0 is connected to ravb.  Reconnecting to ravb
>> ravb Waiting for PHY auto negotiation to complete......... TIMEOUT !
>> Half
>>
>> ARP Retry count exceeded; starting again
>> ravb:0 is connected to ravb.  Reconnecting to ravb
>> ravb Waiting for PHY auto negotiation to complete... done
>> ravb: 1000Base/Full
>> Using ravb device
>> TFTP from server 192.168.1.100; our IP address is 192.168.1.40
>> Filename 'xen-h3ulcb.uImage'.
>> Load address: 0x48080000
>> Loading: ###########################################################
>>          25.4 MiB/s
>> done
>> Bytes transferred = 852368 (d0190 hex)
>> => tftp 0x48000000 Image-r8a7795-h3ulcb-xen-local.dtb
>> ravb:0 is connected to ravb.  Reconnecting to ravb
>> ravb Waiting for PHY auto negotiation to complete..... done
>> ravb: 1000Base/Full
>> Using ravb device
>> TFTP from server 192.168.1.100; our IP address is 192.168.1.40
>> Filename 'Image-r8a7795-h3ulcb-xen-local.dtb'.
>> Load address: 0x48000000
>> Loading: #####
>>          11.7 KiB/s
>> done
>> Bytes transferred = 63778 (f922 hex)
>> => tftp 0x7a000000 h3ulcb-image
>> ravb:0 is connected to ravb.  Reconnecting to ravb
>> ravb Waiting for PHY auto negotiation to complete... done
>> ravb: 1000Base/Full
>> Using ravb device
>> TFTP from server 192.168.1.100; our IP address is 192.168.1.40
>> Filename 'h3ulcb-image'.
>> Load address: 0x7a000000
>> Loading: #################################################################
>>          #################################################################
>>          #################################################################
>>          #################################################################
>>          #################################################################
>>          #################################################################
>>          #################################################################
>>          #################################################################
>>          #################################################################
>>          #################################################################
>>          #################################################################
>>          #################################################################
>>          #################################################################
>>          #################################################################
>>          #################################################################
>>          #################################################################
>>          ########################################
>>          2.1 MiB/s
>> done
>> Bytes transferred = 15843840 (f1c200 hex)
>> => bootm 0x48080000 - 0x48000000
>> ## Booting kernel from Legacy Image at 48080000 ...
>>    Image Name:   XEN
>>    Image Type:   AArch64 Linux Kernel Image (uncompressed)
>>    Data Size:    852304 Bytes = 832.3 KiB
>>    Load Address: 78080000
>>    Entry Point:  78080000
>>    Verifying Checksum ... OK
>> ## Flattened Device Tree blob at 48000000
>>    Booting using the fdt blob at 0x48000000
>>    Loading Kernel Image ... OK
>>    Using Device Tree in place at 0000000048000000, end 0000000048012921
>>
>> Starting kernel ...
>>
>> </end output>
>>
>> I have noticed that the uboot date has not changed from 2015.04 although
>> the build date is from the last time I ran a bitbake command.
>>
>> I do not know what version of uboot should be used without breaking the
>> current compiling versions.
>>
>> Are there any other modifications that I can attempt?
>>
>> Thank you very much for your time and patience.
>>
>> Best regards,
>>
>> Jairo
>>
>>
>>
>>

[-- Attachment #1.2: Type: text/html, Size: 34809 bytes --]

[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: RT Xen on ARM - R-Car series

[Andrii Anisov]

Hello Jairo,

On 08.01.19 19:04, LOPEZ, FUENTES NACARINO Jairo Eduardo wrote:

> I have noticed that the uboot date has not changed from 2015.04
It is not a u-boot date, but upstream version. Renesas derived u-boot from mainline version 2015.04 and put their patches on top of it.

> although the build date is from the last time I ran a bitbake command.
So it should be ok.

> Are there any other modifications that I can attempt?
I'm pretty sure you faced the issue, which is discussed here [1].
You should merge memory nodes in your device tree, as it is done here [2].

Actually the proper way to sort out the root cause is to build XEN with debug and earlyprintk enabled. But it is quite tricky while building XEN using Yocto.
You should pass build command line options for debug build and earlyprintk, but in Yocto command lines are buried in recipes.

BTW, I've got an idea that it would be good to make debug build option as well as earlyprintk configurable through .config only.

[1] https://lists.xenproject.org/archives/html/xen-devel/2017-07/msg02479.html
[2] https://github.com/xen-troops/meta-demo/blob/master/meta-rcar-gen3-xen/recipes-kernel/linux/linux-renesas/r8a7795-salvator-x-xen.dts#L61

-- 
Sincerely,
Andrii Anisov.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: RT Xen on ARM - R-Car series

[LOPEZ, FUENTES NACARINO Jairo Eduardo]

[-- Attachment #1.1: Type: text/plain, Size: 2158 bytes --]

Andrii,

2019年1月16日(水) 16:40、Andrii Anisov さん（andrii.anisov@gmail.com）のメッセージ:

> Hello Jairo,
>
> On 08.01.19 19:04, LOPEZ, FUENTES NACARINO Jairo Eduardo wrote:
>
> > I have noticed that the uboot date has not changed from 2015.04
> It is not a u-boot date, but upstream version. Renesas derived u-boot from
> mainline version 2015.04 and put their patches on top of it.
>
> > although the build date is from the last time I ran a bitbake command.
> So it should be ok.
>
> > Are there any other modifications that I can attempt?
> I'm pretty sure you faced the issue, which is discussed here [1].
> You should merge memory nodes in your device tree, as it is done here [2].
>
> Actually the proper way to sort out the root cause is to build XEN with
> debug and earlyprintk enabled. But it is quite tricky while building XEN
> using Yocto.
> You should pass build command line options for debug build and
> earlyprintk, but in Yocto command lines are buried in recipes.
>
> BTW, I've got an idea that it would be good to make debug build option as
> well as earlyprintk configurable through .config only.
>
> [1]
> https://lists.xenproject.org/archives/html/xen-devel/2017-07/msg02479.html
> [2]
> https://github.com/xen-troops/meta-demo/blob/master/meta-rcar-gen3-xen/recipes-kernel/linux/linux-renesas/r8a7795-salvator-x-xen.dts#L61
>
> --
> Sincerely,
> Andrii Anisov.
>

When I sent that mail I was using the Yocto 2.23 instructions.

Since I didn't get any new information and since I had apparently been able
to safely upgrade the firmware, I attempted to up the version of everything.

I understand that the current yocto environment might be unfriendly to
enable debugging information but I am willing to help in implementing it if
you have an idea. There is nothing more frustrating than being met with a
completely blank output.

When using the Yocto 2.23 instructions I did not come across the issues you
pointed out, so I will take those into consideration should I need to
downgrade the versions again.

Thanks for all the help.

Best regards,

Jairo

>

[-- Attachment #1.2: Type: text/html, Size: 3258 bytes --]

[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: RT Xen on ARM - R-Car series

[LOPEZ, FUENTES NACARINO Jairo Eduardo]

[-- Attachment #1.1: Type: text/plain, Size: 11295 bytes --]

Hello Andrii and everyone still listening,

I am sorry for sending attachments to the mailing list. Below is my github
with the patch I applied and the dts I made to attempt to boot the R-Car H3.

https://github.com/nacarino/xen-arm

As Andrii suggested, I attempted only compiling Xen with the SRCREV
24fb44e971a62b345c7b6ca3c03b454a1e150abe.

The compilation went smoothly but I was unable to get any output from the
R-Car H3 past the "Starting kernel" line.

I have a feeling that my configuration of the serial connection or the
serial configuration somewhere is wrong, however I don't have a clue where
I could start debugging.

Does any one have any ideas?

Thank you all for your patience and cooperation.

Best regards,

Jairo

2018年12月29日(土) 0:22 LOPEZ, FUENTES NACARINO Jairo Eduardo <
jairo@ruri.waseda.jp>:

> Hello Andrii,
>
> Thanks again for responding and for clarifying some of the underlying
> workings of Yocto.
>
> 2018年12月27日(木) 20:07 Andrii Anisov <andrii.anisov@gmail.com>:
>
>> Hello Jairo,
>>
>> On 25.12.18 18:07, LOPEZ, FUENTES NACARINO Jairo Eduardo wrote:
>> > I believe this is the SoC information. If there is any other method of
>> extracting the information, please let me know so I can transmit it.
>>
>> That output gives a full set of required information. Actually, I worried
>> you
>> might have an obsolete SoC revision. But it is not your case.
>>
>
> I am actually very relieved that this is the case.
>
>
>> > I took a look at [1] and decided to start from scratch to attempt to
>> get the minimum workspace functioning.
>>
>> I mainly mentioned point 1 from those limitations. But you have H3 ES2.0
>> so
>> ready for the latest BSP as well.
>>
>> > In previous attempts, I had to modify some recipes to get the
>> compilation working, but this time I would like to confirm with everyone
>> the initial steps before I take them.
>> > ERROR: xen-unstable+gitAUTOINC+9d357cbaf7-r0 do_package: QA Issue: xen:
>> Files/directories were installed but not shipped in any package:
>> >    /usr/lib/libxenfsimage.so
>> >    /usr/lib/libxenfsimage.so.4.12
>> >    /usr/lib/libxenfsimage.so.4.12.0
>> >    /usr/lib/xenfsimage/ext2fs-lib/fsimage.so
>> >    /usr/lib/xenfsimage/ufs/fsimage.so
>> >    /usr/lib/xenfsimage/fat/fsimage.so
>> >    /usr/lib/xenfsimage/iso9660/fsimage.so
>> >    /usr/lib/xenfsimage/reiserfs/fsimage.so
>> >    /usr/lib/xenfsimage/zfs/fsimage.so
>> >    /usr/lib/xen/bin/depriv-fd-checker
>> >    /usr/sbin/xenmon
>> > Please set FILES such that these items are packaged. Alternatively if
>> they are unneeded, avoid installing them or delete them within do_install.
>> > xen: 11 installed and not shipped files. [installed-vs-shipped]
>> > ERROR: xen-unstable+gitAUTOINC+9d357cbaf7-r0 do_package: Fatal QA
>> errors found, failing task.
>> > ERROR: xen-unstable+gitAUTOINC+9d357cbaf7-r0 do_package: Function
>> failed: do_package
>> > ERROR: Logfile of failure stored in:
>> /home/yocto/r-car/build/tmp/work/aarch64-poky-linux/xen/unstable+gitAUTOINC+9d357cbaf7-r0/temp/log.do_package.8954
>> > ERROR: Task 329
>> (/home/yocto/r-car/meta-virtualization/recipes-extended/xen/xen_git.bb <
>> http://xen_git.bb>, do_package) failed with exit code '1'
>> > NOTE: Tasks Summary: Attempted 3642 tasks of which 3641 didn't need to
>> be rerun and 1 failed.
>> > No currently running tasks (2517 of 3653)
>> >
>> > Summary: 1 task failed:
>> >    /home/yocto/r-car/meta-virtualization/recipes-extended/xen/
>> xen_git.bb <http://xen_git.bb>, do_package
>> > Summary: There were 3 WARNING messages shown.
>> > Summary: There were 3 ERROR messages shown, returning a non-zero exit
>> code.
>>
>> It's a known issue. Let's say Yocto's specifics. XEN does evolve so its
>> tools
>> set of libs and apps is being changed. But Yocto tracks all products of
>> compilation and eager to know what to do with each of those files.
>>
>> > I am aware I am using a very old BSP. If there is a slightly better
>> version with which to start with, I would greatly like everyone's opinion.
>>
>> It is mainly not because of BSP, but the meta-virtualization layer
>> version, it
>> describes how to build and install XEN. As you can see in the last patch
>> to
>> xen_git.bbappend [1] it is adjusted for 4.10-rc1. I suppose you already
>> did
>> required changes for the current 4.12-unstable version when saying:
>>
>> > In previous attempts, I had to modify some recipes to get the
>> compilation working
>>
>> Good, you have the BSP with XEN built. Could you please reveal your
>> changes to
>> let me know which XEN you actually built?
>
>
>
> In order to get Xen to compile and to prepare for the boot, I modified the
> meta-demo layer to include the Xen files that were not already included and
> the hand modified dts file. I have attempted to attach these to the mail.
>
> I am not sure if there is an easier way to get that information, but I
> basically did a git show within the tmp/work/aarch64-poky-linux/xen folder
> and got the hash 7f28661f6a7ce3d82f881b9afedfebca7f2cf116 which points to
> the current head of the master branch of the xen.git repository, if I am
> not mistaken.
>
> In the previous email you said, you
>> tried running freshly built BSP with XEN, and it does not show anything to
>> display. But what about the console output for that case?
>>
>>
> Via serial console I get the following output for the images created after
> modifying the dts and applying the patch below:
>
> U-Boot 2015.04 (Jun 22 2018 - 13:36:27)
>
> CPU: Renesas Electronics R8A7795 rev 2.0
> Board: H3ULCB
> I2C:   ready
> DRAM:  3.9 GiB
> MMC:   sh-sdhi: 0, sh-sdhi: 1
> In:    serial
> Out:   serial
> Err:   serial
> Net:   ravb
> Hit any key to stop autoboot:  0
> => setenv bootargs
> => setenv serverip 192.168.1.100
> => tftp 0x48080000 xen-h3ulcb.uImage
> ravb Waiting for PHY auto negotiation to complete...... done
> ravb: 1000Base/Full
> Using ravb device
> TFTP from server 192.168.1.100; our IP address is 192.168.1.40
> Filename 'xen-h3ulcb.uImage'.
> Load address: 0x48080000
> Loading: #############################################################
>          171.9 KiB/s
> done
> Bytes transferred = 886160 (d8590 hex)
> => tftp 0x48000000 Image-r8a7795-h3ulcb-xen-local.dtb
> ravb:0 is connected to ravb.  Reconnecting to ravb
> ravb Waiting for PHY auto negotiation to complete... done
> ravb: 1000Base/Full
> Using ravb device
> TFTP from server 192.168.1.100; our IP address is 192.168.1.40
> Filename 'Image-r8a7795-h3ulcb-xen-local.dtb'.
> Load address: 0x48000000
> Loading: #####
>          11.7 KiB/s
> done
> Bytes transferred = 63810 (f942 hex)
> => tftp 0x7a000000 Image
> ravb:0 is connected to ravb.  Reconnecting to ravb
> ravb Waiting for PHY auto negotiation to complete........ done
> ravb: 1000Base/Full
> Using ravb device
> TFTP from server 192.168.1.100; our IP address is 192.168.1.40
> Filename 'Image'.
> Load address: 0x7a000000
> Loading: #################################################################
>          #################################################################
>          #################################################################
>          #################################################################
>          #################################################################
>          #################################################################
>          #################################################################
>          #################################################################
>          #################################################################
>          #################################################################
>          #################################################################
>          #################################################################
>          #################################################################
>          #################################################################
>          #################################################################
>          #################################################################
>          ############################################
>          2.1 MiB/s
> done
> Bytes transferred = 15911424 (f2ca00 hex)
> => bootm 0x48080000 - 0x48000000
> ## Booting kernel from Legacy Image at 48080000 ...
>    Image Name:   XEN
>    Image Type:   AArch64 Linux Kernel Image (uncompressed)
>    Data Size:    886096 Bytes = 865.3 KiB
>    Load Address: 78080000
>    Entry Point:  78080000
>    Verifying Checksum ... OK
> ## Flattened Device Tree blob at 48000000
>    Booting using the fdt blob at 0x48000000
>    Loading Kernel Image ... OK
>    Using Device Tree in place at 0000000048000000, end 0000000048012941
>
> Starting kernel ...
>
>
> And that is it.
> Please note that although I am getting the images and dtb from tftp, the
> root filesystem is on the microSD card.
>
> I am guessing I should have seen the console output for Xen. I also am not
> sure how to test if dom0 is actually running at this stage.
>
>
>> >  From the log it would seem that the xen_git.bb <http://xen_git.bb> in
>> the meta-virtualization layer is being called and thus the recipe is
>> attempting to compile the newest version of Xen.
>>
>> Right you are. That is the idea. But Yocto's way of BSP compilation makes
>> it
>> tending to break up. You can edit XEN recipe to build it from a specific
>> revision, e.g. 4.10.0-rc1. You should replace `${AUTOREV}` in this line
>> [2]
>> with the correspondent commit-id
>> `24fb44e971a62b345c7b6ca3c03b454a1e150abe` to
>> do so.
>> But I suppose it is not what you really need. Since 4.10.0-rc1 there are a
>> number of changes to scheduling. You might need have those bits up to
>> date for
>> your work.
>>
>
>> > So my second question would be, what version of Xen should I point
>> towards for the board I am using?I guess it is better to use the latest and
>> greatest for your work, so XEN 4.12 unstable should suit you.
>>
>>
> Yes, I have been tracking some of the scheduling changes being done to Xen
> and I would really like to have a relatively newer version up and running.
> However, just having something, anything, running at this stage would be
> great.
>
> Also I do understand that we have our meta-demo layer quite outdated both
>> from
>> XEN and BSP sides. Renesas's 2.x BSPs are baked with Linux kernel 4.9.x,
>> it is
>> really old. Even with LK 4.14, we are using from BSP 3.9, I faced an
>> issue [3]
>> while playing with the latest and greatest XEN.
>>
>> [1]
>> https://github.com/xen-troops/meta-demo/commit/a4178158ca3ebb739c9bc71c517ec7b65f563218
>> [2]
>> https://github.com/xen-troops/meta-demo/blob/master/meta-rcar-gen3-xen/recipes-extended/xen/xen_git.bbappend#L9
>> [3]
>> https://lists.xenproject.org/archives/html/xen-devel/2018-12/msg01373.html
>>
>> --
>> Sincerely,
>> Andrii Anisov.
>>
>
> Thank you very much for all your support.
>
> Best regards,
>
> Jairo
>

[-- Attachment #1.2: Type: text/html, Size: 14402 bytes --]

[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel